Co-Reinforcement Learning for Unified Multimodal Understanding and Generation
Authors: Jingjing Jiang
Chongjie Si
Jun Luo
Hanwang Zhang
Chao Ma
Venue: NeurIPS 2025 spotlight ()
Keywords: Reinforcement Learning
GRPO
Unified Multimodal Understanding and Generation
Primary area: applications

Abstract: This paper presents a pioneering exploration of reinforcement learning (RL) via group relative policy optimization for unified multimodal large language models (ULMs), aimed at simultaneously reinforcing generation and understanding capabilities. Through systematic pilot studies, we uncover the significant potential of ULMs to enable the synergistic co-evolution of dual capabilities within a shared policy optimization framework. Building on this insight, we introduce \textbf{CoRL}, a \textbf{Co}-\textbf{R}einforcement \textbf{L}earning framework comprising a unified RL stage for joint optimization and a refined RL stage for task-specific enhancement. With the proposed CoRL, our resulting model, \textbf{ULM-R1}, achieves average improvements of 7\% on three text-to-image generation datasets and 23\% on nine multimodal understanding benchmarks. These results demonstrate the effectiveness of CoRL and highlight the substantial benefits of reinforcement learning in facilitating cross-task synergy and optimization for ULMs. Code is available at \url{https://github.com/mm-vl/ULM-R1}.

Co-Reinforcement Learning for Unified Multimodal Understanding and Generation
Jingjing Jiang!
Chongjie Si!
Jun Luo²
Hanwang Zhang?
Chao Ma
Shanghai Jiao Tong University
2 Nanyang Technological University
jingjinggjangg21770mmaaa com, {chongjiesi, (ajaomaaajju.... edu.
(junluo, hanwangzhang) Untu. edu.
Abstract
This paper presents a pioneering exploration of reinforcement learning (RL.) via group relative policy optimization for unified multimodal large language mod- els (ULMs), aimed at simultancously reinforcing generation and understanding capabilities. Through systematic pilot studies, uncover the significant poten- tial of ULMs to enable the synergistic co-evolution of dual capabilities within shared policy optimization framework. Building on this insight, we introduce CoRL, a Co- Co-Reinforcement Learning framework comprising unified RL stage for joint optimization and refined RL stage for tasc-sp enhancement. With the proposed CoRL., our resulting model, ULM-RR1, achieves average improve- ments of 7% on three eext-to-image generation datasets and 23% on nine multi- modal understanding benchmarks. These results demonstrate the effectiveness CoRL and highlight the substantial benefits reinforcement learning in fa- cilitating cross-task synergy and optimization for ULMs. Code is available at https: https:///ghhubt com/mmmm//ooooo wwwmmm/UL-------
Introduction
As large foundation models (LFMs) continue advance in their general capabilities and breadth of knowledge, post-training [30, 46, 70, 73, 108] has emerged as a critical paradigm for further refining pretrained LFMss toward specialized applications, thereby facilitating task adaptation and human-aligned behaviors. Recently, reinforcement learning (RI.)-aseed approaches [51, 52, 59, 64, 69, 72, 95] have exhibited considerable promise due their data efficiency and strong alignment abilities. A notable exemplar is DeepSeek-RI [19], which demonstrates that RL with verifiable rewards and the group relative policy optimization (GRPO) algorithm constitutes practical and stable strategy that sidesteps explicit preference modeling [77] and reward model learning [83]. This promising paradigm indicates the significant potential of LFMs to acquire advanced capabilities and generalize effectively without dependence on large-scale, high high-quality supervised data.
In the multimodal Al research community, the prevailing implementation [9, 24, 40, 41, 53, 69, 100, 101] of the GRPO algorithm centers on crafting diverse rule-based reward mechanisms to incentivize long-chain reasoning capabilities of multimodal large language models (MILMss). These initiatives primarily target multimodal understanding, with a particular focus on visual and mathematical reasoning tasks. Conversely, its application to visual generation remains surprisingly limited, with only pioneering explorations [25, 79] suggesting its feasibility. More importantly, extending GRPO unified MILMs (ULMs) [8, 39, 88, 90] capable of concurrently performing visual understanding and generation tasks remains considerably under-explored. Intuitively, ULMs could significantly
"Corresponinng author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

benefit from GRPO owing to their inherent advantages of cross-task synergy and LLM sharing, which enables ULMs to share reward signals across various tasks and effectively mitigate reward imbalance, particularly as GRPO operates by jointly ranking outputs within task-agnostic groups.
This paper aims enhance the understanding and generation capabilities of ULMs without relying on supervised data. We begin with a set of pilot experiments to explore efficient reinforcement learning paradigms. Specifically, we systematically examine four rule-based training strategies: (i) separate RL for individual tasks, (ii) separate RL with weight merging, (iii) cycle RL alternating between tasks, and (iv) unified RL with joint optimization. Our explorations reveal two critical findings. First, direct task task-specific RL fails to achieve the anticipated improvements, particularly in visual generation, and even impairs other abilities. Second, compared with alternative strategies. unified RL showcases comprehensive advantages across tasks. These results demonstrate the synergistic co-evolution of dual capabilities under a shared policy optimization paradigm.
In light of our preliminary findings, we propose CoRL, a co-reinforcement learning framework de- signed synergistically improve the understanding and generation capabilities of ULMs. Specifically, CoRL follows a canittioo-then-S-pcaaiziiue
paradigm and is implemented through a two-stage RL procedure: a unified RL stage for joint optimization of dual capabilities and refined RL stage for task-specific enhancement. In the first stage, the policy ULM is optimized through a unified GRPO algorithm with diverse rewards on a carefully curated dataset spanning both understanding and generation tasks. To effectively guide policy optimization in visual generation, we introduce a bidirectional cycle consistency reward and text-image matching reward, which together promote semantic consistency and faithfulness between synthesized images and their corresponding prompts. The designed rewards complement typical multimodal understanding rewards (ie.. accuracy and format) within a unified group, enabling cross-task joint optimization. In the subsequent stage, we independently reinforce the policy's s understanding and generation capabilities using respective rewards and tailored datasets for task-specific refinement.
Applying the two-stage CoRL procedure to the baseline ULM Janus-Pro [8] yields ULM-R1, unified model with reinforced capabilities in both understanding and generation. To comprehensively assess its performance, we conduct extensive comparisons against of unified MLLMs and dedicated models across both three visual generation and nine multimodal understanding benchmarks. Notably, ULM-R1 achieves substantial gains over its baseline on complex mathematical and logical reasoning tasks, such as WeMath (+15.2) and Logic Vista (+10.6) These results underscore the effectiveness of CoRL, providing compelling empirical evidence for the efficacy of RL in jointly advancing visual understanding and generation tasks.
We summarize our main contributions follows:
We establish that RL with verifiable rewards and GRPO constitutes a data-efficient paradigm for cross-task co-optimization and capability enhancement.
We introduce a co-reinforcement learning framework. CoRL, to synergistically enhance the dual capabilities of ULMss using miifeed-then-refiee RL paradigm.
We demonstrate the effectiveness of CoRL and the advantage of ULM-R1 through extensive qualitative and quantitative experiments across diverse benchmarks.
2
Related Work
Unified Multimodal Understanding and Generation. Recent advancements [8, 39, 67, 71, 76, 82, 86-88, 90, 91, 107] have witnessed increasing attention to jointly model multimodal understanding and visual generation within unified model. Pioneering attempts [12, 17] predominantly rely on continuous diffusion models, integrating external diffussinn decoders for image synthesis. Inspired by autoregressive next-token prediction, growing line of research [8. 31, 39, 47, 57, 71, 82, 85-88, 111] encode visual inputs into discrete tokens and generate images in a fully autoregressive (F-AR) manner. Specifically, this approach employs vector quantized (VQ) tokenizer [14, 93] to convert images into discrete tokens, analogous to text tokenization. To mitigate information loss in VQ discretization, another stream of work [4, 16, 23, 29, 48, 65, 74, 76, 90, 92] explores autoregressive and diffusion (AR-Diff) hybrid modeling approaches. Architecturally, these models typically comprise a vision autoencoder, a text tokenizer, and an L.M.. Given the unified advantage of the F-AR model in generation manner, this work builds upon it to develop co-reinforcemmet learning framework.

---

RL-based Post- TeasnTrainng for MLLMs. Post- Post-training training [108] aims to further enhance the performance of pretrained models for customized upplications and user needs. Recently, RL [78, 84] has emerged as powerful post-training technique, enabling models to learn from feedback and align with human values. RL. in MILLMs can broadly categorized into two paradigms: (1) RI. from human/Al
feedback (RLHF) [34, 54, 61, 68, 77, 81-83, 92, 95, 96, 99, 106, 109] and (2) RL with verifiable reward mechanisms [35, 40, 41, 69,7,, 79, 100J. RLHF involves learning reward models from preference data before RL optimization, whereas the latter directly optimizes models using task-specific reward functions, bypassing explicit preference modeling. For example, DPO [59] is a notable implementa- tion of RLHF and has been adopted by Emu3 [82] and HermesFlow [92] to narrow the performance gap between understanding and generation. contrast, GRPO [64] exemplifies the second paradigm, simplifying reward formulation via group-wise relative advantage estimation. Our work also falls into this paradigm but diverges from prior work such as SimpleAR [79], which utilizes GRPO with external CLIP reward for autoregressive visual generation, and R1-like MILLMs [24, 41, 69, 100] that focus on incentivizing reasoning capabilities. First, our work demonstrates the significant potential of RL in co-optimizing understanding and generation, thereby broadening its applicability beyond reasoning. Moreover, we identify semantic consistency rewards and a co-evolutionary reinforcement strategy as crucial components in enhancing ULMs.
3 Methodology
3.1
Preliminary
Group relative policy optimization (GRPO) [64] is value-free policy optimization algorithm with improved training stability and sample efficiency. Building upon PPO [62], GRPO introduces a group- wise relative advantage approach to bound policy updates while maintaining optimization flexibility. Let denote a policy parameterized by O. Formally, given an input content c, the algorithm first samples group of G outputs (01, ........ og from the current policy Teaia Each output is then evaluated using predefined, verifiable reward functions, yielding the reward set {r1, {?1.72.... .g....,,acl These rewards are subsequently normalized compute group-relative advantages as follows:
...
A; T; - -mean((1)) f2,,,,,,--- r2,n,,,,))..))
(1)
std((fr, r2........."....
After obtaining the advantage set {A1, A ... Aa} via group relative advantage estimation, the policy he is optimized by maximizing the following objective:
(()) = (o Te G
1
Σ
To(o:) A; - B BKL (To ll wret) Tew (or)
(2)
where DKLL denotes the KL-divergence used to constrain the deviation between wo and its reference policy Tref, and / is a regularization coefficient.
3.2 Pilot Exploration
Given the exceptional performance and data ef- ficiency of DeepSeek-R1 ---rro [19], we explore the potential of ULMs enhance understand- ing and generation capabilities without depen-
Benchmark Average Accuracy
44
Beseline
RL Re-Sep (MM2T)
RL-Cycle
79.5
RL-Sep (T2T)
RL-Sep-Merging
RL RL-Unified
MAlver)
Ang. Sore & DPS
79.0
dence on task-specific supervised fine-tuning. To accomplish this, we curate a dataset² com- prising 16K samples sourced from the COCO 2017 training split [38]. Each sample includes real image, an associated caption as a textual prompt for visual generation, and a correspond- ing QA pair for the multimodal understanding task. We adopt CLIP Score [58] as the verifiable reward for image generation, along with a com- bination of formatting correctness and answer
78.5
78.4
78.4
39.1
78.0
39
38
77.5
77.1
77.0
3611
Generction
Understonding
Figure 1: Results of different RL paradigms.
Janus-Pro-ID [8] serves the baseline.
hateps://hugginggaace wttp://atttttttttm////a// ww/xpx/xxxxxxee rft_161
3

---

Prompt Image Question
Text-to- Image Generation
Prompt- Image
re- ee-cettionng
Policy ULM (rg)
Cpca
10
DxL
Group Computation
Per
R
Reyela
Policy Gradient Update
Rym
rff
A
A
Ag
Reward Evaluntion
Reycle
RTD
AAc
Format
Multimodal Understanding
Image-+ Question
7
Open Ended Accuracy Reward
DRL
Group Computation
Multi-Chalce Accuracy Researd
Group Computation
A
Aa
Tyf
A
Aa
Reward Evaluation
Unified RL
S2: Refined RL
Figure 2: Overview of CoRL,, a co-reinforcement learning framework to jointly improve the dual capabilities of ULMs. CoRL. adopts a two-stage RL procedure, comprising unified RI. stage for joint optimization and refined RL stage for task-specific enhancement.
accuracy as the reward for text generation. We investigate four distinct RL paradigms: (i) separate RL, where understanding and generation tasks are independently optimized with their respective reward mechanisms: (ii) separate RI. followed by weight merging, where each task is separately optimized, and the resulting weights are subsequently merged using a Gaussian distribution- based merging strategy [66] to incorporate both abilities; (iii) cycle RL, which employs scheduled alternation between the two tasks throughout the training process; and (iv) unified RL, which both tasks are jointly optimized within a unified paradigm promote the co-evolution of dual capabilities.
observe that (1) direct task -secccccc RL fails to achieve the expected improvements for ULMs, particularly in the visual generation task, and may even impair performance on the other task; and (2) unified RL demonstrates substantial advantages over alternative paradigms. These findings indicate that the dual capabilities that co-evolve within shared training framework contribute enhanced cross-task synergy and knowledge transfer.
As presented Figure
3.3
Co-Reinforcement Learning
3.3.1
Verifiable Reward for Multimodal modeling
In this section, develop suite of verifiable rewards for multimodal modeling, which provide clear and objective feedback to steer ULMs toward generating high-quality image and text outputs.
Bidirectional Cycle Consistency Reward in Text-to-Image Generation. To encourage ULMs to generate images that faithfully depict the concepts and entities described the input prompt, we introduce bidirectional cycle consistency reward Reyele. which measures the consistency between predictions and ground truth both visual and textual spaces. For visual consistency, we adopt LPIPS [104] to assess the patch-level perceptual similarity between the real image Treal and the synthesized image Igen- Textual consistency is implemented
re-captioning manner. Specifically, we first employ BLIP [32] to generate a caption Cre-cap for each synthesized image, and then compute SPICE [1] score between Cre-cap and original prompt Porg to measure semantic fidelity. The combined bidirectional cycle reward is defined as:
Reyck 1 LPIPS Igan) + SPICE(P (Pogg Cre-cap).
This bidirectional reward forms a closed feedback loop that promotes mutual consistency between texts and images, effectively penalizing hallucinated content and reinforcing prompt-aligned visual generation by simultaneously optimizing for both visual and textual consistency. Furthermore, RRyele is normalized the range [0, 1] before being combined to ensure that all rewards operate on comparable scales and prevent any single component from dominating due to scale differences.

---

Text-Image Matching Reward. While CLIP Score [58] provides holistic measure of text-image aligmment, as shown Sec. 3.2, it underperforms due its limited capacity for assessing fine- grained semantics. To address this limitation, we instead propose text-image matching reward RTIM, which leverages the ULM itself evaluate cross-modal alignment at the token level. Given a textual representation T = { ... L E TR.x of the prompt and the corresponding visual representation =
{i1, 22, &L, E R Rlaxd of a generated image, the reward is computed as:
L
L
1
max cos(2 tb) RTIM 2 Li ke(L.
max cos(ts, 8z) L JE
(4)
where L, and Lg are the sequence lengths of the textual and visual tokens, is the embedding dimension, and RTIM also be normalized the range [0, 1]. This reward captures the fine-grained correspondence between textual concepts and visual elements through maximum cosine similarity, ensuring mutual alignment between visual tokens and their most relevant textual counterparts.
Accuracy Reward in Multimodal Question Answering. Accuracy rewards leverage sisk-specific metries to directly evaluate the correctness of ULM predictions. We consider two accuracy rewards tailored to different question types: RMcQ-AAce for multi-choice questions and RoE-Ace for open-ended questions. These rewards follow binary evaluation mechanism, assigning value of when the predicted answer (i.e., the final answer parsed from within <answer> and </answer> tags) matches the ground truth and 0 otherwise.
Format Reward in Text Generation. To encourage ULMs to generate structured and interpretable textual responses, we adopt the format reward [19], which requires the model to enclose its think- ing process inside <think> ... </think>, and provide its final answer within <answer> and </ </answer> tags. The format reward Rormat returns for strict compliance and 0 otherwise.
3.3.2 Unified Reinforcement Learning for Synergistic Multimodal Modeling
As illustrated in Figure the policy ULM first undergoes unified reinforcement learning with diverse rewards across understanding and generation tasks. This unified process aims to jointly enhance its dual capabilities and establish solid foundation for subsequent task-specific refinement.
Reward Function and Training Objective. To ensure diversity and complementarity in reward signals for unified multimodal modeling, we formulate joint reward function as
RUni-SI
Rayele + RTM + À (RAce + Roormat),
(5)
where is a coefficient that balances the two types of rewards. During training, given an input prompt and an image-question pair, the policy model me.w first generates G candidate responses,
((I,T) Ti). (I2, T2)..... (Ig, }. each comprising a synthesized image I and CoT-format solution T. Concurrently, the joint reward function Runi sı evaluates each candidate pair, yielding These rewards are subsequently normalized according to Eq. (1)
the reward set {r1, ........ .GG. compute the corresponding group-relative advantages = (A1, Ag...... Ao}. The new policy model Te is then updated by maximizing the following GRPO-based objective:
..........
Te (o:) Tos (o:) A;, where o; (Is, TT).
Lsi B Tolia T
(6)
Notably, based on empirical findings from recent work [94], we omit the KL-divergence constraint during this stage to improve both optimization efficiency and generalization capability.
Training Data. To support unified RL for synergistic multimodal modeling, we curate a compre- hensive dataset comprising 22K samples?, which follows the data structure established in Sec. 3.2. Each sample includes real image, prompt for visual generation, and a CoT-formal QA pair for multimodal understanding. This balanced data composition facilitates joint optimization of dual capabilities within a unified framework, while preserving the granularity of task sask-spccific supervision.
hntep::/hhugiingfacc http://accc.o///tttttss/////// vl/x2x. 12_22___ft22t
5

---

Table Results on text-to-image generation benchmarks. and d denote models trained using DPO and GRPO strategies. The best performance in each category is highlighted in bold.
GenEval
WISE f DPG t
Model
Scale
Res.
Type
Two Obj. Counting Position Color Attri. Overall Overall Overall
Generation Only
PixArt-o [5]
0.6B
5122
Diff
0.50
0.44
0.08
0.07
0.48
0.47
71.11
SDv1.5 [60]
0.9B
5122
Diff
0.38
0.35
0.04
0.06
0.43
0.32
63.18
SDv2. 1 [60]
0.9B
5122
Diff
0.51
0.44
0.07
0.17
0.50
0.32
68.09
SD3-Medium [15]
2B
5122
Diff
0.94
0.72
0.33
0,60
0.74
0.42
84,08
SDXL [55]
2.6B
1024²
Diff
0.74
0.39
0.15
0.23
0.55
0.43
74.65
DALL-E3 [3]
10242
Diff
0.87
0.47
0.43
0.45
0.67
83,50
LlamaGen
0.8B
256
F-AR
0.34
0.21
0.07
0.04
0.32
65.16
SimpleAR
1.5B
10242
F-AR
0.90
0.28
0.45
0.63
81.97
Unified Understanding and Generation
TokenFlow [57]
8B
256
F-AR
0.60
0.41
0.16
0.24
0.55
73.38
Emu3 [82]
8B
5122
F-AR
0.66
0.39
80.60
Emu3-DPO [82]
5122
F-AR
0.64
81.60
LWM [39]
5122
F-AR
0.41
0.46
0.09
0.15
0.47
Orthus [29]
5122
AR-Diff
0.58
0.27
Janus-Pro [8]
384
F-AR
0.89
0.59
0.79
0.88
0.80
0.35
84.19
ILLUME+ [23]
3B
384
AR-Diff
0.88
0,62
0.42
0.53
0.72
D-DiT [37]
2B
5122
Diff
0,80
0.54
0.32
0.50
0.65
Harmon [87]
1.5B
5122
F-AR
0.86
0.66
0.74
0.48
0.76
0.41
show-o [90]
1.3B
5122
AR-DiII
0.80
0.66
0.31
0.50
0.68
0.35
67.6 78.8
Heom [92]
1.3B
5122
AR-Diff
0.84
0.66
0.32
0.52
0.69
70.22
Janus [85]
1.3B
3842
F-AR
0.68
0.30
0.46
0.42
0.61
0.23
79.68
Janus- Pro [8]
1.5B
3842
F-AR
0.82
0.51
0.65
0.56
0.73
0.26
82.63
ULM-RI
3842
F-AR
0.85
0.71
0.68
0.80
0.77
0.33
83.92
3.3.3
Refined Reinforcement Learning for Task-specific Enhancement
After completing unified RL, as shown in Figure 2, we apply a targeted learning strategy to further enhance the task-specific performance of the policy model. This second-stas optimization leverages task-specific rewards and tailored datasets for individual tasks.
Reward Function and Training Objective. For text-to-image generation, the reward is defined as R1ZI-S2 = Reycle RTIM For multimodal understanding, we define two distinct reward formulations: (1) RMCQ-52 = RMcq-Ace + Rooraa for multiple-choice questions, and (2) RoF.s2 = ROF-Ace R open-ended questions. The training objective in this stage adheres to the standard GRPO formulation in Eq. (2), with the appropriate task-specific reward (RT2152, RMcq-s2, or RoF.s2) replacing A, depending on the task. To ensure stable optimization, we reintroduce the KL-divergence constraint at this stage to limit policy deviation from the reference distribution.
Training Data. For text-to-image generation, continue training on the curated dataset introduced in Sec. 3.2. For multimodal understanding, we utilize two specialized datasets: mcot_rl s___rr_mmr __mcq* for multiple-choice questions and mcot_rl_ r__var'' for open-ended questions. These task-specific datasets enable the model to develop more refined and robust capabilities within each task domain.
4 Experiment
4.1
Experimental Setups
Evaluation Benchmarks. We evaluate visual generation capabilities on the GenEval [18), WISE [50], and DPG-Bench [22] benchmarks. GenEval employs an object-centric evaluation protocol to assess compositional and attribute-level alignment, while DPG-Bench adopts a VQA-based setting to evaluate dense prompt-following and semantic fidelity. WISE provides holistic evaluation of models' world knowledge, considering consistency, realism, and aestheties. We also evaluate multimodal understanding capabilities across diverse benchmarks. Specifically, MMStar [6], MMMU [98], and

---

Table Results on multimodal understanding benchmarks. The best performance within each
category is highlighted in bold. denotes results obtained from our evaluation.
Multi-Choice (MC) 1
Open-Ended (OE) t MMVei POPE Logic
MC&OE Mixed f
Model
LLM
MMMU MMStar Math
Math
Math
Math
Understanding Only
SmoIVLM [49]
Smo1LM2-1.77
38.8
41.7
9.1
33.8
85.5
28.0
43.6
12.6
12.8
SAIL-VL [11]
1we.2.5-1..58
44.1
56.5
14.6
44.2
88.1
30.4
62.8
17.4
17.3
Ovis2 [45]
Qwen2.5.1. 5.1.58
45.6
56.7
9.9
58.3
87.8
34.7
64.1
29.4
17.7
InLemVLL3 [110]
Qwen2.5. 5-1.5B
48.7
61.1
22.9
67.0
90.1
34.7
57.6
24.5
20.2
Qwen2.5-VL [2]
Qwen2.5-25 5-3B
51.2
56.3
22.9
60.0
85.9
40.3
61.2
31.2
21.9
LMM-RI [53]
Qwen2.5
58.0
63.2
41.6
26.4
Unified Understanding and Generation
ILLUMF+ [23]
Qwen2.5.3B
44.3
40.3
87.6
Harmon [87]
Qwen2.5-1.5R
38.9
87.6
VILA-U [88]
LLaMA-2-7B
33.5
85.8
Orthus [29]
Chamelcon-7B
28.2
79.6
UniToken [27]
Chameleon
32.8
46.1
38.5
SGen-VL. [31]
I2-emL
34.2
34.5
85.3
42.7
Show-o [90]
Phi-1.3B
26.7
80.0
HermesFlow [92]
Phi-1,38
28.3
81.4
Janus-Pro [8]
DeepSeck-LLLM
41.0
46.5
9.7
50.0
87.4
28.0
42.5
15.9
14.7
Janus [85]
DespSeck-LLM-I.33
30.5
37.6
3,41
34.3
87.0
23.91
33.7
14.9
13.4t
Janus-Pro [8]
DeepSeck. LLM 1.5B
36.3
43.17
5.91
39.8
86.2
23.91
37.31
13.5
13.41
ULM-R1
DespSock-LLM-11 5B
42.3
47.6
21.1
43.9
88.9
34.5
42.5
25.4
22.0
WeMath (Math We) [56] are used for multi-choice evaluation, while MMVet [97]. POPE [36], and Logic (Logic [89] are used for open-ended evaluation. In addition, employ Math Vista (Math 'T) [43], Math Verse- Vision (Math V5) [102], and Math Vision (Math Vit) [80] to assess complex mathematical reasoning capabilities. covering both multi-chote and open-ended QA formats. On these benchmarks, we compute accuracy using the toolkit VLMEvalKit [13].
Implementation Details. We develop ULM-RI using Janus-Pro- IB [8] the baseline ULM for unified multimodal understanding and generation. To ensure reproducibility and scalability, our RL training is built upon the trl [75] framework. In the unified RI. stage, we employ the AdamW optimizer with an initial learning rate of 4c-6 and batch size of 16. We sample responses for both understanding and generation tasks, and set the reward balancing factor Eq. (5) to 0.8. In the refined RL stage, we sample 16 responses for both multimodal understanding and text-to-image generation tasks. Additionally, we reduce the learning rate le-6 facilitate fine- mee-griieed optimization. All training is conducted on 8 NVIDIA H20 (96G) GPUs. During inference, greedy decoding is used for text generation in multimodal understanding tasks. For text-to-image generation, we employ classifier- freee guidance (CFG) [20] with a guidance weight set to 5. More details on the training data and settings are provided in App.
4.2 Quantitative Results
Text-to-Image Generation. Table presents comprehensive comparison between ULM-R1 and state-of-the-art models across three visual generation benchmarks. Among unified models, our model ranks second on both GenEval and WISE benchmarks. Notably, it achieves balanced performance across diverse task categories within GenEval, with the best score of 0.71 in object counting. When compared with specialized generation-only models, ULM-RI surpasses the top performer SD3- Medium [15] by slight margin (0.77 vs. 0.74 on GenEval). Moreover, ULM-RI shows consistent improvements over its base model across all benchmarks. These results collectively demonstrate the effectiveness and advantage of our CoRL in enhancing visual generation quality.
Multimodal Understanding. Results are shown in Table 2. For mixed QA format evaluation, we continue to apply the Gaussian -iassiinndistibuttion based merging strategy [66] to combine the two task-specific policy models and obtain a final model capable of following both types of instructions. Overall, ULM-R1 markedly outperforms existing unified models across most benchmarks, and substantially narrows the performance gap with leading understanding-onl) MILMs of comparable model scale. More specifically, our model achieves state-of-the-ee of-the-arttttttt performance among unified models

---

Table Comparison between different RL paradigms for ULMs. The cold SFT data is consist of x2x 12x_rft. fft__22,, mcot_rl_mee (22K), and mcot_ry _r____rl_aaaa (22K). #7: CoRL.
Ablated Setting
Stage GenEval
DPG
MMMU
We
MMVet
Logic
Baseline
73.0
82.6
36.3
5.9
39.8
23.9
Cold-SFT
S1
72.8 (-0.3) 82.5 (-0.))
(+4.7) (+10(+
422.
27.9 (+4.0)
2 Unified-RI.
S1
75.9 (+2.9) 83.3 (+0.7)
40.3 (+4.0) 14.0 (+8.1)
42.5 (+2.7)
30.2 (+6.3)
3 + Refined-RR RL (T2I)
S2 75.1 (+2.1) 83.0 (+0.4) /
4 + Refined-RL RL (MM2T-MC)
S2
39.6 (+3.3) 15.8 (+9.9)
5 Refined-RR RL (MM2T-OE)
S2
42.2 (+2.4) 29.5 (+5.6)
+ Refined-RE RL w/ Cold-SFT
S1&S2 74.5 (+1.5) 82.8 (+0.2) 41.8 (+5.5) 22.5 (+16.6) 43.7 (+3.9) 35,9 (+12.0)
7 + Refined-Fi RL w/ Unitied-RL RL S1&S2 77.3 (+4.3) 83.9 (41.3) 42.3 (+6.0) 21.1 (+15.2) 43,9 (+4.1) 34.5 (+10.6)
Jamus-Pro
ULM-RI
Janus-Pro
ULM-RI
Janus-Pro
ULM-RI
photo of a krife and aebra
a photo of four bowls
a photo of four donuts
photo of computer heyboard and a laptop
photo of cell phona and a horse
photo of a green brown airplane
HeLL
Helo
A clear image of blachboard with clean,
An fldhfashtened motal lantern with dark green surface and word weathered brense patina. glass panela written precisely and logibly in with bold, white chalk leccers.
large brown bear partially submerged clear, tranquil body of water. The ripples gently around the bear sits cabmly, enjaying the warmth of the aunlighe.
alightly foggy from years of use..... casting warm light acress dark weeden aurface.
Figure 3: Qualitative comparison of text-to-image generation between Janus-Pro and ULM-RI. The red box marks an exemplary failure case.
on MMStar (47.6), WeMath (21.1). Losic (34.5), and on several mixed-format math benchmarks, including Math Verse (25.4) and Math Vision (22.0). Particularly, ULM-RI demonstrates considerable improvements over its base model in mathematical and logical reasoning tasks, achieving gains of 15.2 on WeMath and 10.6 on Logie These results not only demonstrate the effectiveness of CoRL in enhancing ULMs' understanding capabilities, but also establish that reinforcement learning provides a data-efficient pathway for achieving both robust generalization and sophisticated reasoning capabilities, without the need for large- scale supervised data.
4.3 Qualitative Results
In this section, we first present qualitative comparison between ULM-R1 and Janus-Pro for visual generation, illustrated in Figure The results clearly show that ULM-RI achieves superior text-to- image aligmment and object grounding across diverse prompts, with especially notable improvements spatial arrangement of objects and compositional consistency. Next, as shown in Figure 4, we visu- alize several representative examples of multimodal understanding. Compared to Janus- Pro, ULM-RR exhibits significantly enhanced understanding capabilities, particularly in mathematical reasoning. These comprchensive qualitative results demonstrate the effectiveness of CoRL in simultancously

---

Question: in the diagram, Options: A. 15 B. C. 5m D. No correct answer
wire. What circumforence circle?
Janus-Pro: A
ULM Ru: <think The ciroumfferecce of cirele calcuiated using the formula C zer. Substituting the grvzn
3F
calue, r = Som, we get C= 2x15 10n. The answer is B. <\chink> <unswer> B-c lanswers>
Question: The graph shous an angle a in scandard position with its cerminal side incersecting the figure at P(3/s, s/s). Find the value of tan
Janus-Pro: The vulue is 3/44
ULM-R1: Thhirk-Thhe bangent of un ungle in a right triungle is the ratio of the lengch of the sidle oppusite the ongle to the length the side odjocmt the angle. this cose, the sinle opposite the angle is y-oordionte of the point P. which is 4/5, and the side edjecent to the engle is the x-coordinate of the point P. which 3/5.
Therefore, tun = (4/5)/33/5) #4/3. The onswer is //hink consuer>>4 //nowwrer>>
Question: What is the substance that is developing on these leaves?
Options: Algoe Downy mildew C. Powdery mildew Sooty mould Don't know and don't want to guees
Janus- Pro: B
ULM-R1: <think-Thh green leuves the plant cuvered white poundery substunce. This substuncc
type of mildew that grous on the leaves of piants. Powdery mildow a common disense that offects many plunts, Thergfore, the onswer is Powndery mildew. (jthink comsuersoC lunswer>
Quastion: la the person bigger than the cart
Janus- Pro: Yes, the person bigger than the car. The person standing on the side of the road, while car is on the roud ahoad of him.
ULM--R:: Rai No, the person is not actually bigger than the car. < cantneb-cansseer-Noo |an/uuserrr> Janswer>
Figure 4: Qualitative comparison of multimodal understanding between Janus-Pre Prou and ULM-RI. The red box marks an exemplary failure case.
improving visual generation and multimodal understanding in ULMs. Furthermore, Figures 3 and 4 respectively showcase exemplary failure cases of ULM-RI in visual generation and understanding tasks, providing an intuitive grasp of its limitations. For instance, in the understanding example, it misinterprets commonsense and professional knowledge, leading to an incorrect answer.
4.4 Ablation Study and Discussion
In this section, we primarily evaluate the effectiveness of our RL training strategy and the proposed re- ward functions for text-to-image generation. addition, we discuss the impact of the hyperparameter X and the scalability of CoRL.
Comparison Between Various RL Paradigms. As presented in Table 3, we conduct compre- hensive ablation study to evaluate the effects of different RL paradigms for ULMs. The results reveal two key findings: #2 vs. #1: Unified-RL. effectively enhances both the generation and understanding capabilities of ULMs, whereas Cold-SFT has minimal impact on visual generation. #7 vs. #6: Compared to the de facto paradigm, our CoRL consistently outperforms on visual generation benchmarks while achieving comparable results on multimodal understanding benchmarks. These findings indicate that unified RL provides a robust foundation for task specific refinement, even without reliance on supervised data. Additionally, CoRL consistently outperforms both its baseline and task- sseeiiicc RL variants (#3-#5), achieving improvements of 2.1 points on GenEval (vs. generation-only RL, #3) and 5.3 points on WeMath (vs. understanding-only RL, #4). These results demonstrate the efficacy of CoRL as our final RL paradigm.
Effect of Rewards in Text-to-Image Generation. To evaluate the effectiveness of our proposed rewards for text-to-image generation, we conduct ablation experiments as detailed in Table 4. The results demonstrate that incorporating either reward individually improves performance over the baseline: Reycle yields an increase of 2.1 in average score, while RRIM results an increase of 0.8. Notably, combining both rewards leads to the best overall performance, achieving an average score of 80.6. These findings suggest a modest but complementary effect between Reyele and RRIM. enhancing their joint benefit in enhancing visual generation quality. In addition, we further compare the CLIP score (RCII) and RTIM under our final RL training paradigm. As shown in the table, RTIM achieves better overall performance, especially on the DPG benchmark with dense, long-horizon prompts for image generation, highlighting its superior ability to capture fine-grained semantic alignment compared to the CLIP score.

---

Table 4: Effect of visual generation rewards.
Table Comparison among
Table 6: Impact of X.
Rewards
GenEval
DPG
Avg. 7
visual consistency measures
used Roycle-
GenEval (Gen.)
MMMU
Baseline
73.0
82.6
77.8
(Und.)
Measures
GenEval
DPG
RCLIP
74.2
82.4
78.3 (+0.5)
0,5
77.1
41.0
RTIM
74.1
83.0
78.6 (+0.8)
PSNR
76.0
82.4
0.7
77.3
42.3
Reyele
76.2
83.5
79.9 (+2.1)
MSE
77.1
83.2
0.8
77.3
42.3
Reyele + RCLIP
77.0
83.4
80.2 (+2.4)
SSIM
77.5
83.6
0.9
77.1
43.5
Reyele + RTIM
77.3
83.9
80.6 (+2.8)
LPIPS
77.3
83.9
1.0
76.9
43.0
Impact of Visual Consistency Measures in Reyele- Table provides a more detailed analysis of how different visual consistency measures (PSNR, MSE, SSIM, and LPIPS) used in Reyele affect the quality of visual generation. PSNR and MSE are pixel-level metrics that quantify low-level differences between the generated images, while SSIM and LPIPS assess higher-level perceptual and structural similarities. As shown in the table, SSIM and LPIPS perform better than the other two metries, with LPIPS achieving the best performance (83.9) on the DPG benchmark. This can be attributed to the fact that LPIPS measures image similarity in feature space. making it more robust minor, semantically irrelevant variations and thus better suited to reward high-level consistency.
Impact of hyperparameter X. The factor in Eq. (5) balances the reward scales between the two tasks during unified RL. As shown in Table 6, conduct experiments using different values of À to assess its impact on both generation and understanding performance. The results show that moderate values (~~ 0.8) achieve a balanced trade-off between generation and understanding. Larger values slightly degrade generation performance, indicating that overemphasizing understanding rewards may hinder cross-task optimization.
Scalability of CoRL. To validate the effectiveness of CoRL on other ULMs, as illustrated Table 7, conduct additional experiments using Janus-1.3B [85] and Janus-Pro Pro-7B [8] as the baseline. The results show consistent improvements across both generation and understanding benchmarks, confirming the scalability of CoRL. Notably, Janus-Pro-7B with LoRA tuning achieves smaller gains on the mathematical reasoning benchmark (WeMath) than Janus-1.33 1.3B, suggesting that while CoRL scales well across model size, its enhancement of complex reasoning does not scale linearly.
Table Effectiveness of CoRL on other ULMs. For Janus-Pro-7B. we adopt LoRA tuning [21] to enable efficient training and mitigate memory pressure during unified RL.
Methods
GenEval
WISE
DPG
MMMU
MMStar
Math We
MMVet
POPE
Logic
1.3B
0.61
0.23
79.68
30.5
37.6
3.4%
34.3
87.0
23.9
CoRL (Full Fine-Tuning)
0.64
0.26
80.92
34.6
41.9
16.4
36.9
88.1
27.0
Janus-Pro-7B
0,80
0.35
84.19
41.0
46.5
9.7
50.0
87.4
28.0
+ CoRL (LoRA Tuning)
0.82
0.41
84.97
44.6
49.5
16.0
52.6
88.0
32.4
5
Limitation
Despite the substantial improvements achieved, several limitations remain that warrant further investigation. First, a notable performance gap still exists between generation and understanding tasks of ULMs. Second, our rewards for multimodal understanding are relatively simple and primary. These limitations highlight the need for more sophisticated RL designs that can further enhance understanding capabilities and narrow the performance gap. We hope our work provides valuable insights for future RL research in ULMs.
6 Conclusion
In this work, we investigate how to jointly enhance the understanding and generation capabilities of ULMs, and propose co-reinforcement learning framework (CoRL). Within the proposed CoRL, the policy model follows a Foundation-then-Sye theeiinntttnnSeeiiiiaatiiooo paradigm that involves two-stage RL procedure: unified RL stage for joint optimization and refined RL stage for task-specific enhancement, yielding ULM-RI. Extensive evaluations across diverse understanding and generation benchmarks demonstrate the effectiveness of CoRL and the advantage of ULM-RI.
10

---

Acknowledgements. This work was supported in part by NSFC (62406189. 62322113, 62376156), Shanghai Municipal Science and Technology Major Project (2021SHZIZZ01022, and the Funda- mental Research Funds for the Central Universities.
References
[1] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould, Spice: Semantic propositional image caption evaluation. In ECCV, pages 382-398, 2016.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv:2502.1392 13923, 2025. 7
[3] James Betker, Gabriel Goh, Li Jing. Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions, 2023. URL https://cdn. httpp////cd...peaai..... wwtp:pppws//aaalllaca.a...
pdf. 6
[4] Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang. Tianyi Zhou, Saining Xie, Silvio Savarese, al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset. arXiv:2505.0955 2025.
[5] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao. Enze Xie, Yue Wu, Zhongdao Wang. James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer photorealistic lext-to-image synthesis. arXi::2310.0044 00426, 2023. 6
[6] Lin Chen, Jinsong Li, Xiaoyi Dong. Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models?
arXiv:2403.2033 2024. 6
[7] Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M' A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. arXiv:2405. 2024. 24
[8] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. 2rXiv:2501.1781 1, 2025. 1, 2, 6, 10
[9] Huilin Deng, Ding Zou, Rui Ma, Hongchen Luo, Yang Cao, and Yu Kang. Boosting the generalization and reasoning of vision language models with curriculum reinforcement learning. arXiv:2503.07065 2025.
[10] Linger Deng, Yuliang Liu, Bohan Li, Dongliang Luo. Liang Wu, Chengquan Zhang, Pengyuan Lyu, Ziyang Zhang, Gang Zhang, Errui Ding, al. R-cot: Reverse chain-of- problem generation for geometric reasoning large multimodal models. arXiv:2410. 17885, 2024. 24
[11] Hongyuan Dong, Zijian Kang, Weijie Yin, Xiao Liang, Chao Feng, and Jiao Ran. Scalable vision language model training via high quality data curation. arXiv:2501.059% 00952, 2025.
[12] Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. In ICLR, 2024.
[13] Haodong Duan, Junming Yang. Yuxuan Qiao, Xinyu Fang. Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang. Jiaqi Wang. et al. Vlmevalkit: An open-source toolkit for evaluating large multi- modality models. arXiv:2407.11 1169, 2024.
[14] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers high high- resolution image synthesis. In CVPR, pages 12873 11883,, 2021.
[15] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. In /CML, 2024. 6,7
[16] Lijie Fan, Luming Tang, Siyang Qin. Tianhong Li, Xuan Vang. Siyuan Qiao, Andreas Steiner, Chen Sun, Yuanzhen Li, Tao Zhu, et al. Unified autoregressive visual generation and understanding with continuous tokens. arXiv:2503.1344
2025. 2
[17) Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi calti-granularity comprehension and genera- tion. arXiv:2404.14332 2024.

---

[18] Dhruba Ghosh, Hannaneh Hajishirzi. and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. In NeurlPS, pages 52132-52152, 2023. 6
[19] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang. Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning, arXiv:2501. 12948, 2025. 3,3,5
[20] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. ar 0rXiv2227./25598 2022. 7
[21] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022. 10
[22] Xiwei Hu, Rui Wang. Yixiao Fang. Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with Ilm enhanced semantic alignment, arXiv:2403.05135 2024. 6
[23] Runhui Huang, Chunwei Wang, Junwei Yang. Guansong Lu, Yunlong Yuan, Jianhua Han, Lu Hou, Wei Zhang, Lanqing Hong, Hengshuang Zhao, et al. Illume+: Illuminating unified milm with dual visual tokenization and diffusion refinement. arXiv:2504.0195: 2025. 2, 6,7
[24] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, The Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models.
arXiv:2503.06749 2025. 1,
[25] Dongzhi Jiang, Ziyu Guo, Renrui Zhang, Zhuofan Zong, Hao Li, Le Zhuo, Shilin Yan, Pheng-Ann Heng, and Hongsheng Li. T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot. 2505.0550000703 2025.
[26] Jingjing Jiang, Chao Ma, Xurui Song. Hanwang Zhang, and Jun Luo. Corvid: Improving multimodal large language models towards chain-of-thought reasoning. arXiv:2507.07424, 2025. 24
[27] Yang Jiao, Haibo Qiu, Zequn Jic, Shaoxiang Chen, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding. arXiv:2504.04433 2025. 7
[28] Mehran Kazemi. Hamidreza Alvari, Ankit Anand, Jialin Wu. Xi Chen, and Radu Soricut. Geomverse: A systematic evaluation of large models for geometric reasoning. arXiv:2312.12241, 2023. 24
[29] Siqi Kou, Jiachun Jin, Chang Liu, Ye Ma. Jian Jia, Quan Chen, Peng Jiang, and Zhijie Deng. Orthus: Autoregressive interleaved image-text generation with modality-specinf heads. arXiv:2412.00127 2024.
2, 6, 7
[30] Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming- Hisnn Yang. Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. Llm post-training: A deep dive into reasoning large language models. 2rXiv:2502.21321, 2025. 1
[31] Hao Li, Changyao Tian, Jie Shao, Xizhou Zhu, Zhaokai Wang, Jinguo Zhu, Wenhan Dou, Xiaogang Wang, Hongsheng Li, Lewei Lu, et Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding. arXiv:2412.09604, 2024. 2.7
[32] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In ICML, pages 19730-19742, 2023. 4
[33] Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Mul- timodal arxiv: A dataset for improving scientific comprehension of large vision-language models. urXiv:2403.00231 2024. 24
[34] Lei Li, Zhihui Xie, Mukai Li, Shumian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong, and Qi Liu. Vlfeedback: A large-scale ai feedback dataset for large vision-language models alignment. arXiv:2410.09421,, 2024.
[35] Xinhao Li, Ziang Yan, Desen Meng. Lu Dong. Xiangyu Zeng, Yinan He, Yali Wang. Yu Qiao, Yi Wang, and Limin Wang. Videochat-rl: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv:2504.0695. 2025. 3
[36] Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucination in large vision- languaee models. 2tt.:2335.10355 2023. 7
12

---

[37] Zijie Li. Henry Li. Yichun Shi, Amir Barati Farimani, Yuval Kluger. Linjie Yang. and Peng Wang. Dual diffusion for unified image generation and understanding. arXiv:2501.0025) 002899 2024. 6
[38] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755, 2014.
3. 24
[39] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abheel. World model on million-length video and language with ringattention. In ICIR, 2025. 1, 1,2,6 6
[40] Yuqi Liu, Bohao Peng, Zhisheng Zhong, Zihao Yuc, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg-zero: Reasoning- chain guided segmentation via cognitive reinforcement. MmXiv:2503006522 2025. 1,3
[41] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinfoncement fine-tuning, arXiv:2503.01785, 2025. 1,3
[42] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song- Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, pages 2507-2521, 2022. 24
[43] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hamnanch Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arX0v22310002255, 2023. 7
[44] Pan Lu, Liang Qiu, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In ICLR, 2023. 24
[45] Shiyin Lu, Yang Li, Qing-Gur Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. ar XXxx:2405.20797 2024. 7
[46] Xingtai Lv, Yuxin Zuo. Youbang Sun, Hongyi Liu, Yuntian Wei. Zhekai Chen, Lixuan He, Xuekai Zhu, Kaiyan Zhang, Bingning Wang, et al. Towards a unified view large language model post-training. arXiv:2509.0444 2025.
[47] Chuofan Ma, Yi Jiang, Junfeng Wu, Jihan Yang, Xin Yu, Zehuan Yuan, Bingyue Peng, and Xiaojuan Qi. Unitok: A unified tokenizer for visual generation and understanding. arXiv:2502.20321. 2025. 2
[48] Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. MXXiv:241100797: 2024. 2
[49] Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, et al. Smolvim: Redefining small and efficient multimodal models. arXiv:2504.05299 2025.?
[50] Yuwei Niu, Munan Ning, Mengren Zheng, Bin Lin, Peng Jin, Jiaqi Liao, Kunpeng Ning, Bin Zhu, and Li Yuan. Wise: A world knowledge-infomed semantic evaluation for text-to-image generation. a732v25503.07265, 2025.
[51] OpenAL. Openai 0l system card. arXiv:2412.16720. 2024.
[52] OpenAl. Openai o3 and 04-mini system card, 2025. URL https://cdn, openai com/pdf/ 071777500220--79---80-------8880
and- oo- card. pdf.
[53] Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rale- rl. arXiv:2503.0755. 075336, 2025. 1,7
[54] Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. Strengthening multimodal large language model with bootstrapped preference optimization. In ECCV, pages 382-398, 2024.3
[55] Dustin Podell, Zion English. Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv:2307.01952. 2023. 6
[56] Runqi Qiao, Qiuna Tan, Guanting Dong. Minhui Wu, Chong Sun, Xiaoshuai Song. Zhuoma GongQue, Shanglin Lei. Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human- like mathematical reasoning? arXiv:2407.01284, 2024. 7
13

---

[57] Liao Qu, Huichao Zhang. Yiheng Liu, Xu Wang, Yi Jian, Yiming Gao. Hu Ye. Daniel K Du Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv::412.03066, 2024. 2, 6
[58] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Learning transferable visual models from natural language supervision. ICML, pages 8748-87888 8763, 2021. 3,5
[59] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly reward model. In NeurlPS, pages 53728-53741, 2023. 1,3
[60] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[61] Pritam Sarkar. Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ô Ank, and Tomas Plister. Mitigat- ing object hallucination via data augmented contrastive tuning. arXiv:2405.18653 2024. 3
[62] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv:1707.06344 2017. 3
[63] Dustin Schwenk, Apoory Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- OK VKVQA: A benchmark for visual question answering using world knowledge. In ECCV, pages 146-162, 2022. 24
[64] Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li. Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300 2024. 1,
[65] Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang. Xi Victoria Lin, Luke Zeutlemoyer and Lili
Yu. Lmfusion: Adapting pretrained language models for multimodal generation. arXiv:2412.15788, 2024.
2
[66] Chongjie Si, Jingjing Jiang, and Wei Shen. Unveiling the mystery of weight in large foundation models: Gaussian distribution never fades. arXiv:2501.16611, 2025. 4,
[67] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Au- toregressive model beats diffusion: Llama for scalable image generation. 2:Xi::2406006555 2024. 2.
[68] Zhiqing Sun, Sheng Shen, Shengcao Cao. Haotian Liu. Chunyuan Li. Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu- Yu-Xiong Wang. Yiming Yang, et al. Aligning large multimodal models with factually augmented rihf. arXiv:2309. 2023.
[69] Huajie Tan, Yuheng Ji. Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. rft: Reinforcement tinettnnng for visual reasoning. arXiv:2503.2077 2025. 1,3
[70] Yunlong Tang, Jing Bi, Pinxin Liu, Zhenyu Pan, Zhangyun Tan, Qianxiang Shen, Jiani Liu, Hang Hua, Junjia Guo, Yunzhong Xiao, al. Video-Imm post-training: A deep dive into video reasoning with large multimodal models. arXiv:2510.0503. 2025.
[71] Chameleon Team. Chameleon: Mixed-modal early- farlo-fusion foundation models. arXiv:2405.09818, 2024.
2
[72] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with 1lms. arXiv:2501.12599 2025.
[73] Guiyao Tie, Zeli Zhao, Dingjic Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang. Jiangyue Yan, Yao Su, et al. A survey on post-training of large language models. arXiv:2503.06072. 2025.
[74] Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabhat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv:2412. 141644 2024.
[75] Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching. Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouêdec. Trl: Transformer reinforcement learning. https:/ / / thub. coon/hugginggacc.... tr1, 2020.
14

---

[76] Chunwei Wang, Guansong Lu, Junwei Yang. Runhui Huang. Jianhua Han. Lu Hou, Wei Zhang. and Hang Xu. Illume: Illuminating your Ilms see, draw, and self-enhance. arXiv:2412.06673, 2024. 2
[77] Fei Wang. Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mdpo: Conditional preference optimization for multimodal large language models. arXiv:2406. 1839, 2024. 3,3
[78] Jane Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. 1eXiv:6611.05666 2016. 3
[79] Junke Wang, Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu-Gang Jiang. Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. arXiv:2504. 1 11555, 2025. 1,
[80] Ke Wang. Junting Pan. Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math vision dataset. arXiv:2402.1488 14804, 2024. 7
[81] Weiyun Wang, Zhe Chen, Wenhai Wang. Yue Cao. Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Qiao, et al. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv:2411.10442, 2024."
[82] Xinlong Wang. Xiaosong Zhang, Zhengxiong Luo, Quan Sun. Yufeng Cui, Jinsheng Wang. Fan Zhang. Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv:2409. 18869, 2024. 2, 3, 6
[83] Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified reward model for multimodal understanding and generation. arXiv:2503.0525: 2025. 1,3
[84] Marco A Wiering and Martijn Van Otterlo. Reinforcement learning. Adaptation, learning, and optimiza- tion, 12(3):729, 2012.
[85] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma. Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv:2410. 13848, 2024. 2, 6, 7, 10
[86] Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable multi-modal generators. arXiv:2412.043:3 2024. 2
[87] Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Zhongbua Wu, Qingyi Tao, Wentao Liu, Wei Li, and Chen Change Loy. Harmonizing visual representations for unified multimodal understanding and generation. arXiv:250..2197, 2025, 6.7
[88] Yecheng Wa, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie. Hongxu Yin, Li Yi. al. Vila-u: unified foundation model integrating visual understanding and generation. ICLR, 2025. 2,
[89] Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logievista: Multimodal I1m logical reasoning benchmark in visual contexts. (XXx:2440.04977. 2024. 7
[90] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. In ICLR, 2025. 1, 6,7
[91] Rongehang Xie, Chen Du, Ping Song, and Chang Liu. Muse-vl: Modeling unified vim through semantic 2024. 2
discrete encoding,
[92] Ling Yang, Xinchen Zhang, Ye Tian, Chenming Shang, Minghao Xu, Wentao Zhang, and Bin Cui. Hermesflow: Seamlessly closing the gap in multimodal understanding and generation. 12.... 2025. 2, 3, 6.7
[93] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vegan. 10..022110.04627, 2021. 2
[94] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An source Ilm reinforcement learning system at scale. arXiv:2503. 14476, 2025. 5
15

---

[95] Tianyu Yu. Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng. Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In CVPR, pages 13807-13816, 2024. 1.3
[96] Tianyu Yu, Haoye Zhang. Yuan Yao. Yunkai Dang. Da Chen, Xiaoman Lu, Canqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning milms through open-source ai feedback for super gvt-4v trustworthiness. arXiv:2405.1722D 2024. 3
[97] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv:2308.024: 02490. 2023.7 7
[98] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun. Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipll multimodal understanding and reasoning benchmark for expert agi. In CVPR, pages 9556-9567, 2024. 6
[99] Simon Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Peter Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann Lc- Cun, Yi Ma, et al. Fine-tuning large llngua- models as decision-making agents via reinforcement lcarning. In NeurlPS, pages 110935-1100 19995-110971, 2024. 3
[100] Yufei Zhan, Yousong Zhu, Shurong Zheng, Hongyin Zhao, Fan Yang. Ming Tang, and Jinqiao Wang. Vision-rl: Evolving human-free alignment in large vision-language models via vision-guided reinforce-
ment learning. arXivv:5003. 18013, 2025. 1,3
[101] Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vI: Learning reason with multimodal large language models via step-wise group relative policy optimization. 2rXiv:2503.12937, 2025.
[102] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang. Peng Gao, al. Mathverse: Does your multi-modal Ilm truly the diagrams in visual math problems? urXiv:2403.14624 2024. 7
[103] Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang. Ziyu Guo, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, et al. Mavis: Mathematical visual instruction tuning. arXiv:2407.0873: 2024. 24
[104] Richard Zhang. Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, pages 586-595, 2018. 4
[105] Zhiyuan Zhao, Linke Ouyang, Bin Wang. Siyuan Huang, Pan Zhang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Milm-dataengine: An iterative refinement approach for milm. arXiv:2308. 13566, 2023. 24
[106] Zhiyuan Zhao, Bin Wang. Linke Ouyang. Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucina- tions: Enhancing Ivims through hallucination-awaaee direct preference optimization. 1.Niv:2311116833 2023. 3
[107] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi modal model. arXiv:2408. 11039, 2024. 2
[108] Guanghao Zhou, Panjia Qiu, Cen Chen, Jie Wang, Zheming Yang, Jian Xu, and Minghui Qiu. Reinforced milm: A survey rl-based reasoning in multimodal large language models. arXiv:2504.21277, 2025. 1,
3
[109] Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning, arXiv:2402.11411 2024.
[110] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv:2504. 10479, 2025.
[111] Jialy Zou, Bencheng Liao. Qian Zhang, Wenyu Liu, and Xinggang Wang. Omnimamba: Efficient and unified multimodal understanding and generation via state space models. arXiv:2503. 08686, 2025. 2
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect our paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the last section of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yss]
Justification:
We provide all implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer:
[Yes]
Justification: We provide all links to our data in the footnote of the paper and will soon release our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
http://ddee//eeuuuiiieeeee sonobmiissonooccy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//nips.coom httppp..cppbblicc/cc pepppulic/gguudeeeee guwde//deeeee/e /CodeSummi ssi ssionnolllii. oniioicyy)) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all the experimental details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We use official evaluation protocols provided by the corresponding bench-
marks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by eπror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide all the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httttp////rrrr.....plll///////t
cc. cc/public/Ettiiee
cel.e//deeeecsueeeeleeeeeeeeeeee ines?
Answer: [Yes]
Justification:
We have checked our work.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this in the conclusion and introduction sections.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All are properly referred to.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (c.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No such experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is used for paper polishing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https:/ httppss////urrs.....p /20 for what should or should not be described.
23

---

A Appendix
A.1 Training Data
Training Data for Unified Reinforcement Learning. To support synergistie multimodal modeling during unified RL, we curate dataset (i.e., x2x_ [ex_rft_22 22k) that simultancously involves text-to-image generation and multimodal understanding tasks. As illustrated in Figure 5, each sample includes real image, prompt for generation, and problem for understanding. The real images are sourced from the COCO 2017 train split [38], while the problems and their corresponding solutions are adapted from A-OKVQA [63] and GPT-VQA [105]. addition, prompts are selected from the original COCO captions based on their entity coverage with the problem solutions.
Generation
Generation
Prompt: a girl eating carret
Prompt: an ice cream truck parked fleld flying sky
Question: girl going get carret goes in her threat becouse she will stort doing what?
Options:
A. choking
B. passing our
Question: What type of truck
chowing
D. laughing
Answer the question based on the image and your knowledge.
Answer the question based on the image and your knowledge. Plense write thinking process inside <thinks </ //think> and provide your
Please write thinking process inside <think s/tnink> and provide your
(option Kanswer> s/ </amewer* tags
egg, Aj B/CC/D
(only words) <answer> s/ /answer>> tags.
response MUST striclly follow this format: <think> / think consverropttion letters/ answer>
Your response MUSI strictly follow this format: <think> t.itthink>> canswerPrcnniie answerd/ answert
Understanding (MC)
Understanding (OE)
Figure Illustration of training examples used in unified reinforcement learning.
Training Data for Refined Reinforcement Learning. In this stage, we collect three specialized datasets for task-specific RI.. For text-to-image generation, we continue constructing a dataset (ie.. x2x_rft_16k) with prompts derived from COCO captions. Moreover, we curate mcot_ri_man and mcot_rl_ for multipe-chhocee and open-ended multimodal understanding, respectively. These two datasets are curated top of MCoT-Instruct [26], which encompasses diverse range of multi- science-problem
modal tasks, including mathematical reasoning, science problem solving, and visual commonsense reasoning, across multiple source datasets. Specifically, the source datasets of macot_rr_mee ri_mcq__ com- prise A- A-OKVQA [63], C [7]. SQA-IMG (train) [42], ArxivQA [33], TabMWP (MC) [44], and (MC) [103], while the source datasets of mcot_r ri_vqa include Geom Verse [28].
R-CoT [10], TabMWP (OE) [44], and MAVIS-Instrun (OE) [103].
A.2
Supplementary Experimental Setups
Table 8 provides detailed hyperparameter settings for ULM-RI's RL training.
Table 8: Training hyperparameter setting.
Configuration
Unified Refined RI. (T2I) Refined RI. (MM2T-MC) Refined RI. (MM2T-OF)
Number of sampled outputs (G)
8
16
16
16
Regularization coefficient of Tga (5)
0
0.02
0.02
0.02
Max prompt length
1024
256
1024
1024
Max completion length
512
512
512
Batch size
32
Peak learning rate
4e-6
le-6
le-6
le-6
Epoch
24


Reviews:
[{"review_id": "tH0kXGXd0X", "reviewer": "Reviewer_9e1d", "summary": "# Summary\n\nThe **pilot study** demonstrates the effectiveness of bidirectional RL for synergetic optimization. According to the pilot study, the authors propose a two-stage RL training strategy:\n\n**Stage 1: Unified RL Training**: In the first stage, the authors employ unified RL training using GRPO for both understanding and generation tasks, with Janus-Pro-1B serving as the policy model. The reward functions include:\n\n- $\\mathcal{R}_{\\text{cycle}} = 1 - \\operatorname{LPIPS}(\\mathcal{I}_{\\text{real}}, \\mathcal{I}_{\\text{gen}}) + \\operatorname{SPICE}(\\mathcal{P}_{\\text{org}}, \\mathcal{C}_{\\text{re-cap}})$, where BLIP is used as the re-captioner.\n- $\\mathcal{R}_{\\mathrm{TIM}} = \\frac{1}{2}\\left(\\frac{1}{L_i} \\sum_{j=1}^{L_i} \\max_{k \\in [1, L_t]} \\cos(\\boldsymbol{i}_j, \\boldsymbol{t}_k) + \\frac{1}{L_t} \\sum_{k=1}^{L_t} \\max_{j \\in [1, L_i]} \\cos(\\boldsymbol{t}_k, \\boldsymbol{i}_j)\\right)$, which measures token-level similarity rather than using coarse-grained CLIP scores.\n- Format Reward\n- Accuracy Reward\n\n**Stage II: Task-Specific RL Training**: In the second stage, RL training is tailored for specific tasks, with the following reward combinations:\n- T2I (Text-to-Image): $\\mathcal{R}_{\\text{cycle}} + \\mathcal{R}_{\\text{TIM}}$\n- MCQ (Multiple Choice Questions): $\\mathcal{R}_{\\text{MCQ-Acc}} + \\mathcal{R}_{\\text{Format}}$\n- Open-Ended Text Generation: $\\mathcal{R}_{\\text{OE-Acc}} + \\mathcal{R}_{\\text{Format}}$", "questions": "# Questions\n\n- For $\\mathcal{R}_{\\text{OE-Acc}}$, does this refer to Exact Match?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- The methods introduced in this work are novel.\n- The experiments convincingly demonstrate the effectiveness of the proposed methodology.\n- The ablation studies are comprehensive and robust.\n\n# Weaknesses\n\n- For $\\mathcal{R}_{\\mathrm{TIM}}$, is there any ablation study comparing token-level similarity with the CLIP score?\n- The main claim of this work is that “co-RL is better than task-specific RL.” Why, then, is Task-Specific Refined Reinforcement Learning still necessary?\n- In Section 4.2, the evaluation of T2I generation is based on a non-merged task-specific model, whereas for multimodal understanding, a merged model (two text generation models) is used. What would the results be if a three-to-one merged model were evaluated for both tasks?\n- Please anonymize the Hugging Face URLs for the datasets to avoid potential desk rejection risks.", "was_revised": true, "final_justification": "The responses have addressed most of my concerns. For example, the comparison between patch-token-level similarity and CLIP score in A1, the impact of stage I RL in A2, and the performance of the three-to-one model in A3. However, the understanding and generation directions are not truly unified, as it remains necessary to separate image generation and text generation in ULM-R1, as demonstrated in A3. This limitation prevents me from giving a higher score.", "rebuttal": "Dear Reviewer 9e1d,\n\nThank you for your thorough and constructive review. We appreciate your positive feedback on the novelty and effectiveness of our work, the comprehensiveness of our experiments, and your kind suggestion regarding data anonymization. We respond to your concerns and questions below. \n\n> **W1: Ablation study on $R_\\text{TIM}$ and CLIP score.**\n\n**R1:** Thank you for raising this crucial point. Comparing the CLIP score with $R_\\text{TIM}$ under our final RL training paradigm is an essential ablation. We have conducted additional experiments to complement and improve the ablation study in Table 4. Compared to the CLIP score, $R_\\text{TIM}$ demonstrates better overall performance, especially on the DPG benchmark with dense, long-horizon prompts for image generation, highlighting its ability to better capture fine-grained semantic alignment. \n\n| Reward                                 | GenEval | DPG  | Avg.  |\n|----------------------------------------|---------|------|-------|\n| (#0) Baseline                          | 73.0    | 82.6 | 77.80 |\n| `CLIP score`                           | 74.2    | 82.4 | 78.30 |\n| (#2) $R_\\text{TIM}$                    | 74.1    | 83.0 | 78.55 |\n| `CLIP score +` $R_\\text{cycle}$        | 77.0    | 83.4 | 80.20 |\n| (#3) $R_\\text{TIM}$ + $R_\\text{cycle}$ | 77.3    | 83.9 | 80.60 |\n\n> **W2: The main claim of this work is that “co-RL is better than task-specific RL.” Why, then, is task-specific RL still necessary?**\n\n**R2:** Thank you for this insightful question. Our core argument is not that CoRL (Unified RL) is better than task-specific RL, but rather that Unified RL provides a much stronger foundation upon which task-specific RL can build to achieve the expected results. Specifically, our two-stage CoRL framework is easily understood through a `Foundation-then-Specialization` paradigm, which is both principled and practical for building powerful and versatile unified MLLMs. \n\n- **Stage 1 (Unified RL): The Foundation.** The main goal of this stage is not to outperform our task-specific method or existing leading MLLMs on the target benchmarks, but to build a powerful generalist foundation. By synergistically optimizing both understanding and generation tasks, this stage endows the unified MLLMs with more balanced/comprehensive capabilities that task-specific training alone cannot foster. As demonstrated in our ablation and pilot experiments, this unified RL training provides an overall accuracy boost across the board. \n\n- **Stage 2 (Task-specific/Refined RL): The Specialization.** This second stage is an effective and crucial specialization step, demonstrating the flexibility and extensibility of our CoRL framework across a wide variety of diverse target tasks. The Math$^\\text{We}$ benchmark (with text-rich geometric figures in images) is a perfect example, as it differs significantly from the unified RL data in terms of image types and reasoning types. \n    \n    | Method                          | Math$^\\text{We}$ | Hints                                                            |\n    |---------------------------------|------------------|------------------------------------------------------------------|\n    | Baseline                        | 5.9              | `Starting point`                                                 |\n    | + Unified RL                    | 14.0 (+8.1)      | `Unified RL builds a strong foundation.`                         |\n    | + Task-specific RL              | 15.8 (+9.9)      | `Specialization without a strong foundation is limited.`         |\n    | + Unified RL & Task-specific RL | 21.1 (+15.2)     | `Specialization on a strong foundation yields the best results.` | \n    \n    $\\blacktriangleright$ The results clearly show that our two-stage paradigm is far superior to task-specific methods. This demonstrates that unified RL provides a much stronger starting point for task-specific adaptation. \n\nTherefore, task-specific RL is a necessary and important step in our CoRL framework, leveraging the powerful foundation built by Unified RL to push the boundaries of performance. We will revise our paper to articulate this motivation more clearly. \n\n> **W3: Evaluation setup, and three-to-one merged model results.** \n\n**R3:** Thank you for the question. We are happy to clarify the evaluation setup and provide the corresponding results. \n\n- **Evaluation Setup.** For multimodal understanding, only on the three mixed benchmarks (Math$^\\text{VT}$, Math$^\\text{VS}$, Math$^\\text{Vis}$), whose test sets include both multiple-choice (MC) and open-ended (OE) QA formats, we merged the two task-specific models to obtain a final model capable of following both types of task instructions. \n\n- **Results of Three-to-One Merged Model.** We report the results in the table below. As shown, a naive \"three-to-one\" merge of all task-specific models leads to significant performance degradation in text-to-image generation, likely due to task imbalance or instruction interference or ambiguity. In contrast, its influence on multimodal understanding is less pronounced. \n    \n    | Benchmarks        | Task Type        | Three-to-One Merged | ULM-R1 |\n    |-------------------|------------------|---------------------|--------|\n    | GenEval           | image generation | 0.75                | 0.77   |\n    | WISE              | image generation | 0.29                | 0.33   |\n    | DPG               | image generation | 83.20               | 83.92  |\n    | MMMU              | MC               | 42.0                | 42.3   |\n    | MMStar            | MC               | 47.5                | 47.6   |\n    | Math$^\\text{We}$  | MC               | 21.5                | 21.1   |\n    | MMVet             | OE               | 43.0                | 43.9   |\n    | POPE              | OE               | 88.7                | 88.9   |\n    | Logic$^\\text{VT}$ | OE               | 35.0                | 34.5   |\n    | Math$^\\text{VT}$  | MC & OE          | 42.4                | 42.5   |\n    | Math$^\\text{VS}$  | MC & OE          | 25.4                | 25.4   |\n    | Math$^\\text{Vis}$ | MC & OE          | 22.2                | 22.0   |\n\n> **Q1: For $R_\\text{OE-Acc}$, does this refer to Exact Match?**\n\n**A1:** In the OE tasks, the accuracy reward is based on the correctness of the final extracted answer, not on a strict exact match of the entire generated text (which includes the reasoning process). For instance, for a question whose answer is \"3π\", we parse the final output within the `<answer>3π</answer>` tags and check if it matches the ground truth. This is a binary reward: 1 for a correct final answer, 0 otherwise. \n\n---\n\nWe hope these clarifications and additional results address your concerns. Thank you again for your time and valuable feedback."}, {"review_id": "K3rNGT119p", "reviewer": "Reviewer_5YVq", "summary": "This paper introduces CoRL, a co-evolutionary reinforcement learning framework designed to simultaneously enhance the understanding and generation capabilities of Unified Multimodal Large Language Models (ULMs). The authors' core contribution is a two-stage RL procedure: a unified stage where the model is jointly optimized on both understanding and generation tasks using a composite reward signal, followed by a refined stage for task-specific enhancement. To facilitate this, the paper proposes novel verifiable rewards for text-to-image generation, namely a bidirectional cycle consistency reward  and a text-image matching reward. The authors apply CoRL to a baseline ULM, creating ULM-R1, and demonstrate its effectiveness through extensive experiments. The resulting model shows significant performance improvements over the baseline on a wide array of multimodal understanding and generation benchmarks, particularly in complex reasoning tasks.", "questions": "refer to weakness", "limitations": "refer to weakness", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper presents a compelling and timely investigation into a significant challenge in multimodal AI. The strengths are considerable and warrant its acceptance, though there are a couple of major weaknesses that should be addressed for the final camera-ready version.\n\nStrengths:\n\nThe problem of synergistically training a single model for both multimodal understanding and generation is highly relevant. The proposed CoRL framework, which leverages reinforcement learning for this co-evolution, is an original and promising direction. The initial pilot study provides a solid motivation for pursuing a unified RL approach over separate or cyclical training strategies.\nThe performance gains of ULM-R1 are substantial, especially the notable improvements on complex reasoning benchmarks like WeMath and LogicVista. This demonstrates that the proposed framework does more than just improve surface-level metrics; it enhances deeper reasoning capabilities. The ablation studies effectively validate the contribution of the overall framework and the newly designed reward functions.\nThe paper is well-written and clearly structured. The CoRL framework is explained lucidly, aided by a helpful overview in Figure 2. The methodology and experimental setup are described with enough detail to allow for a good understanding of the work.\nWeaknesses:\n\nThe paper's proposed text-image matching reward, is calculated by \"leveraging the ULM itself to assess cross-modal alignment\". This introduces a significant risk of endogeneity. If the policy model's initial alignment capabilities are imperfect or biased, using it as its own evaluator could create a self-reinforcing feedback loop. The model may learn to generate images that are not objectively better but are simply easier for its own internal representations to score highly. This could lead to convergence on a flawed policy that has learned to \"game\" its own reward system. While the results are strong, it is difficult to ascertain how much of the improvement is genuine and how much might be an artifact of this self-evaluation.\nThe unified reward function, combines four distinct and heterogeneous reward components. The paper lacks a critical discussion on two fronts: 1. The components are derived from different metrics (e.g., perceptual distance like LPIPS, semantic scores like SPICE, and binary accuracy rewards) and likely operate on vastly different numerical scales. Simply summing them, even with a weighting factor \n𝜆, could lead to one component unintentionally dominating the total reward signal if its scale is much larger than the others. The authors do not specify if any form of normalization is applied to each component before summation. 2. The choice of \n𝜆=0.8 is presented without justification. The framework's performance may be highly sensitive to this value. A small change could alter the balance between generation and understanding rewards, potentially leading to drastically different outcomes.", "was_revised": true, "final_justification": "Thanks for the author's detail response. I'll keep the score.", "rebuttal": "Dear Reviewer 5YVq,\n\nWe sincerely thank you for your constructive and insightful review. We are delighted that you found our work to be compelling, novel, and important in multimodal AI, and we deeply appreciate your positive assessment and recommendation for acceptance. We respond to your identified weaknesses and suggestions below.\n\n> **W0: Endogeneity in $R_\\text{TIM}$ due to self-evaluation.** \n\n**R0:** This is a very perceptive and insightful point that touches upon an inherent challenge in self-rewarding RL. We are grateful you raised it. In our work, $R_\\text{TIM}$ was indeed computed using the policy model itself, but the total unified reward is a composite signal. To a certain extent, the potential endogeneity is balanced by the other fully external rewards: $R_\\text{cycle}$, $R_\\text{Acc}$, and $R_\\text{Format}$. This diverse reward structure inherently provides multi-faceted feedback and acts as a form of regularization, preventing the policy model from simply \"gaming\" the self-evaluated $R_\\text{TIM}$ signal. \n\nFurthermore, we conducted additional experiments comparing $R_\\text{TIM}$ with external metrics such as the CLIP score on text-to-image benchmarks. $R_\\text{TIM}$ demonstrates stronger alignment with dense prompts and better downstream performance, suggesting that it captures a more robust alignment signal beyond merely overfitting to internal representations.\n\n|                | GenEval | DPG  | Avg.  |\n|----------------|---------|------|-------|\n| Baseline       | 73.0    | 82.6 | 77.80 |\n| `CLIP score`   | 77.0    | 83.4 | 80.20 |\n| $R_\\text{TIM}$ | 77.3    | 83.9 | 80.60 |\n\n> **W1: Discussion on reward scaling.** \n\n**R1:** Thank you for pointing out this lack of clarity regarding a critical implementation detail. In practice, each reward from $R_\\text{cycle}$, $R_\\text{TIM}$, $R_\\text{Acc}$, and $R_\\text{Format}$ is normalized to the range $[0, 1]$ before aggregation. This ensures that all rewards operate within comparable scales and prevents any single component from dominating due to scale differences. We will clarify this normalization process more explicitly in the revised version. \n\n> **W2: Discussion on the sensitiveness of hyperparameter λ.** \n\n**R2:** Thank you for raising this point. The choice of λ = 0.8 was determined empirically. To provide a more formal justification, we have conducted the following ablation study on the hyperparameter λ. \n\n| λ   | GenEval (Gen.) | MMMU (Und.) | Avg.  |\n|-----|----------------|-------------|-------|\n| 0.5 | 77.1           | 41.0        | 59.05 |\n| 0.8 | 77.3           | 42.3        | 59.80 |\n| 1.0 | 76.9           | 43.0        | 59.95 |\n\nThe results show that λ = 0.8 yields the best generation performance while only slightly affecting understanding capabilities, whereas setting λ = 1.0 slightly boosts understanding at the cost of generation performance. \nMoreover, we agree that the model's sensitivity to λ warrants further analysis, and we will include an ablation sweep in the camera-ready version to explore this sensitivity more systematically.  \n\n---\n\nThank you once again for your constructive feedback and affirmation of our paper. We will incorporate these valuable suggestions and additional ablations in the camera-ready version."}, {"review_id": "zQeO8lFzq9", "reviewer": "Reviewer_wJWf", "summary": "This paper explores how to use reinforcement learning to improve both generation and understanding in unified MLLMs. The authors propose a method called CoRL, which uses a shared policy optimization process to train both capabilities together. CoRL includes two stages: a unified RL stage that jointly improves both generation and understanding, and a refined RL stage that focuses on specific tasks. Based on this approach, they train a model called ULM-R1, which shows clear improvements on T2I generation tasks and multimodal understanding benchmarks.", "questions": "Same as weaknesses, if the authors can resolve my concern from the weaknesses, especially the first one, I will raise my score.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n1. The paper proves that jointly optimizing both generation and understanding with RL rewards designed for both yields better performance than other alternatives, which is intuitive and a step towards unified MLLMs.\n2. The experiment results show great improvements compared to the base Janus-pro model, especially in understanding task.\n\n**Weaknesses**:\n1. In ablation study (Table 3), the SFT cold-start baseline only makes use of understanding data, which can barely help the generation task. The results for comparison on 1&2 and 6&7 seem expected, and they have similar performance on understanding tasks, sometimes SFT cold-start even has better results. The unfairness of the comparison makes the performance gain of unified-RL suspicious.\n2. The experiments only are only conducted on 1.5 B model, the scalability of the method is not verified.", "was_revised": true, "final_justification": "I have read the rebuttals from the authors, and after discussion with authors, I decide to raise my score.", "rebuttal": "Dear Reviewer wJWf,\n\nThank you for your comprehensive review and for recognizing the intuitive nature and significance of our work. We are particularly grateful for your clear feedback and the opportunity to resolve your concerns, especially regarding the fairness of the SFT cold-start baseline. We respond to your concerns and questions below.\n\n---\n\n> **W1: Concerns on SFT cold-start baseline.** \n\n**R1:** Thank you for raising this important point. We would like to clarify that the SFT cold-start baseline was indeed trained on both understanding and generation tasks to ensure a relatively fair comparison. \n\n- **Training Data.** As described in the caption of Table 3, the SFT cold-start data consists of x2x_rft_22k, mcot_r1_mcq (22K), and mcot_r1_vqa (22K). Among them, x2x_rft_22k is the same dataset used in our unified RL training and includes both generation and understanding tasks. Therefore, the SFT cold-start baseline does not only make use of understanding data. However, our initial attempts showed that using only x2x_rft_22k as the SFT cold-start data was almost ineffective. Even worse, it significantly degraded the model’s multimodal understanding capabilities. To address this, we ultimately used x2x_rft_22k, mcot_r1_mcq (22K), and mcot_r1_vqa (22K). \n\n- **Training Objective.** During the SFT cold-start stage, we follow Janus-Pro's training objective and use the cross-entropy loss to simultaneously optimize the model's generation and understanding capabilities. \n\n- **Results.** You made an astute observation. In practice, we did observe that SFT cold-start performs better than unified RL on certain understanding tasks, such as Math$^\\text{We}$ and Logic$^\\text{VT}$. This is likely due to task imbalance and the next-token prediction mode of autoregressive LLMs during SFT training, which is more direct and friendly for optimizing text generation than RL reward feedback. \n\n> **W2: Scalability of CoRL.** \n\n**R2:** We agree that verifying scalability is crucial. To address this, we conducted a new experiment using Janus-1.3B [1] as the baseline, given the limited time during the rebuttal period. The results show significant and consistent improvements across different types of benchmarks. We will incorporate further validation on larger-scale models in our open-source project and the camera-ready version. \n\n|            | GenEval | WISE | DPG   | MMMU | MMStar | Math$^\\text{We}$ | MMVet | POPE | Logic$^\\text{VT}$ |\n|------------|---------|------|-------|------|--------|------------------|-------|------|-------------------|\n| Janus-1.3B | 0.61    | 0.23 | 79.68 | 30.5 | 37.6   | 3.4†             | 34.3  | 87.0 | 23.9†             |\n| + CoRL     | 0.64    | 0.26 | 80.92 | 34.6 | 41.9   | 16.4             | 36.9  | 88.1 | 27.0              |\n\n[1] Chengyue Wu, Xiaokang Chen, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. In CVPR, 2025. \n\n---\n\nWe hope these clarifications and additional results address your concerns. Thank you once again for your time and valuable feedback."}, {"review_id": "8mpN6zaXbh", "reviewer": "Reviewer_hSgy", "summary": "The paper introduces ​​CoRL (Co-Reinforcement Learning)​​, a novel two-stage reinforcement learning framework designed to synergistically enhance ​​multimodal understanding​​ and ​​generation​​ (e.g., text-to-image synthesis) capabilities within unified multimodal large models (ULMs). This work adopts GRPO as the reinforcement learning method and finds that jointly optimizing the two tasks yields better performance compared to optimizing each task separately. The final model achieves state-of-the-art results on multiple benchmarks.", "questions": "As discussed above, overall, this is a paper on a unified model + GRPO. Since the final results are indeed promising, I consider it a paper that meets the passing standard. The main issue lies in the Stage 2 task-specific RL setup, which contradicts the core observations of the paper. If I have misunderstood this aspect, I would welcome further discussion with the authors.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. This paper introduces two novel rewards — bidirectional cycle consistency reward and text-image matching reward. Ablation studies further demonstrate that both rewards play a crucial role in enhancing the model’s text-to-image generation capabilities.\n2. This paper compares the performance of joint optimization versus separate optimization, aligning with the original intention of the unified generation model — achieving synergistic optimization between multimodal understanding and image generation, where the two tasks mutually enhance each other.\n3. The paper is well-written, with clear illustrations and well-designed tables.\n\n**Weaknesses**\n1. The two-stage RL setup in this work seems somewhat peculiar. While the paper advocates for a unified optimization approach, and both ablation studies and pilot exploration demonstrate that unified optimization outperforms separate optimization, stage 2 nevertheless employs task-specific refined reinforcement learning. This inconsistency with the paper’s main thesis is somewhat puzzling.\n2. The paper lacks certain ablation studies, particularly regarding the configuration of the R-cycle. For instance, why was LPIPS chosen as the reward metric for image similarity, rather than alternatives like PSNR, SSIM, or even MSE? Experimental validation supporting this choice would strengthen the argument.\n3. The Related Work section of this paper lacks citations to some relevant papers, such as MetaMorph, LlamaFusion, OmniMamba and BLIP3-o. Including references to these papers would help make this section more comprehensive.", "was_revised": true, "final_justification": "The author's response has addressed my concerns, and I will maintain my original score of 4.", "rebuttal": "Dear Reviewer hSgy,\n\nWe sincerely thank you for your thorough review, thoughtful feedback, and positive remarks! We greatly appreciate your recognition of our novel reward designs, significant results, and clear writing. We respond to your concerns and questions below. \n\n---\n\n> **W1 & Q1: Two-stage RL setup.** \n\n**R1:** Thank you for raising this excellent point. It gives us the opportunity to clarify the core motivation behind the two-stage design of CoRL, which we view as a key strength of our framework rather than a contradiction. Specifically, our two-stage CoRL framework is easily understood through  a `Foundation-then-Specialization` paradigm, which is both principled and practical for building powerful and versatile unified MLLMs. \n\n- **Stage 1 (Unified RL): The Foundation.** The main goal of this stage is not to outperform our task-specific method or existing leading MLLMs on the target benchmarks, but to build a powerful generalist foundation. By synergistically optimizing both understanding and generation tasks, this stage endows the unified MLLMs with balanced and comprehensive capabilities. As shown in our ablation and pilot experiments, this unified RL training provides an overall accuracy boost across the board. \n\n- **Stage 2 (Task-specific/Refined RL): The Specialization.** This second stage is an effective and crucial specialization step, demonstrating the flexibility and extensibility of our CoRL framework across a wide variety of diverse target tasks. The Math$^\\text{We}$ benchmark (with text-rich geometric figures in images) is a perfect example, as it differs significantly from the unified RL data in terms of image types and reasoning types. \n    \n    | Method                          | Math$^\\text{We}$ | Hints                                                            |\n    |---------------------------------|------------------|------------------------------------------------------------------|\n    | Baseline                        | 5.9              | `Starting point`                                                 |\n    | + Unified RL                    | 14.0 (+8.1)      | `Stage 1 builds a strong foundation.`                            |\n    | + Task-specific RL              | 15.8 (+9.9)      | `Specialization without a strong foundation is limited.`         |\n    | + Unified RL & Task-specific RL | 21.1 (+15.2)     | `Specialization on a strong foundation yields the best results.` | \n    \n    $\\blacktriangleright$ The results clearly show that our two-stage paradigm is far superior to task-specific methods. This demonstrates that the unified RL foundation provides a much higher starting point for task-specific adaptation. \n\nIn summary, we argue that this two-stage paradigm effectively resolves the tension between general capability and specialized excellence. It offers a scalable and practical pathway for extending unified MLLMs to a diverse and continually expanding range of applications. We will revise our paper to make this motivation clearer. \n\n> **W2: Ablation study on $\\mathcal{R}_{\\text{cycle}}$.** \n\n**R2:** Thank you for this valuable suggestion. We selected LPIPS for $\\mathcal{R}_{\\text{cycle}}$ primarily because it aligns better with human perceptual judgments than pixel-level metrics such as PSNR and SSIM. LPIPS measures image similarity in a feature space, making it more robust to minor, semantically irrelevant variations and better suited for rewarding high-level semantic consistency. Due to the short rebuttal period, we were only able to provide a comparison between LPIPS and MSE, but we will add a complete ablation study including PSNR and SSIM in the camera-ready version. \n\n|       | GenEval | DPG   | WISE |\n|-------|---------|-------|------|\n| MSE   | 0.77    | 83.15 | 0.30 |\n| LPIPS | 0.77    | 83.92 | 0.33 |\n\nAs shown in the table, LPIPS achieves better performance on both the DPG and WISE benchmarks compared to MSE, indicating its superiority in measuring semantic consistency. \n\n> **W3: More related work.** \n\n**R3:** Thank you for pointing out these relevant works. We have updated the Related Work section to include them. Specifically, we have added MetaMorph, LMFusion, and BLIP3-o to `AR-Diff`, and OmniMamba to `F-AR`. \n\n---\n\nWe hope these clarifications address your concerns. Thank you again for your constructive feedback and affirmation of our paper."}]

Author rebuttal:

Meta-review: The proposed CoRL framework—leveraging Group Relative Policy Optimization (GRPO) to synergistically enhance multimodal understanding and generation within a single model—addresses a critical gap in unified multimodal learning. The two-stage design (unified RL for foundational co-optimization followed by task-specific refinement) proves essential: while joint training boosts generalizability (e.g., +8.1 MathWe accuracy), task-specific tuning further amplifies gains (e.g., +15.2 MathWe), demonstrating that specialization builds upon rather than contradicts the unified foundation.

Reviewers initially raised valid concerns—particularly regarding reward function endogeneity (Reviewer 5YVq) and scalability (Reviewer wJWf)—but authors convincingly resolved these through rebuttal:

-- The Text-Image Matching (TIM) reward’s self-referential risk was mitigated by ​multi-reward regularization​ and empirical validation against CLIP scores, showing TIM’s superiority on dense prompts (DPG benchmark: 83.9 vs. 83.4).

-- Scalability concerns were addressed via ​LoRA-accelerated 7B experiments, where CoRL retained gains (+4.6 MMMU), confirming framework extensibility.

-- The novel ​bidirectional cycle consistency reward​ (validated against MSE/SSIM alternatives) and ​task-aware reward balancing​ (λ=0.8 optimized via ablation) reflect rigorous design.

Notably, all reviewers acknowledged the work’s strength post-rebuttal, with Reviewer 9e1d highlighting "resolved concerns" and Reviewer hSgy praising "clear illustrations and well-designed tables." The 23% understanding gain and 7% generation improvement across 12 benchmarks—including complex reasoning tasks (MathWe) and text-rich generation (DPG)—set a new state-of-the-art for unified multimodal models.

For camera-ready, authors are suggested to expand the related work section per Reviewer hSgy. This work pioneers co-reinforcement learning for multimodal synergy and offers a replicable blueprint for RL-driven multimodal unification.
Decision: Accept (spotlight)