Title: Co-Reinforcement Learning for Unified Multimodal Understanding and Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

GRPO

Unified Multimodal Understanding and Generation
Abstract: This paper presents a pioneering exploration of reinforcement learning (RL) via group relative policy optimization for unified multimodal large language models (ULMs), aimed at simultaneously reinforcing generation and understanding capabilities. Through systematic pilot studies, we uncover the significant potential of ULMs to enable the synergistic co-evolution of dual capabilities within a shared policy optimization framework. Building on this insight, we introduce \textbf{CoRL}, a \textbf{Co}-\textbf{R}einforcement \textbf{L}earning framework comprising a unified RL stage for joint optimization and a refined RL stage for task-specific enhancement. With the proposed CoRL, our resulting model, \textbf{ULM-R1}, achieves average improvements of 7\% on three text-to-image generation datasets and 23\% on nine multimodal understanding benchmarks. These results demonstrate the effectiveness of CoRL and highlight the substantial benefits of reinforcement learning in facilitating cross-task synergy and optimization for ULMs. Code is available at \url{https://github.com/mm-vl/ULM-R1}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ​​CoRL (Co-Reinforcement Learning)​​, a novel two-stage reinforcement learning framework designed to synergistically enhance ​​multimodal understanding​​ and ​​generation​​ (e.g., text-to-image synthesis) capabilities within unified multimodal large models (ULMs). This work adopts GRPO as the reinforcement learning method and finds that jointly optimizing the two tasks yields better performance compared to optimizing each task separately. The final model achieves state-of-the-art results on multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. This paper introduces two novel rewards — bidirectional cycle consistency reward and text-image matching reward. Ablation studies further demonstrate that both rewards play a crucial role in enhancing the model’s text-to-image generation capabilities.\n2. This paper compares the performance of joint optimization versus separate optimization, aligning with the original intention of the unified generation model — achieving synergistic optimization between multimodal understanding and image generation, where the two tasks mutually enhance each other.\n3. The paper is well-written, with clear illustrations and well-designed tables.\n\n**Weaknesses**\n1. The two-stage RL setup in this work seems somewhat peculiar. While the paper advocates for a unified optimization approach, and both ablation studies and pilot exploration demonstrate that unified optimization outperforms separate optimization, stage 2 nevertheless employs task-specific refined reinforcement learning. This inconsistency with the paper’s main thesis is somewhat puzzling.\n2. The paper lacks certain ablation studies, particularly regarding the configuration of the R-cycle. For instance, why was LPIPS chosen as the reward metric for image similarity, rather than alternatives like PSNR, SSIM, or even MSE? Experimental validation supporting this choice would strengthen the argument.\n3. The Related Work section of this paper lacks citations to some relevant papers, such as MetaMorph, LlamaFusion, OmniMamba and BLIP3-o. Including references to these papers would help make this section more comprehensive.\n\nQUESTIONS:\nAs discussed above, overall, this is a paper on a unified model + GRPO. Since the final results are indeed promising, I consider it a paper that meets the passing standard. The main issue lies in the Stage 2 task-specific RL setup, which contradicts the core observations of the paper. If I have misunderstood this aspect, I would welcome further discussion with the authors.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

comments: SUMMARY:
This paper explores how to use reinforcement learning to improve both generation and understanding in unified MLLMs. The authors propose a method called CoRL, which uses a shared policy optimization process to train both capabilities together. CoRL includes two stages: a unified RL stage that jointly improves both generation and understanding, and a refined RL stage that focuses on specific tasks. Based on this approach, they train a model called ULM-R1, which shows clear improvements on T2I generation tasks and multimodal understanding benchmarks.

STRENGTHS AND WEAKNESSES:
**Strengths**:
1. The paper proves that jointly optimizing both generation and understanding with RL rewards designed for both yields better performance than other alternatives, which is intuitive and a step towards unified MLLMs.
2. The experiment results show great improvements compared to the base Janus-pro model, especially in understanding task.

**Weaknesses**:
1. In ablation study (Table 3), the SFT cold-start baseline only makes use of understanding data, which can barely help the generation task. The results for comparison on 1&2 and 6&7 seem expected, and they have similar performance on understanding tasks, sometimes SFT cold-start even has better results. The unfairness of the comparison makes the performance gain of unified-RL suspicious.
2. The experiments only are only conducted on 1.5 B model, the scalability of the method is not verified.

QUESTIONS:
Same as weaknesses, if the authors can resolve my concern from the weaknesses, especially the first one, I will raise my score.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces CoRL, a co-evolutionary reinforcement learning framework designed to simultaneously enhance the understanding and generation capabilities of Unified Multimodal Large Language Models (ULMs). The authors' core contribution is a two-stage RL procedure: a unified stage where the model is jointly optimized on both understanding and generation tasks using a composite reward signal, followed by a refined stage for task-specific enhancement. To facilitate this, the paper proposes novel verifiable rewards for text-to-image generation, namely a bidirectional cycle consistency reward  and a text-image matching reward. The authors apply CoRL to a baseline ULM, creating ULM-R1, and demonstrate its effectiveness through extensive experiments. The resulting model shows significant performance improvements over the baseline on a wide array of multimodal understanding and generation benchmarks, particularly in complex reasoning tasks.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper presents a compelling and timely investigation into a significant challenge in multimodal AI. The strengths are considerable and warrant its acceptance, though there are a couple of major weaknesses that should be addressed for the final camera-ready version.\n\nStrengths:\n\nThe problem of synergistically training a single model for both multimodal understanding and generation is highly relevant. The proposed CoRL framework, which leverages reinforcement learning for this co-evolution, is an original and promising direction. The initial pilot study provides a solid motivation for pursuing a unified RL approach over separate or cyclical training strategies.\nThe performance gains of ULM-R1 are substantial, especially the notable improvements on complex reasoning benchmarks like WeMath and LogicVista. This demonstrates that the proposed framework does more than just improve surface-level metrics; it enhances deeper reasoning capabilities. The ablation studies effectively validate the contribution of the overall framework and the newly designed reward functions.\nThe paper is well-written and clearly structured. The CoRL framework is explained lucidly, aided by a helpful overview in Figure 2. The methodology and experimental setup are described with enough detail to allow for a good understanding of the work.\nWeaknesses:\n\nThe paper's proposed text-image matching reward, is calculated by \"leveraging the ULM itself to assess cross-modal alignment\". This introduces a significant risk of endogeneity. If the policy model's initial alignment capabilities are imperfect or biased, using it as its own evaluator could create a self-reinforcing feedback loop. The model may learn to generate images that are not objectively better but are simply easier for its own internal representations to score highly. This could lead to convergence on a flawed policy that has learned to \"game\" its own reward system. While the results are strong, it is difficult to ascertain how much of the improvement is genuine and how much might be an artifact of this self-evaluation.\nThe unified reward function, combines four distinct and heterogeneous reward components. The paper lacks a critical discussion on two fronts: 1. The components are derived from different metrics (e.g., perceptual distance like LPIPS, semantic scores like SPICE, and binary accuracy rewards) and likely operate on vastly different numerical scales. Simply summing them, even with a weighting factor \n𝜆, could lead to one component unintentionally dominating the total reward signal if its scale is much larger than the others. The authors do not specify if any form of normalization is applied to each component before summation. 2. The choice of \n𝜆=0.8 is presented without justification. The framework's performance may be highly sensitive to this value. A small change could alter the balance between generation and understanding rewards, potentially leading to drastically different outcomes.\n\nQUESTIONS:\nrefer to weakness\n\nLIMITATIONS:\nrefer to weakness", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n# Summary\n\nThe **pilot study** demonstrates the effectiveness of bidirectional RL for synergetic optimization. According to the pilot study, the authors propose a two-stage RL training strategy:\n\n**Stage 1: Unified RL Training**: In the first stage, the authors employ unified RL training using GRPO for both understanding and generation tasks, with Janus-Pro-1B serving as the policy model. The reward functions include:\n\n- $\\mathcal{R}_{\\text{cycle}} = 1 - \\operatorname{LPIPS}(\\mathcal{I}_{\\text{real}}, \\mathcal{I}_{\\text{gen}}) + \\operatorname{SPICE}(\\mathcal{P}_{\\text{org}}, \\mathcal{C}_{\\text{re-cap}})$, where BLIP is used as the re-captioner.\n- $\\mathcal{R}_{\\mathrm{TIM}} = \\frac{1}{2}\\left(\\frac{1}{L_i} \\sum_{j=1}^{L_i} \\max_{k \\in [1, L_t]} \\cos(\\boldsymbol{i}_j, \\boldsymbol{t}_k) + \\frac{1}{L_t} \\sum_{k=1}^{L_t} \\max_{j \\in [1, L_i]} \\cos(\\boldsymbol{t}_k, \\boldsymbol{i}_j)\\right)$, which measures token-level similarity rather than using coarse-grained CLIP scores.\n- Format Reward\n- Accuracy Reward\n\n**Stage II: Task-Specific RL Training**: In the second stage, RL training is tailored for specific tasks, with the following reward combinations:\n- T2I (Text-to-Image): $\\mathcal{R}_{\\text{cycle}} + \\mathcal{R}_{\\text{TIM}}$\n- MCQ (Multiple Choice Questions): $\\mathcal{R}_{\\text{MCQ-Acc}} + \\mathcal{R}_{\\text{Format}}$\n- Open-Ended Text Generation: $\\mathcal{R}_{\\text{OE-Acc}} + \\mathcal{R}_{\\text{Format}}$\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The methods introduced in this work are novel.\n- The experiments convincingly demonstrate the effectiveness of the proposed methodology.\n- The ablation studies are comprehensive and robust.\n\n# Weaknesses\n\n- For $\\mathcal{R}_{\\mathrm{TIM}}$, is there any ablation study comparing token-level similarity with the CLIP score?\n- The main claim of this work is that “co-RL is better than task-specific RL.” Why, then, is Task-Specific Refined Reinforcement Learning still necessary?\n- In Section 4.2, the evaluation of T2I generation is based on a non-merged task-specific model, whereas for multimodal understanding, a merged model (two text generation models) is used. What would the results be if a three-to-one merged model were evaluated for both tasks?\n- Please anonymize the Hugging Face URLs for the datasets to avoid potential desk rejection risks.\n\nQUESTIONS:\n# Questions\n\n- For $\\mathcal{R}_{\\text{OE-Acc}}$, does this refer to Exact Match?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-21"}