Title: Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: multimodal interleaved generation

unified models

reinforcement learning
Abstract: Unified vision-language models have made significant progress in multimodal understanding and generation, yet they largely fall short in producing multimodal interleaved outputs, which is a crucial capability for tasks like visual storytelling and step-by-step visual reasoning.
In this work, we propose a reinforcement learning-based post-training strategy to unlock this capability in existing unified models, without relying on large-scale multimodal interleaved datasets.
We begin with a warm-up stage using a hybrid dataset comprising curated interleaved sequences and limited data for multimodal understanding and text-to-image generation, which exposes the model to interleaved generation patterns while preserving its pretrained capabilities.
To further refine interleaved generation, we propose a unified policy optimization framework that extends Group Relative Policy Optimization (GRPO) to the multimodal setting.
Our approach jointly models text and image generation within a single decoding trajectory and optimizes it with our novel hybrid rewards covering textual relevance, visual-text alignment, and structural fidelity.
Additionally, we incorporate process-level rewards to provide step-wise guidance, enhancing training efficiency in complex multimodal tasks.
Experiments on MMIE and InterleavedBench demonstrate that our approach significantly enhances the quality and coherence of multimodal interleaved generation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a reinforcement learning-based post-training strategy that enables unified vision-language models to generate coherent, interleaved sequences of text and images, introducing a warm-up phase with limited interleaved data and a novel multimodal extension of Group Relative Policy Optimization with hybrid and process-level rewards, and demonstrates significant improvements on dedicated interleaved generation benchmarks without sacrificing general multimodal understanding or generation capabilities.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper addresses an important and underexplored challenge: enabling unified vision-language models to produce truly interleaved multimodal outputs (text and images), which is crucial for applications like visual storytelling and step-by-step reasoning.\n\n2. The method does not require large-scale, high-quality interleaved datasets. Instead, it leverages limited interleaved data and a well-designed warm-up phase, making it practical and accessible for real-world applications.\n\nWeakness:\n\n1. While the method is empirically effective, the authors offer limited theoretical analysis or formal justification for why the proposed multimodal GRPO and reward design are particularly suited to interleaved generation.\n\n2. The process-level reward modeling and the need to generate multiple candidate sequences per prompt in GRPO introduce substantial computational costs, which may limit scalability.\n\n3. The approach relies heavily on the effectiveness and robustness of multiple reward models (textual, visual, and structural) during training, but the authors do not thoroughly analyze how these reward models generalize to out-of-distribution scenarios or novel tasks, potentially limiting the method’s adaptability.\n\n4. Tables 3, 4, 5, and 6 are poorly presented with inconsistent sizes and font styles, making the paper appear unpolished and disorganized.\n\nQUESTIONS:\n1. Process-level rewards and group-wise sampling increase training costs. Can you provide more quantitative analysis on the computational and memory overhead introduced by the GRPO stage (e.g., compared to standard RLHF or supervised fine-tuning)? Please also discuss potential strategies for reducing this overhead, such as more efficient candidate sampling or reward estimation.\n\n2. Could you provide more details or experiments on the robustness and generalization ability of these reward models, especially when applied to out-of-distribution data or tasks beyond the current benchmarks? If not already included, consider testing or discussing how the method would adapt if reward models are imperfect or biased.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBased on unified VLMs, this paper expands their capability to generate interleaved text and images instead of either understanding or generation. The authors utilize GRPO to post-train VLMs with textual, visual and format rewards together with progress rewards after a warmup stage to enable interleaved outputs. They conduct experiments on MMIE and InterleavedBench, and achieve performance gains compared to original VLM base model.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n1. The paper enables the capability of interleaved image and text generation with GRPO post-training for unified VLMs, and achieves improved performance compared to the baseline.\n2. The method is intuitive and easy to implement.\n3. The structure/writing of the paper is easy to follow.\n\n**Weaknesses**:\n1. The experiment is only conducted on Vila-u, which is less powerful than other baselines as shown in Table 5 for multimodal understanding or generation, the generalization of the proposed method on the more powerful unified VLM is not demonstrated or proved.\n2. The compared baselines in main results (Table 1 and Table 2) are somehow outdated, which is lack of more powerful baselines like EMU3 [1] to ensure a more comprehensive evaluation.\n3. There is limited insight the community can gain after reading this paper. It is expected that after GRPO with combined rewards, the model can achieve better performance on interleaved generation for text and images, adding a baseline of SFT only post-train and comparing it with RL post-train may be more insightful.\n\n[1] Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y., Wang, J., Zhang, F., Wang, Y., Li, Z., Yu, Q., Zhao, Y., Ao, Y., Min, X., Li, T., Wu, B., Zhao, B., Zhang, B., Wang, L., Liu, G., He, Z., Yang, X., Liu, J., Lin, Y., Huang, T., & Wang, Z. (2024). Emu3: Next-Token Prediction is All You Need. ArXiv, abs/2409.18869.\n\nQUESTIONS:\n1. Can more comprehensive evaluations be conducted as stated in weaknesses?\n2. Can a baseline of pure SFT post-train be added to provide more insights on the advantage of RL-based post-training at the domain of interleaved text and image generation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a policy optimization framework for unified multimodal models that significantly enhances their capability for interleaved generation. To achieve this objective, the method employs a warm-up stage using hybrid datasets to develop initial interleaved generation capabilities, followed by policy optimization using an extended GRPO framework. The optimization process incorporates a comprehensive multi-type rewards that evaluates textual relevance, visual-text alignment, and structural fidelity to guide the training process effectively. Experimental results on established interleaved generation benchmarks demonstrate the effectiveness of the proposed method, showing substantial improvements in both generation quality and coherence compared to existing approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The construction pipeline for the interleaved data used during warm-up is well-designed and robust.\n- The final performance of interleaved generation demonstrates strong quality and coherence.\n\nWeaknesses:\n- The proposed framework lacks novelty. The GRPO algorithm remains unchanged, still producing token-by-token outputs without any modifications. As a result, this framework is only applicable to autoregressive (AR)-based models such as Janus-Pro. It cannot be applied to other mainstream unified models such as Transfusion[1], MetaQuery [2], or BAGEL [3].\n- The use of multiple rewards is rather straightforward and does not constitute a significant contribution.\n- The interleaved generation primarily focuses on coherence and creativity, but does not demonstrate capabilities such as interleaved reasoning or other complex interleaved scenarios, which limits its applicability.\n\n[1] Zhou, Chunting, et al. \"Transfusion: Predict the next token and diffuse images with one multi-modal model.\" arXiv preprint arXiv:2408.11039 (2024).\n\n[2] Pan, Xichen, et al. \"Transfer between modalities with metaqueries.\" arXiv preprint arXiv:2504.06256 (2025).\n\n[3] Deng, Chaorui, et al. \"Emerging properties in unified multimodal pretraining.\" arXiv preprint arXiv:2505.14683 (2025).\n\nQUESTIONS:\n- Regarding Table 3, I would like to see the performance of the model when applying GRPO directly without the warm-up stage. While the model can generate interleaved outputs even without warm-up (albeit of poor quality), it would be valuable to understand how much the warm-up stage contributes to the final performance by comparing the direct GRPO approach against the full two-stage pipeline.\n\n- The ablation study on group size in GRPO presented in Table 6 appears limited in scope. Given that recent work such as DeepSeek-R1 [1] has explored larger group sizes (8, 12, 16), it would be beneficial to extend this analysis to investigate whether larger group sizes could further improve performance. This expanded ablation would provide better insights into the scalability of the approach and help determine the optimal group size configuration for multimodal interleaved generation tasks.\n\n- The training process for interleaved generation appears to primarily benefit the interleaved output capability without demonstrating clear improvements to the model's core image understanding abilities. Could the authors provide insights into whether this specialized training offers any additional benefits beyond interleaved generation? For instance, does the multi-modal policy optimization process enhance visual reasoning, cross-modal alignment, or other fundamental multimodal understanding capabilities?\n\n[1] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" arXiv preprint arXiv:2501.12948 (2025).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
- This paper proposes a reinforcement learning-based post-training strategy to enable existing unified vision-language models to generate multimodal interleaved outputs.
Specifically, the paper proposes a unified policy optimization framework that extends Group Relative Policy Optimization (GRPO) to the multimodal setting.
- The paper has conducted experiments on two challenging benchmarks, MMIE and InterleavedBench, demonstrating the effectiveness of their proposed approach
- The paper has provided comprehensive ablation studies showing the effectiveness of submodules.

STRENGTHS AND WEAKNESSES:
**Strengths**
- The paper proposes an effective RL training strategy to enable unified vision-language models to perform high-quality interleaved multimodal generation. 
- The paper introduces a hybrid group-wise reward signal composed of textual relevance, visual-text alignment, and structural fidelity—addressing multiple aspects of interleaved generation quality.
- The proposed method is evaluated on two challenging benchmarks, MMIE and InterleavedBench, demonstrating the effectiveness of their proposed approach.

**Weaknesses**
- The computation of the process-level reward (L200) is unclear to me. Is it computed using modality-specific segment as target output, and follows the same reward computation strategy as in Eq. 7? 
- In Table 5, the proposed method underperforms compared to VILA-U on MMVet and POPE

QUESTIONS:
I wonder if the authors have tried the version without the warmup stage, how does the model perform?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03