Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization
Authors: Ming Nie
Chunwei Wang
Jianhua Han
Hang Xu
Li Zhang
Venue: NeurIPS 2025 poster ()
Keywords: multimodal interleaved generation
unified models
reinforcement learning
Primary area: applications
TL;DR: We propose a unified policy optimization framework that supports autoregressive generation across both text and image modalities, enabling enhanced multimodal interleaved generation

Abstract: Unified vision-language models have made significant progress in multimodal understanding and generation, yet they largely fall short in producing multimodal interleaved outputs, which is a crucial capability for tasks like visual storytelling and step-by-step visual reasoning.
In this work, we propose a reinforcement learning-based post-training strategy to unlock this capability in existing unified models, without relying on large-scale multimodal interleaved datasets.
We begin with a warm-up stage using a hybrid dataset comprising curated interleaved sequences and limited data for multimodal understanding and text-to-image generation, which exposes the model to interleaved generation patterns while preserving its pretrained capabilities.
To further refine interleaved generation, we propose a unified policy optimization framework that extends Group Relative Policy Optimization (GRPO) to the multimodal setting.
Our approach jointly models text and image generation within a single decoding trajectory and optimizes it with our novel hybrid rewards covering textual relevance, visual-text alignment, and structural fidelity.
Additionally, we incorporate process-level rewards to provide step-wise guidance, enhancing training efficiency in complex multimodal tasks.
Experiments on MMIE and InterleavedBench demonstrate that our approach significantly enhances the quality and coherence of multimodal interleaved generation.

Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization
Ming Nie!
Chunwei Wang?
Jianhua Han
Hang Xu²
Li Zhang'
School of Data Science, Fudan University
Noah's Ark Lab, Huawei
Abstract
Unified vision-language models have made significant progress in multimodal understanding and generation, yet they largely fall short in producing multimodal interleaved ouiputs, which is a crucial capability for tasks like visual storytelling and step-by-step visual reasoning. In this work, we propose a reinforcement learning- based post-trann teatttraiiing strategy to unlock this capability in existing unified models, without relying on large-scale multimodal interleaved datasets. We begin with a warm-up stage using a hybrid dataset comprising curated interleaved sequences and limited data for multimodal understanding and text-to-image generation, which exposes the model to interleaved generation patterns while preserving its pretrained capabilities. To further refine interleaved generation, we propose unified policy optimization framework that extends Group Relative Policy Optimization (GRPO) to the multimodal setting. Our approach jointly models text and image genera- tion within single decoding trajectory and optimizes it with our novel hybrid rewards covering textual relevance, visual-text aligmment, and structural fidelity. Additionally, we incorporate process-level rewards to provide step-wise guidance, enhancing training efficiency in complex multimodal tasks. Experiments on MMIE and InterleavedBench demonstrate that our approach significantly enhances the quality and coherence of multimodal interleaved generation.
Introduction
In recent years, rapid progress in visual understanding and generation has driven a growing trend towards unifying these capabilities within single multimodal framework. Unified rision-language models aim to perform both understanding tasks (e.g.. g.. visual question answering) and generation tasks (e.g., text-to-imase no-image generation) within one model, representing a pivotal direction in the evolution of vision-language models. This paradigm has seen significant advancement, enabled by the availability of large-scale image-text paired datasets and the scaling of multimodal model architectures. Recent models such as Show-O, VILA-U, and ILLUME [34, 32, 28] demonstrate strong generalization across diverse benchmarks, underscoring the potential of unified frameworks general-purpose AI systems for integrated vision-language reasoning and content creation.
Despite their impressive performance, most unified models still struggle to generate multimodal interleaved contents, which is critical capability for enabling fine- ane-grained reasoning, step-by-step explanation, and context-awarr multimodal synthesis. During inference, these models typically produce either text-only or image-only outputs, constrained by explicit or implicit modality control mechanisms. This limitation arises primarily from the lack of fine-grained supervision and training data to guide dynamic modality transitions. Consequently, current models often default to generating dingle-modalig outputs, failing tasks that require tightly coupled multimodal sequences, such
Li Zhang (lizhangfd@ (udan.edu.cn) is the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

as visual dialogue or sequential storytelling, Addressing this gap is essential for realizing the full potential of unified models in seamless multimodal reasoning and generation.
To this end, we propose a post-training strategy that unlocks multimodal interleaved generation without requiring large-scale high-quality interleaved data. We hypothesize that unified models inherently possess fundamental multimodal generation capabilities acquired during pre-training and supervised fine-tuning (SFT), and that minimal interleaved supervision is sufficient to activate them. Based on this hypothesis, we first construct a hybrid dataset comprising a small amount of interleaved text-image sequences to expose the model to multimodal generation patterns, while incorporating limited SFT data to retain its pretrained strengths in multimodal understanding and conventional sext-t--mmaee generation. After this warm-up stage, the unified model capable of generating basic multimodal interleaved contents conditioned on instructions. However, the outputs often suffer from weak cross-modal alignment, reflecting limited consistency and coherence between text and image.
To further improve multimodal interleaved generation, we propose a reinforcement learning algorithm that formulates generation as a sequential decision-making process and extends Group Relative Policy Optimization [21] (GRPO) to the multimodal setting. While GRPO has been widely explored for text-only modalities, it struggles with multimodal outputs due to challenges like modality switching and hybrid reward attribution. To address this, we propose a unified policy optimization framework that jointly models text and image generation under a single decoding trajectory. This approach enables modality-aware aware decisions and leverages a shared training objective, preserving GRPO's advantages while adapting to multimodal structures. Specifically, we define hybrid group-wise reward signal consisting of three key components: (i) textual reward that evaluates the quality and relevance of the generated text conditioned on the input prompt; (ii) a visual and multimodal reward that jointly assesses image quality and image-text consistency; and (iii) a format reward that promotes structural fidelity by penalizing violations in the expected interleaved output format. In addition, we incorporate process-level reward modeling by assigning intermediate rewards at the interleaved generation steps. This design provides more granular and timely feedback throughout the generation process, which significantly improves the efficiency and effectiveness of policy learning. Such fine-grained supervision is particularly advantageous in complex multimodal generation tasks, where step-wise feedback helps the model perform autoregressive interleaved generation more effectively than relying solely on outcome rewards. We evaluate our approach on two dedicated multimodal interleaved generation benchmarks, namely MMIE [33] and InterleavedBench [14], and demonstrate its effectiveness in enabling unified models to generate coherent, high-quality interleaved outputs.
To summarize, our main contributions are as follows: (i) We introduce a warm-up stage that unlocks the model's latent capability for interleaved text-image generation using only a small amount of curated interleaved data; (ii) We propose a unified policy optimization framework that supports autoregressive generation across both text and image modalities, enabling seamless modality switch- ing within single decoding trajectory; (iii) We design group of hybrid rewards that supervises multimodal generation from multiple aspects, and further enhance learning via process-level level rewards that provide step-wise guidance; (iv) We conduct extensive experiments on two challenging bench- marks, MMIE and InterleavedBench, demonstrating the effectiveness and efficiency of our approach compared to existing unified models.
2 Related work
2.1 Unified understanding and generation
Recently, an increasing number of studies [34, 40, 26, 27, 32, 4, 31] have focused on unified models for both multimodal understanding and generation. Show-o [34], TransFusion [40] and Janus-Pro [4] strategies, while Emu2 [26] employs a fully autoregressive
adopt hybrid ciffusiion-atooegressit architecture that predicts the next multimodal element via classification for text and regression for visual embeddings. Chameleon [27], VILA-U [32], and other works [28, 31] tokenize images and interleave them with text tokens, enabling joint reasoning and autoregressive generation across modalities. Despite these efforts, current unified models often fail to support fine-grained interleaved generation of text and images, largely due to the scarcity of high-quality multimodal interleaved data. This limitation restricts their applicability in tasks that require coherent and context-aware multimodal dialogue and reasoning.

---

2.2 Multimodal interleaved generation
Multimodal learning has rapidly advanced, especially in the integration of text and image modalities. Early models such as DALL-E [19] and Stable Diffusion [16] demonstrated impressive capabilities in generating images from text prompts, but they were primarily designed for unidirectional generation, either text-to-image or image-to-text, without supporting interleaved multimodal outputs. Recently, large vision-language models (LVLMs) have begun to tackle this limitation by enabling interleaved generation, where text and images are jointly generated within a single autoregressive sequence. Approaches such as Chameleon [27]. GILL [9], Anole [5] and ARMOR [24]. have pushed the bound- aries of token-level multimodal modeling. These models adopt unified generation frameworks that allow seamless alternation between modalities, supporting more natural and interactive multimodal communication. However, a key challenge lies in the scarcity of large-scale, fine-grained multimodal supervision data, which limits the full potential of interleaved generation. To address this, we propose a post-training strategy that activates the model's multimodal generation capability without relying on extensive high-quality training data.
2.3 Policy optimization in multimodal models
Policy optimization [15, 18, 21, 8, 22, 30] has become pivotal component in aligning large language and vision vision-language models with human intent. Reinforcement learning from human feedback (RLHF) [15]. exemplified by methods such as PPO [20] and DPO [18], has proven effective in improving text-only generation by leveraging reward signals derived from preference data or learned reward models. Recently, Group Relative Policy Optimization (GRPO) [21] has demonstrated promising results in the post-training of large language models, particularly in domains requiring complex reasoning such mathematics [8, 22, 30]. Its group-wise comparative reward mechanism offers improved sample efficiency and stability, making it a compelling foundation for extension into multimodal interleaved generation. However, extending policy optimization to multimodal settings remains a substantial challenge. First, existing methods [41, 29, 3] are often limited to text-only modalities or apply separate optimization procedures for text and image components, which hinders the development of unified models capable of seamless interleaved generation. Second, most approaches rely on outcome-ories orieetted ood-of-sequent rewards, which are inherently sparse and fail to capture the fine-gre -mnagrriddd, step-wise alignment required for complex multimodal tasks. In this work, we propose a unified policy optimization algorithm, which treats multimodal generation as a single coherent decision-making process for enhanced multimodal policy optimization.
3 Methodology
In this section, we first present the preliminary formulation of unified multimodal models in 3.1. We then introduce warm-up training scheme to equip the model with the capability of generating interleaved text-image outputs while preserving its pretrained knowledge in Sec. 3.2. Next, in Sec. 3.3, we describe our proposed GRPO training framework based on hybrid reward signals tailored for multimodal generation. Finally, we outline the training details in Sec. 3.4.
3.1 Preliminary
Unified multimodal models seek to integrate vision and language by leveraging shared architecture capable of jointly modeling textual and visual modalities in both understanding and generation tasks. Given multimodal input sequence X {x1, ,, }. where each token 2; correspond to either a textual or visual token, the model learns to autoregressively generate an output sequence Y (y1,.. ..i...... }. which is also capable of comprising both modalities. The generation process in unified models is typically modeled as an autoregressive probability distribution over the output tokens:
M
p( p( = II
plun X, 0):
(I)
3

---

Completion Tokenss
Reference model
Policy model
Completion tokend
Text tokens
Discnete wauua tokend
decode
Token-level advantages
Completion
Text encooder
Vision encoder
Stepl: Wash your patatoesi under the cold water//thinio evis>
Textual/Vinnn/FFomad
</vis> <teinoo
Reward models
0.75/0.54/1.0
How to soute potatoes?
0.95/0.76/0.00
Completion
Predict the subsequent
Stepl: Cut the patatoes.
Hybrid rewords
Group relative adwentages
step. The previous steps Wash the potatoes.
Stepl
Trained models
Frozen models
Figure Overview of our reinforcement fine-tuning framework. Multimodal tokens are autore- gressively generated and decoded into completions, with token probabilities used to compute KL divergence along a single trajectory. Hybrid rewards are assigned to each completion, and token-level group relative advantages are calculated to guide policy optimization along with KL regularization.
where 8 denotes the model parameters, and the conditional generation each step relies both the input sequence and the previously generated tokens Yer- The unified model is generally optimized using standard language modeling objective covering multimodal sequences:
L
Lea (t)
+ (tng())loplggll (/)))))))))l.)))))))))
This framework allows for flexible training across various tasks simultaneously, such as captioning, image generation, and more complex multimodal reasoning. However, due to the lack of high-quality training data that explicitly supervises interleaved text-image generation, current unified models often struggle to realize their potential for multimodal generation, leading to limited and inconsistent performance in such tasks.
3.2 Unlocking interleaved generation
To equip unified models with interleaved generation and multimodal reasoning capabilities, we propose warm-up training scheme that activates new functionalities while preserving existing strengths. Unified models are pretrained on large-scale multimodal data comprising both text and images, theoretically equipping them for multimodal interleaved generation. However, they often struggle with smoothly transitioning between modalities and aligning context across text and image. Direct fine-tuning on novel tasks may substantially impair the model's pretrained competencies and induce catastrophic forgetting. To address this, we introduce hybrid data- based warm-up phase that bridges pretraining and task-specific fine-tuning mee-tuning. In this stage, we incorporate large corpus of high-quality supervised fine-tuning (SFT) data, including either curated human-labeled samples or distilled outputs from strong multimodal LLMs.. We further introduce interleaved text-image data to expose the model to multimodal generation patterns, while blending in purely textual data to preserve and reinforce language understanding and generation capabilities. This strategy helps unlock interleaved generation abilities without compromising the model's 's original strengths.

---

3.3 Reinforcement fine-tuning
Once the model is warmed up, we proceed GRPO- optimization phase to further enhance generation quality and cross-modal alignment. During reinforcement fine-tuning, we apply the GRPO algorithm guided by hybrid supervised signal comprising both rule-based metrics and model-based reward, as shown in Figure 1.
Unified policy for multimodal generation. To enhance interleaved text-image generation and multimodal reasoning, we adapt Group Relative Policy Optimization (GRPO) [21], an efficient reinforcement learning algorithm originally proposed for text-only LLMs. GRPO estimates token- level advantages by conducting intra-group comparisons among generated responses conditioned on the same query. Given an input instance X, behavior agent Φ0.r samples batch of G candidate responses [Yi)==1- However, previous applications of GRPO have been limited to optimizing text- only policies. To extend it to unified models capable of generating both text and images, we treat the multimodal generation as single decision process: Y; = {uf .....: Mi lzt ing 1 yin }. and adapt the GRPO framework accordingly. The GRPO optimization objective incorporates clipped surrogate loss term augmented with a KL-penalty to ensure stable policy updates, formulated as:
G
Y
(aRPO
min(Dun (1) Ai,, (4), -,, + +e)...)
(3)
G
where Duni represents the unified KL divergence computed per token across modalities:
To( X 5t) (t) (t)+ +
Te (VV. img Xi, Vi.
(tnc(t) Thol
((s((t).
(4)
( X1s Vi.ga)
Toaa (Vit [ Vi,st)
The advantage A., for the i-th response at time step t is determined by normalizing the rewards obtained across the response group, according to:
Ace r(X, - mewmm (r YI), (i),,.,,(x,,,xxxx)) r(X,))))) std([r(X, Vi), (o.,,,XX,,Y, YG)])
(5)
The hyperparameter controls the permissible deviation from the reference policy. The clipping function constrains the policy ratio within a specified interval, preventing overly large updates. This regularization mechanism enhances training stability and reduces the risk of performance collapse due to aggressive policy shifts.
Hybrid reward signal. Formally, a group of reward models assign a scalar score to each response, producing set of rewards r = {r1,... TG corresponding to the candidates. To guide the optimization of the unified model, design hybrid reward signal that integrates multiple task- specific objectives. The hybrid reward aims to jointly enhance the quality of both generated text and images, well ensure output format consistency.
To better model and optimize the interleaved multimodal outputs, we begin with a probabilistic decomposition of the joint distribution over input prompt X, generated text Yizt, and generated image Yimg. We decompose the joint probability as:
p(Yeats Yingl P(Yaat (X)p((inn img X,, Yrat).
(6)
From this perspective, we design our hybrid reward to align with the two conditional components: The textual reward Ta evaluates o(YzzelX), assessing the relevance and coherence of the generated text given the prompts. The visual rewards targets p(Yimgl X, Yizt). measuring the image's 's quality and its alignment with the text and prompt context. Furthermore, we incorporate a format reward ry to regularize the structure of the generated content. Specifically, we leverage special tokens <think and <vis> to explicitly separate different modalities the content sequence, guiding the model to adhere to consistent and interpretable format. This formatting strategy is illustrated in Figure 1. The textual relevance reward r' is assessed by [355 and the visual reward assessed by ImageReward [36]. To this end, our hybrid reward function is defined as follows:
5

---

MMIE Project-base based learning
Method
Params
Situational analysis
Multi-step reasoning
AVG
MiniGPT-5 [39]
7B
47.63
55.12
42.17
50,92
EMU-2 [26]
39.65
46.12
50.75
45.33
GILLL [9]
7B
46.72
57.57
39.33
51.58
Anole [5]
7B
48.95
59.05
51.72
55.22
Ours (QLoRA)
7B
53.12
60.34
53.28
57.27
Ours
7B
56.87
62.28
54.31
59.50
Table Comparison with existing methods MMIE. The proposed method significantly improves the generation quality of unified models, producing interleaved text-image outputs that are both coherent and aligned with instructions.
= = +x))))))(()+)xx))))))) +ry(X))
(7)
Process-level reward. While outcome supervision provides a single reward at the end of each output, this sparse feedback may be insufficient and inefficient for complex multimodal gen- eration tasks. To address this, we additionally incorporate process-level supervision, which offers intermediate rewards at the end of each modality step. Formally, given set of sam- pled outputs { a process reward model assigns a sequence of rewards to each output: R [(r indew(() ..... (Ki) .... {rG index((1 ..... l indew(Ka))) where inder(j) denotes the end token index of the j-th step, and K2 the total number of modality steps in the i-th output. Each step reward is then normalized using the overall mean and standard deviation across all samples. To compute token-level advantages, we accumulate the normalized rewards from all subsequent steps:
A Σ inder(5)2t Pi ginder(i) The final policy is then optimized by maximizing the GRPO objective using these token-level advantages.
3.4 Training details
We implement VILA-U [32] as our foundation unified model. Thanks to its unified pretraining paradigm, which jointly learns multimodal comprehension and text-to-image generation, the model inherently possesses the potential for multimodal output. Our approach builds on this capability and unlocks interleaved generation with only minimal additional data. During the warm-up stage, we col- lect 0.3M text-image paired samples with interleaved outputs from ActivityNet [1], GenHow To [23] and OpenStory++ [37]. To preserve the model's original multimodal understanding and generation abilities, we further incorporate 1M multimodal understanding samples from EMOVA [2] and IM text-to-image generation samples from JourneyDB [25]. For the GRPO stage, we curate dataset of 0.1M samples (from the same sources as in the warm-up stage) focusing on visual storytelling and multimodal interleaved reasoning facilitate effective policy optimization.
Experiment
4.1 Experimental setup
Data preparation. During the warm-up stage, we collect 0.3M interleaved text-image samples from ActivityNet [1], GenHowTo [23], and OpenStory++ [37]. The data covers diverse multimodal scenar- ios, including temporally grounded actions, step-by-step state transitions, and narrative storytelling, providing rich interleaved patterns for training- (i) ActivityNet [1] is large-scale benchmark for human activity understanding, covering 203 activity classes with about 137 untrimmed videos per class and 1.41 activity instances per video (849 hours total). We use its temporal dense captions [10] to describe fine- fine-grained action segments. For each localized clip, a duration-adaptiic number of

---

IntericavedBench Image coherence
Method
Params
Text quality
Perceptual quality
TIC
Helpfulness
AVG
MiniGPT-5 [39]
1.22
2.45
1.62
2.03
1.77
1.82
EMU-2 [26]
37B
1.26
2.28
1.89
1.34
1.64
1.68
GILL [9]
0.75
3.21
2.25
1.53
1.48
1.84
Ours (QLoRA)
7B
2.33
3.39
3.01
2,75
2.85
2,87
Ours
7B
2.86
3,58
3.25
3.02
2.94
3.13
Table Comparison with existing methods on InterleavedBench, where our method demonstrates superior performance across multiple evaluation metrics.
keyframes is extracted capture visual details, and these keyframes with their captions form in- terleaved training samples. (ii) GenHowTo is a large-scale dataset automatically constructed with 200K image triplets and corresponding textual descriptions. Each triplet contains temporally ordered frames depicting an object's initial statc, the modifying action, and the resulting new state, effectively modeling cause-and-effect visual transitions. These structured triplets naturally support interleaved text-image training for step-by-step visual reasoning. We leverage both the provided action and state prompts, along with their aligned images, construct coherent interleaved text-image samples for visual reasoning tasks. (iii) OpenStory++ [37] is comprehensive dataset emphasizing narrative continuity with instance-level visual segmentation. It extracts keyframes from videos, filters them by aesthetic quality, and generates refined captions via an LLM- annotator to ensure coherence. The resulting keyframe-caption pairs form interleaved text-image sequences for visual storytelling. More detailed data construction process and illustration are provided in the appendix.
Implementation details. As described previously, we adopt VILA-U [32] as our foundation unified model. The language model, vision encoder, and visual decoder are all implemented following VILA-U's original architecture and configuration. All images are resized a fixed resolution of 256 x 256 before being fed into the model. For image generation, we employ classifier-free guidance with guidance scale of improve output fidelity. In GRPO stage, the number of generation G is set to 4 and we train the model for 3k steps. All full-scale experiments are conducted using 32 NVIDIA A100 GPUs. To further demonstrate the compatibility and efficiency of our method, we also implement lightweight version using QLoRA, achieving competitive performance with significantly lower memory and compute requirements.
Evaluation. We evaluate the model's capability for interleaved multimodal generation on two dedicated benchmarks: MMIE[33] and InterleavedBench[I
MMIE consists of 20K carefully curated multimodal queries, covering 3 main categories, 12 domains, and 102 subfields, including mathematics, coding, physics, literature, health, and the arts. It introduces robust automated evaluation protocol using a scoring model fine-tuned on annotated data with systematic criteria, aiming reduce bias and enhance evaluation reliability. InterleavedBench comprises 815 instances across 10 real-world use cases. It defines five key evaluation dimensions: text quality, perceptual quality, image coherence, text-image coherence, and overall helpfulness, offering comprehensive and fine-grained assessment of interleaved generation performance. By encompassing detailed evaluation criteria, InterleavedBench provides a robust framework for measuring a model's ability to generate coherent, visually aligned, and contextually meaningful interleaved outputs.
4.2
Main results
We compare our approach against existing unified models on these challenging benchmarks to evaluate its effectiveness in interleaved multimodal generation. As shown in Table 1, our approach significantly improves the ability of unified models to produce coherent and cettucctionaalignne text-image outputs. These improvements demonstrate that our method not only strengthens the generation quality under limited supervision, but also pushes the performance upper bound of current unified models on interleaved multimodal generation benchmarks (59.5% on MMIE). In particular, our method demonstrates clear advantage on the situational analysis task, which evaluates a model's ability to perform visual storytelling based on given textual and visual prompts. Our model achieves
7

---

wa GRPO
MMIE
InterieavedBench
x
0.51
x
53.31
1.97
59.50
3.13
Table 3: Effects of warm-up stage and GRPO. The warm-up stage enables to generate interleaved multimodal outputs. Building on the warm-up initialization, the proposed GRPO-based policy optimization substantially improves the quality of interleaved generations.
Hybrid rewards
Process reward
MMIE
InterleavedBench
rf
Ft
X
53.56
2.05
X
X
54.62
2.30
/
/
X
57.83
2.79
/
/
59.50
3.13
Table 4: Hybrid reward components ablation. Beginning with the format reward ry, we gradually integrate the textual, visual and process-level rewards. The results demonstrate that the inclusion of complementary reward signals progressively enhances interleaved generation quality.
score of 56.87%, outperforming Anole by over 10%. This advanced performance highlights the effectiveness of our interleaved generation strategy in handling complex, context-rich multimodal reasoning scenarios. We also report our results on InterleavedBench, as shown Table Our method outperforms existing unified models, achieving superior performance with gain of 1.29 over GILL, demonstrating the robustness and generalization ability of our approach.
4.3
Ablation study
Effects of warm-up stage and GRPO. We first investigate the impact GRPO as well as warm-up stage in Table 3. Empirical results demonstrate that the warm-up up stage effectively unlocks the unified model's 's capability generate interleaved multimodal outputs (53.31% % on MMIE and 1.97 on InterleavedBenchh. Prior to this stage, existing unified models were unable to produce such interleaved generations, and thus failed to yield valid results on benchmarks like MMIE. Notably, InterleavedBench includes a subset of tasks that do not require multimodal outputs, which allows baseline models achieve a marginal score (0.51). Building upon the warm-up initialization. our proposed GRPO-based policy optimization significantly enhances the quality of interleaved outputs, leading to consistent improvements across both benchmarks (+6.19% and +1.16). This two-stage strategy proves essential for enabling instruction-following coherent, and structurally aligned text-image sequences.
Rewards components ablation. We then take further step to investigating the components of rewards in the GRPO training paradigm. Starting from the format reward ry. we progressively incorporate the textual reward rt, visual reward re, and finally the process-level reward. As shown in Table 4, the format reward alone does not lead to significant performance gains compared the warm-up stage. However, it plays an essential role in providing explicit supervision for modality switching, which crucial for stable interleaved generation. Adding the textual reward yields a steady improvement of 1.06% MMIE and 0.25 on InterleavedBench, while the visual reward further boosts performance by 3.01 % and 0.49, respectively. Finally, integrating the process-level reward results in the best performance, raising the scores to 59.50% MMIE and 3.13 on InterleavedBench.

---

Understanding
Generation Inuge res.
Method
ILO-Uni.
Params
MME-P13
MMvel [38]
SEEDBench-ing L11
POPE [12]
#Train images
Soore
VILA-U [3]]
x
7B
1401.8
33.5
39.0
85.8
15M
384
0.42
SEED-X
TokenFlo-XL XL [17]
X
13B
1545.9
40.7
68.7
86.8
60M
384
0.55
Janus-Pro Pro
7B
1516.7
45.1
70.1
78.9
72M
384
0.80
Chameleon [27]
/
7B
153.1
8.3
1.48
512
0.39
ARMOR [24]
/
8B
1619.4
53.1
74.8
87.7
5M
256
0.37
1425.2
32.8
0.46
Table 5: Comparison with existing methods on visual understanding and generation benchmarks. ILO-Uni. denotes "Unified models with interleaved text-image output". Our approach maintains comparable performance to baseline models.
KL
G
Tv
MMIE
InterleavedBench
2
ImageReward [36]
53.04
1.72
V
2
ImageReward
55.14
2.27
4
ImageReward
59.50
3.13
CLIP-score 56.94
2.58
Table 6: Aspects of GRPO srper-parameter implementation. We first analyze the effect of the KL-penalty term by duplicating its application during training. Results show that maintaining a KL constraint stabilizes optimization and mitigates policy drift from the initialization, underscoring its critical role. We then ablate the number generations evaluate its impact on learning dynamics, and finally, compare different formulations of the visual reward function.
These results demonstrate the effectiveness of each component and highlight the importance of fine-grained, multi-dimensional reward signals in training high-quality interleaved generation models.
Unified understanding and generation. Additionally, we evaluate our model's performance on standard multimodal understanding and generation tasks (GenEval [7]) to ensure that our method does not lead to performance degradation catastrophic forgetting existing unified capabilities. As shown in Table 5, our approach achieves comparable results the baseline model across these tasks, indicating that both the warm-up stage and our GRPO-base based training strategy are effective in preserving the model's original strengths. Furthermore, we compare method with existing unified models that are capable of generating multimodal interleaved outputs. Results show that our approach achieves competitive performance, highlighting its advantage in enabling high-quality and instruction-aligned interleaved generation without sacrificing general multimodal capabilities.
Aspects of GRPO hyper- ppram-pareeeter implementation. We conduct series of ablation studies to examine the impact of key hyper-paramett choices in our GRPO framework, as shown Table We begin by analyzing the role of the KL.-penalty term, where we duplicate its application during training. Results show that enforcing KL divergence stabilizes training and prevents the model from drifting too far from the initial policy, confirming its importance.
We further ablate the number of generations G for each prompt during training. Increasing the number of sampled generations from 2 to results in notable performance improvement (+4.36% and +0.86), indicating that generating a more diverse set of candidates facilitates more accurate group-wise reward estimation and provides richer learning signals. However, due to computational limitations, we are unable to explore larger generation numbers. particularly resource-intensing in the multimodal setting, as it requires generating both textual and visual outputs during each rollout. This underscores the importance developing more efficient and scalable post-training strategies for multimodal models in future research.
worth noting that GRPO

---

Input: In this tusk, you ane given the first part of on event with both text and on image, and you meed to complete the subsequent parts of the event by genanating text and imges that are nitural continust on of the given first port. The input interlooved!
Ieput: will give you the baginning o
Input: In this task, you are given high-leve god .. You need to predict the subsequent steps to help the user to finsh the task The previous steps one (BEGIN: Rest for a few moments to ebserve moment of pure! peace of the Lorikest Glen.
I I will give you the beginring of 1 story. Continue story with
a story, Continue the story with detailed description / with text and images.
detalled description / with text and limages,
Question: The teom huddled to prepare for the
Question: Amy never thought that she would married.
content is -BESIN
pink does chi.
Output: rhhikk As the referse's is whistle blow, the players broke from
Output: <thinjo But when she met Jack. something charged within her, and she begon to imagine future together.
Output HihinloHead ever to Jock
the huddle with focused determunation. </thinko
Honno's Wild Reserve learn about
Output: ethinlo The person gracefully shifts their weight onto back leg.
conservction efforts
</thunio
<image> </vis> They sprinted pesition.
majestic like
bold
<inage> </vis>
drawing both inward smooth, circular mation, </thinle rimage> <thinkoffeer completing final movement, she stards quietly. </thinio,
eyes sconning field voices
<image>
tolking, loughing.
<thinoo They spent sharing
hours
calling out plays </thinko
sthunlo Join the keeper talks
dreans.v/thnlk
<image> <thinko The
animal encounters to get up up-cless
<vis> <image> </vis>
kicked off, the team luuchee into action with coordinated intersity, </thinlo
experience and despen your understonding of willffs care t/tinkel
Itthrjoo day, Jock surprised her
<image>
cvis> cimage> </vi>>
with a proposal, and without hesitation,
Amy soid yes </thinlo
<vi>> <image> </vis>
<vis> <image> </vis>
Figure 2: Visualizations of multimodal interleaved generation. Qualitative examples illustrate the model's capacity to produce coherent interleaved outputs, smoothly transitioning between text and image modalities within a unified generation process.
Lastly, we compare different choices of visual reward functions ru- Specifically, we find that using ImageReward provides more accurate and consistent supervision signals for image quality than CLIP-score, reinforcing the value of tailored reward models multimodal training.
Visualizations. Figure showcases qualitative examples of interleaved generation, illustrating the model's 's ability to seamlessly alternate between text and image modalities. Notably, the visual outputs align well with the surrounding textual content, reflecting the effectiveness of our policy optimization strategy in producing fine- ane-grained semantically consistent interleaved content. More visualizations and failure case studies are included in the appendix.
5 Conclusion and discussions
Conclusion. We propose an effective training strategy to enable unified vision-language models to perform high-quality interleaved multimodal generation. Through warm-up stage with limited data, we unlock the model's latent multimodal generation capabilities. We further introduce a unified policy optimization framework based on GRPO with hybrid and process-level rewards, which formulates the multimodal generation process as single trajectory from reinforcement learning perspective. Experiments on MMIE and InterleavedBench demonstrate superior performance over existing unified models while preserving general multimodal capabilities, paving the way for more versatile and controllable multimodal generation systems.
Discussion on limitations. Despite the significant performance gains on multimodal interleaved benchmarks, we observe that our method does not bring notable improvements on generalized multimodal understanding and text-to-image generation tasks. We attribute this to the underlying base model, which largely determines performance limits, with GRPO mainly serving to better align text and image content rather than enhancing general capabilities. Future work may benefit from stronger unified architectures and broader reward designs to further enhance model capability.
Acknowledgments
This work was supported in part by National Natural Science Foundation of China (Grant No. 62376060).
10

---

References
[1] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet:
A large-scale video benchmark for human activity understanding. In CVPR, 2015. 6, 21
[2] Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, et al. Emova: Empowering language models to see, hear and speak with vivid emotions. arXiv preprint, 2024. 6
[3] Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. Towards reasoning era: A survey of long chain-of chain-of-thought for reasoning large language models. arXiv preprint. 2025. 3
[4] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint, 2025. 2, 9
[5] Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation. arXiv preprint, 2024. 3,6 6
[6] Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint, 2024. 9
[7] Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. In NIPS, 2023. 9
[8] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint, 2025. 3
[9] Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. In NIPS, 2023. 3, 6,7
[10] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- captioning events in videos. In ICCV, 2017, 6, 21
[11] Bohao Li, Yuying Ge. Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In CVPR, 2024. 9
[12] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision- languagl models. arXiv preprint, 2023. 9
[13] Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, 2024.
[14] Minqian Liu, Zhiyang Xu, Zihao Lin, Trevor Ashby, Joy Rimchala, Jiaxin Zhang, and Lifu Huang. Holistic evaluation for interleaved text-and-image generation. In EMNLP, 2024. 2, 7
[15] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, et al. Training language models to follow instructions with human feedback. In NIPS, 2022.
[16] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint, 2023. 3
[17] Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint, 2024. 9
[18] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NIPS, 2023.
11

---

[19]
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Kero-shot text-to-image generation. In ICML, 2021.
[20] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint, 2017. 3
[21] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint, 2024. 3,
[22] Haozhan Shen, Peng Liu, Jingcheng Li, Chumxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, al. VIm-rl: A stable and generalizable rl-style large vision-language model. arXiv preprint, 2025. 3
[23] Tomáš Souček, Dima Damen, Michael Wray, Ivan Laptev, and Josef Sivic. Genhowto: Learning to generate actions and state transformations from instructional videos. In CVPR, 2024. 6, 21
[24] Jianwen Sun, Yukang Feng, Chuanhao Li, Fanrui Zhang, Zizhen Li, Jiaxin Ai, Sizhuo Zhou, Yu Dai, Shenglin Zhang. and Kaipeng Zhang. Armor vO. 1: Empowering autoregressive multi- modal understanding model with interleaved multimodal generation via asymmetric synergy. arXiv preprint, 2025. 3, 9
[25] Keqjang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: a benchmark for generative image understanding. In NIPS, 2023. 6
[26] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In CVPR, 2024. 2, 6,
[27] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint, 2024. 3,
[28] Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, and Hang Xu. Illume: Illuminating your Ilms to see, draw, and self-enhance. arXiv preprint, 2024. 1,2
[29] Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Multimodal chain-of-thought reasoning: A comprehensive survey. arXiv preprint, 2025.
[30] Yichen Wei. Yi Peng, Xiaokun Wang, Weijie Qiu, Wei Shen, Tianyidan Xie, Jiangbo Pei, Jianhao Zhang, Yunzhuo Hao, Xuchen Song, al. Skywork rlv2: Multimodal hybrid reinforcement learning for reasoning. arXiv preprint, 2025. 3
[31] Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable multi-modal generators. arXiv preprint, 2024.
[32] Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: unified foundation model integrating visual understanding and generation. arXiv preprint, 2024. 1, 2, 6, 7,9
[33] Peng Xia, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang, Wenhao Zheng, Zhaorun Chen, Chenhang Cui, Mingyu Ding, Linjie Li. et al. Mmie: Massive multimodal interleaved comprehension benchmark for large vision-language models. In ICLR, 2025. 2,7
[34] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer unify multimodal understanding and generation. arXiv preprint, 2024. 1,2
[35] Tianyi Xiong, Xiyao Wang, Dong Guo, Qinghao Ye, Haoqi Fan, Quanquan Gu, Heng Huang, and Chunyuan Li. Llava-critic: Learning to evaluate multimodal models. In CVPR, 2025. 5
12

---

[36] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. In NeurlPS, 2023. 5, 9
[37] Zilyu Ye, Jinxiu Liu, Ruotian Peng. Jinjin Cao, Zhiyang Chen, Yiyang Zhang, Ziwei Xuan, Mingyuan Zhou, Xiaoqian Shen, Mohamed Elhoseiny, et al. Openstory++: large-scale dataset and benchmark for instance-aware open domain visual storytelling. arXiv preprint, 2024. 6, 7,
21
[38] Weihao Yu, Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In ICML, 2024. 9
[39] Kaizhi Zheng. Xuehai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-and Usioo-and-language -ad-lang generation via generative vokens. arXiv preprint, 2023. 6,7
[40] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint, 2024.
[41] Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng, Tianyi Zhou, and Cho-Jui Hsich. R R1-zero's s" aha moment" in visual reasoning 2b non-sft model. arXiv preprint, 2025.
13

---

NeurIPS Paper Checklist
BEGIN INSTRUCTIONS
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We make the main claims clearly in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of this paper in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We discuss the experimental setup in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Code would be released after submission and review.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//niss cttpssccccpbccccccceeeeccecccccc httee/guuueeeeeeeuubiiieeee
/Cooeeeodiissseeeeeee wwsssonollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Details of training and testing are specified in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Experiments in this paper conducted in multiple iterations, yielding stable results. We reported the average of these results for consistency and reliability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computational cost and resources are specified in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code Of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: h/tpprip///////.....// wwweeerrrs..c.uuupeeeeeeeee httpppullttthsassiiiiileee
Answer: [Yes]
Justification: The research conducted this paper conforms in every respect with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The Broader Impacts are discussed in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: We follow the safeguards of base unified model VILA-U.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All models and datasets used in this paper have been properly credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.hcoe...c. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: LLM is original component of the core methods in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: : caarrripsscc//ooffeenneeee////// /LLM) for what should or should not be described.
20

---

A Data preparation
During the warm-up stage, we collect 0.3M text-image paired samples with interleaved outputs from ActivityNet [1]. GenHowTo [23] and OpenStory++ [37]. The data comprises diverse multimodal scenarios, including temporally grounded action descriptions, step-by-sted action-state pairs, and narrative visual storytelling. These samples are organized into interleaved text-image sequences to expose the model to rich multimodal generation patterns during training.
ActivityNet. ActivityNet [1] is large-scale video benchmark designed for human activity un- derstanding. It covers a broad spectrum of complex daily activities, featuring 203 activity classes with average of 137 untrimmed videos per class and 1.41 activity instances per video, totaling approximately 849 hours of video content. We utilize the temporal dense captions that describe fine-grained action segments in each video [10]. For each temporally localized clip, we extract set of representative keyframes to capture fine-grained visual details. The number of keyframes is adaptively determined based the duration of the action clip. These keyframes, along with their corresponding dense captions, are subsequently used to construct interleaved training samples.
ActivityNet
GenHowTo
OpenStory++
Query. In this task, you one given the first part of event with both text and on image, and need to complete the subsequent parts of the event by generating text and mages that ore natural inantioii of the given first interleaved content is <EEINN woman kitchen tolking about how cake.
Query: In this task you are given a mult inodal question consisting both text and image.
Query: In this task, you ane given the first part of on event with both text and inage. and need to complete the subsequent parts the event gererating that are natural continustion of the given first The interleaved content (BESIN Write story about golden retriever based reference image.
Your goal to generate interieaved text- image onswer that demonstrated trotes muttimodel reasoning The answer should include both textual and relevant visual content to support reasoning process. <BEGIN> How fish?
by-step
and images
Answer: ethinlo
cutting
<thinio cutting fish outting </thinko
Answer: <thinio There was gentle golden retriever nomed Surny. who brought happiness to his fomily-<thinie:
pumpkin laying them up </thinlo
cimage>
<image> </vis>
ingrediants a mixes
The person
more
pieces a cutting
-tranloSpriigg brought new baby to the family and their dag Surny adopted with ecse and curiosity, </thinko
<vis> <image> </vi>>
together in end, </thinio <vis> <image> </vi>>
<vis> <inage> </vis>
evis> cimage> </vis>
Figure 3: Illustration of data preparation process during the warm-up stage.
GenHowTo. GenHow To is a large-scale dataset automatically constructed with 200K image triplets and corresponding textual descriptions. Each triplet consists of temporally ordered video frames depicting (i) the initial state of an object, (ii) the action that modifies the state, and (iii) the resulting new state of the object. These structured triplets, along with their descriptions, naturally align with interleaved text-image training for modeling step-by-step visual reasoning. We utilize the provided action prompts and state prompts, along with their corresponding images, to construct a series of interleaved text-image samples for visual reasoning tasks, as shown in Figure 3.
OpenStory++. OpenStory++ is comprehensive dataset designed to emphasize narrative continuity around key instances, featuring instance-level visual segmentation. It processes video content by extracting keyframes, evaluating them for aesthetic quality, and generating descriptive captions using BLIP2. These captions are then refined by a Large Language Model (LLM) to ensure coherence and maintain narrative flow. The resulting keyframe-caption pairs are used to build interleaved text-image sequences that support visual storytelling, as shown in Figure 3.
B Failure cases analysis
While our method demonstrates strong performance interleaved multimodal generation tasks, we observe several failure cases that highlight current limitations, as demonstrated in Figure 4. First, in some complex reasoning scenarios, the model may generate hallucinated visual content that does not
21

---

Input: Question: How to Stop Social Media from Damaging Your Offline Behavior?
Output: <thinkeStep 2: Set boundaries for social media usoge. Designate Specific times of day to check social media doesn't interfere with real-life interactions. e/toonssttthiio <vis> <image> </vis>
Method 1-Balancing Online Offine
Social Networks
Step 1: Make offline networking priority
Figure Failure cases analysis about hallucinated visual content.
faithfully correspond to the textual context, indicating limitations in cross-modal alignment. Second, the visual elements produced by the model occasionally fail match the style appearance of the input image, resulting in stylistic inconsistency across modalities. These issues suggest future work could benefit from improved reward modeling that better captures factual consistency, reasoning traceability, and structural fidelity in interleaved generation.
C Broader impacts
The proposed work has the potential to greatly enhance the capability unified models for multimodal interleaved generation. Therefore it could be beneficial for applications such multimodal reasoning and visual storytelling. It could also be used to provide better experience of multimodal interleaved interaction between human and AI system. As for the potential negative impacts, there is risk that the technology could be misused to generate and spread disinformation.
22


Reviews:
[{"review_id": "Yjfl9BTNdt", "reviewer": "Reviewer_7qgK", "summary": "- This paper proposes a reinforcement learning-based post-training strategy to enable existing unified vision-language models to generate multimodal interleaved outputs.\nSpecifically, the paper proposes a unified policy optimization framework that extends Group Relative Policy Optimization (GRPO) to the multimodal setting.\n- The paper has conducted experiments on two challenging benchmarks, MMIE and InterleavedBench, demonstrating the effectiveness of their proposed approach\n- The paper has provided comprehensive ablation studies showing the effectiveness of submodules.", "questions": "I wonder if the authors have tried the version without the warmup stage, how does the model perform?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper proposes an effective RL training strategy to enable unified vision-language models to perform high-quality interleaved multimodal generation. \n- The paper introduces a hybrid group-wise reward signal composed of textual relevance, visual-text alignment, and structural fidelity—addressing multiple aspects of interleaved generation quality.\n- The proposed method is evaluated on two challenging benchmarks, MMIE and InterleavedBench, demonstrating the effectiveness of their proposed approach.\n\n**Weaknesses**\n- The computation of the process-level reward (L200) is unclear to me. Is it computed using modality-specific segment as target output, and follows the same reward computation strategy as in Eq. 7? \n- In Table 5, the proposed method underperforms compared to VILA-U on MMVet and POPE", "was_revised": true, "final_justification": "I think the idea of leveraging a relatively small amount of data with RL to enable interleaved generation is interesting. Although the paper could benefit from a clearer and more detailed explanation of the proposed method. I appreciate the authors' effort during the rebuttal and given that the promised clarifications will be incorporated into the revised version, I would like to maintain my score.", "rebuttal": "**Weakness1**: The computation of the process-level reward (L200) is unclear to me. Is it computed using modality-specific segment as target output, and follows the same reward computation strategy as in Eq. 7?\n\n**A**: Thank you for the question. In our approach, we treat modality-specific segments as the target outputs.\nHowever, the reward computation for each such step $Y^k$ slightly differs from Eq. 7.\nWe compute a modality- and format-specific local reward $r^k$ as follows:\n$$\nr^k = I_{Y^k \\in Y_{\\text{txt}}} \\cdot r_t(X, Y^k) + I_{Y^k \\in Y_{\\text{img}}} \\cdot r_v(X, Y^k) + r_f(X, Y^k).\n$$\nSpecifically, the local reward is only computed based on the specific modality segment, and is then uniformly assigned to all tokens within that segment.\n\nFor credit assignment across steps within the trajectory, we follow a group-wise outcome comparison strategy similar to outcome-level supervision in Eq. 5:\n$$\n\\hat{A}^{k_i}_{i,t} = \\frac{r(X, Y^{k_i}_i) - \\text{mean}\\left( \\{ r(X, Y^{1}_1), \\dots, r(X, Y^{k_1}_1), \\dots, r(X, Y^{k_G}_G) \\} \\right)}{\\text{std}\\left( \\{ r(X, Y^{1}_1), \\dots, r(X, Y^{k_1}_1), \\dots, r(X, Y^{k_G}_G) \\} \\right)}.\n$$\nWe normalize the per-step scores within the group, and compute the relative advantage of each step.\nThis strategy enables a more fine-grained and diverse credit assignment across modality-specific steps.\nWe will clarify this implementation detail more explicitly in the refined version of the paper.\n\n**Weakness2**: In Table 5, the proposed method underperforms compared to VILA-U on MMVet and POPE.\n\n**A**: We thank the reviewer for pointing this out.\nThe slightly lower performance of our method on MMVet and POPE compared to VILA-U is within a reasonable margin of experimental variance (approximately 2% and 1%, respectively), and we believe it does not indicate a fundamental limitation of our approach.\n\nTo further address this concern, we conducted additional experiments using Janus-Pro, a stronger unified model.\nThe results demonstrate consistent improvements across all benchmarks, including MMVet and POPE, validating that our GRPO framework remains effective and generalizable on these benchmarks as well.\n\n| Method           | MME-P  | MMvet | SEEDBench-img | POPE | #Train images | Image res. | GenEval |\n|------------------|--------|--------|----------------|------|----------------|-------------|--------|\n| Janus-Pro        | 1516.7 | 45.1   | 70.1           | 78.9 | 72M            | 384         | 0.80   |\n| Ours (Janus-Pro) | 1552.9 | 45.4   | 71.3           | 80.6 | 16M            | 384         | 0.83   |\n\n**Q1**: I wonder if the authors have tried the version without the warmup stage, how does the model perform?\n\n**A**: We appreciate the reviewer’s insightful suggestion.\nAs requested, we conducted an additional experiment where we directly train the unified model using GRPO without the warm-up stage.\nThe results (**Row 2**) show that generation quality after GRPO is significantly lower compared to the full two-stage pipeline.\nWe attribute this to the model’s poor initial ability to generate interleaved outputs, which limits its capacity to produce high-reward samples during the GRPO stage.\nThese findings underscore the importance of the warm-up stage in providing a reasonable initialization for modality transitions and content generation.\n\n|Warm-up|GRPO|MMIE|InterleavedBench|\n|:----|:----|:----|:----|\n|✗|✗|-|0.51|\n|✗|✓|17.84|1.25|\n|✗|✓ (w/ initialization)|25.96|1.58|\n|✓|✓|59.5|3.13|\n\nTo further investigate this phenomenon, we explored an alternative strategy under the pure GRPO setting by explicitly encouraging the model to sample modality-switching actions through modifications to the sampling policy and reward formulation.\nSpecifically, during the initial training phase, we manually sampled an interleaved content trajectory by adjusting the sampling probability of modality-switching tokens.\nAdditionally, we introduced a multimodal content reward to enhance its relative advantage within the group and incentivize the generation of interleaved content.\n\nHowever, performance (**Row 3**) still fell short compared to the warm-up+GRPO setup, indicating that starting from a cold initialization hinders the model’s ability to discover meaningful interleaved structures solely through reinforcement signals. These results support our two-stage training design and highlight the critical role of the warm-up stage in stabilizing training and guiding the model toward effective interleaved generation."}, {"review_id": "eZoGUMKxku", "reviewer": "Reviewer_Ksvg", "summary": "This paper introduces a policy optimization framework for unified multimodal models that significantly enhances their capability for interleaved generation. To achieve this objective, the method employs a warm-up stage using hybrid datasets to develop initial interleaved generation capabilities, followed by policy optimization using an extended GRPO framework. The optimization process incorporates a comprehensive multi-type rewards that evaluates textual relevance, visual-text alignment, and structural fidelity to guide the training process effectively. Experimental results on established interleaved generation benchmarks demonstrate the effectiveness of the proposed method, showing substantial improvements in both generation quality and coherence compared to existing approaches.", "questions": "- Regarding Table 3, I would like to see the performance of the model when applying GRPO directly without the warm-up stage. While the model can generate interleaved outputs even without warm-up (albeit of poor quality), it would be valuable to understand how much the warm-up stage contributes to the final performance by comparing the direct GRPO approach against the full two-stage pipeline.\n\n- The ablation study on group size in GRPO presented in Table 6 appears limited in scope. Given that recent work such as DeepSeek-R1 [1] has explored larger group sizes (8, 12, 16), it would be beneficial to extend this analysis to investigate whether larger group sizes could further improve performance. This expanded ablation would provide better insights into the scalability of the approach and help determine the optimal group size configuration for multimodal interleaved generation tasks.\n\n- The training process for interleaved generation appears to primarily benefit the interleaved output capability without demonstrating clear improvements to the model's core image understanding abilities. Could the authors provide insights into whether this specialized training offers any additional benefits beyond interleaved generation? For instance, does the multi-modal policy optimization process enhance visual reasoning, cross-modal alignment, or other fundamental multimodal understanding capabilities?\n\n[1] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" arXiv preprint arXiv:2501.12948 (2025).", "limitations": "Yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The construction pipeline for the interleaved data used during warm-up is well-designed and robust.\n- The final performance of interleaved generation demonstrates strong quality and coherence.\n\nWeaknesses:\n- The proposed framework lacks novelty. The GRPO algorithm remains unchanged, still producing token-by-token outputs without any modifications. As a result, this framework is only applicable to autoregressive (AR)-based models such as Janus-Pro. It cannot be applied to other mainstream unified models such as Transfusion[1], MetaQuery [2], or BAGEL [3].\n- The use of multiple rewards is rather straightforward and does not constitute a significant contribution.\n- The interleaved generation primarily focuses on coherence and creativity, but does not demonstrate capabilities such as interleaved reasoning or other complex interleaved scenarios, which limits its applicability.\n\n[1] Zhou, Chunting, et al. \"Transfusion: Predict the next token and diffuse images with one multi-modal model.\" arXiv preprint arXiv:2408.11039 (2024).\n\n[2] Pan, Xichen, et al. \"Transfer between modalities with metaqueries.\" arXiv preprint arXiv:2504.06256 (2025).\n\n[3] Deng, Chaorui, et al. \"Emerging properties in unified multimodal pretraining.\" arXiv preprint arXiv:2505.14683 (2025).", "was_revised": true, "final_justification": "While the paper presents a practical extension of GRPO to multimodal settings, the adaptation relies heavily on existing formulations and does not introduce fundamentally novel techniques. The proposed process-level reward is essentially a form of reward shaping, combining known signals heuristically rather than offering a new design. Key claims, such as model-agnostic applicability, are not empirically supported, which leans to the borderline rejection.", "rebuttal": "**Weakness1**: The proposed framework lacks novelty. The GRPO algorithm remains unchanged, still producing token-by-token outputs without any modifications. As a result, this framework is only applicable to autoregressive (AR)-based models such as Janus-Pro. It cannot be applied to other mainstream unified models such as Transfusion, MetaQuery, or BAGEL.\n\n**A**: \n1. We respectfully disagree with the reviewer’s comment that our GRPO framework is ``unchanged`` and lacks novelty.\nIn this work, we make several substantial extensions beyond the original GRPO algorithm.\n\n* We extend GRPO to the multimodal setting by introducing a unified policy optimization framework, which allows actions from the visual modality to be seamlessly integrated into the overall generation trajectory and jointly optimized alongside textual actions.\nThis enables global optimization over the interleaved sequence, ensuring semantic consistency and facilitating effective gradient propagation across modalities.\n\n* We further design a novel hybrid process-level reward that allows for finer-grained credit assignment and better alignment of intermediate generation steps.\nThese innovations are non-trivial and crucial for effectively adapting GRPO to large-scale multimodal generation tasks.\n\n2. Furthermore, our approach is not inherently tied to autoregressive models.\nWe simply adopt VILA-U as our base model, which operates in an autoregressive fashion.\nIn fact, GRPO strategies have also been successfully applied in diffusion- and flow-matching-based frameworks, such as Flow-GRPO [1].\n\n* Mathematically, regardless of whether generation is performed via AR decoding or diffusion sampling, both paradigms can be abstracted as sampling from a parameterized policy $\\pi_\\theta(y \\mid x)$, where $y$ denotes the generated output (either autoregressive sequence or diffusion sampling trajectory) and $x$ is the input context.\nBoth approaches optimize $\\pi_\\theta(y \\mid x)$ through gradient-based updates guided by relative preferences over grouped samples.\n\n* Therefore, our framework—particularly the design of process-level hybrid rewards—is inherently model-agnostic, and can be seamlessly integrated into a wide range of generation paradigms beyond the AR setting.\nWe will further investigate its applicability to diffusion-based or flow-matching-based generators in future work.\n\nWe hope our response helps clarify the novelty and contributions of our work, as well as eliminate potential misunderstandings—particularly regarding the misconception that our framework is limited to autoregressive models.\n\n**Weakness2**: The use of multiple rewards is rather straightforward and does not constitute a significant contribution.\n\n**A**:\n1. First, we would like to clarify that developing new reward models is not the primary motivation of this work. Our core goal lies in building a unified framework under the GRPO paradigm that enables advanced interleaved multimodal generation.\nTherefore, we deliberately adopt multiple straightforward yet effective reward models as ``functional tools`` to guide policy learning.\nThe innovation of our approach lies in leveraging multimodal hybrid reward signals that provide comprehensive supervision for interleaved multimodal generation.\n\n2. Moreover, for credit assignment, we do propose a novel process-level reward specifically tailored for interleaved multimodal generation.\n\n* Previous works, such as Deepseek-R1, attempted to use process-level rewards, but struggled with the challenge of defining “correct steps” in complex tasks. This ambiguity led to reward diffusion and noisy gradient updates, hindering stable learning.\n\n* In contrast, interleaved multimodal generation offers natural, semantically grounded step boundaries via modality transitions (e.g., text to image), which act as anchors for defining local rewards. This enables clearer credit assignment along the trajectory and improves sample efficiency during optimization. Experimental results in Table 4 (+1.67 on MMIE and +0.34 on InterleavedBench) underscore the significance and contribution of the proposed process-level reward.\n\nTherefore, we believe the choice of reward models does not undermine the novelty or contributions of our work; on the other hand, our innovation lies in how rewards are assigned—particularly through the proposed process-level design tailored for interleaved generation.\n\n**Weakness3 & Q3**: The interleaved generation primarily focuses on coherence and creativity, but does not demonstrate capabilities such as interleaved reasoning or other complex interleaved scenarios, which limits its applicability.\n\n**A**: We thank the reviewer for raising this point.\n\n1. First, we would like to clarify that the benchmarks selected in our main experiments already include scenarios involving interleaved reasoning and other complex interleaved tasks.\nFor example, we have reported the ``Multi-step reasoning\" metric on MMIE benchmark in Table 1, and our results (53.28\\%) outperforms current advanced models.\n\n2. To further strengthen this evaluation, we conduct additional experiments following the MVoT protocol [2], where a suite of maze-specific tasks is used to evaluate the model’s visual reasoning capabilities across multimodal contexts.\nAs shown in the table below, our method achieves substantial performance gains on these benchmarks, further validating its ability to handle complex interleaved reasoning beyond simple coherence and creativity.\n|Method|MAZE|MINIBEHAVIOR|FROZEN-LAKE|\n|:----|:----|:----|:----|\n|MVoT|0.9295|0.9514|0.856|\n|Vila-U|0.7046|0.6527|0.5839|\n|Ours|0.9593|0.9706|0.8742|\n\n**Q1**: Regarding Table 3, I would like to see the performance of the model when applying GRPO directly without the warm-up stage. While the model can generate interleaved outputs even without warm-up (albeit of poor quality), it would be valuable to understand how much the warm-up stage contributes to the final performance by comparing the direct GRPO approach against the full two-stage pipeline.\n\n**A**: We appreciate the reviewer’s insightful suggestion.\nAs requested, we conducted an additional experiment where we directly train the unified model using GRPO without the warm-up stage.\nThe results (**Row 2**) show that generation quality after GRPO is significantly lower compared to the full two-stage pipeline.\nWe attribute this to the model’s poor initial ability to generate interleaved outputs, which limits its capacity to produce high-reward samples during the GRPO stage.\nThese findings underscore the importance of the warm-up stage in providing a reasonable initialization for modality transitions and content generation.\n\n|Warm-up|GRPO|MMIE|InterleavedBench|\n|:----|:----|:----|:----|\n|✗|✗|-|0.51|\n|✗|✓|17.84|1.25|\n|✗|✓ (w/ initialization)|25.96|1.58|\n|✓|✓|59.5|3.13|\n\nTo further investigate this phenomenon, we explored an alternative strategy under the pure GRPO setting by explicitly encouraging the model to sample modality-switching actions through modifications to the sampling policy and reward formulation.\nSpecifically, during the initial training phase, we manually sampled an interleaved content trajectory by adjusting the sampling probability of modality-switching tokens.\nAdditionally, we introduced a multimodal content reward to enhance its relative advantage within the group and incentivize the generation of interleaved content.\n\nHowever, performance (**Row 3**) still fell short compared to the warm-up+GRPO setup, indicating that starting from a cold initialization hinders the model’s ability to discover meaningful interleaved structures solely through reinforcement signals. These results support our two-stage training design and highlight the critical role of the warm-up stage in stabilizing training and guiding the model toward effective interleaved generation.\n\n**Q2**: The ablation study on group size in GRPO presented in Table 6 appears limited in scope. Given that recent work such as DeepSeek-R1 has explored larger group sizes (8, 12, 16), it would be beneficial to extend this analysis to investigate whether larger group sizes could further improve performance. This expanded ablation would provide better insights into the scalability of the approach and help determine the optimal group size configuration for multimodal interleaved generation tasks.\n\n**A**: We appreciate the reviewer’s suggestion and agree that exploring larger group sizes can offer valuable insights into the scalability of GRPO.\nHowever, due to the heavy memory usage of unified multimodal models, especially during group-wise sampling and interleaved generation, we were only able to extend our ablation study up to group size = 6 and 8 within our current hardware constraints.\n\nAs shown in the table below, increasing the group size to 6 yields noticeable improvements, indicating the potential benefit of larger groups.\nHowever, further increasing the group size to 8 brings only marginal gains.\nWe acknowledge the value of further exploration at even larger group sizes (e.g., 12, 16), and we plan to include such extended experiments in future work when more computational resources become available.\n\n|G|MMIE|InterleavedBench|\n|:----|:----|:----|\n|2|55.14|2.27|\n|4|59.5|3.13|\n|6|61.43|3.38|\n|8|61.85|3.42|\n\n[1] Liu, Jie, et al. \"Flow-grpo: Training flow matching models via online rl.\" arXiv preprint, 2505.05470.\n\n[2] Li, Chengzu et al. “Imagine while Reasoning in Space: Multimodal Visualization-of-Thought.” arXiv preprint, 2501.07542."}, {"review_id": "nIkmNwfAMY", "reviewer": "Reviewer_n2df", "summary": "Based on unified VLMs, this paper expands their capability to generate interleaved text and images instead of either understanding or generation. The authors utilize GRPO to post-train VLMs with textual, visual and format rewards together with progress rewards after a warmup stage to enable interleaved outputs. They conduct experiments on MMIE and InterleavedBench, and achieve performance gains compared to original VLM base model.", "questions": "1. Can more comprehensive evaluations be conducted as stated in weaknesses?\n2. Can a baseline of pure SFT post-train be added to provide more insights on the advantage of RL-based post-training at the domain of interleaved text and image generation?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n1. The paper enables the capability of interleaved image and text generation with GRPO post-training for unified VLMs, and achieves improved performance compared to the baseline.\n2. The method is intuitive and easy to implement.\n3. The structure/writing of the paper is easy to follow.\n\n**Weaknesses**:\n1. The experiment is only conducted on Vila-u, which is less powerful than other baselines as shown in Table 5 for multimodal understanding or generation, the generalization of the proposed method on the more powerful unified VLM is not demonstrated or proved.\n2. The compared baselines in main results (Table 1 and Table 2) are somehow outdated, which is lack of more powerful baselines like EMU3 [1] to ensure a more comprehensive evaluation.\n3. There is limited insight the community can gain after reading this paper. It is expected that after GRPO with combined rewards, the model can achieve better performance on interleaved generation for text and images, adding a baseline of SFT only post-train and comparing it with RL post-train may be more insightful.\n\n[1] Wang, X., Zhang, X., Luo, Z., Sun, Q., Cui, Y., Wang, J., Zhang, F., Wang, Y., Li, Z., Yu, Q., Zhao, Y., Ao, Y., Min, X., Li, T., Wu, B., Zhao, B., Zhang, B., Wang, L., Liu, G., He, Z., Yang, X., Liu, J., Lin, Y., Huang, T., & Wang, Z. (2024). Emu3: Next-Token Prediction is All You Need. ArXiv, abs/2409.18869.", "was_revised": true, "final_justification": "I have read the author's rebuttal and further discussion from authors, which addresses my concern.", "rebuttal": "**Weakness1**: The experiment is only conducted on Vila-u, which is less powerful than other baselines as shown in Table 5 for multimodal understanding or generation, the generalization of the proposed method on the more powerful unified VLM is not demonstrated or proved.\n\n**A**: To address this concern, we have conducted extensive experiments on Janus-Pro [1], a more powerful and unified VLM compared to Vila-u, with stronger performance on both multimodal understanding and generation tasks.\nAs shown in the tables below, our proposed method consistently improves performance across various evaluation metrics on Janus-Pro, demonstrating its effectiveness and generalizability beyond a single model architecture.\n\n|Method|MMIE|InterleavedBench|\n|:----|:----|:----|\n|EMU-2|45.33|1.68|\n|EMU-3|50.17|2.19|\n|Ours (Vila-u)|59.5|3.13|\n|Ours (Janus-Pro)|61.76|3.35|\n\n| Method           | MME-P  | MMvet | SEEDBench-img | POPE | #Train images | Image res. | GenEval |\n|------------------|--------|--------|----------------|------|----------------|-------------|--------|\n| Janus-Pro        | 1516.7 | 45.1   | 70.1           | 78.9 | 72M            | 384         | 0.80   |\n| Ours (Janus-Pro) | 1552.9 | 45.4   | 71.3           | 80.6 | 16M            | 384         | 0.83   |\n\n**Weakness2 & Q1**: The compared baselines in main results (Table 1 and Table 2) are somehow outdated, which is lack of more powerful baselines like EMU3 to ensure a more comprehensive evaluation.\n\n**A**: We thank the reviewer for the suggestion.\nHowever, not all unified models currently support interleaved generation, which is the core focus of our method.\nFor completeness, we faithfully report the results of baseline models in Table 1 and Table 2 based on their original papers to ensure fairness and reproducibility.\n\nFurthermore, we have additionally included the performance of EMU3 on standard benchmarks where applicable, to offer a more comprehensive view of model capabilities.\nOur method still demonstrates competitive or superior performance, further confirming its effectiveness.\n\n|Method|MMIE-SA|MMIE-PL|MMIE-MR|MMIE-AVG|InterleavedBench|\n|:----|:----|:----|:----|:----|:----|\n|EMU-2|39.65|46.12|50.75|45.33|1.68|\n|EMU-3|46.14|50.85|52.86|50.17|2.19|\n|Ours|56.87|62.28|54.31|59.5|3.13|\n\n**Weakness3 & Q2**: There is limited insight the community can gain after reading this paper. It is expected that after GRPO with combined rewards, the model can achieve better performance on interleaved generation for text and images, adding a baseline of SFT only post-train and comparing it with RL post-train may be more insightful.\n\n**A**: We thank the reviewer for the valuable suggestion.\nIn the main paper, we have included a comparison in Table 3 where the model is trained only with interleaved data (only w/ warm-up stage) using supervised fine-tuning.\nThe results (53.31 on MMIE and 1.97 on InterleavedBench) show that SFT alone yields suboptimal performance on interleaved generation tasks, indicating that exposure to interleaved data is helpful but not sufficient for optimal performance.\n\nFurthermore, to ensure a fair comparison, we extend the experiment by increasing the SFT data to match the scale used in RL post-training.\nAs shown in the table below (**Row 2**), even under this setting, RL post-training with our combined reward design consistently outperforms SFT, which only gains minimal benefits.\nThe results suggest that reinforcement learning brings non-trivial gains beyond what SFT can achieve, especially in aligning fine-grained generation behavior across modalities.\nWe will highlight these insights more clearly in the revision to better convey the added value of our approach.\n\n|Method|MMIE|InterleavedBench|\n|:----|:----|:----|\n|SFT (Warm-up)|53.31|1.97|\n|SFT (Full)|54.82|2.04|\n|GRPO|59.5|3.13|\n\n[1] Xiaokang Chen, et al. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint, 2025."}, {"review_id": "G9bLsgTThy", "reviewer": "Reviewer_3D4S", "summary": "This paper presents a reinforcement learning-based post-training strategy that enables unified vision-language models to generate coherent, interleaved sequences of text and images, introducing a warm-up phase with limited interleaved data and a novel multimodal extension of Group Relative Policy Optimization with hybrid and process-level rewards, and demonstrates significant improvements on dedicated interleaved generation benchmarks without sacrificing general multimodal understanding or generation capabilities.", "questions": "1. Process-level rewards and group-wise sampling increase training costs. Can you provide more quantitative analysis on the computational and memory overhead introduced by the GRPO stage (e.g., compared to standard RLHF or supervised fine-tuning)? Please also discuss potential strategies for reducing this overhead, such as more efficient candidate sampling or reward estimation.\n\n2. Could you provide more details or experiments on the robustness and generalization ability of these reward models, especially when applied to out-of-distribution data or tasks beyond the current benchmarks? If not already included, consider testing or discussing how the method would adapt if reward models are imperfect or biased.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper addresses an important and underexplored challenge: enabling unified vision-language models to produce truly interleaved multimodal outputs (text and images), which is crucial for applications like visual storytelling and step-by-step reasoning.\n\n2. The method does not require large-scale, high-quality interleaved datasets. Instead, it leverages limited interleaved data and a well-designed warm-up phase, making it practical and accessible for real-world applications.\n\nWeakness:\n\n1. While the method is empirically effective, the authors offer limited theoretical analysis or formal justification for why the proposed multimodal GRPO and reward design are particularly suited to interleaved generation.\n\n2. The process-level reward modeling and the need to generate multiple candidate sequences per prompt in GRPO introduce substantial computational costs, which may limit scalability.\n\n3. The approach relies heavily on the effectiveness and robustness of multiple reward models (textual, visual, and structural) during training, but the authors do not thoroughly analyze how these reward models generalize to out-of-distribution scenarios or novel tasks, potentially limiting the method’s adaptability.\n\n4. Tables 3, 4, 5, and 6 are poorly presented with inconsistent sizes and font styles, making the paper appear unpolished and disorganized.", "was_revised": true, "final_justification": "The authors address most of my concerns during the rebuttal and the additional experiments and explanations improves the manuscript. Therefore, I'm lean to acceptance, but still open to rejection.", "rebuttal": "**Weakness1**: While the method is empirically effective, the authors offer limited theoretical analysis or formal justification for why the proposed multimodal GRPO and reward design are particularly suited to interleaved generation.\n\n**A**: We thank the reviewer for pointing out the lack of theoretical justification.\nTo address this concern, we provide conceptual justifications from two theoretical perspectives to support its design:\n\n1. Unified Policy Optimization: In the main paper, we have introduced a unified policy optimization formulation in Eq. 3 and 4:\n$$\nL_{GRPO}(\\theta) = \\frac{1}{G} \\sum_{i=1}^{G} \\frac{1}{|Y_i|} \\sum_{t=1}^{|Y_i|} \\min [ D_{uni}(t) \\cdot A_{i,t}, \\, \\text{clip}(D_{uni}(t), 1 - \\epsilon, 1 + \\epsilon) \\cdot A_{i,t} ],\n$$\n$$\nD_{uni}(t) = \\frac{\\pi_{\\theta}(y_{i,t}^{txt} \\mid X_i, y_{i, \\leq t})}{\\pi_{\\theta_{old}}(y_{i,t}^{txt} \\mid X_i, y_{i, \\leq t})} \\cdot 1_{t \\leq k}(t) + \\frac{\\pi_{\\theta}(y_{i,t}^{img} \\mid X_i, y_{i, \\leq t})}{\\pi_{\\theta_{old}}(y_{i,t}^{img} \\mid X_i, y_{i, \\leq t})} \\cdot 1_{t > k}(t).\n$$\nThis formulation allows actions from the visual modality to be seamlessly integrated into the overall generation trajectory and jointly optimized alongside textual actions.\nBy treating all modality transitions within a unified policy space, our approach enables global optimization over the entire multimodal sequence.\nThis not only ensures semantic consistency across modalities but also facilitates effective gradient propagation through the interleaved structure, which is critical for learning coherent cross-modal dependencies.\n\n2. Process-Level Reward Design: Previous works focusing on single-modality generation, such as Deepseek-R1, have explored introducing process-level rewards to better align generation with human preferences.\nHowever, they encountered a critical limitation: it is inherently difficult to accurately define and annotate what constitutes a \"correct step\" in complex reasoning or generation processes.\nThis ambiguity in step boundaries leads to reward diffusion, where local credit assignment becomes unclear, resulting in noisy gradient updates and unstable learning dynamics.\nIn contrast, interleaved multimodal generation naturally provides discrete and semantically grounded boundaries through modality switching (e.g., from text to image and back).\nThese modality transitions serve as anchors for defining process-level steps, enabling more accurate and interpretable credit assignment during optimization.\nTo facilitate understanding, we derive the optimization objective as:\n$$\nL(\\theta) = E_{Y \\sim \\pi_\\theta} \\left[ R(Y) \\cdot \\nabla_\\theta \\log \\pi_\\theta(Y) \\right],\n$$\n$$\n\\text{Var} \\left[ R(Y) \\cdot \\nabla_\\theta \\log \\pi_\\theta(Y) \\right] \\propto \\text{Var} \\left[ R(Y) \\right].\n$$\nWhen step-level definitions are incorrect or inconsistent, the resulting reward distribution becomes noisy and ill-posed, leading to unstable learning signals and suboptimal policy updates.\nThis leads to high variance in gradient estimation.\nIn our interleaved multimodal setting, modality transitions provide explicit segmentation of the trajectory, allowing us to define rewards locally per modality segment.\nThis reduces ambiguity in credit assignment and improves sample efficiency during optimization.\n\nWe will include a more detailed theoretical discussion in the revised version to support this rationale.\n\n**Weakness2 & Q1**: The process-level reward modeling and the need to generate multiple candidate sequences per prompt in GRPO introduce substantial computational costs, which may limit scalability. Can you provide more quantitative analysis on the computational and memory overhead introduced by the GRPO stage (e.g., compared to standard RLHF or SFT)? Please also discuss potential strategies for reducing this overhead, such as more efficient candidate sampling or reward estimation.\n\n**A**: We appreciate the reviewer’s concern regarding the potential computational and memory overhead introduced by our GRPO stage.\n\n1. We first would like to clarify that our process-level reward does not introduce additional computational burden—it simply redistributes the original outcome-level reward across different modality-specific steps. Therefore, it incurs negligible additional cost compared to standard RLHF reward computation. Furthermore, our group-wise sampling reduces training overhead compared to standard RLHF methods such as PPO, as it eliminates the need for training a separate critic model, which is often the most computationally intensive component in PPO-based pipelines.\n\n2. To further support our claim, we provide a quantitative comparison of the computational and memory overhead introduced by GRPO relative to standard RLHF and supervised fine-tuning (SFT) in the table below.\nFor fair comparison, the number of samples per iteration is fixed to 4.\nThe results show that GRPO introduces only marginal overhead compared to SFT, and is significantly more efficient than PPO-style RLHF in both compute and memory.\nWe will include this quantitative analysis in the revision.\n|Method|GPU Hours (per 1K steps)|Peak Memory (GB)|\n|:----|:----|:----|\n|SFT|9.7|30|\n|PPO-based RLHF|15.3|58|\n|Our GRPO|11.1|42|\n3. To further reduce cost, we are actively exploring more efficient group-wise candidate sampling strategies (e.g., top-k filtering or nucleus sampling) and lighter reward estimation techniques that approximate process-level rewards without relying on full model evaluations at each step.\n\n**Weakness3 & Q2**: Could you provide more details or experiments on the robustness and generalization ability of these reward models, especially when applied to out-of-distribution data or tasks beyond the current benchmarks? If not already included, consider testing or discussing how the method would adapt if reward models are imperfect or biased.\n\n**A**:\n1. To examine the robustness of reward models, we have included experiments in the main paper (Table 6) that specifically examine imperfect or biased reward models.\nIn this setting, we replace the original reward model with CLIP, which serves as a weaker and less aligned alternative.\nThe results show that while there is a slight performance drop (-2.56 on MMIE and -0.55 on InterleavedBench), our GRPO framework remains stable and continues to outperform baseline methods, highlighting its resilience to noise and bias in the reward signal.\n\n2. To further address this concern, we conduct additional experiments following the MVoT protocol [1], where a suite of out-of-distribution, maze-specific tasks is used to evaluate the model’s visual reasoning capabilities across multimodal contexts.\nAs shown in the table below, without any changes to the reward models, our method achieves substantial performance gains on these challenging benchmarks.\nThis demonstrates the robustness and generalization ability of our reward design beyond the current benchmark tasks.\n|Method|MAZE|MINIBEHAVIOR|FROZEN-LAKE|\n|:----|:----|:----|:----|\n|MVoT|0.9295|0.9514|0.856|\n|Vila-U|0.7046|0.6527|0.5839|\n|Ours|0.9593|0.9706|0.8742|\n\n**Weakness4**: Tables 3, 4, 5, and 6 are poorly presented with inconsistent sizes and font styles, making the paper appear unpolished and disorganized.\n\n**A**: We thank the reviewer for highlighting the formatting inconsistencies in Tables 3, 4, 5, and 6.\nWe acknowledge the issue regarding inconsistent sizes and font styles, and we will carefully address these presentation details in the refined version to ensure clarity and visual consistency throughout the paper.\nWe appreciate the reviewer’s feedback, which will help improve the overall quality and professionalism of our refined version.\n\n[1] Li, Chengzu et al. “Imagine while Reasoning in Space: Multimodal Visualization-of-Thought.” arXiv preprint, 2501.07542."}]

Author rebuttal:

Meta-review: This paper addresses the challenging task of enabling vision-language models to generate interleaved sequences of text and images, which is crucial for applications such as visual storytelling and stepwise reasoning. The authors propose a reinforcement learning approach using Group Relative Policy Optimization (GRPO) complemented by a warm-up stage with curated hybrid datasets. Their novel hybrid reward function guides training to improve the quality and coherence of multimodal generation, which is empirically validated on relevant benchmarks like MMIE and InterleavedBench.

Strengths highlighted by the reviewers include the paper’s focus on an underexplored and significant problem, the practical and effective RL training strategy, and strong empirical performance gains over baselines. The construction of interleaved training data is well-designed and robust, and the authors have addressed many reviewer concerns during rebuttal, such as expanding experimental scope to stronger models and clarifying reward computations. The approach is noted for not requiring large-scale interleaved datasets, making it more accessible for practical use. However, novelty is somewhat limited due to the use of an existing RL algorithm framework, and theoretical justification is minimal. Reviewers also have concerns about generalizing reward models to out-of-distribution scenarios. 

While one reviewer remained unconvinced about the framework’s novelty, others acknowledged the authors’ rebuttal efforts and improved their scores. Overall, the paper stands out for its practical contribution to multimodal interleaved generation despite incremental methodological novelty, warranting acceptance as a solid poster paper at the conference.
Decision: Accept (poster)