UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning

## Abstract

Abstract Unified multimodal large language models such as Show-o and Janus have achieved strong performance across both generation and understanding tasks. However, these models typically rely on large-scale datasets and require substantial computation during the pretraining stage. In addition, several post-training methods have been proposed, but they often depend on external data or are limited to task-specific customization. In this work, we introduce UniRL, a self-improving post-training approach . Our approach enables the model to generate images from prompts and use them as training data in each iteration, without relying on any external image data. Moreover, it enables the two tasks to enhance each other: the generated images are used for understanding, and the understanding results are used to supervise generation. We explore supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) to optimize the models. UniRL offers three key advantages: (1) it requires no external image data, as all training samples are generated by the model itself during training; (2) it not only improves individual task performance, but also reduces the imbalance between generation and understanding; and (3) it requires only several additional training steps during the post-training stage. We evaluate UniRL on top of Show-o and Janus, achieving a GenEval score of 0.77 for Show-o and 0.65 for Janus. Code and models will be released in https://github.com/showlab/UniRL .

## 1 Introduction

Unified multimodal large language models are designed to handle both generation and understanding tasks within a shared parameter space. Recently, models such as Show-o, Janus, and others showo; janus; janusflow; transfusion; emu3 have made rapid progress by adopting various architectural and training strategies. Most existing work focuses on the pretraining stage, leveraging large-scale data to train models built upon large language models (LLMs) phi1.5; phi3; llama or multimodal LLMs (MLLMs) qwen_vl; llava; llava-next. However, these methods often face challenges such as high computational cost and the need for vast amounts of data. Meanwhile, several studies hermesflow; showo_cot; mvot have explored the post-training phase, but they face limitations: some require additional external data hermesflow; dora_cycle; mvot, others target only a single task showo_cot; mvot, or focus on task-specific customization dora_cycle. Therefore, developing effective post-training methods to improve the performance of unified multimodal models remains a significant challenge.

In this work, we propose a self-improving post-training method for unified multimodal models without external image data. We begin by constructing prompts and question–answer (QA) pairs inspired by the GenEval geneval benchmark, which evaluates generation quality by categorizing prompts into six types—such as counting, color, and position—designed to reflect fundamental visual features of natural images. We adopt this categorization to design the prompts used in our training process. In each training iteration, a constructed prompt is fed into the model to generate a group of images. These images, along with the corresponding questions, are then input back into the model to predict answers. The model is optimized using both the predicted and ground-truth answers to improve generation and understanding simultaneously. For optimization, we explore two strategies: supervised fine-tuning (SFT) and reinforcement learning.

In the post-training stage, supervised fine-tuning (SFT) and reinforcement learning have traditionally been widely used. Recently, Group Relative Policy Optimization (GRPO) grpo, a reinforcement learning method, has shown strong performance in large language models (LLMs) deepseek_r1, enabling effective post-training optimization through chain-of-thought (CoT) reasoning. In this work, we explore two optimization strategies—SFT and GRPO—for unified multimodal models and analyze the advantages of each strategy. Previous applications of GRPO typically rely on chain-of-thought outputs to estimate reward distributions. However, most current unified multimodal models lack the ability to produce structured reasoning steps. Therefore, we explore a GRPO-based method that does not rely on reasoning outputs, aiming to improve the performance of unified multimodal models.

Figure: Figure 1: The imbalance between Text-to-Image Generation (T2I) and Multi-modal Understanding (MMU). For the same image, unified multimodal models may struggle to perform both generation and understanding consistently.
Refer to caption: x1.png

For the performance of the unified multimodal models, we define it along three dimensions: (1) image generation capability, (2) image understanding capability, and (3) the balance between the two. While prior work has primarily focused on improving performance for a single task, unified multimodal models frequently exhibit poor alignment between the two capabilities. As shown in Fig. [1](https://arxiv.org/html/2505.23380v1#S1.F1), these models often fail to simultaneously support both tasks on the same image. The model may succeed in generating a plausible image but fail to answer the corresponding question, or fail in generation while answering correctly. Therefore, beyond improving individual tasks, our self-improving method explicitly targets the imbalance between image generation and understanding. To quantitatively assess this imbalance, we design a new evaluation metric. While visual understanding spans a wide range of tasks, including reasoning and mathematical problem solving, we focus on basic visual features of natural images—such as object count, color, and position—in both training and evaluation to ensure alignment between generation and understanding tasks.

Our method offers three main advantages. (1) It does not rely on external image data, using only images generated in real time during training. (2) It improves the performance of individual tasks while also reducing the imbalance between generation and understanding. Specifically, the same set of generated images is used for both tasks, enabling shared supervision that helps improve the imbalance between them. (3) It requires only a small number of training steps during the post-training stage. To evaluate our method, we adopt Show-o and Janus as base models, assessing improvements in both single-task performance and the balance between tasks. Our approach achieves post-training GenEval scores of 0.77 on Show-o and 0.65 on Janus.

Our main contributions are summarized as follows:

- •
To the best of our knowledge, we propose the first self-improving post-training optimization method for unified multimodal models via supervised and reinforcement learning.
- •
We investigate the effectiveness of both supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO) for unified multimodal models, and provide a detailed comparison of their respective performance.
- •
We introduce a new metric to quantify the imbalance between generation and understanding tasks within a single model. Our evaluations across multiple unified models reveal that such imbalance is common.
- •
Our method substantially improves image generation and understanding quality and effectively reduces task imbalance, enabling better consistency between text-to-image generation and multimodal understanding.

## 2 Related work

### 2.1 Unified multimodal understanding and generation

Several recent studies seed-x; wu2023next; team2024chameleon; showo; lwm; sun2023emu; transfusion; CoDI; dreamllm; emu3; janusflow; lumina_mgpt; liquid; llamafusion; anil2023gemini; tokenflow; synergen; Orthus; omniflow; januspro; metamorph have explored unified transformers capable of both generation and comprehension. Chameleon team2024chameleon,Emu3 emu3, Janus janus, JanusPro januspro adopt autoregressive methods for both tasks. SEED-X seed-x incorporates a diffusion model alongside a large language model to support multimodal generation and understanding. While both Show-o showo and JanusFlow janusflow use autoregression for understanding, the former adopts discrete diffusion for generation, whereas the latter employs flow matching. These models primarily focus on the pretraining stage using different methods for different tasks. These approaches require substantial training time and large-scale training data. Some studies Unimod; showo_cot have explored more efficient training or inference strategies. In this work, we take Show-o and Janus as our base model and explore post-training optimization techniques to further improve unified multimodal performance.

### 2.2 GRPO in LLM and MLLMs

Supervised fine-tuning (SFT) and reinforcement learning have been widely adopted in the post-training of large language models (LLMs) gpt_4o; phi1.5; phi3; llama and multimodal LLMs (MLLMs) qwen_vl; llava. Group Relative Policy Optimization (GRPO) is introduced in grpo to optimize LLMs by generating a group of chain-of-thought outputs and applying a task-specific reward function to guide learning. More recently, GRPO has also been explored in the context of MLLMs grpo_vl. In addition to GRPO, other reinforcement learning approaches such as Direct Preference Optimization (DPO) dpo; hermesflow and Proximal Policy Optimization (PPO) ppo have also been applied to both LLMs and MLLMs. Hermesflow hermesflow and Emu3 emu3 use DPO to optimize unified multimodal models. However, PPO requires a separate value network to estimate the baseline, and DPO relies on predefined positive and negative examples, which introduces additional complexity. In our work, we explore SFT and GRPO method to optimize unified multimodal models and improve their generation and understanding performance during the post-training stage.

## 3 Method

In this section, we first introduce the background of unified multimodal models and the GRPO optimization method (Sec.[3.1](https://arxiv.org/html/2505.23380v1#S3.SS1)). We then describe the construction of training prompts (Sec.[3.2](https://arxiv.org/html/2505.23380v1#S3.SS2)), the proposed self-improving framework (Sec.[3.3](https://arxiv.org/html/2505.23380v1#S3.SS3)), and two optimization strategies: SFT and GRPO (Sec.[3.4](https://arxiv.org/html/2505.23380v1#S3.SS4)). We also present a non-end-to-end training approach designed for models that use different image representations in the two tasks (Sec.[3.5](https://arxiv.org/html/2505.23380v1#S3.SS5)). Finally, we introduce a new metric to evaluate the imbalance between the two tasks (Sec.[3.6](https://arxiv.org/html/2505.23380v1#S3.SS6)). The pipeline of training is shown in Fig. [2](https://arxiv.org/html/2505.23380v1#S3.F2).

### 3.1 Preliminary

Unified Multimodal Models. Unified multimodal models can be divided into two main categories: (1) both generation and understanding tasks use autoregressive methods; (2) generation makes use of diffusion or flow matching while comprehension relies on autoregressive methods. We focus on two example models: Show-o showo and Janus janus.

In Show-o, the text tokens are denoted as $T=\{t_{1},\dots,t_{N}\}$ and the image tokens as $I=\{i_{1},\dots,i_{M}\}$. The training objective includes two components: one maximizes the likelihood of each text token given all previous text and image tokens; the other reconstructs masked image tokens based on the remaining image tokens and the full text sequence. The two loss terms are:

$$ $\mathcal{L}_{\mathrm{NTP}}=\sum_{n=1}^{N}\log p_{\theta}\bigl{(}t_{n}\mid t_{1 :n-1},\,i_{1:M}\bigr{)},\quad\mathcal{L}_{\mathrm{MTP}}=\sum_{m=1}^{M}\log p_{ \theta}\bigl{(}i_{m}^{*}\mid i_{1:m-1},\,i_{m+1:M},\,t_{1:N}\bigr{)}.$ (1) $$

Janus uses only the NTP objective, predicting every next token (image or text) in a single autoregressive pass.

Group Relative Policy Optimization(GRPO)
We use a reinforcement learning method called Group Relative Policy Optimization (GRPO) grpo, which optimizes the model by comparing a group of generated outputs and assigning relative rewards. Suppose inputs $x$ come from a distribution $D$. For each $x\sim D$, we sample $M$ candidate outputs $\{y_{j}\}_{j=1}^{M}$ under an old policy $\pi_{\varphi_{\mathrm{old}}}$. We then update our new policy $\pi_{\varphi}$ by maximising the following objective:

$$ $L(\varphi)=\mathbb{E}_{\begin{subarray}{c}x\sim D,\\ \{y_{j}\}\sim\pi_{\varphi_{\mathrm{old}}}\end{subarray}}\Biggl{[}\frac{1}{M} \sum_{j=1}^{M}\min\bigl{(}s_{j}\,A_{j},\,\mathrm{clip}(s_{j},\,1-\delta,\,1+ \delta)\,Z_{j}\bigr{)}\Biggr{]}-\lambda\,D_{\mathrm{KL}}\!\bigl{(}\pi_{\varphi }\,\|\;\pi_{\psi}\bigr{)},$ (2) $$

where

$$ $s_{j}\;=\;\frac{\pi_{\varphi}(y_{j}\mid x)}{\pi_{\varphi_{\mathrm{old}}}(y_{j} \mid x)},\quad A_{j}\;=\;\frac{R_{j}\;-\;\mathrm{mean}(R_{1},\dots,R_{M})}{ \mathrm{std}(R_{1},\dots,R_{M})}.$ $$

$R_{j}$ is the reward for output $y_{j}$, $\delta$ sets the clipping range, $\lambda$ weighs the divergence penalty, and $\pi_{\psi}$ is a fixed reference policy.
In the inner sum, $s_{j}$ measures how the new policy probability differs from the old policy for the same output. Multiplying by the normalised advantage $A_{j}$ gives a gradient direction. Clipping $s_{j}$ to $[1-\delta,\,1+\delta]$ ensures that no single sample drives the update too far. Averaging over $M$ outputs makes the estimate stable across different group sizes. Finally, subtracting $\lambda\,D_{\mathrm{KL}}(\pi_{\varphi}\,\|\,\pi_{\psi})$ discourages large shifts away from a trusted policy, which helps to keep learning smooth.

**Table 1: Examples of prompts constructed for six categories of image-based question–answer tasks: single object, two object, counting, colors, position, and attribute.**
| Category | Prompt | Question | Answer |
| --- | --- | --- | --- |
| Single object | a photo of a bench | What is the main object of the image? | Bench |
| Two object | a photo of a table and a soccer | What are two main objects of the image? | A table and a soccer |
| Counting | a photo of three vases | How many items in this image? | Three vases |
| Colors | a photo of a green fork | What is the color of the object? | Green |
| Position | a photo of a train above of an elephant | What are two objects and what is position relationship between two main items? | The train is above of an elephant |
| Attribute | a photo of a blue chair and a red umbrella | What are two objects and the colors of two objects in the image? | The chair is blue and the umbrella is red |

### 3.2 Prompt and QA pair construction

In our study, we are inspired by the GenEval geneval benchmark to construct prompt and question–answer pairs. The GenEval benchmark evaluates image generation models by dividing spatial relationships into six categories: single object, two objects, counting, colors, color attributes, and position. We create prompts and matching question–answer pairs based on these categories. First, beyond the eighty object classes in the COCO dataset, we use GPT-4o to generate eighty more everyday objects, resulting in 160 objects in total. Second, we follow the GenEval procedure to generate prompts and then formulate the corresponding questions and answers. For example, in the counting category, we use a prompt such as “a photo of three vases”, pair it with the question “How many items are in this image?”, and assign “three vases” as the answer. To ensure a strict separation between training and evaluation, we only use prompts from GenEval for testing. All prompts used for training are independently constructed and do not overlap with those in the GenEval benchmark. Additional examples appear in Tab. [1](https://arxiv.org/html/2505.23380v1#S3.T1).

Figure: Figure 2: Training pipeline of UniRL (a) SFT Optimization: A prompt is used to generate a single image, which, together with the corresponding question, is used to predict an answer and compute the SFT loss. (b) GRPO Optimization: The same prompt generates a set of images, each paired with the same question to produce multiple answers, which are used to compute the GRPO loss.
Refer to caption: x2.png

### 3.3 Self-improving optimization

We propose a self-improving training framework that requires no external image data. In this framework, the image generation and multimodal understanding tasks mutually reinforce each other: the understanding task provides feedback to evaluate and guide image quality, while the generation task supplies training data that enhances the model’s understanding capability.

As shown in Fig. [2](https://arxiv.org/html/2505.23380v1#S3.F2), given a constructed prompt, the model first generates a sequence of image tokens $\mathbf{u}=\{u_{1},\dots,u_{M}\}$. Regardless of whether the image generation module uses a diffusion-based or autoregressive pipeline, we feed the generated image tokens and the prompt back into the model following the original T2I training process to obtain the image logits. Let $\mathbf{z}=\{z_{1},\dots,z_{L}\}$ denote the output logits corresponding to the $L$ target positions to be predicted.

To enable gradient flow through the discrete image tokens, we apply the Straight-Through Gumbel-Softmax (ST-GS) estimator to obtain differentiable approximations:

$$ $\hat{u}_{l}=\mathrm{ST\text{-}GumbelSoftmax}(z_{l}),\quad\text{for }l=1,\dots,L,$ (3) $$

resulting in a completed image token sequence $\hat{\mathbf{u}}$ that includes differentiably sampled tokens. The formulation and implementation details of the Straight-Through Gumbel-Softmax estimator are provided in Sec. [A.2](https://arxiv.org/html/2505.23380v1#A1.SS2). This image sequence, together with the corresponding constructed question $q$, is then used to predict an answer. We use the predicted answers and ground-truth answer to optimize the model exploring two techniques, SFT and GRPO.

### 3.4 SFT and GRPO optimization

SFT optimization. We use the predicted answers and the ground-truth answers to calculate the SFT loss to optimize the models. The SFT loss is computed as:

$$ $\mathcal{L}_{\text{SFT}}=-\sum_{t=1}^{T}\log p_{\theta}(y_{t}\mid y_{<t},\hat{ \mathbf{u}},q),$ (4) $$

where $y_{t}$ is the $t$-th token of the ground-truth answer, and $y_{<t}$ are the preceding tokens. This formulation applies generally to both masked and autoregressive image generation models. The variable $\hat{\mathbf{u}}$ denotes the generated image token sequence, which includes differentiably sampled tokens and serves as input to the answer generation module. This formulation is compatible with both masked and autoregressive image generation models. The SFT loss enables joint optimization of image generation and understanding tasks by allowing gradients to flow through the image tokens.

GRPO optimization. We also adopt Group Relative Policy Optimization (GRPO) to optimize the unified multimodal models. During the GRPO phase, the model takes a prompt $p$ and generates a group of $K$ images $\{\hat{\mathbf{u}}_{1},\dots,\hat{\mathbf{u}}_{K}\}$. Each image $\hat{\mathbf{u}}_{k}$, together with its corresponding question $q$, is then fed back into the model to produce a predicted answer $\hat{a}_{k}$. A reward $r_{k}=r(\hat{a}_{k})$ is computed by comparing the predicted answer to the ground-truth answer $a^{*}$, based on the designed reward function. Similar to SFT, we also apply the Straight-Through Gumbel-Softmax estimator to enable differentiable sampling of image tokens, allowing gradients to propagate through both generation and understanding tasks. The overall GRPO loss is defined as:

$$ $\mathcal{L}(\theta)=\mathbb{E}_{\begin{subarray}{c}x\sim D,\\ \{\hat{a}_{k}\}\sim\pi_{\theta_{\mathrm{old}}}\end{subarray}}\left[-\,\sum_{k= 1}^{K}w_{k}\,\log p_{\theta}\bigl{(}\hat{a}_{k}\mid\hat{\mathbf{u}}_{k},\,q \bigr{)}\right]+\beta\,\mathrm{KL}\!\left(p_{\theta}(a\mid\hat{\mathbf{u}},q) \,\|\,p_{\theta_{\mathrm{ref}}}(a\mid\hat{\mathbf{u}},q)\right),$ (5) $$

where the importance weight $w_{k}$ is computed as:

$$ $w_{k}=\frac{\exp\left(\alpha\,(r_{k}-r_{\mathrm{mean}})\right)}{\sum_{j=1}^{K} \exp\left(\alpha\,(r_{j}-r_{\mathrm{mean}})\right)},$ $$

with $r_{\mathrm{mean}}$ denoting the average reward within the group and $\alpha$ controlling the sharpness of the weight distribution.

The first term in the loss is the GRPO objective, which applies a weighted log-likelihood over the $K$ sampled answers. $\pi_{\theta_{\mathrm{old}}}$ is the fixed sampling policy, typically a frozen copy of the model, while $\pi_{\theta}$ (or $p_{\theta}$) is the current policy being optimized. Rewards are centered by subtracting the group mean $r_{\mathrm{mean}}$, scaled by temperature $\alpha$, and converted into weights $w_{k}$ to emphasize better answers. The second term is a KL penalty, scaled by $\beta$, to keep the updated policy close to a reference model, improving stability and preventing mode collapse.

**Table 2: Examples of reward rules based on key words in generated answers**
| Category | Score | Key Word |
| --- | --- | --- |
| Single object | 1 | bench |
| Two object | 2 | table, soccer |
| Counting | 2 | three, vase |
| Colors | 1 | green |
| Position | 3 | train, above of, elephant |
| Attribute | 4 | blue, chair, |
|  |  | red, umbrella |

Importantly, because the image tokens $\hat{\mathbf{u}}_{k}$ are sampled in a differentiable manner, the reward signals from answer prediction can flow through both the understanding and image generation branches. This enables effective end-to-end optimization of both tasks.

Reward Function.
The reward function assigns scores based on alignment between the predicted answer $\hat{a}_{k}$ and the ground-truth answer $a^{*}$. While the specific reward criteria vary by question category, all rewards are based on two main components:
(1) correct identification of object names, and
(2) accurate prediction of associated attributes, such as number, color, or spatial position.
The detailed reward rules for each category are summarized in Tab. [2](https://arxiv.org/html/2505.23380v1#S3.T2).

### 3.5 End-to-end training vs. non-end-to-end training

Unified multimodal models such as Show-o showo, Emu3 emu3, Chameleon team2024chameleon, and Vila-u vila-u use shared image representations—typically via VQGAN vqgan or custom tokenizers—allowing end-to-end training. Gradients from MMU loss can flow through the T2I module via the Straight-Through Gumbel-Softmax estimator. In contrast, Janus janus and JanusPro januspro use separate representations: discrete tokens for T2I and ViT features vit for MMU. This mismatch prevents effective gradient flow; although gradients can technically propagate, they cannot meaningfully guide the discrete tokens, often leading to unstable training.

To handle such cases, we adopt a non-end-to-end training strategy. For T2I, rewards computed from MMU performance are used to directly optimize the image generation module. Conversely, for MMU, the model generates images, answers the corresponding questions, and uses the rewards to update the MMU branch. In the end-to-end setting, a unified GRPO loss is used to jointly optimize both tasks. In contrast, the non-end-to-end setting applies separate GRPO objectives to T2I and MMU. Although the two modules are trained independently, reward signals still enable cross-task interaction, mitigating the limitations of mismatched representations. Full details and loss formulas are provided in the Sec. [A.3](https://arxiv.org/html/2505.23380v1#A1.SS3).

### 3.6 The balance between image understanding and generation

**Table 3: Performance of the proposed metric for evaluating the imbalance between two tasks.**
| Method | $\text{Accuracy}_{\text{MMU}|\text{T2I}}$ | $\text{Accuracy}_{\text{T2I}|\text{MMU}}$ |
| --- | --- | --- |
| Show-o | 0.83 | 0.70 |
| Janus | 0.67 | 0.68 |
| JanusPro | 0.69 | 0.85 |
| Vila-u | 0.68 | 0.85 |

To evaluate the balance between text-to-image generation (T2I) and multimodal understanding (MMU), we propose a bidirectional evaluation metric with two chains: T2I→MMU and MMU→T2I. In T2I→MMU, the model generates an image from a prompt and answers a question only if the image is correct; we then compute the answer accuracy. In MMU→T2I, the model first answers a question based on a given image; if correct, we regenerate the image from the prompt and evaluate its quality. This setup assesses the consistency between generation and understanding. The conditional accuracies are defined as:

$$ $\text{Accuracy}_{\text{MMU}|\text{T2I}}=\frac{N_{\text{A}\cap\text{I}}}{N_{ \text{I}}},\quad\text{Accuracy}_{\text{T2I}|\text{MMU}}=\frac{N_{\text{A}\cap \text{I}}}{N_{\text{A}}}.$ (6) $$

$N_{\text{I}}$ denotes the number of correctly generated images, $N_{\text{A}}$ the number of correctly answered questions, and $N_{\text{A}\cap\text{I}}$ the number of samples where both are correct. These metrics quantitatively assess the consistency between generation and understanding. $\text{Accuracy}_{\text{MMU}|\text{T2I}}$ measures answer correctness given a correct image, while $\text{Accuracy}_{\text{T2I}|\text{MMU}}$ measures image correctness given a correct answer. This bidirectional formulation evaluates the alignment between the two tasks beyond isolated performance. As shown in Tab. [3](https://arxiv.org/html/2505.23380v1#S3.T3), we apply this metric to several unified models, including Show-o showo, Janus janus, Janus-Pro januspro, and Vila-u vila-u, and observe that task imbalance is consistently present across all models. More details can be seen in Sec. [A.1](https://arxiv.org/html/2505.23380v1#A1.SS1).

## 4 Experiments

### 4.1 Implementation details

We evaluate our method on two base models: Show-o showo and Janus janus. For Show-o, we adopt end-to-end training. The SFT stage uses a batch size of 2 for 5,000 iterations, and the GRPO stage is configured with a group size of 3, a KL coefficient $\beta=0.2$, and 3,000 training steps on 8 H100 GPUs. During inference, we set the guidance scale to 2, diffusion steps to 16, and resolution to 512. For Janus, due to its use of different representations for generation and understanding, we apply a non-end-to-end training strategy. The model is trained with a batch size of 8 for 1,000 iterations using the same KL coefficient. Inference is performed with a guidance scale of 5 and resolution of 384. More implementation details can be seen in Sec. [A.1](https://arxiv.org/html/2505.23380v1#A1.SS1).

### 4.2 Quantitative results

Baselines and our method.
We compare our method against several baselines. Original Model. We evaluate the performance of the pretrained model without any post-training. HermesFlow. hermesflow This method fine-tunes the Show-o model with DPO optimization on the JourneyDB dataset journeydb. We evaluate two variants of our post-training framework:
UniRL (SFT). Our method using supervised fine-tuning (SFT) without reinforcement learning. UniRL (GRPO). Our method using Group Relative Policy Optimization (GRPO). For the Janus model, we only compare our GRPO-based method with the original model, as applying SFT leads to unstable training and failure. The specific reasons are discussed in Sec. [4.4](https://arxiv.org/html/2505.23380v1#S4.SS4).

Text to image generation. For the image generation task, we evaluate model performance using three benchmarks: GenEval geneval, DSG-1K DSG, and CLIP-Score clip. To calculate the CLIP score, we use 2,000 generated images sampled from the GenEval and DSG-1K benchmarks. As shown in Tab. [4](https://arxiv.org/html/2505.23380v1#S4.T4), our method with SFT improves the GenEval score from 0.60 to 0.76, while the GRPO-based variant achieves 0.71—both outperforming the baselines by a large margin. In addition, our approach yields notable gains on DSG-1K and CLIP-Score, demonstrating the effectiveness of our post-training strategy in improving image generation quality. More results will be shown in Sec. [A.4](https://arxiv.org/html/2505.23380v1#A1.SS4).

**Table 4: Comparison of UniRL with baseline methods across Text-to-Image (T2I) benchmarks.**
| Method | GenEval $\uparrow$ | DSG-1K $\uparrow$ | Clip score $\uparrow$ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Single. | Two. | Count. | Colors | Posi. | Attri. | Overall |  |  |  |
| Show-o | 0.96 | 0.64 | 0.67 | 0.81 | 0.25 | 0.37 | 0.60 | 0.77 | 0.331 |
| HermersFlow | 0.97 | 0.67 | 0.65 | 0.77 | 0.28 | 0.42 | 0.61 | 0.78 | 0.334 |
| UniRL(SFT) | 0.99 | 0.93 | 0.62 | 0.89 | 0.55 | 0.68 | 0.77 | 0.82 | 0.337 |
| UniRL(GRPO) | 0.95 | 0.77 | 0.65 | 0.82 | 0.50 | 0.60 | 0.71 | 0.80 | 0.335 |
| Janus | 0.96 | 0.64 | 0.29 | 0.81 | 0.49 | 0.46 | 0.60 | 0.80 | 0.332 |
| UniRL(GRPO) | 0.95 | 0.74 | 0.27 | 0.81 | 0.62 | 0.52 | 0.65 | 0.81 | 0.335 |

Multimodal understanding.
We evaluate the model’s ability to understand natural visual features across six categories: single object, two objects, counting, colors, position, and color attributes. These categories capture core aspects of visual understanding, including object recognition, counting, color perception, and spatial reasoning—fundamental properties of natural images. To assess generalization beyond the training distribution, we use images generated by external models such as JanusPro. The generated images, along with their corresponding questions, are then fed into the target model to evaluate its understanding capabilities on out-of-distribution visual inputs. We compute the accuracy for each categories, allowing a fine-grained analysis of the model’s generalization across different aspects of visual understanding.

As shown in Tab. [5](https://arxiv.org/html/2505.23380v1#S4.T5), our SFT-based method performs slightly worse than the original model, primarily due to overfitting, particularly in the single-object category. In contrast, our GRPO-based method achieves an overall score of 0.79, significantly outperforming all baselines. The underlying reasons are discussed in Sec. [4.4](https://arxiv.org/html/2505.23380v1#S4.SS4), while additional evaluation details and benchmark results are provided in Sec. [A.1](https://arxiv.org/html/2505.23380v1#A1.SS1) and Sec. [A.5](https://arxiv.org/html/2505.23380v1#A1.SS5).

**Table 5: Comparison of UniRL with baseline methods across Multi-modal Understanding(MMU) benchmarks.**
| Method | Single. | Two. | Count. | Colors | Posi. | Attri. | Overall$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Show-o | 0.99 | 0.88 | 0.79 | 0.98 | 0.20 | 0.54 | 0.71 |
| HermesFlow | 0.99 | 0.88 | 0.79 | 0.98 | 0.34 | 0.52 | 0.73 |
| UniRL(SFT) | 0.08 | 0.87 | 0.71 | 0.99 | 0.45 | 0.85 | 0.67 |
| UniRL(GRPO) | 0.92 | 0.88 | 0.77 | 0.94 | 0.46 | 0.72 | 0.79 |
| Janus | 0.93 | 0.93 | 0.92 | 0.97 | 0.38 | 0.74 | 0.84 |
| UniRL(GRPO) | 0.94 | 0.95 | 0.83 | 0.99 | 0.52 | 0.89 | 0.88 |

The imbalance between two tasks.
To evaluate the balance between image understanding and generation, we apply our proposed metric, as shown in Tab. [6](https://arxiv.org/html/2505.23380v1#S4.T6). The prompts used for $\text{Accuracy}_{\text{MMU}|\text{T2I}}$ are taken from the GenEval benchmark, while those for $\text{Accuracy}_{\text{T2I}|\text{MMU}}$ are aligned with the MMU evaluation setup. The results show that our GRPO-based method significantly reduces the imbalance between the two tasks. For certain subtasks, the model achieves nearly 100% accuracy, demonstrating strong consistency between generation and understanding capabilities.

**Table 6: Comparison using our proposed metric to evaluate the imbalance between the two tasks.**
| Method | $\text{Accuracy}_{\text{MMU}|\text{T2I}}$ | $\text{Accuracy}_{\text{T2I}|\text{MMU}}$ |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Single. | Two. | Count. | Colors | Posi. | Attri. | Overall$\uparrow$ | Single. | Two. | Count. | Colors | Posi. | Attri. | Overall$\uparrow$ |  |
| Show-o | 0.98 | 0.89 | 0.92 | 0.94 | 0.10 | 0.61 | 0.83 | 0.96 | 0.66 | 0.67 | 0.85 | 0.18 | 0.41 | 0.70 |
| HermesFlow | 0.99 | 0.85 | 0.90 | 0.96 | 0.23 | 0.57 | 0.83 | 0.97 | 0.68 | 0.69 | 0.84 | 0.24 | 0.41 | 0.71 |
| UniRL(SFT) | 0.11 | 0.99 | 0.97 | 0.99 | 0.83 | 0.98 | 0.81 | 0.99 | 0.95 | 0.58 | 0.93 | 0.60 | 0.74 | 0.80 |
| UniRL(GRPO) | 0.99 | 0.97 | 0.90 | 0.99 | 0.81 | 0.85 | 0.93 | 0.97 | 0.85 | 0.70 | 0.90 | 0.67 | 0.68 | 0.81 |
| Janus | 0.91 | 0.59 | 0.71 | 0.96 | 0.20 | 0.42 | 0.67 | 0.95 | 0.67 | 0.27 | 0.90 | 0.39 | 0.49 | 0.68 |
| UniRL(GRPO) | 0.94 | 0.78 | 0.52 | 0.98 | 0.31 | 0.77 | 0.76 | 0.96 | 0.76 | 0.34 | 0.88 | 0.63 | 0.59 | 0.74 |

### 4.3 Visualization results

Figure: Figure 3: Qualitative comparison of our method with the original models on both text-to-image generation (T2I) and multimodal understanding (MMU) tasks.
Refer to caption: x3.png

As shown in Fig. [3](https://arxiv.org/html/2505.23380v1#S4.F3), we present qualitative comparisons between our method and the original models on both tasks. Our approach produces more visually faithful generations conditioned on prompts and yields more accurate answers for corresponding questions, demonstrating improved consistency across the two tasks. More results will be shown in Sec. [A.6](https://arxiv.org/html/2505.23380v1#A1.SS6).

### 4.4 Analysis of SFT vs. GRPO

From these experimental results, we can draw some key observations.

Observation 1: SFT is more prone to overfitting and memorization than GRPO in the image understanding task.

As shown in Tab. [5](https://arxiv.org/html/2505.23380v1#S4.T5), SFT performs significantly worse than both the original model and GRPO on the single object category, where the training and test sets are fully disjoint in both examples and object types. In contrast, categories like position and colors share abstract features (e.g., position or color attributes), which partially mitigate overfitting. Compared to SFT, GRPO demonstrates better generalization under the same conditions.

Observation 2: In the generation task, SFT outperforms GRPO under an end-to-end training pipeline.

As shown in Tab. [4](https://arxiv.org/html/2505.23380v1#S4.T4), SFT achieves better T2I results than GRPO, with no noticeable overfitting. We hypothesize two reasons for this difference. (1) MMU is more prone to overfitting than T2I, as it mainly learns from text token patterns, while T2I involves more structured image reconstruction. (2) Our training pipeline is end-to-end. The SFT supervision signal for MMU is more direct and localized, whereas for T2I it is more indirect and distributed, which may promote better generalization in image generation. Under indirect supervision in T2I, SFT may optimize more effectively, as its loss is directly based on ground-truth answers. More analysis will be discussed in Sec. [A.7](https://arxiv.org/html/2505.23380v1#A1.SS7).

Observation 3: For non-end-to-end training setups such as Janus, SFT leads to failure in the image generation task.

If we directly use the model’s generated images to supervise T2I at each iteration, training will become unstable. We speculate that early-stage generation errors are repeatedly reinforced, leading to input distribution drift and eventual collapse. Therefore, for unified models with different representations for the two tasks, only the GRPO-based method is effective for improving performance. More analysis will be discussed in Sec. [A.7](https://arxiv.org/html/2505.23380v1#A1.SS7).

Overall, GRPO is a more general and robust optimization strategy compared to SFT. Regardless of the specific task or model architecture—whether the two tasks share the same image representation or not—GRPO consistently improves performance and effectively reduces the imbalance between generation and understanding. In contrast, while SFT tends to perform better on tasks like T2I in end-to-end setups (e.g., Show-o), it is more prone to memorization and overfitting.

## 5 Conclusion

In this work, we propose a self-improving post-training method for unified multimodal models. By combining supervised fine-tuning (SFT) with Group Relative Policy Optimization (GRPO), our approach improves both generation and understanding performance while reducing task imbalance. Experiments on Show-o and Janus validate the effectiveness of our method.

## References

- [1]
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al.
Phi-3 technical report: A highly capable language model locally on your phone.
arXiv preprint arXiv:2404.14219, 2024.
- [2]
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al.
Gemini: A family of highly capable multimodal models.
arXiv preprint arXiv:2312.11805, 1, 2023.
- [3]
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.
Qwen-vl: A frontier large vision-language model with versatile abilities.
CoRR, abs/2308.12966, 2023.
- [4]
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie.
Sft or rl? an early investigation into training r1-like reasoning large vision-language models.
arXiv preprint arXiv:2504.11468, 2025.
- [5]
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin.
Mmdetection: Open mmlab detection toolbox and benchmark.
CoRR, abs/1906.07155, 2019.
- [6]
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan.
Janus-pro: Unified multimodal understanding and generation with data and model scaling.
CoRR, abs/2501.17811, 2025.
- [7]
Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar.
Masked-attention mask transformer for universal image segmentation.
In CVPR, pages 1280–1289. IEEE, 2022.
- [8]
Jaemin Cho, Yushi Hu, Jason M. Baldridge, Roopal Garg, Peter Anderson, Ranjay Krishna, Mohit Bansal, Jordi Pont-Tuset, and Su Wang.
Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation.
In ICLR. OpenReview.net, 2024.
- [9]
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge,
Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
CoRR, abs/2501.12948, 2025.
- [10]
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi.
DreamLLM: Synergistic multimodal comprehension and creation.
In ICLR, 2024.
- [11]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
An image is worth 16x16 words: Transformers for image recognition at scale.
In ICLR. OpenReview.net, 2021.
- [12]
Patrick Esser, Robin Rombach, and Bjorn Ommer.
Taming transformers for high-resolution image synthesis.
In CVPR, pages 12873–12883, 2021.
- [13]
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan.
Seed-x: Multimodal models with unified multi-granularity comprehension and generation.
arXiv preprint arXiv:2404.14396, 2024.
- [14]
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt.
Geneval: An object-focused framework for evaluating text-to-image alignment.
In NeurIPS, 2023.
- [15]
Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng-Ann Heng.
Can we generate images with cot? let’s verify and reinforce image generation step by step.
CoRR, abs/2501.13926, 2025.
- [16]
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghorbani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Elizabeth Proehl, Enoch Cheung, Eric Mitchell,
Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O’Connell, Ian Osband, Ignasi Clavera Gilaberte, and Ilge Akkaya.
Openai o1 system card.
CoRR, abs/2412.16720, 2024.
- [17]
Siqi Kou, Jiachun Jin, Chang Liu, Ye Ma, Jian Jia, Quan Chen, Peng Jiang, and Zhijie Deng.
Orthus: Autoregressive interleaved image-text generation with modality-specific heads.
CoRR, abs/2412.00127, 2024.
- [18]
Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulic, and Furu Wei.
Imagine while reasoning in space: Multimodal visualization-of-thought.
CoRR, abs/2501.07542, 2025.
- [19]
Hao Li, Changyao Tian, Jie Shao, Xizhou Zhu, Zhaokai Wang, Jinguo Zhu, Wenhan Dou, Xiaogang Wang, Hongsheng Li, Lewei Lu, and Jifeng Dai.
Synergen-vl: Towards synergistic image understanding and generation with vision experts and token folding.
CoRR, abs/2412.09604, 2024.
- [20]
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Zichun Liao, Yusuke Kato, Kazuki Kozuka, and Aditya Grover.
Omniflow: Any-to-any generation with multi-modal rectified flows.
CoRR, abs/2412.01169, 2024.
- [21]
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.
Evaluating object hallucination in large vision-language models.
In EMNLP, pages 292–305. Association for Computational Linguistics, 2023.
- [22]
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee.
Textbooks are all you need ii: phi-1.5 technical report.
arXiv preprint arXiv:2309.05463, 2023.
- [23]
Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Li, and Peng Gao.
Lumina-mgpt: Illuminate flexible photorealistic text-to-image generation with multimodal generative pretraining.
CoRR, abs/2408.02657, 2024.
- [24]
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel.
World model on million-length video and language with ringattention.
arXiv preprint, 2024.
- [25]
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.
Llava-next: Improved reasoning, ocr, and world knowledge, 2024.
- [26]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
NeurIPS, 36, 2024.
- [27]
Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, et al.
Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation.
arXiv preprint arXiv:2411.07975, 2024.
- [28]
Weijia Mao, Zhenheng Yang, and Mike Zheng Shou.
Unimod: Efficient unified multimodal transformers with mixture-of-depths.
CoRR, abs/2502.06474, 2025.
- [29]
Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K. Du, Zehuan Yuan, and Xinglong Wu.
Tokenflow: Unified image tokenizer for multimodal understanding and generation.
CoRR, abs/2412.03069, 2024.
- [30]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.
Learning transferable visual models from natural language supervision.
In ICML, pages 8748–8763, 2021.
- [31]
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
In NeurIPS, 2023.
- [32]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
CoRR, abs/1707.06347, 2017.
- [33]
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo.
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
CoRR, abs/2402.03300, 2024.
- [34]
Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu.
Llamafusion: Adapting pretrained language models for multimodal generation.
arXiv preprint arXiv:2412.15188, 2024.
- [35]
Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, Limin Wang, and Hongsheng Li.
Journeydb: A benchmark for generative image understanding.
In NeurIPS, 2023.
- [36]
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.
Emu: Generative pretraining in multimodality.
In ICLR, 2023.
- [37]
Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal.
Any-to-any generation via composable diffusion.
NeurIPS, 36, 2024.
- [38]
Chameleon Team.
Chameleon: Mixed-modal early-fusion foundation models.
arXiv preprint arXiv:2405.09818, 2024.
- [39]
Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu.
Metamorph: Multimodal understanding and generation via instruction tuning.
arXiv preprint arXiv:2412.14164, 2024.
- [40]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.
Llama: Open and efficient foundation language models.
CoRR, abs/2302.13971, 2023.
- [41]
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al.
Emu3: Next-token prediction is all you need.
arXiv preprint arXiv:2409.18869, 2024.
- [42]
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo.
Janus: Decoupling visual encoding for unified multimodal understanding and generation.
CoRR, abs/2410.13848, 2024.
- [43]
Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai.
Liquid: Language models are scalable multi-modal generators.
CoRR, abs/2412.04332, 2024.
- [44]
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua.
Next-gpt: Any-to-any multimodal llm.
arXiv preprint arXiv:2309.05519, 2023.
- [45]
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, Song Han, and Yao Lu.
VILA-U: a unified foundation model integrating visual understanding and generation.
CoRR, abs/2409.04429, 2024.
- [46]
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou.
Show-o: One single transformer to unify multimodal understanding and generation.
arXiv preprint arXiv:2408.12528, 2024.
- [47]
An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, Weijia Xu, Wenbiao Yin, Wenyuan Yu, Xiafei Qiu, Xingzhang Ren, Xinlong Yang, Yong Li, Zhiying Xu, and Zipeng Zhang.
Qwen2.5-1m technical report.
CoRR, abs/2501.15383, 2025.
- [48]
Ling Yang, Xinchen Zhang, Ye Tian, Chenming Shang, Minghao Xu, Wentao Zhang, and Bin Cui.
Hermesflow: Seamlessly closing the gap in multimodal understanding and generation.
CoRR, abs/2502.12148, 2025.
- [49]
Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.
MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI.
In CVPR, pages 9556–9567. IEEE, 2024.
- [50]
Rui Zhao, Weijia Mao, and Mike Zheng Shou.
Doracycle: Domain-oriented adaptation of unified generative model in multimodal cycles.
CoRR, abs/2503.03651, 2025.
- [51]
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy.
Transfusion: Predict the next token and diffuse images with one multi-modal model.
CoRR, abs/2408.11039, 2024.

## Appendix A Technical appendices

In this appendix, we provide additional implementation details of our experiments (Sec. [A.1](https://arxiv.org/html/2505.23380v1#A1.SS1)) and describe the formulation and explanation of the Straight-Through Gumbel-Softmax estimator (Sec. [A.2](https://arxiv.org/html/2505.23380v1#A1.SS2)). We then present more details and the loss formulation for the non-end-to-end training pipeline (Sec. [A.3](https://arxiv.org/html/2505.23380v1#A1.SS3)). Additional results for both tasks, along with extended visualization examples, are provided in Sec. [A.5](https://arxiv.org/html/2505.23380v1#A1.SS5), Sec. [A.4](https://arxiv.org/html/2505.23380v1#A1.SS4), and Sec. [A.6](https://arxiv.org/html/2505.23380v1#A1.SS6). We also offer further analysis comparing SFT and GRPO in non-end-to-end settings (Sec. [A.7](https://arxiv.org/html/2505.23380v1#A1.SS7)). Finally, we discuss the limitations of our work and its broader impact (Sec. [A.8](https://arxiv.org/html/2505.23380v1#A1.SS8) and Sec. [A.9](https://arxiv.org/html/2505.23380v1#A1.SS9)).

### A.1 More implementation details

Training details.
For the SFT or GRPO phase, we set the learning rate to $1\times 10^{-5}$ for both the Janus and Show-o models. We use the constructed prompts as input and randomly select one sentence from the six predefined categories. The constructed prompts serve as the training set, while the prompts from the GenEval benchmark are used as the test set.

Understanding benchmark. For the understanding benchmarks, we use other unified multimodal models to generate images based on the prompts from the GenEval benchmark. We then filter and retain only the correctly generated images. These images, along with the constructed questions, are used as inputs to evaluate the model’s understanding capability. We compute the accuracy as the percentage of predicted answers that match the ground-truth answers.

New metric. For the T2I$\rightarrow$MMU chain, we use the prompts from the GenEval benchmark for evaluation. For the MMU$\rightarrow$T2I chain, we follow the input format of our understanding benchmarks, feeding an external image and corresponding questions for evaluation. To assess the imbalance between the two tasks using our proposed metric, we employ external language models such as Qwen [47] or GPT [16] to judge whether the predicted answers match the ground-truth answers and compute accuracy for each category. For image evaluation, we follow the GenEval protocol, using MMDetection [5] and Mask2Former [7] to determine whether the generated image is correct.

### A.2 Straight-Through gumbel softmax

The Straight-Through Gumbel-Softmax (ST-GS) estimator enables differentiable sampling of discrete tokens, which is crucial for gradient-based optimization in end-to-end training. It approximates the sampling process using a soft relaxation during the backward pass while retaining hard discrete choices during the forward pass.

The Gumbel-Softmax distribution samples a continuous approximation of one-hot vectors using the following formula:

$$ $y_{i}=\frac{\exp\left(\dfrac{\log(\pi_{i})+g_{i}}{\tau}\right)}{\sum\limits_{j =1}^{K}\exp\left(\dfrac{\log(\pi_{j})+g_{j}}{\tau}\right)},$ (7) $$

where $\pi_{i}$ is the unnormalized probability of the $i$-th category, $g_{i}$ is sampled from the $\text{Gumbel}(0,1)$ distribution, $\tau$ is a temperature parameter controlling the smoothness, and $K$ is the number of categories.

In the straight-through version, a hard one-hot vector $\mathbf{z}$ is obtained during the forward pass by applying $\arg\max$ over the sampled logits:

$$ $z_{i}=\begin{cases}1,&\text{if }i=\arg\max\limits_{j}\,y_{j}\\ 0,&\text{otherwise}\end{cases}.$ (8) $$

While the forward pass uses hard selection, the backward pass uses the soft $y_{i}$ values to compute gradients, enabling smooth updates.

We adopt the Straight-Through Gumbel-Softmax in our method to allow gradients from downstream tasks to propagate through discrete image tokens. This is critical for end-to-end training, as it ensures that the generation module receives meaningful gradient signals and participates in joint optimization with the understanding module.

### A.3 Non end-to-end training pipeline

For non-end-to-end training, we decouple the GRPO loss into two separate objectives for the T2I and MMU modules.

T2I optimization.
For the image generation task, we directly optimize the image token logits using rewards computed from MMU performance. The GRPO loss is defined as:

$$ $\mathcal{L}_{\text{T2I}}(\theta)=-\,\sum_{k=1}^{K}w_{k}\,\log p_{\theta}\bigl{ (}\hat{\mathbf{u}}_{k}\mid p\bigr{)}+\beta\,\mathrm{KL}\left(p_{\theta}( \mathbf{u}\mid p)\,\|\,p_{\theta_{\mathrm{ref}}}(\mathbf{u}\mid p)\right),$ $$

where $p$ is the input prompt, $\hat{\mathbf{u}}_{k}$ is the $k$-th sampled image token sequence, and $w_{k}$ is a reward-based importance weight derived from MMU outputs.

MMU optimization.
For the multimodal understanding task, we optimize the answer prediction directly using rewards based on answer correctness. The GRPO loss is given by:

$$ $\mathcal{L}_{\text{MMU}}(\theta)=-\,\sum_{k=1}^{K}w_{k}\,\log p_{\theta}\bigl{ (}\hat{a}_{k}\mid\mathbf{u},\,q\bigr{)}+\beta\,\mathrm{KL}\left(p_{\theta}(a \mid\mathbf{u},q)\,\|\,p_{\theta_{\mathrm{ref}}}(a\mid\mathbf{u},q)\right),$ $$

where $\mathbf{u}$ is the input image token sequence, $q$ is the question, $\hat{a}_{k}$ is the predicted answer, and $w_{k}$ is computed based on the answer reward.

Although the two modules are optimized independently, reward signals still facilitate cross-task influence, allowing the model to benefit from mutual supervision even without gradient sharing.

### A.4 More generation results

In our main experiments using the Show-o model [46], both our method and the baselines adopt a guidance scale of 2 and 16 generation steps during training and inference. Here, we present additional comparison results under a higher guidance scale of 5 and an extended generation length of 50 steps. We observe that our SFT-based method achieves a final score of 0.79, while our GRPO-based method reaches 0.74—both significantly outperforming all baselines.

**Table 7: Comparison of UniRL with baseline methods on the GenEval benchmark under a guidance scale of 5 and 50 generation steps.**
| Method | Single. | Two. | Count. | Colors | Posi. | Attri. | Overall$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Show-o | 0.98 | 0.81 | 0.69 | 0.82 | 0.32 | 0.53 | 0.68 |
| HermesFlow | 0.98 | 0.84 | 0.66 | 0.82 | 0.32 | 0.52 | 0.69 |
| UniRL(SFT) | 1.00 | 0.97 | 0.61 | 0.91 | 0.56 | 0.70 | 0.79 |
| UniRL(GRPO) | 0.96 | 0.80 | 0.67 | 0.86 | 0.50 | 0.67 | 0.74 |

**Table 8: Benchmark results for multimodal understanding tasks.**
| Method | POPE$\uparrow$ | MMMU_val$\uparrow$ |
| --- | --- | --- |
| Show-o | 79.8 | 26.7 |
| UniRL (GRPO) | 78.1 | 25.9 |

### A.5 More understanding results

Since our work primarily focuses on the understanding of basic visual attributes, the main results on tasks such as counting and color recognition have been reported in Sec.[4.2](https://arxiv.org/html/2505.23380v1#S4.SS2). We further evaluate our method on more comprehensive benchmarks, including POPE[21] and MMMU [49], as shown in Tab. [8](https://arxiv.org/html/2505.23380v1#A1.T8). The slightly lower scores can be attributed to the fact that our training data primarily focuses on basic understanding features. Besides, some benchmarks like MMMU focus more on text-based reasoning and numerical calculation, rather than visual understanding. We believe that incorporating a broader range of understanding categories during training, or using a larger model, may help mitigate this limitation in future work.

Figure: Figure 4: Visualization results of our method.
Refer to caption: x4.png

### A.6 More visualization results

We provide additional visualization results in Fig. [4](https://arxiv.org/html/2505.23380v1#A1.F4), including several out-of-distribution examples. These results demonstrate that our method (GRPO) generalizes beyond the specific categories seen during training.

### A.7 More analysis of SFT and GRPO

End-to-end training.
For end-to-end training, supervision for T2I is inherently indirect, as the optimization signal comes from downstream tasks such as answer prediction rather than direct supervision of the generated image. Compared to MMU, T2I is generally less prone to overfitting due to the structured nature of image reconstruction. In this setting, SFT may outperform GRPO, as it leverages the ground-truth answer to directly compute the loss and guide the model toward a clearer optimization direction. This process does not heavily depend on the capacity of the base model; instead, the ground-truth serves as a strong supervision signal that helps SFT converge more efficiently.

In contrast, GRPO optimizes the model by generating a group of candidate answers and computing rewards to update the policy. When the model capacity is limited, the generated answers may all be incorrect, leading to weak or noisy reward signals and thus making optimization more difficult.

For MMU, which primarily relies on learning from text-based inputs, SFT tends to overfit easily by memorizing token patterns. GRPO, on the other hand, benefits from its reward-based training and KL regularization term, which encourages the updated model to remain close to the original pretrained model. This constraint improves generalization and helps prevent overfitting, especially in cases where training data is limited or highly structured.

Non end-to-end training.
For non end-to-end training, applying SFT to T2I is often unstable due to error accumulation from the generation pipeline. The optimization direction can easily become unreliable, leading to training collapse. To mitigate this, one option is to follow prior works [48, 15], which use offline-generated images and filter out incorrect samples to construct a reliable training set for SFT. In comparison, GRPO is more robust in non end-to-end settings, as it directly optimizes with reward signals and demonstrates better stability and performance.

### A.8 Limitations

While our work presents an effective post-training method for unified multimodal models, it also has several limitations.
First, our current focus is limited to basic visual understanding tasks such as counting and color recognition. More complex reasoning abilities, such as mathematical problem-solving or abstract inference, are beyond the scope of this work and are left for future exploration. Second, the training speed per iteration is relatively slow. This is a known limitation of GRPO-based methods, as they require multiple forward passes during inference before optimization.

### A.9 Broader impacts

Our work explores an effective post-training method for unified multimodal models. We believe our method will not bring negative societal impacts.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: Please refer to the weaknesses.

- How sensitive is performance to the reward function design and group size K in GRPO?
soundness: 3
strengths: - The paper identifies a real gap in current u-MLLMs, i.e., the lack of mutual improvement between generation and understanding, and argues convincingly that post-training is an efficient place to address it.
- It is elegant and practically attractive for models to bootstrap its own samples. 
- Text is easy to follow.
summary: The authors introduce UniRL, a self-improving post-training framework for unified multimodal large language models (u-MLLMs) that perform both text-to-image generation (T2I) and multimodal understanding (MMU). In this way, the model is updated by the its self-generated data: the model generates images as well as answers questions about these images, and then optimizes itself using either SFT or GRPO.
weaknesses: - Limited novelty: The overall pipeline is a well-known paradigm in self-training and reinforcement learning.
- Narrow scope: The method is validated only on GenEval-like attributes, which represent low-level visual features. It remains unclear whether the approach generalizes to more complex open-domain tasks (e.g., human actions, reasoning, visual grounding, or captioning).
- The automatic reward relies on keyword or attribute matching. Without semantic or visual similarity measures (e.g., CLIP-based scoring or GPT-based judging), the reinforcement signal is weak and may not scale.
- Although UniRL improves GenEval and balance metrics, the absolute gains are moderate, and the experiments are limited to small model variants. The method does not appear to change the frontier of multimodal learning.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. How does UniRL's performance scale with model size?
soundness: 3
strengths: - **No external training data requirement:** The proposed UniRL uses only model-generated images for post-training, making it more practical and scalable, eliminating the need for expensive real-world data collection and annotation. 

- **Valuable consistency evaluation metric:** The proposed bidirectional metrics, Accuracy(MMU|T2I) and Accuracy(T2I|MMU), is a useful and well-conceived to quantify the alignment between generation and understanding tasks. 

- **Insightful analysis:** This paper provides clear observations explaining why GRPO outperforms SFT for generalization while SFT excels for T2I in end-to-end settings.
summary: This paper proposes a self-improving post-training method (UniRL) for unified MLLMs (u-MLLMs). UniRL consists of both SFT and RL stages, and enables u-MLLMs to generate their own synthetic training data without relying on external datasets. Moreover, the paper introduces a new metric to quantify task imbalance. Experiments on Show-o and Janus demonstrate performance improvements in both tasks over baseline u-MLLMs.
weaknesses: - **Synthetic data reliability and quality control:** While self-generated training data of u-MLLMs is data-friendly, the quality and diversity of generated synthetic data can not be guaranteed. Without guarantees of generated data, the self-training loop risks reinforcing generation biases or error accumulation over iterations. How do you ensure that the self-generated images remain sufficiently diverse and informative across iterations? Is there any mechanism to prevent the accumulation of low-quality or mode-collapsed samples?

- **Limited evaluation and comparison:** The experiments primarily focus on comparisons against baselines, lacking more comprehensive evaluations with leading u-MLLMs of comparable parameter scales and other GRPO-trained u-MLLMs.

- **Reward function design is too empirical:** The reward rules are hand-crafted keyword matching. It lacks universality and may introduce bias toward specific phrasings or synonyms.

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: I would consider raise my score if authors could address my concerns, especially on the evaluation results and details.
soundness: 3
strengths: 1. The proposed self-improving framework is technically sound. By using generated data for iterative training, no external data is required and the mutual reinforcement of T2I and MMU tasks addresses the generation-understanding imbalance pervasive in unified models.
2. The integration of SFT and GRPO  pave the way for further performance improvement in the context of unified models.
3. The overall presentation is clear and easy to follow.
4. The analysis of SFT v.s. GRPO provide valuable insights.
summary: UniRL introduces a self-improving post-training framework for unified multimodal models (e.g., Show-o, Janus) that jointly optimizes text-to-image (T2I) generation and multimodal understanding (MMU) using supervised fine-tuning (SFT) and Group Relative Policy Optimization (GRPO). The core innovation is a closed-loop pipeline where the model generates images from prompts, answers questions about them, and uses the feedback to improve both tasks without external data. Besides, a bidirectional evaluation metric (Accuracy_MMU|T2I, Accuracy_T2I|MMU)  is introduced to quantifies task alignment.
weaknesses: 1. In table 2 and table 4, the performance of Count. underperforms than that of the baseline model. However, the prompts construction and evaluation pipeline  in figure 1 and figure 2  show that the Count. performance could be improved with the proposed framework. Why the performance is worser? More evaluation details should be included to clarify this.
2. The evaluation only foucuses on basic visual attributes (counting, color, position) but neglects complex reasoning (e.g., physics, temporal dynamics). Accordingly, the performance improvement on GenEval could be "hacked" by the constructed prompt suit for iterative self-improving. In other words, the performance gains might be attribute to the iterative optimization on the isomorphic prompt set instead of actual improvement. Are there any ood results to better reflect the effectiveness of the proposed method?
3. More recent unified models should be involved for evaluation and comparison such as BAGEL, SHOW-O2, BLIP-3O, etc.
4. More evaluation on multimodal understanding benchmarks such as MMMU, GQA, MME, etc. , should be included, current results in table 3 only provide limited evaluation scope for the author's claim on improving generation and understanding simutaneously.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In Table 4, UniRL (SFT) AccuracyMMU|T2I for “Single.” is 0.11. Is this a typo or a real phenomenon? If real, what causes such a stark collapse limited to this subtask? Similarly, Table 3 shows UniRL (SFT) “Single” = 0.08 for MMU. Could you share per-class confusion, prompt/QA distributions, and seed variance to rule out evaluation or data-pipeline issues?

2. To address general capability decline, you propose "incorporating a broader set of understanding categories" and "scaling up the model size." For the former, what specific new categories (e.g., math word problems, logical syllogisms) would you prioritize, and how would you design prompts/QA pairs for them to ensure alignment with both T2I and MMU? For the latter, do you have preliminary results or projections on how model scaling (e.g., increasing parameter count from Show-o’s base size) would impact text-based reasoning and numerical calculation performance? Would scaling alone be sufficient, or does it require paired adjustments to the training framework?

3. MMMU’s strength lies in cross-discipline reasoning (e.g., combining visual cues with text-based math). Your current T2I-MMU mutual enhancement relies on basic visual attributes—how would you modify UniRL to enable the model to generate images that support text-based reasoning tasks?
soundness: 3
strengths: 1. UniRL eliminates the need for external image data by using the model’s real-time generated images as training samples. This addresses a critical pain point of existing post-training methods (which often rely on large external datasets like JourneyDB) and reduces data acquisition costs, making it more scalable.

2. Unlike methods that optimize T2I or MMU in isolation, UniRL designs a closed-loop system where T2I and MMU reinforce each other. This not only improves individual task performance (e.g., Show-o’s GenEval score rises from 0.60 to 0.77 via SFT) but also explicitly reduces task imbalance—a long-overlooked issue in unified multimodal models.

3. The proposed bidirectional metric Accuracy_MMU|T2I and Accuracy_T2I|MMU fills a gap in existing evaluations, which often focus on isolated task performance. This metric quantifies how well T2I and MMU align (e.g., whether a correctly generated image leads to a correct answer), providing a more holistic measure of unified model performance.
summary: This paper proposes UniRL, a self-improving post-training framework for unified multimodal large language models (e.g., Show-o, Janus) to address two key limitations of existing methods: heavy reliance on external image data and persistent imbalance between text-to-image generation (T2I) and multimodal understanding (MMU) tasks. UniRL operates without external image data by leveraging the model’s own generated images as training data in each iteration. It enables mutual enhancement between T2I and MMU-generated images are used to train understanding, while understanding results (e.g., answer accuracy) supervise generation.
weaknesses: 1. UniRL only targets basic visual attributes (counting, color, position) and ignores complex multimodal tasks (e.g., mathematical reasoning, abstract scene understanding, or object interaction inference). This restricts its applicability to real-world scenarios where models need to handle diverse, high-level tasks.

2. The credit assignment from MMU rewards to T2I in the decoupled setting is described at a high level; more details on stability tricks, baseline comparisons, and failure cases for SFT are needed.


Decision: Reject