OneFlow: Concurrent Mixed-Modal and Interleaved Generation with Edit Flows

## Abstract

Abstract We present OneFlow , the first non-autoregressive multimodal model that enables variable-length and concurrent mixed-modal generation. Unlike autoregressive models that enforce rigid causal ordering between text and image generation, OneFlow combines an insertion-based Edit Flow for discrete text tokens with Flow Matching for image latents. OneFlow enables concurrent text-image synthesis with hierarchical sampling that prioritizes content over grammar.
Through controlled experiments across model sizes from 1B to 8B, we demonstrate that OneFlow outperforms autoregressive baselines on both generation and understanding tasks while using up to 50% fewer training FLOPs. OneFlow surpasses both autoregressive and diffusion-based approaches while unlocking new capabilities for concurrent generation, iterative refinement, and natural reasoning-like generation.

## 1 Introduction

Native Multimodal Models — models capable of handling both multimodal understanding and generation within a single backbone — have advanced considerably in visual understanding and generation. These models typically employ a unified transformer architecture with next-token prediction to handle both discrete and continuous generation (Team, 2024; Wu et al., 2025a; Ma et al., 2025a; Deng et al., 2025b; Zhou et al., 2025). Recent work like Transfusion (Zhou et al., 2025) and Bagel (Deng et al., 2025b) demonstrates that leveraging modality-specific training objectives within shared architectures can significantly improve performance, particularly on continuous modalities such as vision.

However, both autoregressive (AR) and diffusion-based multimodal approaches face fundamental architectural constraints. Autoregressive models can handle interleaved data but require strict sequential generation — each image must be fully completed before text generation can continue, preventing simultaneous cross-modal refinement. Conversely, diffusion-based multimodal models such as MMaDA (Yang et al., 2025), FUDOKI (Wang et al., 2025), and Unidisc (Swerdlow et al., 2025) enable simultaneous mixed-modal generation but only for predetermined single text-image pairs where modality assignments must be known a priori and rely on independent time schedules for each modality. Neither paradigm supports the simultaneous generation of variable-length interleaved sequences.

We present OneFlow, the first model to achieve simultaneous generation of interleaved data. Unlike autoregressive models that enforce sequential completion of each modality, and unlike diffusion models restricted to fixed length generation, OneFlow combines an insertion-based discrete text generation using Edit Flows with Flow Matching for image generation. This enables concurrent refinement of both text and images with per-image time schedules, using a novel interleaved time schedule.

Through controlled experiments across various model sizes and compute regimes, we demonstrate that OneFlow outperforms both autoregressive (AR) and diffusion baselines on generation and understanding tasks while requiring 50% fewer training FLOPS. Moreover, we find that concurrent mixed modal pretraining yields 4% relative improvement on VQA and 1.5% on image generation over sequential pretraining. We summarize our contributions below.

## 2 OneFlow: Mixed-Modal Generation through Flow Matching

OneFlow handles multiple modalities through a sequence model, where elements in the sequence can be discrete tokens or continuous embeddings, e.g., of images. Concretely, let ${\mathcal{T}}$ denote the space of a single element of the sequence, which can take either a discrete value, up to some fixed vocabulary size $M$, or a continuous value, i.e., ${\mathcal{T}}=[M]\cup\mathbb{R}$.
Then our state space is defined as the set of all possible sequences up to some maximum length $N$, i.e., ${\mathcal{X}}=\bigcup_{n=0}^{N}{\mathcal{T}}^{n}$.

During generation, our model transforms noisy sequences into clean sequences. We do this by combining discrete-valued and continuous-valued generative processes. Specifically, we make use of the Edit Flows (Havasi et al., 2025) framework which enables variable-length sequence generation through the use of edit operations. It starts with a noisy sequence and iteratively applies edits until it is denoised into a generation. We focus on the *insertion* capabilities of Edit Flows, which is conceptually simple yet extremely powerful, as it allows inserting arbitrary number of tokens—and images—into the generated sequence. When images are inserted, we initialize them with noise and then use Flow Matching (Lipman et al., 2024) to generate the image. Since the same model predicts both the text edits and the image denoising, OneFlow achieves variable-length, non-autoregressive joint image and text generation. In the following, we state equations with only intuitive justifications and explanations. Full mathematical details and derivations can be found in Appendix [D](https://arxiv.org/html/2510.03506v2#A4).

### 2.1 Discrete Text Generation via Edit Flows

Edit Flows uses a continuous-time Markov chain (CTMC) to iteratively refine variable-length discrete sequences. We start with an empty sequence $X_{0}=\emptyset$ at time $t=0$, and transform the sequence through insertion operations.
Let ${{\text{ins}}}(x,i,a)$, $x\in{\mathcal{X}},i\in\{1,\dots,n\},a\in{\mathcal{T}}$, be the sequence resulting from inserting the token value $a$ to the right of position $i$ of the sequence $x$, resulting in

$$ ${{\text{ins}}}(x,i,a)=(x^{1},\dots,x^{i},a,x^{i+1},\dots,x^{n}).$ (1) $$

This forms the primitive operation that we use during generation.

During training, we take a data sequence $X_{1}$ and randomly delete tokens with equal probability to obtain $X_{t}$. This defines the process $X_{[0,1]}$ that we will fit to. The probability of each token being deleted is set by a monotonic scheduler $\kappa_{t}$ with $\kappa_{0}=0,\kappa_{1}=1$.

$$ $\mathbb{P}(x^{i}\text{ in }X_{t})=\kappa_{t},\qquad\text{ for each }x^{i}\in X_{1}.$ (2) $$

In preliminary experiments, we tested different $\kappa_{t}$ but found that the linear schedule $\kappa_{t}=t$ works most consistently across our diverse benchmarks. Deleted tokens are removed from the sequence. Noting that on average we retain $\mathbb{E}_{t}[\kappa_{t}]$ fraction of the original tokens, with the linear schedule we retain $50\%$ of the data sequence. This can lead to significant FLOPS savings during training, and tuning the scheduler can save even more if desired.

#### 2.1.1 Parameterization

The parameterization of an Edit Flow model for insertions naturally decomposes into two predictions: (i) how many tokens are missing at the right of position $i$, and (ii) which tokens are missing.
Thus, at each position $i$ of the sequence, our model outputs two quantities

- •
$\lambda^{i}:\mathcal{X}\rightarrow\mathbb{R}^{+}$ is a scalar that predicts the *number of missing tokens* between $i$ and $i+1$.
- •
$Q^{i}:\mathcal{X}\rightarrow[M]$ is a normalized distribution that predicts *what tokens are missing*.

These two predictions form the CTMC rate, so for sampling we would use (up to $o(h)$ error),

$$ $\mathbb{P}\left(X_{t+h}={{\text{ins}}}(X_{t},i,a)\;|\;X_{t}\right)=h\tfrac{\dot{\kappa}_{t}}{1-\kappa_{t}}\lambda^{i}(X_{t})Q^{i}(a|X_{t}).$ (3) $$

The ratio $\tfrac{\dot{\kappa}_{t}}{1-\kappa_{t}}$ dictates the distribution of insertion times according to the schedule $\kappa_{t}$ imposed during training ([2](https://arxiv.org/html/2510.03506v2#S2.E2)), where $\dot{\kappa}_{t}=\tfrac{d\kappa_{t}}{dt}$.
Note that unlike prior work (Havasi et al., 2025), we factor out this ratio $\tfrac{\dot{\kappa}_{t}}{1-\kappa_{t}}$ from the rate predictions and use a simplified model that is independent of $t$. In our practical implementation, we do not feed time values into the network for predicting insertions.
While not theoretically justified, we found this $t$-independence assumption to work better in practice, likely because $X_{t}$ already contains sufficient information for predicting the insertions.

#### 2.1.2 Insertion prediction λ i \lambda^{i}

The main component that determines whether insertions occur is the prediction head $\lambda^{i}$, which is trained by regressing onto the number of missing tokens.
Each position $i$ of the noisy sequence $X_{t}$ has a corresponding number of missing tokens $k^{i}$, which is the number of deleted tokens between $X_{t}^{i}$ and $X_{t}^{i+1}$.
The original Edit Flows loss was constructed through a choice of Bregman divergence (Holderrieth et al., 2024) which results in

$$ $\ell_{\text{Poisson}}(\lambda^{i})=\textstyle\sum_{i}\lambda^{i}(X_{t})-k^{i}\log\lambda^{i}(X_{t}).$ (4) $$

Alternatively, ([4](https://arxiv.org/html/2510.03506v2#S2.E4)) can be interpreted as the negative log-likelihood of a Poisson distribution, so $\lambda^{i}$ is trained to fit a Poisson distribution to model missing token counts $k^{i}$. However, the distribution of $k$ has a very high concentration for $k^{i}=0$. Furthermore, during sampling, the key prediction is whether the missing token count is either zero or nonzero.
As such, we explicitly model the probability of inserting zero tokens.

$$ $\mathbb{P}(k=0)=\pi,\quad\text{ and }\quad\mathbb{P}(k)=(1-\pi)\text{Pois}(k;\lambda_{\text{nonzero}}\;|\;k>0)\quad\text{ for $k>0$}$ (5) $$

where $\pi\in(0,1)$ is the probability of observing zero, and $\lambda_{\text{nonzero}}>0$ is the rate parameter but restricted to only modeling the distribution of nonzero counts.
We train $\pi$ by using a binary cross entropy (BCE) loss to detect if the missing count is zero, and we train $\lambda_{\text{nonzero}}$ using the original loss (([4](https://arxiv.org/html/2510.03506v2#S2.E4))) on nonzero counts.
For sampling ([3](https://arxiv.org/html/2510.03506v2#S2.E3)), we can use the expectation $\lambda^{i}(X_{t})=(1-\pi^{i}(X_{t}))\lambda^{i}_{\text{nonzero}}(X_{t})$.
However, we found that a consistently better sampling strategy is to first sample whether there are zero insertions using $\pi$, then simply use the rate $\lambda^{i}_{\text{nonzero}}(X_{t})$ if there are nonzero insertions. We provide sampling pseudocode in Algorithm [2](https://arxiv.org/html/2510.03506v2#alg2).

#### 2.1.3 Bag-of-tokens prediction Q i Q^{i}

To determine what token to insert at each position, we make use of the output head $Q$ which is a softmax over the discrete vocabulary $[M]$. We use the same Edit Flows loss, which is a sum of cross-entropy loss. Let $\mathcal{A}_{i}$ denote the set of deleted tokens between $X_{t}^{i}$ and $X_{t}^{i+1}$, then for each position $i$, the loss is

$$ $\ell_{\text{tokens}}(Q^{i})=-\textstyle\sum_{a\in\mathcal{A}_{i}}\log Q^{i}(a|X_{t}).$ (6) $$

#### 2.1.4 Combined loss

At each training iteration, we randomly delete tokens from the data sequence, and learn to predict the set of missing tokens at each position, resulting in the total insertion loss:

$$ $\mathcal{L}_{\text{text}}=\mathbb{E}_{t,X_{t}|X_{1}}\left[\frac{1}{n}\sum_{i=1}^{n}\ell_{\text{tokens}}(Q^{i})+\ell_{\text{Poisson}}(\lambda^{i}_{\text{nonzero}})\bm{1}_{[k_{i}>0]}+\ell_{\text{BCE}}(\pi^{i})\right]$ (7) $$

where $n$ is the length of the noisy sequence $X_{t}$. Note that this differs from the original training objective in Edit Flows (Havasi et al., 2025) which additionally weights the loss by the factor $\tfrac{\dot{\kappa}_{t}}{1-\kappa_{t}}$, not affecting the optimal solution. We found that not using this factor produces better results.

### 2.2 Continuous Image Generation via Flow Matching

Following standard practice, we generate images starting from a Gaussian noise of fixed dimension $N_{\text{img}}$, applying a deterministic generation procedure that follows an ordinary differential equation. Let $Y_{t}\in\mathbb{R}^{N_{\text{img}}}$ denote the noisy image, then the generative process is

$$ $\tfrac{d}{dt}Y_{t}=v(Y_{t},t),\qquad Y_{0}\sim\mathcal{N}(0,I),$ (8) $$

where $v:\mathbb{R}^{N_{\text{img}}}\times\mathbb{R}\rightarrow\mathbb{R}^{N_{\text{img}}}$ is a velocity field that determines the direction to transform $Y_{t}$ into a clean sample by $t=1$.
During training, we sample a noise $Y_{0}$ and obtain $Y_{t}$ with a linear schedule $Y_{t}=tY_{1}+(1-t)Y_{0}$. The Flow Matching loss can then be written as

$$ $\mathcal{L}_{\text{image}}=\mathbb{E}_{t,Y_{0},Y_{1}}\left\lVert v(Y_{t},t)-(Y_{1}-Y_{0})\right\rVert^{2}.$ (9) $$

In OneFlow, we use a pretrained autoencoder to map images into latent space. We then design the velocity network $v(\cdot)$ to use a shared Transformer backbone as text but with additional U-Nets to downsample and upsample between the backbone and autoencoder embedding spaces, making use of the same architectural design as Transfusion (Zhou et al., 2025). See illustration in Figure [20](https://arxiv.org/html/2510.03506v2#A3.F20).

### 2.3 Concurrent Mixed-modal Generation

To generate multiple modalities, we simply concatenate them into a single sequence. We now present two multimodal time schedules, an independent schedule that can be used when the number of images is known, and an interleaved schedule that needs to be used when the number of images is arbitrary. OneFlow is designed to work with variable-length text and variable number of images.

#### 2.3.1 Independent mixed-modal generation

We can consider the simple case with a fixed number of images—typically one. In such case, we can generate both the text and image simultaneously by using two time values $t_{\text{text}}$ and $t_{\text{img}}$, where $t_{\text{text}}$ determines the state of the insertion generation process and $t_{\text{img}}$ determines the image generation process.
Following prior work, we simply set independent time schedules, one for the text and one for each image. This allows the modalities to be concurrently generated and be dependent on each other during the generation process. However, this naïve process does not allow us to insert images.

#### 2.3.2 Interleaved mixed-modal generation

A much more complicated setting arises when the number of images is variable and images are being inserted as part of the generation process. Similar to the text-only setting, we start generating from the empty sequence.
We then model image insertion as a special token value 
, which is added to the token prediction output $Q$.
During generation, when the model predicts an image insertion, we insert noise embeddings of dimension $N_{\text{img}}$ into the sequence to represent an inserted image initialized at $t_{\text{img}}=0$.

$$ ${{\text{ins}}}(x,i,\texttt{
})=\big(x^{1},\dots,x^{i},y^{1},\dots,y^{N_{\text{img}}},x^{i+1},\dots,x^{n}\big),\qquad y^{i}\sim\mathcal{N}(0,I).$ (10) $$

Subsequent steps during generation would then simultaneously generate the image embeddings while also performing more insertions into the sequence. However, since the image is generated at a later time, this implies there is a delay between the image time and the text time, i.e. $t_{\text{img}}\leq t_{\text{text}}$, which needs to be taken into account during training.

During training, we need to ensure that the text and image noise levels are consistent with with the ones seen during generation.
Based on the schedule in ([2](https://arxiv.org/html/2510.03506v2#S2.E2)), the time at which an insertion happens is a random variable that has $\kappa$ as its cumulative density function, so the time difference between the inserted image time $t_{\text{img}}$ and the initial text time $t_{\text{text}}$ is given by

$$ $t_{\text{img}}=t_{\text{text}}-\kappa^{-1}(u),\qquad\text{ where }u\sim\text{Unif(0, 1)}.$ (11) $$

We call this the *interleaved time schedule*, which imposes a distributional dependency between the time values $t_{\text{img}}$ for each image and the text time $t_{\text{text}}$.
In order for the model to learn to fully generate all images, during training we sample from an extended time interval, $\tau_{\text{text}}$ from $[0,2]$, since the 
token can be inserted at $\tau_{\text{text}}=1.0$ at the latest, and fully denoised by $\tau_{\text{text}}=2.0$. The probability for each token being in $X_{t}$ is then determined by $\kappa(\min\{1,\tau_{\text{text}}\})$ in place of ([2](https://arxiv.org/html/2510.03506v2#S2.E2)).
We also sample for each image an extended time value $\tau_{\text{img}}=\tau_{\text{text}}-\kappa^{-1}(u)$.
Finally, we determine if an image is deleted from the sequence by checking $\tau_{\text{img}}<0$, and if so, the insertion loss (([7](https://arxiv.org/html/2510.03506v2#S2.E7))) will include the 
 token which the model would learn to insert. Otherwise if $\tau_{\text{img}}\geq 0$, the image is in the sequence and we set $t_{\text{img}}=\min\{1,\tau_{\text{img}}\}$; using the Flow Matching loss ([9](https://arxiv.org/html/2510.03506v2#S2.E9)) to train the velocity. A detailed derivation and in-depth explanation can be found in Appendix [D.1](https://arxiv.org/html/2510.03506v2#A4.SS1).

## 3 Findings and Unlocking New Capabilities in Unified Multimodal

To study the performance of OneFlow against AR, we design a series of controlled experiments. We establish strong baselines by comparing both our approach and AR against other unified multimodal models in the literature. Finally, we explore the novel capabilities that OneFlow enables beyond existing methods. We present our experimental results through five research questions:

| §[3.1](https://arxiv.org/html/2510.03506v2#S3.SS1) | How does OneFlow scale compared to AR? |
| --- | --- |
| §[3.2](https://arxiv.org/html/2510.03506v2#S3.SS2) | What is the impact of mixed-modal vs. sequential pretraining? |
| §[3.3](https://arxiv.org/html/2510.03506v2#S3.SS3) | What emergent behaviors does OneFlow exhibit during generation? |
| §[3.4](https://arxiv.org/html/2510.03506v2#S3.SS4) | How does OneFlow compare to other unified multimodal models? |
| §[3.5](https://arxiv.org/html/2510.03506v2#S3.SS5) | What new capabilities does OneFlow enable beyond existing methods? |

#### 3.0.1 Training stages

Our training consists of two main stages: multimodal pretraining and instruction finetuning. During the pretraining stage, we use a mixture of image understanding and image generation data to learn representations for both image and text. We trained with a sequence length of 512 and a global batch size of 4096. We set the mixed generation probability (the likelihood of concurrently generating clean text and the image) to be either 0 or 0.2.

For finetuning, we use a mixture of VQA, text, and interleaved data to give the model the ability to respond to visual question answering problems. We also fine-tune on image generation data at a higher resolution of 512$\times$512 to improve the model’s image generation capabilities. We study the model’s behavior at the 1B scale for our ablations and controlled experiments, and the scaling trend up to 8B is detailed in Section [3.1](https://arxiv.org/html/2510.03506v2#S3.SS1).

#### 3.0.2 Datasets

For multimodal pretraining, we use image-text pairs from a filtered version of the Conceptual Captions dataset (CC12M (Sharma et al., 2018)), the YFCC dataset (Thomee et al., 2016), and licensed data, for a total of 400M examples.
During instruction finetuning, we use a filtered image portion of the PerceptionLM dataset (Cho et al., 2025), interleaving data from Chameleon (Team, 2024), and a filtered portion of the Cambrian-7M (Tong et al., 2024a) dataset, totaling around 40M examples.

#### 3.0.3 Baselines

To evaluate our model’s performance against existing methods, we compare against two baselines: (1) an autoregressive (AR) + Flow Matching (FM) multimodal model based on Transfusion (Zhou et al., 2025), where text tokens are generated autoregressively and image tokens via FM, and (2) a masked diffusion model based on LLaDA (Nie et al., 2025). For the masked diffusion baseline, we tested two sampling variants: low-confidence and random remasking, with random remasking performing better across all experiments. Unlike Transfusion, we follow Janus-Flow (Ma et al., 2025a) and adopt a dual-encoder setup. For image encoders, we use a pretrained SigLIP2 ViT-SO400M-16@512 (Tschannen et al., 2025) for understanding and an SD3 VAE (Esser et al., 2024) for generation. Following Transfusion, we use U-Net adapters. Figure [20](https://arxiv.org/html/2510.03506v2#A3.F20) illustrates how OneFlow consumes a mixed-modal sequence. To improve performance of the AR model, we follow Chameleon (Team, 2024) and sweep over several generation lengths for optimizing CIDEr score.
For image generation, we use a first order Euler solver with entropy rectifying guidance (Ifriqi et al., 2025), we set the guidance scale to $5.0$ across $50$ sampling steps.

#### 3.0.4 Model Configurations

To investigate scaling trends, we train models at three different sizes – 1B,
3B, and 8B parameters, and token counts. For each benchmark, we plot all results over log-FLOPS curve and regress linear trendlines. Following Transfusion, we also estimate relative compute efficiency by measuring the parity FLOPS ratio: the ratio between the number of FLOPS required by our AR and OneFlow to reach the same level of performance.

#### 3.0.5 FLOPS Estimation

To estimate the FLOPS during training, we track the number of tokens throughout training for all models. During multimodal pretraining, we do not delete images and OneFlow uses on average 50% fewer generated text tokens compared to AR.
We use Flash Attention and follow their FLOPS estimation (Dao et al., 2022).

#### 3.0.6 Evaluation setup

Following Cambrian (Tong et al., 2024a) and PLM (Cho et al., 2025), we group VQA tasks into five groups: General, Knowledge, OCR & Chart, Hard Perception, and Hallucination. We evaluate image generation quality using the FID metric (Heusel et al., 2017) on the COCO-2014 (Lin et al., 2014) validation set, using our base model at 256$\times$256 resolution. To assess prompt alignment, we report CLIPScore (Hessel et al., 2021) and DPG-Bench (Hu et al., 2024). Additionally, we include WISE (Niu et al., 2025) cultural to better understand knowledge-based generation.

### 3.1 OneFlow Scales Better than AR

Figure: Figure 5: Performance of OneFlow vs. AR +FM baseline models at different model scales, data and compute. For text-to-image generation, we report DPG-Bench and FID. For image-to-text caption quality, we report CIDEr and ROUGE. In every benchmark, OneFlow consistently exhibits better scaling laws than AR + FM. Model sizes include 1B, 3B, and 8B.
Refer to caption: x4.png

We first discuss controlled experiments, where we study the performance across different frameworks when trained on the same data sets using the same backbone architecture.
Specifically, we study the performance of OneFlow and AR in controlled settings at various model sizes and token counts. To ensure OneFlow has no advantage in data-constrained settings, we trained both models on 2B image-text pairs over 500k iterations using a batch size of 4096. Both models were initialized from Llama 3.2 1B (AI@Meta, 2024). For AR, the number of tokens predicted during training equals the sequence length, whereas for OneFlow, the number of predicted tokens corresponds to the number of deleted tokens, which on average is 50% of the data sequence. We measure compute efficiency using the parity FLOP ratio (Table [1](https://arxiv.org/html/2510.03506v2#S3.T1)), which quantifies the relative FLOPS required by OneFlow to match AR’s performance. Our results show that OneFlow achieves the largest compute savings on dense prompt alignment and captioning tasks, requiring less than half the FLOPS of AR to reach the same level.

We find that OneFlow scales better than AR on every benchmark. This scaling advantage is especially pronounced on DPG Bench, where OneFlow scales significantly better than AR. For image captioning as well, OneFlow shows a notable performance gap relative to AR. Figure [5](https://arxiv.org/html/2510.03506v2#S3.F5) visualizes the scaling trend, and the final metrics are shown in Tables [2](https://arxiv.org/html/2510.03506v2#S3.T2) and [3](https://arxiv.org/html/2510.03506v2#S3.T3), along with a comparison against other state-of-the-art models. A detailed breakdown of the controlled comparisons among VQA task categories can be found in Table [17](https://arxiv.org/html/2510.03506v2#A2.F17).

**Table 1: Parity FLOP ratio in a controlled experiment. Both models were trained on 2B image-text pairs. Parity FLOP Ratio represents the relative amount of OneFlow FLOPs needed to match the final AR + FM performance.**
|  | DPG $\uparrow$ | FID $\downarrow$ | CIDEr $\uparrow$ | ROUGE $\uparrow$ |
| --- | --- | --- | --- | --- |
| Parity FLOPS Ratio | 0.49 | 0.97 | 0.32 | 0.42 |

### 3.2 Mixed Modal Pre-training Enables Better Generation and Understanding

Figure: Figure 6: Mixed modal vs Sequential pretraining. Mixed modal pretraining vs sequential pretraining. Mixed pretraining achieves 4% relative improvement on VQA tasks and slight improvements on image generation as well.
Refer to caption: x5.png

Figure: Figure 7: Mixed modal concurrent training.
Refer to caption: x6.png

In this section, we study the impact of mixed-modal pretraining by comparing concurrent and sequential training approaches. Specifically, we train two 1B variants: one using sequential pretraining (either text-to-image or image-to-text), and another using the same data but with 20% of examples incorporating concurrent generation.

Our ablation experiment reveals that mixed-modal training significantly improves downstream visual question answering performance (Figure [6](https://arxiv.org/html/2510.03506v2#S3.F6)) and provides a minor boost to image generation. These results demonstrate the benefits of concurrent mixed-modal pretraining. Given these consistent improvements, we use the mixed-modal pretrained model for all subsequent finetuning experiments.

Table: Table 2: Image generation and captioning benchmarks after multimodal pretraining. We find that mixed-modal training consistently improves performance. ^†Evaluated using official open-source model weights. Highlighting denotes best results across all models.

Table: Table 3: VQA performance comparison. OneFlow outperforms AR and Mask models across all benchmarks in controlled experiments using identical finetuning data. Highlighting shows best results in the 1B controlled comparisons. Our results are also competitive with existing autoregressive and discrete diffusion models. ^†Evaluated using official open source weights.

### 3.3 Hierarchical Generation Enables Emergent Reasoning

In Figure [3](https://arxiv.org/html/2510.03506v2#S0.F3.fig1), we present OneFlow’s sampling process when prompted with a visual question. In response to prompts such as "{question} Explain why.", OneFlow generates a reasoning chain before arriving at the final answer, without any Chain-of-Thought (CoT) (Wei et al., 2022) prompting or RL post-training. For example, when asked "Is there a snowboard in the image? Explain why.", the model first implicitly performs visual search by examining the image and searching through likely locations for the snowboard. Similarly, for the math puzzle in (Figure [3](https://arxiv.org/html/2510.03506v2#S0.F3.fig1) bottom), the model first identifies objects in the image that match the prompt description—the green sphere and the large shiny cylinder—before arriving at the final answer.

Our results align with findings in Physics of LLMs (Ye et al., 2024) and MetaMorph (Tong et al., 2024b), where the authors suggest that LLMs precompute reasoning graphs before generating tokens. However, our findings demonstrate that the model can perform the same reasoning chain without autoregressive decoding. This suggests that reasoning capabilities can emerge in non-autoregressive architectures and transfer effectively to OneFlow. We show more example VQA generations compared to the AR baseline in Figure [9](https://arxiv.org/html/2510.03506v2#S3.F9).

Figure: Figure 8: Qualitative comparison of OneFlow and SOTA models. We notice that OneFlow gets the details of the prompt correctly, for instance the polar bear is ‘balancing on a blue barrel’. The visual details of our generation are also better compared to MMaDA possibly due to using continuous image tokens rather than discrete. In the last column, the figure shows OneFlow handles common semantic challenges more effectively, as it was able to generate a building with ‘a door sitting behind a sign’.
Refer to caption: figures/images/image_gen/Bagel/prompt_1_sample_0_task_1.png

Figure: Figure 9: (left) OneFlow first locates the object in the prompt before calculation. (right) OneFlow locates the target (stool), then analyzes color contrast. A red arrow is added for clarity and is not in the original image.
Refer to caption: x7.png

### 3.4 Comparison with State-of-The-Art Unified Models

In Tables [2](https://arxiv.org/html/2510.03506v2#S3.T2) and [3](https://arxiv.org/html/2510.03506v2#S3.T3), we benchmark OneFlow against other autoregressive and diffusion multimodal models. Our models deliver competitive performance across all tasks, rivaling state-of-the-art results despite significant differences in training procedures.

##### Understanding Performance.

On visual question answering, our 1B model nearly matches MMaDA 8B, while our 8B model surpasses MMaDA 8B on most benchmarks (Table [3](https://arxiv.org/html/2510.03506v2#S3.T3)). This is noteworthy because models like MMaDA received extensive post-training with chain-of-thought data and RL finetuning, whereas OneFlow did not. As discussed in Section [3.3](https://arxiv.org/html/2510.03506v2#S3.SS3), we demonstrate that reasoning naturally emerges during generation. Similarly, while FUDOKI leveraged a pretrained multimodal model, OneFlow delivers competitive performance without this advantage. We observe that initializing from a pretrained text-only autoregressive model consistently enhances both prompt alignment for image generation and VQA performance, yielding similar gains as the AR+FM model (Table [5](https://arxiv.org/html/2510.03506v2#A2.T5)).

##### Generation Performance.

For image generation, we measured FID using open-source weights on COCO-2014 (Lin et al., 2014) validation set. Since MMaDA and Bagel received extensive aesthetic finetuning that produces images stylistically distant from real images (yielding higher FID scores). For fair comparison, we benchmark against MMaDA Base and Bagel without thinking mode. We show qualitative comparisons in Figure [8](https://arxiv.org/html/2510.03506v2#S3.F8).

### 3.5 OneFlow Unlocks New Capabilities

#### 3.5.1 Classifier-free guidance improves text detailedness

Figure: Figure 10: Edit Flows with classifier-free guidance produces longer and more detailed answers, improving metrics that involve VLM as a judge (Cheng et al., 2025). Additional examples of classifier-free guidance effects on text generation are shown in Figure [16](https://arxiv.org/html/2510.03506v2#A1.F16).
Refer to caption: figures/images/000000065485.jpg

The use of continuous-time Markov chains allows us to apply classifier-free guidance (CFG) to our model’s insertion rates.
Specifically, given an unconditional prediction $\lambda(X_{t})Q(X_{t})$ and a conditional prediction $\lambda(X_{t}|c)Q(X_{t}|c)$, where $c$ is the prompt and $w$ is the guidance weight, the modified insertion rate is:

$$ $\lambda^{\text{cfg}}(X_{t}|c)=\lambda(X_{t}|c)^{w}\lambda(X_{t})^{1-w}\quad\text{ and }\quad Q^{\text{cfg}}(X_{t}|c)\propto Q(X_{t}|c)^{w}Q(X_{t})^{1-w}.$ (12) $$

As shown in Figures [10](https://arxiv.org/html/2510.03506v2#S3.F10) and [16](https://arxiv.org/html/2510.03506v2#A1.F16), higher CFG values consistently increase the length and detail of generated text. We quantitatively evaluated caption quality and hallucination using CapArena (Cheng et al., 2025) by prompting GPT4-o. Our findings show that increasing CFG leads to more detailed captions, with OneFlow matching AR’s level of detail at a guidance scale of 1. However, this increased detail comes at the expense of hallucinations at very high CFG values.

#### 3.5.2 Simultaneous generation of interleaved text and images

When autoregressive multi-modal models insert an image, they append it at the end of the current generation, fully denoise it, then continue the generation process. However, OneFlow is able to simultaneously denoise images and the text. When the model deems it necessary, it is able to insert an new image in the existing text and denoise it along with the text, as proposed in Section [2.3](https://arxiv.org/html/2510.03506v2#S2.SS3).

To train this model, we took OneFlow 1B Mixed and finetuned it on the interleaved subset of the Chameleon dataset (Team, 2024) for $20000$ steps. This subset contains $17000$ examples that interleave both text and image data.
Figure [4](https://arxiv.org/html/2510.03506v2#S0.F4) shows the generation order of the tokens where two images were generated as part of the answer, with more detailed examples in Appendix [A](https://arxiv.org/html/2510.03506v2#A1) and animated versions in the supplementary material.

## 4 Related Work

Figure: Autoregressive
Refer to caption: x9.png

Table: Table 4: High-level comparison between different frameworks for building unified models of text and image generation.

##### Unified Multimodal Models.

Early unified models like Flamingo (Alayrac et al., 2022) pioneered the integration of vision and language via cross-attention. Emu2 (Sun et al., 2023) advanced this paradigm by employing a unified autoregressive objective. Chameleon (Team, 2024) adopted an early-fusion, AR architecture that unifies images and text as discrete tokens. Subsequent work spans three architectural paradigms: diffusion (Yang et al., 2025; Li et al., 2025; Swerdlow et al., 2025; Shi et al., 2025; Wang et al., 2025), autoregressive (Alayrac et al., 2022; Team, 2024; Sun et al., 2024b; Chen et al., 2025b; Wu et al., 2025a; Yu et al., 2023), and hybrids (Zhao et al., 2024; Zhou et al., 2025; Xie et al., 2024; Tong et al., 2024b; Ma et al., 2025a; Shi et al., 2024b). Transfusion (Zhou et al., 2025) improves upon Chameleon by leveraging AR for text and diffusion for images. The Janus series collectively introduces decoupled visual pathways for understanding and generation (Wu et al., 2024; Ma et al., 2025b; Chen et al., 2025a). BAGEL scales multimodal pretraining using a Mixture‑of‑Transformers (MoT) (Liang et al., 2024; Deng et al., 2025a). In parallel, diffusion‑based models have emerged as promising alternatives: DDiT applies discrete diffusion to text and continuous diffusion to images (Li et al., 2025); UniDisc uses fully discrete diffusion with Llama tokenizer for text and MAGVIT‑v2 (Yu et al., 2024) for images (Swerdlow et al., 2025); FUDOKI uses discrete flow matching instead of masked diffusion, with a SigLIP (Zhai et al., 2023) encoder for understanding and a LlamaGen (Sun et al., 2024a) tokenizer for generation (Wang et al., 2025). MMaDA scales masked diffusion with mixed chain‑of‑thought fine‑tuning and UniGRPO (Yang et al., 2025). MuDDiT combines a pretrained T2I (Meissonic (Bai et al., 2024)) with a lightweight text decoder for MaskGIT‑style discrete diffusion (Chang et al., 2022; Shi et al., 2025). While these models are limited by a fixed generation order or fixed-length outputs, our approach fundamentally differs by simultaneously generating interleaved content and a variable number of images. We summarize the differences between each framework in Table [4](https://arxiv.org/html/2510.03506v2#S4.T4).

##### Discrete Diffusion and Discrete Flow Matching.

Iterative refinement models, including diffusion (Sohl-Dickstein et al., 2015; Ho et al., 2020) and flow models (Liu et al., 2022; Albergo et al., 2023; Lipman et al., 2024), have been adapted for discrete token spaces. Discrete diffusion models typically learn to reverse a corruption process (Austin et al., 2021; Lou et al., 2024), while discrete flow models transport between two distributions with an interpolating scheme (Campbell et al., 2024b; Gat et al., 2024).
These frameworks offer a large design space for discrete generative modeling (Shaul et al., 2024; Wang et al., 2025), and can further be generalized to a large family of continuous-time Markov processes with a unified training recipe (Holderrieth et al., 2024). However, recent works on discrete diffusion have predominantly focused on a simplified mask construction (Sahoo et al., 2024; Shi et al., 2024a; Ou et al., 2024; Zheng et al., 2024). This masking framework, however, cannot be easily applied to variable-length and especially simultaneous interleaved generation.

##### Edit-based Non-autoregressive Language Models.

Early non-autoregressive models for variable-length generation (Gu et al., 2019a, b; Stern et al., 2019; Reid et al., 2022) often relied on multiple models and evaluations to handle edit operations, e.g. by inserting placeholder tokens first and then replacing them with the actual tokens. While later work like Edit Flows (Havasi et al., 2025) improved on this by using a continuous-time framework and removes the need for any placeholder tokens. In order to finetune from existing mask diffusion models, Wu et al. (2025b) and Kim et al. (2025) have also explored inserting only mask tokens, breaking down insertions back into two steps. In contrast, we simply made use of autoregressive models as initialization, modifying only model outputs for insertion predictions with no special tokens used in the model inputs, as this allows us to save on training FLOPS and inference costs. Related to combining discrete and continuous generative processes,
Campbell et al. (2024a) also proposed modeling insertions with a diffusion model for denoising, but only considered singular insertions and required an additional generative model due to not using an interleaved time schedule.
In contrast, our approach considers mixed-modal sequence data, denoising images concurrently with per-image time values, and allows for parallel token insertions.

## 5 Conclusion and Limitations

We introduced OneFlow, a novel non-autoregressive multimodal model that overcomes the fixed-length generation limitations of diffusion models and has better scaling than autoregressive multimodal models.
We introduced mixed-modal generation approaches, which through extensive controlled experiments, improve on benchmarks for both image understanding and image generation.
We also propose a novel approach to interleaved generation that simultaneously denoises images and inserts text tokens, with promising qualitative results. Interleaved generation is still in its infancy and we expect to see more incoming research efforts in constructing large-scale data sets (Awadalla et al., 2024; Laurençon et al., 2023; Zhu et al., 2023) and designing comprehensive benchmarks.

A limitation of requiring bidirectional attention is the lack of key-value caching, which increases inference cost. We do find that OneFlow can already obtain good captioning performance with very few model evaluations—outperforming AR with only 6 sampling steps (Figure [19](https://arxiv.org/html/2510.03506v2#A2.F19)). Therefore, further reducing inference costs, with semi-autoregressive models (Arriola et al., 2025; Gat et al., 2025) or more sophisticated methods, would be an exciting research direction.

## Acknowledgements

We thank Maha Elbayad, Emily Dinan, Xiaochuang Han, Lili Yu, Chunting Zhou, and Melissa Hall for building the Transfusion code base which we built our code upon. We thank Peter Tong and David Fan for pointers on VQA and feedback on the paper. We thank Koustuv Sinha, Sharut Gupta, Marjan Ghazvininejad, Jakob Verbeek, and Amir Bar for feedback on the paper draft and fruitful discussions throughout the project.

## References

- AI@Meta (2024)
AI@Meta.
Llama 3 model card.
2024.
[https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md](https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md).
- Alayrac et al. (2022)
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikołaj Bińkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan.
Flamingo: a visual language model for few-shot learning.
In *Advances in Neural Information Processing Systems*, volume 35, pages 23716–23736, 2022.
- Albergo et al. (2023)
Michael S Albergo, Nicholas M Boffi, and Eric Vanden-Eijnden.
Stochastic interpolants: A unifying framework for flows and diffusions.
*arXiv preprint arXiv:2303.08797*, 2023.
- Arriola et al. (2025)
Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov.
Block diffusion: Interpolating between autoregressive and diffusion language models.
*arXiv preprint arXiv:2503.09573*, 2025.
- Austin et al. (2021)
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg.
Structured denoising diffusion models in discrete state-spaces.
*Advances in neural information processing systems*, 34:17981–17993, 2021.
- Awadalla et al. (2024)
Anas Awadalla, Le Xue, Oscar Lo, Manli Shu, Hannah Lee, Etash Guha, Sheng Shen, Mohamed Awadalla, Silvio Savarese, Caiming Xiong, et al.
Mint-1t: Scaling open-source multimodal data by 10x: A multimodal dataset with one trillion tokens.
*Advances in Neural Information Processing Systems*, 37:36805–36828, 2024.
- Bai et al. (2024)
Jinbin Bai, Tian Ye, Wei Chow, Enxin Song, Qing-Guo Chen, Xiangtai Li, Zhen Dong, Lei Zhu, and Shuicheng Yan.
Meissonic: Revitalizing masked generative transformers for efficient high-resolution text-to-image synthesis.
*arXiv preprint arXiv:2410.08261*, 2024.
- Campbell et al. (2024a)
Andrew Campbell, William Harvey, Christian Weilbach, Valentin De Bortoli, Thomas Rainforth, and Arnaud Doucet.
Trans-dimensional generative modeling via jump diffusion models.
*Advances in Neural Information Processing Systems*, 36, 2024a.
- Campbell et al. (2024b)
Andrew Campbell, Jason Yim, Regina Barzilay, Tom Rainforth, and Tommi Jaakkola.
Generative flows on discrete state-spaces: Enabling multimodal flows with applications to protein co-design.
*arXiv preprint arXiv:2402.04997*, 2024b.
- Chang et al. (2022)
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman.
Maskgit: Masked generative image transformer.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 11315–11325, 2022.
- Chen et al. (2025a)
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan.
Janus-pro: Unified multimodal understanding and generation with data and model scaling.
*arXiv preprint arXiv:2501.17811*, 2025a.
- Chen et al. (2025b)
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan.
Janus-pro: Unified multimodal understanding and generation with data and model scaling.
*arXiv preprint arXiv:2501.17811*, 2025b.
- Cheng et al. (2025)
Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, and Jiajun Chen.
Caparena: Benchmarking and analyzing detailed image captioning in the llm era.
*arXiv preprint arXiv:2503.12329*, 2025.
- Cho et al. (2025)
Jang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi, Triantafyllos Afouras, Tushar Nagarajan, Muhammad Maaz, Yale Song, Tengyu Ma, Shuming Hu, Suyog Jain, et al.
Perceptionlm: Open-access data and models for detailed visual understanding.
*arXiv preprint arXiv:2504.13180*, 2025.
- Dao et al. (2022)
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.
Flashattention: Fast and memory-efficient exact attention with io-awareness.
*Advances in neural information processing systems*, 35:16344–16359, 2022.
- Deng et al. (2025a)
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan.
Emerging properties in unified multimodal pretraining.
*arXiv preprint arXiv:2505.14683*, 2025a.
- Deng et al. (2025b)
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al.
Emerging properties in unified multimodal pretraining.
*arXiv preprint arXiv:2505.14683*, 2025b.
- Esser et al. (2024)
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al.
Scaling rectified flow transformers for high-resolution image synthesis.
In *Forty-first international conference on machine learning*, 2024.
- Gat et al. (2024)
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky T. Q. Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman.
Discrete flow matching.
*Advances in Neural Information Processing Systems*, 37:133345–133385, 2024.
- Gat et al. (2025)
Itai Gat, Heli Ben-Hamu, Marton Havasi, Daniel Haziza, Jeremy Reizenstein, Gabriel Synnaeve, David Lopez-Paz, Brian Karrer, and Yaron Lipman.
Set block decoding is a language model inference accelerator.
*arXiv preprint arXiv:2509.04185*, 2025.
- Gu et al. (2019a)
Jiatao Gu, Qi Liu, and Kyunghyun Cho.
Insertion-based decoding with automatically inferred generation order.
*Transactions of the Association for Computational Linguistics*, 7:661–676, 2019a.
- Gu et al. (2019b)
Jiatao Gu, Changhan Wang, and Junbo Zhao.
Levenshtein transformer.
*Advances in neural information processing systems*, 32, 2019b.
- Havasi et al. (2025)
Marton Havasi, Brian Karrer, Itai Gat, and Ricky T. Q. Chen.
Edit flows: Flow matching with edit operations.
*arXiv preprint arXiv:2506.09018*, 2025.
- Hessel et al. (2021)
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.
Clipscore: A reference-free evaluation metric for image captioning.
In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 7514–7528, 2021.
- Heusel et al. (2017)
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge to a local nash equilibrium.
*Advances in neural information processing systems*, 30, 2017.
- Ho et al. (2020)
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
*Advances in neural information processing systems*, 33:6840–6851, 2020.
- Holderrieth et al. (2024)
Peter Holderrieth, Marton Havasi, Jason Yim, Neta Shaul, Itai Gat, Tommi Jaakkola, Brian Karrer, Ricky T. Q. Chen, and Yaron Lipman.
Generator matching: Generative modeling with arbitrary markov processes.
*arXiv preprint arXiv:2410.20587*, 2024.
- Hu et al. (2024)
Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu.
Ella: Equip diffusion models with llm for enhanced semantic alignment.
*arXiv preprint arXiv:2403.05135*, 2024.
- Ifriqi et al. (2025)
Tariq Berrada Ifriqi, Adriana Romero-Soriano, Michal Drozdzal, Jakob Verbeek, and Karteek Alahari.
Entropy rectifying guidance for diffusion and flow models, 2025.
[https://arxiv.org/abs/2504.13987](https://arxiv.org/abs/2504.13987).
- Kim et al. (2025)
Jaeyeon Kim, Lee Cheuk-Kit, Carles Domingo-Enrich, Yilun Du, Sham Kakade, Timothy Ngotiaoco, Sitan Chen, and Michael Albergo.
Any-order flexible length masked diffusion.
*arXiv preprint arXiv:2509.01025*, 2025.
- Laurençon et al. (2023)
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al.
Obelics: An open web-scale filtered dataset of interleaved image-text documents.
*Advances in Neural Information Processing Systems*, 36:71683–71702, 2023.
- Li et al. (2025)
Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang.
Dual diffusion for unified image generation and understanding.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pages 2779–2790, 2025.
- Liang et al. (2024)
Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, et al.
Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models.
*arXiv preprint arXiv:2411.04996*, 2024.
- Lin et al. (2014)
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.
Microsoft COCO: Common objects in context.
2014.
- Lipman et al. (2024)
Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky T. Q. Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat.
Flow matching guide and code.
*arXiv preprint arXiv:2412.06264*, 2024.
- Liu et al. (2022)
Xingchao Liu, Chengyue Gong, and Qiang Liu.
Flow straight and fast: Learning to generate and transfer data with rectified flow.
*arXiv preprint arXiv:2209.03003*, 2022.
- Lou et al. (2024)
Aaron Lou, Chenlin Meng, and Stefano Ermon.
Discrete diffusion modeling by estimating the ratios of the data distribution.
In *Proceedings of the 41st International Conference on Machine Learning*, pages 32819–32848, 2024.
- Ma et al. (2025a)
Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai Yu, et al.
Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pages 7739–7751, 2025a.
- Ma et al. (2025b)
Yiyang Ma, Xingchao Liu, Xiaokang Chen, Zhiyu Wu, Wen Liu, Zhenda Xie, Zizheng Pan, Xingkai Yu, Liang Zhao, Yuxuan Wang, Jie Liu, and Chong Ruan.
Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2025b.
- Nie et al. (2025)
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, JUN ZHOU, Yankai Lin, Ji-Rong Wen, and Chongxuan Li.
Large language diffusion models.
In *ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy*, 2025.
- Niu et al. (2025)
Yuwei Niu, Munan Ning, Mengren Zheng, Weiyang Jin, Bin Lin, Peng Jin, Jiaqi Liao, Chaoran Feng, Kunpeng Ning, Bin Zhu, et al.
Wise: A world knowledge-informed semantic evaluation for text-to-image generation.
*arXiv preprint arXiv:2503.07265*, 2025.
- Ou et al. (2024)
Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li.
Your absorbing discrete diffusion secretly models the conditional distributions of clean data.
*arXiv preprint arXiv:2406.03736*, 2024.
- Reid et al. (2022)
Machel Reid, Vincent J Hellendoorn, and Graham Neubig.
Diffuser: Discrete diffusion via edit-based reconstruction.
*arXiv preprint arXiv:2210.16886*, 2022.
- Sahoo et al. (2024)
Subham Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov.
Simple and effective masked diffusion language models.
*Advances in Neural Information Processing Systems*, 37:130136–130184, 2024.
- Sharma et al. (2018)
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.
Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.
In *Proceedings of ACL*, 2018.
- Shaul et al. (2024)
Neta Shaul, Itai Gat, Marton Havasi, Daniel Severo, Anuroop Sriram, Peter Holderrieth, Brian Karrer, Yaron Lipman, and Ricky T. Q. Chen.
Flow matching with general discrete paths: A kinetic-optimal perspective.
*arXiv preprint arXiv:2412.03487*, 2024.
- Shi et al. (2024a)
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias.
Simplified and generalized masked diffusion for discrete data.
*Advances in neural information processing systems*, 37:103131–103167, 2024a.
- Shi et al. (2025)
Qingyu Shi, Jinbin Bai, Zhuoran Zhao, Wenhao Chai, Kaidong Yu, Jianzong Wu, Shuangyong Song, Yunhai Tong, Xiangtai Li, Xuelong Li, et al.
Muddit: Liberating generation beyond text-to-image with a unified discrete diffusion model.
*arXiv preprint arXiv:2505.23606*, 2025.
- Shi et al. (2024b)
Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu.
Lmfusion: Adapting pretrained language models for multimodal generation.
*arXiv preprint arXiv:2412.15188*, 2024b.
- Sohl-Dickstein et al. (2015)
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.
Deep unsupervised learning using nonequilibrium thermodynamics.
In *International conference on machine learning*, pages 2256–2265. pmlr, 2015.
- Stern et al. (2019)
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit.
Insertion transformer: Flexible sequence generation via insertion operations.
In *International Conference on Machine Learning*, pages 5976–5985. PMLR, 2019.
- Sun et al. (2024a)
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan.
Autoregressive model beats diffusion: Llama for scalable image generation.
*arXiv preprint arXiv:2406.06525*, 2024a.
- Sun et al. (2023)
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.
Emu: Generative pretraining in multimodality.
*arXiv preprint arXiv:2307.05222*, 2023.
- Sun et al. (2024b)
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.
Generative multimodal models are in-context learners.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 12491–12501, 2024b.
- Swerdlow et al. (2025)
Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, and Katerina Fragkiadaki.
Unified multimodal discrete diffusion.
*arXiv preprint arXiv:2503.20853*, 2025.
- Team (2024)
Chameleon Team.
Chameleon: Mixed-modal early-fusion foundation models.
*arXiv preprint arXiv:2405.09818*, 2024.
- Thomee et al. (2016)
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.
Yfcc100m: The new data in multimedia research.
*Communications of the ACM*, 59(2):64–73, 2016.
- Tong et al. (2024a)
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al.
Cambrian-1: A fully open, vision-centric exploration of multimodal llms.
*Advances in Neural Information Processing Systems*, 37:87310–87356, 2024a.
- Tong et al. (2024b)
Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu.
Metamorph: Multimodal understanding and generation via instruction tuning.
*arXiv preprint arXiv:2412.14164*, 2024b.
- Tschannen et al. (2025)
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al.
Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.
*arXiv preprint arXiv:2502.14786*, 2025.
- Wang et al. (2025)
Jin Wang, Yao Lai, Aoxue Li, Shifeng Zhang, Jiacheng Sun, Ning Kang, Chengyue Wu, Zhenguo Li, and Ping Luo.
Fudoki: Discrete flow-based unified understanding and generation via kinetic-optimal velocities.
*arXiv preprint arXiv:2505.20147*, 2025.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models.
*Advances in neural information processing systems*, 35:24824–24837, 2022.
- Wu et al. (2024)
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Zhenda Xie, Chao Wu, and Chong Ruan.
Janus: Decoupling visual encoding for unified multimodal understanding and generation.
*arXiv preprint arXiv:2410.13848*, 2024.
- Wu et al. (2025a)
Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al.
Janus: Decoupling visual encoding for unified multimodal understanding and generation.
In *Proceedings of the Computer Vision and Pattern Recognition Conference*, pages 12966–12977, 2025a.
- Wu et al. (2025b)
Zirui Wu, Lin Zheng, Zhihui Xie, Jiacheng Ye, Jiahui Gao, Yansong Feng, Zhenguo Li, Victoria W., Guorui Zhou, and Lingpeng Kong.
Dreamon: Diffusion language models for code infilling beyond fixed-size canvas, 2025b.
[https://hkunlp.github.io/blog/2025/dreamon](https://hkunlp.github.io/blog/2025/dreamon).
- Xie et al. (2024)
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou.
Show-o: One single transformer to unify multimodal understanding and generation.
*arXiv preprint arXiv:2408.12528*, 2024.
- Yang et al. (2025)
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang.
Mmada: Multimodal large diffusion language models.
*arXiv preprint arXiv:2505.15809*, 2025.
- Ye et al. (2024)
Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu.
Physics of language models: Part 2.2, how to learn from mistakes on grade-school math problems.
*arXiv preprint arXiv:2408.16293*, 2024.
- Yu et al. (2024)
Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang.
Language model beats diffusion – tokenizer is key to visual generation, 2024.
[https://arxiv.org/abs/2310.05737](https://arxiv.org/abs/2310.05737).
- Yu et al. (2023)
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, Shang-Wen Li, Susan Zhang, Richard James, Gargi Ghosh, Yaniv Taigman, Maryam Fazel-Zarandi, Asli Celikyilmaz, Luke Zettlemoyer, and Armen Aghajanyan.
Scaling autoregressive multi-modal models: Pretraining and instruction tuning, 2023.
[https://arxiv.org/abs/2309.02591](https://arxiv.org/abs/2309.02591).
- Zhai et al. (2023)
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.
Sigmoid loss for language image pre-training.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pages 11975–11986, 2023.
- Zhao et al. (2024)
Chuyang Zhao, Yuxing Song, Wenhao Wang, Haocheng Feng, Errui Ding, Yifan Sun, Xinyan Xiao, and Jingdong Wang.
Monoformer: One transformer for both diffusion and autoregression.
*arXiv preprint arXiv:2409.16280*, 2024.
- Zheng et al. (2024)
Kaiwen Zheng, Yongxin Chen, Hanzi Mao, Ming-Yu Liu, Jun Zhu, and Qinsheng Zhang.
Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling.
*arXiv preprint arXiv:2409.02908*, 2024.
- Zhou et al. (2025)
Chunting Zhou, LILI YU, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy.
Transfusion: Predict the next token and diffuse images with one multi-modal model.
In *The Thirteenth International Conference on Learning Representations*, 2025.
[https://openreview.net/forum?id=SI2hI0frk6](https://openreview.net/forum?id=SI2hI0frk6).
- Zhu et al. (2023)
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi.
Multimodal c4: An open, billion-scale corpus of images interleaved with text.
*Advances in Neural Information Processing Systems*, 36:8958–8974, 2023.

## Appendix A Additional Generation Examples

Figure: A man standing in the middle of a nebula filled with stars.
Refer to caption: figures/generated_images/1.jpg

Figure: Figure 13: Example interleaved generation process showing outdoor exercise. In this example, the image is inserted late in the generation process when the text is almost fully denoised.
Refer to caption: figures/interleaved_figures/lake_walk/decoded_run_0_step_1200_t_0.8007_batch_0.png

Figure: Figure 14: Example interleaved generation process showing flowers.
Refer to caption: figures/interleaved_figures/flowers/decoded_run_11_step_600_t_0.4007_batch_0.png

Figure: Figure 15: Example interleaved generation process showing home types. An animated version is included in the supplementary material.
Refer to caption: figures/interleaved_figures/townhouses/decoded_run_1_step_300_t_0.2007_batch_0.png

Figure: Figure 16: Text generation examples from OneFlow, which allows the use of classifier-free guidance (CFG). We observe that CFG produces longer and more detailed captions and also increased chance of hallucinations. Highlighted text show increased levels of detail when using higher CFG weights.
Refer to caption: figures/images/000000252216.jpg

## Appendix B Additional Experiment Results

### B.1 VQA Average Performance Across Categories

Figure: Figure 17: VQA performance across categories in controlled experiments. The largest gaps appear in General and OCR/Chart tasks.The models are tied on Vision-Centric benchmarks, which primarily depend on the visual encoder rather than text generation.
Refer to caption: x16.png

### B.2 Performance Between AR and OneFlow During Pretraining

Figure: Figure 18: Training curve for OneFlow vs. AR for multimodal pretraining. OneFlow initially starts out lower than AR however it quickly catches up and exceeds AR, most notably on DPG and CIDEr.
Refer to caption: x17.png

### B.3 Pretraining from Scratch vs LLama Init

Table: Table 5: Ablation study comparing LLaMA initialization vs. random initialization. Except for CIDEr, using LLaMA as initialization generally offers benefits, especially for dense prompt image generation (DPG) and for VQA performance. Image generation metrics use CFG=3, and VQA results are averaged across benchmarks.

### B.4 Sampling Steps on Caption Performance

Figure: Figure 19: Performance vs. sampling steps compared to AR. OneFlow achieves parity with the AR model using only 6 sampling steps.
Refer to caption: x18.png

## Appendix C OneFlow Architecture

Figure: Figure 20: Architecture. With a multimodal prompt, OneFlow can produce variable length generations with interleaved text & images in a unified non-autoregressive sequence model, simultaneously generating all modalities with an interleaved time schedule for each generated image and text.
Refer to caption: x19.png

## Appendix D Full Derivations

We provide the derivations of the model here. We briefly summarize the Edit Flow (Havasi et al., 2025) formulation and derivation, and then derive the interleaved time schedule when insertions and image denoising are performed simultaneously.

##### Setup.

We make use of a blank token ${\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\varepsilon}$ to denote empty spaces within a sequence. This token is only used for tracking token deletions during training and is not part of the vocabulary. Let $\mathcal{Z}=\bigcup_{n=0}^{N}({\mathcal{T}}\cup\{{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\varepsilon}\})^{n}$ be an extended space of aligned sequences. Furthermore, define $f_{\text{rm-blanks}}:\mathcal{Z}\rightarrow\mathcal{X}$ as the function that removes all blank tokens from the sequence. Lastly, we define the delta function over sequences $\delta_{z_{1}}(z_{2})=\prod_{i}\delta_{z_{1}^{i}}(z_{2}^{i})$ which is one if all tokens are the same otherwise zero (i.e. Kronecker’s delta function).

##### Continuous-time Markov chain (CTMC).

A CTMC is a continuous-time discrete-space process which iteratively jumps between discrete values, with transitions

$$ $\mathbb{P}(X_{t+h}|X_{t})=\delta_{X_{t}}(X_{t+h})+hu_{t}(x|X_{t})+o(h),$ (13) $$

where $u_{t}$ can be interpreted as a first-order characterization of the transition kernel.
Since with insertions, the sequence lengths of $X_{t}$ can change over time. To simplify notation, Havasi et al. (2025) used an augmented space of $(X_{t},Z_{t})$, where it is basically always enforced that $X_{t}=f_{\text{rm-blanks}}(Z_{t})$. The role of $Z_{t}$ is only for training, to keep track of which tokens are deleted and to compute the loss, and it is neither seen by the model nor used during sampling.

To briefly summarize the construction below, the Flow Matching recipe makes use of a prescribed conditional CTMC that generates single data sequences, which is then marginalized over the data distribution. The resulting marginal CTMC will then sample from the data distribution.

##### Conditional probability path.

Given a data sequence $X_{1}\sim p_{\text{data}}$, we prescribe a conditional probability path over $Z_{t}$ of the same sequence length which interpolates between the empty sequence and this data sequence. We then obtain $X_{t}$ by applying the $f_{\text{rm-blanks}}$ function. Concretely, we can express the conditional probability path as

$$ $\displaystyle p_{t}(X_{t},Z_{t}|X_{1})$ $\displaystyle=p_{t}(X_{t}|Z_{t},X_{1})\cdot p_{t}(Z_{t}|X_{1})$ (14) $\displaystyle=p_{t}(X_{t}|Z_{t})\cdot p_{t}(Z_{t}|Z_{1})$ (15) $\displaystyle=\delta_{f_{\text{rm-blanks}}(Z_{t})}(X_{t})\cdot\left(\prod_{i=1}^{n}(1-\kappa_{t})\delta_{{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\varepsilon}}(Z_{t}^{i})+\kappa_{t}\delta_{X_{1}^{i}}(Z_{t}^{i})\right),$ (16) $$

where $\kappa_{t}$ is a scheduler where $\kappa_{0}=0,\kappa_{1}=1$, and $n$ is the sequence length of $X_{1}$. In English, ([16](https://arxiv.org/html/2510.03506v2#A4.E16)) is a mixture distribution where each token $Z_{t}^{i}$ can either be equal to ${\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\varepsilon}$ with probability $1-\kappa_{t}$ or equal to data value $X_{1}^{i}$ with probability $\kappa_{t}$.

##### Conditional CTMC rate.

As discussed in Havasi et al. (2025), a conditional CTMC that samples from this conditional probability path can be constructed as

$$ $\begin{split}&u_{t}(x,z|X_{t},Z_{t},X_{1})=\left(\sum_{i=1}^{n}\frac{\dot{\kappa}_{t}}{1-\kappa_{t}}(\delta_{X_{1}^{i}}(z^{i})-\delta_{Z_{t}^{i}}(z^{i}))\right)\delta_{f_{\text{rm-blanks}}(z)}(x),\\ &\qquad\qquad\text{ where }x={{\text{ins}}}(X_{t},i,a)\text{ for some $i\in[n]$ and $a\in[M]$}\end{split}$ (17) $$

which denotes the infinitesimal change in probability of going from the state $(X_{t},Z_{t})\rightarrow(x,z)$, constrained to next sequences $x$ that are one token insertion difference from $X_{t}$. In English, ([17](https://arxiv.org/html/2510.03506v2#A4.E17)) assigns a rate of $\frac{\dot{\kappa}_{t}}{1-\kappa_{t}}$ if $Z_{t}^{i}$ is not yet equal to $X_{1}^{i}$; otherwise, it is zero. This ensures that a sample starting with all blanks $Z_{0}=[{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\varepsilon},...{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\varepsilon}]$ at $t=0$ will eventually turn into $X_{1}$ at $t=1$.
This ratio $\frac{\dot{\kappa}_{t}}{1-\kappa_{t}}$ is the infinitesimal rate that each token changes its value, matching the distribution imposed by the scheduler $\kappa_{t}$, and conditioned on that it is still the ${\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\varepsilon}$ token at time $t$.

Figure: Figure 21: During training we construct $Z_{t}$ by replacing tokens with the blank token (${\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\varepsilon}$), with the original tokens used to construct the target bag-of-tokens $\mathcal{A}_{i}$.
Refer to caption: x20.png

##### Training loss.

In order to train a model thats transport sequences via insertions,

$$ $u_{t}^{\theta}(x|X_{t}),\quad\text{ where }x={{\text{ins}}}(X_{t},i,a)\text{ for some $i$ and $a$}$ (18) $$

we would need to marginalize out the auxiliary process $Z_{t}$ and the data $X_{1}$. Havasi et al. (2025) showed this can be done by using a loss based on any Bregman divergence—following the recipe of Holderrieth et al. (2024)—while summing up over all possible sequences $z$ such that $x=f_{\text{rm-blanks}}(z)$. Concretely, given a convex function $\phi$ that defines a Bregman divergence $D_{\phi}(a,b)=\phi(a)-\phi(b)-\langle a-b,\tfrac{\mathrm{d}}{\mathrm{d}b}\phi(b)\rangle$, we can use the loss

$$ $\mathbb{E}_{X_{t},Z_{t}\sim p_{t}(X_{t},Z_{t}|X_{1}),X_{1}\sim p_{\text{data}}}D_{\phi}\Big(\textstyle\sum_{z}u_{t}(\cdot,z|X_{t},Z_{t},X_{1}),u_{t}^{\theta}(\cdot|X_{t})\Big).$ (19) $$

Plugging in the entropy $\phi(u)=\langle u,\log u\rangle$, this results in the Edit Flow loss

$$ $\mathbb{E}_{t,p_{t}(X_{t},Z_{t}|X_{1}),X_{1}\sim p_{\text{data}}}\left[\sum_{x\neq X_{t}}u_{t}^{\theta}(x|X_{t})-\sum_{i=1}^{n}\bm{1}_{[Z_{t}^{i}={\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\varepsilon}]}\frac{\dot{\kappa}_{t}}{1-\kappa_{t}}\log u_{t}^{\theta}({{\text{ins}}}(X_{t},j,X_{1}^{i})|X_{t})\right],$ (20) $$

where $j$ is the position in $X_{t}$ that corresponds to the first non-${\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\varepsilon}$ token on the left of $Z_{t}^{i}$. This ensures that inserting at the $i$-th position corresponds to changing the value of $Z_{t}^{i}$ from ${\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\varepsilon}$ to $X_{1}^{i}$.

##### Loss simplification.

We deviate from Havasi et al. (2025) and use a $t$-independent parameterization. In particular, for sequences $x$ that are one token insertion of $X_{t}$, i.e., $x={{\text{ins}}}(X_{t},i,a)$, we use

$$ $u_{t}^{\theta}\big({{\text{ins}}}(X_{t},i,a)|X_{t}\big)=\frac{\dot{\kappa}_{t}}{1-\kappa_{t}}\lambda^{i}(X_{t})Q^{i}(a|X_{t}),$ (21) $$

where the neural network parameterizes $\lambda$ and $Q$.
Using this parameterization, letting $\mathcal{A}_{j}$ be the set of missing tokens to the right of position $j$ of $X_{t}$,
the training loss ([20](https://arxiv.org/html/2510.03506v2#A4.E20)) can be decomposed into

$$ $\displaystyle\mathbb{E}_{(\dots)}\left(\frac{\dot{\kappa}_{t}}{1-\kappa_{t}}\right)\Bigg(\sum_{j=1}^{n(X_{t})}\lambda^{j}(X_{t})-\sum_{j=1}^{n(X_{t})}\sum_{a\in\mathcal{A}_{j}}\log\left(\lambda^{j}(X_{t})Q^{j}(a|X_{t})\right)\Bigg)$ (22) $\displaystyle=$ $\displaystyle\mathbb{E}_{(\dots)}\left(\frac{\dot{\kappa}_{t}}{1-\kappa_{t}}\right)\sum_{j=1}^{n(X_{t})}\Bigg(\underbrace{\vphantom{\sum_{a\in\mathcal{A}_{j}}}\lambda^{j}(X_{t})-|\mathcal{A}_{i}|\log\lambda^{j}(X_{t})}_{(\ref{eq:poisson_nll})}+\underbrace{\sum_{a\in\mathcal{A}_{j}}\log Q^{j}(a|X_{t})}_{(\ref{eq:token_loss})}\Bigg)+\text{const.}$ (23) $$

which recovers the losses for $\lambda$ and $Q$ in ([4](https://arxiv.org/html/2510.03506v2#S2.E4)) and ([6](https://arxiv.org/html/2510.03506v2#S2.E6)) respectively, after removing the coefficient $\frac{\dot{\kappa}_{t}}{1-\kappa_{t}}$. While keeping this coefficient relates the loss the to an evidence lower bound (Havasi et al., 2025), we found that removing this coefficient in the loss gave better results in practice.

### D.1 Interleaved time schedule

In order to model image insertions, we would make a choice. We can either (i) fully denoise images at the time of insertion, or (ii) insert only noise and denoise later. We choose the latter approach, as this allows simultaneous generation across images and text, and provides the best parallelism as only a single model forward at each step is needed for both modalities.

Without loss of generality, assume there is at most just one image.
Generation starts by advancing the sequence time, denoted $t_{\text{text}}=0$. When the image is inserted, we associate the image with its own time $t_{\text{img}}$.

The main difficulty is that we can not simply set $t_{\text{img}}$ and $t_{\text{text}}$ independently during training, as evidently we always have $t_{\text{text}}\geq t_{\text{img}}$. In fact, an independent scheduler induces the wrong distribution for our insertion prediction, and it will not insert the correct distribution at generation time. Instead, we need to ensure that training and generation see the same distribution of time values. To achieve this, we first note that the image exists in the sequence according to the scheduler $\kappa_{t}$, which means that the *insertion times* are distributed according to

$$ $p(t_{\text{insert}})=\dot{\kappa}_{t},$ (24) $$

where $t_{\text{insert}}$ is the time at which an image is inserted, i.e., $\kappa_{t}$ is the cumulative distribution function (CDF) of the insertion times. Equivalently, to sample the insertion time, we can apply the inverse CDF sampling,

$$ $t_{\text{insert}}=\kappa^{-1}(u),\qquad u\sim\text{Unif}(0,1).$ (25) $$

If we set $t_{\text{img}}=0$ when an image is inserted, then the difference between $t_{\text{text}}$ and $t_{\text{img}}$ is distributed according to the insertion time. This gives us the relation

$$ $t_{\text{text}}-t_{\text{img}}=t_{\text{insert}}$ (26) $$

when $0\leq t_{\text{text}},t_{\text{img}},t_{\text{insert}}\leq 1$. Since $t_{\text{text}}$ will reach 1 before $t_{\text{img}}$, and we want to train for the entire process until $t_{\text{img}}=1$, we can construct an extended time interval

$$ $\tau_{\text{text}}\in[0,2],\qquad t_{\text{text}}=\text{clip}(\tau_{\text{text}}),$ (27) $$

where $\text{clip}(\tau)=\min\{1,\max\{0,\tau\}\}$ clips the time values back into the interval $[0,1]$.

During training, we first sample $\tau_{\text{text}}$, then sample

$$ $\tau_{\text{img}}=\tau_{\text{text}}-\kappa^{-1}(u),\qquad u\sim\text{Unif}(0,1).$ (28) $$

This will sample an extended time for the image in the internal [-1, 2]. If $\tau_{\text{img}}<0$, then it has not yet been inserted, hence it is deleted from the sequence. Otherwise, it is clipped,

$$ $t_{\text{img}}=\text{clip}(\tau_{\text{img}}),$ (29) $$

and we proceed to use the Flow Matching loss ([9](https://arxiv.org/html/2510.03506v2#S2.E9)) to train the image denoising.

## Appendix E Sampling and Training Algorithms

Figure: Algorithm 1 OneFlow interleaved text–image generation.

Figure: Algorithm 2 OneFlow step function. $X$ is the token sequence, $\mathcal{I}$ is the set of image latents each with time $t_{\text{img}}(Y)$.

Figure: Algorithm 3 OneFlow training loss with interleaved schedule

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: - Can you elaborate more on the different candidates for κt (line 112) and provide more information?
- The decision to use a t-independent model is noted to work better in practice despite the theoretical justification for t-dependence. Could you provide an ablation quantifying the performance gap between these two parameterizations? Were there specific tasks where the t-dependent model performed better?
- As shown in Figure 11 and discussed in Sections 3.5 and 4, higher CFG weights consistently lead to longer, more detailed captions but also to an increased chance of hallucinations. Can you discuss this trade-off in more depth?
- The emergent reasoning chains (line 367) are a fascinating finding. Can you elaborate on the conditions or training data that you believe led to this behavior? Is it a general property of the Edit Flow text generation, or is it specific to the multimodal pretraining mixture used?
soundness: 4
strengths: - The proposed OneFlow model requires fewer training FLOPs than compared to AR models, thanks to its non-autoregressive, insertion-based training which only predicts missing tokens.
- The paper provides extensive experiments showing that OneFlow is competitive with or superior to state-of-the-art AR and diffusion models across a diverse range of benchmarks for both image generation and understanding.
- The paper convincingly demonstrates new emergent capabilities, such as implicit reasoning chains without Chain-of-Thought prompting, the application of classifier-free guidance to improve text detail, and the dynamic insertion and denoising of images within a text sequence.
summary: This paper introduces OneFlow, a novel non-autoregressive multimodal model that unifies text and image generation within a single framework and overcomes key limitations of existing autoregressive and diffusion models by combining Edit Flows for variable-length discrete text generation and Flow Matching for continuous image generation. This allows for the concurrent and interleaved generation of variable-length text and a variable number of images, enabling capabilities like simultaneous cross-modal refinement.
weaknesses: - While the integration of Edit Flows with Flow Matching is innovative and effective, the core architectural contribution is the combination of these existing techniques for a new multimodal task. The paper's impressive results are therefore heavily reliant on the pre-established Edit Flows framework, and the fundamental methodological novelties introduced beyond this combination are more incremental.
- As stated by the authors, the bidirectional attention required for non-autoregressive generation prevents the use of key-value caching, making OneFlow's inference slower and more memory-intensive than cached AR sampling, despite requiring fewer steps.
- The 20% mixed-generation probability is used without justification. The performance sensitivity to this key hyperparameter is also unknown, making it hard to determine the optimal data mixture.
- Certain details about hyperparameters are omitted and could use ablations, see Q1 (Ablation on Scheduler) and Q2 (t-Independent Parameterization).

Formatting Concerns:
- Table captions must precede the tables.
- Although clear from the context, the abbreviation VQA is not explicitly clarified.

---

confidence: 2
contribution: 4
initial_rating: 6
presentation: 3
questions: - Will the OneFlow model weights and modeling files be open-sourced? If subsequent researchers cannot build upon this work, it would be detrimental to the unified model community.
- It appears the authors used closed-source fine-tuning data (Line 267). This could potentially impact the reproducibility of the method.
soundness: 4
strengths: - This is a very solid work from both a technical and an engineering perspective. It skillfully combines the recent, popular diffusion text generation techniques with the consistently effective diffusion image generation methods within a single Transformer. I believe this is a strong contribution to both the DLMs and the unified model communities.

- The paper presents comprehensive and rigorous experiments. This includes scaling experiments from 1B to 8B parameters, extensive comparisons against baseline models (for both multimodal understanding and generation), and supplementary ablation studies in Appendix F. This significantly increases the credibility of the paper.
summary: This paper proposes OneFlow, the first non-autoregressive multimodal model that enables concurrent and variable-length text-image generation by combining an insertion-based "Edit Flow" for text with "Flow Matching" for images, which outperforms autoregressive baselines with greater efficiency.
weaknesses: This paper frequently mentions two terms: "interleaved mixed-modal generation" and "concurrent mixed-modal generation." My concerns are primarily centered on these two points.

- Regarding Interleaved Generation: In the introduction, the authors highlight OneFlow's ability to perform interleaved generation with a variable number of outputs as a major contribution. However, in the experiments, the paper only evaluates OneFlow's performance on multimodal understanding and image generation. It notably lacks evaluations on interleaved generation benchmarks, such as OpenING. I believe this experimental omission fails to substantiate the claims made in the introduction.

- Regarding Concurrent Generation: Similarly, the introduction categorizes concurrent mixed-modal generation as a novel capability for unified models. However, after reading the paper, I still do not understand why we need concurrent generation. Is it simply because it leads to better performance (as suggested in Figure 3)? Is there a deeper insight or analysis behind this capability that I missed?

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. For the VQA task, you omit comparisons to recent SOTA models like Show-O2, Mogao, and BAGEL. Do you have preliminary results comparing OneFlow to these models on key VQA benchmarks (e.g., VQAv2, GQA, DocVQA)? 

2. You note that OneFlow’s lack of KV caching increases latency. Have you explored lightweight optimizations (e.g., semi-autoregressive block decoding, sparse bidirectional attention, or model distillation) to reduce inference cost?

3. You finetune on 512×512 images, but many SOTA models (e.g., SD3) support higher resolutions (1024×1024). Has OneFlow been tested on higher-resolution image generation, and if so, how does performance (e.g., FID, detail) scale with resolution?
soundness: 3
strengths: 1. OneFlow fills a critical gap in multimodal generation by pioneering an NAR framework for concurrent, variable-length interleaved text-image generation. The integration of Edit Flow and Flow Matching into a unified transformer backbone is non-trivial: it avoids modality-specific silos and enables cross-modal dependency modeling during generation.

2. The model’s training efficiency is a standout advantage: by using a linear deletion scheduler that retains only 50% of tokens during training, OneFlow reduces training FLOPs by up to 50% compared to AR baselines.

3. OneFlow enables practical, previously unachievable use cases: Unlike AR models that append images to the end of text, OneFlow inserts images dynamically within text (via 
 tokens) and refines them simultaneously; Without CoT prompting or RL post-training, OneFlow generates reasoning chains for visual questions (e.g., object counting, visual search), demonstrating that NAR architectures can support complex reasoning.
summary: This paper presents OneFlow, the first non-autoregressive (NAR) multimodal model designed for variable-length and concurrent mixed-modal (text-image) generation. It addresses key limitations of existing autoregressive (AR) and diffusion-based multimodal models: AR enforces rigid sequential generation (preventing cross-modal refinement), while diffusion models only support fixed-length, pre-specified text-image pairs.
weaknesses: 1. High Inference Cost: OneFlow lacks key-value caching (due to bidirectional attention), leading to higher inference latency and memory usage than AR models. This limits its applicability to low-latency scenarios (e.g., real-time multimodal chatbots). No preliminary optimizations (e.g., semi-autoregressive decoding, sparse attention) are proposed to mitigate this.

2. Incomplete Comparisons to SOTA Models: While the paper compares OneFlow to VQA baselines like Show-O (1.3B), Janus-Pro (1.5B/7B), and MMaDA (8B), it omits critical recent SOTA models that have set new benchmarks in visual question answering. Specifically, there is no comparison to Show-O2, Mogao, or BAGEL; For image generation (Table 1), the paper evaluates OneFlow against AR models (e.g., Transfusion, Janus-Flow) and diffusion models (e.g., MMaDA, FUDOKI) but lacks comparisons to some SOTA works: Show-O and DreamLLM.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: 1- It would be insightful to further evaluate the model’s ability to handle compositional generation and compositional visual question answering, and to benchmark its performance against comparable multimodal baselines in these challenging settings. (e.g. attribute binding, missing entities etc.)

2- What modifications would be needed to extend OneFlow to other modalities like audio or video, given its reliance on Edit Flows for discrete elements and Flow Matching for continuous ones?

3- Beyond the linear κ_t scheduler, what alternatives were tested, and why did linear perform best? Could adaptive schedules further reduce FLOPs while maintaining performance?

4- The paper shows that CFG for text increases detail but also the chance of hallucinations (Figure 11). A more systematic analysis of this trade-off (e.g., a plot of detail vs. hallucination rate across CFG scales) would be beneficial.

5- Please include qualitative examples of failed generations (misplaced images, inconsistent text-image references, incoherent insertions).
soundness: 3
strengths: Novel Integration: The work integrates Edit Flows (insertion operations) with continuous Flow Matching for image latents in a single backbone so both modalities are predicted by the same model and can be denoised jointly.

Rigorous Evaluation: The paper conducts rigorous experiments using a diverse set of benchmarks, metrics, and baselines to thoroughly evaluate the effectiveness of its proposed method. Also the appendix is really rich.

Emergent Reasoning: The observation that OneFlow develops implicit reasoning chains (Figure 5, 19) is a fascinating finding from my point of view.

Writing: The paper is well-written, and the methodology is explained with remarkable clarity and it was so easy for me to follow.
summary: OneFlow is a non-autoregressive multimodal generation framework that jointly handles variable-length text and an arbitrary number of images by combining (i) an insertion-based Edit Flows procedure for discrete tokens and (ii) Flow Matching for image latents. A novel interleaved time schedule couples per-image generation times to the text insertion process so images can be inserted and denoised concurrently with text rather than waiting until an image is completed.
weaknesses: Ablation clarity for interleaved schedule: The interleaved schedule is central. I would like to see ablations for it. The paper describes κ(t)=t (linear) and claims it works well, but sensitivity analysis is missing.

Potential mode of failure for complex interleavings: Examples show 2 images interleaved with text. It is unclear how the model behaves when the number of inserted images is large, or when images must be heavily conditioned on earlier generated text.
Limited operation diversity in Edit Flows: The model only supports insertion operations, omitting deletion and substitution — both fundamental to edit-based generative modeling. (they are even present in the Original Edit Flow Paper)


Decision: Reject