Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents
Authors: Han Lin
Jaemin Cho
Amir Zadeh
Chuan Li
Mohit Bansal
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal LLM
Unified Multimodal LLM
Diffusion Models
Autoregressive Priors
Image Generation
Primary area: deep_learning

Abstract: There is growing interest in integrating high-fidelity visual synthesis capabilities into large language models (LLMs) without compromising their strong reasoning capabilities. Existing methods that directly train LLMs or bridge LLMs and diffusion models usually suffer from costly training since the backbone LLMs have not seen image representations during pretraining. We present Bifrost-1, a unified framework that bridges pretrained multimodal LLMs (MLLMs) and diffusion models using patch-level CLIP image embeddings as latent variables, which are natively aligned with the MLLM's CLIP visual encoder. These patch-level image embeddings are integrated into the diffusion model with a lightweight adaptation of its ControlNet. To retain the original multimodal reasoning capabilities of MLLMs, we equip the MLLM with a visual generation branch initialized from the original MLLM parameters when predicting the patch-level image embeddings. By seamlessly integrating pretrained MLLMs and diffusion models with patch-level CLIP latents, our framework enables high-fidelity controllable image generation with significant training efficiency. Our experiments demonstrate that Bifrost-1 achieves comparable or better performance than previous methods in terms of visual fidelity and multimodal understanding, with substantially lower compute during training. We also provide comprehensive ablation studies showing the effectiveness of our design choices. Project page: https://bifrost-1.github.io.

Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents
Han Lin'
Jaemin Cho'
Amir Zadeh²
Chuan Li
Mohit Bansal'
UNC Chapel Hill
Lambda
Abstract
There is growing interest in integrating high-fidelity visual synthesis capabilities into large language models (LLMs) without compromising their strong reasoning capabilities. Existing methods that directly train LLMs or bridge LLMs and diffusion models usually suffer from costly training since the backbone LLMs have not seen image representations during pretraining. We present BIFROST- 1, unified framework that bridges pretrained multimodal LLMs (MLLMs) and diffusion models using patch-level CLIP image embeddings as latent variables, which are natively aligned with the MI.LM's CL.IP visual encoder. These patch-level image embeddings integrated into the diffusion model with lightweight adaptation of its ControlNet. To retain the original multimodal reasoning capabilities MLLMs, equip the MLLM with a visual generation branch initialized from the original MLLM parameters when predicting the patch-level image embeddings. By seamlessly integrating pretrained MLLMs and diffusion models with patch-level CLIP latents, our framework enables high-fidelity controllable image generation with significant training efficiency. Our experiments demonstrate that BIFROST- I achieves comparable or better performance than previous methods in terms of visual fidelity and multimodal understanding, with substantially lower compute during training. We also provide comprehensive ablation studies showing the effectiveness of our design choices. Project page: https:/ //tpprs///ffott-t/e 1. wiiggittubb. 1o.
Introduction
Humans can naturally process and generate multimodal information such as language, vision, and sound. This ability stems from our integrated cognitive system capable of reasoning, organization, and expression. Inspired by this, building a unified AI system that supports both multimodal understanding and generation has become an active area of research [48, 10, 80, 62, 73, 68]. On the one hand, large language models (LLMs) have achieved impressive results across diverse tasks, including natural language understanding [17, 40, 6, 54], mathematical reasoning [33, 58), and code generation [9, 46), demonstrating strong planning and reasoning abilities [49. On the other hand, recent advances in image and video generation models have enabled high-fidelity visual synthesis [50, 74, 19, 47]. Therefore, effectively and efficiently integrating visual generation capabilities into LLMs, while preserving their reasoning capabilities, has become an active research direction in the pursuit of unified Al systems that bridges language understanding and visual synthesis.
Existing LLM-based image generation methods fall into two main paradigms. First, single- architecture approaches (see Fig. (a)) employ an LLM/MLLM that takes concatenated image and text tokens as inputs. The model is then finetuned, either by updating all LLM parame- ters [12, 67, 42, 41, 62, 71, 10, 68] or via an auxiliary visual generation branch [44, 37, 59], to generate images directly. While this offers unified reasoning and native image synthesis, it typically incurs very expensive computational costs to achieve high-fidelity results. Moreover, without careful
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Image
Image
Image
Cotett-ControiNet A
(VOYVAE
Diffusion
Text
Text
Text
Patch-Level CLIP Latents
A/@
LLM MLLM
LLM / MMLM
MLLM
Vision Gen. Branch
Text
Image
Text
Image
Text
Image
(a) Single Architecture
(b) Bridge w/ 1D Image/Text Tokens
(c) Bridge Patch-Level CLIP Latents
(Chamsleon,
Transfuslon, JanusPro, EMU3, Shaw-o, LMFusion, X-fusion)
(SEED-X, GILL, DreamLLM, MetaMorph, MetaQuery, Lay VPGen)
(Ours: Bifrost-1)
Text Takens
Image Tokens
Patch-Leval CLIP Latents Aligned with MILLM
Figure 1: Comparison of different approaches in using LLM for image generation. (a) Single architecture handling both text and image tokens into the LLM/MLLM. (b) Bridging LLM and Diffusion model with ID sequence (image tokens text tokens). (c) BIFROST-1 I (ours), which bridges MLLM with diffusion models with 2D image tokens aligned with MLLM embeddings.
balancing of training data and objectives, such finetuning can risk degrading the LLM's original language reasoning capabilitiess [71], thereby undermining a primary goal of the integration.
Second, bridging- baaded approaches (see Fig. 1 (b)) use both pretrained diffusion models as well as LLMs, teaching an LLM to guide a diffusion model in generating the final image. The guidance takes the form of detailed text descriptions (e.g., object layouts, extended prompts) generated by the LLM [13, 20, 14], or continuous query tokens also produced by the LLM [31, 23, 63, 51, 1]. Although generally more rraining-efficient than engle-architecture methods, these approaches have their own limitations. Text-based representations can be too sparse to capture fine-grained details of complex scenes, while learning generate effective ID continuous query tokens often requires large connector and end-to-end training, and may not sufficiently convey detailed 2D spatial information.
To address these limitations, introduce BIFROST-1 1 (Sec. 3). As illustrated in Fig. 1(c). BIFROST- 1 uniquely bridges a pretrained multimodal LLM (MLLM) [5] and a diffusion model [32]. Our core innovation lies in using 2D patch-level CLIP [53] latents as the communicative medium. These latents are image emheddings that are natively aligned with the MLLM's own CLIP visual encoder, enabling the MILM generate rich and precise spatial guidance for image synthesis. These patch-level latents are integrated into the diffusion model via a lightweight adaptation of its ControlNet [78]. To retain the original multimodal reasoning capabilities of MLLMs, we equip the MLLMs with a visual generation branch initialized from the original MLLM parameters when predicting the patch-level image embeddings. By seamlessly integrating pretrained MILMS and diffusion models with patch-level CLIP latents, our framework enables high-fidelity controllable image generation with substantial training efficiency. In addition, by fully retaining the original parameters of the MLLM backbone, BIFROST-1 does not suffer from performance degradation of multimodal reasoning capabilities observed some previous works.
We empirically demonstrate the effectiveness of BIFROST- through a series of experiments. First, in Sec. 5.1, we compare different architectures for bridging LLMs and diffusion models, showing that patch-level CLIP latents train more efficiently than the controlled variants with alternative methods. Next, in Sec. 5.2, present the effective scaling of the number of CLIP latent tokens to improve image reconstruction quality. We also qualitatively demonstrate that the image reconstruction quality of BIFROST-I is competitive or superior strong baselines with magnitude higher training computation cost. In Sec. 5.3, we compare BIFROST-1 with state-of-thh of--the-rt art methods on multimodal understanding and generation tasks, showcasing its competitive performance and the preservation of the MLLM backbone's reasoning capabilities. Finally, in Sec. 5.4, we experiment with varying number of MILMM decoding steps, and observe that our method robust as long as the number of decoding steps is larger than 8. In nutshell, our contribution can be summarized as follows:
We propose a novel framework (BIFROST-1) 1) for unified multimodal generation and understanding by bridging pretrained MLLM and diffusion models with fine- -me-ierieed patch-level CLIP latents.

---

By using CLIP latents that are natively aligned with the MLLM, we achieve significantly lower training costs compared to recent works with a single architecture or those that bridge using ID latents or query tokens.
We show qualitatively and quantitatively that BIFROST-1 achieves comparable performance with previous SoTAS on image reconstruction quality and text-to-image generation benchmarks, while well-preserving MLLM's original multimodal understanding ability.
2
Related Work
Unified multimodal LLM architectures for visual generation. The success of large language models (LLMs) such as TS [54] and GPT-3 [6] has driven efforts toward extending language modeling paradigms to multimodal settings. Early works like VL-T5 [11], Sim VLM [69], and Flamingo [3] adapt pretrained LLMs for visual understanding by framing perception tasks as text generation. More recent models also incorporate image generation capabilities into LLMs using either autoregressive objectives [12, 67, 42, 41, 62, 71, 10, 68, 59, 44] or denoising diffusion objective [73, 80], as described Fig. 1 (a). While these approaches enable flexible, end-to-end training, they often inherit their initialization from text-only LMs and require significant resources learn high-quality image generation from scratch. This limitation has motivated a complementary line of research: bridging pretrained LLMs with diffusion models, as discussed next below.
Bridging LLM and diffusion models for visual generation. As described in Fig. 1 (b), several methods decompose the generation process by using LLMs to produce detailed intermediate textual representations, such as object layouts or detailed scene descriptions, which guide a pretrained diffusion model [13, 20, 14]. However, complex visual scenes with rich object interactions often hard to describe purely with text. An alternative strategy is to replace intermediate text with continuous visual representations guide diffusion model [31, 23, 63, 51]. For example, Tong al. [63] starts with an LLM and conduct lcage-scale training to equip it with a SigLIP [77] visual encoder. The image tokens generated autoregressively by the LLM are supervised using cosine similarity loss with ground-truth SigLIP embeddings. These predicted image tokens are then injected into diffusion models via cross-attention. However, teaching LLMs generate visual representations still requires more text-image annotations and computational resources (see Table 1), and is also prone degraded eext-generation quality unless the model carefully balanced and trained on high-qualit text-text and vision-text data, along with extensive compute. Hence, we explore the use of multimodal LLMs (MLLMs) that are already strongly aligned with CLIP embeddings encoding spatial information, by teaching them to generate 2D CLIP embeddings to guide diffusion models through latent ControlNet. As shown Sec. 5.1, we find our method of bridging MILM and Diffusion models with 2D CLIP embeddings and latent ControlNet is significantly more data-efficient in broad training budget scenarios compared to the above baselines.
BIFROST-1
We introduce BIFROST-1, a novel unified multimodal framework that effectively bridges pretrained MLLMs and diffusion models with patch-level CLIP latents which are natively aligned with the MLLM. This enables MLLMs to generate rich and precise spatial guidance for image synthesis while achieving significant training efficiency. In the following, and ControlNct (Sec, 3.1,, how BIFROST-1 I bridges MLLMs and diffusion models via patch-level CLIP latents (Sec. 3.2), and training and inference strategy (Sec. 3.3).
present preliminaries about MLLMs
3.1 Preliminaries
Visual encoding and decoding with MLLMs. For image understanding tasks, given an RGB image 3x recent MILMS commonly employ visual encoders that convert the image into set of d-dimensional continuous (e.g., CLIP [53]) or discrete (c.g., VQVAE [64]) visual embeddings:
{xx) E 1 xW" where H' < H and W < W due to spatial downsampling [5, 81, 2, 15]. These visual embeddings are aligned with the LLMs' embedding space via learned projection layers. For image generation tasks, to enable the MILMM to produce images, the visual embeddings generated by the L.M backbone are passed to visual generation decoder D to render the final image:
D(z). Typically, VQVAEs diffusion models are employed for the decoder, whose visual
3

---

Downsample 20 Reshape
Text Head
Vision Head
Latent ControlNet
<BOP
Patch-Level CLIP Latents
Vision Branch
Pretrained MLLM +
Diffusion Model
HELLO WORLD
HB0T>
+
MLLM w/ auxiiaary vision generation branch
Text Tokenizer
Vision Encoder
MILMM MLP
Vision Gen. MLP
Cute kitten holding "Hello World" sign.
Joint Alin
HELLO WORLD
MILLM QKCV
Vision Gen, OKV
Figure 2: Overview of BIFROST- BIFROST-1 equips the backbone MILMM with a visual generation branch, which a trainable copy of pretrained MLLM parameters (i.e., QKV, MLP. normalization layers) and newly added vision head (i.e., linear layer). The visual generation branch outputs patch-level CLIP latents, which are then downsampled and reshaped into 2D (HxW), provided to latent ControlNet, and finally guiding image generation of pretrained diffusion model. During training. portion of the image patches randomly replaced with learnable mask tokens <M>, During inference, we start with fully masked image tokens and autoregressively predict them.
representation is different from CLIP embeddings; thus, learning to align MLLMs and decoders usually requires significant computation. this work, we extend unCLIP [55] that employs a model to convert single (pooled) CLIP text embedding to single (pooled) CLIP image embedding by teaching MILM to generate patch-level CLIP latents which are natively aligned with the visual encoder of the pretrained MLLM. We find this significantly facilitates image generation training.
ControlNets. ControlNet [79] is designed add spatial controls with image guidance inputs (e.g.. depth, sketch, segmentation maps, etc.) to diffusion models. Specifically, given a pretrained diffusion model Fe. ControlNet adopts similar architecture Fe', whose parameters are initialized from (i.e., 'trainable copy'). ControlNet takes the diffusion timestep i, text prompt Ciext, control image cr (e.g., depth map), and the noisy latents 2t as inputs, and outputs features guide the backbone diffusion Fo for the final image generation. In this work, we propose latent ControlNet that uses patch-level CLIP image embeddings instead control image to guide diffusion models.
3.2 Bridging MLLMs and Diffusion Models with Patch-Level CLIP Latents
BIFROST-1 design summary. We have two main goals: (I) to preserve the multimodal understanding capability of the MILM, while (2) efficiently teaching MLLM to generate latent tokens to guide diffusion models. As illustrated in Fig. 2, BIFROST-I I achieves these goals by having a new visual generation branch initialized from the original MLLM parameters and using patch-level CLIP image embeddings as latent bridge tokens.
Learning to unmask image patch embeddings with visual generation branch. To teach MILMM image generation, we first encode the images using the ML.M's native visual encoder to get patch-level image embeddings and concatenate them with tokens. Following MAR [36], we replace parts of the input image embeddings with a learnable mask token (<M>) and let the MLLM to predict the masked image embeddings. The mask ratio is randomly sampled from truncated normal distribution with a mean of 1.0 and a standard deviation of 0.25, constrained to the range [0.7, 1.0]. For this image embedding prediction task, we introduce visual generation branch (Fig. left), whose parameters are initialized from MLLM parameters (i.e., attention QKV projections, MLP projection layers, and normalization layers) following LMFusion [59]. Just like text head, which is a linear layer toward text embedding space, we use simple linear layer as a vision generation head. By reusing majority of parameters from the pretrained MILMM and randomly initializing only single linear layer as the vision head, we avoid the costly process of realigning image embeddings.

---

Attention across modalities. In Fig. 3, we illustrate how different image and text input tokens can attend each other. In summary, we apply causal masking for text, and we apply full attention for image tokens. All the previous modalities are fully visible to future modality tokens.
Latent ControlNet. To effectively guide diffusion models with patch-level CLIP image embeddings, we create latent ControlNet (Fig. top right), by modifying the original ControlNet architecture from a backbone image diffusion model (e.g., FLUX.1-
NILM Oranch
Valao Gan Dranch
400
N0000
dev [32]) with the following two changes:
We replace the input linear projection layer (R3xd - xd), where d is the embedding di- mension of the backbone image diffusion model, and d' is the embedding dimension of the MILM. To further reduce the number of visual tokens gen- erated by the MLLM, we introduce a lightweight downsampling 2D convolution block to reduce
-20++
<BOT>
sign
Vilan Our pansh
the spatial resolution of the visual embeddings by
factor of before passing to latent ControlNet.
During training, we update only the newly added
Figure 3: Attention mask. X-axis is input and Y-axis is output.
input linear projection, the 2D downsampling con- volution blocks, and the ControlNet for 4 MM-DiT blocks and Single-DiT block, compared to the full FLUX.1 model which contains 19 MM-DiT blocks and 38 Single-DiT blocks in total.
3.3 Training and Inference
Decoupled training. The MILM visual prediction branch and latent ControlNet can be jointly trained or separately. We adopt the second, a decoupled training strategy for the following two reasons: (I) Since we only train small parameters of latent ControlNet, it converges much faster than the MLLM's vision branch. In initial experiments, we find that decoupled training allows us to allocate more compute the MLLM's vision branch, improving overall performance. (2) Since recent diffusion models have large parameters (e.g., FLUX. 1-dev has 12B parameters), end-to-end training significantly increases memory requirements. For the MLLM visual prediction branch, we use the mean squared error (MSE) loss for image patch embedding prediction. For the latent ControlNet, use the original flow-matching loss used in FLUX ControlNet.
Inference of image patch embeddings tokens via masked autoregression. During inference, the MLLM is given text prompt along with fully masked CLIP image embeddings. Following MAR [36], we first randomly sample a generation order of image patches, then iteratively denoise them into clean CLIP image embeddings. Once all image patches are obtained, we give them latent ControlNet to guide the diffusion model.
4 Experiment Setup
Latent ControlNet implementation details. For the latent Control design, we build upon the official FLUX.I-dew ControlNet training implementation from the Hugging Face Diffusers library." Specifically, we train a ControlNet consisting of 4 MM-DiT [19] (DoubleStream) blocks and 1 Single- DiT [52] (SingleStream) block (i.e., num_double dam__odoee___lerr_,,._. num_ num_single__ single_layersse total batch size of 48. All other training hyperparameters, including learning rate, Adam optimizer, and weight decay, are kept identical to the original codebase without any tuning. During latent ControlNet inference, we also retain all default hyperparameters unchanged (e.g., num controlnet_condee t ioninggnnn.....................e scale=0.1 .7, seieddacc__saaee3..e
with
28,
Baselines. For ImageNet experiments, we compare BIFROST-1 1 with two main variants, using the same model initialization and computational budget: (1) the first variant replaces the MLLM-aligned
5

---

Table I: Comparison of different architectures for bridging LLMs and diffusion models in image
generation on ImageNet 256x256.
Method
FID
sFID]
1St
Backbone difffusion model
FLUX.1-dey [32]
51.20
224.82
157.46
2D Learnable query tokens instead CLIP latent
MLLM 2D Learnable Query Tokens Latent ControlNet + FLUX.1-dev
118.69
129.14
9.15
No pre-aligned visual encoder instead of pre-aligned visual encoder
MI..M+ SigT.IP Latent ControlNet FI.UX.I-dev
274.16
304.94
2.69
VAE instead of CLIP latent
MLLM FLUX VAE + Latent ControlNet + FLUX.IIdev
284.51
361.45
1.11
Cross-attention instead of Latent ControlNet
MLLM Pateh-level CLIP Latent + cross-attention FLUX.I-dev
76.32
208 11
26.35
BIFROST-1 (Ours)
MIL.M Patch-level CLIP Latent Latent ControlNet FLUX.. I -edev
25.77
53.67
98.57
CLIP-based embeddings with non-MLL FLUX VAE features for visual generation, and (2) the second variant uses 2D query tokens (extended from MetaQuery [51]) instead of masked
autoregression for visual generation. For SoTA comparison experiments, we compare our method with baselines for unified multimodal understanding and generation including DreamLLM [18],
Chameleon [62], Show-o [73], VILA-U [72]. EMU3 [68], MetaMorph [63], MetaQuery [51], Token-
Flow [24], Transfusion [80], LMFusion [59]. Janus [71], JanusFlow [43], and JanusPro [10].
Training details. We train BIFROST-1 ImageNet [1 [16] for the experiments in Sec. 5.1 and Sec. 5.2. Specifically, the latent ControlNet and BIFROST-1 MLLM in Sec. 5.1 are trained for 2 epochs and 16 epochs respectively, and the latent ControlNet in Sec. 5.2 is trained for only epoch (~225M training steps). For SoTA comparison experiments in Sec. 5.3, we measure the training cost by the total number of training images passed to the model, calculated as the product of the number of training steps and the batch size per step. Our models with Qwen2.5.Vi 3B/7B are trained on 9M and 62M images respectively from the BLIP3-o [8] training dataset. 2 All experiments on ImageNet (Sec. 5.1 and Sec. 5.2) are trained on a single GH200 GPU, and the SoTA comparison experiments in Sec. 5.3 are trained on 16 GB200 GPUs.
Evaluation datasets and metrics. For ImageNet, following previous works [36, 56], we evaluate our model on Fréchet Inception Distance (FID) [27], sFID [45]. and Inception Score (IS) [57]. Then for open prompt evaluation, we follow previous works [10, 51, 63, 59] and report FID scores on MJHQ-30K [35] and 30k randomly sampled images from MSCOCO [38] validation set for visual aesthetic quality, and GenEval [25] and DPG-Bench [28] for prompt alignment, respectively. We use the original prompts from all these four evaluation datasets directly without any prompt rewriting.
5 Results and Discussion
We empirically demonstrate the effectiveness of BIFROST-1 I in various experiments. Specifically, we compare different architectures for bridging L.LMs and diffusion models (Sec. 5.1), training efficiency of different numbers of CLIP latent tokens for image reconstruction (Sec. 5.2), BIFROST-1 with SoTAs on multimodal understanding and generation tasks (Sec. 5.3), and experiments with different MLLM decoding steps (Sec. 5.4).
5.1 Comparison of Design Choices for Bridging LLMs and Diffusson Models
We compare different choices for bridging LLMs and diffusion models in terms of image generation task on ImageNet 256x256. We generate 10K images with classes randomly sampled from Ik categories and compute visual quality metries. In all settings, the MLLM visual generation branch and the Latent ControlNet are trained for 16 and 2 epochs, respectively.
Ptw.Ll-----Petrin----ee Caption

---

FFID
PSNR
LPIPS
0.35
14
0.20
5000c
144
0.25
0.5
5
0.20
03
0.35
0.30-
0.4
Tairing (K)
Training
Figure 4: Image reconstruction scores with different numbers of 2D CLIP latent tokens used within BIFROST-1 on ImageNet for around one training epoch (~266 steps). Results indicate that using more tokens achieves better data efficiency.
BIFROST-1 Backbone diffusion model. In the first and last rows of Table 1, we compare BIFROST-1 I and its backbone diffusion model (FLUX. 1-dev [32]). We find that BIFROST-1 improves FID and sFID, while it hurts IS from the original backbone. The improvement over the baseline mainly comes from adding a few trainable ControlNet blocks to the backbone diffusion model, which enables better adaptation the data distribution.
Patch-level CLIP latent vs. 2D learnable query tokens. MetaQuery [51] is a recent method that bridges LLMs and diffusion models by learning a connector (24-layer transformer encoder) that projects a frozen LLM's hidden representations on a finite number of learnable query tokens, where the connector outputs guide diffusion model via cross attentions. Inspired by this, implement a 2D version of MetaQuery and compare it with BIFROST- 1. To fairly compare methods with similar additional parameters, instead of learning a heavy connector module, directly reshape the L.M representations of learned query tokens as inputs to the latent ControlNet. As shown in Table 1, 2D learnable query tokens hurt performance in all 4 metrics, indicating that learning to align representations between MI.MM and diffusion from scratch requires much more computation than our lete CLIP latent.
Patch-level CLIP latents vs. VAE latents. Here, we compare BIFROST-1 I with a variant that replaces CLIP latents (that are natively aligned with MLLM) with VAE latents (that are not originally aligned with MLLMs). Specifically, we substitute the CLIP visual encoder with the FLUX VAE encoder, and replace our visual decoder (i.e., Latent ControlNet FLUX diffusion model) with the FLUX VAE decoder. Linear projection layers are applied to align the dimensions of the VAE-encoded features with the feature dimension of the MLLM backbone. All other components of our framework and the training strategy are kept the same to ensure a fair comparison. As shown in the 3rd row of Table 1, using VAE features significantly slows down learning. Additionally, this highlights that by leveraging a pretrained diffusion model as the image renderer, BIFROST- effectively reduces the burden on the MLLM side to directly generate high-quality images.
MLLM's native visual encoder vs. non-aligned external visual encoder. In addition, we also compare using the latents from MLLM's native visual encoder (i.e., CLIP) with an external visual encoder (i.e.. SigLIP, as used in MetaMorph (63].. As we can see in the 4th row of Table 1, although using SigLIP achieves better image generation quality than using VAE, it is still significantly worse than using the MLLM's native visual encoder. This indicates the training efficiency of adopting MLLM's native visual encoder compared with non-aligned external visual encoders.
Diffusion model guidance strategy: Latent ControlNet vs. Cross-attention Our method directly injects 2D image tokens generated by the MLLM into the DiT via latent ControlNet. By adding 2D ControlNet latents on top the noisy DiT latents, our method can enforce spatial structures more explicitly and effectively. In contrast, several previous works [63, 51, 8] use cross-attention to condition on image tokens. We conducted an additional experiment comparing these two conditioning strategies. For fair comparison, unfreeze all parameters in DiT when conditioning it via cross- attention and use 64 --genrrree image tokens for both methods. As shown in the 5th row of Table 1, our latent ControlNet achieves better image generation quality (FID, sFID, IS), demonstrating its superior efficiency and effectiveness.
5.2 Image Reconstruction with Patch-level CLIP Latent
In the following, we demonstrate the effectiveness of CLIP Latent-based image representation in image reconstruction experiments. Concretely, we experiment with scaling the number of latent tokens and compare BIFROST-1 I with SoTA models.
7

---

16 Tokens
64 Tokens
144 Tokens
256 Tokens
Real Image
Figure 5: Visual samples for image reconstruction with different numbers of patch-level CLIP tokens generated from MILMM. The Latent ControlNet models with varying numbers of tokens are trained for only epoch on the ImageNet training split.
Real Image
SEED
Emu
Emu2
GPT-4o
MetaQuery
Bifrost-1
Figure 6:
Image reconstruction results.
Scaling the number of CLIP latent tokens. In Fig. 4 we show that image reconstruction quality scales well with the number of patch-level CLIP latent tokens input to the Latent ControlNet. With only epoch of training (~26K steps) on the ImageNet dataset, images represented by 256 (=14x 14) CLIP latent tokens not only achieve higher reconstruction accuracy, measured by rFID, SSIM, PSNR, and LPIPS, but also converge faster compared to those using fewer CLIP latent tokens. In Fig. 5, we show qualitative examples of images generated with different numbers of CLIP latent tokens.
Comparison with SoTA methods. In Fig. 6, we qualitatively compare the image reconstruction quality of our latent ControlNet with various unified models, including SEED [22], EMU [61], EMU2 [60]. GPT-4o [29], and MetaQuery [51]. Specifically, we first encode the images using the visual encoder in Qwen2.5-VL. then the latent ControlNet reconstruct the images based solely on this visual information, without providing any text prompt as additional guidance. Trained exclusively the ImageNet dataset for 3 epochs without exposure any other open-worke images, the reconstructions from the BIFROST- 1 latent ControlNet achieve quality that is competitive with superior to strong baselines such GPT-4o and MetaQuery, demonstrating the efficiency and robustness of our Latent ControlNet design.
5.3 Comparison with SoTA Models
BIFROST-1 1 fully inherits strong visual understanding capabilities of backbone MILM. As BIFROST- keeps the original parameters in the MILLM completely frozen, the trainable vision branch agnostic to the choice of MLLM. This allows BIFROST-1 to fully inherit the visual understanding

---

Table 2: Comparison with state-of-the-an methods on multimodal understanding benchmarks.
Method
Base (M)LMM
MME-Pt
MMB1
SEEDT
MMMU1
MM-Vett
Dr [18]
Vicuna 7B
36.6
Chameleon [62]
Prom Scratch 7B
22.4
8.3
Show-o-512 [73]
Phi-1.5 1.3B
1097.2
26.7
VILA-U [72]
LLaMA-2 7B
1401.8
59.0
33.5
HMU3 [68]
From Scratch 7B
58.5
68.2
31.6
37.2
MetaMorph [63]
I.LMA-33
75.2
71.8
MetaQuery-L [51]
Qwen2.5-VIL
1574.3
78.6
73.8
53.1
63.2
MetaQuery-XLL [51]
Qwen2.5-VL. 7B
1685.2
83.5
76.9
58.6
66.6
TokenFlow-Xi [24]
Qwen-2.5 14B
1551.1
76.8
72.6
43.2
48.2
Transfusion [80]
From Scratch 7B
LMFusion [59]
I.LVA-Nextt 8B
1603.7
72.1
72.5
41.7
Janus [71]
DeepScck-L.IMM 1.5B
1338.0
69.4
63.7
30.5
34.3
JanusFlow [43]
DeepScck-LLLM 1.5B
1333.1
74.9
70.5
29.3
30.9
JanusPro-1B [10]
DeepSeck-LLM 1.5B
1444.0
75.5
68.3
36.3
39.8
JanusPro-7B [10]
DeepSeck-LLMM 7B
1567.1
79.2
72.1
41.0
50.0
BLIP3-o 4B [8]
Qwen2.5-VL 3B
1527.7
78.6
73.8
46.6
60.1
BILP3-o 8B [8]
Qwen2.5-VL 7B
1682.6
83.5
77.5
50.6
66.6
BIFROST-1 1 (Ours)
Qwen2.5-VL 7B
1685.2
83.5
76.9
58.6
66.6
Table 3: Comparison with state-of-the-art methods on multimodal generation benchmarks.
Method
Base (M)LLM
Train Steps
COCO FID
MJHQ FIDI
GenEvalt
DPG-Bencht
EMU [68]
LLaMA 13B
11.66
D.e [18]
Vicuna 7B
8.46
Chameleon [62]
7B
Show-o-512 [73]
Phi-1.5 1.3B
9.24
15.18
0.68
VI2A-U
LLaMA-2 7B
7.69
[68]
From Scratch 7B
80.60
MetaMorph [63]
LLaMA-3 8B
11.8
MetaQuery-LII [51]
Owen2.5-VL 3B
8.87
6.35
0.78
81.10
MetaQuery [51]
Qwen2.5 7B
200M
6.02
82.05
TokenFlow-XX [24]
Qwen-2.5 14B
0.63
73.38
Transfusion [80]
From Scratch 7B
8.70
0.63
LMFusion [59]
LLaVA-Next 8B
8.20
Janus [71]
DespSeck-LLMM 1.5B
100M
8.53
10.10
0.61
JanusFlow [43]
DeepSeck-LLLMM 1.5B
211M
9.51
0.63
80.09
JanusPro-11 IB [10]
DeepScck-LLM
200M
14.33
0.73
82.63
JanusPro-7B [10]
DeepSeck-LLM 7B
194M
13.48
0,80
84,19
BLIP3-o [8]
Qwen2.5-VL. 3B
79.36
BLIP3-0 [8]
Qwen2.5-VI.
0.84
81.60
BIFROST-1 (Ours)
Qwen2.5 VI. 3B
9M
23.02
15.24
0.61
76.41
BIFROST-1 (Ours)
Qwen2.5-V VL 7B
62M
34.35
16.21
0.81
77.67
capabilities of the MILM and easily scale to backbone MLLMs with larger sizes and stronger performance. Table 2 compares the @wwn2.55-VL 7B [5] model with other unified models on image understanding benchmarks, including MME-P [21], MMB [39], SEED [34], MMMU [76], and MM-Vet [75]. Baseline models such JanusPro [10] and MetaMorph [63] fully fine-tune the LLM backbone and therefore tend to lose the original planning and reasoning abilities that these LLMS
acquired through large-scale text pretraining.
BIFROST-1 achieves competitive visual generation quality. Fig. compare BIFROST- with the recent SoTA methods (e.g., JanusPro and MetaQuery) on MJHQ30k prompts. Table compares BIFROST-1 I with other unified models image generation benchmarks, including COCO and MJHQ for visual quality and GenEval and DPG-Benele for prompt following ability. Trained only with 25M image-text pairs for less than two epochs, BIFROST- 1 matches the performance with models trained with much higher compute, including Janus on GenEval benchmark, and outperforms TokenFlow- on DPG-Bench. In addition, we would like to highlight that FID scores heavily depend on choice of diffusion backbones. As also observed in MetaQuery[5 J. diffusion models fine-tuned on aesthetic datasets (e.g., FL.UX. 1-dev) typically achieve worse FID scores compared to models that have not undergone extensive aesthetic fine-tuning (e.g.. SD1.5, SANA).

---

JanusPro
MetaQuery
Bifrost-1 (Ours)
baby town wlstera romantic spedimen, pinktsh white beciground, by Thoraa Körkade Nadia flaeter Anne
angry
squeczee srawbony selective focus, reso ution
dbudy soy body of Koathley, Shutterttok contect winner Phone video whirle ind, Armggeation photo taken upic intricate. sparge Wasseimann, spaiiig
image world
Nanoy
Figure 7: Qualitative comparision with SoTA methods for image generation on MJHQ30K [35].
Table 4: Inference clock time and image generation quality with different VLM decoding steps.
# Decoding Steps
64 (default)
32
16
8
2
Clock Time (seconds)
5.2
2.63s
1.37s
0.66s
0.35s
0.19s
0.09s
FID
18.64
18.49
18.79
18.89
19.97
24.90
60.35
sFID IS f
43.64
43.58
44.52
45.89
50.86
67.23
217.50
156.01
158.93
156.96
158.87
150.56
124.78
83.22
5.4 MILM Inference Time with Varying Decoding Steps
To further assess the effect of decoding steps, conducted experiments with varying numbers of decoding passes on ImageNet. The results are summarized in Tab, 4. We show the inference clock time in seconds, as well image quality metrics including FID, sFID, and IS. As shown, our method remains robust as long as the number of decoding steps is greater than 8. And we can observe that there is clear trade-off between inference speed and image quality when using fewer decoding steps. Importantly, we also highlight that the MLLM decoding time is significantly smaller than the runtime of the diffusion-basedd image generation model. For instance, FL.UX. 1.X.I-dee 1-dev with 12B parameters together with latent ControlNets takes 14.79 seconds to generate a single image with the default 28 denoising steps. Therefore, the MILM decoding time standpoint, users can flexibly select the number of decoding steps based on whether they favor faster inference higher image quality for their particular application.
major bottleneck. From practical
6 Conclusion
We present BIFROST-I, a unified and efficient framework for both multimodal understanding and generation. BIFROST-] bridges pretrained MILMs with diffusion models by introducing patch-level CLIP image embeddings as latent variables, which are naturally aligned with the CLIP visual encoder of the MILM. This design enables high-fidelity, controllable image generation while maintaining strong training efficiency. A key advantage of BIFROST-1 1 is that it fully preserves the original parameters
MILMM backbone, ensuring that its multimodal planning and reasoning capabilities remain intact. Although current model is trained with significantly fewer computational resources than SoTA methods and does not yet surpass them performance, future work focuing on scaling to larger datasets and more powerful MLLM backbones can further realize its potential.
10

---

Acknowledgments
This work was supported by DARPA ECOLE Program No. HR001 1239006,, NSF-AI Engage Institute 12L55 ARO Award W911NF2110220, ONR Grant 200014-2311-3356,, and Bloomberg Data Science PhD Fellowship. The views contained in this article are those of the authors and not of the funding agency.

---

References
[1] AI, B. Gong, C. Zou, D. Zheng, H. Yu, J. Chen, J. Sun, J. Zhao, J. Zhou, K. Ji, L. Ru, L. Wang. Q. Guo, R. Liu, W. Chai, X. Xiao, and Z. Huang. Ming-lite-uni: Advancements in unified architecture architecture for natural multimodal interaction, 2025.
[2] M. AI. The Ilama 4 herd: The beginning of a new era of natively multimodal ai innovation, April 2025. Accessed: 2025-05-12.
[3] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro,
J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira,
O. Vinyals, A. Zisserman, and K. Simonyan. Flamingo: a visual language model for few-shot learning, 2022.
[4] J. Ansel, E. Yang, H. Hc, N. Gimelshein, A. Jain, M. Voznesensky, B. Bao, P. Bell, D. Berard, E. Burovski, G. Chauhan, A. Chourdia, W. Constable, A. Desmaison, Z. Vito, E. Ellison, W. Feng, J. Gong, M. Gschwind, B. Hirsh, S. Huang, K. Kalambarkar, L. Kirsch, M. Lazos, M. Lezcano, Y. Liang, J. Liang, Y. Lu, C. Luk, B. Maher, Y. Pan, C. Puhrsch, M. Reso, M. Saroufim, M. Y. Siraichi, H. Suk, M. Suo, P. Tillet, E. Wang, X. Wang, W. Wen, S. Zhang. X. Zhao, K. Zhou, R. Zou, A. Mathews, G. Chanan, P. Wu, and S. Chintala. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS '24). ACM, Apr. 2024.
[5] S. Bai, Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. J., Ye, X. Zhang, Xie, Cheng, H. Zhang. Z. Yang, H. Xu, and Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 2025.
[6] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[7] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558-3566 2021.
[8] Chen, Z. Xu, X. Pan, Y. Hu, C. Qin, T. Goldstein, L. Huang, T. Zhou, S. Xie, S. Savarese, L. Xue, C. Xiong, and R. Xu. Blip3-o: A family of fully open unified multimodal models- architecture, training and dataset, 2025.
[9] M. Chen et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
[10] X. Chen, Z. Wu, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, and C. Ruan. Janus-pro: Uni- fied multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501. 17811, 2025.
[11] J. Cho, J. Lei, H. Tan, and M. Bansal. Unifying vision-and-languaa tasks via text generation. In ICML, 2021.
[12] Cho, J. Lu, D. Schwenk, H. Hajishirzi, and A. Kembhavi. X-Ixmert: Paint, caption and answer questions with modai transformers. In EMNLP, 2020.
[13] Cho, A. Zala, and M. Bansal Visual programming for step-by-step text-to-image generation and evaluation. Advances in Neural Information Processing Systems, 36:6048-600000 (00000486666,2200 2023.
12

---

[14] S. Datta, A. Ku, D. Ramachandran, and P. Anderson. Prompt expansion for adaptive text-to- image generation. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 3449-3476, Bangkok, Thailand, Aug. 2024. Association for Computational Linguisties.
[15] G. DeepMind. Gemini 2.5 pro, March 2025. Large language model.
[16] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, leee, 2009.
[17] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
[18] R. Dong, C. Han, Peng, 7. Qi, Z. Ge, J. Yang, Zhao, Sun, H. Zhou, Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. In The Twelfth International Conference on Learning Representations, 2023.
[19] P. Esser, S. Kulal, A. Blattmann, R. Entezari, and J. Müller. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint S.Kiv:2033.32006 2024.
[20] W. Feng, W. Zhu, T.-j. Fu, V. Jampani, A. Akula, X. He, S. Basu, X. E. Wang, and W. Y. Wang. Layoutgpt: Compositional visual planning and generation with large language models. arXiv preprint arXiv:2305. 153993 2023.
[21] C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, Lin, Yang, X. Zheng, K. Li, X. Sun, al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
[22] Y. Ge, Y. Ge, Z. Zeng. X. Wang, and Y. Shan. Planting a seed of vision in large language model. arXiv preprint arX 000001, 2023.
[23] Y. Ge, S. Zhao, J. Zhu, Y. Ge, K. Yi, L.. Song, C. Li, X. Ding, and Y. Shan. Seed-x: Multi- modal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404. 14396, 2024.
[24] M. Geyer, O. Bar-Tal, Bagon, and T. Dekel. Tokenflow: Consistent diffusion features for consistent video editing. In The Twelfth International Conference on Learning Representations, 2023.
[25] D. Ghosh, H. Hajishirzi, and L. Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52122-52152, 2023.
[26] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[27] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017.
[28] X. Hu, R. Wang, Y. Fang, B. Fu, P. Cheng, and G. Yu. Ella: Equip diffusion models with Ilm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024.
[29] A. Hurst et al. Gpt-4o system card. arXiv:2410.21276 [cs.CL].
2024.
[30] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitchead, A. C. Berg, W.-Y. Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015-4026, 2023.
[31] Koh, D. Fried, and R. Salakhutdinov. Generating images with multimodal language models. NewrIPS, 2023.
13

---

[32] B. F. Labs.
dedev,
2024.
[33] A. Lewkowycz et al. Minerva: Solving quantitative reasoning problems with language models. arXiv preprini arXiv:2206. 14858, 2022.
[34] B. Li, R. Wang, G. Wang. Y. Ge, Y. Ge, and Y. Shan. Seed-bench: Benchmarking multimodal Iims with generative comprehension. arXiv preprint arXiv:2307.16122, 2023.
[35] D. Li, A. Kamko, E. Akhgari, A. Sabet, L. Xu, and Doshi. Playground v2. 5: Three insights to- wards enhancing aesthetie quality in text-to-image generation. arXiv preprint 11X4::2402..72455 2024.
[36] T. Li, Y. Tian, H. Li, M. Deng, and K. He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37:56424-560 56445, 2024.
[37] C. Liao, L. Liu, X. Wang. Z. Luo, X. Zhang, W. Zhao, J. Wu, L. Li, Z. Tian, and W. Huang. Mogao: An omni foundation model for interleaved multi-modal generation. arXiv preprint arXiv:2505.05472, 2025.
[38] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
[39] Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. Mmbench: your multi-modal model an all-around player? In European conference on computer vision, pages 216-233. Springer, 2024.
[40] Y. Liu et al. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. / /692, 2019.
[41] Lu, C. Clark, Lee, Z. Zhang, S. Khosla, R. Marten, D. Hoiem, and A. Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint arXiv:2312.17172, 2023.
[42] Lu, C. Clark, R. Zellers, R. Mottaghi, and A. Kembhavi. UNIFIED-IO: A unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference Learning Representations, 2023.
[43] Y. Ma, X. Liu, X. Chen, W. Liu, C. Wu, Z. Wu, Z. Pan, Z. Xie, H. Zhang, X. yu, L. Zhao, Y. Wang. J. Liu, and C. Ruan. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation, 2024.
[44] S. Mo, T. Nguyen, X. Huang, S. S. Iyer, Y. Li,, Y. Liu, A. Tandon, E. Shechtman, K. K. Singh, Y. J. Lee, B. Zhou, and Y. Li. X-fusion: Introducing new modality to frozen large language models, 2025.
[45] C. Nash, D. Barrett, and D. Rezende. Generating images with sparse representations. In International Conference on Machine Learning (ICML), pages 7937-7947. PMLR, 2021.
[46] E. Nijkamp al. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. 13474, 2022.
[47] OpenAl. Improving image generation with better captions, 2023. Accessed: 2025-05-11.
[48] OpenAl. Introducing gpt-4o: Image, text, and audio generation, 2024. Accessed: 2025-05-11.
[49] OpenAl. Openai ol team page, 2024. Accessed: 2025-05-11.
[50] OpenAl. Video generation models as world simulators, 2024. Accessed: 2025-05-11.
[51] X. Pan, S. N. Shukla, A. Singh, Z. Zhao, S. K. Mishra, J. Wang, Z. Xu, J. Chen, K. Li, Juefei-Xu,, Hou, and S. Xie. Transfer between modalities with metaqueries. arXiv preprint arXiv:250..06256 2025.
14

---

[52] W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
[53] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In ICMI., 2021.
[54] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
[55] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint wrXiv:2244001222 2022.
[56] S. Ren, Q. Yu, J. He, X. Shen, A. Yuille, and L.-C. Chen. Beyond next-token: Next-x prediction for autoregressive visual generation. arXiv preprint arXiv:2502.203388 2025.
[57] T. Salimans. I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems (NeurIPS), volume 29, 2016.
[58] W. Shi et al. Prm800k: A prompt-rewarded dataset for math reasoning. arXiv preprint arXiv:2305.171166 2023.
[59] W. Shi, X. Han, C. Zhou, W. Liang, X. V. Lin, Zettlemoyer, and Yu. Lmfusion: Adapting pretrained language models for multimodal generation, 2025.
[60] Q. Sun, Y. Cui, X. Zhang, F. Zhang, Q. Yu, Y. Wang. Y. Rao, J. Liu, T. Huang, and X. Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14398-14409, 2024.
[61] Q Sun, Q. Yu, Y. Cui, Zhang, X. Zhang, Y. Wang, H. Gao, J. Liu, Huang, and X. Wang. Emu: Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222, 2023.
[62] C. Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint
24X5v:2405.09818, 2024.
[63] S. Tong, Fan, J. Zhu, Y. Xiong, X. Chen, K. Sinha, M. Rabbat, Y. LeCun, S. Xie, and Z. Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint 1411442412.14164, 2024.
[64] A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning, 2018.
[65] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[66] P. von Platen, S. Patil, A. Lozhkov, P. Cuenca, N. Lambert, K. Rasul, M. Davaadorj, and T. Wolf. Diffusers: State-of-the-ar diffusion models. attps://github com hugn//gggnggace///////o diffusers, 2022.
[67] Wang, A. Yang, R. Men, J. Lin, Bai, Z. Li, Ma, C. Zhou, J. Zhou, and H. Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-t--sequence learning framework. CoRR, 0bs/2202.03052. 2022.
[68] X. Wang, X. Zhang, Z. Luo, Q. Sun, Y. Cui, J. Wang, F. Zhang, Y. Wang, Li, Q. Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409 / 18869, 2024.
[69] 7. Wang, J. Yu, A. W. Yu, Dai, Tsvetkov, and Y. Cao. Simvlm: Simple visual language model pretraining with weak supervision, 2022.
15

---

[70] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowiez, Davison, S. Shleifer, von Platen, C. Ma. Y. Jernite, J. Plu, C. Xu, T. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Transformers: State-of-the-art natural language processing. Proceedings the the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 4845, Online, Oct. 2020. Association for Computational Linguistics.
[71] C. Wu, X. Chen, Z. Wu, Y. Ma, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, C. Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410. 13848, 2024.
[72] Y. Wu, Z. Zhang, J. Chen, H. Tang, D. Li, Y. Fang. L. Zhu, E. Xie, H. Yin, L. Yi, et al. Vila-u: unified foundation model integrating visual understanding and generation. arXiv preprini DrXiv:2409004222 2024.
[73] J. Xie, W. Mao, Z. Bai, D. J. Zhang, W. Wang, K. Q. Lin, Y. Gu, Z. Chen, Z. Yang, and M. Z. Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
[74] Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong, X. Zhang, G. Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint 2024.
arXiv:2408.0660
[75] W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In Forty-first International Conference on Machine Learning, 2023.
[76] X. Yue, Y. Ni, Zhang, Zheng, Liu, G. Zhang, Stevens, D. Jiang, W. Ren, Sun, et al. Mmmu: A massive multi multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9561 2024.
[77] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[78] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. In IEEE International Conference on Computer Vision (ICCV), 2023.
[79] L. Zhang, A. Rao, and M. Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[80] C. Zhou, Yu, A. Babu, K. Tirumala, M. Yasunaga, L. Shamis, Kahn, X. Ma, Zettlemoyer, and O. Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model, 2024.
[81] Zhu, W. Wang, Z. Chen, Z. Liu, Ye, L. Gu, H. Tian, Y. Duan, W. Su, J. Shao, Z. Gao, E. Cui, X. Wang. Y. Cao, Y. Liu, X. Wei, H. Zhang, H. Wang. W. Xu, H. Li. J. Wang, N. Deng, S. Li. Y. He, T. Jiang, J. Luo, Y. Wang, C. He, B. Shi, X. Zhang, W. Shao, J. He, Xiong, W. Qu, P. Sun, P. Jiao, H. Lv, L. Wu, K. Zhang, H. Deng, J. Ge, K. Chen, L. Wang, M. Dou,
I.. Lu, X. Zhu, T. Lu, D. Lin, Y. Qiao, J. Dai, and W. Wang. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025.
16

---

Appendix
A
Formal Description of the BIFROST-1 MLLM Architecture
As illustrated in Fig. 2, we initialize the visual generation branch from the pretrained MILMM by creating a trainable copy of the MLP and attention QKV projection layers [65]. The only component randomly initialized is the vision head, which is a simple linear projection layer. By reusing the majority of parameters from the pretrained MLLM, we avoid the costly process of realigning image embeddings.
Specifically, we use blue color to denote the original frozen MI...M-peecffcc modules, which handle language modeling (LM) and image understanding (Img-U) tasks. These modules remain unchanged during training. In addition, modules for the image generation task (Img-G), which are initialized as trainable copies of the corresponding blue modules.
use yellow color represent the newly introduced trainable
During the image generation training process, first obtain input hidden states for each task. Text inputs x Text are projected through a linear embedding layer to produce Text Images used for image understanding (a Um) and image generation (a (m are both passed through the frozen
MLLM-embedded visual encoder & Undd producing hidden states in Img and Iim
Imng respectively:
h
Text
Linearrext
Text
in
Img-U
E Und (x
Img-U)
Img-G =E in
(x Img Im
During the image generation inference process. hi
lmg--i is initialized from 2D learnable mask tokens.
By using patch-level CLIP image embeddings that natively aligned with the MLLM's visual encoder to represent visual signals in vision generation tasks, we eliminate the need for any additional alignment between the visual generation representation and the MILMM.
For attention processing, we use the frozen MLLM attention layers to compute Q, K, and matrices for the text and image understanding tokens h Text and in ln
Img-U and use the newly added visual
generation branch to process the image generation tokens ft hy Text Text he lk
QKV MLLM (h
ho
Img-U Img-U Img-U_
lek
QK V MLLM
Img-U)
in
Img--G ho
Img-G hg
Img-G by
QKV Ing
(h Img-GG
For language modeling and image understanding tasks, we replicate the standard MLLM attention structure by attending over their respective modalities only. For image generation task, we enable cross-module attention, allowing image generation queries attend jointly over all token types. Specifically:
Text
OMLLM (Attn(he Teat [hk h2 ling ", [h hey Img "I)
to
Img-U ho
OMILM (Attn(h) Img.U ,1 lmg-U hi Textj, [h Amg-U (ox)))))
fo
Img. G
Ormg-G (Attn(h Img-G Img.G [|ak
hg
Img-U Text
[h
Img-Gi h
Img-U:
hy (eT)))))
where denotes concatenation, and O(-) denotes linear output projection layer. We apply causal mask to text and image understanding tokens, and a bidirectional mask to image generation tokens.
For the MLP layers, we follow the same branching: text and image-understanding tokens T and U are passed through the frozen MLLM MLP, while image generation tokens h use the
hin
Img-G
trainable MLP from the generation branch:
Text MLPP
MLPMLLM Text)
Img-U M.
MLPMLLM
(h (memme-))
Img-U
Img-G PM
MLPPig-G
(h
Img-G
Finally, task-specific heads convert the hidden states to output predictions. For language modeling and image understanding, we apply a linear projection to h Text Ing G
MLP: and for image generation, we project
h MLMP
using the vision generation head:
h Text out
TextHead (orp)
Img-G h
VisionHead (me
Img-G
17

---

B Broader Impacts
BIFROST-1 is motivated by the fact that training unified multimodal generation and understanding model that can perform native generation with high visual quality usually requires huge computational cost. By bridging pretrained MILM with pretrained diffusion models, training BIFROST- I can be significantly faster. Therefore, we believe that our work can be strong contribution to efficient unified model training. While our framework can benefit numerous applications in image generation, similar other image generation frameworks, it can also be used for potentially harmful purposes (e.g., creating false information or misleading images). Therefore, should be used with caution in real-world applications.
C Safeguards
BIFROST-1 1 is built upon pretrained MILM (i.e., Qwen2.5-VL) and diffusion models (i.e., FLUX.1- dev) with strong safeguards, and trained publically available image datasets (i.e., MSCOCO and SA1B) that removes unsafe concepts. Therefore, our model avoids the high risk for misuse.
D Limitations
Note that BIFROST-1 is designed bridging method that connects existing MLLMs with diffusion- based image generation models. As such, its performance, output quality, and potential visual artifacts inherently influenced by the capabilities and limitations of the underlying backbone models relies on. For instance, if the diffusion model used as the visual backbone struggles with generating complex, rare, or previously unseen scenes and objects, then BIFROST- 1, which builds upon this foundation, may also exhibit suboptimal image generation results. This dependency highlights the importance of selecting strong and well-generalized base models when applying BIFROST- to real-world or open-domain generation tasks.
E License
We use standard licenses from the community and provide the following links to the licenses for the datasets, codes, and models that we used in this paper. For further information, please refer to the specific link.
PyTorch [4]: BSD-style
HuggingFace Transformers [70]: Apache License 2.0
HuggingFace Diffusers [66]: Apache License 2.0
FLUX.1-dev [32]: Non-Commercial License
Qwen2.5-VL. Non-Commercial License
MSCOCO dataset [38]: CC BY 4.0
CC12M dataset [7]: Permissive Custom License
SAIB dataset [30]: SA-1B Dataset Research License
MJHQ30k dataset [35]: Playground v2 Community License
18

---

Colosseum nigh, beautiuuly Ikminnatd, 20 minutes after etaryy right, dramatic, emational, saturated, resiistic profeasional, winning
A high end residenial swrmming pool with the Attenta stylne serving dynamic statecttheart outdoor luury apa and stunning poutside lighting
Starry night blended into mallstic walkoway, the styls vincent gugh, watercolor parting. amblent
Medieval fanlasy heroic universe, nanural underground unreal engine
Modern egyst with flying spaceships, energy generating pyramicss, gantts rivers the
Realistic, bright
black lian fuluristic shedes mumford, segall, mashup of styles 1990s, solarpunk
Sveampunk detailled complets background
cule beby furedo slyle, body
Aonilla, dresssed Karale Lebbovtzz
The lttle mermaid dienery watercolour
Cinematic, restistic robot humenad
Spidermen
Futurietic soric the hedgehogg, sega, sunglasses,, backdrop
Famlastic wizard who Background of blue
forest background realistic v5, looking, lens flare, evening sky
magic spell
rearing ing foreground facing orange
color, datailed background elaments, victor rizewteev, cark indigo, cute and quirky, leaf patlems, souplee
Lifello, inisescent gloss Beauty Beset, with weter droplets. Hundreds tny tantasy details, beausiful dasign, high contrast, sharpp focus
Detaled painting Van Gogh frank and j. leyandaciior, sold color background. soft colore, contrasting colors
fantissy
EASY RIDER
SNIEORN
GLACIERBAY CRUISE
weering sunglasses white background
logo. traneparent, dragon, television, explosion
Logo girl
travel of Bay Nationaa
vintage easy
Figure 8:
Visualization examples from MJHQ30K dataset.
19

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims we made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include limitation sections in the appendix.
20

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our model architecture is well-illustrated in Sec. 3, and experiment setup is shown in Sec. in detail.
Guidelines:
The answer NA means that the paper does not include experiments.
21

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We use publically available open-sourced datasets and benchmarks for model training and evaluation. We include our code in the supplimentary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /ttpsscccppbbpcc////eeccppcpppcp hwwuublccgguiieee guwdee/ /CodeSubmiii ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
22

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We show training and test details in Sec. 4.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the large training cost for each run, follow previous works [71, 43, 10, 51, 63, 80, 59] and only train the model once and inference once on each evaluation task.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe training compute details in Sec. 4.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
23

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: Satps://o /ttps/////ee...//eeeee ct/ccc//p//o hcc/public/Ethiie www.ceiiiileeciaecccc
Answer:
[Yes]
Justification:
We conform the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed broader impacts in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: Our model aims at bridging pretrained MILM and diffusion models which already have safeguards. We include safeguards information in the appendix.
Guidelines:
24

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We include licenses in the appendix.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.coo///aaaeae com/datasete has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pane se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We include instructions about code and how to the model in the README.md file the supplementary material.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
25

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: When preparing this manuscript, we used LLM only for editing (grammar checking) purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non- non-standard components.
Please refer to our LLM policy (https: hattps://neuriie ///uriuuss.../// www.////eeeeeee///000/1/ for what should or should be described.
26


Reviews:
[{"review_id": "IoTqjkHTCP", "reviewer": "Reviewer_NRRC", "summary": "This paper proposes BIFROST-1, a framework to bridge pre-trained Multimodal Large Language Models (MLLMs) with diffusion models for high-fidelity image generation. The core idea is to use patch-level CLIP image embeddings, which are natively aligned with the MLLM's visual encoder, as the communication medium. The MLLM is trained with a lightweight visual generation branch to predict these embeddings, which then guide a pre-trained diffusion model via a lightweight latent ControlNet. This approach aims to achieve efficient training and strong generation performance while preserving the MLLM's original reasoning capabilities by keeping its backbone frozen.", "questions": "- Could you provide standard quantitative metrics (e.g., GenEval and DPGBench) comparing BIFROST-1 with other leading text-to-image generation models like JanusPro or MetaQuery? The qualitative results in Figure 7 are promising, but objective numbers are needed.\n- Could you provide any quantitative results or qualitative examples demonstrating the model's ability to handle complex compositional prompts that require precise spatial reasoning and object attribute binding?\n- What is the inference latency of your method, particularly the autoregressive step for generating patch embeddings? How does it compare to other methods that generate guidance signals in a single forward pass?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n+ Novel and Clever Bridging Mechanism: The central idea of using natively-aligned, patch-level CLIP latents as the bridge is novel and elegant. It sidesteps the more complex alignment challenges faced by other methods and provides a conceptually simple way to connect the understanding capabilities of MLLMs with the synthesis power of diffusion models.\n+ High Training Efficiency: By freezing the large MLLM and diffusion model backbones and only training lightweight components, the proposed method is highly computationally efficient. The ablation study in Table 1 effectively demonstrates the superiority of this design choice over alternatives.\n+ Preservation of MLLM Capabilities: A key advantage is that the MLLM's original parameters are untouched, ensuring its strong multimodal understanding and reasoning abilities are fully preserved. This is well-supported by the benchmark results in Table 2.\n\n### Weaknesses\n- Insufficient Quantitative Comparison on Image Generation: The paper lacks a direct quantitative comparison against other state-of-the-art text-to-image generation models on standard benchmarks (e.g., DPG-Bench or GenEval). While Figure 7 provides qualitative examples, objective metrics like FID or CLIP Score are necessary to properly situate the model's performance. The Appendix is mentioned, but this critical comparison should be in the main paper for a thorough evaluation.\n- Limited Evaluation of \"Controllability\": While the framework is presented as enabling \"controllable\" generation, the experiments primarily focus on image reconstruction and relatively simple prompts. There is a lack of validation on complex compositional prompts involving multiple objects with specific spatial relationships. This makes it difficult to assess the true extent of control the method provides.\n- Unaddressed Inference Latency: The method relies on an autoregressive masking process to generate the patch embeddings during inference. This iterative process can be slow. The paper completely omits any analysis or reporting of inference speed, which is a significant oversight for a method aiming for practical utility.", "was_revised": true, "final_justification": "After reading the other reviewers' comments and the authors' response, I find that my initial concerns have been well addressed. Although reviewer 98F3 questions the novelty of using patch-level CLIP latents, I do not believe that is the core contribution of this paper. Instead, the key strength lies in how the authors effectively leverage patch-level CLIP latents to achieve structured guidance through a lightweight ControlNet. Overall, I am convinced by the authors’ clarification and experimental validation, and therefore decide to increase my score.", "rebuttal": "We would like to sincerely thank the Reviewer for the feedback. Since the weakness and questions of the Reviewer are closely related, we use “W/Q” to represent the answer to the corresponding weakness and questions.\n\n> ### W1/Q1: Standard quantitative metrics (e.g., GenEval and DPGBench) comparing BIFROST-1 with other leading T2I generation models.\n\nThank you very much for the useful comment. We would like to first kindly point out that we have included standard metrics from GenEval and DPGBench in Appendix Table 1 (in the PDF of the supplementary materials, as mentioned in L284-285 of our main paper). In Appendix Table 1, we measure the training cost by the total number of training images passed to the model, calculated as the product of the number of training steps and the batch size per step. As we can see, our method achieves performance comparable to baselines such as Janus and EMU, while significantly lower training cost (training data 9M from Ours v.s. 100M from Janus).\n\nBeyond the results presented in the appendix, we gained more GPU support after the paper submission deadline, and here we include additional larger-scale experiments trained on the same training dataset from BLIP3-o [1], one of the latest works that connects MLLM with diffusion models for image generation released on May 14, 2025. Specifically, we use 27 million image-text pairs from BLIP3o-Long-Caption dataset, and train the model for 1.7 epochs, followed by fine-tuning on 60k high-quality images in the BLIP3o-60K dataset. In contrast, BLIP3-o is trained on the same 27M dataset for over 10 epochs. We summarize the comparison in the following table:\n\n| Models | MLLM Backbone | Total Pretraining Images (Training Steps×Batch Size) | GenEval | DPGBench |\n| :---- | :---- | :---- | :---- | :---- |\n| BLIP3-o | Qwen2.5-VL 3B | \\~270M | 81% | 79.36% |\n| BLIP3-o | Qwen2.5-VL 7B | \\~270M | 84% | 81.60% |\n| Bifrost-1 (Ours) | Qwen2.5-VL 3B | 9M | 61% | 76.41% |\n| Bifrost-1 (Ours) | Qwen2.5-VL 7B | 46M | 81% | 77.67% |\n\nAs shown, the GenEval score improves significantly from 61% to 81%, and the DPGBench score increases from 76.41% to 77.67%. This demonstrates that higher-quality training data with better annotations, as well as scaled-up training, can lead to meaningful performance gains. Lastly, we would like to emphasize that even with this larger-scale training, our training cost remains much lower than that of prior works, and our method still outperforms previous approaches under the same training budget.\n\n[1] Chen, Jiuhai, et al. \"Blip3-o: A family of fully open unified multimodal models—architecture, training and dataset.\" arXiv preprint arXiv:2505.09568 (2025).\n\n\n--- \n\n> ### W2/Q2: Quantitative results or qualitative examples demonstrating the model's ability to handle complex compositional prompts that require precise spatial reasoning and object attribute binding\n\nThank you very much for this question regarding the model’s ability to handle complex prompts. First, we would like to apologize that, due to the NeurIPS rebuttal policy, we are not allowed to upload any qualitative examples or visualizations. However, we believe the benchmarks used in our evaluation on benchmarks including GenEval and DPGBench already include prompts that assess various capabilities such as object count, spatial and non-spatial relationships, colors, shapes, textures, and attribute binding between objects and their properties.\n\nWe compared our method (Bifrost-1) with BLIP3-o on several out-of-distribution prompts involving rare color–attribute bindings, spatial relationships, and object counts. These include examples such as: “A photo of a blue pizza and a yellow baseball glove”, “A photo of four giraffes”, “A photo of a skateboard above a person”, “A purple computer keyboard and a red chair”. Our results demonstrate that Bifrost-1 generates more accurate and faithful images compared to BLIP3-o. We promise to include additional visualizations and comparisons with such complex prompts in our camera-ready version.\n\nAdditionally, we would like to highlight that due to our design choice of keeping the MLLM fully frozen, the model is able to preserve its strong reasoning capabilities for understanding complex prompts. For example, given a prompt requiring implicit reasoning such as “The national flag of the country where Yellowstone National Park is located,” the MLLM can infer that the intended flag is that of the United States. It can then rewrite the prompt as “The national flag of the United States” before passing it to the visual generation branch. This ability to transform complex or implicit prompts into explicit and visual-friendly ones greatly enhances the model’s overall effectiveness in handling sophisticated user instructions.\n\n--- \n\n> ### W3/Q3: Inference speed for autoregressive masking process\n\nThank you very much for raising this valuable point regarding the inference speed of our model, particularly the autoregressive masking process. We agree that inference speed is an important consideration for practical applications, and that balancing efficiency and accuracy is a critical design trade-off. Below, we first outline the details of our inference procedure and then present new quantitative results using different numbers of decoding steps in the MLLM.\n\n- **Detailed inference procedure:** During inference, we begin with a fully masked 2D token grid, which is fed into the MLLM’s vision generation branch. The accompanying text condition is provided to the frozen MLLM branch to guide the visual token prediction. Prior to decoding, we randomly sample a generation order over the image patch tokens, which determines the sequence in which masked tokens are decoded into clean ones. Following MaskGIT [2], we adopt a cosine schedule to determine how many tokens are unmasked at each decoding step. This schedule, as visualized in Figure 2 of the MaskGIT paper, allocates fewer unmasked tokens in the early stages (when uncertainty is high), and progressively more in later steps as more tokens become available. In our default implementation, we set the number of decoding steps equal to the number of visual tokens for simplicity.\n\n- **Single step decoding v.s. multistep decoding:** In Table 1 of our main paper, we present an ablation study in which patch-level CLIP latents are replaced by 2D learnable query tokens, predicted in a single forward pass. This setup effectively performs a single-step autoregressive prediction and leads to significantly worse generation quality, with an FID of 118.69, compared to our default approach which achieves 25.77.\n\n- **Quantitative Results with Varying Decoding Steps:** To further assess the effect of decoding steps, we conducted experiments with varying numbers of decoding passes on ImageNet. The results, summarized in the table below, show image quality metrics including FID, sFID, and IS. As demonstrated in the table below, our method remains robust as long as the number of decoding steps is greater than 8. There is, however, a clear trade-off between inference speed and image quality when using fewer decoding steps. Importantly, we also highlight that the MLLM decoding time is significantly smaller than the runtime of the diffusion-based image generation model. For instance, FLUX, a 12B parameter model, takes **14.79** seconds to generate a single image with the default 28 denoising steps. Therefore, the MLLM decoding time is not a major bottleneck. From a practical standpoint, users can flexibly select the number of decoding steps based on their specific priorities—whether they favor faster inference or higher image quality for their particular application.\n\n[2] Chang, Huiwen, et al. \"Maskgit: Masked generative image transformer.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n| Number of decoding steps | 64 (default) | 32 | 16 | 8 | 4 | 2 | 1 |\n| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |\n| Clock time | 5.21s | 2.63s | 1.37s | 0.66s | 0.35s | 0.19s | 0.09s |\n| FID (↓) | 18.64 | **18.49** | 18.79 | 18.89 | 19.97 | 24.90 | 60.35 |\n| sFID (↓) | 43.64 | **43.58** | 44.52 | 45.89 | 50.86 | 67.23 | 217.50 |\n| IS (↑) | 156.01 | **158.93** | 156.96 | 158.87 | 150.56 | 124.78 | 83.22 |\n\n--- \nDear Reviewer NRRC,\n\nOnce again, we sincerely thank you for all the comments and valuable feedback. We believe that we have addressed all the questions in depth, in particular: (1) additional experimental results comparing Bifrost-1 with other T2I models on standard quantitative metrics, (2) evaluation of the model’s ability to handle complex prompts beyond GenEval and DPGBench, and (3) inference speed of the autoregressive masking process, including experiments with fewer decoding steps. If the reviewer has any additional questions, please feel free to let us know, and we would be more than happy to address them."}, {"review_id": "KBMTMW2by1", "reviewer": "Reviewer_98F3", "summary": "This paper proposes Bifrost-1, which bridges the MLLMs with diffusion models by patch-level CLIP latents to make the MLLMs generate images. The core innovation is to use 2D patch-level CLIP latents as the communicative medium, instead of 1D tokens. The original MLLM is duplicated into a two-branch network, in which the new branch is trained to predict patch-level CLIP embeddings. The diffusion model is finetuned by ControlNet to decode the generated visual token into an image. Experiments on understanding and generation benchmarks have shown competitive results.", "questions": "1. Due to the core innovation of 2D patch-level CLIP latents appearing to overlap significantly with MetaMorph, the authors may clarify what the significant differences are between these two methods. A clear analysis against MetaMorph would help clarify the novelty.\n\n2. Since the MLLMs already have understanding capabilities, the core evaluation should be image generation. Therefore, I suggest moving the generation benchmark results from the appendix into the main paper to emphasize their importance", "limitations": "yes", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. Bifrost-1 proposes to use 2D patch-level CLIP latents as the communicative medium between the MLLM and diffusion models, which can convey more detailed visual information for high-quality generation.\n\n2. The network is trained efficiently and shows reasonable understanding and generation results.\n\nWeaknesses\n\n1. My main concern about this paper is its significant conceptual overlap with MetaMorph.\n\n1a) Bifrost-1 claims 2D patch-level CLIP latents as its core innovation. However, MetaMorph already uses fine-grained CLIP latents (64 tokens to regress SigLIP image embeddings), as shown in the MetaMorph paper Section 2.1: Tokenizing multimodal data. Therefore, I think the core innovation is not convincing enough.\n\n1b) Both the Bifrost-1 and MetaMorph use the continuous GT CLIP embedding to supervise the generated visual tokens, making the learning objective similar.\n\n1c) Figure 1b misrepresents MetaMorph by labeling it as using 1D image tokens. However, since MetaMorph visual tokens regress SigLIP image embeddings, it should be 2D image tokens. Moreover, the network architecture in Figure 1c is very similar to the structure in Figure 1 in MetaMorph, in which the LLM predicts fine-grained CLIP image tokens to condition the diffusion model.\n\n2. The core motivation of this paper is to enable strong image generation capabilities of MLLMs. However, the results on COCO FID, MJHQ FID, GenEval, and DPG-Bench in Table 1 in the Appendix are not as competitive as previous methods, which undermines its central claim of enabling high-quality generation.", "was_revised": true, "final_justification": "After multiple rounds of discussions, we still have fundamentally different views regarding the contributions of this paper.\n\n1. I am not convinced by the newly claimed contributions and new experiments provided in the author responses. My reasons are detailed in our previous discussions, which are well-supported by related works. Based on this assessment, I recommend rejection. I kindly ask the Area Chairs to read these discussions before making a decision.\n\n2. Even if I were to tentatively accept the newly claimed contributions **(which I do not)**, they differ significantly from the original central claim of \"Patch-level CLIP Latents\". The current paper is structured around that original claim. Accepting it would require a **substantial rewrite** of the paper to properly reflect its true contributions. Such a situation is not suitable for a conference submission, as conferences do not provide a mechanism to supervise or ensure this level of rewriting.\n\nGiven these points, I do not believe the paper is ready for publication at a top-tier venue such as NeurIPS.", "rebuttal": "We would like to sincerely thank the Reviewer for all the valuable feedback. In the rebuttal below, we use “W1/W2” to represent the answer to the bullet points in weaknesses, and “Q1/Q2” to represent the answer to the questions.\n\n> ### W1/Q1: Comparison with MetaMorph\n\nWe sincerely thank the reviewer for raising this important comparison with the related work, MetaMorph. We fully agree that both MetaMorph and our method share the high-level goal of developing a unified framework that decodes LLMs/VLMs outputs with diffusion models. However, we would like to kindly emphasize several key differences between the two approaches:\n\n- **Image Prior Injection Strategy:** Our method directly injects 2D image tokens generated by the MLLM into the DiT latents via a ControlNet-based architecture. By adding 2D ControlNet latents on top of the noisy DiT latents, our method can enforce spatial structures more explicitly and effectively. In contrast, MetaMorph uses cross-attention to condition on image tokens. which behaves more like a global reference image. However, such cross-attention conditioning typically provides weaker spatial guidance than ControlNet and results in lower training efficiency.\n\nTo support our claim that Latent ControlNet is more efficient than cross-attention, we conducted an additional experiment comparing our default ControlNet-based injection with a cross-attention-based variant using the same 64 MLLM-generated image tokens. The cross-attention setup unfreezes all modules in the conditioning branch, resulting in 21% trainable parameters in FLUX.1-dev, compared to only 13% in our design. Both variants were trained on ImageNet for 7 epochs under the same compute budget. As shown below, the ControlNet-based approach achieves better image generation quality (FID, sFID, IS), demonstrating its superior efficiency and effectiveness.\n\n| Condition Injection Strategy | FID (↓) | sFID (↓) | IS (↑)  |\n| :---- | :---- | :---- | :---- |\n| Latent ControlNet (Ours) | **19.21** | **47.08** | **137.25** |\n| Cross Attention  | 76.32   | 208.11 | 26.35 |\n\n- **Training Objective:** MetaMorph generates visual tokens autoregressively (AR) and then visualizes them using a diffusion head. In contrast, we adopt a discrete-diffusion/MAR [1] decoding strategy to decode from masked tokens into clean visual tokens. Specifically, during inference, we randomly sample a generation order over the image patch tokens, which determines the sequence in which masked tokens are decoded into clean ones. Following MaskGIT [2], we adopt a cosine schedule to determine how many tokens are unmasked at each decoding step. This schedule, as visualized in Figure 2 of the MaskGIT paper, allocates fewer unmasked tokens in the early stages (when uncertainty is high), and progressively more in later steps as more tokens become available. As shown in Table 1 of the MAR paper, MAR is significantly more efficient and parallelizable than standard AR, supporting our design decision to prioritize training efficiency. This training objective difference, together with the cross-attention image token injection strategy, made us to classify MetaMorph into Fig1(b). This being said, we fully agree with the reviewer's suggestion that classifying MetaMorph as a 1D image token may be inaccurate. We will update the label of Fig. 1(b) to 'Bridge w/ 1D/2D Image/Text Tokens' in the camera-ready version. Thank you again for pointing out this inaccuracy.\n\n- **MLLM-aligned encoder (Ours) vs. No pre-aligned visual encoder (MetaMorph):** Our method uses the MLLM’s native CLIP visual encoder for visual generation, while MetaMorph uses SigLIP (i.e., ViT-SO400M-14-SigLIP-384) to encode images. Our approach offers three key advantages: \n\n  - (1) Since we initialize the visual generation branch with the same parameters as the original MLLM, the features in the visual generation branch are natively aligned with the MLLM’s CLIP visual encoder.\n\n  - (2) Compared to using new visual encoders such as SigLIP or VAE, our method does not require learning additional projection layers to connect the MLLM with the visual encoder, simplifying the architecture and improving efficiency.\n\n  - (3) The MLLM’s native CLIP encoder encodes a 256×256 image into just 64 patch tokens. In contrast, ViT-SO400M-14-SigLIP-384 first upscales the image to 384×384 and encodes it into 729 patch tokens, requiring additional downsampling or compression to reduce the number of image tokens. MetaMorph uses vanilla bilinear interpolation to compress the features down to 64 tokens, which may lead to information loss. Alternatively, one would need to use learnable downsampling blocks to better preserve image information, adding architectural complexity.\n\nTo support these claims, we provide a new quantitative comparison between our default strategy (MLLM’s native CLIP visual encoder) and the SigLIP encoder. We train both variants with the same compute: 7 epochs for the MLLM vision branch and 1 epoch for ControlNet, on the ImageNet dataset at 256×256 resolution. As shown, our default strategy achieves much better image generation quality compared to the SigLIP-based approach. Additionally, Table 1 in our main paper compares our default strategy with the VAE encoder used in FLUX.1-dev. We also show the numbers for VAE encoder here for completeness.\n\n\n| Visual Encoder | FID (↓) | sFID (↓) | IS (↑) |\n| :---- | :---- | :---- | :---- |\n| MLLM’s Native CLIP Visual Encoder (Ours)  | **19.21** | **47.08** | **137.25** |\n| SigLIP (following MetaMorph)  | 274.16 | 304.94    | 2.69 |\n| VAE Encoder (from FLUX.1-Dev) | 284.51 | 361.45 | 1.11 |\n\n- **Parameter-efficient training (Ours) vs. full-finetuning (MetaMorph):** We insert lightweight ControlNet blocks (13% of FLUX.1-dev) into a frozen DiT, keeping all pretrained weights untouched. In contrast, MetaMorph needs to fine-tune DiT to adapt to new visual tokens, which becomes especially challenging for video generation tasks requiring high fidelity and temporal consistency.\n\n- **MLLM Freezing Strategy:** We keep the MLLM fully frozen to preserve its reasoning and alignment abilities. In contrast, methods like MetaMorph and JanusPro [3] fine-tune the MLLM to enable token generation, risking degraded performance without massive, high-quality data. As shown in Table 3 of the JanusPro paper, their multimodal understanding performance is much lower than that of MetaQuery and BLIP3o, both of which keep the backbone frozen. This supports our design choice to freeze the MLLM.\n\n- **Performance & Reproducibility:** In addition to the attached code in the supplementary material, we have made our model design, training data, and code fully transparent, and will open-source all components upon acceptance. In contrast, MetaMorph is still under legal review, has not released model weights, and lacks evaluation results on standard benchmarks like GenEval or DPGBench, making direct comparisons difficult. We also highlight that our method achieves performance close to SoTA models like MetaQuery and BLIP3o, both generally stronger baselines than MetaMorph, while requiring much lower training cost.\n\nOverall, we have done our best to objectively clarify the differences between our method and MetaMorph, providing detailed explanations and additional experiments to support our claims. We also highly value the contributions of MetaMorph. We hope this explanation helps illustrate the key technical and practical advantages of our method. If the reviewer has any further questions or suggestions, we would be more than happy to address them. Thanks again!\n\n[1] Autoregressive image generation without vector quantization, NeurIPS, 2024\n\n[2] Maskgit: Masked generative image transformer, CVPR, 2022\n\n[3] Janus-pro: Unified multimodal understanding and generation with data and model scaling, arXiv, 2025\n\n--- \n\n> ### W2: Additional quantitative scale-up results\n\nBeyond the results presented in the appendix, we obtained additional GPU resources after the submission deadline and conducted larger-scale experiments using the same training dataset as BLIP3-o [4], which is a recent SoTA method that connects MLLMs with diffusion models for image generation (released on May 14, 2025). Specifically, we use 27 million image-text pairs from the BLIP3o-Long-Caption dataset and train our model for 1.7 epochs, followed by fine-tuning on 60K high-quality images from the BLIP3o-60K dataset. In contrast, BLIP3-o is trained on the same 27M dataset for over 10 epochs. A comparison is summarized in the table below:\n\n| Models | MLLM Backbone | Total Pretraining Images (Training Steps×Batch Size) | GenEval | DPGBench |\n| :---- | :---- | :---- | :---- | :---- |\n| BLIP3-o | Qwen2.5-VL 3B | \\~270M | 81% | 79.36% |\n| BLIP3-o | Qwen2.5-VL 7B | \\~270M | 84% | 81.60% |\n| Bifrost-1 (Ours) | Qwen2.5-VL 3B | 9M | 61% | 76.41% |\n| Bifrost-1 (Ours) | Qwen2.5-VL 7B | 46M | 81% | 77.67% |\n\nAs shown, the GenEval score improves from 61% to 81%, and DPGBench increases from 76.41% to 77.67%, demonstrating that better annotations and scaled-up training lead to meaningful gains. Notably, even with this larger-scale setup, our training cost remains much lower than prior works, while still outperforming existing approaches under the same compute budget.\n\n[4] Blip3-o: A family of fully open unified multimodal models—architecture, training and dataset, arXiv, 2025.\n\n--- \n\n> ### Q2: Paper restructure suggestion\n\nThank you very much for the suggestion. We will move the generation benchmark table from the appendix, along with the new scaled-up experiment results, into the main paper to better emphasize their importance in the camera-ready version.\n\n--- \n\nDear Reviewer 98F3,\n\nOnce again, we sincerely thank you for all the comments and valuable feedback. We believe that we have addressed all the questions in depth. If there’s any additional questions, please feel free to let us know, and we would be more than happy to address them."}, {"review_id": "n4hIShz7MM", "reviewer": "Reviewer_k2H1", "summary": "This paper proposes a framework that bridges pretrained MLLM and diffusion models, enabling high-fidelity image generation with strong training efficiency. By preserving the original MLLM parameters, it maintains multimodal reasoning capabilities while achieving competitive performance.", "questions": "What happens to generation fidelity and diversity as the number of CLIP tokens increases? In the case of VAE representations, increasing the number of tokens often improves reconstruction but can harm the diversity of generated samples. I wonder if similar effects occur here.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n- The overall performance is solid, and the experimental setup is well-structured. The experiments convincingly support the effectiveness of the proposed method.\n- Extensive ablation studies and benchmark evaluations clearly demonstrate the contributions of the proposed modules.\n- The design choices are well justified through ablation studies, including comparisons between CLIP latents and other alternatives like VAE or learnable 2D query tokens.\n\n# Weaknesses\n- Bifrost-1 incorporates a latent ControlNet to bridge patch-level CLIP latents into the diffusion model, but the motivation for introducing a new latent ControlNet structure is not entirely clear. For instance, could an alternative have been to let the MLLM generate a modality (e.g., depth, sketch) that could be consumed by an existing pretrained ControlNet? While this might compromise reconstruction quality, how would it affect generation fidelity?\n- Although the paper emphasizes the training efficiency of the proposed method, it would be better to include a quantitative comparison of training costs with existing baselines.\n- The explanation of some internal components (e.g. vision generation branch) is lacking. A more detailed architectural description would help clarify how each part contributes to the overall system.", "was_revised": true, "final_justification": "I appreciate authors for their rebuttal. The author addresses most of my concerns so I would like to keep my positive stance on this paper.\nBut for W1, I understand author's claim, but I hope there could be better if support experiments or evidence are provided.", "rebuttal": "We would like to sincerely thank the Reviewer for all the valuable feedback. In the rebuttal below, we use “W1/W2/W3” to represent the answer to the bullet points in weaknesses, and “Q1” to represent the answer to the questions.\n\n\n\n> ### W1: Could an alternative have been to let the MLLM generate a modality (e.g., depth, sketch) that could be consumed by an existing pretrained ControlNet? While this might compromise reconstruction quality, how would it affect generation fidelity \n\nThank you very much for bringing up this interesting question! Indeed, one could allow the MLLM to generate an intermediate modality such as depth or sketch, and then reuse a pretrained ControlNet together with an image generation model to render the final image. However, this approach may have several limitations:\n\n- First, the information encoded in a depth or sketch image may be less comprehensive than what is captured by MLLM-generated latents (e.g., the 2048-dimensional CLIP latents from Qwen2.5-VL 3B). Such MLLM-aligned CLIP visual encoder makes it easier to capture richer patch-level image semantics compared to the structural cues available in depth or sketch images. For example, the latents can encode more detailed semantics, including object layouts, colors, textures, and shapes—going beyond simple depth or edge information.\n\n- Second, using a pretrained depth/sketch ControlNet can be more error-prone than training a latent-based ControlNet tailored to the MLLM outputs. This is because the depth or sketch images generated by the MLLM might be imperfect. As a result, strictly following such noisy guidance using a pretrained ControlNet can lead to suboptimal image quality. In contrast, training a ControlNet directly on MLLM-generated latents offers the flexibility to learn how to interpret and leverage those more-or-less noisy latents more effectively than relying on fixed modality-specific guidance. In fact, this is precisely one of the main reasons why we connect the MLLM with DiT through ControlNets. We aim to reduce the burden on the MLLM to generate highly accurate images, as high-level semantics or “blurry” latents are already sufficient to guide the image generation model.\n\n---\n\n> ### W2: Quantitative comparison of training costs with existing baselines\n\nThank you very much for the valuable question regarding training cost. In addition to the comparison of different architectures under the same compute budget in Table 1 of our main paper, we would like to highlight that training cost comparisons with other existing baselines are also provided in Appendix Table 1 (available in the PDF of the supplementary materials, as mentioned in L284-285 of our main paper). In Appendix Table 1, training cost is measured by the total amount of data used, calculated as the product of the number of training steps and the batch size per step. As shown, our method achieves performance comparable to baselines such as Janus and EMU, while requiring significantly less training data (9M for Ours vs. 100M for Janus).\n\nBeyond the results presented in the appendix, we gained more GPU support after the paper submission deadline, and here we include additional larger-scale experiments trained on the same training dataset from BLIP3-o [1], one of the latest works that connects MLLM with diffusion models for image generation released on May 14, 2025. Specifically, we use 27 million image-text pairs from BLIP3o-Long-Caption dataset, and train the model for 1.7 epochs, followed by fine-tuning on 60k high-quality images in the BLIP3o-60K dataset. In contrast, BLIP3-o is trained on the same 27M dataset for over 10 epochs. We summarize the comparison in the following table:\n\n| Models | MLLM Backbone | Total Pretraining Images (Training Steps×Batch Size) | GenEval | DPGBench |\n| :---- | :---- | :---- | :---- | :---- |\n| BLIP3-o | Qwen2.5-VL 3B | \\~270M | 81% | 79.36% |\n| BLIP3-o | Qwen2.5-VL 7B | \\~270M | 84% | 81.60% |\n| Bifrost-1 (Ours) | Qwen2.5-VL 3B | 9M | 61% | 76.41% |\n| Bifrost-1 (Ours) | Qwen2.5-VL 7B | 46M | 81% | 77.67% |\n\nAs shown, the GenEval score improves significantly from 61% to 81%, and the DPGBench score increases from 76.41% to 77.67%. This demonstrates that higher-quality training data with better annotations, as well as scaled-up training, can lead to meaningful performance gains. Lastly, we would like to emphasize that even with this larger-scale training, our training cost remains much lower than that of prior works, and our method still outperforms previous approaches under the same training budget.\n\n[1] Chen, Jiuhai, et al. \"Blip3-o: A family of fully open unified multimodal models—architecture, training and dataset.\" arXiv preprint arXiv:2505.09568 (2025).\n\n---\n\n> ### W3: A more detailed architectural description\n\nThanks for your question. Here we explain our architecture in more detail. \n\n- **Adding a vision generation branch to the MLLM:** Starting from a pretrained MLLM (e.g., Qwen2.5-VL 3B), we create a copy of its parameters (e.g., attention layers, FFN layers) to serve as the visual generation branch. This branch is initialized with the same pretrained weights as the original model. The original MLLM parameters are kept fully frozen, and only the new vision generation branch is trainable.\n\n- **Encoding images using the MLLM’s native visual encoder:** Unlike prior works that use a VAE or CLIP/SigLIP-based encoder to process images, we utilize the MLLM’s native visual encoder during training to encode the images. As shown in Table 1 of our main paper, using the MLLM’s native visual encoder, rather than a VAE as the image encoder, results in significantly better image generation quality under the same computation budget.\n\n- **Masked autoregression (MAR) for visual token prediction:** During training, we apply MAR to teach the model to predict masked visual tokens. Specifically, the training image is first encoded using the MLLM’s visual encoder. Then, a random value α is sampled, and α × (number of visual tokens) are randomly masked. The text condition is passed into the frozen MLLM branch to guide visual token prediction. The model is trained to reconstruct the masked tokens, with the training objective being the mean squared error (MSE) loss between the predicted and ground truth visual tokens in the masked regions.\n\n- **Latent ControlNet for image decoding:** The predicted visual tokens are treated as 2D visual cues or image sketches, which are then passed to a latent ControlNet to generate the final image. The overall architecture closely follows the FLUX.1-Dev ControlNet design, but we make it more lightweight by using only 4 MM-DiT (DoubleStream) blocks and 1 Single-DiT (SingleStream) block (i.e., num_double_layers=4, num_single_layers=1). The training objective matches that of FLUX DiT, using a flow-matching noise scheduler to sample timesteps and predict the flow direction from noise to data.\n\n- **Inference process:** During inference, we begin with a fully masked 2D token grid, which is fed into the MLLM’s vision generation branch. The accompanying text condition is provided to the frozen MLLM branch to guide the visual token prediction. Prior to decoding, we randomly sample a generation order over the image patch tokens, which determines the sequence in which masked tokens are decoded into clean ones. Following MaskGIT [2], we adopt a cosine schedule to determine how many tokens are unmasked at each decoding step. This schedule, as visualized in Figure 2 of the MaskGIT paper, allocates fewer unmasked tokens in the early stages (when uncertainty is high), and progressively more in later steps as more tokens become available. Finally, these decoded tokens are then passed to the latent ControlNet as visual cues to guide final image generation.\n\n\n[2] Chang, Huiwen, et al. \"Maskgit: Masked generative image transformer.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n---\n\n> ### Q1: Generation fidelity and diversity as the number of CLIP tokens increases\n\nThank you again for raising this interesting question! In our current implementation, we do not use additional modules or loss functions to explicitly encourage diversity in the generated objects. Therefore, the observation from VAEs, where diversity decreases as the number of tokens increases, may also apply to our case.\n\nWhile this paper primarily focuses on how to effectively connect MLLMs with diffusion models, we fully agree that promoting diversity in the generated outputs is an important and orthogonal direction worth exploring. We are aware of existing works that address this issue [3], and it would be interesting to investigate whether such methods can be adapted to our architecture. We leave this for future work.\n\n\n[3] Gu, Jiatao, et al. \"Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling.\" Advances in Neural Information Processing Systems 37 (2024): 5498–5527.\n\n\n---\n\nDear Reviewer k2H1,\n\nOnce again, we sincerely thank you for all the comments and valuable feedback. We believe that we have addressed all the questions in depth, including (1) explanation of our design choice of generating image latents and the use of latent ControlNet, (2) quantitative comparison of training costs with existing baselines, (3) a more detailed architectural description, and (4) discussion of the influence of generation fidelity and diversity as the number of CLIP tokens increases. If the reviewer has any additional questions, please feel free to let us know, and we would be more than happy to address them."}, {"review_id": "uwkspmg9Pj", "reviewer": "Reviewer_XePf", "summary": "This paper proposes a novel framework for a unified generation-understanding task model. The framework is built upon a pretrained MLLM and a diffusion-based image generator, with the addition of two modules: a Vision Generation branch and a patch-level latents ControlNet. The authors claim that this framework preserves the reasoning capabilities of the MLLM while also maintaining training efficiency.", "questions": "* As indicated in lines 203-210, why were different training steps used in the experiments in sections 5.1-5.3?\n\n* Are there any additional scale-up results, such as using Qwen2.5-VL 7B as the base model?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n* The experiments show that the proposed architecture injects image generation capabilities with a relatively small training cost while maintaining the MLLM inference ability.\n\n* The paper is well-written and easy to understand.\n\n## Weakness\n\n* The main difference between the proposed method and the one in Fig. 1(b) seems to be the change from 1D tokens to 2D CLIP tokens for the input diffusion model's latent. This appears somewhat trivial. Could you elaborate on the differences between the method proposed in this paper and the one in Fig. 1(b)?\n\n* In Table 2, the proposed method and MetaQuery-L seem to have comparable performance, with no significant improvement in performance.", "was_revised": true, "final_justification": "I would like to thank the authors for the detailed response. Some of my concerns have been addressed, and I will keep my score unchanged.", "rebuttal": "We would like to sincerely thank the Reviewer for the feedback.\n\n> ### W1: The main difference between the proposed method and the one in Fig. 1(b)\n\nWe sincerely thank the reviewer for raising this important comparison with the related works in Fig.1 (b). Below are several key differences between our method and the methods under Fig.1 (b):\n\n- **Image Prior Injection Strategy:** The key difference between our method and those in Fig. 1(b) is that we directly inject 2D image tokens from the MLLM into DiT latents via a ControlNet-based architecture, enabling stronger and more explicit spatial conditioning. In contrast, Fig. 1(b) methods use image/text tokens (e.g., MetaMorph, MetaQuery, DreamLLM) or bounding box layouts (e.g., Layout-GPT, VPGen), typically injecting information via cross-attention, which offers weaker spatial guidance and lower training efficiency.\n\nTo support our claim that Latent ControlNet is more efficient than cross-attention, we conducted an additional experiment comparing our default ControlNet-based injection with a cross-attention-based variant using the same 64 MLLM-generated image tokens. The cross-attention setup unfreezes all modules in the conditioning branch, resulting in 21% trainable parameters in FLUX.1-dev, compared to only 13% in our design. Both variants were trained on ImageNet for 7 epochs under the same compute budget. As shown below, the ControlNet-based approach achieves better image generation quality (FID, sFID, IS), demonstrating its superior efficiency and effectiveness.\n\n| Condition Injection Strategy | FID (↓) | sFID (↓) | IS (↑)  |\n| :---- | :---- | :---- | :---- |\n| Latent ControlNet (Ours) | **19.21** | **47.08** | **137.25** |\n| Cross Attention  | 76.32   | 208.11 | 26.35 |\n\n- **Training Objective:** Some methods in Fig. 1(b), such as DreamLLM and MetaMorph, generate visual tokens autoregressively (AR) and visualize them using a diffusion head. In contrast, we adopt a discrete diffusion / MAR [1] decoding strategy, where masked tokens are iteratively decoded into clean tokens. During inference, we randomly sample a generation order over image patches and follow a MaskGIT-style [2] cosine schedule to progressively unmask tokens. This schedule, as visualized in Figure 2 of the MaskGIT paper, allocates fewer unmasked tokens in the early stages (when uncertainty is high), and progressively more in later steps as more tokens become available. As shown in Table 1 of the MAR paper, MAR is significantly more efficient and parallelizable than standard AR, supporting our design decision to prioritize training efficiency.\n\n- **MLLM-aligned encoder (Ours) vs. No pre-aligned visual encoder:** Our method uses the MLLM’s native CLIP visual encoder for visual generation, while most methods in Fig. 1(b) use non–pre-aligned visual encoders such as SigLIP or VAE. Our approach offers three key advantages:\n\n  - (1) Since we initialize the visual generation branch with the same parameters as the original MLLM, its features are natively aligned with the MLLM’s CLIP visual encoder, which significantly accelerates training.\n\n  - (2) Unlike methods that adopt new visual encoders (e.g., SigLIP or VAE), our approach does not require additional projection layers to connect the MLLM with the visual encoder, simplifying the architecture and improving efficiency.\n\n  - (3) The MLLM’s native CLIP encoder encodes a 256×256 image into just 64 patch tokens. In contrast, non–pre-aligned encoders like SigLIP (e.g., ViT-SO400M-14-SigLIP-384) upscale the image to 384×384 and produce 729 patch tokens, requiring further downsampling or compression. This adds complexity and may lead to information loss.\n\nTo validate our claims, we compare our default strategy (using the MLLM’s native CLIP visual encoder) with the SigLIP encoder under the same compute budget (7 epochs for the MLLM vision branch and 1 epoch for ControlNet) on ImageNet at 256×256 resolution. Our method consistently outperforms the SigLIP-based variant in image quality. For completeness, we also include results with the VAE encoder from FLUX.1-dev, as shown in Table 1 of our main paper, which further highlights our approach’s efficiency and performance advantage.\n\n| Visual Encoder | FID (↓) | sFID (↓) | IS (↑) |\n| :---- | :---- | :---- | :---- |\n| MLLM’s Native CLIP Visual Encoder (Ours)  | **19.21** | **47.08** | **137.25** |\n| SigLIP (e.g., MetaMorph) | 274.16 | 304.94    | 2.69 |\n| VAE Encoder (from FLUX.1-Dev) | 284.51 | 361.45 | 1.11 |\n\n- **Parameter-efficient training (Ours) vs. full-finetuning:** We insert lightweight ControlNet blocks (13% of FLUX.1-dev) into a frozen DiT, keeping all pretrained weights untouched. In contrast, methods in Fig. 1(b) often fully fine-tune DiT to adapt to new visual tokens, which becomes especially challenging for video generation tasks requiring high fidelity and temporal consistency.\n\n- **MLLM Freezing Strategy:** We keep the MLLM fully frozen to preserve its reasoning and alignment abilities. In contrast, methods like MetaMorph and JanusPro [3] fine-tune the MLLM to enable token generation, risking degraded performance without massive, high-quality data. As shown in Table 3 of the JanusPro paper, their multimodal understanding performance is much lower than that of MetaQuery and BLIP3o, both of which keep the backbone frozen. This supports our design choice to freeze the MLLM.\n\nWe hope this helps clarify our technical distinctions from Fig. 1(b). We’re happy to provide further clarification if needed. Thank you again!\n\n[1] Autoregressive image generation without vector quantization, NeurIPS, 2024\n\n[2] Maskgit: Masked generative image transformer, CVPR, 2022.\n\n[3] Janus-pro: Unified multimodal understanding and generation with data and model scaling, arXiv, 2025.\n\n---\n\n> ### W2: The proposed method and MetaQuery-L seem to have comparable performance on multimodal understanding tasks\n\nThank you for this insightful observation! Indeed, the concurrent work MetaQuery and our method achieve the same scores on multimodal understanding tasks, as both use Qwen2.5-VL as the base model and keep it frozen during training. Specifically, MetaQuery uses query tokens to extract semantic information from the frozen multimodal LLM, while our method creates a trainable copy as the vision branch, sharing the same visual encoder. As a result, both approaches are expected to perform similarly on understanding tasks. We will clarify this point in the camera-ready version. Thank you again for highlighting this important detail.\n\n---\n\n> ### Q1: Clarification on the experiments in sections 5.1-5.3\n\nThank you very much for raising this question. Sections 5.1 and 5.2 are conducted on the ImageNet dataset at 256×256 resolution, which contains approximately 1.2M images across 1,000 classes. Specifically, Section 5.1 focuses on text-to-image generation, while Section 5.2 presents image reconstruction experiments. All experiments in these sections are trained on a single GB200 GPU.\n\nIn Section 5.3, we scale up to open-world image-text pairs, using a total of 5M samples. Accordingly, we increase the training resources to 8 H100 GPUs instead of a single GB200.\n\nAdditionally, we observe that our latent ControlNet typically converges much faster than the vision branch in multimodal LLMs due to its lightweight design. As a result, we allocate a smaller training budget to ControlNet compared to the MLLM vision branch.\n\n---\n\n> ### Q2: Additional scale-up results\n\nThank you very much for the valuable question regarding training cost. In addition to the comparison of different architectures under the same compute budget in Table 1 of our main paper, we would like to highlight that training cost comparisons with other existing baselines are also provided in Appendix Table 1 (available in the PDF of the supplementary materials, as mentioned in L284-285 of our main paper). In Appendix Table 1, training cost is measured by the total amount of data used, calculated as the product of the number of training steps and the batch size per step. As shown, our method achieves performance comparable to baselines such as Janus and EMU, while requiring significantly less training data (9M for Ours vs. 100M for Janus).\n\nBeyond the results presented in the appendix, we obtained additional GPU resources after the submission deadline and conducted larger-scale experiments using the same training dataset as BLIP3-o [4], which is a recent SoTA method that connects MLLMs with diffusion models for image generation (released on May 14, 2025). Specifically, we use 27 million image-text pairs from the BLIP3o-Long-Caption dataset and train our model for 1.7 epochs, followed by fine-tuning on 60K high-quality images from the BLIP3o-60K dataset. In contrast, BLIP3-o is trained on the same 27M dataset for over 10 epochs. A comparison is summarized in the table below:\n\n| Models | MLLM Backbone | Total Pretraining Images (Training Steps×Batch Size) | GenEval | DPGBench |\n| :---- | :---- | :---- | :---- | :---- |\n| BLIP3-o | Qwen2.5-VL 3B | \\~270M | 81% | 79.36% |\n| BLIP3-o | Qwen2.5-VL 7B | \\~270M | 84% | 81.60% |\n| Bifrost-1 (Ours) | Qwen2.5-VL 3B | 9M | 61% | 76.41% |\n| Bifrost-1 (Ours) | Qwen2.5-VL 7B | 46M | 81% | 77.67% |\n\nAs shown, the GenEval score improves from 61% to 81%, and DPGBench increases from 76.41% to 77.67%, demonstrating that better annotations and scaled-up training lead to meaningful gains. Notably, even with this larger-scale setup, our training cost remains much lower than prior works, while still outperforming existing approaches under the same compute budget.\n\n[4] Blip3-o: A family of fully open unified multimodal models—architecture, training and dataset, arXiv, 2025.\n\n---\n\nDear Reviewer XePf, \n\nWe sincerely thank you for all the comments and valuable feedback. We believe that we have addressed all the questions in depth. If you have any additional questions, please feel free to let us know, and we would be more than happy to address them."}]

Author rebuttal:

Meta-review: The paper presents a framework that bridges MLLMs and diffusion models using patch-level CLIP latents, a lightweight autoregressive visual branch, and a Latent ControlNet. By freezing the MLLM backbone and training only small auxiliary modules, the approach preserves reasoning ability while enabling efficient and competitive image generation. Experiments show favorable trade-offs among fidelity, reasoning retention, and compute efficiency.

The AC has carefully read the paper, the reviews, the rebuttal, and all post-rebuttal discussions. The AC also appreciates the high-quality, in-depth exchange between the reviewers and authors, which clarified key issues around novelty, positioning, and empirical validation. Reviewer 98F3 raised important concerns about conceptual overlap with prior efforts (e.g., MetaMorph) and the reliance on existing components such as CLIP and ControlNet. The AC highly values this perspective, but after weighing the rebuttal and subsequent discussion, the AC finds that the contribution is not merely incremental. While several modules build on known ideas, their use here is well-motivated toward enabling patch-level CLIP latents to serve as an effective bridge. Rather than being incidental add-ons, these designs are central to making the CLIP-latent interface practical and efficient, consistent with the core positioning of the work. From this perspective, the integration is more significant than the critique suggested. In addition, the other reviewers were overall supportive: they highlighted the practicality of using patch-level CLIP latents as a bridging mechanism, the elegance of freezing the backbone to preserve reasoning ability, the breadth of evaluation, and the strong efficiency benefits relative to comparable systems. Taking all perspectives together, the AC concludes that—even if individual pieces have precedents—the integrated framework is coherent, efficient, and valuable. The balance of evidence supports acceptance.

The AC also suggests that the authors acknowledge related efforts such as X-VILA, which similarly addresses visual information loss in cross-modality alignment; this is a useful connection but not a key weakness of the submission. For the camera-ready. Please (i) incorporate all new results and analyses introduced during rebuttal into the main paper with clear positioning; (ii) refine the novelty discussion and related-work section to clearly delineate differences from MetaMorph and adjacent bridging approaches, and add missing citations; (iii) consolidate ablations/design justifications for key components and report training/inference costs in a reproducible manner; and (iv) briefly discuss limitations and the most promising paths for improvement.
Decision: Accept (poster)