Title: Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal LLM

Unified Multimodal LLM

Diffusion Models

Autoregressive Priors

Image Generation
Abstract: There is growing interest in integrating high-fidelity visual synthesis capabilities into large language models (LLMs) without compromising their strong reasoning capabilities. Existing methods that directly train LLMs or bridge LLMs and diffusion models usually suffer from costly training since the backbone LLMs have not seen image representations during pretraining. We present Bifrost-1, a unified framework that bridges pretrained multimodal LLMs (MLLMs) and diffusion models using patch-level CLIP image embeddings as latent variables, which are natively aligned with the MLLM's CLIP visual encoder. These patch-level image embeddings are integrated into the diffusion model with a lightweight adaptation of its ControlNet. To retain the original multimodal reasoning capabilities of MLLMs, we equip the MLLM with a visual generation branch initialized from the original MLLM parameters when predicting the patch-level image embeddings. By seamlessly integrating pretrained MLLMs and diffusion models with patch-level CLIP latents, our framework enables high-fidelity controllable image generation with significant training efficiency. Our experiments demonstrate that Bifrost-1 achieves comparable or better performance than previous methods in terms of visual fidelity and multimodal understanding, with substantially lower compute during training. We also provide comprehensive ablation studies showing the effectiveness of our design choices. Project page: https://bifrost-1.github.io.


Peer reviews:
comments: SUMMARY:
This paper proposes a novel framework for a unified generation-understanding task model. The framework is built upon a pretrained MLLM and a diffusion-based image generator, with the addition of two modules: a Vision Generation branch and a patch-level latents ControlNet. The authors claim that this framework preserves the reasoning capabilities of the MLLM while also maintaining training efficiency.

STRENGTHS AND WEAKNESSES:
## Strengths

* The experiments show that the proposed architecture injects image generation capabilities with a relatively small training cost while maintaining the MLLM inference ability.

* The paper is well-written and easy to understand.

## Weakness

* The main difference between the proposed method and the one in Fig. 1(b) seems to be the change from 1D tokens to 2D CLIP tokens for the input diffusion model's latent. This appears somewhat trivial. Could you elaborate on the differences between the method proposed in this paper and the one in Fig. 1(b)?

* In Table 2, the proposed method and MetaQuery-L seem to have comparable performance, with no significant improvement in performance.

QUESTIONS:
* As indicated in lines 203-210, why were different training steps used in the experiments in sections 5.1-5.3?

* Are there any additional scale-up results, such as using Qwen2.5-VL 7B as the base model?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework that bridges pretrained MLLM and diffusion models, enabling high-fidelity image generation with strong training efficiency. By preserving the original MLLM parameters, it maintains multimodal reasoning capabilities while achieving competitive performance.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The overall performance is solid, and the experimental setup is well-structured. The experiments convincingly support the effectiveness of the proposed method.\n- Extensive ablation studies and benchmark evaluations clearly demonstrate the contributions of the proposed modules.\n- The design choices are well justified through ablation studies, including comparisons between CLIP latents and other alternatives like VAE or learnable 2D query tokens.\n\n# Weaknesses\n- Bifrost-1 incorporates a latent ControlNet to bridge patch-level CLIP latents into the diffusion model, but the motivation for introducing a new latent ControlNet structure is not entirely clear. For instance, could an alternative have been to let the MLLM generate a modality (e.g., depth, sketch) that could be consumed by an existing pretrained ControlNet? While this might compromise reconstruction quality, how would it affect generation fidelity?\n- Although the paper emphasizes the training efficiency of the proposed method, it would be better to include a quantitative comparison of training costs with existing baselines.\n- The explanation of some internal components (e.g. vision generation branch) is lacking. A more detailed architectural description would help clarify how each part contributes to the overall system.\n\nQUESTIONS:\nWhat happens to generation fidelity and diversity as the number of CLIP tokens increases? In the case of VAE representations, increasing the number of tokens often improves reconstruction but can harm the diversity of generated samples. I wonder if similar effects occur here.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Bifrost-1, which bridges the MLLMs with diffusion models by patch-level CLIP latents to make the MLLMs generate images. The core innovation is to use 2D patch-level CLIP latents as the communicative medium, instead of 1D tokens. The original MLLM is duplicated into a two-branch network, in which the new branch is trained to predict patch-level CLIP embeddings. The diffusion model is finetuned by ControlNet to decode the generated visual token into an image. Experiments on understanding and generation benchmarks have shown competitive results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Bifrost-1 proposes to use 2D patch-level CLIP latents as the communicative medium between the MLLM and diffusion models, which can convey more detailed visual information for high-quality generation.\n\n2. The network is trained efficiently and shows reasonable understanding and generation results.\n\nWeaknesses\n\n1. My main concern about this paper is its significant conceptual overlap with MetaMorph.\n\n1a) Bifrost-1 claims 2D patch-level CLIP latents as its core innovation. However, MetaMorph already uses fine-grained CLIP latents (64 tokens to regress SigLIP image embeddings), as shown in the MetaMorph paper Section 2.1: Tokenizing multimodal data. Therefore, I think the core innovation is not convincing enough.\n\n1b) Both the Bifrost-1 and MetaMorph use the continuous GT CLIP embedding to supervise the generated visual tokens, making the learning objective similar.\n\n1c) Figure 1b misrepresents MetaMorph by labeling it as using 1D image tokens. However, since MetaMorph visual tokens regress SigLIP image embeddings, it should be 2D image tokens. Moreover, the network architecture in Figure 1c is very similar to the structure in Figure 1 in MetaMorph, in which the LLM predicts fine-grained CLIP image tokens to condition the diffusion model.\n\n2. The core motivation of this paper is to enable strong image generation capabilities of MLLMs. However, the results on COCO FID, MJHQ FID, GenEval, and DPG-Bench in Table 1 in the Appendix are not as competitive as previous methods, which undermines its central claim of enabling high-quality generation.\n\nQUESTIONS:\n1. Due to the core innovation of 2D patch-level CLIP latents appearing to overlap significantly with MetaMorph, the authors may clarify what the significant differences are between these two methods. A clear analysis against MetaMorph would help clarify the novelty.\n\n2. Since the MLLMs already have understanding capabilities, the core evaluation should be image generation. Therefore, I suggest moving the generation benchmark results from the appendix into the main paper to emphasize their importance\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes BIFROST-1, a framework to bridge pre-trained Multimodal Large Language Models (MLLMs) with diffusion models for high-fidelity image generation. The core idea is to use patch-level CLIP image embeddings, which are natively aligned with the MLLM's visual encoder, as the communication medium. The MLLM is trained with a lightweight visual generation branch to predict these embeddings, which then guide a pre-trained diffusion model via a lightweight latent ControlNet. This approach aims to achieve efficient training and strong generation performance while preserving the MLLM's original reasoning capabilities by keeping its backbone frozen.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n+ Novel and Clever Bridging Mechanism: The central idea of using natively-aligned, patch-level CLIP latents as the bridge is novel and elegant. It sidesteps the more complex alignment challenges faced by other methods and provides a conceptually simple way to connect the understanding capabilities of MLLMs with the synthesis power of diffusion models.\n+ High Training Efficiency: By freezing the large MLLM and diffusion model backbones and only training lightweight components, the proposed method is highly computationally efficient. The ablation study in Table 1 effectively demonstrates the superiority of this design choice over alternatives.\n+ Preservation of MLLM Capabilities: A key advantage is that the MLLM's original parameters are untouched, ensuring its strong multimodal understanding and reasoning abilities are fully preserved. This is well-supported by the benchmark results in Table 2.\n\n### Weaknesses\n- Insufficient Quantitative Comparison on Image Generation: The paper lacks a direct quantitative comparison against other state-of-the-art text-to-image generation models on standard benchmarks (e.g., DPG-Bench or GenEval). While Figure 7 provides qualitative examples, objective metrics like FID or CLIP Score are necessary to properly situate the model's performance. The Appendix is mentioned, but this critical comparison should be in the main paper for a thorough evaluation.\n- Limited Evaluation of \"Controllability\": While the framework is presented as enabling \"controllable\" generation, the experiments primarily focus on image reconstruction and relatively simple prompts. There is a lack of validation on complex compositional prompts involving multiple objects with specific spatial relationships. This makes it difficult to assess the true extent of control the method provides.\n- Unaddressed Inference Latency: The method relies on an autoregressive masking process to generate the patch embeddings during inference. This iterative process can be slow. The paper completely omits any analysis or reporting of inference speed, which is a significant oversight for a method aiming for practical utility.\n\nQUESTIONS:\n- Could you provide standard quantitative metrics (e.g., GenEval and DPGBench) comparing BIFROST-1 with other leading text-to-image generation models like JanusPro or MetaQuery? The qualitative results in Figure 7 are promising, but objective numbers are needed.\n- Could you provide any quantitative results or qualitative examples demonstrating the model's ability to handle complex compositional prompts that require precise spatial reasoning and object attribute binding?\n- What is the inference latency of your method, particularly the autoregressive step for generating patch embeddings? How does it compare to other methods that generate guidance signals in a single forward pass?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}