I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models
Authors: Zhenxing Mi
Kuan-Chieh Wang
Guocheng Qian
Hanrong Ye
Runtao Liu
Sergey Tulyakov
Kfir Aberman
Dan Xu
Venue: ICML 2025 poster (2025)
Keywords: Diffusion model
Multimodal in-context reasoning
Vision-language model
Vision-language training
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We a novel alignment paradigm that enables multimodal in-context reasoning capabilities in text-to-image diffusion models by integrating vision-language models.

Abstract: This paper presents ThinkDiff, a novel alignment paradigm that empowers text-to-image diffusion models with multimodal in-context understanding and reasoning capabilities by integrating the strengths of vision-language models (VLMs). Existing multimodal diffusion finetuning methods largely focus on pixel-level reconstruction rather than in-context reasoning, and are constrained by the complexity and limited availability of reasoning-based datasets. ThinkDiff addresses these challenges by leveraging vision-language training as a proxy task, aligning VLMs with the decoder of an encoder-decoder large language model (LLM) instead of a diffusion decoder. This proxy task builds on the observation that the **LLM decoder** shares the same input feature space with **diffusion decoders** that use the corresponding **LLM encoder** for prompt embedding. As a result, aligning VLMs with diffusion decoders can be simplified through alignment with the LLM decoder. Without complex training and datasets, ThinkDiff effectively unleashes understanding, reasoning, and composing capabilities in diffusion models. Experiments demonstrate that ThinkDiff significantly improves accuracy from 19.2% to 46.3% on the challenging CoBSAT benchmark for multimodal in-context reasoning generation, with only 5 hours of training on 4 A100 GPUs. Additionally, ThinkDiff demonstrates exceptional performance in composing multiple images and texts into logically coherent images. Project page: https://mizhenxing.github.io/ThinkDiff.

I Think, Therefore I Diffuse:
Enabling Multimodal In-Context Reasoning in Diffusion Models
Zhenxing Mi
Kuan-Chier Wang?
Guocheng Qian Hanrong Ye' Runtao Liu
Sergey Tulyakov3
Kfir Aberman
Dan Xu
(a) Multimodal in-context reasoning generation
(b) Multimodal composing
Predict what the next image is:
monkey
cat
zebra
"The holds board copy DIFFERENT."
Ground truth reasoning answer: flying zebra
Figure 1. (a) Our ThinkDiff reasons over interleaved images (a flying monkey and a flying cat) and text prompts (monkey, cat, and zebra) to generate a logically correct and high-quality image (a flying zebra). The ground truth reasoning answer is provided as a reference for readers. (b) ThinkDiff composes images and texts into coherent and reasonable image.
Abstract
coder. Without complex training and datasets, ThinkDiff effectively unleashes understanding, reasoning, and composing capabilities in diffusion models. Experiments demonstrate that ThinkDiff significantly improves accuracy from 19.2% to 46.3% on the challenging CoBSAT benchmark for multimodal in in-conext reasoning generation, with only hours of training on 4 A100 GPUs. Additionally, ThinkDiff demonstrates exceptional performance in composing multiple images and texts into logically coherent images. Project page: https://miihenxxngggieeee github. io/hiinkDiff..
This paper presents ThinkDiff, novel alignment paradigm that empowers text-to-image diffusion models with multimodal in-context understand- ing and reasoning capabilities by integrating the strengths of vision-language models (VLMs). Ex- isting multimodal diffusion finetuning methods largely focus on pixel-level reconstruction rather than in-context reasoning, and are constrained by the complexity and limited availability of reasoning- based datasets. ThinkDiff addresses these challenges by leveraging lision-language training as a proxy task, aligning VLMs with the decoder of an encoder-decoder large language model (LLM) instead of a diffusion decoder. This proxy task builds on the observation that the LLM decoder shares the same input feature space with diffusion decoders that the corresponding LLM encoder for prompt embedding. As re- sult, aligning VLMs with diffusion decoders can be simplified through alignment with the I.IM de-
1. Introduction
Can diffusion models take "IQ tests"? Figure 1a presents example of visual analogy IQ test. The model is pro- vided with images of a flying monkey and a flying cat, along with text prompts of monkey, cat, and zebra, and asked to generate the next image. A reasonable output image should model's ability
an image of flying zebra, requiring to reason and recognize implicit patterns in context, such as the shared attribute of the flying action in this example.
'Department of Computer Science and Engineering (CSE), The Hong Kong University of Science and Technology (HKUST).
"Thi work was partially done while intemiin Snap. 'Snap Inc. Correspondence to: Zhenxing Mi < connettttt..................
The concept of enabling diffusion models to think and then generate is compelling yet underexplored. Current text-to- image diffusion models (AI, 2024c; Forest, 2024a) excel at generating high-quality images by strictly following explicit prompts, while typically lacking multimodal in-context rea-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Think, Therefore I Diffuse: Enabling Multimodal In- In-Context Reasoning in Diffusion Models
(a) Reconstruction-aaet diffusion finetuning
Encoder-decuder LLM
Text token features
Image enowder
Difitasion loss
Training & Inference
kitty licking
LLM encuder
LLM decoder
Jayfully
Alighar
Diffusion decoder
Shared feature space
A kitty licking
LLM encoder
Diffusion decoder
Text
Text encoder
Diffusion with LLM encoder
Text token features
(b) ThinkDiff vision-language training
Figure 3. Several diffusion models share a language encoder with mecoder-decoder LLMs, allowing aligning with diffu- sion decoders through aligning with LLM decoders.
LLM docoder
VIM
Aligner
Text loss
Training
poorl eooring
Third, instead of readily available image-caption pairs, it requires multimodal reasoning datasets that pair multimodal inputs with logically consistent output images and cover different reasoning tasks. Collecting such datasets is sig- nificantly more complex than captioning images. Existing instruction-guided datasets such as the synthetic Instruct- Pix2Pix (Brooks et al., 2023) dataset primarily focus on im- age editing tasks, lacking the diversity needed for reasoning- based generation tasks. Finally, finetuning diffusion models for reasoning from scratch using limited datasets constrains their performance across a broad range of reasoning tasks.
Text
What the next image? graffini
icor
Drifosiion docoder
Inference
lego
Figure 2. (a) Reconstruction--aan based diffusion finetuning integrates image features using diffusion loss, focusing on pixel-level image reconstruction without reasoning. (b) ThinkDiff aligns a VLM to an LLM decoder by vision- language training on image-caption datasets. In inference (dotted lines), it transfers multimodal in-context reasoning capabilities from the VLM to a diffusion decoder.
To tackle these challenges, we propose ThinkDiff, a novel alignment paradigm to transfer multimodal in in-context rea- soning capabilities from VLMS to diffusion models. Instead of directly aligning VLMs with a diffusion decoder, we de- sign proxy task to align VLMs with large language model (LLM) decoder by rision-language training. The foundation of this proxy task is depicted in Figure 3. Recent diffusion models (AI, 2024b; Chen et al., 2024; Forest, 2024a; AI, 2024c) have adopted the encoder of an encoder-decoder LLM (Raffel et al., 2020) as diffusion models' prompt en- coder. This shared text encoder establishes a shared input feature space for both the diffusion decoder and L.LM de- coder. Therefore, aligning VLM with diffusion decoder can be achieved by the proxy task of aligning a VLM with the LLM decoder by vasion-language training.
soning. Unlocking reasoning capabilities in them can enable them to handle more sophisticated tasks, such as interpreting complex instructions, solving visual analogy problems that require inferring implicit logic relationships, and composing multiple images and text in a logically consistent manner.
With rapid advancements in vision-language models (VLMs) such as CLIP (Radford et al., 2021) and GPT-like models (Radford et al., 2018), we now have powerful tools for advanced multimodal understanding and reasoning, This leads us to a question: can we equip diffusion models with the reasoning capabilities of VL.Ms?
Figure 2b depicts the vision-language training in ThinkDiff. The input images and text prompts are processed by a VLM and aligner network, after which they are fed into LLM decoder. The LLM decoder generates text autoregressively. supervised by cross-entropy loss against ground truth texts. After training, the VLM is aligned to the LLM decoder, and inherently to the diffusion decoder.
Existing multimodal diffusion adapters (Zhang et al., 2023; Ye et al., 2023; Mou et al., 2024) primarily rely on reconstruction--assel diffusion finetuning to incorpo- rate visual conditions into text-to-image diffusion mod- els. Figure 2a illustrates the typical training pipeline of IP-Adapter (Ye et al.. 2023), where the model is finetuned to replicate input images at the pixel level. While effec- tive for pixel-level control and high-fidelity image genera- tion, adapting this finetuning paradigm to support in-context reasoning introduces several challenges. First, this mul- timodal finetuning primarily focuses on lixel-level recon- struction of explicit image inputs rather than performing multimodal reasoning based on input context. Second, the pixel-level reconstruction training does not focus on aligning vision representations with the textual feature space, limiting the model's s ability to reason effectively across modalities.
Our method offers several advantages. First, it fully lever- ages the multimodal in-context understanding and reasoning capabilities of VLMs without requiring expensive training from scratch. Second, by aligning multimodal features to the input space of the L.M decoder through fine-grained text supervision, the model effectively captures rich se- mantic details from multimodal inputs, enabling seamless collaboration between vision and text modalities. Finally, TifnkDiff lightweight, efficient and highly versatile. The
2

---

Think, Therefore I Diffuse: Enabling Multimodal In- In-Context Reasoning in Diffusion Models
vision-language training in it only requires readily avail- able image-caption pairs, eliminating the need for complex reasoning- based datasets while achieving remarkable in- context reasoning capabilities.
inspired unified models for multimodal understanding and generation. These models either finetune LLMs gener- ate image tokens, which are then decoded into images via diffusion decoders (Ge et al., 2024; Pan et al., 2023; Sun et al., 2023; Koh et al., 2024; Wu et al., 2023; Ye et al., 2024), or integrate text, image, and noise tokens within a transformer architecture (Xiao et al., 2024; Shi et al., 2024). They are typically trained end-to-end with diffusion losses or align output image tokens with CLIP text features using cosine similarity losses (Wu et al., 2023; Ye et al., 2024; Tong et al., 2024). While some methods exhibit preliminary reasoning capabilities, these capabilities remain constrained by the limits of diffusion training paradigms, the availabil- ity of reasoning datasets, and the representational limits of CLIP embeddings. In contrast, our method leverages vision-language training to transfer advanced multimodal reasoning capabilities in VLMs to diffusion models.
This paper introduces two variants of ThinkDiff, each us- ing a different VLM. ThinkDiff-LVLM aligns generated tokens of a large vision- language model (LVLM) to diffu- sion models. CLIP vision encoder (Radford et al., 2021) to diffusion models. Our contributions are summarized as follows:
MinnDifff-CIIP aligns image tokens from
We propose ThinkDiff, novel alignment paradigm that equips diffusion models with multimodal in- context reasoning capabilities from VLMs.
ThinkDiff designs a proxy task to align VLMs into a shared feature space of both an LLM decoder and diffusion decoder by vision-language training, fully transferring VLM's reasoning capabilities to diffusion models with efficient training and simple datasets.
2.3. Vision-language Training
We address the poor convergence problem in ThinkDiff for robust feature alignment. After training for only 5 hours 4 A 100 GPUs, ThinkDiff improves state-of- the-art accuracy on the major visual in-context learning benchmark (Zeng et al., 2024) from 19.2% to 46.3 %. It also demonstrates powerful abilities to compose mul- tiple images and texts into logically coherent images.
Vision Vision-language training has proven effective in developing powerful multimodal models. CLIP-like models (Radford et al., 2021; Fang et al., 2023; Girdhar et al., 2023) use con- trastive learning to align image and text embeddings. Recent large vision- lision-language models (LVLMs)(Li et al., 2023; Liu et al., 2023; Zhu et al., 2023; AI, 2024a; Wang et al., 2024b) align CLIP visual features with advanced large language models (LLow) et al., 2020; Achiam al., 2024; AI, 2024a; Yang et al., 2024a) by fine-grained text prediction. This vision-language training enables robust multimodal fea- ture alignment, developing multimodal understanding and reasoning by leveraging powerful LLMs. Inspired by these advancements, our method employs vision- lsiionllaguuag train- ing as a proxy task to bridge VLMs with diffusion models, inheriting their advanced multimodal reasoning capabilities.
2. Related Work
2.1. Diffusion Models
Diffusion models have become powerful tools for text-to- image generation (Ho et al., 2020; Rombach et al., 2022; 2022; For- est, 2024a). Early models, e.g. Stable Diffusion (Rombach et al., 2022), use CLIP (Radford et al., 2021) for prompt embedding, while recent works integrate large language models (LLMs) (Saharia et al., 2022; Chen et al., 2024; AI, 2024c) for complex prompts. Methods such as Control- Net (Zhang et al., 2023), T21-Adapter (Mou et al., 2024), and IP-Adapter (Ye et al.. 2023) introduce structural and image-level controls by reconstruction based fine-tuning. Personalized generation has been enhanced by methods like DreamBooth (Ruiz et al., 2023), and other methods (Gal et al., 2023; Wang et al., 2024a; Li al., 2024; Wang al., 2024c; Qian et al., 2024; Wang et al., 2024d), some of which use interleaved image-text inputs (Pan et al., 2023; Berman & Peysakhovich, 2024). However, these methods focus on reconstruction fidelity rather than in-context reasoning. In contrast, our method equips diffusion models with the multimodal in-context reasoning capabilities of VLMs.
3. Method
3.1. Overview
ThinkDiff employs VLMs to enable diffusion decoders to perform multimodal in-context reasoning. This is achieved by an aligner network that bridges a VLM and a diffusion decoder. As described in Section 1, ThinkDiff simplifies the alignment process by introducing proxy task that aligns the VLM with an LLM decoder using text supervision. This task is based on the shared input feature space between the LLM decoder and diffusion decoder. Figure 2b and Fig- ure 4 illustrate the overall network structure and two model variants, respectively. The multimodal input comprises a set of images {I,} and text tokens {T,). The aligner net- work processes its input token features {a;}} into its output token features [a(]- In training, ThinkDiff generates text tokens [v/}, supervised by ground truth text tokens (v:). In inference, it generates an image l'.
2.2. Unified Understanding and Generation
Recent work on large language models (LLMs) and diffu- sion transformers (Peebles Xie, 2023; Forest, 2024a) has
3

---

Think, Therefore I Diffuse: Enabling Multimodal In- In-Context Reasoning in Diffusion Models
Training
fare Gen with poart cornin
Aligac
deander
(uC) peart sanning
Trainling
CL.P enceeee
Aligzer
LLM drooder
poart aor
cop tion
LLM encoder
Bandun mock
Inference
What
Aliona
Inference
CLIP enooder
Alignce
Difisionn decokkr
girl loughing.
(x) ThinkDiff-LVLM
(b) ThinADiIF-LIIP
Figure 4. (a) In ThinkDiff-LLLM training. the LVLM processes an image and text to generate text tokens and token features, with some token features randomly masked. Unmasked token features are passed to a trainable aligner network and an LLM decoder, predicting masked text tokens supervised by cross-entropy loss. inference, the LLM decoder is replaced by a diffusion decoder. enabling in-context reasoning image generation from interleaved images and texts. (b) In ThinkDiff-CII IL training, CLIP vision model extracts image token features which are then mapped by trainable aligner network. A part of the image caption is encoded by the LLM encoder and concatenated with image tokens. These combined tokens are passed to the LLM decoder to predict the next part of the caption supervised by cross-entropy loss. In inference, the LLM decoder is replaced by a diffusion encoder, allowing coherent image generation based on multimodal context.
Module Overview. ThinkDiff comprises three submodules: source VLM (MMIM., an aligner network (MAx).. and a decoder. The decoder is a LLM decoder (MLLMD) in training and diffusion decoder (MDinD) in inference.
is expressed as: {v/} (M By this training, VLM token features are aligned with the decoder's 's input space, transferring reasoning capabilities from the VLM to MLLMD in training and to .MDmn in inference.
Source VLM. The source VLM generates multimodal to- ken features } capturing the reasoning and understand- ing derived from multimodal inputs and transferring this information to diffusion decoders. The generation is ex- pressed as: } = MMLL(((4), (Ti)). This paper intro- duces two variants of ThinkDiff, each utilizing a different VLM. ThinkDiff -Lff- uses a large aision-languag model (LVLM) to deliver advanced multimodal reasoning capabili- ties while ThinkDifi-CLIP leverages the semantically rich image embeddings provided by a CLIP vision encoder for image understanding, Detailed descriptions of these variants can be found in Sections 3.3 and 3.4.
In inference, the L.M decoder is replaced by a diffusion decoder (MDitd)) which can interpret VLM's outputs and leverage the VLM's multimodal reasoning abilities for im- age generation. ThinkDiff can handle multiple images, texts, or interleaved sequences of images and texts during infer- ence, thanks to their shared feature space. The generated image I' is given by I' .MDDn ({=()).
Loss. We employ a cross-entropy loss between the LLM decoder's generated tokens {v'} and the ground truth text tokens {y,} in training. Let N be the length of {ı/), the loss is defined as: Lust N Σ
log plyl y:).
In the following sections, detail the design of the aligner network and two variants of ThinkDiff.
Aligner network. The aligner network bridges the source VLM with the LLM and diffusion decoder. It transforms token features {a;}, which encapsulate rich reasoning infor- mation, into {x1}- making them interpretable by the L.L and diffusion decoder. This transformation is represented as: {24) MMA((
3.2. Aligner Network
The aligner network MAN is a lightweight module compris- ing two linear layers (CLimar), GELU activation (CGELU) and an RMSNorm layer (Zhang & Sennrich, 2019) (Exomm). Given the VLM's output {x;}, the output { of MAN is:
The decoder operates differently during training and inference. The L.M decoder (MLLMD) is central to derived from an encoder-decoder LLM. In this LLM, the LLM encoder encodes token features and the LIM decoder generates text autoregressively from these token features. In ThinkD- iff training, the VLM token features {a;} are mapped to {o{} by the aligner network. The LLM decoder then treats {xt) as they were outputs from the LLM encoder and autoregressively decodes them into text (1/). This process
Decoder:
ThinkDiff's language training.
{xf} LNorm (Lrinear (CrELU (Linnur ((a))))))
(1)
In training, only MAN is updated. Despite its simplicity, MAN can effectively aligns feature spaces of the powerful VLM and the LLM decoder in the training.
Stable training, Our experiments revealed that without a carefully initialized RMSNorm layer, ThinkDiff encounters

---

Think, Therefore I Diffuse: Enabling Multimodal In- Reasoning in Diffusion Models
convergence issues due to a scale mismatch between the VLM output space and the LLM decoder input space. To ad- dress this, we incorporate an RMSNorm (Zhang & Sennrich, 2019) layer into MMN, initialized with parameters from the LLM encoder's s final RMSNNrm layer. Since the L.M en- coder output space aligns naturally with the LLM decoder input space, this initialization ensures consistent scale align- ment at the start of training, significantly improving training stability and convergence.
generated tokens from the LVLM decoder as input the aligner. This design is motivated by the insight that, in autoregressive models, reasoning is embedded in the genera- tion process. Tokens are generated sequentially, conditioned on both the input context and the prior generated tokens. As a result, the full sequence of generated tokens captures the model's logical reasoning about the input context. By align- ing these generated token features with diffusion models, ThinkDiff-LVLLM ensures that the diffusion models inherit the LVLM's s advanced multimodal reasoning capabilities.
3.3. ThinkDiffLVLMM
Inference for in-context reasoning, In inference, as shown in Figure 4a, the LLM decoder is replaced by a diffusion decoder for image generation. As shown in Figure la and 5, ThinkDiff-LLLLMM effectively leverages the LVL.M's mul- timodal in in-context reasoning capability, using the context of interleaved images {I,} and texts (T,) to generate high- quality, logically coherent images that go beyond simply reconstructing the input content. The generated image I' is:
ThinkDiff-LVLN, incorporates a decoder-only large vision- language model (LVL.M) that excels advanced in-context reasoning tasks, as its VLM. It aligns the deep features of the LVLM's generated tokens to both MIIMD and MDIn
Training. The training framework is illustrated in Figure 4a. The LVLM autoregressively generates text tokens } from an input image and text prompt T. The corresponding token features }2,} are extracted from the LVLM's final RMSNorm layer. These features {2;} are then passed to MAN and MILMDD. where they are decoded into text tokens {v(), supervised by LVLM's generated tokens {y:}. This setup is self-supervised. as both the token features } and the supervision } are all generated by the LVLM itself. This enables the aligner network to accurately transfer information from the LVLM to MLLMD and MDItID.
I' == MM ((T.), (T.)))
(3)
3.4. ThinkDiff-CLIP
ThinkDiff-CLIP employs the vision encoder of CLIP vision-language model (Radford et al., 2021) pretrained on contrastive laigunle tasks, as its VLM. This en- coder produces semantically rich image features, enabling aligned diffusion decoders to generate images based on the semantic understanding of input images.
However, in this setup, token features {I,} have one-to- one correspondence with the supervision text tokens )vi). This may cause the aligner to learn trivial mapping be- tween } and } without truly aligning features. We refer to this issuc as "shortcut mapping".
Training. Figure 4b illustrates the training framework. The model is trained to predict partial captions for an input image. The CLIP vision encoder encodes the input image I into image tokens { which are downsampled via 2D pooling to reduce token count. The aligner network then maps }2; to {a()- Meanwhile, the image caption T is randomly split into two parts: Ti and T2. The first part, Ti, is encoded into text token features [44 } by the LLM encoder. The aligned image tokens { } are concatenated with )t,), and fed to the LLM decoder, which autoregressively predicts text [v{} supervised by the second caption part T2 (tokens {u?)) D). The text generation process is formulated as:
Random masked training. To address the "shortcut map- ping" issue, we introduce random masked training strategy. In this strategy, text tokens {yr } and features {r;} are ran- domly split into two parts: {ut ). {u?) and }. (x.), }. where {w!} } correspond to {x/} } and (v?) correspond to
{x]). Only the first part (x:) } is passed to the aligner and LLM decoder, generating text tokens {v'} supervised by the second part of tokens {u?) }. This breaks the one-to-on one-to-one cor- respondence, encouraging more robust feature alignment. The generated tokens (v/} are computed as:
{ = MMLnoffaa MLLME (T))),
[u/} MMLLMD (fmask (MUTLNN
(4)
where feat denotes concatenation, and MLLMEE is the LLM encoder. The cross-entropy loss is: LALP = * I Logp(y// y²)) After training, the aligned im- age tokens (x/} capture semantic details of the input image and can be interpreted by both MLLMD and MDiID.
where fmask is the random masking and MIVIMG is the LVLM's generation process. The cross-entropy loss is: LiVem = N Σn logp(s/ = v²)
Why use LVLM's generated tokens. Some diffusion mod- els (Liu et al., 2024; Xie et al., 2024) incorporate decoder- only LLMs for prompt encoding but actually treat them as encoders by using the deep features of input tokens. In contrast, ThinkDiff-LVLL uses the deep features of the
Inference. In inference, as shown in Figure 4b, the LLM decoder is replaced by a diffusion decoder for image gener- ation. As shown in Figure 1b, 6, 8, and 13, with an image as input, ThinkDiff- CLIP preserves semantic details of this
5

---

Think, Therefore Diffuse: Enabling Multimodal In- In-Context Reasoning in Diffusion Models
Inputs
bog
car
GT answer: wicker car
SEED-LLaMA
Emu
GILL
ThinkDin--LLM (Ours)
Figure 5. 2-shot evaluation results on CoBSAT. The input structure is similar Figure la. Given multimodal inputs, ThinkDiff-LVL.M accurately captures both implicit attributes (e.g., wicker material) and explicit attributes (e.g. car), and generates a logically correct image (wicker car). In contrast, methods such as SEED-LLaMA (Ge al., 2024), Emu (Sun et al., 2023) and GILL (Koh et al., 2024) produce inaccurate and lower-quality images. The ground truth implicit attribute is highlighted in red for readers' reference. See more results in Appendix Figure 9 and 10.
Table 1. 2-shot CoBSAT accuracy of ThinkDiff-LVLM. It achieves SoTA accuracy on 9 of 10 tasks by large margins, increasing accuracy by more than 20% on Action-I, Color-Il, Action-Il tasks which are particularly hard for other methods,
Color-I
Background-1
Style-1
Action-1
Textre-1
Color-11
Background-II
Style-I1
Action-Il
Texture-II
SEED-LLaMA
0.680
0.348
0.203
0.182
0.196
0.287
0.467
0.297
0.261
0.163
Emu
0.065
0.051
0.057
0.052
0.078
0.062
0.109
0.081
0.092
0.074
GILL 0.171
0.054
0.069
0.063
0.074
0.010
0,043
0.024
0.022
0,040
ThinkDiff-LVLM
0.622
0.349
0.237
0.459
0.290
0,511
0,534
0.340
0.534
0.292
image in the generated image. With multiple input images and text prompts, it seamlessly combines them into a se- mantically coherent image, as both image and text features are well-aligned within a shared feature space. These re- sults highlight ThinkDiff-CLLn compose multimodal context. In contrast, reconstruction- based diffusion finetuming methods like FLUX Ultra (Forest, 2024a), often struggle to simultaneously adhere to image and text prompts. The generation of ThinkDiff- -LII is:
in-context reasoning capabilities of ThinkDiff-LVLM are evaluated on the challenging CoBSAT benchmark (Zeng et al., 2024) and measured by prediction accuracy. More de- tails are in its paper. We assess ThinkDif'CLIIP's reasoning and composition abilities various prompts and images from (Ruiz et al., 2023; Peng et al., 2024; Ye et al., 2023).
ability to understand and
Baselines. We compare ThinkDiff-LVLM with SEED- LLaMA (Ge et al., 2024), Emu (Sun et al., 2023) and GILLI (Koh al., 2024) that can generate images based on image and text inputs. SEED-LLaMA is the previ- ous state-of-the-art (SoTA) model on the CTBSAT bench- mark. We compare ThinkDiff-CLIP with FLUXI.I-pro- Ultra API (Forest, 2024b), which supports image generation from image and text inputs. PLUXI.1-pro-UUT is possibly finetuned by diffusion training and image reconstruction supervision, which differs fundamentally from our method.
I' = Mf (
(5)
4. Experiments
4.1. Implement Details
Base models. We use publicly available FLUX. 1-dev (For- est, 2024a) the diffusion decoder as it employs T5 (Raffel et al., 2020), an LLM, as its prompt encoder. We use the corresponding T5 decoder as .MLLDD- ThinkDiff-LLLM uses Qwen2-VI. (Wang et al., 2024b) as the VLM,, which excels at vision-language reasoning on interleaved images and texts. ThinkDiff-CI employs the vision encoder from the ViT-G/14 model EVA-CLIP (Fang et al., 2023).
4.2. Evaluation Results of ThinkDiff-LYLM
We evaluate ThinkDiff-LVLLM on the 10 multimodal in- context reasoning generation tasks in the CoBSAT, in both 2-shot and 4-shot settings. In each setting. 2 or 4 input images and corresponding texts are provided input, with additional instruction prompt to make the model generate the next image that contains the correct object and attribute, based on in-context reasoning, (see Appendix Section B). Tables and 2 report the accuracy for 2-shot and 4-shot evaluations, respectively. Results of SEED-LLaMA (Ge et al., 2024). Emu (Sun et al., 2023) and GILL (Koh et al., 2024) are token from the CoBSAT (Zeng et al., 2024) paper.
Training and evaluation resources. We use public image- caption datasets for training. ThinkDiff-LVL.M is trained for 25,000 steps on 4 A000 GPUs for 5 hours, with total batch size of 96. ThinkDiff- CLIff is trained for 100,000 steps on 4 A100 GPUs by one day, with a total batch size of 168. See Appendix B for detailed dataset settings. The multimodal
6

---

Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models
Input image
Flux Ultra I
Flux Ultra + +I+T
Ours +
Ours +++++ T+T
"There is wooden boot the water.
"There is a wooden water."
"The dog wearing cowboy
"The dog is wearing cowboy hat."
hat."
Figure 6. Generation results for single image (I) and single image with text prompt (I + T) inputs. Our method effectively integrates semantic details of both image and text modalities to produce coherent images. FLUX excels at replicating the input image but struggles to maintain consistency with additional text prompts. See more results in Figure 11.
Table 2. 4-shot CoBSAT accuracy of ThinkDiff-LLLM shows a 27% average improvement over other methods and a
4.7% increase over its 2-shot results, highlighting its ability to handle complex in-context reasoning. contrast, SEED- LLaMA (Ge et al., 2024), Emu (Sun al., 2023), and GILL (Koh et al., 2024) exhibit reduced performance in 4-shot evaluations, indicating their struggle with increased input complexity. Improvement ratios over SoTA are also provided.
Color-I
Background-1
Style-1
Action-I
Texture-1
Color-I1
Background-II
Style-11
Action-11
Texturo-11
SEED-LLaMA
0.482
0.211
0.141
0.053
0.122
0.252
0.076
0.268
0.207
0.105
Emu
0.063
0.018
0.045
0,048
0.097
0.037
0.122
0.109
0,077
0.088
GILLL
0.106
0.044
0.041
0.073
0.087
0.022
0.059
0.044
0.032
0.067
Ours
0.638
0.362
0.254
0.434
0.317
0.610
0.590
0.432
0.664
0.332
Improvement (A%)
32.4%
71.6%
80.1% %
718.9%
159.8%
142.1 %
676.3%
61.2%
220.8%
216.2%
Training loss (log scale)
80
60
w/o RMSNorm
More importantly, in the more complex 4-shot evaluation (Table 2), ThinkDiff-LV further demonstrates its supe- rior performance, outperforming all methods across every task, with an average accuracy improvement of 27%. No- tably, it also shows a consistent 4.7% accuracy increase over its 2-shot performance, highlighting its ability to effectively leverage additional complex information. In contrast, the accuracy of baselines drops significantly with 4-shot inputs, indicating their difficulties with the increased complexity of multimodal inputs. This underscores that Think not only excels in advanced in-context reasoning but also adapts more effectively complex multimodal inputs. Fig- ures 5, 9, and 10 present the qualitative comparison, where ThinkDiff-LVLM generates both correct and significantly higher-qualij images compared to other methods.
40
RMSNom w/ Default init.
Ours
20
Training steps (x5000)
Figure? 7. Training losses (log scale) of ThinkDiff-LVLM comparing different RMSNorm designs. Disabling RM- SNorm (w/o RMSNorm) or using the default RMSNorm initialization (RMSNorm w/ Default init.) results in signifi- cantly unstable training.
As shown in Table for 2-shot evaluation, ThinkDiff-LVLM achieves SoTA performance on 9 out of 10 tasks, outper- forming other methods by a large margin. Baselines like Emu and GILL perform poorly on most tasks with accuracy below 10%, reflecting the difficulty of these tasks. While SEED-LLaMA performs well on task Color-I, it underper- forms ThiM on other tasks. Notably, TiinkDiff- LVLM exceeds the previous SoTA by over 20% in accuracy on Action-I, Color-II, II, and Action-Il tasks, showcasing its superior in-context reasoning generation capabilities.
Generation quality. We further evaluate the quality of general image-conditioned generation of ThinkDiff-LLLM on 1k images in COCO (Lin et al., 2014). The model is conditioned by image in the experiment. We show the FID, the CLIP image metric (CLIP-I), and the CLIP- Score (CLIP-T) (Hessel et al., 2021) in the Table 4. Our method can achieve much better performance than exist- ing competitors such as SEED-LLAMA, Emu, and GILL. We also evaluate the general text-conditioned generation
7

---

Think, Therefore Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models
Table 3. 2-shot results on CoBSAT ablating models with and without masking, and using deep features of input tokens.
Color-I
Background-I
Style-I
Action-I
Texure-I
Color-1I
Background-II
Stylc-I1
Action-11
Texture-I1
Ours using input tnkens
0.024
0.004
0.03
0.011
0.032
0.007
0.008
0.012
0.019
0.011
Ours w/ masked training
0.548
0.215
0.105
0.256
0.187
0.510
0.338
0.156
0.325
0.228
Ours
0.622
0,349
0.237
0.459
0.290
0,511
0.534
0.340
0.534
0.292
Table 4. Image-conditioned generation quality of Diff
LVLM on CLIP metrics on COCO.
Model
CLIP-I1
CLIP-TI
FID
SBED-LLaMA
0.695
0.546
71.7
Emu
0.443
0.260
554.2
GILL
0.418
0.227
274.5
ThinkDiff-LLMM
0.744
0.590
65.8
Table 5. Text-conditioned generation quality of ThinkDiff- LVLM on GenEval and DPG-Bende
Emu
SEED-LLaMA
Flux (Upperbound)
Ours
GenEvalf DPG-Benchf
3.25
35.4
65.1
39.1
12.4
47.3
82.6
54.8
Figure 8. Results of ThinkDiff-CLIP composing two im- ages. It creatively merge semantic details of both images. See more results in Appendix Figure 12.
of Thin on GenEval (Ghosh et al., 2023) and DPG-Bench (Hu et al., 2024) in Table 5. Our method clearly outperforms Emu and SEED-LLaMA. Flux is used as our diffusion decoder. Therefore, its evaluation is the upper- bound of our model. Please note that Geneval and DPG Bench focus on detailed text-prompt reconstruction, which does not fully evaluate ThinkDiff's s strengths on multimodal reasoning and multimodal-to-ime to-mmag generation.
of our vision-language training, which aligns multimodal features into a shared space, enabling flexible handling of complex multimodal understanding and composing tasks.
Video generation. ThinkDiff-CL.I is agnostic to diffusion decoders, making it versatile for integration with models like VideoX (Yang et al., 2024b), text-to-vides diffusion model. As shown in Appendix Figure 14, a background image is fed to the vision encoder and aligner network, along with a text prompt, to CogVideoX decoder. The model then generates a coherent video by seamlessly integrating images and text. This demonstrates ThinkDiff-CL.. flexibility and broad applicability for multimodal generation tasks.
4.3. Evaluation Results of DhfinDDiffCLIPP
We evaluate ThinkDiff- CLIP on various test cases to demon- strate its ability semantically understand images and able coherent composing of image and text modalities.
Single image + text prompt. Figure 6 and Appendix Fig- ure 11 show results with a single image as input. FLUX Ultra (Forest, 2024b), possibly finetuned by reconstruction- based diffusion training, performs well "copy-pasting" the input image (FLUX Ultra I), but struggles to main- tain coherence when additional text prompt is included (FLUX Ultra + I + T). In contrast, ThinkDiff-CLIP excels at understanding the semantic details of the input image and effectively integrates both image and text to generate logically coherent outputs (Ours + I and Ours + I + T).
4.4. Ablation Study
RMSNorm in the aligner network. As discussed in Sec- tion 3.2, the RMSNorm layer and its initialization are critical for training convergence. Figure 7 compares training losses of three setups: without RMSNorm layer, with default ini- tialization, and with our final design. Without a RMSNorm layer or using default initialization, the training loss fails to converge while with our design, the loss converges to a reasonable value, leading to strong evaluation performance. This comparison validates the effectiveness of our design.
Multiple images + text prompt. ThinkDiff-CLiP is flex- ible and can handle multiple images and text prompts. As shown in Figure 8 and Appendix Figure 13, it can combine semantic details from two images in a reasonable and co- herent manner. Figure 13 further demonstrates that with an additional text prompt (Ours + 21 + T), ThinkDiff-CLIP effectively incorporates the prompt into the generation.
Random masked training strategy. As discussed in Sec- tion 3.3, we introduce a masked training strategy to address the "shortcut mapping" problem in ThinkDiff-LVV ing. In Table 3, we compare the 2-shot accuracy on CoBSAT benchmark for models trained with and without this strategy.
train-
These multimodal generation results highlight the advantage
8

---

I Think, Therefore Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models
Table 6. Ablations of contrastive learning, data scale, different LVLMs, Janus Pro, and Flux Redux. ThinkDiff-LVLM uses Qwen2- VL-7b and 1.7M samples for training.
Color-I
Background-
Style-I
Action-1
Textre-1
Color-11
Background-II
Style-II
Action-I1
Texture-I1
Contrastive
0.414
0.244
0.140
0.202
0.230
0.347
0.346
0.235
0.258
0.231
Janus pro
0.403
0.234
0.378
0,462
0.338
0.313
0.319
0,283
0,549
0.264
Flux Redux
0.042
0.052
0.124
0.106
0.002
0.039
0,046
0.050
0,082
0.004
3.4M data
0.632
0.374
0.233
0.484
0.323
0.469
0,573
0.354
0,523
0.281
Internvi-2.5-8b
0.326
0.108
0.104
0,261
0.111
0.278
0.308
0.163
0,495
0.137
Owen2-VI.-72h
0.656
0.363
0.359
0.361
0.375
0.458
0.617
0.411
0.538
0.338
ThinkDiff-VLLM
0.622
0.349
0.237
0,459
0.290
0.511
0,534
0.340
0,534
0.292
Table 7. Training resources and 4-shot accuracy. Diff LVLM drastically reduces GPU usage and training time and improves accuracy from 0. 192, 0.07, and 0.058 to 0.463.
of using only one token similar to the original Imagebind, we extract 32 semantic tokens from both LVLM and T5 to compute the alignment loss. As shown in Table 6, our ThinkDiff achieves significant improvements on the CoB- SAT benchmark over the ImageBind-ste alignment.
GPU No.
Time /h
Average Acc.
SEED-LLaMA
64 A100
216
0.192
Emu
128 A 100
48
0.070
Different VLMs. We use Qwen2- Qwen2-VL-7B as the LVLM by default, which supports interleaved image and text inputs. We ablates ThinkDiff-LVLNN performance with different LVLMs. As shown in Table 6, InternVL2.5-8B achieves worse performance compared to Qwen2-V Owen2-VL-7B. indicat- ing that a stronger LVL.M can improve the alignment and accuracy, Moreover, with a more powerful LVLM Qwen2- VL-72B, ThinkDiff achieves new SoTA on most tasks.
GILL
2 A6000
48
0.058
ThinkDiff-VLM
A100
5
0.463
Without the random masked training, ThinkDiff-LVLM con- verges quickly but achieves inferior evaluation accuracy, indicating incomplete feature space alignment. In contrast, with the random masked training, the model achieves SOTA accuracy on the evaluation tasks. This validates the criti- cal role of the random masked training for proper feature alignment in ThinkDiff-LVL.M
Data scale. We double the sample size to 3.4M to include more diverse datasets. We train new model for the same steps. Table 6 shows that with more data, our model can generally improve the results in most tasks.
Using generated tokens of LVLM. As discussed in Sec- tion 3.3, LhiL uses deep features of generated tokens from the LVLM effectively transfer reasoning in- formation to diffusion decoders. In this study, we train model using the deep features of input tokens of LVLM for alignment, with these features extracted from the final nor- malization layer of the LVLM. As shown in Table 3, using input token features for alignment leads to a significant per- formance drop, underscoring the critical role of generated tokens in successfully transferring reasoning capabilities.
Janus Pro and Flux Redux. To evaluate Janus Pro (Chen et al., 2025) on CoBSAT, we implemented a two-step workaround since lacks multimodallto-imee capabilittis. Janus Pro converted multimodal inputs into in- termediate textual descriptions, which were then processed through its text-to-image pipeline. As shown in Table 6, ThinkDiff outperforms Janus Pro in most tasks due to its alignment of powerful LVLMs and diffusion decoders, en- abling superior multimodal reasoning and generation. We also evaluated the open-source Flux Redux (Labs, 2024), which supports image inputs on CoBSAT by organizing test cases into a single input image. As shown in Table 6, unlike ThinkDiff, Flux Redux performs poorly on CoBSAT, confirming its lack of reasoning capabilities.
generation
Training time and GPU usage. Table 7 summarizes the training time, GPU requirements, and 4-shot average curacy on CoBSAT for different methods. Our method drastically reduces GPU usage from 128 A100 GPUs to just 4 and cuts training time from 216 hours to only 5 hours. Meanwhile, it achieves a significant improvement in aver- age accuracy, increasing from 0.192, 0.070, and 0.058 to an impressive 0.463. These results highlight the efficiency and effectiveness of our novel alignment paradigm.
5. Conclusion
We introduced ThinkDiff, novel aligmment paradigm equipping diffusion models with multimodal in-context soning of VLMs by vision-language training. ThinkDiff sets new SoTA on the CoBSAT benchmark and excels in vari- ous reasoning tasks. Future work will address its limitations (Appendix A), and extend its capabilities to modalities like audio and video to develop any-to-any foundation models.
Comparison to contrastive learning. We conduct an ex- periment with ImageBind-style (Girdhar et al., 2023) con- trastive alignment. The input of LVLM is an image and a text prompt. It generates token features and text. The input of the T5 decoder is the LVLM's generated text. Instead
9

---

Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models
Acknowledgements
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Con- ceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, pp. 3558- 3568, 2021.
This research is supported part by HKUST-SAIL joint research funding, the Early Career Scheme of the Research Grants Council (RGC) of the Hong Kong SAR under grant No. 26202321 and HKUST Startup Fund No. R9253.
Chen, J., Jincheng, Y., Chongjian, G., Yao, L.,, Xie, E., Wang. Z., Kwok, J., Luo, P., Lu, H., and Li, Z. Pixart-a: Fast training of diffusion transformer for photorealistic lext-to-image synthesis. In ICLR, 2024.
Impact Statement
This paper proposed ThinkDiff, a novel alignment method that enhances text-to-image diffusion models by integrating multimodal in-context reasoning capabilities from vision- language models. By simplifying the alignment process between the VLM and diffusion decoder, ThinkDiff democ- ratizes complex multimodal reasoning generation tasks and make them more accessible and efficient to train. ThinkDiff has potential applications across different fields, such as education, design, and creative industries. However, similar to other text-to-image diffusion models and large vision- language models, ThinkDiff could be potentially misused for generating misleading and harmful content. To mitigate these problems, it is essential to deploy the model responsi- bly and implement robust safeguards to prevent misuse.
Chen, X., Wu, Z., Liu, X., Pan, Z.,, Liu, W., Xie, Z.,, Yu, X., and Ruan, C. Janus-pro: Unified multimodal understand- ing and generation with data and model scaling, arXiv preprint arXiv:2501.18811 2025.
Fang, Y., Wang. W., Xie, B.. Sun, Q.. Wu, L., Wang, X., Huang, T., Wang, X., and Cao, Y. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, pp. 19358-19369, 2023.
Forest, B. back-forest-lbbe fflux, 2024a. GitHub repository.
Flux. https: / http//gttthhhoo//
Forest, B. Flux ultra. httpss//blackforrss.abbs
ai/ultra home, 2024b.
References
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano,
Achiam, J., Adler, S.. and et. al., S. A. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2024.
A. H., Chechik, G., and Cohen-or, D. An image is worth word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
AL M. Llama 3: Vision and edge ai for mo- bile devices.
https://at hett:::/aii.meta... meta com/ blog/llama 3- 2 2024-ectt222--iiieee redge-mobile-deiceee 2024a.
Ge, Y., Zhao, S., Zeng, 7.,, Ge, Y., Li, C.,, Wang, X., and Shan, Y. Making Ilama and draw with seed tokenizer. In ICLR, 2024.
Ghosh, D., Hajishirzi, H., and Schmidt, L. Geneval: An object-focused framework for evaluating text-to-image alignment. Neurips, 2023.
AI. S. model. Deepfloyd if: Text-to-image aiteews///../////////ee// Shepfloyd-if-tee imaeoo--mmmmmmm-e-eee 2024b.
Girdhar, R. El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, Imagebind: One embedding space bind them all. In CVPR, 2023.
AI, S. Stable diffusion 3.5. https://githut: com/ Stability-AIsdd:: 2024c. GitHub repository.
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y. Clipscore: A reference-free evaluation metric for image captioning. In EMNLP, 2021.
Berman, W. and Peysakhovich, A. Mumu: Bootstrapping multimodal image generation from text-to-image data. arXiv preprint arXiv:2406. 18790, 2024.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. NeurIPS, 33:6840-688 6851, 2020.
Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402. 2023.
Hu, X., Wang, R., Fang, Y., Fu, B., Cheng, P., and Yu, G. Ella: Equip diffusion models with 11m for enhanced semantic alignment. arXiv preprint arXiv:2403.05135 2024.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. NeurlPS, 2020.
Koh, J. Y., Fried, D., and Salakhutdinov, R. R. Generating images with multimodal language models. NeurIPS, 36, 2024.
10

---

Think, Therefore Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In SOSP. pp. 61 611-626 2023.
Radford, A., Narasimhan, K., Salimans, T.,, and Sutskever, I. Improving language understanding by generative pre- training. OpenAl, 2018. URL https:/ com/ research/ won///eeaarch/aaguueeeeeeee http://guuuuuuuuuuueiis.e
Labs, B. F.
Flux.
https:
Radford, A., Kim, J. W., Hallacy, C., Ramesh, et al. Learn- ing transferable visual models from natural language su- pervision. In International conference on machine learn- ing. pp. 8748 8763.773.. PMLR, 2021.
//huggingface. co/black- moorest-lbbe FLUX. Redux dev, 2024. Huggingface repository.
J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pp. 19730-19742. PMLR, 2023.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. Exploring the limits transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 (140):1-67, 2020.
Li, Z., Cao, M., Wang, X., Qi, Z., Cheng, M.-M., and Shan, Y. Photomaker: Customizing realistic human photos via stacked embedding. In CVPR, pp. 8640-8650, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models, In CVPR, pp. 10684-10695, 2022.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco:
Common objects in context. In ECCV, 2014.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuming text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. 22500-22510, 2023.
Liu, B., Akhgari, E., Visheratin, A., Kamko, A., Xu, L., Shrirao, S., Souza, J., Doshi, S., and Li, D. Playground v3: Improving text-to-image alignment with deep-fusion large language models. arXiv preprint arXiv2240911099) 2024.
Saharia, C., Chan, W., Saxena, S., Li, L., Whang. J., Denton, E. L.., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image dif- fusion models with deep language understanding. Ad- vances in neural information processing systems, 35: 36479-3660 -3494,, 2022.
Liu, H., Li, C., Wu, Q.. and Lee, Y. J. Visual instruction tuning. In NeurlPS, 2023.
Mou, C., Wang. X., Xie, L., Wu, Y., Zhang, J., Qi, Z., and Shan, Y. 12i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In AAAI, volume 38, pp. 4296-4304, 2024.
Sharma, P., Ding, N., Goodman, S., and Soricut, R. Con- ceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, pp. 2556- 2565, 2018.
Ordonez, V., Kulkarni, G., and Berg, T. Im2text: Describing images using million captioned photographs. NeurlPS, 24, 2011.
Shi, W., Han, X., Zhou, C., Liang, W., Lin, X. V., Zettle- moyer, L., and Yu, L. Llamafusion: Adapting pretrained language models for multimodal generation. arXiv preprint arXiv:2412.15112 2024.
Pan, X., Dong, L., Huang, S., Peng. Z., Chen, W., and Wei, F. Kosmos-g: Generating images in context with multimodal large language models. arXiv preprint arXiv:2310.02992, 2023.
Sun, Q. Yu, Q. Cui, Y., Zhang, F., Zhang, X., Wang, Y.. Gao, H., Liu, J., Huang, T., and Wang. X. Gen- erative pretraining in multimodality. arXiv preprint arXiv:2307.0522,, 2023.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In ICCV, pp. 4195-4205, 2023.
Peng, Y., Cui, Y., Tang. H., Qi, Z., Dong, R., Bai, J., Han, C.. Ge, Z., Zhang, X., and Xia, S.-T. Dreambench++: A human- alinned benchmark for personalized image gener- ation. arXiv preprint arXiv:2406. 16855, 2024.
Tong. S., Fan, D., Zhu, J., Xiong, Y., Chen, X., Sinha, K., Rabbat, M., LeCun, Y., Xie, S., and Liu, Z. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint
2024.
Qian, G., Wang, K.-C., Patashnik, O., Heravi, N., Osta- shev, D., Tulyakov, S., Cohen-Or, D., and Aberman, K. Omni-id: Holistic identity representation designed for generative tasks, arXiv preprint, 2024.
Wang, K.-C,, Ostashev, D., Fang. Y., Tulyakov, S., and Aber- man, K. Moa: Mixture-ou of attennttn for subject-context disentanglement in personalized image generation. In SIGGRAPH Asia, pp. 1-12, 2024a.

---

Think, Therefore Diffuse: Enabling Multimodal In- In-Context Reasoning in Diffusion Models
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution. arXiv preprint 1rXiv:2409.22191, 2024b.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint Att::2304.10592,, 2023.
Wang. Q., Bai, X., Wang. H., Qin, Z., Chen, A., Li, H., Tang, X., and Hu, Y. Instantid: Zero-shot generation in seconds. arXiv preprint arXiv:240..07519, 2024c.
Wang, X., Zhou, X., Fathi, A., Darrell, T., and Schmid, C. Visual lexicon: Rich image features language space. arXiv preprint arXiv:2412.0677
2024d.
Wu, S., Fei, H., Qu, L., Ji, W. and Chua, T.-S. Next- gpt: Any-to-any multimodal Ilm. arXiv preprint arXiv:2309.055/9, 2023.
Xiao, S., Wang, Y., Zhou, J., Yuan, H., Xing, X., Yan, R., Wang, S., Huang, T., and Liu, Z. Omnigen: Unified image generation. arXiv preprint arXiv:2409.1 11300, 2024.
Xie, E., Chen, J., Chen, J., Cai, H., Tang. H., Lin, Y., Zhang, Z., Li, M., Zhu, L., Lu, Y., and Han, S. Sana: Effi- cient high-resolution image synthesis with linear diffu- sion transformer, 2024. URL htw.h:ttp////eee http:::///rxxvoorrr abs/2410 10629.
Yang. A., Yang. B., Zhang, B., Hui, B., Zheng, et al. Qwen2.5 technical report. arXiv preprint arXi::2412.15115, 2024a.
Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J. Yang. Y., Hong, W., Zhang, X., Feng, G., et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:248.00607. 2024b.
Ye, H., Zhang, J., Liu, S., Han, X., and Yang, W. Ip-adapter: Text compatible image prompt adapter for texa-to- diffusion models. arXiv preprint arXiv:2308.0672. 2023.
Ye. H., Huang, D.-A., Lu, Y., Yu, Z., Ping, W., Tao, A., Kautz, J., Han, S., Xu, D., Molchanov, P., et al. X- vila: Cross-modality alignment for large language model. arXiv preprint arXiv:2405. 20345559535, 2024.
Zeng, Y., Kang, W., Chen, Y., Koo, H. I., and Lee, K. Can milms perform text-to-image in-context lcarning? COLM, 2024.
Zhang. and Sennrich, R. Root mean square layer normal- ization. NeurlPS, 2019.
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In ICCV, pp. 3836-3847, 2023.
12

---

Think, Therefore Diffuse: Enabling Multimodal In-Contex Reasoning in Diffusion Models
APPENDIX
A. Limitation
Despite ThinkDiff's strong performance reasoning generation tasks, several limitations remain for future work. First, while substantially outperforms existing methods, ThinkDiff still encounters difficulties with certain complex cases. Enhancing reasoning accuracy may require stronger VLMs, better data quality, advanced diffusion models, and improved training strategies. Second, although this work primarily focuses logical reasoning rather than preserving image fidelity, improving fidelity could expand its applications in tasks like image editing. Finally, more diverse evaluation tasks are needed to better assess reasoning performance and advance research in this area.
B. Dataset Details
For ThinkDiff-LVLM, the training process requires images and their corresponding VLM-generated tokens. We randomly sample 1.7 million images from the CC3M (Sharma al., 2018). CC12M (Changpinyo al., 2021), and SBU (Ordonez et al., 2011) datasets. These images are preprocessed using Qwen2-Vi VI., which generates detailed descriptions based on randomly selected text prompts from a predefined set. The generated text tokens and token features are stored for training the alignment. We generate 64 tokens for each data sample. Data processing is accelerated using the vLLM framework (Kwon et al., 2023).
For ThinkDiff- CII,,,, the training utilizes images and their corresponding captions, sampled from a combination of CC3M (Sharma et al., 2018), CC12M (Changpinyo al., 2021), SBU (Ordonez al., 201
The predefined prompts for DhiM are designed to encourage the VLM to generate detailed descriptions of the image. Below is a list of the prompts we use, some of which are adapted from LLaVA (Liu et al., 2023).
Describe the image concisely.
Provide brief description of the given image.
Offer succinet explanation of the picture presented.
Summarize the visual content of the image.
Give short and clear explanation of the subsequent image.
Share concise interpretation of the image provided.
Present a compact description of the photo's 's key features.
Relay a brief, clear account the picture shown.
Render a clear and concise summary of the photo.
Write terse but informative summary of the picture.
Create a compact narrative representing the image presented.
Generate a prompt that can recreate the image in a 2D diffusion model.
Provide descriptive prompt to reproduce the given image using diffusion model.
Create a prompt suitable for 2D diffusion model to generate the same image.
Summarize the visual details as a prompt for 2D diffusion model.
Write clear prompt guide 2D diffusion model in recreating the image.
13

---

Think, Therefore I Diffuse: Enabling Multimodal In-Contex Reasoning in Diffusion Models
Evaluation CoBSAT. As described in Section 4.2, when evaluating ThinkDiff-LVLM the CoBSAT dataset, instruction prompt guide Qwen2-VL. to generate the next image based multimodal inputs. Qwen2-VL is vision-language model primarily designed to answer questions by text. It does not automatically know that want it to generate the next image and we also do not finetune it for this specific task. Therefore, the instruction prompt is necessary. The instruction prompt used in our evaluation is:
give you several words and pictures. First, please analyse what the next picture is. Then give me detailed diffusion prompt describe the next picture. Please only provide me the detailed prompt and start the answer with 'Create an image'.
C. More High-quality Results
C.1. ThinkDiffLyLMM
Figure and 10 demonstrate more high-quality results of ThinkDiff-LLLM on 2-shot evaluation in CoBSAT benchmark. ThinkDiff-LVLMM can not only generate images with logically correct objects and attributes based on advanced reasoning, but also generate much higher-quality images than SEED-LLaMA (Ge et al., 2024), Emu (Sun et al., 2023), and GILL (Koh al., 2024). These compared methods typically generate wrong images lower quality.
C.2. ThinkDiff-CLIP
Figure 11 shows more results with a single image (I) or single image with a text prompt (I + T) as input. FLUX Ultra (Forest, 2024b) struggles to maintain coherence when an additional text prompt is included (FLUX Ultra + I+ T) while ThinkDiff-CII excels integrating both image and text generate logically coherent images (Ours + I and Ours T).
Figure 12 and 13 shows more results of our ThinkDiff- Ciik handling multiple images and text prompts. ThinkDiff- Phin effectively combines semantic details from two input images in a coherent manner and seamlessly integrates text prompts to guide the generation, showcasing its flexibility and capability for complex multimodal tasks.
D. Video Results of ThinkDiff-LIP
As discussed in Section 4.3, ThinkDiff-CLe CLII integrate Cog VideoX (Yang et al., 2024b) model for text-to-video generation. Figure 14 demonstrates frames of video generation results, validating ThinkDifi-CLAn flexibility and broad applicability for multimodal generation tasks.
14

---

Think, Therefore Diffuse: Enabling Multimodal In- In-Context Reasoning in Diffusion Models
Multimodal Input
Multimodal Input
Multimodal Input
beach
park
purple
vellon
box
seafloor
black
apple
Reasoning output
Reasoning output
Rensoning output
Seed
CII
Seed
Emnu
GILL
Seed
ThinkDiff LVLM
ThinkDiff LVLM
AYLM
Think
GT unswer: lion on seafloor
GT answer: black leaf
GT answer: wicker apple
Multimodal input
Multimodal Input
Multimodal Input
bag
apple
cup
box
apple
leaf
beach
vo Ican
glacier
Rensoning output
Reasoning output
Rensoning output
Seed
Famm
GILL
Seed
Fann
GILL
Seed
Famu
GILL
ThinkDiff- LVLM
ThinkDiff- LVLM
ThinkDiff LVLM
GT unswer: cup
GT answer: pixel leaf
GT answer: bird on glacier
Multimodal input
Multimodal input
Multimodal input
ball
book
car
hat
opple
car
dog
Monkeey
sheep
Reusoning untput
Reusoning output
Reusoning output
Seed
GILL
Seed
Eanu
GILL
Seed
Ermu
GILL
ThinkDiff- LVLM
Think LVLM
ThinkDiff
LVLM
GT answer: plastic
GT answer: lego
GT answer: singing sheep
Figure 9. More 2-shot reasoning results of LLiM on CoBSAT benchmark.
15

---

Think, Therefore Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models
Imput
Inpul
Input
lego watercolor
red
hat
white
leaf
graffiti GT answer graffiti
pink
chair
GT answer pink car
GT answer
plastic chair
car
Input
Input
Input
tiger
zebra
cow
monkey
cow
sheap
bird
tiger tiger in
zebra
GT answer waterfall bird
GT answer zebra in space
Input
Input
Input
zebra
box
monkey
cow
book
lion
cup
GT answer singing cat
answer lion reading
answer pixel cup
Input
Input
Input
wat watercalor
graffit
box
icon
3D
apple GT answer denim apple
lego
sketch
GT answer
answer
lego
sketch
apple
chair
Figure 10. More 2-shot reasoning results of ThinnDDifF-LLLMM on CoBSAT benchmark.
16

---

Think, Therefore Diffuse: Enabling Multimodal In-Conter Reasoning in Diffusion Models
Input image
Flux Ultra +1 I
Flux Ultra + I+T
Ours +1
Ours + IIT
"There big modern yacht on the sea."
"There big modern yacht on the
STOP
STOP
STOP
STOP
STOP
LeT's Sqme
ccppy & paste' Let's reason on the stop sign
copy & paste', Let's reason the stop sign..
Meow
"There speech balloon showing 'Meww"
"There is speech balloon showing 'Meow"" "
"There happily running white dog."
"There is happily running white dog."
"The girl laughing."
"The girl is laughing."
Figure 11. Generation results of single image and text prompt of ThinkDiff-CLN CLIP.
17

---

Think, Therefore Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models
Figure 12. Multiple input image generation results of ThinkDiff-CLIP.
18

---

Think, Therefore Diffuse: Enabling Multimodal In-Conter Reasoning in Diffusion Models
Input images
Ours 21
Text
Ours 21 +
The guitar lies down and floats on the water.
The girl holds the guitar.
The woman and the penguin stand together.
The dog wears red swimsuit.
The woman stands
there.
Figure 13, Generation results for multiple images (2I) and multiple images with text prompt (21 + T) of ThinkDiff-CLIP
19

---

Think, Therefore Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models
Video generation of ThinkDiff-CLLI with multimodal input
The falles beless shite los clea
troved ned johnet pant pooe yebter morh Sullght
comeer felles beher white
risd
le lime be sucue
cutting sorm, pvr nose, The with other care ogat
cesting nore his poee
de pemee gou
The hree the lod coall mallin th sughan The our stie reyped crie turond the najee tervik.
rocthrdd strruhhcc sreee
acis makigg o though
suded part aerfrraace
Video frames
Figure 14, Image + text to video generation results of ThinkDiff-CLPP
20


Reviews:
[{"review_id": "sdlti2U1RQ", "reviewer": "Reviewer_37KH", "summary": "The paper enables diffusion models to perform in-context reasoning across images and text, rather than just reconstructing pixel information. The paper shows two variants --- LVLMs and CLIP. The images generated are of good quality and obtain state of the art performance on various measures.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer 37KH**\n\nWe thank Reviewer 37KH for the insightful comments and suggestions. We provide our responses below.\n\n\n>**Q1. Experiment with Imagebind.**\n\nWe conduct an experiment with ImageBind-style alignment to align the LVLM decoder and the T5 encoder. The input of LVLM is an image and a text prompt. It generates token features and text. The input of the T5 decoder is the LVLM's generated text. Instead of using only one token similar to the original Imagebind, we extract 32 semantic tokens from both LVLM and T5 to compute the alignment loss. This gives stronger capacity to the alignment. As shown in the table below, our ThinkDiff achieves significant improvements on the CoBSAT benchmark over the ImageBind-style alignment.\n\n| Model      | Color-I | Background-I | Style-I | Action-I | Texture-I | Color-II | Background-II | Style-II | Action-II | Texture-II |\n|------------|:-------:|:------------:|:-------:|:--------:|:---------:|:--------:|:-------------:|:--------:|:---------:|:----------:|\n| **ImageBind**  | 0.414   | 0.244        | 0.140   | 0.202    | 0.230     | 0.347    | 0.346         | 0.235    | 0.258     | 0.231      |\n| **Ours**       | **0.622**  | **0.349**        | **0.237**   | **0.459**    | **0.290**     | **0.511**    | **0.534**         | **0.340**    | **0.534**     | **0.292**      |\n\n\n>**Q2. Advantages over ImageBind.**\n\nWhile the contrastive alignment in Imagebind shows promising results in multimodal alignment, ThinkDiff introduces more advantages:\n\n\n1. ThinkDiff exploits the capabilities and knowledge of the LVLM decoder and the LLM (T5) decoder for the alignment, which enables data-efficient and training-efficient alignment, and has been validated in vision-language research areas. On the contrary, Imagebind needs web-scale datasets to train from scratch.\n   \n2. ThinkDiff provides a fine-grained and flexible alignment method. Since it does not compute element-wise distances, the tokens of different modalities can have different lengths. This also supports training strategies such as masked training. The alignment is not element-wise minimization but is the high-level semantic alignment. \n\n3. ImageBind typically uses one token to compute the token distances, which is a coarse alignment. ThinkDiff excels at fine-grained alignment for longer token sequences. Even if using more tokens in ImageBind, the lengths of two types of tokens must be the same.\n\nTherefore, as shown in the experiment in **Q1**, our method can achieve much better accuracy compared to the ImageBind-style alignment.", "claims_and_evidence": "Yes, the claims are supported by the evidence.", "methods_and_evaluation_criteria": "Yes, the proposed method correctly reflects the application at hand.", "theoretical_claims": "There are no theoretical claims in the paper.", "experimental_designs_or_analyses": "The experiments look correct to me.", "relation_to_broader_scientific_literature": "The paper is related to the broader scientific literature.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Please see questions for authors.", "other_comments_or_suggestions": "Please see questions for authors.", "questions_for_authors": "I list down the strengths and the weaknesses of the work here\n\nStrength\n\n- The paper proposes a method that can effectively handle reasoning in diffusion model.\n- The experimental gain is significant, and good for the community to build on.\n\nWeakness\n\n- The major concern appears to be the key aspect of the model. Overall, I feel the method is leveraging the strong powers of the base model---LVLMs or CLIP and use that to learn a multimodal aligner. This idea has been used previously in video and text representation learning, and this work proposes to extend that to multimodal composition. It is not clear why the authors could not use any other way to learn the joint embedding space. For example, ImageBind, Rohit Girdhar et al. performs this step in the image space. What is the key insight that enables this method to do better beyond existing work?\n\n- Same as above, ImageBind by Girdhar et al. is not compared against in the experiments. Why can it not be adapted to the authors' use case? The overall idea is for the model to be compared against a variety of inputs.\n\nOverall, I am leaning towards acceptance, but the distinctions and experiments could be made more robust and convincing. .", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "2OpNUbhjyX", "reviewer": "Reviewer_BZ6o", "summary": "This paper proposes \"ThinkDiff\", a novel method to incorporate VLMs in text-to-image generation pipelines with the goal to improve multimodal understanding and in-context reasoning capabilities. The key lies in aligning the VLM outputs with the diffusion decoder input space, which is done by using the corresponding LLM Decoder as a proxy signal for alignment. This results in a lightweight training algorithm and empirical results demonstrate a) significant improvements for in-context generation on CoBSAT and b) qualitative performance on compositional tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer BZ6o**\n\nWe thank Reviewer BZ6o for the insightful comments and suggestions. We provide our responses below.\n\n>**Q1: Generation quality.**\n\nWe evaluate the general image-conditioned generation of ThinkDiff-LVLM on 1k images in COCO. The models are conditioned by an image in the experiment. We show the FID, the CLIP image metric (CLIP-I), and the CLIPScore (CLIP-T) in the table below. Our method can achieve much better performance than existing competitors such as SEED-LLaMA, Emu, and GILL.\n\n\n| Model            | CLIP-I$\\uparrow$   | CLIP-T$\\uparrow$     | FID$\\downarrow$        |\n|------------------|------------|------------|------------|\n| SEED-LLaMA       | 0.695     | 0.546     | 71.7      |\n| Emu              | 0.443     | 0.260     | 554.2     |\n| GILL             | 0.418     | 0.227     | 274.5     |\n| ThinkDiff-LVLM   | **0.744** | **0.590** | **65.8**  |\n\n>**Q2: Inference time.**\n\nThinkDiff-LVLM replaces the T5 encoder in Flux with an LVLM. The LVLM model is highly optimized in the community framework such as vLLM and adds minimal latency to the inference. Qwen2-VL-7B model takes less than 0.2 seconds for one prompt that consists of both images and texts. Although the T5 encoder (typically costs 0.05 seconds for one prompt) is faster, they both are a marginal overhead compared to the diffusion process of Flux (typically over 2.5 seconds).\n\n>**Q3: ThinkDiff-LVLM vs ThinkDiff-CLIP and their evaluation.**\n\nThinkDiff-LVLM can handle different tasks including general image-conditioned generation, similar to ThinkDiff-CLIP. As shown in **Q1**, we evaluate ThinkDiff-LVLM on the image-conditioned generation task. Our method obtains better CLIP-I, CLIP-T, and FID scores compared to SEED-LLaMA, Emu, and GILL. This shows the general applicability of ThinkDiff-LVLM beyond the reasoning.\n\nOn the contrary, ThinkDiff-CLIP is limited by the CLIP image encoder and the T5 encoder. Therefore, in our experiments in the main paper, we already observed that ThinkDiff-CLIP cannot handle very complex logical questions as ThinkDiff-LVLM does, but it has strong capabilities for multimodal composing. \n\nThe above additional evaluation further shows their respective strengths and weaknesses.", "claims_and_evidence": "Most of the claims in the paper are well-supported by their experiments. However, there is no quantitative evaluation for the quality of generated images as well as analysis of inference compute, which needs to be addressed (see Methods And Evaluation Criteria).", "methods_and_evaluation_criteria": "Certain experiments and benchmarks are missing from the current manuscript.\n\n1. Despite strong results on multimodal in-context generation, the paper lacks in terms of quantitative evaluation outside of the in-context generation settings. For instance, quantitative evaluation of quality of image generation (ex. FID/CLIP scores on zero-shot generation on COCO or subject driven generation on DreamBench) as well as a comparison against contemporaries is missing. However, these are important to ascertain if there are any potential drawbacks arising from the proposed framework and its focus on reasoning.  \n2. While low compute requirements for training is desirable, inference cost is arguably a bigger decision factor for model adoption. Therefore, the paper needs to include an analysis for the inference costs (especially due to the additional multimodal processing in VLMs) to complement their training cost comparisons.\n3.  The paper proposes two models - ThinkDiffLVLM and ThinkDiffCLIP. However, the first is only evaluated on in-context settings while the other only on compositional settings without any specific reasoning for the choice. Evaluation of both methods on both settings is needed to understand their respective strengths and weaknesses.\n\nNevertheless, the framework is simple and effective, and my overall opinion of the paper is generally positive. I believe that addressing these concerns will serve to strengthen the manuscript.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design is sound but not comprehensive enough (see Methods And Evaluation Criteria).", "relation_to_broader_scientific_literature": "The method presents a novel approach to incorporate a VLM's multimodal reasoning ability for in-context image generation with the added advantage of a lightweight training pipeline. The resulting model achieves significant gains on the CoBSAT dataset, compared to existing approaches. With some additional empirical validation, it can be a good contribution for the research community.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper is well structured and easy to follow, while the motivation and approach seem sound.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "xpm1SpdblA", "reviewer": "Reviewer_fnpq", "summary": "This paper introduces **ThinkDiff**, a novel alignment paradigm that enhances text-to-image diffusion models with **multimodal in-context reasoning** capabilities. Instead of traditional pixel-level reconstruction-based fine-tuning, the authors propose aligning vision-language models (VLMs) with the decoder of a **large language model (LLM)**, leveraging vision-language training as a **proxy task**. The key insight is that **LLM decoders and diffusion model decoders share the same input feature space**, allowing the transfer of reasoning capabilities without requiring complex multimodal reasoning datasets.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer fnpq**\n\nWe thank reviewer fnpq for the insightful comments and suggestions. We address major concerns as follows and will add additional literature in the reference.\n\n\n>**Q1: Ablation on different VLMs.**\n\nWe use Qwen2-VL-7B which supports interleaved image and text inputs as the LVLM. The table below further ablates ThinkDiff-LVLM's performance with different LVLMs. \nInternVL2.5-8B achieves a worse performance compared to Qwen2-VL-7B, indicating that a stronger LVLM can improve the alignment and accuracy. Moreover, with a more powerful LVLM Qwen2-VL-72B, ThinkDiff achieves a new SoTA on most tasks. \n\n| Model            | Color-I | Background-I | Style-I | Action-I | Texture-I | Color-II | Background-II | Style-II | Action-II | Texture-II |\n|----|---|----|---|----|---|----|---|----|---|-----|\n| Internvl-2.5-8b  | 0.326   | 0.108        | 0.104   | 0.261    | 0.111     | 0.278    | 0.308         | 0.163    | 0.495     | 0.137      |\n| Qwen2-VL-7b      | 0.622   | 0.349        | 0.237   | **0.459**| 0.290      | **0.511**| 0.534         | 0.340     | 0.534     | 0.292      |\n| Qwen2-VL-72b     | **0.656**| **0.363**   | **0.359**| 0.361    | **0.375** | 0.458    | **0.617**     | **0.411**| **0.538** | **0.338**  |\n\n\n>**Q2: Data scale.**\n\nWe double the sample number to 3.4M to include more diverse datasets. We train a new model for the same steps. The table below shows with more data, our model can generally improve the results in most tasks.\n\n| Data | Color-I | Background-I | Style-I | Action-I | Texture-I | Color-II | Background-II | Style-II | Action-II | Texture-II |\n|----|---|---|----|-----|---|---|---|---|----|----|\n| 1.7M  | 0.622   | 0.349        | **0.237** | 0.459    | 0.29      | **0.511**| 0.534         | 0.340     | **0.534** | **0.292**  |\n| 3.4M  | **0.632**| **0.374**   | 0.233   | **0.484**| **0.323** | 0.469    | **0.573**     | **0.354**| 0.523     | 0.281      |\n\n\n>**Q3: Contrastive loss.**\n\nIn **Q1** and **Q2** of Reviewer 37KH, we use the contrastive loss similar to the ImageBind. It has some disadvantages compared to our method and is inferior in accuracy. However, it still shows potential to effectively combine the explicit contrastive alignment and our implicit alignment in future work.\n\n\n>**Q4: Real-world application.** \n\nOur method not only can handle the tasks defined by the CoBSAT benchmark but also can handle different general tasks. To demonstrate this point, we further evaluate our model on general generation capabilities on other benchmarks, i.e., COCO, GenEval, and DPG-bench. Our method shows clearly better results compared to other competitors. Please refer to the COCO table in Reviewer BZ6o **Q1**, the GenEval, and the DPG-bench tables in Reviewer j31i **Q3**.\n\n>**Q5: Other multimodal fusion techniques.**\n\nThe Flux-pro actually uses a fusion method similar to IP-Adapter which injects image features by attention. As shown in Paper Figures 6, 11, and 13 in the main paper, our method clearly shows advantages in coherently composing different multimodal instructions.\n\n>**Q6. Formal analysis of feature space alignment.**\n\nThe alignment quality of ThinkDiff is implicitly evaluated by the accuracy and consistency of the reasoning benchmark and composing benchmark. Since our alignment is implicit, it is important but not very straightforward to directly analyze the feature space. To investigate the theoretical derivations of the alignment, we may rely on the development of vision-language training in VLMs. We humbly leave this for future work.\n\n>**Q7: Other multimodal reasoning tasks.**\n\nIn this paper we mainly target multimodal generation tasks. The LVLM in our model supports captioning or VQA tasks but the whole method itself does not target them. However, our alignment is a general alignment method and can be possibly applied to other multimodal tasks. This is a very interesting direction for future research. \n\n>**Q8: Other diffusion models.** \n\nOnce aligned, our ThinkDiff can be applied to other models that use T5 as encoder in our experiments. For example, beyond FLUX, we also applied ThinkDiff to CogvideoX in Appendix Figure 14, where a coherent video is generated by seamlessly integrating images\nand text. This demonstrates ThinkDiff's flexibility. Extending ThinkDiff to even more diffusion models is straightforward and is left as future work due to the tight schedule.\n\n\n>**Q9: Failure cases.**\n\nThis anonymous link (https://anonymous.4open.science/r/anonymous-4DF1/failure_case.png) shows two failure cases on CoBSAT. We think the main sources are from two aspects. One is the imperfect of LVLM which gives wrong reasoning, such as the \"cow in the desert\" case. Using a more powerful VLM can address this problem. The other is the imperfect alignment that does not accurately condition diffusion models, such as the second \"white apple\" case. We think a possible way to effectively address this problem is to include high-quality datasets for end-to-end training.", "claims_and_evidence": "The claims made in the paper are generally well-supported by empirical evidence. The authors provide:\n\n- **Quantitative results** demonstrating large accuracy improvements on the CoBSAT benchmark compared to prior methods.\n- **Ablation studies** validating key components of ThinkDiff, such as the importance of aligning generated token features instead of input token features.\n- **Comparisons to existing approaches**, showing that ThinkDiff not only outperforms but also requires significantly fewer resources.\n\nHowever, while the results are strong, further validation on **diverse datasets and real-world applications** would reinforce the claims regarding generalizability.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem:\n\n- The **CoBSAT benchmark** is a well-suited dataset for evaluating multimodal reasoning in diffusion models.\n- The evaluation method follows a structured **2-shot and 4-shot reasoning approach**, ensuring fair comparisons.\n- **Ablation studies** confirm the contribution of the core components of ThinkDiff.\n\nHowever, the work mainly focuses on **image generation tasks**. It would be valuable to evaluate the method on **other multimodal reasoning tasks**, such as **captioning or VQA**, to assess broader applicability.", "theoretical_claims": "The paper does not focus heavily on theoretical derivations but instead introduces a **novel alignment framework**. The core theoretical claim is that aligning **VLMs with LLM decoders naturally aligns them with diffusion model decoders** due to their shared input feature space.\n\nThis is a **reasonable assumption**, but additional theoretical validation or **formal analysis of feature space alignment** could further strengthen the claim.", "experimental_designs_or_analyses": "The experimental design is well-structured, with:\n\n- **Comparisons against multiple baselines**, including SEED-LLaMA, Emu, and GILL.\n- **Ablation studies** examining the effectiveness of masked training, LVLM-generated tokens, and RMSNorm initialization.\n- **Efficiency comparisons**, showing that ThinkDiff achieves better performance with significantly lower computational costs.\n\nHowever, the dataset used for training is relatively **small (~1.7M images from CC3M, CC12M, and SBU)**. Additional results on **larger and more diverse datasets** could provide stronger evidence of generalizability.", "relation_to_broader_scientific_literature": "This work builds on several key areas:\n\n- **Text-to-image diffusion models** (e.g., Stable Diffusion, Imagen).\n- **Multimodal large language models (LLMs)** (e.g., Flamingo, SEED-LLaMA).\n- **Vision-language models (VLMs)** (e.g., CLIP, BLIP-2).\n- **Multimodal reasoning benchmarks** (e.g., CoBSAT).\n\nThinkDiff **extends prior work** by introducing a novel alignment paradigm that allows diffusion models to reason over multimodal inputs rather than simply reconstructing images.", "essential_references_not_discussed": "The paper thoroughly cites relevant prior work but could benefit from discussing:\n\n- **Work on vision-language fine-tuning paradigms** (e.g., InstructBLIP, Kosmos-G).\n- **More literature on feature space alignment in deep learning** (e.g., feature alignment in contrastive learning).\n\nExplicitly comparing ThinkDiff’s **alignment approach** to **existing multimodal fusion techniques** could further strengthen the paper.", "other_strengths_and_weaknesses": "#### **Strengths**:\n\n1.  **Novel multimodal alignment framework**, enabling in-context reasoning in diffusion models.\n2.  **Significant improvement over baselines**, achieving SOTA performance on CoBSAT.\n3.  **Computational efficiency**, requiring fewer GPUs and training hours than competing methods.\n4.  **Clear experimental design**, including thorough ablation studies and efficiency comparisons.\n\n#### **Weaknesses**:\n\n1.  **Limited evaluation on real-world datasets**—performance on **other multimodal tasks** like captioning or VQA is unclear.\n2.  **Theoretical analysis of feature space alignment is minimal**—a deeper mathematical justification could be beneficial.\n3.  **Only tested on a single benchmark (CoBSAT)**—generalizability to other multimodal datasets remains an open question.", "other_comments_or_suggestions": "- The **writing is clear and well-structured**, making it easy to follow the key ideas.\n- The **figures effectively illustrate the concept of multimodal in-context reasoning**.\n- Additional **comparison to multimodal alignment techniques** (e.g., how ThinkDiff differs from IP-Adapter) would be useful.", "questions_for_authors": "1. **Generalization beyond CoBSAT**:\n   - How well do you expect ThinkDiff to generalize to **other multimodal tasks** beyond image generation (e.g., estimation, segment)?\n2. **Scalability to other models**:\n   - How does ThinkDiff perform with **other diffusion models** (e.g., SDXL, DeepFloyd IF)?\n   - Does scaling the LVLM improve results, or does performance saturate?\n3. **Feature space alignment analysis**:\n   - Have you conducted any **quantitative analysis** on the feature space alignment between the **VLM, LLM decoder, and diffusion decoder**?\n   - Could a **contrastive learning-based loss** further improve alignment?\n4. **Ablation on different VLMs**:\n   - Did you test **other VLMs** beyond Qwen2-VL and CLIP?\n   - Would a **stronger LVLM (e.g., GPT-4V)** further improve results?\n5. **Failure cases and limitations**:\n   - Can you provide qualitative examples of **failure cases** where ThinkDiff struggles?\n   - What are the **main sources of error**, and how could future work address them?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MdaswoW2t6", "reviewer": "Reviewer_j31i", "summary": "This paper presents ThinkDiff, a framework that can efficiently and effectively align VLMs with diffusion models. Specifically, the framework design is inspired by the findings that latest diffusion models either use LLMs (e.g., T5) or CLIP as text encoders to guide the output image/video generation, so the paper aligns the embeddings from VLM with the encoded text embeddings with diffusion text encoder. Experiment results on in-context reasoning dataset (CoBSAT) has shown the effectiveness of the proposed ThinkDiff-VLM and ThinkDiff-CLIP models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer j31i**\n\nWe thank Reviewer j31i for the insightful comments and suggestions. We provide our responses as follows and will add additional literature in the reference.\n\n>**Q1: Why alignment?**\n\nThere are several advantages in terms of the alignment compared to text-based methods on the multimodal generation task. We elaborate on them below:\n\n1. An important aspect of multimodal generation is to capture multimodal information in the generator. The aligner in our method is differentiable. It can convey detailed multimodal information that cannot be expressed by texts effectively. However, the text-based methods discard the detailed multimodal information in the middle.\n2. The model is an end-to-end model. The submodels in it can be optimized together to benefit each other.\n3. The features passed through the aligner are more representative and can convey more dense spatial context information than sparse texts.\n\n> **Q2: Text-based evaluation.**\n\nThe CoBSAT benchmark mainly focuses on the correctness of logical reasoning. To better validate the benefits of the alignment that can convey more multimodal information, we evaluate the image-condition generation of ThinkDiff-LVLM on COCO, as shown in the table below. The input of ThinkDiff-LVLM is an image. The input of the text-based method is the texts generated by the LVLM. As shown in this table, the alignment performs better on the CLIP image metric (CLIP-I) and the FID score. This experiment further validates that with the proposed differentiable aligner and dense representations, more multimodal information can be passed to the diffusion decoder, which is important for multimodal generation tasks.\n\n| Metric      | CLIP-I$\\uparrow$ | FID$\\downarrow$   |\n|-------------|------------|-----------|\n| Alignment   | **0.744**  | **65.8**  |\n| Text-based  | 0.728      | 66.3      |\n\n\n\n\n\n> **Q3: Evaluation benchmarks.**\n\nWe evaluate on more datasets such as the COCO, GenEval and DPG-Bench.\n\nThe GenEval:\n\n| Model       | Emu   | SEED-LLaMA | Ours      |\n|-------------|-------|------------|-----------|\n| GenEval$\\uparrow$ | 3.25  | 35.35      | **39.13** |\n\nThe DPG-Bench:\n\n| Model       | Emu   | SEED-LLaMA | Ours      |\n|-------------|-------|------------|-----------|\n| DPG-Bench$\\uparrow$ | 12.4  | 47.3       | **54.8**  |\n\nThe COCO results are in Q2 and Reviewer BZ6o Q1.\n\nCompared to the previous arts, ThinkDiff-LVLM achieves notable gains in all benchmarks.", "claims_and_evidence": "Yes, the claims that ThinkDiff is able to do in-context reasoning is well-supported.", "methods_and_evaluation_criteria": "Yes, the paper evaluate the in-context reasoning ability on CoBSAT dataset.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "- The experiment for in-context reasoning ability on CoBSAT is sound. However, an important 2-step baseline that first uses the VLM to do in-context learning and output the answer in text format, then give this text as input to the diffusion text encoder is not included. This baseline is essential to prove that aligning VLM with diffusion model is better than first use VLM to do in-context reasoning then use diffusion to do generation. I would doubt the necessity of doing such VLM-Diffusion alignment as ThinkDiff if no justification on this point is provided.\n\n- It would make the paper stronger if more evaluation benchmarks on image generation are included, such as GenEval, DPG-Bench.", "relation_to_broader_scientific_literature": "Prior works on diffusion models usually focus on generating high-fidelity images/videos, while not focusing too much on the reasoning ability. There are some recent concurrent works, such as LanDiff [1], start to explore the same direction of adding reasoning abilities to diffusion models.\n\n\n[1] Yin, Aoxiong, et al. \"The Best of Both Worlds: Integrating Language Models and Diffusion Models for Video Generation.\" arXiv preprint arXiv:2503.04606 (2025).", "essential_references_not_discussed": "The references looks complete to me.", "other_strengths_and_weaknesses": "- Strength: the idea of aligning VLM with the text embeddings of diffusion text encoder is novel \n\n- Weakness: As discussed above, a major weakness of this paper is the lack of proof that why we need such alignment instead of just use VLM to do reasoning and use diffusion to do generation separately. If the authors cannot provide a convincing benefits, the necessity of such alignment could be doubtful.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}]

Author rebuttal:

Meta-review: All reviewers found the proposed method effective and the results promising. The rebuttal successfully addressed reviewer comments, and all reviewers recommended acceptance. The authors are encouraged to improve the final version by incorporating reviewer suggestions – particularly by adding further results, providing thorough explanations, and offering in-depth discussions of the method’s advantages and limitations compared to state-of-the-art approaches, as provided in the rebuttal. 

Notably, Reviewer j31i, while leaning slightly toward acceptance, raised two remaining concerns to be addressed in the revision: (1) some benchmark methods may not have been specifically trained or fine-tuned on the CoBSAT dataset, potentially making comparisons less fair; and (2) additional evaluation on GenEval and DPGBench suggests that the method’s strength lies more in reasoning than in precise prompt following, as acknowledged by the authors – these should be clearly articulated in the final version.
Decision: Accept (poster)