Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
Authors: Chunting Zhou
LILI YU
Arun Babu
Kushal Tirumala
Michihiro Yasunaga
Leonid Shamis
Jacob Kahn
Xuezhe Ma
Luke Zettlemoyer
Omer Levy
Venue: ICLR 2025 Oral (2025)
Keywords: multimodal foundation model
multimodal generation and understanding
diffusion
next token prediction
Primary area: foundation or frontier models, including LLMs
TL;DR: Transfusion is a recipe for training a multi-modal model over discrete and continuous data.

Abstract: We introduce Transfusion, a recipe for training a multi-modal model over discrete and continuous data.
Transfusion combines the language modeling loss function (next token prediction) with diffusion to train a single transformer over mixed-modality sequences.
We pretrain multiple Transfusion models up to 7B parameters from scratch on a mixture of text and image data, establishing scaling laws with respect to a variety of uni- and cross-modal benchmarks.
Our experiments show that Transfusion scales significantly better than quantizing images and training a language model over discrete image tokens.
By introducing modality-specific encoding and decoding layers, we can further improve the performance of Transfusion models, and even compress each image to just 16 patches.
We further demonstrate that scaling our Transfusion recipe to 7B parameters and 2T multi-modal tokens produces a model that can generate images and text on a par with similar scale diffusion models and language models, reaping the benefits of both worlds.

Published as a conference paper at ICLR 2025
TRANSFUSION: PREDICT THE NEXT TOKEN AND DIFFUSE IMAGES WITH ONE MULTI-MODAL MODEL
Chunting "hou"*
Lili Yu"
Arun Babu"
Kushal Tirumala"
Michihiro Yasunaga"
Leonid Shamis"
Jacob Kahn"
Xuezhe Ma
Luke Zettlemoyer
Omer Levy"
Work done at Meta University of Southern California
ABSTRACT
We introduce Transfusion, recipe for training multi-modal model over discrete and continuous data. Transfusion combines the language modeling loss function (next token prediction) with diffusion to train a single transformer over mixed- modality sequences. We pretrain multiple Transfusion models up to 7B parameters from scratch on a mixture of text and image data, establishing scaling laws with respect to variety of uni- and cross-modal benchmarks. Our experiments show that Transfusion scales significantly better than quantizing images and training a language model over discrete image tokens. By introducing modality-speciffee coding and decoding layers, we can further improve the performance of Transfusion models, and even compress each image to just 16 patches. We further demonstrate that scaling our Transfusion recipe to 7B parameters and 2T multi-modal tokens produces model that can generate images and text on par with similar scale diffusion models and language models, reaping the benefits of both worlds.
INTRODUCTION
Multi-modke generative models need to be able to perceive, process, and produce both discrete elements (such as text or code) and continuous elements (e.g. image, audio, and video data). While language models trained on the next token prediction objective dominate discrete modalities (OpenAl al. 2024, Dubey et al. (2024), diffusion models (Ho et al.) [2200] Rombach et al.) (022a) and their generalizations (Lipman et al. (022)) are the state of the art for generating continuous modalities (Dai al. 2023 Esser et al. 2024b Bar-Tal et al. 2024). Many efforts have been made to combine these approaches, including extending a language model to use a diffusion model as tool, either explicitly
(Liu et al. 2023) or by grafting a pretrained diffusion model onto the language model (Dong et al. 2023: Koh et al., 2024). Alternatively, one can quantize the continuous modalities (Van Den Oord cet al. 2017 7) and train a standard language model over discrete tokens (Ramesh et al. 2021] Yu et al. 2023). simplifying the model's 's architecture at the cost of losing information. In this work, we show it is possible to fully integrate both modalities, with no information loss, by training a single model to both predict discrete text tokens and diffuse continuous images.
2022:
We introduce Transfusion, a recipe for training a model that can seamlessly generate discrete and continuous modalities. We demonstrate Transfusion by pretraining transformer model 50% text and 50% image data using a different objective for each modality: next token prediction for text and diffusion for images. The model is exposed to both modalities and loss functions at each training step. Standard embedding layers convert text tokens to vectors, while patchification layers represent each image sequence of patch vectors. We apply causal attention for text tokens and bidirectional attention for image patches. For inference, we introduce a decoding algorithm that combines the standard practices of text generation from language models and image generation from diffusion models. Figure illustrrtte Transfusion.
In a controlled comparison with Chameleon's 's discretization approach CChameleon Team 2024). we show that Transfusion models scale better in every combination of modalities. In text-to-image
"qqual contribution.

---

Published as a conference paper at ICLR 2025
cute
cat
<BOL>
What
color
is
its
nose
Transformer
cute
cat
<BOL>
<EOI>
What
color
is
its
nose
Figure high-level illustration Transfusion. single transformer perceives, processes, and produces data of every modality. Discrete (text) tokens are processed autoregressively and trained on the next token prediction objective. Continuous (image) vectors are processed together in parallel and trained on the diffusion objective. Marker BOI and EOI tokens separate the modalities.
generation, we find that Transfusion exceeds the Chameleon approach at less than a third of the compute, measured by both FID and CLIP scores. When controlling for FLOPs. Transfusion achieves approximately 2x lower FII) scores than Chameleon models. We observe a similar trend in image-to-text generation, where Transfusion matches Chameleon at 21.8% of the FLOPs. Surprisingly, Transfusion is also more efficient at learning text-to-text prediction, achieving perplexity parity on text tasks around 50% to 60% of Chameleon' FLOPs.
Ablation experiments reveal critical components and potential improvements for Transfusion. We observe that the intra-image bidirectional attention important, and that replacing it with causal attention hurts text-to-image generation. We also find that adding U-Net down and up blocks to encode and decode images enables Transfusion compress larger image patches with relatively small loss to performance, potentially decreasing the serving costs by up to 64x.
Finally, we demonstrate that Transfusion can generate images similar quality to other diffusion models. We train from scratch 7B transformer enhanced with U-Net down/up layers (0.27B parameters) over 2T tokens: IT text tokens, and approximately epochs of 692M images and their captions, amounting to another IT patches/tokens. Figure shows some generated images sampled from the model. the GenEval (Ghosh et al., 2023) benchmark, our model outperforms other popular models such DALL-E and SDXL; unlike those image generation models, it can generate text, reaching the same level of performance as Llama I on text benchmarks. Our experiments thus show that Transfusion is promising approach for training truly multi-modal models.
BACKGROUND
Transfusion is a single model trained with two objectives: language modeling and diffusion. Each of these objectives represents the state of the in discrete and continuous data modeling, respectively. This section briefly defines these objectives, as well as background on latent image representations.
2.1 LANGUAGE MODELING
Given a sequence of discrete tokens y = y/,,... yo. from a closed vocabulary V. a language model predicts the probability of the sequence P(y). Standard language models decompose P(y) into product of conditional probabilities I[: Pe (yx (9<4). This creates an autoregressive classification task, where the probability distribution of each token is predicted conditioned the prefix of sequence Yev using a single distribution Po parameterized by 9. The model can be optimized by minimizing the cross-entropy between Py and the empirical distribution of the data, yielding the standard next-token prediction objective, colloquially referred to as LM loss:
LIM Ey -- Σ log (o
Once trained, language models also be used to generate text by sampling token by token from the model distribution Ps, typically using temperature and top-p truncation.
2.2 DIFFUSION
Denoising diffusion probabilistic models ka. DDPM diffusion models) operate on the principle of learning to reverse a gradual noise-addition process (Ho et al.) (020). Unlike language models that typically work with discrete tokens (y)) diffusion models operate continuous vectors (x), making
2

---

from Xo using single sample of Published as a conference paper at ICLR 2025
(8)
(b)
(c)
(d)
Trans usion
(c)
(f)
(h)
Figure 2: Generated images from 7B Transfusion trained on 2T multi-modal tokens. Captions are: (a) A bread, an apple, and knife on a table. (b) A corgi. (c) Three spheres made of glass falling into ocean. Water is splashing. Sun is setting. (d) A wall in royal castle. There two paintings the wall. The one on the left a detailed oil painting the royal raccoon king. The one on the right detailed oil painting of the royal raccoon queen. (e) A kangaroo holding beer, wearing ski goggles (f) "Transfusion" bird made of wheat bread. (h) cloud in the shape of two bunnies playing with ball. The ball is made of clouds too.
and passionately singing silly songs.
written on the blackboard. (g) an egg and a
them particularly suited for tasks involving continuous data like images. The diffusion framework involves processes: forward process that describes how the original data is turned into noise, and reverse process of denoising that the model learns to perform.
Forward Process From mathematical perspective, the forward process defines how the noised data (which serves as the model input) is created. Given a data point X Ho et al. (2020) define a Markov chain that gradually adds Gaussian noise over T steps, creating sequence of increasingly noisy ver- sions X1, X., Xr. Each step of this process is defined by ((x:/xr-1) where 3e increases over time according predefined noise schedule (see below). This process can be reparameterized in way that allows us to directly sample x Gaussian noise IO,
N(x:: /1 Bext- 3,I).
X; = Vā + -1-
Here, 04 = IT' B.), providing a useful abstraction over the original Markov chain. In fact, both the training objective and the noise scheduler eventually expressed (and implemented) in these terms.
The diffusion model is trained perform the reverse process Po (x [xe). learning to denoise the data step by step. There are several ways to do so; in this work, we follow the approach Ho et (2020) and model the Gaussian noise in Equation as proxy for the cumulative noise at step L. Specifically, a model (-(-) with parameters is trained to estimate the noise given the noised data x; and timestep In practice, the model often conditions on additional contextual information c, such as caption when generating an image. The parameters of the noise prediction model are thus optimized by minimizing the mean squared error loss:
Reverse Process
LDDPM = Exo. [||e Ex(Xe, t, c)
Noise Schedule When creating a noised example x, (Equation ãg determines the variance the noise for timestep f. In this work, adopt the commonly used cosine scheduler Nichol Dhariwal (2021), which largely follows /or zs cos( f I with some adjustments.
/)

---

Published as a conference paper at ICLR 2025
<B01>
<EOl>What
VAE Decoder
A
Linear
or
U-Net Up
Transformer
Linear
U-Net Down
Noising
VAE Encoder
<EOl>
Figure We convert images and from latent representations using a pretrained VAE, and then into patch representations with either simple linear layer or U-Net down blocks.
What
Figure Expanding on the causal mask, Trans- fusion allows patches of the same image to con- dition on each other.
Inference Decoding is done iteratively, pealing away some of the noise at each step. Starting with pure Gaussian noise at Xy, the model co (x,, f, c) predicts the noise accumulated at timestep
The predicted noise is then scaled according the noise schedule, and the proportional amount of predicted noise is removed from x, to produce X;-1- In practice, inference is done over fewer timesteps than training. Classifier-fre frefieer-free guidance (CFG) (Ho & Salimans 2022) is often used to improve generation by contrasting the prediction of the model conditioned the context with the unconditioned prediction, at the cost doubling the computation.
2.3 LATENT IMAGE REPRESENTATION
Early diffusion models worked directly in pixel space (Ho al. 2020). but this proved computation- ally expensive. Variational autoencoders (VAEs) (Kingma & Welling 2013) can save compute by encoding images into lower-dimensional latent space. Implemented as deep CNNs, modern VAEs are trained on combination of reconstruction and regularization losses (Esser et al. (02T). allowing downstream models like latent diffusion models (LDMs) (Rombach et al. on compact image patch embeddings; e.g. represent every 8x8 pixel patch an 8-dimensional
2022a) to operate efficiently
vector. For autoregressive language modeling approaches (Ramesh et al, 2021 Yu et al. (022).. images must be discretized. Discrete autoencoders, such as rector-quantized VAES (Q-VAE) (Van Den Oord et al. 2017). achieve this by introducing a quantization layer (and related regularization losses) that maps continuous latent embeddings discrete tokens.
TRANSFUSION
Transfusion a method for training a single unified model to understand and generate both discrete and continuous modalities. Our main innovation is demonstrating that use separate losses for different modalities language modeling for text, diffusion for images over shared data and parameters. Figure illustrates Transfusion.
Data Representation We experiment with data spanning two modalities: discrete text and continu- ous images. Each text string tokenized into a sequence of discrete tokens from fixed vocabulary, where each token is represented as an integer. Each image is encoded as latent patches using VAE (see $2.3 where each patch represented as continuous vector; the patches are sequenced left-to-right hop-to-bottom to create sequence of patch vectors from each image. For mixed-modal examples, we surround each image sequence with special beginning of image (BOI) and end of image (EOI) tokens before inserting it to the text sequence; thus, arrive single sequence potentially containing both discrete elements (integers representing text tokens) and continuous elements (vectors representing image patches).
4

---

Published as a conference paper at ICLR 2025
Model Architecture The vast majority of the model's parameters belong to single transformer, which processes every sequence, regardless of modality. The transformer takes a sequence of high- dimensional vectors in R' as input, and produces similar vectors output. To convert our data into this space, we use lightweight modality-specific components with unshared parameters. For text, these are the embedding matrices, converting each input integer to vector space and each output vector into a discrete distribution over the vocabulary. For images, we experiment with two alternatives for compressing local windows of k x k patch vectors into a single transformer vector (and vice versa): (I) simple linear layer, and (2) up and down blocks of a U-Net (Nichol & Dhariwal 2021 Saharia al. 2022). Figure BarreJelllnn Billsstrates the overall architecture.
Transfusion Attention Language models typically use causal masking to efficiently compute the loss and gradients over an entire sequence in a single information from future tokens. While text is naturally sequential, images are not, and are usually modeled with unrestricted (bidirectional) attention. Transfusion combines both attention patterns by applying causal attention to every element in the sequence, and bidirectional attention within the elements of each individual image. This allows every image patch to attend to every other patch within the same image, but only attend to text or patches of other images that appeared previously in the sequence. We find that enabling intra-image attention significantly boosts model performance (see $4.3) Figure shows an example Transfusion attention mask.
pass without leaking
Training Objective To train our model, we apply the language modeling objective LIM to pre- dictions of text tokens and the diffusion objective LDDPM to predictions of image patches. LM loss is computed per token, while diffusion loss is computed per image, which may span multiple elements (image patches) in the sequence. Specifically, we add noise e to each input latent image Xo according to the diffusion process to produce X₁ before patchification, and then compute the image-level diffusion loss We combine the two losses by simply adding the losses computed over each modality with balancing coefficient X:
Transtusion LLM + A LDDPM
(4)
This formulation is a specific instantiation of a broader idea: combining a discrete distribution loss with continuous distribution loss to optimize the same model. We leave further exploration of this space, such as replacing diffusion with flow matching (Lipman et al. 2022), to future work.
Inference Reflecting the training objective, our decoding algorithm also switches between two modes: LM and diffusion. In LM mode, we follow the standard practice of sampling token by token from the predicted distribution. When we sample a BOI token, the decoding algorithm switches to diffusion mode, where we follow the standard procedure of decoding from diffusion models. Specifically, we append a pure noise Xr in the form of n image patches to the input sequence (depending on the desired image size), and denoise over T' steps. At each step t. we take the noise prediction and use it to produce x;-1, which then overwrites x; in the sequence; i.c. the model always conditions on the last timestep of the noised image and cannot attend to previous timesteps. Once the diffusion process has ended, we append an EOI token to the predicted image, and switch back to LM mode. This algorithm enables the generation of any mixture of text and image modalities.
4 EXPERIMENTS
We demonstrate in series of controlled experiments that Transfusion is viable, scalable method for training a unified multi-modal model. The setup of our experiments is detailed in Appendix B.
4.1 SETUP
Evaluation We evaluate model performance on collection of standard uni-modal and cross- modal benchmarks (Table Appendix). For text-to-text text, we measure perplexity on 20M held-out tokens from Wikipedia and the C4 corpus (Raffel et al., 2019), as well as accuracy on the pretraining evaluation suite of lllama 2 (Touvron et al.) 2023b). For text-to-image, we use the MS-COCO benchmark (Lin et al., 2014), where we generate images on randomly selected 30k prompts from validation set and measure their photo-realism using zero-shot Frechet Inception Distance (FID)
Ergo, downstream tokens condition on noisy images during training. See B.2 for further discussion.
5

---

Published as a conference paper at ICLR 2025
(Heusel et al. 2017) well as their alignment with the prompts using CLIP score RRadford et al. 2021) We also evaluate the model's 's ability to generate image captions; we report CIDEr (Vedantam et al. 2015) scores on the Karpathy test split of MS-COCO (Lin et al. 2014). These evaluations
provide signal for investigation of scaling laws (.2) and ablations ($4.3) To compare with recent work in diffusion models, we evaluate largest scale model ($4.4) also on GenEval (Ghosh et al.]
2023), a benchmark that examines a model's 's ability to generate an accurate depiction of the prompt.
Baseline At the time of writing, the prominent open-science method for training single mixed- modal model that can generate both text and images is to quantize images into discrete tokens, and then model the entire token sequence with a standard language model (Ramesh et al.) 202T Yu iet al. 2022 2023). We follow the recipe of Chameleon (Chameleon Team 2024) to train a family of data- and compute-controlled baseline models, which we can directly compare our Transfusion models. The key difference between Chameleon and Transfusion is that while Chameleon discretizes images and processes them as tokens, Transfusion keeps images in continuous space, removing the quantization information bottleneck. To further minimize any confounding variables, we train the VAES for Chameleon and Transfusion using exactly the same data, compute, and architecture, with the only differentiator being the quantization layer and codebook loss of Chameleon's 's VQ-VAF (see details below). Chameleon also deviates from the Llama transformer architecture, adding query-key normalization, post-normalization, denominator loss, and lower learning rate of le-4 to manage training instability, which incur an efficiency cost (see $42
Data For almost all of our experiments, we sample 0.5T tokens (patches) from two datasets a 1:1 token ratio. For text, we use the Llama 2 tokenizer and corpus (Touvron et al. 2023b) of 2T tokens. For images, we use a collection of 380M licensed Shutterstock images and captions. Each image is center-cropped and resized to produce a 256 256 pixel image. We randomly order the image and captions, ordering the caption first 80% of the time.
In one experiment 3.5B caption-image pairs 256 patches per image). To diversify, we add 220M publicly available images with captions, prefiltered to not contain people. To rebalance the distribution, we upsample 80M Shutterstock images containing people. We also add data from Conceptual 12M (CC12M) (Changpinyo et al. 2021). reaching total mixture of 692M image-caption pairs per epoch. Finally, we upweight the portion of high-aesthetic images in the last 1 % of the training schedule.
we scale up the total training data to 2T tokens (IT text tokens and about
Latent Image Representation We train a 86M parameter VAE following Esser et al. (002)) We use CNN encoder and decoder, and latent dimension 8. The training objective is combines reconstruction and regularization losses ( (Appendix[ C For VQ-VAE training, we follow the same setup described for VAE training, except we replace LXL with the standard codebook commitment loss with B = 0.25 Van Den Oord et al. 2017). We use a codebook of 16,384 token types.
Model Configuration 0.37B, 0.76B, 1.4B, and 7B parameters - following the standard settings from Llama (Touvron et al. 2023a) (Table in Appendix). In configurations that use linear patch encoding ($4.2 and $4.3). the number of additional parameters is insignificant, accounting for fewer than 0.5% of total parameters in every configuration. When using U-Net patch encoding ($4.3 and $4.4), these parameters add up to 0.27B additional parameters across all configurations, while this is substantial addition of parameters to smaller models, these layers amount to only a 3.8% increase of the 7B configuration, almost identical to the number of parameters in the embedding layers.
To investigate scaling trends, we train models at five different sizes 0.160.
Optimization We use AdamW (3, =0.9, 32 =0.95, c - with learning rate of 3e-4, warmed up for 4000 steps and decaying to 1.5e-5 using cosine scheduler. We train on sequences of 4096 tokens in batches of 2M tokens for 250k steps, reaching 0.5T tokens in total. In our large-scale experiment ($4.4), we train with batch size of 4M tokens over 500k steps, totalling 2T tokens. We set the coefficient in the Transfusion objective (Equation 4 to following preliminary experiments.
Inference In text mode, we use greedy decoding for generating text. Ranked classification is used for the Llama evaluation suite. For image generation, we follow the standard of 250 diffusion steps (the model is trained on 1,000 timesteps). We follow Chameleon and use CFG with a coefficient of 5
We follow common practice for ablations and use only 5k examples to compute FID and CLIP in $43 Removing these deviations in preliminary experiments encountered optimization instabilities in Chameleon.

---

Published as a conference paper at ICLR 2025
16
Transfusion Chameleon
Transfusion Chemeleon
16
PR.
Transfusio Chameleon
20720
1C123 FLOPs
22
10
16923
12122
20
Ic=22
FLOPs
C4 Perplexity
Wikipedia Perplexity
Llama 2 Eval Suite Accuracy
129
Transfusion Chameleon
Transfusion Chemeleon
Transfusion Chameleon
FLONS
FLOP
PLDIS
MS-COCO CIDEr
MS-COCO 30k FID
MS-COCO 30k CLIP
Figure 5: Performance of Transfusion and Chameleon models at different scales, controlled for parameters, data, and compute. All axes are logarithmic.
in the controlled comparison experiments ($4.2) This value is suboptimal for Transfusion, and so we use a CFG coefficient throughout the ablation experiments (64.3) and follow the standard practice of tuning the coefficient for each benchmark in our large scale experiment ($4.4)
4.2 CONTROLLED COMPARISON WITH CHAMELEON
We run a series of controlled experiments to compare Transfusion with Chameleon at different model sizes (N) and token counts (D), using the combination of both as a proxy for FLOPs (6ND). For simplicity and parameter control, the Transfusion variant in these experiments uses simple linear image encoder/decoder with patch size 2x2, as well as bidirectional attention. For each benchmark, plot all results log-metric over log-FLOPP curve and regress linear trendlines. We also estimate relative compute efficiency by measuring the parity FLOP ratio: the ratio between the number FLOPs required by Transfusion and Chameleon reach the same level of performance.
Figure 5 visualizes the scaling trends, and Table T shows the results of the largest models in this controlled setting and their estimated parity FLOP ratio. In every benchmark, Transfusion consistently exhibits better scaling laws than Chameleon. While the lines are close to parallel, there is a significant gap Transfusion's favor. The difference in compute efficiency is particularly striking in image generation, where FID Transfusion achieves parity with Chameleon using 34x less compute.
Surprisingly, text-only benchmarks also reveal better performance with Transfusion, even though both Transfusion and Chameleon model text in the same way. We investigate this phenomenon by ablating the various changes leading up Transfusion and Chameleon from the original Llama recipe. Table shows that while Transfusion does come at non-zero cost to text performance, the Chameleon recipe suffers from both the stability modifications made to the architecture and from the introduction of image tokens. Training on quantized image tokens degrades text performance
C4
Wiki PPL (4)
Liama
MS-COCO
Model
PPL ({)
Ace (?)
CDr (1)
FID (L)
CLIP (1)
Transfusion
7.72
4.28
61.5
27.2
16.8
25.5
Chameleon
8.41
4.69
59.1
18.0
29.6
24.3
Parity FLOP Ratio
0.489
0.526
0.600
0.218
0.029
0.319
Table 1: Performance of the largest (7B) Transfusion and Chameleon models in controlled setting. Both models were trained on 0.5T tokens. Parity FLOP Ratio is the relative amount of Transfusion FLOPs needed match the results of Chameleon 7B.

---

Published as a conference paper at ICLR 2025
C4 PPL (4)
Wiki PPL ())
Llama
Model
Batch
Ace (f)
Llama
Text Tokens
10.1
5.8
53.7
Transfusion
+ Diffusion
IM Image Patches
(+0.3) 10.4
(+0.2) 6.0
(-1.0) 52.7
Chamelcon
Stability Modifications
1M Text Tokens
(+0.9) 11.0
(+0.5) 6.3
(-1.8) 51.9
+ Loss on Image Tokens
1M Image Tokens
(+0.8) 11.8
(+0.5)
(-3.0) 48.9
Table 2: Performance of the 0.76B Transfusion and Chameleon models on text-only benchmarks, compared to the original Llama recipe.
C4 PPL (L)
Wiki PPL (1)
Llama Acc (4)
MS-COCO
Enc/Dee
Attention
CDr (1)
FID (1)
CLIP (t)
Causal
10.4
6.0
51.4
12.7
61.3
23.0
Linear
Bidirectional
10.4
6.0
51.7
16.0
20.3
24.0
Causal
10.3
5.9
52.0
23.3
16.8
25.3
U-Net
Bidirectional
10.3
5.9
51.9
25.4
16.7
25.4
Table 3: Performance of 0.76B Transfusion models with and without intra-image bidirectional attention. Patch size is set 2x2 latent pixels.
more than diffusion on all three benchmarks. One hypothesis is that this stems from the competition between text and image tokens in the output distribution; alternatively, is possible that diffusion is more efficient at image generation and requires fewer parameters, allowing Transfusion models to use more capacity than Chameleon to model text. We leave further investigation this phenomenon to future research.
4.3 ARCHITECTURE ABLATIONS
We explore improvements and extensions that are applicable to Transfusion alone in this section.
4.3.1
ATTENTION MASKING
We first examine the necessity of intra-image bidirectional attention. Table shows that enabling this attention pattern beyond the standard causal attention is advantageous throughout all benchmarks, and using both image encoding/decoding architectures. In particular, we notice significant improvement in FID when using linear encoding layers (61.3-+0..3, In the causal-only version of this architecture, there is no flow information from patches that appear later in the sequence to those before; since U-Net blocks contain bidirectional attention within, independent of the transformer's attention mask, this gap less pronounced when they are applied.
4.3.2 PATCH SIZE
Transfusion models can be defined over different sizes of latent pixel patches. Larger patch sizes allow the model to pack more images in each training batch and dramatically reduce inference compute, but may come performance cost. Table 4 and Table in Appendix sheds light these performance trade-offs. While performance does decrease consistently as each image is represented by fewer patches with linear encoding, models with U-Net encoding benefit from larger patches on tasks involving the image modality. We posit that this is due to the greater amount of total images (and diffusion noise) seen during training. We also observe that text performance deteriorates with larger patches, perhaps because transfusion needs to exert more resources (i.e. parameters) to learn how to process images with fewer patches and thus less inference compute.
4.3.3 PATCH ENCODING/DECCDING ARCHITECTURE
Our experiments far indicate an advantage to using the U-Net up and down blocks instead of simple linear layer. One possible reason is that the model benefits from the inductive biases of the U-Net architecure; alternative hypothesis is that this advantage stems from the significant increase in overall model parameters introduced by the U-Net layers. To decouple these two confounders, scale up the core transformer to 7B parameters, while keeping the amount of U-Net parameters
8

---

Published as a conference paper at ICLR 2025
Latent/ Patch
Pixel/ Patch
Patch/ Image
C4 PPL (4)
Wiki PPL ())
Liama
MS-COCO
Enc/Dec
Acc (4)
CDr (f)
FID (L)
CLIP (1)
None
1x1
8x8
1024
10.3
52.2
12.0
21.0
24.0
2x2
16x
256
10.3
51.9
25.4
16.7
25.4
U-Net
4x4
32x32
64
10.7
6.2
50.7
29.9
16.0
25.7
8x8
64x64
16
11.4
6.6
49.2
29.5
16.1
25.2
Table 4: Performance of 0.76B Transfusion models with different patch sizes. Bolded figures indicate global best, underlines indicate best within architecture.
Model Params
Enc/Dec Params
C4 PPL (J)
Wiki PPL (J)
Llama Ace (t)
MS-COCO
Ene/Dec
CDr (t)
FID (1)
CLIP (1)
Linear
0.4%
12.0
7.0
47.9
11.1
21.5
22.4
0.37B
U-Net
71.3%
11.8
6.9
48.8
21.1
18.1
24.9
Linear
0.4%
9.5
5,4
53.8
19.1
19.4
24.3
1.4B
U-Net 19.3%
9.4
5.4
53.4
28.1
16.6
25.7
Linear 0.3%
7.7
4.3
61.5
27.2
18.6
25.9
7B
U-Net
3,8%
7.8
4.3
61.1
33.7
16.0
26.5
Table 5: Performance of linear and U-Net variants of Transfusion across different model sizes. Patch size is set at 2x2 latent pixels. Model parameters refers to the transformer alone.
(almost) constant; in this setting, the additional encoder/decoder parameters account for only 3.8% increase of total model parameters, equivalent to the amount of token embedding parameters.
Table Table 10 (Appendix) shows that even though the relative benefit U-Net layers shrinks the transformer grows, it does not diminish. In image generation, for example, the U-Net encoder/decoder allows much smaller models to obtain better FID scores than the 7B model with linear patchification layers. We observe similar trend in image captioning, where adding U-Net layers boosts the CIDEr score of a 1.4B transformer (1.67B combined) beyond the performance of the linear 7B model. Overall, it appears that there are indeed inductive bias benefits U-Net encoding and decoding of images beyond the mere addition of parameters.
4.4 COMPARISON WITH IMAGE GENERATION LITERATURE
Our experiments thus far have covered controlled comparisons with Chameleon and Llama, but have yet to compare Transfusion's image generation capabilities to those of state-of- state-of-the-art the-art image generation models. To that end, train 7B parameter model with U-Net encoding/decoding layers (2x2 latent pixel patches) over the equivalent of 2T tokens, comprising of 1T text corpus tokens and 3.5B images and their captions. While the Transfusion variant in $4.2 favored simplicity and experimental control, the design choices and data mixture ($4.1) of this variant lean a bit more showease generated images from this model.
towards image generation. Figure and Appendix DeroonddxxDsooweee
We compare the performance of our model to reported results of other similar scale image generation models, as well as some publicly available text generating models for reference. Table shows that Transfusion achieves similar performance
high-perffrming image generation models such as DeepFloyd Stabilityy Al (2024), while surpassing previously published models including SDXL (Podell et al. 2023). While Transfusion does lag behind SD (Esser et al. 2024a), this model leveraged synthetic image captions through backtranslation (Betker et al. (023). which enhances its GenEval performance by 6.5% absolute (0.433 0.433) at smaller scale; for simplicity, our experimental setup only included natural data. Finally, we note that our Transfusion model can also generate text, and performs on par with the Llama models, which were trained on the same text data distribution (44.1)
4.5
IMAGE EDITING
Our Transfusion models, which have been pretrained on text-text, image-text, and text-image data, perform well across these modality pairings. Can these models extend their capabilities generate

---

Published as a conference paper at ICLR 2025
Model Params
Text Tokens
Llama Ace (f)
COCO
Gen
Model
Images
FID (
Eval (t)
Llama
Touvron
2023a)
7.1B
1.4T
66.1
Llama
Touvron el al.
2023b
7.1B
2.0T
66.3
Chameleon (C Con
Team
2024
7.1B
6.0T
5.0B
67.1
26.74
0.39
Imagen Saharia ct al.
2022)
2.6B + 4.7B"
5,0B
7.27
Parti Yu et al. (022)
20B
4.8B
77.23
SD 2.1 Rombach al.
2022b)
0.9B O.1I"
2.3B
0.50
DALL- (Ramesh et al.
2022
4.2B + 1B"
2.6B
10.39
0.52
SDXL (Podell et al. 2023
2.6B + 0.8B"
1.6B
0.55
DeepFloyd (Stability A1 2024
5.5B + 4.7B"
7.5B
6.66
0,61
SD 3
Esser et 2024b
8B 4.7B"
$2.0B
0.68
Transfusion (Ours)
7.3B
1.0T
3.5B
66.1
6.78
0.63
Table 6: Performance of a 7B Transfusion model (U-Net encoder/decoder layers, 2x2 latent pixel patches) trained on the equivalent of 2T tokens, compared similar scale models in the literature. Except Chameleon, all the other models are restricted to generating one modality (either text or image). Frozen text encoder parameters. Parti samples 16 images for every prompt and then reranks with an auxiliary scoring model. SD 3 trains with synthetic caption data, which provides boosts GenEval performance.
Remove cupcake on plate.
Change tomato on right to green olive.
Zebra
Write the word "Zebra" Arial bold.
Change this to cartoon style.
Figure Edited images from fine-tuned 7B Transfusion model.
images based on other images? To investigate, fine-tuned 7B model ($4.4) using dataset of only 8k publicly available image editing examples, where each example consists of an input image, an edit prompt, and an output image. This approach, inspired by LIMA (Zhou et al., (024). allows us to assess how well the model can generalize to image-to-image generation, a scenario not covered during pretraining. Manual examination of random examples from the EmuEdit test set (Sheynin) et al. 2024). shown in Figure 6 and Appendix 4.5) reveals that our fine-tuned Transfusion model performs image edits as instructed. Despite the limitations this experiment, the findings suggest that Transfusion models can indeed adapt to and generalize across new modality combinations. We leave further exploration of this promising direction to future research.
CONCLUSION
This work explores how to bridge the gap between the state of the art in discrete sequence modeling (next token prediction) and continuous media generation (diffusion). We propose a simple, yet objectives, tying each modality to
previously unexplored solution: train single joint model its preferred objective. Our experiments show that Transfusion scales efficiently, incurring little to no parameter sharing cost, while enabling the generation of any modality.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning, Advances in neural information processing systems, 35:23716 2022.
Omer Bar-Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12945 2024.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye. Augustus Odena, Arushi Somani, and Sağnak Taşırlar. Introducing our multimodal models, 2023. URL https: adept ai/
blog/fuyu-P0 8b
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving image generation with better captions, 2023. URL https
//api.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence. pp. 7432-7439, 2020.
Chameleon Team. Chameleon: Mixed-modal carly-fusion foundation models. arXiv preprint arXiv:2405.09875, 2024.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web- scale image-text pre-training recognize long-tail visual concepts. CoRR, abs/2102.08981, 2021.
URL https: http:://aaxxv...o
www.aaiiivv.... org/ abaaab/s/aaaa 08981
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.c 04297, 2020.
Christopher Clark, Kenton Lee, Ming-Wai Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 10044, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1xX3::1803005557 2018
Xiaoliang Dai, Ji Hou, Yao Ma, Sam Tsai, Jialiang Wang. Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309. 15807, 2023.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamlim: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.1 1499, 2023.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamlim: Synergistic multimodal comprehension and creation. In The Twelfth International Conference on Learning Representations, 2024.
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olan Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip
11

---

Published as a conference paper at ICLR 2025
Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hamnah Korevaar, Hu Xu, Hugo Touvron, Ilyann Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Yearala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Begoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer. Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh. Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang. Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra. Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujiwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vladan Petrovic. Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whimey Meers, Xavier Martinet, Xiaodong Wang. Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisemman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang. Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpem. Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Igor Molybog, Igor Tufanov, Irina-Flena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Bapticn Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White,
12

---

Published as a conference paper at ICLR 2025
Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswany, Shaun Lindsay, Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao. Yanjun Chen, Ye Hu, Ye Jia. Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He. Zach Rait, Zachary De Vito, Zef Rosnbrick. Zhaoduo Wen. Zhenyu Yang, and Zhiwei httptp://taxii...ttt//// /ttp:///aaivvvvi org/ abs wwwpaa//.2222
Zhao. The Ilama 3 herd of models, 2024. URL https:
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high- resolution image synthesis. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition. pp. 12873-12883, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024a.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling rectified flow trans- formers for high-resolution image synthesis, 2024b. Bttttts/////xiii 03206
2403
Itai Gat, Tal Remez, Neta Shaul, Felix Kreuk, Ricky TQ Chen, Gabriel Synnaeve, Yossi Adi, and Yaron Lipman. Discrete flow matching. arXiv preprint trXiv:2407.15595. 2024.
Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-ing text-to-image image alignment. Advances in Neural Information Processing Systems, 36, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12590 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36. 2024.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsumori Hashimoto. Diffusion-Im improves controllable text generation. ArXiv, abs/2205.144 14217, 2022.
13

---

Published as a conference paper at ICLR 2025
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02744,, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Shilong Liu, Hao Cheng. Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu. et al. Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:231 1.05437, 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162-8171, PMLR, 2021.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman. Diogo Almeida, Janko Altenschmidt. Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bermadett- Shapiro. Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, tukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano. Rajcev Nayak, Arvind Neelakantan. Richard Ngo. Hyconwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss. Carroll Wainwright, Justin Jay Wang, Alvin Wang. Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren
14

---

Published as a conference paper at ICLR 2025
Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang. Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL
fittps://arxie or org abw. 23777087777777444
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprini arXiv:2307.01952, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2 200: 200.002000000000 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910. 10683, 2019. httt::t://aaxttt org 1910. 10683
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmir, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. URL
/2204
06125 25
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10660 2022b.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-3666 -6449, 2022.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106,
2021.
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv: 09728, 2019.
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. In
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8871-8879, 2024.
Stability AI. If by deepfloyd lab stabilityai, 2024. URL https::/sttabllity. //ews/
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée
Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 00980888.. 2023b.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
15

---

Published as a conference paper at ICLR 2025
Ramakrishna Vedantam, C Lawrence Zitncck and Devi Parikh. Cider: Consensus-basee image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern 2015.
recognition, pp.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tniigg, arXiv preprint arXiv:2309.022. 12591, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL-2019). Association for Computational Linguistics, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
16

---

Published as a conference paper at ICLR 2025
Input
Output
Benchmark
Metric
Wikipedia
Perplexity (4)
Size
Layers
Emb Dim
Att Heads
Text
Text
Perplexity (4)
Llama Eval Suite
Accuracy (1)
0.16B
16
768
12
0.37B
24
1024
16
Image
Text
MS-COCO 5k
CIDEr (t)
0.76B
24
1536
MS-COCO 30k GenEval
FID (4). CLIP (1)
1.4B
2048
Text
Image
GenEval score (f)
7B
32
4096
32
Table 7: An overview of the evaluation suite used in this work.
Table 8:
Model sizes and configurations for both Transfusion and baselines.
A RELATED WORK
Most existing multi-modal models are built on the idea of attaching two or more modality-specific architectures together, often pretraining each component separately in advance. -fate-of-the-ar image and video generation models, for instance, use large pretrained text encoders to represent their input prompts latent space, which can then be used to condition diffusion models (Saharia et al.) 2022). In fact, recent work fuses representations from multiple off-the-shed encoders to enhance performance (Podell et al., 2023 Esser et al. 2024b). similar pattern can be observed in the vision language model literature, where typically pretrained language model is complemented by pretrained mopality-specific encoders/decoders via projection layers to/from the pretrained text space. Examples include Flamingo (Alayrac et al.) 2022) and LLaVA (Liu et al. (0224) for visual understanding, GILL (Koh et al.) (2024) for visual generation, and DreamLLM (Dong et al.) 2024)
for both visual comprehension and generation. In contrast, Transfusion has one unified architecture learned end-to-end to generate both text and images.
Prior work end-to-end multi-modal models includes examples such as Fuyu (Bavishi et al. 2023) which uses image patches as inputs for visual understanding, and Chameleon (Chameleon Team 2024), which converts each image to sequence of discretized tokens and then trains over the combined text-image token sequences. However, these approaches are either restricted to input- level multi-modal tasks, or lag behind state-of-the-we models (i.e. diffusion models) in continuous data generation. Transfusion provides simple, end-to-end solution to multi-modal learning that understands and generates high-quality multi-modal data.
An interesting area of recent acrive research is the application diffusion models and their generaliza- tions to discrete text generation (Li al. 2022 Gat et al. (024). However, this approach has yet to achieve the performance and scale of standard autoregressive language models. Future research in this direction may unlock new ways to fuse discrete and continuous modalities single model.
B EXPERIMENTS
B. SETUP
Evaluation We evaluate model performance on collection of standard uni-modal and cross- modal benchmarks (Table 7). For text-to-text, we measure perplexity on 20M held-out tokens from Wikipedia and the C4 corpus (Raffel et al. 2019), as well as accuracy the pretraining evaluation suite of Llama (Touvron et al., 2023b For text-to-image. we use the MS-COCO benchmark (Lin et al. 2014), where generate images randomly selected 30k prompts from validation and measure their photo-realism using zero-shot Frechet Inception Distance (FID) (Heusel et al.] 2017) as well as their alignment with the prompts using CLIP score (Radforrd et al.) (222)) We also evaluate the model's ability to generate image captions; we report CIDEr (Vedantam et al. 2015 scores on the Karpathy test split of MS-COCO (Lin al.
2014). These evaluations provide signal
"The Llama evaluation suite includes HellaSwag (Zellers et al.
2019 PIQA Bisk et 2020
SIQA Sap
al.
2019), WinoGrande (Sakaguchi et al. (00T), ARC-e and -c CCark et 2018), and BoolQ Clark al.
20T9
We report the average 0-shot task accuracy on these henchmarks.
We follow common practice for ablations and use only 5k examples compute FID and CL.IP in
17

---

Published as a conference paper at ICLR 2025
investigation scaling laws ($4. and ablations ($ ($4.3) To compare with recent literature in diffusion models, we evaluate our largest scale model ($4.4) also on GenEval (Ghosh et al. 2002). a benchmark that examines model'ss 's ability to generate an accurate depiction of the prompt.
Baseline At the time of writing, the prominent open-science method for training single mixed- modal model that can generate both text and images is to quantize images into discrete tokens, and then model the entire token sequence with a standard language model (Ramesh et al.) [202]: Yu (et al.] 2022 2023). We follow the recipe of Chameleon (Chameleon Team 2024) to train a family of data- and compute-controller baseline models, which can directly compare our Transfusion models. The key difference between Chameleon and Transfusion is that while Chameleon discretizes images and processes them tokens, Transfusion keeps images in continuous space, removing the quantization information bottleneck. To further minimize any confounding variables, we train the VABS for Chameleon and Transfusion using exactly the same data, compute, and architecture, with the only differentiator being the quantization layer and codebook loss of Chameleon's 's VQ-VAE (see details below). Chameleon also deviates from the Llama transformer architecture, adding query-key normalization, gost-normalization denominator loss, and lower learning rate of le-4 to manage training instability, which incur an efficiency cost (see $4.2 G
Data For almost all of our experiments, we sample 0.5T tokens (patches) from two datasets a 1:1 token ratio. For text, we use the Llama 2 tokenizer and corpus (Touvron et al. 2023b). containing 2T tokens across diverse distribution of domains, For images, we use a collection of 380M licensed Shutterstock images and captions. Each image is center-cropped and resized to produce a 256x pixel image We randomly order the image and captions, ordering the caption first 80% of the time.
we scale up the total training data to 2T tokens (1T text tokens and about 3.5B caption-image pairs at 256 patches per image). To diversify, we add 220M publicly available images with captions, prefiltered to not contain people. To rebalance the distribution, we upsample 80M Shutterstock images containing people. We also add data from Conceptual 12M (CC12M) (Changpinyo et al. 2021), reaching total mixture of 692M image-caption pairs per epoch. Finally, we upweight the portion of high-aesthetic images in the last 1 % of the training schedule.
In one experiment
Latent Image Representation We train a 86M parameter VAE following Esser et al. (021)) We use CNN encoder and decoder, and latent dimension 8. The training objective is combines reconstruction and regularization losses R Our implementation reduces an image of 256x256 pixels to x2 tensor, where each latent 8-dimensional latent pixel represents (conceptually) pixel patch in the original image, and trains for 1M steps. For VQ-VAE training, we follow the same setup described for VAE training, except we replace LKL with the standard codebook commitment loss with B = 0.25
8 8
Van Den Oord et al.) 2017). We use a codebook of 16,384 token types.
Model Configuration To investigate scaling trends, we train models at five different sizes 0.1 0.37B, 0.76B, 1.4B, and 7B parameters - following the standard settings from Llama (Touvron et al. 2023a). Table Sable@deesn describes each setting in detail. In configurations that use linear patch encoding ($4.2 and $4.3), the number of additional parameters is insignificant, accounting for fewer than 0.5% of total parameters in every configuration. When using U-Net patch encoding ($4.3) and $4.4). these parameters add up 0.27B additional parameters across all configurations; while this is substantial addition of parameters to smaller models, these layers amount to only a 3.8% increase of the 7B configuration, almost identical to the number of parameters in the embedding layers.
Optimization We randomly initialize all model parameters, and optimize them using AdamW (8) =0.9, .3 =0.95. =le-8) with a learning rate of 3e-4, warmed up for 4000 steps and decaying to 1.5e-5 using a cosine scheduler. We train on sequences of 4096 tokens in batches of 2M tokens for 250k steps, reaching 0.5T tokens in total. In our large-scale experiment ($4.4). we train with a batch size of 4M tokens over 500k steps, totalling 2T tokens. We regularize with weight decay of 0.1 and
'Removing these deviations in preliminary experiments encountered optimization instabilities in Chameleon. Depending on the compression rate of the patch encoder (see Model Architecture in $ each image will be represented by either 1024, 256, 64, or elements the sequence. Since the text/image ratio is constant during training, higher compression rates enable training on more images in total, at the cost of less compute per image. See Appendix C for details.
18

---

Published as a conference paper at ICLR 2025
Latent/ Patch
Pixel/ Patch
Patch/ Image
C4 PPL (
Wiki
Liama
MS-COCO
Enc/Dec
PPL (
Acc (4)
CDr (f)
FID (L)
CLIP (1)
None
1x1
8x8
1024
10.3
52.2
12.0
21.0
24.0
2x2
16x
256
10.4
16.0
20.3
24.0
Linear
4x4
32x32
64
10.9
6.3
49.8
14.3
25.6
22.6
8x8
64x64
16
11.7
6.9
47.7
11.3
43.5
18.9
2x2
16x 16
256
10.3
5.9
51.9
25.4
16.7
25.4
U-Net
4x4
32x32
64
10.7
6.2
50.7
29.9
16.0
25.7
8x8
64x64
16
11.4
6.6
49.2
29.5
16.1
25.2
Table 9: Performance of 0.76B Transfusion models with different patch sizes. Bolded figures indicate global best, underlines indicate best within architecture.
Model
A Enc/Dec Params
C4 PPL (L)
Wiki PPL (L)
Llama Acc (f)
MS-COCO
Ene/Dec
Params
CDr (1)
FID (U)
CLIP (t)
Linear U-Net
0.5% 106.1%
14.8
8.8
44.2
6.2
37.6
20.0
0.16B
14.4
8.5
45.7
15.3
18.8
23.9
Linear
0.4%
12.0
47.9
11.1
21.5
22.4
0.37B
U-Net
71.3%
48.8
21.1
18.1
24.9
Linear
0.4%
10.4
51.7
16.0
20.3
24.0
0.76B
U-Net
35.5%
10.3
5.9
51.9
25.4
16.7
25.4
Linear
0.4%
9.5
53.8
19.1
19.4
24.3
1.4B
U-Net
19.3%
9.4
5.4
53.4
28.1
16.6
25.7
Linear 0.3%
7.7
4.3
61.5
27.2
18.6
25.9
7B
U-Net
3.8%
7.8
4.3
61.1
33.7
16.0
26.5
Table 10: Performance of linear and U-Net variants of Transfusion across different model sizes. Patch size is set at 2x2 latent pixels. Model parameters refers to the transformer alone.
clip gradients by norm (1.0). We set the À coefficient in the Transfusion objective (Equation 4 to following preliminary experiments; we leave further tuning of À to future work.
Inference In text mode, we use greedy decoding for generating text. Ranked classification is used for the Llama evaluation suite. For image generation, we follow the standard of 250 diffusion steps (the model is trained on 1,000 timesteps). We follow Chameleon and use CFG with coefficient of in the controlled comparison experiments ($4.2). This value is suboptimal for Transfusion, and so we use CFG coefficient of throughout the ablation experiments (44.3), and follow the standard practice of tuning the coefficient for each benchmark in our large scale experiment ($4.4).
B.2 IMAGE NOISING
Our experiments order 80% of image-caption pairs with the caption first, and the image conditioning on the caption, following the intuition that image generation may be a more data-hungry task than image understanding. The remaining 20% the pairs condition the caption on the image. However, these images noised as part of the diffusion objective. We thus measure the effect of limiting the diffusion noise to a maximum images appear before their captions. Table [T] shows that noise limiting significantly improves image captioning, as measure by CIDEr, while having a relatively small effect (less than 1%) on other benchmarks.
= 500 (half of the noise schedule) in the 20% of cases where
19

---

Published as a conference paper at ICLR 2025
Model Params
Noise Limit
C4 PPL ())
Wiki PPL ())
Llama Ace (f)
MS-COCO
CDr (t)
FID ())
CLIP (t)
10.3
5.9
51.9
16.7
25.4
0.76B
10.3
5.9
52.1
29.4
16.5
25.4
7.8
4.3
61.1
33.7
16.0
26.5
7B
7.7
4.3
60.9
35.2
15.7
26.3
Table 11:
Performance of Transfusion with and without limiting the amount of sampled diffusion noise to maximum of /. = 500 when images appear before the caption. The models are U-Net variants encoding 2x2 latent pixel patches. Metrics that change by over 1% are bolded.
C AUTOENCODER DETAILS
The training objective for our VAE closely follows that of Esser et al. (2021):
LVAE = Li + LIPIPS + 0.5CGAN + 0.2CID + 0.000001
where T., L.1 loss in pixel space, LIISS is perceptual loss based on I.PIPS similarity /hhng et al. (2018), LGAN is patch-based discriminator loss, LID is perceptual loss based on internal features the Moco v2 model Chen et al. (2020), and LKL the standard KL-regularization term to encourage encoder outputs towards normal distribution. We delay the beginning of GAN training (i.e. including the adversarial loss in the loss function) to 50,000 steps, in order to let the VAE achieve sufficiently good reconstruction performance. We a latent dimension of 8.
The training objective for the VQ-GAN matches that of the VAE, with one notable exception: replace the LxL. loss with the standard codebook commitment loss Lecuthook (Van Den Oord ct al. 2017). which encourages encoder outputs and codebook vectors to be close together. We use 0.25, and use loss weighting 1.0. The final loss function for the VQ-VAE is therefore:
LvQ-VAE L1 + Liews + 0.5CGAN 0.2L + Lcodebook
The vector quantization layer is applied after projecting the encoder outputs to 8-dimensional space. Outside of the loss function change and the quantization layer, the training setup for the VAE (for Transfusion) and VQ-VAE (for Chameleon) are the same (e.g. same amount of training compute, same training data, and same encoder/decoder architecture).
D EXAMPLES: IMAGE GENERATION
Figure and Figure show examples of images generated from a Transfusion model trained on 2T multi-modal tokens ($.44)
E EXAMPLES: IMAGE EDITING
Figure 10 show random examples of image editing by fine-tuned 7B Transfusion model.
20

---

Published as a conference paper at ICLR 2025
An armchair in the shape an avocado
bread, apple, and knife on table
A corgi.
human life depicted tirely out fractals
Trans usion
blue jay standing on a large basket of rainhow macarons.
"Transfusion" is written on the blackhoard.
close up photo hu- man hand, hand model. High quality
A cloud in the shape of two bunnies playing with ball. The hall is made of clouds too.
START
the word 'START" on blue t-shirt
Dutch still life an arrangement tulips a fluted vase. The light- ing is subtle, casting gen- tee highlights the flow- ers and emphasizing their delicate details and natu- ral beauty.
A wall in royal castle. There are two paintings on the wall. The one on the left a detailed oil paint- ing of the royal raccoon king. The one on the right a detailed oil painting of the royal raccoon queen.
Three spheres made of glass falling into occan. Water is splashing. Sun is setting.
A transparent sculpture of duck made out of glass.
A chromeplated cat sculp- ture placed Persian rug.
A kangaroo holding beer, wearing ski goggles and passionately singing silly songs.
egg and bird made of wheat bread
Figure Generated images from 7B Transfusion trained on 2T multi-modal tokens.
21

---

Published as a conference paper at ICLR 2025
DIFFUSION
Downtown Seattle at sun- risc. detailed ink wash.
A car made out of vegeta- bles.
A sign that says "Diffu- sion".
A black basketball shoo
with a lightning bolt on it.
espresso machine that makes coffee from human souls, high-contrast paint- ing.
Intricate origami of fox and unicorn in snowy forest.
yellow wall with two framed sketches
A crab made of cheese on plate.
BE EXCELLEET
TOCH OTOOER
single beam of light en- ter the room from the ceil- ing. The beam of light is illuminating an easel. On the easel there a Rem- brandt painting of a rac- coon.
White Cycladic houses with blue accents and vi- brant magenta bougainvil- lea in serene Greek land setting.
The saying "BE EX- CELLENT TO EACH OTHER" written in stained glass window.
dark high contrast render of psychedelic tree of life illuminating dust in mystical cave.
photo of person with the head of cow, wear- ing a tuxedo and black howtie. Beach wallpaper in the background.
Photo of lychee-inspiree spherical chair. with a humpy white exterior and plush interior, set against tropical wallpaper.
An old rusted robot ing pants a jacket rid- ing skis in supermarket.
Film still of long-legged cute big-eye anthropomor- phic cheeschurger wear- ing sneakers relaxing on the couch in sparsely decorated living room.
Figure 8: Generated images from 7B Transfusion trained on 2T multi-modal tokens.
22

---

Published as a conference paper at ICLR 2025
bed under- a blanket.
A small blue book sitting on large red book.
A horse reading book.
A light bulb containing sailboat floats through galaxy.
a monarch butterfly.
rowboat on lake with a bike it.
An expressive oil painting of chocolate chip cookie being dipped in glass of milk, depicted as an ex- plosion of flavors.
An angry duck doing heavy weightlifting at the gym.
An emoji of baby panda wearing red hat, green gloves, red shirt, and green pants.
A tranquil, anime-style koi pond in serene Japanese garden, featur- ing blossoming cherry trees.
massive alien space ship that is shaped like a pret- zel.
graffiti of funny dog on street wall.
CANT BEAR IT
spacious, serene room influenced by modern Japanese aesthetics with a view of cityscape out- side of the window.
raccoon wearing cow- boy hat and black leather jacket is behind the back- yard window. droplets on the window.
A relaxed garlic with a blindfold reading paper while floating in pool of tomato soup.
photo a bear wearing suit and tophat in river in the middle of forest holding a sign that says cant bear it".
Rain
Figure Generated images from 7B Transfusion trained on 2T multi-modal tokens.
23

---

Published as a conference paper at ICLR 2025
#
Change the closest keyboard be all black.
Change the grafliti on the truck into calligraphy writing.
we have mountains on the background?
Replace the airplane with a blackhawk helicopter.
Add blue rug to the floor.
Delete the overhead lights on top of the sink.
Change the roll of thread into a roll of wire.
Change the baseball bat to all brown.
Figure 10: Edited images from fine-tuned 7B Transfusion model.
24


Reviews:
[{"review_id": "XId7DJ8kYO", "reviewer": "Reviewer_BCXk", "summary": "The paper proposes a multi-modal transformer model where text tokens are sampled auto regressively and image \"tokens\" are diffused in a VAE latent space.", "questions": "The structure of the UNet is unclear to me. the paper says \"since U-Net blocks contain bidirectional attention within, independent of the transformer’s attention mask, this gap is less pronounced when they are applied.\" What U-Net architecture uses bidirectional attention? Is the UNet applied to each patch independently, or is patching done after the U-Net?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- problem is well motivated\n- method is simple and clear\n- results are good\n- abalations cover most of my questions", "weaknesses": "the method relies on a pre-trained VAE. It would have been nice to see an ablation done in pixel space or with some type of co-training scheme to avoid a 2-stage training process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "96pmfWt9Ee", "reviewer": "Reviewer_Wjjj", "summary": "This paper proposes Transfusion, a unified model for joint understanding and generation. During training, it incorporates CE loss for text learning and diffusion loss for image learning. In inference, it uses casual attention for autoregressive text generation and bidirectional attention for parallel image generation. Compared to Chameleon, Transfusion attains good performance for visual understanding and generation.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper verifies the feasibility combining different learning targets with one framework for respective understanding and generation.\n\n2. It reveals the scaling performance comparing Transfusion and Chameleon, providing good insights for future works.\n\n3. It provides detailed ablation study and analysis to demonstrate the effectiveness of each component.", "weaknesses": "1. The visual understanding performance of Transfusion is not convincing enough, as the paper doesn't provide any results on related benchmarks, such as MME, MMBench, etc.\n\n2. The image editing performance also needs quantitative results to show its efficacy.\n\n3. Does this framework has the potential to conduct image-text interleaved generation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4DBZdQH9xk", "reviewer": "Reviewer_HRoU", "summary": "The paper introduces Transfusion, a novel approach for training a single unified model capable of understanding and generating both discrete and continuous modalities. The authors demonstrate that by combining the language modeling loss function (next token prediction) with diffusion, they can train a transformer model over mixed-modality sequences. This method allows for seamless integration of discrete text tokens and continuous image data, without the need for information loss through quantization.", "questions": "As shown in Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Originality: The paper introduces a unique approach by combining language modeling and diffusion objectives over shared data and parameters. This method allows for the seamless generation of discrete and continuous modalities.\n\n2. Quality: The paper demonstrates the effectiveness of the Transfusion model through extensive experiments and comparisons with existing methods. \n\n3. Clarity: The paper is well-structured, with clear explanations of the Transfusion method, its architecture, and the training objectives. \n\n4. Significance: The paper addresses a critical challenge in handling both discrete and continuous data modalities.", "weaknesses": "#### More experiments are encouraged.\n1. Results on multimodal Understanding tasks are limited. For example, VQA, Summarization, VideoQA and etc.\n2. I encourage the authors to scale up the model parameters or training data size (Scaling Law on Transfusion is very important), but I understand the computation resources on scaling law are huge. Thus, this is a minor issue.\n\n#### Compare the technical differences of several recent works (e.g., Show-o, MIO, Emu3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FDeROIHcCp", "reviewer": "Reviewer_Huwc", "summary": "The paper introduces Transfusion, a novel approach for training multi-modal models that effectively handle both discrete (text) and continuous (image) data. By combining next token prediction for text and diffusion for images within a single transformer architecture, Transfusion eliminates information loss associated with traditional quantization methods. The model was pretrained on a balanced dataset of text and images, utilizing modality-specific encoding and decoding layers to optimize performance. Experimental results demonstrate that Transfusion outperforms existing methods like Chameleon in various benchmarks, achieving significant efficiencies in both text-to-image and image-to-text generation tasks. The study concludes that Transfusion represents a significant advancement in creating integrated multi-modal generative models.", "questions": "same as weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Innovative Framework: Transfusion introduces a novel approach that seamlessly combines text and image modalities, achieving impressive results in image generation and editing.\n2. Robust Comparative Experiments: The paper includes comprehensive comparative analyses and detailed ablation studies that examine various dimensions, such as model architecture and parameters, providing strong evidence for the framework's effectiveness.\n3. Clear and Fluid Writing: The writing is clear and the structure of the paper is well-organized, making it easy to follow the authors' arguments and findings.", "weaknesses": "Lack of demonstration of multi-modal understanding capabilities: While the paper focuses on image generation and editing capabilities, it does not present extensive benchmarks for the model’s general visual question answering (VQA) abilities. It would be beneficial to include benchmarks related to image understanding, such as VQAV2, SEED Bench and MMMU.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LAcsdZYxhI", "reviewer": "Reviewer_MTCF", "summary": "This paper explores how to bridge the modality gap to inject continuous image generation ability into discrete text-based LLMs by proposing a single transformer model trained with autoregressive modeling and diffusion loss simultaneously. This method of Transfusion avoids image discreteness and still enables input and output-level image tasks. By carefully designing attention masks and input formation, the transformer models text data by AR modeling with causal masking and generates images by denoising with bidirectional attention.\n\nCompared to injecting image comprehension/generation ability by introducing discrete image tokens(Chameleon), the authors demonstrated their approach performs better in various scales in text-to-image generation, image captioning, and even text-only tasks. The large-scale model of Transfusion showed comparable performance to previously published large image generative models.", "questions": "-How does Transfusion take the embedding of noise timestep in the training/inference of image latent denoising process?\n\n-In the training phase, the model calculates the AR loss and diffusion loss in one forward pass using text tokens and 'noised' image latent, which introduces unavoidable discrepancy when the model inference with clear latent. Wouldn't it be more accurate to calculate AR and diffusion loss separately in two model passes?\n\n\n-Is there a reason that the COCO FID scores of StableDiffusion models are not in Table 6?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 4, "strengths": "-They suggested a novel training and inference pipeline for a transformer that allows performing both AR and diffusion modeling with shared parameters.\n\n-The manuscript contains extensive controlled experiments and ablation studies.", "weaknesses": "-The comparison with previously published works only covered text-only & text-to-image tasks and didn't include their performances on image comprehension tasks(e.g. image captioning). \nAn additional comparison for these tasks with previous multi-modal LLMs that process images differently(e.g. input-only with an external encoder like LLaVA, or image quantization like Chameleon) would support the effectiveness of Transfusion's approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)