Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

## Abstract

Abstract We introduce Transfusion, a recipe for training a multi-modal model over discrete and continuous data.
Transfusion combines the language modeling loss function (next token prediction) with diffusion to train a single transformer over mixed-modality sequences.
We pretrain multiple Transfusion models up to 7B parameters from scratch on a mixture of text and image data, establishing scaling laws with respect to a variety of uni- and cross-modal benchmarks.
Our experiments show that Transfusion scales significantly better than quantizing images and training a language model over discrete image tokens.
By introducing modality-specific encoding and decoding layers, we can further improve the performance of Transfusion models, and even compress each image to just 16 patches.
We further demonstrate that scaling our Transfusion recipe to 7B parameters and 2T multi-modal tokens produces a model that can generate images and text on a par with similar scale diffusion models and language models, reaping the benefits of both worlds.

## 1 Introduction

Multi-modal generative models need to be able to perceive, process, and produce both discrete elements (such as text or code) and continuous elements (e.g. image, audio, and video data).
While language models trained on the next token prediction objective dominate discrete modalities (OpenAI et al., ; Dubey et al., ), diffusion models (Ho et al., ; Rombach et al., ) and their generalizations (Lipman et al., ) are the state of the art for generating continuous modalities (Dai et al., ; Esser et al., ; Bar-Tal et al., ).
Many efforts have been made to combine these approaches, including extending a language model to use a diffusion model as a tool, either explicitly (Liu et al., ) or by grafting a pretrained diffusion model onto the language model (Dong et al., ; Koh et al., ).
Alternatively, one can quantize the continuous modalities (Van Den Oord et al., ) and train a standard language model over discrete tokens (Ramesh et al., ; Yu et al., , ), simplifying the model’s architecture at the cost of losing information.
In this work, we show it is possible to fully integrate both modalities, with no information loss, by training a single model to both predict discrete text tokens and diffuse continuous images.

We introduce Transfusion, a recipe for training a model that can seamlessly generate discrete and continuous modalities.
We demonstrate Transfusion by pretraining a transformer model on 50% text and 50% image data using a different objective for each modality: next token prediction for text and diffusion for images.
The model is exposed to both modalities and loss functions at each training step.
Standard embedding layers convert text tokens to vectors, while patchification layers represent each image as a sequence of patch vectors.
We apply causal attention for text tokens and bidirectional attention for image patches.
For inference, we introduce a decoding algorithm that combines the standard practices of text generation from language models and image generation from diffusion models.
Figure 1 illustrates Transfusion.

In a controlled comparison with Chameleon’s discretization approach (Chameleon Team, ), we show that Transfusion models scale better in every combination of modalities.
In text-to-image generation, we find that Transfusion exceeds the Chameleon approach at less than a third of the compute, as measured by both FID and CLIP scores.
When controlling for FLOPs, Transfusion achieves approximately 2$\times$ lower FID scores than Chameleon models.
We observe a similar trend in image-to-text generation, where Transfusion matches Chameleon at 21.8% of the FLOPs.
Surprisingly, Transfusion is also more efficient at learning text-to-text prediction, achieving perplexity parity on text tasks around 50% to 60% of Chameleon’s FLOPs.

Ablation experiments reveal critical components and potential improvements for Transfusion.
We observe that the intra-image bidirectional attention is important, and that replacing it with causal attention hurts text-to-image generation.
We also find that adding U-Net down and up blocks to encode and decode images enables Transfusion to compress larger image patches with relatively small loss to performance, potentially decreasing the serving costs by up to 64$\times$.

Finally, we demonstrate that Transfusion can generate images at similar quality to other diffusion models.
We train from scratch a 7B transformer enhanced with U-Net down/up layers (0.27B parameters) over 2T tokens: 1T text tokens, and approximately 5 epochs of 692M images and their captions, amounting to another 1T patches/tokens.
Figure 2 shows some generated images sampled from the model.
On the GenEval (Ghosh et al., ) benchmark, our model outperforms other popular models such as DALL-E 2 and SDXL; unlike those image generation models, it can generate text, reaching the same level of performance as Llama 1 on text benchmarks.
Our experiments thus show that Transfusion is a promising approach for training truly multi-modal models.

Figure: Figure 1: A high-level illustration of Transfusion. A single transformer perceives, processes, and produces data of every modality. Discrete (text) tokens are processed autoregressively and trained on the next token prediction objective. Continuous (image) vectors are processed together in parallel and trained on the diffusion objective. Marker BOI and EOI tokens separate the modalities.
Refer to caption: x1.png

Figure: ((a))
Refer to caption: extracted/5802141/samples/avocado_armchair.jpg

## 2 Background

Transfusion is a single model trained with two objectives: language modeling and diffusion.
Each of these objectives represents the state of the art in discrete and continuous data modeling, respectively.
This section briefly defines these objectives, as well as background on latent image representations.

### 2.1 Language Modeling

Given a sequence of discrete tokens $y=y_{1},...,y_{n}$ from a closed vocabulary $V$, a language model predicts the probability of the sequence $P(y)$.
Standard language models decompose $P(y)$ into a product of conditional probabilities $\prod_{i=1}^{n}P_{\theta}(y_{i}|y_{<i})$.
This creates an autoregressive classification task, where the probability distribution of each token $y_{i}$ is predicted conditioned on the prefix of a sequence $y_{<i}$ using a single distribution $P_{\theta}$ parameterized by $\theta$.
The model can be optimized by minimizing the cross-entropy between $P_{\theta}$ and the empirical distribution of the data, yielding the standard next-token prediction objective, colloquially referred to as LM loss:

$$ $\mathcal{L}_{\text{LM}}=\mathbb{E}_{y_{i}}\big{[}-\log P_{\theta}(y_{i}|y_{<i} )\big{]}$ (1) $$

Once trained, language models can also be used to generate text by sampling token by token from the model distribution $P_{\theta}$, typically using temperature and top-p truncation.

### 2.2 Diffusion

Denoising diffusion probabilistic models (a.k.a. DDPM or diffusion models) operate on the principle of learning to reverse a gradual noise-addition process (Ho et al., ).
Unlike language models that typically work with discrete tokens ($y$), diffusion models operate over continuous vectors ($\mathbf{x}$), making them particularly suited for tasks involving continuous data like images.
The diffusion framework involves two processes: a forward process that describes how the original data is turned into noise, and a reverse process of denoising that the model learns to perform.

##### Forward Process

From a mathematical perspective, the forward process defines how the noised data (which serves as the model input) is created.
Given a data point $\mathbf{x}_{0}$, Ho et al. () define a Markov chain that gradually adds Gaussian noise over $T$ steps, creating a sequence of increasingly noisy versions $\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{T}$.
Each step of this process is defined by $q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta_{t
}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I})$, where $\beta_{t}$ increases over time according to a predefined noise schedule (see below).
This process can be reparameterized in a way that allows us to directly sample $\mathbf{x}_{t}$ from $\mathbf{x}_{0}$ using a single sample of Gaussian noise $\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$:

$$ $\mathbf{x}_{t}=\sqrt{\bar{\alpha}_{t}}\mathbf{x}_{0}+\sqrt{1-\bar{\alpha}_{t}} \boldsymbol{\epsilon}$ (2) $$

Here, $\bar{\alpha}_{t}=\prod_{s=1}^{t}(1-\beta_{s})$, providing a useful abstraction over the original Markov chain.
In fact, both the training objective and the noise scheduler are eventually expressed (and implemented) in these terms.

##### Reverse Process

The diffusion model is trained to perform the reverse process $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$, learning to denoise the data step by step.
There are several ways to do so; in this work, we follow the approach of Ho et al. () and model the Gaussian noise $\boldsymbol{\epsilon}$ in Equation 2 as a proxy for the cumulative noise at step $t$.
Specifically, a model $\epsilon_{\theta}(\cdot)$ with parameters $\theta$ is trained to estimate the noise $\boldsymbol{\epsilon}$ given the noised data $\mathbf{x}_{t}$ and timestep $t$.
In practice, the model often conditions on additional contextual information $c$, such as a caption when generating an image.
The parameters of the noise prediction model are thus optimized by minimizing the mean squared error loss:

$$ $\mathcal{L}_{\text{DDPM}}=\mathbb{E}_{\mathbf{x}_{0},t,\boldsymbol{\epsilon}} \big{[}||\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\theta}(\mathbf{x}_{t},t ,c)||^{2}\big{]}$ (3) $$

##### Noise Schedule

When creating a noised example $\mathbf{x}_{t}$ (Equation 2), $\bar{\alpha}_{t}$ determines the variance of the noise for timestep $t$.
In this work, we adopt the commonly used cosine scheduler Nichol and Dhariwal (), which largely follows $\sqrt{\bar{\alpha}_{t}}\approx\cos(\frac{t}{T}\cdot\frac{\pi}{2})$ with some adjustments.

##### Inference

Decoding is done iteratively, pealing away some of the noise at each step.
Starting with pure Gaussian noise at $\mathbf{x}_{T}$, the model $\boldsymbol{\epsilon}_{\theta}(\mathbf{x}_{t},t,c)$ predicts the noise accumulated at timestep $t$.
The predicted noise is then scaled according to the noise schedule, and the proportional amount of predicted noise is removed from $\mathbf{x}_{t}$ to produce $\mathbf{x}_{t-1}$.
In practice, inference is done over fewer timesteps than training.
Classifier-free guidance (CFG) (Ho and Salimans, ) is often used to improve generation by contrasting the prediction of the model conditioned on the context $c$ with the unconditioned prediction, at the cost of doubling the computation.

### 2.3 Latent Image Representation

Early diffusion models worked directly in pixel space (Ho et al., ), but this proved computationally expensive.
Variational autoencoders (VAEs) (Kingma and Welling, ) can save compute by encoding images into a lower-dimensional latent space.
Implemented as deep CNNs, modern VAEs are trained on a combination of reconstruction and regularization losses (Esser et al., ), allowing downstream models like latent diffusion models (LDMs) (Rombach et al., ) to operate efficiently on compact image patch embeddings; e.g. represent every 8$\times$8 pixel patch as an 8-dimensional vector.
For autoregressive language modeling approaches (Ramesh et al., ; Yu et al., ), images must be discretized.
Discrete autoencoders, such as vector-quantized VAEs (VQ-VAE) (Van Den Oord et al., ), achieve this by introducing a quantization layer (and related regularization losses) that maps continuous latent embeddings to discrete tokens.

## 3 Transfusion

Figure: Figure 3: We convert images to and from latent representations using a pretrained VAE, and then into patch representations with either a simple linear layer or U-Net down blocks.
Refer to caption: x2.png

Transfusion is a method for training a single unified model to understand and generate both discrete and continuous modalities.
Our main innovation is demonstrating that we can use separate losses for different modalities – language modeling for text, diffusion for images – over shared data and parameters.
Figure 1 illustrates Transfusion.

##### Data Representation

We experiment with data spanning two modalities: discrete text and continuous images.
Each text string is tokenized into a sequence of discrete tokens from a fixed vocabulary, where each token is represented as an integer.
Each image is encoded as latent patches using a VAE (see §2.3), where each patch is represented as a continuous vector; the patches are sequenced left-to-right top-to-bottom to create a sequence of patch vectors from each image.(^1^11While our canonical setting uses a VAE following latent diffusion models, we were also able to demonstrate Transfusion using raw pixel representations in preliminary experiments.)
For mixed-modal examples, we surround each image sequence with special beginning of image (BOI) and end of image (EOI) tokens before inserting it to the text sequence;
thus, we arrive at a single sequence potentially containing both discrete elements (integers representing text tokens) and continuous elements (vectors representing image patches).

##### Model Architecture

The vast majority of the model’s parameters belong to a single transformer, which processes every sequence, regardless of modality.(^2^22We follow Llama’s (Touvron et al., ) flavor of the transformer block, which includes the SwiGLU activation function (Shazeer, ) and RoPE (Su et al., ).)(^3^33While we use the transformer architecture in this work, Transfusion could potentially work with other architectures too, despite its name.)
The transformer takes a sequence of high-dimensional vectors in $\mathbb{R}^{d}$ as input, and produces similar vectors as output.
To convert our data into this space, we use lightweight modality-specific components with unshared parameters.
For text, these are the embedding matrices, converting each input integer to vector space and each output vector into a discrete distribution over the vocabulary.
For images, we experiment with two alternatives for compressing local windows of $k\times k$ patch vectors into a single transformer vector (and vice versa): (1) a simple linear layer,(^4^44We add an embedding of the timestep $t$ to every patch vector before the linear layer.) and (2) up and down blocks of a U-Net (Nichol and Dhariwal, ; Saharia et al., ).(^5^55We replace the U-Net’s AdaLayerNorm with regular layer norm in our implementation.)
Figure 4 illustrates the overall architecture.

##### Transfusion Attention

Language models typically use causal masking to efficiently compute the loss and gradients over an entire sequence in a single forward-backward pass without leaking information from future tokens.
While text is naturally sequential, images are not, and are usually modeled with unrestricted (bidirectional) attention.
Transfusion combines both attention patterns by applying causal attention to every element in the sequence, and bidirectional attention within the elements of each individual image.
This allows every image patch to attend to every other patch within the same image, but only attend to text or patches of other images that appeared previously in the sequence.
We find that enabling intra-image attention significantly boosts model performance (see §4.3).
Figure 4 shows an example Transfusion attention mask.

##### Training Objective

To train our model, we apply the language modeling objective $\mathcal{L}_{\text{LM}}$ to predictions of text tokens and the diffusion objective $\mathcal{L}_{\text{DDPM}}$ to predictions of image patches.
LM loss is computed per token,(^6^66When the input is a BOI token, we do not compute any loss.) while diffusion loss is computed per image, which may span multiple elements (image patches) in the sequence.
Specifically, we add noise $\boldsymbol{\epsilon}$ to each input latent image $\mathbf{x}_{0}$ according to the diffusion process to produce $\mathbf{x}_{t}$ before patchification, and then compute the image-level diffusion loss.(^7^77Ergo, downstream tokens condition on noisy images during training. See §4.3.4 for further discussion.)
We combine the two losses by simply adding the losses computed over each modality with a balancing coefficient $\lambda$:

$$ $\mathcal{L}_{\text{Transfusion}}=\mathcal{L}_{\text{LM}}+\lambda\cdot\mathcal{ L}_{\text{DDPM}}$ (4) $$

This formulation is a specific instantiation of a broader idea: combining a discrete distribution loss with a continuous distribution loss to optimize the same model.
We leave further exploration of this space, such as replacing diffusion with flow matching (Lipman et al., )), to future work.

##### Inference

Reflecting the training objective, our decoding algorithm also switches between two modes: LM and diffusion.
In LM mode, we follow the standard practice of sampling token by token from the predicted distribution.
When we sample a BOI token, the decoding algorithm switches to diffusion mode, where we follow the standard procedure of decoding from diffusion models.
Specifically, we append a pure noise $\mathbf{x}_{T}$ in the form of $n$ image patches to the input sequence (depending on the desired image size), and denoise over $T$ steps.
At each step $t$, we take the noise prediction and use it to produce $\mathbf{x}_{t-1}$, which then overwrites $\mathbf{x}_{t}$ in the sequence; i.e. the model always conditions on the last timestep of the noised image and cannot attend to previous timesteps.
Once the diffusion process has ended, we append an EOI token to the predicted image, and switch back to LM mode.
This algorithm enables the generation of any mixture of text and image modalities.

## 4 Experiments

We demonstrate in a series of controlled experiments that Transfusion is a viable, scalable method for training a unified multi-modal model.

### 4.1 Setup

##### Evaluation

We evaluate model performance on a collection of standard uni-modal and cross-modal benchmarks (Table 2).
For text-to-text, we measure perplexity on 20M held-out tokens from Wikipedia and the C4 corpus (Raffel et al., ), as well as accuracy on the pretraining evaluation suite of Llama 2 (Touvron et al., ).(^8^88The Llama 2 evaluation suite includes HellaSwag (Zellers et al., ), PIQA (Bisk et al., ), SIQA (Sap et al., ), WinoGrande (Sakaguchi et al., ), ARC-e and -c (Clark et al., ), and BoolQ (Clark et al., ). We report the average 0-shot task accuracy on these benchmarks.)
For text-to-image, we use the MS-COCO benchmark (Lin et al., ), where we generate images on randomly selected 30k prompts from validation set and measure their photo-realism using zero-shot Frechet Inception Distance (FID) (Heusel et al., ) as well as their alignment with the prompts using CLIP score (Radford et al., ).(^9^99We follow common practice for ablations and use only 5k examples to compute FID and CLIP in §4.3.)
We also evaluate the model’s ability to generate image captions; we report CIDEr (Vedantam et al., ) scores on the Karpathy test split of MS-COCO (Lin et al., ).
These evaluations provide signal for investigation scaling laws (§4.2) and ablations (§4.3).
To compare with recent literature in diffusion models, we evaluate our largest scale model (§4.4) also on GenEval (Ghosh et al., ), a benchmark that examines a model’s ability to generate an accurate depiction of the prompt.

**Table 1: An overview of the evaluation suite used in this work.**
| Input | Output | Benchmark | Metric |
| --- | --- | --- | --- |
| Text | Text | Wikipedia | Perplexity ($\downarrow$) |
| C4 | Perplexity ($\downarrow$) |  |  |
| Llama 2 Eval Suite | Accuracy ($\uparrow$) |  |  |
| Image | Text | MS-COCO 5k | CIDEr ($\uparrow$) |
| Text | Image | MS-COCO 30k | FID ($\downarrow$), CLIP ($\uparrow$) |
| GenEval | GenEval score ($\uparrow$) |  |  |

##### Baseline

At the time of writing, the prominent open-science method for training a single mixed-modal model that can generate both text and images is to quantize images into discrete tokens, and then model the entire token sequence with a standard language model (Ramesh et al., ; Yu et al., , ).
We follow the recipe of Chameleon (Chameleon Team, ) to train a family of data- and compute-controlled baseline models, which we can directly compare to our Transfusion models.
The key difference between Chameleon and Transfusion is that while Chameleon discretizes images and processes them as tokens, Transfusion keeps images in continuous space, removing the quantization information bottleneck.
To further minimize any confounding variables, we train the VAEs for Chameleon and Transfusion using exactly the same data, compute, and architecture, with the only differentiator being the quantization layer and codebook loss of Chameleon’s VQ-VAE (see details below).
Chameleon also deviates from the Llama transformer architecture, adding query-key normalization, post-normalization, denominator loss, and a lower learning rate of 1e-4 to manage training instability, which incur an efficiency cost (see §4.2).(^10^1010Removing these deviations in preliminary experiments encountered optimization instabilities in Chameleon.)

##### Data

For almost all of our experiments, we sample 0.5T tokens (patches) from two datasets at a 1:1 token ratio.
For text, we use the Llama 2 tokenizer and corpus (Touvron et al., ), containing 2T tokens across a diverse distribution of domains.
For images, we use a collection of 380M licensed Shutterstock images and captions.
Each image is center-cropped and resized to produce a 256$\times$256 pixel image.(^11^1111Depending on the compression rate of the patch encoder (see Model Architecture in §3), each image will be represented by either 1024, 256, 64, or 16 elements in the sequence. Since the text/image ratio is constant during training, higher compression rates enable training on more images in total, at the cost of less compute per image.)
We randomly order the image and captions, ordering the caption first 80% of the time.

In one experiment (4.4) we scale up the total training data to 2T tokens (1T text tokens and about 3.5B caption-image pairs at 256 patches per image).
To diversify, we add 220M publicly available images with captions, prefiltered to not contain people.
To rebalance the distribution, we upsample 80M Shutterstock images containing people.
We also add data from Conceptual 12M (CC12M) (Changpinyo et al., ), reaching a total mixture of 692M image-caption pairs per epoch.
Finally, we upweight the portion of high-aesthetic images in the last 1% of the training schedule.

##### Latent Image Representation

We train a 86M parameter VAE following Esser et al. ().
We use a CNN encoder and decoder, and latent dimension 8.
The training objective is combines reconstruction and regularization losses.(^12^1212See Appendix A for details.)
Our implementation reduces an image of 256$\times$256 pixels to a 32$\times$32$\times$8 tensor, where each latent 8-dimensional latent pixel represents (conceptually) an 8$\times$8 pixel patch in the original image, and trains for 1M steps.
For VQ-VAE training, we follow the same setup described for VAE training, except we replace $\mathcal{L}_{\text{KL}}$ with the standard codebook commitment loss with $\beta=0.25$ (Van Den Oord et al., ). We use a codebook of 16,384 token types.

##### Model Configuration

To investigate scaling trends, we train models at five different sizes – 0.16B, 0.37B, 0.76B, 1.4B, and 7B parameters – following the standard settings from Llama (Touvron et al., ).
Table 2 describes each setting in detail.
In configurations that use linear patch encoding (§4.2 and §4.3), the number of additional parameters is insignificant, accounting for fewer than 0.5% of total parameters in every configuration.
When using U-Net patch encoding (§4.3 and §4.4), these parameters add up to 0.27B additional parameters across all configurations; while this is a substantial addition of parameters to smaller models, these layers amount to only a 3.8% increase of the 7B configuration, almost identical to the number of parameters in the embedding layers.

##### Optimization

We randomly initialize all model parameters, and optimize them using AdamW ($\beta_{1}=$0.9, $\beta_{2}=$0.95, $\epsilon=$1e-8) with a learning rate of 3e-4, warmed up for 4000 steps and decaying to 1.5e-5 using a cosine scheduler.
We train on sequences of 4096 tokens in batches of 2M tokens for 250k steps, reaching 0.5T tokens in total.
In our large-scale experiment (§4.4), we train with a batch size of 4M tokens over 500k steps, totalling 2T tokens.
We regularize with weight decay of 0.1 and clip gradients by norm (1.0).
We set the $\lambda$ coefficient in the Transfusion objective (Equation 4) to 5 following preliminary experiments;
we leave further tuning of $\lambda$ to future work.

##### Inference

In text mode, we use greedy decoding for generating text.
Ranked classification is used for the Llama evaluation suite.
For image generation, we follow the standard of 250 diffusion steps (the model is trained on 1,000 timesteps).
We follow Chameleon and use CFG with a coefficient of 5 in the controlled comparison experiments (§4.2).
This value is suboptimal for Transfusion, and so we use a CFG coefficient of 3 throughout the ablation experiments (§4.3), and follow the standard practice of tuning the coefficient for each benchmark in our large scale experiment (§4.4).

### 4.2 Controlled Comparison with Chameleon

We run a series of controlled experiments to compare Transfusion with Chameleon at different model sizes ($N$) and token counts ($D$), using the combination of both as a proxy for FLOPs ($6ND$).(^13^1313Since Transfusion uses continuous representations of images, it can express a single image with significantly fewer tokens, shortening the average document length and thus the overall quadratic price of attention. Since this fact favors Transfusion, we remove this confounder by using the theoretical FLOP calculation.)
For simplicity and parameter control, the Transfusion variant in these experiments uses simple linear image encoder/decoder with patch size 2$\times$2, as well as bidirectional attention.
For each benchmark, we plot all results on a log-metric over log-FLOPs curve and regress linear trendlines.(^14^1414The smaller Chameleon models perform poorly on image generation and understanding tasks, leading to outlier results that do not correlate with the emerging scaling law of larger Chameleon models. We therefore define minimal performance thresholds below which we remove datapoints: $\leq$100 FID, $\geq$17 CLIP, $\geq$4 CIDEr.)
We also estimate relative compute efficiency by measuring the parity FLOP ratio: the ratio between the number of FLOPs required by Transfusion and Chameleon to reach the same level of performance.

Figure 5 visualizes the scaling trends,
and Table 3 shows the results of the largest models in this controlled setting and their estimated parity FLOP ratio.
In every benchmark, Transfusion consistently exhibits better scaling laws than Chameleon.
While the lines are close to parallel, there is a significant gap in Transfusion’s favor.
The difference in compute efficiency is particularly striking in image generation, where FID Transfusion achieves parity with Chameleon using 34$\times$ less compute.

Surprisingly, text-only benchmarks also reveal better performance with Transfusion, even though both Transfusion and Chameleon model text in the same way.
We investigate this phenomenon by ablating the various changes leading up to Transfusion and Chameleon from the original Llama 2 recipe.
Table 4 shows that while Transfusion does come at a non-zero cost to text performance, the Chameleon recipe suffers from both the stability modifications made to the architecture and from the introduction of image tokens.
Training on quantized image tokens degrades text performance more than diffusion on all three benchmarks.
One hypothesis is that this stems from the competition between text and image tokens in the output distribution;
alternatively, it is possible that diffusion is more efficient at image generation and requires fewer parameters, allowing Transfusion models to use more capacity than Chameleon to model text.
We leave further investigation of this phenomenon to future research.

Figure: ((a)) C4 Perplexity
Refer to caption: x4.png

**Table 3: Performance of the largest (7B) Transfusion and Chameleon models in a controlled setting. Both models were trained on 0.5T tokens. Parity FLOP Ratio is the relative amount of Transfusion FLOPs needed to match the results of Chameleon 7B.**
| Model | C4 | Wiki | Llama | MS-COCO |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| PPL ($\downarrow$) | PPL ($\downarrow$) | Acc ($\uparrow$) | CDr ($\uparrow$) | FID ($\downarrow$) | CLIP ($\uparrow$) |  |
| Transfusion | 7.72 | 4.28 | 61.5 | 27.2 | 16.8 | 25.5 |
| Chameleon | 8.41 | 4.69 | 59.1 | 18.0 | 29.6 | 24.3 |
| Parity FLOP Ratio | 0.489 | 0.526 | 0.600 | 0.218 | 0.029 | 0.319 |

**Table 4: Performance of the 0.76B Transfusion and Chameleon models on text-only benchmarks, compared to the original Llama 2 recipe.**
| Model |  | Batch | C4 | Wiki | Llama |
| --- | --- | --- | --- | --- | --- |
|  | PPL ($\downarrow$) | PPL ($\downarrow$) | Acc ($\uparrow$) |  |  |
| Llama 2 |  | 1M Text Tokens | 10.1 | 5.8 | 53.7 |
| Transfusion | + Diffusion | + 1M Image Patches | (+0.3) 10.4 | (+0.2) 6.0 | (-2.0) 51.7 |
| Chameleon | + Stability Modifications | 1M Text Tokens | (+0.9) 11.0 | (+0.5) 6.3 | (-1.8) 51.9 |
|  | + LM Loss on Image Tokens | + 1M Image Tokens | (+0.8) 11.8 | (+0.5) 6.8 | (-3.0) 48.9 |

### 4.3 Architecture Ablations

Now that we have established that Transfusion is a viable, scalable approach to multi-modal modeling in a controlled environment, we can explore improvements and extensions that are applicable to Transfusion alone.

#### 4.3.1 Attention Masking

We first examine the necessity of intra-image bidirectional attention.
Table 5 shows that enabling this attention pattern beyond the standard causal attention is advantageous throughout all benchmarks, and using both image encoding/decoding architectures.
In particular, we notice a significant improvement in FID when using linear encoding layers (61.3$\rightarrow$20.3).
In the causal-only version of this architecture, there is no flow of information from patches that appear later in the sequence to those before;
since U-Net blocks contain bidirectional attention within, independent of the transformer’s attention mask, this gap is less pronounced when they are applied.

**Table 5: Performance of 0.76B Transfusion models with and without intra-image bidirectional attention. Patch size is set at 2$\times$2 latent pixels.**
| Enc/Dec | Attention | C4 | Wiki | Llama | MS-COCO |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| PPL ($\downarrow$) | PPL ($\downarrow$) | Acc ($\downarrow$) | CDr ($\uparrow$) | FID ($\downarrow$) | CLIP ($\uparrow$) |  |  |
| Linear | Causal | 10.4 | 6.0 | 51.4 | 12.7 | 61.3 | 23.0 |
| Bidirectional | 10.4 | 6.0 | 51.7 | 16.0 | 20.3 | 24.0 |  |
| U-Net | Causal | 10.3 | 5.9 | 52.0 | 23.3 | 16.8 | 25.3 |
| Bidirectional | 10.3 | 5.9 | 51.9 | 25.4 | 16.7 | 25.4 |  |

#### 4.3.2 Patch Size

Transfusion models can be defined over different sizes of latent pixel patches.
Larger patch sizes allow the model to pack more images in each training batch and dramatically reduce inference compute, but may come at a performance cost.
Table 6 sheds light on these performance trade-offs.
While performance does decrease consistently as each image is represented by fewer patches with linear encoding, models with U-Net encoding benefit from larger patches on tasks involving the image modality.
We posit that this is due to the greater amount of total images (and diffusion noise) seen during training.
We also observe that text performance deteriorates with larger patches, perhaps because transfusion needs to exert more resources (i.e. parameters) to learn how to process images with fewer patches and thus less inference compute.

**Table 6: Performance of 0.76B Transfusion models with different patch sizes. Bolded figures indicate global best, underlines indicate best within architecture.**
| Enc/Dec | Latent/ | Pixel/ | Patch/ | C4 | Wiki | Llama | MS-COCO |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Patch | Patch | Image | PPL ($\downarrow$) | PPL ($\downarrow$) | Acc ($\downarrow$) | CDr ($\uparrow$) | FID ($\downarrow$) | CLIP ($\uparrow$) |  |
| None | 1$\times$1 | 8$\times$8 | 1024 | 10.3 | 5.9 | 52.2 | 12.0 | 21.0 | 24.0 |
| Linear | 2$\times$2 | 16$\times$16 | 256 | 10.4 | 6.0 | 51.7 | 16.0 | 20.3 | 24.0 |
| 4$\times$4 | 32$\times$32 | 64 | 10.9 | 6.3 | 49.8 | 14.3 | 25.6 | 22.6 |  |
| 8$\times$8 | 64$\times$64 | 16 | 11.7 | 6.9 | 47.7 | 11.3 | 43.5 | 18.9 |  |
| U-Net | 2$\times$2 | 16$\times$16 | 256 | 10.3 | 5.9 | 51.9 | 25.4 | 16.7 | 25.4 |
| 4$\times$4 | 32$\times$32 | 64 | 10.7 | 6.2 | 50.7 | 29.9 | 16.0 | 25.7 |  |
| 8$\times$8 | 64$\times$64 | 16 | 11.4 | 6.6 | 49.2 | 29.5 | 16.1 | 25.2 |  |

#### 4.3.3 Patch Encoding/Decoding Architecture

Our experiments so far indicate an advantage to using the U-Net up and down blocks instead of a simple linear layer.
One possible reason is that the model benefits from the inductive biases of the U-Net architecure;
an alternative hypothesis is that this advantage stems from the significant increase in overall model parameters introduced by the U-Net layers.
To decouple these two confounders, we scale up the core transformer to 7B parameters, while keeping the amount of U-Net parameters (almost) constant;(^15^1515While we do not scale the U-Net layers with the transformer in these experiments, this is a potentially fruitful avenue for future research.)
in this setting, the additional encoder/decoder parameters account for only a 3.8% increase of total model parameters, equivalent to the amount of token embedding parameters.

Table 7 shows that even though the relative benefit of U-Net layers shrinks as the transformer grows, it does not diminish.
In image generation, for example, the U-Net encoder/decoder allows much smaller models to obtain better FID scores than the 7B model with linear patchification layers.
We observe a similar trend in image captioning, where adding U-Net layers boosts the CIDEr score of a 1.4B transformer (1.67B combined) beyond the performance of the linear 7B model.
Overall, it appears that there are indeed inductive bias benefits to U-Net encoding and decoding of images beyond the mere addition of parameters.

**Table 7: Performance of linear and U-Net variants of Transfusion across different model sizes. Patch size is set at 2$\times$2 latent pixels. Model parameters refers to the transformer alone.**
| Model | Enc/Dec | $\Delta$ Enc/Dec | C4 | Wiki | Llama | MS-COCO |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Params | Params | PPL ($\downarrow$) | PPL ($\downarrow$) | Acc ($\uparrow$) | CDr ($\uparrow$) | FID ($\downarrow$) | CLIP ($\uparrow$) |  |
| 0.16B | Linear | 0.5% | 14.8 | 8.8 | 44.2 | 6.2 | 37.6 | 20.0 |
| U-Net | 106.1% | 14.4 | 8.5 | 45.7 | 15.3 | 18.8 | 23.9 |  |
| 0.37B | Linear | 0.4% | 12.0 | 7.0 | 47.9 | 11.1 | 21.5 | 22.4 |
| U-Net | 71.3% | 11.8 | 6.9 | 48.8 | 21.1 | 18.1 | 24.9 |  |
| 0.76B | Linear | 0.4% | 10.4 | 6.0 | 51.7 | 16.0 | 20.3 | 24.0 |
| U-Net | 35.5% | 10.3 | 5.9 | 51.9 | 25.4 | 16.7 | 25.4 |  |
| 1.4B | Linear | 0.4% | 9.5 | 5.4 | 53.8 | 19.1 | 19.4 | 24.3 |
| U-Net | 19.3% | 9.4 | 5.4 | 53.4 | 28.1 | 16.6 | 25.7 |  |
| 7B | Linear | 0.3% | 7.7 | 4.3 | 61.5 | 27.2 | 18.6 | 25.9 |
| U-Net | 3.8% | 7.8 | 4.3 | 61.1 | 33.7 | 16.0 | 26.5 |  |

#### 4.3.4 Image Noising

Our experiments order 80% of image-caption pairs with the caption first, and the image conditioning on the caption, following the intuition that image generation may be a more data-hungry task than image understanding.
The remaining 20% of the pairs condition the caption on the image.
However, these images are noised as part of the diffusion objective.
We thus measure the effect of limiting the diffusion noise to a maximum of $t=500$ (half of the noise schedule) in the 20% of cases where images appear before their captions.
Table 8 shows that noise limiting significantly improves image captioning, as measure by CIDEr, while having a relatively small effect (less than 1%) on other benchmarks.

**Table 8: Performance of Transfusion with and without limiting the amount of sampled diffusion noise to a maximum of $t=500$ when images appear before the caption. The models are U-Net variants encoding 2$\times$2 latent pixel patches. Metrics that change by over 1% are bolded.**
| Model | Noise | C4 | Wiki | Llama | MS-COCO |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Params | Limit | PPL ($\downarrow$) | PPL ($\downarrow$) | Acc ($\uparrow$) | CDr ($\uparrow$) | FID ($\downarrow$) | CLIP ($\uparrow$) |
| 0.76B |  | 10.3 | 5.9 | 51.9 | 25.4 | 16.7 | 25.4 |
| $\checkmark$ | 10.3 | 5.9 | 52.1 | 29.4 | 16.5 | 25.4 |  |
| 7B |  | 7.8 | 4.3 | 61.1 | 33.7 | 16.0 | 26.5 |
| $\checkmark$ | 7.7 | 4.3 | 60.9 | 35.2 | 15.7 | 26.3 |  |

### 4.4 Comparison with Image Generation Literature

Our experiments thus far have covered controlled comparisons with Chameleon and Llama, but we have yet to compare Transfusion’s image generation capabilities to those of state-of-the-art image generation models.
To that end, we train a 7B parameter model with U-Net encoding/decoding layers (2$\times$2 latent pixel patches) over the equivalent of 2T tokens, comprising of 1T text corpus tokens and 3.5B images and their captions.
While the Transfusion variant in §4.2 favored simplicity and experimental control, the design choices and data mixture (§4.1) of this variant lean a bit more towards image generation.
Figure 2 and Appendix B showcase generated images from this model.

We compare the performance of our model to reported results of other similar scale image generation models, as well as some publicly available text generating models for reference.
Table 9 shows that Transfusion achieves similar performance to high-performing image generation models such as DeepFloyd (Stability AI, ), while surpassing previously published models including SDXL (Podell et al., ).
While Transfusion does lag behind SD 3 (Esser et al., ), this model leveraged synthetic image captions through backtranslation (Betker et al., ), which enhances its GenEval performance by 6.5% absolute (0.433$\rightarrow$0.498) at smaller scale; for simplicity, our experimental setup only included natural data.
Finally, we note that our Transfusion model can also generate text, and performs on par with the Llama models, which were trained on the same text data distribution (§4.1).

**Table 9: Performance of a 7B Transfusion model (U-Net encoder/decoder layers, 2$\times$2 latent pixel patches) trained on the equivalent of 2T tokens, compared to similar scale models in the literature. Except Chameleon, all the other models are restricted to generating one modality (either text or image). ^∗ Frozen text encoder parameters. ^r Parti samples 16 images for every prompt and then reranks with an auxiliary scoring model. ^s SD 3 trains with synthetic caption data, which provides boosts GenEval performance.**
| Model | Model | Text | Images | Llama | COCO | Gen |
| --- | --- | --- | --- | --- | --- | --- |
| Params | Tokens | Acc ($\uparrow$) | FID ($\downarrow$) | Eval ($\uparrow$) |  |  |
| Llama 1 (Touvron et al., ) | 7B | 1.4T | — | 66.1 | — | — |
| Llama 2 (Touvron et al., ) | 7B | 2.0T | — | 66.3 | — | — |
| Chameleon (Chameleon Team, ) | 7B | 6.0T | 3.5B | 67.1 | 26.74 | 0.39 |
| Imagen (Saharia et al., ) | 2.6B + 4.7B^∗ | — | 5.0B | — | 7.27 | — |
| Parti (Yu et al., ) | 20B | — | 4.8B | — | ^r7.23 | — |
| SD 1.5 (Rombach et al., ) | 0.9B + 0.1B^∗ | — | 4.0B | — | — | 0.43 |
| SD 2.1 (Rombach et al., ) | 0.9B + 0.1B^∗ | — | 2.3B | — | — | 0.50 |
| DALL-E 2 (Ramesh et al., ) | 4.2B + 1B^∗ | — | 2.6B | — | 10.39 | 0.52 |
| SDXL (Podell et al., ) | 2.6B + 0.8B^∗ | — | 1.6B | — | — | 0.55 |
| DeepFloyd (Stability AI, ) | 5.5B + 4.7B^∗ | — | 7.5B | — | 6.66 | 0.61 |
| SD 3 (Esser et al., ) | 8B + 4.7B^∗ | — | ^s2.0B | — | — | 0.68 |
| Transfusion (Ours) | 7.3B | 1.0T | 3.5B | 66.1 | 6.78 | 0.63 |

### 4.5 Image Editing

Our Transfusion models, which have been pretrained on text-text, image-text, and text-image data, perform well across these modality pairings.
Can these models extend their capabilities to generate images based on other images?
To investigate, we fine-tuned our 7B model (§4.4) using a dataset of only 8k publicly available image editing examples, where each example consists of an input image, an edit prompt, and an output image.
This approach, inspired by LIMA (Zhou et al., ), allows us to assess how well the model can generalize to image-to-image generation, a scenario not covered during pretraining.

Manual examination of random examples from the EmuEdit test set (Sheynin et al., ), shown in Figure 6 and Appendix 4.5, reveals that our fine-tuned Transfusion model performs image edits as instructed.
Despite the limitations of this experiment, the findings suggest that Transfusion models can indeed adapt to and generalize across new modality combinations.
We leave further exploration of this promising direction to future research.

Figure: ((a))
Refer to caption: extracted/5802141/img_edit_samples/0_remove_cupcake.jpg

## 5 Related Work

Most existing multi-modal models are built on the idea of attaching two or more modality-specific architectures together, often pretraining each component separately in advance.
State-of-the-art image and video generation models, for instance, use large pretrained text encoders to represent their input prompts in latent space, which can then be used to condition diffusion models (Saharia et al., ).
In fact, recent work fuses representations from multiple off-the-shelf encoders to enhance performance (Podell et al., ; Esser et al., ).
A similar pattern can be observed in the vision language model literature, where typically a pretrained language model is complemented by pretrained modality-specific encoders/decoders via projection layers to/from the pretrained text space.
Examples include Flamingo (Alayrac et al., ) and LLaVA (Liu et al., ) for visual understanding, GILL (Koh et al., ) for visual generation, and DreamLLM (Dong et al., ) for both visual comprehension and generation.
In contrast, Transfusion has one unified architecture learned end-to-end to generate both text and images.

Prior work on end-to-end multi-modal models includes examples such as Fuyu (Bavishi et al., ), which uses image patches as inputs for visual understanding, and Chameleon (Chameleon Team, ), which converts each image to a sequence of discretized tokens and then trains over the combined text-image token sequences.
However, these approaches are either restricted to input-level multi-modal tasks, or lag behind state-of-the-art models (i.e. diffusion models) in continuous data generation.
Transfusion provides a simple, end-to-end solution to multi-modal learning that understands and generates high-quality multi-modal data.

An interesting area of recent acrive research is the application diffusion models and their generalizations to discrete text generation (Li et al., ; Gat et al., ).
However, this approach has yet to achieve the performance and scale of standard autoregressive language models.
Future research in this direction may unlock new ways to fuse discrete and continuous modalities in a single model.

## 6 Conclusion

This work explores how to bridge the gap between the state of the art in discrete sequence modeling (next token prediction) and continuous media generation (diffusion).
We propose a simple, yet previously unexplored solution: train a single joint model on two objectives, tying each modality to its preferred objective.
Our experiments show that Transfusion scales efficiently, incurring little to no parameter sharing cost, while enabling the generation of any modality.

## Acknowledgments and Disclosure of Funding

We would like to thank Horace He, Songlin Yang, Jiatao Gu, and Ishan Misra for helpful discussions throughout this project.

## Appendix A Autoencoder Details

The training objective for our VAE closely follows that of Esser et al. []:

$$ $\mathcal{L}_{\text{VAE}}=\mathcal{L}_{1}+\mathcal{L}_{\text{LPIPS}}+0.5 \mathcal{L}_{\text{GAN}}+0.2\mathcal{L}_{\text{ID}}+0.000001\mathcal{L}_{\text {KL}}$ $$

where $L_{1}$ is L1 loss in pixel space, $L_{\text{LPIPS}}$ is perceptual loss based on LPIPS similarity Zhang et al. [], $L_{GAN}$ is a patch-based discriminator loss, $L_{\text{ID}}$ is a perceptual loss based on internal features of the Moco v2 model Chen et al. [], and $L_{\text{KL}}$ is the standard KL-regularization term to encourage encoder outputs towards a normal distribution.
We delay the beginning of GAN training (i.e. including the adversarial loss in the loss function) to 50,000 steps, in order to let the VAE achieve sufficiently good reconstruction performance.
We use a latent dimension of 8.

The training objective for the VQ-GAN matches that of the VAE, with one notable exception: we replace the $\mathcal{L}_{\text{KL}}$ loss with the standard codebook commitment loss $\mathcal{L}_{\text{codebook}}$ [Van Den Oord et al., ], which encourages encoder outputs and codebook vectors to be close together.
We use $\beta=0.25$, and use loss weighting $1.0$.
The final loss function for the VQ-VAE is therefore:

$$ $\mathcal{L}_{\text{VQ-VAE}}=\mathcal{L}_{1}+\mathcal{L}_{\text{LPIPS}}+0.5 \mathcal{L}_{\text{GAN}}+0.2\mathcal{L}_{\text{ID}}+\mathcal{L}_{\text{ codebook}}$ $$

The vector quantization layer is applied after projecting the encoder outputs to 8-dimensional space.
Outside of the loss function change and the quantization layer, the training setup for the VAE (for Transfusion) and VQ-VAE (for Chameleon) are the same (e.g. same amount of training compute, same training data, and same encoder/decoder architecture).

## Appendix B Examples: Image Generation

Figure 7 and Figure 8 show examples of images generated from a 7B Transfusion model trained on 2T multi-modal tokens (§4.4).

## Appendix C Examples: Image Editing

Figure 9 show random examples of image editing by a fine-tuned 7B Transfusion model.

Figure: ((a))
Refer to caption: extracted/5802141/samples/downtown_seattle_sunset.jpg

Figure: ((a))
Refer to caption: extracted/5802141/samples/woman_under_blanket.jpg

Figure: ((a))
Refer to caption: extracted/5802141/img_edit_samples/0_change_cloest_keyboard_to_black.jpg

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: The structure of the UNet is unclear to me. the paper says "since U-Net blocks contain bidirectional attention within, independent of the transformer’s attention mask, this gap is less pronounced when they are applied." What U-Net architecture uses bidirectional attention? Is the UNet applied to each patch independently, or is patching done after the U-Net?
soundness: 3
strengths: - problem is well motivated
- method is simple and clear
- results are good
- abalations cover most of my questions
summary: The paper proposes a multi-modal transformer model where text tokens are sampled auto regressively and image "tokens" are diffused in a VAE latent space.
weaknesses: the method relies on a pre-trained VAE. It would have been nice to see an ablation done in pixel space or with some type of co-training scheme to avoid a 2-stage training process.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness
soundness: 3
strengths: 1. This paper verifies the feasibility combining different learning targets with one framework for respective understanding and generation.

2. It reveals the scaling performance comparing Transfusion and Chameleon, providing good insights for future works.

3. It provides detailed ablation study and analysis to demonstrate the effectiveness of each component.
summary: This paper proposes Transfusion, a unified model for joint understanding and generation. During training, it incorporates CE loss for text learning and diffusion loss for image learning. In inference, it uses casual attention for autoregressive text generation and bidirectional attention for parallel image generation. Compared to Chameleon, Transfusion attains good performance for visual understanding and generation.
weaknesses: 1. The visual understanding performance of Transfusion is not convincing enough, as the paper doesn't provide any results on related benchmarks, such as MME, MMBench, etc.

2. The image editing performance also needs quantitative results to show its efficacy.

3. Does this framework has the potential to conduct image-text interleaved generation?

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: As shown in Weaknesses.
soundness: 4
strengths: 1. Originality: The paper introduces a unique approach by combining language modeling and diffusion objectives over shared data and parameters. This method allows for the seamless generation of discrete and continuous modalities.

2. Quality: The paper demonstrates the effectiveness of the Transfusion model through extensive experiments and comparisons with existing methods. 

3. Clarity: The paper is well-structured, with clear explanations of the Transfusion method, its architecture, and the training objectives. 

4. Significance: The paper addresses a critical challenge in handling both discrete and continuous data modalities.
summary: The paper introduces Transfusion, a novel approach for training a single unified model capable of understanding and generating both discrete and continuous modalities. The authors demonstrate that by combining the language modeling loss function (next token prediction) with diffusion, they can train a transformer model over mixed-modality sequences. This method allows for seamless integration of discrete text tokens and continuous image data, without the need for information loss through quantization.
weaknesses: #### More experiments are encouraged.
1. Results on multimodal Understanding tasks are limited. For example, VQA, Summarization, VideoQA and etc.
2. I encourage the authors to scale up the model parameters or training data size (Scaling Law on Transfusion is very important), but I understand the computation resources on scaling law are huge. Thus, this is a minor issue.

#### Compare the technical differences of several recent works (e.g., Show-o, MIO, Emu3).

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: same as weakness
soundness: 3
strengths: 1. Innovative Framework: Transfusion introduces a novel approach that seamlessly combines text and image modalities, achieving impressive results in image generation and editing.
2. Robust Comparative Experiments: The paper includes comprehensive comparative analyses and detailed ablation studies that examine various dimensions, such as model architecture and parameters, providing strong evidence for the framework's effectiveness.
3. Clear and Fluid Writing: The writing is clear and the structure of the paper is well-organized, making it easy to follow the authors' arguments and findings.
summary: The paper introduces Transfusion, a novel approach for training multi-modal models that effectively handle both discrete (text) and continuous (image) data. By combining next token prediction for text and diffusion for images within a single transformer architecture, Transfusion eliminates information loss associated with traditional quantization methods. The model was pretrained on a balanced dataset of text and images, utilizing modality-specific encoding and decoding layers to optimize performance. Experimental results demonstrate that Transfusion outperforms existing methods like Chameleon in various benchmarks, achieving significant efficiencies in both text-to-image and image-to-text generation tasks. The study concludes that Transfusion represents a significant advancement in creating integrated multi-modal generative models.
weaknesses: Lack of demonstration of multi-modal understanding capabilities: While the paper focuses on image generation and editing capabilities, it does not present extensive benchmarks for the model’s general visual question answering (VQA) abilities. It would be beneficial to include benchmarks related to image understanding, such as VQAV2, SEED Bench and MMMU.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: -How does Transfusion take the embedding of noise timestep in the training/inference of image latent denoising process?

-In the training phase, the model calculates the AR loss and diffusion loss in one forward pass using text tokens and 'noised' image latent, which introduces unavoidable discrepancy when the model inference with clear latent. Wouldn't it be more accurate to calculate AR and diffusion loss separately in two model passes?


-Is there a reason that the COCO FID scores of StableDiffusion models are not in Table 6?
soundness: 2
strengths: -They suggested a novel training and inference pipeline for a transformer that allows performing both AR and diffusion modeling with shared parameters.

-The manuscript contains extensive controlled experiments and ablation studies.
summary: This paper explores how to bridge the modality gap to inject continuous image generation ability into discrete text-based LLMs by proposing a single transformer model trained with autoregressive modeling and diffusion loss simultaneously. This method of Transfusion avoids image discreteness and still enables input and output-level image tasks. By carefully designing attention masks and input formation, the transformer models text data by AR modeling with causal masking and generates images by denoising with bidirectional attention.

Compared to injecting image comprehension/generation ability by introducing discrete image tokens(Chameleon), the authors demonstrated their approach performs better in various scales in text-to-image generation, image captioning, and even text-only tasks. The large-scale model of Transfusion showed comparable performance to previously published large image generative models.
weaknesses: -The comparison with previously published works only covered text-only & text-to-image tasks and didn't include their performances on image comprehension tasks(e.g. image captioning). 
An additional comparison for these tasks with previous multi-modal LLMs that process images differently(e.g. input-only with an external encoder like LLaVA, or image quantization like Chameleon) would support the effectiveness of Transfusion's approach.

Meta-review: **Summary**. The paper introduces Transfusion, a unified multi-modal model trained jointly on text and image data. By combining next-token prediction for text and diffusion-based learning for images within a single transformer architecture, it bridges the modality gap without quantizing images into discrete tokens. The study establishes scaling laws for this architecture and evaluates its performance against other multi-modal models such as Chameleon. The authors demonstrate that Transfusion achieves state-of-the-art results in text-to-image generation, while maintaining competitive performance on text-only tasks.

**Strengths**.
(1) Innovative Idea: The combination of autoregressive modeling for text and diffusion processes for images within one model is novel and effectively addresses the modality gap.
(2) Comprehensive Experiments: Extensive ablation studies and comparisons with state-of-the-art models provide strong evidence for the validity and robustness of the proposed method.
(3) Scaling Insights: The paper provides insights into scaling laws, which are valuable for guiding future research in multi-modal modeling.
Clear Presentation: The manuscript is well-structured, with detailed explanations of methods and results, making it accessible to a broad audience.

**Weaknesses**.
(1) Limited Multi-Modal Understanding Results: The paper lacks comprehensive benchmarks on tasks like visual question answering (e.g., VQAv2, MMEU), which are critical for assessing multi-modal understanding.
(2) Image Editing Evaluation: Quantitative results for image editing tasks are not sufficiently detailed, limiting the assessment of this capability.
(3) Training Discrepancies: The reliance on noised image latents during training for image understanding tasks introduces discrepancies with inference, though this was partially addressed in the rebuttal. Comparisons with Recent Methods: Technical differences with some contemporary works (e.g., Show-o, MIO, Emu3) are not discussed thoroughly.

**Reasons**
The paper presents a very novel and effective method for multi-modal modeling for joint understanding and generation, supported by extensive experiments and insights into scaling behavior. Despite some limitations in demonstrating comprehensive multi-modal understanding and minor training discrepancies, the strengths far outweigh the weaknesses. The clarity of presentation and significance of contributions to the field justify its acceptance.

Decision: Accept (Oral)