LMFusion: Adapting Pretrained Language Models for Multimodal Generation
Authors: Weijia Shi
Xiaochuang Han
Chunting Zhou
Weixin Liang
Xi Victoria Lin
Luke Zettlemoyer
LILI YU
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal
Primary area: applications

Abstract: We present LMFusion, a framework for empowering pretrained text-only large language models (LLMs) with multimodal generative capabilities, enabling them to understand and generate both text and images in arbitrary sequences. LMFusion leverages existing Llama-3's weights for processing texts autoregressively while introducing additional and parallel transformer modules for processing images with diffusion. During training, the data from each modality is routed to its dedicated modules: modality-specific feedforward layers, query-key-value projections, and normalization layers process each modality independently, while the shared self-attention layers allow interactions across text and image features. By freezing the text-specific modules and only training the image-specific modules, LMFusion preserves the language capabilities of text-only LLMs while developing strong visual understanding and generation abilities. Compared to methods that pretrain multimodal generative models from scratch, our experiments demonstrate that, LMFusion improves image understanding by 20% and image generation by 3.6% using only 50% of the FLOPs while maintaining Llama-3's language capabilities. We also demonstrate that this framework can adapt existing vision-language models with multimodal generation ability. Overall, this framework not only leverages existing computational investments in text-only LLMs but also enables the parallel development of language and vision capabilities, presenting a promising direction for efficient multimodal model development.

LMFusion: Adapting Pretrained Language Models for Multimodal Generation
Weijia Shi""
Xiaochuang Han ".
Chunting Zhou f Weixin Liang*
Xi Victoria Lin
Luke Zettlemoyer "ff Lili Yu
" "University of Washington
/FAIR Meta
SStanford University
swj04190ux. edu
xhan770uw. edu
liliyu@meta.com
Abstract
We present LMFusion, a framework for empowering pretrained text-only large language models (LLMs) with multimodal generative capabilities, enabling them understand and generate both text and images in arbitrary sequences. LMFusion leverages existing Llama-3's weights for processing texts autoregressively while introducing additional and parallel transformer modules processing images with diffusion. During training, the data from each modality is routed to its dedi- cated modules: modality-speciift feedforward layers, query-key-value projections, and normalization layers process each modality independently, while the shared self-attention layers allow interactions across text and image features. By freezing the text-specific modules and only training the image-specific modules, LMFusion preserves the language capabilities text-only LLMs while developing strong visual understanding and generation abilities. Compared to methods that pretrain multimodal generative models from scratch, our experiments demonstrate that LM- Fusion improves image understanding by 20% and image generation by 3.6% while maintaining Llama-3's language capabilities. We also show that this framework adapt existing vision-language models with multimodal generation ability.
Introduction
Over the past few years, have seen significant progress in multimodal generative models capable of understanding and generating interleaved text and images in arbitrary sequences [ 3 Models like Transfusion (4. Chameleon [5]. and Unified-IO (6] 7 demonstrate the potential of unified architectures that seamlessly handle both image and text modalities. However, these models typically train from scratch, demanding significant computational resources achieve proficiency across all modalities. The computational cost of mastering even a single modality is substantial-training state-of-the-art text-only large language models (LLMs) like Llama-3 [ requires training over 15 trillion tokens.
Given these computational demands, investigate alternative paradigm that reuses and adapts existing pretrained LLMs [T]. We address a fundamental research question: How to preserve the text-only performance of pretrained LLMs while equipping them with visual understanding and generation abilities? Our experiments show that naive finetuning of text-only LLMs on multimodal data leads significant degradation of their language processing capabilities.
To address this challenge, we introduce LMFusion, a framework that enhances a pretrained text-only LLM, Llama-3 [8] with multimodal capabilities by building upon the recipe of Transfusion [4] Drawing from recent and parallel work on modality separation [12] [3] 15), LMFusion integrates the original Llama modules pretrained for language processing while introducing additional dedicated transformer modules for visual understanding and generation tasks. As shown in Figure I we
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

cat
with
secrets to
keep
[BOI]
VAE Decoder
Head
U-Net Upsampler
TTest Module
Imagu Module
FFN
FFN
Self-Attention
X.V
Input Embedding
U-Net Downsampler
with secrets
keep
Encoder
+nolee
Figure I: Overview of LMFusion. It uses modality-specific FFNs and QKV projections to process text and image data separately: the text "A cat with secrets to keep" goes to the text module, while the image patches of the cat goes to the image module. the self-attention layer, text and image representations can attend to previous contexts across the modality boundaries. Both modules initialized from Llama-3, with the text module frozen to preserve language capabilities while the image module trained on image data. Layer normalization and residual connections are folded into the QKV and FFN modules. A special BOI token separates different modalities.
employ modality-speciffi query- weey--key-value (QKV) projections and feed -orward networks (FFNs) to process and image data separately while still allowing for cross-modal interactions the joint self-attention layer. By freezing the text modules while finetuning the image modules, we preserve the language-only capabilities of pretrained LLMS while giving head start to the learning of visual understanding and generation. Compared to pretraining multimodal generative models from scratch, this approach avoids the need to include text-only data in the training process, significantly reducing the computational demands.
To evaluate the effectiveness of approach, we conduct comprehensive experiments comparing LMFusion with Transfusion in controlled settings. Specifically, we initialize our LMFusion archi- tecture with a pretrained Llama-3 8B model and continue training on the same image data as in Transfusion [4]. Compared to Transfusion, LMFusion achieves a 20% improvement in image understanding and 3.6% improvement in image generation. also preserves Llama-3's text-only per- formance that outperforms Transfusion by 11.6%. Figure 2rreesnts images generated by LMFusion. Additionally, we further demonstrate that this framework can adapt existing vision- laiiunlanguugg models (e.g., LLaVA) with multimodal generation ability.
Through ablation studies, we analyze the key architectural decision for LMFusion: separating both self-attention and FFNs for different modality data while freezing weights for the pretrained language modality. We show that naive finetuning of the dense pretrained LLMs multimodal data (no separation) leads to a catastrophic forgetting of their original language capabilities. Furthermore, deep separation proves be more effective than shallow separation (using modality-speciffe FFNs only), with both approaches outperforming models with no separation.
2 Background: Transfusion
Transfusion is single unified multimodal model that is capable of text generation, image understanding, and image generation tasks, by jointly predicting next tokens in language and diffusing image representations. Given a multimodal input (a. ",) Transfusion jointly learns do language modeling 2. on at and image diffuusion 2$2. on in Its architecture is same standard Transformer [16] with an additional U-Net that projects image representations down and up before and after diffusion.

---

sfafetune g48 jaccly
Beasthid
Tartle ardenster.
machin that wffer for human souds
1886 yerbbok pieto with srimaa dussed as fumans.
siambou nn rize
vesticti farkit
START
Mankes
bikkiit
board
aniieo
Lible
aphinn made glam Figure 2: Generated images from LMFusion fine-tuned on aesthetically appealing images for im- proved quality.
An diephurt to
destoris
vat.
up photo
2.1
Language Modeling
Given sequence of discrete language tokens x 21" ..... 244 language model 0 represents its joint probability by P(x"") = II: Pa x ). This formulation sets an autoregressive task, where each token wt" predicted based preceding tokens w The language model is learned by minimizing the cross-ente cross-entropy between Po and the observed data distribution, which is commonly referred as the LM loss:
LIM = 1- log Po(a:"
(1)
Optionally, if there exists image data preceding the language tokens (e.g., image-caption data), Transfusion adds the representation of !ms as additional condition to the objective. More details of representing !mg presented below.
2.2 Image Diffusion
Given raw image, Transfusion first encodes the image into sequence of continuous latent represen- tation nm with pretrained and frozen VAE tokenizer [18]. It then employs Denoising Diffusion Probabilistic Models (i.c., DDPM) to learn to reverse gradual noise-addition process added in the forward process [19]. In the forward diffusion process, Gaussian noise N/ I) is added to the image representation img over T steps, creating sequence noisy image representations Lp, 1.,..., Dr. Specifically, each step t, the noisy image representation is given by:
at
img
/ ar img
(2)
20 In the reverse process, the diffusion model 6a(-) with parameters 0 learns to predict the added noise e given the noisy data x. at timestep f and context !M that can include text prompts such as captions to the image diffusion:
Here ar follows a common cosine schedule
LDDPM
en (x
ing
(2)
The Transfusion architecture contains U-Net downsampler and upsampler reduce the dimension
of imR The U-Net downsampler transforms the image into fewer patches before the main Trans- former modules while the upsampler projects them back to the original dimension of !ng after the Transformer.
"Siiilar to " this context can also include image representations x ""W under image editing setup. We omit it in the notation for simplicity.

---

2.3 Training Objective
During training, Transfusion optimized predict both the LM loss on the text input x and the diffusion loss on the image input " These two losses are combined using hyperparameter A:
Transfusion
LLM A LDDPM
(4)
3
LMFusion
One notable feature of Transfusion that it has the same architecture mainstream LLMs (c.g.. Llama [2I]) while being capable of text generation, image understanding, and image generation together, through an end-to-end training (Equation 4p. 4 trains Transfusion from scratch using language-only and image-caption data. However, such training from scratch requires substantial computational resources, and its performance on language-ond
tasks still lags behind the pretrained, text-only LLMs. In this work, we aim to effectively adapt pretrained, text-only LLMs handle image understanding and generation tasks. Specifically, we build on an open-weigh LLM,, Llama-3 (8) and continue training it with the Transfusion objectives to handle both modalities. Since Transfusion uses shared parameters for its language modeling and image diffusion objectives, the key challenge is to prevent Llama-3's strong text-only performance from dropping while optimizing for its new image capabilities.
3.1 Model Architecture
In response to the challenge above, we propose LMFusion, framework that combines pretrained, text-only Llama model with a dedicated image transformer for visual generation and understanding. enabling each modality be processed through independent weights. By freezing the text modules while finetuning the visual modules, we preserve its language-only capabilities while giving the learning of visual understanding and generation a boost start.
LMFusion is decoder-only model consisting of N transformer layers. As shown in Figure central the design are the modality-specific attention layer and Feed-Forward Network (FFN), each handling only data from its corresponding modality. Without loss of generality, we describe LMFusion below a configuration with single transformer layer, folding residual connections and layer normalization directly into the self-attention and FFN. The inputs to the model are text tokens img ad and noisy image representations Ft Virgaeims vi 0.e. We use blue for text-specific modules and red for image-specific modules.
Input projection The input text tokens x are projected by linear embedding layer to sequence of text hidden states h The noisy image a.; are projected sequence of image representations ho via U-Net downsampler.
Pro (xx
(5)
in
h JUNet-Down, (z, t)
Then the text hidden states h or image hidden states ims are fed into the following attention layer. Modality self-attention We create separate attention matrices for each modality. Specif- ically, the text hidden states h and image hidden states h are converted into their respective queries, keys, and values via separate Q, K,, V matrices. The pre-attention layer normalization also modality-specific and is folded into the QKV functions.
h h h
QKV text
EXT
img
ing hik
QKV img
(h
(8)
We enable cross-modal attention by concatenating the queries, keys, and values from both image and text modalities into unified sequences. The ditention-weighter values at text and image tokens then projected back into the hidden state dimension using separate O weights for each modality. img h
+ M
txt ho
O texi
(softmax
(h img 1w))
(9)
Va.
img [
h IT
+ M
too
img
O img
(softmax
(10)

---

where denotes concatenation. M represents hybrid attention mask same as in Transfusion [4] with causal mask applied to text tokens and di-directional mask applied image tokens. This design allows for self-attention within and across modalities, encouraging cross-modality integrations.
-speeity-speciffic feed-forward network After the attention layer, we employ modality-speciffe FFNs to process text and image data separately. The pre-FFN layer normalization is also modality- specific and is folded in the FFN functions.
FFN text
(ho
(11)
ft FFN
img
FFN img
(
img)
(12)
Output projection are projected either to logits in text via language model's output layer, or to predicted noise in image via a U-Net upsampler.
Finally, after N layers of self-attention and FFNs, the resulting hidden states
Plogits
LM- MMM-eadkoxt
(PFFN)
(13)
Cpred
ENNct-Upin
(h t, h img
(14)
Same as Transfusion, the output Plagits and Epred are passed through the language modeling loss (Equation pb and DDPM loss (Equation 3/ respectively. All parameters in the text modules along with sell-attention and FFN parameters in the image modules are initialized from the pretrained Llama model. During optimization, we decouple the learning rates for the text and image parameter
groups: a text learning rate, Text, is used for Prooear QKV leu.. Otest. F LM-Heeelex! } and an
image learning rate, Thm-- for { JUNet-Downa QKV 'img" Oung. FFNN UNet- Uiket-UPimmg To preserve the model's performance on text-only benchmarks, we use Thext = 0 (freezing text modules) for our main experiments and explore different configurations in
Experiments
4.1 Training Setup
Data Following Transfusion [4], we use the same collection of 380M Shutterstock image-caption data, where each image is center-cropped and resized to 256 x 256 pixels. We order the captions before images (i.e., emphasizing image generation conditioned on texts) 80% of the time, and order the images before captions for the rest.
Model Details For image tokenization, we use the same VAE encode as Transfusion to compress an image of 256 x 256 pixels into 32 x 32 x 8 tensor. These tensors then passed into 2-block U-Net downsampler (17 to further reduce dimensions, resulting in sequence of 256 patches (tokens). Both text-specific and image-specific Transformer modules are initialized from the pretrained Llama- 3 8B model [8] The U-Net downsampler and a corresponding U-Net upsampler, totaling 0.27 billion parameters 4 are trained from scratch. Like Transfusion, LMFusion uses maximum context length of 4096 tokens.
our main experiments, to preserve the language-only performance, we freeze the text modules (Thex 0) while training only the image modules using an AdamW optimizer (3, = 0.9, 32 = 0.95, 6 = 1 x 10-8) with learning rate Vimage x 10-4, The learning rate follows a cosine decay schedule with 4000-step warmup period before gradually decreasing to 1.5 x 10-5 5 The model trained using 128 H100 GPUs over 4 days.
Optimization
4.2 Evaluation Setup
We compare our model with both the original Transfusion 7B model trained from scratch 4 and the Transfusion model initialized from the same LLaMA model. 3 The original Transfusion was trained for 250K steps on 0.25T language-only tokens (text data) and 0.25T image-captions tokens (image data). Since we freeze and reuse the text module from existing text-only models during training, we exclude text data from training process while maintaining language capabilities. In the first
hhttps://huggingfacce https///wweo///abbaaaaa////eeeee
TTransfusion 7B and Llama-3 3 8B have the same Transformer sizes. The size difference is due to the different vocabularies, which affects input and output embedding layers.
5

---

Language-only Evaluation
Image Understanding
Image Generation
(without | with CFG)
Model
HellaSwagt
SIQAt
WinoGrandet
CIDEr t
FID
CLIP t
Llama 3
60.0
48.1
72.8
Transfusion
51.0
42.3
64.3
32.0
14.4
8.70
22.1
24.4
LMFusion (0.5x FLOPS)
60.0
48.1
72.8
38.3
13.9
8.75
22.0
24.4
LMFusion (1.0s FLOP))
60.0
48.1
72.8
38.4
14.0
8.61
22.1
24.4
Table I: Results across text-only benchmarks, image understanding and image generation. LMFusion preserves Llama-3's text performance while adding strong image understanding and generation capabilities. Image generation results are obtained without classifier-free guidance or with a CFG factor of 1.55, Additionally, we show detailed analyses of our full LMFusion recipe vs. the vanilla Transfusion recipe initialized from Llama-3 in Figure 4 and F Figure 5
configuration, we use the amount of 0.25T image data in Transfusion while leaving out the text data (approximately half the total FLOPs of Transfusion), while in the second configuration, we match Transfusion's 's total FLOPs. Following Transfusion, we evaluate LMFusion on language-only, image understanding, and image generation tasks.
Language-only: We evaluate the model's language abilities using four tasks from the standard Llama evaluation suite [8], including Hellaswag (22]. PIQA [23]. SIQA (24). and WinoGrande [25]. We report accuracy on these benchmarks. Image Generation: For evaluating image generation, we use the MS-COCO benchmark (26). We generate images for 30K randomly selected prompts from the validation set and measure the Frechet Inception Distance (FID) 27 and CLIP scores [28]. Our image generation results include versions obtained without classifier-free guidance and with CFG coefficient of I 1.55 or 1.6. Image Understanding: We evaluate the models' ability to generate image descriptions using the test split of MS-COCO [26]. reporting CIDEr scores (29).
4.3
Results
Table compares two variants of LMFusion against Transfusion. On language-only benchmarks, LMFusion keeps the strong performance of Llama-3 since we freeze text modules. For image understanding, LMFusion substantially surpasses Transfusion, with 20% improvement. In image generation tasks, LMFusion also shows superior results in both FID and CLIP scores. Furthermore, in $5 we show from Figure 4 that LMFusion outperforms Transfusion initialized from Llama-3 (i.e., dense model with no separation) during the training process. In Figure 3] we benchmark the performance of LMFusion and Transfusion throughout the training [ We observe consistent advantage of LMFusion over Transfusion during the entire training. for image captioning and generation. These results suggest that LMFusion effectively leverages the pretrained language modules from Llama while developing strong image abilities. Although LMFusion has twice as many parameters Transfusion, it uses same training FL.OPs since only half of the parameters activated for each input token from an arbitrary modality.
5 Analysis
Central to LMFusion our modality separation techniques, which employs the design of modality- specific modules and decoupled learning rates for language and image modules. Our architectural ablation ($5.1) demonstrates the importance of the design for maintaining model performance across both modalities. We further showcase that this recipe could be used for adapting pretrained vision-language models.
5.1
Architecture Ablations
5.1.1
Experimental Design
To evaluate different design choices, we conduct ablation studies using small-scale variants of LMFusion. Our analysis focuses on the impact modality separation by comparing three designs:
'For the image generation results plotted throughout the training, we use smaller subset of 5K prompts and without classifier free guidance.

---

Image Und.
Image Gen.
Model
Base LLM
MMMU
ChanQA
RealWoridQ 1
MME-P 1
FID
EMU-3
31.6
51.8
57.4
12.8
Show-O
Phil-1.5 1.3B
27.4
1435.7
9.2
Janus
DeepSeek 1.3B
30.5
1338.0
8.5
Chameleon
28.4
0.0
19.6
26.7
MetaMorph
LLaMA-3.1 8B
41.8
37.1
58.3
11.8
Transfusion
8.7
LLaVAFusion
LLaVA-Next 8B
41.7
69.5
60.0
1603.7
8.2
Table 2: Comparison of multimodal models across image understanding and generation capa- bilities. Models are evaluated on various image understanding benchmarks and image generation quality (FID). The models without base LLM are pretrained from scratch.
CIDEr
CLIP
FID
22
20
LMFusion
35
Transfusion
30
20
30
18
Performance
HellaSwag
WinoGrande
PIQA
78
05
70
50
60
45
60
72
50
100
200
250
50
100
200
250
50
100
200
250
Steps (K)
Figure 3: Evaluation of LMFusion and Transfusion during training. LMFusion keeps the text performance of Llama throughout training, while achieving better image understanding ability (CIDEr) and image generation quality (CLIP, FID).
(1) separation (a single dense model), (2) shallow separation (using mopality- FFNs only), and (3) deep separation (using both modality-specifie FFNs and attention mechanisms, our final LMFusion).
No separation (dense model) We begin our experiments with the dense Llama-3 8B model trained using the Transfusion recipe. This dense model maintains a unified structure where most components are shared across modalities (a single set of QKV, O and FFN process both texts and images), with the exception of U-Net upsampler and downsampler. For training, we use a text learning rate (Thea) for the components initialized from the text-only LLM { (Projex QKV, O, FFN, LM-Head }. and an image learning rate Ting for { [UNet- DoweDDD UNet-UPing }. To investigate the impact of learning rate decoupling, we experiment with various learning rate ratios Thos
E (0, 0.1, 1). with
Thngge
a constant image learning rate Vimage
10-4, the same as the main experiments. ratio of represents standard continual pretraining where all components share the same learning rate, while ratio indicates a complete freezing of text-related components.
Shallow separation (modality-speciffi FFNs only) We explore simplified variant of LMFusion that separates only FFNs into text-specific and image-specific modules-a common approach in mixture-of-experts architectures [3] 30]. In this setup, we use single shared attention mechanism (QKV, O) for processing both image and text data. For training, we employ separate learning rates: Vhext for text-related components [Project> QKV, O, FFN LM-Heeddexl } and Vlm for image-related components {
FFNing. Unet- SinetUppmeg Upmmg). }. We experiment with various
learning rate ratios
Thange (0,0.1,
Deep separation (modality-specifiic FFNs and attention) Our L.Fusion, as described in section 3 represents deep separation design where both FFNs and attention mechanisms modality-specffe
7

---

HellaSwag
Peromanc
15
15
14 100
180
Seags [K]
Figure 4: Performance of naive Llama-3 finetuning (no separation) with varying Ir ratio Thed
When directly finetuning the Llama-3 model for multimodal generation, using the same learning rate for both text and image components (Ir ratio = 1) substantially reduces its text-only performance. Lowering the learning rate for the text component relative to the image component (Ir ratio 1) helps preserve language performance but slows down the acquisition multimodal abilities.
HalleSong
Peefomarce
10
Sepacation
Shollow Separsice
Deep Squrrtiin
300
150
110
Seaps (K]
Figure 5: Performance of no separation (dense model), shallow separation (modality-speciffie FFNs only), and deep separation (modality-speciffi FFNs and attention) when text modules are frozen. Deep modality separation outperforms shallow separation and no separation.
While our primary configuration freezes text modules during training. we also analyze the impact of different learning dynamics by varying the learning rate ratio That. across {0, 13.
Thraage
In the ablation study, all models are trained for 250K training steps with sequence length of 4,096 tokens and batch size of 250K tokens. The training data comprised 0.03T text-only tokens and 0.03T image-caption tokens. All other hyperparameters remained consistent with those employed in our main experiments.
5.1.2 Results
Naive finetuning of dense pretrained LLMs for multimodal generation compromises their original language capabilities. When directly finetuning Llama-8B (no separation) using the Transfusion recipe, we observe significant performance trade-offs between image and text capabilities Figure 4). With equal learning rates for text and image components ( (_eer.. 1), the model shows "yrrgg
continuous improvement in image understanding and generation. However, this comes at substantial cost to language capabilities, with performance
HellaSwag dropping by 15% initially. While language performance improves during training, never recovers to the original Ilaaa-3 model's level, maintaining persistent 7% gap.
To mitigate this issue, we explore setting Thraa
< 1, which allows us train image-specific modules (U-Nets) with a regular learning rate while preserving text abilities using smaller learning rate for the general Transformer components. Figure 4]shows this improves language-only benchmark performance, reducing the gap from 7% to 2% when the ratio is O.I. However, for dense models, this improvement comes at the cost of consistently reduced image capabilities. Overall, while learning rate decoupling offers some mitigation to the text performance drop, training dense pretrained LLMss without modality separation remains suboptimal.
Thest
Deep Modality Separation Outperforms Shallow Separation. In Figure we compare three architectures: no separation (dense), shallow separation (modality-ppecific FFNs only), and deep separation (modality-specific FFNs and attention). We set Thaa 0 (freezing the text module)
Thrags across all models to maintain Llama-3's 's text performance. Both separation approaches significantly outperform the dense model on image benchmarks. While shallow separation performs slightly worse image understanding, the performance gap widens notably in image generation tasks.
Additionally, deep separation with Iaa 0 has the same amount of tunable parameters no haus
separation with Thee = 1. Despite the intrinsic advantage of modality separation for text-only tasks,
Mang?

---

for image understanding and generation, we still observe that deep separation (blue curve in Figuure 5) are better than no separation (blue eurve in Figuure 4). These results show that modality separation crucial for adapting pretrained language-only LLMS for multimodal generation.
5.2 LLa Vussion::::: extending LMFusion to vision-language models
LMFusion continues training the language-only pretrained LLM Llama with the Transfusion recipe. Can this recipe be extended to on vision-language models (VLMs) such as LLaVA 31
32 and
Qwen-VI. 33 well? In this section, we extend the recipe of LMFusion to VL.Ss, preserving their multimodal understanding capabilities while introducing image generation abilities. Specifically, we build LLaVA-NeXT (32), freezing its transformer parameters and integrating dedicated, image-specific transformer module trained in parallel. We use the same data and model settings as LMFusion. We refer this model VLaVoos and demonstrate its image understanding performance on MMMU (4], MME- AMME-Perception 35 ChartQA [66, and Real Real WorldOA (37], as well as its image generation results. For baselines, compare VA against EMU-3 [8), Show-O (39]. Janus [40,, Chameleon [41], MetaMorph [42], and Transfusion [4]. As shown in Table 2 LLaVAFusion LLAVAFusion demonstrates strong performance in both image understanding
and generation when compared to other unified multimodal LMs. This demonstrates that LMFusion
is promising extension not only to language-only LLMs but also to VLMs, enhancing the multimodal generation capabilities in both cases.
6 Related Work
Unified Models for Multimodal Generation: Recent work has extensively explored unified frame-
works for multimodal generation, including text generation, image understanding, and image gen- eration. While texts are commonly represented as discrete tokens across models, approaches to representing amages-especiane
for image generation-vayy significantly. For instance, methods in (6 43 5 44 Iل]. represents images using vector-quantized discrete tokens [45] 46 47]. An
alternative method, adopted by [48] 49], employs continuous embeddings that require a separate diffusion model for decoding. In this work, we build upon Transfusion [4. which integrates au- toregressive generation for texts with diffusion for images within single model. Model Sparsity:
Model sparsity through Mixture of Experts (MoE) [50] 30 51 52 has proven highly effective in improving LLM training efficiency. This approach has recently been extended to multimodal
53 54 55), particularly to address potential conflicts between different modalities. For 57 replace standard Transformer FFNs with modality experts, enabling separate processing paths for different modalities. Our work takes this concept further by using modality-speciff attention mechanisms. Concurrent work [15] demonstrates the effectiveness of this deeper separation in multimodal pretraining and image generation. Reuse of LLMs in Multimodal Training: Based on the strong language capabilities of LLMs, some recent models on multimodal generation initializes their models from pretrained, language-only LLMs. For example, [9] 10, 1
models 12
example, recent efforts
JTl 58) continued training upon the weights of language-oni LLMs (2 or vision LLMs without visual generation capabilities (33] The main focus of our work is to effectively reuse pretrained LLMs for multimodal generation, particularly with the Transfusion recipe, without any compromise the LLMs' existing text-only capabilities
44
7 Conclusion
We present LMFusion, framework designed to equip LLMS with multimodal generative capabilities. By using Llama-3 for text generation and integrating parallel transformer modules for image diffusion, LMFusion efficiently reuses compute of pretrained LLMs. LMFusion's modular design enables independent developments of language and vision modules, de-risking the complexities associated with large-scale, joint-moaality pretraining. In this work, reuse only the pretrained language components, which still requires substantial compute to train the image generation module from scratch. Future work could explore reusing pretrained image generation components well.
'Concurrent to our work. 15 tackles multimodal generation via joint attention mechanism between DiT structure (59) for images and frozen Llama-3 (8) for texts.

---

References
[1] Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang. Liang Zhao, Jian- jian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309.1 11499, 2023.
[2] Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36, 2024.
[3] Xi Victoria Lin, Akshat Shrivastava, Liang Luo, Srinivasan Iyer, Mike Lewis, Gargi Ghosh, Luke Zettlemoyer, and Armen Aghajanyan. Moma: Efficient early-fusion pre-training with mixture of modality- aware experts, 2024.
[4] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 1039, 2024.
[5] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint 24X.v:2405.09888. 2024.
[6] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi- modall tasks. In The Eleventh International Conference on Learning Representations, 2022.
[7] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26439-26455, 2024.
[8] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint 20772 2024.
[9] Yuying Ge, Sijie Zhao, Ziyun Zeng. Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making Ilama see and draw with seed tokenizer. arXiv preprint 100.022310.01218, 2023.
[10] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint 230.:2307.05222.. 2023.
[11] Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: unified foundation model integrating visual understanding and generation. arXiv preprint 2xxv:2409.04422 2024.
[12] Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vasion-language models with sparse mixture of experts. arXiv preprint arXiv:2303.07226 2023.
[13] Junyi Chen, Longteng Guo, Jianxiang Sun, Shuai Shao, Zehuan Yuan, Liang Lin, and Dongyu Zhang. Eve: Efficient vision-language pre-training with masked prediction and modality- moe. ArXiv, abs/2308. 11971, 2023.
[14] Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen tau Yih, Luke Zettlemoyer, and Xi Victoria Lin. Mixture-of-traastee of-.rrrasfoorrrs sparse and scalable architecture for multi-modal foundation models, 2024.
[15] Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Chase Lambert, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving text-to-image alignment with deep-fusion large language models, 2024.
[16] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
10

---

[17] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and cmmputer-assisted intervention-MICCAl 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part 18, pages 234-241. Springer, 2015.
[18] Diederik Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
[19] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[20] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference machine learning, pages 8162-8171. PMLR, 2021.
[21] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint (txXxx22302.137// 2023.
[22] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-4800, 2019.
[23] Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 7432-7439, 2020.
[24] Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP,, pages
2019.
[25] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106,
2021.
[26] Tsung- Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Ziticck. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[27] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning. pages 8748- 11838876..22 PMLR, 2021.
[29] Ramakrishna Vedantam, Lawrence Zitnick, and Devi Parikh. Cider: Consensus-basee image description evaluation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4566-4575, 2015.
[30] Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experr language models. arXiv preprint 2tt::24900000000 2024.
[31] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[32] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lec. Llava-next: Improved reasoning, oer, and world knowledge, January 2024.
[33] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12940 12966,, 1(2):3, 2023.
11

---

[34] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
[35] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
[36] Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 263-2279, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[37] xAI. Realworldqa, January 2024.
[38] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang. Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li. Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang. Jingjing Liu, Yonghua Lin, Tiejun Huang, and Zhongyuan Wang. Emu3: Next-token prediction is all you need, 2024.
[39] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation, 2024.
[40] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation, 2024.
[41] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2024.
[42] Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal under- standing and generation via instruction tuning, 2024.
[43] Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, al. Scaling autoregressive multi-modal models:
Pretraining and instruction tuning. arXiv preprint Ntt::233900259,, 2(3), 2023.
[44] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijic Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 125.8, 2024.
[45] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[46] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[47] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pateenn Recognition, pages 1152311532, 2022.
[48] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 14398-14409, 2024.
12

---

[49] Yuying Ge, Sijie Zhao. Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi guattigraauulritty comprehension and generation. arXiv preprint arXiv:2404. 14396, 2024.
[50] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mixture-of -ffteeesof-experts layer. arXiv preprint arXiv: 1XX1:1701.06638. 2017.
[51] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, Journal of Machine Learning Research, 23(120):1-39, 2022.
[52] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. arXiv preprint arXiv:2006. 16668,, 2020.
[53] Clare Lyle and Razvan Pascanu. Switching between tasks can cause ai to lose the ability to learn. Nature, 633(8076)7745-777, 2024.
[54] Bin Lin, Zhenyu Tang, Yang Ye. Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-Ilava: Mixture of experts for large vision-language models. ArXiv, ab5/2401.159477, 2024.
[55] Wanggui He, Siming Fu, Mushui Liu, Xierui Wang, Wenyi Xiao, Fangxun Shu, Yi Wang, Lei Zhang, Zhelun Yu, Haoyuan Li, Ziwei Huang, Leilei Gan, and Hao Jiang. Mars: Mixture of auto-regressive models for fine-grained text-to-image synthesis, 2024.
[56] Wenhui Wang. Hangbo Bao, Li Dong, and Furu Wei. Vlmo: Unified vision- Lailuae pre- training with cesttur-of-modallyy--eeeeee
ArXiv, abs/2111. 1.02588, 2021.
[57] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv: 10442, 2022.
[58] Wanggui He, Siming Fu, Mushui Liu, Xierui Wang. Wenyi Xiao, Fangxun Shu, Yi Wang, Lei Zhang, Zhelun Yu, Haoyuan Li, al. Mars: Mixture of auto-regressive models for fine-grained text-to-image synthesis. arXiv preprint 240 2024.
[59] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 4195 4205, 2023.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately describe our core contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the Conclusion section ($7)
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
No formal results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The code will be open-sourced and we describe all model experiments in detail.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
Data and code will be open open-sourced with instructions
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.cm
Sublit//uuides/Cddeeubmiii
olicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ l/pipp.cc/public/guiice http://pwwwuutttuuteeeeeo//utaan /CodeSubmeiiiccec ietbommsssiio awiionooliccoo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See hyperparameter in $4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: NA
Guidelines: We show the model performance with multiple checkpoints Figure
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, report computer resources in detail in $4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http::::////ips...ee wwwparips.ccppuuui
Answer: [Yes]
Justification:
The research conforms with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss these in
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Yes, describe safeguards in 4.11
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we describe licenses in $4.1
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer: [Yes]
Justification:
Yes, documentation is provided throughout the text.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
There are no crowdsourcing experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There crowdsourcing experiments research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
19

---

The core method development in this research does not involve LLMs in non-standard components.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non standaand components.
Please refer to our LLM policy hattps://neurips 220 for what should or should be described.
20


Reviews:
[{"review_id": "wQgD4RgkdB", "reviewer": "Reviewer_axpy", "summary": "This paper proposes a framework called LMFusion, which aims to endow pre-trained text-only large language models (LLMs) with multimodal generation capabilities. The framework integrates pre-trained Llama models for language processing while introducing additional transformer modules to achieve visual understanding and generation capabilities. During the training process, the data from each modality is routed to its dedicated modules. Text-specific modules are frozen, and only image-specific modules are trained. Experimental results show that LMFusion outperforms Transfusion across various benchmarks while maintaining the language capabilities of Llama.", "questions": "- Please compare the training overhead of LMFusion and Transfusion to highlight the advantages of LMFusion.", "limitations": "No major limitations and potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n+ This paper is well-written and technically solid.\n+ LMFusion retains the language processing capabilities of pre-trained LLMs by freezing text-specfic modules.\n+ The experimental results show that, compared to naive fine-tuning pre-trained LLMs, LMFusion achieves significant improvements in image understanding and validates the effectiveness of the deep separation design scheme.\n\nWeaknesses:\n\n- LMFusion has twice as many parameters as Transfusion.\n- LMFusion requires training image-specific modules from scratch, which incurs training overhead. It is unclear how much training overhead LMFusion can reduce compared to Transfusion.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your time and constructive feedback! We address each of your comments below:\n\n> LMFusion has twice as many parameters as Transfusion.\n\nWhile LMFusion has twice the total parameters of Transfusion, **it has the same training and inference FLOPs as Transfusion.** Our modality-specific routing ensures that for any given input token, only the corresponding modules are activated: text tokens are processed by the frozen text modules, and image tokens are processed by the trainable image modules.\n\n> LMFusion requires training image-specific modules from scratch, which incurs training overhead. It is unclear how much training overhead LMFusion can reduce compared to Transfusion.\n\n**LMFusion requires substantially less training overhead compared to Transfusion.** As shown in Figure 3, with less than one-third of Transfusion's training FLOPs, LMFusion achieves better performance than Transfusion across most image generation and understanding tasks while maintaining text performance.\n\nWe hope our responses have addressed the reviewer's concerns. Please let us know if you may have any questions."}, {"review_id": "jLeZ7zYLjQ", "reviewer": "Reviewer_FnMX", "summary": "This paper introduces LMFusion, a unified model that extends pretrained text-only large language models (LLMs), specifically Llama-3, with multimodal generative capabilities. LMFusion achieves this by integrating modality-specific modules for vision (e.g., separate FFNs and QKV projections for image inputs) while keeping the original language model frozen to preserve its textual capabilities. The architecture supports cross-modal interactions through shared self-attention layers and uses diffusion-based modules for image generation. Compared to the unified model Transfusion, the approach demonstrates strong gains in image understanding and generation, while maintaining superior language modeling performance. Ablation studies validate the importance of modality separation and freezing.", "questions": "1. Could you share how LLaVAfusion performs on standard image understanding benchmarks (e.g., VQAv2, MMMU) compared to the base LLaVA-Next-8B?\nAdditionally, while not directly claimed in the paper, I'm curious:\nHave you observed any effects—positive or negative—of jointly training for both image generation and image understanding on the model's performance in understanding tasks?\nAny insight or ablation in this direction would help clarify the broader implications of your approach.\n\n2. The paper observes that naive finetuning of dense pretrained LLMs for multimodal generation compromises their language capabilities, which motivates the modality separation approach in LMFusion. However, this might also suggest that the proposed architecture—by strictly separating modalities and freezing the language model—does not fully align image and text representations in a shared semantic space.\nHow do the authors view this trade-off between preserving language ability and achieving deep cross-modal alignment?\nAre there any signs that the current setup limits joint reasoning or compositional understanding across modalities?", "limitations": "The paper does not include an explicit discussion of its limitations. A limitations section would be valuable, particularly to clarify:\n\n1. How the proposed approach compares conceptually and practically to specialist large multimodal models (e.g., Qwen-2.5-VL, GPT4o, Gemini) in terms of image-text understanding and joint reasoning.\n\n2. How LMFusion's diffusion-based image generation compares to specialized diffusion models (e.g., Flux, Stable diffusion 3), both in terms of quality and flexibility.\n\nA discussion on these points would help position LMFusion more clearly in the current landscape and provide insight into the trade-offs between generality, specialization, and efficiency.", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Clarity and Structure:\nThe paper is clearly written and well-organized. The method is easy to follow, and the architecture is explained with sufficient diagrams and high-level intuition.\n\n2. Improved Performance over Transfusion:\nThe method shows consistent improvements over Transfusion on language-only tasks as well as image understanding and image generation metrics.\n\nWeaknesses:\n\n1. Lack of conceptual novelty: The architectural design of separating modality-specific FFNs and QKV projections while sharing attention is highly derivative of existing multimodal systems(e.g. Flamingo[1]). Freezing base LLM is also a common practice in adaptation-based setups\n\n2. Incomplete comparison set: The baseline comparison is limited to Transfusion. It omits stronger or more recent baselines like VILA-U[2], Janus-Pro[3], MUSE-VL[4] and so on. \n\n3. Weak Evaluation Scope: The paper does not report any results of LMFusion on widely-used multimodal benchmarks such as MMMU, VQAv2, or GQA. These are essential for validating real-world image understanding performance, particularly for academic credibility. For image generation, it only reports on MS COCO and lack results on common text-to-image generation benchmark like GenEval. The paper also lacks a systematic discussion about related specialist models like common large multimodal models and diffsuion-based generative models.\n\n4. Weak analysis for LLavaFusion:\nThe extension of LMFusion to LLaVA (LLavaFusion) is only briefly mentioned, with no substantial analysis and qualitative examples.\n\n[1] Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS 2022.\n\n[2] VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation. ICLR 2025.\n\n[3] Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling. Arxiv 2024.\n\n[4] MUSE-VL: Modeling Unified VLM through Semantic Discrete Encoding. Arxiv 2024.", "was_revised": true, "final_justification": "Most of my concerns remain unaddressed after the rebuttal, and the authors’ response has raised additional issues:\n\n1. *Implementation Discrepancy*: The rebuttal highlights a key discrepancy between the implementations of LMFusion and LLavaFusion. Specifically, LMFusion leverages a learned image generation encoder for image understanding, whereas LLavaFusion avoids this due to observed performance degradation. This inconsistency raises concerns about the general effectiveness of the proposed alignment strategy between image generation and pre-trained LLMs in the current framework.\n\n2. *Framework Limitation and Imbalance in Focus*: The paper presents the finding that a deep modality separation is required to prevent image generation from degrading language and image understanding performance. While this is an interesting observation, it also reflects a fundamental limitation of the framework in handling diverse multimodal tasks. Furthermore, as a work on multimodal generation, the paper disproportionately focuses on language and image understanding where their performance is inherited from their base frozen LLM/MLLM. The paper pays relatively little attention and analysis devoted to image generation. From Table 1, image generation performance shows only marginal improvement under the same LLM backbone, and the paper fails to explore the reasons or implications behind this. In the rebuttal, the authors promise to provide more analysis in the revision, but they do not present any during the rebuttal phase, which leaves me unconvinced about this commitment.\n\n3. *Lack of Broader Comparison*: The authors do not compare their method with proprietary unified models such as GPT-4o or Gemini, nor with specialist image generation models like Flux or Stable Diffusion 3. This omission limits the reader’s understanding of how the proposed method fits into the broader landscape of multimodal generation research.\n\nOverall, I find this work to be a less successful attempt at integrating image generation capabilities into pre-trained LLMs. The paper does not offer sufficiently novel insights or contributions to the field, particularly in its current form. While I acknowledge the effort of this work, especially the analysis of modality separation in multimodal generation, I remain skeptical about whether it meets the threshold for a NeurIPS main conference paper. I believe this work would be more appropriately presented in a workshop setting.", "rebuttal": "We thank the reviewer for your time and constructive feedback! We address each of your comments below:\n\n> Lack of conceptual novelty: The architectural design of separating modality-specific FFNs and QKV projections while sharing attention is highly derivative of existing multimodal systems(e.g. Flamingo[1]). Freezing base LLM is also a common practice in adaptation-based setups\n\nWhile we agree that our work builds on established principles like freezing the base LLM, we believe our contribution is distinct from prior methods and offers a novel, systematic analysis. \n\n- **Architectural Distinction from Flamingo:** Models like Flamingo insert new cross-attention layers into a frozen LLM to incorporate final visual features output from a separate vision encoder. Our approach is different. We employ deep fusion, where each LLM layer can attend to every layer of the T2I model, enabling deep interactions between the T2I model and LLM.\n\n- **Systematic Analysis of Modality Separation:** Our core contribution is not just the final architecture, but the systematic investigation into the importance of modality separation when adapting a pretrained LLM for unified, diffusion-based generation. As detailed in our ablation studies (Section 5.1), we explicitly compare three distinct designs: no separation, shallow separation and deep separation.\n\n> Weak Evaluation Scope: The paper does not report any results of LMFusion on widely-used multimodal benchmarks such as MMMU, VQAv2, or GQA.\n\nWe appreciate these suggestions for broader evaluation. In this paper, we focus on demonstrating LMFusion's core capabilities: image generation, preservation of language capabilities, and modality separation, with controlled comparison to Transfusion. While we provide preliminary image understanding results for LMFusion, we demonstrate the framework's broader applicability through LLaVAFusion (Table 2), which achieves highly competitive performance on comprehensive image understanding benchmarks, validating the effectiveness of our modality-specific design. Additionally, our method has been successfully inherited and applied to create real-world deployable models like Bagel [1] when trained on more comprehensive datasets, further demonstrating the practical value of our approach. We believe these results collectively validate both the technical soundness and real-world applicability of the LMFusion framework.\"\n\n\n> Weak analysis for LLavaFusion: The extension of LMFusion to LLaVA (LLavaFusion) is only briefly mentioned, with no substantial analysis and qualitative examples.\n\nDue to the page limit, we didn't include qualitative examples of LlavaFusion and focused on the controlled study of the LMFusion framework. Thanks for the suggestions. We will include them in the next version of the paper.\n\n> Could you share how LLaVAfusion performs on standard image understanding benchmarks (e.g., VQAv2, MMMU) compared to the base LLaVA-Next-8B? Additionally, while not directly claimed in the paper, I'm curious: Have you observed any effects—positive or negative—of jointly training for both image generation and image understanding on the model's performance in understanding tasks? Any insight or ablation in this direction would help clarify the broader implications of your approach.\n\nOur LLaVAFusion model is designed to fully preserve the understanding capabilities of the base LLaVA-Next-8B model. As shown in Table 2, the performance is identical to LLaVA-Next-8B due to our deep modality separation method, a key advantage of our framework.\n\nIn our LMFusion experiments, we find that the vanilla design without modality separation would have image generation and image understanding compete for capacity. With the deep modality separation, with the same FLOPs, the compete problem disappears. We cannot include a link to new figures in the rebuttal, but compared to the existing Figure 4, the figure of deep modality separation under the same setup is significantly better. We will include it in the next version. For our LlavaFusion experiments, we have not yet explored joint training for image generation and understanding tasks. This is a very interesting future direction and we will include it in the revised version of the paper.\n\n> The paper observes that naive finetuning of dense pretrained LLMs for multimodal generation compromises their language capabilities, which motivates the modality separation approach in LMFusion. However, this might also suggest that the proposed architecture—by strictly separating modalities and freezing the language model—does not fully align image and text representations in a shared semantic space. How do the authors view this trade-off between preserving language ability and achieving deep cross-modal alignment? Are there any signs that the current setup limits joint reasoning or compositional understanding across modalities?\n\nOur modality-specific routing ensures that while the parameters remain separate, the representations can still interact meaningfully through the attention mechanisms. The frozen text modules provide a **stable semantic anchor**, while **the trainable image modules learn to align with this established space.** This design preserves the rich linguistic knowledge encoded in the pretrained LLM while enabling effective cross-modal understanding.\n\n> How the proposed approach compares conceptually and practically to specialist large multimodal models (e.g., Qwen-2.5-VL, GPT4o, Gemini) in terms of image-text understanding and joint reasoning.\n\nOur framework is designed to be backbone-agnostic and can incorporate any VLM as its foundation. While we experimented with LLaVA in this paper, the approach can be extended to other open-source VLMs such as Qwen-2.5-VL. This flexibility allows our method to leverage the strongest available multimodal understanding capabilities while adding unified generation functionality. Though one limitation would be that the base multimodal VLM needs to be open-source in our case, making extensions to GPT4o/Gemini difficult.\n\n> How LMFusion's diffusion-based image generation compares to specialized diffusion models (e.g., Flux, Stable diffusion 3), both in terms of quality and flexibility.\n\nLMFusion offers greater flexibility compared to specialized diffusion models like Flux and Stable Diffusion 3, as evidenced by our ability to lead to truly interleaved/omni models like Bagel. However, a direct comparison of image generation quality would be challenging due to differences in training data, a factor that significantly impacts model performance in real-world applications. Specialized diffusion models are typically trained on curated, high-quality image datasets optimized specifically for generation, while our approach prioritizes the integration of generation capabilities within a unified multimodal framework. \n\nWe hope our responses have addressed the reviewer's concerns. Please let us know if you may have any questions.\n\n[1] Deng et al., 2025. Emerging Properties in Unified Multimodal Pretraining. https://arxiv.org/abs/2505.14683"}, {"review_id": "KhU9fiCzeS", "reviewer": "Reviewer_edtZ", "summary": "This paper proposes LMFusion, a method that enables image generation capability for LLMs or VLMs. To address the forgetting problem caused by joint multimodal training, LMFusion designs independent parameters for the image and text modalities, completely decoupling the two modalities. Experimental results show that this approach can introduce image generation capability without compromising language ability.", "questions": "1. **LLaVAFusion**\nIn section 5.2, the author proposed LLaVAFusion, empowering LLaVA with image generation capability. \nBased on my understanding, the authors feed the image tokens used for visual understanding in the instruction into the original model, while the image tokens used for image generation are sent to the newly decoupled attention and FFN layers.\nHave the authors tried feeding the visual understanding tokens into the newly established decoupled attention and FFN layers as well? This would be more consistent with the modality-specific claim made by the authors.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. **Paper writing**\nThe paper is well written, with a clear motivation and methodology. The figures and tables in the experimental section are also presented very clearly.\n\n2. **Reasonable idea.**\nSeparating parameters between different modalities is a very reasonable idea, and it has indeed achieved good results.\n\n### Weakness\n\n1. **Parameter size.**\nIntroducing modality-specific layers significantly increases the model size. According to the paper, the attention and FFN layers are duplicated for each modality, effectively doubling the parameter count and turning an 8B model into a 16B model. This is my major concern in this paper. If adding more modality (like audio, 3D), the number of parameters will continue to grow accordingly.\n\n2. **Unfair comparision**\nAs mentioned above, LMFusion is a 16B model. Therefore, the authors should make a comparison with models of similar size (around 16B).\n\n3. **Speed and Memory**\nIt would be helpful if the authors could provide a detailed comparison of training and inference time, FLOPs, as well as GPU memory consumption, for models that share parameters and separate parameters for different modalities.", "was_revised": true, "final_justification": "The author's response has addressed most of my concerns. I believe LMFusion is a very forward-looking work and should be accepted.", "rebuttal": "We thank the reviewer for your time and constructive feedback! We address each of your comments below:\n\n> Parameter size. Introducing modality-specific layers significantly increases the model size. According to the paper, the attention and FFN layers are duplicated for each modality, effectively doubling the parameter count and turning an 8B model into a 16B model. This is my major concern in this paper. If adding more modality (like audio, 3D), the number of parameters will continue to grow accordingly.\n\nWhile LMFusion has twice the total parameters of Transfusion, the active parameters remain unchanged for any given input, **resulting in the same inference FLOPs as Transfusion.** Our modality-specific routing ensures that for any given input token, only the corresponding modules are activated: text tokens are processed by the frozen text modules, and image tokens are processed by the trainable image modules. We acknowledge that the total model size would grow with additional modalities, but the active model size remains constant.\n\n\n\n\n> Unfair comparision As mentioned above, LMFusion is a 16B model. Therefore, the authors should make a comparison with models of similar size (around 16B). Speed and Memory It would be helpful if the authors could provide a detailed comparison of training and inference time, FLOPs, as well as GPU memory consumption, for models that share parameters and separate parameters for different modalities.\n\nWe respectfully disagree that this constitutes an unfair comparison, as the inference FLOPs of LMfusion are identical to Transfusion while training FLOPs are one-third less than Transfusion, as shown in Figure 3. **Despite having twice the total parameters, LMFusion is computationally equivalent to Transfusion during inference and significantly more efficient during training.**  We thank the reviewer for the suggestion and will include detailed comparisons in the final version of the paper.\n\n\n> LLaVAFusion In section 5.2, the author proposed LLaVAFusion, empowering LLaVA with image generation capability. Based on my understanding, the authors feed the image tokens used for visual understanding in the instruction into the original model, while the image tokens used for image generation are sent to the newly decoupled attention and FFN layers. Have the authors tried feeding the visual understanding tokens into the newly established decoupled attention and FFN layers as well? This would be more consistent with the modality-specific claim made by the authors.\n\nThis is a great point. In the interleaved setup, visual tokens are routed based on their intended use: visual tokens for image generation are processed by the image generation tower, while tokens for visual understanding are processed by the VLM tower. We have not yet explored routing visual understanding tokens through the image generation tower, but this would be interesting future work. We will include a discussion of this approach in the future work section.\n\nWe hope our responses have addressed the reviewer's concerns. Please let us know if you may have any questions."}, {"review_id": "m9SinF33Xe", "reviewer": "Reviewer_LX3F", "summary": "This work introduces LMFusion, a method to enable pre-trained language models to generate and understand images in addition to text. LMFusion achieves this by starting from a pre-trained language model (Llama-3) and extending it with a parallel transformer module that is also initialized from the same Llama-3 model, with each module possessing separate modalities-specific modules. \n\nThe final version of LMFusion:\n* is more compute efficient than comparable methods pretraining multimodal generative models from scratch\n* achieves strong performance in image generation and understanding\n* retains the capabilities of the original LLM used to seed the model\n\nFinally, the same methodology is validated on existing VLMs.", "questions": "1. In the Transfusion paper, it is explained that in the setting with the image coming before text, the noising process on the images is only limited to half of the noise schedule, as fully noising the images (which condition the subsequent text) has a clearly negative effect. Did you do the same operation here?\n2. Is a full copy of Llama completely necessary as image module? Would it be possible to decouple the parameter count for the 2 modalities instead of having the same?\n3. Mostly out of curiosity, I was wondering if you had experimented inference with different kind of multi-modal sequences (e.g. multiple instances of text-image-text-image)", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- This work is clearly written and the presented methodology is both well motivated (re-using available pre-trained weights can severely cut the amount of FLOPs needed) and justified by prior literature in the field\n- The findings are corroborated by a good amount of evaluations for both the text and image domains (including both image generation and understanding)\n- The components of the final model (e.g. the relative learning rates, the type of separation) are also described clearly and their choice is also based on ablations over each individual component\n\nWeaknesses:\n- As a minor comment: while the work is well executed and of high significance, it does not seem particularly original", "was_revised": true, "final_justification": "The authors have responded to points and questions raised by myself and other reviewers. I believe my original assessment is appropriate for this work.", "rebuttal": "We thank the reviewer for your time and very insightful feedback! We answer each of your questions below:\n\n> In the Transfusion paper, it is explained that in the setting with the image coming before text, the noising process on the images is only limited to half of the noise schedule, as fully noising the images (which condition the subsequent text) has a clearly negative effect. Did you do the same operation here?\n\nYes, to ensure a fair and direct comparison, we closely followed the training protocol of Transfusion, which includes using the limited noise schedule when an image serves as a condition for subsequent text generation. Future work may further explore other upper bound values of the noise schedule for images as conditions.\n\n> Is a full copy of Llama completely necessary as an image module? Would it be possible to decouple the parameter count for the 2 modalities instead of having the same?\n\nThis is an excellent point. A full copy is not strictly necessary. Our framework is flexible and could accommodate differently sized image modules. The cross-attention mechanism allows for effective interaction between modules, and a simple linear projection could be added to merge representations if the hidden dimensions were different. We believe this is a promising direction for future work to optimize the parameter-performance trade-off.\n\n> Mostly out of curiosity, I was wondering if you had experimented inference with different kind of multi-modal sequences (e.g. multiple instances of text-image-text-image)\n\nYes, we experimented with multi-modal sequences for image editing tasks. Due to the page limit of the submission, we did not include these results but will incorporate the following paragraph and qualitative examples in the final version of the paper.\n“We fine-tuned LMFusion on a dataset of 8K image editing examples, each consisting of an original input image, a prompt detailing the desired edit, and a resulting image that reflects the specified changes. We then applied the fine-tuned LMFusion to input images and editing prompts from the MagicBrush [1] test set. Qualitative results show that LMFusion performs effectively in these image-editing scenarios, complementing its strong capabilities in text-only, image understanding, and image generation tasks.\"\n\n[1] MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing"}]

Author rebuttal:

Meta-review: The paper presents LMFusion, a framework designed to equip pre-trained text-only large language models (LLMs), such as Llama-3, with multimodal generation capabilities, particularly for image generation and understanding.

Key Contributions:

•	Modular architecture: LMFusion introduces parallel transformer modules for image and text, initialized from the same LLM but trained independently.

•	Decoupled training: Text modules are frozen while image modules are trained, effectively preventing modality interference and language capability degradation.

•	Efficiency and performance: LMFusion is more compute-efficient than training multimodal models from scratch and achieves strong results across benchmarks.

•	Generalizability: The methodology is demonstrateded on both LLMs and existing VLMs.

Reviewers' overall attitudes:

•	most reviewers acknowledged the novelty, effectiveness, and forward-looking nature of the work.

•	The author’s rebuttal was well-received, resolving most concerns.

•	Most Reviewers emphasized the framework’s ability to retain language capabilities while adding image generation, marking it as a significant advancement in multimodal AI.
Decision: Accept (poster)