Jointly Training Large Autoregressive Multimodal Models
Authors: Emanuele Aiello
LILI YU
Yixin Nie
Armen Aghajanyan
Barlas Oguz
Venue: ICLR 2024 poster (2024)
Keywords: Large Multimodal Models; Joint Training; Interleaved Image-Text Generation; Autoregressive Models
Primary area: generative models

Abstract: In recent years, advances in the large-scale pretraining of language and text-to-image models have revolutionized the field of machine learning. Yet, integrating these two modalities into a single, robust model capable of generating seamless multimodal outputs remains a significant challenge. To address this gap, we present the Joint Autoregressive Mixture (JAM) framework, a modular approach that systematically fuses existing text and image generation models. We also introduce a specialized, data-efficient instruction-tuning strategy, tailored for mixed-modal generation tasks. Our final instruct-tuned model demonstrates unparalleled performance in generating high-quality multimodal outputs and represents the first model explicitly designed for this purpose.

Published as a conference paper at ICLR 2024
JOINTLY TRAINING LARGE AUTOREGRESSIVE MULTI- MODAL MODELS
Emanuele Aiello Lili PPliteenicc di Torino, Meta AI
Lili Yu Yixin Nie Armen Aghajanyan ? Barlas Oguz
ABSTRACT
In recent years, advances in the sarg--scale pretraining of language and text-to- image models have revolutionized the field of machine learning. Yet, integrating these two modalities into a single, robust model capable of generating seamless multimodal outputs remains significant challenge. To address this gap, we present the Joint Autoregressive Mixture (JAM) framework, a modular approach that systematically fuses existing text and image generation models. We also introduce a specialized, data-efficient instruction-tuning strategy, tailored for mixed- modal generation tasks. Our final instruct-tune tuned model demonstrates unparalleled performance in generating high-qualit multimodal outputs and represents the first model explicitly designed for this purpose.
INTRODUCTION
, have made remarkable strides in generating highly detailed images, paralleling the achievements of Diffusion Models Nichol et al.] (2022); Ramesh al. (2022); Rombach et (2022). These models bear architectural resemblance to Large Language Models (LLMs), yet their training regimen is tailored for paired image-text data. LLMs on the other hand (Brown et al. 2020; Zhang et al. 2022 Touvron et al. 2023) are limited to text-based output, thus lacking multimodal generative capabilities despite their proficiency in textual tasks. The subfield of Multimodal Large Models has emerged in recent years l'simpoukelli bring together the disparate strengths of vision and language models. Despite important advances in this direction, these models still predominantly generate one modality, thereby constraining their expressiveness. This study aspires to break this limitation by developing a multimodal model capable of generating integrated text and image outputs.
Autoregressive text-to-image models, as exemplified by works such as Yu et al. (2023 2022)
al. (2021): Alayrac et al. (2022); Li et al. (2022a) in the quest
To achieve this objective, we conduct comprehensive empirical investigation into the fusion of two specialized autoregressive, decoder-only, large transformer models, each designed for unique tasks (one for text-to-imanl and a text only model). We introduce a set of methods under the umbrella of the Joint Autoregressive Mixture (JAM) framework. In building this framework, we take advantage of the inherent architectural compatibility of autoregressive text-to-image models with LLMs, allowing us to do deep model fusion and joint training in ways which would otherwise not possible. Our modular and data-efficient solution allows for deep, rapid and effective integration of continually evolving large models, using less than 1 % of the original pretraining data for both parent models.
Our contributions to this study are twofold. First, we establish the feasibility of blending autoregres- sive text-to-image models with LLMs into a unified architecture that retains the core strengths of each while revealing new, emergent capabilities. Second, instruction tuning, utilizing text-based instructions and a custom-curated dataset designed explicitly for image generation. The result is first-of-its-king -ind large multimodal model which can coherently generate long-form content with interleaved text and images.
present innovative strategies for multimodal
2
METHODS
To tackle the challenge of creating unified model that excels vision-language generative tasks, we propose to combine two autoregressive decoder-only architectures. Our primary image-text model is
work done as an intern in Meta AI, correspondence <emamucle.aiello@eee <cmamade.nclll@@oliiooo

---

Published as a conference paper at ICLR 2024
and
Figure 1: Selected sample generated by our instruction-tuned JAM-Cross model. The model can complex mixed-modal outputs with coherent alignment between generated text and images.
CM3leon Yu et al. 2023), trained on 2.4T image-text caption tokens. contrast, using the same architecture, our LLM (Molybog et al., (2023) has been trained 1.4T text tokens. Both models have 7B parameters, we provide additional architectural details in Section [.1] Our overall methodology develops in stages. In the first stage (Sect. (.1), we first combine and align the models. In the second stage (Sect. [.2). we explore new directions for instruction tuning focused on interleaved image-text generation.
2.1 CONTINUED PRETRAINING
We combine the two pretrained models into a singular, cohesive structure in our proposed framework. This composite model is fine-tuned using hybrid dataset comprising both text-only and image-text samples within our continued pretraining phase. The central motivation behind this approach is to seamlessly merge the capabilities of two pretrained models, capitalizing on the unique strengths of each. The training procedure is data-efficient since the original pretrained models are typically trained on trillions of tokens. contrast, our procedure only uses 50 billion tokens, corresponding to 1.3% of the total data used during the pretraining.
2.1.1 MODEL MERGING
The concept of model merging has been previously utilized to combine models that share identical optimization trajectories (Kaddour et al. 2022), models that are trained on identical datasets but have independent optimizations (for instance, Matena & Raffel (2022); Wortsman al. (2022); Ainsworth et al. (2022)). A consistent approach across these studies is to combine models without any training. Our approach diverges from this convention; view the merged model powerful
initialization for subsequent training on mixed-modal data. The weights of the averaged model are defined as:
1 I Panerage Oum + 0img
Where Owm and 0img represent the weights of the LLM and the text-to-image model respectively. In this study, explore weights merging specifically multimodal decoder-only large transformer models, and notably, on an unprecedented scale, involving models trained on trillions of tokens from diverse datasets. In the following sections, refer our average model
2

---

Published as a conference paper at ICLR 2024
Shared Embedding
Linear
Cross Atcention block
Cross Attention block
No
BX
block
block
Shared Enbedding
Figure 2: JAM-Cross, architecture overview. The cross-attention blocks are interleaved between the original LLM block and the Text-Image blocks, and the output embedding between the two branches are concatenated and then projected to the output embedding dimension.
2.1.2 WIDTH CONCATENATION
Our second approach employs the pretrained weights to initialize a wider architecture. Our new model has hidden dimensions jijnint 8192, which is doubled with respect to one of the two original models dum = damg = 4096. We keep the same number layers the original architectures. The resulting architecture has 26B parameters, initialized starting from the pretrained weights of our backbones. The token embedding input/output projections and the learned positional emheddings of the two initial models concatenated the hidden dimension. The attention weights (e.g query projection) W gccommined E R. Rdjooee Xdjoint are initialized as:
W qillem
W g.ilm
W c.combined
(2)
W Wigam
W Wyjamg
Where W glls W g.img di Xdim represent the weights for the query projection of generic attention layer. All the other weights (FFNs and output projections) are initialized following the same logic. We also experiment with slight variations of the approach:
W g, W qiing
W Wg,overage
W combined
W Wqaaverage
Instead of copying the two models' parameters, we use the average to initialize half of the new parameters. We name the resulting model JAM-Width.
2.1.3 CROSS MODEL FUSION
We propose embed cross- coss-attentionn layers between the foundational models facilitate seamless information interchange while preserving the original models' knowledge. Given two decoder-only transformers models Tim and Tamg- we introduce bi-directional cross-attention mechanism that enables the layers of one model to attend to the corresponding layer's output of the other model. This approach allows for a progressive exchange information at different representation levels. For a specific layer let the models produce sequences of hidden states Hum,i for Tum and H Hing. for Timg where these hidden states are outputs from layer l. The output of the cross-attention mechanism (Heross,) from Tsmg Tum for given layer is evaluated as:
Qcross,1
W Wy.HHum K Kerass, = W Wa.HHimg..t
Veross.!
W Wo,.Hing.i-
(4)
Qeross. KT croseil
Heross, Softmax
Verosa.
(5)
Where Wg, WR, W., represent the query, key, and value projection weights of the newly inserted cross-attention layers. A symmetric process is applied for the reverse direction Tim - Timg- We shared input-output projection layer, initializing the weights of the text tokens from the LLM input embedding and the weights of the image tokens from the image-text model. We insert new linear projection layer that takes the concatenation of the two model's output embeddings input.

---

Published as a conference paper at ICLR 2024
Figure 2 illustrates schematic of our model configuration. We refer the model resulting from this approach as AAM-Coss.. Additional architectural details and the underlying design choices can be found in Sect. 3.1
The ablation study for the optimal frequency of inserting new layers is presented
in Sect. 33
2.2 MUL.TIMODAL CONVERSATIONAL INSTRUCT TUNING
Supervised fine-tuning is fundamental tool to leverage the abilities of large pretrained models. Recently instruct tuning has been extended to a multimodal setting (Liu et al. 20233 Dai et al. [023); ; however, all the existing approaches are focused on visual understanding abilities. In this work, we study instruction tuning tailored to interleaved image-text generation.
We collect a small and curated mrixed-modas dataset to teach our JAM model to support textual explanations with coherent images. Since in the first stage, the model has been trained on image-text captions and text-only data; we train on interleaved image-text data during this phase. Our approach inspired by the Superficial Alignment Hypothesis from L.MA (Zhou, 2023), which posits that model's foundational knowledge and skills are entirely learnt during the pretraining. Instruction tuning is then used to guide the model in selecting the appropriate subdistribution of format used when interacting with users. Our results demonstrate that the model can quickly learn the style of images and text from a small curated dataset, suggesting that LIMA hypotesis holds not only for learning the text style but also for images. In experiments, we consider two slightly different instruction tuning settings, we introduce a small portion of the image-text Shutterstock data with retrieval augmentation and we find this approach beneficial preserve the generated image quality when generating with retrieval augmentation. Sect presents a comparison between these two strategies. We train using standard supervised procedure without leveraging any reinforcement learning or human preference strategy. In this instruction-tunn previous methods (Koh al., (223a) that rely only on image-text caption and no instruction tuning, our experimental results confirm the benefits of training with interleaved image-text data.
phase, we leverage interleaved image-text data in contrast to
EXPERIMENTS
3.1 EXPERIMENTAL DETAILS
Tokenizers For images, we use the VQ-VAE tokenizer from Gafni al. (0022). The image resolution is set to 256 x 256, 1024 tokens represent each image, and the vocabulary has a size of 8192. Our text tokenizer is the same that have been used train the parent models, trained over the Zhang et al. (2022) data for text. We introduce the additional <break> token used by CM3leon to identify a modality break.
Image- Text Autoregressive Model We adopt CM3leon as the image-text autoregressive backbone. The model has a standard decoder-only architecture with some peculiarities: no bias terms, dropout, and learnable parameters for layer norms. It has been trained on 2.4T image-text tokens and uses sequence length 4096.
LLM As an LLM backbone, select model with the same architecture as CM3leon, trained in Molybog et al. (2023) this allows us to experiment with a broader range of approaches, such as weight averaging and width concatenation. The model is trained 1.4T text data with 2048 context length, and we further fine-tuned with a 4096 context length using only 30B text tokens.
Objective In all our experiments, we employ the CM3 objective introduced in Aghajanyan et al. (2022); this objective accepts the original sequence as input or transforms it into an infilling instance by masking specific spans and relocating them the end of the document. Then, the model is optimized for minimizing the standard autoregressive loss - log ((unpput). This objective allows for optional bidirectionally and increases the versatility of the model that can be used for both infilling or standard autoregressive generation. We prevent the objective from masking across the modality <break> tokens.
Retrieval Augmentation We employ multimodal retrieval augmentation introduced in Yasunaga ct al. (2022) for our training procedure. The retrieval procedure employs a dense retriever r, a memory bank M and specifically selected retrieval strategy. The retriever takes an input query
4

---

Published as a conference paper at ICLR 2024
Figure 3: Samples generated by our JAM-Cross instruct tuned model. (Left generated without retrieval augmentation; Right generated with retrieval augmentation)
and returns relevance score r(x, m) for each candidate document m E M. Each multimodal document is split between text and images and fed to the corresponding modality-ppecific VIT-B-32 CLIP encoder (Radford et al., 2021). The two embeddings are then averaged to form the documents' vector representation. We then use Maximum Inner Product Search (MIPS) over the memory bank to obtain a list of candidates. When sampling retrieved documents, we prioritize the diversity of the sampled documents by skipping candidates with score r(z, m) 2 0.9. We apply query dropout to regularize the training, dropping 20% of tokens from the input sequence
Training - - Aligmment Phase During the continued pretraining, we train for approximately 50B multimodal tokens. Our initial learning rate is Ir 3 x 10-5 use 500 warm-up steps. We set our optimal batch size to 8M tokens, this hyperparameter is borrowed from the mixed-modal scaling laws introduced in AAghaaanyan al. (023). The total number of training steps is 5960. This training procedure takes approximately one day on 256 80GB A100s for all models. We select the last checkpoint for all the different JAM models, which is always the one with the lowest average validation perplexity (PPL).
Training- Instruct Tuning Our instruct tuning training procedure is data efficient we train with our instruction tuning mixed corpora. The initial learning rate is set to 1 x 10-5, and we use 300 warm-up steps and batch size of 1M. The instruction tuning procedure takes less than hours on 64 80GB A100s, we train for 15 epochs over our mixture of datasets and manually select the best checkpoint corresponding to the 9th epoch. Following Zhou et al. (2023), we notice that the validation PPL doesn't 't correlate with the quality of the responses.
Decoding Strategies We implement a mixed-modal decoding strategy for our interleaved generation. The model starts generating text tokens until a modality <break> token is detected, then image is sampled. The generation process alternating the two modalities continues iteratively until a <eos> token is sampled. As result model is able generate free-form multimodal documents. We employ temperature sampling, a common technique used in autoregressive model (e.g Ramesh et al. (2022) control the randomness of the prediction by modifying the softmax temperature We pair this technique with TopP sampling introduced in Holtzman et al. (2019) consisting of sampling from the top-ranked tokens with cumulative probability exceeding a predefined threshold TP. We also employ classifier-free guidance (CFG (Gafni et al. 2022) ) for sampling images. This technique allows to condition the sampling procedure, blending the logits from an unconditional sample with the logits from a conditional sample. The procedure is mathematically described as
logits,, logits, uncond + at (logits cond logits (ncoma)
(6)
where logits cond = (t,, ((tt,,t. [t,)) and logits uncond = T(t,, < mask >); T represent the transformer model, mask represent the absence of the input text, f.r are the conditional input tokens, ty
5

---

Published as a conference paper at ICLR 2024
the output tokens and de is the scaling factor for CFG. Thanks to the CM3 objective, our training procedure allows our models to sample with CFG without further fine-tuning. Inspired by Yu et al. (2023) we complement this technique to boost the generation quality, Our samples generated using a temperature value r = 1, Tp is set between 0.8 and 1, and we use classifier-free guidance with values 3.5 and 4. In contrast to other approaches, we don't make of the computationally expensive clip-reranking (Ramesh et al. 20211] Yu et al. 2022 Gafni et al. (022) or constrastive
decoding (Li
2022b
eu 2023).
3.1.1 DATASETS
Shutterstock We randomly sample a subset 30B tokens from CM3leon Yuu et al. 2023) pretrain- ing data. The data consists of legally acquired image-caption pairs from Shutterstock, commerciil online platform offering images with ownership attribution and clear licensing terms.
Text corpora We use 30B text tokens sampled from mixture of several publicly available data, and we reuse the data used for training other common open-source LLM following the same preprocessing of (Touvron et al. 2023). The datasets are: English CommonCrawl (Tourron et al., 2023), C4 (Raffel ct al. 2020), Wikipedia, Books3 from ThePile (Gao et al. 2020), and arXiv.
LIMA We use the 1k dataset present in Zhou et al. 2023 which features various curated prompts
and responses.
wikiHow We collect an interleaved image-text dataset sampling 3000 articles from WikiHow, an online wiki publication that usually curates apposite images for each article. We sample balanced articles from each category to ensure diversity; moreover, we leverage the platform's ' community ratings to filter each article's quality, sampling only those with score greater than 90/ /000. For each article, we use the title (c.g.. 'How to make ..?') ) as prompt, we modify the phrase 'This article...' with "The following answer.'. Furthermore, we restrict the number of images per sample, to fit our 4096 context length.
3.2 CONTINUED PRETRAINING RESULTS
In the initial stage of continued pretraining, we evaluate the performance across various JAM models. Our primary objective is to ensure minimal performance degradation post-merging, relative to the parent models. Managing both image and text processing within single model poses significant challenges. This evaluation seeks to quantify the retention of original performance in our different JAM models, benchmarked against the parent models specialized in individual modalities.
3.2.1 TEXT MODALITY
For the text modality, we compare the zero-shot performance on some common sense reasoning tasks: PIQA (Bisk et al., 2020), ARC-Challenge, ARC-Fasy (Clark et al. 2018), StoryCloze (Mostafazadehi ctt al., 2016), Winograd, and Winogrande (Sakaguchi et al. 2021). We also report some recent influential LLM (Brown al. 2020 Touvron et al.) 2023), and our LLM (Molybog et al.) 2023) fine-tuned with 4k context as a reference. Results are presented Table [ The JAM- uniform reaches slightly better text-only performance than JAM- Width however, it crucial remark that this approach consolidates the functionalities of both parent models within a constrained 7B parameter space. Our findings reveal that the intrinsic knowledge of the parent models can be recovered mainly from the parameter average utilizing only a minimal portion of the original pretraining data. The ALMCrross model yields the best results, aligning with our primary LLM. This highlights the strength of our bidirectional cross-attention mechanism against other baselines.
3.2.2 IMAGE-TEXT MODALITY
To assess the performance of our different baselines over the image-text modality, we compare them using the validation perplexity (PPL) on MS-COCO dataset (Lin et al., 2014). We believe this metric robustly correlates with performance subsequent tasks, such image generation and captioning. Furthermore, it provides a reliable reference point for comparing different autoregressive models sharing identical tokenizer. Results reported in Table 2 Diverging from results

---

Published as a conference paper at ICLR 2024
Table 1: Zero Shot Text Comparison on Common Sense Reasoning Tasks
Model
Size
PIQA ARC-C ARC-E StoryCloze Winograd Winogrande
GPT-3
175B
81.0
51.4
68.8
70.1 1
LLaMa
79.8
47.6
72.8
70.1
LLM-4k
7B
76.7
45.9
67.7
79.3
83.9
66.2
JAM-Uniform
7B
62.4
28.5
42.6
63.5
47.8
49.7
1AM-Widte
26B
57.8
31.4
31.6
54.7
50.2
51.9
JAM-Cross
19B
75.4
41.6
67.2
79.8
81.0
66.0
Table 2: Tmage-Text Comparison
Table 4: Ablations JAM-Cross Model
Model
Size
MS-COCO PPL
C-Attn Size Wikipedia MS-COCO PPL
CM3
2.7B
200.1
RA-CM3
2.7B
193.1
13B
7.86
153.2
CM3leon
760M
168.8
26B
7.53
152.4
CM3leon
7B
149.0
2
19B
7.18
149.0
16B
8.55
151.7
7B
177.5
LAM-Widie
26B
159.5
JAM-Cross
19B
147.6
Table 5: Ablations Instruction Tuning
Table 3: Ablations JAM-Width Model
Shutterstock MS-COCO PPL
Init.
Wikipedia PPL MS-COCO PPL
190.2
Copy
7.34 9.0
159.5
164.5
Average
175.4
on the text-only modality, the JAM-Width model exhibits enhanced performance over the JAM- Uniform model in the image-text domain. Specifically, the JAM-Width model demonstrates superior efficacy in retaining image-text performance relative to text-only performance. Conversely, despite trade-off.
decline in performance, the JAM- AAM-Uniform model remains a good parameters-perforeeee Interestingly. our JAM-Cross model not only reaches the best PPL between the JAM strategies but also surpasses our foundational image-text model, CM3leon. We hypothesize that such advancement can be attributed to integrating novel textual capabilities coupled with an augmented parameter count inherent to the combined architecture. Based on empirical evidence, the JAM-Cross emerges as the best strategy to combine two pretrained autoregressive models.
3.2.3 INTERLEAVED GENERATION
Our instruct-tumed JAM-Cross model reaches a high-quality level of image-text generated output. To demonstrate its ability generate coherent modality interleaved responses, show extensive set of generated samples in Figure 3 and Section B The samples generated with retrieval are obtained from the model instruct-tuned with mixture pretraining image-text Shutterstock data along with our corpora of instruct-tuning datasets, while the samples generated without retrieval are obtained from the model instruct tuned only on our instuuct-tuning set. The generated samples show coherent image and text integration, demonstrating unprecedented abilitics at this novel task. Overall we find our retrieval augmented solution to more effective than standard image sampling, boosting image quality. We further report several qualitative comparisons with the most relevant previous work GILL (Koh et al. 2023a) that features mixed-moda generation. model and source generations for the GILL model from the original paper. From this comparison (Figure 4). it's immediate to notice how our model has better overall quality of responses. The generated text is more complete and exhaustive, while the generated images are more relevant to the text context. We remark that our method the first capable of such coherent and interleaved generation with focus on instruction tuning and that our fine-tuning procedure is effective in
use our setrivaal-augment
JAM-Cross
7

---

Published as a conference paper at ICLR 2024
Figure 4: Qualitative comparison with previous interleaved generation models. Compared to GILL, model is able to generate more complete and precise answers. Results for GILL are sourced from Koh et al. (2023a).
efficiently learning the style of the dataset, not only for text but for images. Our model paves the way toward a larger adaption of mixed-modal generation in real-world use cases.
3.3 ABLATION STUDY
We compare the two approaches for the width concatenation model: copying the original models' weight or using the average to initialize the new parameters. Results (Table show that copying the weights is more effective than averaging them retain the original model capabilities. The ablation study for the cross-attention model is presented in Table 4 We ablate the frequency of inserting cross-attention layers and the impact of not using any cross-attention layers. These experiments are performed training with 25B tokens, all the other parameters are the same as reported in Sect. [..1 We remark that this an even shorter training setting concerning our 50B tokens total training and that the difference in performance increases the training progresses. We further ablate the contribution of image-text pretraining data in the instruction tuning procedure Table 5 The results indicate the importance of using pretraining data mixed in the instruction tuning procedure to preserve the MS-COCO PPL. We do not report WikiHow PPL since analyzing the models shows that it doesn't t correlate with generation quality similarly to Zhou al. (2023).
4 RELATED WORKS
Generative Text-to- Image Models The field of generative text-to-image models has recently been
dominated by diffusion models (Sohl-Dickstein et al. 2015 Ho al. 2020). Recent enhancements
have used pretrained text representations (Ramesh et al. [2022] Nichol et al. 2022) like CLIP (Radford
al.] (2021) improve the generation quality. Concurrently developing diffusion-basse generative models, significant steps have been made by autoregressive token models (Esser et al. 2021 Gafni
al. 2022). These models encode images into discrete latent space (Van Den Oord et al.
2017
8

---

Published as a conference paper at ICLR 2024
and can be processed as a standard requence-to-sqqueee modeling task, enabling the borrowing of techniques used from Large Language Models. A critical element that has been found beneficial in boosting text-to-image generative models is retrieval augmentation (Chen et al. 2022 Yasunaga et al. 2022). Yasunaga et al. 2022) propose to prefix decoder-only models, such as Aghajanyan
2022), with retrieved images during training, resulting in a huge efficiency gain for the training procedure. Yu et al. (2023), scale this strategy to reach state-of-art performance in image
generation using 5x less training compute. In this work, we borrow their model as our text-to-image autoregressive backbone.
Multimodal Language Models The multimodal language model field has recently seen consid- erable development. Several prior works have focused on connecting language models to visual encoders. (Tsimpoukelli et al.] [021]] Mokady et al. [022]] Najdenkoska et al.) 2023] Li et al. (023)) These methods typically train mapping network between pretrained image encoder and language model. Flamingo (Alayrac et al. 2022) introduces cross attention into a frozen LLM to inject visual features and trains a large corpus of image-text pairs. In this work, we similarly use cross attention to bridge the two models; however, our mechanism is bidirectional between the vision and language models, while for Flamingo, the visual knowledge is injected in the language model and not vere- CM3 (Aghajanyan et al. 2022) is trained on a large corpus of structured HTML; it introduces the Casually Masked Language Modeling objective we adopt to train our models. [Koh et al.] (2023b) propose a multimodal language model capable of processing arbitrarily interleaved image and text inputs and generating interleaved output of text and retrieved image. Subsequently, on the same line of work, GILL|Koh et al. (2023a) proposes to ground an LLM to text-to-image model, using a mapping network and freezing the pretrained models, introducing the possibility of generating retrieving images output. Similarly to GILL, (Sun al. (2023b) propose to model different modalities in an autoregressive way with a single model they call Emu. Differently from our work, they employ EVA-CLIP (Sun et al. 2023a) encoder generate visual embeddings and Stable Diffusion RRobbach et al. 2022) conditioned on the generated image tokens to decode images.
Instruction Tuning Instruction tuning aims to teach language models to follow natural language instructions. Several methods have been proposed for instruction tuning, using existing NLP datasets converted in instruction formats Wei et al. (2021) |) Chung et al. (2022), or using LLMs like GPT-4 to generate instruction data with better diversity Wang et al. (2022) Honovich et al. 2022 Recently, LIMA Zhou et al., (2023) demonstrated that 1,000 carefully curated samples are enough to reach competitive results compared to bigger instruction-tuninn -uttinnnntttnng datasets. The authors hypothesize that most of the knowledge is learned during the pretraining, and the instruction tuning teaches the style to interact with the users. In this work, we explore using a small set of multimodal instruction tuning data to fine-tune our model, verifying the effectiveness of a small dataset this multimodal setting tailored to image generation. Several vision language works adopt instruction tuning for multimodal tasks-focused user interactions optimized for visual content understanding Liu et al. (2023) Dai et al.
(2023) Ye et al. (2023) Zhu al. (2023). Unlike previous works, we explore instruction tuning focused mixed-mode modal generation, paving the way for more significant adaptation of multimodal models that can generate interleaved image-text output.
5 CONCLUSIONS
In this work, we have presented novel methodologies for combining pretrained autoregressive models, demonstrating the viability of synthesizing the knowledge of two distinct models into a cohesive structure with extended capabilities. Our exploration validates that the integrated model can be adeptly fine-tuned using our tailored instruction-tumii this end, we pioneered creating specialized dataset centered on instruction tuning for this particular task. Nevertheless, the proposed study is limited to 7B parameter models with the same architecture. Future works may consider scaling the models' size and asymmetrically applying our cross-fusion method to bridge models of varying sizes. Increasing the context length and delving into multi-turn conversations could further represent an interesting exploration direction. In conclusion, our study sets the foundation for substantial advancements in the realm of multimodal autoregressive models. The fusion of text-to-image im-i generation with large language models paves the way for sophisticated systems capable of interleaved image-text interactions, enriching the landscape of conversational AI.
procedure for interleaved image-text generation. To

---

Published as a conference paper at ICLR 2024
REFERENCES
Armen Aghajanyan, Bernie Huang. Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. Cm3: causal masked multimodal model of the internet. arXiv preprint arXiv:220.0075200 2022.
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for generative mixed-modal language models. arXiv preprint 2301.033,, 2023.
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2022.
Jean-Bapa Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. Proceedings of the AAAI conference on artificial intelligence. volume 34, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval-augmented text-to-image generator. arXiv preprint arXiv:2209. 144911, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction pntructioo--inetuneed language models. arXiv preprint arXiv:2210.1 1416, 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.05553 2018.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpont vision-language models with instruction tuning. arXiv preprint arXiv:2305. 06500, 2023.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. In European Conference on Computer Vision, pp. 89-106. Springer, 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-663 -8511, 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference Learning Representations, 2019.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:22/2.09668 2022.
10

---

Published as a conference paper at ICLR 2024
Jean Kaddour, Linging Liu, Ricardo Silva, and Matt J Kusner. Questions for flat-minima optimization of modern neural networks. arXiv preprint arXiv:2202.00661, 2, 2022.
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. arXiv preprint arXiv:2305. 2023a.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal generation. arXiv preprint arXivv2301.13822, 2023b.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-imane image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888-12900. PMLR, 2022a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
Bootstrapping language-image pre-
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint 20 2022b.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, arXiv preprint arXiv:2304.0844 2023.
Michael S Matena and Colin A Raffel. Merging models with weighte averaging. Advances in Neural Information Processing Systems, 35:17033-17716, 2022.
Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning, arXiv preprint XXX:::211100975555 2021.
Igor Molybog, Peter Albert, Moya Chen, Zachary DeVito, David Esiobu, Naman Goyal, Punit Singh Koura, Sharan Narang, Andrew Poulton, Ruan Silva, et al. A theory adam instability in large-scale machine learning. arXiv preprint arXiv:2304.09871 2023.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Van- derwende, Pushmeet Kohli, and James Allen. corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv:1604.01669 2016.
Ivona Najdenkoska, Xiantong Zhen, and Marcel Worring. Meta learning to bridge vision and language models for multimodal few-shot learning. arXiv preprint 2rXhv:2302.14794 2023.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob
Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text- text-guided diffusion models. In International Conference on Machine Learning, pp. 16784-16804, PMLR, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555) 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8833 PMLR, 2021.
11

---

Published as a conference paper at ICLR 2024
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv e-prints, pp. arXiv-2204, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106,
2021.
Jascha Sohl- Sohl-Diccstten, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. pp. 2256 2265. PMLR, 2015.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprint 2r3.v:2303.15389 2023a.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint ar Xrv: 231::2307.00222 05220,22222222222 2023b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothéc Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Maria Tsimpoukelli, Jacob L. Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-212, 2021.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint 12. 10560, 2022.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2021.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes Ari Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23995823999. PMLR, 2022.
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Rich James, Jure Leskovee, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. Retrieval-au augmented multimodal language modeling. arXiv preprint arXiv:2211.12566 2022.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. /4/78, 2023.
Jiahui Yu, Yuanzhong Xu. Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. Transactions on Machine Learning Research, 2022.
Lili Yu, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. Scaling autoregressive multi- models: Pre-training and instruction tuning. arXiv preprint 1xXi:2201.007200 2023.
12

---

Published as a conference paper at ICLR 2024
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- trained transformer language models. arXiv preprint PrXiv:2205.01066 2022.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less more for alignment. arXiv preprint arXiv:2305. 12206, 2023,
Deyao Zhu, Jun Chen, Xiaogian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint 1055: 2023.
A LIMITATIONS
The generation quality our proposed model presents some limitations. Our JAM frameworks rely on LLMs and text-to-image autoregressive models, inheriting their strengths and limitations, such as the potential for hallucinations and biases in image generation. These limitations may be addressed by improving and leveraging better autoregressive backbones.
Moreover, our instruct-tuning procedure focuses on a specific wiki-style single-tum question answer- ing. Most of the time, the model generates single or, most, two images, while longstanding goal could be to generate entire multimodal documents with >2 images. This is partially bottlenecked by the training data and the context length of 4k tokens; given that each image takes 1024 tokens, generate longer outputs it's still challenging.
Furthermore, our model is more effective in generating interleaved image text when the question style aligns with the dataset style and when the images refer to concrete objects. While the model usually struggles to represent more abstract concepts. We showcase some examples of this behavior in Figure [ W When the subject of the text dialogue are more abstract concepts such as "time travel", "music", ", or "research", the generated images fail to capture the textual context and enrich the model output faithfully.
B GENERATED EXAMPLES
This section shows several samples generated by our instruction-tune models using both retrieval augmentation and standard image generation.
13

---

Published as a conference paper at ICLR 2024
coserrot who mented where you
nas Hesnspen are pezzis must and hove
would
they aire and one the
dio east my growier and hew they charged oounte lowr benef the cearse of ettery for befter locs what end people whot would obo travel ancent times try to make
I sould also go to fatu to ene hoa sicley nat change and weat aidd changes has andergone wouil slso loe what kind rmpactt stvencos socitt..
techneleiion people hove
has plyyi patar bacane beker
ungg snd fecornn femder work bow entegu posto set Portie por then Whe en of lere nigk Yamole Segle
ofvered
part ond carbleeed
proghte
extrynn
eigiann Ta ha yoscc canoiaal tit cricts picctic and
mack Auyoo
heing
dram bette work
buainit
sang mend prgrass
dieet iniuety Thes plary naturaa putee slovngg
ea practias dural & le betuear mosi of feurrt doee kit the praciss plail Faely, out molk nan urendd
reasorch pater soddemic cres that recorcher celuomaa "aak ancrom mas to forsuct antiee comert Ubatly thaae myposort the nai Mosof
Wredect
Gie
reterr hae tarhag princcoo coral las stuncite our the pat weur moir ane to dight numero wal voal
paper
grrccs and apur
Desente
sourne
momoit ayd menalcsee tn chat will mes recsand
Figure 5: Generated samples Failure Cases
14

---

Published as a conference paper at ICLR 2024
neammer underteed
bagartin part hap long ths
cecces or deigs
eary tap Tuslogiing fil hreenin perer bariie oel fuch callory ouly
werld you
foree
sasee of tet ser senoied odly
mose perigr the elae
saur pecpres of yndr dueersincs ce peaioee Iollewee
your cudily
huve
Figure 6: Generated samples without retrieval (left), retrieval augmentation (right).
15

---

Published as a conference paper at ICLR 2024
trii my drg can Teach and other bask inttractions
for commands and "shay" cartis sheps
deg. are shape
amacite
To get yoar dlg to an vase your hord the or and make devand rg notion with your hand. Ofhen calied fhords command" your dag a asppoatte ortampne, your das ekoui undiretont nur the sommand onr corressnnde hand eahtanyou yrie
in whetther mation be caneer
your
renfancumant
but your tame
easoaaded
shart wit penti resc and ynakelly ra se your leee vouue when he falkes the command, are happy with hu pert formarcce oee tell "Good enly potriiu..
Beginm ni commands with the hard nation el hel you to cahoblot the movement os command
Hoe cn sprive schencticl frow ledg??
There severdl thirg' you coe your thmestted ieeeidden on dae sugentiind book sortenstical concapta than
Enirodactin dgettr
Eirrodaciion Geueatry
Some
Arthhetic Bos
Hyportale
Scliing Equat ane
You demees woth
unierrttio other dlydre and calcaia
And futor wfliee
saling arcrd
0f1
0=000
2
Figure 7: Generated samples retrieval augmentation
16

---

Published as a conference paper at ICLR 2024
prrpert
Thene ore thing are whan prepmin offept ade tates ore fex augeshore
Diey
cheare the right fege ard anderttad whoh you need to oo get the
encreje just
poreets
Chasse majer
intorttee that
find that like.
oien oegiotaiin and wir feams Theie you le fog Neww other people have fov They in oito
Work hed
grarns gymun war of 3.0 en bethee-s soally necedd
silk
repare in glar as then when
cur 3 grode cverae age heachera coutudirs
Take wass doa
your iniens ond fommly was aayy riee how selous
deg dtuited by chusd hopping that eato your
the
loeue lell dky mare aurriced opt oo lut dayt are louttie that chare beught serciced down they dan't ther onneec el the iyy rdlento you nend u wewe sll hut day wen alto moke
modlinns The
Lsing i nomonade turker sauraee
duduriood fasied
daga ane enolabee in buklet or non grocery shores and they on subethccc for nert nom reopes. furchene
doyt
mest livers son boak your vagr hot dhp cell bus mathicnn adtan, or hespet.
Chasss day musccingg mik a cuick Seaze sing for dag of tealmals fast ond sqanatie thes boyt tig.. arocary
esercing
Taste your dogs for
with tabkapee eb
up prupheeer orge-rr oponn pepeer, seciering mny
wenring dayt
swnun qut rume.
Houe T stat srwhhard
Snweboard to lonat tere are ateae he you mombaard
den sp chacl 6ift to to the the maunian You nead to bay di cranioer Fi rocker
the
muntte you arrivet Al lowe wideed Smach Tetter de the
you oreand ge unty
dali to bave to rike ther 1VT
the
3. Now that you'ne oi the the mournin you ride aranocers dan sourss oo sowitill yoursed to sepp your when you nent to. Tr daster you om park ate the add de the wnwboord wer to fors you fume popr some key that
fact
Figure 8: Generated samples - without retrieval augmentation
17

---

Published as a conference paper at ICLR 2024
How from Bomine's
the soianon Doruno "Large Pizz" men frem that and this endence online.
Click the "Pizaf" mone and select "Lorge Pizze?
Enter the desined quintiyy the "Quann Hy box, For coumple. 2.
Chack "ind PPecage"
Provict your rome, number
popment
7. Confirm the ender not for the poss be delvored.
How can weed table kitcher?
panda (thi probuble the comman style arobably shoppot)
ruble
Bulld ruble with hile
why but there sople that live
Bulld a pool sacke juot realy letchha vabke but can und the ancher) buffer (thess one unally brus wood and outs henvy Goad you have çaarts)
Ramamber, when bailding any 1ab importort part frome. loge one just gaing held the frams, choose your legt boned
propere an coccads tourt?
reletwely y eayy buttee avecado that thidoer moy'' cropy, atzap chappel bread mase wrt whnt or tyo groi. wort to vayan you use weget butter rocess instard
evasado with
loper
get
Hest pan the steve until In's har. Placc
with touater the beitom yeu wert to teart the broad in over or will The frssh oa on ddel place to pat the eathed butme to map from enoking. ariopy
medium
Figure 9: Generated samples - without retrieval augmentation
18

---

Published as a conference paper at ICLR 2024
skulitula is raaty and sittin breadden zld medis with aggt beand eadinoess cnlan, lerosx end tanats duse The di slas oo cranbll in before porr
Mako ladi diaidale
secnuct plmach)))
hesh
dio nevurich
urie acuspes.
perti andiar ma cnd then 5-10 miunns dos sding your harts Oriani mem commen atperimnnt
other
cortader.
Oruh tanatoo thes
pikcos
oon goin witt my Niki preppre ny bockpock)
Hore one some ips to get sterted presaring your bookkook for trek.
Poot yoar emantide hhhhiggh compact
Pocks
main
Use innetholle. systwight motemaa
begs
Oot for remonable atraas, besa or other soomaer ka as you can ligtten your load while hiking.
Pocce yoar datkee is torp callpehia bag they mankto space.
idjuet connving oppcrity socording
Ube aaterproo bag taa held everyhinn
woturppee bog calaskeedd trik.
10. Shart yoar trac wht frash cother dan't T eenory or dimy whil tiklin.
Figure 10: Generated samples without retrieval augmentation
19

---

Published as a conference paper at ICLR 2024
What I with Tarte?
euggeshinn wit Thil::
Soveyt impressive skiin artiouttee,
castury.
The Playants, located the
Thiy
Chunch of fesil ousipned fo kauh compicted 1988 as Gathicutte church lestral the church ditric the
Paccs bun Nurcs er a1. Mark's Square, si besutiful spure lochtd in the heam of the city, foong the Bas
The Torie bome condtnucted 1878, eatres sary exompee of Thalie anchshecure, creened archterts sajletea Actille Vill.
lorger becutifu
Thiy
Gandon. bested
Sur Frunc soo a fun city vest with et of lque ettractera to wanh Some of the mext pepoker ottract ene include the Boldee date bridge, the asent cruine Buy Area
Sijero, inclucing Smsthaekle Mussun
luearriity cit.
nfu view of the diy. Then die lot of differett exseum Hanaryy the Asas am Museun, Getty Muxumm You cas sls vieit populer ructorre
City,
througe Separe repan-ace city's
Square buey chepping
The fastunar
sofia reprossnt the culture siveritty
the city. Thane concarte hald the Leton Squars Pizzo The notlek,, Pridayy, and Seturdaye dur the cummer you disa etterd
Hon con 1 lorm skandborrinng?
Skatsboardig fun, invohes lot movement good exercies Thene waye ikam dicheboord You hale kassens from ol sketer, worch inttructinal videss necd kowilt criicess etc. Try findl fried that shctes and isks ride with you or visit Leom aron pro sicher
bumpy
Find profossiondl skater to teach you Look online for skoter camgi try caling smound your lool skatoperkk You night even know shoter the ii to teach you how to sont, Ongorize ta lerson with the penzee and make oun you know what you dcsech to get out Some
dehers may lilinn he you for free
you if you hata there domaminng coomple.
Organize lesson
yeu knew evectly you froe
tarm you befonce
probubly alse
skandoorrd
Figure 11: Generated samples retrieval augmentation
20

---

Published as a conference paper at ICLR 2024
Heguu helks owe and yop tee tes the the word ined to teo mnser Oie lenee dewecee e leuah way secgla figet bearc welert Gongle i to Nus ca from
achietia
With
troel
lejet you
igh weet budget
Hon to make nange los??
Mergn lew ki link the oa and sugent yeu peeile hhoee for surver day soll fome epyyd plan prrsoealiion
ful
Repare the requi
for ziaeunal
Mia the sogur nanae Joia Masc apoi reparr toyther unil masura muutt You Haad eho sorae soun doas
4. four the oborane Ints pas You any
How yet owa'? wait lie
foyear thes Pooteer
hertee Hoaoadd then better proteet ply ches
Andoorve gene
ren
effertyy
stjecvve
netio hortyeets sereene Beecty you frmm mutiie
voring mtretggiss
Hhe cherboard cary sputige The higin poie expande wheter you Tris Aerter Mosents
deettyy
apanent "h over Angioaes Sollio Zacaty Lawn abat mrorgia
cunnig pourhall
Figure 12: Generated samples retrieval augmentation
21

---

Published as a conference paper at ICLR 2024
Bacturiin it pusder doarts world more he sockrtted now was arlars Thene and iu start plevin baok laker
shot wine baterboll
mertroo
alcona chaaetiall and hath cama sloing Bakertal arear oae suort clls toan quot, lunning how wark as
will
hookertel
laetu bithus for the lit
be oa to
badertall nurted
sow yowerd leckcoun many Hore repenitiin coacher
barurbal bel frogccs DVPl WiP one popular
Hov boadering
Pashing the hand occemsan won problen for seginera sera bea de Here are fee thangs lees wi ed you dendap year techeigee
just gerting your foct the rick, or pathing orig cits part maler kald. Thask mail stretegie alll help you be streagh end coriiaacca before bogeer meres,
through
probably
just sprking an anglc ano al tim, get good on that sefenc rg for example right stom be workin an coudde averhard anp and only "iiee year tine You wally cen't loom opar receiigee out in cagle ouidin You sec to seriica cand etartly in ancen deeelop dkills det odds 30 minunca or preshic evey make same you omealo cocasora need multile
myrack noch que??
There are four tardamerhi that noud get better of behere start getting petter at dlebk pertaaao ander ussitaal routin of cifferee caro cou they read clibbing
hase
Thare
erarting veall good cone peried tim som casd Nuctr bellade at vau cana sctrory with your Long your shale body we whan you mnildng
preferably mermal
morovemerts
meady
pudiing hand
soand
actudy sllig bedy
suldy.
Figure 13: Generated samples retrieval augmentation
22


Reviews:
[{"review_id": "yUyzuJET6R", "reviewer": "Reviewer_qfqe", "summary": "JAM, novel methodologies for combining pretrained autoregressive image2text and text2image models, demonstrates remarkable new abilitys to generate interleaved image-text sequence.", "questions": "My rating is 4. But as only 3 and 5 are available, I choose 5 for the novelty in combining two separate models to empower better multimodal capability. But this work have too many deficiencies too.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This work proposes a novel approach to bridge two separate image2text and text2image models into a new unified model that can generate both image and text.\n\nThe resulting model demonstrates remarkable ability on generating interleaved image-text sequence under adequate qualitative evaluation compared with GILL.", "weaknesses": "**Method is restricted.**\n1. The method is not general and nearly impossible for the community to follow. Because your method requires two identical image2text and text2image models, but nearly all available image-to-text and text-to-image models are of different architectures. \n2. Thus, the only way for the community to test your method's effectiveness is to first pretraining two **identical** separate models. I think this is rather inefficient and prohibitive.\n\n**Experiments seem too casual.**\n1. Your paper focuses on multimodal models that generate both images and texts, but how can the performance of pure language tasks be your main table and experiments? **This table even becomes the most adequate part of your experiments**. This is not aligned with your motivation and focus.\n2. Experiments about multimodal ability are too casual and not taken seriously. Your all multimodal experiments, except for the qualitative cases, are Table 2-5 on Page 7. Among them, three are ablation and Table 2 is your main table for multimodal performance. However, Table 2 uses MS-COCO as dataset and PPL as metric, which rarely serves as a **main** evaluation of multimodal ability. The baselines also lack a lot. The experiment quantity and quality is too bad to be a ICLR submission.\n\n**Inadequate literature review.** \n\nYour focus and motivation is to develop a multimodal model able to generate both image and text outputs. The field has witnessed an emergence of such kind of models. To name a few, Emu [1], SEED-LLaMA [2,3] and DreamLLM [3], etc. Among them, SEED-LLaMA [2,3] and DreamLLM [3] are recent work and can be arguably ignored, but Emu [1] is released even earlier than the blog of CM3Leon, the work you heavily follow. However, your literature review and discussion totally ignore such multimodal unified modeling work that generate both image and text, which I think are **the most closely related to your work** and should not be ignored.\n\n[1] Generative pretraining in multimodality\n\n[2] Planting a seed of vision in large language model\n\n[3] Making LLaMA SEE and Draw with SEED Tokenizer\n\n[3] Dreamllm: Synergistic multimodal comprehension and creation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YoPvgKL1sZ", "reviewer": "Reviewer_jvv1", "summary": "The authors work on combining two pre-trained autoregressive models to form a new decoder armed with multimodal generation capability. More specifically, the authors combined text-to-image decoder and text decoder, and formed a decoder that can seamlessly generate text+image.", "questions": "Did authors compare their decoding strategy with contrastive decoding and clip-reranking, in terms of both efficiency and performance?\n\nCan users control alpha_c, tao and other critical hyper-parameters during decoding? So the tool can be more balanced in generating text+image, or more leaning towards text or image generation.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "To the best of my knowledge, this work is, if not the earliest one, among the pioneering works that explore fusing two decoder-only models of different modality into one, to enable seamless image+text generation. Previous works have considered merging the token spaces from different modalities, and use single decoder to enable generating both modalities (like AudioPaLM); However, the idea of fusing two decoders, and arm the new decoder with the capability of generating high-quality multimodal output is new. \n\nAs a pioneering exploration, the authors explored three different approaches to fuse two decoders, and all the three methods all seem to be successful. The authors also conduct ablation studies to better understand the three approaches they have proposed. Empirically, the authors show the feasibility of fusing two decoders from multiple modalities.\n\nThe authors collect a small and curated mixed-modal dataset for the purpose of multimodal conversational instruction tuning, and also confirmed that training with interleaved image-text data is beneficial.", "weaknesses": "On Reasoning/Understanding: \n\nOne weak point is that JAM models (both JAM-Uniform, Width or Cross) are all much weaker compared to LLaMA and GPT-3, while LLaMA is even smaller in terms of model size compared to JAM-Width and JAM-Cross. This is acceptable as this work is not focusing on reasoning and understanding.\n\nOn Scaling up: \n\nCompared to GPT-3, the JAM model is still pretty small. Scaling up the model size could potentially help with the performance in terms of common sense reasoning and also generation. \n\nSingle image generation:\n\n Interleaved generation is one innovation of this work. By reading the decoding strategies, it seems that one single image will be sampled if <break> token is detected, and then the model will continue text generation unless the <eos> token is sampled. By looking at all the samples provided, it seems that this is the case.One possibility is that there could be multiple text requests to generate multiple images at once.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QEdPg3UPFU", "reviewer": "Reviewer_Ghzd", "summary": "In this work, the authors study three ways in which a pre-trained text-only decoder LLM $\\mathcal{T}\\_{llm}$ and a text-image LLM (trained on VQ-VAE tokens) $\\mathcal{T}\\_{img}$ with the same architecture can be merged as a multi-modal LLM capable of interleaved text-image generation called Joint Autoregressive Mixture (JAM). The three ways of combining the two models are: 1. **Model Merging** in which the weights $\\theta\\_{llm}$ of $\\mathcal{T}\\_{llm}$ and the weights $\\theta\\_{img}$ of $\\mathcal{T}\\_{img}$ are averaged as initialization weights $\\theta\\_{avg} = 1/2 \\theta\\_{llm} + 1/2 \\theta\\_{img}$ for a model of the same parameter size (7B in the paper) (JAM-Uniform) 2. **Width Concatenation** in which the weight matrices are concatenated to form matrices of double size (per matrix dimension), both copying and averaging (JAM-Width) 3. **Cross Model Fusion** in which both  $\\mathcal{T}\\_{llm}$ and $\\mathcal{T}\\_{img}$ are augmented with cross-attention layers that permit a bi-directional flow of information, generalizing the Flamingo model (JAM-Cross). In each way, the authors train the resulting JAM model using the CM3 loss and perform instruction tuning using a small curated dataset of examples (following the LIMA paper).", "questions": "- Shouldnt we have $\\mathbf{H}\\_{llm,l}$ and $\\mathbf{H}\\_{img,l}$ in Eq. 4 instead of $\\mathbf{H}\\_{llm,l}$ and $\\mathbf{H}\\_{llm,l}$?\n- \"We prevent the objective from masking across the modality `<break>` tokens.\": Does the CM3 objective relocate masks at the end of each modality or at the end of the total document?\n- What is $q$ in “The retriever takes an input query x and returns a relevance score $r(q, m)$”, the authors intended the $x$ variable?\n- “We leverage our text-to-image backbone’s modifications introduced in Yu et al. (2023).” What does this phrase mean in the paragraph?\n- “The two embeddings are then averaged to form the documents’ vector representation”: if a document contains multiple text and images, shouldn't it have more than two embeddings?\n- \"We prioritize the diversity of the sampled documents by skipping candidates with a score r(q, m) ≤ 0.9.”. Reading the original paper, the candidates with a score higher than 0.9 are dropped, not the opposite as here.\n- “we is data efficient we train”, bad phrasing.\n- In classifier-free guidance, $t_x$ refers to the tokens generated before the image token to be sampled?\n- I don't understand if the final text-to", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of bi-directional cross-attention layers between two generative backbones is an elegant approach for performing model merging. Indeed, the authors obtained SOTA results with JAM-Cross on MS-COCO (147.6 PPL), also showcasing the positive influence of the pre-trained joint text decoder.\n- Comparing the three ways of performing model merging while demonstrating the superiority of JAM-Cross between the three sheds light on what is the best way of performing such an operation.", "weaknesses": "- While the experimental section is fair, pointing to a new state-of-the-art over CM3leon, the improvement is only 1.4 PPL points with an increase of the model size of more than double (19B vs 7B of CM3leon) and increased training time. The authors suggest that minimal performance degradation post-merging should be studied in the evaluation part. While I agree minimal degradation is a necessary condition, it is not sufficient: given that the final model performs the same tasks as CM3leon, improving only 1.4 PPL points does not justify the necessity of jointly training with a text LLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fsmHSFxRbh", "reviewer": "Reviewer_eaYf", "summary": "The paper introduces the Joint Autoregressive Mixture (JAM) framework, aimed at integrating large-scale pre-trained text and image models to generate cohesive multimodal outputs. Leveraging the architectural compatibility of autoregressive text-to-image models with Large Language Models (LLMs), the authors propose a systematic approach to model fusion and joint training. The JAM framework exhibits superior performance in generating high-quality multimodal outputs, integrating text and images seamlessly, and represents a significant step towards advanced multimodal conversational systems.", "questions": "- Why does LLaMA have blanks in Table 1?\n- Is Table 1 suggesting that 7B LlaMa is better than all the proposed JAM models in reasoning?\n- Can we have the baseline results in Table 2 for the Wikipedia dataset as well?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "- The authors have proposed novel methods for combining pretrained AR models.\n- This represents a great contribution towards building multi-modal conversation agents.\n- The specialized instruction-tuning strategy demonstrates efficiency and effectiveness.\n- The results shown are noticeably better.", "weaknesses": "- PPL is not defined anywhere in the draft.\n- Retrieval augmented presentation, the last section of page 4, can be improved. a small schema can be helpful. \n- Instruction tuning lacks enough details. For example, in the introduction, the efficiency for even 1% of original pretraining data is stated but not mentioned anywhere else in the draft.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)