DreamLLM: Synergistic Multimodal Comprehension and Creation
Authors: Runpei Dong
Chunrui Han
Yuang Peng
Zekun Qi
Zheng Ge
Jinrong Yang
Liang Zhao
Jianjian Sun
Hongyu Zhou
Haoran Wei
Xiangwen Kong
Xiangyu Zhang
Kaisheng Ma
Li Yi
Venue: ICLR 2024 spotlight (2024)
Keywords: Multimodal Large Language Models
Large Language Models
Generative Models
Vision Language
Representation Learning
GPT
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: DreamLLM is a family of versatile Multimodal Large Language Models (MLLMs) empowered with the learning synergy between multimodal comprehension and creation.

Abstract: This paper presents DreamLLM, a learning framework that first achieves versatile Multimodal Large Language Models (MLLMs) empowered with frequently overlooked synergy between multimodal comprehension and creation. DreamLLM operates on two fundamental principles. The first focuses on the generative modeling of both language and image posteriors by direct sampling in the raw multimodal space. This approach circumvents the limitations and information loss inherent to external feature extractors like CLIP, and a more thorough multimodal understanding is obtained. Second, DreamLLM fosters the generation of raw, interleaved documents, modeling both text and image contents, along with unstructured layouts. This allows DreamLLM to learn all conditional, marginal, and joint multimodal distributions effectively. As a result, DreamLLM is the first MLLM capable of generating free-form interleaved content. Comprehensive experiments highlight DreamLLM's superior performance as a zero-shot multimodal generalist, reaping from the enhanced learning synergy. Project page: https://dreamllm.github.io.

Published as a conference paper at ICLR 2024
DREAMLLM: S YNERGISTIC MULTIMODAL COMPREHENSION AND CREATION
Runpei Dong 12xt
Chunrui Han 3* Yuang Peng 41
Zekun Qi
Zheng Ge
Jinrong Yang 5t
Liang Zhao 3
Jianjian Sun 3
Hongyu Zhou" 3
Haoran Wei 3
Xiangwen Kong
Xiangyu Zhang 3:
Kaisheng Ma 49
Li 467 %:
Xi' an Jiaotong University
IInstitute for Interdisciplinary Information Core Technology (IIISCT)
MEGVII Technology *Tsinghua University *HUST "Shanghai Qi Zhi Institute
Shanghai Artificial Intelligence Laboratory
ABSTRACT
This paper presents DREAMLLM, learning framework that first achieves versatile Multimodal Large Language Models (MLLMs) empowered with frequently over- looked synergy between multimodal comprehension and creation. DREAMLLM operates on two fundamental principles. The first focuses on the generative model- ing of both language and image posteriors by direct sampling in the multimodal space. This approach circumvents the limitations and information loss inherent to external feature extractors like CLIP, and more thorough multimodal understand- ing is obtained. Second, DREAMLLM fosters the generation of raw, interleaved documents, modeling both text and image contents, along with unstructured layouts. This allows DREAMLLM to learn all conditional, marginal, and joint multimodal distributions effectively. As result, DREAMLLM is the first MLLM capable of generating free-form interleaved content. Comprehensive experiments highlight DREAMI. MM' s superior performance as zero-shot multimodal generalist, reaping from the enhanced learning synergy. Project page: dreaml 1m. gi thub. io.
INTRODUCTION
"What I cannot create, do not understand."
Richard Feynman, on his blackboard the time of his death, 1988
Content comprehension and creation in multimodality are crucial and among the ultimate courses of machine intelligence (Sternberg, 1985; Legg & Hutter, 2007). To this end, Multimodal Large Language Models (MLLMs) (Alayrac et al., 2022; Hao et al., 2022; Huang et al., 2023) have emerged extensions of the successful GPT-style Large Language Models (LLMs) (Brown et al., 2020; Zhang et al., 2022; OpenAI, 2022; 2023a;b; Chen et al., 2023b; Touvron et al., 2023a;b) into visual realm. Recognized foundation models (Bommasani et al., 2021), MILMS have achieved unprecedented progress in multimodal comprehension capabilities. These advanced models typically enhance LLMs by incorporating images as multimodal inputs, such CLIP features (Radford et al., 2021), to facilitate language-output multimodal comprehension. Their aim is to capture multimodal conditional or marginal distributions via a language posterior. However, multimodal creation, which involves generating images, texts, or both, necessitates universal generative model that simultaneously learns language and image posteriors-currentyy underexplored.
Until very recently, some concurrent works have shown success in conditional image generation using MLLMs (Koh al., 2023; Sun al., 2023b). As depicted in Fig. 1, these methods compel MILMS to produce either discrete continuous conditional embeddings that explicitly align with pretrained CLIP encoder, which could later be used by a pretrained Stable Diffusion (SD) (Rombach et al., 2022) model for image generation. However, due to inherent modality gap (Liang et al., 2022), CLIP
"Euul contribution. Work partially done during the internship IISCT and MEGVII. ! Project leaders. CCorresponding authors.

---

Published as a conference paper at ICLR 2024
A. image
Abgement
Contastivo All Ar ment.
-0
+0
Text
Emboddinn
Image
MLLM
MLLM
MLLM
Visual Emboddinng
Inference
Stoam
(a) CLIP-iike
(b) Flamingo
(c) GILL/Emu-like
(d) DreamLLM (Ours)
Figure 1: Conceptual comparison of vasion-language (VL.) foundation models. (a) CLIP-like
models (Radford et al., 2021; Yu et al., 2022a; Li et al., 2023e) take advantage of two towers that explicitly align VL representations. (b) Flamingo/BLIP-iike models (Alayrac et al., 2022: Li et al., 2022; 2023d; Huang et al., 2023) encode VL representations into a unified manifold space using singular MILM. However, these models lack full autoregressivity, as they only output language. (c) Concurrent MLLMs (Koh et al., 2023; Sun et al., 2023b) align visual outputs with CLIP representations, but this alignment occurs in an intermediate space, not a raw data space. Consequently, models such as Emu necessitate a second-stage fine-tuning of Stable Diffusion (Rombach et al., 2022) for raw image generation. These models also fall short in generating raw interleaved documents. (d) Our DREAMLLM, instead, generates raw language and image inputs in unified regreseeree manner, inherently enabling interleaved generation. Only non-autoregressive generation loss is noted.
semantics focus predominantly on modality-shared information, often overlooking
knowledge that could enhance multimodal comprehension. Consequently, these studies have not fully realized the potential learning synergy between multimodal creation and comprehension, have shown only marginal improvements in creativity, and remain deficient in multimodal comprehension.
In this work, we introduce DREAMLLM, universally learning image and text posteriors with expected creation & comprehension synergy, based on the following two de-facto designing principles:
i. Generate Everything as It Is Different from existing works that generate intermediate image representations like CLIP embeddings during training. DREAMLLM not only takes all modali- ties raw data as inputs but also outputs in truly end-to-end fashion (f.e., outputs are identical to inputs, see Fig. 1). The challenge lies in enabling MLLMS to learn the image posterior without compromising their comprehension capabilities. To address this, introduce dream queries, a set of learnable embeddings that encapsulate the semantics encoded by MLLMs. This approach avoids altering the output space of MLLMs. Raw images are then decoded by the SD image decoder conditioned on these semantics. In this fashion, the pretrained SD acts as the score function (Ho et al., 2020). The image posterior is thus modeled by direct sampling in the pixel space, facilitated by score distillation (van den Oord et al., 2018; Poole et al., 2023).
i. Interlcaved Generative Pre-Training (I-GPT) DREAMLLM is trained to generate interleaved multimodal corpora from the internet (Zhu et al., 2023b), both encoding and decoding interleaved image-text multimodal inputs. Unlike encoding multimodal inputs as in existing methods, decoding interleaved multimodal outputs challenging due to the complex interleaving layout structures and the long-contex, requirement of images. Our approach tackles the interleaved layout learning using a unique <dream> token that predicts the placement of images within texts. Harnessing PREAMLMM'S 's causal nature, all contents are generated with history multimodal contexts of any length. This interleaved generative pretraining (I-GPT) inherently forms all joint, marginal, and conditional distributions of images and texts in the document, leading to learning synergy that grounds DREAMLLM's 's comprehension in creation and vice versa.
Extensive experiments across various vision- languaee comprehension, content creation, and language- only tasks demonstrate DREAMLLM's 'sd superior performance as a zero-shot multimodal generalist. For instance, DREAMLLM- 7B achieves an 8.46 FID on MS-COCO and sets a new standard with 49.1/35.9 scores on MMBench and MM- evaluations, respectively. Moreover, we delve into the learning synergy between comprehension and creation, revealing decent in-context generation capabilities. With Z-GPT pretraining, DREAMLLM generates interleaved documents following human prompts after supervised fine-tuning on instruction instruction-folloving data curated with GPT-4. To knowledge, this work is the first to enable MLLMs to create free-form interleaved content with learning synergy on both sides. As a foundational learning framework. DREAMLLM is adaptable across all modalities, laying a promising foundation for future multimodal learning research.
2

---

Published as a conference paper at ICLR 2024
2 BACKGROUND & PROBLEM STATEMENT
Autoregressive Generative Modeling Given the joint probability distribution (w) over sequence w = [we](=] with length T. the canonical causal generation (Mikolov et al., 2010: Radford et al.. 2018; 2019) of every token Wt by 0-parameterized language model F is modeled as Po (w) II Po (o(we//=c For multimodal comprehension, the sequence could contain K ordered images I = {= interleaved with words. The k-th image is processed as patch embeddings with visual encoders (-) like CLIP, which will then be encoded by projector Mc (e.g. a linear layer (Huang et al., 2023) or DETR- (Carion et al., 2020)/Perceiverllkk (Jaegle et al., 2021) Resampler (Alayrac et al., 2022)) into L-length visual embeddings V; = {ve Let K (t) be the image number before the t-th word token. The maximum likelihood estimation (MLE)
minimize
( = =1,,5 w, 1) = EE, (w, wes: V<K()), Vk(0) - Mg o M Ho((k(o)) (I)
Diffusion Models Diffusion Models (DMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020) are probabilistic generative models that learn the latent structure of data z 1 through continuous- T-timestamps information diffusion. DMs involve a forward or diffusion process q that smoothly converts data to Gaussian noise. Given the initial datapoint 21 g(2,) and diffusion rate 3r == 1-ag, this process can be defined as marginal distribution q Z := (v/az1,3111) and the perturbed data distribution is ((z)) := q(z, (z)(z by integrating out data density qz)). A reversed denoising probability flow p is used for generating data from noise zr ~0 1) as Markov Chain with transition approximated by Gaussian model Pe (Z)-)//)) == .V (z) of which relates to an optimal MSE denoiser since q 22, Z1) ) is Gaussian with enough timestamps (Feller, 1949; Sohl- Dick-Dickttein et al., 2015). Ho et al. (2020) show that the optimization with the evidence lower (g(z.. bound (EL BBO) can be simplified by training a denoising U-Net eg (z. () parameterized with & that estimates the conditional expectation EE|E ~ (O, I))2,] (Bao et al., 2022). Let C be the conditional embeddings, and the perturbed data Ze = v Z + VI T.E, the minimization objective is
LDM 2) == E... [lleç(2; 1) - -<]
(2)
Since 6g (Z+;)) -0,8 (Z1; 1) as derived from Tweedie's (Efron, 2011; Luo, 2022), it is equivalent to denoising score matching of Vz, log pe (2:) (Hyvärinen, 2005; Vincent, 201 1), thus DMs are also called score-function based generative models (Song & Ermon, 2019; 2020; Song et al., 2021; 2023).
2.1 How CAN WE USE MLLMS FOR DIFFUSION SYNTHESIS THAT SYNERGIZES BOTH SIDES?
Multimodal signals typically exhibit modality-speciffc information that has distinct structure but complementary semantics (Dong et al., 2023). This complementary property allows to utilize deep language comprehension to enhance -rosslmodal image generation (Saharia et al., 2022). However, the potential of multimodal creation to improve comprehension remains largely unexplored.
Existing strategies (Koh et al., 2023; Sun et al., 2023b; Ge et al., 2023) integrate successful Diffusion Models with MILMS by aligning the semantic spaces of conditional embeddings between CLIP CCUI and MLLMs CMLLM The objective is to minimize alignment loss Lalign D(Mg o CMLLM CCLIP), employing distance metric (---)) and condition projector My. However, CLIP models primarily learn modality-shared semantics, often overlooking modality-specific information due to modality gap (Liang et al., 2022: Liu et al., 2023f). This explicit alignment with CLIP's intermediate output space may induce more conflicts than synergies, as MLLMs are forced to generate semantically reduced information, deviating from their original output space. To circumvent these issues, we propose alternative learning methodologies (See Fig. 2), which we elaborate in the ensuing sections.
Our aim is to leverage MILMS to model distributions via direct pixel space sampling. Here, the pretrained SD functions as score metric, distilling the learned data distribution. This approach is similar to Score Distillation Sampling (Poole et al., 2023) (SDS, also known as Score Jacobian Chaining (Wang et al., 2023a)). In this context, image posterior is learned in DeepDream-lic manner (Mordvintsev et al., 2015), using MLLMS' conditional parameterization.
Learning Objective
Conditional Embeddings Rather than converting the output space of MLLMs align with CLIP, we propose to query MLLMs using learned embeddings. Consequently, MLLMs-enrichee! semantics serve as diffusion conditioning, and the distribution is implicitly modeled through synthesis sampling.
3

---

Published as a conference paper at ICLR 2024
Interleaved Documents
ny cute Stanese cat.
U-Net
Soom Dusllaiion
beautsful
</>>
like cat." ?
Projection
<dream>
D g 0
0
Causal Multimodal Large Language Model (MLLM)
0
4-0 0
10000
+ 40
"The has beautsfut blue eyes, l6kes
dream queries
Projection
Word Embodding
25 on her coay sest." "I
Visusl Encoder
Embadding
embeddings
I Lske cuto Stamese cat.
has
Interence Stream
beaubiful </>>
Figure 2: Overview of of our DREAMLLM framework. Interleaved documents serve as input, decoded to produce outputs. Both text and images are encoded into sequential, discrete token embeddings for the MLLM input. A special <dream> token predicts where to generate images. Subsequently, series of dream queries are fed into the MLLM, capturing holistic historical semantics. The images synthesized by the SD image decoder conditioned queried semantics. The synthesized images are then fed back into the MLLM for subsequent comprehension.
3 DREAMLLM
We introduce DREAMLLM, a universal learning framework that facilitates both MLLM's compre- hension and creation capabilities. Our DREAMLLM is built with causal decoder-only LLM Fg as the model foundation, i.e. Vicuna (Chiang et al., 2023) based on LLaMA (Touvron al., 2023a) trained on ShareGPT (Zheng et al., 2023). We adopt OpenAl's CLiP-Large (Radford et al., 2021) 1) as the visual encoder Hg. followed by linear layer Mc for visual embedding projection. To synthesize images, we use Stable Diffusion (SD) (Rombach et al., 2022) as the image decoder, and the condition projector M& is also linear layer. An overview of the architecture depicted in Fig. 2.
3.1 END-TO-END INTERLEAVED GENERATIVE PRETRAINING (I-GPT)
All natural documents can be regarded carriers of text-image interleaved information. Text-only, images-only, and text-image pairs data, on the other hand, can be seen as special cases of interleaved corpora with different modality compositions. Thus, it is critical to empower the model with the capability to learn and generate free-form interleaved documents that form all possible distributions.
Interleaved Structure Learning To model the interleaved structure, the interleaved sequence is operated by extending a new special <dream> token before images. During training. DREAMLLM trained predict this <dream> token that indicates where an image emerges, and the conditional image synthesis is performed afterward, as introduced next. During inference, DREAMLLM will generate an image on its "free will" when this token is predicted.
Conditional Synthesis through Score Distillation To avoid the possible conflicts of CLIP semantics and MLLMs stated in Sec. 2.1, carefully design a different learning objective and conditional embeddings. Formally, we introduce a series learnable dream queries with length Q: 1 Considering the t-th token predicted as <dream> token, the conditional embeddings LDREAMLLM for the (K(t) + 1)-th image synthesis can be obtained by causally querying the previous sequences:
(K(+)+1
DREAMILM = Fe (d, x V<K())+1). K1
Thus, the denoising score matching with latent z is motivated the similar formulation to Eq. (2):
CDREAMLLM (0, d, S, vb 2) == F=-2000.))e-N10.. (z; CDREAMILLM - -<1l
(4)
where is not updated since the SD is frozen. Eq. (4) can also be viewed as a generalized formulation of textual inversion (Gal et al., 2023), but all condition embeddings are learnable by model-seeking. From the perspective of score distillation (van den Oord et al., 2018), the KL divergence defined by conditions and the pre-learned score function is equivalently minimized for distilling (Hinton et al.,
4

---

Published as a conference paper at ICLR 2024
Table I: Zero-shot multimodal comprehension evaluation of image-to-text captioning, general VQA, text-related VQA, and comprehensive benchmarks. denotes non-zero-shot results for VQA. DREAMLLM-7B trained using the SFT data constructed by LLaVA-1.5 (Liu et al., 2023b).
Captioning
VQA
Comprehensive
Method
COCO 12Paragraph
VQAr2
OKVQA
VizWiz TexiVQ
MMBench MM-Vet
Comprehension Only MLLMs
MetaLM (Hao et al., 2022)
41.1
11.4
Kosmos-1 (Huang et ul., 2023)
51.0
29.2
Flamingo-9P (Alayrac et al., 2022)
79.4
44.7
28.8
OF-9B (Awadalla et al. 2023)
65,5
52.7
37.8
27.5
29.1
4,6
21.8
LLaVA-7B (Liu et al., 2023c)
28.9
38.7
23.8
MLLMs for Comprehension & Creation
CM3Leon-7B (Yu et al., 2023a)
61.6
10.5
47.6
23.8
37.6
Emu- (Sun al., 20236)
117.7
DREAML.MM-T (Ours)
115.4
17.4
56.6
44.3
45.8
34.9
49,9
35,9
DREAMI.LM-7B' (Ours)
103.7
8.4
72.9
52.2
49.3
41.8
58.2
36.6
2015) learned probability density in conditional image synthesis:
min DREAMLLM : E, CURLSSLLL
DKL (g(z(-1/z,, Z1, CDREAMLLM) pe pe(Z)-1/zi)]
(5)
DM
O.d.C.P
Universal Multimodal Generative Modeling An interleaved document sequence X
}
contains both words w [ and images I { The autoregressive nature forms all possible conditional distributions, such as image conditional multimodal comprehension p( [w] text-to-image synthesis ([[w)), The images processed visual embeddings V for causal comprehension. Assuming that the pretrained SD is an optimal score function, Eq. (5) thus could be viewed as MLE optimization for the synthesis posterior. Different from Eq. (1), the targeted sequence now could be both encoded images words. The objective is thus unified to the MLE of all eousally-conditionn posteriors in arbitrary forms:
cDREMMLMM (e (0, d. v. x) == -E, loog pe (x1 xx])]. MLLM
(6)
3.2 MODEL TRAINING
In this work, we consider three-stage training procedure. It can be summarized as follows, and the implementation details, like training data, can be found in Table 13 in Appendix C.
I Alignment Training This stage is used to alleviate the gap in multimodality, facilitating the adaptation of multimodal inputs LLMs. The linear visual projector, linear condition projector, and learnable dream embedddnggs are pretrained for cross-modal manifold alignment among frozen LLMs, visual encoder, and SD. We approximately 30M image-text pairs data, training both image-to-text comprehension and text-to-image synthesis.
II Z-GPT Pretraining Following alignment, the LLM undergoes an unfrozen process for I-GPT pretraining (detailed in Sec. 3.1). This critical stage facilitates the learning of joint vision-language distributions via generative modeling. Training incorporates approximately 2M selectively filtered documents from MMC4-Core (Zhu et al., 2023b), adhering to a CLIP score threshold of 0.25. Furthermore, we use 2M paired data samples from LAION400M (Schuhmann et al., 2021), captioned by BLIP (Li et al., 2022) (i.e., BLIP-LAION), enhance text-to-image training and potentially mitigate the impact of some low-quality noisy images and texts from sMMC4.
III Supervised Fine-tuning This stage enables the model to perform general multimodal prehension and creative tasks following human instructions (Ouyang et al., 2022). We utilize approximately 80K visual instruction tuning data collected by Liu al.. For instruction-folloiing content creation. - is prompted with document summaries or image captions, collecting approximately 20K instruction-followee 20K image synthesis data from BLIP captioned LAION400M (Instruct-BLIP BLIP-LAION).
document synthesis from MMC4 (InstructMMC4) and
EXPERIMENTS
DREAMLLM is versatile multimodal generalist that excels zero-shot in-context vision- language comprehension and synthesis tasks. In this section, we conduct systematic evaluations for demonstration. See qualitative results in Appendix and implementation details in Appendix C.
5

---

Published as a conference paper at ICLR 2024
4.1 MULTIMODAL COMPREHENSION
Multimodal comprehension enables humans to interact with agents conditioned on both words and visual content. We evaluate the multimodal vision and language capabilities of DREAMLLM across several benchmarks, including image-to-text captioning on COCO (Karpathy & Fei-Fei, 2017) and ImagePParagraph (Krause et al., 2017), general visual question answering (VQA) on VQAv2 (Goyal et al., 2019), OKVQA (Marino al., 2019), VizWiz (Gurari al., 2018), and text-related VQA on TextVQA (Singh et al., 2019). Additionally, we conducted a zero-shot evaluation on the recently developed benchmarks of MMBench and MM- Vet to assess the model's performance in complex multimodal tasks. The results are presented in Table I (See Table 5, and Table 6 in Appendix A), All metrics and data splits are listed in Table 14 Appendix C. We find that i) DREAMLLM outperforms other MLLMs across all benchmarks. Notably, DREAMLLM-7H surpasses concurrent MLLMs with image synthesis capabilitiess by a significant margin, achieving + 11666 higher accuracy on VQ compared to Emu- 13B. ii) On comprehensive benchmarks like MMBench and MM- Vet, DREAMLLM achieves state-of-the-art performance against all 7B counterparts. Detailed analysis revealed superior spatial/relation reasoning capabilities in DREAMLLM compared to other MLLMs, likely a result of its image synthesis learning. See qualitative results and comparisons on multimodal dialogue Table 11, Table 12, Fig. 10, Fig. 11, and Fig. 12, in Appendix B.
4.2 TEXT-CONDITIONAL IMAGE SYNTHESTS
Text2Image one of the most commonly used techniques for creative content gener- ation that follows human's fabulous imag- inations through free-form languages.
Table 2: Zero-shot text-to-image generation FID on MS-COCO LN-COCO. LM denotes language model based methods, MG denotes multimodal generation meth- ods, and FIG denotes free-form interleaved generation methods. is fine-tuned SDv2. on our state I data." denotes perreval-augmentattoo (Sheynin et al., 2023). denotes results after stage alignment training.
We assess text-conditional image synthe- sis on the MS-COCO validation set (Lin et al., 2014) and LN-COCO, the COCO subset of Localized Narratives (Pont- Tuset et al., 2020), following prior works (Xu et al., 2018; Yu et al., 2022b). The MS-COCO dataset primarily con- tains high-level image abstractions with shorter captions. whereas LN-COCO pro- vides more comprehensive image descrip- tions (Yu et al., 2022b). DREAMLLM samples images per text prompt on MS- COCO by CLIP score ranking, following previous works (Ramesh et al., 2022). On LN-COCO, DREAMLLM samples one image per prompt without CLIP ranking since the text is too long and exceeds the CLIP length limit. Note that Parti sam- ples 16 images per prompt with CoCa (Yu et al., 2022a). Our evaluation metric is the zero-shot Fréchet Inception Distance (FID) (Heusel et al., 2017). the results of which are presented in Table We note three key observations: i) Our DREAMLLM shows signifi- cant FID improvement over the StableDiffusion baseline after stage-I alignment, reducing the score by 3.67 and 11.83 on MS-COCO and LN- LN-COCO, respectively. Further, FID improvements of 3.97 and 13.73 are achieved after pretraining and supervised fine-tuning. The substantial improvement on LN-COCO underscores DREAMLLM'S superior capability in processing long-context information. ii) When compared to prior specialist models, DREAMLLM delivers competitive results based on the SD image decoder. iii DREAMLLM consistently outperforms concurrent MLLMM--bsee image synthesis methods. For instance, DREAMLLM-7B surpasses Emu-13B by significant 3,20 FID on MS-COCO. See qualitative results on text-to-image synthesis in Fig. 13 and Fig. 14 in Appendix B.
Method
LM MG FIG MS-COCO LN-COCO
Texe2mmage Specialists
Retrieval Result (Yu et al.)
x
x
17.97
33.59
DALL-F (Ramesh et al.) CogView (Ding el al.)
x
X
~28
x
X
X
27.1
VieWiew2 (Ding et al.)
X
X
X
24.0
SDv2.1 (Rombach al.)
X
X
12.43
34.26
SDv2.1 (Rombuch al.)
X
X
11.91
25.35
GLIDE (Nichol al.)
X
X
12.24
Make-A-Scent (Gafni et al.)
X
X
11.84
DALL-E 2 (Ramesh et al.)
X
X
10.39
Muse-3B (Chang et al.)
X
7.88
Imagen-3.4B (Saharia et al.)
X
7.27
Par (Yu et al.)
/
X
X
7.23
15.97
Multimodal Large Language Models
133 (Aghajanyan al.)
/
/
29.56
GILL-88B (Koh et al.)
/
/
x
12.20
Emu-13B (Sun et al.)
/
/
11.66
CM3Lcon-7B* (Yu al.)
/
/
10.82
DREAMLLM-7B' (Ours)
/ 5
>
8.76
22.42
DREAMLLM-7B (Ours)
8.46
20.53
4.3 MULTIMODAL JOINT CREATION & COMPREHENSION
Free-form Interleaved Document Creation Leveraging the interleaved generative modeling from I-GPT, DREAMLLLM now generate interleaved documents in free-form manner. In

---

Published as a conference paper at ICLR 2024
Help me urste the infroduction and sUJuetratione about Vienna cáky, show ceveral high-guaissty photographs about the beauteful scenes and buéldings such as Viensa Internatsosal Opera House.
The Austrias capstal, Fienna, long known the Csty of Breans, and there is better place to expersence than thés romantic csty during the holiday season. The city's historic center is surrounded by collection of beautifut buildinga on aquare mile. Stroll along the famous Ring Bouleverd and advire the archisectare. Some the busldings dale back to the the most
18th centary. The Vienna State Opera House is besutiful bastdinge, Founded 1869, opera house home of the Vienna State Opera and the ansual Fsenna Opera Ball. <drean, Cosnepolstan flair, ax sclectsc win of archstaatural styles and a réch cultonal herétage come together in Pienna The csty sa home to a nunder of museuns, palaces and churches. The Sacher Porte and the Wiener Schnitzel are local specialliss. A vistt to the romantsc Cbriatmas Narket in Psensa is also snique expersence. 22
house. There mobsle phone on dsning table dining roon, and you are in the living room, Frite short article adout how would you uulh through the house to grab the aobile phone with your robotéc are step by step. Include several high-qualsty photographa to ellastrate your procedure.
Imagime you are robot agent
As a robol agent in the house, I would oalh through the house to grub the mobile phone is the diniing room as shoun én the smage below. <drean, 1>
First, would look around the Vivsng obstactes people that need to awoid. I would then walk towerda the dixing room and stop before entering the roon. Nesst, would look for the mobele phone on the table, as shown the smage below. <drean_2>
to make sare the area safe and there
Once I have identified the phone, would then move closer to it and extend my robot to grab it, shown in the image betow. <drean_3>
Once phone se grabbed, would then bring ét back Isving room and place st on the table where belongs.
Human Instruction
DreamLLM Generation
Selected DREAMLLM instruction following interleaved content creation examples. Note that each image is created automatically at the location decided by DREAMLLM, and then it will be fed back as multimodal comprehension input for the following content generation.
Figure 3:
Fig. 3, we showcase the generated interleaved contents based on human instructions. It demonstrates that: i) DREAMI.LM can generate meaningful content per the instructions. ii) The system can autonomously create images at any specified location by predicting the proposed <dream> tokens, thereby eliminating the need for additional human intervention. This more user-friendly approach compared to systems like Emu, which necessitate human input for image generation locations.
Image Quality Document quality can be influenced by factors such as text content, image quality (including image-text alignment), and illustration positioning. To assess the quality of generated documents, we utilized held-out instruction-follot following subset from the constructed InstreutMMC4 as a demonstrative tool. This subset comprises 15K documents across 30 MMC4-defined topics, with 500 samples per topic. We began by evaluating image quality using FIDD on this subset, generating each image based on the corresponding ground truth texts. The results revealed that when using only matched text inputs for image synthesis, SD achieved FID score of 74.77. In contrast, our DREAMLLM significantly outperforms SD with an FID score of 36.62.
Human Evaluation We perform a comprehensive human evaluation to assess the quality of the generated samples. We randomly selected 150 samples (5 per topic) for instruction-followii ment generation, mixing the generated and ground truth MMC4 documents without any identifying information. Five unbiased volunteers were then asked to determine whether the given samples were supported. Given the presence of duplicate and low-quality images in MMC4, the supportive rate for MMC4 was only 77.24%. In contrast, DREAMLLM model achieves supportive rate of 60.68%. surpassing the 30% Turing test requirement. This result indicates that the generated documents contain high-quality images placed logically, demonstrating the effectiveness of our model.
docu-
7

---

Published as a conference paper at ICLR 2024
5 DISCUSSIONS
5.1 SYNERGY BETWEEN CREATION & COMPREHENSION?
To elucidate the synergy between multi- modal creation and comprehension, we make the comparison among three meth- ods with DREAMLLM architecture, each utilizing identical training data yet differ- ing in their learning objectives: a) the Creation-only baseline, focused solely on image synthe- sis; b) the Comprehension-onll baseline, dedicated to word generation exclusively; the Joint-learning method, which is the default setting of DREAMLLM learning both image and language modeling.
Table 3: Concrete analysis of the synergy between mul- timodal comprehension and creation (image synthesis). ID denotes whether the interleaved dataset is used during the second stage of pretraining.
ID Laiggn
MM-Vet
VQAv2
COCO
Stable Diffusion
x
12.43
textdnccument-condttiee
Creation-only Creation-anly
x
8.50
8.57
Comprchension-onle
X
X
31.0
55.1
Comprehension-onle
X
34.4
54.3
Joint-leaming
35,9
56.6
8.46
Joint-lcaming
N/A
N/A
N/A
Quantitative Analysis As per Table 3, the following observations are made: i) The powerful language comprehension of LLLMs significantly enhances the performance of text-to-image specialists like SD, as evidenced by the impressive 8.50 FIID (line 1). ii) The use of interleaved data, such as MMC4, can potentially boost multimodal comprehension performance (line 4). iii) The proposed I-GPT further synergizes comprehension and creation with improved performance (line 5). iv) When incorporating CLIP alignment loss Calign stated in Section 2.1, our DREAMLLM fails converge but rather ends in collapsing point (line 6). This indicates that the queries are adaptively learning the true data distributions, where CLIP semantics are in conflict with MLLLM-encoded semantics.
Qualitative Analysis In Fig. 4, we compare answers some examplar VQA tasks from comprehension-onie learning modules, respectively. It can be seen that: i) The joint-learning method exhibits superior multimodal comprehen- sion, particularly in identifying subject re- lationships and attributes like object size. ii) In multimodal comprehension scenar- ios involving multiple image inputs, the joint-learning approach demonstrates en- hanced precision. This improved perfor-
Oeeiis dhat's differes Jelwers Thare bai andente? Paaee
wrtlting Ineadd
stoppiy rattig pasn. Made Mere lmags of bosis
and joint
posting hat wathr wilutig and Croineg the fert dast
bebenn
Aerowr & He seats hour bilts The sepleace tor
the postur baper the
bern Pasd setioo saituss cteins panstt fredas and
Figure 4: Qualitative comparison. Answer A: answer from comprehension-only models w/o interleaved train- ing; Answer B: answer from joint-learning models.
mance a natural outcome Z-GPT pretraining, allowing better modeling of multimodal correlations in various interleaved documents.
Multimodal In-Context Generation Multimodal in-context generation a critical emerging capa- bility for MILMs (Bommasani et al., 2021; Alayrac et al., 2022). While significant strides have been made in-context visual question answering, in-context image synthesis remains relatively lacking in exploration. The multimodal context-conditionn demonstrated in Fig. 5, offer promising insights into this domain. Tasks such as in-context image edition, subject-driven image generation, and compositional generation, however, pose significant
image synthesis capabilities of DREAMLLM, as
Mullmodia Inpui
DreamiL M
Input
Mulimoda Imput
Teach
prini fer wiiing HOAA
Tekly
(8) In-context Image Edilion
(b) in-context Subjont-DDivan Gonoration
(c) In-context Compositional Ganeration
Figure 5: Selected DREAMLLM in-context image generation examples. The X in multimodal inputs are replaced accordingly by the text prompts shown under the generated images. We show the results of the SD baseline in (c) with only the text prompt X for a comparison.
8

---

Published as a conference paper at ICLR 2024
challenges in zero-shot setting. particularly without downstream fine-tuning in DreamBooth (Ruiz et al., 2023) or attention modification techniques as in Prompt2Prompt (Hertz et al., 2023). Despite these hurdles, Fig. 5 illustrates DREAMLLM's ability to generate images conditioned on the provided image context. This capability suggests promising potential for DREAMILMM in maintaining subject, identity, and semantic context, thereby paving new way for resolving these complex tasks.
5.2 WHAT IS LEARNED BY DREAMLLM?
Dream Query Attention DREAM-
LLM, the conditional embedding is de- rived from MLLMs with some learned dream queries. Fig. 6 demonstrates vi- sualization of the learned cross-attention mechanism between these queries and the diffusion latent. Similar to (Hertz et al., 2023), visualize the attention map av- eraged across all timestamps. It is seen that: i) The query attention is structured, disentangled, and semantically-orientee This is evidenced by the fact that distinct queries adeptly capture different subject and background semantics. ii) Despite varying prompts, attention patterns exhibit remarkable similarity as shown in Fig. 6 (a) and (b). This contrasts with the token attentions from the original SD, which are typically text-token dependent. We postulate that this arises from the model's ' causal nature, leading consistent semantic structure order.
cst ot
polan
glaza
Irage
image
arem Uitecatert
oWam guNvN LiNe Ialent
Figure 6: Cross-attention of dream queries and the dif- fusion U-Net latent. Similar to (Hertz al., 2023), the 64 queries can be viewed as 64 "words". Each attention map computed as the cross-attention between each query and the latent feature in the U-Net. The 64 queries are ordered as 8 grid sequentially, and each attention map is the result averaged across all timestamps.
RELATED WORKS
Rapid developments have been witnessed in extending LLMs like LLaMA (Touvron et al., 2023a) to multimodal comprehension that enables human interaction with both words and visual content. One line work is built by system integration of LLMs with various functioning agents where language acts as general interface (Wu et al., 2023; Gupta & Kembhavi, 2023; Yang et al., 2023b; Liang et al., 2023; Shen et al., 2023; Yang et al., 2023a; Surís et al., 2023), and remarkable success has been demonstrated in such plugin-style frameworks. Another line of work instead explores training LLMs to consume and understand multimodal inputs (Hao et al., 2022; Huang et al., 2023; Chen al., 2023b) with parameter-efficien tuning (Hu et al., 2022; Alayrac et al., 2022: Li et al., 2023d; Zhang et al., 2023e; Zhu et al., 2023a; Ye et al., 2023) and instruction tuning (Xu et al., 2023b; Liu et al., 2023c; Dai et al., 2023a). More recently, some approaches have been developed towards visual-interactive multimodal comprehension by precise referring instruction tuning (Zhao et al., 2023a; Peng et al.. 2023; Chen al., 2023a; Zhang et al., 2023g). For cross-modal creation, early works generally tokenize the visual contents into discrete VQ codebooks (van den Oord et al., 2017; Wang et al., 2022; Sun et al., 2022; et al., 2023; Diao et al., 2023; Yu et al., 2023a). Recent works instead explore incorporating MLLMs for image synthesis using text-to-image models such Stable Diffusion, and the objective is to generate conditional embeddings that align pretrained CLIP text (i.e., CLIP) or CLIP variant embeddings (Koh al., 2023; Ge et al., 2023; Sun et al., 2023a;b).
CONCLUSIONS
How can the learning synergy between multimodal content understanding and creation emerge? In this paper, we present DREAMLLM, a learning framework for developing MLLMs that not only comprehends but also creates multimodal content via diffusion models. Through score distillation of conditional-image synthesis distributions, we avoid the need for intermediate representation targets that may bring information loss. The employment of interleaved documents further enriches the multimodal distributions, fostering the learning of multimodal encoding and decoding. Our extensive empirical evaluations across diverse VL benchmarks demonstrate the effectiveness of DREAMLLM and the emerging learning synergy between multimodal content understanding and creation. Besides, this work initiates the first step towards free-form interleaved content creation. As a general learning framework, we hope it will spur further research in the multimodal machine learning field.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This research is supported by the National Natural Science Foundation of China (20211710187)
REFERENCES
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. CM3: A causal masked multimodal model of the internet. CoRR, (bs/2201.07520) 2022. 6
Bap-tattstteeee Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katic Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooci, Marianne Montciro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2022. 1, 3, 8, 24
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. In Int. Conf. Comput. Vis. (ICCV), 2015. 30
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Yitzhak Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and I Ludwig Schmidt. Openflamingo: An open-source framework for training large autoregressive vision-language models. CoRR, abs/2308.011111 390, 2023. 5, 22, 23, 26, 27, 28
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In Int. Conf. Learn. Represent. (ICLR), 2022.
James Betker, Goh Gabriel, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lec, Yufci Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, Yunxin Jiao, and Aditya Ramesh. Improving image generation with better captions. 2023. 26, 31
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In AAAI Conf. Artif. Intell. (AAAI), 2020. 22, 30
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bemstein, Jcannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quiney Davis, Dorottya Demszky, Chris Donahuc, Moussa Doumbouya, Esin Durmus, Stefamo Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei. Chelsea Finn. Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang, Thomas Ieard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al. On the opportunitics and risks of foundation models. CoRR, abs/2108.07255 2021. 1,8
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Fric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Adv. Neural Inform. Process. Syst. (NeurIPS). 2020. 1. 22. 30
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunicr, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Eur. Conf. Comput. Vis. (ECCV), 2020. 3
Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, José Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. In Int. Conf. Mach. Learn. (ICML), 2023. 6
Hila Chefer. Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen Or. Attend and-excite Attention tleention-base semantic guidance for texaaee diffusion models. ACM Trans. Graph. (48:)::48::11488: 2023. 31
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng 7hu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. CoRR, ahs/2306...5111 2023a. 9
10

---

Published as a conference paper at ICLR 2024
Xi Chen, Xiao Wang. Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman. Adam Gryener, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong. Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V. Thapliyal, James Bradbury, and Weicheng Kuo. Pali: A jointly-scaled multilingual inguage-image model. In Int. Conf. Learn. Represent. (ICLR), 2023b. 1, 9
Wei-Liin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chathot impress- ing gpt-4 with 90%* chatgpt quality, March 2023. URL https: / /ttss//mmssss.///m//m/ or wrggbbllg////---//gg 00-30-3-c-0aaa 4, 22, 26, 29, 31
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roherts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi. Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao. Parker Baress, Yi Tay, Noam Shazeer. Vinodkumar Prab- hakaran, Emily Reif. Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lovskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hycontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. CoRR, abs/2204.00311 2022. 22, 30
Paul F. Christiano. Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Adv. Neural Inform. Process. Syst. (NIPS), 2017, 30
Christopher Clark, Kenton Lee, Ming- Ming-Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Roolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACI-HIT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), 2019. 22, 30
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In Adv. Neural Inform. Process. Syst. (NeurlPS), 2023a. 9, 22.2..
Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. Plausible may not be faithful: Probing object hallucination in vision-language pre- pre-training. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, 2023b. 23
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory- efficient exact attention with 10-awareness. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2022. 29
Shizhe Diao, Wangchunshu Zhou, Xinsong Zhang, and Jiawei Wang. Write and paint: Generative vision- language models are unified modal learners. In Int. Conf. Learn. Represent. (ICLR), 2023. 9
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. Cogview: Mastering text-to-imes text-to-image generation via transformers. In Adv. Neural Inform. Process. Syst. (NeurlPS), 2021. 6
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. In Adv. Neurul Inform. Process. Syst. (NeurIPS). 2022.
Runpei Dong, Zhanhong Tan, Mengdi Wu, Linfeng Zhang, and Kaisheng Ma. Finding the task-optimal low-bit sub-distribution in deep neural networks. In Int. Conf. Mach. Learn (ICMI), 2022. 25
Runpei Dong, Zekun Qi. Linfeng Zhang, Junbo Zhang, Jianjian Sun, Zheng Ge, Li Yi, and Kaisheng Ma. Autoencoders as cross-modal teachers: Can pretrained 2d image transformers help 3d representation learning? In Int. Conf. Learn. Represent. (ICLR), 2023. 3,
Nan Du, Yanping Huang. Andrew M. Dai, Simon Tong, Dmiiry Lepikhin. Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P. Bosma, Zongwei Zhou, Tao Wang. Yu Emma Wang, Kellic Webster, Marie Pellat, Kevin Robinson, Kathleen S. Mcier-Hellster, Toju Duke, Lucas Dixon, Kun Zhang, Quoe Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with ofxturs-of-experts. In Int. Conf. Mach. Learn. (ICML), 2022. 30
Bradley Efron. Tweedie's 's formula and selection bias. Journal of the American Statistical Association, 106(496): 1602-1614, 2011.
11

---

Published as a conference paper at ICLR 2024
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Mach. Learn. Res. (JMLR), 23:120:1-12:39 2022. 30
William Feller. On the theory of stochastic processes, with particular reference applications. In Proceedings of the [First] Berkeley Synposium on Mathematical Statistics and Probability. The Regents the University of Californi, 1949. 3
Weixi Feng, Xuehai He. Tsu-Jui Fu, Varun Jampani, Arjun R. Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In Int. Conf. Learn. Represent. (ICLR), 2023. 31
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In Eur. Conf. Comput. Vis. (ECCV), 2022.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. Int. Conf. Learn, Represent. (ICLR), 2023. 4,
Pong Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter V2: parameter-efficient visual instruction model. CoRR, abs/2304. 15010, 2023. 22, 23
Yuying Ge. Yixiao Ge. Ziyun Zeng. Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. CoRR, abs/2307 abs/2307.08041, 2023. 3.9
Rohit Girdhar, Alaacidin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind one embedding space bind them all. In IEEE/CVF Conf. Compur. Vis. Pattern Recog. (CVPR), 2023. 32
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng. Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt. A vision and language model for dialogue with humans. CoRR, 0as/2305,04790, 2023. 22, 23
Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Stee Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. Int. J. Comput. Vis. (IJCV), 1274444398 2019. 30
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. CoRR, abs/2300.0669/) 2023. 23
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In JEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023. 9
Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2018. 6, 30
Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang. Shuming Ma, and Furu Wei. Language models are general-purpose interfaces. CoRR, abs/2206. 063366 2022. 1, 9,
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language undcrstanding. In Int. Conf. Learn. Represent. (ICLR), 2021. 22, 30
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross-attention control. In Int. Conf. Learn. Represent. (ICLR), 2023. 9
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Adv. Neural Inform. Process. Syst. (NIPS), 2017, 6, 30
Geoffrey F. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, abs/1503.0253) 2015. 4
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop Deep Generative Models and Downstrean Applications, 2021. 30
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Adv. Neural Inform. Process. Syst. (NeurlPS), 2020. 2,3
12

---

Published as a conference paper at ICLR 2024
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Cao, Alexey A. Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David Fleet, and Tim Salimans. Imagen vidco: High definition vidco generation with diffusion models. CoRR, abs/2210.02303, 2022a. 31
Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2022b. 31
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rac, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. CoRR, abs/2203. 15556, 2022. 30
Yining Hong. Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-1lm: Injecting the 3d world into large language models. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2023. 32
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation large language models. In Int. Conf. Learn. Represent. (ICLR), 2022.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv. Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som. Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models. CoRR, 1.w.2302.14045, 2023. 1, 2, 3, 5, 9, 22, 24
Huggingface. Transformers agent, 2023.
URL https: https://hugginggaccec http:////hggiinffff.oooooootttte /
fransformers.c ormers_agentss. Accessed: 2023-07-20. 23
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. J. Mach. Learn. Res.
(JMLR), 6069 2005. 3
Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman. Alexander Herzog, Daniel
Ho, Julian Ibarz, Alex Irpan, Eric Jang. Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahm, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang- Huei Lee, Yuheng Kuang, Sally Jesmonth, Nikhil J. Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu. Do as I can, not as say: Grounding language in robotic affordances. In Annu. Conf. Robot. Learn. (CoRI.), 2022. 32
Kuang-Huei
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2017. 31
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and João Carreira. Perceiver: General perception with iterative attention. In Int. Conf. Mach. Learn. (ICML), 2021. 3
Chao Jia, Yinfei Yang, Ye Xia. TinTing Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun-Hsuan Sung, Zhen Li. and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Int. Conf. Mach. Learn (ICML), 2021. 31
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.0838 2020. 30, 31
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. IEEE Trans. Puttern Anal. Mach. Intell. (TPAMI), 39(4)6664-67 676. 2017. 6. 30
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2023. 1, 3, 6,
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In Int. Conf. Learn Represent (ICLR), 2021. 32
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hierarchical approach for generating descriptive image paragraphs. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2017. 6,30 30
Shane I Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds Mach., 17 (4):391-444, 2007.
13

---

Published as a conference paper at ICLR 2024
Alexander C. Li. Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. In Int. Conf. Comput. Vis. (ICCV), 2023a.
Bo Li. Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi- model with io-context instruction tuning. CoRR, 0bs/22305.03726. 2023b. 23
Dongxu Li, Junnan Li, and Steven C. H. Hoi. Blip-diffuuion: Pre-trained subject representation for controllable text-to-image generation and editing. In Adv. Neural Inform. Process. Syst. (NeurlPS), 2023c. 25
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified languaa understanding and generation. In Int. Conf. Mach. Learn. (ICML). 2022, 5, 26. 29
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. Hoi. BLIP-2: bootstrapping inguage-image pre- training with frozen image encoders and large language models. In Int. Conf. Mach. Learn. (ICML), 2023d. 2, 9, 23, 27.28
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In IEEE/CVF Conf. Comput. Vis. Pattern Recog, (CVPR), 2023c. 2
Yifan Li. Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina-
tion in large vision-language models. CoRR, abs/2305. 10355, 2023f. 23, 30
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. LIm-grounded diffusion: Enhancing prompt understanding of toxt-to-image diffusion models with large language models. CoRR, 15522305.136555 2023. 31
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y. Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2022. 1,3
Yaoho Liang. Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lui, Lei Ji, Shaoguang Mao. Yun Wang. Linjun Shou, Ming Gong, and Nan Duan. Taskmatrix.ai ai: Completing tasks by connecting foundation models with millions apis. CoRR, abs/2303.16434 2023.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Liu, and Tsung Yi Lin. Magic3d: High-resolution text-to-3d content creation. In JEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023. 31
Tsung-Yr Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Eur. Conf. Comput. Vis. (ECCV), 2014. 3,30
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi- modal model with robust instruction tuning. CoRR, abs/2306. 14565, 2023a. 23
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. CoRR, 100/231,,,3744, 2023b. 5, 22, 23, 24, 29
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Ady. Neurul Inform. Process. Syst. (NeurIPS), 2023c. 5, 9, 22, 23, 26, 27, 28, 29
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-2 Zero-shot one image to 3d object. In Int. Conf. Comput. Vis. (ICCV), 2023d. 31, 32
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? CoRR, abs/2307 062281, 2023e. 22, 30
Zhengzhe Liu, Peng Dai, Ruihui Li, Xiaojuan Qi, and Chi- Wing Fu. ISS: image as stepping stone for text-guided 3d shape generation. In Int. Conf. Learn. Represent. (ICLR), 2023f.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic repre- sentations for aasion-and-language tasks. In Ady. Neural Inform. Process. Syst. (NeurIPS), volume 32, 2019. 29
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. UNIFIED-IO: unified model for vision, language, and multi-modal tasks. Int. Conf. Learn. Represent. (ICLR), 2023.
Calvin Luo. Understanding diffusion models: A unified perspective. CoRR, abs/2208. 119700 2022. 3
14

---

Published as a conference paper at ICLR 2024
Haley MacLeod, Cynthia L. Bennett, Meredith Ringel Morris, and Edward Cutrell. Understanding blind people's experienees with camputer-generater captions of social media images. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI '17, pp. 5988-5999 New York, NY, USA, 2017. Association for Computing Machinery. ISBN 978 9781450346559 23
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: visual question answering benchmark requiring external knowledge. In JEEE/CVF Conf. Comput. Vis. Pottern Recog. (CVPR). 2019. 6, 30
Tomás Mikolov, Martin Karafiát, Lukás Burget, Jan Cernocký, and Sanjecv Khudanpur. Recurrent neural network based language model. In Annu. Conf. Int. Speech Commun. Assoc. (INTERSPEECH), 2010. 3
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: representing scenes as neural radiance fields view synthesis. Commun. ACM, 65(1)999-106, 2022. 31
Alexander Mordvintsev, Christopher Olah, and Mike Tyka, Inceptionism: Going deeper into neural networks.
2015. URL /tttss://ai... http:::://aa..ggllleee///o////se googleblog.cooooooooooooooooggoo com/2015 5/06/ inceptionism- sm- deeeeddeeee-iinne neural. aluuall. html.
3
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen, GLIDE: towards photorcalistic image generation and editing with text-guided diffusion models. In Int. Conf. Mach. Learn. (ICML), 2022. 6, 26, 36
OpenAl. Introducing chatgpt. 2022. URL httphsss//peeaa..co/////g///////
1.30
OpenAl. Gpt-4v(ision) system card. 2023a. URL /tt.://tt/////eeaaaaaa/////a//e/ 26, 27, 28, 30
1,
OpenAT. GPT-4 technical report. CoRR, abs.aaa3./220747.44444 2023b. wtt.a//ttss/ooggaaaaac////aaaaaa 26, 28,
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 1m2text: Describing images using million captioned photographs. In Adv. Neural Inform. Process. Syst. (NIPS), 2011. 29
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L.. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano. Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback. Adv. Neural Inform. Process. Syst. (NeurlPS), 2022. 5, 30
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan La, and Jun-Yan Yan Zhu. Zero-shot image-to-image translation. ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH 2023, Los Angeles, CA, USA, August 6-10, 2023, pp. 11:1-11:11, 2023. 31
William Peebles and Saining Xic. Scalable diffusion models with transformers. CoRR, abs/2212.09748, 2022. 31
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei. and Julien Launay. The refinedweb dataset for falcon LLM: outperforming curated corpora with web data, and web data only. CoRR, abs/2306.00 116, 2023. 31
Zhiliang Peng. Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma. and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. CoRR, abs/2306.14822 2023. 9
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller. Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. CoRR. abs/2307.01952, 2023.
2023. 31
Jordi Pont- Pont-Tuset, Jasper R. R. Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In Eur. Conf. Comput. Vis. (ECCV), 2020. 2020. 6, 30
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d difffusion. In Int. Conf. Learn. Represent. (ICLR), 2023. 2, 3, 31.
Zekun Qi. Runpei Dong. Guofan Fan, Zheng Ge. Xiangyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining. In Int. Conf. Mach. Learn. (ICML.), 2023a. 32
15

---

Published as a conference paper at ICLR 2024
Zekun Qi, Muzhou Yu, Runpei Dong, and Kaisheng Ma. VPP: efficient conditional 3d generation via voxel-point progressive representation. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2023b. 32
Zekun Qi. Runpei Dong, Shaochen Zhang, Haoran Geng, Chunrui Han, Zheng Ge. He Wang. Li Yi, and Kaisheng Ma. Shapellm: Universal 3d object understanding for embodicd interaction. CoRR, ab$/2402.17766, 2024. 32
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. 3, 30
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8)99 2019. 3, 30
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn. (ICML), 2021.
1. 2, 4, 29
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann. H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring. Susannah Young. Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po- Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang. Jonathan Uesato, John Mellor, Trina Higgins, Antonia Creswell, Nat McAlcese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Flena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguma Kuncoro, Aida Nematzadch, Elena Grihovskaya, Domenic Donato, Angeliki I Lazaridou, Arthur Mensch, Jean-Baptiste I essiaau Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy. Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs/21 12.1 1446. 2021. 30
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Iyaa Sutskever. Zero-shot text-to-image generation. In Int. Conf. Mach. Learn. (ICML), 2021. 6, 26, 36
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. CoRR, abs/2204.06122 2022. 6, 26, 36,36
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. 23
Robin Rombach. Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In JEEE/CVF Conf. Comput. Vis. Pattern Recog, (CVPR), 2022. 1, 2, 4, 6, 30, 31
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In /EEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023. 9, 25
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2022. 3, 6, 26, 31,
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: adversarial winograd schema challenge at seale. Commun. ACM, 64(9):99-106, 2021. 22, 30
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019. 22, 30
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow. Alexander M. Rush, Stella Bider- man, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoit Sagot, Niklas Muennighoff, Alhert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Mapje Major,
16

---

Published as a conference paper at ICLR 2024
Iz Beltagy, Huu Nguyen. Lucile Saulnier. Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon. Yacine Jernite Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifcoluwa Adclani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100 2022. 31
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: open dataset of clip-filtered 400 million image-text pairs. CoRR, abs/21 1.02114, 2021. 5, 29
Christoph Schuhmann, Andreas Köpf. Richard Vencu, Theo Coombes, and Romain Beaumont, Laion coco: 600m synthetic captions from laion2b-en, 2023. URL https://laaion ai ai/blog/laion- coco/. 29
Yuzhang Shang. Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusson models. In JEEE/CVE Conf. Comput. Vis. Puttern Recog. (CVPR), 2023. 25
Piyush Sharma, Nan Ding. Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymedd, image alt-text dataset for automatic image captioning. In ACL, 2018, 29
Yongliang Shen, Kaitao Song, Xu Tam, Dongsheng Li, Weiming Lu, and Yucting Zhuang. Hugginggpt: Solving Al tasks with chatgpt and its friends in huggingface. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2023.
Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. knn-diffusion: Image generation via large-scale retrieval. In Int. Conf. Learn. Represent. (ICLR), 2023. 6
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to-video generation without tixt- data. In Int. Conf. Learn. Represent. (ICLR), 2023a. 31
Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Turii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, and Yaniv Taigman. Text-to-4d dynamic scene generation. In Int. Conf. Mach. Learn. (ICML). 2023b. 31
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2019. 6, 30
Jascha Sohl- bish--Dickstein. Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. Int. Conf. Much. Learn. (ICML), 2015. 3
Yang Song and Stefano Ermon. Generative modeling by estimating gradients the data distribution. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2019. 3
Yang Song and Stefano Ermon. Improved techniques for training score-hased generative models. In Adv. Neural
Inform. Process. Syst. (NeurIPS), 2020. 3
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Int. Conf. Learn. Represent. (ICLR), 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In Int. Conf. Mach. Learn (ICML), 2023. 3, 25
Robert Sterbberg. Beyond IQ: A triarchic theory of human intelligence. CUP Archive, 1985.
Qingfeng Sun, Yujing Wang, Can Xu, Kai Zheng, Yaming Yang, Huang Hu, Fei Xu, Jessica Zhang, Xiubo Geng. and Daxin Jiang. Multimodal dialogue response generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL, 2022. 9
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. EVA-CLIP: improved training techniques for CLIP at scale. CoRR, abs/2303. 15389, 2023a. 9
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. CoRR, abs/2307.0520 2023b. 1, 3, 6, 24
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. CoRR, abs/2303.08 128, 2023. 9
17

---

Published as a conference paper at ICLR 2024
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from single image with diffusion prior. In Int. Conf. Comput. Vis. (ICCV), 2023. 31
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-followinn llma model. https://githhue thub. com/ 2023. 31
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Bap- tiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. 1, 4, 9, 22, 29, 31
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferree Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee. Diana Liskovich, Yinghai Lu, Yuning Mao. Xavier Martinet, Todor Mihaylov. Pushkar Mishra, Igor Molybog. Yixin Nie, Andrew Poulton, Jeremy Reizenstein. Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanic Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307. 0.09888, 2023b. 1, 1,
Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Adv. Neural Inform. Process. Syst. (NIPS), 2017. 9
Aäron den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu. George van den Driessche, Edward Lockhart, Luis C. Cobo. Florian Stimberg. Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Hciga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis. Parallel wavenet: Fast high-fidelity speech synthesis. In Int. Conf. Mach. Learn (ICML), 2018. 2,
Ramakrishna Vedantam, Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. /EEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2015. 30
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Comput., 23(7):
1661-1674, 2011. 3
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Ych, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023a. 3, 31
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In Int. Conf. Mach. Learn. (ICMI.), 2022. 9
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou, Self-consistency improves chain of thought reasoning in language models. In Int. Conf. Learn Represent. (ICLR), 2023b. 31
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In Adv. Neural Inform. Process. Syst. (NeurIPS), 2023c. 31
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. In Int. Conf. Learn. Represent. (ICLR), 2022a. 30
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian lehter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of thought prompting elicits reasoning in large language models. In Ady Neural Inform. Process. Syst. (NeurIPS), 2022b. 31
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. CoRR, abs/2303.04671, 2023. 9
18

---

Published as a conference paper at ICLR 2024
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023a. 32
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Ham Zhang, Zhe Gan, Xiaolci Huang, and Xiaodong He. Attngan: Fine-grained text image generation with attentional generative adversarial networks. In IEEE/CVF Conf. Compul. Vis. Pattern Recog. (CVPR), 2018. 6
Zhiyang Xu, Ying Shen, and Lifu Huang. Multinnstrcct: Improving molti-moddl zero-shot learning via instruction tuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL) (Volume I: Long Papers), 2023b. 9
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt41ools: Teaching large language model to use tools via self-instruction. In Adv. Neural Inform. Process. Syst. (NeurlPS), 2023a. 9
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang, An empirical study of GPT-3 for few-shot knowledge-based VQA. In AAAI Conf. Artif. Intell. (AAAI), 2022. 24
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azamasab, Faisal Ahmed, Zicheng I Liu, Ce Liu, Michael 7eng. and Lijuan Wang. MM-REACT; prompting chatgpt for multimodal reasoning and action. CoRR, ahs/2303.1 133.11133 11381, 2023b. 9, 23
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang. Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post- training quantization for large-scale transformers. In Adv. Neural Inform. Process.
Syst. (NeurIPS), 2022. 25
Qinghao Yc, Haiyang Xu, Guohai Xu, Jiabo Yc, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengeheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hchong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality. CoRR, 14172304.14178, 2023. 93
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini. and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. T. Mach. Learn. Res. (TMLR), 2022a. 2, 6
Jiahui Yu. Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Han Zhang. Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. T. Mach. Learn Res. (TMLRR) 2022, 2022b. 6, 26, 37
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang. Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, Shang- Wen Li, Susan Zhang, Richard James, Gargi Ghosh, Yaniv Taigman, Maryam Faral-Z Asli Celikyilmaz, Luke Zettlemoyer, and Armen Aghajanyan. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. CORR, abs/2309. 025991, 2023a. 5, 6,
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. CoRR, htw/2308.02499 2023b. 22, 23, 30
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentenec? In Proceedings of the 57th Conference of the Association for Computotional Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume Long Papers, 2019, 22, 30
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam. Zixuan Ma. Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong. and Tang. GLM-130B: an open bilingual pre-trained model. In Int. Conf. Learn. Represent. (ICLR), 2023. 22, 31
Junbo Zhang, Runpei Dong, and Kaisheng Ma. CLIP--O3D: learning free open-world 3d scene representations from dense CLIP. In Int, Conf. Compur. Vis. Worksh. (ICCV Workshop), 2023a, 32
Junbo Zhang, Guofan Fan, Guanghan Wang. Zhengyuan Su, Kaisheng Ma, and Li Yi. Language-ussisted 3d feature learning for semantic scene understanding. In AAAI Conf. Artif. Intell. (AAAI), 2023b. 32
Linfeng Zhang, Xin Chen, Runpei Dong, and Kaisheng Ma. Region-aware knowledge distillation for efficient image-to- tommage translation. In Brit, Mach. Vis. Conf. (BMVC), 2023c. 31
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Int. Conf. Comput. Vis. (ICCV), 2023d. 31
19

---

Published as a conference paper at ICLR 2024
Renrui Zhang. Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning language models with zero-init attention. CoRR, 0ts/2303.16199, 2023e. 9
Renrui Zhang, Liuhui Wang. Qiao, Peng Gao, Hongsheng Learning representations from 2d pre-trained models image-to-point masked autoencoders. In IEEF/CVE Conf. Comput. Vis. Pattern Recog. (CVPR), 2023f. 32
Shilong Zhang. Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on sgion-of-interest. CoRR, abs/2307 03601, 2023g. 9
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: open pre-trained transformer language models. CoRR, abs/2205.0 010068 2022. 1,31
Liang Zhao, En Yu, Zheng Ge, Jinnong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, and Xiangyu Zhang. Chatspot: Bootstrapping multimodal Ilms via precise referring instruction tuning. CoRR, abs/2307.094. 2023a. 9,31
Yanli Zhao, Rohan Varma, Chien-Chin Huang. Shen Li, Min Xu, and Alban Desmaison. Intro- ducing pytorch fully sharded data parallel (fsdp) api, 2023b. URI. https: //tt://c/tttc...o/// httpp/yoor..roo/blogg...e.. introducing
pytorch- introducing--ytoorc--
sully-sharded-dee dateeedddd-----eeeeeeeeeee-eed-- parallel-api/ Accessed: 2022-03-14. 29
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang. Joseph E. Gonzalez, and Ion Stoica. Judging lmm-as-a-jude with mt-bench and chatbot arena. CoRR, hlx/2306.05683 2023. 29
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In Inf. Conf. Learn. Represent. (ICLR), 2023. 31
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. CoRR, abs/2304.10592, 2023a. 22, 23
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal C4: an open, billion-scale corpus of images interleaved with text. In Ady, Neural Inform. Process. Syst. (NeurIPS), 2023b. 2, 29, 30
20

---

Published as a conference paper at ICLR 2024
CONTENTS
Introduction
Background Problem Statement
2.1
How can we use MLLMs for Diffusion synthesis that synergizes both sides?
3
DREAMLLM
3.1
End-to-End Interleaved Generative Pretraining (I-GPT)
3.2
Model Training
4
Experiments
4.1
Multimodal Comprehension
4.2
Text-Conditional Image Synthesis
4.3
Multimodal Joint Creation & Comprehension
5
Discussions
5.1
Synergy between creation & comprehension?
5.2
What learned by DREAMLLM?
6
Related Works
7
Conclusions
Additional Experiments
22
A.1
Additional Natural Language Understanding Results
22
A.2
Additional Multimodal Comprehension Results
22
A.3
In-Context Multimodal Comprehension
24
A.4
Subject-Driven Image Generation
................................
25
A.5 Additional Ablation Study
25
A.6 Additional Discussions Prompt Rewriting Strategy
26
B Additional Qualitative Examples
26
Implementation Details
C.1 Training Data & Hyper-Parameters
29
C.2 DREAMLLM Model
29
C.3 Evaluation Benchmarks
30
Additional Related Works
30
Limitations, Failure Cases Future Works
31
21

---

Published as a conference paper at ICLR 2024
Table 4: Zero-shot natural language processing evaluation. We report the 5-shot result on MMLU and the relative performance of DREAMLLM compared to base LLM Vicuna-7B.
Commonsense Reasoning
Reading BoolQ
Multitask
Method
PIQA
SIQA
HellaSwag
WinoGrande
MMLU
Language Only Large Language Models (LLMs)
GPT-3 (Brown et al., 2020)
81.0
78.9
70.2
60.5
43.9
PaLM-540B (Chowdhery et al., 2022)
82.3
83.4
81.1
88.0
69.3
ALaMA-7B (Touvron et al., 2023a)
79.8
48.9
76.1
70.1
76.5
35.1
7B (Chiang al., 2023)
77.7
47.5
75.7
67.5
73.9
45.0
Multimodal Large Language Models (MLLMs)
MetaLM (Hao et al., 2022)
72.3
53.5
56.
62.2
Kosmos-1 I (Huang et al., 2023)
72.9
50.0
54.8
56.4
DREAMIL.M-78 (Ours)
78.6-15 48.8+1.3
77.4+1.7
68.5+1.0
75.2+1.3
41.8-3.2
Table Zero-shot multimodal comprehension evaluation on MMBench (Liu et al., 2023e) dev set. LR: Logical Reasoning, AR: Attribute Reasoning, RR: Relation Reasoning, FP-C: Fine- Perception (Cross Instance), FP-S: Fine- Fine-grained Perception (Single Instance), CP: Coarse Perception. DREAMLLM is trained using the SFT data constructed by LA- (Liu et al., 2023b).
Method
LR
AR
RR
FP-S
FP-C
CP
Overall
OpenFlamingo-998 (Awadalla et al., 2023)
4.2
15.4
0.9
8.1
1.4
5.0
6.6
MMGPT-7B (Gong et al., 2023)
2.5
26.4
13.0
3.4
20.8
15.3
MiniGPT-4-7B (Zhu et al., 2023a)
7.5
31.3
4.3
30.3
9.0
35.6
24.3
InstructBLIP-7B (Dai et al., 2023a)
14.2
46.3
22.6
37.0
21.4
49.0
36.0
VisualGLM (Zeng et al., 2023)
10.8
44.3
35.7
43.8
23.4
47.3
38.1
LLaVA-7B (Liu al., 2023c)
16.7
48.3
30.4
45.5
32.4
40.6
38.7
LLaMA-Adapter V2 (Gao et al., 2023)
11.7
35.3
29.6
47.5
38.6
56.4
41.2
MiniGPT-4-13B (Zhu et al., 2023a)
20.8
50.7
30.4
49.5
26.2
50.7
42.3
DREAMLLM-75 (Ours)
15.8
53.7
60.9
53.2
40.0
58.3
49.9
DREAMLLM-7B (Ours)
23.3
67.2
47.8
58.6
54.4
70.5
58.2
A ADDITIONAL EXPERIMENTS
A.1 ADDITIONAL NATURAL LANGUAGE UNDERSTANDING RESULTS
We evaluate the natural language processing capabilities of DREAMLLM post-multimodal adap- tation learning via zero-shot experiments language-only tasks. These included commonsense reasoning (PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), Wino- Grande (Sakaguchi et al., 2021)), reading comprehension (BoolQ (Clark et al., 2019)), and general multi-task benchmark (MMLU 5-shot (Hendrycks et al., 2021)) As Table illustrates, DREAMLLM outperforms the Vicuna baseline most language benchmarks. This suggests that DREAMLLM's multimodal adaptation does not compromise the language learning model's (LLM) capabilities. When compared prior Multimodal Language Learning Models (MLLMs), DREAMLLM demonstrates superior performance, although this may be attributed to the higher baseline results. This finding suggests that more robust LLM base model could yield improved results.
A.2 ADDITIONAL MULTIMODAL COMPREHENSION RESULTS
Detailed Comprehensive Comparison The evaluation results on MMBench (Liu et al., 2023e) and MM-Vet (Yu et al., 2023b) are presented in Table 5 and Table 6, respectively. The key observations from these results are as follows: i) Our DREAMLLM-77 outperforms all other 7B MLLMs, setting benchmark in overall performance. Notably, it even exceeds the performance of some 13B models, including LLaVA and MiniGPT-4. ii) A detailed capability evaluation reveals DREAMLLM's superior performance in fine- graingr understanding and relational/spatial comprehension. This advantage likely due to DREAMLLM's unique learning synergy, where image distributions are comprehended not solely through iaguage--postrrin comprehension but also through creation.
22

---

Published as a conference paper at ICLR 2024
Table 6: Zero-shot multimodal comprehension evaluation of core VL capabilities on MM-Vet (Yu et al., 2023b). denotes compositional systems with OpenAI GPT and various interfaces. Rec: General Visual Recognition, OCR: Optical Character Recognition, Know: Knowledge, Gen: Language Generation, Spat: Spatial Awareness, Math: Arithmetic Math. DREAMLLM is trained using the SFT data constructed by LLaVA-1.5 (Liu al., 2023b).
Method
Rec
OCR
Know
Gen
Spat
Math
Total
TF Agent-GPT-4+ (Huggingface, 2023)
18.2
3.9
2.2
3.2
12.4
4.0
13,4.10.5
MM-ReAct-GPT-3.55 (Yang et al., 2023b)
24.2
31.5
21.5
20.7
32.3
26.2
07.9+t..1
MM-RcAct-GPT-4 (Yang ct al., 2023b)
33.1
65.7
29.0
35.0
56.8
69.2
44.6:t022
LLaMA-Adapter v2-7B (Gao al., 2023)
16.8
7.8
2.5
3.0
16.6
4.4
13.6.0.22
OpenPlamingo-99 (Awadalla et al., 2023)
24.6
14.4
13.0
12.3
18.0
15.0
21.8+001
MiniGPT-4-8B (Zhu ct al., 2023a)
27.4
15.0
12.8
13.9
20.3
7.7
02.1+0.11
BLIP-2-12B (Li al., 2023d)
27.5
11.1
11.8
7.0
16.2
5.8
22,410.2
MiniGPT-4- 14B (Zhu et al., 2023a)
29.9
20.4
22.1
22.2
3.8
24.4.10.4
Otter-9B (Li et al., 2023b)
28.4
16.4
19.4
20.7
19.3
15.0
24.6.10.2
InstructBLIP- 14B (Dai al., 2023a)
30.8
16.0
9.8
9.0
21.1
10.5
25.6+00.
InstructBLIP-SK (Dai al., 2023a)
32.4
14.6
16.5
18.2
18.6
7.7
26.2+:0.2
LLaVA-7B (LLaMA-2) (Liu et al., 2023c)
32.9
20.1
19.0
20.1
25.7
5.2
28.11004
LLaVA-13B (LLaMA-2) (Liu et al., 2023c)
39.2
22.7
26.5
29.3
29.6
7.7
32.910.1
DREAMLLM-7B (Ours)
41.8
26.4
33.4
33.0
31.0
11.5
35.9+0.1
DREAMLLM-7B (Ours)
42.0
28.1
33.2
33.8
32.0
11.5
36.610.1
Table 7: Zero-shot visual hallucination evaluation on POPE (Li et al., 20230) using MS-COCO val set. Yes denotes the proportion of answering "Yes" to the given question, which is better if is more close to 50%. Objects that do not exist in the image are sampled with three different strategies. Random: random sampling, Popular: top-k most frequent objects in MS-COCO (k- 3), Adversial: objects are first ranked based on co-occurring frequencies, then top-k frequent ones are sampled.
POPE
Model
Accuracy
Precision
Recall
FI-Score
Yes (%)
mPLUG-Owl-7P (Ye al., 2023)
53.97
52.07
99.60
95.63
LLaVA-13B (Liu et al., 2023c)
50.37
50.19
99.13
MMGPT-7B (Gong al., 2023)
50.10
50.05
100,00
66.71
Random
MiniGPT-4-144B (Zhu et al., 2023a)
79.67
78.24
82.20
80.17
52.53
InstnuctBLIP-143 (Dai et al., 2023a)
88.57
84.09
95.13
89.27
56.57
DREAMLLM-7B (Ours)
86.36
85.92
87.93
86.91
52.75
mPLUG-OW1-7B (Ye et al., 2023)
50.90
50.46
99.40
66.94
98.57
1LaVA-13B (Liu et al., 2023c)
49.87
49.93
99.27
66.44
99.40
MMGPT-7B (Gong al., 2023)
50.00
50.00
100.00
66.67
100.00
Popular
MiniGPT-4-14B (Zhu et al., 2023a)
69.73
65.86
81,93
73.02
62.20
InstmectLIP--144 14B (Dai et al., 2023a)
82.77
76.27
95.13
84.66
62.37
DREAMLLM-77 (Ours)
80.07
75.74
88.47
81.61
58.40
mPLJUG-Ww1-78 (Ye et al., 2023)
50.67
50.34
99.33
66.82
98.67
LLaVA-13B (Liu et al., 2023c)
49.70
49.85
99.07
66.32
99.37
MMGPT-7B (Gong et al., 2023)
50.00
50,00
100.00
66.67
100.00
Adversarial
MiniGPT-4-14B (Zhu et al., 2023a)
65.17
61.19
82,93
70.42
67.77
InstructB1.IP-14B (Dai et al., 2023a)
72.10
65.13
95.13
77.32
73.03
DREAMIIL.-77 (Ours)
72.63
67.07
88.93
76.47
66.30
Visual Hallucination Visual hallucination, a phenomenon where MLLMs generate non-existent objects or identities in images, significantly compromises their multimodal comprehension capabili- ties (Dai et al., 2023b; Liu et al., 2023a; Gunjal et al., 2023) and may pose safety risks (MacLeod et al., 2017; Rohrbach et al., 2018). We assess the robustness of DREAMLLM against visual hallucination using the recently developed POPE benchmark (Li et al., 2023f), Refer to Table for detailed comparison with concurrent comprehension-only MLLMs. Our results indicate that DREAMLLM-P7 7B exhibits robustness to visual hallucination, matching or surpassing the performance of 13B counterparts. Remarkably, DREAMLLM achieves the best second-best performance in the most challenging setting. We posit that this robust anti-hat anti-hallucination property stems from a deep understanding of object concepts and semantics fostered by multimodal creation learning.
23

---

Published as a conference paper at ICLR 2024
Table 8: Few-shot multimodal comprehension evaluation. & is the number in-context examples.
i denotes methods using the RICES sample selection approach (Yang et al., 2022). DREAMLLM-7B'
trained using the SFT data constructed by LLaVA-1.5 (Liu et al., 2023b).
VQAv2
VizWiz
Method
k=2
k=4
k=8
k=2
k=4
k=8
Comprehension Only MILLMS
Kosmos-1 (Huang et al., 2023)
51.4
51.8
51.4
31.4
35.3
39.0
Flamingo-9P (Alayrac et al., 2022)
56.3
58.0
34.9
39.4
MLLMs for Comprehension Creation
Emu- 14Bt (Sun et al., 2023b)
56.4
58.4
59.0
37.8
41.3
43.9
DREAMLLM 7B (Ours)
58.1
59.2
59.4
46.1
46.7
46.8
DREAMLLM-7B (Ours)
73.8
74.4
73.8
49.8
50.3
49.7
A.3 IN-CONTEXT MULTIMODAL COMPREIEENSION
Few-Shot Evaluation In Table 8, we show the results of few-shot (i.e., k-shot and we set k=2, 8) evaluation by promoting models with a small number of training examples in context. The results demonstrate the strong in-context learning performance of DREAMLLM compared to Emu and Flamingo. It shows that DREAMILLM's effectiveness leveraging in-context knowledge.
Qualitative Examples In Fig. 7, we present qualitative instances of in-context comprehension using DREAMLLM. The illustrations indicate that DREAMLLM, when prompted with specific examples, efficiently executes in-context comprehension in the required formats and logic.
Input Prompt
Completion
woolly ly Mammo th.
This Velsh Corgs.
This ta Koala.
This is
Claude Nonet.
What is the tstle of this painting? Answer: The Son of Man.
Uhen this mas terpiece pasnted? Answer: 1818.
Who painted this?
Answer:
Joker
Fálm name: Harry Potter and the Prisoner Azkaban
FS name: 2049
FEIm name:
Figure 7: Selected DREAMLLM in-context multimodal comprehension examples.
24

---

Published as a conference paper at ICLR 2024
Ixpul Imagge
Ipp Imaqu
Fpp Image
Mie gonz
Aung
tte beach
in blue
grem
dog
top Aut
ruse ansctsiin
in Pract of
Figure 8: Selected zero-shot subject-driven image generation examples with DREAMLLM. The results demonstrate that DREAMLLM able to perform zero-shot subject-driven image generation while preserving image subject details and following generation instructions.
Table 9: Ablation studies and inference latency DREAMLLM. The zero-shot FID on MS-COCO 30K is reported. The inference latency is tested on NVIDIA A800 devices.
(a) The number of <dream> queries.
(b) Inference latency versus dif- ferent number of diffusion steps.
No. Queries COCOFiL
Steps
DREAMLLM
SD
32
9.56
50
3,65s
3.46s
64
8.46
100
7.02s
6.84s
14.24
150
10.41s
10.22s
A.4 SURJECT-DRIVEN IMAGE GENERATION
Image consistency important when generating interleaved content or performing controllable image generation tasks (Gal et al., 2023; Ruiz et al., 2023). However, MMC4 does not have such image consistency property, which leads unsatisfactory image consistency results as shown in Fig. 3. To further verify the effectiveness and potential of DREAMLLM in generating consistent images that preserve subject features, we fine-tune stage pretrained DREAMLLM on subject representation learning data constructed by following the recently proposed controllable image generation method ILII-Diffusion (Li et al., 2023c). We fine-tune DREAMLLM on this small ~270K samples data for 20 epochs, and the results shown in Fig. 8. It demonstrates the effectiveness and promising potential of applying DREAMLLM for image-consistent generation.
A.5 ADDITIONAL ABLATION STUDY
Query Number In Table 9a, we show the results of DREAMLLM using different numbers of the proposed learnable queries. iee., <dream> queries. The results show that 64 queries achieve the best result, while 128 may be too many, which may impact the performance. However, the choice of query number is also related to the choice training data size and diffusion model. For example, given more data and stronger diffusion model image decoder, queries more than 64 may be better.
Inference Latency In Table 9b, we present a comparison of real-time inference latency between DREAMLLM and SD. Relative to SD, DREAMLLM introduces a marginal latency cost of 0.2s on average. This is because the latency primarily stems from the computational demands of the diffusion U-Net demoising rather than the text condition embedding. To enhance inference efficiency, potential strategies could include the adoption of Consistency Models (Song et al., 2023) or the implementation of model compression techniques such as quantization (Yao et al., 2022: Dong et al., 2022; Shang al., 2023).
25

---

Published as a conference paper at ICLR 2024
Table 10: Language processing and multimodal comprehension & creation capability comparison to the seerrite-hhenggnneral baseline.
Language Processing
Multimodal Processing
Method
PIQA
SIQA
HellaSwag
WinoGrande
BoolQ
MMLU
VQAv2
MM-Vet
COCO
Vicuna-78 7B (Chiang et al., 2023)
77.7
47.5
75.7
67.5
73,9
45.0
rewrite-then-eeeeee
78.2
48.5
75.8
68.3
77.4
43.1
54.2
34.1
11.91
DREAMLLM-77 (Ours)
78.6
48.8
77.4
68.5
75.2
41.8
56.6
35.9
8,46
A.6
ADDITIONAL DISCUSSIONS ON PROMPT REWRITING STRATEGY
Very recently, OpenAl has released DELLE-3 (Betker et al., 2023), which proposes to improve generated image quality by rewriting descriptive and better prompts with GPT-4. This product has demonstrated great success in leveraging LLMs language-output agents. However, generally requires a large amount of high-quality data and is limited when applied to condee-ondiiional genera- tion tasks. For instance, DALLE-3 necessitates the initial training of bespoke image captioning specialist capable of producing high-quality descriptive captions, followed by model training in a data-rich environment featuring these written captions. This process is non-trivial, hinging heavily on the availability of substantial volumes of high-quality data. Moreover, such disjoint systems cannot guarantee learning synergy. In contrast, our exploration of DreamLLM has essentially unveiled the significant potential of LLMs to attain a comprehensive understanding of multimodality that genuinely comprehends modalities beyond mere language.
To make comparison regarding language processing and multimodal comprehension capabilities to this rewrite-ther generate baseline method, we conduct a preliminary study. Given the absence of optimal dataset holding improved prompts,
modify the original MMC4 by using <dream> start & end tokens before and after the specific text prompt that has the highest CLIP similarity to a specific image, which can be used as text prompts for image generation. In this setting, we only train the LLMs to output texts, and no image decoders are involved during training. During inference, when the model outputs texts encompassed by the <dream> tokens, the texts are used for an off-the-shelf SD image decoder for generating images. After training, we test the model's language processing and multimodal capabilities. The results show that i) the rewrite-then- eenereee method achieves similar performance to DREAMLLM This demonstrates that both methods won't 't impact the language capability, which is as expected. ii) the performance of the rewrite-ther geeen-geeettee baseline falls short when compared to DREAMLLM, particularly in the context of text-to- to-image generation on the COCO dataset. This underlines the efficacy of the synergistic learning approach inherent in DREAMLLM, suggesting its potential superiority over the baseline methodology.
B ADDITIONAL QUALITATIVE EXAMPLES
Multimodal Dialogue In Tables 11 and 12, we present qualitative comparative analysis of VQA results between our model, DREAMLLM, and other state-of-the fet-the-art art models: GPT-4 (OpenAI, 2023a;b), LLaVA (Liu et al., 2023c), BLIP-2 (Li et al., 2022), and OpenFlamingo (Awadalla et al., 2023). The key findings are as follows: i) DREAMLLM surpasses GPT-4 in providing more detailed and precise responses to given questions. ii) While LLaVA (Liu et al., 2023c) also offers detailed responses, it frequently introduces imaginary elements not present in the image. In contrast, DREAMLLM delivers more accurate answers, effectively avoiding this visual hallucination issue. This observation aligns with our earlier findings in Table 7, which underscore the robustness of DREAMLLM against visual hallucination. Furthermore, we showcase additional qualitative results of the multimodal dialogue in Fig. 10, Fig. 11, and Fig. 12. These figures illustrate DREAMLLM'S proficiency in comprehending and generating long-context multimodal information in arbitrary input and output formats.
Text-condition Image Synthesis In Fig. 13 and Fig. 14, we show the image examples of DREAM- LLM using the same prompts from previous works for a cross reference and comparison, including DALL-E (Ramesh et al., 2021), DALL-E 2 (f.e., unCLIP) (Ramesh et al., 2022), GLIDE (Nichol et al., 2022), Imagen (Saharia et al., 2022), and Parti (Yu et al., 2022b). Similar to Parti, we have extended some prompts with new sub--uuuupeep sub-prompts construct examples from different prompts.
26

---

Published as a conference paper at ICLR 2024
Visual input example, Extreme Ironing:
Soow triss fow hanwersa www.--ooeeeeelaaaeeeeeoeo. wwtp:///www////////e////////// www.e//e.e.........eeeee.eeeeee. 1pe
User
What unusual about this image?
DREAMLLM (Ours)
The unusual aspect this image is that a man ironing clothes while sitting on the trunk of parked traffic. This unconventional because ironing clothes typically done indoors, on flat surface, and away from busy streets where cars are pussing by. The man's actions ironing clothes while on the trunk car and amidst traffic create an unusual and interesting scene.
[Start new conversation. and clear history] What's 's happening the scene?
User
DREAMLLM (Ours)
In the scene, man is ironing clothes while standing on the back of a car on busy city street. The street filled with traffic and the exposed close moving vehicles. This unusual activity of ironing clothes while standing on an sutomobile also draws attention to the man's determination to accomplish his task despite the challenging urban enviromment.
dangers of being
User
What unusual about this image?
LLaVA (Láu et al., 2023c)
The unusual aspect this image is a man ironing clothes on the back of minivan or would usually iron clothes in a more stationary and safe location, such as home, using regular ironing hoard. scene depicted in the image is peculiar as involves makeshift ironing setup on vehiele, which can be both unsafe and unconventional. Additionally. is not cleur how the man is able to maintain halance and stability while ironing clothes in such an unstable environment.
This not typical place perform this activity,
[Shart new conversation and clear the history] scene?
User LLaVA (Liu et al,, 2023c)
What's 's happening
The image depicts holding a portable ladder, seemingly ironing clothes while standing on the vehicle. Around the scene, there are other cars, traffic light, and fire hydrant. Additionally. there two other people in the vicinity, one close to the man on the car and another further away. The man on the ladder appears be attracting attention due to his unconventional choice of ironing his clothes on top moving car. The city street around him is bustling with activity, adding to the unique nature of the scene.
standing top of yellow SUV in busy city street. He is
User
What unusual about this image?
GPT-4 (OpenAl, 2023a;h)
The unusual thing about this image that man ironing clothes ironing board attached to the roof of a moving taxi.
User
Can you explain this meme in detail?
BIJI-22 eti 20223)
is sitting the back yellow cab
User
Can you explain this meme in detail?
OpenFlamingo (Awadalla et ul., 2023)
The man is drying his clothes on the hood of his car.
Table 11: Visual question answering example comparison of DREAMLLM to LLaVA (Liu et al., 2023c), GPT-4 (OpenAI, 2023a;b), BLIP-2 (Li et al., 2023d), and OpenFlamingo (Awadalla al., 2023). This table format follows OpenAl (2023a;b).
27

---

Published as a conference paper at ICLR 2024
Visual input example, Chicken Nugget Map:
wart trom spacs innrred how bo
User DREAMLLM (Ours)
Can you explain this meme in detail??
The meme presents humorous visual in which close-up of plate of fried chicken. specifically chicken nuggets, uppears to be map of the world. The context suggests that someone has made this map by arranging the chicken nuggets resemble and outlines of various countries. The image is accompanied by a caption that reads, "Sometimes I just look at pictures of the earth from space and I marvel at how beautiful is all is." This hints that the person who created the "map" might have been inspired the beauty of our planet from space, and they have used the similar visual pattern to create an amusing and creative representation
shapes
woeld using chicken nuggets.
User
Can you explain this meme detail?
LLaVA (Liu et al., 2023c)
The meme in the image is creative and humorous take on food, with a focus on chicken nuggets, center
universe. The meme begins with the phrase "Sometimes just look at pictures of the Earth from space and marvel at how beautifull it all is... The main image features tray of cooked chicken nuggets, with the chicken pieces positioned to resemble the continents and islands world map. The chicken nuggets represent the Earth, and the various locations depicted in the photo are actually chicken nugget versions different places. The punchline in smaller text at the bottom, reading "I mean, it's not the real Earth, but how beautiful it is all is." This lighthearted and entertaining meme combines food and humor, using the image chicken nuggets create funny and imuginative representation of the world.
playfully suggests that the
meme appears
User GPT-4 (OpenAI, 2023a;b)
Can you explain this meme?
This meme is a joke that combines two unrelated things: pictures of the earth from space and chicken nuggets. The text of the meme suggests that the image below beautiful picture of the earth from space. However, the image is actually of chicken nuggets arranged to vaguely resemble map
world. The humor this meme comes from the unexpected juxtaposition of the text and the image. The text sets up expectation of majestic image of the earth, but the image is actually something mundane and silly.
User
Can you explain this meme detail!! sometimes just look pictures of the earth from space and marvel how beautiful it is
BLII-22 (Li et al., 20234)
User
Can you explain this meme in detail?
OpenFlamingo (Awadalla et al., 2023)
It's a picture chicken nugget on the International Space Station.
Table 12: Visual question answering example comparison of DREAMLLM to LLaVA (Liu et al., 2023c), GPT-4 (OpenAI, 2023a;b), BLIP-2 (Li et al., 2023d), and OpenFlamingo (Awadalla al., 2023). This table format follows OpenAl (2023b).
28

---

Published as a conference paper at ICLR 2024
Table 13: Training recipes for DREAMLLM. The three training stages are introduced in Section 3.2.
Stage I: Alignment training, Stage II: T-GPT pretraining, Stage III: Supervised fine-tuning.
Stage I
Stage II
Stage III
Config
Alignment
Z-GPT
SFT
Training Hyper- Hyper-Parameetrss Parameters
Optimizer
AdamW
AdamW
Adam W
Leurning Rate
2e-3
2c-5
4e-5
Weight Decay
0.0
0.0
0.0
Training Epochs
1
1
3
Warmup Ratio
0,003
0.003
0,003
Learning Rate Scheduler
Cosine
Cosine 8
Cosine 8
Batch Size Per GPU
8
Maximum Token Length
2048 X
2048
2048
Unfreeze LLM
Training Data
① LLa VAPretrai (558K)
① MMC4 (2M)
LLaVAnnttreet (80K/665K)
LAPOO (8M)
@ BLIP-LAION (2M)
2 InstructMMM4 (20K)
Dataset
LAION400M (1IM)
(20K)
L.AION- COCO (1IM)
Data Size
30M
4M Interleave/Pair
120K
Data Type
Pair
Instruction
Training Cost
GPU Device
128 x NVIDIA A800
128 NBVIDI A800
128XNVIDIA A800 ~=.5h
Training Time
~6h
~ 11hh
C IMPLEMENTATION DETAILS
C.I TRAINING DATA & PARAMEERRS
In Table 13, we list the detailed training dataset usage and hyper-parameters. The training data constructed based on the following datasets: a) LAION400M (Schuhmann et al., 2021), b) LAION- COCO (Schuhmann et al., 2023), c) MMC4 (Zhu al., 2023b), d) BLIP-LAION (Li et al., 2022) which is filtered and caption by BLIP (Li et al., 2022), e) LLaVAPretrain (Liu et al., 2023c) which contains 558K image-text pairs from BLIP-captioned CC3M (Sharma et al., 2018). SBU (Ordonez et al., 2011), and LAION400M filtered by LLaVA, f) LLaVAInstruct, which contains 80K/665K visual instruction-followinng data constructed by LLaVA (Liu al., 2023c) and LLaVA-1.5 (Liu et al., 2023b), and g) InstructMMC4, which is our instruction-followin interleaved document generation data curated by prompting GPT-4 to generate instruction based on the text contents of MMC4. h) Instruct-BLIP. BLIP-LAION, which is our instruction-followin image synthesis data. Similar to InstructMMM4, it is curated by prompting GPT-4 to generate instructions based on image captions. Unless otherwise specified, randomly sample the indicated number of instances from each dataset during the training process.
C.2 DREAMLLM MODEL
Language Model We use LLaMA-I (Touvron et al., 2023a) trained on ShareGPT (Zheng et al., 2023) as as the default LLM (i.e., Vicuna-7B' (Chiang et al., 2023)) following Liu et al. (2023c) to endow its instruction-followingg capacity. During training, we use Flash Attention (Dao et al., 2022) and PyTorch FSDP (Zhao et al., 2023b) accelerate training efficiency.
Visual Encoder The visual encoder is the publicly available OpenAI CLIP-L/14 (Radford et al., 2021) model, which is frozen during the whole process. The images are resized to 224 resolution to align with the CLIP pretraining settings, resulting in a sequence of 256 total tokens for each image. Following prior VL practice (Lu al., 2019; Liu et al., 2023c), we append special <IMG> token before the image sequence and special <IMG/> at the end of the sequence.
Vieuna-7B v1.1: https://huggingfacceeo ca/
29

---

Published as a conference paper at ICLR 2024
Table 14: Overall descriptions of the evaluation benchmarks for evaluating capabilities, including VL comprehension, content creation, and natural language processing (NLP).
Dataset
Task description
Eval Split
Metric
COCO (Karpathy Fei-Fei, 2017)
Scene description
test
CIDÉr (Vedantam al.,,,113 2015)
Creation VL C..pprreeensii
Image2Paragraph (Krause et al., 2017)
Scene description
test
CIDEr (Vedantam 2015)
VQAv2 (Goyal et al., 2019)
Scene understanding QA
test-dev
VQA Acc (Antol et al. 2015)
OKVQA (Marino et al., 2019)
Extemal knowledge QA
val
VQA Ace (Antol ct 2015)
VizWiz (Gurari
2018)
Scene understanding
test-dev
VQA Ace (Antol ul., 2015)
TextVQA (Singh et al., 2019)
Text reading QA
val
VQA Ace (Aniol et al., 2015)
MM-Vet (Yu et al., 2023b)
Multimodal Comprehension
GPT4 Eval (Yu et al., 2023b)
MMBench (Liu
2023c)
Multimodal Comprchension
GPT-3.5 Eval
et al., 2023e)
POPE (Li et al., 20230)
Visual Hallucination
Acc, Fl-score, Recall, Precision
MS MS-COCO (Lin al., 2014)
Text- Text-Condiee Conditional Image Synthesis
val-30K
FID (Heusel et al., 2017)
LN-COCO (Pont-Tuset et al., 2020)
Text- Conditional Image Synthesis
val
FID (Heusel et al., 2017)
MMC4 (Zhu et al., 2023b)
Doc Conditional Image Synthesis
held-out
FID (Heusel et al., 2017)
SIQA (Sap et al., 2019)
Commonsense Reasoning
dev
Acc
PIQA (Bisk et al., 2020)
Commensense Reasoning
dev
NLP
HellaSwag (Zellers et al., 2019)
Commonsense Reasoning
dev Acc
WinoGrande (Sakaguchi et al., 2021)
Commonsense Reasoning
dev Acc
BoolQ (Clark et al., 2019)
Reading Comprehension
dev Acc
MMLU (Hendrycks et al., 2021)
Aggregated Comprehension
test
Diffusion Image Decoder We adopt SDv2. (Rombach et al., 2022) trained on 512x512 resolution as the default diffusion image decoder. Same the visual encoder, the SD model is frozen without any modifications or training throughout the whole process. When constructing the SD target to compute the MSE loss, we resize the images to 512 resolution fit its pretraining configuration.
Dream Query We use dream queries to gather semantic context from MLLMs as introduced before in Sec. 3. Without specifications, use 64 learnable query embeddings. It is both efficient and effective in generating high-quality images. In order to predict when to generate images, we also introduce the special <dream> token, which is appended before the dream query sequence. A <dream/> is appended at the end of the sequence, similar to image inputs.
Classifie--Free Guidance Classifier-free guidance (CFG) (Ho Salimans, 2021) has been demon- strated successful in generating photo-realistic contents at the cost of acceptable generation diversity, seş (xa,t, t, 0), where @ is a special "empty" condition representation and s is the condition scale. The larger guidance scale generally improves image authenticity while decreasing diversity. We only adopt CFG during inference, and the scale is set to 7.5 by default and 2.0 for MS-COCO text-conditional image generation.
This technique modifies the objective by é := (1 ()E (x,,t, C) (
C.3 EVALUATION BENCHMARKS
Systemic evaluations of DREAMLLM regarding VI. comprehension, content creation, and NLP capabilities have been conducted. See the used benchmarks and datasets listed in Table 13. During the evaluation, we use the prompt templates listed in Fig. 9.
D ADDITIONAL RELATED WORKS
Large Language Models A flourishing era of Natural Language Processing (NLP) driven by LLMs is being experienced, with the parameter size growing over 100B according to the scaling law (Kaplan al., 2020). The GPT series of models, starting with GPT-1 (Radford al., 2018) and followed by GPT-2 (Radford et al., 2019), made significant advancements in few-shot learning by scaling up the number of parameters to 175 billion GPT-3 (Brown et al., 2020). This breakthrough garnered a lot of attention and paved the way for further research and development in the field. Since then, researchers have focused on developing LLMs by improving the scaling strategy. Several notable efforts include Gopher (Rae et al., 2021), GaLM (Du et al., 2022), FLAN (Wei et al., 2022a), Switch- Transformer (Fedus et al., 2022), Chinchilla (Hoffmann et al., 2022), and PaLM (Chowdhery et al., 2022). Besides, instruction- based tuning techniques are explored for aligning with human prefer- ences (Christiano et al., 2017: Ouyang et al., 2022). Such success of LLMs has been further solidified by the production release of ChatGPT (OpenAI, 2022) and the highly anticipated GPT-4 (OpenAI,
30

---

Published as a conference paper at ICLR 2024
2023a;b). Meanwhile, in the community, the open-source LLMs are achieving remarkable progress in language capabilities compared to their close-source counterparts. For example, OPT (Zhang et al., 2022), BLOOM (Scao et al., 2022), GLM (Zeng et al., 2023), LLaMA (Touvron et al., 2023a:b), and Falcon (Penedo et al., 2023) all raised great attention and are heen widely deployed. Other methods attempt to learn from distillation, such as Alpaca (Taori al., 2023) and Vicuna (Chiang et al., 2023).
Text- Coxt-Conditinall Content Creation with Diffusion Models The recent surge in Al g-geerrtted content (AIGC) has been primarily driven by diffusion-based methods, particularly in the realm of text-conditional content creation. Saharia al. (2022) have achieved astonishing advancements in high-resolution image synthesis through large-scale pretrained language models and cascaded DMs. Another paradigm, such as SD, focuses on latent spaces and demonstrates superior efficiency and performance (Rombach et al., 2022; Ramesh et al., 2022; Peebles & Xie, 2022; Podell et al., 2023). Recently, Lian et al. (2023) propose to enhance the reasoning capability by constructing layouts with LLMs. DALLE-3 (Betker et al., 2023) leverages LLMs agents and proposes to generate images by incorporating GPT-4 for providing high-quality and detailed prompts that facilitate image synthesis. Motivated by the great success in 2D, a series of works have significantly propelled the 3D synthesis development (Mildenhall ct al., 2022; Liu et al., 2023d; Lin et al., 2023; Wang et al., 2023c; Tang al., 2023) based on Score Distillation Sampling (SDS) (Poole et al., 2023; Wang al., 2023a) that utilizes pretrained 2D DMs. For text-to-video/4D synthesis, the expansion of pretrained spatial to a trapoalltempora factorized U-Net with joint image and video data training has yielded significant success (Ho et al., 2022a;b; Singer et al., 2023a:b).
E LIMITATIONS, FAILURE CASES & FUTURE WORKS
Limitations While DREAMLLM has made significant strides toward the development of versatile, creative, and foundational MILMs,, it still has several limitations.
Model scale. The primary constraint pertains the scale of the LLMs utilized. Current evaluations mainly employ 7B LLMs as the base model, and despite the impressive results garnered, the potential benefits of larger model sizes, such 65B or 130B (Kaplan al., 2020), are worth future exploration.
Training data. The second challenge relates to the quality and quantity of training data (Jia et al., 2021). As the model size and capabilities scale up, a corresponding increase in data is crucial. However, the procurement and refinement of high-quality training data present substantial logistical and financial hurdles. For instance, the open-source interleaved dataset MMC4 contains significant amount of noise in the form of text and images, like commercial advertisements. This noise could adversely affect the model's output language and image style.
Prompt sensitivity. The sensitivity of LLMs to human prompts is a known issue (Wei et al., 2022b; Wang et al., 2023b; Zhou et al., 2023), a challenge that extends to MILMs. For instance, MLLMs' propensity for detailed responses necessitates tailored prompting to elicit concise and short answers, which is particularly useful when addressing Visual Question Answering (VQA) tasks.
Failure Cases The main failure cases of DREAMLLM are observed for multiple image-based content creations. For instance, when presented with two images and composite instruction such as "A and B", DREAMLLM sometimes generates a single subject that amalgamates the characteristics of A and B. This output aligns more closely with the directive "A like B". This phenomenon is not unique to DREAMLLM, but is also observed in specialized compositional generation methodologies, such as StructureDiffusion (Feng et al., 2023; Chefer et al., 2023). This recurring issue may be attributed to the inherent complexity of compositional generation tasks, compounded by the severe scarcity of data specific to this domain.
Future Works As simple and general multimodal learning framework, our future work aims to enhance the DREAMLLM framework by integrating fine- fine-grained visual comprehension via methods like precise referring instruction tuning (Zhao et al., 2023a). We also plan to expand beyond visual and linguistic content comprehension and generation. Several promising research directions include:
Exploring applications of in in-context generation capabilities of DREAMLLM to complex tasks such image-to-image translation (Isola al., 2017; Zhang al., 2023e;d; Parmar al., 2023).
31

---

Published as a conference paper at ICLR 2024
System Prompt
A chat between curious user and on ortificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
(a) Image Captioning (Shert)
USER: Based on the image, give the inage coplion brielly.
<IMAGE>
Please summarize object in one sentence within words.
ASSISTANT:
image depicts
<ANSWER>
(b) Image Captioning (Long)
USER: Based on the image, please describe the inage in detai.
<IMAGE>
Please describe the image in detail.
ASSISTANT:
The inage depich
<ANSWER>
(c) VQA (Short)
USER: Based on the inage, please answer the question.
<IMAGE> <QUESTION>
Please provide an accurate anower within one word.
ASSISTANT:
The answer is:
<ANSWER>
(d) VQA (Long)
USER:
exom, plecse answer according to image question.
Please provide accurate and detailed answer.
ASSISTANT:
<ANSWER>
(e) Visual Hallucination
USER:
image, please objectively accurately
object exils. <IMAGE>
image?
ASSISTANT:
The onswer in
<ANSWER>
Figure 9: Prompt templates. (a) Short image captioning includes COCO captioning, and (b) long image captioning includes ParaerPar..raph. (c) Short VQA includes VQAv2. VizWiz. OKVQA, and VAA (d) Long VQA includes MMBench and MM-Vet. (e) Visual hallucination includes POPE. <IMAGE> is the image representation, <QUESTION: INN> denotes each specific question, <ANSWER> the generated answer, and <OBJECT> is a specific object name POPE.
The extension of the DREAMLLM model to video data possessing long-term term context is natural, capitalizing on the model's ability to encapsulate temporal knowledge through context consistency. This would give rise to video comprehension and text-to-video generation capabilities.
Utilizing DREAMLLM's 's context consistency feature for geometry geometry-preserving preserving tasks, including 3D content creation (Poole al., 2023; Qi et al., 2023b; Liu et al., 2023d), representation learn- ing (Dong et al., 2023; Qi et al., 2023a; Zhang et al., 2023a:f), scene comprehension (Zhang et al., 2023b; Hong et al., 2023), and embodied artificial inteligence (Ichter et al., 2022; Qi et al., 2024).
addition, since the diffusion model used as the image decoder, it would be interesting to explore DMs-based zero-shot perception such as classification (Li et al., 2023a) and segmentation (Xu al., 2023a) with the goal of unifying all perception tasks with image synthesis/generation.
Striving to achieve unified multimodal zero-shot generalist by extending the scope various modalities using techniques such as ImageBind (Girdhar et al., 2023) and exploring content creation models other modalities like audio (Kong et al., 2021).
32

---

Published as a conference paper at ICLR 2024
Oae gise Atore intreducticn thip envie?
Se yau feal Cangercus the piotore?
Godfasher,
larye pasnied lsper sarssig of colired reols and stsses, shick is piece of artuerd natker than a real sciest. iseninate objest, the seper does iisie the rggessnistion cepartant frughtewing
feericon deam Jiin directes by Preecto Pors Csspote, It ts the first teacellmnt in The Codfatker inslegy, whrch slav mecistes The Soljather Per 12 (1976) The dagfatter fort
dlrger- siger thes this wank of erd is mod tupabic of enonyy hare.
aciely regarded as unsterpiace and misziase is the eafia geset, exploring the story of the Orrisons family ffcttsoal dnerica crize fartly follows tranaorrettin Corterne, the gurayea on of the famby's gging dim,
tito Cerizons, as be rises ts pazer and eosstnally tabes the cantle of kia father. The file festores sirony
Ahat
painting!
Snosse
Corteres, faceso az Richeet Cirisens, Juess tao foory for ioone, Astert Desiii ar Fon Begen, and Stens Ecatos fup Diilone, The donjalher to hove /er sis scente aesses, necerable tines, dsllgzood film iedasiry.
The of this pointing is "Stri acth Peart Surving."
tepact
gise intreductisn this painting!
food. Phoo we en slinstrration of panda starvey sust and iesnting the esesios lste the Sel/sther.
"Pee usts Frori Sarrieg" is psintony by the Firmerr 29ta unntery. The patnitng /esurr e pirl acoring a peert carrig ou a Aeotscarf, with ber Acot rstted to the ztde. The wee of light and skodew, will the vatriculs the surreunding avtrenentt are choracteristss of Foreser's 's ssine passting style.
As silentratson of panda starveg dresking earties ltke frd/athur.
Photogmuph
Vhat anctionst safur eas thiz store brieg sencetye
DREAM
The lunug disrs, 2ith viz ahtmriesi crterjul interisr and deoaying away of Proseg serchandtse, ce tring seus of 29, seeder, and sazlaigi to kumn scciety. falking isto the zisre exstar faetagge accctenstt and
Phocograph e: DREAM
Diancy frand This antisaal vslae on slao pronte sengrastass and crtalimiy, so dnidees can ongaee sctt the cercheitize is pretecting and plagito cut theer fuarets timey sterees. The store prenctes unique and splcftcag stepping esprrstzaa beth fastorsng jostisve meeting oveanable personal uppriisetss and interesitos.
Phat is urctten theze shotographa? What difference these plocest is detett sbout the differeace writtee words and stales.
childven
Frstarsse: Senerate som kybopality Eliastretsoes threny-threet cxrchattia
In the the photegragha, A and 8t the vestern code Sholograph
"Brear" to writtes differentiy, pososkly mist a blee and pisk baotgrened, shite Photopraph the word appears statige style. The difference in the style of the werd ost the bacigrond vaayests that each shotograph provesla sifferent approsch, pozachts atpnttydeg defferent thanes. Anticionally, the presence of a "Ile" it Photograpt E supgente vorietton is the corgrred only baway Photograph J.
is. onage of a bay of Dianey trandst nerchorzize, utth pecture of
stoney chanscter baskgesund.
about bég Diasey loke Nickey Rosau?
Figure 10: Qualitative examples of multimodal dialogue between human and DREAMLLM. Various modalities can be used as inputs or outputs, and multi-round dialogue is shown.
33

---

Published as a conference paper at ICLR 2024
Photograph of axs
Photograph of son B:
Rho is the in photo and aho vs the in phodo B?
eon photo Albert Finatein, and een photo Lsonel Hesai.
Imagine what kind of conversation will there tuo people have shen they sst in the cane room? Please give posssble dialogue script.
If dibert Ssnstein and Lionel Meses vere to sst is the same roon, they might have faecsnating consersation about their experiences, knouledge, and perapentives on varsous topses such as accence, phyases, and their respective sports. Bere is pozssbie dinlogue script:
Esnstein: Good afternoon, Sr. Meees. Hou are you?
Hello, Dr. Einsteín. doing weli. you for hoving se.
Esnstein: pleases escounter fellow genise world of sporte, could your tnsights gaze.
Messs: Thank sin, As understanding of the gaze, and the ability to adopt to different aidustions. try lears from eoch game and work on my ueoksesses.
player, feel that the key to success va having strong mindses,
Esnalein: Indeed, have said sistiar things sbout the satare of intelligence and knoutedge, Learning from past experiences Ve crucisi to personal grouth and understsnding. It reminds me of fanous guote by the Gerwas phitosopher Johasa Gottléeb Féchte: "If you would be a genina, think deeply and tong, and incessantly about one thing.
Meass: understond value of costinson Learning, and try to apply st én my tracsing and the fseid. aluoya trying isprous ay skills teare froe ay colleagues
club.
fseid. ay opsnion, passson sntegral part of very sepressed by your passson greatness, "The passion for
Esnsteis: That wonderfu? attitude have.
It fuets and espaciatly evident is my field payakea. For Kustasoe, truth is the foundation for the scsences."
Meass: It is great feetiig to know that I as playing sport that full of passtonate sndiusduats. I on proud to be part of that world and do part in contrebuting to st.
Finslein: I as sure that gos, Hr. Nesss, would agree that having passion for knoviedge can lead to dsacoveries that change the wortd. For exopple, my theory of relatiesty has had profosnd sepact os our undersianding of space, time, and groviby. was result of decades of conterplation and research.
world of sports and uorld of sesesce are not so different uhes comes to parsuct escellence.
vn presence of genvas you, Dinstein.
Finsteis: feel way, and would like thank you for opportsaity engage conversation about respective fields. The world aondross place, gratefus
of
Figure 11: Qualitative examples of multimodal dialogue between human and DREAMLLM. Various modalities can be used as inputs or outputs, and multi-round dialogue shown.
34

---

Published as a conference paper at ICLR 2024
you explosn why fonny. about carefullg stop by step.
Leddy bear unerpented and vourtal environtent for staffed bear. Typsoelly, tesdy hears found sndoors, chsil''s room or on cosch. Sussesng poole are associated usth hanas actsustses, so seesig a reddy besr pool evokes sense of heeer, as it's as omesing justaposision of the bean's typically stonsmate and non oparte husan- centered environnest of post. Second, the allame
image fweey for
sovering posi,
natare
holding up one sto hando, itself from drouning, adding to the humar. fastly, the fact that the bear is auieming with anile sha face, as farther contribstes to the crusing ssture of the acens. The combination of the unfaniliar enveroneent and the bear's expresssone create Janny sed ensertainsing smage.
could interpmated
geatare of surprise
enjoying staeff.
Vou. That's aill hoppen if Leddy bears sais. IncInde sone ligh-gpality ixagea.
you urite article seddy bears suimsing?
who
If toddy bears can sats, it's tskely that they sould try to sinic hansn behavior in the uater. They wolld libely float their backa, sopoasible
certais their luga, eves ahonid't 't ienginel Tmegsne longhing uith deleght ar it floate on sta back the uster. Imagsne the scene usth Lonon drop poot fiest and a cusn cap on teddy bear and et sould be eves more adorabie. In the seage, you'l3 11 see broun teddy bean searing cap, flosting
their breath underuster. of course, 6e's oith teddy hear fleating is ailuation, thst dosse't awarsng cap and goggles, and
teddy bears woild
you
auieming pool.
smage amaing The seddy besr flosting back the and what's and yellow shirt wsth lorge broun beily. This nakes the bear look even more forry and cute, and even more likely to sake you awste.
feelinn of joy Aappsness seyone smage. the bear searing
That 's snteresting. So, shet do you tlsnk a resl bear Looks iake? Show ee high-quaiity 0SLR photograph of bear de forest.
bear, such btack. The exact color can sory depending the species bear and the regson where d2 Isves. Bears have chick for and are ussally large and essullr with atvong Ideda, abich they ase for eliabing, digging, and foraging for food. the seage, you'll
black
typicaily shaggy hoir
biond,
in forwat, aslking through the
The taken 0018 absch allous for lesel detsil, smage. The bean appeors to be starsng seto the distance, focussng consthong the distence. The smage captures the bear's naturai habitat, absch is a prins esapple of the forest envtroneent uhere these anteats image hightights bear's ls sise atrength, along with wetased oild nature. photogruph excellent asuppl of wildlife photography, eaptoring real-lifa that bath usauill straking and informative.
and
Figure 12: Qualitative examples of multimodal dialogue between human and DREAMLLM. Various modalities can be used as inputs outputs, and multi-round dialogue is shown.
35

---

Published as a conference paper at ICLR 2024
(a)
small bard with a pale yellow underside light broun creun and back gray tasl and usng tops
top of tail feather yellow black eyes and black strip eyes
(b)
anachaár the shape of avccado
(c)
riding {"An ghotorealsstii style", Art", "as charcoal skatek", "as golden relsef")
style of Pop
8858888
punda nnd scien/ist misssgg sparkling chemicals, art station
obere flke verv scene", "fraw sonl")
(e)
futuristic esty where I vaporuave style", "made of uster"" "Besjing opera style")
("Sn synthouve style",
robots sedctating dn siparsena netreat
Figure 13: DREAMLLM text-conditional image generation examples with prompts from (a-b) DALL- E (Ramesh al., 2021), (c-d) DALL-E (Ramesh et al., 2022), (e-f) GLIDE (Nichol et al., 2022).
36

---

Published as a conference paper at ICLR 2024
(a)
scuipture duck where X
("made out of transparent: glass"), "lade of woo!", "made of wood", "made of paper"))
(b)
coople of glasses are setting ds a table.
squirret ts enstde giant bright shiny crystsl hall in on the surface of hise acsas. There feu clouds
An art gallery Kaspiaying Honet paintings, The ant gallery ss flooded. tobots are going arosnd the gollery using paddie bourds.
oll-m-cenoaz passting blue furpy bright yeltew
retos the explocteg meceting assels blse,
ztora
(d)
acetand villege acta Connecting narth with fisee-like the ssayiey dearse
sky
rolling biue hills.
Long wsndee beach Aod balloon londing frost of
fseld
("Ixtenbul", "Auxtin"" "Besying", "LA")
heat--ap track at the hase of the Great Pyranid
wonden dack merlonkinn conntain saileg
Figure 14: DREAMLLM text-conditional image generation examples with prompts from (a-c) Imagen and Draw Bench (Saharia et al., 2022), (d-f) Parti (i.e., PartiPrompts or P2) (Yu et al., 2022b).
37


Reviews:
[{"review_id": "YqLz0OxUjV", "reviewer": "Reviewer_h8i6", "summary": "The paper introduces \"DREAM LLM\", a novel learning framework designed to enhance Multimodal Large Language Models (MLLMs). The model is designed to 1) The model late-fused three pretrained models, a text-LLM, an image generation model (SD), a CLIP model. Unlike conventional methods that utilize intermediate image representations, DREAM LLM leverages score distillation techniques with a diffusion image generation model inputs raw data modalities and outputs them in the same format. 2) The model is also trained on interleaved multimodal corpora sourced from the internet. \nThis leads to superior performance in several benchmarks and the ability to generate free-form interleaved content.", "questions": "1. Can the model do k-shot learning for image understanding task?\n2. table4 shows amazing results that the model is better at text task than the pretrained text LLM, does the author have any hypothesis why?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The model late-fused three pretrained models, a text-LLM, an image generation model (SD), a CLIP model. With the designed dream query and score distillation with SD model, the model is superior than earlier models that utilize intermediate image representation. \n2. What's more, the model is trained on carefully collected various datasets, including, various text-image datasets, interleaved multimodal corpora sourced from the internet, instruction datasets. With joint-training, it also shows synergy of text and image, understanding and creation. \n3. The model shows very comprehensive experiment results in various benchmarks like MS-COCO, MMBench, and MM-Vet, in different setup zero-shot understanding, image generation, interleaved generation, etc.", "weaknesses": "1. The model is trained on very rich data, and it is not clear how does those data contribute to the zero-shot evaluation. \n\na. several dataset used by the model are derived from COCO datasets, e.g Laion-COCO, LLaVaInstruct, etc. How do we know if there is data leakage in the training datasets. This applies to the results in table 1, as well as in table2. \n\nb. Is the model in table 1 after instruction tuning? \n\nc. if you continue training SDv2.1 with the collected dataset, what are the MS-COCO, and LN-COCO FID number?  \n\n2. Some model details are not clear. \n\na. how is the multi token dream query implemented. For decoder-only transformer training, every token needs a loss (or a score). What us the loss of each query token during training, and are they generated sequentially or altogether during inference?\n\nb. in the interleaved multimodal joint training, for the second/third image generation, do they condition on both the image1 dream query and image1 visual encoder? Or just image1 visual encoder. \n\nc. for the stage1, stage2, stage3, what's the final loss? Do they both have L_DM (formula 5) and L_MLLM (formula 6)? Is there a weight? \n\nd. for the I-GPT training, does the visual projector, condition projector, dream embedding get updated? or only the MLLM transformer got updated?\n\ne. in section 5.1, the L_CLIP is not clear. Which two embeddings are used to calculate the loss?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lq5NctUOaD", "reviewer": "Reviewer_t2eT", "summary": "This paper proposes a framework to allow multimodal large language models combining multimodal comprehension and generation. To enable image generation, instead of fitting CLIP feature, this work directly optimizes the diffusion model's objective to achieve modeling multimodal posteriors. The training pipeline is comprised of three stages: alignment pretraining, interleaved generative pretraining, and supervised finetuning. Extensive experiments on multimodal comprehension and creation have shown the superiority of the proposed model.", "questions": "Will including samples from training datasets as in-context examples improve the performance during evaluation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This model proposed a unified framework for joint multimodal comprehension and generation which shows impressive performance on various tasks, demonstrating the benefits of synergizing these two tasks.\n2. The usage of score distillation avoids the possible information loss to greatly improve the image generation ability.\n3. The proposed training pipeline enables the free-form interleaved generative ability of multimodal models.\n4. The experiments are comprehensive and convincing.", "weaknesses": "1. For free-form interleaved generation, it is important to ensure the consistency between related images. However, as we can view the model as replacing the CLIP text encoder of stable diffusion with a much more powerful LLM for the image generation aspect, the control of the generated image is still limited. As we can see from the Figure 3, the phones in generated samples have large discrepancy.\n2. The paper does not demonstrate the in-context comprehension ability of the model.\n3. Ablation studies on the choices, combination ways, and the importance of filtering process of the training datasets are not shown, which might provide insights for future study.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o0PXDyZuPH", "reviewer": "Reviewer_jWvU", "summary": "This paper proposes a framework that unifies generation of text and images. Specifically, it utilizes a newly introduced <dream> token to encode the representations that will later be forwarded to the diffusion model to decode to images. The authors have conducted extensive experiments across multiple tasks and benchmarks to showcase the ability of the proposed method.", "questions": "See weakness .", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper proposes a unified and promising framework for multimodal generation, strong performances are reported. The proposed approach to integrate diffusion models with LLM seems reasonable and could inspire following works in this area.", "weaknesses": "The major concern I have is the necessity to utilize the token from LLM for image decoding. What is is going to be if you let the LLM to first output the image description, then extract it and feed it directly to the diffusion model? In this case, the original text encoder of diffusion models are leveraged. The results in Table 2 show that the specialists still outperform dreamLLM, which means the above naive alternative could potentially perform better? In addition, is it possible that directly use output text can also alleviate the loss of LLM's original power?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)