Emu: Generative Pretraining in Multimodality
Authors: Quan Sun
Qiying Yu
Yufeng Cui
Fan Zhang
Xiaosong Zhang
Yueze Wang
Hongcheng Gao
Jingjing Liu
Tiejun Huang
Xinlong Wang
Venue: ICLR 2024 poster (2024)
Keywords: Generative pretraining
Multimodal generalist
Foundation models
In-context multimodal learning
Primary area: generative models

Abstract: We present Emu, a multimodal foundation model that seamlessly generates images and text in multimodal context. This omnivore model can take in any single-modality or multimodal data input indiscriminately (e.g., interleaved image, text and video) through a one-model-for-all autoregressive training process. First, visual signals are encoded into embeddings, and together with text tokens form an interleaved input sequence. Emu is end-to-end trained with a unified objective of classifying the next text token or regressing the next visual embedding in the multimodal sequence. This versatile multimodality empowers the leverage of diverse pretraining data sources at scale, such as videos with interleaved frames and text, webpages with interleaved images and text, as well as web-scale image-text pairs and video-text pairs. Emu can serve as a generalist multimodal interface for both image-to-text and text-to-image tasks, supporting in-context image and text generation. Across a broad range of zero-shot/few-shot tasks including image captioning, visual question answering, video question answering and text-to-image generation, Emu demonstrates superb performance compared to state-of-the-art large multimodal models. Extended capabilities such as multimodal assistants via instruction tuning are also demonstrated with impressive performance.

Published as a conference paper at ICLR 2024
EMU: GENERATIVE PRETRAINING IN MULTIMODALITY
Quan Sun
Qiying Yua.-
Yufeng Cui
Fan Zhang'
Xiaosong Zhang
Yueze Wang
Hongcheng Gao'
Jingjing Liu
Tiejun Huang!.
Xinlong Wang
Beijing Academy of Artificial Intelligence
Tsinghua University
Peking University
Code & Demo: https://gi http://ggtttb...// httam//aaiiiiiii /Emu
ABSTRACT
We present Emu, multimodal foundation model that seamlessly generates images and text in multimodal context. This omnivore model can take in any single- modality or multimodal data input indiscriminately (e.g.. interleaved image, text autoregressive training process. First,
and video) through a one- me--ooel--or- visual signals are encoded into embeddings, and together with text tokens form an interleaved input sequence. Emu end-to-end trained with a unified objective of classifying the next text token regressing the next visual embedding in the multimodal sequence. This versatile multimodality empowers the leverage diverse pretraining data sources scale, such as videos with interleaved frames and text, webpages with interleaved images and text, as well as web-scale image-text pairs and video-text pairs. Emu serve generalist multimodal interface for both image-to-text and rext-to-image tasks, supporting in-context image and text generation. Across broad range of rero-shot/feww-shot tasks including image captioning, visual question answering, video question answering and text-to-image generation, Emu demonstrates superb performance compared to ofte-of-theeare large multimodal models. Extended capabilities such as multimodal assistants via instruction tuning are also demonstrated with impressive performance.
INTRODUCTION
With text corpus at massive scale, Large Language Models (LLMs) (Brown et al. 2020; Chowdh ery et al. 2022 Touvron al.,
(023) with straightforward training objectives such next-word- prediction learn to understand, reason, and generate text with unprecedented accuracy and fluency, paving the way for diverse real-life applications (Schulman et al. 2022) unthinkable a decade ago.
Recent studies (Alayrac et al. 2022; Driess et al. 2023 Hao et al. (022)
2 further investigate Large
Multimodal Models (LMMs) beyond LLMs. Flamingo (Alayrac et al. (002) connects a powerful language model with a pretrained vision encoder and inserts learnable layers to capture cross-modality dependencies, demonstrating strong abilities multimodal zero-shot and in-contexi learning. Recent
works (Li et al. 2023b Dai et al. 2023 Huang et al. 2023 Liu et al. [203bb] Zhu et al. 2023a Ye let al. 2023 Li et al. 2023a] Gong et al. 2023) adopt this framework and build LMM by docking a vision encoder with an LLM.
The prevailing training objective in such LMMs predicting the next text token (Alayrac et al. 2022. Hao al. 2022 HHuang et al. 2023 Zhu et al. 2023a Liu al. 2023b Li et al.) (023a). typically with a frozen vision encoder and no supervision for the vision part, which highly restricts model capacity. Besides, these LMMs mostly trained image-text pairs or documents, while overlooking video data as a potential scalable source of interleaved multimodal data. Documents interleaved with images (e.g., textbooks, webpages) provide intuitive representation of complex concepts, and have proved to be effective in empowering models with multimodal in-context learning ability (Alayrac et al. 2022 Zhu et al. 2023b). Videos, which usually contain interleaved image frames and subtitles (Figure 3), are an abundant source of multimodal data that naturally contains dense visual signals and encodes stronger cross-modal correlations with text than regular multimedia
"Equal contribution. Correspondence wangxinlong @ bagi.ac.cn.

---

Published as a conference paper at ICLR 2024
Image Captioning
Image Question Answering
Input Prompt
Mom's Far ily Calendar
The image shows photo of
Question: Can you tell me about this image? Answer:
mavie poster "Fight Club," poster stunding to each other, with the words "Fight Club' written hold letters ahove them. The backyround poster ominous, giving impression gritty and intense movie.
Completion
this image the cover of book called "Mom's l'amily Calendar" by Sandra Boynton. The book is designed help moms track of their family's 's schedule and activities.
Input Prompl
Completion
In-comext Completion peetio
There godzälla.
There is chinchilla.
There are three pandas.
This the first image:
This the sccond image:
The animal in finst image surrounded plant the second imuge:
Input Prompt
Completion
Imngge-oo -t- Emage
Input Prompt
Completion
Text-in Image
An elder man is playing guitar, canvas, impressionism.
Input Prompt
Complction
woman video wearing pair Apple VR gongles while sitting in airplane. She appears to be using the goggles to watch something, possibly view of the airplane's surroundings
Video Question Answering
360-degree
Question: woman doing
The unusual aspect of this video that features giraffe standing on diving board and jumping swimming pool. This not a common sight, as giraffes are not typically found swimming pools.
Question: What is umusual about this video? Answer:
Figure 1: Emu as generalist interface for diverse vision-languas applications, such as image captioning, image/video question answering, in-context image-to-text and text-to-image generation, and image blending, More examples in Appendix B
2

---

Published as a conference paper at ICLR 2024
documents. Furthermore, public videos (especially user-generated clips) possess richer content diversity than Common Craw from which current training datasets mainly originate.
In this work, we introduce Emu, large multimodal model that Iearns from both video and image data interleaved with text, under a unified objective of predicting the next visual or text token in an autoregressive fashion. To take advantage of rich web-scale data with an omnivore capacity, formulate diverse sources of multimodal data (e.g.. videos with subtitles, webpages with images and text) into a unified format of interleaved image embeddings and text tokens (videos are converted into an interleaved sequence of randomly-selected frames and subtitles). Specifically, visual signals are first encoded into embeddings via a visual representation model EVA-CLIP (Sun al. (023). instead of being converted into discrete tokens. These visual embeddings together with text tokens constitute an interleaved multimodal input sequence, which will be fed into Emu for training.
We pretrain Emu on these multimodal data sequences under simple unified objective: predicting the next element multimodal sequence. Different from existing LMMs that compute the predict- the-next loss on text tokens only, in training Emu, all input elements including both discrete text tokens and continuous image embeddings are accounted for loss computation. We adopt the cross- entropy classification loss for discrete text tokens, and the l2 regression loss for continuous visual embeddings. As raw images typically lack the left-to-right causal dependency as in language, Emu does not perform image generative pretraining in the original pixel space. Instead, visual embeddings are transformed into causal latent space via Causal Transformer, which accepts the image encodings generated by EVA-CLIP as input, and outputs N tokens that capture the causal dependency of the given image (as illustrated in Figure
Pretrained with the unified objective and diverse data forms, Emu can serve generalist interface for both image-to-text and text-to-image tasks by performing various types of completion in a multimodal sequence. As illustrated in Figure [ Emu accepts multimodal prompts (e.g., text, image, video, or their interlcaved sequence) and generates multimodal response (for image generation, visual embeddings are decoded by fine-tuned diffusion model). Further, Emu demonstrates impressive capabilities such as in-context text and image generation (the 2nd block of Figure D image blending (the 5th row of Figure [ that combines a cat and a tiger into real-looking tiger-cat). video understanding (the last block of Figure D). and real-world knowledge grounding (Section 5.4).
We evaluate Emu on a broad range of zero-shot and few-shot tasks including image captioning. visual question answering. video question answering, and text-to-image generation. For qualitative demonstration, we also build am effective multimodal assistant via instruction tuning on multimodal conversational data. The instruction-tuna Emu assistant can effectively follow human instructions and interact with users via multimodal response.
2 EMU: PREDICT THE NEXT IN MULTIMODALITY
2.1 ARCHITECTURE
Emu is a large-scale multimodal model that performs completion in multimodality, i.e., perceiving interleaved multimodal input and generating outputs varying in modalities. As illustrated in Figure 2 Emu consists of four parts: Visual Encoder, Causal Transformer, Multimodal Modeling, and Visual Decoder. We leverage pretrained EVA-CLIP (Sun et al.) 2023), LLaMA (Touvron et al. (023) and Stable Diffusion (Rombach et al. 2022) to initialize the Visual Encoder, the Multimodal Modeling LLM and the Visual Decoder, respectively.
Given any sequence of interleaved image, text and video, we first encode the image into dense visual features via EVA-CLIP, then transform the encodings into fixed number of N visual causal embeddings via Causal Transformer. Similarly, we encode a video of T frames into T x N visual causal embeddings. Two special image tokens [MGG] and [/IMG] are prepended and appended for each image or frame, respectively, to represent the beginning and end of the encoded image/frame embeddings. The visual causal embeddings are combined with text tokens to form multimodal sequences that are fed into the Multimodal Modeling LLM for unified autoregressive modeling. We append 
 and 
 tokens to the start and the end of each sequence. In inference, we fine-tune the Visual Decoder to decode the visual embeddings into realistic image.
https::/cccmoneeeeec /org/
3

---

Published as a conference paper at ICLR 2024
Encoder (EVA-CLIP)
Causal Transformer

 [IMG]
[/IG]] An emu egg that will hatch into a baby emu [IMG]
[/MG) 

Multimodal Modeling with LLM (LLaMA)
Classification baby emu [IMG]
Regression
[MG]]
An
emu
egg
that
will
hatch
into
Decoder
(Stable Diffusion)
Figure 2: Emu unifies the modeling of different modalities in auto-regressive manner. Visual signals are first encoded into embeddings, and together with text tokens form an interleaved sequence. The training objective is to either classify the next text token regress the visual embedding. In inference, regressed visual embeddings are decoded into a realistic image via a fine-tuned latent diffusion model.
Causal Transformer. Auto-regressively modeling images in raster order is counter- and has demonstrated satisfactory performance, which may be attributed the fact that images naturally possess 2D structures and are not perceived as sequential signals like text. To better capture the characteristics of images and achieve unified modeling of different modalities, we propose a Causal Transformer module to transform 2D spatial visual signals to ID causal sequences in latent space Z. Specifically, given an image 1 with its encodings g(T) from EVA-CLIP as condition, Causal Transformer accepts randomly initialized embeddings {e1, C2...... .,,...,. eN) embeddings [z1, 22, .2, }
input, and outputs N that capture the causal dependency of the given image. The architecture of Causal Transformer is similar to the decoder of Transformer (Vaswani et al. (017). with each block consisting causal self-attention layer, cross-attention layer, and a feed-forward layer. The cross-attention layer aggregates visual information from the image embeddings extracted from EVA-CLIP, where the visual embeddings are treated as keys and values, and the outputs from the previous causal attention layer serve queries.
Visual Decoder. We use a latent diffusion model to decode visual embeddings into images, and adopt the weights of Stable Diffusion (Rombach al.) 2022) for initialization. Specifically, we feed N visual embeddings generated by Emu into the diffusion model as conditions for image decoding. We replace the linear projections of the cross-attention modules in Stable Diffusion with new linear layers that accommodate the dimension of Emu and Stable Diffusion.
2.2 TRAINING ORJECTIVE
Given an unlabeled web-scale corpora D consisting of interleaved multimodal sequences æ (R., where x can be vision-language sequences of various forms, such as image-text pairs, image-text interlcaved documents, or vidcos with subtitles. x can be a signal unit (text or image token) from any arbitrary modality. We first convert all continuous 2D signals (images and video frames) into 1D causal latent embedding sequences using Causal Transformer, then insert them back into the corresponding places in the sequence The resulting sequence is represented as u = (uj U2 ... umm). where u, can be either a discrete text token, or a visual embedding that captures causal dependency with neighboring visual embeddings.
We approximate the likelihood of the web-scale corpora p(x) with p(u), and maximize the likelihood in unified auto-regressive manner follows:
max
log P(u:/ui..... (4 = p(z)
(1)
WED =1
4

---

Published as a conference paper at ICLR 2024
Two types of losses are adopted to optimize this objective. For discrete text tokens, cross-entropy loss is used to supervise classification in the predefined vocabulary with a language modeling head. For continuous visual embeddings, &g regression loss is adopted with separate regression head.
2.3 GENERALIST INTERFACE
The unified auto regressegees modeling of different modalities endows Emu with a powerful ability to serve as a multimodal generalist that can perform any types of completion multimodal sequence, Le., accepting multimodal sequence as input, and outputting signals across vision and language modalitics. For example, given two examples as the prompt, Emu automatically infers and completes the corresponding task given a new input, as shown in the second block of Figure !
Specifically, given multimodal context, if the expected output format text, Emu will use the language modeling head to generate discrete text tokens. If the desired output is image, we will append a [IMG] token at the end of the input sequence, then Emu will autoregressively generate N visual embeddings that will then be sent to the visual decoder for decoding into a real-worle image.
EMU TRAINING
We pretrain Emu with web-scale data across modalities in various forms, including image-text pairs (LAION-2B (Schuhmann et al., 2022), LAION-COCO (lai) b)), interleaved images-text data (MMC4 (Zhu al. 2023b)), video-text pairs (WebVid-10N) (Bain al. 2021)), and our collected in- terleaved video-text data YYT-Storyboard-1 IB). All these data are formulated as multimodal sequences, from which Emu learns under the objective of peedit-thheneet-eeeeeee in unified auto-regressive manner. After pretraining, we finetune decoder to transform visual embeddings into realistic images.
3.1 DATA
Image-text Pairs. We use the image-text pairs from LAION-2B (Schuhmann et al. 2022 and LAION-COCO (lai (b) for pretraining. LAION-2B provides images paired with noisy alt-texts from the web, and LAION-COCO its 600M subset that is captioned by BLIP (Li et al. 2022).
Video-text Pairs. WebVid-10M (Bain al. 2021) is an extensive dataset consisting of large collection of short videos with textual descriptions These videos are sourced from materials websites with diverse contents and a strong correlation between text and video. We use heuristic rules to remove irrelevant metadata (e.g resolution of the original video, camera parameters).
Interleaved Image and Text. Large-scale image-text interleaved data plays crucial role in unlocking the in-context learning ability of multimodal models. We leverage the Multimodal-CS (MMC4) dataset (Zhu et al.) 2023b), an expanded version of the text-only C4 (Raffel et al., 2020p. MMC4 comprises collection of approximately 75 million million images and 38 billion tokens in total. From each document, we sample random subsequence of I.= 1024 take up to the first N = images. Additionally, we randomly sample N = 5 images along with their corresponding sentences to construct a subsequence L L= 512.,
documents, with 400
Interleaved Video and Text. Videos with subtitles also present promising and scalable source of interleaved multimodal data. We introduce the YT-Storybbard-IB dataset which collects 18 million videos and their corresponding subtitles from YouTube using the video-ids provided by the YT-Temporal-1 1B dataset (Zellers et al.) 2022). Instead of raw videos. we collect storyboard images (about 1.8 billion images total), set of thumbnails provided by the YouTube website for quick video viewing. The combination of storyboard thumbnails and subtitles creates a natural interleaved sequence of video and text ordered by timestamps, as in Figure 3 More details are in Appendix A.I.1
3.2 PRETRAINING
We initialize Emu's Visual Encoder with the 1B version of EVA-01-CLIP (Sun et al..
2023). and
Multimodal Modeling L.LM with the 13B version of LLaMA Touvron et al.) 2023). LLaMA is decoder-only Transformer (Vaswani et al. 2017) and EVA-0I-CLIP is a 40-layer ViT (Dosovitskiy
http::///www.yuutube.
5

---

Published as a conference paper at ICLR 2024
1:19
1:21
1:26
1:38
1:15
L:20
1:25
1:30
1:35
the limaale puts the cag stallow pit male bird
covers with lerven for
incubution of
the mule even safeguards the clucks till they uttained
this an cimu egg. which weight 600 800
of
Video Story board Images with Subtitles
Sort by timestamp
Interfeaved Video-text Data
female puts the egg shallow pir the male bind
male eoge
egz
male even safegrards
weight 600 800 grams
protection
chicks till they altained maturity
Interleaved video-text data. The combination of storyboard thumbnails and subtitles captions creates natural interleaved sequence of video and text that is ordered by the timestamps.
Figure 3:
al. 2021). The Causal Transformer comprises 12 blocks, each of which consists of causal self sell-antention layer, a cross-attention layer, and a feed-forward layer. Random initialization used for Causal Transformer. The total number of parameters of Emu 14B and is trained mndto-eend
We use a batch size of 128 for image-text pair data, 64 for interleaved image-text data, 16 for vidco- text pair and interleaved video-text data. Detailed pertaining hyperparameters are in Appendix A.III] For each video, we randomly sample 8 frames for pretraining, and all images/frames are resized into 224 2244 resolution. For image-text pair and interleaved data, we randomly put each image before or after its corresponding sentence. We train the model on 128 NVIDIA 80G-A 100 GPUs for 10k steps with around 82M samples (150B tokens in total), and the pretraining takes approximately 2 days.
3.3 VISUAL DECODING
After pretraining, we tune the visual decoder with both LAION-COCO b and LAION- Aesthetics (lai (a high-aesthetics quality subset of LAION-5B (Schuhmann al. 2022 image-text pair datasets under text-to-image task. Specifically, We initialize the diffusion model with Stable Diffusion v1.5. We freeze the Visual Encoder, Multimodal Modeling LLM in Emu, and the VAE in diffusion model during training, with only the parameters of U-Net updated. For each training sample, append the [IMG] token to the end of the input and feed it into the Multimodal Modeling LLM, which will then generate N visual embeddings in an auto-regressive manner. These visual causal embeddings are fed into Image Decoder as the condition for image generation training.
We follow the model setups of Stable Diffusion v1.5. We train the diffusion model with 32 A100-40G GPUs for 15k iterations. Detailed hyperparameters are Appendix A.2 To further improve sample quality, we randomly drop image embeddings condition by 10% of the time during training to enable classifier-free guidance (Ho & Salimans) 2022).
INSTRUCTION TUNING
Language instruction tuning has helped pretrained language models align with user inten- tions (Ouyang et al. 2022) Wang et al. 2022c Taori et al. 2023 Zheng et al. 2023) and generalize to unseen tasks (Wei al. 2022]Chung
al., (022). We apply multimodal instruction tuning on
Emu to align it with human instructions through supervised finetuning on publicly available datasets,
including language instructions from ShareGPT (Zheng et al. 2023) and Alpaca (Taori et al. (023).
image-text instructions from LLaVA (Liu et al. 2023b), and video instructions from VideoChat (Li ct al. 2023c) and Video-ChatGPT (Maaz et al. 2023). Dataset details can be found in Appendix B.
In instruction tuning, we freeze all parameters of pretrained Emu, and fine-tune low-rank adaption (LoRA) module (Hu et al. 2022). The main focus instruction tuning is to align the model with natural language instructions, which are less relevant to vision features. Thus, attach LoRA modules only to the self-attention layers the Multimodal Modeling LLM, and add no adaptation to the Vision Encoder. Training details can be found in Appendix B.1

---

Published as a conference paper at ICLR 2024
Table Zero-shot comparison, 'indicates that the zero-shot prompt built by using two examples from the task, where their corresponding images have been removed. Emu-I is the instruction-uun -undedd Emu model. The best results are bold and the second best are underlined.
Image: Text Tasks
Video Text Tasks MSRVITQA
Models
coco
NoCaps
Flickr30K
VQAv2
OKVQA
VizWiz
VisDial
MSVDQA
NEXTQA
Per-task Finetuning
PALI-X-55B
149.2
126.3
86.0
66.1
70.9
47.1
38.3
Metal M
82.2
58.7
43.3
41.1
11.4
Kosmos-1
84.7
67.1
51.0
29.2
Flamingo-9B
79.4
61.5
44.7
28.8
30.2
13.7
Emu
112.4
96.5
72.0
34.2
47.4
8.3
19.6
Emu
52.9
42.8
34.4
47.8
34.3
17.8
23.4
Emu-1
120.4
108.8
32.2
Emu-1
62.0
49.2
38.3
51.1
37.0
21.2
19.9
All instruction-tuning data are packed with this template:
<System Message>
[USER]
<Instruction: [ASSISTANT]
<Answer>,
(2)
where [USER] and [ASSISTANT] are special tokens initialized from the embeddings of words 'user' and 'assistant', respectively. <System Message> varies depending the specific task (Appendix [B.2). <Inst ruct ion> and <Answer> are actual slots for human instructions and assistant answers, and only <Answer> accounted for loss computation.
EVALUATION
We evaluate Emu a broad range of vision-language tasks including image captioning (MS-
COCO (Chen et al. 2015)) ), image question answering (VQAv2 (Goyal et al. 2017), 2 OKVQA (Marino al. 2019). VizWiz (Gurai et al. 2018)), visual dialog (VisDial (Das et al. (017)) video question
answering (MSRVTTQA (Xu et al. 2017), MSVDQA (Xu et al.) 2017). NexIQA (Xiao et al. 2021 and text2image generation(MS- COCO (Lin al. 20144). Details are described Appendix C.I
evaluate our pretrained and instruction-tuned models in zero-shot and few-shot settings.
5.1 ZERO- EVALUATION
In the zero-shot setting, the model is tested on tasks and datasets never encountered during training. Task-specifit prompts are used indicate different tasks to perform, without any additional tuning for model parameters.
Multimodal Understanding. Table presents the zero-shot multimodal understanding performance of Emu and Emu-I (the instruction-tuned model). For zero-shot evaluation of Emu, we adopt the multimodal Srain-of-Thoughhe prompting technique following Huang et al. (2023), which first asks the model to generate caption for visual content before outputting the final result. Additionally, we evaluate using the same strategy following Flamingo (Alayrac et al. 2022), where two text-only examples from the task used as prompts (results indicated by an For more detailed information regarding the evaluation, please refer to Appendix
On COCO captioning task, Emu achieves impressive zero-shot CIDEr score (Vedantam al. 2015 of 1 12.4, which outperforms other LMMs by large margin. In a wide range of image and video question answering tasks, Emu consistently surpasses LMMs like Kosmos-1 and Flamingo- Notably, Emu achieves an accuracy of 34.4% on the complex VizWiz VQA dataset, versus Kosmos- I's 29.2% and Flamingo-9B's 's 28. 28.8%. Emu-I the instruction-tune notable improvements. Remarkably, even with only 14B parameters, Emu-l can outperform much larger-scale Flamingo-800 model in several tasks such as VQAv2 (62.0% vs. 56.3%). VizWiz (38.3% vs. 31.6%), and MSVDQ. (37.0% vs. 35.6%).
Emu model that achieves
Text2image Generation. We evaluate the zero-shot image generation ability on the validation set of MS-COCO (Lin al.] (2014). Following (Ramesh et al. (202I), randomly sample 30k prompts from the validation set and calculate the zero-shot FID (Heusel et al. 2017). The results are shown in For the generation of both Emu and SDv1.5.
Table 2
PNDM (Liu at. 2022) scheduler with

---

Published as a conference paper at ICLR 2024
Table 3: Few-shot comparison. k is the number of in-context examples, and used the same example selection approach (i.e. RICES Yang et al. 2022b) as Flamingo (Alayrac et al. 2022).
VQAv2
VizWiz k=4
MSVDQA k=4
MSRVTTQA k=4
Models
A=2
k=4
k=8
k=2
k=8
k=2
k=8
k=2
k=8
Kosmos-1
51.4
51.8
51.4
31.4
35,3
39.0
Flamingo-99
56.3
58.0
34,9
39,4
36.2
40.8
18.2
23.9
PALI-X
56.9
57.1
Emu
56.4
58.4
59.0
37.8
41.3
43.9
36.0
37.1
39.8
21.2
21.8
24.1
Table 4: Zero-shot evaluation regarding each core VL capability of MM-Vet (Yu et al.
2023b).
Model
Rec
OCR
Know
Gen
Spat
Math
Total
LLaMA-Adapter v2-7B (Gao
2023
16.8
7.8
2.5
3.0
16.6
4.4
13.6::0.2
MiniGPT-4-14B (Zhu et al.
2025a
29.9
16.1
20.4
22.1
22.2
3.8
24.4:10..4
InstructBLIP-1445 (Dai cl al
2023
30.8
16.0
9.8
9.0
21.1
10.5
25.6++0.3
DreamLLLM-7B (Dong al.
20231
41.8
26.4
33.4
33.0
31.0
11.5
35.9+-0.1
LLaVA-65B [u 2023
39.2
28.2
26.2
28.3
33.0
15.0
35.5+003
Emu-1-144 -14B
45.5
19.2
36.7
35.9
25.2
3.8
36.311..3
better generation quality. The scaling factor is set to 5.0 and 3.0 for Emu and SDv1.5 respectively, as these settings yield the best performance for both models. Emu achieves better performance compared a concurrent work GILL (Koh et al. 2023a), which also generates images with LLMs. However, model is inferior to SDv1.5 in terms of FID. This is probably because the condition space (image embeddings) of our visual decoder deviates lot from the condition space (text embeddings) of the diffusion model used as initialization, and our model is trained for a relatively short 15k steps.
50 steps. We also adopt classifier-free guidance (Ho & Salimans) 2022)
5.2 FEW-SHOO EVALUATION
Table 2: Zero-shot text-to-iman results on MS-COCO validation set. 30k samples are randomly sampled for evaluation.
In few-shot evaluation, the model prompted with task-specific prompts and several examples collected from the training data evaluate its in-context learn- ing ability, Evaluation details can be found in Ap- pendix Table presents the performance of the pretraining model Emu in image and video question answering tasks under the few-shot (A: 2, 4, 8) evaluation setting. We use the Retrieval In-Context Example Selection (Yang et al., 2022b) approach following Flamingo (Alayrac et al. 2022). Emu demonstrates superior performance to Flamingo-99 and Kosmos-1 under almost all scenarios. For ample, Emu achieves a VQAv2 accuracy of 58.4% and VizWiz 41.3% under the 4-shot setting, surpass- ing Flamingo-9B by +2. 1% and +6.4%, respectively. For video-text tasks, Emu demonstrates strong performance as well, such as 4-shot 21.8% v.s. Flamingo's 18.2% on the MSRVTTQA benchmark. Additionally, we can observe positive corre- lation between the number of shots (k 0, 2, 4, 8) and the performance of Emu. These results demonstrate Emu's 's remarkable in mn-coneext learning ability.
Models
FID(J)
unimodal generation models
GLIDE NNichol al. 2021
12.24
Make- Scene Galni et al
2022
11.84
DALL-E 2 Ramesh et al. 2022
10,39
SDv1.5 Rombach et al
9.93
Imagen Saharia et al.
7.27
Parti Yuu 20226
7.23
multimodal generation models GILL (Koh 2023a
12.20
Emu (ours)
11.66
5.3
IN-THE-WILD EVALUATION
Table presents zero-shot evaluation results on the in-the-wild benchmark MM-Vet YYu et al., 2023bb.
We report the mean and std of evaluation runs following Yu et al. (202bb). For each core capability, the average score is reported. Emu-I exhibits sfate-of-the--n in an-the-will capability, and outperforms LLaVA-65B (Lu et al. 2023) in Rec, Know, Gen abilities and the total score.
5.4
QUALITATIVE EVALUATION
Beyond quantitative benchmarks, we conduct adequate qualitative evaluation of Emu. Emu demon- strates impressive capabilities that cannot be evaluated standard benchmarks, including real-world
8

---

Published as a conference paper at ICLR 2024
knowledge grounding (upper right of Figure interleaved multi-image understanding (left side of Figure 4. detailed video understanding (lower right of Figure 4), multimodal assistant (Figure 5 multi-turn dialogue (Figure ( image blending (Figure Zp. and (in-context) text-to-image genera- tion. For in-context text-to-image generation, Emu can generate context-related images (in the first two rows of Figure 8 the generated images share the oil painting style in context, compared with the corresponding images generated without context in the first two rows of Figure (9). and follow context-related instructions, shown in the 4th row of Figure [ The multimodal in-context ability of Emu is responsible for this brand-new ability of image generation.
We also compare Emu with other state-of-the- multimodal assistants in terms of the ability to perform typical image captioning tasks (Figure[To TO) and follow human instructions (Figure [T] In Figure 11 we test a slightly difficult instruction, and only Emu response properly to list 8 books written by Agatha Christie and then recommend one.
6 RELATED WORK
Multimodal pretraining (Radford et al. 2021 Tia al. 2021 Sun et al. 2023 Chen et al. 2020
Kim et al.] 2021| Wang et al. 2022d a by Cho et al.] [021] Li et al. 2021; Yu et al. 2022a Chen
al., 2023c; Lu et al. (2022) learns pross-modaa interactions from large-scale multimodal data. Flamingo (Alayrac et al. 20222) bridges powerful yet private pretrained vision and large language models and first demonstrates remarkable multimodal zero-shot and few-shot behaviors. With the increasing impact (Schulman et al.) 2022) and accessability (Touvron et al.] (023) of LLMs, recent
work has also considered building multimodal models based on LLMs (Li et al. 2023b Driess et al.) 2023 Huang et al. [023 Dai et al. 2023) Ye et al. [003] Zeng et al. 2023 Koh et al. 2023b), such as BLiP-series (Li et al. 2023b Dai et al., 2023) that connect frozen vision and language pretrained models with Q Q-Former to bridge the modality gap. These LMMs commonly use predicting the
next text token as the training objective and exert no supervision for vision data (Hao et al. [022] Huang al. 2023 Zhu et al. 2023a Liu et al., 2023b Ye et al. 2023). Instead, Emu unifies the modeling of vision and language with the objective of predicting the next visual or text token in an autoregressive manner, and further explores videos new source of interleaved image-text data. This unified modeling leads to a generalist interface for diverse multimodal tasks that output
either image or text. Emerging recent studies (Zhu al. 2023a; Liu al. 2023b Maaz al. 2023; Li et al. 2023c Liu et al. 2023a; Li et al.) 2023a Chen et al., 2023b a) attempt to build powerful visual multimodal assistants based on L.MSs through constructed conversation data. We also instruction-un cierrcctiontuueeeae Emu using publicly available datasets and build multimodal assistant that aligns well with human instructions on both images and videos.
7 LIMITATIONS AND FUTURE TOPICS
Emu shares the well-acknowledged constraints inherent in other LLMSs and LMMs, including suscep- tibility to both visual and language hallucinations, slow auto-regressive inference speed, a cessation of knowledge updates after pretraining, and a potential for generating non-factual content. Besides, Emu predominantly focused on English-language data. As a result, the model's proficiency in languages other than English is currently delicate, and users should exercise caution when applying it in such contexts. Addressing challenges related hallucination, enhancing inference speed, and expanding multilingual capabilities are crucial areas for future exploration and improvement.
8 CONCLUSION
In this work, we present Emu, Large Multimodal Model trained with unified autoregressive objective of predicting the next element, including both visual and textual tokens. Apart from commonly used image-text pairs and interleaved documents, we explore another scalable data source of image-text interleaved data, i.e., video. Emu trained under such unified objective and diverse data can serve as generalist interface that is capable of performing diverse multimodal tasks, such as image captioning, image/video question answering, and text-to-image generation, together with new abilities like in-context text and image generation, and image blending. We also build a multimodal assistant instruction-tuned on Emu, which exhibits excellent human-aligned abilities such as multi-turn dialogue. We hope that our work will inspire the community to continue exploring the potential of diverse multimodal data at web-scale and also the generative pretraining beyond vision and language.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENTS
Emu is currently in a preliminary stage and has been developed solely for research purposes. Its usage in specific applications is not recommended until comprchensive risk analyses have been conducted and corresponding mitigation strategies thoroughly explored. The ensuing discussion outlines potential risks and corresponding mitigation strategies of Emu, acknowledging the necessity for further research efforts to comprehensively assess associated risks.
POTENTIAL RISKS
The ethical considerations associated with Emu primarily stem from two key aspects: 1) model initialization: the Multimodal Modeling module of Emu is initialized from an open-sourced large language model LLaMA (Touvron et al. 2023), the Visual Decoder module initialized from Stable Diffusion (Rombach al., 2022), and the Vision Encoder is initialized from EVA-CLIP (Sun) et al. 2023). Consequently, Emu inherits the potential risks of generating harmful and biased information, including offensive language, propagation of social biases and stereotypes, and the generation of inappropriate content such as pornography and child abuse. 2) Pretraining data. The pretraining data of Emu are publicly available and they are sourced from the Internet, where bias and harmful information are prevalent. Besides, the datasets sourced from the Internet (such as Common Crawl) may include links to images with personal information, potentially compromising privacy and containing sensitive content like faces, medical images, or other personal data.
MITIGATION STRATEGIES
is crucial to reiterate that Emu is designed exclusively for preliminary academic research and should not be deployed in specific applications without rigorous risk analyses and mitigation strategy exploration. Deployment in production environments warrants a more thorough investigation into model behavior and potential biases.
Given the extensive size of pre-training datasets and the associated training costs, curating datasets and developing models for widespread use exceeds the scope of single research paper. However, we are open to discussing mitigation strategies to help address ethical concerns.
Short-term approaches include: 1) relying on prompting mitigate any biases and harmful outputs, 2) implementing rule-based filtering, human oversight, and evaluation to identify and block harmful information, 3) employing discriminator model capable of classifying harmful information for enhanced blocking, 4) Emu itself can be finetuned to become multimodal discriminator.
In the long term, strategies involve: social or public policy interventions, such as regulatory frame- works and guidelines; 2) thoughtful product design, especially regarding user interface decisions; 3) advancements in Al Ethics of powerful large models, including the development of better benchmarks and improved mitigation strategies.
Additionally, to address privacy concerns, methods exist for obfuscating or generating personal human attributes like faces (Yang al.) 2202aa] Maximov et al. 2020), ensuring anonymity without compromising the quality of learned representations. While this avenue is worth exploring, it is currently beyond the scope of this paper.
In conclusion, Emu is presently model intended for preliminary research purposes only, and deployment should be deferred until the aforementioned issues thoroughly considered and addressed. Caution must be exercised before transitioning to production environments.
REFERENCES
Laion-aesthetics.
Laion coco: 600m synthetic captions from laion2b-en. [tcpps///aa.oo..al/ll////aaiiiip
Jean- Capttttpistte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob I.. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Rieardo Barreira, Oriol Vinyals,
10

---

Published as a conference paper at ICLR 2024
Andrew Zisserman, and Karén Simonyan. Flamingo: visual language model for few-shot leam- ing. In NeurlPS, 2022. URL http:p/pp//pppprr.........eeeeep httpapppp_iiiaaspppp///////////
html
Anas Awadalla, Irena Gao. Joshua Gardner, Jack Hessel. Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Komblith, Pang Wei Koh, Gabriel Iharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo, 2023. URLhttps: / o 10. 528 /ezooo.. 7733589
Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1728-1738, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1990 2020.
Delong Chen, Jianfeng Liu, Wenliang Dai, and Baoyuan Wang. Visual instruction tuning with polite flamingo. arXiv preprint arXiv:2307.01003 2023a.
Kegin Chen, Zhao Zhang. Weili Zeng. Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal 2023b.
I1m's referential dialogue magic. arXiv preprint arXiv:2306.
Xi Chen, Josip Djolonga. Piotr Padlewski. Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang. Yi Tay, al. Pali-x: On scaling up multilingual vision and language model. arXiv preprint ar 2iv: 18565, 2023c.
Xinlei Chen, Hao Fang, Tsung- Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 2eXiv:1504.0033 2015.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, volume 12375 of Lecture Notes Computer Science, pp- 104-120. Springer, 2020. doi: 1111000/978-3-030-5588558 978-3- 3-80--3300 58577-8 8_
URLihttss. /doo//////////////o/ .or./r////oo.. 1007/
Jaemin Cho, Lei, Hao Tan, and Mohit Bansal. Unifying vision-and- language tasks via text generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 1931 1942. PMLR, 2021. URL httpp::/ppoceeeingggg mlr. Axress/v133 h113399/hh22aatmmmm.m
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02:: 000001.......... 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetueed language models. arXiv preprint arXiv:2210.11412 2022.
Wenliang Dai. Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general- purpose Assion-lagguag models with instruction tuning. CoRR, abs/2305.06500 2023. doi: 10..855//2XX..23355.665 URL |RL
//doi.org/// 48550/ arxiv 2305 06500
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh. Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra, Visual dialog. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 326 000-333,220 2017.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi. Zheng Ge. Jinrong Yang, Liang Zhao. Jianjian Sun, Hongyu Zhou, Haoran Wei. et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXiv:2309./ 1499, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov. Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URI. https://<penrewt net. www.moormmmmmmm d cbr cbFdNTTy

---

Published as a conference paper at ICLR 2024
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, al. Palm-e: An embodied multimodal language model, arXiv preprint arXiv:2303.03378, 2023.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In European Conference on Computer Vision, pp. 89-106. Springer, 2022.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, al. Llama-adapter v2: Parameter-efficien visual instruction model. arXiv preprint arXiv:2304.1500 15010, 2023.
Tao Gong, Chengqi Lyu, Shilong Zhang. Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. arXiv preprint 2305:2300.04490, 2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEF conference computer vision and pattern recognition, pp- 3608-3617, 2018.
Yaru Hao, Haoyu Song. Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang. Shuming Ma, and Furu Wei. Language models are general-purpose interfaces. arXiv preprint arXiv:2206.066: 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv: 2rXiv2207..22598 2022.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: /ttpp:////peervveee..o net Eorum? ddzeeeeeeee
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language 2023.
models. arXiv preprint 2rXiv:2302.14000
Chao Jia, Yinfei Yang, Ye Xia, Yi- Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le. Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and Dision-languaa representation learning with noisy text supervision. In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 4904-4916. PMLR, 2021. UtL http:::///occcceceeeeoooc ngs. mlr. wtress//w199/1122222 /tt.:/ae................o//////.
httml
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event. volume 139 of Proceedings Machine
Learning Research, pp. 5583-5594. PMLR. 2021. URL http:://proeeeeeeeeeee mir. wwwpppeessssee. htt.://ceeddia.......re////////
kim2lk. 1.mm2k.htmll
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. Xixiv preprint arXiv:2305. 17216, 2023a.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). International Conference on Machine Learning, ICML 2023, 23-29 July 2023. Honolulu, Hawaii USA, volume 202 Proceedings of Machine Learning Research, pp. 17283- 17300. PMLR, 2023b.
htt.:::ppp//ppcccccingg. mlr.
wttp://w....p...pepe.e.. pwwps///ss/ wttpesss/v22//ch222... kch23a. 3tm.ml
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:230S.03722. 2023a.
12

---

Published as a conference paper at ICLR 2024
Junnan Li, Ramprasaath R. Selvaraju. Akhilesh Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven Chu- Hong Hoi. Align before fuse: Vision and language representation learning with momentum dis- tillation. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jen- nifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Con- ference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, vir- tual, pp. 9694-9705, 2021. URL https:/pproceediineeee 5052597562 01223395544997777777788811811119
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference Machine Learning, pp. 12888- 12900. PMLR, 2022.
Junnan Li. Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint (1Xii:23301.2597 2023b.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Vidcochat: Chat-centric vidco understanding. arXiv preprint arXiv:2305.06:3 2023c.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and CLawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer. 2014.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacooh, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304, 2...0:2304.08885, 2023b.
Luping Liu,, Yi Ren. Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models manifolds. arXiv preprint 2rXiv:2202.009 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Jiasen Lu. Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: unified model for vision, language, and multi-modal tasks. arXiv preprint wxXiv:2066009116 2022.
Yadong Lu,, Chunyuan Li, Haotian Liu, Jianwei Yang. Jianfeng Gao, and Yelong Shen. An empirical study of scaling instruct-tuned large multimodal models. arXiv preprint arXiv:2309.09955 .9.95999 2023.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arX1v:2306.05422, 2023.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Maxim Maximov, Ismail Elezi, and Laura Leal- Taixé. Ciagan: Conditional identity anonymization generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5447-5456, 2020.
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Malt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Hcidari, Yue Liu, et al. Anymal: An efficient and scalable any-modality augmented language model. arXiv preprint arXiv:2309./600)
2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Boh McGrew, Tlya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint
2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730- 27744, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Viriual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMI.R 2021. URL. http://prcceed ngs. mm r, v9ees//119/radeeee rt..
1a.htm)
13

---

Published as a conference paper at ICLR 2024
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to- text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831. PMIR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint XXiv:2204.06155 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479- 36494, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5h: An open large-scale dataset for training next generation image-text models. arXiv preprini arXiv:2210.08400, 2022.
John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng. Juan Felipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. Chatgpt: Optimizing language models for dialogue. OpenAl blog, 2022.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 1538899, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An insirwction-following llama model. https://github 2023.
com/tatsu- lab/ stan alpaca
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-basee image description evaluation. In Proceedings the IEEE conference computer vision and pattern recognition, pp. 4pp45566-4577 2015.
Jianfeng Wang. Zhengyuan Yang. Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 14100, 2022a.
Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma. Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. International Conference on Machine Learning, pp. 23318-23344 PMLR, 2022b.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Han- nanch Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212. 10560, 2022c.
Zirui Wang. Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov. and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022d, URL https //apenreview.netee wwtp:entt/foonuoe 1d-.ruunmi
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du. Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL. https:://penrest net. formm? www.rrmmmmmggg/rrooroo d=gorgG
14

---

Published as a conference paper at ICLR 2024
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question-answering explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9777-9786, June 2021.
Dejing Xu, Zhou Zhao, Jun Xiao, Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1645-1653, 2017.
Kaiyu Yang. Jacqueline Yau, Li Fei- Fei, Jia Deng, and Olga Russakovsky. A study of face obfuscation in imagenet. In International Conference on Machine Learning, pp. 25313 25330.. PMLR, 2022a.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022b.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang. Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
Jiahui Yu, Zirui Wang. Vijay Vasudevan, Legg Yeung, Mojtaba Scyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.0190 2022a.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint Arxiv22206. 10789, 2022b.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint 230900230,,,,5 2023a.
Weihao Yu, Zhengyuan Yang. Linjie Li, Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang. and Li- juan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint
2308.22308.02490 2023b.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjac Yu, Yanpeng Zhao, Mohammadroza Salchi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 16375-16387, 2022.
Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, and Tao Kong. What matters in training gpt4-style language model with multimodal inputs? arXiv preprint arXiv:2307.000 224.22307. 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing. Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging lin-as-a-judge with mt- mt-bench and chatbot arena. 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023a.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint 06X9v92304.06939 2023b.
15

---

Published as a conference paper at ICLR 2024
A EMU TRAINING
A.1 PRETRAINING
A.1.1 DATASET DETAILS
Image-text Pairs. The LAION-2B dataset is the english subset of Laion-5B (Schuhmann et al.
2022) and contains large-scale image-text pairs data. LAION-COCO (ai b is captioned 600M images from LAION-2B with an ensemble of BLIP (Li al. 2022) and CLIP (Radford et al., 202T)
models. Whereas the text in LAION-COCO (lai b) exhibits enhanced fluency and relevance to the associated images, it has insufficient text diversity and a potential loss of high-level semantic information, including world knowledge contents presented in the original LAION-2B dataset. Thus,
we employ both the LAION-2B and LAION-COCO (lai b) datasets during Emu pretraining.
Video-text Pairs. Webvid- 10M (Bain al. 2021) dataset contains a diversity of content with strong correlation between text and video. However, we found that certain amount of the data contained irrelevant metadata information (e.g. (egg.resolution of the original video, camera parameters). To prevent the model from being influenced by these irrelevant details, we use heuristic rules to remove these content. Firstly, we build a word list consisting of irrelevant information. This word list is then utilized as filtering mechanism to process the raw video text descriptions obtained from the original dataset. As a result, approximately million datasets requiring cleaning are identified. Subsequently, specific rules are designed based on this list to identify and eliminate any words of irrelevant information within the text. Finally, the cleaned texts are subjected to rewriting using the Vicuna- 13B (Zheng et al. 2023), thereby ensuring its fluency and enhancing the overall quality.
Interleaved Image and Text. Multimodal-Ca (Zhu et al. 2023b) is used as interleaved image-text data in pretraining. Following OpenFlamingo (Awadalla al. 2023). we filter images based on CLIP similarity score to ensure the relevance of the images and text in each document. Specifically, any image with CLIP similarity score below the threshold of 0.32 for all text in the same document is discarded. From each document, we sample a random subsequence of L = 1024 and take up to the first N 5 images included in the sampled sequence. This process results in long text with the inclusion of multiple images. Additionally, we randomly sample N = 5 images along with their corresponding sentences to construct subsequence of L 512. This approach yields N = image-text pairs.
Interleaved Video and Text. Videos with interleaved subtitles text represent a valuable and scalable source of multimodal data that has received limited attention thus far. In our study, we introduced YT-Storyboarde IB dataset, which collected storyboard images from YouTubee, utilizing the video-ids provided by the YT-Temporal-II1 dataset, which encompasses a vast collection of 18 million videos, equating to total of 1.8 billion storyboard images. Specifically, for each video, we crawl the storyboard images and subtitles files directly. Where the sampling time between storyboard images is fixed, so the start time of each image can be determined by the order. Subtitle files record the content of each subtitle, as well as the start and end times. Therefore, storyboard images and subtitles can be sorted according to their timestamps and adjacent subtitles can be merged to form an interleaved video-text sequence. By opting to collect storyboard images instead of raw video data, we eliminate the necessity of video decoding. Moreover, this approach leads substantial 20-fold reduction in data storage costs, resulting in increased download efficiency.
1.1.2 TRAINING DETAILS
We report the detailed training hyperparameter settings of Emu during the pretraining in Table 5
1.1.3 ABLATIONS ON INTERLEAVED VIDEO-TEXT DATA
We conduct ablation experiments to study the effectiveness of the YT-Storyboard- IB dataset. Smaller model size, batch size and shorter training schedule are used compared to the final Emu model
to reduce the training cost. The model consists of a large-sized EVA-CLIP (Sun et al. 2023) as visual encoder and LLaMA-7B (Touvron et al. 2023) as multimodal modeling module. Image- text and video-text pairs are used as common datasets, including LAION-COCO Lai 6). LAION- 2B (Schuhmann al., 2022) and Web 10M (Bain et al., 2021), with batch size of 256, 256, 32, respectively. YT-Story board- 1B batch size of 32 is used. Models are trained for 7.2k gradient steps on image-text and video-text pairs and additional 2.4k gradient steps on YT-Stooryboard113
16

---

Published as a conference paper at ICLR 2024
Table Summary of pretraining hyperparameters of Emu.
Configuration
Emu Pretraining
Vision encoder weight init.
EVA-CLIP (Snn et al.
2023
Large language model weight init.
LLaMA-13B
Touvron al.
2023
Causal transformer weight init. Vision encoder peak learning rate Large language model peak learning rate
random init.
4c-5
3e-5
Causal transformer peak learning rate Warmup ratio Learning rate schedule
le-4
0.2
cosine decay
Optimizer
AdamW (Lshchilov & Hutter 2017
Optimizer hyper-paramenn ppeer-parameters
81. _2.
0.9, 0.98, 1e-6
Weight decay
0.05
Input image resolution
224x224
Iterations
10k
LAION-2B Schuhmann et al. 2022 LAION-
COCO (lai b). MMC4
(Zhu (023b),
Data
Webvid- 10M (Bain at
2021), YT-Storyboard-
1B (Zellers et al. 2022
Batch size per dataset
128. 128, 64, 16, 16
Table 6: Quantitative comparison of w/wo interleaved video and text data during pre-training. 1: the number of in-context examples, and we used the same example selection approach (i.e. RICES (Yang et al. 2022b) Flamingo (Alayrac al. (2022). indicates that the zero-shot prompt is built by using two examples from the task, where their corresponding images have been removed.
COCO
OKVQA
MSVDQA
MSRVTTQA
Setting
k=0
k= *
k=4
k=0 *
k=4
k=0
k=4
Emu-7B w/o YT-Storyboard--11 BB
110.8
43.9
44.6
30.2
31.1
16.9
19.9
Emu-7B w/ YT-Storyboard- -B
112.9
42.3
45.7
30.8
34.9
17.9
20.8
With YT-Storyboard-I incorporated in the pretraining stage, Emu-7B achieves better zero-shot performance on MS-COCO (Chen et al. 2015), MSVDQA (Xu al. (017) and MSRVTTQA (Xu et al.) 2017). Besides, YT-Storyboard-1B also brings stronger in-context learning capability under 4-shot evaluation.
A.2 VISUAL DECODING
A.2.1 DATASET DETAILS
LAION-Aesthetics (lai a) is the subset of LAION-5B (Schuhmann et al. 2022) which have relatively
high aesthetics quality while LAION-COCO (lai b) has relatively high image-text correlation. To empower the visual decoder to possess the ability of decoding visual embeddings with both high quality and high relevance text prompts, visual decoding training. More specifically, we filter all text prompts with length greater than 150 to preserve large enough batch size and prevent the GPU memory overflow. This rule discards about 8% of LAION-Aesther and 0.01 % of LAION-COCO data, which has little effect on data diversity.
both LAION-COCK and LAIÖN-Aesthetics for
A.2.2 TRAINING DETAILS
The detailed training setups listed in Table
17

---

Published as a conference paper at ICLR 2024
Table Summary of Emu visual decoder training hyperparameters.
Configuration
Visual Decoder
Weight
Stable Diffusion v1.5
Batch size
50 4 x 8
Iterations
15k
warm up to Ic-4 for the first 5k, decrease to 5c-5 and Ic-5 10k and 14k
Learning rate
Input image resolution
512 x 512
Objective
e-prediction
Optimizer
AdamW (Loshchlovv & Hutter 2017)
Optimizer
81 92. c= 0.9, 0.999, 1c-8 le-2
Weight decay
LAION-COCO Aesthetics 7:2
lai
LAION-
Data
Data ratio
B INSTRUCTION TUNING
B. DATASET AND TRAINING DETAILS
We collect publicly available language, image and video instruction datasets for instruction tuning.
Language instructions: ShareGPT contains about 70K user dialogues with ChatGPT or GPT-4, and Alpaca (Taori et al.] 2023) dataset contains 52K instruction-following data generated using self-instruct (Wang al. 2022c from OpenAl's text taxxt-davnci--00
Image instructions: use LLaVA (Liu et al. 2023b dataset consisting of three types visual instructions, conversation, detailed description, and complex reasoning, with total number of 158K image-text instruction folloowin samples. In our preliminary experiments, we found the instruction-tuned model often generates instruction-irrelevvn detailed descriptions of the image. Thus, remove the detailed description subset of LLaVA. We also find a bad pattern 'on top of the back of" in the model's response, and we filter all data that contains this pattern. The resulting 130K LLaVA subset is used for instruction tuning.
Video instructions: we use VideoChat-11K (Li al. 2023c) and a subset of Video-ChatGPT. 100k (Maaz al. (203)) as our video-instruction dataset. VideoChat- K dataset built from WebVid- 10M consisting of 7K detailed video descriptions and 4K video conversations. Miteo-ChatGGTT-- 100k is built from ActivityNet, and we sample an around 30K subset that includes only videos under one minute.
We use a batch size of 128 and train for 10K steps, with epoches for ShareGPT, Alpaca and LLaVa VA datasets, and 60K samples for video-instrueet instruction data. The learning rate linearly warms up le-5 in the first 500 steps, then decays to zero with a cosine schedule. The overall instruction tuning stage takes around 16 hours with 16 A 100-80G GPUs. We attach LoRAs (Hu al. 2022) all linear projections of the self-attention layer, with the LoRA rank and alpha being 16.
B.2 SYSTEM MESSAGES
We use different system messages for instruction, image-instruction and video-instruction datasets, shown in Table
18

---

Published as a conference paper at ICLR 2024
Table 8: Summary of the prompting template.
Task Type
<ystemm Message>
Language Instruction Datasets
None
You are a helpful assistant and you
will be presented with an image: [IMG] ImwgeContent [/IMG] You will be able [/IMG)
Image Instruction Datasets
to see the image after provide it to you. Please answer my questions based the given image.
You are a helpful assistant and you will be presented with video consisting of multiple chronological images: [IMG) ImageContent I/IMG) You will be able to see the video after I provide it to you. Flease answer my questions based on the given video.
Video Instruction Datasets
Table 9: Summary of the evaluation benchmarks.
Dataset
Task
Split
Metric
COCO Text2lmage
Text-to-Image Generation
Val
FID(J)
COCO Caption
Scene description
Test
CIDEr(1)
Image
VQAv2
Scene understanding QA
Test-dev
VQA acc.(f)
OKVQA
External knowledge QA
Val
VQA acc.(f)
VizWiz
Scene understanding QA
Test-dev
VQA acc.(f)
VisDial
Image Dialogue
Val
NDCG(1)
MSVDQA
Event understanding QA
Test
Top-1 acc.(f)
Video
MSRVTTQA
Event understanding QA
Test
Top- acc.(f)
NexIQA
Temporal/Causal QA
Test
WUPS(1)
C EVALUATION
C.I BENCHMARKS
Emu excels at performing diverse types of completion in multimodal sequences by accepting multi- modal prompts, including text, images, videos, or their combinations, and generating comprehensive multimodal responses. To evaluate the capabilities of Emu, conduct extensive benchmark tests covering various tasks, which are summarized in Table 9 Specifically, we meticulously select benchmarks that encompass multimodal image/video and language tasks, including text-to-image generation, visual question answering for both images and videos, and image-based visual dialogue. When benchmarking OKVQA, we use VQAv2 evaluation code and stem the answers using Porter stemming to consolidate answers following Marino et al. (2019). For other tasks, we either submit our results for evaluation on the official website or use standard evaluation code.
C.2 ZERO-SHOT EVALUATION
Prompt Template. To ensure that the model outputs answers in the required style for the benchmark tests, we prompt Emu and Emu-I with task-specific templates, shown in Table I0] For each type of task, we have developed dedicated templates to structure the model's output. In these templates, "(question)" will be replaced with the question from the
task, "(history
https://githbbs
lab/vv
19

---

Published as a conference paper at ICLR 2024
question)" will be replaced with the historical question from the multi-turn visual dialogues, and similarly "history answer" will be replaced with the historical annotated answer from the multi- turn visual dialogues. Then, the image/video will be added before the text input to the model. Additionally, we implement post-processing techniques to filter out commonly occurring redundant phrases such "it is", "it's", "aa, "an", and "the". Furthermore, the model is required to output "unanswerable" for questions that cannot be answered in the VizWiz dataset. To achieve this, we augment the template by adding the phrase "is the answer known?" and prompt the model to respond with either "yes" or "no" by constraining the model generation. If the model responds with "no", we immediately return the answer as "unanswerable". On the other hand, if the model responds with "yes", we proceed to prompt the model to provide a valid answer.
Multimodal Chain-of-Thought Prompting. To enhance the capabilities of the pretrained model, ofain--f--Toughh prompting technique. Initially, when presented with
we utilize the Multimodal an image or video, we employ a prompt to guide the model in generating descriptive caption. Subsequently, the model is given both the caption and task-specific prompt generate the final result. The complete prompt template is shown in Table TO, where the " "(capt ion) tag in template will be replaced with the descriptive text generated by Emu. The experimental results demonstrate that this test-time technique effectively improves the model's performance without any additional data, leveraging the inherent capability of the model itself.
Text-only Examples Prompting. To ensure a fair comparison with Flamingo, we include results obtained through text-only examples prompting, denoted by asterisk (*) in Table We adopt the same approach as Flamingo in selecting examples (i.e., RICES). This involves utilizing two text-only examples from the task prompts, without any accompanying images (similar to the few-shot text prompts). During the evaluation process, we observed that this approach effectively formats the model's 's output, regardless of the label format of the datasets and the evaluation metrics employed. enabling a more accurate reflection of its true performance.
C.3 FEW-SHOT EVALUATION
In the few-shot evaluation settings, we incorporate a few example samples as prefixes in the template and connected the few-shot examples using ". AAditioonally like Flamingo, employ the Retrieval In-Context Example Selection (RICES) approach to select the few-shot examples.
To implement RICES, begin by randomly selecting 5000 training samples for each dataset. Then, using the pretrained EVA-CLIP model, we extract features from both the training set im- ages/videos and the test set images/videos. For each test sample, we select examples from the training set based on the highest cosine similarity using the extracted features, including them in the prompt. For the video-text task, we retrieve similar videos from the training set by comparing the mean of frame-level visual features extracted from our pretrained EVA-CLIP model.
Furthermore, we discover that the support video examples didn't 't require too many frames, which could exceed the LLM's context length limit. Therefore, we sample 8 frames for the given video and only 2 frames for the corresponding support video examples.
D COMPARISON WITH RELATED WORK
The results are presented in Table II where Emu achieves state-of-the-art results on 5 out of benchmarks evaluated.
CM3Leon Yu et al. (2023a) has similar motivation with to train a unified image-to-text and text-to-image model. The most significant difference lies in that CM3Leon discretizes images, but we directly input and output image continuous features. We can find that CM3L Leon performs much worse than Emu on image-to-ter tasks.
AnyMAL MooMALLMoon have relatively weak performance on image to text tasks.
Moon et al. (2023) a large-scale Multimodal LLM that can process any modality, but
20

---

Published as a conference paper at ICLR 2024
Table 10: Summary of the prompting template.
Model
Type
Template
Image Captioning
describing the image in detail. the image shows
Image QA
a picture of (caption). based on the picture, (question) short answer:
picture of (caption). based the picture, (history question) short answer: (history answer). ... based on the picture, (question) short answer:
Emu
Image Dialog
Video Event understanding QA
a video of (caption). a question about
the video: [question] answer:
Video Temporal/Causal QA
video of (caption). a question about
the video: (question) short answer:
[USER] please provide an accurate and concise description of the given image. [ASSISTANT] the image depicts a photo of
Image Captioning
[ based the content of the image and commen sense, please provide an accurate answer consisting of only one word or phrase. is:
Image QA
[ASSISTANT]; the answer
Emu-I
[USER] (history question) [ASSISTANT] : (bistory answer).<\s> ... [USER]: (question) [ASSISTANT]:
Image Dialog
[USER] based on the content the video and common sense, please provide an accurate answer consisting of only one word or phrase. (question) [ASSISTANT]: the answer is:
Video Event understanding QA
[ based on the content of the video and common sense, please provide an accurate short answer. [question] [ASSISTANT]:
Video Temporal/Causal QA
answer
Table 11: Zero-shot comparison with concurrent work.
Models
COCO (Image2Text)
COCO (Tex12lmage)
VQAv2
OKVQA
VizWiz
VisDial
CM3Leon
61.6
10.82
47.6
23.8
37.6
22.6
AnyMAL-13B
99.5
59.6
33.1
24.4
Emu
112.4
11.66
52.0
38.2
34.2
47.4
Emu
52.9
42.8
34.4
47.8
Emu-1
120.4
57.2
43.4
32.2
43.0
Emu-1 *
62.0
49.2
38.3
51.1
E QUALITATIVE CASES
21

---

Published as a conference paper at ICLR 2024
This the first
What's the name of this character?
The character's name Mewtwo.
This is the second image.
What's the difference?
Who
difference first image, there is an alpaca standing in front of fence, standing in front of fence. Alpacas and llomas both members different species. Alpacas are native South America, while llamas are native Central and South America.
images
image from
the anime series Sate Night: Unlimited Dlade
image,
Works.
camelid family, they are
Tell characters in the image,
The charscters the image SpongeBab SquarePants, Parrick Star, and Squidward Tentacles.
This is first image.
This the second image.
Step by stop describe the cooking rocipe shown in the video?
This is the third image.
THE
shows step- by-step guide make chocolate milksshkee, First, the person in the video adds ice cubes to blender. Then, they add two scoops chocolate vanilla lee cream to the blender. After that, they three tablespoons chocolate syrup and two tablespoons of cocoa powder to the blender. Finally. they blend all ingredients together until the mixture is smooth and creumy. The video ends with person pouring the chocolate milksshaee glass and garnishing with whipped cream and
and ene scoop of
This the fourth image. Deseribe all images.
book of the Namelles". The second image is the cover of the of the book "A and of My Dreams". The fourth image the cover the book "The Key MindFulness""
image
image cover
chocolate sprinkles. Overall, the vidco provides simple and easy recipe for making delicious chocolate milkshake.
Figure 4: Examples of interleaved multi-image understanding(left side), real-world knowledge grounding(upper right), detailed video understandingllower right).
22

---

Published as a conference paper at ICLR 2024
Mahatma Gandhi Spinning Yarn the Late 1920s
What does the image show? Describe all details including the figures, texts and numbers.
image shows a black white photograph Mahstmu Gandhi spinning yarn in the 1920s. Gandhi sitting on the ground spinning wheel front of him. wearing white shawl and appears focused spinning the yarn. There other figures objects visible in the image.
Kaacther grvet
Explain this meme to me.
This meme features cat with mouth open and the caption "This is face when teacher gives surprise test." The meme is meant to be humorous and is often used to express frustration annoyance with unexpected situations.
Where in the image?
The image shows the Twelve Apostles on the Great Ocean Road in Victoria, Australia.
Which heavier?
The red ball is heavier than the blue ball.
Figure 5: Examples of multimodal assistant.
23

---

Published as a conference paper at ICLR 2024
What are the things I should be cautious about when visit here?
There are few things you should be cautious about when visiting the lake:
Weather: The weather at the lake can unpredictable, with sudden changes temperature, wind, and precipitation. Make
check the weather forecast before your visit and pack appropriate clothing and gear.
2. Wildlife: The lake home variety of wildlife, including bears, wolves, and moose. Make designated trails and keep your distance from any wildlife you encounter.
on
Water quality: The lake's water quality can vary dopending on the time year the weather conditions. Make
check with local authorities before swimming boating the lake
is safe
4. Navigation: The lake is large and can difficult navigale, especially for inexperienced bosters. Make sure
familiarize yourself with the lake's layout and any potential hazards before venturing out on the water.
5. Safety equipment: Make sure to bring any necessary safety equipment, such as life jackets, first aid kits, and communication devices, case emergeneies.
By taking these precautions, you
safe and enjoyable visit the lake.
Do you have any suggestions about safety equipment?
Yes, here are few suggestions for safety equipment:
1. Life jackets: Make sure to bring a life jacket for each person your group, espacially if you plan on boating or swimming
lifesaver of emergencies, Make include items such as bandages, antiseptic wipes, and pain relievers. Communication devices: Bring reliable communication device, such as a cell phone or walkie-talkie, in case of emergencies.
First aid kit: well-stocked first
Navigation equipment: If you plan besting kayaking lake, make bring map, compass, and GPS device help you navigale the waters. By bringing these safety equipment, ensure a safe and enjoyable visit to the lake.
Can you poem the image?
Sure, here's poem about the image:
The serene streiches out before me. vast expanse crystal-clear water.
The shines down, casting golden glow, As take in the breathtaking view.
The gentle waves lap against the shore. Creating a soothing symphony of sound.
foel
peace tranquility
As bask in the beauty of this place.
Figure 6: Examples of multi-turn dialogue.
24

---

Published as a conference paper at ICLR 2024
Input Prompt
Completion
Figure Examples of image blending.
Input Prompt
Completion
A portrait of old man. He has a bushy beard and wears a straw hat. His face is lowards the left of the frame, but looking towards the camera.
This an oil painting depicting sunflowers. There are many sunflowers in vase. And there are red and yellow sunflowers.
This is image of cute puppy. The puppy sitting and resting the lawn, surrounded by many flowers plants.
This is an image of a big house.
There is cule animal sitting front of a white background. The animal's body has two colors and looks very cute. body is hidden behind its head and it's looking at camera.
An image of the animal the first image is sitting and resting the lawn, surrounded by many flowers and plants.
Figure 8: Examples of in-context text-to-image generation.
25

---

Published as a conference paper at ICLR 2024
Input Prompt
Completion
This an image of cute The puppy sitting and resting on the lawn, surrounded by many flowers and plants.
This image of big house
A photo stalk pink bedroom.
A Mercedes C- Class speeds down highway.
large top of sunny mountain in style
image of dog wearing a pair of glasses.
Figure 9; Examples of text-to-image generation.
26

---

Published as a conference paper at ICLR 2024
Input Image
Input Text
Describe the image detail.
Model Output
Emu
The image is a painting by Claude Monet titled "Impression, Sunrise", depicts a serene scene the sun rising over the water. The painting is done in loose, impressionistic style, with visible brushstrokes than details. colors are muted and soft, with shades bluc, pink, and yellow dominating canvas.
focus on capturing the essence the scene rather
In the foreground of the painting, there is a small boat bobbing up und down on the water. The sun is just peeking over the horizon, casting a warm, golden light over the scene. The water is calm and still, reflecting the colors of the sky above.
In the background of the painting, there are a few buildings visible, but they are mostly obscured by the mist and haze of the early morning. The sky mix of pink and blue, with few wispy clouds drifting lazily across the sky.
Overall, the painting is beautiful and serene depiction of peaceful morning on the water. The soft colors and loose brushstrokes evoke sense of calm and tranquility, inviting the viewer take moment to appreciate the beauty of nature.
LLaVA
The image caplivating scene of small floating on body water, possibly buy or a river, during u beautiful sunset. boal positioned towards center the picture, surrounded by serene atmosphere.
There are several other bosts scattered throughout the image, some closer to edges of the frame, while others are positioned more towards the center. The presence of multiple boats adds a sense depth and interest to the seene.
The sunset casts a warm glow on the water and the boats, creating a picturesque and tranquil environment for those on small boats to enjoy the view.
mPLUG-Owl
The painting depicts beautiful sunrise the occan, creating stunning and serene atmosphere. The scene against backdrop of blue-gray sky, with bright yellow pecking from behind cloud.
The water in the foreground of the painting is calm and peaceful, gently reflecting the sun's rays. Several sailbouts be floating various positions, adding the tranquil atmosphere of this early morning moment. addition boats, there several birds flying different directions, enjoying the beautiful view.
InstructBLIP
In this painting, Claude Monet's famous impressionist artwork titled "Sunrise" takes center stage. The painting depicts the beautiful sunrise over the horizon, with its warm orange and yellow hues spreading across the sky. The water is also a significant part of the painting, with several bouts suiling along it. The sunrise is captured in the middle of the painting, with boats scuttered across the scene. In the background, the boats and the water are surrounded by various colors, with some areas featuring a lighter, brighter tone. The sunrise creates screne and peaceful atmosphere, while the boats add touch of activity to the secnc.
Figure 10: Comparison of Emu with other methods on image captioning task.
27

---

Published as a conference paper at ICLR 2024
Input Image
Input Text
Tell me 8 books written by the woman and recommend one to me.
Model Output
Emu
Here books written Agatha Christie:
"The Mysterious Affair Styles" (1920)
"The Sccret Adversary" (1922)
"The Murder on the Links" (1923)
"The Man in the Brown Suit" (1924)
"The Secret of Chimneys" (192.5)
"The Murder of Roger Ackroyd" (1926)
"The Big Four" (1927)
8. "The Murder at the Vicarage" (1930)
Recommendation: "The Murder of Roger Ackroyd" (1926)
"The Murder of Roger Ackroyd" is considered by many to be Agatha Christic's greatest mystery novel. It classic whodunit with a twist ending that will keep you guessing until the very end.
LLaVA
The woman in the image, Agutha Christie, has written numerous books, totaling up to 66 novels, including mysteries and thrillers. One of her most famous and highly recommended books is "And Then There Were None." " This book is classic whodunit novel that tells the story of ten strangers who are lured murdered one by onc, according to the rules of the famous nursery rhyme. The intricate plor, unique setting, and engaging characters make an excellent read for anyone who enjoys mysteries and suspense.
isolated island and then
mPLUG-Owl
Agatha Christie was prolifie British author who wrote over 80 novels, short stories, and plays. Some of her most famous works include:
1. Murder on the Orient Express (1934): A classic mystery novel about the investigation
of a murder luxury train.
2. And Then There Were None (also known Ten Little Indians) 10 Little Indian, 2015)
The Murders The Rue Morgue (first published in 90s) A detective slory about series of murders Puris.
A MurDER IS ANNOUNCED (published the 40's), play that adupted for movie and later became TV mini-series. It'll give you glimpse into her writing style.
InstructBLIP
the spiderwick chronicles
Figure 11: Comparison of Emu with other methods in term of following human instructions.
28

---

Published as a conference paper at ICLR 2024
Input Vidco
Input Text
What's the woman doing the video?
Model Output
Emu
The woman in the vidco is wearing pair of Apple VR goggles while sitting in airplanc. She appears to be using the goggles watch something, possibly vidco or 360-degree view of the airplane's surroundings.
Video-ChaGGPT
The woman is sitting in plane and looking out of the window.
ImageBind-LIMM
The woman is looking at her cell phone, possibly using it to take picture view something on the screen.
Figure 12: Comparison of Emu with other methods in term of following human instructions.
29


Reviews:
[{"review_id": "BS2ciyj4ak", "reviewer": "Reviewer_yynn", "summary": "The paper presents a novel large multimodal model, Emu that takes as input the interleaved visual and textual data and generates images and text. The key contributions of the paper are three-folds: leveraging web-scale video data as a new source of interleaved data, proposing a novel objective of predicting-the-next-element in an unified autoregressive manner, and the model architecture enabling learning under such objective. The pretrained, or instruct-tuned models show remarkable performance on diverse multimodal tasks, which are shown qualitatively and quantitatively in the paper.", "questions": "I am not sure about the composition of each mini-batch: does each mini-batch comprise heterogeneous samples with a batch size of 128+128+64+16+16=352, or homogeneous samples with a batch size of at most 128? And the authors said that they pretranied Emu only for 10k steps while the total number of training samples is 82M. This means that the model has seen at most 4.3% of the training data during pretraining. Is this a typo? otherwise, please explain how it was possible to learn so well with such a small amount of data.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "+ I agree with that the importance of video as a data source for learning large multimodal models has been overlooked so far. Leveraging videos as interleaved data will definitely provide much diverse supervision signals and facilitate scaling up training data\n+ The writing and presentation of the paper is really good. The paper is overall well-written and easy to read. Especially, the authors describe all the details of the model architecture and training/inference procedures.\n+ The experimental results validate the effectiveness of the proposed method.", "weaknesses": "Emu applies the regression loss to latent embeddings computed by the Causal Transformer, whose parameters are randomly initialized and also learned during pretraining. I was surprised that the training went well with the proposed objective, because I think that without additional constraints, the model may easily fall into a degenerate case, like the Causal Transformer always outputting constant vectors. Please elaborate on the mechanism of the proposed l2 regression loss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g0ZTFeMDA4", "reviewer": "Reviewer_qjEg", "summary": "The paper introduces \"Emu\", a multimodal foundation model adept at generating both images and text in a multimodal context. Unlike traditional models that focus solely on text, Emu can accept various data inputs, including image, text, and video, through a unified autoregressive training process. Visual data is converted into embeddings, which, when combined with text tokens, form an integrated input sequence. The model's training objective is to either classify the next text token or regress the next visual embedding in this sequence. Emu's strength lies in its ability to utilize a wide range of pretraining data sources, such as videos combined with text or web pages with intermingled images and text. This versatility makes Emu suitable for tasks like image captioning, visual question answering, and text-to-image generation, where it outperforms other leading multimodal models. Emu's advanced features include in-context generation of text and images, image blending, video comprehension, and knowledge grounding. The model's effectiveness is further showcased as a multimodal assistant that can interact with users using both text and visuals.", "questions": "what is the FID before and after the visual decoding training?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The model shows ability to do versatile generation and strong in-context learning capability.", "weaknesses": "Certain model details is not clear:\n1. How does causal transformer convert an image as multiple visual tokens. In section 2, is {z_1, z_2, ... z_N} the same as g(I). \n2. Is the N visual embeddings for the image decoder the same as the visual ebmedding after the causal transformer. \n\nPotential data issue impact the \"zero-shot\" restult:\nEMU is trained with Laion-COCO, which has image caption in the style of COCO. How does that impact results in table 1? Is the zero-shot results as good if removing Laion-COCO?\n\nMiss baselines:\nOther multimodal baselines for FID such as https://arxiv.org/abs/2309.02591 should be added to table 2.\n\nSuboptimal image generation capbility: \nAs shown in table2, there is performance loss due to regression to visual embeddings, instead of directly optimizing generating the best images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yHpZbut245", "reviewer": "Reviewer_qbfH", "summary": "The paper presents a novel multimodal foundation model named Emu, designed to handle both images and text in a multimodal context. The primary innovation lies in its autoregressive training process which can accommodate single-modality or multimodal data inputs like images, text, and video. Visual signals are initially encoded into embeddings, which, along with text tokens, form an interleaved input sequence. Emu is trained end-to-end with a unified objective which alternates between classifying the next text token and regressing the next visual embedding within the multimodal sequence. This model's flexibility allows it to harness diverse pretraining data sources, including videos with interleaved frames and text, webpages with interleaved images and text, as well as web-scale image-text pairs and video-text pairs. By doing so, it demonstrates superior performance across a range of zero-shot and few-shot tasks like image captioning, visual question answering, and text-to-image generation.\n\nEmu's architecture is divided into four components: the Visual Encoder, Causal Transformer, Multimodal Modeling, and Visual Decoder. The Visual Encoder and Causal Transformer handle the conversion of images and videos into a format compatible with the Multimodal Modeling LLM (Large Language Model). Specifically, the visual data is first encoded into dense visual features, which are then transformed into a fixed number of visual causal embeddings. These visual causal embeddings, along with text tokens, are fed into the Multimodal Modeling LLM, which performs unified autoregressive modeling. Post-training, a Visual Decoder is fine-tuned to convert visual embeddings back into realistic images. The training data spans web-scale collections of image-text pairs, video-text pairs, interleaved image-text data, and interleaved video-text data, with Emu being pretrained on these multimodal data sequences under a unified objective of predicting the next element in a multimodal sequence.\n\nThe evaluation of Emu covered a broad spectrum of vision-language tasks, demonstrating its effectiveness and superior performance in comparison to other large multimodal models in many instances. In zero-shot evaluations, Emu showcased remarkable results in tasks like COCO captioning and VizWiz VQA. In few-shot evaluations, Emu continued to exhibit strong performance across various image and video question answering tasks. The qualitative evaluation also underlined Emu's real-world knowledge grounding, detailed video understanding, and in-context text-to-image generation capabilities. Moreover, the paper also explores an \"in-the-wild\" evaluation and instruction tuning to align Emu with human instructions further. Through its innovative architecture and comprehensive evaluation, Emu lays down a significant milestone in the journey towards more capable and versatile multimodal AI models.", "questions": "- The paper could benefit from a deeper discussion on the limitations of Emu, and potential strategies for overcoming these limitations in future work. Maybe the authors can add the same?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-organized and the problem is clearly defined. The authors provide a comprehensive introduction to the problem and the proposed solution, Emu, which appears to be novel and well thought out.\n\n- The unified objective for both text and visual data seems to be a promising approach to handle multimodal tasks, and the autoregressive training process is well justified.\n\n- The authors have undertaken extensive evaluations including zero-shot, few-shot, and in-the-wild evaluations, showcasing the model's performance across a variety of tasks. The performance of Emu, particularly in zero-shot tasks, appears to be impressive and competitive with other state-of-the-art models.\n\n- The qualitative evaluation provided a good insight into the model's capabilities in real-world scenarios, which is a strong point of this paper. It’s interesting to see the model's performance in text-to-image generation, image blending, and real-world knowledge grounding.\n\n- Incorporation of various pre-training data sources, including videos with subtitles and image-text interleaved data, could contribute to the model's strong performance and is a positive aspect of this work.", "weaknesses": "- The paper misses some key and very relevant comparative works like Cm3Leon (https://arxiv.org/abs/2309.02591), AnyMAL(https://arxiv.org/abs/2309.16058) etc. These papers should be compared against and explained how the authors work differes from the same.\n- It would be beneficial to see a discussion on the scalability of Emu with respect to the size and diversity of training data, and how the model might perform with fewer resources or less diverse data.\n- Autoregressive models are know to be notoriously slow at inference time due to the sequential nature of their execution. A comparative study of the same and ideas to make it better could help the paper.\n- The image generation models are capable of generating harmful content like pornography, child abuse etc. These also have human faces which are not in compliance with privacy laws in several states and countries and the authors dont talk about taking any steps to ensure compliance to the same. This should potentially be addressed to ensure the model isnt misused.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "thBUHzoNxY", "reviewer": "Reviewer_paGX", "summary": "This paper presents Emu, a foundation model designed for multimodal tasks, capable of handling text, image, and video inputs. It uses a unified autoregressive training approach to encode visual signals and text tokens into an interleaved sequence. Trained on diverse data sources, Emu is versatile in both image-to-text and text-to-image tasks, and performs well in zero-shot and few-shot inference. It also shows promise in extended applications like multimodal assistants, using instruction tuning.", "questions": "- I am curious if the pre-trained model can do more complex tasks such as the in-context subject-driven generation introduced in the paper [1]. Given that the model is general enough to handle prompts such as \"Given the a few images about a common subject {subject_name}: {image_1}, {image_2}, {image_3}. Generate a new rendition of this same subject, in the Hall of Mirrors in Versailles.\". If would be amazing if such complex task can be done in zero-shot.\n\n[1] Subject-driven text-to-image generation via apprenticeship learning", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is a strong paper that investigates a unified multimodal pre-training recipe on creating generative model that inputs and produces interleaved image and text sequence, capable of multimodal understanding and image generation.\n- The model is comprehensively evaluated, on a wide variety of tasks, both generative and discriminative.", "weaknesses": "- The comparison of image-text and video-text tasks should be also made with other state-of-the-art multimodal models, such as GPT-4V, PaLI-X (55B). It is okay to underperform given that those models are (likely) larger in the model size. The important part is to give reader a full picture of how this work compares to those important (private) models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)