Making LLaMA SEE and Draw with SEED Tokenizer
Authors: Yuying Ge
Sijie Zhao
Ziyun Zeng
Yixiao Ge
Chen Li
Xintao Wang
Ying Shan
Venue: ICLR 2024 poster (2024)
Keywords: Multimodal Large Language Model
Discrete Visual Tokenizer
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: The great success of Large Language Models (LLMs) has expanded the potential of multimodality, contributing to the gradual evolution of General Artificial Intelligence (AGI). A true AGI agent should not only possess the capability to perform predefined multi-tasks but also exhibit emergent abilities in an open-world context. However, despite the considerable advancements made by recent multimodal LLMs, they still fall short in effectively unifying comprehension and generation tasks, let alone open-world emergent abilities. We contend that the key to overcoming the present impasse lies in enabling text and images to be represented and processed interchangeably within a unified autoregressive Transformer. To this end, we introduce $\textbf{SEED}$, an elaborate image tokenizer that empowers LLMs with the ability to $\textbf{SEE}$ and $\textbf{D}$raw at the same time. We identify two crucial design principles: (1) Image tokens should be independent of 2D physical patch positions and instead be produced with a $\textit{1D causal dependency}$, exhibiting intrinsic interdependence that aligns with the left-to-right autoregressive prediction mechanism in LLMs. (2) Image tokens should capture $\textit{high-level semantics}$ consistent with the degree of semantic abstraction in words, and be optimized for both discriminativeness and reconstruction during the tokenizer training phase. With SEED tokens, LLM is able to perform scalable multimodal autoregression under its original training recipe, i.e., next-word prediction. SEED-LLaMA is therefore produced by large-scale pretraining and instruction tuning on the interleaved textual and visual data, demonstrating impressive performance on a broad range of multimodal comprehension and generation tasks. More importantly, SEED-LLaMA has exhibited compositional emergent abilities such as multi-turn in-context multimodal generation, acting like your AI assistant. The code (training and inference) and models are released in https://github.com/AILab-CVC/SEED.

Published as a conference paper at ICLR 2024
MAKING LLAMA SEE AND DRAW WITH SEED TOKENIZER
Yuying Ge'",iiji Zhao'r, Ziyun Zeng', Yixiao Gel.2t, Chen Li?, Xintao Wang Ying Shan 1.2 Tencent AT Lab, ARC Lab, Tencent PCG
Chatting with SEED-LLOMA
of Gukkon Gute Brkdge?
Luckl cool imai Can tam anol pa miiting
tutcted Golden Gate fridge diapply impressod, me show sicture.
Frsartt not bringing dog Luchy alrg for viec. This it larkly
locks gookt Can moke Furoy suggtoog?
liwe tak painting
tucky's
adventure for th painting?
upon time, Lucky was misiion
seillin
eaploring until
found patch graes
whene he could by down and take in the beaury of the lirdge As he there, lie put sunglasses to protect eyes
sun adrvire
Figure I: The introduced SEED-LLaMA, a multimodal AI assistant, demonstrates emergent ability in the multi-turn in-context image and text generation given multimodal instructions.
ABSTRACT
The great success of Large Language Models (LLMs) has expanded the potential of multimodality, contributing the gradual evolution of General Artificial Intelli- gence (AGI). A true AGI agent should not only possess the capability to perform predefined multi-tasks but also exhibit emergent abilities in an open-world con- text. However, despite the considerable advancements made by recent multimodal LLMs. they still fall short in effectively unifying comprehension and generation tasks, let alone open-world emergent abilities. We contend that the key to over- coming the present impasse lies enabling text and images to be represented and processed interchangeably within a unified autoregressive Transformer. To this end, we introduce SEED, an elaborate image tokenizer that empowers LLMs with the ability to SEE and Draw the same time. We identify two crucial design principles: (1) Image tokens should independent 2D physical patch positions and instead be produced with ID causal dependency, exhibiting intrinsic interde- pendence that aligns with the left-to-right autoregressive prediction mechanism LLMs. (2) Image tokens should capture high-level semantics consistent with the degree of semantic abstraction in words, and be optimized for both discriminative. ness and reconstruction during the tokenizer training phase. With SEED tokens, LLM is able to perform scalable multimodal autoregression under its original training recipe, i.e., next-word prediction. SEED-LLaMA is therefore produced by large-scale pretraining and instruction tuning on the interleaved textual and visual data, demonstrating impressive performance on broad range of multimodal comprehension and generation tasks. More importantly, SEED-LLaM, has ex- hibited compositional emergent abilities such as multi-turn in-context multimodal generation, acting like your AI assistant. The code (training and inference) and models are released wtt.ttts///giiieco
com/ com/AILal AILab SVC/SEED
"Equal Contribution.
Corresoondence yi www.aaeeeeeeeetccom

---

Published as a conference paper at ICLR 2024
Original Image
Image
Next word Prediction
Semanticcally Aliommen
Encode
1D Causal Dependency
Generate
Multimodal LLM
Tokenize
De-Tolanize
Discrete Vision Codes
Vision
Text
Vision
Features
Noise
(a) SEED Visual Tokenizer
(b) Multimodal Autoregression with SEED visual tokens
Figure 2: (a) SEED is discrete image tokenizer, producing quantized visual codes with ID causal dependency and high-level semantics. (b) With SEED tokenizer, LLM is able to perform scalable multimodal autoregression interleaved visual and textual data with nxxt-wod--predcttin objective.
INTRODUCTION
In recent years, Large Language Models (Touvron et al. 2023 Brown et al. 2020; Chowdhery et al. 2022) (LLMs) pre-trained on massive text corpus with straightforward training objectives such as next-word prediction have exhibited remarkable abilities to understand, reason, and generate texts across a variety of open-ended tasks. Recent studies further exploit the strong generality of LLMs to improve visual understanding generation tasks, collectively referred to as Multimodal LLM (MLLM). While these studies have contributed to technological advancements, MILMS have yet to achieve the remarkable success of LLMs in terms of emergent capabilities. We have made a bold assumption that the premise for the emergence of multimodal capabilities is that text and images can be represented and processed interchangeably in unified autoregressive Transformer.
We posit that proper visual tokenizer is the key as it can facilitate the follow-up multimodal training by (i) easing the semantic alignment between visual and word tokens, and (ii) enabling LLM's original training recipe (i.e., next-word prediction) for multimodal data without specific adaptation for visual tokens. Representing images as a sequence of discrete IDs is naturally compatible with the autoregressive training objective of LLMs. But unfortunately, works (Ramesh et al. 1 Ding al., 202T) that utilize discretized visual tokens for multimodal tasks have receded from prominence, as such models generally rely on super-scale training to converge, leading to substantial training costs. Moreover, we empirically found that the dominant tokenizer VQ-VAE Van Den Oord et al. 2017) in existing works captures too low-level information for L.LMs to effectively perform multimodal comprehension tasks. Existing image tokenizers fail to meet the requirements of unifying the generation of images and texts and facilitating multimodal training.
To this end, we introduce SEED, a VQ-based image tokenizer that produces discrete visual codes with ID causal dependency and necessary high-level semanties for both visual comprehension and generation tasks, as shown in Fig. P (a). The off-the-shelf LLMs can be readily equipped with SEED by treating discrete visual tokens new words and updating the vocabulary. We would like to emphasize the design principles of SEED. (1) Why causal- causal-depenteee dependent tokens? Existing visual tokens (e.g., from VQ-VAE or CLIP-ViT (Sun al. (2023a) are generated using 2D context, which is incompatible with the unidirectional attention in dominant LLMs and counterintuitive for text- to-image tasks requiring raster order prediction. Thus, we convert 2D raster-ordered embeddings into a sequence of semantic codes with ID causal dependency. (2) Why high-level semantics? Since visual and textual tokens in LLMs are expected be interoperable searoprrable-sshriing weights and training objectives- they should encompass the same degree of semantics to prevent misalignment, i.e., the high-level semanties inherently present in words.
Specifically, the SEED tokenizer is composed of a VIT encoder (Dosovitskiy et al. 2020). Causal Q-Former, VQ Codebook [Van Den Oord et al. 2017), multi-layer perceptron (MLP), and a UNet decoder (Ronneberger al. 2015). The ViT encoder and UNet decoder are directly derived from the pre-trained BLIP-2 (Li et al. 2023c) and unCLIP- SD model (Rombach et al. 2022 Ramesh et al. 2022), respectively. (1) Tokemize: Causal Q-Former converts 2D ordered features produced by the ViT encoder into a sequence causal semantic embeddings, which are further discretized by the VQ Codebook. (2) De-Tokenize: The discrete visual codes are decoded into generation embedding
2

---

Published as a conference paper at ICLR 2024
via MLP. The generation embedding is aligned with the latent space of unCLIP-SD so that realistic images with consistent semantics can be generated using the off-the the-shelf SD-UNet.
We further present SEED-LLaMA by equipping the preitedined LLM Touvron et al. 2023 with SEED tokenizer. SEED-LLaMA is pretrained on multimodal data, including image-text parrs, video- text pairs, and interleaved image-text data, toward the training objective of next-word prediction as shown in Fig. (b). Such an pasy-to-implemt
and unified proxy task facilitates scalable multimodal pretraining. We further apply multimodal instruction tuning to align SEED-LLaMA with human instructions through supervised fine-tuning. Our model demonstrates extensive emergent abilities such as multi turn in-context image and text generation given multimodal instructions as shown in Fig. I We also benchmark on broad range of tasks including image captioning. image/video question answering, and text-to-image generation, receiving competitive performance.
In summary, our contributions are three-fold. (1) We introduce SEED, an advanced image tokenizer, designed based on the insights that visual tokens compatible with LLMs should capture high-level semantics while being generated with ID causal dependency. The tailored SEED improves the scalability of subsequent multimodal training. (2) We present SEED-LLaMA, composed of a pre- trained LLM and SEED tokenizer, through large-scale multimodal pretraining and instruction tuning under the word-word-ppe preddicteiitionn training objective. It successfully unified multimodal comprehension and generation tasks in one framework. (3) LLED- shows competitive results existing multimodal tasks (e.g., text-to-image, image-to-text) and further demonstrates emergent abilities in multi-turn in-context multimodal understanding, reasoning, and generation.
2 RELATED WORK
MILMS for Comprehension and Generation. With the impressive success of Large language
models (Touvron et al.) 2023] Brown et al.) 2020 Chowdhery et al. 2022) (LLMs), recent studies
work on Multimodal LLM (MLLM) to improve visual comprehension through utilizing the strong
generality of I.MMs. Previous work Yee et al. 2023 Li et al. 2023c; Zhu al.
2023a Zhang et al.,
2023b Gao et al. 2023; Liu et al. 2023b; Alayrac et al. 2022, Driess et al. 2023) align visual
features of pre-trained image encoder with LLMs on image-text datasets. However, these work commonly use the prediction of the next text token as the objective, thus can only output texts.
To empower LLMs with the image generation ability, Cog View (Ding et al. 2021 pre-trains a visual tokenizer by reconstructing image pixels, and fine-tunes GPT (Brown et al. 2020) with the objective of next-token prediction. GILL (Koh et al.) (2033a) learns a mapping between the embeddings of a LLM and frozen lext-to-image generation model. Both work aim to generate images with LLMs, without being explicitly designed for unifying multimodal comprehension and generation.
Our concurrent works (Sun et al. 2023b Lili et al. 2023) both perform multimodal autoregression including the generation of images and texts. CM3Leon (Lili et al.) 2023) utilizes discrete visual codes from a image tokenizer (Gafni et al., 2022) pre-trained on image pixel reconstruction and
performs to-ee-t and text-to-image autoregression. However, it yields suboptimal performance in visual comprehension tasks (c.g.. CIDEr 61.6 vs. ours 126.9 on COCO image captioning) because the image tokenizer captures too low-level information. Emu (Sun et al. 2023b) employs continuous visual representations and is pre-trained on interleaved multimodal sequences through classifying the next text token or regressing the next visual embedding. For image generation, Emu further fine-tunes a SD model to accommodate the output representations from the LLM. By contrast, we pre-train discrete image tokenizer, where the visual codes can be decoded to realistic images using the off-the-shelf SD model, and perform multimodal autoregressive with a unified wext-word-predictie objective, which facilitates scalable multimodal training.
Visual Tokenizer. Visual tokenizer aims to represent images as a sequence of discrete tokens. Previous work (Van Den Oord et al., 2017: Ramesh et al. 2021 Esser et al. 2021 Gu et al.) 2022) trains a Vector Quantized Variational AutoEncoders (VQ-VAE) by reconstructing image pixels, which captures only low-level details such as color, texture and edge. Beit v2 (Peng et al. 2022) trains a visual tokenizer through reconstructing high-level features from the teacher model, but its visual codes from 2D features of vision transformer (Dosovitskiy al. 2020) incompatible with the unidirectional attention in dominant LLMs for image generation. By contrast, we present SEED tokenizer, which produces discrete visual codes with 1D causal dependency and high-level semantics.
3

---

Published as a conference paper at ICLR 2024
SD Decoder
SEED Dokooeen
Generation Embedding
Reconstruct
Generated Image ge
Causel Codes
Reconstruct
Codebook
Contrastive ....
Text Embeddings
Image Embedding
Embeddings
Causal Q-Farmer
Original Image
VIT Encoder
Encoder
Image Encoder
Learned Queries
SEED Tokenize
"A dog sits on the grass"
Figure 3: Overview of SEED tokenizer, which produces discrete visual codes with causal dependency and high-level semantics. The generation embedding from visual codes can be decoded to realistic images with the frozen unCLIP (Ramesh et al. (2022) SD, which is conditioned on image embedding.
3 METHOD
3.1 SEED TOKENIZER
As shown in Fig. B. the SEED tokenizer consists of the following components for tokenization and
de-tokenization:
(1) ViT encoder (Dosovitskiy et al. 2020), which is derived from the pre-trained BI.IP-2 (I.i et al.
2023c) and is frozen, providing visual features the inputs of the Causal Q-Former.
(2) Causal Q-Former, which is trained (Enq. D to convert 2D raster-ordered features (16x 16 tokens)
produced by the ViT encoder into sequence of causal embeddings (32 tokens).
(3) VQ Codebook (Van Den Oord et al. 2017), which is trained (Enq. 2 to discretize the causal embeddings from Causal Q-Former quantized visual codes (32 tokens) with causal dependency. (4) Multi-layer perceptron (MLP), which is trained (Enq. 3 to decode the visual codes from VQ Codebook into generation embedding (1 token) for the alignment with the latent space of the pre- trained unCLIP-SD (Ramesh et al., 2022) conditioned on image embedding.
(5) UNet decoder (Ronneberger et al. (2015). which is derived from the unCLIP-SD to encode realistic images from the generation embedding, and is kept frozen.
We pre-train SEED tokenizer on image-text pairs including CC3M (Sharma al. 2018), Unsplash
(Luke Chesser 2023), LAION-COCO (Christoph et al.) 2022) and MS-COCO Chen et al. 2015).
3.1.1 TRAINING STAGE I: CAUSAL Q-FORMER
we first train Causal Q-former extract fixed number output features with causal dependency from from the image encoder through contrastive learning. Specifically, a set number of learnable query embeddings (32 tokens) and features of a pre-trained ViT encoder (Sun) et al. 2023a) are fed into the Causal Q-former to encode fixed number of causal embeddings (32 tokens) of the input image. The query embeddings
As shown in Fig. 3
interact with only previous queries through self-attention layers with causal mask, and interact with frozen image features through coss-attention layers. We adopt contrastive learning to maximize the similarity between the final causal embedding è and text features / of the corresponding caption, while minimizing the similarity between the final causal embedding and text features of other captions in batch, using Noise-Contrastive Estimation Oord al. (0018) below.
expl b/T)
exp(t] C:/T)
Leo -log
log D exp(t 6//7)
(1)
D exp(x t//T)
where B is the number of the hatch size and the learnable temperature parameter.
4

---

Published as a conference paper at ICLR 2024
processing
SEED Tokenize
SEED De- Te-enkenize
It usually
takes
half
month from bud to bloom
IMG
Multimodal LLM
Nv-S
Fr-2
N+L
AMO
7
IMB
Ni-I
Ir/2
/IVG
usuelly
takees
half
a
month
from
bud
bloom
Figure 4: Overview of the multimodal autoregressive pretraining interleaved visual and textual data for SEED-LLaMA. Visual inputs are pre-processed into discrete tokens to conserve computational resources. Given the multimodal discrete sequence. unified next-word-predieee objective is em- ployed. During inference, visual codes are decoded into a realistic image by SEED De- Tokenization.
3.1.2 TRAINING STAGE II: VISUAL TOKENIZE AND DE-TOKENIZE
As shown in Fig. we train VQ codebook to discretize the causal embeddings (32 tokens) into quantized visual codes (32 tokens) for visual tokenization, and MLP to decode the visual codes into generation embedding codebook is set as 8192. Specifically, causal embeddings from the frozen Causal Q-Former are first fed into a linear projection layer to reduce the dimensions as C" for improving codebook utilization following previous work Peng et al. (2022). A quantizer looks up the nearest neighbor embeddings Z in the codebook for C" and obtains the corresponding codes. We employ decoder, which is a multi-layer Transformer (Dosovitskiy
token) for visual de-tokenization. The vocabulary size of the VQ
al. 2020). to reconstruct the continuous causal embeddings C from discrete codes as Creeon We employ a MI.P to reconstruct the image embedding I (1 token) of a frozen unCLIP-SI from discrete codes as generation embedding G. As shown in Enq. 2 during training, besides the original VQ objective Van Den Oord al. (207)) (the first and second term). we maximize the cosine similarity between reconstructed causal embeddings Crecon and causal embeddings (the third term), and minimize the MSE loss between generation embedding G and image embedding of unCLIP-SD (the fourth term).
Lvo = || C*) 2 2 + ||C' sg| sg[ |
cos(C reeen CC) + LMIP
(2)
CMLP
MSE(G,
(3)
where denotes the stop- coopggradeent operation. During inference, the generation embedding G are fed into the off-the-shelf SD-UNet to decode realistic images.
3.2
SEED-LLAMA
3.2.
TRAINING STAGE I: MULTIMODAL PRETRAINING
As shown in Fig. SEED-LLaMA adopts unified next-word-rrediitton training objective interleaved visual and textual data. Specifically, visual inputs are first discretized into a sequence of causal codes by SEED tokenizer. Then the interleaved visual codes and text tokens fed into the pretrained LLM for performing multimodal autoregression, where the visual codes are treated as new words and the vocabulary of the LLM is updated accordingly. We maximize the likelihood in unified autoregressive manner as follows:
L(U) Σ log P (w: Ui-kk 1;
where u, represents visual code or text token, and & denotes the the parameters of the transformer. We initialize SEED-LLaMA from a pre-trained LLM, and add 8192 visual codes to the vocabulary. The embedding layer and decoder head layer in the transformer are expanded and the parameters of added visual codes randomly initialized.
For efficiency, we first train LLED-LLLMA using LoRA (Hu et al. 2021 tuning and together optimize the parameters of the embedding layer and decoder head layer due to the added visual codes. We

---

Published as a conference paper at ICLR 2024
Table 1: Evaluation of Image-Text Retrieval. Causal codes are quantized causal embeddings.
Flick33KK (IK test set)
COCO (SK lest set)
Model
Image + Text
Text - Image
Image + Text
Text - Imagee
R@
R@5
R8:0
R@1
R@5
R@10
R@m
R@1
R@5
R@ 10
R@1
R@5
R
R@m
BLIP-2 alal 20236
98.4
99.7
96.5
92.9
95.3
82.7
SEED (causal embeelingg)
91.0
99.5
100.0
79.3
94.8
97.1
93.6
74.2
93.1
96.7
59.0
82.8
89.2
82.5
SEED (causal code)
85.4
98.3
99.6
73.7
92.3
95.7
90.8
66.9
89.3
94.4
53.2
78.8
86.6
78.2
Imput
Reconstructio
Figure 5: Reconstruction images of SEED tokenizer (i.e., original image + SEED tokenize + causal visual codes ++ SEED de-tokenize >> reconstructed image).
then merge the parameters of LoRA onto the LLM backbone and fine-tune all parameters except for the embedding layer. We freeze the embedding layer since we observe that fine-tuning it together with other parameters can lead to unstable training loss, which is also reported in BLOOM (Scao) al. (2022) and GLM-1300 (Zeng et al., 2022). We preprocess the images and videos into discrete tokens beforehand to conserve computational resources. We perform pretraining using two versions of LLM, Vicuna-7B and Llama2-chat-133 with 64 A100-40G GPUs, and yield SEED-LLaMA-SB (144 hours) and SEED-LLaMA- 14B (216 hours), respectively. See Appendix, for details.
3.2.2 TRAINING STAGE II: MULTIMODAL INSTRUCTION TUNING
We perform multimodal instruction tuming on SEED-LLaMA to align it with human instructions through supervised finetuning on public datasets. The details of datasets can found in Appendix, C We fine-tune LoRA module on the pre-trained SHED-LLaMA with the template as below,
USER:
<Instruction> ASSISTANT:
<Answer>
(5)
Only the content of <Answer> is accounted for loss. The overall instruction tuning phase takes 16 hours for SEED-LLaMA and 27 hours for SEED-LLaMA-144 with 32 A A100-80G GPUs.
3.3 SEED TOKENIZER
EXPERIMENT
Evaluation of Causal Embeddings. We evaluate the performance of Causal Q-Former the image- text retrieval using COCO (Lin et al.) 2014) and Flickr30K (Young et al. 2014). The performance measured by Recall@ (R @ K). Note that adopt the dual-stream paradigm for inference and remove the image-text-matching (ITM) re-rank module in BLIP-2 for fair comparison. As shown in Tab. our Causal Q-former achieves better results than BLIP-2 in terms of an aggregated metric Recall @ mean. It demonstrates that the output query embeddings with causal dependency do not drop performance than the output embeddings with bi-directional attention in BLIP-2.
Evaluation of Causal Codes. We evaluate causal codes on the image-text retrieval, where the reconstructed embeddings from causal codes used for retrieval. As shown in Tab. [ discrete codes exhibit competitive performance compared to BLIP-2, which demonstrates that the discrete codes from SEED tokenizer capture high-level semantics, which are suitable for visual comprehension.
We visualize the reconstructed images of SEED tokenizer in Fig. Through obtaining the generation embedding from the causal visual codes, realistic images can be generated using the frozen SD-UNet, which maintain consistent semantics with inputs. The above evaluation and visualization demonstrate the versatility of SEED visual tokens for both comprehension and generation tasks.

---

Published as a conference paper at ICLR 2024
Table 2: Comparison for multimodal comprehension. "Image Gen" denotes whether the model generate images besides texts, and "-I" denotes the instruction tuned model. The best results are bold and the second best are underlined.
Image:
Madels
Size
Image Gen
SEED Bench
COCO
VQAv2
OKVQA
VizWiz
MSVDQA
MSRVTTQA
NEXTQA
Flamingo (A (Alaymic ml
2022
9B
79.4
51.8
44.7
28.8
42.7
30.2
13.7
23.0
BLIP-2 (Li
InstructBL
LLrel a
2023c
8.1B
34,5
58.8
41.8
22.1
Kosmos-1
1.68
84.7
$1.0
29.2
Kosmos-2
Feng al.
2125
1.68
45.6
$4.4
MctaLLM
1.78
82.2
IDEFICS
mitreliaa ot
2023
808
91.8
60.0
45.2
36.0
IDETICS
80B
117.2
37.4
36.9
26.2
53.2
CM3 con
Talt al. 2023
7B
61.6
47.6
23.8
37.6
Emu
202.30
148
112.4
52.0
38.2
34.2
47.3
18.8
8.3
19.6
Emu-l
Sun et
20233
14B
117.7
40.0
34.7
35.4
58.0
32.4
14.0
6.8
SEEDALLOMA
51
123.6
44.2
29.2
21.5
42.2
11.5
5.0
14.3
SEED+LLaMA-1
8B
124.5
66.2
45.9
55.1
51.5
40.9
30.8
24.9
SEED&LLaMA
148
125.0
48.1
27.1
23.3
46.0
13.9
3.7
11.3
SEED-LLaMA-1
14B
126.9
63.4
43.2
49.4
53.7
45.2
35.3
24.7
you generate image cartoon garcen?
lolk succl Let's nan ca Cookic Can Coolis road book??
You, it's the blcsom
there any Itable country for traveling see this scene!
cherry losserr,, and diatiaation for bourbil dusing the scring seasons can placos the binerom bloom.
looks like Coolie evors readieg Now make Cookie cook food kichen.
Can you show me dese piotre thès flower?
Eoodde troks lie euzelent bett
pourtie plying plane.
looks so bow.ttffll Can you wiiee poem about this Towe
Sure, Pere' ca powm foryou:
cheerer Noveom,
low these paintingt you write about Cookee/
Your potal's cance broast to caro.
Your ginkk and white luos, light sec. Jembbot seriing
Once upen time, theme curteus adverturuu
found
dosided
You bioonn just wook twek.
Asshe vadd sie disowored la love for learning and deciddd to become a chet She pracices cooling and baking entry day and somvetting
But poist lipeett worth lifetine kinow.
bring boy
became
Sit'ss
Sie
up piano strted alainn with her ranura talent, the becaee fono plonid entertated everpone withiee become sucnesshe sowia, stventueee
cherish your beauts, though
you.
fute-
Your memory will Enrecer mode
Figure 6: Qualitative examples of multi-turn pulti-turr in-context image and text generation by SEED-LLAMA given multimodal instructions.
4.1
SEED-LLAMA
4.1.1
QUANTITATIVE EVALUATION
Multimodal Comprehension. We evaluate SEED-LLaMA on a wide range of multimodal com- prehension tasks including image captioning and image/video question answering. Details of these
7

---

Published as a conference paper at ICLR 2024
Impus Prompt
Stylized Imagee Generation
boat the lake
Generation
Ingut Prompt
Generation
Promot
Generation
Image Blending
Input Prompt
snowy
Multimodal Composition
In-context Generation
Imput Prompt
The animal the sitting the enviranment of Image.
Generation
This is the first image.
This the second image.
Figure Qualitative examples of compositional image generation by SEED-LLaMA.
benchmarks and evaluation metrics are provided in Appendix, [ As shown in Tab, 2 our SEED- LLaMA achieves competitive performance in both the image and video understanding tasks compared with MILMS that use continuous visual representations. The results demonstrate tokenizer can generate discrete visual codes with high-level semantics, which facilities the visual comprehension. We can observe that pretraining from a LLM with larger model size improves performance SEED-Bench and instruction tuning further contributes enhanced results. Note that as pointed out by recent work (Liu et al., 2023c Li et al., (023b), previous VQA benchmarks listed in Tab. are not tailored for evaluating MLLMs with open-from output, since they require exact match between the model prediction and the target word or phrase. The qualitative examples of multimodal comprehension provided in Appendix,
our SEED
Text-to-image Generation. We evaluate the text-to-image generation MS- IS-COCO (Chen et al. 2015) and Flickr30K (Young et al. 2) and compute the pair-wise CLIP similarity score as the evaluation metric following GILII. (Koh al. 2023b). As shown in Tab. TO images generated by SEED-LLaMA from textual descriptions show higher similarity with the ground-truth images. The results demonstrate that SEED-LLaMA generates images that are highly correlated with text prompts via frozen SD-UNet. We show qualitative examples of text-to-image generation in Appendix.
4.1.2 EMERGENT ABILITY
Multi-turn In-context Multimodal Generation. As shown Fig. and Fig. 6 given multimodal instructions including images and open-form texts from a SEED-LLaMA respond with synthesized image (e.g.. dog in front of the Golden Gate Bridge). sequentially generated images (e.g., cartoon cat in different scenes), instruction-followed image (e.g., closer look-up of cherry
8

---

Published as a conference paper at ICLR 2024
blossom), various forms of texts via creation and real-world knowledge (e.g., story, a poem and flower identification). The results illustrate the impressive capability of SEED-LLaMA in reasoning and generating long-context multimodal content.
Compositional Image Generation. As shown in Fig. our SEED-LLaMA can realize a variety of zero-shot compositional image generation below,
Stylized Image Generation. SEED-LLaMA can take a text prompt and style reference image as inputs and produce output image that adheres to both the style and text prompt.
Image Blending. SEED-LLaMA can take two images as inputs and generate an image that blends the visual components of the input images.
Multimodal Composition. SEED-LLaMA can take an image prompt and a text prompt as inputs and generate composite image that combines the multimodal inputs.
In-context Generation. SEED-LLaMA can take images, their textual references, and text prompts as inputs and generate context-related images.
4.2 ABLATION STUDY
Generation Embedding. The generation embedding of SEED is aligned with the image embedding of unCLIP-SD, and can be decoded realistic images with the AMCLIP-SD-UNE tokenizer SEED text through aligning the generation embeddings with the text embeddings (77 tokens) of SD (Rombach et al. 2022) conditioned on texts. As shown in Tab. [O] the similarity between the reconstructed images of SEED text and original images drop heavily. The semantic representations of texts can not fully preserve the rich visual information of images. The visual comparison the the reconstructed images between SEED and SEED are provided in Appendix. A
We also train visual
Causal Visual Codes vs. Bilateral Visual Codes. We train a Causal Q-Former to convert 2D features produced by the ViT encoder into a sequence of causal semantic embeddings, which are further discretized as causal visual codes. To verify whether the causal visual codes are necessary for compatibility with LLM, we train a visual tokenizer SEED which produces bilateral visual codes from a pre-trained Q-Former with bilateral self-attention. We then pre-train SEED -LLM" and SEED-LLM" on image-text pairs and evaluate the text-to-image generation COCO test set. Given 5000 captions of COCO, SEED -LLM only generates 2134 images successfully while SEED-LLMM generates 4997 images (Failure cases occur when the model predicts a number of visual tokens not equal to 32). The results demonstrate that the non-causal codes lead to highly unstable model performance since they contradict with the left-to-right autoregressive mechanism of LLM.
SEED-LLaMA Pretraining. We first train SEED-LLaMA using LoRA tuning, and then merge the pa- rameters of LoRA with the original LLM and fine-tune all parameters except for the embedding layer.
To explore whether fully fine-tuning helps, evalu- ate the performance of the model before and after fully fine-tuning image captioning and text-to-image -image generation, with evaluation metric CIDEr and clip similarity score. Tab. 3 shows that fully fine-tuning the LoRA tuned model enhances model's capability for both image comprehension and generation.
Table 3: Evaluation of image captioning and text-to-image generation COCO test set.
Pretraining
Captioning
Generation
LoRA
124.5 125.0
68.87
LoRA + Fully
69.07
CONCLUSION
We present SEED, a discrete image tokenizer, designed based on the premise that visual tokens compatible with L.MSs should capture high-level semantics while being generated with ID causal dependency. SEED enables L.LMs to be trained with multimodal data following the original recipe of text (i.e., next-word prediction), which is mature and scalable. We further present SEED-LLaMA via multimodal pretraining and instruction tuning SEED tokenizer. SEED-LLaMA not only exhibits remarkable performance across multimodal comprehension and image generation tasks, but also demonstrates extensive compositional emergent abilities. We hope that SEED would draw increased attention to visual tokenizers. A more rational visual tokenizer could substantially reduce the complexity of multimodal LLM training.
the interleaved visual and textual data with

---

Published as a conference paper at ICLR 2024
REFERENCES
lean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 23736, 2022.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1728 1738, 2021.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pp. 190-2000 2011.
Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Schuhmann Christoph, Köpf Andreas, Vencu Richard, Coombes Theo, and Beaumont Romain. Laion coco: 600m synthetic captions from laion2b-en. [EB/OL], 2022. http://lli..aaaoolo//oooooo/ooo/
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 14:11922-19983 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint 11X1:20010.11929, 2020.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03377 2023.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 12873-12883 2021.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. In European Conference Computer Vision, pp. 89-106. Springer, 2022.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficien visual instruction model. arXiv preprint 2304v /5010, 2023.
10

---

Published as a conference paper at ICLR 2024
Yash Goyal, Tejas Khot, Douglas Summers-Ster Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6900-6901199 091446913,200 2017.
Yuchao Gu, Xintao Wang, Yixiao Ge, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Rethinking the objectives of vector-quantized tokenizers for image synthesis. arXiv preprint arXiv:2212.03185, 2022.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2018.
Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models general-purpol interfaces. arXiv preprint 220..063666. 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0960 09685, 2021.
Shaohan Huang. Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302. 14045, 2023.
Ting-Har Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pp. 1233-1239, 2016.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. arXiv preprint arXiv:2305. 2023a.
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. arXiv preprint arXiv:2305.17216. 2023b.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang. Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023a.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023b.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman pre- training for unified vision- language understanding and generation. In International Conference on Machine Learning, pp. 12888-12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: 2: Bootstrapping language-iman -mmgg pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 125599 2023c.
11

---

Published as a conference paper at ICLR 2024
Yu Lili, Shi Bowen, Pasunuru Ram, Miller Benjamin, Golovneva Olga, Wang Tianlu, Babu Arun, Tang Binh, Karrer Brian, Sheynin Shelly, Ross Candace, Polyak Adam, Howes Russ, Sharma Vasu, Xu Jacob, Singer Uriel, Li (AI) Daniel, Ghosh Gargi, Taigman Yaniv, Fazel-Zarandi Maryam, Celikyilmaz Asli, Zettlemoyer Luke, and Aghajanyan Armen. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. 2023.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning, Transactions the Association for Computational Linguistics, 1:635-651, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.0844 2023b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li. Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06200 2023c.
Ali Zahid Luke Chesser, Timothy Carbone. Unsplash. https//gghhub.commusupshsssddda
2023.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.0544 055244 2023.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp. 3195-320000 9994,,20,,77 2019.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947-952 952. IEEE, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03744. 2018
Junting Pan, Keqiang Sun, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative image understanding. arXiv preprint arX:h:2307.007111 2023.
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208. 06366, 2022.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models the world. arXiv preprint arXiv:2306. 14824, 2023.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot texl-to-image generation. In International Conference on Machine Learning, pp. 8821-8833 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/15, 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 241. Springer, 2015.
12

---

Published as a conference paper at ICLR 2024
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXh::2211.051000 2022.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pp. 146-162, Springer, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 2556-2565, 2018.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanprect Singh. Textcaps: dataset for image captioning with reading comprehension. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 11 16, pp. 742-758. Springer, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303.15388 2023a.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint (iti::2000.05222 2023b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprini arXiv:2302.3397, 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Zijie J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. arXiv preprint arXiv:22100 10. 10210./4890 14896, 2022.
Shenggiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal Ilm. Xrxiv preprint arXiv:2309.055//1 2023.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 9777-9786. 2021.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5288-5296 2016.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1686-1697, 2021.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. /4/78, 2023.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
13

---

Published as a conference paper at ICLR 2024
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. GIm-1306 An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414, 2022.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: manually annotated dataset for instruction-guided image editing. arXiv preprint arXiv:2306. 10012, 2023a.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint wwx.v:2303./6199 2023b.
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint TrXiv:2306.17107 2023c.
Bo Zhao, Boya Wu, and Tiejun Huang. Svit: Scaling up visual instruction tuning. arXiv preprint arXiv:2307.04087, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vasion-language understanding with advanced large language models. arXiv preprint 105552 2023a.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. arXiv preprint 2rXiv:2304.06939 2023b.
14

---

Published as a conference paper at ICLR 2024
(b)
(c)
Figure 8: (a) Input image. (b) Reconstruction images of SEED tokenizer, which is aligned with the feature space of a SD conditioned on text embeddings. (c) Reconstruction images of SEED tokenizer, which is aligned with the feature space of SD conditioned on image embedding.
A SEED TOKENIZER
The generation embedding of SEED is aligned with the image embedding of unCLIP (Ranesh et al. 2022) SD, and can be decoded to realistic images with the tokenizer SEED through aligning the generation embeddings with the text embeddings (77 tokens) of SD (Rombach et al., 2022). and the generation embeddings can be decoded to images with the SD-UNet. The visual comparison of the the reconstructed images between SEED and SEED shown in Fig. 8 We can observe that compared with SEED text the images reconstructed by SEED can better preserve the visual information of the original images.
We also train a visual
B PRETRAINING
B.1 PRETRAINING DATA
As shown in Tab. we utilize diverse categories of datasets as pretraining data, which can be summarized as follows.
2018). Unsplash (Luke Chesser 2023). LAION-COCO (Christoph et al. 2022) and MS-COCO (Chen et al. 2015). filtered the samples these datasets based on image resolution, aspect ratio, and vixtal-textual similarity. We randomly place images or text at the forefront, in order to achieve the generation of captions based on images and vice versa.
Image-text Pairs. We use the image-text pairs from CC3M (Sharma
Video-text Pairs. We use large-scale dataset VibVVddInn 10M (Bain et al. (021]) containing videos and captions. We implemented heuristic rules to exclude extraneous metadata, such as the resolution of the original video and camera parameters. We sample four frames each video for training. Each frame is discretized into 32 codes. The input sequence of video is <IMG> the token of frame I </IMG> <IMG> the token frame 2 </IMG> <IMG> the token of frame </IMG> <IMG> the token of frame 4 </IMG>.
Interleaved Image and Text. We publicly available MMC4 Zhu et al. 2023b) and OBELISC (Laurençon et al. 2023) datasets, which were extracted and thoroughly filtered from Common Crawl. Specifically, employ the MMC4-core split, consisting of 7.3 million samples, and the complete OBELISC dataset. containing 141 million samples. For documents in MMC4, we create a sequence of length 1024 and randomly shuffle the order of images and their corresponding texts (those with the
15

---

Published as a conference paper at ICLR 2024
Table 4: Description of pretraining datasets of SEED-LLaMA.
Dataset Name
Dataset Description
COCO Caption
0.5M image-text pairs with human- captions. Specifically, Karpathy train split is used.
Chen et al.
2015
CC3M
3.3M image-text pairs from the web.
Sharma et al.
2018
Unsplash
4.8M image-text pairs, in which images are composed of high-quality
Luke Chesser
2023)
Unsplash photos.
LATON-COCO
600M image-text pairs, where the caption is generated by the BLIP
Christoph et al.
2022
(Li et al. 2022).
101M irage-interleaved documents collected from Common Crawl. We use the mmc4-core split which is consist of 7.3M documents. We randomly shuffle the order of images and their corresponding text (those with the highest CLIP score).
MMC4
(Zhu et al.
2023b
OBELISC
141M amage-interleavve documents collected from Common Crawl.
Laurençon et al. 2023)
8M video-text pairs, we have implemented heuristic rules to exclude extraneous metadata,such as the resolution of the original video and camera parameters.
WebVid
Bain al. 2021
Table 5: Summary of pretraining hyperparameters
Configuration
SEED 8B
SEED 14B
Vision encoder
EVA-CLIP
LLM
Vicuna-7B
LLaMA2- 13BB
Training Strategy
LoRA + Fully fine-tuning
Peak learning rate
1.5e-4
Warmup ratio
0.03
LR schedule
Cosine decay Adam W
Optimizer
Optimizer hyper- paraaeerrre
B1,32, 6= 0.9, 0.98, le-6 224 224
Image resolution
Weight decay
0.05
Iterations
30k + 10k
(MS-COCO, CC3M, Unsplash), LAION-COCO, OBELISC, MMC4, WebVid 160, 128, 1024, 1024, 200
Data
Sequence length per dataset Batch size per dataset
146, 180, 26, 26, 116
46, 56, 8, 8, 36
Sample ratio per dataset
4.5%, 54.5%, 9.1%, 27.3%, 4.5%
highest CLIP score). As for OBELISC, generate sequence of length 1024 based the order of data in the dataset.
B.2 PRETRAINING HYPERPARAMETERS
We report the detailed pretraining hyperparameters of SEED-LLaMA in Tab.
16

---

Published as a conference paper at ICLR 2024
C INSTRUCTION TUNING
We summarize the datasets and their prompts for supervised instruction tuning of SHED-LLaMA in Tab. [ and Tab. 7 Note that MagicBrush (Zhang et al. 2023a) contains both the single-turn and multi-turn scenarios, and we only use the single-turn for multimodal prompt image generation.
D EVALUATION
D.1 BENCHMARKS
In order assess the multimodal comprehension and image generation ability of SEED-LLaMA, we evaluate SEED-LLaMA on 10 benchmarks shown in Tab. 8 For the evaluation of image generation, we adopt the CLIP-ViT-L/IA generated image. When evaluating SEED--Bench, we adhere to the official guidelines, selecting the option with the highest log likelihood as the response for each multi-choice question (MCQ). For the evaluation video tasks, uniformly sample frames for MSVDQA and MSRVTTQA, and 8 frames for NEXTQA. For the other tasks, we follow the evaluation procedures prior works (Li al. 2023c Sun et al. 2023b) and either submit the results to the official server (VQAv2, VizWiz) or assess them using the official code ourselves.
calculate the CLIP score between the ground-truth image and the
D.2 PROMPT TEMPLATES
We summarize the prompt templates used for evaluating SEED-LLaMA in Tab. 9 As the pre-trained SEED-LLaMA with size of 8B and 14B adopt different LLM (Vicuna-7B and Llama2-chat- their prompts differ accordingly.
D.3 EVALUATION OF IMAGE RECONSTRUCTION
We further evaluate image reconstruction COCO and Flickr30K dataset. SEED first discretizes input images into causal codes (32 tokens) and obtain generation embedding (1 token), which are fed into the unCLIP-SD-UNe for reconstruction. We follow GILL (Koh et al. 2023a) to compute the CLIP similarity score as the metric to evaluate the semantic consistency. As shown in Tab. I0 compared with the upper bound unCLIP-SD, SEED only slightly drops performance.
E QUALITATIVE COMPARISON
For the qualitative comparison of multi-turn in-context image and text generation, as shown in Fig. we can observe that our SEED-LLaMA exhibits exceptional emergent ability in generating long-context multimodal content, while Emu [Sun al. al.] (2023b) and Next-GPT Wu al. (2023) that employ continuous representations to unify comprehension and generation within LLM show less competitive performance. They generate images with incorrect semantics or context, or can not follow instruction accurately in multi-turn dialogue.
F QUALITATIVE CASES
More examples of multi-turn in-context multimodal generation and compositional image generation are shown in Fig. 10 and Fig.
I1 Note that generating images with multimodal prompt is not an
emergent ability since SEED
LaMA is fine-tuned on corresponding paired data such Instruct-
Pix2Pix (Brooks et al.) 2023). We showcase qualitative examples text-to-image generation by
SEED-LLaMA can generate realistic images that aligns with the text prompts. We further provide qualitative examples of multi- modal comprehension by SEED-LLaMA in Fig. [3 Fig. 14 and Fig. [5] SEED-LLaMA can realize in-context multi-imas understanding, real-world knowledge grounding, complex reasoning, story creation and video understanding.
SEED-LLaMA in Fig. 12 Given various textual descriptions,
17

---

Published as a conference paper at ICLR 2024
Table 6: Description of datasets in the instruction tuning of SEED-LLaMA.
Task
Dataset Name
Dataset Description
Type
JourneyDB
It contains 4429K Midjourney images, with text prompt, image caption, and QA pairs.
Single-turn
(Pan 2023
DiffusionDB
It contains 14 million images generated by Stable Diffusion using prompts by real users.
Single-turn
Texl-to-Image
(Wang et al.
2022
Generation
LAION-Aesthetics
It contains several collections of subsets from LAION 5B with high visual quality.
Single-turn
VIST
It contains photos in 20K sequences, aligned to both caption and story language.
Multi-turn
(Huang et al.
2016
Instructpix2pin pix
contains text editing instructions and the correpoondigg images. with 454K samples.
Single-turn
Multimodal Prompt Image
(Brooks et al.
2023
Generation
MagicBrush (Zhang al.
It contains 10K manually annotated triplcts (source image, instruction, target image).
Single-turn
2023a
LLaVA
We use 58K multi turn conversations between assistant and a person.
Multi-turn
(Liu et al.
2023b
Image
SVIT
It contains conversations, complex reasoning, referring QA and detailed image description.
Multi-turn
Conversation
(Zhao et
2023
LLaVAR
It contains 16K multi-turn conversations, cach with QA pairs for text-rich images.
Multi-turn
(hhag al.
2023c
Multi-Image
GSD
It contains 141K pairs of images with text describing the differences.
Single-turn
Understanding
(Li et al.
2023a
VSR
contains texts describing spatial
Single-turn
(Liu et al.
2023a
relations in the image, with 7K training samples.
Image Captioning
COCO Caption
contains image-text pairs with human- captions, with 82K training samples.
Single-turn
(Chen ct al. 2015)
TextCaps
It requires the model comprehend and reason the text in images, with 21K training samples.
Single-turn
Sidorov et al.
2020
VQAv2
A dataset for open-ended image question answering. with 82K training samples.
Single-turn
(Goyal al.
2017
OKVQA
It contains questions that require outside knowledge
Single-turn
(Marino 2019
with 9K training samples.
A-OKVQA
It is successor of OKVQA containing more challenging questions, with 17K training samples.
Single-turn
(Schwenk al.
2022
GQA
It contains questions for image understanding and reasoning, with 30K training samples.
Single-tur
Image QA
(Hudson & Manning 2019
VizWiz
contains visual questions asked hy people who are blind, with 20K training samples.
Single-turn
(Gurari et al.
2018
TextVQA
contains questions that require models read text in the image, with 800K training samples.
Single-turn
(Singh et al
2019
OCR-VQA
It contains questions that requires reasoning about text to answer, with 173K training samples.
Single-turn
Mishra et al.
2019
Video
Video-ChatGPP MMaz al. 2023
It contains of 100K video-instruction pairs via manual and semi-automated pipeline.
Single-turn
Conversation
ActivityNet (Caba Heilbron al. 2013
It contains 200 different types of activities from YouTube, with 10K training videos.
Single-turn
Next-QA
It contains 52K QA pairs of videos grouped into causal, temporal and descriptive questions.
Single-turn
(Xiao et al. 2021
MSVD
It contains videos from YouTube with descriptions, containing 1.2K training samples.
Single-turn
Video QA
[Chnn & Dolan 2011
MSR-VTT
contains videos from YouTube with descriptions, containing 19K training samples.
Single-turn
(Xu et al. 2016
iVQA
is vidco QA dataset with mitigated language biases, containing 6K training samples.
Single-turn
Yang et al. 2021
18

---

Published as a conference paper at ICLR 2024
Table 7: Details of prompt templates used in supervised instruction tuning of SEED-LLaMA.
Type
Prompt
Text-to-Image Generation
USER: [caption)
Please generation an Image.nnASSIITTANT: (image)
Multimodal Prompt
USER: [imagel] {instruction) Please generation an image.
Image Generation
VASSIITANT: (image2)
Image Conversation
USER: (image) [question] /AASSISTANT: (answer)
Multi-Image
USER: This is the first image. [imagel) This is the second image.
Understanding
(image2) (question) IaA.:I//ASSSSSSANNIAAI (answer)
USER: [image] Please provide an accurate and concisedescription of the given image. IASISSTASTT:TT: [caption)
Image Captioning
USER: (image) [question) Please provide an accurate answer consisting
Image QA
of only one word or phrase..nASSSSSAA ANSSSTTANT::: [answer]
Video Conversation
USER: [vidco) (guestiion//AASSSSSAAA)
[answer)
USER: (video) {question) Please provide an accurate answer
Video QA
consisting of only one word or pbrase..nASSSSTTT. AnASSISAAN: (answer)
GPT LL
Goidin
ky bal Ling dug bacls slng
oeplovent
glee dedeer dng
Bane are glaene
SOTD ALARAA
Meviir
mhy late mun alon and
iries ta wandenn arand
tes prrniing dirigg Canyre
budyy' pevier tooiiiin arto
pleery the anddand dissing
loald olglas the beady
Can covi candalaad
uie inddedt appncst
and sow Bhy or lou duly lave camae plaa bects
Figure 9: Comparison of multi-tum co-tentext image and text generation between and MLLMS that unify comprehension and generation.
19

---

Published as a conference paper at ICLR 2024
Lam planning Sercalona, Can recommend some
cartoon rhanacter
User
places to stat-for me?
SEED MA
Of uuusel Bancalona bcautiful city with offer.
Hore are some places should definitcly wiatt
1. La Sagrade tam loc This a famous cathedda designed by
Antoni Souct It's must see attractio Barcelono.
2.
unique srece,
great restaurent, strwet performees. tiic
thi is man photo B.
tabe
stroll and onjoy the sights and sounds.
4i Eaua Betllor Another Coucll creation, this hove lenswnn is uniquue architecture and color focide, It's must see any architecture lover
S. Barceloneta
you're looking place popudar
spot lscal fies of the hope enjoy p!
Who the A who the photo
Thanks for sour sugpestion zm interested Sagroda Familia Can you show petire?
EquardFansi,
Albett Ehistin.
you generate a cartoon image man
spcctacular going you
try
Borceiono, lar snack the chuma, which freed dough pastrs otten sprinic ed with sugar Another popular snace is the croquette. which a small ball of broading fille with moat or fash. You can also ty the pan con tomate, which simpll dish mode brewd, homatnes, and ollue oll you' feeling adventurour, you
Yes,
delidous
Imagine what kind of comorrsaion will the cartoon character and man have when they in the same room? Please give a posuble dualegue script
Camonn Charater "Hey, Fineteint i'm Spongedtth
Squaee spange who lives pineopple under stors?"
squid Borceiont Enjoy!
Enstain,
theoretical
thwory of nelativ that csuld change the understand the universe."
Jm Internded Churms you show me wheel
Cartoon Character "Whwi That sounds smaring: g Five newer been to the sea, but I've been to the bottom of the ocean in submarnee Maybe can help you with your theory! Whet need
you
that?"
Cartoon Character
bubble
travel through the water Maabe thrugh time tooi Let' give it try"" E Entton: "'n not that's the right way to about but willing anything
travel
point. we'' give
Figure 10: Qualitative examples of multi-turn in-context image and text generation by SEED-LLaMA
given multimodal instructions.
20

---

Published as a conference paper at ICLR 2024
Input Prompt
Generation
Input Promot
Generation
Make in Hong Kong
Insert a train
Multimodal Prompt Generation
Input Prompt
Generation
Input Prompt
Generation
Replace the with vegetables
Make green grass
Input Promet
Generation
Input Prompt
Generation
Color flowers while
thunderstorm
Input Prompt
Mount Rainier
A girl in front
A cartoon image
Mount Rainier moonlight.
A wooden house
Mount Rainier.
In-context Generation
Generetion
Input Prompt
animal the first image surrounded the flower in the second
Generration
first immage.
second image.
Figure 11: Qualitative examples of compositional image generation by SEED-LLaMA.
21

---

Published as a conference paper at ICLR 2024
A cybernatic locomotive on rainy day.
cat reading book in the forest.
A black shepherd wearing red beret.
dog wearing bike.
Renalissace-sstyl portrait space, detailed background, refiective helmet.
photo meal, with shallow depth field, elegant plating, and soft lighting.
Aerial photography winding river through auturn forests, with vibrant red and orange foliage.
Cyberpunk cityscape with towering
skyscrapers, neon signs, and flying cars.
cybernetic soldier, enhanced with advanced wnapons systems tactical analysis software,
couple beach, walking hand-in-hand, with the sun setting over horizon and crashing their feet.
Portrait of young nordic age 25, freckled skin, neck tatoo, eyes 35mm lens, photography, ultra details.
captivating a seed cracking open and sprouting under sunlight.
Two flamingos are in love in a sunset, concept high quality.
and inviting library filled with books reading
soaring hot air balloon flosting over landscape.
Kingfisher bird, by Akihito Yoshida, Cool Color
Figure 12: Qualitative examples of text-to-image generation by SEED-LLaMA.
22

---

Published as a conference paper at ICLR 2024
DO NOT PASS
This first image. What's the difference??
This the second
What should I do if see this sign?
you sign, you pass. important to follow traffic signs and signals to ensure your safety
image field while the second image shows zebra standing grassy
running
of others
Tell me the differrnce between these two species detail.
Horses zebras are both members Equidae family, but they belong to different species. that bred for thousands of years for their strength, speed, intelligence. Zebras, animals that are known for their distinctive black stripes, their surroundings and avoid predatoos. Zebras are groups
Horses domesticated
other
What is unusual about this image?
The unusual aspect of this image that features dog space suil, which not common sight.
Tell me the characters in the image.
What's picture?
SpongeBob SquarePants, Patrick Star.
image depicts painting horse. The man wearing a red jacket and a white horse and white, and it's jumping over hill. There's also background. painting quite detailed and captures the movement of the horse and the The colors jacket and white hat the rider standing out against the green the mountain. The painting is quita large, taking up significant portion image. dynamic and action packaad scene.
riding
he's holding
with the red
Which heavier?
white
The
Who he?
The the painting Napcleon Bonsparte. French military political rase to prominence during the French Revolution. military prowess and conquests of much Europe. this painting, he's depicted riding horse, which symbol his military power and conquests, The painting captures moment in history when Napoleon was at the height of his power.
known
image?
The Arc de Triomphe is in the middle of the image
Figure 13: Qualitative examples of multimodal comprehension by SEED-LLaMA.
23

---

Published as a conference paper at ICLR 2024
What's the name those characters?
This the first
This the second
Jerry.
Tell based these 3 images detail.
Explain detail What they are doing.
In the image, Tom and Jerry are engaged in comical chase
Tom, the holding knife and appears to be chasing Jerry, the mouse, who is trying to evade him. The knife in Tom's hand adds element of danger and tension the scene, suggesting that the chase is not just playful
This is the third image.
Once upon a time, in far-off galaxy, there was lonely dog named dreamed of being astronaut and exploring the cosmos. One day, his wish came true when chosen be the first space. soared through the stars, he marveled at the beauty
always
potentially life-threatening Jerry. The background of the image shows living room, which adds a domestic setting to chase. Overall, the image captures moment of suspense and action in the ongaing rivalry
Jerry.
Earth, he realized that he missed the company of his fellow canines. So, set out on journey find other dogs shared space exploration. Along the way, he met friendly alien to the moon. Max eagerly accepted the offer and soon found himself standing on the moon's surface, staring the From that moment on, Max knew that he had calling would spend rest of his life exploring the wonders of the universe.
for
trip
space.
What's in this picture?
Albert Einstein.
important achievements
Albert Einstein was a theoretical physicist who widely regarded scientists of the 20th century, He best known his theory relativity and the famous E-mcc. awarded Nobel in Physics in 1921.
influential
Eiffel made metal.
Figure 14: Qualitative examples of multimodal comprehension by SEED-LLaMA.
24

---

Published as a conference paper at ICLR 2024
Can you describe this video details?
Sure, this video, we see a man riding a down street. is wearing helmet and backpack, and side the enjoying man's looks sturdy well-maintained, The street he riding appears relatively quiet, only few pedestrians visible background. Overall, this video captures peaceful enjoyable street.
street, he passes by seems be riding with
man riding his bike
How many people are the video?
There are two people in the video man and woman.
where are they?
The man worna are sitting a looking out
What they doing?
They drinking talking each other.
What unusual about this video?
One unusual aspect of the video is that the man riding small train on track in the middle of forest, which not common sight. Additionally, the train appears to be old and rusty, which adds to the unusual nature of the video.
Figure 15; Qualitative examples of multimodal comprehension by SEED-LLaMA.
25

---

Published as a conference paper at ICLR 2024
Table 8: Summary of the evaluation benchmarks.
Dataset
Task
Split
Metric
coco al. 2014
Tex--toImmage Generation
Karpathy test
CLIP score (1)
Flickr30K (Young et al.
2014
Text-to-Image Generation
test
CLIP score (1)
COCO Caption (Chen al.
2015
Scene Description
test
CIDEr (*)
VQAv2 (Goyal
2017
Scene Understanding QA test-dev
VQA acc. (f)
Image
VKVQA MMarino 2019
External Knowledge QA
val
VQA acc. (f)
VizWiz Gurari al.
2018
Scene Understanding QA
test-dev
VQA (f)
SEED-Bench (Li
2023b
Comprehensive QA
dim 1-9
MCQ acc. (1)
MSVDQA (Chen & Dolan
2011
Event Understanding QA
test
Top-I acc. (1)
Video
MSRVTTQA (Xu et al.
2016
Event Understanding QA
test
Top-I acc. (
NEXTQA Yang et al.
2021
Temporal/Causal QA
test
WUPS (1)
Table
Summary the prompting template for evaluating SEED-LLaMA.
Model
Type
Template
Image Captioning
(image)
SEED-LLaMA
(image) USE:: (question) Please provide accurate
8B
Image QA
consisting only one word or phrase. InSSISTANT::
USEd:o)USER: [question] Please provide an accurate answer consisting of only one word phrase. Aatrase.InASSSSTANT (image)
Video QA
Image Caption
SEED-LLaMA
(image) Please provide an accurate answer consisting of only one word phrase based on the image.nn
14B
Image QA
Question: Question:(question) \n Answer:
(video) Please provide an accurate answer consisting only one word or phrase based on the video.An
Video QA
Question: \n Answer:
USER: (image] provide an accurate and con-
Image Caption
SEED-LLaMA-I
cise description the given image.InASSISTANN
8B 14B
USER: |image| [question| Please provide an accurate answer
Image QA
consisting of only one word phrase.InASSISTAANT
USER: [video] [question] Please provide an accurate answer consisting only one word or pbrrase.nAASSSTAANN
Video QA
Table 10: Evaluation of Image Generation.
Model
COCO
Flickr30K
Image-to-image
unCL.IP (Ramesh al. 2022
SD
79.30
79.55
SEED text
68.23
65.22
SEED
77.35
76.52
Text-to-image
GILL Koh et al.
2023b
67.45
65.16
Emu (unn al.
2023b
66.46
64.82
LLaMA
69.07
65.54
SEED -LEM-LL
70.68
66.55
26


Reviews:
[{"review_id": "SCfwSnUuRp", "reviewer": "Reviewer_y9Sh", "summary": "Making the language model to see the words is one of the key research direction. This paper present new image tokenization on this direction. Specifically, unlike prior attempts that uses simple 2d style image tokenization (usually VQ-VAE), this paper propose SEED, which makes image embedding to be left-to-right 1d tokenization similar to the text while keeping semantic meaning of images but discarding low-level information. This paper claim that capturing too low-level information hider the performance of LLMs to effectively perform multimodal comprehension.", "questions": "See weakness section", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "(1) Unifying vision and text representation is one of the hot research topic. \n(2) The assumption behind the proposal is reasonable. \n(3) The paper is generally well-written.", "weaknesses": "(1) The methodology is not religiously explained and is not self-contained. Especially, section 3.1 is hard to follow. There are no equation, and it is hard to track which components are trained on which objective function. \n\n(2) In section 4.1, they compared SEED tokenization on image-text retrieval. As described in the paper, SEED generally outperform BLIP-2, in some case BLIP-2 exceed the proposed method. However, there are no explanation on this point. Similar criticism can be applied for the analysis on Table 3. \n\n(3) Regarding the Figure 7, I'm afraid the actual prompt is hard to imagine. \n\n(4) The proposed method contains several components, including image encoder, codebook, text encoder, and generation module. However, the importance of all components is less discussed, making me hard to access the importance of the specific choice of each component. \n\n(5) I'm afraid that I found a statement in the introduction is not fully validated. \"Moreover, we empirically found that the dominant tokenizer VQ-VAE (Van Den Oord et al., 2017) in existing works captures too low-level information for LLMs to effectively perform multimodal comprehension task\". Could you please clarify again which results support the above statement? \n\n(6) While the paper motivate to learn good 1D representation is key to incorporate visual information into pre-trained LLMs, but less discuss on why we should make the representation discrete rather than continuous. Table1 also seems to show that the continuous representation is generally better than discrete representation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OFcjdHZdiE", "reviewer": "Reviewer_i2L6", "summary": "This work introduces an image tokenizer, which is capable of discretizing images into a series of tokens. These image tokens are transformed by Q-Former into pseudo-causal tokens that can serve as the input for Large Language Models (LLMs), most importantly, they can also act as the target. This allows the model to unify both visual understanding and generation tasks.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is the first (at least to my knowledge) to use an image tokenizer to unify visual understanding and generation tasks in LLM, providing a feasible pipeline.", "weaknesses": "1. The authors claim that the use of q-former can establish a causal dependency, which I find questionable. This is because the attention in the visual encoder stage is bidirectional, leading to potential information leakage.\n\n2. Regarding the visual understanding task results shown in Table 3, why does SEED-LLaMA-I (14B) perform no better (or nearly the same) as SEED-LLaMA-I (8B) on some Image-Text Tasks? Does the proposed method not yield much gain on larger models, or has it already reached saturation?\n\n3. In Table 2, SEED-LLaMA-I achieves good results. However, I believe that CLIP similarity does not effectively reflect the quality of generation. Fréchet Inception Distance (FID) is a widely accepted better evaluation method, but unfortunately, this paper does not provide it.\n\n4. Regarding Section 4.3 (Causal Visual Codes vs. Bilateral Visual Codes), the authors mention that some mode collapse may occur for generation tasks, but what about understanding tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wCe12Ato6W", "reviewer": "Reviewer_TFPg", "summary": "The paper presents a method (SEED) to augment large language models (LLMs) with the capability of processing and generating visual data, i.e., images. The core contribution of SEED is a quantized tokenizer that learns to encode images into discrete visual tokens which can be again decoded using a pre-trained generative model. Once the tokenizer is trained, a LLM is trained and fine-tuned on interleaved image-text data such that the LLM can both process and generate the visual tokens which make it applicable to a variety of vision language tasks.", "questions": "- How did you decide on using 32 tokens per image and 8192 codes? \n- Can you confirm that you are using a start and end token for images as shown in Fig. 4? Do you use the same start/end tokens for both images and videos? This information should be included in the paper.\n\nSuggestions:\n- It would help the read flow to already mention the codebook size in section 3.1.2 instead of only in 3.2.1 (i.e., 8192 visual codes).\n- When Table 2 is first discussed in Sec. 4.1, it is unclear what $\\text{SEED}^{\\text{text}}$ refers to. A short description and reference to the ablation in Sec. 4.3. would better facilitate immediate understanding for the reader. Similarly for the \"I\" suffix referencing the instruction-tuned model.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- SEED is well motivated in learning a 1D token representation that better aligns with the auto-regressive generative process of LLMs.\n- Architectural choices are reasonable and have been validated with ablation studies to the most extend, i.e., text vs visual embedding reconstruction, embedding vs. discrete code, causal vs. bilateral visual codes, full fine-tuning after LoRA, instruction fine-tuning.\n- The quantitative evaluation convinces in either surpassing or being competitive in image-text, video-text tasks as well as generative image-to-image and text-to-image tasks.\n- The qualitative results showcase some interesting multi-modal capabilities of SEED, including compositional and in-context image generation.\n- Publishing both code and checkpoints of large-scale models enables future research and empowers the open-source community.", "weaknesses": "- Some architectural design choices are missing an explanation.\n    - In general, it would help the clarity of the paper if all loss functions would be written out.\n    - What is the reason behind using two different image encoders, one for encoding the input to the causal q-former (BLIP-2 ViT) and one for the image generation conditioning (unCLIP-SD vision encoder)? This requires loading more weights into memory during training so an explanation is needed. Can we use the unCLIP-SD vision encoder for both cases?\n    - How important is the contrastive loss between the vision and text embeddings? An ablation could justify it's inclusion.\n    - Why was the original VQ codebook loss replaced by a simple reconstruction loss with cosine similarity? How are collapsing codebooks avoided? Are there any stop-gradient operation in the loss for the codes?\n- The arguments and ablation for using causal vs. bilateral visual codes is not convincing. In general, previous work on VQ models have demonstrated that transformers can learn complex and even low-level dependencies of non-causal codes. Enforcing a causal mask in the q-former restricts the information flow to fully utilize the tokens efficiently and effectively. The argument made in Sec. 4.3 is that the LLM struggles with generating the correct sequence length for images which should always be 32. It is surprising that this happens because the start token for images (as in Fig. 4) should be a clear signal to the LLM that 32 image tokens follow. In practice however, it would be straightforward to enforce the generation of 32 image tokens after the start token is observed by restricting the possible output tokens. How do these two models compare when the number of image tokens is enforced to be always 32?\n- It is not clear how videos are being processed. Are individual frames used to train the tokenizer or are multiple frames passed to the causal q-former? If multiple frames are passed, how do you adjust the reconstruction loss for the generative embedding (1 embedding per frame from unCLIP-SD vs. one embedding per video from your tokenizer)? Do you simply append the encoding of multiple frames when passing videos to the LLM?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)