Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization
Authors: Yang Jin
Kun Xu
Kun Xu
Liwei Chen
Chao Liao
Jianchao Tan
Quzhe Huang
Bin CHEN
Chengru Song
dai meng
Di ZHANG
Wenwu Ou
Kun Gai
Yadong MU
Venue: ICLR 2024 poster (2024)
Keywords: Vision Language Learning
Large Language Model
Pretraining
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Recently, the remarkable advance of the Large Language Model (LLM) has inspired researchers to transfer its extraordinary reasoning capability to both vision and language data. However, the prevailing approaches primarily regard the visual input as a prompt and focus exclusively on optimizing the text generation process conditioned upon vision content by a frozen LLM. Such an inequitable treatment of vision and language heavily constrains the model's potential. In this paper, we break through this limitation by representing both vision and language in a unified form. Specifically, we introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language that LLM can read. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. Coped with this tokenizer, the presented foundation model called LaVIT can handle both image and text indiscriminately under the same generative learning paradigm. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously. Extensive experiments further showcase that it outperforms the existing models by a large margin on massive vision-language tasks. Our code and models are available at https://github.com/jy0205/LaVIT.

Published as a conference paper at ICLR 2024
UNIFIED LANGUAGE-VISION PRETRAINING IN LLM WITH DYNAMIC DISCRETE VISUAL TOKENIZATION
Yang Jin', Kun Xu², Kun Xu², Liwei Chen², Chao Liao?, Jianchao Tan", Quzhe Huang'. Bin Chen", Chenyi Lei", An Liu", Chengru Song', Xiaoqiang Lei², Di Zhang?, Wenwu Ou?, Kun Gai², Yadong Mu 'Peking University "Kuaishou Technology jinyestu. pocyysst..pu..c. eku..du.ccn (xukunxkxi, 8yxu828)) 0 @ma com, myd8pku.
ABSTRACT
Recently, the remarkable advance of the Large Language Model (LLM) has in- spired researchers to transfer extraordinary reasoning capability to both vision and language data. However, the prevailing approaches primarily regard the visual input as a prompt and focus exclusively on optimizing the text generation process conditioned upon vision content by frozen LLM. Such an inequitable treatment vision and language heavily constrains the model's potential. In this paper, we break through this limitation by representing both vision and language in unified form. Specifically, we introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like foreign language that LLM can read. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the im- age. Coped with this tokenizer, the presented foundation model called La VIT can handle both image and text indiscriminately under the same generative learning paradigm. This unification empowers LaVIT to serve an impressive generalist interface to understand and generate multi-modal content simultaneously. Exten- sive experiments further showcase that it outperforms the existing models by a large margin on massive vision-language tasks. Our code and models are avail- able at https: /ttt://gttttt com/ watpm//yy0000///II//
INTRODUCTION
The large language models (LLMs) (Brown et al., 2020; Touvron et al., 2023) nowadays have demonstrated impressive advances in various linguistic applications. Profiting from the knowl- edge in the massive text corpus, they possess exceptional understanding capabilities and serve as a general- pprppss interface to complete a wide range of real-world tasks. Such success has moti- vated researchers to investigate the Multi-modal Large Language Models (MLLMs), which aim at extending the powerful pure-text LLMs to process multi coltiittodlitty inputs. As shown in Figure 1-(a), the prevalent approaches mostly leverage an adapter architecture (e.g., the Resampler (Alayrac et al., 2022), linear projection (Liu et al., 2023), or Q-Former (Li et al., 2023)) to map the visual features encoded by pre-trained vision backbone (Radford al., 2021) to the semantic space of LLM.
Despite achieving preliminary results zero-shot multi-modal understanding, they still suffer from inherent design deficiencies. The training objective of prior methodologies (Li et al., 2023; Huang et al., 2023; Zhu et al., 2023) is centered on predicting textual deseriptions dependent on visual con- tent, where the visual parts are merely regarded as prompts without any supervision. The inequitable treatment of different modal inputs severely constrains the model's potential, limiting them to only performing comprehension tasks like generating text based on images. Moreover, most of these methods completely delegate the responsibility of vision-languall adapter with limited trainable parameters, which fails to leverage the remarkable reasoning capabili- ties of L.L to learn the interaction across different modalities. Although the recent concurrent work
alignment to the newly added
Work done during an internship at Kuaishou Technology. 1 Cooresponning Author.

---

Published as a conference paper at ICLR 2024
IMAGE
DAAGE
DANGE
Regansse
Gougliviie
Vixcal Tanndeer
MILLM
Viuaa Ehooder
MLLM
Viezal Enosder
MLLM
Modal Adapter
Qfermir
Siezal Tekenisey
TENT
TRXT
TEXT
Ing token
Text siken
Diante lag siken
(4) Atapter-Style
(b) Emu
Ours
The comparisons between different Ms..Ms. (a) The adapter-style methods rely adapter network to project visual features into the semantic space of LLM. During training, visual tokens are merely treated as the prompt to guide text generation. (b) The concurrent work Emu adopts the regression loss for visual features and jointly trains with textual tokens. (c) We craft a visual tokenizer to represent images in the same discrete format as text so as to indiscriminately optimize them under a unified generative objective.
Figure 1:
Emu (Sun et al., 2023) proposes to unlock the text-pretrained LLM by regressing the next visual em- bedding during pre-training (Figure 1-(b),, the inconsistent optimization objectives for image and text are not conducive to unified multi-modal modeling.
In this work, we introduce LaVIT (Language- Language-VIsioon Vision Transformer), a novel general-purpose multi- modal foundation model that inherits the successful learning paradigm of LLM: predicting the next image/text token in an auto-regressive manner. Our insight is that by employing unified objective to indiscriminately treat tokens from different modalities, the model can seamlessly achieve "any-to- any" multi-modal comprehension and generation. However, the original LLM is specifically crafted to process discrete textual tokens. When dealing with physical signal inputs, such as images, becomes imperative to embrace a representation seamlessly compatible with text tokens. There- fore, we propose to translate the image into a sequence of tokens like a foreign language that LLM can comprehend, that both images and texts can be handled simultancously under the unified generative objective without any specific architectural modification, as shown in Figure 1-(c).
To achieve this goal, crucial element lies in the development of an efficient visual tokenizer for encoding images, which we contend should adhere to the following principles: (i) discrete visual token: While language models rely on text tokens defined by dictionary, prior visual tokens, like those derived from ViT, consist of continuous feature vectors encoding a patch. In approaches such as masked image modeling (He al.. 2022) or masked feature prediction (Wei et al., 2022), re- gressive objectives on continuous features or raw visual pixels are employed for self-supervised pretraining. Here, we advocate for quantizing the visual tokens into a discrete form, aligning them with the next-token prediction objective in language models. This form is particularly advantageous when the target distribution for the next token is multi-mode. (ii) dynamic token allocation. Given the varying semantic complexity of different images, employing fixed length of tokens to encode all images is compute-unecooomical Moreover, as a key difference from textual tokens, visual patches exhibit notable interdependence, making it considerably more straightforward to deduce one token from others. This renders the next-token paradigm less effective in learning visual knowl- edge through self-supervision. Thus we argue for the moken-megging ensure the least redundancy among visual patches, thereby rendering a dynamic token number for different images.
Following the aforementioned two crucial fundamentals, LaVIT introduces novel dynamic visual tokenization mechanism consisting of selector and a merger to process images. The token selector first decides which visual patches carry informative semanties and are necessary to be selected to encode the whole image. In order to maximally preserve the image details, the token merger further compresses the unselected patches onto the retained ones according their feature similarity. Such a design enables each retained visual token to contain high-level semantics from multiple similar patches and thus reduce the redundancy among tokens. This selecting and merging strategy will produce dynamic sequence length varying from the image content itself. The retained visual to- kens are further quantized into discrete codes by a learnable codebook (Esser et al., 2021), which will serve the supervision signals for visual tokens during pre-training. Empowered by this vi- sual tokenizer, our LaVIT can be trained with a simple yet unified objective: predicting the next image/text token in the multi-modal sequence. After pre-training, LaVIT can serve a multi-modal generalist to perform both mulli-modai comprehension and generation without further fine-tuning (See Figure 2). The key contributions of this work are summarized as:
2

---

Published as a conference paper at ICLR 2024
Next Imago/Text Token Prediction
[Boa]
Understanding
Teatial Tileess
skateboard
Multimodal Language Model (IaV
LaVIT
Tokenizer
carge
DMGG
tobens
Generation
Text Tokenizer
LaVIT
Visual Token
The antelopes
man is playtng ball
Textual Token
cating
Benkee
Mostt: medal TTalooe
(a) Pre-training
(b) Inference
Figure 2: Given an image-text pair. the image is tokenized into discrete tokens and concatenated with text tokens to form a multi-modal sequence. Then, LaVIT is optimized under unified generative objective. After training, it can achieve both zero-shot multi-modal comprehension and generation.
We introduce LaVIT, new effecttve, general-purpose multi-modal foundation model that goes beyond the traditional adapter-based architectures. By transforming images into a sequence of discrete tokens like foreign language that LLM can comprehend and generate, both modalities can be associated indiscriminately under a unified generative training paradigm.
The developed visual tokenizer can produce discrete visual tokens with dynamic length to reduce the interdependence among visual patches, which enhances the representation compatibility of image and text in LLM and improves computational efficiency.
Our LaVIT showcases the extraordinary multi-modal understanding and generation potential. It can take any modality combinations as input and perform impressive in-context generation of both images and text. As demonstrated by extensive experiments, LaVIT achieves state-of-the-ant zero-shot performance on a wide range of vision-language tasks.
RELATED WORK
Vision-Language Pre-training. Researchers have extensively investigated vision-language pre- training (VLP). The pioneer works (Radford et al., 2021; Jia et al., 2021) primarily employ dual- encoder with contrastive objectives to learn the generic pross-modal aligned representations. Re- cently, the rapid progress of large language models (Chowdhery et al., 2022: Touvron et al., 2023) has motivated researchers to delve into the exploration of augmenting LLM towards vision language tasks. The majority of these works adopt adapter-style network (Zhang et al., 2023) that serves as an intermediate bridge connecting the pre-trained vision encoder and frozen language model. For instance, Flamingo (Alayrac al., 2022) develops Perceiver Resampler to generate text-aligned visual representations. Follow-up methods (Li et al., 2023; Zhu et al., 2023) mainly adopt the Q- Former to project the visual semanties to the LLM's input space. However, visual inputs these methods (Huang et al., 2023; Alayrac et al., 2022) are only considered as the prompt and not in- volved in the optimization, which heavily restricts the model potential.
Vector Quantization in Computer Vision. Vector quantization (Gray, 1984; Nasrabadi & King, 1988) is widely used in image-generative models. The VQ-VAE (Van Den Oord et al., 2017) and DALL-E (Ramesh et al., 2021) proposed to convert an image into a set of discrete codes in learn- able discrete latent space by learning to reconstruct the original image pixels. Models like VQ- GAN (Esser et al., 2021) and VÊT-VQGAN (Yu et al., 2021) leverage adversarial and perceptual objectives to further enhance the image generation quality. The BEIT series of works also adopts the quantized visual codes as the supervision in mask image modeling (Peng et al., 2022; Wang et al., 2023). However, most of these methods tokenize the image into token sequence with fixed length (c.g., 512 1024). Such long sequence will invariably result in excessive computa- tional burden. On the contrary, our proposed visual tokenizer reduces the redundancy among image patches and supports dynamic token length, thus enabling efficient multi-modal inference.
3

---

Published as a conference paper at ICLR 2024
Codebook Embeddiiggs
Visual Part Features
Ncaresi Neighbor Lookup
blocks
Feed Forward
(8))
Quantize
Visual Features
Quantized Embodddiggs
Token Selector Mergor
Recemetimnct Dhecrbsrr
Unused Pre- Pre-Training
X
Attention
Value
Query
Reconstruct
Cansal Astention
Visual Encoder
Unselected Patches
Xy
Image
Visuil Semantic Features
[x]
Selocted Patchics
(a) Dynamic Visual Tokenizer
(b) The Token Merger
Figure 3: (a) The pipeline of the proposed dynamic visual tokenizer. It employs token selector to select the most informative patches and token merger to compress the information of discarded patches onto the retained ones. The whole tokenizer is trained by maximally reconstructing the semantics of the input image. (b) The detailed architecture of token merger.
METHOD
This work proposes leverage the extraordinary reasoning potential of the large language model to facilitate the modeling of both vision and language modalities. In pursuit of this goal, the key component to represent these two modalities learning recipe (i.e., next-token prediction). As shown in Figure 2, we develop a visual tokenizer (Section 3.1) 1) to convert the non-ling non-linguistic image to the input that LLMss can comprehend. receives the vision features from pre-trained vision encoder and outputs sequence of discrete visual tokens possessing word-like high-level semanties. Coped with the crafted tokenizer, the visual input can be integrated with textual tokens to compose multi- modal sequence, which is subsequently fed into large language models under a unified regrrsegeess training objective (Section 3.2).
uniform form, so as to exploit L.LM's successful
3.1 STAGE-1: DYNAMIC VISUAL TOKENIZER
Given an image E C is first partitioned into HW/P2 non-overlapping patches, where P is the patch size. These patches are fed into a pre-trained ViT encoder (Fang et al., 2023) to produce sequence the patch features X {.,,,.,, Then, straightforward way to encode images is to directly quantize the N patch-level embeddings into discrete tokens as the input to LLMs. This will result in a long visual sequence and bring superfluous computational burden since many visual patches may contain repetitive and trivial background semantics. These redun- dant patches demonstrate a discernible interdependence, thereby diminishing the efficacy of the next-token paradigm in learning visual knowledge via self- self-supervision. Consequently. the proposed tokenizer aims to produce visual tokens with dynamic length according to the complexity of the image content itself. As illustrated in Figure 3, comprises token selector and token merger.
Token Selector The token selector takes the N letch-level features X input. aims to estimate the importance each image patch and selects the most informative ones that are competent enough to represent the semantics of the whole image. Inspired by (Rao et al., 2021). it is implemented as a lightweight module consisting of several MLP layers to predict a distribution æ E Nx2 where π, MLP(x;). By sampling from the distribution IT, binary decision mask M {0. 13 can be generated, which indicates whether to remain the corresponding image patch. To relax the sampling to be differentiable, the Gumbel-Softmax trick (Maddison et al., 2016) is applied x:
exp((log Tij + Gi,))TT)
Tij
1=1 exp( Tar GTrr)/T)
4

---

Published as a conference paper at ICLR 2024
where G j is the noise sampled from Gumbel distribution, is the temperature to control smooth- ness. Then, the binary decision mask M can be sampled from स for end-to-end training.
Token Merger According to the generated decision mask, total N image patches can be par- titioned into retained and dropped groups, with T' and N - T patches respectively, denoted as x, = {= and Xa= = { - Instead of directly discarding Xa. we develop a token merger to deal with it to maximally preserve the detailed semantics of the input image. As shown in the right of Figure 3. the token merger will progressively compress the information of discarded Xa onto the retained XX, according to their semantic similarity. Concretely, it consists of L stacked blocks, each of which has a causal self-attention layer. a cross-attention layer, and a ford-forward layer. In the causal self-attention layer, each token in X, attends to its previous tokens with causal mask. This helps to convert 2D raster- features from the ViT encoder into a sequence with causal de- pendency, thus ensuring consistency with textual tokens in LLMs. We found this strategy can result in better performance than bi-directional self-attention. The cross-attention layer treats the retained tokens X, as the query and merges tokens in Xy based on their similarity in the embedding space. Formally. this layer calculates an update of X, by:
AX, softmax (QKT /VD)
where D denotes the dimension of hidden state, Q WoX, E RTT.D,K= KK x7) and V = Xd E R(N-T)x2 To parallelize the computation, we adopt the predicted decision mask M to control the cross-attention scope between tokens without directly partitioning them into two groups After L successive token merger blocks, we can obtain the final merged visual tokens X, = { Each token implicitly encodes high-level semantics from several image patches possessing similar visual patterns, which we refer to visual part features X,, The token selector and merger work together to dynamically adjust the visual token sequence length to accommodate images with different content complexity.
Vector Quantization and Training The generated visual part features Xx, are then passed into a quantizer. It tokenizes X, to a sequence of discrete visual codes V = {u; )Tm by looking up a learnable codebook = {c& K where K is codebook size. To be specific, the ish visual code is calculated by assigning d; in X, to its closest neighbourhood code in C:
v. arg min
/ | (z:) - (2()))l2, U, E [0, K- 1].
where l2 indicates the L2 norm. Based on the indexing visual codes, we can obtain the quantized embeddings {Co,) ) which is fed into a decoder to reconstruct the original visual semantic fea- tures X = {z, L. The insight behind this design is that the reconstruction quality of the image semantics depends selecting the most informative patches (token selector), along with maximally preserving the visual details only through the remained tokens (token merger). Thus, both token selector and merger can be effectively updated by encouraging a higher semantic reconstruction quality. The final training objective of the visual tokenizer is defined as:
N Σ (1 al--cos(fi, x:()) + X(p
N
1
Lookenizer
Σ M)),
(4)
N
where cos(1;; (2,, at roc) calculates the cosine similarity between the reconstructed and real visual embed- dings, P is a pre-defined rate that controls the target mean percentage of the retained visual tokens
and À is signals for visual tokens in the following pre-training.
be 2. Finally, the tokenized discrete codes (u, [u will serve as the supervision
Decoding to Pixels The proposed visual tokenizer is capable of reconstructing visual features of input images that contain high-level semantics to represent the image content but lose the pixel-level details. To recover the original pixel space, we employ a conditional de-noising U-Net eg (Rombach et al., 2022) to infill the visual details after training the visual tokenizer. Specifically, it takes the reconstructed Tree as the condition to progressively recover image from a Gaussian noise. Follow- ing Rombach et al. (2022), the parameters 0 of this U-Net are optimized by e prediction:
(5) where 2 is the latent state of image x in the diffusion process. We present some pixel decoding results by the trained denoising U-Net in Figure 7 of the appendix. During inference, the generated visual tokens from La can be decoded into realistic images by this U-Net.
Le = ((0,,)), I||e eø(zt, t, [lce
5

---

Published as a conference paper at ICLR 2024
3.2 STAGE-2: UNIFIED GENERATIVE MODELING
Given an image-texi pair, the 2D image can be tokenized into ID sequence with causal dependency and then concatenated with text tokens to form multi-modal sequence y (yI, 1/22 (y1,,/2;-, ys)- For differentiating between two modalities, two special tokens [IMG] and [/IMG] are inserted into the beginning and end of the image token sequence respectively. indicating the start and end of image content. To empower LaVIT with the capability to generate both text and images, we employ two different concatenation forms, i.e., [image, text] and [text: image]. When the image used as a condition (on the left) to generate text, we use the continuous visual features X, from the token merger instead of quantized visual embeddings as the input to LLMs. Such design mitigates the loss of detailed information caused by vector quantization, which is crucial for fine-grained multi- modal understanding tasks like visual question answering. Our LTV adopts the general Language Modeling (LM) objective to directly maximize the likelihood of each multi-modal sequence in an auto-regressive manner:
plu) = Σ Σ log Po
MEDi=l
Since both image and text are already represented as discrete token IDs, we can use the cross-entropy to supervise the token prediction at each location for both two modalities with a shared prediction head. The complete unification in representation spaces and the training paradigms can help LLMs better learn multi-modal interaction and alignment. When LaVIT is pre-trained, it possesses the capacity to perceive images akin foreign language, comprehending and producing them like text. Nevertheless, most of the existing works merely regard images as prompts to guide the generation of text with supervision, restricting them to solely performing image-to-text tasks.
3.3 MODEL PRE-TRAINING
The LaVIT undergoes a two-stage pre-training procedure on web-scale
corpora.
Tokenizer Training. Following the existing MLLMs, the ViT-G/14 of EVA-CLIP (Fang et al., 2023) is employed as the visual encoder. The visual codebook size is empirically set to K 16384. We adopt L 12 transformer blocks for both token merger and decoder in tokenizer. During training, this encoder is kept frozen and only the parameters of the selector, merger, and codebook are updated. is trained for 50K steps about 100M images from LAION- 400M (Schuhmann et al., 2021) with the batch size of 2048 and 1/3. After training the tok- enizer, the conditional U-Net for pixel decoding is initialized from the Stable Diffusion v1.5 (Rom- bach et al., 2022) and finetuned 20k steps on the same dataset. The whole stage-1 training only requires pure image data without corresponding captions.
Stage-1:
Stage-2: Unified Vision-Language Pre-training. Based on the trained visual tokenizer, all the im- ages can be tokenized into discrete codes that are amenable to the next token prediction. We utilize the raw 7B version of LLaMA (Touvron al., 2023) the default LLM. For image-to-text com- prehension (i.e., [image, text)), we employ about 93M samples from Conceptual Caption (Sharma et al., 2018; Changpinyo et al., 2021), SBU (Ordonez et al., 201 2), and BLIP-Capfilt (Li et al., 2022). For the text-to- to---age- synthesis (i.e., [text, image)). an additional 100M image-text pairs from the -AION-Aestheticc (A high-acsthetics image subset of LAION-5B (Schuhmann et al., 2022)) used following Stable Diffusion. Moreover, to reduce catastrophic forgetting of the reasoning capac- ity in training LLM,, employ the English text corpus from Redpajama (Computer, 2023) dataset and mix it with the above image-text pairs to form the multi-modal input sequence.
EXPERIMENTS
In this section, comprehensive experiments are conducted to systematically validate the effectiveness of LaVIT on a wide range of vision- language tasks. Specifically, we mainly evaluate the model's zero-shot multi-modal understanding and generation capacity.
4.1 ZERO-SHOT MULTIMODAL UNDERSTANDING
We first quantitatively evaluate the zero-shot multi-modal understanding capacity of LaVIT on Image Captioning (NoCaps (Agrawal et al., 2019), Flickr30k (Plummer al., 2015))and Visual

---

Published as a conference paper at ICLR 2024
Image Captioning
Visual Question Answering
Method
Nocaps
Flickr
VQAv2
OKVQA
GQA
VizWiz
Flamingo-3B (Alayrac et al., 2022)
60.6
49.2
41.2
28.9
Flamingo-9B (Alayrac et al., 2022)
61.5
51.8
44.7
28.8
OpenFlamingo-9E (Awadalla al., 2023)
59.5
52.7
37.8
27.5
MetaLM (Hao et al., 2022)
43.4
41.1
11.4
Kosmos-1 (Huang
2023)
67.1
51.0
29.2
Kosmos-2 (Peng al., 2023)
80.5
51.1
BIJI-2 (Vicuna-7B)
al., 2023)
107.5
74.9
41.3
25.3
BLIP-2 (Vicuma- (Li et al., 2023)
103.9
71.6
32.3
19.6
CM3Leon-7B (Yu et al., 2023)
47.6
37.6
Emu (LLaMA--13B) (Sun et al., 2023)
52.0
38.2
Ours (LLaMA-7B)
114.2
83.0
66.0
54.6
46.8
38.5
Table I: Overview of zero-shot evaluation multi-modal understanding tasks. Compared with previous methods, our LaVIT achieved the best performance on both benchmarks.
Question Answering (VQAv2 (Goyal et al., 2017), OKVQA (Marino al., 2019). GQA (Hud- son Manning, 2019), VizWiz (Gurari et al., 2018)). For visual question answering tasks, we simple prompt: "Question: {} Answer: f)" The widely-used CIDEr score and VQA
accuracy are employed as metrics to evaluate captioning and question answering, respectively.
The detailed performance compar- isons are shown in Table 1. As observed. LaVIT surpasses all the existing MLLMs by a large mar- gin on these understanding tasks. For example, it achieves CIDEr score of 83.0 on the Flickr30k test dataset, compared to 61.5 and 74.9 the Flamingo-9B and BLIP-2 (Vicuna-7B) under the same scale of model size, respectively. The per- formance superiority (54.6% v.s. 44.7% of Flamingo- 9B) further showcases the excel- lent multi-modal understanding ca- pacity of LaVIT, since this bench- mark contains questions requiring commonsense knowledge and rea- soning about the content of images. Emu (Sun et al., 2023) also leverages the L.M jointly model the vision and language, the di- rect feature regression objective for visual inputs makes it incompatible with text input. Therefore, despite using more training data (2.6B image-text pairs and 3.8B web-crawled data) and larger LLM (LLaMA 13B), it still achieves inferior performance to ours all evaluation benchmarks.
Method
Model Type
FID())
Text2maage Specialist:
DALL-E (Ramesh et al., 2021)
Autoregressive
28.0
Cog View (Ding al., 2021)
Autoregressive
27.1
SD (Rombuch et al., 2022)
Diffusion
12.6
GLIDE (Nichol et al., 2021)
Diffusion
12.2
al., 2022)
Diffiusion
10.4
Make-A-Scene (Gafni et al., 2022)
Autoregressive
11.8
MUSE-7,66 7.6E-77.BBB6 (Chang et al., 2023)
Non- Non-Autoregressive
7.9
Imagen-3.44 3.4B (Saharia
2022)
Diffusion
7.3
Purti-20B
2022)
Autoregressive
7.2
Multimodal Large Langauge Model: GILL (OPT-6.7B) (Koh et ul., 2023)
OKVQA
LLM
12.2
Emu (LLaMA-13B) (Sun et al., 2023)
LLM
11.7
CM3Leon-7B (Yu et al., 2023)
LLM
10.8
Ours (LLaMA-7B)
L.M
7.4
Table 2:
The zero-shot text-to text-to-image generation performance of different models MS ASS-COO--30E evaluation benchmark.
It is worth noting that, although the concurrent method
4.2 ZERO-SHOT MULTIMODAL GENERATION
Since the proposed visual tokenizer can represent images discrete tokens, LaVIT possesses the capability to synthesize images by auto-regressively generating visual tokens like text. We first quantitatively evaluate the model's zero-shot cet--ooniitonal image synthesis performance on the validation set of the MS-COCO benchmark (Lin et al., 2014). The detailed image generation proce- dure is presented Appendix Following the standard setting like previous text-to-image synthesis works, we randomly sample 30k text prompts and calculate the zero-shot FID metric between real images and generated ones. The detailed comparative results are shown in Table 2. can be seen that LaVIT outperforms all the other multi-modal language models. Compared with the concurrent work Emu, it makes a 4.3 FID improvement with a smaller LLM model, demonstrating excellent vision-languee alignment capability. In addition, LaVIT achieves comparable performance with state-of-ther the---rrtt--tttt text2image specialists Parti (Yu et al., 2022), while only using much fewer training data (e.g., 0.2B v.s. 2B training image-text pairs compared to Parti).

---

Published as a conference paper at ICLR 2024
Ours
Stable Dillision v2.1
A photo of light oune space traveling golasy sailing the hight
delevision madee water that displays inagee of ciyyscape night. Text-to-Image Generation compared with Stable Diffusion
On the broeh
On tup a Wanden Ploce
On the boach
On top o wooden flace
Image+ Text
Image+ Text
polioe
police
Wwaring pink glasoss
Image Image
Iraage Immaee
(b) The Image Generation with multi-modal prompts LaVIT
Figure 4: The qualitative examples of multi-modal image synthesis.
Generation via Multi-modal Prompt LaVIT can seamlessly accept several modality combina- tions (e.g., text, image+text, image+image) as prompts to generate corresponding images with- out any fine-tuning. Figure 4 showcases some examples of the multi-modal image generation re- sults. Our LaVIT produce high-quality images that precisely reflect the style and semantics of the given multi-modal prompts, which demonstrates the strong multi-modal modeling potential of LaVIT. More interestingly, it can modify the original input image by the input multi-modal prompt (e.g., in the last example two prompt images with a dog or cat generate a dog's portrait with the cat's whisker). This capability cannot attained by conventional image generation models like Stable Diffusion in the absence of additional fine-tuned downstream data (Ruiz et al., 2023).
4.3 ABLATION STUDY
In this study, we investigate the impact of various component designs in LaVIT on downstream performance. All the ablations were conducted on part of pre-training data by using the clip ViT- L/14 (Jia et al., 2021) as the visual encoder due to the costly training resources.
Token Classification or Feature Regression? When jointly training vision and language via gener- ative training in text-oriented LLM, it is crucial to select the appropriate optimization objectives for the 2D raster- visual input. When quantizing the continuous visual tokens into the discrete form, it is convenient to use the cross-entropy loss for supervising the next visual token prediction akin to textual tokens. contributes to aligning them together in the LLM. To validate the superiority of the proposed visual quantization, we change the optimization objective of visual tokens to regressing the next visual embeddings by employing regression head like Emu (Sun et al., 2023). Table 3a summarizes the results of different training objectives. As observed, adopting the regression loss for the next visual token prediction will severely degrade the model performance.
conjecture that such uniform objective for both vision and language
8

---

Published as a conference paper at ICLR 2024
Token Num:
Token Num: LOB
Code: 8992
Code: 4107
Toleen Num:: 79
Token Num: 9/7
Cods: 9146
Code: 15116
Figure 5: The visualization dynamic visual tokenizer (left) and learned codebook (right). Our tokenizer can dynamically select the most informative patches based on the image content and the learned codebook can produce visual codes with high-level semantics.
Dynamic Fixed Token Length. Given the extracted visual features, straightforward way is to tokenize all the patch embeddings into the visual tokens, which results in a fixed token length (i.e., 256). We compare the impact of fixed and dynamic tokenization strategies in terms of training time, computation overhead, and zero-shot performance on multi-modal understanding. As shown in Table 3b, the dynamic visual tokenizer achieves superior performance while only requiring 94 tokens on average for the input images, about 36% of the fixed one. Given that the attention computation LLM exhibits a quadratic relationship with respect to the token length, this sparsification can accelerate the training time by 40% and reduce the computational cost in inference.
Setting
Flickr
VQAv2
OKVQA
Setting
Num
Time
Flickr
VQAv2
OKVQA
Regression 60.4
53.6
41.9
Fixed
256
30h
71.1
56.5
46.4
Classification
73.2
57.1
47.0
Dynamic
94
18h
74.0
57.7
47.6
(a) Ablations different training objectives.
(b) Ablations effect different tokenization strategies.
Table 3: The ablations of different optimization objectives for visual tokens and tokenization strate- gies. The num and time in Table 3b indicate the mean visual token number and pre-training time.
4.4 QUALITATIVE ANALYSIS
We visualize some examples processed by the proposed dynamic tokenizer. As shown in Figure 5, the token selector capable of dynamically selecting the most informative image patches that are competent enough to represent the semantics of the whole image. Visual patches that contain repet- itive or trivial background semantics are filtered during this procedure, thereby reducing redundant information and improving computing efficiency. We also visualize the image patches that belong to the same visual code in Figure 5. As observed, the learned discrete codes can convey explicit vi- sual semantics and group the image patches with similar patterns together. For instance, code 4107 represents a part of skateboard, and code 9146 indicates the texture of a giraffe, which strongly demonstrates the interpretability of the learned codebook.
CONCLUSION
This paper presents the LaVIT, a new general-purpose foundation model that is capable of simulta- neously performing multi-modal understanding and generation. Beyond the previous adapter-based methods, it inherits the successful auto-regressive generative learning paradigm of LLMs by rep- resenting both vision and language in a unified discrete token representation via a dynamic visual tokenizer. Through optimization under the unified generative objective, LaVIT can treat images as a foreign language, comprehending and generating them like text. Extensive experimental results further demonstrate the LaVIT's superior capability to serve as multi-modal generalist.
Acknowledgement: This research work supported by National Key R&D Program of China (200)).

---

Published as a conference paper at ICLR 2024
REFERENCES
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning scale. In Proceedings of the IEEE/CVF international conference computer vision, pp. 8948-8957, 2019.
Man-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01130, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020..
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- eration via masked generative transformers. arXiv preprint arXiv:2301 300000000444 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204. 02311, 2022.
Together Computer. Redpajama: An open source recipe to reproduce laaaa training dataset, 2023.
URL https: //tpth/ com t oge togetoeeeeeeeoooeoooo teapeehercomme wwwwehewwwwwweeeeeeeeeeeeeeeeeee a ama Data.
Ming Ding, Zhuoyi Yang. Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, al. Cogview: Mastering text-to-image generation transformers. Advances in Neural Information Processing Systems, 34: 19112-9982355555 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference computer vision and pattern recogni- tion, pp. 12873-1222 12883, 2021.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19358-19369 2023.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. European Conference on Computer Vision, pp. 89-106. Springer, 2022.
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Robert Gray. Vector quantization. IEEE Assp Magazine, 1(2):4 29, 1984.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
10

---

Published as a conference paper at ICLR 2024
Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are general-purpose interfaces. arXiv preprint arXiv:2206.06336, 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 15979-15988, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and
Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12598, 2022.
arXiv preprint
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302. 14045, 2023.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. arXiv preprint 2rXiv:2305..72211 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888-12900. PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.125977 2023.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Visi 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint AXXi::334400488 2023.
Chris Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv: 2016.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Nasser Nasrabadi and Robert A King. Image coding using vector quantization: A review. IEEE Transactions communications, 36(8):957-971, 1988.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 /2. 10741, 2021.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using million captioned photographs. Advances in neural information processing systems, 24, 2011.

---

Published as a conference paper at ICLR 2024
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208. 06366, 2022.
Zhiliang Peng,
Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma. and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting to-sentence models. In Proceedings of the IEEE international conference computer vision, pp- 2641-2649, 2015.
correspondences for richer image-
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pp. 8821-8831 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint wxXiv:2204006125 1(2):3, 2022.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:1343:7 13937-13949, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 15:36479-36499 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-2523 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernyme,, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 2556-2565, 2018.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307.0553 05700222,, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971, 2023.
12

---

Published as a conference paper at ICLR 2024
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal,
Owais Khan Mohammed, Saksham Singhal, Subhojit Som, al. Image foreign language: Beit pretraining for vision and vision-language tasks. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 19175-19186, 2023.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14648-14658, 2022.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. In International Conference on Learning Representations, 2021.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang. Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi-modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2339.0255), 2023.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init atten- tion. arXiv preprint 1rXiv:2303.1619, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vasion-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
13

---

Published as a conference paper at ICLR 2024
A
THE DETAILS OF IMAGE SYNTHESIS
Given a multi-modal prompt (image, text, or their combinations), LaVIT first tokenizes it as a se- quence of discrete tokens. By appending the special [IMG] (image start token) the end of the prompt and feeding into the LLM, can auto-regressively generate a sequence of visual tokens until reaches the special [/IGG] (image end). These yielded discrete visual tokens are further re- constructed into a feature map by the decoder of the proposed visual tokenizer, which reflects the high-level semantics of the synthetic image. Finally, the conditional denoising U-Net will take this feature map as the condition to progressively recover image pixels from a Gaussian noise. Taking the text prompt as an example, we illustrate the entire image synthesis procedure in Figure 6.
Visual Tokens
pIMG]
Reconstrurt
Visual Features
LaVIT
Tokenizer Decoder
Conditioning
[LMG]]
Textual Tokens
Up Sample
Denoising U-Net
Text Tokenizer
Caussian Noise
Generated Image
A man is playing temnis ball
Recover to Pixel Space
The detailed image synthesis process for the textual prompt by LaVIT. Generation for prompts in other modalities is similar.
Figure 6:
During the text-to-image evaluation on MS-COCO, we use the top-k sampling to generate image tokens and set the maximum sampled tokens 300, and softmax temperature 1.0. Besides, the Classifier-Fre Free Guidance (CFG) (Ho & Salimans, 2022) strategy is also employed to enable the model to yield more prompt-aligned image tokens. Specifically, we replace the input multi-modal prompt with the [IMG] token to conduct unconditional sampling. Both conditional and prompt- conditioned visual token streams are generated in LaVIT. Hence, the real sampling logits for fuh image token yr in the generation is blend of both unconditional and conditional logits:
(bx) Iyy, [[MG]) Octy (A(ye |prompt) Iyya [IMG))
The larger acfg will encourage the model to generate samples highly related the input prompt while
sacrificing some diversity. We set Oefg to 1.5 when evaluating the zero-shot text-to-image synthesis on MS MS-COCO. LaVIT generates 4 samples for each prompt and uses CLIP- VIT/L model to select the best one according the image-prompt matching score.
B ADDITIONAL EXPERIMENTAL ANALYSIS
B. THE IMPACT OF VISION-LANGUAGE PRE-TRAINING ON LLM
During pre-training, the weights of LLM are unlocked jointly optimize the visual and textual tokens. Since new modality is included in text-oriented LLM, without setting appropriate hyper- parameters, its original language capability may be impaired. In this section, we discuss how to mitigate the forgetting of knowledge in L.M when undergoing the vision-language pre-training. We chose the massive multitask language understanding benchmark (MMLU) (Hendrycks et al., 2020) to quantitatively measure the model's common knowledge and reasoning ability. MMLU is a knowledge-intensin sciences. It is commonly used benchmark in evaluating the Large Language Model. Following LLaMA (Touvron et al., 2023), we evaluate the model the 5-shot setting. The detailed experimen- tal results reported in Table 4.
dataset covering various domains, including humanities, STEM, and social
Through exploration, we conclude some useful strategies for conducting unified vision-language pre-training in LLM:: 1) The Learning rate of the LLM part is the most important hyper-parameter.
14

---

Published as a conference paper at ICLR 2024
Setting
LRt
LRg
Text
Multi-modal
Humanities
STEM
Social Sciences
Other
Average
LLaMA-7B
3e-4
100%
0%
34.0
30.5
38.3
38.1
35.1
LaVIT-7B
1.c-4
1.5c-4
66%
33%
26.7
26.1
25.7
29.8
27.1(-8.0)
LaVIT-7B
1.5c4
5o-5
50%
50%
31.4
27.2
31.8
34.5
30.9(-4.2
LaVIT-7B
1.5e-4
5e-5
66%
33%
33.5
30.3
35.2
33.6[ (1.5)
Table The influence of learning rate and data proportion on the MMLU benchmark. Here, LRy and LR, indicate the learning rate for the visual and LLM parts in LaVIT. LaVIT learns excellent multi-modal modeling capability with only slight drop of original common knowledge.
Datuset
Sampling prop
C4 Github
80%
4%
Wikipedia
4%
Books
4%
ArXiv
4%
StackExchange 4%
Table 5: The used English text corpus proportion of different subsets in Redpajama.
Although using large learning rate (e-a level) for tuning the LLM part will speed up its conver- gence on solving the rision-language task, it also speeds up the forgetting of its original learned common knowledge (8.0 performance drop compared to original LLaMA). To balance these, we suggest learning rate of e-5 level of LLM and e level of visual. 2) Text corpus data is also important. Lack of text data will also degrade the LLM's ability. Therefore, suggest proportion of 2:1 (L:VL) to maintain its original learned knowledge. In our experiments, the used English text corpus is from the Redpajama (Computer, 2023) dataset. The detailed proportion for each subset of Redpajama is presented in Table 5. Finally, we set the learning rate of L.M to 5e-5 and 1.5e for other parts in LaVIT, which only slightly affected LLM''s language ability (-1.5 MMLUU,, but achieved the best results on both multimodal understanding and generation tasks.
B.2 MORE ABLATION STUDY
Like ablations the main text, the following experiments are conducted on part of pre-training data by using clip ViT-L/14 (Jia et al., 2021) the visual encoder.
Bi-directional or Causal Attention? The token merger in the proposed dynamic visual tokenizer includes the self-attention layer to model the interaction among the visual tokens. To convert 2D raster-ordered visual features from the ViT encoder into a sequence with causal dependency like text, causal self-attention is utilized to enforce the restriction that each visual token can solely attend to its preceding ones. To validate the effectiveness, we replace causal attention with bi-directional attention in the token merger. As shown in Table 6, causal attention outperforms bi-directional self- attention across all multi-modal understanding tasks. We conjecture the inferior performance can be attributed partial information leakage, the preceding token has visibility of the subsequent one, thereby impairing the optimization of next-token prediction in LLM.
Setting
Flickr
VQAv2
OKVQA
Bi- Bi-directional Attention
69.9
55.6
44.1
Causal Attention
73.2
57.3
46.4
Table 6: The effect of different attention manners in the token merger on downstream tasks.
Type of Visual Input Embeddings. For multi-modal understanding where the image is treated as a condition on the left (i.e., [image, text)), the visual representations input to LLM can quantized vectors from the codebook the continuous features yielded by token merger. We investigate the impact of these two different visual input forms in Table 7a. As observed, the continuous visual features achieve better multi-modal understanding performance. This phenomenon is reasonable as quantized embeddings represent the centroid of continuous visual features, potentially leading to the loss of detailed visual content. Therefore, for input sequences with order iimage, text], we utilize the
15

---

Published as a conference paper at ICLR 2024
Original
Reconstruction
Original
Reconstruction
Original
Reconstruction
Figure 7: The image reconstruction results from discrete visual tokens by the denoising U-Net.
outputs from the token merger as input to LLM, whilst regarding the corresponding discrete visual codes as the supervision signal for predicting the next visual tokens.
Setting
Flickr
VQAv2
OKVQA
Setting
Flickr
VQAv2
OKVQA
Quantized
70.3
53.0
44.8
Frozen LIM
63.0
52.7
44.3
Continuous
72.7
56.4
46.2
Unlock LLM
72.1
57.1
47.9
(a) The effect of visual input embeddings forms.
(b) The effect of unlocking LLM training.
Table 7: The ablation studies of visual input embeddings format and the unlock of LLM during pre-training. The performance is evaluated on the zero-shot setting.
Setting
Flickr
VQAv2
OKVQA
w/o Token Merger
70.8
51.9
42.1
w/ Token Merger
72.7
56.4
46.2
Table 8: The effect of the proposed token merger on downstream multi-modal understanding tasks.
The role of Token Merger. The token merger aims to compress the semantics of unselected visual patches onto the retained ones according their feature similarity. Compared to directly discarding these unselected patches, this merge mechanism not only reduces the redundancy among patches but also maximally preserves the visual details. The ablation study explore the role of our token merger shown in Table From the presented results, can be observed a distinct performance drop in multi-modal understanding tasks when eliminating the token merger module.
The Effect of Unlock LLM. In order to ascertain the efficacy of utilizing LLM for learning the interaction between vision and language, devise ablation experiment wherein the entirety of LLM remains frozen while solely optimizing the parameters of the token merger during training. The detailed comparison results are shown Table 7b. One can observe that freezing the language model will restrict its powerful multi-modal modeling potential and thus result in noticeable per- formance degradation. Hence, unlocking the LLM under a unified training objective will contribute to adequately bridging the semantics between vision and language.
C MORE VISUALIZATIONS
The quality of pixel decoding In Figure 7, visualize some pixel decoding examples by the trained denoising U-Net. can be seen that, given the tokenized discrete visual tokens, the original input images be successfully recovered. The reconstructed examples exhibit high degree of
16

---

Published as a conference paper at ICLR 2024
Code 15090: Donut
Code 13014: Human Arm
Code 2223: Railway
Code 7260: Car
Code 12908: Toilet
Code 15235: Bus
Figure 8: The visualization for the leaned discrete visual codes.
Configuration
VL Pre-training
Tokenizer training
Visual Encoder
EVA-CLIP ViT-G/14
EVA-CLIP VIT-G/14
LLM init
LLaMA-1-7B
Optimizer
AdamW
AdamW
Optimizer Hyperparameters
B1 0.9, Bg = 0.95, 1e-6
3 - 0.9, B2 0.99, 1 2048
Global batch size
2048
Peak learning rate of LLM
5e-5
Peak learning rate of Visual Part
1.5e-4
2e-4
Learning rate schedule
Cosine
Cosine 50K
Training Steps
20K
Warm-up steps
2k
4K
Weight decay
0.1
0.01
Grudient clipping
1.0
1.0 224 224
Input image resolution
224 224
Input sequence LLM
2048
Numerical precision
6
b
GPU Usage
256 NVIDIA A100
64 NVIDIA A100 12h
Training Time
30h
Table 9: The detailed training hyperparameters of LaVIT
semantic and general structure the original images. This consistency validates that our dynamic tokenizer can efficiently encode visual information using generated discrete tokens.
The Learned Codebook We also provide more visualization examples for the learned codebook in Figure 8, where the image patches belonging to the same discrete visual code are arranged to- gether. One can observe that the learned visual code can encode explicit high-level visual semantics. For example, code 13014 represents the human arm, code 7260 shows the part of the car, code 15235 represents the bus, and code 15090 indicates the donuts. The high distinctiveness of the learned codebook facilitates the unified vision-language pre-training in LLM.
Multi-Modal Image Synthesis
illustrate more image synthesis results Figure 9, 10, 11 and 12. As presented, LaVIT produces high-quality, content-rich, and appearance-diverse image sam- ples that precisely reflect the prompt's semantics. Importantly, LaVIT can produce images well aligned with long and more complex text descriptions, while the samples generated by stable dif- fusion do not reflect the subtle details in some complicated prompts. Besides, LaVIT also supports the multi-modal in-context image generation. As illustrated in Figure 9, the synthetic samples can
17

---

Published as a conference paper at ICLR 2024
This oil painting
An oil painting dog
This is dog
The animal in image sitting in the place image
This the first image
This is the image
This dog this hat
This dog
This is hat
Input Multi-modal Prompt
Generation
Figure The examples of multi-modal in-context image synthesis.
fully reflect the semantics of input multi-modal context, which contributes to the diversified editing of the input image through the multi-modal control.
Multi-modal Understanding We also visualize some mulli-moda understanding examples in Figure 13. As observed, LaVIT is capable of understanding the visual content, generating concise textual captions to depict the image, and answering diverse questions about detailed visual clues.
D PRE-TRAINING DETAILS
The detailed training
settings are reported in Table 9.
E LIMITATIONS
Since LaVIT is based on the Large Language Model, it inherits LLM's original language halluci- nation limitations, c.g., generating some nonexistent knowledge and making some factual errors. Besides, LaVIT trained only on web-scale image-text pairs, wherein the text descriptions are noisy and short, thereby rendering LaVIT insufficient for effectively modeling extensive text-image correspondences. We believe this limitation can be alleviated when leveraging more high-quality and long image-text interleaved documents.
18

---

Published as a conference paper at ICLR 2024
A small cacius wearing straw hat and sunglasses in the Sahara desert.
high contrast photo of panda dressed as an astronaut sits at a table in a photorealistic style
A lovely dog is reading thick book
Downtown Shanghai at sunrise. Detailed wash.
A transparent sculpture of duck made of glass. The sculpture front of painting of landscape
A high contrast photo of an astronaut riding horse in the forest.
de ofe
super math wizard cut, richly textured oil painting
Figure 10: The examples of text-to-image synthesis.
19

---

Published as a conference paper at ICLR 2024
heavy metal tiger standing on rooftop while singing on an electric guitar under a spotlight.
a bird wearing headphones and speaking into a high-end microphone in recording studio.
A section of the Great Wall the mountains. detailed charcoal sketch
Taj Mahal with reflection. detailed
charcoal sketch.
A blue jay standing large basket of rainbow macarons.
cat drinking pint of beer, DSLR photo, art station, rendering
richly lextured oil painting a young badger delicutely sniffing a yellow ruse next to a trunk. A small waterfall can be seen in the background.
light bulb with ballerina it
Cute adorable little goat, unreal engine, cozy interior lighting, station, detailed digital painting
M
O
A painting of two rabbits in the style of American Gothic, wearing the same clothes as in the original
Figure 11: The examples of text-to-image synthesis.
20

---

Published as a conference paper at ICLR 2024
In the snow
On top a purple rug
In chief outfit
Wearing yellow shirt
Wearing red hat
city background
Wearing
Wearing santa
In the snow
On of purple
chicf outfic
In the
On the beach
With the background
On the beach
Holding santa hat
On top of weoden floor
On of parple
chief outfit
ploice
Wearing
With city in the background
Wearing a black hat Wearing santa
Figure 12: The examples of image synthesis with multi-modal prompt.
Image caption
playing with bannna peol
playing
carrying purple fiihee in its mouth
back of boat the water
STOP
Visual QA
How many zebras
they daing image? A: Skiing and competing.
car?
Q:
this image? A: Three
A: On the side read next
of the image?
A: Toilet
Figure 13: The qualitative evaluation examples model's multi-modal understanding performance.
21


Reviews:
[{"review_id": "3XB3MN7KAB", "reviewer": "Reviewer_6o4w", "summary": "This work proposed to unify the pre-training objectives for visual and textual tokens while adapting the LLM for vision-language tasks. To this end, they first train a visual codebook to quantize the visual embeddings and assign each visual token a discrete label. To reduce the length of visual tokens, a token selector is proposed to select important visual tokens dynamically, and the token merger is used to merge dropped tokens with selected ones. The visual tokens and text tokens are then concatenated and fed to the LLM, which is trained for classification-based language modeling tasks. The authors conduct extensive experiments to evaluate the proposed methods, and the experimental results show that the proposed methods achieve state-of-the-art performance.", "questions": "Refer to the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The experimental results demonstrate that the proposed method achieves a significant improvement compared to previous works.\n- The improvements from each proposed component is well supported by the ablation study results.", "weaknesses": "- The authors train a visual codebook to assign discrete labels to visual tokens, which can be computationally costly. In the context of efficiently adapting LLM for vision-language tasks, it would be interesting to explore whether existing codebooks [1, 2, 3] can be utilized to provide useful supervision for training LLM.\n\n- Hard to be completely fair when compared against other baselines. For instance, compared to BLIP-v2, this work adds additional English text corpus and further fine-tunes the LLM. The significant performance improvement over previous work may be attributed to the additional training data and trainable parameters.\n\n- The implementation details of the ablation studies could be clearer. It is confusing why the proposed method achieves different results in Tables 3, 6, and 7 when doing ablation studies. \n\n- The authors claim that the learned codebook encodes high-level semantic concepts. However, the visualization results of codebooks in Figure 5 may suggest that the same code groups image patches based on low-level visual patterns, such as color and textures. It would be interesting to see additional image-level visualization results, similar to those in [4], to demonstrate that the codebook encodes high-level semantic concepts.\n\n- Minor suggestions/typos: I noticed several typos while reading the paper: \n\n  - Figure 1 (a) caption: Adater -> adapter\n  - The symbols X and when calculating K and V of Equation (2) should be X_d.\n\n---\n- [1] Chen, Yuxiao, et al. \"Revisiting multimodal representation in contrastive learning: from patch and token embeddings to finite discrete tokens.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n- [2] Esser, Patrick, Robin Rombach, and Bjorn Ommer. \"Taming transformers for high-resolution image synthesis.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n- [3] Bao, Hangbo, et al. \"Beit: Bert pre-training of image transformers.\" arXiv preprint arXiv:2106.08254 (2021)\n- [4] Peng, Zhiliang, et al. \"Beit v2: Masked image modeling with vector-quantized visual tokenizers.\" arXiv preprint arXiv:2208.06366 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r8DlgVJ0C5", "reviewer": "Reviewer_K5nK", "summary": "This paper proposes an image tokenization method for visual-language model (VLM) pretraining, called dynamic discrete visual tokenization. The proposed method has two key characteristics:\n* Discrete: Continuous feature vectors from each image patch are mapped to a learnable codebook, resulting in discrete visual tokens compatible with text tokens. The discrete visual tokens make it possible to pre-train VLMs using the same autoregressive language modeling paradigm as text-only LMs.\n* Dynamic: The number of visual tokens is dynamically adjusted depending on the image content using a token selector and token merger. The token selector chooses the most crucial patches in the image, and the token merger reacquires the visual semantics of the unimportant patches, respectively.\n\nThe proposed method is pre-trained in two steps:\n1. Tokenizer learning: The tokenizer is pre-trained to reconstruct the original image from the quantized compressed visual tokens.\n2. Vision-language model learning: The VLM is pre-trained to predict the next visual/text token in a sequence, using the same autoregressive language modeling paradigm as text-only LMs.\nThe proposed method achieves high performance on multimodal understanding and generation tasks, with the relatively small size of model parameters and training data requirements. The paper also shows that quantization and token compression are effective techniques for improving the efficiency of VLM pre-training.\n\nOverall, this paper presents a novel and effective method for VLM pretraining. The proposed method looks simple yet effective and achieves state-of-the-art results on various multimodal tasks.", "questions": "1. What is the computational (speed) improvement during inference using a dynamic token length? It may be important information because training and inference efficiency may vary.\n\n2. There are a few things that could be improved in the manuscript. Please see the following suggestions.\n\n- Figure 1 (a): `Adater-Style` -> `Adapter-Style`\n- The first line in Section 3: `the Large language model` -> `the large language model`\n- The last line in `Token Selector` of Section 3.1 : `the binary decision M` -> `the binary decision mask M`\n- Equation 3: $||l_{2}(\\hat{x_i})-l_{2}(c_{j})||$ -> $||\\hat{x_i}-c_{j}||_2$\n- The 2nd line in Section 4.4: `the token predictor` -> `the token selector`", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-organized and easy to read, and the advantages of the proposed tokenizer are clearly described and demonstrated through various experiments and evaluations. In particular, the method for improving the inefficiency of the conventional vision-text fusion method is effective, and this study can be considered primary research for further improvement.", "weaknesses": "There are a few suggestions for the ablation study.\n1. Compare the performance of text generation using merged continuous embeddings and quantized visual tokens. The loss of detailed information due to quantization is likely to affect the performance of text generation. However, it would be interesting to see how much of a difference there is.\n2. Show the performance difference with and without token merger. It would help to understand the importance of the token merger module.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7L7TnbitmB", "reviewer": "Reviewer_L5Te", "summary": "This paper proposes the idea of treating images and text equally, transforming images into a series of discrete tokens to input into LLM, and training LLM from scratch to complete understanding and generation. The experimental part of the paper yielded so-called advanced results.", "questions": "Refer to weakness.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes to convert the image into a series of discrete tokens and adjust the token length according to the image content.\n2. The visualization of discrete tokens is displayed in Experiments to help readers understand the semantic content carried by tokens.\n3. Better results than the baseline method are achieved in the experiment.", "weaknesses": "1. The novelty of this paper is limited and the ideas of the paper have already been explored in previous work. For example, end-to-end training LLM has been explored in Emu[MR1], and the transformation of visual information into discrete tokens have been explored in [MR2, MR3, MR4, MR5].\n\n2. The generation process relies on a stable diffusion model, which makes the verification of the visual generation capabilities of the proposed model limited. Do the image generation results rely on strong stable diffusion, or are they more attributable to the discrete labels generated by the model? It is unclear to what extent the good results achieved are due to stable diffusion.\n\n3. Visual token merger part is not well validated. How much additional complexity does this stage bring to the model? Are different codebooks needed for data in different domains (animation, painting, natural landscape, etc.)? Why is a complex merger mechanism needed? Can a simple attention map from a pre-trained model indicate the importance of individual image patches?\n\n3+. BTW, the processing of the decoder (using the stable diffusion model) is the same as Emu's [MR1] strategy. A reference to the corresponding location and some justifications are required.\n\n4. In the experimental results in Table 3, using a larger number of tokens (fixed) achieves worse results, and more explanations need to be added.\n\n\n\n\n[MR1] Generative Pretraining in Multimodality. 2023.\n[MR2] Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks. 2023.\n[MR3] Visual Transformers: Token-based Image Representation and Processing for Computer Vision. 2020.\n[MR4] Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training. 2023.\n[MR5] Beit v2: Masked image modeling with vector-quantized visual tokenizers. 20022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LoN85FAvHI", "reviewer": "Reviewer_54hM", "summary": "This paper presents a vision-language framework with unified objectives for both language and image.\nThe crux of the proposed method is the design of a dynamic discrete visual tokenization module.\nIn particular, the proposed LaVIT leverages a dynamic token sparsification approach to embed the given image with fewer vision tokens.\nAfter that, the learned tokenizer is employed to generate discrete vision tokens.\nIn this way, the proposed framework can achieve a unified pre-training objective.", "questions": "- Is it possible that the impressive performance of the proposed method is due to more training data, especially for the vision tokenization codebook?\n\n- Why do the authors still use the unselected tokens as keys and values? \nOne intuitive approach for these tokens is simply discarding them.\nMoreover, these tokens should contribute minorly to the final performance and are redundant as authors claimed.\n\n- For the causal attention, which tokens are seen as previous ones? \nAs there is actually no sequence for an image, using causal attention for image encoding is thus not convincing.\n\n- What is the function of the second half of Eqn. 4?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed method achieves very impressive performance on several vision-language tasks such as VQA and image captioning.\n- The strategy to pre-train the vision-language models with a unified objective seems interesting.\n- The method can generate high-quality images based on the given textual prompt.", "weaknesses": "- The structure of the writing, especially the method section, is very confusing to me.\nIn fact, the key to the proposed method is the learning of the vision token codebook.\nThis codebook and token selector are built on the first stage of learning.\nHowever, the authors' description of this part makes it look like an end-to-end training framework, which confuses me a lot.\nI would suggest the authors carefully revise the structure of this section.\n\n- The authors claim their method can achieve in-context learning capability. \nHowever, I cannot find any experiments pertaining to this merit.\n\n- The authors did not provide a good explanation of why inequitable treatment for language and vision harms multi-modal learning.\n\n- More details about the language-vision pre-training part are required.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)