SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3. Few-shot classification accuracy on miniImageNet using different  $\mathrm{SPAE}_{\mathrm{PaLM}}$  layers.
that is optimal for LLM in-context learning. Overall,  $\mathrm{SPAE}_{\mathrm{PaLM}}$  yields  $+25\%$  and  $+32\%$  average accuracy improvement over the state-of-the-art on the 2-way and 5-way benchmarks in Tab. 1.
Reconstruction quality. We compare the image and video reconstruction quality using the tokens produced by SPAE and the VQGAN baseline used in state-of-the-art image [7, 25, 6] and video generation [45]. We use FID [16], Inception Score (IS) [33], and LPIPS [48] to compare with the image VQGAN from MaskGIT [7] on the ImageNet validation set, and FVD [36] to compare the 3D-VQGAN from MAGVIT [45] on the Kinetics-600 validation set. The results are presented in Tab. 2. While SPAE may have more lossy reconstruction compared to VQGAN when using a similar number of tokens, this is compensated by going into deeper layers. At the bottom of Tab. 2, we showcase the scalability of our model by training on the ImageNet-21k dataset with 13M images and list the comparable variant from LDM [32] as a reference.
Token pyramid visualization. We visualize the tokens produced by SPAE in Fig. 4, where we show the raw pyramid or histogram of tokens with top frequencies for the first four layers, with reconstructed images from layer 5 and 6. We have the following findings.
Figure 4. Examples of pyramid image tokenization and reconstruction by a 6-layer SPAE. We show the raw pyramid or histogram of most frequent tokens for the first four layers, and reconstructed images from layer 5 and 6. In the pyramid, we use darker cells to show tokens with higher CLIP similarity to the original image. For non-English sub-word tokens, we show automatic translation for reference in italic fonts below the original token. Circled tokens are mentioned in Section 4.2. See full pyramid visualizations in the Appendix.
Table 2. Comparison of reconstruction quality between SPAE and VQGAN baselines used in state-of-the-art image [7, 25, 6] and video [45] generation models.
Resolution	Method	# Layers : # Tokens	Image (ImageNet ILSVRC2012 [10])	Video (Kinetics-600 [5]) FVD↓
FID↓	IS↑	LPIPS↓
128×128	VQGAN	1: 256	5.48	119.69	0.13	6.79
SPAE (ours)	5: 341	9.49	109.46	0.17	52.28
6: 597	4.41	133.03	0.12	6.35
256×256	VQGAN	1: 256	4.04	163.95	0.21	-
SPAE (ours)	6: 597	3.60	168.50	0.19	-
VQGAN (LDM [32], OpenImages)	1: 256	5.15	144.55	-	-
SPAE (ours, ImageNet-21k)	6: 597	3.08	173.79	0.19	-
First, the SPAE tokens are organized in a pyramid structure, with every layer comprising semantically related tokens to the image. The few tokens in the top layers seem to capture the primary theme of the image. For instance, in Fig. 4, the token presso (highlighted in orange) represents the espresso machine and other tokens like blender refer to related regions. Layer 3 and Layer 4 reveal additional details about localized objects. For example, the token Thermo refers to the thermometer in the top-left region, while stove appears in the bottom-right area. In addition to nouns, related verbs also show up, including pouring, refill, spill, and brew.
Second, it is worth noting that the CLIP model has an English-only vocabulary. However, thanks to the multilingual vocabularies and embeddings from the LLM, SPAE's semantic guidance is able to map to similar concepts in other languages, such as koffie in Dutch and kaffe in Danish as corresponding terms to the concept of coffee.
Third, similar to RQ tokens [22], SPAE tokens can reconstruct the image with progressively refined details when more layers, and thus tokens, are utilized. Fig. 4 shows Layer 5 begins to produce a reasonable reconstruction while Layer 6 further enhances the level of detail and smoothness.
Visual question answering. Tab. 3 provides quantitative results on the visual question answering (VQA) task. We compare with the baseline Frozen [35] method on the RealFast-VQA [35] benchmark for few-shot learning. As shown, SPAE consistently outperforms Frozen. Unlike Frozen, SPAE training does not require backpropagation through the LLM.
Table 3. Few-shot VQA performance on Real-Fast-VQA.
Inner Shots	1	3	5
Frozen [35]	7.8	10.1	10.5
\( SPAE_{PaLM} \) (ours)	14.3	15.9	15.1
4.3 Qualitative Studies
This section explores the capability of a frozen PaLM 2, trained solely on language tokens, in performing multimodal tasks using in-context learning. We adopt a two-stage decoding process for image generation. In stage one, we use AR decoding to produce the first 5 SPAE layers with task-specific conditions. Stage two is a task-agnostic NAR decoding process for layer 6 conditioned on the first 5 layers.
Image to text and VQA. We examine two tasks involving visual-text reasoning (1) image captioning on COCO [26] captions; and (2) visual question answering (VQA) on COCO-QA [31]. For both
Baseline: A man in a suit standing in front of a white wall.  
SPAE L1: A man in a red jacket and black pants standing on a snowy mountain.  
SPAE L2: A man in a red jacket skiing down a snowy mountain.  
SPAE L3: A man skiing down a snowy mountain.  
SPAE L4: A person skiing down a snowy mountain.  
SPAE L5: A person skiing down a mountain.  
SPAE L6: A person skiing down a mountain.
Baseline: A group of people are standing in a field.  
SPEA L1: A group of people are standing in a room.  
SPEA L2: A kitchen with a stove, sink, and refrigerator.  
SPEA L3: A kitchen with a stove, sink, and refrigerator.  
SPEA L4: A kitchen with a stove, sink, and refrigerator.  
SPEA L5: A kitchen with a stove, sink, and cabinets.  
SPEA L6: A kitchen with a sink, stove, and refrigerator.
Baseline: A man is standing on a rock in the middle of a river.  
SPAE L1: A man is standing on a rock in the middle of a river.  
SPAE L2: A man is wearing a coat and a hat.  
SPAE L3: A man is holding a small dog.  
SPAE L4: A teddy bear is sitting on a bed.  
SPAE L5: A teddy bear is sitting on a bed.  
SPAE L6: A teddy bear is sitting on a bed.
Baseline: A man and a woman are sitting on a bench in a park.  
SPEA L1: A man is holding a baby in his arms.  
SPEA L2: A group of people are standing in a line.  
SPEA L3: A group of people in costumes at a Halloween party.  
SPEA L4: A group of people are dressed up in costumes for Halloween.  
SPEA L5: a group of people dressed in costumes at a party  
SPEA L6: a table with a bowl of fruit and a vase of flowers
SPAE: A pizza with pepperoni and cheese on a white plate.
SPAE: A man in a suit and tie standing next to a woman in a wedding dress.
SPAE: A train is stopped at a station.
O: what is the riding in the A: Baseline
Q: what bear walking through tall grass A: Baseline: Siberian
Q: how many different wines are lined up in glasses on an outdoor table  
A: SPAE: 5
Q: how many computer screens are displayed with one image  
A: SPAE: 3
Figure 5. Qualitative samples of image-to-text generation: image captioning and VQA. We compare between different layers of SPAE (L1-L6) and a baseline model without semantic guidance or pyramid SAQ.
Figure 6. Examples of text-to-image generation on MNIST using SPAE with a frozen PaLM 2 model. We use SPAE to tokenize 50 handwritten images as the context and ask PaLM 2, an LLM trained solely on text tokens, to answer complex queries that require generating digit images through SPAE as the output.
tasks, we provide 10 unique training examples as prompts. In the case of VQA, 10 different answers are presented to form a 10-way 1-shot setup.
We compare SPAE to a baseline model trained with the same frozen language codebook but without the proposed semantic guidance or pyramid SAQ. As shown in Fig. 5, when fed with baseline tokens, the LLM randomly hallucinates a caption or guesses an answer simply based on the question. Similar hallucination can happen if we only use the first two layers of SPAE or five words to represent an image, as it provides insufficient context for captioning. Reasonable captions start to appear with 4 layers or 85 words, while complex scenes may still need the full 6 layers of 597 words.
LLM generating MNIST images. Fig. 6 shows a few image generation examples on MNIST [11]. The frozen LLM learns about handwritten digit images through 50 context samples tokenized by SPAE trained on MNIST. Each sample consists of a preamble "an image of  $k$ " and the lexical tokens representing an image of digit  $k$ . Then we can ask the LLM to answer questions with digit images. Specifically, with a query of "an image of  $1 + 7$ ", we can use progressive AR decoding with the LLM to produce a token sequence that can be decoded into an image of 8 by SPAE. We test with complex questions requiring mathematical reasoning or common sense knowledge, and the LLM is able to respond correctly. In addition, the generated digit images appear different from all context samples. This demonstrates the cross-modal reasoning capability enabled by SPAE and a frozen LLM, with images generated over the text-only interface.
Conditional image denoising. To the best of our knowledge, there have been no successful attempts that demonstrate generic image generation capability using a frozen LLM. To this end, we define a simpler denoising setup to explore the capability of LLMs. Fig. 7 demonstrates the conditional image denoising tasks, e.g., image outpainting, deblur, inpainting, location translation, rotation, etc. Note that, in order to generate images for each task, we utilize 10 pairs of noisy examples with corruption rates ranging from  $50\%$  to  $20\%$ , as discussed in Section 3.2. The full context, which is omitted in Fig. 7, can be found in the Appendix.
Figure 7. Examples of conditional image denoising. We compare different decoding strides for both stages. Yellow and blue boxes indicate the selected results. The LLM is provided with ten pairs of noisy examples like Fig. 2, which are deferred to the Appendix.
Figure 8. Ablation examples with reconstructed image and semantic tokens for models listed in Tab. 4. For non-pyramid tokens, we show a  $4 \times 4$  crop from the first layer corresponding to the region indicated by the black box. For pyramid tokens, we use the third layer which consists of  $4 \times 4$  tokens.
The top rows of Fig. 7 compare the generation from different decoding strides with the same set of context examples. Single-step decoding with infinity stride fails to produce a reasonable image, which validates the proposed progressive generation approach.
In Fig. 9, we qualitatively compare SPAE with baseline methods VQGAN and LQAE using the same in-context denoising procedure. As shown, VQGAN fails to produce reasonable images, in part because many words in the LLM output are out of its vocabulary. LQAE only produces vague object contours but cannot recover any visual details. On the contrary, SPAE can generate reasonable images.
Conditional video denoising and other tasks. Due to space constraints, we show the examples in the Appendix.
The results in Tab. 4 and Fig. 8 verify the effectiveness of the proposed designs in SPAE, as evaluated by reconstruction quality (FID, IS, LPIPS) and semantic relevance (CLIP score,
few-shot classification accuracy). We have the following findings. First, simply using a frozen codebook negatively affects the reconstruction results, but with semantic guidance it performs comparably with the original VQGAN while producing meaningful lexical words. Second, RQ hurts reconstruction quality with a frozen codebook. This is different from RQ's standard setup [22] where the codebook is learned. Third, SAQ improves both quality and semantic similarity, where the pyramid enables representation with much fewer tokens. This allows for accommodating more examples within the fixed and constrained in-context length. Finally, per-layer semantic thresholds benefit understanding and the dynamic semantic loss weight helps reconstruction. The perceptual loss leverages a trained network with access to classification labels, but removing it results in a surprising improvement in classification accuracy while greatly hurting the reconstruction.
Figure 9. Comparison on conditional image denoising with different tokenizers. All models use the same decoding setup with the same ten pairs of prompt images available in the Appendix.
Table 4. Ablation studies on codebook, training objective, quantization method, and token structure. Classification accuracy is evaluated under the mini-ImageNet 5-way 1-shot setup.
Method	# Layers : # Tokens	FID↓	IS↑	LPIPS↓	CLIP↑	Classification Accuracy↑
Baseline VQ	1: 256	5.48	119.69	0.13	n/a	19.6
+ frozen codebook	1: 256	7.44	101.39	0.17	0.1464	19.5
+ semantic loss	1: 256	5.17	124.41	0.13	0.1518	46.2
+ 2-layer RQ [22]	1: 256	11.94	89.01	0.22	0.1595	56.2
2: 512	6.05	113.93	0.15	0.1547	-
+ 2-layer SAQ	1: 256	12.30	93.33	0.21	0.1613	56.6
2: 512	5.08	125.27	0.14	0.1595	-
+ 6-layer pyramid SAQ (SPAE)	1: 1	-	-	-	0.1879	52.0
2: 5	-	-	-	0.1868	64.2
3: 21	-	-	-	0.1815	65.1
4: 85	-	-	-	0.1711	58.5
5: 341	9.49	109.46	0.17	0.1604	46.3
6: 597	4.41	133.03	0.12	0.1577	-
no per-layer thresholds	6: 597	4.33	122.25	0.11	0.1650	59.4 (layer 3)
no dynamic semantic weight	6: 597	9.00	85.14	0.19	0.1847	65.1 (layer 3)
no perceptual loss	6: 597	40.47	33.41	0.20	0.1994	69.5 (layer 3)
Our work unveils the untapped potential of frozen Large Language Models (LLMs) in tackling multimodal understanding and generation tasks involving images and videos, without requiring explicit training on these modalities. This is achieved by a new method, SPAE, which converts between visual content and lexical tokens of variable length, imbued with rich semantic meaning. Our findings show the great potential of harnessing the vast knowledge and reasoning capabilities of LLMs in the field of computer vision, transcending the limitations of language-only tasks.
Limitations. More tokens are required to achieve the same level of reconstruction when using the frozen language codebook, compared to the existing VQGAN models with learned codebooks. The capability of in-context learning is significantly constrained by the acceptable sequence length. Although our results suggest the plausibility of image generation, the resolution, quality, and diversity is far from the recent text-to-image models trained on large image and text data.
Broader impact. Our paper showcases the untapped potential of frozen LLMs in multimodal understanding and generation tasks involving images and videos, without requiring explicit training on these modalities. As an initial research proof-of-concept, we focus on in-context learning, which has limitations in learning context and constrained capabilities. Consequently, there is still a substantial gap to the recent specialized models for text-to-image (e.g., Stable Diffusion) or image-to-text that have been specifically trained using billions of text-image pairs.
The potential impact of our research lies in its influence on future studies, specifically in the area of integrating vision modalities into the LLMs. For instance, our work can be extended to explore finetuning or adapter tuning of LLMs on large-scale text-image datasets. Future research in these directions may implicate ethical issues around fairness and transparency. We have found that the generated tokens occasionally include slang terms or words that create inappropriate connotations related to the subject depicted in the image or video. Such concerns must be thoroughly considered and effectively addressed prior to deploying this method in real-world applications.
Acknowledgments and disclosure of funding. The authors would like to thank anonymous reviewers and area chairs their insightful comments, and to Siamak Shakeri, Sergey Ioffe, Jay Yagnik, and Boqing Gong for their valuable feedback and constructive discussions. This project is funded in part by Carnegie Mellon University's Mobility21 National University Transportation Center, which is sponsored by the US Department of Transportation.
[42] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. TMLR, 2022. 1
[43] Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual ChatGPT: Talking, drawing and editing with visual foundation models. arXiv:2303.04671, 2023. 2
[44] Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, et al. Natural language to code generation in interactive data science notebooks. arXiv:2212.09248, 2022. 3
[45] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. MAGVIT: Masked generative video transformer. In CVPR, 2023. 2, 4, 6, 7
[46] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion-tokenizer is key to visual generation. arXiv:2310.05737, 2023. 2
[47] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Trans. on Audio, Speech, and Language Processing, 30:495-507, 2021. 3, 4
[48] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018. 6
[49] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXiv:2205.01068, 2022. 2