SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lijun Yu†* Yong Cheng† Zhiruo Wang‡ Vivek Kumar† Wolfgang Macherey†
Yanping Huang† David A. Ross† Irfan Essa† Yonatan Bisk‡ Ming-Hsuan Yang†
Kevin Murphy† Alexander G. Hauptmann‡ Lu Jiang†‡
†Google, ‡Carnegie Mellon University
In this work, we introduce Semantic Pyramid AutoEncoder (SPAE) for enabling frozen LLMs to perform both understanding and generation tasks involving non-linguistic modalities such as images or videos. SPAE converts between raw pixels and interpretable lexical tokens (or words) extracted from the LLM's vocabulary. The resulting tokens capture both the semantic meaning and the fine-grained details needed for visual reconstruction, effectively translating the visual content into a language comprehensible to the LLM, and empowering it to perform a wide array of multimodal tasks. Our approach is validated through in-context learning experiments with frozen PaLM 2 and GPT 3.5 on a diverse set of image understanding and generation tasks. Our method marks the first successful attempt to enable a frozen LLM to generate image content while surpassing state-of-the-art performance in image understanding tasks, under the same setting, by over  $25\%$ .
Large language models (LLMs) empowered by Transformers [38] have achieved remarkable progress in addressing a broad spectrum of Natural Language Processing (NLP) tasks [4, 8, 28, 2]. With the continuous increases in model size and training data, LLMs are gradually becoming more versatile and agnostic to specific tasks, unlocking new capabilities in solving complex AI tasks [42], like question answering, code generation, reasoning, mathematics problem-solving, and understanding humor, among various other applications [2, 28].
LLMs capture rich conceptual knowledge about the world in their lexical embeddings. This raises a question: if provided with the appropriate visual representations as input, are frozen LLMs capable of solving tasks in visual modalities? Very recently, there have been notable advancements in extending the capabilities of frozen LLMs to tackle image understanding and retrieval tasks [21, 27]. However, generating a different modality using a frozen LLM that has not been explicitly trained on that modality has proven to be challenging and has had little success.
To facilitate LLMs for such cross-modal tasks, we propose to learn a vector quantizer to map an image, or some other non-linguistic ("foreign") modality, to the token space of a frozen LLM. This effectively translates the image into a language that the LLM can comprehend, enabling us to leverage the generative abilities of the LLM to perform image understanding and generation tasks without having to train on image-text pairs. Specifically, our new approach is that, given an image prompt, convert it to a token space with our learned encoder, use the LLM to generate suitable lexical tokens, and convert back to pixel space with our learned decoder.
*Work partially done during a research internship at Google Research.
Figure 1. Framework of the proposed SPAE model. An image is encoded into a pyramid of lexical tokens capturing semantic concepts and appearance details necessary for reconstruction.
We introduce a novel Semantic Pyramid AutoEncoder (SPAE) that produces a lexical word sequence that (1) carries rich semantics, and (2) retains fine details for signal reconstruction. In contrast to the majority of VQ-VAE approaches [37], our encoder maps to an interpretable discrete latent space, i.e., words. As depicted in Fig. 1, SPAE tokens have a multi-scale representation arranged in a pyramid structure. The upper layers of the pyramid comprise semantic-central concepts, while the lower layers prioritize appearance representations that captures the fine details for image reconstruction. This design enables us to dynamically adjust the token length to accommodate various tasks, such as using fewer tokens for understanding tasks and more tokens for generation tasks.
We verify the plausibility of our approach in an extreme setting of in-context learning [4], without any parameter updates to the LLM. Our SPAE model is trained standalone, without backpropagating through any language model. We evaluate our approach on image understanding tasks including image classification, image captioning, and visual question answering. We showcase a promising direction to image generation with LLMs by utilizing in-context denoising techniques. Our method is LLM-agnostic and has been tested with PaLM 2 [2] and GPT-3.5 [28], suggesting compatibility with arbitrary LLMs.
The main contributions of this work are summarized as follows:
- This is the first successful method, to the best of our knowledge, that uses a frozen language model, trained solely on language tokens, to directly generate image content through in-context learning.
- We introduce a new SPAE tokenizer producing interpretable representations of semantic concepts and fine-grained details in the form of multilingual linguistic tokens with adjustable lengths.
- We evaluate our method on visual understanding and generation tasks, and notably, our approach outperforms the best-published few-shot image classification accuracy [27] by an absolute  $25\%$  under the same in-context setting.
Multimodal generation with LLMs. Advances have been made to expand the capabilities of LLMs beyond language. For example, Visual ChatGPT [43] uses ChatGPT to generate prompts and executes multimodal tasks through another model, e.g., generating image from text prompts by Stable Diffusion [32]. FROMAGe [21] feeds CLIP [30] embeddings to OPT [49] for image understanding and retrieval. However, it requires backpropagation through the LLM and does not support image synthesis. This work enables a standalone frozen LLM to understand and generate other modalities which are unseen in training.
Tokenization via vector quantization. VQ-VAE [37] compresses data into a discrete latent space defined by a codebook via vector quantization. VQGAN [14] enhances the reconstruction quality with adversarial and perceptual objectives. These discrete latent quantities, often referred to as tokens, are widely used to learn generative transformer models for image [32, 7], video [45, 15, 39], image-video [46], and audio [3, 9]. Our SPAE model is built upon the VQGAN framework and applicable to different modalities.
Tokenization into lexical representations. The codebooks in typical VQGANs are learned jointly with the encoder and decoder stacks, which are not directly interpretable via natural languages. LQAE [27] replaces the learned codebook with frozen word embeddings from BERT [12] to connect with an English vocabulary. However, the LQAE tokens seldom contain semantic concepts in an image, and the reconstruction quality is worse than that with a learned codebook. Our SPAE quantizes an input sample into semantically related tokens in a multilingual vocabulary while preserving the high reconstruction quality of a VQGAN for generative tasks. In addition, SPAE tokens are organized in a multi-layer coarse-to-fine pyramid for flexible usage in different tasks.
Few-shot learning with LLMs. In-context learning [4, 8, 2] facilitates LLMs for few-shot learning via the text interface without parameter updates. This approach is commonly employed to assess the performance of LLMs on numerous NLP benchmarks, e.g., classification and question answering [41], mathematical reasoning [24], and code generation [44], which yields competitive results to their fine-tuned counterparts. However, existing few-shot vision-language understanding and generation frameworks [1, 21] still require LLM parameter updates. In contrast, our work inherits the in-context learning ability from frozen LLMs.
Our goal is to model an image, or some other non-linguistic modality (e.g., video or audio), as a language sequence that LLMs can comprehend. Semantic Pyramid AutoEncoder (SPAE) generates a lexical word sequence with dynamically adjustable length that carries rich semantics and retains fine details for signal reconstruction. To work with a frozen LLM via in-context learning, we introduce a progressive in-context denoising method to facilitate image generation. We use the image modality in this section to introduce our SPAE model in 2D, and later showcase the results of a 3D variant with the video modality in our experiments.
3.1 Semantic Pyramid AutoEncoder
Our SPAE model extends the VQ-VAE [37] framework, which comprises an encoder, a quantizer, and a decoder. The CNN encoder maps an image  $\mathbf{I} \in \mathbb{R}^{H \times W \times 3}$  into continuous embeddings  $\mathbf{Z} \in \mathbb{R}^{h \times w \times c}$ . Each element  $\mathbf{z} \in \mathbf{Z}$  is then passed through the quantizer, which assigns it to the closest entry in a codebook, resulting in the quantized embedding. Let  $\hat{\mathbf{Z}}$  represent the quantized embeddings for the entire image. The CNN decoder receives  $\hat{\mathbf{Z}}$  as input and generates the reconstructed image  $\hat{\mathbf{I}}$ . Below we highlight the design differences in SPAE.
As illustrated in Fig. 1, SPAE generates lexical tokens arranged in a pyramid structure, which contains semantic concepts in the upper layers and appearance with progressively refined details in the lower layers. We introduce a semantic loss to encourage the usage of conceptually relevant tokens.
Frozen language codebook. To generate lexical tokens, we utilize a pretrained LLM codebook  $\mathbb{C} = \{(k,\mathbf{e}(k))\mid k\in \mathbb{T}\}$  and freeze it during training, where  $\mathbb{T}$  is a subset of the LLM vocabulary. Here,  $\mathbf{e}(\cdot)$  produces the text embedding for a sub-word  $k$  which may be obtained from any layer of the LLM. Since the codebook is aligned with the language vocabulary, we use the terms "token" and "word" interchangeably.
Token pyramid. The SPAE quantizer produces  $D$  layers of tokens where the tokens at layer  $l$  are denoted as  $\mathbf{k}_l \in \mathbb{T}^{h_l \times w_l}$ . Prior works use Residual Quantization (RQ) to generate multi-layer tokens [22, 47]. In these methods, tokens from all layers have uniform shapes and do not carry specific semantic meanings. In contrast, we propose a pyramid token structure by enforcing the constraint  $h_l \leq h_{l+1} \wedge w_l \leq w_{l+1}$ . The pyramid structure is purposefully designed to concentrate semantics within the within the upper layers of the pyramid. This design allows for representing semantic concepts with notably fewer tokens, e.g., as few as five tokens for understanding tasks. The high token efficiency stems from the pyramid structure, as a conventional layer without pyramid structures needs a minimum of  $hw$  tokens (e.g., 256) to represent the image. Token efficiency is crucial for in-context learning as it enables the accommodation of more examples within the context. A dilation subsampler  $\mathbf{P}(l)$  is used, which selects the positions for quantization at layer  $l$  as
$$
\mathbf {P} (l) = \left\{\left(h ^ {\prime} i - \left\lceil \frac {h ^ {\prime}}{2} \right\rceil + 1, w ^ {\prime} j - \left\lceil \frac {w ^ {\prime}}{2} \right\rceil + 1\right) \mid (i, j) \in ([ 1, h _ {l} ] \times [ 1, w _ {l} ]) \cap \mathbb {Z} ^ {2} \right\}, \tag {1}
$$
where  $h' = \frac{h_D}{h_l}$ , and  $w' = \frac{w_D}{w_l}$  are the downsample ratios.
For each embedding  $\mathbf{z}$  at position  $(x,y)$ , we obtain its discrete tokens sequentially from layer 1 to  $D$ . At layer  $l$ , if  $(x,y) \in \mathbf{P}(l)$ , the quantizer assigns discrete token  $k_l = \arg \min_{k \in \mathbb{T}} \| \mathbf{z}_l - \mathbf{e}(k) \|_2^2$ ,
where  $\mathbf{z}_l$  is the current layer embedding, calculated from
$$
\mathbf {z} _ {l} = \mathbf {z} + \sum_ {i = 1} ^ {l - 1} \mathbf {1} _ {(x, y) \in \mathbf {P} (i)} (\mathbf {z} - \mathbf {e} (k _ {i})). \tag {2}
$$
The quantized embedding reconstructed with the first  $l$  layers is given by the average of the existing token embeddings as
$$
\hat {\mathbf {z}} _ {\leq l} = \frac {\sum_ {i = 1} ^ {l} \mathbf {1} _ {(x , y) \in \mathbf {P} (i)} \mathbf {e} \left(k _ {i}\right)}{\sum_ {i = 1} ^ {l} \mathbf {1} _ {(x , y) \in \mathbf {P} (i)}}. \tag {3}
$$
Using the input of  $\hat{\mathbf{Z}}_{\leq l}$  from tokens up to layer  $l$ , the decoder can progressively reconstruct the image with dynamic token lengths, resulting in gradually improved quality with refined appearance details. We term this approach as Streaming Average Quantization (SAQ) due to its resemblance to computing the average on streaming data, where  $\hat{\mathbf{z}}_{\leq l + 1} = \hat{\mathbf{z}}_{\leq l} + \frac{1}{\hat{l} + 1}\mathbf{e}(k_{l + 1}),\hat{l} = \sum_{i = 1}^{l}\mathbf{1}_{(x,y)\in \mathbf{P}(i)}$
RQ [22, 47] is applicable but yields worse results in this context, as revealed by our ablation studies. This can be attributed to (1) varying scales of embeddings in residual layers, potentially dividing the codebook into multiple parts, and (2) misalignment in the summation of word embeddings, which undermines learning semantically meaningful tokens in later layers.
Semantic loss. We encourage the semantic similarity between the image  $\mathbf{I}$  and each lexical token  $k$  denoted by  $s(\mathbf{I}, k)$ . During training, we build per-layer candidate token pools as
$$
\mathbf {C} _ {l} (\mathbf {I}) = \{k \in \mathbb {T} \mid s (\mathbf {I}, k) \geq \rho_ {l} \}, \tag {4}
$$
where  $\rho_{l}$  is a threshold. We set  $\rho_{l} \geq \rho_{l+1}$  to allow deeper layers to have a larger pool of candidate tokens while sacrificing some semantics.
To define the similarity score, this paper employs a pretrained CLIP model [29]. In more details, let  $f_{\mathcal{I}}$  and  $f_{\mathcal{T}}$  be a pair of image and text CLIP embedding functions. We precompute the text feature for each token  $k \in \mathbb{T}$  as
$$
f _ {\mathcal {T}} ^ {\prime} (k) = \frac {1}{| \mathbf {p} |} \sum_ {i = 1} ^ {| \mathbf {p} |} f _ {\mathcal {T}} \left(\mathrm {p} _ {i} (k)\right), \tag {5}
$$
where  $\mathbf{p}$  is a list of prompt templates, such as "a photo of ...". During training, we extract the image feature  $f_{\mathcal{I}}(\mathbf{I})$  and compute the dot-product similarity as  $\mathbf{s}'(\mathbf{I}, k) = f_{\mathcal{I}}(\mathbf{I}) \cdot f_{\mathcal{T}}'(k)$ . The similarity score is then normalized to account for the varying scales across different images.
$$
\mathbf {s} (\mathbf {I}, k) = \frac {\mathbf {s} ^ {\prime} (\mathbf {I} , k) - \operatorname* {m i n} _ {j} \mathbf {s} ^ {\prime} (\mathbf {I} , j)}{\operatorname* {m a x} _ {j} \mathbf {s} ^ {\prime} (\mathbf {I} , j) - \operatorname* {m i n} _ {j} \mathbf {s} ^ {\prime} (\mathbf {I} , j)}. \tag {6}
$$
We define the semantic loss for the encoder parameters  $\theta_{e}$  as
$$
\mathcal {L} _ {\text {s e m a n t i c}} \left(\theta_ {e}; \mathbf {I}\right) = \underset {l \in [ 1, D ^ {\prime} ]} {\mathbb {E}} \underset {\mathbf {z} _ {l}} {\mathbb {E}} \underset {c \in \mathbf {C} _ {l} (\mathbf {I})} {\mathbb {E}} - \log \frac {\exp \left(- \| \left(\mathbf {z} _ {l} - \mathbf {e} (c) \| _ {2} ^ {2}\right)\right)}{\sum_ {k \in \mathbb {T}} \exp \left(- \| \mathbf {z} _ {l} - \mathbf {e} (k) \| _ {2} ^ {2}\right)}, \tag {7}
$$
where we randomly sample semantically similar target codes  $c$  for each layer embedding in the first  $D'$  layers.
Appearance loss. Using an improved objective from [45], the appearance loss is calculated as:
$$
\mathcal {L} _ {\text {a p p e a r a n c e}} \left(\theta_ {e}, \theta_ {d}; \mathbf {I}\right) = \| \mathbf {I} - \hat {\mathbf {I}} \| _ {2} ^ {2} + \beta \sum_ {l = 1} ^ {D} \| \mathbf {Z} - \operatorname {s g} (\hat {\mathbf {Z}} _ {\leq l}) \| _ {2} ^ {2} + \lambda \mathcal {L} _ {\mathrm {G A N}} + \eta \mathcal {L} _ {\text {P e r c e p t u a l}} + \phi \mathcal {L} _ {\text {L e C A M}}, \tag {8}
$$
where  $\mathcal{L}_{GAN}$ ,  $\mathcal{L}_{\text{Perceptual}}$ , and  $\mathcal{L}_{\text{LeCAM}}$  are the VQGAN [15], perceptual [19], and LeCAM [34] losses. In addition,  $\mathrm{sg}(x)$  is the stop-gradient operation. The appearance loss is applied to both the encoder  $\theta_e$  and decoder parameters  $\theta_d$ , excluding the frozen codebook embedding.
To stabilize the training and balance between appearance and semantics, we add a dynamic weight for the semantic guidance loss as  $w = \mathrm{sg}\left(\frac{\mathcal{L}_{\mathrm{appearance}}(\mathbf{I})}{\mathcal{L}_{\mathrm{semantic}}(\mathbf{I})}\right)$ . The total training loss excluding the GAN discriminator is
$$
\mathcal {L} _ {\mathrm {S P A E}} \left(\theta_ {e}, \theta_ {q}\right) = \underset {\mathbf {I}} {\mathbb {E}} \left[ \mathcal {L} _ {\text {a p p e a r a n c e}} \left(\theta_ {e}, \theta_ {q}; \mathbf {I}\right) + \alpha w \mathcal {L} _ {\text {s e m a n t i c}} \left(\theta_ {e}; \mathbf {I}\right) \right]. \tag {9}
$$
Figure 2. An example of the conditional image denoising task for high resolution synthesis. The context comprises images randomly corrupted in the token space.
3.2 Progressive In-Context Decoding
While our method is more effective when backpropagating through LLMs by prompt [23] or adapter tuning [17, 18], this work focuses on verifying the plausibility in an extreme setting of in-context learning [4]. We demonstrate that LLMs are capable of performing new tasks in foreign modalities without any parameter updates. Specifically, a set of  $K$  examples  $\{(\mathbf{u}^i,\mathbf{v}^i)\}_{i = 1}^K$  are fed to the LLM to learn a new task and answer a query  $\hat{\mathbf{u}}$  with
$$
\hat {\mathbf {v}} \sim \mathbb {P} _ {\mathrm {L L M}} (\cdot \mid \hat {\mathbf {u}}; \left\{\left(\mathbf {u} ^ {i}, \mathbf {v} ^ {i}\right) \right\} _ {i = 1} ^ {K}). \tag {10}
$$
Sampling  $\hat{\mathbf{v}}$  by a single-pass autoregressive decoding is suboptimal due to the distributional shift in the representation and the presence of exceptionally long sequences, e.g., an image is quantized into over 500 tokens. To this end, we use a progressive decoding method.
We generalize Eq. (10) into a multi-pass process, where the LLM learns to generate one segment of the target sequence at a time. The segment generated from the  $t$ -th pass is
$$
\hat {\mathbf {v}} _ {t} \sim \mathbb {P} _ {\mathrm {L L M}} (\cdot \mid [ \hat {\mathbf {u}}, \hat {\mathbf {v}} _ {<   t ^ {\prime}} ]; \left\{\left([ \mathbf {u} ^ {i}, \mathbf {v} _ {<   t ^ {\prime}} ^ {i} ], \mathbf {v} _ {t} ^ {i}\right) \right\} _ {i = 1} ^ {K}), \tag {11}
$$
where  $[\cdot, \cdot]$  indicates concatenation.  $t'$  controls the length of previous segments to condition on, with two common cases: (1) a progressive autoregressive (PAR) process with  $t' = t$ , where each decoded segment conditions on all previously decoded ones; (2) a progressive non-autoregressive (PNAR) process with  $t' = 0$  to sample each segment independently, which greatly reduces the sequence length requirement for the LLM. In practice, we use PAR to generate the first few token layers given task-specific conditions, followed by PNAR to generate the remaining token layers conditioned on the previous layers in an unconditional latent refinement process.
The learning capacity of an in-context setup is far from sufficient for a modality that has not been seen during training. So far, there have been no successful attempts in the literature demonstrating that a frozen LLM can generate image content. For low-resolution images, LLMs can produce images directly using in-context learning, as will be demonstrated with  $32 \times 32$  MNIST images [11]. For higher resolutions, the context length restricts the number of examples. For instance, a context window of 8k tokens can only hold less than a dozen  $128 \times 128$  images. Therefore, we operate in a denoising subspace to synthesis beyond  $32 \times 32$  resolution. Fig. 2 illustrates one example, with detailed definitions in the Appendix.
To verify the compatibility with different LLMs, we train two variants of SPAE, namely  $\mathrm{SPAE}_{\mathrm{PaLM}}$  and  $\mathrm{SPAE}_{\mathrm{GPT}}$ . The  $\mathrm{SPAE}_{\mathrm{PaLM}}$  codebook is taken from the input embedding layer of a PaLM 2-S checkpoint with a 65k vocabulary of the most frequent sentence pieces. The PaLM 2-L API [2] is used for in-context learning with  $\mathrm{SPAE}_{\mathrm{PaLM}}$ .  $\mathrm{SPAE}_{\mathrm{GPT}}$  uses a byte-pair encoding vocabulary with 99k UTF-8 tokens (https://github.com/openai/tiktoken), where we obtain the contextual token embeddings from OpenAI text-embedding-ada-002 (https://platform.openai.com/docs/models/embeddings). For a fair comparison with prior works [27], we use  $\mathrm{SPAE}_{\mathrm{GPT}}$  with the GPT 3.5 text-davinci-003 API (https://platform.openai.com/docs/models/gpt-3-5).
We configure SPAE to encode a  $128 \times 128$  image into a token pyramid of 6 layers where each layer has  $2^{k} \times 2^{k}$  tokens and  $k = [0,1,2,3,4,4]$ . Additionally, we train a video-based SPAE model on the Kinetics-600 dataset [5], and further details can be found in the Appendix. We apply semantic guidance loss to the first five layers, with thresholds of 0.98, 0.95, 0.9, 0.85, and 0.8. The CLIP with a ViT-L/14 [13] vision backbone is used. We use 80 prompt templates from the zero-shot ImageNet
Table 1. Few-shot classification accuracy on the mini-ImageNet benchmarks.  $\mathrm{SPAE}_{\mathrm{GPT}}$  and  $\mathrm{SPAE}_{\mathrm{PaLM}}$  are trained using different vocabularies and embedding sources, with different prompt templates for in-context learning. They show the broad compatibility of SPAE but are not for a comparison between the LLMs. The best performance with GPT is in italics while the overall best is in bold.
Method	Task Induction	✓	✓	✓	✓	✓	✓	Avg
# Layers : # Tokens	Inner Shots Repeats	1	1	3	5	1	1	5
2-Way Classification
Frozen [35]	-	1.7	33.7	66	66	63	65	51.3
LQAE [27]	1: 256	GPT 3.5	1.5	35.2	68.2	69.8	68.5	68.7	53.97
\( SPAE_{GPT} \) (ours)	2: 5	GPT 3.5	5.3	77.2	84.4	86.0	79.4	77.2	69.51
\( SPAE_{PaLM} \) (ours)	2: 5	PaLM 2	32.2	84.0	88.5	88.4	85.1	83.6	77.74
\( SPAE_{PaLM} \) (ours)	3: 21	PaLM 2	27.9	84.8	92.5	92.6	84.8	85.2	79.03
5-Way Classification
Frozen [35]	-	0.9	14.5	34.7	33.8	33.8	33.3	26.26
LQAE [27]	1: 256	GPT 3.5	1.0	15.7	35.9	36.5	31.9	36.4	29.04
\( SPAE_{GPT} \) (ours)	2: 5	GPT 3.5	4.3	63.0	63.4	60.6	61.9	62.1	53.91
\( SPAE_{PaLM} \) (ours)	2: 5	PaLM 2	23.6	64.2	68.0	69.9	63.4	62.0	58.76
\( SPAE_{PaLM} \) (ours)	3: 21	PaLM 2	20.2	65.1	73.7	74.3	66.4	67.0	61.86
classification setup to precompute the CLIP text embeddings for the vocabulary. In addition, we use the Adam [20] optimizer with loss weights  $\alpha = 1$ ,  $\beta = 0.33$ ,  $\lambda = 0.1$ ,  $\eta = 0.1$ ,  $\phi = 10^{-4}$  and a learning rate of  $10^{-4}$  following a linear warmup/cooldown and root square decay schedule. Following the prior work [27], SPAE is trained on the ImageNet ILSVRC2012 [10] dataset. We train with a batch size of 256 for 450k steps. Further details can be found in the Appendix.
4.2 Main Evaluation
Few-shot image classification. We evaluate the in-context image understanding capability with a frozen LLM on the mini-ImageNet [40] few-shot classification benchmark. A set of tokenized images and class labels are fed to the language model as context for classification of a new image. Following [35, 27], we evaluate 14 settings controlled by four factors regarding the content of each test case: (1) task induction: whether including a preamble to specify the output space; (2) number of ways: the number of categories; (3) number of inner shots: the number of unique examples for each category; (4) number of repeats: the number of times that each unique example is repeated.
We compare SPAE with the state-of-the-art methods Frozen [35] and LQAE [27]. As shown in Tab. 1,  $\mathrm{SPAE}_{\mathrm{GPT}}$  consistently outperforms LQAE, both using the same GPT 3.5 model and in-context format, while using only  $2\%$  of its tokens. Fig. 3 shows the performance trend when using different number of  $\mathrm{SPAE}_{\mathrm{PaLM}}$  layers across six settings with task induction and 0 repeats.  $\mathrm{SPAE}_{\mathrm{PaLM}}$  with 3 layers achieves the best performance which balances between sufficient semantics and an image sequence length