SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs
Authors: Lijun Yu
Yong Cheng
Zhiruo Wang
Vivek Kumar
Wolfgang Macherey
Yanping Huang
David A Ross
Irfan Essa
Yonatan Bisk
Ming-Hsuan Yang
Kevin Patrick Murphy
Alexander G Hauptmann
Lu Jiang
Venue: NeurIPS 2023 spotlight (2023)
Keywords: multimodal
generation
large language model
TL;DR: A language-grounded quantization method that enables in-context few-shot multimodal generation with frozen LLMs.

Abstract: In this work, we introduce Semantic Pyramid AutoEncoder (SPAE) for enabling frozen LLMs to perform both understanding and generation tasks involving non-linguistic modalities such as images or videos. SPAE converts between raw pixels and interpretable lexical tokens (or words) extracted from the LLM's vocabulary. The resulting tokens capture both the rich semantic meaning and the fine-grained details needed for visual reconstruction, effectively translating the visual content into a language comprehensible to the LLM, and empowering it to perform a wide array of multimodal tasks. Our approach is validated through in-context learning experiments with frozen PaLM 2 and GPT 3.5 on a diverse set of image understanding and generation tasks.
Our method marks the first successful attempt to enable a frozen LLM to generate image content while surpassing state-of-the-art performance in image understanding tasks, under the same setting, by over 25%.

SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs
Lijun Yu
Yong Cheng?
Zhiruo Wang!
Vivek Kumart
Wolfgang Macherey
Yanping Huang
David A. Ross
Irfan Essal
Yonatan Bisk
Ming-Hsuan Yang"
Kevin Murphy
Alexander G. Hauptmann
Lu Jiang
Google, *Carnegie Mellon University
Abstract
In this work, we introduce Semannti Pyramid AutoEncoder (SPAE) for enabling frozen LLMs perform both understanding and generation tasks involving non- linguistic modalities such as images or videos. SPAE converts between raw pixels and interpretable lexical tokens (or words) extracted from the LLM's vocabulary. The resulting tokens capture both the semantic meaning and the fine-grained details needed for visual reconstruction, effectively translating the visual content into a language comprehensible to the LLM, and empowering it to perform a wide array of multimodal tasks. Our approach is validated through in-context learning experiments with frozen PaLM and GPT 3.5 on diverse set of image understanding and generation tasks. Our method marks the first successful attempt to enable a frozen LLM to generate image content while surpassing state-of-the-an performance in image understanding tasks, under the same setting, by over 25%.
Introduction
Large language models (LLMs) empowered by Transformers [38] have achieved remarkable progress addressing a broad spectrum of Natural Language Processing (NIP) tasks [4, 8, 28, 2]. With the continuous increases in model size and training data, LLMs gradually becoming more versatile and agnostic to specific tasks, unlocking new capabilities in solving complex Al tasks [42], like question answering, code generation, reasoning, mathematics problem-solving, and understanding humor, among various other applications [2, 28].
LLMs capture rich conceptual knowledge about the world in their lexical embeddings. This raises question: if provided with the appropriate visual representations as input, are frozen LLMs capable of solving lasks in visual modalities? Very recently, there have been notable advancements in extending capabilities of frozen I.LMs to tackle image understanding and retrieval tasks [21, 27]. However, generating a different modality using a frozen LLM that has not been explicitly trained on that modality has proven to be challenging and has had little success.
To facilitate LLMs for such cross-modal tasks, we propose to learn a vector quantizer to map an image, or some other lin-linguistic ("foreign"") modality, the token space of frozen LLM. This effectively translates the image into language that the LLM can comprehend, enabling us to leverage the generative abilities of the LLM perform image understanding and generation tasks without having to train on mmage-text pairs. Specifically, our new approach is that, given an image prompt, convert token space with our learned encoder, use the L.M to generate suitable lexical tokens, and convert back to pixel space with our learned decoder.
Work partially done during a research internship at Google Research.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

LLM codebook
CLIP
Reconstructed image
Layer
Semantic
Layer
breach eert, eeple Top & loyers (8S tollens)
layers
Decoder
branch,
bird, seld olae ..
Encoder
loyers (342 takenns)
Meir:
layers
breath entt sagl kivd voll alib. wings fL. uft,... (597 tokens)
Layer
2000
layers
Figure 1. Framework of the proposed SPAE model. An image encoded into a pyramid of lexical tokens capturing semantic concepts and appearance details necessary for reconstruction.
We introduce a novel Semantic Pyramid AutoEncoder (SPAE) that produces lexical word sequence that (1) carries rich semantics, and (2) retains fine details for signal reconstruction. In contrast the majority of VQ- VAE approaches [37], our encoder maps to an interpretable discrete latent space, i.e. words. As depicted Fig. 1, SPAE tokens have a multi-scale representation arranged in a pyramid structure. The upper layers of the pyramid comprise semantic-central concepts, while the lower layers prioritize appearance representations that captures the fine details for image reconstruction. This design enables us dynamically adjust the token length accommodate various tasks, such using fewer tokens for understanding tasks and more tokens for generation tasks.
We verify the plausibility of our approach in an extreme setting of in-context learning [4], without any parameter updates to the LLM. Our SPAE model is trained standalone, without backpropagating through any language model. We evaluate our approach on image understanding tasks including image classification, image captioning, and visual question answering. We showcase promising direction to image generation with LLMs by utilizing in-context denoising techniques. Our method is LLM-agnostic and has been tested with PaLM [2] and GPT-3.5 [28), suggesting compatibility with arbitrary LLMs.
The main contributions of this work are summarized as follows:
This is the first successful method, to the best of our knowledge, that uses a frozen language model, trained solely language tokens, directly generate image content through in-context learning. We introduce a new SPAE tokenizer producing interpretable representations of semantic concepts and fine-grat one-grained details in the form of multilingual linguistic tokens with adjustable lengths.
We evaluate our method on visual understanding and generation tasks, and notably, our approach outperforms the best-published few-shot image classification accuracy [27] by absolute 25% under the same in-context setting.
2 Related Work
Multimodal generation with LLMs. Advances have been made expand the capabilities of LLMs beyond language. For example, Visual ChatGPT [43] uses ChatGPT to generate prompts and executes multimodal tasks through another model, e.g., generating image from text prompts by Stable Diffusion [32]. FROMAGe [21] feeds CLIP [30] embeddings to OPT [49] for image understanding and retrieval. However, it requires backpropagation through the LLM and does not support image synthesis. This work enables a standalone frozen L.M to understand and generate other modalities which are unseen in training.
Tokenization via vector quantization. VQ-VAE [37] compresses data into discrete latent space defined by a codebook via vector quantization. VQGAN [14] enhances the reconstruction quality with adversarial and perceptual objectives. These discrete latent quantities, often referred to as tokens, are widely used to learn generative transformer models for image [32, 7], video [45, 15, 39). image-video [46], and audio [3, 9]. Our SPAE model is built upon the VQGAN framework and applicable to differrnt modalities.

---

Tokenization into lexical representations. The codebooks in typical VQGANs are learned jointly with the encoder and decoder stacks, which are not directly interpretable via natural languages. I.QAE [27] replaces the learned codebook with frozen word embeddings from BERT [12] to connect with an English vocabulary. However, the LQQE tokens seldom contain semantic concepts in an image, and the reconstruction quality is worse than that with a learned codebook. Our SPAE quantizes an input sample into semantically related tokens in a multilingual vocabulary while preserving the high reconstruction quality of a VQGAN for generative tasks. In addition, SPAE tokens are organized in multi-layer coarse-to-fine pyramid for flexible usage in different tasks.
Few-shot learning with LLMss In-context learning [4, 8. 2] facilitates LLMs for few-shot learning via the text interface without parameter updates. This approach is commonly employed to assess the performance of LLMs on numerous NLP benchmarks, e.g., classification and question answering [41], mathematical reasoning [24], and code generation [44], which yields competitive results to their fine-tuned counterparts. However, existing few-shot vision- risoon-lagguage understanding and generation frameworks [1, 21] still require LLM parameter updates. In contrast, our work inherits the in-context learning ability from frozen L.MSs.
3 Method
Our goal is model image, or some other non-linguistic modality (e.g.. video or audio), language sequence that LLMs can comprehend. Semantic Pyramid AutoEncoder (SPAE) generates a lexical word sequence with dynamically adjustable length that carries rich semantics and retains fine details for signal reconstruction. To work with a frozen LLM via in-context learning, we introduce a progressive in-context denoising method to facilitate image generation. We use the image modality in this section to introduce our SPAE model in 2D, and later showcase the results of a 3D variant with the video modality in our experiments.
3.1 Semantic Pyramid AutoEncoder
Our SPAE model extends the VQ-VAE [37] framework, which comprises an encoder, a quantizer, and a decoder. The CNN encoder maps an image I E R "x into continuous embeddings Z E Rhxwxe Each element 2 E Z is then passed through the quantizer, which assigns it to the closest entry in codebook, resulting in the quantized embedding. Let Z represent the quantized embeddings for the entire image. The CNN decoder receives Z input and generates the reconstructed image I. Below we highlight the design differences in SPAE.
As illustrated in Fig. 1, SPAE generates lexical tokens arranged in pyramid structure, which contains semantic concepts in the upper layers and appearance with progressively refined details in the lower layers. We introduce semantic loss to encourage the usage of conceptually relevant tokens. Frozen language codebook. To generate lexical tokens, we utilize a pretrained L.M codebook C =((k, e(k)) & E T) and freeze it during training, where T is subset of the LLM vocabulary. Here, e(-) produces the text embedding for a sub-word k which may be obtained from any layer of the LLM. Since the codebook is aligned with the language vocabulary, we use the terms "token" and "word" interchangeably.
Token pyramid. The SPAE quantizer produces 1) layers of tokens where the tokens at layer I are denoted as kj- E Thix,, Prior works use Residual Quantization (RQ) to generate multi-layer tokens [22, 47]. In these methods, tokens from all layers have uniform shapes and do not carry specific semantic meanings. In contrast, we propose pyramid token structure by enforcing the constraint h4 < h1+1 A Wi S wi+1. The pyramid structure is purposefully designed to concentrate semantics within the within the upper layers of the pyramid. This design allows for representing semantic concepts with notably fewer tokens, e.g., as few as five tokens for understanding tasks. The conventional layer without pyramid
high token efficiency stems from the pyramid structure, structures needs a minimum of hw tokens (e.g.. 256) to represent the image. Token efficiency is crucial for in-context learning it enables the accommodation of more examples within the context. A dilation subsampler P(]) is used, which selects the positions for quantization at layer 1 as h²'
P(I) = (hh'
+ 1, w'j
+ (i,5) E ((1, Au] x [1, wn)) n 2²),
(I)
2
where h/' = hiz and w' = wan wi
are the downsample ratios.
hy
For each embedding z at position (x) we obtain its discrete tokens sequentially from layer to D. At layer I, if (z,y) e P()), the quantizer assigns discrete token ky arg minkeT ||z - e(
3

---

where z is the current layer embedding, calculated from
2 ΣE= ((..))P()( e(k:))
The quantized embedding reconstructed with the first / layers is given by the average of the existing token embeddings as
Ei=1 (())))())()))))
2gi E. L(w))()))
(3)
Using the input of Zei from tokens up to layer I, the decoder can progressively reconstruct the image with dynamic token lengths, resulting in gradually improved quality with refined appearance details. We term this approach as Streaming Average Quantization (SAQ) due to its resemblance to computing
the average on streaming data, where ă = ăgi + [+1 ek(+1), E (()))
RQ [22, 47] is applicable but yields worse results in this context, revealed by our ablation studies. This can be attributed to (1) varying scales of embeddings in residual layers, potentially dividing the codebook into multiple parts, and (2) misalignment in the summation of word embeddings, which undermines learning semantically meaningful tokens later layers.
Semantic loss. We encourage the semantic similarity between the image and each lexical token denoted by sII, k). During training, we build per-layer candidate token pools as
Cx(I) = {k E T s(I, 2 pt),
(4)
where A is threshold. We set P Pt+1 allow deeper layers to have larger pool of candidate tokens while sacrificing some semantics.
To define the similarity score, this paper employs pretrained CLIP model [29]. In more details, let JI and JT be pair of image and text CLIP embedding functions. We precompute the text feature for each token A E T as
1 E!Pl (p)) |pl
(k(
(5)
where p list of prompt templates, such as photo ... During training, we extract the image feature fz and compute the dot-product similarity as s' kl(1,)) = fz(1) (k).. The similarity score is then normalized to account for the varying scales across different images.
s' s'(I,k) miny s' (J
II, k)
(6)
max, s'(I.)) miny s' (I.
We define the semantic loss for the encoder parameters 0g as
exp(-Iles - e(c)
Csemantic (0;;
E
E
E
- log
(7)
IE(LDD
cEC.(1)
KET exp(- |zz e(k)
where we randomly sample semantically similar target codes for each layer embedding in the first D' layers.
Appearance loss.
Using an improved objective from [45], the appearance loss is calculated as:
Cappeurance 0 1- 1 + D Z sg (2g)) + A Crrpaa-CCCCCttooooooonnottottiit
+LLICAM. (8)
where LGAN. Pereeppuaa, and LLECAM are the VQGAN [15], perceptual [19], and LeCAM [34] losses. addition, sg(r) is the stop-gradient operation. The appearance loss is applied to both the encoder 0. and decoder parameters lg excluding the frozen codebook embedding.
To stabilize the training and balance between appearance and semanties, we add a dynamic weight Lappearaneu (1)
for the semantic guidance loss as sg
Lsemancie (I) ). The total training loss excluding the GAN
discriminator
LSPAE(O., 0,) ( [Lappearancca (0c, 0g: I) + oow Cormantit (0.:
(9)

---

Quary
Iinuu
Outgut
Figure 2. An example of the conditional image denoising task for high resolution synthesis. The context comprises images randomly corrupted in the token space.
3.2 Progressive In-Context Decoding
While our method is more effective when backpropagating through LLMs by prompt [23] or adapter tuning [17, 18], this work focuses verifying the plausibility in an extreme setting of in-context learning [4]. We demonstrate that LLMs are capable of performing new tasks in foreign modalities without any parameter updates. Specifically, set of K examples {(u', v') are fed to the LLM to learn a new task and answer query û with
v ~ PLLM(-1 ú; (t: v')',,
(10)
Sampling v by a single-pass autoregressive decoding is suboptimal due to the distributional shift in the representation and the presence of exceptionally long sequences, e.g., image is quantized into over 500 tokens. To this end, we use a progressive decoding method.
We generalize Eq. (10) into multi-pass process, where the LLM learns to generate one segment of the target sequence at a time. The segment generated from the t-th pass is
V ~ PL [ú, Veel: {(u',, ve D)
(11)
where [ indicates concatenation. controls the length of previous segments to condition on, with two common cases: (1) a progressive autoregressive (PAR) process with t' = t, where each decoded segment conditions on previously decoded ones; (2) a progressive non-autoregressive (PNAR) process with L' = 0 to sample each segment independently, which greatly reduces the sequence length requirement for the I.M.. In practice, we use PAR to generate the first few token layers given task-specific conditions, followed by PNAR to generate the remaining token layers conditioned on the previous layers in an unconditional latent refinement process.
The learning capacity of an in-context setup is far from sufficient for a modality that has not been seen during training. So far, there have been no successful attempts in the literature demonstrating that a frozen LLM can generate image content. For low-resolution images, LLMs can produce images directly using in-context learning, as will be demonstrated with 32x32 MNIST images [11]. For higher resolutions, the context length restricts the number of examples. For instance, context window of 8k tokens can only hold less than a dozen 128 x 128 images. Therefore, we operate in
denoising subspace to synthesis beyond 32x32 resolution. Fig. 2 illustrates one example, with detailed definitions in the Appendix.
Experimental Results
4.1 Experimental Settings
To verify the compatibility with different LLMs, we train two variants of SPAE, namely SPAEPMI and SPAEGPT. The SPAERAL codebook is taken from the input embedding layer of a PaLM 2-S checkpoint with a 65k vocabulary of the most frequent sentence pieces. The PaLM 2-L API [2] is used for in-context learning with SPAEPaLM- SPAEGPT uses a byte-pair encoding vocabulary with 99k UTF-8 tokens (https: //gos:///gttull com/ wwtp//ppiantttteeeeeeeeeeo token embeddings from OpenAl text eeeeettemeddnnn--ee -02 ptttps://platie
where we obtain the contextual
openai com/ docs/modee pocs/models/// emdeddiggssdddddsgs....mm For a fair comparison with prior works [27], we use SPAEGPT with the openai. com htt///dcs/mooee /wwomo//gottnn...../
GPT 3.5 text-davinci-00S API pttttps:/plattooo
We configure SPAE to encode a 128 128 image into a token pyramid of 6 layers where each layer has 2* x 2 tokens and ào = [0, 2, 3, 4, 4). Additionally, we train a video-based SPAE model on the Kinetics-600 dataset [5], and further details can be found in the Appendix. We apply semantic guidance loss to the first five layers, with thresholds of 0.98, 0.95, 0.9, 0.85, and 0.8. The CLIP with ViT-L/14 [13] vision backbone is used. We use 80 prompt templates from the shot ImageNet
5

---

Table 1. Few-shot classification accuracy on the mini- ImageNet benchmarks. SPAEGPT and SPAEPALM are trained using different vocabularies and embedding sources, with different prompt templates for in-context learning. They show the broad compatibility of SPAE hut not for a comparison between the L.Mss. The hest performance with GPT is in italies while the overall hest in hold.
Task Induction
V
V
<
Method
# Layers
Inner Shots
1
1
3
5
1
Avg
# Tokens
Repeats
0
0
0
0
1
3
5
2- 2-Wyy Classification
Frozen [35]
1.7
33.7
66
66
63
65
63.7
51.3
LQAE [27]
1: 256
G.T
1.5
35.2
68.2
69.8
68.5
68.7
65.9
53.97
SOPT (ours)
G.5
5.3
77.2
84.4
86.0
79.4
77.2
77.1
69.57
SPAEPILM (ours)
2:
PaLM
32.2
84,0
88.3
88,4
85.1
83.6
82,4
77.74
SPAEPaLM (ours)
3: 21
PaLM 2
27.9
84.8
92.5
92.6
84.8
85.2
85.4
79.03
5- 5-Way Classification
Frozen [35]
0.9
14.5
34.7
33.8
33.8
33.3
32.8
26.26
LQAE [27]
256
3.5
1.0
15.7
35.9
36.5
31.9
36.4
45.9
29.04
SPAF (ours)
5
GPT
4.3
63.0
63.4
60.6
61.9
62.1
62.1
53.91
SPAEPaI (ours)
2:
PaLM 2
23.6
64.2
68.0
69.9
63.4
62.0
60.2
58.76
SPAEPar M (ours)
21
PaLM 2
20.2
65.1
73.7
74.3
66.4
67.0
66.3
61.86
classification setup to precompute the CLIP text embeddings for the vocabulary. In addition, we use the Adam [20] optimizer with loss weights = 1, 3 0.33, À 0.1, 17 = 0.1, 10-1 and learning rate of 10-4 following linear warmup/cooldown and root square decay schedule. Following the prior work [27], SPAE is trained on the ImageNet ILSVRC2012 [10] dataset. We train with batch size of 256 for 450k steps. Further details can be found in the Appendix.
4.2 Main Evaluation
Few-shot image classification. We evaluate the in-context image understanding capability with a frozen LLM on the mini-ImageNet [40] few-shot classification benchmark. A set of tokenized images and class labels are fed to the language model as context for classification of new image. Following [35, 27), we evaluate 14 settings controlled by four factors regarding the content of each test case: (1) task induction: whether including preamble specify the output space; (2) number of ways: the number of categories; (3) number of inner shots: the number of unique examples for each category; (4) number of repeats: the number of times that each unique example is repeated.
We compare SPAE with the state-of-the-ar methods Frozen [35] and LQAE [27]. As shown in Tab. 1, SPAEoFT consistently outperforms LQAE, both using the same GPT 3.5 model and in-context format, while using only 2% of its tokens. Fig. 3 shows the performance trend when using different number of SPAFpaLM lay- ers across six settings with task induction and 0 repeats. SPAEPaLM with 3 layers achieves the best performance which balances between suf- ficient semantics and an image sequence length that is optimal for LLM in-context learning. Overall, SPAEPaLM yields +25% and +32% average accuracy improvement over the state-of-the-an on the 2-way and 5-way benchmarks in Tab. 1.
100
2/way
75
2/way 3hcho
223
76.20
65,25
2oway
5-way 1-4hot
25
5-way 3-shot
5-way S-shot
1:1
2:5
3:21
4:
85
5: 541
Average
#Layers Tokons
Figure 3. Few-shot classification accuracy on mini- ImageNet using different SPAEPALM layers.
Reconstruction quality. We compare the image and video reconstruction quality using the tokens produced by SPAE and the VQGAN baseline used in state-of-the- image [7, 25, 6] and video generation [45]. We use FID [16], Inception Score (IS) [33], and LPIPS [48] compare with the image VQGAN from MaskGIT [7] on the ImageNet validation set, and FVD [36] to compare the 3D-VQGAN from MAGVIT [45] on the Kinetics-600 validation set. The results are presented in Tab. 2. While SPAE may have more lossy reconstruction compared to VQGAN when using a similar number of tokens, this compensated by going into deeper layers. At the bottom of Tab. 2, we showcase the scalability of our model by training on the ImageNet-21k dataset with 13M images and list the comparable variant from LDM [32] as a reference.
Token pyramid visualization. We visualize the tokens produced by SPAE in Fig. 4, where we show the raw pyramid or histogram of tokens with top frequencies for the first four layers, with reconstructed images from layer 5 and 6. We have the following findings.

---

Orig
Layer
Oiginal
Orginal
Loper
logerG
Lxper
Eagers 1-4
Layer
sglie
pridio
Layers
Figure 4. Examples of pyramid image tokenization and reconstruction by a 6-layer SPAE. We show the pyramid or histogram of most frequent tokens for the first four layers, and reconstructed images from layer and 6. In the pyramid, we use darker cells to show tokens with higher CLIP similarity to the original image. For non- Non-English sub-word tokens, we show automatic translation for reference in italic fonts below the original token. Circled tokens are mentioned in Section 4.2. See full pyramid visualizations in the Appendix.
Table 2. Comparison of reconstruction quality between SPAE and VQGAN baselines used in state-of-the-art
image [7, 25, 6] and video [45] generation models.
Image
Video
Resolution
Method
# Layers Tokens
(ImageNet 1LSVRC2012(10)
(Kinetios-600 (5]
FID4
ISt
LPIPS
FVD1
VQGAN
1: 256
5,48
119.69
0.13
6.79
128 128
5: 341
9.49
109.46
0.17
52.28
SPAE (ours)
6: 597
4.41
133.03
0.12
6.35
VQGAN
1: 256
4.04
163.95
0.21
SPAE (ours)
597
3.60
168.50
0.19
256x256
(OGGANNDMMSS Openimages)
17236
5.13
1444.5
SPAF (ours, ImageNes-21k)
6: 597
3.08
173.79
0.19
First, the SPAE tokens are organized in a pyramid structure, with every layer comprising semantically related tokens to the image. The few tokens in the top layers seem capture the primary theme of the image. For instance, in Fig. 4, the token presso (highlighted in orange) represents the espresso machine and other tokens like blender refer related regions. Layer 3 and Layer 4 reveal additional details about localized objects. For example, the token Thermo refers to the thermometer in the top-left region, while stove appears in the bottom-right area. In addition to nouns, related verbs also show up, including pouring, refill, spill, and brew.
Second, it is worth noting that the CLIP model has an English-only vocabulary. However, thanks to the multilingual vocabularies and embeddings from the LLM, SPAE's semantic guidance is able to map to similar concepts in other languages, such as koffie in Dutch and kaffe in Danish as corresponding terms to the concept of coffee.
Third, similar to RQ tokens [22]. SPAE tokens can reconstruct the image with progressively refined details when more layers, and thus tokens, are utilized. Fig. 4 shows Layer 5 begins to produce a reasonable reconstruction while Layer 6 further enhances the level of detail and smoothness.
Visual question answering. Tab. 3 provides quantitative results on the visual question answering (VQA) task. We compare with the baseline Frozen [35] method on the Real- Fast- VQA [35] benchmark for few-shot learning. As shown, SPAE consistently outperforms Frozen. Unlike Frozen, SPAE training does not require backpropagation through the LLM.
Table Few-shot VQA performance on Real-Fast- VQA.
Inner Shots
Frozen [35]
7.8
10.1
10.5
SPAERam (ours)
14.3
15.9
15.1
4.3 Qualitative Studies
This section explores the capability of a frozen PaLM 2, trained solely on language tokens, in performing multimodal tasks using in-context learning. We adopt a two-stage decoding process for image generation. In stage one, we use AR decoding to produce the first SPAE layers with task-specific conditions. Stage two is task-agnostic NAR decoding process for layer 6 conditioned on the first 5 layers.
Image to text and VQA. We examine two tasks involving visual-text reasoning (1) image caption- ing on COCO [26] captions; and (2) visual question answering (VQA) on COCO-QA [31], For both
7

---

Beseine
mountait
SPCC 12:
GPG 1::
Aman in red jockett rbiing down snoray mountains
standing foont of to will red jocket brank stred
SRAC group of people stancing
Dasalne:
SPRE Mither with stove, sinit, and andretiigeoto: rigeoooo SPAC L3: Mishen with a slove, simt, and refrgaaaoo
standing field
SPAE L3,
SPAE 14,
SPA2
pcrsonn skäng dien a snowy mounsaar.
parssnn sking
Aman sk Fng down showe mountaalss
skilig
mounters
mounsain
kishhn stone, airt, cabrness.
Mither with store,
2o0n, figorotoo
retoproten
Busolinc man stand
rack the middle of rives SRAF 11: A mon in iontanding on mork the th dotte of shoee SPAZ mon in soooriiigg coes and nat.
Baseine man and womann are sitting on a banch part. DPA2 Lt: man is holding baby in hhs emms.
group of people etancing in lite
SPAE man lo hokiing small dog.
grop of people instttmn Halloweon party
SPAE beddx bear arting on bas.
group people
tedis
on
dressed
sable
of flowers
SINE;
SINE: standingg neat to woman in wooding cres.
train station.
whter roking in respty poo Baccl nn Like SPM comeboard
wive deor wallin Shroag col gress Hisuliie sherion
kew dffiverr ser-eeddeinn gleeee Wlsses ouiderr modle
peppeeent cheers of white plate.
ot
sneeec sevecoee direlged Alm wae deage
SPAD grinly
Figure 5. Qualitative samples of image-to-fett generation: image captioning and VQA. We compare between different layers of SPAE (I.1-16) and a baseline model without semantic guidance pyramid SAQ.
Context
an image of {
Query
Generation
0
/
2
3
4
5
6
7
8
9
an image of 147
8
0
/
2
3
4
5
6
7
&
2
an image of the last
digit 5+7
5
0
1
2
3
4
5
6
7
8
9
an image of the aquare root of 4
LLM
2
0
1 2
3
4
6
7
g
9
1
2
3
4
5
6
7
8
9
an image of the number of continents in the world
Figure 6. Examples of text-to-image generation on MNIST using SPAE with frozen PaLM 2 model. We use SPAE to tokenize 50 handwritten images as the context and ask PaLM 2, an LLM trained solely on text tokens, to answer complex queries that require generating digit images through SPAE as the output.
tasks, provide 10 unique training examples prompts. In the case of VQA, 10 different answers are presented to form a 10-way 1-shot setup.
We compare SPAE to baseline model trained with the same frozen language codebook but without the proposed semantic guidance or pyramid SAQ. As shown in Fig. 5, when fed with baseline tokens, the LLM randomly hallucinates a caption or guesses an answer simply based on the question. Similar hallucination can happen if we only use the first two layers of SPAE or five words to represent an image, it provides insufficient context for captioning. Reasonable captions start to appear with 4 layers or 85 words, while complex scenes may still need the full 6 layers of 597 words.
LLM generating MNIST images. Fig. 6 shows few image generation examples on MNIST [11]. The frozen LLM learns about handwritten digit images through 50 context samples tokenized by SPAE trained on MNIST. Each sample consists of preamble "an image of k" and the lexical tokens representing an image of digit k. Then we can ask the LLM to answer questions with digit images. Specifically, with query of "an image of 1+7", we can use progressive AR decoding with the LLM to produce a token sequence that can be decoded into an image of 8 by SPAE. We test with complex questions requiring mathematical reasoning or common sense knowledge, and the LLM is able to respond correctly. In addition, the generated digit images appear different from all context samples. This demonstrates the cross-modal reasoning capability enabled by SPAE and frozen LLM, with images generated over the text-only interface.
Conditional image denoising. To the best of our knowledge, there have been successful at- tempts that demonstrate generic image generation capability using a frozen L.M.. To this end, we define a simpler denoising setup explore the capability of LLMs. Fig. 7 demonstrates the condi- tional image denoising tasks, e.g., image outpainting, deblur, inpainting, location translation, rotation, etc. Note that, in order to generate images for each task, we utilize 10 pairs of noisy examples with corruption rates ranging from 50% to 20%, as discussed in Section 3.2. The full context, which is omitted in Fig. can be found in the Appendix.
8

---

Stride
256
64
32
2
Stage 1: PAR Outpainting m>Layer
Stage PNAR
Task-agnostic
=>Layer
Figure 7. Examples of conditional image denoising. We compare different decoding strides for both stages. Yellow and blue boxes indicate the selected results. The I.I.M provided with pairs of noisy examples like Fig. which are deferred to the Appendix.
Original
1180
Pestrul
1200
bag
Par/
pel
GB
bires
Palcem
100A
188u
4611
Praliv
Eent
Nestor Peetee Trolor
ardebnok
guidance
semanii goidance
semantic guidance
2-4ayer
2ayye
6-bayer pyramrd SAQ.
Figure 8. Ablation examples with reconstructed image and semantic tokens for models listed in Tab. For non-pyramid tokens, show a 4x4 crop from the first layer corresponding to the region indicated by the black box. For pyramid tokens, we the third layer which consists of 4 x tokens.
The top rows of Fig. 7 compare the generation from differ- ent decoding strides with the same set context examples. Single-step decoding with infinity stride fails to produce reasonable image, which validates the proposed progressive generation approach.
Input
VOGAA
LOAE
SPAE
we qualitatively compare SPAE with baseline meth- ods VQGAN and LQAE using the same in-context denoising procedure. As shown, VQGAN fails to produce reasonable images, in part because many words in the LLM output are of its vocabulary. LQAE only produces vague object con- tours but cannot recover any visual details. On the contrary, SPAE can generate reasonable images.
In Fig.
Conditional video denoising and other tasks. Due space constraints, we show the examples in the Appendix.
Figure 9, Comparison on conditional image denoising with different tokeniz- ers. All models use the same decoding setup with the same ten pairs of prompt images available in the Appendix.
4.4 Ablation Studies
The results in Tab. 4 and Fig. verify the effectiveness of the proposed designs in SPAE, evaluated by reconstruction quality (FID, IS, LPIPS) and semantic relevance (CI.PP score, few-shot classification accuracy). We have the following findings. First, simply using frozen codebook negatively affects the reconstruction results, but with semantic guidance it performs comparably with the original VQGAN while producing meaningful lexical words. Second, RQ hurts reconstruction quality with frozen codebook. This is different from RQ's standard setup [22] where the codebook is learned. Third, SAQ improves both quality and semantic similarity, where the pyramid enables representation with much fewer tokens. This allows for accommodating more examples within fixed and constrained in-context length. Finally, per-layer semantie thresholds benefit understanding and the dynamic semantic loss weight helps reconstruction. The perceptual loss leverages trained network with access classification labels, but removing results surprising improvement in classification accuracy while greatly hurting the reconstruction.

---

Table 4, Ablation studies codebook, training objective, quantization method, and token structure. Classifica-
tion accuracy evaluated under the mini-lmageNNe 5-way I-shot setup.
# Layers
Classification Accuracyt
Method
FID
IST
LPIPS!
CLIPT
# Tokens
Baseline VQ
1: 256
5.48
119.69
0.13
n/a
19.6
+ frozen codebook
I: 256
7.44
101.39
0.17
0,1464
19.5
semantic loss
I: 256
5.17
124.41
0.13
0.1518
46.2
T: 256
11.94
89.01
0.22
0.1395
56.2
2-layer RQ [22]
512
6.05
113.93
0.15
0.1547
T: 2.56
12.30
93.33
0.2T
0.1613
$6.6
2-layer SAQ
2: 512
5.08
125.27
0.14
0.1595
1:
0.1879
52.0
2:
0.1868
64.2
6-layer pyramid SAQ
0.1815
65.1
(SPAE)
4: 85
0,1711
58.5
5: 341
9.49
109.46
0.17
0.1604
46.3
597
4.41
133.03
0.12
0.1577
no per-layer thresholds
6: 397
4.33
122.25
0.11
0.1650
59.4 (layer 3)
no dynamic semantic weight
6: 597
9.00
85.14
0.19
0.1847
65.1 (layer
no perceptual loss
597
40.47
33.41
0.20
0.1994
69.5 (layer 3)
5
Conclusion
Our work unveils the untapped potential of frozen Large Language Models (LLMs) tackling multimodal understanding and generation tasks involving images and videos, without requiring explicit training on these modalities. This is achieved by a new method, SPAE, which converts between visual content and lexical tokens of variable length, imbued with rich semantic meaning. Our findings show the great potential of harnessing the vast knowledge and reasoning capabilities of LLMs in the field of computer vision, transcending the limitations of language-only tasks.
Limitations. More tokens are required to achieve the same level of reconstruction when using the frozen language codebook, compared to the existing VQGAN models with learned codebooks. The capability of in-context learning is significantly constrained by the acceptable sequence length. Although our results suggest the plausibility of image generation, the resolution, quality, and diversity far from the recent text-to-image models trained on large image and text data.
Broader impact. Our paper showcases the untapped potential of frozen LLMs in multimodal understanding and generation tasks involving images and videos, without requiring explicit training on these modalities. As initial research proof-of-concept, limitations in learning context and constrained capabilities. Consequently, there is still a substantial gap the recent specialized models for text-to-image (e.g.. Stable Diffusion) or image-to-text that have been specifically trained using billions of text-image pairs.
focus in-context learning, which has
The potential impact of research lies in its influence future studies, specifically in the area of integrating vision modalities into the LLMs. For instance, our work can be extended to explore finetuning adapter tuning of LLMs large-scale text-image datasets. Future research in these directions may implicate ethical issues around fairness and transparency. We have found that the generated tokens occasionally include slang terms or words that create inappropriate connotations related to the subject depicted in the image or video. Such concerns must be thoroughly considered and effectively addressed prior to deploying this method in real-world applications.
A Asnnowledgments and disclosure of funding. The authors would like to thank anonymous re- viewers and area chairs their insightful comments, and Siamak Shakeri, Sergey loffe. Jay Yagnik, and Boqing Gong for their valuable feedback and constructive discussions. This project is funded in part by Carnegie Mellon University's Mobility21 National University Transportation Center, which is sponsored by the US Department of Transportation.
10

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc. Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch. Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurlPS, 2022. 3
[2] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. PaLM 2 technical report. arXiv:2305. 10403, 2023.
3,5
[3] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. AudioLM: a language modeling approach to audio generation. JEEE/ACM Trunsactions on Audio, Speech, and Language Processing, 2023. 2
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020. 1, 3,5
[5] Joao Carreira, Eric Noland, Andras Banki- Horvath, Chloc Hillier, and Andrew Zisserman. A short note about Kinetics-600. arXiv:/808.01 2018. 7
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Murphy, William Freeman. Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. In JCML, 2023. 6,7
[7] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: Masked generative image transformer. In CVPR, 2022. 2, 6.7
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma. Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv:2204.02311, 2022. 1,3
[9] Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv:2210. 13438, 2022.
[10] Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hierarchical image database. In CVPR, 2009. 6.
[11] Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, (9(6): 142, 2012. 5, 8
[12] Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pro-training of deep
bidirectional transformers for language understanding. In NAACL, 2019. 3
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. In ICLR, 2020. 5
[14] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high resouuttoootioo image synthesis. In CVPR, 2021. 2
[15] Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang. David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time- annestgoottcc VQGAN and sime-sensitive transformer. In ECCV, 2022. 2.4
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rulc converge to local nash cquilibrium. In NeurlPS, 2017.
[17] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzehski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-effiiient transfer learning for nip. ICLR, 2019. 5
[18] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2021. 5
[19] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016. 4

---

[20] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv::142..6980 2014.
[21] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multi- modal generation. In JCML, 2023. 1, 3.
[22] Doyup Lec, Chihcon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image gencration using residual quantization. In CVPR, 2022. 3, 4, 7, 9, 10
[23] Brian Lester, Rami AI- Afo,, and Noah Constant. The power of scale for parameter-efficien prompt tuning. In EMNLP, 2021. 5
[24] Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramascsh, Ambrose Slone. Cem Anil. Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. In NeurlPS, 2022.
[25] Jose Lezama, Tim Salimans, Lu Jiang. Huiwen Chang. Jonathan Ho, and Irian Essa. Discrete predictor- corrector diffusion models for image synthesis. In ICLR, 2023. 6,
[26] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, 2014. 7
[27] Hao Liu, Wilson Yan, and Pieter Abbeel. Language quantized autoencoders: Towards unsupervised text-image alignment. arXiv:2302.00900 2023. 1, 2, 5,
[28] OpenAI. GPT-4 technical report. ar arXiv:2393308877, 2023. 1,2
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML., 2021.
[30] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-lo-image generation. ICML, 2021. 2
[31] Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering.
In NeurlPS, 2015. 7
[32] Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Björ Ommer. High-rcsolution image synthesis with latent diffusion models. In CVPR, 2022. 2, 6,7
[33] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen, Improved techniques for training gans. In NewrIPS, 2016. 6
[34] Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. In CVPR, 2021.
[35] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. NeurlPS, 2021. 6, 7
[36] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier. Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. 181.07781 2018. 6
[37] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS, 2017. 2,
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. 1
[39] Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Moham- mad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arX.v:2210.02.399 2022. 2
[40] Oriol Vinyals. Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. In NewrIPS, 2016. 6
[41] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. SuperGLUE: A stickier benchmark for ceneral-purposs language understanding systems. In NewrIPS. 2019. 3
12

---

[42] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. TMLR, 2022.
[43] Chenfei Wu, Shengming Yin. Weizhen Qi, Xiaodong Wang, Zecheng Tang. and Nan Duan. Visual ChatGPT: Talking, drawing and editing with visual foundation models. arXiv:2303.0467/ 2023.
[44] Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, al. Natural language code generation interactive data science notebooks. arXiv:2212.09248, 2022. 3
[45] Lijun Yu, Yong Cheng. Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann. Ming-H Hsuan Yang, Yuan Hao, Irfan Essa, et al. MAGVIT: Masked generative video transformer. In CVPR, 2023. 2, 4, 7,
[46] Lijun Yu. José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diflusion-tokenizer tokenizer is key to visual generation. orXiv:2310.0573: 2023. 2
[47] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Trans. on Audio, Speech, and Language Processing, 30:495 507, 2021. 3,
[48] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metrie. In CVPR, 2018. 6
[49] Susan Zhang. Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, OPT: Open pre-trained transformer language models. arXiv:2205.0 2022,
13


Reviews:
[{"review_id": "KiSkZkVt1k", "reviewer": "Reviewer_ekNF", "summary": "The paper introduces SPAE, a method that aligns visual representation with a fixed LLM representation. SPAE effectively captures semantics and visual fine-grained textures for a range of cross-modality tasks. The paper showcases good and solid results in few-shot learning and reconstruction/generation tasks.\n\n\n\n\n", "questions": "- Is it possible to perform zeroshot tasks with the visual encoder and LLM, it would be a good example to compare SPAE with other contrastive based models.\n- How critical is the LLM used in this paper, can we switch LLM with other language models and how will that impact the performance?\n", "limitations": "None", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and well motivated.\n- The proposed idea is interesting, cross-modality with codebook is not a novel idea, but using the fixed pre-trained LLM as dictionary is an interesting idea and seems working well.\n- The results are solid and convincing.", "weaknesses": "- The quantitive comparisons are not fair comparisons; as the language model sued in SPAE is much bigger and trained on much more data than the baselines. One interesting baseline is to replace the heavy LLM to smaller LMs and discuss the impact.\n- Lack of comparisons to other works on generation tasks. It would be good to include comparison to other generative models with same prompt.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer ekNF for the detailed feedback. Please see our response below.\n\n\n> W1: fairness of the quantitative comparisons due to the LM size.\n\nWe kindly note that this seems to be a misunderstanding. In Table 1, both SPAE$\\_\\text{GPT}$ and the baseline LQAE [27] are based on **the same GPT 3.5 model and in-context format** (as mentioned at L217-218 in the manuscript). Our approach outperforms the baseline LQAE by an improvement of over 15\\% when utilizing the identical language model.\n\nNevertheless, to eliminate the LLM dependency during SPAE training, we also train a SPAE$\\_\\text{RoBERTa}$ model, which uses precisely the same embedding as in LQAE. Under the same in-context evaluation protocol with GPT 3.5 for miniImageNet 5-way 1-shot classification, SPAE$\\_\\text{RoBERTa}$ achieves 36.5\\% accuracy, which is significantly better than LQAE with 15.7\\%. This result further validates the effectiveness of the SPAE over the baseline LQAE.\n \n\n> W2: comparisons on generation tasks.\n\n“Our work marks the first successful attempt to enable a frozen LLM to generate image content” (L10-11) through in-context learning, a point acknowledged by other reviewers. Therefore, it is challenging to directly compare to other methods under the same setting. To overcome this challenge, we compare SPAE to the baselines (VQGAN [12], LQAE) using the same prompt images and progressive in-context denoising (proposed in Sec. 3.2).\n\nThe comparison is shown in Fig. 18 in [the rebuttal pdf](https://openreview.net/attachment?id=sIoDsBBe1G&name=pdf). As shown, VQGAN fails to produce reasonable images, mainly because many words in the LLM output are out of its vocabulary. LQAE only produces vague object contours but cannot recover any visual details. On the contrary, SPAE can generate better images.\n\n\n> Q1: Is it possible to perform zero-shot tasks with the visual encoder and LLM?\n\nThe LLMs used in our paper are trained on text-only data. This differs from the zero-shot cross-modal methods, such as contrastive-based models, that are trained on large-scale image-text data. Consequently, the employed LLMs do not understand images, making it non-trivial to perform zero-shot cross-modal tasks. Our paper shows this can be compensated by providing a few image examples via in-context learning.\n\n\n> Q2: LLM flexibility.\n\n“Our method has been tested with PaLM 2 and GPT-3.5, suggesting compatibility with arbitrary LLMs.” (L46-47) Per L186-194, each language model uses different tokenizers with different vocabularies, and we extract word embeddings for SPAE from different layers of the model. These setup variations further suggest the compatibility of our approach. "}, {"review_id": "E6FMV2Fe79", "reviewer": "Reviewer_mRhs", "summary": "This paper presents a novel multi-modal generation method named SPAE which leverages semantic autoencoder and in-context denoising for semantic reconstruction. The SPAE converts raw pixels into interpretable lexical tokens from the LLM's vocabulary, capturing both semantic and fine-grained visual details. The paper conducts in-context learning experiments using frozen PaLM-2 and GPT 3.5 on a diverse set of image understanding and generation tasks for showing the effectiveness.", "questions": "See weakness", "limitations": "The paper demonstrates the generation under the low-resolution setting. The higher resolution (e.g. 1024x1024, 2048x2048) remains unexplored. Can you provide some comparisons with diffusion, such as high resolution and complex scenes?", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "● Semantic Pyramid AutoEncoder (SPAE) is introduced for enabling frozen LLM to perform both understanding and generation tasks involving non-linguistic modalities, such as images or videos.\n● Different from previous work, SPAE produces interpretable representations of semantic concepts and fine-grained details in the form of multilingual linguistic tokens by leveraging frozen LLM.\n● The proposed method outperforms the best-published few-shot image classification accuracy by an absolute 25% under the same in-context setting.", "weaknesses": "1. The 5-shot evaluation on ImageNet classification is somewhat unfair. It is not clear about the performance with GPT-3.5 under this setting. The paper only reports the results of PaLM-2.\n2. The hyperparameters in the paper are quite tricky (e.g. threshold for each layer, the weight of the loss).  Besides, the effect of dynamic weight used in the final objective is not ablated. \n3. Since SPAE is only trained on ImageNet, it is not unknown whether this method scales well on large-scale datasets (e.g. Laion-400M, Laion-2B, COYO-700M, etc).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the constructive comments from reviewer mRhs and the assessment that our method “produces interpretable representations” and “outperforms the best-published few-shot image classification.”\nWe respond to each question below.\n\n> W1: Missing 5-way miniImageNet with GPT 3.5?\n\nFollowing the reviewer’s suggestion, we add the 5-way classification on the mini-ImageNet benchmark in Tab. R2.1, which compares SPAE$\\_\\text{GPT}$ and LQAE [27] with “the same GPT 3.5 model and in-context format” (L217-218). The results are consistent with the observation in Table 1 in the paper that SPAE$\\_\\text{GPT}$ significantly outperforms LQAE.\n\n**Table R2.1.  5-way classification accuracy on the mini-ImageNet benchmark with GPT 3.5. The results are supplementary to Tab. 2 in the paper.**\n\n|                       |         |          |          |          |          |          |          |           |\n|-----------------------|---------|----------|----------|----------|----------|----------|----------|-----------|\n| Task Induction        |         | yes      | yes      | yes      | yes      | yes      | yes      | Avg       |\n| Inner Shots           | 1       | 1        | 3        | 5        | 1        | 1        | 1        |           |\n| Repeats               | 0       | 0        | 0        | 0        | 1        | 3        | 5        |           |\n| LQAE [27]                | 1       | 15.7     | 35.9     | 36.5     | 31.9     | 36.4     | 45.9     | 29.04     |\n| **SPAE$_\\text{GPT}$** | **4.3** | **63.0** | **63.4** | **60.6** | **61.9** | **62.1** | **62.1** | **53.91** |\n\n\n> W2: hyperparameters (e.g., threshold for each layer, the weight of the loss) and ablation on the effect of dynamic weight.\n\nIt’s worth highlighting that most weight hyperparameters in the training loss Eq. (8) closely follow the hyperparameters in the previous works [44, 12]. Following the reviewer’s suggestion, we discuss the two new hyperparameters presented in this paper: the layer-wise threshold and the dynamic weight used in Eq. (9).\n\nRegarding the layer-wise threshold, we train SPAE using a uniform threshold for every layer. As shown in Tab. R2.2, this variant attains comparable reconstruction quality but yields worse understanding performance. The result indicates that layer-wise thresholding might not be absolutely necessary, but it appears beneficial for understanding tasks.\n\nTo analyze the impact of the dynamic weight, we train an SPAE variant using a constant weight of 1 instead. As shown in Tab. R2.2, removing the dynamic weight results in an imbalance between semantics and appearance, which leads to much worse reconstruction quality. The results substantiate the contribution of dynamic weight for the generation task.\n\nWe will clarify these points in the revised version.\n\n**Table R2.2 Ablation experiments on training hyperparameters. We evaluate reconstruction quality on ImageNet, along with miniImageNet 5-way 1-shot classification accuracy with PaLM 2.**\n\n|                               | Reconstruction |        |        | Understanding |\n|-------------------------------|----------------|--------|--------|---------------|\n| Method                        | FID↓           | IS↑    | LPIPS↓ |  5-way 1-shot Accuracy↑ |\n| SPAE  (Baseline)                        | 4.41           | 133.03 | 0.12   | 65.1      |\n| &nbsp; &nbsp; uniform threshold  | 4.33           | 122.25 | 0.11      | *59.4*      |\n| &nbsp; &nbsp; constant weight              | *9.00*           | *85.14*  | *0.19*    | 65.1      |\n\n\n> W3: scalability to larger datasets.\n\nTo test its scalability, we train SPAE at 256x256 resolution on a much larger dataset ImageNet-21k which consists of 13M images and is 10x larger than ImageNet. We selected ImageNet-21k because we could not acquire copyright permission to train on the datasets recommended by the reviewer.\n\nThe results in Tab. R2.3 show the model performance on the standard ImageNet validation set. These results provide empirical evidence of the SPAE model’s scalability across a large dataset of 13 million images from ImageNet21k.\nIn Tab. R2.3, the performance in the Latent Diffusion Model (LDM) paper [32] is also listed for reference. Note that direct comparison to LDM might not be appropriate due to using a distinct dataset (OpenImages).\n\nIn addition, we train SPAE on Kinetics-600, which consists of 384k videos with around 96M frames. The results are in Tab. 3 in the paper and Fig. 16 in the supplementary material.\n\n**Table R2.3 Comparison of reconstruction quality and semantic relevance on ImageNet-val.**\n\n| Method   | Resolution | Dataset      | FID↓ | IS↑    | CLIP↑  |\n|----------|------------|--------------|------|--------|--------|\n| SPAE     | 128x128    | ImageNet     | 4.41 | 133.03   | 0.1577 |\n| SPAE     | 256x256    | ImageNet-21k | 3.08 | 173.79   | 0.1637 |\n|          |            |              |      |        |        | \n| LDM [32] | 256x256    | OpenImages   | 5.15 | 144.55   | n/a    |\n\n\n> L1: higher resolution such as 1024x1024.\n\nAs we acknowledged in L312-313, “the capability of in-context learning is significantly constrained by the acceptable sequence length” of the LLM. In practice, the largest resolution we tried is 256x256, as shown in Fig. 17 of [the rebuttal pdf](https://openreview.net/attachment?id=sIoDsBBe1G&name=pdf).\n\nIt is worth highlighting that our method is just “the first successful attempt to enable a frozen LLM to generate image content” (L10-11). This point has also been acknowledged by Reviewer 1ht3. Scaling to higher resolutions or comparing with diffusion models trained on extensive image-text datasets falls beyond the paper’s scope, necessitating further research. For example, future studies could explore using SPAE to jointly finetune LLMs on larger text-image training datasets to generate high-resolution images and complex scenes. We will clarify this point in the revised version."}, {"review_id": "r01imZc56U", "reviewer": "Reviewer_2BCt", "summary": "This paper proposes Semantic Pyramid AutoEncoder (SPAE) which enables frozen LLMs to perform both understanding and generation tasks involving non-linguistic modalities such as images or videos through in-context learning. The authors first introduce SPAE, a pyramid tokenizer that produces interpretable representations of semantic concepts and fine-grained details. In addition, the authors also propose a new progressive prompting method that facilitates the in-context generation of long cross-modal sequences. The proposed method outperforms SoTA in few-shot image classification accuracy by a large margin.", "questions": "please refer to the weakness part. ", "limitations": "The authors have described the limitations of this work. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed approach does not require further tuning of LLM. It is thus very portable.\n- The paper is well-written and the approach, motivations, and experiments are well-derived and clear.\n- The paper presents an interesting algorithm to leverage LLMs to address non-linguistic tasks, such as how to combine pretrained models and how well it would perform, even without tuning LLM and its codebook.\n- Impressive few-shot results on image classification.\n", "weaknesses": "Although this paper has shown how a frozen LLM can generate image content or realize visual understanding tasks. The tasks the authors choose to evaluate in this paper are still limited (only few-shot image classification on ImageNet). \nCan the authors provide more quantitative experiments to validate the SPAE’s generalization to various tasks (e.g., caption, VQA…)? Besides, there are only qualitative examples of image-to-image generation, which only prove that SPAE “can” generate image content, not that it can generate well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 2BCt for the thoughtful comments. Please see our response below.\n\n> W: quantitative results on VQA.\n\nFollowing the reviewer’s suggestion, we provide quantitative results on the VQA task in Tab. R3.1.\nWe compare with the baseline Frozen [35] method on the Real-Fast-VQA benchmark for few-shot learning. As shown, SPAE consistently outperforms Frozen. It is worth mentioning that, unlike Frozen, SPAE training does not require backpropagation through the language model.\n\n**Table R3.1 Few-shot VQA performance on Real-Fast-VQA.**\n\n| Inner Shots   | 1    | 3    | 5    |\n|---------------|------|------|------|\n| Frozen [35]        | 7.8  | 10.1 | 10.5 |\n| SPAE | 14.3 | 15.9 | 15.1 |\n\n\n> W: generation capability.\n\nOur contribution is a proof-of-concept that, with SPAE, an LLM trained on text tokens “can” indeed generate image content. This marks the first successful attempt to enable a frozen LLM to generate image content through in-context learning. To the best of our knowledge, there have been no standard evaluation setups for such in-context image generation. In Fig. 18 in [the rebuttal pdf](https://openreview.net/attachment?id=sIoDsBBe1G&name=pdf), we qualitatively compare in-context denoising generation results with our baseline methods VQGAN and LQAE. The results validate that SPAE is the only method capable of generating reasonable image content through in-context learning. We will clarify this point in the revised version."}, {"review_id": "4rvGPwDIWF", "reviewer": "Reviewer_1ht3", "summary": "This paper introduces the Semantic Pyramid AutoEncoder (SPAE) for enabling frozen Large Language Models (LLMs) to perform understanding and generation tasks involving non-linguistic modalities such as images or videos. SPAE converts raw pixels to lexical tokens extracted from the LLM's vocabulary, allowing the LLM to perform multimodal tasks. The proposed method is evaluated using in-context learning experiments with frozen PaLM 2 and GPT-3.5, demonstrating success in image understanding and generation tasks. The main contributions include the first successful method using a frozen language model for image content generation, a new SPAE tokenizer producing interpretable representations, a new progressive prompting method for long cross-modal sequences, and evaluation on visual understanding and generation tasks.", "questions": "Figure 8 in the appendix is intriguing, but the experiments are only conducted on the MNIST dataset. It would be compelling to see the experimental results on datasets like MS-COCO.", "limitations": "addressed", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. Compared to LQAE, the method presented in this paper demonstrates superior performance in image reconstruction.\n\n2. By interacting with large language models such as GPT-3.5 and PaLM-2, this paper effectively utilizes the in-context learning capabilities of these models to generate corresponding images, offering an intriguing research direction.", "weaknesses": "1. This paper's experiments are limited to 128x128 resolution, significantly lower than the mainstream 256x256 clarity.\n\n2. To achieve comparable performance with VQGAN, a larger latent space (# tokens) is necessary.\n\n3. Is it possible to attain the same performance as VQGAN + frozen codebook + semantic guidance and an expanded latent space (# tokens)? The need for a pyramid structure comes into play here. Whether the pyramid structure is necessary requires experimental support.\n\n4. The VAE proposed in this paper employs perceptual loss and utilizes a VGG network with supervised pre-training on ImageNet during the training process. In contrast, other methods like LQAE do not use perceptual loss. A strategy similar to ViT-VQGAN should be adopted here, using a version without perceptual loss for training on the comprehension task.\n\n5. There is a lack of model variants in the ablation experiments on the understanding tasks.\n\n6. In lines 203-204, it is mentioned that \"We train with a batch size of 256 for 450k steps, which takes 1.4k TPUv3-hours.\" This training cost is larger than that of both VQGAN and RQ-VAE. It would be interesting to investigate the impact of the training cost on the performance of the VAE proposed in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the detailed comments from reviewer 1ht3 and the acknowledgment of our contributions as “the first successful method using a frozen language model for image content generation,” where our method “demonstrates superior performance in image reconstruction” and offers “an intriguing research direction.”\nWe respond to each question below.\n\n> W1: 256x256 resolution.\n\nWe train an SPAE model to tokenize 256x256 images with a downsampling factor of 16, which yields the same number of tokens as 128x128. As shown in Table R4.1, SPAE outperforms VQGAN in reconstruction FID. More importantly, unlike the VQGAN tokens, SPAE tokens carry semantic information, as measured by the CLIP score, and have been demonstrated to benefit understanding tasks (see Table 1 in the paper).\n\nIn Fig. 17 in [the rebuttal pdf](https://openreview.net/attachment?id=sIoDsBBe1G&name=pdf), we visualize the in-context conditional image generation at 256x256. \n\n**Table R4.1 Comparison of reconstruction quality and semantic relevance at different resolutions.**\n\n|Resolution|Method|FID↓|IS↑|LPIPS↓|CLIP↑|\n|-|-|--|-|-|-|\n|128x128|VQGAN|5.48|119.69|0.13|n/a|\n||SPAE|4.41|133.03|0.12|0.1577|\n|256x256|VQGAN|4.04|163.95|0.21|n/a|\n||SPAE|3.60|168.50|0.19|0.1637|\n\n\n> W2: SPAE needs larger latent space than VQGAN.\n\nAs recognized by reviewer ekNF and mRhs, SPAE tokens capture “both semantic and fine-grained visual details.” Compared to VQGAN, which only captures visual details, more bandwidth is expected to store the additional semantic information while achieving on-par or better reconstruction quality.\n\n\n> W3: is it possible to attain the same performance as VQGAN + frozen codebook + semantic guidance? The necessity of pyramid structure?\n\nComparing the third model (VQGAN + frozen codebook + semantic guidance) and SPAE (6-layer pyramid SAQ), Tab. 4 and Tab. R4.2 show that SPAE performs better in both understanding (accuracy 65.1 vs. 46.2) and reconstruction (FID 4.41 vs. 5.17). In particular, SPAE uses significantly fewer tokens to represent an image (using 21 vs. 256 tokens) for the understanding task. The high token efficiency results from the pyramid structure because a conventional layer without pyramid structures needs a minimum of 256 tokens to represent the image. The pyramid design facilitates the representation of images using much fewer tokens, a vital factor for in-context learning as it enables the accommodation of more examples within the provided context. We will clarify this point in the revised version.\n\nIn Fig. 19 in [the rebuttal pdf](https://openreview.net/attachment?id=sIoDsBBe1G&name=pdf), SPAE shows better reconstruction quality and more relevant tokens than the third model.\n\n**Table R4.2 Ablation results on miniImageNet classification with PaLM 2, supplementary to Tab. 4.**\n\n|Method|#Layers:#Tokens|CLIP↑|5-way 1-shot Accuracy↑|\n|-|-|-|-|\n|VQGAN|1:256|n/a|19.6|\n|+frozen codebook|1:256|0.1464|19.5|\n|++semantic guidance (the third model)|1:256|0.1518|46.2|\n|+++2-layer RQ|1:256|0.1595|56.2|\n|+++2-layer SAQ|1:256|0.1613|56.6|\n|+++6-layer pyramid SAQ|3:21|0.1815|**65.1**|\n\n\n> W4: removing perceptual loss for understanding tasks.\n\nAs suggested, we train an SPAE variant without perceptual loss. As shown in Tab. R4.3, removing the perceptual loss greatly hurts the reconstruction quality as expected but results in even better classification accuracy. We hypothesize that this could be attributed to the fact that, compared to the standard model, this variant focuses much less on appearance and allocates more learning capacity for semantics, which leads to better understanding performance. Nevertheless, the results in Table R4.3 show that SPAE consistently outperforms the baseline LQAE when no perceptual loss is used.\n\n**Table R4.3 Ablation study on the perceptual loss with ImageNet reconstruction and mini-ImageNet classification (3 token layers, PaLM 2).**\n\n||FID↓|IS↑|2-way Avg Accuracy↑|5-way Avg Accuracy↑|\n|-|-|-|-|-|\n|LQAE|||53.97|29.04|\n|SPAE|**4.41**|**133.03**|79.03|61.86|\n|SPAE w/o perceptual loss|39.83|33.23|**79.81**|**64.89**|\n\n\n> W5: performance of ablated model variants on understanding tasks.\n\nAs suggested, we evaluate all model variants in the ablation experiments (Tab. 4) on the miniImageNet 5-way 1-shot image classification benchmark. These results are shown in Tab. R4.2. We will add these results in the revised version.\n\n\n> W6: performance with regard to training cost.\n\nWe would like to mention that all models used in the ablation study in Tab. 4, including VQGAN and RQ-VAE variants, are trained using the same setup for fair comparisons. \nIn Fig. 20 in [the rebuttal pdf](https://openreview.net/attachment?id=sIoDsBBe1G&name=pdf), we show the training curves of FID, IS, LPIPS, and CLIP score of SPAE and VQGAN.\nAs shown, within 40% of the training steps, SPAE shows better FID than the final VQGAN checkpoint.\nThe CLIP score keeps improving as the training proceeds, while the LPIPS saturates quite early.\n\n\n > Q1: Figure 8 in the appendix is intriguing; what about the experimental results on datasets like MS-COCO?\n\nThanks for the encouraging comments regarding Fig 8. As we acknowledged in L312-313, “the capability of in-context learning is significantly constrained by the acceptable sequence length” of the LLM. Dealing with the complex scenes on MS-COCO falls beyond the paper’s scope of “in-context learning” because it needs the finetuning of LLMs on a larger dataset. We hope our paper could facilitate future studies to explore using SPAE to jointly finetune LLMs on larger text-image training datasets to generate images with enhanced fidelity."}]

Author rebuttal:
Dear reviewers, ACs, and SACs,

Thank you for your time in handling this submission. We thank all reviewers for their thoughtful comments. We appreciate the reviewers’ acknowledgment that our idea/algorithm is perceived as “interesting” (ekNF, 2BCt). Our method SPAE is “the first successful method using a frozen language model for image content generation” (1ht3), produces “interpretable representations” (mRhs, 2BCt, 1ht3), captures “both semantic and fine-grained visual details” (mRhs, eKnf), and is “very portable” (2BCt). Regarding the result, the reviewer's comments highlight that our method “outperforms SoTA in few-shot image classification accuracy by a large margin” (2BCt), “outperforms the best-published few-shot image classification accuracy by an absolute 25%” (mRhs), “demonstrates superior performance in image reconstruction” (1ht3), and is “solid and convincing” (ekNF). 

We provide detailed responses below each review, with additional evidence and explanations. The PDF file for the figures is attached to this general response.

Meta-review: The reviewers find the proposed approach interesting, especially the idea of using a pre-trained LLM’s vocabulary as the codebook making the model interpretable. The paper presents the first approach to use Frozen LLM for image generation. The reviewers are intrigued by this research direction. They also appreciate the computationally efficiency of the proposed approach since it does not require fine-tuning of LLMs. The reviewers are impressed by the experimental results. They found the paper to be well written and well motivated.

The reviewers had raised some concerns, but the rebuttal successfully addressed most of them and all reviewers recommend acceptance. The authors are encouraged to improve the final paper version by following reviewer recommendations.
Decision: Accept (spotlight)