Title: SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs
Venue: NeurIPS 2023
Accepted: yes
Keywords: multimodal

generation

large language model
Abstract: In this work, we introduce Semantic Pyramid AutoEncoder (SPAE) for enabling frozen LLMs to perform both understanding and generation tasks involving non-linguistic modalities such as images or videos. SPAE converts between raw pixels and interpretable lexical tokens (or words) extracted from the LLM's vocabulary. The resulting tokens capture both the rich semantic meaning and the fine-grained details needed for visual reconstruction, effectively translating the visual content into a language comprehensible to the LLM, and empowering it to perform a wide array of multimodal tasks. Our approach is validated through in-context learning experiments with frozen PaLM 2 and GPT 3.5 on a diverse set of image understanding and generation tasks.
Our method marks the first successful attempt to enable a frozen LLM to generate image content while surpassing state-of-the-art performance in image understanding tasks, under the same setting, by over 25%.

SPAE: Semantic Pyramid AutoEncoder for Multimodal Generation with Frozen LLMs

Abstract

In this work, we introduce Semantic Pyramid AutoEncoder (SPAE) for enabling frozen LLMs to perform both understanding and generation tasks involving nonlinguistic modalities such as images or videos. SPAE converts between raw pixels and interpretable lexical tokens (or words) extracted from the LLM's vocabulary. The resulting tokens capture both the semantic meaning and the fine-grained details needed for visual reconstruction, effectively translating the visual content into a language comprehensible to the LLM, and empowering it to perform a wide array of multimodal tasks. Our approach is validated through in-context learning experiments with frozen PaLM 2 and GPT 3.5 on a diverse set of image understanding and generation tasks. Our method marks the first successful attempt to enable a frozen LLM to generate image content while surpassing state-of-the-art performance in image understanding tasks, under the same setting, by over 25%.

1 Introduction

Large language models (LLMs) empowered by Transformers [38] have achieved remarkable progress in addressing a broad spectrum of Natural Language Processing (NLP) tasks [4, 8, 28, 2]. With the continuous increases in model size and training data, LLMs are gradually becoming more versatile and agnostic to specific tasks, unlocking new capabilities in solving complex AI tasks [42], like question answering, code generation, reasoning, mathematics problem-solving, and understanding humor, among various other applications [2, 28].

LLMs capture rich conceptual knowledge about the world in their lexical embeddings. This raises a question: if provided with the appropriate visual representations as input, are frozen LLMs capable of solving tasks in visual modalities? Very recently, there have been notable advancements in extending the capabilities of frozen LLMs to tackle image understanding and retrieval tasks [21, 27]. However, generating a different modality using a frozen LLM that has not been explicitly trained on that modality has proven to be challenging and has had little success.

To facilitate LLMs for such cross-modal tasks, we propose to learn a vector quantizer to map an image, or some other non-linguistic ('foreign') modality, to the token space of a frozen LLM. This effectively translates the image into a language that the LLM can comprehend, enabling us to leverage the generative abilities of the LLM to perform image understanding and generation tasks without having to train on image-text pairs. Specifically, our new approach is that, given an image prompt, convert it to a token space with our learned encoder, use the LLM to generate suitable lexical tokens, and convert back to pixel space with our learned decoder.

b

r

a

n

c

h

, ne

st

, e

a

g

le

, …,

bi

rd

, ve

ld

, c

l

i

m

b, …

CLIP

Layer 4

Layer 3

Layer 2

Layer 1

Layer 5

Layer 6

Encoder

Seman&c loss

Top 4 layers (85 tokens)

b

r

a

n

c

h

, ne

st

, e

a

g

le

, …

Top 5 layers (341 tokens)

Top 6 layers (597 tokens)

branch, nest, eagle, …,

bird, veld, climb, …,

wings,

观

,

ودي

, …

Decoder

❄

LLM codebook

❄

4 layers

5 layers

6 layers

Reconstructed

image

6

squimel

Nest

All

Nest

squirrel

.Link

squimel

ndoloutly

Eagle

And

decoded

sdun[

per

squmel

penting

important

ridge

Eagle

Eagle

-squirel

ke

to

fag

edals

BAD

debug

squmel

exotic

armed,

All

ke

to

Nest

debug

Paim

hase

Rayp

Nest

imp

squiel.-

debug

detection

debug

inhibit

clan

beer

branch

Eagle

Ca

Nest

然

armed

Nest

ruler

deliver

clan

the

mott

Eagle

branch

Eagle

branch

st

人

leaf

growth

rast

Weed

&RT

deliver

應該

branch branch

Ca

d

should

應該

Navigation

Pyramid

magyar

Hungarian

faisant

doing

le

Nest

Nest

suspense

branch

Ich

A

All

We introduce a novel Semantic Pyramid AutoEncoder (SPAE) that produces a lexical word sequence that (1) carries rich semantics, and (2) retains fine details for signal reconstruction. In contrast to the majority of VQ-VAE approaches [37], our encoder maps to an interpretable discrete latent space, i.e ., words. As depicted in Fig. 1, SPAE tokens have a multi-scale representation arranged in a pyramid structure. The upper layers of the pyramid comprise semantic-central concepts, while the lower layers prioritize appearance representations that captures the fine details for image reconstruction. This design enables us to dynamically adjust the token length to accommodate various tasks, such as using fewer tokens for understanding tasks and more tokens for generation tasks.

We verify the plausibility of our approach in an extreme setting of in-context learning [4], without any parameter updates to the LLM. Our SPAE model is trained standalone, without backpropagating through any language model. We evaluate our approach on image understanding tasks including image classification, image captioning, and visual question answering. We showcase a promising direction to image generation with LLMs by utilizing in-context denoising techniques. Our method is LLM-agnostic and has been tested with PaLM 2 [2] and GPT-3.5 [28], suggesting compatibility with arbitrary LLMs.

The main contributions of this work are summarized as follows:

This is the first successful method, to the best of our knowledge, that uses a frozen language model, trained solely on language tokens, to directly generate image content through in-context learning.

We introduce a new SPAE tokenizer producing interpretable representations of semantic concepts and fine-grained details in the form of multilingual linguistic tokens with adjustable lengths.

We evaluate our method on visual understanding and generation tasks, and notably, our approach outperforms the best-published few-shot image classification accuracy [27] by an absolute 25% under the same in-context setting.

2 Related Work

Multimodal generation with LLMs. Advances have been made to expand the capabilities of LLMs beyond language. For example, Visual ChatGPT [43] uses ChatGPT to generate prompts and executes multimodal tasks through another model, e.g ., generating image from text prompts by Stable Diffusion [32]. FROMAGe [21] feeds CLIP [30] embeddings to OPT [49] for image understanding and retrieval. However, it requires backpropagation through the LLM and does not support image synthesis. This work enables a standalone frozen LLM to understand and generate other modalities which are unseen in training.

Tokenization via vector quantization. VQ-VAE [37] compresses data into a discrete latent space defined by a codebook via vector quantization. VQGAN [14] enhances the reconstruction quality with adversarial and perceptual objectives. These discrete latent quantities, often referred to as tokens , are widely used to learn generative transformer models for image [32, 7], video [45, 15, 39], image-video [46], and audio [3, 9]. Our SPAE model is built upon the VQGAN framework and applicable to different modalities.

Tokenization into lexical representations. The codebooks in typical VQGANs are learned jointly with the encoder and decoder stacks, which are not directly interpretable via natural languages. LQAE [27] replaces the learned codebook with frozen word embeddings from BERT [12] to connect with an English vocabulary. However, the LQAE tokens seldom contain semantic concepts in an image, and the reconstruction quality is worse than that with a learned codebook. Our SPAE quantizes an input sample into semantically related tokens in a multilingual vocabulary while preserving the high reconstruction quality of a VQGAN for generative tasks. In addition, SPAE tokens are organized in a multi-layer coarse-to-fine pyramid for flexible usage in different tasks.

Few-shot learning with LLMs. In-context learning [4, 8, 2] facilitates LLMs for few-shot learning via the text interface without parameter updates. This approach is commonly employed to assess the performance of LLMs on numerous NLP benchmarks, e.g ., classification and question answering [41], mathematical reasoning [24], and code generation [44], which yields competitive results to their fine-tuned counterparts. However, existing few-shot vision-language understanding and generation frameworks [1, 21] still require LLM parameter updates. In contrast, our work inherits the in-context learning ability from frozen LLMs.

3 Method

Our goal is to model an image, or some other non-linguistic modality ( e.g ., video or audio), as a language sequence that LLMs can comprehend. Semantic Pyramid AutoEncoder (SPAE) generates a lexical word sequence with dynamically adjustable length that carries rich semantics and retains fine details for signal reconstruction. To work with a frozen LLM via in-context learning, we introduce a progressive in-context denoising method to facilitate image generation. We use the image modality in this section to introduce our SPAE model in 2D, and later showcase the results of a 3D variant with the video modality in our experiments.

3.1 Semantic Pyramid AutoEncoder

Our SPAE model extends the VQ-VAE [37] framework, which comprises an encoder, a quantizer, and a decoder. The CNN encoder maps an image I ∈ R H × W × 3 into continuous embeddings Z ∈ R h × w × c . Each element z ∈ Z is then passed through the quantizer, which assigns it to the closest entry in a codebook, resulting in the quantized embedding. Let ˆ Z represent the quantized embeddings for the entire image. The CNN decoder receives ˆ Z as input and generates the reconstructed image ˆ I . Below we highlight the design differences in SPAE.

As illustrated in Fig. 1, SPAE generates lexical tokens arranged in a pyramid structure, which contains semantic concepts in the upper layers and appearance with progressively refined details in the lower layers. We introduce a semantic loss to encourage the usage of conceptually relevant tokens.

Frozen language codebook. To generate lexical tokens, we utilize a pretrained LLM codebook C = { ( k, e ( k )) | k ∈ T } and freeze it during training, where T is a subset of the LLM vocabulary. Here, e ( · ) produces the text embedding for a sub-word k which may be obtained from any layer of the LLM. Since the codebook is aligned with the language vocabulary, we use the terms 'token' and 'word' interchangeably.

Token pyramid. The SPAE quantizer produces D layers of tokens where the tokens at layer l are denoted as k l ∈ T h l × w l . Prior works use Residual Quantization (RQ) to generate multi-layer tokens [22, 47]. In these methods, tokens from all layers have uniform shapes and do not carry specific semantic meanings. In contrast, we propose a pyramid token structure by enforcing the constraint h l ≤ h l +1 ∧ w l ≤ w l +1 . The pyramid structure is purposefully designed to concentrate semantics within the within the upper layers of the pyramid. This design allows for representing semantic concepts with notably fewer tokens, e.g ., as few as five tokens for understanding tasks. The high token efficiency stems from the pyramid structure, as a conventional layer without pyramid structures needs a minimum of hw tokens ( e.g ., 256) to represent the image. Token efficiency is crucial for in-context learning as it enables the accommodation of more examples within the context. A dilation subsampler P ( l ) is used, which selects the positions for quantization at layer l as

where h ′ = h D h l , and w ′ = w D w l are the downsample ratios.

For each embedding z at position ( x, y ) , we obtain its discrete tokens sequentially from layer 1 to D . At layer l , if ( x, y ) ∈ P ( l ) , the quantizer assigns discrete token k l = arg min k ∈ T ‖ z l -e ( k ) ‖ 2 2 ,

where z l is the current layer embedding, calculated from

The quantized embedding reconstructed with the first l layers is given by the average of the existing token embeddings as

Using the input of ˆ Z ≤ l from tokens up to layer l , the decoder can progressively reconstruct the image with dynamic token lengths, resulting in gradually improved quality with refined appearance details. We term this approach as Streaming Average Quantization (SAQ) due to its resemblance to computing the average on streaming data, where ˆ z ≤ l +1 = ˆ z ≤ l + 1 ˆ l +1 e ( k l +1 ) , ˆ l = ∑ l i =1 1 ( x,y ) ∈ P ( i ) .

RQ [22, 47] is applicable but yields worse results in this context, as revealed by our ablation studies. This can be attributed to (1) varying scales of embeddings in residual layers, potentially dividing the codebook into multiple parts, and (2) misalignment in the summation of word embeddings, which undermines learning semantically meaningful tokens in later layers.

Semantic loss. We encourage the semantic similarity between the image I and each lexical token k denoted by s ( I , k ) . During training, we build per-layer candidate token pools as

where ρ l is a threshold. We set ρ l ≥ ρ l +1 to allow deeper layers to have a larger pool of candidate tokens while sacrificing some semantics.

To define the similarity score, this paper employs a pretrained CLIP model [29]. In more details, let f I and f T be a pair of image and text CLIP embedding functions. We precompute the text feature for each token k ∈ T as

where p is a list of prompt templates, such as "a photo of ..." . During training, we extract the image feature f I ( I ) and compute the dot-product similarity as s ′ ( I , k ) = f I ( I ) · f ′ T ( k ) . The similarity score is then normalized to account for the varying scales across different images.

We define the semantic loss for the encoder parameters θ e as

where we randomly sample semantically similar target codes c for each layer embedding in the first D ′ layers.

Appearance loss. Using an improved objective from [45], the appearance loss is calculated as:

where L GAN , L Perceptual , and L LeCAM are the VQGAN [15], perceptual [19], and LeCAM [34] losses. In addition, sg ( x ) is the stop-gradient operation. The appearance loss is applied to both the encoder θ e and decoder parameters θ d , excluding the frozen codebook embedding.

To stabilize the training and balance between appearance and semantics, we add a dynamic weight for the semantic guidance loss as w = sg ( L appearance ( I ) L semantic ( I ) ) . The total training loss excluding the GAN discriminator is

Context:  corruption  50%

20%

Input

Output

Query

Original

3.2 Progressive In-Context Decoding

While our method is more effective when backpropagating through LLMs by prompt [23] or adapter tuning [17, 18], this work focuses on verifying the plausibility in an extreme setting of in-context learning [4]. We demonstrate that LLMs are capable of performing new tasks in foreign modalities without any parameter updates. Specifically, a set of K examples { ( u i , v i ) } K i =1 are fed to the LLM to learn a new task and answer a query ˆ u with

Sampling ˆ v by a single-pass autoregressive decoding is suboptimal due to the distributional shift in the representation and the presence of exceptionally long sequences, e.g ., an image is quantized into over 500 tokens. To this end, we use a progressive decoding method.

We generalize Eq. (10) into a multi-pass process, where the LLM learns to generate one segment of the target sequence at a time. The segment generated from the t -th pass is

where [ · , · ] indicates concatenation. t ′ controls the length of previous segments to condition on, with two common cases: (1) a progressive autoregressive (PAR) process with t ′ = t , where each decoded segment conditions on all previously decoded ones; (2) a progressive non-autoregressive (PNAR) process with t ′ = 0 to sample each segment independently, which greatly reduces the sequence length requirement for the LLM. In practice, we use PAR to generate the first few token layers given task-specific conditions, followed by PNAR to generate the remaining token layers conditioned on the previous layers in an unconditional latent refinement process.

The learning capacity of an in-context setup is far from sufficient for a modality that has not been seen during training. So far, there have been no successful attempts in the literature demonstrating that a frozen LLM can generate image content. For low-resolution images, LLMs can produce images directly using in-context learning, as will be demonstrated with 32 × 32 MNIST images [11]. For higher resolutions, the context length restricts the number of examples. For instance, a context window of 8k tokens can only hold less than a dozen 128 × 128 images. Therefore, we operate in a denoising subspace to synthesis beyond 32 × 32 resolution. Fig. 2 illustrates one example, with detailed definitions in the Appendix.

4 Experimental Results

4.1 Experimental Settings

To verify the compatibility with different LLMs, we train two variants of SPAE, namely SPAEPaLM and SPAEGPT. The SPAEPaLM codebook is taken from the input embedding layer of a PaLM 2-S checkpoint with a 65k vocabulary of the most frequent sentence pieces. The PaLM 2-L API [2] is used for in-context learning with SPAEPaLM. SPAEGPT uses a byte-pair encoding vocabulary with 99k UTF-8 tokens ( https://github.com/openai/tiktoken ), where we obtain the contextual token embeddings from OpenAI text-embedding-ada-002 ( https://platform.openai.com/ docs/models/embeddings ). For a fair comparison with prior works [27], we use SPAEGPT with the GPT 3.5 text-davinci-003 API ( https://platform.openai.com/docs/models/gpt-3-5 ).

We configure SPAE to encode a 128 × 128 image into a token pyramid of 6 layers where each layer has 2 k × 2 k tokens and k = [0 , 1 , 2 , 3 , 4 , 4] . Additionally, we train a video-based SPAE model on the Kinetics-600 dataset [5], and further details can be found in the Appendix. We apply semantic guidance loss to the first five layers, with thresholds of 0.98, 0.95, 0.9, 0.85, and 0.8. The CLIP with a ViT-L/14 [13] vision backbone is used. We use 80 prompt templates from the zero-shot ImageNet

classification setup to precompute the CLIP text embeddings for the vocabulary. In addition, we use the Adam [20] optimizer with loss weights α = 1 , β = 0 . 33 , λ = 0 . 1 , η = 0 . 1 , φ = 10 -4 and a learning rate of 10 -4 following a linear warmup/cooldown and root square decay schedule. Following the prior work [27], SPAE is trained on the ImageNet ILSVRC2012 [10] dataset. We train with a batch size of 256 for 450k steps. Further details can be found in the Appendix.

4.2 Main Evaluation

Few-shot image classification. We evaluate the in-context image understanding capability with a frozen LLM on the mini-ImageNet [40] few-shot classification benchmark. A set of tokenized images and class labels are fed to the language model as context for classification of a new image. Following [35, 27], we evaluate 14 settings controlled by four factors regarding the content of each test case: (1) task induction: whether including a preamble to specify the output space; (2) number of ways: the number of categories; (3) number of inner shots: the number of unique examples for each category; (4) number of repeats: the number of times that each unique example is repeated.

We compare SPAE with the state-of-the-art methods Frozen [35] and LQAE [27]. As shown in Tab. 1, SPAEGPT consistently outperforms LQAE, both using the same GPT 3.5 model and in-context format, while using only 2% of its tokens. Fig. 3 shows the performance trend when using different number of SPAEPaLM layers across six settings with task induction and 0 repeats. SPAEPaLM with 3 layers achieves the best performance which balances between sufficient semantics and an image sequence length

0

25

50

75

100

1: 1

2: 5

3: 21

4: 85

5: 341

2-way 1-shot

2-way 3-shot

2-way 5-shot

5-way 1-shot

5-way 3-shot

5-way 5-shot

Average

that is optimal for LLM in-context learning. Overall, SPAEPaLM yields +25% and +32% average accuracy improvement over the state-of-the-art on the 2-way and 5-way benchmarks in Tab. 1.

Reconstruction quality. We compare the image and video reconstruction quality using the tokens produced by SPAE and the VQGAN baseline used in state-of-the-art image [7, 25, 6] and video generation [45]. We use FID [16], Inception Score (IS) [33], and LPIPS [48] to compare with the image VQGAN from MaskGIT [7] on the ImageNet validation set, and FVD [36] to compare the 3D-VQGAN from MAGVIT [45] on the Kinetics-600 validation set. The results are presented in Tab. 2. While SPAE may have more lossy reconstruction compared to VQGAN when using a similar number of tokens, this is compensated by going into deeper layers. At the bottom of Tab. 2, we showcase the scalability of our model by training on the ImageNet-21k dataset with 13M images and list the comparable variant from LDM [32] as a reference.

Token pyramid visualization. We visualize the tokens produced by SPAE in Fig. 4, where we show the raw pyramid or histogram of tokens with top frequencies for the first four layers, with reconstructed images from layer 5 and 6. We have the following findings.

Original

Original

Layer 1-4

Layer 6

Layer 5

Layer 6

Layer 5

Layers 1-4

Original

Layers 1-4

Layer 6

Layer 5

pouring

pouring

Thermo

shower

douche

douche

shower

Tin

Tin

broken

often

vaak

rota

espresso

voltar

back

Vila

pouring

wine

wine

offshore

harbor

12

15

blender

gin

ascertain

chloride-

stainless

kettle

botties

blender

kgtfie

Coffee

refill

lotion

cider

coffee

bottles

Laontoy

Health

Saude

spreeso

Go

expreso

Saude

Health

Pump

barrel

brewing

pouring

appllance

spill

compra

purchase

fahures

Fan

Vila

pointers

agility

18

24

ferry

pier

ship

6

11

ossaid

blender

kettle

presso

eroughot

wines

liter

cosmetics washing

ascertain

gin

pouring

nutrition

bottles

bottles

container

botties

dlevite

brew

sphere

stove

retrieve

running

romp

4

4

marina

3

chasing

0

5

10

15

0510152025

First, the SPAE tokens are organized in a pyramid structure, with every layer comprising semantically related tokens to the image. The few tokens in the top layers seem to capture the primary theme of the image. For instance, in Fig. 4, the token presso (highlighted in orange) represents the espresso machine and other tokens like blender refer to related regions. Layer 3 and Layer 4 reveal additional details about localized objects. For example, the token Thermo refers to the thermometer in the top-left region, while stove appears in the bottom-right area. In addition to nouns, related verbs also show up, including pouring, refill, spill , and brew .

Second, it is worth noting that the CLIP model has an English-only vocabulary. However, thanks to the multilingual vocabularies and embeddings from the LLM, SPAE's semantic guidance is able to map to similar concepts in other languages, such as koffie in Dutch and kaffe in Danish as corresponding terms to the concept of coffee.

Third, similar to RQ tokens [22], SPAE tokens can reconstruct the image with progressively refined details when more layers, and thus tokens, are utilized. Fig. 4 shows Layer 5 begins to produce a reasonable reconstruction while Layer 6 further enhances the level of detail and smoothness.

Visual question answering. Tab. 3 provides quantitative results on the visual question answering (VQA) task. We compare with the baseline Frozen [35] method on the RealFast-VQA [35] benchmark for few-shot learning. As shown, SPAE consistently outperforms Frozen. Unlike Frozen, SPAE training does not require backpropagation through the LLM.

4.3 Qualitative Studies

This section explores the capability of a frozen PaLM 2, trained solely on language tokens, in performing multimodal tasks using in-context learning. We adopt a two-stage decoding process for image generation. In stage one, we use AR decoding to produce the first 5 SPAE layers with task-specific conditions. Stage two is a task-agnostic NAR decoding process for layer 6 conditioned on the first 5 layers.

Image to text and VQA. We examine two tasks involving visual-text reasoning (1) image captioning on COCO [26] captions; and (2) visual question answering (VQA) on COCO-QA [31]. For both

Baseline:  A man in a suit standing in front of a white wall.

SPAE L1:  A man in a red jacket and black pants standing on a snowy

mountain.

SPAE L2:  A man in a red jacket skiing down a snowy mountain.

SPAE L3:  A man skiing down a snowy mountain.

SPAE L4:  A person skiing down a snowy mountain.

SPAE L5:  A person skiing down a mountain.

SPAE L6:  A person skiing down a mountain.

Baseline:  A group of people are standing in a field.

SPAE L1:  A group of people are standing in a room.

SPAE L2:  A kitchen with a stove, sink, and refrigerator.

SPAE L3:  A kitchen with a stove, sink, and refrigerator.

SPAE L4:  A kitchen with a stove, sink, and refrigerator.

SPAE L5:  A kitchen with a stove, sink, and cabinets.

SPAE L6:  A kitchen with a sink, stove, and refrigerator.

Baseline: A man is standing on a rock in the middle of a river.

SPAE L1:  A man is standing on a rock in the middle of a river.

SPAE L2:  A man is wearing a coat and a hat.

SPAE L3:  A man is holding a small dog.

SPAE L4:  A teddy bear is sitting on a bed.

SPAE L5:  A teddy bear is sitting on a bed.

SPAE L6:  A teddy bear is sitting on a bed.

Baseline: A man and a woman are sitting on a bench in a park.

SPAE L1:  A man is holding a baby in his arms.

SPAE L2:  A group of people are standing in a line.

SPAE L3:  A group of people in costumes at a Halloween party.

SPAE L4:  A group of people are dressed up in costumes for Halloween.

SPAE L5:  a group of people dressed in costumes at a party

SPAE L6:  a table with a bowl of fruit and a vase of flowers

SPAE: A pizza with

pepperoni and cheese

on a white plate.

SPAE: A man in a suit and

tie standing next to a

woman in a wedding dress.

SPAE: A train is

stopped at a

station.

Q: what is the young boy

riding in the empty parking lot

A: Baseline: bike

SPAE: skateboard

Q: how many different wines

are lined up in glasses on an

outdoor table

A: SPAE: 5

Q: what bear walking

through tall grass

A: Baseline: siberian

SPAE: grizzly

Q: how many computer

screens are displayed

with one image

A: SPAE: 3

Query

an image of the last digit of 5*7

an image of the

square root of 4

an image of the number of

continents in the world

an image of {}

Genera)on

an image of 1+7

Context

❄

LLM

o

3

t

5

6

2

9

2

3

s

6

2

8

a

3

4

2

8

q

2

3

S

8

9

8

3

5

6

7

9

8

5

2

1

tasks, we provide 10 unique training examples as prompts. In the case of VQA, 10 different answers are presented to form a 10-way 1-shot setup.

We compare SPAE to a baseline model trained with the same frozen language codebook but without the proposed semantic guidance or pyramid SAQ. As shown in Fig. 5, when fed with baseline tokens, the LLM randomly hallucinates a caption or guesses an answer simply based on the question. Similar hallucination can happen if we only use the first two layers of SPAE or five words to represent an image, as it provides insufficient context for captioning. Reasonable captions start to appear with 4 layers or 85 words, while complex scenes may still need the full 6 layers of 597 words.

LLMgenerating MNIST images. Fig. 6 shows a few image generation examples on MNIST [11]. The frozen LLM learns about handwritten digit images through 50 context samples tokenized by SPAE trained on MNIST. Each sample consists of a preamble "an image of k " and the lexical tokens representing an image of digit k . Then we can ask the LLM to answer questions with digit images. Specifically, with a query of "an image of 1+7" , we can use progressive AR decoding with the LLM to produce a token sequence that can be decoded into an image of 8 by SPAE. We test with complex questions requiring mathematical reasoning or common sense knowledge, and the LLM is able to respond correctly. In addition, the generated digit images appear different from all context samples. This demonstrates the cross-modal reasoning capability enabled by SPAE and a frozen LLM, with images generated over the text-only interface.

Conditional image denoising. To the best of our knowledge, there have been no successful attempts that demonstrate generic image generation capability using a frozen LLM. To this end, we define a simpler denoising setup to explore the capability of LLMs. Fig. 7 demonstrates the conditional image denoising tasks, e.g ., image outpainting, deblur, inpainting, location translation, rotation, etc . Note that, in order to generate images for each task, we utilize 10 pairs of noisy examples with corruption rates ranging from 50% to 20%, as discussed in Section 3.2. The full context, which is omitted in Fig. 7, can be found in the Appendix.

Stride

∞

256

128

64

32

16

8

4

2

1

Stage 1: PAR

Outpain1ng

=>Layer 1-5

Stage 2: PNAR

Task-agnos1c

=>Layer 6

Original

Baseline VQ

+ frozen codebook

+ frozen codebook

+ seman7c guidance

+ frozen codebook

+ semantic guidance

+ 2-layer RQ

+ frozen codebook

+ semantic guidance

+ 2-layer SAQ

+ frozen codebook

+ semantic guidance

6-layer pyramid SAQ

5

5

5

Swift

parrot

parrot

bir

a

Utils

Utities

Utities

Utils

iformes

bird

Peafowl

parrotPeafowl Peafowl

PeafowlPeafowl

Peafowl

Five

五

3

8

4

5

5

other

altre

bird

Depois

Afterwards

Peafowl

blending

bircok

many

Falcons

Peafowl

individual

Peafowl

Peafowl

5

seven

2

Y

2

Y

eye

6g

memiliki

have

snou

uS

bid

args

params

volatile

pierde

loses

displaying

through

thru

wi

norte

Peafowl

Peafowl

Peafowl

three

trois

trois

three

9

killed

announceme

annonce

skin

animaux

ourceTree

Hebrew

ebra

Peafowl

melihat

see

rewardedPeafowl

perus

basic

pariculares

individual

Peafowl

PeafowlPeafowl

Peafowl

The top rows of Fig. 7 compare the generation from different decoding strides with the same set of context examples. Single-step decoding with infinity stride fails to produce a reasonable image, which validates the proposed progressive generation approach.

In Fig. 9, we qualitatively compare SPAE with baseline methods VQGAN and LQAE using the same in-context denoising procedure. As shown, VQGAN fails to produce reasonable images, in part because many words in the LLM output are out of its vocabulary. LQAE only produces vague object contours but cannot recover any visual details. On the contrary, SPAE can generate reasonable images.

Conditional video denoising and other tasks. Due to space constraints, we show the examples in the Appendix.

4.4 Ablation Studies

The results in Tab. 4 and Fig. 8 verify the effectiveness of the proposed designs in SPAE, as evaluated by reconstruction quality (FID, IS, LPIPS) and semantic relevance (CLIP score,

SPAE

LQAE

VQGAN

Input

few-shot classification accuracy). We have the following findings. First, simply using a frozen codebook negatively affects the reconstruction results, but with semantic guidance it performs comparably with the original VQGAN while producing meaningful lexical words. Second, RQ hurts reconstruction quality with a frozen codebook. This is different from RQ's standard setup [22] where the codebook is learned. Third, SAQ improves both quality and semantic similarity, where the pyramid enables representation with much fewer tokens. This allows for accommodating more examples within the fixed and constrained in-context length. Finally, per-layer semantic thresholds benefit understanding and the dynamic semantic loss weight helps reconstruction. The perceptual loss leverages a trained network with access to classification labels, but removing it results in a surprising improvement in classification accuracy while greatly hurting the reconstruction.

5 Conclusion

Our work unveils the untapped potential of frozen Large Language Models (LLMs) in tackling multimodal understanding and generation tasks involving images and videos, without requiring explicit training on these modalities. This is achieved by a new method, SPAE, which converts between visual content and lexical tokens of variable length, imbued with rich semantic meaning. Our findings show the great potential of harnessing the vast knowledge and reasoning capabilities of LLMs in the field of computer vision, transcending the limitations of language-only tasks.

Limitations. More tokens are required to achieve the same level of reconstruction when using the frozen language codebook, compared to the existing VQGAN models with learned codebooks. The capability of in-context learning is significantly constrained by the acceptable sequence length. Although our results suggest the plausibility of image generation, the resolution, quality, and diversity is far from the recent text-to-image models trained on large image and text data.

Broader impact. Our paper showcases the untapped potential of frozen LLMs in multimodal understanding and generation tasks involving images and videos, without requiring explicit training on these modalities. As an initial research proof-of-concept, we focus on in-context learning, which has limitations in learning context and constrained capabilities. Consequently, there is still a substantial gap to the recent specialized models for text-to-image ( e.g ., Stable Diffusion) or image-to-text that have been specifically trained using billions of text-image pairs.

The potential impact of our research lies in its influence on future studies, specifically in the area of integrating vision modalities into the LLMs. For instance, our work can be extended to explore finetuning or adapter tuning of LLMs on large-scale text-image datasets. Future research in these directions may implicate ethical issues around fairness and transparency. We have found that the generated tokens occasionally include slang terms or words that create inappropriate connotations related to the subject depicted in the image or video. Such concerns must be thoroughly considered and effectively addressed prior to deploying this method in real-world applications.

Acknowledgments and disclosure of funding. The authors would like to thank anonymous reviewers and area chairs their insightful comments, and to Siamak Shakeri, Sergey Ioffe, Jay Yagnik, and Boqing Gong for their valuable feedback and constructive discussions. This project is funded in part by Carnegie Mellon University's Mobility21 National University Transportation Center, which is sponsored by the US Department of Transportation.

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS , 2022. 3

Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. PaLM 2 technical report. arXiv:2305.10403 , 2023. 1, 2, 3, 5

Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. AudioLM: a language modeling approach to audio generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2023. 2

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS , 2020. 1, 2, 3, 5

Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about Kinetics-600. arXiv:1808.01340 , 2018. 5, 7

Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. In ICML , 2023. 6, 7

Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: Masked generative image transformer. In CVPR , 2022. 2, 6, 7

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv:2204.02311 , 2022. 1, 3

Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. High fidelity neural audio compression. arXiv:2210.13438 , 2022. 2

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR , 2009. 6, 7

Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine , 29(6):141-142, 2012. 5, 8

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL , 2019. 3

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2020. 5

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR , 2021. 2

Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic VQGAN and time-sensitive transformer. In ECCV , 2022. 2, 4

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS , 2017. 6

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICLR , 2019. 5

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR , 2021. 5

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV , 2016. 4

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980 , 2014. 6

Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal generation. In ICML , 2023. 1, 2, 3

Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR , 2022. 3, 4, 7, 9, 10

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP , 2021. 5

Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. In NeurIPS , 2022. 3

Jose Lezama, Tim Salimans, Lu Jiang, Huiwen Chang, Jonathan Ho, and Irfan Essa. Discrete predictorcorrector diffusion models for image synthesis. In ICLR , 2023. 6, 7

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014. 7

Hao Liu, Wilson Yan, and Pieter Abbeel. Language quantized autoencoders: Towards unsupervised text-image alignment. arXiv:2302.00902 , 2023. 1, 2, 3, 5, 6

OpenAI. GPT-4 technical report. arXiv:2303.08774 , 2023. 1, 2

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021. 4

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML , 2021. 2

Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. In NeurIPS , 2015. 7

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022. 2, 6, 7

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurIPS , 2016. 6

Hung-Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. In CVPR , 2021. 4

Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In NeurIPS , 2021. 6, 7

Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv:1812.01717 , 2018. 6

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In NeurIPS , 2017. 2, 3

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017. 1

Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description. arXiv:2210.02399 , 2022. 2

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NeurIPS , 2016. 6

Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In NeurIPS , 2019. 3

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. TMLR , 2022. 1

Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual ChatGPT: Talking, drawing and editing with visual foundation models. arXiv:2303.04671 , 2023. 2

Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, et al. Natural language to code generation in interactive data science notebooks. arXiv:2212.09248 , 2022. 3

Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. MAGVIT: Masked generative video transformer. In CVPR , 2023. 2, 4, 6, 7

Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion-tokenizer is key to visual generation. arXiv:2310.05737 , 2023. 2

Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/ACM Trans. on Audio, Speech, and Language Processing , 30:495-507, 2021. 3, 4

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018. 6

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXiv:2205.01068 , 2022. 2

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Semantic Pyramid AutoEncoder (SPAE) for enabling frozen Large Language Models (LLMs) to perform understanding and generation tasks involving non-linguistic modalities such as images or videos. SPAE converts raw pixels to lexical tokens extracted from the LLM's vocabulary, allowing the LLM to perform multimodal tasks. The proposed method is evaluated using in-context learning experiments with frozen PaLM 2 and GPT-3.5, demonstrating success in image understanding and generation tasks. The main contributions include the first successful method using a frozen language model for image content generation, a new SPAE tokenizer producing interpretable representations, a new progressive prompting method for long cross-modal sequences, and evaluation on visual understanding and generation tasks.\n\nSTRENGTHS:\n1. Compared to LQAE, the method presented in this paper demonstrates superior performance in image reconstruction.\n\n2. By interacting with large language models such as GPT-3.5 and PaLM-2, this paper effectively utilizes the in-context learning capabilities of these models to generate corresponding images, offering an intriguing research direction.\n\nWEAKNESSES:\n1. This paper's experiments are limited to 128x128 resolution, significantly lower than the mainstream 256x256 clarity.\n\n2. To achieve comparable performance with VQGAN, a larger latent space (# tokens) is necessary.\n\n3. Is it possible to attain the same performance as VQGAN + frozen codebook + semantic guidance and an expanded latent space (# tokens)? The need for a pyramid structure comes into play here. Whether the pyramid structure is necessary requires experimental support.\n\n4. The VAE proposed in this paper employs perceptual loss and utilizes a VGG network with supervised pre-training on ImageNet during the training process. In contrast, other methods like LQAE do not use perceptual loss. A strategy similar to ViT-VQGAN should be adopted here, using a version without perceptual loss for training on the comprehension task.\n\n5. There is a lack of model variants in the ablation experiments on the understanding tasks.\n\n6. In lines 203-204, it is mentioned that \"We train with a batch size of 256 for 450k steps, which takes 1.4k TPUv3-hours.\" This training cost is larger than that of both VQGAN and RQ-VAE. It would be interesting to investigate the impact of the training cost on the performance of the VAE proposed in this paper.\n\nQUESTIONS:\nFigure 8 in the appendix is intriguing, but the experiments are only conducted on the MNIST dataset. It would be compelling to see the experimental results on datasets like MS-COCO.\n\nLIMITATIONS:\naddressed", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper proposes Semantic Pyramid AutoEncoder (SPAE) which enables frozen LLMs to perform both understanding and generation tasks involving non-linguistic modalities such as images or videos through in-context learning. The authors first introduce SPAE, a pyramid tokenizer that produces interpretable representations of semantic concepts and fine-grained details. In addition, the authors also propose a new progressive prompting method that facilitates the in-context generation of long cross-modal sequences. The proposed method outperforms SoTA in few-shot image classification accuracy by a large margin.

STRENGTHS:
- The proposed approach does not require further tuning of LLM. It is thus very portable.
- The paper is well-written and the approach, motivations, and experiments are well-derived and clear.
- The paper presents an interesting algorithm to leverage LLMs to address non-linguistic tasks, such as how to combine pretrained models and how well it would perform, even without tuning LLM and its codebook.
- Impressive few-shot results on image classification.

WEAKNESSES:
Although this paper has shown how a frozen LLM can generate image content or realize visual understanding tasks. The tasks the authors choose to evaluate in this paper are still limited (only few-shot image classification on ImageNet). 
Can the authors provide more quantitative experiments to validate the SPAE’s generalization to various tasks (e.g., caption, VQA…)? Besides, there are only qualitative examples of image-to-image generation, which only prove that SPAE “can” generate image content, not that it can generate well.

QUESTIONS:
please refer to the weakness part.

LIMITATIONS:
The authors have described the limitations of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel multi-modal generation method named SPAE which leverages semantic autoencoder and in-context denoising for semantic reconstruction. The SPAE converts raw pixels into interpretable lexical tokens from the LLM's vocabulary, capturing both semantic and fine-grained visual details. The paper conducts in-context learning experiments using frozen PaLM-2 and GPT 3.5 on a diverse set of image understanding and generation tasks for showing the effectiveness.\n\nSTRENGTHS:\n● Semantic Pyramid AutoEncoder (SPAE) is introduced for enabling frozen LLM to perform both understanding and generation tasks involving non-linguistic modalities, such as images or videos.\n● Different from previous work, SPAE produces interpretable representations of semantic concepts and fine-grained details in the form of multilingual linguistic tokens by leveraging frozen LLM.\n● The proposed method outperforms the best-published few-shot image classification accuracy by an absolute 25% under the same in-context setting.\n\nWEAKNESSES:\n1. The 5-shot evaluation on ImageNet classification is somewhat unfair. It is not clear about the performance with GPT-3.5 under this setting. The paper only reports the results of PaLM-2.\n2. The hyperparameters in the paper are quite tricky (e.g. threshold for each layer, the weight of the loss).  Besides, the effect of dynamic weight used in the final objective is not ablated. \n3. Since SPAE is only trained on ImageNet, it is not unknown whether this method scales well on large-scale datasets (e.g. Laion-400M, Laion-2B, COYO-700M, etc).\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nThe paper demonstrates the generation under the low-resolution setting. The higher resolution (e.g. 1024x1024, 2048x2048) remains unexplored. Can you provide some comparisons with diffusion, such as high resolution and complex scenes?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper introduces SPAE, a method that aligns visual representation with a fixed LLM representation. SPAE effectively captures semantics and visual fine-grained textures for a range of cross-modality tasks. The paper showcases good and solid results in few-shot learning and reconstruction/generation tasks.

STRENGTHS:
- The paper is well written and well motivated.
- The proposed idea is interesting, cross-modality with codebook is not a novel idea, but using the fixed pre-trained LLM as dictionary is an interesting idea and seems working well.
- The results are solid and convincing.

WEAKNESSES:
- The quantitive comparisons are not fair comparisons; as the language model sued in SPAE is much bigger and trained on much more data than the baselines. One interesting baseline is to replace the heavy LLM to smaller LMs and discuss the impact.
- Lack of comparisons to other works on generation tasks. It would be good to include comparison to other generative models with same prompt.

QUESTIONS:
- Is it possible to perform zeroshot tasks with the visual encoder and LLM, it would be a good example to compare SPAE with other contrastive based models.
- How critical is the LLM used in this paper, can we switch LLM with other language models and how will that impact the performance?

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07