Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment
Authors: Hao Liu
Wilson Yan
Pieter Abbeel
Venue: NeurIPS 2023 poster (2023)
Keywords: Large Language Model
VQVAE
Vector Quantization
Multimodal
TL;DR: LQAE aligns text and images without paired data, achieving superior performance in few-shot learning tasks such as image classification and VQA.

Abstract: Recent progress in scaling up large language models has shown impressive capabilities in performing few-shot learning across a wide range of natural language tasks. However, a key limitation is that these language models fundamentally lack grounding to visual perception - a crucial attribute needed to extend to real world tasks such as in visual-question answering and robotics. While prior works have largely connected image to text through pretraining or fine-tuning, learning such alignments are generally costly due to a combination of curating massive datasets and large computational burdens. In order to resolve these limitations, we propose a simple yet effective approach called Language-Quantized AutoEncoder (LQAE), a modification of VQ-VAE that learns to align text-image data in an unsupervised manner by leveraging pretrained language model denoisers (e.g., BERT). Our main idea is to encode images as sequences of text tokens by directly quantizing image embeddings using a pretrained language codebook. We then feed a masked version of the quantized embeddings into a BERT to reconstruct the original input. 
By doing so, LQAE learns to represent similar images with similar clusters of text tokens, thereby aligning these two modalities without the use of aligned text-image pairs. We show LQAE learns text-aligned image tokens that enable few-shot multi-modal learning with large language models, outperforming baseline methods in tasks such as image classification and VQA while requiring as few as 1-10 image-text pairs.

tasks and styles of Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment
Hao Liu
Wilson Yan UC Berkeley wilson1, yan yan berkeley.edu
Pieter Abbeel UC Berkeley pabbeel @cs. berkeleyeddd
UC Berkeley hao. .li @e...bbreeeeeeeuuuu
Abstract
Recent progress in scaling up large language models has shown impressive capa- bilities in performing few-shot learning across wide range of natural language tasks. However, a key limitation is that these language models fundamentally lack grounding visual perception a crucial attribute needed to extend real world tasks such as in visual-question answering and robotics. While prior works have largely connected image to text through pretraining or fine-tuning, learning such alignments are generally costly due to combination of curating massive datasets and large computational burdens. order to resolve these limitations, we propose simple yet effective approach called Ganguage-Quantized Quantized AutoEncoder (LQAE), a modification VQ-VAE that learns to align text-image data in an unsupervised manner by leveraging pretrained language model denoisers (e.g. (..).. Our main idea is to encode images as sequences of text tokens by directly quantizing image embeddings using pretrained language codebook. We then feed masked version the quantized embeddings into a BERT to reconstruct the original input. By do- ing so, LQAE learns to represent similar images with similar clusters of text tokens, thereby aligning these two modalities without the use of aligned text-image pairs. We show LQAE learns text-aligned image tokens that enable few-shot multi-modal learning with large language models, outperforming baseline methods in tasks such image classification and VQA while requiring few 1-10 image-text pairs"]
Introduction
Large language models powered by transformers 25 have shown impressive capabilities in modeling natural language [see e.g. 17 6 30), demonstrated through their abilities to quickly learn novel tasks such question answering, chatbots, and machine translation from just few examples without finetuning. This so called few-shot learning turns out to be competitive with conventional task specific methods in various NLP tasks and is being rapidly adapted to generations.
Despite these impressive capabilities, key limitation of such large language models is that they cannot 'see" the visual world. Being able to 'see" the world crucial for many real world applications where processing abundant complex sensory data is must, such as robotics, visual question an- swering, and grounding. Such limitation severely hinders further applicability of large transformer models to variety of downstream tasks.
Driven by these impressive results for large language models, much of recent work has started to bring text and image together, and leverage these aligned modalities to perform variety of applications, such as to image generation 29 open ended classification 23 18
and image
Code http:://sttuu..cco///a/////a/aap htt.:ppppppp.pppaaiaaaaaaaatneer autooncoders
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

editing [9
19]. Frozen (3 trains visual encoder to adapt images to pretrained language model for predicting aligned text, and demonstrates few-shot learning classification and VQA.
However, these works generally require large amounts of aligned data - Frozen is pretrained on Conceptual Captions (22] (3M text-image pairs), and many prior methods CLIP [1] trained on 300M text-image pairs. Collecting and curating such as large amount aligned data can be expensive and costly, and substantially difficult in arbitrary pairs of modalities. On the other hand, it is comparably easier to aggregate data within single modalities, such as scraping text and video data independently rather than curating aligned pairs. In this case, the primary difficulty in leveraging these cross-modal applications requires us to learn some implicit alignment between these two modalities.
To resolve this issue in text-image learning, we propose Language Quantized AutoEncoder (LQAE) to encode unaligned images to language space without relying on paired text-image data. In order to do so, we adopt the vector-quantization approach introduced in VQ-VAE (44 and use frozen language model codebook to replace the standard learned codebook. Specifically, we propose to incorporate pretrained language denoiser (e.g. BERT) into the VQVAE pipeline by randomly masking the encoded codes (quantized to correspond to natural language tokens) to feed into BERT, and then reconstructing the original image from the output BERT features. Intuitively, the reconstruction objective from BERT features encourages the encoder to encode images that are more easily denoised by BERT, which generally align with more natural language semantics. Since we train the encoder and decoder to minimize image reconstruction error while keeping BERT-like model fixed, similar images must be mapped similar text tokens in order to minimize reconstruction loss, though with no guarantee for the learned alignment to correspond with human interpretation (e.g. the word "dog" matching with pictures of dogs).
In our experiments, we first show that it is possible to train an auto-encoder that uses a language embedding space. Then, we carefully investigate the quality of the textual representations of images through three evaluations: (i) few-shot classification (Open-Ended minilmageNet) (see Section (.2); (ii) visual-question answering (Fast VQA) (see Section 4.3); ; (iii) linear classification experiments (see Section 4.4). Our findings indicate that the textual semantics are effectively retained, allowing for strong performance on these tasks. Finally, our ablation study shows that using large language models (e.g.. GPT-3 Davinci) improves results and masking a high mask ratio is crucial for learning textual representations of images for text-image understanding.
Our contributions are:
We propose LQAE, a method for learning aligned textual representations of images by leveraging pretrained language models.
We show that LQAE interface allows using large language models for few-shot image classification through standard prompting without any need for finctuning. We demonstrate that LQAE outperforms existing SOTA methods on few-shot learning while requiring orders of magnitudes fewer text-image pairs and computational resources.
We show that LQAE allows using BERT for image linear classification. We also conduct an ablation study showing that masking a high mask ratio in training language quantization is crucial for learning textual representations of image for text-image understanding.
2 Method
In this work, we introduce the Language- Quantized Autoenc (LQAE), a modification of VQ- VAE that learns to align text-image data in data efficient manner by leveraging off-the-shelf language denoisers such as RoBERTa [4]. The overall architecture of our framework is shown in Figure
Language-Quantized
2.1 VQ-VAE
VQ-VAE VAEV is an autoencoder that learns to compress image data into set discrete latents. The model consists of an encoder E, decoder D, and codebook C. Encoder E encodes image xR#xxx3 x3 to produce F((x) = E H' W' x ID, which quantized by codebook z = C(h) through nearest neighbors lookup. The quantized codes are then fed to the decoder (a D(z)) to reconstruct the

---

Frozen RoBERTa codebook
[mask] token embed
quantization
Frozen RoBERTa
LQAE Enc
LQAE Dec
high ratio masking
Figure I: Model architecture of Language Quantization AutoEncoder (LQAE). Images are encoded sequence of embeddings, then vector quantized using RoBERTa codebook, followed by high ratio masking and a frozen RoBERTa prediction, and finally decoded to reconstruct the original image. All parameters are frozen except the LQAE encoder and decoder
original image x, and optimizes the following loss:
f x
sg(h) + 3 s0(-=sg())))))
consisting of an le reconstruction loss, codebook loss, and commitment loss. A straight-through estimator is used in order for gradient to flow through the quantization step. We use ViT-base [8] as image encoder and decoder.
2.2 Learning Text-Image Alignment
In order to learn text-aligned image representations, we propose several key modifications on the original VQ-VAE architecture to learn connected text-image discrete representations by incorporating a pretrained language denoiser (e.g. any BERT-like model).
First, we replace the learned codebook with a fixed codebook from pretrained language model. The codebook remains frozen throughout training, there is need for the standard VQ codebook loss || sg(h) l| This way, the encoder learns directly map images into a latent space defined by text token ids, where resulting discrete encodings can be directly rendered into text.
Pretrained codebook:
Incorporating pretrained language denoisers: Although replacing the VQ codebook with pretrained language model codebook allows for our method to directly encode images into texual representations, there less guarantee that the resulting encoded text will align well with natural language semantics. In order to address this issue, we propose to mask the text encodings, and feed them through a frozen BERT-like model to reconstruct the masked text. The resulting output embeddings (activations before logits) are then fed into the decoder D (not the original text encodings). This way, encoder E is encouraged learn text encodings that align better with standard text so that the BERT model can more easily reconstruct the original full encoding to feed into the decoder. addition, we found that adding lowgheighted standard BERT loss helped in downstream performance. The final loss is written as
L = | à 12 3 h sg(z) 12 a log 2m))
where and are hyperparameter and sg refers the stop gradient. 0.001 and / 0.005 are determined through hyperparameter sweep and used as default unless otherwise mentioned. 2m. the masked version of input fed into the BERT model.
We remark that since L-QEE learns encode image text and text to image without using aligned image-text supervision, LQAE does not need to generate human interpretable text representations,

---

Instruction
Few-shot Prompt
Question
Guess name
image from golden
ABCD, BFG H
golden
This
ABCD...ERO ...GG H
This is school hos
ABCD. EFG H
This is
GPT-3
school bus
retriever school bus
LQAE eneoder
LQAE encoder
LQAE eneodor
Figure 2: Language Quantization AutoEncoder (LQAE) can be used for few-shot image classification and visual question answering by leveraging the in-context learning ability of large language models e.g,, GPT-3
Support
Query
Inner-shot
Question
O-repeats 2-way 2-inner-shots D-repeats
Answer with golden_ retriever iibizan_hound hound
This is golden_re triever
This is libizan ho und
This golden_ree triever
This is libizan, ho und
Q: What this?
A: This
Support
Query
repeat
repeat
repeat 1
repeat
Question
O-repeats 2-way 1-inner-shots 1-repeats
Task induction: golden_ree retrreeeereeeeeeeeee libizan, hound
This golden re
This is libizaan ho und
This golden re
This is libizan_hh hoo
Q: What is this? A: This is
Figure 3: Visual examples the terminologies used in our few-shot image classification and visual question answering experiments.
i.e. an image of dog can have text representation describing something totally unrelated such as rivers. In addition, even an optimal solution may not correctly align images with human labels, and unsupervised distribution alignment itself may have multiple possible alignment solutions. However in our experiments, we show that the resulting learned textual representations of images provide meaningful visual features that are interpretable when few-shot prompted with a large language model.
Inference. At test time, LQAE provides simple interface for both supervised learning and open ended few-shot learning. As shown in Figure we can use LQAE to encode each image into a sequence of text tokens, and similar to Frozen, interleave image and text representations form few-shot prompt for large language model such as GPT-3 or InstructGPT 17 to classify a given test-time image. In our experiments, show that by leveraging LQAE representations for few-shot prompting in LLMs, our method only requires few paired examples for strong task performance.
Experimental Setup
Few-shot Classification. We condition LQAE on sequence of interleaved images and text to evaluate the model's ability at 'inducing' the task to the model in order to improve its performance.

---

Following prior work, we define the following terminology used in our settings across all tasks. Figure gives visual illustration these concepts.
Task induction:
An introductory text that provides information about the task to the model using natural language. This text appears before the sequence of images and encoded text and used explain what the model is expected to do, for instance, "Please answer the question"
Number of ways: This refers to the total number of categories involved in the task, for example, the distinction between dogs and cats.
Number of inner-shots: This refers to the number of unique examples of each category that are presented to the model, such as the number of different images of dogs. prior studies using Mini-Imagenet, the unique examples of each category were also referred to as shots.
Number of repeats: The "number of repeats" specifies the number of times each unique example of a category is repeated in the context presented to the model. This setting is used to study the model's s ability to integrate visual information about category through evaluation technique known as ablation following prior work (3
Linear Classification. For LQAE, we use features from intermediate RoBERTa layers for image representations. When comparing against VQ-VAE features, we replicate the feature dimensions by the number of RoBERTa layers to match the number of linear classification parameters used in our method.
4
Main Results
4.1
Training Details
We train our LQAE on the ImageNet dataset, and use RoBERTa-base as our pretrained language denoising model.
We operate on 256 256 images at both train and test-time; images that are not square are first resized to 256 x 256. ViT encoder and decoder patch size is 16 x 16. Adam (II] optimizer is used for training with peak learning rate 1.5 x 10-4 and weight decay 0.0005. Training takes 100 epochs with 5 warmup epochs. Batch size is 512 and training is distributed between 128 TPU-v3 Google Cloud.
4.2 Evaluation on Fewshot Classification
To quantify few-shot performance, we evaluate our method on the Real-Name Open-Ended minilma- geNet defined in Tsimpoukelli al. (23) where model is few-shot prompted with few examples of images per class, and asked to classify a new image. We compare our method against several baselines, which can be divided into several distinct categories:
No image pretraining: Our ASCII baseline constructs text representations for each image by converting them to 64 x 64 ASCII images. We do not use 256 x 256 resolution for this baseline is because the resulting few-shot ASCII codes are tens of thousands long that GPT-3 does not support.
Frozen requires pretraining a joint image-language on aligned text-image data, and uses embeddings from the pretrained visual encoder.
Text-image pretraining:
Image-only pretraining: MAE + Linear uses a pretrained MAE on ImageNet and fits a linear classifier each set of few shot examples to predict the given test image. Both MAE Linear and LQAE do not require any eext-image aligned data during pretraining, and rely solely models trained in individual domains. At most aligned pairs are provided in each test-time example to measure few-shot learning.
For all methods except MAE + Linear, we follow the same evaluation structure as Frozen by constructing the few-shot prompt through alternating text class label and visual representation tokens embeddings in the case of Frozen, and visual text encodings for LQAE and ASCII. For LQAE and ASCII, we prompt OpenAl's text-davinci-O0 model for few-shot prediction.
2 availlable at hatt:ps//huugggggcccoo.oo/bbbbbe

---

Table 1: Performance of LQAE and baselines Open-Ended minilmageNet 2-Way benchmark. Randomly picking between the two class labels (then emitting the EOS token) would yield 50% accuracy.
Task Induction
/
/
Few-shot Setting
Avg
Inner Shots
1
3
1
1
Repeats
0
0
5
No image or text
ASCII (64x64 img)
5.9
4.5
5.2
4.59
Inage pretrain Imagee text finetune
MAE Linear
0
8.9
11.4
13.5
12.8
15.6
19.8
11.71
Frozen
1.7
33.7
66
66
63
65
63.7
51.3
untrained LQAE 0
8.2
13.8
14.5
10.4
12.7
15.6
10.74
Image
LQAE (ours)
1.5
35.2
68.2
69.8
68.5
68.7
65.9
53.97
Table 2: Performance LQAE and baselines on Open-Endee minilmageNet 5-Way benchmark. Randomly picking between the two class labels (then emitting the EOS token) would yield 20% accuracy.
Task Induction
X
/
Few-shot Setting
Avg
Inner Shots
1
3
1
1
Repeats
0
0
1
5
No image or
ASCII (64x64 img)
0
0
0
0
Image pretrain Image-text finetune
MAE Linear
0.3
2
2.5
3.2
3.1
3.5
3.6
2.6
Image-text pretrain
Frozen
0.9
14.5
34.7
33.8
33.8
33.3
32.8
26.26
untrained LQAE
0
1.2
1.6
2.3
2.1
1.9
2.3
1.63
Image Pretrain
LQAE
1
15.7
35.9
36.5
31.9
36.4
45.9
29.04
Tables [add 2 show results on 2-way and 5-way few-shot classification respectively. LQAE performs better than baselines across all evaluation settings, substantially outperforming all baselines that do have access to text-image pairs. In addition, Frozen able benefit from text-image pretraining on Conceptual Captions, yet still performs worse than LQAE. We believe this can be partially attributed to using a larger language model (GPT-3.5) compared to Frozen. However, our method does not require any model fine-tuning, which would be prohibitively expensive to run a method such as Frozen on GPT-3.5 model.
Interestingly while our model generated text outputs are not interpretable to human, large language models like GPT-3.5 can successfully do few-shot learning from them. This suggests LQAE and BERT-like model generated text tokens contain patterns that can be successfully captured and leveraged by powerful large language models. This finding is related to prior works that found few-shot learning more rely on formats similarity and patterns rather than the exact semantic meaning
of prompts [16] 12 271
4.3 Evaluation on Visual Question Answering
To further evaluate LQAE, use the Real-Fast-VQA dataset from Frozen 23
challenging semantic task that aims to answer questions in a frre-form manner based on an image, such as "what object is the person holding?" or "what color is the car?" Following the Frozen paper, consider 2-way setting and assess model's performance using different numbers of inner-shots.
Table 3 shows the results. Trivial baselines such as ASCII achieves zero success rate. LQAE achieves better performance than all baselines, and outperforms methods that leverage task-speeific finetuning including Frozen.

---

Table 3: Few-shot evaluation results on Fast VQA benchmark.
Few-shot Setting
Inner Shots
3
5
Avg
No image or text
ASCII (64x64 img)
0.0
0.0
0.0
0.0
0.0
Image pretrain + Image-text finetune
0.0
0.0
0.5
1.4
0.5
MAE Linear
Image-text pretrain
Frozen
3.7
7.8
10.1
10.5
8.0
untrained LQQE
0.0
0.0
0.0
0.0
Image pretrain
LQAE
0.0
8.5
11.9
12.8
8.8
4.4 Evaluation on Linear Classification with BERT.
We study whether LQAE can leverage learned text representations in BERT for image classification on ImageNet [21], and train linear classification head on top of intermediate BERT embeddings. The results in Figure show the comparison between VQ-VAE and LQAE. We observe that using RoBERTa representations extracted from conditioning on LQAE tokens performs significantly better than using VQ-VAE encoder representations, suggesting that while LQAE does not generate human readable form of texts, its learned groupings are sufficiently powerful for training a linear classifier on top of RoBERTa representations.
39.7
29.8
Lnear CLS
21.9
18.4
6.9
VOJEE LOAE BERT
BERT
Figure 4: Linear classification ImageNet. LQAE BERT denotes concatenating BERT intermediate embeddings based on LQAE input: LQAE encoder denotes concatenating LQAE encoder intermediate embeddings; LQAE encoder BERT denotes combining LQAE BERT and LQAE encoder together.
4.5 Model Variations and Ablations
In the following section, we evaluate different variations of our default model. We present these results in Table
In Table row (A), we experiment removing L.2 normalization when finding nearest neighbor code in vector quantization step. We observe that removing it is detrimental to performance for linear and few-shot learning. This observation aligns well with similar experiments in Yu et al. (28), which may be helpful for learning by providing better coverage over language codebook usage.
In Table row (B), we observe that using pretrained RoBERTa model leads to significantly better results than using randomly initialized language model, suggesting the important of incorporating the language prior in LQAE.
In Table&roo (D), we observe that, contrary to standard VQ-VAEs, introducing an entropy regu- larization quantized codes does not help. We hypothesize that this may be due the fact that the entropy regularization provides more beneficial gradient signal over the codebook rather than encoding embeddings, however, the codebook frozen for LQAE.
In Table 3 rows (E), we vary the weight of BERT loss a. We observe that using larger BERT loss weight improves linear classification but hurts few-shot classification. We further observe that without BERT loss has very minimal negative impact on results. This suggests that image reconstruction alone is sufficient for models to learn to map images to texts, further regularization through BERT loss may not help.
In Table rows (F), we experiment with using vector quantization before decoder input, such that decoder's input are codes from RoBERTa codebook. We observe that doing so no significant benefit. Therefore for simplicity opted not use in default model.
7

---

Table Comparison of variations of LQAE. The metrics are linear classification with BERT-like models on ImageNet, and sew- classification using GPT-3 on mini- mini-lmageNet.
Variation
Entropy
Trained BERT
I.2
BERT loss
Decoder STE % Code
GPT-3 size
Linear
2-way
S-way
Default
0.0
true
trse
0.001
falie
100
Davinci (175B)
35.60)
53.97
29.04
(A)
false
30.30
52.45
27.42
(B)
false
11.80
1.03
0.51
(D)
0.5
30.70
50.45
26.54
0.00
34,80
52.45
28.51
(F)
1.00
36,90
40.45
20.93
(F)
true
34,80
54.53
30.01
(G)
50
N/A
21.00
5.56
Curie(6. 7B)
N/A
46.55
22,80
(H)
Babbage(1
23.85
14.70
VQ RoBERTa w/ Davinci 175B
N/A
3.24
0.00
In Table rowws (G), we vary the percentage of LQAE codes used in GPT-3 based few-shot image classification. We do so by always keeping the first certain percentage tokens. While partially remove tokens reduce the amount LQAE tokens still perform quite well, suggesting that LQAE tokens may have redundant information. We further observe that keeping 50% or fewer leads to significant drop in performance.
information representing images, it is observed that keeping 75% of
In Tows (H), we vary GPT-3 model size from default largest 175B model (Davinci) to smaller models (Curie and Baggage). The results show that larger model consistently perform better. We note that LQAE with 6.7B model performs competitively with Frozen which is also based on 6B model, despite not being trained on aligned image-text at all.
In Table 4 rows (I), we experiment using assigning VQ-VAE tokens to RoBERTa codebook codes. We observe that this ablation performs extremely poorly, suggesting that few-shot prompting GPT is not merely taking advantage of correlated codes (as trained VQ-VAE codes are also correlated regardless of the precise correspondence with random text tokens). As result, although the text encodings learned may look garbled, they do indeed contain non-arbitrary language structure that GPT able leverage.
demonstrates the crucial role of a higher masking ratio for achieving optimal downstream performance. While standard language denoisers like BERT typically employ a masking ratio of 15%, our findings indicate that the LQAE performance peaks at approximately 50% masking ratio, highlighting the need for an increased level of masking.
3
100
Accuracy
27.6.
Few-ehot Acouracy
400
38.5
25.5
240
27.1
200
Figure 5: High mask ratio is crucial for LQAE results. (left): Linear classification result on ImageNet.
(right): 5-way and 2-way few-shot image classification results Mini-ImageNNe..
5 Related Work
Large language models (L.MM) have achieved remarkable success for many natural language under- standing tasks 7 Following this success, a large body of work aims to adapt language models to

---

multi-modal tasks. This stands in contrast to traditional multimodal models [00 29 inter alia] that are trained using extensive amounts of paired image-text data.
Finetuned Language Models. One direction of research to adapt language models to multi-modal tasks is to directly finetune the pretrained model weights. Tsimpoukelli et al. [23] directly finetune language models on visual tasks such as reasoning across discrete sequences and few-shot image classification, demonstrating that knowledge acquired from text transfer to non-linguistic settings. Similarly, Ziegler et al. (32 and Chen et al. 4 show that using a large pre-trained language model as decoder can improve a captioning performance under regimes with more limited training data. Cho et al. [5] proposes using open-ended text generation as an objective for task-general multi-modal models. Generally, these methods require undergoing pretraining on a large corpus of aligned text- image data, after which finetuning is performed for each specific downstream task. While these approaches have shown to produce excellent results on a variety of visual tasks, the resulting models are highly specialized and cannot learn new concepts, or adapt to new tasks with just a few examples, unlike our proposed method, which does so through few-shot prompting and in co-tonext learning with large language models. We accomplish this by first leveraging pretraining on single modal (unaligned) data and then only in-context learn text-image alignment only given a few aligned examples.
Frozen Language Models. Given the computational burden of even finetuning large language models, another direction of related work focuses on using frozen language models. Frozen [23] demonstrates that finetuning visual encoder features to align a frozen pretrained language model achieves non-trivial performance in few-shot image classification and visual question answering. Recently, there have been notable advancements in the field of using text models for image-related tasks, demonstrated by LLaVa 33 and MiniGPT4 [3I] In their respective approaches, these models finetune linear projection layer to align image features with text models. In contrast, our method establishes unique connection between image and text tokens using VQ, resulting in a fundamentally distinct approach. This orthogonal nature of our method sets it apart from previous approaches. An advantage of our approach lies in its superior parameter and computational efficiency, training the VQ module does not involve large language models (LLMs) nor large visual backbones (ViT), which reduces the overall number of parameters required. Additionally, this characteristic of our method renders it compatible with black API-based L.L.s, further extending its versatility and applicability.
6 Conclusion
In this work, we presented Language Quantization AutoEncoder (LQAE), a VQ-VAE style model based on BERT that learns to map images between image and text modalities by using pretrained BERT models. We demonstrated that by leveraging pretrained language denoising models, we can first learn an alignment between text and image data in an unsupervised manner without the use of any rext-image aligned pairs. Then, we can few-shot prompt pretrained large language model with as few as 1-5 pairs text-image examples of our learned text encodings of images to perform classificationand visual question answering, achieving accuracy competitive or exceeding prior works which pretrain on millions of pairs. Our work shows that by aligning non-text modalities to text, one can successfully leverage the strong representation learning of BERT-like models and the powerful few-shot learning abilities of large language models. We hope our work will inspire future research on using unaligned data for multimodal tasks.
Limitations and Future work.
Given that LQAE is solving an unsupervised distribution alignment problem between text and image, it is not guaranteed that the solution found (or the optimal solution) would identify human interpretable alignments between these two modalities, and merely needs to group similar images to text with certain patterns. In this work, seek to address this issue by realigning using GPT through providing few-shot true text-image alignment pairs. Although this alignment allows to solve downstream visual tasks such as classification, the text may still not be human interpretable, which may be of vital important to some domains such as healthcare.
representation
In addition, due to a lack of compute resources, we found it difficult to scale up our models. There are two dimensions of sealing that could lead to very interesting outcomes. One using bigger image encoders and decoders. Another one is using bigger BERT-like model. We hypothesize that both will

---

improve results significantly because larger BERT-like model's has more text knowledge and larger image decoder means more model capacity to decode images.
Lastly, although our work focuses primarily on learning unsupervised alignment between text and image modalities, autoencoder to one modality to map to a second modality. Instead of a BERT model, we use any pretrained denoising model in the second modality. We believe this be very promising direction with many potential cross-modal applications in a wide variety of fields.
method can fully generalize to two arbitrary modalities we can train
Acknowledgment
This research has been partially funded by the Office of Naval Research under grant N00014-21- 1-2769. We express our gratitude to the BAIR communities for their insightful discussions and feedback. Our appreciation extends to the anonymous reviewers and area chairs for their constructive critiques. We are also thankful to Google TPU Research Cloud for providing us access to their TPUs.
References
[1] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: visual language model for few-shot learning, arXiv preprint arXiv:2204. /4/98, 2022.
[2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[3] H. Chang, H. Zhang, J. Barber, A. Maschinot, J. Lezama, L. Jiang, M.-H. Yang, K. Murphy, W. T. Freeman, M. Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. arXiv preprint urXiv2231.00700 2023.
[4] J. Chen, H. Guo, K. Yi, B. Li, and M. Elhoseiny. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18030-18040, 2022.
[5] J. Cho, J. Lei, H. Tan, and M. Bansal. Unifying vision lamluuuaee tasks via text generation. In International Conference on Machine Learning, pages 1931-1942. PMLR, 2021.
[6] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. 2022.
[7] Devlin, M.-W. Chang, K. Lee, and Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810000800 2018
[8] A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 1 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[9] R. Gal, O. Patashnik, H. Maron, A. H. Bermano, G. Chechik, and D. Cohen-Or. Stylegan-nada- Clip-guided domain adaptation of image generators, ACM Transactions on Graphics (TOG), 41 (4):1-13, 2022.
[10] X. Geng, H. Liu, I. Lee, D. Schuurmans, S. Levine, and P. Abbeel. Multimodal masked autoencoders learn transferable representations. arXiv preprint arXiv:2205. 14204, 2022.
[11] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
[12] A. K. Lampinen, I. Dasgupta, S. C. Chan, K. Matthewson, M. H. Tessler, A. Creswell, J. L. McClelland, J. X. Wang, and Hill. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02322 2022.
10

---

[13] H. Liu, C. Li, Q. Wu, and Y. Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08885. 2023.
[14] Y. Liu, M. Out, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zeulemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1eXiv:9907.11660 2019.
[15] C. Meng, Y. Song, Song, J. Wu, J.-Y. Zhu, and S. Ermon. Sdedit: Image synthesis and editing with stochastic differential equations, arXiv preprint arXiv:2108.01072 2021.
[16] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprini arXiv:2202. 2837, 2022.
[17] L. -..uugggg.... J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang. S. Agarwal, K. Slama, A. Ray, al. Training language models to follow instructions with human feedback. arXiv preprini arXiv:2203.02/11 2220022555, 2022.
[18] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[19] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[20] S. Ravi and H. Larochelle. Optimization as a model for few-shot learning. 2016.
[21] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 (11:):11122555 15252, 2015.
[22] P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned. hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018
[23] M. Tsimpoukelli, J. L. Menick, S. Cabi, Eslami, O. Vinyals, and F. Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34: 200-212, 2021.
[24] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning, Advances in neural information processing systems, 30, 2017.
[25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, I. u. Kaiser, and
I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. htRpshtts:://// htt.:ttsss/pprccddngg...ott neurips Paper. pdf
ccc/paper/201/Ffile http://000///ee/33f6e2222232////
[26] O. Vinyals, C. Blundell, T. Lillicrap, D. Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
[27] A. Webson and E. Pavlick. Do based models really understand the meaning of their prompts? arXiv preprint arXiv:2109.01247, 2021.
[28] J. Yu. X. Li, J. Y. Koh, H. Zhang, R. Pang. J. Qin, A. Ku, Y. Xu, J. Baldridge, and Y. Wu. Vector-quantized image modeling with improved vggan. arXiv preprint a7X1v:2110.04627, 2021.
[29] Yu, Y. Xu, J. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, Ku, Y. Yang, B. K. Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
[30] Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, Chen, C. Dewan, M. Diab, X. Li, Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and I.. Zettlemoyer. Opt: Open pre-trained transformer language models. arXiv preprint arXiv: Ar05v-2205.8 2005002205.0066,,, 2022.
11

---

[31] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 2023.
[32] Z. M. Ziegler, L. Melas-Kyriazi, S. Gehrmann, and A. M. Rush. Encoder-agnostic adaptation for conditional language generation. arXiv preprint arXiv: 190.00908 2019.
12

---

A Few-shot MinilmageNee
The dataset construction is based on MinilmageNet (26], following the method of Tsimpoukelli et al. 23 A 256 x 256 image size is used so that the ViT encoder generates 256 tokens.
We use the same subset of ImageNet classes, referred to S, that was utilized in previous research on meta-learning with Minilmagenet [20] 23]. All of the images used come from the validation set of ImageNet. We follow the process used in Tsimpoukelli et al. (33 generate 2-way question with n inner-shots, as follows:
1. Select two classes, O1 and cg2 from a set S.
2. Choose n images, of 1... of n + 1, from class and n images, t² 1...... v from class Cg-
3. Combine the two sets of images into sequence of 2n support images, [uft, uf" ... , v.].
4. Assign label: The label used is the first class name from the ImageNet dataset.
B Examples of Encoded Image
Figure 6: Examples of image-to-text generation using our method. The images are sampled from the ImageNet dataset. Left. Randomly sampled image from ImageNet. Right. Model-generater text based on the image.
Imput Image
Decoded Encoder Output Tokens (Words)
Output Image
Why Butterlyy UK tobaceu PE appraisal COMAN Dr Janeeti Bazmos Future Re Amberoothouth UK EssexCHOuff Rem tan MillsORE Avery Ellie682 NewmanReberi Robinson poundooth
rye McD Nicholson AISSATHAATERKKINNG M. 3oothunktam3300app Tr
1952
Robomors21 mouu Melbourn DoomnosaeeeDvis Stewart pound Smithounds CHANT
Berry
Romieraniice Taylar spylessness 198iocial Cump RECAT
Tetmom lishID tieen pointer Chapman BrisollothbereggAALL.
OUTHEAD Joseph JackOTinss
Scort Birmingham Kurt gent Seegpliving/Fandnngnnnoo
Pearl Art PS 317 Automarksation Dieletion PaintRam Clifford Polaris Gary oullincav 1986 BeaeonllnnEaaaaodreeeee Radotarty Byme Montgomeryoseroneee cant th anxiety Mull 132 gingerney Bradley SampO TOMT IMINTERI FY Top Th Bortonerri
1941 Jeseph
ArnollonedOOHNPER THEMKER Crane license Higgins Bermarlonculp May Banana Sens Lowe Sue 153 Research
Lennon Manning Cakearmeyyson Malcolmenter moceuchmachhne Mothers undert coest
Suatonsuperene
Rutherford Walkinsenta tissce Quim Tiboa Cooper
Wallace folded Totcommitte
Morris Thailand festala
Insovationll Bir Proderick Fag 1700 Bradley Burton cop Moore
Oilyy Michelle Trever S6sts bombs funciman Robbic kan October
Firesini Stepheninth prints Fundsotit Nicholsorord
sausage Lilihimet frrmm Stalnn ANNdomin Gill
amend Elleninated elvesmal unbeat Gill Air Mass
masseinnN.Tott NoT Onlackson Ang.ak/nnnnnnnWWWW/W WNSSSe/WIIISSSIIIIIIII illumination Android Les coun Faven band Hard Ed Colts Tid dor SeottnningTTTR Ltah misioostH:ENCY FINAL, THEYlenc
sights card falls infumschen drinksugisgimosp riflemast Whit Bonnielict weaknessesenne LancisodesP shiny Bangladesh
FilylPintus Collull frog Roduchiniremon
interpreted www.iiiininnannnieeee
Kiss loft Plato Gh Mono Autaul Lem mipple weird63 digull archelIPL. trickisan
remorkably BannkaallyAANiaaa GANTery Bristel Teryrell oo textspec intimZE: Ph
Live Cathedralish Paul Gonn Diness inbammm Indigo
gingerPhoenix SweetCraigards fiveots Canson Franklin
ExraDesttl BSC Randall Angel Baltimore192
FmokliippomSSU UPP Hamilton legions Gong Michael
poisoppl Valeri Francoiiiss
excruciaingiaciiman Briaaeeaamininn Clarkique Church ohe 235 Gffze Puulibsokinnlankke Extremenddeneenns liquor
Gray unchvoltidaaard Dingadandisk escaped sophalt City Tun blocked Levy Eastern 24lkeelelle Northwestern colkett Tanras
Max Dar Marriont aath burg Ut Astenton Stewart
Pick Ken Kan taxissur Hayden Cyr stimulation
tunckeley ParkwaySmull Chaseirrass Toronto burnunck Jocater Patscucks
13

---

Figure 7: Examples of image-to-text generation using our method. The images are sampled from the ImageNet dataset. Left. Randomly sampled image from ImageNet. Right. Model-generated text based on the image.
Input Image
Decaded Enewder Output Tokens (Words)
Output Image
expuisions Heatheniful Mauritbi Transaction RS Tournament
Ryan Ryu Kes frenzy Dhadelphia Sou bations fontsagers
PialConnell HSBCicro Nomoco Smy Vagantemic Twiner
Pebwlankkspomis reply Soc Vaughnt Inion Will Tammy
PorthanCal plysb defer Cucha bERO Tok Comp" Rob mountingttonaal Cabe nan Ryan LinksLL Cory Speneer sched tops Carz bless Jan BD BB seventh Arts Es ooders Eva commanddenAnddy Victor Galaxy grin pourod Boz bowl Becs Hill Andy StreamokeMiller Milo Brewery Jeremy fishes Third Gii Jolly Hawks meask/Ta To coumamentberrot
Shop Ken Ken milk
jacket XTssssss hoeath todkler thope chopping Hogther Hundooth Homs Mam32 Spark subsidiary Disonko&C HY Reilly Pwrug Brookenny Silver undeniably Mog Harrison aunriverpool
LeeGrey porieerpollPammiiie Pammbielli Hock William Laura 184
Space Reynolstern galeway Boots BrookBiton Piaf Gary
herrendousates
Rocre engineers
guitar froe Shanamn Mannit 105em Reserved EGji Meadowales 31 Mull South Tom BermettaniimmmmmA Sw Hood KR milk(K Smoke Dil Bros Mienttra Thorther
Old k county Wood
Microsisurty Neweasile bloom Brond Monn Hol HolEnt
SilchooArundMMalll Bry Meganheddar55/ 151 ATP466artz
physics Burn Iromuan lay dst datGm
to Mania Corporation FresnepaeenSAA Chrysler Holcings Signal
Mijesty Fmerson Trinktad Jiventas Pocition SucTrump oeb
Trinidad ascended Camuais Kusovo Preferences
PetroleumXXSeriil Umuador mund Aircraft Es Sabetic Hyhrid
ASP robor Hut elscemmnicziiossheliiitt WhassApp
Hispunies comprehens Superior aleptionjoicc116
Soundors
Incre precip blaff mulaca GL GL toburco lump Tramerrptt Chero
LLC precip 1983 Exampleted whaleahlenburgneinnae Womanepspe Fiftychin Sohnnhyrccy
Church Posk
Dale Lad John/ohm aimed BusenJohmneyBuren BurenAlan John Dale
Morgan Jarreti Jareellanyaar Attorncy Marthassrry Jareel
Jareett 11 11 Jarrett Samantha Jureet Burke Mitchell
Mitchell Alexander Andy Mischell Lilly
Kelly Allen
Lestic Lestic Leslic Allenany PP Ibs Wem 62omm 7addr
A89292 lbs Bub pel depesits sonsoancseCON
RochestesCON 10 Company
WillS
HokiingaiissnionalCAAAA ICANICASSS vegetableacio Jeunne
commbinoidpres uncupl Vie 0162019Nss
Transciptrraaaanccaee Au Quebec Qucbecques
Afrcc se su See du les connahis BA Fall Monaco nop hydrobal
prin Lelsppeli Ts Cammated CanalOr Collee timed
appearedpeg Colombia python tray Columbia Smooth
paith pul cafe nroma Sutton Al Simony Testingima Simpson
Loasundor neighbor Oct Mens Slate apost brun CLS ET dncuments equality Momoso blacks Monocso prompi
MMuude Tekcomemp Sar3aaalnCommmmnse
14


Reviews:
[{"review_id": "53eZXpq6oE", "reviewer": "Reviewer_FHFJ", "summary": "The proposed approach, LQAE, addresses the lack of visual grounding in large language models by aligning text and image data in an unsupervised manner. It encodes images as sequences of text tokens by quantizing image embeddings using a language codebook and reconstructing the original input with a masked version of the quantized embeddings and BERT. LQAE enables few-shot multi-modal learning, outperforming baseline methods in tasks like image classification and visual question answering with as few as 1-10 image-text pairs.", "questions": "See weaknesses.", "limitations": "The authors have discussed limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a simple but effective method to align text and image representation with pretrained models.\n2. The proposed method allows image to be used in the same way as text, which could be adopted for downstream tasks.\n3. The method doesn't require image-text pairs by unidireictionally align the image to text embeddings.", "weaknesses": "1. The proposed method may lack explanability compared to methods that use contrastive learning to train on text-image pairs.\n2. I'm concerned about the technical novelty as there have been a lot of works trying to align and mix visual and text tokens, including BEiT, Parti, etc.\n3. The trained representation cannot transfer between models, e.g., between RoBERTa and BERT.\n4. The downstream tasks are focused on few-shot settings. How does LQAE perform when doing text-image fine-tuning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We address your concerns below.\n\n> The proposed method may lack explainability compared to methods that use contrastive learning to train on text-image pairs.\n\nInterpretability is indeed an existing flaw in the method, as we do not use text-image pairs. However, we believe that it is a promising direction in future work to potentially train LQAE models with mixed unpaired and paired data for more sample (paired data) efficient learning than existing methods. In addition, LQAE shows promise for a more general framework of leveraging single-modality denoisers to align arbitrary multi-modal pairs, and can be useful in scenarios where unpaired data is much easier to acquire than paired data.\n\n> Technical novelty over prior works that try to align and mix visual and text tokens and does text-image finetuning, including BEiT, Parti, etc. \n\nWe would like to emphasize that the novelty of our work over prior methods centers around enabling vision-language capabilities without requiring text-image pairs - in contrast to nearly every prior vision-language work (Frozen, Flamingo, GPT-4, Parti, BLIP, etc.). In addition, our method does not require fine-tuning, or backpropagating through an LLM, which can be expensive (e.g. 175B LLM), or impossible (no weights available and only through API / prompting). Although BeIT does not require text-image pairs, the method is unimodal in images, and does not contain any language information, unlike LQAE.  \n\n> The trained representation cannot transfer between models, e.g., between RoBERTa and BERT.\n\nYes, this is a limitation regarding our model. However, to the best of our knowledge  we do not believe that this is a common property of most vision-language models, such as Frozen or GPT-4, where both vision and language representations are model / finetuning specific.\n"}, {"review_id": "EBLwLLGgOV", "reviewer": "Reviewer_Z64R", "summary": "This work introduces a method for unsupervised image-text alignment named LQAE. The central concept revolves around utilizing a VQ-VAE framework, replacing the conventional codebook with frozen token embeddings extracted from a Language Model (LM). This approach aims to establish meaningful correspondences between images and their associated textual descriptions. An additional step of masking and filling after decomposing the image into token embeddings is added to ensure consistency and improve the likelihood of the generated texts. There is a comprehensive range of experiments, including an insightful ablation study that thoroughly analyzes each component's importance and respective roles in the alignment process. Notably, the results are better than alternative solutions, even in challenging few-shot settings. By harnessing the power of the GPT model to extract patterns from the text associated with each image, the proposed method achieves remarkable performance in accurately classifying the images.", "questions": "1) During the analysis of the generated text, it appeared to me that most of the generated words were named entities. Do the authors have any insights into why the model tends to use named entities more frequently than standard, more common words? Is the model possibly relying on named entities due to their higher specificity, allowing for greater diversity and (possibly) orthogonality compared to other words, thus enabling more detailed descriptions? \n\n2) There’s a typo on line 163, an extra \"is\".\n\n3) Do the authors plan on releasing the code upon acceptance?", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- One of the notable strengths of this paper is its comprehensive and detailed ablation study, which consistently enhances its soundness. The authors meticulously analyze and dissect the different components of their proposed method, providing valuable insights into the architectural choices. The thorough investigation of the encoding process and its subsequent integration with the language model is particularly commendable. \n\n- Additionally, the paper is well-crafted and exhibits a diverse range of experiments that effectively validate the proposed method. The method itself showcases architectural innovation, setting it apart from its competitors in the field, but since I’m not very familiar with the related literature, my confidence in this regard is not high.", "weaknesses": "Regarding the experimental setup, a weakness I see is that the results don’t have statistical significance reported, given that only a single run for each setting is shown. This somewhat limits the assessment of the robustness of the method. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We address your concerns and questions below.\n\n> Are results statistically significant with one run?\n\nWe have conducted multiple runs on the few-shot prompting experiments. Across three independent runs with temperature = 0, the average scores are 29.039, 29.038, 29.039, indicating the evaluation is reliable with very low statistical variance. Training LQAE should be consistent across runs on such large datasets, due to time constraints, we were not able to finish multiple runs of LQAE training. However, we plan to incorporate such results in the final version. \n\n>  Do the authors have any insights into why the model tends to use named entities more frequently than standard, more common words?\n\nAs the reviewer has hypothesized, we do believe that the tendency to use named entities is indeed to the long-tailed nature of the subset of the named entity vocabulary - to access a greater number of and more diverse codebook vectors for better reconstruction.\n\n> Do the authors plan on releasing the code upon acceptance?\n\nYes, we plan to release all code on acceptance.\n"}, {"review_id": "pMZBQYETQO", "reviewer": "Reviewer_wNXC", "summary": "The paper proposes Language-Quantized AutoEncoder (LQAE), a representation learning approach that quantizes the latents to the codebook/vocabulary of a language model, here, a BERT model. During training, the quantized latents are masked and decoded by the frozen BERT model to implicitly impose language structure onto the learned latents. Once the vision encoder is learned on unpaired image-only data, the latent representation can be used for few-shot prompting of LLMs since the latent can be directly translated to text.", "questions": "1. How do gradients propagate through the network? Are straight-through operations applied when an embedding is replaced by a codebook entry? E.g., how does the gradient of the reconstruction loss travel back to the image encoder? Similarly, how does it work for the MLM loss? Since there seems to be no stop-gradient applied on the last term of the loss Eq. I presume the gradient travels both to the \"target\" $z$ as well as to the inputs $z_m$ with a stright-through applied to reach the image encoder? Please correct me if this understanding is wrong and elaborate.\n2. Intuitively it makes sense that BERT should guide the embeddings $h$ to follow a language structure. In theory, however, what exactly prevents the embeddings to \"get stuck\" and collapse always to the same codebooks? In VQ-VAE this problem is solves by re-initializing unused codebooks to be close to frequently used ones, but this is not possible here. What guarantees that a diverse set of tokens/codebook entries are being used?\n3. On first sight, Figure 3 looks like two duplicate figures. Switching the example might help better illustrating the difference in the two task configurations.", "limitations": "- In the end, it comes down to practical application and efficacy of the proposed LQAE. On paper, it has the nice property of not requiring paired data. In practice, this can rather become a limitation as models that make use of said large-scale paired data can learn relations more effectively and ultimately outperform this approach (e.g. [A]). It would be interesting to know to what extend scale of the model is the limiting factor here.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The proposed idea of using a BERT model to guide the latent representation to have a language-like structure is novel and interesting.\n- Extensive ablations evaluate the model from different perspective and allow insights into the model decisions.\n- LQAE is competitive and performs better on downstream tasks like few-shot classification and VQA compared to Frozen [23]", "weaknesses": "- One key motivation is that paired data is more costly to obtain than single modality data. While true, large-scale paired data such as LAION is cheap to collect. Downstream task performance is far from models trained on paired data such as (Open)CLIP.\n- Frozen is portrayed as the image-text pretrained model to beat although more recent models such as Flamingo [A] outperform it by a large margin.\n- The task generalization that can be achieved with the proposed model could have been highlighted more by evaluating on more tasks (see [A] for a reference).\n- Some details of the method are not clear, specifically around the loss function and gradient propagation. See questions below.\n\nReferences:\n[A]: Alayrac et al., Flamingo: a Visual Language Model for Few-Shot Learning, NeurIPS 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your review. We address some of your questions and concerns below.\n\n> One motivation of LQAE is that paired data is more costly to collect, but paired data is still easy to collect at a large-scale such as LAION. \n\nWhile one can source paired data from the internet such as LAION, it suffers from low text quality and needs extensive cleaning for well-aligned text-image pairs. Unpaired image data is much more widely available. Finally, future works can consider combining paired data and unpaired data in LQAE, in a semi-supervised manner, to leverage both costly paired data and unpaired data. \n\n> Flamingo outperforms both Frozen and LQAE, and has more experimental evaluations\n\nAs shown in the Flamingo figure 1, the main two experiments there are VQA and few-shot prompting. In this paper, we include both experimental evaluations. While Flamingo is capable of doing more tasks than LQAE, we note that it is significantly more costly to learn than LQAE, as Flamingo finetunes LLM weights, while Frozen and LQAE keep LLM weights frozen, and would not be a fair comparison.\n\n> How does the straight-through estimate work with reconstruction loss? \n\nYes, straight-through estimates are applied during the quantization step. The reconstruction loss back-propogates through the decoder, last layer BERT features,, quantization / straight-through, and encoder. The MLM loss backpropagates through the BERT, quantization / straight-through, and encoder.\n\n> How does LQAE prevent codebook from collapsing?\n\nIntuitively, our high masking ratio helps prevent codebook collapse during training. In some sense, it is similar to dropping out codebook vectors and prevents over-reliance on specific codes.\n"}, {"review_id": "UHFNNJuCYI", "reviewer": "Reviewer_Vpea", "summary": "This study focuses on the translation of images into a sequence of discrete language tokens, aiming to enhance the understandability for language models, such as GPT-3. By achieving this, the vision-language tasks can be seamlessly translated into LLM tasks using zero-shot or few-shot learning techniques.", "questions": "would be better to evaluate also on other tasks such as vision-Language tasks such as VQA or captioning. And compare with MiniGPT-4, and see if your method has a better visual encoding performance.", "limitations": "adequately discussed the limitation", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. the paper has a very clear presentation, and the paper writing is good. It well demonstrates their method and experimental results\n2. the paper has a good motivation, and it attempts to solve the image understanding tasks by translating the images into quantized tokens for LLM understanding.", "weaknesses": "1. the experimental results do not look strong enough. the best performance only reaches 54 top1 accuracy for 2-way image classification. and the model performance does not scale up with more training data. training on 75% data performs worse than 50% training data. \n\n2. The idea does not seem very natural.  convert the vision features into quantized tokens through Frozen RoBERTa codebook will lose a lot of visual information. Compared to those models such as MiniGPT-4 or LLaVa, they translate the images into soft prompts that LLM can be more easily understandable  through CLIP vision encoder.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We address your concerns below.\n\n\n> Q1: Top1 accuracy is only best at 54%, and the model does not scale with more training data \n\nWe believe that there may be a misunderstanding. If you are referring to Figure 5, the x-axis is the masking ratio during training, and not the percentage of training data. We apologize for the confusion, and will update the figure caption to clarify. These results reflect similar to those trends observed in other related papers (e.g. MAE), where there is generally a “sweet spot” in terms of optimal masking ratio. Regarding 54% top1 accuracy, the value for this metric is computed as an average over several different evaluation settings. Table 1 shows a more fine-grained breakdown for the 54%, where we can see the average is greatly pulled down by outliers. Generally, LQAE can achieve 65-70% accuracy on the task. \n\n> Q2: The idea does not seem very natural over models such as MiniGPT-4 or LLaVa\n\nAlthough LQAE diverges from the standard paradigm of learning vision-language models, it introduces several key benefits over existing approaches. Firstly, LQAE is far more sample efficient (text-image pairs) compared to prior methods, which are generally trained on millions or hundreds of millions of paired examples. LQAE’s pretraining phase only requires unpaired data, and only a few examples (1-5) are needed during inference. This property is especially useful in domains where paired data may be potentially difficult to collect or curate. Secondly, LQAE presents a more computationally efficient method than prior works, as we do not require backpropagation through an LLM that could be potentially 10-100Bs of parameters. Promising future directions may involve scaling up our model, or investigating sample efficiency trade-offs by incorporating small amounts of paired data during training. Overall, we believe that our work provides a different methodology around vision-language learning that would be of interest to the NeurIPS community as a whole.\n\n> Q3: Comparison against MiniGPT-4 or LLaVa\n\nWhile we do discuss both of these works in the Related Works section, as per Neurips guidelines, we consider these two papers as concurrent works - as both papers were released April 17 and April 20 and within the specified 2 month period prior to the Neurips submission deadline. In addition, we would like to clarify the difference in data and computational requirements between LQAE and LLaVa / Mini-GPT4 as outlined in our response to Q2, which makes them less meaningful baselines to compare against."}]

Author rebuttal:

Meta-review: This paper proposes LQAE for unsupervised image-text alignment. The central concept lies in the use of a VQ-VAE framework, replacing the conventional codebook with frozen token embeddings extracted from a language model. The authors have done a nice job of rebuttal. After rebuttal, it received scores of 5666, ranging from borderline accept to weak accept. 

Reviewers are generally happy about the paper, commenting that (1) the proposed idea is interesting and novel; (2) the experiments are comprehensive with detailed ablation study; (3) the paper is well written and presented. The additional results on MiniGPT-4 etc. makes the evaluation more complete. On the other side, reviewers also commented that (1) the idea of converting vision features into text tokens seems not very natural, and this process will lost a lot of visual information. (2) The proposed method lacks explainability, and by the end, web-crawled image-text data is not that hard to acquire, weakening the motivation of the paper. 

Overall, the proposed idea in this paper looks interesting and novel, and experiments are solid. On balance, the AC thinks that the merits of the paper outweigh the flaws, and therefore, would like to recommend acceptance of the paper.
Decision: Accept (poster)