LG-VQ: Language-Guided Codebook Learning
Authors: Guotao Liang
Baoquan Zhang
Yaowei Wang
Yunming Ye
Xutao Li
Wanghuaibin
Luo Chuyao
kolaye
luolinfeng
Venue: NeurIPS 2024 poster (2024)
Keywords: Codebook Learing
VQ-GAN
Vector-quantized Image Modeling
Primary area: generative_models
TL;DR: We utilize pre-trained text semantics to guide the codebook to learn rich multi-modal knowledge to improve the performance of multi-modal downstream tasks

Abstract: Vector quantization (VQ) is a key technique in high-resolution and high-fidelity image synthesis, which aims to learn a codebook to encode an image with a sequence of discrete codes and then generate an image in an auto-regression manner. 
  Although existing methods have shown superior performance, most methods prefer to learn a single-modal codebook (\emph{e.g.}, image), resulting in suboptimal performance when the codebook is applied to multi-modal downstream tasks (\emph{e.g.}, text-to-image, image captioning) due to the existence of modal gaps.
  In this paper, we propose a novel language-guided codebook learning framework, called LG-VQ, which aims to learn a codebook that can be aligned with the text to improve the performance of multi-modal downstream tasks. Specifically, we first introduce pre-trained text semantics as prior knowledge, then design two novel alignment modules (\emph{i.e.}, Semantic Alignment Module, and Relationship Alignment Module) to transfer such prior knowledge into codes for achieving codebook text alignment.   
  In particular, our LG-VQ method is model-agnostic, which can be easily integrated into existing VQ models. Experimental results show that our method achieves superior performance on reconstruction and various multi-modal downstream tasks.

LG-VQ: Language-Guided Codebook Learning
Guotao Liang!,, Baoquan Zhang" Yaowei Wang', Xutao Li', Yunming Ye', Huaibin Wang'
Chuyao Luo', Kola Ye', Linfeng Luo
Harbin Institute of Technology, Shenzhen, ?Peng Cheng Laboratory, SiFar Company [lianggt, wangyw]@pcl..ad ac.
(238951062, 225051022)@stu.htt.edde {baoquanzhang, lixutao, yeyuaming3-hit..ddu.. {luochuyao.des dalian, kolaygm, 11f100810002055ggmala..cc com
Abstract
Vector quantization (VQ) is a key technique in high-resolution and high-fidelity image synthesis, which aims learn codebook encode an image with a sequence of discrete codes and then generate an image in an auto-regression manner. Although existing methods have shown superior performance, most methods prefer to learn single-mddal codebook (e.g., image), resulting in suboptimal performance when the codebook applied to multi-modal downstream tasks (e.g. text-to- image, image captioning) due to the existence of modal gaps. In this paper, we propose a novel language-guided codebook learning framework, called LG-VQ, which aims to Iearn codebook that can be aligned with the text to improve the performance of multi-modal downstream tasks. Specifically, we first introduce pre- rre-trained text semantics prior knowledge, then design two novel alignment modules (i.e., Semantic Aligmment Module, and Relationship Alignment Module) transfer such prior knowledge into codes for achieving codebook text alignment. In particular, our LG- VQ method is model-agnostic, which can be easily integrated into existing VQ models. Experimental results show that method achieves superior performance on reconstruction and various multi-modal downstream tasks.
Introduction
In recent years, with the growing development of various
task scenarios 37
36 38
unified modeling of visuals and language has sparked considerable interest. Vector Quantization (VQ)-based image modeling technique, exemplified by VQ-VAE (43) and VQ-GAN 9 has emerged as pivotal approach the realm of unified modeling. The VQ methodology two-stage generation paradigm. In the initial stage, trainable discrete codebook is employed to quantize continuous image features into a discrete token sequence to finish the reconstruction task. Subsequently, the codebook is utilized for various downstream tasks by generative models 42 37]
4S typically follows
Learning a robust codebook during the initial stage is crucial for optimizing performance down- stream tasks. At present, lots of VQ methods have been proposed to achieve robust code represen- tation 15 14 12). For instance, VQ-GAN [ introduces an adversarial training loss to learn a perceptually rich codebook. Some other works consider improving the codebook representation from the perspective of addressing the problem of codebook collapse [53] 52).
Although existing methods have shown superior performance, most methods only focus learning single-moda codebook contains more low-level information (e.g., image's 's pixel, edge, and texture), resulting in suboptimal performance when the codebook is applied to multi-modal downstream tasks
CCoresponniig Authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(e.g., text-to- toae [37, image captioning [36] VQA (24)). That is because the codebook lacks
high-level semantics and the existence of modal gaps.
To address the above issue, we propose a novel codehook learning method (ie., multi-modal codebook learning), called Language-Guiin Guided vQ (LG-VQ). The novelty lies in utilizing pre-trained text semantics as supervised information to guide the codebook to learn abundant multi-modal knowledge.
Specifically, we first employ cross-modal pre-trained model (i.e., CLIP (2] to encode text semantics. Then, propose two novel semantic supervision modules to transfer the text semantics into codebook, i.e., Seman- tic Alignment Module, and Relationship Alignment Module. Within the semantic alignment module, we enhance the consistency between the semantic repre- sentations of the codebook and text through global se- mantic alignment and masked text prediction. On the other hand, simply aligning the text and codebook in the holistic semantic space cannot satisfy more com- plex reasoning tasks like image captioning and VQA. Inspired by some VQA techniques 226 40 24, the se- mantic relationships between words play a very impor- tant role in various tasks of natural language processing (See Fig. T.. Based on this fact, we further propose to transfer the semantic relationships between words into codes to achieve better alignment between the codes and words. Such a text-aligned codebook helps alleviate modal gaps and improve codebook performance on cross-modal tasks.
woman
holding.
racket
graas
Q: What is this woman holding?
Figure 1: To answer the question, one not only needs to identify "women" and "racket" but also understand the semantic relation- ship between them ("holding").
The contributions of this work are summarized as follows:
We point out the limitations of existing methods in learning an expressive codebook since they learn single-modal codebook. We propose novel multi-modal codebook learning method, named LG-VQ, which can enable the codebook to effectively retain fine-grained reconstruction information while aligning with the text.
Resorting to pre-trained text semantics, we propose two novel semantic supervision modules, i..., Semantic Alignment Module and Relationship Alignment Module, effectively learn text-aligned codebook. The advantage of such alignment modules is the abundant context and relationship semantics contained in pre-trained text can be sufficiently leveraged for enhancing multi-modal codebook learning.
We conduct comprehensive experiments four public datasets, which shows that IG-VQ method outperforms various state-of-the-ar models on reconstruction and various cross-modal tasks (e.g., text-to-image, image captioning. VQA).
2 Related Works
2.1 Vector Quantization for Image Generation
Vector quantization (VQ) is designed to learn a codebook, which aims to encode continuous image features into discrete sequence. Then, the learned codebook can be utilized for various downstream tasks. Oord et al. replaces the prior distribution of Variational Autoencoder (VAE) with discrete deterministic distri- bution (i.e., codebook). To further improve the performance of VQ, various models are proposed to learn more expressive codebook ( 48 2 17 15 2T For example, VQ-GAN [9] addresses the issue of image blur generated by VQ- VAE through the introduction of an adversarial training loss. However, the above methods do not tackle the codebook collapse issue. To address the issue, many novel methods are proposed from the perspective of regularization (33,, codebook update [3]] codebook transfer [5I). Recently, inspired by the large language models (LLMs), instead of mapping images to the visual code tokens, some works attempt map the images to the word tokens of L.MMs by viewing images as "foreign languages" " [2 50, (66). However, because of the inherent differences
4 first propose a novel VQ method called VQ-VAF. This method innovatively
between vision and language, these works have difficulty assigning correct semantic words to images.

---

Emcy
Languge-Guiie Guidee
VQ-VAF
Input
Cross Anterton Decoder fa.6."
Emcoder E),
Quantize (0)
Codchook
Information
Vianil Fecosle
Dag
Decoder
The
gel
cariigs
86
Output X
Figure 2: The overall architecture of the proposed LG-VQ method. The right part of the figure the basic VQ-VAE module, left language-guide module, which consists of three losses: global semantic aligmment (Cysa), masked text prediction (Emsp), and relationship alignment supervision (Cras). Here, pre-trained text information guides discrete code tokens of image learning rich semantic knowledge based on three losses.
Compared with the aforementioned methods, approach focuses feature space (i.e, learning text-aligned codebook). We use pre-trained text semantics to supervise the codebook learning. The advantage is that the rich semantic information from the text can be fully exploited for more robust codebook learning so that the codebook can not only retain more reconstruction information but also be able understand and match text. More importantly, our method is model-agnostic, which can be easily integrated into existing VQ models.
alignment in
2.2 Vision-Language Representation Learning
Vision-language Pre-training (VLP) aims to learn multi-modal representations from large-scale image- text pairs that can improve vision-language downstream tasks, for example, VQA[[] Early methods such LXMERT 41). UNITER (4] employ pre-trate pre-trained object detectors to extract image region features, and fuse image features with text by a cross modal encoder to achieve the lision-language representation learning. Although these methods achieve superior performance on downstream tasks, they require high-resolution input images and pre-trained object detectors. To remove the object detectors, large number of researchers focus learning separate representations for image and text [32 16 18]. For instance, CLIP 332 learns robust representation for each image and text using contrastive learning based large-scale image-text pair data.
In this paper, we propose to employ pre-trained text semantics as supervised information to guide codebook learning. Its advantage is that abundant multi-modal knowledge contained in text can be fully leveraged for robust codebook learning. Additionally, we design novel relationship alignment module to inject semantic relationships between words into codes.
3 Methodology
3.1 Preliminaries: VQ-VAE
VQ-VAE 43 as pioneering work the VQ research domain, aims to learn discrete codebook to encode images into discrete token sequences through an Encoder-Decodes framework. As illustrated in Fig. rightt the VQ-VAE consists of visual encoder E., (-) with parameter 8e, token decoder Do. (.) with parameter 0a, a quantizer Q(-). and a codebook defined as = {ex 1K k=1 that consists of learnable K entries Eą E IRd, with dimension dg. Given an input image æ R fx WxX XC where H,
W. and C' represent the height, width, and channel of the image respectively. The visual encoder (.,(.) learns to convert the original image into grid features 2 Fo, (a) E T # w. xd, and f the down-sampling factor. The quantizer Q(-) looks up the nearest neighbor in the codebook for each

---

grid representation 2, E E. in 2, using the following equation:
(()) = argmin 2| ex
cxE2
The token decoder Da (-) is used to reconstruct the original image by H (D) where Z is
discrete code tokens of whole image obtained by Eq. [1 During training, the visual encoder Eg, (+), codebook 2, and token decoder Da. (-) are jointly optimized by minimizing the following objective:
Lug = | 2 || |s (a)] 21 + - Ee. (z) [c)->g[[]]] 2
(2)
where, the first term is reconstruction loss, which measures the difference between the original image and the reconstructed image V. sg[:] represents the stop-gradient operator, and the second term codebook loss, which encourages the codebook to be close grid features. The third term is the "commitmentt loss" 43), where w serves as a parra--arameter. However, existing VQ-based methods mainly focus on the learning of single-modal codebook, thereby limiting their applicability to multi-modal downstream tasks.
3.2 Proposed Method: LG-VQ
Existing works attempt to improve codebook reconstruction capabilities obtain better performance on downstream tasks. However, ignoring modal differences results in suboptimal performance when the codebook is applied to cross-modal tasks. To address this issue, we propose utilize the pre- trained text semantics as supervised information to learn a text-aligned codebook. Its advantage is abundant semantic information from text can be fully exploited for more robust codebook learning to improve the performance of reconstruction and cross cross-modal tasks. The comprehensive architecture of the proposed LG- VQ method is illustrated in Fig. left. It consists of two supervision modules: Semantic A Alignment Module (i.e., fasa and Lmmp)), and Relationship Alignment Module (i.e,, Lras). The first module encourages global semantic consistency between the codebook and text. The second module aims to transfer the rich semantic relationship between words into codes. Next, we introduce these two modules in detail.
3.2.1 Semantic Alignment Module
Considering that paired image and text data have consistent semantic information and the missing information of masked data can be completed from the other modality, we propose global semantic aligmment, which aims to enhance the consistency of global semantics between text and visual codes, and masked text prediction, which uses visual codes to restore the masked words. Next, we discuss how to align text and codebook in the semantic space.
Text Information Encoder: Instead of jointly training text and codebook from scratch, we employ a pre-trained cross-modal model CLIP [32] to encode text information. Its advantage is that such text information already has good cross- cross-modal semantic knowledge and is beneficial for codebook learning. Specifically, for a given text description of an image t {WSOT, W1, W2 ... Wn-2, WEOT), ), where w, denotes the i-th word, wsoT and w E'OTT represent the [start] token and [end] token, respectively, and n is text sequence length. We use the text encoder of pre-trained CLIP model to obtain whole sequence embedding T E R.nd;;
T = {ESOT, Swi Ewg:" Ew,,, E = CLIP(().
Similar to CLIP, we use the egor to represent the global context feature of the sequence. Global Semantic Alignment aims to align text and image visual codes in the global semantic space. For getting the global representation of visual codes, we employ vision transformer (ViT) fe... [ to encode the discrete codes of image. Specifically, given an image, we firstly obtain the discrete codes of image Z by Eq. []] Then, we introduce learnable global token [CT.S] at the beginning to form a token sequence 7. where global token [CT.S] is employed to capture the image's global context information. We feed the sequence into fe.. get a new visual code representation, that is:
Zet [(ecLs,,,, ,64
fo.. (Ze)-
Finally, we employ InfoNCE 29 which maximizes the similarity between visual and text in the global representation, as our learning objective, where B is the batch size, s(,,)) -) is cosine similarity:
Σ log exp(s(ecls:<Eort
Lysa
(5)
ΣjEB exp(s(eeuss) (goT))

---

Masked Text Prediction: To further enhance the semantic alignment, we propose to use discrete visual codes to reconstruct the masked words from more fine- grainer perspective. refer to Fig. lleft. Formally, for given fixed-length text sequence of n. - 2, we first randomly sample the masking ratio r from a truncated Gaussian distribution [19]. Subsequently, we randomly mask out r- (n - 2) words and replace them with learnable [mask;] tokens based on their positions i. Next, self-attention module [44) is employed to learn adaptive masked word embeddings based on unmasked words. The resulting adaptive masked sequence is denoted as T {ESOT, mi1 euyz 1 EEOT J. where m, is the mask token embedding at the i-th position the sequence, Following this, a cross attention decoder fe... (:.-) is employed to predict the masked word tokens given the discrete visual codes Zu obtained by Eq. 4 Finally, we add a cross-entropp loss H(:.-) between the ground word tokens and the output of the decoder. Let Ymek denote one-hot vocabulary distribution where the ground-truth word token has a probability of 1, So (Zet; Tinsk) denote the predicted probability of model for masked word tokens. That is:
Emty -E( Tmok H (ymsks fo.. Tm)s))..
(6)
3.2.2 Relationship Alignment Module
While the two aforementioned loss functions for achieving good alignment at holistic semantic space have demonstrated initial promise. they cannot satisfy more complex reasoning tasks like image captioning and VQA. Inspired by some VQA techniques 226 40 24 [. the semantic relationships between pre- trained words play very important role in complex text reasoning tasks. For instance, to answer question ("What is this woman holding?"), one needs to fully understand the visual objects "women", "racket", and semantic relationship between them ("holding"). Based on the above fact, we propose to transfer the semantic relationship between words into codes. Such semantic relationships enable the model better understand the image for addressing complex reasoning tasks.
as shown in Fig D
But unfortunately, there is an issue there is align- ment between words and codes. Thanks for the above two losses that have provided semantic alignment of text and visual codes. To achieve the above idea, as shown in Fig. 3] first use Zue to align with words. Then, we inject semantic relationships between words into the initial codebook Z, instead of the Zut- Its ad- vantage is it can prevent codes from collapsing into a single point for learning more diverse representations by relationship limiting. Then, Zuz primarily serves the purpose of aligning words and codes, but is a crucial step for subsequent processes. Specifically, given any two words of a sentence, we use pre-trained word em- bedding (32] to encode words, ew: and Cuy. We employ cosine similarity to find the index of the code from Zet that is most similar to the word. Then, one can get code embedding from Z based on the index:
06
Step I: Aliggmment
Z
2
S
Step Marping
0's
Figure 3: Illustration of relationship align- ment module, words, then inject the semantic relationship of two words into Z codes.
Zu to align with two
Ez2 2 (argmaar s(ew; =]), E.E
Ez₁ Z(argmaa s(Ewy) e2),
e.EZutta
Next, consider cosine similarity measure of semantic relationships between words and leverage it to establish corresponding relationships between codes achieving semantic relationship transfer. Finally, utilize the following loss function learning objective:
Lras
Σ (w1))0y)
(o(ewy,Crey) s(c,)),2,)) 22,
(8)
3.2.3 Training Objective
We use three hyperparameters (i.e., a, B, and 7) control three losses, respectively. Finally, the overall objective function is:
L Log + aLgoo + BLmtp + Cras.
(9)
5

---

Table 1: Results of image reconstruction TextCaps, CelebA-HO, CUB-200, and MS-COCO. "VQ-VAE+LG" denotes considering our method LG-VQ based on VQ-VAE.
TextCaps
CQ
CUB-200
MS-COCO
Models
FID PSNR1
FID PSNR1
FID4 PSNR1
FID, PSNRT
VQ- VAE
82.31
21.96
41.45
25.57
54.92
24.38
86.21
23,55
VAE+LE
81.93
21.95
40.53
25.04
36.55
25.60
79.54
23.40
VQ-GAN
24.08
19.64
5.66
24.10
3.63
22.19
14.45
20.21
VQ-GAN+LG
20.35
19.92
5.34
23.75
3.08
22.47
10.72
20.50
CVQ
16.35
20.24
5.19
23.15
3.61
22.29
9.94
20.48
CVQ+LG
15.51
20.21
4.90
24.48
3.33
22.47
9.69
20.71
Experiments
4.1 Experimental Settings
Evaluation Metrics. As our method is model-agnostic, we choose recent models, including VQ-VAE
43], VQ-GAN 19]. and CVQ [53] as our backbone network. Following existing works [52 59
evaluate the reconstruction image quality on two evaluation metrics, i.e., Fréchet Inception Distance (FID) [13] which evaluates the perceptual similarity of reconstructed images and original images, and Peak Signal-to-noise Ratio (PSNR) (1O) is employed to measure the pixel-level similarity between the reconstructed and original images.
Dataset. We evaluate our method on four public datasets, including TextCaps (39] CelebA-HQ 23
CUB-200 [45, and MS-COCO (20). For CelebA-HQ, CUB-200, and MS-COCO datasets,
publicly available image captions, CelebA-HQ from 477, CUB-200 from 34 MS-COCO from B Implementation Details. Following VQ-GAN 9 all images are reshaped 256 x 256 for recon- struction and generation. Down- wown-samplnng factor f is set to 16. The codebook size K is 1024. The batch size is 8. In our experiments, we maintain consistent parameter settings between our method LG-VQ and the chosen backbone networks (i.e., VQ-VAE 43 VQ-GAN 99. and CVQ (53) for fair comparison. For each image, we randomly select text from multi-text for training. Since our method introduces additional text and pre-trained CLIP model, for a fair comparison, we select VQCT [5T] as the baseline for various downstream tasks. VQCT extracts many risual-related words from large amount of text and designs a novel codebook transfer network based on the pre-trained CLIP model to learn the visual codebook.
Table 2: Ablation study of our three loss functions on TextCaps and CUB-200.
Table Results (Recall @ 1) of masked word prediction on CelebA-HQ and CUB-200. "Mask-1" denotes that text randomly masked one word.
TextCaps FID
CUB-200
Setting
FID4
(i)
Baseline
24.08
3.63
Dataset
Recall @
(ii)
Lassa
23.01
3.39
Mask-1 I
99.55
(iii)
Littp
21.54
3.49
CelebA-HQ
Mask-3
99.24
(iv)
+ Lenti Lras
20.77
3.32
Mask-1
83.65
(V)
+ Lonty + Lyaa
20.46
3.34
CUB-200
Mask-3
80.17
(vi)
+ Lint Eggn Lras
20.35
3,08
4.2 Discussion of Results
Table I illustrates the image reconstruction performance of our model compared to the backbone model on multiple datasets. It can be observed that our method LG-VQ outperforms all compared methods on most evaluations, which suggests that our method is extremely effective and has strong generality. Compared with FID, our PSNR improvement is marginal, this is reasonable in the VQ research domain, which widely exists in previous VQ methods 21 [52 (33. The main reason is that PSNR only measures the pixel-level similarity of the images, while FID can effectively measure the diversity and semantic similarity of image generation. Compared with backbone models, the key difference lies in that our method introduces well pre-trained text semantics, which is beneficial to learning more expressive codebook. This shows the effectiveness our method. We also provide qualitative comparison of the image reconstruction performance of different methods, please refer to

---

Appendix A.3 Fig.
10
4.3 Ablation Study
Are our three loss functions both effective? In 2 we conduct an ablation study to show the effectiveness of the proposed three loss functions. Specifically, the VQ-GAN serves as the baseline model (i.e., without introducing any loss). We do not conduct a separate experiment on Cras because this module requires code and words to be well aligned. Based on the results from (i) ~ (vi), we draw several key conclusions: Firstly, each loss function plays a erucial role in improving the performance of image reconstruction. Secondly, the performance of (iii) outperforms (ii) by a large margin on TextCaps. This is reasonable because TextCaps's 's texts are richer and more diverse than CUB-200, it can provide more knowledge for more fine-grained alignment between codes and text, which is useful for the learning of a more robust codebook. Thirdly, analyzing the results of (iii) and (iv), injecting word-level semantic relationships into codes is beneficial, which confirms our motivation. Furthermore, the performance of (v) outperforms (i), which is reasonable because the abundant semantic knowledge from pre-trained text can be fully exploited for learning more robust codebook representation. This supports the motivation of learning a multi-modal codebook (i.e., aligned with text). Finally, comparing the results of (vi) with (i)~(v), fully considering all losses achieves the best performance, indicating the effectiveness of our method.
Can our global semantic supervision align vision and language? In Fig. we provide several image-to-text retrieval cases on CelebA-HQ and CUB-200 datasets based on VQ-GAN+I.G. From the figure, can be observed that our method can accurately retrieve text very similar to the image content, achieving the alignment of vision and language. For example, row 2 examples show that our method can precisely understand some key attributes of images (e.g., "gray hair", "necktic", "big nose" and "chubby") and retrieve similar text. This suggests that the codes learned through our method have obtained good alignment with the text, which verifies the effectiveness of our method. Moreover, such alignment is beneficial for learning robust code representations and enhancing performance in downstream tasks.
Tep-1 mertoea kedl polke auarag and
Top I cenderdd bust
mel vihor on begg pity anrk
aech Macc doi which
lesd Look Toy resioval kedt
wago
Tep-l rerieved lest
Thi vinua las code toun) No
The panca cosd glity suoninng work Hir
The markir 260 Aalaanttuuua bup cnde
notee mneed weetingg modtict
yoain work)
OT
Top wisl mys ree, und lg:
ppp vord MI A.b
Figure 4: Examples of the top-1 most similar text selected on image-to-tent retrieval task.
Figure 5: Examples of the top-1 word predicted on masked word prediction task.
Can our codebook accurately predict masked words? To answer this question, we conduct word prediction task on test data based on VQ-GAN+LG by randomly masking one word or three words of text, as shown in Table 3 We use Recall@) I as the evaluation metrie [18]. From the table, our method demonstrates accurate predictions of masked words, confirming the effectiveness of our approach. Fine- tineeggained word prediction can help the codebook better understand the text semantics, which is crucial for improving the performance of the downstream task. Additionally, several examples in Fig. [5 demonstrat our method's ability that accurately predict subject words (e.g., wings, eyes) and verbs (e.g., has, is, and smiling), further affirming its strong multi-modal understanding capabilities.
Can our codebook learn the word semantic relationships? In Fig. 6. we visualize the cosine similarity between words and the cosine similarity between codes aligned with the words for a certain sample based on VQ-GAN+LG. From the figure, we can see our codes can learn consistent relationships with word semantics compared with VQ-GAN. For example, the similarity "code 33" vs "code 232" (0.46) resembles "wings" vs "chest" (0.49). In addition, we provide a quantitative similarity evaluation between codes and words Table 4 From the results, we can find that our codes indeed achieve consistent semantic relationships with words.
Table 4: Results of similarity evalu- ation between codes and words on CUB-200 all test data.
Method VQ-GAN VO-GAN+LG
MSE. 0.6374
0.0351
Is our method effectively learning more diverse code representation? Following (52), we directly feed each codebook embedding En (size: 1 1 x 256) into the decoder Deg(.) to generate codebook
7

---

image (size: 16 x 16 x 3). Then, we concatenate all codebook images to form big image with 32 x patches. Finally, we visualize the result of VQ-GAN and our LG-VQ on TextCaps and MS-COCO as shown in Fig. 7 This visualization suggests that our method enables the model to learn more diverse code representations and improve codebook usage.
48
95
0:22
SUM
6:30
C8
042
0:55
444
044
440
07
0F
41
070
0.9
06
(a) word similarity
(b) LG-VQ code similarity
(c) VQ-GAN code similarity
Figure 6: Visualization of words similarity and image codes similarity aligned with the word. We extract some representative words from the text as a demonstration.
VQ-GAN
VQ-GAN-LG
VQ-GAN
VO-GAN
(a) The usage of codebook on VQ-GAN is 18.62% and VQ-GAN+LG is 43.58% TextCaps
(b) The usage of codehook on VQ-GAN is 40.09% and iQ-GAN+LG 97.89% MS-COCO
Figure 7: Visualization of the codebook of VQ-GAN and LG-VQ on TextCaps and MS-COCO.
4.4 Application
4.4.1 Image Generation
Following 9737 we conduct image generation downstream tasks (i.e., text-to-image, semantic synthesis, unconditional generation, and image completion) to fully validate the effectiveness of the learned codebook on CelebA-HQ.
Text-to-Image In Table 5 compare LG-VQ with the state-of-the-at models CelebA-HQ dataset for text-to-image. From the results, our LG-VQ method outperforms baseline methods by large margin. This is reasonable due the incorporation of pre-trained text knowledge enabling a comprehensive understanding of the text, which suggests our method's s effectiveness. Moreover, we provide some synthesis examples comparing the results of LG-VQ with baseline methods in Figure & showing the performance in the text-to-image task. From the figure, we can see our method only comprehensively understands the given text conditions but also excels in generating realistic images compared with baseline methods. For instance, our method can capture the "glasses", "man", "long black hair". and "no beard" key attributions.
Semantic Synthesis. Following [9]. we compare with existing semantic synthesis models in Table 6 Our method achieves the best performance, which suggests our method's effectiveness. We provide some examples in Appendix Figure 13
Unconditional Generation and Image Completion. Following 9 we conduct unconditional image generation and Image Completion on CelebA-HQ dataset, as shown in Table and the results, we can see that our method can significantly improve the performance of VQ-GAN, which is reasonable because pre-trained text can provide rich semantic knowledge for learning more robust codebook representation. This suggests the effectiveness of our method. We provide some examples in Appendix Figure 18 and Figure [7
and Table I0, From
8

---

Condition
VQ-GAN
VQ-GAN-LG
CVQ
CVO.LG
The wortae
bowa hait. and
ied cyebooss She
lecaiing glasess and is wormg
blar Spaikk
The Ien Iun Jana Blakk bast Helam
Figure Text-to-image synthesis and semantic image synthesis on CelebA-HQ. Text with background color emphasizes generated details
4.4.2 Visual Text Reasoning
we the learned codebook to conduct two visual text reasoning tasks: 1) image captioning on CUB-200; and 2) visual question answering (VQA) on COCO-QA [35] For the experimental setting, please refer to Appendix A
Follow
27 1
Image captioning. Following [27]. we conduct the image captioning task on the CUB-200 dataset. We compare two recent work V2L Tokenizer (56) and VQCT [5]]. We select VQ-GAN our backbone network. The results are shown in Table IT For the results, we can see that our LG-VQ method outperforms the performance of VQ-GAN. This is reasonable because the pre-trained text provides rich context and relationship semantics for codebook learning, which verifies our motivation for learning text-aligned codebook to improve the performance of the codebook on cross-modal tasks. On the other hand, the V2L. Tokenizer and VQCT cannot achieve very good performance because is di difficult to assign correct semantic language tokens to images. Compared with the V2I. Tokenizer, our method utilizes pre-trained text semantics as supervised information. Its advantage can make the codebook learn semantic information consistent with the text (i.e., learning a text- aligned codebook). And, our method is model-agnostic, which can be easily integrated into existing vQ models.
Visual Question Answering. We select VQ-GAN and VQCT [1 as the baseline. We conduct the VQA task on the COCO-QA 351 using the codebook trained on the MS-COCO dataset. The results that our LG-VQ method significantly improves
shown in Table 9 From the results, we the performance of VQ-GAN on VQA task (approximately 8.32% on Accuracy). That is reasonable due introduce pre-trained text semantics enable to obtain codebook aligned with the text, which helpful for comprehensively understanding the given text question. This confirms our motivation and the effectiveness of our method.
Table 5: Results of text-to-image CelebA-HQ
Table 6: Result (FID.) of semantic syn- thesis on CelebA-HQ.
Text-to-Image
Model
FID.
Semantic Synthesis
Model
Unite and Conqu 4
26.09
FID
Corgi 54
19.74
Reg- VQ [52
15.34
LAFITE 55
12.54
VOCT [51]
14.47
VQ-GAN
15.29
VQ-GAN
11.53
CVQ
13.23
CVQ
11.04
VQ-GAN+LG
12.61
VQ-GAN+LG
11.46
CVQ+LG
12.33
CVQ+LG
11.03
4.4.3 Visual Grounding
49 to validate the effectiveness of the We conduct a visual grounding task on refcoco dataset learned LIS-COCOss codebook. Following the same metric used [5]. prediction right the IoU between the grounding truth box and the predicted bounding box is larger than 0.5. We select VQ-GAN and VQCT (1 as the baseline. The results are shown Table g From the results, we can see that the performance of our method consistently outperforms VQ-GAN and VQCT, which suggests its effectiveness. We also provide qualitative comparison Appendix Figure 19 For the experimental setting, please refer to Appendix A.I

---

Table 7: Result (FIDJ) of un- conditional image generation on CelebA-HQ
Table Result (FIDL) of visual grounding on refcoco dataset using MS-COCO's codebook.
Table 9: Results of (Aceu- racy and WUPS 46] VQA on
COCO-QA QA 35 dataset using
MS-COCO' s codebook.
CelebA-Hi HQ
Model
FID4
Visual Grounding
VQA
Model
Setting
Style ALAE 33
19.2
Accuracy(O.5)
Accuracy1
WUPS
DC-VAE 31
15.8
VQ-GAN
9.14
VQCT 51
40.42
82.06
VQ-GAN
10.2
VOCT 51
9.46
VQ-GAN
37.82
83,22
LQ-VQ
9.1
LG-VQ
9.62
LG-VQ
40.97
83.56
Table 10: Result (FIDL) of im- age completion CelebA-HQQ
Table 11: Results of image captioning on CUB-200.
Image Captioning
Model
CelebA-IIQ
BLEU41
ROUGE-L1 METEOR1 CIDEr-D1
Model
FIDJ
VQ-GAN
1.29
33.40
24.47
93.62
VQ-GAN
9.02
V2L. Tokenizer [56
1.59
30.65
25.76
104.14
LG-VQ
8.14
VQCT 51
1.38
26.50
24.63
98.22
Improve 9.76%
LG-VQ
1.69
34.73
25.78
102.77
5
Conclusions
In this paper, we propose novel codebook learning method, named LG-VQ. LG-VQ is a model- agnostic method and can easily be integrated into existing VQ models. In particular, we propose incorporate pre-trained text semantics into the codebook by novel supervision modules, i.e., semantic and relationship. Quantitative and qualitative experiments demonstrate the strong generality of our method, showing its ability improve the performance of the codebook in modsl tasks. Limitations. In our current paper, we suppose each word aligns with a code, but it fails to capture some more complex relationships between words and codes (e.g., one code aligns with multiple words). In the future, we plan to investigate the relationships between codes and words. Moreover, although our results show that the performance of VQ in visual text reasoning tasks can be significantly improved. its results still far lower than the performance of image captioning VQA models. Broader impact Our paper shows that learning a mudli- codebook (i.e., a text-aligned codebook) not only significantly improve the performance of reconstruction but also the performance of the codebook on cross- modal tasks. The potential impact of our research lies in its influence on future studies, specifically in the area of unified modeling of multi-modal understanding and generation. For instance, our work can be extended to interact with LLMs to improve multi modill understanding and generation capabilities. In particular, exploited to produce some erroneous and unethical information, which needs to be handled carefully before employing model in practical applications.
model can be used to generate images or text. It may be
Acknowledgement
This work was supported by the Shenzhen Peacock Program under Grant No. ZX20230597, NSFC under Grant No. 62272130 and Grant No. 62376072, and the Shenzhen Science and Technology Program under Grant No. KCXFZ202 1020163403005. Project of PCL (PCL2023A08) and the National Science Foundation of China: Multi-source Cross- platform Video Analysis and Understanding for Intelligent Perception Smart City: U20B2052.
was also supported by the Major Key
References
[1] Stanislaw Antol and Aishwarya Agrawal. Vqa: Visual question answering. In ICCV, pages 2425-2433, 2015.
[2] Huiwen Chang and Han Zhang. Maskgit: Masked generative image transformer. CVPR, pages 11315-1133 11325,, 2022.
10

---

[3] Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 1504.00333 2015.
[4] Yen-Chun Chen and Linjie Li. Uniter: Universal image-text representation learning. In ECCV, pages 104-120. Springer, 2020.
[5] Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to-end visual grounding with transformers. Proceedings of the IEEE/CVF International Conference Compuler Vision, pages 1769-1779, 2021.
[6] Ming Ding and Zhuoyi Yang. Cogview: Mastering text-to-image generation via transformers. NeurlPS, 34; 14:19922-19835, 2021.
[7] Xiaoyi Dong and Jianmin Bao. Peco: Perceptual codebook for bert pre-training of vision transformers, In AAAI, volume 37, pages 552-560, 2023.
[8] Alexey Dosovitskiy and Lucas Beyer. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[9] Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873-1 12883, 2021.
[10] Fernando Fardo and Victor H Conforto. formal evaluation of psnr as quality measurement parameter for image segmentation algorithms. arXiv preprint arXiv: 1605.076 16, 2016.
[11] Shuyang Gu and Dong Chen. Vector quantized diffusion model for text-to-image synthesis, In CVPR, pages 10696-10706, 2022.
[12] Yuchao Gu and Xintao Wang. Rethinking the objectives of vector-quantized tokenizers for image synthesis. arXiv preprint 1rXiv:2212.03/85, 2022.
[13] Martin Heusel and Hubert Ramsauer. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 30, 2017.
[14] Mengqi Huang and Zhendong Mao. Not all image regions matter: Masked vector quantization for autoregressive image generation. In CVPR, pages 2002-2011, 2023.
[15] Mengqi Huang and Zhendong Mao. Towards accurate image coding: Improved autoregressive image generation with dynamic vector quantization. In CVPR, pages 22596-22605, 2023.
[16] Chao Jia and Yinfci Yang. Scaling up visual and vision-language representation learning with noisy text supervision. In JCML, pages 4904-4916. PMLR, 2021.
[17] Doyup Lee and Chiheon Kim. Autoregressive image generation using residual quantization. In CVPR, pages 1522-11532, 2022.
[18] Junnan Li and Ramprasaath Selvaraju. Align before fuse: Vision and language representation learning with momentum distillation. volume 34, pages 9694-9705, 2021.
[19] Tianhong Li and Huiwen Chang. Mage: Masked generative encoder to unify representation learning and image synthesis. In CVPR, pages 2142-2152, 2023.
[20] Tsung- Yi Lin and Michael Maire. Microsoft coco: Common objects in context. In ECCV, pages 740-755. Springer, 2014.
[21] Xinmiao Lin and Yikang Li. Catch missing details: Image reconstruction with frequency augmented variational autoencoder. In CVPR, pages 1746-1745, 2023.
[22] Hao Liu and Wilson and Yan. Language quantized autoencoders: Towards unsupervised text-image alignment. arXiv preprint arXiv:2302.00902. 2023.
[23] Ziwei Liu and Ping Luo. Deep learning face attributes in the wild. ICCV, pages 3730-3738. 2015.

---

[24] Jiasen Lu and Jianwei Yang. Hierarchical question image co-attention for visual question answering. NeurIPS, 29, 2016.
[25] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11-20, 2016.
[26] Tomas Mikolov and Kai Chen. Efficient estimation of word representations vector space. arXiv preprint arXXi::3001.370 2013.
[27] Ron Mokady and Amir Hertz. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734 2021.
[28] Varun K Nagaraja, Vlad Morariu, and Larry Davis. Modeling context between objects for referring expression understanding. In Computer Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 792-807, Springer, 2016.
2016: 14th European
[29] Aaron den Oord and Yazhe Li. Representation learning with contrastive predictive coding. arXiv preprini / 0077 2018.
[30] Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive generative autoencoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 823-832, 2021.
[31] Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent au- toencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11104-1411,, 2020.
[32] Alec Radford and Jong Wook Kim. Learning transferable visual models from natural language supervision. ICMI., pages 7748-8763. PMLR, 2021.
[33] Ali Razavi and Aaron Van den Oord. Generating diverse high-fidelity images with vq-vae-2. NeurlPS, 32, 2019.
[34] Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. Learning deep representations of fine-grained visual descriptions. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 49-58, 2016.
[35] Mengye Ren and Ryan Kiros, Exploring models and data for image question answering. NeurlPS, 28, 2015.
[36] Yuchen Ren and Zhendong Mao. Crossing the gap: Domain generalization for image captioning. In CVPR, pages 2871-2880 2023.
[37] Robin Rombach and Andreas Blattmann. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[38] Robin Rombach and Andreas Blattmann. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[39] Oleksii Sidorov and Ronghang Hu. Textcaps: dataset for image captioningwith reading comprchension. 2020.
[40] Mohammed Suhail and Abhay Mittal. Energy-based based learning for scene graph generation. In CVPR, pages 13936-13945. 2021.
[41] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint 10Xiv:1908.07490, 2019.
[42] Aaron Van den Oord and Nal Kalchbrenner. Conditional image generation with pixelcnn decoders. NeurlPS, 29, 2016.
[43] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurlPS, 30. 2017.
12

---

[44] Ashish Vaswani and Noam Shazeer. Attention is all you need. NeurIPS, 30, 2017.
[45] Catherine Wah and Steve Branson. The caltech-ucsd birds-200-2011 dataset. 2011.
[46] Zhibiao Wu and Martha Palmer. Verb semantics and lexical selection. arXiv preprint emp- lg/94406033, 1994.
[47] Weihao Xia, Yujuu Yang, Jing-Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and manipulation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 2256-2265, 2021.
[48] Jiahui Yu and Xin Li. Vector-quantizee image modeling with improved vqgan. arXiv preprint 1rX.v2210..04627. 2021.
[49] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016.
[50] Lijun Yu and Yong Cheng. Spae: Semantic pyramid autoencoder for multimodal generation with frozen llms. arXiv preprint arXiv:2306. 7842, 2023.
[51] Baoquan Zhang, Huaibin Wang, Chuyao Luo, Xutao Li, Guotao Liang, Yunming Ye, Xiaochen Qi and Yao He. Codebook transfer with part-of-speech for vector-quantized image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7757-7766, 2024.
[52] Jiahui Zhang and Fangneng Zhan. Regularized vector quantization for tokenized image synthesis. In CVPR, pages 18467-18476, 2023.
[53] Chuanxia Zheng and Andrea Vedaldi. Online clustered codebook. In ICCV, pages 22798-22807
2023.
[54] Yufan Zhou, Bingchen Liu, Yizhe Zhu, Xiao Yang, Changyou Chen, and Jinhui Xu. Shifted diffusion for text-to-image generation. In Proceedings the IEEE/CVE conference on compuler vision and pattern recognition, pages 10157-10166, 2023.
[55] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Towards language-fre training for text-to-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17907-17917, 2022.
[56] Lei Zhu and Fangyun Wei. Beyond text: Frozen large language models in visual signal comprehension. arXiv preprint vrXiv:2403.07874 2024.
13

---

A Appendix supplemental material
A.1 Experiment Details
Semantic image synthesis, unconditional generation and image completion: All models follow the default setting of VQ-GAN-TTansffrmef Specifically, the vocabulary size, embedding number, and input sequence length are 1024, 1024, and 512, respectively. The layers and heads of the transformer are both 16. The semantic image synthesis experiments are conducted on I 4090 GPU with a batch size of 15 and one day of training time. The unconditional generation and image completion experiments training time.
conducted 2 4090 GPUs with batch size of 36 and day of
Text-to-image generation: All models follow the default setting of VO-Diffusionf) Specifically, the layers of the transformer are 19 with dimension 1024. The diffusion step is 100. The training epoch is 90 for all models. The experiments conducted 4090 GPU with batch size of 24 and two days training time.
Image captioning: Inspired by ClipCap [27] use the trained codes replace the ClipCap's prefix embeddings. The model framework is shown in Fig. (a). The training epoch is 100 for all models. The experiments conducted
4090 GPUs with batch size of 60 and day of training time. Visual question answering: The COCO- QA (35] dataset is automatically generated from captions in the Microsoft COCO dataset (20). There are 78,736 train questions and 38,948 test questions in the dataset, These questions are based on 8,000 and 4,000 images respectively. There are four types of questions including object, number, color, and location. Each type takes 70%. 7%. 17%. and 6% of the whole dataset, respectively. All answers in this data set are single word. Following the image captioning task, we use the last hidden embedding to do VQA, shown in Fig. (b). Following the (24), we report classification accuracy and Wu-Palmer similarity (WUPS). The training epoch is 50 for all models. The experiments are conducted on 4090 GPUs with batch size of 60 and one day of training time.
Visual Grounding:
The refcoco dataset [49 includes 19,994 images with 50,000 referred objects.
Each object has more than one referring expression, and there 142,210 referring expressions in this dataset. There are two commonly used split protocols for this dataset. One is RefCOCOg-goope (5), and the other is RefCOCOggmd (28). We follow RefCOCOgumd [28] to split the dataset. The train set has 42,404 expressions, the validation set has 3,811 expressions, and the test set has 3,785 expressions. Following [5], concatenate the image codes and text tokens and feed them into learnable transformer with coordinate regression layers (ie., FNN) to predict the object box. The training epoch is 100 for all models. The experiments are conducted on 2 4090 GPUs with batch size of 30 and several hours of training time.
The
errings END
train
GPT
GPT
Encoder
The girl wears errings.
VQ
Encodder
what coming down
the tracks.
Image captioning
(b) VQA
Figure 9: The architecture of the visual text reasoning based GPT.
thups://github.co
"http://gittuh.com/Con wwtp://wwwwwoooooooooooooooooo.o www.
14

---

A.2 Experimental comparison with VQCT
We provide comparisons with VQCT [1 for image reconstruction in Table 12 Moreover, also provide experimental results on further integrating our method into VQCT to verify its effectiveness and versatility. The results are shown in Table I3
Table 12: Comparison of VQCT and our method on reconstruction
Model
Codebook Size
#Tokens
CelebA-HQ
CUB-200
MS-COCO
VOCT
6207
512
5.02
2.13
9.82
VQ-GAN+LG
1024
256
5.34
3.08
10.72
CVQ+LG 1024
256
4.90
3.33
9.69
Table 13: Comparison of reconstruction and VQA on VQCT and VQCT+LG on the MS-COCO
dataset.
Image Reconstruction
VQA
Model
FID..
Accuracy
VOCT
9.82
40,42
VQCT+LG
9.57
40.64
A.3 More Examples and Qualitative Results
We provide more examples of image reconstruction Fig. 10 image-to-text retrieval in Fig. [] and Fig. 12] We also provide more image synthesis results in Fig. 13 for semantic image synthesis,
and text-to-image synthesis Fig. 14 We provide some examples of image captioning in Fig. [5]
and VQA in Fig. 16 We provide some examples of unconditional generation in Fig. [8. and image
completion in Fig. 17
We also provide qualitative comparison of visual grounding Fig. 19
Original Irage
VO-GAN
VQ-GAN-LG
CVQ
CVQ+LG
Ooiginal Image
VQ-GAN
VQ-GAN-LG
CVQ
CVC-LG
Cakbi-ii
MS-COCO
CU 500-500
32
TeaCap
6150
Figure 10: Reconstruction from different models four datasets. The red-color boxes highlight reconstruction details.
15

---

Top-5
took with
lol.
bookd that says yornng diced his
theordl
buots opened
succher to
open book on
busic wat pages that liikt open paus saps the fanend the top.
taz book opan so poac 160 where actioon kender couda inners.
bood is opeadd to page with bealling fire porr
hoxt oith pagas that look apa open tou page that saye faaura top. piges opens page
io open to page dlous ltaddng.
Top-5 setroved text
Groun Truth text
roo starinng zo jouporer rocilioss with screas showkia "oouantiag"
comppaer scren woth the ingile page foe coteaioos
fenesda computer menixe with windknce up
upua sigs senocn
lyyaski moceiio: machook compaier siplliing windoons updint
agpiuss
wiledo protissional
dell suncker But Saaws the login and pessoced coler.
satrieved
wills
hotti frais
gourdaas
dheeap the takel battle bea roederer champpanne.
<loicep of libel on burlle of sindever chatapigee
gridon burt of parines cason poor oropped with poin rhown.
a lubdi for loais saukerr hruk promner chaneeeaaeeee chamesune.
en qualte bottl wils yellow and good chanzhecd scanoo
rask mederws champagre coocted mitos, Stuee bo (iqu)))
veave cliqgue time dhumpagge
Ber's Chil Bood
Top 5 uttrieved text
Groua Truth text
choseag the sign ousale ber's ehil bewil bright yellow and red.
bee's sign condde
yelion rugs beacee lahhnng hangtag
ras pellow lign Ser reterratt sggs bot'
chill ydllow woub besoi black.
the romile chill beed yallow and mallag haslydall bend ii
Figure 11: Examples of the top-5 most similar text selected on Textcaps based on VQ-GAN+LG. The bold text means the same as the ground truth result.
Top-5
Groond Thrth
this Made berd with rod lond.
staee dar black bad wids conns in load doi a Mack thd with blcc and tii hoad and rod rape. wings
a large brack bid with a will and brisht not crrors
trnpge covers pointed napo.
polnted
Top retheved text
Ground Truth text
groenisk bisd velld black and epectia..
a fasks yalloo hed wash urea wiines and tut and brack owvers.
diss Hid with Riee yellow iiaak
a yolliorr with greoo cvoor and htadi syering
a greae
croes yellow and belly.
the has winga that groms and has yelhow belly
groviih yellow hesl with solil buadi esoiie eyening.
yallan bust with
track cunna, and blick chack puidica.
reall yotow blid, with
conws, and abeet beat.
Top 5 retuieved text
Guound Truck tost
bright enwage hird wash bark wings, blackk apdd an cewige beoll.
seall beight venege bist dark back wings and
cmagg
ccoagg
enagge
wings
omange conna, friecii thonat
with hruuet das bid iirel tlock and very beok.
waall hid
the brighn enage hast we blak winngs and whor painaed hacc
this bastild bis with Mack wage and
hedl hes vingo that ure sane and energe betty
Top-5 retrieved lext
Gloondd Truth
yutioe black hind black bead, yetter bodty, black vitage white
p-llan und bslad bid wits back band, yellow boods Mack witg witk viakz wituftura, und soherr this back the has croea, pellow belty,
wrneghars, and sham, ditn back hilt.
begin yolua onal
parchadl the af heaach
blck long yellow black dhowwandd sareel pellow und blads festhrr covared lert.
loal with
body
pelioe belly da bidd vilg But Madi la yellow belly
Figure 12: Examples of the top-5 most similar text selected CUB-200 based VQ-GAN+LG. The bold text means the same as the ground truth result.
16

---

Semarmt
VO-GANILG
CVQ
Setzanti
VO-GANILG
CVQILG
VO
CVQILG
Figure 13: Semantic image synthesis on CelebA-HE
Coudition
GANGAN
CVQ-LG
Conditicn
VO-GAN-LG
CVQHLG
Coudition
GAN--GG
CVQILG
ais usse. and pakcokis. Swik
lppkk
Mack luir stright bak: and oval fee Mie weas Iputick
been
neen dightty spea wears Iqutick.
The scaro loo cencer estu,
He Hecko big igs. massche
Be
young
was fhy makeeg.
Figure 14: More tox-tto-imae generation on CelebA-HQ.
VQ-GAN: this bird black and white, and has gray black breast.
DOOG:AN: this bard has wings that are black and white with black spots.
this bird brown back, and black feet.
and
Our: small bind with pointed wing bar.
long,
this bird has white light white brown bluse crown and is brown thrmat, and also his very short crenshaw.
VQ-GAN:
VQ-GAN; this small brown in color and
lintl bind brown syes, long black boak, brown crown and beak.
wings,
wings crown and short point orange bill.
Figure 15: Image Captioning CUB-200 based VQ-GAN and VO-GAN+LG.
17

---

Q waat is the person
Qull as sitting the counter differen stages outiing with knit??
Q what the aus swinging it ball on teenis count
Q. what are culbing their heads and necks wyether?
Q: wist the color the log!
Q: how mar small
with e ccwboy hat rkding trying get cow?
childrer thtt are exjoying
small saack?
GT: home
GD cames
GT:
CT gralla.
GT white
GT: foor
VO-CINN motongede
dephonts
Dar: haroo
Our vegrtables
nopur
Out giniftt
Our
Figure 16: VQA on COCO-QA based VQ-GAN and VQ-GAN+LG.
CVQ-LG
Figure 17: Image completion on CelebA-HQ.
T O
m
t77
l E
Figure 18: Examples of unconditional image generation on CelebA-HQ based on VQ-GAN+LG.
18

---

player un right of the tine in front
May bisycle
pensuz with geggles sttling
bottom gimalls
Day gray stirt standing
zan in black en skis
VOGAN
Figure 19: Examples of visual grounding refcoco. Blue boxes the ground tround-truth, red boxes the model predictions.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
See the last paragraph in Section 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See the second paragraph in Section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
Our work mainly involves empirical contributions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide our code in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification:
We provide our code and dataset information in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cc/ubblic guddesSCodeeubmissionn
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (htips) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed experimental settings in Section 4.1 and Appendix A In the supplementary materials, we provide our code, which contains more details of the model and parameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: See the third paragraph "Visual Question Answering" in Section 4.4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss our model's 's computational resources in Appendix A.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See the last paragraph in Section 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
We are using publicly available datasets for all experiments.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all used public datasets in Section 4 and Appendix A.I are publicly available. They under non-commercial license.
All datasets
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssitheode. seterssiihhhddeccc/aaaaatttt has curated licenses for some datasets. Their licensing guide can help determine the license of
dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code and pre-trained model will be released later for the assets. We are using publicly available datasets for all experiments. No personally identifiable information is involved.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human subjects were involved in experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No human subjects were involved in experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "KSd4akPbfa", "reviewer": "Reviewer_8ZRy", "summary": "This paper addresses the issue that current VQ codebook lacks multimodal information, and proposes to introduce language guidance to learn a more comprehensive one. It develops semantic alignment module and relationship alignment module with multiple additional losses to regularize the codebook with additional image captions and pretrained text encoders. The proposed method works on the codebook learning independently and can be applied to any VQ-based backbones. It is evaluated on various downstream applications and achieves enhanced performance compared to its baselines.", "questions": "- The proposed method requires the training images to have gt captions. How do you generate captions for image-only datasets?\n\n- Is there a chance that the proposed method can be applied to other 2-stage generation frameworks, such as text-to-image latent diffusion's (especially the VQ-regularized) VAE? There is only language guidance on the diffusion process but not on the bottleneck latents/codebook.", "limitations": "- The limitations and border impacts are discussed in the paper Sec. 5.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses the lack of multimodal information in current codebooks, which lacks of notice in previous work, and proposes novel modules to incorporate language information and enhances their semantic alignments.\n\n- The proposed method is evaluated on various tasks and reaches good results surpassing baselines. Thorough ablation studies and analyses into its learned space are also performed.", "weaknesses": "- The proposed method involed additional language information (not only the pretrained CLIP model but also the raw language materials) during training, and in some downstream applications like the text-to-image generation, while most baselines/backbones don't incorporate such input at all. This might raise concerns on the comparison fairness and it's better to be specially noted.\n\n- More non-language-conditioned image generation results should be displayed and compared with baselines/backbones, since the same reason above. According to the paper the language-guided codebook not only can parse additional text condition input but also itself has learned a better latent space overall, so this needs to be more clearly demonstrated. Figs. 13 and 17 show image-to-image translation without language, while they're not compared with any other methods. The most basic unconditional image generation should be evaluated and compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 8ZRy for the positive review and valuable feedback. We would like to address your questions below one by one.\n\n> Q1: The proposed method involved additional language information (not only the pretrained CLIP model but also the raw language materials) during training. This might raise concerns on the comparison fairness.\n\nThank you again for taking the time to provide your valuable comments. For a fair comparison, we select VQCT[1], a novel work recently accepted and published by CVPR-2024, as the baseline. VQCT extracts many visual-related words from **a large amount of text**, and designs a novel codebook transfer network based on **the pretrained CLIP model** to learn the visual codebook. We provide comparisons for image reconstruction, semantic image synthesis, image captioning, and VQA in Table R1.1 below.  From these results, we can see that our method outperforms VQCT in all tasks, which suggests the effectiveness of our method. We also provide experimental results on integrating our method into VQCT further to verify the effectiveness and versatility of our method. Please refer to [the rebuttal pdf](https://openreview.net/attachment?id=QfryItnYya&name=pdf). We will include these experimental results and detailed discussion in our paper.\n\n**Table R1.1 Comparison of VQCT and our method on reconstruction, semantic synthesis, and visual text reasoning tasks.**  \n\n| Model     | Codebook Size | #Tokens | CelebA-HQ | CUB-200  | MS-COCO  |\n| --------- | :-----------: | :-----: | :-------: | :------: | :------: |\n| VQCT      |     6207      |   512   |   5.02    | **2.13** |   9.82   |\n| VQ-GAN+LG |     1024      |   256   |   5.34    |   3.08   |  10.72   |\n| CVQ+LG    |     1024      |   256   | **4.90**  |   3.33   | **9.69** |\n\n|       | Semantic Synthesis on CelebA-HQ |                 | Image Captioning on CUB-200 |                  |                   |   VQA on MS-COCO   |\n| ----- | ------------------------------- | --------------- | --------------------------- | ---------------- | ----------------- | :----------------: |\n| Model | FID$\\downarrow$                 | BLEU4$\\uparrow$ | ROUGE-L$\\uparrow$           | METEOR$\\uparrow$ | CIDEr-D$\\uparrow$ | Accuracy$\\uparrow$ |\n| VQCT  | 14.47                           | 1.38            | 26.50                       | 24.63            | 98.22             |       40.42        |\n| Our   | **11.46**                       | **1.69**        | **34.73**                   | **25.78**        | **102.77**        |     **40.97**      |\n\n\n\n\n\n> Q2: Figs. 13 and 17 show image-to-image translation without language, while they're not compared with any other methods. The most basic unconditional image generation should be evaluated and compared.\n\nWe appreciate your feedback to help us further improve the quality of our paper. Actually, we have provided the quantitative evaluation for Figure 13 in Table 5, row 2 of our original paper. Following your suggestions, we report the results for Figure 17, as shown in Table R1.2 below. Additionally, we conduct the unconditional image generation in Table R1.3 below. From the results, we can observe that our method achieves the best performance, which suggests our method's effectiveness. Moreover, we also provide some generation examples in Figure A of [the rebuttal pdf](https://openreview.net/attachment?id=QfryItnYya&name=pdf). We will include these experimental results in our paper.\n\n**Table R1.2 Comparison of image completion on CelebA-HQ.**  \n\n| Model     | FID$\\downarrow$ |\n| --------- | :-------------: |\n| VQ-GAN    |      9.02       |\n| VQ-GAN+LG |      8.14       |\n\n**Table R1.3 Comparison of unconditional image synthesis on CelebA-HQ.**  \n\n| Model         | FID$\\downarrow$ |\n| ------------- | :-------------: |\n| Style ALAE[2] |      19.2       |\n| DC-VAE[3]     |      15.8       |\n| VQ-GAN     |      10.2       |\n| VQ-GAN+LG     |       9.1       |\n\n\n\n> Q3: The proposed method requires the training images to have gt captions. How do you generate captions for image-only datasets?\n\nThis is a good question. One possible way is to use existing pretrained visual-language models (VLMs) (e.g., MiniGPT-v2 [4], or ShareGPT4V [5]) to generate image captions. More importantly, by employing specific prompt strategies with these VLMs, we can obtain more detailed captions that contain richer and more useful knowledge. We believe that exploring this approach may further improve VQ-based model. We are very interested in exploring it in future work.\n\n\n\n> Q4: Is there a chance that the proposed method can be applied to other 2-stage generation frameworks, such as text-to-image latent diffusion's (especially the VQ-regularized) VAE? \n\nYes, our method can be applied to other 2-stage generation frameworks. To verify this, following LDM [6], we use our method to conduct a text-to-image generation on the CUB-200 dataset. The experimental results are shown in Table R1.4 below. From the results, we can see that our method can further improve performance, verifying the versatility and effectiveness of our method. We also provide a qualitative comparison in Figure B of [the rebuttal pdf](https://openreview.net/attachment?id=QfryItnYya&name=pdf). \n\n**Table R1.4 Comparison of text-to-image on CUB-200 based on LDM.**  \n\n|         | Text-to-Image on CUB-200 |\n| ------- | :----------------------: |\n| Methods |     FID$\\downarrow$      |\n| LDM     |          35.13           |\n| LDM+LG  |          34.75           |\n\n\n\n------\n\n[1] Zhang B, et al. Codebook Transfer with Part-of-Speech for Vector-Quantized Image Modeling. CVPR 2024.\n\n[2] Parmar G, et al. Dual contradistinctive generative autoencoder. CVPR 2021.\n\n[3] Pidhorskyi S, et al. Adversarial latent autoencoders CVPR 2020.\n\n[4] Chen J, et al. Minigpt-v2. arXiv 2023.\n\n[5] Chen L, et al. Sharegpt4v: Improving large multi-modal models with better captions. arXiv 2023.\n\n[6] Rombach R, et al. High-resolution image synthesis with latent diffusion models. CVPR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wGzNCL2akA", "reviewer": "Reviewer_heW4", "summary": "This paper proposed a new codebook learning method to improve the\nperformance of multi-modal downstream tasks.\nThe proposed method can be easily integrated into existing VQ models and shows performance improvements in various cross-modal tasks such as Image Generation and Visual Text Reasoning.", "questions": "In the experiments section, there is only a performance comparison among VQ models such as VQ-VAE and VQ-GAN. A comparison with existing image generation models or visual text reasoning models is needed.", "limitations": "The authors discussed both the limitations and potential negative social impacts in the Conclusion section.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to follow.\n\nThe proposed method can be easily integrated into existing VQ models.", "weaknesses": "The main concern about this paper is its novelty. Although the method of enhancing performance by incorporating language-guided information has been proposed, there is concern that the performance improvement may come from the image-text knowledge of the pre-trained CLIP model rather than the proposed method itself.\n\nThe performance improvement is marginal. \nLG-VQ method outperforms baseline methods in several tasks but performance improvement looks marginal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and valuable feedback. We are glad that the reviewer found that our paper is well-written and easy to follow. Please see below for our responses to your comments.\n\n> Q1: The main concern about this paper is its novelty. \n\nWe would like to highlight the novelty of our paper from the following three aspects,\n\n- i. We **point out the limitations of existing VQ-based methods, which is that they learn a single-modal codebook**. This motivates us to propose a novel multi-modal codebook learning framework. This idea is very reasonable and novel.\n- ii. Our LG-VQ framework is designed **for the first time**, to utilize pretrained text semantics as supervised information to learn a text-aligned codebook. For the novelty, our whole framework design is not trivial and very challenging. Because of the need to fully understand the relationship between text and vision and the characteristics of various downstream tasks.   We thereby carefully design two alignment modules.  Moreover, these two modules are not independent of each other. The semantic alignment module serves the relationship alignment module. Finally, our method can be easily integrated into existing VQ-based models. \n- iii. From the experimental results, our method can **consistently improving performance** of existing VQ-based model codebook representation, reconstruction, and various downstream tasks. \n\nBased on the above three points, we believe that our work brings new insights for VQ-based model.\n\n> Q2: The performance improvement may come from the image-text knowledge of the pre-trained CLIP model rather than the proposed method itself.\n\nWe select VQCT [1], a novel work recently accepted and published by CVPR-2024, as the baseline.  VQCT extracts many visual-related words from **a large amount of text**, and designs a novel codebook transfer network based on **the pretrained CLIP model** to learn the visual codebook. Detailed experimental results are provided in Table R2.1 below. The results suggest that the performance improvement comes from our method rather than only the text knowledge of the pre-trained CLIP model. We also provide experimental results on integrating our method into VQCT further to verify effectiveness and versatility of our method. Please refer to [the rebuttal pdf](https://openreview.net/attachment?id=QfryItnYya&name=pdf).\n\n**Table R2.1 Comparison of VQCT and our method on reconstruction, semantic synthesis, and visual text reasoning tasks.**  \n\n| Model     | Codebook Size | #Tokens | CelebA-HQ | CUB-200  | MS-COCO  |\n| --------- | :-----------: | :-----: | :-------: | :------: | :------: |\n| VQCT      |     6207      |   512   |   5.02    | **2.13** |   9.82   |\n| VQ-GAN+LG |     1024      |   256   |   5.34    |   3.08   |  10.72   |\n| CVQ+LG    |     1024      |   256   | **4.90**  |   3.33   | **9.69** |\n\n|       | Semantic Synthesis on CelebA-HQ |                 | Image Captioning on CUB-200 |                  |                   |   VQA on MS-COCO   |\n| ----- | ------------------------------- | --------------- | --------------------------- | ---------------- | ----------------- | :----------------: |\n| Model | FID$\\downarrow$                 | BLEU4$\\uparrow$ | ROUGE-L$\\uparrow$           | METEOR$\\uparrow$ | CIDEr-D$\\uparrow$ | Accuracy$\\uparrow$ |\n| VQCT  | 14.47                           | 1.38            | 26.50                       | 24.63            | 98.22             |       40.42        |\n| Our   | **11.46**                       | **1.69**        | **34.73**                   | **25.78**        | **102.77**        |     **40.97**      |\n\n\n\n\n\n> Q3: The performance improvement is marginal.\n\nWe would like to emphasize that the performance improvement of our method is not marginal. Here are the detailed performance gains of our VQ-GAN+LG compared to the baseline VQ-GAN:\n\n* Image Reconstruction (FID): Improvement of **1.87% to 15.49%**\n\n* Text-to-Image: Improvement of **17.52%**\n\n* Image Captioning: Improvement of **3.98% to 31.01%**\n\n* VQA (Accuracy): Improvement of **8.32%**\n\n- Image Completion: Improvement of **9.76%** (as detailed in our response to reviewer 8ZRY Q1 Table R1.2)\n- Unconditional Image Generation: Improvement of **10.78%** (as detailed in our response to reviewer 8ZRY Q1 Table R1.3)\n\n\n\n> Q4: Compared with other models in image generation and visual text reasoning.\n\nFollowing your suggestions, we provide a comparison with other models on the text-to-image (in Table R2.2) and VQA (in Table R2.3). From the results, we can see that our method achieves superior performance in image generation.  On VQA task, our method can significantly improve the performance of  VQ-GAN, which suggests our method is effective. But its results are still far lower than the performance of specialized VQA-based models. **This is shortcoming of the VQ-based model** and does not mean that our method is ineffective. We have acknowledged this limitation in Section 5 of our paper. \n\n**Table R2.2 Comparison of text-to-image.**  \n\n|                     | Text-to-image on CelebA-HQ |\n| ------------------- | :------------------------: |\n| Model               |      FID$\\downarrow$       |\n| Unite and Conqu [1] |           26.09            |\n| Corgi [2]           |           19.74            |\n| LAFITE [3]          |           12.54            |\n| VQ-GAN+LG           |           12.61            |\n| CVQ+LG              |         **12.33**          |\n\n**Table R2.3 Comparison of VQA.**  \n\n|              |   VQA on COCO-QA   |\n| ------------ | :----------------: |\n| Model        | Accuracy$\\uparrow$ |\n| HieCoAtt [4] |       62.50        |\n| OAM-VQA [5]  |       75.22        |\n| VQ-GAN       |       37.82        |\n| VQ-GAN+LG    |       40.97        |\n\n\n\n------\n\n[1] Nair N G, et al. Unite and conquer. CVPR 2023.\n\n[2] Zhou Y, et al. Corgi. CVPR 2023.\n\n[3] Zhou Y, et al. LAFITE. CVPR 2022.\n\n[4] Lu J, et al. HieCoAtt. NIPS 2016.\n\n[5] Li P, et al. OAM-VQA. AAAI 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jlAXM9SL1e", "reviewer": "Reviewer_YeAE", "summary": "This paper considers the limitations in learning an expressive codebook and \nproposes a novel multi-modal codebook learning method. The authors propose two semantic supervision modules and three losses. Experiments demonstrates the effectiveness of their method.", "questions": "1. How can the proposed method be used in downstream tasks, and will it incur additional computational overhead?", "limitations": "As stated by the authors, this work assumes each word aligns with a code, which offers a limited representation.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThis work is easy to follow, and its motivation is interesting. \n2.\tExperiments demonstrates the effectiveness of the method.", "weaknesses": "1. It is not easy to obtain major objects and relations between different objects. For example, in Figure 1, ‘the net’, the people beside the court, and the chairs are also very obvious objects. Is it necessary to extract the information of these objects as well?\n\n2. The method shown in Figure 1 uses additional detectors to extract the visual features of the corresponding objects. However, there is no description of these detectors in the article, making it unclear.\n\n3. The authors consider the relationship between text and feature embedding. The effectiveness of the method should be further validated on grounding tasks. (e.g., refcoco)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and valuable feedback. We are glad that the reviewer found that our paper is easy to follow and motivation is interesting. Please see below for our responses to your concerns and questions.\n\n> Q1: It is not easy to obtain major objects and relations between different objects. For example, in Figure 1, ‘the net’, the people beside the court, and the chairs are also very obvious objects. Is it necessary to extract the information of these objects as well?\n\nIn fact, we assume that image captions contain the image's main objects, and the pretrained text semantics contain the relationship between objects. Moreover, we do not consider objects that exist in the image but not in the captions (e.g. 'the net' you mentioned), because it is not easy to obtain these objects as you mentioned.\n\n\n\n> Q2: The method shown in Figure 1 uses additional detectors to extract the visual features of the corresponding objects. However, there is no description of these detectors in the article, making it unclear.\n\nWe would like to claim that we do not use any additional detectors. In our paper, we only introduce image captions. The main purpose of Figure 1 is to illustrate the motivation of our introduction of the relationship alignment module.\n\n\n\n>  Q3: The authors consider the relationship between text and feature embedding. The effectiveness of the method should be further validated on grounding tasks. (e.g., refcoco)\n\nFollowing your suggestions, we conduct visual grounding task on refcoco dataset using the codebook trained on the MS-COCO dataset. The results are shown in Table  R3.1 below, which confirm our method's effectiveness. We also provide a qualitative comparison in Figure C of [the rebuttal pdf](https://openreview.net/attachment?id=QfryItnYya&name=pdf). We will include these experimental results in our paper. \n\n**Table R3.1 Comparison of Visual Grounding on refcoco dataset.** \n\n|           | Visual Grounding on refcoco |\n| --------- | :-------------------------: |\n| Model     |   Accuracy(0.5)$\\uparrow$    |\n| VQ-GAN    |            9.14             |\n| VQ-GAN+LG |            9.62             |\n| Improve   |          **5.25%**          |\n\n> Q4: How can the proposed method be used in downstream tasks, and will it incur additional computational overhead?\n\nWe provide experiment details on how to leverage the learned codebooks for various downstream tasks in Appendix A.1 of our original paper. For all downstream tasks, we follow established experimental settings from existing works. For example, we follow VQ-GAN-Transformer for semantic image synthesis and VQ-diffusion for text-to-image.\n\nIt does not incur any additional computational overhead when it is used for various downstream tasks. We provide the training parameters for reconstruction and downstream tasks as well as the approximate training time cost (seconds), as shown in Table R3.2 below. From the results, we can observe that our method only adds extra training parameters (i.e., 7.8M) and time (i.e., 0.04s) during reconstruction training, but both are negligible.\n\n**Table R3.2 Model training parameters and time cost in different tasks.**\n\n|           | Image Reconstruction |                       | Semantic Image Synthesis |                       |   Text-to-Image   |                       |\n| --------- | :------------------: | :-------------------: | :----------------------: | :-------------------: | :---------------: | :-------------------: |\n| Methods   |  #Train Params [M]   | Train time (sec/iter) |    #Train Params [M]     | Train time (sec/iter) | #Train Params [M] | Train time (sec/iter) |\n| VQ-GAN    |         74.9         |         0.83          |           204            |         0.46          |        496        |          0.6          |\n| VQ-GAN+LG |       74.9+7.8       |         0.87          |           204            |         0.46          |        496        |          0.6          |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PsVeFBMqcF", "reviewer": "Reviewer_WVeZ", "summary": "This paper points out that most of existing vector quantization methods learn a single-modal codebook, resulting in suboptimal performance when the codebook is applied to multi-modal downstream tasks. To address this issue, the authors propose a new language-guided codebook learning framework, called LG-VQ, which learns a codebook that can be aligned with the text to improve the performance of multimodal downstream tasks. Specifically, with the pretrained text semantics as prior knowledge, two novel alignment modules are devised to transfer such prior knowledge into codes for achieving codebook text alignment. The proposed method is evaluated on reconstruction and various multi-modal downstream tasks, and achieves superior performance.", "questions": "please try to address the weaknesses.", "limitations": "n/a.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1)\tThe authors point out the limitation of existing single-modal quantization methods and propose a novel multi-modal codebook learning method.\n2)\tTwo semantic supervision modules are designed to effectively learn text-aligned codebook.\n3)\tComprehensive experiments have been conducted and prove the effectiveness of the proposed method.\n4)\tThe paper is well-written and easy to follow.", "weaknesses": "- The details of the relationship alignment module are not clear. Fig. 3 is not easy to understand. The sec. 3.2.2 should be reorganized to clearly introduce the relationship alignment module.\n- There may exist errors in the denominator of Eq. (5).\n- In the ablation study, the performance of only adopting the relationship alignment module (+L_{ras}) should also be evaluated. Could this module be used separately?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for the reviewer’s time and valuable feedback. We are glad that the reviewer found that our paper is easy to follow and well-written. Please see below for our responses to your comment and questions.\n\n> Q1: The details of the relationship alignment module are not clear. Fig. 3 is not easy to understand. The sec. 3.2.2 should be reorganized to clearly introduce the relationship alignment module.\n\n\nThanks for your valuable comment. We provide a more detailed explanation of the relationship alignment module:\n\n**Motivation:**   Our motivation is that when the codebook is applied to the VQA task (e.g. to answer the question \"What is this woman holding?\" in Figure 1), the model needs to identify key visual objects \"women\" and \"racket\" from the image's codes, and also needs to understand the semantic relationship between the objects. However, existing VQ-based methods train codebook independently and do not align text, resulting in their poor performance.\n\n**Our idea:** The pre-trained word embeddings contain rich semantic relationship knowledge, which is beneficial to VQA tasks. Therefore, we propose a relationship alignment module that aims to transfer the semantic relationship between words into codes. which helps the model better understand images for complex reasoning tasks and improves alignment between codes and words.\n\nWe provide a more detailed explanation of Figure 3:\n\n**Input**: given any two words of a sentence (e.g., 'girl' and 'earring'), we encode two words using pretrained CLIP model. \n\n**Alignment**:  we find two codes closest to two words, i.e., code 2 and code 4, based on the similarity between the word embeddings and $Z_{vq}$.\n\n**Mapping:**  we can get the code 2 embedding and code 4 embedding from Z based on index id (i.e., 2 and 4).\n\n**Relationship Transfer ($L_{ras}$):**  we constrain the cosine similarity of code 2 and code 4 embedding to match the cosine similarity between 'girl' and 'earring' embedding.\n\nWe will reorganize Section 3.2.2, improve Figure 3,  and add it to our final version. \n\n\n> Q2: There may exist errors in the denominator of Eq. (5).\n\nThanks for your construction comment. We will fix it in our final version.\n\n\n\n> Q3: In the ablation study, the performance of only adopting the relationship alignment module (+L_{ras}) should also be evaluated. Could this module be used separately?\n\nFollowing your suggestions, we conduction an ablation study on $L_{ras}$ on TextCaps, as shown in Table R4.1. Specifically, we use a layer of MLP network to replace $Z_{vq}$ to learn the alignment between code and word. This result suggests our $L_{ras}$ is effective.\n\n**Table R4.1 Ablation study of our $L_{ras}$ loss function on TextCaps.** \n\n|                                           |       TextCaps       |\n| ----------------------------------------- | :------------------: |\n| Setting                                   | FID$\\downarrow$ |\n| VQ-GAN                                    |        24.08         |\n| + $\\mathcal{L}_{ras}$                     |        23.81         |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## Global Response

We sincerely thank all the reviewers for the thorough reviews and valuable feedback and will incorporate their suggestions into our next revision. We are glad to hear that the motivation is interesting (Reviewer YeAE), the paper is well-written (Reviewer YeAE) and easy to follow (Reviewer heW4, YeAE, and WVeZ), the proposed method is general and can be easily applied to VQ-based models (Reviewer 8ZRy, heW4, and WVeZ). 

We have attempted to address all concerns, which has significantly improved the manuscript. We here summarize and highlight our responses to the reviewers:

* We introduce a novel work VQCT, recently published in CVPR 2024 and utilizing additional language and pretrained CLIP models. We provide detailed experimental comparisons on various tasks to address Reviewer 8ZRy's concern (unfair comparison) and Reviewer heW4's concern (the performance improvement comes from the image-text knowledge of the pre-trained CLIP model rather than our method).
* We provide experimental comparisons on many non-language image generation tasks and other 2-stage generation framework (Reviewer 8ZRy), experimental comparisons with other models in image generation and VQA (Reviewer heW4), experimental comparisons on visual grounding task (Reviewer YeAE), and ablation study on $L_{ras}$, to show the effectiveness of our method. 
* We provide detailed analysis and discussion on Reviewer 8ZRy‘s’ question how do you generate captions for image-only datasets? We think this could be an exciting future work that will further advance the VQ-based community. We also add more detailed explanation about Reviewer heW4's questions (novelty and performance improvement) Reviewer YeAE's questions (Figure 1 and additional computational overhead) and Reviewer WVeZ's questions (the relationship alignment module and Figure 3).
* We provide some examples and experimental comparisons in the rebuttal pdf.

We reply to each reviewer's concerns in detail below their reviews. Please kindly check out them. Thank you and please feel free to ask any further questions. Finally, we feel this paper makes an important, helpful contribution (i.e, addressing the lack of multimodal information in current codebooks). Our method can significantly improve the performance of existing VQ-based methods in codebook representation, reconstruction, and various downstream tasks.

Meta-review: The reviewers collectively appreciate the novel approach of integrating language guidance into codebook learning to enhance multimodal tasks, recognizing the method's effectiveness across various applications. While concerns about comparison fairness, clarity in specific module descriptions, and marginal performance improvements were raised, the reviewers acknowledge that these issues have been mostly resolved by the rebuttal. And in summary, the overall consensus is that the paper is well-written, the method is innovative, and the results are promising.  Based on these strengths, I recommend accepting the paper.
Decision: Accept (poster)