LG-VQ: Language-Guided Codebook Learning

Authors: guotao  liang
baoquan  zhang
yaowei  wang
xutao  li
yunming  ye
huaibin  wang
chuyao  luo
kola  ye
linfeng  luo
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 09:16:02

Abstract: Vector quantization (VQ) is a key technique in high-resolution and high-fidelity image synthesis, which aims to learn a codebook to encode an image with a sequence of discrete codes and then generate an image in an auto-regression manner. 
  Although existing methods have shown superior performance, most methods prefer to learn a single-modal codebook (\emph{e.g.}, image), resulting in suboptimal performance when the codebook is applied to multi-modal downstream tasks (\emph{e.g.}, text-to-image, image captioning) due to the existence of modal gaps.
  In this paper, we propose a novel language-guided codebook learning framework, called LG-VQ, which aims to learn a codebook that can be aligned with the text to improve the performance of multi-modal downstream tasks. Specifically, we first introduce pre-trained text semantics as prior knowledge, then design two novel alignment modules (\emph{i.e.}, Semantic Alignment Module, and Relationship Alignment Module) to transfer such prior knowledge into codes for achieving codebook text alignment.   
  In particular, our LG-VQ method is model-agnostic, which can be easily integrated into existing VQ models. Experimental results show that our method achieves superior performance on reconstruction and various multi-modal downstream tasks.

Main paper: 
 introduction: 
 In recent years, with the growing development of various multi-modal task scenarios [37, 36, 38] , unified modeling of visuals and language has sparked considerable interest. Vector Quantization (VQ)-based image modeling technique, exemplified by VQ-VAE [43] and VQ-GAN [9] , has emerged as a pivotal approach in the realm of unified modeling. The VQ methodology [43] typically follows a two-stage generation paradigm. In the initial stage, a trainable discrete codebook is employed to quantize continuous image features into a discrete token sequence to finish the reconstruction task. Subsequently, the codebook is utilized for various downstream tasks by generative models [42, 37] .Learning a robust codebook during the initial stage is crucial for optimizing performance in downstream tasks. At present, lots of VQ methods have been proposed to achieve robust code representation [15, 14, 7, 12] . For instance, VQ-GAN [9] introduces an adversarial training loss to learn a perceptually rich codebook. Some other works consider improving the codebook representation from the perspective of addressing the problem of codebook collapse [53, 52] .Although existing methods have shown superior performance, most methods only focus on learning a single-modal codebook contains more low-level information (e.g., image's pixel, edge, and texture), resulting in suboptimal performance when the codebook is applied to multi-modal downstream tasks (e.g., text-to-image [37] , image captioning [36] , VQA [24] ). That is because the codebook lacks high-level semantics and the existence of modal gaps.To address the above issue, we propose a novel codebook learning method (i.e., multi-modal codebook learning), called Language-Guided VQ (LG-VQ). The novelty lies in utilizing pre-trained text semantics as supervised information to guide the codebook to learn abundant multi-modal knowledge.Q: What is this woman holding? To answer the question, one not only needs to identify "women" and "racket" but also understand the semantic relationship between them ("holding"). Specifically, we first employ a cross-modal pre-trained model (i.e., CLIP [32] ) to encode text semantics. Then, we propose two novel semantic supervision modules to transfer the text semantics into codebook, i.e., Semantic Alignment Module, and Relationship Alignment Module. Within the semantic alignment module, we enhance the consistency between the semantic representations of the codebook and text through global semantic alignment and masked text prediction. On the other hand, simply aligning the text and codebook in the holistic semantic space cannot satisfy more complex reasoning tasks like image captioning and VQA. Inspired by some VQA techniques [26, 40, 24] , the semantic relationships between words play a very important role in various tasks of natural language processing (See Fig. 1 ). Based on this fact, we further propose to transfer the semantic relationships between words into codes to achieve better alignment between the codes and words. Such a text-aligned codebook helps alleviate modal gaps and improve codebook performance on cross-modal tasks.The contributions of this work are summarized as follows:• We point out the limitations of existing methods in learning an expressive codebook since they learn a single-modal codebook. We propose a novel multi-modal codebook learning method, named LG-VQ, which can enable the codebook to effectively retain fine-grained reconstruction information while aligning with the text.• Resorting to pre-trained text semantics, we propose two novel semantic supervision modules, i.e., Semantic Alignment Module and Relationship Alignment Module, effectively learn text-aligned codebook. The advantage of such alignment modules is the abundant context and relationship semantics contained in pre-trained text can be sufficiently leveraged for enhancing multi-modal codebook learning.• We conduct comprehensive experiments on four public datasets, which shows that our LG-VQ method outperforms various state-of-the-art models on reconstruction and various cross-modal tasks (e.g., text-to-image, image captioning, VQA).2 Related Works 
vector quantization for image generation: 
 Vector quantization (VQ) is designed to learn a codebook, which aims to encode continuous image features into a discrete sequence. Then, the learned codebook can be utilized for various downstream tasks. Oord et al. [43] first propose a novel VQ method called VQ-VAE. This method innovatively replaces the prior distribution of Variational Autoencoder (VAE) with a discrete deterministic distribution (i.e., a codebook). To further improve the performance of VQ, various models are proposed to learn a more expressive codebook [9, 48, 2, 17, 7, 15, 14, 21] . For example, VQ-GAN [9] addresses the issue of image blur generated by VQ-VAE through the introduction of an adversarial training loss. However, the above methods do not tackle the codebook collapse issue. To address the issue, many novel methods are proposed from the perspective of regularization [33] , codebook update [53] , codebook transfer [51] . Recently, inspired by the large language models (LLMs), instead of mapping images to the visual code tokens, some works attempt to map the images to the word tokens of LLMs by viewing images as "foreign languages" [22, 50, 56] . However, because of the inherent differences between vision and language, these works have difficulty assigning correct semantic words to images. Compared with the aforementioned methods, our approach focuses more on multi-modal alignment in feature space (i.e., learning a text-aligned codebook). We use pre-trained text semantics to supervise the codebook learning. The advantage is that the rich semantic information from the text can be fully exploited for more robust codebook learning so that the codebook can not only retain more reconstruction information but also be able to understand and match text. More importantly, our method is model-agnostic, which can be easily integrated into existing VQ models.Vision-language Pre-training (VLP) aims to learn multi-modal representations from large-scale imagetext pairs that can improve vision-language downstream tasks, for example, VQA [1] . Early methods such as LXMERT [41] , UNITER [4] employ pre-trained object detectors to extract image region features, and fuse image features with text by a cross-modal encoder to achieve the vision-language representation learning. Although these methods achieve superior performance on downstream tasks, they require high-resolution input images and pre-trained object detectors. To remove the object detectors, a large number of researchers focus on learning two separate representations for image and text [32, 16, 18] . For instance, CLIP [32] learns a robust representation for each image and text using contrastive learning based on large-scale image-text pair data.In this paper, we propose to employ pre-trained text semantics as supervised information to guide codebook learning. Its advantage is that abundant multi-modal knowledge contained in text can be fully leveraged for robust codebook learning. Additionally, we design a novel relationship alignment module to inject semantic relationships between words into codes.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: See the second paragraph in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: vq-vae: 
 VQ-VAE [43] , as a pioneering work on the VQ research domain, aims to learn a discrete codebook to encode images into discrete token sequences through an Encoder-Decoder framework. As illustrated in Fig. 2 right, the VQ-VAE consists of a visual encoder E θe (•) with parameter θ e , a token decoder D θ d (•) with parameter θ d , a quantizer Q(•), and a codebook is defined as Z = {e k } K k=1 that consists of learnable K entries e k ∈ R dz with dimension d z . Given an input image x ∈ R H×W ×C , where H, W , and C represent the height, width, and channel of the image respectively. The visual encoder E θe (•) learns to convert the original image into grid featuresẐ = E θe (x) ∈ R H f × Wf ×dz and f is the down-sampling factor. The quantizer Q(•) looks up the nearest neighbor in the codebook for each grid representation ẑi ∈ R dz in Ẑi using the following equation:EQUATIONThe token decoder D EQUATIONwhere, the first term is reconstruction loss, which measures the difference between the original image x and the reconstructed image x. sg[•] represents the stop-gradient operator, and the second term is codebook loss, which encourages the codebook to be close grid features. The third term is the "commitment loss" [43] , where ω serves as a hyper-parameter. However, existing VQ-based methods mainly focus on the learning of single-modal codebook, thereby limiting their applicability to multi-modal downstream tasks.Existing works attempt to improve codebook reconstruction capabilities to obtain better performance on downstream tasks. However, ignoring modal differences results in suboptimal performance when the codebook is applied to cross-modal tasks. To address this issue, we propose to utilize the pretrained text semantics as supervised information to learn a text-aligned codebook. Its advantage is abundant semantic information from text can be fully exploited for more robust codebook learning to improve the performance of reconstruction and cross-modal tasks. The comprehensive architecture of the proposed LG-VQ method is illustrated in Fig. 2 left. It consists of two supervision modules: Semantic Alignment Module (i.e., L gsa and L mtp ), and Relationship Alignment Module (i.e., L ras ).The first module encourages global semantic consistency between the codebook and text. The second module aims to transfer the rich semantic relationship between words into codes. Next, we introduce these two modules in detail.Considering that paired image and text data have consistent semantic information and the missing information of masked data can be completed from the other modality, we propose global semantic alignment, which aims to enhance the consistency of global semantics between text and visual codes, and masked text prediction, which uses visual codes to restore the masked words. Next, we discuss how to align text and codebook in the semantic space.Text Information Encoder: Instead of jointly training text and codebook from scratch, we employ a pre-trained cross-modal model CLIP [32] to encode text information. Its advantage is that such text information already has good cross-modal semantic knowledge and is beneficial for codebook learning. Specifically, for a given text description of an image t = {w SOT , w 1 , w EQUATIONFinally, we employ InfoNCE [29] , which maximizes the similarity between visual and text in the global representation, as our learning objective, where B is the batch size, s(•, •) is cosine similarity:L gsa = - i∈B log exp(s(e i CLS , e i EOT ))j∈B exp(s(e i CLS , e j EOT )).(EQUATIONMasked Text Prediction: To further enhance the semantic alignment, we propose to use discrete visual codes to reconstruct the masked words from a more fine-grained perspective, refer to Fig. 2 left. Formally, for a given fixed-length text sequence of n -2, we first randomly sample the masking ratio r from a truncated Gaussian distribution [19] . Subsequently, we randomly mask out r • (n -2) words and replace them with learnable [mask i ] tokens based on their positions i. Next, a self-attention module [44] is employed to learn adaptive masked word embeddings based on unmasked words. The resulting adaptive masked sequence is denoted asT msk = {e SOT , m 1 , e w2 , m 3 , • • • , e EOT },where m i is the mask token embedding at the i-th position in the sequence. Following this, a cross attention decoder f θm (•, •) is employed to predict the masked word tokens given the discrete visual codes Z vt obtained by Eq. 4. Finally, we add a cross-entropy loss H(•, •) between the ground-truth word tokens and the output of the decoder. Let y msk denote a one-hot vocabulary distribution where the ground-truth word token has a probability of 1, f θm (Z vt , T msk ) denote the predicted probability of model for masked word tokens. That is:EQUATIONWhile the two aforementioned loss functions for achieving good alignment at holistic semantic space have demonstrated initial promise, they cannot satisfy more complex reasoning tasks like image captioning and VQA. Inspired by some VQA techniques [26, 40, 24, 1] , the semantic relationships between pre-trained words play a very important role in complex text reasoning tasks. For instance, as shown in Fig 1,  to answer question ("What is this woman holding?"), one needs to fully understand the visual objects "women", "racket", and semantic relationship between them ("holding"). Based on the above fact, we propose to transfer the semantic relationship between words into codes. Such semantic relationships enable the model to better understand the image for addressing complex reasoning tasks.Step 1: AlignmentEQUATIONStep 2: Mapping 0.6 0.5Step 3: ℒ But unfortunately, there is an issue there is no alignment between words and codes. Thanks for the above two losses that have provided semantic alignment of text and visual codes. To achieve the above idea, as shown in Fig. 3 , we first use Z vt to align with words. Then, we inject semantic relationships between words into the initial codebook Z, instead of the Z vt . Its advantage is it can prevent codes from collapsing into a single point for learning more diverse representations by relationship limiting. Then, Z vt primarily serves the purpose of aligning words and codes, but it is a crucial step for subsequent processes. Specifically, given any two words of a sentence, we use pre-trained word embedding [32] to encode words, e wi and e wj . We employ cosine similarity to find the index of the code from Z vt that is most similar to the word. Then, one can get code embedding from Z based on the index:e zi = Z[argmax ez∈Z vt[1:]s(e wi , e z ), :],e zj = Z[argmax ez∈Z vt[1:]s(e wj , e z ), :].Next, we consider cosine similarity as a measure of semantic relationships between words and leverage it to establish corresponding relationships between codes achieving semantic relationship transfer. Finally, we utilize the following loss function as learning objective:L ras = (wi,wj )∈t(s(e wi , e wj ) -s(e zi , e zj )) 2 . (8)We use three hyperparameters (i.e., α, β, and γ) to control three losses, respectively. Finally, the overall objective function is: As our method is model-agnostic, we choose recent models, including VQ-VAE [43] , VQ-GAN [9] , and CVQ [53] as our backbone network. Following existing works [52, 53] , we evaluate the reconstruction image quality on two evaluation metrics, i.e., Fréchet Inception Distance (FID) [13] which evaluates the perceptual similarity of reconstructed images and original images, and Peak Signal-to-noise Ratio (PSNR) [10] is employed to measure the pixel-level similarity between the reconstructed and original images.EQUATIONDataset. We evaluate our method on four public datasets, including TextCaps [39] , CelebA-HQ [23] , CUB-200 [45] , and MS-COCO [20] . For CelebA-HQ, CUB-200, and MS-COCO datasets, we use publicly available image captions, CelebA-HQ from [47] , CUB-200 from [34] , MS-COCO from [3] Implementation Details. Following VQ-GAN [9] , all images are reshaped 256 × 256 for reconstruction and generation. Down-sampling factor f is set to 16. The codebook size K is 1024. The batch size is 8. In our experiments, we maintain consistent parameter settings between our methodLG-VQ and the chosen backbone networks (i.e., VQ-VAE [43] , VQ-GAN [9] , and CVQ [53] ) for a fair comparison. For each image, we randomly select a text from multi-text for training. Since our method introduces additional text and pre-trained CLIP model, for a fair comparison, we select VQCT [51] as the baseline for various downstream tasks. VQCT extracts many visual-related words from a large amount of text and designs a novel codebook transfer network based on the pre-trained CLIP model to learn the visual codebook. Table 1 illustrates the image reconstruction performance of our model compared to the backbone model on multiple datasets. It can be observed that our method LG-VQ outperforms all compared methods on most evaluations, which suggests that our method is extremely effective and has strong generality. Compared with FID, our PSNR improvement is marginal, this is reasonable in the VQ research domain, which widely exists in previous VQ methods [21, 52, 53] . The main reason is that PSNR only measures the pixel-level similarity of the images, while FID can effectively measure the diversity and semantic similarity of image generation. Compared with backbone models, the key difference lies in that our method introduces well pre-trained text semantics, which is beneficial to learning a more expressive codebook. This shows the effectiveness of our method. We also provide a qualitative comparison of the image reconstruction performance of different methods, please refer to Appendix A.3 Fig. 10 .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
ablation study: 
 Are our three loss functions both effective? In Table 2 , we conduct an ablation study to show the effectiveness of the proposed three loss functions. Specifically, the VQ-GAN serves as the baseline model (i.e., without introducing any loss). We do not conduct a separate experiment on L ras because this module requires code and words to be well aligned. Based on the results from (i) ∼ (vi), we draw several key conclusions: Firstly, each loss function plays a crucial role in improving the performance of image reconstruction. Secondly, the performance of (iii) outperforms (ii) by a large margin on TextCaps. This is reasonable because TextCaps's texts are richer and more diverse than CUB-200, it can provide more knowledge for more fine-grained alignment between codes and text, which is useful for the learning of a more robust codebook. Thirdly, analyzing the results of (iii) and (iv), injecting word-level semantic relationships into codes is beneficial, which confirms our motivation. Furthermore, the performance of (v) outperforms (i), which is reasonable because the abundant semantic knowledge from pre-trained text can be fully exploited for learning more robust codebook representation. This supports the motivation of learning a multi-modal codebook (i.e., aligned with text). Finally, comparing the results of (vi) with (i)∼(v), fully considering all losses achieves the best performance, indicating the effectiveness of our method.Can our global semantic supervision align vision and language? In Fig. 4 , we provide several image-to-text retrieval cases on CelebA-HQ and CUB-200 datasets based on VQ-GAN+LG. From the figure, it can be observed that our method can accurately retrieve text very similar to the image content, achieving the alignment of vision and language. For example, row 2 examples show that our method can precisely understand some key attributes of images (e.g., "gray hair", "necktie", "big nose" and "chubby") and retrieve similar text. This suggests that the codes learned through our method have obtained good alignment with the text, which verifies the effectiveness of our method. Moreover, such alignment is beneficial for learning robust code representations and enhancing performance in multi-modal downstream tasks.a jet black bird with a bright yellow-orange color on its head and chest, its beat is short, sharp, and black.Top-1 retrieved text a multicolored bird with an orange and white base color with black tips and a red head with a short beak.The person has mouth slightly open, gray hair, and bags under eyes and is wearing necktie. He is smiling, and chubby.The man has sideburns, bags under eyes, big nose, and double chin and is wearing necktie. He is smiling.Top-1 retrieved text Can our codebook accurately predict masked words? To answer this question, we conduct a word prediction task on test data based on VQ-GAN+LG by randomly masking one word or three words of text, as shown in Table 3 . We use Recall@1 as the evaluation metric [18] . From the table, our method demonstrates accurate predictions of masked words, confirming the effectiveness of our approach. Fine-grained word prediction can help the codebook better understand the text semantics, which is crucial for improving the performance of the downstream task. Additionally, several examples in Fig. 5 demonstrate our method's ability that accurately predict subject words (e.g., wings, eyes) and verbs (e.g., has, is, and smiling), further affirming its strong multi-modal understanding capabilities. Can our codebook learn the word semantic relationships? In Fig. 6 , we visualize the cosine similarity between words and the cosine similarity between codes aligned with the words for a certain sample based on VQ-GAN+LG. From the figure, we can see our codes can learn consistent relationships with word semantics compared with VQ-GAN. For example, the similarity "code 33" vs "code 232" (0.46) resembles "wings" vs "chest" (0.49). In addition, we provide a quantitative similarity evaluation between codes and words in Table 4 . From the results, we can find that our codes indeed achieve consistent semantic relationships with words.Is our method effectively learning more diverse code representation? Following [52] , we directly feed each codebook embedding e k (size: Following [9, 37, 11] , we conduct image generation downstream tasks (i.e., text-to-image, semantic synthesis, unconditional generation, and image completion) to fully validate the effectiveness of the learned codebook on CelebA-HQ.1 × 1 × 256) intoText-to-Image. In Table 5 , we compare our LG-VQ with the state-of-the-art models on CelebA-HQ dataset for text-to-image. From the results, our LG-VQ method outperforms baseline methods by a large margin. This is reasonable due to the incorporation of pre-trained text knowledge enabling a comprehensive understanding of the text, which suggests our method's effectiveness. Moreover, we provide some synthesis examples comparing the results of our LG-VQ with baseline methods in Figure 8 , showing the performance in the text-to-image task. From the figure, we can see our method not only comprehensively understands the given text conditions but also excels in generating realistic images compared with baseline methods. For instance, our method can capture the "glasses", "man", "long black hair", and "no beard" key attributions. Semantic Synthesis. Following [9] , we compare with existing semantic synthesis models in Table 6 .Our method achieves the best performance, which suggests our method's effectiveness. We provide some examples in Appendix Figure 13 . Unconditional Generation and Image Completion. Following [9] , we conduct unconditional image generation and Image Completion on CelebA-HQ dataset, as shown in Table 7 and Table 10 . From the results, we can see that our method can significantly improve the performance of VQ-GAN, which is reasonable because pre-trained text can provide rich semantic knowledge for learning more robust codebook representation. This suggests the effectiveness of our method. We provide some examples in Appendix Figure 18 and Figure 17 . We compare two recent work V2L Tokenizer [56] and VQCT [51] . We select VQ-GAN as our backbone network. The results are shown in Table 11 . For the results, we can see that our LG-VQ method outperforms the performance of VQ-GAN. This is reasonable because the pre-trained text provides rich context and relationship semantics for codebook learning, which verifies our motivation for learning a text-aligned codebook to improve the performance of the codebook on cross-modal tasks. On the other hand, the V2L Tokenizer and VQCT cannot achieve very good performance because it is difficult to assign correct semantic language tokens to images. Compared with the V2L Tokenizer, our method utilizes pre-trained text semantics as supervised information. Its advantage is can make the codebook learn semantic information consistent with the text (i.e., learning a textaligned codebook). And, our method is model-agnostic, which can be easily integrated into existing VQ models. Visual Question Answering. We select VQ-GAN and VQCT [51] as the baseline. We conduct the VQA task on the COCO-QA [35] using the codebook trained on the MS-COCO dataset. The results are shown in Table 9 . From the results, we can see that our LG-VQ method significantly improves the performance of VQ-GAN on VQA task (approximately 8.32%↑ on Accuracy). That is reasonable due to we introduce pre-trained text semantics to enable us to obtain a codebook aligned with the text, which is helpful for comprehensively understanding the given text question. This confirms our motivation and the effectiveness of our method. We conduct a visual grounding task on refcoco dataset [49] to validate the effectiveness of the learned MS-COCO's codebook. Following the same metric used in [5] , a prediction is right if the IoU between the grounding-truth box and the predicted bounding box is larger than 0.5. We select VQ-GAN and VQCT [51] as the baseline. The results are shown in Table 8 . From the results, we can see that the performance of our method consistently outperforms VQ-GAN and VQCT, which suggests its effectiveness. We also provide a qualitative comparison in Appendix Figure 19 . For the experimental setting, please refer to Appendix A.1.  
conclusions: 
 In this paper, we propose a novel codebook learning method, named LG-VQ.LG-VQ is a modelagnostic method and can easily be integrated into existing VQ models. In particular, we propose to incorporate pre-trained text semantics into the codebook by two novel supervision modules, i.e., semantic and relationship. Quantitative and qualitative experiments demonstrate the strong generality of our method, showing its ability to improve the performance of the codebook in cross-modal tasks.Limitations. In our current paper, we suppose each word aligns with a code, but it fails to capture some more complex relationships between words and codes (e.g., one code aligns with multiple words). In the future, we plan to investigate the relationships between codes and words. Moreover, although our results show that the performance of VQ in visual text reasoning tasks can be significantly improved, its results are still far lower than the performance of image captioning or VQA models. Broader impact Our paper shows that learning a multi-modal codebook (i.e., a text-aligned codebook) can not only significantly improve the performance of reconstruction but also the performance of the codebook on cross-modal tasks. The potential impact of our research lies in its influence on future studies, specifically in the area of unified modeling of multi-modal understanding and generation.For instance, our work can be extended to interact with LLMs to improve multi-modal understanding and generation capabilities. In particular, our model can be used to generate images or text. It may be exploited to produce some erroneous and unethical information, which needs to be handled carefully before employing our model in practical applications.A Appendix / supplemental material Visual question answering: The COCO-QA [35] dataset is automatically generated from captions in the Microsoft COCO dataset [20] . There are 78,736 train questions and 38,948 test questions in the dataset, These questions are based on 8,000 and 4,000 images respectively. There are four types of questions including object, number, color, and location. Each type takes 70%, 7%, 17%, and 6% of the whole dataset, respectively. All answers in this data set are single word. Following the image captioning task, we use the last hidden embedding to do VQA, as shown in Fig. 9 Visual Grounding: The refcoco dataset [49] includes 19,994 images with 50,000 referred objects. Each object has more than one referring expression, and there are 142,210 referring expressions in this dataset. There are two commonly used split protocols for this dataset. One is RefCOCOg-google [25] , and the other is RefCOCOgumd [28] . We follow RefCOCOgumd [28] to split the dataset. The train set has 42,404 expressions, the validation set has 3,811 expressions, and the test set has 3,785 expressions. Following [5] , we concatenate the image codes and text tokens and feed them into a learnable transformer with coordinate regression layers (i.  
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See the last paragraph in Section 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: We are using publicly available datasets for all experiments.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite all used public datasets in Section 4 and Appendix A.1. All datasets are publicly available. They are under a non-commercial license.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We provide comparisons with VQCT [51] for image reconstruction in Table 12 . Moreover, we also provide experimental results on further integrating our method into VQCT to verify its effectiveness and versatility. The results are shown in Table 13 . We provide more examples of image reconstruction in Fig. 10 , image-to-text retrieval in Fig. 11 and Fig. 12 . We also provide more image synthesis results in Fig. 13 for semantic image synthesis, and text-to-image synthesis in Fig. 14 . We provide some examples of image captioning in Fig. 15 and VQA in Fig. 16 . We provide some examples of unconditional generation in Fig. 18 , and image completion in Fig. 17 . We also provide a qualitative comparison of visual grounding in Fig. 19 . Our: this bird has a white crown and white throat, brown back, and black feet.VQ-GAN: this bird is black and white, and has a gray crown and black breast.Our: this little bird has a yellow belly, wings, brown eyes, long black beak, brown crown and beak.VQ-GAN: this bird has a white light red body, a white or brown blue crown and is a brown throat, and also has a very short crenshaw.Our: a small bird with a blue head, and long, pointed wing bar.VQ-GAN: this bird has wings that are black and white with black spots.Our: this bird has wings that are brown with a red crown and a short point orange bill.VQ-GAN: this small pter is brown in color and has a red crown. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.https://github.com/CompVis/taming-transformershttps://github.com/microsoft/VQ-Diffusion 


Decision: The reviewers collectively appreciate the novel approach of integrating language guidance into codebook learning to enhance multimodal tasks, recognizing the method's effectiveness across various applications. While concerns about comparison fairness, clarity in specific module descriptions, and marginal performance improvements were raised, the reviewers acknowledge that these issues have been mostly resolved by the rebuttal. And in summary, the overall consensus is that the paper is well-written, the method is innovative, and the results are promising.  Based on these strengths, I recommend accepting the paper.

Reviews: review: paper_summary: paper_summary: This paper addresses the issue that current VQ codebook lacks multimodal information, and proposes to introduce language guidance to learn a more comprehensive one. It develops semantic alignment module and relationship alignment module with multiple additional losses to regularize the codebook with additional image captions and pretrained text encoders. The proposed method works on the codebook learning independently and can be applied to any VQ-based backbones. It is evaluated on various downstream applications and achieves enhanced performance compared to its baselines.
strength_weakness: strength_weakness: - The proposed method involed additional language information (not only the pretrained CLIP model but also the raw language materials) during training, and in some downstream applications like the text-to-image generation, while most baselines/backbones don't incorporate such input at all. This might raise concerns on the comparison fairness and it's better to be specially noted.

- More non-language-conditioned image generation results should be displayed and compared with baselines/backbones, since the same reason above. According to the paper the language-guided codebook not only can parse additional text condition input but also itself has learned a better latent space overall, so this needs to be more clearly demonstrated. Figs. 13 and 17 show image-to-image translation without language, while they're not compared with any other methods. The most basic unconditional image generation should be evaluated and compared.
questions: questions: - The proposed method requires the training images to have gt captions. How do you generate captions for image-only datasets?

- Is there a chance that the proposed method can be applied to other 2-stage generation frameworks, such as text-to-image latent diffusion's (especially the VQ-regularized) VAE? There is only language guidance on the diffusion process but not on the bottleneck latents/codebook.
limitations: limitations: - The limitations and border impacts are discussed in the paper Sec. 5.
paper_summary: paper_summary: This paper proposed a new codebook learning method to improve the
performance of multi-modal downstream tasks.
The proposed method can be easily integrated into existing VQ models and shows performance improvements in various cross-modal tasks such as Image Generation and Visual Text Reasoning.
strength_weakness: strength_weakness: The main concern about this paper is its novelty. Although the method of enhancing performance by incorporating language-guided information has been proposed, there is concern that the performance improvement may come from the image-text knowledge of the pre-trained CLIP model rather than the proposed method itself.

The performance improvement is marginal. 
LG-VQ method outperforms baseline methods in several tasks but performance improvement looks marginal.
questions: questions: In the experiments section, there is only a performance comparison among VQ models such as VQ-VAE and VQ-GAN. A comparison with existing image generation models or visual text reasoning models is needed.
limitations: limitations: The authors discussed both the limitations and potential negative social impacts in the Conclusion section.
paper_summary: paper_summary: This paper considers the limitations in learning an expressive codebook and 
proposes a novel multi-modal codebook learning method. The authors propose two semantic supervision modules and three losses. Experiments demonstrates the effectiveness of their method.
strength_weakness: strength_weakness: 1. It is not easy to obtain major objects and relations between different objects. For example, in Figure 1, ‘the net’, the people beside the court, and the chairs are also very obvious objects. Is it necessary to extract the information of these objects as well?

2. The method shown in Figure 1 uses additional detectors to extract the visual features of the corresponding objects. However, there is no description of these detectors in the article, making it unclear.

3. The authors consider the relationship between text and feature embedding. The effectiveness of the method should be further validated on grounding tasks. (e.g., refcoco)
questions: questions: 1. How can the proposed method be used in downstream tasks, and will it incur additional computational overhead?
limitations: limitations: As stated by the authors, this work assumes each word aligns with a code, which offers a limited representation.
paper_summary: paper_summary: This paper points out that most of existing vector quantization methods learn a single-modal codebook, resulting in suboptimal performance when the codebook is applied to multi-modal downstream tasks. To address this issue, the authors propose a new language-guided codebook learning framework, called LG-VQ, which learns a codebook that can be aligned with the text to improve the performance of multimodal downstream tasks. Specifically, with the pretrained text semantics as prior knowledge, two novel alignment modules are devised to transfer such prior knowledge into codes for achieving codebook text alignment. The proposed method is evaluated on reconstruction and various multi-modal downstream tasks, and achieves superior performance.
strength_weakness: strength_weakness: - The details of the relationship alignment module are not clear. Fig. 3 is not easy to understand. The sec. 3.2.2 should be reorganized to clearly introduce the relationship alignment module.
- There may exist errors in the denominator of Eq. (5).
- In the ablation study, the performance of only adopting the relationship alignment module (+L_{ras}) should also be evaluated. Could this module be used separately?
questions: questions: please try to address the weaknesses.
limitations: limitations: n/a.

Comments: title: Official Comment by Authors
Official Comment by Authors
Official Comment by Authors
Official Comment by Authors
comment: We are thankful for the reviewer’s response and feedback.
Thank you for your response. I have no other concerns with the paper itself, but since I’m not familiar with this domain, I will maintain my current rating and confidence.
Thank you for your response. We will follow your comments and add these discussions to our revised version.
Thanks for the clarification. It's clear now. Please also add this information in the revised paper (the original section of datasets at Ln271 is over concise).
Thank you for your feedback and consideration!
We are thankful for the reviewer’s response and feedback.

> About Q3 image captions: 

In our paper, **we did not generate captions**. 

For CelebA-HQ, CUB-200, MS-COCO datasets, we **used publicly available image captions**, such as: CelebA-HQ from [1], CUB-200 from [2], MS-COCO from [3]. We will add these discussions to our final revised version. 


------

[1] Xia W, Yang Y, Xue J H, et al. Tedigan: Text-guided diverse face image generation and manipulation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 2256-2265.

[2] Reed S, Akata Z, Lee H, et al. Learning deep representations of fine-grained visual descriptions[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 49-58.

[3] Chen X, Fang H, Lin T Y, et al. Microsoft coco captions: Data collection and evaluation server[J]. arXiv preprint arXiv:1504.00325, 2015.
Thank you to the authors for your detailed responses. I believe my concerns have been adequately addressed. I will proceed to revise the final score.
Thank the authors for the detailed information. I think most of my concerns have been well addressed. Overall I'd like to keep my leaning toward acceptance.

I'm having one more question regarding my Q3: I was thinking that you've generated captions for the image-only datasets mentioned in the paper, but in fact you didn't, did you? I'm wondering how the image-only datasets including CelebA-HQ, CUB-200 and MS-COCO were utilized in the experiments.