Unified Lexical Representation for Interpretable Visual-Language Alignment

Authors: yifan  li
yikai  wang
yanwei  fu
dongyu  ru
zheng  zhang
tong  he
  fu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 17:06:25

Abstract: Visual-Language Alignment (VLA) has gained a lot of attention since CLIP's groundbreaking work. 
Although CLIP performs well, the typical direct latent feature alignment lacks clarity in its representation and similarity scores. 
On the other hand, lexical representation, a vector whose element represents the similarity between the sample and a word from the vocabulary, is a natural sparse representation and interpretable, providing exact matches for individual words.
However, lexical representations are difficult to learn due to no ground-truth supervision and false-discovery issues, and thus requires complex design to train effectively.
In this paper, we introduce LexVLA, a more interpretable VLA framework by learning a unified lexical representation for both modalities without complex design. 
We use DINOv2 as our visual model for its local-inclined features and Llama 2, a generative language model, to leverage its in-context lexical prediction ability.
To avoid the false discovery, we propose an overuse penalty to refrain the lexical representation from falsely frequently activating meaningless words.
We demonstrate that these two pre-trained uni-modal models can be well-aligned by fine-tuning on the modest multi-modal dataset and avoid intricate training configurations. 
On cross-modal retrieval benchmarks, LexVLA, trained on the CC-12M multi-modal dataset, outperforms baselines fine-tuned on larger datasets (e.g., YFCC15M) and those trained from scratch on even bigger datasets (e.g., 1.1B data, including CC-12M).
We conduct extensive experiments to analyze LexVLA. 
Codes are available at https://github.com/Clementine24/LexVLA.

Main paper: 
 introduction: 
 The development of Vision-Language Alignment (VLA) models has made great progress [43, 48, 22, 26, 6] since the innovative CLIP [34] effectively learns a shared latent space where text and image are well-aligned. This progress has also boosted related fields like vision-language foundation models [20] , multi-modal understanding [18] , and text-conditional generation [35] . However, CLIP's latent features pose challenges of interpretability issue for analyzing individual factors' impact. Additionally, CLIP's visual model struggles to learn patch-level features, and its text model are trained based on incomplete and biased captions. These challenges reduces its overall effectiveness. On the other hand, the lexical representation is known for its clarity as each dimension corresponds to the similarity between the input and a specific word/token from the vocabulary. Additionally, lexical representation can naturally be made sparse, which means it's efficient for tasks like search or indexing in large-scale retrieval tasks. However, learning lexical representation is difficult. The embedding vector is much larger than the CLIP latent vector as the vocabulary size is usually much larger than CLIP's feature dimension. This poses several challenges: the lack of precise supervision signals; the incomplete and biased text supervision, and the high-dimensional embedding space with low-dimensional targets. Thus using lexical representations in CLIP-style VLA models is tough.Previous lexical representation learning approaches tries to utilize strategies including ℓ 1 regularization [46, 3] to direct encourage sparisity or use ReLU activation [27] to remove negative scores [12, 5, 25] or directly regularize the floating-point operations [31] . However, these intricate training configurations usually introduce extra training difficulty and additional regularization such as log saturation [12, 25] and bag-of-words penalization [48] which potentially limits the VLA capacity.In this paper, we propose LexVLA, a simple yet comprehensive framework for learning a unified lexical representation in the CLIP-style contrastive training pipeline, facilitating the VLA. LexVLA enjoys the following merits in addressing the challenges above. 1) We use single-modal pre-trained models for their unique benefits, including DINOv2 [30] as vision model for its local-inclined features, and Llama 2 [40] as text model to learn lexical representations through in-context prediction tasks instead of traditional caption embeddings. 2) We suggest a unified vocabulary with distinctive codebooks for each modality to maintain the strengths of single-modal pre-trained models by refrain them from learning the same feature space. This allows us to create better VLA models with less multi-modal training data and simpler setups. We also introduce an overuse penalty to prevent excessive activation of irrelevant tokens, improving upon the FLOPs loss [31] which only promotes sparsity. 3) We incrementally train LexVLA to stay aligned with pre-trained models. For text, we fine-tune Llama 2 with LoRA adapter [14] and keep its codebook frozen. For vision, we freeze DINOv2 and train a projector with a self-attention layer and two multi-layer perceptions to the vision codebook, initialized using the text codebook. 4) We introduce PatchDis, a metric for patch-level alignment in visual features, evaluating patch-level classification tasks with image patch features and category text tokens learned by VLA models. We propose PatchDis as a patch-level interpretability metric for VLA models not trained on fine-grained tasks like segmentation or detection. Formally, our LexVLA employs a unified lexical vocabulary with distinct codebooks for text and image modalities, utilizing bi-encoders to individually encode inputs into lexical representations. These encoders, initially set with single-modal pre-trained models, undergo incremental fine-tuning via the standard contrastive learning objective, with the integration of an overuse penalty to promote sparsity and mitigate token over-activation. Emprically, we show the effectiveness of LexVLA on two zero-shot cross-modal retrieval benchmarks. Trained on the CC-12M dataset, it outperforms baselines fine-tuned on larger datasets like YF100m-CLIP + CC-12M and those trained from scratch on even larger datasets totaling 1.1B data, including CC-12M. We provide detailed analysis of LexVLA.Our contributions can be listed as follows:• We emphasize utilizing single-modal pre-trained models for vision-language alignment tasks to benefit from their unique properties that cannot learned by contrastive objectives. To exemplify, we use DINOv2 for its local-inclined features and Llama 2 for its in-context capacity. For the text, we use LoRA to fine-tune the Llama 2 on in-context lexical prediction task, followed with a mapper to get the text lexical representation. We initialize codebooks as Llama 2's codebook, freeze the text codebook while fine-tuning the visual codebook. We train LexVLA with the standard contrastive objectives along with the proposed overuse penalty to encourage sparsity while preventing meaningless activation.• We effectively learn a unified lexical representation with unique codebooks for vision and language modalities to refrain from the weakening of pre-trained capabilities. • We propose an overuse penalty to encourage sparse embedding and prevent meaningless activation. • We enjoy superior retrieval performance with less multi-modal training data, and achieve better patch-level interpretable VLA model with global supervision signals, quantitatively surpassing CLIP-style and lexical methods with our proposed patch-level interpretability metric PatchDis. 
related works: 
 Vision-Language Alignment VLA has been a challenging problem in deep learning. The innovation of CLIP [34] has inspired a plethora of research efforts in combining visual and language modalities [21, 44, 26, 37, 6, 43] . With the rise of large language models (LLMs) [1, 40, 38] , it's become common to use CLIP to get visual data from images and combine it with pretrained LLMs [24, 8, 20, 2, 49] . However, recent studies have identified visual encoders as potential bottlenecks in these models, posing issues like identifying actions and spatial relationships [41, 15] , ignoring attributes and states [45, 10] , introducing hallucination problems [39] , etc. Nevertheless, progress in using other visual models instead of CLIP has been slow. Tong et al. [39] discovered a drop in instruction-following ability when integrating DINOv2 [30] into LLaVA's [24] structure.To address this, they combined features from CLIP and DINOv2. Similarly, Zhai et al. [47] used a locked DINO for contrastive tuning, which necessitated significant aligned data and computational resources. In contrast, we use a locked DINOv2 for lexical VLA, demonstrating the superiority over CLIP with smaller amount of multi-modal training data. It is important to note that our work differs significantly from recent zero-shot cross-modal retrieval models, such as BEiT-3 [42] . Our approach emphasizes developing specific lexical representations for images and text, whereas general retrieval models lack this interpretable representation.Lexical representation is popular in information retrieval [12, 13] thanks to its interpretability and efficiency. However, adapting this approach to vision-language models is challenging because of the differences between the two modalities. Previous approaches rely on complex training stages (two [25] or three [5] ), large scale multi-modal training data [5] to train effectively and additional Bag-of-Words (Bow) restrictions [25, 48] to prevent semantic misalignment. In contrast, we in this paper propose a single fine-tuning stage to align uni-modal pre-trained models in the output lexical space with less multi-modal training data and without BoW restriction. In this paper, our model shall be equipped with the generalizable ability for zero-shot cross-modal evaluation, rather than fine-tuning on each specific target cross-modal training dataset.Additionally, it is important to note that lexical representation differs fundamentally from codebook strategies, as highlighted in [11] . While our aim is to develop interpretable lexical representations for images and visual patches at the vocabulary level, offering greater flexibility, codebook strategies focus on creating a unified yet non-explicit semantic code for images and text at the codebook level, which lacks the desirable traits of interpretability and high retrieval efficiency.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Yes, we discussed the limitations in Sec. 4.3. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
lexvla: 
 Problem setup Our target is to learn the lexical alignment between text and image modalities. Typically, the lexical representation s i ∈ R V , i ∈ {img, txt} is a score vector, whose element indicates the similarity between the sample and the corresponding word from the vocabulary V with V = |V|. The contrastive lexical alignment aims to train the lexical encoders for each modality such that the similarity ⟨s img , s txt ⟩ for positive image-text pairs are maximized and that for negative pairs are minimized, while preserving the correct lexical correspondence with the vocabulary.Overview Our model, as in Fig. 2 , learns a single lexical representation for VLA (Sec. 3.1). We encode each modality separately using uni-modal encoders (Sec. 3.2) adapted in Sec. 3.3.Vocabulary Inspired by tokenization techniques [36, 19] , we initialize with the tokenizer vocabulary and refine it by removing unused and meaningless tokens, reducing the size from 32,000 to 17,149.Codebooks The one-hot embedding is semantic-less and assume the same distance between different words, which is obviously not a good embedding. In this paper, we use the output codebook of language models as a well pre-trained and semantic-rich codebook, inducing lexical codes of 4096-dim vectors. However, the text codebook is not optimized for visual features. Basically, during the pre-training of language model, the visual world is not observable, thus we may have a gap between text-only embeddings and the visual-embeddings. To this end, we propose to use the same vocabulary V but a unique codebook Z i , i ∈ {img, txt} for text and image modalities, respectively. As the language codebook is pre-trained on large corpus, we freeze Z txt to enjoy the well-trained text embeddings. Z img is initialized with the weights of Z txt and fine-tuned along with the training of lexical encoders to enable the adaptation for visual features.Sparse representation Thanks to the inherent similarity measurement in lexical representations, it is natural to turn a dense output vector into a sparse lexical representation. Typically we could utilize the following strategies: 1) Top-k thresholding: select the most important k items and discard the rest; 2) Value thresholding: keep items with value above the threshold and discard the rest. In this paper, we use the value thresholding strategy. We select items with values greater than 1/ √ V , which means we consider only the items with above-average signals as informative.Given the image as x img and text as x txt , the lexical representation is achieved by:EQUATIONWithout loss of generality, we ignore the subscript i in the following definitions unless otherwise specified. We split the lexical encoder in three stages: 1) The single-modal pre-trained feature extractor f , takes as input x and as output a feature sequence y := f (x) ∈ R n×di , where n is the sequence length and d i is the modal-dependent feature dimension; 2) The projector g, projects y to the lexical feature sequence z := g(y) ∈ R n×d , where d is the lexical feature dimension; 3) The mapper h, maps z to the lexical representation s = h(z) ∈ R V .In this paper, we restrict the lexical representation to be non-negative and ℓ 2 -normalized, i.e., ∥s∥ 2 = 1, s i ≥ 0. Different modalities share the same V with a unique codebook Z i ∈ R V ×d , i ∈ {img, txt}.Captioning? In VLA training data pair, the text is typically a caption of the corresponding image. However, it is well-known that caption only captures a partial observation of the semantics in the image, and cannot serve as a perfect target to represent the image. As it is the only accessible ground-truth signal, previous approaches mainly train the text encoders to represent the caption embedding via the output hidden state [9, 48, 12, 25] or [CLS] token embeddings [34] , and use it as the target for the corresponding image. Hence, the learned alignment is biased and leads to false-elimination of image patterns that are not observed in the captions.Predicting! Inspired by the powerful Large Language Models (LLMs), we would like to investigate can we unleash the inherent knowledge of LLMs for lexical representation? The answer is yes.As the auto-regressive LLMs [1, 40] are not naturally to summarize the previous tokens, it is irrational to directly use the predicted token as the embedding for raw text inputs. A straightforward way to activate LLM is to use prompt like "summarize the sentence of [TEXT] in one word:" to activate LLM's summarization capacity and use the output token as the text embedding. But it is still the caption-style summarization, and cannot fully utilize the LLM.LLM as lexical predictor Given the in-context learning capacity of LLM, we propose to adapt LLM as a lexical predictor. Specifically, our input for the LLM is:The focus of "The man is riding a white horse." lies on important words:"man", "riding", "white", "horse". The focus of "[TEXT]" lies on important words:The [TEXT] is the input caption. Then we adopt the output token as the text embedding. This prompt includes two parts: 1) One in-context example to guide the LLM to identify the important words in the caption, adapting the LLM to lexical prediction task; 2) The question prompt to ask the LLM the important words of input caption. In this design, the output token of the LLM naturally performs the lexical prediction task to calculate the similarity between the input caption and every word in the vocabulary, and thus serves as a powerful proxy to get the lexical representation.Realization we utilize Llama 2 [40] as our text encoder f txt . The input text follows the above in-context prompts. We use the output of the predicted token embedding as y txt . As our lexical codebook is Llama 2's output codebook, we do not need g txt , thus z txt = y txt . To implement h txt , we first calculate dot-product attention between each text token z txt ∈ R 1×d and the text lexical codebook Z txt . Then we use the elu1p activation [48] to transform the attention score into non-negative values and then normalize it, yielding global lexical representation s txt . Formally,EQUATIONwhere elu1p(x) = (x + 1) • 1 {x≥0} + e x • 1 {x<0} and Normalize is the ℓ 2 normalization operator. For word-level lexical representation, we directly use the word code in Z txt followed by Eq. (2).We adopt DINOv2 [30] as our visual backbone to implement f img . Given input x img , we first flatten it into patches x img = [x 1 , . . . , x n-1 ] and input to f img to getEQUATION)where CLS is the "class" token in DINOv2. Subsequently, we implement g img with an adapter includes a self-attention layer and 2 multi-layer perceptions to post-process the DINOv2 features.To implement h img , similar to the text encoder but in a more fine-grained manner, we first calculate dot-product attention between each image patch token z img,i ∈ R 1×d and the image lexical codebook Z img , followed by the elu1p activation. In the end, we use max-pooling to aggregate patch representations and then normalize, yielding global lexical representation s img . Formally,EQUATIONThe image patch lexical representation follows the same process but replace the max-pooling operator with selecting the corresponding patch location.Contrastive objective We use the InfoNCE loss [29] with learnable temperature τ for cross-modal retrieval over a batch of N text-image pairs (x img , x txt ) as the major objective:EQUATIONOveruse penalty In lexical representation learning, the FLOPs loss [31] is widely used to encourage the output sparsity. Given the modality-specified lexical representation matrix S = {s i,j }, i ∈ {1, . . . , N }, j ∈ {1, . . . , V } in current mini-batch, the FLOPs loss is defined as:EQUATION)The FLOPs loss aims to reduce FLOPs to induce sparsity. But we noticed that sometimes this penalty pushes the encoder to take shortcuts that falsely activate rarely used and image-unrelated tokens. This semantic deviation leads to false activation and affects the interpretability of lexical representation.To prevent using meaningless tokens too much, we adapt the FLOPs loss with a weighting method. Essentially, we aim to penalize tokens that are used too often across examples. We measure this via the normalized average activation value across the vocabulary, and propose the overuse penalty as:EQUATIONObjective The full objective contains Eq. ( 5) and Eq. ( 7) for each modality:EQUATIONwhere λ I and λ T are two regularization weights for image and text representations, respectively.Incremental fine-tuning For text part, we adopt LoRA adapters [14] to fine-tune the Llama 2. For vision part, we freeze the DINOv2, and only train the projector and vision codebook.The core target of lexical representation is to achieve an interpretable feature representation for input modality. For the text input, the interpretability is straightforward to check. However, existing literature do not have a quantitative metric to measure the patch-level interpretability of visual lexical representation. In this paper, we propose the PatchDis metric to evaluate the patch-level discrimination tasks, inspired by the patch-level segmentation task but designed for models which are not trained from fine-grained alignment tasks like segmentation or detection.Basically, PatchDis evaluates the classification task in the patch-level features. For any given VLA model, we use its text encoder to input the class names and obtain the text embeddings of all classes. Similarly, we input the testing image to the visual encoder to obtain all patch features. Then we could use the model-defined metric to calculate the similarity between each patch feature and the class embeddings, and predict the class of each patch from the largest similarity. Then for each class, we could aggregate all patches of one class as the patch-level prediction for the class. We use the ground-truth segmentation of the class as the target, and use mIoU as the metric to evaluate the patch-level interpretability of the VLA models in the zero-shot patch-level discrimination task.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: (1) We used publicly available benchmark datasets and have provided references for each one for open access. (2) According to the rules of the authors' organization, the code will be available only after the official publication.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 Datasets We use CC-12M [4] for training, a dataset consisting of 12.4 million image-text pairs. We successfully download 9.2M pairs and use this subset as our training set. For evaluation, we use Flickr30k [33] and MSCOCO [23] to evaluate zero-shot cross-modal retrieval tasks.Table 1 : Zero-shot cross-modal retrieval. indicates variants of our LexVLA. CLIP 1 is the original CLIP [34] ; results denoted by (•) 2 are reported in VDR [48] ; results denoted by (•) 3 are reported in STAIR [5] . "Data" is the multi-modal alignment training data size; "Latent" means direct latent feature alignment methods; "Lexical" indicates lexical feature alignment methods. R@K, the recall ratio within top-K items.Setting Model Data MSCOCO Flickr30k image-to-text text-to-image image-to-text text-to-image R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 Latent CLIP 2 15M 20.8 43.9 55. 7 Implementation We use DINOv2 [30] base model and Llama 2 [40] 7B model as our backbones.We use Adam optimizer [17] with learning rate 5e -4 and cosine decayed, batch size of 6,144, precision of BFloat16 for 12 epochs. We initialize τ as 0.07, and clip the logits larger than 100 as in [34] . We use 8 A100 GPUs of 40GB memory to train LexVLA. We quadratically warmup λ in the fisrt 2k steps and then freeze as [31] . We set λ I as 5e -4 and λ T as 1e -3. The trainable parameters in LexVLA is 109 M in total, including 70M for vision codebook, 17M for vision projector (19.76% compared with DINOv2), 21M for Llama adaptor (0.30% compared with Llama 2).Evaluation We conduct experiments on zero-shot cross-modal retrieval tasks on Flickr30k and MSCOCO based on the splits in [16] following previous approaches. We use the R@K, recall ratio within top-K items, as the evaluation metric.Competitors We compare LexVLA with the latent feature alignment methods, including the original CLIP [34] model trained on different dataset scales and followup CLIP-style alignment models, including FILIP [43] , CLIP-BERT [48] , DeCLIP [22] , SLIP [26] , and ProtoCLIP [6] . We also compare with previous lexical representation learning models that focus on zero-shot cross-modal retrieval tasks, including VDR [48] and STAIR [5] . Note that all the competitors are trained by larger datasets. Specifically, the models trained on 15M data uses YFCC15M [21] , and models trained on 1.1B data [5] contains several datasets including CC-12M.Variants For LexVLA, we report the following variants in addition to our final model: 1) (BoW), which selects only the nouns, adjectives, and non-auxiliary verbs in a caption (Bag-of-Words) instead of the LLM-based lexical predictor. 2) (CLIP), which uses the original CLIP visual model, instead of the proposed DINOv2 based encoder; 3) (FLOPs), which uses FLOPs loss as sparsity penalty.3) (d), which uses the top-k thresholding strategy. We test d = 512, the same activated dimension as CLIP to make a fair comparison. The average activated dimension of the final LexVLA is 1081. Note that LexVLA surpasses CLIP with less activated values, see Fig. 5 for detailed analysis. We report the results in Table 1 . 1) LexVLA performs significantly better than both latent feature alignment methods and previous lexical alignment methods when trained on a similar amount of multi-modal examples (12M vs. 15M). This shows the effectiveness of LexVLA. 2) Compared to CLIP trained on much larger multi-modal datasets (0.4B and 1.1B), LexVLA still performs better, even when using the same activated feature dimension as CLIP. This further demonstrates LexVLA's efficiency in text-image alignment with much less training data. 3) LexVLA outperforms STAIR in most metrics and is comparable in two others, despite using significantly less training data (12M vs. 1.1B). This highlights LexVLA's training efficiency. 4) The results of (BoW) are poor, due to the BoW method limits the model's ability to use information effectively. It struggles with vocabulary [7] and semantic [32] mismatches. LexVLA aims to address these limitations by leveraging language models for lexical prediction rather than relying solely on captioning. 5)The significant improvements of LexVLA over (CLIP) indicate that using DINOv2 as the visual backbone is superior. 6) LexVLA with FLOPs loss performs worse than our proposed overuse penalty, and activates meaningless tokens as in Fig. 6 . In summary, these results demonstrate the effectiveness and efficiency of our method. In this part, we evaluate if LexVLA learns the accurate lexical representation. We conduct the following experiments: 1) Quantitatively, we use the proposed PatchDis metric to evaluate the fine-grained patch-level visual lexical representations on MSCOCO 2017 [23] ;2) Qualitatively, we visualize the global lexical representation of images, local lexical representation of image patches, and a PatchDis visualization for the activated patches for the given category.We report the zero-shot PatchDis mIoU performance on MSCOCO 2017 validation dataset. We compare LexVLA with public available baselines CLIP (0.4B) and VDR (15M), as well as random discrimination, and report the performance in Tab. 2. LexVLA clearly shows the superiority on the patch-level interpretability with the weak global training signal. In contrast, the latent alignment CLIP, due to no supervision on patch-level features, performs slightly better than random guessing. VDR enjoys better interpretability, but significantly worse than our LexVLA. LexVLA variant, using the original CLIP visual model, provides a better and more interpretable Figure 5 : Retrieval in different sparse levels. We compare LexVLA with VDR in the same sparse levels and with CLIP as a proxy of dense latent alignment. The first row is results in the Flickr30K dataset, and the second in the MSCOCO dataset. The first to the third columns show the settings of Recall@1, Recall@5, and Recall@10, respectively. Purple symbols represent CLIP.lexical representation of visual patches compared to both the original CLIP and previous lexical methods. This further confirms the effectiveness of our proposed approach. We also visualize the predicted patches of different classes in Fig. 3 . LexVLA successfully activates the correlated patches in the image for different classes, even for the small-scale objects.We visualize the lexical representations of the global image and local patches in Fig. 4 . For the bottom row, we randomly select one object in the image and then annotate the patches that cover the object to calculate the lexical representation on patches in the free form. We use word cloud [28] to illustrate the lexical representation where larger word indicates larger lexical value. These results clearly suggest that the learned lexical representation correctly correlates with the semantics of the image/patch, showcasing the effectiveness of LexVLA.LexVLA under different sparsity One of the targets for lexical representation is to learn a sparse representation for potential benefits in large scale retrieval tasks. To test the robustness of learned representations in LexVLA, we test the retrieval performance of LexVLA in different sparsity levels.Particularly, the sparsity ratio is calculated via ℓ 0 (s i )/V , the ratio of non-zero elements in the lexical representation s i . We compare LexVLA with VDR on the same sparsity ratios. We also visualize the CLIP's performance as a proxy of dense latent representation.Results are shown in Fig. 5 . 1) LexVLA is robust against the sparsity ratio even in a very high ratio (98.27%, 296 activated tokens, which is marked with black vertical dotted line in Fig. 5 ), and then starts to get damaged as the sparse ratio approaches 1; 2) Compared with VDR, LexVLA achieves better performance in all sparse level, indicating a consistent improvement. 3) LexVLA enjoys superior performance with less activated feature dimension as CLIP (296 vs 512) in almost all cases, showcasing our model's effectiveness.Overuse penalty The widely used FLOPs loss aims to encourage the sparsity of learned lexical representation. However, we empirically find that it will falsely activate meaningless tokens as in Fig. 6 . Specifically, the model tends to activate certain semantically unrelated tokens across different s i (like '@' in the image and 'contemporary' in the text, respectively), implicitly treating these tokens as latent dimensions. Our proposed overuse penalty inhibits these frequently activated words more aggressively, and leads to more semantic-correlated lexical representations in Fig. 6 . This enhanced interpretability further improves the retrieval performance in most cases, as in Tab. 1.Limitations The major limitation of LexVLA is the use of vocabulary from large language model. We design our lexical vocabulary based on the Llama 2's tokenizer, and sometimes it splits a word into several sub-word tokens, making the resulting token not a complete word. Although we filter out many meaningless tokens, it still introduces the gap between our vocabulary and the perfect word-level vocabulary. Given that random initialize a word-level vocabulary and additionally learn a projector from sub-word tokens to word-level tokens works poorly, we regard designing a word-level vocabulary that still benefit from the LLMs as a future work.The advancements made in cross-modal retrieval, as demonstrated in this work, hold significant potential for various fields and applications. One immediate impact lies in enhancing information retrieval systems, where users can efficiently search and retrieve multimodal content such as images and text. This can greatly benefit industries such as e-commerce, digital libraries, and multimedia databases, enabling more intuitive and accurate searches. While the advancements in cross-modal retrieval offer numerous benefits, one concern is related to privacy and security, as the integration of multimodal data may raise issues regarding data protection and user consent. It's essential to mitigate these potential negative impacts through robust privacy measures. 
conclusion: 
 In this paper, we propose LexVLA, a unified lexical vision-language alignment framework.LexVLA effectively learns a unified lexical representation with unique codebooks for vision and language modalities, with the aid of single-modal pre-trained models. We propose to incremental fine-tune LexVLA with the standard contrastive objective penalized by the proposed overuse objective to prevent meaningless activation and encourage sparse embedding. LexVLA refrains from the complex training configurations, and effectively achieves the patch-level interpretability with only global supervision signals. We introduce a patch-level interpretability metric, PatchDis, to quantify this on the zero-shot cross-modal benchmark dataset. We demonstrate the effectiveness of LexVLA on two zero-shot cross-modal retrieval benchmark datasets with significantly smaller multi-modal training dataset. We also provide detailed analysis on LexVLA.A Appendix / supplemental material A.1 Model hyperparametersOur hyperparamters selection for the detail can be found in Tab. 3  
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We provided all the details in Sec. 4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We didn't include error bars because calculating them would take too much computing power. We're also following the approach of previous studies that didn't include them.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We provided all the required information in Sec. 4.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We discussed both potential positive societal impacts and negative societal impacts in Sec. 4.3.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work, towards the accurate lexical alignment between text and images, poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have credited all the assets we used, followed their licensing rules, and will not distribute or repurpose them.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We do not have new assets in the submission.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: We do not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: We do not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The DINOv2 [30] base model used in LexVLA is pretrained with self-supervised learning, leveraging a large dataset of 142 million diverse, high-resolution images. It employs a teacher-student architecture with multi-crop views and strong data augmentations to encourage robustness and capture both global and local features. During pretraining, the model uses a cosine similarity loss between student and teacher outputs to ensure consistent feature representations across different augmentations. This model supports downstream performance on detection and segmentation tasks.The Llama 2 model [40] used in LexVLA is pretrained with self-supervised learning on approximately 2 trillion tokens from a diverse mix of publicly available datasets and web-sourced content. It employs a dense transformer architecture, and during training, it focuses on token prediction accuracy and generalization. This model supports effective performance across various language tasks.For more pretraining details, please refer to the original papers of these two pretrained models.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Yes, they correctly describe the paper's contributions and scope. We propose a unified lexical alignment model that aligns text and visual data based on single-modal pretrained models and introduces unique codebooks, an overuse penalty, and enjoys patch-level interpretability.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: Our research conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: After discussion, this submission received 3 positive scores. After rebuttal, the major concerns about interpretability of the proposed approach, result visualization, experimental evaluation, and comparison of difference frameworks were comprehensively solved. After reading the paper, the review comments and the rebuttal, the AC think the remaining concern is about the lack of critical evaluation of the proposed approach, which is encouraged to added to the camera-ready version.

Reviews: review: paper_summary: paper_summary: This paper presents LexVLA, a vision language alignment method integrating a pretrained vision model and a pretrained language model. To retain the original capabilities of pretrained single-modal models, it adopts a unified lexical representation with unique codebooks. Moreover, the vision model is tuned with a projector, and the text model is tuned with LoRA. A metric for patch-level alignment is proposed to evaluate interpretability. Experiments are conducted on retrieval benchmarks.
strength_weakness: strength_weakness: - Even though a new metric is proposed, the effectiveness of its reflection on interpretability is not verified quantitatively or qualitatively.
questions: questions: - How accurately or reliably does the proposed PatchDis metric evaluate/reflect the interpretability of patch-level visual lexical representations?
limitations: limitations: While multiple technical contributions have been made in this paper, some of the components lack rigorous verification.
paper_summary: paper_summary: The paper proposes LexVLA, a more interpretable VLA framework that learns a unified lexical representation for both modalities without complex design.
LexVLA uses DINOv2 as the visual model and Llama 2 as the language model, proposing an overuse penalty to avoid false discoveries.
LexVLA outperforms baselines on cross-modal retrieval benchmarks, even when fine-tuned on a modest dataset.
Extensive experiments were conducted to analyze LexVLA's performance.
strength_weakness: strength_weakness: 1. I can't quite get the novelty of this work. The lexical representation mentioned in the paper is somehow a way to select important information and then map it to the code book. However, the codebook strategy was explored [1]. Especially the visual part, where does the concept of Lexical come in? Can the author elaborate more on this?
2. In Table 1, the improvement is pretty limited in the bottom block compared to using CLIP in the last and first blocks. It makes readers question whether the performance was gained by the DINOv2 representation.
3. The alignment was tested on only one task, it will be more interesting to test on other multimodal tasks such as zeroshot classification, or even grounding since it has DINOv2 representation.


[1] Duan, Jiali, et al. "Multi-modal alignment using representation codebook." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
questions: questions: Please address the questions in the weakness.
limitations: limitations: None
paper_summary: paper_summary: The authors propose a method based on lexical representation for Visual-Language Alignment (VLA). The method relies on aligning two strong unimodal models, namely DINOv2 for the visual modality and Llama 2 for the text modality. Each backbone is fine-tuned with a few adapters or additional layers. The two modalities use separate codebooks mapping to a joint vocabulary. The authors also propose an overuse penalty to limit the excessive activation of irrelevant tokens. Finally, the authors introduce the PatchDis metric to measure patch-level alignment. Evaluation on zero-shot cross-modal retrieval datasets shows state-of-the-art performance of the method with the compared baselines. Additional experiments on the patch-level representation and sparsity showing the effectiveness of the method are also reported.
strength_weakness: strength_weakness: - The vocabulary is based on the Llama tokenizer which, as stated in the limitations, may split words into meaningless sub-word tokens and may also lack longer relevant words.
- The latent baselines for zero-shot cross-modal retrieval do not include recent methods such as BEiT-3 [Wang, Wenhui, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal et al. "Image as a foreign language: Beit pretraining for vision and vision-language tasks." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19175-19186. 2023.]
- One main difference with the compared methods could be the use of the DINOv2 visual backbone and the Llama 2 textual backbone, it is possible the proposed method benefits from these strong backbones. All methods' visual and text backbones (and their potential pretraining) should be discussed in detail to enable the readers to properly judge the merit of the proposed method
questions: questions: - Have the authors explored a simpler approach of just selecting the nouns, adjectives, and non-auxiliary verbs in a caption instead of the LLM-based lexical predictor? How many keywords are extracted by the LLM on average per caption? Does it vary with the length of the caption?
- Eq (3), what is x with any index?
- Table 1: it would be good to also indicate the amount of (unimodal) pretraining data (if any) used for each method e.g. the amount of data used for DINOv2 and LLama 2 for the proposed method. What are the test splits used for this experiment? Commonly, results are reported based on the splits in [Karpathy, Andrej, and Li Fei-Fei. "Deep visual-semantic alignments for generating image descriptions." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3128-3137. 2015.]. If these are not used it would be good to use them as well.
- Figure 3: it would be good to provide the class <-> color mapping here.
- Figure 4: bottom row, how were the local patches selected?
- Figure 5: what does the vertical black dotted line represent? How were the different sparsity level selected?
- The authors mention in the limitations that their “lexical vocabulary based on the Llama 2’s tokenizer (...) splits a word into several sub-word tokens.” does that also mean that some rather rare long words would not appear in the vocabulary? Have the authors studied what are these missing words? Further down the authors state “Given that random initialize a word-level vocabulary and additionally learn a projector from sub-word tokens to word-level tokens works poorly, we regard designing a word-level vocabulary that still benefit from the LLMs as a future work.”, it seems the author did conduct some experiments towards that. Even if the results were not conclusive it would be interesting to share what was tried and what was the performance.

Typos etc:
- p2-l52: missing space “​​LexVLAto”
limitations: limitations: Yes

Comments: comment: Dear Reviewer a8Qd,

We would like to know if our previous response addressed your concern. We have reported the result of the alternative evaluation in another comment section, which shows LexVLA (CLIP) provides a better and more interpretable lexical representation of visual patches compared to the original CLIP. If there are any additional comments or suggestions you would like to share, we would be more than happy to discuss them further.

Thank you again for your valuable feedback.


Best regards,

The Authors
Thanks for the opportunity to address your further concerns in response to our rebuttal.

## Regarding the alternative evaluation:
Thank you for the suggestion. We report the PatchDis evaluation result of LexVLA (CLIP)  as follows:
| **Model**      | **mIoU**  |
|----------------|-----------|
| Random Dis.    | 5.0       |
| CLIP           | 5.3       |
| VDR            | 12.6      |
| *LexVLA (CLIP)*| *13.9*      |
| LexVLA         | **36.3**  |

These results clearly show that our LexVLA (CLIP) provides a better and more interpretable lexical representation of visual patches compared to the original CLIP and previous lexical method. This further confirms the effectiveness of our proposed approach.

## Regarding the results in Table 1
Thank you for your careful review. We would like to supplement that the results in Table 1 demonstrate the effectiveness of LexVLA, even with significantly less multi-modal training data (12M vs. 1.1B). 
Particularly, while CLIP (1.1B) outperforms LexVLA (CLIP) in MSCOCO image-to-text results, it underperforms in MSCOCO text-to-image and Flickr30k image-to-text and text-to-image experiments. Moreover, our full LexVLA model outperforms CLIP (1.1B) in all settings.
I have read all the reviews and the responses from the authors, they have addressed some of my concerns and I believe integrating most of the discussion would improve the paper enough to update my rating to `5: Borderline accept`.
The authors have addressed my concerns. I will keep my previous rating.
Thank you for your rebuttal! 

Upon examining the comparison between the original CLIP model and its modified version, I observed that there is no definitive winner; the original CLIP showed superior performance on MSCOCO, while the new Lexical version excelled on Flickr30k. 

This outcome is somewhat anticipated, as the aim is to find an exact match between the tokens. However, this might not hold true in retrieval tasks due to the presence of a lot of redundant information in the background. I believe that an alternative evaluation setting is needed to showcase the full potential of this model.
Dear reviewers:

There are diverging reviews for this paper. Supporting reviewers, please take a look at the other reviews. For the opposing reviewer, CyPa, please carefully read the rebuttal and share any remaining concerns, if any.

Thanks