Unified Lexical Representation for Interpretable Visual-Language Alignment
Authors: Yifan Li
Yikai Wang
Yanwei Fu
Dongyu Ru
Zheng Zhang
Tong He
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-modal
Alignment
Retrieval
Sparse Retrieval
Lexical Representation
Primary area: machine_vision
TL;DR: We introduce LexVLA, a interpretable Visual-Language Alignment framework by learning a unified lexical representation for both modalities without complex design.

Abstract: Visual-Language Alignment (VLA) has gained a lot of attention since CLIP's groundbreaking work. 
Although CLIP performs well, the typical direct latent feature alignment lacks clarity in its representation and similarity scores. 
On the other hand, lexical representation, a vector whose element represents the similarity between the sample and a word from the vocabulary, is a natural sparse representation and interpretable, providing exact matches for individual words.
However, lexical representations are difficult to learn due to no ground-truth supervision and false-discovery issues, and thus requires complex design to train effectively.
In this paper, we introduce LexVLA, a more interpretable VLA framework by learning a unified lexical representation for both modalities without complex design. 
We use DINOv2 as our visual model for its local-inclined features and Llama 2, a generative language model, to leverage its in-context lexical prediction ability.
To avoid the false discovery, we propose an overuse penalty to refrain the lexical representation from falsely frequently activating meaningless words.
We demonstrate that these two pre-trained uni-modal models can be well-aligned by fine-tuning on the modest multi-modal dataset and avoid intricate training configurations. 
On cross-modal retrieval benchmarks, LexVLA, trained on the CC-12M multi-modal dataset, outperforms baselines fine-tuned on larger datasets (e.g., YFCC15M) and those trained from scratch on even bigger datasets (e.g., 1.1B data, including CC-12M).
We conduct extensive experiments to analyze LexVLA. 
Codes are available at https://github.com/Clementine24/LexVLA.

Unified Lexical Representation for Interpretable Visual-Language Alignment
Yifan Life Yikai Wang't Yanwei Fu
Dongyu Ru²
Zheng Zhang² Tong Helt
Fudan University Amazon Web Services yifanli23@mn fudan. edu. en, cn, yi-kai. wang@outlook. com,
yanweifudfudan edu en, {rudongyu, zhaz, htong] (amazz)mmmzooccoo com
Abstract
Visual- Lasual-Language Alignment (VLA) has gained lot of attention since CLIP's groundbreaking work. Although CLIP performs well, the typical direct latent feature alignment lacks clarity its representation and similarity scores. On the other hand, lexical representation, vector whose element represents the similarity between the sample and a word from the vocabulary, is a natural sparse represen- tation and interpretable, providing exact matches for individual words. However, lexical representations are difficult to learn due to no ground-truth supervision and false-discovery issues, and thus requires complex design to train effectively. In this paper, we introduce Lex VLA, a more interpretable VLA framework by learning unified lexical representation for both modalities without complex design. We use DINOv2 as our visual model for its local inclinedd features and Llama 2, a generative language model, leverage its in-context lexical prediction ability. To avoid the false discovery, we propose an overuse penalty to refrain the lexical rep- resentation from falsely frequently activating meaningless words. We demonstrate that these two pre-trained uni-modal models can be well-aligned by fine-tuning the modest multi-modal dataset and avoid intricate training configurations. On cross-modal retrieval benchmarks, Lex VLA, trained on the CC-12M multi-modal dataset, outperforms baselines fine-tuned on larger datasets (e.g., YFCC15M) and those trained from scratch on even bigger datasets (e.g.. 1.1B data, including CC- 12M). We conduct extensive experiments analyze LexVLA. Codes are available at httpa///wwooo.oommooooooo//////.
Introduction
The development of Vision-I Lnnuuage Alignment (VLA) models has made great progress +5
48
22 26
since the innovative CLIP (34] effectively learns a shared latent space where text and image are well-aligned. This progress has also boosted related fields like vision-language foundation models 200] multi-modal understanding [18], and text-conditional generation (35). However, CLIP's latent features pose challenges of interpretability issue for analyzing individual factors' impact. Additionally, CLIP's visual model struggles learn patch-level features, and its text model are trained based on incomplete and biased captions. These challenges reduces its overall effectiveness.
"Work completed during internship at AWS Shanghai AI Lab. 1 "Corresponding authors.
!Dr. Fu is with School of Data Science in Fudan, Fudan ISTBI-ZJNU Algorithm Center for Brain-inspired Intelligence, Shanghai Key Lab of Intelligent Information Processing, and Technology Innovation Center of Calligraphy and Painting Digital Generation. Ministry Culture and Tourism. China.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

sword io
maing
soldier sea voj sword
tor tribe medieval
Fodieval na ar couple
history
Lex can generate a lexical representation of the input image (the first word cloud figure), or to pick some patches of the image for local lexical information representation (the second word cloud figure, with the seleted patches boxed in red), and to select the most relevant patches of the image given the text content (the rightmost figure, with caption 'horse', with the second-to-las figure is the ground-truth mask).
Figure I:
On the other hand, the lexical representation known for its clarity as each dimension corresponds to the similarity between the input and specific word/token from the vocabulary. Additionally, lexical representation can naturally be made sparse, which means it's efficient for tasks like search or indexing in large-scale retrieval tasks. However, learning lexical representation is difficult. The embedding vector is much larger than the CLIP latent vector as the vocabulary size is usually much larger than CLIP's feature dimension. This poses several challenges: the lack of precise supervision signals; the incomplete and biased text supervision, and the high- eigh-dimensional embedding space with low-dimensional targets. Thus using lexical representations in CLIP-style VLA models tough.
Previous lexical representation learning approaches tries to utilize strategies including & regu- direct encourage sparisity or use ReLU activation [27] to remove negative 25 or directly regularize the floating- pointt operations B1. However, these intricate training configurations usually introduce extra training difficulty and additional regularization such as log saturation
larization 46
3
scores 12
5
25 and bag-of-words penalization (48) which potentially limits the VLA capacity,
In this paper, we propose LexVLA, simple yet comprehensive framework for learning unified lexi- cal representation in the CLiP-style contrastive training pipeline, facilitating the VLA. Lex VLA enjoys the following merits in addressing the challenges above. 1) We use single-modal pre-trained models for their unique benefits, including DINOv2 30) as vision model for its local-inclined features, and Llama 2 [40] as text model learn lexical representations through in-context prediction tasks instead of traditional caption embeddings. 2) We suggest a unified vocabulary with distinctive codebooks for each modality to maintain the strengths of single- modal pre-trained models by refrain them from learning the same feature space. This allows us to create better VLA models with less multi-modal training data and simpler setups. We also introduce an overuse penalty to prevent excessive activation of irrelevant tokens, improving upon the FLOOPs loss (31) which only promotes sparsity. 3) We incrementally train LexVLA to stay aligned with pre-trained models. For text, we fine-tune Llama 2 with LoRA adapter [4] and keep its codebook frozen. For vision, we freeze DINOv2 and train a projector with self-attention layer and two multi-layer perceptions to the vision codebook, initialized using the text codebook. 4) We introduce PatchDis, metric for patch-level alignment in visual features, evaluating patch-level classification tasks with image patch features and category text tokens learned by VLA models. We propose PatchDis as patch-level interpretability metric for VLA models not trained on fine-grained tasks like segmentation or detection.
Formally, our VLA employs a unified lexical vocabulary with distinct codebooks for text and image modalities, utilizing bi-encoders individually encode inputs into lexical representations. These encoders, initially set with single-modal pre-trained models, undergo incremental fine-tuning via the standard contrastive learning objective. with the integration of an overuse penalty to promote sparsity and mitigate token over-activation. Emprically, we show the effectiveness of Lex VLA on two zero-shot cross-modal retrieval benchmarks. Trained on the CC-12M dataset, it outperforms baselines fine-tuned on larger datasets like YF100m-CIL + CC-12M and those trained from scratch on even larger datasets totaling 1.1B data, including CC-12M. We provide detailed analysis of Lex VLA.
Our contributions can be listed as follows:
We emphasize utilizing single-modal pre-trained models for vision-language alignment tasks to benefit from their unique properties that cannot learned by contrastive objectives. To exemplify, we use DINOv2 for its local-inclined features and Llama 2 for its in-context capacity.

---

Adapter
DINOS
Mappor
parameters
vose of folower.
Prozen
Objective
Loss function
LLM
Latent text festure
bloh
Mappor
Shared
Lesical representation
Vocabulary
Framework of VLA. We learn a unified lexical representation with distinct codebooks for text and visual modalities. For the image, adopt the frozen DINOv2, learn adapter and mapper to get visual lexical representation. For the text, we use LoRA to fine-tune the Llama on in-context lexical prediction task, followed with mapper get the text lexical representation. We initialize codebooks as Llama 2's codebook, freeze the text codebook while fine-tuning the visual codebook. We train Lex VLA with the standard contrastive objectives along with the proposed overuse penalty to encourage sparsity while preventing meaningless activation.
Figure 2:
We effectively learn a unified lexical representation with unique codebooks for vision and language modalities to refrain from the weakening of pre-trained capabilities.
We propose an overuse penalty to encourage sparse embedding and prevent meaningless activation. We enjoy superior retrieval performance with less multi-modal training data, and achieve better patch-level interpretable VI.A model with global supervision signals, quantitatively surpassing CLIP-style and lexical methods with our proposed patch-level interpretability metric PatchDis.
2
Related works
Vision-Language Alignment
VLA has been challenging problem in deep learning. The inno- 34 has inspired a plethora of research efforts in combining visual and language
vation of CLIP
modalities
21
26
37
6 43
With the rise of large language models (LLMs) []] 40 38].
it's become common to use CLIP to get visual data from images and combine with pretrained However, recent studies have identified visual encoders as potential bot-
LLMs 24
20
49
tlenecks in these models, posing issues like identifying actions and spatial relationships [4] [5],
ignoring attributes and states [5 [0], introducing hallucination problems (39), etc. Nevertheless,
progress in using other visual models instead of CLIP has been slow. Tong et al. [39] discovered drop in cestruction-followiii To address this, they combined features from CLIP and DINOv2. Similarly, Zhai et al. [47 used a locked DINO for contrastive tuning, which necessitated significant aligned data and computational resources. contrast, we use a locked DINOv2 for lexical VLA, demonstrating the superiority over CLIP with smaller amount of multi-moda training data. is important note that work differs significantly from recent zero-shot cross-modal retrieval models, such as BETT-3 Our approach emphasizes developing specific lexical representations for images and text, whereas general retrieval models lack this interpretable representation.
ability when integrating DINOv2 into LLaVA's 24 structure.
Multi-modal lexical representation Lexical representation is popular in information retrieval [12
13 thanks its interpretability and efficiency. However, adapting this approach vision-language models is challenging because of the differences between the two modalities. Previous approaches
rely on complex training stages (two (25] or three [5)), large scale molti-modal training data (5) to train effectively and additional Bag-of- Words (Bow) restrictions [25] 48 to prevent semantic
misalignment. In contrast, we this paper propose single fine-tuning stage to align uni-modal pre-trained models in the output lexical space with less multi-modal training data and without BoW restriction. In this paper, our model shall be equipped with the generalizable ability for zero-shot crosl-modal evaluation, rather than fine-tuning each specific target cross modal training dataset.

---

Additionally, it is important to note that lexical representation differs fundamentally from codebook strategies, as highlighted in [.] While our aim to develop interpretable lexical representations for images and visual patches at the vocabulary level, offering greater flexibility, codebook strategies focus on creating a unified yet non-explicit semantic code for images and text at the codebook level, which lacks the desirable traits of interpretability and high retrieval efficiency.
3 LeXVLA
Problem setup Our target is to learn the lexical alignment between text and image modalities. Typically, the lexical representation 8; RV, E fimg, txt] is score vector, whose element indicates the similarity between the sample and the corresponding word from the vocabulary V with The contrastive lexical alignment aims to train the lexical encoders for each modality such
V= V that the similarity (8img, Stax) for positive image-text pairs are maximized and that for negative pairs minimized, while preserving the correct lexical correspondence with the vocabulary.
Overview Our model, as in Fig- learns single lexical representation for VLA (Sec. 3.1
We
encode each modality separately using uni-modal encoders (Sec. (.2) adapted Sec. 3.3
3.1 Lexical representation
Vocabulary Inspired by tokenization techniques 36 19,, we initialize with the tokenizer vocabulary and refine it by removing unused and meaningless tokens, reducing the size from 32,000 to 17.149.
Codebooks The one-hot embedding is semantic-less and assume the same distance between different words, which is obviously not good embedding. In this paper, we use the output codebook of language models as a well pre-trained and semantic-rich codebook, inducing lexical codes of 4096-dim vectors. However, the text codebook is not optimized for visual features. Basically, during the pre-train mre-training of language model, the visual world is not observable, thus we may have a gap between text-only embeddings and the iisual-ebeeddings. To this end, we propose to use the same vocabulary V but unique codebook Z, E (img, txt] for text and image modalities, respectively. As the language codebook is pre-trained on large corpus, we freeze Ztzt to enjoy the well-trained text embeddings. Zing is initialized with the weights of Zewt and fine-tuned along with the training of lexical encoders to enable the adaptation for visual features.
Sparse representation Thanks the inherent similarity measurement lexical representations, is natural to turn a dense output vector into a sparse lexical representation. Typically we could utilize the following strategies: 1) Top-k thresholding: select the most important & items and discard the rest; 2) Value thresholding: keep items with value above the threshold and discard the rest. In this paper, we use the value thresholding strategy. We select items with values greater than 1/ \/VV, which means we consider only the items with above-average signals as informative.
3.2 Lexical encoder
Given the image as Ding and text as Birtt the lexical representation achieved by:
s: == e (xx) h4 o g, f(())). i e {img. txt).
(1)
Without loss of generality, we ignore the subscript in the following definitions unless otherwise specified. We split the lexical encoder in three stages: 1) The single-modal pre-trained feature extractor f. takes as input æ and as output feature sequence y == f(x) gnxdd where the sequence length and d, is the modal-dependent feature dimension: 2) The projector g, projects y to the lexical feature sequence z g(y) nx where d is the lexical feature dimension; 3) The mapper h, maps z to the lexical representation s = h(z) R.V.
In this paper, restrict the lexical representation to be non negatega and & ny-orralleed, i.c., || 1, 2 0. Different modalities share the same V with unique codebook Z, IR Vxd ; E {img. txt).

---

3.2.1 Lexical text encoder
Captioning? In VLA training data pair, the text is typically a caption of the corresponding image. However, it well-known that caption only captures a partial observation of the semantics in the image, and cannot serve a perfect target to represent the image. As it is the only accessible ground-truth signal, previous approaches mainly train the text encoders to represent the caption embedding via the output hidden state [9] 48 12 25 or [CLS] token embeddings [4., and use
as the target for the corresponding image. Hence, the learned alignment is biased and leads to false-elimination of image patterns that are not observed in the captions.
Predicting! Inspired by the powerful Large Language Models (LLMs), we would like to investigate can we unleash the inherent knowledge of LLMs for lexical representation? The answer is yes.
40) are naturally to summarize the previous tokens, is irrational to directly use the predicted token as the embedding for raw text inputs. A straightforward way to activate LLM is use prompt like "summarize the sentence of [TEXT] in one word:" to activate LLM's summarization capacity and use the output token as the text embedding. But it is still the caption-style summarization, and cannot fully utilize the LLM.
As the auto-regressive LLMs I
LLM as lexical predictor Given the in-context learning capacity of LLM, we propose to adapt LLM lexical predictor. Specifically, our input for the LLM is:
The focus of "The man is riding a white horse." lies on important words:"man mman", "riding". "white", "horse". The focus of " [[TEX]]" " lies on important words:
The [TEXT] is the input caption. Then we adopt the output token as the text embedding. This prompt includes two parts: 1) One in-context example to guide the LLM to identify the important words in the caption, adapting the LLM to lexical prediction task; 2) The question prompt to ask the LLM the important words of input caption. In this design, the output token of the LLM naturally performs the lexical prediction task to calculate the similarity between the input caption and every word in the vocabulary, and thus serves as a powerful proxy to get the lexical representation.
Realization utilize Llama 2 40 as text encoder fizt. The input text follows the above in-context prompts. We use the output of the predicted token embedding as Yezt- As our lexical codebook is Llama 2's output codebook, we do not need gtzt, thus ztæt Vizt- To implement Maz, we first calculate dot-product attention between each text token ztprt E Rixd and the text lexical codebook Zent. Then we use the elulp activation [48] to transform the attention score into non-negative values and then normalize it, yielding global lexical representation Stxt- Formally,
$ tzt Normalize o clulp(zizz Zixe)
where elulp(z) = (a + 1) 1 (a20) + c" [(xc0) and Normalize is the l, normalization operator. For word-level lexical representation, directly use the word code in Zizt followed by Eq. 2
3.2.2 Lexical visual encoder
We adopt DINOv2 [30 as our visual backbone to implement feng. Given input aing. we first flatten it into patches Tsmg [xi...., In-1] and input to fimg to get
Yimg fimg (CCLS; Tamg]) E gaxd
where CLS is the "class" token in DINOv2. Subsequently, we implement gimg with an adapter includes a self-attention layer and 2 multi-layer perceptions to post-process the DINOv2 features. To implement himg. similar to the text encoder but in more fine-grained manner, we first calculate dot-product attention between each image patch token Zimg. E 1xd and the image lexical code- book Zimg, followed by the elulp activation. In the end, we use max-pooling to aggregate patch representations and then normalize, yielding global lexical representation Sing. Formally,
8 smg Normalize Max-Pool elu p( Zing Z Z
The image patch lexical representation follows the same process but replace the max- pooling operator with selecting the corresponding patch location.
5

---

3.3 Train LexVLA
Contrastive objective We use the InfoNCE loss with learnable temperature for cross-modal retrieval over a batch of N text-image pairs (Zimg Tirt) as the major objective:
exp(axb) /T) log Dj=1 exp(a:b] /T)
l2b
(a, b) {Fiiggs Ltatt, (Fezt: Tsmg]].
(5)
Overuse penalty In lexical representation learning, the FLOPs loss 31 is widely used encourage the output sparsity. Given the modality-specified lexical representation matrix S = {8ig), N,...,NN.J. (1,.-.;W V) in current mini-batch, the FLOOS loss is defined as:
(FLOPS
86j
66)
N
The FLOPs loss aims to reduce FLOPs to induce sparsity. But we noticed that sometimes this penalty pushes the encoder to take shortcuts that falsely activate rarely used and image-unrelated tokens. This semantic deviation leads to false activation and affects the interpretability of lexical representation.
To prevent using meaningless tokens too much, we adapt the FLOPs loss with weighting method. Essentially, we aim to penalize tokens that are used too often across examples. We measure this via the normalized average activation value across the vocabulary, and propose the overuse penalty as:
N
3
N
J.J
overuse
V
NV
$tj/N
Si./
/
Sijz
1=1
k=
j=li=l
Objective
The full objective contains Eq. 55) and Eq. 7 for each modality:
Lezi lize + A Dveruee X overuee
(8)
where d and ÀT two regularization weights for image and text representations, respectively.
Incremental fine-tuning For text part, we adopt LoRA adapters [4] to fine-tune the Llama 2. For vision part, freeze the DINOv2, and only train the projector and vision codebook.
3.4 PatchDis: interpretability metric
The core target of lexical representation is to achieve an interpretable feature representation for input modality. For the text input, the interpretability is straightforward to check. However, existing literature do not have a quantitative metric to measure the patch-level interpretability of visual lexical representation. In this paper, we propose the PatchDis metric evaluate the patch-level discrimination tasks, inspired by the patch-level segmentation task but designed for models which are not trained from fine-grained alignment tasks like segmentation detection.
Basically, PatchDis evaluates the classification task in the patch-level features. For any given VLA model, we use its text encoder to input the class names and obtain the text embeddings of all classes. Similarly, we input the testing image to the visual encoder to obtain all patch features. Then we could use the model-defined metric to calculate the similarity between each patch feature and the class embeddings, and predict the class of each patch from the largest similarity. Then for each class, could aggregate all patches of one class as the patch-level prediction for the class, We use the ground- -oruun-truthhhae segmentation of the class as the target, and use mloU as the metric to evaluate the patch-level interpretability of the VIA models in the zero-shot patch-level discrimination task.
4 Experiments
We use CC-12M 4 for training, dataset consisting of 12.4 million image-text pairs. We successfully download 9.2M pairs and use this subset as our training set. For evaluation, we use Flickr30k (33) and MSCOCO 23 to evaluate zero-shot modal retrieval tasks.
Datasets

---

Table Zero-shot cross-modal retrieval. Q indicates variants our LexVLA. CLIP² is the original CLIP 34I; : results denoted by (-)² are reported in VDR [8]; results denoted by (-)² are reported in STAIR [5]. "Data" is the multi-modal alignment training data size; "Latent" means direct latent feature alignment methods; "Lexical" indicates lexical feature alignment methods. R@ K, the recall ratio within top-K items.
MSCOCO text-to-iman to-image
Flickr30k
Setting Model
Data
image-lo-text
image-to-text
text-to-image
R@l
R@5 R 10
R@I R@5 Ræ 10 R@l
R@s R@ 10 R@ R@s R@ 10
CLIP
15M
20.8
43.9
55.7
13.0
31.7
42.7
34,9
63.9
75.9
23.4
47.2
58.9
FILIP²
15M
21.6
46.7
59.0
13.7
31.7
41.6
46.3
74.4
83.2
30.7
58.2
68.6
CLIP-BERT
15M
23.9
47.8
60.3
13.6
33.8
45.1
44.1
71.2
80.7
27.8
54.7
65.9
Latent
DeCLIP
15M
25.3
51.2
63.4
16.6
35.2
45.4
51.3
80.7
88.5
35.5
63.0
73.0
SLIP
15M
27.7
52.6
63.9
18.2
39.2
51.0
47.8
76.5
85.9
32.3
58.7
68.8
ProtoCLIp
15M
30.2
55.1
66.5
16.9
37.9
49.4
CLIP
0.4B
52.4
76.7
84.6
33.1
58.4
69.0
81.8
96.2
98.8
62.1
85.6
91.8
CLIP3
1.IB
53.4
78.3
85.6
36.2
62.2
72.2
79.6
95.5
98.1
63.0
86.7
92.5
VDR²
15M
30.9
54.5
65.4
17.4
38.1
49.7
51.0
79.3
86.7
32.4
60.1
70.7
Lexical
STAIR
I.IB
57.7
80.5
87.3
41,4
65.4
75.0
81.2
96.1
98.4
66.6
88.7
93.5
Q(BoW)
12M
17.9
34.9
45.2
10.4
24.3
33.1
30.6
56.2
66.3
17.7
36.4
44.9
(CLIP)
12M
51.8
75.5
84.1
36.8
62.5
72.7
82.9
96.2
98.7
65.2
88.3
93.2
Lexical
(FL.Ps)
12M
56.2
80.0
87.4
39.0
65.7
75.6
84.2
96.6
98.7
67.4
89.4
94.1
(512)
12M
56.4
79.9
87.5
38.1
64.6
74.9
84.5
97.3
99.0
65.7
89.3
93.8
LexVLA
12M
55.4
80.6
88.3
39.8
66.3
76.2
83.9
67.8
90.2
94.2
Implementation We use DINOv2 30 base model and Ilama 2 [40] 7B model as our backbones. We use Adam optimizer 117 with learning rate 5e 4 and cosine decayed, batch size of 6.144, precision of 1 for 12 epochs. We initialize r as 0.07, and clip the logits larger than 100 as 34 We use A100 GPUs of 40GB memory train LcxVLA. We quadratically warmup in the fisrt 2k steps and then freeze as [31] We set \ as 5e - 4 and \T as le - 3. The trainable parameters LexVLA is 109 M in total, including 70M for vision codebook, 17M for vision projector (19.766 76% compared with DINOv2), 21M for Llama adaptor (0.30% compared with Llama 2).
4.1 Zero-shot cross-modal retrieval
Evaluation
We conduct experiments on zero-shot cross- moosll retrieval tasks on Flickr30k and MSCOCO based on the splits in [16] following previous approaches. We use the R@K, recall ratio within top-K items, the evaluation metric.
Competitors We compare VLx with the latent feature alignment methods, including the original CLIP (34) model trained on different dataset scales and followup CLIP-style alignment models, including FILIP 43). CLIP-BERY [48), DeCLIP (22]. SLIP (26). and ProtoCLIP [6]. We also compare with previous lexical representation learning models that focus on zero-shot cross-modal retrieval tasks, including VDR
48 and STAIR (5). Note that all the competitors are trained by larger datasets. Specifically, the models trained on 15M data uses YFCC15M [21), and models trained on 1.1B data (5) contains several datasets including CC-12M.
Variants For LexVLA, we report the following variants in addition to our final model: 1) Q(BoW), which selects only the nouns, adjectives, and non-auxiliary verbs in a caption (Bag-of- -f-ooss-Words) instead of the LLM-based lexical predictor. 2) Q(CLIP) which uses the original CLIP visual model, instead of the proposed DINOv2 based encoder; 3) Q(FLOPs) which uses FLOPs loss as sparsity penalty. 3) Qd)), which uses the top-k thresholding strategy. We test d = 512, the same activated dimension as CLIP to make fair comparison. The average activated dimension of the final Lex VLA 1081. Note that Lex LexVLA VLA surpasses CLIP with less activated values, Fig. for detailed analysis.
7

---

Person
TV
Posted plant
Couch
Image
GT mask
Patch pred mask
Image
GT mask
Patch pood mask
Image
GT mask
Patch prod mask
Figure 3: PatchDis visualization. The same color indicates the same category. LexVLA correctly
predicts the corresponding region, even for the small-scale objects, like the bottle in the first image.
sky man Mountains snow
boaa
man
grass
So
sunner Farmssk
PUNES noun ountaan
sea
street
Houses
Sa
birdd
OH storm
dove
eleer
Fake
eee
ben
camera
ca
ro
homes Faet
ves
sel
Aw Lings dmp
birds vinging
shooting ing
anda
ggnnt telè
houses house roadshed
b bird
phone prannbblee
Brank
boaa
Es
boait
an animals
Figure Visualization of the image lexical representation obtained by VLX. Larger word indicates larger lexical value. The first row represents the complete image, and the second row represent local patches (boxed in red). LexVLA learns well-aligned lexical representation for both image and patches without local supervision.
Results We report the results in Table ! 1) LexVLA performs significantly better than both latent feature alignment methods and previous lexical aligmment methods when trained on a similar amount of multi-modal examples (12M vs. 15M). This shows the effectiveness of Lex VI.A. Compared to CL.P trained on much larger multi-modal datasets (0.4B and 1.1B), LexVLA still performs better, even when using the same activated feature dimension as CLIP. This further demonstrates
LexVLA's efficiency in text-image alignment with much less training data. 3) Lex VLA outperforms STAIR in most metrics and is comparable two others, despite using significantly less training data (12M vs. I.IB). This highlights Lex VLA's training efficiency. 4) The results of Q(BoW) are poor, due to the BoW method limits the model's ability to use information effectively. It struggles with vocabulary 7 and semantic [32] mismatches. Lex VLX aims to address these limitations by leveraging language models for lexical prediction rather than relying solely on captioning. 5) The significant improvements of Lex VLA over Q(CLIP) indicate that using DINOv2 the visual backhone is superior. 6) Lex VLA with FLOPS loss performs worse than our proposed overuse penalty, and activates meaningless tokens in Fig. 6 In summary, these results demonstrate the effectiveness and efficiency of our method.
4.2 Lexical representation analysis
In this part, we evaluate if Lex learns the accurate lexical repre- sentation. We conduct the following experiments: 1) Quantitatively, we use the proposed PatchDis metric to evaluate the fine-grained patch-level visual lexical representations 2) Qualitatively, we visualize the global lexical representation im- ages, local lexical representation of image patches, and a PatchDis visualization for the activated patches for the given category.
Table 2: PatchDis results.
Model
mIoU
MSCOCO 2017 [23]:
Random Dis.
5.0
CLIP
5.3
VDR
12.6
Q(CLIP)
13.9
LexVLA
36.3
PatchDis evaluation We report the zero-shot PatchDis mIoU per- formance on MSCOCO 2017 validation dataset. We compare LexVLA with public available baselines CLIP (0.4B) and VDR (15M). as well as random discrimination, and report the performance in Tab. Lex clearly shows superiority on the patch-level interpretability with the weak global training signal. contrast, the latent alignment CLIP, due to no supervision on patch-level features, performs slightly better than random guessing. VDR enjoys better interpretability, significantly worse than our VLA. Lex VLA variant, using the original CLIP visual model, provides better and more interpretable

---

Flickzu0
Flkke22K Beolis
MSCTCCO Becelg
Bealigh
MSCOCO Recillal0
Anoage Spart
Arcrag Sparts Opadratt
Figure 5: Retrieval in different sparse levels. We compare Lex with VDR in the same sparse levels and with CLIP as a proxy of dense latent alignment. The first is results in the Flickr30K dataset, and the second in the MSCOCO dataset. The first to the third columns show the settings of Recall @ I, @e and Recall 10, respectively. Purple symbols represent CLIP.
lexical representation of visual patches compared to both the original CL.IP and previous lexical methods. This further confirms the effectiveness our proposed approach. We also visualize the LexVLA successfully activates the correlated patches
predicted patches of different classes in Fig. 3 the image for different classes, even for the small-scale objects.
Lexical visualization We visualize the lexical representations of the global image and local patches in Fig. 4 For the bottom row, we randomly select one object in the image and then annotate the patches that cover the object calculate the lexical representation on patches in the free form. We use word cloud [28) to illustrate the lexical representation where larger word indicates larger lexical value. These results clearly suggest that the learned lexical representation correctly correlates with the semantics of the image/patch, showcasing the effectiveness of Lex VL..
4.3 Further analysis
LexVLA under different sparsity One of the targets for lexical representation is to learn a sparse representation for potential benefits in large scale retrieval tasks. To test the robustness of learned representations in Lex LL,, we test the retrieval performance of Lex VLA in different sparsity levels. Particularly, the sparsity ratio calculated via (si)/VV, the ratio of non-zero elements in the lexical representation st. We compare LLX with VDR on the same sparsity ratios. We also visualize the CLIP's performance as proxy of dense latent representation.
Results are shown in Fig. 5 1) LexVLA is robust against the sparsity ratio even in a very high ratio (98.27%, 296 activated tokens, which is marked with black vertical dotted line in Fig. (5. and then starts to get damaged as the sparse ratio approaches 1; 2) Compared with VDR, Lex VLA achieves better performance in all sparse level, indicating consistent improvement. 3) Lex enjoys superior performance with less activated feature dimension as CLIP (296 vs 512) in almost all cases, showcasing our model's effectiveness.
Overuse penalty The widely used FLOPs loss aims encourage the sparsity of learned lexical representation. However, we empirically find that will falsely activate meaningless tokens as in Specifically, the model tends to activate certain semantically unrelated tokens across different 9; (like @ in the image and 'ontemppraryy' in the text, respectively). implicitly treating these tokens latent dimensions. Our proposed overuse penalty inhibits these frequently activated words more aggressively, and leads to more semantic-corrella lexical representations in Fig. 6 This enhanced interpretability further improves the retrieval performance in most cases, in Tab.
Fig. 6
The major limitation of LexVLA is the use of vocabulary from large language model. We design our lexical vocabulary based on the Llama tokenizer, and sometimes it splits a word into several sub-word tokens, making the resulting token not complete word. Although we filter out many meaningless tokens, still introduces the gap between our vocabulary and the perfect word-level vocabulary. Given that random initialize word-level vocabulary and additionally learn
Limitations

---

Ovenuse
FLOPS
rE0o eda
circle
Tancc
amura
Streat
S SUm
throne
wassite i
boy
garbage
S
te bath
sink
cave
bath
damm
PoRovaste
castle
ddump
winter
winter
AKKS ky
Toilet with indefinite
conterptr ar
gransport
sport
Capcion
cow stands the back large truck
A picture of young stording on snowboard.
snow
end becouse
of bright lighT.
s1nk bath
Contampora a
GOW
SRI Mh
icon p
cattle
slope
15
Figure 6: Visualization of lexical representations of LexVLA w/ FLOPs loss and w/ proposed overuse penalty, respectively. The first row represents images while the second is caption. LexVLA w/ FLOPs loss usually falsely activates similar meaningless tokens frequently and 'contemporary' outlined by green boxes as examples). LexVLA w/ overuse penalty significantly mitigate this false activation issue.
shorteut (see '@'
projector from sub-word tokens to word-level tokens works poorly, we regard designing word-level vocabulary that still benefit from the LLMs as a future work.
Broader impacts The advancements made in cross-modal retrieval, demonstrated in this work, hold significant potential for various fields and applications. One immediate impact lies in enhancing information retrieval systems, where users efficiently search and retrieve multimodal content such as images and text. This can greatly benefit industries such as e-commerce, digital libraries, and multimedia databases, enabling more intuitive and accurate searches. While the advancements in cross modal retrieval offer numerous benefits, one concern is related to privacy and security, as the integration of multimodal data may raise issues regarding data protection and user consent. It's essential to mitigate these potential negative impacts through robust privacy measures.
5 Conclusion
In this paper, we propose Lex,LA, unified lexical vision- lision-language alignment framework. LexVLA effectively learns a unified lexical representation with unique codebooks for vision and language modalities, with the aid of single-modal pre-trained models. We propose to incremental fine-tune LexVLA with the standard contrastive objective penalized by the proposed overuse objec- tive to prevent meaningless activation and encourage sparse embedding. LexVLA refrains from the complex training configurations, and effectively achieves the patch-level interpretability with only global supervision signals. We introduce patch-level interpretability metric, PatchDis, quantify this on the zero-shot cross cross-modal benchmark dataset. We demonstrate the effectiveness of Lex VLA on two shot cross- cross-modal retrieval benchmark datasets with significantly smaller multi-modal training dataset. We also provide detailed analysis on Lex VLA.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-2377 2373366..77.. 2022.
[3] Usha Bhalla, Alex Oesterling, Suraj Srinivas, Flavio P Calmon, and Himabindu Lakkaraju. In- terpreting clip with sparse linear concept embeddings (splice). arXiv preprint UrXiv:2402.10376 2024.
[4] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 3558-3568, 2021.
10

---

[5] Chen Chen, Bowen Zhang, Liangliang Cao, Jiguang Shen, Tom Gunter, Albin Madappally Jose, Alexander Toshev, Jonathon Shlens, Ruoming Pang, and Yinfei Yang. Stair: Learning sparse text and image representation in grounded tokens. arXiv preprint arXiv:2301.13088 2023.
[6] Delong Chen, Zhao Wu, Fan Liu, Zaiquan Yang, Yixiang Huang, Yiping Bao. and Erjin Zhou. Prototypical contrastive language image pretraining. arXiv preprint arXiv:2206. 10996, 2022.
[7] W Bruce Croft, Donald Metzler, and Trevor Strohman. Search engines: Information retrieval in practice, volume 520. Addison- Addison-Wesley Reading, 2010.
[8] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances Neural Information Processing Systems, 36, 2024.
[9] Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04805, 2018.
[10] Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante- Bonilla, Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, et al. Dense and aligned captions (dac) promote compositional reasoning in vl models. Advances in Neural Information Processing Systems, 36, 2024.
[11] Jiali Duan, Liqun Chen, Son Tran, Jinyu Yang, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Multi-modal aligmment using representation codebook. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15651-15660, 2022.
[12] Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. Splade: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2288-2292, 2021.
[13] Luyu Gao, Zhuyun Dai, and Jamie Callan. Coil: Revisit exact lexical match in information retrieval with contextualieed inverted list. arXiv preprint r:Xiv:2104.07186, 2021.
[14] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[15] Amita Kamath, Jack Hessel, and Kai-Wei Chang. What's" up" with linguanlanguage models? investigating their struggle with spatial reasoning. arXiv preprint (tti:2311..9988.. 2023.
[16] Andrej Karpathy and Li Fei-Fei. Deep semual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128-3137, 2015.
[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[18] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026. 2023.
[19] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, 2018.
[20] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19745 PMLR, 2023.
11

---

[21] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: data efficient contrastive uunuuge-image pre-training paradigm. arXiv preprint arXiv:21 10.05208, 2021.
[22] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: data efficient contrastive language-image pre-training paradigm. In International Conference on Learning Representations, 2022.
[23] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th Europeun Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt V 13, pages 740-755. Springer, 2014.
[24] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[25] Ziyang Luo, Pu Zhao, Can Xu, Xiubo Geng, Tao Shen, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. Lexlip: image-text sparse retrieval. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1120--12217, 2023.
Language-imae pre-training for large-scale
[26] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets dunguage-image pre-training. In European conference computer vision, pages 529-544. Springer, 2022.
[27] Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference machine learning (ICML-10), pages 807-814, 2010.
[28] Layla Oesper, Daniele Merico, Ruth Isserlin, and Gary D Bader. Wordcloud: a cytoscape plugin to create visual semantic summary of networks. Source code for biology and medicine, 6(1):7, 2011.
[29] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03744 2018
[30] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[31] Biswajit Paria, Chih-Kuan Yeh, Ian E.H. Yen, Ning Xu, Pradeep Ravikumar, and Barnabás Póczos. Minimizing flops to learn efficient sparse representations. In International Conference on Learning Representations, 2020.
[32] ME Peters, M Neumann, M lyyer, M Gardner, C Clark, K Lee, and L Zettlemoyer. Deep contextualized word representations. corr. arXiv preprint arXiv: 1802.0563, 2018.
[33] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer mmage-to-sentenee models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2649, 2015.
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-84 8763. PMLR, 2021.
[35] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[36] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, 2016.
12

---

[37] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprint arXiv: 15389, 2023.
[38] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, lean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv: 1805, 2023.
[39] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms, arXiv preprint arXiv:2401.06209, 2024.
[40] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[41] Fei Wang. Liang Ding, Jun Rao, Ye Liu, Li Shen, and Changxing Ding. Can linguistic knowledge improve multimodal alignment in vision-language pretraining? arXiv preprint arXiv:2308. /2898, 2023.
[42] Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign lan- guage: Beit pretraining for vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19175-19186, 2023.
[43] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine- srne-grained interactive language-image pre-training. In International Conference on Learning Representations, 2022.
[44] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2005.01917, 2022.
[45] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of of-dsrds, and what to do about it? In The Eleventh International Conference Learning Representations, 2022.
[46] Hamed Zamani, Mostafa Dehghani, W Bruce Croft, Erik Lealned-Miller, and Jaap Kamps. From neural re-ranking to neural ranking: Learning a sparse representation for inverted indexing. In Proceedings of the 27th ACM international conference on information and knowledge management, pages 497-506, 2018.
[47] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Patlern Recognition, pages 18123-18133, 2022.
[48] Jiawei Zhou, Xiaoguang Li, Lifeng Shang, Xin Jiang, Qun Liu, and Lei Chen. Retrieval- based disentangled representation learning with natural language supervision. In The Twelfth International Conference on Learning Representations, 2024.
[49] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models, arXiv preprint 2304 10592, 2023.
13

---

A
Appendix supplemental material
A.1 Model hyperparameters
Our hyperparamters selection for the detail can be found in Tab. 3
Table Hyperparameters for training LexVLA.
Hyperparameter
Value
Batch size
6144
Learning rate
5e-4
Vocabulary size
17149
Training epochs
12
Max temperature
100.0
Warm-up iterations
1000
LR Scheduler
Cosine
Adam B1
0.9
Adam B2
0.999
Adam E
le-6
Penalty lambdas
5c-4
Penalty A2
le-3
Mora_alpha
16
Lora_r
8
Lora_dropont dropout
Lora
0.05
A.2 Pretraining details of backbone models
The DINOv2 33 base model used in LcxVLA is pretrained with self-supervised learning, leveraging a large dataset of 142 million diverse, high- nigh-resolution images. It employs a teacher-student architec- ture with multi-crop views and strong data augmentations encourage robustness and capture both global and local features. During pretraining, the model uses a cosine similarity loss between student and teacher outputs
consistent feature representations across different augmentations. This model supports downstream performance on detection and segmentation tasks.
The Llama model 44 used in LexVLA is pretrained with self-supervised learning on approximately trillion tokens from diverse mix publicly available datasets and web-sourced content. It employs dense transformer architecture, and during training, it focuses on token prediction accuracy and generalization. This model supports effective performance across various language tasks.
For more pretraining details, please refer to the original papers of these two pretrained models.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, they correctly describe the paper's 's contributions and scope. We propose a unified lexical alignment model that aligns text and visual data based single-modal pre- trained models and introduces unique codebooks, an overuse penalty, and enjoys patch-level interpretability.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, discussed the limitations in Sec.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Yes, provided all the information needed to reproduce the main experimen- tal results Sec. Jad Scc.A.I
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: (1) We used publicly available benchmark datasets and have provided refer- ences for each one for open access. (2) According the rules of the authors' organization, the code will be available only after the official publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
aublicc/guiees/Cdeeebbimsss www.:s/wbbbsssooooooooooo.... for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/mipss.cc/pullc/giidde
swsssssssiniyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provided all the details in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We didn't it include error bars because calculating them would take too much computing power. We're also following the approach of previous studies that didn't 't include them.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided all the required information in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///neiiipppeeeeeee weeuurpss.ccpubliiee htcc/puublc//thhiccciiileell
Answer: [Yes]
Justification:
Our research conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discussed both potential positive societal impacts and negative societal impacts in Sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work, towards the accurate lexical alignment between text and images, poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have credited all the assets we used, followed their licensing rules, and will not distribute or repurpose them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not have new assets in the submission.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "RijXMyLjBF", "reviewer": "Reviewer_KFJh", "summary": "This paper presents LexVLA, a vision language alignment method integrating a pretrained vision model and a pretrained language model. To retain the original capabilities of pretrained single-modal models, it adopts a unified lexical representation with unique codebooks. Moreover, the vision model is tuned with a projector, and the text model is tuned with LoRA. A metric for patch-level alignment is proposed to evaluate interpretability. Experiments are conducted on retrieval benchmarks.", "questions": "- How accurately or reliably does the proposed PatchDis metric evaluate/reflect the interpretability of patch-level visual lexical representations?", "limitations": "While multiple technical contributions have been made in this paper, some of the components lack rigorous verification.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow.\n- The content is rich. An architecture, an objective, and a metric are proposed.\n- Inserting lightweight components to tune vision and language models to learn lexical representation while refraining from original capability degradation is intuitive. \n- The LexVLA can be applied to various architectures.\n- Experiments are conducted on multiple benchmarks.", "weaknesses": "- Even though a new metric is proposed, the effectiveness of its reflection on interpretability is not verified quantitatively or qualitatively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Q1. How accurately or reliably does the proposed PatchDis metric reflect the interpretability of patch-level visual lexical representation? \nThank you for your concern. We have discussed and analyzed this in the main paper. Our proposed PatchDis is a direct metric for assessing the interpretability of patch-level visual lexical representation. Here’s how:\n\n**1) Interpretability of Lexical Representation:** Lexical representation is inherently interpretable, as each dimension corresponds to the activated value of a word/token in the vocabulary. This has been elaborated on the Introduction (P1-L29) and Related Work (P3-L91), particularly referencing works on lexical representation [1, 2, 3].\n\n**2) PatchDis reflects patch-level interpretability:** \nFor a lexical representation of selected patches to be considered interpretable, it should accurately reflect local semantic information without being influenced by non-selected patches or incorrect semantics. In this context, interpretability indicates how well the image patches are represented with clear and understandable semantics.\n\nOur new metric, PatchDis, is explicitly designed to measure how effectively visual patches represent their intended semantics. As discussed in Sec. 3.4 and Sec. 4.2., PatchDis allows us to measure and compare the interpretability of patch-level features across different models, providing a quantitative evaluation of visual feature interpretability at the patch level.\n\n**3) Visualization and Evaluation:** Our visualization of PatchDis, based on the popular MSCOCO 2017 dataset, provides a qualitative reflection of its effectiveness in terms of interpretability.  The segmentation results offer detailed insights into how the model makes matching decisions. This aligns with the definition of interpretability by Biran and Cotton [4], which refers to the degree to which an observer can understand the cause of a decision.\n\nIn summary, the evaluation of lexical representation, both quantitative and qualitative, directly reflects this interpretability. \n\nWe hope these points address your concern, and we will update these information accordingly if accepted.\n\n\n[1] Formal, Thibault, Benjamin Piwowarski, and Stéphane Clinchant. \"SPLADE: Sparse lexical and expansion model for first stage ranking.\" Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2021.\n\n[2] Luo, Ziyang, et al. \"Lexlip: Lexicon-bottlenecked language-image pre-training for large-scale image-text sparse retrieval.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n\n[3] Zhou, Jiawei, et al. \"Retrieval-based Disentangled Representation Learning with Natural Language Supervision.\" The Twelfth International Conference on Learning Representations. 2024.\n\n[4]  Biran, Or, and Courtenay Cotton. \"Explanation and justification in machine learning: A survey.\" IJCAI-17 workshop on explainable AI (XAI). Vol. 8. No. 1. 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OQXNj0iejA", "reviewer": "Reviewer_a8Qd", "summary": "The paper proposes LexVLA, a more interpretable VLA framework that learns a unified lexical representation for both modalities without complex design.\nLexVLA uses DINOv2 as the visual model and Llama 2 as the language model, proposing an overuse penalty to avoid false discoveries.\nLexVLA outperforms baselines on cross-modal retrieval benchmarks, even when fine-tuned on a modest dataset.\nExtensive experiments were conducted to analyze LexVLA's performance.", "questions": "Please address the questions in the weakness.", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is easy to follow.\n2. The framework does not require complex design or training configurations, making it more accessible and efficient.\n3. LexVLA outperforms baselines on cross-modal retrieval benchmarks, even when compared to models trained on larger datasets.\n4. Ablation demonstrates the decision choice and effectiveness of proposed components.", "weaknesses": "1. I can't quite get the novelty of this work. The lexical representation mentioned in the paper is somehow a way to select important information and then map it to the code book. However, the codebook strategy was explored [1]. Especially the visual part, where does the concept of Lexical come in? Can the author elaborate more on this?\n2. In Table 1, the improvement is pretty limited in the bottom block compared to using CLIP in the last and first blocks. It makes readers question whether the performance was gained by the DINOv2 representation.\n3. The alignment was tested on only one task, it will be more interesting to test on other multimodal tasks such as zeroshot classification, or even grounding since it has DINOv2 representation.\n\n\n[1] Duan, Jiali, et al. \"Multi-modal alignment using representation codebook.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Q1-1. Is lexical representation a way to select important information and map it to the codebook?\n\nThank you for your question. We respectfully disagree with this characterization. Lexical representation is not a codebook strategy in [1]. Learning a unified codebook for multi-modal data is fundamentally different from learning a lexical representation. Here’s how they differ:\n1. **Purpose and Approach:**  As discussed in Sec. 3, our goal is to learn the interpretable lexical representation for images and image patches. In contrast, the codebook strategy [1] aims to learn unified, but uninterpretable and non-explicit-semantic-meaningful codes for image and text.\n2. **Alignment Mechanism:** As explained in Sec. 3.1, our alignment is conducted at the vocabulary level, not the codebook level. We focus on aligning indexes, which provides more flexibility when training our lexical encoders compared to training codebook learners. In contrast,  the codebook strategy [1] attempts to align the codebook embeddings.\n3. **Enhanced Interpretability and Efficiency:** As mentioned in the Sec. 2, the lexical representation is widely studied in the field of text information retrieval. This method has been employed in vision-language models due to its good interpretability and high retrieval efficiency. The dense codebook strategy does not meet these properties.\n\n\n[1] Duan, Jiali, et al. \"Multi-modal alignment using representation codebook.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n## Q1-2. Where does the concept of Lexical come in regarding the visual part?\nAs we introduced on P4, L134-L137, the lexical representation is the output of the visual encoder.  And on P3, L102-L104, this output is a score vector, whose element indicates the similarity between the image and the corresponding word from the vocabulary.\n\n## Q2. Limited improvement over CLIP? \nThank you for raising this concern. We would like to clarify that our improvements over CLIP are significant and demonstrate our advantages over competitors. Our main paper and the rebuttal PDF provide comprehensive experiments and analysis to support this. Specifically, as discussed in the analysis of the experimental results (L238-L248)\n1. **Comparison of different frameworks:** Comparing LexVLA(CLIP) with dense CLIP, our method shows comparable or even better performance. This indicates that our approach achieves well-aligned lexical representations with significantly less multimodal training data (12M) compared to CLIP (1.1B).\nAdditionally, our model ensures semantic correctness of the tokens, which is more challenging than merely aligning latent features, as discussed on P1, L32-L36.\n2. **Further Enhancement with DINOv2 Features:**  Replacing CLIP with the DINOv2 encoder, which provides local-inclined features, further improves our results. This demonstrates that good interpretability contributes positively to retrieval performance. This also aligns with our motivation.\n3. **Overall Improvement:** Our full model shows substantial improvement over CLIP in our experimental setup, offering superior retrieval performance and better interpretability, as detailed in Table 1 of the rebuttal PDF file. \n\n## Q3. More interesting to test the alignment on grounding multimodal tasks.\n\nThank you for your suggestion. Our work primarily focuses on learning interpretable lexical representation for  images, our experiments are extensive and support our claims and contributions. Importantly, our paper does include multiple multimodal tasks. Specifically:\n\nAs stated in Section 3.4 and Section 4.2, we proposed a grounding task called PatchDis. This task is inspired by patch-level segmentation but is designed for models that are not trained on fine-grained alignment tasks such as segmentation or detection. Our experiments, as reported in Table 2, demonstrate that our model performs significantly better on this grounding task. This highlights the effectiveness of our approach.\n\nWe agree that exploring additional multimodal downstream tasks would be interesting. We are keen to investigate further applications of our proposed LexVLA in the future. Since our current work is self-contained, this could be a valuable direction for future research.\n\nWe will clarify these points further in our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JsgFCDf47f", "reviewer": "Reviewer_CyPa", "summary": "The authors propose a method based on lexical representation for Visual-Language Alignment (VLA). The method relies on aligning two strong unimodal models, namely DINOv2 for the visual modality and Llama 2 for the text modality. Each backbone is fine-tuned with a few adapters or additional layers. The two modalities use separate codebooks mapping to a joint vocabulary. The authors also propose an overuse penalty to limit the excessive activation of irrelevant tokens. Finally, the authors introduce the PatchDis metric to measure patch-level alignment. Evaluation on zero-shot cross-modal retrieval datasets shows state-of-the-art performance of the method with the compared baselines. Additional experiments on the patch-level representation and sparsity showing the effectiveness of the method are also reported.", "questions": "- Have the authors explored a simpler approach of just selecting the nouns, adjectives, and non-auxiliary verbs in a caption instead of the LLM-based lexical predictor? How many keywords are extracted by the LLM on average per caption? Does it vary with the length of the caption?\n- Eq (3), what is x with any index?\n- Table 1: it would be good to also indicate the amount of (unimodal) pretraining data (if any) used for each method e.g. the amount of data used for DINOv2 and LLama 2 for the proposed method. What are the test splits used for this experiment? Commonly, results are reported based on the splits in [Karpathy, Andrej, and Li Fei-Fei. \"Deep visual-semantic alignments for generating image descriptions.\" In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3128-3137. 2015.]. If these are not used it would be good to use them as well.\n- Figure 3: it would be good to provide the class <-> color mapping here.\n- Figure 4: bottom row, how were the local patches selected?\n- Figure 5: what does the vertical black dotted line represent? How were the different sparsity level selected?\n- The authors mention in the limitations that their “lexical vocabulary based on the Llama 2’s tokenizer (...) splits a word into several sub-word tokens.” does that also mean that some rather rare long words would not appear in the vocabulary? Have the authors studied what are these missing words? Further down the authors state “Given that random initialize a word-level vocabulary and additionally learn a projector from sub-word tokens to word-level tokens works poorly, we regard designing a word-level vocabulary that still benefit from the LLMs as a future work.”, it seems the author did conduct some experiments towards that. Even if the results were not conclusive it would be interesting to share what was tried and what was the performance.\n\nTypos etc:\n- p2-l52: missing space “​​LexVLAto”", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors proposed an effective and interpretable Lexical Representation approach for Visual-Language Alignment\n- The proposed method is described clearly\n- The experimental results show state-of-the-art performance in comparison to the baseline selected", "weaknesses": "- The vocabulary is based on the Llama tokenizer which, as stated in the limitations, may split words into meaningless sub-word tokens and may also lack longer relevant words.\n- The latent baselines for zero-shot cross-modal retrieval do not include recent methods such as BEiT-3 [Wang, Wenhui, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal et al. \"Image as a foreign language: Beit pretraining for vision and vision-language tasks.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19175-19186. 2023.]\n- One main difference with the compared methods could be the use of the DINOv2 visual backbone and the Llama 2 textual backbone, it is possible the proposed method benefits from these strong backbones. All methods' visual and text backbones (and their potential pretraining) should be discussed in detail to enable the readers to properly judge the merit of the proposed method", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1. Tokenizer-based vocabulary is not perfect.\nThanks. Please note that LexVLA has achieved SOTA performance in most experiments, demonstrating its effectiveness. While it is nontrivial to design a perfect vocabulary to handle all the corner cases generally, we would take it as an important future work and already thoroughly discussed in the limitation parts (L289-295). Nevertheless, our main contributions are still well supported by our empirical evidence here, and our work is self-contained.\n\n## W2. Compare with BEiT-3?\nThanks for suggesting the paper of BEiT-3 [1]. We will address how it significantly differs from our work (and thus not included in our original submission), while we will cite and discuss it in the related work in the revision. Particularly,\n\n1) Differences in Goals and Training Strategies:  BEiT-3 uses a masked data modeling strategy to develop a general-purpose model utilizing both single- and multi-modal data; in contrast our approach focuses on developing a specific lexical representation for images and texts. \nAs our target is to learn interpretable lexical representation for images/texts, and BEiT-3 has no such interpretable representations, comparing our method with BEiT-3 would not be informative for analyzing the effectiveness of lexical methods.\n\n2) The comparison between lexical methods and dense methods is to demonstrate that introducing lexical representation does not degrade alignment performance, as shown in the retrieval task. Hence we compare with the contrastive-trained methods, i.e., CLIP and its variants, which are more relevant for assessing the effectiveness of lexical representation.\n\n## W3. The proposed method possibly benefits from strong backbones?\nThanks for this comment. But actually, we didn't propose a general alignment method for any two visual and textual backbones. LexVLA does employ the unique properties of DINOv2 and Llama2. Specifically, as highlighted in our first contribution (L68-L70), we utilize single-modal pre-trained models for vision-language alignment tasks to exploit their unique properties, which cannot be fully captured by contrastive objectives alone. The analysis of the results in Sect 4.1 and 4.2 reflects the advantages of the chosen backbone. As we use standard DINOv2 and Llama2, we will elaborate their pretraining details into Supplementary of our paper as suggested.\n\n## Q1-1. Simpler baseline? \nThanks for your suggestion. We actually do have explored the approach you suggested, which is one typical and simple method called Bag-of-Words (BOW). The results were poor. This aligns with findings in previous studies [2, 3, 4], hence we did not include these results in our submission. We report results in the rebuttal PDF.\n\nThe BOW method limits the model's ability to use information effectively. It struggles with vocabulary [5] and semantic [6] mismatches. LexVLA aims to address these limitations by leveraging language models for lexical prediction rather than relying solely on captioning.\n\n## Q1-2. How many keywords are extracted? Does it vary with the length of the caption?\nWe want to clarify that as stated in Sec. 3.2, when we use a LLM as a lexical predictor, the caption is mapped into a token embedding $y_{text}$, which is then mapped to a lexical representation $s_{text}$.\nThe token embedding is not made up of specific keywords but a compact representation. \n\n## Q3. Details on pretraining data and test splits?\n\n1) All pretrained models are public released models. Llama 2 is trained on 2 trillion tokens and DINOv2 is trained on 142M images.\n\n2) Following previous approaches, the test splits are from [7]. \n\nWe will include these information in the revision.\n\n## Q4. Questions about Figures.\n\n1) Fig 3: Thanks. We have added the label mapping in the rebuttal PDF.\n\n2) Fig 4: In the bottom row, we randomly select one object in the image and then annotate the patches that cover the object to calculate the lexical representation on patches in the free form.\n\n3) Fig 5: a) Vertical black dotted line: The vertical black dotted line represents the sparse level of our final LexVLA model, which is reported (98.27\\%, 296 activated tokens) on P9, L278.\nb) Different sparsity levels: As on P4, L122-L127, sparse levels can be selected via Top-k or Value thresholding. The sparse levels in Fig. 5 were selected using Value thresholding.\n\n## Q5. Question about lexical vocabulary.\nThanks, we tokenize all the words in the test set, and found that even the most frequently split words did not appear often, and most of them retained semantics, such as:\nbathroom -> bath, room;\nmotorcycle -> motor, cycle;\nsidewalk -> side, walk.\nWhile corner cases exist, LexVLA has demonstrated effectiveness and achieved SOTA performance in most experiments. We would take it as an important future work and have thoroughly discussed it in the limitations section of our paper (L289-295).\nPlease note that our main contributions are well supported, and our work is self-contained. \n\n## Q2 & Typos\nThanks!\nThe $x$ in Eq(3) should have index $img$.\nWe will revise all the typos pointed out by the reviewers in the final version and continue to proofread the manuscript to reduce any potential confusion.\n\n\n\n[1] Wang et al. “Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks.” CVPR 2023\n\n[2] Gao et al. \"COIL: Revisit exact lexical match in information retrieval with contextualized inverted list.\" arXiv:2104.07186.\n\n[3] Xiong, et al. \"End-to-end neural ad-hoc ranking with kernel pooling.\" SIGIR 2017\n\n[4] Formal et al. \"SPLADE: Sparse lexical and expansion model for first stage ranking.\" SIC 2021\n\n[5] Croft et al. Search engines: Information retrieval in practice. Vol. 520. Reading: Addison-Wesley, 2010.\n\n[6] Peters, E. et al. “Deep Contextualized Word Representations.” ArXiv abs/1802.05365\n\n[7] Karpathy et al \"Deep visual-semantic alignments for generating image descriptions.\" CVPR 2015", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our sincere gratitude to all the reviewers for their insightful and constructive feedback on our paper. We are delighted that our work has been positively received, and we appreciate the time and effort each reviewer has put into evaluating our submission.

We are particularly grateful for the recognition of several strengths: our paper’s clarity and ease of understanding (Reviewer KFJh, Reviewer a8Qd, Reviewer CyPa), the rich content and novel contributions including the architecture, objective, and metric (Reviewer KFJh, Reviewer CyPa), and the intuitive approach of inserting lightweight components without degrading original capabilities (Reviewer a8Qd, Reviewer CyPa). The flexibility of the LexVLA framework (Reviewer KFJh, Reviewer a8Qd) and the comprehensive experiments conducted on multiple benchmarks (Reviewer KFJh, Reviewer a8Qd) were also appreciated. Additionally, the straightforward design and efficiency of our framework (Reviewer a8Qd), its superior performance on cross-modal retrieval benchmarks (Reviewer KFJh, Reviewer CyPa), and the clear demonstration of the effectiveness of our proposed components (Reviewer KFJh, Reviewer a8Qd) were well-received. Lastly, we are pleased that our method’s effectiveness and interpretability and the state-of-the-art performance compared to baselines (Reviewer CyPa) have been acknowledged.

We also extend our gratitude to the Area Chair for their guidance and support throughout the review process.

Thank you once again for your valuable feedback.

Meta-review: After discussion, this submission received 3 positive scores. After rebuttal, the major concerns about interpretability of the proposed approach, result visualization, experimental evaluation, and comparison of difference frameworks were comprehensively solved. After reading the paper, the review comments and the rebuttal, the AC think the remaining concern is about the lack of critical evaluation of the proposed approach, which is encouraged to added to the camera-ready version.
Decision: Accept (poster)