Title: Unified Lexical Representation for Interpretable Visual-Language Alignment
Venue: NeurIPS 2024
Accepted: yes
Keywords: Multi-modal

Alignment

Retrieval

Sparse Retrieval

Lexical Representation
Abstract: Visual-Language Alignment (VLA) has gained a lot of attention since CLIP's groundbreaking work. 
Although CLIP performs well, the typical direct latent feature alignment lacks clarity in its representation and similarity scores. 
On the other hand, lexical representation, a vector whose element represents the similarity between the sample and a word from the vocabulary, is a natural sparse representation and interpretable, providing exact matches for individual words.
However, lexical representations are difficult to learn due to no ground-truth supervision and false-discovery issues, and thus requires complex design to train effectively.
In this paper, we introduce LexVLA, a more interpretable VLA framework by learning a unified lexical representation for both modalities without complex design. 
We use DINOv2 as our visual model for its local-inclined features and Llama 2, a generative language model, to leverage its in-context lexical prediction ability.
To avoid the false discovery, we propose an overuse penalty to refrain the lexical representation from falsely frequently activating meaningless words.
We demonstrate that these two pre-trained uni-modal models can be well-aligned by fine-tuning on the modest multi-modal dataset and avoid intricate training configurations. 
On cross-modal retrieval benchmarks, LexVLA, trained on the CC-12M multi-modal dataset, outperforms baselines fine-tuned on larger datasets (e.g., YFCC15M) and those trained from scratch on even bigger datasets (e.g., 1.1B data, including CC-12M).
We conduct extensive experiments to analyze LexVLA. 
Codes are available at https://github.com/Clementine24/LexVLA.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method based on lexical representation for Visual-Language Alignment (VLA). The method relies on aligning two strong unimodal models, namely DINOv2 for the visual modality and Llama 2 for the text modality. Each backbone is fine-tuned with a few adapters or additional layers. The two modalities use separate codebooks mapping to a joint vocabulary. The authors also propose an overuse penalty to limit the excessive activation of irrelevant tokens. Finally, the authors introduce the PatchDis metric to measure patch-level alignment. Evaluation on zero-shot cross-modal retrieval datasets shows state-of-the-art performance of the method with the compared baselines. Additional experiments on the patch-level representation and sparsity showing the effectiveness of the method are also reported.\n\nSTRENGTHS:\n- The authors proposed an effective and interpretable Lexical Representation approach for Visual-Language Alignment\n- The proposed method is described clearly\n- The experimental results show state-of-the-art performance in comparison to the baseline selected\n\nWEAKNESSES:\n- The vocabulary is based on the Llama tokenizer which, as stated in the limitations, may split words into meaningless sub-word tokens and may also lack longer relevant words.\n- The latent baselines for zero-shot cross-modal retrieval do not include recent methods such as BEiT-3 [Wang, Wenhui, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal et al. \"Image as a foreign language: Beit pretraining for vision and vision-language tasks.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19175-19186. 2023.]\n- One main difference with the compared methods could be the use of the DINOv2 visual backbone and the Llama 2 textual backbone, it is possible the proposed method benefits from these strong backbones. All methods' visual and text backbones (and their potential pretraining) should be discussed in detail to enable the readers to properly judge the merit of the proposed method\n\nQUESTIONS:\n- Have the authors explored a simpler approach of just selecting the nouns, adjectives, and non-auxiliary verbs in a caption instead of the LLM-based lexical predictor? How many keywords are extracted by the LLM on average per caption? Does it vary with the length of the caption?\n- Eq (3), what is x with any index?\n- Table 1: it would be good to also indicate the amount of (unimodal) pretraining data (if any) used for each method e.g. the amount of data used for DINOv2 and LLama 2 for the proposed method. What are the test splits used for this experiment? Commonly, results are reported based on the splits in [Karpathy, Andrej, and Li Fei-Fei. \"Deep visual-semantic alignments for generating image descriptions.\" In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3128-3137. 2015.]. If these are not used it would be good to use them as well.\n- Figure 3: it would be good to provide the class <-> color mapping here.\n- Figure 4: bottom row, how were the local patches selected?\n- Figure 5: what does the vertical black dotted line represent? How were the different sparsity level selected?\n- The authors mention in the limitations that their “lexical vocabulary based on the Llama 2’s tokenizer (...) splits a word into several sub-word tokens.” does that also mean that some rather rare long words would not appear in the vocabulary? Have the authors studied what are these missing words? Further down the authors state “Given that random initialize a word-level vocabulary and additionally learn a projector from sub-word tokens to word-level tokens works poorly, we regard designing a word-level vocabulary that still benefit from the LLMs as a future work.”, it seems the author did conduct some experiments towards that. Even if the results were not conclusive it would be interesting to share what was tried and what was the performance.\n\nTypos etc:\n- p2-l52: missing space “​​LexVLAto”\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes LexVLA, a more interpretable VLA framework that learns a unified lexical representation for both modalities without complex design.\nLexVLA uses DINOv2 as the visual model and Llama 2 as the language model, proposing an overuse penalty to avoid false discoveries.\nLexVLA outperforms baselines on cross-modal retrieval benchmarks, even when fine-tuned on a modest dataset.\nExtensive experiments were conducted to analyze LexVLA's performance.\n\nSTRENGTHS:\n1. The paper is easy to follow.\n2. The framework does not require complex design or training configurations, making it more accessible and efficient.\n3. LexVLA outperforms baselines on cross-modal retrieval benchmarks, even when compared to models trained on larger datasets.\n4. Ablation demonstrates the decision choice and effectiveness of proposed components.\n\nWEAKNESSES:\n1. I can't quite get the novelty of this work. The lexical representation mentioned in the paper is somehow a way to select important information and then map it to the code book. However, the codebook strategy was explored [1]. Especially the visual part, where does the concept of Lexical come in? Can the author elaborate more on this?\n2. In Table 1, the improvement is pretty limited in the bottom block compared to using CLIP in the last and first blocks. It makes readers question whether the performance was gained by the DINOv2 representation.\n3. The alignment was tested on only one task, it will be more interesting to test on other multimodal tasks such as zeroshot classification, or even grounding since it has DINOv2 representation.\n\n\n[1] Duan, Jiali, et al. \"Multi-modal alignment using representation codebook.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\nQUESTIONS:\nPlease address the questions in the weakness.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper presents LexVLA, a vision language alignment method integrating a pretrained vision model and a pretrained language model. To retain the original capabilities of pretrained single-modal models, it adopts a unified lexical representation with unique codebooks. Moreover, the vision model is tuned with a projector, and the text model is tuned with LoRA. A metric for patch-level alignment is proposed to evaluate interpretability. Experiments are conducted on retrieval benchmarks.

STRENGTHS:
- The paper is well-written and easy to follow.
- The content is rich. An architecture, an objective, and a metric are proposed.
- Inserting lightweight components to tune vision and language models to learn lexical representation while refraining from original capability degradation is intuitive. 
- The LexVLA can be applied to various architectures.
- Experiments are conducted on multiple benchmarks.

WEAKNESSES:
- Even though a new metric is proposed, the effectiveness of its reflection on interpretability is not verified quantitatively or qualitatively.

QUESTIONS:
- How accurately or reliably does the proposed PatchDis metric evaluate/reflect the interpretability of patch-level visual lexical representations?

LIMITATIONS:
While multiple technical contributions have been made in this paper, some of the components lack rigorous verification.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13