ContextBLIP: Doubly Contextual Alignment for Contrastive Image Retrieval from Linguistically Complex Descriptions (2024)

Abstract: Image retrieval from contextual descriptions (IRCD) aims to identify an image within a set of minimally contrastive candidates based on linguistically complex text. Despite the success of VLMs, they still significantly lag behind human performance in IRCD. The main challenges lie in aligning key contextual cues in two modalities, where these subtle cues are concealed in tiny areas of multiple contrastive images and within the complex linguistics of textual descriptions. This motivates us to propose ContextBLIP, a simple yet effective method that relies on a doubly contextual alignment scheme for challenging IRCD. Specifically, 1) our model comprises a multi-scale adapter, a matching loss, and a text-guided masking loss. The adapter learns to capture fine-grained visual cues. The two losses enable iterative supervision for the adapter, gradually highlighting the focal patches of a single image to the key textual cues. We term such a way as intracontextual alignment. 2) Then, ContextBLIP further employs an inter-context encoder to learn dependencies among candidates, facilitating alignment between the text to multiple images. We term this step as inter-contextual alignment. Consequently, the nuanced cues concealed in each modality can be effectively aligned. Experiments on two benchmarks show the superiority of our method. We observe that ContextBLIP can yield comparable results with GPT-4V, despite involving about 7,500 times fewer parameters. Our code is available at https://github.com/LHL3341/ContextBLIP.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 16240–16258 August 11-16, 2024 ©2024 Association for Computational Linguistics
1 Introduction
Text-to-image retrieval is a fundamental crossmodal task that aims to search images for textual queries. Early studies relied on convolutional
∗Equal Contribution. †Corresponding author.
neural networks (CNN) (Lecun et al., 1998) and Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) to learn to extract image and text features and then proceeded to align the representations of two modalities for retrieval. Recent proliferated large vision-language models (VLMs), such as vision Transformers (ViTs) (Dosovitskiy et al., 2020), UNITER (Chen et al., 2020) and CLIP (Radford et al., 2021), which are trained on large-scale short text-image corpus, have made remarkable progress for retrieving images from sentences with few objects and simple linguistic.
However, in the real world, natural languages are highly contextual (Fodor, 2001; Krojer et al., 2022) with long utterances. Context, including perceptual and temporal cues, plays a pivotal role in grounding the implication of a linguistically complex text (Li et al., 2023b). Figure 1 demonstrates such a case that identifies an image from six very similar candidates with a grammatically complicated description. Two major challenges of the retrieval are: 1) A model needs to understand nuanced textual cues, such as “hand is blurry”, “eyes are almost shut”, and “looking at...” across three grammatically complex sentences and align them with various context
16240
cues in each image. 2) Long-range dependencies among candidate images need to be captured to perform cross-modal reasoning for further alignment. The above interesting and challenging task is known as image retrieval from contextual descriptions (IRCD) (Krojer et al., 2022).
Despite the great success of VLMs, they can hardly tackle the above two challenges, and significantly lag behind human performance. Existing VLMs applied to text-image retrieval mainly include contrastive-based ones such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), randomly mask-based ones such as M3AE (Geng et al., 2022) and MaskVLM (Kwon et al., 2022), and attention-based ones such as ALBEF (Li et al., 2021) and BLIP (Li et al., 2022). These models have some limitations. 1) The former two focus more on high-level semantic alignment, while the fine-grained contextual cues may be largely ignored. 2) Existing mask-based ones randomly remove image patches, without specifically learning to concentrate on the key objects associated with text tokens, e.g., “hand” and “eyes” in Figure 1. 3) Dependencies among images are not specifically considered.
Previous state-of-the-art NDCR (Li et al., 2023b) proposed for IRCD divided the complex alignment into multiple simple ones and then combined them for final retrieval. However, the performance is highly dependent on the candidate’s distributions and is poor for fine-grained alignment on static images with a large variance. We observe that NDCR can hardly capture the key contextual cues in grammatically complex long sentences. Further, it also lacks zero-shot capability. Consequently, NDCR still suffers from the two challenges of IRCD. Details are discussed in Table 1 of Experiments.
To this end, we introduce ContextBLIP, a novel doubly contextual alignment scheme for the challenging IRCD task based on BLIP (Li et al., 2022). 1) Specifically, our ContextBLIP comprises a multiscale adapter, a matching loss, and a text-guided masking loss. The learnable adapter, which is inserted into frozen BLIP, aims to capture higherlevel and lower-level visual features of the candidate images. The two losses enable iterative supervision during the training stage, gradually allowing the adapter to highlight the focal patches of a single image to the linguistically complex textual cues. Such a way is termed as intra-contextual alignment that aims to tackle the first challenge issue of the IRCD task. 2) Then, we further fine-tune
ContextBLIP with a temporal Transformer to learn dependencies among candidate images, facilitating alignment between text to multiple images. This step is termed as inter-contextual alignment that aims to address the second challenge issue of IRCD. Experiments on a public benchmark show the effectiveness of the proposed ContextBLIP. The main contributions of this paper are listed as follows.
• We propose ContextBLIP, a simple yet effective method that relies on a doubly contextual alignment scheme for IRCD. It consists of a multi-scale adapter, a matching loss, and a text-guided masking loss, to learn to align the nuanced visual and textual cues, thus effectively tackling the first challenge of IRCD.
• We apply ContextBLIP for the zero-shot IRCD, and further fine-tune ContextBLIP with a temporal Transformer to learn the dependencies among different candidates, thus properly addressing the second challenge of IRCD.
• We conduct extensive experiments under various settings to show the superiority of our method. Our ContextBLIP can achieve comparable performance with proliferated GPT4V (Yang et al., 2023) under various prompts. We also evaluate our ContextBLIP on a very recent benchmark MMVP-VLM and the results further confirm the effectiveness of the proposed method.
2 Related Work
Vision-language models (VLMs): VLMs (Radford et al., 2021; Li et al., 2021; Wang et al., 2022; Li et al., 2022, 2023a; Zhai et al., 2023; Fang et al., 2023; Xu et al., 2023; Sun et al., 2023) have shown great potential on image-text retrieval (Lin et al., 2014; Krishna et al., 2017; Plummer et al., 2015). However, tuning these models for new tasks is expensive. Adapters (Houlsby et al., 2019; Gao et al., 2021; Zhang et al., 2021; Chen et al., 2022; Lu et al., 2023), which can be inserted into a pretrained VLM model, facilitate efficient adaptation to new tasks. The key difference between our adapter and previous ones: ours comprises multiple down-projection adapter layer (DPAL) that connect to the same up-projection adapter layer (UPAL). Each DPAL is inserted into distinct VLM layers, while the UPAL resides outside of the VLM. Such a design can effectively capture multi-level
subtle cues. While previous ones include multiple pairs of DPAL and UPAL inserted in VLM. Masked Image Modeling (MIM): MIM, such as ViBERT (Lu et al., 2019) and MAE (He et al., 2022), refers to predicting the missing pixels in an image by using the surrounding pixels as context. MIM has been applied for various tasks, such as robust learning (Wang et al., 2023) and generation target (Bao et al., 2021, 2022). Recent studies, such as MaskVLM (Kwon et al., 2022), M3AE (Geng et al., 2022), VL-BEiT (Bao et al., 2022) extend MIM for both image and language masking. The key difference between ours and existing ones is: we employ a text-guided masking scheme that learns to generate masks under iterative supervision with two losses, thus gradually highlighting the focal patches of a single image to the key textual cues. While the previous ones randomly remove image patches, without specifically learning to focus the key visual objects associated with text tokens.
3 Our ContextBLIP
3.1 Overall Architecture
Figure 2(a) presents the overall architecture of the proposed ContextBLIP for the challenging IRCD task, which consists of an intra-context encoder based on frozen BLIP, a multilayer perceptron (MLP) module, and an image decoder. The overall procedure can be described in three steps. (1) Pretraining: We first pre-train the proposed ContextBLIP on the large-scale COCO (Lin et al.,
2014) and VG (Krishna et al., 2017) datasets. Aiming at tackling the first challenge of IRCD, ContextBLIP includes three key ingredients, i.e., a multi-scale adapter (Figure 2(b)), and two cosupervision losses including image-text matching (ITM, Litm) loss and text-guided masked image modeling (TMIM, Ltmim) loss. (2) Intra-context Alignment: We directly apply pre-trained ContextBLIP for zero-shot IRCD (Figure 2(d)) on the public benchmark (e.g., IMAGECODE), effectively aligning the nuanced visual and textual cues. (3) Intra- and Inter-context Alignment: Then, we further fine-tune ContextBLIP (Figure 2(e)) on an IRCD benchmark with an inter-context encoder to learn long-range dependencies among image candidates, thus effectively addressing the second challenge of IRCD. Next, we detail four key components of our method, including the multiscale adapter, Litm, Ltmim and the inter-context encoder.
3.2 Multi-Scale Adapter
Our multi-scale adapter, which resides in the intracontext encoder of ContextBLIP, aims to learn to align nuanced visual and textual cues from tiny areas of an image and within linguistically complex descriptions, respectively. Figure 2 (b) illustrates the architecture of the proposed adapter. It comprises multiple down-projection adapter layer (DPAL) and a up-projection adapter layer (UPAL). DPALs are inserted into different ViT layers of frozen vanilla BLIP, while the UPAL resides out-
side of ViT layers. All DPALs connect to the same UPAL, such that both higher-level and lower-level features of candidate images can be effectively captured. We give the detailed formulation as follows.
For a pair of image-text (I, T ), I ∈ Rh×w×c, T ∈ R1×t, where h,w, c are height, width and the number of channels of the image, t is the number of tokens in sentence T . We split an image into p × p patches and then augment them with positional encoding. We feed these image patches into the m-layer ViT. The intermediate visual representations generated at the l-th layer of the ViT can be denoted as X l = [xl1, · · · , xli, · · · , xlp2 ], where l ∈ [1,m], i ∈ [1, p2] and xli ∈ Rd. Here d indicates the representation dimension of the ViT at the l-th layer. We feed X l into a down-project adapter layer (DPAL) for mapping to lower-dimensional space. The output of DPAL can be expressed as follows,
X̃ l = DPAL(X l), (1)
where DPAL is an MLP network and X̃ l ∈ Rp2×d̃. Here d̃ = d/δ, where δ is the downsampling rate of DPAL in our adapter. We use the same rate for all DPALs and obtain the output representations from other DPALs. We aggregate these representations by simply concatenating or adding. Then, the aggregated representations X̃ will be fed into the proposed up-projection adapter layer (UPAL) for up-projection mapping. The output of the UPAL can be expressed as follows,
Y = UPAL(X̃), (2)
where Y ∈ Rp2×d and UPAL is an MLP network. Finally, we add the output of ViT to Y to obtain the final representations of the image for crossmodal matching. The textual query is encoded by frozen BERT (Devlin et al., 2019) of vanilla BLIP and then is fed into the fusion layer. We feed the representations of the fusion layer’s output to the scorer of the intra-context encoder to get a matching score e. By doing so, our multi-scale adapter can facilitate fine-grained interactions between subtle visual regions and linguistic concepts.
3.3 Co-supervision under Litm and Ltmim We train the proposed adapter with two losses, i,e., Litm and Ltmim, that offer collaborative supervision to highlight key contextual cues in two modalities. Our ContextBLIP performs two separate forward computations to calculate Litm and Ltmim. We detail them as follows.
Step 1: Computing Litm: We sequentially feed a pair of image-text into the intra-context encoder and the MLP-based scorer, and obtain the matching score for the pair. The matching loss Litm can be expressed as follows.
Litm = 1
3N
3N∑
i=1
CrossEntropy(ei, qi), (3)
where ei ∈ R2, i ∈ [1, 3N ], indicates the matching score of the i-th image-text pair, and qi ∈ R2, i ∈ [1, 3N ], refers to the groundtruth label that consists of 0 and 1. Here N is the training batch size. Inspired by vanilla BLIP, we additionally generate 2N pairs of negative samples based on cosine distances to N pairs. By distinguishing much more similar image candidates, our ContextBLIP can learn to align the nuanced textual and visual context concealed in tiny areas and within complex descriptions. Step 2: Generating masks and computing Ltmim: We rely on cross-attentions outputted by the fusion layer to generate the text-guided image masking matrix. Figure 2 (c) demonstrates the generation procedure. Specifically, we manually define a mask ratio π,π ∈ [0, 1], to determine the number of patches to be masked. Top π patches with the highest attention scores will be masked and then we can get the masking matrix. We remove the patches according to the mask matrix, feed the masked image to the intra-context encoder, and then use a Transformer-based decoder to reconstruct the image. The pixel-level reconstruction loss Ltmim based on mean squared error (MSE) for a N -size training batch can be expressed as follows.
Ltmim = 1
N
1
µ
N∑
i=1
µ∑
j=1
S∑
s=1
MSE(yijs, ŷijs), (4)
where µ is the number of masked patches and S is the number of pixels in each patch. Here ŷijs and yijs refer to the original and corresponding reconstructed pixel respectively for the s-th pixel of the j-th masked patch in i-th instance of a training batch. Step 3: Iterative refinement: The total loss of our ContextBLIP can be formulated as:
L = Litm + Ltmim. (5) Details are available in Appendix A.3.
We iteratively perform the above steps supervised by the two losses to learn to update the parameters of the multi-scale adapter, and refine the textguided matrix, allowing the proposed ContextBLIP to gradually concentrate on the focal visual contextual keys associated with the textual cues.
3.4 Inter-context Encoder The task of image retrieval from contextual descriptions (IRCD) requires understanding long-range contextual dependencies among candidate images. Keeping this in mind, we introduce a simple yet effective inter-context encoder, which aims to capture rich interactions between candidate images, as well as contextual alignment between a textual query to multiple images. We employ a two-layer Transformer that stacks on top of the intra-context encoder. The underlying design principle is general, and more advanced encoders can be used here for inter-context alignment. Thus, we can effectively tackle the second challenging issue of IRCD.
4 Experiments
4.1 Experimental Settings We conduct experiments on four datasets, including large-scale COCO (Lin et al., 2014) and Visual Genome (VG) (Krishna et al., 2017) for pretraining, IMAGECODE (Krojer et al., 2022) for zero-shot and fine-turning, and MMVP-VLM(Tong et al., 2024) for evaluating our fine-tuned ContextBLIP. We pre-train ContextBLIP on 4 × A100 GPU cards, and other experiments on a RTX3090 GPU card. During the pre-training stage, we configure adapter downsampling rate δ as 2, the mask ratio π as 0.25. We use vanilla BLIP-129M checkpoint as our backbone, which involves 223M parameters. We implement our model on the PyTorch platform.
We select eight strong baselines including CLIP (Radford et al., 2021), UNITER (Chen et al., 2020), ViBERT (Lu et al., 2019), OFA (Wang et al., 2022), ALBEF (Li et al., 2021), BLIP (Li et al., 2022), BLIP-2 (Li et al., 2023a), and NDCR (Li et al., 2023b). Note that NDCR is the previous state-of-the-art method on IMAGECODE. We follow the previous work (Krojer et al., 2022) to use accuracy as the evaluation metric. The IMAGECODE dataset involves three categorizations in the test set, including “Video” which indicates the candidate images are collected from video frames, “Im-
More hyperparameters are available in Appendix A.1
age” which represents the ones that are constructed based on static images, and “All” is the hybrid of the above two datasets.
4.2 Main Results
Zero-shot on IMAGECODE: We per-train the proposed ContextBLIP on COCO and VG. Table 1 reports the comparisons between ours and the baselines for zero-shot ContextBLIP on the IMAGECODE dataset. Equipped with our multi-scale adapter that only involves 2.4M parameters, our ContextBLIP achieves state-of-the-art performance on all test instances and video frames. Compared to the existing CLIP that uses global information in text and images for alignment, our ContextBLIP obtains 8.6% higher accuracy. Our method also outperforms existing UNITER and VILBERT by 10.2% and 10.7%, respectively. The two methods employ random masks for cross-modal alignment. We attribute the improvement to intra-context alignment based on multi-scale adapter and text-guided masking. The former learns both higher- and lowerlevel visual features, and their rich interactions at each level. The latter enables our ContextBLIP to concentrate on focal contextual cues. We also observe that existing BLIP-2 performs better than ours by 2.1 points on static images, and this is not a surprise as BLIP-2 is 50 larger than ContextBLIP. Fine-tuned on IMAGECODE: The right side of Table 1 reports the comparisons of our fine-tuned ContextBLIP to baselines. Our ContextBLIP can achieve state-of-the-art accuracy on test sets of “All” and “Image” and this further confirms the
Context Quantities Spatial Negation Occlusion Nuances Co-reference Meta Properties
CLIP (Zero-shot) 13.3 23.7 20.6 11.7 14.1 9.9 25.3 29.2 BLIP (Zero-shot) 15.4 30.9 27.7 11.4 16.7 11.4 28.9 12.5 Ours (Zero-shot) 17.3 39.2 36.9 19.0 19.1 11.0 37.4 25.0
CLIP (Fine-tuned) 19.2 30.9 30.5 17.3 18.6 14.8 32.5 33.3 NDCR (Fine-tuned) 21.9 30.9 31.9 25.1 23.3 18.9 30.1 37.5 Ours (Fine-tuned) 25.1 39.2 31.9 25.2 23.7 19.7 36.1 37.5
Table 2: Comparison of challenging samples in the IMAGECODE benchmark under zero-shot and fine-tuned settings. The samples involve challenging contextual alignment such as “Context”, “Nuances” and “Co-reference”.
Image Size Params
Questionnaire Progress: 6/300 Can you see the key "Z" in the image? Yes No
This question is not good Answers are too ambiguous
Back Next
Figure 8. Example of user study interfac . The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question.
74 76 78 80 82 84 86 IN-1k ZeroShot Performance
20
25
30
35
40
Av er
ag e
Pe rfo
rm an
ce o
n M
M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14
SigLIP ViT-SO-14
DFN ViT-H-14
DFN ViT-H-14
MetaCLIP ViT-L-14 MetaCLIP ViT-H-14
EVA01 ViT-g-14
EVA02 ViT-bigE-14+
Figure 9. Correlation between ImageNet-1k Zero-shot and
MMVP-VLM average. The area of each bubble corresponds to the model’s number of parameters. A higher ImageNet-1k zeroshot performance does not necessarily imply superior performance in MMVP-VLM.
in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentuates the value of MMVP-VLM as an evaluation metric, which probes into visual patterns such as orientation – aspects that are pivotal for downstream tasks and go beyond what is captured by ImageNet accuracy alone.
D. Visual Patterns for CLIP
Here, we provide the full description of visual patterns that pose
challenges to all CLIP-based models.
• ☼ Orientation and Direction: Questions about the direction something is facing or moving, such as the direction the dog or duck is facing, or the orientation of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the image. • L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet.
• Quantity and Count: Questions about the number of objects
or features present in the image.
• , Positional and Relational Context: This aspect refers to the model’s ability to understand the position and relationship of objects or elements within an image in relation to each other and their surroundings. • h Color and Appearance: Questions regarding the color of certain objects or elements. • Ô Structural and Physical Characteristics: This category involves the model’s ability to identify and analyze the physical attributes and structural features of objects in an image. • k Text: Questions related to text or symbols present in the image. • Viewpoint and Perspective: Questions concerning the per-
13
Image Size
Params (M)
IN-1k ZeroShot ☼ Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2242 427.6 75.5 13.3 13.3 20.0 20.0 13.3 53.3 20.0 6.7 13.3 19.3 OpenAI ViT-L-14 [43] 3362 427.9 76.6 0.0 20.0 40.0 20.0 6.7 20.0 33.3 6.7 33.3 20.0 SigLIP ViT-SO-14 [66] 2242 877.4 82.0 26.7 20.0 53.3 40.0 20.0 66.7 40.0 20.0 53.3 37.8 SigLIP ViT-SO-14 [66] 3842 878.0 83.1 20.0 26.7 60.0 33.3 13.3 66.7 33.3 26.7 53.3 37.0 DFN ViT-H-14 [10] 2242 986.1 83.4 20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53.3 39.3 DFN ViT-H-14 [10] 3782 986.7 84.4 13.3 20.0 53.3 33.3 26.7 66.7 40.0 20.0 40.0 34.8 MetaCLIP ViT-L-14 [62] 2242 427.6 79.2 13.3 6.7 66.7 6.7 33.3 46.7 20.0 6.7 13.3 23.7
MetaCLIP ViT-H-14 [62] 2242 986.1 80.6 6.7 13.3 60.0 13.3 6.7 53.3 26.7 13.3 33.3 25.2
EVA01 ViT-g-14 [54] 2242 1 36.4 78.5 6.7 26.7 40.0 6.7 13.3 66.7 13.3 13.3 20.0 23.0
E A02 ViT-bigE-14+ [54] 2242 5044.9 82.0 13.3 20.0 66.7 26.7 26.7 66.7 26.7 20.0 33.3 33.3
Table 1. Performance of various CLIP based models on different visual patterns in MMVP-VLM benchmark. Models scaled up in resolution show minimal improvement, whereas a slight advantage is observed when scaling up the network. For each
visual pattern, ImageNet-1k Zero-shot accuracy and MMVP average, we use light gray to highlight the best performance. For most of the visual patterns, all CLIP-based methods show struggle, as evident from the scores. We use symbols for visual patterns due to space limit: ☼: Orientation and Direction, Û: Presence of Specific Features, L: State and Condition, : Quantity and Count, ,: Positional and Relational Context, h: Color and Appearance, Ô: Structural and Physical Characteristics, k: Texts, : Viewpoint and Perspective.
0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1.5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance on visual patterns. If CLIP performs poorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual pattern.
tion between the underperformance of CLIP and MLLMs’ visual incapability? To explore this, we categorize questions from MMVP into these visual patterns summarized and calculate each MLLM’s performance on these patterns.
In Figure 6, we plot CLIP’s performance and MLLMs’ performance for each visual pattern. When the CLIP vision encoder underperforms on a certain visual pattern, the MLLM tends to exhibit similar shortcomings. Open-source models such as LLaVA 1.5 [30] and InstructBLIP [8] that explicitly use the CLIP vision encoder display a strong correlation in performance.
Further, we calculate the Pearson Correlation Coefficient between the CLIP model and MLLM’s performance on each visual pattern. Results show that LLaVA 1.5 and InstructBLIP all possess a coefficient score greater than 0.7. This high score indicates a strong correlation that weaknesses in visual pattern recognition in the CLIP model are transferred to MLLMs. More details on the Pearson Correlation Coefficient can be found in Appendix C.
4. Mixture-of-Features (MoF) for MLLM Based on our exploration in earlier sections, a natural question arises: If open-sourced MLLM’s visual shortcomings come from the CLIP vision encoder, how do we build a more competent visual encoder? In this section, we take initial steps to answer the question by studying Mixtureof-Features (MoF). We start with additive MoF that mixes
CLIP features and vision-only SSL model features. Results
show that each encoder presents unique advantages and limitations when employed as the pretrained model in MLLM (Section 4.2). We subsequently propose Interleaved MoF that integrates the features from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model’s ability to follow instructions (Section 4.3).
4.1. Experiment Setting
We adopt LLaVA [30, 31] as the framework to study visual encoders in MLLM. LLaVA uses a pretrained CLIP encoder and trains an adapter to align visual tokens with language tokens in the LLM. (See left side of Figure 7). We use DINOv2 [42] as the vision-only SSL model in our work because it is currently the most scalable vision-only model. Our exploration includes the use of two visual encoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ensure consistent and fair comparisons, we train and finetune our model with the same experiment setting in LLaVA. We include the additional experimental details in Appendix A.
4.2. Additive MoF
We add a pretrained DINOv2 encoder into MLLM and mix the CLIP pretrained encoder with it. We use a coefficient α to control the portion of CLIP features and 1 − α to control the amount of DINOv2 features and linearly add them
age ize Params (M) IN 1k ZeroShot ☼ Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2 427.6 75. 13.3 13.3 20.0 20.0 13.3 5 .3 20.0 6.7 13.3 19.3 OpenAI ViT-L-14 [43] 3362 427.9 76.6 0.0 20.0 40.0 20.0 6.7 20.0 33.3 6.7 33.3 20.0 SigLIP ViT-SO-14 [66] 2242 877.4 82.0 26.7 20.0 53.3 40.0 20.0 66.7 40.0 20.0 53.3 37.8 SigLIP ViT-SO-14 [66] 3842 878.0 83.1 20.0 26.7 60.0 33.3 13.3 66.7 33.3 26.7 53.3 37.0 DFN ViT-H-14 [10] 2242 986.1 83.4 20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53.3 39.3 DFN ViT-H-14 [10] 3782 986.7 84.4 13.3 20.0 53.3 33.3 26.7 66.7 40.0 20.0 40.0 34.8 MetaCLIP ViT-L-14 [62] 2242 427.6 79.2 13.3 6.7 66.7 6.7 33.3 46.7 20.0 6.7 13.3 23.7
MetaCLIP ViT-H-14 [62] 2242 986.1 80.6 6.7 13.3 60.0 13.3 6.7 53.3 26.7 13.3 33.3 25.2
EVA01 ViT-g-14 [54] 2242 1136.4 78.5 6.7 26.7 40.0 6.7 13.3 66.7 13.3 13.3 20.0 23.0
EVA02 ViT-bigE-14+ [54] 2242 5044.9 82.0 13.3 20.0 66.7 26.7 26.7 66.7 26.7 20.0 33.3 33.3
Table 1. Performance of various CLIP based models on different visual patterns in MMVP-VLM benchmark. Models scaled up in res lutio sh w minimal improvement, whereas a slight adva tage is observed when scaling up the network. For each
visual pattern, ImageNet-1k Zero-shot accuracy and MMVP average, we use light gray to highlight the best p rformance. F r m st of the visual patterns, all CLIP-based methods show struggle, as evident from the scores. We use symbols for visual patterns due to space limit: ☼: Orientation and Direction, Û: Presence of Specific Features, L: State and Condition, : Quantity and Count, ,: Positional and Relational Context, h: Color and Appearance, Ô: Structural and Physical Characteristics, k: Texts, : Viewpoint and Perspective.
0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1.5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance on visual patterns. If CLIP performs poorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual pattern.
tion between the r erfor a ce of CLIP nd MLLMs’ visual incapability? To explore this, we categorize questions from MMVP into these visual patterns summarized and calculate each MLLM’s performance on these patterns.
In Figure 6, we plot CLIP’s performance and MLLMs’ performance for each visual pattern. When the CLIP vision encoder underperforms on a certain visual pattern, the MLLM tends to exhibit similar sho tco ings. Open-source m dels such as LLaVA 1.5 [30] and InstructBLIP [8] that explicitly use the CLIP vision encoder display a strong correlation in performance.
Further, we calculate the Pearson Correlation Coefficient between the CLIP m del and MLLM’s perfo mance on each visual pattern. Results show that LLaVA 1.5 and InstructBLIP all possess a coefficient score greater than 0.7. This high score indicates a strong correlation that weaknesses in visual pattern recognition in the CLIP model are transferred to MLLMs. More details on the Pearson Correlation Coefficient can be found in Appendix C.
4. Mixture-of-Features (MoF) for MLLM Based on our exploration in ea lier sections, a natural question arises: If open-sourced MLLM’s visual shortcomings come from the CLIP vision encoder, how do we build a more competent visual encoder? In this section, we take initial steps to answer the question by studying Mixtureof-Features (MoF). We st rt with additive MoF that mixes CLIP featur s and vision-only SSL model features. Results show that each encoder presen s unique advantages and lim-
itations when employed as the pretrained el in MLLM
(Section 4.2). We subsequently propose Interleaved MoF that integrates the features from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model’s ability to follow instructions (Section 4.3).
4.1. Experiment Setting
We adopt LLaVA [30, 31] as the framework to study visual encoders in MLLM. LLaVA uses a pretrained CLIP encoder and trains an adapter to align visual tokens with language t kens in the LLM. (See left side of Figure 7). We use DINOv2 [42] as the vision-only SSL model in our work because it is currently the most scalable vision-only model. Our exploration i cludes the use of two visual encoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ensure consistent and fair comparisons, we train and finetune our model with the same experiment setting in LLaVA. We include the additional experimental details in Appendix A.
4.2. Additive MoF
We add a pretrained DINOv2 encoder into MLLM and mix the CLIP pretrained encoder with it. We use a coefficient α to control the portion of CLIP features and 1 − α to control the amount of DINOv2 features and linearly add them
6
Orientation and Direction a rabbit facing right a rabbit facing left
Presence of Specific Features tulips no tulips
State and Condition butterfly with wings open butterfly with wings closed
Quantity and Count 1 drink 2 drinks
Positional and Relational Context glasses on the right of the slipper glasses on the left of the slipper Color and Appearance light blue sky dark blue sky Structural Characteristics some fruits cut in half uncut fruits Texts “11:54” “11:59” Viewpoint and Perspective flowers seen from above flowers seen from the side Model chooses the correct image based on the text Model chooses the wrong image based on the text MMVP-VLM Benchmark Figure 5. Examples from MMVP-VLM. MMVP-VLM consists of image pairs across nine visual patterns. The examples in the figure are from EVA01 ViT-g-14 model [54], one of the largest CLIP models that also fails to choose the right image given the text description. emerged from CLIP-blind pairs (Section 3.1), (ii) whether these visual patterns pose challenges for CLIP-based models with massive scaling up (Section 3.2), and (iii) the correlation between failure patterns in CLIP models and those in MLLMs (Section 3.3). 3.1. Visual Patterns in CLIP-blind Pairs Having identified the CLIP-blind pairs, we summarize systematic visual patterns that the CLIP vision encoders might consistently misinterpret. It is too abstract to directly capture systematic visual patterns in the CLIP-blind pairs. Therefore, we turn to the questions and options from the MMVP benchmark. With these questions, we transform abstract visual patterns in images into clearer, language-based descriptors that are easier to categorize. In this work, we use GPT-4 [41] to categorize general patterns by prompting it with the following:
User I am analyzing an image embedding model. Can you go through the questions and options, trying to figure out some general patterns that the embedding model struggles with? Please focus on the visual features and generalize patterns that are important to vision models [MMVP Questions and Options]
We identify 9 visual patterns: ☼ Orientation and Direction
Û Presen e of Specific Fe tures
L State and Condition Quantity and Count , Positional and Relational Context h Color and Appearance Ô Structural and Physical Characteristics k Text Viewpoint and Pers ct ve
These visual patterns suggest that CLIP vision encoders
overly focus on high-level semantic understanding, overlooking intricate details of the visual world. Full descriptions of the visual patterns can be found in Appendix D. 3.2. The MMVP-VLM Benchmark CLIP-based models have developed rapidly since the introduction in the first paper [43]. We want to test whether these visual patterns still impose challenges to the more recent CLIP models [10, 54, 62, 66], which significantly scale up in terms of training data and model size. In doing so, we introduce a new benchmark: MMVP-VLM to systematically study if CLIP models handle this visual pattern well.
We distill a subset of questions from the MMVP benchmark into simpler language descriptions and categorize them into visual patterns. To maintain a balanced number of questions for each visual pattern, we add a few questions, if needed, to ensure that each visual pattern is represented by 15 text-image pairs. Examples of pairs are shown in Figure 5. A pair is deemed correctly answered if the model can accurately match both image-text combinations.
We evaluate MMVP-VLM on a variety of CLIP models [10, 43, 54, 62, 66]. These models vary in aspects like size, training data, and methodology. As evidenced in Table 1, increasing network size and training data only aids in identifying two visual patterns – “color and appearance” and “state and condition”. The rest of the visual patterns continue to challenge all CLIP-based models. We also find that the ImageNet-1k zero-shot accuracy is not a definitive indicator of a model’s performance regarding visual patterns. This underscores the necessity for additional evaluation metrics, such as MMVP-VLM, to accurately assess the model’s capabilities in areas beyond image classification.
3.3. How CLIP’s Errors Affect MLLMs
After analyzing the visual patterns that CLIP models struggle with, we pose the following question: Is there a correla-
5
Questionnaire Progress: 6/300 Can you see the key "Z" in the image? Yes No This question is not good Answers are too ambiguous Back Next Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question.
74 76 78 80 82 84 86
IN-1k ZeroShot Per ormance
20
25
30
35
40
Av er
ag e
Pe rfo
rm an
ce o
n M
M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14
SigLIP ViT-SO-14
DFN ViT-H-14
DFN ViT-H-14
MetaCLIP ViT-L-14 MetaCLIP ViT-H-14
EVA01 ViT-g-14
EVA02 ViT-bigE-14+
Figure 9. Correlation between I ageNet-1k Zero-shot and MMVP-VLM average. The area of each bubble corresponds to the model’s number of parameters. A higher ImageNet-1k zeroshot performance does not necessarily imply superior performance in MMVP VLM.
in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This
distinction accentuates the value of MMVP-VLM as an evaluation
metric, which probes into visual patterns such as orientation – as-
pects that are pivotal for downstream tasks and go beyond what is captured by ImageNet accuracy alone.
D. Visual Patterns for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models.
• ☼ Orientation and Direction: Questions about the direction something is facing or moving, such as the direction the dog or duck is facing, or the orientation of the school bus.
• Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the image.
• L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet.
• Quantity and Count: Questions about the number of objects
or fe ures prese t n the image.
• , Positional and Rela ional Context: This aspect refers to the model’s ability to understand the position and relationship of objects or elements within an image in relation to each other and their surroundings.
• h Color and Appearance: Questions regarding the color of certain objects or elements.
• Ô Structural and Physical Characteristics: This category involves the model’s ability to identify and analyze the physical
attributes and structural features of objects in an image.
• k Text: Questions related to text or symbols present in the
image.
• Vi wpoin and Perspectiv : Questions concerning the per-
13
Image Size Par ms (M) IN-1k Zer Shot ☼ Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2242 427.6 5.5 13.3 13. 20.0 20. 13.3 53. 20.0 6.7 13.3 9 OpenAI ViT-L-14 [43] 362 427.9 6.6 0.0 20.0 40. 20. 6.7 20.0 33.3 6.7 33.3 20 0 SigLIP ViT-SO-14 [66] 2242 877.4 82.0 26.7 20.0 53.3 40.0 20. 66.7 40.0 20. 53.3 37 8 SigLIP ViT-SO-14 [66] 3842 878.0 3.1 20.0 26.7 60.0 33.3 13. 66.7 33.3 26.7 53.3 37 0 DFN ViT-H-14 [10] 2242 986.1 83.4 20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53. 39 DFN ViT-H-14 [10] 3782 986.7 84.4 13.3 20.0 53.3 33. 26.7 66.7 40.0 20. 40. 34 8 MetaCLIP ViT-L-14 [62] 2242 4 7.6 9.2 13.3 6.7 66 7 6.7 33.3 46.7 20.0 6.7 13.3 2 7
MetaCLIP ViT-H-14 [62] 2242 986.1 80.6 6.7 13.3 60.0 13.3 6.7 53.3 26.7 13.3 33. 25 2
EVA01 ViT-g-14 [54] 2242 1136.4 78.5 6.7 26.7 40.0 6.7 13.3 66.7 13.3 13. 20.0 3
EVA02 ViT-bigE-14+ [54] 2242 5044.9 82.0 13.3 20.0 66.7 26.7 26.7 66.7 26.7 20.0 33.3
Table 1. Performance of various CLIP based models on differe t visual patterns in MMVP-VL benchmark. Models scaled up in r solut on show minimal i pr vement, wher as a slight advantage is observed wh n scaling up the network. For each
visual pattern, Imag Net-1k Zero-shot accuracy nd MMVP average, we use light gray to highlight the best performance. For most of the visual p tterns, all CLIP-based methods show str ggle, as evid nt from the scores. We use symbols for visual patt rns due to space limit: ☼: Or entation and Direction, Û: Presence of Spe ific Features, L: State and Condition, : Quantity and Count, ,: Positional and Relational Context, h: Color and Appe rance, Ô: Structural and Physical C ara teristics, k: Texts, : Viewpoint and Perspective.
0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1.5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance on visual patterns. If CLIP performs oorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual pattern.
tion between the underp rformance of CLIP and MLLMs’ visual incap bility? To explore this, we categorize questions from MMVP into these visual patterns summarized and calculate ea h MLLM’s performance on these patterns.
In Figure 6, we plot CLIP’s performance and MLLMs’ performance for each visu l pattern. Whe the CLIP vision encoder u derperforms on a certain visual pattern, the MLLM tends to exhibit similar shortcomings. Ope - ourc models such a LLaVA 1.5 [30] and Ins ructBLIP [8] that explicitly use the CLIP vision encoder display a strong correlation in performance.
Further, we calculate the Pearson Co relation Coefficient betw en the CLIP model and MLLM’s performance on each visu l patt n. Results show hat LLaVA 1.5 and InstructBLIP all possess a coefficient score greate than 0.7. This high score indicates a strong correlation that weaknesses in vi ual pattern recognition in the CLIP model are transferred to MLLMs. More details on the Pearson Cor elation Coefficient can be fou d in Appendix C.
4. Mixture-of-Features (MoF) for MLLM Based on our expl ration in earli r sections, a natural question arises: If open-sourc d MLLM’s visual hortcomings come from the CLIP vision encoder, how do we build a more co petent visual encoder? In this section, we take initial steps to answer the ques ion by studying Mixtureof-Features (MoF). We start with ddit ve MoF that mixes CLIP features and vision- nly SSL model featur s. Result show that each encoder presents unique dvantages and lim-
itations when employed as the pretrained model in MLLM
(Section 4.2). We subsequently propose Interleaved MoF that integr es he features from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model’s ability to fo low instructio (Section 4.3).
4.1. Experiment Setti g
We adopt LLaVA [30, 31] as the fr mework to study vis al encoders in MLLM. L aVA uses a pretrained CLIP encoder a d trains an dapter to align visual tokens with language tokens in th M. (See left sid of Figure 7). We use DINOv2 [42] as the vision-only SSL model in our work because it is currently the most scalable vision-on y model. Our exploration includes the use of two visual enco rs: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ens e con ist nt and fair compa is ns, we train and finetune our model with the same experim nt setting in LLaVA. We include the addition l experim ntal details in Appendix A.
4.2. Additive MoF
We add a pretrained DINOv2 encoder i to MLLM and mix th CLIP pretrained ncoder with it. We use a coefficient α to control the portion of CLIP features and 1 − α to control the amount of DINOv2 features and line rly add them
6
Questionnaire Progress: 6/300 Can you see the key "Z" in the image? Yes No This question is not good Answers are too ambiguous Back Next Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question.
74 76 78 80 82 84 86 IN-1k ZeroShot Performance
20
25
30 35 40
Av er
ag e
Pe rfo rm an ce o n M M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14 SigLIP ViT-SO-14 DFN ViT-H-14 DFN ViT-H-14
MetaCLIP ViT-L-14 MetaCLIP ViT-H-14
EVA01 ViT-g-14
EVA02 ViT-bigE-14+
Figure 9. Correlation b tween ImageNet-1k Zero-shot and MM P-VLM avera e. The area of each bubble corresponds to the model’s number of parameters. A higher ImageNet-1k zeroshot performance does not necessarily imply superior performance
in MMVP-VLM.
in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentuates the value of MMVP-VLM as an evaluation metric, which probes into visual patterns such as orientation – aspects that are pivotal for downstream tasks and go beyond what is captured by ImageNet accuracy alone.
D. Visual Patterns for LIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models. • ☼ Orientation and Direction: Questions about the direction something is facing or moving, such as the direction the dog or duck is facing, or the orientation of the school bus.
• Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the image. • L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet. • Quantity and Count: Questions about the number of objects or features present in the image. • , Positional and Relational Context: This aspect refers to the model’s ability to under tand the positio and relationship of bjects or el ments within an im ge in relation to each other
and their surroundings. • h Color nd Appearance: Questions regarding th color of
certain objects or elements. • Ô Structural and Physical Characteristics: This category
in olves the model’s ability to identify and an lyze the physical att ibutes and structur l featur s of objects in an image. • k Text: Que tions related to text or symbols present in the image. • Viewpoint and Perspective: Questions concerning the per-
13
Questionnaire Progress: 6/300 Can you see the key "Z" in the image? Yes No This question is not good Answ rs are too ambiguous Back Next Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question.
74 76 78 80 82 84 86 IN-1k ZeroShot Performance
20 25 30 35 40 Av er ag e Pe rfo rm an ce o n M M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14 SigLIP ViT-SO-14 DFN ViT-H-14 DFN ViT-H-14 MetaCLIP ViT-L-14 MetaCLIP ViT-H-14 EVA01 ViT-g-14 EVA02 ViT-bigE-14+
Figure 9. Correlation between ImageNet-1k Zero-shot and MMVP-VLM average. The area of each bubble corresponds to the model’s number of parameters. A higher ImageNet-1k zeroshot performance does not necessarily imply superior performance in MMVP-VLM.
in models with superior ImageNet-1k Zero-shot performance, this
trend does not necess rily hold for MMVP-VLM accuracy. This
distinction accentuates the value of MMVP-VLM as an evaluati n metric, which probes into visual patt ns such as orientation – spects that are pivotal for d wnst eam tasks and go beyond what is captured by ImageNet accuracy alone.
D. Visual Patterns for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models. • ☼ Orien ation and Direction: Questions about the direction something is facing or m ving, such as t direction the dog or duck is facing, or the orienta io of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the image.
• L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet.
• Quantity an Count: Questions about the number of objects or features present in the image.
• , Positional and Relational Context: This aspect refers to the m del’s ability to understand the position and relationship of objects or elements within an image in relation to each other and their surroundings.
• h olor a d Appearance: Questions r garding the color of certain objects or elem ts.
• Ô St uctural and Physical Characteristics: This category
inv lves the mod l’s ability to de tify and analyze the physical
attri u es a d structural features of objects in an image.
• k Text: Questions relat d to text or symbols pr sent in the m ge.
• Viewpoi t and Perspectiv : Questions concerning the per-
13
Image Size Params (M) IN-1k ZeroShot ☼ Û L , Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2242 427.6 75.5 13.3 13.3 20.0 20. . 53. 20.0 6.7 13.3 9. OpenAI ViT-L-14 [43] 3362 427.9 76.6 0.0 20.0 40.0 20. 6.7 20.0 33.3 6.7 33.3 20.0 SigLIP ViT-SO-14 [66] 2242 877.4 82.0 6.7 20.0 53.3 40.0 20. 66.7 40.0 20. 53.3 37.8 SigLIP ViT-SO-14 [66] 3842 878.0 83.1 2 .0 26.7 60.0 33.3 13. 66.7 3.3 26.7 53.3 37.0 DFN ViT-H-14 [10] 2242 986.1 83.4 20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53. 39. DFN ViT-H-14 [10] 3782 986.7 4.4 3.3 20.0 53.3 33. 26.7 66.7 40.0 20. 40. 34.8 MetaCLIP ViT- -14 [62] 2242 427.6 79.2 13.3 6.7 66.7 6.7 33.3 46.7 20.0 6.7 13.3 2 .7
MetaCLIP ViT-H-14 [62] 2242 986.1 80.6 6.7 13.3 60.0 13.3 6.7 53.3 26.7 13.3 3. 25.2
EVA01 ViT-g- 4 [54] 2242 1136.4 78.5 6.7 2 . 40.0 6.7 13.3 66.7 13.3 13. 20.0 3.
EVA02 ViT-bigE-14+ [54] 2242 50 .9 82.0 13.3 20.0 66.7 26.7 26.7 66.7 26.7 20.0 33.3 .
Table 1. Performance of va i us CLIP based mod ls on different visual patt ns in MMVP-VLM benchmark. Models scaled up in resolution show minimal improveme t, whereas a sligh dvant ge i observed when scaling up the network. For each
visual pattern, ImageNet-1k Zero shot accuracy and MMVP average, w use light gray to high the best performanc . For most f the visual patterns, al CLIP-bas d m thods show struggle, a evident from the scor s. We use symbol for visual patterns due to pac limit: ☼: Orientation a d Direction, Û: Presence of Specific F atures, L: State and Condition, : Quantity and Count, ,: Positional and Relational Context, h: Color and Appearance, Ô: Structu l and Physical Char cteristics, k: Texts, : Viewpoint and Persp ctiv .
0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1.5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance on visual patterns. If CLIP performs poorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual patter .
tion between the u derperformance of CLIP and MLLMs’ visual incap b lity? To ex lore this, we categorize questions from MMVP into these visual patterns summarized and calculate each MLLM’s p rformance on these pa terns.
In Figure 6, we plot CLIP’s performance and MLLMs’ performance for ach visual patt rn. When the CLIP vision encoder underperforms on a certain vi u l pattern, the MLLM tends to xhib t similar shortc mings. Open-s urce models such as LLaVA 1.5 [30] and Instr ctBLIP [8] that explicitly use the CLIP vision encoder display a trong correlation in perfo m nce.
Further, we calculate the P arson Correlation Coefficient between the CLIP model and MLLM’s perf rmance on each visual p tter . Re lts show that LLaVA 1.5 and InstructBLIP all possess a co fficient scor gr ater than 0.7. This high score indicates a strong co r lation th weaknesses i visual patt rn recog ition in the CLIP model are transferred to MLLMs. More details on he Pearson Co relation Coefficient can be fou d in Appendix C.
4. Mixture-of-F atures (MoF) for MLLM Based on our expl ration in earli r s ctions, a natural question arises: If open-sourced MLLM’ visual shortcomings come from the CLIP vision encoder, how do we build a more co petent visual encoder? In this sec ion, we take initial steps to a swer the question by studying Mixtureof-Features (MoF). We start with additive MoF th t mixes CLIP features and vi o -only SSL model featur s. Results show that each enc der present unique advantages d lim-
itations when employed as th pretrained model in ML M
(Sectio 4.2). We subsequently propo Inter eaved MoF that integrates the f atur s from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model’s ability to follow instruc ions (Section 4.3).
4.1. Experiment Setting
We adopt LLaVA [30, 31] as the framewo k to study visual ncoders in MLLM. L aVA uses a pretrained CLIP encoder and trains an d pter to align visual tokens with langu g tokens in the LLM. (See left side o Figure 7). We use DINOv2 [42] as the vision-only SSL model i our work because it is currently th mos scal ble vision-only m de . Our exploration includes the use of two visual encoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To en sur consist nt and fair compa is ns, we train and fi etune our model with the same experim nt setting in LLaVA. We include the a di ion l xperim ntal details in Appen ix A.
4.2. Additive MoF
We add a pretrained DINOv2 encoder i t MLLM and mix the CLIP pretrained encode with it. We us a coefficient α t cont ol the p rtion of CLIP eatures and 1 − α to control the amount of DINOv2 features and line rly add them
6
MMVP Averag
DFN ViT-H-14 (Fang et al., 2023) 224 2 986.1M 20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53.3 39.3
MetaCLIP ViT-H-14 (Xu et al., 2023) 224 2 986.1M 6.7 13.3 60.0 13.3 6.7 53.3 26.7 13.3 33.3 25.2 EVA02 ViT-bigE-14+ (Sun et al., 2023) 224 2 5044.9M 13.3 20.0 66.7 13. 26.7 6 . 26.7 20.0 33.3 3 .3
BLIP (Li et al., 2022) 2242 223M 13.3 6.7 40.0 20.0 26.7 66.7 46.7 20.0 46.7 31.9 NDCR (Li et al., 2023b) 2242 440M 6.7 40.0 60.0 26.7 13.3 8 33.3 2 .0 26 7 37.0 ContextBLIP (Ours) 2242 240M 26.7 46.7 60.0 40.0 46.7 93.3 40.0 20.0 53.3 47.4
Table 3: Comparisons on challenging visual patterns on the MMVP-VLM benchmark. We foll w the previous work (Tong et al., 2024) to use the symbols, such as
Questionnaire Progress: 6/300
Can you see the key "Z" in the image?
Yes No
This question is not good Answers are too ambiguous
Back Next
Figure 8. Example of user study interface. The questions in the us r stu y are randomly shuffled to void any otenti l bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question.
74 76 78 80 82 84 86 IN-1k ZeroShot Performance
20
25
30
35
40
Av er
ag e
Pe rfo
rm an
ce o
n M
M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14
SigLIP ViT-SO-14
DFN ViT-H-14
DFN ViT-H-14
MetaCLIP ViT-L-14 MetaCLIP ViT-H-14
EVA01 ViT-g-14
EVA02 ViT-bigE-14+
Figure 9. orrelation between ImageNet-1k Zero-shot and MMVP-VLM average. The area of each bubble corresponds to the model’s number of parameters. A higher ImageNet-1k zeroshot performance does not necessarily imply superior performance in MMVP-VLM.
in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentuates the value of MMVP-VLM as an valuation metric, which probes into visual patterns such as orientation – aspects that are pivotal for downstream tasks and go beyond what is captured by ImageNet accuracy alone.
D. Visual Patterns for CLIP Here, we provide the full description of visual patterns that pose challe ges to all CLIP-based models.
• ☼ Orientation and Direc ion: Ques ns about th direction
something is facing or moving, such as the direction the dog or duck is facing, or the orientation of the school bus. • Û Presence of Specific Features: Questions that focus on the xistence or non-existence of certain elem nts or fe tures in th
image. • L State and Condition: Questions t at pertain to the state or
condition of an object, such as whether a flag is blowing in the wind or if the round is wet. • Quantity and Count: Questions about the umber of objects r features present in the image. • , Positional and Relational Context: This aspect refers to the model’s ability to understand the position and relationship of objects or elements within an image in relation to each other and their surroundings. • h Color and Appearance: Questions regarding the olor of certain objects or elements. • Ô Structural and Physical Characteristics: This category involves the model’s ability to identify and analyze the physical attributes and structural f atures f objects in an image. • k Text: Questions related to text o symbols present in the image. • Viewpoint and Perspective: Questions concerning the per-
13
and Image Size Params (M) IN-1k ZeroShot Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2242 427.6 75.5 13.3 13.3 20.0 20.0 13.3 53.3 20.0 6.7 13.3 19.3 OpenAI ViT-L-14 [43] 3362 427.9 76.6 0.0 20.0 40.0 20.0 6.7 20.0 33.3 6.7 33.3 20.0 SigLIP ViT-SO-14 [66] 2242 877.4 82.0 26.7 20.0 53.3 40.0 20.0 66.7 40.0 20.0 53.3 37.8 SigLIP ViT-SO-14 [66] 3842 878.0 83.1 20.0 26.7 60.0 33.3 13.3 66.7 33.3 26.7 53.3 37.0 DFN iT-H-14 [10] 2242 986.1 83.4 20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53.3 39.3 DFN ViT-H-14 [10] 3782 986.7 84.4 13.3 20.0 53.3 33.3 26.7 66.7 40.0 20.0 40.0 34.8 MetaCLIP ViT-L- 4 [62] 2242 427.6 79.2 13.3 6.7 66.7 6.7 33.3 46.7 20.0 6.7 13.3 23.7 MetaCLIP ViT-H-14 [62] 2242 986.1 80.6 6.7 13.3 60.0 13.3 6.7 53.3 26.7 13.3 33.3 25.2 EVA01 ViT-g-14 [54] 2242 1136.4 78.5 6.7 26.7 40.0 6.7 13.3 66.7 13.3 13.3 20.0 23.0 E A02 ViT-bigE-14+ [54] 2242 5044.9 82.0 13.3 20.0 66.7 26.7 26.7 66.7 26.7 20.0 33.3 33.3
Table 1. Performance of various CLIP based models on different visual patterns in MMVP-VLM benchmark. Models scaled up in resolution show minimal improvement, whereas a slight advantage is observed when scaling up the network. For each
visual pattern, ImageNet-1k Zero-shot accuracy and MMVP average, we use light gray t highlight the best perform nce. For most of the visual patterns, all CLIP-based methods show struggle, as evident from the scores. We use symbols for visual patterns due to space limit: ☼: Orientation and Direction, Û: Presence of Specific Features, L: State and Condition, : Quantity and Count, ,: Positio l and Relational Context, h: Color and Appearance, Ô: Structural and Physical Characteristics, k: Texts, : Viewpoint and Perspective.
0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1.5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance on visual patterns. If CLIP performs poorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual pattern.
tion between the u derperformance of C IP and MLLMs’ visual incap bility? To explore this, we categorize questions from MMVP into these visual patterns summarized and calculate each MLLM’s performance on these patterns.
In Figure 6, we plot C ’s performance and MLLMs’ performance for each visual pattern. When the CLIP vision encoder underperforms on a certain visual pattern, the MLLM tends to exhibit similar shortcomings. Open-source models such as LL VA 1.5 [30] and I structBLIP [8] t at explicitly use the CLIP vision encoder display a strong correlation in performance.
Further, w calculate the Pearson Correlation Coefficient between the CLIP model and MLLM’s performance on each visual pattern. Results show that LLaVA 1.5 and InstructBLIP all possess a coefficient score greater than 0.7. This high score indicates a strong correlation that weaknesses in visual pattern recognition in the CLIP model are transferred to MLLMs. More details on the Pearson Correl tion Coefficient can be found in Appendix C.
4. Mixture-of-Fe tures ( o ) for MLLM Based on our exploration in earlier sections, a natural question ari s: If open-sourced MLLM’s visual shortcomings
com fro the CLIP vision ncoder, how do we build a
more competent visual encoder? In this section, we take initial steps to answer the question by studying Mixtureof-Features (MoF). We start with additive MoF that mixes CLIP features and vision-only SSL model features. Results show that each encoder presents unique advantages and limitations when employed as the pretrained model in MLLM (Section 4.2). We subsequently propose Interleaved MoF that integrates the features from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model’s ability to follow instructions (Section 4.3).
4.1. Experiment Setting
We adopt LLaVA [30, 31] as the fra ework to study visual encod s MLLM. LL VA us s a pretrained CLIP encoder and trains an adapter to align isual tokens with language tokens in the LLM. (See left side of Figure 7). We use DINOv2 [42] as the vision-only SSL model in our work because it is currently the most scalable vision-only model. Our exploration incl es the use of two visual encoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ensure consistent and fair comparisons, we train and finetune our model with the same experiment setting in LLaVA. We include the additional experimental details in Appendix A.
4.2. Addit ve MoF
We a d a pretrained DINOv2 encoder into MLLM and mix the CLIP pretrained encoder with it. We use a coefficient α to control the portion of CLIP features and 1 − α to control the amount of DINOv2 features and linearly add them
6
, to indicate 9 challenging visual patterns. More
details can be found in Appendix B. The best and second-best results are marked in bold and underline.
superiority of our method. ContextBLIP outperforms exiting CLIP and OFA by 18.7 and 26.4 points on static images, showing the effectiveness of both intra-context and inter-context alignment. We also observe that the proposed ContextBLIP performs slightly worse than the previous stateof-the-art NDCR by 1.7 points on video frames. However, NDCR pays m re attent on to the c ossmodal alignment of video frames and complex text, while the ability of static images is largely underexplored. We observe that ours is better than NDCR by 14.2 points on “Image”, with nearly half fewer parameters. We also attribute this to the model’s capability to handle distributio s of the candidat image, where static ones present a large variance. Comaprison of challenging samples of IMAGECODE: We compare our method with the existing CLIP and BLIP on the 200 challenging samples highlighted in IMAGECODE. These samples are manually labeled with high-quality annotations. Table 2 shows that our ContextBLIP con istently performs best under the fine-tuned setting on various scenarios, and performs best on most scenarios under the zero-shot setting. These results further confirm the superiority of our ContextBLIP i tackling the two challenges of the IRCD task. Comparisons on MMVP-VLM Benchmark: We compare our ContextBLIP with a very recent
benchmark MMVP-VLM (Tong et l., 2024), which aims to valuate how well a VLM model handles various challenging visual patterns. These manually defin d p tterns such as specific features, and positional and relational context, require a model to capture contextual details of visual cues or perform cross-modal reasoning. Table 3 shows tha our ContextBLIP, which is fine-tuned on the IMAGECODE dataset, achieves the best ccuracy on most of the patterns. Further, the proposed ContextBLIP involves much fewer parameters, e.g, the number of parameters in MetaCLIP ViT-H-14 (Xu et al., 2023) is nearly 25 times more than ContextBLIP, while average accuracy is lower th n ours by 22.2 points. Zero-shot Fine-tuned w/o Multi-scale Adapter 15.9 22.7 w/o Ltmim 18.2 23.0 w/o Inter-context Encoder - 23.4 ContextBLIP (Ours) 18.8 24.4
Table 4: Ablation study on the IMAGECODE dataset.
4.3 Ablation Study We conduct an abl tion study IMAGECODE to measure the contribution of each component to IRCD. Table 4 reports the results. Under both zero-
16245
shot and fine-tuned settings, we observe that the removal of the multi-scale adapter leads to a significant performance decrease, i.e., 2.9 points and 1.7 points, indicating the effectiveness of the adapter. Under the fine-tuned setting, it shows that the removal of the inter-context encoder leads to a 1 point performance drop, suggesting the effectiveness of long-range dependencies for the retrieval.
4.4 Sensitivity Analysis
We conduct experiments for sensitivity analysis on IMAGECODE under the zero-shot setting. Masking ratio: To evaluate how mask ratio π affect the retrieval, we configure π as 0.25, 0.50 and 0.75, respectively. Table 5 shows lower masking can increase the accuracy, e.g., from 75.4% to 79.5%. This interesting finding on the IRCD task does not align with previous studies that advocated for higher masking ratios (He et al., 2022; Geng et al., 2022; Bao et al., 2022; Kwon et al., 2022). One possible underlying reason is the challenging alignment in ICRD require more dense visual cues among similar candidates.
Position of adapter inserted: We analyze the impact of adapters inserted into different layers on the retrieval. We include three cases, i.e., inserting both down- and up-projection layers in the top layer of BLIP, inserting down-projection layers in 3-th, 6-th, 9-th, 12-th layers, and inserting down-projection layers in each layer. Table 6 shows inserting adapters at multiple layers of BCLIP achieved the highest overall accuracy. This suggests there exists a tradeoff for the number of layers to be inserted. Downsampling rate δ: We evaluate how δ affects the retrieval. We configure δ as 1, 2, 4, and 8, respectively. Table 7 shows that our ContextBLIP
achieves the best performance when δ is set as 2. The results present a large variance, e.g., 77.0 for δ = 8 and 79.5 for δ = 2. This suggests that a small rate δ leads to better performance.
4.5 Case Study
Figure 3 two cases to visually show why our ContextBLIP performs better for the challenging IRCD task. Both the zero-shot case and fine-tuned case demonstrate that our ContextBLIP not only yields the highest matching score for the golden candidate image. More importantly, it is also capable of aligning the key context cues in two modalities. For example, our method can understand textual cues “two girls in the frame” in the long query,
and yield more accurate alignment to the 7-th, 8-th, and 10-th candidate images with higher matching scores. Equipped with the proposed inter-context encoder, our ContextBLIP can accurately identify the 1-st candidate with the highest score.
4.6 Comparisons with GPT-4V
We compare our ContextBLIP with the OpenAI multi-modal large language model (MLLM) GPT4V (Yang et al., 2023). We randomly sample 50 instances three times from the test set of IMAGECODE, and use a prompt in the form of “Which image fits the text description best, please output the serial number of the image:< text_query >" to identify the image. Table 8 shows that our model can achieve comparable results to GPT-4V, despite involving about 7,500 times fewer parameters. This also aligns with some existing studies (Tong et al., 2024) that GPT-4V may fail to understand subtle cues. Figure 4 demonstrates how we prompt GPT-4V for the challenging IRCD task. Table 9 reports the matching scores of three instances selected from the test set of IMAGECODE. For the challenging “ID 8129”, we observe that GTP-4V performs poorly.
4.7 More discussion We also compare our ContextBLIP with the previous state-of-the-art NDCR (Li et al., 2023b) model
More cases are provided in Appendix D.2
on the IRCD task. We follow the existing NDCR to divide the linguistic complex descriptions into multiple segments with different lengths. We are interested in such a setting and evaluate how well the proposed ContextBLIP performs over sentences that are split from the same long description. Figure 10 illustrates that the proposed ContextBLIP consistently outperforms the existing NDCR under various sentence lengths.
5 Conclusion
This paper presents ContextBLIP, a simple yet effective doubly contextual alignment scheme for the challenging IRCD. Our model comprises a multiscale adapter, a matching loss, and a text-guided masking loss. The adapter learns to capture finegrained visual cues. The two losses enable iterative supervision for the adapter, gradually highlighting the focal patches of a single image to the key textual cues. Then, ContextBLIP further employs an inter-context encoder to learn dependencies among candidates, facilitating accurate alignment between text to multiple images. Consequently, the nuanced cues concealed in textual and visual modalities can be effectively aligned. Experiments on two benchmarks show the effectiveness of our method. We observe that our ContextBLIP can yield comparable results with GPT-4V, despite involving about 7,500 times fewer parameters. In the future, we plan to extend our method to text-to-video retrieval.
6 Limitation
The adaptive mask ratio is worth considering in the future, as a fixed masking ratio in our paper may not dynamically adapt to different cross-modal interactions. The proposed method may also have limitations for fine-grained retrieval for long videos, as pre-training on long videos is time-expensive and requires very large GPUs.
Acknowledgement
We thank Benno Krojer for providing the benchmarks and assisting in analyzing the leaderboard scores. This work was supported by the National Natural Science Foundation of China (No.62372125).
References Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang,
Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile visionlanguage model for understanding, localization, text reading, and beyond.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2021. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254.
Hangbo Bao, Wenhui Wang, Li Dong, and Furu Wei. 2022. Vl-beit: Generative vision-language pretraining. arXiv preprint arXiv:2206.01127.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In ECCV, pages 104–120.
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. 2022. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.08534.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. 2023. Data filtering networks.
Jerry Fodor. 2001. Language, thought and compositionality. Royal Institute of Philosophy Supplement, 48:227–242.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Jiao Qiao. 2021. Clip-adapter: Better vision-language models with feature adapters. ArXiv, abs/2110.04544.
Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel. 2022. Multimodal masked autoencoders learn transferable representations. arXiv preprint arXiv:2205.14204.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked autoencoders are scalable vision learners. In CVPR, pages 16000–16009.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In ICML, pages 2790–2799.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904–4916.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123:32–73.
Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy. 2022. Image retrieval from contextual descriptions. arXiv preprint arXiv:2203.15867.
Gukyeong Kwon, Zhaowei Cai, Avinash Ravichandran, Erhan Bas, Rahul Bhotika, and Stefano Soatto. 2022. Masked vision and language modeling for multi-modal representation learning. arXiv preprint arXiv:2208.02131.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. IEEE, 86(11):2278–2324.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping language-image pretraining for unified vision-language understanding and generation. In ICML, pages 12888–12900.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS, volume 34, pages 9694–9705.
Yunxin Li, Baotian Hu, Yunxin Ding, Lin Ma, and Min Zhang. 2023b. A neural divide-and-conquer reasoning framework for image retrieval from linguistically complex text. arXiv preprint arXiv:2305.02265.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2023. Video-llava: Learning united visual representation by alignment before projection.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV, pages 740– 755.
Haoyu Lu, Mingyu Ding, Yuqi Huo, Guoxing Yang, Zhiwu Lu, Masayoshi Tomizuka, and Wei Zhan. 2023. Uniadapter: Unified parameter-efficient transfer learning for cross-modal modeling. arXiv preprint arXiv:2302.06605.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS, volume 32.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer imageto-sentence models. In ICCV.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In ICML, volume 139, pages 8748– 8763.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint arXiv:2401.06209.
Haoqing Wang, Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhi-Hong Deng, and Kai Han. 2023. Masked image modeling with local multi-scale reconstruction. In CVPR, pages 2122–2131.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In ICML, volume 162, pages 23318–23340.
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. 2023. Demystifying clip data. arXiv preprint arXiv:2309.16671.
Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 9(1):1.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. arXiv preprint arXiv:2303.15343.
Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. 2021. Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling.
A Implementation details
A.1 Training Hyper-parameters Pre-training. We conducted pre-training on 4 Nvidia A100 GPUs for about 10 hours, randomly sampling 2242 patches from images and using RandAugment for data augmentation.
Fine-tuning. Compared to Krojer et al.’s multistep approach involving +Context Batch, +Context Module, and +Temporal Embeddings, we streamline Krojer et al.’s procedure by directly fine-tuning our model on IMAGECODE and then separately training a contextual modeling module between
the backbone and the prediction head to enable contextual comparison and reasoning.
During the first stage, we conduct full fine-tuning for 20 epochs. After that, we add a two-layer transformer as a context/temporal module, which is identical to Krojer et al.. To optimize the module while keeping other parameters fixed, we freeze all parts of the model except the inserted module and train it alone for 5 epochs.
We set the batch size to 360 images (36 batches of 10-image-set) and use the Adam (Kingma and Ba, 2014) optimizer with learning rate of 2e-6 and weight decay of 0.01. We conduct the experiment on one NVIDIA Geforce 3090 for about one day. All other baselines also adopted the same settings.
A.2 Dataset Details
Table 13 provides statistics for the datasets used in pre-training and fine-tuning. It includes three datasets: COCO, VG (Visual Genome), and IMAGECODE. The number of images (#image) and texts (#text) for each dataset are listed.
A.3 Model Details
In section 3.2, the scorer is a 1x768 linear layer. In section 3.3 (Step 2), the decoder is a four-layer transformer, with each layer having twelve attention heads and a feature dimension of 768.
In section 4.2, for CLIP, ViLBERT, and UNITER, we utilized the models with added context modules and temporal embeddings as proposed by Krojer et al. Our-Context model incorporated
a context module in the same manner, while other models did not have such a module added. For ALBEF, due to its pretraining on images with a resolution of 256x256, we conducted experiments using a slightly higher resolution (224x224) compared to other models.
B Validation Performance
Comparison with state-of-the-art methods on the valid set of IMAGECODE is shown on Table 15.
C Additional Case Studies
For tasks involving high image similarity and detailed textual descriptions, such as contrastive image retrieval, the challenge is considerable. We will demonstrate the superior performance of our MIM adaption method in more scenarios and provide the following examples to further illustrate the doubly contextual alignment capabilities of our model.
Figure 5 presents examples of two zero-shot experiments. In the left image, when two men are facing each other, our model assigns significantly higher matching scores compared to other images. In the right image, despite the difficulty in estimating the proportion of the man’s right eye visible, our model assigns obviously higher scores to the first two images showing the right eye. In contrast, BLIP assigns similar scores to all ten images in both samples, indicating its difficulty in attending to the textual cues used to align the intra-contexual information.
Figure 6 showcases examples of fine-tuning experiments. The left image in Figure 6 illustrates an example of the Quantity phenomenon, with the subheading "Two thumbs on the egg." We observe that our model infers significantly higher confidence scores for images 2 and 3 (matching the description of thumb quantity) and infers higher BLIP confidence scores for images 0, 2, and 3 (where the description of thumb quantity does not match). Thus, we demonstrate that our model indeed performs better in dual-contexual alignment.
The right image in Figure 6 demonstrates an example of the Meta-property phenomenon, with the subheading "The man’s face is blurry." We find that our model infers higher confidence scores for images 2, 3, 7, and 8 (all relatively matching the description) while BLIP infers higher confidence scores for images 1, 2, 3, and 8 (where image 1 is clear but does not match the description). Hence, it indicates that our model indeed performs better in
understanding intra-contexual alignment. In conjunction with the main text, we demonstrate the advantages of our model in two challenges: better intra-contexual alignment, better inter-contexual alignment. It can be seen that in many scenarios, our model exhibits better dualcontextual alignment capabilities compared to BLIP.
D More about GPT-4V Experiment
D.1 Sampling Details
Due to GPT-4V’s high computational demands, we created distinct test datasets from IMAGECODE’s static and video-shot sections. We randomly chose 50 samples from each, repeating thrice to avoid bias. Each sample included a ground-truth image plus nine adjacent ones, since GPT-4V processes up to ten images simultaneously. We prompted
GPT-4V in form of "Which image fits the text description best? Please output the serial number of the image:< text_query >" to select the matching image. We used three random seeds: 1, 10, 100, to explore the datasets with the text length distribution (in tokens) illustrated in the Figure 7.
In this experiment, using Python scripts with the API interface of the gpt-4-vision-preview model, we make a request to GPT-4V. If encounter a refusal to answer, we use the browser version to ask again until a result is obtained.
D.2 More Comparison with GPT-4V
This part provides some more examples of GPT4V prediction errors, where our model predicts correctly. The first two samples are from video shot set, and the last two are from static pictures.
Image Size Params
Questionnaire Progress: 6/300 Can you see the key "Z" in the image? Yes No This question is not good Answers are too ambiguous Back Next Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question.
74 76 78 80 82 84 86 IN-1k ZeroShot Performance
20
25
30
35
40
Av er
ag e
Pe rfo
rm an
ce o
n M
M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14
SigLIP ViT-SO-14
DFN ViT-H-14
DFN ViT-H-14
MetaCLIP ViT-L-14 MetaCLIP ViT-H-14
EVA01 ViT-g-14
EVA02 ViT-bigE-14+
Figure 9. Correlation between ImageNet-1k Zero-shot and MMVP-VLM average. The area of each bubble corresponds to the model’s number of parameters. A higher ImageNet-1k zeroshot performance does not necessarily imply superior performance in MMVP-VLM.
in models with superior ImageNet-1k Zero-shot performance, this
trend does not necessarily hold for MMVP-VLM accuracy. This
distinction accentuates the value of MMVP-VLM as an evaluation
metric, which probes into visual patterns such as orientation – aspects that are pivotal for downstream tasks and go beyond what is captured by ImageNet accuracy alone.
D. Visual Patterns for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models.
• ☼ Orientation and Direction: Questions about the direction something is facing or moving, such as the direction the dog or duck is facing, or the orientation of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the image. • L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet. • Quantity and Count: Questions about the number of objects or features present in the image. • , Positional and Relational Context: This aspect refers to the model’s ability to understand the position and relationship of objects or elements within an image in relation to each other and their surroundings. • h Color and Appearance: Questions regarding the color of certain objects or elements. • Ô Structural and Physical Characteristics: This category
involves the model’s ability to identify and analyze the physical
attributes and structural features of objects in an image.
• k Text: Questions related to text or symbols present in the image. • Viewpoint and Perspective: Questions concerning the per-
13
Image Size
Params (M)
IN-1k ZeroShot ☼ Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2242 427.6 75.5 13.3 13.3 20.0 20.0 13.3 53.3 20.0 6.7 13.3 19.3 OpenAI ViT-L-14 [43] 3362 427.9 76.6 0.0 20.0 40.0 20.0 6.7 20.0 33.3 6.7 33.3 20.0 SigLIP ViT-SO-14 [66] 2242 877.4 82.0 26.7 20.0 53.3 40.0 20.0 66.7 40.0 20.0 53.3 37.8 SigLIP ViT-SO-14 [66] 3842 878.0 83.1 20.0 26.7 60.0 33.3 13.3 66.7 33.3 26.7 53.3 37.0 DFN ViT-H-14 [10] 2242 986.1 83.4 20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53.3 39.3 DFN ViT-H-14 [10] 3782 986.7 84.4 13.3 20.0 53.3 33.3 26.7 66.7 40.0 20.0 40.0 34.8 MetaCLIP ViT-L-14 [62] 2242 427.6 79.2 13.3 6.7 66.7 6.7 33.3 46.7 20.0 6.7 13.3 23.7 MetaCLIP ViT-H-14 [62] 2242 986.1 80.6 6.7 13.3 60.0 13.3 6.7 53.3 26.7 13.3 33.3 25.2 EVA01 ViT-g-14 [54] 2242 1 36.4 78.5 6.7 26.7 40.0 6.7 13.3 66.7 13.3 13.3 20.0 23.0 E A02 ViT-bigE-14+ [54] 2242 5044.9 82.0 13.3 20.0 66.7 26.7 26.7 66.7 26.7 20.0 33.3 33.3
Table 1. Performance of various CLIP based models on different visual patterns in MMVP-VLM benchmark. Models scaled up in resolution show minimal improvement, whereas a slight advantage is observed when scaling up the network. For each
visual pattern, ImageNet-1k Zero-shot accuracy and MMVP average, we use light gray to highlight the best performance. For most of
the visual patterns, all CLIP-based methods show struggle, as evident from the scores. We use symbols for visual patterns due to space
limit: ☼: Orientation and Direction, Û: Presence of Specific Features, L: State and Condition, : Quantity and Count, ,: Positional and
Relational Context, h: Color and Appearance, Ô: Structural and Physical Characteristics, k: Texts, : Viewpoint and Perspective.
0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1.5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance on visual patterns. If CLIP performs poorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual pattern.
tion between the underperformance of CLIP and MLLMs’ visual incapability? To explore this, we categorize questions from MMVP into these visual patterns summarized and calculate each MLLM’s performance on these patterns.
In Figure 6, we plot CLIP’s performance and MLLMs’ performance for each visual pattern. When the CLIP vision encoder underperforms on a certain visual pattern, the MLLM tends to exhibit similar shortcomings. Open-source models such as LLaVA 1.5 [30] and InstructBLIP [8] that explicitly use the CLIP vision encoder display a strong correlation in performance.
Further, we calculate the Pearson Correlation Coefficient between the CLIP model and MLLM’s performance on each visual pattern. Results show that LLaVA 1.5 and InstructBLIP all possess a coefficient score greater than 0.7. This high score indicates a strong correlation that weaknesses in visual pattern recognition in the CLIP model are transferred to MLLMs. More details on the Pearson Correlation Coefficient can be found in Appendix C.
4. Mixture-of-Features (MoF) for MLLM Based on our exploration in earlier sections, a natural question arises: If open-sourced MLLM’s visual shortcomings come from the CLIP vision encoder, how do we build a
more competent visual encoder? In this section, we take
initial steps to answer the question by studying Mixture-
of-Features (MoF). We start with additive MoF that mixes
CLIP features and vision-only SSL model features. Results show that each encoder presents unique advantages and limitations when employed as the pretrained model in MLLM (Section 4.2). We subsequently propose Interleaved MoF that integrates the features from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model’s ability to follow instructions (Section 4.3).
4.1. Experiment Setting
We adopt LLaVA [30, 31] as the framework to study visual encoders in MLLM. LLaVA uses a pretrained CLIP encoder and trains an adapter to align visual tokens with language tokens in the LLM. (See left side of Figure 7). We use DINOv2 [42] as the vision-only SSL model in our work because it is currently the most scalable vision-only model. Our exploration includes the use of two visual encoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ensure consistent and fair comparisons, we train and finetune our model with the same experiment setting in LLaVA. We include the additional experimental details in Appendix A.
4.2. Additive MoF
We add a pretrained DINOv2 encoder into MLLM and mix the CLIP pretrained encoder with it. We use a coefficient α to control the portion of CLIP features and 1 − α to control the amount of DINOv2 features and linearly add them
6
age ize Params (M) IN 1k ZeroShot ☼ Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2 427.6 75. 13.3 13.3 20.0 20.0 13.3 5 .3 20.0 6.7 13.3 19.3 OpenAI ViT-L-14 [43] 3362 427.9 76.6 0.0 20.0 40.0 20.0 6.7 20.0 33.3 6.7 33.3 20.0 SigLIP ViT-SO-14 [66] 2242 877.4 82.0 26.7 20.0 53.3 40.0 20.0 66.7 40.0 20.0 53.3 37.8 SigLIP ViT-SO-14 [66] 3842 878.0 83.1 20.0 26.7 60.0 33.3 13.3 66.7 33.3 26.7 53.3 37.0 DFN ViT-H-14 [10] 2242 986.1 83.4 20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53.3 39.3 DFN ViT-H-14 [10] 3782 986.7 84.4 13.3 20.0 53.3 33.3 26.7 66.7 40.0 20.0 40.0 34.8 MetaCLIP ViT-L-14 [62] 2242 427.6 79.2 13.3 6.7 66.7 6.7 33.3 46.7 20.0 6.7 13.3 23.7 MetaCLIP ViT-H-14 [62] 2242 986.1 80.6 6.7 13.3 60.0 13.3 6.7 53.3 26.7 13.3 33.3 25.2 EVA01 ViT-g-14 [54] 2242 1136.4 78.5 6.7 26.7 40.0 6.7 13.3 66.7 13.3 13.3 20.0 23.0 EVA02 ViT-bigE-14+ [54] 2242 5044.9 82.0 13.3 20.0 66.7 26.7 26.7 66.7 26.7 20.0 33.3 33.3
Table 1. Performance of various CLIP based models on different visual patterns in MMVP-VLM benchmark. Models scaled up in res lutio sh w minimal improvement, whereas a slight adva tage is observed when scaling up the network. For each
visual pattern, ImageNet-1k Zero-shot accuracy and MMVP average, we use light gray to highlight the best p rformance. F r m st of the visual patterns, all CLIP-based methods show struggle, as evident from the scores. We use symbols for visual patterns due to space limit: ☼: Orientation and Direction, Û: Presence of Specific Features, L: State and Condition, : Quantity and Count, ,: Positional and
Relational Context, h: Color and Appearance, Ô: Structural and Physical Characteristics, k: Texts, : Viewpoint and Perspective.
0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1.5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance on visual patterns. If CLIP performs poorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual pattern.
tion between the r erfor a ce of CLIP nd MLLMs’ visual incapability? To explore this, we categorize questions from MMVP into these visual patterns summarized and calculate each MLLM’s performance on these patterns.
In Figure 6, we plot CLIP’s performance and MLLMs’ performance for each visual pattern. When the CLIP vision encoder underperforms on a certain visual pattern, the MLLM tends to exhibit similar sho tco ings. Open-source m dels such as LLaVA 1.5 [30] and InstructBLIP [8] that explicitly use the CLIP vision encoder display a strong correlation in performance.
Further, we calculate the Pearson Correlation Coefficient between the CLIP m del and MLLM’s perfo mance on each visual pattern. Results show that LLaVA 1.5 and InstructBLIP all possess a coefficient score greater than 0.7. This high score indicates a strong correlation that weaknesses in visual pattern recognition in the CLIP model are transferred to MLLMs. More details on the Pearson Correlation Coefficient can be found in Appendix C.
4. Mixture-of-Features (MoF) for MLLM Based on our exploration in ea lier sections, a natural question arises: If open-sourced MLLM’s visual shortcomings come from the CLIP vision encoder, how do we build a more competent visual encoder? In this section, we take initial steps to answer the question by studying Mixture-
of-Features (MoF). We st rt with additive MoF that mixes
CLIP featur s and vision-only SSL model features. Results
show that each encoder presen s unique advantages and lim-
itations when employed as the pretrained el in MLLM (Section 4.2). We subsequently propose Interleaved MoF that integrates the features from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model’s ability to follow instructions (Section 4.3).
4.1. Experiment Setting
We adopt LLaVA [30, 31] as the framework to study visual encoders in MLLM. LLaVA uses a pretrained CLIP encoder and trains an adapter to align visual tokens with language t kens in the LLM. (See left side of Figure 7). We use DINOv2 [42] as the vision-only SSL model in our work because it is currently the most scalable vision-only model. Our exploration i cludes the use of two visual encoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ensure consistent and fair comparisons, we train and finetune our model with the same experiment setting in LLaVA. We include the additional experimental details in Appendix A.
4.2. Additive MoF
We add a pretrained DINOv2 encoder into MLLM and mix the CLIP pretrained encoder with it. We use a coefficient α to control the portion of CLIP features and 1 − α to control the amount of DINOv2 features and linearly add them
6
Orientation and Direction a rabbit facing right a rabbit facing left
Presence of Specific Features tulips no tulips
State and Condition butterfly with wings open butterfly with wings closed
Quantity and Count 1 drink 2 drinks
Positional and Relational Context glasses on the right of the slipper glasses on the left of the slipper Color and Appearance light blue sky dark blue sky Structural Characteristics some fruits cut in half uncut fruits Texts “11:54” “11:59” Viewpoint and Perspective flowers seen from above flowers seen from the side Model chooses the correct image based on the text Model chooses the wrong image based on the text MMVP-VLM Benchmark Figure 5. Examples from MMVP-VLM. MMVP-VLM consists of image pairs across nine visual patterns. The examples in the figure are from EVA01 ViT-g-14 model [54], one of the largest CLIP models that also fails to choose the right image given the text description. emerged from CLIP-blind pairs (Section 3.1), (ii) whether these visual patterns pose challenges for CLIP-based models with massive scaling up (Section 3.2), and (iii) the correlation between failure patterns in CLIP models and those in MLLMs (Section 3.3). 3.1. Visual Patterns in CLIP-blind Pairs Having identified the CLIP-blind pairs, we summarize systematic visual patterns that the CLIP vision encoders might consistently misinterpret. It is too abstract to directly capture systematic visual patterns in the CLIP-blind pairs. Therefore, we turn to the questions and options from the MMVP benchmark. With these questions, we transform abstract visual patterns in images into clearer, language-based descriptors that are easier to categorize. In this work, we use GPT-4 [41] to categorize general patterns by prompting it with the following: User I am analyzing an image embedding model. Can you go through the questions and options, trying to figure out some general patterns that the embedding model struggles with? Please focus on the visual features and generalize patterns that are important to vision models [MMVP Questions and Options]
We identify 9 visual patterns: ☼ Orientation and Direction Û Presen e of Specific Fe tures L State and Condition Quantity and Count , Positional and Relational Context h Color and Appearance Ô Structural and Physical Characteristics k Text Viewpoint and Pers ct ve
These visual patterns suggest that CLIP vision encoders
overly focus on high-level semantic understanding, overlooking intricate details of the visual world. Full descriptions of the visual patterns can be found in Appendix D. 3.2. The MMVP-VLM Benchmark CLIP-based models have developed rapidly since the introduction in the first paper [43]. We want to test whether these visual patterns still impose challenges to the more recent CLIP models [10, 54, 62, 66], which significantly scale up in terms of training data and model size. In doing so, we introduce a new benchmark: MMVP-VLM to systematically study if CLIP models handle this visual pattern well. We distill a subset of questions from the MMVP benchmark into simpler language descriptions and categorize them into visual patterns. To maintain a balanced number of questions for each visual pattern, we add a few questions, if needed, to ensure that each visual pattern is represented by 15 text-image pairs. Examples of pairs are shown in Figure 5. A pair is deemed correctly answered if the model can accurately match both image-text combinations.
We evaluate MMVP-VLM on a variety of CLIP models [10, 43, 54, 62, 66]. These models vary in aspects like size, training data, and methodology. As evidenced in Table 1, increasing network size and training data only aids in identifying two visual patterns – “color and appearance” and “state and condition”. The rest of the visual patterns continue to challenge all CLIP-based models. We also find that the ImageNet-1k zero-shot accuracy is not a definitive indicator of a model’s performance regarding visual patterns. This underscores the necessity for additional evaluation metrics, such as MMVP-VLM, to accurately assess the model’s capabilities in areas beyond image classification.
3.3. How CLIP’s Errors Affect MLLMs
After analyzing the visual patterns that CLIP models struggle with, we pose the following question: Is there a correla-
5
Questionnaire Progress: 6/300 Can you see the key "Z" in the image? Yes No This question is not good Answers are too ambiguous Back Next Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question.
74 76 78 80 82 84 86 IN-1k ZeroShot Performance
20
25 30 35 40 Av er ag e Pe rfo rm an ce o n M M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14 SigLIP ViT-SO-14 DFN ViT-H-14 DFN ViT-H-14 MetaCLIP ViT-L-14 MetaCLIP ViT-H-14 EVA01 ViT-g-14 EVA02 ViT-bigE-14+
Figure 9. Correlation between I ageNet-1k Zero-shot and MMVP-VLM average. The area of each bubble corresponds to the model’s number of parameters. A higher ImageNet-1k zeroshot performance does not necessarily imply superior performance in MMVP VLM.
in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentuates the value of MMVP-VLM as an evaluation metric, which probes into visual patterns such as orientation – aspects that are pivotal for downstream tasks and go beyond what is captured by ImageNet accuracy alone.
D. Visual Patterns for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models. • ☼ Orientation and Direction: Questions about the direction something is facing or moving, such as the direction the dog or duck is facing, or the orientation of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the image. • L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet.
• Quantity and Count: Questions about the number of objects or fe ures prese t n the image.
• , Positional and Rela ional Context: This aspect refers to the model’s ability to understand the position and relationship of objects or elements within an image in relation to each other and their surroundings.
• h Color and Appearance: Questions regarding the color of certain objects or elements.
• Ô Structural and Physical Characteristics: This category involves the model’s ability to identify and analyze the physical attributes and structural features of objects in an image.
• k Text: Questions related to text or symbols present in the image.
• Vi wpoin and Perspectiv : Questions concerning the per-
13
Image Size Par ms (M) IN-1k Zer Shot ☼ Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2242 427.6 5.5 13.3 13. 20.0 20. 13.3 53. 20.0 6.7 13.3 9 OpenAI ViT-L-14 [43] 362 427.9 6.6 0.0 20.0 40. 20. 6.7 20.0 33.3 6.7 33.3 20 0 SigLIP ViT-SO-14 [66] 2242 877.4 82.0 26.7 20.0 53.3 40.0 20. 66.7 40.0 20. 53.3 37 8 SigLIP ViT-SO-14 [66] 3842 878.0 3.1 20.0 26.7 60.0 33.3 13. 66.7 33.3 26.7 53.3 37 0 DFN ViT-H-14 [10] 2242 986.1 83.4 20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53. 39 DFN ViT-H-14 [10] 3782 986.7 84.4 13.3 20.0 53.3 33. 26.7 66.7 40.0 20. 40. 34 8 MetaCLIP ViT-L-14 [62] 2242 4 7.6 9.2 13.3 6.7 66 7 6.7 33.3 46.7 20.0 6.7 13.3 2 7 MetaCLIP ViT-H-14 [62] 2242 986.1 80.6 6.7 13.3 60.0 13.3 6.7 53.3 26.7 13.3 33. 25 2 EVA01 ViT-g-14 [54] 2242 1136.4 78.5 6.7 26.7 40.0 6.7 13.3 66.7 13.3 13. 20.0 3 EVA02 ViT-bigE-14+ [54] 2242 5044.9 82.0 13.3 20.0 66.7 26.7 26.7 66.7 26.7 20.0 33.3
Table 1. Performance of various CLIP based models on differe t visual patterns in MMVP-VL benchmark. Models scaled up in r solut on show minimal i pr vement, wher as a slight advantage is observed wh n scaling up the network. For each
visual pattern, Imag Net-1k Zero-shot accuracy nd MMVP average, we use light gray to highlight the best performance. For most of the visual p tterns, all CLIP-based methods show str ggle, as evid nt from the scores. We use symbols for visual patt rns due to space limit: ☼: Or entation and Direction, Û: Presence of Spe ific Features, L: State and Condition, : Quantity and Count, ,: Positional and
Relational Context, h: Color and Appe rance, Ô: Structural and Physical C ara teristics, k: Texts, : Viewpoint and Perspective.
0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1.5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance on visual patterns. If CLIP performs oorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual pattern.
tion between the underp rformance of CLIP and MLLMs’ visual incap bility? To explore this, we categorize questions from MMVP into these visual patterns summarized and calculate ea h MLLM’s performance on these patterns.
In Figure 6, we plot CLIP’s performance and MLLMs’ performance for each visu l pattern. Whe the CLIP vision encoder u derperforms on a certain visual pattern, the MLLM tends to exhibit similar shortcomings. Ope - ourc models such a LLaVA 1.5 [30] and Ins ructBLIP [8] that explicitly use the CLIP vision encoder display a strong correlation in performance.
Further, we calculate the Pearson Co relation Coefficient betw en the CLIP model and MLLM’s performance on each visu l patt n. Results show hat LLaVA 1.5 and InstructBLIP all possess a coefficient score greate than 0.7. This high score indicates a strong correlation that weaknesses in vi ual pattern recognition in the CLIP model are transferred to MLLMs. More details on the Pearson Cor elation Coefficient can be fou d in Appendix C.
4. Mixture-of-Features (MoF) for MLLM Based on our expl ration in earli r sections, a natural question arises: If open-sourc d MLLM’s visual hortcomings come from the CLIP vision encoder, how do we build a more co petent visual encoder? In this section, we take initial steps to answer the ques ion by studying Mixture-
of-Features (MoF). We start with ddit ve MoF that mixes
CLIP features and vision- nly SSL model featur s. Result
show that each encoder presents unique dvantages and lim-
itations when employed as the pretrained model in MLLM (Section 4.2). We subsequently propose Interleaved MoF that integr es he features from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model’s ability to fo low instructio (Section 4.3).
4.1. Experiment Setti g
We adopt LLaVA [30, 31] as the fr mework to study vis al encoders in MLLM. L aVA uses a pretrained CLIP encoder a d trains an dapter to align visual tokens with language tokens in th M. (See left sid of Figure 7). We use DINOv2 [42] as the vision-only SSL model in our work because it is currently the most scalable vision-on y model. Our exploration includes the use of two visual enco rs: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ens e con ist nt and fair compa is ns, we train and finetune our model with the same experim nt setting in LLaVA. We include the addition l experim ntal details in Appendix A.
4.2. Additive MoF
We add a pretrained DINOv2 encoder i to MLLM and mix th CLIP pretrained ncoder with it. We use a coefficient α to control the portion of CLIP features and 1 − α to control the amount of DINOv2 features and line rly add them
6
Questionnaire Progress: 6/300 Can you see the key "Z" in the image? Yes No This question is not good Answers are too ambiguous Back Next Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question. 74 76 78 80 82 84 86 IN-1k ZeroShot Performance 20 25 30 35 40 Av er ag e Pe rfo rm an ce o n M M VP OpenAI ViT-L-14 OpenAI ViT-L-14 SigLIP ViT-SO-14 SigLIP ViT-SO-14 DFN ViT-H-14 DFN ViT-H-14 MetaCLIP ViT-L-14 MetaCLIP ViT-H-14 EVA01 ViT-g-14 EVA02 ViT-bigE-14+
Figure 9. Correlation b tween ImageNet-1k Zero-shot and MM P-VLM avera e. The area of each bubble corresponds to the model’s number of parameters. A higher ImageNet-1k zeroshot performance does not necessarily imply superior performance in MMVP-VLM.
in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentuates the value of MMVP-VLM as an evaluation metric, which probes into visual patterns such as orientation – aspects that are pivotal for downstream tasks and go beyond what is captured by ImageNet accuracy alone.
D. Visual Patterns for LIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models. • ☼ Orientation and Direction: Questions about the direction something is facing or moving, such as the direction the dog or duck is facing, or the orientation of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the image. • L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet. • Quantity and Count: Questions about the number of objects or features present in the image.
• , Positional and Relational Context: This aspect refers to the model’s ability to under tand the positio and relationship of bjects or el ments within an im ge in relation to each other
and their surroundings. • h Color and Appearance: Questions regarding th color of
certain objects or elements. • Ô Structural and Physical Characteristics: This category
in olves the model’s ability to identify and an lyze the physical att ibutes and structur l featur s of objects in an image. • k Text: Que tions related to text or symbols present in the image. • Viewpoint and Perspective: Questions concerning the per-
13
Questionnaire Progress: 6/300 Can you see the key "Z" in the image? Yes No This question is not good Answ rs are too ambiguous Back Next Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question. 74 76 78 80 82 84 86 IN-1k ZeroShot Performance 20 25 30 35 40 Av er ag e Pe rfo rm an ce o n M M VP OpenAI ViT-L-14 OpenAI ViT-L-14 SigLIP ViT-SO-14 SigLIP ViT-SO-14 DFN ViT-H-14 DFN ViT-H-14 MetaCLIP ViT-L-14 MetaCLIP ViT-H-14 EVA01 ViT-g-14 EVA02 ViT-bigE-14+ Figure 9. Correlation between ImageNet-1k Zero-shot and MMVP-VLM average. The area of each bubble corresponds to the model’s number of parameters. A higher ImageNet-1k zeroshot performance does not necessarily imply superior performance in MMVP-VLM.
in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentuates the value of MMVP-VLM as an evaluati n metric, which probes into visual patt ns such as orientation – spects that are pivotal for d wnst eam tasks and go beyond what is captured by ImageNet accuracy alone.
D. Visual Patterns for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models. • ☼ Orien ation and Direction: Questions about the direction something is facing or m ving, such as t direction the dog or duck is facing, or the orienta io of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the image. • L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet. • Quantity an Count: Questions about the number of objects or features present in the image. • , Positional and Relational Context: This aspect refers to the m del’s ability to understand the position and relationship of objects or elements within an image in relation to each other and their surroundings. • h olor a d Appearance: Questions r garding the color of certain objects or elem ts.
• Ô St uctural and Physical Characteristics: This category inv lves the mod l’s ability to de tify and analyze the physical attri u es a d structural features of objects in an image.
• k Text: Questions relat d to text or symbols pr sent in the m ge.
• Viewpoi t and Perspectiv : Questions concerning the per-
13
Image Size Params (M) IN-1k ZeroShot ☼ Û L , Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2242 427.6 75.5 13.3 13.3 20.0 20. . 53. 20.0 6.7 13.3 9. OpenAI ViT-L-14 [43] 3362 427.9 76.6 0.0 20.0 40.0 20. 6.7 20.0 33.3 6.7 33.3 20.0 SigLIP ViT-SO-14 [66] 2242 877.4 82.0 6.7 20.0 53.3 40.0 20. 66.7 40.0 20. 53.3 37.8 SigLIP ViT-SO-14 [66] 3842 878.0 83.1 2 .0 26.7 60.0 33.3 13. 66.7 3.3 26.7 53.3 37.0 DFN ViT-H-14 [10] 2242 986.1 83.4 20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53. 39. DFN ViT-H-14 [10] 3782 986.7 4.4 3.3 20.0 53.3 33. 26.7 66.7 40.0 20. 40. 34.8 MetaCLIP ViT- -14 [62] 2242 427.6 79.2 13.3 6.7 66.7 6.7 33.3 46.7 20.0 6.7 13.3 2 .7 MetaCLIP ViT-H-14 [62] 2242 986.1 80.6 6.7 13.3 60.0 13.3 6.7 53.3 26.7 13.3 3. 25.2 EVA01 ViT-g- 4 [54] 2242 1136.4 78.5 6.7 2 . 40.0 6.7 13.3 66.7 13.3 13. 20.0 3. EVA02 ViT-bigE-14+ [54] 2242 50 .9 82.0 13.3 20.0 66.7 26.7 26.7 66.7 26.7 20.0 33.3 .
Table 1. Performance of va i us CLIP based mod ls on different visual patt ns in MMVP-VLM benchmark. Models scaled up in resolution show minimal improveme t, whereas a sligh dvant ge i observed when scaling up the network. For each
visual pattern, ImageNet-1k Zero shot accuracy and MMVP average, w use light gray to high the best performanc . For most f the visual patterns, al CLIP-bas d m thods show struggle, a evident from the scor s. We use symbol for visual patterns due to pac limit: ☼: Orientation a d Direction, Û: Presence of Specific F atures, L: State and Condition, : Quantity and Count, ,: Positional and
Relational Context, h: Color and Appearance, Ô: Structu l and Physical Char cteristics, k: Texts, : Viewpoint and Persp ctiv .
0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1.5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance on visual patterns. If CLIP performs poorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual patter .
tion between the u derperformance of CLIP and MLLMs’ visual incap b lity? To ex lore this, we categorize questions from MMVP into these visual patterns summarized and calculate each MLLM’s p rformance on these pa terns.
In Figure 6, we plot CLIP’s performance and MLLMs’ performance for ach visual patt rn. When the CLIP vision encoder underperforms on a certain vi u l pattern, the MLLM tends to xhib t similar shortc mings. Open-s urce models such as LLaVA 1.5 [30] and Instr ctBLIP [8] that explicitly use the CLIP vision encoder display a trong correlation in perfo m nce.
Further, we calculate the P arson Correlation Coefficient between the CLIP model and MLLM’s perf rmance on each visual p tter . Re lts show that LLaVA 1.5 and InstructBLIP all possess a co fficient scor gr ater than 0.7. This high score indicates a strong co r lation th weaknesses i visual patt rn recog ition in the CLIP model are transferred to MLLMs. More details on he Pearson Co relation Coefficient can be fou d in Appendix C.
4. Mixture-of-F atures (MoF) for MLLM Based on our expl ration in earli r s ctions, a natural question arises: If open-sourced MLLM’ visual shortcomings come from the CLIP vision encoder, how do we build a more co petent visual encoder? In this sec ion, we take initial steps to a swer the question by studying Mixture-
of-Features (MoF). We start with additive MoF th t mixes
CLIP features and vi o -only SSL model featur s. Results
show that each enc der present unique advantages d lim-
itations when employed as th pretrained model in ML M (Sectio 4.2). We subsequently propo Inter eaved MoF that integrates the f atur s from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model’s ability to follow instruc ions (Section 4.3).
4.1. Experiment Setting
We adopt LLaVA [30, 31] as the framewo k to study visual ncoders in MLLM. L aVA uses a pretrained CLIP encoder and trains an d pter to align visual tokens with langu g tokens in the LLM. (See left side o Figure 7). We use DINOv2 [42] as the vision-only SSL model i our work because it is currently th mos scal ble vision-only m de . Our exploration includes the use of two visual encoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To en sur consist nt and fair compa is ns, we train and fi etune our model with the same experim nt setting in LLaVA. We include the a di ion l xperim ntal details in Appen ix A.
4.2. Additive MoF
We add a pretrained DINOv2 encoder i t MLLM and mix the CLIP pretrained encode with it. We us a coefficient α t cont ol the p rtion of CLIP eatures and 1 − α to control the amount of DINOv2 features and line rly add them
6
MMVP Averag
OpenAI ViT-L-14 (Radford et al., 2021) 224 2 427.6M 13.3 13.3 20.0 20.0 13.3 53.3 20.0 6.7 13.3 19.3
OpenAI ViT-L-14 (Radford et al., 2021) 336 2 427.9M 0.0 20.0 40.0 20.0 6.7 20.0 33.3 6.7 33.3 20.0
SigLIP ViT-SO-14 (Zhai et al., 2023) 224 2 8 7.4M 6.7 20.0 53.3 40.0 20.0 6 . 40.0 20.0 53.3 37.8 SigLIP ViT-SO-14 (Zhai et al., 2023) 384 2 878.0M 20.0 26.7 60.0 33.3 13.3 66.7 33.3 26.7 53.3 37.0
DFN ViT-H-14 (Fang et al., 2023) 224 2 986.1M 20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53.3 39.3 DFN ViT-H-14 3782 986.7M 13.3 20.0 53.3 33 3 26.7 66.7 40.0 20.0 40.0 34.8 MetaCLIP ViT-L-14
(Xu et al., 2023) 224 2 427. M 13.3 6.7 66.7 6.7 33.3 46.7 20.0 6.7 13.3 23.7
MetaCLIP ViT-H-14 (Xu et al., 2023) 224 2 986.1M 6.7 13.3 60 0 13.3 6. 53 3 26.7 13.3 3 . 25.2
EVA01 ViT-g-14 (Sun et l., 2023) 224 2 1136.4M 6.7 26.7 40.0 6.7 13.3 66.7 13.3 13.3 20.0 23.0
EVA02 ViT-bigE-14+ (Sun et l., 2023) 224 2 5044.9M 13.3 20.0 66.7 13.3 26.7 66.7 26.7 20.0 33.3 33.3
BLIP (Li et al., 2022) 2242 223M 13.3 6.7 40.0 20.0 26.7 66.7 46.7 20.0 46.7 31.9 NDCR (Li et al., 2023b) 2242 440M 26.7 40.0 60.0 26.7 13.3 86.7 33.3 20.0 26.7 37.0 ContextBLIP (Our ) 2242 240M 26.7 46.7 60.0 40.0 46.7 93.3 40.0 20.0 53.3 47.4
Table 16: Comparison with various VLMs on different visual patterns in MMVP-VLM benchmark. The best and second-best results are marked in bold and underline. We identify 9 visual patterns:
Questionnaire Progress: 6/300
Can you see the key "Z" in the image?
Yes No
This question is not good Answers are too ambiguous
Back Next
Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question.
74 76 78 80 82 84 86 IN-1k ZeroShot Performance
20
25
30
35
40
Av er
ag e
Pe rfo
rm an
ce o
n M
M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14
SigLIP ViT-SO-14
DFN ViT-H-14
DFN ViT-H-14
MetaCLIP ViT-L-14 MetaCLIP ViT-H-14
EVA01 ViT-g-14
EVA02 ViT-bigE-14+
Figure 9. Correlation between ImageNet-1k Zero-shot and MMVP-VLM average. The area of each bubble corresponds to
the model’s number of par meter . A high r ImageNet-1k z r -
shot performance does not necessarily imply superior performance
in MMVP-VLM.
in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentua es the value of MMVP-VLM as an evalua ion metric, which probes into visual patterns such as orientation – asects that are pivotal for downstream tasks and go beyond what is captured by ImageNet accuracy alone.
D. Visual Patterns for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models.
• ☼ Orientation and Direction: Questions about the direction
something is facing or moving, such as the direction the dog or duck is facing, or the orientation of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the image. • L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the win or if t e ground is wet. • Quantity and Count: Questions about t e number of objects or features present in the image. • , Positional and Relational Context: This aspect refers to the
model’s ability to understand the position a d relationship of
objects or elements within an image in relation to each other
and their surroundings.
• h Color nd Appear nce: Questions regarding the color of
certain objects or elements. • Ô Structural and Physical Characteristics: This category involves th model’s ability to identify and analyze the physical attributes an structural features of objects in an image. • k Text: Questions related to text or symbols present in the image. • Viewpoint and Perspective: Questions concerning the per-
13
: Orientation and irection,Image
Siz Params (M) IN-1k ZeroShot ☼ Û L , h Ô k MMVP Average
OpenAI ViT-L-14 [43] 2242 427.6 75.5 13.3 13.3 20.0 20.0 13.3 53.3 20.0 6.7 13.3 19.3 OpenAI ViT-L-14 [43] 3362 427.9 76.6 0.0 20.0 40.0 20.0 6.7 20.0 33.3 6.7 33.3 20.0 SigLIP ViT-SO-14 [66] 2242 877.4 82.0 26.7 20.0 53.3 40.0 20.0 66.7 40.0 20.0 53.3 37.8 SigLIP ViT-SO-14 [66] 3842 878.0 83.1 20.0 26.7 60.0 33.3 13.3 66.7 33.3 26.7 53.3 37.0 DFN ViT-H-14 [10] 2242 986.1 83.4 20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53.3 39.3 DFN ViT-H-14 [10] 3782 986.7 84.4 13.3 20.0 53.3 33.3 26.7 66.7 40.0 20.0 40.0 34.8 MetaCLIP ViT-L-14 [62] 2242 427.6 79.2 13.3 6.7 66.7 6.7 33.3 46.7 20.0 6.7 13.3 23.7 MetaCLIP ViT-H-14 [62] 2242 986.1 80.6 6.7 13.3 60.0 13.3 6.7 53.3 26.7 13.3 33.3 25.2
EVA01 ViT-g-14 [54] 2242 1136.4 78.5 6.7 26.7 40.0 6.7 13.3 66.7 13.3 13.3 20.0 23.0
EVA02 ViT-bigE-14+ [54] 2242 5044.9 82.0 13.3 20.0 66.7 26.7 26.7 66.7 26.7 20.0 33.3 33.3
Table 1. Performance of various CLIP based models on different visual patterns in MMVP-VLM benchmark. Models
scaled up in resolution show minimal improvement, whereas a slight advantage is observed when scaling up the network. For each
visual pattern, ImageNet-1k Zero-shot accuracy and MMVP average, we use light gray to highlight the best performance. For most of the visual patterns, all CLIP-based methods show struggle, as evident from the scores. We use symbols for visual patterns due to space limit: ☼: Orientation and Direction, Û: Presence of Specific Features, L: State and Condition, : Quantity and Count, ,: Positional and Relational Context, h: Color and Appearance, Ô: Structural and Physical Characteristics, k: Texts, : Viewpoint and Perspective.
0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1.5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance n visual patterns. If CLIP performs poorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual pattern.
tion between the underperformance of CLIP and MLLMs’ visual incapability? To explore this, we categorize questions from MMVP into these visual patterns summarized and calculate each MLLM’ pe forman e on these pattern .
In Figure 6, we plot CLIP’s performance and MLLMs’ performance for each visual pattern. When the CLIP vision encoder underperforms on a certain visual pattern, the MLLM tends to exhibit similar shortcomings. Open-source models such as LLaVA 1.5 [30] and InstructBLIP [8] that explicitly use the CLIP vision encoder display a strong correlation in performance.
Further, we calculate the Pearson Correlation Coefficient between the CLIP model and MLLM’s performance on each visual pattern. Results show that LLaVA 1.5 and InstructBLIP all possess a coefficient score greater than 0.7. This high score indicates a strong correlation that weaknesses in visual pattern recognition in the CLIP model are transferred to MLLMs. More details on the Pearson Correlation Coefficient can be found in Appendix C.
4. Mixture-of-Features (MoF) for MLLM Based on our exploration in earlier sections, a natural question arises: If open-sourced MLLM’s visual shortcomings come from the CLIP vision encoder, how do we build a more competent visual encoder? In this section, we take initial steps to answer the question by studying Mixtureof-Features (MoF). We start with additive MoF that mixes CLIP features and vision-only SSL model features. R sults show that each encoder presents unique advantages and limitatio s when em lo ed as the pretrain d model in MLLM (Section 4.2). We subsequently propose Interleaved MoF that integrates the features from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model ability to follow instru tions (Section 4.3).
4.1. Experiment Setting
We adopt LLaVA [30, 31] as the framework to study visual encoders in LLM. LLaVA uses a pretrained CLIP encoder and trains an adapter to align visual tokens with language tokens in the LLM. (See left side of Figure 7). We use DINOv2 [42] as the vision-only SSL model in our work because it is currently the most scalable vision-only model. Our exploration includes the us of two visual ncoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ensure consistent and fair comparisons, we train and finetune our model with the same experiment setting in LLaVA. We include the additional experimental details in Appendix A.
4.2. Additive MoF
We add a pretrained DINOv2 encoder into MLLM and mix the CLIP pretrained encoder with it. We use a coefficient α to control the portion of CLIP features and 1 − α to control the amount of DINOv2 features and linearly add them
: Presence of Specific Features, Image Size Params (M) IN-1k ZeroShot ☼ Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2242 427.6 75 13.3 13.3 20.0 20.0 13.3 53.3 20.0 6.7 13.3 19.3 OpenAI ViT-L-14 [43] 3362 427.9 76.6 0.0 20.0 40.0 20.0 6.7 20.0 33.3 6.7 33.3 20.0 SigLIP ViT-SO-14 [66] 2242 877.4 82.0 26.7 20.0 53.3 40.0 20.0 66.7 40.0 20.0 53.3 37.8 SigLIP ViT-SO-14 [66] 3842 878.0 83.1 20.0 26.7 60.0 33.3 13.3 66.7 33.3 26.7 53.3 37.0 DFN ViT-H-14 [10] 2242 986.1 83.4 20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53.3 39.3 DFN ViT-H-14 [10] 3782 986.7 84.4 13.3 20.0 53.3 33.3 26.7 66.7 40.0 20.0 40.0 34.8 MetaCLIP ViT-L-14 [62] 2242 427.6 79.2 13.3 6.7 66.7 6.7 33.3 46.7 20.0 6.7 13.3 23.7 MetaCLIP ViT-H-14 [62] 2242 986.1 80.6 6.7 13.3 60.0 13.3 6.7 53.3 26.7 13.3 33.3 25.2 EVA01 ViT-g-14 [54] 2242 1136.4 78.5 6.7 26.7 40.0 6.7 13.3 66.7 13.3 13.3 20.0 23.0
EVA02 ViT-bigE-14+ [54] 2242 5044.9 82.0 13.3 20.0 66.7 26.7 26.7 66.7 26.7 20.0 33.3 33.3
Table 1. Performance of various CLIP based models on different visual patterns in MMVP-VLM benchmark. Models scaled up in resolution sh w minimal improvement, whereas a slight advantage i observed when scaling p the netw rk. For each
visual pattern, ImageNet-1k Zero-shot accuracy and MMVP average, we use light gr y to highlight the be t perfor ance. For mos of the visual patterns, all CLIP-based methods show struggle, as evident from the scores. We use symbols for visual patterns due to space limit: ☼: Orientation and Direction, Û: Presence of Specific Features, L: State and Condition, : Quantity and Count, ,: Positional and Relational Context, h: Color and Appearance, Ô: Structural and Physical Characteristics, k: Texts, : Viewpoint and Perspective.
0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1.5 Ge ini GPT4
Figure 6. CLIP and MLLM’s performance on visu l patter s. If CLIP performs poorly n a visual patter such s “ ☼ orien ation”, MLLMs also underperform on the visual pattern.
tion between the underperformance of CLIP and MLLMs’ visual incapability? To explore this, we categorize questions from MMVP in o th se visu l patterns sum arized and calculate each MLLM’s performance on these patterns.
In Figu e 6, we plot CLIP’s performance and MLLMs’ performance for each visual pattern. hen the C IP vision encoder underperforms on a certain visual pattern, the MLLM tends to xhibit similar hortcomings. Open-source models such as LLaVA 1.5 [30] and InstructBLIP [8] that explicitly use the CLIP vision e coder dis lay a strong correlation in performance.
Further, we calc l te the Pearson Correlatio Coefficient between the CLIP model and MLLM’s performance on each visual pattern. Results show that LLaVA 1.5 and InstructBLIP all possess a coefficient score greater than 0.7. This high score indicates a strong correlation that weaknesses in visual pattern recognition in the CLIP model are transferred to MLLMs. More details on the Pearson Correlation Coefficient can be found in Appendix C.
4. ixture-of-Features (MoF) for MLLM Bas d on our xplo ation in earlier sections, a natural question arises: If open-sourced MLLM’s visual shortcomings come from the CLIP vision encoder, how do we build a more competent visual encoder? In this section, we tak initial steps to answer the question by studying Mixtureof-Features (MoF). We start with additive MoF that mixes CLIP features and vision-only SSL model features. Results show that each encoder presents un que advantages and limitations when employed as the pret ained model in MLLM (Section 4.2). We subsequently propose Interleaved MoF that integrates the features from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model’s ability to follow instructions (Section 4.3).
4.1. Experiment Setting
We adopt LLaVA [30, 31] as the framework to study visual encoders in MLLM. LLaVA uses a pretrained CLIP encoder and trains an adapter to align visual tokens with l nguage tokens in the LLM. (See left side f Figure 7). We use DINOv2 [42] as the vis on-only SSL model in our work because it is currently the most calable vision-only model. Our exploration includes the use of two visual encoders: CLIP-ViT-L-14 [43] and DI V2-ViT-L-14 [42]. To ensure consistent and fair comparisons, we train a d finetune our model with the same experiment setting in LLaVA. We include the additional experimental details in Appendix A.
4.2. Additive MoF
We add a pretrained DINOv2 encoder into MLLM and mix the CLIP pretrained encoder with it. We use a coefficient α to control the portion of CLIP features and 1 − α to control the amount of DINOv2 features and linearly add them
6
: State and Condition,
Orientation and Direction a rabbit facing right a rabbit facing left
Presence of Specific Features tulips no tulips
State and Condition butterfly with wings open butterfly with wings closed
Quantity and Count 1 drink 2 drinks
Positional and Relational Context glasses on the right of the slipper glasses on the left of the slipper Color and Appearance light blue sky dark blue sky Structural Characteristics some fruits cut in half uncut fruits Texts “11:54” “11:59” Viewpoint and Perspective flowers seen from above flowers seen from the side Model chooses the correct image based on the text Model chooses the wrong image based on the text MMVP-VLM Benchmark Figure 5. Examples from MMVP-VLM. MMVP-VLM co sists of image pairs across nine visual patterns. The examples in the figure are from EVA01 ViT-g-14 model [54], one f the largest CLIP models that also fails to choose the right image given the text description.
emerged from CLIP-blind pairs (Section 3.1), (ii) hether thes visual patterns pose challenges for CLIP-based models with massive scaling up (Section 3.2), and (iii) the correlation between failure patterns in CLIP models and those in MLLMs (Section 3.3).
3.1. Visual Patterns in CLIP-blind Pairs
Having identified he CLIP-blind pairs, w summarize systematic visual patterns that the CLIP vision encoders might consistently misinterpret. It is too abstract to directly capture systematic visual patterns in the CLIP-blind pairs. Th refore, we turn to the questions and options from the MMVP benchmark. With these questions, we transform abstract visual patterns in images into clearer, language-based descriptors that are easier to categorize.
In this work, we use GPT-4 [41] to categorize general patterns by prompting it with the following:
User I am analyzing an image embedding model. Can you go through the questions and options, trying to figure out some general patterns that the embedding model struggles with? Please focus on the visual features and generalize patterns at ar import nt to vision models [MMVP Questions and Options]
We identify 9 visual patterns: ☼ Orientation and Direction
Û Presence of Specific Features
L State and Condition
Quantity and Count , Positional and Relational Context h Color and Appearance Ô Structural and Physical Characteristics k Text Viewpoint and Perspective These visual patterns suggest that CLIP vision encoders
overly focus on high-level semantic understanding, overlooking intricate details of the visual world. Full descriptions of the visual patterns can be found in Appendix D.
3.2. The MMVP-VLM Benchmark
CLIP-based models have developed rapidly since the introduction in the first paper [43]. We want to test whether these visual pat erns still impose challenges to the more recent CLIP models [10, 54, 62, 66], which significantly scale up in terms of training data and model size. In doing so, we introduce a new benchmark: MMVP-VLM to systematically study if CLIP models handle this visual pattern well.
We distill a subset of questions from the MMVP benchmark into simpler language descriptions and categorize them into visual patterns. To maintain a balanced number of questions for each visual pattern, we add a few questions, if needed, to ensure that each visual pattern is represented by 15 text-image pairs. Examples of pairs are shown in Figure . A pair is deemed correctly answered if the model can accurately match both image-text combinations.
We evaluate MMVP-VLM on a variety of CLIP models [10, 43, 54, 62, 66]. These models vary in aspects like size, training data, and methodology. As evidenced in Table 1, increasing network size and training data only aids in identifying two visual patterns – “color and appearance” and “state and condition”. The rest of the visual patterns continue to challenge all CLIP-based models. We also find that the ImageNet-1k zero-shot accuracy is not a definitive indic tor of a model’s performance regarding visual patterns. This underscores the necessity for additional evaluation metrics, such as MMVP-VLM, to accurately assess the model’s capabilities in areas beyond image classification.
3.3. How CLIP’s Errors Affect MLLMs
After analyzing the visual patterns that CLIP models struggle with, we pose the following question: Is there a correla-
5
: Quantity and Count,
Questionnair Progress: 6/300 Can you see the key "Z" in the image?
Yes No
This question is not good Answers are too ambiguous
Back Next
Figure 8. Example of user study interface. The questi s in th user study re randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question.
74 76 78 80 82 84 86 IN-1k ZeroShot Performance
20
25
30
35
40
Av er
ag e
Pe rfo
rm an
ce o
n M
M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14
SigLIP ViT-SO-14
DFN ViT-H-14
DFN ViT-H-14
MetaCLIP ViT-L-1 MetaCLIP ViT-H-14
EVA01 ViT-g-14
EVA02 ViT-bigE-14+
Figure 9. Correlation between ImageNet-1k Zero-shot and
MMVP-VLM aver ge. The area of each bubble corresponds to the model’s number of parameters. A higher ImageNet-1k zeroshot performance does not necessarily imply superior performance in MMVP-VLM.
in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentuates the value of MMVP-VLM as an evaluation metric, which probes into visual patterns such as orientation – as-
pects that are pivotal f r downstream tasks and go beyond wha is
captured by ImageNet accuracy alone.
D. Visual Patterns for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models.
• ☼ Orientation and Direction: Questions about the direction something is facing or moving, such as the direction the dog or duck is facing, or the orientation of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of ertain elements or features in the image. • L St te and Condition: Questions that pertain to the state or condition of an object, such as whether flag is blowing in the wind or if the ground is wet.
• Quantity and Count: Qu stions about the number of objects
or features present n the image.
• , Positional and Relational Cont xt: This aspect refers to the model’s ability to understand the position and relationship of objects or elements within an image in relation to each other and their surroundings. • h Color and Appearance: Questions regarding the color of certain objects or elements. • Ô Structural and Physical Characteristics: This category involves the model’s ability to identify a d analyze the physical attributes and structural features of objects in an image. • k Text: Questions related to text or ymbols present in the
image.
• Viewpoint and Perspective: Questions concerning the per-
13
: Positi l and Relational Context,
Image Size Params (M) IN-1k ZeroShot ☼ Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 22 2 427.6 75.5 13.3 13.3 20.0 20.0 13.3 53.3 20.0 13.3 19.3 OpenAI ViT-L-14 [43] 3362 427.9 76. 0.0 20.0 40.0 20.0 6.7 20.0 33.3 6.7 33.3 20.0 SigLIP ViT-SO-14 [66] 2242 877.4 82.0 26.7 20.0 53.3 40.0 20.0 66.7 40.0 20.0 53.3 37.8 SigLIP ViT-SO-14 [66] 3842 878.0 83.1 20.0 26.7 60.0 33.3 13.3 66.7 33.3 26.7 53.3 37.0 DFN ViT-H-14 [10] 2242 986.1 83.4 20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53.3 39.3 DFN ViT-H-14 [10] 3782 986.7 84.4 13.3 20.0 53.3 33.3 26.7 66.7 40.0 20.0 40.0 34.8 MetaCLIP ViT-L-14 [62] 2242 427.6 79.2 13.3 6.7 66.7 6.7 33.3 46.7 20.0 6.7 13.3 23.7 MetaCLIP ViT-H-14 [62] 2242 986.1 80.6 6.7 13.3 60.0 13.3 6.7 53.3 26.7 13.3 33.3 25.2
EVA01 ViT-g-14 [54] 2242 1136.4 78.5 6.7 26.7 40.0 6.7 13.3 66.7 13.3 13.3 20.0 23.0
EVA02 ViT-bigE-14+ [54] 2242 5044.9 82.0 13.3 20.0 66.7 26.7 26.7 66.7 26.7 20.0 33.3 33.3
Table 1. Performance of various CLIP based models n different visual patterns in MMVP-VLM benchmark. Mod l
scaled up in resolution show minimal improvement, whereas a slight advantage is observe when scaling up the network. For each
visual pattern, ImageNet-1k Zero-shot accuracy and MMVP average, we use light gray to highlight the be t performance. For most of the visual patterns, all CLIP-based methods show struggle, as evident from the scores. We use symbols for visual patterns due to space limit: ☼: Orientation and Direction, Û: Presence of Specific Features, L: State and Condition, : Quantity and Count, ,: Positional and Relational Context, h: Color and Appearance, Ô: Structural and Physical Characteristics, k: Texts, : Viewpoint and Perspective.
0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1.5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance on visual patterns. If CLIP performs poorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual pattern.
tion between the underperformance of CLIP and MLLMs’ visual incapability? To explore his, w categorize questions from MMVP into these visual patter s summ riz d and calculate each MLLM’s performance on these patterns.
In Figure 6, we plot CLIP’s performance and MLLMs’ performance for each visual pattern. When th CLIP vision encoder underperforms on a certain visual pat n, the MLLM tends to exhibit similar shortcomings. Ope -source models such as LLaVA 1.5 [30] and Instru tBLIP [8] that explicitly use the CLIP vision encoder display a stro g correlation in performance.
Further, we calculate the Pearson Correla ion Coefficient between the CLIP model and MLLM’s performance on each visual pattern. Results show that LLaVA 1.5 and InstructBLIP all possess a coefficient score greater t an 0.7. This high score indicates a strong correlation that weaknesses in visual pattern recognition in the CLIP mo el are transferred to MLLMs. More details on the Pearson Corr - lation Coefficient can be found in Appendix C.
4. Mixture-of-Features (M F) for MLLM Based on our exploration in earlier sections, a n tural question a i es: I open-sourced MLLM’s visual shor comings come from the CLIP vision encoder, how d we build a more competent visual e oder? I this sec on, we take initial steps to answer the question by studying Mixtureof-Features (MoF). We start with dditive MoF th mixes CLIP features and vision-only SSL model features. Results show that each encoder pr sents unique advan age and li - itations whe employed as the pretrain d model i MLLM (Section 4.2). We subsequently propose Interleaved MoF that integrates the features from both CLIP and SSL into MLLM to enhance visual grounding without co promising the model’s ability to follow instructions (Section 4.3).
4.1. Experiment Setting
We adopt LLaVA [30, 31] as the framework to stu y v - sual encoders in MLLM. LLaVA uses a pretrained CLIP enc der and trains a adapter to align visual tok ns with language tokens in the LLM. (See left side of Figure 7). W use DINOv2 [42] as the vision-only SSL mo el in our work because it is currently the most scalable vision-only model. Our exploration includes the use of two visual encoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ensure consis ent and fair comparisons, we train and finetune our model with the same experiment setting in LLaVA. We include the additional experimental details in Appendix A.
4.2. Additive MoF
We add a pretrained DINOv2 encod r into MLLM and mix the CLIP pretrai ed ncoder with it. We use a coefficient α to c ntrol the portion of CLIP features and 1 − α to control the amount of DINOv2 features and linearly add them
6
: Color and Appearance,
Questionnaire Progress: 6/300
Can you see the key "Z" in the image?
Yes No
This question is not good Answers are too ambiguous
Back Next
Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question.
74 76 78 80 82 84 86 IN-1k Zer Shot Perform nce
20
25
30
35
40
Av er
ag e
Pe rfo
rm an
ce o
n M
M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14
SigLIP ViT-SO-14
DFN ViT-H-14
DFN ViT-H-14
MetaCLIP ViT-L-14 MetaCLIP ViT-H-14
EVA01 ViT-g-14
EVA02 ViT-bigE-14+
Figure 9. Correlation bet een ImageNet-1k Zero-shot an MMVP-VLM average. The area of each bubble corresponds to
the model’s number of parameters. A higher ImageNet-1k zero-
shot p rformance oes n t necessarily imply superior performance
in MMVP-VLM.
in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentuates the value of MMVP-VLM as an evaluation metric, which probe into visual patterns such as orientation – aspects that are pivotal for downstream tasks and go beyond what is captured by ImageNet accuracy alone.
D. Visual Patter s for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models.
• ☼ Orientation and Direction: Questions about the direction something is facing or moving, such as the direction the dog or duck is facing, or the orientat n of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of c rtain elements or f atures in the image. • L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet. • Quantity and Count: Questions about the number of objects or features pres nt in the imag . • , Positional and Relational Context: This aspect refers to the model’ ability to understand the position and relationship of
objects or elements within an image in relation to each other
and their surroundings.
• h Color and Appearance: Questions regarding the color of certain objects or elements. • Ô Structural and Physical Characteristics: This category involves the model’s ability to identify and analyze th physical attributes and structural features of objects in an image. • k Text: Questions related to text or symbols present in the image. • Viewpoint and Perspective: Questions concerning the per-
13
: Structural and Physical Charac eristic ,
Questionnaire Progress: 6/300 Can you see the key "Z" in the image?
Yes No
This question is not good Answ rs are too ambigu us
Back Next
Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose nswers for the VQA question as well as potential concerns for the VQA questio .
74 76 78 80 82 84 86 IN 1k ZeroShot Performance
20
25
30
35 40 Av er ag e Pe rfo rm an ce o n M M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14
SigLIP ViT-SO-14
DFN ViT-H-14
DFN ViT-H-14
MetaCLIP ViT-L-14 MetaCLIP ViT-H-14
EVA01 ViT-g-14
EVA02 ViT-bigE-14+
Figure 9. Correlation between Im g Net-1k Zero-shot and MMVP-VLM average. The area of each bubble corresponds to the model’s number of param ers. A higher ImageNet-1k z roshot performance does not necessarily imply superior performance in MMVP-VLM.
in models with superior ImageN t-1k Z ro-shot performance, this
trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentuates th value of MMVP-VLM as an valuati m tric, which probes into visual pa terns suc as orientation – aspects that are pivotal for downstrea tasks and go beyond what is captured by ImageNet accuracy alone.
D. Visual Patterns for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models.
• ☼ Orientation and Direction: Questions about the direction something is facing or moving, such as the direction the dog or duck is facing, or the orientation of th school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the i age. • L S ate and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet. • Quantity and Count: Questions about the number of objects or f atures present i the image. • , Positional and Relational Context: This aspect refers to the model’s ability to understand the position and relationship of objects or elements within an image in relation to each other and their surroundings. • h Color and App a ance: Questions regarding the color of
certain obje ts or el ments.
• Ô Structural and Physical Characteristics This category
involves the model’s ability to identify and analyze the physical attributes and structural features of objects in an image. • k Text: Questions related to text or symbols present in the image. • Viewpoint and Perspective: Questions concerning the per-
13
: Text and Image Size Params (M) IN-1k Zer Shot ☼ Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2242 427.6 75.5 1 .3 13 3 20.0 20.0 13.3 53.3 20.0 6.7 13.3 19.3 OpenAI ViT-L-14 [43] 3 62 427.9 76.6 0. 20.0 40.0 20.0 6.7 20.0 33.3 6.7 33.3 20.0 SigLIP ViT-SO-1 [66] 2242 877.4 82.0 26.7 20.0 53.3 40.0 20.0 66.7 40.0 20.0 53.3 37.8 SigLIP ViT-SO-14 [66] 3842 878.0 83.1 20.0 26.7 60.0 33.3 13.3 66.7 33.3 26.7 53.3 37.0 DFN ViT-H-14 [10] 2242 986.1 83.4 20.0 26.7 73.3 26.7 26.7 66.7 46.7 13.3 53.3 39.3 DFN ViT-H-14 [10] 3782 986.7 84.4 1 .3 20.0 53.3 33.3 26.7 66.7 40.0 20.0 40.0 34.8 MetaCLIP ViT-L-14 [62] 2242 427.6 79.2 13.3 6.7 66.7 6.7 33.3 46.7 20.0 6.7 13.3 23.7 MetaCLIP ViT-H-14 [62] 2242 986.1 80.6 6.7 13.3 60.0 13.3 6.7 53.3 26.7 13.3 33.3 25.2
EVA01 ViT-g-14 [54] 2242 136.4 78.5 6.7 26.7 40.0 6.7 13.3 66.7 13.3 13.3 20.0 23.0
EVA02 ViT-bigE-14+ [54] 2242 5044.9 82.0 1 .3 20.0 66.7 26.7 26.7 66.7 26.7 20.0 33.3 33.3
Table 1. Perform nce f various CLIP based odels on ifferent visu l pattern in MMVP-VLM b chmark. Models
caled up in resoluti n show minimal improvem nt, hereas a slight advantage is observed when scaling up the network. For each
visual pattern, Ima eNet-1k Zero-sho accuracy and MMVP average, we use light gray t highlight the best performance. For most of th visual patterns, all CLIP-based meth ds show struggle, as evident from the scores. We use symbols for visual patterns due to space limit: ☼: Ori ntation and Direction, Û: Pres nce of Specific Features, L: State and Condition, : Quantity and Count, ,: Positional and Relational Context, h: Color and Appearance, Ô: Structural and Physical Characteristics, k: Texts, : Viewpoint and Perspective.
0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1.5 Gemini GPT4
Figure 6. CLIP and MLLM’s performanc on visual patterns. If CLI perf rms poorly on a visual pa tern uch as “ ☼ orientation”, MLLMs also underperform on the visual pattern.
tion between the underp rformanc of CLIP and MLLMs’ visual incapability? To ex l re this, w categ riz questions from MMVP into these visu l patterns summarized and calculate each MLLM’s performance on these patterns.
In Fig re 6, w plot CLIP’s erformance and MLLMs’ p rfo mance for each visua pattern. When the CLIP vision encoder underperforms on a certain visual pattern, the MLLM tends to exhibit similar shortcomings. Open-source models such s LLaV 1.5 [30] and I structBLIP [8] at expli itly use the CLIP vision e coder displ y a strong correlation in performance.
Further, we calculate the P arson Correlation Coefficient between the CLIP model and MLLM’s performance on each visual pattern. Results show that LLaVA 1.5 and Instruc BLIP all possess a coefficient score greater th n 0.7. This high score i dicates a strong correlat on t at weaknesse in visual patter recogniti n th CLIP model are transferred to MLLMs. More details on he Pearson Correlation Coefficient can be found in Appendix C.
4. Mixture-of-Features (MoF) for MLLM Based on our exploration in earlier secti n , a natural question arises: If open-sourced MLLM’s visual shortcomings come from th CLIP vi ion encoder, how do we build a more competent visual encoder? In this section, we take initial steps to answer the question by studying Mixtureof-Features ( oF). We start with additive MoF that mixes CLIP features and vision-only SSL model features. Results show that each encoder p sents u ique dvantag s and limtations wh em l yed as the pretrained model in MLLM (Section 4.2). We subsequently propose Interleaved MoF that integrates the features fr b th CLIP and SSL into MLLM t enh nce visual groundi g without compromising the m del’s ability to follow instructio s (Section 4.3).
4.1. Experiment Setting
W a opt [30, 31] as the framework to study visual encoders in MLLM. LLaVA u es a pretrained CLIP
coder and trains an adapter to align visual tok ns w th language tokens in the LM. (See left side of Figure 7). We use DINOv2 [42] as i i l SSL model in our work beca se it is c rr ntly the most scalable vision-only model. Our exploration includes the use of two visual encoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ensure consistent and fair comparisons, we train and finetune our model with the same experim t setting in LLaVA. We include the additional experimental details in Appendix A.
4.2. Additive MoF
We add a p etrained DINOv2 encoder i to MLLM and mix the CLIP pretrained encoder with it. We use a coefficient α to control the po tion of CLIP fe tures and 1 − α to control the amount of DINOv2 features and linearly add them
6
: Viewpoint and Perspective.
Method Params All Video Image
Qwen-VL-Chat (Bai et al., 2023) 7B 9.1 8.4 11.3 Video-LLaVA (Lin et al., 2023) 7B 11.3 11.8 9.1 Co textBLIP 240M 8.5 28.7 81.2
Table 17: Comparisons with NDCR, QWen-VL-Chat (Bai t al., 23) and Vide -LLaVA (Lin et al., 2023) on IMAGECODE: We compare our model with the previous NDCR and two recently releas d MLLMs QWe - VL-Chat (Bai et al., 2023) and Video-LLaVA (Lin et al., 2023) on the challenging instances of IMAGECODE. Experimental r sults show t at th propo ed Cont xtBLIP consist ntly p rforms best amo g these baselines.
D.3 Comparison with More MLLMs See Table 17 and Table 18.
16252
Query : Two men facing each other. PBLIP [0.100, 0.101, 0.101, 0.100, 0.100, 0.100, 0.099, 0.100, 0.100, 0.100] (×) POurs [0.076, 0.089, 0.094, 0.090, 0.084, 0.099, 0.142, 0.121, 0.093, 0.113] (✓)
Query : The man's right eye is 40% visible. PBLIP [0.100, 0.099, 0.103, 0.099, 0.099, 0.099, 0.099, 0.105, 0.099, 0.099] (×) POurs [0.145, 0.129, 0.105, 0.086, 0.090, 0.083, 0.084, 0.085, 0.105, 0.089] (✓)
Figure 5: Zero-shot cases from the test set. Our model has advantages over BLIP in both confidence scores.
Query : Two thumbs on egg, no blur, bottom right blue dial has just started to be covered.
Key contextual cue : Two thumbs on egg.
PBLIP’ [0.187, 0.034, 0.196, 0.242, 0.025, 0.045, 0.065, 0.048, 0.025, 0.132] (×) POurs’ [0.086, 0.036, 0.412, 0.22 , 0.016, 0.031, 0.045, 0.034, 0.025, 0.095] (✓)
PBLIP [0.118, 0.047, 0.104, 0.134, 0.079, 0.131, 0.095, 0.105, 0.075, 0.111] (×) POurs [0.097, 0.063, 0.137, 0.131, 0.068, 0.107, 0.106, 0.111, 0.087, 0.092] (✓)
Query : The man's face is visible but very blurry. He is punching downward but you can't see any of his elbow because it is blocked by his fist. His fist is visible but blurry.
Key contextual cue : The man's face is very blurry.
PBLIP’ [0.011, 0.117, 0.141, 0.39 , 0.006, 0.002, 0.046, 0.084, 0.199, 0.003] (×) POurs’ [0.021, 0.113, 0.194, 0.215, 0.008, 0.001, 0.051, 0.2 , 0.194, 0.003] (✓)
PBLIP [0.063, 0.076, 0.193, 0.262, 0.016, 0.017, 0.063, 0.063, 0.229, 0.019] (×) POurs [0.087, 0.067, 0.24 , 0.207, 0.014, 0.008, 0.069, 0.126, 0.168, 0.014] (✓)
Figure 6: Two cases from the test set after fine-tuning. Our model outperforms BLIP in confidence scores for both compound and simple proposition texts.

heading: Future Work
text: In the future, we plan to extend our method to text-to-video retrieval. 6 Limitation
The adaptive mask ratio is worth considering in the future, as a fixed masking ratio in our paper may not dynamically adapt to different cross-modal interactions. The proposed method may also have limitations for fine-grained retrieval for long videos, as pre-training on long videos is time-expensive and requires very large GPUs. Acknowledgement
We thank Benno Krojer for providing the benchmarks and assisting in analyzing the leaderboard scores. This work was supported by the National Natural Science Foundation of China (No. 62372125). References Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang,
Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile visionlanguage model for understanding, localization, text reading, and beyond. Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2021. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106. 08254. Hangbo Bao, Wenhui Wang, Li Dong, and Furu Wei. 2022. Vl-beit: Generative vision-language pretraining. arXiv preprint arXiv:2206. 01127. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In ECCV, pages 104–120. Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun He, Tong Lu, Jifeng Dai, and Yu Qiao. 2022. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205. 08534. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929. Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. 2023. Data filtering networks. Jerry Fodor. 2001. Language, thought and compositionality. Royal Institute of Philosophy Supplement, 48:227–242. Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Jiao Qiao. 2021. Clip-adapter: Better vision-language models with feature adapters. ArXiv, abs/2110. 04544. Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel. 2022. Multimodal masked autoencoders learn transferable representations. arXiv preprint arXiv:2205. 14204. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked autoencoders are scalable vision learners. In CVPR, pages 16000–16009. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In ICML, pages 2790–2799. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904–4916. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412. 6980. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123:32–73. Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy. 2022. Image retrieval from contextual descriptions. arXiv preprint arXiv:2203. 15867. Gukyeong Kwon, Zhaowei Cai, Avinash Ravichandran, Erhan Bas, Rahul Bhotika, and Stefano Soatto. 2022. Masked vision and language modeling for multi-modal representation learning. arXiv preprint arXiv:2208. 02131. Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. IEEE, 86(11):2278–2324. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. Blip-2: Bootstrapping language-image pretraining with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping language-image pretraining for unified vision-language understanding and generation. In ICML, pages 12888–12900. Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. In NeurIPS, volume 34, pages 9694–9705. Yunxin Li, Baotian Hu, Yunxin Ding, Lin Ma, and Min Zhang. 2023b. A neural divide-and-conquer reasoning framework for image retrieval from linguistically complex text. arXiv preprint arXiv:2305. 02265. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2023. Video-llava: Learning united visual representation by alignment before projection. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV, pages 740– 755. Haoyu Lu, Mingyu Ding, Yuqi Huo, Guoxing Yang, Zhiwu Lu, Masayoshi Tomizuka, and Wei Zhan. 2023. Uniadapter: Unified parameter-efficient transfer learning for cross-modal modeling. arXiv preprint arXiv:2302. 06605. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS, volume 32. Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer imageto-sentence models. In ICCV. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In ICML, volume 139, pages 8748– 8763. Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 15389. Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. 2024. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint arXiv:2401. 06209. Haoqing Wang, Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhi-Hong Deng, and Kai Han. 2023. Masked image modeling with local multi-scale reconstruction. In CVPR, pages 2122–2131. Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. OFA: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In ICML, volume 162, pages 23318–23340. Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. 2023. Demystifying clip data. arXiv preprint arXiv:2309. 16671. Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309. 17421, 9(1):1. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. arXiv preprint arXiv:2303. 15343. Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. 2021. Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling. A Implementation details
A. 1 Training Hyper-parameters Pre-training. We conducted pre-training on 4 Nvidia A100 GPUs for about 10 hours, randomly sampling 2242 patches from images and using RandAugment for data augmentation. Fine-tuning. Compared to Krojer et al. ’s multistep approach involving +Context Batch, +Context Module, and +Temporal Embeddings, we streamline Krojer et al. ’s procedure by directly fine-tuning our model on IMAGECODE and then separately training a contextual modeling module between
the backbone and the prediction head to enable contextual comparison and reasoning. During the first stage, we conduct full fine-tuning for 20 epochs. After that, we add a two-layer transformer as a context/temporal module, which is identical to Krojer et al. To optimize the module while keeping other parameters fixed, we freeze all parts of the model except the inserted module and train it alone for 5 epochs. We set the batch size to 360 images (36 batches of 10-image-set) and use the Adam (Kingma and Ba, 2014) optimizer with learning rate of 2e-6 and weight decay of 0. 01. We conduct the experiment on one NVIDIA Geforce 3090 for about one day. All other baselines also adopted the same settings. A. 2 Dataset Details
Table 13 provides statistics for the datasets used in pre-training and fine-tuning. It includes three datasets: COCO, VG (Visual Genome), and IMAGECODE. The number of images (#image) and texts (#text) for each dataset are listed. A. 3 Model Details
In section 3. 2, the scorer is a 1x768 linear layer. In section 3. 3 (Step 2), the decoder is a four-layer transformer, with each layer having twelve attention heads and a feature dimension of 768. In section 4. 2, for CLIP, ViLBERT, and UNITER, we utilized the models with added context modules and temporal embeddings as proposed by Krojer et al. Our-Context model incorporated
a context module in the same manner, while other models did not have such a module added. For ALBEF, due to its pretraining on images with a resolution of 256x256, we conducted experiments using a slightly higher resolution (224x224) compared to other models. B Validation Performance
Comparison with state-of-the-art methods on the valid set of IMAGECODE is shown on Table 15. C Additional Case Studies
For tasks involving high image similarity and detailed textual descriptions, such as contrastive image retrieval, the challenge is considerable. We will demonstrate the superior performance of our MIM adaption method in more scenarios and provide the following examples to further illustrate the doubly contextual alignment capabilities of our model. Figure 5 presents examples of two zero-shot experiments. In the left image, when two men are facing each other, our model assigns significantly higher matching scores compared to other images. In the right image, despite the difficulty in estimating the proportion of the man’s right eye visible, our model assigns obviously higher scores to the first two images showing the right eye. In contrast, BLIP assigns similar scores to all ten images in both samples, indicating its difficulty in attending to the textual cues used to align the intra-contexual information. Figure 6 showcases examples of fine-tuning experiments. The left image in Figure 6 illustrates an example of the Quantity phenomenon, with the subheading "Two thumbs on the egg. " We observe that our model infers significantly higher confidence scores for images 2 and 3 (matching the description of thumb quantity) and infers higher BLIP confidence scores for images 0, 2, and 3 (where the description of thumb quantity does not match). Thus, we demonstrate that our model indeed performs better in dual-contexual alignment. The right image in Figure 6 demonstrates an example of the Meta-property phenomenon, with the subheading "The man’s face is blurry. " We find that our model infers higher confidence scores for images 2, 3, 7, and 8 (all relatively matching the description) while BLIP infers higher confidence scores for images 1, 2, 3, and 8 (where image 1 is clear but does not match the description). Hence, it indicates that our model indeed performs better in
understanding intra-contexual alignment. In conjunction with the main text, we demonstrate the advantages of our model in two challenges: better intra-contexual alignment, better inter-contexual alignment. It can be seen that in many scenarios, our model exhibits better dualcontextual alignment capabilities compared to BLIP. D More about GPT-4V Experiment
D. 1 Sampling Details
Due to GPT-4V’s high computational demands, we created distinct test datasets from IMAGECODE’s static and video-shot sections. We randomly chose 50 samples from each, repeating thrice to avoid bias. Each sample included a ground-truth image plus nine adjacent ones, since GPT-4V processes up to ten images simultaneously. We prompted
GPT-4V in form of "Which image fits the text description best? Please output the serial number of the image:< text_query >" to select the matching image. We used three random seeds: 1, 10, 100, to explore the datasets with the text length distribution (in tokens) illustrated in the Figure 7. In this experiment, using Python scripts with the API interface of the gpt-4-vision-preview model, we make a request to GPT-4V. If encounter a refusal to answer, we use the browser version to ask again until a result is obtained. D. 2 More Comparison with GPT-4V
This part provides some more examples of GPT4V prediction errors, where our model predicts correctly. The first two samples are from video shot set, and the last two are from static pictures. Image Size Params
Questionnaire Progress: 6/300 Can you see the key "Z" in the image? Yes No This question is not good Answers are too ambiguous Back Next Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question. 74 76 78 80 82 84 86 IN-1k ZeroShot Performance
20
25
30
35
40
Av er
ag e
Pe rfo
rm an
ce o
n M
M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14
SigLIP ViT-SO-14
DFN ViT-H-14
DFN ViT-H-14
MetaCLIP ViT-L-14 MetaCLIP ViT-H-14
EVA01 ViT-g-14
EVA02 ViT-bigE-14+
Figure 9. Correlation between ImageNet-1k Zero-shot and MMVP-VLM average. The area of each bubble corresponds to the model’s number of parameters. A higher ImageNet-1k zeroshot performance does not necessarily imply superior performance in MMVP-VLM. in models with superior ImageNet-1k Zero-shot performance, this
trend does not necessarily hold for MMVP-VLM accuracy. This
distinction accentuates the value of MMVP-VLM as an evaluation
metric, which probes into visual patterns such as orientation – aspects that are pivotal for downstream tasks and go beyond what is captured by ImageNet accuracy alone. D. Visual Patterns for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models. • ☼ Orientation and Direction: Questions about the direction something is facing or moving, such as the direction the dog or duck is facing, or the orientation of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the image. • L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet. • Quantity and Count: Questions about the number of objects or features present in the image. • , Positional and Relational Context: This aspect refers to the model’s ability to understand the position and relationship of objects or elements within an image in relation to each other and their surroundings. • h Color and Appearance: Questions regarding the color of certain objects or elements. • Ô Structural and Physical Characteristics: This category
involves the model’s ability to identify and analyze the physical
attributes and structural features of objects in an image. • k Text: Questions related to text or symbols present in the image. • Viewpoint and Perspective: Questions concerning the per-
13
Image Size
Params (M)
IN-1k ZeroShot ☼ Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2242 427. 6 75. 5 13. 3 13. 3 20. 0 20. 0 13. 3 53. 3 20. 0 6. 7 13. 3 19. 3 OpenAI ViT-L-14 [43] 3362 427. 9 76. 6 0. 0 20. 0 40. 0 20. 0 6. 7 20. 0 33. 3 6. 7 33. 3 20. 0 SigLIP ViT-SO-14 [66] 2242 877. 4 82. 0 26. 7 20. 0 53. 3 40. 0 20. 0 66. 7 40. 0 20. 0 53. 3 37. 8 SigLIP ViT-SO-14 [66] 3842 878. 0 83. 1 20. 0 26. 7 60. 0 33. 3 13. 3 66. 7 33. 3 26. 7 53. 3 37. 0 DFN ViT-H-14 [10] 2242 986. 1 83. 4 20. 0 26. 7 73. 3 26. 7 26. 7 66. 7 46. 7 13. 3 53. 3 39. 3 DFN ViT-H-14 [10] 3782 986. 7 84. 4 13. 3 20. 0 53. 3 33. 3 26. 7 66. 7 40. 0 20. 0 40. 0 34. 8 MetaCLIP ViT-L-14 [62] 2242 427. 6 79. 2 13. 3 6. 7 66. 7 6. 7 33. 3 46. 7 20. 0 6. 7 13. 3 23. 7 MetaCLIP ViT-H-14 [62] 2242 986. 1 80. 6 6. 7 13. 3 60. 0 13. 3 6. 7 53. 3 26. 7 13. 3 33. 3 25. 2 EVA01 ViT-g-14 [54] 2242 1 36. 4 78. 5 6. 7 26. 7 40. 0 6. 7 13. 3 66. 7 13. 3 13. 3 20. 0 23. 0 E A02 ViT-bigE-14+ [54] 2242 5044. 9 82. 0 13. 3 20. 0 66. 7 26. 7 26. 7 66. 7 26. 7 20. 0 33. 3 33. 3
Table 1. Performance of various CLIP based models on different visual patterns in MMVP-VLM benchmark. Models scaled up in resolution show minimal improvement, whereas a slight advantage is observed when scaling up the network. For each
visual pattern, ImageNet-1k Zero-shot accuracy and MMVP average, we use light gray to highlight the best performance. For most of
the visual patterns, all CLIP-based methods show struggle, as evident from the scores. We use symbols for visual patterns due to space
limit: ☼: Orientation and Direction, Û: Presence of Specific Features, L: State and Condition, : Quantity and Count, ,: Positional and
Relational Context, h: Color and Appearance, Ô: Structural and Physical Characteristics, k: Texts, : Viewpoint and Perspective. 0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1. 5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance on visual patterns. If CLIP performs poorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual pattern. tion between the underperformance of CLIP and MLLMs’ visual incapability? To explore this, we categorize questions from MMVP into these visual patterns summarized and calculate each MLLM’s performance on these patterns. In Figure 6, we plot CLIP’s performance and MLLMs’ performance for each visual pattern. When the CLIP vision encoder underperforms on a certain visual pattern, the MLLM tends to exhibit similar shortcomings. Open-source models such as LLaVA 1. 5 [30] and InstructBLIP [8] that explicitly use the CLIP vision encoder display a strong correlation in performance. Further, we calculate the Pearson Correlation Coefficient between the CLIP model and MLLM’s performance on each visual pattern. Results show that LLaVA 1. 5 and InstructBLIP all possess a coefficient score greater than 0. 7. This high score indicates a strong correlation that weaknesses in visual pattern recognition in the CLIP model are transferred to MLLMs. More details on the Pearson Correlation Coefficient can be found in Appendix C. 4. Mixture-of-Features (MoF) for MLLM Based on our exploration in earlier sections, a natural question arises: If open-sourced MLLM’s visual shortcomings come from the CLIP vision encoder, how do we build a
more competent visual encoder? In this section, we take
initial steps to answer the question by studying Mixture-
of-Features (MoF). We start with additive MoF that mixes
CLIP features and vision-only SSL model features. Results show that each encoder presents unique advantages and limitations when employed as the pretrained model in MLLM (Section 4. 2). We subsequently propose Interleaved MoF that integrates the features from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model’s ability to follow instructions (Section 4. 3). 4. 1. Experiment Setting
We adopt LLaVA [30, 31] as the framework to study visual encoders in MLLM. LLaVA uses a pretrained CLIP encoder and trains an adapter to align visual tokens with language tokens in the LLM. (See left side of Figure 7). We use DINOv2 [42] as the vision-only SSL model in our work because it is currently the most scalable vision-only model. Our exploration includes the use of two visual encoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ensure consistent and fair comparisons, we train and finetune our model with the same experiment setting in LLaVA. We include the additional experimental details in Appendix A. 4. 2. Additive MoF
We add a pretrained DINOv2 encoder into MLLM and mix the CLIP pretrained encoder with it. We use a coefficient α to control the portion of CLIP features and 1 − α to control the amount of DINOv2 features and linearly add them
6
age ize Params (M) IN 1k ZeroShot ☼ Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2 427. 6 75. 13. 3 13. 3 20. 0 20. 0 13. 3 5. 3 20. 0 6. 7 13. 3 19. 3 OpenAI ViT-L-14 [43] 3362 427. 9 76. 6 0. 0 20. 0 40. 0 20. 0 6. 7 20. 0 33. 3 6. 7 33. 3 20. 0 SigLIP ViT-SO-14 [66] 2242 877. 4 82. 0 26. 7 20. 0 53. 3 40. 0 20. 0 66. 7 40. 0 20. 0 53. 3 37. 8 SigLIP ViT-SO-14 [66] 3842 878. 0 83. 1 20. 0 26. 7 60. 0 33. 3 13. 3 66. 7 33. 3 26. 7 53. 3 37. 0 DFN ViT-H-14 [10] 2242 986. 1 83. 4 20. 0 26. 7 73. 3 26. 7 26. 7 66. 7 46. 7 13. 3 53. 3 39. 3 DFN ViT-H-14 [10] 3782 986. 7 84. 4 13. 3 20. 0 53. 3 33. 3 26. 7 66. 7 40. 0 20. 0 40. 0 34. 8 MetaCLIP ViT-L-14 [62] 2242 427. 6 79. 2 13. 3 6. 7 66. 7 6. 7 33. 3 46. 7 20. 0 6. 7 13. 3 23. 7 MetaCLIP ViT-H-14 [62] 2242 986. 1 80. 6 6. 7 13. 3 60. 0 13. 3 6. 7 53. 3 26. 7 13. 3 33. 3 25. 2 EVA01 ViT-g-14 [54] 2242 1136. 4 78. 5 6. 7 26. 7 40. 0 6. 7 13. 3 66. 7 13. 3 13. 3 20. 0 23. 0 EVA02 ViT-bigE-14+ [54] 2242 5044. 9 82. 0 13. 3 20. 0 66. 7 26. 7 26. 7 66. 7 26. 7 20. 0 33. 3 33. 3
Table 1. Performance of various CLIP based models on different visual patterns in MMVP-VLM benchmark. Models scaled up in res lutio sh w minimal improvement, whereas a slight adva tage is observed when scaling up the network. For each
visual pattern, ImageNet-1k Zero-shot accuracy and MMVP average, we use light gray to highlight the best p rformance. F r m st of the visual patterns, all CLIP-based methods show struggle, as evident from the scores. We use symbols for visual patterns due to space limit: ☼: Orientation and Direction, Û: Presence of Specific Features, L: State and Condition, : Quantity and Count, ,: Positional and
Relational Context, h: Color and Appearance, Ô: Structural and Physical Characteristics, k: Texts, : Viewpoint and Perspective. 0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1. 5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance on visual patterns. If CLIP performs poorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual pattern. tion between the r erfor a ce of CLIP nd MLLMs’ visual incapability? To explore this, we categorize questions from MMVP into these visual patterns summarized and calculate each MLLM’s performance on these patterns. In Figure 6, we plot CLIP’s performance and MLLMs’ performance for each visual pattern. When the CLIP vision encoder underperforms on a certain visual pattern, the MLLM tends to exhibit similar sho tco ings. Open-source m dels such as LLaVA 1. 5 [30] and InstructBLIP [8] that explicitly use the CLIP vision encoder display a strong correlation in performance. Further, we calculate the Pearson Correlation Coefficient between the CLIP m del and MLLM’s perfo mance on each visual pattern. Results show that LLaVA 1. 5 and InstructBLIP all possess a coefficient score greater than 0. 7. This high score indicates a strong correlation that weaknesses in visual pattern recognition in the CLIP model are transferred to MLLMs. More details on the Pearson Correlation Coefficient can be found in Appendix C. 4. Mixture-of-Features (MoF) for MLLM Based on our exploration in ea lier sections, a natural question arises: If open-sourced MLLM’s visual shortcomings come from the CLIP vision encoder, how do we build a more competent visual encoder? In this section, we take initial steps to answer the question by studying Mixture-
of-Features (MoF). We st rt with additive MoF that mixes
CLIP featur s and vision-only SSL model features. Results
show that each encoder presen s unique advantages and lim-
itations when employed as the pretrained el in MLLM (Section 4. 2). We subsequently propose Interleaved MoF that integrates the features from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model’s ability to follow instructions (Section 4. 3). 4. 1. Experiment Setting
We adopt LLaVA [30, 31] as the framework to study visual encoders in MLLM. LLaVA uses a pretrained CLIP encoder and trains an adapter to align visual tokens with language t kens in the LLM. (See left side of Figure 7). We use DINOv2 [42] as the vision-only SSL model in our work because it is currently the most scalable vision-only model. Our exploration i cludes the use of two visual encoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ensure consistent and fair comparisons, we train and finetune our model with the same experiment setting in LLaVA. We include the additional experimental details in Appendix A. 4. 2. Additive MoF
We add a pretrained DINOv2 encoder into MLLM and mix the CLIP pretrained encoder with it. We use a coefficient α to control the portion of CLIP features and 1 − α to control the amount of DINOv2 features and linearly add them
6
Orientation and Direction a rabbit facing right a rabbit facing left
Presence of Specific Features tulips no tulips
State and Condition butterfly with wings open butterfly with wings closed
Quantity and Count 1 drink 2 drinks
Positional and Relational Context glasses on the right of the slipper glasses on the left of the slipper Color and Appearance light blue sky dark blue sky Structural Characteristics some fruits cut in half uncut fruits Texts “11:54” “11:59” Viewpoint and Perspective flowers seen from above flowers seen from the side Model chooses the correct image based on the text Model chooses the wrong image based on the text MMVP-VLM Benchmark Figure 5. Examples from MMVP-VLM. MMVP-VLM consists of image pairs across nine visual patterns. The examples in the figure are from EVA01 ViT-g-14 model [54], one of the largest CLIP models that also fails to choose the right image given the text description. emerged from CLIP-blind pairs (Section 3. 1), (ii) whether these visual patterns pose challenges for CLIP-based models with massive scaling up (Section 3. 2), and (iii) the correlation between failure patterns in CLIP models and those in MLLMs (Section 3. 3). 3. 1. Visual Patterns in CLIP-blind Pairs Having identified the CLIP-blind pairs, we summarize systematic visual patterns that the CLIP vision encoders might consistently misinterpret. It is too abstract to directly capture systematic visual patterns in the CLIP-blind pairs. Therefore, we turn to the questions and options from the MMVP benchmark. With these questions, we transform abstract visual patterns in images into clearer, language-based descriptors that are easier to categorize. In this work, we use GPT-4 [41] to categorize general patterns by prompting it with the following: User I am analyzing an image embedding model. Can you go through the questions and options, trying to figure out some general patterns that the embedding model struggles with? Please focus on the visual features and generalize patterns that are important to vision models [MMVP Questions and Options]
We identify 9 visual patterns: ☼ Orientation and Direction Û Presen e of Specific Fe tures L State and Condition Quantity and Count , Positional and Relational Context h Color and Appearance Ô Structural and Physical Characteristics k Text Viewpoint and Pers ct ve
These visual patterns suggest that CLIP vision encoders
overly focus on high-level semantic understanding, overlooking intricate details of the visual world. Full descriptions of the visual patterns can be found in Appendix D. 3. 2. The MMVP-VLM Benchmark CLIP-based models have developed rapidly since the introduction in the first paper [43]. We want to test whether these visual patterns still impose challenges to the more recent CLIP models [10, 54, 62, 66], which significantly scale up in terms of training data and model size. In doing so, we introduce a new benchmark: MMVP-VLM to systematically study if CLIP models handle this visual pattern well. We distill a subset of questions from the MMVP benchmark into simpler language descriptions and categorize them into visual patterns. To maintain a balanced number of questions for each visual pattern, we add a few questions, if needed, to ensure that each visual pattern is represented by 15 text-image pairs. Examples of pairs are shown in Figure 5. A pair is deemed correctly answered if the model can accurately match both image-text combinations. We evaluate MMVP-VLM on a variety of CLIP models [10, 43, 54, 62, 66]. These models vary in aspects like size, training data, and methodology. As evidenced in Table 1, increasing network size and training data only aids in identifying two visual patterns – “color and appearance” and “state and condition”. The rest of the visual patterns continue to challenge all CLIP-based models. We also find that the ImageNet-1k zero-shot accuracy is not a definitive indicator of a model’s performance regarding visual patterns. This underscores the necessity for additional evaluation metrics, such as MMVP-VLM, to accurately assess the model’s capabilities in areas beyond image classification. 3. 3. How CLIP’s Errors Affect MLLMs
After analyzing the visual patterns that CLIP models struggle with, we pose the following question: Is there a correla-
5
Questionnaire Progress: 6/300 Can you see the key "Z" in the image? Yes No This question is not good Answers are too ambiguous Back Next Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question. 74 76 78 80 82 84 86 IN-1k ZeroShot Performance
20
25 30 35 40 Av er ag e Pe rfo rm an ce o n M M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14 SigLIP ViT-SO-14 DFN ViT-H-14 DFN ViT-H-14 MetaCLIP ViT-L-14 MetaCLIP ViT-H-14 EVA01 ViT-g-14 EVA02 ViT-bigE-14+
Figure 9. Correlation between I ageNet-1k Zero-shot and MMVP-VLM average. The area of each bubble corresponds to the model’s number of parameters. A higher ImageNet-1k zeroshot performance does not necessarily imply superior performance in MMVP VLM. in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentuates the value of MMVP-VLM as an evaluation metric, which probes into visual patterns such as orientation – aspects that are pivotal for downstream tasks and go beyond what is captured by ImageNet accuracy alone. D. Visual Patterns for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models. • ☼ Orientation and Direction: Questions about the direction something is facing or moving, such as the direction the dog or duck is facing, or the orientation of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the image. • L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet. • Quantity and Count: Questions about the number of objects or fe ures prese t n the image. • , Positional and Rela ional Context: This aspect refers to the model’s ability to understand the position and relationship of objects or elements within an image in relation to each other and their surroundings. • h Color and Appearance: Questions regarding the color of certain objects or elements. • Ô Structural and Physical Characteristics: This category involves the model’s ability to identify and analyze the physical attributes and structural features of objects in an image. • k Text: Questions related to text or symbols present in the image. • Vi wpoin and Perspectiv : Questions concerning the per-
13
Image Size Par ms (M) IN-1k Zer Shot ☼ Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2242 427. 6 5. 5 13. 3 13. 20. 0 20. 13. 3 53. 20. 0 6. 7 13. 3 9 OpenAI ViT-L-14 [43] 362 427. 9 6. 6 0. 0 20. 0 40. 20. 6. 7 20. 0 33. 3 6. 7 33. 3 20 0 SigLIP ViT-SO-14 [66] 2242 877. 4 82. 0 26. 7 20. 0 53. 3 40. 0 20. 66. 7 40. 0 20. 53. 3 37 8 SigLIP ViT-SO-14 [66] 3842 878. 0 3. 1 20. 0 26. 7 60. 0 33. 3 13. 66. 7 33. 3 26. 7 53. 3 37 0 DFN ViT-H-14 [10] 2242 986. 1 83. 4 20. 0 26. 7 73. 3 26. 7 26. 7 66. 7 46. 7 13. 3 53. 39 DFN ViT-H-14 [10] 3782 986. 7 84. 4 13. 3 20. 0 53. 3 33. 26. 7 66. 7 40. 0 20. 40. 34 8 MetaCLIP ViT-L-14 [62] 2242 4 7. 6 9. 2 13. 3 6. 7 66 7 6. 7 33. 3 46. 7 20. 0 6. 7 13. 3 2 7 MetaCLIP ViT-H-14 [62] 2242 986. 1 80. 6 6. 7 13. 3 60. 0 13. 3 6. 7 53. 3 26. 7 13. 3 33. 25 2 EVA01 ViT-g-14 [54] 2242 1136. 4 78. 5 6. 7 26. 7 40. 0 6. 7 13. 3 66. 7 13. 3 13. 20. 0 3 EVA02 ViT-bigE-14+ [54] 2242 5044. 9 82. 0 13. 3 20. 0 66. 7 26. 7 26. 7 66. 7 26. 7 20. 0 33. 3
Table 1. Performance of various CLIP based models on differe t visual patterns in MMVP-VL benchmark. Models scaled up in r solut on show minimal i pr vement, wher as a slight advantage is observed wh n scaling up the network. For each
visual pattern, Imag Net-1k Zero-shot accuracy nd MMVP average, we use light gray to highlight the best performance. For most of the visual p tterns, all CLIP-based methods show str ggle, as evid nt from the scores. We use symbols for visual patt rns due to space limit: ☼: Or entation and Direction, Û: Presence of Spe ific Features, L: State and Condition, : Quantity and Count, ,: Positional and
Relational Context, h: Color and Appe rance, Ô: Structural and Physical C ara teristics, k: Texts, : Viewpoint and Perspective. 0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1. 5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance on visual patterns. If CLIP performs oorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual pattern. tion between the underp rformance of CLIP and MLLMs’ visual incap bility? To explore this, we categorize questions from MMVP into these visual patterns summarized and calculate ea h MLLM’s performance on these patterns. In Figure 6, we plot CLIP’s performance and MLLMs’ performance for each visu l pattern. Whe the CLIP vision encoder u derperforms on a certain visual pattern, the MLLM tends to exhibit similar shortcomings. Ope - ourc models such a LLaVA 1. 5 [30] and Ins ructBLIP [8] that explicitly use the CLIP vision encoder display a strong correlation in performance. Further, we calculate the Pearson Co relation Coefficient betw en the CLIP model and MLLM’s performance on each visu l patt n. Results show hat LLaVA 1. 5 and InstructBLIP all possess a coefficient score greate than 0. 7. This high score indicates a strong correlation that weaknesses in vi ual pattern recognition in the CLIP model are transferred to MLLMs. More details on the Pearson Cor elation Coefficient can be fou d in Appendix C. 4. Mixture-of-Features (MoF) for MLLM Based on our expl ration in earli r sections, a natural question arises: If open-sourc d MLLM’s visual hortcomings come from the CLIP vision encoder, how do we build a more co petent visual encoder? In this section, we take initial steps to answer the ques ion by studying Mixture-
of-Features (MoF). We start with ddit ve MoF that mixes
CLIP features and vision- nly SSL model featur s. Result
show that each encoder presents unique dvantages and lim-
itations when employed as the pretrained model in MLLM (Section 4. 2). We subsequently propose Interleaved MoF that integr es he features from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model’s ability to fo low instructio (Section 4. 3). 4. 1. Experiment Setti g
We adopt LLaVA [30, 31] as the fr mework to study vis al encoders in MLLM. L aVA uses a pretrained CLIP encoder a d trains an dapter to align visual tokens with language tokens in th M. (See left sid of Figure 7). We use DINOv2 [42] as the vision-only SSL model in our work because it is currently the most scalable vision-on y model. Our exploration includes the use of two visual enco rs: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ens e con ist nt and fair compa is ns, we train and finetune our model with the same experim nt setting in LLaVA. We include the addition l experim ntal details in Appendix A. 4. 2. Additive MoF
We add a pretrained DINOv2 encoder i to MLLM and mix th CLIP pretrained ncoder with it. We use a coefficient α to control the portion of CLIP features and 1 − α to control the amount of DINOv2 features and line rly add them
6
Questionnaire Progress: 6/300 Can you see the key "Z" in the image? Yes No This question is not good Answers are too ambiguous Back Next Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question. 74 76 78 80 82 84 86 IN-1k ZeroShot Performance 20 25 30 35 40 Av er ag e Pe rfo rm an ce o n M M VP OpenAI ViT-L-14 OpenAI ViT-L-14 SigLIP ViT-SO-14 SigLIP ViT-SO-14 DFN ViT-H-14 DFN ViT-H-14 MetaCLIP ViT-L-14 MetaCLIP ViT-H-14 EVA01 ViT-g-14 EVA02 ViT-bigE-14+
Figure 9. Correlation b tween ImageNet-1k Zero-shot and MM P-VLM avera e. The area of each bubble corresponds to the model’s number of parameters. A higher ImageNet-1k zeroshot performance does not necessarily imply superior performance in MMVP-VLM. in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentuates the value of MMVP-VLM as an evaluation metric, which probes into visual patterns such as orientation – aspects that are pivotal for downstream tasks and go beyond what is captured by ImageNet accuracy alone. D. Visual Patterns for LIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models. • ☼ Orientation and Direction: Questions about the direction something is facing or moving, such as the direction the dog or duck is facing, or the orientation of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the image. • L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet. • Quantity and Count: Questions about the number of objects or features present in the image. • , Positional and Relational Context: This aspect refers to the model’s ability to under tand the positio and relationship of bjects or el ments within an im ge in relation to each other
and their surroundings. • h Color and Appearance: Questions regarding th color of
certain objects or elements. • Ô Structural and Physical Characteristics: This category
in olves the model’s ability to identify and an lyze the physical att ibutes and structur l featur s of objects in an image. • k Text: Que tions related to text or symbols present in the image. • Viewpoint and Perspective: Questions concerning the per-
13
Questionnaire Progress: 6/300 Can you see the key "Z" in the image? Yes No This question is not good Answ rs are too ambiguous Back Next Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question. 74 76 78 80 82 84 86 IN-1k ZeroShot Performance 20 25 30 35 40 Av er ag e Pe rfo rm an ce o n M M VP OpenAI ViT-L-14 OpenAI ViT-L-14 SigLIP ViT-SO-14 SigLIP ViT-SO-14 DFN ViT-H-14 DFN ViT-H-14 MetaCLIP ViT-L-14 MetaCLIP ViT-H-14 EVA01 ViT-g-14 EVA02 ViT-bigE-14+ Figure 9. Correlation between ImageNet-1k Zero-shot and MMVP-VLM average. The area of each bubble corresponds to the model’s number of parameters. A higher ImageNet-1k zeroshot performance does not necessarily imply superior performance in MMVP-VLM. in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentuates the value of MMVP-VLM as an evaluati n metric, which probes into visual patt ns such as orientation – spects that are pivotal for d wnst eam tasks and go beyond what is captured by ImageNet accuracy alone. D. Visual Patterns for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models. • ☼ Orien ation and Direction: Questions about the direction something is facing or m ving, such as t direction the dog or duck is facing, or the orienta io of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the image. • L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet. • Quantity an Count: Questions about the number of objects or features present in the image. • , Positional and Relational Context: This aspect refers to the m del’s ability to understand the position and relationship of objects or elements within an image in relation to each other and their surroundings. • h olor a d Appearance: Questions r garding the color of certain objects or elem ts. • Ô St uctural and Physical Characteristics: This category inv lves the mod l’s ability to de tify and analyze the physical attri u es a d structural features of objects in an image. • k Text: Questions relat d to text or symbols pr sent in the m ge. • Viewpoi t and Perspectiv : Questions concerning the per-
13
Image Size Params (M) IN-1k ZeroShot ☼ Û L , Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2242 427. 6 75. 5 13. 3 13. 3 20. 0 20. 53. 20. 0 6. 7 13. 3 9. OpenAI ViT-L-14 [43] 3362 427. 9 76. 6 0. 0 20. 0 40. 0 20. 6. 7 20. 0 33. 3 6. 7 33. 3 20. 0 SigLIP ViT-SO-14 [66] 2242 877. 4 82. 0 6. 7 20. 0 53. 3 40. 0 20. 66. 7 40. 0 20. 53. 3 37. 8 SigLIP ViT-SO-14 [66] 3842 878. 0 83. 1 2. 0 26. 7 60. 0 33. 3 13. 66. 7 3. 3 26. 7 53. 3 37. 0 DFN ViT-H-14 [10] 2242 986. 1 83. 4 20. 0 26. 7 73. 3 26. 7 26. 7 66. 7 46. 7 13. 3 53. 39. DFN ViT-H-14 [10] 3782 986. 7 4. 4 3. 3 20. 0 53. 3 33. 26. 7 66. 7 40. 0 20. 40. 34. 8 MetaCLIP ViT- -14 [62] 2242 427. 6 79. 2 13. 3 6. 7 66. 7 6. 7 33. 3 46. 7 20. 0 6. 7 13. 3 2. 7 MetaCLIP ViT-H-14 [62] 2242 986. 1 80. 6 6. 7 13. 3 60. 0 13. 3 6. 7 53. 3 26. 7 13. 3 3. 25. 2 EVA01 ViT-g- 4 [54] 2242 1136. 4 78. 5 6. 7 2. 40. 0 6. 7 13. 3 66. 7 13. 3 13. 20. 0 3. EVA02 ViT-bigE-14+ [54] 2242 50. 9 82. 0 13. 3 20. 0 66. 7 26. 7 26. 7 66. 7 26. 7 20. 0 33. 3. Table 1. Performance of va i us CLIP based mod ls on different visual patt ns in MMVP-VLM benchmark. Models scaled up in resolution show minimal improveme t, whereas a sligh dvant ge i observed when scaling up the network. For each
visual pattern, ImageNet-1k Zero shot accuracy and MMVP average, w use light gray to high the best performanc. For most f the visual patterns, al CLIP-bas d m thods show struggle, a evident from the scor s. We use symbol for visual patterns due to pac limit: ☼: Orientation a d Direction, Û: Presence of Specific F atures, L: State and Condition, : Quantity and Count, ,: Positional and
Relational Context, h: Color and Appearance, Ô: Structu l and Physical Char cteristics, k: Texts, : Viewpoint and Persp ctiv. 0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1. 5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance on visual patterns. If CLIP performs poorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual patter. tion between the u derperformance of CLIP and MLLMs’ visual incap b lity? To ex lore this, we categorize questions from MMVP into these visual patterns summarized and calculate each MLLM’s p rformance on these pa terns. In Figure 6, we plot CLIP’s performance and MLLMs’ performance for ach visual patt rn. When the CLIP vision encoder underperforms on a certain vi u l pattern, the MLLM tends to xhib t similar shortc mings. Open-s urce models such as LLaVA 1. 5 [30] and Instr ctBLIP [8] that explicitly use the CLIP vision encoder display a trong correlation in perfo m nce. Further, we calculate the P arson Correlation Coefficient between the CLIP model and MLLM’s perf rmance on each visual p tter. Re lts show that LLaVA 1. 5 and InstructBLIP all possess a co fficient scor gr ater than 0. 7. This high score indicates a strong co r lation th weaknesses i visual patt rn recog ition in the CLIP model are transferred to MLLMs. More details on he Pearson Co relation Coefficient can be fou d in Appendix C. 4. Mixture-of-F atures (MoF) for MLLM Based on our expl ration in earli r s ctions, a natural question arises: If open-sourced MLLM’ visual shortcomings come from the CLIP vision encoder, how do we build a more co petent visual encoder? In this sec ion, we take initial steps to a swer the question by studying Mixture-
of-Features (MoF). We start with additive MoF th t mixes
CLIP features and vi o -only SSL model featur s. Results
show that each enc der present unique advantages d lim-
itations when employed as th pretrained model in ML M (Sectio 4. 2). We subsequently propo Inter eaved MoF that integrates the f atur s from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model’s ability to follow instruc ions (Section 4. 3). 4. 1. Experiment Setting
We adopt LLaVA [30, 31] as the framewo k to study visual ncoders in MLLM. L aVA uses a pretrained CLIP encoder and trains an d pter to align visual tokens with langu g tokens in the LLM. (See left side o Figure 7). We use DINOv2 [42] as the vision-only SSL model i our work because it is currently th mos scal ble vision-only m de. Our exploration includes the use of two visual encoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To en sur consist nt and fair compa is ns, we train and fi etune our model with the same experim nt setting in LLaVA. We include the a di ion l xperim ntal details in Appen ix A. 4. 2. Additive MoF
We add a pretrained DINOv2 encoder i t MLLM and mix the CLIP pretrained encode with it. We us a coefficient α t cont ol the p rtion of CLIP eatures and 1 − α to control the amount of DINOv2 features and line rly add them
6
MMVP Averag
OpenAI ViT-L-14 (Radford et al. , 2021) 224 2 427. 6M 13. 3 13. 3 20. 0 20. 0 13. 3 53. 3 20. 0 6. 7 13. 3 19. 3
OpenAI ViT-L-14 (Radford et al. , 2021) 336 2 427. 9M 0. 0 20. 0 40. 0 20. 0 6. 7 20. 0 33. 3 6. 7 33. 3 20. 0
SigLIP ViT-SO-14 (Zhai et al. , 2023) 224 2 8 7. 4M 6. 7 20. 0 53. 3 40. 0 20. 0 6. 40. 0 20. 0 53. 3 37. 8 SigLIP ViT-SO-14 (Zhai et al. , 2023) 384 2 878. 0M 20. 0 26. 7 60. 0 33. 3 13. 3 66. 7 33. 3 26. 7 53. 3 37. 0
DFN ViT-H-14 (Fang et al. , 2023) 224 2 986. 1M 20. 0 26. 7 73. 3 26. 7 26. 7 66. 7 46. 7 13. 3 53. 3 39. 3 DFN ViT-H-14 3782 986. 7M 13. 3 20. 0 53. 3 33 3 26. 7 66. 7 40. 0 20. 0 40. 0 34. 8 MetaCLIP ViT-L-14
(Xu et al. , 2023) 224 2 427. M 13. 3 6. 7 66. 7 6. 7 33. 3 46. 7 20. 0 6. 7 13. 3 23. 7
MetaCLIP ViT-H-14 (Xu et al. , 2023) 224 2 986. 1M 6. 7 13. 3 60 0 13. 3 6. 53 3 26. 7 13. 3 3. 25. 2
EVA01 ViT-g-14 (Sun et l. , 2023) 224 2 1136. 4M 6. 7 26. 7 40. 0 6. 7 13. 3 66. 7 13. 3 13. 3 20. 0 23. 0
EVA02 ViT-bigE-14+ (Sun et l. , 2023) 224 2 5044. 9M 13. 3 20. 0 66. 7 13. 3 26. 7 66. 7 26. 7 20. 0 33. 3 33. 3
BLIP (Li et al. , 2022) 2242 223M 13. 3 6. 7 40. 0 20. 0 26. 7 66. 7 46. 7 20. 0 46. 7 31. 9 NDCR (Li et al. , 2023b) 2242 440M 26. 7 40. 0 60. 0 26. 7 13. 3 86. 7 33. 3 20. 0 26. 7 37. 0 ContextBLIP (Our ) 2242 240M 26. 7 46. 7 60. 0 40. 0 46. 7 93. 3 40. 0 20. 0 53. 3 47. 4
Table 16: Comparison with various VLMs on different visual patterns in MMVP-VLM benchmark. The best and second-best results are marked in bold and underline. We identify 9 visual patterns:
Questionnaire Progress: 6/300
Can you see the key "Z" in the image?
Yes No
This question is not good Answers are too ambiguous
Back Next
Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question. 74 76 78 80 82 84 86 IN-1k ZeroShot Performance
20
25
30
35
40
Av er
ag e
Pe rfo
rm an
ce o
n M
M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14
SigLIP ViT-SO-14
DFN ViT-H-14
DFN ViT-H-14
MetaCLIP ViT-L-14 MetaCLIP ViT-H-14
EVA01 ViT-g-14
EVA02 ViT-bigE-14+
Figure 9. Correlation between ImageNet-1k Zero-shot and MMVP-VLM average. The area of each bubble corresponds to
the model’s number of par meter. A high r ImageNet-1k z r -
shot performance does not necessarily imply superior performance
in MMVP-VLM. in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentua es the value of MMVP-VLM as an evalua ion metric, which probes into visual patterns such as orientation – asects that are pivotal for downstream tasks and go beyond what is captured by ImageNet accuracy alone. D. Visual Patterns for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models. • ☼ Orientation and Direction: Questions about the direction
something is facing or moving, such as the direction the dog or duck is facing, or the orientation of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the image. • L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the win or if t e ground is wet. • Quantity and Count: Questions about t e number of objects or features present in the image. • , Positional and Relational Context: This aspect refers to the
model’s ability to understand the position a d relationship of
objects or elements within an image in relation to each other
and their surroundings. • h Color nd Appear nce: Questions regarding the color of
certain objects or elements. • Ô Structural and Physical Characteristics: This category involves th model’s ability to identify and analyze the physical attributes an structural features of objects in an image. • k Text: Questions related to text or symbols present in the image. • Viewpoint and Perspective: Questions concerning the per-
13
: Orientation and irection,Image
Siz Params (M) IN-1k ZeroShot ☼ Û L , h Ô k MMVP Average
OpenAI ViT-L-14 [43] 2242 427. 6 75. 5 13. 3 13. 3 20. 0 20. 0 13. 3 53. 3 20. 0 6. 7 13. 3 19. 3 OpenAI ViT-L-14 [43] 3362 427. 9 76. 6 0. 0 20. 0 40. 0 20. 0 6. 7 20. 0 33. 3 6. 7 33. 3 20. 0 SigLIP ViT-SO-14 [66] 2242 877. 4 82. 0 26. 7 20. 0 53. 3 40. 0 20. 0 66. 7 40. 0 20. 0 53. 3 37. 8 SigLIP ViT-SO-14 [66] 3842 878. 0 83. 1 20. 0 26. 7 60. 0 33. 3 13. 3 66. 7 33. 3 26. 7 53. 3 37. 0 DFN ViT-H-14 [10] 2242 986. 1 83. 4 20. 0 26. 7 73. 3 26. 7 26. 7 66. 7 46. 7 13. 3 53. 3 39. 3 DFN ViT-H-14 [10] 3782 986. 7 84. 4 13. 3 20. 0 53. 3 33. 3 26. 7 66. 7 40. 0 20. 0 40. 0 34. 8 MetaCLIP ViT-L-14 [62] 2242 427. 6 79. 2 13. 3 6. 7 66. 7 6. 7 33. 3 46. 7 20. 0 6. 7 13. 3 23. 7 MetaCLIP ViT-H-14 [62] 2242 986. 1 80. 6 6. 7 13. 3 60. 0 13. 3 6. 7 53. 3 26. 7 13. 3 33. 3 25. 2
EVA01 ViT-g-14 [54] 2242 1136. 4 78. 5 6. 7 26. 7 40. 0 6. 7 13. 3 66. 7 13. 3 13. 3 20. 0 23. 0
EVA02 ViT-bigE-14+ [54] 2242 5044. 9 82. 0 13. 3 20. 0 66. 7 26. 7 26. 7 66. 7 26. 7 20. 0 33. 3 33. 3
Table 1. Performance of various CLIP based models on different visual patterns in MMVP-VLM benchmark. Models
scaled up in resolution show minimal improvement, whereas a slight advantage is observed when scaling up the network. For each
visual pattern, ImageNet-1k Zero-shot accuracy and MMVP average, we use light gray to highlight the best performance. For most of the visual patterns, all CLIP-based methods show struggle, as evident from the scores. We use symbols for visual patterns due to space limit: ☼: Orientation and Direction, Û: Presence of Specific Features, L: State and Condition, : Quantity and Count, ,: Positional and Relational Context, h: Color and Appearance, Ô: Structural and Physical Characteristics, k: Texts, : Viewpoint and Perspective. 0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1. 5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance n visual patterns. If CLIP performs poorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual pattern. tion between the underperformance of CLIP and MLLMs’ visual incapability? To explore this, we categorize questions from MMVP into these visual patterns summarized and calculate each MLLM’ pe forman e on these pattern. In Figure 6, we plot CLIP’s performance and MLLMs’ performance for each visual pattern. When the CLIP vision encoder underperforms on a certain visual pattern, the MLLM tends to exhibit similar shortcomings. Open-source models such as LLaVA 1. 5 [30] and InstructBLIP [8] that explicitly use the CLIP vision encoder display a strong correlation in performance. Further, we calculate the Pearson Correlation Coefficient between the CLIP model and MLLM’s performance on each visual pattern. Results show that LLaVA 1. 5 and InstructBLIP all possess a coefficient score greater than 0. 7. This high score indicates a strong correlation that weaknesses in visual pattern recognition in the CLIP model are transferred to MLLMs. More details on the Pearson Correlation Coefficient can be found in Appendix C. 4. Mixture-of-Features (MoF) for MLLM Based on our exploration in earlier sections, a natural question arises: If open-sourced MLLM’s visual shortcomings come from the CLIP vision encoder, how do we build a more competent visual encoder? In this section, we take initial steps to answer the question by studying Mixtureof-Features (MoF). We start with additive MoF that mixes CLIP features and vision-only SSL model features. R sults show that each encoder presents unique advantages and limitatio s when em lo ed as the pretrain d model in MLLM (Section 4. 2). We subsequently propose Interleaved MoF that integrates the features from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model ability to follow instru tions (Section 4. 3). 4. 1. Experiment Setting
We adopt LLaVA [30, 31] as the framework to study visual encoders in LLM. LLaVA uses a pretrained CLIP encoder and trains an adapter to align visual tokens with language tokens in the LLM. (See left side of Figure 7). We use DINOv2 [42] as the vision-only SSL model in our work because it is currently the most scalable vision-only model. Our exploration includes the us of two visual ncoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ensure consistent and fair comparisons, we train and finetune our model with the same experiment setting in LLaVA. We include the additional experimental details in Appendix A. 4. 2. Additive MoF
We add a pretrained DINOv2 encoder into MLLM and mix the CLIP pretrained encoder with it. We use a coefficient α to control the portion of CLIP features and 1 − α to control the amount of DINOv2 features and linearly add them
: Presence of Specific Features, Image Size Params (M) IN-1k ZeroShot ☼ Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2242 427. 6 75 13. 3 13. 3 20. 0 20. 0 13. 3 53. 3 20. 0 6. 7 13. 3 19. 3 OpenAI ViT-L-14 [43] 3362 427. 9 76. 6 0. 0 20. 0 40. 0 20. 0 6. 7 20. 0 33. 3 6. 7 33. 3 20. 0 SigLIP ViT-SO-14 [66] 2242 877. 4 82. 0 26. 7 20. 0 53. 3 40. 0 20. 0 66. 7 40. 0 20. 0 53. 3 37. 8 SigLIP ViT-SO-14 [66] 3842 878. 0 83. 1 20. 0 26. 7 60. 0 33. 3 13. 3 66. 7 33. 3 26. 7 53. 3 37. 0 DFN ViT-H-14 [10] 2242 986. 1 83. 4 20. 0 26. 7 73. 3 26. 7 26. 7 66. 7 46. 7 13. 3 53. 3 39. 3 DFN ViT-H-14 [10] 3782 986. 7 84. 4 13. 3 20. 0 53. 3 33. 3 26. 7 66. 7 40. 0 20. 0 40. 0 34. 8 MetaCLIP ViT-L-14 [62] 2242 427. 6 79. 2 13. 3 6. 7 66. 7 6. 7 33. 3 46. 7 20. 0 6. 7 13. 3 23. 7 MetaCLIP ViT-H-14 [62] 2242 986. 1 80. 6 6. 7 13. 3 60. 0 13. 3 6. 7 53. 3 26. 7 13. 3 33. 3 25. 2 EVA01 ViT-g-14 [54] 2242 1136. 4 78. 5 6. 7 26. 7 40. 0 6. 7 13. 3 66. 7 13. 3 13. 3 20. 0 23. 0
EVA02 ViT-bigE-14+ [54] 2242 5044. 9 82. 0 13. 3 20. 0 66. 7 26. 7 26. 7 66. 7 26. 7 20. 0 33. 3 33. 3
Table 1. Performance of various CLIP based models on different visual patterns in MMVP-VLM benchmark. Models scaled up in resolution sh w minimal improvement, whereas a slight advantage i observed when scaling p the netw rk. For each
visual pattern, ImageNet-1k Zero-shot accuracy and MMVP average, we use light gr y to highlight the be t perfor ance. For mos of the visual patterns, all CLIP-based methods show struggle, as evident from the scores. We use symbols for visual patterns due to space limit: ☼: Orientation and Direction, Û: Presence of Specific Features, L: State and Condition, : Quantity and Count, ,: Positional and Relational Context, h: Color and Appearance, Ô: Structural and Physical Characteristics, k: Texts, : Viewpoint and Perspective. 0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1. 5 Ge ini GPT4
Figure 6. CLIP and MLLM’s performance on visu l patter s. If CLIP performs poorly n a visual patter such s “ ☼ orien ation”, MLLMs also underperform on the visual pattern. tion between the underperformance of CLIP and MLLMs’ visual incapability? To explore this, we categorize questions from MMVP in o th se visu l patterns sum arized and calculate each MLLM’s performance on these patterns. In Figu e 6, we plot CLIP’s performance and MLLMs’ performance for each visual pattern. hen the C IP vision encoder underperforms on a certain visual pattern, the MLLM tends to xhibit similar hortcomings. Open-source models such as LLaVA 1. 5 [30] and InstructBLIP [8] that explicitly use the CLIP vision e coder dis lay a strong correlation in performance. Further, we calc l te the Pearson Correlatio Coefficient between the CLIP model and MLLM’s performance on each visual pattern. Results show that LLaVA 1. 5 and InstructBLIP all possess a coefficient score greater than 0. 7. This high score indicates a strong correlation that weaknesses in visual pattern recognition in the CLIP model are transferred to MLLMs. More details on the Pearson Correlation Coefficient can be found in Appendix C. 4. ixture-of-Features (MoF) for MLLM Bas d on our xplo ation in earlier sections, a natural question arises: If open-sourced MLLM’s visual shortcomings come from the CLIP vision encoder, how do we build a more competent visual encoder? In this section, we tak initial steps to answer the question by studying Mixtureof-Features (MoF). We start with additive MoF that mixes CLIP features and vision-only SSL model features. Results show that each encoder presents un que advantages and limitations when employed as the pret ained model in MLLM (Section 4. 2). We subsequently propose Interleaved MoF that integrates the features from both CLIP and SSL into MLLM to enhance visual grounding without compromising the model’s ability to follow instructions (Section 4. 3). 4. 1. Experiment Setting
We adopt LLaVA [30, 31] as the framework to study visual encoders in MLLM. LLaVA uses a pretrained CLIP encoder and trains an adapter to align visual tokens with l nguage tokens in the LLM. (See left side f Figure 7). We use DINOv2 [42] as the vis on-only SSL model in our work because it is currently the most calable vision-only model. Our exploration includes the use of two visual encoders: CLIP-ViT-L-14 [43] and DI V2-ViT-L-14 [42]. To ensure consistent and fair comparisons, we train a d finetune our model with the same experiment setting in LLaVA. We include the additional experimental details in Appendix A. 4. 2. Additive MoF
We add a pretrained DINOv2 encoder into MLLM and mix the CLIP pretrained encoder with it. We use a coefficient α to control the portion of CLIP features and 1 − α to control the amount of DINOv2 features and linearly add them
6
: State and Condition,
Orientation and Direction a rabbit facing right a rabbit facing left
Presence of Specific Features tulips no tulips
State and Condition butterfly with wings open butterfly with wings closed
Quantity and Count 1 drink 2 drinks
Positional and Relational Context glasses on the right of the slipper glasses on the left of the slipper Color and Appearance light blue sky dark blue sky Structural Characteristics some fruits cut in half uncut fruits Texts “11:54” “11:59” Viewpoint and Perspective flowers seen from above flowers seen from the side Model chooses the correct image based on the text Model chooses the wrong image based on the text MMVP-VLM Benchmark Figure 5. Examples from MMVP-VLM. MMVP-VLM co sists of image pairs across nine visual patterns. The examples in the figure are from EVA01 ViT-g-14 model [54], one f the largest CLIP models that also fails to choose the right image given the text description. emerged from CLIP-blind pairs (Section 3. 1), (ii) hether thes visual patterns pose challenges for CLIP-based models with massive scaling up (Section 3. 2), and (iii) the correlation between failure patterns in CLIP models and those in MLLMs (Section 3. 3). 3. 1. Visual Patterns in CLIP-blind Pairs
Having identified he CLIP-blind pairs, w summarize systematic visual patterns that the CLIP vision encoders might consistently misinterpret. It is too abstract to directly capture systematic visual patterns in the CLIP-blind pairs. Th refore, we turn to the questions and options from the MMVP benchmark. With these questions, we transform abstract visual patterns in images into clearer, language-based descriptors that are easier to categorize. In this work, we use GPT-4 [41] to categorize general patterns by prompting it with the following:
User I am analyzing an image embedding model. Can you go through the questions and options, trying to figure out some general patterns that the embedding model struggles with? Please focus on the visual features and generalize patterns at ar import nt to vision models [MMVP Questions and Options]
We identify 9 visual patterns: ☼ Orientation and Direction
Û Presence of Specific Features
L State and Condition
Quantity and Count , Positional and Relational Context h Color and Appearance Ô Structural and Physical Characteristics k Text Viewpoint and Perspective These visual patterns suggest that CLIP vision encoders
overly focus on high-level semantic understanding, overlooking intricate details of the visual world. Full descriptions of the visual patterns can be found in Appendix D. 3. 2. The MMVP-VLM Benchmark
CLIP-based models have developed rapidly since the introduction in the first paper [43]. We want to test whether these visual pat erns still impose challenges to the more recent CLIP models [10, 54, 62, 66], which significantly scale up in terms of training data and model size. In doing so, we introduce a new benchmark: MMVP-VLM to systematically study if CLIP models handle this visual pattern well. We distill a subset of questions from the MMVP benchmark into simpler language descriptions and categorize them into visual patterns. To maintain a balanced number of questions for each visual pattern, we add a few questions, if needed, to ensure that each visual pattern is represented by 15 text-image pairs. Examples of pairs are shown in Figure. A pair is deemed correctly answered if the model can accurately match both image-text combinations. We evaluate MMVP-VLM on a variety of CLIP models [10, 43, 54, 62, 66]. These models vary in aspects like size, training data, and methodology. As evidenced in Table 1, increasing network size and training data only aids in identifying two visual patterns – “color and appearance” and “state and condition”. The rest of the visual patterns continue to challenge all CLIP-based models. We also find that the ImageNet-1k zero-shot accuracy is not a definitive indic tor of a model’s performance regarding visual patterns. This underscores the necessity for additional evaluation metrics, such as MMVP-VLM, to accurately assess the model’s capabilities in areas beyond image classification. 3. 3. How CLIP’s Errors Affect MLLMs
After analyzing the visual patterns that CLIP models struggle with, we pose the following question: Is there a correla-
5
: Quantity and Count,
Questionnair Progress: 6/300 Can you see the key "Z" in the image?
Yes No
This question is not good Answers are too ambiguous
Back Next
Figure 8. Example of user study interface. The questi s in th user study re randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question. 74 76 78 80 82 84 86 IN-1k ZeroShot Performance
20
25
30
35
40
Av er
ag e
Pe rfo
rm an
ce o
n M
M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14
SigLIP ViT-SO-14
DFN ViT-H-14
DFN ViT-H-14
MetaCLIP ViT-L-1 MetaCLIP ViT-H-14
EVA01 ViT-g-14
EVA02 ViT-bigE-14+
Figure 9. Correlation between ImageNet-1k Zero-shot and
MMVP-VLM aver ge. The area of each bubble corresponds to the model’s number of parameters. A higher ImageNet-1k zeroshot performance does not necessarily imply superior performance in MMVP-VLM. in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentuates the value of MMVP-VLM as an evaluation metric, which probes into visual patterns such as orientation – as-
pects that are pivotal f r downstream tasks and go beyond wha is
captured by ImageNet accuracy alone. D. Visual Patterns for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models. • ☼ Orientation and Direction: Questions about the direction something is facing or moving, such as the direction the dog or duck is facing, or the orientation of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of ertain elements or features in the image. • L St te and Condition: Questions that pertain to the state or condition of an object, such as whether flag is blowing in the wind or if the ground is wet. • Quantity and Count: Qu stions about the number of objects
or features present n the image. • , Positional and Relational Cont xt: This aspect refers to the model’s ability to understand the position and relationship of objects or elements within an image in relation to each other and their surroundings. • h Color and Appearance: Questions regarding the color of certain objects or elements. • Ô Structural and Physical Characteristics: This category involves the model’s ability to identify a d analyze the physical attributes and structural features of objects in an image. • k Text: Questions related to text or ymbols present in the
image. • Viewpoint and Perspective: Questions concerning the per-
13
: Positi l and Relational Context,
Image Size Params (M) IN-1k ZeroShot ☼ Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 22 2 427. 6 75. 5 13. 3 13. 3 20. 0 20. 0 13. 3 53. 3 20. 0 13. 3 19. 3 OpenAI ViT-L-14 [43] 3362 427. 9 76. 0. 0 20. 0 40. 0 20. 0 6. 7 20. 0 33. 3 6. 7 33. 3 20. 0 SigLIP ViT-SO-14 [66] 2242 877. 4 82. 0 26. 7 20. 0 53. 3 40. 0 20. 0 66. 7 40. 0 20. 0 53. 3 37. 8 SigLIP ViT-SO-14 [66] 3842 878. 0 83. 1 20. 0 26. 7 60. 0 33. 3 13. 3 66. 7 33. 3 26. 7 53. 3 37. 0 DFN ViT-H-14 [10] 2242 986. 1 83. 4 20. 0 26. 7 73. 3 26. 7 26. 7 66. 7 46. 7 13. 3 53. 3 39. 3 DFN ViT-H-14 [10] 3782 986. 7 84. 4 13. 3 20. 0 53. 3 33. 3 26. 7 66. 7 40. 0 20. 0 40. 0 34. 8 MetaCLIP ViT-L-14 [62] 2242 427. 6 79. 2 13. 3 6. 7 66. 7 6. 7 33. 3 46. 7 20. 0 6. 7 13. 3 23. 7 MetaCLIP ViT-H-14 [62] 2242 986. 1 80. 6 6. 7 13. 3 60. 0 13. 3 6. 7 53. 3 26. 7 13. 3 33. 3 25. 2
EVA01 ViT-g-14 [54] 2242 1136. 4 78. 5 6. 7 26. 7 40. 0 6. 7 13. 3 66. 7 13. 3 13. 3 20. 0 23. 0
EVA02 ViT-bigE-14+ [54] 2242 5044. 9 82. 0 13. 3 20. 0 66. 7 26. 7 26. 7 66. 7 26. 7 20. 0 33. 3 33. 3
Table 1. Performance of various CLIP based models n different visual patterns in MMVP-VLM benchmark. Mod l
scaled up in resolution show minimal improvement, whereas a slight advantage is observe when scaling up the network. For each
visual pattern, ImageNet-1k Zero-shot accuracy and MMVP average, we use light gray to highlight the be t performance. For most of the visual patterns, all CLIP-based methods show struggle, as evident from the scores. We use symbols for visual patterns due to space limit: ☼: Orientation and Direction, Û: Presence of Specific Features, L: State and Condition, : Quantity and Count, ,: Positional and Relational Context, h: Color and Appearance, Ô: Structural and Physical Characteristics, k: Texts, : Viewpoint and Perspective. 0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1. 5 Gemini GPT4
Figure 6. CLIP and MLLM’s performance on visual patterns. If CLIP performs poorly on a visual pattern such as “ ☼ orientation”, MLLMs also underperform on the visual pattern. tion between the underperformance of CLIP and MLLMs’ visual incapability? To explore his, w categorize questions from MMVP into these visual patter s summ riz d and calculate each MLLM’s performance on these patterns. In Figure 6, we plot CLIP’s performance and MLLMs’ performance for each visual pattern. When th CLIP vision encoder underperforms on a certain visual pat n, the MLLM tends to exhibit similar shortcomings. Ope -source models such as LLaVA 1. 5 [30] and Instru tBLIP [8] that explicitly use the CLIP vision encoder display a stro g correlation in performance. Further, we calculate the Pearson Correla ion Coefficient between the CLIP model and MLLM’s performance on each visual pattern. Results show that LLaVA 1. 5 and InstructBLIP all possess a coefficient score greater t an 0. 7. This high score indicates a strong correlation that weaknesses in visual pattern recognition in the CLIP mo el are transferred to MLLMs. More details on the Pearson Corr - lation Coefficient can be found in Appendix C. 4. Mixture-of-Features (M F) for MLLM Based on our exploration in earlier sections, a n tural question a i es: I open-sourced MLLM’s visual shor comings come from the CLIP vision encoder, how d we build a more competent visual e oder? I this sec on, we take initial steps to answer the question by studying Mixtureof-Features (MoF). We start with dditive MoF th mixes CLIP features and vision-only SSL model features. Results show that each encoder pr sents unique advan age and li - itations whe employed as the pretrain d model i MLLM (Section 4. 2). We subsequently propose Interleaved MoF that integrates the features from both CLIP and SSL into MLLM to enhance visual grounding without co promising the model’s ability to follow instructions (Section 4. 3). 4. 1. Experiment Setting
We adopt LLaVA [30, 31] as the framework to stu y v - sual encoders in MLLM. LLaVA uses a pretrained CLIP enc der and trains a adapter to align visual tok ns with language tokens in the LLM. (See left side of Figure 7). W use DINOv2 [42] as the vision-only SSL mo el in our work because it is currently the most scalable vision-only model. Our exploration includes the use of two visual encoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ensure consis ent and fair comparisons, we train and finetune our model with the same experiment setting in LLaVA. We include the additional experimental details in Appendix A. 4. 2. Additive MoF
We add a pretrained DINOv2 encod r into MLLM and mix the CLIP pretrai ed ncoder with it. We use a coefficient α to c ntrol the portion of CLIP features and 1 − α to control the amount of DINOv2 features and linearly add them
6
: Color and Appearance,
Questionnaire Progress: 6/300
Can you see the key "Z" in the image?
Yes No
This question is not good Answers are too ambiguous
Back Next
Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose answers for the VQA questions as well as potential concerns for the VQA question. 74 76 78 80 82 84 86 IN-1k Zer Shot Perform nce
20
25
30
35
40
Av er
ag e
Pe rfo
rm an
ce o
n M
M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14
SigLIP ViT-SO-14
DFN ViT-H-14
DFN ViT-H-14
MetaCLIP ViT-L-14 MetaCLIP ViT-H-14
EVA01 ViT-g-14
EVA02 ViT-bigE-14+
Figure 9. Correlation bet een ImageNet-1k Zero-shot an MMVP-VLM average. The area of each bubble corresponds to
the model’s number of parameters. A higher ImageNet-1k zero-
shot p rformance oes n t necessarily imply superior performance
in MMVP-VLM. in models with superior ImageNet-1k Zero-shot performance, this trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentuates the value of MMVP-VLM as an evaluation metric, which probe into visual patterns such as orientation – aspects that are pivotal for downstream tasks and go beyond what is captured by ImageNet accuracy alone. D. Visual Patter s for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models. • ☼ Orientation and Direction: Questions about the direction something is facing or moving, such as the direction the dog or duck is facing, or the orientat n of the school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of c rtain elements or f atures in the image. • L State and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet. • Quantity and Count: Questions about the number of objects or features pres nt in the imag. • , Positional and Relational Context: This aspect refers to the model’ ability to understand the position and relationship of
objects or elements within an image in relation to each other
and their surroundings. • h Color and Appearance: Questions regarding the color of certain objects or elements. • Ô Structural and Physical Characteristics: This category involves the model’s ability to identify and analyze th physical attributes and structural features of objects in an image. • k Text: Questions related to text or symbols present in the image. • Viewpoint and Perspective: Questions concerning the per-
13
: Structural and Physical Charac eristic ,
Questionnaire Progress: 6/300 Can you see the key "Z" in the image?
Yes No
This question is not good Answ rs are too ambigu us
Back Next
Figure 8. Example of user study interface. The questions in the user study are randomly shuffled to avoid any potential bias. Users choose nswers for the VQA question as well as potential concerns for the VQA questio. 74 76 78 80 82 84 86 IN 1k ZeroShot Performance
20
25
30
35 40 Av er ag e Pe rfo rm an ce o n M M VP
OpenAI ViT-L-14 OpenAI ViT-L-14
SigLIP ViT-SO-14
SigLIP ViT-SO-14
DFN ViT-H-14
DFN ViT-H-14
MetaCLIP ViT-L-14 MetaCLIP ViT-H-14
EVA01 ViT-g-14
EVA02 ViT-bigE-14+
Figure 9. Correlation between Im g Net-1k Zero-shot and MMVP-VLM average. The area of each bubble corresponds to the model’s number of param ers. A higher ImageNet-1k z roshot performance does not necessarily imply superior performance in MMVP-VLM. in models with superior ImageN t-1k Z ro-shot performance, this
trend does not necessarily hold for MMVP-VLM accuracy. This distinction accentuates th value of MMVP-VLM as an valuati m tric, which probes into visual pa terns suc as orientation – aspects that are pivotal for downstrea tasks and go beyond what is captured by ImageNet accuracy alone. D. Visual Patterns for CLIP Here, we provide the full description of visual patterns that pose challenges to all CLIP-based models. • ☼ Orientation and Direction: Questions about the direction something is facing or moving, such as the direction the dog or duck is facing, or the orientation of th school bus. • Û Presence of Specific Features: Questions that focus on the existence or non-existence of certain elements or features in the i age. • L S ate and Condition: Questions that pertain to the state or condition of an object, such as whether a flag is blowing in the wind or if the ground is wet. • Quantity and Count: Questions about the number of objects or f atures present i the image. • , Positional and Relational Context: This aspect refers to the model’s ability to understand the position and relationship of objects or elements within an image in relation to each other and their surroundings. • h Color and App a ance: Questions regarding the color of
certain obje ts or el ments. • Ô Structural and Physical Characteristics This category
involves the model’s ability to identify and analyze the physical attributes and structural features of objects in an image. • k Text: Questions related to text or symbols present in the image. • Viewpoint and Perspective: Questions concerning the per-
13
: Text and Image Size Params (M) IN-1k Zer Shot ☼ Û L , h Ô k
MMVP Average
OpenAI ViT-L-14 [43] 2242 427. 6 75. 5 1. 3 13 3 20. 0 20. 0 13. 3 53. 3 20. 0 6. 7 13. 3 19. 3 OpenAI ViT-L-14 [43] 3 62 427. 9 76. 6 0. 20. 0 40. 0 20. 0 6. 7 20. 0 33. 3 6. 7 33. 3 20. 0 SigLIP ViT-SO-1 [66] 2242 877. 4 82. 0 26. 7 20. 0 53. 3 40. 0 20. 0 66. 7 40. 0 20. 0 53. 3 37. 8 SigLIP ViT-SO-14 [66] 3842 878. 0 83. 1 20. 0 26. 7 60. 0 33. 3 13. 3 66. 7 33. 3 26. 7 53. 3 37. 0 DFN ViT-H-14 [10] 2242 986. 1 83. 4 20. 0 26. 7 73. 3 26. 7 26. 7 66. 7 46. 7 13. 3 53. 3 39. 3 DFN ViT-H-14 [10] 3782 986. 7 84. 4 1. 3 20. 0 53. 3 33. 3 26. 7 66. 7 40. 0 20. 0 40. 0 34. 8 MetaCLIP ViT-L-14 [62] 2242 427. 6 79. 2 13. 3 6. 7 66. 7 6. 7 33. 3 46. 7 20. 0 6. 7 13. 3 23. 7 MetaCLIP ViT-H-14 [62] 2242 986. 1 80. 6 6. 7 13. 3 60. 0 13. 3 6. 7 53. 3 26. 7 13. 3 33. 3 25. 2
EVA01 ViT-g-14 [54] 2242 136. 4 78. 5 6. 7 26. 7 40. 0 6. 7 13. 3 66. 7 13. 3 13. 3 20. 0 23. 0
EVA02 ViT-bigE-14+ [54] 2242 5044. 9 82. 0 1. 3 20. 0 66. 7 26. 7 26. 7 66. 7 26. 7 20. 0 33. 3 33. 3
Table 1. Perform nce f various CLIP based odels on ifferent visu l pattern in MMVP-VLM b chmark. Models
caled up in resoluti n show minimal improvem nt, hereas a slight advantage is observed when scaling up the network. For each
visual pattern, Ima eNet-1k Zero-sho accuracy and MMVP average, we use light gray t highlight the best performance. For most of th visual patterns, all CLIP-based meth ds show struggle, as evident from the scores. We use symbols for visual patterns due to space limit: ☼: Ori ntation and Direction, Û: Pres nce of Specific Features, L: State and Condition, : Quantity and Count, ,: Positional and Relational Context, h: Color and Appearance, Ô: Structural and Physical Characteristics, k: Texts, : Viewpoint and Perspective. 0
10
20
30
40
50
60
70
A cc
ur ac
y
CLIP InstructBLIP LLaVA 1. 5 Gemini GPT4
Figure 6. CLIP and MLLM’s performanc on visual patterns. If CLI perf rms poorly on a visual pa tern uch as “ ☼ orientation”, MLLMs also underperform on the visual pattern. tion between the underp rformanc of CLIP and MLLMs’ visual incapability? To ex l re this, w categ riz questions from MMVP into these visu l patterns summarized and calculate each MLLM’s performance on these patterns. In Fig re 6, w plot CLIP’s erformance and MLLMs’ p rfo mance for each visua pattern. When the CLIP vision encoder underperforms on a certain visual pattern, the MLLM tends to exhibit similar shortcomings. Open-source models such s LLaV 1. 5 [30] and I structBLIP [8] at expli itly use the CLIP vision e coder displ y a strong correlation in performance. Further, we calculate the P arson Correlation Coefficient between the CLIP model and MLLM’s performance on each visual pattern. Results show that LLaVA 1. 5 and Instruc BLIP all possess a coefficient score greater th n 0. 7. This high score i dicates a strong correlat on t at weaknesse in visual patter recogniti n th CLIP model are transferred to MLLMs. More details on he Pearson Correlation Coefficient can be found in Appendix C. 4. Mixture-of-Features (MoF) for MLLM Based on our exploration in earlier secti n , a natural question arises: If open-sourced MLLM’s visual shortcomings come from th CLIP vi ion encoder, how do we build a more competent visual encoder? In this section, we take initial steps to answer the question by studying Mixtureof-Features ( oF). We start with additive MoF that mixes CLIP features and vision-only SSL model features. Results show that each encoder p sents u ique dvantag s and limtations wh em l yed as the pretrained model in MLLM (Section 4. 2). We subsequently propose Interleaved MoF that integrates the features fr b th CLIP and SSL into MLLM t enh nce visual groundi g without compromising the m del’s ability to follow instructio s (Section 4. 3). 4. 1. Experiment Setting
W a opt [30, 31] as the framework to study visual encoders in MLLM. LLaVA u es a pretrained CLIP
coder and trains an adapter to align visual tok ns w th language tokens in the LM. (See left side of Figure 7). We use DINOv2 [42] as i i l SSL model in our work beca se it is c rr ntly the most scalable vision-only model. Our exploration includes the use of two visual encoders: CLIP-ViT-L-14 [43] and DINOV2-ViT-L-14 [42]. To ensure consistent and fair comparisons, we train and finetune our model with the same experim t setting in LLaVA. We include the additional experimental details in Appendix A. 4. 2. Additive MoF
We add a p etrained DINOv2 encoder i to MLLM and mix the CLIP pretrained encoder with it. We use a coefficient α to control the po tion of CLIP fe tures and 1 − α to control the amount of DINOv2 features and linearly add them
6
: Viewpoint and Perspective. Method Params All Video Image
Qwen-VL-Chat (Bai et al. , 2023) 7B 9. 1 8. 4 11. 3 Video-LLaVA (Lin et al. , 2023) 7B 11. 3 11. 8 9. 1 Co textBLIP 240M 8. 5 28. 7 81. 2
Table 17: Comparisons with NDCR, QWen-VL-Chat (Bai t al. , 23) and Vide -LLaVA (Lin et al. , 2023) on IMAGECODE: We compare our model with the previous NDCR and two recently releas d MLLMs QWe - VL-Chat (Bai et al. , 2023) and Video-LLaVA (Lin et al. , 2023) on the challenging instances of IMAGECODE. Experimental r sults show t at th propo ed Cont xtBLIP consist ntly p rforms best amo g these baselines. D. 3 Comparison with More MLLMs See Table 17 and Table 18. 16252
Query : Two men facing each other. PBLIP [0. 100, 0. 101, 0. 101, 0. 100, 0. 100, 0. 100, 0. 099, 0. 100, 0. 100, 0. 100] (×) POurs [0. 076, 0. 089, 0. 094, 0. 090, 0. 084, 0. 099, 0. 142, 0. 121, 0. 093, 0. 113] (✓)
Query : The man's right eye is 40% visible. PBLIP [0. 100, 0. 099, 0. 103, 0. 099, 0. 099, 0. 099, 0. 099, 0. 105, 0. 099, 0. 099] (×) POurs [0. 145, 0. 129, 0. 105, 0. 086, 0. 090, 0. 083, 0. 084, 0. 085, 0. 105, 0. 089] (✓)
Figure 5: Zero-shot cases from the test set. Our model has advantages over BLIP in both confidence scores. Query : Two thumbs on egg, no blur, bottom right blue dial has just started to be covered. Key contextual cue : Two thumbs on egg. PBLIP’ [0. 187, 0. 034, 0. 196, 0. 242, 0. 025, 0. 045, 0. 065, 0. 048, 0. 025, 0. 132] (×) POurs’ [0. 086, 0. 036, 0. 412, 0. 22 , 0. 016, 0. 031, 0. 045, 0. 034, 0. 025, 0. 095] (✓)
PBLIP [0. 118, 0. 047, 0. 104, 0. 134, 0. 079, 0. 131, 0. 095, 0. 105, 0. 075, 0. 111] (×) POurs [0. 097, 0. 063, 0. 137, 0. 131, 0. 068, 0. 107, 0. 106, 0. 111, 0. 087, 0. 092] (✓)
Query : The man's face is visible but very blurry. He is punching downward but you can't see any of his elbow because it is blocked by his fist. His fist is visible but blurry. Key contextual cue : The man's face is very blurry. PBLIP’ [0. 011, 0. 117, 0. 141, 0. 39 , 0. 006, 0. 002, 0. 046, 0. 084, 0. 199, 0. 003] (×) POurs’ [0. 021, 0. 113, 0. 194, 0. 215, 0. 008, 0. 001, 0. 051, 0. 2 , 0. 194, 0. 003] (✓)
PBLIP [0. 063, 0. 076, 0. 193, 0. 262, 0. 016, 0. 017, 0. 063, 0. 063, 0. 229, 0. 019] (×) POurs [0. 087, 0. 067, 0. 24 , 0. 207, 0. 014, 0. 008, 0. 069, 0. 126, 0. 168, 0. 014] (✓)
Figure 6: Two cases from the test set after fine-tuning. Our model outperforms BLIP in confidence scores for both compound and simple proposition texts