Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models
Authors: Shizhan Gong
Yankai Jiang
Qi Dou
Farzan Farnia
Venue: ICML 2025 poster (2025)
Keywords: vision-language model
representation learning
CLIP
Primary area: deep_learning->other_representation_learning
TL;DR: Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings

Abstract: Vision-language models, such as CLIP, have achieved significant success in aligning visual and textual representations, becoming essential components of many multi-modal large language models (MLLMs) like LLaVA and OpenFlamingo. However, numerous studies have identified CLIP's limited fine-grained perception as a critical drawback, leading to substantial failures in downstream MLLMs. In contrast, vision-centric foundation models like DINOv2 demonstrate remarkable capabilities in capturing fine details from images. In this work, we propose a novel kernel-based method to align CLIP's visual representation with that of DINOv2, ensuring that the resulting embeddings maintain compatibility with text embeddings while enhancing perceptual capabilities. Our alignment objective is designed for efficient stochastic optimization. Following this image-only alignment fine-tuning, the visual encoder retains compatibility with the frozen text encoder and exhibits significant improvements in zero-shot object recognition, fine-grained spatial reasoning, and localization. By integrating the aligned visual encoder, downstream MLLMs also demonstrate enhanced performance. The code and models are available at https://github.com/peterant330/KUEA.

Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models
Shizhan Gong' Yankai Jiang? 2 Qi Dou
Farzan Farnia
Abstract
notable example is CLIP (Radford et al., 2021) and its vari- ants (Zhai et al., 2023; Sun et al., 2023), which are trained on extensive datasets of paired text and images to link im- ages with their corresponding textual descriptions. These models demonstrate exceptional generalizability and zero- shot performance on various downstream tasks, including classification (Saha et al., 2024), segmentation (Yu et al., 2023), and object detection (Vidit et al., 2023). Beyond functioning as standalone tool, CLIP's vision encoder has been incorporated into several Multi- Large Language Models (MLLMs), such as LLaVA (Liu et al., 2024), Open- Flamingo (Awadalla et al., 2023), BLIP-2 (Li et al., 2023a), and Qwen- VL (Bai et al., 2023), serving as an integral com- ponent for visual feature extraction.
Vision-language models, such as CLIP, have achieved significant success in aligning visual and textual representations, becoming essen- tial components of many multi-modal large lan- guage models (MLLMs) like LLaVA and Open- Flamingo. However, numerous studies have iden- tified CLIP's limited fine-grained perception as a critical drawback, leading to substantial fail- ures in downstream MILMs. In contrast, vision- centric foundation models like DINOv2 demon- strate remarkable capabilities in capturing fine details from images. In this work, we propose novel kernel- based method to align CLIP's vi- sual representation with that of DINOv2, ensur- ing that the resulting embeddings maintain com- patibility with text embeddings while enhancing perceptual capabilities. Our alignment objective is designed for efficient stochastic optimization. Following this image-only alignment fine-tuning, the visual encoder retains compatibility with the frozen text encoder and exhibits significant im- provements in zero-shot object recognition, fine- grained spatial reasoning, and localization. By in- tegrating the aligned visual encoder, downstream MLLMs also demonstrate enhanced performance. The code and models are available https: //ggttuu.bb.g wtt.:/copppeeeratttt. /KUEA.
However, because CLIP is trained with global supervision from image captions, it struggles to learn finer pixel-level details such as color (Thrush et al., 2022) and spatial rela- tionship (Liu et al., 2023; Kamath et al., 2023). This limita- tion affects CLIIP's performance on vision-focused tasks and can also impair the fine-grained perception capabilities of downstream MLLMs. Numerous studies have highlighted these issues (Yuksekgonul al., 2023; Guo et al., 2024). For instance, Jiang et al. (2023) examined feature visualization from deeper layers and found that it emphasizes global im- age properties while neglecting intricate details. Similarly, Tong et al. (2024) reported that current MLLMs struggle with simple visual pattern questions, such as counting, color identification, and viewpoint recognition.
Vision-only self-supervised learning protocols, such as DINO (Caron et al., 2021) and MAE (He et al., 2022), produce vision-centric representations that are highly effec- tive for visual grounding. Researchers are exploring these models to address the inherent limitations of CLIP. Some studies have combined multiple models as vision encoders for MLLMS (Jiang et al., 2023; Tong et al., 2024; Shi et al., 2024; Shen et al., 2024). However, this approach introduces considerable computational overhead. Other methods focus on using vision-only self-supervised learning to fine-tune the CLIP encoder (Wu et al., 2023; Covert et al., 2025), which can enhance its localization abilities. Nevertheless, these fine-tuning techniques risk disrupting the alignment between image and text representations, potentially under-
1. Introduction
Vision-language Models (VLMs) have made significant strides and transformed the field of computer vision. A
'Department of Computer Science and Engineering, The Chinese University of Hong Kong. Hong Kong SAR, China "Shanghai Artificial Intelligence Laboratory, Shanghai, China. Cor- respondence Shizhan Gong <szgong22 @ www.eeeee.eeuu.....e edu.hk>. Yankai Jiang <jiangyankai pww.aepppg.p..o..o.. en>, Qi Dou <qi- dou www.k.ccuuukkk....... edu.hk>, Farzan Farnia farnia wwweececee..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
baseline abigned
LellyOA
DEXC-VE Pure
CLIP
regularization
1540
(OCR)
1s (x).
k2 (X2,,
:
CuP
...
K2 Xn)
Retoocdp stes
aligg
65.00
ZA ONIO
(x))
Refcoco+
ccco
(x))
(2 Xn))
ReACOLO
lava wild
VQA-V2
Figure 1. Main claim of this work. We propose kernel-based alignment framework, which is able enhance the visual representation of CLIP via maage-only fine-tuning, Morcover, the improvement can be transferred to downstream multi modal large language models.
mining CLIP's zero-shot performance. Additionally, there efforts to integrate region-based task Ioss (Zhong al., 2022; Wan et al., 2024) or distillation loss from vision mod- els (Salehi et al., 2023) during CLIP''s training, which re- quires re-training the models and incurs high computational costs. Most critically, all these methods produce visual em- beddings that differ significantly from the original CLIP embeddings, leading to incompatibility with downstream models trained on the original embeddings. Consequently, all downstream models (e.g., MILMM)) would need to be re-tuned from scratch to align visual-text embeddings. The cost makes these solutions impractical in real applications.
fication and dense prediction tasks, without requiring fine- tuning of the text encoder. Subsequently, we integrate the aligned CLIP vision encoder into two pre-trained MLLMs, LLaVA (Liu et al., 2024) and OpenFlamingo (Awadalla et al., 2023), evaluating their performance across several standard VQA benchmarks. This also results in significant enhancements over the original CLIP, even without fine- tuning the large language model (LLM) component. Our main contributions can be summarized as follows.
We introduce a kernel-based alignment method that effec- tively aligns two sets of embeddings while preserving the integrity of the original representation space.
In this paper, we present an innovative approach to address the limitations of CLIP embeddings by aligning them with
We implement the alignment using CLIP and DINOv2. With minimal image-only fine-tuning, the aligned CLIP visual encoder shows substantial improvements while pre- serving compatibility with the text encoder and ensuring zero-shot generalizability.
embeddings from vision-centric models. Our method in-
volves fine-tuning CLIP's vision branch to align with the representations of target models, such as DINOv2 (Oquab et al., 2024), while maintaining compatibility with CLIP's text embeddings, which is totally frozen and untouched (see Fig. 1). Given the significant differences between the fea- ture spaces of target models and CLIP, direct alignment of the visual representations could disrupt the alignment with text representations. Instead, we propose to align the embed- dings in the kernel space, which preserves the integrity of the original feature space while allowing for flexible adjust- ments of similarities among samples based on their visual details. As a result, this alignment could enhance the vision encoder's 's ability to recognize fine- Ene-grained visual patterns. In addition, we design optimization objective that can be handled by stochastic optimization, enabling our frame- work to scale effectively to real-world datasets with minimal computational hardware requirements.
Evaluation across various vision-language benchmarks shows that the enhancements in visual representations can be effectively inherited to downstream MLLMs.
2. Related Work
Kernel Methods in Representation Learning. Kerel methods (Hofmann et al., 2008) are algorithms that use the kemne trick, which allows linear algorithms to work in high-ddmensional feature spaces without explicit data trans- formation. is widely used in deep representation learning. For example, He Ozay (2022) utilized kernel-based met- rics for evaluating the similarity between two embeddings. Several work (Allen-Zhu & Li, 2020; He & Ozay, 2022: Zhou et al., 2024) leveraged kernel functions for knowl- edge distillation. Dehdashtian et al. (2024) proposed to di-bias CLIP's image and text representations in reproduc- ing kernel Hilbert spaces for better faimesss. Kernel methods have also been applied to evaluating the fidelity (Binkowski et al., 2018), diversity (Friedman & Dieng, 2022; Jalali et al.,
We examine the enhancements achieved through the pro- posed alignment in both vision-centric tasks and visual question answering (VQA). Experiments conducted on multiple CleP-bbnc (LAION-AI, 2022) and probing bench (Covert et al., 2025) reveal that the CLIP after align- ment demonstrates improved accuracy in zero-shot classi-
2

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
2023; Rezaei et al., 2024; Ospanov & Farnia, 2024; Jalali et al., 2024; Ospanov al., 2024a;b), and novelty (Zhang et al., 2024a;b) of generative models. Our method can be interpreted as aligning visual representations of CLIP and other vision models in the kernel space, through kernel trick.
with coefficient 7, constant offset c, and degree d:
Ap (x) =- (1x y + c)²²
(I)
We call kernel function normalized if (xx, x) = 1 for any vector E IR A kernel function can be normalized via:
Fine-tuning Vision-Language Models. Fine-tuning of VLMs has been widely applied improve the model per- formance on specific downstream dataset (Zhou et al., 2022: Zhang et al., 2022; Gao et al., 2024) domain-specific applications (Cao et al., 2024; Gong et al., 2024). Be- yond the accuracy gain, studies have also discovered that well-designed fine-tuning scheme can promote some desired properties of the model, such as adversarial robustness (Mao et al., 2022; Schlarmann et al., 2024), fairness (Shen et al., 2023), and visual interpretability (Gong al., 2025). In this work, we leverage fine-tuning to enhance the general visual ability of the VI.Ms. Instead of focusing on single datasets, we show the enhancement exhibit good generalizability.
x(x,))
k(x,y) Vk(xx, ((xxx))k(3 xy.
(2)
Note that for each valid kernel function k. there exists a feature map Φb Rdd -> IR* such that for every input vectors x, y, the following holds:
k(x, y) (x(x), x(y)),
where (--) denotes the inner product in R 8 is usually much greater than d, indicating that é maps the input fea- tures into higher-dimennional kernel space.
3.2. Kernel- Embedding Alignment
Enhancement of CLIP Visual Representations. Several studies have proposed methods to address the limitations of CLIP's visual embeddings and enhance its fine-grained capabilities. For instance, Salehi et al. (2023) trained the CLIP encoder using multi-task losses supervised by pseudo labels generated from other vision encoders. Similarly, Jiang et al. (2023) and Tong et al. (2024) complemented CLIP with DINOv2 as the vision encoder for downstream MLLMs. Shi et al. (2024) and Shen et al. (2024) further expanded this approach to incorporate a wider variety of vision encoders. Currently, there are limited explorations fo- cusing on the fine-tuning phase. Covert et al. (2025) applied masked fine-tuning to CLIP's vision encoder to enhance its localization capabilities. However, their fine-tuned vision encoders became incompatible with the text encoder and downstream LLMs. The method most closely related to ours DIVA ( (Wang et al., 2025), which refines CLIP representa- tions using only images with the help of diffusion models. Comparing to DIVA, our method demands significantly less computation and achieves better sero-shot performance.
CLIP consists of two components: an image encoder fo and text encoder fu. These encoders transform each language-image pair (T;, 1;) into their respective represen- tations fo(Ls), (T;) E Rd. The target model is vi- sion encoder (e.g., DINOv2) that generates the representa- tion (T:) E R2d for the image 4. While CLIP effectively achieves visual-language alignment, struggles to capture fine-grained visual details, an area where the target model excels. As noted by Tong et al. (2024), samples that carry similar semantic meanings but differ in visual details exhibit high similarity in the CLIP feature space, while showing low similarity in the DINOv2 feature space. The value of kernel function is a measure of similarity between two sam- ples. The kernel matrix of the target model reflects how samples are arranged in feature spaces based on their visual similarities, whereas the kernel matrix of CLIP only cap- tures semantic similarities. Consequently, it is natural idea to align the kernel matrix of CLIP with that of the target model, as it can encourage the arrangement of samples in the feature space to better represent their visual patterns, thus alleviating the limitations of CLIP (as illustrated in Fig. 1). Due to the large sample size, directly minimizing the distance between the two kernel matrices can be com- putationally infeasible. To address this, we propose the following optimization objective for alignment fine-tuning:
3. Method
3.1. Kernel Function and Kernel Matrix
Consider a function A : Rd x IR + R that assigns a simi- larity score k(x,y) y) E R to every pair of vectors x, y E IRd The function & qualifies as a kernel function if and only if, for any of samples " Xn, E IR,, the resulting ker- nel matrix K= = [k(x., x,) )) is positive semi-definite. This kernel matrix well captures the pairwise similarities among the samples. Common examples of kernel functions include Gaussian kernel, cosine kernel, and polynomial ker- nel. this work, we mainly focus on the polynomial kernel
min # Le.lyr-Doin
E
- (foo(f()))) - +((0(4))
(4) where ki and k2 are the kernel functions for CLIP and the target model, which may take different forms with differ- ent parameters to reflect the variation in the feature space. According to Hoeffding's 's inequality (Hoeffding, 1994), the empirical estimate of the objective function is an unbiased estimator, and the following proposition implies that the gra- dient w.r.t 0 can also be computed through limited samples.
3

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
Table Accuurcy evaluation on zero-shot image classification benchmarks of CLIP models w/wo alignment. Projection means training a linear layer to map DINOv2 representations into the CLIP feature space. Feature means directing aligning the representation pairs subjecting a linear transformation. DIVA refers to the method proposed by Wang et al. (2025), Kernel is our proposed method.
Zero-shot Datasets
Alignment
ImngeNNet
CIFARIO
CIFARION
Caffech
FER
OxfordPets
DTD
RESISC
EuroSAT
PCAM
ImageNet-S
ImageNet-O
Average
Strategies
Zero-shot
VIT-B-16
CLIP
67.72
89.98
65.61
82.17
46.35
89.04
44.95
58.19
55.89
50.70
48.28
42.30
61.22
projection
68.96
96.24
75,30
73.67
27.19
80.70
34.26
33.27
40.44
50.02
54.73
30.00
54.17
feature
67.84
90.47
66.46
82.19
46.36
89.02
45.00
58.29
56.04
50.79
48.31
42.60
61.41
kernel
67.84
91.13
67.38
82.20
46,82
89.23
45,43
57.73
56,85
54.02
48.18
43,50
62.04
CLIP
74.90
95,20
71.08
83.30
50.00
93.21
55.21
63.35
62.65
52.00
59.59
32.25
65.26
V1T--11-333 ViT-L-14
projection
70.42
95.10
74.14
80.31
24.51
80.02
37.02
37.03
32.48
50.02
56.04
28.05
54.07
feature
75.16
95.77
75.13
83.58
49.62
93.35
55.73
63.13
63.11
51.34
58.08
34.20
65.73
DIVA
74.84
95.10
70,96
84.01
49.47
93.57
55.05
63.48
62.93
53.79
59.53
32.05
65.45
kernel
75.52
96.27
77.04
84.32
50.31
93.40
55.74
63.83
63.83
52.68
59.67
34.90
66.54
CLIP
75.82
94.49
71.11
83.42
49.04
93.70
55.64
63.73
61.46
60.68
61.03
32.75
66.10
projection
71.26
94.87
77.35
72.33
23.35
79.39
39.20
42.06
35.06
50.02
57.53
28.65
54.53
feature
75.36
95.45
74.16
83.45
50,24
93.79
56,22
64.37
61.22
59.82
60.40
32.55
66.52
DIVA
75.54
93.74
70.73
83.63
50.07
93.32
55.48
63.19
60.13
59.52
56.90
31.65
65.30
kernel
76.30
95.95
74.92
83.53
50.21
93.79
56.33
64.43
61.93
59.65
61.31
35.60
67.13
Proposition 3.1. Assume that both i, and 22 take values in the range 1]., Let (ma) for M pairs of images independently sampled from the data distribution. Assume ki (Jo(Im)): So (m(m)) ) is L-Lipschitz 8 for any sampled pairs. Define the sample-wise gradient
from straying too far from their original direction. This term is represented by the Le distance between the visual representations before and after the alignment fine-tuning The final optimization objective is expressed follows:
S m S M represent
min -EI,ddy-Dem (fo()(
(6)
1(0; Imp Imz)
- ~ka(0((),, BI, [[] So, (T+)
(Jo((()))) so(mma)) - ka(g( ( (m)): (lm(((())))))))))))))))))))))(
and the true expected gradient is defined as B[((0)]
where Og the parameters of the pre-trained CLIP and frozen during the alignment phase. w is coefficient balanc- ing the two loss terms. The regularization term ensures that the alignment process does not cause the aligned embed- dings to deviate significantly from the original embeddings. The following proposition illustrates how this regularization helps preserve the language -mage alignment.
II (ki (I)),, Then, for every < c < 8L, we have
M
P
Σ
t(O: Imn (ma) E(t())
2c
51212 + 2)
2
1
< exp
(5)
Proposition 3.2. (Schlarmann et al., 2024) For every language-image pair (T, I), (f) (T)lla l S X holds, then we will have
As result, utilize stochastic optimization methods, such as mini-batch gradient descent, to solve the problem efficiently. In each iteration, we sample batch of data pairs and minimize the difference in kernel functions specif- ically for those pairs. This approach enables the alignment framework to scale effectively datasets of real-world size.
cos( (1),((T)) cos(ff(1).
2A
max max (1)fo(1) fl. (1)ll2)
3.3. Regularization for Visual-language Alignment
The proposition shows this regularization helps preserve the language- image alignment without the need of incorporating any text data during the fine-tume fine-tuning phase.
To ensure the alignment with the text branch is preserved, we introduce regularization to prevent the aligned features

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
Table Summary of zero-shot image-to-text and text-to-image retrieval performance on Flickr30K (Young et al., 2014) and COCO benchmark (Chen et al., 2015) datasets. Our alignment will not sacrifice the image-text alignment property of the original CLIP.
Image-to-Tie Text Retrieval
Text-to-lmage Retrieval
Vision
FKickr3KK R@5
MSCOCO
Flickr30K R@5
MSCOCO R@s
Encoder
R@1
R@10 10
R@1
R@5
R@ 10
R@1 1
R@
R@1
R@
w/o align
77.90
94.30
97.40
48.36
72.68
81.76
60.64
83.82
90.36
31.80
55.91
66.97
VIT-B-16
w/ aligm
78.10
94.10
97.50
48.06
72.86
81.78
60.92
84.30
90.62
32.12
56.51
67.15
w/o align
81.40
96.20
98.70
50.64
74.20
82.96
63.62
86.36
91.86
34.51
59.21
69.37
VIT-L-14
DIVA
78.70
94.80
98.30
49.88
74.06
82.82
59.80
83.36
89.40
34.21
58.76
68.97
w/ align
83.00
96,90
99.00
51.72
75.52
83.30
64.78
87.20
92.32
35.98
61.08
71.02
w/o align
83.00
96.60
99.00
52.12
76.12
83.82
64.78
87.92
93.06
35.65
60.30
70.66
VTT-1--14-366
DIVA
80.20
95,90
98.30
52.34
76.48
84.16
61.20
84.72
90.80
35.77
60.29
70.44
align
84.60
96.80
99.10
53.48
77.64
85.30
67.08
88.98
93.62
37.61
62.48
72.46
of CLIP, the incremental alignment step is quite lightweight and can be performed on image-only datasets.
Table Accuracy evaluation on counting, spatial reasoning, and caption recognition tasks of CLIP models w/wo alignment.
Training Schemes. The kernel we use is the normal- ized polynomial kernel of degree 3, which has been com- monly adopted by several well-known studies in the litera- ture (Stein et al., 2023; Kang et al., 2023). We also explore different kernel choices in the ablation studies. For DINOv2, set the hyper-parameter of kernel 7 1/dimemmt and 1, while for CLIP, they are set as trainable. More detailed settings for each experiment can be found in the Appendix B.1. With two 4090 GPUs, the alignment of ViT-
Vision Encoder
svhn
glsrb
clevr distance
clevr counts
ViT-B-16
16
31.31
43.34
22.37
21.21
+align
27.40
44.35
22.40
21.30
Zero- shot
VIT-I.-14
57.02
50.55
20.21
19.43
+align
59.63
52.53
23,39
21.11
ViT-L-14-336
56.03
52.41
18.93
20.05
+align
57.65
53.43
20.91
20.90
ViT-B-16
45.02
57.13
31.19
23.68
L--14 takes around 30 hours, which is efficient and hardware- friendly compared to the pre-training phase of CLIP.
+align
49.27
58.30
30.92
24.02
Linear
ViT-L-14
65,20
72.94 24
22.97
41.25
Probe
+align
69.39
74.51
30.82
49.67
ViT-L-14-336
61.49
70.17
28.43
53.07
4.2. Improvements on Vision-centric Tasks
+align
70.02
71.77
34.65
55.32
We begin with vision-centric tasks, which can be done by CLIP alone, to verify whether alignment with DINOv2 em- beddings can enhance the visual representation of CLIP.
4. Experiments
Zero-shot Object Recognition. We first test the zero-shot accuracy of the aligned CLIP model on standard object recognition benchmarks. We experiments on a diverse benchmarks composed of 12 datasets, including (1) com- mon objects: ImageNet (Deng et al., 2009), CIFAR-II 10, CIFAR-100 (Krizhevsky et al., 2009), Caltech101 (Fei-Fei et al., 2004): (2) fine- fine-grained objects: OxfordPets (Parkhi et al., 2012), DTD (Cimpoi al., 2014), FER2013 (Good- fellow et al., 2013); (3) domain-specific applications: PCAM (Veeling et al., 2018), RESISC45 (Cheng et al., 2017), EuroSAT (Helber et al.. 2018); and (4) out-of- distribution benchmarks: ImageNet-O (Hendrycks et al., 2021), ImageNet-Sketch (Wang et al., 2019). We adopt the standard CLIP-Benchhmrk (LAION-AL, 2022) as the pipeline for evaluation. The results are presented in Table I, For zero-shot object recognition, performance depends on both perceptual ability and compatibility with the text en- coder. After aligning with DINOv2 representations, we
4.1. Implementation Details
Models. The above framework is actually flexible enough to align any two sets of embeddings. In this study, we primarily examine the alignment of visual representations from CLIP and vision-focused models. We experiment with several versions of CLIP, including ViT-B-16, ViT-L-14, and ViT- L-14-336, all pre-trained by OpenAI. For the vision-centric target model, we employ DINOv2 (Oquab et al., 2024) with ViT-L-14 as the backbone, after registration (Darcet al., 2024). This setup allows for aligned embedding pairs to originate from models with different architectures cor- respond to images of varying resolutions, demonstrating the flexibility and generalizability of our proposed framework.
Training data. We utilize the training set of ImageNet- IK (Deng et al., 2009) as our training data, which contains 1. 1.28M images. Compared to the original training process
5

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
Table Results on probing benchmark. We report macro-
averaged recall for both local probing and global probing.
Vision Encoder
local
global
wo allggement
w/o align
44.63
52.61
VIT-B-16
w/ align
45.25
52.94
w/o align
46.40
54.51
ViT-L-14
aliggnent
w/ align
47.44
55.33
w/o align
46.05
55.13
ViT-14-336
w/ align 46.65
56.09
Figure 2. Visualization of CLIP encoder's attention maps. At- tention maps can show more fine-grained details after alignment.
observe improvements across most datasets. Specifically, the average zero-shot accuracy increases by 0.82%. 1.28%, and 1.03% with alignment for ViT-B, ViT-L., and ViT-L-336, respectively. This indicates that our proposed alignment enhances visual representation while maintaining compat- ibility with the CLIP text encoder. The improvements are particularly notable for images with low resolution (e.g., CIFAR) or tiny objects (e.g., EuroSAT), highlighting en- hanced fine-grained perception capabilities. Through exper- iments with different CLIP backbones, we find that these improvements are sechitcctur-agnostie consistently boost- ing classification accuracy. Furthermore, the enhancements hold for both the ImageNet dataset and zero-shot datasets, demonstrating that alignment does not compromise the gen- eralizability of CLIP. Even when fine-tuned on relatively small training data, the model still exhibits strong zero-shot performance on out-of-distribution datasets.
explore replacing DINOv2 with other vision models, such as MLCD (An et al., 2024). The results show improved zero-shot accuracy for all these pairs, demonstrating the generalizability of the proposed framework.
Image-to-text and Text-to-image Retrievals. To further demonstrate that the proposed alignment framework does not compromise the image-text alignment or the general- izability of CLIP, we present zero-shot image-to-text and text-to-image retrieval performance in Table These ex- periments were conducted on both the Flicker30K (Young et al., 2014) and MSCOCO (Chen al., 2015) datasets. The results indicate that CLIP maintains strong performance in both retrieval tasks even after alignment. This reinforces our assertion that the image-text alignment is effectively preserved, despite the improvements in visual capability.
To demonstrate the superiority of the proposed kernel-basse alignment, we compare our method with two straightfor- ward baselines: (1) a linear projector trained to map the DINOv2 representation into the CLIP representation space, and (2) replacing kernel-basee alignment with feature-bant based alignment (details available in Appendix B.2). However, we observe limited improvements or even significant drops in zero-shot accuracy. The representation spaces of DI- NOv2 and CLIP can vary significantly, making it challeng- ing to directly align them while preserving compatibility with the text embeddings. In contrast, our kesnel-based alignment focuses on aligning the relative relationships among samples, which helps maintain the maero-structure of the feature space. We also compare our approach with DIVA (Wang et al., 2025), and the results indicate that our method achieves better zero-shot performance. Furthermore, DIVA and our proposed alignment are orthogonal, they can be combined for complementary benefits.
Counting, Spatial Reasoning, and Caption Recognition. As highlighted in the literature (Tong et al., 2024), the CLIP encoder often struggles with specific tasks, including count- ing, spatial relationship inference, and caption recognition, which serves as major motivation for our work. To in- vestigate how alignment with DINOV2 can alleviate these issues, we conduct experiments on four related benchmarks: (1) SVHN (Netzer et al., 2011), which composed of natural scene images with digits and numbers; (2) GTSRB (Stal- Ikamp et al., 201 2), datasets for recognition of German traffic sign; (3) CLEVR Distance (Johnson et al., 2017), which composed of images with multiple objects and the task is to determine the distance between the closest ob- jects; and (4) CLEVR Counts, a benchmark that requires the model to count the number of objects within the images. We again follow the evaluation protocol of which formulates all four benchmarks as classification tasks. We assess the improvements through both zero-shot classifi- cation and linear probing. The results, presented in Table 3, indicate that alignment significantly enhances CLIP's 's perfor- mance on text or spatial-relate tasks, particularly after lin- ear probing. For example, when using ViT-L- 14, alignment improves the average accuracy by 1.58% for zero-shot classi-
In Appendix C.2, we extend our framework to three addi- tional models that incorporate an intermediate embedding layer linking modalities, albeit with implementations dif- fering from CLIP: SigLIP (Zhai et al., 2023), DFN (Fang et al., 2023), and MetaCLIP (Xu et al., 2024). We also
6

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
Table Evaluation of LLaVA on vision-language benchmarks. The performance can improved by using CLIP after alignment
as the visual encoder, and it can be further improved through parameter-efficiee line-tuning (peft) of the LLM.
Vision Encoder
A12D
text-ocr
text-pure
pope
tallyQA
RefCOCOS
RefCOCO+
RefCoco
vQa-02
llava-wild
Blaa-cocc
Average
baseline
52.91
55.37
41.16
84.64
60.39
51.47
13.89
12.73
15.16
75.07
44.13
75.07
48.50
VIT-L
+align
53.66
55.70
41.53
84.92
60.39
51.50
14.15
12.89
15.37
75.10
44.67
76.60
48.87
+peft
53.63
54.54
41.82
84.35
57.19
53,93
56.06
52.74
61.27
74.71
44.30
64.84
58.28
+both
54.79
55.03
42.13
84.37
59.87
58.02
56.47
52.82
62.25
74.87
43.70
65.33
59.14
ViT-L-336 333
baseline
53.27
57.82
48.63
85,45
61.45
51.47
55.19
51.97
60.49
76.40
42.97
75.60
60.06
+align
52.95
57.94
48.53
85.57
61.25
51.47
55.66
52.78
60.54
76.34
48.40
79.07
60.88
+peft
52.69
57.79
48.50
85.41
61.91
54.14
56.76
53.81
61.98
75.88
44.13
69.13
60.18
+both
52.78
58.18
48.87
85.70
62.21
56.22
57.68
55.74
63.37
75.93
44.90
78.83
61.70
fication and 5.51% for linear probing. Compared to common object recognition, tasks like caption recognition, counting, and spatial reasoning rely more heavily on the quality of the visual representation. This is where lision-centric models such DINOv2 significantly outperform CLIP. After align- ment, the CLIP representation demonstrates enhanced visual capabilities, resulting in better performance on these tasks. Additionally, we include results from MMVP-VLM (Tong al., 2024) in Appendix C.4, which provides insights from more diverse but relatively smaller benchmark.
The attention maps are generally similar w/wo alignment, as the regularization term prevents significant changes in the model's weights. However, we still observe that the attention maps after alignment are sharper and highlight more fine-grained features. This indicates that the aligned model is better at capturing details from the input image, resulting in improved localization ability.
4.3. Improvements on MLLMs
We then show that the enhancements in visual representation can be transferred to downstream tasks. Specifically, we re- place the vision encoder of several MLLMs with the aligned CLIP encoder and evaluate improvements on relevant bench- marks. The LLM component can either be entirely frozen fine-tuned. Importantly, our alignment maintains the similarity of visual representations before and after the pro- cess, eliminating the need for an additional vision- Laniuan alignment step. As a result, fine-tuning can be conducted efficiently using techniques such as LoRA (Hu et al., 2022).
Localization Ability. We further evaluate the localization ability of CLIP visual encoders w/wo alignment. Specif- ically, we utilize the probing benchmark introduced by Covert et al. (2025). This evaluation involves freezing the visual encoder and training a classification head to predict the union of labels for each patch (local probing) or the entire image (global probing) using a binary cross-entropy loss. The experiments
conducted on the MSCOCO dataset (Lin et al., 2014). Following the original setup, we report the maero-averaged recall to account for class imbal- ances. The results, shown in Table 4, reveal that after align- ment with DINOv2, the CLIP visual encoder demonstrates improved localization ability, achieving higher recall for both local and global probing. This enhancement can be at- tributed to the strong perceptual capabilities of the DINOv2 encoder, which are transferred to CLIP encoder via align- ment. Furthermore, the improvements are consistent across different CLIP encoder architectures. While the gains may not be as significant compared to those reported by Covert et al. (2025), their approach disrupts the connection with text representations, leading loss of CLIP's zero-shot capability. In contrast, our method enhances performance without compromising the textual alignment.
Evaluation on LLaVA. We first conduct experiments on LLaVA- 1.5--BB (Liu al., 2024). We utilize diverse bench- marks which comprises of multiple tasks including (1) open- ended visual question answering: VQAv2 (Goyal et al., 2017) and TextVQA (Singh et al., 2019); (2) localiza- tion: RefCOCO, RefCOCO+, RefCOCOg (Kazemzadeh et al., 2014; Yu et al., 2016); and (3) closed-set prediction: VSR (Liu et al., 2023), TallyQA (Acharya et al., 2019), POPE (Li al., 2023b), and AI2D (Kembhavi et al., 2016). We also perform GPT-aided evaluation, LLaVA-bench (Liu et al., 2024). We replace the visual encoder with our aligned version and experiment with both VIT-L-114 and VIT-L-114- 336 versions of CLIP encoder. To better illustrate the en- hancements, we further fine-tune the LLM component of the models. We skip the feature alignment step and conduct only visual instruction tuning using LoRA. For the training data,
We visualize the attention maps from the penultimate layer of the CLIP visual encoder for several examples in Fig 2.

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
(a)
(d)
75
68
Accuracy (%)
67
66.54
70
Accuracy (%6)
67
66.54
00,01
Accuracy (%)
67
65.54
66-23
66.24
Accuracy
65-
65.25
66
66.26
66.19
65.26
20
$5.25
$7.78
65
0
1
2
baseline
W!
w/o
0.0
0.1
0.5
1.0
5.0
10.0
basoline
poly. Gaussian cosine kenel function
# of epochs
w/wo regularization
coefficieut
Figure 3. Ablation studies. Average zero-shot accuracy across 11 datasets are reported as evaluation metrics: (a) Effects of training
epochs; (b) Effects of the regularization term; (c) Effects of the coefficients of alignment: (d) Effects of different kernel functions.
Table 6. Evaluation results on OpenFlaming across seven rision-languan datasets using 0, 4, and 16 in-context examples. The performance can he improved by using the CLIP visual encoder after alignment as the visual encoder, without fine-tuning L.M part. We report CIDEr for COCO and Flicker-30K, ROC AUC for HatefulMemes, and VQA accuracy for the rest.
Shots
Vision Encoder
COCO
Flickr-30K
VQAv2
OK-VQA
TextVQa
VizWiz
HatefulMemes
Average
CLIP
74.40
54.15
43.54
28.72
25.42
17.80
51.03
42.15
0
+align
76.01
54.42
43.71
29.04
25.92
17.91
52.86
42.84
CLIP
82.56
58.72
45.40
31.81
29.03
23.05
48.24
45.54
4
+align 83.95
60.03
45.57
31.82
29.43
23.12
49.22
46.16
90.42
62.68
45.59
32.12
29.83
35.22
50.62
16
+align 90.89
63.85
45.73
32.43
30.33
35.74
48.85
we utilize the LLa data mixture (Liu et al., 2024), which contains 665k examples and is the tuning dataset for the original LLaVA. We also report metrics from performing the same fine-tuning on the original model to control for the effects fine-tuning. The results, shown in Table 5, indi- cate improvements even when the vision encoder simply replaced with our aligned version, without fine-tuning the LLM component. This suggests that the alignment lead to improvement while preserving the compatibility of CLIP with the LLM component. The improvements are further amplified when we fine-tune the LLM component, resulting in an average score increase of 1.66%, for model with VIT- L-14-336 as the visual encoder. This is particularly evident in the localization benchmarks, with 3.05% improvement average. Furthermore, the enhancements cannot be solely at- tributed to LLM fine-tuning, fine-tuning LLM alone does not yield such great gains. These experiments demonstrate that our proposed alignment enhances visual capabilities, especially fine-grained perception, and these improvements can be effectively transferred to downstream applications.
and HatefulMemes (Kiela et al., 2020), For each dataset, we sample few in-context demonstrations from the training split uniformly at random, and prompt the model to give answers to the test samples. We use the OpenFlamingo-3B (Instruct) version of the model. The results are shown in Table 6. For all experiments, we replace the visual encoder to our aligned version, without fine-tuning the LLM part. We discover the alignment improves most of the metrics, showing the generalization of our proposed framework.
4.4. Ablation Studies
We study the impact of various design choices through ab- lation studies, focusing factors such as training epochs, loss function design, and the kernel functions utilized. Zero- shot object recognition accuracy serves as our evaluation metric, with results illustrated in Fig.3. Key takeaways in- clude: (1) the method demonstrates optimal performance with very short training duration; (2) the regularization term is crucial for preserving visual- languaga alignment; (3) the performance remains stable regardless of the coefficient and (4) different kernels all contribute positively, with the polynomial kernel yielding the best results. A more comprehensive analysis can be found in Appendix C.7.
Evaluation OpenFlamingo. We extend the evalua- tion to another popular MILM,, OpenFlamingo (Awadalla et al., 2023). We follow the evaluation pipeline of the original paper, which test the in-comtext-lcaming ability of the MLLMS in several VQA benchmarks, including COCO (Chen al., 2015), Flicker-30K (Young et al., 2014), VQAv2 (Goyal et al., 2017), OK- OK-VQA (Marino et al., 2019), TextVQA (Singh et al., 2019), VizWiz (Gurari al., 2018),
5. Limitations & Conclusion
In this study, we introduce a novel kernel-based embedding alignment strategy designed to align two sets of embeddings.
8

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
Our evaluations on vision-centric tasks and MLLM bench- marks demonstrate that this alignment enhances the visual capabilities of CLIP, with benefits that inherit to downstream applications. Unlike most existing research which enhances the visual encoder but requires re-tuning of the L.M,, our approach opens up a new research avenue to facilitate im- provements through lightweight adaptation, making these advancements more accessible to a broader community.
8084, 2019.
Allen-Zhu, Z. and Li, Y. Towards understanding ensem- ble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.098 2020.
An, X., Yang, K., Dai, X., Feng, Z.,, and Deng, J. Multi-label cluster discrimination for visual representation learning. In European Conference on Computer Vision, pp. 428- 444. Springer, 2024.
Due to the lack of computational resources, we align the embedding on relatively small-scale dataset, and only evaluate the performance on small MLLMs. It can be left as a future work to conduct the fine-tuning on a larger scale datasets (Gadre et al., 2024) and verify the effects on MILMS with larger size (e.g., 70B). Another limitation is that this work focuses on the alignment between CLIP and DINOv2. There are MLLMs that utilize vision coders other than CLIP (Chen et al., 2024), and other of vision-centric models (Bao et al., 2022) available. We would extend the current pipeline to help align other embeddings pairs and bring improvements to more MLLMs.
Awadalla, A., Gao, I., Gardner, J., Hessel, J., Hanafy, Y., Zhu, W., Marathe, K., Bitton, Y., Gadre, S., Sagawa, S., Jitsev, J., Kornblith, S., Koh, P. W., Ilharco, G., Worts- man, M., and Schmidt, L.. Openflamingo: An open- source framework for training large autoregressive vision- language models. arXiv preprint arXiv:2308.013390 2023.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versatile vision- language model for understanding, localization, text read- ing. and beyond. arXiv preprint arXiv:2308. /2966, 1(2): 3, 2023.
Acknowledgements
The work of Farzan Farnia is partially supported by a grant from the Research Grants Council of the Hong Kong Spe- cial Administrative Region, China, Project 14209920, and is partially supported by CUHK Direct Research Grants with CUHK Project No. 4055164 and 4937054. Also, this work is supported in part by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. T45-401/22-N). Finally, the authors would like to thank the anonymous reviewers for their insightful suggestions and feedback.
Bao, H., Dong, L., Piao, S., and Wei, F. BEit: BERT pre- training of image transformers. In International Confer- ence on Learning Representations, 2022. URL https: / //openreview.rr net/ forum?id=p BhZSz5904.
Binkowski, M., Sutherland, D. J., Arbel, M., and Gret- ton, A. Demystifying mmd gans. arXiv preprint 2018.
Cao, Q. Xu, Z.., Chen, Y., Ma, C.,, and Yang, X. Domain prompt learning with quaternion networks. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26637-26646, 2024.
Impact Statement
Our work enhances the capabilities of multi-modal large language models by improving visual perception through the alignment of visual representations. As shown in the experiments, we improve the performance on both general- purpose computer vision benchmarks and domain-specific applications including remote sensing and medical diagno- sis. Therefore, Our method has the potential to significantly advance applications in areas such as autonomous systems and assistive technologies. This progress can lead more accurate and context-aware AI systems, ultimately benefit- ing industries like healthcare, transportation, and education.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Chen, X., Fang. H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollár, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: arXiv: 0aee100040005555 2015.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q. Zhu, X., Lu, L.,, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185 24185-24198, 2024.
References
Acharya, M., Kafle, K., and Kanan, C. Tallyqa: Answering complex counting questions. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 8076

---

Kernel-based
A Unsupervised Alignment of CLIP and DINOv2 Embeddings
Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the Proceed- ings of the IEEE, 105(10):1865-1888) 2017.
Gong, S., Lei, H., Dou, Q., and Farnia, F. Boosting the visual interpretability of clip via adversarial fine-tuning. In International Conference Learning Representations, 2025. URL https: ptttpps://ppeeeviiieee httpp/ppppreiieeeeeeeeeeeeeeeeee net.:.ewwweee/.eeeee.e.. ntt.e/rueet//tttttt/ 1 d-kkuu TvzxPRp.
2017.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Proceed- ings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. Challenges in representation learn- ing: A report on three machine learning contests. In Neural information processing: 20th international con- ference, ICONIP 2013, daegu, korea, november 3-7, 2013. Proceedings, Part III 20, pp. 117-124. Springer, 2013.
Covert, I., Sun, T., Zou, J., and Hashimoto, T. Local- ity alignment improves vision-language models. In- ternational Conference on Learning Representations, 2025. URL https::/openrev///ee wtwp://weeetrtorrreeeeeoee id==ssVptHTPN.
Darcet, T., Oquab, M., Mairal, J., and Bojanowski, P. Vi- sion transformers need registers. In The Twelfth In- ternational Conference on Learning Representations, httplpeeeeveeeeeeeeeeeooeee net/
Goyal, Y., Khot, T., Summers-Stay D., Batra, D., and Parikh, D. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
2024. URL
id=2dn03LIiJI.
Dehdashtian, S., Wang, L., and Boddeti, V. N. Fairerclip: Debiasing clip's zero-shot predictions using functions in rkhss. arXiv preprint arXiv:2403. 15593, 2024.
Gross, D. Recovering low-rank matrices from few coeffi- cients in any basis. IEEE Transactions on Information Theory, 57(3):1548 1566, 2011.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A lcage-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Guo, Z., Xu, R., Yao, Y., Cui, J., Ni, Z., Ge, C., Chua, T.-S., Liu, Z., and Huang. G. Llava-uhd: an Imm perceiving any aspect ratio and high-resolution images. In European Conference on Computer Vision, pp. 390-406. Springer, 2024.
Fang, A., Jose, A. M., Jain, A., Schmidt, L., Toshev, A., and Shankar, V. Data filtering networks. arXiv preprint arXiv:2309.1742 2023.
Gurari, D.., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., and Bigham, J. P. Vizwiz grand challenge: Answering visual questions from blind people. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 360036008000 1000-3617,20 2018.
Fei-Fei, L., Fergus, R., and Perona, Learning genera- tive visual models from few training examples: An in- cremental bayesian approach tested on 101 object cate- gories. Computer Vision and Pattern Recognition Work- shop, 2004.
He, B. and Ozay, M. Feature kernel distillation. Interna- tional Conference on Learning Representations, 2022.
Friedman, D. and Dieng, A. B. The vendi score: A diversity evaluation metric for machine learning. arXiv preprint urXiv:2210.0244 2022.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smymis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang, J., al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Introduc- ing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. In IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, pp. 204 207. IEEE, 2018
Gao, P., Geng, S., Zhang, R., Ma. T., Fang, R., Zhang, Y., Li, H., and Qiao, Y. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2):5811 1-595, 2024.
Hendrycks, D., Zhao K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. CVPR, 2021.
Gong, S., Zhong, Y., Ma, W. Li, J., Wang. Z., Zhang, Holistic
J., Heng, P.-A., and Dou, Q. 3dsam- adaptation of sam from 2d to 3d for promptable tumor segmentation. Medical Image Analysis, 10:1033,4, 2024.
Hoeffding, W. Probability inequalities for sums bounded random variables. The collected works of Wassily Hoeffd- ing, pp. 409 409-426, 1994.
10

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
Hofmann, T., Schölkopf, B., and Smola, A. J. Kernel meth- ods in machine learning. 2008.
Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., and Testuggine, D. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems, 33: 261 1-244, 2020.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y.,
Wang. S., Wang, I.., and Chen, W. LoRA: Low-rank adap- tation of large language models. In International Confer- ence on Learning Representations, 2022. URL https: //openrevies htt.:/eee//ooreeeeee dew.uummidddneeeeeeeeeeee
Kohler, J. M. and Lucchi, A. Sub-sampled cubic regulariza- tion for non-convex optimization. In International Con- ference on Machine Learning, pp. 1895-1904. PMLR, 2017.
Jalali, M., Li, C. T., and Farnia, F, An Coormation-thhorett
evaluation of generative models in learning multi- distributions. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Jalali, M., Li, C. T., and Farnia, F, An information-theoretic evaluation of generative models in learning multi-modal distributions. Advances in Neural Information Processing Systems, 36, 2024.
LAION-AI. Clip benchmark: Clip-like model evaluation, 2022. URL https:/ /ttt::///ghubbb//// com/LAION LAION-AL CLIP benchmark.
Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-iman image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19930-19749 PMLR, 2023a.
Jiang, D., Liu, Y., Liu, S., Zhao, J., Zhang, H., Gao, Z., Zhang, X., Li, J., and Xiong, H. From clip to dino: Visual encoders shout in multi-modal large language models. 2023.
arXiv preprint arXiv:2310.0880
Li, X., Grandvalet, Y.. Davoine, F., Cheng, J., Cui, Y.. Zhang, H., Belongie, S., Tsai, Y.-H., and Yang, M.-H. Transfer learning in computer vision tasks: Remember where you come from. Image and Vision Computing, 93: 103853, 2020.
Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and ele- mentary visual reasoning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2901-29100 2017.
Li, Y., Du, Y., Zhou, K., Wang. J., Zhao, X., and Wen, J.-R. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Naturul Language Processing, 2023b. URL https: //openreview. net forum? wwt.oiddoooooxx.....
Kamath, A., Hessel, J., and Chang, K.-W. What's "up" with vision-language models? investigating their strug- gle with spatial reasoning. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: http///pppp/eeeeeeee/eeeeeeeeeee foruf??u?? 1d=
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D.,, Dollár, P., and Ziticck C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Kang, M., Shin, J., and Park, J. Studiogan: A taxonomy and benchmark of gans for image synthesis. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 45(12):15725-11542, 2023, doi: 10.1109/TPAM..2023. 3306436.
Liu, F., Emerson, G., and Collier, N. Visual spatial reason- ing. Transactions of the Association for Computational Linguistics, 1 11:635-651 2023.
Kazemzadeh, S., Ordonez, V., Matten, M., and Berg, T. Referitgame: Referring to objects in photographs of natu- ral scenes. In Proceedings of the 2014 conference on em- pirical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
Kembhavi, A., Salvato, M., Kolve, E.,, Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth dozen images. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 335-251. Springer, 2016.
Mao, C., Geng, S., Yang. J., Wang, X., and Vondrick, C. Understanding zero-shot adversarial robustness for large- scale models. arXiv preprint arXiv:22 2022.

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok-vqa: A visual question answering benchmark requir- ing external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Salehi, M., Farajtabar, M., Horton, M., Faghri, F, Pouransari, H., Vemulapalli, R., Tuzel, O., Farhadi, A., Rastegari, M., and Mehta, S. Clip meets model zoo ex- perts: Pseudo-supervision for visual enhancement. arXiv preprint arXiv:2310. 14 108, 2023.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B.,
Schlarmann, C., Singh, N. D., Croce, F., and Hein, M. Robust clip: Unsupervised adversarial fine-tuning of vi- sion embeddings for robust large vision-language models. arXiv preprint arXiv:2402. 12336, 2024.
Ng, A. Y., et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. Granada, 2011.
Shen, L., Chen, G., Shao, R., Guan, W., and Nie, L. Mome: Mixture of multimodal experts for generalist multimodal large language models. arXiv preprint arXiv:2407. 2024.
Oquab, M.. Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fermandez, P., Haziza, D., Massa, F., El- Nouby, A., al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, pp. 1-31, 2024.
Shen, X., Du, C., Pang, T., Lin, M., Wong, Y., and Kankan- halli, M. Finetuning text-to-imaae diffusion models for fairness. arXiv preprint 1xXiv:231.007044 2023.
Ospanov, A. and Farnia, F. On the statistical complexity of estimating vendi scores from empirical data. arXiv preprint ar xrii:24400.2171, 2024.
Shi, M., Liu, F., Wang, S., Liao, S., Radhakrishnan, S., Huang, D.-A., Yin, H., Sapra, K., Yacoob, Y.. Shi, H., et al. Eagle: Exploring the design space for multi- modal Ilms with mixture of encoders. arXiv preprint arXiv:2408. 159988 2024.
Ospanov, A., Jalali, M., and Farnia, F. Dissecting clip: Decomposition with a schur complement- based approach. arXiv preprint arXiv:2412. 18644, 2024a.
Singh, A., Natarajan, V., Shah, M.. Jiang, Y., Chen, X., Batra, D.,. Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Ospanov, A., Zhang, J., Jalali, M., Cao, X., Bogdanov, A., and Farnia, F. Towards a scalable reference- evalua- tion of generative models. Advances in Neural Informa- tion Processing Systems, 111120092-120922 2024b.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V. Cats and dogs. In IEEE Conference Computer Vision and Putlern Recognition, 2012.
Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. Man vs. computer: Benchmarking machine learning algo- rithms for traffic sign recognition. Neural networks, 32: 323-332, 2012.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 7768-8763. PMLR, 2021.
Stein, G., Cresswell, J., Hosseinzadeh, R., Sui, Y., Ross, B., Villecroze, V., Liu, Z.., Caterini, A. L., Taylor, E., and Loaiza-Ganema G. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Informtion Processing Systems, 36:3732-3784 2023.
Ranzinger, M., Heinrich, G., Kautz, J., and Molchanov, P. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12490-12500, 2024.
Sun, Q. Fang, Y., Wu, L., Wang, X., and Cao, Y. Eva- clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15388 2023.
Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A.. Kiela, D., and Ross, C. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/VV Conference on Computer Vision and Puttern Recognition (CVPR), pp. 5238-5248, June 2022.
Rezaei, P., Farnia, F., and Li, C. T.. Be more diverse than the most diverse: Online selection of diverse mixtures of generative models. arXiv preprint 17222:2412.17622, 2024.
Saha, O., Van Horn, G., and Maji, S. Improved zero-shot classification by adapting vlms with text descriptions. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17542-17552, 2024.
Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal lms.. In Proceedings of the IEFF/CVE
12

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
Conference on Computer Vision and Puttern Recognition, pp. 3868-9578, 2024.
Vision-ECCV 2016: 14th European Conference, Amster- dam, The Netherlands, October I 11-1, 2016, Proceed- ings, Part I1 14, pp. 69-85. Springer, 2016.
Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., and Welling, M. Rotation equivariant CNNs for digital pathol- ogy. June 2018.
Yu, Q. He, J., Deng, X., Shen, X., and Chen, L.-C. Con- volutions die hard: Open-vocabulary segmentation with single frozen convolutional clip. Advances in Neural Information Processing Systems, 36:32215-32233 2023,
Vidit, V., Engilberge, M., and Salzmann, M. Clip the gap: A single domain generalization approach for object de- tection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3219-3229, 2023.
Yuksekgonul, M., Bianchi, E,, Kalluri, P., Jurafsky, D., and Zou, When and why vision-language models be- have like bags-of-words, and what do about it? In International Conference on Learning Representations, 2023. URL httph:/tp////...... http///pppeeeeeeeeeeeeeeeeeeeeee net n id Id-KRUUvxhuua
Wan, B., Tschannen, M., Xian, Y., Pavetic, F., Alab- dulmohsin, I., Wang, X., Pinto, A. S., Steiner, A. P., Beyer, L., and Zhai, X. Locca: Visual pretraining with location-aware captioners. In The eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: https////pppevvieeeeeeeeeee net/ net/forum? forum? id= id=j JHHKAEHAE
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-trat pre-training. In Proceed- ings of the IEEE/CVF International Conference Com- puter Vision, pp. 11975-11986, 2023.
Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning robust global representations by penalizing local predic- tive power Advances in Neural Information Processing Systems, 32, 2019.
Zhang, J., Jalali, M., Li, C. T., and Farnia, F. Identification of novel modes in generative models via fourier-based differential clustering, 2024a. URL https:://axxiiii org/ aaaaaa/s/a/0 /2405. 02700.
Zhang, J., Li, T., and Farnia, F. An interpretable evalua- tion of entropy- based novelty of generative models. arXiv preprint urXiv:2402.1 /7277, 2024b.
Wang, W., Sun, Q., Zhang, F., Tang, Y., Liu, J., and Wang, Diffusion feedback helps clip see better. In International Conference on Learning Representations, 2025. URL httts:://tpeeeeeeeee http:///ppeeeeieeeeeeeeeeeee htt/ id=t
Zhang, R., Zhang, W., Fang. R., Gao, P., Li, K., Dai, J., Qiao, Y., and Li, Tip-adapter: Training-free free adaption of clip for few-shot classification. In European conference on computer vision, pp. 493-510. Springer, 2022.
Wu, S., Zhang, W. Xu, L., Jin, S., Li, X., Liu, W., and Loy, C. C. Clipself: Vision transformer distills itself for open-vocabulary dense prediction. arXiv preprint arX1v:2310.01403, 2023.
Zhong, Y., Yang. J., Zhang, P., Li, C., Codella, N., Li, L. H., Zhou, L., Dai, X., Yuan, L., Li, Y., et al. Regionclip: Region-based language-image pretraining. In Proceed- ings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16793-16803. 2022.
Xu, H., Xic, S., Tan, X., Huang, P.-Y.. Howes, R., Sharma, V., Li, S.-W., Ghosh, G., Zettlemoyer, L., and Feicht- enhofer, C. Demystifying CLIP data. In The Twelfth International Conference on Learning Representations, 2024. URL httph:tt//////tt /ttpp:://opeeeee http:///ppeeeeeeeeeeeemee forum? 10-5BCFInfE10,
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. International Jour- nal of Computer Vision, 130(9):2337-22
2022.
Xuhong, L., Grandvalet, Y., and Davoine, F. Explicit in- ductive bias for transfer learning with convolutional net- works. In International conference on machine learning, pp. 2825 2834. PMLR, 2018
Zhou, Z., Shen, Y., Shao, S., Gong, L.,, and Lin, Rethink- ing centered kernel alignment in knowledge distillation. arXiv preprint arXiv: 2024.
Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Lin- guistics, 2:67-78, 2014.
Yu, L., Poirson, P., Yang. S., Berg, A. C., and Berg, L. Modeling context referring expressions, In Computer
13

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
A. Proofs of Propositions
A.I. Proof of Proposition 3.1.
This is a conclusion derived from Vector Bernstein Inequality (Gross, 2011; Kohler Lucchi, 2017), which shows that -1, XM are independent vector-valued random variables with common dimension d and that each one is centered, uniformly bounded and also the variance is bounded above:
B| mm 0 and ||Xmll2 $ μ as well as [m |1 S o².
Let
Z
1 M
Σ Xm=
(8)
then we have for 0 < <<<<o 0²/1/
S exp (-
(-80f
Me²
+ 1)
1
P(I 2 2
(9)
802 4
By plugging Xm with 1(0; Ta Img)) [O), we have
E
[t(0; Li Amm) B[{(O)]
(10)
Imy my
Dam
[e(0; Im1 1mz) - B B]](0)|12 < 8L,
(11)
El|it(; Ima) [()()] < 64L".
(12)
Therefore, we have:
Me²
1 4
P
Σ
t(0: Im [ma) - E[t(0)] 2c) S exp(
+
(13)
M
51212
A.2. Proof of Proposition 3.2.
The proposition and the proof adapted from (Schlarmann et al., 2024). We have
g())
(o)
So(1)
(1)
So(1)
(T),((T))))))))))))))))))))))))) - cos (oosSS(()) = (TT()) (Tll
S
(
(T,l2 (
For which we can get the two upper bounds:
Seo So(T) I
1
[1] (1)lall 11, (T) fo(T)ll2),
(14)
(T)ll2
[Ifo(
1 ITfo(T)
(T)
fa(1)
[11/6(1)]22 - (1 + 11, (T) fo(1)12)
(15)
[o,, (T)ll2 ([fo(T)lla
(T)ll
According the triangle inequality:
(1 (166,)) fo)ll)2.
(16)
therefore, the upper bound holds:
2 2 (aos( - cos( min (TT(( I[,, (T)ll) -1)(())))
(17)
Remarks. This proposition demonstrates that minimizing the L.2 distance between the visual representations before and after the alignment fine-tuning help directly preserve the visual-text alignment. This is different from previous work such as Xuhong et al. (2018); 1) Li al. (2020), which penalizes the Lo-morm of parameter changes during the fine-tuning phase. While their method effectively prevents overfitting to the target domain in transfer learning, our direct feature-based penalty measure aims to maintain zero-shot compatibility while enhancing fine- grained visual capabilities.
14

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
B. More Implementation Details
B.1. Hyper-parameters Setup.
Here we provide more information on the implementation details. Specifically, the hyper-paramette used for the alignment fine-tuning listed in Table All the experiments are conducted on NVIDIA GeForce RTX 4090 GPUs.
Table Detailed hyper-parameter setups.
Hypee-parameeters
ViT-B-16
ViT-L-14
ViT-L-14-336
coefficient w
0,5
0.5
1.0
number of GPUs
batch size
128
64
training epochs
2
2
optimizer
AdamW (Loshchilov, 2017)
weight decay
le-4
B learning rate
(0.9, 0.999)
1e-5
scheduler
CosineAnnellingLR
warm-up steps
1400
2800
5600
B.2. Illustration on Feature-based Alignment.
A vanilla way to align the CL.IP embedding fo(T:) with the embedding of the target model g(T;) is to minimize the L.2 distance between (1:) and g(4:) subject to a linear transformation R E jddxd'
min E
[w]]ff(Ii) - +R9(4)))1+ (lf(()) - fa, (T.1)).
(18)
R.R. Ie-Dimn
call this method feature-based embedding alignment. This approach also make sense and lead to certain improvements compared with the original visual embeddings of CLIP. However, the representation space of CLIP and the target model often vary significantly, which can hardly be adjusted via linear transformation. As result, it is sub-optimal solution. On the contrary, our kernel-based embedding alignment provide more flexible solution. Firstly, comparing with absolute position within the feature space, the relative position among samples is more important. We would like the embeddings of two samples with similar visual characteristics to be close to each other. Using kemnel function as supervision can directive promote this effects. Secondly, kernel function measures the similarity in the high- sigh-dimensional kernel space. Alignment of embeddings within the kernel space will not lead to drastic changes of embeddings the original feature space, making the image embeddings after alignment still compatible with text encoder and downstream modules. Moreover, our kernelbbaeed alignment can actually be interpreted as aligning the representations in the kernel space. The feature transformation é for common kemnels usually non-linear. It renders more flexibility compared the feature- alignment which only use linear transformation.
B.3. Implementation Details for LLM's Fine-tuning.
In this section, elaborate the implementation details of the LLM fine-tuning of LLaVA, which used further demonstrate the enhancement of the vision encoder with alignment. We employ the official implementation from LLaVA for LoRA fine-tuning. The training is conducted on a mixture of LLaVA-1.5 data for one epoch, using the following LoRA configuration: r = 128 and a = 256. The training is executed in bf16 format across four NVIDIA GeForce RTX 4090 GPUs, with a batch size of per device. To address the small batch size, we apply a gradient accumulation step of 32. The optimizer used AdamW W (Loshchilov, 2017), set with a learning rate of 2e-4 and weight decay of
15

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
C. Additional Experimental Results
C.1. Loss Curve for the Alignment Term.
We visualize the alignment loss (with moving average smoothing) during the fine-tuning phase in Fig. 4. We discover the loss term is stably decreasing during the fine-tuning phase, showing that the alignment term is indeed effectively optimized improve the similarity between CLIP and DINOv2 representations.
1.06
1.01
batch-wrise 1o6s (75)
1.02-
3.9
0.6
www
500
1000
1500
2000
2500
Imining stegs
Figure 4. Visualization of the alignment loss curve. The alignment loss is stably decreasing during the alignment fine- fine-tuning phase.
C.2. Results for VLMs other than CLIP.
To evaluate the generalization of our proposed framework, we conducted experiments on three additional models that feature an intermediate embedding layer connecting modalities, but with implementations distinet from CLIP: SigLIP (Zhai et al., 2023), DFN (Fang al., 2023), and MetaCLIP (Xu al., 2024). For all the VLMs, utilized the ViT-L- 14 version, maintaining the same implementation details as those for CLIP ViT-L-14. We assessed zero-shot accuracy across 12 datasets as our evaluation metric, with results presented in Table 8. The findings indicate that our alignment fine-tuning enhances zero-shot performance for all three models, yielding average increases of 0.45%, 0.48%, and 0.87% respectively. These results highlight the generalizability of our framework, demonstrating that the alignment is applicable to VLMs beyond CLIP. Despite ongoing advancements CLIP research, our proposed alignment remains adaptable and beneficial for these more advanced VLMs.
Table 8. Accuracy evaluation on zero-shot image classification benchmarks of vision-language models w/wo alignment. We
experiments with ViT-L-14 version of SigLIP (Zhai et al., 2023), DFN (Fang et al., 2023), and MetaCLIP (Xu et al., 2024).
Zero-shot Datasets
ImageNet
CIFARIO
CIF
CalTeeh
OxfordPets
RESISC
EuroSAT
PCAM
ImageNet-S
ImageNet-O
FER
DTD
Vision Encoder
Average Zero-shot
SigLip +align
82.02
96.87
84.27
86.01
49,90
95.31
71.06
72.59
62.65
50.55
74.02
29.60
70.26
82.18
96.88
84.89
86.11
49.16
95.42
71.06
72.84
65.54
50.11
73.92
31.90
70.71
DFN
81.50
98.32
88.15
85.57
42.70
95.58
66.06
73.25
64.94
63.18
68.33
39.30
71.40
+align
81.28
98.12
87.82
85.49
44.27
95.39
66.70
73.76
65.44
62.90
68.40
42.35
71.88
MetaCLIP
75.58
95.67
77.72
85.74
37.96
93.81
62.34
68.52
60.41
70.35
65,05
28.90
67.86
+align
76.60
96.42
80.11
85.49
40.39
93.68
62.82
69.25
60.35
69.76
65.12
32.65
68.73
16

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
C.3. Results for Other Target Models.
We also investigate the potential of replacing DINOv2 with alternative vision encoders demonstrate the flexibility of our proposed framework. Specifically, employed MLCD (An et al., 2024) as the target model, with results displayed in Table 9. The findings suggest that switching the target model from DINOv2 to other vision encoders can lead to significant improvements, highlighting the generalizability of our framework. As vision-centrie self-supervised learning progresses and larger image datasets are developed, we expect to see the emergence of even more powerful vision encoders in the future. Our research establishes a foundation for utilizing these advanced encoders to enhance the performance of iision-language models and potentially benefit downstream MLLMs.
Table Accuracy evaluation on zero-shot image classification benchmarks CLIP models w/wo alignment. use MLCD instead of DINOv2 as the target model for alignment. The results show the alignment still show improvement when using different target model.
Zero-shot Datasets
ImageNet
CIFARIO
CIFARIOO
Calfech
OxfordPets
RESISC
Au
PCAM
ImageNet-S
ImageNer-O
Vision Encoder
PER
DTD
Average
Zero-shot
VIT-L-14
74.90
95.20
71.08
83.30
50.00
93.21
55.21
63.35
62.65
52.00
59.59
32.25
65.26
+align
75.32
96.03
74.90
83.86
49.67
93.43
55.59
64.62
63.44
52.28
59.99
34.70
66.26
ViT-L--14-336
75.82
94.49
71.11
83.42
49.04
93.70
55.64
63.73
61.46
60.68
61.03
32.75
66.10
+align
76.18
95.90
76.70
83.53
49.00
94.06
56.60
63.90
61.54
60.66
61.57
36.30
67.25
C.4. Results on MMVP-VLM benchmarks.
We conduct experiments on the MMVP-VLM benchmarks (Tong et al., 2024). The benchmark consists of "CLIP-blind pair" images, which the CLIP vision encoder recognizes as similar, despite notable visual differences. It is designed to test whether the CLIP model can differentiate between these pairs. The pairs encompass range visual patterns, including orientation and direction, the presence of specific features, state and condition, quantity and count, positional and relational context, color and appearance, structural and physical characteristics, text, and viewpoint and perspective. The results detailed in Table 10. Findings indicate that the proposed alignment improves accuracy by 2.97%, 2.96%, and 2.22% for three variants of CLIP models. These results demonstrate that aligning with DINOv2 helps address the limitations of CLIP in these challenging cases.
Table 10, Performance of CLIP on various visual patterns MMVP-YLM benchmark. Alignment with DINOv2 embeddings greatly overcomes CLIP's original shortcomings in of perceiving visual details. Symbols for visual patterns (Tong et al., 2024) inherited: @ Orientation and Direction, Q: Presence of Specific Features, Ø State and Condition, fà: Quantity and Count, 9: Positional
and Relational Context,
Color and Appearance, Structural and Physical Characteristics, A: Texts, O Viewpoint and Perspective.
Vision Encoder
Q
11
$
A
Average
align 6.67
0.00
26.67
13.33
13.3
20.00
13.33
0.00
20.00
12.59
VIT-B-16
w/ align
6.67
0.00
20.00
13.33
20.00
26.67
26.67
0.00
26.67
15.56
w/o align
6.67
13.33
20.00
13.33
6.67
53.33
26.67
6.67
13.33
17.78
VIT-L-14
w/ align
13.33
20.00
20.00
20.00
6.67
53.33
33.33
13.33
20.00
20.74
w/o align
0.00
20.00
40.00
20.00
6.67
20.00
33.33
0.00
33.33
19.26
VIT-L-14-336
w/ align
6.67
26.67
40.00
13.33
6.67
40.00
26.67
13.33
20.00
21.48
C.5. Results on MMVP benchmarks.
We also conducted experiments on the MMVP benchmarks (Tong et al., 2024), which are similar to the MMVP-VLM benchmark but evaluate through VQA on MLLMs. For this experiment, we used LLaVA-1.5-7B with CLIP 14-33-14-336 as the visual encoder. The results are shown in Table 11. We found that the proposed alignment framework significantly
17

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
enhances the performance of LLa leading marked reduction in error rates for these challenging cases. Improvements observed simply by replacing the vision encoder with the aligned version, and the gains even more pronounced when we fine-tuned the LLM component as well. These results further confirm that the enhanced visual capabilities of CLIP can be effectively transferred to downstream MLLMs.
Table 11. Performance of LLaVA MMVP benchmark. The performance improves when we replace the visual encoder with the aligned version. This indicates that alignment enhances visual capabilities, and this effect can be carried over to downstream tasks.
Vision Encoder
CLIP
CLIP+align
CLIP+align+peft
Accuracy (%)
23.33
24.67
34.00
C.6. Additional Baselines
In this section, compare our method with additional baselines, which also entail integrating knowledge from multiple experts to build a stronger visual encoder.
Comparison with AM-RADIO (Ranzinger et al., 2024). AM-RADIO distills knowledge from multiple teacher models into a student model, delivering strong performance across various downstream tasks. However, the original AM-RADIO model is trained on DataComp-1B, a dataset consisting of 13 billion samples. This training process requires computational resources comparable to the pre-training of CLIP, making it highly resource-intensive. contrast, our proposed kernel-based aligmment method achieves strong accuracy and generalizability through fine-tuning on relatively small datasets, such as ImageNet-1k, over just a few epochs. To provide clearer comparison, follow the AM-RADIO training protocol and train a model on ImageNet-Iie Ik using CLIP and DINOv2 as teacher models (both employing ViT-L-14 backbones), with another VIT-L-144 as the student model. We then evaluate its zero-shot classification performance, as shown in Table 12. The results reveal that AM-RADIO struggles to generalize to out-o--distribution datasets under this setup. In contrast, our aligmment method achieves both strong performance and superior generalizability, even with limited data and computational resources.
Table 12. Comparison between alignment fine-tuning and AM-RADIO. Our method represent better performance when the training data is limited.
Zero-shot Datasets
ImageNet
CIFARIO
CIFARI00
CalTech
OxfordPets
RESISC
Eu
PCAM
ImageNet-S
ImogeNer-O
Vision Encoder
PER
DTD
Average Zero-shot
AM-RADIO
75.38
95.04
71.48
78.47
25.66
83.62
45.37
38.48
31.35
50.02
51.34
69.45
58.21
Ours
75.52
96.27
77.04
84.32
50.31
93.40
55.74
63.83
63.83
52.68
59.67
34.90
66.54
Comparison with Additive-MoF (Tong et al., 2024). We also compare our method with Additive-MoR, a straightforward approach combining visual representations from different visual encoders for MLLM applications. This comparison conducted on vision-language benchmarks using the LLaVA implementation, with the results presented in Table 13. While Additive-MoF achieves better performance in certain tasks, such object detection, it performs suboptimally in others, such as open-ended visual question answering. As highlighted in the original paper (Tong et al., 2024), simply combining CLIP and DINOv2 features involves an inherent trade-off between visual grounding accuracy and mustruction-followe capability. In contrast, our alignment fine-tuning approach enhances visual capabilities while maintaining strong alignment with the textual embedding space. Additionally, our framework does not require full re-tuning of the LLM component, making particularly beneficial in resource- limited settings.
18

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
Table Comparison between Additive-MoF and ours on vision-language benchmarks. Our alignment fine-tuning approach achieves
enhanced visual capabiliiiie without compromising alignment to the textual embedding space
Vision Encoder
A12D
text-ocr
text-puie
pope
aallyAA
vSr
RefcocOg
ReicOCO+
ReCCOCO
v9a-v22
llava-wild
flava-coco
Average
Additive-Moo
51.36
45.93
18.66
86.58
63.18
51.47
68.44 44.44
65.34
69.69
75.12
41.54
74.50
59.32
Ours
53.95
58.89
50.51
86.05
62.14 14
52.55
58.68
57.74
65.79
76.97
52.20
78.90
62.86
C.7. Analysis of Ablation Studies
We conduct several ablation studies to examine the effects several design choices in our framework. The experiments are performed with CLIP ViT-L-14, and average zero-shot accuracy across 11 datasets used evaluation metrics.
68
75
68
Accuracy (%g
67-
70
Accunicy (%)
67
Accuracy (%) (P6)
67
66
06.51
06.54
66.31
Accuracy
65
65.34
66
06.54
66.9
66
00.54
6611
65-
60
65
65-
57.78
64
55
64
64
baseline
10.0
baseline
poly. Goussion cosine
A uf epechts
w/wu regulatization
coefficieni W
kemel function
Figure 5. Ablation studies. Average zero-shot accuracy 11 datasets reported evaluation metrics: Effects of training
epochs; (b) Effects of the regularization term; (c) Effects of the coefficients of alignment; (d) Effects of different kernel functions.
Effects of Training Epochs. We experiments with fine-tuning for more epochs to see how it affects the performance. The results are shown in Fig. (a). We find that the performance stabilizes after training for epochs. Further training leads to no improvements. This finding highlights the effectiveness of our proposed method, which achieves strong performance with minimal training time.
Effects of Regularization. We also conducted an ablation study on the regularization term in Eq. 6, specifically fine-tuning the CLIP model using only the kernel alignment term. The results are shown in Fig. 5 (b). Without this regularization term, we observe a significant drop in performance, with zero-shot accuracy even worse than that of the original CLIP model. This indicates that the regularization is crucial for maintaining alignment with the text embeddings. Only when both terms are utilized does the alignment phase achieve an effective balance between fine fine-graine grained visual capability and compatibility with text semantics.
Effects of Coefficients. We then tested various coefficients for the alignment term in Eq. 6, experimenting with values of 0.1, 0.5, 1.0, 5.0, and 10.0. The results are presented in Fig. 5 (c). We found that performance remains relatively stable across these coefficient changes. In our experiments, a coefficient of 1.0 yielded the best performance, effectively balancing alignment with the DINOv2 representation and compatibility with the text encoder.
Selection of Kernel Function. For the main experiment, we employed normalized polynomial kernel alignment and also tested the Gaussian kernel and cosine kernel, with results shown in Fig. (d). Our findings indicate that all three kernels lead to performance improvements, with the polynomial kernel achieving the best results. The Gaussian kernel encounters gradient vanishing issues, complicating the optimization process, while the cosine kernel lacks additional parameters to flexibly address differences in feature dimensions and structures, making them sub-optimal compared to the polynomial kernel, One potential avenue for future improvement is to explore more diverse kernel functions or to combine multiple kernel functions as a compositional kernel for alignment.
Effects of the dataset size, We investigate how the size of the fine-tuning dataset impacts model performance. To this end,
19

---

Kernel-based Unsupervised Alignment of CLIP and DINOv2 Embeddings
perform ablation studies using 25% and 50% of the ImageNet dataset for fine-tuning. The results, presented in Table 14, indicate that alignment fine-tuning consistently enhances model performance, when only 25% of the samples used, demonstrating the method's efficiency. Furthermore, the performance improvement grows the amount of fine-tuning data increases, highlighting the potential for further gains when larger datasets are utilized.
Table 14, Ablation study in terms of fine-tuning data size. We conduct the alignment fine-tuning with different proportions of ImageNet data. The model demonstrates consistent performance gain, and the performance improves as more data incorporated.
Zero-shot Datasets
ImageNer
CIFARIO
CIFARI00
CalTech
OxfordPets
RESISC
EuroSAT
PCAM
ImageNNet-S
ImageNet-O
FER
DTD
Proportion
Average Zero-shot
0%
74.90
95.20
71.08
83,30
50.00
93.21
55.21
63.35
62.65
$2.00
59.59
32.25
65.26
25%
75.08
95.84
74.13
83.89
50.59
93.40
55.32
64.33
61.54
52.02
59.96
35.30
66.03
50%
75.34
95.87
76.05
83.91
50.29
93.59
55.90
64.41
64.00
52.53
59.98
34.75
66.48
100%
75.52
96.27
77.04
84.32
50.31
93.40
55.74
63.83
63.83
52.68
59.67
34.90
66.54
20


Reviews:
[{"review_id": "bal9ZnHdli", "reviewer": "Reviewer_ue7L", "summary": "This paper employs a kernel-based method to align CLIP's visual representation with DINOv2's representation. The goal is to enhance the CLIP vision encoder's fine-grained perception capabilities. The resulting aligned visual encoder demonstrates improvements in zero-shot object recognition, fine-grained spatial reasoning, and localization abilities.\n\n## Update after rebuttal\nMy major concern is addressed during the rebuttal. Thanks the authors for the additional experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer ue7L for the thoughtful and constructive feedback on our work. The following is our response to the comments and questions in the review.\n___\n> **Comparison with DINOv2 encoder**\n\nWe would like to clarify that our numerical evaluation of the fine-tuned CLIP using alignment with DINOv2 performs experiments of the following two types of image-centric models:\n1. Text-image cross-modality tasks such as zero-shot classification and image-text retrieval: We note that DINOv2 is not applicable to these tasks, since they require a text encoder that is not existing in DINOv2. This is why we did not report the performance of DINOv2.\n2. Image-only tasks: In these experiments, our goal is to highlight that aligning CLIP with image-modality expert DINOv2 enhances CLIP's visual performance. Per the reviewer's suggestion, we include DINOv2's performance on linear probing and probing bench. Results show DINOv2 significantly outperforms CLIP on purely perceptual tasks (e.g., spatial reasoning, locality probing), with alignment successfully narrowing this gap. For tasks requiring both visual and multimodal abilities (e.g., caption recognition), the aligned model leverages strengths from both models, outperforming each individual model.\n\nIn both settings, we aim to show that aligning with modality experts improves multi-modal models, that is why we aligned CLIP with DINOv2 rather than the opposite, and focused on the comparison with the original CLIP. We will include the additional comparison with DINOv2 in the revised text.\n|Task|svhn|gtsrb|clevr distance|clevr counts|probing local|probing global|\n|-|-|-|-|-|-|-|\n|CLIP|65.20|72.49|22.97|41.25|46.40|54.51|\n|DINOv2|49.00|61.25|47.81|65.43|49.75|57.01|\n|ours|69.39|74.51|30.82|49.67|47.44|55.33|\n___\n> **The explanation for additional PEFT**\n\nThank you for pointing this out. Since we have fine-tuned the visual encoder, we also attempted to finetune the downstream component so that we can achieve even a stronger improvement. Because our alignment method ensures the visual encoder does not deviate significantly from the original model, fine-tuning the downstream component can be efficiently performed using PEFT. Following the reviewer’s suggestion, we further experimented with standard two-stage training, and compared the CLIP encoder w/wo alignment. In our experiments, we observed that this led to even greater performance and confirmed that the alignment improves the MLLMs performance. \n|Task|AI2D|text-ocr|text-pure|pope|tallyQA|vsr|RefCOCOg|RefCOCO+|RefCOCO|vqa-v2|llava-wild|llava-coco|average|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|w/o align|53.27|57.82|48.63|85.45|61.45|51.47|55.19|51.97|60.49|76.40|42.97|75.60|60.06|\n|w/ align|53.95|58.89|50.51|86.05|62.14|52.55|58.68|57.74|65.79|76.97|52.20|78.90|62.86|\n\nDue to the time limit, we could not finish the experiment with DINOv2. However, [[Karamcheti et al., 2024](https://arxiv.org/abs/2402.07865)] has shown that DINOv2’s MLLM performance does not match that of CLIP, which may due to its image-only pre-training protocol.\n___\n> **Explanations  on “projection” and “feature” methods**\n\nThe term “projection” in Table 1 refers to the case with a trained projection head to project the DINOv2 features into the CLIP feature space. Specifically, we freeze the CLIP and DINOv2 encoders, and train a projection head $P(\\cdot)$, to minimize the ${\\mathbb{E}}[\\Vert P(g(I_i)) - f(I_i) \\Vert^2]$. Then we use $P(g(I_i))$ as the visual representation to conduct zero-shot classification. The term “feature” refers to aligning the CLIP and DINOv2 representations directly, subject to a linear transformation. Formally, we fine-tune the CLIP encoder to minimize the following objective:\n\n$\\min_{\\theta, \\mathbf{R}} {\\mathbb{E}}[w\\Vert f_\\theta(I_i) - \\mathbf{R}g(I_i)\\Vert_2^2 + \\Vert f_\\theta(I_i) - f_{\\theta_0}(I_i)\\Vert_2^2]$\n\nWe will make it more clear in the revised text.\n___\n> **Effects of the regularization term**\n\nRegarding the reviewer's suggestion, we would like to note that there is an ablation study of the regularization term in Fig. 3(b). Removing the regularization term results in a significant drop of the zero-shot classification performance, which shows its importance for maintaining the visual-text alignment. For a more detailed evaluation, we have run a new experiment where we plugged the encoder fine-tuned without regularization into LLaVA without fine-tuning the LLM. The results below also indicate the performance drop. We will include the additional results in the revised paper.\n|Task|AI2D|text-ocr|text-pure|pope|tallyQA|vsr|RefCOCOg|RefCOCO+|RefCOCO|vqa-v2|llava-wild|llava-coco|average|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|w/o reg.|52.10|50.22|29.90|83.47|57.73|51.47|13.72|11.23|12.38|71.32|38.57|70.47|45.22|\n|w/ reg.|53.66|55.70|41.53|84.92|60.39|51.50|14.15|12.89|15.37|75.10|44.67|76.60|48.87|\n___\n**Reference**\n- Karamcheti et al. \"Prismatic vlms: Investigating the design space of visually-conditioned language models.\" ICML (2024).", "claims_and_evidence": "While the evidence presented is clear, it is not fully convincing in supporting the paper's claims.\n\nClaim 1: The aligned encoder shows improvements on vision-centric tasks.\n\nEvidence 1: The paper evaluates performance on zero-shot object recognition, image-to-text and text-to-image retrievals, counting, spatial reasoning, and caption recognition and localization ability. However, the tables primarily compare results to the original CLIP encoder. The claim would be more convincing if comparisons with the DINO v2 encoder were included for all these vision-centric tasks.\n\nClaim 2: The aligned encoder shows improvements on MLLMs.\n\nEvidence 2: The paper evaluates performance on LLaVA and OpenFlamingo. However, for Table 5, the distinction between baseline and PEFT is unclear. Since the baseline LLaVA-1.5-7B already undergoes two-stage training (pretraining and SFT), the rationale for conducting additional PEFT using the same tuning dataset needs explanation.", "methods_and_evaluation_criteria": "Yes, the kernel-based method makes sense. What if we removed the regularization term? How would that affect the MLLM performance?", "theoretical_claims": "The two proofs are correct.", "experimental_designs_or_analyses": "The experimental design is validity. The ablation study is solid.", "relation_to_broader_scientific_literature": "As mentioned in the paper, kernel methods have also been applied to evaluating the fidelity, diversity, and novelty of generative models.", "essential_references_not_discussed": "All are discussed.", "other_strengths_and_weaknesses": "Strength: use kernel methods to combine the strength of CLIP and DINOv2.\n\nWeakness: no weakness.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. Can you compare the performance of MLLM using CLIP encoder, DINOv2 encoder and your aligned encoder following the standard 2 stage method?\n\n2. Can you please explain more on “projection” and “feature” methods in table 1?\n\n3. This paper use the kernel method to align CLIP to DINOv2. What about aligning the DINO v2 with CLIP model? Using DINO v2 as baseline, use kernel method to align it with the CLIP to get the encoder.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1LVUod7dIi", "reviewer": "Reviewer_3gYu", "summary": "The paper aims to enhance the fine-grained perception capability of CLIP by aligning with DINOv2 in the kernel space. It finds that such a practice can resolve the feature space conflict problem when aligning two different forms of features. Improvements are observed in image-text retrieval benchmarks and MLLM benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 3gYu for the thoughtful and constructive feedback on our work. The following is our response to the comments and questions in the review.\n___\n> **Significance of the improvements**\n\nWe would like to highlight that our alignment fine-tuning achieves consistent improvements on several kinds of vision-centric benchmarks. We improve the performance of CLIP by 1.28%, 1.34%, 5.51%, and 2.96% on zero-shot classification, image-text retrieval, linear probing, and MMVP-VLM benchmark, respectively. For the MLLM benchmarks, we focus on consistent gains across multiple benchmarks and aim to highlight that our kernel-based approach provides a principled and efficient way to enhance model performance, even under resource-constrained settings. Meanwhile, following the suggestions of Reviewer ue7L, we have conducted experiments with standard two-stage training of LLaVA, which shows even greater improvements (under the response for Reviewer ue7L: *The explanation for additional PEFT*). Overall, we think our proposed alignment fine-tuning provides considerable improvements with very low demand for data and computational resources.\n___\n> **Comparison with AM-RADIO models**\n\nWe thank the reviewer for recommending the related work. AM-RADIO [[Ranzinger et al., 2024](https://arxiv.org/abs/2312.06709)] distills knowledge from multiple teachers into a student model, which achieves superior performance on multiple downstream tasks. However, the original AM-RADIO model is trained with DataComp-1B, which is composed of 13B samples. This training process incurs a computational cost equivalent to the pre-training of CLIP, making it resource-intensive. In contrast, our proposed kernel-based alignment focuses on achieving strong accuracy and generalizability through fine-tuning on relatively small datasets, such as ImageNet-1k, for only a few epochs. To provide a fair comparison under our settings, we trained AM-RADIO on ImageNet-1k using CLIP and DINOv2 as teacher models (both with ViT-L-14 as backbones) and another ViT-L-14 as the student model. We evaluated its zero-shot classification performance and observed that AM-RADIO does not generalize well to out-of-distribution datasets under this setup. In contrast, our alignment method demonstrates both strong performance and superior generalizability, even with limited data and computational resources. We will include the results and discussion in the revised version of the paper.\n\n\n| Task  |imagenet|cifar10|cifar100|caltech|fer|pets|dtd|resisc|eurosat|pcam|imagenet-s|imagenet-o|average|\n|-----|-----|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|-----|\n|AM-RADIO|75.38|95.04|71.48|78.47|25.66|83.62|45.37|38.48|31.35|50.02|51.34|69.45|58.21|\n|ours           |75.52|96.27|77.04|84.32|50.31|93.40|55.74|63.83|63.83|52.68|59.67|34.90|66.54|\n\n\n___\n**Reference**\n\n- Ranzinger et al. \"Am-radio: Agglomerative vision foundation model reduce all domains into one.\" CVPR (2024).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Different from prior works, it presents a new method to align with multiple teacher models.", "essential_references_not_discussed": "AM-RADIO: Agglomerative Vision Foundation Model, CVPR 2024", "other_strengths_and_weaknesses": "**Strengths**\n- The paper is clearly written and easy to follow. The illustrations and tables are also clear.\n- The method is well-motivated and can resolve the feature conflict problem of different multiple teacher models.\n- The experimental results are comprehensive and support the claims.\n\n**Weaknesses**\n- The improvement is slightly marginal compared with the CLIP baseline on MLLM evaluation benchmarks. Morover, the authors does not follow a standard LLaVA training pipeline, e.g., skipping the alignment stage and using LoRA rather than fully fine-tuning. I understand that it is may because the authors are limited by computational resources. But considering the severe fluctuation in MLLMs evaluation results, larger improvements are expected to be observed.\n- More methods should be compared under the authors' settings, such as the AM-RADIO models, which use simple feature alignment to distill features of multiple teacher models.\n\n[1] AM-RADIO: Agglomerative Vision Foundation Model, CVPR 2024", "other_comments_or_suggestions": "See weaknesses", "questions_for_authors": "See weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WLjs5FgwLf", "reviewer": "Reviewer_u8xU", "summary": "In this paper, the author proposed a kernel-based method to tune CLIP models for fine-grained visual information. Specifically, they propose a kernel based objective to align features from the original CLIP and another visual model that is capable of focusing on fine-grained details. Additionally, they introduce a regularization to prevent the tuning deviates from the original too far. Experiments are conducted on visual-centric tasks and MLLM-related tasks. Multiple ablations are also conducted to validate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer u8xU for the thoughtful and constructive feedback on our work. The following is our response to the comments and questions in the review.\n___\n> **Preserving the original CLIP space in the alignment with DINOv2**\n\nWe would like to clarify that our objective function for the alignment task is designed to minimize the changes to the CLIP embedding while aligning it with DINOv2. We agree that we have to make some modifications to the CLIP model to have a higher alignment with the DINOv2. Meanwhile, to minimize the changes to CLIP in alignment, our proposal is to match only the kernel similarity matrices of the CLIP and DINOv2 embeddings, which only focus on the **relative positioning** of the points. In other words, the proposed objective function does not require the absolute position of CLIP embedded point to be identical to that of DINOv2, but the goal is to only obtain similar similarity scores between the two embeddings. In this sense, our alignment method attempts to mitigate the changes to the CLIP model as much as possible. \n___\n> **Relationship with L2SP**\n\nOur regularization minimizes the $L_2$ difference between visual embeddings of the fine-tuned and original encoders, directly preserving visual-text alignment (as demonstrated in Proposition 3.2). We note that this is different from L2SP that penalizes the $L_2$-norm of parameter changes. While L2SP effectively prevents overfitting to the target domain in transfer learning, our direct feature-based penalty aims to maintain compatibility with the text encoder while enhancing fine-grained visual capabilities. We will include the discussion in the revised paper.\n___\n> **Clarification on training loss**\n\nThe choice of the polynomial kernel function is inspired by the literature of generative model evaluation and the KID metric [[Bińkowski et al., 2018](https://arxiv.org/abs/1801.01401)], which is an established metric to measure the similarity between two sets of representations. In the standard KID evaluation, the kernel function is commonly chosen to be polynomial degree 3. This choice has been commonly adopted by other well-known studies in the literature, e.g. [[Stein et al., 2023](https://arxiv.org/abs/2306.04675)],  [[Kang et al., 2023](https://ieeexplore.ieee.org/abstract/document/10224337)]. We have also drawn the kernel loss in Fig. 4 of the Appendix, which remains at a reasonable magnitude throughout training. \n\nFurthermore, we would like to clarify that the choice of poly-3 kernel function is only to have a reasonable similarity score between **already embedded** samples, and the structure of the kernel similarity matrix is supposed to be governed by the choice of embedding (DINOv2 in our work), not by different kernel functions. Considering this, we think our alignment method is flexible and allows the application of different embedding models. For example, we have already attempted to align with MLCD embedding in Appendix C.3.\n___\n> **Additional baseline on imagenet fine-tuning**\n\nWe thank the reviewer for pointing this out. We agree that comparison with a vanilla ImageNet-fine tuned CLIP (without kernel-based alignment) will shed light on how important the role of kernel-based alignment is in the improvement. To address this comment, we tested the following baseline: fine-tuning the CLIP visual encoder with contrastive loss, where the paired text for each image is “This is a photo of {*class name*}”. While this approach improves ImageNet zero-shot classification, it deteriorates the generalization of the model to other datasets.\n \n|methods|imagenet zero-shot|average zero-shot across 11 dataset|\n|-|-|-|\n|baseline|84.10|57.12|\n|ours|75.52|66.54|\n\nIf the reviewer has any specific fine-tuning method in mind, we would be happy to run and report the results during discussion. \n___\n> **Ablation on the dataset size**\n\nThank you for the suggestion. We have conducted ablation studies using 25% and 50% of the imagenet for fine-tuning. The results show that our alignment fine-tuning consistently improves the scores even with 25% of the samples. We will include the results and analysis in the revised paper.\n|proportion of data|imagenet zero-shot|average zero-shot across 11 dataset|\n|-|-|-|\n|0%|74.90|65.26|\n|25%|75.08|66.03|\n|50%|75.34|66.48|\n|100%|75.52|66.54|\n___\n> **Regarding projection heads**\n\nWe would like to clarify that we do not use any projection heads when computing the kernel; it is computed directly on the representations produced by the models. Meanwhile, in Table 1,  we also show experiments on aligning two models with different representation dimensions (e.g., CLIP-B and DINOv2-L), where we observe similar improvements. \n___\n**Reference**\n- Bińkowski et al. \"Demystifying MMD GANs.\" ICLR (2018).\n- Stein et al. \"Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models.\" NeurIPS (2023).\n- Kang et al. \"StudioGAN: a taxonomy and benchmark of GANs for image synthesis.\" TPAMI (2023).", "claims_and_evidence": "Some claims are supported but another important one is not. See below for details", "methods_and_evaluation_criteria": "The evaluation criteria generally makes sense. But additional baselines are required. See below for details.", "theoretical_claims": "Some claims are theoratically supported.", "experimental_designs_or_analyses": "The experimental design makes sense", "relation_to_broader_scientific_literature": "The proposed method could have some boarder relations with general visual and multimodal representation learning.", "essential_references_not_discussed": "See below for details.", "other_strengths_and_weaknesses": "Pros:\n\n1. Experimental analysis is thorough, covering CLIP-based and MLLM-based tasks.\n\n2. Some claims of the proposed method are supported by theory.\n\nCons:\n\n1. One major claim of the proposed method, in line 88-92 in the introduction, is not well supported. Why aligning features in the kernel space can “preserve the integrity of the original feature space while adjusting the similarities among samples based on their visual details.“ is not explained in the paper. In my understanding, the original feature space will be changed once the encoder is trained (which produces features in the space) regardless of what training objectives are used. In addition, the preservation seems to mainly come from the regularization term instead of the kernel objective in the method. The regularization term, however, is not new and was previously proposed by L2SP [1][2]. More analysis is needed to support the claim.\n\n2. The training loss risks being very small. The polynomial kernel takes a degree of 3 in experiments. The objective in Equation 4 further involves a square operation. I am concerned that the loss from the equation could be very small. It could lead to inefficient training, overflow, or gradient vanishing. In addition, this could lead to very limited choices of kernel functions and their coefficients (the authors found gradient vanishing using gaussian kernels in the supplementary.)\n\n3. Additional baseline is required for vision-centric tasks. The baseline CLIP seems to be directly taken from the pretrained version. However, other baselines including the proposed method are tuned on the ImageNet. This raises concern whether the improvement compared with CLIP is from the fine-tuning dataset or the method itself. The CLIP-ViT fine-tuned on ImageNet should also be involved. \n\n[1] Xuhong L I, Grandvalet Y, Davoine F. Explicit inductive bias for transfer learning with convolutional networks[C]//International conference on machine learning. PMLR, 2018: 2825-2834.\n\n[2] Li X, Grandvalet Y, Davoine F, et al. Transfer learning in computer vision tasks: Remember where you come from[J]. Image and Vision Computing, 2020, 93: 103853.", "other_comments_or_suggestions": "Difference between kernel-based objectives and distillation-based objectives (I am just considering this point as an additional opinion not a major weakness.) From the information perspective, the improvement gains mainly come from learning from DINOv2 on the fine-tuning dataset. The kernel-based objective does not introduce additional information to improve the model (same for other distillation-based objectives). The difference between the proposed method and others are just the objective formats. Why or under what circumstances the proposed method is better than others should be further investigated. In my opinion, the fine-tuning dataset could be such a variable. Whether the proposed method is still better than others in a smaller or larger dataset is encouraged to explore.", "questions_for_authors": "Are there projection heads for CLIP or DINOv2 before they are computed in the kernel? It seems that their original dimensions are the same. What if we want to align two encoders with different dimensions and what will be the impact if applying projection?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers recognise the authors'  method to better align VLM image-representations with those of vision only models in order to  efficiently improve sensitivity to image features, particularly fine grained features, not fully captured by the original text alignment objective. This is exemplified and evaluated using CLIP's image representations with DINOv2's. Recognised strengths of the paper include:
* The paper is clearly written and easy to follow. The illustrations and tables are also clear.
* The alignment objective is well-motivated and can resolve the feature conflict problem when aligning two different forms of features. 
* The experimental design is appropriate and the ablation study is solid.
* The experimental results are comprehensive and support the claims.

Identified limitations and concerns include:
* Some explanations could have been made clearer to reviewers not familiar with kernel methods and representational alignment literature.
* The improvements are consistent across conditions, but at times small in size. 

Note that additional results presented in discussion by the authors show additional improvements across a range of conditions which will strengthen the paper by their inclusion. Moreover, the theoretical explanations, justifications and results appear sound and the approach novel. However, no one reviewer championed the paper all only recommending "weak accept", and I have reflected this in my recommendation.
Decision: Accept (poster)