When Kernels Multiply, Clusters Unify: Fusing Embeddings with the Kronecker Product
Authors: Youqi WU
Jingwei Zhang
Farzan Farnia
Venue: NeurIPS 2025 poster ()
Keywords: Embedding Fusion
Multimodal Representations
Kernel Methods
Random Projection
Kronecker Product
Primary area: deep_learning

Abstract: State-of-the-art embeddings often capture distinct yet complementary discriminative features: For instance, one image embedding model may excel at distinguishing fine-grained textures, while another focuses on object-level structure. Motivated by this observation, we propose a principled approach to fuse such complementary representations through *kernel multiplication*. Multiplying the kernel similarity functions of two embeddings allows their discriminative structures to interact, producing a fused representation whose kernel encodes the union of the clusters identified by each parent embedding. This formulation also provides a natural way to construct *joint kernels* for paired multi-modal data (e.g., image–text tuples), where the product of modality-specific kernels inherits structure from both domains. We highlight that this kernel product is mathematically realized via the *Kronecker product* of the embedding feature maps, yielding our proposed *KrossFuse* framework for embedding fusion. To address the computational cost of the resulting high-dimensional Kronecker space, we further develop *RP-KrossFuse*, a scalable variant that leverages random projections for efficient approximation. As a key application, we use this framework to bridge the performance gap between cross-modal embeddings (e.g., CLIP, BLIP) and unimodal experts (e.g., DINOv2, E5). Experiments show that RP-KrossFuse effectively integrates these models, enhancing modality-specific performance while preserving cross-modal alignment. The project code is available at https://github.com/yokiwuuu/KrossFuse.

When Kernels Multiply, Clusters Unify: Fusing Embeddings with the Kronecker Product
Youqi Wu" yqwu24@cse. cuhk, edu hk
Jingwei Zhang" juzhang22@cse cuhk. edu. hk
Farzan Farnia"
farnia@cse. cuhk, edu. hk
Abstract
State-of-the-art embeddings often capture distinct yet complementary discrimina- tive features: For instance, one image embedding model may excel at distinguishing fine-grained textures, while another focuses on object-level structure. Motivated by this observation, propose principled approach to fuse such complementary representations through kernel multiplication. Multiplying the kernel similarity functions of two embeddings allows their discriminative structures to interact, producing a fused representation whose kernel encodes the union of the clusters identified by each parent embedding. This formulation also provides natural way to construct joint kernels for paired multi-modal data (e.g.. image- text tu- ples), where the product of -speciff kernels inherits structure from both domains. We highlight that this kernel product is mathematically realized via the Kronecker product of the embedding feature maps, yielding our proposed Kroxs- Fuse framework for embedding fusion. To address the computational cost of the resulting -igh-dimensional Kronecker space, we further develop JP--KrosFFuss a scalable variant that leverages random projections for efficient approximation. As key application, we use this framework bridge the performance gap between cross-modal embeddings (e.g., CLIP, BLIP) and unimodal experts (e.g., DINOv2, E5). Experiments show that KrossFusee effectively integrates these models, enhancing modality-speciff performance while preserving cross-modal alignment. The project code available at wt/https:///
Introduction
The representation learning literature features a wide range of embedding models, each excelling distinct and often complementary discriminative features. For example, one image encoder may specialize in distinguishing fine-grained categories such as dog breeds, while another captures broader semantic distinctions such as traffic signs. This contrast is illustrated Figure ! the DINOv2 image embedding [53] tends to form clearer clusters for dog breeds but shows less separation among traffic signs, whereas the CLIP image embedding [60] exhibits the opposite trend, achieving better separation for traffic signs while mixing the two dog breeds in the kernel heatmaps. Such observations motivate the following question: how can we systematically fuse multiple embeddings to obtain single representation that combines the discriminative strengths of all its parent embeddings?
In this work, we view each embedding as inducing kernel similarity function that assigns a similarity score to every pair of samples. Evaluating this function over a reference dataset produces a kernel similarity structure-a matrix that reflects how the embedding perceives relationships among samples
Department of Computer Science & Engineering, The Chinese University of Hong Kong
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Image samples clustering
Kernel mat CLIP embedding
Kernel matrix Ky: DINOv2 embedding
Kernel matrix Kj @ Kg= CLIP DINOv2
70
Cluster Scores:
NMI: 75.8 ARI: 68.6
NMI: 71.6 ARI: 59.4
NMI: 84.4 ARI: 76.5t
Figure 1: Heatmaps of RBF kernel similarity matrices for an image dataset with four groundtruth clusters (two dog classes in ImageNet and two traffic sign classes in GTSRB) (left) K₃ for CLIP, (middle) K₂ for DINOv2, (right) K1 @ K2 elementwise product for CLIP and DINOv2's Kronecker product. Unlike CLIP and DINOv2, their Kronecker produet could cluster the four image classes.
and which samples it tends to cluster together. This kernel-based view allows us to analyze and combine embeddings directly in the kernel space, where our goal is to construct fused representation whose similarity structure reflects a strict union of the parent cluster structures: two samples should appear similar only if all parent embeddings agree that they are similar.
natural and principled way to capture this "all must agree" logic is through kernel multiplication, defining the fused kernel kohn.. as the product of the individual kernel similarity functions ky, and ky, of embeddings 4/1 and 1/2
Ke (x. y) = Kip (I, v) Rip (z. y)-
This formulation provides simple yet effective mechanism for combining the discriminative patterns of multiple embeddings: assuming normalized kernel similarity scores bounded by 1, the fused similarity becomes small whenever any parent embedding separates the two samples" P The empirical effect of this operation is illustrated in Figure [ (right). where multiplying the CLIP and DINOv2 kernels clearly separates all four classes that cach model alone fails to isolate.
feature representation whose kernel similarity function equals the product of the individual kernels is obtained by taking the Kronecker product of the individual embedding feature maps, which is well known in the kernel methods literature 65 7 Building on this insight, we introduce KrossFuse Fuse, a general framework for embedding fusion based on Kronecker Kronecker-product embeddings. Unlike simple concatenation of embedding feature vectors, which corresponds to the summation of individual kemels and lacks the discussed cluster unification property, KrossFuse yields a representation whose kernel unifies sample clusters across parent embeddings" similarities as displayed Figures T
Beyond fusing representations within a single modality, KrossFuse also offers a principled way to construct joint kernels for multi-modal data (e.g., image-text tuples). Such joint kernels are applicable to generative and retrieval systems (e.g., text-to-image models) that rely on paired data for training and evaluation, where the product of moodality-speciff domains. Figure 2 iilustrrates this effect in a text-to- eext-to-image generation setting with SD-XL Turbo model [64] while the kernel structures of the DINOv2 image, CLIP text, and their concatenated feature vectors fail to clearly separate the six clusters defined by three professions (firefighter, chef, police officer) and two genders (male, female), their Hadamard product-using the kernel of the Kronecker-fusee embeddings-can can distinctly separate all six groups
kernels captures the joint structure across
While conceptually elegant, the Kronecker formulation incurs a significant computational cost: the dimensionality of the fused embedding equals the product of the input dimensions (for instance, fus- ing 512-dimensional CLIP and 768-dimensional DINOv2 embeddings yields a 119.216-dimmeniomn vector). To address this scalability barrier, we propose RP-KrossFuse tension that efficiently approximates the Kronecker feature space while preserving kernel similarities. This approach retains the theoretical properties of KrossFuse while making it practical for large-scale, high-dimensional embeddings.
random-projection random- projection based ex-
Forr non negative kernels such as the Gaussian RBF or normalized even- wenn-eegree polynomial kernels, the near-zero kernel similarity score directly implies separation, whereas for kernels that can take negative values (e.g.. linear or cosine similarity kernels), the fusion operates through orthogonality if two samples are nearly orthogonal in any parent space, their fused similarity remains near zero, thus preserving separation.

---

Samples from the six text prompt clusters
Kernel matrix Ky (Text)
Kernel matrix + Ky (concatenation)
A skilled portrait picture of male firefighter
A skilled portrait image of female firefighter
skilled portrait photograph male chef
x
A skilled portrait image of female chef
A skilled portrait photograph of male police officer
skilled portrait photo female police officer
Two samples from SD-XL generated images for each text prompt cluster
Kernel matrix Kj (Image)
Kernel matrix Kj Ky (kronecker product)
Kernel similarity heatmaps for (text,image) data with underlying clusters. While the kernel matrix of the concatenated CLIP text and DINOv2 image embeddings blur cluster boundaries, the kernel matrices' Hadamard product (for Kronecker- embedding) separates all the groups.
Figure 2:
CUI Nomki
cfar100
cifer10.
average
Kroes une Korell CORPSE
Towies
magenet
PeNDve Horrel
food701
WP-K/soaFuse
caltech101
Figure The Kronecker product fusion of embeddings in our proposed KrossFuse: The RP- KrossFuse fusion (implemented with Random Projection) of CLIP and DINOv2 could improve the averaged few-shot classification accuracy over CLIP benchmark image datasets.
We further apply the KrossFuse framework to address the performance gap between cross-modal and uni-modal embeddings. Cross-modal models such CLIP, ALIGN (32, and BLIP 38 achieve aligmment across modalities but often lag behind modality-specific experts such as DINOv2 and E5 [74] on domain-speeific benchmarks. The embedding fusion in this setting poses a significant challenge: the uni-modal expert lacks an embedding map for the other modality (c.g.. DINOv2 provides no text encoder). We demonstrate that KrossFuse naturally extends to this case by defining symmetrized embedding for the shared modality and an imputed constant map for the missing modality, ensuring that the Kronecker product remains well-defined and balanced across domains.
We perform several experiments to demonstrate that RP-KrossFus effectively fuses CLIP with uni- modal image (DINOv2) and text RoBERTaa embeddings. achieving competitive modality- specific accuracy while preserving strong cross-modal alignment. For example, Figure shows the improved results the KrossFuse fusion DINOv2 and CLIP few-shot classification over the CLIP model. Our results suggest the AP-KKossFuss framework provides a scalable, training-free ffee mechanism unify embedding structures. summary, our contributions are:
We propose KrossFuse, embedding fusion framework that applies the principle of kernel multiplication to unify the cluster structures of embeddings.
We develop RroKrosFuss,, scalable, aandom-rojecttinn.o mates the Kronecker feature space.
extension that efficiently approxi-
We apply KrossFuse fuse cross-modal and uni-modal embeddings, demonstrating that it can enhance modality-speciff performance while preserving cross-modal alignment.

---

2 Related Work
Cross-modal Embeddings. Recent advances in cross-modal embeddings have bridged the gap
between visual and textual modalities. CLIP [60] is a pioneer model in this field using a con-
trastive learning approach, enabling remarkable zero-shot capabilities. Subsequent models, such as
ALIGN 332 and FLORENCE [79], scaled datasets and refined learning objectives for improved per-
formance. Variants like CoCa 78 introduced captioning objectives, while BLIP and BLIP-2 [38 (3] employed bootstrapping techniques to generate synthetic image-text pairs. Recent works, including OpenFlamingo [5], PaLI [10] and ImageBind [18), extended capabilities to few-shot scenarios and multilingual, multi-modal scaling. However, a trade-off could exist: strong mross-modal alignment often comes at the price of relatively lowering the performance in single modalities. Our work focuses on this trade-off and explores fusing cross-modal and uni-modal strengths into unified embedding.
Combining Representation of Embeddings. Unifying the strengths of different embedding models has been studied in the representation learning literature. The EVA [15 and X-CLIP 45 methods enhance modality-specific performance while preserving cross-modal capabilities. Bertalign [44 aligns multilingual embeddings via parallel corpus supervision, and ImageBind [88] unifies six
modalities using image- saage-paired data. VLMo 6 balances modality-speciffc traits with cross- mooal interaction, while FLAVA (67] and UniCLIP 37 propose unified frameworks for joint representation learning. Different from these training-basee strategics, our method offers a training-fite -eeeing-free approach by applying the Kronecker product of the involved embeddings. By leveraging the symmetrized Kronecker product, our proposed KrossFuse aims to fuse cross-modal and uni-modal embeddings, preserving alignment while improving single-modality performance. To our knowledge, the task of fusing modsl and uni-modal embeddings has not been explored exclusively in the literature.
Random Projections and Kronecker Products. Random projection is a well-established method for dimensionality reduction. The
(JL) lemma 34 guarantees distance preservation in random projection with high probability. Sparse random projections [ and fast JL transforms [2) have been shown to further improve the computational efficiency. The Kronecker product combined with random projection has not been utilized in embedding fusion in the context of cross-modal embeddings. In the existing literature, random projections are applied separately before computing the Kronecker product. We unify the two operations in the RP-KrossFuse approach by using the Hadamard product of random projected embeddings and demonstrate the approximation of the kernel matrix of the Kronecker product output.
Kernel Embedding Methods for Generative AI Kernel embeddings are used for evaluation and
guidance of generative models. The Kernel Inception Distance (KID) [8] introduced kernel -base
evaluation, followed by entropy-based measures including diversity metries [7 [7 17 31 81 54 56
82). Kernel formulations are extended distributed [75) and online evaluation settings [26 25 63
as well as explainable (29 20 [9) and evversity-guidee [30] embedding. It is relevant to explore how KrossFuse could be extended to these frameworks through its product-kernel formulation.
3 Preliminaries
3.1 Kernel Functions and Feature Maps
Consider a data vector x E X. function X X - IR is called kernel function if for every N and set of points TI E X. the following kernel matrix Kx E _g will be positive semi-definite (PSD):
2(x1,21)
k(11,2n)
Kx =
:
21)
k(zn,
The Moore-Aronszajn Theorem implies that &: is a kernel function if there is feature map @ X - R= such that k( y) (ő(x), Φ(v)) is the inner product (denoted by ()) of the representations Φ(z) and d(y). An example is the linear kernel kpm (a, y) = x provided by the standard inner product. Another example is the Gaussian (RBF) kernel function with bandwidth parameter B > 0:
||
012
k(x,, 8) = exp
B

---

The Schur product theorem shows that for every two kernel functions k1, k2, their product k(z, y) := (z, y)ka (z, y) will also be a kernel function. It can be seen that the feature map of the product kernel & is the Kronecker product of the feature maps Φ1, $2 of the kernels kı k2, i.e.:
(xx)
(a) (2(2)
In the above @ denotes the Kronecker product which for matrix A E Rmxd with entries as 's and matrix B E Rxxl defined as:
01,1 B
A B
=
am, B
am
3.2 Cross-Modal Embedding Maps and Kernel Spaces
Consider two data domains X' (e.g. for image modality) and T (e.g. for text modality). We call an embedding map v5 = ( (5T) cross-modal for these two domains if offers modality-specific maps Øx : X + Z and Ør T + that respectively map a vector x E X' in the first domain and a vector E T in the second domain to a shared embedding space i.e. we have (x)(x), (T(() E Z.The standard cross-modal embeddings are trained such that their outputs for relevant paired data point (z,,t t) is properly aligned. This property can be mathematically formulated as for a proper kernel function le : 2 x 2 -> IR the kernel function A (vbx (z), (t)) is supposed to attain high values for relevant paired sample (x, t) ~ Px,r drawn from a ground-truth joint distribution PX,T.
4 FuossFuse: Kronecker Fusion of Embeddings
4.1 Fusing Uni-modal Embeddings using their Kronecker Product
Consider two uni-modal embedding maps 71 : X' + 3, and 72 X - Z2. To fuse ?I and 72. analyze the kernel similarity functions of the two embeddings. Considering kernel functions k1 : 2 x Z + R and k2 Z2 x Z2 + R operating in the embedding spaces, each of the embeddings 71 and 7/2 provide a kernel function for inputs 2, y E X:
(x, v) kn (n)
(r,( = ka(r2(()) (2())
Note that if Φi, d2 denote the feature maps of kernels ki, k2, i.e. kI y) = (0) (z), di (v)) and (2(2, y) = (d2(z), (2()), then the feature map of ky, and by, will be 01 71 and Φ2 o 72. respectively.
In fusing the embeddings, we set the fused kernel function to be the product of the marginal kernel functions ko, - As shown in Figure D this implies that the similarity score between inputs x; y will be low if either of the kernel functions assign low similarity score, i.e, that embedding distinguishes the input types. As a result, the inputs x, will be clustered differently in the kernel-based method their kernel similarity score is minor according to at least one of the embeddings. In the following proposition, we show that the Kronecker product of the embeddings' feature map Φ1 " and Ø2 o 72 possesses the mentioned kernel product property. We defer the proof of the theoretical statements to the Appendix.
Proposition 1. Consider feature maps Φ1 - 62 Z2 - Rd and their corresponding kernel finctions ky, k2. Then, given the kernel functions defined in ②). the product kernel function ky, (x, y) knn (x,, 8) has the feature map @nnm X dd defined using the Kronecker product:
om (x) = Φn (m())) @ (1(((2)))
Therefore, the feature map to combine the two embeddings follows from the Kronecker multiplication of Φi 7I and 22 72, which maps an input E X to space of dimension di d2, i.e. the product of the dimensions of maps Φi and Φ2..
Remark The discussed Kronecker product combination of two embeddings can be further extended multiple embeddings "1i.... Tom Assuming kernel functions k1, k2...., Km (with feature maps
5

---

$1...., pm) ) to operate on the m embedded data, the feature map corresponding to the unified product kernel I (e(()()), (e())) will be
Py (x) = (o((((z)) E Rdiidard...
The above implies that, using the above feature map, the resulting kernel function could distinguish the dissimilarity of inputs v, y. if one of the embeddings can differentiate the two data points.
4.2 Extending Kross use for Kronecker Fusion Uni-modal and Cross-Modal Embeddings
We earlier discussed how to fuse unimodal representations and via their Kronecker product, such that the kernel similarity function of the fused embedding the product of their kernels. However, the challenge in combining moosl embedding & (sxx, (/r) operating on two modalities in X, and a uni-modal embedding y (7:x) of only the modality is the missing operator of y to apply the nons-shared modality T. For example, if suppose v represents the CLIP cross-modal model applying to image X' and text T domains and y denotes the DINOv2 embedding applying to single image X, then we do not have the text part "r multiply to the text part of CLIP model.
To apply the Kronecker product-based fusion of the embeddings, we propose the KrossFuse method, where we define the following symmetrized cross-modal embedding 7 = (Ø,,x, (,,T) to play the role of the uni-modal embedding (7x) in the Kronecker fusion process:
C Vā ,(>x(=))
xxx(2)
V2 d + (1x(x=))
C 2d
3tr((
V
2d
Here, C> is defined as a hyperparameter constant deternining the constant similarity score of every data points in the missing modality. Note that denotes the feature map of the given kernel function for the single-modality embedding, and d denotes the dimension of ø's output vector.
Given the symmetrized cross-modal embedding 7 which also applies to the non-shared modality T, KrossFuse combines the cross-modal embedding e = (txx, tT) (e.g. CLIP) and the uni-modal embedding (e.g. DINOv2) by taking the Kronecker product of v and in each modality as:
Ex(x) = 6 V xx(x)) (x),
(4)
Er(() (o())==(V7700 @ (1)
(5)
Given the combined cross-modal embedding in (4 and kernel function k( y) (ő(z), ((y) for feature map 6, the following inner product will hold for every inputs x, xf E X from the shared modality and inputs t, E T from the non non-shared modality:
Proposition 2.
(Ex(x), (x(()))x(()))) = =<(xx(x), (x), (c ((((((()))
ET(1) = CC CV-00)(() (Ex(e), FT(()) = ((C.)((x)).)))))
As a result, for the merged KrossFuse embedding, the inner product (i.e. the resulting kernel function) between the transformation of two inputs from either of the modalities will be the product of C and the kernel function of the cross-man modal embedding (e.g. CLIP model), except the case of two inputs from the shared embedding where the value will be added to the product of the kernels from both the cross-modal and uni-modal embeddings.
Remark The KrossFuse framework can be similarly applied to fuse a cross-modar embedding with multiple uni-moda embeddings. Such multi- enntddiibbddiingg fusion involves the Kronecker product of the modified embedding for every uni-modal embedding in the unification. Therefore, the KrossFuse algorithm can be applied to fuse each of the modalities of a cross-modal embedding with specialized uni-modal models. For example, each of the text and image embedding of CLIP can be merged with text-specific (e.g. E5) and image-specific (e.g. DINOv2) models.

---

Algorithm Kernel Feature Fusion of Two Embeddings
1: Input: Image samples
{ First image encoder y1, Second image encoder 72, Kernel maps
@1, 02. Projected dim /
2: U1 -Unnffrm[- v33 v33 d. xi /vi
3: U2 Uniformi v3, v do /vi
4: Initialize Zin E LRNxx1
5: for batch 3 in { do
6: V1.8, Φ1 (1 (B²) )) 12,8 t (2(12(Be))
7: Zime 11,,8, U1) O (1/2,8, U2)
8: end for
9: Return 7
RP-KrossFuse: Scalable Embedding Fusion via Random Projection
Although the Kronecker product in KrossFuse can combine multiple embeddings, the dimension of the merged feature vector will be the product of the dimension of individual models, which would be computationally challenging in standard applications. To lower the computational costs, we propose a scalable application of random projection that preserve the inner product (kernel function values) with limited feature size. The proposed extension, which we call RP-KrossFuse, applies random projection each cross-modality embedding (modified cross-modality embedding for an original uni-modal embedding). To do this, for each of the cross-modality embeddings t (#x,1, tT, 1 and 2/2 = (#x,2, 2),,2), we generate random matrix U; E R whose entries are independent random variables with uniform distribution over -3,, v3), that has unit variance. Then, the RP-KrossFuse embedding of each of inputs x E X' and E T will be
(x
V1 (U (U)))x(()) @ (U2(2.(())
(6)
(t) = v7 (U1bLT(t)) @ (U2V2r(1))
(7)
In the above, @ denotes the element-wist wise Hadamard product. This formulation leads to Algorithm T for uni-modal fusion and Algorithm 2 for cross-modal fusion. Note that computing the above RP-
KrossFuse embeddings requires ([(d) d + d2)) operations. Also, the output has / dimensions, which for properly bounded I will be significantly cheaper than didz dimensions in the Kronecker product of the two embedded vectors. Theorem shows the above method can preserve the KrossFuse inner products with high probability.
Theorem 1. Consider n input pairs (t), Suppose max (Ixx,., (z:) (x:)x,;<x)))1 ( S B is morm-bounder for every j E {1,2] and index i E {L..... .n.. Then, for any 0 we have with probability at least --8: 8:
2Ba log (og(2n?/0)
(1(4 (T(()) (iaa((ti) (i..)(t))) (1ar(t))(vaar(t))) s
2B2 log (2n2/8)
(xx) Tx((x)))))))))))))))))))>))))T v),))(x;)))))))))))))))))))))))) 012x)()))) (1. V
In the Appendix, we have also proved Theorem 2 on the extension of the random projection approach to infiniteddimensional shift-invariant kernels, e.g., RBF kernels, via the framework of random Fourier features 61.. We defer the discussion of the shift-invariant kernels to the Appendix A.4
6 Numerical Results
To evaluate the effectiveness of our proposed fusion methods, KrossFuse and RP- KrosssFuse, we
performed several numerical experiments regarding unimodal and cross-modal embedding tasks
(see appendix for implementation details). We tested the proposed and baseline fusion methods in
application to the following embedding models: cross-modal (image,tex)) embeddings of CL.IP (60]
OpenCLIP 28 and LIg (80), image modality embeddings of DINOv2 [53]. Unicom [ and
7

---

20
20
70
CLIP
20
20
70
DINOV2
70
20
70
70
KrossFuse
20
20
70
70
20
20
70
70
Figure 4: Clustering results and kernel matrix heatmaps for CLIP, DINOv2, and KrossFuse on ImageNet dog breeds and GTSRB dataset. While CLIP could not fully separate all the dog categories and DINOv2 struggled in clustering traffic signs, the KrossFuse fusion captured the clusters.
text modality embeddings of S-RoBERTa 62 E5 numerical results are provided appendix C
74] Unless otherwise specified, the additional
Visualization of kernel similarity matrix for the Kronecker product of embeddings. To visualize how the Kronecker product of two embeddings can capture the clusters identified by each of the embeddings, we present the numerical results of performing kernel-basee clustering on the following
image datasets: CUB-200-2011 ten dog breeds from ImageNet- IK 13. GTSRB (69 and typographic attack images by introducing
(73, Oxford Flowers [11], DTD [1, Image- Woof [23] consisting of
Image-Woof
mislabeled red text into 10 ImageNet subclasses following the reference (47). The visualizations
of the kemnel matrices for these datasets highlight how CLIP and DINOv2 could assign differently
structured kernel similarity scores in different image types. We present three representative image clustering results, visualized with kernel matrix heatmaps in Figure In the Image- Moage-WWoff dataset, the DINOv2 similarity map could visually capture the existing three clusters, while the CLIP embedding did not completely separate the three clusters. In contrast, GTSRB dataset, the CLIP embedding could differentiate the three groundtruth clusters in its kernel matrix, while the DINOv2 assigned kernel matrix did not display the three clusters. On the other hand, the Kronecker product kernel matrix in KrossFuse shows clear block -ick--daaonal structures on both datasets.
the three traffic sign classes from
Unimodal Classification Results. To assess the performance of the RP-Krossfuse fusion of repre- sentations CLIP and DINOv2, we performed linear probe classification the following standard image datasets: ImageNet [3), GTSRB [69], and SVHN [50). as well as on out-of-distribution (OOD) benchmarks: ImageNet-A [22) and ImageNet-R (2I]. The results are shown in Table T
where KrossFuse is compared with the four baselines (see detailed implementation in appendix B.2
(1) KPoMRP, which utilizes the Kronecker product of marginal random projections; (2) GATE, simplified implementation of the cfature-of-Experrtr (MoE) (66] paradigm; (3) ATTN, a lightweight feature-level attentional fusion method [84]: and (4) COMM, a MLP projector fusion framework proposed in 33 Notably, RP-KrossFuse obtained 84. 1 1% accuracy on ImageNet, which was improv- ing upon the DINOv2, CLIP image embeddings and also the baselines. While DINOv2 performed competitively on OOD benchmarks. RP-KrossFuss reached better accuracy scores on GTSRB and SVHN. We note that CLIP and DINOv2 would likely excel on certain image categories, and the RP- RP-KrossFuse method seems to consistently fuse the strengths of the two embeddings.
For the text experiments, used the SentEval toolkit [2] evaluate the RP- RP-KrossFuse embeddings compared to CLIP and S-RoBERTa on the following NLP classification benchmarks: MR (58], CR (24). SUBJ [57) MPQA (76). SST2 (68), TREC 772 and MRPC [1.. Comprehensive evaluation can be found in 2
Zero-shot mrdal Alignment. To test whether the improved unimodal performance of our fused embeddings comes without compromising CLIP's zero-shot cross-modal alignment, we visual- ized the cosine similarity distributions of positive and negative image-text pairs on the MSCOCO
8

---

Table 1: Linear probe evaluation of embeddings on various image benchmarks (IN: ImageNet).
'Projection dimension of RP- ProssssFFuse and KPoMRP: typically 3000 (except 5000 for IN).
Embedding
IN
GTSRB
SVHN
IN-A
IN-R
CLIP
60
73.2
83.1
63.6
23.2
60.0
DINOv2
53
83.3
72.5
60.5
48.5
68.8
GATE 66
81.4
82.2
66.0
38,9
59.1
ATTN
84
79.5
77.3
64.6
38.9
61.5
KPoMRP
79.4
71.8
49.4
34.8
55.2
COMM 33
82.7
76.7
65.5
44.7
63.3
UP-KKrosFuuee
84.1
82.7
47.6
67.4
Table 2: Linear probe evaluation of frozen features of variants of CLIP, Sroberta, RP-KrossFuse and three baselines using the SentEval toolkit on text benchmarks. Test accuracy (%) are based on
5-fold cross-validation. The projection dimension of KPoMRP and RP- UP-KrossFuse is 3000.
Embedding
Arch
Fused
MR
CR
SUBJ
MPQA
SST2
TREC
MRPC
Avg
CLIP 60
ViT-B/32
75.8
83.1
92.5
86.4
82.0
83.0
70.1
81.8
ViT-L/14
78.1
85.3
93.8
87.0
83.9
86.4
67.7
83.2
KPPMRP
ViT-B/32
73.8
78.5
86.6
82.1
80.3
74.8
70.8
78.1
ViT-L/14
72.5
80.3
86.0
83.1
74.0
78.2
68.1
77.5
GATE 66
ViT-B/32
V
84.8
87.2
94.4
88.5
91.8
89.3
65.5
85.9
VIT-L/14
84.3
87.8
94.5
88.3
91.4
89.3
67.6
86.2
ATTN 84
ViT-B/32
85.7
86.3
93.4
88.8
91.9
88.5
66.2
85.8
VIT-L/14
V
85.7
85.9
94.3
87.8
92.4
86.5
65.9
85.5
FusssssFFuse
ViT-B/32
85.8
88.7
94.4
89.1
89.7
95.0
73.6
88.0
VIT-L/14
86.0
88.1
94.8
89.3
89.8
95.2
73,6
88.1
SRoBERTa 62
TF-1.24
85.1
86.8
93.7
87.7
89.1
93.2
68.1
86.2
validation dataset in Figure 5 The overlapping curves suggest that KrossFuse could preserve the geometric distribution of both positive and negative image text pairs. We report zero-shot image- to-text and text-to-image retrieval results on MSCOCO 42 and Flickr30k [77] in table 6 of the appendix. We can observe that the differences between CLIP and RP-KrossFuse are mostly below 1 1%, suggesting that RP Pi--KrssFFue maintains strong zero shot cross-modal alignment, with retrieval performance comparable to CLIP.
Cross-modal Few-shot Learning. To evaluate how EP-KrossFuse applies cross-modal few- shot learning scenarios, we used samples from one modality enhance few shot learning another modality following the work in [43 which simu- lates data-limited learning tasks. This cross-modal adaptation task is enabled by multimodal founda- tion models like CLIP, which map different modal- ities into shared representation space, thereby allowing text samples to augment image samples. For generic object and scene image benchmarks, we adopt the standard prompt "a photo of [class]" extend this
CLP juitt
CUP Seout
1800
1000
500
9033
Figure 5: The cosine similarity distributions in MSCOCO.
for the text modality. To further show the benefits of enhanced text representations, evaluation to the MSCOCO dataset, where ground-truth captions can be directly used as text samples. As shown in Table our method of cross modality could perform better than CLIP and the other baselines, particularly in and shot cases, highlighting the gains reached by the fusion method in cross-modal transfer learning.
Ablation Studies. To test the effect of the different RP-KrossFuse components, we evaluated the classification accuracy ImageNet and the average accuracy across seven NLP benchmarks in SentEval. shown in Figure 6 (a) Fusing CLIP with image expert embeddings (UniCom, DINOv2) could also improve performance over the individual CLIP embedding. (b) Fusing text-

---

Table Cross-mode few-shot classification results across datasets. "Ours" RRP-KrossFuse (proj.
dim. 3000); "T""TT" denote image/text domains.
Shots
Method
Callech [6
Food [
DTD 11
Aireraft 46
ImageNet
13
MSCOCO
42
Average
CLIP 60 (l)
70.9
37.8
35.4
14.6
24.3
8.7
32.0
CLIP (I+T)
78.9
58.7
44.9
17.8
33.8
44.3
DINOv2 53
84.3
57.9
47.2
15.4
54.0
16.4
45.8
Ours ()
84.6
55.7
48.3
19.4
51.8
21.5
46.9
Ours (I+T)
86.0
64.6
51.7
20.3
54.9
53.5
CL.IP 660 (1)
78.9
44.2
38.4
CLIP 60 (I+T)
82.7
60.7
47.3
19.8
36,0
47.2
49.0
2
DINOv2 53 (1)
88.3
63.4
57.3
17.3
61.9
23.1
51.9
57.3
23.6
55.2
Ours (I+T)
90.1
68.0
59.5
24.8
62.1
51.5
59.3
CLIP 60 (I)
83.3
57.7
51.9
20.6
36.8
23.9
45.7
CLIP [60 (1+T)
84.6
64.8
52.0
21.1
42.4
$7.5
53.7
DINOv2
90.4
69.8
64.0
20,9
67.0
38.5
58.4
Ours (1)
90.8
71.8
64.4
28.1
66.6
52.8
62.4
Ours (I+T)
91.1
73.8
65.0
28.2
67.2
58.1
63.9
CLIP 60 (l)
84.5
65.5
53.7
24.2
42.1
44,9
52.5
CLIP (1+T)
85.8
68.7
54.6
24.6
45.2
61.2
56.7
8
DINOv2 5R (1)
91.4
73.0
69.2
24.5
70.4
53.6
63.7
Ours ()
92.0
75.9
69.2
31.7
70.4
65.7
Ours (I+1)
92.2
76.9
69.4
31.7
70.7
61.9
67.1
(d)
84.30
90-
92
66.23
83 99
Acciroor (4)
88
18.00
83.29
86
84
73.40
wrusiCom
w/DROV2
W/SmRERRS
uneef
Conti
who Imge
Fised wiwe Text Embudding
Figure 6: Ablation Studies. (a) (b) Effect of fusing different image and expert embeddings. (c) Effect of kernel function. (d) Effect of random projected dimension.
expert embeddings (S-RoBERTa, E5) could bring considerable gains compared to the CLIP text encoder. (c) All kernel-based fusions led to performance improvements, with the Cosine and RBF kernels yielding slightly higher gains than the linear kernel. (d) Increasing the random projection dimension would lead to gradually saturated accuracy, suggesting RP-KrossFuse could converge to KrossFuse as the dimension reaches around 3000.
7 Conclusion and Limitations
The proliferation of powerful embedding models across vision, language, and other modalities underscores the need for principled methods to unify complementary representations. This work introduced KrossFuse, Kronecker-produu framework for embedding fusion grounded in the kernll product principle, and its scalable variant RP-KrossFuse, which employs random projection to efficiently approximate the high- -ibhemineniio Kronecker feature space. The proposed formulation provides a simple, training-fiee free mechanism to integrate embeddings from diverse sources succh as cross-modal and domain-speciie models-while preserving the discriminative structure of each.
Despite its efficiency and generality, RP-KrossFuse introduces a few additional hyperparameters, primarily the scaling coefficient C and the projection dimension! 1. The fused embedding may require a higher projection dimension (e.g., 3,000) than individual encoders such as CLIP or DINOv2. While this difference may affect direct dimensional comparability. it aligns with standard practice in representation learning. where embeddings varying sizes are routinely compared. The use of random projection is deliberate design choice that enables scalability without the prohibitive cost of operating in the full Kronecker feature space (on the order of 8 x 10 dimensions). When strict dimensional parity compactness is desired, post-hoc dimensionality reduction via PCA can be applied to the fused embedding. Future work may extend this framework to non-vin visual modalities, multi-way way fusion scenarios, attention-based adaptation modules that refine the fused embedding for task-specific objectives.
10

---

Acknowledgments
The work of Farzan Farnia is partially supported by grant from the Research Grants Council of the Hong Kong Special Administrative Region, China, Project 14209920, and is partially supported by CUHK Direct Research Grants with CUHK Project No. 4055164 and 4937054. The authors acknowledge the support from the Hong Kong Research Grants Council (RGC) and the Hong Kong PhD Fellowship Scheme (HKPFS) award supporting Youqi Wu's research. Also, the authors sincerely thank the anonymous reviewers for their insightful comments and constructive suggestions.
References
[1] Dimitris Achlioptas. Database-friendly random projections: coins. Journal of computer and System Sciences, 66(4):671-687, 2003.
with binary
[2] Nir Ailon and Bernard Chazelle. The fast johnson-llnneenstruss transform and approximate nearest neighbors. SIAM Journal on computing, 39(1):302-322, 2009.
[3] Xiang An, Jiankang Deng, Kaicheng Yang, Jaiwei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Unicom: Universal and compact representation learning for image retrieval. arXiv preprint arXiv:2304.0588/ 2023.
[4] Reyhane Askari Hemmat, Melissa Hall, Alicia Sun, Candace Ross, Michal Drozdzal, and Adriana Romero-Soriano. Improving geo-diversity of generated images with contextualized vendi score guidance. In European Conference on Computer Vision, pages 213-229, Springer, 2024.
[5] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.00 3900, 2023.
[6] Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with mfxture-of-modallee modality-experts. Advances in Neural Information Processing Systems, 35: 32897-32912, 2022.
[7] Alain Berlinet and Christine thomas-Agnan Reproducing Kernel Hilbert Spaces in Probability and Statistics. Springer, New York, 2004. ISBN 978-1-4419-9996-9.
[8] Mikolaj Binkowski, Danica J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. International Conference on Learning Representations, 2018. doi: 10.48550/
18Xiv1180100140
[9] Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. components with random forests. In European Conference on Compuler Vision, pages 446-461. Springer, 2014.
discriminative
[10) Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint 2rXiv:2209.06773 2022.
[11] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[12] Alexis Conneau and Douwe Kiela. Senteval: An evaluation toolkit for universal sentence representations. arXiv preprint 10.3:v/800000..000 2018
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.

---

[14]
William B Dolan and Chris Brockett. Automatically constructing corpus of sentential para- phrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). 2005.
[15] Yuxin Fang. Wen Wang. Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19358-19369, 2023.
[16] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 Conference on Computer Vision and Pattern Recognition Workshop. pages 178-178. IEEE, 2004.
[17] Dan Friedman and Adji Bousso Dieng, The vendi score: A diversity evaluation metric for machine learning. arXiv preprint arXiv:2210.02410, 2022.
[18] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180-15190, 2023.
[19] Shizhan Gong, Yankai Jiang, Qi Dou, and Farzan Farnia. Kernel-based unsupervised embedding alignment for enhanced visual representation in vision-language models. Proceedings of the 42nd International Conference Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 19912-19931. PMLR, 13-19 Jul 2025.
[20] Shizhan Gong, Haoyu Lei, Qi Dou, and Farzan Farnia. Boosting the visual interpretability of clip via adversarial fine-tuning, In International Conference Learning Representations, 2025. URL ptttts:://peenrviee..co. wwt.:orrmuuuee id=khulvzxxPpp.
[21] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8340-8349, 2021.
[22] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15262-15271, 2021.
[23] Jeremy Howard. Imagewoof: a subset of 10 classes from imagenet that aren't so easy to classify, https://gt gi thub. http:///////aaaa//mgeeeeeeee imagewoof March 2019. GitHub repository.
[24] Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the Tenth ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pages 168-177. ACM, 2004.
[25] Xiaoyan Hu, Ho-Fung Leung, and Farzan Farnia. A multi-armed bandit approach to online selection and evaluation of generative models. In Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Research, pages 1864-1872 PMLR, 2025.
[26] Xiaoyan Hu, Ho-Fung Leung, and Farzan Farnia. PAK-UCB contextual bandit: An online learning approach to prompt-aware selection of generative models and LLMs. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 14447-24481 PMLR, 2025.
[27] Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 2:193-218. 1985.
[28] Gabriel Ilarroo, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021.
12

---

[29] Mohammad Jalali, Bahar Dibaci Nia, and Farzan Farnia. Towards an explainable comparison and alignment of feature embeddings. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 26757-26796. PMLR, 2025.
[30] Mohammad Jalali, Haoyu Lei, Amin Gohari, and Farzan Farnia. Sparke: Scalable prompt-aware diversity and novelty guidance in diffusion models via RKE score. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025.
[31] Mohammad Jalali, Azim Ospanov, Amin Gohari, and Farzan Farnia. An information- approach to diversity evaluation of prompt-based generative models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 7634-7643, October 2025.
[32] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and with noisy text supervision. International Conference on Machine Learning, pages 4904 09144916, 2021.
representation learning
[33] Dongsheng Jiang, Yuchen Liu, Songlin Liu, Jin'e e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li. and Hongkai Xiong. From clip to dino: Visual encoders shout in multi-modal large language models, 2024.
[34] William B Johnson, Joram Lindenstrauss, and Gidcon Schechtman. Extensions of lipschitz maps into banach spaces. Israel Journal of Mathematics, 14(2):129-138, 1986.
[35] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[36] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32, 2019.
[37] Janghyeon Lee, Jongsuk Kim, Hyounguk Shon, Bumsoo Kim, Seung Hwan Kim, Honglak Lee, and Junmo Kim. Uniclip: Unified framework for contrastive language-iman pre-training. Advances in Neural Information Processing Systems, 15:1008-1019 2022.
[38] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision- vision-language language understanding and generation. In International Conference Machine Learning, pages 12888- PMIR, 2022.
[39] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models, 2023. URL https //arxiv. www./ogg///bbbbeeeeeeeeeee aas///////////// 12597
[40] Xianhang Li, Yanqing Liu, Haoqin Tu, Hongru Zhu, and Cihang Xie. Openvision: A fully-open, cost-effective family of advanced vision encoders for multimodal learning. arXiv preprint arXiv:2505.04601, 2025.
[41] Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 75112-1227 2022.
[42] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740-755. Springer, 2014.
[43] Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, and Deva Ramanan. Multimodality helps unimodality: Cross-modal few-shot learning with multimodal models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19325-19337, 2023.
13

---

[44] Lei Liu and Min Zhu. Bertalign: Improved word embedding- based sentence alignment for chinese-englise parallel corpora of literary texts. Digital Scholarship in the Humanities, 38(2): 621-634, 12 2022. ISSN 2055-7671. doi: 10. 10.1093/llcffqac(089)
[45] Yiwei Ma. Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. X-clip: End-to- end multi-grained contrastive learning for video-text retrieval. In Proceedings of the 30th ACM International Conference on Multimedia, pages 638 0888647,220 2022.
[46] Subhransu Maji. Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft. arXiv preprint orXiv:/306.55/ 2013.
[47] Joanna Materzyúska, Antonio Torralba, and David Bau. Disentangling visual and written concepts in clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16410-16419, 2022.
[48] Aaron F. McDaid, Derek Greene, and Neil Hurley, Normalized mutual information to evaluate overlapping community finding algorithms, 2013.
[49] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metries for generative models. In International conference machine learning. pages 7176-7185, PMLR, 2020.
[50] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning, In NIPS workshop on deep learning and unsupervised feature learning. volume 201 page 4. Granada, 2011.
[51] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729, IEEE, 2008.
[52] Teng Niu, Shiai Zhu, Lei Pang, and Abdulmotaleb El Saddik. Sentiment analysis on multi-view social data. In MultiMedia Modeling: 22nd International Conference, MMM 2016, Miami, FL, USA, January 4-6, 2016, Proceedings, Part II 22, pages 15-27. Springer, 2016
[53] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07192 2023.
[54] Azim Ospanov and Farzan Farnia. Do vendi scores converge with finite samples? truncated vendi score for finite-sample convergence guarantees. In Proceedings of the Forty-first Conference on Uncertainty in Artificial Intelligence, volume 286 of Proceedings of Machine Learning Research, pages 3272-3299. PMLR, 2025.
[55] Azim Ospanov, Jingwei Zhang, Mohammad Jalali, Xuenan Cao, Andrej Bogdanov, and Farzan Farnia. Towards scalable reference-free evaluation of generative models. In Advances in Neural Information Processing Systems, 2024.
[56] Azim Ospanov, Mohammad Jalali, and Farzan Farnia. Scendi score: Prompt-aware diver- sity evaluation via schur complement of clip embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 16927-16937, October 2025.
[57] Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, pages 271-278, 2004.
[58] Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorizaaiinnnn with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 115-124, 2005.
[59] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 20/2 IEEE Conference on Computer Vision and Pattern Recognition, pages 3498-3505 IEEE, 2012.
14

---

[60] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages 8748-8763. 2021.
[61] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
[62] N Reimers. Sentence-bert, Sentence embeddings using siamese bert-networks. arXiv preprint arXiv: 10084, 2019.
[63] Parham Rezaei, Farzan Farnia, and Cheuk Ting Li. Be more diverse than the most diverse: Optimal mixtures of generative models via mixture-UCB bandit algorithms. In The Thirteenth International Conference on Learning Reprexentations, 2025.
[64] Axel Sauer, Dominík Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024.
[65] Bernhard Schölkopf and Alexander J. Smola. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge, MA, 2002. ISBN 978-0- 262-19475-4.
[66] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv: 1701.06538.8.. 2017.
[67] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages I 15638-15650, 2022.
[68] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, 2013.
[69] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks, 32: 323-332, 2012.
[70] Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. Mobileclip: Fast image-text models through multi-modal reinforced training. In Proceedings of the JEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15963-15974, 2024.
[71] Nguyen Xuan Vinh, Julien Epps, and James Bailey. Information theoretic measures for clus- terings comparison: a correction for chance necessary? In Proceedings of the 26th annual international conference machine learning, pages 1073-1080, 2009.
[72] Ellen M Voorhees and Donna M Tice. Building question answering test collection. In Proceed- ings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 200-207, ACM, 2000.
[73] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[74] Liang Wang, Nan Yang, Furu Wei, Xiaojun Huang, and Binxing Ma. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint 12Xiv:2212003333, 2023.
[75] Zixiao Wang, Farnia Farzan, Zhenghao Lin, Yunheng Shen, and Bei Yu. On the distributed evaluation of generative models. In Proceedings of the IEEF/CVE International Conference on Computer Vision Workshops, pages 7644-7653, 2025.
15

---

[76] Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2-3)1165-2 2005.
[77] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the association for computational linguistics, 2:67-78, 2014.
[78] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[79] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432 2021.
[80] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. Proceedings of the IEEE/CVF international conference on computer vision, pages 1195--1996, 2023.
[81] Jingwei Zhang, Cheuk Ting Li, and Farzan Farnia. An interpretable evaluation of entropy- -bssdd novelty of generative models. arXiv preprint arXiv:2402.17287, 2024.
[82] Jingwei Zhang, Mohammad Jalali, Cheuk Ting Li, and Farzan Farnia. Unveiling differences in generative models: A scalable differential clustering approach. In Proceedings of the Computer Vision and Puttern Recognition Conference, pages 8269-8278, 2025.
[83] Yuhui Zhang, Elaine Sui, and Serena Yeung-Levy. Connect, collapse, corrupt: Learning cross- mrosl tasks with uni-modal data. arXiv preprint arXiv:2401.0856) 2024.
[84] Taoxu Zhao, Meisi Li, Kehao Chen, Liye Wang, Xucheng Zhou, Kunal Chaturvedi, Mukesh Prasad, Ali Anaissi, and Ali Braytee. Enhancing sentiment analysis through multimodal fusion: bert-dinov2 approach, 2025.
16

---

A Proofs
A.1 Proof of Proposition
To show the proposition, we only need to note the following about the inner product of the feature map ,22,0 for two samples x, y:
(n)) (r) (0i(M1())) @ (
( (m((W)))))))) @ (xs(1(()))
())())))) (1((()))
(d2(11(1)) ((
kyn (2,, v) @ (x,1)
kn (x, v) (x, 9)
A.2 Proof of Proposition] 2
To show the proposition, validate the claimed identities one by one. For the first equation, note that
(xx(x), = (x()x(x)) @ xn)((a))
o o
(d(Vx(x)) ((x((+))) R (xxx(1)
= ((Xx(=), (xxx(()
d 2d 2d 1 2 2 (
A(Wx(x)
= =(Vx ((Vx()),Xx(x') (C+ (2+)
For the second identity, note that
(tr))),r,(t) (o(V)(t)) @ ) (o(U)(t)) @ (
(d(V)(1) (VV((())) @ ((x((()))((1))
(VP((t), VT (t. (+x((()))))(
(VV((),, vT(( C
2d 2d
(VVr(t), (t)) C
Finally, for the last identity, we can complete the proof as:
(Ex(x), (o(Ux(*)) @ x()) (d(Vr(()) @ )t)((1)
(0(Vx(x)) (())(()) (1xx()))nn((1)
((Vx(x), 2 (T)T)(((1)
1 C
k(VT((), vT(())
2 à 2d
(VVx(x),VT()) C.
A.3 Proof of Theorem 1
To show this statement, we observe that the Kronceker product embedding can be written as:
17

---

(12x(x))))11xx01)) - (Vix((x)) @ (2xx())) (Vix(Z)) & (2.((()))
(ax) (w)x(1))) e (exx()))2xx()))
(Vixx(()) ( (Lx(()) (V2x(1) (22x(1)))
Eu, [ Tu (ubx)(())))) Bus (2;x(a)) JuJ (2wax((3))
= Eun (1)xx(()) un ((Ix(1))) (V2.x(ar)) Tu u2 (1z))
where in the above i1 E Rd and u2 E R are independent random vectors with zero mean and covariance matrix Ta xd, and Idy can seen that each row of the matrix U1 and U/ with uniformly-drawn entries over 1-v3, v3) satisfies this property. As result, for every row u1 of U₁ and row U2 of U2 (with randomly drawn entries with the specified distribution) we have:
uu.. U UX..u.. (2:) "x uz, (z)) = V1.2.( (122x(22))
Note that have (x;) (xx(a)) other hand, we know that each random variable in this empirical mean is bounded as |xx,a, (z:) T (15)) < a,, ( (2:)ll2 I B. Therefore, we can apply Hoeffding's s inequality to show that
+ "x ox (2:) T M2. (zz). On the
-le²
IP ((xx(x)) vx(xx) - (x;) (1 2 < exp
2B
Setting 8 = 2n exp
-lea 213
which implies that
2B log(2n"/ /8)
shows that for every 1 S i,j s
n we have that
P( xx
(xx)) (xx)
(x(x,)) (xx))xxxxx(() 112.xx (x:) (T 2 2
2B² log (og(2n//0)
8
Thus, applying the union bound all n² pairs (i,j) fl......?? shows that
2 (og(2n2/0)
P Vi.j (xx) (x,))) - (x;)) v12))(5))))) s
2 1 -
1
We can similarly prove the above result for the other modality:
2B log(2n? 2/)
P Vi. (tx))t)))tt) (t.) (1ar(t)) ( sv
1 1-166 8.
This completes the proof.
A.4 Extending KrossFuse to Infinite-dimension Shift-Invariant Kernels
As we discussed in the main text, feasible application of KrossFuse requires feature map á X' + IR* mapping to a finit-ddimensinnll space with s < co. However, this assumption does not apply to popular shift-invariant kernels including the Gaussian (RBF) kernel (gusssion(())) y)
exp(-1|= vll//202) and the Laplace kernel hipplaco()) (z, v) = exp(-Ila v/l1/1)-
To extend the KrossFuse application to general shift-invariant kernel k(z, y) <(x y) without any assumption on the finiteness its feature map, we propose the application of the random Fourier feature (RFF) framework in [1]. Note that if shift-invarial map k(z,,) v) <(x ((z--)) for k. X R satisfies the positive semi definief property a kernel function, Bochner's 's theorem shows that the Fourier transform k Rd + IR of к will take real nog-nieative values everywhere, R(w) 2 O. Vi. Note that we define the Fourier transform as follows where (w, æ) denotes the standard inner product in the X' space.
R(w) (2m)d / <(x) (iw x)))
(8)
18

---

Given the above definition. it can be seen that the synthesis equation implies k(0) I. dw which means valid probability density function (PDF) for every normalized shift-invariant kernel & satisfying k(r,,)) = x(0) = 1.
Therefore, the synthesis equation shows that
k(z, v) k( y)
J R(w) exp (i (x v))dw
J ((w) cos(w (x (x- dw
EunR cos( (x - -1)]
E
cos(ww) 2) cos(w v) + sin sin(wT (ww x) sin( U)
[coos (wo و), sin sin(w" T x)] T [cos(w] T v) sin (w w)]l
= E
Note that given a single embedding 7 X' - Rdd, the above characterization leads to the standard RFF framework, where for RFF feature size E N, we draw IID random samples wi...... wp K and define the following RFF proxy map Φ4 : IRd + IR22:
(x)=
1 Vr
coo cos 2), sin [w] :),... (o,( 2), sin =)]
However, the application of the RFF framework the Kroncker product of the embeddings " and 22 remains unclear. In this work, we propose a joint sampling of RFF features for the Kronceker product of embeddings. More specifically, suppose we consider shift-invariant kernel functions ki (xx,x))
(x x²) for embedding map and k2(x,x²) = N2 (2 x') for embedding map 72- Then, we consider the joint probability density function M (2) Ri (wr) R2(W2) for independent variables W1,42. 42.
For applying the RFF framework the Kronecker product of the embeddings under chift-invaraint kernels, we propose IID sampling (wf", w((") ~ M, i... we draw the r samples jointly, instead of generating them separately for each samples and consider the grid-based pairing of the drawn samples. Given the r drawn samples (w )
i==1 we define the following joint RFF feature map:
(=1, 1 cos (21 wi (1) + zy ww(() sin wi)) + 2ω w/((")
(r) cos [2] (xT 2Φ7 wlrr) sin(2) w(th) 2 w
(10)
Similar to the standard case in the single embedding application of Fourier features, we can prove the following proposition:
Theorem 2. Consider two embedding maps 7/1 X' + Rd and 7/2 : X + Rd applied with shift- invariant kernel similarity functions ky Rd x dd + R and k2 : B2d2 Rd + R. respectively. Assume (wi we Yi= are drawn independently according to Ri X Rg, and define the proxy feature map Gr Rdi x Rd + 2R2" as in (10). Then, for every I, E X and every d > O, the following holds with probability at least d:
(6) ())r
2 2og (2/8)
ky ((1()))))))) k2 (172(()))))1)) (Ør (m(x), (. (11)))
r
Proof. Given that ki (2,8) = xi (x - 3) is a shift-invariant kernel that satisfies ki (0) 1, we can deduce that
kn ((()(t) = BuynaR [os((WT (m)(2)) ((me))m(()))) (.))))
19

---

Similarly, we can observe that (1))())))2))))))))) Bugnitig (ws( ra(1))i] which results in
(1(1(()) (=(2(()))
= Buj=Ri
exp(iwn] (m) - -1 (v)))] Buugnig (exp(is)) (12(21) r2(1)))
= )o) x
exp exp [
- (11 (in) (12(2) - (2((1))
exp iw (71 m (v)) + in (12(1) - re(x)))
= (wy) x8²
exp ( (w0 (1 +2) (w) (v) + we (2(0))
(2 x 2
(u (ar XR2
cos (w] (1) m2 (v) + wp ra(v)))
+ i E (w)) 42 J xR
sin [w (1 ip (2))) (w] (1) + w/ r2(1)))
cos (u] w3)))))))))))))))))))))))))))))) (v) wp ra(v)))
(w
Buw XR2 cos ( -os)) (w nvv + r2))
sin (x) + W2 (=())) sin (w) 7/ (1) w/ (
Therefore, since ccos(-) $ 1, the application of Hoeffding's inequality implies that:
(21(t)) ka (22(()))221)) (or (21(1)) (21())))200))) 20)
= P
cos (w] (z) + wi (2())) - (w) + (uun x
Σ
coos [oos((u))) (iui)) (x) w 72(x)) - (wf(TT(()) + w 22 (v)) 2()
m(v)
#=1
-2re2
2xexp
4
=2cxp
2
e= / 2l0r(2/6) If we let 8 2 2exp( -
i.e..,
then we can equivalently write
r
P p( ([ (1)), m(v) ka (a(22(z)) re()) (m(()) or (1)))))))))))
2 2log(2/8) (2/8)
S 8
which completes the proof.
Following Theorem 2, we can extend the random projection fusion of uni-modal embeddings to the settings with shift-invariant kernels that possess an anfiite-dimensionn feature map. In this case, the fusion will consider the jointly-drawn RFFs according to (10) of embeddings ?1 with shift-invariant kernel b and 72 with a shift-invariant kernel 22. This 2r-dimensional vector will be the proxy fusion of the embedding maps (using the proves that the proxy kernel function of this RFF-based fusion is, with high probability, close to the product of marginal kernel functions that is the kernel function of the Kronecker product of the embeddings.
kernel similarity functions K1, k2). Theorem I also
To further extend this discussion to the KrossFuse fusion of the cross-modal (7x,7T) and uni-modal x/x we derive the following formulation. We first generate the RFF features in (TO) to obtain the 2r-dimensional map (x) for the shared modality. Extending the RP-KrrosFuse
20

---

definition to shift-invarian kernels, we obtain the following fused embeddings Vx X' + R and का. T + R for the uni-modal embedding dxx functioning only only the modality X and cross-modal embedding y = (Tx;22))
5x(x) == 22
VC cos (1x (x) T w/ + cos
cos
(7x [Tf" + Mx(x)T=x))
VCcos( (1) Tw) - cos
(xx(1)
xx (x) w(").
vCcos((xx(1) Tw) + sin
(x + (x() ww/"
VCcos(Pi (1) ww) - sin
(x(x)) + xx (x] ww"". (x((1)
VCsin(Ux w(") cos
(x) (x) Twb")
VCsin()x((1) w( cos
(2x(x) (x)) wg").
(xxin(vx(x)) w(") sin
(xx(z) (1) + (x) T()")) w
(Csin(Ux(()) Tw?" w( sin
(1x(#) w( (1) + dx (x)T w
(vx ) + cos
(xx(T)T[) + xx xx( (z) ww!") (xx(x) ww)" 1x (x) ww!")
(C ( w((') cos
(C (1) T Tw(²) + sin (Ccos(ex [w(")) sin
(xx(() w(t + (x) Tw)")
(x(x)) w/ + xx(()) [T) Tub").
VCsin(Vx(()) (x) wh?)) + cos
(x(x)) w/ + xx()) (x)T Tw)"). (xx(x) + xx()) (z) Tw)"). (xx(x) + xb (x) waf").
VTsin Tu/") cos
VCsin(ixx (x) whr)) + sin VCsin(ux(x) Twb")) sin (xx(x) + x/x (x) wfr')
and
(T(( 2r
(T(() VC + cos (m((1) Tuu")) .T - cos (17(()TT[()) vC + sin (Tr(()TT(9)
vC - sin (T7(()/w]]
((r((t Tua cos (tr(t)/w[fl w(ll) sin 1r (t) (tr(t)TT() wfat) - sin (t) (of
cos
VC + VC+
cos (r((1TTff) VC - cos
vC + sin (1r(1) Tw("))
VC VC- - sin (Tr(t) Taf")).
cos (vr(())T(ff adtt) cos ( (t) Tuf")) ssn (Trr(t]T(f*)) sin (vr( Tof))
A.5 Algorithm of Fusing Cross-modal Embeddings and Uni-modal Embeddings
21

---

Algorithm 2 Kernel Feature Fusion of Cross-modal Embeddings and Uni-modal Embeddings
1: Input: Image samples { Text samples {y Cross-modal encoder 7/, Uni-image encoder and Uni-text encoder ya. Kernel maps Φ1, Φ2 9s. Constant Projected dim
2: Uj ~ Uniform[- [ -3, v3) de. X1 /v1, for i = 1,2,3.
3: Initialize ~ E 12N,xx x1 Zie E 12
4: for batch B2 in {a;} do
Process Image Samples
5: V1,B, + Φ1 (n (B2)). V2,B t 22(12(22)))) V3.B. + c. 1x,,
6: 422.8. - concat(cl c1 + 12,,,, c1 - 122
7:
ZB_ + (1,,B, U1) @ (w/2,8, U2) 2 (1',,,UUs)
8: end for
9: for batch By in {uj} } do
D Process Text Samples
10:
VI, + क1 (71 (B,)), 8, + c. 1 V3.s + )b((a(B,))
11:
V.,., + concat(ct cl + Bs, c1 - v3,
12:
Z t U1) @ (12,8, U2) @ (4% Uz)
13:
end for
Return 7img, 7
B Implementation Details
B.1 Experiments Setup
We tested several pre- traineeined embeddings in our experiments, including multiple variants of CLIP 60
DINOv2 (3), Unicom [3. Sroberta [62] E5 74), and Siglip [0 Unless otherwise specified, we primarily report results in the main text using standard Sroberta. The projection matrices U; we generate once using the uniform distribution over [-v3, v3) that has unit variance. The projection dimension / is 3000 for all datasets except 5000 in the case of ImageNet. The parameter C' has been determined using cross-valdiation over {10 3, 10 1102..... 10 All experiments were run on RTX-4090 GPUs.
(VTT-B/B//)) VVNOv2(V1T-B/14) and
B.2 Baselines Setup
Our experiments analyze the fusion of cross-modal and uni-modal representations where we utilize the pretrained expert encoders without additional fine-tuning. our numerical analysis, we consider the last-layer output for every attempted encoder. To the best of our knowledge, the task of fusing cross-modal and uni-modal embeddings has not been exclusively analyzed in the literature. Therefore, we emphasize that the baseline methods discussed in our analysis are fusion methods proposed for two unimodal embeddings, which cannot be applied the zero-shot classification tasks.
The baseline methods in our analysis are: (1) Kronecker Product of Marginal Random Projection (KPoMRP). This baseline applies an independent random projection separately to each embedding output, followed by a Kronecker product to obtain the fused representation. Note that our proposed RP RP-KrossFuse functions differently and samples the random features jointly for the output of the two embeddings. (2) Gated Fusion. This baseline represents the application of the Mixture-of-Exxe Experts (MoE) method discussed in [66), where a gating mechanism dynamically controls the contribution of different feature sources. Specifically, each feature is first passed to an MLP layer, then the outputs are passed through a sigmoid gating function to compute dynamic fusion weight. (3) Attentional Fusion. The baseline follows the self-attend oolf-attntional fusion framework proposed in [84]. Projected features from each encoder are concatenated and passed through self-attention module to dynamically aggregate
information. (4) COMM. This baseline follows the fusion framework proposed in (3). They employ
an MI.P layer to project the features of DINOv2 and concatenate the output features with that of CLIP.
22

---

Additional Experimental Results
C.1 Unimodal Clustering
We present the complete set of our numerical results of kernel-based clustering the image datasets: CUB-200-2011 77]. Oxford Flowers [TT) DTD (T]. Image-Woof [3] consisting of ten dog breeds from ImageNet-I1 IK [33, GTSRB 66 and typographic attack images by introducing mislabeled red text into 10 ImageNet subclasses following the reference [47] Table f reports the clustering performance scores of different methods on six image datasets, evaluated by Normalized Mutual Information 48 (NMI,, Adjusted Mutual Information 771 (AMI), and Adjusted Rand Index [27] (ARI). Across all datasets and metrics, KrossFuse consistently performs better than individual CLIP and DINOv2, reaching the highest scores across the datasets. Notably, KrossFuse shows improvements on known challenging dataset cases such as Typo-Attacked ImageNet, ImageNet-Dogs and DTD, indicating its capability in capturing discriminative features for clustering tasks. The following figures further illustrate the detailed clustering results these datasets. On top of it, visualize the kernel matrices, as well as the distribution of the embeddings using t-SNE and UMAP, providing qualitative insights into the effectiveness of each method in separating different classes.
Table
Metric
Method
CUB200
Flowers102
DTD
ImageNet-Dogg
GTSRB
Typo-Attacked ImageNet
CL.IP
63.2
80.0
50.9
49.7
49.7
20.1
NMI
DINO>2
85.2
98.7
60.5
86.7
40.2
81.9
KrossFuse
85.6
99.1
62.9
88.3
50.0
87.4
CLIP
45.7
73.2
47.7
49.2
46.4
19.3
AMI
DINOv2
78.3
98.2
57.8
86.6
36.2
81.7
KrossFuse
79.0
98.8
60.4
88.2
46.7
87.3
CLIP
21.1
55.5
37.1
18.6
DINOv2
55.9
94.6
28.1
84.8
12.3
65.9
KrossFuse
56.3
97.0
36.4
86.3
19.5
79.6
23

---

STIHL
sessell gleve
OHERRAA te Blagee
CLP
cassente player
Sespecka Sester
ahurso
cosselll dlada
didirch
evelt Beek
fremch horm
Cranch Deat
DINOVZ
per Sade
Franch hoam
Presenlie degaer
aicurosi
Soaft Foom
sessllle Diew
haneahhate
Te-h
KrossFuse
Sohidingsse
Bee Beaw
Figure 7: Clustering results of CLIP, DINOv2 and KrossFuse embeddings for typographic attacked images (red text is the misleading labels that simulate the attack) from 10 ImageNet classes. KrossFuse could cluster the attacked image classes like DINOv2 while CLIP is mislead by text.
24

---

Typo-attacked ImageNet Dataset
CLIP Kernel
t-SNE
UMAP
NMI: 20.1
AMI: 19.3
ARI: 10.4
DINOv2 Kernel
t-SNE
UMAP
NMI: 81.9
AMI: 81.7
ARI: 65.9
KrossFuse Kernel
t-SNE
UMAP
NMI: 87.4
AMI: 87.3
ARI: 79.6
Figure 8: Comparison among CLIP, DINOv2 and KrossFuse embeddings for typographic attacked images from 10 ImageNet classes. (Left) Heatmaps of RBF kernel similarity matrices, (Middle) t-SNE visualization, (Right) UMAP visualization. FusssFuse could cluster the attacked image classes like DINOv2 while CLIP is mislead by text.
25

---

20
20
20
STOP
STOP
STOP
20
20
20
STOP
STOP
STOP
CLIP
20
20
20
STOP
STOP
STOP
120
20
50
50
50
120
120
120
50
50
50
50
50
50
STOP
STOP
STOP
STOP
STOP
STOP
DINOv2
STOP
STOP
STOP
20
20
120
120
20
50
50
50
50
120
50
120
120
120
50
120
20
20
20
STOP
STOP
STOP
20
20
20
STOP
STOP
STOP
KrossFuse
20
20
20
STOP
STOP
STOP
120
200
120
50
50
50
120
120
50
50
50
120
120
120
50
50
50
Figure 9: Clustering results of CLIP, DINOv2 and KrossFuse embeddings for GTSRB dataset with eight clusters. KrossFuse could cluster the eight image classes like CLIP while DINOv2 can't t distinguish them.
26

---

GTSRB Dataset
CLIP Kernel
t-SNE
UMAP
NMI: 49.7
AMI: 46.4
ARI: 18.6
DINOv2 Kernel
t-SNE
UMAP
NMI: 40.2
AMI: 36.2
ARI: 12.3
KrossFuse Kernel
t-SNE
UMAP
400%
NMI: 50.0
AMI: 46.7
ARI: 19.5
Figure 10: Comparison among CLIP, DINOv2 and KrossFuse embeddings for GTSRB dataset with eight clusters. (Left) Hcatmaps of RBF kernel similarity matrices, (Middle) t-SNE visualization. (Right) UMAP visualization. KrossFuse could cluster the eight image classes like CLIP while DINOv2 can't distinguish all of them.
27

---

CLIP
DINOv2
KrossFuse
Figure 11: Clustering results of CLIP, DINOv2 and KrossFuse embeddings for ImageNet-dog breeds dataset. FuossFuse could cluster them like DINOv2 while CLIPP can't distinguish all of them.
28

---

ImageNet-Dogs Dataset
CLIP Kernel
t-SNE
UMAP
NMI: 49.7
AMI: 49.2
ARI: 37.1
DINOv2 Kernel
t-SNE
UMAP
NMI: 86.7
AMI: 86.6
ARI: 84.8
KrossFuse Kernel
t-SNE
UMAP
NMI: 88.3
AMI: 88.2
ARI: 86.3
Figure 12: Comparison among CLIP, DINOv2 and KrossFuse embeddings for ImageNet-dog breeds
dataset. (Left) Heatmaps of RBF kernel similarity matrices, (Middle) t-SNE visualization, (Right) UMAP visualization. KrossFuse could cluster the different dog classes like DINOv2 while CLIP can't 't distinguish all of them.
29

---

WWW
CLI
DINOW
KrossFuse
Figure 13: Clustering results of CLIP, DINOv2 and KrossFuse embeddings for DTD dataset. Kross-
Fuse could cluster them like DINOv2 while CLIP can't distinguish all of them.
30

---

DTD Dataset
CLIP Kernel
t-SNE
UMAP
6M0
NMI: 50.9
AMI: 47.7
ARI: 27.2
DINOv2 Kernel
t-SNE
UMAP
415
4006
NMI: 60.5
AMI: 57.8
ARI: 28.1
KrossFuse Kernel
t-SNE
UMAP
6% 2
NMI: 62.9
AMI: 60.4
ARI: 36.4
Figure 14: Comparison among CLIP, DINOv2 and KrossFuse embeddings for DTD dataset. (Left) Hcatmaps of RBF kernel similarity matrices, (Middle) t-SNE visualization, (Right) UMAP visualiza- tion. KrossFuse could cluster the different texture classes like DINOv2 while CLIP can't 't distinguish all of them.
31

---

CLIP
DINOv2
Kuse
Figure 15: Clustering results of CLIP, DINOv2 and KrossFuse embeddings for CUB200 dataset.
KrossFuse could cluster them like DINOv2 while CLIP can't distinguish all of them.
32

---

CUB200 Dataset
CLIP Kernel
t-SNE
UMAP
NMI: 63.2
AMI: 45.7
ARI: 21.1
DINOv2 Kernel
t-SNE
UMAP
4.00
NMI: 85.2
AMI: 78.3
ARI: 55.9
KrossFuse Kernel
t-SNE
UMAP
2000
8+-
NMI: 85.6
AMI: 79.0
ARI: 56.3
Figure 16: Comparison among CLIP, DINOv2 and KrossFuse embeddings for CUB200 dataset. (Left) Heatmaps of RBF kernel similarity matrices, (Middle) t-SNE visualization, (Right) UMAP visualization. KrossFuse could cluster the different bird classes like DINOv2 while CLIP can't distinguish all of them.
33

---

CLIP
DINOv2
KxossFus
Figure 17: Clustering results of CLIP, DINOv2 and KrossFuse embeddings for Flowers 102 dataset. KrossFuse could cluster them like DINOv2 while CLIP can't distinguish all of them.
34

---

Flowers Dataset
CLIP Kernel
t-SNE
UMAP
NMI: 80.0
AMI: 73.2
ARI: 55.5
DINOv2 Kernel
t-SNE
UMAP
NMI: 98.7
AMI: 98.2
ARI: 94.6
KrossFuse Kernel
t-SNE
UMAP
NMI: 99.1
AMI: 98.8
ARI: 97.0
Figure 18: Comparison among CLIP, DINOv2 and KrossFuse embeddings for Flowers 102 dataset. (Left) Heatmaps of RBF kernel similarity matrices, (Middle) t-SNE visualization, (Right) UMAP visualization. KrossFuse could cluster the different flower classes like DINOY2 while CLIP can't distinguish all of them.
35

---

C.2 Cross-modal Clustering
While our method achieves comparable performance to CLIP in zero shot image-text retrieval tasks, we observed an interesting phenomenon in cross-modal clustering: despite normalization the unit hypersphere. image and text embeddings tend to form separate clusters due to the modality gap 41 33
This separation persists even when both modalities represent semantically identical concepts, indicating a systematic misalignment the embedding spaced To address this challenge, we propose a learned unitary transformation applied to the normalized text embeddings. This rotation operation preserves the geometric structure within each modality while aligning the text embedding space with the image embedding space. Our approach ensures that the semantic relationships are maintained across modalities, enhancing the model's ability to perform cross-modal tasks effectively. We validated our method on the MSCOCO dataset, where we applied t-SNE visualization to demonstrate the elimination of the modality gap. As shown in Figure [9 after applying our unitary transformation, image and text embeddings of the same semantic concepts well-aligned the shared embedding space while maintaining their internal structure.
Images
60
Images Texts
t-SNE dimension 2
40
Texts
t-SNE dimension 2
40
20
22
-20
-40
-40
-60
-60
-60
-20
40
60
-40
-20
60
t-SNE dimension
t-SNE dimension
Figure 19: t-SNE visualization of image and text embeddings on MS COCO dataset. Left: image embeddings (red circle) and text embeddings (blue triangular) form separate clusters due to modality gap. Right: After applying our unitary transformation text embeddings, the two modalities align well in the embedding space while preserving their internal structure, enabling effective cross-modal clustering.
In the following figures, we showcase the clustering results on the COCO dataset after applying transformation. This transformation aligns the embedding spaces, enhancing the clustering of semantically similar image-text pairs.
'we emphasize that this step is limited to the mross-modal clustering setting and was not used in any of the other experiments in the paper.
36

---

Large commercial cargo plane sits tarmac next to radar equipment.
A side view of plane flying in the clear blue sky.
white airplane parked on runway under blue sky.
An airplame leuving trail the
Wite
Two planes in runway and another plane in the sky.
Some people high visibility jackets putting suitcases onto conveyer belt from a container.
black and white image of shipyard with some boats. A boat traveling into water filled tunnel.
couple airplane flying under cloudy sky.
Figure 20: Post-transformation alignment of image and text embeddings on the COCO dataset. The cluster samples shown are all related to planes, demonstrating complete alignment of visual and textual data.
A man his skis snowy slope.
A person skiing through snowy forest trees.
A is his skiing gear, while posing for group of beginner snow skiers having class
male skier going down a hill group of people riding skis snowy surfuce
A woman with snowboard with a man standing next to her ski slope.
person standing snow wearing snow suit and skis
woman posing the standing on skis.
Figure 21: fost-transformation alignment of image and text embeddings on the COCO dataset. The
cluster samples shown are all related to ski, demonstrating complete alignment of visual and textual data.
37

---

person paruchute surling large body of water
There is blurry photo of'a surfer walking out of the water
man riding a blue surfboard on top of wave.
the kid on surbourd riding a little wave
man surfing
suit riding
surfhoard.
man wearing a wet suit riding the wave
man dog riding surf board the water.
A man blue raft attempting to catch ride large wave.
Figure 22: Post-transformation alignment of image and text embeddings on the COCO dataset. The cluster samples shown are all related to surf, demonstrating complete alignment of visual and textual data.
MAST S
A man standing on clay tennis court with racquet.
beautiful young woman swinging tennis racquet on a lennis court.
woman prepares to serve tennis ball.
man holding tennis racket the air
sinsonissoon
Son sson
man hitting tennis ball with tennis racquet.
Venus Williams swinging at tennis at
game.
young boy holding tenmis ruoquet on a court.
man holding tennis racquel top of tennis court.
woman holding tennis raequet while wearing glasses.
Figure 23: Post-rransformation alignment of image and text embeddings on the COCO dataset. The cluster samples shown are all related to tennis, demonstrating complete alignment of visual and textual data.
38

---

C.3 Text Linear Probe Results
To validate the effectiveness of RP-KrossFuse for improving the uni-modal performance, we also conducted linear probe experiments text domain. We used the SentEval toolkit [12 to evaluate the RP-KrossFuse embeddings compared to CLIP, Sroberta and other baselines on the following NLP classification benchmarks: MR [58], CR [24]. SUBJ [57]. MPQA [76]. SST2 [8]. TREC 72 and MRPC [4]. The linear probe results are provided in the Table 5 The results show that uP-KrossFuse significantly enhances CLIP's text understanding capabilities, achieving an average improvement of 6% over CLIP itself. In comparison the baseline, our method demonstrates superior performance across all evaluated tasks.
Table 5: Linear probe evaluation of frozen features of variants of CLIP, Sroberta, RP- K.-KrrosFuuse and three baselines using the SentEval toolkit on text benchmarks. Test accuracy (%) are based on 5-fold cross-validation.
Embedding
Arch
Fused
MR
CR
SUBJ
MPQA
SST2
TREC
MRPC
Avg
CLIP 60
ViT-B/32
75.8
83.1
92.5
86.4
82.0
83.0
70.1
81.8
ViT-L/14
78.1
85.3
93.8
87.0
83.9
86.4
67.7
83.2
KPoMRP
ViT-B/32
73.8
78.5
86.6
82.1
80.3
74.8
70.8
78.1
ViT-L/14
72.5
80.3
86.0
83.1
74.0
78.2
68.1
77.5
GATE 66
ViT-B/32
84.8
87.2
94.4
88.5
91.8
89.3
65.5
85.9
ViT-L/14
84.3
87.8
94.5
88.3
91.4
89.3
67.6
86.2
ATTN 84
ViT-B/32
85.7
86.3
93.4
88.8
91.9
88.5
66.2
85.8
ViT-L/14
85.7
85.9
94.3
87.8
92.4
86.5
65.9
85.5
RP-KrossFuse
ViT-B/32
85.8
88.7
94.4
89.1
89.7
95.0
73.6
88.0
VIT-J/14
86.0
88.1
94.8
89.3
89.8
95.2
73.6
88.1
SRoBERTa 62
TF-L24
85.1
86.8
93.7
87.7
89.1
93.2
68.1
86.2
C.4 Zero Shot Image-to-text and Text-to-image Retrievals
To evaluate the cross modal alignment ability of ours UP-KrossFuse method, we conduct zero-shot image-to-text and text-to-image retrieval experiments on MSCOCO (2) and Flickr30k [77]. In our experiments, we utilize several variants CLIP, including the base, large, and large with 336 pixel models, well as the large and huge versions of OpenCLIP, to ensure comprehensive comparison. The results in table shows that the differences between CLIP and RP- KrossFuse are mostly below 1%, suggesting that RP KrossFuse maintains strong shot cross cross-modal alignment, with retrieval performance comparable to CLIP.
C.5 Improvement of Image Representation vs Alignment of Zero Shot Classification
We performed linear probe and zero-shot classification tasks on multiple datasets covering different vi- sual recognition settings: ImageNet (13), CIFAR-IO. CIFAR-100 [35], Caltech101 [16), Food101 (9) Oxford Flowers [5T], Oxford-IIIT Pet (59 and DTD [I] The detailed results are shown in Ta- ble 7 As shown in Table 7 a). RP-K Pr-KrrssFuse improves the averaged linear probe accuracy over several image benchmarks from 83.3% to 91.2%. more detailed breakdown of per-dataset image improvement is visualized in Figure left where we observe more than 10% increases on 4 out of 8 datasets. Notably, on the ImageNet dataset, RP-KrossFuse irpved over CLIP from 73.2% to 84. 1%, suggesting considerable improvement in the image representation alignment with the actual label. On the other hand, the averaged zero-shot accuracy of Ka-KKrosFFuse drops by 0.5% compared to CLiP(see Table a)). This is indeed expected because we involved uni-moda expert's embedding which is not aligned with the other modality whereas CLIP is trained to align image and text explicitly. Also, we note that most lower scores are underperforming by at most 1% (on out of 8 datasets), which are relatively outweighed by the gains in uni-modal representations shown in Figure 2aaure22000 (left). For example, for ImageNet where RP-KrrosFuse suffers the gap of 1,57% the boost in uni-modal representation of 10. 90% is the second highest accuracy the datasets.
39

---

Table 6: Zero-shot image-text retrieval performance on Flickr30K and MSCOCO. We report Re- call@ K (%) for Image-Text and lexe- retrieval. Fuse denotes our RP-KrossFuse fusion method. Superscripts denote the backhone variant: ViT-B/32, ViT-L/14, ViT- 1/14@ 336px, H ViT-H/14.
Flickr30K
MSCOCO
Method
Image TTexee-TTxtt R@5
Text > Image
Image- Trreee-TTxxc
Texx
Text + -Image
R@I
R@ 10
R@
R@5
R@ 10
R@y
R@5
R@ 10
R@I
R@5
R@ 10
CLIP
94.3
97.8
$7.9
82.9
48.4
73.8
81.6
29.8
54.0
65.0
F R
75.7
94.6
97.7
58.0
82.5
89.0
48.9
72.7
81.3
29.1
53.3
64.4
CLIPE
85.9
97.3
99.2
64.5
87.1
91.9
56.9
79.6
86.7
35.7
60.4
70.5
Fuse
85.0
97.0
99.0
64.6
86.6
91.9
56.5
79.0
86.2
35.7
60.3
70.2
CLIP
87.7
98.5
99.4
66.8
88.9
93.3
57.3
80.2
87.5
35.9
60.7
70.7
Fuse
87.2
98.2
99.4
67.0
88.5
93.0
57.9
80.3
87.0
35.8
60.3
70.4
OpenCLIP4
98.5
74.9
92.4
61.8
83.6
45.5
70.4
Fuse Opent.
89.0
98.4
99.4
74.6
92.5
95.6
62.3
83.8
90.0
45.2
70.2
78.8
OpenCI.p 90.7
99.2
99.7
77.6
94.2
96.6
66.1
86.5
91.9
48.5
72.8
81.0
Fuse Openll
91.0
99.3
99.8
77.5
94.0
96.7
66.3
86.5
92.0
48.4
72.7
81.1
Table 7: Comparison of CLIP and RP-KrossFuse in the linear probe and zero-shot classification setting. LP: linear probe. ZS: zero shot.
(a) Average.
(b) ImageNet.
(c) CIFAR-10.
LP
ZS
LP
ZS
LP
ZS
CLIP
83.3
69.4
CLIP
73.2
57.9
CLIP
95.0
88.8
RP- KrossFuse
91.2
68.9
RP-KrossFuse
84.1
56.3
RP-KrossFuse
98.6
88.4
+7.9
-0.5
A
+10.9
-1.6
A
+3.6
-0.4
(d) CIFAR-100.
(e) Caltech101.
(f) Food101.
LP
ZS
LP
LP
ZS
CLIP
80.0
61.7
CL.IP
92.6
85.3
CLIP
87.3
79.2
RP- FrossssFuuee
90.5
61.0
ProsssssFueee
95.6
85.2
RP-KrossFuse
90.9
78.5
+10.5
-0.7
A
+3.0
A
+3.6
-0.7
(g) Oxford Flowers.
(h) Oxford-IIIT Pet.
(i) DTD.
LP
ZS
LP
LP
ZS
CLIP
84.9
63.5
CLIP
87.9
75.5
CLIP
65.3
43.0
RP- KrossFuse
99.7
62.8
KrosssFFuse
95.3
75.7
RP- Pe-KrossFuse
75.2
43.3
+14.8
-0.7
A
+7.4
+9.9
+0.3
C.6 Analysis of Ablation Studies
To test the effect of the different RP-KrossFuse components, evaluated the classification accuracy on ImageNet and the average accuracy across seven NLP benchmarks SentEval.
Effect of Image Expert Embeddings. To assess the impact of incorporating additional image expert embeddings on the performance of the image modality, we conducted a linear probe on ImageNet by fusing the CLIP embedding with UniCom B as an alternative image expert. As illustrated in Figure 25 a),, integrating UniCom with CLIP increases the accuracy from 73.2% to 79.17%, demonstrating nearly 6% improvement over using CLIP alone. Furthermore, fusing CLIP with DINOv2 yields even higher accuracy than with UniCom, indicating that the effectiveness of OP-KKrssFuse is closely related to the quality of the expert embeddings employed. Specifically,
40

---

vs.
Rnwerr112
+16.00
CTD
+0.27
ImageNet
+10.90
OxfordPets
+0.20
10.50
Caltech101
OxfordPtts
Flowors102
Food101
+3.69
.0.70
Fuod101
GIAR10
-3.60
4.77
CIRAR1D0
Caltecha01
+3.00
-157
Imagehet
Absolutte Improvement
Abvolutw Change
Figure 24: Comprehensive comparisons of RP-KrossFuss and CLIP in the linear probe and shot classification settings. (Left) RP-KrossFUse is able to gain consistent improvements over CLIP in linear probe on all datasets. (Right) RP-KrossFuse' declines in zero shot accuracy are mostly under 1 1%, which are far outweighed by the gains in linear probe.
(a)
(b)
(d)
66.23
24
85 89
Accrsor (86)
88:02
60-
06
50
82
B2
$1.40
75.20
B0
70
80
SIRRERS
600
Uneal
CoSDA
Gaunwah
1000
2000
3000
4000
5000
Fused wwe Imge Eeport Embedd
Floed wlw Text Enbuudcinn
Kamal Pundtion
Figure 25: Ablation Studies. (a) (b) Effect of fusing different image and text expert embeddings. (c) Effect of kernel function. (d) Effect of random projected dimension.
since DINOv2 provides stronger representations than UniCom, the overall performance gain is more pronounced when DINOv2 used as the expert embedding.
Effect of Text Expert Embeddings. To evaluate the generalizability of our approach with different text experts, replace RoBERTa with ES the RP- KrossFuse framework and conduct linear probe experiments using the SentEval toolkit on seven NLP benchmarks: MR [58]. CR (24). SUBJ [57]. MPQA 76. SST2 68), TREC (72), and MRPC [4. As shown Table g incorporating either RoBERTa E5 as an additional text expert consistently improves the performance over the original CLIP text encoder, with E5 yielding an average accuracy gain of 7.4%. These results demonstrate that RP- Ur-osrssFuee can effectively leverage various strong text experts to enhance text representation.
Table 8: Linear probe evaluation of CLIP, E5, SRoBERTa and Uu-KrrssFuse using the SvnlEval toolkit text benchmarks. Test accuracy (%) based 5-fold cross- validatii........ CLIP+E5 and CLIP+SRoBERTa are RP-KrossFuse methods with different text embeddings.
Embedding
MR
CR
SUBJ
MPQA
SST2
TREC
MRPC
Avg
CLIP
75.8
83.1
92.5
86.4
82.0
83.0
70.1
81.8
E5
87.1
91.1
94.7
90.2
92.0
95.7
73.1
89.1
CLIP+E5
86.8
91.3
94.3
90.5
91.9
95.6
74.1
89.2
SRoBERTa 85.1
86.8
93.7
87.7
89.1
93.2
68.1
86.2
CLIP+SROBERTa 85.8
88.7
94.4
89.1
89.7
95.0
73.6
88.0
Effect of Cross-modal Embeddings. To assess the generalization capability of RP-KrossFuse in enhancing unimodal performance while maintaining cross-modal alignment, we further conducted linear probe and zero-shot experiments on the ImageNet dataset using several multimodal embedding
41

---

baselines. As shown in Table RP-K ur-KrossFuse consistently improves linear probe accuracy across different backbones, indicating stronger unimodal representation, while maintaining comparable zero-shot performance, demonstrating preserved cross-modal alignment. These results highlight the generality and effectiveness of FrssrossFuse when applied to various cross-modal embedding architectures.
Table 9: Comparison of linear probe and zero-shot accuracy (%) between baseline cross-modal embeddings and their KrossrssFuse counterparts on ImageNet.
Linear Probe
Zero-Shot
Embedding Model
Baseline
RP- FroKrossFuse
Baseline
RP-KrossFuse
SigLIP
8O
81.7
83.7
73.1
72.8
MobileCLIP 77
82.3
84.5
71.1
70,9
Open Vision 40
79.3
84.2
65.9
65.1
Effeet of Kernel Function. To validate the role of kernel function RP-KrossFuse framework, we conducted linear probe experiments among different kernel functions. As shown in Figure ())c), those kernel-bass fusions can led to almost 10% classification accuracy improvements on ImageNet dataset, with the Cosine and RBF kernels yielding slightly higher gains than the linear kernel. In the main text, we use cosine kernel for classification and RBF kernel for clustering.
Effect of Random Projected Dimension. We investigate how the choice of random projection dimension affects the performance of RP-KrrssFuse. As shown in Figure d5),), increasing the projection dimension leads to a steady improvement in classification accuracy, which gradually saturates as the dimension becomes larger. Notably, when the dimension reaches around 3000, the performance of RP-KrossFus closely approaches that of the KrossFuse, indicating that a sufficiently high projection dimension can effectively preserve the information required for optimal fusion.
Effect of Hyperparameter C. The constant C' in Equation (3) balances the influence of uni-modal embeddings on the fused shared-modality kernel similarity and maintains balance between similarity scores for shared and non-shared modalities. Specifically, when considering images and text as shared and non-shared modalities, a smaller increases the weight of the image uni-modal embedding. but reduces the balance between (image, image), (image, text), and (text, text) similarity scores. To empirically validate this effect, we evaluated the cosine similarity distributions of image-text pairs on the MSCOCO validation set under different values of C. As shown in Figure 26 smaller values of C lead to slightly larger separation between the similarity curves of CLIP and RP-KrossFuse, whereas larger C' results in more overlapping curves, indicating improved modal alignment.
C.7 Computational Efficiency of RP-KrossFuse
We provide additional analysis of the computational efficiency of our proposed RP- Kr-KrossFuse scheme. Theoretically, RP-KrossFuse reduces the computational complexity from (d)dzl) (naive random projection) to O(dd, + dz)() by leveraging structured random projections, which enables more scalable multimodal fusion. Regarding inference efficiency, it is important to note that, assuming sufficient memory, our method allows for the two embedding models to operate in parallel, thereby avoiding increased inference latency. To support this, we conducted benchmarking experiments on two NVIDIA RTX 3090 GPUs with a batch size of 128. The original KrossFuse pipeline required 491.3 ms per batch and 18.48 18.406 GB of peak memory, while Rr-KrossFuse (with / = 5000) achieved comparable performance with just 413.1 ms per batch and significantly reduced memory footprint of 3.39 GB. Notably, the fusion step's ' overhead was reduced from 22.0 ms to only 0.14 ms. For reference, the baseline CLIP and DINOv2 models individually required 73.6 ms and 392.1 ms per batch, respectively.
C.8 Concatenation-Out Only Baseline
We additionally analyze a concatenation-only baseline to better understand the advantages of the pro- posed FrssrossFuse fusion strategy. From a theoretical perspective, concatenating two embeddings corresponds to defining a kernel similarity function that the sum of the individual kernels. This formulation implies that two samples will have high similarity in the fused space only if both original
42

---

(
2500
(pos)
Kroorsssuee (pos)
2000
CL (neg)
RP-KrossFuse (neg)
Samples
1500
1000
500
0
0.2
0.4
0.6
0.8
Casine Similarity
(b)
2500
CLiP (pos)
AP-KrossFuse (pos)
2000
CLI (neg)
Krosssss (neg)
Samples
1500
1000
500
-0.2
0.0
0.2
0.4
0.6
0.8
Cosine Similarity
(c)
CL/P (pos)
RP-Krosskuse (pos)
2000
CLIP (neg)
FP-KrossFuse (neg)
Samples
1500
1000
500
0
0.2
0.0
0.2
0.4
0.6
0.8
Casine Similarity
Figure 26: The cosine similarity distributions of positive/negative image text pairs MSCOCO dataset across various values hyperparameter C. (a) C = 0.1. (b) C 0.5. (c) C 1.
43

---

Table 10: Benchmarking of inference and fusion efficiency.
Method
Time/Batch (ms)
Peak Memory (GB)
CLIP
73.6
2,077
DINOv2
392.1
3.199
KrossFuse Pipeline
491.3
18.406
KrossFuse (Overhead)
22.0
17.203
Parallel RP-K EP-KrossFuse Pipeline
413.1
3.387
PuP-KrssFuse (Overhead)
0.14
1.729
embeddings assign them high similarity, effectively modeling the intersection of the semantic con- cepts captured by each modality. In contrast, the Kronecker based fusion employed in RP-Kin OP-KrossFuse leads to a product of kernel similarities, which enables distinguishing two samples as long as either embedding distinguishes them, thereby modeling the union of their semantic representations.
In supervised classification settings, the concatenation baseline is expected to perform comparably to the stronger of the individual embeddings. This is because the concatenated feature vector preserves the full representational capacity of both embeddings, while the limited VC dimension of the linear probe mitígates overfitting. However, in unsupervised or weakly supervised scenarios such as clustering. the product-base Kronecker fusion can better capture complementary cross-modal structures, leading to improved representation quality and broader concept coverage.
Empirically, compared the concatenation baseline with OP-KrrssFuse across multiple tasks. For supervised ImageNet linear probing, both methods achieve similar performance (concatenation: 83.6%, RP-KrossFuse: 84.1% which aligns with theoretical expectations. In multimodal classi- fication tasks, SrP-KrosFuuee consistently outperforms the concatenation baseline on the MVSA dataset [52], achieving 74.3% vs. 71.3% on MVSA-Single and 66.6% 63.7% MVSA-Multipl.. The largest gains are observed in unsupervised clustering tasks, where RP-KrossFuse shows clear improvements across datasets, summarized in Table
Table 11: Comparison between the concatenation-only baseline and RP-KrossFuse on unsupervised clustering tasks, evaluated using NMI, AMI, and ARI metrics (%).
Method
Metric
Flowers102
DTD
ImageNet-Dog.
GTSRB
Typo-Attacked IN
Concatenation
NMI
98.4
60.5
76.3
44.4
RP-KrossFuse
NMI
99.1
62.9
88.3
50.0
87.4
Concatenation
AMI
97.9
57.9
76.1
40.2
43.9
RP-KrossFuse
AMI
98.8
60.4
88.2
46.7
87.3
Concatenation
ARI
93.9
35.3
28.4
FusKKrssFuuse
ARI
97.0
36.4
86.3
19.5
79.6
C.9 Application of RP-KrossFusee Text-to-Image Diffusion Models
To further demonstrate the practical utility of the proposed RP-KrossFuse embedding fusion, apply it to conditional image generation with text-to-image diffusion models. Specifically, we adapt the Vendi Score Guidance (VSG) framework proposed by Askari et al. [4. which guides the reverse diffusion process using the Vendi diversity score of the generated samples. In the original implementation. CLIP embeddings were used compute the Vendi diversity score. We replace this embedding with the Kronecker-fuue fused representation of DINOv2 and CLIP, thereby enriching the diversity guidance with complementary visual and multimodal semanties.
Experiments were conducted on the ImageNet dataset using a class-conditional Diffusson Transformer (DīT-XL/2) the backbone. Two guidance configurations were compared: (1) the original CLIP- based contextualized Vendi Score Guidance (c- (S-V and (2) our KrossFuse-based version, which employs the fused CLIP and DINOv2 embeddings. As reported in Table [2] incorporating RP- KrossFuse improves both diversity and fidelity of the generated samples. Specifically, the diversity 366 and Coverage 49 increase notably, while the quality metrics Precision [36 and Density 49 also improve over the CLIP-only baseline.
44

---

Table 12: Comparison of Vendi Score diversity guidance methods for text-to-image diffusion on
ImageNet using DiT-XL/2. Metrics follow 33 49]
Diversity Guidance Method
Precision
Recall
Density
Coverage
c-VSG Guidance (CLIP)
0.913
0.413 0.484
1.206
0.552
KrossFuse (CLIP DINOv2)
0.932
1.252
0.613
45

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
The abstract and introduction clearly summarize our proposed method and its contributions. These claims directly supported by experimental findings across multiple benchmarks.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Our paper includes a "Conclusion and Limitations" subsection in the main
text.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
46

---

Answer: [Yes]
Justification:
Our paper provide complete proofs in the section A. of appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Our paper ensures full reproducibility. The experiments details are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
47

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We submitted the code as the part of the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/publiccguu..eee CoodeSumeeieeeeee ssi iessonPolll molyicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details are given in Sec. B. of the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We report the average performance over multiple runs, but do not include error bars or statistical significance tests. The observed variance across runs was small, so we focused reporting mean results for clarity.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
48

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All experiments are run on two RTX-4090 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The paper conforms with the NeurIPS Code of Ethics and does not pose any potential harm.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We do not foresee any negative societal impacts beyond those generally associated with aarge-scale representation learning.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
49

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all models and datasets we used in the paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
50

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
51

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buctps://neure wwwparippps..coooeeeeeeeee for what should or should be described.
52


Reviews:
[{"review_id": "hoGuF2vXrs", "reviewer": "Reviewer_kcnR", "summary": "This paper introduces a method to integrate cross-modal embedding with modality-specific embeddings called RP-KrossFuse. The main idea is to combine the embeddings using Kronecker product, so that the sample-pairwise similarity scores of the fused embeddings is the product of cross-modal embedding similarity and single-modal embedding similarity. This property allows the fused embeddings to form fine-grained clusters that inherit the discriminative information of both parent embeddings. Considered the extremely high dimension of the Kronecker product, the authors further proposes the application of random projection to lower the dimension of the fused embedding while preserving the similarity relationships. Theorems are provided to show that random projected features can preserve such relationship with high probability. Extensive experiments are conducted to test the fused embedding, including image classification, text classification, zero-shot image-text alignment and cross-modal few-shot learning. The results indicate that the fused embedding combined the strength of cross-modal embedding and modality-specific embedding.", "questions": "What is the implementation of the kernel map $\\phi$?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Combining representations from different models is an interesting question with broad potential applications. The paper provide an effective solution to this problem. The training-free nature of the proposed method improves its utility in real-world scenarios.\n2. The theoretical argument that randomly projected features preserve sample-wise similarity relationships is a strong foundation for the proposed method.\n3. The experiments are extensive with acceptable results. Despite that the fused embedding performs occasionally underperforms either cross-modal embedding or modality-specific embedding on certain tasks, the overall results indicate that the fused embedding is generally better than any of the individual representations.\n4. The paper is written in a decent way, having good structure and being easy to follow. Although the \"Numerical Results\" section has a minor issue (see Weakness 1), the execution is otherwise solid.\n\nWeaknesses:\n1. In section 6 (Numerical Results), 'see appendix' appears too frequently. Introducing the experiment without showing the result in the main paper affects reading experience. The author should carefully decide which result to be the most important and to be presented in the main text.\n2. The authors should report the actual dimension $l$ they choose in implementation, as Figure 5 (d) clearly shows that higher dimension leads to better performance. On one hand, the dimension of the fused embedding determines the computational cost and the efficiency of the method. On the other hand, a larger dimension used in the fused embedding may raise concerns regarding whether certain comparison is fair. For instance, a larger linear layer with more parameters could be applied in the linear probing experiments, leading to better performance compared to training a small linear layer on the CLIP embedding.", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer kcnR for the constructive and thoughtful feedback on our work. We are delighted that the reviewer finds our work to provide “an effective solution” and “written in a decent way”. Below is our response and clarifications regarding the questions and comments in the review:\n\n**1- Referrals to the Appendix in the main text**\n\nWe thank the reviewer for pointing this out. In the revision, we will keep the more important numerical results on the zero and few shot classification and clustering performance in the main text and reduce the Appendix referrals to only one time at the end of the numerical results section.\n\n**2- The role of random projection dimension $l$ in the RP-KrossFuse method**\n\nWe appreciate the reviewer’s comment regarding the importance of the random projection dimension $l$ in the RP-KrossFuse method. We would like to clarify that we have used dimension $l=3000$ for all datasets except $l=5000$ in the case of ImageNet. In the revised version of the paper, we will clearly state the value of $l$ alongside the tables and figures.\n\nTo further explain the role and selection of dimension $l$, we note that a key challenge of the Kronecker product fusion is the resulting high dimensionality: the fused representation has size $2d_1d_2$ for embeddings of dimensions $d_1$ and $d_2$. For example, combining CLIP and DINOv2 embeddings leads to a fused dimension close to 800,000 ($2 \\times 512 \\times 768$).\n\nTo address this, we introduce random projection in RP-KrossFuse, which is *not to ensure fairness* in comparison with the original embeddings, *but to reduce the computational overhead* to meet the computational budget. If computationally feasible, we would even prefer to use the full Kronecker product without random projection. However, this would incur significant memory and compute costs. \n\nAlso, we highlight that our proposed RP-KrossFuse scheme reduces the computational complexity from $O(d_1d_2 l)$ (naive random projection) to $O((d_1 + d_2)l)$, enabling more efficient fusion. To empirically validate this efficiency gain, we conducted benchmarking experiments on an NVIDIA RTX 3090 GPU with a batch size of 128. The original KrossFuse pipeline required 491.3 ms per batch and 18.406 GB of peak memory, while RP-KrossFuse (with $l = 5000$) achieved comparable performance with just 413.1 ms per batch and a significantly reduced memory footprint of 3.39 GB. Notably, the fusion step’s overhead was reduced from 22.0 ms to only 0.14 ms. For reference, the baseline CLIP and DINOv2 models individually required 73.6 ms and 392.1 ms per batch, respectively.\n\n\n|                 Method                | Time/Batch (ms) | Peak Memory (GB) |\n|---------------------------------------|-----------------|------------------|\n|                  CLIP                 |      73.6      |      2.077       |\n|                 DINOv2                |      392.1     |      3.199       |\n| KrossFuse Pipeline |      491.3     |      18.406      |\n| KrossFuse (Overhead) |      22.0      |      17.203      |\n| Parallel RP-KrossFuse Pipeline |      413.1     |      3.387       |\n|     RP-KrossFuse (Overhead)     |       0.14      |      1.729       |\n\n\n**3-** *“a larger linear layer with more parameters could be applied in the linear probing experiments, leading to better performance compared to training a small linear layer on the CLIP embedding”*\n\nWe would like to clarify in the linear probe classification experiments, we did not use any non-linear activation functions applied to the random projection features. Therefore, in the comparison with the CLIP model, even if we first apply a linear transformation (denoted by $g_w$) to transform the CLIP features to a higher dimension and then apply the linear probe (denoted by $f_{\\theta}$) to the linearly-transformed features, the composition $g_w \\circ f_{\\theta}$ still remains to be a linear function of the CLIP features, and therefore the linear-probe classification accuracy will not improve over applying it directly to the CLIP embedding. \n\n\n\n**4- Question on the kernel map $\\phi$**\n\nWe thank the reviewer for their question regarding the kernel map $\\phi$. In our experiments, we evaluated two kernel functions:\n1) The normalized linear (cosine similarity) kernel, implemented as $\\phi(x) = \\frac{x}{\\Vert x \\Vert_2}$. This implementation is straightforward and directly normalizes the input embeddings.\n2) The Gaussian (RBF) kernel, for which we used random Fourier features as described in Appendix Section A.4.\nWe would like to emphasize that our implementation of the Gaussian kernel map $\\phi$ does not simply apply RP-KrossFuse to a low-dimensional vector of precomputed random Fourier features. Instead, we designed the implementation to unify the generation of random Fourier features and the random projection in a single step. This unified approach effectively simulates operating in a feature space as if the number of Fourier features were infinite, while still being computationally efficient.\nThis design allows RP-KrossFuse to retain the expressivity of the Gaussian kernel without incurring the cost typically associated with large numbers of explicit random features."}, {"review_id": "x5F1tT4wcl", "reviewer": "Reviewer_MP5Z", "summary": "This paper introduces RP-KrossFuse, a method to fuse cross-modal embeddings with uni-modal embeddings to transfer the capabilities of both sets of embeddings into one unified set. The key motivation is that cross-modal embeddings enable cross-modal tasks like zero-shot image classification and cross-modal retrieval, however their unimodal capabilities take a hit, whereas uni-modal embeddings excel within their domain and modality, however are not applicable to cross-modal tasks. RP-KrossFuse utilizes random projection-based Kronecker product to integrate cross-modal embeddings with their uni-modal counterparts in an efficient manner. There are several experiments conducted to showcase the benefits of the fused embeddings on both uni-modal and cross-modal tasks.", "questions": "- In C.2 there is a learned unitary transformation that maps the text embeddings onto the space of the image embeddings. What are the downstream implications of such a method? Both [Udandarao et al](https://www.mlmi.eng.cam.ac.uk/files/2021-2022_dissertations/understanding_and_fixing_the_modality_gap_in_vision-language_models_reduced.pdf) and [Schrodi et al](https://arxiv.org/pdf/2404.07983) showed that the downstream effects of aligning the embeddings might not be the most effective for improving downstream task performance? There should be a discussion added that addresses the key motivation for performing this unitary transformation if it indeed does not positively affect downstream performance.\n- Is there a baseline where the embeddings are directly just concatenated and then there is a learnable projection at the end that is tuned? My concern with the current baselines is that they are all quite involved, and there is no simple \"concatenation-only\" baseline that has been tested.", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The motivation and outlined problem that the paper tackles is important and well-justified. Enabling representations to perform well across both cross-modal and uni-modal tasks is extremely significant, especially in an era where the community is moving towards unified end-to-end models.\n- The proposed KrossFuse and RP-KrossFuse approaches are novel to the best of my knowledge, and introduce an efficient method to fuse embeddings.\n- The method is theoretically grounded.\n\nWeaknesses:\n- The embedding models used in the paper are somewhat outdated. While there is definitely good signal from using these models like CLIP/OpenCLIP, there are much more advanced and newer vision encoders that have been released, which improve downstream performance using better data and training recipes. Some examples include [AIMv2](https://arxiv.org/pdf/2411.14402), [OpenVision](https://arxiv.org/abs/2505.04601) (although admittedly OpenVision was only released a week prior to the NeurIPS deadline, so I don't expect the authors to use those models for experiments), [MobileCLIP](https://arxiv.org/abs/2311.17049) etc.\n- A key important weakness in my eyes regarding the clustering results and the downstream performance results is that they are not directly comparable due to the differences in data distribution that was used for training the CLIP models and the DinoV2 models. See [WebSSL](https://arxiv.org/pdf/2504.01017). Hence, directly making claims about the differences being about cross-modal and uni-model encoders is somewhat flawed in my opinion, since the differences stem not necessarily only from the training recipe but the training data distribution used. I would urge the authors to add a discussion and caveating their claims surrounding the differences.\n- The paper does not compare with any methods that directly try to fuse the benefits of cross-modal and uni-modal recipes, at training time. Some exemplar methods include [SLIP](https://arxiv.org/abs/2112.12750), [TIPS](https://arxiv.org/abs/2410.16512), [SILC](https://arxiv.org/abs/2310.13355), [SigLIP-2](https://arxiv.org/abs/2502.14786). The reason those methods might be more beneficial and preferred at inference time is due to the inference time cost: for any new samples, running the RP-KrossFuse approach incurs extra cost since it requires forwarding through two encoder models, as compared to just forwarding through one model in the case of these approaches.", "was_revised": true, "final_justification": "The rebuttal addressed all my concerns. Further, upon reading the other reviews, I do not find any significant flaws / drawbacks in the paper. Therefore, I am leaning towards accepting the paper.", "rebuttal": "We would like to thank Reviewer MP5Z for the constructive and thoughtful feedback on our work. We are glad to hear that the reviewer finds our work’s motivation “well-justified” and our proposed method to be “theoretically grounded” . Below is our response and clarifications regarding the questions and comments in the review:\n\n**1- The method’s application to recent embeddings models**\n\nWe appreciate the reviewer’s suggestion regarding the application of our fusion method to more recent embedding models. We would like to emphasize that RP-KrossFuse is a general method that can be applied to fuse any combination of unimodal and cross-modal embeddings. Our focus on CLIP and its variants is due to CLIP’s widespread adoption and integration in many AI and machine learning frameworks.\n\nHaving said that, we agree that evaluating the method on newer cross-modal embeddings is valuable. In fact, we have already included results on the more recent SigLIP model in Appendix C.6 as part of our ablation studies.\nAdditionally, during the rebuttal period, we conducted further experiments on the reviewer’s suggested models, OpenVision and MobileCLIP. Below, we report the Linear-Probe and Zero-Shot classification results on ImageNet. These results also support the effectiveness of RP-KrossFuse when applied to more recent embedding architectures. We will include these results in the revised paper.\n\n| Model | Linear Probe (%) | Zero-Shot (%) |\n|-------|-----------------|--------------|\n| MobileCLIP | 82.3 | 71.1 |\n| RP-KrossFuse (DINOv2 + MobileCLIP) | 84.5 | 70.9 |\n| OpenVision | 79.3 | 65.9 |\n| RP-KrossFuse (DINOv2 +OpenVision) | 84.2 | 65.1 |\n\n\n**2- Role of different training data distributions of the embedding models in their Kronecker product fusion**\n\nWe fully understand the reviewer’s point regarding the differing training data distributions of CLIP and DINOv2, which naturally result in different behaviors when clustering or distinguishing between samples. In fact, this observation results in a key motivation behind our proposal to fuse these embeddings using the Kronecker product.\nAs discussed in the introduction, the kernel similarity score of our fused representation is the product of the individual similarity scores from the two embeddings. This means that two samples will be distinguishable in the fused embedding space as long as either CLIP or DINOv2 assigns them sufficiently small similarity scores.\n\nGiven that CLIP and DINOv2 are trained on different datasets, it is expected that they will specialize in distinguishing different semantic or visual concepts. This diversity enables the Kronecker fusion to effectively combine their complementary strengths. For example, Figure 1 in the paper shows that CLIP is more effective at separating traffic sign categories, while DINOv2 performs better at differentiating between dog breeds. When we apply Kronecker fusion, the resulting representation could successfully distinguish both traffic signs and dog breeds, which neither model could fully achieve on its own.\n\nTherefore, we argue that the difference in training data distributions between the embedding models is not a limitation, but rather could be a motivation for fusion. The Kronecker product allows us to construct a fused embedding that inherits the union of the discriminative capabilities of the models. We will revise the introduction to make this intuition more clear, and we also kindly refer the reviewer to Figures 8 and 10 in the Appendix, which provide additional experiments supporting this point across more diverse sample types.\n\n**3- Comparison with the training-based fusion methods**\n\nWe appreciate the reviewer’s comment regarding the comparison with training-based fusion approaches. To clarify, the baselines GATE and ATTN included in our study are indeed training-based fusion methods. However, we understand that the reviewer’s comment refers to a training-time integration, retraining one embedding model from scratch while conditioning on the other, potentially enabling lower inference-time complexity. We would like to raise two key points in response:\n\nFirst, regarding inference efficiency, it is important to note that, assuming sufficient memory, our method allows for the two embedding models to operate in parallel, thereby avoiding increased inference latency. To support this, we conducted benchmarking experiments on an NVIDIA RTX 3090 GPU with a batch size of 128. The individual CLIP and DINOv2 models required 73.6 ms and 392.1 ms per batch, respectively. Our parallel RP-KrossFuse pipeline achieved a total inference time of 413.1 ms, which is only slightly higher than the slower of the two models. Notably, the fusion step itself adds just 0.14 ms of overhead.\n\n|                 Method                | Time/Batch (ms) | Peak Memory (GB) |\n|-----|---------------|-------|\n|                  CLIP                 |      73.6      |      2.077       |\n|                 DINOv2                |      392.1     |      3.199       |\n| Parallel RP-KrossFuse Pipeline |      413.1     |      3.387       |\n|     RP-KrossFuse ( Overhead)     |       0.14      |      1.729       |\n\nSecond, a core advantage of our training-free fusion approach is its potential for improved out-of-distribution (OOD) generalization. Since RP-KrossFuse does not rely on training over a specific dataset, it avoids overfitting to a particular data distribution. In contrast, training-based methods often show sensitivity to their training data. For instance, Table 5 of [1, SLIP] demonstrates a 7.2\\% performance gap on ImageNet linear probing: 65.4\\% when trained on CC3M, 73.7\\% on CC12M, and a drop to 66.5\\% when trained on YFCC15M. This highlights the dependence of training-based fusion models on the choice of pretraining data.\n\nWe will clarify these points and include the new results in the text.\n\n**4- Unitary transformation in Appendix C.2 experiments**\n\nWe would like to clarify that the unitary transformation mentioned in Appendix C.2 was applied solely for the cross-modal clustering experiments reported in that subsection. None of the experimental results presented in the main text or other parts of the appendix involve this transformation.\n\nThe motivation for applying a unitary transformation in Appendix C.2 is due to the known “modality gap” between the image and text embedding spaces of CLIP, as discussed in previous work [2]. To enable meaningful cross-modal clustering via spectral clustering, it was necessary to learn and correct this modality misalignment. The unitary transformation serves this purpose by aligning the modalities in a shared space before clustering.\n\nWe emphasize that this step is limited to the cross-modal clustering setting and was not used in any of the other experiments in the paper. We will make this more clear in the revised paper.\n\n**5- \"concatenation-only\" baseline**\n\nWe thank the reviewer for raising the point about the concatenation-only baseline. Theoretically, concatenating two embeddings results in a kernel similarity function that corresponds to the *sum* of the individual kernels. This implies that two samples will have high similarity in the fused space only if both original embeddings deem them similar, effectively modeling the *intersection* of the embeddings’ concepts. In contrast, our proposed Kronecker fusion leads to a *product* of the kernel similarities, which enables distinguishing two samples as long as either embedding distinguishes them, modeling the union of their concepts.\n\nIn terms of numerical performance, we note that the concatenation baseline is expected to perform comparably to the stronger of the individual embeddings in supervised classification settings. This is because the concatenated vector retains all entries from both embeddings, and the limited VC dimension of the linear probe helps prevent overfitting.\n\nHowever, in unsupervised or weakly supervised scenarios such as clustering, the product-based Kronecker fusion can better capture the complementary structure from both embeddings. This often results in improved performance, as it reflects broader concept coverage. To empirically validate this, we conducted several experiments comparing the concatenation baseline with RP-KrossFuse across various tasks:\n\n1. For supervised learning tasks like ImageNet linear probing, both methods perform comparably (concatenation: 83.6%, RP-KrossFuse: 84.1%), which is theoretically expected. \n2. In multimodal classification tasks, RP-KrossFuse shows clear advantages over the concatenation on  MVSA dataset [3] (Sentiment Analysis on Multi-view Social Data):\n\n   - MVSA-Single: 74.3% vs 71.3%\n   - MVSA-Multiple: 66.6% vs 63.7%\n\n3. Most notably, in unsupervised clustering tasks, RP-KrossFuse significantly outperforms concatenation across the datasets. We have measured the clustering score, Adjusted Rand Index, Adjusted and Normalized Mutual Information (ARI, AMI and NMI), and the results are in the following table.\n\n| Method        | Metric | Flowers102 | DTD   | ImageNet-Dogs | GTSRB | Typo-Attacked ImageNet |\n|--|---|---|--|----|--|------|\n| Concatenation | NMI    | 98.4       | 60.5  | 76.3          | 43.8   | 44.4                 |\n| RP-KrossFuse  | NMI    | 99.1       | 62.9  | 88.3          | 50.0   | 87.4                 |\n| Concatenation | AMI    | 97.9       | 57.9  | 76.1          | 40.2   | 43.9                 |\n| RP-KrossFuse  | AMI    | 98.8       | 60.4  | 88.2          | 46.7   | 87.3                 |\n| Concatenation | ARI    | 93.9       | 35.3  | 64.9          | 14.1   | 28.4                 |\n| RP-KrossFuse  | ARI    | 97.0       | 36.4  | 86.3          | 19.5   | 79.6                 |\n\nReferences:\n\n[1] Mu et al. Slip: Self-supervision meets language-image pre-training. ECCV 2022.\n\n[2] Liang et al. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. NeurIPS 2022.\n\n[3] Niu et al. Sentiment analysis on multi-view social data. MMM 2016."}, {"review_id": "2vF1pMgRH1", "reviewer": "Reviewer_9JZK", "summary": "This paper proposes a method to effectively combine two modality spaces using the Kronecker product. Specifically, it aims to merge cross-modal embeddings (CLIP) with unimodal embeddings (DINO) to leverage the advantages of both methods. To do this, the paper introduces Kronecker fusion to incorporate the unimodal embedding's missing modality. It also proposes using Random Projection to reduce the  computational cost of the Kronecker product space. The proposed method demonstrated highly effective performance improvements, outperforming both CLIP and DINO on various downstream tasks.", "questions": "Are there other domains where these integrated embeddings could be highly effective? For example, could Vision-Language Models (VLMs) or text-to-image diffusion models achieve better performance by using these integrated embeddings?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n- The paper is well-written and easy to understand. \n- The proposed methods are sound, and practical scalability is well-considered. \n- Experimental results show effectiveness of proposed method on various downstream tasks\n\nWeakness\n- Further experiments are needed to determine if the method performs well with more recent cross-modal embeddings, such as SigLiP.\n- Evaluation on more downstream tasks is required. Please refer to the questions.", "was_revised": true, "final_justification": "I think this is a good paper that shows consistent performance gains across many uni- and multi-modal scenarios. While there are some worries about the efficiency of this method, I think it is not a strong drawback.", "rebuttal": "We would like to thank Reviewer 9JZK for the constructive and thoughtful feedback on our work. We are pleased to hear that the reviewer finds our work “well-written” and the proposed method to be  “sound” . Below is our response and clarifications regarding the questions and comments in the review:\n\n**1- Experiments on the recent SigLIP embedding**\n\nIn the Appendix C.6, we have already presented the numerical results of the zero-shot and Linear-probe classification accuracy of SigLIP embeddings on ImageNet. We will move these results to the main text to address the reviewer’s comment. In the rebuttal period, we also ran more experiments on recent embedding models MobileCLIP[1] and OpenVision[2], and the results support the method:\n\n| Model | Linear Probe (%) | Zero-Shot (%) |\n|-------|-----------------|--------------|\n| MobileCLIP | 82.3 | 71.1 |\n| RP-KrossFuse (w/ MobileCLIP) | 84.5 | 70.9 |\n| OpenVision | 79.3 | 65.9 |\n| RP-KrossFuse (w/ OpenVision) | 84.2 | 65.1 |\n\nIn the revised paper, we will add these results to the main text.\n\n\n**2- Application of the proposed method on text-to-image diffusion models**\n\nWe thank Reviewer 9JZK for the suggestion on showing the practical application of our proposed embedding fusion to diffusion models. To demonstrate the value of the proposed fusion in conditional image generation, we used our approach to adapt and improve the Vendi score diversity guidance proposed in [3]. The Vendi score guidance method guides the reverse diffusion process using the Vendi diversity score of the generated data. In the original implementation in [3], the authors used the CLIP embedding for computing the Vendi score in diversity guidance, which we updated to the Kronecker product of DINOv2 and CLIP.\nIn the experiment, we generated ImageNet samples using a class-conditional Diffusion Transformer (DiT-XL/2) with both the CLIP-based Vendi score guidance from [3], using CLIP as the feature extractor, and the KrossFuse embedding (CLIP fused with DINOv2). We found that the diversity of the generated samples, measured with Recall [4] and Coverage [5] scores, increased with the KrossFuse embedding, while the quality scores Precision [4] and Density [5] also improved over the original CLIP-based Vendi score guidance. We will include this experiment in the received manuscript.\n\n\n| Diversity Guidance method       | Precision | Recall | Density | Coverage |\n|---------------------------------|-----------|--------|---------|----------|\n| c-VSG Guidance (CLIP)        | 0.913     | 0.413  | 1.206   | 0.552    |\n| KrossFuse (CLIP and DINOv2) | 0.932  | 0.484  | 1.252   | 0.613    |\n\n\n\n\nReferences:\n\n[1] Vasu et al. Mobileclip: Fast image-text models through multi-modal reinforced training. CVPR 2024.\n\n[2] Li et al. Openvision: A fully-open, cost-effective family of advanced vision encoders for multimodal learning. arXiv preprint (2025).\n\n[3] Askari et al. Improving geo-diversity of generated images with contextualized vendi score guidance. ECCV 2024.\n\n[4] Kynkäänniemi et al. Improved precision and recall metric for assessing generative models. NuerIPS 2019.\n\n[5] Naeem et al. Reliable fidelity and diversity metrics for generative models. ICML 2020."}, {"review_id": "ahoLdwG1Oz", "reviewer": "Reviewer_tf3Z", "summary": "The paper proposes a novel approach for fusing uni-modal and multi-modal representations by integrating the kernel properties of the Kronecker product into the fused representation. To mitigate the multiplicatively growing dimensionality, the paper approximates the Kronecker product of the embeddings with the Hadamard product on randomly up-projected representations. By utilizing a random matrix with identity covariance for random projection, the paper derives a probabilistic upper bound for the proposed approximation. Technical claims are supported by empirical evaluations and ablations on classification tasks, showing that the proposed embeddings preserving the classification abilities of the uni-modal embeddings while retaining the zero-shot abilities of the multi-modal embeddings.", "questions": "1. How would the fused representation compare to the unimodal representation under the same kernel transformation? For example, how would the fused representation compare to the L2-normalized DINOv2 representation in the linear probing experiment?\n2. Are any transformations applied to shift the range of the kernels to positive values? If not, would doing so benefit the linear or cosine kernels?\n3. Are the comparisons done on an equal dimensionality footing of the fused embedding space? If not, what are the results if the fused dimensionality is equal to the uni- or multi-modal embeddings? \n4. The experiments mainly focus on self-supervised unimodal representations. Would one expect to observe a similar gain in accuracy when fusion is performed with a supervised unimodal representation?\n5. Can you comment on how the modality gap is affected by the proposed fusion?", "limitations": "As mentioned in the \"Major weaknesses,\" there is no dedicated \"Limitations\" section. As limitations, one could mention (i) introducing additional hyperparameters that need tuning, such as $l$ and $C$, (ii) experiments being limited to image classification tasks, and (iii) inconsistency in gain of accuracy compared to the stronger fused component.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* The method addresses a problem of potentially high significance. Fusing representations of varying modalities, especially those of foundational models, could impact a broad range of applications.\n* The lead-up to the RP-KrossFuse method is logically coherent. The design choices are well-motivated, and the extension via random projections is skillfully integrated.\n* The empirical results demonstrate improvement over individual components and simple, alternative fusion strategies, including ensemble and attention-based methods.\n\nMajor weaknesses:\n* The clarity of the text can be improved. Tables 1 and 2 should include citations for the previous methods. For Tables 1 and 2 and Figures 4 and 5, the captions lack sufficient detail about the experimental setup. For example, the figures omit information about which dataset is used. Additionally, lines 265–266 lack citations for the baseline methods and provide only a brief explanation, making it difficult to understand them without consulting the appendix. Similarly, line 225 mentions a range for the random matrix but does not clarify why it was chosen and can only be understood by referring to the appendix. The notation $\\psi_{1,2,X}$ etc. in Theorem 1 is apparently not defined, etc. The main text should be self-contained and clear without requiring the reader to refer to the appendix for key details. Other clarity issues: The notion of \"properly aligned\" in line 153 remains vague. Why is the embedding in Eqs. 4 and 5 suddenly E when Greek letters were used before.\n* Lines 49–51 build the motivation for using the Kronecker product as a fusion method. The claim is that if the fused kernel is the product of the individual component kernels, then the fused representation will identify two samples as similar only if they are deemed similar by both component kernels, which would lead to better class separation. This assumption only holds if the range of the kernel functions is non-negative. For example, with cosine or linear kernels, if both individual component kernels output negative values near their lower bound, indicating dissimilarity in both components, the Kronecker product would yield a high similarity score, contradicting the original motivation. Furthermore, Figure 5c shows that the cosine kernel, which violates the assumption, outperforms the Gaussian kernel, for which the assumption holds.\n* I may have missed this, but it seems like the value of l, i.e. the dimensionality of the projected features is not specified for the experimental comparison. Hence it is unsure if we are really comparing apples and apples here, in the sense that the projected, fused embeddings may not have the same dimensionality as the uni-modal ones that are being compared to. Unless I overlooked something, e.g., the CLIP and DINOv2 variants considered have differing embedding dimensions so RP-KrossFuse cannot be fairly compared against both. A comparison on an equal dimensionality footing seems quite essential, however.\n* Given that the modality gap is a known issue, as the paper discusses in the appendix (Line 630f), I am surprised that the paper does not assess this issue in more detail, e.g. to clarify whether the modality gap is affected by the proposed fusion strategy.\n* According to the checklist, the appendix is said to include a dedicated \"Limitations\" section. However, no such section is present. Further, \"Section 7: Limitations and Conclusions\" does not address the limitations of the work.\n\nMinor weaknesses:\n* Line 518 defines an upper bound for the absolute value of the dot product. However, in the equation following Line 519, the bound is applied to the dot product itself, not its absolute value. Moreover, the denominator of the Hoeffding inequality's exponent should be a function of $2B$, and not $B$, as the dot product would range between $[-B, B]$. Additionally, it is unclear why the denominator of the exponent in the Hoeffding equation is not squared.\n* During fusion, the method introduces additional nonlinearities in the form of kernels. It is unclear how the fused representation compares to unimodal representations with the same nonlinearity applied.\n* The experiments are limited to classification tasks. It remains unclear how the fused representation would generalize to dense prediction tasks such as semantic segmentation.\n* While the fused representation yields robust accuracy across different datasets, the gain over the stronger fused component is not consistent.\n* The paper is a bit sloppily written in various places. There are a fair number of typos (\"Kroncker\", \"simialrty\", ...) and the \"S-RoBERTa\" approach is capitalized in numerous different ways. \n* The answer to question 7 in the checklist is not convincing. Averaging the results is *not* a form of establishing statistical significance. Similarly, the answers to questions 9 and 10 are not convincing.", "was_revised": true, "final_justification": "The answers to my concerns as well as those of the other reviewers have clarified a number of important points in a concise fashion. From my perspective, with these additions as well as the added experiments, no significant concerns remain, hence I am upgrading my recommendation to accept. I believe this paper would be a valuable addition to the literature.", "rebuttal": "We would like to thank Reviewer tf3Z for the constructive and thoughtful feedback on our work. We are glad that the reviewer finds our work to address “a problem of potentially high significance” and the design choices “well-motivated” . Below is our response and clarifications regarding the questions and comments in the review:\n\n**1- Writing improvements**\n\nWe thank Reviewer tf3Z for pointing out the needed citations and information in Tables 1,2 and the captions of Figures 4,5. We would like to clarify that the notation $\\psi_{1,2}$ denotes the KrossFuse embedding of embeddings $\\psi_{1}$ and $\\psi_2$ which we will state in Theorem 1. The term “properly aligned” refers to how alignment is supposed to be measured in a cross-modal embedding (e.g. via cosine similarity or Euclidean distance). We will make the term clearer in the revision.\n\n**2- Range of kernel values in the KrossFuse fusion**\n\nWe appreciate the reviewer’s comment on the potential negative values in the kernel inner products and whether they fit into our argument for the product of kernel functions. As already pointed out by the reviewer, this concern does not apply to the Gaussian (RBF) kernel, which always takes positive values. We note that in all our clustering experiments, we have used the Gaussian kernel, which ensures non-negative values.\n\nIn the case of the cosine similarity kernel, the kernel inner product may indeed take negative values. However, we note that for both CLIP and OpenCLIP, the cosine similarity is always non-negative on MS-COCO and ImageNet samples (as also shown in reference [1]). On the other hand, the cosine similarity for DINOv2 can take negative values on ImageNet samples.\n\nThe question raised by the reviewer can be viewed as follows: What is the interpretation of the product kernel fusion when kernel similarity scores take negative values? Our answer is that in such cases, the product kernel fusion distinguishes sample types based on the **orthogonality** of their embedded vectors. Please note that we do not claim that this is the universally correct way to define similarity for all embeddings, but rather that it is a natural consequence of fusing kernels via multiplication. As such, the effectiveness of the product kernel fusion depends on whether the orthogonality condition is proper for the fused embeddings, i.e. should clustering within the embedding space be performed based on the orthogonality of the vectors? If an embedding has such a property, then applying the framework with cosine similarity can improve the numerical results. If this is not the case, the framework can instead be used with Gaussian, Laplace, or (normalized) even-degree polynomial kernels, all of which yield positive values bounded by 1, where our intuition on similarity score will directly apply.\n\n**3- The value of random projection dimensionality $l$ in the experiment**\n\nWe would like to clarify that we have used dimension $l=3000$ for all the datasets except $l=5000$ in the case of ImageNet. We will make this clear in the revision. \n\n**4- Fairness of RP-KrossFuse comparison with embedding of different dimensions**\n\nIf we have correctly understood the reviewer’s comment, it concerns the fairness of comparing embeddings with different dimensions. We respectfully disagree with the notion that such comparisons are inherently unfair. For example, DINOv2 and CLIP already have different embedding dimensions (768 for DINOv2 and 512 for CLIP), yet their comparison is widely accepted in the literature. Following the same logic, it would be inconsistent to deem a comparison between RP-KrossFuse and these individual embeddings as unfair solely due to dimensional differences.\n\nTo further clarify our motivation for using random projection in RP-KrossFuse, our goal is to reduce the high dimensionality of the original Kronecker fusion (nearly 800,000 for CLIP and DINOv2) only for the sake of computational efficiency in downstream applications. If enough compute and memory were available, we would even prefer to use the full KrossFuse representation over the projected one. As highlighted in our ablation studies, a lightweight random projection with computational cost $O((d_1 + d_2)l)$ and projection dimension $l = 5000$ is sufficient to preserve the performance of full KrossFuse in the ImageNet case.\n\n\nContinuing our response, if we are constrained to reduce the fused embedding to a bounded dimension, e.g., $l = 512$ to match the lower of the original embedding sizes, then using a pure random projection may not improve the signal-to-noise ratio (SNR). Under such dimensional constraints required by the reviewer's fairness criterion, we would instead apply PCA dimensionality reduction, which provides better SNR at the cost of increased computational cost. Of course, performing PCA directly on the full Kronecker fusion (size $\\approx$800,000) is computationally prohibitive. Therefore, in our rebuttal experiments, we applied PCA to reduce the RP-KrossFuse embedding (from 5000 to 512 dimensions) using ImageNet training data. The resulting linear-probe accuracy was 83.7% for the 512-dimensional PCA-reduced embedding, compared to 84.1% for the original RP-KrossFuse at 5000 dimensions. For reference, the linear-probe accuracies of the original CLIP and DINOv2 image encoders are 73.2% and 83.3%, respectively.\n\nTo summarize, we apply random projection in RP-KrossFuse solely to reduce computational cost for downstream tasks. We do not believe that exact dimension matching is a strict requirement for fair comparison between embeddings. However, if dimensionality parity is required, we recommend using PCA for compression, as it offers improved SNR, although at a higher computational cost. We will include this discussion in the revised manuscript.\n        \n**5- Modality Gap for the fused embedding vs. original CLIP embeddings**\n\nTo address the reviewer’s comment, we have measured the modality gaps (as defined in [1]) of the fused embeddings over the MS-COCO dataset. We made the measurement with our used RP-KrossFuse fused embedding with dimension $5000$. The results are in the following table. \n\n\n| Method            | Modality Gap |\n|-------------------|:------------:|\n| CLIP              | 0.875 |\n| RP-KrossFuse      | 0.853 |\n\n\n**6- Limitations**\n\nTo address the reviewer’s comment, we will discuss the limitations of the KrossFuse framework in more detail, including the potential cross-validation needed for parameter $C$ and $l$. However, we would like to highlight the following results in the Appendix: the text-image retrieval and clustering results,  the text classification results, that extend the evaluation beyond only the image classification task. Still, we will discuss the further evaluation of the KrossFuse framework in non-image domains in the limitation section.  \n\n**7- Typo in the bound of Theorem 1**\n\nWe thank the reviewer for catching the typo in the theorem’s equation regarding the power of $B$. We will correct it in the revision. \n\n**8- Question 1 (Results for $L_2$-normalized DINOv2)**\n\nTo answer the reviewer’s question, we tested the performance of the $L_2$-normalized DINOv2 embedding and the numerical results are in the following table (continuing Table 1 in the main text). As seen, the numerical performance on linear-probe classification results are mostly similar between normalized and unnormalized DINOv2. \n\n| Method | ImageNet | GTSRB | SVHN | IN-A | IN-R |\n|--------|:--------:|:-----:|:----:|:----:|:----:|\n| DINOv2 (unnormalized) | 83.3 | 72.5 | 60.5 | 48.5 | 68.8 |\n| Normalized DINOv2 | 81.8 | 73.1 | 59.9 | 47.4 | 68.1 |\n| RP-KrossFuse (DINOv2 + CLIP) | 84.1 | 82.7 | 66.9 | 47.6 | 67.4 |\n\n\n**9- Question 4 (Experiments beyond self-supervised embeddings)**\n\nTo address this question, we performed zero-shot and linear-probe classification experiments on ImageNet1K by applying RP-KrossFuse to the ImageNet-22k-supervised-trained image model  DeiT-III [2] .The numerical results are in the following table:  \n\n| Model | Linear Probe (%) | Zero-Shot (%) |\n|-------|-----------------|--------------|\n|  DeiT-III | 86.5 | NA |\n| CLIP | 73.2 | 57.9 |\n| RP-KrossFuse ( DeiT-III + CLIP) |  86.9| 57.5  |\n\n\n\nReferences:\n\n[1] Liang et al. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. NeurIPS 2022.\n\n[2] Touvron et al. Deit iii: Revenge of the vit. ECCV 2022."}, {"review_id": "7SyEai2psx", "reviewer": "Reviewer_m3ub", "summary": "This paper focuses on the performance trade-off between cross-modal embeddings and single-modality embeddings in the field of multi-modal learning. Cross-modal embeddings excel at aligning representations across modalities, but may underperform compared to specialized single-modality embeddings on modality-specific tasks. To address this, the authors propose the KrossFuse method, whose core idea is to leverage the product of kernel functions and fuse the feature maps of different modal embeddings through the Kronecker product. This method allows the fused embedding to achieve fine-grained distinction between samples if either of the original embeddings can distinguish them. The paper further introduces the scalable RP-KrossFuse method. RP-KrossFuse cleverly applies random projections to reduce the dimensionality of the fused representation while approximately preserving the kernel similarity values of KrossFuse. RP-KrossFuse maintains CLIP's zero-shot cross-modal alignment capability and also achieves superior performance over baselines like CLIP and DINOv2 in cross-modal few-shot learning tasks.", "questions": "1.\tWhile this study provides theoretical grounds concerning kernel function products and Kronecker products, a more thorough theoretical analysis is still needed regarding the deeper mechanisms of Kronecker product in semantic information fusion and the practical impact of approximation errors from random projections on fine-grained feature distinction capabilities, with the aim of transcending the current preliminary explanations based on kernel function products. Please provide a more intuitive and in-depth explanation from an information-theoretic or geometric perspective. Furthermore, the practical impact of approximation errors introduced by random projection on fine-grained feature distinction capabilities is not sufficiently discussed in the paper. Please analyze in detail under what conditions this approximation error might become significant and potentially lead to performance degradation.\n2.\tThe paper emphasizes RP-KrossFuse's \"training-free\" characteristic and computational efficiency. However, there is a lack of direct quantitative comparison with existing training-based fusion methods regarding inference time and resource consumption. Please provide specific resource consumption for RP-KrossFuse at typical dataset scales, and quantify this against single-modality embedding methods such as CLIP or DINOv2, to fully demonstrate its efficiency advantages.\n3.\tIn Figures 1 and 3, the paper utilizes kernel matrix heatmaps to demonstrate KrossFuse's clustering capability. Although the text description claims that the Kronecker product can better separate classes, the visual effect of the figures themselves lacks sufficient persuasiveness in terms of discriminability, making it difficult for readers to intuitively perceive the significant advantage of RP-KrossFuse. Please improve the visual effectiveness of the charts.\nIf the authors can provide a more in-depth and intuitive theoretical explanation, offer a more detailed analysis and discussion of the error effects of random projection, and improve the visual effectiveness of the figures, this will significantly enhance the paper's theoretical depth and comprehensibility, and I may increase the clarity rating by one point.\n4.\tIf the CLIP visual encoder, widely used in LLaVA and other vision-language models, were to be replaced with RP-KrossFuse's fused image embeddings, would this significantly enhance the model's visual understanding capabilities? Since RP-KrossFuse aims to bridge the performance gap between cross-modal embeddings like CLIP and single-modality expert embeddings like DINOv2, theoretically, its fused image representations should possess stronger fine-grained visual recognition capabilities while maintaining cross-modal alignment. Have the authors considered or preliminarily explored such a replacement? For instance, in downstream tasks for VLMs, such as visual question answering, image captioning, or instruction following, could using RP-KrossFuse image embeddings lead to performance improvements, especially in scenarios requiring detailed visual discrimination?\nIf the authors can provide preliminary experimental results or compelling theoretical analysis demonstrating that RP-KrossFuse fused image embeddings can indeed enhance the visual capabilities of VLMs (such as LLaVA), especially in tasks requiring stronger visual detail understanding, I will increase the significance and quality ratings.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strengths：\n1. The paper proposes a novel, training-free embedding fusion method called KrossFuse, which is based on the Kronecker product. This contrasts with most existing fusion strategies that rely on additional training, offering a new and more efficient paradigm for multimodal representation learning.\n2. The paper proposes that when the fused kernel function is the product of two original kernel functions, its feature map is the Kronecker product of these two original feature maps. To address the problem of missing non-shared modalities in single-modality embeddings, the paper solves this by constructing a symmetrized cross-modal embedding, enabling the application of Kronecker product fusion while maintaining unchanged similarity scores between shared and non-shared modalities.\n3. In RP-KrossFuse, the integration of random projection with the Kronecker product, achieved through the Hadamard product, aims to efficiently approximate high-dimensional kernel spaces; this represents a novel and practical technical combination.\n4. The paper conducted detailed ablation studies to analyze the impact of different expert embeddings, kernel function types, and random projection dimensions on performance. This helps in understanding the role of each component in RP-KrossFuse\n\nWeaknesses:\n1. In Figures 1 and 3, the paper utilizes kernel matrix heatmaps to demonstrate KrossFuse's clustering capability. Although the text description claims that the Kronecker product can better separate classes and capture the clustering structures of different embeddings, the visual effect of the figures themselves does not always clearly support this claim. It is difficult for readers to intuitively perceive the significant advantage of KrossFuse compared to single embeddings, and its clarity and distinctiveness are not highly convincing. This visual ambiguity may weaken the persuasiveness of the experimental results and affect the paper's clarity.\n2. While this study provides theoretical grounds concerning kernel function products and Kronecker products, a more thorough theoretical analysis is still needed regarding the deeper mechanisms of Kronecker product in semantic information fusion and the practical impact of approximation errors from random projections on fine-grained feature distinction capabilities, with the aim of transcending the current preliminary explanations based on kernel function products.\n3. The paper emphasizes RP-KrossFuse's \"training-free\" characteristic and computational efficiency. However, it lacks a direct quantitative comparison with existing training-based fusion methods regarding actual training/inference time and resource consumption. Although computational cost is mentioned, a specific comparison with CLIP or single-modality embeddings is not provided, which makes the intuitive understanding of its efficiency advantages insufficient.", "was_revised": true, "final_justification": "After careful consideration of their responses and alignment with fellow reviewers, I will maintain my original rating.", "rebuttal": "We would like to thank Reviewer m3ub for the constructive and thoughtful feedback on our work. We are glad that the reviewer finds our work to propose a “novel and practical technical combination”. Below is our response and clarifications regarding the questions and comments in the review:\n\n**1- Quantitative comparisons on clustering results for Figures 1 and 3**\n\nWe thank the reviewer for pointing this out. Originally, we decided to limit the evaluation of Figure 1 in the introduction to the qualitative comparison of kernel heatmaps. To address the reviewer’s concern, we have measured the clustering score, Adjusted Rand Index, Adjusted and Normalized Mutual Information (ARI, AMI and NMI), and the results are in the following table, indicating the better clustering of the Kronecker product fusion in comparison to each of the CLIP and DINOv2 embeddings.\n\n\n| Method | NMI | ARI | AMI |\n|--------|:---:|:---:|:---:|\n| CLIP | 75.8 | 68.6 | 75.8 |\n| DINOv2 | 71.6 | 59.4 | 71.6 |\n| RP-KrossFuse | **84.4** | **76.5** | **84.4** |\n\n\n**2- Theoretical analysis of Kronecker product in semantic information fusion**\n\nWe appreciate the reviewer’s point on the theoretical analysis of the role of Kronecker product in combining the semantic information of the two embeddings. As explained in the text (Propositions 1 and 2), the Kroncker product fusion leads to the product of the two kernel functions. A simple but somewhat insightful example to understand how the semantic features can be merged is when we have a 2-dimensional feature vector $[x,y]$ where each of the two embeddings can capture the information of only one of the coordinates, i.e. $\\psi_1([x,y]) = f(x)$  and  $\\psi_2([x,y]) = g(y)$ for mutivaraibel functions $f$ and $g$. Then, the Gaussian kernel fusion of the two embeddings will lead to the following kernel functions\n\n$$k_{fuse}([x,y],[x’,y’]) =  \\exp(\\frac{||f(x)-f(x’)||^2 + ||g(y)-g(y’)||^2}{\\sigma^2})$$\n\nAs seen, while neither of the embeddings contained the full information, the kernel function of the Kronecker product captures the role of both the coordinates. We will discuss this simple example in the revised text to better highlight how the fused embedding can capture the semantic information of both the embeddings.\n\n**3- Comparison with training-based fusion methods**\n\nWe appreciate the reviewer’s comment regarding the comparison with training-based fusion approaches. To clarify, the baselines GATE and ATTN included in our study are indeed training-based fusion methods. However, we understand that the reviewer’s comment could more generally refer to a training-time integration, retraining one embedding model from scratch while conditioning on the other, potentially enabling lower inference-time complexity. \n\nRegarding inference efficiency, it is important to note that, assuming sufficient memory, our method allows for the two embedding models to operate in parallel, thereby avoiding increased inference latency. To support this, we conducted benchmarking experiments on an NVIDIA RTX 3090 GPU with a batch size of 128. The individual CLIP and DINOv2 models required 73.6 ms and 392.1 ms per batch, respectively. Our parallel RP-KrossFuse pipeline achieved a total inference time of 413.1 ms, which is only slightly higher than the slower of the two models. Notably, the fusion step itself adds just 0.14 ms of overhead.\n|                 Method                | Time/Batch (ms) | Peak Memory (GB) |\n|---------------------------------------|-----------------|-----------------|\n|                  CLIP                 |      73.6      |      2.077       |\n|                 DINOv2                |      392.1     |      3.199       |\n| Parallel RP-KrossFuse Pipeline |      413.1     |      3.387       |\n|     RP-KrossFuse ( Overhead)     |       0.14      |      1.729       |\n\nAlso, a core advantage of our training-free fusion approach is its potential for improved out-of-distribution (OOD) generalization. Since RP-KrossFuse does not rely on training over a specific dataset, it avoids overfitting to a particular data distribution. In contrast, training-based methods often show sensitivity to their training data. For instance, Table 5 of [1, SLIP] demonstrates a 7.2\\% performance gap on ImageNet linear probing: 65.4\\% when trained on CC3M, 73.7\\% on CC12M, and a drop to 66.5\\% when trained on YFCC15M. This highlights the dependence of training-based fusion models on the choice of pretraining data.\nWe will clarify these points and include the new results in the revised manuscript.\n\n\n\n\n**4- Practical applications of the proposed method**\n\nWe thank Reviewer 9JZK for the suggestions on showing the practical application of our proposed embedding fusion. We agree that the application of KrossFuse fusion to VLMs could be highly interesting. In the rebuttal period, we did not manage to finish the visual instruction tuning of LLAVA-v1.5-7B. However, we performed another experiment on the application of the framework in guided diffusion models as explained in the following paragraph.\n\nTo demonstrate the value of the KrossFuse fusion in conditional image generation, we used it to adapt and improve the Vendi score diversity guidance proposed in [2]. The Vendi score guidance method guides the reverse diffusion process using the Vendi diversity score of the generated data. In the original implementation in [3], the authors used the CLIP embedding for computing the Vendi score in diversity guidance, which we updated to the Kronecker product of DINOv2 and CLIP.\n\nIn the experiment, we generated ImageNet samples using a class-conditional Diffusion Transformer (DiT-XL/2) with both the CLIP-based Vendi score guidance from [2], using CLIP as the feature extractor, and the KrossFuse embedding (CLIP fused with DINOv2). We found that the diversity of the generated samples, measured with Recall [3] and Coverage [4] scores, increased with the KrossFuse embedding, while the quality scores Precision [3] and Density [4] also improved over the original CLIP-based Vendi score guidance. We will include this experiment in the received manuscript.\n\n\n| Diversity Guidance method       | Precision | Recall | Density | Coverage |\n|---------------------------------|-----------|--------|---------|----------|\n| c-VSG Guidance (CLIP)        | 0.913     | 0.413  | 1.206   | 0.552    |\n| KrossFuse (CLIP and DINOv2) | 0.932  | 0.484  | 1.252   | 0.613    |\n\nReferences:\n\n[1] Mu et al. Slip: Self-supervision meets language-image pre-training. ECCV 2022.\n\n[2] Askari et al. Improving geo-diversity of generated images with contextualized vendi score guidance. ECCV 2024.\n\n[3] Kynkäänniemi et al. Improved precision and recall metric for assessing generative models. NuerIPS 2019.\n\n[4] Naeem et al. Reliable fidelity and diversity metrics for generative models. ICML 2020."}]

Author rebuttal:

Meta-review: This paper proposes a method for effectively combining two modality spaces using the Kronecker product. The authors introduce Kronecker fusion to address the missing modality in unimodal embeddings and propose leveraging Random Projection to reduce the computational overhead of the Kronecker product space. During the discussion, reviewers acknowledged that the design choices are well-motivated, the method addresses a problem of potentially high significance, and the experimental results demonstrate consistent performance gains across numerous uni- and multi-modal scenarios. The primary concerns raised by reviewers included the motivation behind Kronecker product fusion, the complexity of random projection for dimensionality reduction in RP-KrossFuse, and the choice of reduced dimension in RP-KrossFuse. The authors’ rebuttal effectively addressed most of these concerns, leading to unanimous acceptance by the end of the discussion. Consequently, the Area Chair (AC) recommends acceptance.
Decision: Accept (poster)