Towards an Explainable Comparison and Alignment of Feature Embeddings
Authors: Mohammad Jalali
Bahar Dibaei Nia
Farzan Farnia
Venue: ICML 2025 poster (2025)
Keywords: Feature embeddings
Interpretable Machine Learning
Kernel methods
Spectral Analysis
Embedding Alignment
Primary area: general_machine_learning->unsupervised_and_semisupervised_learning
TL;DR: We propose the SPEC method for interpretable comparison and alignment of embeddings, identifying how two embeddings cluster a reference dataset differently via spectral analysis.

Abstract: While several feature embedding models have been developed in the literature, comparisons of these embeddings have largely focused on their numerical performance in classification-related downstream applications. However, an interpretable comparison of different embeddings requires identifying and analyzing mismatches between sample groups clustered within the embedding spaces. In this work, we propose the Spectral Pairwise Embedding Comparison (SPEC) framework to compare embeddings and identify their differences in clustering a reference dataset. Our approach examines the kernel matrices derived from two embeddings and leverages the eigendecomposition of the difference kernel matrix to detect sample clusters that are captured differently by the two embeddings. We present a scalable implementation of this kernel-based approach, with computational complexity that grows linearly with the sample size. Furthermore, we introduce an optimization problem using this framework to align two embeddings,  ensuring that clusters identified in one embedding are also captured in the other model. We provide numerical results demonstrating the SPEC's application to compare and align embeddings on large-scale datasets such as ImageNet and MS-COCO. The project page is available at [https://mjalali.github.io/SPEC/](https://mjalali.github.io/SPEC/).

Towards an Explainable Comparison and Alignment of Feature Embeddings
Mohammad Jalali'
Bahar Dibaei Nia? 2 Farzan Farnia'
Abstract
Alayrac et al., 2022; Li et al., 2023; Oquab et al., 2024). The application of pre-trained embeddings has enabled scalable solutions for many downstream tasks, particularly in scenar- ios where the available sample size and compute resources are significantly limited. In such cases, the embedded data can be used train simple models, such as linear or k- nearest neighbors (KNN) classifiers, to achieve satisfactory results. Additionally, features extracted by standard em- bedding models are widely employed for the automated evaluation of generative models (Heusel et al., 2017: Hessel et al., 2021; Stein et al., 2023; Ospanov et al., 2025), provid- ing accurate rankings of generative modeling architectures without requiring time-intensive human assessments.
While several feature embedding models have been developed in the literature, comparisons of these embeddings have largely focused on their numerical performance in dlussification-relate downstream applications. However, an inter- pretable comparison of different embeddings re- quires identifying and analyzing mismatches be- tween sample groups clustered within the embed- ding spaces. In this work, we propose the Spectral Pairwise Embedding Comparison (SPEC) frame- work to compare embeddings and identify their differences in clustering reference dataset. Our approach examines the kernel matrices derived from two embeddings and leverages the eigende- composition of the kernel difference matrix to de- tect sample clusters that are captured differently by the two embeddings. We present scalable implementation of this kernel-based approach, with computational complexity that grows linearly with the sample size. Furthermore, we introduce an optimization problem using this framework to align two embeddings, ensuring that clusters identified in one embedding are also captured in the other model. We provide numerical results demonstrating the SPEC's application to compare and align embeddings on large-scale scale datasets such as ImageNet and MS-COCO. The project page is available at https://mjalali. https:/mmjialligiiubbb//SSPCCC githu..i//SPEC/.
While recent advancements in the machine learning commu- nity have introduced various embedding models that achieve remarkable results on standard image, text, and video do- mains, comparisons of these embeddings have primarily focused on evaluating their performance in standard down- stream tasks, such as classification accuracy on benchmark datasets (e.g. ImageNet). However, such comparisons often lack interpretability and do not reveal how differently the embeddings behave in recognizing various sample types. A more fine-grained comparison is necessary to disclose explainable differences between embedding models, partic- ularly in identifying which samples are clustered differently according to the models. Understanding these differences can aid in interpreting and debugging embeddings and can also be leveraged to align multiple embeddings. Further- more, interpreting the discrepancies of embeddings can be used to select representation models for downstream tasks.
In this work, we propose a spectral approach called Spec- tral Pairwise Embedding Comparison (SPEC) for the fine- grained comparison of two embeddings. The SPEC frame- work detects differences in sample clusters assigned by two embeddings, identifying major data groups that are clus- tered differently by one embedding compared to the other. To achieve this, we extend the well-established kernel princi- pal component analysis (Kernel-PCA) approach (Schölkopf et al., 1998), that leverages the eigendecomposition of the kernel similarity matrix, and propose analyzing the princi- pal eigenvectors of the difference of embeddings' kernel matrices to interpret the differences in cluster assignments. Our analysis suggests that the SPEC framework could effec- tively detect cluster differences between two embeddings.
1. Introduction
Several mainstream frameworks in computer vision and nat- ural language processing rely on embedding models to map raw image and text inputs into spaces with semantically meaningful features (Radford et al., 2021; Liu et al., 2023;
'The Chinese University of Hong Kong "Sharif Uni- versity of Technology. Correspondence to: Moham- mad Jalali <mjalali24@csee www.e@eec.c.......e.ee. Bahar Dibaei Nia ebahar.dibacinia. @ sharif.edu>>, Farzan Farnia <far- nia@ www.ccchk..u..kk.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Towards Explainable Comparison and Alignment of Feature Embeddings
An overview of SPEC method
Kernel matrix for Embedding X
Embedding X (CLIP)
UMAP Visualization of -UIPO2
Output: Eigenvectors to interpret and visualize differently captured clusters. Eigenvalues to measure the difference.
CLIP
Spectral approach for Pairwise Embedding Comparison (SPEC) using
Kernel matrix for
Embedding Y
Embedding Y
Kemee Difference matrix
(DINO22)
Reference Dataset
(FFHQ
Top 3 SPEC identified Clusters for CLIP DINOv2
Top 3 SPEC identified Clusters for DINOv2 CLIP
Cluster
Cluster #2
Cluster #3
Cluster
Cluster
Cluster
Figure 1. Overview of the Spectral Pairwise Embedding Comparison (SPEC) framework: The SPEC performs eigendecomposition of the difference of kernel matrices following the two compared embeddings (e.g.. DINOv2 and CLIP image embeddings) on a given reference dataset. Every eigenvector be interpreted as differently captured sample cluster by the embeddings, and the corresponding cigenvalue quantifies the difference between the cluster frequencies in the embedding spaces.
particularly when the clusters can be separated using the principal eigendirections of the kernel matrices.
cuss scalable computations of this distance and its gradient with respect to the embedding parameters. Using the power method and the calculated left and right eigenvectors of the differential covariance matrix, we enable gradicnt- based op- timization of the distance measure for aligning embedding models. This gradient based approach leads to a method we call SPEC-align, aligning embedding models by min- imizing their differences in clustering reference dataset. SPEC-align is particularly useful for aligning cross- embeddings, such as CLIP (Radford et al., 2021), with a state-of ofttte--rttttttt single-modality embedding. The alignment can improve the performance of cross-modality embeddings in capturing concepts specific to individual modalities.
To address the computational challenges of performing cigendecomposition on large-scale datasets, we develop a scalable implementation of SPEC. A direct eigendecompo- sition of the kernel difference matrix requires O(r²)) computations for dataset with n samples, which is compu- tationally expensive for large datasets. Assuming bounded feature dimension d for the applied kernel function, we prove that the eigenspace of the kernel difference matrix can be computed using O + d3) operations, resulting scalable algorithm under a moderate dimension value. Furthermore, we extend this scalable computation method to shift-invariant kernel functions, e.g. the Gaussian (RBF) kerel,, by employing the framework of random Fourier fea- tures (RFF) (Rahimi & Recht, 2007), where the size of RFF proxy-feature map can be controlled for a more efficient application of SPEC.
Finally, present numerical experiments several stan- dard image and text embeddings using benchmark datasets. Our results demonstrate the scalability of the SPEC frame- work in revealing differences in sample clusters across em- beddings over large-scale datasets. In our experiments, we tested the SPEC algorithm's 's application with both cosine similarity and shift-invariant Gaussian kernels, where we leverage random Fourier features for the latter case. Addi- tionally, we discuss the application of SPEC-align to align the CLIP model with single-modality embeddings. The em- pirical results highlight the effectiveness of SPEC-align in reducing the differences between CLIP's image embeddings
We also explore the application of the SPEC framework to detine distance measure between two embeddings. We define the SPEC-diff distance as the spectral radius of the kernel difference matrix, which aims to quantify the weight of the most differently captured cluster in one embedding that not strongly clustered by the other model. We dis-
2

---

Towards an Explainable Comparison and Alignment of Feature Embeddings
and specialized image-domain embeddings. The following is summary of our work's main contributions:
Also, Vargas et al. (2024) discuss a spectral approach the comparison of embeddings, where they consider the PCA- principal components of the collective embedded vectors of two representations. On the other hand, our work considers the kernel difference matrix, which better suits the compar- ison of embeddings in different dimensions and leads to a distance metric for their alignment.
Proposing the SPEC framework for explainable compari- son of two embeddings,
Providing scalable SPEC implementation with linearly growing computational cost to the sample size,
A different yet related line of work is the evaluation of gen- erative models. (Binkowski et al., 2018; Jalali et al., 2023; Ospanov et al., 2024; Jalali et al., 2024; Ospanov & Farnia, 2025; Wang et al., 2023b; Hu et al., 2025a; Rezaci et al., 2025; Hu et al., 2025b;c) leverage the eigenspectrum of ker- nel matrices to quantify diversity. The papers (Jiralerspong et al., 2023; Zhang et al., 2024) explore novelty evaluation, analyzing how generated samples differ from those of a reference model. In particular, (Zhang et al., 2024; 2025) propose a spectral method for measuring the entropy of the novel modes of generative model with respect to a reference model.
Developing the gradient- based SPEC-align method to align two embeddings and matching their sample clusters,
Demonstrating the successful application of SPEC in com- paring and aligning embeddings on benchmark datasets.
2. Related Work
Spectral Clustering, Kernel PCA, and Random Fourier Features. Kernel PCA (Schölkopf et al., 1998) is a widely established method for dimensionality reduction that relies on the eigendecomposition of the kernel matrix. Several studies (Bengio et al., 2003b;a) have explored the relation- ship between kernel PCA and spectral clustering. Also, the analysis of random Fourier features (Rahimi & Recht, 2007) for performing scalable kernel PCA has been stud- ied by Chitta et al. (2012); Ghashami et al. (2016); Ullah al. (2018); Sriperumbudur & Sterge (2022); Gedon al. (2023). In our work, we introduce a spectral approach for comparing two embeddings, leveraging the random Fourier features framework to address computational challenges.
Embeddings Alignment. There are many works on embed- ding alignment for multimodal models (Bellagente et al., 2023; Lu et al., 2024; Han et al., 2024; Wang et al., 2023c; Girdhar et al., 2023; Grave al., 2019). Salman et al. (2024); Eslami & de Melo (2025) introduced a method, which demonstrates that adversarial perturbations can force text embeddings to align with any image in multimodal models, exposing security vulnerabilities in vision- vision-language learning. Ye et al. (2024) proposed ModalChorus, an in- teractive system that visualizes and corrects misalignments in multi-modal embeddings, improving interpretability and optimization. Focusing fine-grained alignment, Yin et al. (2024) introduced a method for explicitly aligning individ- ual word embeddings with corresponding visual features, leveraging cross-modal attention to refine token-image asso- ciations. In contrast, our work focuses on aligning embed- dings in a kernel setting to match their sample clusters.
Evaluation and Comparison of Embeddings. Embedding evaluation is typically conducted using a limited set of down- stream tasks (Chen et al., 2013; Santos et al., 2020; Perone al., 2018; Choi et al., 2021). Existing NLP benchmarks (Gao et al., 2021; Reimers & Gurevych, 2019) focus limited tasks. Muennighoff et al. (2023) introduces MTEB, standardizing text embedder evaluation across diverse NLP tasks. In Image embeddings, Kynkäänniemi et al. (2023); Stein et al. (2023) compared different image embeddings and showed how they can influence different tasks, specif- ically the evaluation of generative models. Another line of research is probing methods (Belinkov, 2022; Pimentel et al., 2020; Adi et al., 2017; Rogers et al., 2021), which an- alyze model embeddings by training small models on them to understand what information is encoded. These methods help assess how well embeddings capture specific features, although they are not focused on embedding comparison.
3. Preliminaries
3.1. Embedding maps and spaces
Consider a data vector x E X' in the space X. An em- bedding map vb : X + S maps an input x to the em- bedding space S, which is supposed to provide a more meaningful representation of the input data vector. Through- out this work, we focus on the problem of characterizing and interpreting the differences of two embedding maps wi X + Si and 1/2 : X + S2, which can map the input x E X' to different embedding spaces Sy,S2.
Darrin et al. (2024) propose new metric for comparing embeddings without labeled data and propose the concept of information sufficiency (IS) to quantify the required infor- mation to simulate one embedding from another. Our work offers a complementary. explainable method for comparing embeddings by detecting different sample clusters assigned by embeddings and providing a method for aligning them.
3.2. Kernel Functions and Covariance Matrix
A kernel function k X' - IR maps two inputs x' to a similarity score k(zxx') = (((z),((z)) E [0, 1] that is the
3

---

Towards Explainable Comparison and Alignment of Feature Embeddings
inner product of the representation of x' characterized by X' RRd This definition implies that for every sequence of data points 21,...,,2 In E X, the following kernel matrix is positive semi-definite (PSD):
the kernel matrix of the second embedding with K (n.n)
[ka (1))())), (22 (1.))=(111*
Definition 1. We define the normalized kernel difference matrix Ag1,02 E R as follows:
21) : 21)
( [n)
Apr.v == 1 (KKps -KVz)
(Kws
K
00
k(zn Fnn
We propose the framework of Spectral Pairwise Embed- ding Comparison (SPEC) where the two embeddings vn and 1/2 are compared using the eigendirections of the kernel difference matrix Apr, As we will show, the principal eigenvectors be interpreted as the clusters of samples assigned by embedding s/1 that are less strongly grouped by the second embedding 2/2. In what follows, we first show theoretical result supporting the mentioned property of Apr.v 's eigenvectors. Next, we provide a scalable compu- tation method for computing the eigenspace of Agh.s that linearly scales with the sample size n.
Well-known examples of kernel functions include the cosine- similarity kernel krosine (ar, y) = o² y and the Gaussian (RBF) kernel defined for bandwidth or as
||
<G y) exp ( 11 v
20
Both these examples are normalized kernels where k(x, = holds for every x E X. Note that the kernel matrix in (1) can be written as = 40 where E Raxd contains (xx;) as its ith row for i E (1...., ,). Then, the kernel covariance matrix Cx E Raxd can be defined by reversing the matrix multiplication order as:
Theorem I proves that under the following two conditions on the sample index set I C (1,...,,)) n). the eigndirections of Ag can separate the clustered sample indices from the rest of samples. Note that the notation Ie denotes the complement index set of I, and KII, J] denotes the sub- matrix of K with rows in I and columns in J.
1 T Cx
n Σ (()))()))) 72
1
(2)
n2
Therefore, Cx = = and K = T T share the same
n
non-zero eigenvalues, since they represent the products of matrices with flipped multiplication orders.
Condition 1: Suppose the sample set Xy characterized by index set I are separated from the rest of samples by embedding 4h, where the normalized block kernel matrix 1 K [I, I"] has an 61 bounded Frobenius norm:
4. SPEC: A Spectral Identification of Embeddings' Mismatches
n
|| K [I. [I.
S 61
Consider set of data points F1.-.- In E X and two
embedding maps th X + S and thy : X + Sz. Also, suppose kn Si x S + IR and la S2 x S2 + R are kernel functions to be applied to the embedding spaces for 4/, 4/2, respectively.
Condition 2: Suppose the sample X characterized by index set I are weakly grouped by embedding 1/2, where the normalized block kernel matrix 1 K [I, Z] has an e2-bounded 62-operator norm (which is the maximum eigenvalue for this PSD matrix)
To compare the two embeddings, note that their correspond- ing spaces S₁ and S2 may have different dimensions, and therefore sample-specific comparison of the embedded vectors di (ar), (2((x) for each individual data point x will not provide a meaningful comparison of the embedding maps. Therefore, a more relevant approach is to com- pare the embeddings' outputs over the entire set of data (r1...., and investigate which structures are dissimilar between the sets of embedded data following the embed- dings. Here, we consider spectral approach and partic- ularly focus the difference of kernel matrices between the two embeddings. In the following, we discuss how the eigenspace of the kernel difference matrix can help identify the differently clustered points by the two embeddings. In the following, we denote the kernel matrix the first em-
|| III|| S
Theorem 1. Consider the kernel difference matrix Agr, in (3). Suppose Conditions 1 and 2 hold. Let Vi....y Vn. be the unit-norm eigenvectors of Ngr. corresponding to eigenvalues >ty..., ...... For every i E (I,....,n) we de- fine X7 and XF" to be the closest eigenvalue of Agg [I and Awn w [Ie,I% to di. Then, the following holds for { 4(G + c2):
Σ (A: - ( xF) 1 -Z) SE
Proof. We defer the proof of the theoretical statements to the Appendix A.
(n.a) bedding with K [Ka (1/1 (mx),, ), thi (2r))) (1.))(1.)) and
4

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Corollary 1. In the setting of Theorem suppose is an eigenvector of s Se for eigenvalue X whose gap with the maximum eigenvalue of the sub-matrix Agr.vz [Io, I9] satisfies x Amax (Av, s/ (I 2 >>>00 O. Then.
Note that the computation of the matrix I Ten can be per- formed with ( + d2)²) computations, linearly grow- ing in sample size, and the eigendecomposition of I' To can be handled via ( + d2)"), depending on the dimen- sions of the kernel feature maps Φ1, Φ2, and finally the eigen- vector mapping from F s... to AgI.vz will be (n1(d1 + d2))). Therefore, the entire eigenvector computation of A yi vi can be handled using (n(d, + d2)² + (d, + dz)²) computations. The algorithm 1 contains the main steps in computing SPFiC-eigenvectors using the above approach. As detailed in this algorithm, the computation of the differ- ential kernel covariance matrix can be run over samples in a cascade, avoiding the need to store a large dataset.
2 2
M
7
The above corollary proves that if an eigenvalue À of the kernel difference matrix A21,2a is sufficiently large, such that its gap with the maximum eigenvalue of the block A0g [ I9] (with the complement of samples in clus- tered by 1/1 yet not by 122) is higher than the threshold X, then the enteriis of the corresponding unit-norm eigen- vector will be bounded, reflecting the lack of Ie samples in the differentially clustered samples by embedding 1/1 and 2/2. Based the above theoretical results, we propose considering the principal eigenvectors of the kernel differ- ence matrix, and using their significant-valus entries to find the subset of samples clustered by embedding 1 but not grouped by 1/2.
Remark 1. Unlike the x n kerne difference ma- trix Asawwy E R the differential covariance matrix I Ter,22 E (d)d)+)))x((())) is not symmetric matrix. Still, T vi.0z will only have real eigenvalues, where the non-zero eigenvalues are shared with Ap, In Appendix A.3, we further provide Cholesky decomposition-basee method to reduce the computation of I T'n'''''s 's non-zero eigen- values and eigenvectors to the eigendecomposition of a (dl + d2) x (dl + dg diimensionl symmetric matrix.
Since the kernel difference matrix is of size n xn, a standard eigendecomposition will cost (n²) computations. Propo- sition shows that the computation cost will be lower for embeddings with bounded feature maps. In fact, this result shows the computation of the eigenspace can be performed using linearly growing computation cost O(n).
Applying the standard linear and cosine-similarity kernels, the kernel feature dimension will match that of the em- bedding, which is usually bounded by 1000 for standard image and text embeddings. In the case of shift-invariant kernels, e.g. the Gaussian (RBF) kernel, whose feature dimension is infinite, we utilize the random Fourier fea- tures (RFFs) (Rahimi & Recht, 2007; Sutherland & Schnei- der, 2015) to reduce the dimension of the kernel feature dimension with a proxy kernel function characterized by the RFFs. Following the RFF framework, given ker- nel function k(æ, v) s(x - y) that is normalized i.e. k()) = 1, we draw a number m independent Fourier fea- tures W1, Wm ~ R from probability density function R which denotes the Fourier transform of k defined as
Proposition 1. Consider the kernel difference matrix Aoh. 2b in (3). This matrix shares the same non-zero eigen- values with the following matrix:
C
Cer.vz
E (((1-d))((++d))) (4)
Ter.v=
- Srib
-O
where Car E Rddxdd, Cup E RRdxxdz are the kernel covari-
ance matrices V1, 12, respectively, and Cer v2 E Red xd2 is the cross-covarian matrix, defined as:
Cyn =
ri (z)
1 (2 /
R(w)
n((x) exp(-i((u )d
n
Σ
2 (v) (2)
Then, the RFF method approximates the shift-invariant ker- nel k(z, 3) z h(2, 8) (o(z). k(v)) where
n
Cpr.vz
Φ (xi( 02 (v2())))
x(x)
cos(wT T 2), sin x) ., cos(w,T sin (iT(w.))))) m
1
We also note that for every (right) cigenvector v E Rd of matrix To Te in (4), which we call the differential co- variance matrix, we can find the corresponding eigenvector of kernel difference matrix Ap as follows:
Theorem 2. Consider normalized shifl-invariani kernel (1 = K1 xz (x - v) and (a',,, v/) = K2 22(x' -y'). Then, drawing m Fourier features w(") ~ R and w/ Roz we form the RFF-proxy kernel functions functions k Rz. Then, consid- ering eigenvalues Ai...., An and eigenvectors Vi,..., Fn of proxy Ner for every > 0. the following holds with
(x(1)())) (2(()()1))
u=
v
(x))) (2(2((n)))
5

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Algorithm 1 Spectral Pairwise Embedding Comparison (SPEC)
Since SPEC-diff is only function of Ag1. non-zero eigenvalues, Proposition SPEC-diff 122) p(T (b)))) is equal to the spectral ra- dius of the differential covariance matrix I ae therefore, it is a symmetric pseudo-distance whose computation cost scales linearly with sample size n.
shows that
1: Input: Sample set (xI,..., ).). embeddings th and 4/2. kernel feature maps Φ1 and Φ2
2: Initialize Co. 0d,xd, xd,, Cay
Xd2
Ch =0d 00dxxd x22
3: for i (l,....,
do
While the SPEC-diff measure can be used to quantify the mismatches of two embeddings, can be further optimized in the training fine-tune finettuning of an embedding map 1/1,0" parameters 0 in order to align the embedding's 's clusters with another reference embedding 122. The optimization problem to be solved for such an alignment of the embeddings will be the following, which we call the SPEC-align problem:
4: Update Cwn + Cor + 1 (x:))
5: Update Cwuy + Coty +
6: Update Cah. f Ca on (22))
7: end for
8: Construct I oh as in Equation (4)
9: Compute eigenvalues Aiddy+d +dg and eigenvectors
Vi:di+da of noo-symmettr matrix I Tee.,2
min ((11,0) + 3 SPEC- (V1,0, 1/22)
(6)
BEB
10: for E (1...., d1 + d2) do
11: Map eigenvector u; - [ (X)) (2)))
In the above, L( .a) denotes the original loss function of training embedding 11 e and 3 denotes the coefficient of the penalty function SPEC-diff (22)), penalizing the mis- match with reference embedding 122. To apply a gradient- based optimization algorithm to solve (6), one needs to effi- ciently compute the gradient of SPEC-diff (tbi (v/,,, 122) with respect to parameter O. The following proposition shows that the gradient computation can be run in O(n.) over batch size n.
12: end for
13: Output: Eigenvalues Ai.... Adi+ eigenvectors
U1. tdy
probability at least - 8:
Σ Aerne Di - Av.ll <
128 (3/()
Proposition 2. Consider the definitions in (3)) Then, assuming a unique top eigenvalue (in terms of absolute value) for T ai,0, d2 with the left and right eigenvectors Utsf; Wrights we will have:
Theorem 2 shows the eigenvectors of the RrF-y kernel difference Agr,y provide a proxy for the eigenspace of the target kernel difference function Ag Note that the differential covariance matrix of the PFoXy-RFF kernel, the dimension of I v0z will be 4m 4m, that is finite for every shift-invariant kernel. As result, can apply the eigenspace correspondence in Proposition 1 to the proxy kernel function and reduce the computational complexity to Omm2n + m²) for m REF features and n samples.
V VoSPEC-dif( = (1udon araa Mright)) (7)
Therefore, the above proposition suggests computing the top left and right eigenvector of the (di +d22 (di + d2) kernel difference matrix, which can be computed using the power method, and subsequently to take the gradient of the scalar function u 99....v. Wright| which is the absolute value of the mean of the function value for each individual sample x TI,...., In. This property is especially suitable for applying stochastic gradient methods.
5. SPEC-based Quantification of Embedding Differences
As discussed earlier, the eigenspace of the kernel differ- ence matrix Agr,yz provides information on the differ- ently clustered samples by the two embeddings. There- fore, the SPEC approach motivates measuring the dif- ference of two embeddings using the eigenspectrum of Aggnnin Here, we specifically focus on the spectral ra- dius of Agi.02 ie.. its maximum absolute eigenvalue p( ((A)),y)) = maxigign (Ar(A)).))))) (p(A) denotes A's spectral radius). Note that Ag1.vz is by definition a symmet- ric matrix with a zero trace, and therefore its eigenvalues all real and add up to O. The following definition states the difference measure, which we call SPEC-diff score:
Remark 2. Another viable difference function follows the f-rm of the eigenvalues of Ag (note that SPEC-diff is the -nom of A 's cigenvalues). Since Ag d is symmetric matrix with real eigenvalues, the squared-E2- norm of its eigenvalues is its Frobenius norm-squared:
||A vb 12
(Kep (I:, - kaps (Z (2t, 2;))²
An advantage of the above embedding distance function is the possibility of performing stochastic optimization, where given mini-batch of B samples, we can consider the esti- Ba mation 1 Σij=1 (kaz (2,, -k))) (r, 22,)² in the align-
SPFC-diff (1/1, 122 = p(Apr.
ment process. This kernel difference momennus-norm-heee
6

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Cluster
Cluster
Cluster
Visualization top Clusters
eigenvalue= 0.025
eigenvalue 0.023
eigenvalue 0.019
DINOv2
CLIP
DINOv2- CLIP
eigenvalue 0.0334
eigenvalue 0.026
eigenvalue 0.013
CLIP
DINOv2
CLIP- - DINOv2
33
AMI:076 0,0036 DINOv2 SPEC
DINOv2 CLIP
CLIP SLUP-DIA DINON
pecuuit Maan
Figure 2. Comparison of different embeddings on 15K samples from the AFHQ dataset, consisting of 5K cats, 5K wildlife, and 5K dogs. The number at the top of cach image represents the eigenvalue of the corresponding SPEC cluster. The last two images in each row show the UMAP representation of the SPEC clusters for each embedding individually.
aligmment of embeddings has also been explored in the con- current work by Gong et al. (2025).
2021), and V2-V2 (Wang et al., 2023a). All embeddings were extracted using pre-trained models, and standard pre- processing was applied for uniformity across datasets.
6. Numerical Results
Experimental settings. In our experiments, we computed the SPEC differential kernel covariance matrix using 2000 independent random Fourier features for Gaussian kernel. To determine the Gaussian kernel bandwidth we followed the kernelbbaeed evaluation of generative models in (Jalali et al., 2023; Ospanov et al., 2024) and selected the embeddings bandwidths such that the difference between top eigenvalue is less than 0.01. We provide the detailed SPEC algorithm in Algorithm The experiments were performed two RTX-4090 GPUs.
In this section, first discuss the experimental settings and then apply the SPEC algorithm compare different image and text embeddings across various large-scale datasets. Finally, we explore the use of the SPEC-align method to match the sample clusters of the embeddings.
Datasets. In our experiments on image data, we used four datasets: AFHQ (Choi et al., 2020) (15K animal faces in categories of cats, wildlife, and dogs), FFHQ (Karras al., 2019) (70K human-face images), ImageNet- (Deng et al., 2009) (1.4 million images across 1,000 labels), and MS-COCO 2017 (Lin et al., 2015) 10110K samples of di- verse scenes with multiple objects). Additionally, similar to (Materzynska et al., 2022), we created custom dataset derived from 10 selected classes from ImageNet-Ik -1, where overlaid text labels directly images.
SPEC comparison of different embeddings. To evaluate SPEC, we compared various image embeddings using the AFHQ dataset. As shown in Figure 2, we employed SPEC for pairwise comparisons to analyze the difference between these embeddings. We reported the top images that corre- spond the maximum entries of the top three eigenvectors in the SPEC approach. Subsequently, alized the top 100 samples (with maximum entries) from each of the top 10 eigenvectors (i.e. SPEC-identified clus- ters). To confirm whether these samples were clustered by the first embedding and not by the second embedding. we used UMAP maps (McInnes al., 2018) to validate
found and visu-
Embeddings. The feature embeddings tested in this study include the image embeddings: CLIP (Radford et al., 2021), DINOv2 (Oquab et al., 2024), Inception-V33 (Szegedy et al., 2016), and SWAV (Caron et al., 2021), and the text embed- dings: RoBERTa (Liu al., 2020), CLIP (Radford al.,

---

Towards Explainable Comparison and Alignment of Feature Embeddings
SPEC Principal Clusters for ELIP-DINOY,
phoosiall pleyee
church
chawed
pays
pare
Cluster #1 eigenval=0.030
Cluster #2
Cluster #3
Cluster
eigenval=0.021
eigenval=0.006
eigenval=0.013
SPEC Principal Clusters for D
Mendahoal hemm
Cluster #1
Cluster #2 eigenval=0.024
Cluster #3
Cluster #4
cigenval-0.034
eigenval=0.016
eigenval=0.011
Figure 3. Top 4 SPEC-identified clusters comparing CLIP and DINOv2 embeddings on 10 ImageNet classes with overlaid text labels.
the SPEC-identified different-captured sample groups by the two embeddings. In the Appendix, we further provide the t-SNE (Van der Maaten & Hinton, 2008) and PaCMAP (Wang et al., 2021) plots of the FFHQ and AFHQ exper- iments. Also, we have analyzed the found clusters using violin plots to visualize normalized distances between data points within each cluster. The plots also suggest that the first embedding can cluster the points more strongly com- pared to the second embedding. Also, we ran the K-means clustering algorithm 50 times on each of the embedding's features and computed the averaged (across the 50 runs) Adjusted Mutual Information (AMI) (Vinh et al., 2009) be- tween the K-means labels and the SPEC-identifedd labels. The results indicate that the first embedding aligns more strongly with K-Means labels.
the cluster-specific samples. A similar experiment per- formed on SPEC clusters from the DINOv2 - CLIP on the FFHQ dataset (Figure 17). Details are in Appendix B.5.
To further evaluate SPEC's performance in comparing em- beddings, we apply a typographic attack on CLIP embed- dings. As studied by Materzynska et al. (2022), CLIP prior- itizes text added custom dataset over the image content. We selected 10 classes from the ImageNet- IK dataset and overlaid different text labels directly onto the images. The top four SPEC-identified clusters are presented in Figure 3, where we observe that CLIP clusters are based on the over- laid text, whereas DINOv2 clusters them based on visual features. We also compared CLIP and DINOv2 embeddings under the same settings on the Image dataset (Howard, 2019), which consists of various dog breeds from ImageNet- K. SPEC principal clusters show that DINOv2 primarily clusters images based on dog breeds, whereas CLIP groups them based on the animals' gestures. Additional details are provided in Figure 15 of Appendix B.4.
Furthermore, to highlight clustering differences between embeddings, we conducted sanity check on two of the top five SPEC clusters from the DINOV2 - CLIP on AFHQ. We computed the center of the top four images in each cluster in both DINOv2 and CLIP embeddings. Then, we calculated the cosine similarity between the center and set of eight test images: four additional images from the same cluster and four random images that do not belong to the cluster. As shown in Figure 16, DINOv2 well sepa- rates the cluster images from random samples, assigning the highest similarity scores to cluster-specific samples while keeping random samples significantly lower. However, in CLIP. some random images rank higher in similarity than
SPEC comparison of embeddings on different image and text datasets. To check the performance of SPEC on text embeddings, we generated 10K samples from GPT- 4o across different categories, including profession, object, gender, and emotion. We compared CLIP and RoBERTa text embeddings in Figure 4 and observed that the top four clusters in CLIP focused on objects in sentences, while RoBERTa clustered based on profession and gender. We
8

---

Towards Explainable Comparison and Alignment of Feature Embeddings
SPEC Principal Clusters for CLIP- ROBERTa
UMAP visualization of top SPEC clusters
Cluster
Cluster 83
CLIP
RoBERTa
aigroval.nutss
digenvoli 2.14
olgencal 2,00
A serious female artist is adjusinn a camera.
calm female tencher sctting an airplane.
[ scrious ferale doctor carrying frying pan.
determineed male engineer hokiing
locused lemale doctor adjusting airplane.
An angry male carpenter adjusting frying pan.
A happy female docter adjusting photograph calm female pioo designing photograph.
A curious male teacher adjusing airplane.
AA determined female chefis adjusting frying pon
calm female dector inspecting airplane.
A uized male artist adjusting frying
SPEC Principal Clusters for RoBERTa-CLL CLIP
RoBFRTa
CLIP
Cluster
Cluster
eigrnvalttins happy carponter caryying a desk.
calmn firefigttrr inspecting camera.
holding photograph.
calin female carpenter is holding syrings
An excited mule carpenter carrying laptop.
happy male arth inspecting frying pun. happy astist carrying desk.
An excited lnefighter bolding a desk.
focused carpenter holding camera. An excited mule carpenter holding camera.
tird fenale firefighter holding dosk.
cunous male artist is inspecting camera.
Figure 4. Top 4 SPEC- PrEC-identified clusters by comparing CLIP and RoBERTa text embeddings on a dataset generated from GPT-4o.
Kernel
SPEC-align
DINOv2 Kemel
Figure 5. Comparison of Kernel matrices after using SPEC-align match the sample clusters of CLIP to DINOv2.
also noted from the UMAP visualization of the samples that the top clusters of each embedding are not well clustered by the other, indicating that they focus on different aspects of the sentences. We also compared CLIP with E5 and observed the same results in Figure 13 of the Appendix B.3. In Figure 18, we compared different image embeddings on the ISSCOOO 2017 training set with 120K samples which we discuss in the Appendix B.4.
aalgg's performance, we conducted an experiment similar to (Oquab et al., 2024), where feature quality was assessed by training simple classifier on frozen backbone without fine-tuning its weights. In this setting, SPEC-align CLIP achieved 73.93% top-1 accuracy ImageNet- IK, outperforming the standard CLIP model, which reached 67.20%. For reference, DINOv2 achieved 78, 78.99% on the same task, indicating that SPEC-align brings CLIP substan- tially closer to DINOv2 performance.
Aligning embeddings using SPEC-align In this section, we discuss how use the SPEC-align method align the differential kernel covariance of two embeddings. As shown in the comparison of CLIP and DINOv2 in Figures 3 and 15, DINOv2 captures certain clusters that CLIP fails to distin- guish. To improve CLIP's performance, we aligned it with DINOv2 using the ImageNet training set. Specifically, we added an alignment term to the CLIP loss, as formulated in (6), and computed the gradient using (7). Learning param- eters are detailed in Appendix B. and additional results using Remark 2 provided in Appendix B.I.
7. Conclusion
In this paper, we proposed the spectral SPEC approach the comparison of embedding maps. The SPEC method aims to identify groups of samples clustered by one embed- ding model which is not grouped by another model. We formulated a scalable algorithm with ((n) computations to apply SPEC to a dataset of size n. We also discussed the application of SPEC for measuring the mismatches of two embeddings and their alignment. We note that the SPEC ap- proach operates based on the assumption that the differently clustered samples can be detected by the spectral method. Extending the clustering- based approach to non-spectral clustering frameworks will be interesting for future explo- ration. In addition, extending the framework to compare cross-modal embeddings such as CLIP and BLIP is future direction for this work.
We provide the kernel matrices for the four clusters in this experiment in Figure 5, corresponding to the results in Fig- ure 3. Notably, the SPEC-aligned CLIP kernel captures the top four clusters based on image content rather than the overlaid text labels. The clusters and their UMAP vi- sualizations are shown in Figure 26. To further evaluate

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Acknowledgments
Binkowski, M., Sutherland, D. J., Arbel, M., and Gret-
ton, A. Demystifying MMD GANs. arXiv preprint 2018.
The work of Farzan Farnia partially supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China, Project 14209920, and is partially supported by CUHK Direct Research Grants with CUHK Project No. 4055164 and 4937054. Finally, the authors sincerely thank the anonymous reviewers for their useful feedback and constructive suggestions.
Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments, 2021.
Chen, Y., Perozzi, B., AI-Rfou, R., and Skiena, S. The expressive power of word embeddings, 2013. URL attps::/arxiv.orgg wwt.://gggaabaa...oo...oo.a att.//aa...111331 3226.
Impact Statement
Chitta, R., Jin, R., and Jain, A. K. Efficient kernel cluster- ing using random fourier features. 2012 IEEE 12th International Conference on Data Mining, pp. 161-170. IEEE, 2012.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Choi, H., Kim, J., Joe, S., and Gwon, Y. Evaluation of BERT and ALBERT sentence embedding performance on downstream NLP tasks. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 5482- 5487, 2021. doi: 10. N0.110091CPR488822221...0.
References
Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., and Goldberg, Y. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. International Conference on Learning Representations, 2017.
Choi, Y., Uh, Y., Yoo, J., and Ha, J.-W. StarGAN v2: Di- verse image synthesis for multiple domains. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Alayrac, J.-B., Donahue, Luc, P., Miech, A., Barr, Has- son, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Si- monyan, K. Flamingo: visual language model for few-shot learning. ArXiv, abs/2204. 14198, 2022.
Darrin, M., Formont, P., Ayed, B., Cheung, C., and Piantanida, P. When is embedding model more promising than another? In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL https: /ttt::://poee http:///ppeeeiieeeee net forum? id=VqF2 7iiccl..
Belinkov, Y. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 277):207-219 04 2022. ISSN 0891-2017. doi: 10. N0l1166/coli..0000 00422.
Deng. J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei- Fei, L. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Bellagente, M., Brack, M., Teufel, H., Friedrich, F., Deis- croth, B., Eichenberg, C., Dai, A. M., Baldock, R.. Nanda, F., Schramowski, P., Kersting, K., and Weinbach, S. Multifusion: Fus- ing pre-trained models for multi-lingust multi-modal image generation. In Advances in Neural Information Processing Systems, volume 36, pp. 59502-59521, Cur- ran Associates, Inc., 2023.
Eslami, S. and de Melo, G. Mitigate the gap: Improv- ing cross-modal alignment CLIP. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /pp.revieew net wtt.:/ounttrrutt forum? id-aPTGvFqile.
S., Oostermeiijer K., Cruz-Salinas,
Gao, T., Yao, X., and Chen, D. SimCSE: Simple con- trastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894-6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v/2021.. 555. URL https: http://acaatthhoooo org 20g/2021.0 emn21mmmmmpp...mmmmmmmmm mmain.55/..
Bengio, Y., Vincent, P., Paiement, J.-F., Delalleau, O., Ouimet, M., and Roux, N. Spectral clustering and kernel PCA are learning eigenfunctions, volume 1239. Citeseer, 2003a.
Bengio, Y., Vincent, P., Paiement, J.-F, Delalleau, O., Ouimet, M., and LeRoux. N. Learning eigenfunctions of similarity: linking spectral clustering and kernel PCA. Technical report, Technical Report 1232, Departement d' I'Inormatique et Recherche Oprationnelle. ... 2003b.
Gedon, D., Ribeiro, A. H., Wahlström, N., and Schön, T. B. Invertible kemnel PCA with random fourier features. IEEE Signal Processing Letters, 30:563-567, 2023.
10

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Ghashami, M., Perry, D.J., and Phillips, J. Stramiin kemel principal component analysis. In Artificial intelligence and statistics, pp. 1365-1374. PMLR, 2016.
Hu, X., Pick, L.,, fung Leung, H., and Farnia, F. Prompt- wise: Online learning for cost-aware prompt assignment in generative models. 2025c.
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., and Misra, I. Imagebind: One embedding space to bind them all. In CVPR, 2023.
Jalali, M. Li, C. T., and Famia, F. An information-theoreii evaluation of generative models in learning multi-modal distributions. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: / //openrevew... net/ forum? wtw.mmmmPdddd22fP.....
Gong, S., Jiang, Y., Dou, Q- and Farnia, F. Kernl- based unsupervised embedding alignment for enhanced visual representation in vision- laniugee models. In Forty- second International Conference on Machine Learning, 2025. URL https:://// http///ppneeeeeeeeeeeeeeeeeeeeee met/forum Ad=0fLoGPTLo1.
Jalali, M., Ospanov, A., Gohari, A., and Farnia, F. Condi- tional Vendi Score: An informatiio-ttheoret approach to diversity evaluation of prompt-based generative mod- els. arXiv preprint 2...v:22411.22811. 2024. URL 02817.
Grave, E., Joulin, A., and Berthet, Q. Unsupervised align- ment of embeddings with wasserstein procrustes. In Chaudhuri, K. and Sugiyama, M. (eds.), Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 1880-1890. PMLR, 16-
Jiralerspong, M., Bose, J., Gemp. I., Qin, C., Bachrach, Y., and Gidel, G. Feature likelihood score: Evaluating the generalization of generative models using samples. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023. URL httph:://tppeeeeeeeeeeeee net for.mformm? -d--12k22o1111.
18 Apr 2019. URL https: httpt::/tpspeee http:////oceeeeeeeeeeeoo mllr. press/ /Ve5s/v883 vv9 ggaveelralleaaeeee html.
Karras, T., Laine, S., and Aila, T. A style-based generator architecture for generative adversarial networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4396-4405, 2019. doi: 10.1109) CVPR,2019.0045
Gross, D. Recovering low-rank matrices from few coeffi- cients in any basis. IEEE Transactions on Information Theory, 57(3):1548-155555 2011.
Han, J., Gong, K., Zhang, Y., Wang. J., Zhang, K., Lin, D., Qiao, Y., Gao, P., and Yue, X. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Kohler, J. M. and Lucchi, A. Sub-sampled cubic regulariza- tion for non-convex optimization. In International Con- ference Machine Learning. pp. 1895-1904 PMLR. 2017.
Kynkäääniemi, T., Karras, T., Aittala, M., Aila, T., and Lehtinen, J. The role of ImageNet classes in Fréchet inception distance. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: //openreview.co net forum? dl 4 w.8.
Hessel, J., Holtzman, A., Forbes, M., Bras, R. L., and Choi, Y. CLIPScore: reference-free evaluation metric for image captioning. In EMNLP, 2021.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium, Advances in neural information processing systems, 30, 2017.
Li, J.. Li, D., Savarese, S., and Hoi, S. BLIP-2: boot- strapping eegguugee-magee pre-training with frozen image encoders and large language models. In ICML, 2023.
Lin, T.-Y,, Maire, M., Belongie, S., Bourdev, I.., Girshick, R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., and Dollár, P. Microsoft COCO: Common objects in context, 2015.
Howard, J. ImageWoof: a subset of 10 classes from imagenet that aren't easy to classify, March 2019. URL. https: / / www.tttw..ooo.tt.eeeettttttteete fastai/ imagenettet wiwetewoa.........
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tun- ing. In Advances in Neural Information Processing Sys- tems, volume 36, pp. 34892-34916. Curran Associates, Inc., 2023.
Hu, X., fung Leung, H., and Farnia, F. A multi-armed bandit approach online selection and evaluation of generative models, 2025a. URL https: /ttp:s///axvvvvvo. abs/ 2406.01 07451.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. RoBERTa: A robustly optimized BERT pretraining ap- proach, 2020. URL https: httpss:/opperreieew opwprevieeeeeeeeeee wtt.eiiwwweeeeeeee.e........ee.e forum?1d=-sxSSO44tus
Hu, X., Leung, H.-f., and Farnia, F. Pak-ucb contextual bandit: An online learning approach to prompt-aware selection of generative models and 1lms. In Focto- International Conference on Machine Learning, 2025b.

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Lu, S., Li, Y., Chen, Q-G., Xu, Z., Luo, W., Zhang, K., and Ye, H.-J Ovis: Structural embedding alignment for multimodal large language model. arXiv:2405.20797, 2024.
Pimentel, T., Valvoda, J., Maudslay, R. H., Zmigrod, R., Williams, A., and Cotterell, R. Information- probing for linguistic structure. In Proceedings of the 58th Anmual Meeting of the Association for Computa- tional Linguistics, pp. 4609-4622, Online, July 2020. Association for Computational Linguistics.
Materzynska, J., Torralba, A., and Bau, D. Disentangling visual and written concepts in clip. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision, 2021.
McInnes, L., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:8802..0 03466,346, 2018.
Rahimi, A. and Recht, B. Random features for large-scale kemel machines. Advances in neural information pro- cessing systems, 20, 2007.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv: 1609.0070.07844 07843, 2016.
Reimers, N. and Gurevych, Sentence-BERT: Sentence em- beddings using Siamese BERT- In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP- IJCNLP), pp. 3982-3992. Association for Computa- tional Linguistics, November 2019. doi: 10.18653/v1/ D19- 1410. URL. https::/aclanthott D19 1410/.
Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB: Massive text embedding benchmark. In Vla- chos, A. and Augenstein, I. (eds.), Proceedings of the 17th Conference of the European Chapter of the Asso- ciation for Computational Linguistics, pp. 2014-2037, Dubrovnik, Croatia, May 2023. Association for Compu- tational Linguistics. doi: 10. 10.18653/11/2023.colmmmii 148. URL https:://// /a/aclanni /ataaaaaaaoooy.oooo///./. eacl Bacl-maing ii.iii... 1
org/
Rezaei, P., Farnia, F., and Li, C. T. Be more diverse than the most diverse: Optimal mixtures of generative models via mixture-uch bandit algorithms, 2025. URL https: //arxiv. org. abs /2422 12411.1.6222 17622.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fermandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnacve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. Dinov2: Learning robust visual features without supervision. 2024.
Rogers, A., Kovaleva, O., and Rumshisky, A. primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Lin- guistics, 8:842-866 01 2021. ISSN 2307-387X.
Ospanov, A. and Farnia, F. Do vendi scores converge with finite samples? truncated vendi score for finite-sample convergence guarantees. In The 41st Conference on Un- certainty in Artificial Intelligence, 2025.
Salman, S., Shams, M. M. B., and Liu, X. Unaligning every- thing: Or aligning any text to any image in multimodal models. arXiv preprint arXiv:2407.00 157,
2024. 202,4
Santos, J., Consoli, B., and Vieira, R. Word embed- ding evaluation in downstream tasks and semantic analo- gies. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4828-4834, Marseille, France, May 2020. European Language Resources As- sociation. ISBN 979-10-95546-344, URL https: / //aclanthology . htwp:oo....rg/22...1111. 1.594/.
Ospanov, A., Zhang, J., Jalali, M., Cao, X., Bogdanov, A., and Farnia, F. Towards a scalable reference- free evaluation of generative models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: h http//ppneeeeeeeeeoeeeee net./ forum? hetmmmdddxxxxxxxexmmmmm
Ospanov, A., Jalali, M., and Farnia, F. Scendi score: Prompt- aware diversity evaluation via schur complement of clip embeddings. In Proceedings of the IEFF/CVE Interna- tional Conference on Computer Vision, 2025.
Schölkopf, B., Smola, A., and Müller, K.-R. Nonlinear Kernel Eigenvalue Problem. Neural Computation, 129):1129-13199 July 1998. ISSN 0899-7667. doi: 10. (0011620009976669300007666 https: / deeeeeeploree.ooo Leee org/ soorg//ocumen... 6790375. Conference Name: Neural Computation.
Component Analysis
URL
Perone, C. S., Silveira, R., and Paula, T. S. Eval- uation of sentence embeddings in downstream and linguistic probing tasks. ArXiv, abs/1806.06259, 2018. URL http:/://ap..ssaiiisssseeoeo org/CorppuSDD:4306001
Sriperumbudur, B. K. and Sterge, N. Approximate kernel PCA: Computational versus statistical trade-off. The Annals of Statistics, 50(5):2713-2736 2022.
12

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Stein, G., Cresswell, J., Hosseinzadeh, R., Sui, Y., Ross, B., Villecroze, V., Liu, Z.., Caterini, A. L.., Taylor, E.,, and Loaiza-Ganem, G. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems, volume 36, pp. 3732-3784. Curran Associates, Inc., 2023.
Ye, Y., Xiao, S., Zeng, X., and Zeng, W. Modalchorus: Vi- sual probing and alignment of multi-modal embeddings via modal fusion map. IEEE Transactions Visualiza- tion and Computer Graphics, 2024.
Yin, Y., Zhao, Y., Zhang, Y., Lin, K., Wang. J., Tao, X., Wan, P., Zhang, D., Yin, B., and Zhang, W. Sea: Super- vised embedding alignment for token-level visual-textual integration in mllms. arXiv preprint arXiv:2408.11811 2024.
Sutherland, D. J. and Schneider, J. On the entor of ran- dom fourier features. In Proceedings of the Thirty-Fire Conference on Uncertainty in Artificial Intelligence, pp. 862-871, 2015.
Zhang, J., Li, C. T., and Farnia, F. An interpretable evalu- ation of antropy-based novelty of generative models. In Proceedings of the 41st International Conference on Ma- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 59148-59111 PMLR, 21-27 Jul 2024.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vi- sion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Ullah, E., Mianjy, P., Marinov, T. V., and Arora, R. Stream- ing kernel PCA with of (n) random features. Advances in Neural Information Processing Systems, 31, 2018.
Zhang, J., Jalali, M., Li, C. T., and Farnia, F. Unveiling differences in generative models: A scalable differential clustering approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025. URL https: / /arxivvv..////////a///aa//////v/ www./ooroogs////o.. 2405. 02700.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
Vargas, M., Cannon, R. T., Engel, A. W., Chiang, T. Y., and Sarwate, A. D. Understanding generative content with embedding models, 11 2024. URL https: //ww. osti, gov
Vinh, N. X., Epps, J., and Bailey, J. Information theoretic measures for clusterings comparison: is correction for chance necessary? In Proceedings of the 26th annual international conference on machine learning, pp. 1073- 1080, 2009.
Wang, L.., Yang, N., Huang, X., Yang, L.,, Majumder, R., and Wei. F. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00366 2023a.
Wang, Y., Huang, H., Rudin, C., and Shaposhnik, Y. Un- derstanding how dimension reduction tools work: An empirical approach to deciphering t-sne, umap, trimap, and pacmap for data visualization. Journal of Machine Learning Research, 22(201):1-73, 2021. URL http: //jmlr, org/ http:/ppppss//2/2/22 1061.. html.
Wang, 7., Farnia, F, Lin, Z., Shen, Y., and Yu, B. On the distributed evaluation of generative models. arXiv preprint ar arXiv:2310.11711 2023b.
Wang, Z., Zhao, Y., Cheng, X., Huang, H., Liu, J., Tang, L., Li, I.., Wang, Y., Yin, A., Zhang, Z.,, and Zhao, Z. Connecting multi-modal contrastive representations. In Proceedings of the 37th International Conference on Neu- ral Information Processing Systems, NIPS '23. Curran Associates Inc., 2023c.
13

---

Towards Explainable Comparison and Alignment of Feature Embeddings
A. Proofs
A.1. Proof of Theorem
For simplicity, we adopt the following notations in this proof:
K
1
K
11
[]
K
1
K v) [I,
K2
wh
[Io _T]
n1
N1
K11
1
K v/
[I,
K : n
K 1/2
IIII%
22
(2)
- K 1/2 II
1
"
A11 == Agr.va [I,
A12 = A No [I, [
A22
Apr.va
[T I
As result, the following holds by definition:
An i1 K
A12 K -k12, A22 K2 -22
According Condition 1, we know the Frobenius norm bound || || S1. Also, due to Condition 2, know the (2-operator norm bound || ll2 S e2. Note that the matrix 1.2 K2 is positive semi-definite (PSD). Therefore, the Schur complement of its block representation following indices in I and I [1, .l....,.. - must be a PSD matrix, i.c.
61.
K - K (2)
Therefore, the above Schur complement has a non-negative trace, implying that
Tr K (2)
(2)
K
k(2)
20 0
Tr
nT(K22
(2) K
(2)
K 12
<T((K())
S
<Tr
(122))
Therefore, we will have the following:
= Tr - Kizz)
2 Tr 22
Tr (2
k(2)
-1
k-(2)
Tr K2
(2)
1
1)K
(2)
Amax
(2)
1
Tr K-211 T
K22)
Amax (k
Tr
KK
(2) 12
K
|
la
||<(2111 ||
|
12
The above means that Condition 2 implies
k
<
k(22
52
12
As result, we can apply Young's 's inequality to show that
||12l1
2
(1
[
+
|
(1) 2
< 22 + (2)
F
Therefore, we will have the following:
A11
0
2
A Agr.v2
< 4(c + 62)
0
A22
14

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Now, since A 12,22 is symmetric matrix, we apply spectral decomposition write it as A01.v2 Vdiag ( where every V, of matrix V is eigenvector of A wo with corresponding eigenvalue d sorted A1 2 2 An Note that the cigenvalues are real and sum up to 0 as the trace of Abiwb is zero. Then, we can write:
Σ Aprnva As Vill2
(A Villé
i=1
Σ V. (Aprsve - -Aprne)) (Api - )
Tr v. (Apines - Appnvs) T (Api - Aprrwa)
Σ
Tr v.V. (Aprnes - (or) (Aprses Apprve))
i=1
v.v. (Aor.os - (osr T (Aer Awhr)a))
Tr
Tr (Aotnes - Astioss) (Apinvs - Aprnvs))
Agnnas - ApniNa 12
< (ee + 62)
As result, write
1(e + 62) 2
A - Amis vill2
> Aniv Avv, || Avv,
(A. x7) 1 + (As - x7 [
In the above, the last inequality holds know for every PSD matrix A and vector we have |AAv-vvllz 2 11y-\|1lvll. where Aj is the eigenvalue of A with the minimum absolute difference [λj Al. Therefore, the proof of Theorem 1 is complete.
A.2. Proof of Proposition 1
Note that we can write A Ap.. using the following matrix multiplication:
A Apr.ya
4
7
In the above, we define Pe. gaxd; to be the embedding of dataset æ1.... In with embedding map 1h, i.c., its ith row will be &i (4/ (I:)), and similarly we let T.,, E d to be the embedding of dataset with embedding map t/ with its ith row being d2 (x;)). Therefore, if we define A = [ Agr.e. AB.
Φvn]] and B = [ e1
then we will have T
On the other hand, we know that for every matrix E +d) and B E +ddd AB and B4 share the same
15

---

Towards Explainable Comparison and Alignment of Feature Embeddings
non-zero eigenvalues. In this case, the matrix BA sharing the non-zero eigenvalues with Agi, AB be calculated
TT
BA
V1
T
42
V/2
4 11
$
41
1/2
P 41
Ci
CT
Cata
P1,42
In addition, for every eigenvector v (corresponding to a non-zero eigenvalue) of v1,02 BA, we have that u Av an eigenvector of Ap1.ta AB which
(o((b(((1))
(2((>(011)
u1
P
pvn] v=
(1(((( (2((2((1)))
Therefore, the proof is complete.
A.3. Computation of Eigenvectors and Eigenvalues of I aa
Considering the notations in the previous proof, note that the defined differential covariance matrix I po can be written as follows:
Ci
Adxx
Od, xda
Cen
I Virva
CT
-Cer
Oagxd:
- -Layxda dd
CT
Cyto
Matrix D
Matrix C'
In the above, is a diagonal matrix with #1 diagonal entries, and C represents the joint kernel covariance matrix of the concatenation of the embedding vectors (01 (x)), th2 (v)2(x))] that is PSD matrix. Since C (d, + +22 x (d1 + d2) symmetric PSD matrix, we can apply the Cholesky decomposition to write C ZT for an upper-triangular matrix Z E (d)) Therefore, have I 11 DZ Z. While I 11.12 DZ Z is a symmetric matrix, note that the multiplication-orde flippe-- θ ZDZ dd)) matrix θ shares the same non-zero eigenvalues with r wa and for each eigenvector w of O, the vector u D2 w will be the corresponding eigevector of I v1. To conclude, one can perform the eigendecomposition for the symmetric matrix θ and then convert the non-zero eigenvalues and eigenvectors to compute those of p1 resulting in the following procedure:
will be symmetric matrix. Therefore, the symmetric
Cet
1. Form the PSD matrix C
CT V1.
Cez
2. Apply Cholesky decomposition to write C 7T for upper-triangular matrix Z E R(4++)))((+4a)))
Tai Xds
Od, xdg
3. Compute the symmetric matrix =
ZT
Odzxdi
-Id
4. Perform eigendecomposition on symmetric matrix ④ to compute di + d2 pairs of eigenvalues and eigenvectors dd+dd
(Ai, w;) d
Idxxd,
xd2
5. For each i (I,...di di + d2). compute u,
2 ZTw;
Odxxds
- - xd2
16

---

Towards Explainable Comparison and Alignment of Feature Embeddings
A.4. Proof of Theorem 2
As stated in the theorem, consider independent random Fourier features wi....,, wm R where the proxy feature map is:
((x) mm
cos(w] r), sin(w] z),, cos(wi sin sin(
Based the assumption, the shift-invariant kernel k(r,, y) s(x y) is normalized where (1) for every X. Therefore, using the Fourier synthesis equation k<r; y) RR [cos(w> (xx y)) EE [cos(w] 2)) cos(w" T y) +
sin(w" sin(w" y)]. yy]. The RFF-proxy kernel function can be viewed as
1 m
k( v)
(ws (x 8)).
As result, if we consider kernel matrix Kyran where kip un (sr, v) cos(uz? (4/1 (x)) - 1/i (w))), can simplify the proxy kernel matrix
m
K
K
s/1
72
171
72
where that Buwyrrp [w: Kag, is drawn from the Fourier transform R.
Also, | ae || < 1 holds, because the kernel function is assumed be normalized and [kg (z, v)1 < 1 for every x, Noting that the Frobenius norm || || can be written as the Euclidean norm of the vectorized matrix, the application of Vector Bernstein inequality (Gross, 2011; 1; Kohler Lucchi, 2017) proves for any S S
1
1 KKm ||- 26) < exp
8 - m-me²
K Kernw! m
32
Therefore, we will have
Re,
Ker
26)
< exp
88 mc²
n
32
Similarly, we can show that for the embedding 2ha will have
me²
P
K
K
2c) I
exp
32
17

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Next, we note that
71
Σ
A - -.1.112 A1 =
Ap
Σ
Aut. V. - -Ass. Vi/12
71
(Apin - -Tvinva)) 8/112
7
(Atrntt - -Rmr T
)( (Mvin Rorr ):
Tr (VT (Aorios Rere (( (Neinvs Avirvv)) )..)
==1
Tr (Apiee - Avenws)) (Apenita - -Notanols viv?? Σ
Th (A( - -osrwaa) T
( (Aerioos - Retrests)) ) a
[
piNT))
Tr (Apt.es - -pprviz) T
(Aeriios - (eerdss))
1.-
|2
A Agiin
- 2 Key - Rep 1| + K - Ryzl ||
The last line in the above incqualities follow from Young's ' incquality showing that || + Bl s 2|| B Then,
setting 8 exp mnc implying
32 (2ee/22)/4/0)) we will have
P ([| Rv
Kv 1|=
32 log 10)
2
PP
R
1
Kuz S
32 log (/o//22////0)
2 1
8 2
n
2'
m
where by applying the union bound we can show
P (
Rv
Ker
11- S
32 log 1oloo(22)) /8))
and
1 K
Kuz
5
32 log((22 10g(22!/4/8))
2 1-8
72
n
m
n
n
m
which shows that
128 log (2c2//1/1/)
A Aprio v. - -
2 -
The above completes the theorem's 's proof given that 201/1 2.57 <.
A.5. Proof of Proposition? 2
To show this statement, we leverage the fact that the eigenvalues of matrix a.,,, are real, as they are shared with the symmetric matrix Ag Then, we can leverage the Jordan canonical form to write the matrix PrLo as follows:
I PLLe -U
In the above, matrix U E tdd includes the right generalized eigenvectors of matrix I T a. as its rows, and U-1 includes the left generalized eigenvectors of matrix I V1..1, in its rows. Also, is the Jordan normal form containing one block matrix on the diagonal for every cigenvalue. Assuming that the eigenvalue with the maximum absolute value (which determines the spectral radius) has multiplicity 1, the Jordan canonical form has only one diagonal entry Amax for the top eigenvalue, and we can write the decomposition
T To = (mmma)))) J( bm-xmaa U -1 1( imax) Amaxx Ujeft 11 right
18

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Time (sec)
Computation Method
d 1000
d -20000
d =4000
=6000
d =800
d =10000
Direct (Algorithm 1)
13
35
72
Cholesky Decomposition-aasee (Appendix A.3)
11
22
33
47
64
85
Table 1. Time taken for Direct method and Cholesky Decomposition method at different values of d.
Due to the bi-orthogonality of Ujets Wright with the rest of right and left generalized eigenvectors, respectively, we will have
Amax (I (T.... = Utet I de Hright
Taking the partial derivative with respect to & from the above identity proves the proposition.
B. Additional Numerical Results
In this section, we provide additional numerical results on embedding comparisons and alignment using the SPEC framework, further illustrating its effectiveness in identifying clustering differences across various datasets and embedding methods.
B.1. Cholesky decomposition--aseed method for eigenvectors and eigenvalues computation
As explained in Remark 1, the computation of the eigenvalues and eigenvectors of the differential covariance matrix T vi can be efficiently reduced to the eigendecomposition of a symmetric matrix through the use of Cholesky decomposition. We followed the explanation in the Appendix A.3 and compared the runtime between Algorithm and the Cholesky decomposition--asse method on the Gaussian Kernel with different RFF. As shown in Table 1, the Cholesky method is more scalable and was able to handle d 10K, whereas the vanilla implementation of Algorithm using Pytorch's 's eig command (for general square matrices) could not efficiently scale beyond dimension d = 5000.
Furthermore, in Figures 6, 7, we present the top four SPEC-identified clusters comparing CLIP with DINOv2 using different values of Random Fourier Features (RFF) r. ranging from 1000 to 6000, with Cholesky decomposition applied to the MS-COCO dataset. We observe that for r S 3000, the clusters are not well-separated and show some inconsistencies. increases, these inconsistencies diminish, and by r 6000, the clusters are clean and well-separated. We also provide the t-SNE and UMAP representations of the top 10 SPEC-identifee clusters for CLIP and DINOv2 for 2000, 6000. in Figures 8, 9.
r
B.2. Ablation Study on the Comparison of Visualization Algorithms
To further validate that the SPEC method can distinguish between two embeddings, supplemented the UMAP visualization in Figure 2 with additional techniques, PacMAP (Wang al., 2021) and t-SNE, These visualizations assess how well the clusters identified by SPEC align across different dimensionality reduction methods. As shown in Figure 10, the alternative techniques confirm that SPEC correctly isolates distinct clusters in one embedding, whereas the other embedding fails to produce clean separations.
B.3. Comparison of text embeddings
Synthetic text dataset. To evaluate the performance of SPEC on text embeddings, we generated dataset of 10K text samples using GPT-4o, covering diverse categories such as profession, objects, gender, and emotions. We then applied SPEC to compare CLIP, RoBERTa, and E5 text embeddings. As shown in Figures 12 and 13, CLIP primarily clusters sentences based on objects mentioned in the text, whereas RoBERTa organizes them according to profession and gender. For instance, CLIP's first cluster consists of sentences related to cameras and photography, while RoBERTa's 's first cluster groups sentences about female firefighters and female carpenters. This suggests that CLIP embeddings do not cluster professions based on gender but excel at grouping objects, which aligns with its training focus. Additionally, the t-SNE visualization reveals that the principal clusters identified in one embedding are not well-separated in the other, indicating that each model captures different semantic aspects of the text. addition to previous experiments, we used MS-COCC 2017 train set captions( 120K samples) to compare RoBERTa and E5-L-V2 embeddings. As shown in Figure 14, we can observe that E5 managed to cluster captions where two or more animals are interacting.
19

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Cluster #1
Cluster #2
Cluster #3
Cluster #5
I I= == 1000
r t= 2000
3000
T= 4000
== = 5000
I= 6000
Figure 6. Top 4 SPEC-identified clusters comparing CLIP DINOv2 with different Random Fourier features (RFF) ranging from 1000
6000 using Cholesky decomposition on MS-COCO dataset.
20

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Cluster #1
Cluster #2
Cluster #3
Cluster #5
1000
T=
I=
I r 2000
3000
T= 4000
== = 5000
I= 6000
Figure 7. Top 4 SPEC-identified clusters comparing DINOv2 CLIP with different Random Fourier features (RFF) ranging from 1000
6000 using Cholesky decomposition on MS-COCO dataset.
21

---

Towards Explainable Comparison and Alignment of Feature Embeddings
SPEC Principal Clusters for CLIP-DINOv2
# of Random Fourier Features: 2000
Cluster #1 cigenval-0.007
Cluster #2 eigenval=0.005
Cluster #3 eigenval=0.004
Cluster #4
cigenval=0.003
I-SNE visualization of top 10 SPEC clusters with CLIP embedding
UMAP visualization of top 10 SPEC clusters with CLIP embedding
t-SNE visualization of top 10 SPEC
UMAP visualization of top 10 SPEC
clusters with DINOv2 embedding
clusters with DINOVZ embedding
-10
20
# of Random Fourier Features: 6000
Cluster #1 eigenval=0.007
Cluster #2 eigenval=0.005
Cluster #3 eigenval=0.004
Cluster #4 eigenval=0.003
visualization of top
visualization top embedding
I-SNE visualization of top SPEC clusters with DINOv2 embedding
visualization top
clusters CLIP embedding
clusters with DINOvZ embedding
10
-10
37
-00
-20
-30
Figure 8. Top 4 SPEC-identified clusters comparing CLIP DINOv2 embeddings MS-COCO dataset using Cholesky decomposition with different Random Fourier Features (2000 and 6000). The second row shows the t-SNE and UMAP representation the top 10 SPEC-identified clusters for each embedding.
22

---

Towards Explainable Comparison and Alignment of Feature Embeddings
SPEC Principal Clusters for DINOv2-CLIP
# of Random Fourier Features: 2000
Cluster #1 eigenval=0.007
Cluster #2 eigenval-0.005
Cluster #3 el eigenval-0.004
Cluster #4 eigenval=O.00s
E-SNE visualization of top SPEC clusters with DINOv2 embedding
UMAP visualization top 10 SPEC clusters with DINOv2 embedding
I-SNE visualization top SPEC embedding
UMAP visualization of top 10 SPEC clusters with CLIP embedding
-10
of Random Fourier Features: 6000
Cluster #1 eigenval=0.007
Cluster #2 eigenval=0.005
Cluster #3
Cluster #4 eigenval=0.003
eigenval=0.004
I-SNE visualization top 10 SPEC clusters with DINOv2 embedding
UMAP visualization top SPEC clusters with DINOv2 embedding
top embedding
UMAP visualization top 10 SPEC
embedding
1.5
5.0
255
-10
Figure 9. Top 4 SPEC-identified clusters comparing DINOv2- CLIP embeddings on MS-COCO dataset using Cholesky decomposition with different Random Fourier Features (2000 and 6000). The second shows the t-SNE and UMAP representation of the top 10 SPEC-identified clusters for each embedding.
23

---

Towards Explainable Comparison and Alignment of Feature Embeddings
UMAP Visualization of top Clusters
PacMAP Visualization of top 10 Clusters
t-SNE Visualization of top 10 Clusters
10
DINOv2:
DINOv2 - CLIP
-5-
-20
-15
-10
10
CLIP
30
CLIP
CLIP- DINOv2
-10
-20
20
0-
DINOv2
-20
-20
-10
0
10
20
30
40
Figure Comparison of SPEC-identiiied clusters across different visualization methods (PacMAP, (-SNE, UMAP) Figure experi- ment. One embedding shows clear cluster separation, while the other fails to distinguish groups.
24

---

Towards an Explainable Comparison and Alignment of Feature Embeddings
The prompt used to generate the text dataset: "You are an expert prompt optimizer for lext-to-image models. Text-lo-image models take text prompt as input and generate images. Your task generate a prompt describing person in [Profession]; [Emotion], and [Gender] performing [Action] with [Object]. You can randomly choose the categories from the attributes: Professions: Chef. doctor. journalist, scientist, carpenter, engineer. pilot, artist, teacher, firefighter. Emotions: Excited. calm, angry, serious, curious, confident, focused, determined, happy, tired. Genders: Male, female. Actions: Designing, adjusting, sitting, crouching, climbing, carrying, holding, standing, inspecting.jugae photograph, syringe, golden throne, desk, guitar, airplane."
Objects: Camera. frying pan, painting, laptop,
Real world text dataset. To further compare the text embeddings, we validated our approach on a large-scale real text dataset: WikiText-2 (Merity et al., 2016). We split the dataset into 10K samples, each containing 100 tokens. Then, we used SPEC to compare CLIP and RoBERTa embeddings. As shown in F We observed that RoBERTa better clustered Military Operations& & Infrastructure, Ecology& & Species Biology, Historical Figures, and Music, while CLIP embeddings more strongly clustered Entertainment & Sports and Science
In addition to t-SNE plots, we also examined the distribution of pairwise distances within each cluster to verify that one embedding successfully captured these clusters while the other was less inclined to do so. Also, we ran the K- means clustering algorithm 50 times on each of the embedding's 's features and computed the averaged (across the 50 runs) Normalized Mutual Information (NMI) between the K-means labels and the SPEC-identified labels. The results demonstrate that one embedding achieved considerably stronger alignment with KMeans labels.
B.4. Comparison of image embeddings
To explore the capability of CLIP's image embedding, we analyze the top 4 SPEC-identified clusters comparing CLIP and DINOv2 embeddings on ImageNet- Ik dog breeds in Figure 15, highlighting their different clustering strategies. CLIP primarily groups dogs based on their posture and gestures rather than their breed. For example, in cluster two, all dogs are standing, but they belong to different breeds. This suggests that CLIP focuses more on high-level visual features like body position and orientation. In contrast, DINOv2 forms clusters based on dog breeds, grouping visually similar dogs together regardless of their posture. The last row presents the t-SNE representation of the top 10 SPEC-identified clusters for each embedding, further illustrating their distinct clustering behaviors.
We analyzed the clustering behavior of different embeddings on 120K samples from the MS-COCO 2017 dataset and observed similar trends in how different models organize visual concepts in Figure 18. For instance, SWAV demonstrates a strong ability to cluster grid-like images, suggesting its emphasis on structural patterns in images. Meanwhile, CLIP excels at differentiating activities like surfing, capturing finegrained semantic details that may not be as distinct in SWAV or DINOv2. However. CLIP struggles to cluster certain sports, such as tennis, as effectively as SWAV or DINOv2, highlighting its relative limitations in capturing specific action-based similarities. These findings further illustrate the varying strengths of different embeddings in organizing visual content.
We also analyzed SPEC on 70,000 samples from the FFHQ dataset. As observed in Figure 19, DINOv2 better distinguishes images where two people are present, with one appearing incompletely, as a cluster. In contrast, CLIP is more effective at identifying children as a distinct group.
B.5. Comparing similarity ranking for SPEC clusters
In Figure 16, the leftmost images show the top 4 samples of SPEC-identified clusters on AFHQ. The first and second clusters correspond to black cats and black & white cats, respectively. For each cluster, cosine similarity is computed between the mean of these samples and both 4 additional cluster members (green- bordered) and 4 random images (red-bordered). Images are sorted left to right by similarity scores. The first row represents DINOv2, the second CLIP. Unlike DINOv2, CLIP ranks some random cats more similar to the cluster mean than the actual cluster members.
For the FFHQ dataset, shown in Figure 17, the first cluster corresponds to people wearing graduation caps, and the second to people wearing sunglasses. While DINOV2 maintains significant similarity gap, clearly distinguishing the clusters, CLIP assigns higher similarity to some samples that lack these defining features.
25

---

Towards Explainable Comparison and Alignment of Feature Embeddings
SPEC Principal Clusters for RoBERTa- CLIP
Cluster
Clustes
Claster
Cluster E- 8466
End cae
Eveaon
WWI Siaal Railaay Contrration (1916)
Haliine of Pygeballic sepuncteti
Lant Rosdbery's Peorttiiios
Codepay's "Clods"s"C"osss "Chock""
Ostoran Water Saune Destreriinn (1818)
Life Cyclect of Zygohilta sespunctatic
Anchiaatd 5th Farl Reecery
Kihanro'ss Dlamends World Tour
Onli Decripton
Brashtect's
Love"
Slavery Misiaas (1893s)
Oribi Schasier
Diedkare bat's Reign
Markesk''
WW1 Noval Raid Plzating (993 18)
Kekapo Pacrot Charackabslies
Neolilic Loug Bariow Folkioee
Whitney Hosdterss Resiows
St. Nacrie Raid Pan (1922)
Nony Miner Tasenoryy
Aricamedaa' Roman Connection
Beysnes""Fraaem Drod'
Marjung Sun Battle Bkcrran War, 1953)
Taway Nure Shark Ensiag
Coldran Stones Bodiseovery
The Famils Jench Alben Recoption
Fot Glanvilk's
Filse Morel Frailing
Feru
Destrction
Skmna''s
Glantili's
(189),
degans (Stditoo Pingus)
Märquer's Histurical
Magadidtu Read Moderttozionn (2015-1) 15)
Anyfosteraaso Fergal Fanily
Kining Goffrenn's Iefluence
Castiny
Am
+SNE vissalization of top 10 SPEC clusten with BoBERT, anbeadiin
+NNE vissalization of top SPEC dusttn with CLIP eubrelling
Noemalized dister Distances top 32 SPEC-idnniGee cuuier RoBERTa-CLIP
Duakers
Ceder
RoBERTa KMeans & SPEC NME: 0.82 0.0004
CLIP KMeans & SPEC :M1: ILODIS
SPEC Principal Clusters for ELIP-ROBERTE
Claster Auiiaa
Cluster
3. 6l0)
2. 601
Tristas (hand) Carze Highight
Hydrootatic equilitriam strlbrr Astrephysin)
Cardinal Appoisinctto of Septatbeer 1756
Reger
Ross and Rachel: Caltaral Anelysi of Friceds
Xenen compounde and their chemial (Chmbirry)
Milasese Cardital Unde: Benede: XIV
Hamode 2009 poetuason flres Yankon)
Chilaa E.C. in the 1550: Fini Division Triumph
Antiminy axiites and halldes (Chonistr)
Papell Legates Appoinee 1799
Harricas Hisknn wroding moses
Ona Nota: Mass Anthany's Fivotal Tratation
Tripical syclane formation from exeurly
Eivech Candinah Under Clement (136)
Track and field event clastication
The
Marringe
Liiayyd Réinioon laland
trapical skpreasin
Pope
Cardinabae
Felare's playing
Midge
Sparish
Roger
Nowconte United Reerves 17-88 Contril Lagu
Impact of apricaltare on frsh vidile labitts
Pape Cheneer NIV (1343-1754)
Regir Wederer 2004
Odyyay Namber Tiee
Exiinet Taemn Rétnion blaad
Antini Cakonaa Branciione - Cardinal
2017 Azarbullan Grand Pris
Wrewiling Champiindepe Ennough the Agas
tap pentection
Urbano Parreciori Rutlic Sep 1756). Cartinal
Fockeer Rodiick chraliy "21-3 head-to-head recerd
embedding
Nosmalized Intra-dister Distances top 50 SPECideniffee cluster RoBERT&-CLIP
Olider
Olderr
Cadara
Clatts
CLIP KMeans & SPEC NMI: : 0.63 0.0011
RoBERTa KMeans & SPEC NMI: 0.32 = 0.0003
Figure 11. Top 4 SPEC-identifedd clusters by comparing CLIP and RoBERTa text embeddings on the WikiText-2 dataset with the visualization of the top SPEC-identified clusters using I-SNE.
26

---

Towards Explainable Comparison and Alignment of Feature Embeddings
SPEC Principal Clusters for CLI''ROBEER
Cluster 12
Cluster
Cluster d4
serious fenslle adjustngg
determined teacher stting atptinn
carrynng frying pas.
ttsjecting serious fomole ongineer aajusting poltetng
determined male enginerr is holfinn careern
dettemined male teacher string alrplae
ungry make carpetier adjusting fiying pan
A happy fenale dretor to haafuoning- phatograph.
coriiido femall cargerer extin airplone
A huppy fanak docice is oljustig a fiylgg pan.
coim frmall journeli ingpeting paluting
A calm femalz piat doagging photegraph
focuedd femals dectur adjuning sirplane.
A happp male aritt odjistiss frying
calin male you tallst isspecting painting.
desernined
aljuiing pleourghh
adiustin sitingg skplanc.
pan determined adjueing fryleg pan
chef's
fomale
pointing
detertised fanale enginces bokiing cemera.
An enery fetrale
icadasiing pabeting
happy lecule artist is adjustng photigraph
tied fanale chef sitting en akylane.
focsurd enginuer is inepecting frying pan verion male scientic inspecting frying pen calts fimale pilor adjusing
foctaod engineer inpaaing painting
An male journolls: is bokiing photograph.
colnn terralt dector insrecting airglone
determined make dector hoiling painting
calm male engincer holling phonorrahh
determine male holding slrplre
Soctsed maal cusineer curying paintizz
saNE visualization 10 siec. cabbllding
visualization SPEC
b-NE vivsalization sirt."
visuaiiaaiinn top clusters RoBERT eabelding
embolding
canbeddiig
100
II-
-15
-20
25
-50
199
-12
Normalized Intra-cluster Distances of top SPEC-identified
RoBERTa
Normaliee coctuntee Dafantos
RoBERTa
ROBERTO Intair Mean: 1.07
Clusters
KMeans AMI:
KMeans
AME: 0.0010
SPEC Principal Clusters for RoBERTa CLIP
fonell
Zappy carponeer carrying lapoop.
desi.
plotograph
junnalie
corpester holding syttage.
A happy male ingestig flying pan.
colnn joumallst holding photographh
encited femule Bevlighter holding a
A foosed male cargester haiding comma.
A tappy male wille corrinng doak.
focaeo femole jouraalk holding a galber
A iirdi female didiigher is holding dok.
An csriied male carpenter holding camma
A curiuss male antie is inppeting comes.
Acin fimole jummale is inspecting comers
A happy famale firdiggter holding guiter
A contidezi male carpenter' hoiding caners
An excited muk artit carrying dok.
An saary mak jeumalist bolding phatograph.
hobling corpester carrsigg laptop.
happy
bolling
confidest
inpecting
isurslist sitting angry male joumallst helding a fiyiag pai.
es pester bobling caniera.
excited antit holding laptog.
A caln Renele repenter holingg dest.
A calnn male cnr penter is holiing cariem.
An angry male wrihs inspecting comera. A happy male withe hokling pointing
cain fomele jurnllst is carrying pointigg An engry timele jummitee is carriin gaitor
An angry fomale ourpatrr sanding dask.
A happy male corpenter holding frying pan
USNE visualization of top SPBC
UMAP visualization top 18 SPEC
eSNE vissalization op SPEC dlusten with CLP ewhelding
UMAP visualisation of top SPEC clusten with CLI emhedding
clsttes with BonERTs embelding
dusttes with HottERT embedding
Normalized Intra-cluster Distances of top 10 SPEC-identified clusters RoBERTa- CL.IP
Horallord Disances
CLIP
RoBERTa Inter-cluster Mean: 0.90
CLIP Inter-dlustr Mean: 1.03
Clusters
RoBERTa KMeans SPEC AMI: 0.72 0.0010
CLIP KMeans & SPEC AMI: 0.0014
Figure 12. Top 4 SPEC-identifees clusters by comparing CLIP and RoBERTa text embeddings dataset of 10K samples generated from GPT-4o with the visualization the top SPEC-identified clusters using t-SNE and UMAP.
27

---

Towards Explainable Comparison and Alignment of Feature Embeddings
SPEC Principal Clusters for CLIP E5
An limak doigting phutograph.
photograph.
An angry male sniid i mecsting syringe
cales male adenaal climbing gullen theeee.
An angry mile aniic herrryynng frying
mak
A happy male engineer udjuningg syringe. wria
desersized make dovasing
scrioss make scientia lng
calin fanale enpinees desigting platiarraph.
angry
angay scleniis designiis pobkes tired fernle sclestist sitting on goiéee throse.
famale
An eugry male plet is desigting a plotegraph.
A mals juursalss inpaaiing syrage
rited femal: ademnisi carrynng pristr
A dedeermned male antic designing plotegrogk
A dearmined male wienis is hoking a qrings.
An angry male ortist desigling geldee throse.
curioua fermie dacier is cary ing liying
male erginarii dasignng phutograph
desigring phuttgraph
A confidenn male joumalis imsecting usinge.
tinad male wtentic donigning grlkee thease.
An snary fetail chef carrying frying pui. detemmind
huppy engineer
priken
fanale aicntii boking phoingraph.
coufident female
adjusing syringe
conficent make atentin
polken flrees.
caryleg putar. curioss femol ockmiii is carrylag caroers.
serioys
A heapy male docioc idjesiin photograh.
curioos mals joumslia kadjjiinng syrlegn
A cales male sccecit standing golden throse.
e-Ni vouulication of top sPIC
UMAP visualization of top to SPEC clusteos with CLIP enbedding
1-NE visualization of top 10 SPIC
UMAP sisualization of top SPCC cluster with embedding
clantere with CLPP embeddinng
clusten with (ts embedding
Normalized Intra-cluster Distances of top SPEC-identified clusters CLIP
Normalzed intra cluser Distonizs
E5
ES inter-cluster Mean: 0.99
cuiber
Guaters
Clusters
CLIP KMeans & SPEC AMI 0.68 a 0.0003
E5 KMeans & SPEC AME: 0.43 a 0.0022
SPEC Principal Clusters for E5- CLIP
Claster #1
Clister
Cluster #3
Clester
foased femalee Brvlighter carrying laptop
licussed mude forfightr is holding tepoop A calm male findiightr is holding a paintng.
augre femede cacrying typop
female merigttr iocurying grings
A sariona famals Heviightr cimbing comera.
As angry fomale pilat hulding phocograph.
dinel fumale dustor bs adjuoiing photograph.
A esciaal kanal firelihher standing painting.
happy male findiightrr is standing liptop
fomads dhaf'a holding syrings.
dind female destor designnng frying pan.
Sanal
holding laptop.
fietiahter
fesall
adiuitnng
fouale fteffiitter lueidig guita..
focused fleffihter croschingg
augrs
desigting caneeaa.
scientist odjesting
A focsed fonale firefighter crcuching careeaa
detee nined male efigher holding camen
An aegre Senale pilot corrying desk.
dind fimale frelighter hokding camena
A caln fonals fiefighter bodling& laptop.
As excited male firlighter croaching guiar. fieffllter
As fonal artist adiating lastop.
died female gllc currynnz photogrgh.
A focused fiefighter liddigg
fonnll cheff desigting
dire feuslle teocher
A confider: frrefighter holdag logtog.
costident fheffghter siting punar.
aegre
caniing pointing
holding phetograph.
LSNE Ssualication top
UMAP vivullizaion SPEC
MSNE viesalization sanc clustons with CLIP anbelldin
visullizaiion embedding
embedding
enbedding
30
Normalized Intra-cluster Distances top CLIP-E5
Interclucter 0.99
CLIP Inter cluster Mean: 1.04
Narmaltaad
Ouoder
Cleier
Cale
Orler
Coler
Olver
Sudeer
Cleier
Duekr
Clustors
CLIP KMcans& SPEC AMI 0.55 0.0008
ES KMeans SPEC AME: 0.32 0.0004
Figure 13. Top 4 SPEC-identifee clusters by comparing CLIP and E5-Large-Va V2 text embeddings on dataset of 10K samples generated from GPT-4o with the visualization the top 10 SPEC-identified clusters using t-SNE.
28

---

Towards an Explainable Comparison and Alignment of Feature Embeddings
SPEC Principal Clusters for RoBERTa- E5
Closter
chain sirting Eprraed
with brrg dipaaoidiii
buildingg
white
litclen. white suilet sitting bothroom window.
parked front
A wommn and wone kids sitting on couch.
a bothhruu with a woskan doox and Like wall.
sits next bus stup
A white toilet sitring under bathroom window.
A large refrigerator Uleeur sitting in kiichen.
white bath with doublo sink
A yellow bas driving down street past cass.
A dog sitting in frent daurway.
refrigerator freezer strtnn in
roum
and large mirnor
passenger bus parked frost building.
A small black dog sitting front of TV.
kichen.
butlecor with sink toilet and mirrer
A red doulde dacker bas nest to park.
sitting looking
window.
building A black chair sittng next to refrigerator A white soilet sitingg under buthi sink.
wooden bencht sitting
lange
buthrub
A blue and yellow double decker
street.
A white sefr liceeet sitting a kischen.
a buthroon with a sink and two mirrors
A road with cars and buses near building
A benchee sitring inside building.
bethrooms with sinks mirror
A car the street next Bus.
A person sittng oc a wonden bench cutside.
white sink dittigg under a window.
a small bathrosm with mirror and sink
bas parked front buikding
t-E visualization of top 10 SPIC embedding
I-SNE visualization of top SPEC embedding
SPEC Principal Clusters for E5 RoBERTa
mun standing nest giralle snow
Two zeleas standing logether
outside
socme mectt some vegples sitting on plate.
zuy jumps in
slate
mon holds snowboard on snowy day.
Two giratfes standing front their stables.
A cut pizza sits plate in front of person. close of plate of
man who niing skatchoard down the street.
riling beard
of'a beut.
prussy
fling through
skateboard
A man that on snowboard sanding in the stow.
Africon butfale and zebeas standing a sovanna.
A very close up picture of some food on table.
mas on stuteboard performing miik.
picture of
ski's standing the mon holding noraaal snow.
At least three rebeas standing together affiet iomtley
A sitting table where
there
skateboard performing miik.
close up ofa plaza a plate un table.
mon skateboard performing trick.
man riding a surtboard in the ocean
Two zebras stand in from: a large building.
close of plate of food with pirxa
man in the air a statehoard.
standing soowhrand non raling a sarthsandd in the waves
gleaffe A adhra standing soar smotlar selea.
checkerol A picture of's plate pieca on tublie
plate plaza
coverr
daing a their board A man rilling i skateburrd doing trick.
E-NE visualization of top 10 SPEC clusters embeddinn
+SNE visualization of top 10 SPEC clusters with CLIP embedding
Figure 14. Top 4 SPEC-identified clusters by comparing RoBERTa and E5-Large-V V2 text embeddings on MS-COCO 2017 train captions ( 120K prompts) with the visualization of the top 10 SPEC-identified clusters using t-SNE.
29

---

Towards Explainable Comparison and Alignment of Feature Embeddings
SPEC Principal Clusters for
Cluster #1 eigenval=0.082
Cluster #2 eigenval=0.011
Cluster #3 eigenval=0.009
Cluster #4 cigenval-0.006
SPEC Principal Clusters for DINOv2-CLIP
Cluster #1 eigenval=0.016
Cluster #2 eigenval=0.013
Cluster #3
Cluster #4 eigenval=0.009
eigenval=0.011
Comparison of top 10 SPEC-identified clusters for CLIP-DINOv2
Comparison of top 10 SPEC-identified clusters for DINOv2 CLIP
UMAP visualization CLIP
UMAP visualization of DINOv2
UMAP visualization of DINOx2
UMAP visualization of CLIP
33
-10
-30
Normalized Intra-cluser Distances of 10 SPEC-identFied cluxtens DINOV2
Normalized Intro Distances identidiedd clustes DINOV2 CLIP
toam
KMfaan & DINOv2 KMeans SPEC AMI: 8.23 0.0003
DINOv2 KMicns SPEC AMI: 0.0008
KMeans SPEC AMT 0L0024
Figure 15. Top SPEC-identified clusters comparing CLIP and DINOv2 embeddings on ImageNet- Ik dog breeds. The last row shows the UMAP representation of top 10 SPEC-identifed clusters for each embedding.
30

---

Towards Explainable Comparison and Alignment of Feature Embeddings
DINOV2
0.5627
0.1418
-0.0611
CLIP
DINOV2
0.8707
0.8681
0.1897
CLIP
0.9692
0.8653
0.9429
0.9290
0.9197
0,9008
Figure 16. Comparing similarity ranking for SPEC clusters in DINOV2-CLIP the AFIQ dataset. The leftmost images show the top samples of two SPEC-identified clusters. Cosine similarity is computed with 4 cluster members (red-hordered), sorted hy score. Unlike DINOv2, CLIP ranks some random samples higher than cluster members.
and 4 random images
DINOv2
0.9480
0.9245
0.9101
0.8869
0.2676
0.2388
0.2272
CLIP
0.6206
0.8182
0.8120
0.8124
0.8093
0.7892
0.6868
0.5790
DINOv2
0,8463
0.7801
0.7643
0.7558
0.511
0.4491
0.4584
0.4012
CLIP
0,8223
0.2266
0.221
0.5249
0.5012
07740
07345
0.6582
Figure 17. Comparing similarity ranking for SPEC clusters in DINOV2-CLIP on the FFHQ dataset. The leftmost images show the top samples of SPEC-identified clusters. Cosine similarity computed with 4 cluster members (green-bordered) and 4 random images (red-hordered), sorted hy score. Unlike DINOv2, CLPP ranks some random samples higher than cluster members.
31

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Comparing Different Image Embeddings using SPEC MS-COCO dataset
Cluster #1
Cluster #2
Cluster #3
Cluster #1
Cluster #2
Cluster #3
CLIP DinoV2
DinoV2 CCIP
CLIP SWAV
SWAV CLIP
Inception SWAV
SWAV Inception
Figure 18, Comparing Different embeddings on the 120K samples from MS-COCO 2017 dataset.
Cluster Cluster
Cluster
UMAP Visualization of top 10 Clusters
eigenvalue
eigenvalue 0.004
eigenvalue 0.003
DINOv2
CLIP
DINOv2 CLIP
eigenvalue 0.010
eigenvalue 0.007
eigenvalue 0.005
DINOw2
CLIP DINOv2
DINOv2 KMeans & SPEC AME: 0,50 0.0002
CLIP KMeans & SPEC AMI: 0.0004
CLIP KMeans & SPEC AMI 0.67 0.0007 DINOV2 KMeans SPEC AME 0.33 0,0003
DINOv2 - CLIP
CLIP- - 2200
Figure 19. Comparing embeddings on 70K FFHQ samples. Top numbers show SPEC cluster cigenvalues. Last two images per row
display UMAP representations of SPEC clusters for each embedding.
32

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Cluster
Cluster
Cluster
UMAP Visualization top Clusters
eigenvalue 0.015
eigenvalue 0.008
eigenvalue 0.006
Inception
CLIP
Inception - CLIP
eigenvalue
eigenvalue
eigenvalue
CLIP
Inception
CLIP - Inception
Inception- CLIP
Inceptiin KMaans & SPEC AMI: 0.76 0.0022
CLIP KMeam SPEC AMI: 0.20 0.0001
CLIP- Inception
CLIP KMeans & SPIC AMI: 0.80 0.0026
Inception KMeans SPLC AMI 022 0.0022
Figure 20. Comparing emheddings on 70K FFHQ samples. Top numbers show SPEC cluster eigenvalues. Last two images per row display UMAP represettttoo of SPEC clusters for each embedding.
B.6. SPEC-align Experiments
SPEC-align using f2-Norm of Kernel Difference Matrix. As mentioned Remark 2, another choice of SPEC-diff uses the f--norm of the eigenvalues of the kernel difference matrix #2 Similar to the experiment Figure 5, we aligned CLIP to DINOv2 using this fg-norm-based objective. Figure 22 shows the CLIP kernel before and after alignment, alongside the target DINOv2 kernel. We also plot the SPEC-diff score over iterations, demonstrating successful convergence of the alignment process.
SPEC-align Finetuning Experiments Parameters. The following parameters were used in our experiment. We used the OpenCLIP GitHub repository (link) and used the MS-COCO 2017 training set, which consists of 120K pairs of texts and images. We use SPEC-align with the following parameters and chose MANOv2-Vit-B/L. as our reference model.
Comparison Kernel matrices for SPEC-align. As shown in Figure 26, the SPEC-aligned CLIP kernel captures the top four clusters based image content rather than the overlaid text labels. Furthermore, according the t-SNE of the models, the SPEC-align cluster of fish is close to the cluster of images with overlaying text of fish showing that SPEC-align captured the similarity of text and image in this experiment while clustered based on the ground truth (cluster of images).
Aligning text embeddings. In addition, we aligned CLIP text features to the T5-XL model. In Figure 25, we can observe that the CLIP kernel has become more similar to T5-XL, and the SPEC-diff is also decreasing.
Clusters Comparison of SPEC-align. We provide additional results by comparing the top Kernel-PCA (Gaussian RBF kernel) clusters of CLIP, DINOV2, and SPEC-align CLIP, We used the CLIP aligned with DINOv2 on the ImageNet training set. We compare the clustering of these embeddings with ImageWoot Woof and the text-overlaid dataset in Figure 3. In Figure 23, we observe the top 8 clusters the text-overlaid dataset. DINOv2 clusters are based the images, while CL.IP clusters are based on images and texts and in some cases fail to cluster based on the image. On the other hand, SPEC-align CLIP clusters based on images while focusing on the images with the same text, as expected. The top clusters of Image Woof in Figure 24 also show that CLIP clusters the dogs based on the gesture or their interaction with humans or the number of dogs, while DINOv2 clusters them only based on their breed. But SPEC-align CLIP clusters dogs based on their breeds while
33

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Cluster #1
Cluster #2
Cluster #3
UMAP Visualization of top 10 Clusters
eigenvalue 0.051
eigenvalue 0.021
eigenvalue 0.014
DINOv2
SWAV V CAONE
cigenvalue = 0.027
eigenvalue 0.021
eigenvalue 0.017
SWAY
POONIG
SWAV- DANDINANN
SWAV- DINOv2
DINOV KMeans SPEC AMP: 9.78 uJeus DINOS KMeans SPEC AMI IUSI 0.0000
DINOv2 SWAV
DINOv2 KMeans SPEC AMI: 1.00 0,0000
SWAV KMrans & SPEC 0.8 0.0006
cuatr Mear
Cuter
Figure 21. Comparison of different embeddings on 15K samples from the AFHQ dataset, consisting 5K cats, 5K wildlife, and 5K dogs. The number at the top of cach image represents the cigenvalue of the corresponding SPEC cluster. The last two images in each row show the UMAP representation the SPEC clusters each embedding individually.
34

---

Towards Explainable Comparison and Alignment of Feature Embeddings
CLIP Kernel
SPEC-align (fy-norm) CLIP Kemel
DINOv2 Kemel
05
SPEC-diff (42-Norm) during aligning
0.5
SPEC-diff (lz-Norm)
CLIP aligning to DINOv2
0.4
0.3
0.2
0.1
0
200
400
600
800
1000
Number of Iterations
Figure 22. SPEC-align using the éu-norm of the cigenvalues of the kernel difference matrix. Left: original CLIP kemel. Center: aligned CLIP kernel using f2-norm-baat based SPEC-align. Right: target DINOv2 kernel. Bottom: SPEC-diff score (f2-Norm) over alignment iterations.
35

---

Towards Explainable Comparison and Alignment of Feature Embeddings
CLIP Clusters
parsciada
SPEC-align Clusters
Branch baan
Porashuto
Casselle
gas puup
GOR SASS
paleze
Pa
DINOv2 Clusters
Figure 23. Top 8 Kernel-PCC -Cnnel-CCAAA (Gaussian RBF kernel) clusters for CLIP, DINOV2, and CLIP aligned with DINOv2, trained on the ImageNet dataset.
36

---

Towards Explainable Comparison and Alignment of Feature Embeddings
CLIP Clusters
SPEC-align Clusters
DINOv2 Clusters
Figure 24. Top Kernel-PCA (Gaussian RBF kernel) clusters for CLIP, DINOv2, and CLIP aligned with DINOv2, trained on the ImageNet dataset.
37

---

Towards Explainable Comparison and Alignment of Feature Embeddings
Parameter
Value
accum freg
alignment.loss.weiie
0.1
site
128
0.9
coca. contrastive loss. bessht distributed
1.0
True
epochs Ir
10
1e-05
Ir ir.scheduler
cosine
model
ViT-B-32
name
Vit-B-32.laion2be 1r.feeeee.
precision
amp
pretrained
laion2b e16
seed wd
0.2
Table 2. Configuration parameters used in the experiments.
Table Linear evaluation frozen features on fine-grained benchmarks.
Model
Architecture
Data
Imagenet- 1K
OpenCLIP
ViT-B/32
LAION 400M
73.50
SPEC-align OpenCLIP
ViT-B/32
LAION 400M
76.45
DINOv2
Vit-B/14
LVD-142M
78.99
focusing on the gesture the number of dogs their interactions with humans.
38

---

Towards Explainable Comparison and Alignment of Feature Embeddings
SPEC-diff score during aligning
CLIP aligning to T5-XL
SPEC-diff
6.7
66-
1050
2000
2003
4200
Number of Iterations
CLIP Kemel
SPEC-align
T5-XI, Kerne
3000
4000
Figure 25. Comparison of Kernel matrices after using SPEC-align match the sample clusters of CLIP T5-XL with measuring SPEC-diff during the training.
39

---

Towards Explainable Comparison and Alignment of Feature Embeddings
t-SNE visualization of top SPEC-identified clusters for CLIP DINOv2
CLIP
DINOv2
1
0
-1
-2-
-6-
-3
-4
-2
2
6
121
SPEC-align CLIP with DINOv2
2
-2
CLIP Kernel
SPEC-align CLIP Kernel
DINOv2 Kernel
OM0022
000033
Figure 26. Comparison of Kernel matrices after using SPEC-align align CLIP DINOv2.
40


Reviews:
[{"review_id": "vR6FylyjkN", "reviewer": "Reviewer_hTAA", "summary": "The authors derive a method for identifying clusters which are strongly clustered by one encoder and weakly clustered by another encoder. The run time of the method scales linearly with the number of samples in the dataset. This is deployed on a few image datasets for some established image foundation models, and a collection of text samples for some established language foundation models. The authors also propose and briefly demonstrate that the metric can be deployed for fine-tuning to transfer the clustering of one encoder into another.\n\n## update after rebuttal\n\nI am happy to increment my score 3->4 after clarifications and requested changes from the authors during the rebuttal period.\n\n*Note on Adjusted Mutual Information for new results*\n\nIt would make sense to either use NMI or AMI for all the results; it is redundant to show both, and inconsistent to swap between them. AMI is generally better than NMI because it accounts for the coincident information measurements due to chance. It's especially important when the number of clusters differs between clusterers to compare, hence that's when it becomes critical to use AMI instead of NMI. If you're in the situation where this sometimes happens, I recommend you swap to using AMI for *all* results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer hTAA for the thoughtful and constructive feedback on our work. Below is our response to the comments and questions in the review: ([Our Numerical results are shown in this link](https://github.com/ICML6204/ICML6204/blob/main/ICML_Rebuttal.pdf))\n\n**1- Experiments on GPT-4o generated text data**\n\nWe generated a dataset using GPT-4o, covering categories including professions, emotions, genders, actions, objects. We used the bellow prompt to generate. In the revised manuscript, we will explain the generation process.\n\nThe prompt for generating the text dataset: “You are an expert in text-to-image models. Text-to-image models take a text prompt as input and generate images. Your task is to generate a prompt describing a person in [Profession] [Emotion], and [Gender] performing [Action] with [Object].\n\n**2- Additional experiment for real text datasets**\n\nTo further address the reviewer’s comment, we also validated our approach on a large-scale real text dataset: WikiText-2. We split the dataset into 10K samples, each containing 100 tokens. Then, we used SPEC to compare CLIP and RoBERTa embeddings. The results can be found in Figure 2. We observed that RoBERTa better clustered Military Operations, Species Biology, Historical Figures, and Music, while CLIP embeddings more strongly clustered Sports and Science. \n\nWe also examined the distribution of pairwise distances within each cluster to verify that one embedding successfully captured these clusters while the other was less inclined to do so. Also, we ran the K-means clustering algorithm 50 times on each of the embedding's features and computed the averaged (across the 50 runs) Normalized Mutual Information (NMI) between the K-means labels and the SPEC-identified labels. The results demonstrate that one embedding achieved considerably stronger alignment with KMeans labels.\n\n**3- Evaluation on SPEC-diff**\n\nTo show that SPEC-diff effectively measures embedding similarity in SPEC-Align, we considered the SPEC-Align of CLIP to DINOv2 and tracked the SPEC-diff scores while plotting their kernel matrices. The results in Figure 3 show that as the kernel matrices look more similar, the SPEC-diff value decreases. The results suggest the SPEC-diff's utility for quantifying embedding alignment in SPEC-Align.\n\n**4- Explanation on the statement in Line 156**\n\nWe will make the explanation clear that since the multiplication order in $C=\\frac{1}{n}\\Phi^\\top \\Phi$ and $C=\\frac{1}{n}\\Phi \\Phi^\\top$ is flipped, they share the same non-zero eigenvalues, and those eigenvectors are in one-to-one correspondence.\n\n**5- The choice of Gaussian vs. cosine similarity kernel functions**\n\nThe choice of kernel function determines the similarity measure used for the clustering algorithm. The cosine similarity looks only at the angle between the vectors, while the Gaussian kernel concerns the Euclidean distance between the input points. The choice of the kernel function depends on the input embeddings and their induced geometry in the embedding space. As we discussed in the text, our analysis can be efficiently applied to both kernels.\n\n**6- Relationship to Laplacian spectral clustering**\n\nIn the revised draft, we will make the connection to Laplacian spectral clustering more clear. One main difference between the proposed SPEC clustering and Laplacian spectral clustering is the usage of kernel matrix in SPEC (similar to Kernel PCA) vs. the Laplacian (kernel minus the diagonal degree matrix) in spectral clustering. We will explain this difference in the draft.  \n\n**7- Additional results on embedding alignment using SPEC-Align**\n\nWe aligned the CLIP text embedding with the T5-XL model. In Figure 6 of the rebuttal link (last page), we observe that the CLIP kernel has become more similar to T5-XL, and the SPEC-diff is also decreasing.\n\n**8- Symmetry of SPEC-diff**\n\nWe would like to clarify that SPEC-diff is a symmetric measure because it is defined as the *spectral radius* (eigenvalue with maximum absolute value) of the difference of the two kernel matrices. Since the definition concerns the spectral radius (and not the maximum eigenvalue), the SPEC-diff is symmetric with respect to the embedding order. We will make this point clear in the writing.\n\n**9- Clarification on kernel values**\n\nWe would like to clarify that the paper’s main results suppose a normalized kernel function where for every $x$, we have $k(x,x)= \\langle \\phi(x), \\phi(x) \\rangle=1$, i.e. $\\Vert \\phi(x) \\Vert = 1$. Following Cauchy-Schwarz inequality, $| k(x,x’)|\\le 1$ holds for every $x,x’\\in\\mathcal{X}$. We will make this point clear in the revised draft. \n\n**10- Typos and writing improvements**\n\nWe thank the reviewer for pointing them out. We will correct them in the revised draft.\n\n**11-Change from CLIP to OpenCLIP**\n\nFor alignment experiments, we utilized the widely-cited open source OpenCLIP github repository. We note that in Table 1, the CLIP backbones are the same.", "claims_and_evidence": "Okay.", "methods_and_evaluation_criteria": "The text dataset used is a bit opaque. It is not clear why GPT-4o was used to generate the text samples instead of using an existing text dataset, and it is also unclear what prompting was used to generate this text data.\n\nIt would be advantageous to include more quantitative evaluation. For example, does low SPEC-diff on a dataset imply that two models have similar performance at the task established by the labels for that dataset?", "theoretical_claims": "The proofs are in the appendix. I checked \"A.2. Proof of Proposition 4.3\", which I take to be correct.\n\nSome steps in the main text eluded me and could be made clearer. For instance, L156 makes sense having read Proof of Proposition 4.3, but wouldn't otherwise. Also, it was unclear why a Gaussian RBF kernel needed to be used when other parts of the work were happy to support the kernel used being cosine-similarity.", "experimental_designs_or_analyses": "Okay.", "relation_to_broader_scientific_literature": "The work is novel and relevant. This tool has the potential to be useful for the model interpretability community.", "essential_references_not_discussed": "The relationship to Laplacian spectral clustering (e.g. Ng, et al, NeurIPS 2001) wasn't really discussed. I think it would be helpful to articulate the similarities and differences between the SPEC method and spectral clustering to the reader.", "other_strengths_and_weaknesses": "**Strengths**\n\nIt was not initially obvious that this eigen-based clustering comparison would scale linearly with dataset size, but the proof and algorithm made it clear this was the case. This scaling is important for the utility of the method, since it is desirable to be able to deploy at scale.\n\n**Weaknesses**\n\n- The work on fine-tuning using SPEC seems a bit rushed within the paper, possibly it was a late addition and space was limited. I think the paper would benefit from further experiments exploring this component of the work. The fact that this can be done to induce clusterings from one encoder into another *without having to align their embedding spaces* is certainly a strength, but it is not highlighted in the paper.\n\n- The paper should emphasize clearly that the distance metric is asymmetric.\n\n- Fig 1 caption should show more detail about which classes and samples are selected to show in the tSNE plots.", "other_comments_or_suggestions": "- L133 left. The matrix is positive semi-definite because k is an inner product, so $k(\\cdot,\\cdot)\\in[0,1]$. However the more general statement for a kernel function only constrains its output to the reals. I think the implication could be made more clear by adding the missing step, e.g. on L131 by specifying $<\\phi(x),\\phi(x')>\\in[0,1]$.\n\n- L068 right. should be \\citet not \\citep if the citation is part of the text, as it is here. It is bad English grammar to write that something is in (a parenthetical description). As author names appear in the text, you should replace this with \"has been studied by \\citet{ref1,ref2,ref3,ref4,ref5}.\"\n\n- L077 right. in this case the citation of Gao (2021) should be parenthetical, \\citep, instead of textual. The Muennighoff citation is correctly textual \\citet. However this sentence doesn't make sense anyway - how is Muennighoff (2023) offering something within the work of Gao (2021)?\n\n- L080, L099, L103, L107, L369 citations should also be \\citet\n\n- L145 left. Bracket should go around the whole fraction (use `\\left(` and `\\right)`)\n\n- L164 left. Should be $\\phi_1$ and $\\phi_2$, not $\\phi_1$ twice.\n\n- L180 left. Please hyperlink here to the proof within the Appendix. Similarly for other deferred proofs.\n\n- L190 right. Missing subscript command on $\\Gamma\\Psi\\Psi$.\n\n- L191 right. if -> of\n\n- L215 right. shift-invarian -> shift-invariant\n\n- L266 left. Consider replacing *Embeddings’* with *Embedding*\n\n- L380 right. No citation for ImageWoof dataset. (There's no paper for it so usually people cite the imagenette github repo.)\n\n- L711 Need to adjust so the 1 and T in the sub/superscripts above one another don't run together. At the moment they look like one big, combined dagger symbol!\n\n\n**References**\n\nBe careful with casing (often messed up in the bibtex file and has to be manually corrected). Some names need to be title cased: Fréchet, Vendi, ImageNet. Some initialisms need to be changed from lower to upper case: GAN, PCA, COCO, BERT, CLIP. But the worst citation formatting is `Ro{bert}a`, which has the wrong casing and literal curly braces (also for `{bert}`)!", "questions_for_authors": "Why use this generated text data instead of an established dataset?\n\nWhy change from CLIP to OpenCLIP for Table 1?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "baBP6baYMZ", "reviewer": "Reviewer_iKx5", "summary": "This paper proposes a method for comparing the embedding spaces for pairs of models by constructing PSD kernel matrices of each embedding space and studying the differences of these PSD matrices. The theoretical section discusses how, under some assumptions, the eigendecomposition of this difference between the PSD matrices carries the information about which clusters differ between the embeddings. The authors then perform an experimental analysis of their results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer iKx5 for the thoughtful and constructive feedback on our work. Below is our response to the comments and questions in the review: ([Our Numerical results are shown in this link](https://github.com/ICML6204/ICML6204/blob/main/ICML_Rebuttal.pdf))\n\n**1- Quantitative evaluation of the SPEC method**\nWe thank the reviewer for the suggestions. Following the suggestion, we have analyzed the cluster distributions using violin plots to visualize normalized distances between data points. The plots also suggest that one embedding can cluster the points more strongly. Additionally, we performed KMeans clustering on the embedding features and computed normalized mutual information (NMI) between SPEC labels and KMeans clusters. The results indicate that one embedding reaches a stronger alignment and correlation with the KMeans labels.\n\n**2- Referring to the Appendix figures in the main text**\n\nWe thank the reviewer for pointing this out. We will move the referrals to Figures 8, 9, and 12 to the Appendix in the revision.\n\n**3- Experimental Validation of Assumptions: Conditions 1 and 2**\n\nWe analyzed Conditions 1 and 2 in the experiment of Figure 2 in the main text. The numerical results are presented in the Rebuttal Figures 1 and 2.\n\n**4- References in Sections 1 and 3**\n\nWe thank the reviewer for pointing this out. We will cite references related to the spectral clustering, kernel PCA, alignment of embeddings, and explainability of PCA approach to embedding analysis in the revised sections 1 and 3.\n\n**5- SPEC method vs. using classification accuracy for embedding comparison**\n\nWe appreciate the reviewer’s point on using the labeled samples by embedding models to explain their differences. We note that the selection of the label set can significantly influence the result of this approach. On the other hand, the SPEC method performs unsupervised learning to identify the soft-labels (i.e., eigenvectors) for performing the comparison of embeddings. Therefore, the SPEC method can avoid any biases that might be introduced by using a given label set. Also, we highlight that the SPEC method performs a soft clustering, which is different from a hard labeling of the samples assigning each data point to only one label. We will discuss these points in the revised introduction and conclusion.", "claims_and_evidence": "The claims in the paper are supported by evidence, although the evidence could be made stronger. It seems that the author's primary mechanism for verifying their method is via tSNE plots of those points which are in line with the principal eigenvectors of the difference matrix. The idea here is that if these points are in tight clusters in the tSNE plot, then they must have a similarly nice structure in the embedding space. Although I agree with this argument in principle, I have two concerns with it. First, this is a qualitative measure and therefore cannot tell the full story. For example, Figure 1's third tSNE plot is not very clear. Which side is supposed to be better clustered? \nMy second concern is a more theoretic one about relying on tSNE outputs. As far as I know, given a tSNE output, it's not clear what can be confidently said about the corresponding input. Thus, if the tSNE output is clusterable, this does not necessarily guarantee that the input has a similarly nice structure. In short, the tSNE plots are not particularly convincing and, even if they were, I'm not sure what information can be gleamed from them.\n\nSimilarly, I am not sure that Figures 8 and 9 are the best format in which to make the point the authors are making. I agree that if the center of a cluster's nearest neighbors belong to tha same cluster, then the cluster potentially has a nicer structure than one whose centroid has nearest neighbors from other clusters. However, I again have two concerns with this being a definitive statement. First, it could certainly be the case that it is purely by luck that this structure emerges. For example, suppose there is a cluster, $C_1$, which is perfectly separable from the other ones except for two points which land within it from another cluster $C_2$. Then $C_1$ is precisely the kind of cluster the authors are looking for. However, these two points which belong to $C_2$ could be in the center of $C_1$, thereby skewing the author's nearest-neighbors-to-the-center metric. (I also have slight concerns about figures showing results being relegated to the appendix but discussed in the main body of the paper, but it's not a big deal).\n\nRather than the metrics the authors utilized, I would instead be much more interested in metrics which are (a) quantitative and (b) global. For example, one could fit k-means to the clusters found by SPEC and evaluate the cost of the k-means clusters. Additionally, one could visualize the variance of the embedding space by showing violin plots of the distances of points in the clusters to the center. These kinds of plots would unambiguously make the authors' point by showing that SPEC finds clusters which are compact and separable in one embedding space but have poor structure in the other one. As it stands now, this point (which is the premise of the paper) has not been shown unambiguously by the results.", "methods_and_evaluation_criteria": "The datasets and models used in the paper are reasonable for proving the authors' point.", "theoretical_claims": "The theoretical claims are reasonable and follow with what I would expect, having worked in kernel-based ML for a bit of time now. However, I would be curious whether the authors can evidence experimentally that their assumptions (Conditions 1 and 2) are reasonable to expect in practice. Specifically, what are the bounds on $\\varpesilon_1$ and $\\varepsilon_2$ on the datasets the authors used in the paper? What is the value of $\\zeta$ which the authors obtain and is the bound in Theorem 4.1 supported experimentally?", "experimental_designs_or_analyses": "My thoughts on the experimental design and analysis are outlined above.", "relation_to_broader_scientific_literature": "I am not well-acquainted with the broader scientific literature regarding explainability via studying the clusterability of the embedding spaces. However, this work is clearly in line with the themes from kernel-based ML literature. I appreciate the author's reference to Kernel PCA, as they are essentially doing PCA using the difference matrix rather than the kernel matrices themselves. Perhaps some references to explainability via principal components of the embedding distribution would be nice to add.", "essential_references_not_discussed": "All of the essential references seem present. I would, however, note that the way the references are included is slightly strange. There are 0 references on the first page and then a littany of them in the related work section. It would seem appropriate to include some references in Sections 1, 3 and 4 to back up the statements which are made. Here's an example sentence which would do well to have a few references to support it: \"Understanding these differences can aid in interpreting and debugging embeddings and can also be leveraged to align multiple embeddings.  Furthermore, interpreting the discrepancies between embeddings can be utilized to select representation models for downstream applications such as generative model evaluation.\"", "other_strengths_and_weaknesses": "A strength of this paper that I have not focused on enough in the review is its clear description and intuitive idea. I am surprised this has not been done before and it feels like a very natural way to establish the differences between learned representations. As soon as it was described in the intro, I understood the point and it was obvious to me that it should work reasonably well.\n\nOne weakness which I feel needs to be mentioned is that I am not convinced that this actually aids in explainability. Certainly, the method makes it clear which samples one model has clustered better than another model, but couldn't this be evaluated using the classification accuracy on those samples? Why is this method necessary in a way that distinguishes it from other standard explainability measures?", "other_comments_or_suggestions": "No other comments come to mind.", "questions_for_authors": "The questions for the authors can be found scattered throughout the review. Perhaps the most pressing one is in finding quantitative metrics and analysis which support the claims. I would be interested in seeing analyses of the distances from points to the cluster centers across the sets of points identified by SPEC. The absence of quantitative supporting evidence leaves the paper less convincing than it otherwise would be.\n\nThe second question that would be good to address would be how this differs in a measurable way from other explainability measures which compare between models. What does SPEC tell the user which is complementary to, for example, evaluating the classification accuracy of various samples between models? This would again require quantitative evidence to back it up.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IwMF6kFokO", "reviewer": "Reviewer_7vC9", "summary": "The work is proposed to allow an interpretable comparison of feature embeddings from different methods through a Spectral Pairwise Embedding Comparison (SPEC) on clustering and also proposes an approach to align one embedding with another. The work is relevant for a wide audience where interpretability and flexibility with embeddings is desired.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 7vC9 for the thoughtful and constructive feedback on our work. Below is our response to the comments and questions in the review: ([Our Numerical results are shown in this link](https://github.com/ICML6204/ICML6204/blob/main/ICML_Rebuttal.pdf))\n\n**1- API endpoints for the embeddings used in the work**\n\nWe would like to clarify that we have used open-source embeddings in our numerical experiments from their main repositories. Specifically, the embeddings discussed in the paper are in the following repositories, for which we will include the links in the revised paper:\n\nDINOv2 downloaded from: https://huggingface.co/docs/transformers/en/model_doc/dinov2\n\nCLIP downloaded from: https://huggingface.co/docs/transformers/en/model_doc/clip\n\nSWAV downloaded from: https://huggingface.co/lixiangchun/imagenet-swav-resnet50w2\n\nRoBERTa downloaded from: https://huggingface.co/docs/transformers/en/model_doc/roberta\n\nE5 downloaded from: https://huggingface.co/intfloat/e5-base-v2\n\nInception V3 downloaded from: https://huggingface.co/docs/timm/en/models/inception-v3\n\n\n**2- alignment_loss_weight parameter in Section B.4.1**\n\nWe thank the reviewer for pointing this out. The alignment_loss_weight parameter refers to the $\\beta$ hyperparameter in Equation 6, which we have set as $0.1$ (mentioned in the Appendix) in addition to the parameters of the OpenCLIP Github repository. We will make this clear in the revision.\n\n**3- Additional experiments for text embeddings’ alignment**\n\nWe thank the reviewer for the suggestion. To further address the reviewer's comment, we aligned the CLIP text embedding with the T5-XL model. In Figure 6 of the rebuttal link (last page), we can observe that the CLIP kernel has become more similar to T5-XL, and the SPEC-diff is also decreasing.\n\nT5-XL model huggingface: https://huggingface.co/google/t5-v1_1-xl\n\n**4- role of $\\beta$ penalty on the alignment**\n\nIn our experiments, we conducted a grid search for $\\beta$ values ranging from 0.05 to 1. During our experiments, we observed that when $\\beta$ is large ($\\beta$ > 0.5), the consistency between CLIP text and image alignment decreases, and aligning should occur gradually. For our experiment, we ultimately used $\\beta$ = 0.1.", "claims_and_evidence": "The contributions are centered around the proposed SPEC method for comparing two embeddings, an O(n) implementation for the same along with an alignment method to align one embedding with another. Evidence has been provided using methods, experiments and insights which are discussed in other comments.", "methods_and_evaluation_criteria": "- The study uses four widely used benchmark image based datasets, a dataset constructed by overlaying text labels on the ImageNet-1k dataset and text based datasets generated by GPT-4o.\n- Embeddings used for the evaluation are fairly representative of the commonly used embedding methods (CLIP, DINOv2, etc.) \n- The methods are described in detail -- the algorithm has been described for spectral pairwise embedding comparison (SPEC) using the kernel covariance matrices and the loss for aligning embedding maps of one approach with another by adding a penalty term based on the SPEC-diff term that penalizes the mismatch between the with the reference embedding.", "theoretical_claims": "I have not validated the proofs for the theoretical claims. However, the framework definition, formulation and proofs are consistent.", "experimental_designs_or_analyses": "- Experiments for embedding comparisons have been performed in Section 6 (and supplementary material) where the experiment settings have been reported. Figure 1 shows the top-3 clusters based on the approach1 (DINOv2, CLIP, SWAV) in comparison to approach 2 (CLIP, DINOv2, DINOv2) which is validated through the tSNE representation of each embedding individually. The pairwise SPEC embedding approach provides insights like 'CLIP ranking random samples higher as compared to DINOv2' which is qualitatively shown in Figure 8,9. Experiments have also been performed by performing a typography attack to highlight that CLIP clusters are based on overlaid text whereas DINOv2 focuses on overlaid text.\n- The alignment of CLIP with DINOv2 to improve CLIP's performance on MS-COCO 2017 is highlighted in figure 3, 12 where the alignment of Kernel matrices of CLIP to DINOv2 is performed to obtain the SPEC-align CLIP Kernel using the penalty based alignment objective discussed in the methods.", "relation_to_broader_scientific_literature": "Text and Image embeddings are widely used in many applications to perform downstream tasks or to just store objects for retrieval purposes. The interpretability of these embeddings is sought for in multiple areas like healthcare where situations like domain shift or bias in embeddings are undesirable. The pairwise embeddings comparison approach could allow the users to take a well informed decision for choosing the embeddings for the specific application and embedding-alignment approach could be beneficial in adapting embeddings for obtaining specific outcomes. This work is (probably) a first step towards providing insights on the commonly used embedding methods through experiments on the selected datasets.", "essential_references_not_discussed": "I am not aware of any essential references that have not been discussed.", "other_strengths_and_weaknesses": "The paper is well-written and readable. References have been provided to point to the existing literature where needed. The framework of embeddings comparison and alignment is defined well and has been theoretically and experimentally validated extensively.", "other_comments_or_suggestions": "- It would be good to provide the API endpoints for the embeddings used in this work. The methods have been cited but the endpoints would help in reproducibility of the approach.\n- As there have been several insights provided by comparing embedding methods like CLIP and DINOv2 by highlighting cases where the embeddings have some limitations. It would be good to provide some general guidelines about how to use the SPEC approach for the broader audience which may be interesting in comparing their embeddings on general tasks. For instance, the insight provided in Subsection \"SPEC comparison of embeddings on different image and text datasets.\" regarding the RoBERTa clustering based on gender and profession could have fairness implications for the wider audience. It would be good to provide a detailed implementation for further validation of the approach in other domains (and datasets) and ensure reproducibility of the results.\n- In Section B.4.1, does the 'alignment_loss_weight' parameter refer to the β term in Eqn: 6? It may be good to clarify the difference between 'clip_contrastive_alignment_loss_weight' and 'alignment_loss_weight'. I can see that the 'coca_contrastive_loss_weight' is mentioned in the OpenCLIP github repository but the alignment terms may need some clarification.\n- In the last line of B.1,  'was' can be removed from '..observe that E5 was managed to cluster captions...'\n- For the alignment approach, there could be some explanation on the benefits of alignments other than CLIP->DINOv2. For instance, it would be good to see some experiments on text embedding alignment and/or to see the behavior of embeddings upon performing other types of alignment. It would also be good to discuss how the role of β penalty on the alignment.", "questions_for_authors": "I have mentioned my comments in other sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cL8hIWmnOu", "reviewer": "Reviewer_KUAe", "summary": "The authors propose the Spectral Pairwise Embedding Comparison (SPEC) framework for comparing feature embeddings in an explainable manner. The goal is to identify differences in how two embeddings cluster data points, rather than relying solely on downstream performance metrics. The main contributions are: 1) a spectral analysis approach leveraging eigendecomposition of differential kernel matrices to detect mismatches in clustering behavior, 2) a scalable implementation that reduces computational complexity, 3) the SPEC-align method for aligning embeddings by minimizing clustering differences, and 4) numerical results demonstrating the effectiveness of SPEC on benchmark datasets such as ImageNet and MS-COCO. The study shows that SPEC can reveal embedding discrepancies and improve cross-modality alignment, such as enhancing CLIP embeddings with single-modality features.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer KUAe for the thoughtful and constructive feedback on our work. Below is our response to the comments and questions in the review: ([Our Numerical results are shown in this link](https://github.com/ICML6204/ICML6204/blob/main/ICML_Rebuttal.pdf))\n\n**1- Comparison with existing embedding alignment techniques**\n\nA distinction of our proposed alignment approach is that SPEC-Align operates directly on kernel matrices, enabling cluster-level comparison between encoders without requiring pointwise embedding alignment. The standard Procrustes and Wasserstein distance-based alignments usually enforce alignment in the sample level, which is a stronger alignment requirement compared to SPEC-align. This cluster-centric perspective is particularly advantageous when the embeddings have different topologies that should not be forcefully aligned or when only cluster-assignment consistency (rather than pointwise matching) is needed for the downstream task. We will include this discussion in the revised text.\n\n**2- Difference kernel matrix and differential kernel matrix**\n\nWe thank the reviewer for pointing this out. We will update the paper and consistently use the term “difference kernel matrix” to avoid any confusion.\n\n**3- Non-spectral clustering methods such as hierarchical clustering or DBSCAN**\n\nWhile the current SPEC framework follows spectral clustering for kernel-based embedding comparison, extending non-spectral clustering methods such as hierarchical clustering or DBSCAN [1] to perform cluster-based embedding comparison will be an interesting direction for future exploration. We will discuss this future direction in the conclusion section.\n\n[1] Ester et al., \"A density-based algorithm for discovering clusters in large spatial databases with noise\" KDD 1996\n\n**4- SPEC-align computational complexity and scalability in sample size**\n\nFollowing Proposition 5.1, we showed that SPEC-align can be computed using the power method on a matrix with size $(d_1+d_2)\\times (d_1+d_2)$, and the gradient calculation can be run with only $O(\\max\\lbrace n_B, (d_1+d_2)^2\\rbrace)$ computations ($O((d_1+d_2)^2)$ is the cost of one run of the power method), considering a batch size $n_B$ and embedding dimensions $d_1$ and $d_2$. We will explain the complexity of SPEC-align in the revised text.", "claims_and_evidence": "The authors claim that the SPEC framework enables explainable comparisons of feature embeddings by identifying clustering differences between them. They support this claim through theoretical derivations using spectral analysis. They show that the eigendecomposition of the differential kernel matrix highlights mismatches in clustering behavior. The paper also introduces SPEC-align, which aligns embeddings by minimizing these differences.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited for the problem of explainable embedding comparison. Furthermore, the evaluation is robust. The authors used well-established benchmark datasets such as ImageNet and MS-COCO. The scalability of SPEC is validated through efficient computation techniques, making it feasible for large datasets.", "theoretical_claims": "The theoretical claims presented in the paper appear well-founded. Specifically, the authors provide rigorous proofs demonstrating that the eigendecomposition of the differential kernel matrix effectively identifies clustering differences between embeddings. Moreover, they establish the scalability of their method by proving that the computational complexity grows linearly with the sample size.", "experimental_designs_or_analyses": "1. The authors evaluated the SPEC framework using well-known benchmark datasets, including ImageNet and MS-COCO.\n2. The authors compared different feature embeddings using the eigendecomposition of differential kernel matrices to identify differences in clustering behavior.\n3. The scalability of the approach is validated by implementing an optimized computation strategy for large datasets.", "relation_to_broader_scientific_literature": "The SPEC framework proposed by the authors leverages the eigendecomposition of differential kernel matrices to compare embeddings. This aligns with spectral clustering and diffusion maps, which are widely used in manifold learning within the existing scientific literature.", "essential_references_not_discussed": "Most of the essential references were cited in the paper.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper presents a novel spectral framework (SPEC) for comparing embeddings.\n2. The SPEC-align method provides a structured way to align embeddings, resulting in improved cross-modality performance.\n3. The scalability of the approach is well-addressed. This makes it feasible for large datasets.\n4. The authors demonstrated the effectiveness of SPEC in detecting clustering differences across various embeddings and experiments.\n\nWeaknesses:\n\nThe paper could benefit from direct comparison with existing embedding alignment techniques.", "other_comments_or_suggestions": "1. Overall the paper is well-written.\n2.  \"eigendecomposin\" → \"eigendecomposition\"\n3. consistently use \"difference kernel matrix\" or \"differential kernel matrix\" to avoid confusion. Can you check this?", "questions_for_authors": "1. The paper uses both \"difference kernel matrix\" and \"differential kernel matrix.\" Are these terms interchangeable, or do they refer to distinct concepts?\n2. The SPEC framework focuses on spectral analysis for comparing embeddings. How would it perform against non-spectral clustering methods such as hierarchical clustering or DBSCAN?\n3. The paper discusses the computational efficiency of SPEC, but how does SPEC-align scale in high-dimensional spaces with very large datasets? \n4. The paper introduces SPEC-align for embedding alignment, but how does it compare to existing alignment techniques such as Procrustes analysis or Wasserstein distance-based alignment?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a spectral framework to compare and contrast feature embeddings for interpretable ML.  The methodology is novel and has wide potential applications.  The general concensus of the reviewers is that this submission adds to the growing and important literature on interpretable machine learning.
Decision: Accept (poster)