Intrinsic Dimension Correlation: uncovering nonlinear connections in multimodal representations
Authors: Lorenzo Basile
Santiago Acevedo
Luca Bortolussi
Fabio Anselmi
Alex Rodriguez
Venue: ICLR 2025 Poster (2025)
Keywords: intrinsic dimension
nonlinear correlation
multimodal representations
representation similarity
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a novel nonlinear correlation method, based on intrinsic dimension estimation, that is able to capture correlations between paired multimodal representations.

Abstract: To gain insight into the mechanisms behind machine learning methods, it is crucial to establish connections among the features describing data points. However, these correlations often exhibit a high-dimensional and strongly nonlinear nature, which makes them challenging to detect using standard methods. This paper exploits the entanglement between intrinsic dimensionality and correlation to propose a metric that quantifies the (potentially nonlinear) correlation between high-dimensional manifolds. We first validate our method on synthetic data in controlled environments, showcasing its advantages and drawbacks compared to existing techniques. Subsequently, we extend our analysis to large-scale applications in neural network representations. Specifically, we focus on latent representations of multimodal data, uncovering clear correlations between paired visual and textual embeddings, whereas existing methods struggle significantly in detecting similarity. Our results indicate the presence of highly nonlinear correlation patterns between latent manifolds.

Published as a conference paper at ICLR 2025
INTRINSIC DIMENSION CORRELATION: UNCOVERING NONLINEAR CONNECTIONS IN MULTIMODAL REPRESENTATIONS
Lorenzo Basile 1.2
Santiago Acevedo 3 Alex Rodriguez
Luca Bortolussi 1
Fabio Anselmi 1,1
Uniiersiity of Trieste
2 AREA Science Park
SISSA
*MIT
$ICTP
lorenzo. waammo.basiiie basile@phd. units.it
ABSTRACT
To gain insight into the mechanisms behind machine learning methods, it cru- cial to establish connections among the features describing data points. However, these correlations often exhibit high-dimensional and strongly nonlinear nature, which makes them challenging to detect using standard methods. This paper exploits the entanglement between intrinsic dimensionality and correlation to propose metric that quantifies the (potentially nonlinear) correlation between high-dimensional manifolds. We first validate our method on synthetic data controlled environments, showcasing its advantages and drawbacks compared to existing techniques. Subsequently, we extend our analysis to large-scale applications in neural network representations. Specifically, focus on latent representations of multimodal data, uncovering clear correlations between paired visual and textual embeddings, whereas existing methods struggle significantly in detecting similarity. Our results indicate the presence of highly nonlinear correlation patterns between latent manifolds.
INTRODUCTION
Modern machine learning models have the remarkable ability to extract subtle patterns from complex datasets and use them to perform a wide variety of tasks in an astonishingly accurate way. However, to date, we still lack complete and accurate understanding of their inner workings, especially the case of deep neural networks. An active field of research in the interpretability of neural networks focused on characterizing and quantifying the similarity between different models. To this aim, many works (Raghu al. 2017 Kornblith et al. 2019; Nguyen et al., (021) evaluate the statistical correlation between the latent representations produced by the models. This quantification is key because it allows, for example, disentangle or tie together different aspects of data representations, allowing a better interpretation of how the model makes its decisions. Moreover, assessing the similarity between representations of different models is particularly useful to determine whether the latent spaces are compatible, meaning that information extracted by one model can be successfully transferred others.
This point is crucial particularly when data points are represented by multiple interrelated modal- ities, such as visual and textual. In recent times, it has been shown (Norelli et al., 2023; Moayeri et al. 2023) that it is possible to build multimodal vision-language models starting from pranted unimodal encoders,
an effort to match the outstanding performance of state-of-the-an vision- language models such as CLIP (Radford et al. 2021). These findings indicate that modern deep models can produce compatible representations when evaluated on aligned text-image data, hinting strong functional similarity between those. However, we find that standard latent similarity met- rics such as CKA (Kornblith et al. (2019) and Distance Correlation (Székely et al. 2007 Zhen et al.) 2022) find very low structural correlation between paired multimodal representations. We hypoth- esize that this is due to the strongly nonlinear nature these correlations, making them difficult to analyze with standard methods.
Our work tackles this challenge by introducing a novel metric, dubbed Intrinsic Dimension Corre- lation (orCor) that leverages the concept of intrinsic dimension (Ta), i.e. the minimum number of

---

Published as a conference paper at ICLR 2025
Full dataset
xy projection, 17 = 2
ld =
correlation(xy.c 2)
1fCor =
X xZ projection, 1 =
2
y lfCor 0
X
correlation(x,,))
Figure 1: Example usage of ToCo:: consider 3D dataset in which the points lie the surface of a cylinder, hence whose intrinsic dimension (7,) is We want to assess the correlation between the 2D set of coordinates Ty (which also has set z. Intuitively, as æ and describe a circle, it is evident that knowledge of z (encoded by color) very informative in determining the x coordinate (e.g., yellow point sure be found in the central region of the axis), but not in determining the hence, the correlation coefficient is 0.5, according to equation 3 Conversely, when estimating the correlation between zz (whose Ia is 1, bottom-right panel) and we can see that having access (which is now represented by color) does not give any information on the value nor 2, hence the correlation
2, shown the top-right panel) and the ID
variables required to describe the data, to quantify the mutual information between high- dimensional data manifolds. Intuitively, the metric is based the concept that if two data representations correlated, the intrinsic dimension of a dataset created by concatenating the features of these repre- sentations is reduced, because the information in representation other. An example is provided in Fig. [ It is worth noting that, to compute the correlation between two representations, a simpler approach based the difference between the embedding dimension and the intrinsic dimension of the concatenated dataset would not suffice, since this would measure the amount of correlated features, regardless of whether these correlations are intra or inter repre- sentation. Computational experiments indicate that our method is effective at detecting nonlinear relationships where traditional methods often fail.
describe some aspects of the
Thus, the main contributions of this work can be summarized follows:
We propose novel perspective that links the concepts of correlation and intrinsic dimen- sion. The key idea that, since the La serves as proxy for the information content of dataset, the Ia of dataset created by merging two datasets will represent their joint information.
Building on this idea, we propose LoCor, a novel correlation metric based on intrinsic di- mension estimation, able to unveil nonlinear correlation between high-dimensional data manifolds, even of unpaired dimensions. We evaluate our metric on synthetic data, show- casing its strengths and limitations in comparison with existing methods.
We then consider more complex scenarios and quantify the correlation between the repre- sentations learned by different deep neural networks engaged in large datasets. Specifically, we focus on multimodal data, demonstrating that we can find strong evidence correlation where standard methods struggle identify any.
2

---

Published as a conference paper at ICLR 2025
2 BACKGROUND
2.1 CORRELATION IN LATENT REPRESENTATIONS
Understanding whether different neural networks can learn process data in similar ways is crucial point when trying to make sense of their results. Despite the inherent difficulty in defining what it means for two neural networks to be similar (or, at least, to behave similarly), research in this field has made significant progress in the last few years. A well-established approach to test if two neural networks are similar is that of measuring the statistical correlation between the data representations (or embeddings) learned by them. Recent proposals in this direction include Singular Value Canonical Correlation Analysis (SVCCA) (Raghu al. 2017). Projection Weighted CCA (PWCCA) (Morcos et al. 2018)) Centered Kernel Alignment (CKA) (Kormblith et al., 2011), Distance Correlation (dCor) (Székely et al., 2007] (Zhen et al., (022). Aligned Cosine Similarity
(Hamilton et al. (016) and Representation Topology Divergence (RTD) (Barannikov et al.) 2022),
among others. For a more complete summary of current approaches to neural network similarity
measurement, we defer the reader to Klabunde al. (2023).
These techniques have been widely employed to gain a deeper understanding of various aspects of the way neural models process information: for instance to quantify how different vision architec- tures encode spatial information (Raghu et al. [021] Nguyen et al.) (221]) or the relation between
learning disentangled features and adversarial robustness (Zhen et al.] 2022). Taking slightly dif-
ferent approach, Davari et al.] (2023) provides an in-depth analysis of the sensitivity of CKA to
transformations that occur frequently in neural latent spaces, showcasing the importance of gather- ing results from a broader range of similarity metrics to obtain reliable information.
2.2 MULTIMODAL LATENT SPACE ALIGNMENT
An example that highlights the importance of assessing similarity, quantified via correlation mea- sures, between neural representations is further demonstrated by the recent empirical findings related to the so-called latent communication. This concept, introduced by |Moschella et al. (2023). builds on the idea that it is possible to transfer knowledge between latent spaces, even when they are pro- duced by different models and on different data modalities, provided that some semantic alignment between the data exists (for example, images and their textual descriptions). The feasibility of this knowledge transfer was shown in Moschella et al. (2023) through the introduction of relative repre- sentations, where each point of the original representation is mapped according to its distance from a set of fixed anchor points. Using this alternative representation of data, the authors show that is possible to stitch (Lene& & Vedaldi 2015) together encoders and decoders coming from different models, with little to no additional training
Furthermore, numerous recent studies have demonstrated that large state-of-the-art visual and tex- tual encoders can produce transferable representations when evaluated on aligned data (i.e., the same data or data that share some semantics, such as mmage-caption pairs). Indeed, simple linear trans- formation is usually enough to map one latent space into another (Moayeri et al. 2023 Merullo al. 2023] Maiorca et al., 2023 Lähner & Moeller 2024), at least in terms of performance specific downstream task, c.g., classification. It is worth noting that, to perform the aligmment of the data, one assumes prior knowledge about the semantic correlation between the data representations (in order to define the anchor points). Hence, while these findings suggest a remarkable similarity between compatible latent spaces, the problem of detecting these correlations without relying on any downstream evaluation is still an open problem. Indeed, our investigation into the connection between aligned textual and visual embeddings reveals a very weak correlation using existing meth- ods, calling for the development of methods that allow the identification of nonlinear correlations in high-ding gigh-dimessional spaces.
2.3 INTRINSIC DIMENSION
The concept of the intrinsic dimension (La) of a dataset widely used in data analysis and Machine Learning. Before providing a more formal definition, imagine a dataset where your data points the cities around the globe described by their 3D Cartesian coordinates. We will say that the embedding dimension of this dataset is three. However, anyone familiar with cartography would agree that nearly the same information can be encoded with only two coordinates (latitude and
3

---

Published as a conference paper at ICLR 2025
longitude). Therefore, its La would be equal two. Indeed, one of the definitions of Ia is the minimum number of coordinates needed to represent the data with minimal information loss. A complementary definition is the dimension of the manifold in which the data lic, which in this case would be a sphere.
A possible way of estimating the Ia is to find meaningful projection (i.e., with minimal information loss) into the lowest dimensional space possible. A classical method for doing that is Principal Com- ponent Analysis (Wold et al.) 1987). but has the drawback that the intrinsic dimension it estimates is only correct if the manifold in which the data lie is hyperplane. Therefore, the development of methods that can estimate the Ia in nonlinear manifolds an active research field. Typically, these approaches infer the La from the properties of distances to the Nearest Neighbors. While full review of these methods is out of the scope of this work (the interested reader referred to Cam- padelli et al. (2015)). it is worth mentioning the Maximum Likelihood estimator (MLE) (Levina & Bickel| 2004), the Dimensionality from Angle and Norm Concentration (DANCo) approach (Ceruti et al. 2014) the TwoNN (Facco al. 2017). The last is the one employed in this work since is particularly fast and behaves well even in the case of datasets with a high non-uniformity on the density of points. A brief description of TwoNN is provided in Appendix A.
More recently, several studies have estimated the intrinsic dimension of neural representations, demonstrating that Ia is valuable tool for understanding the geometry of the latent manifolds produced by deep models. This concept was initially explored in Ansuini et al. (2019), where the authors estimated Ia across different layers of CNNs, gaining insights into the sequential informa- tion flow within these models. Later, Valeriani et al. (2023) and Cheng et al. (0023) analyzed the representations of transformer models, across different domains, while Kvinge et al. (2023) stud- ied the internal Ia of generative diffusion models. In slightly different direction, Brown et al.) (022) unveiled a connection between generalization and the Ia of the hidden representations, while Kaufman & Azencot (2023) studied the relation between Ia and curvature in latent manifolds.
CORRELATION THROUGH INTRINSIC DIMENSION
The intrinsic dimension of dataset is closely linked to the correlations among the various fea- tures that define the data points. These correlations determine the regions in which the data points can exist, thereby shaping the underlying manifold. Let us consider the simplest example: a two- dimensional dataset. If the two variables are uncorrelated, their linear correlation coefficient (R²) approaches zero while, if one feature a linear function the other, It2 becomes equal to one. The two scenarios differ by the La of the data manifold: the first case corresponds to plane (Ia 2), while the second corresponds to a line (la = 1). we examine a slightly more complex case, the advantage of exploiting the La for correlation becomes evident. Let us consider spiral-shaped dataset embedded in two dimensions: it has R² se due to the nonlinear nature of the correlation be- tween the two variables, while the behavior of the Ia is identical to the one observed the linearly correlated dataset, as reported in detail in section 4.1
A formal framework that connects correlation and intrinsic dimension comes from information the- ory [Romano et al.) 2016). In this field, mutual information is the fundamental metric that quantifies the relationship between simultaneously sampled random variables. Mutual information makes natural candidate to serve as a measure of correlation between data, and Horibe (1985) defined a correlation coefficient between two random variables X and Y (known as Normalized Mutual In- formation NMI) as:
1(X,Y) Y)
(X H(Y))
where 1(-,-) is the mutual information and (-) is the entropy.
In high dimensional datasets, computing the mutual information is challenging due to the curse of dimensionality, so we introduce the intrinsic dimension as a proxy. Mutual information can be expressed in terms of entropies as:
I(X,) = H(X) + H(Y) - H(XYY)
(2)
Recent research (Bailey et al. 2022 Ghosh & Motani 2023) has shown that intrinsic dimension is a
metric that can satisfy the most desirable properties of entropy, while being easy and fast compute
4

---

Published as a conference paper at ICLR 2025
provide test where this relationship is exact). We exploit this relation to define our correlation coefficient LgCor, based on the NMI formulation, but using intrinsic dimension as replacement for entropy. Given two standardized datasets (of possibly different dimensionality) X E RR,di and Y E IRm,da, we define their LaCor as:
on large-scale data (in Appendix A.2
(CCo(XXYY) Y) (Ia(X) + Ia() - Ia(X @Y) max (Ta((),
where @ denotes row-wise concatenation. Although various other versions of NMI have been pro- posed over time (for complete survey on variations of NMI defer the reader to Vinh et al. (2009) ) we adopt the max normalization since it allows an interpretation as the fraction of features of the most informative dataset that can be predicted using the less informative one (see Fig. f an intuitive explanation of this concept).
In practical applications, intrinsic dimension calculated using estimators, which can be prone to errors. To mitigate this, we assign a p-value to the observed correlation, employing permutation test (Davison Hinkley 1997 on la(X @ Y). Specifically, we estimate the La of several indepen- dent samples of the joint dataset, created by concatenating the two original datasets and randomizing the pairings to disrupt any existing correlations. This process allows us to determine a p-value that represents the probability of the hypothesis of the two datasets being uncorrelated as p 3+1 where T. is the number of estimates lower than the original joint la and S is the total number of permuted samples considered.
L+1
RESULTS
In this section, we begin by assessing our proposed LçCor measure in simple, controlled settings using synthetic data. Then, we move to applications latent representations generated by different neural networks on various datasets. We begin with a straightforward example that underscores the difficulty faced by standard methods in identifying nonlinear relationships, then progress more extensive applications involving visual representations and multimodal text-image representations. For some experiments, we do not report the correlation p-value returned by our method. In such cases, the p-value is always the lowest possible according to the permutation test outlined in section (5FI. where S' the number of permutations, 100 most of our experiments).
4.1 SYNTHETIC EXPERIMENTS
As a first step, we produce three toy datasets, displayed in the Appendix in Fig. Such datasets are made of 5000 observations of two variables that are either linearly correlated, uncorrelated, or nonlinearly correlated. The first setting simply obtained by arranging i and y on a straight line, in the second case both random variables sampled independently from a normal Gaussian distribution, while the last dataset contains data arranged on a spiral curve.
We report correlation results in Table T aggregating results over 10 independent random sam- plings of the datasets. In the simpler cases (linear and random data) our method agrees with linear correlation and distance correlation, correctly identifying a very strong correlation in linear data and the lack thereof in random data. The spiral dataset constitutes a more tricky testbed: while there is clear correlation between the two variables, it a highly nonlinear one, and the linear correlation coefficient is around 0. Even Distance Correlation, despite being a nonlinear method, fails to find any strong signal of correlation, returning value very close to Instead, our method correctly identifies the strong dependency between the two variables, with mean correlation coefficient of 0.98, determined with high confidence, as witnessed by the p-value consistently equal to 0.01. We note here that our method relies on an intrinsic dimension estimator (in this paper, we mainly em- play TwoNN (Facco et al. 2017). but we also provide results on synthetic data using MLE (Levina) & Bickel 2004) in section A.3. in the Appendix), and it inherits substantial properties from it. On the negative side, Ia estimators are not oracles, and they can return values that slightly differ from what one would expect (c.g., Ia lower than 1 in linear dataset higher than 2 in the random case), or even totally fail (when the In becomes large enough, the estimator is also affected by the curse of dimensionality). Conversely, the choice of employing TwoNN makes method extremely
5

---

Published as a conference paper at ICLR 2025
efficient (the correlation coefficient can be obtained with just 3 calls to the estimator), allowing it to scale easily to large and high- dimensional datasets.
Table 1: Correlation in two-variable datasets. (R²): linear correlation coefficient; (dCor): distance correlation coefficient; (la(); intrinsic dimension of the concatenated (2D) dataset; (IaCor): correlation coefficient (g) returned by our method; (p-value): significance of the correlation detected by our method (100 shuffles). Results reported mean + std over 10 independent random samplings of data, with the exclusion of p- p-luue, which is reported as a range.
Data
R
dCor
La@
Ia Cor
p-value
Linear
1.00 + 0.00
1.00 0.00
0.99 + 0.02
1.00 + 0.00
0.01
Random
0.00 + 0.00
0.00 + 0.00
2.02 + 0.04
-0.02 + 0.04
[0.03, 0.99]
Spiral
0.01 0.00
0.02 + 0.00
1.01 + 0.02
0.98 0.02
0.01
Then, we switch to a higher-dimennional setting, in which we test our method on correlating pairs of synthetic datasets of 4 variables. We construct first random dataset containing 4 independently sampled variables, and then consider correlation scenarios: in the first, we compute its correlation with another likewise randomly sampled dataset, hence we expect no correlation; in the second, we randomly sample 2 variables of the second dataset, while binding the other 2 to the original dataset through trigonometric functions, establishing partial correlation between the two; finally, in the third scenario we set all variables of the second dataset to be trigonometric functions of the original variables, thus making the datasets completely correlated, although in strongly nonlinear fashion. We compare our [oCor against Distance Correlation and Canonical Correlation Analysis (CCA) (Hotelling 1936), which the direct analogous of R² in multivariate correlation, and serves as a linear baseline. As we report in Table while all methods agree in correctly finding no corre- lation in the first scenario, both CCA and dCor fail in the partially and completely correlated cases, while Coro returns substantially higher correlation coefficients, showcasing its robustness to non- linear correlations. In the Appendix (sections AA..2 and A.3.3), we report additional results in this setting, obtained applying CoCor to differently sampled and noisy data.
Table Correlation between high dimensional synthetic datasets various scenarios. We consider a first dataset with 5000 observations of 4 variables x, and z, all sampled independently from a Gaussian distribution with mean 0 and standard deviation т, and assess correlations in different scenarios. In scenario A absence of correlation we compute the correlation between such dataset and another dataset built by independently sampling corresponding w', x', y and z' from the same distribution. In scenario B partial correlation, we randomly sample y and 2', while we set w/ = cos(us w + y) and s' sin(r). Finally, in scenario C- - complete correlation, we set w' cos(w + y), x' sin(x), s/ sin(xz) z' cos(y). Results are reported mean + std over 10 independent random samplings of data, with the exclusion of p-value, reported as a range.
Correlation
CCA
dCor
La
CoCor
p-value
(A) Absent
0.02 = 0.01
0.00 : 0.00
8.07
0.17
0.01 = 0.05
[0.25, 1.00]
(B) Partial
0.02 + 0.01
0.00 0.00
6.41
+ 0.10
0.35 - 0.03
0.01
(C) Complete
0.02 + 0.00
0.01 + 0.00
4.88 0.06
0.67 + 0.02
0.01
4.2 A MOTIVATING EXAMPLE ON NEURAL REPRESENTATIONS
Across different layers, neural networks encode information in complex, high-dimensional repre- sentations that differ significantly from the simpler datasets discussed earlier in this manuseript. In particular, deep models are structured to learn nonlinear functions of the input data, typically through the use of nonlinear activation functions like ReLU. This suggests that the representations produced by different networks on the same data can be correlated in complex, nonlinear ways. Consequently, methods used to detect such correlations need to be capable of capturing this degree of nonlinearity.
To illustrate this phenomenon, we showcase a simple example: we consider a randomly initialized multilayer perceptron (MIP), made of 15 fully connected layers of 784 neurons, followed by

---

Published as a conference paper at ICLR 2025
10-
0.8
Correlation
0.6
0.5
dCor
(lneer)
CKA (RBF)
0.3
SVCCA
0.8
06
04
LeakyReLU slope
Noniinarity
Figure 2: Average correlation results with different methods between MNIST data and their final representations computed by randomly initialized MLP, with variable degree of activation linearity, increasing on the x axis. Shaded area represents standard deviation over 10 runs with independent random initialization of MLP weights.
LeakyReLU activation. LeakyReLU a parametric activation function, whose behavior depends parameter called slope: if the slope is 1, it behaves like the identity function, rendering MLP a linear function of the input, while lower slope values make the network nonlinear, with corresponding to the standard ReLU. We feed MLP with the MNIST (LeCun et al. 1998) dataset at increasing degrees of nonlinearity (which corresponds to decreasing the slope) and compute the correlation between the representation the final layer and the input data, both with method and with established baselines (SVCCA, Distance Correlation, linear kernel CKA, and RBF kernel CKA).
As we report in Fig. 2. our method is weakly affected by the increasing nonlinearity in the corre- lation, as it consistently returns correlation coefficients above 0.75. Existing baselines capture high correlation in linear or lineir cases (high slope), but the signal tends to degrade quickly as the slope decreases. This especially true for SVCCA, as it is linear method, but even nonlinear alter- natives see the initial correlation fade when the activation becomes ReLU, reaching values below 0.5.
Our method estimates a proxy of the normalized mutual information between representations, being therefore largely insensitive to nonlinear transformations of the data (as opposed to other metrics like CKA or dCor) and making optimal to detect statistical dependencies between representa- tions. However, this insensitivity makes it a less informative geometrical similarity index, since the two representations will have a high mutual information regardless of the nature of the nonlinear transformation.
4.3 IMAGENET REPRESENTATIONS
Moving to more realistic setting, we test our method on measuring similarity between ImageNet (Russakovsky et al. 2015) embeddings coming from different neural encoders. We consider vari- cty pre-trained architectures, including CNNs (ResNet-18 (He et al. 2016) and EfficientNet-B0
Tan & Le 2019)), ) four variants of Vision Transformers (Dosovitskiy al. 2020) (including a ViT- CNN hybrid) and three ViT-based self supervieee vision-language models (CLIP.ViT-E (Radford)
al. 2021). SigLIP-ViTI VIT-B (Zhai et al. 2023) and ELIP-ViT- (Li al. 2022)). For all the models, we consider the output the last hidden representation produced by the encoder, before the clas- sification head: in supervised VITs this choice corresponds to the last class token, while in CNNs

---

Published as a conference paper at ICLR 2025
to the pooler output. For CLIP, SigLIP and BLIP, consider instead the image representation in the shared vision-language space. Since some of the correlation methods we employ as baselines
dCor
loCor
1.0
EfficienNNat
8:42
0:42
0.2
EfficerNNat
6:22
5:34
2:83
0:22
0.5
0.77
0.78
SIUUP
0.34
5:49
8:14
2:51
3.55
3.53
9.07
0.63
SUUP
6:01
6:41
8.
3:41
8.
8.81
81
0.40
WIT--32--86-
0.42
0.40
0:55
0.57
2.42
VIT---2--044
0.02
0:92
0:02
5.00
0.05
9.92
ViFeybrid
9.4
6:54
5:57
57
5:43
0.5
0.51
0.64
VITaybrdd
6:91
0:51
46
A6
A.
0.84
84
0707
VIT
04
ResVet
6:42
2:43
8:43
9.47
0:00
Redtet
8:87
8:05
0:07
79
02
CUP-
0.29
5:46
9.46
6.32
1.31
1.32
CLIP
0:77
0:31
93
1-92
9:58
9.51
9.79
9.55
Diiceentee
VITa 228-102244
364
VINL16-22
Restee
CLPP
Effiientee
SgUL
Viaa 18--48-244
viie 2r 384
Viaapand
174-162224
Restet
BLP
Figure 3: Correlation results on ImageNet representations, obtained using: left Distance Correlation (dCor); right LoCor (ours). Both methods are able to detect non-negligible correlation. More base- line results are reported in the Appendix.
(namely, CKA) are particularly expensive in terms of memory requirements, for all our experiments randomly sample subset of 30000 data points.
We report the correlation results produced by our method in Fig. 3 along with the correlation scores returned by Distance Correlation (dCor), as it is the baseline method that most closely matches ours in terms mean off-diagonal correlation (LaCor mean: 0.87, dCor mean: 0.50). Detailed results for SVCCA (mean: 0.46), linear kernel CKA (0.43) and RBF kernel CKA (0.44) are reported in the Appendix in Fig. 8 In this setting, all models are computing embeddings for the same data points, hence we would expect significant correlation to be present for any given pair of models. Indeed, all methods are clearly capturing such correlation (even if with higher variance than I,Cor), including SVCCA, which suggests that relevant component of this correlation is actually linear.
4.3.1 COARSE ALIGNMENT
The previous section demonstrated that method effectively identifies strong correlations within perfectly aligned datasets of representations. We now aim to explore how the performance of the method might vary when applied coarsely aligned data. To test this, utilize the inherent class information of ImageNet data. Specifically, we randomly shuffle the embeddings generated by a model while keeping the labels unchanged, and then compare this modified dataset with the original dataset prior to shuffling. In other words, given a point index of class C. we pair it with another randomly chosen point of class C₁ with the condition that C. Cy.
This allows us to assess the robustness of our correlation estimation in less ideal conditions: we ex- pect the correlation signal to decrease. as alignment is crucial property for all correlation methods to detect similarities. However, as we report in Table 3 we are still able to identify correlations with high confidence, as demonstrated by the low p-values, The values of the correlation suggest that the number of features needed to perform the classification task is between 50 and 75%. On average, C,Cor returns a correlation of 0.62, while baselines sit the range 0.31 - 0.47. Interestingly, we observe that the models that exhibit lower correlation are CLIP, BLIP and SigLIP, all contrastive vision- Janoull models. In two the three cases, IςCor is even surpassed by Distance Correlation and CKA. While we have clear explanation for such behavior, these results align with very recent results by Ciernik et al. (2024). highlighting the impact of the pre-training objective on representa- tion similarity. For reference, also report the Cor results obtained when of the two datasets
8

---

Published as a conference paper at ICLR 2025
is freely shuffled (irrespective of class labels). As witnessed by the high p-values, LaCor correctly reports a lack of correlation in such case.
Table 3: Correlation between ImageNet representations when exact alignment is broken. Results shown, in terms of p-value and correlation coefficient, for the coarse alignment case, in which data shuffled while preserving the labeling. IaCor significantly outperforms all baseline methods, with the exception of two models. The last two columns report the LaCor coefficient and p-value for the fully shuffled case, in which no constraint is enforced on labels, and correlation is not present, as identified by high p-values. All results are reported as means (excluding p-values, reported as ranges) over independent shufflings. Standard deviations are not reported as they are lower than 0.02 in all cases.
Model
LgCor
p-val.
SVCCA
CKA (lin.)
CKA (RBF)
dCor
IaCor (rand)
p-val. (rand)
RfficienNNet
0.61
0.01
0.25
0.17
0.19
0.25
-0.10
0.997
SigLIP
0.53
0.01
0.26
0.58
0.59
0.59
-0.01
0.89
ViI-B-16
0.27, 0.98
VTT 32
0.68
0.01
0.40
0140
0.40
0.47
0.22
0.25.0.99
0.63
0.01
0.51
0.55
0.09,
WITL
0.73
0.01
0.40
0.45
0.43
0.55
0.37
0.18,
ResNet
0.66
0.01
0.22
0.34
0.35
0.37
0.39
0.16, 0.91
CLIP
0.53
0.01
0.59
0.61
0.62
0.21
0.08, 0.85
BLIP
0.50
0.01
0.28
0.37
0.39
0.39
-0.10
00228, 0.95
Average
0.62
0.01
0.31
0.42
0.42
0.47
0.17
[0.08,0.10 0.99
4.4 MULTIMODAL REPRESENTATIONS
We now shift our focus to a multimodal context, where we examine the similarities between hidden spaces learned by text and image encoders. We use three datasets, N24News (Wang et al.) 2022), MS-COCO 2014 (Lin et al.] 2014) and Flickr30k (Young et al.] 2014), all consisting of image- caption pairs. This analysis will help us understand how textual and visual representations correlate when evaluated on related multimodal content. Images are encoded using representative subset of the vision models introduced in section 4.3 two CNNs (EfficientNet-BO and ResNet-188,, two ViTs (ViT-B-16 and ViT-hybrid), and the visual branch of CLIP. For text we employ five architectures, all taken pre-trained: BERT (Devlin al. 2019), both cased and uncased, ALBERT (Lan al.) [2019), Electra (Clark al. 2020) and finally the text encoder of CLIP. For all text models, we consider the last representation of the class token.
We report the correlation results obtained on N24News in Fig. comparing our method against Dis- tance Correlation (dCor), which is once again the closest-performing baseline method. Our C,Cor yields a mean off-diagonal correlation of 0.66, which is noticeably higher than those of baseline methods, in the range (0.25 0.29). In fact, the correlation heatmaps for all baseline methods reveal clear block structure, such methods are able capture correlation only among same- modality encoders, but fail on cross-min modal correlation. Full results for baseline methods are available in the Appendix (Fig. (9) Instead, our method returns significant correlation values even across differ- ent modalities, in accordance with previous findings (Maiorca et al. 2023) that proved N24News representations to be transferable across models and modalities. Experiments on Flickr30k and MS- COCO confirm the behavior observed for N24News, as discussed in the Appendix (section A.5.2).
Moreover, in sections A.5.3 and A.6 we provide additional experiments on the discussed datasets to show the discriminative power of CoCor in cases with low or no correlation, while section A.7 contains additional similarity results on N24News, obtained using RTD (Barannikov et al.) 2022).
4.5 COMPUTATIONAL RESOURCES
We performed all the computations on NVIDIA A100 GPU, equipped with 40GB of RAM. The main computational hurdle of our method is the estimation La through TwoNN: our implementa- tion follows closely that of Ansuini al. (2019). which we translated to PyTorc to enable GPU ac- celeration. With this setup, ToCor runs in the order of 1s on two -i4--iimnssioal datasets of 30000 points. Just like correlation method, all the representation similarity baselines employ greatly benefit from GPU acceleration: we used them in their Py PoTch implementations provided by Mi- randa (222) (SVCCA), Maiorcal (2024) (CKA) and Zhen et al. (202)) (Distance Correlation), with

---

Published as a conference paper at ICLR 2025
dCor
CoCor
ALBERT
0.72
0.33
0:25
0.11
2.855
0.30
ATBERT
0.72
8:53
9.62
9.17
014
0.39
0.35
0.32
5:37
9:53
0.39
Becro
3:06
0:71
5:678
0.3
Elettaa
65
0.6
CLPNT
0.2
0.26
9.27
02
0.22
0.8
0.33
CLIPT
0.50
0.20
0:33
0.36
2:42
0.74
EfficennNee-
3,003
0.14
6.12
8.66
0.17
0.54
0.54
0.63
0.42
EMiceentiet
3.88
0.57
0.53
0.64
0.08
0.03
2:48
0.43
0IT-0.033
0.17
0.34
0.711
02
3:39
07
033
0.5
VIT
337
0.6
82
0.72
0:5
0.55
2:49
9.82
0:73
INAGE
ViReyend
6:35
0.078
0.22
2:38
3:67
0.50
WTT-tybrl
0.57
0.67
0.03
0.50
3.76
9:53
3:33
9:23
ALBERT
nett
Beetro
ViT UT-myarrd
RotWel
CUPV
ALBERT
MCRT-U
Bearo
Enfomntted
MF
WiTrytenn
Pesseet
44
TXXT
MAGE
TEX7
IMAGE
Correlation results on N24News representations, obtained using: left Distance Corre- lation (dCor); right IaCor (ours). Like all other baselines evaluate, dCor is only able to spot correlations between encoders of the same modality, while L,Cor reveals significant correlation for all model pairs.
Figure 4:
minor adaptations. Pretrained models were obtained from the Transformers library by HuggingFace (Wof et al. 2020), details on the checkpoints we employed are provided the Appendix (section A.8.. Our code is available at https::: /tte:://gtttttt com http://www..baaieeeeeeeeeeeeeeee
DISCUSSION
Our work introduces Intrinsic Dimension Correlation (IşCor), novel and robust method for detect- ing complex nonlinear correlations in high-dimensional spaces. Due to its flexibility, this method can be employed in wide range of applications, from natural language processing to computer vision and beyond (including other fields of science, like physics), offering a new type of analysis address how machine learning models represent data. Remarkably, our results show the effective- ness of the method to detect a correlation signal in multimodal data where we know correlation should exist but where standard methods struggle to identify any.
Limitations Among the possible drawbacks of the method, it is worth mentioning that is fully dependent on the precision of the Ia estimator, so, if the Ia is wrongly predicted the method will fail find correlations. This would likely happen when the Las involved are big, so even cest-generation estimators will be affected by the curse of dimensionality.
Future directions Our method lays the foundations for many interesting future research avenues. For example, can be used to disentangle data representations by minimizing the correlation be- tween representations from two more datasets, through the concept of Total Correlation (Watan- abe 1960). This approach could be highly relevant applications such as multimedia analysis, cross-modal retrieval, and data fusion, potentially resulting in more interpretable neural networks.
In its present form, TaCor is only applicable to datasets with uniform Ia. However, some realistic datasets can present more than one manifold with different intrinsic dimensionalities. With the development of methods that allow estimating in reliable way these local Las (Allegra et al. 2020) Dyballa & Zucker 2023). we devise that I,Cor be applied to quantify the correlations locally. example in the Appendix AA.9.
We provide a prroof-of-conce
In conclusion, tions but also paves the way for innovative solutions in representation learning and interpretability, making it valuable tool across many fields of machine learning.
method not only enhances the understanding of high-dimensional data correla-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We acknowledge AREA Science Park for making the supercomputing platform ORFEO avail- able for this work. We also thank Sanketh Vedula for insightful discussions on statistical testing- This work is supported by the PNRR project iNEST (Interconnected Nord-Est Innovation Ecosys- tem) funded by the European Union Gext-GenerationEU (Piano Nazionale di Ripresa Resilienza (PNRR) Missione 4 Componente 2, Investimento 1.5 - D.D. 1058 23/06/2022, ECS 00000043, CUP J43C22000320006)
REFERENCES
Michele Allegra, Elena Facco, Francesco Denti, Alessandro Laio, and Antonictta Mira. Data seg- mentation based on the local intrinsic dimension. Scientific reports, 10(1):16449, 2020.
Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide 7occolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Sys- tems, 32, 2019.
James Bailey, Michael E. Houle, and Xingjun Ma. Local intrinsic dimensionality, entropy and sta- tistical divergences. Entropy, 24(9):1220, 2022.
Serguei Barannikov, Ilya Trofimov, Nikita Balabin, and Evgeny Burnaev. Representation topology divergence: method for comparing neural network representations. In International Conference on Machine Learning, pp. 1607-1626. PMLR, 2022.
Bradley CA Brown, Jordan Juravsky, Anthony I. Caterini, and Gabriel Loaiza-Ganem. Relating regularization and generalization through the intrinsic dimension of activations. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022.
Paola Campadelli, Elena Casiraghi, Claudio Ceruti, and Alessandro Rozza. Intrinsic dimension estimation: Relevant techniques and a benchmark framework. Mathematical Problems in Engi- neering, 2015:1-21, 2015.
Claudio Ceruti, Simone Bassis, Alessandro Rozza, Gabriele Lombardi, Elena Casiraghi, and Paola Campadelli. Danco: An intrinsic dimensionality estimator exploiting angle and norm concentra- tion. Pattern recognition, 47(8):2569-2581, 2014.
Emily Cheng, Corentin Kervadec, and Marco Baroni. Bridging information- and geometric compression in language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12397-12420, 2023.
Laure Ciernik, Lorenz Linhardt, Marco Morik, Jonas Dippel, Simon Kornblith, and Lukas Mut- tenthaler. Training objective drives the consistency of representational similarity across datasets. arXiv preprini arXiv:2411.05550 2024.
Kevin Clark, Minh-Thame Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre- training text encoders as discriminators rather than generators. In ICLR, 2020. URL https: BtvB
//openzeview. ped fid=ridd=lli
MohammadReza Davari, Stefan Horoi, Amine Natik, Guillaume Lajoie, Guy Wolf, and Eugene Belilovsky. Reliability of CKA similarity measure in deep learning. In The Eleventh Intera- tional Conference on Learning Representations, 2023. URL
net/
forum? 8=888vyxc6666
Anthony Christopher Davison and David Victor Hinkley. Bootstrap methods and their application. Number 1. Cambridge university press, 1997.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-traine Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, 2019.
11

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Reprexentations, 2020.
Luciano Dyballa and Steven W Zucker. lan: Iterated adaptive neighborhoods for manifold learning and dimensionality estimation. Neural Compatation, 35(3):453-524, 2023.
Elena Facco, Maria d' Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimen- sion of datasets by a minimal neighborhood information. Scientific reports, 7(1):12140, 2017.
Rohan Ghosh and Mehul Motani. Local intrinsic dimensional entropy. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7714-7721, 2023.
William Hamilton, Jure Leskovec, and Dan Jurafsky. Diachronic word embeddings reveal statis- tical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 1489-155 1501, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Yasuichi Horibe. Entropy and correlation. IEEE Transactions on Systems, Man, and Cybernetics, (5):641-642, 1985.
Harold Hotelling. Relations between two sets of variates. Biometrika, 28(3/4):321-777, 1936.
lya Kaufman and Omri Azencot. Data representations' study of latent image manifolds. In Inter- national Conference on Machine Learning, pp. 15928-15945. PMLR, 2023.
Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich. Similarity of neural network models: survey of functional and representational measures. arXiv preprint (rXiv:2305.06323 2023.
Simon Kombliihh Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference machine learning, pp. 3519- 3529. PMLR, 2019.
Henry Kvinge, Davis Brown, and Charles Godfrey. Exploring the representation manifolds of stable diffusion through the lens of intrinsic dimension. arXiv preprint (30.:v233220930001 2023.
Zorah Lähner and Michael Moeller. On the direct alignment of latent spaces. In Proceedings of UniReps: the First Workshop on Unifying Representations in Neural Models, volume 243 of Proceedings of Machine Learning Research, pp. 169. PMLR, 2024. URL https://
proceedings .ml wttps/.pp...//////aesamm 244 html
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2019.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86( 96(11))2278-2324 1998.
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equiv- ariance and equivalence. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2015.
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. Advances in neural information processing systems, 17, 2004.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman -mage pre- training for unified languaeee understanding and generation. In International conference on machine learning, pp. 12888-1200 PMLR, 2022.
12

---

Published as a conference paper at ICLR 2025
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Valentino Maiorca. Latentis: Your gateway to latent space communication. https://githu Com/F F wll.yysslaatetteeeeeeeeee
2024.
Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and Emanuele Rodolà. Latent space translation via semantic alignment. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. Linearly mapping from image to text space. In The Eleventh International Conference on Learning Representations, 2023.
Brando Miranda. Ultimate anatome, the ultimate pytorch library to analyze internal representation www./ooaaaaaao. 90/
of neural networks.
2021.
Mazda Moayeri, Keivan Rezaei, Maziar Sanjabi, and Soheil Feizi. Text-to-concept (and back) via cross-model alignment. In International Conference on Machine Learning, pp. 25037-25060. PMLR, 2023.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in neural information processing systems, 31, 2018.
Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. In The Eleventh International Conference on Learning Representations, 2023. URL https::/
openreview.nee htt/://orummmmdd/mme nwieGu
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In Interna- tional Conference on Learming Representations, 2021. URL
forum?2d==KNcAAA8EY
Antonio Norelli, Marco Fumero, Valentino Maiorea, Luca Moschella, Emanuele Rodola, and Francesco Locatello. Asif: Coupled data turns unimodal models to multimodal without train- ing. Advances in Neural Information Processing Systems, 37, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexcy Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34:121 16 121222 2021.
Simone Romano, Oussama Chelly, Vinh Nguyen, James Bailey, and Michael E Houle. Measur- ing dependency via intrinsic dimensionality. In 2016 23rd international conference on pattern recognition (ICPR), pp. 1207-1212. IEEE, 2016.
Olga Russakovsky, Jia Deng. Hao Su. Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211 1-22,, 2015.
Gábor J Szěkely, Maria Rizzo, and Nail K Bakirov. Mcasuring and testing dependence by corre- lation of distances. The Annals of Statistics, pp. 2769-2794 2007.
13

---

Published as a conference paper at ICLR 2025
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114 PMLR, 2019.
Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Al- berto Cazzaniga. The geometry of hidden representations of large transformer models. Advances in Neural Information Processing Systems, 37, 2023.
Nguyen Xuan Vinh, Julien Epps, and James Bailey. Information theoretic measures for clusterings comparison: a correction for chance necessary? In Proceedings of the 26th annual international conference on machine learning, pp. 1073-1080, 2009.
Zhen Wang, Xu Shan, Xiangxie Zhang, and Jie Yang. N24news: A new dataset for multimodal news classification. 13th International Conference on Language Resources and Evaluation Conference, LREC 2022, pp. 6768 0768-6775. European Language Resources Association (ELRA), 2022.
Satosi Watanabe. Information theoretical analysis of multivariate correlation. IBM Journal of re- search and development, 4(1):66-8)) 82,6662211 1960.
Svante Wold, Kim Eshensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing: system demonstrations, pp. 38-45, 2020.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockemmaier. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975-11986, 2023.
Xingjian Zhen, Zihang Meng, Rudrasis Chakraborty, and Vikas Singh. On the versatile uses of par- tial distance correlation in deep learning. In Proceedings the European conference on computer vision (ECCV), 2022.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 THE TWONN INTRINSIC DIMENSION ESTIMATOR
The TwoNN method (Facco al. 2017) used in this work is local intrinsic dimension (La) timation technique. Its distinguishing feature is that, by focusing solely on the first two nearest neighbors, it minimizes the size of the Ia-dimensional hyperspheres which the density is sumed to be constant. This method is based on the distribution functions of neighborhood distances, which depend on la-
Specifically, for each point x in the dataset, it considers the first and second nearest-neighhor dis- tances, denoted as ri (z) and re(2). respectively. Assuming that the dataset is locally uniform within the range of second nearest neighbors. it has been shown by Facco et al. (2017) that the distribution of the ratio μ (2(2)) (rrrrr())) follows:
f = 1a-11
(4)
And using the corresponding cumulative distribution function, P(u), one can write:
In 1111-PUU P(w] In (15)
La
(5)
This relation allows for the estimation of La by fitting the set S = {1nn with a straight line passing through the origin. Here, (p) represents the empirical cumulative distribution, computed by sorting the values of μ in ascending order.
- In [1
pemp ()))(
(μ+]}
A.2 CORRELATION THROUGH THE INTRINSIC DIMENSION: AN EXACT EXAMPLE.
In order to provide an intuition of the relationship between the intrinsic dimension and the entropy of data set, let us imagine data sets generated in this way: 1) Sample multivariate Gaussian of dimension d and identity covariance matrix. This can be done by simply generating d series of Gaussian distributed numbers with standard deviation equal 1. 2) Embed them in higher dimensional space of dimension D and apply a random angle rotation around a random vector of this dimension.
The entropy of such a dataset could be approximated by the differential entropy of a multivariate Gaussian distribution
d log (2xe) + log det = d log(2 2
1 2
H - 2
(6)
where the last equality derives from the use of an identity covariance matrix.
Another interesting property of such dataset is that, since have generated the dataset through rotation, the quantity can be recovered as the number of non-zero eigenvalues of the covariance matrix. Please note that this is equivalent identifying the intrinsic dimension using PCA.
Now let us imagine that we generate three datasets in this way (X,, X₂ and X3) each of them with its own intrinsic dimension d1, d2, da and embedding dimension Di, D2. D3. While datasets Xi and X2 are completely independent, dataset X3 is built using one of the series employed for building dataset X1.
We will use PCA for estimating the La and apply equation 3 to compute the correlation between datasets these datasets. In the case of datasets X1 and X2, being fully independent, we will have Ia(Xx)= = d1, Ix(X2) = d2, and Ta(X, @ X2) = d1 + dz. By applying PCA, we can recover our datasets as multivariate normal Gaussian, therefore obtaining:
= H(X,)+ (X2)) - HXx, e X2) =
di
1 log(2 + + log(2re) d d log(2TT)
max
max( ( log (og(2 & log(2we))
di + d2 - (di + d2)) I.Cor X2) =0 (7)
max (d1, d2)
15

---

Published as a conference paper at ICLR 2025
The situation is identical for datasets X2 and X3 but, for X2 and X3. the situation changes bit. In this case, applying PCA to the combined dataset X₃ @ X3 will also provide a multivariate Gaussian, but with dimension equal to d1 d3 1, therefore obtaining & = LaCor(X,
nax(da,ds)
A.3 ADDITIONAL RESULTS ON SYNTHETIC DATA
In Fig. 5 we display the two-variable synthetic datasets we used in section 4. as a first validation for our method.
Linear
Random
Spiral
4 = 0.98
= 2.05
= 1.02
Figure 5: Synthetic datasets, each associated with its intrinsic dimension.
A.3.1 RESULTS USING THE MLE ESTIMATOR
IaCor fundamentally powered by intrinsic dimension estimator. In the main text, we employed TwoNN (Facco et al., 2017), but this is only one of the possible choices. To highlight this point, provide here results the synthetic experiments of section 4.1 using the Maximum Likelihood Estimator (MLE) by Levina Bickel (2004) instead TwoNN. MLE relies fundamentally on hyperparameter k the number of nearest neighbors consider for each data point. As show in Fig. 66 & can significantly impact the estimated Ta and, in general, there is no clear strategy to 100 in our experiments, obtain a reasonable trade-off between efficiency (the lower k the faster MLE is) and accuracy. We report in Table the
choose Based on these curves, we opt for &
LaCor results with MLE the same datasets of Table I
Linear
Random
Noll scoudadd
Tockn
1.2
3.16
egedddd
120-
120
3.00
135
230
145
3.76
100-
2.50
103
Figure 6: Intrinsic dimension estimation with the MLE estimator, varying the number neearest neighbors k. Horizontal red and green lines report the estimate found by TwoNN and the expected Ia data, respectively.
In Table 5| report the results of IoCor equipped with MLE La estimation on the same datasets employed in Table 2 We observe that Coro equipped with MLE produces slightly worse results with respect TwoNN, as La is estimated less accurately (expected values would be 8, 6 and respectively for scenarios A, B and C).
16

---

Published as a conference paper at ICLR 2025
Table 4: Correlation in the two-variables datasets of Table | (Ia intrinsic dimension of the concatenated (2D) dataset, computed with MLE; (LrC) correlation coefficient (g) returned by our method; (p-value): significance of the correlation detected by our method (100 shuffles). Results are reported as mean + std over 10 independent random samplings of data, with the exclusion of p-value, which is reported range.
Data
Ia
La Cor
p-value
Linear
1.03 = 0.00
1.00 + 0.00
0.01
Random
2.07 + 0.00
0.01 + 0.00
(0.04.0.7))
Spiral 1.02 0.00
1.06 + 0.00
0.01
Table 5: Correlation between high dimensional synthetic datasets in the scenarios of Table 2 when TwoNN is replaced by MLE in the computation of LaCor. Results are reported as mean + std over 10 independent random samplings of data, with the exclusion of p-value, which is reported as range.
Correlation
Ia
CaCor
p-value
(A) Absent
7.52 0.01
0.17 0.00
[0.17, 0.93]
(B) Partial
6.68 = 0.01
0.27 = 0.00
0.01
(C) Complete
5.81 + 0.01
0.38 0.01
0.01
A.3.2 RESULTS ON DIFFERENT DATA DISTRIRUTION
In T we provide the results for the same experiment of Table when data are sample from uniform distribution in [-T, x instead of Gaussian distribution. From this point, we switch back the TwoNN estimator.
Table 6: Correlation between high dimensional synthetic datasets in the scenarios of Table 2 when the original dataset is sampled independently and uniformly in [- æ).. Results are reported as mean + std over 10 independent random samplings of data, with the exclusion of p-value, which is reported as a range.
Correlation
CCA
dCor
IaO
Ia Cor
p-value
(A) Absent
0.02 = 0.01
0.00 : 0.00
7.21 + 0.12
0.15 0.04
[0.09, 0.96)
(B) Partial
0.17 = 0.05
0.06 0.00
5.74 0.11
0.48 - 0.04
0.01
(C) Complete
0.23 = 0.02
0.21 0.00
4.03 = 0.06
0.92 = 0.02
0.01
17

---

Published as a conference paper at ICLR 2025
A.3.3 ROBUSTNESS NOISE
To evaluate how LoCor behaves in more complex and noisy setting, we consider Scenario C (com- plete correlation) from Table 2 and perturb the second dataset (x', y', w', with Gaussian noise with mean and increasing standard deviation, up to 3 times larger than the scale the dataset (all variables are obtained through sinusoidal functions, hence they are constrained in [-1, We report Fig. 7 the correlation results obtained with C,Cor, averaging the correlation over 5 independent random noise samplings for each standard deviation value. We also report the range of p-values for each noise magnitude, showing that correlation detected with high confidence approximately until the point where noise and signal have similar amplitude.
0.7
0.6
0.4
l,Cor
0.3
0.4
0.2
0.1
0.2
0.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Gaussian noise
Figure 7: LaCor results (correlation coefficient and p-value) on completely correlated data from Table 2 scenario C, perturbed with random Gaussian noise of variable magnitude.
18

---

Published as a conference paper at ICLR 2025
A.4 ADDITIONAL IMAGENET CORRELATION RESULTS
We provide detailed results for latent correlation in ImageNet (Fig.
using SVCCA, linear kernel
CKA and RBF kernel CKA.
A
CKA (linear)
B
CKA (RBF)
SiguP- 22
04
0.41
0.5
0.47
6:31
6.6
0.66
SigUP:
53
0.41
0.42
0.51
0.46
0.53
0.61
0.40
VIT-0-22-366
0.3
6.41
0.50
0.46
0.50
6:33
0.31
0.33
VIFS-12 J84
0:
0.42
0.62
2
0.51
6.61
0.34
0.39
0.35
VIT-aybrid-
VIRL-18-22C-
63
5:47
0.6
0.50
0.53
6.30
0.1
0.30
VITL-16-224-
0:31
3.46
0.63
0.41
0.57
0.38
0.44
0.38
CUP
0
3.8
0.36
0.38
5.47
0.44
6.48
6.63
CUP
05
5.81
0.38
4.30
0.48
0.44
0.43
0.45
Eficannkett
Siguue
Vm18-10-224
ayperid
WT 16-244
Resseet
CL.0
Effcceettent
WT& 16-224
364
VIT hydbidd
WITL-66-234
Ressdet
SVCCA
EfficientNet
2:30
0:44
0.66
3:42
00
2.40
0.42
0.51
VIT---66-224
0.44
2:38
0.54
3.47
0:54
0.1
0
0.54
WTThybrdd
0:42
2:37
0.47
0.00
0.54
0.30
0.42
6.52
ResNet
0.43
2:34
0.4
0.42
3.36
0.43
0.29
0.4
-02
RUP
0:51
5.63
0.54
0.55
3.52
0.55
04
0.58
SHLUP
16-244
VI-8322.0 2044
WThytolld
WTT-16-724
Reester
CUP
BUP
130
Figure 8: Additional baseline correlation results on ImageNet representations, obtained using: (A) Linear CKA; (B) RBF CKA: (C) SVCCA
19

---

Published as a conference paper at ICLR 2025
A.5
ADDITIONAL CORRELATION RESULTS ON MULTIMODAL DATASETS
A.5.1
ADDITIONAL BASELINE RESULTS ON N24NEWS
In Fig. we provide additional results for SVCCA, linear kernel CKA and RBF kernel CKA on N24News representations. These methods perform similarly to Distance Correlation, and only capture high correlation between representations belonging to the same data modality.
A
CKA (linear)
B
CKA (RBF)
BERT-C
BERT-C
BERT-U
Ekcoro
CLINT
CUPT
EmicientNet-C
EfficintNetto
MAAE
ST-hybrid-e.99)
VIT-hy-id
ResMett
ResNet.
CLPV
837
CUPV
ALBEST
BERtO
Blecra
CUPT
Effckentile
VIT
RemMet
GLINN
ALBEST
Beette
CUPT
VI
Rattee
TEXT
IMAGe
TEXT
MAGE
SVCCA
ALSERT
0,3
0.29
0.26
23
0.21
0.15
5.1b
03
0.2
BERT-C
0.5
0.25
0.27
32
0.21
0.17
0.37
0.27
0.22
TEXT
BERTAU
023
0.55
0.26
0.22
0.21
0.17
0.37
0.27
0.22
Electra
028
0.27
0.8
0.29
0.2
0.15
0.36
0.18
0.38
CUPT
028
0.32
0.52
0.26
0.45
0.23
0.25
0.18
0.3
EffciintNett 0:2 0.21
0.2
0.2
0.25
0.55
0.35
0.4
0.34
VT- 0.18 0.27 0.17
0.15
0.23
0.36
0.35
0.31
0.35
BAAGE
0.1 0.27
0.17
0.16
0.23
0.35
0.
0.31
0.37
ResNet- 0.16 0.27
0.17 0.16
0.18
0.4
01
0.31
0.25
CLPV- 0.2
0.22
0.2
8.18
6.3
9.34
33
0.37
23
ALBERT
BBACC
Eletrr
CUPT
Erfeckethlt
Net hybridd
Rostht
CLIR
TEXT
IMAG
Figure 9: Additional baseline correlation results N24News representations, obtained using: (A) Linear CKA; (B) RBF CKA; (C) SVCCA
20

---

Published as a conference paper at ICLR 2025
A.5.2 RESULTS ON OTHER MULTIMODAL DATASETS
In Fig. 10 and Fig. [1 we report the correlation results we obtain respectively on Flickr30k (Young et al. 2014) and MS-COCO (in et al. 2014). two multimodal datasets containing images and the corresponding captions. We employ the same models used for N24News (section (.4). As for the previous dataset, we observe that CoCor outperforms previous baselines, which significantly struggle to find similarities between embeddings of different modalities.
A
IgCor
3.0
ALBERT
8:43
0:24
6:74
0.42
0-4
1.54
BERTS
2:01
5.3
6.78
0.76
0.53
0.61
0.5
6.5
T2XT
Eettr
2.26
3.24
6:64
0.65
0.45
9:5
0.52
CUP
3.24
5:75
0:04
0.57
0.57
0.03
Emionneer
14
VIT
6/62
4:49
6.45
6:65
6:56
CLB3
5.4
0.22
3.79
MAVGE
Vi--yyord
5:02
0.67
1.2
Roshet
6.4:
6:57
6:61
0.76
0.22
0.74
ALBENTT
BEnT
Elerree
S1-byyorld
Restee
CLPN
TEXT
HAACE
CKA (linear)
CKA (RBF)
ALSERT
0.23
1:3
0.18
3.22
AUDEST
622
0.22
E22
0L17
DCNT-C
1.52
3.20
0.1
0.13
BERTC
DOTAL
0U5
022
3.50
CAT5
2:2
60
122
0,01
8,00
Cedre
1.2
2.12
2.24
CADS
622
0.23
0.14
0001
CL
0.26
1.33
0.32
6:2
0.24
CUNT
912
1.24
9:055
0:024
3.21
0A4
ERlceddtet 0.02 622
0471
0:03
032
0:226
3.24
6.46
6:2 0482 0004
025
9.004
0:020
3.27
6.4
034
ViTRydndd 3:3 6:2
0492 0007
0.29
6.28
Pen
9313
4.22
1.2
654
0/4
Resiter GO05 6:2 0.08
0003
022
CLEN 8M
3:20
6:29
CLPV--00
6:7
0.13
0.44
ALBERT
QUITT
ALSION
CuPr
Resden
Place
TEXT
IPAcz
SVCCA
E
dCor
ALPENT
8:0
078
111
025
011
837
6:7
AURE4T
611
6.0
239
017 6081 1377 E355 5:1 EBM
6:22
2.1
038
3
622
81
8344
6:1
3.2
3:
0.11
227
117 631
832
6:17
BERTA
6:1
0.3
037
332
022
531
838
6:18
322
627
221
031 0081 045
500
6.7
Betto
628
0.79
277
37X
025
818
31X
Henio
622
011
310
201
6:3
6.5
2.31
033
027
621
827
6:7
527
CUP
317
63
0371
2.58
327
633
821
6.3
Miceetter
601
0.2
5.2
037
3.7
8.31
seniter J008
632
C401
3.24
027
E31
6:44
0.18
537
635
327
6.18
WT pamt 6:38
0.20
1M
AMC
637
233
6.18
327
031
3
634
0.2
355
844
Redker
011
618
217
12
Redket ooe4 632
2014
0.28
2M
0.22
122
618
227
635
3.0
CLPN 0o04
637
612
0.38
ASMES
BERIC
PEBOC
Kextra
Cues
Eicleder
Tbxr
Figure 10: Correlation results on Flickr30k representations, obtained using: (A) our method IaCor; (B) linear kernel CKA: (C) RBF kernel CKA; (D) SVCCA; (E) Distance Correlation
21

---

Published as a conference paper at ICLR 2025
A
IgCor
ALSERT
2:90
2:34
6:42
6.82
0.43
0.51
0.47
6.5
3.49
0.5
0.49
0.66
3.05
2:85
2
0.72
0.45
0.55
0.52
0.24
0.54
5:79
0.29
5.43
0.45
0.33
0.62
0.64
6.4
VIT
9:31
8:32
0.34
0.33
3.4
0.82
0.25
0.04
0.79
MANE
1.2
ResKet
6.51
0.54
6:49
0.64
0.66
0.74
CUPV
22
R,22
1:34
6:42
0.19
0.81
9.14
AUBI
Plecte
NY
Restie
GLIV
TEx7
PAAE
CKA (linear)
CKA (RBF)
0:41
2:23
0:37
312
Edes 500
EENL o0R
AUMe-1
e:7
0.44
2:4
0:3
5331
C476
0.00
B64T.C
6:39
5:47
0:3
3.30
631
3.2
621
BENTC
3:
821
CJ5
317
621
6:24
0:40
034
011
912
8:
0.03
6314
0114
6:7
0:1
224
3.77
5.11
C409
5:03
0.29
3:
624
0.21
0:01
5333
CAZT
9007
KT
6:14
0:00
0.07
0:11
9.0
2:
0:1
9:00
mm
0.18
3.64
6:3
51
2:57
0.05
CM
VIT.2
2:2
6.18
3.42
032
0.01
6:22
0.13
9:01
0.4
MM
0:29
2:1
9.0
ViThyend oort
6:22
0014
9307
Nestet 2014
0.24
233
3.4
6M
0.3
SST
Resket 0.01
6:35
0.18
9307
0.4
CPV 2.9
2:7
82m
2:00
6:1
241
IV Gooe
6:28
618
8:00
ALMEI
Wees
GUPT
ALASTT
RESSC
Teer
PAAGE
TExT
IMAce
SVCCA
E
dCor
ALEERT
8.20
330
032
AUDE
6:2
032
E:5
033
0001
6.27
337
023
0.5
3:30
824
GA4
BOTTAL
337
BERTUU
8.5
0.5
Dedre
1.29
031
Diesrr
3:5
624
0.26
0.03
1.23
0.11
Effckendee 0.01
6.29
0,33
0000
3.00
6:2
WT-tyott-
VTTRodedd
119
6.2
0.16
6.2
120
CLIVA
0.55
9103
RAMEET
ALSSENT
TEXT
Figure 11: Correlation results on MS-COCO representations, obtained using: (A) our method IaCor; (B) linear kernel CKA; (C) RBF kemnel CKA; (D) SVCCA; (E) Distance Correlation
22

---

Published as a conference paper at ICLR 2025
A.5.3 PARTIAL MULTIMODAL CORRELATION
In multimodal setting, l.Cor adapts to the strength of the correlation between textual and visual representation. In this experiment, we consider again the Flickr30k dataset, which provides 5 human annotated captions per image. Instead of computing the correlation between each image represen- tation and a single corresponding caption set, we consider here an enriched textual representation, containing the concatenated encoding for relation by shuffling an increasing number of textual representations. As we report in Fig. [ CaCor decreases almost linearly with the number of perturbed textual representations. Instead, the p-value stays the minimum (0.01) until the last step, where all caption embeddings are shuffled and, consequently, correlation is totally lost.
captions. Then, we progressively perturb the cor-
lcCor
p-value
0.4
0.40
0.3
lyCor
0.30
0.2
0.1
0.20
0.0
1
4
Shuffled caption sets
Correlation between visual and textual representations Flickr30k, considering all captions per image and shuffling increasing number of textual encoding sets. Representations computed using CLIP-VIT (images) and BERT (text).
Figure 12:
23

---

Published as a conference paper at ICLR 2025
A.6 CORRELATION BETWEEN DIFFERENT DATASETS
In this section, provide assessment of LaCor on unimodal image representations of different input datasets, computed using CLIP-ViT-B As we report in Fig. [3 LaCor scores significantly fall off- off-diagonal (on different input data), and the corresponding p-values increase, indicating lack of correlation.
lCCor
p-value
ImageNet-
1.000
0.246
0.074
0.173
ImageNet
0.010
0.703
0.564
0.168
N24News-
0.246
1.000
0.072
0.167
N24News
0.703
0.010
0.376
0.634
0.66
MS-COCO
0.074
0.072
1.000
0.068
MS-COCO
0.564
0.376
0.010
0.772
0.44
Fllckr30k 0.173
0.167
0.068
1.000
Flicke330k
0.168
0.634
0.772
0.010
00.0
ImageNet
N24News
MS COCO
Flickr30k
ImageNct
N24Nows
MS COCO
Flickr30k
Correlation results between representations produced by the CLIP vision encoder on different image datasets, reported in terms of LaCor coefficient (left) and p-value (right).
Figure 13:
A.7 MULTIMODAL RTD SCORES
In this section, report the divergence scores computed using Representation Topology Divergence (RTD) (Barannikov et al. 2022) on N24News encodings. As reported in Fig. [4 RTD captures stronger similarities within the visual domain (off-diagonal mean divergence is 26.8), while it finds weaker similarities in the textual and cross-modal settings, with means slightly above 45.
RTD
-00
ALBERT
BERT-C
49
24
30
44
45
44
38
70
TEXT
Electra
27
35
57
55
44
31
40
EfficientNet.
76
41
27
37
30
IMAGE
ViT-hybrid
78
40
55
22
16
30
28
10
CLIRA
34
ALBERT
BERT
BERTA
Elestra
CURT
Efficenthett
VIT
WT--hybrid
Resthet
CLIPU
TEXT
MAGE
Figure 14: RTD (Barannikov al. w.r.t. previous figures as RTD produces a divergence score, and not similarity score.
2022
scores N24News representations. Colormap is inverted
24

---

Published as a conference paper at ICLR 2025
A.8 MODEL DETAILS
All models we employ are taken pre-trained from the HuggingFace trans formers Wolf et al. (020)) library. We report in Table 7 the full list of pre-trained models employed in this work, associated with the name of the corresponding checkpoint in the library.
Table 7: Reference guide for pre- rreitrained model checkpoints in HuggingFace t transformest Wolf et al. (020) library.
Name in the paper
pre-trained checkpoint name
ALBERT
albert/aaleerttbass-22
BERT-C
google-bert/berrt--ba---aee
BERT-U
goog///-bsrr//errtbbaaa-uuaaaa
Electra
CLIP (-T/-V)
apenai/clapyy1--baaaaaatttt
BLIP
EfficientNet
goggla//fffccieennntt----00....3
SigLIP
ViT-B-16-224 (ViT)
goggle/wit-bbaeeeattchh---22
ViT-B-32-384
soqgle//it-baae--atchh333333
ViT-hybrid
ViT-L-16-224
spogee/iit-laare-aachhh-22222
ResNet
mi microsoft/rreseet-18
25

---

Published as a conference paper at ICLR 2025
A.9 LOCAL IdCOR
In this section, we provide proof-of-concept illustration of a local version of LaCor, that employs IAN (Dbbllla & Zucker 2023), local La estimator.
Local lcCor
-10
0.8
JoCor
0.4
-0.2
-0.2
Figure 15: Local LoCor with IAN estimator. It be seen that in the center of the spiral, where the data is nearly uniformly distributed, the correlation is low, while in the outer region, where correlations force the data to be in the arms of the spiral, the Cor index is
26


Reviews:
[{"review_id": "PodccTWOWC", "reviewer": "Reviewer_xZgg", "summary": "The paper introduces a new metric, Intrinsic Dimension Correlation (IdCor), designed to detect nonlinear correlations between features describing data points, in particular the latent representations output by deep networks. IdCor leverages \"intrinsic dimension\" (Id) as a proxy for information and estimates correlation using a normalized mutual information approach. By quantifying correlation based on the minimal number of variables required to represent data, IdCor aims to provide insight into nonlinear dependencies that standard methods may not achieve. The authors test their method on a variety of synthetic datasets and deep neural networks, including multimodal models.", "questions": "- How does IdCor perform under different levels of added noise, especially when applied to real-world, noisy datasets?\n\n- Why is Id+ not included in Table 2?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- IdCor introduces an interesting approach of measuring nonlinear correlations by combining the notions of mutual information and intrinsic dimensionality.\n\n-  The paper conducts experiments using both synthetic and real-world datasets.\n\n- The paper is well-written and organized.\n\n- Interesting discussion on latent space alignment.", "weaknesses": "__Choice of Id estimator:__ Not enough reasons are given for the choice of using TwoNN. Given that this is the main point of the paper, benchmarking it against other more popular methods for Id estimation would be strongly advised, e.g., Levina & Bickel's. Moreover, through lines 179--185, the authors list several references that estimated the Id found in the representations output by deep networks. I believe it would be helpful to list which methods those authors used for computing Id and possibly using the same ones to compare against using TwoNN.\n\n__Id assumptions:__ The proposed measure IdCor assumes that the intrinsic dimensionality is roughly constant throughout the data. That is unlikely to be true for most real-world data (in line with what the authors point out in lines 312--313). A simple example is one in which we have clustered data but different clusters might have different dimensionalities -- a reasonable assumption if one takes into account that a clustered representation might be clustered precisely due to the fact that different \"concepts\" may require different topologies. E.g., the dimensionality of the space of images of dogs is likely to much much higher than that of images of soccer balls due to the different symmetries and \"directions\" of variability (e.g. color) between the two objects encountered in natural scene data. Here are a couple of recent papers that directly consider this possibility and propose solutions for dealing with this dimensional heterogeneity:\n\n[1] Allegra, M., Facco, E., Denti, F., Laio, A., & Mira, A. (2020). \"Data segmentation based on the local intrinsic dimension\". Scientific reports, 10(1), 16449. \n\n[2] Dyballa, L., Zucker, S., (2023). “IAN: Iterated Adaptive Neighborhoods for manifold learning and dimensionality estimation”, Neural Computation, 35 (3): 453-524. \n\nThe first one is in fact by the same group as that of the TwoNN method. So I would consider both including this limitation in Discussion section, and proposing a way to adapt the IdCor computation that would allow for multiple dimensionalities to be present within the same dataset. (E.g., considering each subset with homogeneous dimension at a time and later aggregating the results somehow seems like a reasonable candidate strategy.)\n\n__Synthetic results:__ I believe the authors jump prematurely into the deep net section without sufficiently exploring the synthetic data first (important, because it provides ground truth). For example: since the data is randomly sampled, how many times has this experiment been repeated. Then, values reported in Table 2 should show the mean and standard. dev. across multiple random seeds, not just one instance. (as far as I understand it, although multiple permutations were considered to compute the p-values, they all came from the same data set, correct?)  How many points are sampled? (Important when estimating dimensionality.) How robust are the results w.r.t. the average sampling density? How is the sampling done? Uniformly or drawn from a Gaussian distribution? (both should be tested) Why not show the results for several values of Id? I think the results with synthetic data can be made considerable more interesting but adding these. These modifications are easy to implement (see [2] for examples), and it would be a much stronger way of empirically testing the robustness of the proposed method. (I would also suggest to move these results into the Results section.)\n\n__Figure 2:__ the points plotted should also be the mean across several different weight initializations of the MLP, with std. dev. shown as error bars.\n\n__Figure 3:__ Authors show that the value of IdCor are high when the same dataset is compared across multiple representations output from networks. I believe it would be important to show that if the dataset is changed, then IdCor should go down considerable (this would work as a sanity check that the method is not simply giving a high correlation between any two high-dimensional datasets). The multimodal results in Table 3 do not help in showing this, since the values produced by IdCor are still the highest (or close to) every case.\n\n__Minor points:__\n\nLine 239-240: It is confusing to call these datasets \"2D\", given that two of them are actually 1-D manifolds. What is common between them is that they are all embedded in 2D space;  this should be made more clear. Line 196: for the same reason, it is inadvisable to call the spiral shaped dataset simply a \"2D dataset\". It is a 1-D manifold embedded in 2-D ambient space. This is in fact what was shown in Fig. 1, bottom-right: the Id of the circle is correctly labeled as 1.\n\nThe paper could elaborate more on how IdCor’s detection of nonlinear correlations leads to interpretability benefits, providing examples of how this information could aid in model evaluation or adjustment. In other words, clarifying how quantifying similarity between different models (lines 36--37, 107)) can help interpreting them?\n\nTwoNN should be defined either in the main text or the Appendix since it is the method underlying all Id computations used in IdCor.\n\nLines 163-164: PCA might not be correct even if the data lie in a hyperplane: for example, a circle (1-dimensional) would lie on a hyperplane yet would require 2 principal component coordinates to be embedded. Thus, PCA can find the minimum number of dimensions required to embed the data (dimension of the ambient space), but not necessarily their intrinsic dimension (in the topological sense).\n\nThe methods listed in lines 166--169 can be used for dimensionality reduction but not for dimensionality estimation. Unlike PCA, which produces a way to rank dimensions based on their explained variance, all of these require the user to pre-specify the desired dimension.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W5QBZz78JM", "reviewer": "Reviewer_2zy4", "summary": "The paper presents Intrinsic Dimension Correlation ($I_{d}$Cor), a novel similarity metric aimed at detecting correlations between high-dimensional data representations, particularly in nonlinear, multimodal contexts. Unlike traditional similarity metrics that rely on pairwise distances or explicit mappings, $I_{d}$Cor leverages intrinsic dimensionality (ID) as a proxy to assess correlation. It operates by measuring the reduction in combined intrinsic dimensionality when two datasets are concatenated, capturing their shared structural complexity. The paper applies $I_{d}$Cor to multimodal datasets, showing potential in revealing non-linear correlations between different modalities of data corresponding to the same source. $I_{d}$Cor demonstrates the ability to detect correlations that traditional metrics like CKA, CCA, and Distance Correlation (dCorr) might miss.", "questions": "1. Have you run experiments where you examine more than two nearest neighbors of each point when computing intrinsic dimension?\n2. What is the exact computational complexity of your proposed approach?\n3. Why is only the p-value reported for coarse vs. random alignment? I would like to see the IdCor value as well.\n4. Are the p-values consistently the lowest possible in all the multimodal heatmaps?\n5. For the N24News dataset, do you treat the headline as the corresponding text component or the caption?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Innovative Use of Intrinsic Dimensionality for Correlation Measurement: Extending traditional distance correlation by looking at intrinsic dimension correlation is a novel perspective in representation similarity metrics. \n2. Focus on Multimodal and Nonlinear Correlation: $I_{d}$Cor is specifically designed to tackle non-linear correlations in multimodal data, an area where traditional similarity metrics often struggle. \n3. $I_{d}$Cor incorporates permutation testing to establish the statistical significance of its correlation scores. This approach adds robustness to the metric, allowing users to differentiate between meaningful correlations and spurious ones with a quantifiable p-value.", "weaknesses": "I believe the current version of the manuscript requires several improvements, but I’ve summarized my primary concerns below:\n\n1. **Insufficient Evaluation:** $I_{d}$Cor’s evaluation lacks rigor in assessing fundamental metric properties. I would like to see a comprehensive study on the metric’s invariance properties (e.g., scaling, rotation, linear/non-linear transformations, translations, subset translations, etc.). Additionally, Ding et al. [1] proposed a comprehensive set of sensitivity and specificity tests for validating new similarity metrics. Performing these tests on $I_{d}$Cor would help establish its standing among existing metrics.\n\n\n2. **Lack of Comparison to Recent, Relevant Metrics:**  $I_{d}$Cor is compared only to CCA and CKA, yet several newer similarity metrics have since emerged, such as Representation Topology Divergence (RTD) [2], Graph-Based Similarity (GBS)[3], and Inner Product Similarity[4]. You could even look at Deep CCA [5]. These metrics are known to capture complex structural and topological differences, often outperforming CKA and CCA. Benchmarking $I_{d}$Cor against these advanced metrics would clarify its effectiveness and limitations, especially in high-dimensional and multimodal contexts. \n\n3. **Ambiguity in Purpose:** The paper’s positioning of $I_{d}$Cor as either a structural or functional similarity metric is unclear. It is referred to as a correlation metric, but intrinsic dimension calculation relies on distances between data points, suggesting a structural basis. However, the metric is evaluated on its ability to detect similarities between multimodal representations (which, according to Maiorca et al. [6], differ primarily by affine transformations), leaning toward functional similarity. The choice of benchmarking $I_{d}$Cor against structural similarity metrics like CKA (known to be limited to scaling and orthogonal invariances) instead of functional metrics contributes to this ambiguity. I recommend clarifying the intended classification in the paper.\n\n4. **Limitations of TwoNN:** \n- $I_{d}$Cor ’s reliance on intrinsic dimension estimation through second-nearest neighbor (TwoNN) consistency severely limits its insight into global structural alignment. I find it hard to see how $I_{d}$Cor could accurately quantify correlation between two spaces as a global value when it relies only on 2-NN consistency to assess similarity. Many fundamental changes can be made to a representation space while maintaining 2-NN consistency. Since the metric is “automatically invariant to any transformation that preserves the local neighborhood structure of data points,” it would miss changes that impact global structure but not local neighborhoods. This narrow focus makes $I_{d}$Cor unreliable in scenarios requiring a functional or comprehensive structural alignment.\n- The paper evaluates $I_{d}$Cor on multimodal alignment with spaces that are already trained and tend to have good nearest neighbor consistency. I would like to see tests on spaces with high local but low global consistency to see if  $I_{d}$Cor can still perform well. This limitation raises the concern that  $I_{d}$Cor may be reporting high correlations due to a lack of sensitivity to broader transformations rather than genuinely revealing correlations among multimodal spaces. Additional experiments with controlled transformations (such as subset translations) are necessary to determine whether  $I_{d}$Cor’s high correlation scores reflect meaningful similarity or an inability to detect fundamental changes in the representation space.\n\n\n    \n    \n\n\n\n[1] Frances Ding, Jean-Stanislas Denain, and Jacob Steinhardt. Grounding representation similarity\nthrough statistical testing. Advances in Neural Information Processing Systems, 2021. \n\n[2] Serguei Barannikov, Ilya Trofimov, Nikita Balabin, and Evgeny Burnaev. Representation topology\ndivergence: A method for comparing neural network representations. Proceedings of\nthe 39th International Conference on Machine Learning, 2022\n\n[3] Zuohui Chen, Yao Lu, Jinxuan Hu, Wen Yang, Qi Xuan, Zhen Wang and Xiaoniu Yang. Graph-Based Similarity of Neural Network Representations. arXiv 2021\n\n[4] Wei Chen, Zichen Miao, and Qiang Qiu. Inner product-based neural network similarity. In\nThirty-seventh Conference on Neural Information Processing Systems, 2023.\n\n[5] Galen Andrew, Raman Arora, Jeff Bilmes, Karen Livescu.  Deep Canonical Correlation Analysis.  Proceedings of the 30th International Conference on Machine Learning, PMLR 28(3):1247-1255, 2013.\n\n[6] Valentino Maiorca, Luca Moschella, Antonio Norelli, Marco Fumero, Francesco Locatello, and\nEmanuele Rodola. Latent space translation via semantic alignment. Thirty-seventh Conference\non Neural Information Processing Systems, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4QpwZrlmPr", "reviewer": "Reviewer_KtPo", "summary": "This paper propose a way to define a proxy metric for mutual information to measure the relationship between two high dimensional distribution. The idea is fairly simple. Given distribution X and Y, and joint distribution X \\times Y, it uses manifold learning based dimension reduction algorithm to reduce the dimension of X, Y and X \\times Y. Intuitively, if X \\times Y can be reduced to the same dimension of X or Y, then X and Y must be highly related. Moreover, the relationship between X and Y can be nonlinear due to the nonlinear dimension reduction algorithm. \n\nThey author shows clean and well illustrated toy example to illustrate how the metric identify the nonlinear relationship between data. And show the model can be applied to analyze the distance between two neural representations and multi-model neural representations.", "questions": "I don't have any questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Simple and well-motivated idea. I really like this idea! \n2. Writing is good and easy to read.\n3. Potentially influential application for deep learning and many field like neural science (distance between artificial/biological neural representation).", "weaknesses": "1. I think you want to show the discriminative power of I_dCor. This is probably the most important weakness. i.e. show I_dCor is not just a high score in general. If it maps different neural representation trained on imagenet to be close, then it should also map neural representations pre-trained on different dataset to be far away. So it would nice if you can come up with a task or metric that measure the discriminative power of your method I_dCor compared to other method. For example, construct subset of Imagenet using class label, map through a pretrained encoder, and use I_dCor to measure distance between the subset. You can use the distance metric and maybe just nearest neighbor to classify the label of each subset. Then show the I_dCor is superior then other measure. This might not be the best idea, but I believe you guys can come up with something better.\n\n2. The assumption might be too strong. Like the author mentions, this idea is depends on the manifold learning algorithm for non-linear dimension reduction. The manifold learning must be able to identify the manifold from a high dimensional ambient space. But personally I think it's fine if you show the method actually works well, but you need more analyze to define \"work well\" than what you currently have. For example, see 1.\n\n3. The analysis in figure 3 could be less ambiguous. I think you want to say the different (neural) representation of ImageNet is consistently similar under I_dCor, i.e. the similarity score has small variance. But it has a much bigger variance under a different metric under other metrics. I think this could be better shown than using two tables. \n\n4. I'm not sure if figure 2 is that meaningful. Because these different metric are under different scale right? As the network becomes more nonlinear, other metric decrease, so is I_dCor. So I'm not sure what it really says.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QkSGsoiuAI", "reviewer": "Reviewer_ixgw", "summary": "The authors introduce a novel metric for determining the correlation of high dimensional datasets. They propose to use normalized mutual information, whereby the mutual information, expressed in the form of entropies, is replaced with the intrinsic dimension, which they calculate using the Two-NN method. This metric is then tested on synthetic data against CCA and dCor, showing that IdCor can effectively find correlation in non-linear settings, and can do so better than dCor. The authors then go on to show this in the context of large vision models, showing the correlation between various large vision models, and then extend into a multi-modality setting, showing a correlation between vision models and LLMs on a multi-modality dataset.", "questions": "In table 3/Section 4.2.1 \"Coarse Alignment\", it is seen that IdCor is outperformed by other methods for CLIP and SigLIP. The authors do give an explanation as to why, stating that these two use a self-supervised loss, which have no explicit notion of class. This is a good explanation, but I think this should be discussed a little bit further. Is it that any model with no explicit notion of class/using a self-supervised loss will cause IdCor to be less performant (meaning this should be put as a limitation), or is this a fluke of these two specific examples, and on other methods with self-supervised loss, IdCor outperforms other metrics? Either a theoretical explanation of why this is the case, or a few more examples using self-supervised loss on coarsely aligned data could help greatly towards ensuring that future works do not encounter problems with IdCor by inadvertantly using it incorrectly.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "From an originality/novelty perspective, the paper is quite novel in that this is a new use case for the relationship between intrinsic dimension and entropy, being substituted into a mutual information context.\n\nThe paper is quite clear, and it is very easy to see not only how various parts are put together, but the rationale between combining them as well. This is especially apparent in the beginning of Part 3: \"Correlation through Intrinsic Dimension\", where the authors give a very clear example why ID is needed, and the relationship betwen ID and NMI. \n\nThe authors do a very good job anticipating many of the questions readers might have, and explaining abnormalities in the results. In the results, they address several oddities that appear, such as the apparent lack of comparative performance on SigLIP and CLIP for coarsely aligned imagenet.\n\nOn the topic of results, the results are very good, with IdCor demonstrating very good performance, and a clear picture is painted as to its ability to deal with nonlinearity when obtaining a correlation metric. \n\nThe authors have a very substantial list of references, and are hitting many of the related and necessary works for understanding what the authors are trying to do. \n\nThe combination of these come together to create a very high-quality paper. Moreover, I believe this could have a moderate to highly significant impact on the field, and I can see how this new metric can certainly help with discovering the underlying mechanics of deep neural networks.", "weaknesses": "While the work is generally very clear, there is one place where a slight modification would help significantly. \n\nWhile the reasoning behind using IdCor and not Id by itself is made fairly clear, the inclusion of the intrinsic dimension itself in table 1 could cause some confusion in this matter. A quick explanation of why Id is not sufficient (I believe no more than a sentence or two would be needed) should be all that is needed to ensure that it is clear that IdCor gives a better estimate, and is more useful. \n\nOther than that, there are only a few very minor grammatical mistakes. The most pressing is Section I, last paragraph, bullet point 1,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JZ9RtV6MbZ", "reviewer": "Reviewer_i5Pr", "summary": "This paper introduces a novel metric for latent space analysis based on intrinsic dimension correlation. With respect to previous metrics, this paper proposes a method that takes care of data nonlinearities and computations efficiency, two crucial aspects of real-worls problems.\n\nI would like to thank the authors for this interesting paper. Overall, I find the paper interesting and relevant to the scope of the conference.", "questions": "1) Related to the weakness 1, I would like to know whether exists a formulation for more than two modalities.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1) The paper explores the latent space from an information-theoretic point of view, which is very valuable considering that nowadays lots of papers just do so from architectural and neural points of view.\n2) The method proposed in the paper consider the nonlinearity of the data, representing a crucial advancement with respect to previous methods that mainly focus on linear structures.\n3) Results of the method seem consistent across different types of representations, ranging from different models to multimodal.\n4) The paper is readable and easily understandable.", "weaknesses": "1) Although the authors claim that the proposed method can be extended to more than two modalities, no mathematical formulation is presented for this in the method section. Indeed, equations (1,2,3) focus on X and Y, and I am curious if it is possible to extend this to three or more.\n2) A graphical representation, which is lacking, of the differences between the various methods (like figure 1) may help the reader better understand that method and its advantages.\n3) The authors could maybe insert a downstream example in which their method makes the difference. This would make the paper more effective and also highlight the importance of the proposal to practical researchers.\n\nMinor concerns that do not affect my rating but that the authors could improve:\n1) In scientific writing, the Saxon genitive should be avoided, even though I know that both some LLMs and Grammarly suggest using it. I would personally remove the usage of the Saxon genitive in the paper.\n2) Usually, figures of results are presented putting on the left the old method and on the right the proposed one. In this paper, the authors often do the opposite, which may confuse the reader. I suggest changing to the conventional method of results presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)