Spectral Analysis of Representational Similarity with Limited Neurons
Authors: Hyunmo Kang
Abdulkadir Canatar
SueYeon Chung
Venue: NeurIPS 2025 poster ()
Keywords: representational similarity
centered kernel alignment
random matrix theory
computational neuroscience
Primary area: neuroscience_and_cognitive_science

Abstract: Understanding representational similarity between neural recordings and computational models is essential for neuroscience, yet remains challenging to measure reliably due to the constraints on the number of neurons that can be recorded simultaneously. In this work, we apply tools from Random Matrix Theory to investigate how such limitations affect similarity measures, focusing on Centered Kernel Alignment (CKA) and Canonical Correlation Analysis (CCA). We propose an analytical framework for representational similarity analysis that relates measured similarities to the spectral properties of the underlying representations. We demonstrate that neural similarities are systematically underestimated under finite neuron sampling, mainly due to eigenvector delocalization. To overcome this, we introduce a denoising method to infer population-level similarity, enabling accurate analysis even with small neuron samples. Theoretical predictions are validated on synthetic and real datasets, offering practical strategies for interpreting neural data under finite sampling constraints.

Spectral Analysis of Representational Similarity with Limited Neurons
Hyunmo Kang" Department of Physics Johns Hopkins University
Abdulkadir Canatar"
Center for Computational Neuroscience Flatiron Institute
Baltimore, MD 21218 hkang560jh.es edu
New York, NY 10010
acanttarffaccrronhssttttt.....
SucYeon Chung
Department of Physics
Harvard University
Cambridge, MA 02138
sueyeonchunggg hwrvardrddc.m.. edu
Abstract
Understanding representational similarity between neural recordings and compu- tational models is essential for neuroscience, yet remains challenging to measure reliably due to the constraints on the number of neurons that can recorded simultaneously. In this work, we apply tools from Random Matrix Theory to investigate how such limitations affect similarity measures, focusing on Centered Kernel Alignment (CKA) and Canonical Correlation Analysis (CCA). We pro- pose an analytical framework for representational similarity analysis that relates measured similarities to the spectral properties of the underlying representations. We demonstrate that neural similarities are systematically underestimated under finite neuron sampling, mainly due to eigenvector delocalization. Moreover, for power-law population spectra, we show that the number of localized eigenvectors scales as the square root of the number of recorded neurons, providing simple rule of thumb for practitioners. To overcome sampling bias, we introduce denois- ing method infer population-level similarity, enabling accurate analysis even with small neuron samples. Theoretical predictions are validated on synthetic and real datasets, offering practical strategies for interpreting neural data under finite sampling constraints.
Introduction
Understanding how artificial neural networks relate to biological neural activity remains one of the central challenges in computational neuroscience [13, 50, 40]. As deep learning models become increasingly sophisticated at matching human-level performance interest in whether these models actually learn representations that mirror those found in the brain [53, 26, 25, 45, 35]. However, fundamental obstacle stands in the way of making this comparison: while artificial networks can be analyzed their entirety, neuroscientists can only record from a small subset of neurons in any given brain region [10, 51, 47J. This sampling limitation poses critical challenge for the field. When we measure the similarity between model and neural representations using standard techniques like Canonical Correlation Analysis (CCA) Centered Kernel Alignment (CKA), how much does our limited neural sample size distort the true relationship? Addressing this
complex tasks, there is growing
"qqual contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

issue is critical, given that these metrics increasingly inform model selection and neuroscientific interpretation [42, 39].
Our work provides the first rigorous theoretical framework for understanding how neuron sampling affects representational similarity measures. Our analysis reveals that measuring CCA and CKA with limited number of recorded neurons systematically underestimates the true population-level similarity. This underestimation stems primarily from eigenvector delocalization [l, 15, 4J-a phenomenon where sample eigenvectors become increasingly misaligned with their population counterparts as the number of recorded neurons decreases.
Our analysis proceeds two parts. First, in the forward problem, we investigate how neuron sub- sampling from the full underlying population distorts the population eigencomponents and how this distortion affects the computed similarity measures. Second, in the backward problem, we ask whether observations from finite number of neurons can be used reliably infer the population representational similarity.
1.1 Our Contributions
Eigencoopponet--iis Analysis of Representation Similarity: We show how neuron sub-sampling alters the eigenvalues and eigenvectors of the Gram matrix, leading to systematic underestimation of CCA/CKA due eigenvector delocalization.
Backward Inference via Denoising Eigenvectors: We introduce a denoising method that leverages population eigenvalue priors (e.g., power-law) to infer the true population similarity from limited data, substantially correcting the sampling bias.
Validation Real Neural Data: Applying our framework primate visual cortex recordings confirms that even modest neuron counts can lead to severe underestimation of brain similarity and that our method effectively recovers the missing signal.
1.2 Related Works
Representation similarity measures expressed in terms of eigencomponents were presented in detail by Korbblith al. [29], who showed that CCA, CKA, and linear regression scores can all be written in terms of the eigenvalues and eigenvectors of the Gram matrices.
key question is how these similarity measures behave under different kinds of noise. Broadly, there are two primary noise sources:
Additive noise, which arises from trial-to-trial variability and measurement error. In many studies, repeated trials and averaging can substantially mitigate this type of noise.
2. Sampling noise, which occurs because we can only record from limited subset of neurons rather than the entire population. Consequently, the sample eigenvectors and eigenvalues differ from their population counterparts.
In this work, we focus on the latter issue- sampling siise--ince we assume trial averaging already reduces the additive noise manageable level.
One approach to address sampling noise is by studying the moments of the Gram matrix [28, 14]. While these methods provide way to approximate the effect of sampling the scalar values of certain similarity measures, they do not directly offer an interpretable description of what happens to the underlying eigencomponents Recent work by [41] provides bounds on representation similarity measures when the number of sampled neurons limited. However, these bounds are tight only under the assumption of a white Wishart model (i.c., all population eigenvalues are 1). For more realistic data, where cigenvalues often decay according to nower-law(primary visual cortex for mice in [49] and human visual cortex fMRI in [20]), these bounds can become too loose to be practically informative.
Instead, we directly investigate how sampling noise affects both the eigenvalues and eigenvectors of the sample Gram matrix using random matrix theory [43, 7, 8). Extensive results exist for white Wishart matrices and low-rank "spiked" models, including the Baik-Ben Arous-Péchts (BBP) phase transition [4], which reveals that sample eigenvectors often serve as poor estimators their population counterparts. These ideas have been extended to canonical correlation analysis [36, 9]. However, the

---

lower-law-like spectra observed in neural data have not yet received comparable attention. Our work attempts bridge this gap by studying sampling noise in representations with strongly decaying eigenvalues, which are ubiquitous in neural datasets.
2 Notation & Problem Setup
We use bold fonts for matrices and bracket notation for vectors We use a tilde to denote quantities related their population values.
consider two centered population activations X E PxNx and Y E RRN,, with N₂ and N, neurons, recorded in response fixed set of stimuli of size P. Centered means that we subtract the column-wise mean. Their corresponding population Gram matrices are given by Σ. = XX and Σ, = YY with eigendecomposition:
P Σ2
P Σ, = Σ Pa (waXWall
Σ
de (asl 1=1
where Ài Na and [ [ma) are respectively their eigenvalues and eigenvectors, and the eigenvectors mutually orthogonal, i.e. ūú) dij and (wa (.x)) dab.
The sample activations X E PxNN and Y E PxN, are assumed to be generated from the population ones by random projection X XR where R E NxN is random matrix with Gaussian i.i.d entries. Their Gram matrices are defined as Eg = XX and E, = YY with eigendecomposition:
Σ, Σ de (ux Σ, Σ pa wwaXwal-
(2)
Random projections serve as an effective approach for sampling nigh-dimensional data due their geometry-preservee properties [31] and are popular method in analyzing neural dynamics from limited recordings [19]. This assumption allows us to treat sample Gram matrices as structured random Wishart matrices (see SL.A.I).
Noting that both population and sample eigenvectors reside R.P. we define self-overlap matrices between sample and population eigenvectors for each representation
Q (F( [1,
2% [B( [1,)
and cross-overlap mutrices between the eigenvectors of two representations as
Min := B((us[wa))) Min := (ū.
(4)
Expectations are over different instances of neuron sampling via random projections. The cross- overlap M between two population eigenvectors is deterministic, hence does not require averaging.
2.1 Common Representational Similarity Measures
Here, we review common representational similarity measures and show that these measures can be expressed in terms of the average quantities presented above.
Canonical Correlation Analysis (CCA) is an algorithm that sequentially finds a set of orthonormal
vectors } and (b,, } for which the correlation coefficients Pa. = corr corr(Xa,, (Xa,, Yb,,) for two matrices X, Y are maximized [23]. The squared sum these coefficients gives the CCA similarity (CCA En PZ). and can be expressed in terms of the overlap matrix Mi, [5, 29]
N
(2s ( min NN N
No
N₂
Mia
CCA
(5)
min N,)
t=1
For vectors E N (alb) denotes their inner product (a T b) and laXb| their outer product (ab T).

---

CCA is sensitive to perturbations when the condition number of X or is large [21]. To enhance robustness, Singular Value CCA (SVCCA) performs CCA on the truncated singular values of X and Y [44]. In this approach, the sum of the overlap matrix M is truncated to include only the first few components. To avoid confusion, from now on, we will refer to SVCCA truncated to the top ten components for both and Y as CCA, ie (SV)CCA = To J Σ Σ Mia.
Centered Kernel Alignment (CKA) is summary statistic of whether two representations agree on the (dis)similarrty between a pair of examples based on their dot products [16]. CKA is defined as
Tr Ey and essentially measures the angle between two Gram matrices, In terms of spectral 2g y components, it can be expressed as:
P ΣΣ
A
Pa
CKA
Maa.
(6)
b= 1
Note that CKA is very similar to CCA but with additional (normalized) cigenvalue weighting. CKA will be the main focus of our work.
Representational Similarity Analysis (RSA) a popular method in neuroscience used to compare different brain regions in response to the same set of stimuli [30]. It is similar to CKA, except RSA compares pair-wise Euclidean distances instead of pair-wise inner products. Recent work has established its equivalence to CKA when RSA is combined with an extra centering step [52]. Therefore, our analyses are directly applicable to (centered-)RSA.
3 Theoretical Background
Treating Eg and Σy as random matrices deseribed in Sec. 2, we leverage results from random matrix theory [43] to compute deterministic equivalents of average CCA and CKA in the asymptotic limit. Defining fu- = P/N, and 4y P/Ng. we consider the limit P, NN, Ny - oc by keeping qu fy ~ O1).
Both similarity measures depend the cross-overlap between sample eigenvectors Mia defined in Eq. (4). Asymptotically Mia decouples as [7]
Mia = Σ Q7 Ma,Q% J.b
where the self-overlaps Qi and Q% can be computed analytically [32]. The self-overlap matrix for X can be expressed in terms of the resolvent matrix G(2) = (a - Σ)- given by:
Qú C lim Im G5 - -in),
(8)
4-+++
where C is a constant and the resolvent G(2) has a deterministic equivalent defined by the following self-consistent equation
5y
1 TrG(2).
G Gzf(2) 2 X1 (1 + (()((2) - 1)) g(2) = P
(9)
We provide a detailed derivation of these results SI.A. Here, we note that the complex function g(2) and Eq. (8) can be solved numerically (see SL.D for details).
Main result. Asymptotically, we obtain an analytical formula for sample CCA (Eq. (5)) and sample CKA (Bq. (6)) by replacing the cross-overlap matrix Mia with its deterministic equivalent (Bq. (7)).
Several remarks are in order:
While the theory for CCA and CKA should generally apply the cases where both models are sampled, henceforth, we fix one of the models to be deterministic for practical reasons. Often, neural similarity measures are applied to compare biological data with limited neuron recordings to an
IIn the original SVCCA formulation [44], components are typically retained to explain fixed proportion of variance. Our theoretical analysis applies regardless of the specific truncation criterion.

---

artificial model where the entire population is available. For example fixing model Y implies that its self-overlap Q" is just an identity matrix, hence simplifying Eq. (7) to M = Q*M.
'Te analytical formula for CCA and CKA depends only on the population quantities. However, since the self-overlap matrix Qij in Eq. (8) explicitly depends on individual eigencomponents, its deterministic equivalent specifically depends on the expected sample eigenvalue for the 1t component (EA,) and the population eigenvalue for the j"h component (Λz). The latter makes it harder to apply the theory when the population eigenvalues cannot be observed. We discuss this issue further in Sec. 4.2.
Sample Eigenvalues: Theoretical values of individual sample eigenvalues B[A.] can be predicted given the population eigenvalues by solving the following integral equation [43]
1 lim Im g(> in))
p()) di
p p())
(10)
E(A.
where p(A) is the deterministic equivalent of the empirical eigenvalue density (see SLAA.2). Computing E[A,] this way may be problematic due to numerical instabilities. Alternatively, one can exploit the fact that each single-trial eigenvalue concentrates around this mean with trial-to-tre triat fluctuations of O(1 /V /V) [43] and simply replace [[[s]] with a single-trial observation in the large P limit. We provide a detailed account of this approximation in SI.A.5.
Sample Eigenvectors:
Unlike eigenvalues, the sample eigenvectors (u, ū,)² exhibit trial-to-trial fluctuations that persist even as - dc (see SI.A.6). Still, we can compute the mean value of the overlap represented by the squared overlap Qú in Eq. (3).
BBP Phase Transition: In addition to inevitable fluctuation the sample eigenvectors, their mean behavior can still differ markedly from that of the population eigenvectors. A classic example is the Baik-Ben Arous-Péche (BBP) phase transition [4]. Consider a population Gram matrix with one large "spike" eigenvalue and the rest equal to 1. Depending whether the spike strength exceeds a critical threshold determined by P/N, the sample eigenvector associated with it can either have an (1) with the true eigenvector (localized) can be completely uncorrelated (delocalized). We depict this transition in Fig. la.
Finally, numerically test the theoretical prediction for self-overlap given by Eq. (8) on the eigenvectors of deep neural network activations. We extract layer activations from pre-trained 18 on CIFAR-10 images and subsample N neurons through random projection. In Fig. 1b, we show the self-overlap Qi for the first few eigenvectors of the layer activations and demonstrate perfect match with theory. As the number of neurons decreases, the number delocalized eigenvectors increases since fewer eigenvectors have self-overlap Qa zs 1.
Numerical Confirmation:
The effect of eigenvector delocalization' is reflected in the CKA between the sampled and population layer activations as shown in Fig. lc. The alignment completely misleading when small numbers of neurons are sampled, which poses significant problem for practical purposes.
a)
b) 10?
C) 10
Ogermeet
10
CKA
OG cah
Sole > (Laako))
Spke c (Ddlodaadd)
M- 5000
OGi (Taal
(feeer)
10a
10'
10a
10'
10a
10
Index eigenvector
Neuron size N
Figure 1: a) Illustration of eigenvector delocalization in BBP phase transition. b) Self-overlap Qi: between sample and population eigenvectors for ResNet18 activations. c) CKA between population and sample activations when N neurons are sampled. The gray- region represents the standard deviation empirical CKA across different random samplings.
'While eigenvalues also change, we show in SL.F that the dominant factor in reduced CKA is the eigenvector delocalization, especially for fast decaying spectra.
5

---

Applying Theory to Representation Similarity
4.1 Forward Problem: Impact Neuron Sampling on Similarity
In the forward problem, we assume that the population eigenvalues and eigenvectors are known. The first step is to obtain the typical sample eigenvalues by running a single-trial numerical simulation. We then move on to the eigenvectors by computing Q using Eq. (8). Finally, we calculate the overlap between the two systems, M, using Eq. (7). Having these components allows us to evaluate both CCA and CKA as functions the number neurons N.
As illustrated in Fig. 2, the theoretical predictions obtained from this eigen-decomposition match the observed CCA and CKA across different values of N. Notice that CKA decreases when the number of neurons N is reduced. As discussed above, both of these effects can be explained by the delocalization of cigenvectors.
0.60
0.52
00 0.44
0.35
0:35
0:32
Thezry Samole DKA
Theory Sumplee CCA
6:25
Nouron dize
Ncuron s2c
Figure 2: Comparison of sample vs population measures for CKA and CCA: Error bars represent empirical sample similarity and dotted lines the theoretical predictions. The black dotted line marks the true population similarity which is set close to 0.5 for both measures. Solid lines indicate inferred true similarity from samples. Sample similarity is lower due to eigenvector delocalization, while our method consistently provides a closer estimate of the true value.
4.2 Backward Problem:
Inferring Population Similarity from Limited Neurons
Just like in our earlier analysis, inferring the population representational similarity begins with estimating the eigenvalues of the underlying population. general, this is difficult because sample eigenvalues can deviate substantially from their population counterparts. Moreover, if N < P, there are N zero eigenvalues in the sample covariance matrix, further complicating the problem.
However, if we adopt parametric form, we can often achieve significant improvements in accu- racy [41]. Here, we assume a power-law spectrum of the form À i-1-?, and develop a numerical method based on random matrix theory that reliably infers the true decay rate of population eigen- values based on only the sample eigenvalues (see SI.C for detailed analysis). One can also consider more sophisticated eigenvalue models (e.g. broken pawer-law [41]) are also possible.
While it is possible to estimate the population eigenvalues for general spectra [33], they require estimating each eigenvalue individually and are hence computationally expensive. Here, we only consider the power-law spectrum because 1) we only need to estimate a single parameter and 2) we derive a closed-form expression for the population eigenvalues (see SI.C for derivation) and 3) it has been shown to be relevant for biological systems [49].
After estimating the population eigenvalues {A:). we address the eigenvectors by computing the self-overlap matrix Q using Eq. (8). Since every population eigenbasis produces the same mean self-overlap, estimated population eigenvalues {},} are sufficient find Q
Our final goal is to estimate the population cross-overlaps M, which are required to infer the true population similarity between two systems. Here, we propose constrained optimization problem to invert the forward relationship M = Q M using the estimated Q and the observed M, as shown in Alg. 1.

---

Two challenges arise in this naive approach. First, eigenvector statistics do not self-average [43], so the empirical cross-overlap M deviates from its expected value. This discrepancy can be partially mitigated by trial averaging or statistical bootstrapping. Second, the self-overlap Q is not invertible unless P & N. As result, it is impossible to recover the entire matrix M. Intuitively, only the first few eigenvectors are well-localized; the rest delocalize and lose information, so we can only reliably retrieve the corresponding columns of M.
While the constrained optimization provides point estimates of M, does not directly quantify their uncertainty. To assess the reliability of these estimates, we additionally derive confidence intervals under maximum likelihood estimation (MLE) framework. This allows us to estimate statistical uncertainty for each M jar well as for composite quantities such CKA and CCA (see SI.G).
Algorithm 1 Inferring Pop. Cross-Overlap M
Require:
{>; ) Sample eigenvalues P: # of stimuli, N: # of neurons
6: Step 2: Compute Self-overlap matrix
1:
7:
Q < function((X)) P.N)
2: M IPPx; sample cross-overlap
8: Step 3: Optimize Population Similarity
3: Step 1: Estimate Population Eigenvalues
9: Mest +- -rr argming || Q- Q-M]|r
4:
Assume power-law ansatz: X x x-11
10:
s.t. Mij E [0, 1] for Vi,j
5: Find y that best explains {M}
return Mest
4.2.1 Up to How Many Eigenvectors Can We Resolve for Given N, P?
Consider power-law spectrum, which decays relatively quickly. Under such spectrum, only the leading sample eigenvectors tend to be well-localized, as shown in Fig. 3(Left). we run the backward algorithm, we observe that for given N, P. we can reliably recover only those initial components that remain localized, as shown in Fig. 3(Right).
Practical implication. For power-law population spectra Å x i-1-2, the critical localization index scales as i i"NN ~ + VN (see SI.H). This provides simple rule of thumb: with N recorded 1+2 VN leading eigenvectors. Conversely, to stably recover vB
vs
neurons, one can reliably resolve roughly
the top i components, it is sufficient record N (1+7)2 h2 neurons.
Inferned
102
N-
Delocalized
160
Localized
Loca lizea
N=
Sample Index
Figure 3: Left: Participation ratio (P.R.) of self-overlap (1/ E, Q.). indicating the onset of eigenvec- tor delocalization, for a power-law spectrum X; i-1. For fixed N, increasing P marginally affects the leading eigenvectors. By contrast, for fixed P, increasing N makes more eigenvectors localized. Only sample eigenvectors below the black horizontal line localized (P.R. 1). Heuristically, Mio can be recovered reliably for only indices below this line. Right: Each column shows the 5-trial averaged M, the theoretical prediction of M, the inferred population overlap Mest. and the actual population overlap M. With fewer neurons N, sample eigenvectors become delocalized, causing large discrepancies. Nevertheless, our inference method successfully recovers the dominant overlaps, which are enough for global similarity measures such as CKA and CCA.
We can explicitly truncate these eigenvectors by taking a partial inverse of Q (see SI.E). However, this approach can be numerically unstable and might produce values of My outside the [o, 1] range.
7

---

Additionally, Fig. 3(Left) demonstrates that, under a power-law of the same exponent, varying P has a subtler effect on these leading indices than varying N, which significantly affects localization.
4.2.2 Why This Is Sufficient for Inferring Population Similarity
Although our denoising approach only manages to recover the leading few eigencomponents (those that remain localized), it is precisely these components that matter most for similarity measures like CKA and (SV)CCA. As shown in Fig. 2, these metrics are governed primarily by the initial eigenvalues and eigenvectors. Thus, even with very limited number of neurons, estimating those leading components is sufficient for practical purposes.
Note that for CKA (and not CCA), there an alternative approach to infer population similarity called the moment--asee estimator [22, 28, 14], which computes the similarity using unbiased statistics. This method is more suitable for estimating similarity with small datasets but lacks theoretical insight. In contrast, our approach provides an analytical framework for studying how spectral properties precisely alter the observed similarity, but it assumes sufficiently large datasets so that all biases are negligible.
5
Experiments
5.1 Synthetic Data with a Known Population Gram Matrix
We first evaluate approach synthetic dataset where the population Gram matrix is fully specified, allowing us to directly compare our estimated similarity measures against the ground-truth population values.
Fig. illustrates that our forward and backward procedures work well. In the forward approach, show that the eigencomponent- based analysis matches the empirical results closely. In the backward approach, even with an extremely limited number of neurons (N s 20), our method infers population similarity close to the actual value, despite the observed sample similarity being substantially lower.
Since the population eigenvectors are known, we can also verify how well the inferred overlaps match the true overlaps. Specifically, Fig. 3(Right) displays the top-left 10 x 10 block of each matrix: the empirical M, the theoretical M (second column), the inferred population overlap Mest (third column), and the actual population overlap M (fourth column). In this example, we set Σ, = Ey. and hence the actual population cross-overlap should be the identity matrix. However, with fewer neurons, the sample eigenvectors become more delocalized, as evident in the first column. The theoretical prediction of this phenomenon (second column) aligns closely with the empirical observation. Notably, even with severely limited neurons, our backward-inferene method recovers cross-overlap matrix Mest (third column) much closer to the true identity than the naive observed M.
5.1.1 Sampling Neurons Can Change Representation Similarity Ranking
Next, we showcase synthetic example in which sampling can lead to a reversal in the similarity rankings of models. Specifically,, we construct two models:
Model has significant overlap with the Brain its first 3 population eigenvectors.
Model has significant overlap with the Brain on the next eigenvectors.
We set the total population (SV)CCA of Model 2 to be higher than that of Model 1. However, as neurons are sampled, eigenvectors corresponding to larger indices (smaller cigenvalues) tend to delocalize more. Hence, the empirical cross-overlap M for Model 2 deteriorates faster, causing its (SV)CCA to drop more than that of Model 1. Eventually, Model overtakes Model 2 in the sample-based (SV)CCA ranking, illustrated in Fig. 4(Left).
Fig. 4(Right) presents the empirical and population cross-overlaps of the two models (each compared to the Brain). We set P = 200 and N == 30, and all population eigenvalues follow a plwer-law with exponent -1.2. Model 2's higher-dimensional overlaps delocalize more strongly, producing an apparent discrepancy that flips their observed ranking once neuron sampling is taken into account.
8

---

Emp. M
023
5.4
Model
0.44
0:22
0.42
Model 2
538
Noddl
Pop.
Figure 4: Left: Sample-basck CCA ranking flips despite Model 2 having a larger population CCA than Model 1. The decrease in Model 2's CCA is more pronounced due to its stronger reliance on higher-indexed eigenvectors, which become more delocalized with limited neuron sampling. Right: Empirical vs. population cross-overlaps for Model vs. Brain and Model vs. Brain. Here, P 200
and N 30. All three population eigenvalue spectra follow power-law with exponent -1.2. Although Model 2's true overlap is higher at the population level, it relies on higher-indexed (smaller eigenvalue) components, which delocalize more severely in the sample.
5.2 Brain Data
Finally, we apply our denoising framework to real neural recordings in the primate visual cortex, comparing them against various computational model predictions. (for experimental details see SL.D)
In Fig. 5, we illustrate a scatter plot of the representation similarity for different models compared to neural responses from V2 cortex [18, 46], given an artificially limited neuron count of N 20 out of 103 neurons. The z-axis corresponds to the observed sample CKA or CCA, while the y-axis is our inferred population measure. Observe that our inference method consistently produces higher population similarity estimates than the naive sample estimates. In particular, certain models that appear to have lower similarity (when judged by the raw, sample- base metric) can actually exhibit higher true similarity to the brain once sampling effects are taken into account.
Region: V2
Region:
9.55
Pop
0.4
Pop
6.3
0.2
6.2
comet
0:30
CCA
Figure 5: Scatter plots of observed sample similarity vs. inferred population similarity for multiple models compared to V2 cortex, using only N 20 neurons (out of larger set). (Left) CKA results; (Right) CCA results. The dotted line indicates equality. Notice that the inferred population similarity is consistently higher than the naive sample-based measure, demonstrating how limited neuron sampling can lead to underestimation of the true model-brain correspondence.
6 Conclusion and Outlook
We have presented an eigencomponent-basel analysis of how sampling a finite number of neurons affects similarity measures, including CCA and CKA. By applying methods from Random Matrix Theory, we established that this limited sampling systematically underestimates similarity because of eigenvector delocalization in the sample Gram matrices. Our framework provides:

---

Forward Analysis:
Predicting how Population eigenvalues and eigenvectors will manifest under neuron sampling, thus explaining the observed drop in similarity.
Backward Inference: Denoising algorithm to infer Population representation similarity from limited data, overcoming the biases introduced by sampling noise.
We validated our approach on both synthetic and real datasets. the synthetic experiments, where the population Gram matrices were fully known, we showed that our method reliably recovers the true population overlaps and similarity values, even in regimes with very few neurons. Importantly, we highlighted a striking effect of sampling: under certain configurations, the ranking of two models with respect to the brain can be inverted when only a limited set of neurons is recorded. In real datasets from primate visual cortex, our method consistently produced higher population similarity estimates than naive sample-based methods, underscoring that the observed decrease in similarity is largely a sampling artifact.
Moreover, for representations with power-law eigenspectra, we identified universal scaling law: the number of well-localized eigenvectors grows as the square root of the number of recorded neurons. can estimate how many principal
This N behavior offers a practical rule of thumb-researchee components can be reliably resolved for a given neuron count, or conversely, how many neurons are needed to capture a desired number of components (see SI.H). This scaling bridges theoretical predictions with experimental design, guiding how to interpret and plan neural recording studies under finite sampling constraints.
Future Directions. There are several promising avenues for extending our work. First, it would be valuable to explore more sophisticated spectral priors-suue such as broken power-law spectra- to account for multiple functional subpopulations in the data, each contributing a distinct spectral structure. Second, while we have focused sampling noise, future work should incorporate explicit models of additive noise that arise real-time neurophysiological recordings, relaxing the assumption that trial averaging eliminates most of it. Third, improved denoising methods could be developed by adopting Bayesian approaches to model the joint distribution of sample eigenvectors and population accurate recovery of the population eigenspaces. Finally, as
eigenvectors [38], thus allowing we outline in SI.B, our framework naturally extends to regression settings, where sampling-inducel distortions in eigencomponents can adversely affect regression scores, much like their impact representational similarity measures.
Overall, our results suggest that practical neuroscience studies must account for sampling-inducee eigenvector delocalization when interpreting representational similarity. By unveiling the intrinsic biases introduced by limited neuron sampling and proposing a systematic solution, we aim to provide neuroscientists and machine learning researchers with more reliable tools for comparing computational models and neural data.
7 Limitations
Our framework assumes that neural responses arise from Gaussian (linear) projections of latent population codes. This yields analytical tractability via random matrix theory, but real data can exhibit non- Gau-Sassiinn statistics, nonlincarities, and stimulus-depender covariances, which may reduce the guantitative accuracy of our estimators when higher-order dependencies dominate.
In addition, the similarity measures we study rely on specific symmetry assumptions; our method currently models rotational invariance only, ignoring other relevant symmetries (e.g., translation, scaling, permutation). It also does not yet cover dynamic reeerrelved///raectorry or nonlinear similarity metries. Extending the theory these richer classes remains crucial area for future research.
Acknowledgments and Disclosure of Funding
This work was supported by the Center for Computational Neuroscience at the Flatiron Institute of the Simons Foundation, well as by Sloan Research Fellowship and Klingenstein-Simoon Award (to S.C.). All experiments were performed on the high-performance computing cluster at the Flatiron Institute.
10

---

References
[1] Amol Aggarwal, Charles Bordenave, and Patrick Lopatto. Mobility edge for lévy matrices,
2023. URL https://arxiiy org/abs/2211 09458.
[2] Alexander Atanasov, Jacob A. Veavatone-Veth.. and Cengiz Pehlevan. Scaling and renormaliza- tion high-dimensional regression, 2024. URL http://///axvv.or//aaaaaae
00592.
[3] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, 121(27):a2311878122 2024.
[4] Jinho Baik, Gerard Ben Arous, and Sandrine Peche. Phase transition of the largest eigenvalue for non-null complex sample covariance matrices, 2004. URL https://arxii: http::://aaxv.o/p//eeee org/ abs/ math/0403022.
[5] Ake Bjorek and Gene H Golub. Numerical methods for computing angles between linear subspaces. Mathematics of computation, 27(123):579-594 1973.
[6] Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pages 1024-1034. PMLR, 2020.
[7] Joël Bun, Jean-Philippe Bouchaud, and Mare Potters. Overlaps between eigenvectors of correlated random matrices. Physical Review E, 98(5):052145, 2018
[8] Joël Bun, Jean-Philippe Bouchaud, and Mare Potters. Cleaning large correlation matrices: Tools from random matrix theory. Physics Reports, 666: 1-19,, January 2017. ISSN 0370-1573. doi: w...01//..hysrep.2200 005.
URL http: hxtp//xxxx/// doi orgooo////// 10. 1016/j whysrep. 2016 10.
[9] Anna Bykhovskaya and Vadim Gorin, Nigh-dimensional canonical correlation analysis, 2025.
URL WRR.hhttss//arxxi.orrr wwtp:/o////////////e/a 16393.
[10] Mingbo Cai, Nicolas W Schuck, Jonathan W Pillow, and Yael Niv. A bayesian method for reducing bias in neural representational similarity analysis. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016. URL http:://////r cccppparr/ooBB///eeooooooleeaeae
neurips. .dff.
[11] Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks.
Nature communications, 12(1):2914, 2021.
[12] Abdulkadir Canatar, Jenelle Feather, Albert Wakhloo, and Yeo Chung. A spectral theory of neural prediction and alignment. Advances in Neural Information Processing Systems, 36,
2024.
[13] Matteo Carandini, Jonathan B Demb, Valerio Mante, David J Tolhurst, Yang Dan, Bruno A Olshausen, Jack Gallant, and Nicole C Rust. Do we know what the early visual system does? Journal of Neuroscience, 25(46): 10077-10977771 2005.
[14] Chanwoo Chun, Yeon Chung, and Daniel D. Lee. Estimating the spectral moments of the kernel integral operator from finite sample matrices, 2024. URL https: //arxi/...ivvi/i org/ abs/ 2410. 17998.
[15] P. Cizeau and J. P. Bouchaud. Theory of lévy matrices. Phys. Rev. E, 50:11001122, Sep 1994. doi: 10. 10. ww..100/yeeee......88... URL dostps://iing aps 50. 1810.
[16] Nello Cristianini, John Shawe-Taylor, Andre Elisseeff, and Jaz Kandola. On kernel-target alignment. Advances in neural information processing systems, 14, 2001.
[17] Arthur Erdélyi. Higher transcendental functions. Higher transcendental functions, page 59, 1953.

---

[18] Jeremy Freeman, Corey M. Ziemba, David J. Heeger, Eero P. Simoncelli, and J. Anthony Movshon. A functional and perceptual signature of the second visual area primates. Nature Neuroscience, (67)),97--981, Jul 2013. ISSN 1546-1726. doi: 10.1038/mn.342. URL
https://doi.org// www.///...////......// www./////.......
[19] Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. theory of multineuronal dimensionality, dynamics and measurement. BioRtiv, page 214262. 2017.
[20] Raj Magesh Gauthaman, Brice Ménard, and Michael F. Bonner. Universal scale-free free representa- tions in human visual cortex, 2024. URL peattts://arxxe org/ abs /2409 06843..
[21] Gene H Golub and Hongyuan Zha. The canonical correlations of matrix pairs and their numerical computation. Springer, 1995.
[22] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert- norms. In Sanjay Jain, Hans Ulrich Simon, and Etsuji Tomita, editors, Algorithmic Learning Theory, pages 63-77, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg. ISBN 978-3-540-31696
[23] H Hotelling. Relations between two sets of variates. Biometrika, 1936.
[24] Arthur Jacot, Berfin Simsek, Francesco Spadaro, Clément Hongler, and Franck Gabriel. Implicit regularization of random feature models. In International Conference on Machine Learning, pages 4631 16400. PMLR, 2020.
[25] Alexander JE Kell, Daniel LK Yamins, Erica N Shook, Sam V Norman Haimanne and Josh H McDermott. A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron, 98(3):630-644, 2018
[26] Seyed-Mahdi Khaligh-Razen Razavi and Nikolaus Kriegeskorte. Deep supervised. but not unsuper- vised, models may explain it cortical representation. PLoS computational biology, 10(1)): e1003915, 2014.
[27] Antti Knowles and Jun Yin. Anisotropic local laws for random matrices. Probability Theory and Related Fields, 169:257-352. 2017.
[28] Weihao Kong and Gregory Valiant. Spectrum estimation from samples, 2017. URL https: //arxiv. org/ htse//a//16600 00061.
[29] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. International Conference on Machine Learning, pages 2519-35299 PMLR, 2019.
[30] Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, page 4, 2008
[31] Subhaneil Lahiri, Peiran Gao, and Surya Ganguli. Random projections of random manifolds. arXiv preprint arXiv:1607.0433/ 2016.
[32] Olivier Ledoit and Sandrine Péché. Eigenvectors of some large sample covariance matrix ensembles. Probability Theory and Related Fields, 151(1):233-264 2011.
[33] Olivier Ledoit and Michael Wolf. Spectrum estimation: unified framework for covariance matrix estimation and pea in large dimensions. Journal of Multivariate Analysis, 139:360-384, 2015.
[34] Olivier Ledoit and Michael Wolf. Numerical implementation of the quest function, 2016. URL https://arxte: wwwreiivv.rrr/o.o abs/abas/11/////// 05870.
[35] Grace W Lindsay. Convolutional neural networks as a model the visual system: Past, present, and future. Journal of cognitive neuroscience, 33(10):2017-2031, 2021.
12

---

[36] Zongming Ma and Fan Yang. Sample canonical correlation coefficients of high-dimensional random vectors with finite rank correlations, 2022. URL 03297.
[37] Vladimir Alexandrovich Marchenko and Leonid Andreevich Pastur. Distribution of eigenvalues for some sets of random matrices. Matematicheski Sbornik, 114(4):507-536, 1967.
[38] Rémi Monasson and Dario Villamaina. Estimating the principal components of correlation matrices from all their empirical eigenvectors. EPL (Furophysics Letters). 112(5):50001, December 2015. ISSN 1286-4854. doi: 10. 00.1009029555507/1 12/50000. URL http://dx. doi. foi.org/100 10. 1209 /0000 0000///0256607/112/0000000
[39] Alex Murphy, Joel Zylberberg, and Alona Fyshe. Correcting biased centered kernel alignment measures in biological and artificial neural networks, 2024. URL htaps:///arxiyy org/ //bbs/ 2405.01012
[40] Thomas Naselaris, Kendrick Kay, Shinji Nishimoto, and Jack Gallant. Encoding and decoding in fmri. Neuroimage, 56(2):400-410, 2011.
[41] Dean A. Pospisil and Jonathan W. Pillow. Revisiting the high- geometry of pop- ulation responses visual cortex. bioRxiv, 2024. doi: 0LL.11122244...6.656600000 URL
nttps://stt //www. biw.xiv.....oo..ooo org/ content/ wawaarl//////////222
02. 16. 580726.
[42] Dean A. Pospisil, Brett W. Larsen, Sarah E. Harvey, and Alex H. Williams. Estimating shape distances neural representations with limited samples, 2023. URL. URR.https://arxiv org// abs/ hat//2310051
[43] Marc Potters and Jean-Philippe Bouchaud. A First Course in Random Matrix Theory: for Physicists, Engineers and Data Scientists. Cambridge University Press, 2020.
[44] Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
[45] Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience, 22(11):1711-1770, 2019.
[46] Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007, 2018.
[47] Heiko H Schütt, Alexander D Kipnis, Jöm Diedrichsen, and Nikolaus Kriegeskorte. Statistical inference on representational geometries. eLife, 12:e82566, aug 2023. ISSN 2050-084X. doi: N0.7554/eLife.82566 URL httppp::/dii.orgg///// 7554/eLife 82566.
[48] Jack W Silverstein and Sang-Il Choi. Analysis of the limiting spectral distribution of large dimensional random matrices. Journal of Multivariate Analysis, 54(2):295-309 1995.
[49] Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Harris. High- digh-diimesiooall geometry of population responses in visual cortex. Nature, 571(7765): 361-365, 2019.
[50] Marcel AJ van Gerven. A primer encoding models in sensory neuroscience. Journal of Mathematical Psychology, 76:172-188 2017.
[51] Alexander Walther, Hamed Nili, Naveed Ejaz, Arjen Alink, Nikolaus Kriegeskorte, and Jörn Diedrichsen. Reliability of dissimilarity measures for multi-voxel pattern analysis. Neurolmage. 137:188-200 1001 2016.
[52] Alex H Williams. Equivalence between representational similarity analysis, centered kernel alignment, and canonical correlations analysis. In UniReps: 2nd Edition of the Workshop Unifying Representations Neural Models, 2024. URL httpss://ppenreeeee htt.eewwwwwwweeeeeeeeeeeeeeeeeee net/forum? id= id=zMdnnFasgC.
13

---

[53] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. cerformance-opimizee hierarchical models predict neural responses higher visual cortex. Proceedings of the national academy of sciences, 11 111623):8619-8622, 2014.
NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction clearly state the paper's main claims regarding how limited neuron sampling affects similarity measures (CCA and CKA). The contributions- cigencomponent-wis analysis of representational similarity, backward inference via denois- ing cigenvectors, and validation on real neural data-are outlined accurately.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer:
[Yes]
Justification:
The paper explicitly discusses limitations and future directions in the conclu- sion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
14

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
The paper provides detailed mathematical formulations and theoretical deriva- tions for each result, with clear notation established in Section 2. It includes references
Justification:
to supplementary information (SI sections) for complete proof' and additional theoretical details.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: The paper provides sufficient methodological details to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
15

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The authors provide a Github repository that allows reproducing the figures.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c cc/
public/ gww.dee/Codeeuueeegeeeee
wws.sssssiciicyylco for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /ewdde//Coeeeuumiiaa //nips. paips.cc/pie wttpc//pppppppppiieceeeeee.lppea gutgg/udee///e// /CodeSubmisseee wwssssooiiiyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper specifies experimental parameters such as sample sizes (P, N values), eigenvalue decay rates, and modeling assumptions. The figures include clear descriptions of the experimental conditions, and the text refers to supplementary information for additional experimental details.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
16

---

Answer: [Yes]
Justification: The paper reports error bars in multiple figures showing standard error across different random samplings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етгог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The supplementary materials include the full codebase along with specific information about computational resources required.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 'tnnt make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurs
Answer: [Yes]
Justification: The paper's research focus on the theoretical analysis of neural representations does not raise immediate ethical concerns.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
17

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper mainly develops theoretical framework for understanding common representational similarity measures.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
18

---

The paper cites original sources for datasets and prior work, including specific references to the primate visual cortex recordings (Freeman 2013, Schrimpf 2018) and previous research on representational similarity measures.
Justification:
Guidelines:
existing assets. The answer NA means that the paper does
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswith thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
We do not generate data train model.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
19

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: LLMs not used components of the research methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy ttttps://nuuriii http:///eeeeeeccceeeeeeeeeeeeeee /20p:///eeeeeeeee//225////////// for what should or should not be described.
20

---

A Detailed Derivation of the Main Result
A.1 The Sample Gram Matrix
Let X E RP denote the true population matrix with P samples and N₂ neurons. We consider sampling only in the neuron/feature axis. The sample data X R PxN, is obtained by applying an N₂ N₂ random projection matrix R. on X
XR., (Ra)ij N ( N₂. )
X
(S1)
The population and sample Gram matrices and their corresponding eigencomponents denoted as
=
d [..)
Σg = Xx = Σ A (us)uull
(S2)
1=1
In Random Matrix Theory (RMT), it is often convenient to consider matrices of the form M VCW /C, where W = RR T is a random Wishart matrix and C is a deterministic square matrix. We first put Σ, into this form simplify written terms of the SVD components of X -
calculations [27]. The sample Gram matrix can be
Σ = XR. = UA (v (V.RR..VV A1/UT,
(S3)
where Å E ERP, is a diagonal matrix, and U E RPx and V E RAN,,, orthogonal matrices. Since deterministic orthogonal transformations of Wishart matrices again Wishart matrices, get:
Σ, = WW
(S4)
where W_ = V VRRRRVV is random Wishart matrix with aspect ratio 62 N./N2. We divide our discussion into two cases:
When P Ñz. the eigenvalue matrix can be completed to P P-matrix by padding and replacing W. with a Wishart matrix with 9r = P/Nz. Using the orthogonality of U, this allows us express Dg
Σ. ( "UXX//""" (U)
(S5)
where W. is Wishart matrix with aspect ratio qa- P/Ng.
When P < Nz, the eigenvalue matrix and the Wishart matrix can written as
A = N o). W. =
R R
R
R,),
(S6)
where the P P matrix 7 the non-zero part of Å. and R. PXN, R₂ (NN x, are two projection matrices. Plugging these back in, we arrive at the same form the previous case.
In both cases, the statisties of Σ, does not depend explicitly on Ñ..
A.2 Eigenvalue statistics of sample Gram matrices
One of the main objectives of RMT is to understand the eigenvalue distribution of random matrices in terms of deterministic quantities [43]. Here, we review some classical results on the eigenvalue
statistics of random matrices of the form Σ / VEWVE where W is P x N Wishart matrix
21

---

with ratio P. Here, Σ and E are the sample and population Gram matrices, and they have the N. following cigendecompositions
Σ
X luXuul
= A (uXi
(S7)
1=1
We denote their (discrete-)eigenvaalle Jeigenvalue distribution by p()) and p(A):
1 P 5 8(2 Ac), v()) = P Σ
(()) =
5(( Az).
(S8)
We define the resolvent of the random matrix X and its trace as
14 1u
G(2) (a Σ)
(S9)
X
The Stieltjes transform of the empirical spectral distribution is defined as
((A) - dd = P Gr)
g (=(z)= (<))==}
(S10)
In the large P limit, this quantity is self-averaging and there is deterministic equivalent g(2) g²))) given by the self-consistent equation
(())
a(>) - 1(1 f q + gag(2))
di,
(S11)
which only depends on the deterministic eigenvalues P.. (A) and the ratio P/N [43]. In practical applications, 2= (A) is often replaced with the uniform measure over the population eigenvalues {}} as defined in Eq. (S8). This remarkable result was first obtained [37] for white Wishart matrices (for which Σ = D).
Due to the equivalence g(2) ~ >=g(2) in large P limit, these two integrals are equivalent
/ ((A) dx
P-200 - ---X1 (())
2-x
dî,
(S12)
q + gag(z))
from which one can obtain the density of the limiting spectral density using the inversion formula [8]
p(A) lim Tm Tmg(A> in))
(S13)
7-00
The Stieltjes transform also connects to the effective regularization in ridge regression [6. 24, 11, 2]. We define a new function x(2) as
x(2)
g(2) 2- -1 (2 +
(S14)
1- q q2)(2)'
and express Eq. (S11) in terms of this quantity:
x(())
x())
g
dā = - (2-1 x(z)-t).
(S15)
k<²)
Then, we obtain a new self-consistent equation for K
x(=) =-)====0 = m(z) /
qu i (d)dx
(S16)
À + x(2)
which is also known as the Silverstein equation [48]. Expressing this in terms of the discrete population eigenvalues, and evaluating it at X, we get
P
=
(S17)
which is exactly the equation for the renormalized ridge parameter in [11, 2] with the scaling
X + NA.
22

---

A.3 Eigenvector statisties of sample Gram matrices and the self-overlap matrix
This result from Eq. (S11) can also be generalized to the resolvent matrix itself [27, 8], which becomes diagonal in the population eigenbasis:
u uua
G(2) =
(S18)
X
2₂ q q=0(=))
where the integral over eigenvalues is replaced by the discrete measure over population cigenvalues. This allows to study the eigenvector statistics by analyzing the quantity
tu وun X3
(ū G(=))ū))
(S19)
2 2- 1, q + g=g(=))
In the large P limit, the sum empirical eigenvalues becomes integral:
/ (2(.2))
تūg G(z)(ūz) p000
=- p(A)d),
(S20)
where we defined Qy),, P (us (ū,)) is the overlap between the hth sample cigenvector and the jth population eigenvector. Now, we can obtain Q(),, X3) using the following inversion formula
1 lim Im (ū,/G(A, in))
Q(,,;))= xP(h:) 9-01
(S21)
Using the equivalence in Eq. (S19) and evaluating this expression explicitly:
Q(xx)))=
(S22)
3 ②) X ()A)) + [
we get an explicit formula for eigenvector overlaps [32, 8], where p(>r) is given by Eq. (S13) and (2) is its Hilbert transform:
6(2) = p.v. / (())
2 dA.
(S23)
and can be obtained from the Stieltjes transform via
lim g(2 in) = h(2) + imp(2). --+
(S24)
A.4 Overlap formula for two Gram matrices
Here, we provide a short review of the work by Bun et al. [7] which derives an overlap formula between eigenvectors from random matrices. We consider observations from two representations a X E PxN; and Y E PxN, in response to a common set of inputs of size P. Their sample Gram matrices have decompositions:
P
= =
Aa lugXual,
YY = Σ Ha WWal-
(S25)
We assume that X and are observations sampled from the underlying population features X through independent random projections. The corresponding population
I PxN, and Y I Px Ny Gram matrices are decomposed as:
P =y = Σ Paa (aaXWal
Σ. = = Σ A -
(S26)
1=1
23

---

We consider two sample data matrices X E RR &xNN, and Y RPX,, In Sec. A.I, we showed that the sample Gram matrices can be expressed in terms of the population ones as:
Σ, = Σ W. Σ... by = by En,
(S27)
where the Wishart matrices W. and w, have aspect ratios qr P/N2 and gy p/N,, respectively. Resolvents of the sample Gram matrices are
U (ui ui
P
Wa Xwal
(z)z)= = (2 Ex)
Gy(z)) (z' -Ey)
Ey)
(S28)
z' pi
We want to compute
1 サp(2,20) = E
{ Ir (G. = =E
P p2
P (u wa)
?
(S29)
):) Ma)
1,0=1
where the expectation is over random realizations of sample Gram matrices [7]. In the limit P - oc. as empirical eigenvalues become continuous, this object approaches deterministic function
2 (22222) / (p))))p(() (a A) -p) pdddd, M(),, Po) [P (uelwa)
(S30)
Here, (A),), Py(H) are the eigenvalue densities of Σg, Σy given by Eq. (S13). The function ( Ma) E P (u, (wa)² denotes the expected overlap between two eigenvectors associated with eigenvalues ); and Mas and it is the central object for our analysis since it directly appears in CCA and CKA. This quantity can be obtained by computing (A{ - in,, Pa + in/), collecting the term proportional to qpy' and taking the limit 1/ 0 [7]:
/ (A: X A (N) (Mo 1 11 (n))p)()) M(,, 1)dd dp (4+ 212 7
(), in. Ma + in))
(Ma 41) + 1//2
/
(A)
ppp((H))
dp +
(): 212 + 172 (Ha - 12) + 1p22
x² (Ppo( (Ma)) + (...)
(S31)
n.y -u
To simplify, will assume that the population eigenvectors form complete set of basis:
I
ja.iall
(S32)
Then each resolvent in Eq. (S29) can be expressed in these bases:
Gz(2)
Σ
ū Xū,
(2))
(z): (ū. (=)),,)
G2(() = Σ
T XWo, (x),,,
3% = (wa G=)=)(o)).
(S33)
a.b
where Ty and क are the matrix elements of resolvents Gz (2) and Gzy(') in their respective deterministic bases. Then, Eq. (S29) simplifies to
2 E
1
Σ ijabb
1 Σ
B [10 (S34)
P
I.j.a.b
24

---

where we defined the deterministic overlap matrix elements Cjs := (ū: wa)) In the second equality, we assumed that the two resolvents are independent, reducing the problem computing the expected resolvent of single Gram matrix.
As discussed around Eq. (S19), the resolvent G2 has limiting value for P - that is diagonal in the corresponding deterministic basis [8], and its matrix elements are given by:
dij
(2( - A, + (g,)
O(P-1/2).
(S35)
where (2) satisfies the consisttenn condition in Eq. (S11).
In order to compute the overlap M (Ai,)a)) we use Eq. (S31) and collect the term proportional to vpr.. Thanks to Eq. (S34) and Eq. (S35), this term simplifies to:
Σ 3.b
lim Im (A: ii)) C3..
lim Im D (Ma in') n1
(S36)
Defining
(b))))))= 1 lim Im $ (Ma inn)
Qa(di : TPp n-0
lim Im (,, -i))
(Pp((a)) 70
(S37)
we get an equation for M as
1 P
M(), Ma)
(S38)
j.b
Here, Qz and Q2 were already calculated in Eq. (S22). Identifying the following quantities
Qij = E (i, ū2))² =
p Qo(As, 24).
1
Qnd = E (walkir 12b)2
1
Qu(Ha.
p
Mia = (u:) wa) P M (dc.Haa) Mo), Mio == (ūrl =Cha
(S39)
we get our main result [7]:
M Q
Zodi O P [ (z) A todeSgba } + (A
quHtabi
Q¹ P P[ño(1 - (/) Pa + qyHla + P (Ha))
(S40)
A.5 Statistics of sample eigenvalues and its concentration properties
As discussed in the main text, the practical usage of Eq. (S22) requires computing the expec- tation value of individual sample eigenvalues. Eq. (S40), treating 7th biggest sample eigenvalue as deterministic and plugging = 1/VP
For sufficient conditions, we can show that the sample resolvent g(=) self-averages. In this case, the sample eigenvalue density p(A) converges in law. Here, we show that practical use Eq. (S21), Eq. (S40), we can treat 5th largest eigenvalue effectively as deterministic in its most probable position.
Specifically, we demonstrate that for large number of eigenvalues P. the most probable i-th largest cigenvalue X; satisfies
f p(A) dA P
(S41)
and that the fluctuations around this most probable is O(1/VP).
25

---

Consider set of P eigenvalues 1'1,>2,... Ap} drawn independently from the probability density P()). We order these eigenvalues in descending order:
A(1) 2 A 2 ... 2 A(P)
where A() denotes the 1t largest eigenvalue. To find the most probable value X, for the 1th largest cigenvalue, we focus on the probability that exactly cigenvalues exceed threshold Å. we define
F(A) = J x(X)
then the probability that exactly out of P samples exceed À is given by the binomial expression
F(), = (9) [F(A)] [F(() P-
We determine the threshold X that maximizes F(,, P,ú) by setting its derivative (with respect X) to zero. From this calculation, one obtains the simple condition
i P
F(A)
Equivalently, since F(A) SX p(X') dx, the most probable hhh eigenvalue X; satisfies
d(A) = 7
Now we calculate approximations for fluctuation around this most probable position. Let's analyze P,i) near λ; Write >; dλ and expand F(A) in a Taylor series about Ae
dF
dF
F(A) = F(A, EA) ~ F(A) +
5A
+
(o\)²
dA
2
dx2
Ax
Since F(A:) f. and A; is determined by maximizing F(X, P) the first derivative of F at A; vanishes:
dF
0,
dA
A
thus
F(A)
(AA))
P 2
(We expect ("())) O since F(X) decreases with X.)
Substituting this expansion back into ( [F()) (I- P-i we find that dominant depen- dence on dA appears in Gaussian-like factor
exp (
This indicates that À is peaked sharply around A; with variance
of PF
In summary, most probable i-th largest eigenvalue A(o) is determined by
d())d P
A
with fluctuation O(1/VP).
26

---

A.6 Statistics of sample eigenvalues and its concentration properties
Note that unlike eigenvalue density converges in law, eigenvector statistics Eq. (S21) noisy even when P + do [43]. In this case, we define the Q matrix as the expectation over different trials as in Eq. (S39). Equivalently, this could be obtained by averaging over small eigenvalue interval, which could be done by plugging in a small r 1/VP to extract the pole. Note that this 1/vP is also obtained by analyzing fluctuation around the most probable i-th biggest eigenvalue as above. This is essentially averaging over a Cauchy distribution centered with width 17. Thus, for practical usage of Eq. (S40), we simply plug this most likely i-th eigenvalue [8], with 7j = 1/vP.
B Relation to regression-based similarity measures
Regression Score is not representational similarity measure but is commonly used for scoring model closeness to the brain [46, 12]. Here, we discuss how our theoretical analysis for the overlap matrix M can also be applied to the regression setting. Regression score measures how well model's ' activations X predict neural responses via a linear probe. Concretely, one performs ridge regression on training subset (X 1Xipp 11p)) of size p < P. obtaining:
(p) Y 3())-
(S42)
3(p) arg min 11p8 - X1.ple areg 11311
(S43)
Then the regression score gives the neural prediction error,
Ep(p) = [K(p) - X
(S44)
X
Note that this error can be decomposed to each error mode, where Eg(p) = E, We(p) where (x(p): 2 w;
(x)-
The quantity W, denotes the projection of target labels on the hth model eigenvalue and hence can be expressed in terms of cigencomponents, W = Ej Ex A Mej. However, calculating W; assumes that there is access population-level eigenvalues and poses problem with limited data. In future work, we would like to test whether our analyses help improve the reliability of regression-ban based similarity methods.
C Theory of Power-Law Spectrum
Here, we consider the case where the population spectrum obeys power-law:
h:
d
k= P.
(S45)
where we normalized eigenvalue indices explicitly by P. For large P. the population density becomes:
- 2 8 Nz) ~
1 p /// 5(ā do)dk,
(S46)
k=1
We change the variables μ == R for which get:
8 1-/0 dk.
du = -s--s dk
(S47)
p
In the limit P + oo, the density becomes
1 1 f n -1-1// 5(ā 1a) dp = [1,00,,) 8=
(S48)
8
where we defined E [0, 1] for notational convenience. Note that, in this definition, the expectation value of diverges.
27

---

C.1 Solving the Stieltjes transform
Next, we need to solve the self consistent equation for the Stieltjes transform Eq. (S11) which reads:
+ g(2) / x(1 1-1--+ qag(2)) di, p(A)= = >
p(A)
(S49)
This integral has a closed-form solution expressed in terms of hypergeometric functions [3]. To
evaluate this integral, is convenient work in terms of the following quantities:
= 1-
g' : g'==2g(2).
w
(S50)
1 3g
Then the integral equation becomes
g' = >>k] 7-1-7
/
R u²
dd = -
du
1 u
= - BB(x + +,,
(S51)
where made change of variables == n// in the first line and used the integral definition of the incomplete Beta function
B(2;a,)) /-- u ua-1(1 ddu.
(S52)
The integral solution reported in [3] can be obtained from Eq. (S51) by using the following identity [17] in terms of hypergeometric functions:
21
B(2; a, b)
2 F1 a by 1;
(S53)
a
We rewrite the self-consistent equation by replacing the definition of
g' = - (1+ 30
1 3g' +,,
(S54)
While this equation is exact, it is possible to solve for g. To obtain analytical solution for the self-consistent equation, we need to expand the r.h.s. to leading order in g':
= y B (w; + 7, 0) + g'B7 (1 - ww BB "(w; 1 + +,,00 + +0
(S55)
which can be truncated to the linear order provided that 3g' 1. Solving for g', we get
+ 7, 0) -YW ~B
(w: "B
g'
(S56)
By
81 T-w -W (w; 1 + 74 0)
We also provide power series expansion of the incomplete beta function:
00
- + ((cot(TT) i), when w > 1 when 1,
B (w; 7,
m=
(S57)
nty w
which will be helpful when we implement these functions numerically. In terms of the power series, the solution becomes:
((Co((TT) - i) D
n-7 w
g'
1 By (T(cco(TT)) - 1)w- v Ene
w
w 1-w
(cot(TT7) - f)w + 2n0 1 1n
(S58)
1 + By (Co(((TTTT) - i)w- 7 -n=
n-y w
where we simplified the denominator in the last line using
Σn
I-w
28

---

Next, we compute the sample eigenvalue density p()) and its Hilbert transform ((A) by computing
g' (A in) A inj
lim g(A in7) 7-0
lim 7-00
b(A) + imp()))
(S59)
This is an extremely tedious calculation that we perform using Mathematica. Furthermore, we expand the results in & and, assuming < 1, keep only the linear term. In this regime, the leading order behavior of p()) and (AA) looks like:
))
P(A) = 7 1- 97
2
Σ
20 + y
x-m) + +0(0")
n=1
b(2) x-1 111 - -X-1_ 3
-x-1
1 1-
+ 2779 Ty\ 1 (cot((T)) - 1) - (7 + 1) cot(TT) + (9))
(S60)
1-7
Here, we did not include higher-order terms for (() to avoid clutter.
Finally, the formula for estimating sample eigenvalues Eq. (S41) for which obtain explicit formula:
1 x- ))
3(A.gn)) = J P(A)dX X - 97² A Σ
(S61)
Here, the semi-colon separates sample-related arguments that have empirical access to (A;, q) from the only population-relatt rellted quantity, 7. Hence, using the following relation [34, 8]
8 (A,, (:7): 9:7
(S62)
we can either predict the shape of empirical eigenvalues given the decay rate of population spectrum (forward), or infer the population decay rate given the empirical observations of eigenvalues (back- ward). Finally, we numerically test our theory and obtain perfect agreement with empirical data in Fig. SI.
16
309
Eigenvalue Av
251112227
107
12
10-4
7511.200
5011-110
2
12
300 1.
(2.402)
20-4
200 (1 1023
156 D. 1035
10-1
200 12 4013
159 12 22.4022
N 500 12.201)
N 500 12, (2.400)
N 2000 11.100)
11 1000 12.400)
10-4
A 2-4
A2 eigenvector
cigenvector
Figure S1: For population spectrum with oς - (Left) and Å 2-4 (Right), show the spectra of the empirical eigenvalues for different N. Black solid line indicates the true eigenvalue decay. The numbers in parentheses in the legend indicate the inferred true decay rate from population of N. In the regime s < 2 > 0.5), the empirical eigenvalues are always overestimated (Left), and in the regime > 2 (y 0.5) they are always underestimated (Right).
D Experimental Details
Code for all experiments is publicly available in this Github repository. All experiments were done using a single A100 GPU.
29

---

D.1
Synthetic Data
For the synthetic experiments, we generate a population activation matrix in RPxA whose Gram matrix follows chosen spectral distribution (c.g., power-law). We then form the sample activation matrix by projecting onto random subset (or random linear subspace) of size N, yielding & This procedure enables us to directly control the underlying population eigenvalues and eigenvectors, facilitating clean comparisons between sample-leeel and population- sirlarar measures.
D.2
Brain Data
We employ a set of publicly available neural recordings from primate visual cortex (e.g., V2) and compare these against the representations of various vision models, similarly the methodology in [12]. In total, we evaluate 32 models spanning supervised, self-supervised, and adversarially trained architectures, including well-known families such as ResNet, DenseNet, MobileNet, EfficientNet, and Vision Transformers. We extract intermediate-layer activations for each model on the same set of visual stimuli used in the neural recordings, applying the standard preprocessing routines (e.g., image resizing, ImageNet normalization).
Within each model, we select one more representative layers (e.g.. post-ReLU or transformer blocks). We then compute Gram matrices from those activations, matching the dimensionality of the neural dataset. In scenarios where the dataset contains more neurons than wish to analyze, project the data into a liver-dmeesiooal subspace of size N. Finally, we compute representational similarity (e.g., CKA (SV)CCA) between these model-derived Gram matrices and the neural Gram matrices, both in their raw (sample) forms and using our denoising procedure for hackward inference.
E
Another denoising method: truncated inverse
We utilize a truncated Singular Value Decomposition (SVD) to obtain regularized estimate of M:
M
VΣ-1 UM, trune
(S63)
where Q(#) UEVT is the SVD of ((a),, and Emmbe is the truncated inverse of the singular values, defined as:
if T, otherwise,
Σ
-1 Irunc
(S64)
0
F
Sample CKA with population eigenvalue term
We demonstrate that eigenvector delocalization is the dominant factor causing the decrease in sample CKA for rapidly decaying spectra such as power-law or exponential distributions.
In Fig. S2, we consider population with eigenvalues following A; i-1 and in Fig. S3, examine a population with eigenvalues following X; e These experiments confirm that while sampling affects both eigenvalues and eigenvectors, the systematic underestimation of CKA is predominantly caused by eigenvector delocalization rather than changes to the eigenvalue distribution. This holds true for both power-law and exponential eigenvalue spectra, which are common in neural data.
G Confidence Interval under Maximum Likelihood Estimation
Although it is hard to calculate the uncertainty of the estimator from constrained optimization, can compute using maximum likelihood estimation for Mja-
30

---

1.30
Norma Eigenvalus Adl
0.95
Drop due
eigenvector
deloca intoralaaiion
1011
Population
0.75
10-1 N-100
Pop. CKA.
Sample pop. elgenva
0.70
10a Neurrn size N
Eigenvalue Index ())
Figure S2: Left: Normalized sample cigenvalues for P= 100 and X i-12 with varying N. The first few terms of normalized eigenvalues remain relatively stable despite neuron sampling. Right: Sample CKA between brain and model with identical representations (true CKA = 1) as neurons sampled. The deviation is primarily due to eigenvector delocalization, shown by the close match between observed sample CKA and CKA calculated with population eigenvalues but sample eigenvectors.
10
Normalized Elgenvalue
eigenwattor
delachiization
10a
10-1
0.95
16-9
Repuration
N-10
0.34
N-21
Sample CKA Sumple OKA
10
Nod6
0.93
0.32
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
100
107 Neuron N
107
Eigenvalue Index
Figure S3: Left: Normalized sample eigenvalues for P 100 and X; e with varying N. As with the power-law case, the dominant normalized eigenvalue components remain relatively stable under sampling. Right: Sample CKA behavior with exponential eigenvalue decay. The observed sample CKA closely tracks the hybrid CKA (population eigenvalues with sample eigenvectors), confirming that eigenvector delocalization primarily drives the CKA reduction.
G.1 Problem Setup
For two sets of eigenvectors [ (unobserved) and {lua))e- (observed), estimate con- fidence intervals for Mja (u, va)
based empirical eigenvectors {la("))
) across trials
Our neuron-wise sampling model assumes:
12!0)
10 Vinluy (uj), &0 ~ iidd Σ
(S65)
j=1
where Q E PpP is fixed across trials.
Projecting onto loa) gives:
Va
(1)
o -
(S66)
The squared overlaps follow:
M 10
(vol2
22
(S67)
31

---

G.2
Confidence Intervals (CI)
G.2.1
Single-Trial Case
The negative log-likelihood for a single trial is:
Mia 0
Lo (M.
In
t0
(S68)
For confidence interval on Maa. profile likelihood:
(mpofle(m) (M 05 McM
exp (-aa(Maa) with Mr.
(S69)
The (1 -1-aa a:) CI is:
CM {m [0, 1] -2 (Lprofite (m) Lmaxx) xi.1-a]
(S70)
Algorithm 2 Profile CI for MR (Single Trial)
Input: Mia. Q. index k, level 1
2: Output: [MMower Ma Muppprr]
3: Set xi,1-a- Compute Lmax by minimizing &a in Eq. (S68). grid in [0, 1] do
4: for
5: Minimize La (M..) subject to Ma m and S Mia S (j # k).
6: Set A(m) 2 log Lprotiee (m))) Lmax).
7: end for
8: Return the smallest and largest with ((m)
G.2.2 Multiple Trials
For T trials with the same Q, the joint negative log-likellnn legelikeiihodd is:
T
P
(1)
goint (M..))
In 0
+
(S71)
0ia
1=1
The joint profile likelihood and CI follow analogously:
CT = {m E [0, 1] Ar(m) S x1,1-a)
(S72)
Algorithm Profile CI for Mra (Multiple Trials)
1: Input: (M
|i=1 Q, index k. level a, optional weights wa
2: Output: [M]ower (Mpppr _Mupper
3: Set = =xi1-a- Compute (T) by minimizing foint
4: for m on a grid in [0, do
5: Minimize fuiint (M. subject to Mr. = m and 0 S Mío s 1 $ k). L (m)/Lmn)
6: Set Ar(m) 2 log
end for
8: Return the smallest and largest m with Ar(m)= S T.
32

---

G.3 Cls for CKA and CCA
For weighted functionals like CKA or CCA:
P P
S(M)
= Σ
Σ
wyro's Mja
(S73)
jim a=
Apply the delta method:
1. Compute gradient:
ar 8M,a
wyw'
2. Estimate Fisher information: I 111
722 gloii (M)
3. Calculate variance: Var(ʃ)
V I VS
4. CI: f(M) + 21-0/2 Var(f)
1.0
0.8
0
Sample CKA
Sumple ECA
0.1
Pop. CKA
Pop. CCA
0.9
0.9
Figure S4: MLE estimation and confidence interval of trials. Here P 100. and the two population Gram matrices had power-law eigenvalues with exponent 1.2. Left: For CKA. Blue dots are the empirical sample CKA, where the orange dotted line is the theoretical line for sample CKA. The green solid line is the estimated population CKA using MLE, and the blue shades are the 95% confidence interval. Right: Same analysis but for CCA.
H Eigenvector Delocalization via Support Mergers
H.1 Setup and diagnostic
Let Σ E RxP have eigenvalues {A: K- From N i.idd. samples X = 21/2z with Zg ~N(0, 1), the sample covariance is S XX convenient diagnostic for support components is
P
B(x) = 1
(S74)
whose monotonicity between its poles x; = 1/A; tracks gaps vs. support. Intervals with (x) < 0 correspond gaps; support edges occur where the local monotonicity changes (tangency/nffeccion)) See [8] for related transform pictures.
H.2 From support mergers to eigenvector delocalization
Decreasing N is akin to increasing effective noise level: nearby spikes mix first (Dyson Brownian motion intuition). Thus leading (well-separated) spikes remain isolated, while deeper ones merge into common bulk. Empirically, the diagonal overlaps Qu stay near 1 up index i' and then drop sharply; the drop aligns with the point where a local neighborhood can no longer sustain two separated support components.
'Sign conventions vary: conclusions invariant under global sign flip.
33

---

Figure S5: Counting components via B. Blue segments on the vertical axis correspond to support intervals (adapted from Fig. 6 of [8]).
N
ds
d2
d
+
V
BN
bi
N
dBij
1
N
dv; =
N
Aj
Vj
2N
(h:
2;) )
1=1
j#i
Figure S6: Heuristic. Additive Dyson Brownian motion shown; our multiplicative setting, the same local-mixing picture applies: smaller inter-spike spacing X; A = earlier merger.
H.3 Two-peak approximation and the VN law
Assume power-law population spectrum
A - > 0,
(S75)
so the poles of B x; 1/A, i,,,, Between two consecutive poles z; and I++1
(a > 6), approximate locally
Bid (x)
(S76)
A local merger occurs when + 1 is tangent to the rational term (equivalently, equal value and slope at xx (b, a)). Solving the tangency system gives the critical sample size
1 (31++)
No
(S77)
+ 1)2+7 - 1-14 ,1+9
For N > Nr. the two local components near and Ji+ remain separated; for N < N2, they merge.
Remark H.1 (Key: V scaling of cigenvector delocalization). Using (i + 1) - -
+ (1+7) + o(2)) and (i + ()(()))) ##(+)) (1)() o(if(1+1)) Eq. (S77) yields
8
N
2 | o(1)]
(1 + >)
Hence the delocalization threshold (where Qu drops) occurs
1 1 VN. I
('(N)
V8
Interpretation: the number of population-aligner eigenvectors grows only like vN; beyond i", local support components have merged and the corresponding sample eigenvectors are mixed with the bulk. This /N law the main practical takeaway. See Fig. $7.
34

---

10a
Samplestue
Pando
Po10to Pw1354
Pwz20
Pol1000
10
Reuron Size Nin RI-31
Crttical Cclosuk
Delocalized
Theary (yl
60
Localizer zed
10
12 Sample Index
Figure S7: L.eft: PR of sample eigenvector when population eigenvalue followed power-law with exponent 1.2. Vertical lines with colors are theoretical predictions for the critical index from the two- peak approximation. The black horizontal dashed line corresponds to PR = 2. which roughly marks the index at which the eigenvector becomes delocalized. Right: Critical index i', where population eigenvalue followed power-law with exponent -1 y- Solid lines theoretical predictions from the two-peak approximation, i' I+x VN. Markers are empirical critical index when PR is 2.
v8
35


Reviews:
[{"review_id": "H5FNOW5fSE", "reviewer": "Reviewer_BZoy", "summary": "Measuring representational similarity between computational models and neural recordings has become a staple methodology in computational neuroscience. In practice, although we can access the entire population of neurons in computational models, we can record only a limited number of neurons in the brain, hoping they satisfactorily represent the larger population. In this work, the authors apply Random Matrix Theory to investigate how this sampling limitation impacts similarity measures. They find that similarity scores from popular measures are systematically underestimated under finite neuron sampling, and they propose a method to infer the population similarity. Theoretical predictions are validated on both synthetic and real datasets.", "questions": "- Fig. 2: How many sampling is done per neuron size N?\n- Why is figure 5 used before figure 4 in the text?\n- The proof that CKA or CCA are underestimated differently across models is strong enough evidence to motivate the usefulness of the proposed correction. That being said, it would have further strengthened the paper to push further the empirical investigation in section 5.2 to see if any of the previously established insights change when applying the proposed correction.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths \n- This paper tackles an important problem and contributes a practically useful methodology for the community. Representational similarity metrics are broadly used to infer good computational models of the brain, and we need to be able to trust those measures despite the practical limitations when recording neural activities.\n- The paper is well written and well illustrated.\n- The investigation carried out in the paper is thorough and convincing. \n\n### Weaknesses\n- I do not see any major weakness in this paper.", "was_revised": true, "final_justification": "I have read all the reviews and discussions. I did not have major concerns, and while some reviewers have raised important limitations (scope), the authors have convincingly addressed them. I believe this paper deserves to be accepted.", "rebuttal": "We thank the reviewer for their kind review and hope our responses addressed the reviewer’s questions.\n\n**Questions:**\n\n**1:**  We thank the reviewer for catching this. In Fig.2, we had 20 trials of sampling neurons, and we will mention it in the caption.\n\n**2.** We are sorry for the confusion. We have fixed the organization now.\n\n**3.** We thank the reviewer for this suggestion and agree that application to other data would be valuable. We are actually aiming to release a codebase that can be generalized to any dataset. We will also try to include some new experiments on additional datasets in our revised manuscript."}, {"review_id": "kcmvAmSLff", "reviewer": "Reviewer_iYBp", "summary": "This paper analyze the representation similarity in neural networks with limited neurons, which is a common scenario in neuroscience as the recorded neurons are often limited in number. The authors develop a theoretical framework based on Random Matrix Theory to analyze how finite neuron sampling systematically underestimates similarity measures like CCA and CKA. They propose a denoising method to infer population-level similarity from limited samples and validate their approach on both synthetic and real neural data.", "questions": "See the `Weaknesses` section.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1. The research question of this paper is important.\n2. The authors develop a theoretical framework based on Random Matrix Theory to analyze the effects of limited neuron sampling on similarity measures, which is a novel approach.\n\n## Weaknesses\n**Minor concerns**\n1. My primary concern is that applying CCA and CKA to analyze neural representational similarity may be fundamentally inappropriate, as these `representational metrics completely ignore the temporal dynamics of neural activity`. This limitation has been extensively discussed in recent similarity measures literature [1-3], highlighting two critical issues:\n- Different dynamical systems can yield similar static representations when sampled at discrete time points, and\n- Similar underlying dynamics can manifest as different static representations depending on sampling timing.\n\n    However, both CCA and CKA are incapable of addressing these two scenarios, making the comparison between neural recording potentially misleading.\n\n2. The scalability of the authors' approach can be constrained by its dependence on the spectral decomposition properties specific to CCA/CKA. The method cannot easily extend to:\n- Dynamic similarity metrics\n- Non-linear similarity metrics\n- Information-theoretic approaches\n  This raises concerns about the broader impact of this work: if the field is moving away from `representation similarity metrics` toward other directions above, then developing sophisticated corrections for potentially obsolete metrics may have limited long-term value for computational neuroscience.\n\n3. The assumption that neuron sampling can be modelled as `random projections` is mathematically convenient but can be a dangerous oversimplification. In practice, electrode placement is often anatomically constrained, and recorded neurons may be spatially clustered or biased toward certain sub-regions. This could violate the independence assumptions underlying the random matrix theory.\n\n**Minor issues**\n1. The notation switches between tildes and hats for population vs. estimated quantities, which could be clarified for consistency.\n\n**Reference**\n1. Ostrow, M., Eisen, A., Kozachkov, L., & Fiete, I. (2023). Beyond geometry: Comparing the temporal structure of computation in neural circuits with dynamical similarity analysis. Advances in Neural Information Processing Systems, 36, 33824-33837.\n2. Kamiya, S., Kitazono, J., & Oizumi, M. (2024, March). Koopman Operator Based Dynamical Similarity Analysis for Data-driven Quantification of Distance between Dynamics. In ICLR 2024 Workshop on Representational Alignment.\n3. Zhang, S., Ye, Z., Yan, Y., Song, Z., Wu, Y., & Wu, J. (2025). KoopSTD: Reliable Similarity Analysis between Dynamical Systems via Approximating Koopman Spectrum with Timescale Decoupling. In Forty-second International Conference on Machine Learning.", "was_revised": true, "final_justification": "I recommend borderline accept", "rebuttal": "We thank the reviewer for their thorough review. We hope our response addresses the reviewer’s concerns and look forward to discussing it further.\n\n**Weakness:** \n\n**1&2:** We first note that similarity measures are guided by the symmetries and structure present in the data. For instance, CKA and Procrustes treat two representations as similar if they are related by an orthogonal transformation. Depending on the underlying data-generating process, this may or may not be an appropriate symmetry to assume, and in some cases, alternative metrics may be more suitable.\n\nWe acknowledge that both CCA and CKA are static similarity measures and therefore do not account for temporal dynamics or nonlinear structure in the data. However, our work explicitly targets static representations, which remain widely used and scientifically relevant—for example, in studies of early visual cortex, where neurons are often modeled via their time-averaged firing rates. In such settings, the assumption of Poisson-distributed spiking and the use of rate-based representations are standard practice in neuroscience.\n\nRegarding the concern about long-term relevance and impact, while our theoretical analysis focuses on CKA, the core phenomenon we study—sampling-induced spectral bias—is not unique to CKA or even to kernel methods. Any approach that summarizes high-dimensional neural activity using operators such as covariance matrices, Gram matrices, or cross-spectral matrices will exhibit analogous finite-sample biases in their eigenspectra. Thus, we believe our framework lays the foundation for extensions to more complex or dynamic similarity metrics in future work.\n\n\n**3:** We agree that modeling neuron sampling as i.i.d. random projections is a convenient baseline but can be unsuitable when anatomical constraints induce spatial clustering and correlated activity, potentially violating the independence assumptions behind our RMT analysis. Our goal in this paper is to foreground—apparently for the first time—that neuron‑wise sampling itself reshapes the spectrum and can bias CKA/CCA; this effect has not been examined thoroughly. We now state this assumption explicitly and position our results as a baseline. Extending the analysis to structured, non‑i.i.d. sampling (e.g., spatially biased or block‑correlated selections) is an important direction, and we plan to pursue it in follow‑up work.\n\nPlease also see our response to Reviewer fmbc Q.1\n\n**Minor issues:**\n\nWe thank the reviewer for pointing out this confusion. In our notation, tildes correspond to population quantities; however, we have another quantity,  $\\hat{\\tilde M}$, which denotes the estimated value of the population $\\tilde M$ from sample quantities. We will abandon this notation and instead refer to it as $\\tilde M_{est}$.\n\nPlease also see our response to Reviewer HcPT Q.4b."}, {"review_id": "97TT5ebVGZ", "reviewer": "Reviewer_fmbc", "summary": "This paper demonstrates that in certain theoretically tractable settings, subsampling neural populations systematically underestimates neural similarity.", "questions": "1. From line 89, it appears that subsampling is modeled using a random projection matrix $R$. However, it's not immediately clear to me why this is an appropriate model for subsampling. Intuitively, I would expect subsampling to involve selecting specific units rather than mixing information across all units, as a random Gaussian projection does. Could you clarify the rationale behind this modeling choice?\n2. In the literature, one typically encounters similarity metrics which are normalized by the maximum explainable variance (i.e., similarity is measured relative to a noise ceiling). See, for example, equation 1 [here](https://arxiv.org/pdf/2502.16238). If you were to use this metric in your setting, would you still be systemically underestimating the neural similarity? \n3. There has been much recently work on similarity metrics between dynamical systems, starting with [DSA](https://arxiv.org/pdf/2306.10168) but see also [this](https://arxiv.org/abs/2410.24070) work. Do you think your theoretical insights would carry over into this setting as well? If that is too difficult to answer, do you think Procrustes-based similarity metrics also suffer from this systematic bias? It would be nice to see some discussion on this front.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "This paper tackles an important question at a good time. It is mostly well-written, and the motivation is very clear. \n\nThe main weakness is that it really helps to be a physicist when reading this paper, which I am not (I am coming from dynamical systems / control theory / ML).", "was_revised": true, "final_justification": "The authors addressed all of my concerns. I am maintaining my score of 5 (accept).", "rebuttal": "We thank the reviewer for their positive comments and for raising important concerns. We hope our responses clarify the reviewer’s questions, and we are happy to address any follow-up questions.\n\n**Strength & Weakness:**\n\nWe are aware that bracket notation is non-standard in the ML community, but we decided to use it because of how it was formulated in the original paper. We apologize for the inconvenience and will try to include a more detailed introduction to bracket notation in the appendix.\n\n**Questions:**\n\n**1:** This is an excellent question and one of the main limitations of our work. Indeed, it is not clear if this is a good model for neurophysiological recordings. We have several reasons to consider this:\n\n1. It is analytically easy to study since there is ample work on Wishart matrices.\n2. It has been debated whether neural data can be regarded as random projections. Often, neural data involves convolution and may potentially mix with each other. Also, large-area recordings such as fMRI automatically act as random projections as they involve averaging many units.\n3. Neural recordings have been found to be high-dimensional in the sense that individual neurons represent population-level information, hinting that the recorded neurons are already mixed.\n\nAlthough currently we struggle to generalize this to other sampling methods, we want to mention that this method may potentially be used to check whether the individual units are mixed or not. RMT is a powerful tool that can make precise predictions about how spectral quantities change as a function of sample size. One may employ cross-validation techniques to see if the change in the spectra by adding more neurons matches the RMT prediction and potentially informs practitioners. We leave this point to future work.\n\n**2:** We thank the reviewer for raising this insightful point.\n\nFirst, we can certainly use our method for estimating the noise ceiling by measuring the similarity between repeated trials of the same neural stimulus. Since both the noise ceiling (self-similarity) and neural similarity monotonically decrease with neuron sampling, it is not immediately clear whether their ratio still remains underestimated.  On the other hand, normalizing with the noise ceiling may also result in inflated similarity scores since it is extremely sensitive to neuron sampling.\n\nWhile we could not make this analysis due to time constraints, it will certainly be interesting to explore this problem.\n\nAdditionally, we aim to extend this theory to random projections in which the matrix elements are correlated instead of independent, as in the Wishart case.\n\nWe will discuss these points in detail in a separate **Limitations** section.\n\n**3:** We thank the reviewer for bringing this interesting point to our attention. We certainly intend to extend such RMT analysis to other similarity measures, and will provide a more detailed discussion of these extensions in the future directions section.\n\nWhile an RMT treatment of DSA itself seems difficult because of its complex nature, we would like to mention that this approach may potentially transfer to comparing delay-embedded representations with CKA when there are limited data points or the timeseries is sparsely sampled. However, we note that this still would not capture the entire non-linear nature of DSA.\n\nExtension to Procrustes-based similarity measures is a very interesting idea!  While CKA depends on the Frobenius norm of the cross-covariance, Procrustes distance relies on its nuclear norm (see Pospisil et al. [Ref a]), and we think it is reasonable to expect that a similar bias also exists in Procrustes distance.\n\nInspired by the reviewer’s question and the cited papers, we are currently conducting simple experiments on ring attractors with finite sampling and hoping to report empirical results on whether such biases exist in DSA and Procrustes.\n\n[a] Pospisil et.al. Estimating shape distances on neural representations with limited samples"}, {"review_id": "esqQxcMyOP", "reviewer": "Reviewer_HcPT", "summary": "This paper studies how sampling a subset of neurons in a large population (modeled as Gaussian projection) affects estimates of the alignment between the sampled representation and some fixed reference population. Its main result applies existing characterizations of eigenvector overlaps of Wishart matrices to describe how eigenvector delocalization induces bias in plug-in estimators of representational alignment. The authors then leverage this characterization to develop a corrected estimator.", "questions": "1. The main technical limitation I see in this paper is that the authors do not provide a full characterization of the bias of their proposed algorithm. The justification provided in 4.2.2 for why it works is heuristic in nature. It would be useful if, even just for the setting of power law data, you could give a more complete theoretical analysis of when the algorithm works. For instance, fixing a particular population decay exponent, how many components does one have to accurately estimate in order to estimate the CKA/CCA within some desired error tolerance, and when can the constrained optimization used to estimate $\\tilde{M}$ succeed in estimating that number of components? How does the choice of algorithms used to estimate the power law exponent and the overlap matrix affect the final estimate of the CKA/CCA? The lack of clarity around this issue is my main reason for recommending rejection. \n\n2. The authors mention moment-based unbiased estimators for the CKA in lines 231-235, but do not compare these methods to their proposed estimator. Adding a few such comparisons would help the reader choose a method. In particular, it would be useful to include those alternative estimators in Figure 4.\n\n3. The series of approximations made in Appendix C is not clearly motivated. Can you elaborate on why each of the assumptions made in order to render the computation tractable make sense in the context of the neuroscience data of interest? \n\n4. I have several formatting suggestions, which I think would clarify the presentation:\n\n    1. I think it would be helpful if the authors stated their main theoretical result (deterministic equivalents for CCA and CKA) as a proposition or theorem, since as written the lede is somewhat buried in Section 3 (the main result is stated in Lines 134-135 as \"Plugging in the formula for expected $M_{ia}$ in Eq. 5 and Eq. 6, we get an analytical formula for CCA and CKA, respectively\"). \n\n    2. The paper employs somewhat non-standard notation for sample and population covariance matrices, denoting them by $\\Sigma$ and $\\tilde{\\Sigma}$, respectively. Unless the authors have a good reason for not doing so, I suggest that they adopt the more standard notation of $\\hat{\\Sigma}$ and $\\Sigma$, where quantities estimated from samples are distinguished from their population counterparts by a circumflex.", "limitations": "On the whole, I think there is room for the authors to better address the limitations of their work. One point I wish they remarked upon more clearly is the fact that they do not consider the effect of random stimulus sampling; a sentence or two in the discussion would suffice. Some more prose could also be devoted to the limitations of the proposed algorithm.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "On the whole, I think that this paper is timely, and will certainly be of interest to the community of researchers working on representational alignment. In my view, its main weakness is that the connection between the theoretical results and the proposed algorithm for obtaining corrected estimates of alignment is not so strong. For instance, so far as I can tell the authors do not have a guarantee that it produces an asymptotically unbiased estimate. This is an important limitation because the authors make quite a bit of the fact that their approach comes with more theoretical insight relative to unbiased estimators based on moment computations (Line 233). Most of my questions below are focused on this point. I also have a few suggestions for how the authors could improve the clarity of their manuscript, but those are minor concerns in comparison.", "was_revised": true, "final_justification": "The authors have done a good job of addressing my concerns, and have adequately responded to the other referees. I have given a longer description of why I favor acceptance in a comment below.", "rebuttal": "We thank the reviewer for their thorough review. We hope our response addresses the reviewer’s concerns and look forward to discussing it further.\n\n**Strength and Weaknesses:**\n\n***(Theoretical Insight)*** Moment-based unbiased estimators for CKA are plug-in estimators that correct for finite-sample bias arising from both input sampling and neuron sampling. While these methods are useful in practical settings, they offer limited theoretical insight—for example, they do not theoretically reveal how CKA varies with the number of neurons or how the eigenspectrum structure influences the finite-sample CKA.\n\nOur work is complementary to these methods in that it enables an analytical understanding of representational similarity, specifically in the regime of limited neuron sampling. For example, consider recording $N$ neurons from the brain. An unbiased estimator can correct for finite-sample bias when comparing neural recordings to a model, ideally yielding a value close to the true similarity. In contrast, the random matrix theory allows one to analyze the spectral properties of the sample Gram matrix to infer, for instance, how many additional neurons are needed to resolve further eigenvectors. This type of inference is not possible with plug-in estimators.\n\n***(Asymptotic unbiasedness)*** We are grateful to the reviewer for raising this issue, which led us to improve our theoretical analysis. Here, we discuss the bias in our proposed algorithm and will provide a more detailed analysis in our manuscript.\n\nThe random matrix theory, which underlies our analysis, is valid in the limit $P, N \\to \\infty$ ($q = P/N \\in (0,\\infty)$) and is asymptotically unbiased by construction. In practice, for finite $N$ , the theoretical prediction for the self-overlap matrix $Q$ has order $1/\\sqrt{N}$ fluctuations (see Ref A). Propagating this error to the estimated CKA, we also expect a similar $\\mathcal{O}(1/\\sqrt{N})$ fluctuation. We will provide additional experiments to demonstrate this.\n\nIn summary, we emphasize again that our method provides a predictive theory of CKA and complements unbiased estimators. In practice, we believe unbiased estimators are useful, especially for very small data, and should always be compared to RMT predictions. As the reviewer rightfully pointed out, the current version lacks a comparison between two methods, and we now include unbiased estimators in our experiments (see below).\n\nIn Section 4, we will elaborate on this in more detail and also include a separate subsection discussing the differences with unbiased estimators. We are also re-running all our experiments with unbiased estimators and will include the plots in the revised version (see below).\n\n[a] Bun et al., On the overlaps between eigenvectors of correlated random matrices\n\n\n**Questions:**\n\n**1:** *(illustrative example)*\n\nConsider two identical representations whose population eigenvalues follow a power law with exponent $\\gamma = -1.2$ and $P = 100$. In this setting, the top $5 \\times 5$ block of  \n$\n\\frac{\\tilde{\\lambda_i}}{\\sqrt{\\sum_{j=1}^P \\tilde{\\lambda_j}^2}} \\cdot\n\\frac{\\tilde{\\mu_a}}{\\sqrt{\\sum_{b=1}^P \\tilde{\\mu_b}^2}} \\cdot\n\\tilde{M_{ia}}\n$\nalready accounts for about 95\\% of the CKA. While this is a simple toy case, the takeaway is that when eigenvalues decay quickly, the leading components dominate the CKA; with sufficiently many recorded neurons, these are precisely the localized components that can be estimated accurately.\n\n*(From heuristic to quantitative uncertainty.)*\n\nWe agree that the above argument is still heuristic. Rather than attempting a full bias analysis on constrained optimization, we provide confidence intervals for $\\tilde{M_{ia}}$ using a maximum likelihood estimation (MLE) approach.  \n\nFirst assume that we had the correct parametric form of the population eigenvalues, so that we estimated them accurately and thus obtained the self-overlap matrix $Q$ (as shown in Appendix C). Under a Gaussian ansatz(see Ref a where  \n$\n\\ket{u_i^{(t)}} = \\sum_{j=1}^P \\epsilon_{ij}^{(t)} \\sqrt{Q_{ij}} \\ket{\\tilde{u_j}}\n\\quad \\text{with} \\quad \\epsilon_{ij}^{(t)} \\sim \\mathcal{N}(0,1),\n$\n(see [Ref A]) \nthe squared overlaps follow $M_{ia}^{(t)} \\sim \\sigma_{ia}^2 \\chi_1^2$, where $\\sigma_{ia}^2 = \\sum_{j=1}^P Q_{ij} \\tilde{M_{ja}}$.\n\nUsing profile likelihood on the negative log-likelihood  \n$\n\\ell_a(\\tilde{M_{\\cdot a}}) = \\frac{1}{2} \\sum_{i=1}^P \\left[ \\ln \\sigma_{ia}^2 + \\frac{M_{ia}}{\\sigma_{ia}^2} \\right],\n$\nand running MLE, we obtain $(1 - \\alpha)$ confidence intervals that translate directly to CKA/CCA via the delta method. (We added an Appendix for algorithms and also comparisons between true similarity, MLE estimates, and confidence intervals.)\n\nIn practice, we used constrained optimization rather than unconstrained MLE since each matrix element lies in $[0, 1]$, which yields slightly more stable results in extremely small neuron regimes (e.g., $N = 10$).\n\n*(How many components are well localized.)*\n\nWe quantify the number of well-localized eigenvectors by analyzing when sample eigenvalue support components merge. For power-law population spectra $\\tilde{\\lambda_i} = i^{-1 - \\gamma}$, we use a local two-peak approximation of the blue function  \n$\n\\mathfrak{B}(x) = \\frac{1}{x} + \\frac{1}{N} \\sum_{i=1}^P \\frac{1}{\\frac{1}{\\tilde{\\lambda_i}} - x}\n$\n(see [Ref b]) to derive the critical neuron size $N_i^\\star$ at which adjacent components coalesce.\n\nThis yields a delocalization threshold $i^\\star \\sim \\frac{1 + \\gamma}{\\sqrt{8}} \\sqrt{N}$, implying that only $O(\\sqrt{N})$ leading eigenvectors remain reliably localized — precisely those whose diagonal overlaps $Q_{ii}$ stay near 1 before dropping sharply at $i^\\star$.\n\n\n[a] Monasson et al., *Estimating the spectrum of large covariance matrices from random subsamples*\n\n[b] Bun et al., *Cleaning large correlation matrices*\n\n\n\n\n**2:** \nWe thank the reviewer for this suggestion. We now include the CKA predictions using unbiased estimators in all our experiments. Our method almost perfectly agrees with the unbiased estimator in synthetic dataset experiments. \n\nIn Fig.4, we found that our method tends to slightly overestimate CKA compared to the unbiased estimator, but it does not change the overall picture. We will include these figures in our appendix. We now confirm this behavior also for other regions (V1, V4, IT), and will include their plots in the appendix.\n\n\n\n\n**3.:** The approximation $\\beta \\mathfrak{g}' \\ll 1$ is mainly used to obtain an analytical solution (see ref.[3]). It essentially corresponds to the large eigenvalue $\\lambda \\gg 1$ ***and*** small $q\\ll 1$ limit. \n\nTo see this, we note that $\\mathfrak{g} \\sim 1/z$ for $z\\gg1$. Since  $\\mathfrak{g}'=z\\mathfrak{g} \\sim \\mathcal{O}(1)$ remains order 1, the condition $\\beta \\mathfrak{g}' \\ll 1$ implies $\\beta \\sim q\\ll 1$. This can be also seen from Eq.S60 where the small $q$ correction to $\\rho(\\lambda)$ is given by an infinite power series in terms of $\\lambda^{-1}$, which is negligible for large eigenvalues.\n\nThe limit $\\lambda \\gg 1$ can be relevant to neural data analysis since CKA is dominated by large eigenvalues, and the limit $q\\ll 1$ is relevant when we study small deviations from the population density. Furthermore, we numerically confirm that our final formula leads to precise results for the power-law distribution, which seems to be relevant in many neuroscience experiments.\n\nWe will include a detailed explanation of the reasoning behind our assumptions in Appendix C.\n\n\n\n\n**4.: ** (a) We thank the reviewer for this suggestion. We now state our main result as a proposition. (b) We made this choice because we mainly work with sample quantities, and with the hat, the equations looked too crowded. We can certainly change the notation if it would help readability.\n\n\n\n\n**Limitations:** We thank the reviewer for pointing this issue out. Some works handle this using estimators, but an RMT treatment seems to be difficult and beyond the scope of this work. We will explicitly mention this issue in a separate Limitations section."}]

Author rebuttal:

Meta-review: This submission studies the behavior of CCA and CKA when subsampling neurons under the assumption that the subsampling happens through a Gaussian random projection of the population activity and the eigenvalues of the population Gram matrix exhibit power-law decay. It proposes an algorithm to infer the population similarity values from limited data, evaluates it in a synthetic setting, demonstrating that it can draw accurate conclusions where the naive estimator would fail, and analyzes its behavior on brain data.

Reviewers fmbc and BZoy were initially supportive of acceptance and had only minor concerns. Reviewer HcPT was initially concerned that the proposed method is heuristic and its bias is difficult to characterize. While the authors agreed that the bias is difficult to characterize, they showed that it is asymptotically unbiased and gave a method to derive confidence intervals, which the reviewer found convincing. Reviewer iYBp felt that the enterprise of measuring representational similarity was itself flawed since static representational similarity measures have limitations when applied to dynamical systems such as the brain, and the assumption of subsampling by random projections may not be accurate if the units of subsampling are actually axis-aligned neurons. Although these are both valid concerns, as Reviewer HcPT notes, these representational similarity measures are widely used and there exist problems where they do give useful insight, and the assumption of Gaussianity could still give useful results in a non-Gaussian world. Reviewer iYBp was ultimately convinced and raised their score to borderline accept.

Overall, this submission provides a novel and theoretically-justified (if still somewhat heuristic) algorithm to address bias when measuring similarity between subsampled representations. The underlying theory is interesting, and the algorithm is clearly better than naive estimation, making it likely that experimentalists will find it valuable for practical applications. I thus support its acceptance.
Decision: Accept (poster)