Estimating Dimensionality of Neural Representations from Finite Samples
Authors: Chanwoo Chun
Abdulkadir Canatar
SueYeon Chung
Daniel Lee
Venue: ICLR 2026 Poster (2026)
Keywords: Dimensionality
estimator
neuroscience
Primary area: applications to neuroscience & cognitive science

Abstract: The global dimensionality of a neural representation manifold provides rich insight into the computational process underlying both artificial and biological neural networks. However, all existing measures of global dimensionality are sensitive to the number of samples, i.e., the number of rows and columns of the sample matrix. We show that, in particular, the participation ratio of eigenvalues, a popular measure of global dimensionality, is highly biased with small sample sizes, and propose a bias-corrected estimator that is more accurate with finite samples and with noise. On synthetic data examples, we demonstrate that our estimator can recover the true known dimensionality. We apply our estimator to neural brain recordings, including calcium imaging, electrophysiological recordings, and fMRI data, and to the neural activations in a large language model, and show that our estimator is invariant to the sample size. Finally, our estimators can additionally be used to measure the local dimensionalities of curved neural manifolds by weighting the finite samples appropriately.

Published as a conference paper at ICLR 2026
ESTIMATING DIMENSIONALITY OF NEURAL REPRESEN- TATIONS FROM FINITE SAMPLES
Chanwoo Chun
Abdulkadir Canatar
SueYeon Chung ,3 Daniel Lee
1 Harvard University
2 New York University
8 Flatiron Institute
4 Coreell Tech
ABSTRACT
The global dimensionality of a neural representation manifold provides rich insight into the computational process underlying both artificial and biological neural networks. However, all existing measures of global dimensionality are sensitive to the number of samples, i.e., the number of rows and columns of the sample matrix. We show that, in particular, the participation ratio of eigenvalues, a popular measure of global dimensionality, is highly biased with small sample sizes, and propose a bias-corrected estimator that is more accurate with finite samples and with noise. On synthetic data examples, we demonstrate that our estimator can recover the true known dimensionality. We apply our estimator to neural brain recordings, including calcium imaging, electrophysiological recordings, and fMRI data, and to the neural activations large language model, and show that our estimator is invariant to the sample size. Finally, our estimators can additionally be used to measure the local dimensionalitics of curved neural manifolds by weighting the finite samples appropriately.
INTRODUCTION
How does population of a million neurons encode an input or stimulus? This question is central in neuroscience and machine learning (ML). In standard geometric view, the population response to a stimulus is vector in high-dimensional space whose axes are individual neurons' activation levels. Varying the stimulus forms a set of representations, a neural manifold, in this space. basic question in this framework is: what is the dimensionality of that manifold? Although fundamental, many dimensionality estimators are sensitive to sample size (the number of stimuli and recorded ncurons) and measurement noise. Local (intrinsic) dimensionality estimators exist that invariant to sample size, such as the TwoNN method, but they cannot measure global dimensionality. and they are often highly sensitive noise (Facco et al., 2017; Denti et al., 2022). Despite the long history and proven utility of global dimensionality in neuroscience and ML, there is no estimator that is resistant to both finite sample size and noise (Mineault et al., 2024). We aim to close this gap.
Global dimensionality, understood as an effective rank given by the number of nonzero or effectively nonzero singular values of data matrix, provides rich insights. has been used understand computation in brains and deep networks, to quantify classification (Chung et al., 2018; Cohen et al., 2020; Sorscher et al., 2022) and regression performance (Zhang, 2002: Caponnetto and De Vito, 2007; Bach, 2013), to train linear probes in artificial neural networks (Shah et al., 2025), and to design brain-computer interface (BCT) decoders (Willett et al., 2021; Menendez et al., 2025; Willsey et al., 2025). Cohen et al. (2020), Chou et al. (2025), and Sorscher et al. (2022) relate global dimensionality to the linear separability of manifolds and observe that dimensionality tends to decrease across successive processing stages in both the convolutional neural network and visual cortex, which improves linear separability. Estimating global dimensionality is also useful for interpretability research in large language models (LLMMs). For example, linear probes applied to intermediate L.M layers can reliably classify harmful versus non-harmful content (Shah et al., 2025; Kantamneni et al., 2025; Smith al., 2025), so tracking global dimensionality across layers yields important insights for AI safety and interpretability (Shah et al., 2025). In BCI, it is a standard practice to estimate
"qqual contribution.
Work done while Weill Comell Medical College.

---

Published as a conference paper at ICLR 2026
the global dimensionality of neural activations to first understand the encoding scheme of the motor cortex and utilize this insight to guide the design of BCI decoders (Willett et al., 2021; Menendez et al., 2025; Willsey et al., 2025). Beyond these applications, analyzing global dimensionality has long been staple in neuroscience, MI., genomics (Poernic et al., 2016), and behavior science (Woo al., 2023) for characterizing collective hehavior in large systems.
Global dimensionality estimated from finite data is systematically biased. In neuroscience, the observed activation matrix has shape P x Q (stimuli x neurons) and effectively a random submatrix of a much larger, unobserved matrix. One can present only a small subset of stimuli and record only a subset of neurons. Counting large singular values of the observed matrix, or of its covariance, is highly sensitive to the available numbers of rows and columns, widely noted challenge across many fields (Woo et al., 2023; Poernic et al., 2016; Mazzucato et al., 2016; Mineault et al., 2024). Common workarounds include subsampling to produce saturation curves that are checked visually as the matrix approaches the full dataset size (Woo et al., 2023), and ad-hoc extrapolation when saturation is not observed (Lehky et al., 2014).
We address this problem with a principled estimation-theoreti approach. We correct the finite- sample bias of widely used global dimensionality metric, the participation ratio (PR) of covariance cigenvalues. The PR is a soft count of nonzero singular values and is widely used in neuroscience and machine learning (Meissner- Bernardd et al., 2024; Fortunato et al., 2024; Harvey et al., 2024; Beiran et al., 2023; Niemeyer et al., 2022; Susman et al., 2021; Gao et al., 2017; Rajan et al., 2010; Kirsanov al., 2025; Schrage et al., 2024; Kuoch et al., 2024; Yerxa al., 2023; Sorscher et al., 2022; Fort Cohen et al., 2020; Harvey et al., 2024). However, just like other global dimensionality estimators, the existing PR estimators exhibit substantial finite-sample bias (Mazzucato et al., 2016; Menendez et al., 2025). Previously, there have been non-rigorous attempts at correcting the bias (Dahmen al., 2020; Pospisil and Pillow, 2024), but they require strong assumptions on distributions, only account for row sampling, and provide biased estimators of the numerator and denominator of PR. Here, we present rigorous bias-corrected estimator of PR by deriving the unbiased estimators of the numerator and denominator that require only very weak assumptions and account for both row and column sampling. We also provide an extension that removes the bias contributed by noise, performs sampling correction via importance sampling, is applicable to sparse matrices, and a variation that measures local dimensionality, which is resistant to noise, unlike the existing popular local dimensionality estimator, TwoNN (Denti et al., 2022).
et al., 2022; Mel and Ganguli. 2021;
2 DEFINITIONS AND PROBLEM SETUP
Here, we provide an informal formulation of the problem setup. A more rigorous and thorough problem formulation is presented in the Supplementary section Sec. A.
2.1 REPRESENTATION MATRIX
In neuroscientific and ML experiments, the neural representation data takes the form of matrix E RfxQ, where each row represents an input (stimulus), and each column represents a single feature (neuron). We assume that the neural activation P.a. is given by hypothetical map that maps an input z; and a parameter Wa that parameterizes the neuron:
p Ln
Φ(z,, w.,)
(1)
As it will be evident later, our dimensionality estimator is agnostic of Φ and the distributions of I;; and w.. Nonetheless, we define this generative process to establish clear mental framework.
2.2 PARTICIPATION RATIO
We are interested in measuring the dimensionality of the activation matrix in the limit where the number of neurons and inputs approach infinity. Let us denote the matrix in this limit as g(ox) e pl xQ(>>) where p(x)) Q(xc) - oo. A dimensionality measure should quantify how many non- zero eigenvalues the covariance matrix (()) qi ((<)) has. In the most strict sense, this given by the rank of ((e)), However, the rank is sensitive small eigenvalues that one might want
2

---

Published as a conference paper at ICLR 2026
to ignore. Therefore, softer count of eigenvalues, participation ratio (PR) % has become popular measure of dimensionality,, which is defined as the following:
(2,,,² 7o = Σ
where }A:} is the set of all eigenvalues of K(,), Suppose D number of the eigenvalues all take single value c, and the rest are all zero. In this case, "oo is D, which equals the matrix rank. However, there is an additional small eigenvalue e, then the PR changes only by a small amount 7o s D (e), whereas the matrix rank abruptly becomes + 1. Therefore "ro forms a lower bound on the matrix rank. It has been shown that "Yo number of the largest eigenvalues typically explain 70-80% of the variance in the data (Gao et al., 2017).
We introduce equivalent ways of expressing the PR, which will be useful in the later sections. We the fact that the sum of the eigenvalues of positive semi-definite matrix is given by its trace, and the sum of the squares of the eigenvalues is given by the trace of its square:
P K(00))
2
P tr (((x))2)
=
where
Eag 1yki = $ (
(2)
and (+) is a notation for averaging over all free indices, e.g.
ΣΣ
708 fjlr
fjlr
of summands
sjlr ag
The second equality in Eqn. 2 obtained by simply expanding the matrix notation in terms of the activations, e.g. $ (0() ia = Φ(x,, Wa).
We consider the which is a common practice in neuroscience and ML. Consider neural manifold in Q( where each point in the manifold is row vector of ((x), This centering operation simply shifts the manifold such that its center of mass is at the origin. The dimensionality y of the centered manifold given by
where each column (neuron) is centered before computing the dimensionality,
A
==
B
where
A =
2
and
B
LaB U1j0)
2 -2 ma0 (isst + (Dae
1gir
Note that we use the notation To refer the uncentered dimensionality, and y to refer to the centered dimensionality. We study for the rest of the paper. The centering above (subtracting each column's mean, i.e. centering across stimuli) gives rise to the tusk dimensionality, which measures the dimensionality of the stimulus-driven neural variance. An alternative centering of each row (subtracting the mean across neurons for each stimulus) gives the neuron dimensionality. The two task centering throughout the main paper. See Sec. A for precise
generally not equal; definitions and guidance on which to use.
2.3 SAMPLE MATRIX AND NAIVE ESTIMATOR
The sample activation matrix & E P is random submatrix of ((=)), obtained by selecting the P rows and Q columns of (=o) independently and uniformly at random, and then collecting the entries the intersections of the selected rows and columns. most neuroscience experiments, one can only observe a subset of neurons in a given brain region, which corresponds to the column sampling. In both neuroscience and MI. experiments, one can only present a subset of stimuli (e-g., it is impossible to present all possible natural images), which corresponds to the row sampling. In terms of the generative process defined in Eqn. 1, the sampling the submatrix equivalent to independently drawing P stimuli {x;}e_ and Q neuronal parameters {w, 1 from some distributions px and Pw, respectively, i.e. 24 px, and w. pw-
3

---

Published as a conference paper at ICLR 2026
In the literature, the PR is commonly estimated by simply substituting pl()) with + in Eqn. 3:
Aguive Bruive
Ynaive
where
A naive
(1 Aajj ) - 2
+
( 153r
Buaive
1585 -
(1
+ +10
and
ni9 Figkt
42 sa Pa 113.
Note that the only difference between Eqn. and Eqn. is that the Eqn. 4 is computed the submatrix 4 whereas Eqn. 3 is computed on the true matrix x(x²) However, this naive estimator is very sensitive to the number of observed neurons Q and stimuli presented P. In the next sections, we identify the source of the sample-size -ee-ssiiii-eeeiiveeeeeeeeee bias in this estimator, and then propose estimator that corrects the bias.
3 BEHAVIOR OF NAIVE ESTIMATOR
3.1 SCALING LAW OF DIMENSIONALITY OF SAMPLE MATRIX
We find that the naive PR estimate, i.e., the dimensionality of the sample matrix, scales
Eg
Ynaive
P Q
under the assumptions that the norms of the rows (and also columns) of (( are uniform and centered, and P, Q, and 7 are large (see Sec. B for details). This provides remarkably intuitive insight: "nniiv is approximately a harmonic mean of P, Q. and 7, up to a constant factor, much like the law of parallel resistance. An example generative process that can satisfy these assumptions is 6(æ, w) x Gw where G is some constant matrix with large dimensionality, æ and w are isotropic random vectors with finite variance.
3.2 BIAS NAIVE ESTIMATOR
Although the simple substitution in Ynuive is intuitive, it leads to heavily biased estimation of This because both the numerator and denominator Thaive are biased estimates the numerator and denominator of y: EE., [Anaive] j A and Exp [Baivv]] * B where Ex [-] denotes the average over the independent uniform sampling of the submatrix. fact, each term in Anaive (or Bruive) is biased
estimate of corresponding term in A (or B). For example, Es
of 118j
oa. Uijj )
which
the first terms of Anaive and A. If we write this inequality explicitly in terms of the matrix entries i Φ(x,, Wa), we have
ΣΣ o(x; Wa)oolz, wa) * Er,w [o(z, w)]] og
Exp
PaQa
where Ey,w0 [-] denotes the average over the distributions of a and w. Consider decomposing the sum in the LHS into the terms where none of the indices coincide, and the rest:
asea Es
ΣΣ ( o43
6(a,, w.) Po( wa)² 2 rest
Moving the expectation inside the sum over unequal indices, for the first term, we have Es [ø(2;, w. ) (r,,, wa) 2] for i * j and a * B. However, since the row sampling and column sampling are both independent, ((x;,)) and o(Iz..) are independent for i * j. and Φ(, w.) and o(,, wa) are also independent for a + B. Therefore, the first term factorizes to Es [0(2; )²] Es [0(2;, wg) which is simply Ez,., [ō(2,, u=) 2 the quantity we want to esti- mate. However, the "rest" term is non-zero, i.c. Es [rest] of O, contributing to the bias. Note that in the "rest" term, the indices Φ(x, wa) (2 wa)² are not all unequal, making d(I,, wa)2 and (0 2 correlated, the expectation cannot be factorized, contributing bias.
4

---

Published as a conference paper at ICLR 2026
4 BIAS-CORRECTED GLOBAL DIMENSIONALITY ESTIMATOR
Having identified the previous section that overlapping indices in a sum contribute to the bias in each term of Ansive (or ) we now know that unbiased estimators of and / can be found by simply averaging over unequal indices. Let us first define notation for averaging over unequal indices for both rows and columns:
of summands Σ Lijhh E..
of "jlr"
both
denotes a summation restricted mutually distinct values of the free indices. For example, E.e Lig excludes all summands where any pair among i, j. and coincides. When the subscript pattern
forces certain indices to be equal, as in 8/40
ag
the summation runs over the free indices only.
here Eij Then, our unbiased estimators of and B are
Aboth
oB Viijs/
-2 2
-2 (iist) both
+
a. 131r
and
buth
both
Booth
2
..8 Pistr
Finally, we define estimator for the true dimensionality simply the ratio of Anoth and Boot::
Anoth
"hooth
Buoth
Note that even if X and Y are unbiased estimators of X and Y, X// is biased estimate of X/Y. The ratio operation introduces small but inevitable bias, which cannot be reduced further a straightforward manner (see Sec. D). In general, however, the bias contributed by the ratio operation negligible compared to that contributed by biases in the numerator and denominator. We provide a detailed theoretical analysis of the bias and variance of these estimators in Sec. D.
If one desires, one can correct only the bias contributed from row sampling by summing over unequal row indices, but still summing over all column indices. This could be useful if one has full observation of neurons, but the inputs are sampled. In similar manner, one can only correct the bias contributed by column sampling. We refer these estimators "how and Yeol. respectively.
4.1 IMPLEMENTATION
The main challenge in implementation is in performing the sum unequal indices. Vectorizing or parallelizing the sum is highly non-trivial. Consider, for example, a sum of the following form, Eii Wifk for some tensor To vectorize this sum, one needs to express it in terms of regular sum that sums over all indices. After some algebra, the unequal sum can be expanded as:
Σ R.j.A Wijk
Σ Wiyk Pjk
Σ uss Σnsy - Σ uijs +
Σ Wiit
where now the vectorized calculation is possible for each sum on a computer. One can simply use the einsum subroutine for each sum. In our case, we have four row indices (.. and two column indices {a, 3), each set required to be mutually distinct, yielding six sets of constraints. This makes the expanded forms considerably longer than the 3-index example above; the complete, implementable expressions for all estimators (*both- Trow, and %ool are provided in Sec. A.
4.2 EXTENSIONS
4.2.1
NOISE CORRECTION
In many scenarios, the sample matrix is corrupted by an additive or multiplicative noise. This is inevitable in neural recordings. We show that we correct the bias from the additive or multiplicative noise (or both simultancously). as long as two sample matrices are obtained over two trials for fixed sets of stimuli and neurons. An alternative naive approach would be perform N trials
5

---

Published as a conference paper at ICLR 2026
and take element- average of the N sample matrices, before passing it the dimensionality estimator. However, performing multiple trials is typically expensive, and the bias contributed by the noise would be O (1/VN) with this naive method. In contrast, our method, inspired by Stringer et al. (2019), only requires N trials, and the bias contributed by the noise O (1/P (1//+1/Q) much more efficient than the alternative method. There, we assume the sample matrix in 1-th trial is generated by
(xI; Wa) + (7 Wa ft)
where 6 is sampled independently across trial, and [,(n(x, w, c)] 0. If ( Way ()) Col(xs wa)
and e, is zero-mean, then this models multiplicative noise: T = (1 + (e) p( Wa).
ia
of $ To correct for the bias due noise, we can simply redefine Vijkl Vijkl where (22) and ((2) are the sample matrices from two trials. The rest of the calculation can be performed as explained in the previous section obtain the dimensionality estimates. See Sec. for the formal derivation, Sec. D for its bias- variance analysis, and Sec. A.4 for a discussion of which noise structures are neutralized by this construction, and the N-trial extension for correlated noise.
of
(2)
4.2.2 IMPORTANCE SAMPLING AND LOCAL DIMENSIONALITY
In real life, the distribution of neurons Pw in a given brain region could be different from the distribution of the observed neurons goh. Similarly, the distribution of the intended set of stimuli Px may be different from that of the available set stimuli pobs. In this case, importance-sampling (IS) weights r(z) (x(x)) (2)(())))) (x) and c(w) Pw(w)/ // (w) can be incorporated into the unequal-inde sums obtain the bias-corrected estimator 18 bothh the full derivation and a concrete
V1 example are given in Sec. A.5.
There exists simpler, yet less rigorous, implementation. Suppose S = { is set of scaling factors, each of which determines how much we want to weight a given neural response. We now define weighted sum notation:
Eighr Ea, 8188818888888 08
Ailt
both
8,8,515r
03 S has two free
Each free row index contributes exactly one scaling factor. For instance, of row indices and so the numerator contains 8:8j Vijaj and the denominator normalizes by Q(Q 1) Σi, 818j. Then the weighted dimensionality is given by
Uijas both
S
S
S both
a8 1827
s85l
ijlr
S Tooth
both
both S
S
2
o. Viglr
hoth
hoth
both
One can also estimate the local dimensionality, i.e., intrinsic dimensionality, around a given point in the representation space by assigning large weights the neighboring points and small weights to distant points. In conjunction with the noise correction method explained above, our local dimension- ality estimator is resistant to the additive/mullillicatiee noise, unlike popular local dimensionality estimators (see Sec. 5).
4.2.3 ESTIMATING FROM SPARSE SAMPLE MATRIX
Sometimes, the observed entries of an underlying matrix do not form a complete matrix due to an unstructured sparsity, e.g., missing neurons for certain trials in neural recording, and a real-life user- item matrix in recommender system. If we precisely define the "# of summands" in Equation (5) as the number of summands that do not include any missing entry, Booth are unbiased estimators of A and B, even when the sample matrix is sparse. However, one should assume that the occurrence of missing entry and the sampling of w are independent.
ensure that the Aboth and

---

Published as a conference paper at ICLR 2026
4.2.4 ESTIMATING FINITE-SIZE SIZE UNDERLYING MATRIX
In some cases, the underlying matrix from which submatrix is sampled may not be infinitely large. but of finite size, R x C, where R 2 P and C' Z Q. In this case, the rows and columns of Φ are sampled without replacement from the finite underlying matrix G. This makes the row (and also column) sampling process no longer independent, requiring an alternative formulation of the estimators. We present unbiased estimators of the numerator and denominator of the PR of G in the Sec. A.6. These estimators require the knowledge of R and C.
4.3 SYNTHETIC DATA
We first verify our estimator on synthetic data by testing how quickly different estimators converge to their true dimensionality. Here, we considered the following noisy linear generative process:
T ia
I₁ Wa Czax
(6)
We sample Q feature variables Wa Jo=1 and inputs { independently from A/ Iz), and the noise term {Gaa} from (N0, o² to form a P Q sample matrix O In the limit when P, 2 - oc, the true, frese-free,
should approach d, which we refer to as the true dimensionality y for this setup. Note that finite size & obtained by the above process can be seen random, noisy submatrix of an underlying infinite matrix.
40
Yralee
Vou
30
30
Yea
20
20
10
10
102
10'
102
10a
Figure 1: Different dimensionality estimates of the linear model with d = 50 and noise variance
of 0.2. Left: varying number of stimuli P with fixed Q 100 neurons. Right: varying number of neurons Q with fixed P = 200 stimuli. The horizontal line marks the true dimensionality y 50.
We find that our estimator "hoth is able to recover the true dimensionality across wide ranges of finite P and Q (Figure 1). Note that our estimator did require any information about the distributions of x and w, or the map 6. We do observe a very small bias in our estimator when P or Q is very small, due to the nonlinear effect of taking the ratio of unbiased estimators, as described earlier in Section However, this bias is negligible compared to the bias in the other estimators.
4.4 BRAIN DATA
In real data scenarios, the true dimensionality 7 is unknown, and we can only assess the estimators' performance based how quickly they converge to the dimensionality calculated using the entire dataset. Here, we test our estimator on multiple neural datasets, all using natural image stimuli:
1. Mouse recorded with calcium imaging Stringer et al. (2019)
2. Macaque V4 recorded with microelectrode arrays (local field potential) Papale et (2025)
3. Macaque IT recorded with microelectrode arrays (spike-sorted) Majaj et (2015)
4. Human IT fMRI data Hebart et al. (2023)
By subsampling from each dataset, we vary the number of neuronal units Q or the number of natural images P, and apply the dimensionality estimators. Figure 2, we report our results as a function of the number of subsamples sampled from the full dataset.
They are either ROIs (calcium imaging), spiking units (Ephys), electrodes (Ephys), voxels (fMRI).
7

---

Published as a conference paper at ICLR 2026
V1 Calcium imaging [Stringer et al. (2019))
IT Electrodes (spike-sorted) [Majaj & Hong et al. (2015))
Qm5424
P-2800
Q-66
P- P200
15.0
15.0
00
12.5
60
60-
Yrer
10.0
10.0
18
Yeam
1000
2000
2500
5000
7500
1000
2000
3000
50
100
(Images)
(RDIs)
P (Images)
(Splking units)
V4 Electrodes (LFP) [Papale et al. (2025))
IT AMR (Hebart et al. (20231)
Q-153
P-22248
Q-2232
P-9840
18
200
16
16
180
14-
14-
160
160
12
140
140
120
120
10000 P (Images)
20000
100
200
2500
5000
7500
10000
1000
2000
3000
(Electrodess
(Images)
(Vexels)
Figure 2: Dimensionality estimates on four different neural recording datasets for varying number of stimuli P, and neural activation units Q. by subsampling from the full dataset. Top left: Mouse V1 (Stringer et al., 2019); Top right: Macaque IT (Majaj et al., 2015); Bottom left: Macaque V4 (Papale et al., 2025); Bottom right: Human IT (Hebart et al., 2023). Note that the estimators converge to different plateau values: Tnaive retains residual bias from both P and Q sampling: "row (or Yool) corrects only row (or column) sampling bias and so plateaus at an intermediate value; Tboth remains remarkably constant across P and Q variations (grey dotted line extends Yboth the max P).
Most notably, the empirical mean of Thouh is consistently least sensitive to the number of samples, compared to that of the other estimators (Figure 2). Note that when varying the number of rows P. the Trow that corrects the bias due to row sampling is practically invariant, but it is sensitive (Figure 2). The opposite is true for Yool- Finally, Thaie is sensitive to both P and Q and is the most biased. These results indicate that our estimator useful regardless of neural recording modality and can capture the underlying dimensionality with relatively much smaller number of samples.
Q
4.5 ARTIFICIAL NEURAL NETWORKS
We also apply our estimator to artificial neural networks. In this case, one has access to the entire population of neurons; hence, correcting for overlapping column indices should not make any difference. However, there may be input-limited cases where only few exemplars of a particular class can be accessed.
Here, we consider this case for evaluating the dimensionality of hidden layers of large language models (L.LM). We use the FLORES+ dataset NLLB Team et (2024) for multilingual machine translation and extract representations from the hidden layers of pretrained L1 ama3 base model Grattafiori et al. (2024). The FLORES+ dataset contains 483 sentences translated to over 200 languages. Here, we picked 9 languages (see Sec. C for details) and extracted their hidden layer representations from the LLM. Since sentences have different lengths, we use the representation of the last token of each sentence. In Fig. 3, we report the average dimensionality across all languages against the input sampling ratio.
Since fix the number of neurons and subsample the inputs only, expect column correction to have a tiny effect compared to row correction. Fig. 3a, we indeed observe that Yool performs as poorly as the naive estimator and that Vrow performs well as estimator.
Next, we show how the average dimensionality changes from layer to layer in Fig. 3b when the sample size is small. While the naive estimator significantly underestimates the dimensionality,
8

---

Published as a conference paper at ICLR 2026
Dimensionality
35
30
Tisalee
25-
Troos
Took
Toosh
0-
10-1
10"
0.00
0.25
0.50
0.75
1.00
P Sampling Ratio
Layer Depth Ratio
Figure 3: Estimating the task dimensionality of LLM features for different languages. a) We calculate the dimensionality the last layer for each language separately and report its average as a function of the input sampling ratio. In this example, all layers have Q 4096 dimensional representations, and each language has a total of = 483 sentences. The error bars represent the standard deviation for 50 random draws. b) The dimensionality profile across layers when the sampling ratio is 0.1.
preserves the overall profile of dimensionality across layers. However, our estimator reveals more fine-gre rreeggained features of the layerwise dimensionality that are hidden otherwise. We provide additional details about experiments in Sec. C.
Finally, we would like to comment on the interesting behavior in Fig. 3b, where the dimensionality increases towards the mid-layers and decreases again. This behavior was observed previously Valeriani et al. (2023) and was recently reported in Skean et al. (2025) by using matrix-bass entropy measures Giraldo et al. (2014), which includes the logarithm of the dimensionality special case.
EXTENSION: LOCAL DIMENSIONALITY ESTIMATION
In the earlier section, we described the procedure of weighing samples based on importance. Here we demonstrate that one can extend this framework to measure local dimensionality of manifold, adopting a method inspired by Recanatesi et al. (2022). From given data point T (an arbitrary row vector of (), one can measure the distance to the rest of the data points in a manifold dataset. One can then discard the data points that are further than some predefined distance r, giving them zero weights and giving the points inside the local ball radius of uniform weights. The resulting weighted dimensionality estimate is denoted Thoth To.r). One can obtain the average local dimensionality
of given manifold by taking the average of Tooth
To.-7 r)
over all available o 's:
local boohh
(r)
Σ
local Tooth
where Pi:: denotes ith row vector of Φ. In our experiments, we use the Mahalanobis distance with a local metric. See Sec. E for formal definition of the local dimensionality estimator and distance metric. One can similarly define >ylccu (r). 7oocall (r), and 1 (r).
To confirm that our estimator can indeed capture the true local dimensionality, we test on synthetic dataset. The synthetic dataset is created using a random Fourier feature (RFF) model whose covariance kernel converges to the radial basis function (RBF) kernel the limit where the number of features (neurons) approaches infinity. A noisy RFF model is defined as
o(x, {w, b]) sin(æ + b)
where E IRd is sampled independently from normal distribution / (0, Ia), b is sampled inde- pendently from uniform distribution 24 (-T/2,
20 π 1/2). The data matrix & e IR PxQ is created by presenting number of inputs (x's) sampled from A (0, o2 Q number of random features, e is the noise term. We emphasize
such that for trial t, (t)) o(x;, {wa, ba)) + Cj where G,a,, that the noise term G, is sampled independently across all and t from / (0, of). In the limit of infinite P and Q, the true noise-free local dimensionality of the Q-dimensional representation
manifold should approach d.

---

Published as a conference paper at ICLR 2026
a.
b. V1 Electrodes (LFP) [Papale et al. (2025))
Random Fourier features
TwcNN
15-
Local dimansionallyy
Local dimensionaiiss
10-
5-
10
1.2
1.4
1.6
1.1
1.4
Local
Figure 4: a. Estimating the local dimensionality of the random Fourier feature model using TwoNN, local (r), and local (r), while varying the radius of the local ball for the latter two estimators. Signal- to-noise ratio is approximately 3.33 (a, 0.3). b. Estimating the local dimensionality of the macaque VI LFP measured with electrode arrays (Papale et al., 2025).
We find that our local dimensionality measure Jocal (r) recovers the true local dimensionality 7ooch
despite the presence of noise decrease the local ball radius to the smallest allowable length (Figure 4). The smallest radius determined by requiring that at least four data points lie inside a local ball. Recall that our estimator requires at least four distinct indices, so four the minimum allowed dataset size. In contrast, the widely adopted local dimensionality estimator TwoNN significantly overestimates the local dimensionality due its susceptibility to noise. The naive local estimator maive
local (r) also fails to recover the true dimensionality, since it underestimates when there are only
local ball. This highlights the need to correct for bias due to the small sample size, especially when measuring local dimensionality.
small number of samples
We also apply the local dimensionality estimators the real brain dataset, V1 LFP measurements from macaque (Papale et al., 2025). We find that the estimate from our estimator Jocal (r) in the small radius limit is much smaller than the TwoNN estimate (Figure 4).
DISCUSSIONS
In this paper, resolve the timely issue of the sensitivity of the global dimensionality estimator to the sample size and measurement noise. this end, we correct the bias in the PR, a popular synthetic dataset, real neuronal
measure of global dimensionality. We then apply our estimators datasets of various recording modalities, and a large language model representation, to show that our estimator is indeed resistant to the sample size. As extensions, also show how local dimensionality using our dimensionality estimators.
measure
The PR characterizes the eigenvalue spectrum based the first and second order statistics of the spectrum. To recover more information about the spectrum, one can estimate the higher spectral moments using the same basic principles employed here (Chun et al., 2025).
7 CODE AVAILABILITY
We provide our http::////ttttto////////////aaaa
in
LIMITATIONS
For measuring local dimensionality, estimator Jocal (r) requires computing pairwise distances. which dominates at O(P²Q) time and O(P2 + PQ) memory; sweeping over r radii brings the total to O(rP2Q) time and OrP + Q)) memory. This is more expensive than TwoNN, though the computation can easily parallelized over multiple threads. The global estimator Taah shares the same asymptotic time complexity the naive estimator: O P, Q)² max (P, Q))-
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank Jonathan D. Victor for the discussion. Partial support for this work was provided by the Simons Foundation. DDL was also supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean Government (MSIT) (No. RS- 2024-00457882, National AI Research Lab Project). S.C. was partially supported by Sloan Research Fellowship. a Klingenstein-Simen Simons Award, and the Samsung Advanced Institute of Technology project, "Next Generation Deep Learning: From Pattern Recognition to AI." All experiments were performed using the Flatiron Institute's high-performance computing cluster.
REFERENCES
Francis Bach. Sharp analysis of low-rank kernel matrix approximations. In Conference on learning theory, pages 185-209. PMLR, 2013.
Manuel Beiran, Nicolas Meirhaeghe, Hansem Sohn, Mehrdad Jazayeri, and Srdjan Ostojic. Parametric control of flexible timing through low-dimensional neural manifolds. Neuron, 11165::39-773.e8, 2023. ISSN 0896-6273. doi: http://loo.oy///...vvveeeeooo222 URL https: /www. sciencedirect com/ /ttpa//c//aattccc//////666223332
Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computationnal Muthematics, 7(3):331-368, 2007.
Chi-Ning Chou, Royoung Kim, Luke A. Arend, Yao Yuan Yang, Brett D. Mensh, Won Mok Shim, Matthew G. Perich, and Yuo Chung. Geometry linked to untangling efficiency reveals structure and computation in neural populations. bioRxiv, 2025. doi: 10. 10.1100//024....6.. 582157. URL https: http:/wwww.o.rooooo www.xiv.ivvvv.......v.... wttp:/wwwwwwwwww/..o.o/ooo wttgg//onneee//erlll/oo//ee 02.26. 582157.
Chanwoo Chun, Yeon Chung, and Daniel Lee. Estimating the spectral moments of the kernel integral operator from finite sample matrices. In The 28th International Conference Artificial Intelligence and Statistics, 2025.
Yeon Chung, Daniel D Lee, and Haim Sompolinsky. Classification and geometry of general perceptual manifolds. Physical Review X, 8(3):031003, 2018.
Uri Cohen, Yeon Chung, Daniel Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11(1):746, 2020.
David Dahmen, Stefano Recanatesi, Gabriel K Ocker, Xiaoxuan Jia, Moritz Helias, and Eric Shea- Brown. Strong coupling and local control of dimensionality across brain areas. Biorxiv, pages 2020-11, 2020.
Francesco Denti, Diego Doimo, Alessandro Laio, and Antonietta Mira. The generalized ratios intrinsic dimension estimator. Scientific Reports, 12(1):20005, 2022.
Flena Facco, Maria d'Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic di- mension of datasets by minimal neighborhood information. Scientific reports, 7(1):12140, 2017.
Stanislav Fort, Ekin Dogus Cubuk. Surya Ganguli, and Samuel S. Schoenholz. What does deep neural network confidently perceive? the effective dimension of high certainty class manifolds and their low confidence boundaries. ArXiv, abs/2210.05546, 2022. URL https://apt //api. semanticschole wttp:oopoo///cceCCC..... 22:288153::22222525500
Cátia Fortunato, Jorge Bennasar- Vázquez, Junchol Park, Joanna C. Chang, Lee E. Miller, Joshua T. Dudman, Matthew Perich, and Juan Gallego. Nonlinear manifolds underlie neural population activity during behaviour. bioRxiv, 2024. doi: 10.1101/2023.07. 17..1.98.455...
Peiran Gao, Eric Trautmann, Byron Yu, Gopal Santhanam, Stephen Ryu, Krishna Shenoy, and Surya Ganguli. A theory of multineuronal dimensionality. dynamics and measurement. bioRxiv, 2017. doi: 10.1 10.1101/214262
11

---

Published as a conference paper at ICLR 2026
Luis Gonzalo Sanchez Giraldo, Murali Rao, and Jose C Principe. Measures of entropy from data using infinitely divisible kernels. IEEE Transactions on Information Theory, 61(1):535-555) 548, 2014.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint arXi::2407.21785 2024.
Sarah E. Harvey, David Lipshutz, and Alex H. Williams. What representational similarity measures wwtp:/www........o/oo//e///be 08117.089.
imply about decodable information, 2024. URL
Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkam, and Chris Baker. Things-data, multimodal collection of large-scale datasets for investigating object representations in human brain and behavior. el.ife, 12:e82580, feb 2023. ISSN 2050-084X. doi: 10.7554/eLife.8258.
Subhash Kantamneni, Joshua Engels, Senthooran Rajamanoharan, Max Tegmark, and Neel Nanda. Are sparse autoencoders useful? case study in sparse probing, arXiv preprint urXiv:2502.16681 2025.
Artem Kirsanov, Chi-Ning Chou, Kyunghyun Cho, and Suc Chung. The geometry of prompt- ing: Unveiling distinct mechanisms of task adaptation in language models. arXiv preprint arXiv:2502.0800) 08000, 2025.
Michael Kuoch, Chi-Ning Chou, Nikhil Parthasarathy, Joel Dapello, James J DiCarlo, Haim Sompolin- sky, and Sue Yeon Chung. Probing biological and artificial neural networks with dask-dependent neural manifolds. In Conference on Parsimony and Learning, pages 395 418. PMLR, 2024.
Sidney R Lehky, Roozbeh Kiani, Hossein Esteky, and Keiji Tanaka. Dimensionality of object representations in monkey inferotemporal cortex. Neural computation, 26(10):2135-216,, 2014.
Najib Majaj, Ha Hong, Ethan A Solomon, and James J DiCarlo. Simple learned weighted sums of inferior temporal neuronal firing rates accurately predict human core object recognition performance. Journal of Neuroscience, 13402):13402-1344 2015.
Luca Mazzucato, Alfredo Fontanini, and Giancarlo La Camera. Stimuli reduce the dimensionality of cortical activity. Frontiers systems neuroscience, 10:11, 2016.
Claire Meissner- Bernard, Friedemann Zenke, and Rainer W Friedrich. Geometry and dynamics of representations in precisely balanced memory network related to olfactory cortex. eLife, December 2024. doi: 10.7554/elife.66333.22 URL /tttp://ddk doi. org.oo....///............. 96303.2. 2.
Gabriel Mel and Surya Ganguli. A theory of high dimensional regression with arbitrary correlations between input features and target functions: sample complexity, multiple descent curves and hierarchy of phase transitions. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7578-7587, PMLR, 18-24 Jul 2021.
Jorge Aurelio Menendez, Jay A Hennig, Matthew D Golub, Emily R Oby, Patrick T Sadtler, Aaron P Batista, Steven M Chase, Byron M Yu, and Peter E Latham. A theory of brain-computer interface learning via low-dimensional control. bioRxiv, pages 2024-04, 2025.
Patrick Mineault, Niccolò Zanichelli, Joanne Zichen Peng, Anton Arkhipov, Eli Bingham, Julian Jara-Ettinger, Emily Mackevicius, Adam Marblestone, Marcelo Mattar, Andrew Payne, al. Neuroai for ai safety. arXiv preprint arXiv:2411.18526 2024.
James E Niemeyer, Poornima Gadamsetty, Chanwoo Chun, Sherika Sylvester, Jacob P Lucas, Hongtao Ma. Theodore Schwartz, and Emre RF Aksay. Seizures initiate in zones of relative hyperexcitation in zebrafish epilepsy model. Brain, 145(7)2344-2266), 2022.
12

---

Published as a conference paper at ICLR 2026
NLLB Team, Marta R. Costa-jussa, James Cross, Onur Celebi, Maha Elbayad, Kenneth Heafield, Kevin Hefferman, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao. Vedanuj Goswami, Francisco Guzman, Philipp Kochn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. Scaling neural machine translation to 200 languages. Nature, (0:8801888111-11 doi org.orgggoooooo /11.///..000 10. 1038/ s41586 024 07335-x.
2024. ISSN 1476-4687. URL
https: //oii.//oi//////
Paolo Papale, Feng Wang, Matthew W Self, and Pieter R Roelfsema. An extensive dataset of spiking activity to reveal the syntax of the ventral stream. Neuron, 2025.
Ivan Pocrnic, Daniela AL Lourenco, Yutaka Masuda, Andres Legarra, and Ignacy Misztal. The dimensionality genomic information and its effect on genomic prediction. Genetics, 203(1):
573-581, 2016.
Dean A Pospisil and Jonathan W Pillow. Revisiting the high-dimensional geometry of population responses in visual cortex. bioRxiv, pages 2024-02, 2024.
Kanaka Rajan, L Abbott, and Haim Sompolinsky. Inferring stimulus selectivity from the spatial structure of neural network dynamics. J. Lafferty, C. Williams, J. Baylor, R. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems, volume 23, Curran Asso- ciates, Inc., 2010. URL https: http:::::////cpeppeeeeeoe 2010/ f e 9b httpa/////26600000022aaa11666622
newrips cc wwcppppeeraaeeeeeeeeeeeeeeeeeeee filess eelles/ /pssprr////// pdf.
Stefano Recanatesi, Serena Bradde, Vijay Balasubramanian, Nicholas
Steinmetz, and Eric Shea-Brown. A scale-dependent measure of system dimensionality. Patterns, 3(8):100555 2022. ISSN 2666-3899, doi: http:///teet.cooeeeeeeeaeeaaa//a
URL https: 60x.
//www. www.::sseeeneeireet........ com/
Linden Schrage, Kazuki Irie, and Haim Sompolinsky. Neural representational geometry of concepts in large language models. In NeurlPS 2024 Workshop on Symmetry and Geometry in Neural Representations, 2024.
McNair Shah, Saleena Angeline, Adhitya Rajendra Kumar, Naitik Chheda, Kevin Zhu, Vasu Sharma, Sean O' O'Brien, and Will Cai. The geometry of harmfulness in Iims through subconcept probing. arXiv preprint arXiv:2507.2114), 2025.
Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations language models. arXiv preprint arXiv:2502.00 02001.1........... 2025.
Lewis Smith, Senthooran Rajamanoharan, Arthur Conmy, Callum McDougall, János Kramár, Tom Lieberum, Rohin Shah, and Neel Nanda. Negative results for saes on downstream tasks and depri- oritising sae research (gdm mech interp team progress update# 2), march 2025. URL https://www. lesswrong. hwmp/oosssuxcAAMMPPPttBBBBiiese forr wtto--saeessoo-doossseaaa---eeee Less- Wrong post, 2025.
Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. Neural representational geometry under- lies few-shot concept learning. Proceedings of the National Academy of Sciences, 119(43): e2200800119, 2022.
Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D Harris. High-dimensional geometry population responses in visual cortex. Nature, 37166665):311335, 2019.
Lee Susman, Francesca Mastrogiuseppe, Naama Brenner, and Omri Barak. Quality of internal representation shapes learning performance in feedback neural networks. Phys. Rev. Res., 3: 013176, Feb 2021. doi: 10.11039hhys
URL https::/1iit aps.
13

---

Published as a conference paper at ICLR 2026
Lucrezia Valeriani, Diego Doimo, Francesca Cuturello, Alessandro Laio, Alessio Ansuini, and Alberto Cazzaniga. The geometry of hidden representations of large transformer models. Advances in Neural Information Processing Systems, 11551234-12222 2023.
Francis R Willett, Donald T Avansino, Leigh Hochberg, Jaimie M Henderson, and Krishna V Shenoy. High-performance brain-to-text communication via handwriting. Nature, 593(7858): 249 254, 2021.
Matthew S Willsey, Nishal Shah, Donald T Avansino, Nick V Hahn, Ryan M Jamiolkowski, Foram В Kamdar, Leigh R Hochberg, Francis R Willett, and Jaimie M Henderson. A high- performance brain-computer interface finger decoding and quadcopter game control in an individual with paralysis. Nature Medicine, 31(1):96-100) 104, 2025.
Theodosia Woo, Xitong Liang. Dominic A Evans, Olivier Fernandez, Friedrich Kretschmer, Sam Reiter, and Gilles Laurent. The dynamics of pattern matching in camouflaging cuttlefish. Nature, 619(7968):122-115 2023.
Thomas Yerxa, Yilun Kuang, Eero Simoncelli, and Yue Chung. Learning efficient coding of natural images with maximum manifold capacity representations. In Advances in Neural Information Processing Systems, volume 36, 2023.
Tong Zhang. Effective dimension and generalization kernel learning. Advances in Neural Information Processing Systems, 15, 2002.
14

---

Published as a conference paper at ICLR 2026
A DERIVATION OF BIAS CORRECTED ESTIMATORS OF DIMENSIONALITY
A.1 DEFINITIONS
KERNEL INTEGRAL OPERATOR
We assume there are latent variables, x and w, associated with stimulus and neuron, respectively. To model the sampling of stimulus, assume there is distribution px the set X' of all stimulus latent variables. Similarly, to model the sampling neurons, we assume there is distribution fw over the set W of all neuron latent variables. Then, assume the entry of the sample matrix is given by a map o X' x W-
t0 ( wa).
(S1)
We let P and Q denote the number of sampled stimuli and neurons in this paper, i.e. + R PxQ. Assume @ is
with respect to px and pw. Now, define the associated kernel functions
k(x,x')= (pw(ww) ((x, w)o(2', (ol((/'w))) w), and
(S2)
k(w, w') = / dpx((x) ó(x, w) w').
(S3)
There is kernel integral operator that is associated to k. Tc (px, + (2x, (t, defined as
Tw = f dpx(o) ((,
(S4)
Ty is trace operator and is analogous to the population covariance matrix Kan introduced earlier. The tuple (ø, px, Pw) uniquely defines a generative process of data.
PARTICIPATION RATIO
The eigenvalues {1 of Th are implicitly defined by
The; Xie Vi
(S5)
where e; (2 X'), and ) forms an orthonormal set. There countably infinite number of eigenvalues. Then, the participation ratio of these eigenvalues is defined as
((dpx())(xx,2)) 7o = (E x,) = [d v)²
(S6)
We refer to as the true effective dimensionality of Ts. If n number of X take a constant value and the rest are O, then the participation ratio is n, matching the definition of rank. However, if decrease one of the positive cigenvalues to be smaller than c (but still positive), then the participation ratio is between n and n. reflecting the fact that the dimensionality is effectively slightly less than n. Therefore, the participation ratio forms a lower bound on the rank, and is "softer" than the rank.
Typically, the participation ratio is computed on the eigenvalues (n) } of a sample covariance matrix K = q T
(E, 1h: tr ( E,
tr (K)2
=
(S7)
A.2 CENTERED KERNEL
Suppose ko(x, y) a centered kernel, where activations for each neuron is centered:
k,( = dpp(w) (x, (v.
where
6x w) ò(z, w) dpx(2) b(z,w)
15

---

Published as a conference paper at ICLR 2026
Similarly, (e, y) is another centered kernel, where activations for each stimulus is centered:
kc (2 xz) =
where
o5(2, w) 010,
-(,) w) - (pw(()
Before centering, did not matter whether we computed the dimensionality with ie
( (ddw(()) k(w, w)
(S dpx(()) k(x,x))² (dxx(x) (v) k(z,))² J ((pp(w) k(w, u))
?
Not only the ratio, but also the numerators and denominators of the two expressions match respectively. However, the centered dimensionalities are different
? lask + neuron
where
(J (ddw(w)) ke(w, w)) dpw(w) (u) koww, u)?
( I (ex hi (2,
fnsk
and
neuron
Guidance on centering. The two centered dimensionalities measure conceptually different quanti- ties. Task dimensionality task (row centering via k.)) subtracts from each neuron's 's response its mean across stimuli, thereby measuring the dimensionality of the stimulus-drive variance in neural space. This is the default throughout the main paper and is the natural choice when the scientific question concerns how the stimulus set is coded. Neuron dimensionality 7 nouron (column centering via k.) subtracts from each stimulus representation its mean across neurons, measuring the dimensionality of the neuron-response patterns. This is more appropriate when the question concerns the structure of neural covariation rather than stimulus coding. general 7 task * y neuron, the two coincide only when the data matrix has special symmetric structure (e.g.. when k k and the row and column distributions are identical). The bias-corrected estimators -aak and Tboth are derived identically to az Tooh, with the respective centering applied before computing Vigte"
For now, let us focus on the numerator and the denominator of task Note that the numerator can be expressed in terms of the original kernel & as:
= (a) 2
= -[ M(ss)) M(r,,) -2 (xx(a)) M(x,,)) / dpx(x))px(()) (), (f dpx(a)ddx(() Ko,.)) 2
The denominator can be expanded as:
B B= / dpx(a)dpx(())kkek) ka k)
= =f dxx(x)dpxx()) 2(x,))= (dx(())dox())dxx()) / 2)x,))((2. v)+ (f dpx(ordox(() (o)eel()(())))))))))) M(c.)))
16

---

Published as a conference paper at ICLR 2026
A.3 DERIVATION OF THE ESTIMATORS
The goal is to derive an unbiased estimator of each one of the six terms (five unique terms) above. The five unique terms are
7 - (fapxlo)Mass))
8 8= | dpx((t) k(x,)) / dpx(x))dxx()) k(r,y).
8 = I (px(())px(()) Ma, Et = ( k(x, 2))2,v),
and t -(doxx(odpxx()) ()))()) k(.v)
With these terms, the numerator of task is
A 8 27 +1,
and the denominator is
== 3 2F + 85.
Consider 7 an example. We start with deriving naive biased estimator, since is easiest.
1
ΣΣ i5
(a,, w.) ( w
naive
P2Q²
We arrived at the above by simple derivation:
p²Q = (Tr(K))²=
ΣΣ 2
(S8)
ΣΣ 22 22
22
22
ΣΣ Ó( w.) ww))o/l...... wg)
(S9)
jg
The reason havee is biased, i.c. Et # (thave) where (1)g is the average over all submatrices, is the following:
11
1
ΣΣ o8
o(z;, wa ( wp)2
p2c2
(2chtwa>
bbx wa) ) 2000, wz)² + ΣΣ
(0(z), wA 2 wz)2)
22Q²
$ a
ΣΣ
(,( w. )Z wa)z)
+ ΣΣ
(plozes (()c)))))))))
(4)
(P (Q - 1) (k(r, x)) ammpx
Q-1
(k(o,,0)2)
+
PQ
PQ
xxpp
P-1
(k(w,, w)2)
(S10)
PQ
PQ
20-Px W0-PP
Note that in the first term, (M(2, ar) x~px = Zl, so the leading order term in (tlave) is unbiased.
However, the rest of the terms contribute to O + + + bias. This shows that we can derive the
17

---

Published as a conference paper at ICLR 2026
unbiased estimator by simply summing over disjoint indices, and normalizing by the number of summands:
ΣΣ (xs wa)2 wa)²
both P(P-1)Q(Q) 1)
Then have ( (nhrt) 4 at the following unbiased estimators:
P. Applying the same logic to all five terms P. C. 84, and 89, arrive
both P(P-1)Q(Q 1)
Σ igf afB
aB Wijf
2 both
((P-1)(P-))Q(QQ-1 Σ Σ
ifitlaps
P(P-1)Q(QQ - ΣΣ
1
o8
both
19új
iff ax8
moth
2P
and
(z1 az
Σ
both
P(P-1) ()P-
#
where
Ujtt = 2.. D P13
4 r/8.
The remaining challenge is computing the sums over disjoint indices. The implementation is challenging, since computing this sum over loop can be slow on computer. Therefore, one needs re-express disjoint sum into a linear combination of regular sums. For example:
Σnz=Ery
ry
Σ T
This becomes non-trivial as the number of indices increases. Consider an example with three indices:
Σ
Eris
rish (1- (1 6zx) (1 - 55k)
Tisk
Σ ijjk
Tisk (1 85k -.n Būj + 28(gh)
Σ Tisk
Σ
Σ
riys - Σ
raj + Σ rss-
riss
F.j.k
i.
Using this technique, we find that the terms can be expanded into the following. Let
1
1
Σ
08
Σ
Pijlr
QQQ
fittr
Q
-1)
0
then
18

---

Published as a conference paper at ICLR 2026
Eis Fuijj - -Ei Fitti
both
11P-1)
Eist Taill - -2 Pijis- - Eij Tiijj - 2 2 Tiii
both
P(P - (P)(P- - 2)
Eij Tijij - Ei Vitti
3
P1P-1)
Eijt Paist - 2 -2E Fiiij - Eij Tijej + > 2raa
both
and
P(P - 1) (P - -2)
Eiitm Tizim - Eiji (raast + 2rijst) + Eig (Pays 8rajis + 2rajas - 6 Ee Passi P(P 1) (P 2) (P- (P-3) 3)
both
Finally, the unbiased estimators of the numerator and the denominator of "aask are
+5 Aboth tboth 20 both
and
Booth
"both
2f +
bothh
By simply taking the ratio of these unbiased estimators, we obtain our dimensionality estimator
Aboth
Tboth
Booth
Deriving Arow and Brow can be achieved by simply summing over the columns with the regular sum, but the rows with disjoint indices: Let
T'yer =
1
Σ
Q2
sjir
09
Then,
Eij Paass Ei Thist
1
P(P- -
Eig rig 2 Enj 1881 Eis riiss + 22Er1 r'us
row
P(P (p
Enjthys Ei T'itt
3 Fow
P(P
Eog -2 2n, Plass Eis N'1is Ee 2r/mn
4 Tow
and
P(P - 1) (P -2)
Esstm Pestm 2 (rist 2r' + Eng (Piiss + 8r/iss + 2r(ses) - -6Σ, Prit
5 row
P 1) 2) (P 3)
Then the row-corrected estimators given by
A Arow
Yoow
2/2 2ow
Tows
19

---

Published as a conference paper at ICLR 2026
Brow
*row
2f row
+
and
Arow Brow
Trow
Similarly, Acol and Beol can be achieved by simply summing over the rows with the regular sum, but the columns with disjoint indices:
e1
Tújj
2
Tiifls
p3
43
col
p2
Tijejo
Pijjts
and
p3
Σ Tijlr. i.ddr
col
Note that the unbiased estimators for Tacuron are given by redefining aa as [ ka / [ haa [ Trs, i.e. simply compute everything with the transposed data ma- trix.
All the estimators can be straightforwardly implemented using the einsum operation. We use the cinsum function provided in JAX.
A.4 NOISE CORRECTION DERIVATION
Suppose that for each stimulus H; and neuron w.. the observed activation in trial / is
Φ(2, wa) + 77 Wa: (t),
where is sampled independently across trials, and E, [n( w, e)] 0 for all æ and w. Additive noise corresponds to v not depending on di multiplicative noise corresponds to ry(2, w, e) = Φ(x w) with E[e] 0.
Cross-product construction. Given two independent trial matrices ((1) and (22), redefine the elementary quartic tensor as
(2) kg 13
Giki
Claim. For indices (i,j, () that are all distinct and a + 3,
ad Uijkt]
6(2i, w.) ((I5, w.) (x ws) p(m, wg).
Proof. Since e and e2 are independent, the two factors $ (1) Φ (they involve disjoint sets of noise realizations once conditioned on { and {wa)). For the row factor:
(2) and D (2) are also independent
(1) ② (2) = E (Dia + (1)) + -
sia Dja + dia Emm
(2)
+ B(min
(1)
dya Bin
B [E = Dja
=0
20

---

Published as a conference paper at ICLR 2026
where used [] 0, E[n O, and independence of the two trials. The column factor gives the same result. Multiplying gives the claimed expectation, so the bias-corrected estimators Abot and Booth applied to the cross-pross pros-pproduct matrix unbiased estimators of A and B from Eq. Eq. (3). The remaining bias is therefore purely of order (11/P + /Q), arising from the unequal-index averaging.
Correlated noise. If 61 and (2 are correlated across trials (e.g., slow drift, gain modulation), then (17("))(22)) o O, and the above cancellation is incomplete. Specifically, for i (diagonal terms, which appear in booh and bohh)
(1) (2) Tia Tia
SC = Oia +
The residual bias from correlated noise contributes O(1/P+1/Q) multiplied by factor proportional to the cross-trial noise covariance. Using N trials and averaging over all N(N - 1)/2 cross-trial pairs reduces this contribution by a factor of 1/N, O11NN).
Summary of noise structures. The cross-product construction eliminates bias contributed by independent additive and multiplicative noise noise whose realizations 61. C2 are uncorrelated across the two trials - reducing the noise-induced bias to O(1/P 1/Q). Noise that is correlated across trials (e.g., slow neural drift, gain modulations, or stimulus-locker hemodynamic variation in IMRI) is not eliminated: the residual bias proportional to the cross-trial noise covariance (7 Using N 2 trials and averaging all N(N - 1)/2 cross-trial pairs reduces this correlated- contribution by factor of 1/N.
A.5 IMPORTANCE SAMPLING
In practice, sampling distributions of stimuli and neurons may not match the intended distributions. For instance, electrophysiology probes tend to capture neurons from spatially clustered cortical layers or depths, and stimulus sets may be imbalanced across categories. These biases can introduce design-specific errors not removed by the unequal-indes averaging alone.
We resolve this via importance sampling. Suppose the latent variables are drawn from the true pw. but the observed sample matrix is generated under biased (observed) distributions v; ~ obs and wa ~ PW where the supports of px and jaw are subsets of those of pobe and pebs, respectively. Define the importance weights
distributions T; px and
px(x)
(w( ohs (w)
r(x)
dww)=
obs (
We introduce a weighted unequal-index summation
#r.X.
#o.W
Σ
Σ
as
o
both
# of summands
where the row sum iterates over the observed stimulus set X, and the column sum iterates over the observed neuron W.. each excluding colliding indices. The importance-samplinn sunpnee-oeaeiiinnno--tttedd corrected estimator is then
IS
18 +
IS
Uiijj
2
Cigtt
IS both
both
both
both
15
0. Cijis
2
ag Aglr
both
both
buth
whose numerator and denominator are unbiased estimators those of 7. As a concrete example, one aims to estimate the PR of V1 representations but recordings are biased toward neurons with specific tuning orientations, one can construct a histogram over tuning orientations to estimate piy assume uniform true distribution Pw, and compute c(w) accordingly. Similarly, stimulus-set imbalances across object categories can be corrected by setting r(x) to the inverse of the category sampling frequency.
21

---

Published as a conference paper at ICLR 2026
A.6 ESTIMATING THE PR OF A FINITE- SIZE UNDERLYING MATRIX
Suppose the sample matrix & E PxQ is obtained by uniformly sampling the rows and columns of G E RRxC without replacement. The goal is to estimate the PR of the matrix G' with column-wise
centering:
71 21
7
2t
where
2g = (905))
(
29 - (0
= ( and =
a9 where Pijlr Gi Gja Gia Gra, and the bracket here means simply
the free indices.
The following are the unbiased estimators, i.e. tg
12 9 ig. ty. and ty. of these terms respectively:
R(R-1)C(C-1) Σ P(P-1)Q(Q-1)
RC(C-1)
Σ
o8 H8jj
Σ
Σ
o08 Viii
9
R2C²
+
PQ
($ 0
0
+ P(P-1)Q
R(R-1)C
Σ
RC PQ
Σ Σ
(S11)
1
2(R--1)(R-))(CC-1) Σ Σg
a8
C
((P-1)(P-))QQQ--
aij
-(R-1)C(C-1) Σ Σ
RC(C--1)
2 U + (
PQ(Q-1) ΣΣ
o8
P(P-1)Q(Q-1)
1888 ag
iデj ax$
1
R(R- ()
Σ Σ
+
C
P(P-1)(P-2))
ifjl
R(R-1)C
RC
ΣΣ P(P-1)Q
(2 Uiy
00 +
18jj
+
PQ
ΣΣ
U
Hi
(S12)
ifj
R(R-1)C(C-1) P(P-1)Q(Q-1)
RC(C-1)
1 RCC
Σ Σ
o8 196
+
ΣΣ
PQ(Q-1)
($1 0Ø3
+ P(P-1)Q
R(R-1)C
Σ
RC PQ
Σ
Σ
(S13)
1
((R- Σ Σ
a8
R
((P-1)(P-))QQQ-
iigtt
iデj$1 a
R(R-1)C(C-1) Σ Σ
RC(C-1)
(2
000 + (00
PQ(Q-1) ΣΣ
P(P-1)Q(Q-1) iンj a#9
ag
1
R(R- C Σ Σ
CCC
P(P- P(P-1)(P-2)
R(R-1)C P(P-1)Q ifj
RCC
ΣΣ
(2
aa +
+
ΣΣ
(S14)
198j
PQ
22

---

Published as a conference paper at ICLR 2026
and
R(R R[R-I][R-2/(R-)((CC-) Σ
09 RAC
Σ
aD
P(P- PIP-11(9-3(P-3]QQQ-] #Ffllr ax$
R(R-I)(R-)((CC-) Σ
Σ
(2 08 figt
+ 4 9g
2(P-1)(P-))((Q-1)
#1 of
R(R-1) Σ
RC
Σ
ag mis
ΣΣ
+
P(P-1)Q(Q-1)
PQ(Q
2 2088
i#j a7
af
((R-))(-2)(R-2)) Σ
Σ
RCC
P(P (P--))(-2)(PP))) 3)Q
ifitlt
R(R- Σ Σ
+ P(P-1)(P-2]C
2
Vijl
aa
+ 4 1 ad
R(R-1)C ΣΣ
RC PQ
4
00 Btj
Σ
(S15)
P(P-1)Q
17j
.9 where Mijtr = iin Djn 1 2.3.
To implement this function as a computer program, we need to vectorize the summations over unequal indices in an efficient manner. For that, we provide the following expansions of these recurring sums. Suppose
a3 Gglr
. ijlr
Σ
Tiyh Q(Q
QQQ 1)
fily
23

---

Published as a conference paper at ICLR 2026
Then
ΣE oz9
ag 3883
Σ
(S16)
PQ(Q-1)
P
ΣΣ
03
Σ Tajjj Σ
1855
Titl
i#j a78
(S17)
1( -1)
P(P 1)
ΣΣ
.99
Σ
Tiijj Σ
Viity
Pisii
1
(S18)
P(P-1)Q(Q- -1)
11)-1)
ΣΣ
Σ Tijij Σ
195)
Tiil
itj 0*8
1
(S19)
P(P-1)Q(Q 1)
P(P 11)
ΣΣ
05 rigl
Σ Tuil 2 Σ Tiaij Σ
rags +2 > Tea
#
hddd
i.j
1.j
(S20)
P(P - 1)(P -)) Q(Q)(QQ)))) 1)
P(P - -1](P- (1)
ΣΣ
0.
Σ
Tijil 2 Σ
Fitij Σ Pijij +2 +2E rias
G58t
3,dd
(S21)
P(P 1)(P 2) Q(Q 1)
P(P 1)
ΣΣ
08
( 0
Σ
Tijlm -2 "
(rase + 2rigst
P(P - 1)(P 2) -3)
iidlm
(rass + +8rig1 + 2 - 6 Σ rssss Σ 2
(S22)
In g. Eg. 13. tg. and ty. we also have version of each expression (one whole line) above, where we have E. instead of Eos3. In that case, one can simply replace righr शाΦ (Ea,s Wiftr Ea (jfr) with "! appears.
sglr = Q Ea
aa and plug into the R.H.S. of each expression above wherever r Aitr
A.6.1 EXAMPLE DERIVATION
As an example, here we derive 13 an estimator for 'g" which is the denominator of the PR without the centering. This also corresponds the second moment of the eigenvalues of G. The following derivation of to should illuminate the rest of the derivation, which readers can easily deduce.
Let us start with the expression of what we want estimate
1 (99f5))
ΣΣ aB Pifty
R²C
3.
Now consider the following sum and its expansion:
24

---

Published as a conference paper at ICLR 2026
ΣΣ a9 Bijs
ΣΣ
8
ΣΣ axd
ΣΣ Jigt
ΣΣ
RºC²
RºC²
Jijaj
Fistr
+
9 9153
The goal estimate each term without bias.
aB Let us try estimating Eijz Eays Piges with the following estimator, which we derived earlier for the infinite underlying matrix case:
ΣΣ Σ
PPP P(P-1)Q(Q-1) 1
1j8j
we take the expected value of have
o(x wa Wa)o(, wa)o(as w3) w3) ΣΣ ifj a
P(P 1(Q-1)Q(Q 1)
(ó(z, w w) w) u)dly. u)) I.N,
= -x Gwx Guy Gu)
ΣΣ 1jhll ar3ew
(100,)0000)))0,) (w))) (0)) Gia
where is vector and (+) is scalar value indicating the ith dimension of x (similar for god) wll)) and ul ()))), Here, we defined the generative process
ia Φ w.) x Gwa
where z; and Wa are randomly generated one-hot vectors. I; and wa are drawn such that they select the rows and columns of without replacement. Therefore, unlike the original setup, { are not independent, and so are {w., 1
Since is one-hot vector, (010 0, if 7f k. The same logic applies y, and Also note that z (1) (0) 0, if and w/ ( = 0, if a 0. Therefore, the above simplifies to
ΣΣ P(P-1)Q(C
Σ ΣΣ i.f az3
( (w))))((0)
a0 Pijig
Ejjjj
igj ax3
2.g.0,M
ΣΣ
R(R-1)C(C-1)
Pefij
1nf ag
ab Therefore, we have estimated Ejj Easo Pijs but with the extra factor R(R 1)C(C-T Therefore, we know that the following is an unbiased estimator ΣiØs Eazo Pig.
1 RR - 1((-1) ΣΣ 154j"
a9
P(P-1)Q(Q-1) 84 3
a6 The estimators of Σ, Entg Peiii Σij, Ea Pi and Σ; Ea g can be derived in the same manner. The same logic applies deriving all terms for the estimators of f7 12
g" 14 and 15 g-
25

---

Published as a conference paper at ICLR 2026
B APPENDIX FOR SCALING LAW OF THE NAIVE ESTIMATOR
Here, we quantify the expected value of the inverse of the naive estimator. In other words, quantify the expected dimensionality of the sample matrix & E R PxQ. For simplicity, we do not consider the centering, and assume ? large and
close to J. 7
We start with the second-order Taylor series expansion of the ratio of some random variables h and 1, Le. 1 around L, where H and are the expected value of and I respectively.
( 11
+ L
1
H dp
L
HL (onl + 5n51) + I
(o + +0f))
where 6; = (1) H, 62 = (1) L, Tn = Cov(h, 1), and of = Var (1)-
Since quantify the expected value of the inverse of the naive estimator, I corresponds to the numerator of "naivv and h corresponds to the denominator of "niive i.e.,
ΣΣ is 0.8
ban wa Mas w wa)(,,
P2
and
h = p2Q² ΣΣ
ba, wa Waldlay wa wa) wg)k(, wa).
The indices run over the samples {ar) and {wa) 1 that are independently drawn from px and Pw respectively. After some algebra, we arrive at the following expressions for the quantities relevant for quantifying
() above.
1 ((M(o.,0))) ((M(o.0)))((M00))) +
1 -MMO,)))))
( ((kttv: w)²) - (k(2, ())))
52 = P
Q
1
- (Mxx.))2) ((M)),))3)
1 ((k(w,,)))) -
- (M(z.w3))
5n p
Q
4 ((H(E.0))(00000)) (k(u, w((x,w))) (K(2, (=)x)
Oih
P
4 ((ktu,,))kkuu (kk(u, (k(co.))))(((x.00))
(k(w,
t2
(k(I, + p (k(z, (c) (k(x, 2 (xxxz))) (k(z, v)²)
4
of = p
6 (k(w,w) )=)*+ + 4
(klww, (((w,w))
2 [ilw, w))² (k(w, w)²)
2
Q
Q
and
-² = 4 ((te.)))kk()) -
4 (((t:,.)(k(uu00)) - (M(u, m) 3)))))
0
+
Plugging the above into the approximate formula for (4). we get
26

---

Published as a conference paper at ICLR 2026
Pvb + )
(1-2)
+
Q
Yuaive
(M(uc, ( w)
4
(k<(,,)) y) k(zz (((x,,))k(z,, x²)
4
-1
Py
(k(x,,)² (k(
Q7
(klus w)))
(k(w,
1
1
1
1
2
6 ?
+ 4
P
7
Pb
Q
72
where v ( (k(z,)))) and ته = (k(w.w))
which are the participation ratios of k(x, æ) and k(w, w) with respect to px and pw, respectively.
Now we take the large y assumption, which simplifies the above to
1 P
Thaive
QU
Intuitively, &b is the effective fraction of stimuli that evokes large overall response in the neural population (i.e., the effective fraction of rows with large norms), and v is the effective fraction of neurons that has large overall response to the stimuli (i.c., the effective fraction of columns with large norms). If we assume k(x, .)) and k(w, w) are constants (t = 3 1), then we have
-
Tiuive J
large class of generative process has k(x,,x) and lew,,, w) that are constants. For example, consider the linear process where sampling of & is modeled as a random projection of some fixed ground truth matrix G E _
4 x Gw..
where x, (RxR) and wa widely used to approximate matrix sampling in physics and random matrix theories, and has zb 2 1.
independently across and a. This process is
C EXPERIMENTAL DETAILS
We provide all our code and necessary data to produce the figures with our supplementary material. Here, provide additional results from experiments.
C.I NEURAL DATA EXPERIMENTS
In all our experiments, we computed the dimensionality subsampled activations over many iterations (Stringer: 50, Ma Maja jang: 500, TVSD: 250, and ThingsFMRI: 50).
Here, we show that the bias in our estimator arises from the nonlinear nature of the division operation used in computing dimensionality. Note that the dimensionality is defined as the ratio between A and as in Eq. (3). In Fig. $1, the first row shows our estimator when dimensionality is calculated by averaging the ratio, ( ) while the second row shows the ratio of the averaged numerator and denominator, (4) While our estimator gives unbiased estimators of A and B separately, it remains (18] biased for dimensionality.
27

---

Published as a conference paper at ICLR 2026
MajajHoog
MajajHong IT
10
17.5
15.0
8
12.5
Blased
10.0
5.0
10
10
10"2
100
12
Unbiased
10:
08
10
10a
10-1
10
Figure $1: Bias due to nonlinearity in the definition of dimensionality.
C.2 LLM EXPERIMENTS
For our experiments, we use the pretrained Llama 3 base model from Grattafiori et al. (2024). We extract its hidden representations nine different languages from the Flores dataset NLLB Team et al. (2024): English, French, Japanese, Korean, Russian, Ukranian, Turkish, Kazakh, and Greek.
D BIAS VARIANCE ANALYSIS
Here we quantify the bias and variance of the "naive and Booh in the context where both the rows and columns are sampled. P and Q are the number of sampled rows and columns, respectively. We will ignore the centering operation for simplicity. We start by considering the general behavior of the ratio of estimators.
Suppose an unbiased estimate and B is an unbiased estimate of Suppose 8,4 and og are the biases of A and B, respectively, e.g. 8 = (A) o² and g are the variances of A and B, respectively, and o BB the covariance of and Then the following are the bias and variance of A/ B as an estimator of a/b, up to the second-order approximations:
A
1 ab
1 (0 + 68) 6)) (oAR SΛdn) + 1
bias
-on
b
2
var
B
~
b
a
+
13 TB - AAB
ab
In our case, a = (k(I,
and b = (k(2, y)
A
6A
OAB + 5Δ6B
o + + (k<z, v)
bias
B
(b(2,
(k(x,))2)
(k(x, (((,)))))((())))))))
(k(z, v)2)
28

---

Published as a conference paper at ICLR 2026
A
o9
o
OAB
var B
(k(z,,0)) +
(Mx,,))e)
- 2
x)) (k(2,
D. NAIVE ESTIMATOR
We have the following for the naive estimator Yhaiie::
1 ( (k(w, - (k(zz, ()() 1 ((Kle,27)) (M(2))))*
A p
6B P ((M) (e(2..0)))
+ 1
- (M(2)))))
4
(b(2, - (k(z; u)²) (B(x))))(((,x)))))))))){ (b(x,,))) +
- (klww, w))
AAB
p
(k(,,w)) ( ( (b(x. ()()
@
0 = P (k(1, + 4
6
7 (M(x, (1),,,z))
2 (k)) (M(z, w)²))
P
p
- 2 (k(w))w)) 2 ((lu,w))) wk( 2 + 2 (k(w), w) )² (k(w w)²)
4
2
4 oB -
( (M(r,))))) + 1 Ō (k(u:,))k(u, (k(ur,w)))
Plugging the above into the earlier expressions, the bias and variance of the naive estimator are given by
bias (Yinaive) = 47 (p (c-c)+ o0( (a
((--)) -7 (
( 21)
Pvb + 1 ) and
Qτ)
var (Thaive) 2 4 22 (1
+ 26'
4 32
+ē + 28 28 -27 - 1) 1+22
P 1
Q
where
c= ( ((((1,))))
(k(z,v)<k(z; xx)
d
(k(a,, 3)²)
(k(x,))?) (k(2), x))
[6( w)2)
(k(w, (,0,, w)
d' (k(w, w)2) (k(w,, w)
(M(,, w)2)
+ ( (k(w, wo)
3 xk)), ())))
and
(ktw w)))
29

---

Published as a conference paper at ICLR 2026
D.2 OUR ESTIMATOR
We have the following for our estimator Tboth"
54 = 0
dB 0
P (M(O,)))M (((MMe..))MMa)))))))))) (k(x,x)) - (k(z) v)²) (x)x)
(k((uu)) w) (k(w,, ) (()))(((())))))) (b(x. (k)2)
((luu)))((e
+ x
d3 P
4 ((Ml2,2)3) ( + Q
((K(u,, ( ((w (b(x)
0B - = (M(x - -K((I
+ 4 Φ ((ktu,,)k(uu (K()))()))))))))))) 2)))
Plugging the above into the carlier expressions, the bias and variance of our estimator are given by
bias (7boch) xe 47 ( (c- ( (6-2)) and
var (7both) ) PJ 4 22 (
+ 4 1"
+ 2d
+ - 27
P 36
D.3 COMPARISON OF THE BIAS OF 'NAIVE AND 7BOTH In summary, we have the following for the biases:
c) (- Q (c ()) -7
1
1 Pub Pbb
(( ②
)
2) 2
bias (Yiaaive) 4r
+ (p(pp
+0
Q
and bias (7hoth) = 47 ( (c d) + _/ ( (
+
Note that the first terms 4y (c- + (ö こ1) in bias (?haive) is entirely the leading order
term of bias (Nbah). This means that the second term 7 (7 1) ( P Q is the additional bias in Thaive- Here we show that the first term is often small, and the second term tends to large, which is removed in bias (Mom).. Let us inspect the term d' that contributed the bias in Ybah.
d ((((z.0))
(k(x, k), 2²) (M(Z, v))) (k(IT, (x)x,))))
(k(z, v22,,))222
(((k,,)))) y)²), is constant for all y. then c = 1. This is also when e' = Therefore, when (k(æ, v)?) constant,
30

---

Published as a conference paper at ICLR 2026
0.
Note that (k(x,,) )) svvp is constant for all y, if the kernel and data distribution are matched by global symmetry (rotational or translational). For example, consider any dot-product kernel of the form
k(x, v) = (x y)
where h + I, with æ and sampled uniformly from a unit sphere. Then c -- c' = 0, so the bias in our estimator is dramatically smaller than that of the naive estimator. Similarly, consider any translation invariiant kernel of the form
k v) = h1(2- -ox-yy) y)
with x and sampled from a translation-invariant distribution, egg. uniform measure on torus. Then d' O.
E LOCAL DIMENSIONALITY ESTIMATION
The sample neural manifold is simply the set of row vectors of the sample matrix 4
M M [
where o; E R is the ith row vector of 4. Suppose the distance metric defined the sample representation space R& denoted as d (t,-). Define the ball in the representation space centered around Φn E R& with radius r:
Bl r) = ( M d(ō, Φ0) sr).
Let Tboth (S) denote dimensionality measured finite set S of sample representations in R Then, the local dimensionality estimate for a radius r is defined as
Jocal Yboth
(r)
Tboth (B( r))
IM
P.E
Now, let us define the distance metric. For two representation vectors 6, 2o E R2, the squared Mahalanobis distance defined as:
(0- M (5-
where .M is a positive-definite metric. We want to preferably select representation along the tangent directions of oo so that we faithfully capture the local dimensionality. We estimate a local repre- sentation metric by (Φo). the covariance of the k-nearest neighbors of do. Then we define M as (0o) the pseudoinverse of Σ (øo).
31


Reviews:
[{"review_id": "Q8341Br33C", "reviewer": "Reviewer_kPsS", "summary": "The paper examines the problem of estimating the manifold dimensionality (effective rank) formed from neural data under stimuli, how current approaches may be biased especially in small sample sizes, and a new bias-corrected estimator. The authors demonstrate this metric on diverse neural data and a LLM.", "questions": "- How does this apply without inputs (e.g., spontaneous activity)? \n\n- Is treating the sample activation matrix as a random submatrix always appropriate? Neurons are certainly subsampled, but in many organisms much more of the network is directly observable. If we observe active neurons, is it likely to be a random subsample, or simple most neurons that are responsive to a certain input stimulus? And most samples on the stimulus input side are not randomly chosen either. \n\n- How much does the bias term contribute to the difference in estimators? E.g., how biased is biased? Line 312 denotes some bias as negligible without giving a value or order of magnitude. Line 358 seems to imply that 'most bias' means most change across number of samples. Figure 4 indicates that the new estimator is also biased?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Very well written and organized. It tackles an important and broad problem in computational neuroscience with some applications to general ML models. Math appears correct and well derived.", "weaknesses": "- Some jumps in the flow (e.g. how do line 266 and line 268 connect?) detracted from the clarity. \n\n- Some portions of the results are unclear or unexplained. Figure 2 shows how some estimators don't change as a function of the number of samples included. But some of the naive and other estimators are stable, though the difference in value of dimensionality is different. Which is correct, given no ground truth? (See Q, top right and bottom left). \n\n- Discussion and Limitations are too short; how feasible is this? How computationally expensive is it in practice?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jUW7G0cXBO", "reviewer": "Reviewer_aVtZ", "summary": "The paper studies estimates of global manifold dimensionality from finite neural response matrices. It shows that the standard participation ratio (PR) is systematically biased in small-sample regimes and proposes a bias-corrected PR estimator that removes terms arising from index collisions in finite submatrices. Concretely, the sample matrix is modeled as a random submatrix of an underlying infinite matrix, obtained by independently and uniformly sampling stimuli and neurons. The authors derive unbiased estimators for the numerator and denominator of PR by averaging only over unequal indices and define γ_both as their ratio, with row-only and column-only variants (γ_row, γ_col). They further give a two-trial noise correction that cancels additive or multiplicative noise without requiring many repeats, and a weighted version that yields a local, noise-robust intrinsic dimensionality when weights select neighbors. Experiments on synthetic data recover ground-truth dimensionality, and applications to mouse V1 calcium imaging, macaque electrophysiology, human IT fMRI, and LLM activations show reduced dependence on P and Q compared to naive PR and TwoNN. Implementational tricks for unequal-index summations are provided.", "questions": "* How does γ_both behave under non-uniform sampling where inclusion probabilities differ by layer or receptive field location. Can your weighted estimator correct such sampling bias if weights are set to inverse inclusion probabilities?\n* The two-trial noise correction assumes independence of trial noise. How sensitive is γ_both to trial-to-trial drift or slow fluctuations that are shared across trials?\n* For the LLM study, can you report layerwise γ with and without centering and with language-balanced sampling to ensure that row sampling is not conflated with sentence-length distribution?\n* The supplement mentions different centered kernels for task vs neuron centering. Can you clarify in the main text when each is appropriate?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Originality: Clear identification of finite-sample bias sources in PR and a general correction via unequal-index averaging. Weighted and two-trial extensions broaden utility\n* Quality: Solid derivations, sensible baselines, coverage across multiple data modalities\n* Clarity: Motivations, assumptions, and estimator definitions are well laid out; limitations are stated.\n* Significance: If adopted, this can standardize dimensionality reporting and reduce misleading sample-size effects across empirical work.", "weaknesses": "* The estimator assumes independent, uniform sampling of rows and columns. In practice, electrophysiology probes capture spatially clustered units and often bias toward specific cortical layers or depths; calcium imaging has field-of-view and sparsity biases; fMRI voxels are not neuron samples at all. This undermines the \"random submatrix of the infinite one\" assumption and could introduce design-specific bias not removed by unequal-index averaging. Please analyze or simulate non-uniform, block-dependent, or layer-biased sampling and show robustness or an adjusted estimator that incorporates known inclusion probabilities via importance weights\n* The two-trial correction addresses additive or global multiplicative noise across trials. Real recordings can have correlated, heteroskedastic noise across neurons/time. Clarify which noise structures are neutralized by the cross-trial construction and which remain.\n* No variance estimates or confidence intervals are given for γ_both. A delta-method approximation for a ratio of U-statistics or a simple bootstrap over rows and columns would help readers gauge reliability.\n* The local estimator requires radius sweeps and unequal-index summations. Please report time and memory as functions of P and Q\n* No comparisons to other global measures used in practice (under finite samples) are made. No ablations included that vary manifold curvature, anisotropy, and rank-plus-tail spectra", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "njRKWTfqW1", "reviewer": "Reviewer_uXtA", "summary": "This submission proposes a new estimator of the Participation Ratio (PR), a well-known tool to measure the global dimensionality in Machine Learning or Neuroscience. This novel estimator is proven to be less biased and more robust to measurement noise than the state-of-the-art. The main impact is that it needs fewer measurements to yield a precise estimate.", "questions": "- Suggestions: Adding pointers to the appendices in the main body will make the submission sharper. I also suggest numbering more equations to allow these precise cross-references. Another option would be to formulate results as Propositions whose proofs are given in the appendix... but I can live without this formalism. \n\n-  Q1: Fig. 1. What is the value of Q when P is varying (left)?  What is the value of $P$ when $Q$ is varying (right)?\n\n- Q2: Why $\\gamma_{col}$ ($\\gamma_{row}$) is a scaled copy of $\\gamma_{both}$ when $P$ (resp. $Q$) varies?\n\n- Q3: I am not convinced by the use of Mahalanobis distance to define the nearest neighbours. Typically, NN-based Local dimensionality estimators use plain Euclidean distance. What is the value of k for learning the local covariance matrix? You mention 4 neighbours in the main body.\n\nTypos:\n- Line 760: \"Consider $\\hat{t}^5$...\" but the following details the comptutation of $t^1$.\n- Line 907: Similarly", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "I see the following strengths:\n- The topic fits exactly the scope of ICLR, ie. learning representation. This new estimator is helpful in gauging the 'quality' of a learned representation.\n- The paper is well written, postponing the gory technical details in the appendices. At first sight, reading the main body, I had the impression that the exposition was too simple, but the appendices evidence that the rationale is strong. I suggest to add more cross-references to the appendices. \n- The approach is deductive: first an analysis of the biases in the naive approach, then the deduction of the modifications to patch this caveat.\n- The new estimator is not entirely unbiased, but the study shows that the bias is significantly smaller\n- This main property is also demonstrated with convincing experiments.\n- The computation of the new estimator is expensive. Yet, the submission proposes an efficient implementation.  \n- The submission is dense as it also proposes two extensions: robustness to measurement noise with a double trial, and an integration of a weighting mechanism to estimate local dimensionalities, which can then be averaged into a global dimensionality.", "weaknesses": "- W1: The experimental protocol only compares the new estimator to the naive PR estimate and Two-NN. There are many more estimators based on nearest neighbours, like Gride from Denti et al. (cited in the submission), the Hill estimator for local dimensionality and the likes (MLE, MoM, TLE, see the works of Michael Houle, James Bailey -- not cited in the submission), not to mentioned the `insane' estimator based on diffusion models (see A Geometric View of Data Complexity: Efficient Local Intrinsic Dimension Estimation with Diffusion Models, Kamkari et al., NeurIPS 2024).\n\n- W2: I expected some justifications about the noise correction based on double trials. On the same token, I do not understand how appendix B is related to the main body.\n\n- W3: I am not sure that the dimensionality estimator should be the arithmetic mean of local estimates. Some works argue that the harmonic mean is better. See for instance, page 11 of *A Survey of Dimension Estimation Methods*, Binnie et al.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)