PCA++: How Uniformity Induces Robustness to Background Noise in Contrastive Learning
Authors: Mingqi Wu
Qiang Sun
Archer Y. Yang
Venue: NeurIPS 2025 spotlight ()
Keywords: Contrastive Learning
Principal Component Analysis
Uniformity Regularization
Self-Supervised Learning
Representation Learning
Primary area: general_machine_learning
TL;DR: This paper proposes PCA++, a uniformity-constrained contrastive PCA for robust signal recovery under strong background noise.

Abstract: High-dimensional data often conceal low-dimensional signals beneath structured background noise, limiting standard PCA. Motivated by contrastive learning, we address the problem of recovering shared signal subspaces from positive pairs--paired observations sharing the same signal but differing in background. Our baseline, PCA+, uses alignment-only contrastive learning and succeeds when background variation is mild, but fails under strong noise or high-dimensional regimes. To address this, we introduce PCA++, a hard uniformity-constrained contrastive PCA that enforces identity covariance on projected features. PCA++ has a closed-form solution via a generalized eigenproblem, remains stable in high dimensions, and provably regularizes against background interference. We provide exact high-dimensional asymptotics in both fixed-aspect-ratio and growing-spike regimes, showing uniformity’s role in robust signal recovery. Empirically, PCA++ outperforms standard PCA and alignment-only PCA+ on simulations, corrupted-MNIST, and single-cell transcriptomics, reliably recovering condition-invariant structure. More broadly, we clarify uniformity’s role in contrastive learning—showing that explicit feature dispersion defends against structured noise and enhances robustness.

PCA++: How Uniformity Induces Robustness to Background Noise in Contrastive Learning
Mingqi Wu
Qiang Sun University of Toronto MBZUAI
Archer Y. Yang
McGill University
McGill University Mila
Mila mingqi. wu@mail mmg.il...m...ccc.
qsunstats@@mail. com Corresponding author
archer rang@mcgill...
Corresponding author
Abstract
nigh-ddimessionll data often contain diw-dimesiional signals obseured by struc- tured background noise, which limits the effectiveness of standard PCA. Moti- vated by contrastive learning, subspaces from positive pairs, paired observations sharing the same signal but differing in background. Our baseline, PCA+, learning and succeeds when background variation is mild, but fails under strong noise or high-dimensional regimes. To address this, introduce PCA++, hard uniformity-coostrrineed contrastive PCA that enforces identity covariance on pro- jected features. PCA++ has closed-form solution via generalized eigenproblem, remains stable in high dimensions, and provably regularizes against background in- terference. We derive exact high-dimensional asymptotics both fixed- exed-aspect-ratie and growing-spiks regimes, showing uniformity's role in robust signal recovery. Empirically, PCA++ outperforms standard PCA and alignment-only PCA+ on sim- ulations, corrupted-MNIST and single-cell transcriptomics, reliably recovering andition-iivvrialn structure. More broadly, we clarify uniformity's role in con- trastive learning, showing that explicit feature dispersion defends against structured noise and enhances robustness.
address the problem of recovering shared signal
alignment-only contrastive
1 Introduction
Real-world data often hide simple, low-dimensional signal beneath layers of structured noise and random variation. In genomics, batch effects blur true biological differences [32, 21]. In medical imaging, scanner differences obscure clinical signals [18]. In finance, the performance of individual assets is masked by market-wide trends [17, 9]. Yet standard Principal Component Analysis (PCA) indiscriminately captures all dominant directions, failing distinguish the signal of interest from unwanted backgrounds.
Alignment-only contrastive learning and PCA+. Contrastive learning [20, 37, 11, 13] offers natural remedy. By comparing paired datasets X, X* e gaxd with samples in dimensions cancel out unwanted vari-
that share the same signal but experience different backgrounds ation. Motivated by this idea, we introduce PCA+, a vanilla contrastive PCA method that forms a "contrastive" covariance S Jn (X X + X X) from positive pairs, and applies ordinary PCA to S; At the population level, PCA+ perfectly recovers the shared signal subspace. In finite-sample, low-dimensional regimes, PCA+ again succeeds. However, as background spikes (eigenvalue (4g.) increase relative to signal spikes (Figure 1, left) or the ambient dimension grows relative (Figure 1, right), its estimated directions can drift into buckground-dominge -ominated components and miss the true signal.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

PCA PCA+ vs. PCA++
PCA vs. PCA+ vs. PCA+
1.0
1.0
Subspace Estimation Error
0.8
0.6
0.4
Sulspace
0.4
0.2
0.2
0.0
PCA
FCA
POA 15
0.0
-8- PCA
P0A
0.30
0.35
0.40
0.45
0.50
0.55
0.60
0.65
0.25
0.50
0.75
1.00
1.25
1.50
1.75
Relative Strength (
Aspect Ratio (din)
for standard PCA, PCA+, PCA++ Results for Example 3.3. Left: (varying relative strength of the signal /4.1/ (7..) As background strength grows, PCA+ deteriorates sharply while PCA++ keeps its error uniformly low. Right: (varying aspect ratio d/n) Across all regimes, PCA++ outperforms both PCA and PCA+.
Figure 1: Subspace estimation
Hard uniformity constraint and PCA++ To guard against this failure mode, propose PCA++ which augments the contrastive objective with a "hard uniformity" constraint [50]: the projected features must have identity covariance. Equivalently, PCA++ solves
VV( VeR
max
s.t. v
(1.1)
where S, = XXX X is the usual sample covariance. This constraint-forcingg the learned features to be uniformly distributed-ace like an "even-spread" regularizer, protecting the signal representation from distortion by strong background noise. We show that PCA++ admits a closed-form solution via generalized eigenproblem and remains stable (with low-rank truncation of S,)) even when d > Empirically, PCA++ maintains tight alignment with the ground-trun signal across a wide range of d/n, outperforming both standard PCA and PCA+ (Figure 1).
Theoretical guarantees. We analyze PCA+ and PCA++ under a tractable contrastive factor model where each paired observation decomposes orthogonal signal and background. Our main results include: (1). Population-leve consistency of PCA+, plus umite-sample bounds on its subspace error-and and a precise characterization of when strong background spikes overwhelm it. (2). Exact high-dimensional asymptotics for PCA++ in two regimes: (i) Fixed aspect ratio d/n - c > O, under Baik-Ben Arous-Péché (BBP) detectability condition [5]. we derive a closed-form limit for the subspace error; (ii) Growing-spite regime: when each spike scales with d/n, the limiting subspace estimation simplifies to function of the weakest signal's 's effective signal-to ugnal-to-noise-raue noisoc-rrrtio. These analyses quantify how the uniformity constraint regularizes against background interference.
Broader impact on contrastive learning. Our work also sheds light on an open question in contrastive learning theory: the practical role uniformity. Recent theoretical results [50, 39] show that contrastive losses naturally encourage uniformity while preserving meaningful class structures. Yet, the precise theoretical reason uniformity improves downstream performance aeffomance-ppariiulallly high-dimensional wwwirrnmeent-remmaiee unclear. We rigorously show how explicit uniformity constraint can defend against structured noise high-dimensional regimes- why uniformity enhances robustness and generalization.
noisy,
In summary, our contributions the community include:
A novel contrastive PCA framework that leverages paired observations to perfeetly isolate shared signal subspaces at the population level and provably bounds finite-samin sameesssmle subspace recovery eиror under a tractable linear factor model.
2. A hard uniformity-ccontrainnee PCA++ algorithm that enforces identity covariance on pro- jected features, admits a closed-form generalized eigenproblem solution, and remains stable even when background eigenvalues and ambient dimension grow arbitrarily large.
3. Exact high-dimensional asymptotic characterizations of PCA++ in both fixed aspect-ratio and growing- regimes, yielding closed-form limits for subspace estimation error that quantify how uniformity regularizes against structured noise.

---

4. A theoretical explanation for uniformity's 's power in contrastive learning, rigorously show- ing how enforcing feature dispersion defends signal recovery from strong background interference and clarifies uniformity's 's role as robust regularizer.
1.1 Related work
Foundations of contrastive learning. Early theory framed InfoNCE through mutual information (MI) maximization [37. 31, and initial analyses linked the contrastive objective to downstream class separability guarantees [43]. But MI alone proved insuffciint-tighher MI bounds sometimes hurt performance 48]. A major advance came with the aligument- uniformity framework of Wang and Isola [50], showing that InfoNCE naturally pulls positive pairs together while spreading features uniformly on the hypersphere. This insight inspired methods like Barlow Twins [52] and VICReg [7], which explicitly shape feature covariances, and it was further extended by Chen et al. [12), who parameterized alignment and uniformity terms directly. More recently, Parulekar et al. [39] proved that InfoNCE promotes uniformity in the population limit, and other work has highlighted how inductive biases steer the final representations [42, 22]. Yet precise, theory-grounded explanation for why uniformity bolsters booustness-especialle under structured noise-remains open, and motivates our current study.
miigument-uniformiiy
Spectral views. Another strand casts contrastive learning as a form of spectral learning. The spectral contrastive loss [23, 24] directly manipulates feature covariances, and in fact it and InfoNCE share the same population solution [29]. This spectral view connects CL to graph-based dimensionality reduction (e.g. t-SNE and spectral clustering) [47].
Linear models and high-dimensional analysis. In parallel, tractable linear and asymptotic models have shed light on CL's strengths. For instance, Ji et al. [28] showed that CL recovers latent signal directions more reliably than classic unsupervised methods, while Bansal et al. [6] used Gaussian mixtures to prove InfoNCE can identify optimal low-dimensional subspaces. Our work also builds on tractable model, embedding linear contrastive factor model in high-dimensional random matrix theory to isolate how a hard uniformity constraint protects signal recovery under strong, structured backgrounds.
foreground-backkrruund cPCA. A significant line of work, initiated by cPCA [1], seeks directions of high variance in foreground dataset that absent in separate background dataset. This methodology, including subsequent probabilistic and sparse extensions [33, 8, 54, 53), is fundamen- tally designed for case-control settings in which a pure background dataset is explicitly available. Formally, these methods take as input two distinct datasets: a target dataset X, which contains both signal and nuisance variation, and background-only dataset Y. which contains only nuisance variation. In contrast, our setting is structurally different. The proposed methods (PCA+ and PCA++) address the positive-pair scenario, which is central modern self-supervised learning (e.g., SimCLR [11]. MoCo [25]). Instead of relying on a clean background sample. we are given paired observations X)X*) that share the same latent signal but are corrupted by independent background variations. The objective is to identify directions that are shared across positive pairs, i.e., invariant to nuisance variability, without requiring an explicit background dataset. See additional discussion in Appendix J.
Canonical correlation analysis (CCA), Although there is conceptual connection between our work and CCA [26]. the two approaches differ fundamentally. Both aim to identify shared structure between paired datasets, but they impose different objectives and constraints, which leads to substantial performance differences in noisy, high-dimensional settings. A detailed discussion of the relationship to CCA is provided in Appendix K.
2 Contrastive Factor Model
We observe paired sigh-dmmensional samples ((x, x, x;)) generated by a contrastive factor model, which decomposes each point into a shared signal, an independent background, and noise:
X
Aw
Bh, + S
Aw; + Bh
i= i=
(2.1)
where E oa and B Rdx have orthonormal columns spanning the signal and background subspaces, respectively; w; B is the shared signal factor, while h, hf e R" are the independent background factors; E,, e/ E Rd are independent random noise.
3

---

Equivalently, we may write
AA. Vaaavas). B = V val............................. AR VB.m
each column Va.j (resp. Vn.j) ) is a principal direction with variance AA, (resp. AB.). )) By contrasting each pair (Ny xf), which share the same signal Aw; but differ in background Bh; Bh, and noise, goal is to isolate and recover the low low-dimensional signal subspace spanned by the columns of A.
To make the model analytically tractable, we impose three assumptions:
Assumption 2.1 (Orthogonal signal and background) The column spaces of the signal and back- ground looding matrices are mutually orthogonal, i.e., span(A) 1 (B)
Assumption 2.2 (Gaussian latent factors) Wi across i.
NOO.)). h, K N(O. Im), all independent
Assumption 2.3 (Isotropic noise) e, N(O, La), independent of 2a wi, W
On the orthogonality assumption. Orthogonality assumption is made primarily for analytical tractability by forcing the signal and background into disjoint subspaces. Our framework's 's core mechanism is, however, robust to its violation. The key insight is that even if signal and background subspaces overlap, the contrastive energy (as analyzed in Lemma F.1) of the shared directions remains strictly positive, while that of pure background directions is zero. This allows PCA++ to distinguish the full signal space from the background. The main effect of overlap is a reduction in the generalized eigenvalue for the shared directions, but they remain detectable.
the Gaussian latent factor and noise assumption. This assumption was also made for analytical convenience, as it allows for the clean derivation of exact constants and closed-form error rates. However, we expect the core results to hold more generally for sub-Gaussian distributions. Many of the key results from random matrix theory that rely (e.g.. in Lemmas G.4 and G.5) have well-known extensions beyond the Gaussian case, often requiring only a few finite moments.
On the isotropic noise assumption. We fix the noise variance to one without loss of generality, since any other scale can be absorbed into the singular values of and B. We discuss the potential to relax these assumptions in Appendix E.
3 PCA+: Contrastive PCA via Alignment Only
key insight from [50, 12, 39] is that contrastive objectives implicitly promote two complementary geometric forces in representation space: Alignment (bringing positive pairs close), and Uniformity (evenly spreading all features on the hypersphere). Specifically, for a feature map f X + R general spectral contrastive learning objective [24] with uniformity teafformitt-weighhtt r 0s
(f) = (t.)) [1100) f(x") [ r [F00000000000
(3.1)
In this work, consider closely related objective, adopting the alignment term from [23]:
L(f
E f()) + (x.x")
E [())
(3.2)
aligument
uniformity
Both objectives (3.1) and (3.2) capture the alignment-uniformiy trade-off observed in many popular contrastive learning frameworks [11, 46, 51].
In this section, we "turn off" uniformity (r = in (3.2) and study linear encoder that optimizes only alignment under our contrastive factor model; the following section then analyzes the effect of enforcing uniformity. We show that an "alignment-only" method (PCA+)-i.e. applying PCA to the contrastive covariance-can successfully recover the signal subspace when background variation mild. However, in high-dimensional regimes with strong background noise, this pure-alignment approach can fail completely. This failure highlights the necessity of uniformity-ty constraint: by

---

enforcing feature dispersion, we regain robust signal recovery even when background components overwhelmingly dominate.
To isolate the role of alignment contrastive learning, we study a linear encoder
f(x) V'x, Ve RR
and maximize alignment between positive pairs (x,x;) without a uniformity objective (r 0) in (3.2):
min - (V V= 11
o max tr ((xxxxxV
v'
where - 2i= (V'x) (V x7) = x= focused setup, orthogonality constraint VTV = I is necessary. Note that _ x is not be symmetric, so its eigenvalues can complex and eigenvectors need not be orthogonal. A simple fix is to symmetrize
is the empirical version of EE((VEVVT xVT((xx))) and X (XI...., (x)....... x with X E R To prevent representational collapse in such an alignment-
S = 2n
(x x+
which is real-symmetric and shares the same population expectation (shown in Theorem 3.1). Plug- ging $ into the alignment objective yields
max tr(V V),, VIVEL
(3.3)
which exactly PCA on the "contrastive" covariance S We call this method PCA+-our baseline that captures pure alignment under orthogonality constraint.
The estimated signal subspace is taken be the span of the top eigenvectors of S We first show that S is an unbiased estimator of the true signal covariance:
Theorem 3.1 (Unbiasedness of the contrastive covariance estimator) Under Assumptions 2.1- 2.3, the contrastive covariance estimator S satisfies E[S;] AAT
In other words. in expectation, the contrastive covariance matrix S perfectly cancels out both background and noise, recovering the population signal covariance AA
Having established the behaviour of PCA+ estimator at the population level, we now turn to its finite- sample performance. Our main result is non-asymptotic upper bound on the subspace estimation error of PCA+, incurred by applying PCA to the contrastive covariance matrix S Denote the true subspace span by columns of A by UA = span(v'.. Yaxl and let UA be our estimator. We measure the estimation error between U4 and UA via their subspace distance based on the principal angles between UA and UA The definition the subspace distance is provided Appendix A.
Theorem 3.2 (Finite-sample performance of PCA+) Suppose Assumptions 2.1 hold. and that the sample size obeys:
C
(K2h, md max(k, m).. (da.1 AB.1 + 1)) log + d),
A.
for a sufficiently large universal C. Then, with probability at least 1 O((n + d) the distance between the estimated subspace UA and the true subspace UA satisfies:
Dist(UA,
$
1 V
-
dg
m +
Vasidan
max(k,
Ask
+ (Vaai
log 1/2 (2 d).
This bound shows that, when the smallest signal spike A4,8 is large relative to the largest background spikes dg. and the noise level, PCA+ recovers the signal subspace accurately from finite data. However, if the background strength grows or the eigengap AA shrinks, the error can blow up. Indeed, in the case below, standard PCA+ fails completely to align with the true
extreme signal:
5

---

Example 3.3 (One-signal, one-background) Let k = m 1 in model (2.1) and
A [dgnes and B = |
where e1 == (I.0....,0) and e2 (0, (0.1,.....,0) 0) are the first two standard basis vectors in R4.
Theorem 3.4 (Failure under strong background) Under this model with d/n (0, +00) as n, d + +00, the leading PCA+ direction D1 S satisfies
AA111
lim (f))2)) S a
VAB.IC
This result highlights& a limitation of PCA+: when the leading background eigenvalue becomes sufficiently large, Vda. AA.1/ Vc, the top empirical eigenvector fails to align with the true signal direction, with the alignment remaining bounded away from one. To address this issue, we propose constrained contrastive PCA method in the following section that incorporates uniformity suppress background interference and achieve robust signal recovery even under strong background conditions.
Enforcing Uniformity: PCA++ and Its Analysis
The previous section exposed a flaw of pure- alignalig PCA (PCA+): it can no longer recover the true signal when background noise is too strong. We now show that enforcing perfect uniformity- ie. constraining the projected features to have identity covariance acttc powerful regularizer, neutralizing even very strong background.
Concretely, introduce the hard-unin saudd-uniformity contrastive PCA (PCA++):
Lr(V' VVV.
s.t.
VV 4, untformity
(4.1)
VERUM
alignment
where S, X X. The hard constraint VTS,V enforces 1 (Vx)(VV i.e. perfect uniformity of the projected features. (3.1).
The PCA++ objective (4.1) can be solved in one shot via a single generalized eigenvalue decomposition (see proof in Appendix B.1), making both conceptually simple and computationally efficient. Specifically, one computes
v A,S, j [1,.......dd
(4.2)
for (dj, vyl. where s 1o2 (XT X* and S,, 1 The top-k generalized eigenvalues Vi...., Vt (those with largest 2,) form the columns of the PCA++ solution e Raxt When S, is invertible, (4.2) reduces to the ordinary eigenproblem for S For implementation details see Algorithm 1 in Appendix B.2, and for full derivation of generalized eigenvalue solvers refer to [38, 19].
In very high dimensions, 5, is often severely ill-conditioned, so solving (4.2) directly may be numerically unstable. Figure 2 (left) shows how that, even with the uniformity constraint, the generalized eigenvectors can drift away from the true signal when d > n. To restore stability, we replace S, by its rank-s approximation (S,) = E where (dg, v))= are the top-s eigenpairs of S n. and is tuning parameter. We then solve
ITVV), VERUNE
VT(S,).V== I.
(4.3)
By construction, (S,); well-conditioned on its s-dimensional leading subspace and discards the small, unstable directions, enforcing uniformity only where S,, is reliable. As Figure (right) shows, an appropriate s dramatically reduces error, whereas including too many dimensions re-introduces noise. Computationally, one simply replaces S, with (S s), in Algorithm I to obtain stable solution. We provide detailed guidelines for choosing the value of this crucial hyperparameter Appendix C.
Furthermore, this truncation strategy offers significant computational advantage. Since we only need find the top-s generalized eigenvectors, we can avoid a full decomposition and instead employ efficient iterative solvers, such as the implicitly restarted Lanczos method (IRLM). This makes the

---

Untruncated + vs. Truncated PCA+
PCA++ with Varying Truncation
Subspace Estimation Ertor
0.8
0.6
0.4
0.4
0.2
0.2
PCA
Truncited PCA tam 6.a
Trureted Pilce 6201
0.0
Untrnnaeeed PlAce
Trincted RCA+ (6:21
0.0
Trunsded
0.25
0.50
0.75
1.00
1.25
1.50
1.75
0.25
0.50
0.75
1.00
1.25
1.50 1.75
Aspect Ratio (din)
Aspect Ratio (din)
Figure 2: Effect of covariance truncation PCA++ Results for Example 3.3. Left: As d/n increases, truncated PCA++ remains stable and accurate while untruncated PCA++ deteriorates sharply. Right: Truncated PCA++ with varying truncation ranks (fixed = 2: or as 0.1d, 0.2d, 0.4d of feature dimension d).
approach scalable to very high-dimensional data, and the details of its computational complexity are discussed in Appendix D.
will analyze the truncated uniformity-constrane contrastive PCA two complementary high- dimensional asymptotic regimes using tools from random matrix theory and the spiked covariance model [30, 5, 40]:
Fixed aspect ratio regime. The signal eigenvalues (Aa.)) and background eigenvalues (Ag.i remain constant, while the aspect ratio d/n - c 0 both n, d co.
II. Growing-spike regime. Both signal spikes AA, and background eigenvalues ARj diverge with n.d co, and the scaled aspect ratios udy. CA. and d ndg, CBJ, with CA.1
CAA < co and CB.1 <... CB. < 0o.
Our analysis for the following high-dimensional regimes builds upon the asymptotic alignment between sample and population principal components of S, = f X under model (2.1), as formalized in Lemma G.4. Denote the true population spikes and directions of E[xx ] by [O Va
and j(AR, ) |=1 sorted in descending order, and let [(λ,,, PA.)) and 1(AR,, Pa.)) be the corresponding sample eigenpairs obtained by matching each empirical eigenvalue of S,, to its nearest population counterpart magnitude. This matching preserves the separation between signal and background components in our high-dimensions analysis.
I. Fixed aspect ratio regime. We now study the regime d/n - > 0 as both n, d - co and ask: when do the sample eigenvalues "stick out" of the Marcenko-Pastue bulk so that both signal and background directions remain identifiable?
Assumption 4.1 (Detectable spikes) The population eigenvalues (dA.)) and (Aa,) all dis- finct and satisfy Aa.j. An.j 2 vc for every j.
Remark. This condition corresponds to the classical BBP threshold, fundamental requirement in -igh--imensional PCA for spiked covariance models. When the signal strength falls below this threshold, the leading eigenvalues of the sample covariance matrix are absorbed into the noise bulk of the Marcenko-Pasiee distribution, rendering the signal statistically undetectable [5, 40]. This phenomenon reflects an intrinsic information-theored limit of signal detection via PCA in the high- dimensional regime, rather than limitation specific to our PCA+ method. Under this assumption, the hard-unite hard-uniformity constrained PCA (PCA++) admits simple and exact characterization of its asymptotic subspace error:
Theorem 4.2 (Asymptotic subspace error under hard uniformity) Under Assumptions 2.1-2.3
and 4.1, and 2 k. let Ul be the top-k subspace returned by the hard-uniformity PCA in (4.3) and
7

---

UA be the true population signal subspace. Then as n, d - +o0 with n/d c (0, +00),
disu(UA u28 - 1 - cd
a.s.
1 cd
We can see that when the weakest signal spike dAA Vc, the error 1- 1-
is small, reflecting
1tc
accurate recovery; As dax Ve grows, the error vanishes. Conversely, increasing the aspect ratio d/n makes recovery harder and increases the limiting error. Crucially, this result highlights the power of the uniformity constraint. Unlike the pure-alignment method PCA+ (Example 3.3 and Theorem 3.4), which can be overwhelmed by strong background spikes, the hard-uniformity estimator PCA++ continues to recover the signal subspacc-albed with controlled nigh-dimensional bven when background eigenvalues grow arbitrarily large.
IL Growing-spike regime. In the growing nrowing-aspect-reg aspect-ratio regime, where the spikes grow proportion- ally to d/n such that their "effective signal-to-noiss ratios" converge to finite, distinct constants, no longer require the lower-bound from Assumption 4.1: the growth of the eigenvalues ensures they remain asymptotically separable from the noise bulk.
Assumption 4.3 (Distinct growing spikes) The population eigenvalues (daJ) jat and " are all distinct.
This assumption was made primarily for analytical convenience. With distinct eigenvalues, we can cleanly map each sample eigenvector to a population counterpart using existing results (Lemmas G.4 and G.5). Without this assumption, if set of eigenvalues were identical, we would instead estimate subspace for those directions. Within that estimated subspace, it would be difficult to distinguish which basis vectors correspond to signal and which to background just by looking at S, alone, making it harder to explain how the uniformity constraint works per-direction basis. The key insight is that even the standard covariance Sn has degencrate subspaces (i.c., multiple identical eigenvalues mixing signal and background components), the contrastive covariance S resolves this ambiguity. Since S has asymptotically zero energy on pure background directions, the still correctly identify the signal subspace and separate from the background. We discuss the potential to relax this assumption in Appendix E.3.
Remark.
generalized eigenvalue problem
The following theorem shows that, the limiting subspace estimation error is simple function of the weakest signal's effective SNR, CAA nāns"
Theorem 4.4 Under Assumptions 2.1 and 4.3, and 2 k, as n, +00,
dist uA)²
CAR + CAk
a.s.
As CA* decreases (i.e. the signal grows stronger relative to the ambient dimension d/n), the error vanishes. Moreover, by plugging Ca.j ada into the error expression from Theorem 4.2, one recovers exactly the same form in Theorem 4.4, confirming full consistency between the two high-dimensional analyses.
Experiments
Simulation studies. We designed suite of simulations to test our theoretical predictions and to compare standard PCA, the alignment-only method PCA+, our uniformity-constree (PCA++), and the corresponding high-dimensional theory. In every scenario, paired samples were drawn from the linear contrastive factor model (Eq. (2. 1)), and performance was quantified by the sine of the principal angle between the estimated and true signal subspaces, averaged over 50 independent
PCA++ +
runs.
We first explored the role of truncation in stabilizing PCA+ when 1000 and AA.1 10, A.,1 500. Solving the generalized eigenproblem without truncation leads to erratic signal estimates once d/n exceeds about I (Figure 2, left). However, projecting the sample covariance onto its top-s subspace
8

---

with s 2 fully restores recovery. We then swept over (2, 0.1d, 0.2d, 0.4d) and found that moderate truncation (e.g. s 0.Id) achieves the best trade-off: discards the unstable noise directions while preserving enough dimensions to enforce the uniformity constraint (Figure 2, right).
Next, turned the asymptotic regimes characterized by Theorems 4.2 and 4.4. Fixing 500 and varying the aspect ratio d/n, we embedded five-dimensional signal subspace (variances [50, 25, 20, 15, 10) in the first five coordinates and an orthogonal liveddimensionll background (variances [500, 400, 300, 200, 100]) in the last five. Applying PCA++ with truncation rank s 10, we again computed the sine of the largest principal angle the true signal, averaged over fifty runs.
Under the fixed-aspeet-ratte regime, the empirical PCA++ errors trace our closed-form curve almost exactly, while the alignment-only PCA+ method diverges d/n increases (Figure left). In the growing-spike regime-whie -whnn both d and spike variances are multiplied by ten-PCA++ continues adhere the simple limit THe but PCA+ collapses (Figure 3, right). These results confirm that theoretical predictions capture the precise high-dimensional behavior of PCA++ ++ in both regimes. Further details and additional simulation results can be found in Appendix H.
Fixed aspect ratio regime
Growing-spike regime
1.0
1.0
Subspace Estimation Error
Sulspace Extimation
0.4
0.4
0.2
0.2
0.0
Theory Prsem
0.0
PCA+
PCA Theory Prodlned
0.25
0.50
0.75
1.00
1.25
1.50 1.75
0.25
0.50
0.75
1.00
1.25
1.50
1.75
Aspect Ratio (din)
Aspect Ratio (cin)
Figure 3: Empirical validation of theoretical predictions for PCA++. Left: Validation in the fixed aspect ratio regime for Theorem Right: Validation the growing-spinn swwiing-spike regime for Theorem 4.4.
Corrupted MNIST Data. To visually assess signal disentanglement from structured background noise, we created synthetic dataset of 5,000 paired images by superimposing MNIST digits [16] or '1') onto distinct ImageNet [15] "grass" patches. Figure 4 compares the 2D embeddings obtained from standard PCA, PCA+, and PCA++. While standard PCA fails to separate classes and PCA+ shows only partial, misaligned separation, PCA++ clearly distinguishes the digits, with separation predominantly along its first learned eigenvector. This visually confirms that the uniformity constraint in PCA++ enables effective isolation of the true digit signal from the background.
PCA
PCA+
PCA++ (s= 10)
Pigeztin unto Exebreededrr
Digit
Digit
Digit 0
Digit1
Digit O
Digit
10.0
-15
-75
-50 -2.5 610 25
53
Figure 4: embeddings noisy higit-over-grass images, standard
to separate classes. Contrastive PCA+ shows partial, misaligned separation. In contrast, our PCA++ achieves clear class separation predominantly along its first eigenvector, highlighting its superior ability to isolate the true signal and background noise.
Single-cell RNA sequeneing data. We evaluated PCA++ on single-cell RNA-seq data from [31], comprising 14,619 control and 14,446 After matching 9,268 cells per condition via donor identity and local structural alignment, we extracted the top 50 components with both PCA and PCA++ and visualized each embedding (in
PBMCs across eight immune cell types.

---

Figure 5) using UMAP [35]. Standard PCA often separates control and stimulated cells of the same type, even when transcriptional changes are minimal, while PCA++ yields tightly overlapping clusters for invariant populations (e.g., CD4 T cells, B cells, NK cells), while still reflecting biologically meaningful dispersion in responsive cell types (e.g., monocytes Figure of the appendix). This demonstrates PCA++'s 's ability to isolate condition-invarian Further details and additional simulation results can be found in Appendix I.
structure in complex single-cell data.
(a)
(b)
(c)
soll san lots DAY!
-10
180 248 129)
CP1
15. 1A00
CD4 2811
-10
-12
16
-10
-12
(cl)
(e)
(PCA+e)
(f)
CD4 Colls (311
Niccelk
Bubb (c2) Co (rinn)
MBC ocho bein))
-15
-16
-15 -10
-15 -10 0 10 15
Figure 5: PCA vs. PCA++ embeddings. We apply PCA and PCA++ to matched control and stimulated PBMCs (9,268 cells each) from the [31] dataset and visualize the top 50 components using UMAP. (a-c) show PCA embeddings of CD4 T cells. B cells, and NK cells. where control and stimulated cells are often separated despite minimal biological response. (d-f) show the same cells under PCA++, where alignment across conditions improves, highlighting PCA++'s ability to isolate stable, condition-invee invaarian structure.
Discussion and Future Work
This work revisits classical dimensionality reduction through modern contrastive perspective, showing that alignment alone is insufficient for robust signal recovery in high-dimensional settings with strong structured noise. We demonstrate that incorporating an explicit uniformity constraint-a fundamental ingredient of recent contrastive learning nuisance variation. The proposed method, PCA++, recovers latent signal subspaces in regimes where both standard PCA and alignment-only contrastive PCA (PCA+) provably fail.
principled safeguard against
Uniformity robustness mechanism. A key conceptual contribution of this work is clarifying the role of uniformity in contrastive learning. Prior studies emphasized its geometric effects (e.g., feature dispersion and isotropy), but its statistical role under structured noise was unclear. Our analysis shows that uniformity suppresses directions aligned with dominant background spikes, acting as a spectral filter that reweights principal directions based on their gegnal-tobbcckrooue ratio rather than their raw variance.
Broader implications and future work. Although studied in linear setting, the insights extend beyond PCA: (a) Self-superviser learning. PCA++ formalizes the benefit of uniformity in SimCLR- style losses and explains why explicit control of feature dispersion improves robustness to nuisance variation. (b) Multiview learning. Our paired-data setting parallels CCA, but we show that variance- based alignment alone is insufficient under antagonistic background structure. The success of PCA++ suggests that modern representation learning in multiview settings may benefit from hard covariance constraints. Future extensions include contrastive sparse PCA [55], contrastive kernel PCA [44] and tensor PCA [2] for nonlinear/multiway data, and spectral clustering variants [36] for community detection. These directions promise expand uniformity-constrr high-dimensional settings.
inference across diverse
10

---

Impact Statement
This paper aims advance the field of machine learning. While there may be societal impacts, none require specific attention here.
Acknowledgments
We sincerely thank Yixuan Li for invaluable assistance with the real data analysis. This work was partially supported by NSERC Discovery Grants NGGPI--2024006880 and (GPIN-2018-00 (018-06484) and FRQNT Team Research Project Grant (FRONT 327788).
References
[1] Abid, A., Zhang, M. J., Bagaria, V. K., and Zou, J. (2018). Exploring patterns enriched in dataset with contrastive principal component analysis. Nature Communicutions, 91):2113.
[2] Anandkumar, A., Ge, R., Hsu, D. J., Kakade, S. M.. Telgarsky, M., et al. (2014). Tensor decompositions for learning latent variable models. Journal of Machine Learning Research, 15(1):2773-2832.. 6
[3] Bachman, P., Hjelm, R. D., and Buchwalter, W. (2019). Learning representations by maximizing mutual information across views. Advances in Neural Information Processing Systems, 32. 1.1
[4] Bai, Z. D. and Yin, Y. Q. (1993). Limit of the smallest eigenvalue of large dimensional sample covariance matrix. The Annals of Probability, 1275 - 1294. G
[5] Baik, J. and Silverstein, J. W. (2006). Eigenvalues of large sample covariance matrices of spiked population models. Journal of Multivariate Analysis, 9366): 1408. 1, 4, 4, G.4
[6] Bansal, P., Kavis, A., and Sanghavi, S. (2024). Understanding contrastive learning via Gaussian mixture models. arXiv preprint arXiv:2411.035 1.1
[7] Bardes, A., Ponce, J., and LeCun, Y. (2021). VICReg: Varianceinvvriaccccovvaiiee regulaa ization for self superviee learning. arXiv preprint arXiv:2 (09000 1.1
[8] Boileau, P., Hejazi, N. S., and Dudoit, S. (2020). Exploring high-dimensional biological data with sparse contrastive principal component analysis. Bioinformatics, 36(11):3422-3400 1.1
[9] Campbell, J. Y., Lettau, M., Malkiel, B. G., and Xu, Y. (2001). Have individual stocks become volatile? an empirical exploration of idiosyncratic risk. The Journal of Finance, 56(1):1-43.
[10] Candès, E. J.. Li, X., Ma, Y., and Wright, J. (2011), Robust principal component analysis? Journal of the ACM, 58(3):1-37. I.2
[11) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. (2020). A simple framework for contrastive learning of visual representations. International Conference on Machine Learning, pages 1597- 1607. PMLR. 1, 1.1,3
[12] Chen, T., Luo. C., and Li, L. (2021a). Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 34:1 1834-1184445 1.1,3
[13] Chen, X. and He, K. (2021). Exploring simple siamese representation learning. In Proceedings of the IEEE/VVF Conference on Computer Vision and Pottern Recognition, pages 15750-15758.
[14] Chen, Y., Chi, Y., Fan, J.. Ma, C., al. (2021b). Spectral methods for data science: A statistical perspective. Foundations and Trends in Machine Learning. 14(5):566-88) 806. G.2
[15] Deng, J., Dong, W., Socher, R., Li, L.JJ., Li, K., and Fei-Fei, L. (2009). ImageNet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. IEEE, 5, I.I

---

[16] Deng, L. (2012). The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142. I.1
[17] Fama, E. F. and French, K. R. (1993). Common risk factors in the returns on stocks and bonds. Journal of Financial Economics, 33(1):3-56.) 1
[18] Fortin, J.-P., Parker, D., Tunç, B., Watanabe, T., Elliott, M. A., Ruparel, K., Roalf, D. R., Satterthwaite, T. D., Gur, C., Gur, E., al. (2017). Harmonization of multi-site diffusion tensor imaging data. Neuroimage, 161:149-170.
[19] Ghojogh, B.. Karray, F., and Crowley, M. (2019). Eigenvalue and generalized eigenvalue problems: Tutorial. arXiv preprint
4
[20] Gutmann, M. and Hyvärinen, A. (2010). Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of The International Conference on Artificial Intelligence and Statistics, pages 377.304.MML. Workshop and Conference Proceedings.
[21] Haghverdi, I.,, Lun, A. T., Morgan, M. D., and Marioni, J. C. (2018). Batch effects in single-cell RNA- sequuncing data are corrected by matching mutual nearest neighbors. Nature Biotechnology, 36(5):421-47.. 1
[22] HaoChen, J. Z. and Ma. T. (2022). theoretical study of inductive biases in contrastive learning. arXiv preprint 1xXi::2211.16699 1.1
[23] HaoChen, J. Z., Wei, C., Gaidon, A., and Ma, (2021). Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems, 34:5000-500 5011. 1.1,
[24] HaoChen, J. Z., Wei, C., Kumar, A., and Ma, T. (2022). Beyond separability: Analyzing the linear transferability of contrastive representations related subpopulations. Advances Neural Information Processing Systems, 35:26889-26902. 1.1,
[25] He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. (2020). Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 9729-9738. 1.1
[26] Hotelling, H. (1936). Relations between two sets of variates. Biometrika, 28(3-)):321-377 1.1
[27] Hubert, L. and Arabie, P. (1985). Comparing partitions. Journal of Classification, 2(1):193-218.
I.2
[28] Ji, W., Deng, Z., Nakada, R., Zou, J., and Zhang, L. (2023). The power of contrast for feature learning: A theoretical analysis. Journal of Machine Learning Research, 24(330):1-78. 1.1
[29] Johnson, D. D., Hanchi, A. E., and Maddison, J. (2023). Contrastive learning can find an optimal basis for approximately view-invariant functions. In The International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. 1.1
[30] Johnstone, I. M. (2001). On the distribution of the largest eigenvalue in principal components analysis. The Annals of Statistics, 29(2):295-33) -27299953777..0 4
[31] Kang. H. M.. Subramaniam, M., Targ, S., Nguyen. M., Maliskova, L., McCarthy, E., Wan, E., Wong S., Byrnes, L., Lanata, C. M., et al. (2018). Multiplexed droplet single-cell RNA-sequencing using natural genetic variation. Nature Biotechnology. 36(1): (6(1):89-994 94. 5.2
[32] Leck, J. T., Scharpf, R. B., Bravo, H. C., Simcha, D., Langmead, B., Johnson, W. E., Geman, D., Baggerly, K., and Irizarry, R. (2010). Tackling the widespread and critical impact of batch effects in high-throughput data. Nature Reviews Genetics, 11(10):733-73,
[33] Li, D., Jones, A., and Engelhardt, B. (2020). Probabilistic contrastive principal component analysis. arXiv preprint arXiv:2012.07977, 1.1
12

---

[34] McAllester, D. and Stratos, K. (2020). Formal limitations on the measurement of mutual information. International Conference Artificial Intelligence and Statistics, pages 875-884. PMLR. 1.1
[35] McInnes, L., Healy, J., and Melville, J. (2018). UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03420. 5, 1.2
[36] Ng, A., Jordan, M., and Weiss, Y. (2001). On spectral clustering: Analysis and an algorithm. Advances in Neural Information Processing Systems, 14. 6
[37] Oord, A. v. d., Li, Y., and Vinyals, O. (2018). Representation learning with contrastive predictive coding, arXiv preprint arXiv: 1, 1.1,3
[38] Parlett, B. N. (1998). The symmetric eigenvalue problem. SIAM. 4
[39] Parulekar, A., Collins, L., Shanmugam, K., Mokhtari, A., and Shakkottai, (2023). Infonce loss provably learns cluster-preserving representations. In Neu, G. and Rosasco, L.. editors, Proceedings of Thirty Sixth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pages 1914-1961. PMLR. 1, 1.1,3
[40] Paul, D. (2007). Asymptotics of sample eigenstructure for large dimensional spiked covariance model. Statistica Sinica, pages 1617-1642. 4, 4, G.4
[41] Salloum, R. and Kuo, C.-C. J. (2022). cPCA++: An efficient method for contrastive feature learning. Pattern Recognition, 124:108378.
[42] Saunshi, N., Ash, J., Goel, S., Misra, D., Zhang, C., Arora, S., Kakade, S., and Krishnamurthy, (2022). Understanding contrastive learning requires incorporating inductive biases. In International Conference on Machine Learning, pages 19250-19286. PMLR. 1.1
[43] Saunshi, N., Plevrakis, O., Arora, S., Khodak, M., and Khandeparkar, H. (2019). A theoretical analysis of contrastive unsupervised representation learning. In International Conference Machine Learning, pages 5628-500 5637. PMLR. 1.1
[44] Schölkopf, B., Smola, A., and Müller, K.-R. (1998). Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10(5):1299-133
[45] Shen, D... Shen, H., Zhu, H., and Marron, J. (2016). The statistics and mathematics of high dimension low sample size asymptotics. Statistica Sinica, 26(4):1747. G.S. G
[46] Sohn, K. (2016). Improved deep metric learning with multi-class n-pair loss objective. In Lee, D., Sugiyama, M., Luxburg, U., Guyon, I., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc.
[47] Tan, Z., Zhang, Y., Yang, J., and Yuan, Y. (2024). Contrastive learning is spectral clustering on similarity graph. In International Conference on Learning Representations. 1.1
[48] Tschannen, M., Djolonga, J., Rubenstein, P. K., Gelly, S., and Lucic, M. (2019). On mutual information maximization for representation learning. arXiv preprint (tt.://977.33222 1.1
[49] van der Maaten, L. and Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9:2579 2600. I.2
[50] Wang, T. and Isola, P. (2020). Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929-9939, PMLR.
1.1,3
[51] Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. (2018). Unsupervised feature learning via non- parametric instance discrimination. In 2018 IEEEJCVF Conference on Computer Vision and Pattern Recognition, pages 3733-3742.
[52] Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. (2021). Barlow Twins: Self-supervised learning via redundancy reduction. International Conference on Machine Learning, pages 123 12302000. PMLR. 1.1
13

---

[53] Zhang, E. and Li, D. (2025). Contrastive functional principal component analysis. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 22380-22380 1.1
[54] Zhou, Q. Gao, Q. Wang, Q. Yang, M., and Gao, X. (2023). Sparse discriminant pea based on contrastive learning and class-specificity distribution. Neural Networks, 167:775-786. 1.1
[55] Zou, H., Hastie, T., and Tibshirani, R. (2006). Sparse principal component analysis. Journal of Computational and Graphical Statisties, 15(2):265-22) 286.. 6
14

---

Appendix
PCA++: How Uniformity Induces Robustness to Background Noise in Contrastive Learning
A
Principal Angles
Denote the true subspace by UA == span(v........ VARi and let UA an estimator obtained from given learning method. We measure the estimation error between UA and Ur via their principal angles, which capture the maximal deviations between corresponding directions in the two subspaces:
Definition A.1 (Principal angles) Let U and U' be k-dimensional subspaces Rf. Let the columns of U e Baxk and U' Baxk form orthonormal bases for 'I and U', respectively. Write the singular values of U U" in descending order as 0 2 02 2 ... 2 2 0. The principal angles 8 between U and U' are defined by
Oj arccos((r)). for j I....,k.
These angles capture the worst-case misalignment between the subspaces. A common aggregate distance is
distsin.H (u, 44) sin O|| ||diag(sin ( ... sin 02)1..
where || Il is any unitarily invariant matrix norm (e.g., the operator Frobenius norm).
Remark A.2 This metric is invariant under orthogonal transformations and relates directly other subspace distances commonly employed in the literature, such as the projection Frobenius norm IUUT U'U'TlI l|.. Thus, it offers an interpretable measure to evaluate subspace estimation accuracy, particularly useful in high-ddieensoonl analysis.
B Generalized Eigenvalue Problem
B.1 Equivalence between (4.1) and (4.2)
Use variable V' instead of V in (4.1) (the reason will become clear later), and introduce symmetric multiplier Rixk for (4.1). The Lagrangian is
L(V", M) = tr(V" V*) tt[M(V' S, -4]].
Since M is symmetric, we may rewrite
L(V, M) tr(v'T 5* tr(MV' S V") + tr(M).
We differentiate L(V', M) w.r.t. V' and then set v 0, obtain
V = 2 V 2 V =
S = V.M.
(B.1)
Meanwhile, multiply V"T to both sides of (B.1), we obtain
VTT V'S, V'M
M VTS
where we used V'SSV'' = 4. The objective function we want to maximize is IT(V'S; which equal to tr(M). The matrix M V"T S is k k real symmetric matrix. Therefore, it is orthogonally diagonalizable. Let Q be k x orthogonal matrix (Q Q eQ I)) such that QT MQ = A, where diag(d,, d2...., ... d2) is a diagonal matrix containing the eigenvalues of M.
Define new matrix = V'Q. We check if V satisfies the constraint:
VTS V (V'')TSs(VV' Q V - 100'0==Q4400 Q=Q=1
So V also satisfies the constraint. Now substitute V' = VQT into equation (B.1):
S,,
Post-multiply by Q:
=
15

---

We get
V S.VA
(B.2)
we write V (Vi, F2...., Ve), then equation (B.2) can be expressed column by column as: V SySai for I,...,k.
This is precisely the generalized eigenvalue equation. Since the objective function is
ur(V" tr(M)
To maximize this sum, we must choose the k generalized eigenvectors Vj that correspond to the k largest generalized eigenvalues 2j. The solution matrix V will have these eigenvectors as its columns. Typically, the full generalized eigenvalue problem V A,S,vy is solved for all possible eigenvectors and eigenvalues. Then, these eigenvalues are sorted in descending order, d A2 ... As. and the eigenvectors Vi....,% V corresponding to the & largest eigenvalues are chosen to form the columns V (V))...,,
B.2 Generalized eigenvalue solver
To compute the hard-uniformity PCA (PCA++) in (4.1), we must solve the generalized eigenvalue problem
1 A,S Vje (l....d..
where S (XT x X X) and x Equivalently, in matrix form,
V SVA.
where V (Pi..... va) contains the eigenvectors columns, A diag(d)..... is diagonal matrix of eigenvalues.
A stable and efficient procedure is as follows:
Algorithm 1 Generalizzd eigenvalue solver
1. Compute cigendecomposition: S,V, = V,A2-
2. Compute + V,(A, + ela)-1/2 where & small regularizer to handle nearly zero eigenvalues.
3. Project the contrastive covariance: M < R S.
4. Compute eigendecomposition: MU = UA.
5. Recover generalized eigenvectors V + V and generalized eigenvalues A.
C On Selecting Hyperparameter s
The choice in truncated PCA++ controls a fundamental trade-off between the stability of the generalized eigenproblem and the effectiveness of the uniformity constraint:
A small ensures that the inverse of the truncated covariance (S,), is well-conditioned, promoting numerical stability. However, it may discard dimensions needed to enforce uniformity effectively, potentially biasing the result.
A large enforces uniformity
larger subspace but risks instability if S, is ill- conditioned, as including directions with near-zero eigenvalues can amplify noise. As shown in experiments (Figure left).
Based on this trade-off, we propose to combine the following practical strategies for selecting
16

---

Information-based criterion (lower bound): As in standard PCA, one can determine minimum s by examining the cumulative variance explained by the eigenvalues of the sample covariance S. For example, choose large enough to capture significant portion (e-g., 90%) of the total variance. This ensures the uniformity constraint operates on most meaningful directions.
Stability-based criterion (upper bound); To ensure numerical stability, one can monitor the condition number of the truncated matrix (S,),, which the ratio of its largest to its s-th eigenvalue (A1 (i((Ss)/dss a), .)). One should choose s such that this ratio remains below a reasonable threshold, avoiding severe ill-conditioning.
In our experiments (c.g., Figure 2, right), we found that PCA++ is robust across a reasonable range of
D Computational Complexity
The computational cost of PCA++ has two main components:
1. Covariance matrix formation:
We compute the sample covariance S, and the contrastive covariance S For data matrices of size d (samples features), forming these d matrices requires matrix multiplications with a complexity of O(nd)). ). In the common high-dimensional setting where n, this can optimized to O(dn²) by first computing the X Gram matrix.
2. Generalized eigenvalue problem (GEP) solution: The cost is dominated by the initial truncated cigendeccoposition of S, to rank Using iterative solver like the IRLM, as implemented in scipy, this step has a complexity of approximately O(sd²). Since the truncation rank is typically much smaller than d, this step is highly efficient.
Overall, the complexity comparable to standard PCA, primarily driven by the feature dimension
Empirical Scalability To demonstrate its practical performance, we benchmarked the runtime of PCA++ while varying the number of samples (n) and features (d). The results confirm our theoretical analysis.
Table I: Computational cost of PCA++ (in seconds)
n'd
100
1000
5000
10000
100
0,002
0.157
2.288
9,492
1000
0.003
0.204
3.781
15,666
5000
0.009
0.662
13.919
53,610
10000
0.017
1.310
26.433
100.656
Setup: Truncation rank s = 10, top k = eigenvectors estimated. Benchmarked on an Intel Xeon CPU @ 2.20GHz.
The benchmarks show that the runtime is dominated by the feature dimension d, scaling quadratically, while it scales nearly linearly with the number of samples n. This profile makes PCA++ computation- ally feasible for typical high-dimensional datasets with tens thousands of features, confirming its practical scalability.
E Discussion of Theoretical Assumptions
On Orthogonal signal and background (Assumption 2.1)
Why the core mechanism is robust to overlapping subspaces: The key insight, enabled by the linearity of our factor model in (2.1), is that we can analyze the behavior of the covariance matrices shared basis. Even with overlap, the contrastive covariance S isolates signal components, while the standard covariance S, accumulates variance from both signal and background in the shared directions.
17

---

Let's sketch this for simple case. Because the model is linear, we can define an orthonormal basis that accounts for the overlap. Suppose span(A) and span(B) share a single direction Vo- We can decompose the subspaces as:
Signal space: span(A) is spanned by (vo, VA... VAR-II, where vo is the shared part and VA. are the pure signal directions, orthogonal to Vo-
Background space: span(B) is spanned by (Vo, Vn,1. Vam-th ih. where vas the pure background directions.
The population covariances then become:
Contrastive covariance:
= A
ANAAN F21
Here, A4.0 is the signal variance in direction Vo- Thus, contributions from the pure background directions (Va,)) are still averaged out. The resulting expectation remains spanned only by the signal-related directions (vo, Va.l. VAR-I). The shared direction Vo is not cancelled.
Standard covariance:
E[S (Ma,0 AB
Σ AAAaaaa
Σ AnaVasvos
Here, Ano is the background variance in direction vo- The variance in the shared direction Vo is amplified by both signal and background components.
Intuition-why PCA++ is robust to violations of orthogonality assumption: The robustness of PCA++ stems from how the generalized eigenvalue problem S A interacts with these modified covariance structures. We can analyze this through the lens of the asymptotic contrastive energy for each direction, as defined in our analysis for Lemma F.1.
Pure background directions (Va,)): : The contrastive energy S vR. remains asymptoti- cally zero, so these directions are filtered out.
Shared direction (vo): The contrastive energy 5 strictly positive due to the signal component. However, its variance in the standard covariance is now amplified by both signal (A4,) and background (dg,) components.
The resulting contrastive energy for shared direction will be smaller than that of pure signal spike, but it will still be bounded away from zero. Therefore, the PCA++ objective still detects the shared direction as part the signal space. The fundamental mechanism-iisolaing all directions with intact.
non-zero contrastive
Empirical validation with overlapping subspaces: To provide strong empirical evidence, we new simulations with non-orthogonal signal and background subspaces and will add these to the Appendix. We ran new simulations based on the setting in Section H.4, but introduced overlap between the signal and background subspaces. We aligned two background directions with the two weakest signal directions, creating two-dimensional shared subspace. We tested this under two background noise levels.
Signal variances: [50, 20, 15, 10].
Moderate noise background: [500, 400, 300, (pure background) + [25, 12.5] (shared background).
Large noise background: [500, 400, 300] (pure background) + [100,50] (shared back- ground).
The results for the fixed aspect ratio regime and growing-spike regime are shown in Table 2-5. For demonstration, we compare against our original theoretical predictions (derived under orthogonality).
18

---

Table 2: Fixed aspect ratio with moderate overlapping noise
Aspect Ratio
0.1
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.8
+CA++
0.137
0.207
0.269
0.287
0.317
0.311
0.356
0.388
0.416
0.431
PCA++ theory
0.104
0.179
0.229
0.268
0.301
0.330
0.356
0.379
0.400
0,410
Table 3: Fixed aspect ratio with large overlapping noise
Aspect Ratio
0.1
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.8
PCA++
0.254
0.206
0.278
0.280
0.357
0.302
0.359
0.405
0.391
0.416
PCA++ theory
0.104
0.179
0.229
0.268
0.301
0.330
0.356
0.379
0.400
0,410
Table 4: Growing-spike regime with moderate overlapping noise
Aspect Ratio
0.1
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.8
PCA++
0.121
0.197
0.216
0.261
0.299
0.334
0.369
0.374
0.400
0.426
PCA++ theory
0.100
0.171
0.218
0.256
0.287
0.315
0.339
0.361
0.381
0.391
Table 5: Growing-spike regime with large overlapping noise
Aspect Ratio
0.1
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.8
PCA++
0.138
0.182
0.214
0.262
0.316
0.323
0.363
0.375
0.380
0.397
PCA++ theory
0.100
0.171
0.218
0.256
0.287
0.315
0.339
0.361
0.381
0.391
The results in Table 2-5 show that PCA++ is remarkably robust, even when the orthogonality assumption is violated. In the moderate overlapping noise case, the empirical error of PCA++ continues track the theoretical predictions remarkably well. This demonstrates that when the background variance in the shared subspace is not excessively large, the impact of the overlap is minimal.
Conclusion:
In the large overlapping noise case, we observe a slight increase in estimation error, as expected. This because the background noise in the shared directions becomes strong enough reduce the effective signal-to-noise ratio, making recovery more challenging. Nevertheless, even in this challenging scenario, PCA++ remains stable and successfully recovers the signal subspace with controlled error, confirming that perfect orthogonality is not a practical prerequisite for our method's 's success.
E.2 On the Gaussian latent factor assumption (Assumption 2.2):
This assumption was made for analytical convenience, it allows for the clean derivation of exact constants and closed-form error. However, we expect the core results to hold more generally for Lub-Gaussian distributions.
To provide strong empirical evidence for this claim, we have run new simulations where the Gaussian latent factors (w; h:) and noise (e:) from Assumptions 2.2 and 2.3 were replaced with samples from standardized Beta(2, 2) distribution (a symmetric, bounded, non-Gaussian distribution). We repeated the experiments from Section H.4 (Figure 3) under this new setting.
The results in Table 6-7 show that the empirical performance of PCA++ under Beta-distributed noise continues to align almost perfectly with our theoretical predictions, which were derived under the Gaussian assumption.
Table 6: Fixed aspect ratio with Beta distribution
Aspect Ratio
0.1
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.8
PCA++
0.095
0.195
0.244
0.266
0.295
0.361
0.385
0.381
0.402
0.412
+CA++ theory
0.104
0.179
0.229
0.268
0.301
0.330
0.356
0.379
0.400
0.410
19

---

Table 7: Growing-spike regime with Beta distribution
Aspect Ratio
0.1
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.8
+CA++
0.112
0.171
0.223
0.260
0.296
0.331
0.348
0.360
0.389
0.409
+CA++ theory
0.100
0.171
0.218
0.256
0.287
0.315
0.339
0.361
0.381
0.391
This remarkable consistency provides strong evidence that the Gaussian assumption is a technical choice for analytical clarity rather than a strict requirement for the validity of our results, which appear to exhibit universality.
E.3
On Assumption 4.3 (Distinct growing spikes)
This assumption was made primarily for analytical convenience. Even the standard covariance S, has degenerate subspaces (i.e., multiple identical eigenvalues mixing signal and background components), the contrastive covariance S resolves this ambiguity. Since S has asymptotically zero energy on pure background directions, the generalized eigenvalue problem can still correctly identify the signal subspace and separate from the background.
To empirically validate this claim, we have run a new set of simulations for the setting in Section H.4 (Figure 3) but with:
Signal variances: [50, 50, 20, 15, 10).
Background variances: [500, 500, 300, 50).
The results in Table 8-9 below show that even with these degeneracies, the empirical subspace error for PCA++ continues track our theoretical predictions, which were derived under the distinct spike assumption. This provides strong evidence that the assumption is technical convenience rather than a practical necessity.
Table Fixed aspect ratio regime
Aspect Ratio
0.1
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.8
PCA++
0.120
0.189
0.230
0.262
0.288
0.323
0.361
0.366
0.423
0.418
PCA++ theory
0.104
0.179
0.229
0.268
0.301
0.330
0.356
0.379
0.400
0.410
Table 9: Growing-spike regime
Aspect Ratio
0.1
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.8
PCA++
0.136
0.195
0.228
0.270
0.307
0.321
0.353
0.384
0,403
0.384
PCA++ theory
0.100
0.171
0.218
0.256
0.287
0.315
0.339
0.361
0.381
0.391
F
Proofs
F1 Proof of Theorem 3.1
Proof Stacking paired samples into data matrices gives
x WTA HTB BT + %
x= =
H' +7,
(F.1)
where:
X = (AI....,, Xn)
Tn) E nxd
X = (xi,....
xx) nxd
W = (W)..... W..) ) E R txn
H1 = (h.....,
hw)) E m
H' (K....., "K)....,,))) E mx
Z = (EI...., E aad
7' = (E) , E nxd
In the contrastive factor model, the covariance of each observation x; decomposes additively into signal, background, and noise:
1 AA BB la-
20

---

This decomposition ensures that the spectrum of [[xx;;]] directly reflects the signal, background, and noise contributions.
Recall that
S
(X X + +X+
2n
Substituting the matrix form of the generative model (F.1), we obtain the decomposition
S = AA +
2 E, E
where the error matrix E, collects cross-terms:
E (AWWWT- - n4JA AWH B" + AWW
n
BHWTA A BHH BT + BHZ'
(F.2)
Z A HT + zZ
By Assumptions 2.1-2.3, each latent factor matrix has independent, zero-mean Gaussian columns, and the noise matrices are independent with zero mean. Consequently,
E[E, 0,
and hence
16 E AAT,
as claimed.
F.2
Lemma F.
Lemma F.1 formalizes the behavior of the contrastive sample covariance matrix S when projected onto the sample signal and background eigenvectors derived from S.
Lemma F.1 (Contrastive energy of sample directions) Under Assumptions 2.1-2.3 and 4.1, as n,d +o0 with d/n + (0, +0o), for each S k
lim 1 - 2
Aaj - cd +
(1+ vol
lim ndi+tos
a.s.
and-++o Aaj
Aij + cda.jj
Anj
and for each 1 $ j.
lim nd0
PBs a.s.
Ao.j
In other words, in the fixed-c limit the contrastive covariance S retains strictly positive "energy" along each signal direction, but asymptotically vanishes along every background direction.
Proof of Lemma Proof We begin by noting a useful identity. For any vector v e Rd
v S
2n v (x x + x x+Tx) v.
2n T xxTx* + (X"x+)))
(F:3
x xtv.
where the last equality follows from the symmetry property of quadratic forms. Specifically, for any matrix M& e Raxd and vector a E Rd:
a = a TTMMa a.
With this identity in hand, our task reduces to proving the following two limits as n, d +o0 with d/n + C (0, +00):
2 lim 1 111 da 1- -222
(1+ (1+ 1+0)).
lim
T
x
1sisk a.s.,
n
ndi++o
Aa, 1 cd das
1 B.. xxxxtee
lim
PB. O. i5is s a.s.
AB.j
21

---

Step 1: Proof of di
1 17 1 X BB. = =0
And B.jn"
Recall the decomposition
X X nAA =nE,
=AWWT AIJA +AWH" B' +AWZ
(F.4)
BHW -III B' BHIZ'
+ Z WTW A" + H" B' +Z
We analyze each term M in En to show
B. MDB.
R-+000
a.s..
Case = nAA
Since A A'RR is the projection of Pa.j onto the column space of A and PB. itself aligns with the column space o B, Lemma G.4 implies:
V PB.
A ATa.
R-++000
ÔE IR a.s..
VAAA Ae BB
Hence,
AA Vnnj
a.s..
Case 2: M = A(WWT -
Since WWT toikk is of fixed dimension, the strong law of large numbers implies
WWT
n-++00 4 a.s.
n
Hence,
| WWT - 11, |A AAT...lll VB
ST A(WW" nl)A S
a.s..
n
An analogous argument applies to the terms
AH, B VB.j- n
BH , B VB. and BH t H B PB..-
n
n
This shows that all such terms almost surely converge to zero. BHZ'
Case 3:
Since Z' and Pa.j are independent, we can condition on PR. and write
21
Z' Z'
Zn
where Zı...., are i.i.d. Gaussian random variables with mean zero and variance ||
1. These random variables are independent of vn.j. Consequently, H is also independent of Dn, and given Z' P'n. the entries of HZ' HZ'Dn, are i.i.d. random variables with mean zero and variance [|Z'0 11,l. By the strong law of large numbers.
HH FR.
(| DR mlllZ'n,.ll?) + of (mll mn) + mn
n
#--00000
a.s..
Hence,
DT. BHH S fllllll
R=+000
a.s..
where the limit of [|B Dalll2 is bounded almost surely by Lemma G.4, similar argument applies to show
TT B DB,AAZZ
#1000
0 a.s.
72
22

---

Case 4: =
Using the same reasoning as in Case 3, the entries of R' are i.i.d. Gaussian random vari- able with mean zero and variance 1, and are independent of both P..j and Z. Consequently, conditioning on Zra.,. implies that 1 Z Z'n follows a Gaussian distribution with mean zero and variance # By Lemma G.3,
lim LlZDa $ (1 Vo)² < +00 a.s..
Thus
DT 7. V'n
0 a.s..
Case 5: M = Z WT'""A" AT From Cases 1, 2 and 4, we already know
[|A
D,,ll22 110000
0 a.s..
WWT
11-000 1 a.s.,
n
S (1 Vo)2 a.s..
We now these facts to bound
B
Z
WT
A
DB.
1
[lPa,llall' A A.a,llz
| lllllllllww IlllA fa,llz
0 a.s..
Case 6: M =
By the same reasoning in Case 3, the entries of H" BT are ii.d. Gaussian variable with mean zero and variance ||B &g,,ll- According to Lemma G.4, ||B & remains bounded almost surely. Consequently,
- 2J 7 H' DB. S -128 B B..all 2-7000 0 a.s.,
π1
where the convergence follows from the strong law of large numbers and lim J 1 & (1+ ve)² is shown in Case 4.
Putting these cases together, all contributions of terms in equation (F.4) converge zero. Dividing by Âg, does not affect the limit, since ÂB, converges to deterministic value by Lemma G.4. It follows that
A
B.j n
X x DB.
O a.s.,
completing the proof of Step
Ans 1-025 (1+ vo))
Step Proof of limad-+oo 2a,
DT X 2 lim
A.jn"
day 1+027)) des
By the same reasoning employed in Step 1, each term M appearing in equation (F.4) satisfies
MrA.
n=++00
a.s.,
n
with the exceptions AAT and Z WT AT.
Case 1: = AAT
We have
V PAN
- Cd
TT AA VA, = I|A 1
n=4000
An.j
a.s.
+ cd,
DARYY PAs
where the limit follows from Lemma G.4.
23

---

Case 2: M= = W'WTA A
Note that
X = AWW AT + +AWH B' + AWW
+ BHWTA" BHHBB B BHZ
+ WTAT B +ZZ
Following the argument in Step for each of these summands, we obtain
- X' An. + c. -2-0X, 1 17,
c.
a.s.. (F.)
Since LoT X XPas= = 2n.j. follows that n Ag
lim 2 AWZPA,
1
- cd AN
lim AA..- - 1 (7,,z'Z0.n AA.j + ct and-++o
nnf-++00
1 - cd
lim A - (1+ Vo22 A cd Rd-0+00
a.s.,
where the last line follows from Lemma G.3.
Combining the contributions from AAT and Z WT A with the negligible effects of all other terms yields the desired lower bound:
1
1 1 -
lim Rd000000
X
lim nud-
P AA Pa.
lim
- AWZiAN
(F.6)
AA.
A.J
11
dAJ
and-+0o dAJ
da 1- -c
(1+ vo)
lim nnd-
+
da a.s.
Aa.j + cd
This completes the proof of Step and hence the theorem.
F.3 Lemma F.2
We extend the fixed-aspect-ratee analysis (Lemma F.1 and Theorem 4.2) to the growing-spike regime.
Lemma F22 (Contrastive energy in growing-spike regime) Under Assumptions 2.1-2.3 and 4.3, as n, d > +00, the contrustive energy along each sample signal direction satisfies
lim n00 AA.j
S 2 2(1 + cc.))
+ 2(1 + CA.))
for each j k, and
lim
P 0 a.s.
Ag.j
for each background index s
This lemma shows that-even when both dimension and spike strengths grow-the contrastive co- variance S still concentrates nonzero contrastive "energy" every true signal direction while vanishing on all background directions, thus preserving the clean separation needed for accurate subspace recovery under our hard-uniformity constraint.
Proof of Lemma F.2 Proof The proof follows similar lines as that Theorem F.I, with modifica- tions to account for differences in the current setting. Specifically, establish the following key results:
1. As n. 400,
- TAT vn,ll? - 0 a.s. for all 1 j m.
Ao.
2. As dl +00, + LAA TIAATAA PA, [T+a a.s., for all S $ m.
24

---

3. dim ndw.
1 ZZ) CB.j. for 1 $ s m. An analogous result holds for 2A.--
4. limaai++oo nda,
CAs 8T, AWZRAJ (1+ea))"
Using these results along with analogous arguments from the proof of Theorem F.I, we derive the following lower bound:
1 P
lim R,00++00
PA. X
lim nnt- Aaj
AA PA,
lim noo 2aj" An
AA.j
n
+
201 Ca.))
2(1 CA.))
We now detail the proofs of the above assertions separately:
Case 1: (Analysis of Ang |IA |AATralll (anil?)
We have
Aul day
Pz.j
| 1
11=+00
0 a.s..
A
ALE
Pa.i
Aa.
A.A
where the convergence follows from Lemma G.5 and the observation that
dss lim Ano
1 Asi
1
CB.
S 00.
+ Cj
AB.j
I cn.j CA.
Case 2: (Analysis of x SLAATTAAA PF) Similarly, using Lemma G.5, we have
A. PA.j
Aew
ILA PA.lP==
11+++00
(1 Ca.)) a.s..
Au.j
PA.j
Aau
A.
Case 3: (Analysis - Amax (Z²
ndg,
Since Z and ZZT share identical nonzero eigenvalues, Lemma G.3 implies
| ((
n
lim Amax ndd-4000
lim
V = 1.
It then follows that
d
1
lim ndd++40 nāB.
Amax (Z=
lim
Amxx
j
CB.
add++00
nds,
for sssm.. m.
Case 4: (Analysis of ala, OT, AWZAA)))
By arguments analogous those in Case 2 of Step 2 in the proof of Lemma F.I, we deduce that
lim nd0 ndnj
DI AWZiA,
= lim 1
TT ZPA. ZZPAJ
AA
AATiAN DA
Rd-++00
nda.j
2as
CA.
1
2 1
(1 + ca.)
(1 caj)?
CAj
(1+ CA,))?
for s j m, which establishes the claimed inequality.
The proof is thus complete.
25

---

F.4 Proof of Theorem 3.2
Proof We analyze the distance between the estimated signal subspace UA and its population counterpart UA using the Davis-Kahan sin @ theorem (Lemma G.2), Recall equation (F.2)
s = AAT (E, +
where AA represents the signal component and E, represents the error matrix. Furthermore, the signal component is expressed as:
AAT Σ MAPAS
To apply Lemma G.2, need to bound ||Enll2 such that:
|, =
Bounding the error matrix En. Recall that the error matrix En (F.2):
En 1 (A(WWT nla) +WWH" B" AAW
n
BHWTAT BHH B BHZ
Z A H B B +ZZZ)
We bound each term in E, using Lemma G.I. For the first term, since the entries in W are i.i.d. standard Gaussian random variables, we have:
((-w--111) -
1 WW te| $ dA.1
k log(n + d)
provided that n 2 & (oog d), with probability least 1 1-0( d)- 10). Applying similar arguments for the remaining terms, we obtain:
k log(n d)
m log(n d)
max(k, m) log(n d)
| dA.1
AB.
Va
n
n
dlog(n
dloggn
dlog(n dlog²
+
V2B.
n
n
n
=(M M V
k
+ ABA V m
max(k, m)
-
dlog 3/2 (n + d)
+ (Vdni
vdn. + 1) v
log (1 d).
n
Sufficient condition for n.
To satisfy the bound ||E, 1 Madh it is sufficient to assume:
(Ki muh + max(k, madadd. d (da1 + dBn + log(n
(n +d) + k + d
for some large enough constant C> 0. The condition in the theorem is established by observing that:
d dlog (n d)
ddd. log + d)
dAt
Ax
This allows us eliminate the second-to-last term the above expression, and by similar reasoning, the last term can also be omitted.
26

---

Final bound on subspace distance. subspace Us and the true subspace UA satisfies:
Using Lemma G.2, the distance between the estimated
211
2
dist(Ll,, U S 1(AAT) )2 M(A R0.1
AAA
Substituting the bound on ||E,]l2, obtain:
dist(Ul, Un) 5
21.1 V
AB.
m
VA.IdR.1
max(k, m)
AAA
dlo 3/2 dloog + d)
+ (vda. + vdRi +
log 12 + d)
where the last term in the parenthesis, under the assumption:
(aod) is dominated by the second-to-last term
Auid
ddA. log' d)
d
log' (n d).
AA.1
Therefore, the last term in the parenthesis can be omitted, and the final bound becomes:
dist(Tl, Uw
[(
k dAI1
V
λBA
m
max(k, m)
dak
(Vani+ vea +1)
log 1/2 + d).
Thus, the theorem proved.
F,5 Proof of Theorem 3.4
Proof We aim to show that
dA11 Aa.ic
lim (TTe)) 2
Let 2j be the largest eigenvalue of S Then
(1))) e St
2
Hence, it suffices to establish the following two statements as n, d + to0 with d/n + C.E (0, +00)
1. lim ef S = AA.I-
2. lim 21 2 Vance
Once these are established, the desired bound follows immediately.
Step 1: Proof of lim e, S AA.1 Using the identity (F.3), aim to demonstrate instead:
el - = AA.1
lim ad0
Recall the decomposition (F.4):
X x - nAA =nF
A(WW nLAA + +AWH'S B AWZ
+ BHWTAT BHH BT BHZ
+ Z W A + HTT BJ +Z
Following similar approach as in the Step proof of Theorem F.I, we analyze each term in M in E,. For any matrix M E, we demonstrate:
1
-e
0 a.s.,
with the exception of ZT WTAT, which is analyzed separately.
27

---

Case 1: M = nAA
For this case, compute:
[[AAT]=1A'* = Vaniefesl( 14.1
Case 2: = - W T
Here, we write
Vda. N
WZei
w W
R-+000
a.s.,
where Ej represents the first entry of the noise vector Ej. The convergence to zero follows because wy and εy independent standard Gaussian random variables and the strong law of large numbers.
This completes the proof of Step 1.
Step 2:
Proof dt d1 2
Vaace 2
Let
HTT
==
v2
62
V2 H/'
is sufficient to show that
1. lima,d-++oo v v Anic
2. limpyi++os ||v||2 =
Using the identity (F.3) and the decomposition (F.4), we proceed by proving that, for each term M in nAA" and E, the following holds:
v Mv R-+
a.s.,
with the exception of ZT H' B which satisfies
I LI
AB.IC
lim nd-+400
(F.7
2
Case 1: M AAT
From the definition of v, we have
A MaieTT
A
e.TTHT HI
2 IIZT H
AAA
Σ K
2
I|ZT H
The term T|Z HT satisfies
d
Σ
1
Σ W شركر
H
7
nd
Since H and εµ are independent standard Gaussian random variables, the sum + Σラ= K, follows Gaussian distribution with mean and variance Σ n k2. By the strong law of large numbers, this variance converges to almost surely +oo. Consequently, have:
1| H 112 n-+000
a.s.
nd
28

---

Hence,
A ATA V T
11++00
a.s..
2 I1 H
Case 2: M = A(WWT=nR)) R\)A
The arguments for this case are analogous to Case 2 in Step of the proof of Theorem F.1. The same reasoning applies directly to the terms A(WWT n/)AA, " 1 AWH B Da.- Lo, BH H BT.. and 1 BH HT B PBJ- B
Case 3: = Z.
Since v independent Z', the same arguments used in Case and Case of Step proof of Theorem F.1 can be applied here. This reasoning extends to the terms Z'Z', BHZ' and AWZ'.
Case 4: M WTA From the Case 1, we know that
A
a.s..
The rest the proof follows exactly the reasoning Case of Step in the proof of Theorem F.I.
Case 5: = 2 2-2H' H B
We write:
H' - B 1 (ZTHT)
7 HT B"v BTv
V2n
V2n I/TH'TIb
We will show that the first term converges to zero, and the second term satisfies the inequality in (F.7).
Firstly, consider the term Vy
v
VABIeTy
Ap.11
ZT H"
1 HT
AgA1
1
h 11552
[ HrT 1|2 j=1
11++000
Ag.1
a.s..
(F.8)
2
where the convergence follows from the same reasoning as Case 1. Hence,
1
R
B
BT
a.s.
2n
v2
n
Now, for the second term:
HTT
Aacc
lim +000
HT B
lim
112 H"
112
VIn H
add+400
2n
2
It remains demonstrate that limm.d-+4a ||v|12
We have
Z IT
L
Z HTT
||vlß
V2" e2
v2 1Z
H'
V2
V2
H
=1 e, H"T
a.s.,
I HT
where the convergence follows from the same arguments in equation (F.8).
This completes the proof of Step
29

---

F.6 Proof of Theorem 4.2
Proof We begin by considering the sample covariance matrix S, write its eigendecomposition as
S V = V.Ax.
where we order V (DA.I...., PAA. BB......, .......) and A, = (QA.1..... AAx.28.1....... .....). Set A/ be the diagonal matrix whose entries are 2-112 when à > 0 and otherwise, and define whitened matrix. The transformed matrix M, can thus be expressed as:
M A3 V V
The matrix is nonzero only within its upper-left (k m) (k m) submatrix:
M (inkkm)xx(-m))
0T ST
Van 1 B.
Aas A.
Vaaidax
Viuidan
AR PAP
AA
Ax
Al n D n.
Vaseda
Ass
Vigudau
Viyden
211 STS.
SAAAA
B. PB.
B.. n B..m
Vanida
dan
Van
Bm PAA.
PARA
ST Aa
B.n 7p
VaRadai
Agu
Bun
Ban
We now show that, in the limit, this submatrix M(stm)x(km)) vanishes except for the first diagonal term entries, which remain positive by Lemma F.I.
Entries involving background directions. Using Lemma F.1, follows directly that cross terms involving eigenvectors from the background subspace vanish asymptotically, for any i, s
B.
S
D 13,5
at o,
fnnddnj
Agd
and similarly, for 1 sis k. 1 15 m.
S S
SI Ap..
0. 1 1B8
Off-diagonal signal-signal entries. equation (F.6) shows that
Fix 1 siijssk A similar argument as in the derivation of
1
lim Vandaa
ST SASj
lim
AAAATAA VA.
Varday
=;
lim nd0
PT (AWZ + Z WTTTTA ATAJ-
Vnnddn
2n
1. Tj: We have
lim
AATATAA PA.
lim
Pa =0.
Anidas
Viridas
lml
where the last equality follows directly from Lemma G.4.
30

---

2. T2: Similar reasoning to equation (F.5) yields:
1
lim n,--+400
MAWAZ + WTWTA A Das
lim n,d-+00
Pi -x_ XPAi
Vandans
n
Jiidass
n2
lim n,d-+000
AA PA.
lim nd-++oo
D - -z
ZPA.
Vindas
Vanidss
lim
A.
-Z-ZZAZAA
A.d
To demonstrate the almost convergence of the remaining expression zero, apply leave-one-out decoupling approach:
delete of and let 2, be the corresponding eigen- vectors. These eigenvectors satisfy independence from the omitted data point and Davis-Kahan perturbation bounds implies II2; 21 = (
For each sample
Express the term as:
TJJ ZPAJ
Σ
(2) + Op Vn).
where z the 1-th row of Z, is independent of p and x(-))
A.j
Following the proof Lemma G.6, Ex (2) ) ((x] ( ( //n is an average of i.i.d
zero-mean variables with finite variance, hence it converges to zero almost surely by the strong law of large numbers.
Consequently, we establish:
lim
PT. (AWZ Z A'
lim idd++000
- Ad
ZDAA = O,
Myd-a
AAJdA.j
proving that the off-diagonal entries vanish asymptotically.
Positive definite submatrix Mдx We have demonstrated that the asymptotic limit of the matrix Madt-mix(--m) except for its first k diagonal term, which remain strictly positive by Lemma F.1. Consequently, asymptotically, exactly sample eigenvectors correspond to positive eigenvalues, yielding the estimated signal set:
v (PA.I....,AAA))
Evaluate the estimated subspace Finally, we quantify the accuracy of the estimated subspace U relative to the true signal subspace UA. By Lemma G.4, it follows that:
T VAR
lim V
lim
VAVA
1,4 PAX
1 c
cd,2
diag
+ cdi
1 + cd
Hence, the asymptotic squared subspace distance given by:
lim dist(Uu, U22 R.d-+000
lim max sin (arccoso ad- 1sjsk
= lim max 1 o3 md 1sisk
1 1 cd
1 + cd,11
31

---

F.7 Proof of Theorem 4.4
Proof The proof follows directly from the arguments detailed in the proof of Theorem 4.2, in conjunction with Lemma G.5 and Lemma F.2. Accordingly, the algorithm asymptotically yields:
V CA........ P.R)
Consequently, we have:
lim V diag (1 cA.1) 1122
and the squared subspace distance asymptotically satisfies:
lim dist(Ll, UA)²= lim max sin sin (arccos or)
n,d-+000
ndd-+40 Isjsk
lim max of
nd-- 1sjsk
1
CAR
1 + CAR
G Preliminary Lemmas
This section collects technical tools that will be invoked repeatedly in the proofs of our main results.
Lemma G.1 (Lemma 3.5 in [14)) Assume that the entries in matrices R and Z Rabon are i..d. standard Gaussian random variables, ifn klog' (n+d), with probability at least 1 1-0(nnd))1)) 19) one has
dlog(n + d)
WZ || i
n
Ww - -0/1 llog(n + d)
1 -zzz T0/
dlo((n + d)
loog2 (n d)
The next lemma is convenient corollary of the classical Davis-Kail Kahan sin O theorem.
Lemma G.2 (Corollary 2.8 in [14]) Let M and M M + be two X n real symmetric matrices. We express the eigendecomposition of M and M as follows
M' E ( (1)( N U.T (
UT
Here, (2i1f-1 (resp. (2)511) denote the eigenvalues of M (resp. M'), and vj (resp. vi) stands for the eigenvector associated with the eigenvalue dj (resp. 27). Additionally, we take
A == diag ([d1,... .aab..)),,,,)) E
. Rr
U= [vi..... .vr. Ra
The matrices U", U., A", and A are defined analogously. Suppose that [2]] 2 [M 2... 2 |27 > W ... 1121 and [iil 2 |l 2 ... 2 lall (ie., the eigenvalues are sorted by their magnitudes). If ||] < (I 1/ v2)(1)1 ( then
21
211
(is(u,u)) v2I sin s
|1:11 - -,,,,
| 141-12;,1
where U (resp. U() ) is the subspace spanned by U (resp. U*)
32

---

By combining Theorem 2 and Remark 1 of [4], we obtain the following lemma.
Lemma G.3 Assume that the feature vector x has i.i.d entries with zero mean, unit variance, and bounded 4-th moment. As n, d + 00, 0 c < 00,
lim ii (''XXXXX/X) X/n) (1- vo)² a.s.
nd-++00
lim Amax (X" X/n) =1 vo)2 a.s.
an-++00
where 2 denotes the smallest positive eigenvalue.
Lemma G.4 (15) and [40]) Under model (2.1), let > top k m eigenvalues. Then, as d/n + c with n, d + +00,
Aktm Vc (0, +00) denote the
1 + dj
2 + 1
a.S.
dj
cd,²
a b
a Vvv.B a.s.,
1+ cd
for any deterministic unit vectors a, a,be Rd and i m.
Lemma G.5 (Theorem 5.1 in [45]) Under model (2.1), let di 12 ... ARom denote the top k + m eigenvalues. Then, as n, d + too with & - E (0, + +00) and deam - co,
2 - (1 cg) a.s.,
- (1+ 1( i) a.s.,
for 1 S i,j k m.
Proof The original statement Theorem 5.1 in [45] covers explicitly the results of both the sample cigenvalues and the alignment between the j-th sample and population cigenvectors when = j. Here, we extend the arguments handle the off-diagonal scenario j. Let
(f....... Paj)". j=
denote the cigenvectors of the sample covariance. Without loss of generality, we adopt the standard Euclidean basis (e for the population eigenvectors, setting v, ey. Under this choice, it suffices to demonstrate that, as d +00,
Prj a.s.,
for all # j. We first consider the scenario with = 1. By Theorem 5.1 (equation (8.21)) of [45], follows directly that:
6+m
4=000
a.s..
Hence, we conclude 9.,1 a.s. for all # 1, proving the claim for j= Next, for the case, j = 2, equations (8.23) and (8.24) Theorem 5.1 from [45] imply, as n, d + to0
++ Σ
a.s.,
a.s..
1 ++
Consequently, we have 942 0 a.s. for all # 2. Furthermore, from equation (8.17) in [45], have:
a.s..
1+
which directly implies D1,2 - 0 a.s. as n1, +00. Thus, the result also holds for j=
The above arguments can be iteratively applied to subsequent indices j = 3, k reasoning. Hence, we obtain the general result for arbitrary i and j. completing the proof.
k m by analogous
33

---

Lemma G.6 (Asymptotic orthogonality of Gaussian projections) Let a, b E R be unit vectors satisfying a b and let Buxd be random matrix with entries Xij iid N(O, 1). Then:
x Xb n...
as n, d - 400.
Proof We first rewrite the expression as:
" -a x Xb -(xa) (Xb) Σ
(Xa))(Xb))
n Fl
Let 2 = (X)), so that:
1
-a X Xb
Σ 2/-
F=
Each row Xj. E Rd of X is an i.d. sample from N(O, 1.). Then:
(Xa) x7,a, (Xb); XT.b.
and thus:
t = (X (x}b).
Since x,. NN(,, La), and a, E Bd are fixed vectors, the random variables X and XJ.b are jointly Gaussian with:
= b[ and Cov(XJaa, a
Hence, a_ and b are uncorrelated cero-mean Gaussian random variables and thus independent. Therefore, since both are independent and centered,
E[a] (I((),))) b)] B[X],a] E[ b] O.
Also, Zn are i.idd with finite variance. Then by the strong law of large numbers,
2
E(z,] =.
Thus,
aa X Xb 25 O.
H
Additional Details for Numerical Simulations (Section 5)
Below we describe in additional details of the synthetic data configurations and evaluation procedures underlying numerical experiments. In all experiments measure the subspace estimation error as the sine of the principal angle between the estimated and true signal spaces, averaged over 50 Monte Carlo trials.
H.1 One-signal-one mackground-mottel--vryyigg background strength
We generated n 2000 paired samples in d 800 dimensions from the linear contrastive factor model (Eq. 1)..)). The true signal subspace is one-dimensional
1): A /2A.1. OJJ places a spike of magnitude 1x,1 10 in coordinate 1. B = [0, V o..., _OT, background variation is confined to the orthogonal coordinate 2 (m 1). with strength A varied so that A A ranges over (0.3125.... 0.666). For each method (PCA, PCA+, PCA++), we compute the sine of the principal angle between the estimated vector and the true signal direction (the first standard basis vector e1). fixed truncation rank this one-dimensional signal case). Results in Figure (left) show that, background strength grows, PCA collapses to noise, PCA+ eventually drifts onto the background axis, while PCA+ remains tightly aligned with the true signal.
used to stabilize PCA++ (though truncation does not affect
34

---

H.2 One-signal-one background-moote-varyyyye aspect ratios
We fixed 500 and varied d so that d/n spans (0.1...... 1.8). Both signal and background spikes remain at AA.1 10 in coordinate and Ag.1 = 500 in coordinate 2. We apply ordinary PCA, PCA+, and PCA++ with truncation (right) shows that while PCA and PCA+ errors grow unbounded for large d/n, PCA++ enor remains uniformly low.
The sine of the principal angle e₁ is plotted versus d/n. Figure
H.3 Stability of PCA++ truncation effects
Figure 2 (left) illustrates the instability of untruncated PCA++ for = 1000 as d/n increases from 0.1 to 1.8. With the same one-dimensional signal (ΛA,1 10) and background (Aæ,1 500), direct solution of the generalized eigenproblem yields erratic directions at high aspect ratios. Replacing the sample covariance S, by its rank-s approximation with
restores stable recovery.
Figure 2 (right) examines PCA++'s sensitivity to s. Again with n 1000, A = 10, d = 500, and d/n e [0.1, 1.8], vary {2, 0.1d, 0.2d, 0.4d). Moderate truncation (e.g. 0.1d) best balances bias and variance, discarding noisy directions while retaining the subspace needed to enforce uniformity.
H.4 Empirical high-dimensional asymptotics
We n 500 and let d/n vary for the rixed-aspect-ratio regime (Theorem 4.2), using a five- dimensional signal & 5. The signal loading matrix A R was defined such that the signal occupied the first five feature dimensions:
AAAI
V AA,
0
0
O
JdA,3
0
A
0
0
AAAA
0
0
: 0
0
0
with the signal component variance per dimension [dA.1 A4,2, A AA A4,5] 25, 25, 20, 10].
The true signal subspace UA is spanned by Ie es). The background loading matrix B R was defined orthogonally to A, affecting the last five feature dimensions
0
0
0
0
0
O
Ans
O
0
AAp
0
0
A
0
0
0
AA.11
0
0
with the signal component variance per dimension [dR,1, Ap,, dn AR. [500, 400, 300, 200, 100). Truncation rank is 10. For the growing-spike regime (Theo- rem 4.4), we scale both d and all spikes by a factor of 10. As shown Figure 3, the empirical subspace error of PCA++ tracks the theoretical prediction almost exactly, as the spikes and dimension diverge, whereas PCA+ consistently fails to align with the true signal.
H.S Empirical validation with fixed aspect ratio
To assess performance under a fixed aspect ratio of d/n = 0.4, we conducted simulations comparing standard PCA, PCA+, PCA++, and the theoretical prediction. Data were generated from the linear contrastive factor model (Eq. (2.1)) with signal and background each occupying five orthogonal directions p-dimensional space. The signal subspace consisted of the first five canonical axes, with variances [1..1 dA,2 14.3, A das] [20, 20, 15, 10, 10], while the background spanned the last five
35

---

axes with variances [Ag.1. Ag2, AB3,, Ag., A85] [500, 500, 200, 100, 100] in the "large noise" regime and [100, 100, 20, 25] in the "mild noise" regime. We varied the sample size n {100, ,00,5000000 5000), keeping d = 0.4 n, and performed 50 independent trials for each setting. Subspace error was measured by the largest principal angle between the estimated and true signal subspaces.
Under large background noise (Table 10), both PCA and PCA+ fail completely, yielding error near one, whereas PCA++ achieves low error (0.304 at n 100, dropping to 0.212 = 5000) with small variability (standard deviation from 0.037 to 0.005). Even when background noise is mild (Table 11), PCA+ only improves its average error at large n (0.222 at = 5000) but remains highly unstable (sd = 0.062). In contrast, PCA++ matches or betters PCA+'s 's mean error (0.212) while reducing its standard deviation to 0.004. These results demonstrate that PCA++ not only lowers estimation error but also yields far more consistent recovery of the signal subspace than either PCA or the alignment-only method.
Table 10: Fixed Aspect Ratio (d/n 0.4) under Large Background Noise
Setting
PCA
PCA+
PCA++
PCA++ theory
100, p 40
0,999 (0)
0.999 (0)
0.304 (0.037)
0.205
π 500, = 200
0.999 (0)
0.999 (0)
0.225 (0.010)
0.205
π 5000, p = 2000
0.999 (0)
0.999 (0.001)
0.212 (0.005)
0.205
Table 11: Fixed Aspect Ratio (d/n 0.4) under Mild Background Noise
Setting
PCA
PCA+
PCA++
PCA++ theory
100, 40
0.999 (0)
0.996 (0.011)
0.293 (0.038)
0.205
л 500, p = 200
0.999 (0)
0.713 (0.287)
0.227 (0.010)
0.205
5000, p 2000
0.999 (0)
0.222 (0.062)
0.212 (0.004)
0.205
I Additional Details for Real Data Applications (Section 5)
L11 Corrupted MNIST data
This section details the experimental setup for the qualitative evaluation on noisy MNIST digits, the results of which are presented in Figure The aim was to visually assess signal disentanglement from structured background noise using a dataset of n 5000 paired synthetic images (x,, x,
Data generation. The signal components were images of digits '0' and '1' from the MNIST dataset [16]. Each original 28 x 28 grayscale MNIST image had its pixel values normalized to the range [0. 1]: let M, denote such processed digit. Backgrounds were sourced from the "grass" synset of the ImageNet dataset [15]. Full ImageNet images were converted to grayscale, from which 28 x 28 patches were randomly cropped. Each cropped grass patch was then also normalized its pixel values ranged from [0. 1j. Let G; be independently sampled and processed grass patches. Paired images were then constructed through pixel-wise addition:
3 M,
x 0.25 M, + G,-
This setup uses the same MNIST digit M; for both x; and its positive pair x but with different grass backgrounds, and with the MNIST signal scaled down by factor of 0.25 in the x view. Each resulting synthetic image was flattened into a p 784 dimensional vector. Finally, before applying the PCA methods, the set of all x; vectors (forming matrix X) was globally mean-centered (i.e. the mean vector of X was subtracted from each row x,). Similarly, the set of all x vectors (forming matrix X+) was globally mean-centered.
I.2 Single-cell RNA sequencing data
We investigate the impact of immune stimulation on peripheral blood mononuclear cells (PBMCs) using single-cell RNA sequencing data from [31]. This dataset includes scRNA-seg profiles from
36

---

8 human donors collected under two conditions: control and interferon- (IFN-B) stimulation. The processed data comprise 14,619 control cells and 14,446 stimulated cells, spanning curated immune cell types. These include both IFN-6-responsive populations (e.g., monocytes) and cell types with minimal transcriptional response (e-g., B cells), providing a natural testbed for studying condition- invariant structure. While cells are not explicitly paired in the raw data, shared donor identities and consistent annotations allow post hoc matching for comparative analysis. Each cell is labeled with a curated cell type, offering stable reference to assess shifts in population structure under perturbation. Our goal is to demonstrate that the proposed PCA++ method more effectively captures biologically meaningful, condition inttaiiinn structure in real-world single-cell data. To this end, we apply both PCA and PCA++, and visualize the resulting embeddings by applying UMAP [35] to the top 50 components of each projection. We compare the structures side-by side-by-side across control and stimulated conditions to assess the preservation and alignment of cell-type clusters.
To prepare the dataset for contrastive analysis, we first filtered the gene space to retain only infor- mative and biologically relevant features. We excluded non-coding genes (e.g., IncRNA, miRNA, pseudogenes) by annotating Ensembl gene IDs using the mygene API and removing entries with non-protein-coding biotypes. We then selected the top 2,000 highly variable genes (HVGs) using the Seurat v3 method implemented in Scanpy. This yielded a final feature space shared across 29,065 cells, comprising 14,619 control and 14,446 stimulated cells.
Because cells are not explicitly paired in the original dataset, we constructed cell pairs post hoc using donor identity and local structural alignment. For each donor, we performed independent normalization, tng-trrasformtiion, and PCA on cells from the control and stimulated conditions. We clustered cells within each condition using Leiden clustering and computed the centroids of each cluster in the 50-dimensional PCA space. Clusters were matched across conditions by cosine similarity of their centroids. Within each matched pair of clusters, we applied nearest-neighhor matching in PCA space to identify at most 300 well-aligned cell pairs. This strategy ensured both inter-condition comparability and intra-donor consistency, resulting in 9,268 matched control and 9.268 matched stimulated cells across all 8 donors.
To evaluate the stability of cellular identity across conditions, we apply PCA and our proposed PCA++ to the full set of matched control and stimulated cells. Both methods produce 50-dimensional latent embeddings, which we visualize using UMAP. In the PCA++ framework, the stimulated cells are treated as the target dataset and the control cells as the target-plus dataset, enabling the method to extract components that emphasize shared structure while suppressing condition sppecffcc variation. This design aligns with our goal of capturing invariant cell populations under perturbation.
The left panel of Figure 6 shows the UMAP projection of PCA embeddings. While cell types and stimulated cells of
are well-separated, the embeddings exhibit clear batch effects- the same type often appear as disconnected clusters. This suggests that PCA captures both cell identity and stimulation-dri -dritiionndrvennnnnn variance. In contrast, the right panel of Figure 6 shows UMAP on PCA++ embeddings. Here, stimulated and control cells are more co-localized, especially for stable populations like CD4 T cells and B cells, indicating that PCA++ better preserves condition-invarian structure. Although some separation remains for highly responsive populations (e.g., monocytes), PCA++ enables clearer alignment of shared cellular manifolds across conditions.
To better interpret the results, we highlight key observations from Figure 5, which contrasts UMAP visualizations derived from PCA (a. b, c) and PCA++ (d, e, f) projections. In the PCA embeddings, cell types are well separated in both control and stimulated conditions; however, control and stimulated cells of the same type frequently occupy non-overlapping regions. This indicates that standard PCA captures both cell identity and same cell type. In contrast, PCA++ (d, e, f) produces embeddings where control and stimulated cells are better aligned within each cell type. Although the overall separation between cell types becomes slightly less distinct, the coherence of shared structure across conditions improves markedly. For example, invariant populations such as CD4 T cells, B cells, and NK cells remain tightly clustered regardless of stimulation, highlighting PCA++'s ability to emphasize stable transcriptional programs. Importantly, these findings are consistent with known biology: these cell types are relatively insensitive to IFN-B stimulation and exhibit minimal transcriptional changes. In contrast, more responsive cell types such as monocytes appear more dispersed, reflecting biologically meaningful variation rather than projection pojection-induced induced noise. These results demonstrate that PCA++ more effectively
variation, leading to separation even within the
37

---

disentangles cell identity from stimulation effects, enabling clearer interpretation of shared cellular structure across perturbed and unperturbed states.
we have also included a comprehensive comparison across several methods, including standard PCA, alignment-only PCA+ (which lacks the uniformity constraint), PCA++, as well as additional baseline dimensionality reduction techniques (UMAP [35], t-SNE [49], Robust PCA [10]). All methods were evaluated on the same single-cell RNA-seq data, focusing on their ability recover cell-type groupings in both control and stimulated conditions.
The adjusted Rand index [27, ARI] (lower is better in this context) for each method is summarized in Table 12.
Table 12: Performance comparison across different cell types
Cells
B Cells
NK Cells
All
PCA
0.2084
0.2977
0,2502
0.1478
PCA+
0.0036
0.0242
0.0299
0.014
PCA++
0.0008
0.005
0.0049
0,0172
U-MAP 0.2031
0.2506
0,2078
0.1392
t-SNE
0.0233
0.0967
0.0477
0.0217
Robust PCA
0.0399
0.1907
0.1314
0.0443
As shown, both PCA+ and PCA++ dramatically reduce the ARI compared standard PCA and other popular methods, indicating much better mixing of the two conditions within cell types. Importantly, adding the uniformity constraint in PCA++ further improves the results over PCA+, especially in B cells and NK cells. These quantitative results highlight the utility of both the alignment and uniformity components. PCA++ achieves superior separation of invariant populations compared alternative approaches.
PCS): Ctrl (tadad)
(PCA++ PCs): Stim (taded)
Danchiiccall
Devemnnnaie jainn
2014- Maroodka (df)
rocriser Vonagtaa 200t
C0n6- Moncoyees salm))
FOORReee Mesoogtte (ylnng
UMAR2
Megelarrpeeeee (dit)
phoryveyes (ste))
OO6'T NK
UMAP1
Figure 6: UMAP visualizations of PCA (left) vs. PCA++ (right) on all cells. Both plots show UMAP projections of the top 50 components applied to all matched PBMCs. The PCA embedding (left) separates control and stimulated cells-even even within the same cell type-duc duc confounding condition-spenn speciifc variance. In contrast, the PCA++ embedding (right), computed using stimulated cells the target and control cells auxiliary input, better aligns the two conditions. Invariant populations such as CD4 T cells and cells remain coherent across stimulation, while more responsive types (e.g., monocytes) exhibit increased spread, reflecting true biological variability.
J Additional Discussion on cPCA
Although there is fundamental distinction between cPCA and PCA++, one reviewer suggested that cPCA could nonetheless be adapted to address the problem solved by PCA++. Specifically, the reviewer proposed constructing a synthetic foreground and background via xy = / and X; (x-x**) x/-x /2, and then applying cPCA to the pair (x,, Ab). While this construction yields a cPCA-like method that theoretically valid at the population level, our analysis and experiments demonstrate that it substantially less stable and reliable in finite-sample regimes compared to PCA++
38

---

We can see that at the population level, the signal is perfectly isolated. Under the contrastive factor model, the population covariances of the synthesized data are:
Σ = B AAT + +2 2
1 Σ, B[xb,,]]] BB"
Therefore, the difference Σ, - = AAT perfectly recovers the signal covariance. The practical challenge, however, arises from finite-sample estimation error. Define foreground matrix X, (X + x/2 and background matrix X, (X )/2. The sample covariances of the synthesized data are Σ xTxx//n and Σn X3 X,/n, respectively.
The reviewer's proposed approach relies on the matrix subtraction Ê, aL. this subtraction of two large, noisy matrices can amplify noise, potentially overwhelming the true signal and leading to unstable eigenvectors.
Regarding cPCA:
Regarding cPCA++: A cPCA-like like extension, referred to cPCA++ [41], has also been proposed in prior work. This method identifies contrastive directions by solving a generalized eigenvalue problem of the form E Ey, where , and Σ, are the empirical covariance matrices of the foreground and Σ,'E,
background data, respectively. As demonstrate in our paper (Figure 2), directly inverting Σ, is numerically unstable when dimension d is large. The reviewer might suggest stabilizing this by using truncated pseudoinverse, similar this procedure is equivalent to projecting the foreground data X, onto the principal subspace of the background data X, and then performing PCA. Since the synthesized background X contains no signal A by construction, its principal subspace is also signal-free. Projecting the foreground onto this signal-free subspace would annihilate the very signal we aim recover. Therefore, a truncated cPCA+ would fundamentally fail.
PCA++ However, this reveals deeper conceptual issue:
Empirical validation:
To investigate these different approaches empirically, we performed a new set of experiments compar- ing PCA++ against alternative methods proposed by the reviewer.
1. Baselines on synthesized data: Following the reviewer's suggestion, we evaluated methods that operate synthesized foreground X, = (X + )/2 and background X, = (X x/2 data. This includes standard cPCA and cPCA+ (based on difference of covariances). This allows to directly test the stability of the petibractttee-decooppss approach versus our "decompose-rommccross- from-croos--covvrinee method.
2. CCA:
We also introduced CCA as a canonical and highly relevant baseline. We apply by treating the paired data matrices, X and X*, the two views. In our model, the shared signal is the sole source of population- CCA. which finds directions of maximal correlation, is theoretically suited for recovering the signal subspace and serves as a strong benchmark.
correlation between these views. Therefore,
We consider the following experiment setup:
The experiment follows the n = 500, k fixed aspect ratio setting from Sec E.4. For 'cPCA', set 1. This is the most principled choice as it yields an unbiased population estimator
Experimental setup:
B Σ사 AAT BB" 12 1a/2 BB 12/22 AA
We tested two scenarios: one with moderate background noise and one with strong back- ground noise.
Signal variances: [50, 25, 20, 15, 10).
Moderate noise background: [100, 50, 40, 30, 20].
Large noise background: [500, 400,300, 200, 100).
39

---

Experimental results (subspace error):
The results are provided in the following Table and 14. As the results clearly show, the cPCA adaptation is highly unstable even with its optimal hyperparameter setting 1. This, combined with the conceptual issue in the truncated cPCA++ approach, demonstrates that these methods are not well-suited for this problem. In contrast, PCA++ method remains stable, highlighting the robustness of our GEP formulation.
Aspect Ratio
0.1
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.8
cPCA
0.293
0.819
0.430
0.977
0.284
0.626
0.813
0.581
1.000
0.964
cPCA++
1.000
1.000
1.000
1.000
1.000
1.000
1.000
0.999
0.999
1.000
CCA
0.160
0.488
0.999
1.000
1.000
1.000
1.000
0.999
1.000
1.000
PCA++
0.125
0.212
0.250
0.275
0.311
0.347
0.375
0.388
0.415
0.401
PCA++ theory
0.104
0.179
0.229
0.268
0.301
0.330
0.356
0.379
0.400
0,410
Table 13: Performance comparison across different aspect ratios with moderate noise background.
Aspect Ratio
0.1
0.3
0.5
0.7
0.9
1.1
1.3
1.5
1.7
1.8
cPCA
1.000
1.000
0.999
0,941
0.998
0.999
0.998
1.000
0.998
0.996
cPCA+
1.000
1.000
1.000
1.000
1.000
1.000
1,000
1,000
1.000
1.000
CCA
0.154
0.369
0.999
1.000
1.000
1.000
1.000
1.000
1.000
1.000
PCA++
0.156
0.185
0.230
0.283
0.321
0.339
0.379
0.353
0.410
0,447
PCA++ theory
0.104
0.179
0.229
0.268
0.301
0.330
0.356
0.379
0.400
0.410
Table 14: Performance comparison across different aspect ratios with large noise background.
K
Additional discussion the relationship to CCA
While both CCA and PCA++ methods aim to find shared structure between paired datasets, they differ crucially their objectives and constraints, leading to significant performance gaps in noisy, high-dimensional settings.
Objective functions and constraints:
Standard CCA: Seeks two distinct projection matrices, U and V. that maximize the correlation between the projected views U X and V The objective for the leading component is:
u Tx Xv)
max
(u (X X))(v ( (X+TX+)V)
The key instability lies in the denominator, which normalizes by variance from both views and X*). In our problem setting, both views contain large, independent background and noise components, making this normalization highly susceptible to noise amplification. Although one can also revise CCA for the positive-pair setting by imposing constrain U V, the objective becomes Generalized Rayleigh Quotient, which, to our knowledge, does not have the same closed-form solution as standard CCA.
PCA++ Our formulation makes two critical changes that enhance stability
Shared projection space: It enforces U V, searching for single, shared subspace that captures the signal common to both views.
Robust normalization: Instead of normalizing by the noisy variance from both views, PCA++ maximizes the shared covariance v (X X+)v subject to a hard uniformity con- straint on only one view v (X X)y = 1. This constraint acts powerful regularizer, anchoring the solution to a more stable variance structure and preventing distortion from the background and noise present in the second view X*.
40

---

Empirical Performance:
This difference in formulation has a dramatic impact on performance. As our new experiments (provided in Table 13 and 14 of Appendix J) demonstrate:
Standard CCA is highly unstable the settings we study. It fails to recover the signal, with its performance collapsing as dimensionality or noise increases.
PCA++ remains stable and robust, successfully recovering the signal subspace across all tested regimes.
This confirms that our specific choice of a shared projection space and single-view uniformity constraint is essential for reliable signal recovery in this problem setting.
41

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We support each point by either theory experiment.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper relies on certain assumptions such as linear model structure and noiseless or Gaussian data. It discusses the limitations arising from these assumptions and how results might be affected when they are violated.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specfficatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
42

---

Answer: [Yes]
Justification:
See Appendix F and G.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
43

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code and data are publicly released at time of submission. The experiments are based on standard datasets and use well-established implementations as described the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ httppp/biiii/gggieedddeeeeeaaiii
guw.e//ieeeeeeee/eeegeeee//ee Cooe/eueeeeeeeeeeee wwt.sSuuuiisiiiini...... ssionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix H
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
The differences are often significantly large.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
44

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: The experiments do not involve heavy computation, and thus detailed compu- tational resources are not necessary for reproducibility. Relevant implementation details can found Appendices B.2 and H.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpp//:///uuussscccooo http//rwwppp...cpp.pb.///ccaaaaa
Answer: [Yes]
Justification:
We obey all aspects of the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This is theory-oriented paper with no societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
45

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not involve such models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi twcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
46

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
We introduce no new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not use crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do have such studies.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
47

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ /ttpss////uuis.../// /ctccooeeeeeenee/22255////////// for what should or should be described.
48


Reviews:
[{"review_id": "uj0Fb2Zvsp", "reviewer": "Reviewer_r13i", "summary": "The paper introduces a new contrastive PCA algorithm, PCA++, that isolates shared signals in positively related datasets under strong background noise. PCA++ enforces hard uniformity constraints in the projected features. And it results in closed-form solutions with stability and non-asymptotic guarantees. The paper compares this approach with alignment only contrastive PCA and reports improved performance on synthetic (simulation, MNIST) and one real-world (sequencing data) experiments. The paper further provides justification and proof for the importance of uniformity in improving performance and makes PCA robust to high background noise.", "questions": "* In Figure 3, only classical PCA seems to be considered as a baseline for comparison. This limited real-world experiment and the considered baselines make it hard to assess the utility of the method in real-world experiments.  \n\n\n* I encourage authors to consider citing an important early work on contrastive PCA[1] and comparing the method with existing baselines. \n\n[1] Contrastive Principal Component Analysis https://arxiv.org/pdf/1709.06716", "limitations": "While the authors have mentioned extensions to other existing contrastive learning methods, the limitations of this work are not clearly outlined.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**S1**: The paper presents a well-motivated problem. It presents the classical and alignment-only contrastive PCA and their limitations with examples, setting up a perfect ground for the proposed method, PCA++. In addition, the paper is easier to follow, with a clear outline of its contribution.  \n \n**S2**: The paper presents a novel contribution to contrastive learning theory by analysing the role of uniformity constraints on projected features in high-dimensional strong noise setting. This result could have an impact on other machine learning areas.  \n \n \n**S3**: The paper precisely shows the limitation of PCA  and the utility of the proposed techniques, with the real-world experiments showing isolating noise and extracting shared space. Specifically, the synthetic experiments showed clear improvement in error trends with aspect ratio and truncation levels.  \n\n**Weakness**\n\n**W1**: While the authors have mentioned the work could be extended to other branches of contrastive learning, the authors did not discuss the limitations of the work, for instance, with regards to the assumption or settings considered (Assumptions 4.2, and 4.3), and how real-world data conforms to these assumptions.  \n \n \n**W2**: The simulation experiment shows the effect of varying aspect ratio d/n, and truncation etc.. However, there seems to be a missing discussion on how these factors affect the performance of PCA++ in real-world experiments (RNA sequencing). \n \n \n**W3**: I think the only real-world experiment presented in the paper ( using sequencing data ) is limiting in judging how important the uniformity constraint is in contrastive PCA. I think it would be important to at least show how well the alignment-only PCA+ does on the data. This limited real-world experiment and the considered baselines make it hard to assess the utility of the method in real-world experiments.", "was_revised": true, "final_justification": "My main concern, also shared by other reviewers, is the comparison of the paper with prior contrastive PCA work. The authors have responded with clarification on the differences from existing contrastive PCA works. And I am satisfied with the responses, and I will maintain my original score and confidence.", "rebuttal": "> ... the authors did not discuss the limitations of the work, for instance, with regards to the assumption or settings considered (Assumption 4.2, and 4.3), and how real-world data conforms to these assumptions.\n\n**A**: We thank the reviewer for their thoughtful questions regarding the assumptions underlying our main theoretical results. We agree that discussing the scope and necessity of these assumptions is important. We address each point below and will add these clarifications to the camera-ready manuscript.\n(For a discussion of the paper's broader limitations, such as the linear model, please see the detailed response to Reviewer `[9ppt]`.)\n\n- **On Assumption 4.1** (BBP detectability threshold):\nThis assumption, $\\lambda > \\sqrt{c}$, is a classical and fundamental requirement in high-dimensional PCA for any spiked covariance model. As we note on lines 216-217, any signal spike weaker than this threshold becomes statistically indistinguishable from the noise bulk of the Marcenko-Pastur distribution. This is a fundamental limit of signal detection via PCA in this regime, not a limitation specific to our `PCA++` method.\n\n- **On Assumption 4.3** (distinct eigenvalues):\nThis assumption was made primarily for analytical convenience. With distinct eigenvalues, we can cleanly map each sample eigenvector to a population counterpart using existing results (Lemmas D.4, D.5). **Without this assumption**, if a set of eigenvalues were identical, we would instead estimate a *subspace* for those directions. Within that estimated subspace, it would be difficult to distinguish which basis vectors correspond to signal and which to background just by looking at $S_n$ alone, making it harder to explain how the uniformity constraint works on a per-direction basis.\n\n**The key insight** is that even if the standard covariance $S\\_n$ has degenerate subspaces (i.e., multiple identical eigenvalues mixing signal and background components), the **contrastive covariance $S\\_{n}^{+}$ resolves this ambiguity**. Since $S\\_{n}^{+}$ has asymptotically zero energy on pure background directions, the generalized eigenvalue problem can still correctly identify the signal subspace and separate it from the background.\n\n--- \n\nTo empirically validate this claim, we have run a new set of simulations for the setting in Sec E.4 (Figure 4) but with:\n\n*   Signal variances: `[50, 50, 20, 15, 10]`\n*   Background variances: `[500, 500, 300, 50, 50]`\n\nThe results below show that even with these degeneracies, the empirical subspace error for `PCA++` continues to track our theoretical predictions, which were derived under the distinct spike assumption. This provides strong evidence that the assumption is a technical convenience rather than a practical necessity.\n\n**Fixed aspect ratio regime**\n\n| Aspect Ratio | 0.1 | 0.3 | 0.5 | 0.7 | 0.9 | 1.1 | 1.3 | 1.5 | 1.7 | 1.8 |\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| PCA++ | 0.120 | 0.189 | 0.230 | 0.262 | 0.288 | 0.323 | 0.361 | 0.366 | 0.423 | 0.418 |\n| PCA++ theory predicted | 0.104 | 0.179 | 0.229 | 0.268 | 0.301 | 0.330 | 0.356 | 0.379 | 0.400 | 0.410 |\n\n**Growing-spike regime**\n\n| Aspect Ratio | 0.1 | 0.3 | 0.5 | 0.7 | 0.9 | 1.1 | 1.3 | 1.5 | 1.7 | 1.8 |\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| PCA++ | 0.136 | 0.195 | 0.228 | 0.270 | 0.307 | 0.321 | 0.353 | 0.384 | 0.403 | 0.384 |\n| PCA++ theory predicted | 0.100 | 0.171 | 0.218 | 0.256 | 0.287 | 0.315 | 0.339 | 0.361 | 0.381 | 0.391 |\n\nWe will add a remark clarifying the role of these assumptions and include this new empirical validation in the appendix to strengthen our claims. Thank you again for pushing us to be more precise on these important details.\n\n\n\n\n> ... I think it would be important to at least show how well the alignment-only PCA+ does on the data...\n\n**A**:  Thank you for your thoughtful feedback regarding the limitations of our real-world experiments and the importance of evaluating the uniformity constraint in contrastive PCA.\n\nIn response, we have included a comprehensive comparison across several methods, including standard `PCA`, alignment-only `PCA+` (which lacks the uniformity constraint), `PCA++`, as well as additional baseline dimensionality reduction techniques (`U-MAP`, `t-SNE`, `Robust PCA` [Candes et al., 2011]). All methods were evaluated on the same single-cell RNA-seq data, focusing on their ability to recover cell-type groupings in both control and stimulated conditions. \n\nThe Adjusted Rand Index (ARI; lower is better in this context) for each method is summarized in the table below:\n\n|  | Cells | B Cells | NK Cells | All |\n| :--- | ---: | ---: | ---: | ---: |\n| `PCA` | 0.2084 | 0.2977 | 0.2502 | 0.1478 |\n| `PCA+` | 0.0036 | 0.0242 | 0.0299 | 0.014 |\n| `PCA++` | 0.0008 | 0.005 | 0.0049 | 0.0172 |\n| `U-MAP` | 0.2031 | 0.2506 | 0.2078 | 0.1392 |\n| `t-SNE` | 0.0233 | 0.0967 | 0.0477 | 0.0217 |\n| `Robust PCA` | 0.0399 | 0.1907 | 0.1314 | 0.0443 |\n\nAs shown, both `PCA+` and `PCA++` dramatically reduce the ARI compared to standard PCA and other popular methods, indicating much better mixing of the two conditions within cell types. Importantly, adding the uniformity constraint in `PCA++` further improves the results over `PCA+`, especially in B cells and NK cells. These quantitative results highlight the utility of both the alignment and uniformity components. `PCA++` achieves superior separation of invariant populations compared to alternative approaches. We will includ these expanded results and discussion in the revised manuscript.\n\n> I encourage authors to consider citing an important early work on contrastive PCA[1] and comparing the method with existing baselines. \n\n**A**: We thank the reviewer for their suggestion to cite contrastive PCA (cPCA) [1] and for the prompt to compare with existing baselines. We will add this important citation and others from the cPCA literature to our related work section.\n\nRegarding a direct comparison, it is important to note that our work and the cPCA method in [1] address **fundamentally different problem** settings:\n\n*   **cPCA:** This method requires two distinct datasets: a \"foreground\" dataset containing the signal of interest and an explicit \"background\" dataset. Its goal is to find variance in the foreground that is absent in the background.\n*   **Our work (`PCA++`):** We operate in the standard self-supervised learning setting, where we are given only **paired positive views $(X, X^{+})$** of the data, where both views contain a mixture of the foreground signal and background context. Our method does not require access to a pure background dataset.\n\nBecause our setting does not provide the separate background dataset that cPCA requires, their method cannot be applied, which prevents a direct comparison.\n\n---\n\nFor a more comprehensive discussion of the distinctions between our framework and the broader cPCA literature, we respectfully refer the reviewer to our detailed response to Reviewer `[uGvG]`, where we elaborate on these points. Thank you again for bringing this important line of work to our attention."}, {"review_id": "pXzED5lIT0", "reviewer": "Reviewer_uGvG", "summary": "This paper utilizes ideas from contrastive learning literature to propose PCA methods for robust signal recovery under strong background noise. Firstly, the authors introduce PCA with alignment-only contrastive learning (they name it ), analyze the finite-sample performance of the proposed method PCA+, and identify some cases (high background spikes relative to signal spikes or ambient dimension comparable to the number of samples) where PCA+ is not enough for robust signal recovery. To solve this issue, they propose to use a “hard uniformity” constraint, leading to a PCA method that they named PCA++. Then, the authors provide high-dimensional characterizations of the proposed method (PCA++) and a theoretical explanation for the uniformity’s power in contrastive learning. Finally, the paper includes experimental results with simulated data (based on contrastive factor model), corrupted MNIST, and single-cell transcriptomics, indicating PCA++ outperforms standard PCA.", "questions": "1. Are the authors aware of the existence of the mentioned papers about \"contrastive PCA\"?\n\n2. What is the additional contribution of this work compared to the existing contrastive learning and contrastive PCA literature?", "limitations": "Limitations of the proposed method (PCA++) are not discussed explicitly in the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "### Strengths\n1. An easily understandable paper\n2. Studies an interesting and relevant problem: robust signal recovery under strong background noise.\n3. Provides theoretical and empirical results regarding the proposed PCA methods. \n4. Gives a high-dimensional analysis of a uniformity-constrained estimator together with some insights regarding the role of uniformity in contrastive learning, which is valuable from my point of view. \n\n### Weaknesses\n- The main problem with this paper is its positioning in the literature. With the current presentation of the paper, it sounds like this is the first paper to study contrastive PCA. However, there exists some work studying contrastive PCA in the literature (see the examples below). Furthermore, these highly relevant papers are not mentioned or cited in this paper. Therefore, evaluating the contributions of this work is challenging without any comparison with the prior contrastive PCA literature. Due to this major issue, I recommend rejection.\n- Even if the presentation of the paper is fixed by adding the relevant citations and so on, the fundamental differentiating point of this work compared to the contrastive PCA literature seems to be \"uniformity constraint\", which is also an existing idea in the contrastive learning literature (e.g., see [41] from the references of the submitted paper). Therefore, I think the originality/contribution of this work is questionable at this point.\n\n**Example papers about \"contrastive PCA\" in the literature:**\n- \"Contrastive principal component analysis\", A Abid, MJ Zhang, VK Bagaria, J Zou, arXiv:1709.06716, 2017.\n- \"Probabilistic contrastive principal component analysis\", D Li, A Jones, B Engelhardt, arXiv:2012.07977, 2020.\n- \"Exploring high-dimensional biological data with sparse contrastive principal component analysis\", P Boileau, NS Hejazi, S Dudoit, Bioinformatics, 2020.\n- \"Sparse discriminant PCA based on contrastive learning and class-specificity distribution\", Q Zhou, Q Gao, Q Wang, M Yang, X Gao, Neural Networks, 2023.\n- \"Contrastive Functional Principal Component Analysis\", E Zhang, D Li, AAAI, 2025.", "was_revised": true, "final_justification": "The authors properly addressed my concerns and promised to fix the issues that I mentioned in my reviews and comments. I have no further questions/concerns left, so I am raising my rating to 4. My reason for not increasing the rating further is that the initial submission missed (lacked any discussion or citation of) significantly related literature completely, which was and still is a major problem that can only be solved in the camera-ready version.", "rebuttal": "We sincerely thank Reviewer `[uGvG]` for their detailed feedback and for identifying a problem in our paper's positioning. We agree that our failure to cite and discuss the previous \"contrastive PCA\" literature (e.g., Abid et al., 2017; Li et al., 2020, Boileau et al. 2020, Zhou et al. 2023, Zhang et al. 2025, etc) was a major oversight. We see now how this created an inaccurate impression of our work's novelty, and we are grateful for the chance to correct this problem and clarify our specific contributions.\n\nOur response is structured to directly address the reviewer's two primary concerns:\n\n\n1. **Fundamental difference in problem formulation**: Our work (`PCA++`) addresses a very different, and arguably more common, problem setting than the previous contrastive PCA literature (`cPCA`) cited by the reviewer.\n\n2. **Novelty and significance of our contribution**: Our core contribution is a novel and rigorous high-dimensional analysis that yields new theoretical insights into the role of uniformity, not the invention of the uniformity concept itself.\n\n--- \n\n1. **Clarifying our problem setting vs. cPCA**\n\n\nThe reviewer's main concern stems from a misunderstanding of the problem setting, which was caused by our poor positioning. We will rectify this in the manuscript. There is a fundamental distinction:\n\n**Input Data**:\n \n- `cPCA`: Two distinct datasets -- a  **foreground/target** dataset  ($X$) containing both signal and background, and an explicit **background-only** dataset ($Y$). \n\n- `PCA++`: Paired \"positive views\" $(X, X^{+})$ that share a signal but have different background instances.\n\n**Objective**:\n\n- `cPCA`: Find directions of variance in $X$ that are not present in $Y$. (e.g., via covariance subtraction $\\Sigma\\_{X} - \\alpha \\Sigma\\_{Y}$).\n\n- `PCA++`: Find directions that are **shared/invariant** between the positive views $(X, X^{+})$\n\n**Typical Scenario**:\n\n- `cPCA`: Comparing cases vs. controls; comparing two different experimental conditions.\n\n- `PCA++`: Self-supervised learning from data augmentations (standard setting of modern contrastive learning, e.g., SimCLR, MoCo); analyzing paired measurements (e.g., our single-cell data).\n\n\nThis distinction is critical. The methods cited by the reviewer are **not applicable** to our problem setting, as they require access to a pure background dataset, which is unavailable in the standard positive-pair contrastive learning framework. Our work is expressly designed for this latter, widely used scenario.\n\n**Action**: We will revise our **Introduction and Related Work** sections. We will begin by clearly delineating these two distinct \"contrastive\" paradigms. We will properly cite and discuss the foreground-background cPCA literature (Abid et al., etc.) and explicitly state that our work addresses the positive-pair setting, thereby correctly positioning our contribution within the context of modern contrastive learning [8, 10, 41].\n\n--- \n\n2. **On the novelty and significance of our contributions**\n\nThe reviewer suggests our contribution is questionable because the \"uniformity constraint\" is an existing idea. We respectfully but firmly disagree. Our novelty lies not in proposing the concept of uniformity, but in the following non-trivial contributions:\n\n**A. Principled Motivation and Failure Analysis**: We do not simply add a uniformity constraint. Our paper's narrative is a core part of our contribution:\n\n- We first establish a natural baseline, alignment-only `pca+`.\n\n- We then provide a **novel theoretical failure analysis (Thm 3.4)**, proving that this intuitive approach *provably fails* when background noise is strong -- a key insight that motivates the need for a stronger regularizer.\n\n- This directly leads to `PCA++`, where the uniformity constraint is introduced as a **principled and necessary remedy** to this specific, proven failure mode.\n\n**B. New, Rigorous High-Dimensional Theory**: Our primary contribution is a deep theoretical analysis that goes far beyond prior work.\n\n- **Exact Asymptotic Limits**: Instead of loose non-asymptotic bounds, we are the first to provide **exact, closed-form asymptotic limits** for the subspace recovery error using powerful tools from random matrix theory (Thm 4.2, 4.3). This offers a much sharper and more precise understanding.\n\n- **Proving *Why* Uniformity Works**: Our theory provides the first quantitative proof of how uniformity confers robustness. By comparing the error formula for `PCA+` (which depends on background strength) with that of `PCA++` (which is **provably independent of background strength**), we demonstrate a precise, mathematical mechanism. While Wang and Isola [41] noted that InfoNCE *implicitly* encourages uniformity, our work provides a rigorous explanation for *why* this is so beneficial in the presence of structured noise.\n\n**Action**: We will sharpen the language in our abstract and introduction to emphasize that our core technical novelty lies in the **exact high-dimensional characterization of a uniformity-constrained estimator** and the **new insights this theory provides** into the noise-suppressing role of uniformity in contrastive learning.\n\n\n--- \n\n3. **On Discussing Limitations**\n\nWe agree that a clear discussion of limitations is essential and will add a dedicated \"Limitations and Future Work\" section to the revised manuscript.\n\nThe primary limitation of our work—which is also its key analytical strength—is the adoption of a linear contrastive factor model. While real-world data often exhibits non-linearities, this deliberate choice was crucial for achieving our paper's main goal: to provide a precise, quantitative explanation of how uniformity provides robustness in contrastive learning.\n\nThis tractable linear structure enabled us to:\n\n1.  Derive exact, closed-form asymptotic limits for the subspace estimation error (Theorems 4.2 and 4.4), a stronger result than the looser upper bounds common in more general settings.\n2.  Precisely isolate and demonstrate the mechanism by which the uniformity constraint filters out strong, structured background noise—a level of clarity that would be difficult to achieve with more complex, non-linear models.\n\nOur work thus serves as a foundational, theoretical study that provides concrete insights into the role of uniformity.\n\n**Future Work:**\nAn important next step is to extend this framework to handle non-linearities. As the reviewer might anticipate, a natural direction is to develop a kernelized version of `PCA++`, which we plan to explore in future work. This would involve applying the same principles of contrastive alignment and hard uniformity in a reproducing kernel Hilbert space, potentially offering the same robustness benefits for non-linear data.\n\nWe will add this discussion to the paper to clearly scope our contributions and outline promising future directions.\n\n**Action:** We will add this discussion to the paper to clearly scope our contributions and outline promising future\ndirections.\n\nWe are confident that once our paper is revised to properly position our work, the novelty and significance of our contributions will be clear. Our work tackles a different problem than prior cPCA and provides a new, deep theoretical analysis that has not been previously shown. We believe the reviewer's concerns, while stemming from a real flaw in our initial manuscript's presentation, are fully addressable. We hope they will re-evaluate our work based on these clarifications and planned revisions."}, {"review_id": "GJyXvkCko5", "reviewer": "Reviewer_9ppt", "summary": "This paper introduces PCA++, a contrastive Principal Component Analysis method designed to robustly isolate shared signal subspaces from paired observations under strong background noise. The key innovation is the addition of a hard uniformity constraint , which requires the projected features to have identity covariance. Leveraging a generalized eigen decomposition with low-rank truncation for stability, PCA++ admits a closed-form solution and provides rigorous non-asymptotic and high-dimensional guarantees under a linear contrastive factor model.", "questions": "See weaknesses.", "limitations": "1. Some assumptions might be quite strong and not hold for real-world applications.\n2. This method doesn't consider data with non-linear structure.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.\tThis paper introduces a hard uniformity constraint and leverages paired observations to isolate shared signal subspaces, leading to a closed-form generalized eigenproblem solution.\n2.\tThe authors provide a theoretical explanation for how the uniformity constraint enhances robustness in contrastive learning, especially under structured noise.\n3.\tNumerical experiments on synthetic datasets, corrupted MNIST, and real single-cell RNA-seq data demonstrate the effectiveness of the proposed algorithm. \n\nWeaknesses:\n\n1.\tThe assumption which requires orthogonal signal and background subspaces is quite strong and may not hold in many real-world applications.\n2.\tThe method is limited to linear data, whereas many real-world datasets exhibit nonlinear structures.\n3.\tThe strategy for selecting the hyperparameter s is not reasonable. \n4.\tThe limitations of PCA++ are not clearly discussed in the paper.\n5.\tSome typos.", "was_revised": true, "final_justification": "All my concerns have been solved.", "rebuttal": "We thank Reviewer `[9ppt]` for their constructive feedback and for recognizing the paper's strengths. We agree with their assessment of the weaknesses and will address each point to significantly improve the manuscript.\n\n---\n\n**On the Orthogonality Assumption**:\n\n\n> The assumption which requires orthogonal signal and background subspaces is quite strong and may not hold in many real-world applications.\n\n**A**: \nWe thank the reviewer for raising this crucial point. We agree this assumption is strong and adopted it for analytical tractability. However, the core mechanism of `PCA++` is robust to violations of this assumption.\n\n*Why the core mechanism is robust to overlapping subspaces*: The key insight, enabled by the *linearity of our factor model (Eq. 2.1)*, is that we can analyze the behavior of the covariance matrices in a shared basis. Even with overlap, the contrastive covariance $S\\_{n}^{+}$ isolates signal components, while the standard covariance $S\\_n$ accumulates variance from both signal and background in the shared directions.\n\nLet's sketch this for a simple case. Because the model is linear, we can define an orthonormal basis that accounts for the overlap. Suppose `span(A)` and `span(B)` share a single direction $v\\_0$. We can decompose the subspaces as:\n\n-   **Signal space:** `span(A)` is spanned by $ \\\\{ v\\_0, v\\_{A,1}, ..., v\\_{A,k-1} \\\\} $, where $v\\_0$ is the shared part and $v\\_{A,i}$ are the pure signal directions, orthogonal to $v\\_0$.\n\n-   **Background space:** `span(B)` is spanned by $ \\\\{ v\\_0, v\\_{B,1}, ..., v\\_{B,m-1} \\\\} $, where $v\\_{B,j}$ are the pure background directions.\n\nThe population covariances then become:\n\n-   **Contrastive covariance:** \n$$E[S_{n}^+] = \\lambda\\_{A} v\\_{0} v\\_{0}^\\top + \\sum\\_{i=1}^{k-1} \\lambda\\_{A, i} v\\_{A, i} v\\_{A, i}^\\top. $$\nHere, $\\lambda\\_{A,0}$ is the signal variance in direction $v\\_{0}$.\nThus, contributions from the pure background directions $\\\\{v\\_{B,j}\\\\}$ are still averaged out. The resulting expectation remains spanned only by the signal-related directions $ \\\\{ v\\_0, v\\_{A,1}, ..., v\\_{A,k-1} \\\\} $. The shared direction $v\\_{0}$ is *not* cancelled.\n\n-   **Standard covariance:** \n$$E[S_{n}] = (\\lambda\\_{A, 0}+\\lambda\\_{B, 0}) v\\_{1} v\\_{1}^\\top + \\sum\\_{i=1}^{k-1} \\lambda\\_{A, i} v\\_{A, i} v\\_{A, i}^\\top + \\sum\\_{i=1}^{m-1} \\lambda\\_{B, i} v\\_{B, i} v\\_{B, i}^\\top.$$ \nHere, $\\lambda\\_{B, 0}$ is the background variance in direction $v\\_{0}$. The variance in the shared direction $v\\_{0}$ is amplified by both signal and background components.\n\n**Intuition** -- why `PCA++` is robust to violations of orthogonality assumption: The robustness of `PCA++` stems from how the generalized eigenvalue problem $S\\_{n}^{+} v = \\lambda S\\_{n} v$ interacts with these modified covariance structures. We can analyze this through the lens of the asymptotic contrastive energy for each direction, as defined in our analysis for **Lemma C.1**.\n\n*   **Pure background directions ($v\\_{B,j}$):** The contrastive energy $v\\_{B,j}^T S\\_{n}^{+} v\\_{B,j}$ remains asymptotically zero, so these directions are filtered out.\n*   **Shared direction ($v\\_0$):** The contrastive energy $v\\_0^T S\\_{n}^{+} v\\_0$ is strictly positive due to the signal component. However, its variance in the standard covariance is now amplified by both signal ($\\lambda\\_{A,0}$) and background ($\\lambda\\_{B,0}$) components.\n\nThe resulting contrastive energy for shared direction will be smaller than that of a pure signal spike, but it will still be bounded away from zero. Therefore, the `PCA++` objective still detects the shared direction as part of the signal space. The fundamental mechanism—isolating all directions with non-zero contrastive energy—remains intact.\n\n--- \n\n**Empirical Validation with Overlapping Subspaces**:\n\nTo provide strong empirical evidence, we ran new simulations with non-orthogonal signal and background subspaces and will add these to the Appendix. \nWe ran new simulations based on the setting in Sec E.4, but introduced overlap between the signal and background subspaces. We aligned two background directions with the two weakest signal directions, creating a two-dimensional shared subspace. We tested this under two background noise levels.\n\n*   **Signal variances:** `[50, 25, 20, 15, 10]`\n*   **Moderate noise background:** `[500, 400, 300]` (pure background) + `[25, 12.5]` (shared background)\n*   **Large noise background:** `[500, 400, 300]` (pure background) + `[100, 50]` (shared background)\n\nThe results for the fixed aspect ratio regime are shown below. For this demonstration, we compare against our original theoretical predictions (derived under orthogonality).\n\n**Fixed Aspect Ratio with Moderate Overlapping Noise**:\n\n| Aspect Ratio | 0.1 | 0.3 | 0.5 | 0.7 | 0.9 | 1.1 | 1.3 | 1.5 | 1.7 | 1.8 |\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| `PCA++` | 0.137 | 0.207 | 0.269 | 0.287 | 0.317 | 0.311 | 0.356 | 0.388 | 0.416 | 0.431 |\n| `PCA++` theory predicted | 0.104 | 0.179 | 0.229 | 0.268 | 0.301 | 0.330 | 0.356 | 0.379 | 0.400 | 0.410 |\n\n**Fixed Aspect Ratio with Large Overlapping Noise**\n\n| Aspect Ratio | 0.1 | 0.3 | 0.5 | 0.7 | 0.9 | 1.1 | 1.3 | 1.5 | 1.7 | 1.8 |\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| `PCA++` | 0.254 | 0.206 | 0.278 | 0.280 | 0.357 | 0.302 | 0.359 | 0.405 | 0.391 | 0.416 |\n| `PCA++` theory predicted | 0.104 | 0.179 | 0.229 | 0.268 | 0.301 | 0.330 | 0.356 | 0.379 | 0.400 | 0.410 |\n\n**Growing-spike regime with Moderate Overlapping Noise**\n\n\n| Aspect Ratio | 0.1 | 0.3 | 0.5 | 0.7 | 0.9 | 1.1 | 1.3 | 1.5 | 1.7 | 1.8 |\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| `PCA++` | 0.121 | 0.197 | 0.216 | 0.261 | 0.299 | 0.334 | 0.369 | 0.374 | 0.400 | 0.426 |\n| `PCA++` theory predicted | 0.100 | 0.171 | 0.218 | 0.256 | 0.287 | 0.315 | 0.339 | 0.361 | 0.381 | 0.391 |\n\n**Growing-spike regime with Large Overlapping Noise**\n\n\n| Aspect Ratio | 0.1 | 0.3 | 0.5 | 0.7 | 0.9 | 1.1 | 1.3 | 1.5 | 1.7 | 1.8 |\n|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| `PCA++` | 0.138 | 0.182 | 0.214 | 0.262 | 0.316 | 0.323 | 0.363 | 0.375 | 0.380 | 0.397 |\n| `PCA++` theory predicted | 0.100 | 0.171 | 0.218 | 0.256 | 0.287 | 0.315 | 0.339 | 0.361 | 0.381 | 0.391 |\n\n\n**Conclusion**: The results show that `PCA++` is remarkably robust, even when the orthogonality assumption is violated. In the moderate overlapping noise case, the empirical error of `PCA++` continues to track the theoretical predictions remarkably well. This demonstrates that when the background variance in the shared subspace is not excessively large, the impact of the overlap is minimal.\n\nIn the large overlapping noise case, we observe a slight increase in estimation error, as expected. This is because the background noise in the shared directions becomes strong enough to reduce the effective signal-to-noise ratio, making recovery more challenging. Nevertheless, even in this challenging scenario, `PCA++` remains stable and successfully recovers the signal subspace with controlled error, confirming that **perfect orthogonality is not a practical prerequisite for our method's success**. We will add this discussion and the full experiment to the paper.\n\n\n---\n\n**On Selecting Hyperparameter $s$**:\n\n> The strategy for selecting the hyperparameter s is not reasonable.\n\n**A**: We thank the reviewer for this important practical point. We agree that our paper must provide clearer guidance on choosing the truncation rank $s$.\n\nThe choice of $s$ in truncated `PCA++` controls a fundamental trade-off between the stability of the generalized eigenproblem and the effectiveness of the uniformity constraint. We will add a new subsection to the Appendix with practical guidelines for selecting s, centered on the following trade-off:\n\n*   A **small $s$** ensures that the inverse of the truncated covariance $(S\\_n)\\_s$ is well-conditioned, promoting numerical stability. However, it may discard dimensions needed to enforce uniformity effectively, potentially biasing the result.\n*   A **large `s`** enforces uniformity over a larger subspace but risks instability if $S\\_n$ is ill-conditioned, as including directions with near-zero eigenvalues can amplify noise. As shown in our experiments (Figure 2, left).\n\nBased on this trade-off, we propose to **combine the following practical strategies** for selecting $s$:\n\n-  **Information-based criterion (lower bound):** As in standard PCA, one can determine a minimum $s$ by examining the cumulative variance explained by the eigenvalues of the sample covariance $S\\_n$. For example, choose $s$ large enough to capture a significant portion (e.g., 90\\%) of the total variance. This ensures the uniformity constraint operates on the most meaningful directions.\n\n-  **Stability-based criterion (upper bound):** To ensure numerical stability, one can monitor the **condition number** of the truncated matrix $(S\\_n)\\_s$, which is the ratio of its largest to its $s$-th eigenvalue $(\\lambda\\_1(S\\_n) / \\lambda\\_s(S\\_n))$. One should choose $s$ such that this ratio remains below a reasonable threshold, avoiding severe ill-conditioning.\n\nIn our experiments (e.g., Figure 2, right), we found that `PCA++` is robust across a reasonable range of $s$. We will incorporate these guidelines into Section 4 of the paper to make the method more accessible.\n\n---\n\n**On Limitations and Typos**:\n\n> The limitations of PCA++ are not clearly discussed in the paper.\n\n**A**:  We agree with the reviewer that this was an omission. As detailed in our response to Reviewer `[uGvG]`, we will add a dedicated **Limitations**  section to the paper, focusing on the assumptions underpinning our theoretical model.\n\n>  Some typos.\n\n**A**: We will carefully proofread the manuscript and correct all identified errors.\n\nWe are confident these substantial revisions will fully address the reviewer's concerns and strengthen the paper. We are grateful for their thoughtful feedback."}, {"review_id": "4ujSbdnG03", "reviewer": "Reviewer_KG4u", "summary": "This paper proposes PCA++, a PCA algorithm for contrastive learning, which can robustly extract signal from background under a broad class of scenarios. This is achieved by enforcing feature uniformity via a hard constraint. A baseline PCA+ method is first proposed. Theoretical analysis of both PCA+ and PCA++ are provided, as well as a closed form solution. Numerical experiments verify the improved performance in contrastive learning tasks.", "questions": "1. Please include discussion on the computational cost and scalability.\n2. Please discuss the restrictiveness of the model assumptions, and their implications on real-world data scenarios.", "limitations": "Please see above.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strength\n1. The proposed PCA++ framework addresses the challenge in contrastive learning in a robust manner with closed form.\n2. This paper is very technically solid, and has contribution in the theoretical analysis of high-dimensional data analysis.\n3. The insight on the relationship between uniformity and contrastive learning is valuable.\n4. Numerical experiments cover different aspects of the method and scenarios.\n\nWeakness\n1. The assumptions 1-3 are somewhat restrictive, and may be unrealistic in real scenarios, in particular the orthogonality in Assumption 1. The authors should discuss the dependency and sensitivity of the results on these assumptions, either by theoretical analysis or numerical experiments.\n2. The authors should discuss the computational cost of the proposed framework, and it scalability to large datasets.", "was_revised": true, "final_justification": "I appreciate the authors' response to my questions, and will maintain the previous score.", "rebuttal": "> The assumptions 1-3 are somewhat restrictive, and may be unrealistic in real scenarios, in particular the orthogonality in Assumption 1. The authors should discuss the dependency and sensitivity of the results on these assumptions, either by theoretical analysis or numerical experiments.\n\n**A**: We thank the reviewer for their insightful questions regarding the assumptions of our model. We agree that discussing their scope and necessity is crucial. We address the main points below and will add these clarifications to the revised manuscript.\n\n- **On the orthogonality assumption**: (Assumption 2.1)\nWe acknowledge that this is a strong assumption, made primarily for analytical tractability. Our framework's core mechanism is, however, robust to its violation. The key insight is that even if signal and background subspaces overlap, the **contrastive energy** (as analyzed in Lemma C.1) of the shared directions remains strictly positive, while that of pure background directions is zero. This allows `PCA++` to distinguish the full signal space from the background. The main effect of overlap is a reduction in the generalized eigenvalue for the shared directions, but they remain detectable. For a more detailed technical sketch and experimental validation, we respectfully refer the reviewer to our response to Reviewer `[9ppt]`.\n\n- **On the Gaussian latent factor assumption**: (Assumption 2.2)\nThis assumption was also made for analytical convenience, as it allows for the clean derivation of exact constants and closed-form error rates. However, we expect the core results to hold more generally for sub-Gaussian distributions. Many of the key results from random matrix theory that we rely on (e.g., in Lemmas D.4 and D.5) have well-known extensions beyond the Gaussian case, often requiring only a few finite moments. We chose to present the Gaussian case for clarity and precision, but relaxing this assumption is a natural direction for future theoretical work, which would likely involve more complex proofs.\n\n    To provide strong empirical evidence for this claim, we have run new simulations where the Gaussian latent factors $(w\\_i, h\\_i)$ and noise $(\\epsilon\\_i)$ from Assumptions 2.2 and 2.3 were replaced with samples from a standardized **$Beta(2,2)$ distribution** (a symmetric, bounded, non-Gaussian distribution). We repeated the experiments from Sec E.4 (Figure 4) under this new setting.\n    \n    The results below show that the empirical performance of `PCA++` under Beta-distributed noise **continues to align almost perfectly** with our theoretical predictions, which were derived under the Gaussian assumption.\n    \n    **Fixed Aspect Ratio with Beta distribution**\n    \n    | Aspect Ratio | 0.1 | 0.3 | 0.5 | 0.7 | 0.9 | 1.1 | 1.3 | 1.5 | 1.7 | 1.8 |\n    |:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n    | `PCA++` | 0.095 | 0.195 | 0.244 | 0.266 | 0.295 | 0.361 | 0.385 | 0.381 | 0.402 | 0.412 |\n    | `PCA++` theory predicted | 0.104 | 0.179 | 0.229 | 0.268 | 0.301 | 0.330 | 0.356 | 0.379 | 0.400 | 0.410 |\n    \n    **Growing-spike regime with Beta distribution**\n    \n    | Aspect Ratio | 0.1 | 0.3 | 0.5 | 0.7 | 0.9 | 1.1 | 1.3 | 1.5 | 1.7 | 1.8 |\n    |:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n    | `PCA++` | 0.112 | 0.171 | 0.223 | 0.260 | 0.296 | 0.331 | 0.348 | 0.360 | 0.389 | 0.409 |\n    | `PCA++` theory predicted | 0.100 | 0.171 | 0.218 | 0.256 | 0.287 | 0.315 | 0.339 | 0.361 | 0.381 | 0.391 |\n    \n    This remarkable consistency provides strong evidence that the Gaussian assumption is a technical choice for analytical clarity rather than a strict requirement for the validity of our results, which appear to exhibit universality. We will add these new findings to the appendix to empirically support the robustness of our framework.\n\n- **On other assumptions**: (e.g., Assumption 4.1, 4.3)\nWe have also provided a detailed justification for the high-dimensional assumptions regarding the BBP threshold and distinct eigenvalues in our response to Reviewer `[r13i]`. The key takeaway is that the BBP threshold is a fundamental limit of PCA-based detection, while the distinct eigenvalue assumption is a technical convenience that we validate empirically.\n\nWe believe these clarifications properly scope our theoretical contributions, and we will incorporate this discussion into the final manuscript. Thank you again for your valuable feedback.\n\n--- \n\n\n\n\n\n\n>  The authors should discuss the computational cost of the proposed framework, and it scalability to large datasets.\n\n\n**A**: Thanks for the suggestion. We will add a detailed analysis of the computational cost and scalability of `PCA++` to the revised manuscript.\n\n\n---\n\n**Computational Complexity**\n\nThe computational cost of `PCA++` has two main components:\n\n\n1.  **Covariance matrix formation**: We compute the sample covariance $S\\_n$ and the contrastive covariance $S\\_{n}^{+}$. For data matrices of size $n \\times d$ (samples × features), forming these $d \\times d$ matrices requires matrix multiplications with a complexity of $O(nd^{2})$. In the common high-dimensional setting where\n$d\\gg n$, this can be optimized to\n$O(dn^{2})$ by first computing the $n\\times n$ Gram matrix.\n\n2.  **Generalized eigenvalue problem (GEP) solution**: The cost is dominated by the initial truncated eigendecomposition of  $S\\_n$ to rank $s$. Using an iterative solver like the Implicitly Restarted Lanczos Method (IRLM), as implemented in `scipy`, this step has a complexity of approximately $O(sd^{2})$. Since the truncation rank $s$ is typically much smaller than $d$, this step is highly efficient.\n\nOverall, the complexity is comparable to standard `PCA`, primarily driven by the feature dimension $d$.\n\n---\n\n**Empirical Scalability**\n\nTo demonstrate its practical performance, we benchmarked the runtime of `PCA++` while varying the number of samples ($n$) and features ($d$). The results confirm our theoretical analysis.\n\n***Computational Cost of `PCA++` (in seconds)***\n\n| $n, d$ | 100 | 1000 | 5000 | 10000 |\n|:---|:---:|:---:|:---:|:---:|\n| **100** | 0.002 | 0.157 | 2.288 | 9.492 |\n| **1000** | 0.003 | 0.204 | 3.781 | 15.666 |\n| **5000** | 0.009 | 0.662 | 13.919 | 53.610 |\n| **10000** | 0.017 | 1.310 | 26.433 | 100.656 |\n\nSetup: Truncation rank $s=10$, top $k=5$ eigenvectors estimated. Benchmarked on an Intel Xeon CPU @ 2.20GHz.\n\nThe benchmarks show that the runtime is dominated by the feature dimension $d$, scaling quadratically, while it scales nearly linearly with the number of samples $n$. This profile makes PCA++ computationally feasible for typical high-dimensional datasets with tens of thousands of features, confirming its practical scalability."}]

Author rebuttal:

Meta-review: This paper introduces PCA++, a contrastive PCA framework specifically designed for datasets with strong background noise. The paper is well-motivated by an important problem for the contrastive learning audience, with technically solid theoretical results and strong empirical evidence. Three reviewers recommended acceptance while one reviewer recommended rejection in the original review. The negative recommendation was primarily due to mispositioning this paper in the contrastive dimension reduction literature, which is a fair comment given the use of same terminology, as contrastive PCA does exist in the other literature for case-control studies. During the rebuttal,  such concern was addressed by distinguishing between contrastive learning, a self-supervised learning method, and contrastive dimension reduction for case-control studies, as confirmed by the reviewer. Finally, all reviewers recommend acceptance, which I agree with. 

However, in the camera-ready version, the authors should clarify the distinction between contrastive learning and contrastive dimension reduction and cite related references to better position this paper.
Decision: Accept (spotlight)