Contrastive dimension reduction: when and how?
Authors: Sam Hawke
Yueen Ma
Didong Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Contrastive methods
dimension estimation
case-control data
dimension reduction
Primary area: evaluation

Abstract: Dimension reduction (DR) is an important and widely studied technique in exploratory data analysis. However, traditional DR methods are not applicable to datasets with with a contrastive structure, where data are split into a foreground group of interest (case or treatment group), and a background group (control group). This type of data, common in biomedical studies, necessitates contrastive dimension reduction (CDR) methods to effectively capture information unique to or enriched in the foreground group relative to the background group. Despite the development of various CDR methods, two critical questions remain underexplored: when should these methods be applied, and how can the information unique to the foreground group be quantified? In this work, we address these gaps by proposing a hypothesis test to determine the existence of contrastive information, and introducing a contrastive dimension estimator (CDE) to quantify the unique components in the foreground group. We provide theoretical support for our methods and validate their effectiveness through extensive simulated, semi-simulated, and real experiments involving images, gene expressions, protein expressions, and medical sensors, demonstrating their ability to identify the unique information in the foreground group.

Contrastive dimension reduction: when and how?
Sam Hawke
YueEn Ma
Department of Biostatistics University of North Carolina at Chapel Hill shawke@unc.ed.
Department of Statisties & Operations Research University of North Carolina at Chapel Hill myueen&unc.edd
Didong Li
Department of Biostatistics
University of North Carolina Chapel Hill sidongli@unc.ed edu
Abstract
Dimension reduction (DR) is important and widely studied technique in ex- ploratory data analysis. However, traditional DR methods are not applicable to datasets with contrastive structure, where data split into foreground group interest (case or treatment group), and a background group (control group). This type of data, common in biomedical studies, necessitates contrastive dimension duction (CDR) methods to effectively capture information unique to or enriched the foreground group relative to the background group. Despite the development of various CDR methods, two critical questions remain underexplored: when should these methods be applied, and how the information unique the foreground group be quantified? In this work, we address these gaps by proposing hypoth- esis test determine the existence of contrastive information, and introducing a contrastive dimension estimator (CDE) to quantify the unique components in the foreground group. We provide theoretical support for our methods and validate their effectiveness through extensive simulated, semi-simulated, and real experiments involving images, gene expressions, protein expressions, and medical sensors, demonstrating their ability to identify the unique information in the foreground group
Introduction
figh-dimensiioal datasets are ubiquitous in the era of big data, arising from applications such as image data (Shorten and Khoshgoftaar 2019), gene/protein expressions (Bhola and Singh 2018). and medical data from wearable devices (Cho et al. 2021 Banaee et al.) (2013), to name a few. Dimension reduction (DR) crucial in these contexts for various reasons (Fan al. 2014); improving data visualization, reducing computational cost for downstream analysis (Fan and Li 2006). minimizing noise (Thudumu al. 2020). and enhancing interpretability (Johnstone and Titterington, 2009).
DR has been a vibrant research area for several decades, leading to the development of numerous methods (Ayesha al. 2020). Principal component analysis (PCA, (ootellin (1933)), for instance, maximizes the variance in the reduced space and is widely applied across different domains (Jirsa al. 1994 Novembre and Stephens 2008 Pasini 2017). Other methods, such as Multi-Dimensionn Scaling (MDS, Torgerson| (1952)) and Isomap (Tenenbaum et al. 2000), focus on preserving pairwise Euclidean and geodesic distances between samples, respectively. Probabilistic PCA (PPCA, Tipping and Bishop 1999)) incorporates uncertainty quantification through probablistic model. Additionlly,
estimaaiion 38th Conference on Neural Information Processing Systems (NcurIPS 2024).
All code available at

---

techniques such as t-distributed Stochastic Neighborhood Embedding (tSNE. Van der Maaten and Hinton (2008) and Uniform Manifold Approximation (UMAP, McInnes et al. 2018)), preserve the local probability distribution and manifold topological structure in the data, respectively.
In recent years, new type of data has garnered attention, particularly in biomedical research involving case-control studies. Here, data are divided into a foreground group (case or treatment group) and a background group (control group). The objective is identify low-dimensional representations unique to or enriched in the foreground group. This scenario necessitates contrastive dimension reduction (CDR) methods. For example, in a benchmark dataset for CDR on mouse protein expressions (Higuera et al., (015),, the foreground group consists of mice who received a shock therapy. while the background group includes mice that did not. The goal is isolate the protein expression patterns unique to the foreground group, i.e., the shocked mice.
To address this problem, several CDR methods have been developed. For example, Contrastive PCA (CPCA, [Aidd et al.] (2018) extends PCA by maximizing the variation in the foreground while minimizing variation in the background. Additionally, Probabilistic Contrastive PCA (PCPCA, Li et al. (2020) generalizes both CPCA and PPCA by incorporating a probabilistic model into the contrastive setting, enabling statistical inference. Furthermore, the Contrastive Latent Variable Model (CLVM, SSevesoon et al. (2019) introduces latent variables specifically designed for this contrastive setting. Moreover, the Contrastive Variational Autoencoder (CVAE. Abid and Zou (2019) and Contrastive Variational Inference (CVI, |Weinberger et al. (2023)) both leverage deep learning approaches detect nonlinear patterns the data. Collectively, these methods aim isolate patterns unique to the foreground data relative to the background. These methods have been successful with a variety of datasets, identifying patterns that are often scientifically meaningful and opening doors to new scientific discoveries. For example, studies using these methods on mouse protein expression data have successfully identified subgroup structures, such as distinguishing between mice with and without Down Syndrome, and highlighted key proteins responsible for these differences. We reference various applications and discoveries using these methods in section and appendix [D]
Despite the development of these CDR methods, two critical questions remain underexplored: when should these CDR methods be applied, and how many unique components exist in the foreground group? The first question can be reframed as follows: given foreground and background groups, can we determine whether there exists unique information in the foreground group? The second question pertains to estimating the number of unique representations to the foreground, termed contrastive dimension in this paper. This is a counterpart of the intrinsic dimension estimation problem, with a rich literature (Camastra and Staiano 2016 Levina and Bickel, 2004), but in the contrastive context.
In this paper, we answer the first question by proposing hypothesis test determine the existence of unique information in the foreground data. To answer the second question, we introduce the rigorous notion of contrastive dimension and provide a consistent estimator of this quantity. We provide theoretical support of our methods and validate their effectiveness through extensive simulated, semi-simulated. and real experiments involving images, gene expressions, protein expressions, and medical sensors.
The rest of the paper is organized as follows. In section we provide more background on CDR
methods and present our definition of contrastive dimension. In section we present our solutions to
the above questions: a hypothesis test and estimator for the contrastive dimension. In section 4 we prove the consistency of our proposed estimator of the contrastive dimension and establish a finite-sample error bound. We demonstrate the effectiveness of the methods simulated and semi-
simulated data in section 5) and real data in section 6 Finally. in section 7 we discuss the strengths and weaknesses of our methods and identify potential directions for further study.
2 Background
In this section, we introduce our notations and provide a more detailed overview of various existing CDR methods. Throughout this paper, we denote the foreground data as X = {x1,-.. .nn C R with dimension p and sample size ng. and background data as Y = {y1, ... BBny C RP with the same dimension p but potentially different sample size Ny- We do not make any assumptions the ratio of sample sizes between groups, n2 ny. For simplicity, we also assume each dataset is centered separately. Next, we briefly overview some existing CDR methods.
Contrastive Principal Component Analysis (CPCA). CPCA (Abid et al., 2018) was proposed to uncover the now-dimensional structure that is unique to the foreground data X relative to the
2

---

background data Y. Taking one- one-dimensional CPCA as example, the objective is to find direction, represented by a unit vector v E IR,, that explains more variance in the foreground among all " directions, and less variance in the background. Let Cx Di= zany and Cy "y شز= زلار be the sample covariance matrices of two datasets. Then, CPCA solves the following optimization problem:
argmax v C' -pu Cyu= argmax A. C'u, |=||=1
|||||=1
where 7 E [0, oo] a tuning parameter, called the contrastive parameter, and = Cx - Cy is called the contrastive covariance matrix. The optimal v is simply the top eigenvector of C. When y = 0, CPCA coincides with PCA, and when 7 = oo, CPCA finds the direction that explains the least variance in the background group.
Contrastive Latent Variable Model (CLVM). CLVM (Severson et al. 2019 a latent variable model, similarly proposed to discover patterns enriched in the foreground group X:
2; S2, + Wt. + Es. V// S2y + Ej:
where 2;, 2j Redt and 1; E RC are the latent variables, and Eg E R are the noise terms. The factor loading S E R represents the space shared between the foreground and background groups, while the factor loading W R represents the space unique the foreground group. The primary goal of CLVM is to identify W, which represents the snreground-speccfn information.
Probabilistic Contrastive Principal Component Analysis (PCPCA). PCPCA (Li 2020) was proposed as a probabilistic model to extend CPCA, allowing for uncertainty quantification:
W + Ez; = Wzy + Ey:
where W represents the factor loading. and Ex; noise terms. However, PCPCA utilizes an unconventional strategy for estimating W. Instead of maximizing the joint likelihood, which would be appropriate if the foreground and background groups are assumed to share the same space, the PCPCA objective to maximize p(X
p(Y]W)>+ where 7 E [0, oo] is the contrastive hyperparameter, as in CPCA. When 7 = 0, PCPCA reduces to Probablistic PCA (Tipping and Bishop 1999) on the foreground data, while PCPCA reduces CPCA as the noise level goes zero.
Contrastive VAE (CVAE). CVAE (Abid and Zou 2019) considers a broadened view of the data- generating process, in which the foreground data X follows an arbitrary probability distribution with unknown parameters 6, conditional on salient variables and irrelevant variables z. The background data Y are assumed not to have the salient variables, as the CVAE model is given by
2t fe(-|s;, 21), 8/ f0(-
The problem here is to learn the parameters of f. which CVAE does by training two probabilistic encoders go, (s|zz and go. (z|z) (salient and irrelevant) to infer and respectively, from the observed features, and decoder network f-(-) which reconstructs the original samples from æ [s, 2] in the foreground data, or y [0, 2'} in the background data.
Contrastive Variational Inference (CVI). CVI was proposed in Weinberger for the purpose of disentangling the split between treatment and control groups for single-cell data, To this end, CVI posits a lata-generating process with latent variables Σn. representing shared information between the two groups, and In. representing information unique to the foreground group. Each gene expression each sample of the foreground group is assumed to follow a generative process depending on on and t, (both with (N 1) priors), but in the background group fn 0 is assumed, similarly to CVAE. The posterior distribution of CVI is analytically intractable, so is approximated using variational inference.
(2023) specifically
Each of the methods above has a similar goal: to discover variation unique to the foreground group relative to the background. However, none of these methods enables detection of whether there exists such variation at all. Additionally, for each method, the assumed reduced dimension (or number of latent features unique to the foreground) was treated as tuning parameter required as input to run the method, without an easy-to- tompleeettt method to choose such a dimension.
Motivated by this gap, we consider the following linear model:
2 + Ei, Uj Wi + Eg. i 1, Bay j = 1,...,
........
(1)
3

---

where 2, E V2, W/ E Vy for some low-dimensional linear subspaces Vz, V/ C R", and Ey represent the noise. We let z represent foreground and y represent background.
However, defining and identifying the information that is truly unique to the foreground requires more rigorous approach. For example, in CLVM, even if we estimate nonzero W, it may not present information unique to the foreground if' S and W are not appropriately distinguished. This motivates the need for precise definition of the contrastive subspace and contrastive dimension:
Definition 1. Under the above notation, we define the contrastive subspace to be Vay == Proivy (Va) and the contrastive dimension, denoted by day. to be the dimension of the contrastive subspace: day dim(Va v-
The rationale of our definition is that the contrastive subspace contains the information unique to the foreground data, guaranteed by the projection to V the orthogonal complement of Vg. In order to determine the existence of unique information in the foreground group, it is desirable to have hypothesis test for testing Ho day versus H1 day 0. Furthermore, an estimator for day is valuable for providing suggestion to use for the reduced dimension parameter in the CDR models presented above.
Remark 1. Based on model and definition 7 V C V is equivalent to dzy 0.
Remark 2. Even minute departure of Vz from Vy results in a nonzero day-
Because we are interested in detecting where the foreground differs from the background, we allow framework to detect
small departure of V2 from Vy, although bigger departure will be easier to detect. This motivates our study of the principal angle between V/2 and V, in section 3
It is worth noting that the new tasks we are proposing, namely determining whether there is unique information in the foreground group and quantifying how much, depart from the traditional CDR task of estimating the space representative of such unique information, assuming it exists. In this paper, we focus on our solution to these new tasks, starting with manageable assumptions about the data.
3 Methods
In this section, we present our methods to address the questions of whether there exists unique information in the foreground group and how many dimensions there unique to the foreground group. First, we detail the hypothesis test of H, day = O versus H₃ day > O. We introduce a contrastive version of the bootstrap resampling scheme, based on the null assumption that day 0, which is equivalent to the assumption of V C Vy, Next, we introduce the CDE of day- In both methods, we make the very reasonable sample size assumptions that nt, dz and my >> dy. many existing studies have observed a small intrinsic dimension (Pope et al. (02T). The inference in both of these problems based on the notion of principal angles between two linear subspaces YYe and Lim 2016):
Definition 2. Let and V be two subspaces of R with dimensions di and d2, then the principal angles, denoted by (U, V), i = 1, 1.... min(d1, d2) are defined recursively as follows
max cos 0 (U, V)) V.EUmeVV 1.0
To be clear, the n; and v; solving the maximization problem are not uniquely defined; however, the principal angles 0; are uniquely defined. The principal angles measure the "difference" between two subspaces. For instance, if two spaces coincide, all principal angles are zero; when two spaces are orthogonal to each other, all principal angles are /2. Uo and Vo are orthogonal matrices spanning U and V, respectively, then the calculation of the principal angles is given by the following equation
(U, V)) o, (U, Vo).
where (:(+) denotes the i-th singular value of a matrix. As result, we adopt this notion to study the relation between Vn and Vg for our purposes.

---

3.1 Hypothesis test
We develop hypothesis test for testing Hu dry versus H1 day > 0 under the version of m with
with the following assumptions,
zi= Sgz + Eis Vj Sywj + Eji where S₂ IRpxd, and Sy -Rxxd are full rank, 24 ~ Na. (0, 1). w, Na, (0, Ble.] Eley] 0, Cov(e,) o.I. and Cov(E,) = =
(3)
First, we assume max dy) < p/2, which is very weak assumption given that p is often much bigger than the intrinsic dimension in high-dimensional datasets.
Remark 3. With m [ we do not make any assumptions the relations between intrinsic dimensions d, and dy However, by definition 7 dz > dy implies that day > O, indicating that there must be some unique information in the foreground space. Therefore, for our hypothesis test, we focus on the more challenging but important case of dg S dy-
An equivalent formulation of the null and alternative hypotheses in this model is Ho C(S,) C C(Sy) versus H1 C(S²) o C(S,), where C(.) denotes the column space of matrix. Motivated by the assumption that V C Vy under Ho, we draw a sample of size fy with replacement from [fn,...,In,, ). and we draw sample of size ny with replacement from (.1,..., 11.,-11,. V1.... Vn, }- We repeat this bootstrap procedure B (B = 1000 throughout this paper) times and compare the largest angle between Ve and V, in the original data with the distribution of the largest angles between those subspaces in the resampled data. The exact calculation is deferred to section 3.2
Algorithm 1: Contrastive Bootstrap Hypothesis Test
Input: Foreground [2, Background {y, )/=1 dimensions d, S dy. bootstrap parameter
Output: p-value for test of Ho V c. Vy in model [ Å < (1, y, da, dy): for b 1 to B do
// Compute as in Algorithm
J. + sample with repl.(x: nx): )
// Resample foreground Resample background, pooled
y+ t sample with repl. (z. y) n2y)
gg + (2 Vv, de, dy);
// Compute as in Algorithm
p
return p
We demonstrate with simulations in section 5 that this hypothesis test produces conservative results, and in section 7 we discuss a potentially more powerful likelihood-basse alternative. Additionally, this hypothesis test requires input intrinsic dimension estimates for dy and dy. for which there exists a rich literature with a variety of sophisticated methods (Campadelli et al. 2015). This conservatism is designed to reduce the likelihood of false positives, but it can also lead to some signals not being detected by the hypothesis test. One explanation for this phenomenon could be that the test's resampling technique might not handle nonlinear structures in the data effectively (e.g. the data lie on a nonlinear manifold). Because V, and V,, are assumed to be linear subspaces, a nonlinear pattern obscure the differences the test is designed detect.
3.2 Contrastive dimension estimator (CDE)
We start from the following lemma link our target, dzy- and principal angles between Vj and Vg- Lemma 1. dzy = dimoprojy. Vx) = # (i: (V Vy)) > 0} + max(d, - dyy 0).
As result, to construct an estimator of drys it suffices to estimate Vz, Vy. and calculate the corre- sponding principal angles. Based on eq. (T). V. and V,, consist of the top d, and dy eigenvectors of Σ, and Eg. respectively, which can be estimated by sample covariance matrices Ex 1 Ei= Z. and E.y Tu Σド yiu] Thus, we propose the following estimator:
no
1
Jen = # {i: 00 (0 (eigidd, ($mm) eigiad, (S,))) > c) + -mxx(d, dg. 0),
(4)
5

---

where > O is small toleration due to randomness from finite samples, and eign.d refers to the span of the first d eigenvectors. Let Ü, = eigidd, 1:d_ Σx and Uy == eigind, (5y) by eq. (2, we have
X (0.) (U Uy), leading to algorithm
Algorithm 2: Contrastive Dimension Estimator
Input: Foreground (21,...,, Eng F: ), Background {yi..... Vn, dimensions du. dy tolerance
Output: day
(E,, Eny (n, Ei 2, ny Ei= "y'Ei= ):
// Compute sample covariances
2 (Uns Uy) (iigind (Sez); , eigi:dy (Emz))
// Compute sample eigenvectors
3 UDVT - Uy (A....., Amin dg) dDag(D);
// SVD
// Singular values
day #li: 23 < 1 - e) max(da dy,
return dzy
Note that algorithm 2 takes intrinsic dimensions dy and dy as input. In practice, the user must choose method estimate the intrinsic dimension of both the foreground and background datasets. This flexibility allows for the incorporation of different intrinsic dimension estimators to complement our approach (Bac et al.. 2021 Pope al., 2021).
Some may only be interested in estimating the contrastive dimension dzy and not in the hypothesis test for dzy = O. These users can skip the hypothesis testing step. However, the hypothesis test offers a direct way to assess uncertainty, guiding users on whether it is worthwhile to proceed with contrastive dimension reduction methods.
In the next section, we study the asymptotic and finite-sample behavior of this estimator day.
4 Consistency and finite sample error bound
In this section, we provide theoretical support for the estimator dzz presented above, first by estab- lishing consistency, and then by constructing finite-sample error bound for it.
Theorem 1. Assume that the second moments Σg and My are finite for both groups, and that the top de + 1 eigenvalues of Eg and top dy + 1 eigenvalues of My are distinct. Then, proposed estimator dry is consistent: day +200 day-
ny +0
The assumption of distinct eigenvalues is common in the literature (Vershynin 2018 and ensures the identifiability of eigenvectors for each covariance matrix. Because we are interested in convergence to the space spanned by the top eigenvectors, rather than individual eigenvectors themselves, this assumption might be relaxed to the eigengap assumption that the dy and (dN + 1 1)th eigenvalues of Σx are distinct, and that the th and (dy + 1)th eigenvalues of E, are distinct. See section 7 for more discussion.
To describe the finite sample error bound, we recall that U2 and U, are spanned by the top d, and dy eigenvectors of Eg and Ey, respectively, and singular values of W = UT_ Uy, denoted by A = o, (W)) I, m, where m = min(dg. d,y, are the cosines principal angles between U- and Uy, In practice, estimating the contrastive dimension for providing a finite sample error bound. Instead, we report the singular values to account for more uncertainty and provide a more nuanced understanding. Thus, present the following finite sample error bound (and hence the convergence rate) of these estimated singular values rather than day-
integer can be somewhat restrictive
Theorem In addition to all assumptions in theorem ! we assume I and y model 7 are sub- Sab--aassian. More precisely, assume that there exist constants Kz, K2 2 1 such that
(2, 1l 5 K,E ()) and 11 vill I Ky u) for any v E RR

---

where 11-1lgh is defined as in Section 3.4 of Vershynin 2018). least 1 20 we have
Then, for any u > 0, with probability at
by 1 SC
Σ kE(aNy)
Vd K2 (VP+*
p
|E On 1/2 nl 1/2)
max j=
nhk
n
(5)
where C is absolute, positive constant, 8, is the minimum eigengap among the top d. + 1 eigenvalues E.. and ||+| denotes spectral norm.
To show the finite sample error bound, we need the Sabb-Gaussiantt assumption to ensure that the tails of the distribution are not too heavy. This assumption is also common and encompasses large class of distributions (Vershynin) (0018).
Having established the theoretical support for the estimator, in the next section we focus on applying our methods to simulated data.
Simulations
To demonstrate the validity of our methods in practice, apply them to four simulations for different purposes. The first two examples are purely simulated, fully under our control, with day = 0 for the first one and dzy 0 for the second one. Because these first two simulations are purely synthetic, repeat them both 100 times and report the means and standard deviations of the p-value, dzy. and the four smallest singular values. The third simulation steps away from the ground truth being completely known by combining synthetic image data with known dimension, with real, grassy noise image data, elucidating the methods' performance in an interpretable setting. The final simulation is variation on the third one, in which the synthetic data are replaced with MNIST images (Deng 2012).
We describe the experimental settings of the simulations in more detail below and present results, including the p-value from test day. and the smallest four singular values, shown in Table 4 We denote the estimates with hat, needed. In all these examples, the threshold c = 0.1.
We do not compare dry with other estimators of contrastive dimension because, to our knowledge, no such estimators have been proposed in the literature.
Table 1: Summary of simulation results
Setup
d,
dy
day
p-value
day
smallest singular values
Simulation 1
6
8
0.814
0.44
0,901, 0,939, 0.961, 0.975 0.059, 0.106, 0.159, 0.215
Simulation 2
6
8
6
0.029
6
Sim. 3: Disk Line
4
20
1
0.000
1
0.490, 0.975, 0.981, 0.988
Sim. 4: MNIST Line
11
14
1
0.000
1
0.407, 0.904, 0.955, 0.975
Simulation 1. In this first simulation, we generate data from the specific case of model with variance o. of = 0.25, C(S.) C C(S,) such that day = 0, nz nyy 100. Because day 0 is the assumption of Ho for the hypothesis test in Section 3 we repeat this simulation 100 times to empirically demonstrate the conservative type I error rate (defined by the proportion of p-values below a) versus pre-specified a E (0, 1). We also repeat this simulation but with nz = ny = 200, and we present both plots of empirical type error rate in Figure 1
As shown in the first row of Table both dzy and the p-value agree, average, with the ground truth that dzy = 0, showing that in this simulated setting, our methods do not erroneously detect a signal when there is none. However, given that the average smallest singular value value observed is 0.901, this simulation suggests that may be prudent to use a choice of greater than 0.1, i.e. a cutoff for the singular value that is below 0.9. The standard deviation for day among the 100 replications is 0.52, which also suggests that a lower cutoff may be appropriate. The standard deviation of the p-value is O. 05153, and of each of the smallest singular values is 0.025, 0.013, 0.009, and 0.007, respectively. Both plots of observed type error in Figure demonstrate that, in this setting, the hypothesis test conservative. We discuss more powerful potential alternative in Section 7
7

---

Observed Type Error Rate
0.75
0.75
0.50
0.50
0.25
025
0.00
0.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
alpha
alpha
Figure 1: Hypothesis test Type error plots. Left: f2 ny 100. Right: f2 n2y 200.
B
C
Figure 2: Samples for simulations 3 (AB) and 4 (CD). AC: add line. BD: add grass.
Simulation 2. The only difference from Simulation is that day = 6. Because the singular values all well below the cutoff of 0.9, the standard deviation of dry among the 100 trials O; every trial achieved the correct estimate. Our hypothesis test produced small p-value of 0.029 on average (standard deviation 0. 0.029), which significance level of a 0.05 leads one to the correct conclusion of rejecting Ho and concluding that dæy > 0. Moreover, the six smallest singular values 0.059, 0.106, 0.159, 0.215, 0.279, 0.367 (standard deviations 0.023, 0.028, 0.032, 0.034, 0.038, and 0.050, respectively). all well below the threshold of 0.9, illustrating that CDE correctly identifies directions unique to the foreground data.
also useful to evaluate the performance our methods on datasets where the ground truth is only partially known. So, in this simulation, construct semi-synthetic image dataset by generating 5000 disks of random centers (cz, Cy) E (5...., 24]² and radii E (1,...,1)) uniformly, in the 28 by 28 grayscale pixel space.
Simulation 3.
Of these 5000 images, we draw a white line of random length beginning at the leftmost pixel of the sixth row on 2500 images to create the foreground (Fig. 2 As result, d, = 4. parameterized by Cz. Cy. T, and length of line, For background, we superimpose the remaining 2500 images on grass images noise (Fig. 2 so that dy d,- Although d, is not known, dry is known to be one (the unique parameter being the length the line). Here, we used the method of moments estimator (Amsaleg et al. 2018), leading to d, 20.
Our method only gives one singular value (0.529) below the cutoff of 0.9, leading correct estimate of d 1. Additionally, our hypothesis test produces p-value O, which suggests to reject Hy in favor of H1 day > O. This simulation illustrates that our methods produce favorable results when one of the groups has an estimated intrinsic dimension.
Simulation 4. To assess our methods in a simulation setting where both d, and d, are unknown, we consider very similar simulation to the previous one, but where start with 5923 handwritten 0's from the MNIST dataset (Deng) 2012) instead of randomly generated disks. We further randomly split them into two sets, create foreground by adding lines (Fig. (C), and background by imposing on grass images (Fig. (D), leading to the corrupted MNIST data proposed in Abid et al. (018). Using method of moments, d, 11 and 4, 14. Crucially, the contrastive dimension day 1 is still known, allowing us to assess the results of our methods when d, and dy are both unknown.
CDE again accurately estimates exactly one singular value (0.256) below the cutoff of 0.9, leading to correct estimate of dzy 1. Furthermore, our hypothesis test produces p-value of 0, leading to the correct conclusion that day > a significance level of a 0.05. After successful applications of our methods to various settings, it is time to apply them real datasets.

---

Real data experiments
In this section, we apply our methods to real datasets previously studied the literature to validate their effectiveness and interpret the results. In T we summarize relevant characteristics of each dataset along with the results. For each dataset, the intrinsic dimension estimation was performed with the method of moments estimator (Bac ct al. 2021). Note that in each case where dz > dy. the p-value is not reported because the hypothesis test does not apply. The cutoff for the singular values set to 0.9 in all experiments.
Table 2: Summary of Results for Real Data Experiments
Dataset
p
n
n
d₂
dy
p- Value
dzy
4 Smallest Singular Values
Noisy MNIST
784
5000
5000
20
20
0.008
0.095, 0.315, 0.705, 0.846
Mouse Protein
77
270
135
5
4
4
0.319, 0.472, 0.886, 0.919
mHealth
23
6451
3072
6
10
0.000
0.394, 0.995, 0.997, 0.999
BMMC
500
4501
1985
26
30
0.073
17
0.042, 0.123, 0.182, 0.306
Small Molecule
1000
3096
2831
12
11
0.627, 0.699, 0.894, 0.925
ECCITE-Seq
1000
18343
2386
41
32
13
0.850, 0.857, 0.874, 0.877
Pathogen
1000
4481
3240
24
25
0.161
0.259, 0.345, 0.528, 0.700
Perturb-Seq
1000
24913
8907
49
47
28
0.038, 0.102, 0.131, 0.176
CelebA: Glasses
1000
13193
189406
21
28
0.138
0.215, 0.972, 0.975, 0.984
CclebA: Hat
1000
9818
192781
22
28
0.063
2
0.702. 0,803, 0.924, 0.938
Corrupted MNIST. The corrupted MNIST dataset was previously studied extensively by Abid et al. (018)) Abid and Zou (2019). Severson et al. (2019), and et al.] (2020). The foreground group depicts handwritten 0's and I's corrupted by images of grass, and the background contains only the grassy noise (no handwritten digits). In this case, the digits are unique to the foreground, supported by our p-value of 0.008. Furthermore, the contrastive dimension estimate for this dataset is dzy 5. in line with the commonly believed dimension of digits and (Pope et al.) 202T).
Mouse Protein. This dataset (Higuera et al. 2015) is a benchmark in the literature, considered in Abid et al. (2118) Li et al. (2020), and Severson et al. (019)) The foreground group contains the protein expressions of mice, some with and some without Down Syndrome, who were subjected to shock therapy, while the background group consists of protein expressions of mice without Down Syndrome who did not receive shock therapy. The intrinsic dimension estimates of dz 5 and Ły = 4 require that day > 0. In this setting, where the foreground space is bigger than the background space, the foreground group necessarily carries unique information. Nevertheless, the CDE estimate for this dataset of day 4, which aligns with the observation of day 2 in the literature.
mHealth. The mHealth dataset (Banos et al., 2015), studied in Abid et al. (2018); Severson et al. (2019), consists of measurements made from medical sensors while subject performs certain actions. While the foreground group contains measurements taken of a subject either squatting or cycling, the background measures subject lying down. Our p-value < 0.001 and d = 1. This estimate explains the result in Abid et al. (2018), duplicated here in figure 3 showing that a single direction suffices to distinguish subgroups the foreground data.
BMMC. This is a single-cell RNA sequencing dataset (Zheng et al., 2017 studied in Abid al. (2018), Abid and Zou (2019), Severson et al. (2019), Li et al. (2200). and Weinberger et al. (2023). Here, the foreground group consists of gene expressions measured on bone marrow mononuclear cells (BMMCs) of patients with acute myeloid leukemia before and after receiving transplant. The background group consists of healthy patients. Previous studies found that the 3rd and 4th PCs can
separate pre- and tost-transplant groups (Fig. 3). supported by our dzy = 17 > 4. However, the p-value of 0.073 stands somewhat in contradiction to this result, maybe due the conservative nature of this hypothesis test, as illustrated in figure and discussed in section 7
Small Molecule. This is dataset of cell line responses to small-molecule therapy (McFarland et al. 2020) and used in Weinberger et al.) (2023). Here, the foreground group contains measurements from 24 cell lines treated with idasanutlin, while the background are the same cell lines treated with

---

the control dimethyl sulfoxide. de 12 and d, = 11 imply that day > O. Nontheless, day = 4. suggesting that CDR method with reduced dimension of 4 appropriate.
This ECCITE-Seq dataset (Mimitou et al. 2019) was studied in Weinberger et al. (2023), containing joint RNA and surface protein expressions of pooled CRIPSR screens. The foreground group consists of cell transcriptomes of perturbed cells, while the background is a set of control cells. d₂ 41 and dz = 32 imply that day > 0. Therefore, the application of contrastive methods is suitable. Moreover, the CDE estimate of dry 13 helpfully gauges the number of directions unique to the foreground, recommending reduced dimension of 13.
ECCITE-Seq.
Pathogen Data. Studied by Weinberger et al. (2023), this dataset (Haber et al.) 2017) contains gene expressions in the epithelial cells of mice infected with either Salmonella or H. Poly (foreground) or healthy mice (background). day 10, suggesting that contrastive method might fit well with reduced dimension of 10. However, the p-value of 0.161 suggests that linear contrastive method may not be appropriate. In fact, WWinberger et al. (2023) shows that only CVI, deep learning based approach, works for this dataset, compared with linear methods, which aligns with our results.
Perturb-Seq: Studied by Weinberger et al. (2023), , this dataset (Adamson et al., 2016) includes gene expressions measured of cells subjected to -edditted perturbations (foreground) and cells treated with control guides (background). The hypothesis test is not applicable since dz > dy. Regardless, CDE provides helpful estimate for the amount of information unique to the foreground group, suggesting the efficacy of CDR methods.
CelebA: The CelebA dataset (Liu et al., 2015) was studied in Abid and Zou (2019) and with images of celebrities. We perform two experiments, with foreground being those wearing glasses and hat, respectively. In both, the background group is the remaining images. In the former case, day = 1, while in the latter day 2. However, both cases have a somewhat large p-value, perhaps due to the conservative nature of the hypothesis test or nonlinearity.
7 Conclusion and Future Work
In this paper, we have introduced a hypothesis test determines whether linear CDR model is appropriate for a given dataset and an estimator of contrastive dimension, representing the dimension of the subspace unique to the foreground group relative to the background. This estimate of contrastive dimension can be used as the reduced dimension in downstream analysis with CDR method. Meanwhile, there are still some open problems left to future work.
The problem of estimating intrinsic dimension is challenging in its own right (Camastra and Staiano, [2016] Campadelli et al. (2015) Fefferman al. 2016). While methods for it have been developed, there is not single accepted standard for it. Fortunately, since day - dzy < de dx + d., - d our day is robust when d₂ and dy are mis-specifying d, and dy in the Lipschitz-1 sense.
Intrinsic Dimension Estimator:
Hypothesis Test. Our hypothesis test introduces the notion the contrastive bootstrap to allow for resampling under the assumption of the null hypothesis that the foreground space is subset of the background space. However, a more powerful likelihood ratio test is applicable. Under model 3 the denominator is exactly from the solution of PPCA Tipping and Bishop [199)). However, the numerator involves a nontrivial constrained optimization problem. Naive gradient step method is neither effective nor scalable. An interesting future direction is resolve this optimization issue.
Uniqueness of Eigenvalues. In section 4 we mentioned that the distinctness of eigenvalues ensures the identifiability of eigenvectors for each covariance matrix. However, this assumption can be relaxed since our focus is not on estimating each individual eigenvector but rather on identifying the eigenspace spanned by the top d. and top d, eigenvectors of Σ,, and Egy
Extension to Nonlinear Setting. In section we mentioned that the data lying on nonlinear
manifold could lead to the apparent inconsistency between the estimator dzy being nonzero and the y-value produced by the hypothesis test being large. To appropriately handle the nonlinear case, future work may consider a nonparametric extension analogous to the extension of CLVM (Severson) et al. 2019) to CVAE (Abid and Zou| 2019).
10

---

Acknowledgments and Disclosure of Funding
SH was supported by NIH grants T32ES007018 and UMI TR004406; DL was supported by NIH grants R01 AG079291, R56 LM013784, RO1 HL149683, and UMI TR004406, R01 LM014407, P30 ES010126.
References
Abid, A., Zhang. M. J.. Bagaria, V. K., and Zou, J. (2018). Exploring patterns enriched a dataset with contrastive principal component analysis. Nature communications, 91):2113.
Abid, A. and Zou, J. (2019). Contrastive variational autoencoder enhances salient features. arXiv preprint 19Xi::/902.04000
Adamson, B., Norman, T. M., Jost, M., Cho, M. Y., Nuñez, J. K., Chen, Y., Villalta, J. E., Gilbert, L. A.. Horlbeck, M. A., Hein, M. Y., et al. (2016). A multiplexed single-cell crispr screening platform enables systematic dissection of the unfolded protein response. Cell, 167(7):187-1182.
Amsaleg, L., Chelly, O., Furon, T., Girard, S., Houle, M. E., Kawarabayashi, K.-i,, and Nett, (2018). Extreme--aluu-theoree estimation of local intrinsic dimensionality. Data Mining and Knowledge Discovery, 1786))176
Anderson, T. W. (1963). Asymptotic theory for principal component analysis. The Annaly of Mathematical Statisticy, 34(1):122-148.
Ayesha, S., Hanif, M. K., and Talib, R. (2020). Overview and comparative study of dimensionality reduction techniques for high dimensional data. Information Fusion, 59:44-58.
Bac, J., Mirkes, E. M.. Gorban, A. N., Tyukin, I. and Zinovyev, A. (2021). Scikit-dimension: a python package for intrinsic dimension estimation. Entropy, (31)))1366.
Banace, H., Ahmed, M. U.. and Loutfi, A. (2013). Data mining for wearable sensors in health monitoring systems: review of recent trends and challenges. Sensors, 13(12)17472-17500.
Banos, O., Villalonga, C., Garcia, R., Saez, A., Damas, M., Holgado-Temrinne Terriza, A., Lee, S., Pomares, H., and Rojas, (2015). Design, implementation and validation of novel open framework for agile development of mobile health applications. Biomedical engineering online, 14:1-20.
Bhola, A. and Singh, S. (2018). Gene selection using high dimensional gene expression data: an appraisal. Current Bioinformatics, 13(3):22--23..
Camastra, F. and Staiano, A. (2016). Intrinsic dimension estimation: Advances and open problems. Information
Sciences, 328:26-41.
Campadelli, P., Casiraghi, E.. Ceruti, C., and Rozza, A. (2015). Intrinsic dimension estimation: Relevant techniques and benchmark framework. Mathematical Problems in Engineering. 2015:1-21.
Cho, S., Weng, C., Kahn, M. G., Natarajan, K., et al. (2021). Identifying data quality dimensions for person- generated wearable device data: Multi-method study. JMIR mHeallh and uHealthh 9 (12)e31618.
Deng, L. (2012). The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 1916)144
Fan, J., Han, F,, and Liu, H. (2014). Challenges of big data analysis. National science review, 1())2993-14.
Fan. and Li, R. (2006). Statistical challenges with high dimensionality: Feature selection in knowledge discovery. arXiv preprint math/0602133.
Fefferman, Mitter, S., and Narayanan, (2016). Testing the manifold hypothesis. Journal of the American
Mathematical Sociery, 29(4):883-100 1049.
Haber, A. L., Biton, M., Rogel, N., Herbst, R. H., Shekhar, K., Smillie, C., Burgin, G., Delorcy, T. M., Howitt, M. R., Katz, Y., et al. (2017). A single-cell survey of the small intestinal epithelium. Nature, 551(7680):333339)
Higuera, C., Gardiner, K. J., and Cios, K. J. (2015). organozggn feature maps identify proteins critical to learning in mouse model of down syndrome, PloS one, 1016)200

---

Hotelling, H. (1933). Analysis of a complex of statistical variables into principal components. Journal of educational psychology. 24(6):417.
Jirsa, V. K., Friedrich, R., Haken, H., and Kelso, J. S. (1994). A theoretical model of phase transitions in the human brain. Biological cybernetics, 71:27-35.
Johnstone, I. M. and Titterington, D. M. (2009). Statistical challenges of high-dimensional data.
Levina, E. and Bickel, (2004). Maximum likelihood estimation of intrinsic dimension. Advances in neural information processing systems, 17.
Li. D., Jones, A.. and Engelhardt, B. (2020). Probabilistic contrastive principal component analysis. arXiv preprint arXiv:2012.07977
Liu, Z., Luo, P., Wang, X., and Tang, X. (2013). Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vixion, pages 3730-3738.
McFarland, J. M.. Paolella. B. R.. Warren, A., Geiger-Schuller, K., Shibue. T., Rothberg, M., Kuksenko, O.. Colgan, W. N., Jones, A., Chambers, E., et al. (2020). Multiplexed single-cell transcriptional response profiling to define cancer vulnerabilities and therapeutic mechanism of action. Nature communications, 11(1):429..
McInnes, L.,, Healy, J., and Melville, J. (2018). Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1=Xivv/802.03422
Mimitou, E. P., Cheng, A., Montalbano, A., Hao, S., Stoeckius, M., Legut, M., Roush, T., Herrera, A.. Papalexi, E.,, Ouyang, Z., al. (2019). Multiplexed detection of proteins, transcriptomes, clonotypes and crispr perturbations in single cells. Nature methods, 16(5)=499442.
Novembre, J. and Stephens, M. (2008). Interpreting principal component analyses of spatial population genetic variation. Nature genetics, 40(5):66 649.
Pasini, G. (2017). Principal compenent analysis for stock portfolio management. International Journal of Pure and Applied Mathematics, 15()):153-167.
Pope, P., Zhu, C., Abdelkader. A., Goldblum, M., and Goldstein, T. (2021). The intrinsic dimension of images and its impact on learning, arXiv preprint arXiv:2104.08894,
Severson, K. A., Ghosh, S., and Ng, K. (2019). Unsupervised learning with contrastive latent variable models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 4862-4869.
Shorten, C. and Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep leamning Journal of big data, 6(1):1 1-8.
Tenenbaum, J. B., Silva, d., and Langford, J. C. (2000). global geometric framework for nonlinear dimensionality reduction. science, 290053000:2319-2233
Thudumu, S., Branch, Jin, J., and Singh, (2020). comprehensive survey anomaly detection techniques for high dimensional big data. Journal of Big Data, 7:1-30.
Tipping, M. E. and Bishop, C. M. (1999). Probabilistic principal component analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology, 6136)611-622.
Torgerson, W.S. S. (1952), Multidimensional scaling: I. theory and method. Psychometrika, 17(4):401-41,,
Van der Maaten, I., and Hinton, G. (2008). Visualizing data using t-sne. Journal of machine learning research, 9(11).
Vershynin, R. (2018). High-dimensions probability: An introduction with applications in data science, volume 47. Cambridge university press.
Weinberger. E., Lin, C.. and Lee, S.-L. (2023). Isolating salient variations of interest single-cell data with contrastivevi. Nature Methods,
Ye, and Lim, L.-.. (2016). Schubert varieties and distances between subspaces of different dimensions. SIAM Journal on Matrix Analysis and Applications, 37(3):117--1199 176-1197.
Zheng, G. X., Terry. J. M., Belgrader, P., Ryvkin, P., Bent, Z. W., Wilson, R., Ziraldo, S. B., Wheeler, T. D., McDermott, G. P., Zhu, J., et al. (2017). Massively parallel digital transcriptional proffiiin of single cells. Nature communications, 8414.
12

---

A Licensed Assets
1. BMMC data (C-BY-00
2. ECCITE-Seq data Open Data Commons Open Database License (ODbL) v1.0
3. mHealth data CC BY-4.0
4. Mouse Protein Expression data CC BY-4.0
5. Pathogen data Open Data Commons Open Database License (ODbL) v1.0
6. Perturb-seq data Open Data Commons Open Database License (ODbL) v1.0
7. Small-molecule data CC BY-4.0
8. MNIST data CC BY-SA 3.0 license
9. contrastive Github repository -BD 3-Clause License
10. contrastive Github repository -MIT License
11. pepea Github repository MIT License
12. python scikit dimension package BSD 3-Clause license
13. phthon anndata package BSD 3-Clause License
14. python scanpy package BSD 3-Clause License
15. python requests package Apache-2.0 license
16. python pillow package open source HPND License
Every contributor of new assets to this paper is author. The authors have all mutually agreed to use the CC BY-4.0 license, as described in the README file of the Github in appendix
B Data and availability
All code and access to data pre-processing, well the original data sources, available Github at the following link.
we report the approximate time and memory requirements for each experiment. All experiments were run on a Linux-based virtual computer with 6500 conventional compute cores delivering 13,000 threads. We used only 1 core.
In table 3
Table 3: Approximate Time and Memory Requirements per Experiment
Experiment
Time
Memory
Simulation
5-15 minutes
4 GB
Simulation 2
5-15 minutes
4 GB
Simulation 3
hour
4 GB
Simulation 4
hour
4 GB
Corrupted MNIST
hour
4 GB
Mouse Protein
5-15 minutes
4 GB
mHealth
1-5 minutes
4 GB
BMMC
hour
160 GB
Small Molecule
1 hour
4 GB
ECCITE--Sq
hours
GB
Pathogen Data
1 hour
4 GB
Perturb-Seq
2 hours
160 GB
CelebA: Glasses
6 hours
190 GB
CelebA: Hat
6 hours
190 GB
13

---

C Proofs
C.1 Proof to lemma
Lemma 1. day dim(Proiv, Vo) # 0.(V V₂)) > 0} + max(ds dyy 0).
Proof. Fix 22 Vu C IP, and let dim = dVm(V dy. Let dim N [22) =
Let U1..... Ug be an orthonormal basis for V n Vg. follows that u1..... Up E V2 n V/ are the first k principal vectors for both V₂ and Vy, corresponding to principal angles 81 ... = 02 = 0.
Furthermore, for every i = ,,...,,, we know that Projvy u, = because uy Vy.
Case 1: Assume dN I dy. In this case, there are dz h nonzero principal angles corresponding to principal vectors we denote by uk+1..... ud, E Va and Uk+1,...,, Ud, E Vy, For each i k- 1,...,d,, dz. know that Projy,- u, + O, because u, $ Vg. In fact, because the u, are orthogonal, Projy, us+1....., Projv,, dd, are a basis for Projy, Vg- Therefore, dim(Projy, Vg = # (i
Oc(V,,, Vi)) > 0}.
Case Assume dz > dy. this case, there are d, k nonzero principal angles. Similar to the above argument,
extend the basis for V2 starting with the principal vectors ug+1, .... Udy Va corresponding to the principal angles. However, because there are only dy principal angles, we denote by *d,+1..... Udz the remaining vectors necessary to specify orthogonal basis for V., Because the principal angle maximizes u, Vú, the remaining vectors udd +1y-... Hdd are all orthogonal to Vy- Therefore, Projy, uk+1..... Projv, Ud, Udy .1..... ud. is basis for Projy, V.. It follows that dim(Proiv, V.) # (i: (V Vúy) 0} + dn dy, desired.
C.2 Proof to theorem
Theorem 1. Assume that the second moments E, and Ey are finite for both groups, and that the top dz + eigenvalues of E.z and top dy + eigenvalues of F.g are distinct. Then, our proposed estimator dzy is consistent: day 12-00 day.
Proof. We know by Anderson 1963) that the sample matrix of eigenvectors UN converges to the true matrix of eigenvectors U, in probability as n, + co, and similarly, Ü, - U, ny + co. Let m min (d,, dy); and consider the following functions.
f1 Roxd= Rpxdy dad
(Un, Uy) + U
fa Rdd xd, - Rmm
+ (A1,..., Am) where Aj is the jth singular value of
It is clear that is continuous. Due Weyl's inequality for singular values, f2 is continuous with respect to the spectral norm. The contrastive dimension is defined to be d = #(,, < 1}. while its estimator is defined by day # < - e]. Therefore, by the continuous mapping theorem, it holds that X + probability. Taking e - gives Jay > days as desired.
C.3 Proof to theorem
Theorem 2. In addition to all assumptions in theorem 7 we assume x and y in model 7 Sub-Gaussian More precisely, assume that there exist constants K1, K2 2 such that
(x 1)ll I K,E u) and 11 1l I K2L (M, u) for any v e g
14

---

where
1,1,, is defined as in Vershynin 2018). Then, for any >> O. with probability at least - 22 "0
we have
Sc SC
E kE(xNt
da Kb
(vPt*
p+u
|_ = -O(n + f 11/1/22
max
71
where C' is an absolute, positive constant, ò is the minimum eigengap among the top d+ eigenvalues of E, and |I-Il denotes spectral norm.
Proof. Let W = V where U is the matrix of the top dı cigenvectors of Σ, and V is the matrix of the top dy eigenvectors of Ey. We use hats to represent sample estimates of these matrices and let W = V. We would like a samite-sample error upper bound on maxj=l.....m o
o, (W)
max m
oj
W
oy (W)
W w| by Weyl's inequality for v-values
| - V. 8 - -
1TV - 0 -
-
|1|| - 0/| |||||
V - ||| + 0 -
v + 1 U
Enn - En
En, - En
< 23/2
Vd
V dy
82
8y
The final inequality holds due to the Davis-Kahar theorem, presented in Vershynin (2018). Note that here dg is the smallest eigengap of Ex, and õ, is the smallest eigengap of >ig
For & E {a, v), with probability at least 1 - 2e-", we have that
Eg S CCK2
(v++t u
p + u
|Ex
n&
where C is absolute, positive constant, and K2 || constants, as in Vershynin 2018). Substituting these upper bounds on Eyz - Σn yield the bound claimed.
Ky
| the sub Sau-Gaussian
D Additional experimental details
The code for all experiments made available at the anonymous Github link provided appendix B To allow for reproducibility, test. Sometimes we set the seed to be 1; other times we set the seed to be 42. There was no particular reason behind the choice for any seed. In every experiment requiring intrinsic dimension estimation, we used the method of moments estimator from the sci-kit dimension package. Other estimators are possible and may give slightly different results, but chose this estimator because found it to give the most stable and reasonable results. our experience, we found some of the other estimators give estimates as low 0 and high the number of features, of which neither estimate is sensible in the contexts we considered.
set seed for every experiment in which perform the bootstrap
D.1 Simulations
For simulations 1 and 2, S,r and Sy were randomly chosen to be orthogonal matrices in Rxx and igd,,, respectively. Therefore, the covariance matrices in this simulation are given by Σx
15

---

S + I and Ey = SyS, + I respectively. Notably, these covariance matrices violate the assumption of distinct eigenvalues presented in section However, the space spanned by the top d, eigenvectors of Ex is still identifiable, because the d, eigenvalue is 1 + og while the (dz + 1)th eigenvalue o² (similarly for Ey)) As discussed in section 7 this is the necessary identifiability condition.
We designed simulations 3 and to allow for an interpretable ground truth of day 1 while mimicking real image data. Figure Srgure@illusrraaa illustrate the data-generating process for both groups in both simulations. Every image represented by 28 by 28 matrix in the grayscale space, where 0 represents black, and 255 represents white. Each of these image matrices was converted to a vector in R 784, In both simulations, the foreground group was created by drawing a white line on top; each pixel in this line was set to 255, while the background group was created by averaging the original (disk or MNIST 0) image entrywise with an image of grass from the corrupted MNIST dataset. The was no specific intent behind choosing to draw a horizontal line in the sixth row of each image; drawing line of random length going any single direction from any starting pixel could have worked. However, we chose to fix the starting pixel because we wanted to be sure not to introduce a second parameter. That said, simulations which other parameters are introduced, leaving a different ground truth day are possible. For example, creating the foreground group by drawing white rectangle of random width and height (w, h) E {1....,24] 22.) beginning at the top-left pixel would lead to a scenario in which day = 2.
D.2 Real data experiments
The Mouse Protein, mHealth, BMMC, Small Molecule, ECCITE-Seq, Pathogen, Perturb-Seq. and CelebA experiments all required additional pre-processing. To ensure that our pre-processing aligned with previous analyses, we used the code available for pre-processing these datasets associated with the papers analyzing them. To be specific, we pre- pre-processed the Mouse Protein and mHealth datasets using the code associated with /Abid et al. (2018). We pre-processed the Small Molecule, ECCITE-Seq, Pathogen, and Perturb-Seq datasets using the code associated with Weinberger et al. (2023). We downloaded the BMMC dataset using instructions and code from Weinberger al. (2023) and pre-processed it using the code from Li et al. (2020). Code files for replicating the pre-proces pre-processing we did for each experiment
available on the anonymous Github in appendix B To pre-process the CelebA dataset, we first used the PIL. package in Python to resize the images from 178 by 218 pixels to 64 by 64 pixels. From there, performed PCA (Anderson 1963) on the resized images and took the top 1000 directions.
In section 6 we made reference to results standing in agreement with results from previous analyses. In particular, we claimed that for the BMMC dataset, our observation of y.y > was consistent with the analysis in Li et al. (2020), in which a figure was published illustrating a distinction between the pre- and posttransplant groups in the BMMC data using the third and fourth PCPCA directions. We duplicated that graph in the left panel of figure 3 for transparency. We also claimed that for the mH-ealth dataset, our observation of day 1 was consistent with the analysis in Abid ct al. (2018), in which a figure was published illustrating a single direction differentiating the two groups of squatting and eycling. We duplicated this illustration in the right panel of figure here for convenience.
For clarity on the image datasets used in section 6 we provide sample images from the foreground and background groups of both the Corrupted MNIST and CelebA datasets. The images for the Corrupted MNIST application are in figure the top row has examples of images in the foreground group, while the bottom row has examples of images in the background group. The images for the CelebA application are in figure the top-left two are foreground (glasses), while the bottom-left two are background (glasses). Similarly, the top-right two are foreground (hat), while the bottom-right two are background (hat).
D.3 Practical Implications for Downstream Tasks
To explore how the methods we presented in this paper can be used effectively in downstream data analysis tasks, we conducted further analysis with the corrupted MNIST dataset used in section 6 We performed CPCA with dimensions d 1,2,3, ... 10, then ran logistic regression with the
16

---

0.9
cPCA (unstandardized)
200
Squattin
Pretransplant Posttransplant
Cycling
150
100
PCPC4
cPC2
50
0-
-50-
-3100
-20
100
0
100
100
PCPC3
cPC
Figure 3: Left, PCPCA for BMMC Data. Right, cPCA for mHealth Data.
Figure 4: Corrupted MNIST Sample Images. Top row, foreground. Bottom row, back
outcome (response) variable being the digit (0 vs. 1) imposed on the grass images. The input was the contrastive principal components (cPCs) based on CPCA with different d. The results are as follows:
Table 4: Summary of simulation results
Dimension
1
2
3
5
6
7
10
Accuracy 0.583
0.941
0.941
0.943
0.945
0.947
0.946
0.947 0.947
0.947
The accuracy increases with the number of cPCs and then plateaus 5, where the accuracy is 0.945,
after which the accuracy increases only slightly. This provides indirect evidence that if one can simply contrastive dimension reduction methods with the dimension equal in this case). One may argue, however, that = 2 is optimal, resembling an elbow point. Our principal angles (measured by singular values) align with this observation. As shown in Table 2 the smallest four singular values are 0.095, 0.315, 0.705, and 0.846. The two smallest singular values suggest that there are more prominent contrastive dimensions, leading to satisfactory classification performance for d 2. While our suggested cutoff 0.9 gives dzy 5, cutoff 0.7 gives dry, = 2. This cutoff is subjective and can be used in conjunction with classification accuracy, particularly if response variable is available, improve decision making.
estimator (5
17

---

UN
Figure 5: CelebA Sample Images. Top-left, foreground (glasses), Top-right, foreground (hat).
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately describe the contribution our methods. We clearly identify the scope of the paper in the abstract and introduction and address the concerns in the rest of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In section we detail the limitations of the hypothesis test we present, and we suggest an alternative for more powerful test. We also discuss the need for further investigation into the robustness of CDE to misspecification of intrinsic dimension estimates.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
18

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We have two theoretical claims about our estimator, and we state the assump- tions and provide complete and correct proofs for them in appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all code and information necessary to access and pre-process the data in the anonymous Github repository at the link in appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same
19

---

dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g.. to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All code and instructions to obtain and pre-process datasets are in the anony- mous Github repository, for which we provide the link in appendix B
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/ public/guides:
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nniss /mipsscc/ppllie http:ccccccgbccueeeeeeeeeeeeeeee www.ddee/eoooobbbiiaceeee wwwpssbmmsssoonPoooooo ssioooicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
20

---

Justification: We specify all parameters for replicating our experiments, including choice of intrinsic dimension estimator, number of bootstrap samples, cutoff (tolerance), and variance.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the p-value result from the hypothesis test developed for each simulation and experiment.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, in appendix B we include approximate time and memory required to replicate simulations and experiments, including a description of the machine on which the experiments were run.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
21

---

9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:://hoorrss.ccooolll////aaap
Answer: [Yes]
We have reviewed the NeurIPS Code of Ethics and conform to in every re- spect, including, but limited to, data-related concerns, societal impact, impact mitigation, model licensing, and reproducibility.
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our methods mostly concern the applicability of other dimension reduction methods designed to analyze case-control data. While is possible that our method might have societal impact, we choose focus more on method development than downstream effects difficult to observe at this stage in the work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our method simply provides p-value and an estimate for parameter to be used in other dimension reduction methods. Because the output consists of merely handful numbers to be used as a guideline for future analysis, poses extremely low risk for misuse.
22

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:
[Yes]
Justification: We properly cite and credit all licenses to all code, data, and packages in appendix A
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pww.::s...aaseiiiia...... thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
All of our code is available the anonymous Github.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
23

---

Answer: [NA]
Justification:
Our research does not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our research does not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "gYheKo8dsu", "reviewer": "Reviewer_e5iP", "summary": "The authors address two key challenges using two datasets, categorized as foreground and background: 1) Determine whether contrastive dimensionality methods are appropriate for application to such a pair of datasets and 2) Quantify the unique information present in the foreground data.", "questions": "Please see above", "limitations": "Please see weakness section", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The text is mostly clear including two theorems and two algorithms.\n- The background section is well-written, offering a comprehensive overview of relevant literature and methodologies", "weaknesses": "--Hypothesis Test:\nThe assumptions, such as $V_x \\subset V_y$ under $H_0$, lack proper justification.\nThe hypothesis test relies on estimating the intrinsic dimension, which is inherently challenging. The authors employ other methods for these estimates but fail to compare the accuracy of their estimates to these methods. These methods could also be used to test $d_{xy} = 0$.\nThe overall presentation of the hypothesis test is inadequate. The rationale for choosing this specific test and how calculating the singular values of the custom $\\hat{V}_y$ improves the $p$ estimate remains unclear. The authors do not justify, explain, discuss, or comment on their choices.\n--Experimental Evaluation:\nThe method lacks comparison to other approaches: For intrinsic dimension estimation, there's no clear way to compare against other methods due to the absence of ground truth. Alternative evaluations should be designed, such as training GANs to a priori upper-bound the intrinsic dimension of generated data by the dimension of the latent noise vector, as done in [1].\nThe authors don't compare their work to other \"contrastive dimensionality reduction\" methods. They could compare (i) expressed variance in the foreground group among different methods, (ii) expressed variance in the background group, and (iii) expressed variance common across groups. Alternatively, they could propose other ways to quantify performance and compare their method to existing literature.\nThe intrinsic dimension estimates are compared to those of (Pope et al., 2021), which aren't ground truth values. This makes it difficult to compare the inferred dimension to the actual one.\nThe practical implications of the algorithms aren't tested. How can these methods use dimensionality reduction to identify foreground sets? How can they be applied to related downstream classification tasks? How can they be used to compare different experiments?\nDespite being motivated by \"large scale data,\" the experiments are limited to small dimensions (e.g., MNIST), with no consideration of scalability crucial in today's datasets.\nThere's no exploration of other datasets or discussion on how intrinsic dimensions were estimated.\n--Terminology: The terminology is often unclear and potentially misleading. For example, \"unique information in the foreground group\" refers to low-rank components, and terms like \"contrastive information\" and \"contrastive dimension\" could be misinterpreted in the context of self-supervised learning. Alternative terms like \"complementary information\" may be more appropriate. Even the title is misleading as \"contrastive dimension reduction\" is closely related to contrastive learning, a self-supervised dimensionality reduction method.\nTitle: The title doesn't accurately reflect the paper's content and focus.\n--Missing Conclusion: The paper lacks a conclusion section.\nLanguage Issues: The manuscript contains several typos and linguistic errors, including repeated words in the abstract (e.g., \"with with\").\n[1] Pope, P., Zhu, C., Abdelkader, A., Goldblum, M., & Goldstein, T. The Intrinsic Dimension of Images and Its Impact on Learning. In International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Hypothesis Test**\n\nWe appreciate your feedback and the opportunity to clarify our approach.\n\nFirstly, based on Definition 1 and the notation we introduce following Model 1, the statements $d_{xy} = 0$ and $V_x \\subset V_y$ are indeed equivalent. We have clarified this at the end of Section 2.\n\nWe chose the method of moments estimator from the sci-kit dimension package for estimating intrinsic dimension due to its stability and reasonable results, as mentioned in Appendix D. Other estimators we considered are listed in the following table. These methods occasionally produced extreme estimates (as low as 0 or as high as $p$), which were not meaningful in our contexts.\n\n| Method | Correlation Dimension    | Dimensionality from Angle and Norm Concentration    | Expected Simplex Skewness    | Fisher Separability    | kNN (k=10)    | PCA    | Manifold-Adaptive Dimension Estimation     | Maximum Likelihood    | TwoNN   |\n|---------------|------|------|------|------|------|------|------|------|------|\n| Reference      | Campadelli, et al. (2015) | Ceruti, et al. (2012) | Johnsson, et al. (2014) | Albergante, et al. (2019) | Rozza, A., et al. (2012) | Fan, et al. (2010) | Farahmand, et al. (2007) | Levina and Bickel (2004) |  Facco, et al. (2017) |\n\nWe acknowledge that estimating intrinsic dimension is inherently challenging, and this impacts the hypothesis test. Our heuristic explanation is as follows: under $H_0$, $V_x \\subset V_y$. By resampling background data $y$ from both the foreground and background (while resampling $x$ only from the foreground), we create a resampled dataset where $V_x \\subset V_y$ holds, regardless of its validity in the original data. Comparing the original data to the resampled data allows us to detect if $V_x \\not\\subset V_y$.\n\nWe recognize the limitations of our hypothesis test. As discussed in section 7, a likelihood-based test may be possible, but the required optimization is challenging and slow. We presented our hypothesis test as it avoids these optimization challenges and is computationally efficient.\n\n\n**Experimental Evaluation**\n\n*Comparison with other methods*\n\nWe did not compare our method to other estimates of contrastive dimension because, to our knowledge, there are no existing methods in the literature specifically for contrastive dimension estimation. We have clarified this in Section 5 and 6 in the revised manuscript.\n\n*Intrinsic dimension estimation of a single group*\n\nWe acknowledge that intrinsic dimension estimation for a single dataset is an interesting problem, but it is not the focus of our paper. Our method is compatible with various intrinsic dimension estimators. For instance, using GANs to upper-bound the intrinsic dimension, as suggested, is a viable approach. Once the estimates of $d_x,d_y$ are obtained, our method can be used to find $\\hat{d}_{xy}$. This flexibility allows the incorporation of different intrinsic dimension estimators, including GANs, to complement our approach. We have clarified this After Algorithm 2.\n\n*Use of contrastive DR methods*\n\nOur purpose is not to design a new contrastive DR method. Instead, we provide suggestions on whether contrastive DR methods should be used and the appropriate hyperparameter for the reduced dimension. \n\n*Practical implications and downstream tasks*\n\nOur method aids in selecting the reduced dimension, which can then be used by contrastive DR methods for various applications. \n\n*Large scale data considerations*\n\nThe ten real datasets we considered have all been used in previous literature to demonstrate the usefulness of contrastive DR. While Table 2 might suggest dimensions of at most 1000, this is due to pre-processing steps like selecting the most variable genes, consistent with prior analyses of these datasets, as described in Appendix D.2. All intrinsic dimension estimates were made using the method of moments estimator from the sci-kit dimension package because it provided the most stable and reasonable results, as discussed in Appendix D.\n\n**Terminology and Title**\n\nThank you for the helpful feedback. Our aim for the title and terminology was to reflect our focus on contrastive dimension reduction methods, such as CPCA and CLVM, and to study when it is appropriate to use them and how to improve the selection of the reduced dimension parameter.\n\nWe acknowledge that terms like \"contrastive information\" and \"contrastive dimension\" might be misinterpreted within the context of self-supervised learning, where \"contrastive learning\" is a widely recognized concept. However, our work is rooted in the specific field of contrastive dimension reduction, which, while sharing some terminology with self-supervised learning, addresses different research questions and methodologies.\n\nGiven that our work is inspired by methods like CPCA and CLVM, we have chosen terminology consistent with the established literature in this area. We believe this is the most straightforward way to communicate our contributions to researchers familiar with these methods. \n\n\n**Missing Conclusion**\n\nThank you for your feedback. We understand the importance of having a clear conclusion section. In our current manuscript, Section 7, titled \"Discussion,\" serves this purpose. In this section, we first summarize the contributions of our paper and then discuss possible future work. However, to make it clearer and to address your concern, we have changed the title of Section 7 to \"Conclusion and Future Work\" in the revised manuscript.\n\n**Typos**\n\nThanks for catching the typos, we have fixed them in the revision.\n\nThank you once again for your valuable feedback. We are happy to provide further clarification or discuss any additional questions you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8djmLwefgp", "reviewer": "Reviewer_xiqq", "summary": "The paper proposes to examine the existence and estimate the number of contrastive dimensions between the foreground and background groups. The authors provide a formal definition of contrastive dimensions and propose a hypothesis test method to examine the existence of contrastive dimensions. The authors also propose a contrastive dimension estimator to estimate the number of contrastive dimensions, and provide theoretical analysis on the consistency and finite sample error bound of the proposed estimator. Experiments are conducted on synthetic semi-synthetic and real-world datasets to evaluate the performance of the proposed hypothesis test and contrastive dimension\nestimator.", "questions": "1. Could you explain in more detail the practicality of the formulation? Can this formulation address the example given in the weaknesses section?\n2. Could you conduct additional experiments to assess the effectiveness of the proposed methods in selecting hyperparameters for downstream contrastive dimension reduction tasks?\n3. Could you justify the necessity of a hypothesis test when simply detecting whether $\\hat{d}_{xy}=0$ seems to be effective?", "limitations": "Yes", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation is novel, and the proposed contrastive dimension estimator is theoretically sound.\n2. The paper is well-written, and the experimental codes are made available, which make the paper easy to follow and reproduce.", "weaknesses": "1. The formulation of contrastive dimension seems impractical. For instance, consider the foreground group $X\\in\\mathbb{R}^3$ drawn from $\\mathcal{N}(\\mathbf{0}, diag(2, 1, \\sigma_\\varepsilon))$ and the background group $Y\\in\\mathbb{R}^3$ drawn from $\\mathcal{N}(\\mathbf{0}, diag(1, 1, \\sigma_\\varepsilon))$, where $\\sigma_\\varepsilon$ denotes the standard deviation of the noise terms. Both the foreground $Z_i$ and background $W_j$ occupy the same subspace, specifically the column space \n$\\mathcal{C}\\left(\\left[ {\\begin{array}{ccc} 1&0&0 \\\\\\\\ 0&1&0\\\\\\\\ \\end{array} } \\right]^\\top\\right)$\n. The contrastive dimension is expected to be $dim\\left(\\mathcal{C}\\left(\\left[1\\quad 0\\quad 0\\right]^\\top\\right)\\right)$, yet according to the definition 1, the contrastive dimension is $dim(V_{xy})=0$ .\n2. In section 2, the paper claims that a significant application of the proposed methods is to determine the number of dimensions for contrastive dimension reduction. However, the experiments provide limited discussion on the effectiveness of these methods in selecting hyperparameters for downstream contrastive dimension reduction tasks. This lack of detail undermines the practical utility of the proposed\napproaches.\n3. The paper mentions that the hypothesis test is conservative, and the experimental results indicate that the hypothesis test does not offer significant advantages over simply detecting whether $\\hat{d}_{xy}=0$. However, the paper does not justify the need for a hypothesis test when the contrastive dimension estimator appears to be effective on its own.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The practicality of the formulation of contrastive dimension**\n\nThank you for the opportunity to clarify. This example highlights the differences between our approach and methods like CPCA and CLVM.\n\nIn your example, from the CLVM perspective, $\\Sigma_x = SS^\\top + WW^\\top + \\sigma_{\\epsilon} \\mathrm{I}$ and $\\Sigma_y = SS^\\top + \\sigma_{\\epsilon}\\mathrm{I}$, where $S =  [1,0,0;0,1,0]^\\top$ and $W =[1,0,0]^\\top$. In this situation, both groups lie in the same 2-dimensional subspace spanned by $[1,0,0]$ and $[0,1,0]$. So, there is no low-dimensional subspace unique to the foreground; the foreground data simply has a larger variance in the direction of $[1,0,0]$ but the background also contains information in this direction. Methods like CPCA and CLVM would identify the span of $[1,0,0]$ as the space unique to the foreground, which is not valid in this case. Without further assumptions on the relationship between $S$ and $W$ in CLVM, finding a nonzero $W$ does not necessarily mean $W$ captures the linear subspace unique to the foreground—if $W = S$, it is not truly unique.\n\nIn contrast, our definition gives a zero contrastive dimension in this example, which seems more reasonable. This shows our approach quantifies unique information geometrically, different from CPCA and CLVM. We are not claiming our method is more correct; rather, it is a different way to quantify unique information that might be more meaningful in certain applications. This geometric perspective may inspire new contrastive dimension reduction methods, an interesting direction for future work. We discuss this in the revised manuscript.\n\n**The effectiveness of the proposed methods in selecting hyperparameters for downstream tasks.**\n\nMeasuring the effectiveness of selecting the reduced dimension hyperparameter $d$ for downstream contrastive dimension reduction tasks is indeed challenging. Contrastive dimension reduction often serves as a pre-processing step for further data analysis tasks, such as classification, clustering and visualization. \n\nOur proposed method is the first in the literature to define and estimate the contrastive dimension of a given dataset. Therefore, measuring the effectiveness of selecting the reduced dimension hyperparameter for downstream tasks is tricky. In the examples provided in the paper, we demonstrated how our methods can determine (i) whether using contrastive dimension reduction methods is appropriate, and (ii) a suggestion for the reduced (or contrastive) dimension hyperparameter in these methods.\n\nTo address your request for additional experiments, we conducted further analysis using the corrupted MNIST dataset used in Section 6 in our manuscript. Here, the estimated contrastive dimension is 5. We performed CPCA with dimensions $d = 1, 2, 3, \\ldots, 10$, and then ran logistic regression with the outcome (response) variable being the digit (0 vs. 1) imposed on the grass images. The input was the contrastive principal components (cPCs) based on CPCA with different $d$. The results are as follows:\n\n| Dimension (d) | 1    | 2    | 3    | 4    | 5    | 6    | 7    | 8    | 9    | 10   |\n|---------------|------|------|------|------|------|------|------|------|------|------|\n| Accuracy      | 0.58 | 0.94 | 0.94 | 0.94 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 |\n\nWe observed that the accuracy increases with the number of cPCs and then plateaus at 5, where the accuracy is 0.95. This provides indirect evidence that if one trusts our estimator, one can simply run contrastive dimension reduction methods with the dimension equal to our estimator (5 in this case).\n\nNote that one may argue $d=2$ is optimal, resembling an elbow point. Although there is no objective way to decide, our principal angles (measured by singular values) align with this observation. As shown in Table 2 of our manuscript, the smallest four singular values are 0.095, 0.315, 0.705, and 0.846. The two smallest values, 0.095 and 0.315, suggest two more prominent contrastive dimensions, leading to satisfactory classification performance for $d=2$. If we use a cutoff of 0.7, our estimator would be $\\hat{d}_{xy}=2$ as well. However, if we choose $d=5$, it aligns with our previously suggested cutoff of 0.9. The cutoff is subjective and can be used in conjunction with classification accuracy, particularly if a response variable is available, to improve decision making.\n\nAdditionally, it's important to note that in the broader literature on intrinsic dimension estimation (not specifically contrastive), there is no gold standard for validating the estimated dimension. Our work similarly provides an estimator and suggests a practical approach for its application, but the validation of such estimators in an unsupervised context remains a challenging and ongoing area of research.\n\n\n**The necessity of a hypothesis test and the relationship between the test and the contrastive dimension estimator**\n\nThe decision to use the hypothesis test is user-dependent. If you want to assess uncertainty, the hypothesis test provides a direct measure of whether it is worth moving forward with applying a contrastive dimension reduction method. If the test rejects the null hypothesis, it indicates that there is unique information, and estimating the contrastive dimension can guide the selection of the number of low-dimensional representations for downstream tasks.\n\nOn the other hand, if users are satisfied with a point estimate of the contrastive dimension, they can rely solely on the estimator, which we have shown to be consistent and have a known error bound. We are not claiming that one must use both approaches; it depends on the user's needs and the specific application. We have clarified this in the revised manuscript, section 3.2.\n\nThank you once again for your valuable feedback. We are happy to provide further clarification or discuss any additional questions you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mrVZh94d1e", "reviewer": "Reviewer_Casz", "summary": "The paper deals with a contrastive dimension reduction (CDR) techniques which seeks to find a unique low-dimensional pattern that only exists in the foreground data compared to background data. Authors point out that despite of recent developments of related techniques such as contrastive principal component reduction (CPCA), there lacks a rigorous guideline to determine when and how those techniques should be utilized. Hence, they propose a hypothesis testing method to determine whether the foreground data actually have unique information which background data do not and an estimator of the number of CPCA PCs. The authors then investigate theoretical aspect of the estimator including the consistency and finite sample error bound. They also conduct simulation study and real data analysis to demonstrate the performance of the proposed method.", "questions": "1. As one of the possible reasons for the discrepancy mentioned as the 1st weakness, the authors conjecture that it might be due to the nonlinearity of the underlying unique pattern. However, if there actually exists nonlinear pattern, why does the estimator succeed to capture it while hypothesis testing fails? This point needs to be elaborated.\n\n2. From the formulation, it seems that the number of samples $n_x$ and $n_y$ should always be larger than $d_x$ and $d_y$ respectively to correctly estimate the dimension. Can you elaborate on the relationship between the sample size and intrinsic dimension with possible implications to the method?\n\n3. Under the CLVM model which has a close relationship with CPCA, background data only has shared information implying that $d_x$ > $d_y$. However, the manuscript also concerns a case where $d_x$ < $d_y$ but $d_{xy}$ > 0 assuming the model (1). Can this be understood as suggesting a different viewpoint from CLVM to understand the CDR?", "limitations": "The authors well addressed the potential limitations and future research goals of the study. They included the possibility of introducing likelihood ratio testing method to overcome the weakness of the permutation test pointed out as the 3rd weakness of the manuscript.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Contribution of introducing hypothesis testing and estimator of number of PCs to CDR literature.\n\n2. Mathematically rigorous formulation to capture the amount of information which only exists in the foreground data.\n\n3. Theoretical proof of the consistency and error bound of the estimator.\n\n4. Extensive real data analysis with interpretation.\n\n5. Well organized and well written manuscript.", "weaknesses": "1. Discrepancy between the p-value obtained from permutation test and the estimator happens in some cases. The authors might need to conduct additional simulation study to investigate the reason behind this discrepancy.\n\n2. The simulation study seems to be too simple without repetitive iterations to check the stability of the proposed method. More extensive simulation is recommended. Authors need to at least provide the average of the repeatedly estimated value.\n\n\n3. The proposed hypothesis testing relies on the permutation method which makes the result unstable and further theoretical investigation difficult.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Discrepancy between the p-value obtained from permutation test and the estimator happens in some cases.**\n\nThank you for the feedback. We would like to clarify that the hypothesis test is conservative, and therefore it is possible to observe a high $p$-value even when the estimator indicates that $d_{xy} > 0$. This conservativeness is designed to reduce the likelihood of false positives, but it can also mean that some signals are not detected by the hypothesis test. We have clarified this in the revised manuscript after proposing the hypothesis test.\n\n**Nonlinear as one of the possible reasons for the discrepancy**\n\nThank you for the question. The hypothesis test is conservative, and its reliance on resampling can lead to discrepancies when detecting nonlinear patterns. While the dimension estimator and the hypothesis test both use linear methods, they are applied differently. The estimator directly measures the dimension, which can sometimes capture variance explained by nonlinear patterns as part of the overall dimensionality. In contrast, the hypothesis test compares distributions under the assumption that $V_x \\subset V_y$.\n\nTo elaborate, under $H_0$ it holds that $V_x \\subset V_y$. A hypothesis test that resamples the background data $y$ from both the foreground and the background (while resampling $x$ only from the foreground) creates a resampled dataset in which $V_x \\subset V_y$ regardless of whether this assumption held true in the original data. Therefore, if $V_x \\not\\subset V_y$ in the original data, comparing the original data to the resampled data allows us to detect the desired signal.\n\nHowever, the hypothesis test’s resampling technique might not handle nonlinear structures in the data effectively (e.g., if the data lie on a nonlinear manifold). This can result in the test failing to detect a signal even when the estimator captures it, as the nonlinear structure can obscure the differences the test is designed to detect. We have added a remark to discuss such a discrepancy in the revised manuscript.\n\n\n\n**Additional simulations**\n\nThank you for the feedback. We have repeated simulations 1 and 2 both 100 times and report the mean (standard deviation) of the results in the following tables, and updated our tables in the simulation section of the manuscript. \n\n| Setup  | $d_{xy}$ | $\\hat{d}_{xy}$ | p-value | 4 smallest singular values |\n|--------|----------|----------------|---------|-----------------------------|\n| Sim. 1 | 0        | 0.44 (0.52)    | 0.81 (0.15) | 0.90 (0.02), 0.94 (0.01), 0.96 (0.01), 0.97 (0.01) |\n| Sim. 2 | 6        | 6 (0)          | 0.03 (0.03) | 0.06 (0.02), 0.11 (0.03), 0.16 (0.03), 0.22 (0.03) |\n\nWith known ground truth, the estimator $\\hat{d}_{xy}$ works perfectly for Simulation 2, while it makes some errors for Simulation 1. As we mentioned in the paper, these results indicate that a cutoff of 0.9 might be too close to 1, and a lower cutoff may be more appropriate. However, the optimal choice of cutoff could depend on the variance magnitude and the sample size.\n\nIn these simulations, we used a sample size of $n_1 = n_2 = 100$ and a variance of $\\sigma_x^2 = \\sigma_y^2 = 0.25$. Larger sample sizes might be necessary to detect smaller angles between the subspaces $V_x$ and $V_y$. Future work could explore how varying these parameters affects the performance of the estimator and the hypothesis test, potentially leading to more precise guidelines for selecting cutoffs in different scenarios.\n\n\n**The permutation method involved in the hypothesis test**\n\nThank you for the comments. We agree that the result from this hypothesis test can be unstable, which is why we repeat the resampling procedure 1000 times for every experiment, as discussed in Section 3.1. As mentioned in the discussion section, a likelihood-based test may be possible; however, the optimization required for it is challenging and slow. We presented the hypothesis test because it avoids the need for challenging optimization and is computationally efficient. Nevertheless, we believe that exploring a likelihood-based approach or other hypothesis tests is an interesting direction for future work.\n\n\n**The assumption of $n_x>d_x$ and $n_y>d_y$**\n\nThank you for this question. It is correct that $n_x > d_x$ and $n_y > d_y$ are required to correctly estimate $d_{xy}$. This assumption is very reasonable, as many existing studies observed a small intrinsic dimension. For instance, Pope et.al., 2021 studied high-dimensional image data where the intrinsic dimensions typically range from 10 to 50, which is not excessively high. We have clarified this point in the revised manuscript.\n\n**Relationship between\n $d_x$ and $d_y$. Can this be understood as suggesting a different viewpoint from CLVM to understand the CDR?**\n\nThank you for this question. Yes, our notion of CDR can be understood from a viewpoint that differs from CLVM. CLVM assumes that $d_x \\geq d_y$, ensuring unique information in the foreground group. One of our major motivations is to avoid making any assumptions about the relationship between $d_x$ and $d_y$. Note that $d_x > d_y$ implies $d_{xy} > 0$. However, even when $d_x \\leq d_y$, unique information in the foreground group may still exist, which CLVM would overlook. In contrast, our method can detect such information based on our definition. We have clarified this distinction in the revision as a remark. \n\nThank you once again for your valuable feedback. We are happy to provide further clarification or discuss any additional questions you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uAI9bfvUyo", "reviewer": "Reviewer_Bh4k", "summary": "The authors consider the problem of contrastive dimension reduction/estimation. In a nutshell, this problem involves a background dataset and a foreground dataset, and it is of interest to determine if the foreground dataset contains submanifolds/subspaces that are inherently different than those in the background. While there's past work on determining the differences of the foreground from the background, this paper asks if such a difference exists in the first place. This is posed as a hypothesis testing problem. An intuitively reasonable test is proposed and it's shown that the test is consistent. The authors also derive finite sample bounds on the error, resorting to commonly used sub-Gaussianity assumptions. They demonstrate the utility of the test through synthetic as well as real data.", "questions": "- line 81 : Do you assume a relation between $n_x$ and $n_y$, like, for instance $n_y \\gg n_x$?\n- line 83 : Do we center $X$, $Y$ separately? Or, in practice, do we assume $X$ and $Y$ to have the same centers?\n- line 131 : Should $Z_i$ and $W_i$ be lower case? If not, why not?\n- line 312 : Please state any assumptions on the dimension of $V_x$, $V_y$. If no assumptions are made, please note that.\n- Defn 1 : Even a minute departure of $V_x$ from $V_y$ would result in a non-zero $d_{xy}$. But you deal with that in the sequel. It may be worth noting that.\n- Defn 2 : This doesn't uniquely define $u_i$'s and $v_j$'s does it? But are $\\theta_i$'s uniquely defined?\n- line 174 : Again, if there are any assumptions on the ratio $n_x : n_y$ would be good to note here. \n- eqn 4 : Does the expression $\\text{eig}_{1:d_x}$ mean the span of the first $d_x$ eigenvectors? Please define that somewhere in the text.\n- Thm 2 : It would be good to include the definition of $\\| \\cdot \\|_{\\psi_2}$, or a reference to the section where it's defined in Vershynin's book.", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The manuscript identifies a new problem in an otherwise known sub-area of dimension reduction. A solution is proposed and it is justified both asymptotically and for finite sample scenarios, using available, and accessible mathematical tools. The paper is well written, motivates the problem well, and demonstrates the solution nicely on synthetic/real world examples. I liked reading the paper and I think it would be useful for future readers.", "weaknesses": "I think the manuscript has minor weaknesses. The most important one of those is that the proposed test itself is not very surprising, but I don't know if that's really a weakness -- after all it's backed up theoretically, and simple is good.\nOther than that, I list a few questions below, which should be easy to address by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough and insightful feedback on our manuscript. We appreciate the opportunity to clarify and improve our work. Below, we have summarized your questions and our responses:\n\n**Line 81 - Assumption on $n_x$ and $n_y$ and Line 174 - Ratio $n_x : n_y$**\n\nThank you for the question and opportunity to clarify. We do not assume any specific relationship between $n_x$ and $n_y$. We have clarified this in the revised version right after we introduce the notation of $n_x$ and $n_y$. \n\n**Line 83 - Centering of $X$ and $Y$**\n\nThank you for your question. We center $X$ and $Y$ separately in our analysis. We have clarified this in the manuscript.\n\n**Line 131 - Typographical Error**\n\nYou are correct; $Z_i$ and $W_i$ should be in lowercase. We have corrected this typographical error in the revision.\n\n**Line 312 - Assumptions on Dimensions of $V_x$ and $V_y$**\n\n\nThank you for the feedback and the opportunity to clarify. For the purpose of the contrastive dimension estimator, we do not make any assumptions on the dimension of $V_x$ and $V_y$. However, because $d_x > d_y$ implies that $d_{xy} > 0$, for the purpose of the hypothesis test, we do make the assumption that $d_x \\leq d_y$: In cases where $d_x > d_y$, there is no need for a hypothesis test, and the $p$-value can be considered to be $0$.\n\n**Definition 1 - Sensitivity to Departures Between $V_x$ and $V_y$**\n\nYou are absolutely correct that even a minute departure of $V_x$ from $V_y$ results in a non-zero $d_{xy}$. We have noted this in a remark right after the definition of $d_{xy}$, which motivates our study of the singular values. \n\n**Definition 2 - Uniqueness of $u_i$, $v_j$, and $\\theta_i$**\n\nIndeed, $u_i$ and $v_j$ are not uniquely defined, but $\\theta_i$ are uniquely defined, so principal angles are well-defined. We have clarified this right after the definition of principal angles. \n\n**Equation 4 - Definition of $\\mathrm{eig}_{1:d_x}$**\n\nYes, the expression $\\mathrm{eig}_{1:d_x}$ refers to the span of the first $d_x$ eigenvectors. We have defined this notation clearly in the revision.\n\n**Theorem 2 - Definition of $\\| \\cdot \\|_{\\psi_2}$**\n\nWe have included a reference to the relevant section in Vershynin's book to provide clarity in the revision.\n\nThank you once again for your valuable feedback. We hope these clarifications and corrections will improve the clarity of our manuscript and we are available to provide further clarification or address any additional questions you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable feedback. This global response covers common questions and concerns. Further detailed responses are provided in the individual rebuttals.

**Hypothesis Test (e5ip, xiqq, Casz)** 

Regarding the assumption of $V_x \subset V_y$ versus $d_{xy} = 0$ under $H_0$, we revised the manuscript to clarify that the statements $V_x \subset V_y$ and $d_{xy} = 0$ are equivalent. 

Another question focused on the need for a hypothesis test given that an estimator is available. The decision to use the hypothesis test is user-dependent. The hypothesis test may be used as a direct measure of whether applying a contrastive DR method is worthwhile and provides a measure of uncertainty. If the test rejects the null hypothesis, it indicates that there is unique information; estimating the contrastive dimension can guide the selection of the reduced dimension parameter. 


A third question involved the discrepancy possible between the results of the estimator and of the hypothesis test. We clarify that the hypothesis test is conservative, so a high p-value is possible even when the estimator indicates that $d_{xy} > 0$. This conservatism is designed to reduce the likelihood of false positives but can also lead to some signals not detected by the hypothesis test. We mentioned nonlinearity as a possible explanation for why the hypothesis test may fail to detect a signal. To elaborate, the hypothesis test’s resampling technique might not handle nonlinear structures in the data effectively (e.g., if the data lie on a nonlinear manifold). This can result in the test failing to detect a signal even when the estimator captures it, as the nonlinear structure can obscure the differences the test is designed to detect.  We discussed the possibility of using a likelihood ratio test as an alternative in the future, which could address some of these limitations. However, the optimization required for the likelihood ratio test is more complicated and computationally intensive. 

**Comments Regarding Experiments (e5ip, Casz)** 

One concern was the difficulty of intrinsic dimension estimation. We agree it's a challenging problem. We tried various methods but used the method of moments estimator in our analyses because it produced the most sensible results.

Another concern was the stability of the method in Simulations 1 and 2 and the need to repeat the simulation and report a measure of variation for the results. To address this concern, we repeated these simulations under the same settings 100 times and report the mean (std) of the results in the table below. 

| Setup  | $d_{xy}$ | $\hat{d}_{xy}$ | p-value | 4 smallest singular values |
|--------|----------|----------------|---------|-----------------------------|
| Sim. 1 | 0 | 0.44 (0.52)  | 0.81 (0.15) | 0.90 (0.02), 0.94 (0.01), 0.96 (0.01), 0.97 (0.01) |
| Sim. 2 | 6 | 6 (0) | 0.03 (0.03) | 0.06 (0.02), 0.11 (0.03), 0.16 (0.03), 0.22 (0.03) |

With known ground truth, the estimator $\hat{d}_{xy}$ works perfectly for Simulation 2, while it makes some errors for Simulation 1. As we mentioned in the paper, these results indicate that a cutoff of 0.9 might be too close to 1, and a lower cutoff may be more appropriate. However, the optimal choice of cutoff could depend on the sample size: Larger sample sizes might be necessary to detect smaller angles between the subspaces $V_x$ and $V_y$. Future work could explore how varying these parameters affects the performance of the estimator and the hypothesis test, potentially leading to more precise guidelines for selecting cutoffs in different scenarios.

**Practical Implications for Downstream Tasks (e5ip, xiqq)**

A third issue discussed was how our methods can be used effectively in downstream data analysis tasks. To address this concern, we conducted further analysis using the corrupted MNIST dataset used in Section 6 of our manuscript. Here, the estimated contrastive dimension is 5. We performed CPCA with dimensions $d = 1, 2, 3, \ldots, 10$, and then ran logistic regression with the outcome (response) variable being the digit (0 vs. 1) imposed on the grass images. The input was the contrastive principal components (cPCs) based on CPCA with different $d$. The results are as follows:

| Dimension (d) | 1 | 2 | 3 | 4 | 5 | 6 | 7  | 8  | 9 | 10 |
|---------------|------|------|------|------|------|------|------|------|------|------|
| Accuracy | 0.58 | 0.94 | 0.94 | 0.94 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 | 0.95 |

The accuracy increases with the number of cPCs and then plateaus at 5, where the accuracy is 0.95, providing indirect evidence that one can simply run contrastive dimension reduction methods with the dimension equal to our estimator (5 in this case). One may argue, however, that $d=2$ is optimal, resembling an elbow point. Our principal angles (measured by singular values) align with this observation. As shown in Table 2 of our manuscript, the smallest four singular values are 0.095, 0.315, 0.705, and 0.846. The two smallest suggest two more prominent contrastive dimensions, leading to satisfactory classification performance for $d=2$. While our suggested cutoff of 0.9 gives $\hat{d}_{xy}=5$, a cutoff of 0.7 gives an estimate of 2. This cutoff is subjective and can be used in conjunction with classification accuracy, particularly if a response variable is available, to improve decision making.

It's important to note that in the broader literature on intrinsic dimension estimation (not specifically contrastive), there is no gold standard for validating the estimated dimension. Our work similarly provides an estimator and suggests a practical approach for its application, but the validation of such estimators in an unsupervised context remains a challenging and ongoing area of research.

We would like to thank the reviewers once again for their valuable feedback. We are open to providing further clarification or discussing any additional questions you may have.

Meta-review: This paper introduces a data-driven hypothesis test to evaluate the suitability of a linear contrastive dimension reduction (CDR) model, along with an estimator for contrastive dimension. This estimate is then suggested as the reduced dimension in downstream analyses involving CDR methods. The work tackles a new problem within the well-established sub-area of dimension reduction. The authors investigate the estimator's consistency and finite sample error bounds, offering a theoretical assessment of its robustness. Validation is conducted through both synthetic examples and real-world applications. The paper is generally well-organized and well-written.

While the technical aspects appear sound, there still are some major weaknesses related to the value and practicality of the proposed CDR formulation, as well as the lack of comparison with SOTA methods to evaluate the performance of their contrastive dimension estimator. Additionally, the study is limited to linear structures in data, which the authors have acknowledged as an open problem for future work.

In the rebuttal, the authors attempted to address these issues, arguing that their work offers an alternative approach for quantifying unique information, which could be valuable in specific applications and provide complementary insights. The new formulation of contrastive dimension is motivated by ensuring that the detected signal represents unique features of the foreground. They clarified that they did not compare their method to other estimates of contrastive dimension because no existing SOTA methods specifically address contrastive dimension estimation, which appears to be correct after a non-extensive review. Nonetheless, they conducted additional numerical experiments, including one to demonstrate the effectiveness of their proposed methods in selecting the dimension hyperparameter $d$ for downstream data analysis tasks. The automatic selection of $d$ needs further discussion.

Despite the aforementioned weaknesses, which were partially addressed in the rebuttal, this paper introduces novel ideas that could contribute positively to the CDR literature. It is supported by a mathematically rigorous formulation, with codes provided to facilitate replication by the research community. Furthermore, the application of this work is particularly beneficial when dealing with datasets with a contrastive structure (e.g., in biomedical studies), where data are split into a foreground group of interest and a background group, making it potentially valuable to practitioners.

Given these considerations, the decision leans toward acceptance. However, for the camera-ready version, the authors should carefully revise the paper to position better the value of the new definition of contrastive dimension. In particular, they should further discuss the following points:

- The differences and similarities between the new task and the one addressed in traditional CDR literature.
- Whether it is possible to solve the new task, possibly with some simple post-processing techniques, using existing CDR methods.
- Potential hints on how the proposed theory could be extended to nonlinear methods.
Decision: Accept (poster)