Contrastive dimension reduction: when and how?

Authors: sam  hawke
yueen  ma
didong  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 19:30:11

Abstract: Dimension reduction (DR) is an important and widely studied technique in exploratory data analysis. However, traditional DR methods are not applicable to datasets with with a contrastive structure, where data are split into a foreground group of interest (case or treatment group), and a background group (control group). This type of data, common in biomedical studies, necessitates contrastive dimension reduction (CDR) methods to effectively capture information unique to or enriched in the foreground group relative to the background group. Despite the development of various CDR methods, two critical questions remain underexplored: when should these methods be applied, and how can the information unique to the foreground group be quantified? In this work, we address these gaps by proposing a hypothesis test to determine the existence of contrastive information, and introducing a contrastive dimension estimator (CDE) to quantify the unique components in the foreground group. We provide theoretical support for our methods and validate their effectiveness through extensive simulated, semi-simulated, and real experiments involving images, gene expressions, protein expressions, and medical sensors, demonstrating their ability to identify the unique information in the foreground group.

Main paper: 
 introduction: 
 High-dimensional datasets are ubiquitous in the era of big data, arising from applications such as image data (Shorten and Khoshgoftaar, 2019) , gene/protein expressions (Bhola and Singh, 2018) , and medical data from wearable devices (Cho et al., 2021; Banaee et al., 2013) , to name a few. Dimension reduction (DR) is crucial in these contexts for various reasons (Fan et al., 2014) : improving data visualization, reducing computational cost for downstream analysis (Fan and Li, 2006) , minimizing noise (Thudumu et al., 2020) , and enhancing interpretability (Johnstone and Titterington, 2009) .DR has been a vibrant research area for several decades, leading to the development of numerous methods (Ayesha et al., 2020) . Principal component analysis (PCA, Hotelling (1933) ), for instance, maximizes the variance in the reduced space and is widely applied across different domains (Jirsa et al., 1994; Novembre and Stephens, 2008; Pasini, 2017) . Other methods, such as Multi-Dimensional Scaling (MDS, Torgerson (1952) ) and Isomap (Tenenbaum et al., 2000) , focus on preserving pairwise Euclidean and geodesic distances between samples, respectively. Probabilistic PCA (PPCA, Tipping and Bishop (1999) ) incorporates uncertainty quantification through a probablistic model. Additionally, techniques such as t-distributed Stochastic Neighborhood Embedding (tSNE, Van der Maaten and Hinton (2008) ) and Uniform Manifold Approximation (UMAP, McInnes et al. (2018) ), preserve the local probability distribution and manifold topological structure in the data, respectively.In recent years, a new type of data has garnered attention, particularly in biomedical research involving case-control studies. Here, data are divided into a foreground group (case or treatment group) and a background group (control group). The objective is to identify low-dimensional representations unique to or enriched in the foreground group. This scenario necessitates contrastive dimension reduction (CDR) methods. For example, in a benchmark dataset for CDR on mouse protein expressions (Higuera et al., 2015) , the foreground group consists of mice who received a shock therapy, while the background group includes mice that did not. The goal is to isolate the protein expression patterns unique to the foreground group, i.e., the shocked mice.To address this problem, several CDR methods have been developed. For example, Contrastive PCA (CPCA, Abid et al. (2018) ) extends PCA by maximizing the variation in the foreground while minimizing variation in the background. Additionally, Probabilistic Contrastive PCA (PCPCA, Li et al. (2020) ) generalizes both CPCA and PPCA by incorporating a probabilistic model into the contrastive setting, enabling statistical inference. Furthermore, the Contrastive Latent Variable Model (CLVM, Severson et al. (2019) ) introduces latent variables specifically designed for this contrastive setting. Moreover, the Contrastive Variational Autoencoder (CVAE, Abid and Zou (2019) ) and Contrastive Variational Inference (CVI, Weinberger et al. (2023) ) both leverage deep learning approaches to detect nonlinear patterns in the data. Collectively, these methods aim to isolate patterns unique to the foreground data relative to the background. These methods have been successful with a variety of datasets, identifying patterns that are often scientifically meaningful and opening doors to new scientific discoveries. For example, studies using these methods on mouse protein expression data have successfully identified subgroup structures, such as distinguishing between mice with and without Down Syndrome, and highlighted key proteins responsible for these differences. We reference various applications and discoveries using these methods in section 6 and appendix D.Despite the development of these CDR methods, two critical questions remain underexplored: when should these CDR methods be applied, and how many unique components exist in the foreground group? The first question can be reframed as follows: given foreground and background groups, can we determine whether there exists unique information in the foreground group? The second question pertains to estimating the number of unique representations to the foreground, termed contrastive dimension in this paper. This is a counterpart of the intrinsic dimension estimation problem, with a rich literature (Camastra and Staiano, 2016; Levina and Bickel, 2004) , but in the contrastive context.In this paper, we answer the first question by proposing a hypothesis test to determine the existence of unique information in the foreground data. To answer the second question, we introduce the rigorous notion of contrastive dimension and provide a consistent estimator of this quantity. We provide theoretical support of our methods and validate their effectiveness through extensive simulated, semi-simulated, and real experiments involving images, gene expressions, protein expressions, and medical sensors.The rest of the paper is organized as follows. In section 2, we provide more background on CDR methods and present our definition of contrastive dimension. In section 3, we present our solutions to the above questions: a hypothesis test and an estimator for the contrastive dimension. In section 4, we prove the consistency of our proposed estimator of the contrastive dimension and establish a finite-sample error bound. We demonstrate the effectiveness of the methods on simulated and semisimulated data in section 5 and real data in section 6. Finally, in section 7, we discuss the strengths and weaknesses of our methods and identify potential directions for further study. 
background: 
 In this section, we introduce our notations and provide a more detailed overview of various existing CDR methods. Throughout this paper, we denote the foreground data as X = {x 1 , • • • , x nx } ⊂ R p with dimension p and sample size n x , and background data as Y = {y 1 , • • • , y ny } ⊂ R p with the same dimension p but potentially different sample size n y . We do not make any assumptions on the ratio of sample sizes between groups, n x : n y . For simplicity, we also assume each dataset is centered separately. Next, we briefly overview some existing CDR methods.Contrastive Principal Component Analysis (CPCA). CPCA (Abid et al., 2018) was proposed to uncover the low-dimensional structure that is unique to the foreground data X relative to the background data Y . Taking one-dimensional CPCA as an example, the objective is to find a direction, represented by a unit vector v ∈ R p , that explains more variance in the foreground among all directions, and less variance in the background. Let C X = 1 nx nx i=1 x i x ⊤ i and C Y = 1 ny ny j=1 y j y ⊤ j be the sample covariance matrices of two datasets. Then, CPCA solves the following optimization problem:argmax ∥v∥=1 v ⊤ C X v -γv ⊤ C Y v = argmax ∥v∥=1 v ⊤ Cv,where γ ∈ [0, ∞] is a tuning parameter, called the contrastive parameter, and C = C X -γC Y is called the contrastive covariance matrix. The optimal v is simply the top eigenvector of C. When γ = 0, CPCA coincides with PCA, and when γ = ∞, CPCA finds the direction that explains the least variance in the background group.Contrastive Latent Variable Model (CLVM). CLVM (Severson et al., 2019) is a latent variable model, similarly proposed to discover patterns enriched in the foreground group X:x i = Sz i + W t i + ε i , y j = Sz j + ε j ,where z i , z j ∈ R k and t i ∈ R t are the latent variables, and ε i , ε j ∈ R p are the noise terms. The factor loading S ∈ R p×k represents the space shared between the foreground and background groups, while the factor loading W ∈ R p×t represents the space unique to the foreground group. The primary goal of CLVM is to identify W, which represents the foreground-specific information.Probabilistic Contrastive Principal Component Analysis (PCPCA). PCPCA (Li et al., 2020) was proposed as a probabilistic model to extend CPCA, allowing for uncertainty quantification:x = W z x + ε x , y = W z y + ε y ,where W represents the factor loading, and ε x , ε y are noise terms. However, PCPCA utilizes an unconventional strategy for estimating W . Instead of maximizing the joint likelihood, which would be appropriate if the foreground and background groups are assumed to share the same space, the PCPCA objective is to maximize p(X|W ) p(Y |W ) γ , where γ ∈ [0, ∞] is the contrastive hyperparameter, as in CPCA. When γ = 0, PCPCA reduces to Probablistic PCA (Tipping and Bishop, 1999) on the foreground data, while PCPCA reduces to CPCA as the noise level goes to zero.Contrastive VAE (CVAE). CVAE (Abid and Zou, 2019) considers a broadened view of the datagenerating process, in which the foreground data X follows an arbitrary probability distribution f with unknown parameters θ, conditional on salient variables s and irrelevant variables z. The background data Y are assumed not to have the salient variables, as the CVAE model is given byx i ∼ f θ (•|s i , z i ), y j ∼ f θ (•|0, z ′ j ). The problem here is to learn the parameters θ of f , which CVAE does by training two probabilistic encoders q ϕs (s|x) and q ϕz (z|x) (salient and irrelevant) to infer s and z, respectively, from the observed features, and a decoder network f θ (•) which reconstructs the original samples from x = [s, z] in the foreground data, or y = [0, z ′ ] in the background data.Contrastive Variational Inference (CVI). CVI was proposed in Weinberger et al. (2023) specifically for the purpose of disentangling the split between treatment and control groups for single-cell data. To this end, CVI posits a data-generating process with latent variables z n , representing shared information between the two groups, and t n , representing information unique to the foreground group. Each gene expression in each sample of the foreground group is assumed to follow a generative process depending on z n and t n (both with N (0, I) priors), but in the background group t n = 0 is assumed, similarly to CVAE. The posterior distribution of CVI is analytically intractable, so it is approximated using variational inference.Each of the methods above has a similar goal: to discover variation unique to the foreground group relative to the background. However, none of these methods enables detection of whether there exists such variation at all. Additionally, for each method, the assumed reduced dimension (or number of latent features unique to the foreground) was treated as a tuning parameter required as input to run the method, without an easy-to-implement method to choose such a dimension.Motivated by this gap, we consider the following linear model:EQUATIONwhere z i ∈ V x , w j ∈ V y for some low-dimensional linear subspaces V x , V y ⊂ R p , and ε i , ε j represent the noise. We let x represent foreground and y represent background.However, defining and identifying the information that is truly unique to the foreground requires a more rigorous approach. For example, in CLVM, even if we estimate a nonzero W , it may not present information unique to the foreground if S and W are not appropriately distinguished. This motivates the need for a precise definition of the contrastive subspace and contrastive dimension:Definition 1. Under the above notation, we define the contrastive subspace to be V xy := Proj V ⊥ y (V x ) and the contrastive dimension, denoted by d xy , to be the dimension of the contrastive subspace:d xy := dim(V xy ).The rationale of our definition is that the contrastive subspace contains the information unique to the foreground data, guaranteed by the projection to V ⊥ y , the orthogonal complement of V y . In order to determine the existence of unique information in the foreground group, it is desirable to have a hypothesis test for testing H 0 : d xy = 0 versus H 1 : d xy > 0. Furthermore, an estimator for d xy is valuable for providing a suggestion to use for the reduced dimension parameter in the CDR models presented above.Remark 1. Based on model 1 and definition 1, V x ⊂ V y is equivalent to d xy = 0.Remark 2. Even a minute departure of V x from V y results in a nonzero d xy .Because we are interested in detecting where the foreground differs from the background, we allow our framework to detect even a small departure of V x from V y , although a bigger departure will be easier to detect. This motivates our study of the principal angle between V x and V y in section 3.It is worth noting that the new tasks we are proposing, namely determining whether there is unique information in the foreground group and quantifying how much, depart from the traditional CDR task of estimating the space representative of such unique information, assuming it exists. In this paper, we focus on our solution to these new tasks, starting with manageable assumptions about the data.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: In section 7, we detail the limitations of the hypothesis test we present, and we suggest an alternative for a more powerful test. We also discuss the need for further investigation into the robustness of CDE to misspecification of intrinsic dimension estimates. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methods: 
 In this section, we present our methods to address the questions of whether there exists unique information in the foreground group and how many dimensions there are unique to the foreground group. First, we detail the hypothesis test of H 0 : d xy = 0 versus H 1 : d xy > 0. We introduce a contrastive version of the bootstrap resampling scheme, based on the null assumption that d xy = 0, which is equivalent to the assumption of V x ⊂ V y . Next, we introduce the CDE of d xy . In both methods, we make the very reasonable sample size assumptions that n x > d x and n y > d y , as many existing studies have observed a small intrinsic dimension (Pope et al., 2021) . The inference in both of these problems is based on the notion of principal angles between two linear subspaces (Ye and Lim, 2016) : Definition 2. Let U and V be two subspaces of R p with dimensions d 1 and d 2 , then the principal angles, denoted byθ i (U, V ), i = 1, • • • , min(d 1 , d 2 ) are defined recursively as follows cos(θ i (U, V )) = max ui∈U,vi∈V,∥ui∥=∥vi∥=1,u ⊤ i uj =v ⊤ i vj =0,∀j<i u ⊤ i v iTo be clear, the u i and v i solving the maximization problem are not uniquely defined; however, the principal angles θ i are uniquely defined. The principal angles measure the "difference" between two subspaces. For instance, if two spaces coincide, all principal angles are zero; when two spaces are orthogonal to each other, all principal angles are π/2. If U 0 and V 0 are orthogonal matrices spanning U and V , respectively, then the calculation of the principal angles is given by the following equationEQUATIONwhere σ i (•) denotes the i-th singular value of a matrix. As a result, we adopt this notion to study the relation between V x and V y for our purposes.We develop a hypothesis test for testing H 0 : d xy = 0 versus H 1 : d xy > 0 under the version of model 1 with the following assumptions,EQUATIONwhere S x ∈ R p×dx and S y ∈ R p×dy are full rank, z i ∼ N dx (0, I), w i ∼ N dy (0, I),E[ε i ] = E[ε j ] = 0, Cov(ε i ) = σ 2x I, and Cov(ε j ) = σ 2 y I. First, we assume max(d x , d y ) < p/2, which is a very weak assumption given that p is often much bigger than the intrinsic dimension in high-dimensional datasets. Remark 3. With model 1, we do not make any assumptions the relations between intrinsic dimensions d x and d y . However, by definition 1, d x > d y implies that d xy > 0, indicating that there must be some unique information in the foreground space. Therefore, for our hypothesis test, we focus on the more challenging but important case of d x ≤ d y .An equivalent formulation of the null and alternative hypotheses in this model isH 0 : C(S x ) ⊂ C(S y ) versus H 1 : C(S x ) ̸ ⊂ C(S y ), where C(•) denotes the column space of a matrix. Motivated by the assumption that V x ⊂ V y under H 0 , we draw a sample of size n x with replacement from {x 1 , . . . , x nx }, and we draw a sample of size n y with replacement from {x 1 , . . . , x nx , y 1 , . . . , y ny }. We repeat this bootstrap procedure B (B = 1000 throughout this paper) times and compare the largest angle between V x and V y in the original data with the distribution of the largest angles between those subspaces in the resampled data. The exact calculation is deferred to section 3.2.Algorithm 1: Contrastive Bootstrap Hypothesis Test We demonstrate with simulations in section 5 that this hypothesis test produces conservative results, and in section 7 we discuss a potentially more powerful likelihood-based alternative. Additionally, this hypothesis test requires as input intrinsic dimension estimates for d x and d y , for which there exists a rich literature with a variety of sophisticated methods (Campadelli et al., 2015) . This conservatism is designed to reduce the likelihood of false positives, but it can also lead to some signals not being detected by the hypothesis test. One explanation for this phenomenon could be that the test's resampling technique might not handle nonlinear structures in the data effectively (e.g. if the data lie on a nonlinear manifold). Because V x and V y are assumed to be linear subspaces, a nonlinear pattern can obscure the differences the test is designed to detect.Input: Foreground {x i } nx i=1 , Background {y j } ny j=1 , dimensions d x ≤ d y , bootstrap parameter B Output: p-value for test of H 0 : V x ⊂ V y in model 1 λ 1 ← λ 1 (x, y, d x , d y ) ; //We start from the following lemma to link our target, d xy , and principal angles between V x and V y . Lemma 1.d xy = dim(Proj V ⊥ y V x ) = #{i : θ i (V x , V y ) > 0} + max(d x -d y , 0). As a result, to construct an estimator of d xy , it suffices to estimate V x , V y , and calculate the corresponding principal angles. Based on eq. ( 1), V x and V y consist of the top d x and d y eigenvectors of Σ x and Σ y , respectively, which can be estimated by sample covariance matrices Σ x := 1 nx nx i=1 x i x ⊤ i and Σ y := 1 ny ny j=1 y j y ⊤ j . Thus, we propose the following estimator:EQUATIONwhere ϵ > 0 is a small toleration due to randomness from finite samples, and eig 1:d refers to the span of the first d eigenvectors. Let U x := eig 1:dx Σ x and U y := eig 1:dy Σ y , by eq. ( 2), we have λ i := cos(θ i ) = σ i ( U ⊤ x U y ),( Σ x , Σ y ) ← (n -1 x nx i=1 x i x ⊤ i , n -1 y ny i=1 y i y ⊤ i ) ; // Compute sample covariances ( U x , U y ) ← eig 1:dx ( Σ x ), eig 1:dy ( Σ y ) ; // Compute sample eigenvectors U DV ⊤ ← U ⊤ x U y ; // SVD ( λ 1 , . . . , λ min{dx,dy} ) ← diag(D) ; // Singular values d xy ← #{j : λ j < 1 -ϵ} + max(d x -d y , 0) return d xyNote that algorithm 2 takes intrinsic dimensions d x and d y as input. In practice, the user must choose a method to estimate the intrinsic dimension of both the foreground and background datasets. This flexibility allows for the incorporation of different intrinsic dimension estimators to complement our approach (Bac et al., 2021; Pope et al., 2021) .Some users may only be interested in estimating the contrastive dimension d xy and not in the hypothesis test for d xy = 0. These users can skip the hypothesis testing step. However, the hypothesis test offers a direct way to assess uncertainty, guiding users on whether it is worthwhile to proceed with contrastive dimension reduction methods.In the next section, we study the asymptotic and finite-sample behavior of this estimator d xy .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: We have two theoretical claims about our estimator, and we state the assumptions and provide complete and correct proofs for them in appendix C. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
consistency and finite sample error bound: 
 In this section, we provide theoretical support for the estimator d xy presented above, first by establishing its consistency, and then by constructing a finite-sample error bound for it.Theorem 1. Assume that the second moments Σ x and Σ y are finite for both groups, and that the top d x + 1 eigenvalues of Σ x and top d y + 1 eigenvalues of Σ y are distinct. Then, our proposed estimator d xy is consistent:d xy nx→∞ ----→ ny→∞ d xy .The assumption of distinct eigenvalues is common in the literature (Vershynin, 2018) and ensures the identifiability of eigenvectors for each covariance matrix. Because we are interested in convergence to the space spanned by the top eigenvectors, rather than individual eigenvectors themselves, this assumption might be relaxed to the eigengap assumption that the d x th and (d x + 1)th eigenvalues of Σ x are distinct, and that the d y th and (d y + 1)th eigenvalues of Σ y are distinct. See section 7 for more discussion.To describe the finite sample error bound, we recall that U x and U y are spanned by the top d x and d y eigenvectors of Σ x and Σ y , respectively, and singular values ofW := U ⊤ x U y , denoted by λ i = σ i (W ), i = 1, • • • , m, where m = min(d x , d y ), are the cosines of principal angles between U x and U y . In practice, estimating the contrastive dimension as an integer can be somewhat restrictive for providing a finite sample error bound. Instead, we report the singular values to account for more uncertainty and provide a more nuanced understanding. Thus, we present the following finite sample error bound (and hence the convergence rate) of these estimated singular values rather than d xy .Theorem 2. In addition to all assumptions in theorem 1, we assume x and y in model 1 are sub-Gaussian. More precisely, assume that there exist constants K x , K y ≥ 1 such that ∥⟨x, v⟩∥ ψ2 ≤ K x E ⟨x, v⟩ 2 and ∥⟨y, v⟩∥ ψ2 ≤ K y E ⟨y, v⟩ 2 for any v ∈ R p where ∥•∥ ψ2 is defined as in Section 3.4 of Vershynin (2018) . Then, for any u > 0, with probability at least 1 -2e -u , we haveEQUATION)where C is an absolute, positive constant, δ • is the minimum eigengap among the top d • + 1 eigenvalues of Σ • , and ∥•∥ denotes spectral norm.To show the finite sample error bound, we need the sub-Gaussianity assumption to ensure that the tails of the distribution are not too heavy. This assumption is also common and encompasses a large class of distributions (Vershynin, 2018) .Having established the theoretical support for the estimator, in the next section we focus on applying our methods to simulated data.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: We provide all code and information necessary to access and pre-process the data in the anonymous Github repository at the link in appendix B. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
simulations: 
 To demonstrate the validity of our methods in practice, we apply them to four simulations for different purposes. The first two examples are purely simulated, fully under our control, with d xy = 0 for the first one and d xy > 0 for the second one. Because these first two simulations are purely synthetic, we repeat them both 100 times and report the means and standard deviations of the p-value, d xy , and the four smallest singular values. The third simulation steps away from the ground truth being completely known by combining synthetic image data with known dimension, with real, grassy noise image data, elucidating the methods' performance in an interpretable setting. The final simulation is a variation on the third one, in which the synthetic data are replaced with MNIST images (Deng, 2012) .We describe the experimental settings of the simulations in more detail below and present results, including the p-value from our test d xy , and the smallest four singular values, as shown in Table 4 . We denote the estimates with a hat, if needed. In all these examples, the threshold is ϵ = 0.1.We do not compare d xy with other estimators of contrastive dimension because, to our knowledge, no such estimators have been proposed in the literature. 2 x = σ 2 y = 0.25, C(S x ) ⊂ C(S y ) such that d xy = 0, n x = n y = 100.Because d xy = 0 is the assumption of H 0 for the hypothesis test in Section 3, we repeat this simulation 100 times to empirically demonstrate the conservative type I error rate (defined by the proportion of p-values below α) versus pre-specified α ∈ (0, 1). We also repeat this simulation but with n x = n y = 200, and we present both plots of empirical type I error rate in Figure 1 .As shown in the first row of Table 4 , both d xy and the p-value agree, on average, with the ground truth that d xy = 0, showing that in this simulated setting, our methods do not erroneously detect a signal when there is none. However, given that the average smallest singular value value observed is 0.901, this simulation suggests that it may be prudent to use a choice of ϵ greater than 0.1, i.e. a cutoff for the singular value that is below 0.9. The standard deviation for d xy among the 100 replications is 0.52, which also suggests that a lower cutoff may be appropriate. The standard deviation of the p-value is 0.153, and of each of the smallest singular values is 0.025, 0.013, 0.009, and 0.007, respectively. Both plots of observed type I error in Figure 1 demonstrate that, in this setting, the hypothesis test is conservative. We discuss a more powerful potential alternative in Section 7. Simulation 2. The only difference from Simulation 1 is that d xy = 6. Because the singular values were all well below the cutoff of 0.9, the standard deviation of dxy among the 100 trials was 0; every trial achieved the correct estimate. Our hypothesis test produced a small p-value of 0.029 on average (standard deviation 0.029), which at a significance level of α = 0.05 leads one to the correct conclusion of rejecting H 0 and concluding that d xy > 0. Moreover, the six smallest singular values are 0.059, 0.106, 0.159, 0.215, 0.279, 0.367 (standard deviations 0.023, 0.028, 0.032, 0.034, 0.038, and 0.050, respectively), all well below the threshold of 0.9, illustrating that CDE correctly identifies 6 directions unique to the foreground data.Simulation 3. It is also useful to evaluate the performance of our methods on datasets where the ground truth is only partially known. So, in this simulation, we construct a semi-synthetic image dataset by generating 5000 disks of random centers (c x , c y ) ∈ {5, . . . , 24} 2 and radii r ∈ {1, . . . , 10} uniformly, in the 28 by 28 grayscale pixel space.Of these 5000 images, we draw a white line of random length beginning at the leftmost pixel of the sixth row on 2500 images to create the foreground (Fig. 2A ). As a result, d x = 4, parameterized by c x , c y , r, and length of line. For background, we superimpose the remaining 2500 images on grass images as noise (Fig. 2B ) so that d y > d x . Although d y is not known, d xy is known to be one (the unique parameter being the length of the line). Here, we used the method of moments estimator (Amsaleg et al., 2018) , leading to d y = 20.Our method only gives one singular value (0.529) below the cutoff of 0.9, leading to a correct estimate of d xy = 1. Additionally, our hypothesis test produces a p-value of 0, which suggests to reject H 0 in favor of H 1 : d xy > 0. This simulation illustrates that our methods produce favorable results when one of the groups has an estimated intrinsic dimension.Simulation 4. To assess our methods in a simulation setting where both d x and d y are unknown, we consider a very similar simulation to the previous one, but where we start with 5923 handwritten 0's from the MNIST dataset (Deng, 2012) instead of randomly generated disks. We further randomly split them into two sets, to create foreground by adding lines (Fig. 2C ), and background by imposing on grass images (Fig. 2D ), leading to the corrupted MNIST data proposed in Abid et al. (2018) . Using method of moments, d x = 11 and d y = 14. Crucially, the contrastive dimension d xy = 1 is still known, allowing us to assess the results of our methods when d x and d y are both unknown. CDE again accurately estimates exactly one singular value (0.256) below the cutoff of 0.9, leading to a correct estimate of d xy = 1. Furthermore, our hypothesis test produces a p-value of 0, leading to the correct conclusion that d xy > 0 at a significance level of α = 0.05. After successful applications of our methods to various settings, it is time to apply them to real datasets.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: All code and instructions to obtain and pre-process datasets are in the anonymous Github repository, for which we provide the link in appendix B.Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: Our methods mostly concern the applicability of other dimension reduction methods designed to analyze case-control data. While it is possible that our method might have a societal impact, we choose to focus more on method development than downstream effects difficult to observe at this stage in the work. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
real data experiments: 
 In this section, we apply our methods to real datasets previously studied in the literature to validate their effectiveness and interpret the results. In Table 2 , we summarize relevant characteristics of each dataset along with the results. For each dataset, the intrinsic dimension estimation was performed with the method of moments estimator (Bac et al., 2021) . Note that in each case where d x > d y , the p-value is not reported because the hypothesis test does not apply. The cutoff for the singular values is set to 0.9 in all experiments. 2020). The foreground group depicts handwritten 0's and 1's corrupted by images of grass, and the background contains only the grassy noise (no handwritten digits). In this case, the digits are unique to the foreground, supported by our p-value of 0.008. Furthermore, the contrastive dimension estimate for this dataset is d xy = 5, in line with the commonly believed dimension of digits 0 and 1 (Pope et al., 2021) .Mouse Protein. This dataset (Higuera et al., 2015) is a benchmark in the literature, considered in Abid et al. (2018) , Li et al. (2020) , and Severson et al. (2019) . The foreground group contains the protein expressions of mice, some with and some without Down Syndrome, who were subjected to shock therapy, while the background group consists of protein expressions of mice without Down Syndrome who did not receive shock therapy. The intrinsic dimension estimates of d x = 5 and d y = 4 require that d xy > 0. In this setting, where the foreground space is bigger than the background space, the foreground group necessarily carries unique information. Nevertheless, the CDE estimate for this dataset of d xy = 4, which aligns with the observation of d xy ≥ 2 in the literature.mHealth. The mHealth dataset (Banos et al., 2015) , studied in Abid et al. (2018) ; Severson et al. (2019) , consists of measurements made from medical sensors while a subject performs certain actions.While the foreground group contains measurements taken of a subject either squatting or cycling, the background measures a subject lying down. Our p-value < 0.001 and d xy = 1. This estimate explains the result in Abid et al. (2018) , duplicated here in figure 3 , showing that a single direction suffices to distinguish subgroups in the foreground data.BMMC. This is a single-cell RNA sequencing dataset (Zheng et al., 2017) Here, the foreground group consists of gene expressions measured on bone marrow mononuclear cells (BMMCs) of patients with acute myeloid leukemia before and after receiving a transplant. The background group consists of healthy patients. Previous studies found that the 3rd and 4th PCs can separate pre-and post-transplant groups (Fig. 3 ), supported by our d xy = 17 > 4. However, the p-value of 0.073 stands somewhat in contradiction to this result, maybe due to the conservative nature of this hypothesis test, as illustrated in figure 1 and discussed in section 7.Small Molecule. This is a dataset of cell line responses to small-molecule therapy (McFarland et al., 2020) and used in Weinberger et al. (2023) . Here, the foreground group contains measurements from 24 cell lines treated with idasanutlin, while the background are the same cell lines treated with Regardless, CDE provides a helpful estimate for the amount of information unique to the foreground group, suggesting the efficacy of CDR methods.CelebA: The CelebA dataset (Liu et al., 2015) was studied in Abid and Zou (2019) and with images of celebrities. We perform two experiments, with foreground being those wearing glasses and hat, respectively. In both, the background group is the remaining images. In the former case, d xy = 1, while in the latter d xy = 2. However, both cases have a somewhat large p-value, perhaps due to the conservative nature of the hypothesis test or nonlinearity. 
conclusion and future work: 
 In this paper, we have introduced a hypothesis test to determines whether a linear CDR model is appropriate for a given dataset and an estimator of contrastive dimension, representing the dimension of the subspace unique to the foreground group relative to the background. This estimate of contrastive dimension can be used as the reduced dimension in downstream analysis with a CDR method.Meanwhile, there are still some open problems left to future work. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our method simply provides a p-value and an estimate for a parameter to be used in other dimension reduction methods. Because the output consists of merely a handful of numbers to be used as a guideline for future analysis, it poses extremely low risk for misuse.Answer: [NA] Justification: Our research does not involve human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Estimator. The problem of estimating intrinsic dimension is challenging in its own right (Camastra and Staiano, 2016; Campadelli et al., 2015; Fefferman et al., 2016) . While methods for it have been developed, there is not a single accepted standard for it. Fortunately, since Hypothesis Test. Our hypothesis test introduces the notion of the contrastive bootstrap to allow for resampling under the assumption of the null hypothesis that the foreground space is a subset of the background space. However, a more powerful likelihood ratio test is applicable. Under model 3, the denominator is exactly from the solution of PPCA Tipping and Bishop (1999) . However, the numerator involves a nontrivial constrained optimization problem. Naive gradient step method is neither effective nor scalable. An interesting future direction is to resolve this optimization issue.d xy -d xy ≤ d x -d x + d y -d y ,In section 4 we mentioned that the distinctness of eigenvalues ensures the identifiability of eigenvectors for each covariance matrix. However, this assumption can be relaxed since our focus is not on estimating each individual eigenvector but rather on identifying the eigenspace spanned by the top d x and top d y eigenvectors of Σ x and Σ y .Extension to Nonlinear Setting. In section 6, we mentioned that the data lying on a nonlinear manifold could lead to the apparent inconsistency between the estimator d xy being nonzero and the p-value produced by the hypothesis test being large. To appropriately handle the nonlinear case, a future work may consider a nonparametric extension analogous to the extension of CLVM (Severson et al., 2019) to CVAE (Abid and Zou, 2019) .C.1 Proof to lemma 1 Lemma 1.d xy = dim(Proj V ⊥ y V x ) = #{i : θ i (V x , V y ) > 0} + max(d x -d y , 0). Proof. Fix V x , V y ⊂ R p , and let dim(V x ) = d x , dim(V y ) = d y . Let dim(V x ∩ V y ) = k.Let u 1 , . . . , u k be an orthonormal basis for V x ∩ V y . It follows that u 1 , . . . , u k ∈ V x ∩ V y are the first k principal vectors for both V x and V y , corresponding to principal anglesθ 1 = • • • = θ k = 0.Furthermore, for every i = 1, . . . , k, we know that Proj V ⊥ y u i = 0, because u i ∈ V y .Case 1: Assume d x ≤ d y . In this case, there are d x -k nonzero principal angles corresponding to principal vectors we denote by u k+1 , . . . , u dx ∈ V x and v k+1 , . . . , v dx ∈ V y . For each i = k + 1, . . . , d x , we know that Proj V ⊥ y u i ̸ = 0, because u i / ∈ V y . In fact, because the u i are orthogonal,Proj V ⊥ y u k+1 , . . . , Proj V ⊥ y u dx are a basis for Proj V ⊥ y V x . Therefore, dim(Proj V ⊥ y V x ) = #{i : θ i (V x , V y ) > 0}.Case 2: Assume d x > d y . In this case, there are d y -k nonzero principal angles. Similar to the above argument, we can extend the basis for V x starting with the principal vectors u k+1 , . . . , u dy ∈ V x corresponding to the principal angles. However, because there are only d y principal angles, we denote by u dy+1 , . . . , u dx the remaining vectors necessary to specify an orthogonal basis for V x . Because the principal angle maximizes u ⊤ i v i , the remaining vectors u dy+1 , . . . , u dx are all orthogonal to V y . Therefore, Proof. We know by Anderson (1963) that the sample matrix of eigenvectors U x converges to the true matrix of eigenvectors U x in probability as n x → ∞, and similarly, U y → U y as n y → ∞. Let m = min (d x , d y ), and consider the following functions.Proj V ⊥ y u k+1 , . . . , Proj V ⊥ y u dy , u dy+1 , . . . , u dx , is a basis for Proj V ⊥ y V x . It follows that dim(Proj V ⊥ y V x ) = #{i : θ i (V x , V y ) > 0} + d x -d y , as desired.f 1 : R p×dx × R p×dy → R dx×dy (U x , U y ) → U ⊤ x U y f 2 : R dx×dy → R m A → (λ 1 , . . . , λ m ) ⊤where λ j is the jth singular value of A It is clear that f 1 is continuous. Due to Weyl's inequality for singular values, f 2 is continuous with respect to the spectral norm. The contrastive dimension is defined to be d xy = #{λ j < 1}, while its estimator is defined by d xy = #{ λ j < 1 -ϵ}. Therefore, by the continuous mapping theorem, it holds that λ → λ in probability. Taking ϵ → 0 gives d xy → d xy , as desired.Theorem 2. In addition to all assumptions in theorem 1, we assume x and y in model 1 are sub-Gaussian. More precisely, assume that there exist constants K 1 , K 2 ≥ 1 such that ∥⟨x, v⟩∥ ψ2 ≤ K 1 E ⟨x, v⟩ 2 and ∥⟨y, v⟩∥ ψ2 ≤ K 2 E ⟨y, v⟩ 2 for any v ∈ R p where ∥•∥ ψ2 is defined as in Vershynin (2018) . Then, for any u > 0, with probability at least 1 -2e -u , we haveEQUATION)where C is an absolute, positive constant, δ is the minimum eigengap among the top d + 1 eigenvalues of Σ, and ∥•∥ denotes spectral norm.Proof. Let W = U ⊤ V , where U is the matrix of the top d x eigenvectors of Σ x and V is the matrix of the top d y eigenvectors of Σ y . We use hats to represent sample estimates of these matrices and letW = U ⊤ V .We would like a finite-sample error upper bound on max j=1,...,m σ j W -σ j (W ) . max j=1,...,mσ j W -σ j (W ) ≤ W -W by Weyl's inequality for σ-values = U ⊤ V -U ⊤ V + U ⊤ V -U ⊤ V ≤ U ⊤ V -U ⊤ V + U ⊤ V -U ⊤ V = U • V -V + U -U • ∥V ∥ = V -V + U -U ≤ V -V F + U -U F ≤ 2 3/2   d x Σ nx -Σ x δ x + d y Σ ny -Σ y δ y  The final inequality holds due to the Davis-Kahan theorem, as presented in Vershynin (2018) . Note that here δ x is the smallest eigengap of Σ x , and δ y is the smallest eigengap of Σ y .For k ∈ {x, y}, with probability at least 1 -2e -u , we have thatΣ nx -Σ x ≤ CK 2 k p + u n k + p + u n k ∥Σ k ∥where C is an absolute, positive constant, and K x = ∥x∥ ψ2 , K y = ∥y∥ ψ2 are the sub-Gaussian constants, as in Vershynin (2018) . Substituting these upper bounds on Σ n k -Σ k yield the bound claimed.The code for all experiments is made available at the anonymous Github link provided in appendix B.To allow for reproducibility, we set a seed for every experiment in which we perform the bootstrap test. Sometimes we set the seed to be 1; other times we set the seed to be 42. There was no particular reason behind the choice for any seed. In every experiment requiring intrinsic dimension estimation, we used the method of moments estimator from the sci-kit dimension package. Other estimators are possible and may give slightly different results, but we chose this estimator because we found it to give the most stable and reasonable results. In our experience, we found some of the other estimators to give estimates as low as 0 and as high as the number of features, of which neither estimate is sensible in the contexts we considered.For simulations 1 and 2, S x and S y were randomly chosen to be orthogonal matrices in R p×dx and R p×dy , respectively. Therefore, the covariance matrices in this simulation are given by Σ x = S x S ⊤ x + σ 2 x I and Σ y = S y S ⊤ y + σ 2 y I, respectively. Notably, these covariance matrices violate the assumption of distinct eigenvalues presented in section 4. However, the space spanned by the top d x eigenvectors of Σ x is still identifiable, because the d x th eigenvalue is 1 + σ 2x , while the (d x + 1)th eigenvalue is σ 2x (similarly for Σ y ). As discussed in section 7, this is the necessary identifiability condition.We designed simulations 3 and 4 to allow for an interpretable ground truth of d xy = 1 while mimicking real image data. Figure 2 illustrates the data-generating process for both groups in both simulations. Every image is represented by 28 by 28 matrix in the grayscale space, where 0 represents black, and 255 represents white. Each of these image matrices was converted to a vector in R 784 . In both simulations, the foreground group was created by drawing a white line on top; each pixel in this line was set to 255, while the background group was created by averaging the original (disk or MNIST 0) image entrywise with an image of grass from the corrupted MNIST dataset. The was no specific intent behind choosing to draw a horizontal line in the sixth row of each image; drawing a line of random length going in any single direction from any starting pixel could have worked. However, we chose to fix the starting pixel because we wanted to be sure not to introduce a second parameter. That said, simulations in which other parameters are introduced, leaving a different ground truth d xy are possible. For example, creating the foreground group by drawing a white rectangle of random width and height (w, h) ∈ {1, . . . , 24} 2 beginning at the top-left pixel would lead to a scenario in which d xy = 2.The Mouse Protein, mHealth, BMMC, Small Molecule, ECCITE-Seq, Pathogen, Perturb-Seq, and CelebA experiments all required additional pre-processing. To ensure that our pre-processing aligned with previous analyses, we used the code available for pre-processing these datasets associated with the papers analyzing them. To be specific, we pre-processed the Mouse Protein and mHealth datasets using the code associated with Abid et al. (2018) . We pre-processed the Small Molecule, ECCITE-Seq, Pathogen, and Perturb-Seq datasets using the code associated with Weinberger et al. (2023) . We downloaded the BMMC dataset using instructions and code from Weinberger et al. (2023) and pre-processed it using the code from Li et al. (2020) . Code files for replicating the pre-processing we did for each experiment are available on the anonymous Github in appendix B. To pre-process the CelebA dataset, we first used the PIL package in Python to resize the images from 178 by 218 pixels to 64 by 64 pixels. From there, we performed PCA (Anderson, 1963) on the resized images and took the top 1000 directions.In section 6 we made reference to our results standing in agreement with results from previous analyses. In particular, we claimed that for the BMMC dataset, our observation of d xy ≥ 4 was consistent with the analysis in Li et al. (2020) , in which a figure was published illustrating a distinction between the pre-and posttransplant groups in the BMMC data using the third and fourth PCPCA directions. We duplicated that graph in the left panel of figure 3 for transparency. We also claimed that for the mHealth dataset, our observation of d xy = 1 was consistent with the analysis in Abid et al. (2018) , in which a figure was published illustrating a single direction differentiating the two groups of squatting and cycling. We duplicated this illustration in the right panel of figure 3 here for convenience.For clarity on the image datasets used in section 6, we provide sample images from the foreground and background groups of both the Corrupted MNIST and CelebA datasets. The images for the Corrupted MNIST application are in figure 4 ; the top row has examples of images in the foreground group, while the bottom row has examples of images in the background group. The images for the CelebA application are in figure 5 ; the top-left two are foreground (glasses), while the bottom-left two are background (glasses). Similarly, the top-right two are foreground (hat), while the bottom-right two are background (hat).To explore how the methods we presented in this paper can be used effectively in downstream data analysis tasks, we conducted further analysis with the corrupted MNIST dataset used in section 6. We performed CPCA with dimensions d = 1, 2, 3, . . . , 10, then ran logistic regression with the The accuracy increases with the number of cPCs and then plateaus at 5, where the accuracy is 0.945, after which the accuracy increases only slightly. This provides indirect evidence that if one can simply run contrastive dimension reduction methods with the dimension equal to our estimator (5 in this case). One may argue, however, that d = 2 is optimal, resembling an elbow point. Our principal angles (measured by singular values) align with this observation. As shown in Table 2 , the smallest four singular values are 0.095, 0.315, 0.705, and 0.846. The two smallest singular values suggest that there are two more prominent contrastive dimensions, leading to satisfactory classification performance for d = 2. While our suggested cutoff of 0.9 gives dxy = 5, a cutoff of 0.7 gives dxy = 2. This cutoff is subjective and can be used in conjunction with classification accuracy, particularly if a response variable is available, to improve decision making. NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.All code available at https://github.com/myueen/contrastive-dimension-estimation 38th Conference on Neural Information Processing Systems (NeurIPS2024). 


Decision: This paper introduces a data-driven hypothesis test to evaluate the suitability of a linear contrastive dimension reduction (CDR) model, along with an estimator for contrastive dimension. This estimate is then suggested as the reduced dimension in downstream analyses involving CDR methods. The work tackles a new problem within the well-established sub-area of dimension reduction. The authors investigate the estimator's consistency and finite sample error bounds, offering a theoretical assessment of its robustness. Validation is conducted through both synthetic examples and real-world applications. The paper is generally well-organized and well-written.

While the technical aspects appear sound, there still are some major weaknesses related to the value and practicality of the proposed CDR formulation, as well as the lack of comparison with SOTA methods to evaluate the performance of their contrastive dimension estimator. Additionally, the study is limited to linear structures in data, which the authors have acknowledged as an open problem for future work.

In the rebuttal, the authors attempted to address these issues, arguing that their work offers an alternative approach for quantifying unique information, which could be valuable in specific applications and provide complementary insights. The new formulation of contrastive dimension is motivated by ensuring that the detected signal represents unique features of the foreground. They clarified that they did not compare their method to other estimates of contrastive dimension because no existing SOTA methods specifically address contrastive dimension estimation, which appears to be correct after a non-extensive review. Nonetheless, they conducted additional numerical experiments, including one to demonstrate the effectiveness of their proposed methods in selecting the dimension hyperparameter $d$ for downstream data analysis tasks. The automatic selection of $d$ needs further discussion.

Despite the aforementioned weaknesses, which were partially addressed in the rebuttal, this paper introduces novel ideas that could contribute positively to the CDR literature. It is supported by a mathematically rigorous formulation, with codes provided to facilitate replication by the research community. Furthermore, the application of this work is particularly beneficial when dealing with datasets with a contrastive structure (e.g., in biomedical studies), where data are split into a foreground group of interest and a background group, making it potentially valuable to practitioners.

Given these considerations, the decision leans toward acceptance. However, for the camera-ready version, the authors should carefully revise the paper to position better the value of the new definition of contrastive dimension. In particular, they should further discuss the following points:

- The differences and similarities between the new task and the one addressed in traditional CDR literature.
- Whether it is possible to solve the new task, possibly with some simple post-processing techniques, using existing CDR methods.
- Potential hints on how the proposed theory could be extended to nonlinear methods.

Reviews: review: paper_summary: paper_summary: The authors address two key challenges using two datasets, categorized as foreground and background: 1) Determine whether contrastive dimensionality methods are appropriate for application to such a pair of datasets and 2) Quantify the unique information present in the foreground data.
strength_weakness: strength_weakness: --Hypothesis Test:
The assumptions, such as $V_x \subset V_y$ under $H_0$, lack proper justification.
The hypothesis test relies on estimating the intrinsic dimension, which is inherently challenging. The authors employ other methods for these estimates but fail to compare the accuracy of their estimates to these methods. These methods could also be used to test $d_{xy} = 0$.
The overall presentation of the hypothesis test is inadequate. The rationale for choosing this specific test and how calculating the singular values of the custom $\hat{V}_y$ improves the $p$ estimate remains unclear. The authors do not justify, explain, discuss, or comment on their choices.
--Experimental Evaluation:
The method lacks comparison to other approaches: For intrinsic dimension estimation, there's no clear way to compare against other methods due to the absence of ground truth. Alternative evaluations should be designed, such as training GANs to a priori upper-bound the intrinsic dimension of generated data by the dimension of the latent noise vector, as done in [1].
The authors don't compare their work to other "contrastive dimensionality reduction" methods. They could compare (i) expressed variance in the foreground group among different methods, (ii) expressed variance in the background group, and (iii) expressed variance common across groups. Alternatively, they could propose other ways to quantify performance and compare their method to existing literature.
The intrinsic dimension estimates are compared to those of (Pope et al., 2021), which aren't ground truth values. This makes it difficult to compare the inferred dimension to the actual one.
The practical implications of the algorithms aren't tested. How can these methods use dimensionality reduction to identify foreground sets? How can they be applied to related downstream classification tasks? How can they be used to compare different experiments?
Despite being motivated by "large scale data," the experiments are limited to small dimensions (e.g., MNIST), with no consideration of scalability crucial in today's datasets.
There's no exploration of other datasets or discussion on how intrinsic dimensions were estimated.
--Terminology: The terminology is often unclear and potentially misleading. For example, "unique information in the foreground group" refers to low-rank components, and terms like "contrastive information" and "contrastive dimension" could be misinterpreted in the context of self-supervised learning. Alternative terms like "complementary information" may be more appropriate. Even the title is misleading as "contrastive dimension reduction" is closely related to contrastive learning, a self-supervised dimensionality reduction method.
Title: The title doesn't accurately reflect the paper's content and focus.
--Missing Conclusion: The paper lacks a conclusion section.
Language Issues: The manuscript contains several typos and linguistic errors, including repeated words in the abstract (e.g., "with with").
[1] Pope, P., Zhu, C., Abdelkader, A., Goldblum, M., & Goldstein, T. The Intrinsic Dimension of Images and Its Impact on Learning. In International Conference on Learning Representations.
questions: questions: Please see above
limitations: limitations: Please see weakness section
paper_summary: paper_summary: The paper proposes to examine the existence and estimate the number of contrastive dimensions between the foreground and background groups. The authors provide a formal definition of contrastive dimensions and propose a hypothesis test method to examine the existence of contrastive dimensions. The authors also propose a contrastive dimension estimator to estimate the number of contrastive dimensions, and provide theoretical analysis on the consistency and finite sample error bound of the proposed estimator. Experiments are conducted on synthetic semi-synthetic and real-world datasets to evaluate the performance of the proposed hypothesis test and contrastive dimension
estimator.
strength_weakness: strength_weakness: 1. The formulation of contrastive dimension seems impractical. For instance, consider the foreground group $X\in\mathbb{R}^3$ drawn from $\mathcal{N}(\mathbf{0}, diag(2, 1, \sigma_\varepsilon))$ and the background group $Y\in\mathbb{R}^3$ drawn from $\mathcal{N}(\mathbf{0}, diag(1, 1, \sigma_\varepsilon))$, where $\sigma_\varepsilon$ denotes the standard deviation of the noise terms. Both the foreground $Z_i$ and background $W_j$ occupy the same subspace, specifically the column space 
$\mathcal{C}\left(\left[ {\begin{array}{ccc} 1&0&0 \\\\ 0&1&0\\\\ \end{array} } \right]^\top\right)$
. The contrastive dimension is expected to be $dim\left(\mathcal{C}\left(\left[1\quad 0\quad 0\right]^\top\right)\right)$, yet according to the definition 1, the contrastive dimension is $dim(V_{xy})=0$ .
2. In section 2, the paper claims that a significant application of the proposed methods is to determine the number of dimensions for contrastive dimension reduction. However, the experiments provide limited discussion on the effectiveness of these methods in selecting hyperparameters for downstream contrastive dimension reduction tasks. This lack of detail undermines the practical utility of the proposed
approaches.
3. The paper mentions that the hypothesis test is conservative, and the experimental results indicate that the hypothesis test does not offer significant advantages over simply detecting whether $\hat{d}_{xy}=0$. However, the paper does not justify the need for a hypothesis test when the contrastive dimension estimator appears to be effective on its own.
questions: questions: 1. Could you explain in more detail the practicality of the formulation? Can this formulation address the example given in the weaknesses section?
2. Could you conduct additional experiments to assess the effectiveness of the proposed methods in selecting hyperparameters for downstream contrastive dimension reduction tasks?
3. Could you justify the necessity of a hypothesis test when simply detecting whether $\hat{d}_{xy}=0$ seems to be effective?
limitations: limitations: Yes
paper_summary: paper_summary: The paper deals with a contrastive dimension reduction (CDR) techniques which seeks to find a unique low-dimensional pattern that only exists in the foreground data compared to background data. Authors point out that despite of recent developments of related techniques such as contrastive principal component reduction (CPCA), there lacks a rigorous guideline to determine when and how those techniques should be utilized. Hence, they propose a hypothesis testing method to determine whether the foreground data actually have unique information which background data do not and an estimator of the number of CPCA PCs. The authors then investigate theoretical aspect of the estimator including the consistency and finite sample error bound. They also conduct simulation study and real data analysis to demonstrate the performance of the proposed method.
strength_weakness: strength_weakness: 1. Discrepancy between the p-value obtained from permutation test and the estimator happens in some cases. The authors might need to conduct additional simulation study to investigate the reason behind this discrepancy.

2. The simulation study seems to be too simple without repetitive iterations to check the stability of the proposed method. More extensive simulation is recommended. Authors need to at least provide the average of the repeatedly estimated value.


3. The proposed hypothesis testing relies on the permutation method which makes the result unstable and further theoretical investigation difficult.
questions: questions: 1. As one of the possible reasons for the discrepancy mentioned as the 1st weakness, the authors conjecture that it might be due to the nonlinearity of the underlying unique pattern. However, if there actually exists nonlinear pattern, why does the estimator succeed to capture it while hypothesis testing fails? This point needs to be elaborated.

2. From the formulation, it seems that the number of samples $n_x$ and $n_y$ should always be larger than $d_x$ and $d_y$ respectively to correctly estimate the dimension. Can you elaborate on the relationship between the sample size and intrinsic dimension with possible implications to the method?

3. Under the CLVM model which has a close relationship with CPCA, background data only has shared information implying that $d_x$ > $d_y$. However, the manuscript also concerns a case where $d_x$ < $d_y$ but $d_{xy}$ > 0 assuming the model (1). Can this be understood as suggesting a different viewpoint from CLVM to understand the CDR?
limitations: limitations: The authors well addressed the potential limitations and future research goals of the study. They included the possibility of introducing likelihood ratio testing method to overcome the weakness of the permutation test pointed out as the 3rd weakness of the manuscript.
paper_summary: paper_summary: The authors consider the problem of contrastive dimension reduction/estimation. In a nutshell, this problem involves a background dataset and a foreground dataset, and it is of interest to determine if the foreground dataset contains submanifolds/subspaces that are inherently different than those in the background. While there's past work on determining the differences of the foreground from the background, this paper asks if such a difference exists in the first place. This is posed as a hypothesis testing problem. An intuitively reasonable test is proposed and it's shown that the test is consistent. The authors also derive finite sample bounds on the error, resorting to commonly used sub-Gaussianity assumptions. They demonstrate the utility of the test through synthetic as well as real data.
strength_weakness: strength_weakness: I think the manuscript has minor weaknesses. The most important one of those is that the proposed test itself is not very surprising, but I don't know if that's really a weakness -- after all it's backed up theoretically, and simple is good.
Other than that, I list a few questions below, which should be easy to address by the authors.
questions: questions: - line 81 : Do you assume a relation between $n_x$ and $n_y$, like, for instance $n_y \gg n_x$?
- line 83 : Do we center $X$, $Y$ separately? Or, in practice, do we assume $X$ and $Y$ to have the same centers?
- line 131 : Should $Z_i$ and $W_i$ be lower case? If not, why not?
- line 312 : Please state any assumptions on the dimension of $V_x$, $V_y$. If no assumptions are made, please note that.
- Defn 1 : Even a minute departure of $V_x$ from $V_y$ would result in a non-zero $d_{xy}$. But you deal with that in the sequel. It may be worth noting that.
- Defn 2 : This doesn't uniquely define $u_i$'s and $v_j$'s does it? But are $\theta_i$'s uniquely defined?
- line 174 : Again, if there are any assumptions on the ratio $n_x : n_y$ would be good to note here. 
- eqn 4 : Does the expression $\text{eig}_{1:d_x}$ mean the span of the first $d_x$ eigenvectors? Please define that somewhere in the text.
- Thm 2 : It would be good to include the definition of $\| \cdot \|_{\psi_2}$, or a reference to the section where it's defined in Vershynin's book.
limitations: limitations: N/A

Comments: title: Response to author rebuttal
comment: Thank you for the rebuttal. I'll keep my rating.
Thank you for your responses, that would help further clarify the paper. My suggestions were minor, and I already gave a high score. I'll keep it as is.
Thank you for your continued feedback. We understand the importance of demonstrating the practicality of our formulation.

To clarify, our focus is not on proposing a new contrastive dimension reduction (CDR) method, but rather on determining whether or not to use a CDR method and identifying the appropriate dimension if it should be used. Existing CDR methods such as CLVM, CPCA, cVAE, can be seen as complementary, rather than competing, as they serve different purposes when analyzing case-control data.


The motivation behind our definition of contrastive dimension (CD) is to ensure that the detected signal represents features truly unique to the foreground, especially in contexts where existing approaches might mistakenly identify shared features as unique. For example, consider gene expression data for diseased individuals (foreground, $x$) and healthy individuals (background, $y$). In the CLVM framework, where $\Sigma_x = SS^\top + WW^\top + \sigma^2 I_p$ and $\Sigma_y = SS^\top + \sigma^2 I_p$, the matrix $W$ is interpreted as representing genes unique to the foreground. A naive approach to defining CD might rely solely on $W$, say, the rank of $W$. However, $W$ does not necessarily contain information unique to the foreground. For example, when $S = W$, the same genes are present in both groups, and they are not truly unique to the foreground. In contrast, our method would yield a CD of zero, which is more aligned with the correct genomic interpretation—these genes are not unique to the diseased group but shared across both groups. 

It is important to note that CLVM does not discuss this issue and does not define any notion of contrastive dimension in their work. This potential for misinterpretation is why we defined CD in a way that specifically ensures the detected signal represents features that are truly unique to the foreground group. 

We do not wish to claim that this is the only way to define CD. Potential alternative definitions could be interesting and useful in different contexts. However, we believe that our definition is useful in certain cases because, by projecting onto $V_y^\perp$, we ensure that any detected signal is in a direction that is different from the space spanned by the background data, making it unique to the foreground group. As far as we know, ours is the first approach to explicitly define and estimate contrastive dimension.

We believe this illustrates the utility of our formulation, particularly in applications like genomics where understanding the true uniqueness of features is critical. We are happy to provide further clarification or discuss ways to improve our work.
Thank you for your thoughtful reconsideration and for taking the time to review our responses. We appreciate your support and are glad our clarifications were helpful. We are happy to provide any additional clarification or engage in further discussion if needed.
Thanks to the authors for the rebuttal. However, it does not answer my question about the practicality of their formulation, and it is not clear why the new formulation seems more reasonable (at least from the submitted manuscript or the rebuttal). I would like to keep the score unchanged.
After considering the authors' responses and reviewing the other evaluations and rebuttals, I've decided to increase my score.