Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Authors: Lei Yan
Xin Zhang
Qing Mai
Venue: ICML 2025 poster (2025)
Keywords: High-dimensional inference
EM algorithm
Mixture model
Primary area: general_machine_learning->clustering
TL;DR: The paper presents mixPFC, a model combining subspace clustering with sufficient dimension reduction for heterogeneous data, achieving clustering, subspace estimation, and variable selection with theoretical guarantees.

Abstract: Scientific and engineering applications are often heterogeneous, making it beneficial to account for latent clusters or sub-populations when learning low-dimensional subspaces in supervised learning, and vice versa. In this paper, we combine the concept of subspace clustering with model-based sufficient dimension reduction and thus generalize the sufficient dimension reduction framework from homogeneous regression setting to heterogeneous data applications. In particular, we propose the mixture of principal fitted components (mixPFC) model, a novel framework that simultaneously achieves clustering, subspace estimation, and variable selection, providing a unified solution for high-dimensional heterogeneous data analysis. We develop a group Lasso penalized expectation-maximization (EM) algorithm and obtain its non-asymptotic convergence rate. Through extensive simulation studies, mixPFC demonstrates superior performance compared to existing methods across various settings. Applications to real world datasets further highlight its effectiveness and practical advantages.

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Lei Yan
Xin Zhang'
Qing Mai
Abstract
dimensionality. First, real-world data often lie in union of multiple subspaces with unknown membership, reflect- ing underlying latent sub- sub-populations. Second, PCA, as an unsupervised method, equates variation with informa- tion and thus disregards the specific relationship between Y and X. Third, in the high-dimensional setting where p is much larger than the sample size n, the subspace estimated by PCA can be highly unreliable, or even orthogonal the true subspace (Baik & Silverstein, 2006; Paul, 2007).
Scientific and engineering applications are of- ten heterogeneous, making it beneficial to ac-
count for latent clusters or sub-populations when learning low-dimensional subspaces super- vised learning, and vice versa. In this paper, we combine the concept of subspace clustering with model-based sufficient dimension reduction and thus generalize the sufficient dimension duction framework from homogeneous regres- sion setting to heterogeneous data applications. In particular, we propose the mixture of princi- pal fitted components (mixPFC) model, a novel framework that simultaneously achieves cluster- ing, subspace estimation, and variable selection, providing unified solution for high- -ineesinnnsi heterogeneous data analysis. We develop group Lasso penalized algorithm and obtain its non-asymptotic conver- gence rate. Through extensive simulation stud- ies, mixPFC demonstrates superior performance compared existing methods across various set- tings. Applications to real world datasets fur- ther highlight its effectiveness and practical ad- vantages.
Addressing the challenges of heterogeneity and high di- mensionality is crucial for statistical and machine Iearn- ing methods. Subspace clustering is a family of pow- erful methods for clustering the data into multiple sub- spaces (Vidal, 2011; Soltanolkotabi Candés, 2012). Most subspace clustering methods generalize PCA and factor analysis from single subspace union of multiple sub- spaces (Agarwal & Mustafa, 2004; Vidal et al., 2005; Yan & Pollefeys, 2006; Tron & Vidal, 2007; Favaro et al., 2011; Elhamifar & Vidal, 2013). However, like PCA, these meth- ods ignore the response, leading to inevitable loss of regres- sion relevant information. Furthermore, subspace cluster- ing methods often assume clean observations with small random noise (Kanatani, 2001; Vidal et al., 2005), mean- ing data points are expected to lie nearly exactly within a subspace. When the observations are subject to significant random errors, these methods frequently fail to accurately identify clusters.
(EM)
1. Introduction
Conversely, sufficient dimension reduction (SDR) provides supervised framework by projecting X onto low- dimensional subspace while preserving all relevant regres- sion information. Formally, we have
Reducing high-dimensional data a low-dimensional rep- resentation is one of the most important steps in multivari- ate statistics and various applied sciences. Typically, this is achieved by projecting data onto a single -inddmmnnional subspace. Among unsupervised dimension reduction meth- ods, principal component analysis (PCA) is probably the most popular one. However, when applied the context of regressing univariate response on p-dimensional pre- dictor X, PCA faces three critical limitations: data hetero- geneity, loss of information on regression, and the curse of
Y Y
where Ps is the projection matrix onto S. The intersec- tion of all the subspaces satisfying (1) is called the central subspace. Recent advances in deep learning- SDR methods (Banijamali et al., 2018; Liang et al., 2022; Kapla et al., 2022; Huang et al., 2024; Chen al., 2024) have demonstrated the potential to capture complex nonlinear structures in high-dimensional data.
'Department of Statistics, Florida State University, Tallahas- see, Florida, United States. Correspondence to: Xin Zhang <henry sta..fsu.edu>.
While SDR guarantees regression sufficiency (Li, 1991; Cook Forzani, 2008), it overlooks the heterogeneous nature of scientific and engineering applications. Incor-
Proceedings of the 1nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
porating heterogeneity into SDR has the potential en- hance dimension reduction by uncovering latent subpopu- lations. Motivated by this, we integrate the subspace clus- tering concept with model-based SDR.
ishnan et al., 2017). Unlike many existing proofs in high-dimensional EM algorithms, our analysis does not require sample splitting (Kwon et al., 2019) and allows a relatively general model. Specifically, we derive a non-asymptotic convergence rate for a two- mixture principal fitted components model with un- known mixing proportions and without restrictions on the minimum angle between subspaces.
Contributions. In this paper, we propose a novel mixture of principal fitted component (mixPFC) model, designed for simultaneous clustering, variable selection, and dimen- sion reduction. has the following major contributions.
2. Mixture of Principal Fitted Components
Supervised Subspace Clustering: By extending subspace clustering into supervised framework, mixPFC identifies subspaces that preserve regression information, addressing both heterogeneity and pre- dictive accuracy. This contrasts sharply with unsuper- vised approaches (Elhamifar & Vidal, 2013; Ji et al., 2017; Cai al., 2022). Leveraging response informa- tion, mixPFC allows for exact overlap between sub- spaces, overcoming limitations of classical methods that require separation conditions (e.g., minimal angle condition in Soltanolkotabi & Candés (2012)). Build- ing upon SDR, mixPFC targets central subspaces rather than subspaces where X resides. This ensures the low-dimensional representation of X preserves all the information relevant to regression. Moreover, un- like most subspace clustering methods that separate clustering from subspace estimation, mixPFC per- forms both tasks jointly in a unified framework.
We extend the framework of sufficient dimension reduction by introducing a latent variable to model heterogeneity in data. In particular, we consider univariate (continuous or discrete) response Y E R, multivariate predictor X E RP, and latent categorical variable W E {1,2,.... K). We aim to estimate K subspaces Sar. w- = 1,..., K, such that
W = ww) Y (Ps., X,W
(2)
Pr(W X,X) Pr(W SY,)sX)
where S - --= S. C RP offers the usual SDR seen in the literature and each Ps.. is the projection matrix onto Sw to capture the relationship between Y and X within cluster w. Let B E Roxd denote a basis matrix of S. Then, the projected data , X contains all relevant infor- mation in X to be combined with response information Y for clustering data into K clusters. When W is observable, the smallest space S. is known as the conditional central subspace, and is building block for studying the partial central subspace (Chiaromonte et al., 2002). However, our problem is much more challenging because W is latent and has to be inferred from data. Moreover, unlike existing par- tial/conditional central subspace methods, we further incor- porate variable selection for high-dimensional studies.
High-Dimensional Estimation: We develop a group penalized expectation- maximization (EM) algorithm for the mixPFC model. Extending SDR to high di- mensions is a challenging and nascent research area (Lin et al., 2018; 2019; Tan al., 2018; 2020; Zeng et al., 2024). Existing approaches often require in- verting p x p matrices or estimating parameters in a pi-dimensional space, which poses scalability chal- lenges. Our model, designed to estimate multiple heterogeneous subspaces in different unknown sub- populations, is much more complicated. To address this challenge, we formulate the subspace estima- tion convex optimization over an approximately p-dimensional parameter space. We further incor- porate a group Lasso penalty (Yuan & Lin, 2006) for coordinate- -indeneeddedeeedeeett variable selection (Chen et al., 2010).
We propose the mixPFC model, as a generative mixture of principal fitted components (PFC),
X / (Y, W = =w) ~ N Nww + I f(Y) A), Pr(W = w) = Two w w= K,
(3)
where Maw RP is the center of each cluster, T.. E Rox is the coefficient matrix that represents the relationship be- tween Y and X in each cluster. f(-) = (ff.))..... f((+))" R ++ R99 is a set of pre-specified fitting functions that in- troduces non- relationships,
A E RPxx is symmetric positive definite matrix, and m.. > is the mixture proba- bilities with Σ KK= T. 1.
Theoretical Guarantees: We establish theoretical re- sults for the proposed group penalized EM algorithm. While classical EM theories only guaranteed asymp- totic convergence to fixed point, we derive non- asymptotic result that mixPFC converges geometri- cally to a fixed point that is within statistical preci- sion of the unknown true parameter. This stronger type of guarantee has emerged only recently (Balakr-
The mixPFC model unifies and generalizes many model- based clustering and model-based SDR approaches. When K 1, the mixPFC reduces to the PFC model (Cook & Forzani, 2008), which further reduces to the probabilis- tic principal component analysis (PCA) model (Tipping & Bishop, 1999) by restricting A = I²I, and replacing f(Y)
2

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
vision.
S
Finally, the following proposition identifies the key param- eter for fitting mixPFC.
Proposition 2.1. Under model (3), the smallest subspaces satisfying (2) are S span - ). W = 1,..., K. Consequently, du = (im(Sw) = rank([ and d = dim(E
Subspace Clustering
Mixture of PFC
K Sw) S <> dw-
Figure 1. The mixPFC model enhances subspace clustering by considering how the response (solid line) changes as the predictor varies within each cluster: linear response effects in the first mix- ture and non- non-linear in the second.
The rank of Tw RPx0, du could be smaller than g. the number of functions in f. Our model-based SDR ap- proach for handling heterogeneity in data is now rigorously connected to the central subspace notion in general (2). Based on the maximum likelihood estimation (MLE) for PFC model parameters (Cook Forzani, 2008), we derive the MLE for S., and, more importantly, penalized EM algorithm for nigh-dimensiooal data.
as latent variables V/ ~ L)) Therefore, our mixPFC, by allowing general covariance structures and incorporat- ing response information, a supervised generalization of the probabilistic PCA. On the other hand, we completely remove the effect of the response (YY), the mixPFC be- comes the Gaussian mixture model (GMM) (McLachlan et al., 2019; Cai et al., 2019). If we further have A = I'Ip. mixPFC is a model-based interpretation of K-means clus- tering (Forgy, 1965; MacQueen et al., 1967). When Y is categorical, the mixPFC model reduces to the mixture dis- criminant analysis model (Hastie Tibshirani, 1996; Fra- ley & Raftery, 2002).
3. Group-Penalized EM Algorithm
Let {(,,, Y.) be n independent data points from mixPFC (3). 0 (A, Twi Mlo: Sw, w = I..... K) be the set of unknown model parameters. In low dimensions, all the parameters can be estimated by the EM algorithm.
The EM algorithm aims to maximize the log-likelihood of X Y over 0, by iteratively alternating between an Expectation-step (E-step) and a Maximization-ster (M- step). The conditional log-likelihood X Y
The mixPFC can also be viewed extension of sub- space clustering. Figure demonstrates the advantage of mixPFC over subspace clustering. In subspace clustering. we have a fully unsupervised problem on X, but in mixPFC we further have the response Y to guide our clustering. which can be very informative. Consequently, subspace clustering requires a minimal angle condition for identifia- bility (Soltanolkotabi & Candés, 2012), but in mixPFC we allow non-overlap, partial overlap, and complete overlap between Sj and S₁ for any k). With the response Y, even when S₁ = Sy. we can still have r * T₂ to ensure cluster identifiability, although sT) = sT) A more concrete demonstration is given in Figure 2, where we have conducted two toy example simulations based on the proposed model (3). The first toy example in Figure 2 (a) has S₁ = S2, but the response Y has differrntt relationships with X in the two clusters. The subspace clustering would completely fail to identify the subspace or to cluster data while mixPFC works well and produces near perfect sub- space estimation and clustering results. In Figure (b), we have another simulation where S₁ 1 S2, which is an ideal setup for subspace clustering methods. We applied our pro- posed method and two popular subspace clustering meth- ods: random sample consensus (RANSAC, Tron & Vidal (2007)) and sparse subspace clustering (SSC, Elhamifar & Vidal (2013)). The clustering errors by subspace clustering methods, 8.4% (SSC) and 13.0% (RANSAC), are reduced by Pix 3% thanks to the additional response super-
K
((0) = Σ log
Σ
T N
(X;
Au Twf,, 4)
im
where f; = f(Y.) and N μ4, A) is the probability density function of multivariate normal distribution with mean μ and covariance A. In the E-step, we compute the expecta- tion of the log- likeeihoooo function of 0 with respect to the conditional distribution of W given ((X; K;))
(())
New ( (log(T))
(4)
1=1
+ log N NX; Muw + Ti A))]
where Yow (0(0) = Pr(W; = w X1, Yz). Assuming the cluster means fau are equal, the estimated probability Now (0()) is given by
Now (1
exp 1/2 DED
(t)
Ä (w) - ~0 ff]
Then, in the M-step, we
update @
argmax, (())@()) by maximizing (4).
3

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Response
Response
Response
00
Predictor Projected onto S (=S²)
Predictor P Projected onto S ( 21
P Predictor Projected onto S (15,)
(a) Toy Example
(b) Toy Example
200. The proposed mixPFC method works well both extremes of setups: (a) Subspaces coincide, i.e, Si = S2, but mixtures are well-separated by response variability. this case, subspace clustering completely fails. (b) Subspaces orthogonal to each other, i.e. S .L Sz. For both examples, we plot the response versus the estimated linear reductions of the predictors based on the mixPFC method.
Figure 2. Two toy simulation examples. with = 6 and two mixtures of size nh
The above standard EM algorithm infeasible for high- dimensional problems. The inverse of p x p covariance matrix A serves the cornerstone of the EM algorithm. In high dimensions, it is impractical to use A repeat- edly in the EM updates. However, it can be seen that the probabilities Vou (0(() are evaluated on the linear function of X5: (1)
timates in EM updates are computed as
New (10) - -1!0) - -2!0
U 6
Yow (1)) (X())) ) F)T.
T By Proposition 2.1, A (T, ) is contained in the central subspace. Hence, there is no loss of information to first project X; onto the central subspace and then calculate the probability based on reduced predictors, avoiding the p p matrix inversion A A1. Specifically, given basis matrix 3 Rexd of S, focus the reduced predictors 3 (Y))
where 1 I/n E3 f.. lem,
EEi Yemr (001) Vin (10) X, and f Then we solve the convex optimization prob-
B
argmin B ERRxy
(t B..)
ge < E Rd. This is mixture linear regression problem Z = fYY) + &. where Z = X,
(6)
T w. and & ~ (O,, A') with A 3 AB E Rdxd Then the updating equation for the probabilities can be simplified to
B + B B2
where À > 0 is tuning parameter and the L2,1 penalty B
Σm (2) 22)) is coordinate- independent (Chen et al., 2010). The problem in (6) is con- vex. We develop a groupwise coordinate descent algorithm to solve it efficiently. Note that B E IRPX''s are naturally rank deficient, with q 2 du- Therefore, at the convergence of the penalized EM algorithm 1, top-d left singular vectors of B be the subspace esti- mates S.p.
(2) exp
Now (0())=1
(Z [AA0 A (1) (.)) ((+) 100) (A
Σ
π
jf ww
(5)
the span of the
A f, x +1.
As shown in the original PFC paper (Cook & Forzani, 2008), subspace estimation remains consistent under mis- specification f(Y), provided f(Y) is sufficiently corre- lated with the true function. Our mixPFC model inher- its this property, ensuring validity across a broad class of functions. In practice, polynomials splines standard choices. The initialization method, selection of K, along with mixPFC-ISO an alternative algorithm tailored for isotropic covariance matrices- is detailed in Section B of the appendix. The code is available on GitHub at https: / //gittts //gttbub. cow/ eeillllnnnnneeeee monn-ly. 1y //mm xPFC.
Given B, the closed-form updates for A", Ag. and The are straightforward to derive. The most challenging part re- maining is how to obtain an accurate estimator the cen- tral subspace.
For high-ddmennsionll predictors, we consider the following groupwise penalized estimation of the basis matrix of each subspace B. First of all, we recognize that span( span( A -T = span " U,,,, where Σ,,, = cov (X W w) E PPx and U.. = cov(X, f(Y) W = w) E Rex9 are the covariance matrices. The iterative sample es-
4

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Algorithm Penalized EM algorithm for mixture PFC
Input: Data {(X,, Y=)) fitting function f(-) Initialize (0%) and center f;
repeat
#
E-Step: Yow (ioo(0(1) (0(1) =
Σ jZ a expl((2) - A ()))) DE)
To
M-Step: ((++1) = argmin Bw 2 tr tr(B Bw) - -tr((U.)) ) Bw] } + 2 Bu
=((+1) 1/n Di= Nio (0()) (01)), p = 1/E 1ri ( Σm=1 Yiw 0 O
f
until converge Output: R.. Sm
4. Theory
the theoretical properties of a simplified version of Algo- rithm I, which is detailed in Algorithm 4 in the appendix. Let 0* denote the true value of 0, and x(t) represent the estimate of 0 at the t-th iteration. The true parameter space is defined as
4.1. Preliminary
We begin this section with some notations. For num- bers and b, a V b and A b means max(a, by and min(a, b). For an integer n. [n] denotes the set {l....,n}, For a vector x = zp)" ||x|| = Ei=1 [xal. VE -'=1 o7. For matrix A (ar)),
o = =10" : E (CT, 1- 1.,11-c,,, cx), vec ll S sq.
| P S Ma. | S M,. w 1,
and ||x|| mmin(A) (A) and max ( (A) represent the smallest and largest singular values of A. SA denote the column space of A. The Frobenius norm, and spectral norm of A are de- fined as | = (tr(AA)), ||||| = O max (A). The Frobenius inner product between two matrices A and B is (A, F) tr(AaB) B). For matrices A and B with same column rank d, the distance between subspaces is SB) = ||PA- PBllF/v2d. For a set A S (I. A and [A| denote its complement and car- dinality. For two sequences of positive numbers }an) and {bn}. an $ bn or an = O(b,) means an/bn S C < co, and an = o(b,) means that /n/bn - 0 as + Co. Let gp4-1 be the unit sphere. For a positive integer < p3g), let (()) {u E RP9 ||u | x
where each condition has a natural interpretation. The con- dition कi E (Ca, --) ensures each latent cluster has sufficiently large sample size. The sparsity condition vec |lo S sq reflects group sparsity structure, and Till I M,, are used in the literature on mixture linear regression (Yi & Caramanis, 2015; Wang et al., 2024). The parameter B is the true solution to the optimization prob- lem (6), and defined later in the text.
Since o² 1 and K 2, the conditional probability Pr(W, 1 X,, Y,) is simplified as
(n) -1 (1/m 1) expl [X 1 11/(TT2 -1/2(T2+T,)6)) T.) (T2 r1)f,} +1,
(Vsq + 2qv (//33)] |us ll2 + /sgllulla, for some Si C [Pq] with S 3sq} and [s(s) = ((s)1 {uip E (. For matrix A IRxx0 define || SuPueBex*, (et (A, F.
; Y2) = - ma (0). The following quantities are used repeatedly in the theoretical analysis:
and let x2 = Pr(W, = 2/0 x,
We conduct the theoretical analysis under the assumption that Y, is fixed, f(.) is known, and fa = O. We focus on the case where K = 2, common assumption in high- dimensional EM algorithm analysis (Cai et al., 2019; Wang al., 2024). We further assume that A o²I, with o known. Treating the covariance matrix as known param- eter is also standard in theoretical studies of simpler models such as mixture linear regression (Klusowski et al., 2019; Wang et al., 2024) and the Gaussian mixture model (Xu et al., 2016; Cai et al., 2019). Without loss of generality, we set x² 1. Under these assumptions, we re-define the parameter as 0- (T,, I T2). since T2 = क and S,, (T ,,), 2. With this setup, we analyze
(0)
Yho (
(o)(0) = E. TT (0))
n i=1
(.) =
New (O), (= =((ū
x=l
(..( =
= Yow OXX, =0) (0))
where the expectation is with respect to X,, 1, 1.2.... ... n. We define B (E%) - w. where Σx E
and U is = (0') Let 300 and Bi represent top- du left singular vectors of B and B Then have
S S3.- Let (Q) = ( (0), )), U (0), ()) w
5

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
1 represent the sample-based estimates and (0) (0). U (0), En (0), tion counterpart.
(C3) There exists sufficiently large constant M3 > which does not depend on n,p,8, such that oh (B > Ms 2 / (lgg
1.2) be the popula- To quantify the differences be- tween two sets of parameters, we introduce a distance (f(M(O:), M ())). defined as
(C4) Q 2 Ci(co, Co Mo; = L...., 4) for a constant that is only depends on cu. Mr. Co. and M,, 1,...,4, 4, and C. < C2(M2) for a constant only de- pends on M2.
max f| Tw (01) Tu ( (02) V (e - U. (02)l ==1,2
v (0)) m. (02 B lF)
Let Ω Vu((T= (oiTT:-rjEE(C rJ denote the sig-
nal strength of the mixture PFC model, where Σσ 1/n Ei= f.f.. We define the contraction basin (f)) as the set:
(C5) n > C3sq log(p) for a sufficiently large constant C3.
Condition (C1) is mild since f, is g-dimensional vector, where is a small fixed number that does not grow with n and Condition (C2) ensures the initialization lies within the contraction basin, which guarantees the estimates pro- duced at each step of the EM algorithm stay in the con- traction basin. It is a common condition in mixture mod- els (Cai al., 2019; Wang et al., 2024). Condition (C3) requires that the nonzero singular values of B are suf- ficiently separated from zero. This is a standard assump- tion in the theoretical analysis of high dimensional SDR problems (Zeng et al., 2024). Condition (C4) has two re- quirements. The first one is that the signal strength is larger than a constant that does not depend on n and p such that the two mixtures are distinguishable. This sumption is widely used in mixture linear model (Zhang et al., 2020; Wang et al., 2024). The second is that, for the parameters Twr within the contraction basin, the dis- tance | - is bounded by the signal strength multi- plied by a universal constant independent of n and p. Con- dition (C5) is a common assumption in high dimensions on the relationship among n. p, s to guarantee consistent estimation (Meinshausen & Yu, 2009; Cai et al., 2019). Specifically, it implies that the restrictive eigenvalue con- dition inf unf 1 |u" E X.X? ul > holds with high probability for a positive constant Ti-
cu (0') - : the E (co, 1 - cn), - Fallo Cr
(1 Ca) $ tr(6 (T) - II)
< (1 + C2)
vec m ) E [()), 11,2]
where co S Cx and (T) = T - (T + /22. The contraction basin requires that θ is not far away from the true parameter 0- Under the conditions shown later, an initialization (() within the contraction basin guarantees that all subsequent estimators ((() remain in the contraction basin throughout the iterative process.
4.2. Main Results
We need some technical conditions before stating the theo- retical results.
(C1) The singular values of Σx 1/n Ei= f.f. satisfy
that M1 S minn (Er)= S max Et) S M2, and M3 I min |fall2 I max1 |fulla S M4.
(C2) The initialization ((0) satisfies that (r() 0*) V
[ (0) BillF V [B Bill < rl, and
vee (T r E [(s), with r < 1c0 cz l/ A 62
C. A
1 e (V/Ca 1/(4/M14MM + daa
) a² =
Next, we state the main result for the subspace es- timation error of mixPFC in Theorem 4.1, with its proof provided in Section C in the appendix.
222 /VMi b M/M2 + [Ma+ + /Ma/2//VMM
Theorem 4.1. Under conditions (CI)-(C5), there exists a constant 0 < 1/2, such that B satisfies, with probability at least 1 - o(1,
(log n) logp
[ - -Bille S (0) 0*) D 0 - B v | -Bill) +
computational evror
staiiticall error
Consequently, for 2 (-log n log bog(n(dr 00) 0r) v ( Bi F V B B)
sqa sqa 2 logp
[+() k B FF D(S 30 Spz.) )ax)s S
n
6

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Theorem 4.1 is the first theoretical result in high- dimensional heterogeneous SDR. Compared the high- dimensional PFC result in Zeng et al. (2024), our con- vergence rate is slower by factor of log n, reflecting the added complexity of unknown cluster labels. Importantly, Theorem 4.1 I holds under unequal proportions and arbitrary subspace angles, making it highly non-trivial. Even in low- dimensional settings, existing EM theory often requires ad- ditional assumptions such as equal proportions (Gaussian mixtures (Xu et al., 2016)), or symmetric coefficients (mix- tures of linear regression (Zhu et al., 2017)). Additionally, our analysis does not rely on sample splitting, a common technique in the literature (Yi et al., 2014; Yi & Carama- nis, 2015; Zhang et al., 2020) that divides the data into many batches and uses new batch of samples in each iteration to make random samples and current parameter estimates independent. Sample splitting, while theoreti- cally convenient, is suboptimal in practice as it decreases estimation efficiency and is rarely used in real-world ap- plications. Recent work by Wang et al. (2024) derived a rate of (s(log n)) p/n without data splitting for mix- ture of linear regression. However, the mixture of PFC is inherently more complex. Our rate is slower by factor of q3/2 due to the dependence on the g-dimensional vec- tor f(Y). Similarly, compared to Gaussian mixture model (Cai et al., 2019), the convergence rate is slower by a factor of /²/2 log n due to the involvement of Your in Ew,()) and function f(Y).
5. Numerical Results
5.1. Simulations
We compare the mixPFC and mixPFC-ISO against ex- isting methods in clustering accuracy, subspace estima- tion, and variable selection. Since no existing method si- multaneously classifies the data and estimates subspaces, we evaluate our methods against subspace clustering ap- proaches for clustering error rates and high- -igh-iimensional SDR methods for subspace estimation and variable selec- tion. The subspace clustering methods considered include LSA (Yan & Pollefeys, 2006), SSC (Elhamifar & Vidal, 2013), LRSC (Favaro et al., 2011), GPCA (Vidal et al., 2005), and RANSAC (Tron & Vidal, 2007). GPCA is ap- plied only to the important variables due to computational constraints. Additionally, K-means and hierarchical clus- tering, are included and applied to both X and XoY, where the i-th row of Xo Y is defined as Y, x X, For variable se- lection and subspace estimation accuracy, we include Las- soSIR (Lin et al., 2019), SEAS-SIR, and SEAS-PFC (Zeng et al., 2024). Clustering results are presented in this section, with subspace estimation and variable selection results pro- vided in Section A of the appendix.
We consider four settings for central subspaces, denoted as models MI-M4, to examine different configurations of mixtures. Models M1-M3 have K = 2 mixtures with dif- ferent degrees of overlap between two subspaces. Specifi- cally, the subspaces are identical in M1, orthogonal in M2, and oblique in M3. Model M4 randomly generates multi- ple subspaces (K > 2). which tend to be nearly orthogonal to each other sine 8 = 10. The dimension dw of each sub- space is for M1 and M2, and 2 for M3 and M4. The active set is defined as An = {l...., 6} for MI-M3, and Aur = {1..... 10} for M4. Across all models, we set the = 0, f(Y) (Y, [Y))?, and nw = 1/K. After the basis matrices B.r of central subspaces are generated ac- cording to M1-M4 (parameters provided in Section A of the appendix), we set F,, = A, where The E R Xq links the central subspace and function f(Y). Imbalanced clusters and non-linear functions are also examined in Sec- tion A of the appendix.
Starting with an initial value within the contraction basin, Theorem 4.1 shows that the proposed algorithm converges to the true parameters at rate containing both computa- tional and statistical errors. The computational error, ex- pressed as x' <'(d=(0 0-) v [ -Bill v | F))ls), diminishes exponentially as t 00 since 0 < < 1/2. The statistical error, V " n22 p/g./n. rep- resents the irreducible estimation error and persists regard- less of the number of EM iterations. For sufficiently large
|[B[]) Bill V | B }} the computation error becomes negligi- ble relative to the statistical error. Beyond this step, addi- tional iterations do not improve the estimators. Notably, since this threshold grows only logarithmically with n, the Algorithm 1 achieves accurate estimation in practice within a limited number of iterations.
t 2 (- log k)) log (n(rr (0(0) 0*) V
The sample size is fixed at n. 200K with p 1000, and for each simulation setting, 100 independent datasets are generated. To explore the influence of differ- ent covariance structures, we examine four configurations: 0. p. Ip. AR.33)), AR((.5), where AR(r) represents the ruto-regressive covariance structure (A), = - for
The computational cost per EM iteration is OnnKpg + n n33g² + KTTpqq + n n2²²), where T denotes the num- ber of iterations to solve the penalized optimization prob- lem (6). Given that q a small number that does not grow with n. P, K, the overall complexity of Algorithm 1 is (log(n)(nK3) + KTnp + Knp?)) with the dominated term (log(n) (n) K Knp²) from covariance estimation. This re-
i = I....,
Table 1 summarizes clustering error rates. As expected, mixPFC achieves substantially lower error rates than all subspace clustering methods across most model settings. When K = 2, mixPFC has error rates of around 10%
mains tractable even for large K or p.
7

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
across all settings, dropping below 5% in certain cases. Notably, mixPFC-ISO demonstrates superior performance with even lower enror rates for model M2. For M2 with A = 0.1Ip, subspace clustering methods exhibit compara- ble or slightly lower error rates. This is likely due to favor- able conditions for subspace clustering methods, where the subspaces are orthogonal and the random errors are mini- mal. When K 2. error rates remain impressively low, under 3% for K 3, probably due enhanced signal strength from setting rise to around 15,, likely due to the increased difficulty in generating high-quality initial values for larger K.
expected to be heterogeneous.
The dataset is randomly partitioned into 80% training and 20% testing samples, with 100 repetitions. Table 2 reports the prediction mean squared errors (PMSE) and the number of selected variables @ for each method, with the number of clusters set to and 5 for Nutlin-3 and AZD6244 when using mixPFC. For both compounds, mixPFC significantly reduces prediction error compared to homogeneous meth- ods, suggesting heterogeneity in the data. Notably, mixPFC does not select more variables than Lasso and the three ho- mogencous SDR methods.
10. When K = 5, error rates
Figure 4 shows summary plots of the response against re- duced predictors projected onto each subspace for Nutlin, Within each cluster, the response exhibits approximately linear relationships with the projected predictors. The lack of clear patterns when points are projected onto subspaces outside their cluster further highlights the heterogeneity of data. The plot for AZD6244 and additional real data anal- ysis are provided in Section A in the appendix.
5.2. Real Data Analysis
The Australian Institute of Sport (AIS) dataset, available in the R package dr, contains lean body mass data for 102 male and 100 female athletes. The objective is to investi- gate the relationship between lean body mass and predic- tors, including height, weight, and red cell count. Given that body composition varies between males and females (Bredella, 2017), the AIS data likely includes two distinct subpopulations. Figure 3 (a) shows summary plots for males and females when sex is observed, highlighting dis- tinct fitted lines for each group. Figure 3 (b) demonstrates that mixPFC effectively identifies the two subpopulations, achieving an error rate of 0.074.
Figure 4. The scatter-plot the response Y versus x for the drug Nutlin. The solid line is fitted using samples in the given cluster.
6. Discussion
(a) PFC with true sex
(b) mixPFC with estimated sex
In this work, we proposed mixture of PFC model, which combines subspace clustering with SDR methods to han- dle heterogeneous and high-dimensional data. An efficient group Lasso penalized EM algorithm has been developed to simultaneously perform clustering, subspace estimation, and variable selection. Theoretical analysis revealed an encouraging non-asymptotic convergence rate, offering in- sight into the empirical success of the algorithm.
Figure 3. Summary plots for the AIS dataset. (a) Fitted lines for males and females when the variable is observed, illustrate distinct subpopulation trends. (b) Results from mixPFC, which accurately identifies the two subpopulations with an error rate of 0.074.
The Cancer Cell Line Encyclopedia (CCLE) dataset con- tains 8-point dose-response curves for 24 chemical com- pounds across over 400 cell lines, with 18.926 gene ex- pression features for each cell line, accessible at https: //sites. cetoadinntiiin wwtssstitttterrrccllloooeo org/ccle. Due to in- consistencies in cell lines across compounds, we focus on two popular cancer treatments: Nutlin-3 (n 480) and AZD6244 (n 479). Following (Wang al., 2024; Li et al., 2019), we use the logarithm of the area under the dose-response curve as the response, representing drug sen- sitivity. The top p- = 500 genes with the highest absolute correlations with the responses are selected for analysis. Given the inherent complexity of cancer, the CCLE data is
A key aspect of our theoretical framework is its devel- opment for K = 2 cluster scenario. Generalizing theo- ries for multi-cluster EM algorithms remains an important yet challenging direction. Recent advances focus on low- dimensional settings (Yan et al., 2017: Tian et al., 2024), and to the best of our knowledge, no general theory exists for high-dimensional multi-cluster EM. Addressing this open question likely requires fundamentally new tools to handle complex parameter spaces and interactions between K subspaces.
Further discussion and potential extensions are provided in
8

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Table 1. Averages and standard errors of clustering error rates with 200K,, p 1000.
mixPFC-ISO
M
RANSAC
LSA
LRSC
SSC
GPCA
K-means
K-means X XoY
hclust
hclust
MI: K 2, A
0.1I,,
5.0
3.4 (0.7)
48 (0.1)
44.6 (0.2)
47.8 (0.2)
47.9 (0.1)
47.9 (0.1)
44.2 (0:2)
20.2 (0.3)
45.8 (0.2)
14 (0.2)
(1.0
(1.1)
48.1 (0.2)
47.9 (0.2)
48 (0.2)
48 (0.1)
48.1 (0.1)
45.2 (0:2)
29.1 (0.4)
48.1 (0.2)
38.2 (0.5)
6.0 (0.6)
48.0 (0.2)
47.6 (0.2)
48.0 (0.2)
(019
(0.1)
44.7 (0.1)
23.1 (0.3)
47 (0.2)
23,9 (0.3)
4.7 (0.4)
47.9 (0.2)
47.0 (0.2)
47.8 (0.2)
(8.1
(0.1)
44.5 (0.2)
(0.3)
46.6 (0.2)
18.3 (0.3)
M2: K =
0.11, p.
AR(03))
AR(0.5)
(1.5)
16.4 (2.0)
28.3 (1.2)
(0.1)
48.1
(1.2
(0.1)
(0.2)
(0.3)
46,9 (0.2)
19.1 (0.3)
13.9 (0.8)
16.3
47.6 (0.2)
(0.3)
47.7
(.2)
(0.1)
47.5 (0.2)
36.5 (0.3)
47.4 (0.2)
46.2 (0.3)
11.5 (0.9)
46.2 (0.3)
(0.4)
(0.1
(4.9
48.2 (0.1)
37 (0.4)
32.2
44.2 (0.3)
42.6 (0.5)
11.4 (0.9)
43.4 (0.6)
(0.3)
(8.0
(3.4
48.0 (0.2)
33,9 (0.2)
31.1 (0.3)
40.2 (0.3)
34,9 (0.5)
M3:
AR(00,, AR(03)) AR(0.5)
(L.))
3.6 (1.2)
39.7 (0.7)
(0.8)
(0.2)
(3.2)
47.8 (0.2)
44.6
(0.3
44.5 (0.4)
15.1 (0.2)
(1.1
(1.4)
47.8 (0.2)
(0.2
(0.0
47.9 (8.0
45.9 (0.2)
(0.6)
47.9 (0.1)
41.2 (0.4)
(1.4)
(0.2)
47.8 (0.)
(7.5
47.9
(O.I)
(0.3)
47.9 (0.1)
26.3 (0.3)
(0.9)
(0.1) (0.2)
(0.1)
(7.7
47.8 (0.2)
46.7 (0.1)
24.1 (0.3)
47.7 (0.2)
21.1 (0.3)
M4: 3,
AR(0.3)
AR(0.5)
(
(
(1)
(0.5)
18.2
6.5)
(3.3)
(0.2)
(2.2)
(0.4)
29.4 (0.3)
(0.1)
(.1))
(0.2)
(0.2
(..0
(0.5)
(3.4
(0.2)
(3.2)
68.8 (0.3)
(0.4)
(011)
59.8 (0.3)
(0.3)
(..0
(0.3
(3.4
61.8 (0.2)
50.1 (0.3)
(0.3)
(0.4)
2.2 (0.1)
56,9 (0.3)
(0.2)
(0.1
(9.6
63.7 (0.1)
62.8 (0.2)
(0.3)
(0.3)
42.3 (0.5)
M4: = 3,A=
(..IT,11 AR(033), AR(0.5)
(0.3)
0.2 (0.2)
50.1 (0.7)
(0.6)
(2.6
13.8 (0.2)
74.1 (0.2)
63.4 (0.2)
(0.4)
64.8 (0.2)
25.5 (0.2)
(1.2)
(1.6)
75.2 (0.1)
66.7 (0.2)
76.9 (0.1)
75.3 (0.1)
76 (0.1)
65.7 (0.1)
(0.4)
(0.6)
71.5 (0.3)
15.7 (2.0)
73.6 (0.2)
(0.2)
76.8 (0.1)
(0.1)
76 (0.1)
62.7 (0.1)
53.7 (0.3)
72.8 (0.1)
66.1 (0.3)
(2.2)
72.4 (0.2)
(0.2)
77 (0.1)
(0.1)
76.1 (0.1)
63.1 (0.2)
55.4 (0.2)
71.1 (0.1)
62 (0.3)
Electronic Journal of Probability, 13:1000 1034, 2008.
Table The averages the prediction errors, the sparsity level 8, and the corresponding standard errors based on 100 replicates.
Agarwal, P. K. and Mustafa, N. H. k-means projective clustering. In Proceedings of the Twenty-Third ACM Symposium Principles of Database Systems, PODS '04, pp. 155-165, New York, NY, USA, 2004. Association for Computing Machinery.
Nutlin-3
mixPEC
SEAS-SIR
SEAS-PFC
LassoSIR (0.3)
Lasso
PMSE x00
8.9(0.4)
18.7 (0.4)
18.8 (0.4)
(0.3)
31.5(2.5)
48.3
33.3 (1.8)
(2.)
39.1 (1.1)
AZD624
PMSE x100
(1.7)
108.8 (2.2) 107.1 (2.2)
83.2
(1.6(
77.8
(1.5
66.8 (0.7)
(0.7)
Baik, J. and Silverstein, J. W. Eigenvalues of large sample covariance matrices of spiked population models. Jour- nal of multivariate analysis, 97(6): 1182-138824881 2006.
Section G in the appendix.
Balakrishnan, S., Wainwright, M. J., and Yu, B. Statisti- cal guarantees for the em algorithm: From population to sample-based analysis. The Annals of Statistics, 45(1):
Acknowledgements
77-120, 2017.
The authors thank reviewers for constructive comments. The research was partly supported by grant RMS-2053697 from the U.S. National Science Foundation.
Banijamali. E., Karimi, A.-H., and Ghodsi, Deep varia- tional sufficient dimensionality reduction. arXiv preprint arXiv:1812.07666 2018.
Impact Statement
Biernacki, C., Celeux, G.. and Govaert, G. Choosing start- ing values for the EM algorithm for getting the high- est likelihood in multivariate Gaussian mixture models. Computational Statistics & Data Analysis, 41 41(3-4):561- 575, 2003.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential socictal consequences of our work, none which we feel must specifically highlighted here.
Bredella, M. A. Sex Differences in Body Composition, pp. 27. Springer International Publishing, 2017.
References
Adamczak, R. A tail inequality for suprema of unbounded empirical processes with applications to Markov chains.
Cai, J., Fan, J., Guo, W., Wang, S., Zhang, Y., and Zhang, Z. Efficient deep embedded subspace clustering. In 2022

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 21-30, 2022.
Hinton, G. E. and Salakhutdinov, R. R. Reducing the di- mensionality of data with neural networks. Science, 313 (5786):504-577, 2006.
Cai, T. T., Ma, J., and Zhang, L. Chime: Clustering of high- -igh-dimensional Gaussian mixtures with EM algorithm and its optimality. The Annals of Statistics, 47(3):1234- 1267, 2019.
Hsing, T. and Ren, H. An RKHS formulation of the in- verse regression dimension-reduction problem. Annals of statistics, 37(2):726-755, 2009.
Chen, X., Zou, C., and Cook, R. D. Coordinate- independent sparse sufficient dimension reduction and variable selection. The Annals of Statistics, 38(6):3696- 3723, 2010.
Huang. J., Jiao, Y. Liao, X., Liu, J., and Yu, Z. Deep dimension reduction for supervised representation learn- ing. IEEE Transactions on Information Theory, 70(5): 3583-3598, 2024.
Chen, Y., Jiao, Y., Qiu, R., and Yu, Deep nonlinear sufficient dimension reduction. The Annals of Statistics, 12(3):1201-1226, 2024.
Ji, P., Zhang, T., Li, H., Salzmann, M., and Reid, I. Deep subspace clustering networks. Proceedings of the 31st International Conference on Neural Information Pro- cessing Systems, NIPS' 17, pp. 23-32, Red Hook, NY, USA, 2017. Curran Associates Inc.
Chiaromonte, F., Cook, R., and Li, B. Sufficient dimen- sions reduction in regressions with categorical predic- tors. The Annals of Statistics, 30(2):475- - 497, 2002.
Journée, M., Nesterov, Y., Richtárik, P., and Sepulchre, R. Generalized power method for sparse principal compo- nent analysis. Journal of Machine Learning Research, 11:517-553, 2010.
Cook, R. D. and Forzani, L. Principal fitted components for dimension reduction in regression. Statistical Science, 2344):485-501, 2008.
d' Aspremont, A., Ghaoui, L. E., Jordan, M. I., and Lanck- riet, G. R. G. direct formulation for sparse PCA using semidefinite programming. SIAM Review, 49(3):434- 448, 2007.
Kanatani, K. Motion segmentation by subspace separation and model selection. In Proceedings Eighth IEEE Inter- national Conference on Computer Vision. ICCV 2001, volume 2, pp. 586-591 vol.2, 2001.
Dirksen, S. Tail bounds via generic chaining. Electronic Journal of Probability, 20(53):1-29, 2015.
Kapla, D., Fertl, L., and Bura, E. Fusing sufficient di- mension reduction with neural networks. Computational Statistics & Data Analysis, 168:107390, 2022.
Elhamifar, E. and Vidal, Sparse subspace clustering: Al- gorithm, theory, and applications. IEEE transactions pattern analysis and machine intelligence, 35(11):2765- 2013. 2013.
Klusowski, M., Yang, D., and Brinda, W. Estimating the coefficients of mixture of two linear regressions by expectation maximization. IEEE Transactions Infor- mation Theory, 65(6))3515 01133511-5224 2019.
Erichson, N. B., Zheng, P., Manohar, K., Brunton, S. L.,
Kutz, J. N., and Aravkin, A. Y. Sparse principal compo- nent analysis via variable projection. SIAM Journal on Applied Mathematics, 80(2):977-0002, 2020.
Kwon, J., Qian, W., Caramanis, C., Chen, Y., and Davis, D. Global convergence of the em algorithm for mix- tures of two component linear regression. In Beygelz- imer, A. and Hsu, D. (eds.), Proceedings of the Thirty- Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pp. 2055- 2110. PMLR, 25-28 Jun 2019.
Favaro, P., Vidal, R., and Ravichandran, A. A closed form solution to robust subspace estimation and clustering. In CVPR 2011. pp. 1801-1801, 2011.
Forgy, E. W. Cluster analysis of multivariate data: effi- ciency versus interpretability of classifications. Biomet- rics, 21:768-7699 1965.
Kwon, O.-R., Mukherjee, G., and Bien, J. Semi-supervised learning of noisy mixture of experts models. arXiv preprint arXiv:2410.09033 2024.
Fraley, C. and Raftery, A. E. Model-based clustering, dis- criminant analysis, and density estimation. Journal of the American statistical Association, 97(458):611663 2002.
Le, L., Patterson, A., and White, M. Supervised autoen- coders: Improving generalization performance with un- supervised regularizers. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Bisa-Bianchi, N., and Gar- nett, R. (eds.), Advances Neural Information Process- ing Systems, volume 31. Curran Associates, Inc., 2018
Hastie, T. and Tibshirani, R. Discriminant analysis by Gaussian mixtures. Journal of the Royal Statistical So- ciery Series B: Statistical Methodology, 58(1):155-176. 1996.
10

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Ledoux, M. and Talagrand, M. Probability in Ba- nach Spaces: Isoperimetry and Processes, volume 23. Springer Science & Business Media, 1991.
Paul, D. Asymptotics of sample eigenstructure for large dimensional spiked covariance model. Stafistica Sinica, 17(4):1117-1422 2007.
Rudelson, M. and Zhou, S.
Reconstruction from anisotropic random measurements. In Mannor, S., Sre- bro, N., and Williamson, R. C. (eds.), Proceedings of the 25th Annual Conference on Learning Theory, volume 23 of Proceedings of Machine Learning Research, pp. 10.1- 10.24, Edinburgh, Scotland, 25-27 Jun 2012. PMLR.
Li, B., Artemiou, A., and Li, L. Principal support vector machines for linear and nonlinear sufficient dimension reduction. Annals of Statistics, (9(6))3182-3210 2011.
Li, K.-C. Sliced inverse regression for dimension reduc- tion. Journal of the American Stutistical Association, 86 (414):316-32) 337777 1991.
Shao, X. and Zhang. J. Martingale difference correlation and its use in high-dimensional variable screening. Jour- nal of the American Statistical Association, 109(507): 1 1302-1318, 2014.
Li, L., Ke, C.. Yin, X., and Yu, Z. Generalized martin- gale difference divergence: Detecting conditional mean independence with applications in variable screening. Computational Statistics & Data Analysis, 180:107618, 2023.
Sheng, W. and Yin, X. Sufficient dimension reduction via distance covariance. Journal of Computational and Graphical Statistics, 25(1):91-104, 2016.
Li, Q- Shi, R., and Liang, F. Drug sensitivity prediction with high-dimensional mixture regression. PLOS ONE, 14(2):118, 02 2019.
Soltanolkotabi, M. and Candés, E. J. A geometric analy- sis of subspace clustering with outliers. The Annals of Statistics, 40(4):2195-2338 2012.
Li, R., Zhong, W., and Zhu, L. Feature screening via dis- tance correlation learning. Journal of the American Sta- ristical Association, 107(499): ([1499):1129-111 11011119-1111 113991119,21111111 2012.
Székely, G. J., Rizzo, M. L.,, and Bakirov, N. K. Measuring and testing dependence by correlation of distances. The Annals of Statistics, 35(6):2769 2794, 2007.
Liang, S., Sun, Y., and Liang F. Nonlinear sufficient dimension reduction with stochastic neural network. In Proceedings the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc.
Talagrand, M.
The generic chaining: upper and lower bounds of stochastic processes. Springer Science & Business Media, 2005.
Tan, K., Shi, L., and Yu, Z. Sparse SIR: Optimal rates and adaptive estimation. The Annals of Statistics, 48(1):64- 85, 2020.
Lin, Q.. Zhao, Z., and Liu, S. On consistency and sparsity for sliced inverse regression in high dimensions. The Annals of Statistics, 46(2):580-610, 2018.
Tan, K. M., Wang, Z., Zhang, T., Liu, H., and Cook, R. D. A convex formulation for high-dimensional sparse sliced inverse regression. Biometrika, 105(4):76--78, 2018.
Lin, Q., Zhao, Z., and Liu, J. S. Sparse sliced inverse re- gression via lasso. Journal of the American Statisticul Association, 114652)::126-11739 2019.
Tian, Y., Weng, H., and Feng. Y. Towards the theory of unsupervised federated learning: non-asymptotic anal- ysis of federated EM algorithms. In Proceedings of the 41st International Conference Machine Learning. ICML' '2. JMLR.org, 2024.
MacQueen, J. et al. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, volume 1, pp. 281-297. University of California Press, 1967.
Tibshirani, R., Walther, G., and Hastie, T. Estimating the number of clusters a data set via the gap statistic. Jour- nal of the Royal Statistical Society: Series B (Statistical Methodology), 63(2):41 1-23, 2001.
Mai, Q., Yang, Y., and Zou, H. Multiclass sparse discrimi- nant analysis. Statistica Sinica, 29(1)997-111, 2019.
Tipping, M. E. and Bishop, C. M. Probabilistic principal component analysis. Journal the Royal Statistical So- ciety Series B: Statistical Methodology, 61(3):611-622, 1999.
McLachlan, G. J., Lee, S. X., and Rathnayake, S. I. Fi- nite mixture models. Annual review of statistics and its application, 6:355-378, 2019.
Tron, R. and Vidal, R. A benchmark for the comparison of 3-d motion segmentation algorithms. In 2007 IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 1-8, 2007.
Meinshausen, N. and Yu, B. Lasso-type recovery of sparse representations for high-ding figh-dimensional data. The Annals of Statistics, 37(1):246 270, 2009.

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Vershynin, R. Introduction the non-asymptotic analy- sis of random matrices. arXiv preprint 101 2010.
Yi, X. and Caramanis, C. Regularized EM algorithms: unified framework and statistical guarantees. In Pro- ceedings of the 29th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, pp. 1567-1575, Cambridge, MA, USA, 2015. MIT Press.
Vershynin, R. Highh-iimensionl probability: An introduc- tion with applications in data science, volume 47. Cam- bridge university press, 2018.
Yi, X., Caramanis, C.,, and Sanghavi, S. Alternating mini- mization for mixed linear regression. In Xing, E. P. and Jebara, T. (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Pro- ceedings of Machine Learning Research, pp. 613-621, Bejing, China, 22 24 Jun 2014. PMLR.
Vidal, R. Subspace clustering. IEEE Signal Processing Magazine, 28(2):52-68, 2011.
Vidal, R., Ma. Y., and Sastry, S. Generalized principal com- ponent analysis (GPCA). IEEE transactions on pattern analysis and machine intelligence, 17(12))945-1195, 2005.
Yu, Y. Wang, T., and Samworth, R. J. A useful variant theorem for statisticians. Biometrika,
of the 102(2)315-322) 2015.
Wang, N., Zhang, X., and Mai, Q. Statistical analysis for penalized EM algorithm in high-dimensional mixture linear regression model. Journal of Machine Learning Research, 25(222): 2024.
Yuan, M. and Lin, Y. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68 (1):49-67, 2006.
Wang, Z., Gu, Q. Ning, Y., and Liu, H. High dimensional EM algorithm: statistical optimization and asymptotic normality. In Proceedings of the 29th International Con- ference on Neural Information Processing Systems- Vol- ume 2, NIPS' 15, pp. 2521-252222 Cambridge, MA, USA, 2015. MIT Press.
Zeng. J., Mai, Q., and Zhang, X. Subspace estimation with automatic dimension and variable selection in sufficient dimension reduction. Journal of the American Statistical Association, 1 119(545):343-33 355, 2024.
Witten, D. M.. Tibshirani, R., and Hastie, T. penalized matrix decomposition, with applications to sparse princi- pal components and canonical correlation analysis. Bio- statistics, 10(3):515-533 2009.
Zhang, L., Ma, R., Cai, T. T., and Li, H. Estimation, confidence intervals, and large-scale hypotheses testing for high-dimensional mixed linear regression. arXiv preprint arXiv:2011.03:: 1.03598, 2020.
Wu, H.-M. Kernel sliced inverse regression with appli- cations to classification. Journal of Computational and Graphical Statistics, 17(3):590--6) 610, 2008
Zhu, R., Wang, L., Zhai, C., and Gu, Q. High-dimensional auriance-reduced stochastic gradient expectation- maxinization algorithm. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 4180-4188. PMLR, 06-11 Aug 2017.
Xu, J., Hsu, D., and Maleki, A. Global analysis of ex- pectation maximization for mixtures of two gaussians. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS'16, 16, pp. 2684-2692, Red Hook, NY, USA, 2016. Curran Asso- ciates Inc. ISBN 9781510838819.
Zong, B., Song, Q., Min, M. R., Cheng, W., Lumezanu, C., ki Cho, D., and Chen, H. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International Conference Learning Representations, 2018.
Yan, B., Yin, M., and Sarkar, P. Convergence of gradi- ent em on multi-componen mixture of gaussians. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fer- gus, R., Vishwanathan, S., and Garnett, R. (eds.), Ad- vances in Neural Information Processing Systems, vol- ume 30. Curran Associates, Inc., 2017.
Zou, H., Hastic, T., and Tibshirani, R. Sparse princi- pal component analysis. Journal of computational and graphical statistics, 15(2):265-286, 2006.
Yan, J. and Pollefeys, M. A general framework for tion segmentation: Independent, articulated, rigid, non- rigid, degenerate and non-degenerate. In Leonardis, A., Bischof, H., and Pinz, A. (eds.), Computer Vision ECCV 2006, pp. 94-106, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
12

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
This appendix provides additional numerical results, implementation details, and technical proofs supporting the theoretical analysis. Section A presents detailed results from the numerical studies. Section B outlines implementation specifics, including the initialization method, selection of the number of clusters, and the miXPFC-ISO algorithm. The theoretical analysis of Theorem 4.1 relies on two key lemmas, with ancillary results provided in Section F. The proofs of the two key lemmas are detailed in Sections and E, while Section C contains the full proof of the theorem. We discuss potential extensions in Section G.
A. Additional Numerical Results
A.1. Simulations
The parameters for M1-M4 in the simulation section are generated follows.
(MI) The non-zero coefficients of 31 are B1g = =2.3, i 1,... ....... 6, and then set B3 = -B
(M2) The non-zero coefficients of B1 and B2 are B 2.3, S i S 6, and B2. 2.3 for i 1,3,5, B2: 2.3 for
= 2, 4, 6.
(M3) The basis matrices 31 and B2 have two columns. The non-zero rows of 31 and B2 are
2.3
2.3
2.3
2.3
2.3
2.3
(BI)T.6
2.3
223
2.3
-23
2.3
-2.
2.3
-23
-23
223
-23
-2.3
(32)
2.3
2.3
-23
2.3
2.3
2.3
(M4) The basis matrices Pwr w = 1, 1...., K have two columns. For each cluster w, the non-zero elements are generated as (Bw)is ~ Unif([-2.5, 2.1] U
< 10, = 1,2, w = 1,2...., K.
We consider four scenarios of simulations described in Table A.3. The clustering error rates for S1 and S4 are presented in the main paper. The error rates for scenarios S2 and S3 are summarized in Table A.4&A.5.
To assess the subspace estimation and variable selection accuracy, we define the following criteria: the distance be- tween estimated and actual subspaces is defined as D.,. = D(S Sp.) = PP... Pa.. / // w = 1,..., K; error rate ER is the fraction of incorrectly classified samples: true positive rate (TPR) and false positive rate (FPR) When K 2, instead of reporting defined as TPR, = |Aw [AwnAAl A.wl///Aw and TPR AAw (AA5)//AA Et= D K, average TPR and FPR each D... TPR., and FPR,,, we calculate the average subspace distance >" mm FPR,, /K, and TPR and FPR of the union of selected variables across all and FPR = (w. AUA4))/U
K TPR > m TPR KK and FPR
clusters TPR- =
Table A.6&A.7 summarize variable selection and subspace estimation results for scenario S1&S4 with covariance A and AR(3.3). The corresponding results for A 0.1I and AR(0.5) are provided in Table A.S&A.9. Results for S2 and S3 can be found in Table A.10-A.11 and Table 1.12-A.13, respectively. Under scenario S1, mixPFC demonstrates strong performance, effectively identifying the important variables and accurately estimating central subspaces. It achieves true positive rates TPR,, greater than 95%, false positive rates FPR,,. below 1%, and subspace estimation errors D., around 0.3 when A and AR(0.3). However, as variables correlation increases (A = AR((.5)), mixPFC shows reduced accuracy in subspace estimation with Dw, increasing to approximately 0.5, though it remains effective in variable selection. When the covariance matrix contains small elements (A = 0. 0.1I), mixPFC performance declines both variable selection and subspace estimation. This reduction occurs because estimating S. A -r becomes challenging when all elements of A have small magnitude. In such cases, mixPFC-ISO, which assumes isotropic errors, outperforms mixPFC when A 0.11 and I.
In the unbalanced clusters scenario S2, subspace estimation errors are smaller for clusters with more samples, while variable selection and error rates remain consistent with S1. In scenario $3, featuring a nonlinear fitting function, both mixPFC and mixPFC-ISO perform poorly in variable selection and subspace estimation for model M3. Even SEAS-PFC using true clusters offers little improvement. However, the error rates remain well controlled. The performance under M1 and M2
13

---

Heterogencous Sufficient Dimension Reduction and Subspace Clustering
similar S1. In the multi-cluster setting S4, the mixPFC shows 10% 20% reduction in TPR for each cluster compared to S1. But, the union of selected variables typically includes all the important variables.
When compared with LassoSIR, SEAS-SIL SIR, and SEAS-PFC fitted on estimated clusters, mixPFC has similar TPR,, and FPR but larger subspace estimation errors. This is likely because the algorithm incorporates some misleading information from other clusters through "Nie- Among the high-dimensional SDR methods, SEAS-SIR and SEAS-PFC consistently outperformed LassoSIR. Therefore, recommended to refit with SEAS-PFC after classifying the data using mixPFC. Notably, fitting with true clusters offers only marginal improvements typically just a few percentage points points-over over using clusters estimated by mixPFC.
Table A.3. Parameters seltings four sets simulations. linking matrices 7/ = 013) for M2, and 7 for M4.
Scenario
Model
π
Y
f(Y)
S1
M1-M3
(0.5, 0.5)
1nif(-1,1)
(Y, (Y))
M1-M3
(0.7,0.3)
Unif(-1,1)
(Y,1
S3
M1-M3
(0.5, 0.5)
Unif(0,2)
(Y3/3, Y /6)TT)
M4 K 3,5 (1/K, (1.K...... 1/K)
Unif(-1,1)
(Y))")))
Table A.4. Averages and standard errors clustcring error scenario S2 with n 400, p 1000, K
Method
mixPFC-ISO
mixPFC
RANSAC
ISA
LRRCC
SSC
GPCA
K meaas
K-means Y
helust X
helust
MI: a
100.p
1000, A 0.11
AR(03)).
AR(0.5)
ER(%)
7.7 (L.3)
(0.7)
(0.5)
47.3 (7.4 (0.2)
(0.1
33.1
(0.3)
10.3
(1.3)
7.2 (0.6)
48 (0.2)
41.2 (0.4)
(0.4)
(0.9
47.4 (0.2)
(0.1
42.7
43 (0.5)
27.8 (0.9)
6.9 (0.9)
46.4 (0.3)
44.5 (0.3)
(0.1
47.7 (0.2)
45 (0.4)
48.5
(0.7)
46.5 (0.3)
18.8 (0.8)
(0.5)
43.8 (0.4)
45.5 (0.3)
(0.1)
48 (0.1)
44.9 44.9 (0.4)
(0.1)
35.2 (0.6)
45.9 (0.3)
13.5 (0.5)
400,p
1000,
1.11,
AR(0.)
ER(S)
12.9
14.4
29.9
(..1).
47.6
(1.2
(0.2)
(0.3
41.7 (0.3)
45.3
13.9 (0.6)
13.2 (0.5)
14.4 (1.3)
47.4 (0.2)
40.6 (0.5)
(0.5)
(0.2)
(0.9
(0.1
45.9 (0.6)
40.8 (0.6)
30.7 (0.4)
10 (0:7)
44.8 (0.4)
33.6 (0.3)
(8.1
46.1 (0.3)
44.2 (0.4)
(0.2)
45.6 (0.2)
47.2 (0.2)
29.9 (0.7)
9.4 (0.8)
40.8 (0.7)
24.4 (0.2)
(0.2)
46.2 (0.3)
44 (0.4)
(0.2
44.4 (0.3)
(0.2)
31.8 (1.2)
400.p
1000, A
AR(0.3), AR(0.5)
(R(S)
(1.2)
(0.9)
(0.8)
46.1
46.8
(0.5)
(0.))
(0.2)
(0.7
11.7
10.9
(0.7)
47.9 (0.2)
45.8 (0.3)
(0.3)
(0.6
46,9 (0.2)
46.8
(0.7
41.8 (0.6)
29.7 (0.8)
(1)
(0.3)
46.9 (0.2)
(0.1
47.7 (0.2)
44 (0.4)
48 (0.1)
(0.5)
46.7 (0.2)
(0.8)
4.8 (1)
42.4 (0.4)
46.6 (0.2)
48.2 (0.2)
48.1 (0.2)
44,4 (0.4)
48.2
(8.4
46.6 (0.2)
14.7 (0.5)
Table A.5. Averages and standard errors of clustering error rates for scenario S3 with n = 400, 1000, K = =2.
Method
mixPFC-ISO
mixPFC
RANSAC
LSA
LRSC
SSC
GPCA
K-means X
K-means X Y
belust X
helust X
MI: 100,
1000,
A 0.11
AR(0.3),
AR(0.5)
ER(%)
6.1 (0.8)
(0.1)
47.7 (0.2)
42.7 (0.2)
48.3 (0.1)
47.9 (0.2)
(0.2)
(9.9) (0.2)
39.6
42.5 (0.3)
35.8 (0.2)
14.2 (0.5)
(0.8)
(0.2)
47.8 (0.2)
47.9
47.9 (0.2)
(0.9)
44 (0.2)
43.6
47.9 (0.2)
45.4 (0.3)
(0.5)
(0.1)
(0.2)
48.2
(0.2)
(0.2)
(0.2)
(0.2)
47.6 (0.2)
(0.3)
7.8 (0.6)
48
44.3
48.1
(0.1)
(8.2
40.6 (0.1)
39.6
46.3 (0.2)
37.7 (0.2)
M2: 400,
1000,
& 0.11, A,. AR(0.).
ER(S))
6.9 (0.8)
11.1 (1.6)
40 (0.8)
18.7 (1.2)
48.1 (0.2)
16.9 (0.2)
48.2 (0.1)
46.7 (0.2)
40.4 (0.2)
45.3 (0.4)
38.1 (0.2)
21.7 (0.7)
19.9 (0.9)
48 (0.2)
47.9 (0.2)
48 (0.1)
47.9 (0.2)
47.8 (0.2)
47.7 (0.2)
45.2 (0.3)
48 (0.2)
47.2 (0.2)
17.6 (0.8)
47.5 (0.2)
42 (0.2)
47.9 (0.2)
(0.6.
(0.1)
46.3 (0.3)
40.8 (0.2)
47.5 (0.2)
(0.2)
38.4
(0.8
(7.9
(1.4
(9.6
44.3 (0.3)
M3: 100, 1000,
A 0.1I, I,, AR(0.3).
AR(0.5)
ER(%)
17.9 (1.8)
22.6
33.1(11)
11. (0.2)
47.6
12.6 (0.2)
(0.2)
39.9 (0.1)
39.8 (0.2)
39.7 (0.3)
35.2 (0.2)
20 (1.2)
(1.1)
47.6 (0.2)
46.5 (0.2)
47.9 (0.2)
47.3 (0.2)
47.8 (0.2)
42.9 (0.2)
42.1 (0.3)
47.7 (0.2)
43.9 (0.3)
(3.1
47.7 (0.2)
45.5 (0.2)
48.1 (0.1)
47.5 (0.2)
48.1 (0.2)
(0.2)
40.2 (0.3)
47.5 (0.2)
42.6 (2.3
(1.1
(0.3
(0.2)
(0.2)
(0.2)
(0.2)
(0.2)
(0.2)
47.4 (0.2)
(0.3)
14

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Table A.6. Simulations results for scenario SI with v. 400,, p- 1000, K FPR, and subspace distance for each cluster.
table reports averages standard errors TPR,
= AR(0.3)
Method
(%)
(P)...())
Dw 100
TPR,(%))
FPR..(%)
Du. 100
MI
mixPFC
95, 96.7 (1.7, 1.4)
1.6 (0.6,,
30, 29.49.422 (2.2,
98.7, 98.3 (0.7,
0.2, (0.2, 0.3)
35, (1.4, 1.4)
miXPFC-ISO
100, 100 (0, 0)
0.4, (0,00
(0. 0.2)
LassoSIR'
100, 100 (0, 0)
1.5, 1.7 (..1.
30.6,
99.5, 99.7 (0.3, 0.2)
1.3, 0.11
41.1 (1.2, 1.1)
LassoSIR
94.7, (1.8, 1.5)
204.
(2.1,
98.5, 97.7 (0.8, 1.1)
1.2, (0.2,
41.2, (1.5, 1.4)
SEAS-SIR'
100, 100 (0, 0)
0.5.0.6 (0.1...)
7.5,7.5 (0.4,
100, 100 (0,
0.1,0.1
13.9. 14.5 (0.3, 0.4)
SEAS-SIR""
99.1, 99.5 (0.5, 0.4)
0.7. (.21.
10.7, 10.6 (1.3.
99.8, 99.3 (0.2, 0.5)
(0..0.))
15, 15.8 (0.6,
SEAS-PPC
100, 100 (0, 0)
1.2, 1.4 (0.2,
8, (0.4. 0.4)
100, 100 (0, 0)
(2200.2
13.5, 14.2 (0.3, 0.4)
SEAS-PFC"
96.7, 96.8 (1.5, 1.4)
1.2, 1.6 (0.4, 0.3)
14.9, 13.9 (2.4, 2)
99.8, 99.7 (0.2, 0.3)
0.3, 0.5 (0.1..
15.1, 15.7 (I, 1.2)
M2
mixPFC
94.5 (1.4, 1.5)
0.8 0.11
31.7. 1.9)
98.8, 100 (0.5,
0, (0.00.)
37.3, (1.4, 0.9)
mixPFC-ISO
98.7, 98.8 (0.8, 0.6)
0.5. 0.8 (..).
9.2, (1.3, 1.6)
LassoSTR
100, 100 (0, 0)
1.5, 1.3 (..
30.6, 29.8 1)
99.5, 100 (0.3, 0)
(0.1 0.2)
41.2, 29.9 (1.2, I)
"assoSIR"
91.7 (1.7. 1.9)
1.8 (0.2
97, 99 (1.1, 0.5)
2...24
43,5, 32.1 (1.5, (15,111 1.3)
SEAS-SIR'
100, 100 (0. 0)
0.5. 0.6 (0.1.0.1)
7.5 (0.4.
100. 100 (0. 0)
1.1
139. 30.9 (0.3.0.5)
SEAS-STR
92.7 (1.6, 1.7)
0.4, 0.5 (0.1,.)))
19.4, 19.2 (24,
98.2, 97.3 (0.8, I.I)
0.1, 0.
18. (1.3,
SEAS-PFC
100, 100 (0, 0)
1.2, 1.3 (.22,
8, 8.1 0..4,
100, 100 (0, 0)
0.2,0.9
13.5. 28.4 (0.3, 0.5)
SEAS-PPC
92.8, (1.7, 1.7)
L.1. 0.9 0.2
19.6, 19.6 (2.4.
97.5, 97.7 (1.1.
0.2, (0,00)
17.7, 30,9 (1.4, 1.1)
M3
mixPFC
94.3
0.2, (0.1.0.1)
29.5, (2.2, 2.1)
96.8, 98.5 (0.9, 0.6)
0.3, (0.2, 0.1)
30, 35.7 (1.7, 1.4)
mixPFC-ISO
98.8, 97.2 (1, 1.6)
0.7, 0.7 0.21.
21.4, 22.1 (1.7.)
LassoSIR'
100, 100 (0, 0)
6.6, 6.6 0.2).
50,1 (0.8,
99.8, 100 (0.2, 0)
9.8. 8.4 (0.2, 0.2)
69.3, 68.5 (0.5, 0.5)
1 LassoSTR" m
94.8, (1.5, 1.2)
7.6, 7.6,7.2 0.3,
9.8,9.6 (0.3, 1.5)
98.3, 98.7 (0.6, 0.6)
9.3, (0.2, 0.2)
71, 70.3 (0.7, 0.7)
SEAS-SIR'
100, 100 (0. 0)
2.2, 203,
100, 100 (0, 0)
18. (0.4, (0.4, 0.3)
SEAS-SIR"
94.1. (2.5
1.4222 (..2.0.))
23.222.222...2...
95.3, (1.6, 1.6)
0.1.0.2 (0.0)
36.7. (2, 1.)
SEAS-PFC
100, (0, 0)
L.5 (0.20.0))
9.1. (0.....)
100, 100 (0, 0)
0.2. (0.0)
17.7, 30.8 (0.4, 0.3)
SEAS-PFC"
91.5, (1.2 2.2)
1.2, 1.3 (02,
23.6 (2.5
94.3, 93 (1.7, 1.9)
0.2
26.7, 37.6 (2, 1.6)
Table A.7. Simulations results for scenario $4 with 400K,p 1000, K 3,55 The table reports averages and standard errors TPR, FPR, and subspace distance, calculated as the mcan values across K clusters.
AR(0.3)
Method
TPR(S) )
FPR(%)
TPR(S)
FPR(%)
100
TPR(%)
FPR(S)
TPR(S)
FPR(S)
D x 100
M4 K
mixPFC
99.6 (0.1)
(0)
100 (0)
(0)
27.2 (0.4)
86.7 (0.8)
0.3 (0.1)
100 (0)
(0.2)
45.5 (0.6)
miXPFC-ISO
100 (0)
(
100 (0)
0.1
14.7 (0.2)
LassoSIR'
100
(..1)
100 (0)
17.3 (0.3)
53.1 (0.4)
99.4 (0.1)
(0.1)
100 (0)
17 (0.2)
61.2 (0.3)
LassoSIR m
100 (0)
(..1)
100 (0)
(0.3)
52.9 (0.4)
99.4 (0.2)
6.1 (0.1)
100
17.4 (0.3)
61.7 (0.3)
SEAS-SIR
100 (0)
(.11)
100 (0)
(.5
11.3
92
0.3
100 (0)
0.9
43.8 (0.7)
SEAS-SIRE
100
2.31)
100 (0)
(.44
12.1 (0.2)
90.4 (0.6)
0.3
100
0.8
45.4
SEAS-PFC
100 (0)
(0.1)
100 (0)
30.3)
9.7 (0.2)
90.3 (0.8)
0.1 (0)
100
0.3
44 (0.8)
SEAS-PFC
100 (0)
0.8 (0.1)
100 (0)
(0.3)
(0.2)
(0.8)
0.1 (0)
100 (0)
0.3 (0)
45.6 (0.8)
M4 =
mixPFC
98 (0.4)
0.1 (0)
100 (0)
0.4 (0.1)
32.3 (1)
88 (0.7)
0.8 (0.1)
100 (0)
4 (0:4)
50 (1.2)
mixPFC-ISO
97.2 (0.6)
0.2
100 (0)
0.9 (0.2)
(1.4)
LassoSIR'
100 (0)
(0.1
100 (0)
(0.)
(3.6
99.8 (0.1)
(0.1)
100 (0)
27.4
62.5 (0.3)
"
98.3 (0.5)
6.1
100 (0)
51.5 (0.6)
93.7 (1)
6.8 (0.2)
99.8 (0.1)
29.6 (0.6)
69.9 (1.2)
SEAS-SIR
100 (0)
2.5
100 (0)
11.8
11.4 (0.2)
94.6 (0.4)
0.4 (0)
100 (0)
(.8)).2)
42.9 (0.5)
SEAS-SIR"
99.3 (0.2)
2.6
100 100
(25)
(1.3
88.6 (0.8)
(.1
100
3.8
50.3 (1.1)
SEAS-PEC
100 (0)
1.2
100 (0)
(..4).
9.6
91.8 (0.5)
0.3 (0)
100 (0)
1.2
44.3 (0.5)
SHAS-PFCM
98.7 (0.3)
1.1 (0.1)
(0)
(0.4)
(1.3)
86.3 (0.9)
0.5 (0.1)
100
2.5 (0.5)
50.9
15

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Table A.8. Simulations results for scenario SI with n. 400, p 1000, K FPR, and subspace distance for each cluster.
table reports averages standard errors TPR,
0.11
= AR(0.5)
Method
(%)
FP)..(%)
Dw 100
TPR,(%))
FPR...%)
D.. 100
M1
mixPFC
97.5, 98 (0.9, 0.7)
0,0
39, 39.99.222 (1.2, 1.3)
96.3, 96.5 (1.1, 0.8)
0.9, 1.1 (0.1, 0.3)
50.3, 48.3 (1.3, 1.2)
mixPFC-ISO
100, 100 (0, 0)
0.1,
1.6 (0.1,,
LassoSIR
98.8, 99.5 (0.4, 0.3)
0.8, 0.9 (0.1..0)
(3.2.
96.2, 95.3 (0.7, 0.8)
1.1. (0.1,, 0.1)
52.8, 51.7 (1.2, 1.2)
"
(0.7, 0.7)
0.8, 0.9 (0.1..0))
45.3.45.
96, 94.5 (0.8, 1.2)
1.2 (..1,
51.7, 52.8 (1.2, 1.4)
SEAS-SIR'
100, 100 (0, 0)
0.1,
2..22.
100, 100 (0, 0)
(0.1..))
19.7. 19.7 (0.4, 0.4)
SEAS-SIR
100, 100 (0, 0)
(..
202.
99.8, 99.8 (0.2, 0.2) 0.2)
0
20.5, 20.6 (0.6, 0.8)
SEAS-PPC
100, 100 (0. 0)
(,0
2.8, (0222
100, 100 (0. 0)
0.1.0.1
19.8, 19.5 (0.5, 0.4)
SEAS-PFC"
100, 100 (0, 0)
0,2,
3, (0.3, 0.3)
100, 99.2 (0, 0.6)
0.1, (0,00.)
20.5, 20.6 (0.6, I)
M2
mixPFC
87.8, 87.5 (1.8, 1.9)
0.2, 0.2 (..1.
50.6, (1.5, 1.7)
95.5. 100 (1, 0)
2.6 2..1 0.2)
54.2, 34.1 (1.6, 1)
mixPFC-ISO
100, 99.8 (0, 0.2)
(0.00)
9.8, 10.3 (2, 2.1)
LassoSIR'
98.8, 99.3 (0.4, 0.3)
0.8, 1.1 (0.1
45.3, (1.2, 1.2)
96.3, 97.7 (0.7, 0.6)
1.1.3. (0.1.
52,9, 40.4000000 (1.2, 1.1)
LassoSIR
84.7, 84.7 (2.3, 2.3)
1.2, (0.1,.)))
(1.9,
91.7, 96.3 (1.4.0.9)
(0..0..)
33.3,43.7 (1.4, 1.4)
SEAS-SIR'
100, 100 (0. 0)
0.1.
2.9.2.60022.0.
100, 89.5 1.3)
(1..25
19.7. 63.8 (0.4, 0.5)
SEAS-SIR"
86.3, 86.2 (2.2, 2.2)
0.1,0.1
23.1, (311,
89.8 (0.5, 1.5)
(.1.2.4
23.6, (1.2, 0.6)
SEAS-PFC
100, 100 (0, 0)
0,0
2.8, 2.6 (0.22 0.2)
100, 95.7 (0, 0.9)
0.1, 0.
19.8. 59.3 (0.5, 0.5)
SEAS-PFC
85.5, 85.3 (2.3, 2.3)
(.0
23.3, 23.1 (3.1.3.1)
98.2, 92 (0.7.
11.17.00.0.)
23.8, 60.6 (1.3,0.6
M3
mixPFC
92.3, 92.5 1.9)
0.,
39.2, 37.9
1.9)
96.7, (1.1, 0.8)
0.5, 0.9 (0.1, 0.3)
40.5, (1.2, 1.2)
mixPFC-ISO
100, 100 (0, 0)
0.4, 0)
10.8, 10.7 (1.5, 1.5)
LassoSIR
100, 100 (0, 0)
4.5, 4.8 (0.2, 0.2)
57.3, 58.5
0.8)
98.7, 98.5 (0.5, 0.5)
8.8, 8.8, 6.4 (0.2, 0.3)
77.9, 75.3 (0.4, 0.4)
"assoSIR
95.5. 95.3 1.7)
4.6, (0.2.
59.8, (1.3.
96.5, 98.2 (0.9.
9.1. (0.2. 0.3)
78.5, 76 (0.5, 0.4)
SEAS-SIR'
100, 100 (0, 0)
0.
3.
100, 87.5 (0, 1.3)
0.6 (0.1.
34.3, (0.6, 0.9)
SEAS-SIR"
95.2, 96 (1.7, 1.5)
0,0
9.9,9.7
99.5, (0.3, 1.3)
0.7, (..1,
36.6, 53.3 (1, 0.9)
SEAS-PFC
100, 100 (0, 0)
0.0
3,299 (0.1.0.1)
100, 83 1.5)
0.1. (0.0)
33.1, (0.5, 0.9)
SEAS-PFC"M
94.8, 94.7 (1.8, 1.8)
0,0
9.2, 9.1 (2.1,
98.3, 86.3 1.6)
0.,
35.7, 53.2 (1.1,
Table A.9. Simulations results for scenario $4 with n 400K" ,p- = 1000, K 3,5. The table reports averages and standard errors TPR. FPR, and subspace distance, calculated the mean values across K clusters.
A 0.11
= AR(0.55
Method
TPR(%)
FPR(%)
TPR(%)
FPR(%)
D x 100
TPR(%)
FPR(%)
TPR(%)
FPR(%)
D x 100
M4
MixPFC
65.9 (0.9)
(0)
93.5 (0.8)
(0)
62.9 (0.7)
77.9 (0.7)
0.7 (0.1)
99.7 (0.2)
2 (0.2)
57 (0.5)
mixPFC-ISO
100 (0)
0.1 (0)
100
0.3
(..1)
LassoSIR
98.7 (0.2)
4.4 (0.1)
100
12.7 (0.3)
68 (0.3)
96.5 (0.3)
(0.1)
100 (0)
17.3 (0.3)
70.2 (0.3)
LassoSIR
98.8 (0.2)
(0.1)
100
(2.9
(0.2)
96.4 (0.3)
6.1 (0.1)
(0)
(0.3)
70.2 (0.3)
SEAS-SIR
(0)
(
10)
(
12.5 (0.5)
(0.5)
0.6
100
1.7
61.8 (0.2)
SEAS-SIR "TR"
100
(1)))
(0)
(
(0.4.))
75.3 (0.5)
0.5
100
1.5
62.2 (0.2)
SEAS- PFCC
100
(0)
(0)
(
9.1 (0.3)
72 (0.4)
(
100
0.1 (0)
61.9 (0.1)
SEAS-PFC"
(0)
(0)
(0)
0 (0)
8.9 (0.3)
(2.5)
(0)
100
0.1 (0)
61.9 (0.1)
mixPFC
65.7 (0.8)
(0)
97.8 (0.4)
(0)
(0.7)
76.8 (0.7)
(0.1)
100
5.1 (0.4)
65.6 (0.9)
miXPFC-ISO
100 (0)
0.1
100 (0)
0.6
3.8 (0.4)
LassoSIR
98.9 (0.1)
(0.1)
(0)
(0.9)
67.7 (0.3)
97.5 (0.2)
(0.1)
100 100
27.1 (0.4)
70.7 (0.2)
"
98.8 (0.2)
4.4
100 (0)
20.3 (0.3)
68 (0.3)
84.8 (1.4)
6.5 (0.1)
99.9 (0.1)
(0.5)
80.1 (0.9)
SEAS-SIR
99.9 (0)
(0)
100 (0)
(
10 (0.3)
80.3 (0.4)
0.4 (0)
100 (0)
1.7
62.8 (0.1)
SEAS-SIR"
99.8 (0.1)
0
(0)
000)
74.6 (0.4
0.8 (0.1)
100
4
(0.7
SEAS-PECE
100 (0)
(
(0)
000)
(0.3)
77.3 (0.3)
(
100
0,2
61.5
SEAS-PFC
99.9 (0.1)
(
(0)
0
7.7 (0.3)
74.5 (0.7)
0.8 (0.1)
100
3.7
67 (0.6)
16

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Table A.10. Simulations results for scenario S2 with n = 400, 1000, K = 2. The table reports averages and standard errors TPR. FPR, and subspace distance for each cluster.
A 0.11 FPR(S))
A
Method
TPR(%)
D x 100
TPR(%)
FPR(%)
D x 100
M1
mixPFC
97.8, 97.3 (0.7, 0.9)
0. (.0 0)
35.6, 35.9 (1.3, 1.4)
98.2, 98.8 (1, 0.6)
0.5, 1 0.31,
23.7, 23.9 (1.5, 1.6)
MiXPFC-ISO
100, 100 (0, 0)
0.2, 0.3 (0
1.9.
100, 99.8 (0, 0.2)
0.8, (0.2000220 0.2)
8, 8,8.5 (0.8, 0.9)
LassoSIR'
100, 96.3 (0, 0.7)
0.8, 0.8 (0.1.
51.2. 1.3)
100, 100 (0. 0)
(..11..
24.9.41.1 (0.9, 1.2)
T. LassoSTR
97. (0.9, 0.7)
0.9, 0.8 (0.1,
45.1,47.4
98.8, 98.8 (0.6, 0.7)
1.2.6.0
36.1, 1.9)
SEAS-STR'
100, 100 (0, 0)
(0,
(0.1,
100, 100 (0,
6.1,9.8 1.2.0.4
SEAS-SIR""
100, 100 (0. 0)
0.1.0.1
2.8. (0.2. 0.2)
99.8, 99.3 (0.2,
1.2.00)
90,2 (0.7.
SEAS-PFC'
100, 100 (0,
(,0
(0.1,
100, 100 (O, 0)
1.4008 (0.3,
6,4, 9.1 (0.3, 0.3)
SEAS-PFC
100, 100 (0. 0)
00.10.)
2.7. 3..4,, 0.52
98.7,
0.9, (0.2.0.2)
9.9, 10.8 (13,,
M2
mixPFC
89.8, 89 (1.7, 1.6)
0.2, 0.3 (0.1.
43.4, 50.8 (1.9. (19.. 1.6)
97.2, (0.9, 1.3)
(0.19 0.1)
26.6. 37 (1.9, 1.9)
mixPFC-ISO
100, 97.5 (0, 1.I)
0.4, (0.1.0)
9.6, 17.1 (1.7,
100, 98.7 (0, 0.8)
0.2 (0, 0.3)
6,4, 13.3 (0.7, 1.5)
LassoSTR
100, 96.5 (0, 0.7)
0.8, 0.8 (..1,
(1.2,
100, 100 (0
(41.1.)
24,9, 41.3 (0.9, 1.3)
" TM
89.2. 83.7 (1.9. 2.2)
0.9. 0.9 (0.1....
47.5.61 (2.
95.7, 92.2 (1.3, 1.7)
1.7.1.7
34.5 (2.
SEAS-SIR'
100, 100 0)
0,0.1
(0..0..))
100, 100 (0.
15.06.
6.1. 10.4 0.4)
SEAS-SIR"
90.3, 87.7 (1.9, 2.2)
0,0
20.2,21.5
97.3, 93.9 (0.7, 1.5)
(.300.)
16.7, 22.2 (2.,,
SEAS-PFC'
100, 100 (0, 0)
0,0
(0.1.
100, 100 (0, 0)
6.4. 9.8 (0.3,
SEAS-PFC
89.8, 87 (1.9, 2.2)
(,0
21.5,
95.7, 92 1.9)
1,0.7 (0.2.
17.8, 23.5 2.5)
M3
mixPPC
96.3, 93.2 (1.2, 1.2)
0..01
35.9, 40.1 (1.4, 1.8)
98.8, (0.4, 1.2)
(0.,
19.4, (1.4, 1.4)
mixPFC-ISO
100, 100 (0, 0)
0.5. (0.1.)
10.9, 14.3 (1.3, 1.6)
100, 92.7 (0, 2.5)
0.4. 0.4. 0.11.
17.1, (1.3, 2.3)
LassoSIR'
100, 99.3 (0. 0.3)
4.2, 4,2.40..2 (0.2,
48.8, 69.1 (0.8,
100, 100 (0. 0)
(02...))
42.5, 63.1 (0.8, 0.8)
LassoSIR" m
98.5, 98.3 98.3 (0.8, 0.6)
4.6, 4.2 (0.2,
51.8, 70.5 (1.3,
98.8, 99.5 (0.6,
7.2.5.6002,,02..
45.9, 65 (1.2, 0.9)
SEAS-SIR'
100, 100 (0, 0)
0, (..0
3,4.7 (0.2, 0.3)
100, 100 (0,
2.21,.0
7.8. 12.9 (0.2, 0.3)
SEAS-SIR"
97.3 (0.5,
0,3.3 (0.
6.7,8.8 (1.3,
99.2, (0.5, 0.8)
1., (02.0.)
12.3, 18.3 (1.5, 1.6)
SEAS-PFC'
100, 100 (0, 0)
0,,
2.3. (0.1.
100, 100 (0. 0)
0..30.8003.0.) 0.1)
7.5, 11.3 (0.3, 0.3)
SEAS-PFC
97.2 1.2)
(0.1,)
717.
97.5, 97 (1.2, 1.1)
1.7 (0.2.
12.7, 17.1 (1.7, 1.6)
Table A. Simulations results for scenario S2 with TPR. FPR, and subspace distance for each cluster.
400, 1000, K 2. The table reports averages and standard errors of
A AR(00.)
A AR(0.5)
Method
TPR(%)
FPR(%)
D 100
TPR($)
FPR(%)
x 100
MI
mixPFC
96, 98 (1.4, 0.9)
0.1.0
35.7, 34 (1.6, 1.3)
96.2, 95 (0.8, 1.1)
0.8, 0.8, 0.7 (.11.
46.2, 44.4 (1.4, 1.4)
LassoSIR'
100, 97 (0, 0.6)
1.2, (.110.))
33.2, 50.3 1.4)
99.7, 91.3 (0.2.
0.9,0.9 (0.1, 0.1)
40.7, 60.9 (1.2, 1.1)
LassoSIR"
97.3, 98.8 (0.9. 0.5)
1.2, 1.5 (..1.02)
48.3, (2, 1.8)
95.5, 94.8 (0.8, 0.8)
0.9. 0.1)
52.8. 50.2
1.7)
SEAS-SIR
100. 100 (0. 0)
1.2 0.5 (.12.
13.5. 15.5 (0.3.
100, 100 (0. 0)
0.3.0.1 (0.
18.8, 19.8 (0.2,
SEAS-SIR"
100. 100 (0.
0.9, (.2.
15.9, 15.1 (0.6, 0.4)
100, 100 (0. 0)
0.2,0.2
19.5, 0.5),
SEAS-PRC
100, 100 (0. 0)
0.1.
13.4. 14.9 0.4)
100, 100 (0
(...0.1
18.7. 19.7 (0.2, 0.4)
SEAS-PRC."
99.3, 100 (0.7,
0.4, (O.1,,))
16.3, 15.8 (1, 1.I)
100, 100 (0,
0.1, (0,
19.4, 20.1 0.4)
M2
mixPFC
98.5, 99.8 (0.8, 0.2)
0, 1.4 (..40.
32.4. (1.3. 1)
96.3, 97.3 (1.1, 0.9)
(0,
41, 47.5 (1.3, 1.2)
LassoSIR
100, 100 0)
2..
33.2, 42.2 1.1)
99.7, 88.8 (0.2. 1.1)
(.....0
40.8, 61.4 (1.2, 1.2)
LassoSTR
97.7, 97.8 1)
(0..0.))
35.2, 48.7 (1.3, 1.4)
95.8, 85.7 (1.3, 1.5)
(01.0.))
45.9, 64.5 (1.6, 1.2)
SEAS-SIR
100, 100 (0. 0)
12, (02.0.))
13.5, 38.1 (0.3.
100. 74.7 (0, 1.4)
(0.00.)
19,6 (0.2. 0.5)
SEAS-STR
99.3, 97.8 (0.3, 0.8)
(0.3,00)
16.2, 41.3 (0.9,
99.7, 74.2 (0.2, 1.5)
(0,00.0
20.6, 71.3 (0.6, 0.6)
SEAS-PFC
100, 100 (0,
(0.00))
13.4, (0.3. 0.5)
100, 85.2 (0, 1.4)
0.1. (0,
18.9. 64.9 (0.2. 0.7)
SEAS-PRC"
99.3, 97.8 (0.3, 1)
0.1,0.8
16, 36 (1,
99.3, 83.3 (0.3, 1.7)
0.1, (0.
20.8, 66 0.7)
M3
mixPFC
98.8, (0.5, 0.8)
(0.2,.0)
25.1, 36.6 (1.5, I.I)
96.2, 95.7 L.I)
0.1, (0. 0.2)
37,54.3 (1.5,
LassoSIR*
100, 99.3 (0, 0.3)
1..1. (0.2. 0.1)
62.8, 75.9 (0.5,
99.8, 95.2 (0.2, 0.8)
10.9. (0.3. (03.
74.1. 80.2 (0.3, 0.4)
LassoSTR" m
98.2, 98.3 (0.9, 0.6)
(0.2 0.2)
64.8, 76.5 0..)
97.2, 93.8 (1.1, I)
11,5.1 (0.3,
75.3, 80.6 (0.6, 0.5)
SEAS-SIR
100, 99.8 (0, 0.2)
0.6, 0..0.8 (0.2.
18.3,33.1 0.4)
100, 86 (0, 1.3)
0.4.0.1
32.7.54. (0.5, 0.9)
SEAS-SIR" SIR""
97.3, 96.7 (1.2, 1.3)
1.1. 1 0.1)
22.6, 36.1 (1.4, 1.2)
97.5, 84,3 (1, 1.5)
03.0..10,00
56.1 (1.2, 1) 32, (0.5,
SEAS-PFC
100, 99.2 (0, 0.5)
0.1. (0.
17.6, (0.3.
100, 83.8 (0. 1.4)
0.1.0.100,0)
SEAS-PRC"
96.8, 95.3 (1.4,
0.1. 0..
36.3 (1.5. 1.3)
96.5, 81.8 (1.4, 1.6)
0.1
56.3 (1.3,
17

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Table A.12. Simulations results for scenario S3 with n 400, p 1000, K 2. The table reports averages and standard errors of TPR, FPR, and subspace distance for each cluster.
A = 0.11 FPR(%)
A
Method
TPR(%)
D 100
TPR(%)
(PR(%)
x 100
MI
mixPFC
96.8, 95.8 (1.2, 1.4)
(0,00
38, 37.7 (1.3, 1.4)
95.7, 95.7 (1.6, 1.7)
1.4, (0.3 0.3)
36.8, 36 (19.2)
mixPFC-ISO
100, 100 (0, 0)
0.6, 0.6 (0,
2.6, 0.11
99.7, 99.7 (0.3, 0.3)
1.4, 1.5 (0.1, 0.2)
13.2, 13.5 (1, L.I)
LassoSIR'
100, 99.5 (0, 0.3)
(..
38.1,43.8 (1.1...)))
100. 100 (U. 0)
2.5,2.600..00 0.2)
36.5,
1.2)
LassoSTR"
99.7, 99.7 (0.2, 0.2)
1.2, (..90
(1.2,
96.7, 96.2 (1.6, 1.7)
3.1, 0.3)
45.8, 45.3 (1.8, 1.9)
SEAS-SIR'
100, 100 ()
(0,
3...3.7
100, 100 (0. O)
0.9, (..11.) 0.21.
13.1 (3.40,0)) 0..4.
SEAS-SIR
100, 100 ()
(0.00
3,9
98.1, 98.3 (1.3, 1.2)
1.6, (0.22.
17.1, 16 (1.3, 1.4)
SEAS-PFC
(0
02,003
3.5. (0.2, 0.2)
100, 100
(0.2,
12.9 (0.4.04)
SEAS-PFC"
100, (0,
0.3, (0.1,)
3.7,35 (0.2, 0.2)
96.8 (1.7, 1.6)
1.7 (0.3,
19.5, 18.1 (18,
M2
miXPFC
91.7, 90.5 (1.6, 1.7)
0.8, 0.6 (0.3, 0.4)
42.7,45.9
1.9)
95.3, 95 (1.6, 1.4)
1.1, 1.7 (0.2, 0.2)
36, 39.8 (1.7, 1.7)
PFCPFO-
100, 99.5 (0, 0.5)
4,4, 0.8 (1.4.0) 0)
(..17 1.5)
98.3, 97.7 (0.8, 0.9)
2.5 2.5.2 0.4)
16,9. 19.3 (1.5,
LassoSTR'
100, 99.7 0.2)
(...,0)
(1.1.
100, 100 (0, 0)
2.2.2.
36,5,
1.4)
LassoSIR
92.8, 92.3 (1.7, 1.6)
1.4 (0.2.
46.1.49.8 1.9)
95.8, 95.7 (1.3, 1.4)
2.8,
47.1, (1.5, 1.5)
SEAS-SIR'
100, (0, 0)
(0,0)
3.8.
100, (0.
0.9, (0.1.
11.1, 12.9 (0.4, 0.4)
SEAS-SIR"
94.3, 95.8 (1.5, 1.3)
(0.,0
14.5. (2.5.
97. 96.96.8 91. 1.2)
1,1. (0.1.
22.2 (1.7, 1.6)
SEAS-PFC"
100, 100 (0, 0)
3.6 (... 0.2)
100. 100 (U. 0)
0.9, 0.9 (0.2.
10.8, 12.3 (0.4.
SEAS-PFC
92.3, 94.7 (19. 1.5)
0.2, (0.1. 0.1)
14.8, 14 (2.6, 2.4)
96.3, 95.7 (1.3, 1.5)
0.9, 1.2 (0.2, 0.2)
21.1. (177, 18)
M3
mixPFC
407, 1.I)
2.6.4.5 (0.7,
80.3, 81
49.7, (1.2, 1.I)
2.2.,002
73.7, (0.4,
mixPFC-ISO
8288, 84.3 (2.1,)
11,09. 0.13 (0.,0.))
48.8, 48.7 (2.4, 2.2)
53.6 (1.4, 1, 1.5)
4.6,4.1 (0.2, 0.1)
73.2, (0.6, 0.6)
LassoSIR*
(0.6, 0.2)
8.3. (0.2,
67.6, 67.3
69, 67 (1.5, 1.5)
10.1, 10.5 (0.2,
73.5, 73.9 73.9 (0.3, 0.3)
"
69.5, (3.2, 2, 3.3)
9.9,9.3 (..3.
81.6, 80.1 (1.2,
59, (1.3, 1.8)
10.5, 10.9 (0.2,
76.3, 76.4 (6.7, 0.7)
SEAS-SIR
94.7,95.8 (1.4 1.2)
(0
56.2, (1.5. 1.1)
L.7.
70.6 (..1,
SEAS-SIE SIR"
68.3, 67.7 (2.9, 3)
0.5, (0.2,00))
65.4, (1.7,
53.1, (1.2, 1.2)
2.5, (.,, 0.2)
71.9, 71.6 (0.5, 0.4)
SEAS-PFC!
50, (0,
(0,00
70.7 (0.
51, (0.5,
2.222 0.3)
70.8, 70.8 (0.
SEAS-PRC"
43.7,44.7 (0.9, 0.9)
0.2, 0.5 (0.1,
75.7 (0.6,
48.7, 49.2 (0.8, 0.9)
1.8, 1.7 (0.3,
72.3, 72.4 (0.5, 0.5)
Table A.13. Simulations results for scenario S3 with n = 400, = 1000, K = 2. The table reports averages and standard errors of TPR, FPR, and subspace distance for each cluster.
= AR(0.3)
A AR(U.5)
Method
TPR(%)
FPR(%)
D x 100
TPR(%)
FPR(S)
D x 100
MI
mixPFC
96.7, 96.5 (1.1. 1.1)
0.5. 0.2)
35.1, 36.9 (1.6, 1.7)
92.2, 91.3 (1.5, 1.9)
0,4, 0.2)
49.4. 49.1 (1.5, 1.5)
LassoSTR
99.7, 99.7 (0.2, 0.2)
(0.12
41.6, 45.5 (1.2, 1.4)
99, 97.3 (0.4, 0.7)
1.2, 1.4 (0.1,..
47.1, (1.3, 1.2)
LassoSIR
99, 98.5 (0.6, 0.7)
1.2. 0.2)
43.9. 45.5 (1.5, (1.5.155)
94.8, 94.7 (1.2. 1.4)
1.4.44 (0.2..0)
53.3. 52.3 (1.4. 1.4)
SEAS-SIR'
100, 100
1...0.2)
15. 15 (0.4. 0.4)
100, 100
0.2.0.3
20. 20.9 (0.4. 0.6)
SEAS-SIR" m
99.8, 100 (0.2, 0)
1.2.2.2
16.3, 15.7 (0.6. 0.5)
99, (0.6, 0.8)
0.3, 0.
22.3, (1.1, 1.1)
SEAS- (FCC
100, 100 ()
(0..
14.7, 15.3 (0.4, 0.4)
100, 99.8 (0, 0.2)
0,2, 0.2 (0.
20, 20.9 (0.4, 0.6)
SEAS-PFC"
100, 100 0)
(03.
16.2, 16.8 (0.7, 0.8)
98.7, 98.7 (0.7, 1)
0.4, 0.5 (0.2, 0.2)
22.7, 23.7 (1.2, 1.3)
M2
mixPPC
94.2, 99 0.5)
(..1,,
42.3, 36.6 (1.5, 1.3)
72.7, (1.6, 1.5)
40.1,,
52
L
99.7. 100 (0.2.
4...
41.6, 47.9 (1.2, 1.3)
99.2, (0.4, 1.2)
6..66.000....2
47.5. 70.6 (1.3, 12)
"
94.8, 98 (1.2, 0.7)
1.6, 1,6,49 (0.2,
51.3, (1.4, 1.2)
88.5, 85.7 (1.2, 1.4)
6.777,7
60.4, 71.5 (1.3, 1.3)
SEAS-SIR
100, 99.5 (0, 0.3)
(02,
15, (0.4, 0.8)
100, 66.8 (0, 1.5)
0.2, (0.
20, 73.4 (0.4, 0.5)
SEAS-SIR"
98.7, 96.7 (0.7, 1.1)
1.5, (..2.0.2)
46.5 (1.4,
97.3, (0.8. 1.5)
(..
31.1. (1.4. 20, 69.7 (0.5, 0.5)
SEAS-PFC
100, 99.7 (0. 0.3)
1.2.
14.7. 36 (0.4.
100, 73.8 1.7)
(0.00
SEAS-PFC
98.8, 97 (0.6, L.I)
0.6, 1..2,
22.1, 42.1 (1.4, 1.1)
96.7, 1.5)
(0.
31.3, 70.6 (1.5,
M3
mixPFC
50.5, (1.2, 1.1)
0.5, 1.3 (0.1. 0.2)
72.6. 74 (0.4, 0.6)
53.5, (1.3, 1.3)
0.3, 03.110.3 (0.2.
72.3. 73.6 (0.5,
!
73.5 (1.6, 1.5)
9.6, 9.5 (..2.0.2)
74.2, 73.9 (0.3, 0.3)
78.5, 77 (1.4, (1.44.144 1.4)
8.7.9 (0.2.
75.4. 74.1 (0.3.
"assoSTR
70
10.2 (0.2,
76.5,
0.7)
75.8, 75 (1.9, 1.4)
(0.2, 0.2)
77.7, 76.2 (0.7, 0.6)
SEAS-SIR'
90.5. 80.2 (1.3, 1.2)
0.6, 1.3 (0.1.. 0.2)
59.5. 65.7 (1.2,
93. 84.2
0...5.
62.9. 65.7 (1.1,
"EAS-SIR"
82.6, 74.3 (1.9, 1.6)
(0.1,,
67.9, 69.2 (0.9.
87.. 1.4)
0. (0.1..0)
69.5, 68.4 (0.8, 0.7)
SEAS-PFC
89.7, 76.3 (1.5, 1.2)
0.4, 0..1,
53,9, 61.7 (1.4,
93.3, 83.2
0.1,
55.1, 63.2 (1.2,
SEAS-PFC"'
70.5 (2.1.
0..10.
64.4, 68.1 (1.2,
85.7, 80 (2,
0.,
64.9. 67.5 (1.2.
18

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
A.2. Real Data Analysis
For the CCLE data, we evaluate the performance of mixPFC across K 2,.,..., 10. Table A.14 summarizes PMSE and the number of selected variables $ (total number of unique variables selected across all clusters) for different K under mixPFC.
For mixPFC, the prediction error decreases as the number of clusters K increases. However, the reduction becomes less pronounced beyond = 3 for Nutlin-3 and K = 5 and AZID6244. For both responses, mixPFC achieves a significant reduction in prediction error compared to homogencous methods (Table 2), suggesting heterogeneity in the data. The num- ber of selected variables š initially increases and then stabilizes as the number of clusters K grows, indicating that different clusters select different variables. Notably, mixPFC does not select more variables than Lasso and the three homogeneous SDR methods, even for large K. These findings suggest that over-specifieation of K does not significantly impact the pre- diction performance of mixPFC. However, over-specification of K increases computational cost and introduces variability in parameter estimation. Using the tuning method described in Section B of the appendix, the average selected K is 3 for both Nutlin-3 and A7DD244..
Summary plots of the response versus the reduced predictors projected onto each subspace are shown in Figure A.5 for AZD6244. These plots reveal upproximately linear relationships between the response and projected predictors within each cluster for both drugs. Notably, the points in cluster 1 form vertical band when projected onto subspaces S. 2, 4, This pattern coincides with the example in Figure (b) and suggests the first subspace is orthogonal to the remaining subspaces.
Table A.14. The averages of the prediction errors, the sparsity level 3, and the corresponding standard errors based on 100 replicates.
K=
==3
K
K Nutlin-3
==6
==7
K ==8
K
K 10
PMSE x 100
11.9 (0.4)
8.9 (0.4)
8.3 (0.3)
8.4 (0.3)
7.7 (0.3)
7.84 (0.4)
7.36 (0.3)
6.9 (0.3)
6.9 (0.3)
8
16.5 (1.2)
31.5 (2.5)
45.2 (2.7)
52.3 (2.7)
53.9 (2.7)
53.7 (2.6)
52.6
58.2 (2.4)
59.5 (2.1)
AZD6244 (1.7)
PMSE x 100
66.4 (1.5)
58.7 (2.0)
62.2 (2.3)
41.6 (1.6(1.)
36.7 (1)
33.5 (1.0)
32.00
34.7 (1.2) 70.6 (2.3)
8
21.6 (0.6)
53.4 (2.5)
62.2 (2.3)
77.8 (3.2) 77.4 (3.1)
74.1 (2.8)
65.1 (2.7)
67.5 (2.4)
B. Implementation Details
B.1. Initialization and Tuning K
Initialization. To implement the proposed mixPFC algorithm, it is critical to obtain reliable initial values for Yiu- Classical distance-based clustering algorithms, like K K-means and hierarchical clustering, often produce low-quality initial values. Similarly, subspace clustering methods also fail for the high-dimensional mixture of PFC. In the Gaussian mixture model, it is recommended initialize EM with short runs of EM, where the algorithm is stopped early rather than (Biernacki et al., 2003). Given that existing methods are not designed for the model (3), we propose a similar initialization procedure that runs the mixPFC algorithm on transformed data with an early stopping criterion to generate initial estimates of Yow In high-dimensional settings, we transform the original data into a lower-dimensional space by applying distance correlation (dcor) screening (Li et al., 2012) followed by principal component analysis (PCA). Specifically, we first select x (n log n| variables using dcor and then project the data onto the first u principal components of the selected variables. Based on simulation results in Table A.15, we recommend setting u 2 and v 10.
convergence
Selection of number of clusters K. The gap statistic (Tibshirani et al., 2001) is popular technique for selecting the optimal number of clusters for many clustering algorithms. Here, we adapt this approach to enhance its suitability for the mixture model 3. Let Qw represent the orthogonal complement of Bw. G denote the indices of observations in cluster w and n. = G Then define
K
DW
Σ L.EG
Q
O.
VK
D
2nw
The gap statistic is
Gap,, (K) = Fx, [lg (vx)) log(Vk))
19

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
2.-
20-
15"
15-
15-
dluster
50
1xx
7.--
15-
1.5-
-55
-12
1x
Figure A.5. The scatter-ple plot of the response versus x for the drug AZD6244. The solid line fitted using samples the given cluster.
Table A.15. Error rates averaged over 20 datasets generated under model scenario SI using M1-M3. First, use deor to select
|N log(NN) variables and then apply PCA to project the data onto dimension space spanned by principal components.
M1
M2
M3
5
10
15
v
10
15
V
10
15
A = 0.11 mixPFC
u
5.7
2.2
6.3
25.2
27.1
30.0
8.4
3.0
7.3
13.6
4.0
4.5
29.2
31.6
35.0
11.0
11.9
13.5
15.8
13.2
4.5
30.1
38.6
38.1
7.3
7.5
10.5
A
0.1I
mixPFC-ISO
23.8
25.0
24.9
6.7
9.0
11.3
20.6
24.3
19.2
2
24.7
22.1
34.5
11.1
11.3
17.3
18.7
19,8
24.5
3
30.5
28.2
18.7
9.1
21.9
9.2
27.3
22.3
17.2
A = I mixPFC
14.3
10.7
10.8
27.3
37.4
43.5
9.9
9.0
20.1
14.4
11.9
31.6
38.5
41.3
13.9
9,9
8.7
16.1
16.5
18.7
31.8
42.7
45.9
11.9
13.8
20.7
A
mixPFC-ISO
17.3
13.0
11.4
9.4
12.0
15.3
13.9
19.7
17.6
=
18.8
15.0
11.5
15.3
10.2
12,6
24.8
22.1
13.3
u 3
15.0
15.4
17.6
12.2
14.4
13.2
14.7
19.5
8.3
AR(0.3)
mixPFC
21.0
12.3
14.4
27.0
31.2
38.8
12.9
12.3
11.4
13.9
15.5
12.5
23.2
34.3
41.4
20.5
19.1
14.9
22.5
18.9
14.3
29.4
40.9
36.5
26.3
28.4
23.5
20

---

Heterogencous Sufficient Dimension Reduction and Subspace Clustering
where E, denotes the expectation under a sample of size from reference distribution. We estimate E (ooggoVV (Vk)) by an average of B copies log(V which is computed from samples drawn from uniform reference distribution. However, in high dimensions, the gap statistic becomes computationally expensive it requires calculating Vx for B data samples, In practice, we find that bypassing the expectation calculation and directly using VK yields effective results. Let K denote the true number of clusters. To illustrate, we use models M1 (K* = 2) and M4 (K* = 3, 5), with an identity covariance matrix as defined in the simulation section, to generate high-dimensional data. Notably, the two subspaces are identical in model M1. Figure A.6 presents representative plot of the within-cluster dispersion Vx. calculated using the mixPFC Algorithm 1, as a function of the number of clusters. The enror measure VK decreases monotonically the number of clusters K increases, but begins rise once K exceeds the true number of clusters K Based this finding, propose to select the smallest K such that
max (Vx) - VK p(max((Vx) min(Vi))
where p E 10 specified to avoid overestimating K. Table A.16 presents the selected K for models M1 and M4 across four different covariance matrices. The proposed selection method works well most cases, with the exception of K = 5 under AR(0.3) and AR(0.5). The reason is inaccuracies in subspace estimation affect VK, which depends on the orthogonal complement of central subspaces.
4000
8000
4000
6000
2000
2000
3000
Figure A.6. Within-cluster dispersion Vi for K = 1, 2, 10.
Table A.16. The average selected number of clusters over 100 repetitions, using statistic Vx. The standard errors are in parentheses and
Kmax 10.
K*
0.1I
I
AR(0.3)
AR(0.5)
K" 2
(0)
(0)
2
2.1(0)
K" =3
30)
(
2.7 (0)
2.9 (0.1)
K" 5 (0)
4.8 (0)
3.7 (0.1)
3.1 (0.2)
B.2, Implementation of mixPFC
For the mixPFC algorithm 1, the tuning parameter (1) could either be fixed or vary across iterations. In theoretical analysis, to show statistical convergence results, set A (++1) () Cxv (log n)2 p/n, where 0 < к < 1/2 and C2 is positive constant. Note ( is at the order of / (log 2 p/ when is large. Therefore, practice, we fix ( and tune À with cross-validated distance correlation (Székely al., 2007). For fixed A X, the penalized EM algorithm maximizes
K
{(0) -
B 1|.11
where ((0) is the log-likelihood of X The following lemma shows the convergence Algorithm 1.
21

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Lemma B.1. If we X for all the objective function from (7) evaluated at 0(++) is guaranteed be no less than the objective function from (7) evaluated at 0 That is, the sequence of iterates {0(0) jt= generated by Algorithm I monotonically increase the value of the objective function from (7).
Proof. Recall that the conditional Log-likeihhod is
K
{(: X, Y)
Σ logi
Tw NX,, Mu + T To,)f A)].
w=1
The penalized EM algorithm can be viewed as two alternating maximization steps. Consider the following function:
Flg, 0)
(())
B 2,1
where q (q1, (9...... 4x) is arbitrary probability density the unobserved variable W and H(q) -E log(g.) is the entropy of distribution g. It is easy to show
F(g. 0) E(;; Del((llpxxx((xXXX))) - B
where DKL. is the Kullback-Leibler (KI.) divergence. The KL divergence between two distributions is non-negative and is zero when the two distributions are identical. Therefore, the E-step is to choose to maximize F(q. 0):
((++1) argmax F(g, (1)) = PWIX,Y (- Y. )
which is given by the updating function of You- M-step, we maximize F over @(++1) = argmax F F("(1)
K Σ
argmax E Ege [log XX -
B 12,1
argmax ((0 -
Σ
B w 12.11
which is the same the penalized M-step. Through this coordinate ascent strategy, the update (+++)) makes the value of the penalized log-lik likeliidd function converge monotonically. Combining this result with the convergence (Lemma H.1, Zeng et al. (2024)) of the group-wise coordinate descent algorithm 2, we know Algorithm converges when 1 = X.
Recall that Algorithm in the main paper. We update B'+1 u by solving:
argmin tr tr B (((( + | 1ul12,,1 B.. BR.xR 2
1
This optimization solved by the group-wise coordinate descent algorithm proposed in Mai et al. (2019). For ease of presentation, we remove the subscript w in B ((t) and T The algorithm is presented in Algorithm 2.
Tuning X. For two random vectors T E IRP and Z E IR?, the distance correlation deor (T. T, Z) E [0, measures the dependence between the two random vectors (Szεkely et al., 2007). In particular, dcor(T, Z) = 0 if and only and Z are independent. Given observed samples T E Raxp and Z R=*g, the sample distance correlation is denoted as dcor(T. (T, Z). Under the usual SDR model, Sheng & Yin (2016) showed that the distance covariance between Y and 3x is maximized at the central subspace over 3 Rexd such that E3 La. Therefore, Zeng et al. (2024) recommended selecting À by maximizing dcor(y, X,9), where Y _ and X guxp represent the observed response vector and predictor matrix. For the mixture SDR model, adjust for the sample membership Viw and select À by maximizing (D. Y, D,, X,B.), where D m is diagonal matrix with diagonal elements (
22

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Algorithm 2 Group-wise coordinate descent algorithm
Input: { (ōt,) E Rpx,, Útt) (ūn) R2x;
Initialize each row of B as B
(0) = (b) ..... b (0) = E RR; = 1, 1,...,P P and calculate the auxiliary vector
h (0) (0) (111 1 E R2 as as un U1 5 1 11
11 1 ,....
repeat
for to p do
Based on hj 1)-1) in the previous step, update (*))
) as
B) hj (r-1)
(1
A
ززر 1bj
Based on (rr) ..... j-1 B ((--)) ..... B² 1 update the auxiliary vector h 11-1)
as
//11 (r-1) ا Cr 1bu +
O 1= 1......
end for until converge Output: ((r)
B.3. Implementation of mixPFC-ISO
When A I there no need to compute the inverse of the pxp covariance matrix. Instead, we only need to estimate σ². Under this simplification, the parameters of mixture of PFC model (3) reduces to O (a Twi M Swn ,, K) In the E-step, the estimated probability is calculated by
(4) Th
( Ejtn T (x)(
110 P ) 1()))))))
In the M-step, the most important part is the updating formula of r Let c/i =; Viw (((t),, D.,,. be diagonal matrix with diagonal elements Tow (0()), X ROxp be the centered data matrix with rows (X; p yr F E Raaq with rows (f: r)² Through straightforward calculation, the MLE of S is the span of eigenvectors of (1)
fir, corresponding
to the largest dw eigenvalues, where (t)
ft. X PrD Cw. P3 D(F F and du is the dimension of
subspace S,,. Let T (t) (())
(0) Dd. be the eigenvectors corresponding the largest d, eigenvalues of (t)) fft,w" Then,
have
((
DF
To update o², we maximize Q function over 2², while letting other parameters fixed,
( (0-)00=
1 Σ cii (r((Ew)) tr a Emaww)
np
where $:00 X D.. In high dimensions, challenging to accurately estimate the eigenvectors of (
Bit.w To address this issue, we promote sparsity by finding the sparse eigenvectors of " Several algorithms have been proposed to compute sparse eigenvectors (d' Aspremont et al., 2007; Zou et al., 2006: Witten et al., 2009; Journée et al., 2010). We adopt the variable projection method proposed by Erichson et al. (2020) for its computational efficiency and robustness in high dimensions. The algorithm, mixPFC-ISO, and corresponding convergence analysis provided in the supplement.
To obtain sparse estimates $ (t) We adopt the variable projection method proposed by Erichson et al. (2020) for its com-
23

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
putational efficiency and robustness in high dimensions. Specifically, we solve the following problem
(A*, (D) argmin A
2
( fit
(0))) fit,
2 A 17 {(D)( subject to A =
where (+) is a tuning parameter, and & is sparsity-inducing penalty function such Lasso or clastic net, For fixed (t a, the penalized EM algorithm 3 for mixPFC with isotonic enrors maximizes
K
((0)
- a
(8)
w=1
The following result guarantees the convergence of Algorithm 3.
Lemma B.2. If we set (t)) a for all t, the objective function from (8) evaluated at @(++1) guaranteed be no less than the objective function from (8) evaluated at O()) That is, the sequence of iterates (0(0) It generated by Algorithm 3 monotonically increase the value of the objective function from (8).
Proof. The conditional log-likelihood is
K log Te N N(x pw + (Y), w=1
{00 X, X)
The penalized EM algorithm can be viewed as two alternating maximization steps. Consider the following function:
F(g, 0)
H()
Σ (
where q = (q1...., 1Kk is an arbitrary probability density over the unobserved variable W. It is easy to show
Flq.0)
( (X,,,
((
where DKL is the Kullback-Leibler (KL) divergence. The KL divergence between distributions is non-negative and is zero when the two distributions are identical. Therefore, the E-step to choose q to maximize F(g. 0):
(+1) argmax F(g, O =
which is given by the updating function of Yor M-step, we maximize F over
(++1)) arginax ((
argmax EqUO ((0
((V...
argmaa (()) - a w.
Through this coordinate ascent strategy, the update 0 (+1) makes the value the penalized log- liglliili function con- verge monotonically. Combining this result with the convergence of the variable projection algorithm solving the penalized Q-function (Erichson et al., 2020), we know Algorithm converges when (t) a.
24

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Algorithm 3 Penalized EM algorithm for mixture PFC with isotonic errors
Input: Data {(X,, Y=) fitting function f Initialize (a%) repeat
(0)) - -
E-Step:
ww
+ E j
//10 exp(L/663100X - HO
M-Step:
(()))
u xT D PDD to X
n. Pp = =(F
D D 1F
(A", $ (1)) = argmax
(1))
(()
T L - a (
subject to A =1
A s
2
fit
((+1)
2 { (t)
(0)
FFFF F) -
w
(G2) 1] 1 Σw=l w=1
tr (t()
tr (P 3 m)
(++1) 1 -5 E 1s
(t)
Tw
Pie (1++1)
Σ Now ((0)
E: 7iw((001)
until converge
Output: Rw, T
C. Proof of Theorem 1
From this section onward, we focus on the theoretical properties of a simplified version (Algorithm 4) of the mixPFC algorithm presented in the main paper.
Recall the five conditions are:
(CI) The singular values of E 1 Σ.= l.f satisfy that M1 S Omin (Ex) S max (Ex) S M2. and M3 s minigign [f:]|2 I maxigisn ||,/|2 M4.
(C2) The initialization (0)) satisfies that di (0 (V) (0) - Bi
FV
B
-Bill F < rsl, and vec (0) rw) L(s), 2M1
with r < lco - 1 C. A C. 1/(4VM
62
a²
b = M + 2M
an
VM,
2/M
(C3) There exists sufficiently large constant M3 > O, which does not depend on n, p, 8, such that oa(B 2 Ms 2 sqq (2llg)))loo/// logp/n.
ri (C4) Ω (r((TT - - co. Mr On, and M,,i= L...,,4
Clco, Co, Mr. Múi= 1 1,..., 4), which is constant that is only depends on
(C5) n > Casq log(p) for a sufficiently large constant C3.
Before the proof, review some definitions and present properties about Σ and U The parameter of interest is 1 ", r2). Let 0* be the true value of 0, and @(t) be the estimate of at the 1-th iteration. The true parameter space consider is
o' = (0 गं E 1- vl s sq, |B S Ma. lel & Mo. = 1,2),
and the constriction basin Bun (8)) is
Benn (0)) ((0 : Tw (co, 1 - (o): [T, l' S Casl,
(1- Cx) < |r( (T) r
1T) )) < (1+ Ca)
vec - ) E L(s), w 1,,
where Q is the signal strength, co S Cr. and 8m (T) = r - -Te + T2)/2. We define
((s) = {u E R
| ||1 - (v /g + 2qv 3s) |ug llz + /sqllull,
for some S C [pq] with | = 3sg.}
25

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Algorithm 4 Penalized EM algorithm for mixture PFC (simplified version for theoretical analysis)
Input: Data (X,, Y)). fitting function f, d1, d2
Initial value = ( D U and B argmin tr(B
(r((((())) B.)) ( B.,) + +1 [ 2,11 where the
B. 2 w
initial tuning parameter is
dr 010) 0 V
Bills V | Bille + +C,V
(1ogg)) logp
1 =
V
n
repeat E-Step:
Tin (0:))
nw
+ Σ wr yeu ;
exp(([=2(0))-1(X;
I + T (((.))))) T ((Y)))
M-Step:
(1++1)
1 1 (,(1),
1)++1)
Now ( X; (
Σ
Nho (.P(Y)T
Tw
w
n ==1
1=1
9 q* n)² log
( argmin B. 2
(1))
ir(
B (
[ ((+1) = = kx10+Ca)
n
Compute 1(t11 the top-d left singular vectors of ( and then update according to
((++))
(
Σrl -1
(02)
1
Pew (0 (1))()(())))) ir(P po+t)) -ir(Prgto Enn,)))
np
until converge Output: Rw, r w: nM
Let M() = (O), E., (0), w = 1,2]. Note that veo(T*)) lo S sq implies || veo(B*)llo S sg. Define
1
(0)
Now (0),
( (0))
i=1
1
(e)
New 0 O
(w) = (e)
n
1 n
(0)(0)
Tiw (0)x,
(0)
n1
By definition,
((++1) ano 7/ (0(1)
(t+1) E|+(1+11] El (0)) u. (o())
T
((++1) = Ü. ((t))
U((+1) 1) E(Ū(+t) = E )() = U., ( (0)
(1+1) ESm (0)))
((++)) E[ F. (( (0)) E. (10())))
Note that
Ta (0") E( 1
New (0 E E Σ
P(W, w YJ) P(W. wi X, Y)) π gm
im
26

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
We have
U
Ui(0')
B Birimn (0")X:
1
I I(W, = w) x
Σ
x
I(W,- = =,)x,
17
n1
1=1
1
1 n
x f.f T Er.
72
n
where Σ₁ 1/2 Di=1 1i=1 f.f.F Similarly,
n Ex = (0))
El Niw (0')x,X²]
n
Ew [[W, w) [XX
Ew - w F. (ra))
T p) + n T To Sp(Tt)) = n. " + n r (T(T)TTT
Let B = 1 10
U w,
5 be the top-d left singular vectors. Then span(B:)) = span(T)). We further have r
Pr. U ( (T E{) -1 P 3. U (T )
Since min (TT) 2 0, miin (Ip + r Ep (T) 2 1, we can bound the 2-norm as following
B |Bul
|| ( p + क E (T
) x ri Drl12
[pp + r Er w
r w
I + r Ep m. 1
| Erl1a
< Till |Erlz - M,.2..
Define (M.)). M(22)) and (O)) as
max (ITT(( n.. (02) (01) (02) (0)) (O, p.a.:
w=1,2
(ITT (01) (02) (0,) (O2 v (0,) - (o, B p)),
w=1,2
where r|
(A, r. The proof of Theorem 4.1 is based on the following two lemmas.
Lemma C.1. Under conditions (C1) and (C4), i 0 (0))(0)) ). some < Ro 2v(26Cog//o//VCC'
(F) S Ro(d 0*) v B1 - Bille v B - BB]le),
where d (0,0") {[(T] x1 V [[ - Tille [[ - []
Lemma C.2. Suppose 0" E O", under condition (CI) and (C5), there exists a constant Ceon > 0, such that with probability at least o(1),
sq
(log n)2 logp
sup (O) S Ccom OE (0*)
n
The proof of these two lemmas is quite involved and is presented in Sections D and E. We first establish a concentration result for the estimator B, in Section C.1 before proving Theorem 4.1 Section C.2.
27

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
C.I. Concentration of the estimator B
Lemma C.3. Suppose that 0" e and vec vec [ ) B000 Let
9 (logn) logp + 4mO
dF(((t) 0 V Bill B B
((+1) 2 4Coom
for Ko defined before and some constant Coo and ((++1)
solved by
B
argmin
Ir Ir(B w
S(TH))
(rr((())))) B + \
w
B
With probability least 1 - o(1), we have
veo veo (1+1) ((++1 B²)) E L(s);
and
4
2 ((+t) 35q 2/sq 22/3sq2).
B ((+1) B |F dr (M) TO
TO
Proof. Recall that
C(s) {u E IP |ug || S (vsq + 24//3s) lug lla /sqluull.
for some [pd] with |$: 3sq.}
Consider
(t++)) to
= argmin
tr SB (r(((((+)))T Bw + ((++)) | B 122.1.
tr
B
For simplicity, we let w = 1 in the following. To show
vec vec[B (((++)) - BB) E L(s),
note that
(++1) (++1)
12.11 [
tr(((B)) E (1+1)
BT) tr(( ((( - (((((t))BB((t)))
tr((((t)))BB;
2
tri - B (((+)) T )
T
(++
Bi)
tr (B; - 1 ((+))) (B; -
-BiBB(t B(+t))))
( [B] )((+)))),
where we use
1
tr( - )(( (++1) (B; -B( (((+t)))
2
= = (n (1+1) Bi) + 1 (
(((+)
Since (t+1) is symmetric semi-positive definite, we have
tr((;; - ( (i(()) -BiB B((t1)))
vec(BB -
((t1)) vec(Ei(t (1++ (Bi - D (((+1)))
voc(B - (I, & S (((+1) vec - B (++1) 2 0.
28

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Using the fact that B; Ui= 0. we have
((+1) [ (++1) ||,,1 12)
- - 1 ( Bj) - (B( - B(
Lr((B; - ( (1) Bi 0
(Bi B( " -((t))
[Bi B ((+1) (5]+)
((+1) - E Bi + 2 f++ 2 Bi ul(+1) (ūf+t)) u((tl))) [ B ((+11) ((11)
(ū(+1) Bi
))
[ B (
11 Bi U )
= [Bi B
(1)
((+))) (+++)) Bi - - (0(t)) - )) F+ 8+
B B (41))
(+1)) Bi + E UU - (u((tl) U
[ B
Σ ((+1)
|((+1) B (0(+1) -((+t))))
(1)
B B (++1)
( (++1) Σ -U))++
BBi ((+1)
E -
Bi. For a matrix A E R2xg and a set S E [p], As E IRS x2 is the submatrix where rows are in set S. Occasionally, we use the same notation As to represent its 0-extended version A' E RPxx such that A = 0 and
Let u (1+1) B (++1)
A's As. Then
B
12.1 - -Biil2,, = [B] + (
- B
|B + us
(++1)
+ + 1 11a,1 - [Billa,
B + (1+1)
ge
1|,11
1 ( 1|21
|Bi|la
| + ||u
(++1)
||,,1
||u5 (++1) - |
5=
= ||ug (++1) 12.1
||us
(++1)
112211
Therefore
(1ug. (1+1) |1,,1 |
(++1)
12.1) S (1) + (ii) + (Hi).
For a vector x E R2, we have ||x||2 - ||x|1 S / |||2- Then,
Σ
May
Let = 18 + ....... S. Spg),, where S + means the set of indices in S adds p. Then
||uge
||,,1 11 = Σ
(++1)
Σ
(ui
(++1)
Σ
|
(1+1)
1
vec ((+1))ss
/q
j=
And
||0
(1+1) S
11,11 = Σ
(uly
10+1)32 <
Uig
((+1) = (o (++1) ((+1)))sll sslll
iES
j=1
iES j=1
Thus, we have
((++))( 1
vec 1n (s(l1) - (eo(n((+1)))s ),,1)))))))))))))
111) < (1+11)
(1Iu (( ((+1)
2.1 - 111 (+1 12.1) S (1)+
+ +(1) + (11)).
29

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Recall that
(i) = (Bi - -B((B (((-1) { ((-1) E (Ū u((+1)))
= vec - B ) veo( (1(+1)) - -1 (1+1)) BB] veo - B vec( - uf+t)).
We want to bound E norm of vea (51 (S(+))) contains S and the largest sq coefficients of Ū ( U (++ and the largest sq coefficients of (E) b We have
- Bi) and vec( (f(+1) uf Let Si be a set of size 38g, which
(e - ) vec 0 ((+11) - )(
S vvoc((((++1) u 5 )" voc((Bi -Bi -B(+t1)
5 )
|voc(( -(++1) 55 T" voc - -B(+t)) sy))
< || veo((((((t) - -u(++ 55 )|2 veo - -B((1)) )5 )12+
|| (evo(((((tt) ((((t)) - -((+1)) 5p 11 || veo((( -B( -B(+t)) >
By the definition, we have
||| F.s =
sup HERPX4 vec (lc())) 1
vec(()).
Let R.9 such that V3, voc(A)g, s and V
0. To define [(s), we want vectors u that ||u || is bounded. Clearly
v E (s) since V| |1 = 0. Then
1 (voc(A). || ||v1 = ||ll2 = || vvvc()) (A),, 1|2-
| 2
|v1l2
Thus, v/ee( u(+tt) ا5)لl S 110 - 111 By the definition of S,, || vec(
u((+l)) ((ttl))sp>l )lo) I veo((( -U( u((tt) )s 112/ /sq S 110(++]) - u((t1)) )s / /sq. Therefore,
(ec - B vee i ((+1)) - -
< $1(0(tt1) - u((+1) |r. ()
(1+1)
5, 1|2 + /sq
(((+t)) - u((t7) |1, - (e 55ll.
Similarly, we have
B - -B((BB)t))) B( vec (5 (1+1) - - (1+1) Bi}
< veo (S((t)) - (1+1+) Bi Bi); 3, ||e vec (Bi - B (1+1)
)5 )llt
(voc((((tt))) - ( B . ||,0 /eov - -B (B((+1)
33 )112
< <|((5 - -D(+ 1) Bi F.. (ec(u((+))
S, ||2 + - ||
[1(tt)) (2+1) (x+11)
Bill
Billes
vec(u )=y
)5y11
Vsq
Therefore,
(log n) logy
slog n)²
log
[(0] < 20 com|
vec(u (++1) )5 l12 + 20
vec (u ((++1) )5 ll2.
n
n
/5
Using Lemma C.I, we have
[(t)] < (1( (1+1) - -
(++1
- B (((+1) ||
= (I) 10) - Σ Bil
| + U, (0())
Ui|l)) B - B ||
< 2no dr (0 0*) v B
B
B B ((+)))) ||
lq
dr (())) 0-) B
B
= 2k0
ved 1n 122
30

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Combine the bound for terms (i) and (di), we have
( vec (((+))))))))))))))) 1 - vec(u )s, l11)
sq (log n)² log
S 20n
sq vec(u )5, 112
n
sq
sq (logg)) log
20con
veo(u
57 |l1t
n
/sq
dr ( 0*) V B (1)
Bi
2no
/8gl
vec(u
8q
Let
(log log
dr dr(0 0*) V [ Billr
((+1) 2 20cun
2no
n
sq
Then
veof
1 (++1)) 1s;1l 111 - -1 vco(u (t+1) 1s1ll1
/89
vec (u (((11))
)5, 1,,122 +
vec( (ct(
33 1111111 + 59
vec (u ((+1))
2/4
2/9
2/9
1
5q + v
|| (+)((((+1)s
5, 11a
vec(u (t+1)
vec(u ((+)) 1l|2
2/7
which implies
ggl11 S (ec(u((+1)) )5 llz + v² vec(u((+1))
V8
2/ || vec (1+1))
l1 + vec((((t))) 111,
2
2
where we use S C St. Using || (e 1,,1:1 |11 5 /3sgll vec( (u (1+2))) 112. we have
veo((((+1))
5y I (vøq + 2//3s)|
15, 212 vsq veo(u(t+)))ls
Now, we focus on the second result. Let w 1. Recall that
(0+) B (++1)
2.1 [Bll2..))
< ir((( ((( 7 ( (1 Bi)
Bi)
2 Lr([B; ((( (Bi - -B((t))g)
tr(( (++ B B (++
= [Bi -Bi 1++
S (++1)
Bi)
(B -B((11) (((+1) (Bi - B - Bi-B((+))) B
(ū((t1)
2
follows that
(Bi - B (++1) (1(11)
(Bi - B((41))) [Bi - (((+1) (1+11) Bi (((+1) ) +
_1
(9)
2111
(++1)
2
B 2
(1T)
31

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Recall that Ej Uj= = 0. For term (/), since vec(Bi -B] BB++))// vec(B;
B ]|2 [(8) 18 we have
[Bi B ((+1) (++1)
1
Bi 0 0(++))
KBi B (1+1) (1+ Bi - -U 0
{Bi B (++1) Σ
|+
Bi E
(Bj B Uj ú h++
r + (Bi B E - r
< |Bi B B
(++1
B E | 1r. || (++1) -
S 2 B B F MM, (0)), M()))))))
where use the definitions of || ||r norm and dr.,. In the last inequality, the fact
(, u (()), Üw (0 (0('), 1,2) ( Ú(+1), E(++1), 1,2).
For term (I7), using reverse triangle inequality.
[ (1+1)
1l. 22.1
2..1
B Butty
B
1ijkk
==1
1=1
k=1
9
B (1+1)
Bi Bink)
B
(++1) Lik
-Big
j=1
1=1
f=1 k=1
$ || vec
B ((+1) Bi]l
vec
(
(++1) Bi) 111 + vec (Bt(B(t -Bi) s; ll1
v 3sgll vec B (1+1) -Bi),, ll2 + (v 2 22 vec (1+1).
-Bi)3, 1l2
v sq vee B (++1) Bi)ll
< (v38g 38q + 2/ + ) (t(++) -Bi
For the right hand side of (9), we have
[Bi - B 1) (0
(Bi - ( B(t+t))) = veo /
- D( vec(S (++11)
(B; - -B(+1))
|vee
B (++1) (Ig & S (t++l) ()) vec(Bi (((-)))
Σ ( - B
و S +1) Bi - B
where Bi and (t+1) B,k
represent the k-th column of Bj and B ((+1), Recall that
(((+1)
7 1,0 1 (X,; V.).XX".
By Lemma C.2, we know that
8q slog (logn)) 11)2 log
1,00)) (X,, Y,) - -F = =0(V
1=1
with probability least o(1). Thus, 1 n
Σm 71, (X,, Y.) Ti for some positive constant Ti with probability at
32

---

Heterogencous Sufficient Dimension Reduction and Subspace Clustering
least 1 (1). Define the N
(i: 11:01,00 X Y,) 7/23. Then by Lemma F.2,
B
1,k
(1+1)
((+1)
B
B (1++1)
inf
u²
1
1100
(X,, X;X
(5)
n iml
B i,k
1,k
inf
u
Σ
M (X,, X,)Y,)X
(
iEN
2 Bij 1,k - 1,k
1 T2/2
Then
-BiBB (1+11
((+1) (((1))
B - B(+
veo (i Bi, - (B (++1) B1)
1/2
where To = =11/2-7. Combing the above results, we have
Toll B (++1) Bill < (((+1)) - Bi P (M,, M(0))
21 ( 3sg 2 2 [ (++1) - -Bille.
Hence,
4 [ [B(++) Bille I dr,
(v3gq + 2 ((tt)) 35q + 2/50 + 2 3sq2).
To
Tu
C.2. Proof of Theorem
Theorem C.4. Under conditions (CI)-(C5), there exists constant 1/2, such that B satisfies, with probability al least - o(1),
sq (log n²2 log
[ (1) B =0 (d 0*) V B Bill V B -BilF) +
n
Consequently, for (-log k (n r 00) 0*) v [
B V B (0) B²lle)).
(V
sq (log n)2 log
B
B
B lel D 30 S3 )lp
n
Proof. We update (t) by
q" (log 2og
(1)) = KA (1-11) + Cxp
(0) Ci dr(0
B Bil v B - -Bile +xx
(3og n)² log
where Cj = (32(gg)) Thus, have
(() = K'Cr (0 0*) B Bille [
Balle
1- k1+1 +
9d (log 2 logg
C
v
1 - K
Let k = (1 V 128Cog V 4Cn) since
< K0p <
2V(250C64/ we have 0 < K < 1/2. Then define
( [2n²²
1xx² - 4x + 2
4Coy
To -x) )
64Co9
1 1-
0
(Co
Com
2x² 5x 2
To
1- 2x
480
C = 4Co00 + 1-K C
We claim
33

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
2 Kü. Ci 2 4ng. and
ACng To
Co.
16Cn9 xC1 S
iii I-R C* Ocon S O", (Cao To
+ Co)Ccon +
C2 SC To(1-x)
16Cag
The first two inequalities in (7) can be seen from the definition of and C1. For the third, we have
4Co9 + Co)Nu +
16Cug
xC1 = ACo9 + Co)No + 2t S
1
1
k + 7 + k S k.
To
32
For the first inequality in (ti), is equivalent Coon I 1-x-K0 C Since Ki S ni
1-k
1 K0 C*
-
2k C Ceoas
K
1 K
where the last inequality we use the definition of C* For the second inequality in (ti), we have
4Cp9
16Co9
4Co4 TO
16Coy [4Coon 1 - C*]
4ko
+ Co)
C^
Con)coon
TO
To(1 - 13)
To(1 - K)
4Co4
64Coy
64Coq C".
Co
Coun + To(1 - kk)
To
To k))
Use the second inequality in (i),
64Cogno
2%o C1
K
S
2
Then
4Coq
16Co9
4Co9
64Coq
+ Con
Cx
+ Co
Can +
2(1 - x;) C*
TD
To -k)
To
To x)
2k²
5x +2 2
K
C*
+
=²
2x² 4x
+ 2
2(1
k)
Next, we use induction to show the following results
(++1) 2 cCoon| (1og n²
log
4ko de (0 0-) V D
Billy ( -BilLA
Vs
d 0*)
[ (t+1)
Billov
B (t+1) B S
1 1 k+2
sq sop n)2 log
(dr
B
Bille
V
B)lls)
1 1 k
n r E C(s).
dr (
B (++1) Bille
[
-Bil= F S rΩ. vea ((+1)) w
is easy to verify that dr,, satisfies the triangle inequality. Then using Lemma C.1 and C.2
dr (M)
s dr. M(00(()))) + dr, (M, ( M()))) ())
< (o 0 V B B v [ ( B
sq (log n)2 logg
Coon
$ (0
B O
B v [ +
R2 C*
sq (log log
1 k
34

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
where we use Ko and Coon No C* + Coon S C* < (1 +*CC For ( we have
9at 1)ogg)
\ = =2 + Cxp
n
= RC1 dr(00), 0*) B 00)
Bille [B Bille
+ +11 C
q goog n logp + 4ng dr dp(00), 00) B
Bille
12 (0)
B |
2 4o V
n
V8
since (1 Cx 2 4Ccon and Ci 4no-
Note that v3sq 2/5 9q + 2 3sq² < 8 sq². By Lemma C.3, we have
B (1) BillF
2 (1 (vqq + 2/5q + 2 3sq²)
dF (Mr ). (( + TO
70
4
Ko (o (0(0) 8r) B 0 Bille V [
B2(l) Coon
sq sq n2 log
70
n
16
dF ( 0*) | 0))
Bill V BB 0 BBlle
q39 (log 2
sq²
AC1
+ (1+ (1++))xN
70
V5
71
16
0*) V B 0)) - Bill v / - Bb]l))+
Kg
+
T0
TO
4
16
8q (log n) logp
Coon +
9(1 k.)xal
TO
TO
n
(0) - Bille V 1 - B )) + 1- n²
logp
S <n(dF(O ())
1- On
By results in (i), 1g K0 +
164 xC < K. To show 1e Ccon + 10e (1+)) S (1++)*** it is equivalent to show 1/ /50on/((1) (1+
70
R) 70
16g C^ S C Since 1 1/(1 + x) 1 1/(1 k),, the result holds by applying second inequality in (it).
Let T(0) be the top-d left singular vectors of D For matrix A RPx,
|| st AlF,
sup uERPXo
(P A, u)F
sup 4E
(P u, A)
(e
|
A|I S VallAllmss
where we use the fact that vec(P 3 A) E [(s). If (e)(A)) E C(s),
| Al|r,
sup
(A,u)r 2 (A,) = -|
u voc(u))
Then, we have
| Fill 3m U (1 1 E. -1 Pa U
< ||P (, (1)) [T ( E -1 P3;
0,
| (Q
)
Eg -1 -1
[P 0,
E5 -1 P
Ui
Er
115.0
1
< |P 3 PP
P F (70)
Σ F.st
;
(0)) 0 0) ΣS1
Uilmi Σx
11...
35

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
By condition (C3), we have [[B]" Billa s [ Bille S k(dr ((d) 0*) [B Bill [B; BBllr) +
(1~x)Ma Ch (B C". For the first term in the last equality, by Lemma F.4, we have
IP 3
PBi 1F S Cs B Bille
where Cp = vd 4m (B") + 20") (B))).
Note that ( 1 E 7,,000 (X,, E. and 0 S
7,,000) (X,, Y:) S 1. According to Lemma F.3,
( E In S
1 10, S M/M.
M
with probability at least
o1). For the second inequality,
0
0))
m E Uii Er || F.
-i(m)) ||5,
(i(mi)
Mi
1
|
Uj 1n (0:0)) (0:0))
1
|Uillr [ (0))) (xi)-
M1
M1
1
M,MM2
<
| 0
(
[
-1
(11)
M
Mi
Then, there exists a positive constant Co, such that
I Fille S BI - Bill F + dF, (M)
Without loss of generality, we assume C. 2 Therefore, we have
(0
0*) v | Bille v [ -B2le
4
+ Co)d ( ((")) + Co 2 (1)) (v 3sq + 22/9 2 3sq
$ (Co
TO
TO
4
(log n22 logp
< Col
+1)
Ki 5 0-) v B Bill= V B Ballz) Coon V
To
16C
dr(O v B - Bii V B Bill
(1og n) logp
2s
+ +1++1)Cx
V8
To
16C,
$ {Cou(
+ 11mo0
(nCilddr 0*) V B B V D B)
4
16C.
(logn)² logp log
[Col
+10coon +
+x)C
TO
70
n
1- x2
q (3og n)² log
S En(d 10
( Bille v [l Bille) + 1- 1
n
By results in (i), (Caa + Co)ro + 16Ca9 S To show (4Coa + Co) + 16Cag (1 + K)Cx S (1+ k)C", is To
To
To
70
equivalent to show 4Cog + Coo
/11m//11111 (1 + x) 16Cag C2 < C Since 1/(1 k) < 1 < 1/(1 k)., the result holds by
applying second inequality in (ii).
In addition, since (0()) )0)) V B (0) Biil V | 0)) B rst,
dr (0L))
V B (1)
Bill
B (1) B
1 1- x² C",
//3 /q0 n) log p
<x(dr(0
B 0))
B r v B B
k
n
9dd (logn) logp
1uu + (1+
< rΩ,
n
36

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
since rl
1+x T-x
C*
(log n)3 log when is sufficiently large. Then by Lemma F.5, 0(1) Boon (0*).
Next, we assume the following holds for 1-th step,
qºt (log n logp + 4
dr 10 0*) 1 (1( Bille
B ((1)) Bille )
1 2 4Ccon
V
dr (0()) 0²) v |[B Bill F V B BBjl P S x² (dr 0*) v 0 Bille V [ -bll))
1 1 1 COV
sq (log n) logp
1
n
(0t)) 0 [
Bill V I Bzlle S rQ2 vea(T rw) L()).
By Lemma F.5, 00) E Beon (0)).
Then
(log n
n + 4no dr )0) V B
logp
Bill [ -BillE)
4Coon
vs
(1ogg Ploggn)looppp..
4no
4Coon| V
(d(p)(((0) , v B Bills v |B, BBlls)
n
V
1 - ktt1
89p (log 122 logp
1 K
n
4NOx (dF(
,0) V B Bill V B - BBll))
V8
1- kt+1
qdt (log n2 logp
(4Coon + 4/50
C*)
--K K
(dr (O()) 0-) v
B Bille V B B)lle)
1 1 222
( n) 2 log
I SnttlC C1
Ox1
1
n
((++)
Use (i), 4kg - C1x. By the definition of Cx.
f+2
1- x C2
4kD C*) 2 4Ccon
4ND 1- ++
(4Ccon + 1
+
C
K
- K
1 - k K
k(+1
2 40con +4%0
C
1 - K
Then note that
dF. M,)
(M M(0)) dr, (M, (0")) M (θ())))
sq s 2 log
V B Bill B Bīll)) - Coon V
n
ne+1 O³
sq (log n)2 log p
(ds 1 v 8 B V B -Bale) +
s <ku
/q0g n2 logp
Com
n
1- R22
sq n) 2
log
S e+
( 0 [ - -Bi P [ - B;]l))
+
O"
1 -
n
37

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
I-n't since No S and Ko I-x
C + Coon I I- C³. Then by Lemma C.3,
1-x++2
(++1) - Bills
dr. (, + 16q (=x(++1)
T0
TO
4
n: (0 00 0-) v [ (0)
-iill V [ 0 -Ba(()+ + 1 x++
1-x C* slog n2 logp
Kip
+
T0
(log n logp
169
V h 1 (ds (0 0*) v B Bill V B
B
Cooa!
n2
TO
1 - +2
T
(log n²² log
O2
1 K
72
4 K0
16
qkCi (dF(
V
V [
-Bil
F | (0) B:ll p)+
70
TO
4
1
1-
1-
16g - k²+2
sq (log 1 )a logp
Con
+
-
No
C +
Cxly
70
TO
1-K K
TO 1
n
1 x2 0n Bill V [ BBllz) +
3q n² log
S (dr
(0) 0*) v B (0)
n
n
In the last inequality, we use ( Coq + Co)No + 16 CoqnC1 S K. and
4Co9 + Con +
4Co9
16Cbq
1 kT42
1 12
+ Co)
C
C 5
C
TO
-
k
To
1 - K
1 - K
4Co9 + Coo) +
1
1
x1+1
16Cug
1 - k +t2
1 - 12
(32
ki
+
Rk)
C² +
C5
C
TO
32
4
1 K
TO
1
- K
- k
ACo4 + Coo
16Cng 1 - 1- l42
1- k
142
9
k
1+1
Cx <
O*
C*,
To
To
1-K K
1 K
32
1-K
where the right-hand side is greater than C and thus the inequality holds due (ii). Using the same argument,
|
((+1))
Fill s Colli[( Bille + (M. M(0))
Therefore,
dr (++1) 0*) V B ((+1) - -Bill [ -
-Billr
4Co
(0))) + 16Co9 (t+(+ NO 1 (d (0)0) 8t)
Co)dF, (O')),
TO
4Co
+
Co)
B ())
Bill
B ())
B]ll))+
To
- 1+1
8qp (log 1)2 logp
sq (log )a logg
Ceon
-
16Co9
V lettic (d)( 0*) V
[
Bille B BbllF)
1+2 1- xt2
(logn)) log
+
C.V
To
Vs
1 1 k
4Co
16Co9
$ I(
+ +co)no+ +
(dr
0*)
V
B
BBil V |
B [r)+
4Cb
4Co
1-
16Coq 1 - +22
sqa (log n) log
Con
Co)
0* +
1-k Cxly
50
TO
1 k
To
<x--1 55-
(d)(())))))
0) 0*) v [ Bi
F
[B B)
+
1 1+2
C
sq (log n)²
log
1 - K
71
38

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Further, since
1-x x and r52 T-x 1+x
C*
2 []ogg))loog logg
when is sufficiently large,
"
dr(0))1) 0*) V | [ Bille V [B2
((+1) B
1 - xt+2
8q (log n2 log
I 1 (dr(0 0 V [ Bi F V B 0)) -B F)
1 C*
S 1 1 22 C*
(logl n2 logp
k.42
S K ++ rs2 +
r5 S r22
n
n
1 + ki
When 2 log x) log(n(dd 0*) v [ Bille v [B -Bil
S K loR, (n(ds (0 ""vvi B B)))
n(d=(00),0-) Bill=
F v
[ (0) B)(l))
which implies
[ -B BI = - n) log
(op(logg))
With Lemma F.4,
PP Pgge Pæ: l|F
(Vorplogg)) (1 log
318 S3:.)
S Ca/ /d/
B B =0
2d
7
D. Proof of Lemma C.1
D.1. Contraction of weights
In this section, we show |m (0) (00)) ko(dr(0,0") v [B, - Bille v |B, - BBlle). By definition,
(0) Ti (0*) = E
(X, - -1,0- (X, Y.)).
i=1
39

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
When y4 is fixed, we can further simplify the above. Thus, for given i, we bound [B[nee (X,, Y2) 71,00 (X,, Yi]l- Let 5" (m,, vec r2)², vec(T2 FT ),) A , c.. E" + uAg. Then Eo 5", ç. Then
Blm,o(X; Y.) -1,0- (X,0 (Xi, E
OyI Yi)
)dh
oE
E=
Du
an (X, Yi)
An,)duu | {1/ 2 vec(TTz r1) <=Eu
an, Yi)
E
dr.-r,)ddu
θTi
E=
(X,, Vi)
E
E-E." (rrsttiduu
2 Te + rT)) E=E."
E en. (x,, Vi)
(TI - -ii) + 1.' (1l ( veo(T, [2]] ==
dn.e( (x;, Yi)
A rd
f=
(Fl Tn (Xi, Yi)
Argtr,)du
vee 22 + 11)
&=
dn (X,, Y2)
< sup (C
| u +
mm
(1)
dy.ell x Yi)
sup EEB
E
lll2 1 - r1 +ill
① vec([T T2
rí
(11)
@yl.e (X,, Yi)
sup B (0*)
E
llle [T r - r Til |F.
1 vec F1)
(1I1)
Thus, we bound the three terms in the last inequality. Recall that
71. (X,, Yi)
T1 (1 11) exp X,
] (T2 [T) [.)..}
We can decompose X, as the sum of two independent random variables, X; ~d Z + ubf, where Z (O, and independent of and W., Plub = rī) मं and P(*h r मं. Let 0(T) + -T= + r2)/2. Then
1 (T2+Ta) 2+ 2
x, T22 + r.)ff z of,
Therefore, write
m,e(X; M1 (1 7x) ((T) T))ET
By calculation, we have
On d (X,,
cx (Z (T,
Oz
(1) + (2-00))) (Z+
dy,. (X,, Yi)
-m m1) (m + (1-1 (1
Voc((T2 - -T))
i Te + +r)) = (1 (1-m) m exp [Z + (T2
0n,e (X,, Yi)
1 8 (12-1))..
(T (1 - (()
Let T, (T2- r, and T2 8(T).. Let H, be an orthonormal matrix whose first row is T T 112.. Then H,T, [T]1200, where e₁ is the basis vector in the Euclidean space whose first entry I and zero otherwise. Then
(T - r,) = T T T HF' [T
(10)
40

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
where V is the first coordinate of V H Ip) and is standard normal distribution. Then
n X Y2)
exp (Z + T2)
(T + M1) (xp (Tf T2)))²
exp( 2| + T2)))
El (m + (x- 22 + TTT2)))
where E1 standard normal distribution. Note that
T = | (T2 - 11) 2 ((r( - [1)²) 2 a0?
where 61 c(1 - Ca). Similarly, we can show T2 S c252²
Recall that f2 = Vtr((Ts - ri) [ = | [T ri)" Ir. have
Ω2 veo((T) ア² vee E$ ri)²)
- L @ Σt) vec((Tz )
which implies
/2/M2 - r3 - r1 & /2/M1.
Then, when M, <1, 1.
T V ( [r[(T] - TI (T2 - r1)
tr((( F [1) -13
///2/11111 II)
2 - (T; r1)² |5 (T2 - + Fi)lF
2 cl 2/ M2Cil) 2 cz
for some constant that depends M2 and C.. Similarly, we can show V T S cΩ.. Define events
& [I]) ||22||
C1 n2?)
On the event &, ||/T/l12111 + T 2 T we obtain
T 2 02 Using the tail probability of normal distribution,
2
P(E)) s 2exp(
< 2 expp
T
8c
Then
dy, (X,, Yi)
E
iTi
exp(1 T |126| TT)))
=E
( + (1 - 11 exp [ 1/2 + )2)
exp(I
112 + TT(T,)
[T] + (1-m))
. e T 12 + TTT2)))
cxp([|T]
1 2 + TTT2)
E
P())
(T) + (1 - m1) exp
T1 22
+ T T 1)
1
c1S?
c202
exp(
exp
Tf (1 イ }o]
2
2 min f [ (1 m )2
8c7
1
1
2
z exp(
exp(
2
20
80
2
2
exp((
n?²))
(11)
2
802

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
We proceed to bound (IT). Note that
1 (X,,, Yi) Ti (1 T1) a (ve T22 - -1)
expl (Z- o(T))G)²(T, F.)..
f @ (Z + ((T).)
(T) (1 T1) exp ((TT)))((T ())
dn (X,, Yi)
(X,, Yi) &
fi 2 Z
ŌTi
(17.6)
(11.6))
By definition of H.,
60 Z = fT @
H.Z (f 0 H (f. s H.).
For the first term, we have
e Yi)
exp(1 T 112 V1 + TT)
fg @
= E (fs
(m (1 mi) exp | Vi + T2))
exp(I || 11 + T())))
(f, 2 E
V101
(m + (1- exp | 2 T2))
where the last inequality uses the fact that V1 and Vj are independent for any 1 イj S and B[V = 0. Then
exp(1
T 12 11 + TT T2))
E
T 12 + T2))) Vi
[T1 (1
11) exp
exp(1 T
+TT,))
1
E
T 11 + TTT, T T2)
( + (1 (1) exp(
T(T
ll2
exp(1
[T [T 11
1
E
(I 12 + TTTT2) & P(E))
(T) + (1 - m1
[ T
|T
11
exp(11 [
112
1
(1 + TT2))
E4 P(())
(xi + (1 イ1)
e 11121 + T
| 11
exp( TT 1| 21 + TT))))
E
T TT
(α) + (1-x)) exp 122, 2))
|T1l2
For the first term in the last equality, using the fact that on the event &
T c/ /2, it is bounded by
2
3c, 1
exp(
[ (1 n )2]
8
The second term is bounded by
exp( 2 1
min (1 - 12
8c
For the third term, we have
exp(1 T 11 + TT))))
T T2
(m)
+ (1
TT) exp(1 T 11 + TT))))
TTll2
2
exp(1 T1 12 + TTTT))
T T
< T
E
W, =
(M] (1 m1
exp
TT 22 + T
[
w=1
2
1 C1 ^ d
< exp(-
102)
2
8c7
42

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Since (f, @ H e vec(T T l2f ), it follows that,
2
3c, Ω2
[((I.))] < (f @ H je 2|
exp(
+
min N.. (1 m )2}
8
202
1
2
min (1 TT }z
exp(
8c
Cy5l
+
z
exp( (
2
A
804
29
2
2022
3e1
c
< || Ivee / T/il2f 112(
V
exp(-
^
8
861
[llll(( 2
202l
3C1
V
exp(
(
A
Gfeal cff
86g
202l
3r
e (0p)
S Ma(
exp(p
(12)
Gell
8
8c2
We proceed to bound (IL.i). Note that
T2 +
r1
T1
r
T2
[
+
|| S 2
| - Tille- + 2 | - -elll
2
2
2
2
2
1
000
ri r F
2
1
1
1 U2
G.0+ +
Ω (C.
2
M/M
M/M
and similarly [Tz Kronecker product,
F || S (Cs +
1 )2. Therefore | W. S M3(Cb + 2/M,
1 )20??. By the definition of
M/M
f.. 2 8T W w Σ 2 (8T W. W.S. Mf(C + 1 222??
S
2/
where fij is the j-th element of f,. Then
In.el (i, Yi)
B = FE
f 8T))]l
Om
$ x
exp(
^
M
+
2/M M.
4
2
8c
Therefore,
E On.e (X.,, Y2)
|l S (1 - m] [( + [(((T.i))]]
i vec(T) T2 12)
M4
2
2008
301
A of
<
V
eepp -
)s??)
4
G
2
8
8c
1
M&
+
Mexx((
- (1)
Cl
^ 2
200
2 M
2
8c4
301
2
S c3 exp(pp
^
n22))
(13)
8
8
where cs- = max|
M4
4
2
V
2000.
cfcs
)
200
1
Mz(Cs + 1 10b
2/M,
For the term (IIT), note that
T
f
T = 11113- - & MG002
43

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Then, have
E n, x; Yi)
1ll2
vec 12 + T21
10 - (f)- 1 (
E|| mi) ( + (1 T1) exp (Z (1 (12 lll
(1 m))
Tyne [X Yi)
E
|| @ T2ll
ÔT
2
exp( - exp(-(
(1 A
ct
8 c
8c1
Mcaa
exp( ( A c
)
(14)
4c
801
Combing results in (11), (13) and (14), we have
El,,,(X,,Y)) V.) 70 (X,, Y.)] S (xx(dr(0,0') B - B v [B -Bil
(15)
where Kx = ( + +
Mecdl 46
(30 A 2 1022)
D.2. Contraction of matrices U.. We aim to show
(0) U,(0")lle $ (0,
[B] - Bille V [B, B)(l)).
By definition,
(w (e) E( Σ (X,,), - - (X,, X)),,
When is fixed, X; are not identically distributed. Thus, bound the expectation of given Let & (π1, vec(T2 F vec(T2+Di)), Ag = - E. E" +u- Then Eo E" E g. Define the Jacobian matrix J 0f/0x E Rmxx as Jig 0ff/Oxz where x = (#I,.... "T and f (fi...., fml )r, Then
vec(E[(i.. Yi) 71.5- (X,, X.] veocmn,e Y.)X
veo(Eu) du
=E
θ vve((E)
$= Ou
I vee((71. (Xi, Yi),,
E= A An,uy )
F.
① vvccmLe P?)
&= Ar.-r, du +
@vec[T, - -1)) I @vec(y.e (X,, Yi)
E
Arstr, du
@voc(T2 I1)
E=
44

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Therefore,
1l Y.) 7... (I)x, }11F
$ Vol
(efhs(Xx; Yi) - n (X,, x,,"", Hl2
@vec(n),e (X,, X;)
vill sup
E
A 112
(0))
@voc(n,g [1 X.)
vgll
sup (E
E
Ar--r,le 112
vvec [ l1)
voo(mi (X,, Y.)
Voll
sup &E (0*)
E
Arztr,l2
& veo 22 + Ti)
7 sup (O-)
(X,, Yi)
vec" (x.((")
||2|=| - +
Ōm
(1)
am.e (X;, Yi)
/q sup (E
|
veo(T's - r() vec. (x.f()
||2|1 rī
(11)
0n.e (X;, Yi) vvo([T2 F2)
/ sup (EBr.(0")
||
vee vee" (X,{())
||2| rī + T2 r
(T11)
Note that vec(X,f?) vec[(Z +ff)f (f, (Z) vf.)). For the second term (II), we have
0n.e (X,, Yτ)
(1)1-m1) d vee T2 - {1) vee (ocT(XXf,))
T1 -
(Z ((T, - -
(mi + (1- Ti) (x - [2 (f (Z))2+800)) vec (x,f.)
(X,,,(X,,,)) @ (1(11)) (Z) (Z + +b,) (f. 11)
(X,, Yi)
(f. & I,) zzz (f) 1,) dn.e(X, (X,, Yi)
(fe 1, Z (T 11)++
Om
(II.))
(1I
dn.e(X,,Y) Y.)
(f. @ L))(T)ZZ (C. & I,) (X, Yi)
(f. & (TT)) (CT @L)
Om
(TT.66)
Recall that T, (T2 Ti)f, T2 5(D),, H. is an orthonormal matrix whose first row T T 11, V H.Z
45

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
(O, I.). For the first term, we have
B[(T..)]
Dr Yi)
= (f, 0 Ip) E
ZZ"
(P,T @Ip)
mm (1,, Y₁)
(f. @ I, H E
H [(f H,
e Xi Vi)
(f: @ I, H
Vr @ H,
dT
112 TX
+ T2)
(f. @ b
VI" & H. (by(10))
(1 11) exp
T 2 2))
exp(1 T 12 V1 + T2))
(f: @ H Ei
(V? el ( 8 H1H
[11 (1 - 1 expl
Ta 12l12Vi TT
(11.1.0)
exp(1 T 1/ + T(T))
(f. 2 H
L & H,
1 + (1 - π exp { | 12 T))
(11.i.))
Recall that the event & [1]]T]l/211 I 9 } and on the event &, || 21 TT 2 |T |111 N c,523/2 /.. Let
[T 2 + T2 g(t) (m Then
(xp(t)
[[I
exp( T 21 T2)
El
(1- exp | 11221 T2))) (z) 1>1
+
<
("
ITiLE )E)P(E)))
Bb((((
|
(| + - (
+TTT) + (TTT2)² IT ({8))({))
Elg(t)(
|
[l((())) +
(3)
+ Elo(t) 2TT }
IT
{Eig(())(15))
-
(TfTT) Elg(t)
T2t -
} P(E
1
1
4 exp( 1
2
<
(2) + (oe + c))0
exp(
exp(
)(²)) +
2022
min
( i (1 ক }3
2
8e1
30
4
(c) + 2) 4T - m1)
20g02
expl
0??)
+
min(TTi (1 ]
8
og
min
- (1 11}
2 2xxp(=
min (1 - ]1}
8c²
+ 2 + 2000 + 40g!?
8
(c] + c2) + 20
expl
exp(
80y
8c7
1+2(03 + 2 + +
8+ s+ 102
Let C3
We have
12
a (o2).
< <cxppp
(16)
46

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
The term (TT.i.) can be bounded same (11),
2 expl-(22 A oî )
(17)
% Scy
Then we bound the spectral norm of ((T.i)))
E((II.i)]ll
s (ff @ L,H e (f H1HHHl22 (((LLa)))
(I(f e I, H, (1) [[((LA)
s Mil [(LLia)) + M]
s Mfcs exp( ( 0 ^ eी
n + M² 2
exp(-(
6 A of
8c3 )09)
2 _ exp( - s M2 +
-
^ )0
(18)
4 80y
The other three terms are easier to bound. Note that
e Yi)
E[(IL (f, @ I,) El
Zt 20 1' [f. &I))
dn (x, Yi)
(f. 60 H E
Viel 1f +b (TT II).
dm
Using the similar technique in (12), we have
||l| @yL (X,, Yi)
Oyi. (X,, Yi)
Vieif wb² 112 S [E[
V1 Mr
2
20252
301
S Ma
exp(
-
c/
8
Therefore,
[[(Iii)))]
2n,e Yi)
$ (f. e I, ll ||E|
Vio ffa""" 2b "12112 lll2 -.(( @ 12)2
20252 efcss
(30
dt
s MbMb
cexpl((]
(19)
geoh
8 ScG
We proceed to bound (IT.it)).
(X,, Yi)
E[[Iii))] (f. @ Ip) E
8(T)EZ")(E 01)
dT
) (T e 11)
= (f. 2 I,)
dm
Using the similar technique in (12) and || 12 < MC + 2//T
2222 we have
0Yi. (X,,, Yi)
8(T) Villla < Mc 1
2
202Ω²
3c1
&f
E
V
exp
^
)
M
c/
c
8
804
Therefore,
< [f. @ 1,)l2 || (X,, Y.)
8(T) Villl HH, & 12)22
d
2
20,
301
< (C
+ M/
)(
exp(
(20)
cees
862
47

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Finally, we bound (TI.iv),
IIif s Ip) El In (X;, Yi)
UT
^ o )0.) 8c7
S MC +
} c
expl-122
(21)
M/
Combing results (18), (19), (20), and (21), we have
|| E (X, Y.)
2 ve - T1) vec" (x.t") ll2
s ni (1 T1) [((L,))) ||[(4 | [[(((1.))))ll
+
$ 4 Mf + 2
expp(
(exp(-(1)
al ^ eी )oa) + M
v 20252
exp(
-
300
^ ot
) +
2
4
8c1
Gesl
cjes
8
sc8
1
202l2
301
MfGC+ +
)
V
eepp (xp(-((
A
)
2/M1
caess
8
8c4
ot )0?)
MAMoCC& +
exp(
^ (
61
2 M1
2
<ctexp(-
9 ^
)!
(22)
861
where = Ma + ( + M2
Mo
2
V
2020 Gecs
+ MM + 2// 1 r33
V
2c
)
cffs
M1Mo(Ch+ + 2MM }
Next, we bound the first term (!),
On.e OX,, Yi)
vec? vecT(xe (x,,77) 1l2
E
On
Toyye (X,, Yi)
E
(f. @ (p) (p + vf.)) 112
@0n,ee (X,, Ys)
(x,,((,,, Yi)
< (f(,, @ I,)) E
z
| @ I,) E
ubf. 112.
Om
Om
(1.6)
(1.is)
Similarly to (12), we have
@0n,e (X;, Vi)
(I.i) = [I(f & H. E
V
1|2
exp(
Ti 2/ T())
= II(ff & 1, H e E
V1 ll2
(T1 + (1 - m1 exp
T V1 + TTT2)}}²
< M4(
2
2
exp( (- 301
8 ^ of
102))
(23)
C3
According to (11),
(I.i) < M exp(
2
- c1
^ &
)o?))
(24)
of
2
807
48

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Therefore,
Tn. (X,, Yi)
E
vec (X.f.t)
112
θT
202l
3c1
2 103 + Mā expl-
o
c1 c
S M4
exp(
)0²)
)0²)
Geg
8
867
861
3C1
S 07 expl
)S2)),
(25)
8
8c4
where 07 = M4
2cx22) eges
Ma 3.4 + M3
For the third term, using (25) and the fact that ||f @ (T2 - -1)ffll S Mcca2,
Ing (x,
vec "oee((x (x.t7) ll2
E
Vve 22 + I))
Ti (1 Ii
I11. (X,, Yi) vec" (x, Ill2
(T2 F E
2
In.e (x)
[c @ (T2
-i)
|
vec" T (X.f?? !]1|2
M4caa
30
s 8 C7
exp(
(26)
8
Sc1
Combining results (22), (25) and (26), we have
[E((,,,((, Yi) - 11,5-(x;; X;) }|[
$ {v (0, { co { Sgeep((() - c ^ c
)
+ C7 exp((
3e1
^
&
)
804
804
Macal
3e1 e7 exxp(-11
o7
102)
^
8
8
862
Macas
301
^ 2
S (idr(0,00) +
exp
-
n??)
8
8
80
< <xu(d=(0,00) v B1 Bill
B2 Bille).
(27)
where KU volco + o7 + Macal c7] exp(-(23 3e1 ^ 807 117)
D.3. Contraction of covariance matrices We aim to show
[0) - (O')BB & kx(dF(0,0") v - / BB2 - -B:ll).
By definition,
(0) E = E( 1 Σ
Un (X,, Y2) -n.s-(X, (,I,),
49

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Using the same argument before, obtain
El X K.) mh. (X,, X BF
$ Vgll Ei (X,, Yi) - -1.- (X,, X)) Bi
voo(71 (X, Y,) X, Bj)
I Vall
sup
E
A.. 12t
(O))
0m
v x X Bj)
Vall sup (EBon(O')
E
Ar.-r, 122
Ô vve 2 r
veo(Tn,e (x, X Bj)
sup vgl.
E
Arstr,ll
① (ve + +1)
BE (0))
Pen,e(X, Yi)
99 sup BE (0*)
E
vec" (x,x, Bi) 12/ +
dTi
(1)
(X, Yi)
V
sup (E
|
Tvec[T2 -T1) vec veeT (X,x((x
|2/ ri- - P2 Till +
(X, Yi)
sup (EB)..(0))
E
@ve r1) vec T (x,x(xX Bj)
11--7 ri + P2 -al
(111)
We focus on the second term (II). Note that B; = Bi)" & I, Then,
1 rm.e "X,, Y.)
vee (X,X((
#1 (1 11) (e r1)
exp((Z + ((T) (.),,
(6 (Z 8(T)f.) vec (x,x(B)
[Ni + (00-
(x,,((X, Yi)
Zm (f, (Z) ((T)() (Z vf)" ((Z + +f, 3d B I,).
(28)
We can decompose the last line into sum of terms. The term involves three Z is the most complicated one. We consider
in,e(X,,Y,) dī
(f. 8 I,) ( & Ip).
Recall that H; is an orthonormal matrix whose first row is TT/ TT [|2- We further require that Bi, E span(H,,1: H,2)), Bi. span(H., H,,2,, H.,..).... ..., Bi.g; gpan(H,,1:.... H.(g+)):), where Aj: is the j-th row of matrix A. Thus there exists matrix igpx such that Bi H A. Then
z² = (H, HH Bj =Z HHHHHHHAAV
= = V
4+1
Aq1
Vg2
A V M",
i=1
j=1
j=1
50

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
where AsA is the (j, k- element of A and V the j-th element of V. When j > k 1, Ayk and |||| | S
vdjlBil2 = VdM,M2. Therefore,
0r,,,(, Yi)
(f.& If, @I,)
dm
@x, Yi)
(f. ( II
an,e(X, Yi)
@ Ip H,) (M" & HHL))
ÔTL
(X,, Yi)
(f. & I, H. (M, ,
M, H,),
where Mj is the j-th element of M. Therefore, we have
(X,, Yi)
E
(f. & 1,) ZZ
[z" Bi & 1
In.e (X;, Yi)
= (f, & H
E
(M H, M M,H,)
exp(1 T 112 V1 + T2))
(f. I E
VV H,, M2H,,... ..... H,,HH............
11 + (1 ní exp
Ta 1/ + T2))
((1))
Note that
Big(Vi)VV's M&]
A2a
V1VV
E(((V))
+
A1s ViV
dex ViV3
Ash V/3
A
0
Vi3 ASa V/V33
V
A1a
)
A( 17a ViV2 0
A(--11a 13a V
Therefore, for each k; the matrix [[g(V)) Mr] can be written as the sum of a diagonal matrix and matrix with 2k non-zero elements. Then, the x pq matrix .,,,,..., , M, M,H,)) ]] can be written as the sum of two block matrices (F[J.) + F[J2)), where each block has size p x p. In the first matrix E[J,], the k-th block is H,H,, where D is a diagonal matrix. And in the second matrix E[J2], || [|]]]]] |F
the k-th block is A₂ H., where A is matrix that only has 2k non-zero elements. Since || [[]2]]la [[[[A,..... Aq)).. ll,, each element of E[As] only involve E[g(Vi) which can be bounded with the same argument used in (12). The total number of non-zeros elements in E[(A1..... A,)] is q(1 + g). Then we have
E[(AHH,,,.... ..... Allll
3c1
[[Ja]]la < caq(1 + 4) exp(eppppppp
A )
(29)
where es is some positive constant.
Next, we bound || E[Ji]ll2- Since each block of J, is diagonal matrix D2 times the orthonormal matrix H,, rows of Jı orthogonal. Then we can construct its SVD F[J.] FDKT in the following way. Let F E RRPP be the identity matrix, D is a diagonal matrix that elements are l2 norm of rows of J1. and KT is normalized E[J1] where each row is divided by its &g norm. Therefore, || |[J, equals to the largest l norm of rows E[J,]- It is easy to see rows E[J,] and rows of (DD..... .D)).. have same l norm. Thus, we only need to bound the largest lg norm of E[(D..... D2)]. Each row of E (D,,... D2)] has q non-zero elements, which contain either VV or F[g(Vi)
Since F[g can be
51

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
bounded similar (12). We focus on bounding Elg( Recall that the event {1lIT 5 and on the
event &,. [|/TT|122 + Tf T2l 2 T2
T 21 2 c /2. Let / = || | Zi + T2 h(t)
exp(()
[ exp
Then
exp( [ +
El
(TI + (1 (1- mT) exp e T }) 2il
(1 12 + 3 3(TT) T2
IT || - 3 T T (TTT))
= BTh(t)
||
3
(l + TT
||T1112
||T111
BI + TTTa)
|T2|l
The second, third, and fourth terms can be bounded similarly as (12), (16) and (11). For some positive constant we have
(TT(t) T2)
+
Tz T
+ |T
E(h(t)(TTT.))
T
| 1l
C1 A & )(7) S cg exp(
861
Using Lemma F.I,
E[ 11 + T2
T
exp(t)
exp(t)
E
1 (EE))())) +| E
1 (189)
2/2
(T1 + (1 51 exp
(T] (1 m1) exp
1
8
900
4
expl cis
exp( exp(--2
minn(m) mT. (1 Ti }3}
min ( प.. (1 ग )2]
807
8
exp(
c00
) +
4
exp(
00
<
exp(
-
C1
)(29).
8
8c
2 c
8
8c)
Combining the above results, we have
ot
E[(((v) < (cg +
exp(-(
)S?
cf
Sc
Therefore, the 2-norm of [[ (the largest l norm of rows J is bounded by
ci
E Illl ex- c ^
(30)
Sc
Combing results (29) and (30),
(X,,, Y.)
E
(f, & I, B; @ L,lll S Mr[esg(1 + + coog] exp(- ^
82y
The other terms in (28) involve Z at most twice and therefore can be bounded with the same technique in (11), (12), and (16). Therefore, we have
d...e(ii Yi) Vee([2 Ti)
|
vee" (x,x(x ||2 S cnqq exp(-
(31)
82
for some constant C11-
52

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
For the first term (I),
e Yi)
vec' ( = (X,, Yi)
X (X
Bi) @ Lp]
θTi
ÔT
Tn.e (Xi, Yi)
(Z vbf,)' +bt)) Bi Ip),
and the third term (III)
in.g (x, Yi)
0 vee T2 {T) vec? (X,x(xX
(1 - TI) 0 Xi, Yi)
(f: @ (T2 - (T2-Tr)ff) ved (X,X'
2
Om
T1 T1)
0r1 (X,, Yi)
(f. @ (T2 - (Z) + vf.) ((Z + +ff)' Bi @ Ip).
2
We see that both terms only have ZZT and thus can be bounded similarly (11), (12), and (16). Finally, for some positive constant 612 we obtain
d n? )2)((000))
[B[[,,e Y.) m.e. (X, X;X S exp2 - ^
864
Skx(dr(0, 0*) [B: Bille V |B F)
(32)
where Ky 112 exp
)(2). 86q
E. Proof of Lemma C.2
E.1. Covering number of C(s)
We state lemmas that used later.
Lemma E.1 (Rudelson & Zhou (2012), Lemma 11). Let ui...., uM RP9, Let E conv(u,..... uMr) There exists
set L E [M] such that
|4| S 4 Ja |
and vector y' conv(uy,J L) such that
l - y/12 Se.
Lemma E.2 (Rudelson & Zhou (2012), Lemma 21). Let 0, x E RR49 be vectors such that [101]2 = 1, (x, O) * 0, and is not parallel to x. Define & : R + IR by:
(x + Xu, 0)
<(A)
| + Aull2
Assume (a)) has local maximum at 0, then
(x + u. 0)
||u//
2 1
(x. 0)
||x||
Next, we show the following lemma
Lemma E.3. Let 0 1/3p, and d = 26883sg3 then
C(s) 8P4-1 c 2 convv( U Ey(p) n 18P4-1).
(33)
1
where conv denotes the convex hull and Epp = span(e, j J).
53

---

Heterogencous Sufficient Dimension Reduction and Subspace Clustering
Proof. The proof is analogous that for Lemma 13 of Rudelson & Zhou (2012) with some modifications. We assume d pq, otherwise the lemma is trivially true. For vector E IR2,, let T denote the indices of the 38q largest absolute coefficients of u. Then |u 1| S ||u3 |1. We decompose vector E [(s) n SPP-1 as
uy + uy- uuy + KV59 2qv 2gv3s) us l2 + /sgl|u|2) absconv(e) : :JE T)),
where absconv denotes the absolutely convex hull. Since
|
u7c 13 < |ut||i||uT= 1mx ||uge|1
3sq
| 1
< [[V3] +4 24) u5 1|2
v /eq
|u|2
v3sg
||uT|22 1|
s [[[ssg + 25)
uu 11 /sq||u|12
v
[[33/3 2 |7 v3/3
112
we have
||u|| 2 ||u7
12
| 11 <
v3/3 + 2
UT v3/3 |1T122
Let ||u|||2 (2a))) 2 12/ /(4a²) + 1. Then we have ||u 2
We are interested in finding range of that satisfies a²x² + bæ 2 1, which is equivalent to (ar
+2
20
Define
v= (uT + [[v5] sq + 24v 3s)
1l2
s0 [1l2]] absconv(e, : (T)): L(s) n nP
We have ((s) c V c. [s) and V is compact. Therefore, V contains a base of C(s), that is, for any y E C(s)/(0),
there exists À > 0 such that xy E V. For any nonzero vector v E 1P4,, we define
F(v)
V
|
Then function F' is continuous on [(s)/(0) and V. Thus,
((s) = F(C(s)/(0)) = F(V).
By duality, inclusion (33) can be derived by showing the supremum of any linear functional over the left side of (33) does not exceed the supremum the right side of it. Since L(s) = F(V), it is enough show that for any 0 E SP0-1 there exists z' E (0) such that supp(z') < d and F(z') satisfies that
max 0) S 2(F((2))))) vEV
(34)
For given 0, we construct d-sparse vector z' that satisfies (34). Let argmax, (egnaxxee(F(())) (F(v), 0). By the definition of V, there exists set I [pg] such that [Z] 3sq and for some 1/j {1, 11],
z = zI + [(vog + 2g/ 1133, lla /sqllz|aa] Σ a
JEI
where aj E 1), Ejer 0j S 1 and ||z/||2 2 > a sparse vector with szpp < 3sq Let z'
a; = for some E Ie, then 0; for IT/(i) and z. Clearly, (33) holds with d 3sq + 1. the following, we assume
2n
a; (0,1), Vi I To use lemma 1. denote Cpp+1 0, 1pp+1 1 and set
Σ aj.
0pg+1 =
jEI
54

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Then y Zge E conv(u "iMin where u, [[ysj 2gv3s)) 55, lla v m M = U Ie U (pq + 1} aj > 0). We define appt1 because the sum of coefficients must equal in convex combinations. According to lemma E.I, there exists set J C.M such that
|0 S m= 4axxJEM |uy122
c²
AmaxjeM((Vsg + 2qq 2g (//336)1z3 [2g ll + /sqlzz|2 ||
6²
84 + 12s02² + 48qy 3q S
+ sq 28q + 48qy3q
Asq + 8 34 + 12s
16sq + 32sq /agg///qrem + 4889²
E²
2²
120sq² 2²
and a vector y' E j E J')
y' = 1(v/q + (g//3)]lz3 12 vsg|z|12] Σ
Bynjez
JEJ
such that Σjeg 33 1 and
|yy y S e. Set z' zy y'. Then z' E Eg, where J (IUJ)
[pq] and
|J| S [ + T' 3sq m. We have
2 - 1l2 = |z - zz y'12 = |zz== - y 11e = lly y'lla S..
For (33 jj E J'} as above, we extend it to {8, j E I U {rg + } by setting 3, = Off E U U - 13/(3'} and write
z' = zy + +(Vsq 2g 2g/ 35 1l2 + /
Σ
Bym
JEZ-U(P9-11
where 83 E [0, and Σ (jez-u(rg+)) 11 33 = 2f = z', then (34) holds naturally
max(F(v), = (F(z), 0) و 2(F(2)). = -(((())))))
"EV
and d = 3sq m. Otherwise, for some À to be specified, consider the vector
z + A(z' - z)= = zI + [(v5q + 2g/ 3s) 1z5 la /sqlzz12l
Σ
[11 - A)a; + +A3ylngej.
jEI
We have Eje (1))))aa + A33] 1. There exists 8g > 0 such that Vjj E U 1), (1- X)a, + A3, E [0.1]
if | < since
This condition holds by continuity for all j such that ay E (0,
If a, = for some j, then B,- = 0 by definition of M.
Therefore, we have Σjez [[1 -))a) +)ay+ A8,] S which implies z + X(2' - 2) E V. Now consider function (-50, 5u) + R.
(()) = (F + X(z' - -2)), = (z + A(z' - 2), 0) 1z M(z' z)112
Since z = argmaxx veni
( (F(v), ()),)) 0). ((A) attains a local maximum at O. According to lemma E.2,
(z', .)
zz + (z' z). 0) 1 |zz
zl12
|z 12
Iz' z||2
{2, 0) zz
||z||2
2
55

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
follows that
(F(z').) 0.
(x' / 1 2 0)
| la
(z', 0)
| 12
|| 2
zz'
z
2
(F(z).0) 0)
(z/ 0t/l2/112. 0)
|z'|2
{z, O)
|
2 2|
||///2
||a||2
||z
z|12
|
1z' 2 22
| + '|z - 2112
2 l2 +
1z - 2 2
1z 2
2
| 2 + &
||a|/2 +
We know |z/2 2
||zz 1z2
2
+1 2a
where a² + v3/3 + 2/ and b v3/3. Let 11
we have
6a
(F(z'), 0) 1 (F(z), 0) 2 2
Therefore, we construct sparse vector z' such that (34) holds. To derive have
120sq
m <
22
Note that
4a + 212
2hy 2 +
2
36a
4(1 + 2 + 2
2/3/3 4(1 2 +
36(1
2/3 - 2/3/3 x 1/3
9x
where x = 4(1 13/3+32/// Since q 2 1, x 2 4(3 + v3/3). By the derivative
d
1
2/33
- x 1/3) =1
0, when x > 4(3 + v3/3).
dr
3xx
Substitute x = 14 into the numerator
10
1
1
1
1
9x
x²
16(1 + v3/3 + 2//)²
16 14g
2244
Then, we have
120sq
m <
< 26880s0
22
Let Mn be the cardinality of 1/2-net of L(s) n nP--1 We want to bound Maes which will be used in the later proof. With lemma E.3 and using the same argument in Rudelson & Zhou (Settone H.1], we have
Mma $ (1 (1+2/ 2/
pp d
5epq )), s 5d( (pp epq ya exp(dlog( d
d
Let Ca 26883,
log(Moc) < d log(
Jepq = Casqa log(
5epq Casqu
= C2sq llog(
P + log
5e
< Cnaiq log( P sqa
d
sqa
Ca
for some Cue > 0. when p > esq² for sufficiently large If we want to eliminate log 5e/Ca, log( (g)) 2 log 5c/Ca. That the reason we require p esq².
56

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
E.2. Concentration of the matrices U
Recall that (0) = - 71,0 ( Y.). U,(0) 1.4 E B (X,, Xf We want to bound
WU = sup BE
[0) - U,(0)llE..
By definition, we have
WU
sup
sup
(X, X,) n
E X, u)P
(ee( (8*) 71
(n (X, Vi)X, - (X, X,) f, u)r-
sup
sup
BE
Define
n
W sup
(n, X)); - (x),,. E., u).
BE (0*)
71
i=1
Then WU = (up W We use an e-net argument. The first step is approximation. Let voc(u,).... vec(uM...) be 1/2-net of ((s) Spa-1, This means that for any C(s) n SPe-1, there is some in- dex E [Mnel] such that ||v u, < 1/2. We have
Wy S WU + IWU u,
1
s max WU ME|Mal n,
w I|u v/1F
max W u,
W.
JE[Mal
Then MU sup, W < Ma.) u, +1/2WU, which implies WU, < 2 max W Then in the second step. we bound the tail of WU for fixed j. And the third step is union bound, where use the covering number of L(s) O SP9-1. Let {ei be i.i.d. Rademacher variables. Recall that
T (X,,X, Yi) (1-xx)exp) exp [X 1/2(T, + (T) F.).)
(x.)
(1 σz exp Co.y (X.)(
57

---

Heteregeneous Sufficient Dimension Reduction and Subspace Clustering
Then according Lemma S.5 in Wang et al. (2024) and Hölder's inequality, have
E(exp(AWu,))
E(exp(A
sup
(X, Y,)x, Blm,o(X, X,),,) of (y)r))
BE (0+) n i=1
Efexp(
sup
(11,0(X; X; El,,,(X, X,) fft u))r
n B (8r)
2X Elexp(
( X X.X.fff ا
sup
n
BE
1=1
Elexp(
2\ sup
E + (1 m exp Co. (X,)
m
(X, uz)E + (x uidr }/11
OE 2nn (0")
E ( n
4X
sup (0*)
TT1 + (1 क. exp exp((). - (X,f u,)5
T
i=1
E [eep( sup (0-)
4A
n Σ i=1
Ga (X, ny) 1112
n
(1T)
exp(2] ग. is Lipschitz with constant 1-xi < 1-co and (0) = Since Y, f, are fixed, (x, a function defined for random variable X, Let I' = = + (1-1)) r and Z, X, r f, be the centered random variable. Note that Z, ~ N r'f I,) +
To bound (I), we use lemma C.I in Cai et al. (2019). The function (xxx)
(1-+1))g
(1~ I.
58

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Therefore
(7)²
< Elexp(
81 - 00
sup
Co., u))
Co
BE (0))
82 Co
Elexp(
sup
c((X; 1/2(T2 -1))) ,y)r
n
Co
BE
Elexp(
8A co
sup
- +T) (T2 (T) u2)=/11
CO n
BEB
Eleex((
82 Co
sup
(2 - 1/2(T2+ - (i)E)((Z + TT u5)r )]
Co (0*
32X 6o
G (Z,, (T2 - [Z, u)) -1)))),
< Elexp(
sup
n
DE (0*)
i=1
(i0)
Elexp(
32X Co
sup
6/ (T 1/2 Fi)f, (T2 ( 1p)
Co
(0
1=1
exxp
32X co
sup
Σ (Z,, (T2
n
uz) 11
n
cu
00
i=1
(tf)
32X 1 cu
6 (T*, 1/2((T 4,),, (T2 D.G u))
Blexp(
sup
Co
0EB...(0') 1=1
(66)
Note that
T2 Pill== = [T2 - -T--T, - T1) + + Fille
S S 12-15
Fill + |T |Till
(35)
20,0 +:00+2000 2M
We can bound [[T++T] 1i+Fi with same quantity. Therefore, We like to bound
(T, < M((2C((2MM)) 2C,M2+2M5
T2
r 2
[[T
(T2 Pill < [
|| [e[[T T1lla
2
T2 +
s <[[ +
|| (2C + 2M)
2
s 20C + 2.,. + Mc)
20302 + 60,M + 4M3.
We know that |A||2 = maxx,y x" Ay/ A ([x[||lll))))) For (i), we have
[f - 1/2(T, D.),, (T2 - rT))E)
sup B (0*)
- (T2 11 [.))
sup
BEB
sup
(1= +T)) - (T2-11)6))
(0-)
||0])(22 + 6 4M3)
sup
(0
59

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering Let Cn (20 60,M, 4M²) max; || Therefore, (i)*
=Flexp(
32X 1
co
sup
GTT -
T.) (z.f. us>e
7
co
Bc (0*)
Elexp(
32A 1 co
sup
(T' 1/2([2 + ri)f, (T2 T1)E (1)))
sup
Σ
"
(Z,ff uy)
co
Be (8*)
BE
SCn
32A 1 - Enr
< Elexp(
sup
E, (Z.f u,)r
cn
BEB (0-)
i=1
Let that
Z.1(W; = w) = ? F/W, = w) = vec(u,) @ IW, = =w).
Since Z,(WW; = w) NT - Ip). var Z (W, = w)) = || vec ('f & 1,) - M2. Therefore ||c.Z. (e; = w) 11 =
S C.M.. Since e; is independent of Z, Ele, Z. (W; w)] 0. Then, by equation (5.12) in
Vershynin (2010), the moment generating function of I,Z, = w) is
Eiexp(tc,Z.,11))) = w))l s cxp(CMM3))
Then, have
E(exp(
32X 1
CD
Cn1
C (.. u1)F
n
i=1
=mmmxx
exp(
32\
6o
Cn
6 (Z.f, uz)r, expl
32X - c0 En
c (Z, .u).f
n
Co
= -E/max
exp
32A 1 - 60 Cn
Σ
Z.), exp(
32 c0 Cn Σ <zZz) ]
n
Co
1=1
i=1
< E/exp(
32X 1
cD Cn
n
(Z.)] + Elexp(
32A 1 co Cn
Σ GZ))
n
n
Co
n
c
X2 322
0o)2
X2
2eexp(
T2 CM3) = 2exp( 4Cn1),
2
n
n
where Cn 3 (1-0,)² 2 02 Then
32A co
Σ ZZ. u)) <///4 exp( - Cn))
(i) < E(exp
sup 0. (0+)
(36)
Co
When 0 E B (0*), veo (T...... m
D E L(s), which implies vec(T..) = vec(T")) + vec(uw), vec(uw) E ((s). Next, we
bound the second term (ii). Note that
(T'ff uz)rl
fff, < [
vec
vec(u,)]l2 vec vl [|2
=I((fff.' I,,) vec([T 1| S [|f,ff lla vec(T")]l
s My 1 < Mf max [..
60

---

Heteregeneous Sufficient Dimension Reduction and Subspace Clustering
Then
(it)* = Ejexy E/exp(
32X 1 co
sup
Es (Z (T2 - (T) u,)=
co
BEB (0*)
1=1
32X cD
< Ejexp(
Mb max ||f:112
sup DE (0*)
(ZZ (T2 -r.)f()) )]
Co
< Ejexp(
64X cD
M1 max ||f.112
sup
(ZZ, (T² +u))f.) 1112
n
co
OE (00
64A 1
Co Mb max 1
Σ (Z, (T) + 11)
Elexp(
sup (0
n
co
< E exxp(
128A Co
Mp max |
e (Zi r 126) 1144.
1=1
(i.11
Eleep((
128A co
Mn max ||G:
sup
Σ
c (Z, u2 u2f p2n
Cu
(61.2)
Elexp(
128X 1 Co
Mp max ||G:
Σ
n
e
(Z,, rit 1144.
n cD
(i..3
Eleep(
128A
Co
Mb max
||G
sup
Σ c (Z, un fi 13
n
co
vod (
(65.4)
We first focus on the term (ii.4)
Elexp(
128A 1 Co
M, max
|
sup )E((s
6s (Z, f.) |]-
n
Co
Again, we use the e-net argument. Let vec(ūi).....
veo (Me( be a 1/2-net of C(s) n SP
sup Ee)())))E(s)
Σ 1=1
e (Z, uifi
sup EEC(s
|
Σ 1=1
E ZZ; u | lal
/
n c (Z, uf.)
sup vec()))EC(s)
| ||
sup
vec(ū]e (())SSp--
2 sup vec(ur)e((s))
|
max
Σ (Zi ū,f
jE[M
< 20, max jE jE[M
Σ 1m0
(Z, ū,fi)
61

---

Heterogencous Sufficient Dimension Reduction and Subspace Clustering
where in the last equality we use vec(u; = ve vec(TT) and [[] -Fille Cys. Then
128A 1 - Co M₁ max ||
Elcxp(
sup (EC())
c (Z unfi )
n
Co
El (exp(
256 1
co
My max
max JE[Man]
Σ
n
c (Z, ū, |]
π
Co
E e
256 1
cD
Mp
max |1.11,0022
(Z, ü |
n
co
jE[Mal
E| Elexp(
256
1
co
max
|| [30
Σ e 2 )
11
co
M
E22
where 2, (Z, Since El,, w)] ex)(CM3(P)
0 and var[Z,[(W;
w)] s || (e = =w]] s
Therefore, we have
n
(ii.4)
Σ jE[Mml
Elexp(
C2 72
Σ
A expl
Σ Z,), exp( C - - EaZiji
E[max
Cppp n
jE[Mal
i=1
Elexp(
X C12 n
Σ
n
G 2zZ,a== + Σ
Elexp( C2
Σ
c 2.,1
n
JE[Mal
i=1
jE[MAml
i=1
X2 12 CM) = 2.M exp
X2
< 2M i=1
exp( n2
CR.
4C12
X² + S log( P )).
< 22exp(4C12
n
sqp
Let 2, = Then Ele, Zi = =w] O, var [ar[Z,l(s = =)] S Mi With same the argument,
(11.3) = Elexp(
128 - ccp
Mr max | Σ c ZZ rif }
Co
< <F(exp(
128A 1
co
Mb max
||ft 1
Σ
n
G (Z,,) Elexp(
128A CU
Mb max ||6:
n
Cu
n cu
1=1
1=1
X2 1282 (1
< 2 eexpl
2 CMM
4012
Then we have
(i..3 (ii.4) S V2ex(((C12 + C12) + Cma/Asq" log( P )) sqs
With the same argument, we can derive similar bound for (ii.1 (t..2). Then, for some positive constant C12.
X² + 0ma/8sq" log( sg "²2)). 1).
(ii) < 21/4 exp(C12
(37)
Recall that for a bounded random variable X E [a. b] almost surely with E[X] = 0, E[exp(txX) < exp(te(b - /)//8) for any E R. From earlier derivation, we have ff uz)rl S M3 max; ||.
Thus, Ele ,,))r] = 0 and
62

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
fff u
M My max, fll My max; [[ which implies
t MM [max, [[6)1]22
Elepp(te uy)e]] exp(
VI R.
2
For (iii), we have
(idi)*
32\ co
= =Flexp(
sup
G 1/2(T2 +T)e, (T2 - T.) u5)r
Co
(E
i=1
32X Co
< Elexp(
Cn
sup
(T uy)r }
1
cD
BEBim
32X 1 6o
321 1 00
< Elexp(
Čn
(T'f u))rrl Elexp(
C1
c fT'f... u,)r)
Co
Co
X² 512(1
2
< 2expp(
Ch MJ)
n
%
1C1s
Thus
X2
(iii) s 2//4 exp( C13). 32/4exp("
(38)
7
Recall that ve Then we proceed to bound
ur - C(s). There exist u
Rexq such that vec(Tw) = vec(TT)) + vec(vee (w)) and vec(u,) L(s).
(iv) = Elexp(
32Á - Co
Zi (T2 - (Z, u3)=
sup
co
i=1
< Eleep((
64A co
sup
Gi (Z, (2f,) (Z,?" 12>
n
B (0*)
Elexp(
64A Co
sup
G (Z, 1 (Z. u,)F 111/2
n co
(0-
iml
E eexp(
128A 1 Co
sup
(Z. 1, .u)F 11/4
n
Co
BEB (0*)
(io.
Elexp(
128X 1 -
sup
(Z, uzfi u,) =
Co
B.. (
i=
(iv.2)
128A - Co
(Z, TT 1
E exxxp(
sup
Co
B (0*)
(i0.3)
128A Co
Z u f.Y. 112/
Elexp(
sup
BE O
(fr.4)
63

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
For (iv.4), define
Wn =
sup voc (x)
Ex Zi, (Z.f, Uz)
km
sup vec(i))E(()n880-
(vec(u),
G voo(Z.ff) veo(Z:f))") vec(u,)),
where we use
(Z, ūf.)) = 1.' i tr(ff Z,) (ū, 2, )r (vec(ū), vee (ec(Zif,))
Let
W (vec(u),
4 vec (Z,f, vecl poc(Z)f,)))) voc(uy))
and (ūi,..., ".M.. } be a 1/2-net of of [(s) n SP4-1 For any [(s) n 1 let üا be one of the closest point in the 1/2-cover. Then by definition,
v üu
||v ux112 S 2 E (s))
v üg
Therefore,
Wv, S Wann + Wan. Wv.ul
Wi + ((voo(ūx v)/l v112. Σ c voc(Z,f,) voc voc(Z,ff)"), voc(u,)) ūN -vl12
max kE[Mul
1 < max W + W1 Ma)
which implies Wa S (M Wū... Then
sup BE
(Z un (Z, U,)F
| Σ 1=1
e, (Z un / |1 ff)) (Z,f," .u)F
sup vec(ui)E2(s))
I C.sl W. S 20,0 max W kE
where we use su (U))
|n ||r = veo ( ri)ll S Cyfl. Then
128 Co
(bv.4)* < Elexp(
20 max kE|Mall
Win.
Co
Co 20, 128
Elexp(
(Z, (Z, ,
n
co
kE[M
Elexp(
128A 1
co 20,0
4 ZZ ū f.) Z.f.. F))
CO
kE(Mal
To bound G (Z,, f, (Z,fT, uz)F for fixed ü, and uy. we use lemma D.2 in Wang et al. (2015) that states the product of two nub-Gaussian random variables is a sub-exponential random variable. Let || lle, and ||. ll.s denote the sub-exponential and sub-Gaussian norm. Note that (Z,, f.
w) and (Z,fZ" 1y) rl,, = w) are normal distributions with zero mean and variance less than or equal to [f. For X NN, g²), ||X lv2 I C.. For some Cpp > O, we have
||, (Z, f.) F 12) F (W; = w)lle = [[, f.f ( , = u²)l llv
< Ca, max (1, u. f.) (W, = - 1Z uz) (W, = w
I Can (CL, max | + d/ max (
I M....
64

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Note that (Z, ig (Z, U2)F (W; =w] 0 since e, Rademacher random variable independent of Zg. Accord- ing Lemma 5.15 in Vershynin (2010), we obtain
128A 1 -0 20,0 o (Z,, ū,f (Z.f uz)r)l exp 1282>² (1 (1-co)² Elexp(
40 Mf).
co
n²
C
when
128A 1 20 S
C²
co
C... max; [ful
Then
X
1282 (1
co)2
(io.4)* < 2
Σ
exp
40
C
Kr [
1Cu
X2
X²
exp 10
S 2 exp(
40n
Cvesg log( P
n2
π
For term (iv.3), using Lemma D.2 (Wang et al., 2015) and Lemma 5.15 (Vershynin, 2010) again, for some positive constant C, we have
128A 1
Co e, (Z, (Zf u,)
X2 exp( 20
E Elexp((
n
Co
7
when
128A 1
co
71
00
C² + Mb) max; |
Therefore,
X2 (tv.3) S 2expl 4C14). n
We can bound
X2
(vv.3) (iv.4) S V2exp((Oia 014) Cna////// spp).
-
The analysis of 1iv.1) 1) (iv.2) is similar to (iv.3) (iv.4) and we have
(iv) < 1/4 exp(C14 X²
+ Cna/8sq² P sqe
(39)
Combing (36), (37), (38), and (39), the bound for (/) is
X2 (1) < (i) (ii (iit) (to)) < V2exp(Cr + Cna/8sq" P sq
(40)
where C₁ = (On + C12 + C13 + C14) /2. It remains to bound (IT). Using the same argument, for some positive constant
CIT. we have
(TT)² E(cxp( n BE (0*)
2
sup
Ga ( u,)r }
E(exp(
4A
sup
n
G (Z. ff u,ye 11122.
n
B
E eepp
4A
sup
G G (T u2)
AEB
1=1
2
2exp( n
2011).
which implies
X²
(IT) < 22exp( Cn).
(41)
65

---

Heterogencous Sufficient Dimension Reduction and Subspace Clustering
Combining the results (40) and (41), obtain
X2
S (I)- (II) - 2 expp Cu + Caa/ log( p )),
n
5q2
where Cu = C1 + C11. Thus,
Elexp(AW/U) E2x max Wulll jE[Mal
Σ
Elexp(2AWW
ME(
412 Cu Cnei/ /8e// log( P.
412
Mma exp(
)) S 2 eexp Cu 20 log( P sup
n
4X2
< expp(
Cu + 2C s0 log(p)).
Using the Chernoff bounds, we have
- -i(0)-U(00]1r:: > = P(WU >1)
sup ( (0))
4X2 (() S 2exp( Cu 20nes log(p) A1).
Let A Vnsq log(p)/Cu,, 1 (2Cnet CTsq" log(p)/n. Then
P(WU > t) < exp(( - log(p)) s (exp(-log(p)) o(1).
Recall that we require
128A 0o
- 128
sq (gglog()) cp
S C
Co
nCu
co
Therefore as long as n 2 C" sq log(p) for a sufficiently large C we have with probability at least - o(1)
sq log(p)
sup (ū] - (O)()) $
(42)
B (0*)
n
Note that we can get sharper bound when sqa = =(p).
sq log( xg
[0] (2)0)lles $
sup (e*)
n
E.3. Concentration of the weights Tu
We proceed to bound supeeB...
(1))) Recall that
1 71,e(X; ())
Ebn (X,, Y:)),
71
n
and
71,0
c + (1 - T1)
exp (X; 1/2(T2 + P.)E) F.)A)
Co.((X.)
66

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Let W* ... (0)) (0) (0)]. We have
"]exp(AW*))
= Elexp(
sup
T1 21 (1 11) exp (xpp(Corr(X.)
El,,,0 (X;, Xill |
B (0*)
S Elexp( A
(X,,, Yi)l
sup
T1 + (1 - Tι) exp(Co.x(()
ni B
E 1311
Elexp(
sup n2 Be (0)
11 + (1 - nι exp(Co.y(X.))
Apply Lemma S.5 in Wang et al. (2024) to (2),
2X
a
(1) < Elexp(
sup
Ei
exp Ca.y x:)) }]
π
(0*)
1 + (1 Ti
4A Elexp(
T + (1 - n exx Ti m1/1118.
sup
E3
n (0
4A
Σ 4 1112
E Exexp
sup B
(16)
Note that (x:) le 2019) with g(-) 1, we have
is Lipschitz with constant
I-w < 1-ca and (0) 0. By Lemma C.I (Cai et al.,
8A
(i) < Elexp(
sup
(X, - 1/2(T +TI)E) (T2 T
BE (0*)
i=1
161
(Z (T2 - 1122
E Elexp(
sup
Be (
(4.1)
exp(
16A
sup
Σ
TT
ク (T2 + r,),, (T2 112
OE Ban (0))
1=1
(1.2)
Terms (i.1) and (i.2) can be bounded similarly to (37) and (38). We have
(i.1) S 22xpp(2 X2
2sq/2sq
P log(
sqp 2)))
n
X²
(i.2) S V2exp(2 Term (iz) can be bounded easily using properties of sub-Gaussian random variables,
) n
(ii) S V2exp(Cis ))
Thus
X² (T) x (i) (i) < 2cxp(C, + /aa//Asf log( P. )),
n
67

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
where C₁ C11 + C + C13. Term (IT) can be bounded similarly. Combining the results, we have
X? n + Cnersq log( P 80
Elexp(AW>\] & 4exp(C
Using the same Chernoff approach before, have
X² (0) - (0) > 1) P(W* >1) S 4exp(C= (000)-T(00))
P( sup (0+)
+ 1 At).
n
Let À v (osg2log(p)) log(p)/Cr..t (Cnet + 2) S log(p)/n. Then
P(WU t) < 4exp( log(p)) &
o(1).
Therefore, as long as n s" log(p) for sufficiently large C", we have with probability at least o(1)
sq log(p)
sup (0-)
51 (0) (0) <
(43)
n
Still when sq² o(p), we have the following sharper bound
sqe log( L
sup (0) - (0) 2
DE E (0*))
n
E.4. Concentration of covariance matrices
Last, study the concentration of Σn. Recall that
(e) = XXX (0) = 1 2 EX, XX,xXXX,X,XX
Directly applying Lemma C.I (Cai et al., 2019) converts the problem into bounding the product of three sub- Gaussian random variables. While is well known the product of two sub-Gaussian variables is sub-exponential, the product of three or more sub-Gaussian random variables is not necessarily sub-exponential. Therefore, we must use another method than the one used in the concentration of U... We use tail bound for unbounded random processes given in Theorem Adamezak (2008).
Let 2 supeeB... (0+) (() BillB, By definition, have
W
Σ (X,,,X EE,,,(X, ()X, u.
sup re
sup (0
n2
Let
W
Σ
(1, X Ebi,o(X X)XXX,IBB; u)
sup B (0-
Then W sup W.S. Let vec(u,),... vecf vec( (Mmm) denotes the 1/2-net of L(8) n SP4-1 For any 1,19-1,let let uj be one of the closest point in the 1/2-cover. Then by definition,
E L(s) 1,
u E C(s)
NSP9-1 1|v - 1--u,1222 - ~
- u
Therefore,
< max W:E + 1 W je Man]
W & W + IW
W
2
68

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
which implies W < je|Mal W.G. u, We proceed bound W for fixed u, E L(s) SP4-1 To use Theorem (Adamczak, 2008), define
u,
(X.,,)) = =)) x) u,)s E X, Bi uy)rl.
We have
1 sup
n fX,, Yi)l
: 71 BE (0")
It is clear that for every 0 Beon (0*) and every i, (X, Yi)] = 0. Next we show (X;X/Bi, uz) is sub-exponential. Note that
X Bi uz) = voc(u)) )" vee X Bi) = vec vec(u,) 8 x,X,) vec(Bi)
Bi, Σ X Ug (X, (X,, Bi
where uj,k and Bi, are the k--th column of u, and B Clearly, (X,, uz,e) and (X,, Bi are normal. Then
X 1y S (Z,, u,a) + (1(bbf, ll S Cluugalla + C",
and similarly
[X,, Bi Ml & ,
Therefore, by Lemma D.2 (Wang al., 2015)
XX,XX rll S
(X,, (X;; (X,, (i,,,)))
max uy,a)l || II Bi BiN)) S'' <o.
The above results hold for any 0 E Booo (0)). Then
|| sup (0-) [s(X,, Y:)ll
=|| sup dE (..)
(71,0XX Y,),,, Bi, u,)r (0+)
sup
El( (X,, Y.)X.X 1))r]]le
sup 0: (0)
(x:X, Bi us)r llv +
sup (0
EXX Uz) 11
<000
where we use the fact ( < n1 (X, < 1. We verified the two conditions for Theorem 4 (Adamezak, 2008).
Define truncated function and the remaining parts of f(X;, Y:) as
A(X,, Y)) fX,,, T
sup (
( YY) p),
12(X, Y.) /X, Y.) 0c BE
sup
[[(X,, Y:) p),
where 8 Smmmax suPoeB... (0)) (X, Y.] Let Q max; (X,X F.) Since (X.X, u)) sub-
exponential, (((XXXX Bi, Fl > x log n) < 2 exp( log Then
P(Q > 1l < Σ PXXXXX uy)el > r log S 2exp(=cr log n logn).
69

---

Heteregeneous Sufficient Dimension Reduction and Subspace Clustering
Therefore, when log n
have
/
2 log
00
F[Q] P(Q > t)dt
> (Q>ttdd
P(Q t)dt
2 l
P(Q> > t log n ()g(())gg))
c
2 no n
+ 2og
exp(-(ct- 1) log n)
2 2lggn n
+ 2lgo n
exp( ct + 1)dt
2 2lgg n log exp(-1)
C'log n.
Since the above holds for any 0 ( we have C C'logn.
Note that
sup O B (0*)
(X,, S sup (0-)
fiXX, YJ) (X,,
sup OE (0))
(2(Xi, Y,) - EX, Y)]l,
where we use the fact that Eifi] + E[f2] = O. It follows that
E( sup oc (0*)
IXX, Yi)I < El sup (e...(
( EX,, +OIII+
2m1
2m1
2 E
sup
f2 [-1]
AC B (ot))
i=1
By Markov inequality and definition of /2(X;; Yi), we have
P(max kSn
sup (0*)
f2(X, Y.)l > 0)
i=1
E max; suPoeB... If(X,, YiIl
< P(max sup (E.
[/(x,, Y:)l > p)
P
which means
④ Σ
lg = inf > 0; P(max sup ksn (0*)
f2 YDI >) 8
=
1=1
Then, by Proposition 6.8 Ledoux Talagrand (1991),
1
E| mmax K (e
sup
f2(,, Y.)l < Elmax sup (f, Y:I < p S Clog n. (E*).(0")
iml
Thus,
log n
E sup (e*)
Σ im1
fz (X,, YJIl
< E Bmaax
sup
Σ im
fx Y3l < C2
(44)
K
(e
70

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
When (0.)) [ YJ) S (X,X/ u is bounded. We proceed bound /1, by Lemma S.5 (Wang al., 2024) and Lemma C.1 (Cai et 2019),
E sup
Σ fi Yi) (X,, XII
(00))
n
S CE sup
Σ (X,
1 (T2 +Tif,, (T2 IX)) u,)F
AE (0*)
n1
1
S C' E
sup
(X B
ni
iml
(1I)
where (X;X? m bounded for all i. Under the condition (X; Bi, Uz)r is bounded, 6 (X; (T2+Tr))f (T2-
T.)E) (X, Bi, F is sub-exponential for any I and r22 Define set 7 = {t: (t:t0 t" = (vec(T), )², vec (T2) ) E
Boon (0)) Using the argument to derive [(s), we have T C C' convoUUUsste E2(4 8:p4-1 ) where d = Casq". By the definition of constriction basin, we have
veo(TT) vec(T s vec(TT) vec(TT))le + voc(T)) vec(TT)ll s 20,2.
Therefore,
[I(]vc([T]) - vee(TT) [vec([T) vec(T22)
< / vee(T)) - vec (1) + ve)(T2) (e2((2))l
< || vec(Tr) - vec vec(TT) - vec(T2))l2
< 4C,2.
The diameter of T
D = diam(T) = sup d(s,t) S 40,2,
a,IET
where d is the &a distance. Note that
sup- EET
Fl
[x(;
" (T2++T)) (T2
1
u
2
1=1
Caa Cq s Cgle" gl c'o'0
< sup IET
2
1=1
where we use the bound of moments of sub-exponential random variables and Stirling's approximation. By Corollary 5.2 in Dirksen (2015),
(1) C3 ( (T. d2) 1 (T, di)) + Ca (1++))
where (T, d and (T, d2) are Talagrand functional (see Dirksen (2015) for details). Given & E T, let K,, G(X;
(T2 +r),, (T2 - TT) X X Bi u.)F. Dirksen (2015) define
1/2
1
di(s,t) max || Kr, K dz(s, =
|K,, Ka, 1
1=1
Consider the two metric spaces (T, (2) and (T. d1). We have di (s, S Cplls t||2. Then by Theorem 1.3.6 (Talagrand, 2005), (T di) S (Pm(TT Similar result hold for (2( d2). Therefore,
(T) < Csp (
1 7e. 62) mT( (x) + +Ca
1
1+1)
(45)
vn n
71

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
By equation (4) (Dirksen, 2015),
Ta (2, Co / (log ((T, Ca l(ol0 (log
where we use log(1) = We know that
2TT 1+ 2 d 229
1 1 + 2) [2pp
d
d
Then we have
1 n(T,22) SC Sc SC
log N (T, e)de
sc ! Sqp log
1+ 1+2) sq log
2ep Casq?
de
S Csg log
p
&Csq logp,
5q7
when p > C'sq²² for some constant. Similarly,
(T(T, S." (log (TT
C S." /sq0log (1+ sq log
2ep Casq2
de
$ C s
Then according to (45),
sq logg
sq logg
sq 3 n)² log
(I) S C log n(y
S C.V
n
n
n
when n > sqe log(p). Combining with (44), we have
E sup
- Σ f(X, Yill s S.V (0'))
(log n²2 log
C2 log n
S <CV
sq s4 n2
log
i=1
where we use log(n)/n < log(n)/ S Vsq log (oolglolo(n////)) /n. To use Theorem 4 (Adamezak, 2008), we need bound
0² sup
Ell f(x_ Y,)
sup (e
EHJ((,,
BE (0*)
nt
Note that
E[[[(X, Yo)3
= E[(no, (X, Bj - 0/1 X,, XXX,XXXX 1}) = -Elmoo(X, (X, 4; E X) B p]
S F[(((XXX u F| B ur)
since (X uz) is sub-exponential. Then o² < C/n. The last term to bound before applying the theorem is max; s.. (0-) ( Y.)l Use earlier results,
max sup 1 [x, Y.)le S C- log nll (0.)
sup (0
[X,,, Villl
C logn
C'logn
sup (0*)..(0')
XXXX,Bi u 1 n
72

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Let Wi, suppge....0" (0-) 1 Ei=1 (X,, Y.) Then, for 0 <1 and
0, we have the following result
P(Wd, 2 (1 + 17) Fl +1)
t Cl max; Bu (1, Yi)
$ exp
+ exp
2(1 8)02
(- (-Cont
Cont
exp (-Cnnt²) + 3 exp
(olo log
Let 1 = Cty
sq (log n)² l where C7 is sufficiently large. Using union bound, we have
P( max Wi,, (1 17) E(W.,) +t) jEMa
(-
Caru
Maet exp (-Csnt²)
3.m exp
log m
exx
Cneisq log p
Csntt?
exx
(Ceass 2 log P
Cani
sqp
sqe
log
4
S exp(-log] - 3 eexp(- logp)
p
when n >> Ossq" logp for sufficiently large Cg. This means with probability at least - o(1).
max Wi, (11+ (1+ + C.V sq n)) 2
1sg (log n²²
logp
S Cop
JEMa.
n
n
Recall that W S F Then, with probability at least o(1).
sq (log n)² log
sup (e
(1))(()) (0) w S
(46)
n
F. Ancillary Lemmas
We first present some technical lemmas that are used in the proof.
Lemma F.1. Let (1())
(12-6)) [w+(1-w)etps 12, fa(t) (w+(1-w)etjpr fa(t) (w+(1-w)e and fa(t) (w+(1-w)etpz- Then
e¹
1
Si(1) 4min w. =m]]] E R
(1() S min 1 wmm
exp(-a) va
12a
(f(()) < 2mnnf ? (1-
E R
2
sup (f2(t))
exp( xxp(-3a/4), Va 0,
min W. (1 w))z
4 + 12
[fa(t)) < min (22, (1 - ww]
Vt E R,
4 4+ b²
sup (f((1) < min (1, (1 w)) exp(-a/2) Va 2, 0,
(f((t)) min (1 - w1-w)2
Vt E R
8
sup [fa(t) min min (w, (1 w))
Va 2. 0.
73

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Proof. We results (C.1) (C.6) in supplement of Cai al. (2019) prove. Since S1(1) (we- (w=-v\=(1-w)ev33 (tt2)). By (C.1) and (C.2), fi(t)= z
4 (w" (1-w)33 and
min w. 1 w/2 exp(2 a/2). sup (t( = sup (tt/))
1za
1/22a/2
Since [2(1) we 1/24(1 w) 2f2(1/2), using (C.3) and (C.4), have
1/2
[f2(t) & 2)f2(t/22 < min(w?? (1 - w}2}
and
2 min { (w, (1 - w)} exp(
3 a
sup [2(t)] = 2 sup
[2(t/2)] v
12a
1/22n/2
22
2
3
sup
[fa(t)] sup |fa(-t/2)]
min w. (1 w) exp(
22
t
-tza
Note that fa(1)
(/o--/24(1-ww))) (1 4f2(t/2). Then, by (C.5) and (C.6),
21/2)2((b/2)
(b/2)2
4+0 b²
lfa(O)= 4|fs((/2) < min (2, (1 - w/ min (2 (1 - w)
and
4+
sup [fa(t) = sup [fa(1/)]
min { w. (1 w)} cxp(-a//
t2
1/22a/2
Define
Fa(t) [we ++(1 (111wway w)
Then fa(t) 8f (t/2.. Note that
83
1 4 wn (1 w))2
[54(0)= (we-t + - -w)eet3
min (w: - w p(e et)2
which implies
2
[f((t) < min min(w, (1 w) )
2 Twe #Hl1-weepp I (sp(-a/2) when 2 a 2 0. Therefore.
Then note that
8
sup
[fa(t) = sup [Fa(t/2) < min w. (1 w)}2 exp(-a/4). 1/22a/2
t2e
Lemma F.2. Let Lx(s)= = L( The restrictive eigenvalue condition
T Σ uEC,
inf
X.XTu > >0
i=1
holds with probability at least o(1) when n > Cslogp sufficiently large positive constant C.
Proof. Let C(s) conv (U)i Ey(p) n s--1), ). d = Cas. According to Lemma E.3, we have C,(s) n SP C C(s).
We show a stronger conclusion that
inf nEC(s)
u
Σ
X
u >
im
74

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Since X, 7 A + ( A). have
F|X,] f F T + T2 x = T (TT) + (T))
Ax cov X A Ti (Ti) + -r)f
(47)
For any V E C(s) n 8--1, Z, X, is mixture of Gaussian and thus uub-Gaussian. Then Z v X v is sub- exponential. Thus, V
v) vl I C X, Li. Let max; L₁. By Bernstein's inequality (Theorem 2 in Vershynin (2018)), for t > 0.
X.X v LV" E[Xx'1
}
t² D n² L max; L
21 2 2exp(-cmimn
n
nt² nt L L ) - 2exp(= 2exp(-cnnt))
2xp minn
where the last inequality holds when 1 S I, and eı = c.2.2. Let Vi,.... Mii is e-net of C(s) n SP-1, Then according to Lemma E.3, log [J] S CJs log where CJ is positive constant. By union bound,
P
X
I L EXXXTTV
}
(2, s 2| exp(-
$ 22xp(Crslosp-cmm))
Define 11 X.X.. Then, we have for all j E J.
F - 1 |/ v12 <
E E
with probability at least
2 exp C logp cnn
Assume C1 S Tmin (A) S 0 max
A - Cz. For any unit vector we have
v " (T)) s max (2) S M2
Since B| A A A + +
C1 V- E < S M,M,.
Therefore,
Va- -15 / CC2 M2M3 + t.
Then, for any v C(s) (SP-1, there exists Vj such that
v v- 1/v V|v- vyllv Sc
iv-vvlla CC(s) s--1
and
2/2 v,l12 |/ v;)ll2 1/2 / 1/2 vz)ll2-
The right-hand side is upper bounded by
V C2 + M2M3 +t+c
sup vEC(s)
|//22 12,
which implies
|//1/2//22 5 V C2 + Mr +1
sup vEC(s)
75

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Meanwhile, the left-hand side is lower bounded by
| 1
(v-)(v- 2 VC -
| |t//2v//
sup
vEC(+)ng
2 CVC C2
M2 +1
which means
'V CvC2 + M2M3 +1
inf ve
|| 2 VC
1-c
Let C = C2 + M2M,, = C1/2 and
VC1/2-1 -1
< VC+ 01/2+ VC1/2-
When 0 <<< < 1/22. We have
inf vEC(s)nSP
| 2 T1
with probability at least 2 f logp cnnt²) for <T < v1/2. When Cslog p for sufficiently large C, exp(Cys logp ennt?) 2 exp( log 2/p= o(1)
Lemma F33 Let U = EE X.f. The following
|
sup VERPXa VEC(S))
(U, S M,
holds with probability at least o(1) when > Caq log for sufficiently large positive constant C.
Proof. By definition
1 72
||||,
sup VERPXY Cec(
X v)F
sup VERUXq
vec(v) 1,),,.
72
Note that, for any vec(v) E C(s) C(s)- E.vec( (v) (f. (f,, & L = vec(v) 2 I', where I Mंo + TTI Since (f, I,), is sub-Gaussian, have vec(v) -ec(v inequality (Theorem 2.8.1 in Vershynin (2018)), for
S S1. By Bernstein's
P Σ
vec( (re(v))(E (f - vec(v) @ 1 21)
En # CP max, Ic, ) < 2 eexpl-emin
nt² nt o
< 2 22exp(-cmin
Ci
$ 22exp(-cnnt))
where the last inequality holds when t S C1 and e₁ c/C?. Let vec(vi),.... vec(v is 1/2-net of [(s) n SPP-11 Then according to Lemma E.3, log Mnes < O log where Caei is positive constant. By union bound,
P( Σ
(f. - r'r 2t,5v, e C(s) ( SSP9-1)
< 2M
< 2xppppepeeecccccce logg cntt)..
76

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
Therefore for all V E L()) n SP
2 Σ (vee(v,)
1
I,)X,
n
vec (ec(v (f +
(1 @ 1,) +
1=1
MIM. +t.
Then, for any vec(v) E [(s) n gp--1, there exists a v, such that
vec(v v1)
v vgllF - 2' vV- v5lF
E L(s) f(s)) n
and
- - x
v S
-
Σ
X Lf v2)F + (=
X pff v - Vz)F.
1=1
i=1
1
Taking supremum of both sides, we have
f.f. v)r $ 2(MM +t),
sup vEC(s)n8Pa-
n
with probability at least 1 2 exp exp(Cnessp sq logp-annt Lett at least 1 - o(1),
(2 log 1/2 R21
When n > Csq" log p. with probability
2Cnet
X.f. v/r s MB
sup VE
c
Lemma F44. Suppose that B, B Rexq with rank(B) rank(B*) d. Let B3, 3 be the top-d left singular vectors of
B, B e 1284 and 01 2 ... 2 da be the singular values of B*. Assume |B -BI12 S CB. Then
[P - PollE S V2d 40 + 2CB BB Bl|
oz
Proof. Let BTB = MDN denote the singular value decomposition of 3 where M,N E Rdxd and D = diag(w)),... ...,, Wa). Define the principal angles between the subspaces spanned by á and B as (ei,..., Qa)
(cos - wl,..., cos wa), where W1 2 2 wa are the singular values of 3 B. And define sin (3,3)
diag(sin .1...., sin da). Then according to Theorem 3 (Yu al., 2015),
sin (3,,3)l < 2(201 + B - B" (2) min SinfvdlB B B 12, B --- -B
40 + 2CB <
Vd B B-BBae Er
o
Then
PP
-Pale = 113 - 33 || / tr((3TT 33 - 33
Vr + +,00)) - 2 (3 333 3) 12d 2tD(MT
1/2
- V2 d- Σ w/
22
Σ
sin? Φ5
/2ll sin (33 B)
i=1
i=1
< 22 40 + 2CB
Vdl B
-B'
03
77

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
where we use sin(cos w;) v w
2M b 2/ M2 (B -B Bille <r0 vec(T -r.))
Lemma F.5. Let a²
VM."
VM,
L(8), and lca Cr / ( 11 +
21 402
62
then A E (f))
Proof. Recall that
Bonn - To E (cv, 1 co): [T - - S Cas,
- Ca) S tr pm ( (1+ veo -ww [()), 1
where ow (T) T" (T2 + 12) and Ω vtr((Tr Er r.
Since oo E (cr, cx), when | xं r < Ico - c.l. we have c - coo c <m T <1 Gr + +co
-c/l Using
co Cx. गı (co, 1 co). By definition r. |[T, -l I r s CR...
Note that
|n2 - -
- - r())
= -1T
[+))++
[TT -T2--:)'T
(1)
Ir (T2
r
( F1)/2))
(IT)
We have
(1) = veo r - P2 + F2 (T; Erl
< 1 rí T2 r F (T) Er
< 2r / M2Ω.- 2 Marl?.
Since
[[ |IT2-Tr))
|| < [T; - ri F + [T - r3 - I1 + 1| S0+ 20++2M222)
we have
1 + 2 Mr
[ - <
[T2 - <
Ω
VM
VM
And
FillF S 2r 2Mr.
When Q 1,, for w 1,
(IT) = vec - 1T) vee( [ 2 - (T2 + 11)/2 Ex)
< [T Till [[[ 2 (T2 2
+ 2/
<
Ω 2. M2MM,, + 1 + MD)
VMi
+ 2/Mrr
Mar + 1+2 1+2/Mrr 4Mr
VM,
VM,
+ 222
(M2r + 1
VM,
78

---

Heteregeneous Sufficient Dimension Reduction and Subspace Clustering
If 2,
(11) vec (T2 r )" vec(I-ri -T= (T2+
$ 2 - -rille - + (T2 + r/2/2//rll
2/M2r Q MM + r + Mc)
VM1
VMi (M2r 1 1102.
2/
Let a²
2M JM,
.b --2--M
M + 2M2+vM2 and assume 1/(4/M/) < Ca. Then a²r² + br < Ca 1/(4/M)
when
2//d,
1 -1/AA/
+
ba
24
Therefore, have
102 - t (T) - (1) & M2rl? +
1 1+2 2/
(Mrr + 11P
VM
2M
,2 (2 M2 + 2M2 VM2
$
2/M )r + M
S2 S C.s.
VM1
G. Additional Discussion
The mixture PFC method is model-based and belongs to the linear heterogeneous SDR method. In contrast, nonlinear SDR aims to find a vector-valued function such that Y X g(X). Traditional nonlinear SDR methods often combine kernel tricks and linear SDR techniques (Wu, 2008; Hsing Ren, 2009; Li et al., 2011). However, these approaches have common computational challenges, when computing eigenvectors inverse of n X n or p x p matrices, making them infeasible for large-scale ligh-dimmnsionnl data. Deep learning. with its proven success in various domains, offers promising alternatives for nonlinear SDR. The auto-encoder (Hinton & Salakhutdinov, 2006; Zong et al., 2018) is the most representative example of deep learning for unsupervised dimension reduction. Recently, several deep SDR methods have emerged, leveraging the power of deep neural networks to address the above challenges (Banijamali et al., 2018; Liang et al., 2022; Kapla et al., 2022; Huang et al., 2024; Chen et al., 2024).
We suggest two strategies to extend the linear heterogeneous SDR to nonlinear settings through deep learning. The first strategy inspired by Kwon et al. (2024) and addresses semi-supervised scenarios with both labeled data {(X,, Y:) and unlabeled data {(X, ) }. Then model assumes the following structure:
Y T X (X), W w). Pr(W Y, X)- Pr(W Y. gx))
X (W w) Nu Ea),w, 1,, 1...,K
Pr(W- = w W
To
The key idea is to use the Gaussian mixture model on the unlabeled data to infer the joint distribution of (X, W) and then apply any proposed deep SDR method to learn gw- The procedure as follows.
Step 1:
Learn the joint distribution of (X, W) using GMM fitted to the unlabeled data.
Step 2:
Assign labeled data {(X,, Y0 to the clusters defined by using the estimate distribution of W|X.
Step 3:
Estimate the nonlinear SDR using any deep SDR method for each cluster.
Step 4:
Estimate the transition matrix II w = (Tw)
The second strategy combines a compression network and an estimation network, similar to the deep auto-encoding Gaus- sian mixture model (DAGMM) (Zong et al., 2018). The compression network is supervised auto-encoder (Ie et al., 2018),
79

---

Heterogeneous Sufficient Dimension Reduction and Subspace Clustering
designed to perform dimension reduction while preserving the nonlinear SDR structure. The innermost layer incorporates supervised loss ensure the reduced representation g g(X) satisfies the conditional independence condition in nonlinear SDR. Various dependence measures can be used to construct the loss function, such as distance covariance (Székely al., 2007), martingale difference divergence (Shao & Zhang, 2014), and generalized martingale difference divergence (Li et al.,
2023). Then the estimation network uses the learned low-dimensional vector g(X) and the response to predict clusters. Unlike DAGMM, this step employs supervised clustering model rather than the Gaussian mixture model Zong et al. (2018). To evaluate the clustering quality of the estimation network, the log- log-likelihood of the cluster assignments can be computed. Both strategies highlight the potential of deep learning to effectively extend SDR nonlinear, heterogeneous, and high-dimensional settings.
80


Reviews:
[{"review_id": "2c3jRv1B1D", "reviewer": "Reviewer_Htk4", "summary": "The paper presents a mixture of PFC model, which combines sufficient dimension reduction with subspace clustering to deal with heterogeneous high-dimensional data. Moreover, a grouped Lasso based EM algorithm is designed to solve the problem. Nonasymptotic convergence analysis is provided. Empirical results are also shown.\n\n## update after rebuttal: \nThe authors promise to reorganize the literature review and clarify the significance of this work and some issues in the empirical evaluation. Thus, the reviewer is leaning to a borderline accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed feedback. Below, we address each of your concerns.\n## Weaknesses\n> (literature review, novel contribution, EM algorithm) \n\n**Reply**: We acknowledge the current presentation could be streamlined. In the revision, we will reorganize the literature review to focus on clarifying how prior works on subspace clustering lack integration of response information and variable selection.  \nThe novelty lies in three key advances:\n1. Supervised Subspace Clustering\n    - Integrates response information into subspace clustering, enabling exact overlap of subspaces (unlike classical methods requiring separation angles, Soltanolkotabi & Candés (2012 AoS)). \n    - Achieves 30–40% higher clustering accuracy than unsupervised baselines in high-noise regimes (Table 1).\n2. High-Dimensional EM Algorithm: Although our proposed mixPFC algorithm has a familiar structure of the classical EM algorithm, it involves many innovations to adapt to the challenging problem of interest, which distinguishes it from existing EM algorithms.\n    - Our algorithm is built on a new model that seamlessly integrates dimension reduction and subspace clustering, and naturally combines the information from the predictors and the response. Without such a rigorous foundation, it is extremely difficult to derive a reasonable algorithm for our purpose, or the strong theoretical guarantee. We also incorporate variable selection to exclude inactive predictors, a capability absent in classical subspace clustering.\n    - While low-dimensional EM is standard, extending it to high dimensions presents significant challenges. A naive generalization would require inverting the $p\\times p$ covariance matrix in both E-step and M-step, which is practically impossible in high dimensions without additional structural assumptions.\n        - Novel E-step: estimate probabilities $\\gamma_{iw}$ via a low-dimensional mixture linear regression on projected predictor, reducing covariance inversion to $d\\times d$ ($d\\ll p$).\n        - Scalable M-step: replace the non-convex or $p\\times p$ dimensional optimization for subspace estimation with scalable $p\\times q$ ($q\\ll p$) dimensional convex optimization. \n3. Theoretical Results\n    - The high-dimensional EM theory is challenging (Wang et al. (2024 JMLR)). Unlike classical results focusing on asymptotic convergence to fixed points, we derived a non-asymptotic result that mixPFC converges to the **true** parameter. We provided a detailed discussion in our response to Reviewer LLk6's second question and kindly ask Reviewer HtK4 to refer to that section.\n\n> (empirical evaluations)\n\n**Reply**: \n- The datasets and model sizes used in our experiments align with established benchmarks in SDR and mixture model literature ([LI19] and Wang et al. (2024 JMLR)). Due to page limits, we included extensive numerical analyses (subspace estimation errors, variable selection) in Appendix A. \n- Our primary goal was to conceptually demonstrate that incorporating response information significantly improves clustering accuracy over unsupervised methods, which is not explored in the literature. \n\n> (initialization of $\\gamma_{iw}$ )\n\n**Reply**: \n- We recognize that initialization is a prevalent issue in EM algorithms, without complete solutions even for much simpler models. However, we provide a practical solution to make our algorithm feasible.\n- Our strategy (Appendix B.1) follows principled recommendations from [BI03], using short EM runs on predictors reduced via PCA and variable screening to ensure stability in high dimensions.\n- While existing methods are incompatible with our model’s structure, this approach consistently yields reliable initializations (Table A.15). A formal initialization scheme remains future work.\n\n## Questions\n> (comments on subspace clustering methods)\n- **On random errors (L034-036)**: We wrote ''When the observations are subject to **significant** random errors, ...''. Subspace clustering methods struggle under significant random errors(Table 1 M3: SSC error rate >47 under high noise vs. 13.2 under low noise). Our intent was to highlight this limitation, not dismiss robustness to small errors.\n- **On overlapping subspaces (L114-115)**: We apologize for the imprecise phrasing and appreciate the reviewer's clarification. Classical subspace clustering requires a minimal angle condition for identifiability (Theorem 2.8 of Soltanolkotabi & Candés (2012 AoS)). In contrast, mixPFC allows exact overlap by leveraging response information. We will revise to clarify this distinction.\n\n\n[LI19] Lin et al. (2019 JASA) Sparse sliced inverse regression via lasso  \n[BI03] Biernacki et al. (2003 CSDA) Choosing starting values for the EM algorithm for getting the highest likelihood in multivariate gaussian mixture models.", "claims_and_evidence": "yes.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Check a part, not completely. The proofs are too lengthy.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Different from (unsupervised) subspace clustering, the paper attempts to perform (sufficient) dimension reduction and subspace clustering for regression problem, which related to broad literature in statistics.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n+ The paper attempts to address an interesting problem, which involves subspace clustering, supervised dimension reduction and variable selection.  \n\nWeaknesses: \n- The literature review is chaos and the presentation is unclear.  While quite involved and complicated derivations are shown, the novel contribution is weak. The deduction of the EM algorithm is a routine task. \n\n- Empirical evaluations are insufficient and weak.  Most experiments are conducted on toy datasets. \n\n- For the designed algorithm, the initialization of $\\gamma_{i,w}$ is critical. Rather than the mentioned ad hoc tricks to initialize  $\\gamma_{i,w}$, is there more principled way to provide good initialization?", "other_comments_or_suggestions": "- It might be more suitable to submit on a statistics conference, e.g., AISTATS, UAI.", "questions_for_authors": "- The comments on subspace clustering methods are questionable in L034-036 at the right column: the state of the art subspace clustering methods can easily handle random errors. Also, in L114-115: \"in subspace clustering the latent subspaces cannot overlap...\" It is NOT true. \nAs discussed in (Soltanolkotabi & Candes, 2012), subspaces can even be partially overlapping.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KzziDaoUr8", "reviewer": "Reviewer_ooHT", "summary": "The paper introduces a supervised subspace clustering model called mixPFC, which combines sufficient dimension reduction (SDR) with subspace clustering to address three major bottlenecks in traditional methods for high-dimensional heterogeneous data analysis：\n1. The model effectively guides clustering and dimension reduction through the response variable, even when the subspaces are fully overlapping.\n2. It designs an EM algorithm with a grouped Lasso penalty to jointly optimize variable selection and subspace estimation in high-dimensional scenarios, overcoming the limitations of traditional methods on the separation angles of subspaces. \n3. The model allows for arbitrary overlap between subspaces and ensures identifiability by leveraging the nonlinear relationship between the response variable and the projected subspaces. \nExperiments show that it reduces clustering errors by more than 50% on both synthetic and real data, such as cancer genomic data. However, its limitations include overly simplified covariance structure assumptions (e.g., isotropic assumption), sensitivity to initialization, lack of a systematic strategy for cluster number selection, and the need for further validation of its generalizability to multi-cluster settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s thoughtful comments. Below, we address each limitation raised.\n## Weaknesses\n> (isotropic covariance assumption)\n\n**Reply**: \n- The isotropic assumption in Theorem 4.1 was adopted solely to enable tractable analysis of $\\gamma_{iw}$, a common simplification in mixture model theory. We agree that extending the theory to general covariances is a valuable future direction. However, relaxing this would require significant advances beyond the scope of our current framework.\n- While the theory assumes isotropy for clarity, our algorithm imposes no such restriction. Simulations in Section 5.1 demonstrate mixPFC’s robustness under general covariances (e.g., AR(0.3)). This suggests the assumption is not necessary in practice. \n\n> (sensitivity to initialization)\n\n**Reply**\n- Non-Convexity: Like all EM-based methods, we acknowledge that mixPFC inherits sensitivity to initialization—a well-known challenge in non-convex optimization [JA17]. \n- To mitigate this, our initialization strategy (Appendix B.1) follows principled recommendations from [BI03], using short EM runs on predictors reduced via PCA and variable screening to ensure stability in high dimensions. While existing initialization methods for classical mixtures are not directly applicable to our proposed model due to its unique structure, this approach consistently produced high-quality initial values (Table A.15).\n\n> (cluster number selection)\n\n**Reply**\n- Appendix B.1 details our strategy for selecting $K$, which is inspired by the well-accepted gap statistics [TI01] with careful modifications to tailor it to our problem.\n    - Gap statistics are calculated using data projected onto the orthogonal subspaces. \n    - Under high dimensions, estimating the expected within-cluster dispersion becomes computationally prohibitive. Instead, we propose using the observed within-cluster dispersion. Simulations in Appendix B.1 validate our proposal.\n\nWe acknowledge that a summary of this approach in the main text would enhance clarity, and we will add it to the final manuscript if accepted.\n\n> (multi-cluster generalizability)\n\n**Reply**\n- We discuss theoretical challenges under multi-cluster settings in the reply to the second question of Reviewer LLk6. To avoid redundancy, we kindly ask Reviewer ooHT to refer to that response.\n- MixPFC’s performance in multi-cluster settings is rigorously tested in both simulations and real data. \n    - Simulations in Section 5.1 include extensive tests with $K=3,5$ clusters under different noise levels. Appendix A.1 further validates its performance in terms of subspace estimation and variable selection.\n    - In the analysis of CCLE data in Section 5.2 and Appendix A.2, we systematically tested $K=2,3,\\dots,10$ clusters and selected $K=3$ for Nutlin-3 and $K=5$ for AZD6244, demonstrating mixPFC’s ability to adapt to multi-cluster settings.\n\n## Questions\n> (theory for $K>2$)\n\n**Reply**\nWe sincerely thank Reviewer ooHT and Reviewer LLk6 for their insightful critiques on multi-cluster theory. To avoid redundancy, we have provided a detailed discussion of theoretical challenges under multi-cluster settings in our **reply to Reviewer LLk6’s second question**. We kindly ask Reviewer ooHT to refer to that section and welcome any follow-up clarifications.\n\n> (subspace identifiability)\n\n**Reply**\n- Role of Response Information: Unlike classical subspace clustering, mixPFC leverages response information, which disentangles overlapping subspaces using the relationship between predictors and the response. Even when subspaces are identical (e.g., Figure 2(a)), the response provides discriminative signal to recover clusters. Under Model M1 in Section 5.1, where the two subspaces are identical, mixPFC maintains cluster error rates below 10\\% across most covariance settings, whereas classical subspace clustering methods fail.\n- Theoretical Guarantees: Theorem 4.1 establishes convergence to true subspaces **without requiring a minimum separation angle**. Traditional subspace clustering relies on separation assumptions (e.g., angles between subspaces in Theorem 2.8 of Soltanolkotabi & Candés (2012 AoS)).\n\n[JA17] Jain et al. (2017 FnTML)  Non-convex optimization for machine learning.  \n[BI03] Biernacki et al. (2003 CSDA) Choosing starting values for the em algorithm for getting the highest likelihood in multivariate gaussian mixture models.  \n[TI01] Tibshirani et al. (2001 JRSS-B) Estimating the number of clusters in a data set via the gap statistic.", "claims_and_evidence": "The submitted materials include a relatively detailed theoretical analysis.", "methods_and_evaluation_criteria": "The paper introduces the mixPFC model, which integrates supervised sufficient dimension reduction with subspace clustering. Utilizing an EM algorithm with a grouped Lasso penalty, the model achieves joint clustering, dimension reduction, and variable selection in high-dimensional data. The paper also theoretically proves the non-asymptotic convergence rate, thus overcoming the traditional methods' dependence on the separability of subspaces.", "theoretical_claims": "I have checked the method.", "experimental_designs_or_analyses": "I have checked the experiment and analysis.", "relation_to_broader_scientific_literature": "The paper introduces the mixPFC model, which integrates supervised sufficient dimension reduction with subspace clustering, thus overcoming the traditional methods' dependence on the separability of subspaces.", "essential_references_not_discussed": "I did not find.", "other_strengths_and_weaknesses": "The strengths and weaknesses have already been discussed in the preceding sections.", "other_comments_or_suggestions": "No further comments.", "questions_for_authors": "The theoretical analysis in the paper is confined to the binary classification ($K=2$) scenario, while the empirical results validate the effectiveness for multiple clusters ($K>2$). How can we ensure the convergence and statistical error rates of the EM algorithm remain consistent in complex heterogeneous data with multiple clusters ($K>3$)? When the number of clusters K increases with the data size (e.g., K=O(log n)), does the existing theoretical framework still hold? Additionally, is there a risk of decreased subspace identifiability due to increased interactions between clusters?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eYlfYS5n9R", "reviewer": "Reviewer_LLk6", "summary": "This paper proposes a new method, mixPFC, which integrates subspace clustering with model-based SDR to handle heterogeneous, high-dimensional data. The method simultaneously performs clustering, subspace estimation, and variable selection. A group Lasso penalized EM algorithm is developed, and non-asymptotic convergence rates are established. Empirical results demonstrate superior performance over existing methods in simulations and real-world applications.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for raising many critical points. Below, we address each of your concerns.\n## Weaknesses\n> (Gaussian assumption)\n\n**Reply** \n- Gaussian assumption is common in clustering, promoting interpretability and scalability. \n- mixPFC’s theoretical grounding and empirical performance establish it as a new frontier for heterogeneous SDR, with a clear pathway for generalization: e.g., replacing Gaussian distribution in the model with the student-t or skew normal to handle heavy-tailed or skewed data.\n- In our experience, the mixPFC, whose algorithm is derived under Gaussian assumption, is generally robust under non-Gaussian noise (model-misspecification). To show this, we consider the following errors:\n    - multivariate t-distribution: degrees of freedom is 3\n    - multivariate skew normal distribution: shape is 5\n- Under heavy-tailed noise, mixPFC achieves low error rates (e.g., t(3): 3.4 under $0.1\\mathbf{I}$ vs. SSC’s 17.4; 11.0 under AR(0.3) vs. SSC’s 46.3), demonstrating robustness to non-Gaussianity. Under the same setting, mixPFC's performance has no significant changes when we replace normal with skew normal. Full results will be included in the final manuscript.\n\n## Questions\n\n> (choice of $f(Y)$ in practice)\n\n**Reply** \n- As noted in Theorem 3.5 of the original PFC framework (Cook & Forzani (2008 Stat. Sci)), the estimated subspace remains consistent under misspecification of $\\mathbf{f}(Y)$, provided $\\mathbf{f}(Y)$ is sufficiently correlated with the true function. Our mixPFC model directly inherited this property from the PFC model. This property ensures that our method retains validity across a broad class of functions, which can be treated as a tuning parameter to enhance flexibility.\n- In practice, polynomials or splines are standard choices. We use $\\mathbf{f}(Y) = (Y, Y^2, Y^3)^T$ in simulations.\n\n> (theory when $K>2$)\n\n**Reply** \nTheoretical Challenge for $K=2$\n- While classical EM theories only guaranteed asymptotic convergence to a fixed point, we derive a non-asymptotic result that mixPFC converges geometrically to a fixed point that is within statistical precision of the unknown **true** parameter. This type of result has emerged only recently [BA17].\n- Even the low-dimensional EM theory for two mixtures requires additional assumptions such as equal proportions and known covariance (Gaussian mixtures [XU16]), or equal proportions and symmetric coefficients $\\boldsymbol{\\beta}_1 = -\\boldsymbol{\\beta}_2$ (mixtures of linear regression [KL19]). Theorem 4.1 establishes convergence rates under unequal proportions and arbitrary subspace angles.\n- Many established EM theories [KW19] are based on sample-splitting. Sample-splitting divides the data into many batches and uses a new batch of samples in each iteration to make random samples and current parameter estimates independent.\n- Our theoretical analysis does not require sample-splitting and allows a more complex model (which involves subspace estimation and variable selection), making it novel and highly non-trivial. \n\nTheoretical Challenge for $K>2$\n- Establishing theories for multi-cluster EM algorithms is an important but challenging direction. Recent advances focus on low-dimensional settings, see [YA17, TI24].\n- To the best of our knowledge, extending multi-cluster EM theories to high dimensions is an open question. Theoretical analyses are often limited to $K=2$ even for Gaussian mixtures and mixtures of linear regression (Cai et al (2019 AoS); Wang et al. (2024 JMLR)). \n- For $K$ diverges, we believe it requires fundamentally new tools to handle complex parameter spaces and interactions between $K$ subspaces. \n- Empirically, mixPFC works well when $K>2$. Its superior performance is demonstrated through simulations (Section 5.1 and Appendix A.1).\n\n> (computational cost)\n\n**Reply** \n- The total cost per EM iteration is $O(nKpq + nK^3q^3 + KTpnq + nKp^2)$. Typically, $q$ is a small number that does not grow with $n, p, K$. Theorem 4.1 suggests the number of EM iterations is of order $O(\\log(n))$.\n- The total cost of mixPFC is $O(\\log(n)(nK^3 + KTpn + nKp^2))$ with the dominated term $O(\\log(n)Knp^2)$ from covariance estimation, which is tractable when $K$ or $p$ is large.\n\n[BA17] Balakrishnan et al. (2017 AoS) Statistical guarantees for the em algorithm: From population to sample-based analysis  \n[XU16] Xu et al. (2016 NeurIPS) Global analysis of expectation maximization for mixtures of two gaussians.  \n[KL19] Klusowski et al. (2019 IEEE Trans. Inf. Theory) Estimating the coefficients of a mixture of two linear regressions by expectation maximization  \n[KW19] Kwon et al. (2019 PMLR) Global convergence of the em algorithm for mixtures of two component linear regression  \n[YA17] Yan et al. (2017 NeurIPS) Convergence of gradient em on multi-component mixture of gaussians  \n[TI24] Tian et al. (2024 ICML) Towards the theory of unsupervised federated learning: Non-asymptotic analysis of federated em algorithms", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No.", "experimental_designs_or_analyses": "Yes, I checked the numerical results in Section 5.", "relation_to_broader_scientific_literature": "This paper extends the subspace clustering into a supervised framework.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength: The integration of subspace clustering with SDR is innovative, addressing the limitations of unsupervised subspace clustering and homogeneous SDR. The supervised framework leverages response information to improve clustering and dimension reduction.\n\nWeakness: The proposed mixPFC is built upon a Gaussian assumption outlined in Equation (3). This may limit the applicability and \nrobustness of the method because many real-world datasets exhibit non-normal characteristics such as skewness or heavy tails.", "other_comments_or_suggestions": "None.", "questions_for_authors": "1. How to choose or estimate $f(Y)$ in practice?\n\n2. What are the theoretical properties when $K>2$?\n\n3. What is the computational cost associated with mixPFC? I believe that when $K$ is large or $p$ is very high, the mixPFC becomes computationally extensive.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposed a supervised subspace learning model (mixPFC), which combines sufficient dimension reduction with subspace clustering for heterogeneous high-dimensional data. Grouped Lasso based EM algorithm is designed and nonasymptotic convergence analysis is provided. Reviewers consistently raised some critical concerns regarding sensitivity to initialization, the multi-cluster generalizability, the theoretical extension for K>2, Gaussian assumption unsuitable for non-Gaussian real-world datasets, and high computational cost. The authors’ responses clarified some of these concerns, but still some key concerns remain. For example, theoretical challenge for K>2 and Gaussian assumption are not addressed, even though empirical experiments may work on the tested data. As a result, weak accept is recommended.
Decision: Accept (poster)