Errors-in-variables Fr\'echet Regression with Low-rank Covariate Approximation
Authors: Dogyoon Song
Kyunghee Han
Venue: NeurIPS 2023 poster (2023)
Keywords: Frechet regression
principal component regression
non-Euclidean
low-rank matrix
errors-in-variables analysis
TL;DR: We propose a principal component regression framework for non-Euclidean response variables and provide a theoretical analysis for the consistency and the rate of convergence for the proposed estimator.

Abstract: Fr\'echet regression has emerged as a promising approach for regression analysis involving non-Euclidean response variables. However, its practical applicability has been hindered by its reliance on ideal scenarios with abundant and noiseless covariate data. In this paper, we present a novel estimation method that tackles these limitations by leveraging the low-rank structure inherent in the covariate matrix. Our proposed framework combines the concepts of global Fr\'echet regression and principal component regression, aiming to improve the efficiency and accuracy of the regression estimator. By incorporating the low-rank structure, our method enables more effective modeling and estimation, particularly in high-dimensional and errors-in-variables regression settings. We provide a theoretical analysis of the proposed estimator's large-sample properties, including a comprehensive rate analysis of bias, variance, and additional variations due to measurement errors. Furthermore, our numerical experiments provide empirical evidence that supports the theoretical findings, demonstrating the superior performance of our approach. Overall, this work introduces a promising framework for regression analysis of non-Euclidean variables, effectively addressing the challenges associated with limited and noisy covariate data, with potential applications in diverse fields.

Errors-in-variable Fréchet Regression with Low-rank Covariate Approximation
Dogyoon Song"
Kyunghee Han"
Department of EECS University of Michigan
Department of Math, Stat and Comp Sci University of Illinois at Chicago
Ann Arbor, MI 48109, USA
Chicago. IL 60607, USA
dogyoons$umi ch. edu
hankhduic.edd
Abstract
Fréchet regression has emerged as promising approach for regression analysis involving -un-uuclide response variables. However, its practical applicability has been hindered by its reliance on ideal scenarios with abundant and noiseless covariate data. this paper, we present a novel estimation method that tackles these limitations by leveraging the low-rank structure inherent in the covariate matrix. Our proposed framework combines the concepts of global Fréchet regression and principal component regression, aiming to improve the efficiency and accuracy the regression estimator. By incorporating the low-rank structure, our method enables more effective modeling and estimation, particularly in high-dimensional and errors-in-variable regression settings. We provide a theoretical analysis the proposed estimator's 's sarge-sample properties, including comprehensive rate analysis of bias, variance, and additional variations due to measurement errors. Furthermore, numerical experiments provide empirical evidence that supports the theoretical findings, demonstrating the superior performance our approach. Overall, this work introduces promising framework for regression analysis of non-Euclidean variables, effectively addressing the challenges associated with limited and noisy covariate data, with potential applications in diverse fields.
Introduction
Regression analysis is fundamental statistical methodology model the relationship between response variables and explanatory variables (covariates). Linear regression, for example, models the (conditional) expected value of the response variable as a linear function of covariates. Regression models enable researchers and analysts to make predictions, gain insights into how input variables influence the outcomes of interest, and validate hypothetical associations between variables in inferential studies. As a result, regression is widely utilized across various scientific domains, including economics, psychology, biology, and engineering [53 21 29].
In recent decades, there has been a growing interest in developing statistical methods capable of handling random objects in non-Euclidean spaces. Examples of these include functional data analysis 41) statistical manifold learning [31] statistical network analysis (34, and object-oriented data analysis [39]. such contexts, the response variable defined in a metric space that may lack an algebraic structure, making it challenging to apply global, parametric approaches toward regression as in the classical Euclidean setting. To overcome this challenge, (global) Fréchet regression, which models the relationship by fitting the (conditional) barycenters of the responses as a function of covariates, has been introduced [40]. Notably, when the Euclidean metric considered, Fréchet
Equul contribution. To whom correspondence should be addressed: hankhduic. edu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

regression recovers classical Euclidean regression models. For more details on Fréchet regression and its recent developments, we refer readers 30 40
45
27
Nevertheless, most existing research on Fréchet regression has focused on ideal scenarios char- acterized by abundant covariate data that accurately measured and free of noise. In practical applications, however, high- high-dimensional data often arise, which are also susceptible to measurement errors and other forms of contamination. These errors can stem from various sources, such as unreliable data collection methods (e.g. low-resolution probes, subjective self-reports) or imperfect data storage and transmission. The sigh-immensionalty and the presence of measurement errors in covariates pose critical challenges for statistical inference, as regression analysis based on error- pror- covariates may result in incorrect associations between variables, yielding misleading conclusions.
To address these limitations, it is crucial to extend the methodology and analysis of Fréchet regression to tackle high-dimensional prrors-in-variibles problems. In this work, aim leverage the low- rank structure in the covariates to enhance the estimation accuracy and computational efficiency of Fréchet regression. Specifically, we explore the extension of principal component regression to handle errors-in-variables regression problems with non-Euclidean response variables.
1.1 Contributions
This paper contributes to advancing the (global) Fréchet regression of non-Euelidean response variables, with a particular focus on high-dimensional,, errors-in-variables regression.
Firstly, we propose a novel framework, called the regularized (global) Fréchet regression (Section 3 that combines the ideas from Fréchet regression [40] and the principal component regression [32 This framework effectively utilizes the low-rank structure in the matrix of (Euclidean) covariates by extracting its principal components via low-rank matrix approximation. Our proposed method is straightforward to implement, not requiring any knowledge about the groo-gcnerating mechanism.
Furthermore, we provide comprehensive theoretical analysis (Section 4 in three main theorems to establish the effectiveness of the proposed framework. Firstly, we prove the consistency of the proposed estimator for the true global Fréchet regression model (Theorem 1 Secondly, we
investigate the convergence rate the estimator's ' bias and variance (Theorem 2p. Lastly, we derive an upper bound for the distance between the estimates obtained using error-free covariates and those with prrors-i--variable covariates (Theorem 3). Collectively, these results demonstrate that our approach effectively addresses model mis mis-specification and achieves more efficient model estimation by leveraging the low-rank structure of covariates, despite the presence of inherent bias due to unobserved measurement eπrors.
To validate our theoretical findings, we conduct numerical experiments on synthetic datasets (Section 5). We observe that the proposed method provides more accurate estimates of the regression param- eters, especially in high-dimensional settings. Our experimental results emphasize the importance of incorporating the low-rank structure of covariates in Fréchet regression, and provide empirical evidence that aligns with our theoretical analysis.
1.2 Related work
www.aaaiinammmiflodd---ao manifold-vvlued responses were proposed as a generalization of regression for multivariate outputs by Steinke et al. 48 These works provided a foundation for recent developments in regression analysis of non-Fuclidean responses. Later, Hein [50) proposed a Nadaraya-Watson-yy Watson-type kernel estimation of regression model for general metric-spacc-valuee outcomes. Since then, statistical propertics of regression models for some special classes of metric--pace- outcomes, such as distribution functions 23 52 28 and matrix-valued responses [57] 20), have been investigated. Recently, many
Metrieespace-valued variables. Nonparametric regression models for Riemannian
researchers have introduced further advances in Fréchet regression, including [40] 10
37 45 In this
study, we use the global Fréchet regression proposed in [40] as the basis for our proposed method.
irrors-in-variable regression. Much of earlier work on iroor-in-vvariblle (EIV) problems in the
statistical literature can be found in [13 which covers the simulation-extrapolatioo (SIMEX) [66
the attenuation correction method 33 covariate-adjuste
model [46] 19] and the deconvolution kernel method 25 24 18.. The regression calibration method 47), instrumental variable modeling
12 43 and the two-phase study design [9 4 were also proposed when additional data are available

---

for correcting measurement errors. In the nigh-dimensional modeling literature, regularization methods for recovering the true covariate structure can also be utilized [38 17 Despite a diverse body of literature on high-dimensional learning and robust regression modeling, much of it assumes response spaces to be vector spaces endowed with inner products. In this paper, we tackle EIV problems within the Fréchet regression framework. While previous works have explored regression analysis in -on-Euclidean metric spaces, addressing EIV issues in this context remains uncharted.
Principal component regression. The principal component regression (PCR) 32 is statistical technique that regresses response variables on principal component scores of the covariate matrix. The conventional PCR selects a few principal components as the "new" regressors associated with the first leading eigenvalues to explain the highest proportion of variations observed in the original covariate matrix. In functional data analysis, PCR is known to have a shrinkage effect on the model estimate and produce robust prediction performance in functional regression 442 33 Recently, Agarwal et al. 2 investigated the robustness of PCR in the presence of measurement errors on covariates and the statistical guarantees for learning a good predictive model. Unlike prior statistical analyses of EIV problems that often assume known estimable noise distributions, PCR leverages inherent low-rank structures in the covariates without requiring a priori knowledge of measurement error distributions. We adopt PCR as concrete, practical solution to EIV models in non-Euclidean regression, driven by two compelling considerations. Firstly, the prevalence of (approximate) low-rank structures in real-world datasets enhances the practical relevance of our approach. Secondly, intentionally opt for an approach with minimal assumptions regarding covariate errors to ensure broad applicability.
1.3 Organization
In Section 2 we introduce the notation used throughout the paper, and overview the global Fréchet regression framework. Section 3 presents the problem setup, objectives, and our proposed estimator; which we refer to as the regularized Fréchet regression (Definition 4 In Section 4 we discuss theoretical guarantees on the regularized Fréchet regression method in accurately estimating the global Fréchet regression function. S presents the results of numerical "proof-of-concee" experiments that support the theoretical findings. Finally, we conclude this paper with discussions in Section 6 Due to space constraints, detailed proofs of the theorems as well as additional details and discussions of experiments are provided in the Appendix.
2 Preliminaries
2.1 Notation
Let N denote the set of positive integers and lR denote the set of real numbers. Also, let R_ = {o E R : r 2 0}. For E N. we let [n] {I....,n1] n). We mostly use plain letters to denote scalars, vectors, and random variables, but we also use boldface uppercase letters for matrices, and curly letters to denote sets when useful. Note that we may identify vector with its column matrix representation. For matrix X, we let X denote its inverse (if exists) and X denote the Moore- Penrose pseudoinverse of X. Also, we let rowsp (X) ) and colsp (X) denote the row and column spaces of X, respectively. Furthermore, we let spec (X) denote the set of non-zero singular values of X, (x( denote the i-th largest singular value of X, and o (A) == inf (X; > X E N) with the convention inf 0 co. We let 1, (1, 1, .............. 1)" E Rd and let 1 denote the indicator function. We let || denote a norm, and set || |1 - || ||2 (the form for vectors, and the spectral norm for matrices) by default unless stated otherwise. For finite set D, we may identify D with its empirical measure I/D = D
ΣeED 8z, where òS denotes the Dirac measure supported on {x}.
Letting f,g: R R. we write f(x) O(g(x)) as x co if there exist M > 0 and xo > 0 such that [f(x)] < M g(x) for all x 2 zo. Likewise, we write f(x) = ((g(x) if g(x) = (()()))
Furthermore, we write f(x) = o(g(x)) as + oo if lim
f(x) = 0. For sequence of random s(z)
variables X,, and sequence an. we write X, Op(on) as n + oo for any E > there exists M E R.+ and N N such that P( X
> M) < E for all n 2 N. Similarly, we write X,, = (a,)
an
if lim P([ X= > E) = 0 for all E > 0.
3

---

2.2 Global Fréchet regression
Let (X,Y) Y) be a random variable that has a joint distribution Px.y supported on R x M. where R" is the p-dimensional Euclidean space and M = (M, d) is metric space equipped with a distance function di M x M - R. We write the marginal distribution of X as Px, and the conditional distribution of Y given X as Py|x.
Definition 1 (Fréchet regression function), Let (X,Y) be random element that takes value in RP x M. The Fréchet regression function of on X is a function 8² -RP -M such that
(x)x)) arg min (( X ==z]] z),
Yx E susp Px CR.
(1)
yEM
We note that 40 (a:) is the best predictor of Y given X Z. as it minimizes the marginal risk E [Y x under the squared-distance loss. In the literature, (²(xx) is also known as the conditional Fréchet mean of Y given X = r [26]. is important to recognize that the existence and uniqueness of the Fréchet regression function are closely tied to the geometric characteristies of M, and are not guaranteed in general [3] g]. Nonetheless, extensive research has been conducted on the existence and uniqueness of Fréchet means in various metric spaces commonly encountered in practical applications. Examples include the unit circle in R22 [14,, Riemannian manifolds [ S. Alexandrov spaces with non-positive curvature [51] metric spaces with upper bounded curvature
(58, and Wasserstein space [59 35
While modeling and estimating the Fréchet regression function o is often of interest, its global (parametric) modeling may not be straightforward, especially when M lacks useful algebraic structure, such as an inner product. For instance, in classical linear regression analysis with M R, the conditional distribution of Y given X = x is normally distributed with a mean of (²(x) = a + B x and a fixed variance o², where a and / represent the regression coefficients. Similarly, when M possesses linear-algebraic structure, one specify class of regression functions that quantifies the association between the expected outcome and covariates in an additive and multiplicative manner. However, the lack of an algebraic structure in general metric spaces may prevent us from characterizing the barycenter yo (x)) in the same way classical regression analysis determines the expected value of outcomes with changing covariates.
To address this challenge, Petersen and Müllrr [40] recently proposed to exploit algebraic structures in the space of covariates, R", instead of M. Specifically, they consider weighted Fréchet mean as
(zz arg min [[[w(X, X, x)) (Y(Y,)).
yEM
where w. : RP x RD - R is a weight function such that wE,,, x) denotes the influence of & at æ. In particular, Petersen and Müller 40 defined the global Fréchet regression function with a specific choice of w as follows.
Definition 2 (Global Fréchet regression function). Let Y) be random variable in R x M.
Let J B(X) and = Var(X). The global Fréchet regression function of Y on X is a function Pglo RP - -M such that
Pato = arg min E [ (X, (X) (Y.vv)
DEM
where Wyto (X,x) 1 -)= -
pt).
When M is an inner product space (e.g., M = RR, the function fglo restores the standard linear regression model representation over the domain Rº, For this reason, iPglo is commonly referred to
as the global Fréchet regression model for metric-space-valued outcomes 40 37 54].
Remark on Definition 2 One might wonder why the term "global" is used to describe Pgloo as Fréchet regression function. The use of the adjective "global" serves emphasize its distinction from "local" nonparametric regression methods that interpolate data points. Notably, when M is a Hilbert space, Palo reduces to the natural linear models. For instance, if M R. then it follows that
Pglo (x: E Wiglo (X, x²) Y] = a + 3T (a - n)), where a = E and B = E-1 E[(x - 14) Y]. These linear models hold uniformly for the evaluation point x. Similarly, in the case of an L2 space equipped with the squared-distance metric dy. = |!y y' induced by the 1,2 norm, Pglo represents the linear regression model for functional responses. Thus, Pglo establishes a globally defined model that spans the entire space.

---

Problem and methodology
3.1 Problem formulation
Let (X,Y) be random variable I& x M and Px.y be their joint distribution. Let Dn {(X,, Y;) : [r|}} be independent and identically distributed (II) sample drawn from Px,y- Note that we may identify the set D, with its discrete measure (empirical distribution), ef. Section
2.1 We consider the problem of estimating the global Fréchet regression function Pglo (see Definition
2 from data Dn. In this setting, a natural estimator of Pkto would be its sample-analogue estimator.
With NDD -_(,, (X) 1 E X, and Σp. ())D, (X) 4 Ei=i(X, -DD.).
(X; PDD... T,, the sample-analogee estimator GD.. defined as
(z) = arg min yEM
Σ ÜDD (X., &e (X,Y))
n
where 2Xx,)= (D.)) Epp (I (.D.). The statistical properties of pp.. including
the asymptotic distribution, ridge-type variable selection operation, and total variation regularization method have been investigated [40 37 54]
In practice, however, we may only be able access D. {(Z,, Y2) [n]] instead of Dn, where 1,....1
Z: X; + Es,
(5)
denotes error-prone observation of the covariates X by measurement error e. This formulation corresponds to the classical errors-in-variables problem.
Objective. Given a dataset, either Dn, or Dn, our aim is to produce an estimate e of the global Fréchet regression function fglo that the prediction error is minimized. Specifically, evaluate the performance of @ by means of the distance in the response space, (glo())..
3.2 Fréchet regression with covariate principal components
Singular value thresholding, Among various low-rank matrix approximation methods, we consider the (hard) singular value thresholding (SVT). For any R4,, we define the map SVT(A) Ruxx Raxp that removes all singular values that are less than the threshold X. To be precise, SVT(A) can be expressed in terms the singular value decomposition (SVD) as follows:
min(n.p)
min(n.p)
M
8, 4.00 aVD
SVT(A)
Σ
8, 1..1 > A} u.u:" (6)
Regularized Fréchet regression. We introduce variant of the sample-analog estimator of the global Fréchet regression function based on principal components of the sample covariance. To facilitate the description of our proposed estimator, we introduce additional notation here.
Definition 3 (Covariate mean/covariance). For probability distribution v on TRP x M, the covariate mean and covariate covariance with respect to are respectively defined as
Hy = [x(XY))x(X)
and En (Xx)
Recall that finite set D
RP
M may be identified with its empirical distribution; it follows that
HD
1
Σ
and Dp
Σ (a, (D)--))-- (x; TD)
(8)
D
D
ED
(Z.
Definition 4 (Regularized Fréchet regression). Let v be probability distribution on Rp x M and A E 4 The X-regularized Fréchet regression function for is map (A). RP - M such that
p (1) (x)
arg min R((A (y55), where (()()) -x (x, .f (Y,,))
DEM
and w) ( (z',x) = 1 (x' po) (ST(A)(E..)) (ar pv).
(9)
5

---

When D, = (I) E x M:iE [n]} is an IID sample from Px,,, the A-regularized estima- (A) tor PD, subsumes the sample-analogue estimator @D, in (4) as a special case where À = 0.
Connection to principal component regression. Here we remark that when M is Euclidean space, the regularized Fréchet regression function vp( effectively reduces to the principal component regression. Suppose that M = and D = { ys) E Rp x R E } is a given dataset. Then PDD (x) y + 3x (z PD.) where y 1 Ei= y/ and 3x [SVT (A) (Ep.)]' [ E
[n]}
(DD.) (y: m)-. Observe that 32 is exactly the regression coefficient of principal component
regression applied to the centered dataset Dettr = {(x; uD,, Vi D) : i [n]] using & principal components with k = max.e(p] (<a(E) 2 }}.
4 Main results
In this section, we investigate properties of po for 2 O, with a focus on two cases: V = Dr and v = Dn, cf. Section 3.1 By denoting the true distribution that generates (X, Y) as v², we can express Pglo as -/00 To analyze the discrepancy between the estimator u (x)) and (2), examine the relationships depicted in the schematic in Figure ! Our theoretical findings can be summarized as follows: Even in the presence of covariate noises, ( with suitable À > 0 can
Dr, effectively eliminate the noise in to estimate X, thereby reducing the error in estimating Pgloo
Population
Finite-sample
Errors-in-variabbl
40
Unregularized
(glo(()) 30 (x)
PDD (x)
(x)
Dn
(A 0)
Lemma
Theorems I& 2
(Apeendix)
Regularized
(2) PD (x)
(x)
(A 0)
Lemma (Appendix)
Theorem 3
Dr
Figure A schematic for the relationship between the regularized Fréchet regression estimators.
4.1 Model assumptions and examples
We impose the following assumptions for our analysis.
(CO) (Existence) For any probability distribution & and any X E R4, the object up (x) exists (almost surely) and is unique. In particular, inf dl R(y, R( (
(A)
for all where R(y:z) R (y),
(CI) (Growth) There exist Dg > 0, Cg > 0 and 1, possibly depending on x, such that for any probability distribution v and any X E I.+,
(x)) < Dg ddy. ( (z))
[d(y. (x))
!1A (v:z) ((() (0. (A) (x); 2 Cg d(v, po (x)) R (y:z) R! (pb (x):
(x)) 2 Dg
2C De-
(10)
(C2) (Bounded entropy) There exists C. O. possibly depending on y, such that
S 1 + log M2 (B. 55) de S Co,
lim sumsup 8-20
(11)
where B) (V' E M: d(y. v') s 8) and ((S e) is the e-covering number of S.
The formal definition of covering number is provided in Appendix A see Definition

---

Assumption (CO) is common to establish the consistency of an M-estimator [55 Chapter 3.2]; in particular, it ensures the weak convergence of the empirical process 12 to the population process
R/A) implying convergence of their minimizers. Furthermore, the conditions the curvature (CI)
and the covering number[ [umber[C2) (C2) the behavior of the objectives near the minimum in order to obtain rates of convergence; is worth mentioning that (C2) corresponds
(locally) bounded
entropy for every y E M, while (PI) in 40 requires the same condition only with Le (r).
These conditions arise from empirical process theory and are also commonly adopted 40
Here we provide several examples of the space M. in which the conditions (CO) (CI) and (C2) arc satisfied. We verify the conditions in Appendix A see Propositions 2
and
Example Let M = -H, das) a fifiiie-dimennionnll Hilbert space H equipped with the Hilbert- Schmidt metric (Hs(y1, 1222 = (y1 y2, V1 3/2""², e.g. M (IR", d2) where d2 is the la --metric. Example 2. Let M be W, the set of probability distributions on R such that Sg x² dG(x)) < co. equipped with the Wasserstein metric dw defined as
dw (G1, (2),,G)2= S. (CT (CT' dt,
where GT¹ and G52² are the functionns of G1 and G2, respectively. See [40) Section 6]. Example 3. Let M {M& E R M M M and Ma = 1, E [r]} the set of corre- lation matrices of size T. equipped with the Frobenius metric, dx(M, ||M M Example 4. Let M be a (bounded) Riemannian manifold of dimension r, and let dg be the geodesic distance induced by the Riemannian metric.
4.2 Theorem statements
4.2.1
Noiseless covariate setting
We first verify the consistency of the X-regularized Fréchet regression function as follows.
Theorem 1 (Consistency). Suppose that Assumption (C0))holds (Co]
If diam (M) co, then any I such that S (in(),(Ev))) (E{E))) > 0}. and any x E R",
(x)) 4/ (x)) or(1)
(12)
I a
min oz (E,.) 0). then the regularized estimator spD, (1r) effectively
reduces to the same the sample-analog estimator PD. (x) in in the limit -> co. Thus, PD
inherits the consistency of @D.. We provide detailed proof of Theorem I Appendix B
(A) In addition to the consistency of Ppp. in the small À limit, we present an analysis for the convergence
rate of PD. (a) in the following theorem.
Definition 5. The Mahalanobis seminorm of x induced by a positive semidefinite matrix Σ is | (x >1))))'' 1/2
Theorem (Rate of convergence). Suppose that Assumptions (C0) C2) thenfor any x R4 and x E IRP such that || My- -IE, S
hold. If diam (M) < oo.
C.D.
diam (M) 1 vank Y...
(2) (x), //00 (x))) = OO =x)(bx())=) + 20-11 as n + 00, a( a(pb.
(13)
where dy (x) = rank (Σy- -/AA) $ -||2 Mw 11s x2())- (()-
We obtain Theorem 2 by showing a "bias" upper bound df ((p (x), P (x)) (bx and (A)
a "variance" bound d ((p(b)) (x). / (z)) = Or(n (---)_)))) ); see Lemmas and in Appendix C
Here we remark that bx (x) is a monotone non non-decreasing function of X, and if (00) (2,,) ) then
by (x(z) 0. Also, the condition on || p.- ||_.. - is introduced for technical reason, and can be
removed when Dg = co. Note that Condition holds with Dg = oo and a for Examples
and Thus, we have d (x), po (x)) Op (xx +n +
7

---

4.2.2 Error-pront covariate sefting
Given set D = [(.;, i E [n]]] let DD, == [ Σnl E Ruxp, We let X = X and
Z= = X B. for shorthand, and further, we let Xetr (I - 1 X and Zetr - 11,1,,1, denote the 'row-centered' matrices.
Theorem 3 (De-noising covariates). Suppose that Assumptions (CO) and [T]
hold. Then there exists
a constant C > O such that for any X E R4, if E HDD. + rowsp ctr and
1
C. Dg
((A)(X (X ctr A (()) (Zar) (aa)) -1);
|| H.D. I1xion S
(14)
2diam (M)
2 x1
then
2. ||0 - 1---DDll MDD sion
+1> +/
(2) (1)) S SC. d ( (x).
1 - (x))(X (tt) A (A) Zetz)
(15)
C9
Note that the condition on || uy- ||اا- in 14 can be removed when Dg = co. We highlight that ||-X| the quantity ( ( acts as the reciprocal of the signal-to-noise ratio. Here, || X|
quantifies the magnitude of the "noise" in the covariates, while min (x())(Xtt)) measures the strength of the "signal" retained in the X-SVT of the design matrix. We observe that the error bound (15) increases proportionally to the normalized deviation of x from the mean, MDD., which is a reasonable outcome. For the complete version of Theorem and its proof, refer to Appendix D
(Zzz))
Remarks Theorem 3 We avoid imposing distributional assumptions on the noise to ensure broad applicability of the result. Also, the inequality (15) is sharp, as there is a worst- case noise instance that attains equality (up to a multiplicative constant). Despite its generality, this upper bound highlights effective error mitigation in specific scenarios. For instance, consider well-balanced, effectively low-rank covariates X Rxx such that [Xxyl = 22(1) for all i, j and 01 (X) x (X) >> (x) x (x) = O()) where r < n. A is the effective rank of X. Then 01 (x)² xx)==x0 xx |XP//r Z np/r. Additionally, if = X + E where E is a random rows, then | X|| /n + vp with high probability. In the random design scenario where the rows of X and the test point are drawn IID from the same distribution, | 15--4m UDD ||x & I with high probability. Consequently, the upper bound in (15 is bounded by vr/p + vr/n, which diminishes to 0 when r < n Ap.
Z X,
matrix with independent
4.3 Proof sketches
Proof of Theorem
We show that RDp (y: x) weakly converges to R (y: x) in the 6 (MM)sense.
According to (55] Theorem 1.5.4], it suffices to show that (1) R (y)x), - R (1;x) = (1) for
all y E M, and (2) ne!. is asymptotically equicontinuous in probability.
Proof of Theorem 2 We prove upper bounds for the bias and the variance separately.
To control the bias (Appendix Lemma D. we show an upper bound for -
R()
and convert it to restrain the distance between the minimizers d d((())()) (x), ((x)) using the Growth condition (CI) In this conversion, we employ the "peeling technique" in empirical process theory.
To control the variance (Appendix(C) C Lemma 2)
we follow a similar strategy as in Lemma I but with
additional technical considerations. Defining the 'fluctuation variable' Z((A) (y;; == R D. (v:
R (v;):) parameterized by y E .M, we derive probabilistic upper bound for R,A) (P2, (x):
R(A (o (x); by establishing a uniform upper bound for z,(ai (y;x) 2((2) (((x); x)); here, the
Entropy condition (C2) is used. Again, we use the Growth condition (CI) and the peeling technique
to obtain a probabilistic upper bound for the distance d( d() (z), p. (z)).
Proof of Theorem Expressing the difference in the weights u (A) (y: x) - w. (y:: x)) terms of
D.
X and Z, we utilize classical matrix perturbation theory to control R(A) (y; R!A x)) and (A)
transform it to an upper bound on the distance d( (2) D., (x)) PDD (x)) using the Growth condition
(C1)
8

---

Experiments
In this section, we present numerical simulation results to validate and support our theoretical findings. We focus on global Fréchet regression analysis for one one-dimensional distribution functions (Example These simulations cover various conditions, allowing us to evaluate and compare our methodology's s performance with alternative approaches. For a summary of the experimental results, please refer to Figure 2 and Table] Further details about simulation settings and additional results
are provided in Appendix B
Experimental setup. We consider combinations of p é {150, 600} and {100, 200, 400).
The datasets D,, = {(X,, ; E [m]] and D, = {(Z Y,) : E [n]] are generated as follows. Let X; N 0p. Σ) be IID multivariate Gaussian with mean O, and covariance Σ such that spee (Σ)
{Ky > E [p]} is an exponentially decreasing sequenee such that tr (Σ) = Σ₁ Rig = p
and K1 Kip 10 Note that Σ}= K2 - Ky 0.9, and thus, E is effectively low-rank.
We generate Z; following (5) under two scenarios Eij !! N( o²)) and Eij 1P Laplace(O, oz), respectively. Lastly, given X = z, let Y be the distribution function of N (Ha,,(z)) + +n, r²). where (i) (x.)(x) = a + BTx 2 with = 1 and 3 = p 1/2 1p; (ii) 17 ~ (0, of);; and (iii) 7² (G $2). an inverse gamma distribution with shape $1 and scale We performed B 500 Monte Carlo experiments by drawing ! and Di!h) as independent copies of D, and Dn, respectively, for b [B].
Performance evaluation. We assess the in-sample and out-of- out-of-sample performance of the Fréchet regression function estimator by using the mean squared error (MSE) and the mean squared prediction error (MSPE). To this end, we create a "test set" "N = {(XTew Ym 1 E [N]], with N = 1000. The MSE and the MSPE are computed the average of squared detric-distance residuals from the observed responses in the "training set" 'Dn and in the "test set" ") respectively:
R
M)E((
Σ dw (T.p/((X)) and MSPE(
Σ dw (Y, ,p!A1 )x
We report the MSE averaged over B 500 random trials: MSE( E and likewise for MSPE. Furthermore, we evaluate the accuracy and efficiency of the estimator using bias and variance, with detailed definitions deferred to Appendix
)
Simulation results. Our numerical study demonstrates that the proposed SVT method consistently improves both estimation and prediction performance, especially in the errors-nn- inaviiaallles setting. Figure 2highlights how the SVT estimator outperforms the naive errors- in- variables (EIV) estimator, which corresponds to SVT with À 0. The naïve EIV suffers from an intrinsic model bias, called the attenuation effect 13 as regresses responses on error- ppoo- covariates. This leads to a misrepresentattion of the association between responses and true covariates, potentially leading to statistical inference based on a mis-specified model.
REF
MSPE
EIV
Squared Blas
0.26
EVT
Varlanoa
0.30
Varfanne
MSPP
MSPE
0.7
0.5
0.6
08
0.10
Squared
06
0.5
0.08
0.4
0.50
0.25
0.52
0.26
1.04
1.31
1.57
0.0
0.5
1.0
1.5
Threshold
Threshold
Figure 2: Comparison of the prediction performance of yp.. (REF), (0)
40. (EIV), and (X) (SVT)
q D.
(left), and the trade-off between the bias and the variance (right) for B = 500, p 50 and 100.

---

Table 1: Average performance of PDD, (REF),
PD.
(0)
(EIV), and PB. (SVT) in various settings
(A)
(boldface indicates the best). The choice of threshold X for SVT is detailed in Appendix E
Error distribution
Sample size
71
100
1 200
n
100
Criterion
REF
EIV
SVI
REF
EIV
SVT
REF
EIV
SVT
Bias
0.107
0.164
0.312
0.040
0.087
0.189
0.020
0.078
0.136
150
/Var
1.147
0,973
0.531
0.969
0.857
0.414
0.432
0.381
0.295
MSE
0.000
0.154
0.205
0.075
0.334
0.232
0.190
0.255
0.267
MSPE
1.613
1.267
0.686
1.245
1.042
0.513
0.492
0.456
0.412
Bias
0.317
0.352
0.485
0.112
0.170
0.327
0.039
0.091
0.207
Gaussian
Var
0.805
0.731
0.523
1.137
0.970
0,512
0.964
0.845
0.405
300
(O, 0.05?)
MSE
0.000
0.033
0.217
0.000
0.152
0.235
0.075
0.327
0.237
MSPE
1.045
0.956
0.808
1.602
1.267
0.667
1.232
1.025
0.509
Bias
0.568
0.592
0.619
0.311
0.353
0.451
0.104
0.155
0.298
600
v Var
0.663
0.631
0.598
0.799
0.735
0.589
1.135
0.956
0.510
MSE
0.000
0.016
0.067
0.000
0.036
0.112
0.000
0.153
0.208
MSPE
1.075
1.062
1.054
1.039
0.968
0.858
1.602
1.242
0.657
Bias
0.102
0.112
0,286
0.045
0.053
0.171
0.019
0.025
0.121
150
Var
1.155
1.113
0.549
0.971
0.949
0,421
0.431
0,416
0.297
MSE
0.000
0,026
0.187
0.075
0.151
0.218
0.189
0.205
0.253
MSPE
1.633
1.538
0.688
1.254
1.207
0.513
0.489
0.477
0.406
Bias
0.321
0.325
0.491
0.105
0.117
0.344
0.043
0.048
0.198
Laplacian
300
Var
0.805
0.794
0.525
1.140
1.099
0,511
0.960
0.929
0.413
DE(0, 0.05)
MSE
0.000
0,003
0.233
0,000
0.025
0.227
0.076
0.148
0.229
MSPE
1.049
1.034
0.814
1.610
1.521
0.677
1.226
1.169
0.514
Bias
0.566
0.568
0.608
0.312
0.317
0,443
0.102
0.109
0.282
600
/ Var
0.664
0.661
0.614
0,800
0.792
0.606
1.134
1.094
0.521
MSE 0.000
0.001
0.074
0.000
0.004
0.157
0.000
0.025
0,200
MSPE
1.073
1.071
1.059
1.045
1.035
0.872
1.602
1.515
0.662
Remarkably, the SVT estimator achieved a smaller MSPE even compared to the oracle estimator (REF) obtained from the error-free sample. Although the REF estimator had the smallest MSE due to its small bias, we observed its overfitting to the training sample, resulting in poor prediction performance. Notably, even the naïve EIV estimator outperformed the REF estimator in MSPE. We believe this is mainly because the true covariate matrix was nearly singular in our simulation setup, causing multicollinearity issues for the REF. In contrast, measurement errors introduced non-ignorable minimum singular values in the EIV covariate matrix, unintentionally mitigating multicollinearty for the naïve EIV and causing it to behave like ridge regression.
Discussion
This paper has addressed errors-in-varibbee regression of non-Euclidean response variables through the (global) Fréchet regression framework enhanced by low-rank approximation of covariates. Specif- ically, we introduce a novel regularized (global) Fréchet regression framework (Section(3) 3). which combines the Fréchet regression with principal component regression. We also provide compre- hensive theoretical analysis in three main theorems (Section 4. and validate our theory through numerical experiments on simulated datasets. Moreover, our numerical experiments demonstrate empirical evidence of the effectiveness and superiority of our approach, reinforcing its practical relevance and potential impact in non-Euclidean regression analysis.
We conclude this paper by proposing several promising directions for future research. First, it would be worthwhile to explore the large sample theory for selecting the optimal threshold parameter À in the proposed SVT method, in order to characterize the theoretical phase transition of the -taiinar trade-off in the regularized (global) Fréchet regression. Second, we believe that our framework could be extended to errors-in- in-ois-b--varibless Fréchet regression for response variables in broader class of metric spaces, e.g., by leveraging the quadruple inequality proposed by Schötz 44 45). Lastly,
investigating the asymptotic distribution of the proposed SVT estimator would be highly appealing in the statistical literature, as it would enable us to make statistical inferences on the conditional Fréchet
mean in non-Eucheeeee -on-Eullidan spaces 66 g with errors-in intoos-i-vvarablle variables covariates.
10

---

Acknowledgments and Disclosure of Funding
DS acknowledges support from the National Science Foundation under Grant No. CCF-2212326.
References
[1] Bijan Afsari. Riemannian L center mass: Existence, uniqueness, and convexity. Proceedings of the American Mathematical Society, 139(2):555-673, 2011.
[2] Anish Agarwal, Devavrat Shah, Dennis Shen, and Dogyoon Song. On robustness of principal component regression. Journal of the American Statistical Association, 1731575 2021.
[3] Adil Ahidar-Coutio Thibaut Le Gouic, and Quentin Paris. Convergence rates for empirical barycenters in metric spaces: Curvature, convexity and extendable geodesics. Probability Theory and Related Fields, 177(1):323-36) 2020.
[4] Gustavo Amorim, Ran Tao, Sarah Lotspeich, Pamela Shaw, Thomas Lumley, and Bryan Shepherd. Two-phase sampling designs for data validation in settings with covariate measure- ment error and continuous outcome. Journal of the Royal Statistical Society Series A: Statistics in Society, 184(4): 11333688888888 2021.
[5] Marc Arnaudon and Laurent Miclo. Means in complete manifolds: Uniqueness and approxima- tion. ESAIM: Probability and Statistics, 18:185-20 206, 2014.
[6] Dennis Barden, Huiling Le, and Megan Owen. Central limit theorems for Fréchet means in the space of phylogenetic trees. Electronic Journal of Probability, 18(25):1-25, 2013.
[7] Alexandre Belloni, Mathieu Rosenbaum, and Alexandre Tsybakov. Linear and conic pro- gramming estimators in high dimensional errors- virriabbles models. Journal of the Royal Statistical Sociery. Series (Statistical Methodology), pages 939 939-956996991 99699,56,2 2017.
[8] Rabi Bhattacharya and Lizhen Lin. Omnibus CLTs for Fréchet means and nonparametric infer- ence non- -uc-ddnld spaces. Proceedings of the American Mathematical Society, 145(1):413- 428, 2017.
[9] Norman E Breslow and Thomas Lumley. Semiparametric models and two-phase samples: Applications to Cox regression. IMS Collections, 9:65-77, 2013.
[10] Louis Capitaine, Jérémie Bigot, Rodolphe Thiébaut, and Robin Genuer. Fréchet random forests for metric space valued regression with non Fuclidean predictors. arXiv preprint urXiv:/9066 0/741, 2019.
[11] Raymond J Carroll, Helmut Küchenhoff, Frederick Lombard, and Leonard A Stefanski. Asymp- totics for the SIMEX estimator in nonlinear measurement error models. Journal of the American Statistical Association, 91(433):242-255 1996.
[12] Raymond Carroll, David Ruppert, Ciprian M Crainiceanu, Tor D Tosteson, and Margaret R Karagas. Nonlinear and nonparametrie regression and instrumental variables. Journal of the American Statistical Association, 99(467):736-755 -503666--..00 2004.
[13] Raymond J Carroll, David Ruppert, Leonard A Stefanski, and Ciprian M Crainiceanu. Mea- surement Error in Nonlinear Models: Modern Perspective. Chapman and Hall/CRC, 2006.
[14] Benjamin Charlier. Necessary and sufficient condition for the existence of a Fréchet mean on the circle. ESAIM: Probability and Statistics, 17:635-649 2013.
[15] Yan Mei Chen, Xiao Shan Chen, and Wen Li. On perturbation bounds for orthogonal projections. Numerical Algorithms, 73(2):433-444, 2016.
[16] John R Cook and Leonard A Stefanski. Simulation-extrapolaai estimation in parametric measurement error models. Journal of the American Statistical Association, 89(428):1314-132 1994.

---

[17]
Abhirup Datta and Hui Zou. CoCoLasso for high- -ihhemmisniio amor-in-variable regression. The Annals of Statistics, 45(6):2400-244 2017.
[18] Aurore Delaigle, Jianqing Fan, and Raymond Carroll. A design-adaptive local polynomial estimator for the iaroor-in-variablee problem. Journal of the American Statistical Association, 104(485):348-355 2009.
[19] Aurore Delaigle, Peter Hall, and Wen-xin Zhou. Nonparametric covariate-adjusted regression. The Annals of Statistics, 44(5):2190-22) 22000022222222220002 2016.
[20] Shanshan Ding and R Dennis Cook. Matrix variate regressions and envelope models. Journal of the Royal Statistical Society Series B: Statistical Methodology, 80(2):387-408, 2018.
[21] David Donoho. 50 years of data science. Journal of Computational and Graphical Statistics, 26(4):745-766, 2017.
[22] Paromita Dubey and Haos-G Müller. Functional models for time-varying random objects. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(2):275-327, 2020.
[23] Juan José Egozcue, Josep Daunis-I-Estadella, Vera Pawlowsky-Glabn Karel Hron, and Peter Filzmoser. Simplicial regression. The normal model. Journal of Applied Probability and Statistics, 2012.
[24] Jianqing Fan and Elias Masry. Multivariate regression estimation with errors-n--variables: Asymptotic normality for mixing processes. Journal of Multivariate Analysis, 43(2):237-271, 1992.
[25] Jianqing Fan and Young K Truong. Nonparametric regression with errors in variables. The Annals of Statistics, pages 1900-1925, 1993.
[26] Maurice Fréchet Les éléments aléatoires de nature quelconque dans un espace distancié. In Annales de I' I'Institut Henri Poincaré, volume 10, pages 215-310, 1948.
[27] Aritra Ghosal, Wendy Meiring, and Alexander Petersen. Fréchet single index models for object response regression. Electronic Journal of Statistics, 17(1):1044-111; 2023.
[28] Kyunghee Han, Hans-Georg Müller, and Byeong U Park. Additive functional regression for densities as responses. Journal of the American Statistical Association, 2019.
(29) Xuming He, David Madigan, Bin Yu, and Jon Wellner. Statistics at a crossroads: Who is for the challenge. In Workshop report: National Science Foundation. https://www. nsf. gov/apssdm/ddocmmnnsSSSttttttst at a Crossroads Trossroads_Workshopp
_2t1t. pdf. 2019.
[30] Matthias Hein. Robust nonparametric regression with metric-space valued output. Advances in
Neural Information Processing Systems, 22, 2009.
[31] Alan Izenman. Modern Multivariate Statistical Techniques: Regression, Classification, and Manifold Learning. Springer, 2008.
[32] Ian T Jolliffe. A note on the use of principal components in regression. Journal of the Royal Statistical Society Series C: Applied Statistics, 3100:300-333, 1982.
[33] Ioannis Kalogridis and Stefan Van Aclst. Robust functional regression based on principal components. Journal of Multivariate Analysis, 13:393-415. 2019.
[34] Eric D. Kolaczyk. Statistical Analysis of Network Data. Springer, 2009.
[35] Thibaut Le Gouic and Jean-Michel Loubes. Existence and consistency of Wasserstein barycen- ters. Probability Theory and Related Fields, 168:901-917, 2017.
[36] Hua Liang, Wolfgang Härdle, and Raymond J Carroll. Estimation in a semiparametric partially linear grrors-in-varialle model. The Annals of Statistics, 27(5):1519-1535 1999.
[37] Zhenhua Lin and Hans-Georg Müller. Total variation regularized Fréchet regression for metric- space valued data. The Annals of Statistics, 4966):3500-3533 2021.
12

---

[38] Po-Ling Loh and Martin J Wainwright. High- nigh-dimensional regression with noisy and missing data: Provable guarantees with non-convexity. Advances in Neural Information Processing Systems, 24, 2011.
[39] James Stephen Marron and lan L Dryden. Object Oriented Data Analysis. CRC Press, 2021.
[40] Alexander Petersen and Hans- Georg Müller. Fréchet regression for random objects with Euclidean predictors. The Annals of Statistics, 47(2)699-719, 2019.
[41] James O Ramsay and Bernard W Silverman. Functional Data Analysis. Springer, 2nd edition, 2005.
[42] Philip T Reiss and R Todd Ogden. Functional principal component regression and functional partial least squares. Journal of the American Statistical Association, 102(479);944-996, 2007.
[43] Susanne M Schennach. Instrumental variable estimation nonlinear errors-in vaviiablles models. Econometrica, (5(1):201-239)
2007. 2007.
[44] Christof Schötz. Schötz. Convergence rates for the generalized Fréchet mean via the quadruple inequality. Electronic Journal of Statistics, 13:4280-4328 43000-4345, 2019.
[45] Christof Schötz. Nonparametric regression in nonstandard spaces. Electronic Journal of
Statistics, 16(2):4679-4741, 2022.
[46] Damla Şentürk and Hans-Georg Müller. Covariate-adjusted regression. Biometrika, 92(1):75-89, 2005.
[47] Donna Spiegelman, Aidan McDermott, and Bernard Rosner. Regression calibration method for correcting measurrmeent-error bias in nutritional epidemiology. The American Journal of Clinical Nutrition, 65(4): 1997.
[48] Florian Steinke and Matthias Hein. Non-parametri regression between manifolds. Advances in Neural Information Processing Systems, 21, 2008.
[49] Florian Steinke, Matthias Hein, and Bernhard Schölkopf. Nonparametric regression between general Riemannian manifolds. SIAM Journal Imaging Sciences, 3(3):527-563, 2010.
[50] Gilbert W Stewart. On the perturbation of pseudo-inverses, projections and linear least squares problems. SIAM Review, 19(4):634-662, 1977.
[51] Karl-Theodor Sturm. Probability measures metric spaces of nonpositive curvature. Heat Kernels and Analysis on Manifolds, Gruphs, and Metric Spaces: Lecture Notes from Quarter Program on Heat Kernels, Random Walks, and Analysis on Manifolds and Gruphs, 338:357, 2003.
[52] Renáta Talská, Alessandra Menafoglio, Jitka Machalová, Karel Hron, and E Fišerová. Com- positional regression with functional response. Computational Statistics & Data Analysis, 123:66-85,, 2018.
[53] Kristin M Tolle, D Stewart W Tansley, and Anthony JG Hey. The fourth paradigm: Data- intensive scientific discovery [point of view]. Proceedings of the IEEE, 933311333-1 2011.
[54] Danielle C Tucker, Yichao Wu, and Hans-Georg Müller. Variable selection for global Fréchet regression. Journal of the American Statistical Association, pages 1-15, 2021.
[55] Aad W Vaart and Jon A Wellner. Weak Convergence and Empirical Processes. Springer, 2nd edition, 2023.
[56] Roman Vershynin. High-dimensional Probability: An Introduction with Applications in Data Science, volume 47. Cambridge University Press, 2018
[57] Cinzia Viroli. On matrix-variate regression analysis. Journal of Multivariate Analysis, 111:296- 309, 2012.
13

---

[58] Takumi Yokota. Convex functions and barycenter on CAT(I) of small radii. Journal of the Mathematical Society of Japan, 68(3):1297-1323. 2016.
[59] Zemel and V M Panaretos. Fréchet means and procrustes analysis Wasserstein space. Bernoulli, 25(2):932-976, 2019.
14

---

A Verification of the model assumptions
A.1 Additional background
Definition (s-net and covering number), Let (M, d) be a metric space. Let C T be a subset and let g 0. A subset N _ S is called E-nct of S if every point in S is within distance of some point N, i...
vx S, Aro E such that d(æ, 2o) e. The e-covering number of S. denoted by IT(S,),, is the smallest possible cardinality of an E-net of i.e,
h M(S,e) = min k KEN: By...... Ukk E M such that S S ealune))
(16)
where Baly,e) E) = [v' E M d(y. v/ s E} denotes the closed e-ball centered at y E M.
Let (0, 1) == {x E R' |0112 I 1 denote the unit --norm ball in R It well known that for any > 0,
J
(B)(0,1)), 3 2 +11 (2+1)²
(17)
A.2 Example Euclidean space
Proposition 1. The space (H, dhs) defined in Example 7 satisfies Assumptions (CO) C1) and (C2)
Proof of Proposition For any probability distribution and any A R+, let y E, (w]] xX, y]. Then we observe that (v: E,
(2)
[w (X, x) d (Y,
Er w (X, x) a|Y y||²
Ey
(w(a) (X, x)) ||Y 11a1 I1 1
Ey w (2) x) (V- - y(ai)
=R x) + Ily ( 1
As R (yr,) is strictly convex and coercive function, there exists unique minimizer, 1pb Thus, Condition (CO) proved. Furthermore, Condition (CI) is also satisfied with Dr = oc, Cg 1, and 2.
Lastly, for any y H. and any 0,
2 (2+1) S0.5- 5
M(Baus (y, 6), ôs) M7 BBass 1).
where y= dim H. and C > 1 is constant that depends only; see the covering number upper bound (7]. Thus, the integral [1] is bounded as follows:
S/' V 1 log (BL(((()))))))))))) 05) de & f V1+ log C - r log de
1 C Vr f'
-- loge ds
VI I log C + Vr e etdi
Vra
/1 11++lgg C' +
2
See (56 Corollary 4.2.13) for example.
15

---

using the change of variable t - -loge. Therefore, Assumption (C2) holds with C. = vI +1+log log C
2
A.3 Example 2
Set of probability distributions
Proposition 2. The space (W, dw) defined in Example satisfies Assumptions (CO)
C1
and (2)
Proof of Proposition For probability distribution function y W defined R, let Q Q(W) = (Q(y) E W] denote the collection of corresponding quantile functions, where (Q())) (u) y (u) for E [0, 1].
We note that - Eg [w) (A) (X, [0) is bounded linear functional defined (0) 1] because IE, wwb (A) (X,x)² 2 < 2+ [1 (1==- implies that E,, [w]]] (2) xx),)))l(((()))) is bounded.
It follows from the Riesz representation theorem that there exists (()) E [ 1] such that
[ (X, (Q(Y):
gb2] (f( g)2
(18)
for all E 1,2 1]. Then, we have
R E,, [w. (X, (x,x))1((() f( 12 + +1Q(2) f(A'
(19)
which yields that
(x) Q-
|rg min ||Q - (()) QEQ
arg
(20)
The condition (CO) follows from the convexity of (2, || (12). Moreover, the convexity also gives
(Q(v) Q(p (x)), Se (x) ((p)) & 0 for all E W, so that
R$A0 (y; (e))))))))
[1Q(y) (x) 11Q( (x)
11Qy)) (c)!) (x))ll2 2(Q(z) - (x), (()))()))))
(21)
11Q(v) - (()))())))
diy (y. (x))
Therefore, the condition (C1) holds for any arbitrary constant Dg with Cg 1 and 2. Finally, we refer to sup log (Ba (y, 5), D.=) < sup log R(B,, 8), C --
40 Proposition 1] to ensure that for any and any E 0,
(22)
vEW
QEo
holds with an absolute constant C > 0. Technically, this fact comes from the covering number bound for a class of uniformly bounded and monotone functions in L This confirms that the entropy condition (C2)
A.4 Example Set of correlation matrices
Proposition 3. The space (M,dp) defined in Example 3 satisfies Assumptions
CO
CI and (C2)
Proof of Proposition 3 First of all, we note that M is convex subset of S {X Rexr
XT), which is the set of symmetric matrices. It is because M S 0 H where St denotes
the cone of r x positive semidefinite matrices and == [X E St X = 1, Vi E [r]] denotes an affine subspace of S", both of which are convex.
Next, we observe that S equipped with the Frobenius metric de is isometrically isomorphic to Rr(r+1)/2 equipped with the 22-metric. Hence, (M, dr) satisfies Assumptions (CO) (CI) and (C2)
inheriting these properties from the ambient space S", which is established by Proposition We
note that the inheritance of (CO) (CI) relies on the convexity of M, while (C2) is inherited simply
based on the inclusion M C S
16

---

A.5 Example Bounded Riemannian manifold
Proposition 4. The space (M, dg) defined Example 4 satisfies Assumption C2) Riemannian metric is equivalent to the ambient Euclidean metric.
provided that the
Furthermore, let M be the tangent space of M y, and Exp, T M. M be the manifold
exponential map at y. Let gu (2) (u; R (Ey(u), for u T MM If (CO) holds and the
Hessian of gu (u: pp (I), is positive definite, then (C) for some D3 > 0.
Proof of Proposition 3 Since M has finite dimension and is bounded, the bounded entropy condition (C2) follows from the metric equivalence.
(2) (x)) Consider E .M such Suppose that (CO) holds, and let â > 0 be the injectivity radius at 1p
that dyy spv
(A) (x)) < and My Log Log (2) (v). Then we have
R for some y between and Mly Since u., Mly = dl. 4/ (2) (x)) and ( is continuous, the positive definiteness of V Vagg (A) (7y) implies (C1) with =.
(yxx) R
(p/ (x): / (uyi pb (z), 2)) g.²¹) (0: yp (x). u. v 9 (ūy)) By
B Proof of Theorem 1
Proof of Theorem Recall from Definition 4 cf. (9. that for any probability distribution v on RP, any X E R4, and any IRP, the X-regularized Frechet regression function evaluated at is given as the minimizer of a function A!(A) as (x)
(x) = arg min R. (y: x) NEM
where
(y: E. (x w (x, 1 (z' = 1+ (x' [w) SVT (2))) pv).
d dd v)
and
In this proof, we follow similar strategy to that in the proof of [40 Theorem 1]. Specifically, suffices to show SuPyE. (y: x) R (y; converges to zero in probability, due [55
Corollary 3.2.3]. To this end, we show R (y; x) weakly converges R (y:z) in the ex (M)- sense, and then apply [55] Theorem 1.3.6]. Again, according to [55] Theorem 1.5.4], this weak
convergence can be proved by showing that
(SI) R (yi x) R (yr x) = (1) for all y M, and
(A) (S2) D. is asymptotically equicontinuous in probability, i.e., for any E, > 0, there exists 8 >> such that
lim simsup/ P
di (2152)) - R (J2i x) > < 17.
sup
n
In what follows, we prove these two statements, Theorem[]
(SI) and (S2)
thereby completing the proof of
Step 1: proof of (S1) First of all, we observe that
R (y) x) 1 x) = (v: 12 (v): + (R [n (vi x) R. (yr (2:
(23)
:Ti
=:T²
We separately analyze the two terms Ti and T2 below to show Ti = (1) and T2 op(1) as
+
17

---

(i Ti o,()).
Let D, =( Yi) : [n]], and we re-write
Ti - n i=1
1
Σ
DD (X;, - -w (X,,))) d vY
Letting pn = ,D,, En D',,, and Σ((A) = =SVT(A) for shorthand, we observe that
(2) Dr (X - w W (X x)) = (X, - -im) (g E// (xr pmn)
Let x [X, Xn] T
E Rnxx, and note that En = n1 (X - 11,12)))) (x - -1nf())
Then it follows that
Σ (X; Tnn
(()) 2'n
Et
1 1 1nd(n)
2n 2/10
Consider a singular value decomposition of X 1m namely,
min(n.p}
X - 1np = Σ u.u
and observe that Σn = Em $² -. is an eigenvalue decomposition of Σn, Letting
mm)
D = span 1/ : E [p], 0 < S VA be subspace of IR' spanned by the eigenvec-
tors of Σ., corresponding to the nonzero eigenvalues no greater than X, we have
1
1(s,
VAJ m.u.
1 0} 0.1
i=1
Σ
110 < 8, S VRJ faan
i=1 $
2
EI II vua
= n (X 1I ) (x 1x
(24)
where II yas denotes the projection matrix onto the subspace v(Aa) Note that ((,)
and only if min {i E [p) 0 V 0.
Therefore, we have
(2) (x,, - w/ (x.. -d2(),,)) v) D.
T
i=1
diam (M)²
1 1[(X (X- - -1,7t1) (S(A))
Ett (x Pm)
diam (M)² 1 X - 1n 1,T II -I (x pm))
24
(1))
The last line follows from the fact that supsejy (o,(Enn) - -xx(E))) + 0 in probability, and thus, Hy + 0 in probability.
iii Th (1).
18

---

Letting ax; = - (0) (x,,)) (Y,, v)), we decompose T2 follows:
T2 R2, (y: - (y: x) + R., (y:xx) Ry (y:x)
WiD (Xi, w (X:; vY..))
(0)
=:22a
n
(0) (X,, (Vi.))
71
72
Note that T28 converges to in probability by the weak law of large numbers. Now it remains to show T24 = (1). To this end, we note that
D.D. (X,, x) w (X,,z) Vz,(x) Wx())
(0)
(x) -1, (x -bx) + + Et 11),
where
(w((r) = =t - pn)) - Exxx - -H).
(25)
Since Tr, and Σn, respectively converge to and Σ in probability, it is possible verify that [V (x)], [W,((>)) converge to 0 in probability. As a result, T2 also converges to 0 in probability.
All in all, we have (v: R2 (u: = (1), and thus, proved (S1)
Step 2: proof of (S2) For any U1, 1/2 M.
R (y1; R2.. (122:))
()) (X;, (L.(), d dVr
71
"'D, (X
(2)
(X,, m)
(d((i,))) + d(Y; U2)1 (d(Vi,))) - d(Yi,u2)
< 2diam (M) d(yn,32)
(2) Σ
WDD (Xi,
Op (d(31 (d))))8))
where the Op term is independent of yı, 1/2 E M. Therefore,
R (U1:T) - (v2: =p((),
which proves (S2)
C Proof of Theorem 2
In this section, we prove the two claims in Theorem 2 prove lemma that controls the bias in the population estimator (Lemma and in Section C.2 we present and prove lemma that controls the variance the empirical estimator (Lemma
Specifically, in Section C.I we present and
C.1 Bias the population estimator
| (x Tx 1/2
We recall the definition of Mahalanobis seminorm from Definition
19

---

Lemma 1. Suppose that Assumptions [CO) and r
(C))
hold. If
Cg D || py- 1 diam (M)2 vrank Eny-
(26)
then for any X Rt,
(x). $2K0. dx( ((b((z)0z)
(27)
where
1
4 4diam (M)
Ko
log
+1 and
(a- 1) log
Cg. (1-27 2a(a-))) (1-2-(a-11))
bx(z)= = rank (Ev- - 20(h
|| - Hvu 11, x2
Proof of Lemmal 7
// (x) and ((x) 1 (1) throughout this proof, dropping the subscript v². Likewise, we simply write μ = py and E = Eve.
For the sake of brevity, we write (x)
Step 1: A naive upper bound. Observe that for any X E R2, x E TR,, and E M.
R)); (((A) (y)
= =|Ep- (X - -w) 1²) (It - -E(-t) (x -4) (2))dd((.)))
Eg.
I diam (M)2 lE.- [[|X - ul1x- |1:0 - 11 5115.-5:1 )(A)
Cauchy-Schward inequality
I diam (M)2. (Ep- ||X - 1 - ) 1/2 1|x - م (
Jensen's 's inequality
= diam (M)² v rank (I - E())
12 101 --(
(28)
where the last inequality follows from E.. || - u1 - yay = rank (E D()).
We observe that the upper bound in 28 is monotone non-decreasing with respect to À E R4, and it converges to 0 as À + 0. To see this, for any X R4, we let
((A) span fo; i E [p]. 0 S<
where = DP A 2a4o an eigenvdecomposiiionn of Σ. Letting Πy denote the projection
matrix onto the subspace D(A), we note that Σ ())) = Hyya Ml and that (E - A1 =
Hya yy). Thus, rank (Σ I() = dim D(A), and furthermore, we notice that D(A) = {0}
and only X Amin = min (A: A > 0}. Therefore,
X < Amia
(y:) R( =
(() pp(z), Vz. (29)
The observation together with Assumption (CO) implies that di
= =(1) as
X + 0.
Step 2: Controlling risk difference. Next, we move on to determine the order of d d( (x), pxx))
as a function of (x)) - for fixed À E R. We may assume À > Amia for the proof because the lemma is trivial otherwise, cf. (29). Assuming À Amins may decompose the difference in the population objective at (x)(()) and co(x) follows:
(=)=)) R(p(x);x) (x)= ( (z)()) + R( (v(());)) R(p()):))
=:9
{ (p(x)))));)) R(A) (xo
20

---

We observe that both Ri and Rg are non non-negative, due to the optimality of p(x) and (x)) Then, we obtain an upper bound for Ri using similar argument as (8). Specifically,
(x)()) ((((())=)) s 1i
(X, < 2 diam (M) bx(2) d(( (zr), p(x)).
Eu- Ka w (X,x) w (x, x))
{ (())))))))))) - - dY, ((x))) H
(30)
Step 3: Converting risk difference to bias. Lastly, we convert the upper bound 30 to upper bound on the distance dz)(A)(z), (()) using Assumption (C1) To this end, we begin by confirming that
(z):z) R(((();=)) Ey. [(x - P)) .Et (x- pp) --) (
df d(, (()))) }
S diam (M)2 (En ||X ||X--1111 m1l3) 1/2
|| 1|ا
diam (M)² Vrank E || - Mll S C.g. Dg.
Thereafter, we choose an arbitrary K E N and r E R. whose values will be determined later in this proof. Then we obtain the following inequality using the so-called peeling technique:
1 (d (x), p(x)) > 2 bx(x)*}
1 1f2 ba(x)" (z) s 2h 12tt1 bx(x))
2th ba(x)) < (x) s 22 bx(x))
(x): - R({(()))) -(((())))) Cg. (x(r))
(o), z2tt1 2ettittblll bx(xx))
(C1)
(31)
Moreover, we decompose the numerator in the fraction appearing in the upper bound 31 as follows:
Combining
30 with
we have
1 { [d(pp (2) (x), p(x)) > 2k bx(r)²
diam (M) bx(x) (Io()
1 (d(p)) s 2 22t41.bb(0 bx(x)")
k=K Cg (2* (z
4 diam (M) 2-)x))-((a>
(32)
Cs
k=k 2k(a-1)
Note that C diam M 0 is constant independent of X. Letr 1/(a 1), and observe that becomes smaller than 1 for a sufficiently large K. Specifically,
the upper bound 32
K3 >
log
C5 (1-21-1-0--1) 2-(1-1) )
4 diam (M)
)
4 diam (M) Cg
Σ km
26(a-11 < 1.
(a 1) log
K
As result, the inequality "d (N) ((x))) 2 by (2r)r in the indicator function must be false, and we conclude that
(z), S 2k bx(x)=-T.
21

---

C.2 Variance of the empirical estimator
Lemma Suppose that Assumptions (CO) E spec (Ev)) ) it holds that
(CI) and (2)
hold. For any X R4 such that
d ((P Op n 2ia
Proof of Lemma 9 that
Recall from the definition of X-regularized Fréchet regression (Definition 4) and
n (X., df (Y and (w: x) = Exx,y (x, x) d d(r
R R (v:x) " w
(A)
Additionally, define auxiliary function R2 (y; as the "empirical risk with population weight" such that
Rt (v)
Σ (2) we (Xi, de (Y.. v)-
We present the rest of this proof in three steps, outlined follows. In Step show the consistency (A)
of PDD (x), i.e., d( (x), (r).)p..)
(2))
p0 (x)) = (1) as n + co. In Step 2, we define the discrepancy vari- (A)
able Z (w; x) = R (1;=) - R (y; between the finite-sample and the population objectives, cf. (35). and prove a uniform upper bound for - (y; x) that holds neighborhood of //A² (y; x)). Lastly, in Step 3, we utilize the peeling technique from empirical process theory to obtain the desired rate convergence.
Step 1: Consistency, We first claim that d (( (z), (p (z)) = (1( by an argument similar to that used in the proof of Theorem Specifcally, it suffices to show that
(S1') R. (w; R (ur; x) (1()), and
(S2') A (::): M - asymptotically equicontinuous in probability.
Note that we already showed the asymptotic equicontinuity in the proof of Theorem I see (S2) Thus, remains show the pointwise convergence in probability. To show (SI') we decompose BBJ (y: x) - R.AA) ()) as follows.
D. (y:x) - n2 (v) = { (R (1;x) Rin (v:x)) x}} + { [R (11xx) - -ReA (v,z)}
[wp (2) (X., w/ (X1,x)} dP
-//a (gar)
1
W (X,, dVr,,,) Ey [wn (X,, d (Y,,))))) w)l
7
Next, we show that (2) (y, x:) and BB(A) (y;; x)) respectively converge to 0 in probability.
Letting Pn = HD.. En = Ep,, and ((A) = SVT (Sx)) for shorthand, we can write
(1) (X,, xx) w (x), vx() x. X.T (x),
2
similarly
15 where
Vy(al (x)) = --op [ (x - Pm)) + T [ (x 1))
(33)
(x)) = [S/AN]] (xr -a--ith [im) - [ (x- u)).
22

---

Since || pll2 = Op(n-1/2) and | - E(A = 0p(n (if A & spec Σ) inde-
pendent of > 0, we also have [V / (x)l - Op(n and ||W (z))12 - 0p Op(n
This implies that A! (y: = op(1).
Moreover, we note that if || - ela < oc, then the random variable wy-- (X, x) has finite second moment
Ey- Wi- (X,
2 1 + By,
(X - (1) (>x 11)
|| X ullssN -1 11 10
(34)
2 2
1 1 Ey
$ 2
||2 - -113s)
regardless of the value of X > 0. When diam (M) < co, the product w ( x)) dY v) also has finite second moment. Since By: (y: x)) is the sample mean of IID random variables with mean zero and finite variance, it follows that
(VNar) Varl u.y (X1, de((1.8))
(y; Op
=Op(n' (122).
n
Step 2: Uniform control of the fluctuation in objective discrepancy.
For any À E E., and any
(x, y) E IRP x M, we let ((a2 (y; denote the random variable defined as
(v: = R (v: R!!. R (y;
(35)
We observed that
((a) (y: x. (p!
R RLa (v: (1;
R (x);z) R (p! (x);
Rbb)) (2: x)- - (v; x))
nel (1o (x);z) - Ro (p/ (x) }
[{ Ri (v: R! (y:2) }
Ry (4 (z)=z) - R (p!a) (z):2)) }
(2) (X,, ( (Xe, 21) } e (v:x)
(36)
W D.
1=1
=aa("
(2) (X,, (v:)) - Ey. (x(((()))) (()()))))
1=1
-23 (Btar)
where ((A) (2) = (v::)==((YYv)) d (Y
Next, analyze the asymptotic behavior of the terms, 2 x (v:z) and B 2)). Specifically, we establish upper bounds on their magnitudes that hold uniformly over 6-neighborhood of (A)) (z) 2!a (x), which will be used later in Step 3 of this proof.
23

---

Firstly, we observe that for any ó > 0,
sup (1 (v:
w Ba (v (o)): ())
(A) Dr (Xi, - w 12) (X),
(Y..)) (x))
sup
M
(x)) (z)) + |x | (2)ll2
(x)) (x))
x
sup
yEBa (v
=Op 6 nnnn" 1/2 1/2
(37)
where we used the property of ( (z) and W:l (x)
(z) discussed in the paragraph following
Since the stochastic magnitudes of Vilal (z) and w!!') (x) are independent of 6, (37)
implies that there exists c(A) C( (x) > such that for any 8 > 0,
(v: : d(y, pt (x)) < <6) cf . 18-2-122 = 1. (38)
lim inf P sup +2-00
YEM
Furthermore, for any y, 5 E R4 such that 0 S < o, let e (7. d; be defined as an event such that
(), sup { WeM
[ (v: dlu. (x)) 01 s c( .0. ...1 (39)
For any 7 [0,8]. we have (0,0);;z) s En(7,;;)), and thus, lim nnf (())) 1I
Next, we note that
(x ef (y:z) 2) s diam(MM d(y, p (x)) | (X,,
Observe that d Pp- (1)) I diam (M) < co and recall that E, [ (x,x)² <
2 + p p| 1 as shown in Step 1 of this proof, ef. (34). It follows from the uniform entropy condition (C2) Theorem 2.7.11, and Theorem 2.14.2 in [55] that there cxists
D. (x) 0 such that for all 8 E [0, D.).
{
( (y:x)] d(v. p (x)) < 6}]
E sup NEM
$ 2ddam((M) .. 1/2 1 + p|1 11l3 L'' + I(g()(((((()))))) 6), öc) de
$ <C(A), -1/2
(40)
where C.2y = 2 (C. + 1) diam (M) /I+pll= m is independent of 8 > 0 and
n >
Step 3: Concluding the proof. Lastly, we combine the results from Steps 1-2 to show that, for
any 77 > 0, there exist K K(n) > and N N()) 2 such that P ddpp)) (z), (x), p! (z)) >
2k n - < 1) for any n 2 N, where 3 > 0 is an absolute constant that will be determined later in this proof. We prove this claim using the peeling technique, in similar manner we did in the proof of Lemma 1 To avoid cluttered notation, we let A(x) = d(pb)) (a), (N) (x:) in the rest of this proof.
24

---

For any fixed K E N and a sufficiently large n n(K) 2 1 satisfying 2 nー < D. = D..
we observe that
2 n=)) = - 2 D.) + PP(2 8 <
(41)
where we used P(A) & P(B*) + P(An B) to get the inequality. As we know that P (()()) > D. o(1) by Step 1 of this proof, we focus on showing upper bound for the other term, P(2k n- s (z) D.).
Step 3-A: Decomposition of P(2k p S Axx) < D.). For each n. k E N, we define
8n,k
n
(2 8 2 D.;a),
(42)
(N
k-1
ela) n nB, 8 2n A D.: n Ell (2 2 A D2;z)"
Bn,k =
K==K
where set K-1 be the entire event space that G, 6n,K (Bn,k) It worth mentioning that G Rn, and B,,,'' mutually exclusive for any h + k 2 K, and we will use this property when concluding the proof in Step 3-C below.
Now, we observe that
p(2"mn n -8 < D.)
S P a(a) (2 - 2K+1n A D.:
+ +P((2* n - < A(x) < D.) n (((2) (2 n n g A Dz:zz)
-((Bn,) + P
2k n -3 $ A < D.) n 18aKKK
= -P(6n.k) + P
22 n 1-85500 < < 22k+1 n -8 n B=DDD.) n BBnkK
+ P +P((2K+)
g n
2 A(x) < <D.) 18nKKK
and that for every lo > K,
P ((22t11n-0 n < A(x)
D. NBmt s (2 n > (2())
<A(x)< < D.))Bn)t11) ) +P(Bn,tt2)
As result, we have
p(2K n <A(x)< = Σ P(G + p
20
Ep((2n~ =>>((x)) < 2 n A ADD (8mk)
C.a
(43)
25

---

Step 3-B: Controlling Cn.k. Next, we show an upper bound for C,,k Suppose that 2 - S A(x) <
2 n 8 A D. and the event B, occurs. Then it follows from Assumption (CI) that
Cg. (a)a
I R (PD - (p)
x H (ob. (z); / (p. (z):
Dr (10 (x): (x): - Rb (PD., (2);
20
2 (vD x) 2 (r))
cf. (35)
- 2 (1 (x): + 2 (x):
(36
- sup
{ ( (1:²)
+ B (1xx))
2t n nー S d(v. x. < 2k+ n- A D.)}
yEM
I c( (2t+1 - A.)
11/2
1/2
d /1 {133
2 -8
D.)
sup seM
(44)
Therefore, we obtain that for each & 2 K.
C, = P (2 n2^n- n
A < 2 - A D.) n 18mk
P (A(z)) 2 (2 n 0))) n8mkk
-2 n ^ D.) n 1/2 E supyem { (1 ddy.p (2 (x)) < 22- D.
(u:=))
Cg: (2 n n=
& Markov's inequality
(C(A) + c (2 D.) n 1/2
40
(45)
Og (2 n 8)
Step 3-C: Concluding Step 3. Combining
41 43 and 445
have
2((
a)
c
0 2 k(a-1)
P(((x)> > 2 n
22nnnnnnn'0_ 0))
n +
C₂
k=K
P(A()) D.
Σ P(6n..)
K
=o(1) : Step of this proof
Moreover, Bn,,k are mutually exclusive, and thus,
Σ P((m)))= = (U (Un -
(r(U) e
P 22 Di:z))
+
39
R=K
b==
K=K
Finally, we obtain the desired result by letting 3 2(a-1)
D Proof of Theorem 3
In this section, prove Theorem that establishes upper bound on d(p (1) D. (z). PD. (z)). This
section organized as follows. Firstly, in Section D. we present several useful results from matrix perturbation theory as lemmas. Next, in Section D.2 we provide a key lemma (Lemma 6 that
establishes the stability of the weight function when there is covariate noise. Lastly, in Section D.3 state and prove Theorem 4 from which Theorem can be casily derived.
26

---

D.1 Useful lemmas
Definition Let n.p E N and let M Ruxp, The row projection matrix for M, denoted by M E Rpxp, is matrix such that
Io M M.
(46)
and the column projection matrix for M, denoted by Io E [R=x", matrix such that
I = M M
(47)
We recall from (6) that for any À E R.,, the singular value thresholding (SVT) operator SVT(A) is defined such that
min(n.pp
min(n.p)
M
U. is SVD
SVT (M)
Σ
1(; A} u.s!
In the rest of this section, we let ( (VT( (M) for shorthand.
Lemma 3 (Properties of the row/column projection matrices), Let E N, and M E R=xp, For any X R4, the following statements are true.
1. M defines a projection in Rº and rank 1 M(A = rank M (A)
2. Tog defines a projection in R and rank II M rank M
3. M M( M
-Tool M
and M M' TTol (X) M
Tooo
M
Proof. Letr rank M and consider a compact singular value decomposition (SVD) of M:
M
where .1....,8, S, are non-zero singular values of M. Noticing that
M (A) (M) = E1ts > A} uuu
and that M = Er 8, veu," the three conclusions of the lemma follow straightforwardly from the orthonormality of singular vectors.
I = -E Ta 1fs; > A} is the projection onto the row space of (A).
[Tcol M(A)
= E un? 1 1s > }} is the projection onto the column space of M()
Due to the orthonormality of singular vectors,
M M
Σ st Iaii
( DooT 1{s; > 21
Σ sf vuul
Σ mnf 11s >
col
and likewise, M
M
Foo M
(
In addition, collect two classical results from matrix perturbation theory and state them lemmas.
27

---

Lemma 4 55 Theorem 3.2]). Let X, Z E Raxy, Then the following equation is true:
z x = - 1 Z X x x +
11
11x
X
(48)
where IIx I, ! and 2
I 1
Lemma 5 2[115 Theorems 2.4 & 2.5]). Let X, Z E Rux. Then
112 x
S max
(2 x)x 1 -2-)
(49)
Moreover, if rank X = rrank Z, then
| - x S min
(1 x)x |||
(50)
D.2 Stability of the weights under (small) perturbation in covariates
Let Dn ((x,, Mi) E RP M i [n]] and D. = {(z,, E E M: [r]]] be two sets in
RP x MM. We may identify these sets with their empirical distributions. Recall the definition of w from 9 for any probability measure on I2 x M, any À R4, and any æ, x' IRP,
w 2 (al ,x)
(x' -
(VT(
(wv)
where He Exxx (X) and Σ, (ar(x,y) (X), of. Z We define the weight vectors induced by Dn and D, as follows: for any X E R and any x E RP,
T (x) = W'D_ (21
(A) D. (In, x)
E R,,
(51)
Th (x) = (1) D.
(21,2) ... w (1)
(zn x)
E RR..
Dn
Dr
Lemma 6 (Stability of weights). Let D, {(x1)81 E R. M [n]] and D.
(Z, E Rº x MMiE [n]]. Let X = [2] ... znl E Rux and Z = [2] Zn] E
Raxr, For any X E R,, i E Rº satisfies x MDD E rowsp X (ou). then
(x))
(2) WD
(z)
Vn 1 XI| min (X ( e(rr
2 | uDD IIo,, +1)
(52)
D.
where X (In 1
X and a (X) = inf( > A A: A:ic N) (likewise for 2).
Proof of Lemma 0 This proof consists of three steps. In Step 1, we express the weight discrepancy ((A)
(1:) (x)
(A) D. (a:) as a sum of matrix products using projections. In Step 2, we establish upper
"
bounds on the norm of the expression obtained in Step In Step 3, we collect intermediate results together and conclude the proof.
Step 1: Decomposition of the weight discrepancy. First of all, we rewrite (7(2) (x)
w D. (1) in
a compact matrix representation that is presented (60) at the end of this step. To this end, we begin by observing that
JDD..
1 T In: n2
and ED. = n
(X In1Db)) - In/bb)
1 XT T
n
1
etr
Xetr (53)
For given R,,, we let x(A)
(A) ctr
SVT(N (X eetr). and observe that
Fow
1 x X X
Iroo (())
1
x
(A)
X
(54)
(
n
Then it follows that
x (2) T
(A) =
x (2)
x (1)
=n.
[x) (
= n
row
x (*)
Xi
ctr
(X (XT)
row
x (A)
28

---

Therefore, we have
w A
(x) = ln + (X -1nH(
(x - PDD)
= In + + Xetr
I x(2)
X ctr (x
x(A)
(x (DD))
= 1n + cool
x(2)) (x IDD.);
(55)
ctr
where the equality in the last line follows from Lemma
3
Xetr Irrow
x
X ctr
x x
Likewise, we repeat the above for D,, and Z to write
TT, 1n and
Σp. = - Zee Zer-
DDD
Then, we obtain an expression for (x) in a similar form to (55 namely,
xx) 1n + 1Tool Z(A)
(Zdr)
z(A) (x HB )
(56)
Thereafter, we define CzS Cg E Ru so that
-(1
1
Cx = x - V.D.lls..
=
nn
XT
ctr
(x
HDD.)
and
(57)
t x J.D.. |1ip
1
Z
1
(x P.D.)
Vn
ctr
Then we observe that for any R.,,
1
12-11oww (x-MD,,) n-l1
x(A)
row x (A)
X
ctr"
X
-xx (x--uD.) = Vn-lloow X Cr. (58)
x (())
vn
Likewise,
- (x-MO,,) IITow
1
Zer
(
Zesz
1
-xx- -D. /n- -Itow Z Cg (59)
Vn
Vn
2(A)
Consequently, for any x E TRP, we obtain from 55 and (56) with aid of 588 and
59 that
(7())
(x) - whb (x)
Vn 1Ieol
L (etzz)
-1 Fro
Z
Eo Vn-
x(a))
X etr
11 11
(
X cte Cz
ctr
= Vn. col
((A)
Vn. eooo (A) Cz.
x
Lemma
3
Vn. col ((A)
(ē= - cx) + Vn-
[Tool ((A)
MTool x(A) Cr.
(60)
By triangle inequality, we obtain the following upper bound:
(1) (x)
wht (x) S vn.
(2. (Cx-
(Gx-cx)
+ vn
(100)
11ool
cce
(61)
(A)
(
Step 2: Upper bounding the norm. Next, we establish separate upper bounds for the two terms in 61).
(1) The first term in 61.
First of all, we observe from the definition of ca and č. cf. 57). that
1
(x- (D-Mo.) /n X (x HDD.)
C Cr
nn
x xtre ) (x (p-)) vn. [ 1
(15, -VD..)
Vn.
29

---

Then we can upper bound the first term in (61) as follows:
Iooo
(cr
ca)
- vn.
I Z
( 1
x -rr
") (x - HDD. |oa)||t/i
I
Z
7
(HB. - HD..
2
ctr
%
(62)
Next, we consider the orthogonal decomposition of HDD..'
1
MD. =11x (I--D + x + (DD.)
Vn
X Cx 11x +
(x- (63)
If x PD, E rowsp (Xetr), then we obtain the following upper bound for the first term in 66
Zztr - -x ') (x - (.D.)
vn.
2(!!'
[icol _
(Zerr
- -x ctr )) X cco
+ vTl
[|cool
(Zor²
xxxx ))
11x
1 .xx (DD.)
663
=0
[T ((A)
11 (Zer
-X
II
Xar
X ctr
17
X oo. co/
-Lemma
ctr
(A)
1
|
|1
Zear
-2
11
11
Xetr
X
< o(1) (Zerr)
||cxll-
Similarly, the second term in (62) can be bounded by
Vn
cool (A)
Zcer] (UD.
HD.)
1
Z
(A)
|| (Z - x)
vni
1
X|
||
vn
a(1)
(Zetr)
All in all, we obtain
Vn
I col (
(č. - cx) o(1) (Zerr)
12 -XI
(vn |cxll + ||1,|]
(64)
(2) The second term in 61 Letting ((1) = _(a1/²
x()) ctr we observe that
(A)
col
((A)
[ooo (
S max
E(A) x ctr ||
(A)+
E z
(2) ||}
ctr
(A)
Lemma 5
(A)
mmxx { ||
||
1 ||}
1 - XII
min (o()) (a(()(XX(a)))) (Xar), (A1 ettr))
All in all, we obtain the following upper bound:
vn
(Tea) (A)
cool
Ca
<
TT (
|| 5
1z -XI
min (X()) o())
(2
(65)
Step 3: Concluding the proof. We conclude this proof by inserting the upper bounds 64) and from Step 2 into the upper bound (T)) Step Specifically, we obtain
65
(1) D.
(1) (x)
<
2
XI|
(Vi ||z]|| ||c.|+(11...1) min (o(A)(X Xetr),
I -|
Vn ||/|
((A) Zecz)
-
< min {o(>) (X ctr) a()) Z (2vn ||c/l| + [1.ll)..
Lastly, we note that ||ca]| = v (a -_D_)) Σو_ (x (.D.)) = ic--ro,llen -11o-re.llso. and | = Vn.
30

---

D.3
Completing the proof of Theorem 3
Recall that given set D,- {( :ie [n]}, we let Xo. = 10g nnl E Ruxp, In addition, we let
Vy M, d, (v) = [d(
d' v) E ER.
(66)
Recall that we let X = XD, and X D., for shorthand, and further, we let Xetr (In
1 1, X and Zetr (I, 1 1 1, Z denote the 'row-centered' matrices. Here we present and prove the complete version of Theorem 3 Theorem 4 (De-noising covariates). Suppose that Assumptions (0) and C1) CT) if x E HDD + rowsp X ctr and
hold. For any X E R4,
- TD-DD,IlED S //2
Dg
min (on) (X ctr); o(2) (Zar)) ( -1).
(67)
2
2 diam (M)
1 - -X
then
d (o!)) (x), (x)).(//..0) (2))
(x),
1 -X min (Xar)), ( (Zetr)
2 x PDD Ee.
+1
| (5.) +
+1I2B. (vPa)
C
vn
(68)
Proof of Theorem 4 First of all, we recall from (51) that ((A) (
WD (x)
= D. (x1,z)
(2) DD, (Zn x)]
and
[1))()))) (x) = [ D,,
(21, (21, x))
w 1 [2n x2)
D.
In addition, recall that we let for any y E M.
L. (y1.))
d w)1 12n.
Thereafter, we observe that for any y E M and any x E (UD, + rowsp Xar).
(1) 12 (21 xx)
())z) Dr
(2) Dn
(zi, x) -wp, (2) (1)12)) ot (vi.w)
(I) w (x), JB. (v)
1
(1)
(x) d.t (z) (1ll
D,
1A3
N X|
|BB (w)
min (a()) (ttr): x(ا) [etrr] (2 || uD uo.llEo. +1)
vn
(69)
where (a) is due to Cauchy-Schwarz inequality, and (b) follows from Lemma 6
Using shorthand notation R., R!A) R. (() D." Fn =<DD(()) (x), and G. =// (2) that
(x), observe
Rr (G.) - Rn(()n)
Rn() Rn(( + Rm(()) Rn((p))
(0) Ra Rn(G)) - [(n(Gn) + Rn Rn((p)) (pm)= Ru Rn((p))
(8)
1 X
(2 || - Hemllzon +1) 1|0b, (8.) + +1IdB, ((
min (a())) (xrr), (Z)r
Vn
(70)
31

---

where (a) follows from the optimality of Gn. i... Rn((n) 2 Rr (F.).. and (b) is due to (69). Finally, note that if
1 Cg.D2 DE
min (oin (X o(2) (Zar)) -1)
- - Il 2 2 diam (M)
1 - -x
then the upper bound in (70) certifies that Rn (Pn) (x Cg. Dg. Thus, we can use Assumption (CT) convert the risk bound (70) to derive a distance bound between the minimizers:
Rn Rm((x)) RP((Pn)
d pn) s
Cg
which completes the proof.
E Further details on the experiments
Experimental setup in Section 5 We consider combinations of (150,300 600) and n (100, 200, 400). The datasets D, {(X,, Y,) i [n]] and D, {(Z,, Y)) i [n]] generated as follows.
(True covariate X) Let X; N (0p. Σ) be IID multivariate Gaussian with mean Op and covariance Σ such that spec (2) = {K2 > 0 E [p]} is an exponentially decreasing sequence such that
(r))) Σj=1 Rig p. To specific, for each p. we consider an exponentially decreasing sequence
= a1 ... ap = 10 10-3, and then set = E ag') for each j E [p]- Note that
1p/31 Kj/ 1 Ki Ro 0.9, and thus, E effectively low-rank.
IID
(Noisy covariate Z) For the error-prone covariate Z = X + e, we consider two scenarios Sj NO0, oz) and Ej IID Laplace (0,)acee00,0,)) (0, Note that in this setting, we have the signal-to-nois ratio [[]]X())/E(((lll 1/0 We set of 0.052.
~
(Response Y) Given X = x. let Y be the distribution function of N (Ha,s())) + n) =²), where
(a,(()) + B with 11 and 3 p1122. 1pp n (0,
IG(81, 82), an inverse gamma distribution with shape 81 and scale 82.
2 We note that E(r²) s1-1 and Var(r?) (s1 ()1 2)- In particular, when r² 0, this setting corresponds to the classical linear regression model for scalar responses. We set o, = 0.5², and (81, 82) = -18,, 17). In this setting, we have
E (X)) and Var (Ha, ET E(?) = 1 and Var(>?) = 0.25
(Tuning parameter A) For simplicity, we chose universal threshold value
Ån = arg min (A))
AEA
where A is fine grid on (0, / Àι p/n). Then the same threshold Ån was used to evaluate (A)
Des (x)). and MSE De (x)) for all b = I. Therefore, we claim that the performance of the SVT estimator reported in Table I has further room for improvement one substitute ( arg min MSPE(p!!) for each Monte Carlo experiment. Although suboptimal results are reported, we note that the proposed SVT outperforms both the oracle estimator and the naive EIV estimator our simulation study. practice, one may employ crossvvalidation for
Bias (pp(A) ( pas (z)), Var(p (A)
we reported minxeA MSPE(y (A)
better performance. For the MSPE in Table
32

---

Evaluation metrics: bias and variance. We evaluate the accuracy and efficiency of the Fréchet regression function estimator using the bias and the variance. For any given x, we define
B
Bias, (p!!1) = dw (x), p (x)) and Var. (v! = Σ
dw (1 (2), (x))
where E (Dn, D and (x) = arg miny E dw (P (100((2) (2), y) is the Fréchet mean of P (z),.... Puin (x). Note that these definitions are a generalization of the standard bias and variance of the regression function estimator in Euclidean spaces. We evaluate the global performance of the estimator by considering fixed set of evaluation points, GMs fim compute
1, .,... M ). and
Bias (+!A))
Bias am (+l and Var(p!!
1 M
Var (
M
In our experiment, generate the set GM by drawing , RM IID from the same distribution as X, with M 500.
Additional experiment with linear regression models We also conducted an additional experiment for the standard linear regression models with three different metric metrics.
(Model) The linear regression model for (X, Y) E R2 x Rd is defined as Y a + XB + v) and the covariate is contaminated as X + e. We generate X using the effective low-rank model with geometrically decaying spectrum and condition number 10 (see Appendix E in the original submission).
(Parameters) Here, we let = 1a + 0.1 g. 3= d- - 1pxd + 0.1 G. 0.5. Ia) and NN(00..5* 0.5² 1p), with g. G being standard Gaussians.
(a) L1
(b)
(c) Loo
Figure 3: Normalized mean squared prediction (NMSPE) versus threshold À for vector-valued linear regression with three different metrics for V: (a) fmmmeric:; (b) fz-metric; and (c) -me-tri... [SVT]: the
(A) regularized Fréchet regression estimator 4 with best NMSPE; [EIV]: the unregularized Fréchet regression
estimator (0)
fb, with emrons-n--vaiiable covariates Z: [REF]: the unregularized Fréchet regression estimator
(0) PDD with error-free covariates X (oracle).
33


Reviews:
[{"review_id": "MotSRMLUuA", "reviewer": "Reviewer_Jnac", "summary": "The submission 5822 makes a step forward in exploring the Fréchet regression, which is a significant approach for non-Euclidean response variables. Compared to existing work, this research has specific focuses on 1) high-dimensional, 2) errors-in-variables settings, and designed  a novel framework that combines the Fréchet regression and the principal component regression. The analysis in paper 1) proved the consistency of the proposed estimator, 2) investigated the convergence rate, and 3) derived an upper bound for the distance between the the estimates from error-free and errors-in-variables covariates. Some numerical simulations on synthetic datasets are provided.\n\n", "questions": "\n- in experiments, the Gaussian noise added to covariate may be too ideal. Although this assumption is common, could you provide results of other forms of noise (for example, some work on adversarial perturbations)?\n\n- how would proposed theorems extend to concrete models/objectives? providing a practical guide is better for helping readers understand the contribution of this paper\n\n- line 826: ''$\\Lambda$ is a fine grid on ...''  adding more details about how to define the searching grid and the final chosen value of $\\lambda$ are appreciated\n\n- some lines have small typos, e.g., in line 523, it should be Proof of Proposition **4**\n\nI'm not an expert of non-Euclidean regression analysis and would be happy to revise the rating if I missed some key aspects\n\n\n", "limitations": "- authors did not discuss in detail the limitations of the proposed regression framework (while they said \"Yes\" in checklist), but given that this work is based on theoretical analysis, I believe that all Assumptions in main body could clarify the technical limitations\n\n- there's no need to discuss potential negative societal impact", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "\n- this work is well-written and provides results that will I believe be of interest to the community. The idea to design such a regression scheme is novel to me. In particular, the framework and main theorems described in Sec. 3 and Sec. 4 are convincing and well presented. The literature (from what I know) is globally well discussed.\n\n- claims and mathematical derivations seem to be sound and correct, and it clearly state its contributions, notation and results.\n\n- also the R implementation is offered in supplement, which allows for reproducing the key results using the README file\n\n", "weaknesses": "\n- I understand this paper is a theory-based work. However, I still think the numerical simulations are too weak and might not be convincing. \n  - the scale of the experiment is too small. The authors are suggested to consider larger $p$ and $n$, as well as more concrete examples/models as mentioned in Sec. 4.1\n  - the synthesized data ($p<n$) in implementation is *not* high-dimensional, which is the main setting of this work\n  - lack of experiments on real-world benchmarks, e.g., UCI, libsvm datasets\n\n- maybe the motivations of this research should be reconsidered. \n  - why the authors want to combine the *Fréchet regression* and *errors-in-variables setting*, along with the specific *PCR*? The authors only mentioned some advances in PCR research in line 96 and expressed their inspiration from this. Infact, there is whole lot of literature on high dimensional statistical learning and robust modeling. Why not borrow relevant ideas? \n  - compared to the existing work of Fréchet  regression, what are the core challenges (difficulties) of this work, and did the authors draw on some existing frameworks / proof techniques from them?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We value the insightful comments and feedback provided. Our response is organized to address the highlighted weaknesses first, followed by comprehensive point-by-point responses to the specific questions.\n\n### **Weaknesses**\n#### 1\\. Numerical simulations:\n*(A) Scale of experiments, high-dimensional settings, and more concrete examples:*\nWe expanded simulations, incorporating results in the Author Rebuttal report; we added nine configurations in Table R.1, encompassing various dimensions ($n\\in${100, 200, 400} and $p\\in${150, 300, 600}) for easy extrapolation to larger-scale experiments.  \n  \nAdditionally, we extended investigations, analyzing the proposed method's prediction performance in standard regression analysis with Euclidean responses (Example 1 in Sect. 4.1) and other metrics ($\\ell_1$ and $\\ell_{\\infty}$) in $R^d$. Consistent MSPE trends were observed, akin to Figure 2 in the original submission; see Figure R.1 in the Author Rebuttal report.\n\nLastly, our focus on the Wasserstein space (Example 2 in Sect. 4.1) as the primary simulation instance in the original submission aligns with growing interest in machine learning for random objects, enhancing insight into errors-in-variables challenges within related work.\n\n*(B) Absence of real-world benchmarks:*\nAs the reviewer mentioned, this paper primarily focuses on establishing theoretical guarantees for employing singular value thresholding (SVT) in errors-in-variables Fr\\'echet regression analysis for metric-space-valued responses, with a comprehensive analysis. While real-world benchmark experiments are absent, our mathematically rigorous analysis and numerical experiments on synthesized data consistently demonstrate SVT's superior finite-sample performance over the naive EIV estimator, even without leveraging prior knowledge of measurement error distributions.\n\nWe acknowledge the reviewer's point and concur that investigating SVT's performance on real-world datasets offers exciting prospects for future research. Additionally, the inherent flexibility of our two-step EIV Fr\\'echet regression framework (allowing covariate cleansing through means other than SVT) suggests potential for practical improvements, making these avenues enticing for further exploration.\n\n#### 2\\. Reconsidering the motivations:\n*(A) Why PCR?*\nIndeed, there exists a diverse body of literature in high-dimensional learning and robust regression modeling, as acknowledged by the reviewer and elaborated upon in the Author Rebuttal. However, much of this literature assumes response spaces to be vector spaces or endowed with inner product structures. Additionally, prior statistical analyses of EIV problems often assume known or estimable noise distributions.\n\nIn this paper, we propose a two-step approach involving covariate cleansing followed by followed by Fréchet regression. Specifically, we opt for PCR/SVT for the covariate cleansing step, among other methods, to address these challenges in scenarios when distributional knowledge is absent. We appreciate the reviewer's suggestion of exploring other ideas in the literature and agree that this could offer promising avenues for future research.\n\n*(B) Core challenges:*\nOne of the core challenges in our work was analyzing the asymptotic behavior of the $M$-estimator in the global Fr\\'echet regression model combined with SVT. Unlike settings with an algebraic structure on the response space, the global Fr\\'echet regression model in generic metric spaces lacks an explicit form and is defined only pointwisely. Thus, controlling the risk difference involves scrutinizing perturbations of weight functions, influenced by the threshold $\\lambda$ and covariate errors. Consequently, obtaining the bias-variance decomposition in Theorem 2 is not straightforward from existing works; it necessitated intricate technical development, as exemplified by Lemmas 2 and 3. The derivation of Theorem 3 also departs from conventional PCR analyses in [2] due to similar reasons. While drawing on some proof techniques in [41] and [2], we employed more sophisticated arguments to address these unique intricacies and challenges within our framework.\n\n### **Questions**\n#### 1\\. Non-Gaussian noise:\nPlease refer to the Author Rebuttal and its attachment for extra experimental outcomes using Laplacian noise. It's important to note that our proposed method isn't explicitly designed for active robustness, yet it can withstand (sparse) adversarial noise if $|Z-X|$ is significantly smaller than the signal singular value, as indicated by Eq. (16).\n\n#### 2\\. Illustration of Theorems with concrete models:\nIn our paper, we used a basic linear regression model with scalar responses to offer an accessible illustration for readers unfamiliar with metric-spaced-valued responses. For instance, consider the linear regression $Y = \\alpha + \\beta X + \\eta$, where $X$ is confined to a compact interval in $\\mathbb{R}$ (page 4). Then, Theorem 2 demonstrates that bias asymptotically relies on eigenvalues below threshold $\\lambda$, while variance follows a $\\sqrt{n}$ rate. This extends standard principal component regression to metric-spaced-valued regression. Remark 1 on page 7 discusses the application of Theorem 2 in Examples 1, 2, and 3, and Theorem 3's implications are highlighted in the Author Rebuttal report.\n\n#### 3\\. Details for searching grid:\nWe used an 100 evenly spaced search grid in the interval $(0, \\lambda_1 \\sqrt{p/n})$, where $\\lambda_1$ is the maximum eigenvalue of the covariance matrix. MSPE was calculated over this grid, and the value minimizing MSPE was selected. For instance, we obtained $\\lambda=0.3584$ for n=100, p=50 in Figure 2.\n\nDue to page limits, full technical details, simulation settings, implementation, metrics, and result discussions are deferred to Appendix E. However, we recognize the value of summarizing these details in the Experiments section for reader clarity. Upon revision, we'll ensure a clear pointer to technical details."}, {"review_id": "tIVwzAai0e", "reviewer": "Reviewer_ZuE1", "summary": "The proposed method leverages the low-rank structure inherent in the covariate matrix to improve efficiency and accuracy. It combines global Fréchet regression with principal component regression to enable more effective modeling and estimation, especially in high-dimensional and errors-in-variables regression settings. The paper provides a theoretical analysis of the proposed estimator's properties in large samples, including bias, variance, and variations due to measurement errors. Empirical experiments support the theoretical findings, demonstrating the superior performance of the approach.", "questions": "What is the difference between MSE and MSPE, does MSPE stand for Mean Square Prediction Error? The abbreviations in the experiments section have not been explained properly like REV, EIV, SVT, etc. Line 147 can be rewritten as \" .. conditional distribution of Y given X = x is normally distributed .. \"", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths of this paper include:\n\nThe paper introduces a new framework, called regularized (global) Fréchet regression, which combines Fréchet regression and principal component regression. This framework effectively utilizes the low-rank structure in the covariate matrix by extracting principal components through low-rank matrix approximation.\n\nThe paper provides a thorough theoretical analysis with three main theorems. Firstly, it proves the consistency of the proposed estimator for the true global Fréchet regression model. Secondly, it investigates the convergence rate of the estimator's bias and variance. Lastly, it derives an upper bound for the distance between estimates obtained with error-free covariates and those with errors-in-variables covariates. These results establish the effectiveness of the proposed framework in addressing model mis-specification and achieving more efficient model estimation.\n\nNumerical experiments conducted on simulated datasets validate the theoretical findings. The results demonstrate that the proposed method provides more accurate estimates of regression parameters, particularly in high-dimensional settings. The experiments highlight the importance of incorporating the low-rank structure of covariates in Fréchet regression and provide empirical evidence that aligns with the theoretical analysis.", "weaknesses": "The authors should have performed experiments in diverse settings using multiple distance metrics. It would have been interesting to see the relation of MSPE against threshold in different settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable feedback on our work. With gratitude for the positive evaluation, we are dedicated to clarifying and enhancing our contributions by addressing the raised concerns and questions. To facilitate this process, our response is organized to first tackle highlighted weaknesses, followed by a detailed response to the question.\n\n### **Weaknesses**\n#### 1\\. Experiments in more diverse settings:\nWe value the reviewer's suggestion and have extended our numerical investigation to encompass a wider range of scenarios. This includes diverse problem parameters and the introduction of non-Gaussian noise (Laplacian) conditions for the Wasserstein space example. Furthermore, we have examined the standard linear regression model for Euclidean responses, along with variations involving different metrics ($\\ell_1$ and $\\ell_{\\infty}$) within the response space $\\mathcal{Y}=R^d$. We present a summary of these findings in Table R.1 and Figure R.1, included in the attachment to our Author Rebuttal.\n\nSpecifically, Figure R.1 demonstrates consistent trends in mean squared prediction error (MSPE) across all metrics, resembling those observed in Figure 2 of our original submission. Notably, our proposed method (represented in blue) consistently outperforms the naive EIV approach (depicted in red) across all three metrics. It is important to highlight that while the $\\ell_2$ metric enables an explicit linear regression model form, such an explicit form is not available for $\\ell_1$ or $\\ell_{\\infty}$ metrics alike Fr\\'echet regression for the Wasserstein space.\n\n### **Questions**\n#### 1\\. Acronyms / paraphrasing of Line 147:\nWe acknowledge the reviewer's accurate observation regarding the MSPE acronym. The formal definition of Mean Squared Prediction Error can be found in Appendix E (Line 822). Due to page constraints, we deferred the definitions of evaluation metrics and acronyms, including Mean Squared Error (MSE) and Mean Squared Prediction Error (MSPE), to Appendix E of the original manuscript. Other technical details, such as simulation settings, implementation specifics, and evaluation metrics, are also provided in Appendix E.\n\nWe appreciate the reviewer's suggestion and recognize the value of providing concise explanations of these terms within the Experiments section to enhance readers' comprehension. During the revision process for the camera-ready version upon acceptance, we will incorporate these explanations either in the caption of Table 1 or within the main text. Moreover, we will revise line 147, as per the suggestion, for improved clarity."}, {"review_id": "HGJdWO0ctK", "reviewer": "Reviewer_S37y", "summary": "This paper proposes a new method in Frechet regression of non-Euclidean response variables, with a particular focus on high-dimensional, errors-in-variables regression. The idea is to combine the original Frechet regression with Principle Component Regression (PCR). In this way, the low-rank structure in the matrix of (Euclidean) covariates is utilized by extracting its principal components via low-rank matrix approximation. Theoretical analysis of consistency, convergence etc of the proposed method have been provided. Numerical experiments on simulated datasets have also been presented.", "questions": "N/A", "limitations": "no limitations provided", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The proposed method tackles limitations of Frechet estimation such as reliance on ideal scenarios with abundant and noiseless\ncovariate data;\n\n(2) The method utilizes the low-rank structure so that it can be applied to high-dimensional settings;\n\n(3) both theoretical analysis and empirical results are demonstrated.", "weaknesses": "(1) underlines some terms and references which are rarely seen in papers;\n\n(2) line 98: after \"Frechet regression.\", there is an additional phrase \"in this study.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your feedback and positive evaluation. In response, we have prepared both a comprehensive Author Rebuttal report and individual rebuttal to address highlighted weaknesses and questions. We believe these efforts will strengthen our contributions while effectively addressing the concerns raised.\n\n### **Regarding your specific points**\n#### 1\\. Atypical underlines:\nThank you for bringing this to our attention. We will ensure the removal of the underlines in the revision.\n\n#### 2\\. Typo in Line 98:\nWe are grateful for pointing out the typo. During the preparation of the camera-ready revision upon acceptance, we will meticulously review the manuscript and make necessary corrections, including those highlighted by the reviewer."}, {"review_id": "cQAk44ClSa", "reviewer": "Reviewer_BqGr", "summary": "This paper discusses a new method for Fréchet regression that relates non-Euclidean response variable to multivariate predictors. The proposed method leverages the low-rank structure of the covariate matrix to improve the efficiency and accuracy of the estimator, particularly in high-dimensional and errors-in-variables regression settings. The authors provide a theoretical analysis of the estimator's properties and demonstrate its superior performance through numerical experiments.", "questions": "1. In the numerical experiments, does the method still work for the high-dimensional case when $p>n$?\n\n2. How to choose the regularization parameter $\\lambda$ in practice?\n", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors propose the regularized global Fréchet regression framework that can effectively utilize the low-rank structure inherent in the covariate matrix.\n\n2. The proposed method is straightforward to implement and works well in the errors-in variables regression setting.\n\n3. The authors have provided both theoretical analysis and numerical experiments to demonstrate the effectiveness of the proposed method.\n\n\n\n", "weaknesses": "1. The paper looks interesting, but it does not appear to be significantly advancing the state of the art in the field, e.g. [38, 41]\n\n2. In Theorem 2, it is not clear how $\\lambda$ affects the convergence rate in general.\n\n3. No discussions regarding the performance of the three algorithms are given after Table 1 in Section 5.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback and comments on our work. With gratitude for the positive evaluation, we are committed to further elucidating our contributions by addressing the concerns and questions you have raised. To facilitate this process, our response is structured to first address highlighted weaknesses, followed by comprehensive point-by-point responses to your specific questions.\n\n### **Weaknesses**\n#### 1\\. Lack of significant advances:\nWe acknowledge the pioneering nature of [38] and [41] in the realm of Fr\\'echet regression, as highlighted in our Introduction section. Yet, it is crucial to underscore that our paper's primary focus is on addressing Errors-in-Variables (EIV) challenges within the Fr\\'echet regression framework.\n\nWhile there have been commendable strides in statistical and machine learning literature, including notable works such as [10], [38], [41], [46], and [54], that delve into regression analysis for response variables within general metric spaces, we are not aware of any prior investigations into addressing EIV problems in the non-Euclidean regression setting. Our work bridges this gap adeptly, addressing the EIV challenge in the non-Euclidean regression context, relevant to real-world scenarios with latent measurement errors impacting observed covariates.\n\nOur paper also offers substantial theoretical advancements. An inherent challenge we tackled involves analyzing the asymptotic behavior of the $M$-estimator in the global Fr'echet regression model when coupled with the singular value thresholding (SVT) technique. This task is made considerably intricate due to the absence of an explicit form for the global Fr\\'echet regression function in generic metric spaces, where it can be only identified pointwisely. Thus, our endeavor necessitates a detailed examination of risk differences involving weight function perturbations, influenced by the threshold $\\lambda$ and covariate errors. As such, deriving the bias-variance decomposition in Theorem 2 is not straightforward; intricate technical development, exemplified by Lemmas 2 and 3, was essential. Additionally, the derivation of Theorem 3 diverges from conventional PCR analyses in [2] for similar reasons, highlighting our unique approach. While rooted in some proof techniques from [41] and [2], our work employs more sophisticated arguments to address intricate nuances and challenges.\n\nConsidering these factors, we firmly assert that our work significantly advances this field.\n\n#### 2\\. Effects of $\\lambda$ in Theorem 2:\nAs $\\lambda$ increases, the bias (reflected on $b_{\\lambda}$) grows while the variance diminishes. Our analysis captures the bias trend, although the variance aspect may be slightly unclear due to the asymptotic nature of our analysis. Refining our analysis into the finite-sample properties in future research could provide further insights.\n\n#### 3\\. Missing discussions on numerical experiments:\nA thorough analysis of the three algorithms' performance is available in Appendix E. We've incorporated technical details like simulation settings, implementation specifics, evaluation metrics, and further result discussions in Appendix E due to page limitations. We acknowledge the reviewer's recommendation to succinctly describe these aspects in the Experiments section for clarity. In the camera-ready version post-acceptance, we will provide clear references to facilitate easy access to the technical details.\n\n### **Questions**\n#### 1\\. Numerical experiments for high-dimensional cases:\nWe've performed extra simulations to reaffirm the consistency of our numerical results with those in the original manuscript. Specifically, we have extended our investigation to include nine additional combinations, with $n\\in${100, 200, 400} and $p\\in${150, 300, 600}, as presented in Table R.1 attached to the Author Rebuttal. These encompass high-dimensional scenarios, aiding readers in extrapolating numerical trends to even larger-scale experiments. For comprehensive information, please refer to our responses in the Author Rebuttal report.\n\n#### 2\\. Choice of $\\lambda$ in practice:\nIn our numerical analysis, we utilized an evenly spaced search grid in the range $(0, \\lambda_1 \\sqrt{p/n})$, taking into account the maximum eigenvalue of the covariance matrix ($\\lambda_1$), model complexity ($p$), and sample size ($n$). After evaluating mean squared prediction errors (MSPE) across this grid, we selected the parameter value that minimized MSPE.\n\nIn practical contexts, cross-validation could be employed to estimate MSPE, and it is viable to use the sample covariance matrix of noisy covariates to approximate $\\lambda_1$. Importantly, it is noteworthy that certain settings within the penalized regression literature provide theoretically-guided principles for selecting $\\lambda$."}, {"review_id": "CoHF7TiClJ", "reviewer": "Reviewer_urUC", "summary": "The paper attempts to extend regression models by combining error-in-variables in high-dimensional predictors and responses in a metric space. The main method is principal component regression whereby principal components are formed in the predictor space and serve as dimension-reduced predictors. It ties into existing approaches for global object regression and is a valid attempt at a very challenging problem but unfortunately none of the problems is solved: The predictors are not high-dimensional as their dimension is fixed; the errors-in-variables are not mitigated: While a bound is provided where the errors appear, the error-in-variables model is not shown to allow for consistent estimation and why principal components would be a reasonable approach at error mitigation is not elucidated; tuning parameter choice is not investigated.\n", "questions": "Is the bound in (16) sharp? \n\nHigh-dimensional predictors are mentioned but the dimension p is fixed throughout. Can you handle the case of large p where p would increase with sample size ? \n", "limitations": "See the previous comments. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper addresses two difficult topics, errors-in-variables and Frechet regression. Errors-in-variables is a major problem in regression settings and is an important issue for many applications. While much research has been done on errors-in-variables in standard regression settings, it remains a challenging issue and especially so when the responses are in metric spaces. Such new types of responses are also clearly relevant for modern data. An approach through principal component regression would be of interest if it leads to consistent estimation and is implemented in a data-adaptive way. So the topic of this paper is timely from several perspectives.\n\n", "weaknesses": "(1) The background is not well balanced. It emphasizes Frechet regression while it is very superficial about error-in-variables. Since principal component regression is used to address error-in-variables it is then of interest to report how PCR mitigates/addresses or is expected to mitigate error-in-variables even in standard regression setting.\n\n(2) The proposed method does not lead to convergence for errors-in-variables according to (16), but this is what is needed for a method to succeed in this setting. \n\n(3) There is no data-adaptive choice (along with theory justifying it) for the tuning parameter $\\lambda$, however this parameter may be a crucial feature for principal component regression implementations as it determines the number of components to be included as predictors. It is not clear how it should be chosen to mitigate errors-in-variables or when responses are in metric space. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We value the insightful feedback and comments received. In response, we have prepared a comprehensive Author Rebuttal report and individual rebuttals to address the highlighted concerns and questions. Our response is structured to first address the highlighted weaknesses and then provide detailed point-by-point responses to the specific questions.\n\n### **Weaknesses**\n#### 1\\. Background of study:\nIn this paper, we tackle EIV problems within the Fr'echet regression framework. While previous works have explored regression analysis in non-Euclidean metric spaces, addressing EIV issues in this context remains uncharted. This study aims to fill this gap, responding to real-world situations where latent measurement errors impact observed covariates.\n\nWe adopt PCR as a concrete, practical solution to EIV models in non-Euclidean regression, driven by two compelling considerations. Firstly, the prevalence of (approximate) low-rank structures in real-world datasets enhances the practically relevance of our approach. Secondly, we intentionally opt for an approach with minimal assumptions regarding covariate errors to ensure broad applicability. Notably, PCR aligns well with these considerations, due to its inherent utilization of effective low-rank structures and its demonstrated capacity to alleviate covariate errors in the standard regression setting [2]. This has prompted our anticipation that PCR can similarly address EIV challenges in non-Euclidean settings. \n\nWhile there is an extensive EIV literature, conventional techniques often rely on knowledge of error distributions or require prior information to eliminate measurement errors. In response to the reviewer's insightful comment, we focus on \"mitigating\" (possibly non-stochastic) covariate perturbations by retaining only essential assumptions necessary for Fr'echet regression analysis, instead of attempting to completely \"denoise\" EIV effects that may depend on impractical information. Importantly, PCR's advantage lies in its error-mitigation capabilities without necessitating prior knowledge of error distributions. Our method demonstrates effectiveness with theoretical guarantees and practical applicability.\n\nWe value the reviewer's input and acknowledge that these nuances may not have been fully conveyed in our initial presentation. In particular, we recognize the need to elaborate on how/why PCR is expected to mitigate/address EIV challenges. To address this, we plan to enrich the \"Errors-in-variables-regression\" and \"Principal component regression\" paragraphs in the Related Work, providing a more comprehensive overview of literature and our motivations in the upcoming camera-ready version after acceptance.\n\n#### 2\\. Convergence/mitigation of EIV implied by Eq.(16): \nHere we reiterate our response in the Author Rebuttal report. Despite its generality, this upper bound highlights the effective error mitigation in specific scenarios. Consider the following:\n\n1. **Well-balanced, effectively low-rank covariates:**\nSuppose that $X\\in R^{n\\times p}$ satisfies (1) $|X_{ij}|=\\Omega(1)$ for all i,j, and (2) $\\sigma_1(X)\\asymp\\sigma_r(X)\\gg\\sigma_{r+1}(X)\\asymp\\sigma_{n\\wedge p} = O(1)$, where $r\\ll n\\wedge p$ denotes the effective rank of $X$. Then we have $\\sigma_1(X)^2\\asymp\\sigma_r(X)^2\\asymp\\|X\\|_F^2/r\\gtrsim np/r$. \n\n2. **Independent sub-gaussian noise:**\nNext, suppose that $Z=X+E$ where $E$ is a random matrix with independent sub-Gaussian rows. Then $\\|Z-X\\|\\lesssim\\sqrt{n}+\\sqrt{p}$ with high probability due to concentration inequality.\n\nIn the random design scenario where $X$ and $x$ have i.i.d. rows drawn from the same distribution, $\\|x-\\mu_{D_n}\\|_{\\Sigma}\\approx 1$ with high probability. As a result, the upper bound in Eq.(16) is bounded by $\\sqrt{r/p}+\\sqrt{r/n}$, which diminishes to 0 when $r\\ll n\\wedge p$. \n\n#### 3\\. Principled choice of $\\lambda$:\nIn our numerical study, we utilized a uniform search grid for $\\lambda$ selection, opting for the value that minimizes the MSPE. Similarly, in practical scenarios, cross-validation can be employed to identify the optimal $\\lambda$.\n\nRegarding the concern about lacking a data-adaptive principle (and theoretical justification), it's worth noting that the effective estimation of $X$ from $Z$ drives the error-mitigating capability, irrespective of $\\mathcal{Y}$'s metric, under Assumptions (C0)-(C2). The low-rank estimation of $X$ via SVT (Eq. 6) is closely connected to nuclear-norm-regularized low-rank matrix estimation (hard SVT vs. soft SVT). We conjecture based on these observations that the well-studied principles for selecting $\\lambda$ in low-rank matrix estimation could be extended with minor adjustments.\n\n### **Questions**\n#### 1\\. Sharpness of the bound in Eq.(16):\nThe error bound in Eq.(16) is sharp (up to a multiplicative constant) due to the existence of a worst-case noise instance ($E=Z-X$) that achieves equality. This sharpness is inherited from the tightness of the subspace perturbation bound (Davis-Kahan/Wedin). While it might be feasible to achieve a more potent error-mitigating bound by adopting a stricter model assumption, we have refrained from doing so to maintain generality. Thus, Eq. (16) remains sharp within our framework.\n\n#### 2\\. Handling high-dimensional case ($p>n$): \nOur approach, involving low-rank matrix estimation via SVT, effectively addresses the high-dimensional case where $p$ grows with $n$, under the condition that the (effective) rank of $X$ remains appreciably smaller than $n\\wedge p$, ensuring control over $|Z-X|/\\sigma_r(X)$ as seen in Eq. (16).\n\nFurthermore, we have supplemented our response with additional numerical simulation results in the Author Rebuttal report. These results underscore the superior performance of our proposed method compared to the naive Fr\\'echet in EIV settings. Our method consistently achieves better predictive accuracy across diverse scenarios, including the cases with $p > n$."}]

Author rebuttal:
We appreciate the valuable feedback from the reviewers and their constructive comments. Here, we address common themes raised by the reviewers, providing clarity on our methodological and theoretical contributions. We also present supplementary numerical results to reinforce our findings. Detailed responses to each reviewer's comments can be found in dedicated individual rebuttals.

### **Summary of contributions**
We address the errors-in-variables (EIV) regression problem in a non-Euclidean response context through a composite approach, combining Fr\'echet regression (for non-Euclidean responses) and principal component regression (for covariate error mitigation).

Unlike conventional EIV literature that relies on distributional knowledge for covariate error ($\varepsilon_i$ in Eq.(5)), we avoid such assumptions, distinguishing our approach. Rather than aiming for complete elimination of covariate noise -- a challenging task without stringent distributional assumptions -- we focus on "mitigating" errors. This is achieved by estimating the design matrix through low-rank matrix estimation, particularly via principal component regression (PCR), which approximates the design matrix and alleviates errors. A similar approach was explored in standard linear regression setting [2].

The paper underscores efficient use of the design matrix's low-rank structure (covariates), even in arbitrary non-Euclidean metric spaces for the response variable.  Our "regularized" Fr\'echet regression estimator's superiority is supported by three theorems (consistency, convergence rate, and error reduction in variables) and also corroborated by numerical experiments.

### **More background & Rationale for PCR**
A vast literature addresses EIV models. In this work, we adopt PCR for a concrete, practical solution to EIV models in non-Euclidean regression, driven by two compelling reasons. Firstly, the prevalence of (approximate) low-rank structures in real-world datasets renders our approach practically relevant. Secondly, we purposefully choose an approach with minimal assumptions concerning covariate errors for broad applicability.

PCR aligns with these considerations, leveraging inherent low-rank structures effectively and demonstrating error-mitigating capabilities [2]. Notably, PCR stands apart from conventional EIV techniques by not requiring a priori knowledge of measurement error distributions. Moreover, PCR has an extensive presence in the high-dimensional statistics and dimensionality reduction literature.

We appreciate Reviewer 1 (Jnac) and Reviewer 5 (urUC) for their invaluable insights. Acknowledging that these nuances may not have been fully elucidated in our initial presentation, we plan to augment the "Errors-in-variables-regression" paragraph within the Related Work section in the upcoming camera-ready version after acceptance. This revision will offer a more comprehensive overview of high-dimensional and robust modeling, clarifying our motivations and methodology.

### **Implications of Theorem 3**
We refrain from imposing distributional assumptions on $\varepsilon=Z-X$, ensuring the broad applicability of inequality Eq.(16) across diverse scenarios. Also, this bound in Eq.(16) is sharp, exemplified by a worst-case noise instance that attains equality (up to a multiplicative constant).

Despite its generality, this upper bound highlights the effective error mitigation in specific scenarios. Consider the following:
1. *Well-balanced, effectively low-rank covariates:*
Suppose that $X\in R^{n\times p}$ satisfies (1) $|X_{ij}|=\Omega(1)$ for all i,j, and (2) $\sigma_1(X)\asymp\sigma_r(X)\gg\sigma_{r+1}(X)\asymp\sigma_{n\wedge p} = O(1)$, where $r\ll n\wedge p$ denotes the effective rank of $X$. Then we have $\sigma_1(X)^2\asymp\sigma_r(X)^2\asymp\|X\|_F^2/r\gtrsim np/r$. 

2. *Independent sub-gaussian covariate noise:*
Next, suppose that $Z=X+E$ where $E$ is a random matrix with independent sub-Gaussian rows. Then $\|Z-X\|\lesssim\sqrt{n}+\sqrt{p}$ with high probability due to concentration inequality.

In the random design scenario where $X$ and $x$ have i.i.d. rows drawn from the same distribution, $\|x-\mu_{D_n}\|_{\Sigma}\approx 1$ with high probability. As a result, the upper bound in Eq.(16) is bounded by $\sqrt{r/p}+\sqrt{r/n}$, which diminishes to 0 when $r\ll n\wedge p$. 

### **Additional numerical experiments**
Our extended numerical investigation encompasses non-Gaussian (Laplacian) covariate noise  and includes larger-scale experiments, including $p>n$ cases, enabling straightforward extrapolation to larger scenarios. The summarized results in the attachment (Table \ref{tab:sim-wasserstein-high-dim}) affirm the superiority of our proposed method (SVT), evident from consistently lower mean squared prediction errors (MSPE). This added experiment  further underscores SVT's ability to effectively address EIV challenges, enhancing prediction accuracy for EIV Fr\'echet regression, even without prior knowledge of measurement errors.

Significantly, SVT even surpasses error-free-covariate Fr\'echet regression (REF) in MSPE within this experiment. Standard Fr\'echet regression generally falters in high-dimensional settings due to issues like non-invertibility of sample covariance matrix (when $n<p$), or high covariate correlations. Our background study (not detailed here) identified the ill-posed nature of the M-estimation for REF, leading to instability and escalated mean squared errors (MSE). Hence, Table R.1 presents REF's finite-sample performance achieved through pseudo-inverse of the sample covariance matrix. Nevertheless, SVT proves a more dependable predictive model than even this stabilized version of REF.

Furthermore, we conducted experiments in linear regression settings ($\cal{Y}=R^d$) using three metrics: $\ell_1$, $\ell_2$, and $\ell_{\infty}$. Similar trends in MSPE, observed in Figure 2 of the original submission, persist across all these metrics; see Figure R.1 attached.

Meta-review: All reviewers voted for acceptance. They (more or less) all argue that this work contains concepts and results that will be of interest to the community. Even the more critical reviewers acknowledge that during the rebuttal and discussion phase several critical remarks could be addressed in a meaningful way. I fully share this positive over-all impression of the paper, and for me the methodological strengths clearly outweigh possible weaknesses regarding the motivation and some design choices in the experiments. So after going again over the paper, all reviews an, the rebuttal and the discussions, I am convinced that a final version of this paper that takes into account the results of the discussions would make a nice contribution. Therefore, I recommend acceptance of this paper.
Decision: Accept (poster)