Title: Errors-in-variables Fr\'echet Regression with Low-rank Covariate Approximation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Frechet regression

principal component regression

non-Euclidean

low-rank matrix

errors-in-variables analysis
Abstract: Fr\'echet regression has emerged as a promising approach for regression analysis involving non-Euclidean response variables. However, its practical applicability has been hindered by its reliance on ideal scenarios with abundant and noiseless covariate data. In this paper, we present a novel estimation method that tackles these limitations by leveraging the low-rank structure inherent in the covariate matrix. Our proposed framework combines the concepts of global Fr\'echet regression and principal component regression, aiming to improve the efficiency and accuracy of the regression estimator. By incorporating the low-rank structure, our method enables more effective modeling and estimation, particularly in high-dimensional and errors-in-variables regression settings. We provide a theoretical analysis of the proposed estimator's large-sample properties, including a comprehensive rate analysis of bias, variance, and additional variations due to measurement errors. Furthermore, our numerical experiments provide empirical evidence that supports the theoretical findings, demonstrating the superior performance of our approach. Overall, this work introduces a promising framework for regression analysis of non-Euclidean variables, effectively addressing the challenges associated with limited and noisy covariate data, with potential applications in diverse fields.

Errors-in-variables Fréchet Regression with Low-rank Covariate Approximation

Abstract

Fréchet regression has emerged as a promising approach for regression analysis involving non-Euclidean response variables. However, its practical applicability has been hindered by its reliance on ideal scenarios with abundant and noiseless covariate data. In this paper, we present a novel estimation method that tackles these limitations by leveraging the low-rank structure inherent in the covariate matrix. Our proposed framework combines the concepts of global Fréchet regression and principal component regression, aiming to improve the efficiency and accuracy of the regression estimator. By incorporating the low-rank structure, our method enables more effective modeling and estimation, particularly in high-dimensional and errors-in-variables regression settings. We provide a theoretical analysis of the proposed estimator's large-sample properties, including a comprehensive rate analysis of bias, variance, and additional variations due to measurement errors. Furthermore, our numerical experiments provide empirical evidence that supports the theoretical findings, demonstrating the superior performance of our approach. Overall, this work introduces a promising framework for regression analysis of non-Euclidean variables, effectively addressing the challenges associated with limited and noisy covariate data, with potential applications in diverse fields.

1 Introduction

Regression analysis is a fundamental statistical methodology to model the relationship between response variables and explanatory variables (covariates). Linear regression, for example, models the (conditional) expected value of the response variable as a linear function of covariates. Regression models enable researchers and analysts to make predictions, gain insights into how input variables influence the outcomes of interest, and validate hypothetical associations between variables in inferential studies. As a result, regression is widely utilized across various scientific domains, including economics, psychology, biology, and engineering [53, 21, 29].

In recent decades, there has been a growing interest in developing statistical methods capable of handling random objects in non-Euclidean spaces. Examples of these include functional data analysis [41], statistical manifold learning [31], statistical network analysis [34], and object-oriented data analysis [39]. In such contexts, the response variable is defined in a metric space that may lack an algebraic structure, making it challenging to apply global, parametric approaches toward regression as in the classical Euclidean setting. To overcome this challenge, (global) Fréchet regression, which models the relationship by fitting the (conditional) barycenters of the responses as a function of covariates, has been introduced [40]. Notably, when the Euclidean metric is considered, Fréchet

regression recovers classical Euclidean regression models. For more details on Fréchet regression and its recent developments, we refer readers to [30, 40, 22, 45, 27].

Nevertheless, most existing research on Fréchet regression has focused on ideal scenarios characterized by abundant covariate data that are accurately measured and free of noise. In practical applications, however, high-dimensional data often arise, which are also susceptible to measurement errors and other forms of contamination. These errors can stem from various sources, such as unreliable data collection methods ( e.g. , low-resolution probes, subjective self-reports) or imperfect data storage and transmission. The high-dimensionality and the presence of measurement errors in covariates pose critical challenges for statistical inference, as regression analysis based on error-prone covariates may result in incorrect associations between variables, yielding misleading conclusions.

To address these limitations, it is crucial to extend the methodology and analysis of Fréchet regression to tackle high-dimensional errors-in-variables problems. In this work, we aim to leverage the lowrank structure in the covariates to enhance the estimation accuracy and computational efficiency of Fréchet regression. Specifically, we explore the extension of principal component regression to handle errors-in-variables regression problems with non-Euclidean response variables.

1.1 Contributions

This paper contributes to advancing the (global) Fréchet regression of non-Euclidean response variables, with a particular focus on high-dimensional, errors-in-variables regression.

Firstly, we propose a novel framework, called the regularized (global) Fréchet regression (Section 3) that combines the ideas from Fréchet regression [40] and the principal component regression [32]. This framework effectively utilizes the low-rank structure in the matrix of (Euclidean) covariates by extracting its principal components via low-rank matrix approximation. Our proposed method is straightforward to implement, not requiring any knowledge about the error-generating mechanism.

Furthermore, we provide a comprehensive theoretical analysis (Section 4) in three main theorems to establish the effectiveness of the proposed framework. Firstly, we prove the consistency of the proposed estimator for the true global Fréchet regression model (Theorem 1). Secondly, we investigate the convergence rate of the estimator's bias and variance (Theorem 2). Lastly, we derive an upper bound for the distance between the estimates obtained using error-free covariates and those with errors-in-variables covariates (Theorem 3). Collectively, these results demonstrate that our approach effectively addresses model mis-specification and achieves more efficient model estimation by leveraging the low-rank structure of covariates, despite the presence of inherent bias due to unobserved measurement errors.

To validate our theoretical findings, we conduct numerical experiments on synthetic datasets (Section 5). We observe that the proposed method provides more accurate estimates of the regression parameters, especially in high-dimensional settings. Our experimental results emphasize the importance of incorporating the low-rank structure of covariates in Fréchet regression, and provide empirical evidence that aligns with our theoretical analysis.

1.2 Related work

Metric-space-valued variables. Nonparametric regression models for Riemannian-manifold-valued responses were proposed as a generalization of regression for multivariate outputs by Steinke et al. [48, 49]. These works provided a foundation for recent developments in regression analysis of non-Euclidean responses. Later, Hein [30] proposed a Nadaraya-Watson-type kernel estimation of regression model for general metric-space-valued outcomes. Since then, statistical properties of regression models for some special classes of metric-space-valued outcomes, such as distribution functions [23, 52, 28] and matrix-valued responses [57, 20], have been investigated. Recently, many researchers have introduced further advances in Fréchet regression, including [40, 10, 37, 45]. In this study, we use the global Fréchet regression proposed in [40] as the basis for our proposed method.

Errors-in-variables regression. Much of earlier work on errors-in-variables (EIV) problems in the statistical literature can be found in [13], which covers the simulation-extrapolation (SIMEX) [16, 11], the attenuation correction method [36], covariate-adjusted model [46, 19], and the deconvolution kernel method [25, 24, 18]. The regression calibration method [47], instrumental variable modeling [12, 43], and the two-phase study design [9, 4] were also proposed when additional data are available

for correcting measurement errors. In the high-dimensional modeling literature, regularization methods for recovering the true covariate structure can also be utilized [38, 7, 17]. Despite a diverse body of literature on high-dimensional learning and robust regression modeling, much of it assumes response spaces to be vector spaces endowed with inner products. In this paper, we tackle EIV problems within the Fréchet regression framework. While previous works have explored regression analysis in non-Euclidean metric spaces, addressing EIV issues in this context remains uncharted.

Principal component regression. The principal component regression (PCR) [32] is a statistical technique that regresses response variables on principal component scores of the covariate matrix. The conventional PCR selects a few principal components as the 'new' regressors associated with the first leading eigenvalues to explain the highest proportion of variations observed in the original covariate matrix. In functional data analysis, PCR is known to have a shrinkage effect on the model estimate and produce robust prediction performance in functional regression [42, 33]. Recently, Agarwal et al. [2] investigated the robustness of PCR in the presence of measurement errors on covariates and the statistical guarantees for learning a good predictive model. Unlike prior statistical analyses of EIV problems that often assume known or estimable noise distributions, PCR leverages inherent low-rank structures in the covariates without requiring a priori knowledge of measurement error distributions. We adopt PCR as a concrete, practical solution to EIV models in non-Euclidean regression, driven by two compelling considerations. Firstly, the prevalence of (approximate) low-rank structures in real-world datasets enhances the practical relevance of our approach. Secondly, we intentionally opt for an approach with minimal assumptions regarding covariate errors to ensure broad applicability.

1.3 Organization

In Section 2, we introduce the notation used throughout the paper, and overview the global Fréchet regression framework. Section 3 presents the problem setup, objectives, and our proposed estimator, which we refer to as the regularized Fréchet regression (Definition 4). In Section 4, we discuss theoretical guarantees on the regularized Fréchet regression method in accurately estimating the global Fréchet regression function. Section 5 presents the results of numerical 'proof-of-concept' experiments that support the theoretical findings. Finally, we conclude this paper with discussions in Section 6. Due to space constraints, detailed proofs of the theorems as well as additional details and discussions of experiments are provided in the Appendix.

2 Preliminaries

2.1 Notation

Let N denote the set of positive integers and R denote the set of real numbers. Also, let R + := { x ∈ R : x ≥ 0 } . For n ∈ N , we let [ n ] := { 1 , . . . , n } . We mostly use plain letters to denote scalars, vectors, and random variables, but we also use boldface uppercase letters for matrices, and curly letters to denote sets when useful. Note that we may identify a vector with its column matrix representation. For a matrix X , we let X -1 denote its inverse (if exists) and X † denote the MoorePenrose pseudoinverse of X . Also, we let rowsp( X ) and colsp ( X ) denote the row and column spaces of X , respectively. Furthermore, we let spec ( X ) denote the set of non-zero singular values of X , σ i ( X ) denote the i -th largest singular value of X , and σ ( λ ) ( X ) := inf { σ i ( X ) > λ : i ∈ N } with the convention inf ∅ = ∞ . We let 1 n = (1 , 1 , . . . , 1) ⊤ ∈ R d and let 1 denote the indicator function. We let ∥ · ∥ denote a norm, and set ∥ · ∥ = ∥ · ∥ 2 (the ℓ 2 -norm for vectors, and the spectral norm for matrices) by default unless stated otherwise. For a finite set D , we may identify D with its empirical measure ν D = 1 |D| ∑ x ∈D δ x , where δ x denotes the Dirac measure supported on { x } .

Letting f, g : R → R , we write f ( x ) = O ( g ( x )) as x →∞ if there exist M > 0 and x 0 > 0 such that | f ( x ) | ≤ M · g ( x ) for all x ≥ x 0 . Likewise, we write f ( x ) = Ω( g ( x )) if g ( x ) = O ( f ( x )) . Furthermore, we write f ( x ) = o ( g ( x )) as x →∞ if lim x →∞ f ( x ) g ( x ) = 0 . For a sequence of random variables X n , and a sequence a n , we write X n = O p ( a n ) as n →∞ if for any ε > 0 , there exists M ∈ R + and N ∈ N such that P (∣ ∣ X n a n ∣ ∣ > M ) < ε for all n ≥ N . Similarly, we write X n = o p ( a n ) if lim n →∞ P (∣ ∣ X n a n ∣ ∣ > ε ) = 0 for all ε > 0 .

2.2 Global Fréchet regression

Let ( X,Y ) be a random variable that has a joint distribution P X,Y supported on R p ×M , where R p is the p -dimensional Euclidean space and M = ( M , d ) is a metric space equipped with a distance function d : M×M→ R . We write the marginal distribution of X as P X , and the conditional distribution of Y given X as P Y | X .

Definition 1 (Fréchet regression function) . Let ( X,Y ) be a random element that takes value in R p ×M . The Fréchet regression function of Y on X is a function φ ∗ : R p →M such that

We note that φ ∗ ( x ) is the best predictor of Y given X = x , as it minimizes the marginal risk E [ d 2 ( Y, φ ∗ ( X )) ] under the squared-distance loss. In the literature, φ ∗ ( x ) is also known as the conditional Fréchet mean of Y given X = x [26]. It is important to recognize that the existence and uniqueness of the Fréchet regression function are closely tied to the geometric characteristics of M , and are not guaranteed in general [3, 8]. Nonetheless, extensive research has been conducted on the existence and uniqueness of Fréchet means in various metric spaces commonly encountered in practical applications. Examples include the unit circle in R 2 [14], Riemannian manifolds [1, 5], Alexandrov spaces with non-positive curvature [51], metric spaces with upper bounded curvature [58], and Wasserstein space [59, 35].

While modeling and estimating the Fréchet regression function φ ∗ is often of interest, its global (parametric) modeling may not be straightforward, especially when M lacks a useful algebraic structure, such as an inner product. For instance, in classical linear regression analysis with M = R , the conditional distribution of Y given X = x is normally distributed with a mean of φ ∗ ( x ) = α + β ⊤ x and a fixed variance σ 2 , where α and β represent the regression coefficients. Similarly, when M possesses a linear-algebraic structure, one can specify a class of regression functions that quantifies the association between the expected outcome and covariates in an additive and multiplicative manner. However, the lack of an algebraic structure in general metric spaces may prevent us from characterizing the barycenter φ ∗ ( x ) in the same way classical regression analysis determines the expected value of outcomes with changing covariates.

To address this challenge, Petersen and Müller [40] recently proposed to exploit algebraic structures in the space of covariates, R p , instead of M . Specifically, they consider a weighted Fréchet mean as

where w : R p × R p → R is a weight function such that w ( ξ, x ) denotes the influence of ξ at x . In particular, Petersen and Müller [40] defined the global Fréchet regression function with a specific choice of w as follows.

Definition 2 (Global Fréchet regression function) . Let ( X,Y ) be a random variable in R p ×M . Let µ = E ( X ) and Σ = Var( X ) . The global Fréchet regression function of Y on X is a function φ glo : R p →M such that

When M is an inner product space ( e.g. , M = R ), the function φ glo restores the standard linear regression model representation over the domain R p . For this reason, φ glo is commonly referred to as the global Fréchet regression model for metric-space-valued outcomes [40, 37, 54].

Remark on Definition 2 One might wonder why the term 'global' is used to describe φ glo as a Fréchet regression function. The use of the adjective 'global' serves to emphasize its distinction from 'local' nonparametric regression methods that interpolate data points. Notably, when M is a Hilbert space, φ glo reduces to the natural linear models. For instance, if M = R , then it follows that φ glo ( x ) = E [ w glo ( X,x ) · Y ] = α + β ⊤ ( x -µ ) , where α = E [ Y ] and β = Σ -1 · E [ ( X -µ ) · Y ] . These linear models hold uniformly for the evaluation point x . Similarly, in the case of an L 2 space equipped with the squared-distance metric d 2 ( y, y ′ ) = ∥ y -y ′ ∥ 2 2 induced by the L 2 norm, φ glo represents the linear regression model for functional responses. Thus, φ glo establishes a globally defined model that spans the entire space.

3 Problem and methodology

3.1 Problem formulation

Let ( X,Y ) be a random variable in R p × M and P X,Y be their joint distribution. Let D n = { ( X i , Y i ) : i ∈ [ n ] } be an independent and identically distributed (IID) sample drawn from P X,Y . Note that we may identify the set D n with its discrete measure (empirical distribution), cf. Section 2.1. We consider the problem of estimating the global Fréchet regression function φ glo (see Definition 2) from data D n . In this setting, a natural estimator of φ glo would be its sample-analogue estimator.

With ̂ µ D n = E ( X,Y ) ∼D n ( X ) = 1 n ∑ n i =1 X i and ̂ Σ D n = Var ( X,Y ) ∼D n ( X ) = 1 n ∑ n i =1 ( X i -̂ µ D n ) · ( X i -̂ µ D n ) ⊤ , the sample-analogue estimator ̂ φ D n is defined as

where ̂ w D n ( X,x ) = 1 + ( X -̂ µ D n ) ⊤ ̂ Σ -1 D n ( x -̂ µ D n ) . The statistical properties of ̂ φ D n , including the asymptotic distribution, a ridge-type variable selection operation, and total variation regularization method have been investigated [40, 37, 54].

In practice, however, we may only be able to access D n = { ( Z i , Y i ) : i ∈ [ n ] } instead of D n , where

denotes an error-prone observation of the covariates X by measurement error ε . This formulation corresponds to the classical errors-in-variables problem.

Objective. Given a dataset, either D n or ˜ D n , our aim is to produce an estimate ̂ φ of the global Fréchet regression function φ glo so that the prediction error is minimized. Specifically, we evaluate the performance of ̂ φ by means of the distance in the response space, d ( ̂ φ ( x ) , φ glo ( x ) ) .

3.2 Fréchet regression with covariate principal components

Singular value thresholding. Among various low-rank matrix approximation methods, we consider the (hard) singular value thresholding (SVT). For any λ ∈ R + , we define the map SVT ( λ ) : R n × p → R n × p that removes all singular values that are less than the threshold λ . To be precise, SVT ( λ ) can be expressed in terms of the singular value decomposition (SVD) as follows:

Regularized Fréchet regression. We introduce a variant of the sample-analog estimator of the global Fréchet regression function based on principal components of the sample covariance. To facilitate the description of our proposed estimator, we introduce additional notation here.

Definition 3 (Covariate mean/covariance) . For a probability distribution ν on R p ×M , the covariate mean and covariate covariance with respect to ν are respectively defined as

Recall that a finite set D ⊂ R p ×M may be identified with its empirical distribution; it follows that

Definition 4 (Regularized Fréchet regression) . Let ν be a probability distribution on R p ×M and λ ∈ R + . The λ -regularized Fréchet regression function for ν is a map φ ( λ ) ν : R p →M such that

When D n = { ( X i , Y i ) ∈ R p ×M : i ∈ [ n ] } is an IID sample from P X,Y , the λ -regularized estimator φ ( λ ) D n subsumes the sample-analogue estimator ̂ φ D n in (4) as a special case where λ = 0 .

Connection to principal component regression. Here we remark that when M is a Euclidean space, the regularized Fréchet regression function φ ( λ ) ν effectively reduces to the principal component regression. Suppose that M = R and D n = { ( x i , y i ) ∈ R p × R : i ∈ [ n ] } is a given dataset. Then φ ( λ ) D n ( x ) = y + ˆ β ⊤ λ ( x -µ D n ) where y = 1 n ∑ n i =1 y i and ˆ β λ = [ SVT ( λ ) ( Σ D n ) ] † · [ 1 n ∑ n i =1 ( x i -µ D n ) · ( y i -y ) ] . Observe that ˆ β λ is exactly the regression coefficient of principal component regression applied to the centered dataset D ctr n = { ( x i -µ D n , y i -y ) : i ∈ [ n ] } using k principal components with k = max a ∈ [ p ] { σ a (Σ D ctr n ) ≥ λ } .

4 Main results

In this section, we investigate properties of φ ( λ ) ν for λ ≥ 0 , with a focus on two cases: ν = D n and ν = ˜ D n , cf. Section 3.1. By denoting the true distribution that generates ( X,Y ) as ν ∗ , we can express φ glo as φ (0) ν ∗ . To analyze the discrepancy between the estimator φ ( λ ) ν ( x ) and φ glo ( x ) , we examine the relationships depicted in the schematic in Figure 1. Our theoretical findings can be summarized as follows: Even in the presence of covariate noises, φ ( λ ) ˜ D n with a suitable λ > 0 can effectively eliminate the noise in Z to estimate X , thereby reducing the error in estimating φ glo .

4.1 Model assumptions and examples

We impose the following assumptions for our analysis.

(C0) ( Existence ) For any probability distribution ν and any λ ∈ R + , the object φ ( λ ) ν ( x ) exists (almost surely) and is unique. In particular, inf y ∈M : d ( y,φ glo ( x )) >ε R ( y ; x ) > R ( φ glo ( x ); x ) for all ε > 0 , where R ( y ; x ) := R (0) ν ∗ ( y ; x ) .

(C1) ( Growth ) There exist D g > 0 , C g > 0 and α > 1 , possibly depending on x , such that for any probability distribution ν and any λ ∈ R + ,

(C2) ( Bounded entropy ) There exists C e > 0 , possibly depending on y , such that

where B d ( y, δ ) := { y ′ ∈ M : d ( y, y ′ ) ≤ δ } and N ( S, ε ) is the ε -covering number 2 of S .

.

Assumption (C0) is common to establish the consistency of an M-estimator [55, Chapter 3.2]; in particular, it ensures the weak convergence of the empirical process R ( λ ) D n to the population process R ( λ ) ν ∗ implying convergence of their minimizers. Furthermore, the conditions on the curvature (C1) and the covering number (C2) control the behavior of the objectives near the minimum in order to obtain rates of convergence; it is worth mentioning that (C2) corresponds to a (locally) bounded entropy for every y ∈ M , while (P1) in [40] requires the same condition only with y = φ glo ( x ) These conditions arise from empirical process theory and are also commonly adopted [40, 44, 45].

Here we provide several examples of the space M , in which the conditions (C0), (C1) and (C2) are satisfied. We verify the conditions in Appendix A; see Propositions 1, 2, 3, and 4.

Example 1. Let M = ( H , d HS ) be a finite-dimensional Hilbert space H equipped with the HilbertSchmidt metric d HS ( y 1 , y 2 ) = ⟨ y 1 -y 2 , y 1 -y 2 ⟩ 1 / 2 , e.g., M = ( R r , d 2 ) where d 2 is the ℓ 2 -metric.

Example 2. Let M be W , the set of probability distributions G on R such that ∫ R x 2 dG ( x ) < ∞ , equipped with the Wasserstein metric d W defined as

where G -1 1 and G -1 2 are the quantile functions of G 1 and G 2 , respectively. See [40, Section 6].

Example 3. Let M = { M ∈ R r × r : M = M T , M ⪰ 0 and M ii = 1 , ∀ i ∈ [ r ] } be the set of correlation matrices of size r , equipped with the Frobenius metric, d F ( M,M ′ ) = ∥ M -M ′ ∥ F .

Example 4. Let M be a (bounded) Riemannian manifold of dimension r , and let d g be the geodesic distance induced by the Riemannian metric.

4.2 Theorem statements

4.2.1 Noiseless covariate setting

We first verify the consistency of the λ -regularized Fréchet regression function as follows.

Theorem 1 (Consistency) . Suppose that Assumption (C0) holds. If diam( M ) < ∞ , then for any λ ∈ R such that 0 ≤ λ < min { σ i (Σ ν ∗ ) : σ i (Σ ν ∗ ) > 0 } , and any x ∈ R p ,

If λ < σ (0) (Σ ν ∗ ) = min { σ i (Σ ν ∗ ) : σ i (Σ ν ∗ ) > 0 } , then the regularized estimator φ ( λ ) D n ( x ) effectively reduces to the same as the sample-analog estimator ̂ φ D n ( x ) in (4) in the limit n →∞ . Thus, φ ( λ ) D n ( x ) inherits the consistency of ̂ φ D n . We provide a detailed proof of Theorem 1 in Appendix B.

In addition to the consistency of φ ( λ ) D n in the small λ limit, we present an analysis for the convergence rate of φ ( λ ) D n ( x ) in the following theorem.

Definition 5. The Mahalanobis seminorm of x induced by a positive semidefinite matrix Σ is ∥ x ∥ Σ := ( x ⊤ Σ † x ) 1 / 2 .

Theorem 2 (Rate of convergence) . Suppose that Assumptions (C0)-(C2) hold. If diam( M ) < ∞ , then for any λ ∈ R + and x ∈ R p such that ∥ x -µ ν ∗ ∥ Σ ν ∗ ≤ C g · D α g diam( M ) 2 · √ rank Σ ν ∗ ,

ν ν

We obtain Theorem 2 by showing a 'bias' upper bound d ( φ ( λ ) ν ∗ ( x ) , φ (0) ν ∗ ( x ) ) = O ( b λ ( x ) 1 α -1 ) and a 'variance' bound d ( φ ( λ ) D n ( x ) , φ ( λ ) ν ∗ ( x ) ) = O P ( n -1 2( α -1) ) ; see Lemmas 1 and 2 in Appendix C. Here we remark that b λ ( x ) is a monotone non-decreasing function of λ , and if λ < σ (0) (Σ ν ∗ ) then b λ ( x ) = 0 . Also, the condition on ∥ x -µ ν ∗ ∥ Σ ν ∗ is introduced for a technical reason, and can be removed when D g = ∞ . Note that Condition (C1) holds with D g = ∞ and α = 2 for Examples 1, 2 and 3. Thus, we have d ( φ ( λ ) D n ( x ) , φ (0) ν ∗ ( x ) ) = O P ( b λ ( x ) + n -1 2 ) as n →∞ .

4.2.2 Error-prone covariate setting

Given a set D n = { ( x i , y i ) : i ∈ [ n ] } , let X D n := [ x 1 · · · x n ] ⊤ ∈ R n × p . We let X = X D n and Z = X ˜ D n for shorthand, and further, we let X ctr = ( I n -1 n 1 n 1 ⊤ n ) X and Z ctr = ( I n -1 n 1 n 1 ⊤ n ) Z denote the 'row-centered' matrices.

Theorem 3 (De-noising covariates) . Suppose that Assumptions (C0) and (C1) hold. Then there exists a constant C > 0 such that for any λ ∈ R + , if x ∈ µ D n +rowsp X ctr and

then

Note that the condition on ∥ x -µ ν ∗ ∥ Σ ν ∗ in (14) can be removed when D g = ∞ . We highlight that the quantity ∥ Z -X ∥ σ ( λ ) ( X ctr ) ∧ σ ( λ ) ( Z ctr ) acts as the reciprocal of the signal-to-noise ratio. Here, ∥ Z -X ∥ quantifies the magnitude of the 'noise' in the covariates, while min { σ ( λ ) ( X ctr ) , σ ( λ ) ( Z ctr ) } measures the strength of the 'signal' retained in the λ -SVT of the design matrix. We observe that the error bound (15) increases proportionally to the normalized deviation of x from the mean, µ D n , which is a reasonable outcome. For the complete version of Theorem 3 and its proof, refer to Appendix D.

Remarks on Theorem 3. We avoid imposing distributional assumptions on the noise ε = Z -X , to ensure broad applicability of the result. Also, the inequality (15) is sharp, as there is a worstcase noise instance that attains equality (up to a multiplicative constant). Despite its generality, this upper bound highlights effective error mitigation in specific scenarios. For instance, consider well-balanced, effectively low-rank covariates X ∈ R n × p such that | X ij | = Ω(1) for all i, j and σ 1 ( X ) ≍ σ r ( X ) ≫ σ r +1 ( X ) ≍ σ n ∧ p ( X ) = O (1) , where r ≪ n ∧ p is the effective rank of X . Then σ 1 ( X ) 2 ≍ σ r ( X ) 2 ≍ ∥ X ∥ 2 F /r ≳ np/r . Additionally, if Z = X + E where E is a random matrix with independent sub-Gaussian rows, then ∥ Z -X ∥ ≲ √ n + √ p with high probability. In the random design scenario where the rows of X and the test point x are drawn IID from the same distribution, ∥ x -µ D n ∥ Σ ≈ 1 with high probability. Consequently, the upper bound in (15) is bounded by √ r/p + √ r/n , which diminishes to 0 when r ≪ n ∧ p .

4.3 Proof sketches

Proof of Theorem 1. We show that R ( λ ) D n ( y ; x ) weakly converges to R (0) ν ∗ ( y ; x ) in the ℓ ∞ ( M ) -sense. According to [55, Theorem 1.5.4], it suffices to show that (1) R ( λ ) D n ( y ; x ) -R (0) ν ∗ ( y ; x ) = o p (1) for all y ∈ M , and (2) R ( λ ) D n is asymptotically equicontinuous in probability.

Proof of Theorem 2. We prove upper bounds for the bias and the variance separately.

To control the bias (Appendix C, Lemma 1), we show an upper bound for R ( φ ( λ ) ( x ); x ) -R ( φ ( x ); x ) , and convert it to restrain the distance between the minimizers d ( φ ( λ ) ( x ) , φ ( x ) ) using the Growth condition (C1). In this conversion, we employ the 'peeling technique' in empirical process theory.

To control the variance (Appendix C, Lemma 2), we follow a similar strategy as in Lemma 1, but with additional technical considerations. Defining the 'fluctuation variable' Z ( λ ) n ( y ; x ) := R ( λ ) D n ( y ; x ) -R ( λ ) ν ∗ ( y ; x ) parameterized by y ∈ M , we derive a probabilistic upper bound for R ( λ ) ν ∗ ( φ ( λ ) D n ( x ); x ) -R ( λ ) ν ∗ ( φ ( λ ) ν ∗ ( x ); x ) by establishing a uniform upper bound for Z ( λ ) n ( y ; x ) -Z ( λ ) n ( φ ( x ); x ) ; here, the Entropy condition (C2) is used. Again, we use the Growth condition (C1) and the peeling technique to obtain a probabilistic upper bound for the distance d ( φ ( λ ) D n ( x ) , φ ( λ ) ν ∗ ( x )) .

Proof of Theorem 3. Expressing the difference in the weights w ( λ ) ˜ D n ( y ; x ) -w ( λ ) D n ( y ; x ) in terms of X and Z , we utilize classical matrix perturbation theory to control R ( λ ) ˜ D n ( y ; x ) -R ( λ ) D n ( y ; x ) , and transform it to an upper bound on the distance d ( φ ( λ ) ˜ D n ( x ) , φ ( λ ) D n ( x )) using the Growth condition (C1).

5 Experiments

In this section, we present numerical simulation results to validate and support our theoretical findings. We focus on global Fréchet regression analysis for one-dimensional distribution functions (Example 2). These simulations cover various conditions, allowing us to evaluate and compare our methodology's performance with alternative approaches. For a summary of the experimental results, please refer to Figure 2 and Table 1. Further details about simulation settings and additional results are provided in Appendix E.

Experimental setup. We consider combinations of p ∈ { 150 , 300 , 600 } and n ∈ { 100 , 200 , 400 } . The datasets D n = { ( X i , Y i ) : i ∈ [ n ] } and ˜ D n = { ( Z i , Y i ) : i ∈ [ n ] } are generated as follows. Let X i ∼ N p ( 0 p , Σ ) be IID multivariate Gaussian with mean 0 p and covariance Σ such that spec (Σ) = { κ j > 0 : j ∈ [ p ] } is an exponentially decreasing sequence such that tr (Σ) = ∑ p j =1 κ j = p and κ 1 /κ p = 10 3 . Note that ∑ ⌊ p/ 3 ⌋ j =1 κ j /∑ p j ′ =1 κ j ′ ≈ 0 . 9 , and thus, Σ is effectively low-rank. We generate Z i following (5) under two scenarios ε ij IID ∼ N ( 0 , σ 2 ε ) and ε ij IID ∼ Laplace ( 0 , σ ε ) , respectively. Lastly, given X = x , let Y be the distribution function of N ( µ α,β ( x )+ η, τ 2 ) , where (i) µ α,β ( x ) = α + β ⊤ x with α = 1 and β = p -1 / 2 · 1 p ; (ii) η ∼ N ( 0 , σ 2 η ) ; and (iii) τ 2 ∼ IG ( s 1 , s 2 ) , an inverse gamma distribution with shape s 1 and scale s 2 . We performed B = 500 Monte Carlo experiments by drawing D ( b ) n and ˜ D ( b ) n as independent copies of D n and ˜ D n , respectively, for b ∈ [ B ] .

Performance evaluation. We assess the in-sample and out-of-sample performance of the Fréchet regression function estimator by using the mean squared error (MSE) and the mean squared prediction error (MSPE). To this end, we create a 'test set' D new N = { ( X new i , Y new i ) : i ∈ [ N ] } , with N = 1000 . The MSE and the MSPE are computed as the average of squared metric-distance residuals from the observed responses in the 'training set' D n and in the 'test set' D new N ), respectively:

We report the MSE averaged over B = 500 random trials: MSE( φ ( λ ) ν ) = B -1 ∑ B b =1 MSE( φ ( λ ) ν ( b ) ) , and likewise for MSPE. Furthermore, we evaluate the accuracy and efficiency of the estimator using bias and variance, with detailed definitions deferred to Appendix E.

Simulation results. Our numerical study demonstrates that the proposed SVT method consistently improves both estimation and prediction performance, especially in the errors-in-variables setting. Figure 2 highlights how the SVT estimator outperforms the naïve errors-in-variables (EIV) estimator, which corresponds to SVT with λ = 0 . The naïve EIV suffers from an intrinsic model bias, called the attenuation effect [13], as it regresses responses on error-prone covariates. This leads to a misrepresentation of the association between responses and true covariates, potentially leading to statistical inference based on a mis-specified model.

口

REF

60

口

EIV

SVT

0.7

MSPE

0.6

5.

0.4

0

0.25

0.52

0.78

1.04

1.31

1.57

Threshold

MSPE

SquaredBias

0.25

0.20

1Bias/Variance

7

MSPE

E

0.15

SquaredE

0.10

5.

0.05

0.00

0.0

0.5

1.0

1.5

Threshold

Remarkably, the SVT estimator achieved a smaller MSPE even compared to the oracle estimator (REF) obtained from the error-free sample. Although the REF estimator had the smallest MSE due to its small bias, we observed its overfitting to the training sample, resulting in poor prediction performance. Notably, even the naïve EIV estimator outperformed the REF estimator in MSPE. We believe this is mainly because the true covariate matrix was nearly singular in our simulation setup, causing multicollinearity issues for the REF. In contrast, measurement errors introduced non-ignorable minimum singular values in the EIV covariate matrix, unintentionally mitigating multicollinearity for the naïve EIV and causing it to behave like ridge regression.

6 Discussion

This paper has addressed errors-in-variables regression of non-Euclidean response variables through the (global) Fréchet regression framework enhanced by low-rank approximation of covariates. Specifically, we introduce a novel regularized (global) Fréchet regression framework (Section 3), which combines the Fréchet regression with principal component regression. We also provide a comprehensive theoretical analysis in three main theorems (Section 4), and validate our theory through numerical experiments on simulated datasets. Moreover, our numerical experiments demonstrate empirical evidence of the effectiveness and superiority of our approach, reinforcing its practical relevance and potential impact in non-Euclidean regression analysis.

We conclude this paper by proposing several promising directions for future research. First, it would be worthwhile to explore the large sample theory for selecting the optimal threshold parameter λ in the proposed SVT method, in order to characterize the theoretical phase transition of the bias-variance trade-off in the regularized (global) Fréchet regression. Second, we believe that our framework could be extended to errors-in-variables Fréchet regression for response variables in a broader class of metric spaces, e.g., by leveraging the quadruple inequality proposed by Schötz [44, 45]. Lastly, investigating the asymptotic distribution of the proposed SVT estimator would be highly appealing in the statistical literature, as it would enable us to make statistical inferences on the conditional Fréchet mean in non-Euclidean spaces [6, 8] with errors-in-variables covariates.

Acknowledgments and Disclosure of Funding

DS acknowledges support from the National Science Foundation under Grant No. CCF-2212326.

References

Bijan Afsari. Riemannian L p center of mass: Existence, uniqueness, and convexity. Proceedings of the American Mathematical Society , 139(2):655-673, 2011.

Anish Agarwal, Devavrat Shah, Dennis Shen, and Dogyoon Song. On robustness of principal component regression. Journal of the American Statistical Association , 116(536):1731-1745, 2021.

Adil Ahidar-Coutrix, Thibaut Le Gouic, and Quentin Paris. Convergence rates for empirical barycenters in metric spaces: Curvature, convexity and extendable geodesics. Probability Theory and Related Fields , 177(1):323-368, 2020.

Gustavo Amorim, Ran Tao, Sarah Lotspeich, Pamela A Shaw, Thomas Lumley, and Bryan E Shepherd. Two-phase sampling designs for data validation in settings with covariate measurement error and continuous outcome. Journal of the Royal Statistical Society Series A: Statistics in Society , 184(4):1368-1389, 2021.

Marc Arnaudon and Laurent Miclo. Means in complete manifolds: Uniqueness and approximation. ESAIM: Probability and Statistics , 18:185-206, 2014.

Dennis Barden, Huiling Le, and Megan Owen. Central limit theorems for Fréchet means in the space of phylogenetic trees. Electronic Journal of Probability , 18(25):1-25, 2013.

Alexandre Belloni, Mathieu Rosenbaum, and Alexandre B Tsybakov. Linear and conic programming estimators in high dimensional errors-in-variables models. Journal of the Royal Statistical Society. Series B (Statistical Methodology) , pages 939-956, 2017.

Rabi Bhattacharya and Lizhen Lin. Omnibus CLTs for Fréchet means and nonparametric inference on non-Euclidean spaces. Proceedings of the American Mathematical Society , 145(1):413428, 2017.

Norman E Breslow and Thomas Lumley. Semiparametric models and two-phase samples: Applications to Cox regression. IMS Collections , 9:65-77, 2013.

Louis Capitaine, Jérémie Bigot, Rodolphe Thiébaut, and Robin Genuer. Fréchet random forests for metric space valued regression with non Euclidean predictors. arXiv preprint arXiv:1906.01741 , 2019.

Raymond J Carroll, Helmut Küchenhoff, Frederick Lombard, and Leonard A Stefanski. Asymptotics for the SIMEX estimator in nonlinear measurement error models. Journal of the American Statistical Association , 91(433):242-250, 1996.

Raymond J Carroll, David Ruppert, Ciprian M Crainiceanu, Tor D Tosteson, and Margaret R Karagas. Nonlinear and nonparametric regression and instrumental variables. Journal of the American Statistical Association , 99(467):736-750, 2004.

Raymond J Carroll, David Ruppert, Leonard A Stefanski, and Ciprian M Crainiceanu. Measurement Error in Nonlinear Models: A Modern Perspective . Chapman and Hall/CRC, 2006.

Benjamin Charlier. Necessary and sufficient condition for the existence of a Fréchet mean on the circle. ESAIM: Probability and Statistics , 17:635-649, 2013.

Yan Mei Chen, Xiao Shan Chen, and Wen Li. On perturbation bounds for orthogonal projections. Numerical Algorithms , 73(2):433-444, 2016.

John R Cook and Leonard A Stefanski. Simulation-extrapolation estimation in parametric measurement error models. Journal of the American Statistical Association , 89(428):1314-1328, 1994.

Abhirup Datta and Hui Zou. CoCoLasso for high-dimensional error-in-variables regression. The Annals of Statistics , 45(6):2400-2426, 2017.

Aurore Delaigle, Jianqing Fan, and Raymond J Carroll. A design-adaptive local polynomial estimator for the errors-in-variables problem. Journal of the American Statistical Association , 104(485):348-359, 2009.

Aurore Delaigle, Peter Hall, and Wen-xin Zhou. Nonparametric covariate-adjusted regression. The Annals of Statistics , 44(5):2190-2220, 2016.

Shanshan Ding and R Dennis Cook. Matrix variate regressions and envelope models. Journal of the Royal Statistical Society Series B: Statistical Methodology , 80(2):387-408, 2018.

David Donoho. 50 years of data science. Journal of Computational and Graphical Statistics , 26(4):745-766, 2017.

Paromita Dubey and Hans-Georg Müller. Functional models for time-varying random objects. Journal of the Royal Statistical Society Series B: Statistical Methodology , 82(2):275-327, 2020.

Juan José Egozcue, Josep Daunis-I-Estadella, Vera Pawlowsky-Glahn, Karel Hron, and Peter Filzmoser. Simplicial regression. The normal model. Journal of Applied Probability and Statistics , 2012.

Jianqing Fan and Elias Masry. Multivariate regression estimation with errors-in-variables: Asymptotic normality for mixing processes. Journal of Multivariate Analysis , 43(2):237-271, 1992.

Jianqing Fan and Young K Truong. Nonparametric regression with errors in variables. The Annals of Statistics , pages 1900-1925, 1993.

Maurice Fréchet. Les éléments aléatoires de nature quelconque dans un espace distancié. In Annales de l'Institut Henri Poincaré , volume 10, pages 215-310, 1948.

Aritra Ghosal, Wendy Meiring, and Alexander Petersen. Fréchet single index models for object response regression. Electronic Journal of Statistics , 17(1):1074-1112, 2023.

Kyunghee Han, Hans-Georg Müller, and Byeong U Park. Additive functional regression for densities as responses. Journal of the American Statistical Association , 2019.

Xuming He, David Madigan, , Bin Yu, and Jon Wellner. Statistics at a crossroads: Who is for the challenge. In Workshop report: National Science Foundation. https://www. nsf. gov/mps/dms/documents/Statistics_at_a_Crossroads_Workshop_Report_2019. pdf , 2019.

Matthias Hein. Robust nonparametric regression with metric-space valued output. Advances in Neural Information Processing Systems , 22, 2009.

Alan J. Izenman. Modern Multivariate Statistical Techniques: Regression, Classification, and Manifold Learning . Springer, 2008.

Ian T Jolliffe. A note on the use of principal components in regression. Journal of the Royal Statistical Society Series C: Applied Statistics , 31(3):300-303, 1982.

Ioannis Kalogridis and Stefan Van Aelst. Robust functional regression based on principal components. Journal of Multivariate Analysis , 173:393-415, 2019.

Eric D. Kolaczyk. Statistical Analysis of Network Data . Springer, 2009.

Thibaut Le Gouic and Jean-Michel Loubes. Existence and consistency of Wasserstein barycenters. Probability Theory and Related Fields , 168:901-917, 2017.

Hua Liang, Wolfgang Härdle, and Raymond J Carroll. Estimation in a semiparametric partially linear errors-in-variables model. The Annals of Statistics , 27(5):1519-1535, 1999.

Zhenhua Lin and Hans-Georg Müller. Total variation regularized Fréchet regression for metricspace valued data. The Annals of Statistics , 49(6):3510-3533, 2021.

Po-Ling Loh and Martin J Wainwright. High-dimensional regression with noisy and missing data: Provable guarantees with non-convexity. Advances in Neural Information Processing Systems , 24, 2011.

James Stephen Marron and Ian L Dryden. Object Oriented Data Analysis . CRC Press, 2021.

Alexander Petersen and Hans-Georg Müller. Fréchet regression for random objects with Euclidean predictors. The Annals of Statistics , 47(2):691-719, 2019.

James O Ramsay and Bernard W Silverman. Functional Data Analysis . Springer, 2nd edition, 2005.

Philip T Reiss and R Todd Ogden. Functional principal component regression and functional partial least squares. Journal of the American Statistical Association , 102(479):984-996, 2007.

Susanne M Schennach. Instrumental variable estimation of nonlinear errors-in-variables models. Econometrica , 75(1):201-239, 2007.

Christof Schötz. Convergence rates for the generalized Fréchet mean via the quadruple inequality. Electronic Journal of Statistics , 13:4280-4345, 2019.

Christof Schötz. Nonparametric regression in nonstandard spaces. Electronic Journal of Statistics , 16(2):4679-4741, 2022.

Damla ¸ Sentürk and Hans-Georg Müller. Covariate-adjusted regression. Biometrika , 92(1):75-89, 2005.

Donna Spiegelman, Aidan McDermott, and Bernard Rosner. Regression calibration method for correcting measurement-error bias in nutritional epidemiology. The American Journal of Clinical Nutrition , 65(4):1179S-1186S, 1997.

Florian Steinke and Matthias Hein. Non-parametric regression between manifolds. Advances in Neural Information Processing Systems , 21, 2008.

Florian Steinke, Matthias Hein, and Bernhard Schölkopf. Nonparametric regression between general Riemannian manifolds. SIAM Journal on Imaging Sciences , 3(3):527-563, 2010.

Gilbert W Stewart. On the perturbation of pseudo-inverses, projections and linear least squares problems. SIAM Review , 19(4):634-662, 1977.

Karl-Theodor Sturm. Probability measures on metric spaces of nonpositive curvature. Heat Kernels and Analysis on Manifolds, Graphs, and Metric Spaces: Lecture Notes from a Quarter Program on Heat Kernels, Random Walks, and Analysis on Manifolds and Graphs , 338:357, 2003.

Renáta Talská, Alessandra Menafoglio, Jitka Machalová, Karel Hron, and E Fišerová. Compositional regression with functional response. Computational Statistics & Data Analysis , 123:66-85, 2018.

Kristin M Tolle, D Stewart W Tansley, and Anthony JG Hey. The fourth paradigm: Dataintensive scientific discovery [point of view]. Proceedings of the IEEE , 99(8):1334-1337, 2011.

Danielle C Tucker, Yichao Wu, and Hans-Georg Müller. Variable selection for global Fréchet regression. Journal of the American Statistical Association , pages 1-15, 2021.

Aad W Vaart and Jon A Wellner. Weak Convergence and Empirical Processes . Springer, 2nd edition, 2023.

Roman Vershynin. High-dimensional Probability: An Introduction with Applications in Data Science , volume 47. Cambridge University Press, 2018.

Cinzia Viroli. On matrix-variate regression analysis. Journal of Multivariate Analysis , 111:296309, 2012.

Takumi Yokota. Convex functions and barycenter on CAT(1)-spaces of small radii. Journal of the Mathematical Society of Japan , 68(3):1297-1323, 2016.

Y Zemel and V M Panaretos. Fréchet means and procrustes analysis in Wasserstein space. Bernoulli , 25(2):932-976, 2019.

A Verification of the model assumptions

A.1 Additional background

Definition 6 ( ε -net and covering number) . Let ( M , d ) be a metric space. Let S ⊆ T be a subset and let ε > 0 . A subset N ⊆ S is called an ε -net of S if every point in S is within distance ε of some point N , i.e.,

The ε -covering number of S , denoted by N ( S, ε ) , is the smallest possible cardinality of an ε -net of S , i.e.,

where B d ( y, ε ) = { y ′ ∈ M : d ( y, y ′ ) ≤ ε } denotes the closed ε -ball centered at y ∈ M .

Let B r 2 (0 , 1) := { x ∈ R r : ∥ x ∥ 2 ≤ 1 } denote the unit ℓ 2 -norm ball in R r . It is well known 3 that for any ε > 0 ,

A.2 Example 1: Euclidean space

Proposition 1. The space ( H , d HS ) defined in Example 1 satisfies Assumptions (C0), (C1), and (C2).

Proof of Proposition 1. For any probability distribution ν and any λ ∈ R + , let y ( λ ) ν := E ν [ w ( λ ) ν ( X,x ) · Y ] . Then we observe that

As R ( λ ) ν ( y ; x ) is a strictly convex and coercive function, there exists a unique minimizer, φ ( λ ) ν . Thus, Condition (C0) is proved. Furthermore, Condition (C1) is also satisfied with D g = ∞ , C g = 1 , and α = 2 .

Lastly, for any y ∈ H and any ε > 0 ,

where r = dim H and C > 1 is a constant that depends on r only; see the covering number upper bound in (17). Thus, the integral (11) is bounded as follows:

using the change of variable t = -log ε . Therefore, Assumption (C2) holds with C e = √ 1 + log C + √ rπ 2 .

A.3 Example 2: Set of probability distributions

Proposition 2. The space ( W , d W ) defined in Example 2 satisfies Assumptions (C0), (C1), and (C2).

Proof of Proposition 2. For a probability distribution function y ∈ W defined on R , let Q = Q ( W ) := { Q ( y ) : y ∈ W} denote the collection of corresponding quantile functions, where ( Q ( y ) ) ( u ) = y -1 ( u ) for u ∈ [0 , 1] .

We note that f ↦→ E ν [ w ( λ ) ν ( X,x ) ⟨ Q ( Y ) , f ⟩ ] is a bounded linear functional defined on L 2 [0 , 1] because E ν | w ( λ ) ν ( X,x ) | 2 ≤ 2+2 p ∥ ( x -µ ν ) ∥ 2 Σ ν implies that E ν [ w ( λ ) ν ( X,x ) |·∥ Q ( Y ) ∥ 2 ] is bounded. It follows from the Riesz representation theorem that there exists f ( λ ) x ∈ L 2 [0 , 1] such that

for all g ∈ L 2 [0 , 1] . Then, we have

which yields that

The condition (C0) follows from the convexity of ( Q , ∥ · ∥ 2 ) . Moreover, the convexity also gives 〈 Q ( y ) -Q ( φ ( λ ) ν ( x )) , f ( λ ) x ( x ) -Q ( φ ( λ ) ν ( x )) 〉 2 ≤ 0 for all y ∈ W , so that

Therefore, the condition (C1) holds for any arbitrary constant D g > 0 with C g = 1 and α = 2 . Finally, we refer to [40, Proposition 1] to ensure that for any δ > 0 and any ε > 0 ,

holds with an absolute constant C > 0 . Technically, this fact comes from the covering number bound for a class of uniformly bounded and monotone functions in L 2 . This confirms that the entropy condition (C2) holds.

A.4 Example 3: Set of correlation matrices

Proposition 3. The space ( M , d F ) defined in Example 3 satisfies Assumptions (C0), (C1), and (C2).

Proof of Proposition 3. First of all, we note that M is a convex subset of S r := { X ∈ R r × r : X = X ⊤ } , which is the set of r × r symmetric matrices. It is because M = S r + ∩ H where S r + denotes the cone of r × r positive semidefinite matrices and H := { X ∈ S r : X ii = 1 , ∀ i ∈ [ r ] } denotes an affine subspace of S r , both of which are convex.

Next, we observe that S r equipped with the Frobenius metric d F is isometrically isomorphic to R r ( r +1) / 2 equipped with the ℓ 2 -metric. Hence, ( M , d F ) satisfies Assumptions (C0), (C1), and (C2), inheriting these properties from the ambient space S r , which is established by Proposition 1. We note that the inheritance of (C0), (C1) relies on the convexity of M , while (C2) is inherited simply based on the inclusion M⊂S r .

A.5 Example 4: Bounded Riemannian manifold

Proposition 4. The space ( M , d g ) defined in Example 4 satisfies Assumption (C2) provided that the Riemannian metric is equivalent to the ambient Euclidean metric.

Furthermore, let T y M be the tangent space of M at y , and Exp y : T y M→M be the manifold exponential map at y . Let g ( λ ) ν ( u ; y, x ) := R ( λ ) ν ( E y ( u ) , x ) for u ∈ T y M If (C0) holds and the Hessian of g ( λ ) ν ( u ; φ ( λ ) ν ( x ) , x ) is positive definite, then (C1) for some D g > 0 .

Proof of Proposition 3. Since M has finite dimension and is bounded, the bounded entropy condition (C2) follows from the metric equivalence.

Suppose that (C0) holds, and let δ > 0 be the injectivity radius at φ ( λ ) ν ( x ) . Consider y ∈ M such that d ( y, φ ( λ ) ν ( x ) ) < δ , and let u y = Log φ ( λ ) ν ( x ) ( y ) . Then we have

for some ¯ u y between 0 and u y . Since u ⊤ y u y = d ( y, φ ( λ ) ν ( x ) ) 2 and g ( λ ) ν is continuous, the positive definiteness of ∇ 2 g ( λ ) ν (¯ u y ) implies (C1) with α = 1 .

B Proof of Theorem 1

Proof of Theorem 1. Recall from Definition 4, cf. (9), that for any probability distribution ν on R p , any λ ∈ R + , and any x ∈ R p , the λ -regularized Fréchet regression function evaluated at x is given as the minimizer of a function R ( λ ) ν as

where

In this proof, we follow a similar strategy to that in the proof of [40, Theorem 1]. Specifically, it suffices to show sup y ∈M ∣ ∣ R ( λ ) D n ( y ; x ) -R (0) ν ∗ ( y ; x ) ∣ ∣ converges to zero in probability, due to [55, Corollary 3.2.3]. To this end, we show R ( λ ) D n ( y ; x ) weakly converges to R (0) ν ∗ ( y ; x ) in the ℓ ∞ ( M ) -sense, and then apply [55, Theorem 1.3.6]. Again, according to [55, Theorem 1.5.4], this weak convergence can be proved by showing that

(S2) R ( λ ) D n is asymptotically equicontinuous in probability, i.e., for any ε, η > 0 , there exists δ > 0 such that

In what follows, we prove these two statements, (S1) and (S2), thereby completing the proof of Theorem 1.

Step 1: proof of (S1). First of all, we observe that

We separately analyze the two terms T 1 and T 2 below to show T 1 = o p (1) and T 2 = o p (1) as n →∞ .

(i) T 1 = o p (1) .

Let D n = { ( X i , Y i ) : i ∈ [ n ] } , and we re-write

Letting ̂ µ n = µ D n , ̂ Σ n = Σ D n , and ̂ Σ ( λ ) n = SVT ( λ ) ( ̂ Σ n ) for shorthand, we observe that

Let X = [ X 1 · · · X n ] ⊤ ∈ R n × p , and note that ̂ Σ n = 1 n ( X -1 n ̂ µ ⊤ n ) ⊤ ( X -1 n ̂ µ ⊤ n ) . Then it follows that

Consider a singular value decomposition of X -1 n ̂ µ ⊤ n , namely,

and observe that ̂ Σ n = ∑ min { n,p } i =1 s 2 i · v i v ⊤ i is an eigenvalue decomposition of ̂ Σ n . Letting V ( λ ) n := span { v i : i ∈ [ p ] , 0 < s i ≤ √ λ } be a subspace of R p spanned by the eigenvectors of ̂ Σ n corresponding to the nonzero eigenvalues no greater than λ , we have

where Π V ( λ ) n denotes the projection matrix onto the subspace V ( λ ) n . Note that Π V ( λ ) n = 0 if and only if min { i ∈ [ p ] : 0 < s i ≤ √ λ } = ∅ .

Therefore, we have

The last line follows from the fact that sup i ∈ [ p ] ( σ i ( ̂ Σ n ) -σ i (Σ ν ∗ ) ) → 0 in probability, and thus, Π V ( λ ) n → 0 in probability.

(ii) T 2 = o p (1) .

Letting ˜ R n ( y ; x ) = 1 ∑ n w (0) ν ∗ ( X i , x ) · d 2 ( Y i , y ) , we decompose T 2

Note that T 2 B converges to 0 in probability by the weak law of large numbers.

Now it remains to show T 2 A = o p (1) . To this end, we note that

Since ̂ µ n and ̂ Σ n respectively converge to µ and Σ in probability, it is possible to verify that | V n ( x ) | , ∥ W n ( x ) ∥ converge to 0 in probability. As a result, T 2 also converges to 0 in probability.

All in all, we have R ( λ ) D n ( y ; x ) -R (0) ν ∗ ( y ; x ) = o p (1) , and thus, proved (S1).

Step 2: proof of (S2). For any y 1 , y 2 ∈ M ,

where the O p term is independent of y 1 , y 2 ∈ M . Therefore,

which proves (S2).

C Proof of Theorem 2

In this section, we prove the two claims in Theorem 2. Specifically, in Section C.1, we present and prove a lemma that controls the bias in the population estimator (Lemma 1), and in Section C.2, we present and prove a lemma that controls the variance of the empirical estimator (Lemma 2).

C.1 Bias in the population estimator

We recall the definition of Mahalanobis seminorm from Definition 5: ∥ x ∥ Σ := ( x ⊤ Σ † x ) 1 / 2 .

Lemma 1. Suppose that Assumptions (C0) and (C1) hold. If

then for any λ ∈ R + ,

where

Proof of Lemma 1. For the sake of brevity, we write φ ( λ ) ( x ) = φ ( λ ) ν ∗ ( x ) and φ ( x ) = φ (0) ν ∗ ( x ) throughout this proof, dropping the subscript ν ∗ . Likewise, we simply write µ = µ ν ∗ and Σ = Σ ν ∗ .

Step 1: A naïve upper bound. Observe that for any λ ∈ R + , x ∈ R p , and y ∈ M ,

where the last inequality follows from E ν ∗ ∥ X -µ ∥ 2 Σ -Σ ( λ ) = rank ( Σ -Σ ( λ ) ) .

We observe that the upper bound in (28) is monotone non-decreasing with respect to λ ∈ R + , and it converges to 0 as λ → 0 . To see this, for any λ ∈ R + , we let

where Σ = ∑ p i =1 λ i · v i v ⊤ i is an eigenvdecomposition of Σ . Letting Π V ( λ ) denote the projection matrix onto the subspace V ( λ ) , we note that Σ -Σ ( λ ) = Π V ( λ ) ΣΠ V ( λ ) , and that (Σ -Σ ( λ ) ) † = Π V ( λ ) Σ † Π V ( λ ) . Thus, rank ( Σ -Σ ( λ ) ) = dim V ( λ ) , and furthermore, we notice that V ( λ ) = { 0 } if and only if λ < λ min := min { λ i : λ i > 0 } . Therefore,

The observation (29), together with Assumption (C0), implies that d ( φ ( λ ) ( x ) , φ ( x ) ) = o (1) as λ → 0 .

Step 2: Controlling risk difference. Next, we move on to determine the order of d ( φ ( λ ) ( x ) , φ ( x ) ) -as a function of b λ ( x ) -for a fixed λ ∈ R . We may assume λ > λ min for the proof because the lemma is trivial otherwise, cf. (29). Assuming λ > λ min , we may decompose the difference in the population objective at φ ( λ ) ( x ) and φ ( x ) as follows:

We observe that both R 1 and R 2 are non-negative, due to the optimality of φ ( x ) and φ ( λ ) ( x ) . Then, we obtain an upper bound for R 1 using a similar argument as in (28). Specifically,

Step 3: Converting risk difference to bias. Lastly, we convert the upper bound (30) to an upper bound on the distance d ( φ ( λ ) ( x ) , φ ( x ) ) using Assumption (C1). To this end, we begin by confirming that

Thereafter, we choose an arbitrary K ∈ N and r ∈ R + whose values will be determined later in this proof. Then we obtain the following inequality using the so-called peeling technique:

Moreover, we decompose the numerator in the fraction appearing in the upper bound (31) as follows: Combining (30) with (31), we have

Note that C := 4 diam M C g > 0 is a constant independent of λ . Let r = 1 / ( α -1) , and observe that the upper bound in (32) becomes smaller than 1 for a sufficiently large K . Specifically,

As a result, the inequality ' d ( φ ( λ ) ( x ) , φ ( x ) ) > 2 K 0 · b λ ( x ) r ' in the indicator function must be false, and we conclude that

C.2 Variance of the empirical estimator

Lemma 2. Suppose that Assumptions (C0), (C1) and (C2) hold. For any λ ∈ R + such that λ ̸∈ spec ( Σ ν ∗ ) , it holds that

Proof of Lemma 2. Recall from the definition of λ -regularized Fréchet regression (Definition 4) and (9) that

Additionally, we define an auxiliary function ˜ R n ( y ; x ) as the 'empirical risk with population weight' such that

We present the rest of this proof in three steps, outlined as follows. In Step 1, we show the consistency of φ ( λ ) D n ( x ) , i.e., d ( φ ( λ ) D n ( x ) , φ ( λ ) ν ∗ ( x ) ) = o P (1) as n →∞ . In Step 2, we define the discrepancy variable Z ( λ ) n ( y ; x ) := R ( λ ) D n ( y ; x ) -R ( λ ) ν ∗ ( y ; x ) between the finite-sample and the population objectives, cf. (35), and prove a uniform upper bound for Z ( λ ) n ( y ; x ) that holds in a neighborhood of φ ( λ ) ν ∗ ( y ; x ) . Lastly, in Step 3, we utilize the peeling technique from empirical process theory to obtain the desired rate of convergence.

Step 1: Consistency. We first claim that d ( φ ( λ ) D n ( x ) , φ ( λ ) ν ∗ ( x ) ) = o P (1) by an argument similar to that used in the proof of Theorem 1. Specifcally, it suffices to show that

(S2') R ( λ ) D n ( · ; x ) : M→ R is asymptotically equicontinuous in probability.

Note that we already showed the asymptotic equicontinuity in the proof of Theorem 1; see (S2). Thus, it remains to show the pointwise convergence in probability. To show (S1'), we decompose

Next, we show that A ( λ ) n ( y ; x ) and B ( λ ) n ( y ; x ) respectively converge to 0 in probability.

Since ∥ ̂ µ n -µ ∥ 2 = O P ( n -1 / 2 ) and ∥ ̂ Σ ( λ ) n -Σ ( λ ) ∥ = O P ( n -1 / 2 ) (if λ ̸∈ spec Σ ) independent of λ > 0 , we also have | V ( λ ) n ( x ) | = O P ( n -1 / 2 ) and ∥ W ( λ ) n ( x ) ∥ 2 = O P ( n -1 / 2 ) . This implies that A ( λ ) n ( y ; x ) = o P (1) .

Moreover, we note that if ∥ x -µ ∥ Σ < ∞ , then the random variable w ( λ ) ν ∗ ( X,x ) has finite second moment

regardless of the value of λ > 0 . When diam( M ) < ∞ , the product w ( λ ) ν ∗ ( X,x ) · d 2 ( Y, y ) also has finite second moment. Since B ( λ ) n ( y ; x ) is the sample mean of IID random variables with mean zero and finite variance, it follows that

Step 2: Uniform control of the fluctuation in objective discrepancy. For any λ ∈ R + and any ( x, y ) ∈ R p ×M , we let Z ( λ ) n ( y ; x ) denote the random variable defined as

We observed that

where ℓ ( λ ) i ( y ; x ) := d 2 ( Y i , y ) -d 2 ( Y i , φ ( λ ) ν ∗ ( x ) ) .

Next, we analyze the asymptotic behavior of the two terms, A ( λ ) n ( y ; x ) and B ( λ ) n ( y ; x ) . Specifically, we establish upper bounds on their magnitudes that hold uniformly over a δ -neighborhood of φ ( λ ) ( x ) = φ ( λ ) ν ∗ ( x ) , which will be used later in Step 3 of this proof.

Firstly, we observe that for any δ > 0 ,

where we used the property of V ( λ ) n ( x ) and W ( λ ) n ( x ) discussed in the paragraph following (33). Since the stochastic magnitudes of V ( λ ) n ( x ) and W ( λ ) n ( x ) are independent of δ , (37) implies that there exists C ( λ ) 1 = C ( λ ) 1 ( x ) > 0 such that for any δ > 0 ,

Furthermore, for any γ, δ ∈ R + such that 0 ≤ γ < δ , let E ( λ ) n ( γ, δ ; x ) be defined as an event such that

For any γ ∈ [0 , δ ] , we have E n (0 , δ ; x ) ⊆ E n ( γ, δ ; x ) , and thus, lim inf n →∞ P ( E n ( γ, δ ; x ) ) = 1.

Next, we note that

Observe that d ( y, φ ( λ ) ν ∗ ( x ) ) ≤ diam( M ) < ∞ and recall that E ν ∗ [ w ( λ ) ν ∗ ( X,x ) 2 ] ≤ 2 { 1 + p ∥ x -µ ∥ 2 Σ } as shown in Step 1 of this proof, cf. (34). It follows from the uniform entropy condition (C2), Theorem 2.7.11, and Theorem 2.14.2 in [55] that there exists D e = D e ( x ) > 0 such that for all δ ∈ [0 , D e ) ,

where C ( λ ) 2 = 2( C e + 1) · diam( M ) · √ 1 + p ∥ x -µ ∥ 2 Σ is independent of δ > 0 and n ≥ 1 .

Step 3: Concluding the proof. Lastly, we combine the results from Steps 1-2 to show that, for any η > 0 , there exist K = K ( η ) > 0 and N = N ( η ) ≥ 1 such that P ( d ( φ ( λ ) D n ( x ) , φ ( λ ) ν ∗ ( x ) ) > 2 K n -β ) < η for any n ≥ N , where β > 0 is an absolute constant that will be determined later in this proof. We prove this claim using the peeling technique, in a similar manner as we did in the proof of Lemma 1. To avoid cluttered notation, we let ∆( x ) = d ( φ ( λ ) D n ( x ) , φ ( λ ) ν ∗ ( x ) ) in the rest of this proof.

For any fixed K ∈ N and a sufficiently large n = n ( K ) ≥ 1 satisfying 2 K n -β < D ∗ := D g ∧ D e , we observe that

where we used P ( A ) ≤ P ( B c ) + P ( A ∩ B ) to get the inequality. As we know that P ( ∆( x ) ≥ D ∗ ) = o (1) by Step 1 of this proof, we focus on showing an upper bound for the other term, P ( 2 K n -β ≤ ∆( x ) < D ∗ ) .

Step 3-A: Decomposition of P ( 2 K n -β ≤ ∆( x ) < D ∗ ) . For each n, k ∈ N , we define

̸

where we set F n,K -1 to be the entire event space so that G n,K = ( F n,K ) c . It is worth mentioning that G n,k and G n,k ′ are mutually exclusive for any k = k ′ ≥ K , and we will use this property when concluding the proof in Step 3-C below.

Now, we observe that

and that for every k ≥ K ,

As a result, we have

Step 3-B: Controlling C n,k . Next, we show an upper bound for C n,k . Suppose that 2 k n -β ≤ ∆( x ) < 2 k +1 n -β ∧ D ∗ and the event F n,k occurs. Then it follows from Assumption (C1) that

Therefore, we obtain that for each k ≥ K ,

Step 3-C: Concluding Step 3 . Combining (41), (43), and (45), we have

Moreover, G n,k are mutually exclusive, and thus,

Finally, we obtain the desired result by letting β = 1 2( α -1) .

D Proof of Theorem 3

In this section, we prove Theorem 4 that establishes an upper bound on d ( φ ( λ ) ˜ D n ( x ) , φ ( λ ) D n ( x ) ) . This section is organized as follows. Firstly, in Section D.1, we present several useful results from matrix perturbation theory as lemmas. Next, in Section D.2, we provide a key lemma (Lemma 6) that establishes the stability of the weight function when there is covariate noise. Lastly, in Section D.3, we state and prove Theorem 4, from which Theorem 3 can be easily derived.

D.1 Useful lemmas

Definition 7. Let n, p ∈ N and let M ∈ R n × p . The row projection matrix for M , denoted by Π row M ∈ R p × p , is a matrix such that

and the column projection matrix for M , denoted by Π col M ∈ R n × n , is a matrix such that

We recall from (6) that for any λ ∈ R + , the singular value thresholding (SVT) operator SVT ( λ ) is defined such that

In the rest of this section, we let M ( λ ) := SVT ( λ ) ( M ) for shorthand.

Lemma 3 (Properties of the row/column projection matrices) . Let n, p ∈ N , and M ∈ R n × p . For any λ ∈ R + , the following statements are true.

Π row M ( λ ) defines a projection in R p and rank Π row M ( λ ) = rank M ( λ ) .

Π col M ( λ ) defines a projection in R n and rank Π col M ( λ ) = rank M ( λ ) .

M Π row M ( λ ) M † = Π col M ( λ ) and M † Π col M ( λ ) M = Π row M ( λ ) .

Proof. Let r = rank M and consider a compact singular value decomposition (SVD) of M :

where s 1 , . . . , s r are non-zero singular values of M . Noticing that

and that M † = ∑ r i =1 s -1 i · v i u ⊤ i , the three conclusions of the lemma follow straightforwardly from the orthonormality of singular vectors.

Π row M ( λ ) = ∑ r i =1 v i v ⊤ i · 1 { s i > λ } is the projection onto the row space of M ( λ ) .

Π col M ( λ ) = ∑ r i =1 u i u ⊤ i · 1 { s i > λ } is the projection onto the column space of M ( λ ) .

Due to the orthonormality of singular vectors,

and likewise, M † Π col M ( λ ) M = Π row M ( λ ) .

In addition, we collect two classical results from matrix perturbation theory and state them as lemmas.

Lemma 4 ([50, Theorem 3.2]) . Let X , Z ∈ R n × p . Then the following equation is true:

⊥

where Π col X = I n -Π col X and Π row Z ⊥ = I p -Π row Z . Lemma 5 ([15, Theorems 2.4 & 2.5]) . Let X , Z ∈ R n × p . Then

Moreover, if rank X = rank Z , then

D.2 Stability of the weights under (small) perturbation in covariates

Let D n = { ( x i , y i ) ∈ R p ×M : i ∈ [ n ] } and ˜ D n = { ( z i , y i ) ∈ R p ×M : i ∈ [ n ] } be two sets in R p ×M . We may identify these sets with their empirical distributions. Recall the definition of w ( λ ) ν from (9): for any probability measure ν on R p ×M , any λ ∈ R + , and any x, x ′ ∈ R p ,

where µ ν = E ( X,Y ) ∼ ν ( X ) and Σ ν = Var ( X,Y ) ∼ ν ( X ) , cf. (7). We define the weight vectors induced by D n and ˜ D n as follows: for any λ ∈ R + and any x ∈ R p ,

Lemma 6 (Stability of weights) . Let D n = { ( x i , y i ) ∈ R p ×M : i ∈ [ n ] } and ˜ D n = { ( z i , y i ) ∈ R p ×M : i ∈ [ n ] } . Let X = [ x 1 · · · x n ] ⊤ ∈ R n × p and Z = [ z 1 · · · z n ] ⊤ ∈ R n × p . For any λ ∈ R + , if x ∈ R p satisfies x -µ D n ∈ rowsp ( X ctr ) , then

where X ctr = ( I n -1 n 1 n 1 ⊤ n ) X and σ ( λ ) ( X ) := inf { σ i ( X ) > λ : i ∈ N } (likewise for Z ).

Proof of Lemma 6. This proof consists of three steps. In Step 1, we express the weight discrepancy ⃗ w ( λ ) ˜ D n ( x ) -⃗ w ( λ ) D n ( x ) as a sum of matrix products using projections. In Step 2, we establish upper bounds on the norm of the expression obtained in Step 1. In Step 3, we collect intermediate results together and conclude the proof.

Step 1: Decomposition of the weight discrepancy. First of all, we rewrite ⃗ w ( λ ) ˜ D n ( x ) -⃗ w ( λ ) D n ( x ) in a compact matrix representation that is presented in (60) at the end of this step. To this end, we begin by observing that

For given λ ∈ R + , we let X ( λ ) ctr := SVT ( λ ) ( X ctr ) , and observe that

Then it follows that

Therefore, we have

where the equality in the last line follows from Lemma 3: X ctr Π row X ( λ ) ctr X † ctr = Π col X ( λ ) ctr .

Likewise, we repeat the above for ˜ D n and Z to write

Then, we obtain an expression for ⃗ w ( λ ) ˜ D n ( x ) in a similar form to (55), namely,

Thereafter, we define c x , ˜ c x ∈ R n × 1 so that

Then we observe that for any x ∈ R ,

Likewise,

Consequently, for any x ∈ R p , we obtain from (55) and (56) with aid of (58) and (59) that

By triangle inequality, we obtain the following upper bound:

Step 2: Upper bounding the norm. Next, we establish separate upper bounds for the two terms in (61).

(1) The first term in (61) . First of all, we observe from the definition of c x and ˜ c x , cf. (57), that

Then we can upper bound the first term in (61) as follows:

Next, we consider the orthogonal decomposition of x -µ D n :

Similarly, the second term in (62) can be bounded by

All in all, we obtain

(2) The second term in (61) . Letting E ( λ ) := Z ( λ ) -X ( λ ) , we observe that

All in all, we obtain the following upper bound:

Step 3: Concluding the proof. We conclude this proof by inserting the upper bounds (64) and (65) from Step 2 into the upper bound (61) in Step 1. Specifically, we obtain

D.3 Completing the proof of Theorem 3

Recall that given a set D n = { ( x i , y i ) : i ∈ [ n ] } , we let X D n := [ x 1 · · · x n ] ⊤ ∈ R n × p . In addition, we let

Recall that we let X = X D n and Z = X ˜ D n for shorthand, and further, we let X ctr = ( I n -1 n 1 n 1 ⊤ n ) X and Z ctr = ( I n -1 n 1 n 1 ⊤ n ) Z denote the 'row-centered' matrices. Here we present and prove the complete version of Theorem 3.

Theorem 4 (De-noising covariates) . Suppose that Assumptions (C0) and (C1) hold. For any λ ∈ R + , if x ∈ µ D n +rowsp X ctr and

then

Proof of Theorem 4. First of all, we recall from (51) that

⃗ w ( λ ) D n ( x ) = [ w ( λ ) D n ( x 1 , x ) · · · w ( λ ) D n ( x n , x ) ] and ⃗ w ( λ ) ˜ D n ( x ) = [ w ( λ ) ˜ D n ( z 1 , x ) · · · w ( λ ) ˜ D n ( z n , x ) ] . In addition, recall that we let for any y ∈ M ,

Thereafter, we observe that for any y ∈ M and any x ∈ ( µ +rowsp X ) ,

where (a) is due to Cauchy-Schwarz inequality, and (b) follows from Lemma 6.

Using shorthand notation R n = R ( λ ) D n , ˜ R n = R ( λ ) ˜ D n , φ n = φ ( λ ) D n ( x ) , and ˜ φ n = φ ( λ ) ˜ D n ( x ) , we observe that

where (a) follows from the optimality of ˜ φ n , i.e., ˜ R n ( φ n ) ≥ ˜ R n ( ˜ φ n ) , and (b) is due to (69). Finally, we note that if

then the upper bound in (70) certifies that R n ( ˜ φ n ) -R n ( φ n ) < C g · D α g . Thus, we can use Assumption (C1) to convert the risk bound (70) to derive a distance bound between the minimizers:

which completes the proof.

E Further details on the experiments

Experimental setup in Section 5. We consider combinations of p ∈ { 150 , 300 , 600 } and n ∈ { 100 , 200 , 400 } . The datasets D n = { ( X i , Y i ) : i ∈ [ n ] } and ˜ D n = { ( Z i , Y i ) : i ∈ [ n ] } are generated as follows.

(True covariate X ) Let X i ∼ N p ( 0 p , Σ ) be IID multivariate Gaussian with mean 0 p and covariance Σ such that spec (Σ) = { κ j > 0 : j ∈ [ p ] } is an exponentially decreasing sequence such that tr (Σ) = ∑ p j =1 κ j = p . To be specific, for each p , we consider an exponentially decreasing sequence 1 = a 1 > · · · > a p = 10 -3 , and then set κ j = p · a j / ( ∑ j ′ =1 a j ′ ) for each j ∈ [ p ] . Note that ∑ ⌊ p/ 3 ⌋ j =1 κ j /∑ p j ′ =1 κ j ′ ≈ 0 . 9 , and thus, Σ is effectively low-rank.

(Noisy covariate Z ) For the error-prone covariate Z = X + ε , we consider two scenarios ε j IID ∼ N ( 0 , σ 2 ε ) and ε j IID ∼ Laplace ( 0 , σ ε ) . Note that in this setting, we have the signal-to-noise ratio E ( ∥ X ∥ 2 2 ) / E ( ∥ ε ∥ 2 2 ) = 1 /σ 2 ε . We set σ 2 ε = 0 . 05 2 .

(Response Y ) Given X = x , let Y be the distribution function of N ( µ α,β ( x ) + η, τ 2 ) , where

µ α,β ( x ) = α + β ⊤ x with α = 1 and β = p -1 / 2 · 1 p ,

η ∼ N ( 0 , σ 2 η ) ,

τ 2 ∼ IG ( s 1 , s 2 ) , an inverse gamma distribution with shape s 1 and scale s 2 .

We note that E ( τ 2 ) = s 2 s 1 -1 and Var( τ 2 ) = s 2 2 ( s 1 -1) 2 ( s 1 -2) . In particular, when τ 2 = 0 , this setting corresponds to the classical linear regression model for scalar responses. We set σ 2 η = 0 . 5 2 , and ( s 1 , s 2 ) = (18 , 17) . In this setting, we have

E ( µ α,β ( X ) ) = 1 and Var ( µ α,β ( X ) ) = β ⊤ Σ β ≈ 1 ,

E ( τ 2 ) = 1 and Var( τ 2 ) = 0 . 25 2 .

(Tuning parameter λ ) For simplicity, we chose a universal threshold value as

where Λ is a fine grid on ( 0 , √ λ 1 · p/n ) . Then the same threshold ˆ λ n was used to evaluate Bias 2 ( φ ( λ ) ˜ D ( b ) ( x )) , Var( φ ( λ ) ˜ D ( b ) ( x )) , and MSE( φ ( λ ) ˜ D ( b ) ( x )) for all b = 1 , . . . , B . Therefore, we claim that the performance of the SVT estimator reported in Table 1 has further room for improvement if one substitute ˆ λ ( b ) n = arg min λ ∈ Λ MSPE( φ ( λ ) ν ( b ) ) for each Monte Carlo experiment. Although suboptimal results are reported, we note that the proposed SVT outperforms both the oracle estimator and the naive EIV estimator in our simulation study. In practice, one may employ cross-validation for better performance. For the MSPE in Table 1, we reported min λ ∈ Λ MSPE( φ ( λ ) ˜ D n ) .

Evaluation metrics: bias and variance. We evaluate the accuracy and efficiency of the Fréchet regression function estimator using the bias and the variance. For any given x , we define

where ν ∈ {D n , ˜ D n } and φ ( λ ) ν ( x ) := arg min y ∑ B b =1 d W ( φ ( λ ) ν ( b ) ( x ) , y ) 2 is the Fréchet mean of φ ( λ ) ν (1) ( x ) , . . . , φ ( λ ) ν ( B ) ( x ) . Note that these definitions are a generalization of the standard bias and variance of the regression function estimator in Euclidean spaces. We evaluate the global performance of the estimator by considering a fixed set of evaluation points, G M = { x m : m = 1 , . . . , M } , and compute

In our experiment, we generate the set G M by drawing x 1 , . . . , x M IID from the same distribution as X , with M = 500 .

Additional experiment with linear regression models Wealso conducted an additional experiment for the standard linear regression models with three different metric metrics.

(Model) The linear regression model for ( X,Y ) ∈ R p × R d is defined as Y = α + Xβ + η and the covariate is contaminated as Z = X + ε . We generate X using the effective low-rank model with a geometrically decaying spectrum and condition number 10 3 (see Appendix E in the original submission).

(Parameters) Here, we let α = 1 d +0 . 1 · g , β = d -1 / 2 · 1 p × d +0 . 1 · G , η ∼ N (0 , 0 . 5 2 · I d ) and ε ∼ N (0 , 0 . 5 2 · I p ) , with g , G being standard Gaussians.

(a)

L

1

(b)

L

2

(c)

L

∞

.15

L-norm

Lz-norm

6.35

UUOU-"7

0.10

6.25

0.30

9.25

0.30

0.25-

6.25

3eSN

2.17

0.30

0.10

9.09

0:05

0.85

Threshold A

0.65.758859

41a.1a242d.1a.n

0.45

Threshold.A

0.35

1.0

80085010.15 02025030.35040.450505506

Threshold A

0654.74.750.80.850.90.95

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper attempts to extend regression models by combining error-in-variables in high-dimensional predictors and responses in a metric space. The main method is principal component regression whereby principal components are formed in the predictor space and serve as dimension-reduced predictors. It ties into existing approaches for global object regression and is a valid attempt at a very challenging problem but unfortunately none of the problems is solved: The predictors are not high-dimensional as their dimension is fixed; the errors-in-variables are not mitigated: While a bound is provided where the errors appear, the error-in-variables model is not shown to allow for consistent estimation and why principal components would be a reasonable approach at error mitigation is not elucidated; tuning parameter choice is not investigated.\n\nSTRENGTHS:\nThis paper addresses two difficult topics, errors-in-variables and Frechet regression. Errors-in-variables is a major problem in regression settings and is an important issue for many applications. While much research has been done on errors-in-variables in standard regression settings, it remains a challenging issue and especially so when the responses are in metric spaces. Such new types of responses are also clearly relevant for modern data. An approach through principal component regression would be of interest if it leads to consistent estimation and is implemented in a data-adaptive way. So the topic of this paper is timely from several perspectives.\n\nWEAKNESSES:\n(1) The background is not well balanced. It emphasizes Frechet regression while it is very superficial about error-in-variables. Since principal component regression is used to address error-in-variables it is then of interest to report how PCR mitigates/addresses or is expected to mitigate error-in-variables even in standard regression setting.\n\n(2) The proposed method does not lead to convergence for errors-in-variables according to (16), but this is what is needed for a method to succeed in this setting. \n\n(3) There is no data-adaptive choice (along with theory justifying it) for the tuning parameter $\\lambda$, however this parameter may be a crucial feature for principal component regression implementations as it determines the number of components to be included as predictors. It is not clear how it should be chosen to mitigate errors-in-variables or when responses are in metric space.\n\nQUESTIONS:\nIs the bound in (16) sharp? \n\nHigh-dimensional predictors are mentioned but the dimension p is fixed throughout. Can you handle the case of large p where p would increase with sample size ?\n\nLIMITATIONS:\nSee the previous comments.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

comments: SUMMARY:
This paper discusses a new method for Fréchet regression that relates non-Euclidean response variable to multivariate predictors. The proposed method leverages the low-rank structure of the covariate matrix to improve the efficiency and accuracy of the estimator, particularly in high-dimensional and errors-in-variables regression settings. The authors provide a theoretical analysis of the estimator's properties and demonstrate its superior performance through numerical experiments.

STRENGTHS:
1. The authors propose the regularized global Fréchet regression framework that can effectively utilize the low-rank structure inherent in the covariate matrix.

2. The proposed method is straightforward to implement and works well in the errors-in variables regression setting.

3. The authors have provided both theoretical analysis and numerical experiments to demonstrate the effectiveness of the proposed method.

WEAKNESSES:
1. The paper looks interesting, but it does not appear to be significantly advancing the state of the art in the field, e.g. [38, 41]

2. In Theorem 2, it is not clear how $\lambda$ affects the convergence rate in general.

3. No discussions regarding the performance of the three algorithms are given after Table 1 in Section 5.

QUESTIONS:
1. In the numerical experiments, does the method still work for the high-dimensional case when $p>n$?

2. How to choose the regularization parameter $\lambda$ in practice?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

comments: SUMMARY:
This paper proposes a new method in Frechet regression of non-Euclidean response variables, with a particular focus on high-dimensional, errors-in-variables regression. The idea is to combine the original Frechet regression with Principle Component Regression (PCR). In this way, the low-rank structure in the matrix of (Euclidean) covariates is utilized by extracting its principal components via low-rank matrix approximation. Theoretical analysis of consistency, convergence etc of the proposed method have been provided. Numerical experiments on simulated datasets have also been presented.

STRENGTHS:
(1) The proposed method tackles limitations of Frechet estimation such as reliance on ideal scenarios with abundant and noiseless
covariate data;

(2) The method utilizes the low-rank structure so that it can be applied to high-dimensional settings;

(3) both theoretical analysis and empirical results are demonstrated.

WEAKNESSES:
(1) underlines some terms and references which are rarely seen in papers;

(2) line 98: after "Frechet regression.", there is an additional phrase "in this study."

QUESTIONS:
N/A

LIMITATIONS:
no limitations provided
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed method leverages the low-rank structure inherent in the covariate matrix to improve efficiency and accuracy. It combines global Fréchet regression with principal component regression to enable more effective modeling and estimation, especially in high-dimensional and errors-in-variables regression settings. The paper provides a theoretical analysis of the proposed estimator's properties in large samples, including bias, variance, and variations due to measurement errors. Empirical experiments support the theoretical findings, demonstrating the superior performance of the approach.\n\nSTRENGTHS:\nStrengths of this paper include:\n\nThe paper introduces a new framework, called regularized (global) Fréchet regression, which combines Fréchet regression and principal component regression. This framework effectively utilizes the low-rank structure in the covariate matrix by extracting principal components through low-rank matrix approximation.\n\nThe paper provides a thorough theoretical analysis with three main theorems. Firstly, it proves the consistency of the proposed estimator for the true global Fréchet regression model. Secondly, it investigates the convergence rate of the estimator's bias and variance. Lastly, it derives an upper bound for the distance between estimates obtained with error-free covariates and those with errors-in-variables covariates. These results establish the effectiveness of the proposed framework in addressing model mis-specification and achieving more efficient model estimation.\n\nNumerical experiments conducted on simulated datasets validate the theoretical findings. The results demonstrate that the proposed method provides more accurate estimates of regression parameters, particularly in high-dimensional settings. The experiments highlight the importance of incorporating the low-rank structure of covariates in Fréchet regression and provide empirical evidence that aligns with the theoretical analysis.\n\nWEAKNESSES:\nThe authors should have performed experiments in diverse settings using multiple distance metrics. It would have been interesting to see the relation of MSPE against threshold in different settings.\n\nQUESTIONS:\nWhat is the difference between MSE and MSPE, does MSPE stand for Mean Square Prediction Error? The abbreviations in the experiments section have not been explained properly like REV, EIV, SVT, etc. Line 147 can be rewritten as \" .. conditional distribution of Y given X = x is normally distributed .. \"\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission 5822 makes a step forward in exploring the Fréchet regression, which is a significant approach for non-Euclidean response variables. Compared to existing work, this research has specific focuses on 1) high-dimensional, 2) errors-in-variables settings, and designed  a novel framework that combines the Fréchet regression and the principal component regression. The analysis in paper 1) proved the consistency of the proposed estimator, 2) investigated the convergence rate, and 3) derived an upper bound for the distance between the the estimates from error-free and errors-in-variables covariates. Some numerical simulations on synthetic datasets are provided.\n\nSTRENGTHS:\n- this work is well-written and provides results that will I believe be of interest to the community. The idea to design such a regression scheme is novel to me. In particular, the framework and main theorems described in Sec. 3 and Sec. 4 are convincing and well presented. The literature (from what I know) is globally well discussed.\n\n- claims and mathematical derivations seem to be sound and correct, and it clearly state its contributions, notation and results.\n\n- also the R implementation is offered in supplement, which allows for reproducing the key results using the README file\n\nWEAKNESSES:\n- I understand this paper is a theory-based work. However, I still think the numerical simulations are too weak and might not be convincing. \n  - the scale of the experiment is too small. The authors are suggested to consider larger $p$ and $n$, as well as more concrete examples/models as mentioned in Sec. 4.1\n  - the synthesized data ($p<n$) in implementation is *not* high-dimensional, which is the main setting of this work\n  - lack of experiments on real-world benchmarks, e.g., UCI, libsvm datasets\n\n- maybe the motivations of this research should be reconsidered. \n  - why the authors want to combine the *Fréchet regression* and *errors-in-variables setting*, along with the specific *PCR*? The authors only mentioned some advances in PCR research in line 96 and expressed their inspiration from this. Infact, there is whole lot of literature on high dimensional statistical learning and robust modeling. Why not borrow relevant ideas? \n  - compared to the existing work of Fréchet  regression, what are the core challenges (difficulties) of this work, and did the authors draw on some existing frameworks / proof techniques from them?\n\nQUESTIONS:\n- in experiments, the Gaussian noise added to covariate may be too ideal. Although this assumption is common, could you provide results of other forms of noise (for example, some work on adversarial perturbations)?\n\n- how would proposed theorems extend to concrete models/objectives? providing a practical guide is better for helping readers understand the contribution of this paper\n\n- line 826: ''$\\Lambda$ is a fine grid on ...''  adding more details about how to define the searching grid and the final chosen value of $\\lambda$ are appreciated\n\n- some lines have small typos, e.g., in line 523, it should be Proof of Proposition **4**\n\nI'm not an expert of non-Euclidean regression analysis and would be happy to revise the rating if I missed some key aspects\n\nLIMITATIONS:\n- authors did not discuss in detail the limitations of the proposed regression framework (while they said \"Yes\" in checklist), but given that this work is based on theoretical analysis, I believe that all Assumptions in main body could clarify the technical limitations\n\n- there's no need to discuss potential negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}