SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Errors-in-variables Fréchet Regression with Low-rank Covariate Approximation
Dogyoon Song*
Department of EECS  
University of Michigan  
in Arbor, MI 48109, USA  
logyoons@umich.edu
Kyunghee Han*
Department of Math, Stat and Comp Sci  
University of Illinois at Chicago  
Chicago, IL 60607, USA  
hankh@uic.edu
Frechet regression has emerged as a promising approach for regression analysis involving non-Euclidean response variables. However, its practical applicability has been hindered by its reliance on ideal scenarios with abundant and noiseless covariate data. In this paper, we present a novel estimation method that tackles these limitations by leveraging the low-rank structure inherent in the covariate matrix. Our proposed framework combines the concepts of global Frechet regression and principal component regression, aiming to improve the efficiency and accuracy of the regression estimator. By incorporating the low-rank structure, our method enables more effective modeling and estimation, particularly in high-dimensional and errors-in-variables regression settings. We provide a theoretical analysis of the proposed estimator's large-sample properties, including a comprehensive rate analysis of bias, variance, and additional variations due to measurement errors. Furthermore, our numerical experiments provide empirical evidence that supports the theoretical findings, demonstrating the superior performance of our approach. Overall, this work introduces a promising framework for regression analysis of non-Euclidean variables, effectively addressing the challenges associated with limited and noisy covariate data, with potential applications in diverse fields.
Regression analysis is a fundamental statistical methodology to model the relationship between response variables and explanatory variables (covariates). Linear regression, for example, models the (conditional) expected value of the response variable as a linear function of covariates. Regression models enable researchers and analysts to make predictions, gain insights into how input variables influence the outcomes of interest, and validate hypothetical associations between variables in inferential studies. As a result, regression is widely utilized across various scientific domains, including economics, psychology, biology, and engineering [53, 21, 29].
In recent decades, there has been a growing interest in developing statistical methods capable of handling random objects in non-Euclidean spaces. Examples of these include functional data analysis [41], statistical manifold learning [31], statistical network analysis [34], and object-oriented data analysis [39]. In such contexts, the response variable is defined in a metric space that may lack an algebraic structure, making it challenging to apply global, parametric approaches toward regression as in the classical Euclidean setting. To overcome this challenge, (global) Fréchet regression, which models the relationship by fitting the (conditional) barycenters of the responses as a function of covariates, has been introduced [40]. Notably, when the Euclidean metric is considered, Fréchet
*Equal contribution. To whom correspondence should be addressed: hankh@uic.edu
regression recovers classical Euclidean regression models. For more details on Fréchet regression and its recent developments, we refer readers to [30, 40, 22, 45, 27].
Nevertheless, most existing research on Fréchet regression has focused on ideal scenarios characterized by abundant covariate data that are accurately measured and free of noise. In practical applications, however, high-dimensional data often arise, which are also susceptible to measurement errors and other forms of contamination. These errors can stem from various sources, such as unreliable data collection methods (e.g., low-resolution probes, subjective self-reports) or imperfect data storage and transmission. The high-dimensionality and the presence of measurement errors in covariates pose critical challenges for statistical inference, as regression analysis based on error-prone covariates may result in incorrect associations between variables, yielding misleading conclusions.
To address these limitations, it is crucial to extend the methodology and analysis of Fréchet regression to tackle high-dimensional errors-in-variables problems. In this work, we aim to leverage the low-rank structure in the covariates to enhance the estimation accuracy and computational efficiency of Fréchet regression. Specifically, we explore the extension of principal component regression to handle errors-in-variables regression problems with non-Euclidean response variables.
This paper contributes to advancing the (global) Fréchet regression of non-Euclidean response variables, with a particular focus on high-dimensional, errors-in-variables regression.
Firstly, we propose a novel framework, called the regularized (global) Fréchet regression (Section 3) that combines the ideas from Fréchet regression [40] and the principal component regression [32]. This framework effectively utilizes the low-rank structure in the matrix of (Euclidean) covariates by extracting its principal components via low-rank matrix approximation. Our proposed method is straightforward to implement, not requiring any knowledge about the error-generating mechanism.
Furthermore, we provide a comprehensive theoretical analysis (Section 4) in three main theorems to establish the effectiveness of the proposed framework. Firstly, we prove the consistency of the proposed estimator for the true global Fréchet regression model (Theorem 1). Secondly, we investigate the convergence rate of the estimator's bias and variance (Theorem 2). Lastly, we derive an upper bound for the distance between the estimates obtained using error-free covariates and those with errors-in-variables covariates (Theorem 3). Collectively, these results demonstrate that our approach effectively addresses model mis-specification and achieves more efficient model estimation by leveraging the low-rank structure of covariates, despite the presence of inherent bias due to unobserved measurement errors.
To validate our theoretical findings, we conduct numerical experiments on synthetic datasets (Section 5). We observe that the proposed method provides more accurate estimates of the regression parameters, especially in high-dimensional settings. Our experimental results emphasize the importance of incorporating the low-rank structure of covariates in Fréchet regression, and provide empirical evidence that aligns with our theoretical analysis.
Metric-space-valued variables. Nonparametric regression models for Riemannian-manifold-valued responses were proposed as a generalization of regression for multivariate outputs by Steinke et al. [48, 49]. These works provided a foundation for recent developments in regression analysis of non-Euclidean responses. Later, Hein [30] proposed a Nadaraya-Watson-type kernel estimation of regression model for general metric-space-valued outcomes. Since then, statistical properties of regression models for some special classes of metric-space-valued outcomes, such as distribution functions [23, 52, 28] and matrix-valued responses [57, 20], have been investigated. Recently, many researchers have introduced further advances in Fréchet regression, including [40, 10, 37, 45]. In this study, we use the global Fréchet regression proposed in [40] as the basis for our proposed method.
Errors-in-variables regression. Much of earlier work on errors-in-variables (EIV) problems in the statistical literature can be found in [13], which covers the simulation-extrapolation (SIMEX) [16, 11], the attenuation correction method [36], covariate-adjusted model [46, 19], and the deconvolution kernel method [25, 24, 18]. The regression calibration method [47], instrumental variable modeling [12, 43], and the two-phase study design [9, 4] were also proposed when additional data are available
for correcting measurement errors. In the high-dimensional modeling literature, regularization methods for recovering the true covariate structure can also be utilized [38, 7, 17]. Despite a diverse body of literature on high-dimensional learning and robust regression modeling, much of it assumes response spaces to be vector spaces endowed with inner products. In this paper, we tackle EIV problems within the Fréchet regression framework. While previous works have explored regression analysis in non-Euclidean metric spaces, addressing EIV issues in this context remains uncharted.
Principal component regression. The principal component regression (PCR) [32] is a statistical technique that regresses response variables on principal component scores of the covariate matrix. The conventional PCR selects a few principal components as the "new" regressors associated with the first leading eigenvalues to explain the highest proportion of variations observed in the original covariate matrix. In functional data analysis, PCR is known to have a shrinkage effect on the model estimate and produce robust prediction performance in functional regression [42, 33]. Recently, Agarwal et al. [2] investigated the robustness of PCR in the presence of measurement errors on covariates and the statistical guarantees for learning a good predictive model. Unlike prior statistical analyses of EIV problems that often assume known or estimable noise distributions, PCR leverages inherent low-rank structures in the covariates without requiring a priori knowledge of measurement error distributions. We adopt PCR as a concrete, practical solution to EIV models in non-Euclidean regression, driven by two compelling considerations. Firstly, the prevalence of (approximate) low-rank structures in real-world datasets enhances the practical relevance of our approach. Secondly, we intentionally opt for an approach with minimal assumptions regarding covariate errors to ensure broad applicability.
1.3 Organization
In Section 2, we introduce the notation used throughout the paper, and overview the global Fréchet regression framework. Section 3 presents the problem setup, objectives, and our proposed estimator, which we refer to as the regularized Fréchet regression (Definition 4). In Section 4, we discuss theoretical guarantees on the regularized Fréchet regression method in accurately estimating the global Fréchet regression function. Section 5 presents the results of numerical "proof-of-concept" experiments that support the theoretical findings. Finally, we conclude this paper with discussions in Section 6. Due to space constraints, detailed proofs of the theorems as well as additional details and discussions of experiments are provided in the Appendix.
Let  $\mathbb{N}$  denote the set of positive integers and  $\mathbb{R}$  denote the set of real numbers. Also, let  $\mathbb{R}_+ := \{x \in \mathbb{R} : x \geq 0\}$ . For  $n \in \mathbb{N}$ , we let  $[n] := \{1, \ldots, n\}$ . We mostly use plain letters to denote scalars, vectors, and random variables, but we also use boldface uppercase letters for matrices, and curly letters to denote sets when useful. Note that we may identify a vector with its column matrix representation. For a matrix  $X$ , we let  $X^{-1}$  denote its inverse (if exists) and  $X^{\dagger}$  denote the Moore-Penrose pseudoinverse of  $X$ . Also, we let  $\operatorname{rowsp}(X)$  and  $\operatorname{colsp}(X)$  denote the row and column spaces of  $X$ , respectively. Furthermore, we let  $\operatorname{spec}(X)$  denote the set of non-zero singular values of  $X$ ,  $\sigma_i(X)$  denote the  $i$ -th largest singular value of  $X$ , and  $\sigma^{(\lambda)}(X) := \inf \{\sigma_i(X) > \lambda : i \in \mathbb{N}\}$  with the convention  $\inf \emptyset = \infty$ . We let  $\mathbf{1}_n = (1, 1, \ldots, 1)^{\top} \in \mathbb{R}^d$  and let  $\mathbb{1}$  denote the indicator function. We let  $\| \cdot \|$  denote a norm, and set  $\| \cdot \| = \| \cdot \|_2$  (the  $\ell_2$ -norm for vectors, and the spectral norm for matrices) by default unless stated otherwise. For a finite set  $\mathcal{D}$ , we may identify  $\mathcal{D}$  with its empirical measure  $\nu_{\mathcal{D}} = \frac{1}{|\mathcal{D}|} \sum_{x \in \mathcal{D}} \delta_x$ , where  $\delta_x$  denotes the Dirac measure supported on  $\{x\}$ .
Letting  $f, g: \mathbb{R} \to \mathbb{R}$ , we write  $f(x) = O(g(x))$  as  $x \to \infty$  if there exist  $M > 0$  and  $x_0 > 0$  such that  $|f(x)| \leq M \cdot g(x)$  for all  $x \geq x_0$ . Likewise, we write  $f(x) = \Omega(g(x))$  if  $g(x) = O(f(x))$ . Furthermore, we write  $f(x) = o(g(x))$  as  $x \to \infty$  if  $\lim_{x \to \infty} \frac{f(x)}{g(x)} = 0$ . For a sequence of random variables  $X_n$ , and a sequence  $a_n$ , we write  $X_n = O_p(a_n)$  as  $n \to \infty$  if for any  $\varepsilon > 0$ , there exists  $M \in \mathbb{R}_+$  and  $N \in \mathbb{N}$  such that  $P\left(\left|\frac{X_n}{a_n}\right| > M\right) < \varepsilon$  for all  $n \geq N$ . Similarly, we write  $X_n = o_p(a_n)$  if  $\lim_{n \to \infty} P\left(\left|\frac{X_n}{a_n}\right| > \varepsilon\right) = 0$  for all  $\varepsilon > 0$ .
2.2 Global Fréchet regression
Let  $(X,Y)$  be a random variable that has a joint distribution  $P_{X,Y}$  supported on  $\mathbb{R}^p\times \mathcal{M}$ , where  $\mathbb{R}^p$  is the  $p$ -dimensional Euclidean space and  $\mathcal{M} = (\mathcal{M},d)$  is a metric space equipped with a distance function  $d:\mathcal{M}\times \mathcal{M}\to \mathbb{R}$ . We write the marginal distribution of  $X$  as  $P_{X}$ , and the conditional distribution of  $Y$  given  $X$  as  $P_{Y|X}$ .
Definition 1 (Frechet regression function). Let  $(X,Y)$  be a random element that takes value in  $\mathbb{R}^p\times \mathcal{M}$ . The Frechet regression function of  $Y$  on  $X$  is a function  $\varphi^{*}:\mathbb{R}^{p}\to \mathcal{M}$  such that
$$
\varphi^ {*} (x) = \underset {y \in \mathcal {M}} {\arg \min } \mathbb {E} \left[ d ^ {2} (Y, y) \mid X = x \right], \quad \forall x \in \operatorname {s u p p} P _ {X} \subseteq \mathbb {R} ^ {p}. \tag {1}
$$
We note that  $\varphi^{*}(x)$  is the best predictor of  $Y$  given  $X = x$ , as it minimizes the marginal risk  $\mathbb{E}\big[d^2 (Y,\varphi^* (X))\big]$  under the squared-distance loss. In the literature,  $\varphi^{*}(x)$  is also known as the conditional Fréchet mean of  $Y$  given  $X = x$  [26]. It is important to recognize that the existence and uniqueness of the Fréchet regression function are closely tied to the geometric characteristics of  $\mathcal{M}$ , and are not guaranteed in general [3, 8]. Nonetheless, extensive research has been conducted on the existence and uniqueness of Fréchet means in various metric spaces commonly encountered in practical applications. Examples include the unit circle in  $\mathbb{R}^2$  [14], Riemannian manifolds [1, 5], Alexandrov spaces with non-positive curvature [51], metric spaces with upper bounded curvature [58], and Wasserstein space [59, 35].
While modeling and estimating the Fréchet regression function  $\varphi^{*}$  is often of interest, its global (parametric) modeling may not be straightforward, especially when  $\mathcal{M}$  lacks a useful algebraic structure, such as an inner product. For instance, in classical linear regression analysis with  $\mathcal{M} = \mathbb{R}$ , the conditional distribution of  $Y$  given  $X = x$  is normally distributed with a mean of  $\varphi^{*}(x) = \alpha + \beta^{\top}x$  and a fixed variance  $\sigma^2$ , where  $\alpha$  and  $\beta$  represent the regression coefficients. Similarly, when  $\mathcal{M}$  possesses a linear-algebraic structure, one can specify a class of regression functions that quantifies the association between the expected outcome and covariates in an additive and multiplicative manner. However, the lack of an algebraic structure in general metric spaces may prevent us from characterizing the barycenter  $\varphi^{*}(x)$  in the same way classical regression analysis determines the expected value of outcomes with changing covariates.
To address this challenge, Petersen and Müller [40] recently proposed to exploit algebraic structures in the space of covariates,  $\mathbb{R}^p$ , instead of  $\mathcal{M}$ . Specifically, they consider a weighted Fréchet mean as
$$
\varphi (x) = \underset {y \in \mathcal {M}} {\arg \min } \mathbb {E} [ w (X, x) \cdot d ^ {2} (Y, y) ], \tag {2}
$$
where  $w: \mathbb{R}^p \times \mathbb{R}^p \to \mathbb{R}$  is a weight function such that  $w(\xi, x)$  denotes the influence of  $\xi$  at  $x$ . In particular, Petersen and Müller [40] defined the global Fréchet regression function with a specific choice of  $w$  as follows.
Definition 2 (Global Fréchet regression function). Let  $(X,Y)$  be a random variable in  $\mathbb{R}^p\times \mathcal{M}$ . Let  $\mu = \mathbb{E}(X)$  and  $\Sigma = \operatorname {Var}(X)$ . The global Fréchet regression function of  $Y$  on  $X$  is a function  $\varphi_{\mathrm{glo}}:\mathbb{R}^p\to \mathcal{M}$  such that
$$
\varphi_ {\mathrm {g l o}} (x) = \underset {y \in \mathcal {M}} {\arg \min } \mathbb {E} \left[ w _ {\mathrm {g l o}} (X, x) \cdot d ^ {2} (Y, y) \right] \tag {3}
$$
where  $w_{\mathrm{glo}}(X,x) = 1 + (X - \mu)^{\top}\Sigma^{-1}(x - \mu)$ .
When  $\mathcal{M}$  is an inner product space (e.g.,  $\mathcal{M} = \mathbb{R}$ ), the function  $\varphi_{\mathrm{glo}}$  restores the standard linear regression model representation over the domain  $\mathbb{R}^p$ . For this reason,  $\varphi_{\mathrm{glo}}$  is commonly referred to as the global Fréchet regression model for metric-space-valued outcomes [40, 37, 54].
Remark on Definition 2 One might wonder why the term "global" is used to describe  $\varphi_{\mathrm{glo}}$  as a Fréchet regression function. The use of the adjective "global" serves to emphasize its distinction from "local" nonparametric regression methods that interpolate data points. Notably, when  $\mathcal{M}$  is a Hilbert space,  $\varphi_{\mathrm{glo}}$  reduces to the natural linear models. For instance, if  $\mathcal{M} = \mathbb{R}$ , then it follows that  $\varphi_{\mathrm{glo}}(x) = \mathbb{E}\big[w_{\mathrm{glo}}(X,x)\cdot Y\big] = \alpha +\beta^{\top}(x - \mu)$ , where  $\alpha = \mathbb{E}[Y]$  and  $\beta = \Sigma^{-1}\cdot \mathbb{E}\big[(X - \mu)\cdot Y\big]$ . These linear models hold uniformly for the evaluation point  $x$ . Similarly, in the case of an  $L^2$  space equipped with the squared-distance metric  $d^{2}(y,y^{\prime}) = \| y - y^{\prime}\|_{2}^{2}$  induced by the  $L^2$  norm,  $\varphi_{\mathrm{glo}}$  represents the linear regression model for functional responses. Thus,  $\varphi_{\mathrm{glo}}$  establishes a globally defined model that spans the entire space.
3 Problem and methodology
Let  $(X,Y)$  be a random variable in  $\mathbb{R}^p\times \mathcal{M}$  and  $P_{X,Y}$  be their joint distribution. Let  $\mathcal{D}_n = \{(X_i,Y_i):i\in [n]\}$  be an independent and identically distributed (IID) sample drawn from  $P_{X,Y}$ . Note that we may identify the set  $\mathcal{D}_n$  with its discrete measure (empirical distribution), cf. Section 2.1. We consider the problem of estimating the global Fréchet regression function  $\varphi_{\mathrm{glo}}$  (see Definition 2) from data  $\mathcal{D}_n$ . In this setting, a natural estimator of  $\varphi_{\mathrm{glo}}$  would be its sample-analogue estimator. With  $\widehat{\mu}_{\mathcal{D}_n} = \mathbb{E}_{(X,Y)\sim \mathcal{D}_n}(X) = \frac{1}{n}\sum_{i = 1}^{n}X_i$  and  $\widehat{\Sigma}_{\mathcal{D}_n} = \operatorname {Var}_{(X,Y)\sim \mathcal{D}_n}(X) = \frac{1}{n}\sum_{i = 1}^{n}(X_i - \widehat{\mu}_{\mathcal{D}_n})\cdot$ $(X_{i} - \widehat{\mu}_{\mathcal{D}_{n}})^{\top}$ , the sample-analogue estimator  $\widehat{\varphi}_{\mathcal{D}_n}$  is defined as
$$
\widehat {\varphi} _ {\mathcal {D} _ {n}} (x) = \underset {y \in \mathcal {M}} {\arg \min } \left\{\frac {1}{n} \sum_ {\left(X _ {i}, Y _ {i}\right) \in \mathcal {D} _ {n}} \widehat {w} _ {\mathcal {D} _ {n}} \left(X _ {i}, x\right) \cdot d ^ {2} \left(Y _ {i}, y\right) \right] \tag {4}
$$
where  $\widehat{w}_{\mathcal{D}_n}(X,x) = 1 + (X - \widehat{\mu}_{\mathcal{D}_n})^\top \widehat{\Sigma}_{\mathcal{D}_n}^{-1}(x - \widehat{\mu}_{\mathcal{D}_n})$ . The statistical properties of  $\widehat{\varphi}_{\mathcal{D}_n}$ , including the asymptotic distribution, a ridge-type variable selection operation, and total variation regularization method have been investigated [40, 37, 54].
In practice, however, we may only be able to access  $\widetilde{\mathcal{D}}_n = \{(Z_i,Y_i):i\in [n]\}$  instead of  $\mathcal{D}_n$  , where
$$
Z _ {i} = X _ {i} + \varepsilon_ {i}, \quad i = 1, \dots , n \tag {5}
$$
denotes an error-prone observation of the covariates  $X$  by measurement error  $\varepsilon$ . This formulation corresponds to the classical errors-in-variables problem.
Objective. Given a dataset, either  $\mathcal{D}_n$  or  $\widetilde{\mathcal{D}}_n$ , our aim is to produce an estimate  $\widehat{\varphi}$  of the global Fréchet regression function  $\varphi_{\mathrm{glo}}$  so that the prediction error is minimized. Specifically, we evaluate the performance of  $\widehat{\varphi}$  by means of the distance in the response space,  $d\big(\widehat{\varphi} (x),\varphi_{\mathrm{glo}}(x)\big)$ .
3.2 Fréchet regression with covariate principal components
Singular value thresholding. Among various low-rank matrix approximation methods, we consider the (hard) singular value thresholding (SVT). For any  $\lambda \in \mathbb{R}_+$ , we define the map  $\mathrm{SVT}^{(\lambda)}: \mathbb{R}^{n \times p} \to \mathbb{R}^{n \times p}$  that removes all singular values that are less than the threshold  $\lambda$ . To be precise,  $\mathrm{SVT}^{(\lambda)}$  can be expressed in terms of the singular value decomposition (SVD) as follows:
$$
\boldsymbol {M} = \sum_ {i = 1} ^ {\min  \{n, p \}} s _ {i} \cdot u _ {i} v _ {i} ^ {\top} \text {i s a S V D} \quad \Longrightarrow \quad \operatorname {S V T} ^ {(\lambda)} (\boldsymbol {M}) = \sum_ {i = 1} ^ {\min  \{n, p \}} s _ {i} \cdot \mathbb {1} \left\{s _ {i} > \lambda \right\} \cdot u _ {i} v _ {i} ^ {\top}. \tag {6}
$$
Regularized Fréchet regression. We introduce a variant of the sample-analog estimator of the global Fréchet regression function based on principal components of the sample covariance. To facilitate the description of our proposed estimator, we introduce additional notation here.
Definition 3 (Covariate mean/covariance). For a probability distribution  $\nu$  on  $\mathbb{R}^p\times \mathcal{M}$ , the covariate mean and covariate covariance with respect to  $\nu$  are respectively defined as
$$
\mu_ {\nu} := \mathbb {E} _ {(X, Y) \sim \nu} (X) \quad a n d \quad \Sigma_ {\nu} := \operatorname {V a r} _ {(X, Y) \sim \nu} (X). \tag {7}
$$
Recall that a finite set  $\mathcal{D} \subset \mathbb{R}^p \times \mathcal{M}$  may be identified with its empirical distribution; it follows that
$$
\mu_ {\mathcal {D}} = \frac {1}{| \mathcal {D} |} \sum_ {(x _ {i}, y _ {i}) \in \mathcal {D}} x _ {i} \quad \text {a n d} \quad \Sigma_ {\mathcal {D}} = \frac {1}{| \mathcal {D} |} \sum_ {(x _ {i}, y _ {i}) \in \mathcal {D}} (x _ {i} - \mu_ {\mathcal {D}}) \cdot (x _ {i} - \mu_ {\mathcal {D}}) ^ {\top}. \tag {8}
$$
Definition 4 (Regularized Fréchet regression). Let  $\nu$  be a probability distribution on  $\mathbb{R}^p \times \mathcal{M}$  and  $\lambda \in \mathbb{R}_+$ . The  $\lambda$ -regularized Fréchet regression function for  $\nu$  is a map  $\varphi_{\nu}^{(\lambda)}: \mathbb{R}^p \to \mathcal{M}$  such that
$$
\begin{array}{l} \varphi_ {\nu} ^ {(\lambda)} (x) = \underset {y \in \mathcal {M}} {\arg \min} R _ {\nu} ^ {(\lambda)} (y; x), \quad w h e r e \quad R _ {\nu} ^ {(\lambda)} (y; x) = \mathbb {E} _ {(X, Y) \sim \nu} \left[ w _ {\nu} ^ {(\lambda)} (X, x) \cdot d ^ {2} (Y, y) \right] \\ a n d \quad w _ {\nu} ^ {(\lambda)} \left(x ^ {\prime}, x\right) = 1 + \left(x ^ {\prime} - \mu_ {\nu}\right) ^ {\top} \left[ \mathrm {S V T} ^ {(\lambda)} \left(\Sigma_ {\nu}\right) \right] ^ {\dagger} (x - \mu_ {\nu}). \tag {9} \\ \end{array}
$$