SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiaming Qiu Fred Hutchinson Cancer Center jqiu3@fredhutch.org
Xiongtao Dai  
Division of Biostatistics, School of Public Health  
University of California, Berkeley  
xdai@berkeley.edu
We extend metric learning by studying the Riemannian manifold structure of the underlying data space induced by similarity measures between data points. The key quantity of interest here is the Riemannian metric, which characterizes the Riemannian geometry and defines straight lines and derivatives on the manifold. Being able to estimate the Riemannian metric allows us to gain insights into the underlying manifold and compute geometric features such as the geodesic curves. We model the observed similarity measures as noisy responses generated from a function of the intrinsic geodesic distance between data points. A new local regression approach is proposed to learn the Riemannian metric tensor and its derivatives based on a Taylor expansion for the squared geodesic distances, accommodating different types of data such as continuous, binary, or comparative responses. We develop theoretical foundation for our method by deriving the rates of convergence for the asymptotic bias and variance of the estimated metric tensor. The proposed method is shown to be versatile in simulation studies and real data applications involving taxi trip time in New York City and MNIST digits.
With increasing data complexity, it has received increasing attention for capturing the geometric structure of the data space characterized by a Riemannian manifold, which consists of two vital parts: the coordinate system and the Riemannian metric. Manifold learning attempts to find low-dimensional representations as coordinates for the high-dimensional data [23, 27, 7]. Yet existing methods generally assume the local geometry is given, e.g., by the Euclidean distance between ambient data points, leading to non-isometric methods that distort the data geometry in the embedded space. On the other hand, many methods have been proposed to consider the Riemannian metric, including parametric estimation [17, 16, 15]; multi-metric learning [12]; pushforward via the Laplacian operator [21]; pull-back via Jacobian of generative models [1, 2]; Bayesian approach based on arc length of Gaussian process [14]; and dynamics of evolving probability when time-indexed samples are available [24]. It is also possible to recover the manifold in an abstract setting with only similarity measures among data points [9].
We propose to estimate the Riemannian metric utilizing similarity measures between data points when data coordinates are available. Suppose that data are generated from an unknown Riemannian manifold, while the Euclidean distance between the coordinates may not reflect the underlying geometry. We model the observed similarity measures between data points as noise-contaminated intrinsic distances, which are then used to characterize the intrinsic geometry on the Riemannian manifold. The targeted Riemannian metric is estimated in a data-driven fashion, which enables path finding via geodesics and calculus on the manifold in which the data reside.
This problem is closely related to metric learning [3, 26] where a Mahalanobis distance is commonly used to obtain the best distance for classification [11] and clustering [29] tasks. While a global metric is often the focus of earlier works, multiple local metrics [10, 28, 22, 5] are found to be useful because they better capture the data space geometry. Despite the resemblance, our target is to learn the Riemannian metric instead of the distance metric, which fundamentally differentiates our proposal from metric learning. We emphasize the data space geometry rather than obtaining a metric optimal for subsequent supervised learning tasks. The additional smoothness of the Riemannian metric tensor field allows analysis of finer structures, while the coefficient matrix for the Mahalanobis distance is locally constant for distance metric learning.
To formulate the problem, let  $(\mathcal{M}, G)$  be a Riemannian manifold with Riemannian metric  $G$  and induced geodesic distance  $\text{dist}(\cdot, \cdot)$  which measures the true intrinsic difference between points. Knowing the coordinates of data points  $x_0, x_1 \in \mathcal{M}$ , we identify each point via its coordinates (a tuple of real numbers). The noisy measures  $y$  of the intrinsic distance between data points are referred to as similarity measures (equivalently dissimilarity). The response is modeled flexibly, and we consider the following common scenarios: (i) noisy distance, where  $y = \text{dist}(x_0, x_1)^2 + \epsilon$  for error  $\epsilon$ , (ii) similarity/dissimilarity, where  $y = 0$  if the two points  $x_0, x_1$  are considered similar and  $y = 1$  otherwise, and (iii) relative comparison, where a triplet of points  $(x_0, x_1, x_2)$  are given and  $y = 1$  if  $x_0$  is more similar to  $x_1$  than to  $x_2$  and  $y = 0$  otherwise. The binary similarity measurement is common in computer vision [e.g. 6], while the relative comparison could be useful for perceptional tasks and recommendation system [e.g. 25, 4]. We aim to estimate the Riemannian metric  $G$  and its derivatives using the coordinates and similarity measures among the data points.
The major contribution of this paper is threefold. First, we formulate a framework for probabilistic modeling of similarity measures among data on manifold via intrinsic distances. Such a framework also justifies the distance metric learning from a geometrical perspective and unveils its approximation error for the first time as we know. Second, a theoretical foundation is developed for the proposed method including asymptotic consistency. Last and most importantly, the proposed method provides a geometric interpretation for the structure of the data space induced by the similarity measures, as demonstrated in the numerical examples that include a taxi travel and an MNIST digit application.
2 Background in Riemannian Geometry
For brevity, metric now refers to Riemannian metric while distance metric is always spelled out. Throughout the paper,  $\mathcal{M}$  denotes a  $d$ -dimensional manifold endowed with a coordinate chart  $(U,\varphi)$ , where  $\varphi: U \to \mathbb{R}^d$  maps a point  $p \in U \subset \mathcal{M}$  on the manifold to its coordinate  $\varphi(p) = (\varphi^1(p), \ldots, \varphi^d(p)) \in \mathbb{R}^d$ . Without loss of generality, we identify a point by its coordinate as  $(p^1, \ldots, p^d)$ , suppressing  $\varphi$  for the coordinate chart. Upper-script Roman letters denote the components of a coordinate, e.g.,  $p^i$  is the  $i$ -th entry in the coordinate of the point  $p$ , and  $\gamma^i$  is the  $i$ -th component function of a curve  $\gamma: \mathbb{R} \supset [a,b] \to \mathcal{M}$  when expressed on chart  $U$ . The tangent space  $T_p\mathcal{M}$  is a vector space consisting of velocities of the form  $v = \gamma'(0)$  where  $\gamma$  is any curve satisfying  $\gamma(0) = p$ . The coordinate chart induces a basis on the tangent space  $T_p\mathcal{M}$ , as  $\partial_i|_p = \partial/\partial x^i|_p$  for  $i = 1, \ldots, d$ , so that a tangent vector  $v \in T_p\mathcal{M}$  is represented as  $v = \sum_{i=1}^{d} v^i \partial_i$  for some  $v^i \in \mathbb{R}$ , suppressing the subscript  $p$  in the basis. We adopt the Einstein summation convention unless otherwise specified, namely  $v^i \partial_i$  denotes  $\sum_{i=1}^{d} v^i \partial_i$ , where common pairs of upper- and lower-indices denote a summation from 1 to  $d$  [see e.g., 18, pp.18-19].
The Riemannian metric  $G$  on a  $d$ -dimensional manifold  $\mathcal{M}$  is a smooth tensor field acting on the tangent vectors. At any  $p \in \mathcal{M}$ ,  $G(p): T_p\mathcal{M} \times T_p\mathcal{M} \to \mathbb{R}$  is a symmetric bi-linear tensor/function satisfying  $G(p)(v, v) \geq 0$  for any  $v \in T_p\mathcal{M}$  and  $G(p)(v, v) = 0$  if and only if  $v = 0$ . On a chart  $\varphi$ , the metric is represented as a  $d$ -by- $d$  positive definite matrix that quantifies the distance traveled along infinitesimal changes in the coordinates. With an abuse of notation, the chart representation of  $G$  is given by the matrix-valued function  $p \mapsto G(p) = [G_{ij}(p)]_{i,j=1}^{d} \in \mathbb{R}^{d \times d}$  for  $p \in \mathcal{M}$ , so the distance traveled by  $\gamma$  at time  $t$  for a duration of  $dt$  is  $[G_{ij}(\gamma(t))\dot{\gamma}^i(t)\dot{\gamma}^j(t)]^{1/2}$ . The intrinsic distance induced by  $G$ , or the geodesic distance, is computed as  $\text{dist}(p,q) = \inf_{\alpha} \int_0^1 \sqrt{\sum_{1 \leq i,j \leq d} G_{ij}(\alpha(t))\dot{\alpha}^i(t)\dot{\alpha}^j(t)} dt$  for two points  $p, q$  on the manifold  $\mathcal{M}$ , where infimum is taken over any curve  $\alpha: [0,1] \to \mathcal{M}$  connecting  $p$  to  $q$ .
A geodesic curve (or simply geodesic) is a smooth curve  $\gamma : \mathbb{R} \supset [a, b] \to \mathcal{M}$  satisfying the geodesic equations, represented on a coordinate chart as
$$
\ddot {\gamma} ^ {k} (t) + \dot {\gamma} ^ {i} (t) \dot {\gamma} ^ {j} (t) \Gamma_ {i j} ^ {k} \circ \gamma (t) = 0, \text {f o r} i, j, k = 1, \dots , d, \tag {2.1}
$$
where over-dots represent derivative w.r.t.  $t$ ;  $\Gamma_{ij}^{k} = \frac{1}{2} G^{kl}\left(\partial_{i}G_{jl} + \partial_{j}G_{il} - \partial_{l}G_{ij}\right)$  are the Christoffel symbols at  $p$ ; and  $G^{kl}$  is the  $(k,l)$ -element of  $G^{-1}$ . Solving (2.1) with initial conditions produces geodesic that traces out the generalization of a straight line on the manifold, preserving travel direction with no acceleration, and is also locally the shortest path.
Considering the shortest path  $\gamma$  connecting  $p$  to  $q$  and applying Taylor's expansion at  $t = 0$ , we obtain  $\text{dist}(p, q)^2 \approx \sum_{1 \leq i,j \leq d} G_{ij}(p)(q^i - p^i)(q^j - p^j)$ , showing the connection between the geodesic distance and the Mahalanobis distance. Our estimation method is inspired by this approximation, and we will discuss the higher-order terms shortly which unveil finer structure of the manifold.
3 Local Regression for Similarity Measurements
3.1 Probabilistic Modeling for Similarity Measurements
Suppose that we observe  $N$  independent triplets  $(Y_{u},X_{u0},X_{u1})$ ,  $u = 1,\dots ,N$ . Here, the  $X_{uj}$  are locations on the manifold identified with their coordinates  $\left(X_{uj}^{1},\ldots ,X_{uj}^{d}\right)\in \mathbb{R}^{d}$ ,  $j = 1,2$ , and  $Y_{u}$  are noisy similarity measures of the proximity of  $(X_{u0},X_{u1})$  in terms of the intrinsic geodesic distance  $dist(\cdot ,\cdot)$  on  $\mathcal{M}$ . To account for different structures of the similarity measures, we model the response in a fashion analogous to generalized linear models. For  $X_{u0},X_{u1}$  lying in a small neighborhood  $\mathcal{U}_p\subset \mathcal{M}$  of a target location  $p\in \mathcal{M}$ , the similarity measure  $Y_{u}$  is modeled as
$$
\mathbb {E} \left(Y _ {u} \mid X _ {u 0}, X _ {u 1}\right) = g ^ {- 1} \left(\operatorname {d i s t} \left(X _ {u 0}, X _ {u 1}\right) ^ {2}\right), \tag {3.1}
$$
where  $g$  is a given link function that relates the conditional expectation to the squared distance.
Example 3.1. We describe below three common scenarios modeled by the general framework (3.1).
1. Continuous response being the squared geodesic distance contaminated with noise:
$$
Y _ {u} = \operatorname {d i s t} \left(X _ {u 0}, X _ {u 1}\right) ^ {2} + \sigma (p) \varepsilon_ {u}, \tag {3.2}
$$
where  $\varepsilon_1, \ldots, \varepsilon_u$  are i.i.d. mean zero random variables, and  $\sigma: \mathcal{M} \to \mathbb{R}^+$  is a smooth positive function determining the magnitude of noise near the target point  $p$ . This model will be applied to model trip time as noisy measure of cost to travel between locations.
2. Binary (dis)similarity response:
$$
\mathbb {P} \left(Y _ {u} = 1 \mid X _ {u 0}, X _ {u 1}\right) = \operatorname {l o g i t} ^ {- 1} \left(\operatorname {d i s t} \left(X _ {u 0}, X _ {u 1}\right) ^ {2} - \hbar (p)\right) \tag {3.3}
$$
for some smooth function  $\hbar : \mathcal{M} \to \mathbb{R}$ , where  $\mathrm{logit}(\mu) = \log (\mu / (1 - \mu))$ ,  $\mu \in (0,1)$  is the logit function. This models the case when there are latent labels for  $X_{uj}$  and  $Y_{u}$  is a measure of whether their labels are in common or not. The function  $\hbar(p)$  in (3.3) describes the homogeneity of the latent labels for points in a small neighborhood of  $p$ . The latent labels could have intrinsic variation even if measures are made for the same data points  $x = X_{u0} = X_{u1}$ , and the strength of which is captured by  $\hbar(p)$ .
3. Binary relative comparison response, where we extend our model for triplets of points  $(X_{u0}, X_{u1}, X_{u2})$ , where  $Y_u$  stands for whether  $X_{u0}$  is more similar to  $X_{u1}$  than to  $X_{u2}$ :
$$
\mathbb {P} \left(Y _ {u} = 1 \mid X _ {u 0}, X _ {u 1}, X _ {u 2}\right) = \log \mathrm {i t} ^ {- 1} \left(\operatorname {d i s t} \left(X _ {u 0}, X _ {u 2}\right) ^ {2} - \operatorname {d i s t} \left(X _ {u 0}, X _ {u 1}\right) ^ {2}\right), \tag {3.4}
$$
so that the relative comparison  $Y_{u}$  reflects the comparison of squared distances.
<sup>1</sup> See Section S3 of the Supplement for details about solving it in practice.
3.2 Local Approximation of Squared Distances
It is the squared distance that determines the responses in our model (3.1), which is justified by the following local approximation. Proposition 3.1 provides a Taylor expansion for the squared geodesic distance between two geodesics with same starting point but different initial velocities (see Figure S5.1 for visualization). It is the key tool to estimate model (3.1) through local regression. Furthermore, (3.5) characterize the error of approximating geodesic distance with Mahalanobis distance. For a point  $p$  on the Riemannian manifold  $\mathcal{M}$ , let  $\exp_p: T_p\mathcal{M} \to \mathcal{M}$  denote the exponential map defined by  $\exp_p(tv) = \gamma(t)$  where  $\gamma$  is a geodesic starting from  $p$  at time 0 with initial velocity  $\gamma'(0) = v \in T_p\mathcal{M}$ . For notational simplicity, we suppress the dependency on  $p$  in geometric quantities (e.g., the metric  $G$  is understood to be evaluated at  $p$ ). For  $i = 1, \dots, d$ , denote  $\delta^i = \delta^i(t) = \gamma^i(t) - \gamma^i(0)$  as the difference in coordinate after a travel of time  $t$  along  $\gamma$ .
Proposition 3.1 (spread of geodesics, coordinated). Let  $p \in \mathcal{M}$  and  $v, w \in T_p\mathcal{M}$  be two tangent vectors at  $p$ . On a local coordinate chart, the squared geodesic distance between two geodesics  $\gamma_0(t) = \exp_p(tv)$  and  $\gamma_1(t) = \exp_p(tw)$  satisfies, as  $t \to 0$ ,
$$
d i s t (\gamma_ {0} (t), \gamma_ {1} (t)) ^ {2} = \delta_ {0 - 1} ^ {i} \delta_ {0 - 1} ^ {j} G _ {i j} + \delta_ {0 - 1} ^ {i} \left(\delta_ {0} ^ {k} \delta_ {0} ^ {l} - \delta_ {1} ^ {k} \delta_ {1} ^ {l}\right) \Gamma_ {k l} ^ {j} G _ {i j} + O (t ^ {4}), \tag {3.5}
$$
where for  $i,j,k,l,m = 1,\dots ,d$
-  $\delta_0^i = \gamma_0^i (t) - p^i$ ,  $\delta_1^i = \gamma_1^i (t) - p^i$ , and  $\delta_{0 - 1}^i = \delta_0^i -\delta_1^i$ , i.e.,  $\delta_0^i$ ,  $\delta_1^i$  are differences in  $i$ -th coordinates of  $\gamma_0(t)$  and  $\gamma_{1}(t)$  to the origin  $p$ , respectively, and  $\delta_{0 - 1}^i = \delta_0^i -\delta_1^i$  is the coordinate difference between  $\gamma_0(t)$  and  $\gamma_{1}(t)$ ;
-  $G_{ij}$  and  $\Gamma_{kl}^{j}$  are the elements of the metric and Christoffel symbols at  $p$ , respectively.
Proof. See Section S5 in the Supplement.
To the RHS of (3.5), the first term is the quadratic term in distance metric learning. The second term is the result of coordinate representation of geodesics. It vanishes under the normal coordinate where the Christoffel symbols are zero. It inspires the use of local regression to estimate the metric tensor and the Christoffel symbols. For  $X_{u0}, X_{u1}$  in a small neighborhood of  $p$ , write the linear predictor as
$$
\eta_ {u} := \beta^ {(0)} + \delta_ {u, 0 - 1} ^ {i} \delta_ {u, 0 - 1} ^ {j} \beta_ {i j} ^ {(1)} + \delta_ {u, 0 - 1} ^ {k} \left(\delta_ {u 0} ^ {i} \delta_ {u 0} ^ {j} - \delta_ {u 1} ^ {i} \delta_ {u 1} ^ {j}\right) \beta_ {i j k} ^ {(2)}, \tag {3.6}
$$
a function of the intercept  $\beta^{(0)}$  and coefficients  $\beta_{ij}^{(1)},\beta_{ijk}^{(2)}$ , where  $\delta_{u0}^i = X_{u0}^i -p^i$ ,  $\delta_{u1}^i = X_{u1}^i -p^i$ , and  $\delta_{u,0 - 1}^i = \delta_{u0}^i -\delta_{u1}^i$ , for  $i,j,k,l = 1,\ldots ,d$ , and  $u = 1,\dots ,N$ . The intercept term  $\beta^{(0)}$  is included for capturing intrinsic variation (e.g.,  $\hbar(p)$  in (3.3)) and can otherwise be dropped. The link function connects the linear predictor to the conditional mean via  $\mu_u\coloneqq g^{-1}(\eta_u)\approx \mathbb{E}\left(Y_u|X_{u0},X_{u1}\right)$  as indicated by (3.1) and (3.5), where  $\mu_{u}$  is seen as a function of the coefficients  $\beta^{(0)},\beta_{ij}^{(1)}$ , and  $\beta_{ijk}^{(2)}$ . Therefore, upon the specification of a loss function  $Q:\mathbb{R}\times \mathbb{R}\to \{0\} \cup \mathbb{R}^{+}$  and non-negative weights  $w_{1},\ldots ,w_{N}$ , the minimizers
$$
\left(\hat {\beta} ^ {(0)}, \hat {\beta} _ {i j} ^ {(1)}, \hat {\beta} _ {i j k} ^ {(2)}\right) = \underset {\beta^ {(0)}, \beta_ {i j} ^ {(1)}, \beta_ {i j k} ^ {(2)}; i, j, k} {\arg \min } \sum_ {u = 1} ^ {N} Q \left(Y _ {u}, \mu_ {u}\right) w _ {u}, \tag {3.7}
$$
$$
\beta_ {i j} ^ {(1)} = \beta_ {j i} ^ {(1)}, \beta_ {i j k} ^ {(2)} = \beta_ {j i k} ^ {(2)}, \text {f o r} i, j, k, l = 1, \dots , d, \tag {3.8}
$$
are used to estimate the metric tensor and Christoffel symbols, obtaining
$$
\hat {G} _ {i j} = \hat {\beta} _ {i j} ^ {(1)}, \quad \hat {\Gamma} _ {i j} ^ {l} = \hat {\beta} _ {i j k} ^ {(2)} \hat {G} ^ {k l}, \tag {3.9}
$$
where  $\hat{G}^{kl}$  is the matrix inverse of  $\hat{G}$  satisfying  $\hat{G}^{kl}\hat{G}_{kj} = 1_{\{j = l\}}$ . The symmetry constraints (3.8) are the result of the symmetries in the metric tensor and Christoffel symbols, and are enforced by optimizing over only the lower triangular indices  $1\leq i < j\leq d$  without constraints. Asymptotic
<sup>2</sup>See e.g., [19] pp. 131–133 for normal coordinate. See [20] and Proposition S5.1 of the Supplement for coordinate-invariant version of Proposition 3.1.
results provide the positive-definiteness of the metric estimate, as will be shown in Proposition 4.1. To weigh the pairs of endpoints according to their proximity to the target location  $p$ , we apply kernel weights specified by
$$
w _ {u} = h ^ {- 2 d} \prod_ {i = 1} ^ {d} K \left(\frac {X _ {u 0} ^ {i} - p ^ {i}}{h}\right) K \left(\frac {X _ {u 1} ^ {i} - p ^ {i}}{h}\right) \tag {3.10}
$$
for some  $h > 0$  and non-negative kernel function  $K$ . The bandwidth  $h$  controls the bias-variance trade-off of the estimated Riemannian metric tensor and its derivatives.
Altering the link function  $g$  and the loss function  $Q$  in (3.7) enables flexible local regression estimation for models in Example 3.1.
Example 3.2. Consider the following loss functions for estimating the metric tensors and the Christoffel symbols when data are drawn from model (3.2)-(3.4), respectively.
1. Continuous noisy response: use squared loss  $Q(y, \mu) = (y - \mu)^2$  with  $g$  being the identity link function so  $\mu_u = \eta_u$ .
2. Binary (dis)similarity response: use log-likelihood of Bernoulli random variable
$$
Q (y, \mu) = y \log \mu + (1 - y) \log (1 - \mu), \tag {3.11}
$$
and  $g$  the logit link, so  $\mu_u = \mathrm{logit}^{-1}(\eta_u)$ . The model becomes a local logistic regression.
3. Binary relative comparison response: apply the same loss function (3.11) and logit link as in the previous scenario, but here we formulate the linear predictor based on  $\text{dist}(X_{u0}, X_{u2})^2 - \text{dist}(X_{u0}, X_{u1})^2 \approx \eta_{u1} - \eta_{u2}$  and
$$
\mu_ {u} = g ^ {- 1} \left(\eta_ {u 1} - \eta_ {u 2}\right). \tag {3.12}
$$
Locally, the difference in squared distances is approximated by
$$
\begin{array}{l} \eta_ {u 1} - \eta_ {u 2} = \left(\delta_ {u, 0 - 1} ^ {i} \delta_ {u, 0 - 1} ^ {j} - \delta_ {u, 0 - 2} ^ {i} \delta_ {u, 0 - 2} ^ {j}\right) \beta_ {i j} ^ {(1)} + \left(\delta_ {u, 0 - 1} ^ {k} \left(\delta_ {u 0} ^ {i} \delta_ {u 0} ^ {j} - \delta_ {u 1} ^ {i} \delta_ {u 1} ^ {j}\right)\right) \beta_ {i j k} ^ {(2)} \\ - \left(\delta_ {u, 0 - 2} ^ {k} \left(\delta_ {u 0} ^ {i} \delta_ {u 0} ^ {j} - \delta_ {u 2} ^ {i} \delta_ {u 2} ^ {j}\right)\right) \beta_ {i j k} ^ {(2)}, \tag {3.13} \\ \end{array}
$$
for  $\delta_{u2}^{i} = X_{u2}^{i} - p^{i}$  and  $\delta_{u,0-2}^{i} = \delta_{u2}^{i} - \delta_{u0}^{i}, i = 1, \ldots, d$ . Here  $\eta_{u1}$  and  $\eta_{u2}$  are constructed in analogy to (3.6) using  $(X_{u0}, X_{u1})$  and  $(X_{u0}, X_{u2})$  pair respectively.
Examples in Section 5 will further illustrate the proposed method in those scenarios. Besides the models listed, other choices for the link  $g$  and loss function  $Q$  can also be considered under this local regression framework [8], accommodating a wide variety of data. To efficiently estimate the metric on the entire manifold  $\mathcal{M}$ , we apply a procedure based on discretization and post-smoothing, as detailed in Section S3 of the Supplement. In short, kernel smoothing (weighted average) of estimated component functions  $\hat{G}_{ij}$  over some grid points provides a smooth tensor field, which ease the burden of re-estimation for every points on the manifold.
4 Bias and Variance of the Estimated Metric Tensor
This subsection provides asymptotic justification for model (3.2) with  $\mathbb{E}\left(Y_u|X_{u0},X_{u1}\right) = dist\left(X_{u0},X_{u1}\right)^2$  under the squared loss  $Q(\mu ,y) = (\mu -y)^2$  and the identity link  $g(\mu) = \mu$ . The estimator we analyzed here fits a local quadratic regression without intercept and approximates the squared distance by a simplified form of (3.6):
$$
\operatorname {d i s t} \left(X _ {u 0}, X _ {u 1}\right) ^ {2} \approx \eta_ {u} := \delta_ {u, 0 - 1} ^ {i} \delta_ {u, 0 - 1} ^ {j} \beta_ {i j} ^ {(1)}, \tag {4.1}
$$
for  $u = 1, \ldots, N$ . Given a suitable order of the indices  $i, j$  for vectorization, we rewrite the formulation into a matrix form. Denote the local design matrix and regression coefficients as
$$
\begin{array}{l} \mathbf {D} _ {u} = \left(\delta_ {u, 0 - 1} ^ {1} \delta_ {u, 0 - 1} ^ {1}, \dots , \delta_ {u, 0 - 1} ^ {i} \delta_ {u, 0 - 1} ^ {j}, \dots , \delta_ {u, 0 - 1} ^ {d} \delta_ {u, 0 - 1} ^ {d}\right) ^ {T}, \\ \boldsymbol {\beta} = \left(\beta_ {1 1} ^ {(1)}, \dots , \beta_ {i j} ^ {(1)}, \dots , \beta_ {d d} ^ {(1)}\right) ^ {T}, \\ \end{array}
$$
so that the linear predictor  $\eta_u = \mathbf{D}_u^T\beta$ . Further, write  $\mathbf{D} = (\mathbf{D}_1,\dots ,\mathbf{D}_N)^T$ ,  $\mathbf{Y} = (Y_{1},\ldots ,Y_{N})^{T}$  and  $\mathbf{W} = \mathrm{diag}(w_1,\ldots ,w_N)$ , with weights  $w_{u}$  specified in (3.10). The objective function in (3.7) becomes  $(\mathbf{Y} - \mathbf{D}\boldsymbol {\beta})^T\mathbf{W}(\mathbf{Y} - \mathbf{D}\boldsymbol {\beta})$ , and the minimizer is  $\hat{\beta} = \left(\mathbf{D}^T\mathbf{W}\mathbf{D}\right)^{-1}\mathbf{D}^T\mathbf{W}\mathbf{Y}$ , for which we will analyze the bias and variance.
To characterize the asymptotic bias and variance of the estimator, we assume the following conditions are satisfied in a neighborhood of the target  $p$ . These conditions are standard and analogous to those assumed in a local regression setting [8].
(A1) The joint density of endpoints  $(X_{u0}, X_{u1})$  is positive and continuously differentiable.
(A2) The functions  $G_{ij}, \Gamma_{ij}^k$  are  $C^2$ -smooth for  $i, j, k = 1, \ldots, d$ .
(A3) The kernel  $K$  in weights (3.10) is symmetric, continuous, and has a bounded support.
(A4)  $\sup_u\operatorname {var}\left(Y_u|X_{u0},X_{u1}\right) <   \infty .$