Local Fréchet regression with spherical predictors

Chang Jun Im¹ and Jeong Min Jeon²

¹Department of Statistics, Seoul National University, icj803@snu.ac.kr

²Department of Data Science, Ewha Womans University,
jeongmin.jeon@ewha.ac.kr

September 10, 2024

# 1 The directional kernel

In this section, we prove some properties about the directional kernel $L$. The idea of the directional kernel was used in Im et al. (2024), Marco Di Marzio and Taylor (2014), García-Portugués et al. (2013), Hall et al. (1987), and Bai et al. (1988). First, we impose a mild condition about $L$, which was used in Im et al. (2024). Note that the condition L1 implies that $L$ is Legesgue integrable and bounded on $[0, \infty)$. From now on, denote $\mathbb{N}_0$ as $\mathbb{N} \cup \{0\}$.

**Condition L1.** The directional kernel $L: [0, \infty) \to [0, \infty)$ is a non-increasing function. Also, for any given $j \in \mathbb{N}_0$ and $k = 1, 2$, the following holds.

$$0 < \int_{0}^{\infty} L^{k}(r^{2})r^{j}dr < \infty$$

Define a constant $a_{j,k}(L) := \int_{0}^{\infty} L^{k}(r^{2})r^{j}dr$ for every $j \in \mathbb{N}_0$ and $k = 1, 2$.

**Remark 1.1.** There are several examples of the directional kernel satisfying L1.

* Suppose $L(r^2) = e^{-r^2}$ for every $r \ge 0$, which is known as von-Mises kernel. Then $a_{2j,1}(L) = (2j - 1)!!/2^{j+1}\sqrt{\pi}$, $a_{2j,2}(L) = (2j - 1)!!/2^{2j+1}\sqrt{\pi}/2$, $a_{2j+1,1}(L) = j!/2$, $a_{2j+1,2}(L) = j!/2^{j+2}$ for every $j \in \mathbb{N}_0$, where !! denotes the double factorial.

* Suppose $L(r^2) = e^{-r}$ for every $r \ge 0$, which is known as the exponential kernel. Then $a_{j,1}(L) = j!$ and $a_{j,2}(L) = j!/2^{j+1}$ for every $j \in \mathbb{N}_0$.
• Suppose $L(r^2) = 1_{[0,1]}(r)$ for every $r \ge 0$, which is known as the uniform kernel. Then $a_{j,1}(L) = a_{j,2}(L) = (j+1)^{-1}$ for every $j \in \mathbb{N}_0$.

Now we state a Lemma about the major property of $L$. This is an extension of Lemma 1.1 of Im et al. (2024) and Lemma 1 of García-Portugués et al. (2013).

**Lemma 1.1.** Suppose the condition L1 holds. For any given $h > 0$, $j, l \in \mathbb{N}_0$, and $k = 1, 2$, define $\lambda_{h,j,k,l}(L)$ and $\lambda_{j,k,l}(L)$ as below.

$$ \lambda_{h,j,k,l}(L) := \int_{0}^{\frac{\sqrt{2}}{h}} L^k(r^2) r^j (2 - h^2 r^2)^{\frac{l-1}{2}} dr \quad (1.1) $$

$$ \lambda_{j,k,l}(L) := 2^{\frac{l-1}{2}} \int_{0}^{\infty} L^k(r^2) r^j dr = 2^{\frac{l-1}{2}} a_{j,k}(L) \quad (1.2) $$

Then $\lim_{h \downarrow 0} \lambda_{h,j,k,l}(L) = \lambda_{j,k,l}(L)$.

*Proof of Lemma 1.1.* Let $j, l \in \mathbb{N}_0$, and $k = 1, 2$ be given. Define functions $\phi_{h,j,k,l}(r) : [0, \infty) \to [0, \infty)$, $\tilde{\phi}_{h,j,k,l}(r) : [0, \infty) \to [0, \infty)$, and $\phi_{j,k,l}(r) : [0, \infty) \to [0, \infty)$ as below for every $h > 0$.

$$ \phi_{h,j,k,l}(r) := L^k(r^2)r^j(2 - h^2r^2)^{\frac{l-1}{2}}1_{[0, \frac{\sqrt{2}}{h})}(r), \quad \forall r \in [0, \infty) $$

$$ \tilde{\phi}_{h,j,k,l}(r) := L^k(r^2)r^j(2 - h^2r^2)^{\frac{l-1}{2}}1_{[0, \frac{1}{h})}(r), \quad \forall r \in [0, \infty) $$

$$ \phi_{j,k,l}(r) := 2^{\frac{l-1}{2}} L^k(r^2) r^j 1_{[0, \infty)}(r), \quad \forall r \in [0, \infty) $$

To prove Lemma 1.1, we consider two cases.

(Case 1: $l = 0$) Note that $\lim_{h \downarrow 0} \phi_{h,j,k,0} = \lim_{h \downarrow 0} \tilde{\phi}_{h,j,k,0} = \phi_{j,k,0}$ pointwise, and $\phi_{h,j,k,0}$ can be decomposed into $\tilde{\phi}_{h,j,k,0}$ and $\phi_{h,j,k,0} - \tilde{\phi}_{h,j,k,0}$ for every $h > 0$. Since $2 - h^2r^2 \ge 1$ for every $h > 0$ and $r \in [0, 1/h)$, the following holds for every $h > 0$.

$$ 0 \le \tilde{\phi}_{h,j,k,0}(r) \le L^k(r^2)r^j 1_{[0, \frac{1}{h})}(r) \le \sqrt{2}\phi_{j,k,0}(r), \quad \forall r \in [0, \infty) $$

Note that $\phi_{j,k,0}$ is integrable on $[0, \infty)$, which implies the following by DCT.

$$ \lim_{h \downarrow 0} \int_{0}^{\infty} \tilde{\phi}_{h,j,k,0}(r) dr = \int_{0}^{\infty} \phi_{j,k,0}(r) dr = \lambda_{j,k,0}(L) $$

Now it suffices to show the following.

$$ \lim_{h \downarrow 0} \int_{0}^{\infty} \left( \phi_{h,j,k,0}(r) - \tilde{\phi}_{h,j,k,0}(r) \right) dr = 0 $$
Note that the following holds for every $h > 0$.

$$
\begin{aligned}
0 &\le \int_0^\infty \left( \phi_{h,j,k,0}(r) - \tilde{\phi}_{h,j,k,0}(r) \right) dr = \int_{\frac{1}{h}}^{\frac{\sqrt{2}}{h}} L^k(r^2) r^j (2 - h^2 r^2)^{-\frac{1}{2}} dr \\
&\le \int_{\frac{1}{h}}^{\frac{\sqrt{2}}{h}} L^k(r^2) h^2 r^{j+2} (2 - h^2 r^2)^{-\frac{1}{2}} dr \\
&\le \sup_{r \ge \frac{1}{h}} \left( L^k(r^2) r^{j+1} \right) \int_{\frac{1}{h}}^{\frac{\sqrt{2}}{h}} h^2 r (2 - h^2 r^2)^{-\frac{1}{2}} dr = \sup_{r \ge \frac{1}{h}} \left( L^k(r^2) r^{j+1} \right)
\end{aligned}
$$

Also, the following holds by L1.

$$0 \le \sum_{n=1}^{\infty} L^k(n^2)(n-1)^{j+1} \le \sum_{n=1}^{\infty} \left( \int_{n-1}^{n} L^k(r^2)r^{j+1}dr \right) = \int_{0}^{\infty} L^k(r^2)r^{j+1}dr < \infty$$

Therefore, $\lim_{n \to \infty} L^k(n^2)(n-1)^{j+1} = 0$, which implies $\lim_{r \to \infty} L^k(r^2)r^{j+1} = 0$ by L1. By summing these up, we get the following.

$$0 \le \lim_{h \downarrow 0} \int_0^\infty \left( \phi_{h,j,k,0}(r) - \tilde{\phi}_{h,j,k,0}(r) \right) dr \le \lim_{h \downarrow 0} \left( \sup_{r \ge \frac{1}{h}} L^k(r^2) r^{j+1} \right) = 0$$

(Case 2: $l \ge 1$) Note that $\lim_{h \downarrow 0} \phi_{h,j,k,l} = \phi_{j,k,l}$ pointwise. Also, $l \ge 1$ implies $0 \le \phi_{h,j,k,l} \le \phi_{j,k,l}$ for every $h > 0$. Since $\phi_{j,k,l}$ is integrable on $[0, \infty)$, the following holds by DCT.

$$\lim_{h \downarrow 0} \lambda_{h,j,k,l}(L) = \lim_{h \downarrow 0} \int_{0}^{\infty} \phi_{h,j,k,l}(r) dr = \int_{0}^{\infty} \phi_{j,k,l}(r) dr = \lambda_{j,k,l}(L)$$

Considering two cases above, Lemma 1.1 is proved. $\square$

Now denote $S^{d-1} = \{x \in \mathbb{R}^d : \|x\|_2 = 1\}$ as the sphere on a $d$-dimensional plane, where $\| \cdot \|_2$ denotes the Euclidean norm. Denote $\omega_{d-1}$ as the Lebesgue measure in $S^{d-1}$. The next part of this section focuses on computing the following integral inductively for any given real-valued Lebesgue integrable function $h$.

$$\int_{S^{d-1}} h(z) \omega_{d-1}(dz)$$

The main idea is to consider an orthonormal basis of $\mathbb{R}^d$ containing $x$ for any given $x \in S^{d-1}$. Let $x \in S^{d-1}$ be given. By Gram-Schmidt process, we can extend $\{x\}$ to an orthonormal basis $\{x, b_1, \dots, b_{d-1}\}$ of $\mathbb{R}^d$ such that $\det([x, b_1, b_2, \dots, b_{d-1}]) = 1$ and $b_i \in S^{d-1}$ for every $i = 1, 2, \dots, d-1$. By defining $B_x := [b_1, b_2, \dots, b_{d-1}] \in \mathbb{R}^{d \times (d-1)}$, the following holds, where $0_n$ denotes the zero vector of $\mathbb{R}^n$ and $I_n$ denotes the identity matrix of $\mathbb{R}^{n \times n}$ for every $n \in \mathbb{N}$.

$$B'_x B_x = I_{d-1}, \quad B'_x x = 0_{d-1}, \quad B_x B'_x = I_d - xx', \quad \det([x, B_x]) = 1 \quad (1.3)$$
Using this idea, we can define $\mathcal{B}_x$ as the following for every $x \in \mathbb{S}^{d-1}$.

$$ \mathcal{B}_x := \{ \mathcal{M} \in \mathbb{R}^{d \times (d-1)} : \mathcal{M}'\mathcal{M} = \mathbf{I}_{d-1}, \mathcal{M}'\mathbf{x} = \mathbf{0}_{d-1}, \det([\mathbf{x}, \mathcal{M}]) = 1 \} $$

Now suppose $x \in \mathbb{S}^{d-1}$ and $\mathcal{B}_x \in \mathcal{B}_x$ be given. Note that $\mathcal{B}_x\mathcal{B}_x' = \mathbf{I}_d - xx'$ holds. By the orthonormal basis representation, there exists a decomposition for any given $z \in \mathbb{S}^{d-1}$ as the following.

$$ z = x \cos \theta + (\mathcal{B}_x \xi) \sin \theta, \quad \exists \theta \in [0, \pi], \quad \exists \xi \in \mathbb{S}^{d-2} $$

This idea is a modification of the tangent-normal decomposition, which was used in Marco Di Marzio and Taylor (2014), Marco Di Marzio and Taylor (2019), and Di Marzio et al. (2019). Using the idea above, we can get the following Lemmas, which are direct consequences of Lemma 2 and Lemma 3 of García-Portugués et al. (2013).

**Lemma 1.2.** Let $h$ be a real-valued Lebesgue integrable function on $\mathbb{S}^{d-1}$. Then the following holds for every $x \in \mathbb{S}^{d-1}$.

$$ \int_{\mathbb{S}^{d-1}} h(z) \omega_{d-1}(dz) = \int_0^\pi \int_{\mathbb{S}^{d-2}} h(x \cos \theta + (\mathcal{B}_x \xi) \sin \theta) \omega_{d-2}(d\xi) \sin^{d-2}(\theta) d\theta \quad (1.4) $$

**Remark 1.2.** Putting $d = n+1$ and $h(z) = 1$ for every $n \in \mathbb{N}$ and $z \in \mathbb{S}^n$, we can get a recursive formula for the surface area of the unit $n$-sphere, which is denoted as $\mathcal{A}_n$.

$$ \mathcal{A}_n = \mathcal{A}_{n-1} \int_0^\pi \sin^{n-1}(\theta) d\theta, \quad \mathcal{A}_0 = 2, \quad \forall n \in \mathbb{N} $$

By solving the above, we define a constant $\mathcal{A}_n$ as the following for every $n \in \mathbb{N}_0$.

$$ \mathcal{A}_n := \frac{2\pi^{\frac{n+1}{2}}}{\Gamma(\frac{n+1}{2})} = \begin{cases} 2\pi^{\frac{n+1}{2}}/(\frac{n-1}{2})! & (n \text{ is odd}) \\ 2(2\pi)^{\frac{n}{2}}/(n-1)!! & (n \text{ is even}) \end{cases} \quad (1.5) $$

**Remark 1.3.** Let $x \in \mathbb{S}^{d-1}$ and $\mathcal{B}_x \in \mathcal{B}_x$ be given. For the convenience, define $\Phi_{\mathcal{B}_x} : [0, \pi] \times \mathbb{S}^{d-2} \to \mathbb{S}^{d-1}$ as the following.

$$ \Phi_{\mathcal{B}_x}(\theta, \xi) := x \cos \theta + (\mathcal{B}_x \xi) \sin \theta, \quad \forall \theta \in [0, \pi], \quad \forall \xi \in \mathbb{S}^{d-2} \quad (1.6) $$

Then (1.4) is equivalent to the following.

$$ \int_{\mathbb{S}^{d-1}} h(z) \omega_{d-1}(dz) = \int_0^\pi \int_{\mathbb{S}^{d-2}} h(\Phi_{\mathcal{B}_x}(\theta, \xi)) \omega_{d-2}(d\xi) \sin^{d-2}(\theta) d\theta $$
**Lemma 1.3.** Let $n \in \mathbb{N}$ be given. For any $\xi \in \mathbb{R}^n$, denote $\xi_i$ as the i-th component of $\xi$ for every $i = 1, 2, \dots, n$. Then the following holds, where $\mathcal{A}_{n-1}$ is defined as (1.5).

$$ \int_{\mathbb{S}^{n-1}} \xi_i \omega_{n-1}(d\xi) = 0, \quad \int_{\mathbb{S}^{n-1}} \xi_i \xi_j \omega_{n-1}(d\xi) = \begin{cases} 0 & (i \neq j) \\ n^{-1} \mathcal{A}_{n-1} & (i = j) \end{cases}, \quad \forall i, j = 1, 2, \dots, n $$

Therefore, for every $v \in \mathbb{R}^n$ and $M \in \mathbb{R}^{n \times n}$, the following holds.

$$ \int_{\mathbb{S}^{n-1}} \xi' v \omega_{n-1}(d\xi) = 0, \quad \int_{\mathbb{S}^{n-1}} \xi' M \xi \omega_{n-1}(d\xi) = n^{-1} \mathcal{A}_{n-1} \mathrm{tr}(M) $$

Finally, we define a constant $c_{h,j,k}(L)$ for every $h > 0$, $j \in \mathbb{N}_0$, and $k = 1, 2$ as the following, where $\mathcal{A}_{d-2}$ is defined by (1.5).

$$ c_{h,j,k}(L) := \mathcal{A}_{d-2} \int_{0}^{\pi} L^k \left( \frac{1 - \cos \theta}{h^2} \right) \theta^j \sin^{d-2}(\theta) d\theta \quad (1.7) $$

The following Lemma states about the asymptotic property of $c_{h,j,k}(L)$ for any given $j \in \mathbb{N}_0$ and $k = 1, 2$ as $h \downarrow 0$, which is an extension of Lemma 1.2 of Im et al. (2024).

**Lemma 1.4.** Suppose the condition L1 holds. Then the following holds.

$$ \lim_{h \downarrow 0} \frac{c_{h,j,k}(L)}{h^{j+d-1}} = 2^{\frac{j+d-1}{2}} \mathcal{A}_{d-2} a_{j+d-2,k}(L), \quad \forall j \in \mathbb{N}_0, \forall k = 1, 2 $$

*Proof of Lemma 1.4.* Note that the following holds by Lemma 1.2 of Im et al. (2024).

$$ 0 \le \frac{x^2}{2} - (1 - \cos x) \le (1 - \cos x)^2, \quad \forall x \in [-\pi, \pi] \quad (1.8) $$

Similar to Lemma 1.2 of Im et al. (2024), we get the following by applying (1.8).

$$ 2^j (1 - \cos \theta)^j \le \theta^{2j} \le 2^j (1 - \cos \theta)^j + 6^j (1 - \cos \theta)^{j+1}, \quad \forall \theta \in [0, \pi], \forall j \in \mathbb{N}_0 \quad (1.9) $$

Note that (1.9) implies the following.

$$ 2^{\frac{j}{2}} (1 - \cos \theta)^{\frac{j}{2}} \le \theta^j \le 2^{\frac{j}{2}} (1 - \cos \theta)^{\frac{j}{2}} + 6^{\frac{j}{2}} (1 - \cos \theta)^{\frac{j+1}{2}}, \quad \forall \theta \in [0, \pi], \forall j \in \mathbb{N}_0 \quad (1.10) $$

Also, note that the following holds for every $h > 0$, $j \in \mathbb{N}_0$, and $k = 1, 2$, where $\lambda_{h,j+d-2,k,d-2}(L)$ is defined by (1.1).

$$ \begin{aligned} & \int_{0}^{\pi} L^k \left( \frac{1 - \cos \theta}{h^2} \right) (1 - \cos \theta)^{\frac{j}{2}} \sin^{d-2}(\theta) d\theta \\ = & 2h^{j+d-1} \int_{0}^{\frac{\sqrt{2}}{h}} L^k(r^2) r^{j+d-2} (2 - h^2 r^2)^{\frac{d-3}{2}} dr = 2h^{j+d-1} \lambda_{h,j+d-2,k,d-2}(L) \end{aligned} \quad (1.11) $$
Combining (1.10) and (1.11) with Lemma 1.1, the following holds for every $j \in \mathbb{N}_0$ and $k = 1, 2$.

$$ 
\begin{aligned}
\liminf_{h \downarrow 0} \frac{c_{h,j,k}(L)}{h^{j+d-1}} &\ge \liminf_{h \downarrow 0} \frac{2^{j/2} \mathcal{A}_{d-2}}{h^{j+d-1}} \int_0^\pi L^k \left( \frac{1-\cos\theta}{h^2} \right) (1-\cos\theta)^{\frac{j}{2}} \sin^{d-2}(\theta) d\theta \\
&= \liminf_{h \downarrow 0} 2^{\frac{j+2}{2}} \mathcal{A}_{d-2} \lambda_{h,j+d-2,k,d-2}(L) = 2^{\frac{j+d-1}{2}} \mathcal{A}_{d-2} a_{j+d-2,k}(L)
\end{aligned}
$$

$$ 
\begin{aligned}
\limsup_{h \downarrow 0} \frac{c_{h,j,k}(L)}{h^{j+d-1}} &\le \limsup_{h \downarrow 0} \frac{2^{j/2} \mathcal{A}_{d-2}}{h^{j+d-1}} \int_0^\pi L^k \left( \frac{1-\cos\theta}{h^2} \right) (1-\cos\theta)^{\frac{j}{2}} \sin^{d-2}(\theta) d\theta \\
&\quad + \limsup_{h \downarrow 0} \frac{6^{j/2} \mathcal{A}_{d-2}}{h^{j+d-1}} \int_0^\pi L^k \left( \frac{1-\cos\theta}{h^2} \right) (1-\cos\theta)^{\frac{j+1}{2}} \sin^{d-2}(\theta) d\theta \\
&= \limsup_{h \downarrow 0} 2^{\frac{j+2}{2}} \mathcal{A}_{d-2} \lambda_{h,j+d-2,k,d-2}(L) \\
&\quad + \limsup_{h \downarrow 0} 2 \cdot 6^{\frac{j}{2}} \mathcal{A}_{d-2} \lambda_{h,j+d-1,k,d-2}(L) h = 2^{\frac{j+d-1}{2}} \mathcal{A}_{d-2} a_{j+d-2,k}(L)
\end{aligned}
$$

Therefore, the proof is done. □

## 2 The kernel density estimator for spherical data

In this section, we define the kernel density estimator for spherical data and prove some properties. Let $\mathbf{X}$ be a random variable on $\mathbb{S}^{d-1}$ with the density function $f$. Also, let $n \in \mathbb{N}$ and random samples $\mathbf{X}_1, \mathbf{X}_2, \dots, \mathbf{X}_n$ of $\mathbf{X}$ be given. The kernel density estimator $\hat{f}_h$ for spherical variable $\mathbf{X}$ is defined by the following, where $L$ is the directional kernel, $h = h_n > 0$ is the bandwidth parameter depending on $n$, and $c_{h,0,1}(L)$ is the normalizing constant defined by (1.7).

$$ \hat{f}_h(\boldsymbol{x}) := \frac{1}{nc_{h,0,1}(L)} \sum_{i=1}^{n} L\left(\frac{1 - \boldsymbol{X}'_i \boldsymbol{x}}{h^2}\right), \quad \forall h > 0, \forall \boldsymbol{x} \in \mathbb{S}^{d-1} \quad (2.1) $$

This kernel density estimator is an extension of the kernel density estimator defined on Im et al. (2024). Similarly, we introduce some conditions with respect to the density function $f$.

**Condition D1.** There exists a homogeneous extension $\bar{f}$ of $f$ from $\mathbb{S}^{d-1}$ to $\mathbb{R}^d \setminus \{\mathbf{0}_d\}$ defined by the following, where $\mathbf{0}_d$ denotes the zero vector of $\mathbb{R}^d$.

$$ \bar{f}(\boldsymbol{x}) := f\left(\frac{\boldsymbol{x}}{\|\boldsymbol{x}\|_2}\right), \quad \forall \boldsymbol{x} \in \mathbb{R}^d \setminus \{\mathbf{0}_d\} $$

In addition, assume that the gradient vector $\nabla \bar{f}$ and the Hessian matrix $\nabla^2 \bar{f}$ both exist and are continuous on $\mathbb{R}^d \setminus \{\mathbf{0}_d\}$.

**Condition D2.** The density function $f$ is strictly positive.
The condition D1 and D2 are similarly used in Im et al. (2024), Marco Di Marzio and Taylor (2014), Marco Di Marzio and Taylor (2019), and Di Marzio et al. (2019). Since $\bar{f}$ a homogeneous extension of $f$, the following holds.

$$z'\nabla_{\bar{f}}(z) = 0, \quad z'\nabla_{\bar{f}}^2(z)z = 0, \quad \forall z \in \mathbb{S}^{d-1} \qquad (2.2)$$

In addition, the condition D1 and the compactness of $\mathbb{S}^{d-1}$ imply the following, where $\|M\|_2$ denotes the spectral norm of $M$ for every $M \in \mathbb{R}^{m \times p}$.

$$\sup_{z \in \mathbb{S}^{d-1}} |f(z)| < \infty, \quad \sup_{z \in \mathbb{S}^{d-1}} \|\nabla_{\bar{f}}(z)\|_2 < \infty, \quad \sup_{z \in \mathbb{S}^{d-1}} \|\nabla_{\bar{f}}^2(z)\|_2 < \infty \qquad (2.3)$$

Also, we impose a mild condition of the bandwidth parameter $h = h_n > 0$, which is very similar to the real data case.

**Condition B1.** The bandwidth parameter $h = h_n > 0$ is a sequence of positive numbers such that $\lim_{n \to \infty} h_n = 0$ and $\lim_{n \to \infty} nh_n^{d-1} = \infty$.

The following states about the approximation of $f(z)$ for every $z \in \mathbb{S}^{d-1}$.

**Lemma 2.1.** Suppose the condition D1 holds. Let $x \in \mathbb{S}^{d-1}$ and $B_x \in \mathcal{B}_x$ be given. Then for any given $\theta \in [0, \pi]$ and $\xi \in \mathbb{S}^{d-2}$, there exists $\tilde{\theta}_\xi \in [0, \theta]$ depending on $\xi$ such that the following holds, where $\Phi_{B_x}$ is defined by (1.6).

$$f(\Phi_{B_x}(\theta, \xi)) = f(x) + \theta(B_x\xi)'\nabla_{\bar{f}}(x) + \frac{\theta^2}{2}\left(x'\nabla_{\bar{f}}^2(\Phi_{B_x}(\tilde{\theta}_\xi, \xi))x + (B_x\xi)'\nabla_{\bar{f}}^2(\Phi_{B_x}(\tilde{\theta}_\xi, \xi))(B_x\xi)\right)$$

*Proof of Lemma 2.1.* Let $x \in \mathbb{S}^{d-1}$ and $B_x \in \mathcal{B}_x$ be given. Similar to Lemma 2.1 of Im et al. (2024), the following hold by the condition D1 and (2.2) for every $\xi \in \mathbb{S}^{d-2}$.

$$\bar{f}(\Phi_{B_x}(\theta, \xi))\Big|_{\theta=0} = f(x), \quad \left. \frac{\partial}{\partial \theta} \bar{f}(\Phi_{B_x}(\theta, \xi)) \right|_{\theta=0} = (B_x\xi)'\nabla_{\bar{f}}(x) \qquad (2.4)$$

$$\Phi_{B_x}(\theta, \xi)'\nabla_{\bar{f}}(\Phi_{B_x}(\theta, \xi)) = 0, \quad \Phi_{B_x}(\theta, \xi)'\nabla_{\bar{f}}^2(\Phi_{B_x}(\theta, \xi))\Phi_{B_x}(\theta, \xi) = 0, \quad \forall \theta \in [0, \pi] \qquad (2.5)$$

$$\frac{\partial^2}{\partial^2 \theta} \bar{f}(\Phi_{B_x}(\theta, \xi)) = \Phi_{B_x}\left(\theta + \frac{\pi}{2}, \xi\right)' \nabla_{\bar{f}}^2(\Phi_{B_x}(\theta, \xi))\Phi_{B_x}\left(\theta + \frac{\pi}{2}, \xi\right), \quad \forall \theta \in [0, \pi] \qquad (2.6)$$

Combining (2.5) and (2.6), we get the following for every $\theta \in [0, \pi]$ and $\xi \in \mathbb{S}^{d-2}$.

$$\begin{align*}
\frac{\partial^2}{\partial^2 \theta} \bar{f}(\Phi_{B_x}(\theta, \xi)) &= \operatorname{tr}\left(\Phi_{B_x}\left(\theta + \frac{\pi}{2}, \xi\right)' \nabla_{\bar{f}}^2(\Phi_{B_x}(\theta, \xi))\Phi_{B_x}\left(\theta + \frac{\pi}{2}, \xi\right)\right) \\
&\quad + \operatorname{tr}\left(\Phi_{B_x}(\theta, \xi)' \nabla_{\bar{f}}^2(\Phi_{B_x}(\theta, \xi))\Phi_{B_x}(\theta, \xi)\right) \\
&= \operatorname{tr}\left(\nabla_{\bar{f}}^2(\Phi_{B_x}(\theta, \xi))\Phi_{B_x}\left(\theta + \frac{\pi}{2}, \xi\right)\Phi_{B_x}\left(\theta + \frac{\pi}{2}, \xi\right)\right)
\end{align*}$$
$$
\begin{align*}
&+\operatorname{tr}\left(\nabla_{\bar{f}}^2(\Phi_{B_x}(\theta, \boldsymbol{\xi}))\Phi_{B_x}(\theta, \boldsymbol{\xi})\Phi_{B_x}(\theta, \boldsymbol{\xi})'\right) \\
&=\operatorname{tr}\left(\nabla_{\bar{f}}^2(\Phi_{B_x}(\theta, \boldsymbol{\xi}))\left(\boldsymbol{x}\boldsymbol{x}'+(\boldsymbol{B}_x\boldsymbol{\xi})(\boldsymbol{B}_x\boldsymbol{\xi})'\right)\right) \\
&=\boldsymbol{x}'\nabla_{\bar{f}}^2(\Phi_{B_x}(\theta, \boldsymbol{\xi}))\boldsymbol{x}+(\boldsymbol{B}_x\boldsymbol{\xi})'\nabla_{\bar{f}}^2(\Phi_{B_x}(\theta, \boldsymbol{\xi}))(\boldsymbol{B}_x\boldsymbol{\xi}) \tag{2.7}
\end{align*}
$$

Let $\theta \in [0, \pi]$ and $\boldsymbol{\xi} \in \mathbb{S}^{d-2}$ be given. Combining (2.4) and (2.7) with Taylor's remainder theorem and the condition D1, the proof is done. $\square$

The following is the major Lemma for proving the asymptotic properties of the bias and the variance of the kernel density estimator for spherical data.

**Lemma 2.2.** Suppose the conditions L1 and D1 hold. Then the following holds for any given $k = 1, 2$.

$$ \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \mathbb{E} \left[ L^k \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) \right] - c_{h,0,k}(L) f(\boldsymbol{x}) - \frac{c_{h,2,k}(L)}{2(d-1)} \operatorname{tr} \left( \nabla_{\bar{f}}^2(\boldsymbol{x}) \right) \right| = o(h^{d+1}) $$

*Proof of Lemma 2.2.* Let $k = 1, 2$ be given. Also, let $\boldsymbol{x} \in \mathbb{S}^{d-1}$ and $\boldsymbol{B}_x \in \mathcal{B}_x$ be given. Note that the following hold by Lemma 1.2 with the condition L1 for every $h > 0$.

$$
\begin{align}
\mathbb{E}\left[L^k\left(\frac{1 - \mathbf{X}'\mathbf{x}}{h^2}\right)\right] &= \int_{\mathbb{S}^{d-1}} L^k\left(\frac{1 - \mathbf{z}'\mathbf{x}}{h^2}\right) f(\mathbf{z}) \omega_{d-1}(d\mathbf{z}) \tag{2.8} \\
&= \int_0^\pi \int_{\mathbb{S}^{d-2}} f(\Phi_{\mathbf{B}_x}(\theta, \boldsymbol{\xi})) \omega_{d-2}(d\boldsymbol{\xi}) L^k\left(\frac{1 - \cos\theta}{h^2}\right) \sin^{d-2}(\theta) d\theta \nonumber
\end{align}
$$

Note that the following hold by Lemma 1.3 and (2.2).

$$
\begin{align}
\int_{\mathbb{S}^{d-2}} (\mathbf{B}_x \boldsymbol{\xi})' \nabla_{\bar{f}}(\mathbf{x}) \omega_{d-2}(d\boldsymbol{\xi}) &= \int_{\mathbb{S}^{d-2}} \boldsymbol{\xi}' (\mathbf{B}_x' \nabla_{\bar{f}}(\mathbf{x})) \omega_{d-2}(d\boldsymbol{\xi}) = 0 \tag{2.9} \\
\int_{\mathbb{S}^{d-2}} (\mathbf{B}_x \boldsymbol{\xi})' \nabla_{\bar{f}}^2(\mathbf{x}) (\mathbf{B}_x \boldsymbol{\xi}) \omega_{d-2}(d\boldsymbol{\xi}) &= \int_{\mathbb{S}^{d-2}} \boldsymbol{\xi}' (\mathbf{B}_x' \nabla_{\bar{f}}^2(\mathbf{x}) \mathbf{B}_x) \boldsymbol{\xi} \omega_{d-2}(d\boldsymbol{\xi}) = \frac{\mathcal{A}_{d-2}}{d-1} \operatorname{tr} \left( \mathbf{B}_x' \nabla_{\bar{f}}^2(\mathbf{x}) \mathbf{B}_x \right) \nonumber \\
&= \frac{\mathcal{A}_{d-2}}{d-1} \operatorname{tr} \left( \nabla_{\bar{f}}^2(\mathbf{x}) \mathbf{B}_x \mathbf{B}_x' \right) = \frac{\mathcal{A}_{d-2}}{d-1} \operatorname{tr} \left( \nabla_{\bar{f}}^2(\mathbf{x}) (\mathbf{I}_d - \mathbf{x}\mathbf{x}') \right) \nonumber \\
&= \frac{\mathcal{A}_{d-2}}{d-1} \left\{ \operatorname{tr} \left( \nabla_{\bar{f}}^2(\mathbf{x}) \right) - \mathbf{x}' \nabla_{\bar{f}}^2(\mathbf{x}) \mathbf{x} \right\} = \frac{\mathcal{A}_{d-2}}{d-1} \operatorname{tr} \left( \nabla_{\bar{f}}^2(\mathbf{x}) \right) \tag{2.10}
\end{align}
$$

Combining (1.7), (2.8), (2.9) and (2.10) with Lemma 1.3, the following holds for every $h > 0$.

$$
\begin{align*}
& \mathbb{E}\left[L^k\left(\frac{1 - \mathbf{X}'\mathbf{x}}{h^2}\right)\right] - c_{h,0,k}(L)f(\mathbf{x}) - \frac{c_{h,2,k}(L)}{2(d-1)}\operatorname{tr}\left(\nabla_{\bar{f}}^2(\mathbf{x})\right) \\
&= \int_0^\pi \int_{\mathbb{S}^{d-2}} \underbrace{\left(f(\Phi_{\mathbf{B}_x}(\theta, \boldsymbol{\xi})) - \theta(\mathbf{B}_x\boldsymbol{\xi})'\nabla_{\bar{f}}(\mathbf{x}) - \frac{\theta^2}{2}(\mathbf{B}_x\boldsymbol{\xi})'\nabla_{\bar{f}}^2(\mathbf{x})(\mathbf{B}_x\boldsymbol{\xi})\right)}_{(2.A)} \omega_{d-2}(d\boldsymbol{\xi})
\end{align*}
$$
$$ \times L^k \left( \frac{1 - \cos \theta}{h^2} \right) \sin^{d-2}(\theta) d\theta \qquad (2.11) $$

Let $\epsilon > 0$ be given. Since $\nabla_{\tilde{f}}^2$ is uniformly continuous on $\mathbb{S}^{d-1}$, there exists $\delta_\epsilon \in (0, \pi)$ depending on $\epsilon$ such that the following holds.

$$ \|z_1 - z_2\|_2 < \delta_\epsilon \Rightarrow \|\nabla_{\tilde{f}}^2(z_1) - \nabla_{\tilde{f}}^2(z_2)\|_2 < \epsilon, \quad \forall z_1, z_2 \in \mathbb{S}^{d-1} \qquad (2.12) $$

Now suppose $\theta \in [0, \pi]$ and $\xi \in \mathbb{S}^{d-2}$ be given. Note that $\|\Phi_{B_x}(\tilde{\theta}, \xi) - x\|_2 = 2\sin(\tilde{\theta}/2) \le \tilde{\theta}$ and $\|x\|_2 = \|B_x\xi\|_2 = 1$ for every $\tilde{\theta} \in [0, \theta]$. Therefore, combining Lemma 2.1, (2.2), and (2.12) with the condition D1, there exists $\tilde{\theta}_\xi \in [0, \theta]$ depending on $\xi$ such that the following holds.

$$ \begin{aligned} |(2.A)| &= \frac{\theta^2}{2} \left| x' \nabla_{\tilde{f}}^2 (\Phi_{B_x}(\tilde{\theta}_\xi, \xi)) x + (B_x \xi)' \nabla_{\tilde{f}}^2 (\Phi_{B_x}(\tilde{\theta}_\xi, \xi)) (B_x \xi) - (B_x \xi)' \nabla_{\tilde{f}}^2 (x) (B_x \xi) \right| \\ &= \frac{\theta^2}{2} \left| x' \left( \nabla_{\tilde{f}}^2 (\Phi_{B_x}(\tilde{\theta}_\xi, \xi)) - \nabla_{\tilde{f}}^2 (x) \right) x + (B_x \xi)' \left( \nabla_{\tilde{f}}^2 (\Phi_{B_x}(\tilde{\theta}_\xi, \xi)) - \nabla_{\tilde{f}}^2 (x) \right) (B_x \xi) \right| \\ &\le \theta^2 \left\| \nabla_{\tilde{f}}^2 (\Phi_{B_x}(\tilde{\theta}_\xi, \xi)) - \nabla_{\tilde{f}}^2 (x) \right\|_2 \le \begin{cases} \theta^2 \epsilon & 0 \le \theta < \delta_\epsilon \\ 2\theta^2 \sup_{z \in \mathbb{S}^{d-1}} \|\nabla_{\tilde{f}}^2(z)\|_2 & \delta_\epsilon \le \theta \le \pi \end{cases} \end{aligned} \qquad (2.13) $$

Combining (2.11) and (2.13), the following holds for every $h > 0$.

$$ \begin{aligned} |(2.11)| &\le \epsilon A_{d-2} \underbrace{\int_0^{\delta_\epsilon} L^k \left( \frac{1 - \cos \theta}{h^2} \right) \theta^2 \sin^{d-2}(\theta) d\theta}_{(2.B)} \\ &\quad + 2 A_{d-2} \sup_{z \in \mathbb{S}^{d-1}} \|\nabla_{\tilde{f}}^2(z)\|_2 \underbrace{\int_{\delta_\epsilon}^{\pi} L^k \left( \frac{1 - \cos \theta}{h^2} \right) \theta^2 \sin^{d-2}(\theta) d\theta}_{(2.C)} \end{aligned} \qquad (2.14) $$

First, we control the upper bound of (2.B) at (2.14). Note that the following holds by Lemma 1.4 and (1.7) with the condition L1.

$$ \limsup_{h \downarrow 0} \frac{(2.B)}{h^{d+1}} \le \limsup_{h \downarrow 0} \frac{1}{h^{d+1}} \int_0^\pi L^k \left( \frac{1 - \cos \theta}{h^2} \right) \theta^2 \sin^{d-2}(\theta) d\theta = 2^{\frac{d+1}{2}} a_{d,k}(L) \qquad (2.15) $$

Second, we control the upper bound of (2.C) at (2.14). Note that the following holds by (1.7) with Cauchy-Schwarz inequality for every $h > 0$.

$$ \begin{aligned} \left(\frac{(2.C)}{h^{d+1}}\right)^2 &\le \frac{1}{h^{2(d+1)}} \int_{\delta_\epsilon}^{\pi} L^k \left(\frac{1 - \cos \theta}{h^2}\right) \theta^4 \sin^{d-2}(\theta) d\theta \int_{\delta_\epsilon}^{\pi} L^k \left(\frac{1 - \cos \theta}{h^2}\right) \sin^{d-2}(\theta) d\theta \\ &\le \frac{c_{h,4,k}(L)}{A_{d-2}h^{2(d+1)}} \int_{\delta_\epsilon}^{\pi} L^k \left(\frac{1 - \cos \theta}{h^2}\right) \sin^{d-2}(\theta) d\theta \\ &= \frac{2c_{h,4,k}(L)}{A_{d-2}h^{d+3}} \int_{\frac{\sqrt{1-\cos(\delta_\epsilon)}}{h}}^{\frac{\sqrt{2}}{h}} L^k(r^2)r^{d-2}(2-h^2r^2)^{\frac{d-3}{2}} dr \end{aligned} $$
$$
\begin{align}
&\le \frac{2c_{h,4,k}(L)}{\mathcal{A}_{d-2}h^{d+3}} \frac{1}{1-\cos(\delta_\epsilon)} \int_{\frac{\sqrt{1-\cos(\delta_\epsilon)}}{h}}^{\frac{\sqrt{2}}{h}} L^k(r^2)h^2r^d(2-h^2r^2)^{\frac{d-3}{2}}dr \nonumber \\
&\le \frac{2c_{h,4,k}(L)}{\mathcal{A}_{d-2}h^{d+3}} \frac{1}{1-\cos(\delta_\epsilon)} \left( \sup_{r \ge \frac{\sqrt{1-\cos(\delta_\epsilon)}}{h}} L^k(r^2)r^{d-1} \right) \int_{0}^{\frac{\sqrt{2}}{h}} h^2r(2-h^2r^2)^{\frac{d-3}{2}}dr \nonumber \\
&= \frac{2c_{h,4,k}(L)}{\mathcal{A}_{d-2}h^{d+3}} \frac{1}{1-\cos(\delta_\epsilon)} \frac{2^{(d-1)/2}}{d-1} \left( \sup_{r \ge \frac{\sqrt{1-\cos(\delta_\epsilon)}}{h}} L^k(r^2)r^{d-1} \right) \tag{2.16}
\end{align}
$$

Combining (2.16) with the argument of Lemma 1.1 and Lemma 1.4, we get the following.

$$
\begin{align}
\limsup_{h \downarrow 0} \frac{c_{h,4,k}(L)}{\mathcal{A}_{d-2}h^{d+3}} &= 2^{\frac{d+3}{2}} a_{d+2,k}(L), \quad \limsup_{h \downarrow 0} \left( \sup_{r \ge \frac{\sqrt{1-\cos(\delta_\epsilon)}}{h}} L^k(r^2)r^{d-1} \right) = 0 \nonumber \\
&\Rightarrow \limsup_{h \downarrow 0} \frac{(2.C)}{h^3} = 0 \tag{2.17}
\end{align}
$$

Therefore, combining (2.14), (2.15), and (2.17), we get the following.

$$
\limsup_{h \downarrow 0} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \frac{(2.11)}{h^{d+1}} \right| \le \epsilon \mathcal{A}_{d-2} 2^{\frac{d+1}{2}} a_{d,k}(L) \quad (2.18)
$$

Since (2.18) holds for any $\epsilon > 0$, $\sup_{\mathbf{x} \in \mathbb{S}^{d-1}} |(2.11)| = o(h^{d+1})$, which ends the proof. $\square$

The following Theorems state about the bias and the variance of $\hat{f}_h$ respectively.

**Theorem 2.1** (The bias of $\hat{f}_h$). Suppose the conditions L1 and D1 hold. Then the bias of the kernel density estimator is the following.

$$
\sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \mathbb{E} \left[ \hat{f}_h(\mathbf{x}) \right] - f(\mathbf{x}) - \frac{1}{d-1} \frac{a_{d,1}(L)}{a_{d-2,1}(L)} \text{tr} \left( \nabla_{\hat{f}}^2(\mathbf{x}) \right) h^2 \right| = o(h^2)
$$

Proof of Theorem 2.1. First, note that the following holds by (2.1).

$$
\mathbb{E} \left[ \hat{f}_h(\mathbf{x}) \right] = \frac{1}{c_{h,0,1}(L)} \mathbb{E} \left[ L \left( \frac{1 - \mathbf{X}'\mathbf{x}}{h^2} \right) \right], \quad \forall \mathbf{x} \in \mathbb{S}^{d-1}
$$

By Lemma 2.2 with the conditions L1 and D1, the following holds.

$$
\sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| c_{h,0,1}(L) \mathbb{E} \left[ \hat{f}_h(\mathbf{x}) \right] - c_{h,0,1}(L) f(\mathbf{x}) - \frac{c_{h,2,1}(L)}{2(d-1)} \text{tr} \left( \nabla_{\hat{f}}^2(\mathbf{x}) \right) \right| = o(h^{d+1}) \quad (2.19)
$$

Also, the following holds by Lemma 1.4 with the condition L1.

$$
\lim_{h \downarrow 0} \frac{c_{h,0,1}(L)}{h^{d-1}} = 2^{\frac{d-1}{2}} \mathcal{A}_{d-2} a_{d-2,1}(L) > 0, \quad \lim_{h \downarrow 0} \frac{c_{h,2,1}(L)}{c_{h,0,1}(L)h^2} = \frac{2a_{d,1}(L)}{a_{d-2,1}(L)} \quad (2.20)
$$

Combining (2.3), (2.19), and (2.20), the proof ends. $\square$
**Theorem 2.2** (The variance of $\hat{f}_h$). Suppose the conditions L1, D1, and B1 hold. Then the variance of the kernel density estimator is the following.

$$ \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \mathrm{Var}\left[\hat{f}_h(\boldsymbol{x})\right] - n^{-1} h^{-(d-1)} \frac{2^{-(d-1)/2}}{\mathcal{A}_{d-2}} \frac{a_{d-2,2}(L)}{a_{d-2,1}^2(L)} f(\boldsymbol{x}) \right| = o(n^{-1} h^{-(d-1)}) $$

*Proof of Theorem 2.2.* The variance of $\hat{f}_h$ can be expressed as the following for every $h = h_n > 0$ and $n \in \mathbb{N}$.

$$ nc_{h,0,1}^2(L) \mathrm{Var}\left[\hat{f}_h(\boldsymbol{x})\right] = \mathbb{E}\left[L^2\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)\right] - \left(\mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)\right]\right)^2, \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1} \quad (2.21) $$

By Lemma 2.2 with the conditions L1 and D1, the following holds.

$$ \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \mathbb{E}\left[L^k\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)\right] - c_{h,0,k}(L)f(\boldsymbol{x}) - \frac{c_{h,2,k}(L)}{2(d-1)}\mathrm{tr}\left(\nabla_{\hat{f}}^2(\boldsymbol{x})\right) \right| = o(h^{d+1}), \quad \forall k = 1, 2 \quad (2.22) $$

Also, note that (2.3) and (2.22) imply the following.

$$ \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \mathbb{E}\left[L^2\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)\right] - c_{h,0,2}(L)f(\boldsymbol{x}) \right| = O(h^{d+1}), \quad \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)\right] \right| = O(h^{d-1}) \quad (2.23) $$

Combining (2.21) and (2.23), we get the following.

$$ \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| nc_{h,0,1}^2(L) \mathrm{Var}\left[\hat{f}_h(\boldsymbol{x})\right] - c_{h,0,2}(L)f(\boldsymbol{x}) \right| = o(h^{d-1}) \quad (2.24) $$

Also, the following holds by Lemma 1.4 with the condition L1.

$$ \lim_{h \downarrow 0} \frac{c_{h,0,1}(L)}{h^{d-1}} = 2^{\frac{d-1}{2}} \mathcal{A}_{d-2} a_{d-2,1}(L) > 0, \quad \lim_{h \downarrow 0} \frac{c_{h,0,2}(L)h^{d-1}}{c_{h,0,1}^2(L)} = \frac{2^{-(d-1)/2}}{\mathcal{A}_{d-2}} \frac{a_{d-2,2}(L)}{a_{d-2,1}^2(L)} \quad (2.25) $$

Combining (2.3), (2.24), and (2.25) with the condition B1, the proof ends. $\square$

The next step is to calculate the mean integrated squared error (MISE). Note that the MISE is defined as the following for every $h = h_n > 0$.

$$ \mathrm{MISE}[\hat{f}_h] := \int_{\mathbb{S}^{d-1}} \left( \mathbb{E}[\hat{f}_h(\boldsymbol{x})] - f(\boldsymbol{x}) \right)^2 \omega_{d-1}(\mathrm{d}\boldsymbol{x}) + \int_{\mathbb{S}^{d-1}} \mathrm{Var}[\hat{f}_h(\boldsymbol{x})] \omega_{d-1}(\mathrm{d}\boldsymbol{x}) \quad (2.26) $$

Now we state the MISE formula, with defining a score $\mathbf{S}_f$ as the following.

$$ \mathbf{S}_f := \int_{\mathbb{S}^{d-1}} \left[ \mathrm{tr}\left(\nabla_{\hat{f}}^2(\boldsymbol{x})\right) \right]^2 \omega_{d-1}(\mathrm{d}\boldsymbol{x}) \quad (2.27) $$
**Theorem 2.3** (The MISE of $\hat{f}_h$). Suppose the conditions L1, D1, and B1 hold. Then the following holds.

$$ \text{MISE}[\hat{f}_h] = h^4 \frac{1}{(d-1)^2} \frac{a_{d,1}^2(L)}{a_{d-2,1}^2(L)} \mathbf{S}_f + n^{-1} h^{-(d-1)} \frac{2^{-(d-1)/2}}{\mathcal{A}_{d-2}} \frac{a_{d-2,2}(L)}{a_{d-2,1}^2(L)} + o(h^4 + n^{-1}h^{-(d-1)}) \quad (2.28) $$

Therefore, the optimal bandwidth with respect to the asymptotic MISE (AMISE) for the kernel density estimator $h_{\text{AMISE}}$ and the order of the convergence rate of MISE with respect to $h_{\text{AMISE}}$ can be calculated as the following.

$$ h_{\text{AMISE}} = 2^{-\frac{1}{2}} \left[ \frac{(d-1)^3 a_{d-2,2}(L)}{\mathcal{A}_{d-2} \mathbf{S}_f a_{d,1}^2(L)} \right]^{\frac{1}{d+3}} n^{-\frac{1}{d+3}}, \quad \text{MISE}[\hat{f}_{h_{\text{AMISE}}}] = O(n^{-\frac{4}{d+3}}) \quad (2.29) $$

*Proof of Theorem 2.3.* Note that Theorem 2.1 with the conditions L1 and D1 implies the following.

$$ \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \mathbb{E} \left[ \hat{f}_h(\mathbf{x}) \right] - f(\mathbf{x}) + \frac{h^2}{d-1} \frac{a_{d,1}(L)}{a_{d-2,1}(L)} \text{tr} \left( \nabla_{\hat{f}}^2(\mathbf{x}) \right) \right| = O(h^2) \quad (2.30) $$

$$ \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \left( \mathbb{E} \left[ \hat{f}_h(\mathbf{x}) \right] - f(\mathbf{x}) \right)^2 - \frac{h^4}{(d-1)^2} \frac{a_{d,1}^2(L)}{a_{d-2,1}^2(L)} \left[ \text{tr} \left( \nabla_{\hat{f}}^2(\mathbf{x}) \right) \right]^2 \right| = o(h^4) \quad (2.31) $$

Combining and (2.27), (2.30), and (2.31) with the fact that $\int_{\mathbb{S}^{d-1}} f(\mathbf{x})\omega_{d-1}(d\mathbf{x}) = 1$, the following holds for every $h = h_n > 0$ and $n \in \mathbb{N}$.

$$ \left| \int_{\mathbb{S}^{d-1}} \left( \mathbb{E} \left[ \hat{f}_h(\mathbf{x}) \right] - f(\mathbf{x}) \right)^2 \omega_{d-1}(d\mathbf{x}) - \frac{h^4}{(d-1)^2} \frac{a_{d,1}^2(L)}{a_{d-2,1}^2(L)} \mathbf{S}_f \right| \\ \leq \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \left( \mathbb{E} \left[ \hat{f}_h(\mathbf{x}) \right] - f(\mathbf{x}) \right)^2 - \frac{h^4}{(d-1)^2} \frac{a_{d,1}^2(L)}{a_{d-2,1}^2(L)} \left[ \text{tr} \left( \nabla_{\hat{f}}^2(\mathbf{x}) \right) \right]^2 \right| \quad (2.32) $$

$$ \left| \int_{\mathbb{S}^{d-1}} \mathrm{Var} \left[ \hat{f}_h(\mathbf{x}) \right] \omega_{d-1}(d\mathbf{x}) - n^{-1} h^{-(d-1)} \frac{2^{-(d-1)/2}}{\mathcal{A}_{d-2}} \frac{a_{d-2,2}(L)}{a_{d-2,1}^2(L)} \right| \\ \leq \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \mathrm{Var} \left[ \hat{f}_h(\mathbf{x}) \right] - n^{-1} h^{-(d-1)} \frac{2^{-(d-1)/2}}{\mathcal{A}_{d-2}} \frac{a_{d-2,2}(L)}{a_{d-2,1}^2(L)} f(\mathbf{x}) \right| \quad (2.33) $$

By combining (2.32), (2.33), and Theorem 2.2 with the conditions L1, D1, and B1, (2.28) is proved. Note that by differentiating with respect to $h$, (2.28) is minimized when $h > 0$ satisfies the following.

$$ \frac{\partial}{\partial h} \text{MISE}[\hat{f}_h] = 4h^3 \frac{1}{(d-1)^2} \frac{a_{d,1}^2(L)}{a_{d-2,1}^2(L)} \mathbf{S}_f - (d-1) n^{-1} h^{-d} \frac{2^{-(d-1)/2}}{\mathcal{A}_{d-2}} \frac{a_{d-2,2}(L)}{a_{d-2,1}^2(L)} = 0 $$

By some simple calculations, (2.29) is proved. □

Finally, we state and prove the convergence property of the kernel density estimator.
**Theorem 2.4** (Convergence property of $\hat{f}_h$). Suppose the conditions L1, D1, and B1 hold. Then the following holds.

$$ \hat{f}_h(\boldsymbol{x}) - f(\boldsymbol{x}) = o_p(1) \quad \text{as} \quad n \to \infty, \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1} $$

*Proof of Theorem 2.4.* Let $\boldsymbol{x} \in \mathbb{S}^{d-1}$ be given. By Theorem 2.1 with the conditions L1 and D1, $\mathbb{E}[\hat{f}_h(\boldsymbol{x})] - f(\boldsymbol{x}) = O(h^2)$. Therefore, it suffices to prove that $\hat{f}_h(\boldsymbol{x}) - \mathbb{E}[\hat{f}_h(\boldsymbol{x})] = o_p(1)$ as $n \to \infty$. Let $\epsilon > 0$ be given. By Chebyshev's inequality, the following holds for every $h = h_n > 0$ and $n \in \mathbb{N}$.

$$ \mathbb{P}\left(\left|\hat{f}_h(\boldsymbol{x}) - \mathbb{E}[\hat{f}_h(\boldsymbol{x})]\right| > \epsilon\right) \le \epsilon^{-2}\mathbb{V}\text{ar}[\hat{f}_h(\boldsymbol{x})] $$

Note that Theorem 2.2 with the conditions L1, D1, and B1 implies $\mathbb{V}\text{ar}[\hat{f}_h(\boldsymbol{x})] = O(n^{-1}h^{-(d-1)})$. Therefore, the following holds.

$$ \lim_{n \to \infty} \mathbb{P}\left(\left|\hat{f}_h(\boldsymbol{x}) - \mathbb{E}[\hat{f}_h(\boldsymbol{x})]\right| > \epsilon\right) = 0 $$

Since $\epsilon > 0$ is arbitrary, the proof is done. $\square$

## 3 The local constant Fréchet regression for random objects with spherical predictors

In this section, we prove some properties about the local constant Fréchet regression for random objects with spherical predictors. Let $(\Omega, d)$ be a metric space. Consider a random process $(\mathbf{X}, Y) \sim F$, where $\mathbf{X}$ and $Y$ take values in $\mathbb{S}^{d-1}$ and $\Omega$ respectively, and $F$ is the joint distribution of $(\mathbf{X}, Y)$ on $\mathbb{S}^{d-1} \times \Omega$. Also, let $f$ be the marginal density function of $\mathbf{X}$ on $\mathbb{S}^{d-1}$, and $F_Y$ be the marginal distribution of $Y$ on $\Omega$. Now we impose a mild condition about the conditional distribution. Note that the condition C1 is similar to the condition D1.

**Condition C1.** For any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$, the conditional distribution $F_{Y|\mathbf{X}=\boldsymbol{x}}$ of $Y|\mathbf{X} = \boldsymbol{x}$ exists. Also, for any given $y \in \Omega$, assume that there exists a function $g_y: \mathbb{S}^{d-1} \to [0, \infty)$ defined by the following, where $dF_{Y|\mathbf{X}=\boldsymbol{x}}/dF_Y$ denotes the Radon-Nikodym derivative of $F_{Y|\mathbf{X}=\boldsymbol{x}}$ with respect to $F_Y$ for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$.

$$ g_y(\boldsymbol{x}) := \frac{dF_{Y|\mathbf{X}=\boldsymbol{x}}(y)}{dF_Y}, \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1} $$
For any given $y \in \Omega$, there exists a homogeneous extension $\bar{g}_y$ of $g_y$ from $\mathbb{S}^{d-1}$ to $\mathbb{R}^d \setminus \{\mathbf{0}_d\}$ defined by the following.

$$ \bar{g}_y(\mathbf{x}) := g_y\left(\frac{\mathbf{x}}{\|\mathbf{x}\|_2}\right), \quad \forall \mathbf{x} \in \mathbb{R}^d \setminus \{\mathbf{0}_d\} $$

In addition, assume that the family of functions $\{\bar{g}_y : y \in \Omega\}$, the family of gradient vectors $\{\nabla_{\bar{g}_y} : y \in \Omega\}$, and the family of Hessian matrices $\{\nabla^2_{\bar{g}_y} : y \in \Omega\}$ are equicontinuous and pointwise bounded on $\mathbb{R}^d \setminus \{\mathbf{0}_d\}$.

The condition C1 is an application of the conditions used in Petersen and Müller (2019). Since $\mathbb{S}^{d-1}$ is compact, under the condition C1, the family of the gradient vectors $\{\nabla_{\bar{g}_y} : y \in \Omega\}$ and the family of Hessian matrices $\{\nabla^2_{\bar{g}_y} : y \in \Omega\}$ are uniformly equicontinuous and uniformly bounded on $\mathbb{S}^{d-1}$. For any given $y \in \Omega$, the condition C1 implies the following.

$$ \mathbf{x}'\nabla_{\bar{g}_y}(\mathbf{x}) = 0, \quad \mathbf{x}'\nabla^2_{\bar{g}_y}(\mathbf{x})\mathbf{x} = 0, \quad \forall \mathbf{x} \in \mathbb{S}^{d-1} \qquad (3.1) $$

In addition, the condition D1 and C1 with the compactness of $\mathbb{S}^{d-1}$ imply the following, where $\nabla_{\bar{f} \cdot \bar{g}_y}$ and $\nabla^2_{\bar{f} \cdot \bar{g}_y}$ denote the gradient vector and the Hessian matrix of $\bar{f} \cdot \bar{g}_y$ respectively for any given $y \in \Omega$.

$$ \sup_{y \in \Omega} \sup_{\mathbf{z} \in \mathbb{S}^{d-1}} |(f \cdot g_y)(\mathbf{z})| < \infty, \quad \sup_{y \in \Omega} \sup_{\mathbf{z} \in \mathbb{S}^{d-1}} \|\nabla_{\bar{f} \cdot \bar{g}_y}(\mathbf{z})\|_2 < \infty, \quad \sup_{y \in \Omega} \sup_{\mathbf{z} \in \mathbb{S}^{d-1}} \|\nabla^2_{\bar{f} \cdot \bar{g}_y}(\mathbf{z})\|_2 < \infty \quad (3.2) $$

The following states about the approximation of $(f \cdot g_y)(\Phi_{B_x}(\theta, \boldsymbol{\xi}))$ for every $y \in \Omega$, $\mathbf{x} \in \mathbb{S}^{d-1}$, $B_x \in \mathcal{B}_x$, and $\theta \in [0, \pi]$, which is a slight modification of Lemma 2.1 by substituting $f$ to $(f \cdot g_y)$ for any given $y \in \Omega$. The proof is omitted.

**Lemma 3.1.** Suppose the conditions D1 and C1 hold. Let $\mathbf{x} \in \mathbb{S}^{d-1}$ and $B_x \in \mathcal{B}_x$ be given. Then for any given $y \in \Omega$, $\theta \in [0, \pi]$, and $\boldsymbol{\xi} \in \mathbb{S}^{d-2}$, there exists $\tilde{\theta}_{\boldsymbol{\xi}} \in [0, \theta]$ depending on $\boldsymbol{\xi}$ such that the following holds, where $\Phi_{B_x}$ is defined by (1.6).

$$ (f \cdot g_y)(\Phi_{B_x}(\theta, \boldsymbol{\xi})) = (f \cdot g_y)(\mathbf{x}) + \theta(B_x \boldsymbol{\xi})' \nabla_{\bar{f} \cdot \bar{g}_y}(\mathbf{x}) \\ + \frac{\theta^2}{2} \left( \mathbf{x}' \nabla^2_{\bar{f} \cdot \bar{g}_y} (\Phi_{B_x}(\tilde{\theta}_{\boldsymbol{\xi}}, \boldsymbol{\xi})) \mathbf{x} + (B_x \boldsymbol{\xi})' \nabla^2_{\bar{f} \cdot \bar{g}_y} (\Phi_{B_x}(\tilde{\theta}_{\boldsymbol{\xi}}, \boldsymbol{\xi})) (B_x \boldsymbol{\xi}) \right) $$

The following is a generalization of Lemma 2.2, which is crucial for proving the asymptotic properties of the local constant Fréchet estimator. The proof is similar to Lemma 2.2, except for that we use (3.2) instead of (2.3), and the uniform equicontinuity of $\{\nabla^2_{\bar{f} \cdot \bar{g}_y} : y \in \Omega\}$ on $\mathbb{S}^{d-1}$ instead of the uniform continuity of $\nabla^2_{\bar{f}}$ on $\mathbb{S}^{d-1}$. Detailed proof is omitted.
**Lemma 3.2.** Suppose the conditions L1, D1, and C1 hold. Then the following holds for any given $k = 1, 2$.

$$ \sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \mathbb{E} \left[ L^k \left( \frac{1 - \mathbf{X}'\mathbf{x}}{h^2} \right) g_y(\mathbf{X}) \right] - c_{h,0,k}(L) (f \cdot g_y)(\mathbf{x}) - \frac{c_{h,2,k}(L)}{2(d-1)} \text{tr} \left( \nabla^2_{\bar{f} \cdot \bar{g}_y}(\mathbf{x}) \right) \right| = o(h^{d+1}) $$

Next, we impose conditions for $\Omega$, just as Im et al. (2024). The condition R1 is about the boundedness of $\Omega$, and the condition R2 is about the entropy condition of $(\Omega, d)$.

**Condition R1.** The metric space $(\Omega, d)$ is totally bounded, i.e. $\text{diam}(\Omega) < \infty$.

**Condition R2.** Denote $B_d(w, \delta) \subset \Omega$ as the ball of radius $\delta$ centered at $w$ for every $w \in \Omega$ and $\delta > 0$. Also, denote $N(\epsilon, B_d(w, \delta), d)$ as the covering number of $B_d(w, \delta)$ using balls of radius $\epsilon$ for every $\epsilon > 0$, $w \in \Omega$, and $\delta > 0$. Then the following holds.

$$ \sup_{w \in \Omega} \int_{0}^{1} \sqrt{1 + \log N(\delta\epsilon, B_d(w, \delta), d)} d\epsilon = O(1) \quad \text{as} \quad \delta \downarrow 0 $$

As introduced in Im et al. (2024), the Fréchet regression function $m_\oplus$ of Y given $\mathbf{X} = \mathbf{x}$ is defined by the following.

$$ m_{\oplus}(\mathbf{x}) := \underset{y \in \Omega}{\operatorname{argmin}} \mathbb{E} \left[ d^2(Y, y) \middle| \mathbf{X} = \mathbf{x} \right], \quad \forall \mathbf{x} \in \mathbb{S}^{d-1} \qquad (3.3) $$

For the convenience, similar to Im et al. (2024), we define a function $M_\oplus : \mathbb{S}^{d-1} \times \Omega \to \mathbb{R}$ as the following, which might be interpreted as the conditional mean squared distance. Note that for any given $\mathbf{x} \in \mathbb{S}^{d-1}$, $m_\oplus(\mathbf{x})$ is the minimizer of $M_\oplus(\mathbf{x}, \cdot)$.

$$ M_{\oplus}(\mathbf{x}, y) := \mathbb{E} \left[ d^2(Y, y) \middle| \mathbf{X} = \mathbf{x} \right], \quad \forall \mathbf{x} \in \mathbb{S}^{d-1}, \forall y \in \Omega \qquad (3.4) $$

The goal of this paper is to estimate $m_\oplus(\mathbf{x})$ for any given $\mathbf{x} \in \mathbb{S}^{d-1}$. In this section, we introduce the local constant Fréchet regression estimator, which we denote as $\hat{m}_{h,0}$. Let random samples $(\mathbf{X}_i, Y_i) \sim F$, $i = 1, 2, \dots, n$ be given. Just as Im et al. (2024), we define $\hat{m}_{h,0} : \mathbb{S}^{d-1} \to \mathbb{R}$ and $\hat{M}_{h,0} : \mathbb{S}^{d-1} \times \Omega \to \mathbb{R}$ as the following for every $h = h_n > 0$ and $n \in \mathbb{N}$. Note that for any given $\mathbf{x} \in \mathbb{S}^{d-1}$, $\hat{m}_{h,0}(\mathbf{x})$ is the minimizer of $\hat{M}_{h,0}(\mathbf{x}, \cdot)$.

$$ \hat{m}_{h,0}(\mathbf{x}) := \underset{y \in \Omega}{\operatorname{argmin}} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}'_i \mathbf{x}}{h^2}\right) d^2(Y_i, y), \quad \forall \mathbf{x} \in \mathbb{S}^{d-1} \qquad (3.5) $$

$$ \hat{M}_{h,0}(\mathbf{x}, y) := \frac{\sum_{i=1}^{n} L\left(\frac{1-\mathbf{X}'_i \mathbf{x}}{h^2}\right) d^2(Y_i, y)}{\sum_{i=1}^{n} L\left(\frac{1-\mathbf{X}'_i \mathbf{x}}{h^2}\right)}, \quad \forall \mathbf{x} \in \mathbb{S}^{d-1}, \forall y \in \Omega \qquad (3.6) $$
Also, we define $\tilde{m}_{h,0} : \mathbb{S}^{d-1} \to \mathbb{R}$ and $\tilde{M}_{h,0} : \mathbb{S}^{d-1} \times \Omega \to \mathbb{R}$ as the following for every $h = h_n > 0$ and $n \in \mathbb{N}$. Note that for any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$, $\tilde{m}_{h,0}(\boldsymbol{x})$ is the minimizer of $\tilde{M}_{h,0}(\boldsymbol{x}, \cdot)$. In addition, $\hat{m}_{h,0}$ and $\hat{M}_{h,0}$ are actually empirical versions of functions $\tilde{m}_{h,0}$ and $\tilde{M}_{h,0}$ respectively.

$$ \tilde{m}_{h,0}(\boldsymbol{x}) := \underset{y \in \Omega}{\operatorname{argmin}} \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) d^2(Y, y) \right], \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1} \qquad (3.7) $$

$$ \tilde{M}_{h,0}(\boldsymbol{x}, y) := \frac{\mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right) d^2(Y, y)\right]}{\mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)\right]}, \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1}, \forall y \in \Omega \qquad (3.8) $$

For any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$, the goal of this section is to get an upper bound of $d(\hat{m}_{h,0}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x}))$. This can be computed by decomposing into $d(\tilde{m}_{h,0}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x}))$ and $d(\hat{m}_{h,0}(\boldsymbol{x}), \tilde{m}_{h,0}(\boldsymbol{x}))$. Since it is difficult to compute $d(\tilde{m}_{h,0}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x}))$ and $d(\hat{m}_{h,0}(\boldsymbol{x}), \tilde{m}_{h,0}(\boldsymbol{x}))$ directly, instead we compute $\tilde{M}_{h,0}(\boldsymbol{x}, y) - M_{\oplus}(\boldsymbol{x}, y)$ and $\hat{M}_{h,0}(\boldsymbol{x}, y) - \tilde{M}_{h,0}(\boldsymbol{x}, y)$ for every $y \in \Omega$. This method was also used in Im et al. (2024). To do so, first we impose some conditions with respect to $m_{\oplus}$, $\hat{m}_{h,0}$, $\tilde{m}_{h,0}$, $M_{\oplus}$, $\hat{M}_{h,0}$, and $\tilde{M}_{h,0}$, which are also used in Im et al. (2024).

**Condition M1-0.** For any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$, the object $m_{\oplus}(\boldsymbol{x})$ is unique. For any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$ and for every $n \in \mathbb{N}$, $\hat{m}_{h,0}(\boldsymbol{x})$ and $\tilde{m}_{h,0}(\boldsymbol{x})$ exist and are unique, the latter almost surely. In addition, for any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$, the following hold for every $\epsilon > 0$.

$$ \inf_{d(y, m_{\oplus}(\boldsymbol{x})) > \epsilon} \left[ M_{\oplus}(\boldsymbol{x}, y) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) \right] > 0 $$

$$ \liminf_{h \downarrow 0} \inf_{d(y, \tilde{m}_{h,0}(\boldsymbol{x})) > \epsilon} \left[ \tilde{M}_{h,0}(\boldsymbol{x}, y) - \tilde{M}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,0}(\boldsymbol{x})) \right] > 0 $$

**Condition M2.** For any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$, there exist constants $\eta_{\oplus} > 0$, $C_{\oplus} > 0$, and $\beta_{\oplus} > 1$ such that the following holds for every $y \in \Omega$.

$$ d(y, m_{\oplus}(\boldsymbol{x})) < \eta_{\oplus} \Rightarrow M_{\oplus}(\boldsymbol{x}, y) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) \geq C_{\oplus} d(y, m_{\oplus}(\boldsymbol{x}))^{\beta_{\oplus}} $$

**Condition M3-0.** For any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$, there exist constants $\tilde{h}_0 > 0$, $\tilde{\eta}_0 > 0$, $\tilde{C}_0 > 0$, and $\tilde{\beta}_0 > 1$ such that the following holds for every $y \in \Omega$.

$$ 0 < h < \tilde{h}_0, \quad d(y, \tilde{m}_{h,0}(\boldsymbol{x})) < \tilde{\eta}_0 \Rightarrow \tilde{M}_{h,0}(\boldsymbol{x}, y) - \tilde{M}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,0}(\boldsymbol{x})) \geq \tilde{C}_0 d(y, \tilde{m}_{h,0}(\boldsymbol{x}))^{\tilde{\beta}_0} $$

The following is a useful lemma to prove the convergence rate of $d(\tilde{m}_{h,0}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x}))$.

**Lemma 3.3.** Suppose the conditions L1, D1, D2, and C1 hold. Then the following holds.

$$ \sup_{y \in \Omega} \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \frac{\mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right) g_y(\boldsymbol{X})\right]}{\mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)\right]} - g_y(\boldsymbol{x}) \right| $$
$$ - \frac{h^2}{d-1} \frac{a_{d,1}(L)}{a_{d-2,1}(L)} \left\{ \mathrm{tr} \left( \nabla^2_{\bar{g}_y}(\boldsymbol{x}) \right) + 2 \frac{\nabla_{\bar{f}}(\boldsymbol{x})' \nabla_{\bar{g}_y}(\boldsymbol{x})}{f(\boldsymbol{x})} \right\} = o(h^2) $$

Proof of Lemma 3.3. Note that the following hold by Lemma 2.2 and Lemma 3.2 with the conditions L1, D1, and C1.

$$ \sup_{y \in \Omega} \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) g_y(\boldsymbol{X}) \right] - c_{h,0,1}(L) (f \cdot g_y)(\boldsymbol{x}) - \frac{c_{h,2,1}(L)}{2(d-1)} \mathrm{tr} \left( \nabla^2_{\bar{f} \cdot \bar{g}_y}(\boldsymbol{x}) \right) \right| = o(h^{d+1}) \quad (3.9) $$

$$ \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) \right] - c_{h,0,1}(L) f(\boldsymbol{x}) - \frac{c_{h,2,1}(L)}{2(d-1)} \mathrm{tr} \left( \nabla^2_{\bar{f}}(\boldsymbol{x}) \right) \right| = o(h^{d+1}) \quad (3.10) $$

Also, the following holds for every $y \in \Omega$ and $\boldsymbol{x} \in \mathbb{S}^{d-1}$.

$$ \mathrm{tr} \left( \nabla^2_{\bar{f} \cdot \bar{g}_y}(\boldsymbol{x}) \right) = f(\boldsymbol{x}) \mathrm{tr} \left( \nabla^2_{\bar{g}_y}(\boldsymbol{x}) \right) + g_y(\boldsymbol{x}) \mathrm{tr} \left( \nabla^2_{\bar{f}}(\boldsymbol{x}) \right) + 2 \nabla_{\bar{f}}(\boldsymbol{x})' \nabla_{\bar{g}_y}(\boldsymbol{x}) \quad (3.11) $$

Combining (2.3), (3.2), (3.9), (3.10), and (3.11) with the condition C1, we get the following.

$$ \sup_{y \in \Omega} \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) g_y(\boldsymbol{X}) \right] - g_y(\boldsymbol{x}) \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) \right] \right. \\ \left. - \frac{c_{h,2,1}(L)}{2(d-1)} \left\{ f(\boldsymbol{x}) \mathrm{tr} \left( \nabla^2_{\bar{g}_y}(\boldsymbol{x}) \right) + 2 \nabla_{\bar{f}}(\boldsymbol{x})' \nabla_{\bar{g}_y}(\boldsymbol{x}) \right\} \right| = o(h^{d+1}) \quad (3.12) $$

Also, the compactness of $\mathbb{S}^{d-1}$ with the conditions D1 and C1 implies the following.

$$ \sup_{y \in \Omega} \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \nabla_{\bar{f}}(\boldsymbol{x})' \nabla_{\bar{g}_y}(\boldsymbol{x}) \right| < \infty \quad (3.13) $$

Therefore, by combining (2.3), (3.2), (3.10), (3.12), (3.13), and Lemma 1.4 with the conditions L1, D1, and C1, we get the following.

$$ \sup_{y \in \Omega} \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) g_y(\boldsymbol{X}) \right] - g_y(\boldsymbol{x}) \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) \right] \right| = O(h^{d+1}) \quad (3.14) $$

$$ \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) \right] - c_{h,0,1}(L) f(\boldsymbol{x}) \right| = O(h^{d+1}) \quad (3.15) $$

Next, (3.13), (3.14), (3.15), and Lemma 1.4 with the conditions L1 and D2 imply the following.

$$ \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left( c_{h,0,1}(L) f(\boldsymbol{x}) \right)^{-1} = O(h^{-(d-1)}) \quad (3.16) $$

$$ \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| c_{h,0,1}(L) f(\boldsymbol{x}) \left( \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) \right] \right)^{-1} \right| = 1 \quad (3.17) $$

$$ \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \left( \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) \right] \right)^{-1} - \left( c_{h,0,1}(L) f(\boldsymbol{x}) \right)^{-1} \right| = O(h^{-(d-3)}) \quad (3.18) $$
Combining (2.20), (3.12), (3.13), (3.16), (3.17), and (3.18) with the condition L1, we get the following.

$$ \sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \left( c_{h,0,1}(L) f(\mathbf{x}) \right)^{-1} \left\{ \mathbb{E} \left[ L \left( \frac{1-\mathbf{X}'\mathbf{x}}{h^2} \right) g_y(\mathbf{X}) \right] - g_y(\mathbf{x}) \mathbb{E} \left[ L \left( \frac{1-\mathbf{X}'\mathbf{x}}{h^2} \right) \right] \right\} - \frac{h^2}{d-1} \frac{a_{d,1}(L)}{a_{d-2,1}(L)} \left\{ \text{tr} \left( \nabla^2_{\tilde{g}_y}(\mathbf{x}) \right) + 2 \frac{\nabla_{\tilde{f}}(\mathbf{x})' \nabla_{\tilde{g}_y}(\mathbf{x})}{f(\mathbf{x})} \right\} \right| = o(h^2) \quad (3.19) $$

Therefore, by combining (3.14), (3.18), and (3.19), the proof ends. □

Now we state about the approximation of $\tilde{M}_{h,0}(\mathbf{x}, y) - M_{\oplus}(\mathbf{x}, y)$ for every $\mathbf{x} \in \mathbb{S}^{d-1}$ and $y \in \Omega$.

**Lemma 3.4.** Suppose the conditions L1, D1, D2, C1, and R1 hold. Then the following holds.

$$ \sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \tilde{M}_{h,0}(\mathbf{x}, y) - M_{\oplus}(\mathbf{x}, y) - \frac{h^2}{d-1} \frac{a_{d,1}(L)}{a_{d-2,1}(L)} \left\{ \int_{\Omega} d^2(y, w) \text{tr} \left( \nabla^2_{\tilde{g}_w}(\mathbf{x}) \right) dF_Y(w) + \frac{2}{f(\mathbf{x})} \int_{\Omega} d^2(y, w) \nabla_{\tilde{f}}(\mathbf{x})' \nabla_{\tilde{g}_w}(\mathbf{x}) dF_Y(w) \right\} \right| = o(h^2) $$

*Proof of Lemma 3.4.* Fubini's theorem implies the following for every $\mathbf{x} \in \mathbb{S}^{d-1}$, $y \in \Omega$, and $h > 0$.

$$ M_{\oplus}(\mathbf{x}, y) = \int_{\Omega} d^2(y, w) dF_{Y|\mathbf{X}=\mathbf{x}}(w) = \int_{\Omega} d^2(y, w) g_w(\mathbf{x}) dF_Y(w) \quad (3.20) $$

$$ \tilde{M}_{h,0}(\mathbf{x}, y) = \frac{\mathbb{E}\left[L\left(\frac{1-\mathbf{X}'\mathbf{x}}{h^2}\right) M_{\oplus}(\mathbf{X}, y)\right]}{\mathbb{E}\left[L\left(\frac{1-\mathbf{X}'\mathbf{x}}{h^2}\right)\right]} = \int_{\Omega} d^2(y, w) \frac{\mathbb{E}\left[L\left(\frac{1-\mathbf{X}'\mathbf{x}}{h^2}\right) g_w(\mathbf{X})\right]}{\mathbb{E}\left[L\left(\frac{1-\mathbf{X}'\mathbf{x}}{h^2}\right)\right]} dF_Y(w) \quad (3.21) $$

Since the condition R1 implies that $d^2(y, w) \le \text{diam}(\Omega)^2 < \infty$ for every $y, w \in \Omega$, Lemma 3.3 with the conditions L1, D1, D2, and C1 implies Lemma 3.4. □

The next Theorems are direct consequences of Lemma 3.4, which state that $d(\tilde{m}_{h,0}(\mathbf{x}), m_{\oplus}(\mathbf{x}))$ converges to 0 with the convergence rate of $O(h^{2/(\beta_{\oplus}-1)})$ for every $\mathbf{x} \in \mathbb{S}^{d-1}$.

**Theorem 3.1** (Convergence of the bias of $\tilde{m}_{h,0}$). Suppose the conditions L1, D1, D2, C1, R1, and M1-0 hold. Then the following holds for every $\mathbf{x} \in \mathbb{S}^{d-1}$.

$$ \lim_{h \downarrow 0} d(\tilde{m}_{h,0}(\mathbf{x}), m_{\oplus}(\mathbf{x})) = 0 $$

*Proof of Theorem 3.1.* Let $\mathbf{x} \in \mathbb{S}^{d-1}$ be given. Also, let $\epsilon > 0$ be given. By the condition M1-0, there exist constants $\eta > 0$ such that the following hold.

$$ d(y, m_{\oplus}(\mathbf{x})) > \epsilon \Rightarrow M_{\oplus}(\mathbf{x}, y) - M_{\oplus}(\mathbf{x}, m_{\oplus}(\mathbf{x})) \ge \eta, \quad \forall y \in \Omega \quad (3.22) $$
Also, by (3.2), (3.13), and Lemma 3.4 with the conditions L1, D1, D2, C1, and R1, there exists a constant $\delta > 0$ such that the following holds.

$$ \mathrm{diam}(\Omega)^{-2} \left| \int_{\Omega} d^2(y, w) \mathrm{tr} \left( \nabla_{\bar{g}_w}^2(\boldsymbol{x}) \right) \mathrm{d}F_Y(w) \right| \le \sup_{y \in \Omega} \left| \mathrm{tr} \left( \nabla_{\bar{g}_y}^2(\boldsymbol{x}) \right) \right| < \infty \quad (3.23) $$

$$ \mathrm{diam}(\Omega)^{-2} \left| \int_{\Omega} d^2(y, w) \nabla_{\bar{f}}(\boldsymbol{x})' \nabla_{\bar{g}_w}(\boldsymbol{x}) \mathrm{d}F_Y(w) \right| \le \sup_{y \in \Omega} \left| \nabla_{\bar{f}}(\boldsymbol{x})' \nabla_{\bar{g}_y}(\boldsymbol{x}) \right| < \infty \quad (3.24) $$

$$ \sup_{y \in \Omega} \left| \tilde{M}_{h,0}(\boldsymbol{x}, y) - M_{\oplus}(\boldsymbol{x}, y) \right| < \frac{\eta}{2}, \quad \forall h \in (0, \delta) \quad (3.25) $$

Note that the following holds for every $h \in (0, \delta)$ by (3.23), (3.24), and (3.25).

$$ \begin{aligned} & M_{\oplus}(\boldsymbol{x}, \tilde{m}_{h,0}(\boldsymbol{x})) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) \\ & \le \left| M_{\oplus}(\boldsymbol{x}, \tilde{m}_{h,0}(\boldsymbol{x})) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) \right| + \left| \tilde{M}_{h,0}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) - \tilde{M}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,0}(\boldsymbol{x})) \right| \\ & \le \left| \tilde{M}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,0}(\boldsymbol{x})) - M_{\oplus}(\boldsymbol{x}, \tilde{m}_{h,0}(\boldsymbol{x})) \right| + \left| \tilde{M}_{h,0}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) \right| < \eta \end{aligned} \quad (3.26) $$

Therefore, (3.22) and (3.26) imply that $d(\tilde{m}_{h,0}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x})) \le \epsilon$ for every $h \in (0, \delta)$. Since $\epsilon > 0$ is arbitrary, the proof is done. $\square$

**Theorem 3.2** (Convergence rate of the bias of $\hat{m}_{h,0}$). Suppose the conditions L1, D1, D2, C1, R1, M1-0, and M2 hold. Then the following holds for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$, where $C_{\oplus} > 0$ and $\beta_{\oplus} > 1$ are the constant defined in M2.

$$ d(\tilde{m}_{h,0}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x}))^{\beta_{\oplus}-1} \le \frac{2\mathrm{diam}(\Omega)}{C_{\oplus}(d-1)} \frac{a_{d,1}(L)}{a_{d-2,1}(L)} \sup_{y \in \Omega} \left| \mathrm{tr} \left( \nabla_{\bar{g}_y}^2(\boldsymbol{x}) \right) + 2 \frac{\nabla_{\bar{f}}(\boldsymbol{x})' \nabla_{\bar{g}_y}(\boldsymbol{x})}{f(\boldsymbol{x})} \right| h^2 + o(h^2) $$

*Proof of Theorem 3.2.* Let $\boldsymbol{x} \in \mathbb{S}^{d-1}$ be given. Note that the following holds for every $h > 0$ by (3.20) and (3.21).

$$ \begin{aligned} & M_{\oplus}(\boldsymbol{x}, \tilde{m}_{h,0}(\boldsymbol{x})) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) \\ & \le \left| M_{\oplus}(\boldsymbol{x}, \tilde{m}_{h,0}(\boldsymbol{x})) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) \right| + \left| \tilde{M}_{h,0}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) - \tilde{M}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,0}(\boldsymbol{x})) \right| \\ & = \int_{\Omega} \left( d^2(m_{\oplus}(\boldsymbol{x}), w) - d^2(\tilde{m}_{h,0}(\boldsymbol{x}), w) \right) \left( \frac{\mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)g_w(\boldsymbol{X})\right]}{\mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)\right]} - g_w(\boldsymbol{x}) \right) \mathrm{d}F_Y(w) \\ & \le \int_{\Omega} \left| d^2(m_{\oplus}(\boldsymbol{x}), w) - d^2(\tilde{m}_{h,0}(\boldsymbol{x}), w) \right| \left( \frac{\mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)g_w(\boldsymbol{X})\right]}{\mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)\right]} - g_w(\boldsymbol{x}) \right) \mathrm{d}F_Y(w) \end{aligned} $$
$$ \le 2\mathrm{diam}(\Omega)d(\tilde{m}_{h,0}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x})) \sup_{y \in \Omega} \left| \frac{\mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)g_y(\boldsymbol{X})\right]}{\mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)\right]} - g_y(\boldsymbol{x}) \right| \quad (3.27) $$

Also, by Lemma 3.3 with the conditions L1, D1, D2, C1, and R1, the following holds.

$$ \begin{aligned} & \sup_{y \in \Omega} \left| \frac{\mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)g_y(\boldsymbol{X})\right]}{\mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)\right]} - g_y(\boldsymbol{x}) \right| \\ & = \frac{h^2}{d-1} \frac{a_{d,1}(L)}{a_{d-2,1}(L)} \sup_{y \in \Omega} \left| \mathrm{tr}\left(\nabla^2_{\tilde{g}_y}(\boldsymbol{x})\right) + 2 \frac{\nabla_{\tilde{f}}(\boldsymbol{x})'\nabla_{\tilde{g}_y}(\boldsymbol{x})}{f(\boldsymbol{x})} \right| + o(h^2) \end{aligned} \quad (3.28) $$

Combining (3.2) and (3.13) with the conditions D1, D2, and C1, the following holds.

$$ \sup_{y \in \Omega} \left| \mathrm{tr}\left(\nabla^2_{\tilde{g}_y}(\boldsymbol{x})\right) + 2 \frac{\nabla_{\tilde{f}}(\boldsymbol{x})'\nabla_{\tilde{g}_y}(\boldsymbol{x})}{f(\boldsymbol{x})} \right| < \infty \quad (3.29) $$

Note that Theorem 3.1 with the conditions L1, D1, D2, C1, R1, and M1-0 states that there exists $\delta > 0$ such that $d(\tilde{m}_{h,0}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x})) < \eta_{\oplus}$ for every $h \in (0, \delta)$, where $\eta_{\oplus} > 0$ is the constant defined in the condition M2. Therefore, by the condition M2, the following holds for every $h \in (0, \delta)$, where $C_{\oplus} > 0$, $\beta_{\oplus} > 0$ are the constants defined in the condition M2.

$$ C_{\oplus} d(\tilde{m}_{h,0}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x}))^{\beta_{\oplus}} \le M_{\oplus}(\boldsymbol{x}, \tilde{m}_{h,0}(\boldsymbol{x})) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) \quad (3.30) $$

Therefore, by combining (3.27), (3.28), (3.29), and (3.30), the proof is done. $\square$

Next, we state a pointwise upper bound of $|\hat{M}_{h,0}(\boldsymbol{x}, y) - \tilde{M}_{h,0}(\boldsymbol{x}, y)|$ for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$ and $y \in \Omega$ as $n \to \infty$.

**Lemma 3.5.** Suppose the conditions L1, D1, D2, B1, and R1 hold. Then the following holds for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$ and $y \in \Omega$.

$$ \left| \hat{M}_{h,0}(\boldsymbol{x}, y) - \tilde{M}_{h,0}(\boldsymbol{x}, y) \right| = o_p(1) \quad \text{as} \quad n \to \infty $$

*Proof of Lemma 3.5.* Let $\boldsymbol{x} \in \mathbb{S}^{d-1}$ and $y \in \Omega$ be given. Note that the condition R1 implies the following for any given $h = h_n > 0$ and $n \in \mathbb{N}$.

$$ \left| \hat{M}_{h,0}(\boldsymbol{x}, y) - \tilde{M}_{h,0}(\boldsymbol{x}, y) \right| = \left| \frac{\sum_{i=1}^{n} L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right) d^2(Y_i, y)}{\sum_{i=1}^{n} L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)} - \frac{\mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right) d^2(Y, y)\right]}{\mathbb{E}\left[L\left(\frac{1-\boldsymbol{X}'\boldsymbol{x}}{h^2}\right)\right]} \right| $$
$$
\begin{align}
&\le \left( n^{-1} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}'_i \mathbf{x}}{h^2}\right) d^2(Y_i, y) \right) \cdot \left| \left( n^{-1} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}'_i \mathbf{x}}{h^2}\right) \right)^{-1} - \left( \mathbb{E}\left[L\left(\frac{1 - \mathbf{X}' \mathbf{x}}{h^2}\right)\right] \right)^{-1} \right| \nonumber \\
&\quad + \left( \mathbb{E}\left[L\left(\frac{1 - \mathbf{X}' \mathbf{x}}{h^2}\right)\right] \right)^{-1} \cdot \left| n^{-1} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}'_i \mathbf{x}}{h^2}\right) d^2(Y_i, y) - \mathbb{E}\left[L\left(\frac{1 - \mathbf{X}' \mathbf{x}}{h^2}\right) d^2(Y, y)\right] \right| \nonumber \\
&\le \mathrm{diam}(\Omega)^2 \left( \mathbb{E}\left[L\left(\frac{1 - \mathbf{X}' \mathbf{x}}{h^2}\right)\right] \right)^{-1} \cdot \left| n^{-1} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}'_i \mathbf{x}}{h^2}\right) - \mathbb{E}\left[L\left(\frac{1 - \mathbf{X}' \mathbf{x}}{h^2}\right)\right] \right| \nonumber \\
&\quad + \left( \mathbb{E}\left[L\left(\frac{1 - \mathbf{X}' \mathbf{x}}{h^2}\right)\right] \right)^{-1} \cdot \underbrace{\left| n^{-1} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}'_i \mathbf{x}}{h^2}\right) d^2(Y_i, y) - \mathbb{E}\left[L\left(\frac{1 - \mathbf{X}' \mathbf{x}}{h^2}\right) d^2(Y, y)\right] \right|}_{(3.A)} \tag{3.31}
\end{align}
$$

By Chebyshev's inequality, the following hold.

$$
n^{-1} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}'_i \mathbf{x}}{h^2}\right) - \mathbb{E}\left[L\left(\frac{1 - \mathbf{X}' \mathbf{x}}{h^2}\right)\right] = O_p\left(\sqrt{n^{-1}\mathrm{Var}\left[L\left(\frac{1 - \mathbf{X}' \mathbf{x}}{h^2}\right)\right]}\right) \quad (3.32)
$$

$$
(3.A) = O_p\left(\sqrt{n^{-1}\mathrm{Var}\left[L\left(\frac{1 - \mathbf{X}'\mathbf{x}}{h^2}\right) d^2(Y, y)\right]}\right) \quad (3.33)
$$

Note that (2.23) and Lemma 1.4 with the conditions L1, D1, and R1 imply the following.

$$
\mathrm{Var}\left[L\left(\frac{1-\mathbf{X}'\mathbf{x}}{h^2}\right)\right] \leq \mathbb{E}\left[L^2\left(\frac{1-\mathbf{X}'\mathbf{x}}{h^2}\right)\right] = O(h^{d-1}) \quad (3.34)
$$

$$
\mathrm{Var}\left[L\left(\frac{1 - \mathbf{X}'\boldsymbol{x}}{h^2}\right) d^2(Y, y)\right] \leq \mathbb{E}\left[L^2\left(\frac{1 - \mathbf{X}'\boldsymbol{x}}{h^2}\right) d^4(Y, y)\right] \leq \mathrm{diam}(\Omega)^4 O(h^{d-1}) \quad (3.35)
$$

By combining (3.32), (3.33), (3.34), and (3.35), we get the following.

$$
n^{-1} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}'_i \mathbf{x}}{h^2}\right) - \mathbb{E}\left[L\left(\frac{1 - \mathbf{X}' \mathbf{x}}{h^2}\right)\right] = O_p\left(n^{-\frac{1}{2}} h^{\frac{d-1}{2}}\right), \quad (3.A) = O_p\left(n^{-\frac{1}{2}} h^{\frac{d-1}{2}}\right) \quad (3.36)
$$

Therefore, combining (3.16), (3.17), (3.31), and (3.36) with the condition D2, we get the
following.

$$
\left( \mathbb{E} \left[ L \left( \frac{1 - \mathbf{X}' \mathbf{x}}{h^2} \right) \right] \right)^{-1} = O(h^{-(d-1)}), \quad \left| \hat{M}_{h,0}(\mathbf{x}, y) - \tilde{M}_{h,0}(\mathbf{x}, y) \right| = O_p(n^{-\frac{1}{2}} h^{-\frac{d-1}{2}}) \quad (3.37)
$$

The proof is done by the condition B1. □

By using Lemma 3.5, we can get a uniform version which is stated as the below. The proof is
the same as Lemma 3.6 of Im et al. (2024).

**Lemma 3.6.** Suppose the conditions L1, D1, D2, B1, and R1 hold. Then the following holds for every **x** ∈ S<sup>d-1</sup>.

$$
\sup_{y \in \Omega} \left| \hat{M}_{h,0}(\mathbf{x}, y) - \tilde{M}_{h,0}(\mathbf{x}, y) \right| = o_p(1) \quad \text{as } n \to \infty
$$
The next Theorem is a direct consequence of Lemma 3.6, which states that $d(\hat{m}_{h,0}(\boldsymbol{x}), \tilde{m}_{h,0}(\boldsymbol{x}))$ converges to 0 in probability for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$. The proof is the same as Theorem 3.3 of Im et al. (2024).

**Theorem 3.3** (Convergence of the stochastic deviation of $\hat{m}_{h,0}$). Suppose the conditions L1, D1, D2, B1, R1, and M1-0 hold. Then the following holds for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$.

$$d(\hat{m}_{h,0}(\boldsymbol{x}), \tilde{m}_{h,0}(\boldsymbol{x})) = o_p(1) \quad \text{as} \quad n \to \infty$$

Theorem 3.1 and Theorem 3.3 imply the consistency of $\hat{m}_{h,0}(\boldsymbol{x})$ for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$. In order to get the convergence rate of $\hat{m}_{h,0}(\boldsymbol{x})$ for any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$, the following Lemmas are necessary, which are from Theorem 2.7.11 and Theorem 2.14.2 of van der Vaart and Wellner (1996). These Lemmas are also used in Im et al. (2024).

**Lemma 3.7.** Let a metric space $(T, d_T)$ be given. Let $\mathcal{H} = \{h_t : t \in T\}$ be a class of real functions defined on a domain $\mathcal{X}$ such that the following holds for some fixed function $H: \mathcal{X} \to \mathbb{R}$.

$$|h_s(x) - h_t(x)| \le d_T(s, t)H(x), \quad \forall s, t \in T, \forall x \in \mathcal{X}$$

Then for any norm $\|\cdot\|$ of $\mathcal{H}$,

$$N_{\Box}(2\epsilon\|H\|, \mathcal{H}, \|\cdot\|) \le N(\epsilon, T, d_T), \quad \forall \epsilon > 0$$

where $N_{\Box}(\delta, \mathcal{H}, \|\cdot\|)$ and $N(\delta, T, d_T)$ denote the $\delta$-bracketing number of $(\mathcal{H}, \|\cdot\|)$ and the $\delta$-covering number of $(T, d_T)$ respectively for every $\delta > 0$.

**Lemma 3.8.** Let a measurable space $(\mathcal{X}, \mathcal{A})$ and $\mathcal{X}$-valued i.i.d. random variables $Z_1, Z_2, \dots, Z_n$ with the common distribution $P$ be given. Let $\mathcal{H}$ be a class of measurable functions $h: \mathcal{X} \to \mathbb{R}$ with the measurable envelope function $H$. Then there exists a universal constant $C > 0$ such that

$$\mathbb{E}\left[\sup_{h \in \mathcal{H}} \left|\frac{1}{n} \sum_{i=1}^{n} h(Z_i) - \int_{\mathcal{X}} hdP\right|\right] \le \frac{C}{\sqrt{n}} \|H\|_{L_2(P)} \int_0^1 \sqrt{1 + \log N_{\Box}(\epsilon \|H\|_{L_2(P)}, \mathcal{H}, L_2(P))} d\epsilon$$

where $N_{\Box}(\delta, \mathcal{H}, L_2(P))$ denotes the $\delta$-bracketing number of $(\mathcal{H}, L_2(P))$ for every $\delta > 0$.

Now we state the theorem about the convergence rate of $\hat{m}_{h,0}(\boldsymbol{x})$ for any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$.

**Theorem 3.4** (Convergence rate of the stochastic deviation of $\hat{m}_{h,0}$). Suppose the conditions L1, D1, D2, B1, R1, M1-0, and M3-0 hold. Then the following holds for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$, where $\tilde{\beta}_0 > 1$ is the constant defined in M3-0.

$$d(\hat{m}_{h,0}(\boldsymbol{x}), \tilde{m}_{h,0}(\boldsymbol{x}))^{\tilde{\beta}_0 - 1} = O_p(n^{-\frac{1}{2}}h^{-\frac{d-1}{2}}) \quad \text{as} \quad n \to \infty$$
Proof of Theorem 3.4. Let $\boldsymbol{x} \in \mathbb{S}^{d-1}$ be given. Similar to Lemma 3.6 of Im et al. (2024), we define functions $\hat{T}_{h,0}: \mathbb{S}^{d-1} \times \Omega \to \mathbb{R}$ and $U_{h,0}: \mathbb{S}^{d-1} \times \Omega \times \Omega \to \mathbb{R}$ as the following for every $h = h_n > 0$ and $n \in \mathbb{N}$.

$$ \begin{aligned} \hat{T}_{h,0}(\boldsymbol{x}, y) &:= \hat{M}_{h,0}(\boldsymbol{x}, y) - \tilde{M}_{h,0}(\boldsymbol{x}, y), \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1}, \forall y \in \Omega \\ U_{h,0}(\boldsymbol{z}, w, y) &:= L\left(\frac{1 - \boldsymbol{z}'\boldsymbol{x}}{h^2}\right) D_{h,0}(\boldsymbol{x}, w, y), \quad \forall \boldsymbol{z} \in \mathbb{S}^{d-1}, \forall w, y \in \Omega \end{aligned} $$

Just as Lemma 3.6 of Im et al. (2024), the following holds for every $y \in \Omega$, $h = h_n > 0$, and $n \in \mathbb{N}$.

$$ \begin{aligned} & \mathbb{E}\left[\left|L\left(\frac{1 - \mathbf{X}'\boldsymbol{x}}{h^2}\right)\right|\left|\hat{T}_{h,0}(\boldsymbol{x}, y) - \hat{T}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,0}(\boldsymbol{x}))\right|\right] \\ & \le 2\mathrm{diam}(\Omega)d(y, \tilde{m}_{h,0}(\boldsymbol{x}))\left|n^{-1}\sum_{i=1}^{n}L\left(\frac{1 - \mathbf{X}'_i\boldsymbol{x}}{h^2}\right) - \mathbb{E}\left[\left|L\left(\frac{1 - \mathbf{X}'\boldsymbol{x}}{h^2}\right)\right|\right]\right| \\ & \quad + \left|n^{-1}\sum_{i=1}^{n}U_{h,0}(\mathbf{X}_i, Y_i, y) - \mathbb{E}\left[U_{h,0}(\mathbf{X}, Y, y)\right]\right| \end{aligned} \quad (3.38) $$

First, note that the following holds by Markov's inequality and (3.34) with the conditions L1, D1, D2, and B1.

$$ \mathbb{E}\left[\left|n^{-1}\sum_{i=1}^{n}L\left(\frac{1 - \mathbf{X}'_i\boldsymbol{x}}{h^2}\right) - \mathbb{E}\left[L\left(\frac{1 - \mathbf{X}'\boldsymbol{x}}{h^2}\right)\right]\right|\right] \le \sqrt{\mathrm{Var}\left[n^{-1}\sum_{i=1}^{n}L\left(\frac{1 - \mathbf{X}'_i\boldsymbol{x}}{h^2}\right)\right]} = O(n^{-\frac{1}{2}}h^{\frac{d-1}{2}}) $$

Therefore, there exist constants $L_1 > 0$ and $N_1 \in \mathbb{N}$ such that the following holds.

$$ n \ge N_1 \quad \Rightarrow \quad \mathbb{E}\left[\left|n^{-1}\sum_{i=1}^{n}L\left(\frac{1 - \mathbf{X}'_i\boldsymbol{x}}{h^2}\right) - \mathbb{E}\left[L\left(\frac{1 - \mathbf{X}'\boldsymbol{x}}{h^2}\right)\right]\right|\right] \le L_1 n^{-\frac{1}{2}}h^{\frac{d-1}{2}} \quad (3.39) $$

The next step is to get a nice upper bound of the second term of (3.38). Following the steps used in Lemma 3.6 of Im et al. (2024) with Lemma 3.7 and Lemma 3.8, there exist constants $L_2 > 0$ and $N_2 \in \mathbb{N}$ such that the following holds for every $\delta > 0$.

$$ \begin{aligned} n \ge N_2 \quad \Rightarrow \quad & \mathbb{E}\left[\sup_{y \in B_d(\tilde{m}_{h,0}(\boldsymbol{x}), \delta)} \left|n^{-1}\sum_{i=1}^{n} U_{h,0}(\mathbf{X}_i, Y_i, y) - \mathbb{E}\left[U_{h,0}(\mathbf{X}, Y, y)\right]\right|\right] \\ & \le 2L_2\delta n^{-\frac{1}{2}}h^{\frac{d-1}{2}} \int_0^1 \sqrt{1 + \log N(\delta\epsilon, B_d(\tilde{m}_{h,0}(\boldsymbol{x}), \delta), d)} d\epsilon \end{aligned} \quad (3.40) $$

Note that by the condition R2, there exist constants $L_3 > 0$ and $\delta_1 > 0$ such that the following holds.

$$ 0 < \delta < \delta_1 \quad \Rightarrow \quad \sup_{w \in \Omega} \int_0^1 \sqrt{1 + \log N(\delta\epsilon, B_d(w, \delta), d)} d\epsilon \le L_3 \quad (3.41) $$
Therefore, by (3.40) and (3.41), the following holds.

$$
\begin{align}
n \ge N_2, \ 0 < \delta < \delta_1 \quad &\Rightarrow \quad \mathbb{E} \left[ \sup_{y \in B_d(\tilde{m}_{h,0}(\boldsymbol{x}), \delta)} \left| n^{-1} \sum_{i=1}^n U_{h,0}(\boldsymbol{X}_i, Y_i, y) - \mathbb{E} \left[ U_{h,0}(\boldsymbol{X}, Y, y) \right] \right| \right] \nonumber \\
&\le 2L_2 L_3 \delta n^{-\frac{1}{2}} h^{\frac{d-1}{2}} \tag{3.42}
\end{align}
$$

In addition, by (3.37) with the conditions L1, D1, D2, and B1, there exist constants $L_4 > 0$ and $N_3 \in \mathbb{N}$ such that the following holds.

$$
n \ge N_3 \quad \Rightarrow \quad \left( \mathbb{E} \left[ L \left( \frac{1 - \mathbf{X}'\mathbf{x}}{h^2} \right) \right] \right)^{-1} \le L_4 h^{-(d-1)} \quad (3.43)
$$

Now define constants $N_4 := \max\{N_1, N_2, N_3\}$ and $L_5 := L_4(2\mathrm{diam}(\Omega)L_1 + 2L_2L_3)$. Then the following holds by (3.38), (3.39), (3.42), and (3.43).

$$
n \ge N_4, \ 0 < \delta < \delta_1 \quad \Rightarrow \quad \mathbb{E} \left[ \sup_{y \in B_d(\tilde{m}_{h,0}(\boldsymbol{x}), \delta)} \left| \hat{T}_{h,0}(\boldsymbol{x}, y) - \hat{T}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,0}(\boldsymbol{x})) \right| \right] \le L_5 \delta n^{-\frac{1}{2}} h^{-\frac{d-1}{2}} \quad (3.44)
$$

Let $\epsilon > 0$ be given. Suppose constants $\tilde{h}_0 > 0$, $\tilde{\eta}_0 > 0$, $\tilde{C}_0 > 0$, $\tilde{\beta}_0 > 0$ are from the condition M3-0. since $\sum_{k=1}^{\infty} 4^{-k(\tilde{\beta}_0-1)/\tilde{\beta}_0}$ converges, there exists a constant $M \in \mathbb{N}$ such that the following holds.

$$
\frac{4L_5}{\tilde{C}_0} \sum_{k=M}^{\infty} \left( 4^{-\frac{\tilde{\beta}_0-1}{\tilde{\beta}_0}} \right)^k < \frac{\epsilon}{2} \quad (3.45)
$$

Also, by Theorem 3.3 with the conditions L1, D1, D2, B1, R1, and M1-0, there exists $N_5 \in \mathbb{N}$ such that the following holds.

$$
n \ge N_5 \quad \Rightarrow \quad \mathbb{P} \left( d(\hat{m}_{h,0}(\mathbf{x}), \tilde{m}_{h,0}(\mathbf{x})) > \frac{\min\{\tilde{\eta}_0, \delta_1\}}{2} \right) < \frac{\epsilon}{2}, \quad 0 < h < \tilde{h}_0 \quad (3.46)
$$

For the convenience, define $t_n := (nh^{d-1})^{\tilde{\beta}_0/(4\tilde{\beta}_0-4)}$ for every $n \in \mathbb{N}$. Also, we define events $E_n$, $A_{n,k}$, $B_n$ as the following for every $n, k \in \mathbb{N}$. Note that $n \ge N_5$ implies $\mathbb{P}(B_n^c) < \epsilon/2$.

$$
\begin{align*}
E_n &:= \left\{ d(\hat{m}_{h,0}(\boldsymbol{x}), \tilde{m}_{h,0}(\boldsymbol{x})) > 2^{\frac{2M}{\tilde{\beta}_0} (nh^{d-1})^{\frac{-1}{2(\tilde{\beta}_0-1)}}} \right\} = \left\{ t_n d(\hat{m}_{h,0}(\boldsymbol{x}), \tilde{m}_{h,0}(\boldsymbol{x}))^{\frac{\tilde{\beta}_0}{2}} > 2^M \right\}, \\
A_{n,k} &:= \left\{ 2^{k-1} < t_n d(\hat{m}_{h,0}(\boldsymbol{x}), \tilde{m}_{h,0}(\boldsymbol{x}))^{\frac{\tilde{\beta}_0}{2}} \le 2^k \right\}, \quad B_n := \left\{ d(\hat{m}_{h,0}(\boldsymbol{x}), \tilde{m}_{h,0}(\boldsymbol{x})) \le \frac{\min\{\tilde{\eta}_0, \delta_1\}}{2} \right\}
\end{align*}
$$

Since $E_n = \cup_{k=M+1}^{\infty} A_{n,k}$ for every $n \in \mathbb{N}$, note that the following holds by the union bound for every $n \in \mathbb{N}$.

$$
\mathbb{P}(E_n) \le \mathbb{P}(E_n \cap B_n) + \mathbb{P}(B_n^c) \le \sum_{k=M+1}^{\infty} \mathbb{P}(A_{n,k} \cap B_n) + \mathbb{P}(B_n^c) \quad (3.47)
$$
Now define $r_{n,k} := \min\{4^{k/\tilde{\beta}_0}t_n^{-2/\tilde{\beta}_0}, \delta_1/2\}$ for every $n, k \in \mathbb{N}$. Following steps of Lemma 3.6 of Im et al. (2024) with Markov's inequality and (3.44), the following holds for every $n, k \in \mathbb{N}$ such that $n \ge \max\{N_4, N_5\}$ and $k \ge M + 1$.

$$ \mathbb{P}(A_{n,k} \cap B_n) \le \frac{t_n^2}{\tilde{C}_0 4^{k-1}} \mathbb{E} \left[ \sup_{y \in B_d(\tilde{m}_{h,0}(\mathbf{x}), r_{n,k})} \left| \hat{T}_{h,0}(\mathbf{x}, y) - \hat{T}_{h,0}(\mathbf{x}, \tilde{m}_{h,0}(\mathbf{x})) \right| \right] \le \frac{4L_5}{\tilde{C}_0} \left( 4^{-\frac{\tilde{\beta}_0-1}{\tilde{\beta}_0}} \right)^k \quad (3.48) $$

Now define $N := \max\{N_4, N_5\}$. Then by combining (3.45), (3.46), (3.47), and (3.48), we get the following.

$$ n \ge N \Rightarrow \mathbb{P}(E_n) \le \frac{4L_5}{\tilde{C}_0} \sum_{k=M+1}^{\infty} 4^{-\frac{k(\tilde{\beta}_0-1)}{\tilde{\beta}_0}} + \mathbb{P}(B_n^c) < \epsilon $$

Since $\epsilon > 0$ is arbitrary, this implies $d(\hat{m}_{h,0}(\mathbf{x}), \tilde{m}_{h,0}(\mathbf{x}))^{\tilde{\beta}_0-1} = O_p(n^{-\frac{1}{2}h^{-\frac{d-1}{2}}})$ as $n \to \infty$. $\square$

Finally, by Theorem 3.2 and Theorem 3.4, we can get the convergence rate of $\hat{m}_{h,0}(\mathbf{x})$ for every $\mathbf{x} \in \mathbb{S}^{d-1}$. Also, we can get the optimal bandwidth sequence for $h = h_n > 0$.

**Theorem 3.5.** Suppose the conditions L1, D1, D2, C1, B1, R1, R2, M1-0, M2, and M3-0 hold. Then the following holds for every $\mathbf{x} \in \mathbb{S}^{d-1}$, where $\beta_{\oplus} > 1$ and $\tilde{\beta}_0 > 1$ are the constants defined in M2 and M3-0 respectively.

$$ d(\hat{m}_{h,0}(\mathbf{x}), m_{\oplus}(\mathbf{x})) = O(h^{\frac{2}{\beta_{\oplus}-1}}) + O_p((nh^{d-1})^{\frac{-1}{2(\tilde{\beta}_0-1)}}) \quad \text{as } n \to \infty $$

Also, among bandwidth sequences $h = h_n = n^{-\gamma}$ for any constant $\gamma > 0$, the optimal sequence is obtained when $\gamma = \frac{\beta_{\oplus}-1}{4(\tilde{\beta}_0-1)+(d-1)(\beta_{\oplus}-1)}$, and the following holds for every $\mathbf{x} \in \mathbb{S}^{d-1}$ in this case.

$$ d(\hat{m}_{h,0}(\mathbf{x}), m_{\oplus}(\mathbf{x})) = O_p(n^{\frac{-2}{4(\tilde{\beta}_0-1)+(d-1)(\beta_{\oplus}-1)}}) \quad \text{as } n \to \infty $$

# 4 The local linear Fréchet regression for random objects with spherical predictors

In this section, we introduce the local linear Fréchet regression estimator and prove some properties about the local linear Fréchet regression estimator for random objects with spherical predictors. Assume that the setting is the same as that of section 3.

Now we consider the local linear Fréchet regression. The main idea is to consider the case when $\Omega = \mathbb{R}$ and extend to general metric spaces. Let random samples $(\mathbf{X}_i, Y_i) \sim F$, $i = 1, 2, \dots, n$ be
given. Also, let $\boldsymbol{x} \in \mathbb{S}^{d-1}$ and $\boldsymbol{B}_x \in \mathcal{B}_x$ be given. Note that there exist $\theta_i \in [0, \pi]$ and $\xi_i \in \mathbb{S}^{d-2}$ for every $i = 1, 2, \dots, n$ such that the following holds, where $\Phi_{\boldsymbol{B}_x}$ is defined by (1.6).

$$ \mathbf{X}_i = \Phi_{\boldsymbol{B}_x}(\theta_i, \xi_i), \quad \forall i = 1, 2, \dots, n \qquad (4.1) $$

At the beginning of the section, we consider the case when $\Omega = \mathbb{R}$. Note that the local linear regression estimator $\hat{\alpha}_h(\boldsymbol{x})$ is defined by the following, where $L$ is the directional kernel, $h = h_n > 0$ is the bandwidth parameter depending on $n$. This estimator is based on Marco Di Marzio and Taylor (2014).

$$ (\hat{\alpha}_h(\boldsymbol{x}), \hat{\beta}_h(\boldsymbol{x})) = \underset{\alpha \in \mathbb{R}, \beta \in \mathbb{R}^d}{\operatorname{argmin}} n^{-1} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}_i' \boldsymbol{x}}{h^2}\right) \left(Y_i - \alpha - \theta_i (\boldsymbol{B}_x \boldsymbol{\xi}_i)' \boldsymbol{\beta}\right)^2 \\ \text{Subject to } \boldsymbol{x}' \boldsymbol{\beta} = 0 \qquad (4.2) $$

Note that $\{\boldsymbol{\beta} \in \mathbb{R}^d : \boldsymbol{x}'\boldsymbol{\beta} = 0\} = \{\boldsymbol{\beta} \in \mathbb{R}^d : \boldsymbol{\beta} = \boldsymbol{B}_x\boldsymbol{\gamma}, \exists \boldsymbol{\gamma} \in \mathbb{R}^{d-1}\}$. Therefore, by substituting $\boldsymbol{\beta} = \boldsymbol{B}_x\boldsymbol{\gamma}$, (4.2) is equivalent to the following since $\boldsymbol{B}_x'\boldsymbol{B}_x = \mathbf{I}_d$.

$$ (\hat{\alpha}_h(\boldsymbol{x}), \hat{\gamma}_h(\boldsymbol{x})) = \underset{\alpha \in \mathbb{R}, \gamma \in \mathbb{R}^d}{\operatorname{argmin}} n^{-1} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}_i' \boldsymbol{x}}{h^2}\right) \left(Y_i - \alpha - \theta_i \boldsymbol{\xi}_i' \boldsymbol{\gamma}\right)^2 \qquad (4.3) $$

Using the fact that $\boldsymbol{B}_x\boldsymbol{B}_x' = \mathbf{I}_d - \boldsymbol{x}\boldsymbol{x}'$ and $\cos\theta_i = \mathbf{X}_i'\boldsymbol{x}$ for every $i = 1, 2, \dots, n$, we get the following by plugging at (1.6).

$$ \theta_i = \arccos(\mathbf{X}_i'\boldsymbol{x}), \quad \boldsymbol{\xi}_i = \frac{\boldsymbol{B}_x'\mathbf{X}_i}{\sqrt{1 - (\mathbf{X}_i'\boldsymbol{x})^2}}, \quad \forall i = 1, 2, \dots, n \qquad (4.4) $$

Considering (4.3) and (4.4), we define a map $\boldsymbol{H}_x : \mathbb{S}^{d-1} \to \mathbb{R}^d$ as the following.

$$ \boldsymbol{H}_x(\boldsymbol{z}) := \frac{\arccos(\boldsymbol{z}'\boldsymbol{x})}{\sqrt{1 - (\boldsymbol{z}'\boldsymbol{x})^2}}\boldsymbol{z}, \quad \forall \boldsymbol{z} \in \mathbb{S}^{d-1} \qquad (4.5) $$

Combining (4.4) and (4.5), (4.3) is equivalent as the following.

$$ (\hat{\alpha}_h(\boldsymbol{x}), \hat{\gamma}_h(\boldsymbol{x})) = \underset{\alpha \in \mathbb{R}, \gamma \in \mathbb{R}^d}{\operatorname{argmin}} n^{-1} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}_i'\boldsymbol{x}}{h^2}\right) \left(Y_i - \alpha - \boldsymbol{H}_x(\mathbf{X}_i)'\boldsymbol{B}_x\boldsymbol{\gamma}\right)^2 \qquad (4.6) $$

Note that $\hat{\alpha}_h(\boldsymbol{x})$ is an M-estimator of $\tilde{\alpha}_h(\boldsymbol{x})$, which is defined by the following for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$, $\boldsymbol{B}_x \in \mathcal{B}_x$, $h = h_n > 0$, and $n \in \mathbb{N}$.

$$ (\tilde{\alpha}_h(\boldsymbol{x}), \tilde{\gamma}_h(\boldsymbol{x})) = \underset{\alpha \in \mathbb{R}, \gamma \in \mathbb{R}^d}{\operatorname{argmin}} \mathbb{E}\left[L\left(\frac{1 - \mathbf{X}'\boldsymbol{x}}{h^2}\right)\left(Y - \alpha - \boldsymbol{H}_x(\mathbf{X})'\boldsymbol{B}_x\boldsymbol{\gamma}\right)^2\right] \qquad (4.7) $$

Note that (4.6) and (4.7) depend on $\boldsymbol{B}_x \in \mathcal{B}_x$ for any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$. Therefore, it is necessary to prove that $\tilde{\alpha}_h(\boldsymbol{x})$ and $\hat{\alpha}_h(\boldsymbol{x})$ is invariant of the choice of $\boldsymbol{B}_x \in \mathcal{B}_x$ for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$. To prove this, we use the definition and a Lemma respect to Moore-Penrose inverse.
**Remark 4.1.** Let a matrix $A \in \mathbb{R}^{m \times p}$ be given. Then the Moore-Penrose inverse of $A$ is defined as a matrix $A^+ \in \mathbb{R}^{p \times m}$ satisfying the following criteria.

$$AA^+A = A, \quad A^+AA^+ = A^+, \quad (AA^+)' = AA^+, \quad (A^+A)' = A^+A$$

Also, for any given $A \in \mathbb{R}^{m \times p}$, the Moore-Penrose inverse $A^+ \in \mathbb{R}^{p \times m}$ of $A$ exists uniquely.

**Lemma 4.1.** For any given matrices $A \in \mathbb{R}^{m \times p}$ and $B \in \mathbb{R}^{m \times m}$ such that $A'A = I_p$, the following holds.

$$(AA'BAA')^+ = A(A'BAA)^+A'$$

*Proof of Lemma 4.1.* Let matrices $A \in \mathbb{R}^{m \times p}$ and $B \in \mathbb{R}^{m \times m}$ be given such that $A'A = I_p$. For the convenience, denote $P := AA'BAA'$ and $Q := A(A'BAA)^+A'$. Then the following hold.

$$PQP = A(A'BAA)(A'BAA)^+(A'BAA)A' = P \quad (4.8)$$

$$QPQ = A(A'BAA)^+(A'BAA)(A'BAA)^+A' = Q \quad (4.9)$$

$$(PQ)' = \left( A(A'BAA)(A'BAA)^+A' \right)' = A\left( (A'BAA)(A'BAA)^+ \right)'A' = PQ \quad (4.10)$$

$$(QP)' = \left( A(A'BAA)^+(A'BAA)A' \right)' = A\left( (A'BAA)^+(A'BAA) \right)'A' = QP \quad (4.11)$$

Combining (4.8), (4.9), (4.10), and (4.11), the proof is done. $\square$

For the convenience, similar to Im et al. (2024), we define $\tilde{\mu}_{h,0}(x)$, $\tilde{\mu}_{h,1}(x)$, $\tilde{\mu}_{h,2}(x)$, $\tilde{\Sigma}_h(x)$, $\tilde{\sigma}_h(x)$ and their estimators $\hat{\mu}_{h,0}(x)$, $\hat{\mu}_{h,1}(x)$, $\hat{\mu}_{h,2}(x)$, $\hat{\Sigma}_h(x)$, $\hat{\sigma}_h(x)$ as the following for every $x \in \mathbb{S}^{d-1}$, $h = h_n > 0$, and $n \in \mathbb{N}$.

$$\tilde{\mu}_{h,0}(x) := \mathbb{E}\left[L\left(\frac{1 - X'x}{h^2}\right)\right], \quad \tilde{\mu}_{h,1}(x) := \mathbb{E}\left[L\left(\frac{1 - X'x}{h^2}\right)H_x(X)\right] \quad (4.12)$$

$$\tilde{\mu}_{h,2}(x) := \mathbb{E}\left[L\left(\frac{1 - X'x}{h^2}\right)H_x(X)H_x(X)'\right] \quad (4.13)$$

$$\tilde{\Sigma}_h(x) = (I_d - xx')\tilde{\mu}_{h,2}(x)(I_d - xx'), \quad \tilde{\sigma}_h(x) = \tilde{\mu}_{h,0}(x) - \tilde{\mu}_{h,1}(x)'\tilde{\Sigma}_h(x)^+\tilde{\mu}_{h,1}(x) \quad (4.14)$$

$$\hat{\mu}_{h,0}(x) := \frac{1}{n} \sum_{i=1}^{n} L\left(\frac{1 - X_i'x}{h^2}\right), \quad \hat{\mu}_{h,1}(x) := \frac{1}{n} \sum_{i=1}^{n} L\left(\frac{1 - X_i'x}{h^2}\right) H_x(X_i) \quad (4.15)$$

$$\hat{\mu}_{h,2}(x) := \frac{1}{n} \sum_{i=1}^{n} L\left(\frac{1 - X_i'x}{h^2}\right) H_x(X_i)H_x(X_i)' \quad (4.16)$$

$$\hat{\Sigma}_h(x) = (I_d - xx')\hat{\mu}_{h,2}(x)(I_d - xx'), \quad \hat{\sigma}_h(x) = \hat{\mu}_{h,0}(x) - \hat{\mu}_{h,1}(x)'\hat{\Sigma}_h(x)^+\hat{\mu}_{h,1}(x) \quad (4.17)$$
Using (4.12), (4.13), (4.15), (4.16), the normal equations of (4.7) and (4.6) are the following
for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$, $\boldsymbol{B}_{\boldsymbol{x}} \in \mathcal{B}_{\boldsymbol{x}}$, $h = h_n > 0$, and $n \in \mathbb{N}$.

$$
\left[
\begin{array}{c}
\mathbb{E}\left[L\left(\frac{1-\mathbf{X}'\boldsymbol{x}}{h^2}\right)Y\right] \\
\mathbf{B}'_{\boldsymbol{x}}\mathbb{E}\left[L\left(\frac{1-\mathbf{X}'\boldsymbol{x}}{h^2}\right)\mathbf{H}_{\boldsymbol{x}}(\mathbf{X})Y\right]
\end{array}
\right]
=
\left[
\begin{array}{cc}
\tilde{\mu}_{h,0}(\boldsymbol{x}) & \tilde{\mu}_{h,1}(\boldsymbol{x})'\mathbf{B}_{\boldsymbol{x}} \\
\mathbf{B}'_{\boldsymbol{x}}\tilde{\mu}_{h,1}(\boldsymbol{x}) & \mathbf{B}'_{\boldsymbol{x}}\tilde{\mu}_{h,2}(\boldsymbol{x})\mathbf{B}_{\boldsymbol{x}}
\end{array}
\right]
\left[
\begin{array}{c}
\tilde{\alpha}_h(\boldsymbol{x}) \\
\tilde{\gamma}_h(\boldsymbol{x})
\end{array}
\right]
\quad (4.18)
$$

$$
\left[
\begin{array}{c}
n^{-1} \sum_{i=1}^{n} L\left(\frac{1-\mathbf{X}'_{i} \mathbf{x}}{h^{2}}\right) Y_{i} \\
\mathbf{B}'_{x}\left(n^{-1} \sum_{i=1}^{n} L\left(\frac{1-\mathbf{X}'_{i} \mathbf{x}}{h^{2}}\right) \mathbf{H}_{x}(\mathbf{X}_{i}) Y_{i}\right)
\end{array}
\right]
=
\left[
\begin{array}{cc}
\hat{\mu}_{h,0}(\mathbf{x}) & \hat{\mu}_{h,1}(\mathbf{x})^{\prime} \mathbf{B}_{x} \\
\mathbf{B}'_{x} \hat{\mu}_{h,1}(\mathbf{x}) & \mathbf{B}'_{x} \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_{x}
\end{array}
\right]
\left[
\begin{array}{c}
\hat{\alpha}_{h}(\mathbf{x}) \\
\hat{\gamma}_{h}(\mathbf{x})
\end{array}
\right]
\quad (4.19)
$$

Using the fact that $\mathbf{B}_x \mathbf{B}_x' = \mathbf{I}_d - \mathbf{x}\mathbf{x}', \mathbf{B}_x' \mathbf{B}_x = \mathbf{I}_{d-1}$ for every $\mathbf{x} \in \mathbb{S}^{d-1}$ and $\mathbf{B}_x \in \mathcal{B}_x$, the following hold by Lemma 4.1.

$$
\tilde{\Sigma}_h(\mathbf{x})^+ = \mathbf{B}_x (\mathbf{B}_x' \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \mathbf{B}_x', \quad \hat{\Sigma}_h(\mathbf{x})^+ = \mathbf{B}_x (\mathbf{B}_x' \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \mathbf{B}_x' \quad (4.20)
$$

Combining (4.14), (4.17), (4.18), (4.19), and (4.20), $\tilde{\alpha}_h(\mathbf{x})$ and $\hat{\alpha}_h(\mathbf{x})$ can be evaluated by the following for every $\mathbf{x} \in \mathbb{S}^{d-1}$, $h = h_n > 0$, and $n \in \mathbb{N}$ given that $\mathbf{B}_x' \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x$ and $\mathbf{B}_x' \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x$ are invertible for some $\mathbf{B}_x \in \mathcal{B}_x$.

$$
\tilde{\sigma}_h(\mathbf{x})\tilde{\alpha}_h(\mathbf{x}) = \mathbb{E}\left[L\left(\frac{1-\mathbf{X}'\mathbf{x}}{h^2}\right)Y\right] - \tilde{\mu}_{h,1}(\mathbf{x})'\tilde{\Sigma}_h(\mathbf{x})^+\mathbb{E}\left[L\left(\frac{1-\mathbf{X}'\mathbf{x}}{h^2}\right)\mathbf{H}_x(\mathbf{X})Y\right] \quad (4.21)
$$

$$
\hat{\sigma}_h(\mathbf{x})\hat{\alpha}_h(\mathbf{x}) = \frac{1}{n} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}'_i \mathbf{x}}{h^2}\right) Y_i - \hat{\mu}_{h,1}(\mathbf{x})' \hat{\Sigma}_h(\mathbf{x})^+ \left( \frac{1}{n} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}'_i \mathbf{x}}{h^2}\right) \mathbf{H}_x(\mathbf{X}_i) Y_i \right) \quad (4.22)
$$

Therefore, (4.21) and (4.22) imply that $\tilde{\alpha}_h(\mathbf{x})$ and $\hat{\alpha}_h(\mathbf{x})$ is invariant of the choice of $\mathbf{B}_x \in \mathcal{B}_x$
for every $\mathbf{x} \in \mathbb{S}^{d-1}$. Now it is necessary to ensure that $\mathbf{B}_x' \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x$, $\mathbf{B}_x' \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x$ are invertible
and $\tilde{\sigma}_h(\mathbf{x})$, $\hat{\sigma}_h(\mathbf{x})$ are positive almost surely for every $\mathbf{x} \in \mathbb{S}^{d-1}$, $\mathbf{B}_x \in \mathcal{B}_x$, and for any sufficiently
small $h = h_n > 0$ for sufficiently large $n \in \mathbb{N}$. To prove this, we need a stronger version of
Lemma 2.2. The first step is to state and prove an extension of Lemma 1.3, which is the following
Lemma.

**Lemma 4.2.** For any $\boldsymbol{\xi} \in \mathbb{R}^{d-1}$, denote $\xi_i$ as the $i$-th component of $\boldsymbol{\xi}$ for every $i = 1, 2, \dots, d-1$.
Then the following holds for every $\alpha_1, \alpha_2, \dots, \alpha_{d-1} \in \mathbb{N}_0$ and $\beta_1, \beta_2, \dots, \beta_{d-1} \in \mathbb{N}_0$ such that
$\sum_{i=1}^{d-1} \alpha_i = 3$ and $\sum_{i=1}^{d-1} \beta_i = 4$, where $\mathcal{A}_{d-2}$ is defined as (1.5).

$$
\int_{\mathbb{S}^{d-2}} \left( \prod_{i=1}^{d-1} \xi_i^{\alpha_i} \right) \omega_{d-2}(\mathrm{d}\boldsymbol{\xi}) = 0, \quad \int_{\mathbb{S}^{d-2}} \left( \prod_{i=1}^{d-1} \xi_i^{\beta_i} \right) \omega_{d-2}(\mathrm{d}\boldsymbol{\xi}) = \begin{cases} \frac{3A_{d-2}}{d^2-1} & (\exists i \text{ s.t. } \beta_i = 4) \\ \frac{A_{d-2}}{d^2-1} & (\exists i \neq j \text{ s.t. } \beta_i = \beta_j = 2) \\ 0 & (\text{Otherwise}) \end{cases} \tag{4.23}
$$
Therefore, for every $\mathbf{v} \in \mathbb{R}^{d-1}$ and $\mathbf{M} \in \mathbb{R}^{(d-1)\times(d-1)}$, the following holds, where $\mathbf{O}_{d-1}$ denotes the $(d-1) \times (d-1)$ zero matrix.

$$ \int_{\mathbb{S}^{d-2}} \xi(\xi'\mathbf{v})\omega_{d-2}(d\xi) = \frac{\mathcal{A}_{d-2}}{d-1}\mathbf{v}, \quad \int_{\mathbb{S}^{d-2}} \xi(\xi'M\xi)\omega_{d-2}(d\xi) = \mathbf{0}_{d-1} \qquad (4.24) $$

$$ \int_{\mathbb{S}^{d-2}} (\xi\xi')(\xi'\mathbf{v})\omega_{d-2}(d\xi) = \mathbf{O}_{d-1} \qquad (4.25) $$

$$ \int_{\mathbb{S}^{d-2}} (\xi\xi')(\xi'M\xi)\omega_{d-2}(d\xi) = \frac{\mathcal{A}_{d-2}}{d^2-1}(M + M' + \mathrm{tr}(M)I_{d-1}) \qquad (4.26) $$

**Proof of Lemma 4.2.** Since it is trivial for the case when $d=2$, we only consider the case when $d \ge 3$. Let $\alpha_1, \alpha_2, \dots, \alpha_{d-1} \in \mathbb{N}_0$ and $\beta_1, \beta_2, \dots, \beta_{d-1} \in \mathbb{N}_0$ be given such that $\sum_{i=1}^{d-1} \alpha_i = 3$ and $\sum_{i=1}^{d-1} \beta_i = 4$. Note that there exists at least one $i=1, 2, \dots, d-1$ such that $2 \nmid \alpha_i$ by parity. Therefore, the following holds by the symmetry of $\mathbb{S}^{d-2}$.

$$ \int_{\mathbb{S}^{d-2}} \left( \prod_{i=1}^{d-1} \xi_i^{\alpha_i} \right) \omega_{d-2}(d\xi) = 0 \qquad (4.27) $$

Also, by the similar argument from the above, if there exists at least one $i=1, 2, \dots, d-1$ such that $2 \nmid \beta_i$, the following holds.

$$ \int_{\mathbb{S}^{d-2}} \left( \prod_{i=1}^{d-1} \xi_i^{\beta_i} \right) \omega_{d-2}(d\xi) = 0 \qquad (4.28) $$

Therefore, without loss of generality, it is sufficient to evaluate A and B defined as the below.

$$ A := \int_{\mathbb{S}^{d-2}} \xi_1^4 \omega_{d-2}(d\xi), \quad B := \int_{\mathbb{S}^{d-2}} \xi_1^2 \xi_2^2 \omega_{d-2}(d\xi) \qquad (4.29) $$

First, note that the following holds by Lemma 1.3.

$$ \frac{\mathcal{A}_{d-2}}{d-1} = \int_{\mathbb{S}^{d-2}} \xi_1^2 \omega_{d-2}(d\xi) = \int_{\mathbb{S}^{d-2}} \xi_1^2 \left( \sum_{i=1}^{d-1} \xi_i^2 \right) \omega_{d-2}(d\xi) = A + (d-2)B \qquad (4.30) $$

Second, we consider substituting $\xi = (\xi_1, \xi_2, \dots, \xi_{d-1})$ to $\tilde{\xi} = (\tilde{\xi}_1, \tilde{\xi}_2, \dots, \tilde{\xi}_{d-1})$ as the following.

$$ \tilde{\xi}_1 = \frac{\xi_1 + \xi_2}{\sqrt{2}}, \quad \tilde{\xi}_2 = \frac{\xi_1 - \xi_2}{\sqrt{2}}, \quad \tilde{\xi}_i = \xi_i, \quad \forall i = 3, 4, \dots, d-1 \qquad (4.31) $$

Then the following holds by (4.29) and (4.31).

$$ B = \int_{\mathbb{S}^{d-2}} \xi_1^2 \xi_2^2 \omega_{d-2}(d\xi) = \frac{1}{4} \int_{\mathbb{S}^{d-2}} (\tilde{\xi}_1 + \tilde{\xi}_2)^2 (\tilde{\xi}_1 - \tilde{\xi}_2)^2 \omega_{d-2}(d\tilde{\xi}) = \frac{1}{2}(A - B) \qquad (4.32) $$

By combining (4.27), (4.28), (4.30), and (4.32), (4.23) is proved. Also, (4.19) and Lemma 1.3 imply (4.24) and (4.25). To prove (4.26), let $\mathbf{M} \in \mathbb{R}^{(d-1)\times(d-1)}$ be given. Our goal is to evaluate $(i, j)$-th component of $\int_{\mathbb{S}^{d-2}} (\xi\xi')(\xi'M\xi)\omega_{d-2}(d\xi)$ for every $i, j = 1, 2, \dots, d-1$. Let $i, j = 1, 2, \dots, d-1$ be given. We consider two cases.
(Case 1: $i = j$) Note that the following holds by (4.28), (4.30), and (4.32), where $\mathbf{A}_{kl}$ denotes $(k,l)$-th component of a matrix $\mathbf{A} \in \mathbb{R}^{(d-1) \times (d-1)}$ for every $k, l = 1, 2, \dots, d-1$.

$$ 
\begin{aligned}
\left[ \int_{\mathbb{S}^{d-2}} (\boldsymbol{\xi}\boldsymbol{\xi}') (\boldsymbol{\xi}'\mathbf{M}\boldsymbol{\xi}) \omega_{d-2}(\mathrm{d}\boldsymbol{\xi}) \right]_{ii} &= \int_{\mathbb{S}^{d-2}} \xi_i^2 (\boldsymbol{\xi}'\mathbf{M}\boldsymbol{\xi}) \omega_{d-2}(\mathrm{d}\boldsymbol{\xi}) \\
&= \sum_{k=1}^{d-1} \sum_{l=1}^{d-1} \mathbf{M}_{kl} \int_{\mathbb{S}^{d-2}} \xi_i^2 \xi_k \xi_l \omega_{d-2}(\mathrm{d}\boldsymbol{\xi}) \\
&= \sum_{k=1}^{d-1} \mathbf{M}_{kk} \int_{\mathbb{S}^{d-2}} \xi_i^2 \xi_k^2 \omega_{d-2}(\mathrm{d}\boldsymbol{\xi}) \\
&= \frac{\mathcal{A}_{d-2}}{d^2-1} (2\mathbf{M}_{ii} + \mathrm{tr}(\mathbf{M}))
\end{aligned} 
\quad (4.33) $$

(Case 2: $i \neq j$) Note that the following holds by (4.28), (4.30), and (4.32).

$$ 
\begin{aligned}
\left[ \int_{\mathbb{S}^{d-2}} (\boldsymbol{\xi}\boldsymbol{\xi}') (\boldsymbol{\xi}'\mathbf{M}\boldsymbol{\xi}) \omega_{d-2}(\mathrm{d}\boldsymbol{\xi}) \right]_{ij} &= \int_{\mathbb{S}^{d-2}} \xi_i \xi_j (\boldsymbol{\xi}'\mathbf{M}\boldsymbol{\xi}) \omega_{d-2}(\mathrm{d}\boldsymbol{\xi}) \\
&= \sum_{k=1}^{d-1} \sum_{l=1}^{d-1} \mathbf{M}_{kl} \int_{\mathbb{S}^{d-2}} \xi_i \xi_j \xi_k \xi_l \omega_{d-2}(\mathrm{d}\boldsymbol{\xi}) \\
&= (\mathbf{M}_{ij} + \mathbf{M}_{ji}) \int_{\mathbb{S}^{d-2}} \xi_i^2 \xi_j^2 \omega_{d-2}(\mathrm{d}\boldsymbol{\xi}) \\
&= \frac{\mathcal{A}_{d-2}}{d^2-1} (\mathbf{M}_{ij} + \mathbf{M}_{ji})
\end{aligned} 
\quad (4.34) $$

By combining (4.33) and (4.34), (4.26) is proved. $\square$

The next step is to state and prove a generalization of Lemma 2.1.

**Lemma 4.3.** Suppose the condition D1 holds. Then the following holds for every $\mathbf{x}, \mathbf{z} \in \mathbb{S}^{d-1}$.

$$ |f(\mathbf{z}) - f(\mathbf{x}) - \mathbf{H}_x(\mathbf{z})'(\mathbf{I}_d - \mathbf{x}\mathbf{x}')\nabla_{\bar{f}}(\mathbf{x})| \le \arccos^2(\mathbf{z}'\mathbf{x}) \sup_{\mathbf{u} \in \mathbb{S}^{d-1}} \|\nabla_{\bar{f}}^2(\mathbf{u})\|_2 $$

*Proof of Lemma 4.3.* Let $\mathbf{x}, \mathbf{z} \in \mathbb{S}^{d-1}$ be given. Now choose any $\mathbf{B}_x \in \mathcal{B}_x$. Note that there exist $\theta \in [0, \pi]$ and $\boldsymbol{\xi} \in \mathbb{S}^{d-2}$ such that $\mathbf{z} = \Phi_{\mathbf{B}_x}(\theta, \boldsymbol{\xi})$. Then Lemma 2.1 with the condition D1 implies that there exists $\tilde{\theta}_{\boldsymbol{\xi}} \in [0, \pi]$ such that the following holds.

$$ f(\mathbf{z}) - f(\mathbf{x}) - \theta(\mathbf{B}_x\boldsymbol{\xi})'\nabla_{\bar{f}}(\mathbf{x}) = \frac{\theta^2}{2} \left( \mathbf{x}'\nabla_{\bar{f}}^2(\Phi_{\mathbf{B}_x}(\tilde{\theta}_{\boldsymbol{\xi}}, \boldsymbol{\xi}))\mathbf{x} + (\mathbf{B}_x\boldsymbol{\xi})'\nabla_{\bar{f}}^2(\Phi_{\mathbf{B}_x}(\tilde{\theta}_{\boldsymbol{\xi}}, \boldsymbol{\xi}))(\mathbf{B}_x\boldsymbol{\xi}) \right) \quad (4.35) $$

Note that $\|\mathbf{x}\|_2 = \|\mathbf{B}_x\boldsymbol{\xi}\|_2 = 1$. Therefore, by combining (4.4) and (4.35), the following holds.

$$ \left| f(\mathbf{z}) - f(\mathbf{x}) - \frac{\arccos(\mathbf{z}'\mathbf{x})}{\sqrt{1-(\mathbf{z}'\mathbf{x})^2}} (\mathbf{B}_x \mathbf{B}_x' \mathbf{z})' \nabla_{\bar{f}}(\mathbf{x}) \right| \le \arccos^2(\mathbf{z}'\mathbf{x}) \sup_{\mathbf{u} \in \mathbb{S}^{d-1}} \|\nabla_{\bar{f}}^2(\mathbf{u})\|_2 \quad (4.36) $$

Combining (4.5) and (4.36) with the fact that $\mathbf{B}_x\mathbf{B}_x' = \mathbf{I}_d - \mathbf{x}\mathbf{x}'$, the proof ends. $\square$
Now we state and prove an extended version of Lemma 2.2.

**Lemma 4.4.** Suppose the conditions L1 and D1 hold. Then the following hold.

$$ \sup_{x \in \mathbb{S}^{d-1}} \sup_{\mathbf{B}_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} \mathbf{B}'_x \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x}) - \frac{1}{d-1} \mathbf{B}'_x \nabla_{\bar{f}}(\boldsymbol{x}) \right\|_2 = O(h) \quad (4.37) $$

$$ \sup_{x \in \mathbb{S}^{d-1}} \sup_{\mathbf{B}_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} \mathbf{B}'_x \tilde{\boldsymbol{\mu}}_{h,2}(\boldsymbol{x}) \mathbf{B}_x - \frac{f(\boldsymbol{x})}{d-1} \mathbf{I}_{d-1} \right\|_2 = O(h^2) \quad (4.38) $$

Moreover, if the condition D2 holds additionally, then $\mathbf{B}'_x \tilde{\boldsymbol{\mu}}_{h,2}(\boldsymbol{x}) \mathbf{B}_x$ is invertible for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$, $\mathbf{B}_x \in \mathcal{B}_x$, and $h > 0$. Also, the following holds.

$$ \sup_{x \in \mathbb{S}^{d-1}} \sup_{\mathbf{B}_x \in \mathcal{B}_x} \left\| c_{h,2,1}(L) (\mathbf{B}'_x \tilde{\boldsymbol{\mu}}_{h,2}(\boldsymbol{x}) \mathbf{B}_x)^+ - \frac{d-1}{f(\boldsymbol{x})} \mathbf{I}_{d-1} \right\|_2 = O(h^2) \quad (4.39) $$

*Proof of Lemma 4.4.* Let $\boldsymbol{x} \in \mathbb{S}^{d-1}$ and $\mathbf{B}_x \in \mathcal{B}_x$ be given. Combining Lemma 1.2, Lemma 1.3, and Lemma 4.2, the following hold for every $h > 0$. Note that $\mathbf{B}'_x \mathbf{H}_x(\Phi_{\mathbf{B}_x}(\theta, \boldsymbol{\xi})) = \theta \boldsymbol{\xi}$ for every $\theta \in [0, \pi]$ and $\boldsymbol{\xi} \in \mathbb{S}^{d-2}$.

$$ \int_{\mathbb{S}^{d-1}} L \left( \frac{1 - z'x}{h^2} \right) \mathbf{B}'_x \mathbf{H}_x(z) f(x) \omega_{d-1}(dz) \\ = f(x) \int_0^\pi \int_{\mathbb{S}^{d-2}} \boldsymbol{\xi} \omega_{d-2}(d\boldsymbol{\xi}) L \left( \frac{1 - \cos \theta}{h^2} \right) \theta \sin^{d-2}(\theta) d\theta = \mathbf{0}_{d-1} \quad (4.40) $$

$$ \int_{\mathbb{S}^{d-1}} L \left( \frac{1 - z'x}{h^2} \right) \mathbf{B}'_x \mathbf{H}_x(z) \mathbf{H}_x(z)' (\mathbf{I}_d - \mathbf{x}\mathbf{x}') \nabla_{\bar{f}}(x) \omega_{d-1}(dz) \\ = \left[ \int_0^\pi \int_{\mathbb{S}^{d-2}} \boldsymbol{\xi} \boldsymbol{\xi}' \omega_{d-2}(d\boldsymbol{\xi}) L \left( \frac{1 - \cos \theta}{h^2} \right) \theta^2 \sin^{d-2}(\theta) d\theta \right] \mathbf{B}'_x \nabla_{\bar{f}}(x) = \frac{c_{h,2,1}(L)}{d-1} \mathbf{B}'_x \nabla_{\bar{f}}(x) \quad (4.41) $$

$$ \int_{\mathbb{S}^{d-1}} L \left( \frac{1 - z'x}{h^2} \right) \mathbf{B}'_x \mathbf{H}_x(z) \mathbf{H}_x(z)' \mathbf{B}_x f(x) \omega_{d-1}(dz) \\ = \left[ \int_0^\pi \int_{\mathbb{S}^{d-2}} \boldsymbol{\xi} \boldsymbol{\xi}' \omega_{d-2}(d\boldsymbol{\xi}) L \left( \frac{1 - \cos \theta}{h^2} \right) \theta^2 \sin^{d-2}(\theta) d\theta \right] f(x) = \frac{c_{h,2,1}(L)}{d-1} f(x) \mathbf{I}_{d-1} \quad (4.42) $$

$$ \int_{\mathbb{S}^{d-1}} L \left( \frac{1 - z'x}{h^2} \right) \mathbf{B}'_x \mathbf{H}_x(z) \mathbf{H}_x(z)' \mathbf{B}_x \left( \mathbf{H}_x(z)' (\mathbf{I}_d - \mathbf{x}\mathbf{x}') \nabla_{\bar{f}}(x) \right) \omega_{d-1}(dz) \\ = \int_0^\pi \int_{\mathbb{S}^{d-2}} \boldsymbol{\xi} \boldsymbol{\xi}' (\boldsymbol{\xi}' \mathbf{B}'_x \nabla_{\bar{f}}(x)) \omega_{d-2}(d\boldsymbol{\xi}) L \left( \frac{1 - \cos \theta}{h^2} \right) \theta^2 \sin^{d-2}(\theta) d\theta = \mathbf{O}_{d-1} \quad (4.43) $$

Combining (4.40), (4.41), (4.42), (4.43), Lemma 4.3 with the condition D1, the following holds for every $h > 0$. Note that $\| \mathbf{B}'_x \mathbf{H}_x(\Phi_{\mathbf{B}_x}(\theta, \boldsymbol{\xi})) \|_2 \le \theta$ for every $\theta \in [0, \pi]$ and $\boldsymbol{\xi} \in \mathbb{S}^{d-2}$.

$$ \begin{aligned} & \left\| \mathbf{B}'_x \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x}) - \frac{c_{h,2,1}(L)}{d-1} \mathbf{B}'_x \nabla_{\bar{f}}(\boldsymbol{x}) \right\|_2 \\ &= \left\| \int_{\mathbb{S}^{d-1}} L \left( \frac{1 - z'x}{h^2} \right) \mathbf{B}'_x \mathbf{H}_x(z) \left( f(z) - f(\boldsymbol{x}) - \mathbf{H}_x(z)' (\mathbf{I}_d - \mathbf{x}\mathbf{x}') \nabla_{\bar{f}}(\boldsymbol{x}) \right) \omega_{d-1}(dz) \right\|_2 \\ &\le \int_{\mathbb{S}^{d-1}} L \left( \frac{1 - z'x}{h^2} \right) \| \mathbf{B}'_x \mathbf{H}_x(z) \|_2 \left| f(z) - f(\boldsymbol{x}) - \mathbf{H}_x(z)' (\mathbf{I}_d - \mathbf{x}\mathbf{x}') \nabla_{\bar{f}}(\boldsymbol{x}) \right| \omega_{d-1}(dz) \end{aligned} $$
$$
\begin{align}
&\le \sup_{\mathbf{z} \in \mathbb{S}^{d-1}} \|\nabla_{\bar{f}}^2(\mathbf{z})\|_2 \int_0^{\pi} \mathcal{A}_{d-2} L\left(\frac{1 - \cos \theta}{h^2}\right) \theta^3 \sin^{d-2}(\theta) \mathrm{d}\theta = c_{h,3,1}(L) \sup_{\mathbf{z} \in \mathbb{S}^{d-1}} \|\nabla_{\bar{f}}^2(\mathbf{z})\|_2 \tag{4.44} \\
&\qquad \left\| \mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x - \frac{c_{h,2,1}(L)}{d-1} f(\mathbf{x}) \mathbf{I}_{d-1} \right\|_2 \nonumber
\end{align}
$$

$$
= \left\| \int_{\mathbb{S}^{d-1}} L\left(\frac{1-z'x}{h^2}\right) \mathbf{B}'_x \mathbf{H}_x(z) \mathbf{H}_x(z)' \mathbf{B}_x \left( f(z) - f(x) - \mathbf{H}_x(z)'(\mathbf{I}_d - \mathbf{x}\mathbf{x}')\nabla_{\bar{f}}(x) \right) \omega_{d-1}(\mathrm{d}z) \right\|_2 \\
\le \int_{\mathbb{S}^{d-1}} L\left(\frac{1-z'x}{h^2}\right) \left\| \mathbf{B}'_x \mathbf{H}_x(z) \right\|_2^2 \left| f(z) - f(x) - \mathbf{H}_x(z)'(\mathbf{I}_d - \mathbf{x}\mathbf{x}')\nabla_{\bar{f}}(x) \right| \omega_{d-1}(\mathrm{d}z)
$$

$$
\le \sup_{\mathbf{z} \in \mathbb{S}^{d-1}} \|\nabla_{\bar{f}}^2(\mathbf{z})\|_2 \int_0^{\pi} \mathcal{A}_{d-2} L\left(\frac{1 - \cos \theta}{h^2}\right) \theta^4 \sin^{d-2}(\theta) \mathrm{d}\theta = c_{h,4,1}(L) \sup_{\mathbf{z} \in \mathbb{S}^{d-1}} \|\nabla_{\bar{f}}^2(\mathbf{z})\|_2 \quad (4.45)
$$

Combining (4.44) and (4.45), we get the following for every $h > 0$.

$$
\sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \sup_{\mathbf{B}_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} \mathbf{B}'_x \tilde{\mu}_{h,1}(\mathbf{x}) - \frac{1}{d-1} \mathbf{B}'_x \nabla_{\bar{f}}(\mathbf{x}) \right\|_2 \le \frac{c_{h,3,1}(L)}{c_{h,2,1}(L)} \sup_{\mathbf{z} \in \mathbb{S}^{d-1}} \|\nabla_{\bar{f}}^2(\mathbf{z})\|_2 \quad (4.46)
$$

$$
\sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \sup_{\mathbf{B}_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} \mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x - \frac{f(\mathbf{x})}{d-1} \mathbf{I}_{d-1} \right\|_2 \le \frac{c_{h,4,1}(L)}{c_{h,2,1}(L)} \sup_{\mathbf{z} \in \mathbb{S}^{d-1}} \|\nabla_{\bar{f}}^2(\mathbf{z})\|_2 \quad (4.47)
$$

Therefore, (4.46), (4.47), and Lemma 1.4 with the condition L1 imply (4.37) and (4.38). Also, note that the following holds by Lemma 1.2 and Lemma 1.3 for every $\mathbf{u} \in \mathbb{S}^{d-2}$, $\mathbf{x} \in \mathbb{S}^{d-1}$, $\mathbf{B}_x \in \mathcal{B}_x$, and $h > 0$.

$$
\begin{align}
u' \mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x u &= \int_{\mathbb{S}^{d-1}} L\left(\frac{1-z'x}{h^2}\right) \left(u' \mathbf{B}'_x \mathbf{H}_x(z)\right)^2 f(z) \omega_{d-1}(\mathrm{d}z) \nonumber \\
&\geq \left[\inf_{\mathbf{z} \in \mathbb{S}^{d-1}} f(z)\right] \int_{\mathbb{S}^{d-1}} L\left(\frac{1-z'x}{h^2}\right) \left(u' \mathbf{B}'_x \mathbf{H}_x(z)\right)^2 \omega_{d-1}(\mathrm{d}z) \nonumber \\
&= \left[\inf_{\mathbf{z} \in \mathbb{S}^{d-1}} f(z)\right] \int_0^\pi \int_{\mathbb{S}^{d-2}} (u' \boldsymbol{\xi})^2 \omega_{d-2}(\mathrm{d}\boldsymbol{\xi}) L\left(\frac{1-\cos\theta}{h^2}\right) \theta^2 \sin^{d-2}(\theta) \mathrm{d}\theta \nonumber \\
&= \frac{c_{h,2,1}(L)}{d-1} \left[\inf_{\mathbf{z} \in \mathbb{S}^{d-1}} f(z)\right] \tag{4.48}
\end{align}
$$

Note that (4.48) with the condition D2 imply that $\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x$ is both positive-definite and invertible for every $\mathbf{x} \in \mathbb{S}^{d-1}$, $\mathbf{B}_x \in \mathcal{B}_x$, and $h > 0$. Therefore, the following holds for every $\mathbf{x} \in \mathbb{S}^{d-1}$, $\mathbf{B}_x \in \mathcal{B}_x$, and $h > 0$.

$$
\| (\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \|_2 = \left[ \inf_{\mathbf{u} \in \mathbb{S}^{d-2}} u' \mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x u \right]^{-1} \le \frac{d-1}{c_{h,2,1}(L)} \left[ \inf_{\mathbf{z} \in \mathbb{S}^{d-1}} f(\mathbf{z}) \right]^{-1} \quad (4.49)
$$

Also, (4.49) implies the following for every $h > 0$.

$$
\begin{align*}
& \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \sup_{\mathbf{B}_x \in \mathcal{B}_x} \left\| c_{h,2,1}(L) (\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ - \frac{d-1}{f(\mathbf{x})} \mathbf{I}_{d-1} \right\|_2 \\
&= \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \sup_{\mathbf{B}_x \in \mathcal{B}_x} \left\| \frac{(d-1)c_{h,2,1}(L)}{f(\mathbf{x})} (\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \left( \frac{1}{c_{h,2,1}(L)} \mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x - \frac{f(\mathbf{x})}{d-1} \mathbf{I}_{d-1} \right) \right\|_2 \\
&\le (d-1)^2 \left[ \inf_{\mathbf{z} \in \mathbb{S}^{d-1}} f(\mathbf{z}) \right]^{-2} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \sup_{\mathbf{B}_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} \mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x - \frac{f(\mathbf{x})}{d-1} \mathbf{I}_{d-1} \right\|_2 \tag{4.50}
\end{align*}
$$
Combining (4.38) and (4.50), (4.39) holds. □

The following Lemma states and proves the asymptotic property of $\tilde{\sigma}_h(\boldsymbol{x})$ for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$ and for any sufficiently small $h > 0$.

**Lemma 4.5.** Suppose the conditions L1, D1, and D2 hold. Then the following holds.

$$ \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \frac{1}{c_{h,0,1}(L)} \tilde{\sigma}_h(\boldsymbol{x}) - f(\boldsymbol{x}) \right| = o(1) \quad \text{as} \quad h \downarrow 0 $$

*Proof of Lemma 4.5.* Combining (2.1), (2.3), (4.12), and Theorem 2.1 with the conditions L1 and D1, the following holds.

$$ \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \frac{1}{c_{h,0,1}(L)} \tilde{\mu}_{h,0}(\boldsymbol{x}) - f(\boldsymbol{x}) \right| = o(1) \quad \text{as} \quad h \downarrow 0 \quad (4.51) $$

Note that (4.20) implies the following for every $h > 0$.

$$ \begin{aligned} & \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \frac{1}{c_{h,2,1}(L)} \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^+ \tilde{\mu}_{h,1}(\boldsymbol{x}) \right| \\ &= \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \sup_{\boldsymbol{B}_x \in \mathcal{B}_x} \left| \left( \frac{1}{c_{h,2,1}(L)} \boldsymbol{B}_x' \tilde{\mu}_{h,1}(\boldsymbol{x}) \right)' \left( c_{h,2,1}(L) (\boldsymbol{B}_x' \tilde{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ \right) \left( \frac{1}{c_{h,2,1}(L)} \boldsymbol{B}_x' \tilde{\mu}_{h,1}(\boldsymbol{x}) \right) \right| \\ &\le \left( \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \sup_{\boldsymbol{B}_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} \boldsymbol{B}_x' \tilde{\mu}_{h,1}(\boldsymbol{x}) \right\|_2^2 \right) \left( \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \sup_{\boldsymbol{B}_x \in \mathcal{B}_x} \left\| c_{h,2,1}(L) (\boldsymbol{B}_x' \tilde{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ \right\|_2 \right) \end{aligned} \quad (4.52) $$

Combining (1.3), (2.2), (4.52), and Lemma 4.4 with the conditions L1, D1, and D2, the following holds. Note that $\mathbb{S}^{d-1}$ is compact on $\mathbb{R}^d$.

$$ \begin{aligned} & \limsup_{h \downarrow 0} \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \frac{1}{c_{h,2,1}(L)} \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^+ \tilde{\mu}_{h,1}(\boldsymbol{x}) \right| \\ &\le \left( \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \sup_{\boldsymbol{B}_x \in \mathcal{B}_x} \left\| \frac{1}{d-1} \boldsymbol{B}_x' \nabla_{\bar{f}}(\boldsymbol{x}) \right\|_2^2 \right) \left( \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left\| \frac{d-1}{f(\boldsymbol{x})} \boldsymbol{I}_{d-1} \right\|_2 \right) \\ &= (d-1)^{-1} \left[ \inf_{\boldsymbol{z} \in \mathbb{S}^{d-1}} f(\boldsymbol{z}) \right]^{-1} \left( \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \sup_{\boldsymbol{B}_x \in \mathcal{B}_x} \nabla_{\bar{f}}(\boldsymbol{x})' \boldsymbol{B}_x \boldsymbol{B}_x' \nabla_{\bar{f}}(\boldsymbol{x}) \right) \\ &= (d-1)^{-1} \left[ \inf_{\boldsymbol{z} \in \mathbb{S}^{d-1}} f(\boldsymbol{z}) \right]^{-1} \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left\| \nabla_{\bar{f}}(\boldsymbol{x}) \right\|_2^2 < \infty \end{aligned} \quad (4.53) $$

Therefore, by combining (2.20), (4.14), (4.51), and (4.53), the proof ends. □

Before we state and prove the asymptotic property of $\hat{\sigma}_h(\boldsymbol{x})$ for any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$, we need a matrix variable version Chebyshev's inequality.

**Lemma 4.6.** Let $\mathbf{M} \in \mathbb{R}^{m \times p}$ be a random matrix. Also, let $\mathbf{M}_1, \mathbf{M}_2, \dots, \mathbf{M}_n \in \mathbb{R}^{m \times p}$ be independent copies of $\mathbf{M}$. Suppose $\|\mathbb{E}[\mathbf{M}'\mathbf{M}]\|_2 < \infty$. Then the following holds for every $\epsilon > 0$.

$$ \mathbb{P}\left(\left\|\frac{1}{n}\sum_{i=1}^{n} \mathbf{M}_i - \mathbb{E}[\mathbf{M}]\right\|_2 > \epsilon\right) \le \frac{p}{n\epsilon^2} \|\mathbb{E}[\mathbf{M}'\mathbf{M}]\|_2 $$
Proof of Lemma 4.6. Let a random matrix $\mathbf{M} \in \mathbb{R}^{m \times p}$ be given such that $\|\mathbb{E}[\mathbf{M}'\mathbf{M}]\|_2 < \infty$. Also, let $\mathbf{M}_1, \mathbf{M}_2, \dots, \mathbf{M}_n \in \mathbb{R}^{m \times p}$ be independent copies of $\mathbf{M} \in \mathbb{R}^{m \times p}$. Note that the following holds by the property of non-negative definite symmetric matrices, where $\{\mathbf{e}_k\}_{k=1}^p$ denotes the standard basis of $\mathbb{R}^p$.

$$ \begin{aligned} \left\| \frac{1}{n} \sum_{i=1}^{n} \mathbf{M}_i - \mathbb{E}[\mathbf{M}] \right\|_2^2 &= \left\| \left( \frac{1}{n} \sum_{i=1}^{n} \mathbf{M}_i - \mathbb{E}[\mathbf{M}] \right)' \left( \frac{1}{n} \sum_{j=1}^{n} \mathbf{M}_j - \mathbb{E}[\mathbf{M}] \right) \right\|_2 \\ &\le \operatorname{tr} \left( \left( \frac{1}{n} \sum_{i=1}^{n} \mathbf{M}_i - \mathbb{E}[\mathbf{M}] \right)' \left( \frac{1}{n} \sum_{j=1}^{n} \mathbf{M}_j - \mathbb{E}[\mathbf{M}] \right) \right) \\ &= \sum_{k=1}^{p} e_k' \left( \frac{1}{n} \sum_{i=1}^{n} \mathbf{M}_i - \mathbb{E}[\mathbf{M}] \right)' \left( \frac{1}{n} \sum_{j=1}^{n} \mathbf{M}_j - \mathbb{E}[\mathbf{M}] \right) e_k \end{aligned} \quad (4.54) $$

Also, note that the following holds by the independence for every $k = 1, 2, \dots, p$.

$$ \begin{aligned} & \mathbb{E} \left[ e_k' \left( \frac{1}{n} \sum_{i=1}^{n} \mathbf{M}_i - \mathbb{E}[\mathbf{M}] \right)' \left( \frac{1}{n} \sum_{j=1}^{n} \mathbf{M}_j - \mathbb{E}[\mathbf{M}] \right) e_k \right] \\ &= e_k' \mathbb{E} \left[ \left( \frac{1}{n} \sum_{i=1}^{n} \mathbf{M}_i - \mathbb{E}[\mathbf{M}] \right)' \left( \frac{1}{n} \sum_{j=1}^{n} \mathbf{M}_j - \mathbb{E}[\mathbf{M}] \right) \right] e_k \\ &\le \frac{1}{n^2} \sum_{i=1}^{n} \sum_{j=1}^{n} e_k' \mathbb{E} \left[ (\mathbf{M}_i - \mathbb{E}[\mathbf{M}])' (\mathbf{M}_j - \mathbb{E}[\mathbf{M}]) \right] e_k = \frac{1}{n} e_k' \mathbb{E} \left[ (\mathbf{M} - \mathbb{E}[\mathbf{M}])' (\mathbf{M} - \mathbb{E}[\mathbf{M}]) \right] e_k \\ &= \frac{1}{n} e_k' \mathbb{E}[\mathbf{M}'\mathbf{M}] e_k - \frac{1}{n} e_k' \mathbb{E}[\mathbf{M}]' \mathbb{E}[\mathbf{M}] e_k \le \frac{1}{n} e_k' \mathbb{E}[\mathbf{M}'\mathbf{M}] e_k \le \frac{1}{n} \|\mathbb{E}[\mathbf{M}'\mathbf{M}]\|_2 \end{aligned} \quad (4.55) $$

Combining (4.54) and (4.55) with Markov's inequality, the following holds for every $\epsilon > 0$.

$$ \begin{aligned} \mathbb{P}\left(\left\|\frac{1}{n}\sum_{i=1}^{n}\mathbf{M}_i - \mathbb{E}[\mathbf{M}]\right\|_2 > \epsilon\right) &\le \mathbb{P}\left(\sum_{k=1}^{p} e_k' \left(\frac{1}{n}\sum_{i=1}^{n}\mathbf{M}_i - \mathbb{E}[\mathbf{M}]\right)' \left(\frac{1}{n}\sum_{j=1}^{n}\mathbf{M}_j - \mathbb{E}[\mathbf{M}]\right) e_k > \epsilon^2\right) \\ &\le \frac{1}{\epsilon^2} \mathbb{E}\left[\sum_{k=1}^{p} e_k' \left(\frac{1}{n}\sum_{i=1}^{n}\mathbf{M}_i - \mathbb{E}[\mathbf{M}]\right)' \left(\frac{1}{n}\sum_{j=1}^{n}\mathbf{M}_j - \mathbb{E}[\mathbf{M}]\right) e_k\right] \\ &\le \frac{p}{n\epsilon^2} \|\mathbb{E}[\mathbf{M}'\mathbf{M}]\|_2 \end{aligned} \quad (4.56) $$

Therefore, (4.56) ends the proof. $\square$

Now we state and prove the asymptotic property of $\hat{\sigma}_h(\mathbf{x})$ for every $\mathbf{x} \in \mathbb{S}^{d-1}$ and for any sufficiently small $h = h_n > 0$ and $n \in \mathbb{N}$.

**Lemma 4.7.** Suppose the conditions L1, D1, D2, and B1 hold. Then the following holds for every $\mathbf{x} \in \mathbb{S}^{d-1}$.

$$ \left| \frac{1}{c_{h,0,1}(L)} (\hat{\sigma}_h(\mathbf{x}) - \tilde{\sigma}_h(\mathbf{x})) \right| = O_p(n^{-\frac{1}{2}} h^{-\frac{d-1}{2}}) \quad \text{as} \quad n \to \infty $$
Proof of Lemma 4.7. Let $\boldsymbol{x} \in \mathbb{S}^{d-1}$ be given. Combining (2.1), (2.3), (4.12), (4.15), and Theorem 2.2 with the conditions L1, D1, and B1, the following holds by Chebyshev's inequality.

$$ \left| \frac{\hat{\mu}_{h,0}(\boldsymbol{x}) - \tilde{\mu}_{h,0}(\boldsymbol{x})}{c_{h,0,1}(L)} \right| = O_p \left( \sqrt{\mathrm{Var} \left[ \frac{\hat{\mu}_{h,0}(\boldsymbol{x})}{c_{h,0,1}(L)} \right]} \right) = O_p \left( n^{-\frac{1}{2}} h^{-\frac{d-1}{2}} \right) \quad \text{as} \quad n \to \infty \quad (4.57) $$

Note that (4.20) implies the following for every $\boldsymbol{B}_x \in \mathcal{B}_x$, $h = h_n > 0$, and $n \in \mathbb{N}$.

$$ \begin{aligned} & \hat{\mu}_{h,1}(\boldsymbol{x})' \hat{\Sigma}_h(\boldsymbol{x})^{+} \hat{\mu}_{h,1}(\boldsymbol{x}) - \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^{+} \tilde{\mu}_{h,1}(\boldsymbol{x}) \\ &= (B_x' \hat{\mu}_{h,1}(\boldsymbol{x}))' (B_x' \hat{\mu}_{h,2}(\boldsymbol{x}) B_x)^{+} (B_x' \hat{\mu}_{h,1}(\boldsymbol{x})) - (B_x' \tilde{\mu}_{h,1}(\boldsymbol{x}))' (B_x' \tilde{\mu}_{h,2}(\boldsymbol{x}) B_x)^{+} (B_x' \tilde{\mu}_{h,1}(\boldsymbol{x})) \\ &= (B_x' \hat{\mu}_{h,1}(\boldsymbol{x}))' \left( (B_x' \hat{\mu}_{h,2}(\boldsymbol{x}) B_x)^{+} - (B_x' \tilde{\mu}_{h,2}(\boldsymbol{x}) B_x)^{+} \right) (B_x' \hat{\mu}_{h,1}(\boldsymbol{x})) \\ &\quad + (B_x' \hat{\mu}_{h,1}(\boldsymbol{x}) - B_x' \tilde{\mu}_{h,1}(\boldsymbol{x}))' (B_x' \tilde{\mu}_{h,2}(\boldsymbol{x}) B_x)^{+} (B_x' \hat{\mu}_{h,1}(\boldsymbol{x}) + B_x' \tilde{\mu}_{h,1}(\boldsymbol{x})) \end{aligned} \quad (4.58) $$

Also, note that the following hold by (2.3), Lemma 1.2, and Lemma 1.3 with the conditions L1 and D1 for every $k = 1, 2$, $\boldsymbol{B}_x \in \mathcal{B}_x$ and $h > 0$.

$$ \begin{aligned} & \mathbb{E} \left[ L^k \left( \frac{1 - \mathbf{X}'\boldsymbol{x}}{h^2} \right) (B_x' \mathbf{H}_x(\mathbf{X}))' (B_x' \mathbf{H}_x(\mathbf{X})) \right] \\ &= \int_{\mathbb{S}^{d-1}} L^k \left( \frac{1 - \boldsymbol{z}'\boldsymbol{x}}{h^2} \right) (B_x' \mathbf{H}_x(\boldsymbol{z}))' (B_x' \mathbf{H}_x(\boldsymbol{z})) f(\boldsymbol{z}) \omega_{d-1}(d\boldsymbol{z}) \\ &\le \left[ \sup_{\boldsymbol{z} \in \mathbb{S}^{d-1}} f(\boldsymbol{z}) \right] \int_{\mathbb{S}^{d-1}} L^k \left( \frac{1 - \boldsymbol{z}'\boldsymbol{x}}{h^2} \right) (B_x' \mathbf{H}_x(\boldsymbol{z}))' (B_x' \mathbf{H}_x(\boldsymbol{z})) \omega_{d-1}(d\boldsymbol{z}) \\ &= \left[ \sup_{\boldsymbol{z} \in \mathbb{S}^{d-1}} f(\boldsymbol{z}) \right] \int_0^\pi \int_{\mathbb{S}^{d-2}} 1 \omega_{d-2}(d\boldsymbol{\xi}) L^k \left( \frac{1 - \cos \boldsymbol{\theta}}{h^2} \right) \theta^2 \sin^{d-2}(\boldsymbol{\theta}) d\theta = \left[ \sup_{\boldsymbol{z} \in \mathbb{S}^{d-1}} f(\boldsymbol{z}) \right] c_{h,2,k}(L) \quad (4.59) \\ & \left\| \mathbb{E} \left[ L^k \left( \frac{1 - \mathbf{X}'\boldsymbol{x}}{h^2} \right) (B_x' \mathbf{H}_x(\mathbf{X}) \mathbf{H}_x(\mathbf{X})' B_x)^2 \right] \right\|_2 \\ &= \sup_{\boldsymbol{u} \in \mathbb{S}^{d-2}} \int_{\mathbb{S}^{d-1}} L^k \left( \frac{1 - \boldsymbol{z}'\boldsymbol{x}}{h^2} \right) \boldsymbol{u}' (B_x' \mathbf{H}_x(\mathbf{X}) \mathbf{H}_x(\mathbf{X})' B_x)^2 \boldsymbol{u} f(\boldsymbol{z}) \omega_{d-1}(d\boldsymbol{z}) \\ &\le \left[ \sup_{\boldsymbol{z} \in \mathbb{S}^{d-1}} f(\boldsymbol{z}) \right] \sup_{\boldsymbol{u} \in \mathbb{S}^{d-2}} \int_{\mathbb{S}^{d-1}} L^k \left( \frac{1 - \boldsymbol{z}'\boldsymbol{x}}{h^2} \right) \boldsymbol{u}' (B_x' \mathbf{H}_x(\mathbf{X}) \mathbf{H}_x(\mathbf{X})' B_x)^2 \boldsymbol{u} \omega_{d-1}(d\boldsymbol{z}) \\ &= \left[ \sup_{\boldsymbol{z} \in \mathbb{S}^{d-1}} f(\boldsymbol{z}) \right] \sup_{\boldsymbol{u} \in \mathbb{S}^{d-2}} \int_0^\pi \int_{\mathbb{S}^{d-2}} (\boldsymbol{u}'\boldsymbol{\xi})^2 \omega_{d-2}(d\boldsymbol{\xi}) L^k \left( \frac{1 - \cos \boldsymbol{\theta}}{h^2} \right) \theta^4 \sin^{d-2}(\boldsymbol{\theta}) d\theta \\ &= \left[ \sup_{\boldsymbol{z} \in \mathbb{S}^{d-1}} f(\boldsymbol{z}) \right] \frac{c_{h,4,k}(L)}{d-1} \end{aligned} \quad (4.60) $$

Therefore, combining (4.15), (4.16), (4.59), (4.60), Lemma 1.4, and Lemma 4.6 with the conditions L1 and B1, the following hold for every $\boldsymbol{B}_x \in \mathcal{B}_x$.

$$ \| B_x' \hat{\mu}_{h,1}(\boldsymbol{x}) - B_x' \tilde{\mu}_{h,1}(\boldsymbol{x}) \|_2 = O_p(n^{-\frac{1}{2}} h^{\frac{d+1}{2}}) \quad (4.61) $$

$$ \| B_x' \hat{\mu}_{h,2}(\boldsymbol{x}) B_x - B_x' \tilde{\mu}_{h,2}(\boldsymbol{x}) B_x \|_2 = O_p(n^{-\frac{1}{2}} h^{\frac{d+3}{2}}) \quad (4.62) $$
Also, note that the compactness of $\mathbb{S}^{d-1}$ and the condition D1 imply the following.

$$ \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \| \mathbf{B}'_x \nabla_{\bar{f}}(\mathbf{x}) \|_2^2 \le \sup_{x \in \mathbb{S}^{d-1}} \| \mathbf{I}_d - x\mathbf{x}' \|_2 \sup_{z \in \mathbb{S}^{d-1}} \| \nabla_{\bar{f}}(z) \|_2^2 = \sup_{z \in \mathbb{S}^{d-1}} \| \nabla_{\bar{f}}(z) \|_2^2 < \infty \quad (4.63) $$

Also, combining (4.61), (4.62), (4.63), Lemma 1.4, and Lemma 4.4 with the conditions L1, D2, and B1, the following hold for every $\mathbf{B}_x \in \mathcal{B}_x$.

$$ \| \mathbf{B}'_x \tilde{\mu}_{h,1}(\mathbf{x}) \|_2 = O(h^{d+1}), \quad \| \mathbf{B}'_x \hat{\mu}_{h,1}(\mathbf{x}) \|_2 = O_p(h^d) \quad (4.64) $$

$$ \| \mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x \|_2 = O(h^{d+1}), \quad \| \mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x \|_2 = O_p(h^{d+1}) \quad (4.65) $$

Now we fix $\mathbf{B}_x \in \mathcal{B}_x$. Let $\epsilon > 0$ be given. By (4.62), there exist $M > 0$ and $N_1 \in \mathbb{N}$ such that the following holds.

$$ n \ge N_1 \Rightarrow \mathbb{P} \left( \| \mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x - \mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x \|_2 > n^{-\frac{1}{2}} h^{\frac{d+3}{2}} M \right) < \epsilon \quad (4.66) $$

Now we define events $E_n$ and $F_n$ as the following for every $n \in \mathbb{N}$.

$$ E_n := \left\{ \| (\mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ - (\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \|_2 > \left[ \inf_{z \in \mathbb{S}^{d-1}} f(z) \right]^{-2} \frac{2(d-1)^2}{c_{h,2,1}^2(L)} n^{-\frac{1}{2}} h^{\frac{d+3}{2}} M \right\} \\ F_n := \left\{ \| \mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x - \mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x \|_2 \le n^{-\frac{1}{2}} h^{\frac{d+3}{2}} M \right\} $$

Note that Lemma 1.4 with the conditions L1 and B1 imply that there exists $N_2 \in \mathbb{N}$ such that the following holds.

$$ n \ge N_2 \Rightarrow n^{-\frac{1}{2}} h^{\frac{d+3}{2}} M \le \frac{c_{h,2,1}(L)}{2(d-1)} \left[ \inf_{z \in \mathbb{S}^{d-1}} f(z) \right] \quad (4.67) $$

Then the following holds on $F_n$ by (4.48) and (4.67) for every $n \in \mathbb{N}$ such that $n \ge N_2$.

$$ \inf_{\mathbf{u} \in \mathbb{S}^{d-2}} \mathbf{u}' \mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x \mathbf{u} \ge \inf_{\mathbf{u} \in \mathbb{S}^{d-2}} \mathbf{u}' \mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x \mathbf{u} - \| \mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x - \mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x \|_2 \\ \ge \frac{c_{h,2,1}(L)}{d-1} \left[ \inf_{z \in \mathbb{S}^{d-1}} f(z) \right] - n^{-\frac{1}{2}} h^{\frac{d+3}{2}} M \ge \frac{c_{h,2,1}(L)}{2(d-1)} \left[ \inf_{z \in \mathbb{S}^{d-1}} f(z) \right] \quad (4.68) $$

By (4.49), (4.68), and the property of symmetric matrices, $\mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x$ and $\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x$ are both invertible on the event $F_n$ for every $h = h_n > 0$ and $n \in \mathbb{N}$ such that $n \ge N_2$. Now define $N := \max\{N_1, N_2\}$. Note that the following holds on $F_n$ by (4.49), (4.66), and (4.68) with the condition D2 for every $h = h_n > 0$ and $n \in \mathbb{N}$ such that $n \ge N$.

$$ \| (\mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ - (\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \|_2 = \| (\mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^{-1} - (\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^{-1} \|_2 \\ \le \| (\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^{-1} \|_2 \| \mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x - \mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x \|_2 \| (\mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^{-1} \|_2 $$
$$ \leq \left[ \inf_{\mathbf{z} \in \mathbb{S}^{d-1}} f(\mathbf{z}) \right]^{-2} \frac{2(d-1)^2}{c_{h,2,1}^2(L)} n^{-\frac{1}{2}} h^{\frac{d+3}{2}} M \quad (4.69) $$

Therefore, $n \ge N$ implies $\mathbb{P}(F_n) \le \mathbb{P}(E_n^c)$, which is equivalent to $\mathbb{P}(E_n) \le \mathbb{P}(F_n^c) < \epsilon$ for every $n \in \mathbb{N}$. Since $\epsilon > 0$ is arbitrary, the following holds by (4.69) and Lemma 1.4 with the conditions L1 and B1 for every $\mathbf{B}_x \in \mathcal{B}_x$.

$$ \| (\mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ - (\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \|_2 = O_p(n^{-\frac{1}{2}} h^{-\frac{3d+1}{2}}) \quad (4.70) $$

Combining (4.49), (4.58), (4.61), (4.64), (4.70), we get the following for every $\mathbf{B}_x \in \mathcal{B}_x$.

$$ \begin{aligned} & |\hat{\mu}_{h,1}(\mathbf{x})' \hat{\Sigma}_h(\mathbf{x})^+ \hat{\mu}_{h,1}(\mathbf{x}) - \tilde{\mu}_{h,1}(\mathbf{x})' \tilde{\Sigma}_h(\mathbf{x})^+ \tilde{\mu}_{h,1}(\mathbf{x})| \\ & \le \|\mathbf{B}'_x \hat{\mu}_{h,1}(\mathbf{x})\|_2^2 \|(\mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ - (\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+\|_2 \\ & + \|\mathbf{B}'_x \hat{\mu}_{h,1}(\mathbf{x}) - \mathbf{B}'_x \tilde{\mu}_{h,1}(\mathbf{x})\|_2 \|(\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+\|_2 (\|\mathbf{B}'_x \hat{\mu}_{h,1}(\mathbf{x})\|_2 + \|\mathbf{B}'_x \tilde{\mu}_{h,1}(\mathbf{x})\|_2) \\ & = O_p(h^d)^2 O_p(n^{-\frac{1}{2}} h^{-\frac{3d+1}{2}}) + O_p(n^{-\frac{1}{2}} h^{\frac{d+1}{2}}) O_p(h^{-(d+1)}) O_p(h^d) = O_p(n^{-\frac{1}{2}} h^{\frac{d-1}{2}}) \quad (4.71) \end{aligned} $$

Therefore, by combining (4.14), (4.17), (4.20), (4.57), (4.71), Lemma 1.4 with the conditions L1 and B1, the proof ends. □

So far, Lemma 1.4, Lemma 4.5, and Lemma 4.7 with the conditions Condition L1 and Condition B1 imply $\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x$, $\mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x$ are invertible and $\tilde{\sigma}_h(\mathbf{x})$, $\hat{\sigma}_h(\mathbf{x})$ are positive almost surely for every $\mathbf{x} \in \mathbb{S}^{d-1}$, $\mathbf{B}_x \in \mathcal{B}_x$, and for any sufficiently small $h = h_n > 0$ for sufficiently large $n \in \mathbb{N}$. For any given $\mathbf{x} \in \mathbb{S}^{d-1}$, we define the effective kernel $\tilde{W}_{\mathbf{x},h} : \mathbb{S}^{d-1} \to \mathbb{R}$ and its estimator $\hat{W}_{\mathbf{x},h} : \mathbb{S}^{d-1} \to \mathbb{R}$ as the following for every $h = h_n > 0$ and $n \in \mathbb{N}$.

$$ \tilde{W}_{\mathbf{x},h}(\mathbf{z}) := \frac{1}{\tilde{\sigma}_h(\mathbf{x})} L\left(\frac{1 - \mathbf{z}'\mathbf{x}}{h^2}\right) (1 - \tilde{\mu}_{h,1}(\mathbf{x})' \tilde{\Sigma}_h(\mathbf{x})^+ \mathbf{H}_{\mathbf{x}}(\mathbf{z})), \quad \forall \mathbf{z} \in \mathbb{S}^{d-1} \quad (4.72) $$

$$ \hat{W}_{\mathbf{x},h}(\mathbf{z}) := \frac{1}{\hat{\sigma}_h(\mathbf{x})} L\left(\frac{1 - \mathbf{z}'\mathbf{x}}{h^2}\right) (1 - \hat{\mu}_{h,1}(\mathbf{x})' \hat{\Sigma}_h(\mathbf{x})^+ \mathbf{H}_{\mathbf{x}}(\mathbf{z})), \quad \forall \mathbf{z} \in \mathbb{S}^{d-1} \quad (4.73) $$

By (4.72) and (4.73), (4.21) and (4.22) are equivalent as the following for every $\mathbf{x} \in \mathbb{S}^{d-1}$, $h = h_n > 0$, and $n \in \mathbb{N}$.

$$ \tilde{\alpha}_h(\mathbf{x}) = \underset{y \in \mathbb{R}}{\operatorname{argmin}} \mathbb{E} \left[ \tilde{W}_{\mathbf{x},h}(\mathbf{X})(Y - y)^2 \right], \quad \hat{\alpha}_h(\mathbf{x}) = \underset{y \in \mathbb{R}}{\operatorname{argmin}} n^{-1} \sum_{i=1}^{n} \hat{W}_{\mathbf{x},h}(\mathbf{X}_i)(Y_i - y)^2 \quad (4.74) $$

From now on, we consider the case when $(\Omega, d)$ is a general metric space. Similar to Im et al. (2024), we extend (4.74) to $(\Omega, d)$ by substituting $Y - y$ to $d(Y, y)$. Therefore, we define functions $\tilde{M}_{h,1} : \mathbb{S}^{d-1} \times \Omega \to \mathbb{R}$ and $\tilde{m}_{h,1} : \mathbb{S}^{d-1} \to \mathbb{R}$ with their estimators $\hat{M}_{h,1} : \mathbb{S}^{d-1} \times \Omega \to \mathbb{R}$
and $\hat{m}_{h,1}: \mathbb{S}^{d-1} \to \mathbb{R}$ respectively as the following for every $h = h_n > 0$ and $n \in \mathbb{N}$. Note that $\tilde{m}_{h,1}$ and $\hat{m}_{h,1}$ are the miminizers of $\tilde{M}_{h,1}$ and $\hat{M}_{h,1}$ respectively for every $h = h_n > 0$ and $n \in \mathbb{N}$.

$$ \tilde{M}_{h,1}(\boldsymbol{x}, y) := \mathbb{E}\left[\tilde{W}_{\boldsymbol{x},h}(\mathbf{X})d^2(Y, y)\right], \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1}, \forall y \in \Omega \quad (4.75) $$

$$ \tilde{m}_{h,1}(\boldsymbol{x}) := \underset{y \in \Omega}{\operatorname{argmin}} \mathbb{E}\left[\tilde{W}_{\boldsymbol{x},h}(\mathbf{X})d^2(Y, y)\right], \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1} \quad (4.76) $$

$$ \hat{M}_{h,1}(\boldsymbol{x}, y) := n^{-1} \sum_{i=1}^{n} \hat{W}_{\boldsymbol{x},h}(\mathbf{X}_i)d^2(Y_i, y), \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1}, \forall y \in \Omega \quad (4.77) $$

$$ \hat{m}_{h,1}(\boldsymbol{x}) := \underset{y \in \Omega}{\operatorname{argmin}} n^{-1} \sum_{i=1}^{n} \hat{W}_{\boldsymbol{x},h}(\mathbf{X}_i)d^2(Y_i, y), \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1} \quad (4.78) $$

For the convenience, we define $\tilde{\tau}_{h,0}$, $\tilde{\tau}_{h,1}$, $\tilde{\nu}_{h,0}$, $\tilde{\nu}_{h,1}$, $\hat{\nu}_{h,0}$, and $\hat{\tau}_{h,1}$ as the following for every $h = h_n > 0$ and $n \in \mathbb{N}$, where $g_y$ is defined by C1. Note that $\hat{\nu}_{h,0}$ and $\hat{\tau}_{h,1}$ are the estimators of $\tilde{\nu}_{h,0}$ and $\tilde{\nu}_{h,1}$ respectively for every $h = h_n > 0$ and $n \in \mathbb{N}$.

$$ \tilde{\tau}_{h,0}(\boldsymbol{x}, y) := \mathbb{E}\left[L\left(\frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2}\right)g_y(\boldsymbol{X})\right], \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1}, \forall y \in \Omega $$

$$ \tilde{\tau}_{h,1}(\boldsymbol{x}, y) := \mathbb{E}\left[L\left(\frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2}\right)H_{\boldsymbol{x}}(\boldsymbol{X})g_y(\boldsymbol{X})\right], \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1}, \forall y \in \Omega $$

$$ \tilde{\nu}_{h,0}(\boldsymbol{x}, y) := \mathbb{E}\left[L\left(\frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2}\right)d^2(Y, y)\right], \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1}, \forall y \in \Omega $$

$$ \tilde{\nu}_{h,1}(\boldsymbol{x}, y) := \mathbb{E}\left[L\left(\frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2}\right)H_{\boldsymbol{x}}(\boldsymbol{X})d^2(Y, y)\right], \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1}, \forall y \in \Omega $$

$$ \hat{\nu}_{h,0}(\boldsymbol{x}, y) := n^{-1} \sum_{i=1}^{n} L\left(\frac{1 - \boldsymbol{X}'_i \boldsymbol{x}}{h^2}\right) d^2(Y_i, y), \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1}, \forall y \in \Omega $$

$$ \hat{\nu}_{h,1}(\boldsymbol{x}, y) := n^{-1} \sum_{i=1}^{n} L\left(\frac{1 - \boldsymbol{X}'_i \boldsymbol{x}}{h^2}\right) H_{\boldsymbol{x}}(\boldsymbol{X}_i) d^2(Y_i, y), \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1}, \forall y \in \Omega $$

Therefore, $\tilde{M}_{h,1}$ and $\hat{M}_{h,1}$ can be expressed as the following for every $h = h_n > 0$ and $n \in \mathbb{N}$.

$$ \tilde{M}_{h,1}(\boldsymbol{x}, y) = \frac{1}{\hat{\sigma}_h(\boldsymbol{x})} (\tilde{\nu}_{h,0}(\boldsymbol{x}, y) - \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^{+} \tilde{\nu}_{h,1}(\boldsymbol{x}, y)), \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1}, \forall y \in \Omega \quad (4.79) $$

$$ \hat{M}_{h,1}(\boldsymbol{x}, y) = \frac{1}{\hat{\sigma}_h(\boldsymbol{x})} (\hat{\nu}_{h,0}(\boldsymbol{x}, y) - \hat{\mu}_{h,1}(\boldsymbol{x})' \hat{\Sigma}_h(\boldsymbol{x})^{+} \hat{\nu}_{h,1}(\boldsymbol{x}, y)), \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1}, \forall y \in \Omega \quad (4.80) $$

For any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$, the goal of this section is to get an upper bound of $d(\hat{m}_{h,1}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x}))$. This can be computed by decomposing into $d(\tilde{m}_{h,1}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x}))$ and $d(\hat{m}_{h,1}(\boldsymbol{x}), \tilde{m}_{h,1}(\boldsymbol{x}))$, just as section 3. To do so, we compute $\tilde{M}_{h,1}(\boldsymbol{x}, y) - M_{\oplus}(\boldsymbol{x}, y)$ and $\hat{M}_{h,1}(\boldsymbol{x}, y) - \tilde{M}_{h,1}(\boldsymbol{x}, y)$ for every $y \in \Omega$. This method was also used in Im et al. (2024). First, we replace the conditions M1-0 and M3-0 with the conditions M1-1 and M3-1 in order to impose conditions with respect to $\hat{m}_{h,1}$, $\tilde{m}_{h,1}$, $\hat{M}_{h,1}$, and $\tilde{M}_{h,1}$.
**Condition M1-1.** For any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$, the object $m_{\oplus}(\boldsymbol{x})$ is unique. For any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$ and for every $n \in \mathbb{N}$, $\hat{m}_{h,1}(\boldsymbol{x})$ and $\tilde{m}_{h,1}(\boldsymbol{x})$ exist and are unique, the latter almost surely. In addition, for any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$, the following hold for every $\epsilon > 0$.

$$ \inf_{d(y, m_{\oplus}(\boldsymbol{x})) > \epsilon} \left[ M_{\oplus}(\boldsymbol{x}, y) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) \right] > 0 \\ \liminf_{h \downarrow 0} \inf_{d(y, \tilde{m}_{h,1}(\boldsymbol{x})) > \epsilon} \left[ \tilde{M}_{h,1}(\boldsymbol{x}, y) - \tilde{M}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right] > 0 $$

**Condition M3-1.** For any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$, there exist constants $\tilde{h}_1 > 0$, $\tilde{\eta}_1 > 0$, $\tilde{C}_1 > 0$, and $\tilde{\beta}_1 > 1$ such that the following holds for every $y \in \Omega$.

$$ 0 < h < \tilde{h}_1, \quad d(y, \tilde{m}_{h,1}(\boldsymbol{x})) < \tilde{\eta}_1 \quad \Rightarrow \quad \tilde{M}_{h,1}(\boldsymbol{x}, y) - \tilde{M}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \geq \tilde{C}_1 d(y, \tilde{m}_{h,1}(\boldsymbol{x}))^{\tilde{\beta}_1} $$

The following is a useful lemma to prove the convergence rate of $d(\tilde{m}_{h,1}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x}))$. The following is a generalization of Lemma 3.3.

**Lemma 4.8.** Suppose the conditions L1, D1, D2, and C1 hold. Then the following holds.

$$ \sup_{y \in \Omega} \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \frac{\tilde{\tau}_{h,0}(\boldsymbol{x}, y) - \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^+ \tilde{\tau}_{h,1}(\boldsymbol{x}, y)}{\tilde{\sigma}_h(\boldsymbol{x})} - g_y(\boldsymbol{x}) - \frac{h^2}{d-1} \frac{a_{d,1}(L)}{a_{d-2,1}(L)} \text{tr}\left(\nabla^2_{\tilde{\bar{g}}_y}(\boldsymbol{x})\right) \right| = o(h^2) $$

*Proof of Lemma 4.8.* Note that the following hold by (1.4), (3.17), Lemma 1.4, Lemma 2.2, and Lemma 3.2 with the conditions L1, D1, and C1.

$$ \sup_{y \in \Omega} \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \frac{\tilde{\tau}_{h,0}(\boldsymbol{x}, y)}{c_{h,0,1}(L)} - (f \cdot g_y)(\boldsymbol{x}) - \frac{1}{2(d-1)} \frac{c_{h,2,1}(L)}{c_{h,0,1}(L)} \text{tr}\left(\nabla^2_{\tilde{f} \cdot \tilde{g}_y}(\boldsymbol{x})\right) \right| = o(h^2) \quad (4.81) $$

$$ \sup_{\boldsymbol{x} \in \mathbb{S}^{d-1}} \left| \frac{\tilde{\mu}_{h,0}(\boldsymbol{x})}{c_{h,0,1}(L)} - f(\boldsymbol{x}) - \frac{1}{2(d-1)} \frac{c_{h,2,1}(L)}{c_{h,0,1}(L)} \text{tr}\left(\nabla^2_{\tilde{f}}(\boldsymbol{x})\right) \right| = o(h^2) \quad (4.82) $$

Also, the following holds for every $y \in \Omega$ and $\boldsymbol{x} \in \mathbb{S}^{d-1}$.

$$ \text{tr}\left(\nabla^2_{\tilde{f} \cdot \tilde{g}_y}(\boldsymbol{x})\right) = f(\boldsymbol{x})\text{tr}\left(\nabla^2_{\tilde{g}_y}(\boldsymbol{x})\right) + g_y(\boldsymbol{x})\text{tr}\left(\nabla^2_{\tilde{f}}(\boldsymbol{x})\right) + 2\nabla_{\tilde{f}}(\boldsymbol{x})'\nabla_{\tilde{g}_y}(\boldsymbol{x}) \quad (4.83) $$

$$ \nabla_{\tilde{f} \cdot \tilde{g}_y}(\boldsymbol{x}) = f(\boldsymbol{x})\nabla_{\tilde{g}_y}(\boldsymbol{x}) + g_y(\boldsymbol{x})\nabla_{\tilde{f}}(\boldsymbol{x}) \quad (4.84) $$

Now we get the asymptotic property of $B'_{\boldsymbol{x}} \tilde{\tau}_{h,1}(\boldsymbol{x}, y)$ for every $y \in \Omega$, $\boldsymbol{x} \in \mathbb{S}^{d-1}$, $B_{\boldsymbol{x}} \in \mathcal{B}_{\boldsymbol{x}}$, and $h > 0$. Similar to (4.40) and (4.41) of Lemma 4.4, we get the following for every $y \in \Omega$, $\boldsymbol{x} \in \mathbb{S}^{d-1}$, $B_{\boldsymbol{x}} \in \mathcal{B}_{\boldsymbol{x}}$, and $h > 0$.

$$ \int_{\mathbb{S}^{d-1}} L\left(\frac{1 - z'\boldsymbol{x}}{h^2}\right) B'_{\boldsymbol{x}} H_{\boldsymbol{x}}(z)(f \cdot g_y)(\boldsymbol{x}) \omega_{d-1}(dz) = \mathbf{0}_{d-1} \quad (4.85) $$

$$ \int_{\mathbb{S}^{d-1}} L\left(\frac{1 - z'\boldsymbol{x}}{h^2}\right) B'_{\boldsymbol{x}} H_{\boldsymbol{x}}(z) H_{\boldsymbol{x}}(z)' (I_d - \boldsymbol{x}\boldsymbol{x}') \nabla_{\tilde{f} \cdot \tilde{g}_y}(\boldsymbol{x}) \omega_{d-1}(dz) = \frac{c_{h,2,1}(L)}{d-1} B'_{\boldsymbol{x}} \nabla_{\tilde{f} \cdot \tilde{g}_y}(\boldsymbol{x}) \quad (4.86) $$
Just as (4.46) of Lemma 4.4 with (4.85) and (4.86), the following holds for every $y \in \Omega$ and $h > 0$.

$$ \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} B'_x \tilde{\tau}_{h,1}(x, y) - \frac{1}{d-1} B'_x \nabla_{\bar{f} \cdot \bar{g}_y}(x) \right\|_2 \le \frac{c_{h,3,1}(L)}{c_{h,2,1}(L)} \sup_{z \in \mathbb{S}^{d-1}} \| \nabla^2_{\bar{f} \cdot \bar{g}_y}(z) \|_2 \quad (4.87) $$

Therefore, (3.2), (4.87), Lemma 1.4, and Lemma 4.4 with the conditions L1 and D1 imply the following.

$$ \sup_{y \in \Omega} \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} B'_x \tilde{\tau}_{h,1}(x, y) - \frac{1}{d-1} B'_x \nabla_{\bar{f} \cdot \bar{g}_y}(x) \right\|_2 = O(h) \quad (4.88) $$

$$ \sup_{y \in \Omega} \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} B'_x \tilde{\mu}_{h,1}(x) g_y(x) - \frac{g_y(x)}{d-1} B'_x \nabla_{\bar{f}}(x) \right\|_2 = O(h) \quad (4.89) $$

By combining (2.3), (3.2), (4.84), (4.88), and (4.89), we get the following.

$$ \sup_{y \in \Omega} \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} B'_x \left( \tilde{\tau}_{h,1}(x, y) - \tilde{\mu}_{h,1}(x) g_y(x) \right) - \frac{f(x)}{d-1} B'_x \nabla_{\bar{g}_y}(x) \right\|_2 = O(h) \quad (4.90) $$

Note that (1.3), (2.2), (2.3), and Lemma 4.4 with the conditions L1 and D1 imply the following.

$$ \begin{aligned} & \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} B'_x \tilde{\mu}_{h,1}(x) \right\|_2 \\ & \le \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \left\| \frac{1}{d-1} B'_x \nabla_{\bar{f}}(x) \right\|_2 + \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} B'_x \tilde{\mu}_{h,1}(x) - \frac{1}{d-1} B'_x \nabla_{\bar{f}}(x) \right\|_2 \\ & = \frac{1}{d-1} \sup_{x \in \mathbb{S}^{d-1}} \sqrt{\nabla_{\bar{f}}(x)' (\mathbf{I}_d - \mathbf{x}\mathbf{x}') \nabla_{\bar{f}}(x)} + \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} B'_x \tilde{\mu}_{h,1}(x) - \frac{1}{d-1} B'_x \nabla_{\bar{f}}(x) \right\|_2 \\ & = \frac{1}{d-1} \sup_{x \in \mathbb{S}^{d-1}} \| \nabla_{\bar{f}}(x) \|_2 + \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} B'_x \tilde{\mu}_{h,1}(x) - \frac{1}{d-1} B'_x \nabla_{\bar{f}}(x) \right\|_2 = O(1) \quad (4.91) \end{aligned} $$

Combining (2.2), (2.3), (4.91), and Lemma 4.4 with the conditions L1, D1, and D2 imply the following.

$$ \begin{aligned} & \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \left\| \tilde{\mu}_{h,1}(x)' B_x (B_x' \tilde{\mu}_{h,2}(x) B_x)^+ - \frac{1}{f(x)} \nabla_{\bar{f}}(x)' B_x \right\|_2 \\ & \le \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} \tilde{\mu}_{h,1}(x)' B_x \right\|_2 \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \left\| c_{h,2,1}(L) (B_x' \tilde{\mu}_{h,2}(x) B_x)^+ - \frac{d-1}{f(x)} \mathbf{I}_{d-1} \right\|_2 \\ & + \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \left\| \frac{1}{c_{h,2,1}(L)} \tilde{\mu}_{h,1}(x)' B_x - \frac{1}{d-1} \nabla_{\bar{f}}(x)' B_x \right\|_2 \sup_{x \in \mathbb{S}^{d-1}} \left\| \frac{d-1}{f(x)} \mathbf{I}_{d-1} \right\|_2 \\ & = O(1)O(h^2) + O(h)O(1) = O(h) \end{aligned} \quad (4.92) $$

Also, combining (2.3), (4.91), and (4.92) with the condition D2, we get the following.

$$ \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \left\| \tilde{\mu}_{h,1}(x)' B_x (B_x' \tilde{\mu}_{h,2}(x) B_x)^+ \right\|_2 \\ \le \sup_{x \in \mathbb{S}^{d-1}} \sup_{B_x \in \mathcal{B}_x} \left( \left\| \tilde{\mu}_{h,1}(x)' B_x (B_x' \tilde{\mu}_{h,2}(x) B_x)^+ - \frac{1}{f(x)} \nabla_{\bar{f}}(x)' B_x \right\|_2 + \left\| \frac{1}{f(x)} \nabla_{\bar{f}}(x)' B_x \right\|_2 \right) $$
$$
\begin{align}
&= \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \sup_{\mathbf{B}_{\mathbf{x}} \in \mathcal{B}_{\mathbf{x}}} \left( \left\| \tilde{\boldsymbol{\mu}}_{h,1}(\mathbf{x})' \mathbf{B}_{\mathbf{x}} (\mathbf{B}_{\mathbf{x}}' \tilde{\boldsymbol{\mu}}_{h,2}(\mathbf{x}) \mathbf{B}_{\mathbf{x}})^{+} - \frac{1}{f(\mathbf{x})} \nabla_{\bar{f}}(\mathbf{x})' \mathbf{B}_{\mathbf{x}} \right\|_{2} + \frac{1}{f(\mathbf{x})} \| \nabla_{\bar{f}}(\mathbf{x}) \|_{2} \right) \nonumber \\
&= O(h) + O(1) = O(1) \tag{4.93}
\end{align}
$$

Note that the following holds by (1.3), (2.3), (3.1), (3.2), and (3.2).

$$
\begin{align}
\sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \sup_{\mathbf{B}_{\mathbf{x}} \in \mathcal{B}_{\mathbf{x}}} \left\| \frac{f(\mathbf{x})}{d-1} \mathbf{B}'_{\mathbf{x}} \nabla_{\bar{g}_y}(\mathbf{x}) \right\|_{2} &= \sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \frac{f(\mathbf{x})}{d-1} \sqrt{\nabla_{\bar{g}_y}(\mathbf{x})' (\mathbf{I}_d - \mathbf{x}\mathbf{x}') \nabla_{\bar{g}_y}(\mathbf{x})} \nonumber \\
&= \sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \frac{f(\mathbf{x})}{d-1} \| \nabla_{\bar{g}_y}(\mathbf{x}) \|_{2} = O(1) \tag{4.94}
\end{align}
$$

Combining (1.3), (4.20), (4.90), (4.92), (4.93), and (4.94), we get the following.

$$
\begin{align}
& \sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \frac{1}{c_{h,2,1}(L)} \tilde{\boldsymbol{\mu}}_{h,1}(\mathbf{x})' \tilde{\boldsymbol{\Sigma}}_h(\mathbf{x})^+ \left( \tilde{\boldsymbol{\tau}}_{h,1}(\mathbf{x}, y) - \tilde{\boldsymbol{\mu}}_{h,1}(\mathbf{x}) g_y(\mathbf{x}) \right) - \frac{1}{d-1} \nabla_{\bar{f}}(\mathbf{x})' \nabla_{\bar{g}_y}(\mathbf{x}) \right| \nonumber \\
&= \sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \sup_{\mathbf{B}_{\mathbf{x}} \in \mathcal{B}_{\mathbf{x}}} \left| \frac{1}{c_{h,2,1}(L)} \tilde{\boldsymbol{\mu}}_{h,1}(\mathbf{x})' \mathbf{B}_{\mathbf{x}} (\mathbf{B}_{\mathbf{x}}' \tilde{\boldsymbol{\mu}}_{h,2}(\mathbf{x}) \mathbf{B}_{\mathbf{x}})^{+} \mathbf{B}_{\mathbf{x}}' \left( \tilde{\boldsymbol{\tau}}_{h,1}(\mathbf{x}, y) - \tilde{\boldsymbol{\mu}}_{h,1}(\mathbf{x}) g_y(\mathbf{x}) \right) \right. \nonumber \\
& \qquad \left. - \frac{1}{d-1} \nabla_{\bar{f}}(\mathbf{x})' \mathbf{B}_{\mathbf{x}} \mathbf{B}_{\mathbf{x}}' \nabla_{\bar{g}_y}(\mathbf{x}) \right| \nonumber \\
&\le \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \sup_{\mathbf{B}_{\mathbf{x}} \in \mathcal{B}_{\mathbf{x}}} \left\| \tilde{\boldsymbol{\mu}}_{h,1}(\mathbf{x})' \mathbf{B}_{\mathbf{x}} (\mathbf{B}_{\mathbf{x}}' \tilde{\boldsymbol{\mu}}_{h,2}(\mathbf{x}) \mathbf{B}_{\mathbf{x}})^{+} \right\|_{2} \nonumber \\
& \qquad \times \sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \sup_{\mathbf{B}_{\mathbf{x}} \in \mathcal{B}_{\mathbf{x}}} \left\| \frac{1}{c_{h,2,1}(L)} \mathbf{B}_{\mathbf{x}}' \left( \tilde{\boldsymbol{\tau}}_{h,1}(\mathbf{x}, y) - \tilde{\boldsymbol{\mu}}_{h,1}(\mathbf{x}) g_y(\mathbf{x}) \right) - \frac{f(\mathbf{x})}{d-1} \mathbf{B}_{\mathbf{x}}' \nabla_{\bar{g}_y}(\mathbf{x}) \right\|_{2} \nonumber \\
& \qquad + \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \sup_{\mathbf{B}_{\mathbf{x}} \in \mathcal{B}_{\mathbf{x}}} \left\| \tilde{\boldsymbol{\mu}}_{h,1}(\mathbf{x})' \mathbf{B}_{\mathbf{x}} (\mathbf{B}_{\mathbf{x}}' \tilde{\boldsymbol{\mu}}_{h,2}(\mathbf{x}) \mathbf{B}_{\mathbf{x}})^{+} - \frac{1}{f(\mathbf{x})} \nabla_{\bar{f}}(\mathbf{x})' \mathbf{B}_{\mathbf{x}} \right\|_{2} \nonumber \\
& \qquad + \sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \sup_{\mathbf{B}_{\mathbf{x}} \in \mathcal{B}_{\mathbf{x}}} \left\| \frac{f(\mathbf{x})}{d-1} \mathbf{B}_{\mathbf{x}}' \nabla_{\bar{g}_y}(\mathbf{x}) \right\|_{2} \nonumber \\
&= O(1)O(h) + O(h)O(1) = O(h) \tag{4.95}
\end{align}
$$

Therefore, combining (2.20), (3.2), (4.81), (4.14), (4.82), (4.83), and (4.95), the following holds.

$$
\begin{align*}
& \sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \frac{1}{c_{h,0,1}(L)} \left( \tilde{\tau}_{h,0}(\mathbf{x}, y) - \tilde{\boldsymbol{\mu}}_{h,1}(\mathbf{x})' \tilde{\boldsymbol{\Sigma}}_h(\mathbf{x})^+ \tilde{\tau}_{h,1}(\mathbf{x}, y) - \tilde{\sigma}_h(\mathbf{x}) g_y(\mathbf{x}) \right) \right. \\
& \qquad \left. - \frac{1}{2(d-1)} \frac{c_{h,2,1}(L)}{c_{h,0,1}(L)} f(\mathbf{x}) \operatorname{tr} \left( \nabla^2_{\bar{g}_y}(\mathbf{x}) \right) \right| \\
\le & \sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \frac{1}{c_{h,0,1}(L)} \left( \tilde{\tau}_{h,0}(\mathbf{x}, y) - \tilde{\boldsymbol{\mu}}_{h,0}(\mathbf{x}) g_y(\mathbf{x}) \right) \right. \\
& \qquad \left. - \frac{1}{2(d-1)} \frac{c_{h,2,1}(L)}{c_{h,0,1}(L)} \left\{ \operatorname{tr} \left( \nabla^2_{\bar{f} \cdot \bar{g}_y}(\mathbf{x}) \right) - g_y(\mathbf{x}) \operatorname{tr} \left( \nabla^2_{\bar{f}}(\mathbf{x}) \right) \right\} \right| \\
& + \frac{c_{h,2,1}(L)}{c_{h,0,1}(L)} \sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \frac{1}{c_{h,2,1}(L)} \tilde{\boldsymbol{\mu}}_{h,1}(\mathbf{x})' \tilde{\boldsymbol{\Sigma}}_h(\mathbf{x})^+ \left( \tilde{\tau}_{h,1}(\mathbf{x}, y) - \tilde{\boldsymbol{\mu}}_{h,1}(\mathbf{x}) g_y(\mathbf{x}) \right) - \frac{f(\mathbf{x})}{d-1} \mathbf{B}'_{\mathbf{x}} \nabla_{\bar{g}_y}(\mathbf{x}) \right| \\
\le & \sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \frac{1}{c_{h,0,1}(L)} \tilde{\tau}_{h,0}(\mathbf{x}, y) - f(\mathbf{x}) g_y(\mathbf{x}) - \frac{1}{2(d-1)} \frac{c_{h,2,1}(L)}{c_{h,0,1}(L)} \operatorname{tr} \left( \nabla^2_{\bar{f} \cdot \bar{g}_y}(\mathbf{x}) \right) \right|
\end{align*}
$$
$$
\begin{align}
&+ \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \frac{1}{c_{h,0,1}(L)} \tilde{\mu}_{h,0}(\mathbf{x}) - f(\mathbf{x}) - \frac{1}{2(d-1)} \frac{c_{h,2,1}(L)}{c_{h,0,1}(L)} \mathrm{tr} \left( \nabla_{\tilde{f}}^2(\mathbf{x}) \right) \right| \left( \sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} |g_y(\mathbf{x})| \right) \nonumber \\
&+ \frac{c_{h,2,1}(L)}{c_{h,0,1}(L)} \sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \frac{1}{c_{h,2,1}(L)} \tilde{\mu}_{h,1}(\mathbf{x})' \tilde{\Sigma}_h(\mathbf{x})^+ \left( \tilde{\tau}_{h,1}(\mathbf{x}, y) - \tilde{\mu}_{h,1}(\mathbf{x}) g_y(\mathbf{x}) \right) - \frac{f(\mathbf{x})}{d-1} B_{\mathbf{x}}' \nabla_{\tilde{g}_y}(\mathbf{x}) \right| \nonumber \\
&= o(h^2) + o(h^2) + O(h^2)O(h) = o(h^2) \tag{4.96}
\end{align}
$$

Also, (2.3), (2.20), and Lemma 4.5 with conditions L1, D1, and D2 imply the following.

$$
\begin{align}
& \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \frac{c_{h,2,1}(L)}{2c_{h,0,1}(L)} f(\mathbf{x}) - \frac{h^2 a_{d,1}(L)}{a_{d-2,1}(L)} \frac{\tilde{\sigma}_h(\mathbf{x})}{c_{h,0,1}(L)} \right| \nonumber \\
& \le \left| \frac{c_{h,2,1}(L)}{2c_{h,0,1}(L)} - \frac{h^2 a_{d,1}(L)}{a_{d-2,1}(L)} \left| \left( \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} f(\mathbf{x}) \right) + \frac{h^2 a_{d,1}(L)}{a_{d-2,1}(L)} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| f(\mathbf{x}) - \frac{\tilde{\sigma}_h(\mathbf{x})}{c_{h,0,1}(L)} \right| \right| \nonumber \\
& = o(h^2)O(1) + O(h^2)O(1) = o(h^2) \tag{4.97}
\end{align}
$$

Combining (3.2), (4.96), and (4.97), we get the following.

$$
\sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \frac{\tilde{\tau}_{h,0}(\mathbf{x}, y) - \tilde{\mu}_{h,1}(\mathbf{x})' \tilde{\Sigma}_h(\mathbf{x})^+ \tilde{\tau}_{h,1}(\mathbf{x}, y)}{c_{h,0,1}(L)} - \frac{h^2 \tilde{\sigma}_h(\mathbf{x})}{(d-1)c_{h,0,1}(L)} \frac{a_{d,1}(L)}{a_{d-2,1}(L)} \mathrm{tr} \left( \nabla^2_{\tilde{g}_y}(\mathbf{x}) \right) \right| = o(h^2) \quad (4.98)
$$

Note that Lemma 4.5 with the conditions L1, D1, and D2 imply the following.

$$
\sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \frac{c_{h,0,1}(L)}{\tilde{\sigma}_h(\mathbf{x})} = \left( \inf_{\mathbf{x} \in \mathbb{S}^{d-1}} \frac{\tilde{\sigma}_h(\mathbf{x})}{c_{h,0,1}(L)} \right)^{-1} = \left( \inf_{\mathbf{x} \in \mathbb{S}^{d-1}} f(\mathbf{x}) - \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \frac{\tilde{\sigma}_h(\mathbf{x})}{c_{h,0,1}(L)} - f(\mathbf{x}) \right| \right)^{-1} = O(1) \quad (4.99)
$$

Therefore, (4.98) and (4.99) ends the proof. □

Now we state about the approximation of $\tilde{M}_{h,1}(\mathbf{x}, y) - M_{\oplus}(\mathbf{x}, y)$ for every $\mathbf{x} \in \mathbb{S}^{d-1}$ and $y \in \Omega$.

**Lemma 4.9.** Suppose the conditions L1, D1, D2, C1, and R1 hold. Then the following holds.

$$
\sup_{y \in \Omega} \sup_{\mathbf{x} \in \mathbb{S}^{d-1}} \left| \tilde{M}_{h,1}(\mathbf{x}, y) - M_{\oplus}(\mathbf{x}, y) - \frac{h^2 a_{1,1}(L)}{a_{0,1}(L)} \int_{\Omega} d^2(y, w) \mathrm{tr} \left( \nabla^2_{\tilde{g}_w}(\mathbf{x}) \right) dF_Y(w) \right| = o(h^2)
$$

*Proof of Lemma 4.9.* Fubini's theorem and (3.20) imply the following for every $\mathbf{x} \in \mathbb{S}^{d-1}$, $y \in \Omega$, and $h > 0$.

$$
\tilde{M}_{h,1}(\mathbf{x}, y) = \mathbb{E} \left[ \tilde{W}_{\mathbf{x},h}(\mathbf{X}) M_{\oplus}(\mathbf{X}, y) \right] = \int_{\Omega} d^2(y, w) \frac{\tilde{\tau}_{h,0}(\mathbf{x}, w) - \tilde{\mu}_{h,1}(\mathbf{x})' \tilde{\Sigma}_h(\mathbf{x})^+ \tilde{\tau}_{h,1}(\mathbf{x}, w)}{\tilde{\sigma}_h(\mathbf{x})} dF_Y(w) \quad (4.100)
$$

Since the condition R1 implies that $d^2(y, w) \le \mathrm{diam}(\Omega)^2 < \infty$ for every $y, w \in \Omega$, (3.20), (4.100), and Lemma 4.8 with the conditions L1, D1, D2, and C1 imply Lemma 4.9. □
Note that the difference between Lemma 3.4 and Lemma 4.9 is the existence of the term $\frac{2}{f(\boldsymbol{x})} \int_{\Omega} d^2(y, w) \nabla_{\bar{f}}(\boldsymbol{x})' \nabla_{\bar{g}_w}(\boldsymbol{x}) dF_Y(w)$. The next Theorems are direct consequences of Lemma 4.9, which state that $d(\tilde{m}_{h,1}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x}))$ converges to 0 with the convergence rate of $O(h^{2/(\beta_{\oplus}-1)})$ for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$.

**Theorem 4.1** (Convergence of the bias of $\hat{m}_{h,1}$). Suppose the conditions L1, D1, D2, C1, R1, and M1-1 hold. Then the following holds for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$.

$$ \lim_{h \downarrow 0} d(\tilde{m}_{h,1}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x})) = 0 $$

*Proof of Theorem 4.1.* Let $\boldsymbol{x} \in \mathbb{S}^{d-1}$ be given. Also, let $\epsilon > 0$ be given. By the condition M1-1, there exist constants $\eta > 0$ such that the following hold.

$$ d(y, m_{\oplus}(\boldsymbol{x})) > \epsilon \Rightarrow M_{\oplus}(\boldsymbol{x}, y) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) \geq \eta, \quad \forall y \in \Omega \quad (4.101) $$

Also, by (3.2) and Lemma 4.9 with the conditions L1, D1, D2, C1, and R1, there exists a constant $\delta > 0$ such that the following holds.

$$ \mathrm{diam}(\Omega)^{-2} \left| \int_{\Omega} d^2(y, w) \mathrm{tr} \left( \nabla^2_{\bar{g}_w}(\boldsymbol{x}) \right) dF_Y(w) \right| \leq \sup_{y \in \Omega} \left| \mathrm{tr} \left( \nabla^2_{\bar{g}_y}(\boldsymbol{x}) \right) \right| \leq d \sup_{y \in \Omega} \| \nabla^2_{\bar{g}_y}(\boldsymbol{x}) \|_2 < \infty \quad (4.102) $$

$$ \sup_{y \in \Omega} \left| \tilde{M}_{h,1}(\boldsymbol{x}, y) - M_{\oplus}(\boldsymbol{x}, y) \right| < \frac{\eta}{2}, \quad \forall h \in (0, \delta) \quad (4.103) $$

Note that the following holds for every $h \in (0, \delta)$ by (4.102) and (4.103).

$$ \begin{aligned} & M_{\oplus}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) \\ & \leq \left[ M_{\oplus}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) \right] + \left[ \tilde{M}_{h,1}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) - \tilde{M}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right] \\ & \leq \left| \tilde{M}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) - M_{\oplus}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right| + \left| \tilde{M}_{h,1}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) \right| < \eta \quad (4.104) \end{aligned} $$

Therefore, (4.101) and (4.104) imply that $d(\tilde{m}_{h,1}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x})) \leq \epsilon$ for every $h \in (0, \delta)$. Since $\epsilon > 0$ is arbitrary, the proof is done. $\square$

**Theorem 4.2** (Convergence rate of the bias of $\hat{m}_{h,0}$). Suppose the conditions L1, D1, D2, C1, R1, M1-1, and M2 hold. Then the following holds for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$, where $C_{\oplus} > 0$ and $\beta_{\oplus} > 1$ are the constant defined in M2.

$$ d(\tilde{m}_{h,1}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x}))^{\beta_{\oplus}-1} \leq \frac{2\mathrm{diam}(\Omega)}{C_{\oplus}(d-1)} \frac{a_{d,1}(L)}{a_{d-2,1}(L)} \sup_{y \in \Omega} \left| \mathrm{tr} \left( \nabla^2_{\bar{g}_y}(\boldsymbol{x}) \right) \right| h^2 + o(h^2) $$
Proof of Theorem 4.2. Let $\boldsymbol{x} \in \mathbb{S}^{d-1}$ be given. Note that the following holds for every $h > 0$ by (3.20) and (4.100).

$$
\begin{align*}
& M_{\oplus}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) \\
\le & \left[M_{\oplus}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x}))\right] + \left[\tilde{M}_{h,1}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) - \tilde{M}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x}))\right] \\
= & \int_{\Omega} \left(d^2(m_{\oplus}(\boldsymbol{x}), w) - d^2(\tilde{m}_{h,1}(\boldsymbol{x}), w)\right) \left(\frac{\tilde{\tau}_{h,0}(\boldsymbol{x}, w) - \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^{+} \tilde{\tau}_{h,1}(\boldsymbol{x}, w)}{\tilde{\sigma}_h(\boldsymbol{x})} - g_w(\boldsymbol{x})\right) dF_Y(w) \\
\le & \int_{\Omega} \left|d^2(m_{\oplus}(\boldsymbol{x}), w) - d^2(\tilde{m}_{h,1}(\boldsymbol{x}), w)\right| \left|\frac{\tilde{\tau}_{h,0}(\boldsymbol{x}, w) - \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^{+} \tilde{\tau}_{h,1}(\boldsymbol{x}, w)}{\tilde{\sigma}_h(\boldsymbol{x})} - g_w(\boldsymbol{x})\right| dF_Y(w) \\
\le & 2\operatorname{diam}(\Omega) d(\tilde{m}_{h,1}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x})) \sup_{y \in \Omega} \left|\frac{\tilde{\tau}_{h,0}(\boldsymbol{x}, w) - \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^{+} \tilde{\tau}_{h,1}(\boldsymbol{x}, w)}{\tilde{\sigma}_h(\boldsymbol{x})} - g_y(\boldsymbol{x})\right| \quad (4.105)
\end{align*}
$$

Also, by Lemma 4.8 with the conditions L1, D1, D2, C1, and R1, the following holds.

$$
\sup_{y \in \Omega} \left| \frac{\tilde{\tau}_{h,0}(\boldsymbol{x}, y) - \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^{+} \tilde{\tau}_{h,1}(\boldsymbol{x}, y)}{\tilde{\sigma}_h(\boldsymbol{x})} - g_y(\boldsymbol{x}) \right| = \frac{h^2 a_{1,1}(L)}{a_{0,1}(L)} \sup_{y \in \Omega} \left| \operatorname{tr} \left( \nabla^2_{\tilde{g}_y}(\boldsymbol{x}) \right) \right| + o(h^2) \quad (4.106)
$$

Note that Theorem 4.1 with the conditions L1, D1, D2, C1, R1, and M1-1 states that there exists $\delta > 0$ such that $d(\tilde{m}_{h,1}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x})) < \eta_{\oplus}$ for every $h \in (0, \delta)$, where $\eta_{\oplus} > 0$ is the constant defined in the condition M2. Therefore, by the condition M2, the following holds for every $h \in (0, \delta)$, where $C_{\oplus} > 0$, $\beta_{\oplus} > 0$ are the constants defined in the condition M2.

$$
C_{\oplus} d(\tilde{m}_{h,1}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x}))^{\beta_{\oplus}} \leq M_{\oplus}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) - M_{\oplus}(\boldsymbol{x}, m_{\oplus}(\boldsymbol{x})) \quad (4.107)
$$

Therefore, by combining (4.105), (4.106), and (4.107), the proof is done. $\square$

Note that the convergence rate of $d(\tilde{m}_{h,1}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x}))$ is the same as $d(\tilde{m}_{h,0}(\boldsymbol{x}), m_{\oplus}(\boldsymbol{x}))$, but the leading term is slightly different. Next, we state a pointwise upper bound of $|\hat{M}_{h,1}(\boldsymbol{x}, y) - \tilde{M}_{h,1}(\boldsymbol{x}, y)|$ for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$ and $y \in \Omega$ as $n \to \infty$.

**Lemma 4.10.** Suppose the conditions L1, D1, D2, B1, and R1 hold. Then the following holds for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$ and $y \in \Omega$.

$$
\left| \hat{M}_{h,1}(\boldsymbol{x}, y) - \tilde{M}_{h,1}(\boldsymbol{x}, y) \right| = o_p(1) \quad \text{as} \quad n \to \infty
$$

Proof of Lemma 4.10. Let $\boldsymbol{x} \in \mathbb{S}^{d-1}$ and $y \in \Omega$ be given. Note that the following holds by (2.3), Lemma 1.4, and Lemma 2.2 with the conditions L1, D1, B1, and R1.

$$
\mathrm{Var} \left[ \frac{\hat{v}_{h,0}(\boldsymbol{x}, y)}{c_{h,0,1}(L)} \right] \le \frac{1}{nc_{h,0,1}^2(L)} \mathbb{E} \left[ L^2 \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) d^4(Y, y) \right]
$$
$$ \leq \frac{\mathrm{diam}(\Omega)^4}{nc_{h,0,1}^2(L)} \mathbb{E}\left[L^2\left(\frac{1 - \mathbf{X}'\mathbf{x}}{h^2}\right)\right] = O(n^{-1}h^{-(d-1)}) \quad (4.108) $$

Combining (4.108) with Chebyshev's inequality, the following holds.

$$ \left| \frac{\hat{\nu}_{h,0}(\mathbf{x}, y) - \tilde{\nu}_{h,0}(\mathbf{x}, y)}{c_{h,0,1}(L)} \right| = O_p\left( \sqrt{\mathrm{Var}\left[ \frac{\hat{\nu}_{h,0}(\mathbf{x}, y)}{c_{h,0,1}(L)} \right]} \right) = O_p(n^{-\frac{1}{2}}h^{-\frac{d-1}{2}}) \quad (4.109) $$

Also, note that the following holds by (4.59) with the conditions L1, D1, and R1 for every $\mathbf{B}_x \in \mathcal{B}_x$ and $h > 0$.

$$ \begin{aligned} & \mathbb{E}\left[L^2\left(\frac{1 - \mathbf{X}'\mathbf{x}}{h^2}\right)(\mathbf{B}_x'\mathbf{H}_x(\mathbf{X}))'(\mathbf{B}_x'\mathbf{H}_x(\mathbf{X}))d^4(Y, y)\right] \\ & \leq \mathrm{diam}(\Omega)^4 \mathbb{E}\left[L^2\left(\frac{1 - \mathbf{X}'\mathbf{x}}{h^2}\right)(\mathbf{B}_x'\mathbf{H}_x(\mathbf{X}))'(\mathbf{B}_x'\mathbf{H}_x(\mathbf{X}))\right] = \mathrm{diam}(\Omega)^4 \left[\sup_{\mathbf{z} \in \mathcal{S}^{d-1}} f(\mathbf{z})\right]_{c_{h,2,2}(L)} \end{aligned} \quad (4.110) $$

By combining (4.110), Lemma 1.4, and Lemma 4.6 with the conditions L1 and B1, the following holds for every $\mathbf{B}_x \in \mathcal{B}_x$.

$$ \| \mathbf{B}_x' \hat{\nu}_{h,1}(\mathbf{x}, y) - \mathbf{B}_x' \tilde{\nu}_{h,1}(\mathbf{x}, y) \|_2 = O_p(n^{-\frac{1}{2}}h^{\frac{d+1}{2}}) \quad (4.111) $$

Also, the following holds by (2.3), (4.59), Lemma 1.4, and Lemma 2.2 with the conditions L1, D1, R1, and Cauchy-Schwarz inequality for every $\mathbf{B}_x \in \mathcal{B}_x$.

$$ \begin{aligned} \| \mathbf{B}_x' \tilde{\nu}_{h,1}(\mathbf{x}, y) \|_2^2 &= \left\| \mathbb{E} \left[ L \left( \frac{1 - \mathbf{X}'\mathbf{x}}{h^2} \right) \mathbf{B}_x' \mathbf{H}_x(\mathbf{X}) d^2(Y, y) \right] \right\|_2^2 \\ &\le \left( \mathbb{E} \left[ L \left( \frac{1 - \mathbf{X}'\mathbf{x}}{h^2} \right) \| \mathbf{B}_x' \mathbf{H}_x(\mathbf{X}) \|_2 d^2(Y, y) \right] \right)^2 \\ &\le \mathrm{diam}(\Omega)^4 \left( \mathbb{E} \left[ L \left( \frac{1 - \mathbf{X}'\mathbf{x}}{h^2} \right) \| \mathbf{B}_x' \mathbf{H}_x(\mathbf{X}) \|_2 \right] \right)^2 \\ &\le \mathrm{diam}(\Omega)^4 \tilde{\mu}_{h,0}(\mathbf{x}) \mathbb{E} \left[ L \left( \frac{1 - \mathbf{X}'\mathbf{x}}{h^2} \right) \| \mathbf{B}_x' \mathbf{H}_x(\mathbf{X}) \|_2^2 \right] \\ &= \mathrm{diam}(\Omega)^4 \tilde{\mu}_{h,0}(\mathbf{x}) \mathbb{E} \left[ L \left( \frac{1 - \mathbf{X}'\mathbf{x}}{h^2} \right) (\mathbf{B}_x' \mathbf{H}_x(\mathbf{X}))' (\mathbf{B}_x' \mathbf{H}_x(\mathbf{X})) \right] \\ &= O(h^{d-1}) O(h^{d+1}) = O(h^{2d}) \end{aligned} \quad (4.112) $$

Therefore, (4.20), (4.49), (4.61), (4.62), (4.64), (4.65), (4.70), (4.111), and (4.112) with the condition B1 imply the following for every $\mathbf{B}_x \in \mathcal{B}_x$.

$$ \begin{aligned} & \left| \hat{\mu}_{h,1}(\mathbf{x})' \hat{\Sigma}_h(\mathbf{x})^+ \hat{\nu}_{h,1}(\mathbf{x}, y) - \tilde{\mu}_{h,1}(\mathbf{x})' \tilde{\Sigma}_h(\mathbf{x})^+ \tilde{\nu}_{h,1}(\mathbf{x}, y) \right| \\ &= \left| \hat{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x (\mathbf{B}_x' \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \mathbf{B}_x' \hat{\nu}_{h,1}(\mathbf{x}, y) - \tilde{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x (\mathbf{B}_x' \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \mathbf{B}_x' \tilde{\nu}_{h,1}(\mathbf{x}, y) \right| \end{aligned} $$
$$
\begin{aligned}
&\leq \left| \hat{\mu}_{h,1}(\boldsymbol{x})' \boldsymbol{B}_x (\boldsymbol{B}'_x \hat{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ \left( \boldsymbol{B}'_x \hat{\nu}_{h,1}(\boldsymbol{x}, y) - \boldsymbol{B}'_x \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \right) \right| \\
&+ \left| \hat{\mu}_{h,1}(\boldsymbol{x})' \boldsymbol{B}_x \left( (\boldsymbol{B}'_x \hat{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ - (\boldsymbol{B}'_x \tilde{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ \right) \boldsymbol{B}'_x \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \right| \\
&+ \left| \left( \hat{\mu}_{h,1}(\boldsymbol{x})' \boldsymbol{B}_x - \tilde{\mu}_{h,1}(\boldsymbol{x})' \boldsymbol{B}_x \right) (\boldsymbol{B}'_x \tilde{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ \boldsymbol{B}'_x \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \right| \\
&\leq \| \hat{\mu}_{h,1}(\boldsymbol{x})' \boldsymbol{B}_x \|_2 \|(\boldsymbol{B}'_x \hat{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ \|_2 \| \boldsymbol{B}'_x \hat{\nu}_{h,1}(\boldsymbol{x}, y) - \boldsymbol{B}'_x \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \|_2 \\
&+ \| \hat{\mu}_{h,1}(\boldsymbol{x})' \boldsymbol{B}_x \|_2 \|(\boldsymbol{B}'_x \hat{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ - (\boldsymbol{B}'_x \tilde{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ \|_2 \| \boldsymbol{B}'_x \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \|_2 \\
&+ \| \hat{\mu}_{h,1}(\boldsymbol{x})' \boldsymbol{B}_x - \tilde{\mu}_{h,1}(\boldsymbol{x})' \boldsymbol{B}_x \|_2 \|(\boldsymbol{B}'_x \tilde{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ \|_2 \| \boldsymbol{B}'_x \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \|_2 \\
&\leq \| \hat{\mu}_{h,1}(\boldsymbol{x})' \boldsymbol{B}_x \|_2 \|(\boldsymbol{B}'_x \hat{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ - (\boldsymbol{B}'_x \tilde{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ \|_2 \| \boldsymbol{B}'_x \hat{\nu}_{h,1}(\boldsymbol{x}, y) - \boldsymbol{B}'_x \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \|_2 \\
&+ \| \hat{\mu}_{h,1}(\boldsymbol{x})' \boldsymbol{B}_x \|_2 \|(\boldsymbol{B}'_x \tilde{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ \|_2 \| \boldsymbol{B}'_x \hat{\nu}_{h,1}(\boldsymbol{x}, y) - \boldsymbol{B}'_x \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \|_2 \\
&+ \| \hat{\mu}_{h,1}(\boldsymbol{x})' \boldsymbol{B}_x \|_2 \|(\boldsymbol{B}'_x \hat{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ - (\boldsymbol{B}'_x \tilde{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ \|_2 \| \boldsymbol{B}'_x \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \|_2 \\
&+ \| \hat{\mu}_{h,1}(\boldsymbol{x})' \boldsymbol{B}_x - \tilde{\mu}_{h,1}(\boldsymbol{x})' \boldsymbol{B}_x \|_2 \|(\boldsymbol{B}'_x \tilde{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ \|_2 \| \boldsymbol{B}'_x \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \|_2 \\
&= O_p(h^d) O_p(n^{-\frac{1}{2}} h^{-\frac{3d+1}{2}}) O_p(n^{-\frac{1}{2}} h^{\frac{d+1}{2}}) + O_p(h^d) O(h^{-d+1}) O_p(n^{-\frac{1}{2}} h^{\frac{d+1}{2}}) \\
&+ O_p(h^d) O_p(n^{-\frac{1}{2}} h^{-\frac{3d+1}{2}}) O(h^d) + O_p(n^{-\frac{1}{2}} h^{\frac{d+1}{2}}) O(h^{-d+1}) O(h^d) = O_p(n^{-\frac{1}{2}} h^{\frac{d-1}{2}}) \quad (4.113)
\end{aligned}
$$

Combining (4.109), (4.113), and Lemma 1.4 with the condition L1, the following holds.

$$
\begin{aligned}
& \left| \hat{\nu}_{h,0}(\boldsymbol{x}, y) - \hat{\mu}_{h,1}(\boldsymbol{x})' \hat{\Sigma}_h(\boldsymbol{x})^{+} \hat{\nu}_{h,1}(\boldsymbol{x}, y) - \tilde{\nu}_{h,0}(\boldsymbol{x}, y) + \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^{+} \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \right| \\
& \leq \left| \hat{\nu}_{h,0}(\boldsymbol{x}, y) - \tilde{\nu}_{h,0}(\boldsymbol{x}, y) \right| + \left| \hat{\mu}_{h,1}(\boldsymbol{x})' \hat{\Sigma}_h(\boldsymbol{x})^{+} \hat{\nu}_{h,1}(\boldsymbol{x}, y) - \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^{+} \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \right| \\
& = O_p(n^{-\frac{1}{2}} h^{\frac{d-1}{2}}) \quad (4.114)
\end{aligned}
$$

To get the asymptotic bound of $\hat{\sigma}_h(\boldsymbol{x})$, note that the following holds by Lemma 4.5 and Lemma 4.7 with the conditions L1, D1, D2, and B1.

$$
\begin{aligned}
\left| \frac{\hat{\sigma}_h(\boldsymbol{x})}{c_{h,0,1}(L)} - f(\boldsymbol{x}) \right| &\le \left| \frac{\tilde{\sigma}_h(\boldsymbol{x})}{c_{h,0,1}(L)} - f(\boldsymbol{x}) \right| + \left| \frac{\hat{\sigma}_h(\boldsymbol{x}) - \tilde{\sigma}_h(\boldsymbol{x})}{c_{h,0,1}(L)} \right| \\
&= o(1) + O_p(n^{-\frac{1}{2}} h^{-\frac{d-1}{2}}) = o_p(1) \quad (4.115)
\end{aligned}
$$

By combining (4.115), Lemma 1.4, Lemma 4.5 and Lemma 4.7 with the conditions L1, D1, D2, and B1, we get the following.

$$
\left| \frac{1}{\hat{\sigma}_h(\boldsymbol{x})} \right| = \frac{1}{c_{h,0,1}(L)} \left| \frac{\tilde{\sigma}_h(\boldsymbol{x})}{c_{h,0,1}(L)} \right|^{-1} = O(h^{-(d-1)}) O(1) = O(h^{-(d-1)}) \quad (4.116)
$$

$$
\left| \frac{1}{\hat{\sigma}_h(\boldsymbol{x})} \right| = \frac{1}{c_{h,0,1}(L)} \left| \frac{\hat{\sigma}_h(\boldsymbol{x})}{c_{h,0,1}(L)} \right|^{-1} = O(h^{-(d-1)}) O_p(1) = O_p(h^{-(d-1)}) \quad (4.117)
$$
Also, note that the following holds by (2.3), Lemma 1.4, and Lemma 2.2 with the conditions L1, D1, and R1.

$$0 \le \hat{\nu}_{h,0}(\boldsymbol{x}, y) = \mathbb{E}\left[L\left(\frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2}\right) d^2(Y, y)\right] \le \mathrm{diam}(\Omega)^2 \mathbb{E}\left[L\left(\frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2}\right)\right] = O(h^{d-1}) \quad (4.118)$$

Combining (4.20), (4.49), (4.64), (4.112), and (4.118), we get the following for every $\boldsymbol{B}_x \in \mathcal{B}_x$.

$$\begin{aligned}
& |\tilde{\nu}_{h,0}(\boldsymbol{x}, y) - \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^+ \tilde{\nu}_{h,1}(\boldsymbol{x}, y)| \\
& \le |\tilde{\nu}_{h,0}(\boldsymbol{x}, y)| + |\tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^+ \tilde{\nu}_{h,1}(\boldsymbol{x}, y)| \\
& = |\tilde{\nu}_{h,0}(\boldsymbol{x}, y)| + |\tilde{\mu}_{h,1}(\boldsymbol{x})' \boldsymbol{B}_x (\boldsymbol{B}_x' \tilde{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+ \boldsymbol{B}_x' \tilde{\nu}_{h,1}(\boldsymbol{x}, y)| \\
& \le |\tilde{\nu}_{h,0}(\boldsymbol{x}, y)| + \|\tilde{\mu}_{h,1}(\boldsymbol{x})' \boldsymbol{B}_x\|_2 \|(\boldsymbol{B}_x' \tilde{\mu}_{h,2}(\boldsymbol{x}) \boldsymbol{B}_x)^+\|_2 \|\boldsymbol{B}_x' \tilde{\nu}_{h,1}(\boldsymbol{x}, y)\|_2 \\
& = O(h^{d-1}) + O(h^{d+1}) O(h^{-(d+1)}) O(h^d) = O(h^{d-1}) \quad (4.119)
\end{aligned}$$

Finally, combining (4.79), (4.80), (4.116), (4.117), (4.119), Lemma 1.4, Lemma 4.5, and Lemma 4.7 with the conditions L1, D1, D2, and B1, the following holds.

$$\begin{aligned}
& \left| \hat{M}_{h,1}(\boldsymbol{x}, y) - \tilde{M}_{h,1}(\boldsymbol{x}, y) \right| \\
& = \left| \frac{\hat{\nu}_{h,0}(\boldsymbol{x}, y) - \hat{\mu}_{h,1}(\boldsymbol{x})' \hat{\Sigma}_h(\boldsymbol{x})^+ \hat{\nu}_{h,1}(\boldsymbol{x}, y)}{\hat{\sigma}_h(\boldsymbol{x})} - \frac{\tilde{\nu}_{h,0}(\boldsymbol{x}, y) + \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^+ \tilde{\nu}_{h,1}(\boldsymbol{x}, y)}{\tilde{\sigma}_h(\boldsymbol{x})} \right| \\
& \le \left| \frac{1}{\hat{\sigma}_h(\boldsymbol{x})} \right| \left| \hat{\nu}_{h,0}(\boldsymbol{x}, y) - \hat{\mu}_{h,1}(\boldsymbol{x})' \hat{\Sigma}_h(\boldsymbol{x})^+ \hat{\nu}_{h,1}(\boldsymbol{x}, y) - \tilde{\nu}_{h,0}(\boldsymbol{x}, y) + \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^+ \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \right| \\
& \quad + \left| \frac{1}{\hat{\sigma}_h(\boldsymbol{x})} - \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \right| \left| \tilde{\nu}_{h,0}(\boldsymbol{x}, y) - \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^+ \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \right| \\
& = \left| \frac{1}{\hat{\sigma}_h(\boldsymbol{x})} \right| \left| \hat{\nu}_{h,0}(\boldsymbol{x}, y) - \hat{\mu}_{h,1}(\boldsymbol{x})' \hat{\Sigma}_h(\boldsymbol{x})^+ \hat{\nu}_{h,1}(\boldsymbol{x}, y) - \tilde{\nu}_{h,0}(\boldsymbol{x}, y) + \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^+ \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \right| \\
& \quad + \left| \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \right| \left| \frac{1}{\hat{\sigma}_h(\boldsymbol{x})} \right| \left| \hat{\sigma}_h(\boldsymbol{x}) - \tilde{\sigma}_h(\boldsymbol{x}) \right| \left| \tilde{\nu}_{h,0}(\boldsymbol{x}, y) - \tilde{\mu}_{h,1}(\boldsymbol{x})' \tilde{\Sigma}_h(\boldsymbol{x})^+ \tilde{\nu}_{h,1}(\boldsymbol{x}, y) \right| \\
& = O_p(h^{-(d-1)}) O_p(n^{-\frac{1}{2}h^{\frac{d-1}{2}}}) + O(h^{-(d-1)}) O_p(h^{-(d-1)}) O_p(n^{-\frac{1}{2}h^{\frac{d-1}{2}}}) O(h^{d-1}) = O_p(n^{-\frac{1}{2}h^{-\frac{d-1}{2}}}) \quad (4.120)
\end{aligned}$$

Therefore, (4.120) with the condition B1 ends the proof. $\square$

By using Lemma 4.10, we can get a uniform version stated below.

**Lemma 4.11.** Suppose the conditions L1, D1, D2, B1, and R1 hold. Then the following holds for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$.

$$\sup_{y \in \Omega} \left| \hat{M}_{h,1}(\boldsymbol{x}, y) - \tilde{M}_{h,1}(\boldsymbol{x}, y) \right| = o_p(1) \quad \text{as } n \to \infty$$
Proof of Lemma 4.11. Let $\boldsymbol{x} \in \mathbb{S}^{d-1}$ be given. Note the following hold by the condition R1 for every $w_1, w_2 \in \Omega$, $\boldsymbol{B}_x \in \mathcal{B}_x$, $h = h_n > 0$, and $n \in \mathbb{N}$.

$$ 
\begin{aligned}
|\tilde{\nu}_{h,0}(\boldsymbol{x}, w_1) - \tilde{\nu}_{h,0}(\boldsymbol{x}, w_2)| &= \left| \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) \left( d^2(Y, w_1) - d^2(Y, w_2) \right) \right] \right| \\
&\le \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) \left| d^2(Y, w_1) - d^2(Y, w_2) \right| \right] \\
&\le 2\mathrm{diam}(\Omega) d(w_1, w_2) \tilde{\mu}_{h,0}(\boldsymbol{x})
\end{aligned} 
\quad (4.121) $$

$$ 
\begin{aligned}
|\hat{\nu}_{h,0}(\boldsymbol{x}, w_1) - \hat{\nu}_{h,0}(\boldsymbol{x}, w_2)| &= \left| \frac{1}{n} \sum_{i=1}^{n} L \left( \frac{1 - \boldsymbol{X}'_i \boldsymbol{x}}{h^2} \right) \left( d^2(Y_i, w_1) - d^2(Y_i, w_2) \right) \right| \\
&\le \frac{1}{n} \sum_{i=1}^{n} L \left( \frac{1 - \boldsymbol{X}'_i \boldsymbol{x}}{h^2} \right) \left| d^2(Y_i, w_1) - d^2(Y_i, w_2) \right| \\
&\le 2\mathrm{diam}(\Omega) d(w_1, w_2) \hat{\mu}_{h,0}(\boldsymbol{x})
\end{aligned} 
\quad (4.122) $$

$$ 
\begin{aligned}
\| \boldsymbol{B}'_x \tilde{\nu}_{h,1}(\boldsymbol{x}, w_1) - \boldsymbol{B}'_x \tilde{\nu}_{h,1}(\boldsymbol{x}, w_2) \|_2 &= \left\| \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) \boldsymbol{B}'_x \boldsymbol{H}_x(\boldsymbol{X}) \left( d^2(Y, w_1) - d^2(Y, w_2) \right) \right] \right\|_2 \\
&\le \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) \| \boldsymbol{B}'_x \boldsymbol{H}_x(\boldsymbol{X}) \|_2 \left| d^2(Y, w_1) - d^2(Y, w_2) \right| \right] \\
&\le 2\mathrm{diam}(\Omega) d(w_1, w_2) \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) \| \boldsymbol{B}'_x \boldsymbol{H}_x(\boldsymbol{X}) \|_2 \right]
\end{aligned} 
\quad (4.123) $$

$$ 
\begin{aligned}
\| \boldsymbol{B}'_x \hat{\nu}_{h,1}(\boldsymbol{x}, w_1) - \boldsymbol{B}'_x \hat{\nu}_{h,1}(\boldsymbol{x}, w_2) \|_2 &= \left\| \frac{1}{n} \sum_{i=1}^{n} L \left( \frac{1 - \boldsymbol{X}'_i \boldsymbol{x}}{h^2} \right) \boldsymbol{B}'_x \boldsymbol{H}_x(\boldsymbol{X}_i) \left( d^2(Y, w_1) - d^2(Y, w_2) \right) \right\|_2 \\
&\le \frac{1}{n} \sum_{i=1}^{n} \left[ L \left( \frac{1 - \boldsymbol{X}'_i \boldsymbol{x}}{h^2} \right) \| \boldsymbol{B}'_x \boldsymbol{H}_x(\boldsymbol{X}_i) \|_2 \left| d^2(Y, w_1) - d^2(Y, w_2) \right| \right] \\
&\le 2\mathrm{diam}(\Omega) d(w_1, w_2) \left( \frac{1}{n} \sum_{i=1}^{n} L \left( \frac{1 - \boldsymbol{X}'_i \boldsymbol{x}}{h^2} \right) \| \boldsymbol{B}'_x \boldsymbol{H}_x(\boldsymbol{X}_i) \|_2 \right)
\end{aligned} 
\quad (4.124) $$

Note that the following holds by (4.112) for every $\boldsymbol{B}_x \in \mathcal{B}_x$.

$$ \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) \| \boldsymbol{B}'_x \boldsymbol{H}_x(\boldsymbol{X}) \|_2 \right] \le \left( \tilde{\mu}_{h,0}(\boldsymbol{x}) \mathbb{E} \left[ L \left( \frac{1 - \boldsymbol{X}'\boldsymbol{x}}{h^2} \right) \| \boldsymbol{B}'_x \boldsymbol{H}_x(\boldsymbol{X}) \|_2^2 \right] \right)^{\frac{1}{2}} = O(h^d) \quad (4.125) $$

Also, the following holds by (4.59) and Lemma 1.4 with the condition L1 for every $\boldsymbol{B}_x \in \mathcal{B}_x$.

$$ 
\begin{aligned}
\mathrm{Var} \left[ \frac{1}{n} \sum_{i=1}^{n} L \left( \frac{1 - \boldsymbol{X}'_i \boldsymbol{x}}{h^2} \right) \| \boldsymbol{B}'_x \boldsymbol{H}_x(\boldsymbol{X}_i) \|_2 \right] &= \frac{1}{n} \mathrm{Var} \left[ L \left( \frac{1 - \boldsymbol{X}'_i \boldsymbol{x}}{h^2} \right) \| \boldsymbol{B}'_x \boldsymbol{H}_x(\boldsymbol{X}) \|_2 \right] \\
&\le \frac{1}{n} \mathbb{E} \left[ L^2 \left( \frac{1 - \boldsymbol{X}'_i \boldsymbol{x}}{h^2} \right) (\boldsymbol{B}'_x \boldsymbol{H}_x(\boldsymbol{X}))' (\boldsymbol{B}'_x \boldsymbol{H}_x(\boldsymbol{X})) \right] \\
&= \frac{1}{n} \left[ \sup_{\boldsymbol{z} \in \mathbb{S}^{d-1}} f(\boldsymbol{z}) \right] c_{h,2,2}(L) = O(n^{-1} h^{d+1})
\end{aligned} 
\quad (4.126) $$
Combining (4.125), (4.126), and Chebyshev's inequality, the following holds for every $\mathbf{B}_x \in \mathcal{B}_x$.

$$ \left| \frac{1}{n} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}'_i \mathbf{x}}{h^2}\right) \|\mathbf{B}'_x \mathbf{H}_x(\mathbf{X}_i)\|_2 - \mathbb{E}\left[L\left(\frac{1 - \mathbf{X}'_i \mathbf{x}}{h^2}\right) \|\mathbf{B}'_x \mathbf{H}_x(\mathbf{X})\|_2\right] \right| \\ = O_p\left(\sqrt{\mathrm{Var}\left[\frac{1}{n} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}'_i \mathbf{x}}{h^2}\right) \|\mathbf{B}'_x \mathbf{H}_x(\mathbf{X}_i)\|_2\right]}\right) = O_p\left(n^{-\frac{1}{2}} h^{\frac{d+1}{2}}\right) \quad (4.127) $$

Also, combining (4.125) and (4.127), the following holds for every $\mathbf{B}_x \in \mathcal{B}_x$.

$$ \frac{1}{n} \sum_{i=1}^{n} L\left(\frac{1 - \mathbf{X}'_i \mathbf{x}}{h^2}\right) \|\mathbf{B}'_x \mathbf{H}_x(\mathbf{X}_i)\|_2 = O(h^d) + O_p(n^{-\frac{1}{2}} h^{\frac{d+1}{2}}) = O_p(h^d) \quad (4.128) $$

Now note that the following hold by (4.20), (4.121), (4.122), (4.123), and (4.124) for every $w_1, w_2 \in \Omega$, $\mathbf{B}_x \in \mathcal{B}_x$, $h = h_n > 0$, and $n \in \mathbb{N}$.

$$ \begin{aligned} & \left| \tilde{M}_{h,1}(\mathbf{x}, w_1) - \tilde{M}_{h,1}(\mathbf{x}, w_2) \right| \\ &= \left| \frac{1}{\tilde{\sigma}_h(\mathbf{x})} \left| (\tilde{\nu}_{h,0}(\mathbf{x}, w_1) - \tilde{\nu}_{h,0}(\mathbf{x}, w_2)) + \tilde{\mu}_{h,1}(\mathbf{x})' \tilde{\Sigma}_h(\mathbf{x})^+ (\tilde{\nu}_{h,1}(\mathbf{x}, w_1) - \tilde{\nu}_{h,1}(\mathbf{x}, w_2)) \right| \right| \\ &= \left| \frac{1}{\tilde{\sigma}_h(\mathbf{x})} \left| (\tilde{\nu}_{h,0}(\mathbf{x}, w_1) - \tilde{\nu}_{h,0}(\mathbf{x}, w_2)) + \tilde{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x (\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \mathbf{B}'_x (\tilde{\nu}_{h,1}(\mathbf{x}, w_1) - \tilde{\nu}_{h,1}(\mathbf{x}, w_2)) \right| \right| \\ &\leq \left| \frac{1}{\tilde{\sigma}_h(\mathbf{x})} \left| \tilde{\nu}_{h,0}(\mathbf{x}, w_1) - \tilde{\nu}_{h,0}(\mathbf{x}, w_2) \right| \right| \\ &\quad + \left| \frac{1}{\tilde{\sigma}_h(\mathbf{x})} \left| \| \mathbf{B}'_x \tilde{\mu}_{h,1}(\mathbf{x}) \|_2 \|(\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+\|_2 \| \mathbf{B}'_x \tilde{\nu}_{h,1}(\mathbf{x}, w_1) - \mathbf{B}'_x \tilde{\nu}_{h,1}(\mathbf{x}, w_2)\|_2 \right| \right| \\ &\leq \left| \frac{2\mathrm{diam}(\Omega)d(w_1, w_2)}{\tilde{\sigma}_h(\mathbf{x})} \right| \tilde{\mu}_{h,0}(\mathbf{x}) \\ &\quad + \left| \frac{2\mathrm{diam}(\Omega)d(w_1, w_2)}{\tilde{\sigma}_h(\mathbf{x})} \right| \| \mathbf{B}'_x \tilde{\mu}_{h,1}(\mathbf{x}) \|_2 \|(\mathbf{B}'_x \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+\|_2 \mathbb{E} \left[ L\left(\frac{1 - \mathbf{X}'_i \mathbf{x}}{h^2}\right) \|\mathbf{B}'_x \mathbf{H}_x(\mathbf{X})\|_2 \right] \end{aligned} \quad (4.129) $$

$$ \begin{aligned} & \left| \hat{M}_{h,1}(\mathbf{x}, w_1) - \hat{M}_{h,1}(\mathbf{x}, w_2) \right| \\ &= \left| \frac{1}{\hat{\sigma}_h(\mathbf{x})} \left| (\hat{\nu}_{h,0}(\mathbf{x}, w_1) - \hat{\nu}_{h,0}(\mathbf{x}, w_2)) + \hat{\mu}_{h,1}(\mathbf{x})' \hat{\Sigma}_h(\mathbf{x})^+ (\hat{\nu}_{h,1}(\mathbf{x}, w_1) - \hat{\nu}_{h,1}(\mathbf{x}, w_2)) \right| \right| \\ &= \left| \frac{1}{\hat{\sigma}_h(\mathbf{x})} \left| (\hat{\nu}_{h,0}(\mathbf{x}, w_1) - \hat{\nu}_{h,0}(\mathbf{x}, w_2)) + \hat{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x (\mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \mathbf{B}'_x (\hat{\nu}_{h,1}(\mathbf{x}, w_1) - \hat{\nu}_{h,1}(\mathbf{x}, w_2)) \right| \right| \\ &\leq \left| \frac{1}{\hat{\sigma}_h(\mathbf{x})} \left| \hat{\nu}_{h,0}(\mathbf{x}, w_1) - \hat{\nu}_{h,0}(\mathbf{x}, w_2) \right| \right| \\ &\quad + \left| \frac{1}{\hat{\sigma}_h(\mathbf{x})} \left| \| \mathbf{B}'_x \hat{\mu}_{h,1}(\mathbf{x}) \|_2 \|(\mathbf{B}'_x \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+\|_2 \| \mathbf{B}'_x \hat{\nu}_{h,1}(\mathbf{x}, w_1) - \mathbf{B}'_x \hat{\nu}_{h,1}(\mathbf{x}, w_2)\|_2 \right| \right| \\ &\leq \left| \frac{2\mathrm{diam}(\Omega)d(w_1, w_2)}{\hat{\sigma}_h(\mathbf{x})} \right| \hat{\mu}_{h,0}(\mathbf{x}) \end{aligned} $$
$$ + \left| \frac{2\mathrm{diam}(\Omega)d(w_1, w_2)}{\hat{\sigma}_h(\boldsymbol{x})} \right| \|\boldsymbol{B}'_x \hat{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})\|_2 \|(\boldsymbol{B}'_x \hat{\boldsymbol{\mu}}_{h,2}(\boldsymbol{x})\boldsymbol{B}_x)^+\|_2 \left( \frac{1}{n} \sum_{i=1}^n L\left(\frac{1-\boldsymbol{X}'_i\boldsymbol{x}}{h^2}\right) \|\boldsymbol{B}'_x \boldsymbol{H}_x(\boldsymbol{X}_i)\|_2 \right) \quad (4.130) $$

Combining (4.49), (4.64), (4.70), (4.116), (4.117), (4.125), (4.128), Lemma 1.4, Lemma 2.2, and Theorem 2.2 with the conditions L1, D1, D2, and B1, the following hold for every $\boldsymbol{B}_x \in \mathcal{B}_x$.

$$ =O(h^{-(d-1)}) \left( O(h^{d-1}) + O(h^d)O(h^{-(d+1)})O(h^d) \right) = O(1) \quad \text{as} \quad h \downarrow 0 \quad (4.131) $$

$$ =O_p(h^{-(d-1)}) \left( O_p(h^{d-1}) + O_p(h^d)O_p(h^{-(d+1)})O_p(h^d) \right) = O_p(1) \quad \text{as} \quad n \to \infty \quad (4.132) $$

Now let $\epsilon, \delta > 0$ be given. By (4.131) and (4.132), there exists a constant $M > 0$ such that the following hold.

$$ \limsup_{n \to \infty} (4.131) \le M, \quad \limsup_{n \to \infty} \mathbb{P}((4.132) > M) \le \delta \quad (4.133) $$

Define a constant $r := 8^{-1}M^{-1}\mathrm{diam}(\Omega)^{-1}\epsilon$. By the condition R1, there exists a collection of open balls $\{B_d(y_j, r) : j = 1, 2, \dots, N\}$ which cover $\Omega$. Then we define events $E_n, A_{n,1}, A_{n,2}, \dots, A_{n,N}, B_n$, and $F_n$ as the following for every $n \in \mathbb{N}$.

$$ \begin{align*} E_n &:= \left\{ \sup_{y \in \Omega} \left| \hat{M}_{h,1}(\boldsymbol{x}, y) - \tilde{M}_{h,1}(\boldsymbol{x}, y) \right| > \epsilon \right\} \\ A_{n,j} &:= \left\{ \left| \hat{M}_{h,1}(\boldsymbol{x}, y_j) - \tilde{M}_{h,1}(\boldsymbol{x}, y_j) \right| > \frac{\epsilon}{2} \right\}, \quad j = 1, 2, \dots, N \\ B_n &:= \left\{ \sup_{w_1, w_2 \in \Omega, d(w_1, w_2) < r} \left| \left[ \hat{M}_{h,1}(\boldsymbol{x}, w_1) - \tilde{M}_{h,1}(\boldsymbol{x}, w_1) \right] + \left[ \hat{M}_{h,1}(\boldsymbol{x}, w_2) - \tilde{M}_{h,1}(\boldsymbol{x}, w_2) \right] \right| > \frac{\epsilon}{2} \right\} \\ F_n &:= \left\{ (4.131) > M \right\} \cup \left\{ (4.132) > M \right\} \end{align*} $$

Since $B_n^c \cap (\cap_{j=1}^N A_{n,j}^c) \subset E_n^c$ for every $n \in \mathbb{N}$, the following holds by the union bound.

$$ \mathbb{P}(E_n) \le \sum_{j=1}^{N} \mathbb{P}(A_{n,j}) + \mathbb{P}(B_n) \le \sum_{j=1}^{N} \mathbb{P}(A_{n,j}) + \mathbb{P}(B_n \cap F_n^c) + \mathbb{P}(F_n), \quad \forall n \in \mathbb{N} \quad (4.134) $$

Combining (4.131), (4.132), and (4.133), the following holds on the event $F_n^c$ for every $n \in \mathbb{N}$.

$$ \sup_{w_1, w_2 \in \Omega, d(w_1, w_2) < r} \left| \left[ \hat{M}_{h,1}(\boldsymbol{x}, w_1) - \tilde{M}_{h,1}(\boldsymbol{x}, w_1) \right] + \left[ \hat{M}_{h,1}(\boldsymbol{x}, w_2) - \tilde{M}_{h,1}(\boldsymbol{x}, w_2) \right] \right| $$
$$ \leq 2rdiam(\Omega) \times (4.131) + 2rdiam(\Omega) \times (4.132) \leq 4rdiam(\Omega)M = \frac{\epsilon}{2} \quad (4.135) $$

Therefore, (4.135) implies that $B_n \cap F_n^c = \emptyset$ for every $n \in \mathbb{N}$. Also, combining (4.134) and Lemma 4.10 with the conditions L1, D1, D2, B1, and R1, we get the following.

$$ \limsup_{n \to \infty} \mathbb{P}(E_n) \leq \sum_{j=1}^{N} \limsup_{n \to \infty} \mathbb{P}(A_{n,j}) + \limsup_{n \to \infty} \mathbb{P}(F_n) \leq \delta \quad (4.136) $$

Since $\epsilon, \delta > 0$ are arbitrary, (4.136) ends the proof. □

The next Theorem is a direct consequence of Lemma 4.11, which states that $d(\hat{m}_{h,1}(\boldsymbol{x}), \tilde{m}_{h,1}(\boldsymbol{x}))$ converges to 0 in probability for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$. The proof is the same as Theorem 4.3 of Im et al. (2024).

**Theorem 4.3** (Convergence of the stochastic deviation of $\hat{m}_{h,1}$). Suppose the conditions L1, D1, D2, B1, R1, and M1-1 hold. Then the following holds for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$.

$$ d(\hat{m}_{h,1}(\boldsymbol{x}), \tilde{m}_{h,1}(\boldsymbol{x})) = o_p(1) \quad \text{as} \quad n \to \infty $$

Theorem 4.1 and Theorem 4.3 imply the consistency of $\hat{m}_{h,1}(\boldsymbol{x})$ for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$. The following is the theorem about the convergence rate of $\hat{m}_{h,1}(\boldsymbol{x})$ for any given $\boldsymbol{x} \in \mathbb{S}^{d-1}$.

**Theorem 4.4** (Convergence rate of the stochastic deviation of $\hat{m}_{h,1}$). Suppose the conditions L1, D1, D2, B1, R1, M1-1, and M3-1 hold. Then the following holds for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$, where $\tilde{\beta}_1 > 1$ is the constant defined in M3-1.

$$ d(\hat{m}_{h,1}(\boldsymbol{x}), \tilde{m}_{h,1}(\boldsymbol{x}))^{\tilde{\beta}_1 - 1} = O_p(n^{-\frac{1}{2}}h^{-\frac{d-1}{2}}) \quad \text{as} \quad n \to \infty $$

*Proof of Theorem 4.4.* For the convenience, we define functions $\hat{T}_{h,1}: \mathbb{S}^{d-1} \times \Omega \to \mathbb{R}$ and $D_{h,1}: \mathbb{S}^{d-1} \times \Omega \times \Omega \to \mathbb{R}$ as the following for every $h = h_n > 0$ and $n \in \mathbb{N}$.

$$ \hat{T}_{h,1}(\boldsymbol{x}, y) := \hat{M}_{h,1}(\boldsymbol{x}, y) - \tilde{M}_{h,1}(\boldsymbol{x}, y), \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1}, \forall y \in \Omega $$

$$ D_{h,1}(\boldsymbol{x}, w, y) := d^2(w, y) - d^2(w, \tilde{m}_{h,1}(\boldsymbol{x})), \quad \forall \boldsymbol{x} \in \mathbb{S}^{d-1}, \forall w, y \in \Omega $$

Note the following hold by (4.79) and (4.80) for every $\boldsymbol{x} \in \mathbb{S}^{d-1}$, $y \in \Omega$, $h = h_n > 0$, and $n \in \mathbb{N}$.

$$ \hat{T}_{h,1}(\boldsymbol{x}, y) - \hat{T}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) = \frac{1}{\hat{\sigma}_h(\boldsymbol{x})} \left( \hat{\nu}_{h,0}(\boldsymbol{x}, y) - \hat{\nu}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right) - \frac{1}{\hat{\sigma}_h(\boldsymbol{x})} \hat{\mu}_{h,1}(\boldsymbol{x})' \hat{\Sigma}_h(\boldsymbol{x})^+ \left( \hat{\nu}_{h,1}(\boldsymbol{x}, y) - \hat{\nu}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right) $$
$$
\begin{align}
& -\frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \left( \tilde{\nu}_{h,0}(\boldsymbol{x}, y) - \tilde{\nu}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right) + \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \tilde{\boldsymbol{\Sigma}}_h(\boldsymbol{x})^+ \left( \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, y) - \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right) \nonumber \\
&= \left( \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} - \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \right) \left( \tilde{\nu}_{h,0}(\boldsymbol{x}, y) - \tilde{\nu}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right) \nonumber \\
&\quad - \left( \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \tilde{\boldsymbol{\Sigma}}_h(\boldsymbol{x})^+ - \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \tilde{\boldsymbol{\Sigma}}_h(\boldsymbol{x})^+ \right) \left( \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, y) - \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right) \nonumber \\
&\quad + \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \left( \tilde{\nu}_{h,0}(\boldsymbol{x}, y) - \tilde{\nu}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) - \tilde{\nu}_{h,0}(\boldsymbol{x}, y) + \tilde{\nu}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right) \nonumber \\
&\quad - \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \tilde{\boldsymbol{\Sigma}}_h(\boldsymbol{x})^+ \left( \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, y) - \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) - \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, y) + \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right) \nonumber \\
&= \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \left\{ \tilde{\nu}_{h,0}(\boldsymbol{x}, y) - \tilde{\nu}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) - \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \tilde{\boldsymbol{\Sigma}}_h(\boldsymbol{x})^+ \left( \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, y) - \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right) \right\} \nonumber \\
&\quad - \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \left\{ \tilde{\nu}_{h,0}(\boldsymbol{x}, y) - \tilde{\nu}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) - \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \tilde{\boldsymbol{\Sigma}}_h(\boldsymbol{x})^+ \left( \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, y) - \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right) \right\} \nonumber \\
&\quad + \left( \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} - \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \right) \left( \tilde{\nu}_{h,0}(\boldsymbol{x}, y) - \tilde{\nu}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right) \nonumber \\
&\quad - \left( \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \tilde{\boldsymbol{\Sigma}}_h(\boldsymbol{x})^+ - \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \tilde{\boldsymbol{\Sigma}}_h(\boldsymbol{x})^+ \right) \left( \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, y) - \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right) \tag{4.137}
\end{align}
$$

Let $\boldsymbol{x} \in \mathbb{S}^{d-1}$ be given. Define a function $U_{h,1} : \mathbb{S}^{d-1} \times \Omega \times \Omega \to \mathbb{R}$ as the following for every $h > 0$, where $\tilde{W}_{\boldsymbol{x},h}$ is defined by (4.72) for every $h > 0$.

$$ U_{h,1}(\boldsymbol{z}, w, y) := \tilde{W}_{\boldsymbol{x},h}(\boldsymbol{z}) D_{h,1}(\boldsymbol{x}, w, y), \quad \forall \boldsymbol{z} \in \mathbb{S}^{d-1}, \forall w, y \in \Omega \quad (4.138) $$

By (4.72) and (4.138), the following hold for every $y \in \Omega$, $h = h_n > 0$, and $n \in \mathbb{N}$.

$$
\begin{align}
& \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \left\{ \tilde{\nu}_{h,0}(\boldsymbol{x}, y) - \tilde{\nu}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) - \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \tilde{\boldsymbol{\Sigma}}_h(\boldsymbol{x})^+ \left( \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, y) - \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right) \right\} \nonumber \\
&= \frac{1}{n} \sum_{i=1}^{n} \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} L\left(\frac{1 - \mathbf{X}'_i \boldsymbol{x}}{h^2}\right) (1 - \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \tilde{\boldsymbol{\Sigma}}_h(\boldsymbol{x})^+ \mathbf{H}_{\boldsymbol{x}}(\mathbf{X}_i)) D_{h,1}(\boldsymbol{x}, Y_i, y) \nonumber \\
&= \frac{1}{n} \sum_{i=1}^{n} \tilde{W}_{\boldsymbol{x},h}(\mathbf{X}_i) D_{h,1}(\boldsymbol{x}, Y_i, y) = \frac{1}{n} \sum_{i=1}^{n} U_{h,1}(\mathbf{X}_i, Y_i, y) \tag{4.139}
\end{align}
$$

$$
\begin{align}
& \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \left\{ \tilde{\nu}_{h,0}(\boldsymbol{x}, y) - \tilde{\nu}_{h,0}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) - \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \tilde{\boldsymbol{\Sigma}}_h(\boldsymbol{x})^+ \left( \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, y) - \tilde{\boldsymbol{\nu}}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right) \right\} \nonumber \\
&= \mathbb{E} \left[ \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} L \left( \frac{1 - \mathbf{X}' \boldsymbol{x}}{h^2} \right) (1 - \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \tilde{\boldsymbol{\Sigma}}_h(\boldsymbol{x})^+ \mathbf{H}_{\boldsymbol{x}}(\mathbf{X})) D_{h,1}(\boldsymbol{x}, Y, y) \right] \nonumber \\
&= \mathbb{E} \left[ \tilde{W}_{\boldsymbol{x},h}(\mathbf{X}) D_{h,1}(\boldsymbol{x}, Y, y) \right] = \mathbb{E} \left[ U_{h,1}(\mathbf{X}, Y, y) \right] \tag{4.140}
\end{align}
$$

Combining (4.20), (4.122), (4.124), (4.137), (4.139), and (4.140) with the condition R1, we get the following for every $y \in \Omega$, $\mathbf{B}_{\boldsymbol{x}} \in \mathcal{B}_{\boldsymbol{x}}$, $h = h_n > 0$, and $n \in \mathbb{N}$.

$$ \left| \hat{T}_{h,1}(\boldsymbol{x}, y) - \hat{T}_{h,1}(\boldsymbol{x}, \tilde{m}_{h,1}(\boldsymbol{x})) \right| $$
$$
\begin{align*}
&\le \left| \frac{1}{n} \sum_{i=1}^{n} U_{h,1}(\mathbf{X}_i, Y_i, y) - \mathbb{E}\left[U_{h,1}(\mathbf{X}, Y, y)\right] \right| + \left| \frac{1}{\hat{\sigma}_h(\mathbf{x})} - \frac{1}{\tilde{\sigma}_h(\mathbf{x})} \right| \left| \hat{\nu}_{h,0}(\mathbf{x}, y) - \hat{\nu}_{h,0}(\mathbf{x}, \tilde{m}_{h,1}(\mathbf{x})) \right| \\
&\quad + \left| \left( \frac{1}{\hat{\sigma}_h(\mathbf{x})} \hat{\mu}_{h,1}(\mathbf{x})' \hat{\Sigma}_h(\mathbf{x})^+ - \frac{1}{\tilde{\sigma}_h(\mathbf{x})} \tilde{\mu}_{h,1}(\mathbf{x})' \tilde{\Sigma}_h(\mathbf{x})^+ \right) \left( \hat{\nu}_{h,1}(\mathbf{x}, y) - \hat{\nu}_{h,1}(\mathbf{x}, \tilde{m}_{h,1}(\mathbf{x})) \right) \right| \\
&\le \left| \frac{1}{n} \sum_{i=1}^{n} U_{h,1}(\mathbf{X}_i, Y_i, y) - \mathbb{E}\left[U_{h,1}(\mathbf{X}, Y, y)\right] \right| + \left| \frac{1}{\hat{\sigma}_h(\mathbf{x})} - \frac{1}{\tilde{\sigma}_h(\mathbf{x})} \right| \left| \hat{\nu}_{h,0}(\mathbf{x}, y) - \hat{\nu}_{h,0}(\mathbf{x}, \tilde{m}_{h,1}(\mathbf{x})) \right| \\
&\quad + \left\| \frac{1}{\hat{\sigma}_h(\mathbf{x})} \hat{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x (\mathbf{B}_x' \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ - \frac{1}{\tilde{\sigma}_h(\mathbf{x})} \tilde{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x (\mathbf{B}_x' \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \right\|_2 \\
&\quad \times \left\| \mathbf{B}_x' \hat{\nu}_{h,1}(\mathbf{x}, y) - \mathbf{B}_x' \hat{\nu}_{h,1}(\mathbf{x}, \tilde{m}_{h,1}(\mathbf{x})) \right\|_2 \\
&\le \left| \frac{1}{n} \sum_{i=1}^{n} U_{h,1}(\mathbf{X}_i, Y_i, y) - \mathbb{E}\left[U_{h,1}(\mathbf{X}, Y, y)\right] \right| + 2\mathrm{diam}(\Omega)d(y, \tilde{m}_{h,1}(\mathbf{x}))\hat{\mu}_{h,0}(\mathbf{x}) \left| \frac{1}{\hat{\sigma}_h(\mathbf{x})} - \frac{1}{\tilde{\sigma}_h(\mathbf{x})} \right| \\
&\quad + 2\mathrm{diam}(\Omega)d(w_1, w_2) \left( \frac{1}{n} \sum_{i=1}^{n} L\left( \frac{1 - \mathbf{X}_i'\mathbf{x}}{h^2} \right) \| \mathbf{B}_x' \mathbf{H}_x(\mathbf{X}_i) \|_2 \right) \\
&\quad \times \left\| \frac{1}{\hat{\sigma}_h(\mathbf{x})} \hat{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x (\mathbf{B}_x' \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ - \frac{1}{\tilde{\sigma}_h(\mathbf{x})} \tilde{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x (\mathbf{B}_x' \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \right\|_2 \tag{4.141}
\end{align*}
$$

Note that (4.49), (4.61), (4.64), (4.70), (4.116), (4.117), Lemma 1.4, and Lemma 4.7 with the conditions L1, D1, D2, and B1 imply the following for every $\mathbf{B}_x \in \mathcal{B}_x$.

$$
\begin{align*}
& \left\| \frac{1}{\hat{\sigma}_h(\mathbf{x})} \hat{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x (\mathbf{B}_x' \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ - \frac{1}{\tilde{\sigma}_h(\mathbf{x})} \tilde{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x (\mathbf{B}_x' \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \right\|_2 \\
&\le \left\| \frac{1}{\hat{\sigma}_h(\mathbf{x})} \hat{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x \left( (\mathbf{B}_x' \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ - (\mathbf{B}_x' \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \right) \right\|_2 \\
&\quad + \left\| \frac{1}{\hat{\sigma}_h(\mathbf{x})} \left( \hat{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x - \tilde{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x \right) (\mathbf{B}_x' \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \right\|_2 \\
&\quad + \left\| \left( \frac{1}{\hat{\sigma}_h(\mathbf{x})} - \frac{1}{\tilde{\sigma}_h(\mathbf{x})} \right) \tilde{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x (\mathbf{B}_x' \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \right\|_2 \\
&\le \left| \frac{1}{\hat{\sigma}_h(\mathbf{x})} \right| \left\| \hat{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x \right\|_2 \left\| (\mathbf{B}_x' \hat{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ - (\mathbf{B}_x' \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \right\|_2 \\
&\quad + \left| \frac{1}{\hat{\sigma}_h(\mathbf{x})} \right| \left\| \hat{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x - \tilde{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x \right\|_2 \left\| (\mathbf{B}_x' \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \right\|_2 \\
&\quad + \left| \frac{1}{\tilde{\sigma}_h(\mathbf{x})} \right| \left| \frac{1}{\hat{\sigma}_h(\mathbf{x})} \right| \left| \hat{\sigma}_h(\mathbf{x}) - \tilde{\sigma}_h(\mathbf{x}) \right| \left\| \tilde{\mu}_{h,1}(\mathbf{x})' \mathbf{B}_x \right\|_2 \left\| (\mathbf{B}_x' \tilde{\mu}_{h,2}(\mathbf{x}) \mathbf{B}_x)^+ \right\|_2 \\
&= O_p(h^{-(d-1)}) O_p(h^d) O_p(n^{-\frac{1}{2}} h^{-\frac{3d+1}{2}}) + O_p(h^{-(d-1)}) O_p(n^{-\frac{1}{2}} h^{\frac{d+1}{2}}) O(h^{-(d+1)}) \\
&\quad + O(h^{-(d-1)}) O_p(h^{-(d-1)}) O_p(n^{-\frac{1}{2}} h^{\frac{d-1}{2}}) O(h^{d+1}) O(h^{-(d+1)}) = O_p(n^{-\frac{1}{2}} h^{-\frac{3d-1}{2}}) \tag{4.142}
\end{align*}
$$

Also, by combining (4.116), (4.117), (4.128), (4.142), Lemma 1.4, Theorem 2.4, and Lemma 4.7 with the conditions L1, D1, D2, and B1, the following holds for every $\mathbf{B}_x \in \mathcal{B}_x$.

$$
\hat{\mu}_{h,0}(\mathbf{x}) \left| \frac{1}{\hat{\sigma}_h(\mathbf{x})} - \frac{1}{\tilde{\sigma}_h(\mathbf{x})} \right| + \left( \frac{1}{n} \sum_{i=1}^{n} L\left( \frac{1 - \mathbf{X}_i'\mathbf{x}}{h^2} \right) \| \mathbf{B}_x' \mathbf{H}_x(\mathbf{X}_i) \|_2 \right) \times (4.142)
$$
$$
\begin{align}
&= \hat{\mu}_{h,0}(\boldsymbol{x}) \left| \frac{1}{\tilde{\sigma}_h(\boldsymbol{x})} \right| \left| \frac{1}{\hat{\sigma}_h(\boldsymbol{x})} \right| |\hat{\sigma}_h(\boldsymbol{x}) - \tilde{\sigma}_h(\boldsymbol{x})| + \left( \frac{1}{n} \sum_{i=1}^n L \left( \frac{1 - \boldsymbol{X}'_i \boldsymbol{x}}{h^2} \right) \| \boldsymbol{B}'_x \boldsymbol{H}_x (\boldsymbol{X}_i) \|_2 \right) \times (4.142) \nonumber \\
&= O_p(h^{d-1}) O(h^{-(d-1)}) O_p(h^{-(d-1)}) O_p(n^{-\frac{1}{2}} h^{\frac{d-1}{2}}) + O_p(h^d) O_p(n^{-\frac{1}{2}} h^{-\frac{3d-1}{2}}) = O_p(n^{-\frac{1}{2}} h^{-\frac{d-1}{2}}) \tag{4.143}
\end{align}
$$

Let $\epsilon > 0$ be given. By (4.143), there exist constants $N_1 \in \mathbb{N}$ and $L_1 > 0$ such that the following holds.

$$
n \ge N_1 \Rightarrow \mathbb{P}\left( (4.143) > L_1 n^{-\frac{1}{2}} h^{-\frac{1}{2}} \right) < \frac{\epsilon}{3} \quad (4.144)
$$

Also, define events $F_n$ as the following for every $n \in \mathbb{N}$.

$$
F_n := \left\{ (4.143) \le L_1 n^{-\frac{1}{2}} h^{-\frac{1}{2}} \right\}
$$

Now we control the first term of (4.141) on the event $F_n$ for every $n \in \mathbb{N}$. To do so, define $\mathcal{H}_{h,\delta} := \{U_{h,1}(\cdot, \cdot, y) : y \in B_d(\tilde{m}_{h,1}(\boldsymbol{x}), \delta)\}$ for every $h > 0$ and $\delta > 0$. Then $\mathcal{H}_{h,\delta}$ is a class of real functions with the envelope function $H_{h,\delta}$ defined by the following for every $h > 0$ and $\delta > 0$.

$$
H_{h,\delta}(\boldsymbol{z}, w) := 2\mathrm{diam}(\Omega)\delta|\tilde{W}_{\boldsymbol{x},h}(\boldsymbol{z})|, \quad \forall \boldsymbol{z} \in \mathbb{S}^{d-1}, \forall w \in \Omega \quad (4.145)
$$

Note that the following holds by Cauchy-Schwarz inequality for every $h > 0$.

$$
\begin{align}
\mathbb{E} \left[ H_{h,\delta}^2 (\mathbf{X}, Y) \right] &= \left( \frac{2\mathrm{diam}(\Omega)\delta}{\tilde{\sigma}_h(\mathbf{x})} \right)^2 \mathbb{E} \left[ L^2 \left( \frac{1-\mathbf{X}'\mathbf{x}}{h^2} \right) (1-\tilde{\boldsymbol{\mu}}_{h,1}(\mathbf{x})' \tilde{\boldsymbol{\Sigma}}_h(\mathbf{x})^+ \mathbf{H}_x(\mathbf{X}))^2 \right] \nonumber \\
&\le 2 \left( \frac{2\mathrm{diam}(\Omega)\delta}{\tilde{\sigma}_h(\mathbf{x})} \right)^2 \left( \mathbb{E} \left[ L^2 \left( \frac{1-\mathbf{X}'\mathbf{x}}{h^2} \right) \right] + \mathbb{E} \left[ L^2 \left( \frac{1-\mathbf{X}'\mathbf{x}}{h^2} \right) (\tilde{\boldsymbol{\mu}}_{h,1}(\mathbf{x})' \tilde{\boldsymbol{\Sigma}}_h(\mathbf{x})^+ \mathbf{H}_x(\mathbf{X}))^2 \right] \right) \tag{4.146}
\end{align}
$$

Also, note that the following holds by (2.3), (4.20), (4.49), (4.59), (4.64), and Lemma 1.4 with
the conditions L1 and D1 for every $\mathbf{B}_x \in \mathcal{B}_x$.

$$
\begin{align*}
& \mathbb{E} \left[ L^2 \left( \frac{1 - \mathbf{X}'\boldsymbol{x}}{h^2} \right) (\tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \tilde{\boldsymbol{\Sigma}}_h(\boldsymbol{x})^+ \mathbf{H}_x(\mathbf{X}))^2 \right] \\
&= \mathbb{E} \left[ L^2 \left( \frac{1 - \mathbf{X}'\boldsymbol{x}}{h^2} \right) (\tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \mathbf{B}_x (\mathbf{B}_x' \tilde{\boldsymbol{\mu}}_{h,2}(\boldsymbol{x}) \mathbf{B}_x)^+ \mathbf{B}_x' \mathbf{H}_x(\mathbf{X}))^2 \right] \\
&\le \mathbb{E} \left[ L^2 \left( \frac{1 - \mathbf{X}'\boldsymbol{x}}{h^2} \right) \| \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \mathbf{B}_x \|_2^2 \|(\mathbf{B}_x' \tilde{\boldsymbol{\mu}}_{h,2}(\boldsymbol{x}) \mathbf{B}_x)^+\|_2^2 \| \mathbf{B}_x' \mathbf{H}_x(\mathbf{X})\|_2^2 \right] \\
&= \| \tilde{\boldsymbol{\mu}}_{h,1}(\boldsymbol{x})' \mathbf{B}_x \|_2^2 \|(\mathbf{B}_x' \tilde{\boldsymbol{\mu}}_{h,2}(\boldsymbol{x}) \mathbf{B}_x)^+\|_2^2 \mathbb{E} \left[ L^2 \left( \frac{1 - \mathbf{X}'\boldsymbol{x}}{h^2} \right) (\mathbf{B}_x' \mathbf{H}_x(\mathbf{X}))' (\mathbf{B}_x' \mathbf{H}_x(\mathbf{X})) \right] \\
&= O(h^{2(d+1)}) O(h^{-2(d+1)}) O(h^{d+1}) = O(h^{d+1}) \tag{4.147}
\end{align*}
$$

Combining (2.3), (4.116), (4.146), (4.147), Lemma 1.4, and Lemma 2.2 with the conditions L1 and D1, the following holds.

$$
\frac{1}{\delta^2} \mathbb{E} \left[ H_{h,\delta}^2 (\mathbf{X}, Y) \right] = O(h^{-2(d-1)}) (O(h^{d-1}) + O(h^{d+1})) = O(h^{-(d-1)}) \quad (4.148)
$$