SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The last line follows from the fact that  $\sup_{i\in [p]}\left(\sigma_i(\widehat{\Sigma}_n) - \sigma_i(\Sigma_{\nu^*})\right)\to 0$  in probability, and thus,  $\Pi_{\mathcal{V}_n^{(\lambda)}}\rightarrow 0$  in probability.
(ii)  $T_{2} = o_{p}(1)$
Letting  $\tilde{R}_n(y;x) = \frac{1}{n}\sum_{i = 1}^n w_{\nu^*}^{(0)}(X_i,x)\cdot d^2 (Y_i,y)$ , we decompose  $T_{2}$  as follows:
$$
\begin{array}{l} T _ {2} = R _ {\mathcal {D} _ {n}} ^ {(0)} (y; x) - \tilde {R} _ {n} (y; x) + \tilde {R} _ {n} (y; x) - R _ {\nu^ {*}} ^ {(0)} (y; x) \\ = \underbrace {\frac {1}{n} \sum_ {i = 1} ^ {n} \left\{w _ {\mathcal {D} _ {n}} ^ {(0)} (X _ {i} , x) - w _ {\nu^ {*}} ^ {(0)} (X _ {i} , x) \right\} \cdot d ^ {2} (Y _ {i} , y)} _ {=: T _ {2 A}} \\ + \underbrace {\frac {1}{n} \sum_ {i = 1} ^ {n} \left\{w _ {\nu^ {*}} ^ {(0)} (X _ {i} , x) \cdot d ^ {2} (Y _ {i} , y) - \mathbb {E} \left[ w _ {\nu^ {*}} ^ {(0)} (X _ {i} , x) \cdot d ^ {2} (Y _ {i} , y) \right] \right\}} _ {=: T _ {2 B}} \\ \end{array}
$$
Note that  $T_{2B}$  converges to 0 in probability by the weak law of large numbers.
Now it remains to show  $T_{2A} = o_p(1)$ . To this end, we note that
$$
\begin{array}{l} w _ {\mathcal {D} _ {n}} ^ {(0)} (X _ {i}, x) - w _ {\nu^ {*}} ^ {(0)} (X _ {i}, x) = V _ {n} (x) + X _ {i} ^ {\top} W _ {n} (x) \\ \text {w h e r e} \quad \left\{ \begin{array}{l} V _ {n} (x) = - \widehat {\mu} _ {n} ^ {\top} \widehat {\Sigma} _ {n} ^ {\dagger} (x - \widehat {\mu} _ {n}) + \mu^ {\top} \Sigma^ {\dagger} (x - \mu), \\ W _ {n} (x) = \widehat {\Sigma} _ {n} ^ {\dagger} (x - \widehat {\mu} _ {n}) - \Sigma^ {\dagger} (x - \mu). \end{array} \right. \tag {25} \\ \end{array}
$$
Since  $\widehat{\mu}_n$  and  $\widehat{\Sigma}_n$  respectively converge to  $\mu$  and  $\Sigma$  in probability, it is possible to verify that  $|V_{n}(x)|$ ,  $\| W_{n}(x)\|$  converge to 0 in probability. As a result,  $T_{2}$  also converges to 0 in probability.
All in all, we have  $R_{\mathcal{D}_n}^{(\lambda)}(y;x) - R_{\nu^*}^{(0)}(y;x) = o_p(1)$ , and thus, proved (S1).
Step 2: proof of (S2). For any  $y_{1},y_{2}\in \mathcal{M}$
$$
\begin{array}{l} \left| R _ {\mathcal {D} _ {n}} ^ {(\lambda)} \left(y _ {1}; x\right) - R _ {\mathcal {D} _ {n}} ^ {(\lambda)} \left(y _ {2}; x\right) \right| = \left| \frac {1}{n} \sum_ {i = 1} ^ {n} w _ {\mathcal {D} _ {n}} ^ {(\lambda)} \left(X _ {i}, x\right) \cdot \left\{d ^ {2} \left(Y _ {i}, y _ {1}\right) - d ^ {2} \left(Y _ {i}, y _ {2}\right) \right\} \right| \\ \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \left| w _ {\mathcal {D} _ {n}} ^ {(\lambda)} (X _ {i}, x) \right| \cdot | d (Y _ {i}, y _ {1}) + d (Y _ {i}, y _ {2}) | \cdot | d (Y _ {i}, y _ {1}) - d (Y _ {i}, y _ {2}) | \\ \leq 2 \operatorname {diam} (\mathcal {M}) \cdot d (y _ {1}, y _ {2}) \cdot \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left| w _ {\mathcal {D} _ {n}} ^ {(\lambda)} \left(X _ {i}, x\right) \right|\right) \\ = O _ {p} \left(d \left(y _ {1}, y _ {2}\right)\right) \\ \end{array}
$$
where the  $O_p$  term is independent of  $y_1, y_2 \in \mathcal{M}$ . Therefore,
$$
\sup_{y_{1},y_{2}\in \mathcal{M}:  d(y_{1},y_{2}) <   \delta}\left|R^{(\lambda)}_{\mathcal{D}_{n}}(y_{1};x) - R^{(\lambda)}_{\mathcal{D}_{n}}(y_{2};x)\right| = O_{p}(\delta),
$$
which proves (S2).
In this section, we prove the two claims in Theorem 2. Specifically, in Section C.1, we present and prove a lemma that controls the bias in the population estimator (Lemma 1), and in Section C.2, we present and prove a lemma that controls the variance of the empirical estimator (Lemma 2).
C.1 Bias in the population estimator
We recall the definition of Mahalanobis seminorm from Definition 5:  $\| x\|_{\Sigma} \coloneqq \left(x^{\top}\Sigma^{\dagger}x\right)^{1 / 2}$ .
Lemma 1. Suppose that Assumptions (C0) and (C1) hold. If
$$
\left\| x - \mu_ {\nu^ {*}} \right\| _ {\Sigma_ {\nu^ {*}}} \leq \frac {C _ {g} \cdot D _ {g} ^ {\alpha}}{\operatorname {d i a m} (\mathcal {M}) ^ {2} \cdot \sqrt {\operatorname {r a n k} \Sigma_ {\nu^ {*}}}}, \tag {26}
$$
then for any  $\lambda \in \mathbb{R}_{+}$
$$
d \left(\varphi^ {(\lambda)} (x), \varphi (x)\right) \leq 2 ^ {K _ {0}} \cdot \mathbf {b} _ {\lambda} (x) ^ {\frac {1}{\alpha - 1}} = O \left(\mathbf {b} _ {\lambda} (x) ^ {\frac {1}{\alpha - 1}}\right) \tag {27}
$$
$$
K _ {0} = \left\lfloor \frac {1}{(\alpha - 1) \log 2} \cdot \log \left(\frac {4 \mathrm {d i a m} (\mathcal {M})}{C _ {\mathfrak {g}} \cdot (1 - 2 ^ {- (\alpha - 1)})}\right) \right\rfloor + 1 a n d
$$
$$
\boldsymbol {b} _ {\lambda} (x) = \sqrt {\operatorname {r a n k} \left(\Sigma_ {\nu^ {*}} - \Sigma_ {\nu^ {*}} ^ {(\lambda)}\right)} \cdot \| x - \mu_ {\nu^ {*}} \| _ {\Sigma_ {\nu^ {*}} - \Sigma_ {\nu^ {*}} ^ {(\lambda)}}.
$$
Proof of Lemma 1. For the sake of brevity, we write  $\varphi^{(\lambda)}(x) = \varphi_{\nu^*}^{(\lambda)}(x)$  and  $\varphi(x) = \varphi_{\nu^*}^{(0)}(x)$  throughout this proof, dropping the subscript  $\nu^*$ . Likewise, we simply write  $\mu = \mu_{\nu^*}$  and  $\Sigma = \Sigma_{\nu^*}$ .
Step 1: A naïve upper bound. Observe that for any  $\lambda \in \mathbb{R}_+$ ,  $x \in \mathbb{R}^p$ , and  $y \in \mathcal{M}$
$$
\begin{array}{l} \left| R (y; x) - R ^ {(\lambda)} (y; x) \right| \\ = \left| \mathbb {E} _ {\nu^ {*}} \left[ (X - \mu) ^ {\top} \cdot \left(\Sigma^ {\dagger} - \Sigma^ {(\lambda), \dagger}\right) \cdot (x - \mu) \cdot d ^ {2} (Y, y) \right] \right| \\ \leq \operatorname {d i a m} (\mathcal {M}) ^ {2} \cdot \mathbb {E} _ {\nu^ {*}} \left[ \| X - \mu \| _ {\Sigma - \Sigma^ {(\lambda)}} \right] \cdot \| x - \mu \| _ {\Sigma - \Sigma^ {(\lambda)}} \quad \because \text {C a u c h y - S c h w a r z i n e q u a l i t y} \\ \leq \operatorname {d i a m} (\mathcal {M}) ^ {2} \cdot \left(\mathbb {E} _ {\nu^ {*}} \| X - \mu \| _ {\Sigma - \Sigma^ {(\lambda)}} ^ {2}\right) ^ {1 / 2} \cdot \| x - \mu \| _ {\Sigma - \Sigma^ {(\lambda)}} \quad \because \text {J e s s e n ' s i n e q u a l i t y} \\ = \operatorname {d i a m} (\mathcal {M}) ^ {2} \cdot \sqrt {\operatorname {r a n k} \left(\Sigma - \Sigma^ {(\lambda)}\right)} \cdot \| x - \mu \| _ {\Sigma - \Sigma^ {(\lambda)}}, \tag {28} \\ \end{array}
$$
where the last inequality follows from  $\mathbb{E}_{\nu^{*}}\| X - \mu \|_{\Sigma -\Sigma^{(\lambda)}}^2 = \mathrm{rank}\left(\Sigma -\Sigma^{(\lambda)}\right)$
We observe that the upper bound in (28) is monotone non-decreasing with respect to  $\lambda \in \mathbb{R}_+$ , and it converges to 0 as  $\lambda \to 0$ . To see this, for any  $\lambda \in \mathbb{R}_+$ , we let
$$
\mathcal {V} ^ {(\lambda)} := \operatorname {s p a n} \left\{v _ {i}: i \in [ p ], 0 <   \lambda_ {i} \leq \lambda \right\}
$$
where  $\Sigma = \sum_{i=1}^{p} \lambda_i \cdot v_i v_i^\top$  is an eigenv decomposition of  $\Sigma$ . Letting  $\Pi_{\mathcal{V}^{(\lambda)}}$  denote the projection matrix onto the subspace  $\mathcal{V}^{(\lambda)}$ , we note that  $\Sigma - \Sigma^{(\lambda)} = \Pi_{\mathcal{V}^{(\lambda)}} \Sigma \Pi_{\mathcal{V}^{(\lambda)}}$ , and that  $(\Sigma - \Sigma^{(\lambda)})^\dagger = \Pi_{\mathcal{V}^{(\lambda)}} \Sigma^\dagger \Pi_{\mathcal{V}^{(\lambda)}}$ . Thus,  $\operatorname{rank}\left(\Sigma - \Sigma^{(\lambda)}\right) = \dim \mathcal{V}^{(\lambda)}$ , and furthermore, we notice that  $\mathcal{V}^{(\lambda)} = \{0\}$  if and only if  $\lambda < \lambda_{\min} := \min \{\lambda_i : \lambda_i > 0\}$ . Therefore,
$$
\lambda <   \lambda_ {\min } \quad \Longrightarrow \quad R ^ {(\lambda)} (y; x) - R (y; x) = 0 \quad \Longrightarrow \quad \varphi^ {(\lambda)} (x) = \varphi (x), \forall x. \tag {29}
$$
The observation (29), together with Assumption (C0), implies that  $d\big(\varphi^{(\lambda)}(x), \varphi(x)\big) = o(1)$  as  $\lambda \to 0$ .
Step 2: Controlling risk difference. Next, we move on to determine the order of  $d\big(\varphi^{(\lambda)}(x), \varphi(x)\big)$  — as a function of  $\mathsf{b}_{\lambda}(x)$  — for a fixed  $\lambda \in \mathbb{R}$ . We may assume  $\lambda > \lambda_{\min}$  for the proof because the lemma is trivial otherwise, cf. (29). Assuming  $\lambda > \lambda_{\min}$ , we may decompose the difference in the population objective at  $\varphi^{(\lambda)}(x)$  and  $\varphi(x)$  as follows:
$$
\begin{array}{l} R\bigl(\varphi^{(\lambda)}(x);x\bigr) - R\bigl(\varphi (x);x\bigr) = \underbrace{\Bigl\{R\bigl(\varphi^{(\lambda)}(x);x\bigr) - R^{(\lambda)}\bigl(\varphi^{(\lambda)}(x);x\bigr) + R^{(\lambda)}\bigl(\varphi(x);x\bigr) - R\bigl(\varphi(x);x\bigr)\Bigr\}}_{=: \mathfrak{R}_{1}} \\ - \underbrace {\left\{R ^ {(\lambda)} \big (\varphi (x) ; x \big) - R ^ {(\lambda)} \big (\varphi^ {(\lambda)} (x) ; x \big) \right\}} _ {=: \Re_ {2}}. \\ \end{array}
$$
We observe that both  $\Re_1$  and  $\Re_2$  are non-negative, due to the optimality of  $\varphi(x)$  and  $\varphi^{(\lambda)}(x)$ . Then, we obtain an upper bound for  $\Re_1$  using a similar argument as in (28). Specifically,
$$
\begin{array}{l} R \big (\varphi^ {(\lambda)} (x); x \big) - R \big (\varphi (x); x \big) \leq \Re_ {1} \\ = \mathbb {E} _ {\nu^ {*}} \left[ \left\{w _ {\nu^ {*}} ^ {(0)} (X, x) - w _ {\nu^ {*}} ^ {(\lambda)} (X, x) \right\} \cdot \left\{d ^ {2} \left(Y, \varphi^ {(\lambda)} (x)\right) - d ^ {2} \left(Y, \varphi (x)\right) \right\} \right] \\ \leq 2 \operatorname {d i a m} (\mathcal {M}) \cdot \mathfrak {b} _ {\lambda} (x) \cdot d \left(\varphi^ {(\lambda)} (x), \varphi (x)\right). \tag {30} \\ \end{array}
$$
Step 3: Converting risk difference to bias. Lastly, we convert the upper bound (30) to an upper bound on the distance  $d\big(\varphi^{(\lambda)}(x), \varphi(x)\big)$  using Assumption (C1). To this end, we begin by confirming that
$$
\begin{array}{l} R \big (\varphi^ {(\lambda)} (x); x \big) - R \big (\varphi (x); x \big) = \mathbb {E} _ {\nu^ {*}} \left[ \big (X - \mu \big) ^ {\top} \cdot \Sigma^ {\dagger} \cdot (x - \mu) \cdot \left\{d ^ {2} \big (Y, \varphi^ {(\lambda)} (x) \big) - d ^ {2} \big (Y, \varphi (x) \big) \right\} \right] \\ \leq \operatorname {d i a m} (\mathcal {M}) ^ {2} \cdot \left(\mathbb {E} _ {\nu^ {*}} \| X - \mu \| _ {\Sigma} ^ {2}\right) ^ {1 / 2} \cdot \| x - \mu \| _ {\Sigma} \\ = \operatorname {d i a m} (\mathcal {M}) ^ {2} \cdot \sqrt {\operatorname {r a n k} \Sigma} \cdot \| x - \mu \| _ {\Sigma} \\ \leq C _ {\mathbf {g}} \cdot D _ {\mathbf {g}} ^ {\alpha}. \\ \end{array}
$$
Thereafter, we choose an arbitrary  $K \in \mathbb{N}$  and  $r \in \mathbb{R}_+$  whose values will be determined later in this proof. Then we obtain the following inequality using the so-called peeling technique:
$$
\begin{array}{l} \mathbb {1} \left\{d \left(\varphi^ {(\lambda)} (x), \varphi (x)\right) > 2 ^ {K} \cdot \mathfrak {b} _ {\lambda} (x) ^ {r} \right\} \\ = \sum_ {k = K} ^ {\infty} \mathbb {1} \left\{2 ^ {k} \cdot \mathsf {b} _ {\lambda} (x) ^ {r} <   d \left(\varphi^ {(\lambda)} (x), \varphi (x)\right) \leq 2 ^ {k + 1} \cdot \mathsf {b} _ {\lambda} (x) ^ {r} \right\} \\ \leq \sum_ {k = K} ^ {\infty} \mathbb {1} \left\{2 ^ {k} \cdot \mathsf {b} _ {\lambda} (x) ^ {r} <   d \left(\varphi^ {(\lambda)} (x), \varphi (x)\right) \leq 2 ^ {k + 1} \cdot \mathsf {b} _ {\lambda} (x) ^ {r} \right\} \\ \leq \sum_ {k = K} ^ {\infty} \frac {R \left(\varphi^ {(\lambda)} (x) ; x\right) - R \left(\varphi (x) ; x\right)}{C _ {\mathbf {g}} \cdot \left(2 ^ {k} \cdot \mathsf {b} _ {\lambda} (x) ^ {r}\right) ^ {\alpha}} \cdot \mathbb {1} \left\{d \left(\varphi^ {(\lambda)} (x), \varphi (x)\right) \leq 2 ^ {k + 1} \cdot \mathsf {b} _ {\lambda} (x) ^ {r} \right\}. \quad \because (C 1) \tag {31} \\ \end{array}
$$
Moreover, we decompose the numerator in the fraction appearing in the upper bound (31) as follows: Combining (30) with (31), we have
$$
\begin{array}{l} \mathbb {1} \left\{d \left(\varphi^ {(\lambda)} (x), \varphi (x)\right) > 2 ^ {K} \cdot \mathfrak {b} _ {\lambda} (x) ^ {r} \right\} \\ \leq \sum_ {k = K} ^ {\infty} \frac {2 \operatorname {d i a m} (\mathcal {M}) \cdot \mathsf {b} _ {\lambda} (x) \cdot d \big (\varphi^ {(\lambda)} (x) , \varphi (x) \big)}{C _ {\mathbf {g}} \cdot \left(2 ^ {k} \cdot \mathsf {b} _ {\lambda} (x) ^ {r}\right) ^ {\alpha}} \cdot \mathbb {1} \left\{d \big (\varphi^ {(\lambda)} (x), \varphi (x) \big) \leq 2 ^ {k + 1} \cdot \mathsf {b} _ {\lambda} (x) ^ {r} \right\} \\ \leq \frac {4 \operatorname {d i a m} (\mathcal {M})}{C _ {\mathrm {g}}} \cdot \mathrm {b} _ {\lambda} (x) ^ {1 - r (\alpha - 1)} \sum_ {k = K} ^ {\infty} \frac {1}{2 ^ {k (\alpha - 1)}}. \tag {32} \\ \end{array}
$$
Note that  $C \coloneqq \frac{4\mathrm{diam}\mathcal{M}}{C_{\mathrm{g}}} > 0$  is a constant independent of  $\lambda$ . Let  $r = 1 / (\alpha - 1)$ , and observe that the upper bound in (32) becomes smaller than 1 for a sufficiently large  $K$ . Specifically,
$$
K \geq \left\lfloor \frac {1}{(\alpha - 1) \log 2} \cdot \log \left(\frac {4 \operatorname {d i a m} (\mathcal {M})}{C _ {\mathfrak {g}} \cdot (1 - 2 ^ {- (\alpha - 1)})}\right) \right\rfloor + 1 \quad \Longrightarrow \quad \frac {4 \operatorname {d i a m} (\mathcal {M})}{C _ {\mathfrak {g}}} \cdot \sum_ {k = K} ^ {\infty} \frac {1}{2 ^ {k (\alpha - 1)}} <   1.
$$
As a result, the inequality  $d\big(\varphi^{(\lambda)}(x), \varphi(x)\big) > 2^{K_0} \cdot \mathsf{b}_{\lambda}(x)^r$  in the indicator function must be false, and we conclude that
$$
d \left(\varphi^ {(\lambda)} (x), \varphi (x)\right) \leq 2 ^ {K _ {0}} \cdot \mathfrak {b} _ {\lambda} (x) ^ {\frac {1}{\alpha - 1}}.
$$
C.2 Variance of the empirical estimator
Lemma 2. Suppose that Assumptions (C0), (C1) and (C2) hold. For any  $\lambda \in \mathbb{R}_+$  such that  $\lambda \notin \operatorname{spec}\left(\Sigma_{\nu^*}\right)$ , it holds that
$$
d \Big (\varphi_ {\mathcal {D} _ {n}} ^ {(\lambda)} (x), \varphi_ {\nu^ {*}} ^ {(\lambda)} (x) \Big) = O _ {P} \Big (n ^ {- \frac {1}{2 (\alpha - 1)}} \Big).
$$
Proof of Lemma 2. Recall from the definition of  $\lambda$ -regularized Fréchet regression (Definition 4) and (9) that
$$
R _ {\mathcal {D} _ {n}} ^ {(\lambda)} (y; x) = \frac {1}{n} \sum_ {i = 1} ^ {n} w _ {\mathcal {D} _ {n}} ^ {(\lambda)} \left(X _ {i}, x\right) \cdot d ^ {2} \left(Y _ {i}, y\right) \quad \text {a n d} \quad R _ {\nu^ {*}} ^ {(\lambda)} (y; x) = \mathbb {E} _ {(X, Y) \sim \nu^ {*}} \left[ w _ {\nu^ {*}} ^ {(\lambda)} (X, x) \cdot d ^ {2} (Y, y) \right].
$$
Additionally, we define an auxiliary function  $\tilde{R}_n(y;x)$  as the "empirical risk with population weight" such that
$$
\tilde {R} _ {n} (y; x) := \frac {1}{n} \sum_ {i = 1} ^ {n} w _ {\nu^ {*}} ^ {(\lambda)} \left(X _ {i}, x\right) \cdot d ^ {2} \left(Y _ {i}, y\right).
$$
We present the rest of this proof in three steps, outlined as follows. In Step 1, we show the consistency of  $\varphi_{\mathcal{D}_n}^{(\lambda)}(x)$ , i.e.,  $d\bigl (\varphi_{\mathcal{D}_n}^{(\lambda)}(x),\varphi_{\nu^*}^{(\lambda)}(x)\bigr) = o_P(1)$  as  $n\to \infty$ . In Step 2, we define the discrepancy variable  $Z_{n}^{(\lambda)}(y;x)\coloneqq R_{\mathcal{D}_{n}}^{(\lambda)}(y;x) - R_{\nu^{*}}^{(\lambda)}(y;x)$  between the finite-sample and the population objectives, cf. (35), and prove a uniform upper bound for  $Z_{n}^{(\lambda)}(y;x)$  that holds in a neighborhood of  $\varphi_{\nu^{*}}^{(\lambda)}(y;x)$ . Lastly, in Step 3, we utilize the peeling technique from empirical process theory to obtain the desired rate of convergence.
Step 1: Consistency. We first claim that  $d\big(\varphi_{\mathcal{D}_n}^{(\lambda)}(x), \varphi_{\nu^*}^{(\lambda)}(x)\big) = o_P(1)$  by an argument similar to that used in the proof of Theorem 1. Specifically, it suffices to show that
(S1')  $R_{\mathcal{D}_n}^{(\lambda)}(y;x) - R_{\nu^*}^{(\lambda)}(y;x) = o_P(1)$ , and
(S2')  $R_{\mathcal{D}_n}^{(\lambda)}(\cdot ;x):\mathcal{M}\to \mathbb{R}$  is asymptotically equicontinuous in probability.
Note that we already showed the asymptotic equicontinuity in the proof of Theorem 1; see (S2). Thus, it remains to show the pointwise convergence in probability. To show (S1'), we decompose  $R_{\mathcal{D}_n}^{(\lambda)}(y;x) - R_{\nu^*}^{(\lambda)}(y;x)$  as follows.
$$
\begin{array}{l} R _ {\mathcal {D} _ {n}} ^ {(\lambda)} (y; x) - R _ {\nu^ {*}} ^ {(\lambda)} (y; x) = \left\{R _ {\mathcal {D} _ {n}} ^ {(\lambda)} (y; x) - \tilde {R} _ {n} (y; x) \right\} + \left\{\tilde {R} _ {n} (y; x) - R _ {\nu^ {*}} ^ {(\lambda)} (y; x) \right\} \\ = \underbrace {\frac {1}{n} \sum_ {i = 1} ^ {n} \left\{w _ {\mathcal {D} _ {n}} ^ {(\lambda)} (X _ {i} , x) - w _ {\nu^ {*}} ^ {(\lambda)} (X _ {i} , x) \right\} \cdot d ^ {2} (Y _ {i} , y)} _ {:= A _ {n} ^ {(\lambda)} (y; x)} \\ + \underbrace {\frac {1}{n} \sum_ {i = 1} ^ {n} \left(w _ {\nu^ {*}} ^ {(\lambda)} (X _ {i} , x) \cdot d ^ {2} (Y _ {i} , y) - \mathbb {E} _ {\nu^ {*}} \left[ w _ {\nu^ {*}} ^ {(\lambda)} (X _ {i} , x) \cdot d ^ {2} (Y _ {i} , y) \right]\right)} _ {:= B _ {n} ^ {(\lambda)} (y; x)}. \\ \end{array}
$$
Next, we show that  $A_{n}^{(\lambda)}(y;x)$  and  $B_{n}^{(\lambda)}(y;x)$  respectively converge to 0 in probability.
- Letting  $\widehat{\mu}_n = \mu_{\mathcal{D}_n}$ ,  $\widehat{\Sigma}_n = \Sigma_{\mathcal{D}_n}$ , and  $\widehat{\Sigma}_n^{(\lambda)} = \mathrm{SVT}^{(\lambda)}(\widehat{\Sigma}_n)$  for shorthand, we can write
$$
w _ {\mathcal {D} _ {n}} ^ {(\lambda)} (X _ {i}, x) - w _ {\nu^ {*}} ^ {(\lambda)} (X _ {i}, x) = V _ {n} ^ {(\lambda)} (x) + X _ {i} ^ {\top} W _ {n} ^ {(\lambda)} (x),
$$
similarly to (25), where
$$
\begin{array}{l} V _ {n} ^ {(\lambda)} (x) = - \widehat {\mu} _ {n} ^ {\top} \left[ \widehat {\Sigma} _ {n} ^ {(\lambda)} \right] ^ {\dagger} (x - \widehat {\mu} _ {n}) + \mu^ {\top} \left[ \Sigma^ {(\lambda)} \right] ^ {\dagger} (x - \mu), \tag {33} \\ W _ {n} ^ {(\lambda)} (x) = \left[ \widehat {\Sigma} _ {n} ^ {(\lambda)} \right] ^ {\dagger} (x - \widehat {\mu} _ {n}) - \left[ \Sigma^ {(\lambda)} \right] ^ {\dagger} (x - \mu). \\ \end{array}
$$
Since  $\| \widehat{\mu}_n - \mu \|_2 = O_P(n^{-1/2})$  and  $\| \widehat{\Sigma}_n^{(\lambda)} - \Sigma^{(\lambda)} \| = O_P(n^{-1/2})$  (if  $\lambda \notin \operatorname{spec} \Sigma$ ) independent of  $\lambda > 0$ , we also have  $|V_n^{(\lambda)}(x)| = O_P(n^{-1/2})$  and  $\| W_n^{(\lambda)}(x) \|_2 = O_P(n^{-1/2})$ . This implies that  $A_n^{(\lambda)}(y; x) = o_P(1)$ .
- Moreover, we note that if  $\| x - \mu \|_{\Sigma} < \infty$ , then the random variable  $w_{\nu^{*}}^{(\lambda)}(X,x)$  has finite second moment
$$
\begin{array}{l} \mathbb {E} _ {\nu^ {*}} \left[ w _ {\nu^ {*}} ^ {(\lambda)} (X, x) ^ {2} \right] \leq 2 \left(1 + \mathbb {E} _ {\nu^ {*}} \left[ \left| (X - \mu) ^ {\top} \left[ \Sigma^ {(\lambda)} \right] ^ {\dagger} (x - \mu) \right| ^ {2} \right]\right) \\ \leq 2 \left(1 + \mathbb {E} _ {\nu^ {*}} \left[ \| X - \mu \| _ {\Sigma (\lambda)} ^ {2} \cdot \| x - \mu \| _ {\Sigma (\lambda)} ^ {2} \right]\right) \tag {34} \\ \leq 2 \left\{1 + p \| x - \mu \| _ {\Sigma} ^ {2} \right\}, \\ \end{array}
$$
regardless of the value of  $\lambda > 0$ . When  $\mathrm{diam}(\mathcal{M}) < \infty$ , the product  $w_{\nu^{*}}^{(\lambda)}(X, x) \cdot d^{2}(Y, y)$  also has finite second moment. Since  $B_{n}^{(\lambda)}(y; x)$  is the sample mean of IID random variables with mean zero and finite variance, it follows that
$$
B _ {n} ^ {(\lambda)} (y; x) = O _ {P} \left(\sqrt {\frac {\operatorname {V a r} \left[ w _ {\nu^ {*}} ^ {(\lambda)} (X _ {1} , x) \cdot d ^ {2} (Y _ {1} , y) \right]}{n}}\right) = O _ {P} \big (n ^ {- 1 / 2} \big).
$$
Step 2: Uniform control of the fluctuation in objective discrepancy. For any  $\lambda \in \mathbb{R}_+$  and any  $(x,y)\in \mathbb{R}^p\times \mathcal{M}$ , we let  $Z_{n}^{(\lambda)}(y;x)$  denote the random variable defined as
$$
Z _ {n} ^ {(\lambda)} (y; x) := R _ {\mathcal {D} _ {n}} ^ {(\lambda)} (y; x) - R _ {\nu^ {*}} ^ {(\lambda)} (y; x) \tag {35}
$$
We observed that
$$
\begin{array}{l} Z _ {n} ^ {(\lambda)} (y; x) - Z _ {n} ^ {(\lambda)} \left(\varphi_ {\nu^ {*}} ^ {(\lambda)} (x); x\right) \\ = \left\{R _ {\mathcal {D} _ {n}} ^ {(\lambda)} (y; x) - R _ {\nu^ {*}} ^ {(\lambda)} (y; x) \right\} - \left\{R _ {\mathcal {D} _ {n}} ^ {(\lambda)} \left(\varphi_ {\nu^ {*}} ^ {(\lambda)} (x); x\right) - R _ {\nu^ {*}} ^ {(\lambda)} \left(\varphi_ {\nu^ {*}} ^ {(\lambda)} (x); x\right) \right\} \\ = \left[ \left\{R _ {\mathcal {D} _ {n}} ^ {(\lambda)} (y; x) - \tilde {R} _ {n} (y; x) \right\} - \left\{R _ {\mathcal {D} _ {n}} ^ {(\lambda)} \left(\varphi_ {\nu^ {*}} ^ {(\lambda)} (x); x\right) - \tilde {R} _ {n} \left(\varphi_ {\nu^ {*}} ^ {(\lambda)} (x); x\right) \right\} \right] \\ + \left[ \left\{\tilde {R} _ {n} (y; x) - R _ {\nu^ {*}} ^ {(\lambda)} (y; x) \right\} - \left\{\tilde {R} _ {n} \left(\varphi_ {\nu^ {*}} ^ {(\lambda)} (x); x\right) - R _ {\nu^ {*}} ^ {(\lambda)} \left(\varphi_ {\nu^ {*}} ^ {(\lambda)} (x); x\right) \right\} \right] \\ = \frac {1}{n} \sum_ {i = 1} ^ {n} \left\{w _ {\mathcal {D} _ {n}} ^ {(\lambda)} \left(X _ {i}, x\right) - w _ {\nu^ {*}} ^ {(\lambda)} \left(X _ {i}, x\right) \right\} \cdot \ell_ {i} ^ {(\lambda)} (y; x) \tag {36} \\ =: \mathfrak {A} _ {n} ^ {(\lambda)} (y; x) \\ + \frac {1}{n} \sum_ {i = 1} ^ {n} \left(w _ {\nu^ {*}} ^ {(\lambda)} (X _ {i}, x) \cdot \ell_ {i} ^ {(\lambda)} (y; x) - \mathbb {E} _ {\nu^ {*}} \left[ w _ {\nu^ {*}} ^ {(\lambda)} (X _ {i}, x) \cdot \ell_ {i} ^ {(\lambda)} (y; x) \right]\right) \\ =: \mathfrak {B} _ {n} ^ {(\lambda)} (y; x) \\ \end{array}
$$
where  $\ell_i^{(\lambda)}(y;x)\coloneqq d^2\big(Y_i,y\big) - d^2\big(Y_i,\varphi_{\nu^*}^{(\lambda)}(x)\big).$
Next, we analyze the asymptotic behavior of the two terms,  $\mathfrak{A}_n^{(\lambda)}(y;x)$  and  $\mathfrak{B}_n^{(\lambda)}(y;x)$ . Specifically, we establish upper bounds on their magnitudes that hold uniformly over a  $\delta$ -neighborhood of  $\varphi^{(\lambda)}(x) = \varphi_{\nu^*}^{(\lambda)}(x)$ , which will be used later in Step 3 of this proof.
- Firstly, we observe that for any  $\delta > 0$ ,
$$
\begin{array}{l} \sup_{\substack{y\in B_{d}\left(\varphi_{\nu^{*}}^{(\lambda)}(x);\delta\right)}}\left|\mathfrak{A}_{n}^{(\lambda)}(y;x)\right| \\ \leq \frac{1}{n}\sum_{i = 1}^{n}\left|w_{\mathcal{D}_{n}}^{(\lambda)}(X_{i},x) - w_{\nu^{*}}^{(\lambda)}(X_{i},x)\right|\cdot \sup_{y\in B_{d}\bigl(\varphi_{\nu^{*}}^{(\lambda)}(x);\delta \bigr)}\left|d^{2}(Y_{i},y) - d^{2}(Y_{i},\varphi_{\nu^{*}}^{(\lambda)}(x))\right| \\ \leq 2 \operatorname {diam} (\mathcal {M}) \cdot \left\{\frac {1}{n} \sum_ {i = 1} ^ {n} \left\{| V _ {n} ^ {(\lambda)} (x) | + \| X _ {i} \| _ {2} \| W _ {n} ^ {(\lambda)} (x) \| _ {2} \right\} \right\} \\ \times \sup_{y\in B_{d}}\bigl(\varphi_{\nu^{*}}^{(\lambda)}(x);\delta \bigr)  d\bigl(y,\varphi_{\nu^{*}}^{(\lambda)}(x)\bigr) \\ = O _ {P} \left(\delta \cdot n ^ {- 1 / 2}\right), \tag {37} \\ \end{array}
$$
where we used the property of  $V_{n}^{(\lambda)}(x)$  and  $W_{n}^{(\lambda)}(x)$  discussed in the paragraph following (33). Since the stochastic magnitudes of  $V_{n}^{(\lambda)}(x)$  and  $W_{n}^{(\lambda)}(x)$  are independent of  $\delta$ , (37) implies that there exists  $C_1^{(\lambda)} = C_1^{(\lambda)}(x) > 0$  such that for any  $\delta > 0$ ,
$$
\lim  _ {n \rightarrow \infty} P \left(\sup  _ {y \in \mathcal {M}} \left\{\left| \mathfrak {A} _ {n} ^ {(\lambda)} (y; x) \right|: d \left(y, \varphi_ {\nu^ {*}} ^ {(\lambda)} (x)\right) <   \delta \right\} \leq C _ {1} ^ {(\lambda)} \cdot \delta \cdot n ^ {- 1 / 2}\right) = 1. \tag {38}
$$
Furthermore, for any  $\gamma, \delta \in \mathbb{R}_+$  such that  $0 \leq \gamma < \delta$ , let  $\mathfrak{E}_n^{(\lambda)}(\gamma, \delta; x)$  be defined as an event such that
$$
\mathfrak {E} _ {n} (\gamma , \delta ; x) = \left(\sup  _ {y \in \mathcal {M}} \left\{\left| \mathfrak {A} _ {n} ^ {(\lambda)} (y; x) \right|: d \left(y, \varphi_ {\nu^ {*}} ^ {(\lambda)} (x)\right) \in [ \gamma , \delta) \right\} \leq C _ {1} ^ {(\lambda)} \cdot \delta \cdot n ^ {- 1 / 2}\right). \tag {39}
$$
For any  $\gamma \in [0, \delta]$ , we have  $\mathfrak{E}_n(0, \delta; x) \subseteq \mathfrak{E}_n(\gamma, \delta; x)$ , and thus,  $\lim \inf_{n \to \infty} P\big(\mathfrak{E}_n(\gamma, \delta; x)\big) = 1$ .
- Next, we note that
$$
\left| w _ {\nu^ {*}} ^ {(\lambda)} (X _ {i}, x) \cdot \ell_ {i} ^ {(\lambda)} (y; x) \right| \leq 2 \operatorname {d i a m} (\mathcal {M}) \cdot d \big (y, \varphi_ {\nu^ {*}} ^ {(\lambda)} (x) \big) \cdot \left| w _ {\nu^ {*}} ^ {(\lambda)} (X _ {i}, x) \right|.
$$