SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \vec {w} _ {\widetilde {\mathcal {D}} _ {n}} ^ {(\lambda)} (x) - \vec {w} _ {\mathcal {D} _ {n}} ^ {(\lambda)} (x) \\ = \sqrt {n} \cdot \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {c o l}} \cdot \left(\boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top}\right) ^ {\dagger} \cdot \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {r o w}} \cdot \boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top} \cdot \tilde {c} _ {x} - \sqrt {n} \cdot \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {c o l}} \cdot \left(\boldsymbol {X} _ {\mathrm {c t r}} ^ {\top}\right) ^ {\dagger} \cdot \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {r o w}} \cdot \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top} \cdot c _ {x} \\ = \sqrt {n} \cdot \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {c o l}} \cdot \tilde {c} _ {x} - \sqrt {n} \cdot \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {c o l}} \cdot c _ {x} \quad \because \text {L e m m a 3} \\ = \sqrt {n} \cdot \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {c o l}} \cdot \left(\tilde {c} _ {x} - c _ {x}\right) + \sqrt {n} \cdot \left(\Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {c o l}} - \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {c o l}}\right) \cdot c _ {x}. \tag {60} \\ \end{array}
$$
By triangle inequality, we obtain the following upper bound:
$$
\left\| \vec {w} _ {\tilde {\mathcal {D}} _ {n}} ^ {(\lambda)} (x) - \vec {w} _ {\mathcal {D} _ {n}} ^ {(\lambda)} (x) \right\| \leq \sqrt {n} \cdot \left\| \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {c o l}} \cdot (\tilde {c} _ {x} - c _ {x}) \right\| + \sqrt {n} \cdot \left\| \left(\Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {c o l}} - \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {c o l}}\right) \cdot c _ {x} \right\|. \tag {61}
$$
Step 2: Upper bounding the norm. Next, we establish separate upper bounds for the two terms in (61).
(1) The first term in (61). First of all, we observe from the definition of  $c_x$  and  $\tilde{c}_x$ , cf. (57), that
$$
\begin{array}{l} \tilde {c} _ {x} - c _ {x} = \left(\frac {1}{\sqrt {n}} \boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top}\right) ^ {\dagger} \cdot \left(x - \mu_ {\widetilde {\mathcal {D}} _ {n}}\right) - \left(\frac {1}{\sqrt {n}} \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top}\right) ^ {\dagger} \cdot \left(x - \mu_ {\mathcal {D} _ {n}}\right) \\ = \sqrt {n} \cdot \left(\boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top} ^ {\dagger} - \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top} ^ {\dagger}\right) \cdot (x - \mu_ {\mathcal {D} _ {n}}) + \sqrt {n} \cdot \left[ \boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top} \right] ^ {\dagger} \cdot \left(\mu_ {\widetilde {\mathcal {D}} _ {n}} - \mu_ {\mathcal {D} _ {n}}\right). \\ \end{array}
$$
Then we can upper bound the first term in (61) as follows:
$$
\left\| \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {c o l}} \cdot \left(\tilde {c} _ {x} - c _ {x}\right) \right\| = \sqrt {n} \cdot \left\| \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {c o l}} \cdot \left(\boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top} {} ^ {\dagger} - \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top} {} ^ {\dagger}\right) \cdot \left(x - \mu_ {\mathcal {D} _ {n}}\right) \right\| + \sqrt {n} \cdot \left\| \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {c o l}} \cdot \left[ \boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top} \right] ^ {\dagger} \cdot \left(\mu_ {\widetilde {\mathcal {D}} _ {n}} - \mu_ {\mathcal {D} _ {n}}\right) \right\|. \tag {62}
$$
Next, we consider the orthogonal decomposition of  $x - \mu_{\mathcal{D}_n}$ :
$$
x - \mu_ {\mathcal {D} _ {n}} = \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}}} ^ {\mathrm {r o w}} (x - \mu_ {\mathcal {D} _ {n}}) + \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}}} ^ {\mathrm {r o w} \perp} \cdot (x - \mu_ {\mathcal {D} _ {n}}) = \frac {1}{\sqrt {n}} \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top} \cdot c _ {x} + \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}}} ^ {\mathrm {r o w} \perp} \cdot (x - \mu_ {\mathcal {D} _ {n}}). \tag {63}
$$
If  $x - \mu_{\mathcal{D}_n} \in \mathrm{rowsp}\left(\boldsymbol{X}_{\mathrm{ctr}}\right)$ , then we obtain the following upper bound for the first term in (62):
$$
\begin{array}{l} \sqrt {n} \cdot \left\| \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {\mathrm {c o l}}} ^ {\mathrm {c o l}} \cdot \left(\boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top} ^ {\dagger} - \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top} ^ {\dagger}\right) \cdot (x - \mu_ {\mathcal {D} _ {n}}) \right\| \\ \leq \left\| \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {c o l}} \cdot \left(\boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top} ^ {\dagger} - \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top} ^ {\dagger}\right) \cdot \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top} \cdot c _ {x} \right\| \\ + \sqrt {n} \cdot \left\| \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {c o l}} \cdot \left(\boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top} ^ {\dagger} - \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top} ^ {\dagger}\right) \cdot \underbrace {\Pi_ {\boldsymbol {X} _ {\mathrm {c t r}}} ^ {\operatorname {r o w}} ^ {\perp} \cdot \left(x - \mu_ {\mathcal {D} _ {n}}\right)} _ {= 0} \right\| \quad \because \tag {63} \\ \leq \left\| \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {\left(\lambda\right)}} ^ {\operatorname {c o l}} \cdot \left\{- \boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top} ^ {\dagger} \cdot \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}}} ^ {\operatorname {r o w}} \cdot \left(\boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top} - \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top}\right) \cdot \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}}} ^ {\operatorname {c o l}} \cdot \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top} ^ {\dagger} \right\} \cdot \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top} \cdot c _ {x} \right\| \quad \because \text {L e m m a 4} \\ \leq \left\| \left[ \boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda) ^ {\top}} \right] ^ {\dagger} \right\| \cdot \left\| \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}}} ^ {\text {r o w}} \cdot (\boldsymbol {Z} - \boldsymbol {X}) ^ {\top} \cdot \Pi_ {\mathbf {1} _ {n} ^ {\perp}} ^ {\text {r o w}} \cdot \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}}} ^ {\text {c o l}} \right\| \cdot \| c _ {x} \| \\ \leq \frac {\left\| \boldsymbol {Z} - \boldsymbol {X} \right\|}{\sigma^ {(\lambda)} \left(\boldsymbol {Z} _ {\mathrm {c t r}}\right)} \cdot \left\| c _ {x} \right\|. \\ \end{array}
$$
Similarly, the second term in (62) can be bounded by
$$
\begin{array}{l} \sqrt {n} \cdot \left\| \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {c o l}} \cdot \left[ \boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top} \right] ^ {\dagger} \cdot \left(\mu_ {\widetilde {\mathcal {D}} _ {n}} - \mu_ {\mathcal {D} _ {n}}\right) \right\| \leq \frac {1}{\sqrt {n}} \cdot \left\| \left[ \boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)} ^ {\top} \right] ^ {\dagger} \right\| \cdot \| \boldsymbol {1} _ {n} ^ {\top} \cdot (\boldsymbol {Z} - \boldsymbol {X}) \| \\ \leq \frac {1}{\sqrt {n}} \cdot \frac {\| \boldsymbol {Z} - \boldsymbol {X} \|}{\sigma^ {(\lambda)} \left(\boldsymbol {Z} _ {\mathrm {c t r}}\right)} \cdot \| \mathbf {1} _ {n} \|. \\ \end{array}
$$
All in all, we obtain
$$
\sqrt {n} \cdot \left\| \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {c o l}} \cdot \left(\tilde {c} _ {x} - c _ {x}\right) \right\| \leq \frac {\left\| \boldsymbol {Z} - \boldsymbol {X} \right\|}{\sigma^ {(\lambda)} \left(\boldsymbol {Z} _ {\mathrm {c t r}}\right)} \cdot \left(\sqrt {n} \cdot \left\| c _ {x} \right\| + \left\| \mathbf {1} _ {n} \right\|\right) \tag {64}
$$
(2) The second term in (61). Letting  $\pmb{E}^{(\lambda)} \coloneqq \pmb{Z}_{\mathrm{ctr}}^{(\lambda)} - \pmb{X}_{\mathrm{ctr}}^{(\lambda)}$ , we observe that
$$
\begin{array}{l} \left\| \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {c o l}} - \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {c o l}} \right\| \leq \max  \left\{\left\| \boldsymbol {E} ^ {(\lambda)} \cdot \boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda) ^ {\dagger}} \right\|, \left\| \boldsymbol {E} ^ {(\lambda)} \cdot \boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda) ^ {\dagger}} \right\| \right\} \quad \because \text {L e m m a} \\ \leq \left\| \boldsymbol {E} ^ {(\lambda)} \right\| \cdot \max  \left\{\left\| \boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda) ^ {\dagger}} \right\|, \left\| \boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda) ^ {\dagger}} \right\| \right\} \\ \leq \frac {\left\| Z - X \right\|}{\min  \left\{\sigma^ {(\lambda)} \left(X _ {\mathrm {c t r}}\right) , \sigma^ {(\lambda)} \left(Z _ {\mathrm {c t r}}\right) \right\}}. \\ \end{array}
$$
All in all, we obtain the following upper bound:
$$
\sqrt {n} \left\| \left(\Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {c o l}} - \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {c o l}}\right) \cdot c _ {x} \right\| \leq \left\| \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {c o l}} - \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {c o l}} \right\| \cdot \| c _ {x} \| \leq \frac {\| \boldsymbol {Z} - \boldsymbol {X} \|}{\min  \left\{\sigma^ {(\lambda)} (\boldsymbol {X} _ {\mathrm {c t r}}) , \sigma^ {(\lambda)} (\boldsymbol {Z} _ {\mathrm {c t r}}) \right\}} \cdot \sqrt {n} \cdot \| c _ {x} \| . \tag {65}
$$
Step 3: Concluding the proof. We conclude this proof by inserting the upper bounds (64) and (65) from Step 2 into the upper bound (61) in Step 1. Specifically, we obtain
$$
\begin{array}{l} \left\| \vec {w} _ {\widetilde {\mathcal {D}} _ {n}} ^ {(\lambda)} (x) - \vec {w} _ {\mathcal {D} _ {n}} ^ {(\lambda)} (x) \right\| \leq \frac {\| \boldsymbol {Z} - \boldsymbol {X} \|}{\sigma^ {(\lambda)} (\boldsymbol {Z} _ {\mathrm {c t r}})} \cdot \left(\sqrt {n} \cdot \| c _ {x} \| + \| \mathbf {1} _ {n} \|\right) + \frac {\| \boldsymbol {Z} - \boldsymbol {X} \|}{\min \left\{\sigma^ {(\lambda)} (\boldsymbol {X} _ {\mathrm {c t r}}) , \sigma^ {(\lambda)} (\boldsymbol {Z} _ {\mathrm {c t r}}) \right\}} \cdot \sqrt {n} \cdot \| c _ {x} \| \\ \leq \frac {\| \boldsymbol {Z} - \boldsymbol {X} \|}{\min  \left\{\sigma^ {(\lambda)} \left(\boldsymbol {X} _ {\mathrm {c t r}}\right) , \sigma^ {(\lambda)} \left(\boldsymbol {Z} _ {\mathrm {c t r}}\right) \right\}} \cdot \left(2 \sqrt {n} \cdot \| c _ {x} \| + \| \mathbf {1} _ {n} \|\right). \\ \end{array}
$$
Lastly, we note that  $\| c_{x}\| = \sqrt{(x - \mu_{\mathcal{D}_{n}})^{\top}\Sigma_{\mathcal{D}_{n}}^{\dagger}(x - \mu_{\mathcal{D}_{n}})} = \left\| x - \mu_{\mathcal{D}_{n}}\right\|_{\Sigma_{\mathcal{D}_{n}}}$  and  $\| \mathbf{1}_n\| = \sqrt{n}$ .
D.3 Completing the proof of Theorem 3
Recall that given a set  $\mathcal{D}_n = \{(x_i, y_i) : i \in [n]\}$ , we let  $\pmb{X}_{\mathcal{D}_n} := [x_1 \dots x_n]^\top \in \mathbb{R}^{n \times p}$ . In addition, we let
$$
\forall y \in \mathcal {M}, \vec {d} _ {\mathcal {D} _ {n}} ^ {2} (y) := \left[ \begin{array}{l l l} d ^ {2} \left(y _ {1}, y\right) & \dots & d ^ {2} \left(y _ {n}, y\right) \end{array} \right] \in \mathbb {R} ^ {n}. \tag {66}
$$
Recall that we let  $\mathbf{X} = \mathbf{X}_{\mathcal{D}_n}$  and  $Z = X_{\widetilde{\mathcal{D}}_n}$  for shorthand, and further, we let  $\mathbf{X}_{\mathrm{ctr}} = \left(\mathbf{I}_n - \frac{1}{n}\mathbf{1}_n\mathbf{1}_n^\top\right)\mathbf{X}$  and  $Z_{\mathrm{ctr}} = \left(\mathbf{I}_n - \frac{1}{n}\mathbf{1}_n\mathbf{1}_n^\top\right)\mathbf{Z}$  denote the 'row-centered' matrices. Here we present and prove the complete version of Theorem 3.
Theorem 4 (De-noising covariates). Suppose that Assumptions (C0) and (C1) hold. For any  $\lambda \in \mathbb{R}_+$ , if  $x \in \mu_{\mathcal{D}_n} + \mathrm{rowsp} X_{\mathrm{ctr}}$  and
$$
\left\| x - \mu_ {\mathcal {D} _ {n}} \right\| _ {\Sigma_ {\mathcal {D} _ {n}}} \leq \frac {1}{2} \left(\frac {C _ {g} \cdot D _ {g} ^ {\alpha}}{2 \operatorname {d i a m} (\mathcal {M})} \cdot \frac {\min  \left\{\sigma^ {(\lambda)} \left(\boldsymbol {X} _ {\mathrm {c t r}}\right) , \sigma^ {(\lambda)} \left(\boldsymbol {Z} _ {\mathrm {c t r}}\right) \right\}}{\| \boldsymbol {Z} - \boldsymbol {X} \|} - 1\right), \tag {67}
$$
$$
\begin{array}{l} d \left(\varphi_ {\tilde {\mathcal {D}} _ {n}} ^ {(\lambda)} (x), \varphi_ {\mathcal {D} _ {n}} ^ {(\lambda)} (x)\right) \\ \leq \left(\frac {\| \boldsymbol {Z} - \boldsymbol {X} \|}{\operatorname* {m i n} \left\{\sigma^ {(\lambda)} \left(\boldsymbol {X} _ {\mathrm {c t r}}\right) , \sigma^ {(\lambda)} \left(\boldsymbol {Z} _ {\mathrm {c t r}}\right) \right\}} \cdot \frac {2 \cdot \left\| x - \mu_ {\mathcal {D} _ {n}} \right\| _ {\Sigma_ {\mathcal {D} _ {n}}} + 1}{C _ {g}} \cdot \frac {\left\| \vec {d} _ {\mathcal {D} _ {n}} ^ {2} (\tilde {\varphi} _ {n}) \right\| + \left\| \vec {d} _ {\mathcal {D} _ {n}} ^ {2} (\varphi_ {n}) \right\|}{\sqrt {n}}\right) ^ {\frac {1}{\alpha}}. \tag {68} \\ \end{array}
$$
Proof of Theorem 4. First of all, we recall from (51) that
$$
\vec {w} _ {\mathcal {D} _ {n}} ^ {(\lambda)} (x) = \left[ \begin{array}{c c c} w _ {\mathcal {D} _ {n}} ^ {(\lambda)} (x _ {1}, x) & \dots & w _ {\mathcal {D} _ {n}} ^ {(\lambda)} (x _ {n}, x) \end{array} \right] \qquad \text {a n d} \qquad \vec {w} _ {\widetilde {\mathcal {D}} _ {n}} ^ {(\lambda)} (x) = \left[ \begin{array}{c c c} w _ {\widetilde {\mathcal {D}} _ {n}} ^ {(\lambda)} (z _ {1}, x) & \dots & w _ {\widetilde {\mathcal {D}} _ {n}} ^ {(\lambda)} (z _ {n}, x) \end{array} \right].
$$
In addition, recall that we let for any  $y \in \mathcal{M}$ ,
$$
\vec {d} _ {\mathcal {D} _ {n}} ^ {2} (y) = \left[ \begin{array}{c c c} d ^ {2} (y _ {1}, y) & \dots & d ^ {2} (y _ {n}, y) \end{array} \right] \in \mathbb {R} ^ {n}.
$$
Thereafter, we observe that for any  $y \in \mathcal{M}$  and any  $x \in (\mu_{\mathcal{D}_n} + \mathrm{rowsp} X_{\mathrm{ctr}})$ ,
$$
\begin{array}{l} \left| R _ {\widetilde {\mathcal {D}} _ {n}} ^ {(\lambda)} (y; x) - R _ {\mathcal {D} _ {n}} ^ {(\lambda)} (y; x) \right| = \frac {1}{n} \left| \sum_ {i = 1} ^ {n} \left(w _ {\widetilde {\mathcal {D}} _ {n}} ^ {(\lambda)} \left(z _ {i}, x\right) - w _ {\mathcal {D} _ {n}} ^ {(\lambda)} \left(x _ {i}, x\right)\right) \cdot d ^ {2} \left(y _ {i}, y\right) \right| \\ = \frac {1}{n} \left| \left\langle \vec {w} _ {\tilde {\mathcal {D}} _ {n}} ^ {(\lambda)} (x) - \vec {w} _ {\mathcal {D} _ {n}} ^ {(\lambda)} (x), \vec {d} _ {\mathcal {D} _ {n}} ^ {2} (y) \right\rangle \right| \\ \stackrel {(a)} {\leq} \frac {1}{n} \left\| \vec {w} _ {\tilde {\mathcal {D}} _ {n}} ^ {(\lambda)} (x) - \vec {w} _ {\mathcal {D} _ {n}} ^ {(\lambda)} (x) \right\| \cdot \left\| \vec {d} _ {\mathcal {D} _ {n}} ^ {2} (y) \right\| \\ \stackrel {(b)} {\leq} \frac {\left\| \boldsymbol {Z} - \boldsymbol {X} \right\|}{\min  \left\{\sigma^ {(\lambda)} \left(\boldsymbol {X} _ {\mathrm {c t r}}\right) , \sigma^ {(\lambda)} \left(\boldsymbol {Z} _ {\mathrm {c t r}}\right) \right\}} \cdot \left(2 \cdot \left\| x - \mu_ {\mathcal {D} _ {n}} \right\| _ {\Sigma_ {\mathcal {D} _ {n}}} + 1\right) \cdot \frac {\left\| \vec {d} _ {\mathcal {D} _ {n}} ^ {2} (y) \right\|}{\sqrt {n}} \tag {69} \\ \end{array}
$$
where (a) is due to Cauchy-Schwarz inequality, and (b) follows from Lemma 6.
Using shorthand notation  $R_{n} = R_{\mathcal{D}_{n}}^{(\lambda)}$ ,  $\tilde{R}_n = R_{\widetilde{\mathcal{D}}_n}^{(\lambda)}$ ,  $\varphi_n = \varphi_{\mathcal{D}_n}^{(\lambda)}(x)$ , and  $\tilde{\varphi}_n = \varphi_{\widetilde{\mathcal{D}}_n}^{(\lambda)}(x)$ , we observe that
$$
\begin{array}{l} R _ {n} (\tilde {\varphi} _ {n}) - R _ {n} (\varphi_ {n}) \\ = R _ {n} \left(\tilde {\varphi} _ {n}\right) - \tilde {R} _ {n} \left(\tilde {\varphi} _ {n}\right) + \tilde {R} _ {n} \left(\tilde {\varphi} _ {n}\right) - R _ {n} \left(\varphi_ {n}\right) \\ \stackrel {(a)} {\leq} R _ {n} (\tilde {\varphi} _ {n}) - \tilde {R} _ {n} (\tilde {\varphi} _ {n}) + \tilde {R} _ {n} (\varphi_ {n}) - R _ {n} (\varphi_ {n}) \\ \stackrel {(b)} {\leq} \frac {\left\| \boldsymbol {Z} - \boldsymbol {X} \right\|}{\min  \left\{\sigma^ {(\lambda)} \left(\boldsymbol {X} _ {\mathrm {c t r}}\right) , \sigma^ {(\lambda)} \left(\boldsymbol {Z} _ {\mathrm {c t r}}\right) \right\}} \cdot \left(2 \cdot \left\| x - \mu_ {\mathcal {D} _ {n}} \right\| _ {\Sigma_ {\mathcal {D} _ {n}}} + 1\right) \cdot \frac {\left\| \vec {d} _ {\mathcal {D} _ {n}} ^ {2} (\tilde {\varphi} _ {n}) \right\| + \left\| \vec {d} _ {\mathcal {D} _ {n}} ^ {2} (\varphi_ {n}) \right\|}{\sqrt {n}} \tag {70} \\ \end{array}
$$
where (a) follows from the optimality of  $\tilde{\varphi}_n$ , i.e.,  $\tilde{R}_n(\varphi_n) \geq \tilde{R}_n(\tilde{\varphi}_n)$ , and (b) is due to (69).
Finally, we note that if
$$
\| x - \mu_ {\mathcal {D} _ {n}} \| _ {\Sigma_ {\mathcal {D} _ {n}}} \leq \frac {1}{2} \left(\frac {C _ {\mathbf {g}} \cdot D _ {\mathbf {g}} ^ {\alpha}}{2 \operatorname {d i a m} (\mathcal {M})} \cdot \frac {\min \left\{\sigma^ {(\lambda)} (\boldsymbol {X} _ {\mathrm {c t r}}) , \sigma^ {(\lambda)} (\boldsymbol {Z} _ {\mathrm {c t r}}) \right\}}{\| \boldsymbol {Z} - \boldsymbol {X} \|} - 1\right),
$$
then the upper bound in (70) certifies that  $R_{n}(\tilde{\varphi}_{n}) - R_{n}(\varphi_{n}) < C_{\mathbf{g}} \cdot D_{\mathbf{g}}^{\alpha}$ . Thus, we can use Assumption (C1) to convert the risk bound (70) to derive a distance bound between the minimizers:
$$
d \left(\tilde {\varphi} _ {n}, \varphi_ {n}\right) \leq \left(\frac {R _ {n} (\tilde {\varphi} _ {n}) - R _ {n} (\varphi_ {n})}{C _ {\mathrm {g}}}\right) ^ {\frac {1}{\alpha}},
$$
E Further details on the experiments
Experimental setup in Section 5. We consider combinations of  $p \in \{150,300,600\}$  and  $n \in \{100,200,400\}$ . The datasets  $\mathcal{D}_n = \{(X_i,Y_i) : i \in [n]\}$  and  $\widetilde{\mathcal{D}}_n = \{(Z_i,Y_i) : i \in [n]\}$  are generated as follows.
(True covariate  $X$ ) Let  $X_{i} \sim \mathcal{N}_{p}\big(\mathbf{0}_{p}, \Sigma\big)$  be IID multivariate Gaussian with mean  $\mathbf{0}_p$  and covariance  $\overline{\Sigma}$  such that  $\operatorname{spec}(\Sigma) = \{\kappa_j > 0 : j \in [p]\}$  is an exponentially decreasing sequence such that  $\operatorname{tr}(\Sigma) = \sum_{j=1}^{p} \kappa_j = p$ . To be specific, for each  $p$ , we consider an exponentially decreasing sequence  $1 = a_1 > \dots > a_p = 10^{-3}$ , and then set  $\kappa_j = p \cdot a_j / (\sum_{j'=1}^{p} a_{j'})$  for each  $j \in [p]$ . Note that  $\sum_{j=1}^{\lfloor p/3 \rfloor} \kappa_j / \sum_{j'=1}^{p} \kappa_{j'} \approx 0.9$ , and thus,  $\Sigma$  is effectively low-rank.
(Noisy covariate  $Z$ ) For the error-prone covariate  $Z = X + \varepsilon$ , we consider two scenarios  $\varepsilon_{j} \stackrel{IID}{\sim} \mathcal{N}(0, \sigma_{\varepsilon}^{2})$  and  $\varepsilon_{j} \stackrel{IID}{\sim} \mathrm{Laplace}(0, \sigma_{\varepsilon})$ . Note that in this setting, we have the signal-to-noise ratio  $\mathbb{E}(\|X\|_2^2) / \mathbb{E}(\|\varepsilon\|_2^2) = 1 / \sigma_{\varepsilon}^2$ . We set  $\sigma_{\varepsilon}^2 = 0.05^2$ .
(Response  $Y$ ) Given  $X = x$ , let  $Y$  be the distribution function of  $\mathcal{N}\big(\mu_{\alpha,\beta}(x) + \eta, \tau^2\big)$ , where
-  $\mu_{\alpha, \beta}(x) = \alpha + \beta^\top x$  with  $\alpha = 1$  and  $\beta = p^{-1/2} \cdot \mathbf{1}_p$ ,
$\eta \sim \mathcal{N}\big(0,\sigma_{\eta}^{2}\big),$
-  $\tau^2 \sim \mathcal{IG}(s_1, s_2)$ , an inverse gamma distribution with shape  $s_1$  and scale  $s_2$ .
We note that  $\mathbb{E}(\tau^2) = \frac{s_2}{s_1 - 1}$  and  $\mathrm{Var}(\tau^2) = \frac{s_2^2}{(s_1 - 1)^2(s_1 - 2)}$ . In particular, when  $\tau^2 = 0$ , this setting corresponds to the classical linear regression model for scalar responses. We set  $\sigma_{\eta}^{2} = 0.5^{2}$ , and  $(s_1,s_2) = (18,17)$ . In this setting, we have
-  $\mathbb{E}\bigl (\mu_{\alpha ,\beta}(X)\bigr) = 1$  and  $\operatorname {Var}\bigl (\mu_{\alpha ,\beta}(X)\bigr) = \beta^{\top}\Sigma \beta \approx 1,$
-  $\mathbb{E}(\tau^2) = 1$  and  $\operatorname{Var}(\tau^2) = 0.25^2$ .
(Tuning parameter  $\lambda$ ) For simplicity, we chose a universal threshold value as
$$
\hat {\lambda} _ {n} = \underset {\lambda \in \Lambda} {\arg \min} \operatorname {M S P E} (\varphi_ {\widehat {\mathcal {D}} _ {n}} ^ {(\lambda)}),
$$
where  $\Lambda$  is a fine grid on  $(0, \sqrt{\lambda_1 \cdot p / n})$ . Then the same threshold  $\hat{\lambda}_n$  was used to evaluate  $\mathrm{Bias}^2(\varphi_{\widetilde{D}^{(b)}}^{(\lambda)}(x))$ ,  $\mathrm{Var}(\varphi_{\widetilde{D}^{(b)}}^{(\lambda)}(x))$ , and  $\mathrm{MSE}(\varphi_{\widetilde{D}^{(b)}}^{(\lambda)}(x))$  for all  $b = 1, \dots, B$ . Therefore, we claim that the performance of the SVT estimator reported in Table 1 has further room for improvement if one substitute  $\hat{\lambda}_n^{(b)} = \arg \min_{\lambda \in \Lambda} \mathrm{MSPE}(\varphi_{\nu^{(b)}}^{(\lambda)})$  for each Monte Carlo experiment. Although suboptimal results are reported, we note that the proposed SVT outperforms both the oracle estimator and the naive EIV estimator in our simulation study. In practice, one may employ cross-validation for better performance. For the MSPE in Table 1, we reported  $\min_{\lambda \in \Lambda} \mathrm{MSPE}(\varphi_{\widetilde{D}_n}^{(\lambda)})$ .
Evaluation metrics: bias and variance. We evaluate the accuracy and efficiency of the Fréchet regression function estimator using the bias and the variance. For any given  $x$ , we define
$$
\operatorname {B i a s} _ {x} \big (\varphi_ {\nu} ^ {(\lambda)} \big) := d _ {W} \Big (\overline {{\varphi}} _ {\nu} ^ {(\lambda)} (x), \varphi_ {\nu^ {*}} ^ {(0)} (x) \Big) \quad \text {a n d} \quad \operatorname {V a r} _ {x} \big (\varphi_ {\nu} ^ {(\lambda)} \big) := \frac {1}{B} \sum_ {b = 1} ^ {B} d _ {W} \Big (\varphi_ {\nu^ {(b)}} ^ {(\lambda)} (x), \overline {{\varphi}} _ {\nu} ^ {(\lambda)} (x) \Big) ^ {2},
$$
where  $\nu \in \{\mathcal{D}_n,\widetilde{\mathcal{D}}_n\}$  and  $\overline{\varphi}_{\nu}^{(\lambda)}(x)\coloneqq \arg \min_y\sum_{b = 1}^B d_W\bigl (\varphi_{\nu^{(b)}}^{(\lambda)}(x),y\bigr)^2$  is the Fréchet mean of  $\varphi_{\nu^{(1)}}^{(\lambda)}(x),\ldots ,\varphi_{\nu^{(B)}}^{(\lambda)}(x)$ . Note that these definitions are a generalization of the standard bias and variance of the regression function estimator in Euclidean spaces. We evaluate the global performance of the estimator by considering a fixed set of evaluation points,  $\mathcal{G}_M = \{x_m:m = 1,\dots ,M\}$ , and compute
$$
\operatorname {B i a s} ^ {2} \big (\varphi_ {\nu} ^ {(\lambda)} \big) := \frac {1}{M} \sum_ {m = 1} ^ {M} \operatorname {B i a s} _ {x _ {m}} ^ {2} \big (\varphi_ {\nu} ^ {(\lambda)} \big) \quad \text {a n d} \quad \operatorname {V a r} \big (\varphi_ {\nu} ^ {(\lambda)} \big) := \frac {1}{M} \sum_ {m = 1} ^ {M} \operatorname {V a r} _ {x _ {m}} \big (\varphi_ {\nu} ^ {(\lambda)} \big).
$$
In our experiment, we generate the set  $\mathcal{G}_M$  by drawing  $x_1, \ldots, x_M$  IID from the same distribution as  $X$ , with  $M = 500$ .
Additional experiment with linear regression models We also conducted an additional experiment for the standard linear regression models with three different metric metrics.
(Model) The linear regression model for  $(X,Y)\in \mathbb{R}^p\times \mathbb{R}^d$  is defined as  $Y = \alpha +X\beta +\eta$  and the covariate is contaminated as  $Z = X + \varepsilon$ . We generate  $X$  using the effective low-rank model with a geometrically decaying spectrum and condition number  $10^{3}$  (see Appendix E in the original submission).
(Parameters) Here, we let  $\alpha = \mathbf{1}_d + 0.1\cdot \pmb {g},\beta = d^{-1 / 2}\cdot \mathbf{1}_{p\times d} + 0.1\cdot \pmb {G},\eta \sim \mathcal{N}(0,0.5^2\cdot I_d)$  and  $\varepsilon \sim \mathcal{N}(0,0.5^2\cdot I_p)$ , with  $\pmb {g},\pmb{G}$  being standard Gaussians.
(a)  $L_{1}$