SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Observe that  $d\big(y, \varphi_{\nu^*}^{(\lambda)}(x)\big) \leq \text{diam}(\mathcal{M}) < \infty$  and recall that  $\mathbb{E}_{\nu^*}\left[w_{\nu^*}^{(\lambda)}(X,x)^2\right] \leq 2\big\{1 + p\| x - \mu \|_{\Sigma}^2\big\}$  as shown in Step 1 of this proof, cf. (34). It follows from the uniform entropy condition (C2), Theorem 2.7.11, and Theorem 2.14.2 in [55] that there exists  $D_{\mathsf{e}} = D_{\mathsf{e}}(x) > 0$  such that for all  $\delta \in [0, D_{\mathsf{e}})$ ,
$$
\begin{array}{l} \mathbb {E} \left[ \sup  _ {y \in \mathcal {M}} \left\{\left| \mathfrak {B} _ {n} ^ {(\lambda)} (y; x) \right|: d \left(y, \varphi_ {\nu^ {*}} ^ {(\lambda)} (x)\right) <   \delta \right\} \right] \\ \leq 2 \operatorname {diam} (\mathcal {M}) \cdot \delta \cdot n ^ {- 1 / 2} \sqrt {1 + p \| x - \mu \| _ {\Sigma} ^ {2}} \int_ {0} ^ {1} \sqrt {1 + \log \mathfrak {N} \left(B _ {d} \left(\varphi^ {(\lambda)} (x); \delta\right) , \delta \epsilon\right)} d \epsilon \\ \leq C _ {2} ^ {(\lambda)} \cdot \delta \cdot n ^ {- 1 / 2} \tag {40} \\ \end{array}
$$
where  $C_2^{(\lambda)} = 2\left(C_{\mathbf{e}} + 1\right) \cdot \mathrm{diam}(\mathcal{M}) \cdot \sqrt{1 + p\|x - \mu\|_{\Sigma}^2}$  is independent of  $\delta > 0$  and  $n \geq 1$ .
Step 3: Concluding the proof. Lastly, we combine the results from Steps 1-2 to show that, for any  $\eta > 0$ , there exist  $K = K(\eta) > 0$  and  $N = N(\eta) \geq 1$  such that  $P\Big(d\big(\varphi_{\mathcal{D}_n}^{(\lambda)}(x), \varphi_{\nu^*}^{(\lambda)}(x)\big) > 2^K n^{-\beta}\Big) < \eta$  for any  $n \geq N$ , where  $\beta > 0$  is an absolute constant that will be determined later in this proof. We prove this claim using the peeling technique, in a similar manner as we did in the proof of Lemma 1. To avoid cluttered notation, we let  $\Delta(x) = d\big(\varphi_{\mathcal{D}_n}^{(\lambda)}(x), \varphi_{\nu^*}^{(\lambda)}(x)\big)$  in the rest of this proof.
For any fixed  $K \in \mathbb{N}$  and a sufficiently large  $n = n(K) \geq 1$  satisfying  $2^K n^{-\beta} < D_* := D_{\mathfrak{g}} \wedge D_{\mathfrak{e}}$ , we observe that
$$
P \left(\Delta (x) > 2 ^ {K} n ^ {- \beta}\right) = P \left(\Delta (x) \geq D _ {*}\right) + P \left(2 ^ {K} n ^ {- \beta} \leq \Delta (x) <   D _ {*}\right) \tag {41}
$$
where we used  $P(A) \leq P(B^c) + P(A \cap B)$  to get the inequality. As we know that  $P\Big(\Delta (x) \geq D_*\Big) = o(1)$  by Step 1 of this proof, we focus on showing an upper bound for the other term,  $P\big(2^K n^{-\beta} \leq \Delta (x) < D_*\big)$ .
Step 3-A: Decomposition of  $P\big(2^K n^{-\beta} \leq \Delta(x) < D_*\big)$ . For each  $n, k \in \mathbb{N}$ , we define
$$
\mathfrak {F} _ {n, k} = \bigcap_ {k ^ {\prime} = K} ^ {k} \mathfrak {E} _ {n} ^ {(\lambda)} \left(2 ^ {k ^ {\prime}} n ^ {- \beta}, 2 ^ {k ^ {\prime} + 1} n ^ {- \beta} \wedge D _ {*}; x\right), \tag {42}
$$
$$
\mathfrak {G} _ {n, k} = \left(\bigcap_ {k ^ {\prime} = K} ^ {k - 1} \mathfrak {E} _ {n} ^ {(\lambda)} \left(2 ^ {k ^ {\prime}} n ^ {- \beta}, 2 ^ {k ^ {\prime} + 1} n ^ {- \beta} \wedge D _ {*}; x\right)\right) \cap \mathfrak {E} _ {n} ^ {(\lambda)} \left(2 ^ {k} n ^ {- \beta}, 2 ^ {k + 1} n ^ {- \beta} \wedge D _ {*}; x\right) ^ {c},
$$
where we set  $\mathfrak{F}_{n,K - 1}$  to be the entire event space so that  $\mathfrak{G}_{n,K} = \left(\mathfrak{F}_{n,K}\right)^c$ . It is worth mentioning that  $\mathfrak{G}_{n,k}$  and  $\mathfrak{G}_{n,k'}$  are mutually exclusive for any  $k\neq k^{\prime}\geq K$ , and we will use this property when concluding the proof in Step 3-C below.
Now, we observe that
$$
\begin{array}{l} P \left(2 ^ {K} n ^ {- \beta} \leq \Delta (x) <   D _ {*}\right) \\ \leq P \left(\mathfrak {E} _ {n} ^ {(\lambda)} \left(2 ^ {K} n ^ {- \beta}, 2 ^ {K + 1} n ^ {- \beta} \wedge D _ {*}; x\right) ^ {c}\right) \\ + P \left(\left(2 ^ {K} n ^ {- \beta} \leq \Delta (x) <   D _ {*}\right) \cap \mathfrak {E} _ {n} ^ {(\lambda)} \left(2 ^ {k ^ {\prime}} n ^ {- \beta}, 2 ^ {k ^ {\prime} + 1} n ^ {- \beta} \wedge D _ {*}; x\right)\right) \\ = P \left(\mathfrak {G} _ {n, K}\right) + P \left(\left(2 ^ {K} n ^ {- \beta} \leq \Delta (x) <   D _ {*}\right) \cap \mathfrak {F} _ {n, K}\right) \\ = P \left(\mathfrak {G} _ {n, K}\right) + P \left(\left(2 ^ {K} n ^ {- \beta} \leq \Delta (x) <   2 ^ {K + 1} n ^ {- \beta} \wedge D _ {*}\right) \cap \mathfrak {F} _ {n, K}\right) \\ + P \left(\left(2 ^ {K + 1} n ^ {- \beta} \leq \Delta (x) <   D _ {*}\right) \cap \mathfrak {F} _ {n, K}\right) \\ \end{array}
$$
and that for every  $k \geq K$ ,
$$
P \left(\left(2 ^ {k + 1} n ^ {- \beta} \leq \Delta (x) <   D _ {*}\right) \cap \mathfrak {F} _ {n, k}\right) \leq P \left(\left(2 ^ {k + 1} n ^ {- \beta} \leq \Delta (x) <   D _ {*}\right) \cap \mathfrak {F} _ {n, k + 1}\right) + P \left(\mathfrak {G} _ {n, k + 1}\right).
$$
$$
P \left(2 ^ {K} n ^ {- \beta} \leq \Delta (x) <   D _ {*}\right) = \sum_ {k = K} ^ {\infty} P \left(\mathfrak {G} _ {n, k}\right) + \sum_ {k = K} ^ {\infty} \underbrace {P \left(\left(2 ^ {k} n ^ {- \beta} \leq \Delta (x) <   2 ^ {k + 1} n ^ {- \beta} \wedge D _ {*}\right) \cap \mathfrak {F} _ {n , k}\right)} _ {=: \mathfrak {C} _ {n, k}}. \tag {43}
$$
Step 3-B: Controlling  $\mathfrak{C}_{n,k}$ . Next, we show an upper bound for  $\mathfrak{C}_{n,k}$ . Suppose that  $2^{k}n^{-\beta} \leq \Delta(x) < 2^{k+1}n^{-\beta} \wedge D_{*}$  and the event  $\mathfrak{F}_{n,k}$  occurs. Then it follows from Assumption (C1) that
$$
\begin{array}{l} C _ {\mathbf {g}} \cdot \Delta (x) ^ {\alpha} \\ \leq R _ {\nu^ {*}} ^ {(\lambda)} \left(\varphi_ {\mathcal {D} _ {n}} ^ {(\lambda)} (x); x\right) - R _ {\nu^ {*}} ^ {(\lambda)} \left(\varphi_ {\nu^ {*}} ^ {(\lambda)} (x); x\right) \\ \leq \left\{R _ {\nu^ {*}} ^ {(\lambda)} \big (\varphi_ {\mathcal {D} _ {n}} ^ {(\lambda)} (x); x \big) - R _ {\nu^ {*}} ^ {(\lambda)} \big (\varphi_ {\nu^ {*}} ^ {(\lambda)} (x); x \big) \right\} + \underbrace {\left\{R _ {\mathcal {D} _ {n}} ^ {(\lambda)} \big (\varphi_ {\nu^ {*}} ^ {(\lambda)} (x) ; x \big) - R _ {\mathcal {D} _ {n}} ^ {(\lambda)} \big (\varphi_ {\mathcal {D} _ {n}} ^ {(\lambda)} (x) ; x \big) \right\}} _ {\geq 0} \\ = Z _ {n} ^ {(\lambda)} \left(\varphi_ {\nu^ {*}} ^ {(\lambda)}; x\right) - Z _ {n} ^ {(\lambda)} \left(\varphi_ {\mathcal {D} _ {n}} ^ {(\lambda)} (x); x\right) (35) \\ \leq \left| \mathfrak {A} _ {n} ^ {(\lambda)} \left(\varphi_ {\nu^ {*}} ^ {(\lambda)} (x); x\right) \right| + \left| \mathfrak {B} _ {n} ^ {(\lambda)} \left(\varphi_ {\nu^ {*}} ^ {(\lambda)} (x); x\right) \right| (36) \\ \leq \sup  _ {y \in \mathcal {M}} \left\{\left| \mathfrak {A} _ {n} ^ {(\lambda)} (y; x) \right| + \left| \mathfrak {B} _ {n} ^ {(\lambda)} (y; x) \right|: 2 ^ {k} n ^ {- \beta} \leq d \big (y, \varphi_ {\nu^ {*}} ^ {(\lambda)} (x) \big) <   2 ^ {k + 1} n ^ {- \beta} \wedge D _ {*} \big) \right\} \\ \leq C _ {1} ^ {(\lambda)} \cdot \left(2 ^ {k + 1} n ^ {- \beta} \wedge D _ {*}\right) \cdot n ^ {- 1 / 2} + \sup  _ {y \in \mathcal {M}} \left\{\left| \mathfrak {B} _ {n} ^ {(\lambda)} (y; x) \right|: d \left(y, \varphi_ {\nu^ {*}} ^ {(\lambda)} (x)\right) <   2 ^ {k + 1} n ^ {- \beta} \wedge D _ {*} \right\}. \quad \because (39) \\ \end{array}
$$
Therefore, we obtain that for each  $k \geq K$ ,
$$
\begin{array}{l} \mathfrak {C} _ {n, k} = P \left(\left(2 ^ {k} n ^ {- \beta} \leq \Delta (x) <   2 ^ {k + 1} n ^ {- \beta} \wedge D _ {*}\right) \cap \mathfrak {F} _ {n, k}\right) \\ \leq P \left(\left(\Delta (x) ^ {\alpha} \geq \left(2 ^ {k} n ^ {- \beta}\right) ^ {\alpha}\right) \cap \mathfrak {F} _ {n, k}\right) \\ \leq \frac {C _ {1} ^ {(\lambda)} \cdot \left(2 ^ {k + 1} n ^ {- \beta} \wedge D _ {*}\right) \cdot n ^ {- 1 / 2} + \mathbb {E} \left[ \sup  _ {y \in \mathcal {M}} \left\{\left| \mathfrak {B} _ {n} ^ {(\lambda)} (y ; x) \right| : d \big (y , \varphi_ {\nu^ {*}} ^ {(\lambda)} (x) \big) <   2 ^ {k + 1} n ^ {- \beta} \wedge D _ {*} \right\} \right]}{C _ {\mathrm {g}} \cdot \left(2 ^ {k} n ^ {- \beta}\right) ^ {\alpha}} \\ \because (4 4) \& \text {M a r k o v} \text {s} \text {i n e q u a l i t y} \\ \leq \frac {\left(C _ {1} ^ {(\lambda)} + C _ {2} ^ {(\lambda)}\right) \cdot \left(2 ^ {k + 1} n ^ {- \beta} \wedge D _ {*}\right) \cdot n ^ {- 1 / 2}}{C _ {\mathrm {g}} \cdot \left(2 ^ {k} n ^ {- \beta}\right) ^ {\alpha}} \quad \because (4 0) \tag {45} \\ \end{array}
$$
Step 3-C: Concluding Step 3. Combining (41), (43), and (45), we have
$$
\begin{array}{l} P \left(\Delta (x) > 2 ^ {K} n ^ {- \beta}\right) \leq \frac {2 \left(C _ {1} ^ {(\lambda)} + C _ {2} ^ {(\lambda)}\right)}{C _ {\mathbf {g}}} n ^ {- \frac {1}{2} + \beta (\alpha - 1)} \sum_ {k = K} ^ {\infty} 2 ^ {- k (\alpha - 1)} \\ + \underbrace {P \left(\Delta (x) \geq D _ {*}\right)} _ {= o (1) :   \because \text {S t e p 1 o f t h i s p r o o f}} + \sum_ {k = K} ^ {\infty} P \left(\mathfrak {G} _ {n, k}\right). \\ \end{array}
$$
Moreover,  $\mathfrak{G}_{n,k}$  are mutually exclusive, and thus,
$$
\sum_ {k = K} ^ {\infty} P \left(\mathfrak {G} _ {n, k}\right) = P \left(\bigcup_ {k = K} ^ {\infty} \mathfrak {G} _ {n, k}\right) = P \left(\left(\bigcup_ {k = K} ^ {\infty} \mathfrak {G} _ {n} ^ {(\lambda)} \left(2 ^ {k} n ^ {- \beta}, 2 ^ {k + 1} n ^ {- \beta} \wedge D _ {*}; x\right)\right) ^ {c}\right)\rightarrow 0 \quad \because \tag {39}
$$
Finally, we obtain the desired result by letting  $\beta = \frac{1}{2(\alpha - 1)}$ .
D Proof of Theorem 3
In this section, we prove Theorem 4 that establishes an upper bound on  $d\big(\varphi_{\mathcal{D}_n}^{(\lambda)}(x),\varphi_{\mathcal{D}_n}^{(\lambda)}(x)\big)$ . This section is organized as follows. Firstly, in Section D.1, we present several useful results from matrix perturbation theory as lemmas. Next, in Section D.2, we provide a key lemma (Lemma 6) that establishes the stability of the weight function when there is covariate noise. Lastly, in Section D.3, we state and prove Theorem 4, from which Theorem 3 can be easily derived.
Definition 7. Let  $n, p \in \mathbb{N}$  and let  $M \in \mathbb{R}^{n \times p}$ . The row projection matrix for  $M$ , denoted by  $\Pi_{M}^{\mathrm{row}} \in \mathbb{R}^{p \times p}$ , is a matrix such that
$$
\Pi_ {M} ^ {\text {r o w}} := M ^ {\dagger} \cdot M. \tag {46}
$$
and the column projection matrix for  $M$ , denoted by  $\Pi_M^{\mathrm{col}} \in \mathbb{R}^{n \times n}$ , is a matrix such that
$$
\Pi_ {M} ^ {\mathrm {c o l}} := \boldsymbol {M} \cdot \boldsymbol {M} ^ {\dagger}. \tag {47}
$$
We recall from (6) that for any  $\lambda \in \mathbb{R}_+$ , the singular value thresholding (SVT) operator  $\mathrm{SVT}^{(\lambda)}$  is defined such that
$$
\boldsymbol {M} = \sum_ {i = 1} ^ {\min  \{n, p \}} s _ {i} \cdot u _ {i} v _ {i} ^ {\top} \text {i s a S V D} \quad \mapsto \quad \operatorname {S V T} ^ {(\lambda)} (\boldsymbol {M}) = \sum_ {i = 1} ^ {\min  \{n, p \}} s _ {i} \cdot \mathbb {1} \left\{s _ {i} > \lambda \right\} \cdot u _ {i} v _ {i} ^ {\top}.
$$
In the rest of this section, we let  $M^{(\lambda)}\coloneqq \mathrm{SVT}^{(\lambda)}(M)$  for shorthand.
Lemma 3 (Properties of the row/column projection matrices). Let  $n, p \in \mathbb{N}$ , and  $M \in \mathbb{R}^{n \times p}$ . For any  $\lambda \in \mathbb{R}_+$ , the following statements are true.
1.  $\Pi_{M^{(\lambda)}}^{\mathrm{row}}$  defines a projection in  $\mathbb{R}^p$  and  $\mathrm{rank}\Pi_{M^{(\lambda)}}^{\mathrm{row}} = \mathrm{rank}M^{(\lambda)}$ .
2.  $\Pi_{M^{(\lambda)}}^{\mathrm{col}}$  defines a projection in  $\mathbb{R}^n$  and  $\mathrm{rank}\Pi_{M^{(\lambda)}}^{\mathrm{col}} = \mathrm{rank}M^{(\lambda)}$ .
3.  $M\Pi_{M^{(\lambda)}}^{\mathrm{row}}M^{\dagger} = \Pi_{M^{(\lambda)}}^{\mathrm{col}}$  and  $M^{\dagger}\Pi_{M^{(\lambda)}}^{\mathrm{col}}M = \Pi_{M^{(\lambda)}}^{\mathrm{row}}$ .
Proof. Let  $r = \operatorname{rank} M$  and consider a compact singular value decomposition (SVD) of  $M$ :
$$
\boldsymbol {M} = \sum_ {i = 1} ^ {r} s _ {i} \cdot u _ {i} v _ {i} ^ {\top}
$$
where  $s_1, \ldots, s_r$  are non-zero singular values of  $M$ . Noticing that
$$
\boldsymbol {M} ^ {(\lambda)} = \operatorname {S V T} ^ {(\lambda)} (\boldsymbol {M}) = \sum_ {i = 1} ^ {r} \mathbb {1} \left\{s _ {i} > \lambda \right\} \cdot u _ {i} v _ {i} ^ {\top}
$$
and that  $M^{\dagger} = \sum_{i=1}^{r} s_i^{-1} \cdot v_i u_i^\top$ , the three conclusions of the lemma follow straightforwardly from the orthonormality of singular vectors.
-  $\Pi_{M^{(\lambda)}}^{\mathrm{row}} = \sum_{i=1}^{r} v_i v_i^\top \cdot \mathbb{1}\{s_i > \lambda\}$  is the projection onto the row space of  $M^{(\lambda)}$ .
-  $\Pi_{M^{(\lambda)}}^{\mathrm{col}} = \sum_{i=1}^{r} u_i u_i^\top \cdot \mathbb{1}\{s_i > \lambda\}$  is the projection onto the column space of  $M^{(\lambda)}$ .
- Due to the orthonormality of singular vectors,
$$
\begin{array}{l} \boldsymbol {M} \Pi_ {\boldsymbol {M} ^ {(\lambda)}} ^ {\text {r o w}} \boldsymbol {M} ^ {\dagger} = \left(\sum_ {i = 1} ^ {r} s _ {i} \cdot u _ {i} v _ {i} ^ {\top}\right) \left(\sum_ {i = 1} ^ {r} v _ {i} v _ {i} ^ {\top} \cdot \mathbb {1} \{s _ {i} > \lambda \}\right) \left(\sum_ {i = 1} ^ {r} s _ {i} ^ {- 1} \cdot v _ {i} u _ {i} ^ {\top}\right) \\ = \sum_ {i = 1} ^ {r} u _ {i} u _ {i} ^ {\top} \cdot \mathbb {1} \left\{s _ {i} > \lambda \right\} \\ = \Pi_ {M ^ {(\lambda)}} ^ {\mathrm {c o l}}, \\ \end{array}
$$
and likewise,  $M^{\dagger}\Pi_{M^{(\lambda)}}^{\mathrm{col}}M = \Pi_{M^{(\lambda)}}^{\mathrm{row}}$ .
In addition, we collect two classical results from matrix perturbation theory and state them as lemmas.
Lemma 4 ([50, Theorem 3.2]). Let  $X, Z \in \mathbb{R}^{n \times p}$ . Then the following equation is true:
$$
Z ^ {\dagger} - X ^ {\dagger} = - Z ^ {\dagger} \Pi_ {Z} ^ {\operatorname {c o l}} (Z - X) \Pi_ {X} ^ {\operatorname {r o w}} X ^ {\dagger} + Z ^ {\dagger} \Pi_ {Z} ^ {\operatorname {c o l}} \Pi_ {X} ^ {\operatorname {c o l} ^ {\perp}} - \Pi_ {Z} ^ {\operatorname {r o w} \perp} \Pi_ {X} ^ {\operatorname {r o w}} X ^ {\dagger} \tag {48}
$$
where  $\Pi_{\pmb{X}}^{\mathrm{col}^{\perp}} = \pmb{I}_n - \Pi_{\pmb{X}}^{\mathrm{col}}$  and  $\Pi_{\pmb{Z}}^{\mathrm{row}^{\perp}} = \pmb{I}_p - \Pi_{\pmb{Z}}^{\mathrm{row}}$ .
Lemma 5 ([15, Theorems 2.4 & 2.5]). Let  $X, Z \in \mathbb{R}^{n \times p}$ . Then
$$
\left\| \Pi_ {\boldsymbol {Z}} ^ {\operatorname {c o l}} - \Pi_ {\boldsymbol {X}} ^ {\operatorname {c o l}} \right\| \leq \max  \left\{\left\| (\boldsymbol {Z} - \boldsymbol {X}) \boldsymbol {X} ^ {\dagger} \right\|, \left\| (\boldsymbol {Z} - \boldsymbol {X}) \boldsymbol {Z} ^ {\dagger} \right\| \right\}. \tag {49}
$$
Moreover, if  $\operatorname{rank} X = \operatorname{rank} Z$ , then
$$
\left\| \Pi_ {\boldsymbol {Z}} ^ {\operatorname {c o l}} - \Pi_ {\boldsymbol {X}} ^ {\operatorname {c o l}} \right\| \leq \min  \left\{\left\| (\boldsymbol {Z} - \boldsymbol {X}) \boldsymbol {X} ^ {\dagger} \right\|, \left\| (\boldsymbol {Z} - \boldsymbol {X}) \boldsymbol {Z} ^ {\dagger} \right\| \right\}. \tag {50}
$$
D.2 Stability of the weights under (small) perturbation in covariates
Let  $\mathcal{D}_n = \{(x_i, y_i) \in \mathbb{R}^p \times \mathcal{M} : i \in [n]\}$  and  $\widetilde{\mathcal{D}}_n = \{(z_i, y_i) \in \mathbb{R}^p \times \mathcal{M} : i \in [n]\}$  be two sets in  $\mathbb{R}^p \times \mathcal{M}$ . We may identify these sets with their empirical distributions. Recall the definition of  $w_\nu^{(\lambda)}$  from (9): for any probability measure  $\nu$  on  $\mathbb{R}^p \times \mathcal{M}$ , any  $\lambda \in \mathbb{R}_+$ , and any  $x, x' \in \mathbb{R}^p$ ,
$$
w _ {\nu} ^ {(\lambda)} (x ^ {\prime}, x) = 1 + (x ^ {\prime} - \mu_ {\nu}) ^ {\top} \left[ \mathsf {S V T} ^ {(\lambda)} \big (\Sigma_ {\nu} \big) \right] ^ {\dagger} (x - \mu_ {\nu})
$$
where  $\mu_{\nu} = \mathbb{E}_{(X,Y)\sim \nu}(X)$  and  $\Sigma_{\nu} = \mathrm{Var}_{(X,Y)\sim \nu}(X)$ , cf. (7). We define the weight vectors induced by  $\mathcal{D}_n$  and  $\widetilde{\mathcal{D}}_n$  as follows: for any  $\lambda \in \mathbb{R}_+$  and any  $x\in \mathbb{R}^p$ ,
$$
\vec {w} _ {\mathcal {D} _ {n}} ^ {(\lambda)} (x) := \left[ \begin{array}{c c c} w _ {\mathcal {D} _ {n}} ^ {(\lambda)} (x _ {1}, x) & \dots & w _ {\mathcal {D} _ {n}} ^ {(\lambda)} (x _ {n}, x) \end{array} \right] \in \mathbb {R} ^ {n},
$$
$$
\vec {w} _ {\tilde {\mathcal {D}} _ {n}} ^ {(\lambda)} (x) := \left[ \begin{array}{l l l} w _ {\tilde {\mathcal {D}} _ {n}} ^ {(\lambda)} \left(z _ {1}, x\right) & \dots & w _ {\tilde {\mathcal {D}} _ {n}} ^ {(\lambda)} \left(z _ {n}, x\right) \end{array} \right] \in \mathbb {R} ^ {n}. \tag {51}
$$
Lemma 6 (Stability of weights). Let  $\mathcal{D}_n = \{(x_i, y_i) \in \mathbb{R}^p \times \mathcal{M} : i \in [n]\}$  and  $\widetilde{\mathcal{D}}_n = \{(z_i, y_i) \in \mathbb{R}^p \times \mathcal{M} : i \in [n]\}$ . Let  $\mathbf{X} = [x_1 \dots x_n]^\top \in \mathbb{R}^{n \times p}$  and  $\mathbf{Z} = [z_1 \dots z_n]^\top \in \mathbb{R}^{n \times p}$ . For any  $\lambda \in \mathbb{R}_+$ , if  $x \in \mathbb{R}^p$  satisfies  $x - \mu_{\mathcal{D}_n} \in \mathrm{rowsp}\left(\mathbf{X}_{\mathrm{ctr}}\right)$ , then
$$
\left\| \vec {w} _ {\widetilde {\mathcal {D}} _ {n}} ^ {(\lambda)} (x) - \vec {w} _ {\mathcal {D} _ {n}} ^ {(\lambda)} (x) \right\| \leq \frac {\sqrt {n} \cdot \| Z - X \|}{\operatorname* {m i n} \left\{\sigma^ {(\lambda)} \left(X _ {\mathrm {c t r}}\right) , \sigma^ {(\lambda)} \left(Z _ {\mathrm {c t r}}\right) \right\}} \cdot \left(2 \cdot \| x - \mu_ {\mathcal {D} _ {n}} \| _ {\Sigma_ {\mathcal {D} _ {n}}} + 1\right) \tag {52}
$$
where  $\mathbf{X}_{\mathrm{ctr}} = \left(\mathbf{I}_n - \frac{1}{n}\mathbf{1}_n\mathbf{1}_n^\top\right)\mathbf{X}$  and  $\sigma^{(\lambda)}(\mathbf{X})\coloneqq \inf \{\sigma_i(\mathbf{X}) > \lambda :i\in \mathbb{N}\}$  (likewise for  $\mathbf{Z}$ ).
Proof of Lemma 6. This proof consists of three steps. In Step 1, we express the weight discrepancy  $\vec{w}_{\widetilde{\mathcal{D}}_n}^{(\lambda)}(x) - \vec{w}_{\mathcal{D}_n}^{(\lambda)}(x)$  as a sum of matrix products using projections. In Step 2, we establish upper bounds on the norm of the expression obtained in Step 1. In Step 3, we collect intermediate results together and conclude the proof.
Step 1: Decomposition of the weight discrepancy. First of all, we rewrite  $\vec{w}_{\bar{\mathcal{D}}_n}^{(\lambda)}(x) - \vec{w}_{\mathcal{D}_n}^{(\lambda)}(x)$  in a compact matrix representation that is presented in (60) at the end of this step. To this end, we begin by observing that
$$
\mu_ {\mathcal {D} _ {n}} = \frac {1}{n} \boldsymbol {X} ^ {\top} \mathbf {1} _ {n}, \quad \text {a n d} \quad \Sigma_ {\mathcal {D} _ {n}} = \frac {1}{n} \left(\boldsymbol {X} - \mathbf {1} _ {n} \mu_ {\mathcal {D} _ {n}} ^ {\top}\right) ^ {\top} \left(\boldsymbol {X} - \mathbf {1} _ {n} \mu_ {\mathcal {D} _ {n}} ^ {\top}\right) = \frac {1}{n} \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top} \boldsymbol {X} _ {\mathrm {c t r}}. \tag {53}
$$
For given  $\lambda \in \mathbb{R}_+$ , we let  $X_{\mathrm{ctr}}^{(\lambda)} \coloneqq \operatorname{SVT}^{(\lambda)}(X_{\mathrm{ctr}})$ , and observe that
$$
\Sigma_ {\mathcal {D} _ {n}} ^ {(\lambda)} = \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\text {r o w}} \cdot \left(\frac {1}{n} \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top} \boldsymbol {X} _ {\mathrm {c t r}}\right) \cdot \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\text {r o w}} = \frac {1}{n} \cdot \boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda) ^ {\top}} \cdot \boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}. \tag {54}
$$
Then it follows that
$$
\left[ \Sigma_ {\mathcal {D} _ {n}} ^ {(\lambda)} \right] ^ {\dagger} = n \cdot \left[ \boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda) ^ {\top}} \cdot \boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)} \right] ^ {\dagger} = n \cdot \left[ \boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)} \right] ^ {\dagger} \cdot \left[ \boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda) ^ {\top}} \right] ^ {\dagger} = n \cdot \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {r o w}} \cdot \boldsymbol {X} _ {\mathrm {c t r}} ^ {\dagger} \cdot \left(\boldsymbol {X} _ {\mathrm {c t r}} ^ {\top}\right) ^ {\dagger} \cdot \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {r o w}}.
$$
$$
\begin{array}{l} \vec {w} _ {\mathcal {D} _ {n}} ^ {(\lambda)} (x) = \mathbf {1} _ {n} + \left(\boldsymbol {X} - \mathbf {1} _ {n} \mu_ {\mathcal {D} _ {n}} ^ {\top}\right) \cdot \left[ \Sigma_ {\mathcal {D} _ {n}} ^ {(\lambda)} \right] ^ {\dagger} \cdot (x - \mu_ {\mathcal {D} _ {n}}) \\ = \mathbf {1} _ {n} + n \cdot \boldsymbol {X} _ {\mathrm {c t r}} \cdot \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {r o w}} \cdot \boldsymbol {X} _ {\mathrm {c t r}} ^ {\dagger} \cdot \left(\boldsymbol {X} _ {\mathrm {c t r}} ^ {\top}\right) ^ {\dagger} \cdot \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {r o w}} \cdot (x - \mu_ {\mathcal {D} _ {n}}) \\ = \mathbf {1} _ {n} + n \cdot \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {c o l}} \cdot \left(\boldsymbol {X} _ {\mathrm {c t r}} ^ {\top}\right) ^ {\dagger} \cdot \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {r o w}} \cdot (x - \mu_ {\mathcal {D} _ {n}}), \tag {55} \\ \end{array}
$$
where the equality in the last line follows from Lemma 3:  $X_{\mathrm{ctr}} \Pi_{X_{\mathrm{ctr}}^{(\lambda)}}^{\mathrm{row}} X_{\mathrm{ctr}}^{\dagger} = \Pi_{X_{\mathrm{ctr}}^{(\lambda)}}^{\mathrm{col}}$ .
Likewise, we repeat the above for  $\widetilde{\mathcal{D}}_n$  and  $Z$  to write
$$
\mu_ {\widetilde {\mathcal {D}} _ {n}} = \frac {1}{n} \boldsymbol {Z} ^ {\top} \mathbf {1} _ {n} \qquad \text {a n d} \qquad \Sigma_ {\widetilde {\mathcal {D}} _ {n}} = \frac {1}{n} \boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top} \boldsymbol {Z} _ {\mathrm {c t r}}.
$$
Then, we obtain an expression for  $\vec{w}_{\widetilde{\mathcal{D}}_n}^{(\lambda)}(x)$  in a similar form to (55), namely,
$$
\vec {w} _ {\widetilde {\mathcal {D}} _ {n}} ^ {(\lambda)} (x) = \mathbf {1} _ {n} + n \cdot \Pi_ {\mathbf {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {c o l}} \cdot \left(\mathbf {Z} _ {\mathrm {c t r}} ^ {\top}\right) ^ {\dagger} \cdot \Pi_ {\mathbf {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {r o w}} \cdot (x - \mu_ {\widetilde {\mathcal {D}} _ {n}}). \tag {56}
$$
Thereafter, we define  $c_{x},\tilde{c}_{x}\in \mathbb{R}^{n\times 1}$  so that
$$
\begin{array}{l} c _ {x} = \| x - \mu_ {\mathcal {D} _ {n}} \| _ {\Sigma_ {\mathcal {D} _ {n}}} = \left(\frac {1}{\sqrt {n}} \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top}\right) ^ {\dagger} \cdot (x - \mu_ {\mathcal {D} _ {n}}) \quad \text {a n d} \tag {57} \\ \tilde {c} _ {x} = \left\| x - \mu_ {\mathcal {D} _ {n}} \right\| _ {\Sigma_ {\tilde {\mathcal {D}} _ {n}}} = \left(\frac {1}{\sqrt {n}} \boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top}\right) ^ {\dagger} \cdot \left(x - \mu_ {\tilde {\mathcal {D}} _ {n}}\right). \\ \end{array}
$$
Then we observe that for any  $x\in \mathbb{R}^p$
$$
n \cdot \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {r o w}} \cdot (x - \mu_ {\mathcal {D} _ {n}}) = n \cdot \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {r o w}} \cdot \frac {1}{\sqrt {n}} \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top} \cdot \left(\frac {1}{\sqrt {n}} \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top}\right) ^ {\dagger} \cdot (x - \mu_ {\mathcal {D} _ {n}}) = \sqrt {n} \cdot \Pi_ {\boldsymbol {X} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\mathrm {r o w}} \cdot \boldsymbol {X} _ {\mathrm {c t r}} ^ {\top} \cdot c _ {x}. \tag {58}
$$
$$
n \cdot \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {r o w}} \cdot (x - \mu_ {\widetilde {\boldsymbol {D}} _ {n}}) = n \cdot \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {r o w}} \cdot \frac {1}{\sqrt {n}} \boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top} \cdot \left(\frac {1}{\sqrt {n}} \boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top}\right) ^ {\dagger} \cdot (x - \mu_ {\widetilde {\boldsymbol {D}} _ {n}}) = \sqrt {n} \cdot \Pi_ {\boldsymbol {Z} _ {\mathrm {c t r}} ^ {(\lambda)}} ^ {\operatorname {r o w}} \cdot \boldsymbol {Z} _ {\mathrm {c t r}} ^ {\top} \cdot \tilde {c} _ {x}. \tag {59}
$$
Consequently, for any  $x \in \mathbb{R}^p$ , we obtain from (55) and (56) with aid of (58) and (59) that