SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} g \left(\mathbf {W} _ {1: M}; \mathbf {X}, \mathbf {y}\right) - g \left(\mathbf {W} _ {1: M} ^ {\prime}; \mathbf {X}, \mathbf {y}\right) \\ \leq \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \left[ \ell \left(f _ {M} ^ {\mathbf {W}}, y _ {j}\right) - \ell \left(f _ {M} ^ {\mathbf {W} ^ {\prime}}, y _ {j}\right) \right] \right| \\ \leq \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{N n} \sum_ {i, j, m} \left[ \overline {{\sigma}} \left(\mathbf {W}, \widetilde {\mathbf {x}} _ {0} ^ {(j)} \widetilde {\mathbf {x}} _ {i} ^ {(j) \top}\right) - \overline {{\sigma}} \left(\mathbf {W} ^ {\prime}, \widetilde {\mathbf {x}} _ {0} ^ {(j)} \widetilde {\mathbf {x}} _ {i} ^ {(j) \top}\right) \right] \langle \mathbf {v} _ {m}, \widetilde {\mathbf {x}} _ {i} \rangle \right| \\ \lesssim \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \sum_ {m} \| \mathbf {W} _ {m} - \mathbf {W} _ {m} ^ {\prime} \| _ {\mathsf {F r}} \| \mathbf {v} _ {m} \| _ {2} \right| \\ \lesssim \| \mathbf {W} _ {1: M} - \mathbf {W} _ {1: M} ^ {\prime} \| _ {\mathrm {F r}} \sqrt {\frac {K _ {2}}{M}}. \\ \end{array}
$$
Since  $\mathbf{W}_m$  has independent standard Gaussian entries, by Gaussian concentration inequality (Theorem A.1), we have that
$$
\mathbb {P} \left[ \left| g \left(\mathbf {W} _ {1: M}; \mathbf {X}, \mathbf {y}\right) - \mathbb {E} _ {\mathbf {W}} \left[ g \left(\mathbf {W} _ {1: M}; \mathbf {X}, \mathbf {y}\right) \mid \mathbf {X}, \mathbf {y} \right] \right| \geq \varepsilon \mid \mathbf {X}, \mathbf {y} \right] \leq 2 \exp \left(- \frac {M \varepsilon^ {2}}{4 K _ {2}}\right). \tag {39}
$$
For the conditional expectation, by symmetrization, we have
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {W}} \left[ g \left(\mathbf {W} _ {1: M}; \mathbf {X}, \mathbf {y}\right) \mid \mathbf {X}, \mathbf {y} \right] \\ = \mathbb {E} _ {\mathbf {W}} \left[ \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) - \mathbb {E} _ {\mathbf {W}} \left[ \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) \right] \right| | \mathbf {X}, \mathbf {y} \right] \\ \leq 2 \mathbb {E} _ {\mathbf {W}, \boldsymbol {\xi}} \left[ \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \sum_ {j = 1} ^ {n} \xi_ {j} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) \right| | \mathbf {X}, \mathbf {y} \right] \\ \lesssim K _ {1} \sqrt {\frac {\log (d M) \log (n N M)}{n}} \tag {40} \\ \end{array}
$$
for any  $\mathbf{X}$  and  $\mathbf{y}$ , where the last inequality uses Lemma B.3. Combining (39) and (40) and taking  $\varepsilon = 2\sqrt{K_2\log(6 / \delta) / M}$ , we have
$$
\begin{array}{l} \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) - \mathbb {E} _ {\mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) \right] \right| \\ \lesssim \sqrt {\frac {K _ {2}}{M}} \sqrt {\log \frac {6}{\delta}} + K _ {1} \sqrt {\frac {\log (d M) \log (n N M)}{n}}, \tag {41} \\ \end{array}
$$
for any  $\mathbf{X}$  and  $\mathbf{y}$ . Since the right-hand side is irrelevant to  $\mathbf{X}$  and  $\mathbf{y}$ , we get (34). This proves Lemma B.4.
$$
h \left(\mathbf {x} _ {0: N}, \mathbf {y}\right) = \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \mathbb {E} _ {\mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}}, y _ {j}\right) \right] - \mathbb {E} _ {\mathbf {X}, \mathbf {y}, \mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}}, y _ {j}\right) \right] \right|. \tag {42}
$$
For each  $i \in [n]$ , let  $\{\mathbf{X}', \mathbf{y}'\}$  differs with  $\{\mathbf{X}, \mathbf{y}\}$  only on  $i$ -th data point. We have
$$
\begin{array}{l} h (\mathbf {X}, \mathbf {y}) - h \left(\mathbf {X} ^ {\prime}, \mathbf {y} ^ {\prime}\right) \\ \leq \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \right. \frac {1}{n} \sum_ {j = 1} ^ {n} \left[ \right. \mathbb {E} _ {\mathbf {W}} \left[ \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right)\right] - \mathbb {E} _ {\mathbf {W}} \left[ \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j) \prime}; \mathbf {V}\right), y _ {j} ^ {\prime}\right)\right]\left. \right| \\ \leq \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \mathbb {E} _ {\mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \left| f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right) - f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j) \prime}; \mathbf {V}\right) \right| \right] \\ \leq \sup _ {\mathbf {V} \in \mathcal {V} _ {M}} \mathbb {E} _ {\mathbf {W}} \left[ \frac {1}{n N} \sum_ {m \in [ M ], k \in [ N ]} \left[ \left| \overline {{\sigma}} (\mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} ^ {(i)} \widetilde {\mathbf {x}} _ {k} ^ {(i) \top}) \right| + \left| \overline {{\sigma}} (\mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} ^ {(i) \prime} \widetilde {\mathbf {x}} _ {k} ^ {(i) \prime \top}) \right| \right] \| \mathbf {v} _ {m} \| _ {2} \right]. \\ \lesssim \frac {K _ {1}}{n}. \\ \end{array}
$$
Therefore,  $h(\mathbf{x}_{0:N}, \mathbf{y})$  satisfies the bounded difference property with the parameter  $\{L_i\}_{i=1}^n$  uniformly bounded by  $\Theta(K_1/n)$ . By bounded difference inequality (Theorem A.2), there's a constant  $\widetilde{C}$  such that
$$
\mathbb {P} \left[ \left| h (\mathbf {X}, \mathbf {y}) - \mathbb {E} _ {\mathbf {X}} \left[ h (\mathbf {X}, \mathbf {y}) \right] \right| \geq \varepsilon \right] \leq 2 \exp \left(- \frac {\widetilde {C} n \varepsilon^ {2}}{K _ {1} ^ {2}}\right).
$$
Combining with Lemma B.3, we have
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {X}, \mathbf {y}} \left[ h (\mathbf {X}, \mathbf {y}) \right] \\ = \mathbb {E} _ {\mathbf {X}, \mathbf {y}} \left[ \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \mathbb {E} _ {\mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}}, y _ {j}\right) \right] - \mathbb {E} _ {\mathbf {X}, \mathbf {y}, \mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}}, y _ {j}\right) \right] \right| \right] \\ \leq 2 \mathbb {E} _ {\mathbf {X}, \mathbf {y}, \boldsymbol {\xi}} \left[ \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \xi_ {j} \mathbb {E} _ {\mathbf {W}} \left[ \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) \right] \right| \right] \\ \lesssim K _ {1} \sqrt {\frac {\log (d M) \log (n N M)}{n}}. \\ \end{array}
$$
Therefore, by taking  $\varepsilon = 2K_{1}[\log (6 / \delta) / (n\widetilde{C})]^{1 / 2}$ , we have
$$
\begin{array}{l} \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \mathbb {E} _ {\mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}}, y _ {j}\right) \right] - \mathbb {E} _ {\mathbf {X}, \mathbf {y}, \mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}}, y _ {j}\right) \right] \right| \\ \lesssim K _ {1} \sqrt {\frac {\log (d M) \log (n N M)}{n}} + K _ {1} \sqrt {\frac {\log (1 / \delta)}{n}} \\ \end{array}
$$
with probability at least  $1 - \delta$ . This proves Lemma B.5.
$$
\varphi (\mathbf {W} _ {1: M}) = \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \mathbb {E} _ {\mathbf {X}, \mathbf {y}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell (f _ {M} ^ {\mathbf {W}}, y _ {j}) \right] - \mathbb {E} _ {\mathbf {X}, \mathbf {y}, \mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell (f _ {M} ^ {\mathbf {W}}, y _ {j}) \right] \right|.
$$
Similar to the proof of Lemma B.4. Given  $\mathbf{W}_{1:M} = \{\mathbf{W}_m\}_{m=1}^M$  and  $\mathbf{W}_{1:M}' = \{\mathbf{W}_m'\}_{m=1}^M$ , define  $\| \mathbf{W}_{1:M} - \mathbf{W}_{1:M}' \|_{\mathrm{Fr}} = \sqrt{\sum_m \| \mathbf{W}_m - \mathbf{W}_m' \|_{\mathrm{Fr}}^2}$ . We have
$$
\begin{array}{l} \varphi \left(\mathbf {W} _ {1: M}\right) - \varphi \left(\mathbf {W} _ {1: M} ^ {\prime}\right) \\ \leq \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \mathbb {E} _ {\mathbf {X}, \mathbf {y}} \left[ \ell \left(f _ {M} ^ {\mathbf {W}}, y _ {j}\right) - \ell \left(f _ {M} ^ {\mathbf {W} ^ {\prime}}, y _ {j}\right) \right] \right| \\ \leq \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{N n} \sum_ {i, j, m} \mathbb {E} _ {\mathbf {X}, \mathbf {y}} \left\{\left[ \overline {{\sigma}} (\mathbf {W}, \widetilde {\mathbf {x}} _ {0} ^ {(j)} \widetilde {\mathbf {x}} _ {i} ^ {(j) \top}) - \overline {{\sigma}} (\mathbf {W} ^ {\prime}, \widetilde {\mathbf {x}} _ {0} ^ {(j)} \widetilde {\mathbf {x}} _ {i} ^ {(j) \top}) \right] \langle \mathbf {v} _ {m}, \widetilde {\mathbf {x}} _ {i} \rangle \right\} \right| \\ \end{array}
$$
$$
\begin{array}{l} \leq \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \sum_ {m} 2 \sqrt {2} \| \mathbf {W} _ {m} - \mathbf {W} _ {m} ^ {\prime} \| _ {\mathrm {F r}} \| \mathbf {v} _ {m} \| _ {2} \right| \\ \lesssim \| \mathbf {W} _ {1: M} - \mathbf {W} _ {1: M} \| _ {\mathrm {F r}} \sqrt {\frac {K _ {2}}{M}}. \\ \end{array}
$$
Since  $\mathbf{W}_m$  has independent standard Gaussian entries, there is a constant  $\widetilde{C}$  s.t.
$$
\mathbb {P} \left[ \left| \varphi \left(\mathbf {W} _ {1: M}\right) - \mathbb {E} _ {\mathbf {W}} \left[ \varphi \left(\mathbf {W} _ {1: M}\right) \right] \right| \geq \varepsilon \right] \leq 2 \exp \left(- \frac {\widetilde {C} M \varepsilon^ {2}}{K _ {2}}\right). \tag {43}
$$
For the expectation, we have
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {W}} \left[ \varphi \left(\mathbf {W} _ {1: M}\right) \right] = \mathbb {E} _ {\mathbf {W}} \left[ \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) - \mathbb {E} _ {\mathbf {X}, \mathbf {y}, \mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}}, y _ {j}\right) \right] \right| \right] \\ \leq 2 \mathbb {E} _ {\mathbf {X}, \mathbf {y}, \mathbf {W}, \boldsymbol {\xi}} \left[ \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \xi_ {j} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) \right| \right] \\ \lesssim K _ {1} \sqrt {\frac {\log (d M) \log (n N M)}{n}}, \tag {44} \\ \end{array}
$$
where the last inequality is by Lemma B.3. Combining (43) and (44) and taking  $\varepsilon = [K_2\log (6 / \delta) / (M\widetilde{C})]^{1 / 2}$ , we get
$$
\begin{array}{l} \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \mathbb {E} _ {\mathbf {X}, \mathbf {y}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}}, y _ {j}\right) \right] - \mathbb {E} _ {\mathbf {X}, \mathbf {y}, \mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}}, y _ {j}\right) \right] \right| \\ \lesssim \sqrt {\frac {K _ {2} \log (1 / \delta)}{M}} + K _ {1} \sqrt {\frac {\log (d M) \log (n N M)}{n}} \\ \end{array}
$$
with probability at least  $1 - \delta$ . This proves Lemma B.6.
Proof. Given any target function  $f_{\star}$ , by Lemma B.1, with probability larger than  $1 - \delta /2$  over  $\mathbf{W}_{1:M}$ , there exists  $\widetilde{\mathbf{V}} = \{\widetilde{\mathbf{v}}_m\}_{m = 1}^M$  such that
$$
\left\| f _ {\star} - f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N}; \widetilde {\mathbf {V}}\right) \right\| _ {L ^ {2} (P)} \lesssim \sqrt {\frac {\left(d ^ {2} + \log M\right) B \left(f _ {\star}\right) \delta^ {- 1}}{M}}, \tag {45}
$$
$$
\sum_ {m = 1} ^ {M} \left\| \tilde {\mathbf {v}} _ {m} \right\| _ {2} \lesssim \sqrt {B (f _ {\star})} + \sqrt {\frac {B (f _ {\star}) \delta^ {- 1}}{M}} \lesssim 2 \sqrt {B (f _ {\star})} \text {a n d} \sum_ {m = 1} ^ {M} \left\| \tilde {\mathbf {v}} _ {m} \right\| _ {2} ^ {2} \lesssim \frac {B (f _ {\star}) \delta^ {- 1}}{M}. \tag {46}
$$
By our choice of  $K_{1}$  and  $K_{2}$ , let  $f_{\widehat{\mathbf{v}},M}^{\mathbf{W}} = f_M^{\mathbf{W}}(\cdot ;\widehat{\mathbf{V}})$  denote the model trained by (11), and let
$$
f_{\mathbf{v}^{\star},M}^{\mathbf{W}} = \operatorname *{arg  min}_{\mathbf{V}\in \mathcal{V}_{M}}L_{D}\big(f_{M}^{\mathbf{W}}(\cdot ;\mathbf{V})\big) \text{and} f_{\widetilde{\mathbf{v}},M}^{\mathbf{W}} = f_{M}^{\mathbf{W}}\left(\mathbf{x}_{0:N};\widetilde{\mathbf{V}}\right).
$$
Then with probability at least  $1 - \delta$  over  $\mathbf{W}$ ,  $\mathbf{X}$ , and  $\mathbf{y}$ , we have
$$
\begin{array}{l} L _ {D} \left(f _ {\hat {\mathbf {v}}, M} ^ {\mathbf {W}}\right) - L _ {D} \left(f ^ {*}\right) (47) \\ \leq L _ {D} \left(f _ {\hat {\mathbf {v}}, M} ^ {\mathbf {W}}\right) - \widehat {L} _ {D} \left(f _ {\hat {\mathbf {v}}, M} ^ {\mathbf {W}}\right) + \widehat {L} _ {D} \left(f _ {\hat {\mathbf {v}}, M} ^ {\mathbf {W}}\right) - \widehat {L} _ {D} \left(f _ {\mathbf {v} ^ {*}, M} ^ {\mathbf {W}}\right) + \widehat {L} _ {D} \left(f _ {\mathbf {v} ^ {*}, M} ^ {\mathbf {W}}\right) \\ - L _ {D} \left(f _ {\mathbf {v} ^ {*}, M} ^ {\mathbf {W}}\right) + L _ {D} \left(f _ {\mathbf {v} ^ {*}, M} ^ {\mathbf {W}}\right) - L _ {D} \left(f _ {\widetilde {\mathbf {v}}, M} ^ {\mathbf {W}}\right) + L _ {D} \left(f _ {\widetilde {\mathbf {v}}, M} ^ {\mathbf {W}}\right) - L _ {D} \left(f _ {\star}\right) (48) \\ \leq L _ {D} \left(f _ {\widehat {\mathbf {v}}, M} ^ {\mathbf {W}}\right) - \widehat {L} _ {D} \left(f _ {\widehat {\mathbf {v}}, M} ^ {\mathbf {W}}\right) + \widehat {L} _ {D} \left(f _ {\mathbf {v} ^ {*}, M} ^ {\mathbf {W}}\right) - L _ {D} \left(f _ {\mathbf {v} ^ {*}, M} ^ {\mathbf {W}}\right) + L _ {D} \left(f _ {\widetilde {\mathbf {v}}, M} ^ {\mathbf {W}}\right) - L _ {D} \left(f ^ {*}\right) (49) \\ \leq 2 \sup  _ {f \in \mathcal {V} _ {M}} \left| L _ {D} (f) - \widehat {L} _ {D} (f) \right| + \| f _ {\tilde {\mathbf {v}}, M} ^ {\mathbf {W}} - f _ {\star} \| _ {L ^ {2} (P)} (50) \\ \end{array}
$$
$$
\begin{array}{l} \lesssim K _ {1} \sqrt {\frac {\log (d M) \log (n N M)}{n}} + \sqrt {\log \left(\frac {1}{\delta}\right)} \left(\frac {K _ {1}}{\sqrt {n}} + \sqrt {\frac {K _ {2}}{M}}\right) + \sqrt {\frac {(d ^ {2} + \log M) B (f _ {\star}) \delta^ {- 1}}{M}} (51) \\ \lesssim \sqrt {\frac {B (f _ {\star})}{n}} \left(\sqrt {\log (d M) \log (n N M)} + \sqrt {\log (\delta^ {- 1})}\right) + \sqrt {\frac {(d ^ {2} + \log M) B (f _ {\star}) \delta^ {- 1}}{M}} (52) \\ \lesssim \sqrt {\frac {B \left(f _ {\star}\right) \left[ \log (d M) \log (n N M) + \log \left(\delta^ {- 1}\right) \right]}{n}} + \sqrt {\frac {\left(d ^ {2} + \log M\right) B \left(f _ {\star}\right) \delta^ {- 1}}{M}}, (53) \\ \end{array}
$$
where from (48) to (49) we use the definition of  $f_{\widehat{\mathbf{v}},M}^{\mathbf{W}}$  and  $f_{\mathbf{v}^{*},M}^{\mathbf{W}}$ . From (49) to (50), we bound  $\ll \aleph_{\mathrm{u}}$  notes: fix  $\gg L_{D}(f_{\widehat{\mathbf{v}},M}^{\mathbf{W}}) - \widehat{L}_{D}(f_{\widehat{\mathbf{v}},M}^{\mathbf{W}})$  and  $L_{D}(f_{\mathbf{v}^{*},M}^{\mathbf{W}}) - \widehat{L}_{D}(f_{\mathbf{v}^{*},M}^{\mathbf{W}})$  by  $\sup_f |L_D(f) - \widehat{L}_D(f)|$  and use the Lipschitzness of  $\ell(f, y)$ . From (50) to (51), we use Proposition B.1 and Lemma B.1. From (51) to (52), we insert the value of  $K_1$  and  $K_2$  into the equation. This proves Theorem 2.
B.4 Proof of Examples in Section 3
B.4.1 Excess risk of RFMLP
Denote  $\mathcal{F}$  to be the set of all functions in the function class (6) and (7), i.e.,
$$
\mathcal {F} = \left\{f _ {\star} \left(\mathbf {x} _ {0: N}\right) = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {r, s \geq 0} ^ {\infty} \left\langle \mathbf {x} _ {0} ^ {\otimes r} \otimes \mathbf {x} _ {i} ^ {\otimes s}, \mathbf {f} _ {r s} \right\rangle : \mathbf {f} _ {r s} \in \mathbb {R} ^ {d ^ {r + s}} \text {s y m m e t r i c ,} r, s \geq 0 \right\}. \tag {54}
$$
Consider the RFMLP model
$$
f _ {M} ^ {\mathrm {M L P}} \left(\mathbf {x} _ {0: N}; \mathbf {v}\right) = \sum_ {m = 1} ^ {M} \sigma \left(\left\langle \mathbf {w} _ {m}, \operatorname {v e c} \left(\mathbf {x} _ {0: N}\right) \right\rangle\right) \cdot v _ {m}, \quad \left\{\mathbf {w} _ {m} \right\} _ {m \in [ M ]} \sim_ {\mathrm {i i d}} \mathsf {N} (\mathbf {0}, \mathbf {I} / (N + 2)), \tag {55}
$$
where  $\operatorname{vec}(\mathbf{x}_{0:N}) = [\mathbf{x}_0; \mathbf{x}_1; \dots; \mathbf{x}_N; 1] \in \mathbb{R}^{dN + d + 1}$ . For target functions that take forms in (6) and (7), define  $B_{\mathrm{MLP}}(f_\star) = \sum_{k=0}^\infty \widetilde{C}_k \sum_{r+s=k} \| \mathbf{f}_{rs} \|_{\mathrm{Fr}}^2$  with  $\widetilde{C}_k = k^{3.5}(N + 2)^{2k}$ . In case where  $f_\star$  admits multiple representations of the form (7),  $B_{\mathrm{MLP}}(f_\star)$  is the infimum of the right-hand-side over all such representations.
Then we consider the empirical risk minimizer over the RFMLP model:
$$
\widehat {\mathbf {v}} = \arg \min  _ {\mathbf {v} \in \mathcal {V} _ {M}} \widehat {L} _ {D} \left(f _ {M} ^ {\mathrm {M L P}} (\cdot ; \mathbf {v})\right), \quad \widehat {L} _ {D} (f) = \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f \left(\mathbf {x} _ {0: N} ^ {(j)}\right), y _ {j}\right), \tag {56}
$$
where the constrained class  $\nu_{M}^{\mathrm{MLP}}$  gives
$$
\mathcal {V} _ {M} ^ {\mathrm {M L P}} = \left\{\mathbf {v} = \left\{v _ {m} \right\} _ {m = 1} ^ {M}: \sum_ {m = 1} ^ {M} | v _ {m} | \leq K _ {1}, \sum_ {m = 1} ^ {M} v _ {m} ^ {2} \leq K _ {2} / M \right\}. \tag {57}
$$
Proposition B.2 (The sample complexity of RFMLP). Let  $f_{\star} = \arg \min_{f \in \mathcal{F}} L_D(f)$  be the population risk minimizer within the target function class (54). Assume  $M > \delta^{-1}$  and  $n > \log(dM)$ . Take  $K_1 = C \sqrt{B_{\mathrm{MLP}}(f_{\star})}$  and  $K_2 = CB_{\mathrm{MLP}}(f_{\star}) \delta^{-1}$  in (57), with  $C$  being a constant. Let  $\widehat{f_M^{\mathrm{MLP}}} = f_M^{\mathrm{MLP}}(\cdot; \widehat{\mathbf{v}})$  be empirical risk minimizer of model RFMLP. Then for any joint distribution  $\mathsf{P}$ , with probability at least  $1 - \delta$  over  $\{\mathbf{w}_m\}_{m \in [M]}$  sampled according to (55) and  $\{(x_{0:N}^{(j)}, y_j)\}_{j \in [n]} \sim_{iid} \mathsf{P}$ , the excess risk is bounded by
$$
L _ {D} \left(\widehat {f} _ {M} ^ {\mathrm {M L P}}\right) - L _ {D} \left(f _ {\star}\right) \leq \widetilde {\mathcal {O}} \left(\sqrt {B _ {\mathrm {M L P}} \left(f _ {\star}\right)} \left[ \sqrt {\frac {1}{n}} + \sqrt {\frac {d ^ {2} \delta^ {- 1}}{M}} \right]\right). \tag {58}
$$
Proof of Proposition B.2. The proof is basically the same as that of RFA model. We only give a sketch of the proof. Firstly, with a few modifications of the proof, we can show that Lemma B.1 also holds for RFMLP, with  $B(f_{\star})$  replaced with  $B_{\mathrm{MLP}}(f_{\star})$ . Lemma B.2 is slightly different, we have the kernel expansion:
$$
K(\mathbf{x}_{0:N},\mathbf{x}^{\prime}_{0:N}) = \sum_{\ell \in \{0,1\} \cup \{2k\}_{k\geq 1}}\Big\langle \sqrt{c_{\ell}} (N + 2)^{-\ell}\Big[ \operatorname {vec}(\mathbf{x}_{0:N})\Big]^{\otimes \ell},\sqrt{c_{\ell}} (N + 2)^{-\ell}\Big[ \operatorname {vec}(\mathbf{x}^{\prime}_{0:N})\Big]^{\otimes \ell}\Big\rangle .
$$
Therefore we can rewrite  $f_{\star}$  as
$$
f _ {\star} = \sum_ {\ell \in \{0, 1 \} \cup \{2 k \} _ {k \geq 1}} \left\langle \frac {(N + 2) ^ {\ell}}{N \sqrt {c _ {\ell}}} \sum_ {i = 1} ^ {N} \sum_ {r + s = \ell} \widetilde {\mathbf {f}} _ {r s, i}, \sqrt {c _ {\ell}} (N + 2) ^ {- \ell} \left[ \operatorname {v e c} \left(\mathbf {x} _ {0: N} ^ {\prime}\right) \right] ^ {\otimes \ell} \right\rangle , \tag {59}
$$
where  $\langle \widetilde{\mathbf{f}}_{rs,i},[\mathrm{vec}(\mathbf{x}_{0:N}^{\prime})]^{\otimes \ell}\rangle = \langle \mathbf{f}_{rs},\mathbf{x}_0^{\otimes r}\otimes \mathbf{x}_i^{\otimes s}\rangle$  . Thus,
$$
\| f _ {\star} \| _ {\mathcal {H} _ {K}} ^ {2} \leq \sum_ {k = 0} ^ {\infty} \widetilde {C} _ {k} \sum_ {r + s = k} \| \mathbf {f} _ {r s} \| _ {\mathbb {F r}} ^ {2} \quad \text {w i t h} \quad \widetilde {C} _ {k} = k ^ {3. 5} (N + 2) ^ {2 k}.
$$
The right-hand-side gives the formulation of  $B_{\mathrm{MLP}}(f_{\star})$ . Then, a similar version of Proposition B.1 holds for RFMLP model  $f_{M}^{\mathbf{W}}$ , i.e., with probability at least  $1 - \delta$  (w.r.t.  $\mathbf{W}$ ,  $\mathbf{y}$ , and  $\mathbf{X}$ ),
$$
\begin{array}{l} \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) - \mathbb {E} _ {\left(\mathbf {x} _ {0: N}, y\right) \sim \mathsf {P}} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N}; \mathbf {V}\right), y\right) \right| \\ \lesssim K _ {1} \sqrt {\frac {(N + 2) \log (d M) \log (n N M)}{n}} + \sqrt {(N + 2) \log (1 / \delta)} \left(\frac {K _ {1}}{\sqrt {n}} + \sqrt {\frac {K _ {2}}{M}}\right). \tag {60} \\ \end{array}
$$
Then combining all of the above equations and following the proof in Section B.2, we get (58).
Remark 1. The representation in (59) is not unique. With a more careful choice of the representation of the target function  $f_{\star}$ , we can get a better bound for  $B_{\mathrm{MLP}}(f_{\star})$ , which is given by
$$
B _ {\mathrm {M L P}} \left(f _ {\star}\right) = \sum_ {k = 0} ^ {\infty} \widetilde {C} _ {k} \sum_ {r + s = k} \| \mathbf {f} _ {r s} \| _ {\mathbf {F r}} ^ {2} \quad \text {w i t h} \quad \widetilde {C} _ {k} = k ^ {3. 5} [ (N + 2) / 2 ] ^ {2 k}. \tag {61}
$$
B.4.2 Proofs of Examples
Proposition B.3 (Restatement of Example 1). For functions of  $\mathbf{x}_0$  of the form
$$
f _ {\star} (\mathbf {x} _ {0: N}) = \sum_ {k = 0} ^ {\infty} \left\langle \mathbf {x} _ {0} ^ {\otimes k}, \mathbf {A} _ {k} \right\rangle , \mathbf {A} _ {k} \in \mathbb {R} ^ {d ^ {k}},
$$
we have that  $B(f_{\star}) = \sum_{k=0}^{\infty} C_k \| \mathbf{A}_k \|_{\mathbb{F}_r}^2$ . Setting  $M = \Theta(d^2 n)$ , the excess risk bound gives  $\widetilde{\mathcal{O}}(\sqrt{\sum_{k=0}^{\infty} k^{4.5} 4^k \| \mathbf{A}_k \|_{\mathbb{F}_r}^2 / n})$ . Moreover, consider  $f_{\star}(\mathbf{x}_{0:N}) = (\boldsymbol{\beta}^\top \mathbf{x}_0)^p$ . The above excess risk of RFA model and the RFMLP model scales as
$$
\mathrm {R F A}: \widetilde {\mathcal {O}} \Big (\mathrm {P o l y} (p) \sqrt {4 ^ {p} \| \boldsymbol {\beta} \| _ {2} ^ {2 p} / n} \Big), \qquad \mathrm {R F M L P}: \widetilde {\mathcal {O}} \Big (\mathrm {P o l y} (p) \sqrt {((N + 2)) ^ {p} \| \boldsymbol {\beta} \| _ {2} ^ {2 p} / n} \Big).
$$
Proof of Proposition B.3. This follows by direct calculation.
Proposition B.4 (Restatement of Example 2). For  $f_{\star} = \frac{1}{N}\sum_{i=1}^{N}\psi(\langle\beta,\mathbf{x}_i\rangle)$  with  $\psi(z) = z\arctan(z/\eta)$  with  $\eta > 2$  and  $\| \beta \|_2 = 1$ . The excess risk bound of RFA model and the RFMLP model are
$$
\mathrm {R F A}: \widetilde {\mathcal {O}} \Big (\sqrt {\sum_ {k = 1} ^ {\infty} k ^ {4 . 5} (2 / \eta) ^ {2 k} / n} \Big) = \widetilde {\mathcal {O}} (\sqrt {1 / n}), \quad \mathrm {R F M L P}: \widetilde {\mathcal {O}} \Big (\sqrt {\sum_ {k = 1} ^ {\infty} k ^ {4 . 5} [ (N + 2) / 2 \eta ] ^ {2 k} / n} \Big).
$$
Proof of Proposition B.4. Use the power series expansion of  $\psi$ , we have
$$
f _ {\star} = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {k = 1} ^ {\infty} (- 1) ^ {k} \frac {\left\langle \boldsymbol {\beta} , \mathbf {x} _ {i} \right\rangle^ {2 k}}{(2 k - 1) \eta^ {2 k - 1}}.
$$
Plug it into the formula of  $B(f_{\star})$  (10) and  $B_{\mathrm{MLP}}(f_{\star})$  (61), we get
$$
B \left(f _ {\star}\right) = \sum_ {k = 1} ^ {\infty} (2 k) ^ {4. 5} 4 ^ {2 k} \| \beta / \eta \| _ {2} ^ {4 k} = \Theta \Big (\sum_ {k = 1} ^ {\infty} k ^ {4. 5} (2 / \eta) ^ {2 k} \Big),
$$
and  $B_{\mathrm{MLP}}(f_\star) = \Theta(\sum_{k=1}^\infty k^{4.5}[(N+2)/2\eta]^{2k})$ . Therefore, by Theorem 2 and Proposition B.2, we get their excess risk. This proves Proposition B.4.