SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Take  $C = \max(\widetilde{C_1}, \widetilde{C_2})$ . Since  $\mathbb{E}(\| \mathbf{W} \|_{\mathrm{Fr}}) \leq (\mathbb{E} \| \mathbf{W} \|_{\mathrm{Fr}}^2)^{1/2} \leq d + 1$ , by choosing  $R_{\mathbf{W}} = d + 1 + C \sqrt{\log M}$ , the above probability is less than  $1/M^2$ . Then
$$
\begin{array}{l} \left\| f _ {\mathbf {v}} - \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {m = 1} ^ {M} \overline {{\sigma}} (\mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}) \langle \mathbf {v} _ {m}, \widetilde {\mathbf {x}} _ {i} \rangle \right\| _ {L ^ {2} (P)} ^ {2} \leq \frac {3 C R _ {\mathbf {W}} ^ {2} R ^ {2}}{\delta M} + C R ^ {2} \mathbb {P} (\| \mathbf {W} \| _ {\mathsf {F r}} > R _ {\mathbf {W}} / 2) ^ {\frac {1}{2}} \\ \leq \frac {C (\log M + d ^ {2}) R ^ {2}}{\delta M} \\ \end{array}
$$
with probability larger than  $1 - \delta /3$ . This proves Eq. (27).
Step 2. Proof of Eq. (28). By Chebyshev's inequality,
$$
\mathbb {P} \left(\sum_ {m = 1} ^ {M} \| \mathbf {v} _ {m} \| _ {2} - \mathbb {E} \Big [ \| \mathbf {v} (\mathbf {W}) \| _ {2} \mathbf {1} \left\{\| \mathbf {W} _ {m} \| _ {\mathrm {F r}} \leq R _ {\mathbf {W}} / 2 \right\} \Big ] \geq \sqrt {\frac {6 R ^ {2}}{\delta M}}\right)
$$
$$
\leq \frac {\mathbb {E} \left[ \| \mathbf {v} (\mathbf {W}) \| _ {2} ^ {2} \mathbf {1} \left\{\| \mathbf {W} \| _ {\mathrm {F r}} \leq R _ {\mathbf {W}} / 2 \right\} \right] \delta M}{6 R ^ {2} M} \leq \frac {\delta}{3}.
$$
Combining with the fact that  $\mathbb{E}\Big[\| \mathbf{v}(\mathbf{W})\| _2\mathbf{1}\left\{\| \mathbf{W}\|_{\mathrm{Fr}}\leq R_{\mathbf{W}} / 2\right\} \Big]\leq \sqrt{\mathbb{E}\Big[\| \mathbf{v}(\mathbf{W})\|_2^2\Big]}\leq R,$  we have
$$
\mathbb {P} \left(\sum_ {m = 1} ^ {M} \| \mathbf {v} _ {m} \| _ {2} \geq R + \sqrt {\frac {6 R ^ {2}}{\delta M}}\right) \leq \frac {\delta}{3}.
$$
For the second part of (28), Markov inequality gives
$$
\mathbb {P} \left(\sum_ {m = 1} ^ {M} \| \mathbf {v} _ {m} \| _ {2} ^ {2} \geq \frac {6 R ^ {2}}{\delta M}\right) \leq \frac {\mathbb {E} \left[ \| \mathbf {v} (\mathbf {W}) \| _ {2} ^ {2} \mathbf {1} \left\{\| \mathbf {W} _ {m} \| _ {\mathrm {F r}} \leq R _ {\mathbf {W}} / 2 \right\} \right] \delta M}{6 R ^ {2} M} \leq \frac {\delta}{3}.
$$
This proves Eq. (28) and completes the proof of Lemma B.1.
To get the kernel of the RFA model, we have
$$
K (\mathbf {x} _ {0: N}, \mathbf {x} _ {0: N} ^ {\prime}) = \frac {1}{N ^ {2}} \mathbb {E} _ {\mathbf {W}} \Big [ \sum_ {i, j = 1} ^ {N} \sigma (\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \rangle) \sigma (\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} ^ {\prime} (\widetilde {\mathbf {x}} _ {j} ^ {\prime}) ^ {\top} \rangle) \langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \rangle \Big ].
$$
We first consider a single component in the sum, which is
$$
\mathbb {E} _ {\mathbf {W}} \left[ \sigma \left(\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \rangle\right) \sigma \left(\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} ^ {\prime} \left(\widetilde {\mathbf {x}} _ {j}\right) ^ {\prime} \rangle\right) \right] \langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \rangle . \tag {30}
$$
Let  $u_{i,j} = \left\langle \widetilde{\mathbf{x}}_0\widetilde{\mathbf{x}}_i^\top ,\widetilde{\mathbf{x}}_0'(\widetilde{\mathbf{x}}_j')^\top \right\rangle /4$ . Let  $\mathsf{N}_2(\rho)$  denote a bivariate normal distribution with marginals are  $\mathsf{N}(0,1)$  and the correlation is  $\rho \in [-1,1]$ . Then (30) can be expanded as follows:
$$
\begin{array}{l} \left. \mathbb {E} _ {\mathbf {W}} \left[ \sigma \left(\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top} \rangle\right) \sigma \left(\langle \mathbf {W}, \widetilde {\mathbf {x}} _ {0} ^ {\prime} \left(\widetilde {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {\top} \rangle\right) \right] \langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \rangle \right. \\ = \mathbb {E} _ {Z _ {1}, Z _ {2} \sim \mathsf {N} _ {2} (u _ {i, j})} \left[ \sigma (Z _ {1}) \sigma (Z _ {2}) \right] \langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \rangle \\ = \frac {1}{2 \pi} \left(u _ {i, j} \left(\pi / 2 - \operatorname {a r c c o s} u _ {i, j}\right) + \sqrt {1 - u _ {i , j} ^ {2}}\right) \langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \rangle \\ = \frac {1}{2 \pi} \left(1 + \frac {\pi}{2} u _ {i, j} + \sum_ {\ell = 1} ^ {\infty} \frac {(2 \ell - 3) ! !}{(2 \ell) ! ! (2 \ell - 1)} u _ {i, j} ^ {2 \ell}\right) \langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \rangle \\ = \sum_ {\ell \in \{0, 1 \} \cup \{2 k \} _ {k \geq 1}} c _ {\ell} \left\langle \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}, \widetilde {\mathbf {x}} _ {0} ^ {\prime} \left(\widetilde {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {\top} \right\rangle^ {\ell} 4 ^ {- \ell} \left\langle \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {x}} _ {j} ^ {\prime} \right\rangle \\ = \sum_ {\ell \in \{0, 1 \} \cup \{2 k \} _ {k \geq 1}} c _ {\ell} \left\langle 2 ^ {- \ell} \left(\widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}\right) ^ {\otimes \ell} \otimes \widetilde {\mathbf {x}} _ {i}, 2 ^ {- \ell} \left(\widetilde {\mathbf {x}} _ {0} ^ {\prime} \left(\widetilde {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {\top}\right) ^ {\otimes \ell} \otimes \widetilde {\mathbf {x}} _ {j} ^ {\prime} \right\rangle . \\ \end{array}
$$
Here the coefficients  $\{c_{\ell}\}$  satisfy
$$
c _ {0} = 1 / (2 \pi), \quad c _ {1} = 1 / 4, \quad \text {a n d} \quad c _ {2 \ell} = \frac {(2 \ell - 3) ! !}{2 \pi (2 \ell) ! ! (2 \ell - 1)} = O \left(\ell^ {- \frac {5}{2}}\right) \text {f o r} \ell \geq 1.
$$
Therefore, the kernel can be expressed as:
$$
\begin{array}{l} K (\mathbf {x} _ {0: N}, \mathbf {x} _ {0: N} ^ {\prime}) \\ = \frac {1}{N ^ {2}} \sum_ {1 \leq i, j \leq N} \sum_ {\ell \in \{0, 1 \} \cup \{2 k \} _ {k \geq 1}} c _ {\ell} \left\langle 2 ^ {- \ell} \left(\widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}\right) ^ {\otimes \ell} \otimes \widetilde {\mathbf {x}} _ {i}, 2 ^ {- \ell} \left(\widetilde {\mathbf {x}} _ {0} ^ {\prime} \left(\widetilde {\mathbf {x}} _ {j} ^ {\prime}\right) ^ {\top}\right) ^ {\otimes \ell} \otimes \widetilde {\mathbf {x}} _ {j} ^ {\prime} \right\rangle \\ = \sum_ {\ell \in \{0, 1 \} \cup \{2 k \} _ {k \geq 1}} \left\langle \frac {\sqrt {c _ {\ell}}}{N} \sum_ {i = 1} ^ {N} 2 ^ {- \ell} (\widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}) ^ {\otimes \ell} \otimes \widetilde {\mathbf {x}} _ {i}, \frac {\sqrt {c _ {\ell}}}{N} \sum_ {j = 1} ^ {N} 2 ^ {- \ell} (\widetilde {\mathbf {x}} _ {0} ^ {\prime} (\widetilde {\mathbf {x}} _ {j} ^ {\prime}) ^ {\top}) ^ {\otimes \ell} \otimes \widetilde {\mathbf {x}} _ {j} ^ {\prime} \right\rangle . \\ \end{array}
$$
Now we reformulate the target function as
$$
\begin{array}{l} f _ {\star} = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {\ell = 0} ^ {\infty} \sum_ {\max \{r, s \} = \ell} \left\langle \widetilde {\mathbf {x}} _ {0} ^ {\otimes r} \otimes \widetilde {\mathbf {x}} _ {i} ^ {\otimes s}, \mathbf {f} _ {r s} \right\rangle \\ = \frac {1}{N} \sum_ {i = 1} ^ {N} \sum_ {\ell = 2 k, k \geq 0} ^ {\infty} \sum_ {\max  \{r, s \} = \ell \text {o r} \ell - 1} \left\langle (\widetilde {\mathbf {x}} _ {0} \otimes \widetilde {\mathbf {x}} _ {i}) ^ {\otimes \ell} \otimes \widetilde {\mathbf {x}} _ {i}, \widetilde {\mathbf {f}} _ {r s} \right\rangle \\ = \sum_ {\ell = 2 k, k \geq 0} ^ {\infty} \left\langle \frac {2 ^ {\ell}}{\sqrt {c _ {\ell}}} \sum_ {\max  \{r, s \} = \ell \text {o r} \ell - 1} \widetilde {\mathbf {f}} _ {r s}, \frac {\sqrt {c _ {\ell}}}{N} \sum_ {i = 1} ^ {N} 2 ^ {- \ell} (\widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}) ^ {\otimes \ell} \otimes \widetilde {\mathbf {x}} _ {i} \right\rangle , \\ \end{array}
$$
where  $\widetilde{\mathbf{f}}_{rs}$  is a transpose of  $\mathbf{f}_{rs} \otimes \mathbf{1}_{d+1}^{\otimes (2\ell - r - s + 1)}$  with  $\mathbf{1}_{d+1} = (0, \ldots, 0, 1)$ , such that  $\langle \widetilde{\mathbf{x}}_0^{\otimes r} \otimes \widetilde{\mathbf{x}}_i^{\otimes s}, \mathbf{f}_{rs} \rangle = \langle (\widetilde{\mathbf{x}}_0 \otimes \widetilde{\mathbf{x}}_i)^{\otimes \ell} \otimes \widetilde{\mathbf{x}}_i, \widetilde{\mathbf{f}}_{rs} \rangle$  for any  $r \geq 0$  and  $s \geq 0$ . Then by the feature map equivalence property (23), the RKHS norm of  $f^{\star}$  can be bounded as
$$
\begin{array}{l} \| f ^ {\star} \| _ {\mathcal {H} _ {K}} ^ {2} \leq \Big \| \sum_ {\ell = 2 k, k \geq 0} ^ {\infty} \left\langle \frac {2 ^ {\ell}}{\sqrt {c _ {\ell}}} \sum_ {\max \{r, s \} = \ell \mathrm {o r} \ell - 1} \widetilde {\mathbf {f}} _ {r s}, \frac {\sqrt {c _ {\ell}}}{N} \sum_ {i = 1} ^ {N} 2 ^ {- \ell} (\widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}) ^ {\otimes \ell} \otimes \widetilde {\mathbf {x}} _ {i} \right\rangle \Big \| _ {\mathcal {H} _ {K}} ^ {2} \\ = \sum_ {\ell = 2 k, k \geq 0} ^ {\infty} \left\langle \frac {2 ^ {\ell}}{\sqrt {c _ {\ell}}} \sum_ {\max  \{r, s \} = \ell \text {o r} \ell - 1} \widetilde {\mathbf {f}} _ {r s}, \frac {2 ^ {\ell}}{\sqrt {c _ {\ell}}} \sum_ {\max  \{r, s \} = \ell \text {o r} \ell - 1} \widetilde {\mathbf {f}} _ {r s} \right\rangle \\ = \sum_ {\ell = 2 k, k \geq 0} ^ {\infty} 4 ^ {\ell} c _ {\ell} ^ {- 1} \left\| \sum_ {\max  \{r, s \} = \ell \text {o r} \ell - 1} \widetilde {\mathbf {f}} _ {r s} \right\| _ {\mathrm {F r}} ^ {2} \\ \leq \sum_ {k = 0} ^ {\infty} 4 ^ {k} k ^ {4. 5} \sum_ {\max  \{r, s \} = k} \left\| \widetilde {\mathbf {f}} _ {r s} \right\| _ {\mathsf {F r}} ^ {2}. \\ \end{array}
$$
Thus, using again the property (23) with the original feature map of the random feature model, there exists  $\mathbf{v}:\mathbb{R}^{(d + 1)\times (d + 1)}\to \mathbb{R}^{d + 1}$  such that
$$
f _ {\star} = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathbb {E} _ {\mathbf {W}} [ \overline {{\sigma}} (\mathbf {W}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}) \langle \mathbf {v} (\mathbf {W}), \widetilde {\mathbf {x}} _ {i} \rangle ], \text {w i t h}
$$
$$
\mathbb {E} _ {\mathbf {W}} [ \| \mathbf {v} (\mathbf {W}) \| _ {2} ^ {2} ] \leq \sum_ {k = 0} ^ {\infty} 4 ^ {k} k ^ {4. 5} \sum_ {\max  \{r, s \} = k} \| \widetilde {\mathbf {f}} _ {r s} \| _ {\mathbb {F r}} ^ {2}.
$$
Notice that  $\| \widetilde{\mathbf{f}}_{rs}\|_{\mathrm{Fr}}^2 = \| \mathbf{f}_{rs}\|_{\mathrm{Fr}}^2$  by our construction of  $\widetilde{\mathbf{f}}_{rs}$ , so that the right-hand-side of the equation above coincides with Eq. (10). This proves Lemma B.2.
B.2 Preliminary proposition for Theorem 2
To prove Theorem 2, we first present and prove the following proposition that gives a high probability bound for the difference between the empirical risk and the population risk. In the proposition and lemmas below, we denote  $\mathbf{X} = \{\mathbf{x}_{0:N}^{(j)}\}_{j\in [n]}$  and  $\mathbf{y} = \{y_{j}\}_{j\in [n]}$ .
Proposition B.1. Under the setting of Theorem 2. Consider the finite width RF model (25):
$$
f _ {M} ^ {\mathbf {W}} (\mathbf {x} _ {0: N}; \mathbf {V}) = \sum_ {m = 1} ^ {M} \frac {1}{N} \sum_ {j = 1} ^ {n} \overline {{\sigma}} \left(\mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} \widetilde {\mathbf {x}} _ {i} ^ {\top}\right) \left\langle \mathbf {v} _ {m}, \widetilde {\mathbf {x}} _ {i} \right\rangle .
$$
Then with probability at least  $1 - \delta$  (w.r.t.  $\mathbf{W}$ ,  $\mathbf{y}$ , and  $\mathbf{X}$ ), we have
$$
\begin{array}{l} \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) - \mathbb {E} _ {\mathbf {x} _ {0: N}, y} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N}; \mathbf {V}\right), y\right) \right| \\ \lesssim K _ {1} \sqrt {\frac {\log (d M) \log (n N M)}{n}} + \sqrt {\log \left(\frac {6}{\delta}\right)} \left(\frac {K _ {1}}{\sqrt {n}} + \sqrt {\frac {K _ {2}}{M}}\right). \tag {31} \\ \end{array}
$$
The main difficulty of the proof of Proposition B.1 comes from that  $\ell(f_{M}^{\mathbf{W}}, y)$  might be unbounded and that  $\ell(f_{M}^{\mathbf{W}}(\mathbf{x}_{0:N}^{(j)}), y_{j})$  are not independent across  $j$  (since they share the same  $\{\mathbf{W}_{m}\}_{m \in [M]}$ ). So we begin with several lemmas below.
Lemma B.3. Let  $\{\xi_j\}_{j\in [n]}$  be a set of i.i.d. Rademacher random variables. Under the setting of Proposition B.1,
$$
\mathbb {E} _ {\mathbf {X}, \mathbf {y}, \mathbf {W}, \boldsymbol {\xi}} \left[ \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \xi_ {j} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) \right| \right] \lesssim K _ {1} \sqrt {\frac {\log (d M) \log (n N M)}{n}}. \tag {32}
$$
Furthermore, any fixed  $\mathbf{X}$  and  $\mathbf{y}$ ,
$$
\mathbb {E} _ {\mathbf {W}, \boldsymbol {\xi}} \left[ \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \xi_ {j} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) \right| \right] \lesssim K _ {1} \sqrt {\frac {\log (d M) \log (n N M)}{n}}. \tag {33}
$$
Lemma B.4. Under the setting of Proposition B.1. With probability at least  $1 - \delta / 3$  over  $\mathbf{X}$ ,  $\mathbf{y}$ , and  $\mathbf{W}$ ,
$$
\begin{array}{l} \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) - \mathbb {E} _ {\mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) \right] \right| \\ \lesssim \sqrt {\frac {K _ {2} \log (6 / \delta)}{M}} + K _ {1} \sqrt {\frac {\log (d M) \log (n N M)}{n}}. \tag {34} \\ \end{array}
$$
Lemma B.5. Under the setting of Proposition B.1. With probability at least  $1 - \delta / 3$  over  $\mathbf{X}$  and  $\mathbf{y}$ ,
$$
\begin{array}{l} \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \mathbb {E} _ {\mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) \right] - \mathbb {E} _ {\mathbf {X}, \mathbf {y}, \mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) \right] \right| \\ \lesssim K _ {1} \sqrt {\frac {\log (d M) \log (n N M)}{n}} + K _ {1} \sqrt {\frac {\log (6 / \delta)}{n}}. \tag {35} \\ \end{array}
$$
Lemma B.6. Under the setting of Proposition B.1. With probability at least  $1 - \delta / 3$  over  $\mathbf{W}$ ,
$$
\begin{array}{l} \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \mathbb {E} _ {\mathbf {X}, \mathbf {y}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) \right] - \mathbb {E} _ {\mathbf {X}, \mathbf {y}, \mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) \right] \right| \\ \lesssim \sqrt {\frac {K _ {2} \log (6 / \delta)}{M}} + K _ {1} \sqrt {\frac {\log (d M) \log (n N M)}{n}}. \tag {36} \\ \end{array}
$$
The proofs of Lemma B.3, B.4, B.5, and B.6 are contained in section B.2.1. Now assuming they hold, we proceed to prove Proposition B.1.
Proof of Proposition B.1.
Split the left-hand side of inequality (31), we have
$$
\begin{array}{l} \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) - \mathbb {E} _ {\mathbf {X}, \mathbf {y}} \left(\ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y\right)\right) \right| \\ \leq \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}}, y _ {j}\right) - \mathbb {E} _ {\mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}}, y _ {j}\right) \right] \right| \\ + \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \mathbb {E} _ {\mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}}, y _ {j}\right) \right] - \mathbb {E} _ {\mathbf {X}, \mathbf {y}, \mathbf {W}} \left(\ell \left(f _ {M} ^ {\mathbf {W}}, y\right)\right) \right| \\ + \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \mathbb {E} _ {\mathbf {X}, \mathbf {y}} \left(\ell \left(f _ {M} ^ {\mathbf {W}}, y\right)\right) - \mathbb {E} _ {\mathbf {X}, \mathbf {y}, \mathbf {W}} \left(\ell \left(f _ {M} ^ {\mathbf {W}}, y\right)\right) \right| \\ \lesssim K _ {1} \sqrt {\frac {\log (d M) \log (n N M)}{n}} + \sqrt {\log \left(\frac {6}{\delta}\right)} \left(\frac {K _ {1}}{\sqrt {n}} + \sqrt {\frac {K _ {2}}{M}}\right) \\ \end{array}
$$
with probability at least  $1 - \delta$ . Here the last inequality uses Lemma B.4, B.5, and B.6. This proves Proposition B.1.
B.2.1 Proof of auxiliary lemmas
First using Rademacher contraction inequality, since  $\ell(0,y) \leq 1$ , we can center it and only pay an extra term  $1/\sqrt{n}$  in the Rademacher complexity. Then by the Rademacher contraction property (Theorem A.4), the problem boils down to bounding the Rademacher complexity of  $f_M^{\mathbf{W}}$ , which is
$$
\mathbb {E} _ {\mathbf {X}, \mathbf {y}, \mathbf {W}, \boldsymbol {\xi}} \Big [ \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \xi_ {j} f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right) \right| \Big ].
$$
Fix  $\mathbf{X}$ ,  $\mathbf{y}$ , and  $\mathbf{W}$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {\xi}} \left[ \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \xi_ {j} f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right) \right| \right] \\ = \mathbb {E} _ {\boldsymbol {\xi}} \left[ \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \sum_ {m = 1} ^ {M} \left\langle \mathbf {v} _ {m}, \frac {1}{n} \sum_ {j = 1} ^ {n} \xi_ {j} \left[ \frac {1}{N} \sum_ {i = 1} ^ {N} \overline {{\sigma}} (\mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} ^ {(j)} \widetilde {\mathbf {x}} _ {i} ^ {(j) \top}) \widetilde {\mathbf {x}} _ {i} ^ {(j)} \right] \right\rangle \right| \right] \\ \leq K _ {1} \mathbb {E} _ {\boldsymbol {\xi}} \left[ \max  _ {m} \left\| \frac {1}{n} \sum_ {j = 1} ^ {n} \xi_ {j} \left[ \frac {1}{N} \sum_ {i = 1} ^ {N} \bar {\sigma} \left(\mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} ^ {(j)} \widetilde {\mathbf {x}} _ {i} ^ {(j) \top}\right) \widetilde {\mathbf {x}} _ {i} ^ {(j)} \right] \right\| _ {2} \right]. \tag {37} \\ \end{array}
$$
By matrix Bernstein inequality (Theorem A.3), for any fixed  $m$ ,
$$
\mathbb {P} \Big [ \Big \| \frac {1}{n} \sum_ {j = 1} ^ {n} \xi_ {j} \Big [ \frac {1}{N} \sum_ {i = 1} ^ {N} \overline {{\sigma}} (\mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} ^ {(j)} \widetilde {\mathbf {x}} _ {i} ^ {(j) \top}) \widetilde {\mathbf {x}} _ {i} ^ {(j)} \Big ] \Big \| _ {2} \geq \varepsilon \Big ] \leq 2 d \exp \left(- \frac {n \varepsilon^ {2} / 2}{A ^ {2} + K \varepsilon / 3}\right),
$$
$$
\begin{array}{l} A = \max _ {m} \sqrt {\frac {1}{n N ^ {2}} \sum_ {i , j , k} \left[ \overline {{\sigma}} (\mathbf {W} _ {m} , \widetilde {\mathbf {x}} _ {0} ^ {(j)} \widetilde {\mathbf {x}} _ {i} ^ {(j) \top}) \overline {{\sigma}} (\mathbf {W} _ {m} , \widetilde {\mathbf {x}} _ {0} ^ {(j)} \widetilde {\mathbf {x}} _ {k} ^ {(j)}) \left\langle \widetilde {\mathbf {x}} _ {i} ^ {(j)} , \widetilde {\mathbf {x}} _ {k} ^ {(j)} \right\rangle \right]} \\ \lesssim \max  _ {m, i, j} \bar {\sigma} \left(\mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} ^ {(j)} \widetilde {\mathbf {x}} _ {i} ^ {(j) \top}\right), \text {a n d} \\ \end{array}
$$
$$
K = \max _ {i, m} \left\| \frac {1}{N} \sum_ {i = 1} ^ {N} \overline {{\sigma}} (\mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} ^ {(j)} \widetilde {\mathbf {x}} _ {i} ^ {(j) \top}) \widetilde {\mathbf {x}} _ {i} ^ {(j)} \right\| _ {2}.
$$
Using the union bound, we have
$$
\mathbb {P} \Big [ \max _ {m \in [ M ]} \left\| \frac {1}{n} \sum_ {j = 1} ^ {n} \xi_ {j} \Big [ \frac {1}{N} \sum_ {i = 1} ^ {N} \overline {{\sigma}} (\mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} ^ {(j)} \widetilde {\mathbf {x}} _ {i} ^ {(j) \top}) \widetilde {\mathbf {x}} _ {i} ^ {(j)} \Big ] \right\| _ {2} \geq \varepsilon \Big ] \leq 2 d M \exp \left(- \frac {n \varepsilon^ {2}}{A ^ {2} + K \varepsilon / 3}\right).
$$
Therefore, we can bound its expectation with
$$
\mathbb {E} _ {\boldsymbol {\xi}} \left[ \max  _ {m} \left\| \frac {1}{n} \sum_ {j = 1} ^ {n} \xi_ {j} \left[ \frac {1}{N} \sum_ {i = 1} ^ {N} \bar {\sigma} \left(\mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} ^ {(j)} \widetilde {\mathbf {x}} _ {i} ^ {(j) \top}\right) \widetilde {\mathbf {x}} _ {i} ^ {(j)} \right] \right\| _ {2} \right] \lesssim \left[ \sqrt {\frac {\log (d M)}{n}} A + \frac {\log (d M)}{n} K \right]. \tag {38}
$$
Now take expectation over  $\mathbf{X}$ ,  $\mathbf{y}$ , and  $\mathbf{W}$ . Since  $n \geq \log(dM)$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {X}, \mathbf {y}, \mathbf {w}} \Big [ \sqrt {\frac {\log (d M)}{n}} A + \frac {\log (d M)}{n} K \Big ] \\ \lesssim \left(\sqrt {\frac {\log (d M)}{n}} + \frac {\log (d M)}{n}\right) \mathbb {E} _ {\mathbf {X}, \mathbf {y}, \mathbf {W}} \left[ \max  _ {i, j, m} \left| \overline {{\sigma}} \left(\left\langle \mathbf {W} _ {m}, \widetilde {\mathbf {x}} _ {0} ^ {(j)} \widetilde {\mathbf {x}} _ {i} ^ {(j) \top} \right\rangle\right) \right| \right] \\ \lesssim \sqrt {\frac {\log (d M) \log (n N M)}{n}}. \\ \end{array}
$$
Combine this with Eq. (37) and (38), we prove (32).
Fixing any  $\mathbf{X}$  and  $\mathbf{y}$ , only taking expectation over  $\mathbf{W}$ , we get
$$
\mathbb {E} _ {\mathbf {W}} \left[ \sqrt {\frac {\log (d M)}{n}} A + \frac {\log (d M)}{n} K \right] \lesssim \sqrt {\frac {\log (d M) \log (n N M)}{n}}.
$$
Combine this with Eq. (37) and (38), we prove (33). This finishes the proof of Lemma B.3.
Proof for Lemma B.4.
Denote  $\mathbf{X} = \{\mathbf{x}_{0:N}^{(j)}\}_{j\in [n]}$  and  $\mathbf{y} = \{y_{j}\}_{j\in [n]}$  and denote
$$
g \left(\mathbf {W} _ {1: M}; \mathbf {X}, \mathbf {y}\right) = \sup  _ {\mathbf {V} \in \mathcal {V} _ {M}} \left| \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) - \mathbb {E} _ {\mathbf {W}} \left[ \frac {1}{n} \sum_ {j = 1} ^ {n} \ell \left(f _ {M} ^ {\mathbf {W}} \left(\mathbf {x} _ {0: N} ^ {(j)}; \mathbf {V}\right), y _ {j}\right) \right] \right|.
$$
Given  $\mathbf{W}_{1:M} = \{\mathbf{W}_m\}_{m=1}^M$  and  $\mathbf{W}_{1:M}' = \{\mathbf{W}_m'\}_{m=1}^M$ , we define  $\| \mathbf{W}_{1:M} - \mathbf{W}_{1:M}' \|_{\mathbb{F}_r} = (\sum_{m \in [M]} \| \mathbf{W}_m - \mathbf{W}_m' \|_{\mathbb{F}_r}^2)^{1/2}$ . Then we have