SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \| \mathrm {E} [ | g (x, y) | _ {\mathcal {A}} ^ {2} ] ^ {1 / 2} p \| ^ {2} - \left\| \left(\frac {1}{n} \sum_ {i = 1} ^ {n} | g (x _ {i}, y _ {i}) | _ {\mathcal {A}} ^ {2}\right) ^ {1 / 2} p \right\| ^ {2} = \langle p, \mathrm {E} [ | g (x, y) | _ {\mathcal {A}} ^ {2} ] p \rangle - \left\langle p, \frac {1}{n} \sum_ {i = 1} ^ {n} | g (x _ {i}, y _ {i}) | _ {\mathcal {A}} ^ {2} p \right\rangle \\ = \operatorname {E} [ \langle p, | g (x, y) | _ {\mathcal {A}} ^ {2} p \rangle ] - \frac {1}{n} \sum_ {i = 1} ^ {n} \langle p, | g (x _ {i}, y _ {i}) | _ {\mathcal {A}} ^ {2} p \rangle = \operatorname {E} [ \| g (x, y) p \| ^ {2} ] - \frac {1}{n} \sum_ {i = 1} ^ {n} \| g (x _ {i}, y _ {i}) p \| ^ {2}. \\ \end{array}
$$
Since  $(x,y)\mapsto \| g(x,y)p\| ^2$  is a real-valued map, by Theorem 3.3 by Mohri et al. [11], we have
$$
\| \operatorname {E} [ | g (x, y) | _ {\mathcal {A}} ^ {2} ] ^ {1 / 2} p \| ^ {2} - \left\| \left(\frac {1}{n} \sum_ {i = 1} ^ {n} | g (x _ {i}, y _ {i}) | _ {\mathcal {A}} ^ {2}\right) ^ {1 / 2} p \right\| ^ {2} \leq 2 \hat {R} _ {n} (\mathbf {x}, \tilde {\mathcal {G}} (\mathcal {F}, p)) + 3 M \sqrt {\frac {\log (2 / \delta)}{2 n}}.
$$
The inequality  $\| (\sum_{i = 1}^{n}|g(x_i,y_i)|_{\mathcal{A}}^2)^{1 / 2}p\|^2\leq \| \sum_{i = 1}^{n}|g(x_i,y_i)|_{\mathcal{A}}^2\|_{\mathcal{A}}$  completes the proof.
Lemma 4.3 With the same notations in Lemma 4.2, let  $K = 2\sqrt{2} (C + E)$ . Then, we have  $\hat{R}_n(\mathbf{x},\tilde{\mathcal{G}} (\mathcal{F},p))\leq K\hat{R}_n(\mathbf{x},\mathcal{F}p)$ , where  $\mathcal{F}p = \{x\mapsto f(x)p\mid f\in \mathcal{F}\}$ .
Proof Let  $h_i(z) = \| z - y_i p \|^2$  for  $z \in \{f(x)p \mid f \in \mathcal{F}\} \subseteq \mathbb{R}^d$  and  $i = 1, \ldots, n$ . The Lipschitz constant of  $h_i$  is calculated as follows: We have
$$
\begin{array}{l} h _ {i} (z) - h _ {i} \left(z ^ {\prime}\right) = \left\| z \right\| ^ {2} - 2 \left\langle z - z ^ {\prime}, y _ {i} p \right\rangle - \left\| z ^ {\prime} \right\| ^ {2} \\ \leq \left(\| z \| + \| z ^ {\prime} \|\right) \left(\| z \| - \| z ^ {\prime} \|\right) + 2 \| z - z ^ {\prime} \| \| y _ {i} p \| \\ \leq \left(\| z \| + \| z ^ {\prime} \|\right) \left(\| z - z ^ {\prime} \| + \| z ^ {\prime} \| - \| z ^ {\prime} \|\right) + 2 \| z - z ^ {\prime} \| \| y _ {i} p \| \\ \leq (2 C + 2 E) \| z - z ^ {\prime} \|. \\ \end{array}
$$
Thus, we have  $|h_i(z) - h_i(z')| \leq 2(C + E)\|z - z'\|$ . By Corollary 4 by Maurer [40], the statement is proved.
Proof We have
$$
\begin{array}{l} \operatorname {E} \left[ \sup  _ {f \in \mathcal {F} _ {1}} \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \sigma_ {i}, f (x _ {i}) p \rangle \right] = \operatorname {E} \left[ \sup  _ {f \in \mathcal {F} _ {1}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left\langle (\sigma_ {i} p ^ {*}) p, \langle \phi_ {1} (x _ {i}), f \rangle_ {\mathcal {M} _ {1}} p \right\rangle \right] \\ = \operatorname {E} \left[ \sup _ {f \in \mathcal {F} _ {1}} \frac {1}{n} \Big \langle p, \sum_ {i = 1} ^ {n} (p \sigma_ {i} ^ {*}) \left\langle \phi_ {1} (x _ {i}), f \right\rangle_ {\mathcal {M} _ {1}} p \Big \rangle \right] \\ = \operatorname {E} \left[ \sup  _ {f \in \mathcal {F} _ {1}} \frac {1}{n} \left\langle p, \left\langle \sum_ {i = 1} ^ {n} \phi_ {1} \left(x _ {i}\right) \left(\sigma_ {i} p ^ {*}\right), f \right\rangle_ {\tilde {\mathcal {M}} _ {1}} p \right\rangle \right] \\ = \operatorname {E} \left[ \sup  _ {f \in \mathcal {F} _ {1}} \frac {1}{n} \left(\sum_ {i = 1} ^ {n} \phi_ {1} \left(x _ {i}\right) \left(\sigma_ {i} p ^ {*}\right), f\right) _ {\tilde {\mathcal {M}} _ {1}, p} \right] \\ \leq \mathrm {E} \left[ \sup  _ {f \in \mathcal {F} _ {1}} \frac {1}{n} \left\| \sum_ {i = 1} ^ {n} \phi_ {1} \left(x _ {i}\right) \left(\sigma_ {i} p ^ {*}\right) \right\| _ {\tilde {\mathcal {M}} _ {1}, p} \| f \| _ {\tilde {\mathcal {M}} _ {1}, p} \right] (3) \\ \leq \mathrm {E} \left[ \sup  _ {f \in \mathcal {F} _ {1}} \frac {1}{n} \left\langle p, \left| \sum_ {i = 1} ^ {n} \phi_ {1} \left(x _ {i}\right) \left(\sigma_ {i} p ^ {*}\right) \right| _ {\tilde {\mathcal {M}} _ {1}} ^ {2} p \right\rangle^ {1 / 2} \| f \| _ {\tilde {\mathcal {M}} _ {1}} \right] (4) \\ \leq \frac {B _ {1}}{n} \mathrm {E} \left[ \left(\sum_ {i, j = 1} ^ {n} p ^ {*} p \sigma_ {i} ^ {*} k _ {1} \left(x _ {i}, x _ {j}\right) \sigma_ {j} p ^ {*} p\right) ^ {1 / 2} \right] \\ \leq \frac {B _ {1}}{n} \mathrm {E} \left[ \sum_ {i, j = 1} ^ {n} \sigma_ {i} ^ {*} k _ {1} \left(x _ {i}, x _ {j}\right) \sigma_ {j} \right] ^ {1 / 2} (5) \\ = \frac {B _ {1}}{n} \mathrm {E} \Big [ \sum_ {i = 1} ^ {n} \sigma_ {i} ^ {*} k _ {1} (x _ {i}, x _ {i}) \sigma_ {i} \Big ] ^ {1 / 2} = \frac {B _ {1}}{n} \Big (\sum_ {i = 1} ^ {n} \mathrm {t r} k _ {1} (x _ {i}, x _ {i}) \Big) ^ {1 / 2}, \\ \end{array}
$$
where for  $p \in \mathbb{C}^d$  and  $f, g \in \tilde{\mathcal{M}}_1$ ,  $(f, g)_{\tilde{\mathcal{M}}_{1}, p}$  is the semi-inner product defined by  $(f, g)_{\tilde{\mathcal{M}}_{1}, p} = \langle p, \langle f, g \rangle_{\tilde{\mathcal{M}}_1} p \rangle$  and  $|f|_{\tilde{\mathcal{M}}_1}^2 = \langle f, f \rangle_{\tilde{\mathcal{M}}_1}$ . In addition, the inequality (3) is by the Cauchy-Schwartz inequality and the inequality (5) is by the Jensen's inequality. Note that the Cauchy-Schwartz inequality is still valid for semi-inner products. The inequality (4) is derived by the inequality
$$
\| f _ {L} \| _ {\tilde {\mathcal {M}} _ {L}, p} ^ {2} = \left\langle p, \left\langle f _ {L}, f _ {L} \right\rangle_ {\mathcal {M} _ {L}} p \right\rangle \leq \| p \| ^ {2} \| \left\langle f _ {L}, f _ {L} \right\rangle_ {\mathcal {M} _ {L}} \| _ {\mathcal {A}} \leq \| f _ {L} \| _ {\tilde {\mathcal {M}} _ {L}} ^ {2}.
$$
Theorem 4.1 Assume there exists  $D > 0$  such that  $\| k_{1}(x,x)\|_{\mathcal{A}}\leq D$  for any  $x\in \mathcal{A}_0$ . Let  $\tilde{K} = 4\sqrt{2} (\sqrt{D} B_1 + E)B_1$  and  $\tilde{M} = 6(\sqrt{D} B_1 + E)^2$ . Let  $\delta \in (0,1)$ . Then, for any  $g\in \mathcal{G}(\mathcal{F}_1)$ , where  $\mathcal{F}_1$  is defined in Section 3, with probability at least  $1 - \delta$ , we have
Proof For  $f\in \mathcal{M}_1$  and  $x\in \mathcal{A}_0$  , we have
$$
\| f (x) \| _ {\mathcal {A}} = \| \left\langle \phi_ {1} (x), f \right\rangle_ {\mathcal {M} _ {1}} \| _ {\mathcal {A}} \leq \| k _ {1} (x, x) \| _ {\mathcal {A}} ^ {1 / 2} \| f \| _ {\mathcal {M} _ {1}} \leq \sqrt {D} B _ {1}.
$$
Thus, we set  $C$  as  $\sqrt{DB_1}$  and apply Lemmas 4.2, 4.3, and 4.4. Then, for  $p\in \mathbb{R}^d$  satisfy  $\| p\| = 1$  with probability at least  $1 - \delta$ , we have
$$
\| \mathrm {E} [ | g (x, y) | _ {\mathcal {A}} ^ {2} ] ^ {1 / 2} p \| ^ {2} \leq \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} | g (x _ {i}, y _ {i}) | _ {\mathcal {A}} ^ {2} \right\| _ {\mathcal {A}} + \frac {\tilde {K}}{n} \Big (\sum_ {i = 1} ^ {n} \mathrm {t r} k _ {1} (x _ {i}, x _ {i}) \Big) ^ {1 / 2} + \tilde {M} \sqrt {\frac {\log (2 / \delta)}{2 n}}.
$$
$$
\begin{array}{l} \| \mathrm {E} [ | g (x, y) | _ {\mathcal {A}} ^ {2} ] \| _ {\mathcal {A}} = \| \mathrm {E} [ | g (x, y) | _ {\mathcal {A}} ^ {2} ] ^ {1 / 2} \| _ {\mathcal {A}} ^ {2} \\ \leq \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} | g (x _ {i}, y _ {i}) | _ {\mathcal {A}} ^ {2} \right\| _ {\mathcal {A}} + \frac {\tilde {K}}{n} \Big (\sum_ {i = 1} ^ {n} \mathrm {t r} k _ {1} (x _ {i}, x _ {i}) \Big) ^ {1 / 2} + \tilde {M} \sqrt {\frac {\log (2 / \delta)}{2 n}}, \\ \end{array}
$$
Proposition 4.6 We have
$$
\hat {R} _ {n} (\mathbf {x}, \mathcal {F} _ {L} ^ {\mathrm {d e e p}} p) \leq \frac {1}{n} \sup _ {(f _ {j} \in \mathcal {F} _ {j}) _ {j}} \| P _ {f _ {L - 1}} \dots P _ {f _ {1}} | _ {\tilde {\mathcal {V}} (\mathbf {x})} \| _ {\mathrm {o p}} \| f _ {L} \| _ {\mathcal {M} _ {L}} \left(\sum_ {i = 1} ^ {n} \operatorname {t r} (k _ {1} (x _ {i}, x _ {i}))\right) ^ {1 / 2}.
$$
Here,  $\tilde{\mathcal{V}} (\mathbf{x})$  is the submodule of  $\tilde{\mathcal{M}}_1$  generated by  $\phi_{1}(x_{1}),\ldots \phi_{1}(x_{n})$
The following lemma by Lance [33, Proposition 1.2] is used in proving Proposition 4.6. Here, for  $a, b \in \mathcal{A}$ ,  $a \leq b$  means  $b - a$  is Hermitian positive definite.
Lemma A.1 Let  $\mathcal{M}$  and  $\mathcal{N}$  be Hilbert  $\mathcal{A}$ -modules and let  $A$  be an  $\mathcal{A}$ -linear operator from  $\mathcal{M}$  to  $\mathcal{N}$ . Then, we have  $|Aw|_{\mathcal{N}}^2 \leq \|A\|_{\mathrm{op}}^2 |w|_{\mathcal{M}}^2$ .
$$
\begin{array}{l} \mathrm {E} \left[ \sup  _ {f \in \mathcal {F} _ {L} ^ {\mathrm {d e e p}}} \frac {1}{n} \sum_ {i = 1} ^ {n} \langle \sigma_ {i}, f (x _ {i}) p \rangle \right] = \mathrm {E} \left[ \sup  _ {(f _ {j} \in \mathcal {F} _ {j}) _ {j}} \frac {1}{n} \sum_ {i = 1} ^ {n} \langle (\sigma_ {i} p ^ {*}) p, f _ {L} (f _ {L - 1} (\dots f _ {1} (x _ {i}) \dots)) p \rangle \right] \\ = \operatorname {E} \left[ \sup _ {(f _ {j} \in \mathcal {F} _ {j}) _ {j}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left\langle (\sigma_ {i} p ^ {*}) p, \langle \phi_ {L} (f _ {L - 1} (\dots f _ {1} (x _ {i}) \dots)), f _ {L} \rangle_ {\mathcal {M} _ {L}} p \right\rangle \right] \\ = \operatorname {E} \left[ \sup _ {(f _ {j} \in \mathcal {F} _ {j}) _ {j}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left\langle p, (p \sigma_ {i} ^ {*}) \left\langle \phi_ {L} (f _ {L - 1} (\dots f _ {1} (x _ {i}) \dots)), f _ {L} \right\rangle_ {\mathcal {M} _ {L}} p \right\rangle \right] \\ = \operatorname {E} \left[ \sup  _ {(f _ {j} \in \mathcal {F} _ {j}) _ {j}} \frac {1}{n} \left\langle p, \left\langle \sum_ {i = 1} ^ {n} \phi_ {L} (f _ {L - 1} (\dots f _ {1} (x _ {i}) \dots)) (\sigma_ {i} p ^ {*}), f _ {L} \right\rangle_ {\tilde {\mathcal {M}} _ {L}} p \right\rangle \right] \\ \end{array}
$$
$$
\begin{array}{l} = \operatorname {E} \left[ \sup  _ {(f _ {j} \in \mathcal {F} _ {j}) _ {j}} \frac {1}{n} \left(\sum_ {i = 1} ^ {n} \phi_ {L} (f _ {L - 1} (\dots f _ {1} (x _ {i}) \dots)) (\sigma_ {i} p ^ {*}), f _ {L}\right) _ {\tilde {\mathcal {M}} _ {L}, p} \right] \\ \leq \mathrm {E} \left[ \sup  _ {\left(f _ {j} \in \mathcal {F} _ {j}\right) _ {j}} \frac {1}{n} \left\| \sum_ {i = 1} ^ {n} \phi_ {L} \left(f _ {L - 1} \left(\dots f _ {1} \left(x _ {i}\right) \dots\right)\right) \left(\sigma_ {i} p ^ {*}\right) \right\| _ {\tilde {\mathcal {M}} _ {L}, p} \| f _ {L} \| _ {\tilde {\mathcal {M}} _ {L}, p} \right] (6) \\ \leq \mathrm {E} \left[ \sup  _ {(f _ {j} \in \mathcal {F} _ {j}) _ {j}} \frac {1}{n} \left\| P _ {f _ {L - 1}} \dots P _ {f _ {1}} \sum_ {i = 1} ^ {n} \phi_ {1} (x _ {i}) \left(\sigma_ {i} p ^ {*}\right) \right\| _ {\tilde {\mathcal {M}} _ {L}, p} \| f _ {L} \| _ {\tilde {\mathcal {M}} _ {L}} \right] \\ = \operatorname {E} \left[ \sup  _ {\left(f _ {j} \in \mathcal {F} _ {j}\right) _ {j}} \frac {1}{n} \left\langle p, \left| P _ {f _ {L - 1}} \dots P _ {f _ {1}} \sum_ {i = 1} ^ {n} \phi_ {1} \left(x _ {i}\right) \left(\sigma_ {i} p ^ {*}\right) \right| _ {\tilde {\mathcal {M}} _ {L}} ^ {2} p \right\rangle^ {1 / 2} \| f _ {L} \| _ {\tilde {\mathcal {M}} _ {L}} \right] \\ \leq \mathrm {E} \left[ \sup  _ {\left(f _ {j} \in \mathcal {F} _ {j}\right) _ {j}} \frac {1}{n} \left\langle p, \| P _ {f _ {L - 1}} \dots P _ {f _ {1}} | _ {\tilde {\mathcal {V}} (\mathbf {x})} \| _ {\mathrm {o p}} ^ {2} \left| \sum_ {i = 1} ^ {n} \phi_ {1} \left(x _ {i}\right) \left(\sigma_ {i} p ^ {*}\right) \right| _ {\tilde {\mathcal {M}} _ {L}} ^ {2} p \right\rangle^ {1 / 2} \| f _ {L} \| _ {\tilde {\mathcal {M}} _ {L}} \right] (7) \\ \leq \frac {1}{n} \sup  _ {\left(f _ {j} \in \mathcal {F} _ {j}\right) _ {j}} \| P _ {f _ {L - 1}} \dots P _ {f _ {1}} | _ {\tilde {\mathcal {V}} (\mathbf {x})} \| _ {\mathrm {o p}} \| f _ {L} \| _ {\mathcal {M} _ {L}} \operatorname {E} \left[ \left\langle p, \left| \sum_ {i = 1} ^ {n} \phi_ {1} \left(x _ {i}\right) \left(\sigma_ {i} p ^ {*}\right) \right| _ {\tilde {\mathcal {M}} _ {L}} ^ {2} p \right\rangle^ {1 / 2} \right] \\ \leq \frac {1}{n} \sup  _ {\left(f _ {j} \in \mathcal {F} _ {j}\right) _ {j}} \| P _ {f _ {L - 1}} \dots P _ {f _ {1}} | _ {\tilde {\mathcal {V}} (\mathbf {x})} \| _ {\mathrm {o p}} \| f _ {L} \| _ {\mathcal {M} _ {L}} \operatorname {E} \left[ \left(\sum_ {i, j = 1} ^ {n} p ^ {*} p \sigma_ {i} ^ {*} k _ {1} \left(x _ {i}, x _ {j}\right) \sigma_ {j} p ^ {*} p\right) ^ {1 / 2} \right] \\ \leq \frac {1}{n} \sup  _ {(f _ {j} \in \mathcal {F} _ {j}) _ {j}} \| P _ {f _ {1}} \dots P _ {f _ {L - 1}} | _ {\tilde {\mathcal {V}} (\mathbf {x})} \| _ {\mathrm {o p}} \| f _ {L} \| _ {\mathcal {M} _ {L}} \left(\sum_ {i = 1} ^ {n} \operatorname {t r} (k _ {1} (x _ {i}, x _ {i}))\right) ^ {1 / 2}, \\ \end{array}
$$
where the inequality (6) is by the Cauchy-Schwartz inequality and the inequality (7) is by Lemma A.1.
$$
\min _ {(f _ {j} \in \mathcal {M} _ {j}) _ {j}} h (f _ {L} \circ \dots \circ f _ {1} (x _ {1}), \ldots , f _ {L} \circ \dots \circ f _ {1} (x _ {n})) + g _ {1} (P _ {f _ {L - 1}} \dots P _ {f _ {L}} | _ {\tilde {\mathcal {V}} (\mathbf {x})}) + g _ {2} (\| f _ {L} \| _ {\mathcal {M} _ {L}}).
$$
Proof Let  $\mathcal{V}_j(\mathbf{x})$  be the submodule of  $\mathcal{M}_j$  generated by  $\phi_j(x_1^{j-1}), \ldots, \phi_j(x_n^{j-1})$ . Let  $f_j = f_j^{\parallel} + f_j^{\perp}$  where  $f_j^{\parallel} \in \mathcal{V}_j(\mathbf{x})$  and  $f_j^{\perp} \in \mathcal{V}_j(\mathbf{x})^{\perp}$ . Then, for  $j = 1, \ldots, L-1$ , we have
$$
f _ {j} \left(x _ {i} ^ {j - 1}\right) = \left\langle \phi_ {j} \left(x _ {i} ^ {j - 1}\right), f _ {j} \right\rangle_ {\mathcal {M} _ {j}} = \left\langle \phi_ {j} \left(x _ {i} ^ {j - 1}\right), f _ {j} ^ {\parallel} \right\rangle_ {\mathcal {M} _ {j}} = f _ {j} ^ {\parallel} \left(x _ {i} ^ {j - 1}\right). \tag {8}
$$
$$
P _ {f _ {L - 1}} \dots P _ {f _ {1}} | _ {\tilde {\mathcal {V}} (\mathbf {x})} = \prod_ {j = 1} ^ {L - 1} P _ {f _ {j}} | _ {\tilde {\mathcal {V}} _ {j} (\mathbf {x})},
$$
where  $\tilde{\mathcal{V}}_j(\mathbf{x})$  is the submodule of  $\tilde{\mathcal{M}}_j$  generated by  $\phi_j(x_1^{j - 1}),\ldots \phi_j(x_n^{j - 1})$ . In the same manner as Eq. (8), we obtain
$$
\begin{array}{l} P _ {f _ {j}} \sum_ {i = 1} ^ {n} \phi_ {j} \left(x _ {i} ^ {j - 1}\right) c _ {i, j} = \sum_ {i = 1} ^ {n} \phi_ {j + 1} \left(f _ {j} \left(x _ {i} ^ {j - 1}\right)\right) c _ {i, j} = \sum_ {i = 1} ^ {n} \phi_ {j + 1} \left(f _ {j} ^ {\parallel} \left(x _ {i} ^ {j - 1}\right)\right) c _ {i, j} \\ = \sum_ {i = 1} ^ {n} P _ {f _ {j} ^ {\parallel}} \phi_ {j} (x _ {i} ^ {j - 1}) c _ {i, j}. \\ \end{array}
$$
Thus, we have  $P_{f_j}|_{\tilde{\mathcal{V}}_j(\mathbf{x})} = P_{f_j^{\parallel}}|_{\tilde{\mathcal{V}}_j(\mathbf{x})}$ . Furthermore, we have
$$
\big \| f _ {L} \big \| _ {\mathcal {M} _ {L}} = \big \| \big <   f _ {L} ^ {\parallel} + f _ {L} ^ {\perp}, f _ {L} ^ {\parallel} + f _ {L} ^ {\perp} \big > _ {\mathcal {M} _ {L}} \big \| _ {\mathcal {A}} = \big \| \big <   f _ {L} ^ {\parallel}, f _ {L} ^ {\parallel} \big > _ {\mathcal {M} _ {L}} + \big <   f _ {L} ^ {\perp}, f _ {L} ^ {\perp} \big > _ {\mathcal {M} _ {L}} \big \| _ {\mathcal {A}} \geq \big \| \big <   f _ {L} ^ {\parallel}, f _ {L} ^ {\parallel} \big > _ {\mathcal {M} _ {L}} \big \| _ {\mathcal {A}},
$$
where the last inequality is derived from the fact that  $\langle f_L^{\parallel},f_L^{\parallel}\rangle_{\mathcal{M}_L} + \langle f_L^\perp ,f_L^\perp \rangle_{\mathcal{M}_L} - \langle f_L^\perp ,f_L^\perp \rangle_{\mathcal{M}_L}$  is positive and Theorem 2.2.5 (3) by Murphy [34]. As a result, the statement is proved.
Proposition 5.3 For  $j = 1, L$ , let  $[\phi_j(x_1^{j-1}), \ldots, \phi_j(x_n^{j-1})] R_j = Q_j$  be the QR decomposition of  $[\phi_j(x_1^{j-1}), \ldots, \phi_j(x_n^{j-1})]$ . Then, we have  $\| P_{f_{L-1}} \cdots P_{f_1} |_{\tilde{\mathcal{V}}(\mathbf{x})} \|_{\mathrm{op}} = \| R_L^* G_L R_1 \|_{\mathrm{op}}$ .
Proof The result is derived by the identities
$$
\begin{array}{l} \| P _ {f _ {L - 1}} \dots P _ {f _ {1}} | _ {\tilde {\mathcal {V}} (\mathbf {x})} \| _ {\mathrm {o p}} = \| Q _ {L} ^ {*} P _ {f _ {L - 1}} \dots P _ {f _ {1}} Q _ {1} \| _ {\mathrm {o p}} = \| Q _ {L} ^ {*} P _ {f _ {L - 1}} \dots P _ {f _ {1}} [ \phi_ {1} (x _ {1}), \dots , \phi_ {1} (x _ {n}) ] R _ {1} \| _ {\mathrm {o p}} \\ = \left\| R _ {L} ^ {*} \left[ \phi_ {L} \left(x _ {1} ^ {L - 1}\right), \dots , \phi_ {L} \left(x _ {n} ^ {L - 1}\right) \right] ^ {*} \left[ \phi_ {L} \left(x _ {1} ^ {L - 1}\right), \dots , \phi_ {L} \left(x _ {n} ^ {L - 1}\right) \right] R _ {1} \right\| _ {\mathrm {o p}} \\ = \left\| R _ {L} ^ {*} G _ {L} R _ {1} \right\| _ {\mathrm {o p}}. \\ \end{array}
$$
$$
\| P _ {f _ {L - 1}} \dots P _ {f _ {1}} | _ {\tilde {\mathcal {V}} (\mathbf {x})} \| _ {\mathrm {o p}} \leq \| G _ {L} ^ {- 1} \| _ {\mathrm {o p}} ^ {1 / 2} \| G _ {L} \| _ {\mathrm {o p}} \| G _ {1} ^ {- 1} \| _ {\mathrm {o p}} ^ {1 / 2}.
$$
Proof Since  $G_{L}$  is invertible, by Proposition 5.3,  $\| R_{L}^{*}G_{L}R_{1}\|_{\mathrm{op}}$  is bounded as
$$
\| P _ {f _ {L - 1}} \dots P _ {f _ {1}} | _ {\tilde {\mathcal {V}} (\mathbf {x})} \| _ {\mathrm {o p}} = \| R _ {L} ^ {*} G _ {L} R _ {1} \| _ {\mathrm {o p}} \leq \| R _ {L} \| _ {\mathrm {o p}} \| G _ {L} \| _ {\mathrm {o p}} \| R _ {1} \| _ {\mathrm {o p}} = \| G _ {L} ^ {- 1} \| _ {\mathrm {o p}} ^ {1 / 2} \| G _ {L} \| _ {\mathrm {o p}} \| G _ {1} ^ {- 1} \| _ {\mathrm {o p}} ^ {1 / 2},
$$
where the last equality is derived by the identity  $\| R_j\|_{\mathrm{op}}^2 = \| R_jR_j^*\|_{\mathrm{op}} = \| G_j^{-1}\|_{\mathrm{op}}$
B Connection with Generalization Bound with Koopman Operators
Hashimoto et al. [18] derived a generalization bound for the classical neural network composed of linear transformations and activation functions using Koopman operators. The Koopman operator is the adjoint of the Perron-Frobenius operator. In their analysis, they set the final transformation  $f_{L}$  in an RKHS and observed that if the transformation of each layer is injective, the noise is separated through the linear transformations and cut off by  $f_{L}$ . Since the final transformation  $f_{L}$  in our case is in an RKHM, the same interpretation about noise is valid for deep RKHM, too. Indeed, if  $f_{L} \circ \dots \circ f_{1}$  is not injective, then  $G_{L}$  is not invertible, which results in the right-hand side of the inequality (2) going to infinity. The bound derived by Hashimoto et al. also goes to infinity if the network is not injective.
C Experimental Details and Additional Results
We provide details for the experiments in Section 7. All the experiments are executed with Python 3.9 and TensorFlow 2.6 on Intel(R) Core(TM) i9 CPU and NVIDIA Quadro RTX 5000 GPU with CUDA 11.7.
C.1 Comparison with vvRKHS
We set  $k_{j}(x,y) = \tilde{k} (x,y)I$  for  $j = 1,\ldots ,3$  with  $\tilde{k} (x,y) = \mathrm{e}^{-c\sum_{i,j = 1}^{d}|x_{i,j} - y_{i,j}|}$  and  $c = 0.001$  for positive definite kernels. For the optimizer, we used SGD. The learning rate is set as  $1e^{-4}$  both for the deep RKHM and deep vvRKHS. The initial value of  $c_{i,j}$  is set as  $a_{i,j} + \epsilon_{i,j}$ , where  $a_{i,j}\in \mathcal{A}_j$  is the block matrix all of whose elements are 0.1 and  $\epsilon_{i,j}$  is randomly drawn from  $\mathcal{N}(0,0.05)$ . The result in Figure 2 is obtained by 5 independent runs.
C.2 Observation about benign overfitting
We set the same positive definite kernel as Subsection C.1 for  $j = 1,2$ . For the optimizer, we used SGD. The learning rate is set as  $3 \times 1e^{-4}$ . The initial value of  $c_{i,j}$  is the same as Subsection C.1. The result in Figure 2 is obtained by 3 independent runs.
C.3 Comparison to CNN
We set  $k_{j}(x,y) = \tilde{k} (xa_{j},ya_{j})xy^{*}$  for  $j = 1,2$ , where  $a_1$  and  $a_2$  are block matrices whose block sizes are 2 and 4, for positive definite kernels. All the nonzero elements of  $a_j$  are set as 1. We set  $a_1$  and  $a_2$  to induce interactions in the block elements (see Remark 5.2). For the optimizer, we used Adam with learning rate  $1e^{-3}$  for both the deep RKHM and the CNN. The initial value of  $c_{i,j}$  is set as  $\epsilon_{i,j}$ , where  $\epsilon_{i,j}$  is randomly drawn from  $\mathcal{N}(0,0.1)$ .
We combined the deep RKHM and CNN with 2 dense layers. Their activation functions are sigmoid and softmax, respectively. For the CNN layers, we also used the sigmoid activation functions. The loss function is set as the categorical cross-entropy for the CNN. The result in Figure 2 is obtained by 5 independent runs.
C.3.1 Memory consumption and computational cost
Memory consumption We used a CNN with  $(7 \times 7)$  and  $(4 \times 4)$ -filters. On the other hand, for the deep RKHM, we learned the coefficients  $c_{i,j}$  in Proposition 5.1 for  $j = 1, 2$  and  $i = 1, \ldots, n$ . That is, we learned the following coefficients:
-  $n(= 20)$  block diagonal matrices each of whom has four  $(7 \times 7)$ -blocks (for the first layer)
-  $n$  block diagonal matrices each of whom has seven  $(4 \times 4)$ -blocks (for the second layer)
Thus, the size of the parameters we have to learn for the deep RKHM is larger than the CNN. Since the memory consumption depends on the size of learning parameters, the memory consumption is larger for the deep RKHM than for the CNN.
Computational cost In each iteration, we compute  $f(x_{i})$  for  $i = 1, \dots, n$  and the derivative of  $f$  with respect to the learning parameters. Here,  $f = f_{1} \circ f_{2}$  is the network. For the deep RKHM, we compute the product of a Gram matrix (composed of  $n \times n$  block diagonal matrices) and a vector (composed of  $n$  block diagonal matrices) for computing  $f_{j}(x_{i})$  for all  $i = 1, \dots, n$ . Thus, the computational cost for computing  $f(x_{i})$  for all  $i = 1, \dots, n$  is  $O(n^{2}d(m_{1} + m_{2}))$ , where  $m_{1} = 7$  and  $m_{2} = 4$  are the sizes of the block diagonal matrices. For the CNN, the computational cost for computing  $f(x_{i})$  for all  $i = 1, \dots, n$  is  $O(nd^{2}(l_{1} + l_{2}))$ , where  $l_{1} = 7 \times 7$  and  $l_{2} = 4 \times 4$  are the number of elements in the filters. Since we set  $n = 20$  and  $d = 28$ , the computational cost of the deep RKHM for computing  $f(x_{i})$  for  $i = 1, \dots, n$  is smaller than that of the CNN. However, since the size of the learning parameters of the deep RKHM is large, the computational cost for computing the derivative of  $f(x_{i})$  is larger than that of the CNN.
Additional results To compare the deep RKHM to a CNN with the same size of learning parameters (the same memory consumption), we conducted the same experiment as the last experiment in the main text, excepting for the structure of the CNN. We constructed a CNN with  $(28 \times 7 \cdot 20)$  and  $(28 \times 4 \cdot 20)$ -filters (The size of learning parameters is the same as the deep RKHM) and replaced the CNN used in the experiment in the main text. Figure 3 shows the result. The result is similar to that in Figure 2 (c), and the deep RKHM also outperforms the CNN with the same learning parameter size. Since the size of the learning parameter is the same in this case, the computational cost for one iteration for learning the deep RKHM is the same or smaller than that for learning the CNN.
C.3.2 Additional results for benign overfitting
We also observed benign overfitting for the deep RKHM. We compared the train and test losses for the deep RKHM with  $\lambda_{1} = 1$  to those with  $\lambda_{1} = 0$ . The results are illustrated in Figure 4. If we set  $\lambda_{1} = 0$  in Eq. (1), i.e., do not minimize the term in Eq. (2), then whereas the training loss becomes small as the learning process proceeds, the test loss becomes large after sufficient iterations. On the other hand, if we set  $\lambda_{1} = 1$ , i.e., minimize the term in Eq. (2), then the training loss becomes
Figure 3: Comparison of deep RKHM to a CNN with the same size of learning parameter.
Figure 4: Train loss and test loss for the MNIST classification task with  $(\lambda = 1)$  and without  $(\lambda = 0)$  the minimization of the second term in Eq. (1) regarding the Perron-Frobenius operators.
smaller than the case of  $\lambda_{1} = 0$ , and the test loss does not become large even the learning process proceeds. The result implies that the term regarding the Perron-Frobenius operators in Eq. (1) causes overfitting, but it is benign overfitting.