SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\epsilon_1^{(j)}$ 's are new i.i.d. Rademacher variables; the last inequality comes from Proposition 6 in [57] (see Equations (5)-(10) in [57] for more details). We can then get rid of the absolute value by
considering swapping  $f$  and  $h$
$$
\begin{array}{l} \sup  _ {f ^ {(j)}, h ^ {(j)} \in \mathcal {G} ^ {(j)}} \left(\sqrt {2} L _ {\ell} \left| \sum_ {j = 1} ^ {n} \epsilon_ {1} ^ {(j)} \left(f ^ {(j)} \left(\theta_ {1}\right) - h ^ {(j)} \left(\theta_ {1}\right)\right) \right| + \sum_ {i = 2} ^ {N} \epsilon_ {i} \left(\ell \left(f \left(\theta_ {i}\right), \theta_ {i}\right) + \ell \left(h \left(\theta_ {i}\right), \theta_ {i}\right)\right)\right) \\ = \max  \left\{\sup  _ {f ^ {(j)}, h ^ {(j)} \in \mathcal {G} ^ {(j)}} \left(\sqrt {2} L _ {\ell} \sum_ {j = 1} ^ {n} \epsilon_ {1} ^ {(j)} (f ^ {(j)} (\theta_ {1}) - h ^ {(j)} (\theta_ {1})) + \sum_ {i = 2} ^ {N} \epsilon_ {i} \left(\ell (f (\theta_ {i}), \theta_ {i}) + \ell (h (\theta_ {i}), \theta_ {i})\right)\right), \right. \\ \sup _ {f ^ {(j)}, h ^ {(j)} \in \mathcal {G} ^ {(j)}} \left(\sqrt {2} L _ {\ell} \sum_ {j = 1} ^ {n} \epsilon_ {1} ^ {(j)} (h ^ {(j)} (\theta_ {1}) - f ^ {(j)} (\theta_ {1})) + \sum_ {i = 2} ^ {N} \epsilon_ {i} \left(\ell (f (\theta_ {i}), \theta_ {i}) + \ell (h (\theta_ {i}), \theta_ {i}))\right) \right\} \\ = \sup  _ {f ^ {(j)}, h ^ {(j)} \in \mathcal {G} ^ {(j)}} \left(\sqrt {2} L _ {\ell} \sum_ {j = 1} ^ {n} \epsilon_ {1} ^ {(j)} \left(f ^ {(j)} \left(\theta_ {1}\right) - h ^ {(j)} \left(\theta_ {1}\right)\right) + \sum_ {i = 2} ^ {N} \epsilon_ {i} \left(\ell \left(f \left(\theta_ {i}\right), \theta_ {i}\right) + \ell \left(h \left(\theta_ {i}\right), \theta_ {i}\right)\right)\right). \\ \end{array}
$$
We can thus upper bounded  $\mathrm{URad}\left(\mathcal{G}_{|S}^{\ell}\right)$  by
$$
\begin{array}{l} \frac {1}{2} \mathbb {E} _ {\epsilon} \sup  _ {f ^ {(j)}, h ^ {(j)} \in \mathcal {G} ^ {(j)}} \left(\sqrt {2} L _ {\ell} \sum_ {j = 1} ^ {n} \epsilon_ {1} ^ {(j)} \left(f ^ {(j)} \left(\theta_ {1}\right) - h ^ {(j)} \left(\theta_ {1}\right)\right) + \sum_ {i = 2} ^ {N} \epsilon_ {i} \left(\ell \left(f \left(\theta_ {i}\right), \theta_ {i}\right) + \ell \left(h \left(\theta_ {i}\right), \theta_ {i}\right)\right)\right) \\ = \mathbb {E} _ {\epsilon} \sup  _ {g ^ {(j)} \in \mathcal {G} ^ {(j)}} \left(\sqrt {2} L _ {\ell} \sum_ {j = 1} ^ {n} \epsilon_ {1} ^ {(j)} g ^ {(j)} (\theta_ {1}) + \sum_ {i = 2} ^ {N} \epsilon_ {i} \ell (g (\theta_ {i}), \theta_ {i})\right), \\ \end{array}
$$
Repeating this procedure for the other coordinates, we can further have
$$
\operatorname {U R a d} \left(\mathcal {G} _ {| S} ^ {\ell}\right) \leq \sqrt {2} L _ {\ell} \mathbb {E} _ {\epsilon} \sup  _ {g ^ {(j)} \in \mathcal {G} ^ {(j)}} \left(\sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {n} \epsilon_ {i} ^ {(j)} g ^ {(j)} \left(\theta_ {i}\right)\right) \leq \sqrt {2} L _ {\ell} \sum_ {j = 1} ^ {n} \mathbb {E} _ {\epsilon} \sup  _ {g ^ {(j)} \in \mathcal {G} ^ {(j)}} \left(\sum_ {i = 1} ^ {N} \epsilon_ {i} ^ {(j)} g ^ {(j)} \left(\theta_ {i}\right)\right),
$$
which leads to our claim in the lemma.
Proof of Theorem 1. We are now geared up for the complete proof.
Proof. We first introduce the sketch of the proof. We mainly utilize Theorem 2 to attain the claimed results in Theorem 1. Specifically, we set the random sample set  $S = \{\theta_i\}_{i=1}^N$ , the function class  $\mathcal{G}$  as  $\{\theta \mapsto c_m \rho(x_\beta(\theta), \theta)^m\}$  (the assumption  $r_i - f_i(x) \in [b, B]$  indicates that  $\rho(x, \theta) = \min_{i \in [m]} \left\{ \frac{r_i - f_i(x)}{\lambda_i(\theta)} \right\} \geq b \geq 0$  and by the definition in Equation (7),  $\rho_\beta(\theta)$  is thus always  $\rho(x(\theta), \theta)^m$ ;  $x_\beta(\cdot)$  is an  $L$ -layer MLP to be specified later). Applying Theorem 2, we can obtain that with probability at least  $1 - \frac{\delta}{2}$ ,
$$
\sup _ {g \in \mathcal {G}} \mathbb {E} _ {\theta} g (\theta) - \frac {1}{N} \sum_ {i} g (\theta_ {i}) \leq \frac {2}{N} \mathrm {U R a d} (\mathcal {G} _ {| S}) + 3 c _ {m} (B \sqrt {m}) ^ {m} \sqrt {\frac {\ln (4 / \delta)}{2 N}},
$$
where the definition of URad and  $\mathcal{G}_{|S}$  can be found in Definition 5. Simply replacing  $\mathcal{G}$  with  $-\mathcal{G} := \{-g : g \in \mathcal{G}\}$ , we can have the inequality of the other direction with probability at least  $1 - \frac{\delta}{2}$ :
$$
\begin{array}{l} \sup _ {g \in - \mathcal {G}} \mathbb {E} _ {\theta} g (\theta) - \frac {1}{N} \sum_ {i} g (\theta_ {i}) \leq \frac {2}{N} \mathrm {U R a d} \left(- \mathcal {G} _ {| S}\right) + 3 c _ {m} (B \sqrt {m}) ^ {m} \sqrt {\frac {\ln (4 / \delta)}{2 N}} \\ \Rightarrow \sup _ {g \in \mathcal {G}} \mathbb {E} _ {\theta} - g (\theta) - \frac {1}{N} \sum_ {i} - g (\theta_ {i}) \leq \frac {2}{N} \mathrm {U R a d} (- \mathcal {G} _ {| S}) + 3 c _ {m} (B \sqrt {m}) ^ {m} \sqrt {\frac {\ln (4 / \delta)}{2 N}} \\ \Rightarrow \sup _ {g \in \mathcal {G}} \frac {1}{N} \sum_ {i} g (\theta_ {i}) - \mathbb {E} _ {\theta} g (\theta) \leq \frac {2}{N} \mathrm {U R a d} (\mathcal {G} _ {| S}) + 3 c _ {m} (B \sqrt {m}) ^ {m} \sqrt {\frac {\ln (4 / \delta)}{2 N}}, \\ \end{array}
$$
where we apply the property  $\mathrm{URad}\left(-\mathcal{G}_{|S}\right) = \mathrm{URad}\left(\mathcal{G}_{|S}\right)$ . We thus, with probability at least  $1 - \delta$  (as a result of union bound), can upper bound  $\sup_{g\in \mathcal{G}}\left|\mathbb{E}_\theta g(\theta) - \frac{1}{N}\sum_i g(\theta_i)\right|$  by
$$
\begin{array}{l} \left. \max  \left\{\sup  _ {g \in \mathcal {G}} \mathbb {E} _ {\theta} g (\theta) - \frac {1}{N} \sum_ {i} g \left(\theta_ {i}\right), \sup  _ {g \in \mathcal {G}} \frac {1}{N} \sum_ {i} g \left(\theta_ {i}\right) - \mathbb {E} _ {\theta} g (\theta) \right\} \right. \\ \leq \frac {2}{N} \operatorname {U R a d} \left(\mathcal {G} _ {| S}\right) + 3 c _ {m} (B \sqrt {m}) ^ {m} \sqrt {\frac {\ln (4 / \delta)}{2 N}}. \\ \end{array}
$$
For the next step, we will upper bound  $\mathrm{URad}\left(\mathcal{G}_{|S}\right)$  by analyzing the structure of  $c_{m}\rho (x_{\beta}(\theta),\theta)^{m}$  and utilizing the existing bound (see Theorem 3) for Rademacher complexity of MLP  $x_{\beta}$ .
The main idea of controlling  $\mathrm{URad}\left(\mathcal{G}_{|S}\right)$  is to obtain the "partially Lipschitz continuity" that  $|\rho (x_{\beta}(\theta),\theta) - \rho (x_{\beta^{\prime}}(\theta),\theta)|\leq L_{\rho}\| x_{\beta}(\theta) - x_{\beta^{\prime}}(\theta)\|$  for a certain  $L_{\rho} > 0$ ; with the "partially Lipschitz continuity" we can apply Lemma 4 and obtain the desired bound. For simplicity, we denote  $x_{\beta}(\theta),x_{\beta^{\prime}}(\theta)$  respectively as  $x,x^{\prime}$ , and use  $\lambda_{j}$ 's as shorthand for  $\lambda_j(\theta)$ 's. We now expand the difference  $|\rho (x_{\beta}(\theta),\theta) - \rho (x_{\beta^{\prime}}(\theta),\theta)|$  as:
$$
\begin{array}{l} \left| \min  _ {j \in [ m ]} \frac {r _ {j} - f _ {j} (x)}{\lambda_ {j}} - \min  _ {k \in [ m ]} \frac {r _ {k} - f _ {k} \left(x ^ {\prime}\right)}{\lambda_ {k}} \right| \\ = \max  \left\{\min  _ {j \in [ m ]} \frac {r _ {j} - f _ {j} (x)}{\lambda_ {j}} - \min  _ {k \in [ m ]} \frac {r _ {k} - f _ {k} \left(x ^ {\prime}\right)}{\lambda_ {k}}, \min  _ {k \in [ m ]} \frac {r _ {k} - f _ {k} \left(x ^ {\prime}\right)}{\lambda_ {k}} - \min  _ {j \in [ m ]} \frac {r _ {j} - f _ {j} (x)}{\lambda_ {j}} \right\}. \\ \end{array}
$$
If we respectively denote the minima index of the two finite-term minimization as  $j^{*}$  and  $k^{*}$ , we can then upper bound  $|\rho (x_{\beta}(\theta),\theta) - \rho (x_{\beta^{\prime}}(\theta),\theta)|$  by
$$
\begin{array}{l} \max  \left\{\frac {r _ {k ^ {*}} - f _ {k ^ {*}} (x)}{\lambda_ {k ^ {*}}} - \frac {r _ {k ^ {*}} - f _ {k ^ {*}} (x ^ {\prime})}{\lambda_ {k ^ {*}}}, \frac {r _ {j ^ {*}} - f _ {j ^ {*}} (x ^ {\prime})}{\lambda_ {j ^ {*}}} - \frac {r _ {j ^ {*}} - f _ {j ^ {*}} (x)}{\lambda_ {j ^ {*}}} \right\} \\ = \max  \left\{\frac {f _ {k ^ {*}} \left(x ^ {\prime}\right) - f _ {k ^ {*}} (x)}{\lambda_ {k ^ {*}}}, \frac {f _ {j ^ {*}} (x) - f _ {j ^ {*}} \left(x ^ {\prime}\right)}{\lambda_ {j ^ {*}}} \right\} \leq \max  _ {j \in \{j ^ {*}, k ^ {*} \}} \frac {\left| f _ {j} (x) - f _ {j} \left(x ^ {\prime}\right) \right|}{\lambda_ {j}} \\ \leq \max  _ {j \in \{j ^ {*}, k ^ {*} \}} \frac {L _ {f} | x - x ^ {\prime} |}{\lambda_ {j}}. \\ \end{array}
$$
We note there is a special property for  $\lambda_{j}$  when  $j$  is the minima index: as  $\| \lambda \| = 1$ , there must be a certain  $\lambda_{j} \geq 1 / \sqrt{m}$ , and since  $b \leq r_{j} - f_{j}(x) \leq B, \forall j$ , we have
$$
\frac {b}{\lambda_ {j ^ {*}}} \leq \frac {r _ {j ^ {*}} - f _ {j ^ {*}} \left(x ^ {\prime}\right)}{\lambda_ {j ^ {*}}} \leq \frac {B}{1 / \sqrt {m}} \Rightarrow \lambda_ {j ^ {*}} \geq \frac {b}{\sqrt {m} B}. \tag {30}
$$
With this special property, we obtain
$$
| \rho (x _ {\beta} (\theta), \theta) - \rho (x _ {\beta^ {\prime}} (\theta), \theta) | \leq \frac {\sqrt {m} B}{b} L _ {f} | x - x ^ {\prime} |.
$$
$$
\begin{array}{l} \left| c _ {m} \rho \left(x _ {\beta} (\theta), \theta\right) ^ {m} - c _ {m} \rho \left(x _ {\beta^ {\prime}} (\theta), \theta\right) ^ {m} \right| \\ = c _ {m} \left| \rho \left(x _ {\beta} (\theta), \theta\right) - \rho \left(x _ {\beta^ {\prime}} (\theta), \theta\right) \right| \left(\sum_ {k = 1} ^ {m} \rho \left(x _ {\beta} (\theta), \theta\right) ^ {m - k} \rho \left(x _ {\beta^ {\prime}} (\theta), \theta\right) ^ {k - 1}\right) \\ \leq c _ {m} \frac {\sqrt {m} B}{b} L _ {f} | x - x ^ {\prime} | m (B \sqrt {m}) ^ {m - 1} = c _ {m} \frac {m}{b} (B \sqrt {m}) ^ {m} L _ {f} | x - x ^ {\prime} |, \\ \end{array}
$$
which establishes the "partially Lipschitz continuity". The distance function  $\rho(x_{\beta}(\theta), \theta)$  is bounded by Lemma 3. We can then apply Lemma 4 and have
$$
\begin{array}{l} \operatorname {U R a d} \left(\mathcal {G} _ {| S}\right) \leq \sqrt {2} c _ {m} \frac {m}{b} (B \sqrt {m}) ^ {m} L _ {f} n \operatorname {U R a d} \left(\mathcal {G} _ {| S} ^ {\mathrm {M L P}}\right) \\ \leq \sqrt {2} c _ {m} \frac {m}{b} (B \sqrt {m}) ^ {m} L _ {f} n \cdot B _ {w} ^ {L} \| X _ {\theta} \| _ {F} (1 + \sqrt {2 L \ln (2)}). \\ \end{array}
$$
Combining the pieces above, we finally have
$$
\begin{array}{l} \sup  _ {g \in \mathcal {G}} \left| \mathbb {E} _ {\theta} g (\theta) - \frac {1}{N} \sum_ {i} g \left(\theta_ {i}\right) \right| \\ \leq \frac {2}{N} \operatorname {U R a d} \left(\mathcal {G} _ {| S}\right) + 3 c _ {m} (B \sqrt {m}) ^ {m} \sqrt {\frac {\ln (4 / \delta)}{2 N}} \\ \leq c _ {m} (B \sqrt {m}) ^ {m} \left(\frac {2 \sqrt {2} m n}{N b} L _ {f} \cdot B _ {w} ^ {L} \| X _ {\theta} \| _ {F} (1 + \sqrt {2 L \ln (2)}) + 3 \sqrt {\frac {\ln (4 / \delta)}{2 N}}\right), \\ \end{array}
$$
which is the generalization error bound we claim.