SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In proposition 1 below, we show that the condition  $f_{\lambda} \geq \frac{\lambda}{\lambda + H} f$  is exactly equivalent to  $\sqrt{H}$ -square-root-Lipschitzness.
Proposition 1. A function  $f: \mathbb{R} \to \mathbb{R}$  is non-negative and  $\sqrt{H}$ -square-root-Lipschitz if and only if for any  $x \in \mathbb{R}$  and  $\lambda \geq 0$ , it holds that
$$
f _ {\lambda} (x) \geq \frac {\lambda}{\lambda + H} f (x). \tag {35}
$$
Proof. Suppose that equation (35) holds, then by taking  $\lambda = 0$  and the definition in equation (2), we see that  $f$  must be non-negative. For an non-negative function  $f$ , we observe for any  $x \in \mathbb{R}$ , it holds that
$$
\begin{array}{l} \forall \lambda \geq 0, f _ {\lambda} (x) \geq \frac {\lambda}{\lambda + H} f (x) \\ \Longleftrightarrow \forall \lambda > 0, f _ {\lambda} (x) \geq \frac {\lambda}{\lambda + H} f (x) \quad \text {s i n c e} f _ {\lambda} \geq 0 \\ \Longleftrightarrow \inf  _ {\lambda > 0} \frac {\lambda + H}{\lambda} f _ {\lambda} (x) \geq f (x) \\ \Longleftrightarrow \inf  _ {\lambda > 0} \frac {\lambda + H}{\lambda} \inf  _ {y \in \mathbb {R}} f (y) + \lambda (y - x) ^ {2} \geq f (x) \quad \text {b y} \\ \Longleftrightarrow \inf  _ {y \in \mathbb {R}} \inf  _ {\lambda > 0} \left(1 + \frac {H}{\lambda}\right) f (y) + (\lambda + H) (y - x) ^ {2} \geq f (x) \\ \Longleftrightarrow \inf  _ {y \in \mathbb {R}} f (y) + H (y - x) ^ {2} + 2 \sqrt {f (y) H (y - x) ^ {2}} \geq f (x) \quad \text {b y} \lambda^ {*} = \sqrt {\frac {H f (y)}{(y - x) ^ {2}}} \\ \Longleftrightarrow \forall y \in \mathbb {R}, (\sqrt {f (y)} + \sqrt {H} | y - x |) ^ {2} \geq f (x) \\ \Longleftrightarrow \forall y \in \mathbb {R}, \sqrt {H} | y - x | \geq \sqrt {f (x)} - \sqrt {f (y)} \quad \text {s i n c e} f \geq 0. \\ \end{array}
$$
Therefore,  $f$  must be  $\sqrt{H}$ -square-root-Lipschitz as well. Conversely, if  $f$  is non-negative and  $\sqrt{H}$ -square-root-Lipschitz, then the above implies that (2) must hold and we are done.
Interestingly, there is a similar equivalent characterization for Lipschitz functions as well.
Proposition 2. A function  $f: \mathbb{R} \to \mathbb{R}$  is M-Lipschitz if and only if for any  $x \in \mathbb{R}$  and  $\lambda > 0$ , it holds that
$$
f _ {\lambda} (x) \geq f (x) - \frac {M ^ {2}}{4 \lambda}. \tag {36}
$$
Proof. Observe that for any  $x \in \mathbb{R}$ , it holds that
$$
\begin{array}{l} \forall \lambda > 0, f _ {\lambda} (x) \geq f (x) - \frac {M ^ {2}}{4 \lambda} \\ \Longleftrightarrow \inf  _ {\lambda > 0} f _ {\lambda} (x) + \frac {M ^ {2}}{4 \lambda} \geq f (x) \\ \Longleftrightarrow \inf  _ {\lambda > 0, y \in \mathbb {R}} f (y) + \lambda (y - x) ^ {2} + \frac {M ^ {2}}{4 \lambda} \geq f (x) \quad \text {b y} \\ \Longleftrightarrow \inf  _ {y \in \mathbb {R}} f (y) + M | y - x | \geq f (x) \quad \text {b y} \lambda^ {*} = \frac {M}{2 | y - x |} \\ \Longleftrightarrow \forall y \in \mathbb {R}, M | y - x | \geq f (x) - f (y) \\ \end{array}
$$
and we are done.
Finally, we show that any smooth loss is square-root-Lipschitz. Therefore, the class of square-root-Lipschitz losses is more general than the class of smooth losses studied in Srebro et al. 2010.
Definition 3. A twice differentiable function  $f: \mathbb{R} \to \mathbb{R}$  is  $H$ -smooth if for all  $x$  in  $\mathbb{R}$
$$
\left| f ^ {\prime \prime} (x) \right| \leq H.
$$
The following result is similar to to Lemma 2.1 in Srebro et al. 2010:
Proposition 3. Let  $f: \mathbb{R} \to \mathbb{R}$  be a  $H$ -smooth and non-negative function. Then for any  $x \in \mathbb{R}$ , it holds that
$$
\left| f ^ {\prime} (x) \right| \leq \sqrt {2 H f (x)}.
$$
Therefore,  $\sqrt{f}$  is  $\sqrt{H / 2}$ -Lipschitz.
<sup>1</sup>The definition of smoothness can be stated without twice differentiability, by instead requiring the gradient to be Lipschitz. We make this assumption here simply for convenience.
Proof. Since  $f$  is  $H$ -smooth and non-negative, by Taylor's theorem, for any  $x, y \in \mathbb{R}$ , we have
$$
\begin{array}{l} 0 \leq f (y) \\ = f (x) + f ^ {\prime} (x) (y - x) + \frac {f ^ {\prime \prime} (a)}{2} (y - x) ^ {2} \\ \leq f (x) + f ^ {\prime} (x) (y - x) + \frac {H}{2} (y - x) ^ {2} \\ \end{array}
$$
where  $a \in [\min(x, y), \max(x, y)]$ . Setting  $y = x - \frac{f'(x)}{H}$  yields the desired bound. To show that  $\sqrt{f}$  is Lipschitz, we observe that for any  $x \in \mathbb{R}$
$$
\left| \frac {d}{d x} \sqrt {f (x)} \right| = \left| \frac {f ^ {\prime} (x)}{2 \sqrt {f (x)}} \right| \leq \sqrt {H / 2}
$$
and so we apply Taylor's theorem again to show that
$$
| \sqrt {f (x)} - \sqrt {f (y)} | \leq \sqrt {H / 2} | x - y |
$$
which is the desired definition.
B.2 Properties of Gaussian Distribution
We will make use of the following results without proof.
Gaussian Minimax Theorem. Our proof of Theorem 1 and 6 will closely follow prior works that apply Gaussian Minimax Theorem (GMT) to uniform convergence (Koehler et al. 2021; Zhou et al. 2021; Zhou et al. 2022; Wang et al. 2021; Donhauser et al. 2022). The following result is Theorem 3 of Thrapoulidis et al. 2015 (see also Theorem 1 in the same reference). As explained there, it is a consequence of the main result of Gordon (1985), known as Gordon's Theorem.
Theorem 7 (Thrampoulidis et al. 2015; Gordon 1985). Let  $Z: n \times d$  be a matrix with i.i.d.  $\mathcal{N}(0,1)$  entries and suppose  $G \sim \mathcal{N}(0, I_n)$  and  $H \sim \mathcal{N}(0, I_d)$  are independent of  $Z$  and each other. Let  $S_w, S_u$  be compact sets and  $\psi: S_w \times S_u \to \mathbb{R}$  be an arbitrary continuous function. Define the Primary Optimization (PO) problem
$$
\Phi (Z) := \min  _ {w \in S _ {w}} \max  _ {u \in S _ {u}} \langle u, Z w \rangle + \psi (w, u) \tag {37}
$$
and the Auxiliary Optimization (AO) problem
$$
\phi (G, H) := \min  _ {w \in S _ {w}} \max  _ {u \in S _ {u}} \| w \| _ {2} \langle G, u \rangle + \| u \| _ {2} \langle H, w \rangle + \psi (w, u). \tag {38}
$$
Under these assumptions,  $\operatorname*{Pr}(\Phi (Z) < c)\leq 2\operatorname*{Pr}(\phi (G,H)\leq c)$  for any  $c\in \mathbb{R}$
Furthermore, if we suppose that  $S_w, S_u$  are convex sets and  $\psi(w, u)$  is convex in  $w$  and concave in  $u$ , then  $\operatorname{Pr}(\Phi(Z) > c) \leq 2\operatorname{Pr}(\phi(G, H) \geq c)$ .
GMT is an extremely useful tool because it allows us to convert a problem involving a random matrix into a problem involving only two random vectors. In our analysis, we will make use of a slightly more general version of Theorem 7, introduced by Koehler et al. (2021), to include additional variables which only affect the deterministic term in the minmax problem.
Theorem 8 (Variant of GMT). Let  $Z: n \times d$  be a matrix with i.i.d.  $\mathcal{N}(0,1)$  entries and suppose  $G \sim \mathcal{N}(0, I_n)$  and  $H \sim \mathcal{N}(0, I_d)$  are independent of  $Z$  and each other. Let  $S_W, S_U$  be compact sets in  $\mathbb{R}^d \times \mathbb{R}^{d'}$  and  $\mathbb{R}^n \times \mathbb{R}^{n'}$  respectively, and let  $\psi: S_W \times S_U \to \mathbb{R}$  be an arbitrary continuous function. Define the Primary Optimization (PO) problem
$$
\Phi (Z) := \min  _ {(w, w ^ {\prime}) \in S _ {W}} \max  _ {(u, u ^ {\prime}) \in S _ {U}} \langle u, Z w \rangle + \psi ((w, w ^ {\prime}), (u, u ^ {\prime})) \tag {39}
$$
$$
\phi (G, H) := \min  _ {(w, w ^ {\prime}) \in S _ {W}} \max  _ {(u, u ^ {\prime}) \in S _ {U}} \| w \| _ {2} \langle G, u \rangle + \| u \| _ {2} \langle H, w \rangle + \psi ((w, w ^ {\prime}), (u, u ^ {\prime})). \tag {40}
$$
Theorem 8 requires  $S_W$  and  $S_U$  to be compact. However, we can usually get around the compactness requirement by a truncation argument.
Lemma 1 (Zhou et al. 2022, Lemma 6). Let  $f: \mathbb{R}^d \to \mathbb{R}$  be an arbitrary function and  $S_r^d = \{x \in \mathbb{R}^d : \| x \|_2 \leq r\}$ , then for any set  $\mathcal{K}$ , it holds that
$$
\lim  _ {r \rightarrow \infty} \sup  _ {w \in \mathcal {K} \cap \mathcal {S} _ {r} ^ {d}} f (w) = \sup  _ {w \in \mathcal {K}} f (w). \tag {41}
$$
If  $f$  is a random function, then for any  $t \in \mathbb{R}$
$$
\Pr \left(\sup  _ {w \in \mathcal {K}} f (w) > t\right) = \lim  _ {r \rightarrow \infty} \Pr \left(\sup  _ {w \in \mathcal {K} \cap \mathcal {S} _ {r} ^ {d}} f (w) > t\right). \tag {42}
$$
Lemma 2 (Zhou et al. 2022, Lemma 7). Let  $\mathcal{K}$  be a compact set and  $f, g$  be continuous real-valued functions on  $\mathbb{R}^d$ . Then it holds that
$$
\lim  _ {r \rightarrow \infty} \sup  _ {w \in \mathcal {K}} \inf  _ {0 \leq \lambda \leq r} \lambda f (w) + g (w) = \sup  _ {w \in \mathcal {K}: f (w) \geq 0} g (w). \tag {43}
$$
If  $f$  and  $g$  are random functions, then for any  $t \in \mathbb{R}$
$$
\Pr \left(\sup  _ {w \in \mathcal {K}: f (w) \geq 0} g (w) \geq t\right) = \lim  _ {r \rightarrow \infty} \Pr \left(\sup  _ {w \in \mathcal {K}} \inf  _ {0 \leq \lambda \leq r} \lambda f (w) + g (w) \geq t\right). \tag {44}
$$
Concentration inequalities. Let  $\sigma_{\mathrm{min}}(A)$  denote the minimum singular value of an arbitrary matrix  $A$ , and  $\sigma_{\mathrm{max}}$  the maximum singular value. We use  $\| A \|_{op} = \sigma_{\mathrm{max}}(A)$  to denote the operator norm of matrix  $A$ . The following concentration results for Gaussian vector and matrix are standard.
Lemma 3 (Special case of Theorem 3.1.1 of Vershynin 2018). Suppose that  $Z \sim \mathcal{N}(0, I_n)$ . Then
$$
\Pr \left(\left| \| Z \| _ {2} - \sqrt {n} \right| \geq t\right) \leq 4 e ^ {- t ^ {2} / 4}. \tag {45}
$$
Lemma 4 (Koehler et al. 2021, Lemma 10). For any covariance matrix  $\Sigma$  and  $H\sim \mathcal{N}(0,I_d)$ , with probability at least  $1 - \delta$ , it holds that
$$
1 - \frac {\left\| \Sigma^ {1 / 2} H \right\| _ {2} ^ {2}}{\operatorname {T r} (\Sigma)} \lesssim \frac {\log (4 / \delta)}{\sqrt {R (\Sigma)}} \tag {46}
$$
$$
\left\| \Sigma H \right\| _ {2} ^ {2} \lesssim \log (4 / \delta) \operatorname {T r} \left(\Sigma^ {2}\right). \tag {47}
$$
Therefore, provided that  $R(\Sigma) \gtrsim \log(4 / \delta)^2$ , it holds that
$$
\left(\frac {\| \Sigma H \| _ {2}}{\| \Sigma^ {1 / 2} H \| _ {2}}\right) ^ {2} \lesssim \log (4 / \delta) \frac {\operatorname {T r} (\Sigma^ {2})}{\operatorname {T r} (\Sigma)}. \tag {48}
$$
Theorem 9 (Vershynin 2010, Corollary 5.35). Let  $n, N \in \mathbb{N}$ . Let  $A \in \mathbb{R}^{N \times n}$  be a random matrix with entries i.i.d.  $\tilde{\mathcal{N}}(0,1)$ . Then for any  $t > 0$ , it holds with probability at least  $1 - 2\exp(-t^2/2)$  that
$$
\sqrt {N} - \sqrt {n} - t \leq \sigma_ {\min } (A) \leq \sigma_ {\max } (A) \leq \sqrt {N} + \sqrt {n} + t. \tag {49}
$$
Conditional Distribution of Gaussian. To handle arbitrary multi-index conditional distributions of  $y$  given by assumption (B), we will apply a conditioning argument. After conditioning on  $W^T x$  and  $\xi$ , the response  $y$  is no longer random. Importantly, the conditional distribution of  $x$  remains Gaussian (though with a different mean and covariance) and so we can still apply GMT. In the lemma below,  $Z \in \mathbb{R}^{n \times d}$  is a random matrix with i.i.d.  $\mathcal{N}(0,1)$  entries and  $X = Z\Sigma^{1/2}$ .
Lemma 5 (Zhou et al. 2022, Lemma 4). Fix any integer  $k < d$  and any  $k$  vectors  $w_1^*, \ldots, w_k^*$  in  $\mathbb{R}^d$  such that  $\Sigma^{1/2}w_1^*, \ldots, \Sigma^{1/2}w_k^*$  are orthonormal. Denoting
$$
P = I _ {d} - \sum_ {i = 1} ^ {k} \left(\Sigma^ {1 / 2} w _ {i} ^ {*}\right) \left(\Sigma^ {1 / 2} w _ {i} ^ {*}\right) ^ {T}, \tag {50}
$$
the distribution of  $X$  conditional on  $Xw_{1}^{*} = \eta_{1},\dots,Xw_{k}^{*} = \eta_{k}$  is the same as that of
$$
\sum_ {i = 1} ^ {k} \eta_ {i} \left(\Sigma w _ {i} ^ {*}\right) ^ {T} + Z P \Sigma^ {1 / 2}. \tag {51}
$$
B.3 Vapnik-Chervonenkis (VC) theory
By the conditioning step mentioned above, we will separate  $x$  into a low-dimensional component  $W^{T}x$  and the independent component  $Q^{T}x$ . Concentration results for the low-dimensional component can be easily established using VC theory. As mentioned in Zhou et al. 2022, low-dimensional concentration can be established using alternative results (e.g., Vapnik 1982; Panchenko 2002; Panchenko 2003; Mendelson 2017).
Recall the following definition of VC-dimension from Shalev-Shwartz and Ben-David (2014).
Definition 4. Let  $\mathcal{H}$  be a class of functions from  $\mathcal{X}$  to  $\{0,1\}$  and let  $C = \{c_1,\dots,c_m\} \subset \mathcal{X}$ . The restriction of  $\mathcal{H}$  to  $C$  is
$$
\mathcal {H} _ {C} = \left\{\left(h \left(c _ {1}\right),..., h \left(c _ {m}\right)\right): h \in \mathcal {H} \right\}.
$$
A hypothesis class  $\mathcal{H}$  shatters a finite set  $C\subset \mathcal{X}$  if  $|\mathcal{H}_C| = 2^{|C|}$ . The VC-dimension of  $\mathcal{H}$  is the maximal size of a set that can be shattered by  $\mathcal{H}$ . If  $\mathcal{H}$  can shatter sets of arbitrary large size, we say  $\mathcal{H}$  has infinite VC-dimension.
Also, we have the following well-known result for the class of nonhomogenous halfspaces in  $\mathbb{R}^d$  (Theorem 9.3 of Shalev-Shwartz and Ben-David (2014)), and the result on VC-dimension of the union of two hypothesis classes (Lemma 3.2.3 of Blumer et al. (1989)):
Theorem 10. The class  $\{x\mapsto sign(\langle w,x\rangle +b):w\in \mathbb{R}^d,b\in \mathbb{R}\}$  has VC-dimension  $d + 1$
Theorem 11. Let  $\mathcal{H}$  a hypothesis classes of finite VC-dimension  $d\geq 1$ . Let  $\mathcal{H}_2\coloneqq \{\max (h_1,h_2): h_1,h_2\in \mathcal{H}\}$  and  $\mathcal{H}_3\coloneqq \{\min (h_1,h_2): h_1,h_2\in \mathcal{H}\}$ . Then, both the VC-dimension of  $\mathcal{H}_2$  and the VC-dimension of  $\mathcal{H}_3$  are  $O(d)$ .
By combining Theorem 10 and 11, we can easily verify the VC assumption in Corollary 1 for the phase retrieval loss  $f(\hat{y}, y) = (|\hat{y}| - y)^2$ . Similar results can be proven for ReLU regression. To verify the VC assumption for single-index neural nets in Corollary 2, we can use the following result (equation 2 of Bartlett et al. (2019)):
Theorem 12. The VC-dimension of a neural network with piecewise linear activation function,  $W$  parameters, and  $L$  layers has VC-dimension  $O(WL\log W)$ .
We can easily establish low-dimensional concentration due to the following result:
Theorem 13 (Vapnik 1982, Special case of Assertion 4 in Chapter 7.8; see also Theorem 7.6). Suppose that the loss function  $l: \mathcal{Z} \times \Theta \to \mathbb{R}_{\geq 0}$  satisfies
(i) for every  $\theta \in \Theta$ , the function  $l(\cdot, \theta)$  is measurable with respect to the first argument
(ii) the class of functions  $\{z\mapsto \mathbb{1}\{l(z,\theta) > t\} :(\theta ,t)\in \Theta \times \mathbb{R}\}$  has VC-dimension at most  $h$  and the distribution  $\mathcal{D}$  over  $\mathcal{Z}$  satisfies for every  $\theta \in \Theta$
$$
\frac {\mathbb {E} _ {z \sim \mathcal {D}} \left[ l (z , \theta) ^ {4} \right] ^ {1 / 4}}{\mathbb {E} _ {z \sim \mathcal {D}} \left[ l (z , \theta) \right]} \leq \tau , \tag {52}
$$
then for any  $n > h$ , with probability at least  $1 - \delta$  over the choice of  $(z_{1},\ldots ,z_{n})\sim \mathcal{D}^{n}$ , it holds uniformly over all  $\theta \in \Theta$  that
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} l \left(z _ {i}, \theta\right) \geq \left(1 - 8 \tau \sqrt {\frac {h (\log (2 n / h) + 1) + \log (1 2 / \delta)}{n}}\right) \mathbb {E} _ {z \sim \mathcal {D}} [ l (z, \theta) ]. \tag {53}
$$
It is clear that Theorem 1 is a special case of Theorem 6. Therefore, we will prove the more general result here.
Notation. Following the tradition in statistics, we denote  $X = (x_{1},\dots,x_{n})^{T}\in \mathbb{R}^{n\times d}$  as the design matrix. In the proof section, we slightly abuse the notation of  $\eta_{i}$  to mean  $Xw_{i}^{*}$  and  $\xi$  to mean the  $n$ -dimensional random vector whose  $i$ -th component satisfies  $y_{i} = g(\eta_{1,i},\dots,\eta_{k,i},\xi_{i})$ . We will write  $X = Z\Sigma^{1 / 2}$  where  $Z$  is a random matrix with i.i.d. standard normal entries if  $\mu = 0$ .
Throughout this section, we can first assume  $\mu = 0$  in Assumption (A) without loss of generality because if we define  $\tilde{f}:\mathbb{R}\times \mathcal{Y}\times \Theta \to \mathbb{R}$  by
$$
\tilde {f} (\hat {y}, y, \theta) := f (\hat {y} + \langle w (\theta), \mu \rangle , y, \theta), \tag {54}
$$
then by definition, it holds that
$$
f (\langle w (\theta), x \rangle , y, \theta) = \tilde {f} (\langle w (\theta), x - \mu \rangle , y, \theta)
$$
and so we can apply the theory on  $\tilde{f}$  first and then translate to the problem on  $f$ . Similarly, we can also assume  $\Sigma^{1/2}w_1^*, \dots, \Sigma^{1/2}w_k^*$  are orthonormal without loss of generality. This is because we can denote  $W \in \mathbb{R}^{d \times k}$  by  $W = [w_1^*, \dots, w_k^*]$  and let  $\tilde{W} = W(W^T\Sigma W)^{-1/2}$ . By definition, it holds that  $\tilde{W}^T\Sigma \tilde{W} = I$  and so the columns of  $\tilde{W} = [\tilde{w}_1^*, \dots, \tilde{w}_k^*]$  satisfy  $\Sigma^{1/2}\tilde{w}_1^*, \dots, \Sigma^{1/2}\tilde{w}_k^*$  are orthonormal. If we define  $\tilde{g}: \mathbb{R}^{k+1} \to \mathbb{R}$  by
$$
\tilde {g} \left(\eta_ {1}, \dots , \eta_ {k}, \xi\right) = g \left(\left[ \eta_ {1}, \dots , \eta_ {k} \right] \left(W ^ {T} \Sigma W\right) ^ {1 / 2} + \mu^ {T} W, \xi\right), \tag {55}
$$
then  $y = \tilde{g} (x^{T}\tilde{W},\xi)$  and so we can apply the theory on  $\tilde{g}$
We will write the generalization problem as a Primary Optimization problem in Theorem 8. For generality, we will let  $F$  be any deterministic function and then choose it in the end.
Lemma 6. Fix an arbitrary set  $\Theta \subseteq \mathbb{R}^p$  and let  $F:\Theta \to \mathbb{R}$  be any deterministic and continuous function. Consider dataset  $(X,Y)$  drawn i.i.d. from the data distribution  $\mathcal{D}$  according to (A) and (B) with  $\mu = 0$  and orthonormal  $\Sigma^{1 / 2}w_1^*,\dots,\Sigma^{1 / 2}w_k^*$ . Then conditioned on  $Xw_{1}^{*} = \eta_{1},\dots,Xw_{k}^{*} = \eta_{k}$  and  $\xi$ , if we define
$$
\Phi := \sup  _ {\substack {(w, u, \theta) \in \mathbb {R} ^ {d} \times \mathbb {R} ^ {n} \times \Theta \\ w = P \Sigma^ {1 / 2} w (\theta)}} \inf  _ {\lambda \in \mathbb {R} ^ {n}} \left\langle \lambda , Z w \right\rangle + \psi (u, \theta , \lambda \mid \eta_ {1}, \dots , \eta_ {k}, \xi) \tag{56}
$$
where  $P$  is defined in (50) and  $\psi$  is a deterministic and continuous function given by
$$
\begin{array}{l} \psi (u, \theta , \lambda | \eta_ {1}, \dots , \eta_ {k}, \xi) = F (\theta) - \frac {1}{n} \sum_ {i = 1} ^ {n} f (u _ {i}, g (\eta_ {1, i}, \dots , \eta_ {k, i}, \xi_ {i}), \theta) \\ + \langle \lambda , \left(\sum_ {i = 1} ^ {k} \eta_ {i} \left(\Sigma w _ {i} ^ {*}\right) ^ {T}\right) w (\theta) - u \rangle , \tag {57} \\ \end{array}
$$
then it holds that for any  $t\in \mathbb{R}$  we have
$$
\Pr \left(\sup  _ {\theta \in \Theta} F (\theta) - \hat {L} (\theta) > t \mid \eta_ {1}, \dots , \eta_ {k}, \xi\right) = \Pr (\Phi > t). \tag {58}
$$
Proof. By introducing a variable  $u = Xw(\theta)$ , we have
$$
\begin{array}{l} \sup  _ {\theta \in \Theta} F (\theta) - \hat {L} (\theta) = \sup  _ {\theta \in \Theta} F (\theta) - \frac {1}{n} \sum_ {i = 1} ^ {n} f (\langle w (\theta), x _ {i} \rangle , y _ {i}, \theta) \\ = \sup  _ {\theta \in \Theta , u \in \mathbb {R} ^ {n}} \inf  _ {\lambda \in \mathbb {R} ^ {n}} \left\langle \lambda , X w (\theta) - u \right\rangle + F (\theta) - \frac {1}{n} \sum_ {i = 1} ^ {n} f (u _ {i}, y _ {i}, \theta). \\ \end{array}
$$
Conditioned on  $Xw_{1}^{*} = \eta_{1},\dots,Xw_{k}^{*} = \eta_{k}$  and  $\xi$ , the above is only random in  $X$  by our multi-index model assumption on  $y$ . By Lemma 5, the above is equal in law to
$$
\begin{array}{l} \sup  _ {\theta \in \Theta , u \in \mathbb {R} ^ {n}} \inf  _ {\lambda \in \mathbb {R} ^ {n}} \left\langle \lambda , \left(\sum_ {i = 1} ^ {k} \eta_ {i} \left(\Sigma w _ {i} ^ {*}\right) ^ {T} + Z P \Sigma^ {1 / 2}\right) w (\theta) - u \right\rangle + F (\theta) - \frac {1}{n} \sum_ {i = 1} ^ {n} f (u _ {i}, y _ {i}, \theta) \\ = \sup  _ {\theta \in \Theta , u \in \mathbb {R} ^ {n}} \inf  _ {\lambda \in \mathbb {R} ^ {n}} \left\langle \lambda , \left(Z P \Sigma^ {1 / 2}\right) w (\theta) \right\rangle + \psi (u, \theta , \lambda | \eta_ {1}, \dots , \eta_ {k}, \xi) \\ = \sup_{\substack{(w,u,\theta)\in \mathbb{R}^{d}\times \mathbb{R}^{n}\times \Theta \\ w = P\Sigma^{1 / 2}w(\theta)}}\inf_{\lambda \in \mathbb{R}^{n}}\langle \lambda ,Zw\rangle +\psi (u,\theta ,\lambda   |  \eta_{1},\ldots ,\eta_{k},\xi) \\ = \Phi . \\ \end{array}
$$
The function  $\psi$  is continuous because we require  $F, f$  and  $w$  to be continuous in the definitions.
Next, we are ready to apply Gaussian Minimax Theorem. Although the domains in (56) are not compact, we can use the truncation lemmas 1 and 2 in Appendix B.
Lemma 7. In the same setting as Lemma 6, define the auxiliary problem as
$$
\Psi := \sup  _ {\substack {(u, \theta) \in \mathbb {R} ^ {n} \times \Theta \\ \langle H, P \Sigma^ {1 / 2} w (\theta) \rangle \geq \left\| \| P \Sigma^ {1 / 2} w (\theta) \| _ {2} G + \sum_ {i = 1} ^ {k} \langle w (\theta), \Sigma w _ {i} ^ {*} \rangle \eta_ {i} - u \right\| _ {2}}} F (\theta) - \frac {1}{n} \sum_ {i = 1} ^ {n} f (u _ {i}, y _ {i}, \theta) \tag{59}
$$
then for any  $t\in \mathbb{R}$  , it holds that
$$
\Pr \left(\sup  _ {\theta \in \mathcal {K}} F (\theta) - \hat {L} (\theta) > t\right) \leq 2 \Pr (\Psi \geq t). \tag {60}
$$
where the randomness in the second probability is taken over  $G, H, \eta_1, \dots, \eta_k$  and  $\xi$ .
Proof. Denote  $S_r = \{(w, u, \theta) \in \mathbb{R}^d \times \mathbb{R}^n \times \Theta : w = P\Sigma^{1/2}w(\theta)$  and  $\|w\|_2 + \|u\|_2 + \|r\|_2 \leq r\}$ . The set  $S_r$  is bounded by definition and closed by the continuity of  $w$ . Hence, it is compact. Next, we denote the truncated problems:
$$
\Phi_ {r} := \sup  _ {(w, u, \theta) \in \mathcal {S} _ {r}} \inf  _ {\lambda \in \mathbb {R} ^ {n}} \langle \lambda , Z w \rangle + \psi (u, \theta , \lambda | \eta_ {1}, \dots , \eta_ {k}, \xi) \tag {61}
$$
$$
\Phi_ {r, s} := \sup  _ {(w, u, \theta) \in \mathcal {S} _ {r}} \inf  _ {\| \lambda \| _ {2} \leq s} \langle \lambda , Z w \rangle + \psi (u, \theta , \lambda | \eta_ {1}, \dots , \eta_ {k}, \xi). \tag {62}
$$
By definition, we have  $\Phi_r\leq \Phi_{r,s}$  and so
$$
\Pr (\Phi_ {r} > t) \leq \Pr (\Phi_ {r, s} > t).
$$
The corresponding auxiliary problems are
$$
\begin{array}{l} \Psi_ {r, s} := \sup  _ {(w, u, \theta) \in \mathcal {S} _ {r}} \inf  _ {\| \lambda \| _ {2} \leq s} \| \lambda \| _ {2} \langle H, w \rangle + \| w \| _ {2} \langle G, \lambda \rangle + \psi (u, \theta , \lambda | \eta_ {1}, \dots , \eta_ {k}, \xi) \\ = \sup  _ {(w, u, \theta) \in \mathcal {S} _ {r}} \inf  _ {\| \lambda \| _ {2} \leq s} \| \lambda \| _ {2} \langle H, w \rangle + \langle \lambda , \| w \| _ {2} G + \sum_ {i = 1} ^ {k} \eta_ {i} \langle w (\theta), \Sigma w _ {i} ^ {*} \rangle - u \rangle \\ + F (\theta) - \frac {1}{n} \sum_ {i = 1} ^ {n} f (u _ {i}, g (\eta_ {1, i}, \dots , \eta_ {k, i}, \xi_ {i}), \theta) \\ = \sup  _ {(w, u, \theta) \in \mathcal {S} _ {r}} \inf  _ {0 \leq \lambda \leq s} \lambda \left(\langle H, w \rangle - \left\| \| w \| _ {2} G + \sum_ {i = 1} ^ {k} \eta_ {i} \langle w (\theta), \Sigma w _ {i} ^ {*} \rangle - u \right\| _ {2}\right) \\ + F (\theta) - \frac {1}{n} \sum_ {i = 1} ^ {n} f (u _ {i}, g (\eta_ {1, i}, \dots , \eta_ {k, i}, \xi_ {i}), \theta) \\ \end{array}
$$
and the limit of  $s\to \infty$  ..
$$
\Psi_{r}:= \sup_{\substack{(w,u,\theta)\in \mathcal{S}_{r}\\ \langle H,w\rangle \geq \left\| \| w\|_{2}G + \sum_{i = 1}^{k}\eta_{i}\langle w(\theta),\Sigma w_{i}^{*}\rangle -u\right\|_{2}}}F(\theta) - \frac{1}{n}\sum_{i = 1}^{n}f(u_{i},g(\eta_{1,i},\ldots ,\eta_{k,i},\xi_{i}),\theta)
$$
By definition, it holds that  $\Psi_r \leq \Psi$  and so
$$
\Pr (\Psi_ {r} \geq t) \leq \Pr (\Psi \geq t).
$$
$$
\Pr (\Phi > t) = \lim  _ {r \rightarrow \infty} \Pr (\Phi_ {r} > t)
$$
by Lemma 1
$$
\leq \lim  _ {r \rightarrow \infty} \lim  _ {s \rightarrow \infty} \Pr (\Phi_ {r, s} > t)
$$
$$
\leq 2 \lim _ {r \to \infty} \lim _ {s \to \infty} \Pr (\Psi_ {r, s} \geq t)
$$
$$
= 2 \lim  _ {r \rightarrow \infty} \Pr (\Psi_ {r} \geq t)
$$
by Theorem 8
by Lemma 2
$$
\leq 2 \Pr (\Psi \geq t).
$$