SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We also introduce the following additional notations and definitions: First,  $\mathbb{P}_n$  denotes the (random) uniform measure over  $\mathbf{X} = (X_1,\dots ,X_n)$ . Next, following Bartlett et al. (2005), we define the lower and upper isometry remainders of  $(\mathcal{F},\mathbb{P})$  for a given  $n$ . These remainders measure the discrepancy between  $L_{2}(\mathbb{P})$  and a "typical"  $L_{2}(\mathbb{P}_{n})$ , here "typical" means for most of realizations of  $\mathbf{X}$ . These remainders first emerged in the field of metric embeddings, specifically in the definition of quasi-isometries (cf. Ostrovskii (2013)).
In order to introduce these isometry remainders, we first define for each realization of the input  $\mathbf{X}$ , the constants  $\mathcal{I}_L(\mathbf{X})$  and  $\mathcal{I}_U(\mathbf{X})$  as the minimal numbers  $A_X$ ,  $B_X \geq 0$ , respectively, such that the following holds:
$$
\forall f, g \in \mathcal {F}: 4 ^ {- 1} \int (f - g) ^ {2} d \mathbb {P} - A _ {X} \leq \int (f - g) ^ {2} d \mathbb {P} _ {n} \leq 4 \int (f - g) ^ {2} d \mathbb {P} + B _ {X}.
$$
Note that as  $A_X$  and  $B_X$  increase, the geometry of  $L_2(\mathbb{P}_n)$  and  $L_2(\mathbb{P})$  over  $\mathcal{F}$  becomes less similar. For example, in the extreme case of  $A_X = B_X = 0$ , it implies the  $L_2(\mathbb{P})$  and  $L_2(\mathbb{P}_n)$  induce the same topology over  $\mathcal{F}$ . In words, the lower isometry is the minimal threshold that satisfies the following: all  $f, g \in \mathcal{F}$  that are  $\omega(\mathcal{I}_L(\mathbf{X}))$  far from each other in  $L_2(\mathbb{P})$ , must be at least  $\Omega(\|f - g\|)$  far in  $L_2(\mathbb{P}_n)$ . The upper isometry remainder implies the converse. To provide further intuition on these remainders, for instance, observe that  $\mathcal{I}_L(\mathbf{X})$  upper bounds on diameter in  $L_2(\mathbb{P})$  of possible solutions of ERM; namely, one has
$$
\sup _ {f ^ {*} \in \mathcal {F}, \boldsymbol {\xi} \in \mathbb {R} ^ {n}} \mathrm {D i a m} _ {\mathbb {P}} (\{f \in \mathcal {F}: (f (X _ {1}), \dots , f (X _ {n})) = (\widehat {f} _ {n} (X _ {1}), \dots , \widehat {f} _ {n} (X _ {n})) \}) ^ {2} \leq 4 \cdot \mathcal {I} _ {L} (\mathbf {X}).
$$
Finally, the isometry remainders  $\mathcal{I}_L(n),\mathcal{I}_U(n)$  are defined as the "typical" values of  $\mathcal{I}_L(\mathbf{X}),\mathcal{I}_U(\mathbf{X})$
Definition 2. The lower and upper isometry remainders  $\mathcal{I}_L(n),\mathcal{I}_U(n)$  are defined as the minimal constants  $A_{n},B_{n}\geq 0$  (respectively) such that
$$
\operatorname * {P r} _ {\mathbf {X}} (\mathcal {I} _ {L} (\mathbf {X}) \leq A _ {n}) \geq 1 - n ^ {- 1} a n d \operatorname * {P r} _ {\mathbf {X}} (\mathcal {I} _ {U} (\mathbf {X}) \leq B _ {n}) \geq 1 - n ^ {- 1}.
$$
In the classical regime (van de Geer, 2000), it is considered to be a standard assumption that  $\max \{\mathcal{I}_L(n),\mathcal{I}_U(n)\} \lesssim \epsilon_*^2$ . However, in the high dimensional setting, it may happen that the lower isometry remainder is significantly smaller than the upper isometry remainder, e.g.,  $\mathcal{I}_L(n) \lesssim \epsilon_*^2$  and
$\mathcal{I}_U(n) \gg \epsilon_*^2$  (cf. Liang et al. (2020); Mendelson (2014)). We discuss these remainders further in Remark 10 below.
Finally, we remind the reader that  $\widehat{f}_n$  is uniquely defined on the data points  $\mathbf{X}$  when  $\mathcal{F}$  is a convex closed function class, but it may not be unique over the entire  $\mathcal{X}$  (as multiple functions in  $\mathcal{F}$  may take the same values at  $X_1,\ldots ,X_n$ ). In §2.3.1, the results hold for any possible solution of  $\widehat{f}_n$  over  $\mathcal{X}$ , whereas in §2.3.2, we (implicitly) assume that  $\widehat{f}_n$  is equipped with a selection rule such that it is also unique over the entire  $\mathcal{X}$  (e.g., choosing the minimal norm solution (Hastie et al., 2022; Bartlett et al., 2020)); i.e,  $\widehat{f}_n:\mathcal{D}\to \mathcal{F}$ .
2.3.1 The empirical processes approach
Here, we assume that the function class and the noise are uniformly bounded.
Assumption 4. There exist universal constants  $\Gamma_1, \Gamma_2 > 0$  such that  $\mathcal{F}$  is uniformly upper-bounded by  $\Gamma_1$ , i.e.  $\sup_{f \in \mathcal{F}} \|f\|_{\infty} \leq \Gamma_1$ ; and the components of  $\pmb{\xi} = (\xi_1, \dots, \xi_n)$  are i.i.d. zero mean with variance one and are almost surely bounded by  $\Gamma_2$ .
Remark 1. The uniform boundedness assumption on the noise is taken to simplify the proof (which uses of Talagrand's inequality). This can be relaxed to the assumption that the noise is i.i.d. sub-Gaussian, at the price of a multiplicative factor of  $O(\log n)$  in the error term in Theorem 3 below.
Definition 3. Set  $\epsilon_U\coloneqq \max \{\epsilon_*,\tilde{\epsilon}\}$ , where  $\tilde{\epsilon}$  is the solution of
$$
\mathcal {I} _ {U} (n) \cdot \log \mathcal {N} (\epsilon , \mathcal {F}, \mathbb {P}) \asymp n \epsilon^ {4}. \tag {11}
$$
Note that when  $\mathcal{I}_U(n) \lesssim \epsilon_*^2$ ,  $\epsilon_U \asymp \epsilon_*$ , while if  $\mathcal{I}_U(n, \mathbb{P}) \gg \epsilon_*^2$  then  $\epsilon_U \gg \epsilon_*$ . The following is our main result in this approach to the random design setting:
Theorem 3. Set  $\epsilon_V^2 \coloneqq \max \{\epsilon_U^2, \mathcal{I}_L(n)\}$ , then under Assumptions 1,3,4 the following holds with probability of at least  $1 - n^{-1}$ :
$$
\sup _ {f \in \mathcal {O} _ {\delta_ {n}}} \int (f - \mathbb {E} _ {\mathcal {D}} \widehat {f} _ {n}) ^ {2} d \mathbb {P} \lesssim \epsilon_ {V} ^ {2},
$$
where  $\delta_n = O(\epsilon_V^2)$ ; and in particular  $\mathcal{V}(\widehat{f}_n, \mathcal{F}, \mathbb{P}) \lesssim \epsilon_V^2$ .
Theorem 3 is a generalization of Theorem 1 to the random design case, and its proof uses the strong convexity of the loss and Talagrand's inequality. In §5.1 below, we discuss this bound in the context of "distribution unaware" estimators. We remark that this Theorem extends the scope of Caponnetto and Rakhlin (2006) to non-Donsker classes.
An immediate and useful corollary of this result is that if we have sufficient control of the upper and lower isometry remainders, the variance will be minimax optimal:
Corollary 1. Under Assumptions 1,3,4 and  $\max \{\mathcal{I}_L(n),\mathcal{I}_U(n)\} \lesssim \epsilon_*^2$  , the following holds:
$$
\mathcal {V} (\widehat {f} _ {n}, \mathcal {F}, \mathbb {P}) \lesssim \epsilon_ {*} ^ {2}.
$$
In the classical regime, the assumption that  $\max \{\mathcal{I}_L(n),\mathcal{I}_U(n)\} \lesssim \epsilon_*^2$  is considered to be standard in the empirical process and shape constraints literature (cf. van de Geer (2000) and references therein); it holds for many classical models (see Remark 11 below).
Remark 2. Note that Corollary 1 may also be derived directly from Theorem 1 if the noise is assumed to be standard Gaussian. Yet, this corollary holds for any isotropic sub-Gaussian noise – which is significantly more general.
2.3.2 The isoperimetry approach
In order to motivate this part, we point out that just requiring that  $\mathcal{I}_L(n) \lesssim \epsilon_*^2$  is considered to be a mild assumption (see Remark 10 below). However, the upper bound of Theorem 3 depends on the upper isometry remainder as well; we would like to find some conditions under which this dependency can be removed. Moreover, note that the isometry remainders are connected to the geometry of
$(\mathcal{F}, \mathbb{P})$  and not directly to the stability properties of the estimator. Using a different approach, based on isoperimetry, we will upper-bound the variance of ERM based on some "interpretable" stability parameters of the estimator itself. These stability parameters will be data-dependent relatives of the lower isometry remainder. Differently from the previous part, we do not assume that the function class  $\mathcal{F}$  is uniformly bounded by a constant independent of the sample size  $n$ .
First, we introduce the definition of Lipschitz Concentration Property (LCP):
Definition 4. Let  $\mathbf{Z} = (Z_1, \ldots, Z_m)$  be a random vector taking values in  $\mathcal{Z}^{\otimes m}$ .  $\mathbf{Z}$  satisfies the LCP with constant  $c_L > 0$ , with respect to a metric  $d: (\mathcal{Z}^{\otimes m}, \mathcal{Z}^{\otimes m}) \to \mathbb{R}^+$ , if for all  $F: \mathcal{Z}^m \to \mathbb{R}$  is 1-Lipschitz, the following holds:
$$
\Pr \left(| F (\mathbf {Z}) - \mathbb {E} F (\mathbf {Z}) | \geq t\right) \leq 2 \exp \left(- c _ {L} t ^ {2}\right). \tag {12}
$$
The LCP property is also known as the isoperimetry condition (cf. (Bubeck and Sellke, 2023, §1.3)), and it is stronger than being sub-Gaussian (Boucheron et al., 2013), and yet it is significantly less restrictive than requiring normal noise (in which case  $c_{L} = 1 / 2$  (Ledoux, 2001)); for further details see Remark 9 below. Now, we state our first assumption:
Assumption 5.  $\xi$  is an isotropic random vector satisfying (12) with constant  $c_{L} = \Theta(1)$ , with respect to the Euclidean norm in  $\mathbb{R}^n$ .
Recall that  $\epsilon_{*}$  is defined as the stationary point of  $n\epsilon^{2} \asymp \log \mathcal{N}(\epsilon, \mathcal{F}, \mathbb{P})$ , and that the conditional variance of  $\widehat{f}_n$ , which is a function of the realization  $\mathbf{X}$  of the input, is defined as
$$
V \left(\widehat {f} _ {n} | \mathbf {X}\right) := \mathbb {E} _ {\boldsymbol {\xi}} \left[ \| \widehat {f} _ {n} - \mathbb {E} _ {\boldsymbol {\xi}} \left[ \widehat {f} _ {n} | \mathbf {X} \right] \| ^ {2} \right];
$$
that is, we fix the data points  $\mathbf{X}$  and take expectation over the noise.
The formulation of the following definition involves a yet-to-be-defined (large) absolute constant  $M > 0$ , which will be specified in the proof of Theorem 4 (see §3.2 below).
Definition 5. For each realization  $\mathbf{X}$  and  $f^{*} \in \mathcal{F}$ , let  $\rho_{S}(\mathbf{X}, f^{*})$  be defined as the minimal constant  $\delta(n)$  such that
$$
\Pr_ {\boldsymbol {\xi}} \left\{\boldsymbol {\xi} \in \mathbb {R} ^ {n}: \forall \boldsymbol {\xi} ^ {\prime} \in B _ {n} (\boldsymbol {\xi}, M \epsilon_ {*}): \| \widehat {f} _ {n} (\mathbf {X}, \boldsymbol {\xi} ^ {\prime}) - \widehat {f} _ {n} (\mathbf {X}, \boldsymbol {\xi}) \| ^ {2} \leq \delta (n) \right\} \geq \exp \left(- c _ {2} n \epsilon_ {*} ^ {2}\right). \tag {13}
$$
where  $B_{n}(\pmb {\xi},r) = \{\pmb{\xi}^{\prime}\in \mathbb{R}^{n}:||\pmb {\xi} - \pmb{\xi}^{\prime}||_{n}\leq r\}$ , and  $c_{2} > 0$  is an absolute constant.
We also set  $\rho_{S}(\mathbf{X})\coloneqq \sup_{f^{*}\in \mathcal{F}}\rho_{S}(\mathbf{X},f^{*})$ $\rho_S(\mathbf{X})$  measures the optimal radius of stability (or "robustness") of  $\hat{f}_n$  to perturbations of the noise when the underlying function and data points  $\mathbf{X}$  are fixed. This is a weaker notion than the lower isometry remainder; in fact, one can verify that  $\rho_S(\mathbf{X})\lesssim \max \{\mathcal{I}_L(\mathbf{X}),\epsilon_*^2\}$  for every realization  $\mathbf{X}$  (see Lemma 12 for completeness). Now, we are ready to present our first theorem:
Theorem 4. Under Assumptions 1,3,5, the following holds for every realization  $\mathbf{X}$  of the data:
$$
V _ {\mathcal {D}} (\widehat {f} _ {n} | \mathbf {X}) \lesssim \max  \left\{\rho_ {S} (\mathbf {X}, f ^ {*}), \epsilon_ {*} ^ {2} \right\},
$$
and in particular  $\sup_{f^{*}\in \mathcal{F}}\mathbb{E}_{\mathbf{X}}V_{\mathcal{D}}(\widehat{f}_n|\mathbf{X})\lesssim \max \{\mathcal{I}_L(n),c_*^2\} .$
Note that if  $\mathcal{I}_L(n) \lesssim \epsilon_*^2 - a$  very mild assumption – then we obtain that the expected conditional variance is minimax optimal. However, we believe that it is impossible to bound the total variance via the lower isometry remainder alone. Intuitively,  $\widehat{f}_n$  only observes a given realization  $\mathbf{X}$ , and in general, the geometry of  $\mathcal{F}$  may "look different" under different realizations if  $\mathcal{I}_L(n)$  is large (see the discussion in §5.1 for further details).
In our next result, we identify a model under which we can bound the total variance of  $\widehat{f}_n$  by the lower isometry remainder. To state the next assumption, we fix a metric  $d:\mathcal{X}\times \mathcal{X}\to \mathbb{R}^+$  on  $\mathcal{X}$ , and denote by  $d_{n}$  the metric on  $\mathcal{X}^n$  given by  $d_{n}(\mathbf{X},\bar{\mathbf{X}}^{\prime})^{2} = \sum d(X_{i},X_{i}^{\prime})^{2}$ .
Assumption 6.  $\mathbf{X} \sim \mathbb{P}^{\otimes n}$  satisfies (12) with respect to the metric  $d_n(\cdot, \cdot)$ , and with constant  $c_X$  that only depends on  $\mathbb{P}$
Note that it is insufficient to assume that  $X \sim \mathbb{P}$  satisfies an LCP, since this does not imply that  $\mathbf{X}$  satisfies an LCP with a constant independent of  $n$  (w.r.t. to  $d_n(\cdot, \cdot)$ ). However, if  $X \sim \mathbb{P}$  satisfies a concentration inequality which tensorizes "nicely," such as a log-Sobolev or  $W_2$ -transportation cost inequality (cf. (Ledoux, 2001, §5.2, §6.2)), then  $\mathbf{X} \sim \mathbb{P}^{\otimes n}$  does satisfy this LCP property.
Next, we assume that with high probability,  $\widehat{f}_n$  is close to interpolating the observations:
Assumption 7. There exist absolute constants  $c_{I}, C_{I} > 0$ , such that the following holds:
$$
\operatorname * {P r} _ {\mathcal {D}} \left(\sum_ {i = 1} ^ {n} (\widehat {f} _ {n} (X _ {i}) - Y _ {i}) ^ {2} \leq C _ {I} \epsilon_ {*} ^ {2} \cdot n\right) \geq 1 - \exp (- c _ {I} n \epsilon_ {*} ^ {2}).
$$
This assumption is quite common in the study of "rich" high dimensional models (i.e. when the function class  $\mathcal{F}$  depends on  $n$ ; see, e.g., Belkin et al. (2019); Liang and Rakhlin (2020)) which are prominent in the recent benign overfitting literature.
Finally, we introduce another stability notion. Recall the random set  $\mathcal{O}_{\delta} \subset \mathcal{F}$  of almost-minimizers of the empirical loss, as defined in (5) above; note that, in the random design setting,  $\mathcal{O}_{\delta}$  depends on both  $\mathbf{X}$  and  $\pmb{\xi}$ . The random variable  $\mathrm{diam}_{\mathbb{P}}(\mathcal{O}_{\delta})$  can be thought of as measuring the stability of the ERM with respect to imprecision in the minimization algorithm (cf. (Caponnetto and Rakhlin, 2006)). The formulation of the following definition involves another yet-to-be-defined (large) absolute constant  $M' > 0$ , which will be specified in the proof of Theorem 5 (see §6.6 below), as well as the constant  $c_{I}$  from Assumption 7.
Definition 6.  $\rho_{\mathcal{O}}(n,\mathbb{P},f^{*})$  is defined as the smallest  $\delta (n)\geq 0$  such that
$$
\Pr_ {\mathcal {D}} \left(\operatorname {d i a m} _ {\mathbb {P}} \left(\mathcal {O} _ {M ^ {\prime} \epsilon_ {*} ^ {2}}\right) \leq \sqrt {\delta (n)}\right) \geq 2 \exp \left(- c _ {I} n \epsilon_ {*} ^ {2}\right), \tag {14}
$$
where  $c_{I} \geq 0$  is the same absolute constant defined in Assumption 7.
In order to understand the relation between this and the previous stability notions, note that under Assumption 7 and the event  $\mathcal{E}$  of Definition 6, we have that on an event of nonnegligible probability,  $\rho_S(\mathbf{X},f^*)\leq \rho_{\mathcal{O}}(n,\mathbb{P},f^*)$ ; in addition,  $\rho_{\mathcal{O}}(n,\mathbb{P},f^{*})\lesssim \max \{\mathcal{I}_{L}(n),\epsilon_{*}^{2}\}$  (see Lemma 13 below). Under these additional two assumptions and the last definition, we state our bound for the total variance of  $\widehat{f}_n$ :
Theorem 5. Under Assumptions 1,3,5-7, the following holds:
$$
V _ {\mathcal {D}} (\widehat {f} _ {n}) \lesssim c _ {X} ^ {- 1} \cdot \sup  _ {f ^ {*} \in \mathcal {F}} \| f ^ {*} \| _ {L i p} \cdot \max  \{\epsilon_ {*} ^ {2}, \rho_ {\mathcal {O}} (n, \mathbb {P}, f ^ {*}) \},
$$
and in particular one has  $\mathcal{V}(\widehat{f}_n,\mathcal{F},\mathbb{P})\lesssim c_X^{-1}\cdot \sup_{f^*\in \mathcal{F}}\| f^*\|_{Lip}\cdot \max \{\epsilon_*^2,\mathcal{I}_L(n)\} .$
Note that when  $\mathcal{F}$  is a robust learning architecture (i.e.  $\mathcal{F} \subset \{\mathcal{X} \to \mathbb{R} : \|f\|_{Lip} = O(1)\}$ ), our bound is optimal. Interestingly, the assumptions of Theorem 5 coincide with those of the model considered in the recent paper of Bubeck and Sellke (2023). Also note that the last theorem connects the total variance of  $\widehat{f}_n$  to a "probabilistic" threshold for the  $L_2(\mathbb{P})$ -diameter of the data-dependent set of  $\Theta(\epsilon_*^2)$ -approximating solutions of  $\widehat{f}_n$  - two parameters which at first sight are unrelated.
Remark 3. One may suspect that the assumptions of almost interpolation and robustness are incompatible, which would render our theorem vacuous. However, perhaps counter-intuitively, in the high-dimensional setting these assumptions can coexist. For example, interpolation with  $O(1)$ -Lipschitz functions may be possible when the "intrinsic" dimension of  $\mathcal{X}$  is  $\Omega(\log(n))$  (depending on the richness of  $\mathcal{F}$ ), though it is generally impossible when the dimension is  $o(\log(n))$  (this follows from the behaviour of the entropy numbers of the class of Lipschitz functions; cf. Dudley (1999)).
Remark 4. Using Assumptions 1,3,5-6, one may prove the same bound as in Theorem 3, i.e. that  $\mathcal{V}(\widehat{f}_n,\mathcal{F},\mathbb{P})\lesssim \epsilon_V^2$  , without requiring the noise or the function class to be uniformly bounded. The idea is to obtain the crucial concentration bounds in the proof of Theorem 3 by using the LCP properties of  $\pmb{\xi}$  and  $X\sim \mathbb{P}$  along with the robustness of  $\mathcal{F}$ , rather than via Talagrand's inequality.
3 Proof sketches
In this section, we sketch the proofs of less-technical results to give the reader a flavor of our methods. The full proofs are given in the next section. For the proofs we introduce some additional notations. For  $m \in \mathbb{N}$ , we set  $[m] := \{1, \dots, m\}$ . The inner products in  $L_2(\mathbb{P}), L_2(\mathbb{P}^{(n)})$  are denoted by  $\langle \cdot, \cdot \rangle, \langle \cdot, \cdot \rangle_n$ , respectively.
3.1 Sketch of proof of Theorem 1
Here, we sketch a simple proof of a weaker version of our result, namely  $V_{\mathcal{D}}(\hat{f}_n) \lesssim \mathcal{M}(\mathcal{F}, \mathbb{P}^{(n)})$ , under the stronger assumption that
$$
\mathcal {M} (\mathcal {F}, \mathbb {P} ^ {(n)}) \asymp \epsilon_ {*} ^ {2}, \tag {15}
$$
where  $\epsilon_{*}$  solves  $\log \mathcal{N}(\epsilon, \mathcal{F}, \mathbb{P}^{(n)}) \asymp n\epsilon^2$ . (This holds under reasonable assumptions on  $\mathcal{F}$ , but can be dispensed with; see Lemma 1 for the exact characterization.) In §6.1.1, we fill in the details of this sketch, and in §6.1.2, we give the full proof of Theorem 1.
The proof uses the probabilistic method (Alon and Spencer, 2016). Let  $f_{1},\ldots ,f_{N}$  be centers of a minimal  $\epsilon_*$ -cover of  $\mathcal{F}$  with respect to  $L_{2}(\mathbb{P}^{(n)})$ , per Definition 1. First, since for any  $i\in [N]$ , the map  $\xi \mapsto \sqrt{n}\| \widehat{f}_n(\pmb {\xi}) - f_i\| _n$  is 1-Lipschitz, (12) and a union bound ensures that with probability at least  $1 - \frac{1}{2N}$ , for all  $i\in [N]$ ,
$$
\mathbb {E} _ {\boldsymbol {\xi}} \| \widehat {f} _ {n} - f _ {i} \| _ {n} - \| \widehat {f} _ {n} - f _ {i} \| _ {n} \lesssim \sqrt {\frac {\log N}{n}}.
$$
On the other hand, by the pigeonhole principle, there exists at least one  $i^{*} \in [N]$  such that with probability at least  $1 / N$ ,  $\| \widehat{f}_n - f_{i^*}\| _n \leq \epsilon_*$ . Hence, there exists at least one realization of  $\pmb{\xi} \in \mathbb{R}^n$  for which both bounds hold, and thus, deterministically,
$$
\mathbb {E} _ {\boldsymbol {\xi}} \left\| \widehat {f} _ {n} - f _ {i ^ {*}} \right\| _ {n} \lesssim \epsilon_ {*} + \sqrt {\frac {\log N}{n}} \lesssim \epsilon_ {*}
$$
where we used the balancing equation (10). Another application of (12) and integration of tails yields
$$
V (\widehat {f} _ {n}) = \mathbb {E} _ {\mathcal {D}} \| \widehat {f} _ {n} - \mathbb {E} _ {\mathcal {D}} \widehat {f} _ {n} \| _ {n} ^ {2} \leq \mathbb {E} _ {\mathcal {D}} \| \widehat {f} _ {n} - f _ {i ^ {*}} \| _ {n} ^ {2} \lesssim \epsilon_ {*} ^ {2},
$$
implying that the variance of ERM is minimax optimal.
3.2 Sketch of proof of Theorem 4
As is well-known, the Lipschitz concentration condition (12) is equivalent to an isoperimetric phenomenon: for any set  $A \subset \mathbb{R}^n$  with  $\operatorname*{Pr}_{\pmb{\xi}}(A) \geq 1/2$ , its  $t$ -neighborhood  $A_t = \{\pmb{\xi} \in \mathbb{R}^n : \inf_{x \in A} \|x - \pmb{\xi}\|_n \leq t\}$  satisfies
$$
\Pr_ {\xi} \left(A _ {t}\right) \geq 1 - 2 \exp \left(- n t ^ {2} / 2\right). \tag {16}
$$
One sees quickly that this implies that if  $A$  has measure at least  $2\exp (-nt^2 /2)$ , then  $A_{2t}$  has measure  $1 - 2\exp (-nt^{2} / 2)$ .
Let  $\mathcal{E}$  be the event of Definition 5. As in §3.1 above, one obtains via the pigeonhole principle and the definition of  $\epsilon_{*}$  that there exists some  $f_{c} \in \mathcal{F}$  such that
$$
\operatorname * {P r} _ {\boldsymbol {\xi}} (\underbrace {\{\widehat {f} _ {n} \in B (f _ {c} , \epsilon_ {*}) \} \cap \mathcal {E}} _ {A} | \mathbf {X}) \geq \frac {\operatorname* {P r} _ {\boldsymbol {\xi}} (\mathcal {E})}{\mathcal {N} (\epsilon_ {*} , \mathcal {F} , \mathbb {P})} \geq \exp (- C _ {3} n \epsilon_ {*} ^ {2}).
$$
By isoperimetry,  $\operatorname{Pr}_{\pmb{\xi}}(A_{2t}) \geq 1 - 2\exp(-nt^2/2)$ , where  $t = M\epsilon_*/2$  and  $M$  is chosen such that  $(M/2)^2 \geq 2C_3$ ; this fixes the value of the absolute constant  $M$  used in (13).
Applying (13) yields that if  $\pmb {\xi}\in A\subset \mathcal{E}$  and  $\| \pmb{\xi}^{\prime} - \pmb {\xi}\|_{n}\leq M\epsilon_{*} = 2t$ $\| \widehat{f}_n(\pmb {\xi}) - \widehat{f}_n(\pmb {\xi}')\| \leq \rho_S(\mathbf{X},f^*)$  and so  $\| \widehat{f}_n(\pmb {\xi}'') - f_c\| \leq \epsilon_* + \rho_S(\mathbf{X},f^*)$ . This implies
$$
\operatorname * {P r} _ {\boldsymbol {\xi}} (\{\widehat {f _ {n}} \in B (f _ {c}, \epsilon_ {*} + \rho_ {S} (\mathbf {X}, f ^ {*})) \} | \mathbf {X}) \geq \operatorname * {P r} _ {\boldsymbol {\xi}} (A _ {2 t} | \mathbf {X}) \geq 1 - 2 \exp (- n t ^ {2} / 2),
$$
which implies via conditional expectation that  $V(\widehat{f_n} | \mathbf{X}) \lesssim \max \{\rho_S(\mathbf{X}), \epsilon_*^2\}$ , as desired (where we used that  $\epsilon_*^2 \gtrsim \log(n)/n$  (see Lemma 3 below), and therefore  $\exp(-nt^2) = O(\epsilon_*^2)$ ).
Acknowledgements: This work was supported by the Simons Foundation through Award 814639 for the Collaboration on the Theoretical Foundations of Deep Learning, the ERC under the European Union's Horizon 2020 research and innovation programme (grant agreement No 770127), and the NSF (awards DMS-2031883, DMS-1953181). Part of this work was carried out while the first two authors were in residence at the Institute for Computational and Experimental Research in Mathematics in Providence, RI, during the Harmonic Analysis and Convexity program; this residency was supported by the NSF (grant DMS-1929284). Finally, the first two authors also wish to acknowledge Prof. Shiri Artstein-Avidan for introducing them to each other.