SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Notably, the  $\Gamma$ -limit of  $\mathcal{R}_{\lambda}$  itself would be zero at all points of interest. Rescaling to consider  $F_{\lambda}$  instead has fits into the framework of  $\Gamma$ -expansions considered by Braides and Truskinovsky [2008]. Denote
$$
\mathcal {F} = \{f \in \mathcal {B}: f \equiv f ^ {*} \mu - a. e. \} \tag {9}
$$
Proof. Step 1. liminf-inequality. First consider  $f \in \mathcal{F}$  and assume that  $\{f_{\lambda}\}_{\lambda >0}$  is a family of functions such that  $f_{\lambda} \xrightarrow{\text{good}} f$ . Then by the compactness theorem for Barron functions in coarser topologies (Theorem C.3) we have the following:
$$
\liminf_{\lambda \to 0^{+}}F_{\lambda}(f_{\lambda})\geq \liminf_{\lambda \to 0^{+}}[f_{\lambda}]_{\mathcal{B}}\geq [f]_{\mathcal{B}} = F(f).
$$
Now assume that  $f \notin \mathcal{F}$  and that  $f_{\lambda} \xrightarrow{\text{good}} f$ . We need to show that  $F_{\lambda}(f_{\lambda}) \to +\infty$ . Since  $f \notin \mathcal{F}$ , we see that  $\mathcal{R}(f) = \| f - f_d^*\|_{L^2 (\mu)}^2 > 0$ . Denote  $\varepsilon = \sqrt{\mathcal{R}(f)}$  and observe that there exists  $\Lambda > 0$  such that  $\| f_{\lambda} - f\|_{L^2 (\mu)} < \varepsilon / 2$  for all  $\lambda < \Lambda$  by the definition of the notion of convergence.
In particular, we find that
$$
\left\| f _ {\lambda} - f ^ {*} \right\| _ {L ^ {2} (\mu)} \geq \left\| f - f ^ {*} \right\| _ {L ^ {2} (\mu)} - \left\| f _ {\lambda} - f \right\| _ {L ^ {2} (\mu)} \geq \varepsilon / 2
$$
for all  $\lambda <  \Lambda$  by the inverse triangle inequality and thus
$$
\liminf_{\lambda \to 0^{+}}F_{\lambda}(f_{\lambda})\geq \liminf_{\lambda \to 0^{+}}\frac{(\varepsilon / 2)^{2}}{\lambda} = +\infty .
$$
2 It is easy to generalize this to continuous limits, but if preferred, then  $\lambda = \lambda_{n}$  can be taken to be a discrete sequence converging to zero.
Step 2. limsup-inequality. Again, we first consider the case  $f \in \mathcal{F}$ . Set  $f_{\lambda} = f$  for all  $\lambda > 0$  and observe that  $f_{\lambda} \to f$  as  $\lambda \to 0^{+}$  (trivially). By the same argument  $F_{\lambda}(f_{\lambda}) = F(f) = [f]_{\mathcal{B}}$  for all  $\lambda$ , i.e. the constant sequence is a recovery sequence since  $F_{\lambda}(f_{\lambda}) \to F(f)$ .
On the other hand, if  $f \notin \mathcal{F}$ , then  $\mathcal{R}(f) > 0$  and thus  $F_{\lambda}(f) \to +\infty = F(f)$ . Again, we can use the constant sequence as a recovery sequence, somewhat trivially.
Corollary E.4. Assume that  $f_{\lambda} \in \operatorname{argmin}_{f \in \mathcal{B}} \mathcal{R}_{\lambda}$ , i.e.  $f_{\lambda}$  minimizes  $\mathcal{R}_{\lambda}$ . Then there exists  $\hat{f} \in \mathcal{B}$  such that  $f_{\lambda} \xrightarrow{\text{good}} \hat{f}$ .
Proof. Clearly  $f_{\lambda}$  minimizes  $\mathcal{R}_{\lambda}$  if and only if it minimizes  $F_{\lambda} = \lambda^{-1}\mathcal{R}_{\lambda}$ . We note that
$$
[ f _ {\lambda} ] _ {\mathcal {B}} \leq \lambda^ {- 1} \mathcal {R} _ {\lambda} (f _ {\lambda}) \leq \lambda^ {- 1} \mathcal {R} (f ^ {*}) + [ f ^ {*} ] _ {\mathcal {B}} = [ f ^ {*} ] _ {\mathcal {B}}.
$$
In particular, by the compact embedding of Theorem C.3, there exists  $\hat{f} \in \mathcal{B}$  such that  $f_{\lambda} \xrightarrow{good} \hat{f}$  up to subsequence. By the properties of  $\Gamma$ -convergence, we conclude that  $\hat{f}$  is a minimizer of  $F$ .
We present a special case of Corollary E.4 in the setting of Proposition 3.2 which exploits the uniqueness of the minimizer. Recall the definition of the radial average in (3) and  $f_{d}^{*}$  from Proposition 3.2.
Corollary E.5. Assume that  $f^{*}(0) = 1$ ,  $f^{*}(x) = 0$  if  $\| x\| \geq 1$  and  $\mu$  satisfies the conditions of Corollary 3.3. Assume additionally that  $f_{\lambda} \in \operatorname{argmin}_{f\in \mathcal{B}}\mathcal{R}_{\lambda}$ , i.e.  $f_{\lambda}$  minimizes  $\mathcal{R}_{\lambda}$ . Then  $\operatorname{Av}f_{\lambda}\xrightarrow{good}f_{d}^{*}$ .
Proof. Step 1. Clearly  $f_{\lambda}$  minimizes  $\mathcal{R}_{\lambda}$  if and only if it minimizes  $F_{\lambda} = \lambda^{-1}\mathcal{R}_{\lambda}$ .  $\mathrm{Av}f_{\lambda}$  is also a minimizer of  $F_{\lambda}$  since the functional is convex and rotationally symmetric, so by averaging in radial direction, we are taking a (continuous) convex combination of minimizers, which is a minimizer again.
Step 2. We find that
$$
[ \mathrm {A v} f _ {\lambda} ] _ {\mathcal {B}} \leq F _ {\lambda} (\mathrm {A v} f _ {\lambda}) \leq F _ {\lambda} (f _ {\lambda}) \leq F _ {\lambda} \left(f _ {d} ^ {*}\right) = \left[ f _ {d} ^ {*} \right] _ {\mathcal {B}}.
$$
By the compactness theorem for Barron functions, Theorem C.3, there exists  $f \in \mathcal{B}$  such that  $\operatorname{Av} f_{\lambda} \xrightarrow{good} f$  (up to a subsequence). Since  $F_{\lambda} \to F$  in the sense of  $\Gamma$ -convergence, we find that  $f$  is a minimizer of  $F$ . Since  $f$  is also radially symmetric, we find by Proposition 3.2 that  $f \equiv f_{d}^{*}$ .
Step 3. By the exact same logic, we could show that every subsequence of  $\{\mathrm{Av}f_{\lambda}\}$  has a further subsequence which converges to  $f_{d}^{*}$ . By a standard argument in topology, the whole sequence converges.
A similar statement can be proved in the more complicated case where  $f_{\lambda}$  is a neural network with finitely many neurons and  $F_{\lambda}$  uses a finite data set rather than a continuous expectation. In this case, the parameter  $\lambda$  must be coupled to the number of parameters  $m$  and the number of data points  $n$ , such that  $\lambda \rightarrow 0$ , but not too quickly. The proof is a more technically challenging variant of those of Theorem E.3 and Corollaries E.4 and E.5, which utilizes the generalization bound of D.4. To this end, we first introduce a new notion of convergence. That is, we define a notion of convergence from the parameter to function. We define a notion of convergence by saying that  $\theta_{k} := (a_{k}, W_{k}, b_{k}) \xrightarrow{good} f$  iff  $f_{\theta_{k}} \xrightarrow{good} f$  as  $k \to \infty$ .
Theorem E.6. Consider the parameter space  $\Theta_{m} \subseteq \mathbb{R}^{m} \times \mathbb{R}^{m \times d} \times \mathbb{R}^{m + 1}$  of neural networks with a single hidden layer of width  $m$  and the associated functions
$$
f _ {\theta} (x) := b _ {0} + \sum_ {i = 1} ^ {m} a _ {i} \sigma \left(w _ {i} \cdot x + b _ {i}\right)
$$
Let  $m_{n},\lambda_{n}$  scale with  $n$  according to (1). We denote
$$
F _ {n}: \Theta_ {m _ {n}} \rightarrow [ 0, \infty) \quad F _ {n} (\theta) = \frac {\widehat {\mathcal {R}} _ {n , m _ {n} , \lambda_ {n}} (\theta)}{\lambda_ {n}} = \frac {\widehat {\mathcal {R}} _ {n} (f _ {\theta})}{\lambda_ {n}} + R _ {W D} (\theta)
$$
Then almost surely over the choice of data points, we have  $\Gamma - \lim_{n \to \infty} F_n = F$  almost surely with respect to the notion of convergence  $\theta_k \xrightarrow{good} f$  defined above.
Proof. We use  $\mathcal{F}$  as in (9) throughout. In the proof we assume that all stochastic quantities in Corollary D.4 and Theorem E.1 are satisfied with probability at least  $1 - \delta_{n}$  for  $\delta_{n} = n^{-2}$ . The quantity  $\log (\delta_n)$  therefore becomes comparable to  $\log n / n\ll \lambda_{n}$ . Since  $\sum_{n = 1}^{\infty}n^{-2} < \infty$ , we find that all conditions are met for all but finitely many  $n\in \mathbb{N}$  by the Borel-Cantelli Lemma. For questions of asymptotic convergence, we may therefore assume that the statements of both Theorems apply without qualifying for high probability. Note that  $n^{-2}\geq e^{-n}$  for all  $n\geq 2$  as needed for Theorem E.1.
Step 1. liminf-inequality. Again, we consider the cases  $f \in \mathcal{F}$  and  $f \notin \mathcal{F}$  separately. First, when  $f \in \mathcal{F}$ , we apply the same method we did in Theorem E.3. For any sequence of parameters  $\theta_{n} \xrightarrow{\text{good}} f$ , by Theorem C.3 the following holds:
$$
\liminf_{n\to \infty}F_{n}(\theta_{n})\geq \liminf_{n\to \infty}R_{W D}(\theta_{n})\geq \liminf_{n\to \infty}[f_{\theta_{n}}]_{\mathcal{B}}\geq [f]_{\mathcal{B}} = F(f).
$$
The second inequality comes from  $[f_{\theta_n}]_{\mathcal{B}}$  being an infimum of weight decay regularizers with any arbitrary probability measure on parameter space.
Second, when  $f \notin \mathcal{F}$ , we need to show  $\lim \inf_{n \to \infty} F_n(\theta_n) = \infty$  for any  $\theta_n \xrightarrow{good} f$ . We distinguish two prototypical cases:
1.  $[f_{\theta_n}]_{\mathcal{B}} \to +\infty$  as  $n \to \infty$ . In this case  $F(\theta_n) \geq [f_{\theta_n}]_{\mathcal{B}} \to +\infty$  as well by the same logic as above.
2.  $\lim \sup_{n\to \infty}[f_{\theta_n}]_{\mathcal{B}} < + \infty$  . In this case, we take  $\varepsilon \coloneqq \| f - f^{*}\|_{L^{2}(\mu)} / 2 > 0$  . Then there exists  $N\in \mathbb{N}$  such that for all  $n\geq N$  we have
$$
\left\| f _ {\theta_ {n}} - f ^ {*} \right\| _ {L ^ {2} (\mu)} \geq \left\| f - f ^ {*} \right\| _ {L ^ {2} (\mu)} - \left\| f _ {\theta_ {n}} - f \right\| _ {L ^ {2} (\mu)} \geq \varepsilon
$$
for all  $n\geq N$  by definition. Additionally
$$
\begin{array}{l} F _ {n} (\theta_ {n}) \geq \frac {\widehat {\mathcal {R}} _ {n} (f _ {\theta_ {n}}) - \mathcal {R} (f _ {\theta_ {n}})}{\lambda_ {n}} + \frac {\mathcal {R} (f _ {\theta_ {n}})}{\lambda_ {n}} + [ f _ {\theta_ {n}} ] _ {\mathcal {B}} \\ \geq \frac {\widehat {\mathcal {R}} _ {n} \left(f _ {\theta_ {n}}\right) - \mathcal {R} \left(f _ {\theta_ {n}}\right)}{\lambda_ {n}} + \frac {\varepsilon^ {2}}{\lambda_ {n}} + \left[ f _ {\theta_ {n}} \right] _ {\mathcal {B}}. \\ \end{array}
$$
Due to Corollary D.4 and the arguments of Theorem E.1 to control the discrepancy at 0, we have
$$
\widehat {\mathcal {R}} _ {n} \left(f _ {\theta_ {n}}\right) - \mathcal {R} \left(f _ {\theta_ {n}}\right) = O \left(\frac {\log n}{\sqrt {n}}\right).
$$
in this case. Since  $\log n / \sqrt{n}\ll \lambda_{n}$  by assumption, we note that
$$
\lim _ {n \to \infty} \frac {\hat {\mathcal {R}} _ {n} (f _ {\hat {\theta} _ {n}}) - \mathcal {R} (f _ {\hat {\theta} _ {n}})}{\lambda_ {n}} = 0
$$
$$
\lim  _ {n \rightarrow \infty} F _ {n} (\theta_ {n}) \geq \lim  _ {n \rightarrow \infty} \inf  \left(0 + \frac {\varepsilon^ {2}}{\lambda_ {n}} + 0\right) = + \infty .
$$
The same holds in the general case by passing to subsequences.
Step 2. limsup-inequality. As in Theorem C.5, the case  $f \notin \mathcal{F}$  follows from the lim inf-inequality in an essentially trivial fashion. We therefore only consider the case  $f \in \mathcal{F}$ . An approximating sequence in this case is constructed from Theorem C.5 as in Theorem E.1 or Theorem E.3. Namely, we find  $\tilde{\theta}_n$  such that
$$
F _ {n} (\tilde {\theta} _ {n}) \leq \frac {C}{\lambda_ {n} m _ {n}} \left(1 + \frac {\log n}{\sqrt {n}}\right) + [ f ^ {*} ] _ {\mathcal {B}} \quad \Rightarrow \quad \lim  _ {n \rightarrow \infty} F _ {n} (\tilde {\theta} _ {n}) \leq [ f ^ {*} ] _ {\mathcal {B}}.
$$
Remark E.7. The key ingredients for the proofs of both Theorem E.1 and Theorem E.6 are Theorem C.5 and Corollary D.4, but they are combined differently. While they are paired in Theorem E.1 to obtain a precise rate, they occur separately in Theorem E.6: Theorem C.5 is used for the lim supinequality while Corollary D.4 enters in the proof of the lim inf-inequality. Analogously, the condition  $\lambda_{n} \ll \log n / \sqrt{n}$  is used in the proof of the lim inf-condition while the fact that  $\frac{1}{m_n} \ll \lambda_n$  is used in the proof of the lim sup-inequality.
E.3 Proofs of the main theorems
The statements of the Theorems in the main body of the text can easily be deduced from the statements proved in this Appendix.
Proof of Theorem 2.1. Convergence in  $L^p$  holds by Theorem E.1 for  $1 \leq p \leq 2$  and Corollary E.2 (general  $p$ ). The proof of uniform convergence follows from Theorem E.6 in the same fashion that Corollary E.4 follows from Theorem E.3. The explicit bound is obtained from Theorem E.1 with  $\delta_n = \frac{1}{4n^2}$ .
Proof of Corollary 3.3. This follows in the same way as the proof of Theorem 2.1 with modifications as in Corollary E.5.
F Theorem 2.1 for finite data sets
Finally, we note that a version of Theorem 2.1 holds if the data set  $S = \{x_{1},\ldots ,x_{n}\}$  is kept fixed. The proof is a combination of those of Theorems E.3 and E.6, as we deal with a finite approximating neural network, but do not require generalization bounds. The details are left to the reader.
Theorem F.1. We make the following assumptions.
1. Let  $S = \{(x_1, y_1), \ldots, (x_n, y_n)\}$  be a fixed dataset of  $n$  data points in  $x_i \in \mathbb{R}^d$  and labels  $y_i \in \mathbb{R}$ .
2. Let the loss function  $\ell(f, y)$  be the mean squared error  $\ell_{MSE}(f, y) = |f - y|^2$ .
3. Assume that  $\lambda_{m}$  is a sequence of parameters such that  $\lambda_{m} \to 0$ ,  $1 / m \ll \lambda_{m}$  as  $m \to \infty$ .
Consider the regularized empirical risk functional  $\widehat{\mathcal{R}}_m:\mathbb{R}^m\times \mathbb{R}^{m\times d}\times \mathbb{R}^m\to [0,\infty)$
$$
\widehat {\mathcal {R}} _ {m} (a, W, b) = \frac {1}{2 n} \sum_ {i = 1} ^ {n} \ell \left(f _ {(a, W, b)} (x _ {i}), y _ {i}\right) + \frac {\lambda_ {m}}{2} \left(\| a \| _ {2} ^ {2} + \| W \| _ {F r o b} ^ {2}\right).
$$
Then if  $(a, W, b)_{m} \in \operatorname{argmin} \widehat{\mathcal{R}}_{m}$  for all  $m \in \mathbb{N}$ , then every subsequence of  $f_{m} \coloneqq f_{(a, W, b)_{m}}$  has a further subsequence which converges to some limit  $\hat{f}^{*} \in \mathcal{B}$  uniformly on compact subset of  $\mathbb{R}^d$ . The limiting function satisfies
$$
\hat{f}^{*}\in \operatorname *{argmin}_{\{f\in \mathcal{B}:f(x_{i}) = y_{i} \forall i\}}[f]_{\mathcal{B}}.
$$
G Minimum norm interpolation in one dimension
Proof of Proposition 3.1. Any Barron function  $f$  is also Lipschitz-continuous, in particular differentiable almost everywhere and  $f(b) - f(a) = \int_{a}^{b}f^{\prime}(x)\mathrm{d}x$  for all  $a,b\in \mathbb{R}$  by Rademacher's Theorem (see e.g. [Evans and Gariepy, 2015, Section 3.1]). In particular, there exist points  $x^{+},x^{-}\in (a,b)$  such that
$$
f ^ {\prime} (x ^ {-}) \leq \frac {f (b) - f (a)}{b - a} = \frac {1}{b - a} \int_ {a} ^ {b} f ^ {\prime} (x) \mathrm {d} x \leq f ^ {\prime} (x ^ {+}).
$$
If  $f'$  is not constant, both inequalities are satisfied strictly. Under the assumptions, there exists  $a \in (x_0, x_1)$  and  $b \in (x_{n-1}, x_n)$  such that
$$
f ^ {\prime} (a) \leq \frac {f \left(x _ {1}\right) - f \left(x _ {0}\right)}{x _ {1} - x _ {0}} \leq 0 \leq \frac {f \left(x _ {n}\right) - f \left(x _ {n - 1}\right)}{x _ {n} - x _ {n - 1}} \leq f ^ {\prime} (b). \tag {10}
$$
Since the derivative  $f'$  changes sign, we conclude by [Wojtowytsch, 2022, Proposition 2.5] that  $[f]_{\mathcal{B}} = \int_{-\infty}^{\infty} \mathrm{d}|\mu|$  where the Radon measure  $\mu$  is the distributional derivative of  $f'$  and  $|\mu|$  denotes the total variation measure of  $\mu$ . Since  $f'$  is differentiable at  $a, b$ , neither point is an atom of  $\mu$  and thus
$$
\frac {f (x _ {n}) - f (x _ {n - 1})}{x _ {n} - x _ {n - 1}} - \frac {f (x _ {1}) - f (x _ {0})}{x _ {1} - x _ {0}} \leq f ^ {\prime} (b) - f ^ {\prime} (a) \leq \int_ {a} ^ {b} \mathrm {d} \mu \leq \int_ {a} ^ {b} \mathrm {d} | \mu | \leq [ f ] _ {\mathcal {B}}.
$$
Equality holds if and only if  $|\mu| = \mu$ , i.e. if  $\mu \geq 0$  in the sense of signed measures and if and only if the inequality in (10) can only be satisfied with equality. The first condition means that  $f$  must be convex, the second implies that the derivative of  $f$  must be constant in the intervals  $(x_0, x_1)$  and  $(x_{n-1}, x_n)$ . The same can easily be seen for the larger intervals  $(-\infty, x_1)$  and  $(x_{n-1}, \infty)$ .
H Sub-Gaussian random variables
In this Appendix we quickly gather some facts about sub-Gaussian random variables. For the reader's convenience, we provide full proofs. Experts are encouraged to skip ahead to Appendix D.
The first property we observe is a bound of expected maximum value of sub-Gaussian.
Lemma H.1. Let  $\mu$  be  $\sigma^2$ -sub-Gaussian, and for  $i = 1, \dots, n$ , let  $x_i$  be a iid random sample from a law  $\mu$ . Then, the following holds:
$$
\mathbb {E} \big [ \max _ {1 \leq i \leq n} \| x _ {i} \| \big ] \leq \mathbb {E} _ {x \sim \mu} \big [ \| x \| \big ] + \sqrt {2 \log n} \sigma .
$$
Proof. From the sub-Gaussian assumption we have
$$
\log \left(\mathbb {E} _ {x \sim \mu} \big [ \exp \left(\lambda \big (\| X \| - \mathbb {E} \big [ \| X \| \big ]\right) \big ]\right) \leq \frac {\sigma^ {2} \lambda^ {2}}{2}
$$
for some fixed  $\sigma > 0$  and all  $\lambda > 0$ . In particular, Jensen's inequality implies the sub-Gaussian maximal inequality
$$
\begin{array}{l} \mathbb {E} \left[ \max  _ {1 \leq i \leq n} \| x _ {i} \| \right] = \mathbb {E} _ {x \sim \mu} \left[ \| x \| \right] + \mathbb {E} \left[ \max  _ {1 \leq i \leq n} \left(\| x _ {i} \| - \mathbb {E} \| x _ {i} \|\right) \right] \\ \left. \right. \leq \mathbb {E} _ {x \sim \mu} \left[ \| x \| \right] + \frac {1}{\lambda} \log \left(\mathbb {E} \left[ \exp \left(\lambda \max  _ {1 \leq i \leq n} \left(\| x _ {i} \| - \mathbb {E} \| x _ {i} \|\right)\right)\right]\right) \\ \leq \mathbb {E} _ {x \sim \mu} [ \| x \| ] + \frac {1}{\lambda} \log \left(\mathbb {E} \left[ \sum_ {i = 1} ^ {n} \exp (\lambda (\| x _ {i} \| - \mathbb {E} \| x _ {i} \|)) \right]\right) \\ \leq \mathbb {E} _ {x \sim \mu} [ \| x \| ] + \frac {1}{\lambda} \log \left(\sum_ {i = 1} ^ {n} \mathbb {E} \left[ \exp \left(\lambda \left(\| x _ {i} \| - \mathbb {E} \| x _ {i} \|\right)\right) \right]\right) \\ = \mathbb {E} _ {x \sim \mu} [ \| x \| ] + \frac {1}{\lambda} \log \left(n \exp \left(\frac {\lambda^ {2} \sigma^ {2}}{2}\right)\right) \\ \leq \mathbb {E} _ {x \sim \mu} \left[ \| x \| \right] + \frac {\log n}{\lambda} + \frac {\sigma^ {2}}{2} \lambda \\ \end{array}
$$
for all  $\lambda > 0$ . We specifically select  $\lambda = \sqrt{\frac{2\log n}{\sigma^2}}$ , making the bound
Next, we observe the concentration of maximum values among samples of sub-Gaussian distribution.
Lemma H.2. Let  $\mu$  be  $\sigma^2$ -sub-Gaussian, and for  $i = 1, \dots, n$ , let  $x_i$  be a iid random sample from a law  $\mu$ . Then, with probability at least  $1 - \delta$ , the following holds:
$$
\max  _ {1 \leq i \leq n} \| x _ {i} \| \leq \mathbb {E} _ {x \sim \mu} \left[ \| x \| \right] + \sigma \sqrt {2 \log (n / \delta)}.
$$
Proof. For all  $t > 0$ , we observe that
$$
\begin{array}{l} \mu^ {n} \left(\max  _ {1 \leq i \leq n} \| x _ {i} \| \geq \mathbb {E} _ {x \sim \mu} [ \| x \| ] + t\right) = \mu^ {n} \left(\max  _ {1 \leq i \leq n} \exp \left(\lambda (\| x _ {i} \| - \mathbb {E} \| x _ {i} \|\right) \geq \exp (\lambda t)\right) \\ \leq e ^ {- \lambda t} \mathbb {E} \left[ \max  _ {1 \leq i \leq n} \exp \left(\left(\lambda \left(\| x _ {i} \| - \mathbb {E} \| x _ {i} \|\right)\right) \right] \right. \\ \leq e ^ {- \lambda t} \sum_ {i = 1} ^ {n} \mathbb {E} \left[ \exp \left(\left(\lambda \left(\| x _ {i} \| - \mathbb {E} \| x _ {i} \|\right)\right) \right] \right. \\ \leq n \exp \left(- \lambda t + \frac {\lambda^ {2} \sigma^ {2}}{2}\right). \\ \end{array}
$$
For fixed  $t$ , the bound becomes tightest for  $\lambda = t / \sigma^2$  with
$$
\mu^ {n} \left(\max  _ {1 \leq i \leq n} \| x _ {i} \| \geq \mathbb {E} _ {x \sim \mu} [ \| x \| ] + t\right) \leq n \exp \left(- \frac {t ^ {2}}{2 \sigma^ {2}}\right) \leq \delta
$$
$$
- \frac {t ^ {2}}{2 \sigma^ {2}} \leq \log \left(\frac {\delta}{n}\right) \quad \Leftrightarrow \quad t \geq \sigma \sqrt {2 \log (n / \delta)}.
$$
Thus with probability at least  $1 - \delta$ , we have
$$
\max _ {1 \leq i \leq n} \| X _ {i} \| \leq R _ {n} := \mathbb {E} _ {x \sim \mu} \big [ \| x \| \big ] + \sigma \sqrt {2 \log (n / \delta)}.
$$
In the following Lemma, we investigate expectation of squared norm of sub-Gaussian near the tail.
Lemma H.3. Let  $\mu$  be  $\sigma^2$ -sub-Gaussian distribution in  $\mathbb{R}^d$ . For any  $R > \mathbb{E}_{x \sim \mu} \| x \|$ , we have the following:
$$
\mathbb {E} _ {x \sim \mu} \left[ \| x \| ^ {2} 1 _ {B _ {R} (0) ^ {\mathrm {c}}} (x) \right] \leq \exp \left(- \frac {\left(R - \mathbb {E} \| x \|\right) ^ {2}}{4 \sigma^ {2}}\right) \sqrt {2 \pi} \left(\left(\mathbb {E} \| x \|\right) ^ {2} + 2 \sigma^ {2}\right).
$$
$$
\mu \left(\left\{x: \| x \| \geq \left(\mathbb {E} _ {x ^ {\prime} \sim \mu} \| x ^ {\prime} \| + t\right) \right\}\right) \leq \exp \left(- \frac {t ^ {2}}{2 \sigma^ {2}}\right)
$$
as demonstrated in the proof of Lemma H.1 (consider  $n = 1$ ). Thus
$$
\begin{array}{l} \mathbb {E} _ {x \sim \mu} \left[ \| x \| ^ {2} 1 _ {B _ {R} (0) ^ {\mathrm {c}}} (x) \right] \leq \int_ {R} ^ {\infty} s ^ {2} \mu \big (\{\| x \| \geq s \} \big) \mathrm {d} s \leq \int_ {R} ^ {\infty} s ^ {2} \exp \left(- \frac {\left(s - \mathbb {E} \| x \|\right) ^ {2}}{2 \sigma^ {2}}\right) \mathrm {d} s \\ \leq \exp \left(- \frac {\left(R - \mathbb {E} \| x \|\right) ^ {2}}{4 \sigma^ {2}}\right) \int_ {R} ^ {\infty} \exp \left(- \frac {\left(s - \mathbb {E} \| x \|\right) ^ {2}}{4 \sigma^ {2}}\right) d s \\ \leq \exp \left(- \frac {\left(R - \mathbb {E} \| x \|\right) ^ {2}}{4 \sigma^ {2}}\right) \int_ {R} ^ {\infty} \exp \left(- \frac {\left(s - \mathbb {E} \| x \|\right) ^ {2}}{4 \sigma^ {2}}\right) d s \\ = \exp \left(- \frac {\left(R - \mathbb {E} \| x \|\right) ^ {2}}{4 \sigma^ {2}}\right) \sqrt {2 \pi} \left(\left(\mathbb {E} \| x \|\right) ^ {2} + 2 \sigma^ {2}\right). \\ \end{array}
$$
In next Lemma we introduce how the mean of squared norm of sub-Gaussian is concentrated.
Lemma H.4. Assume  $\mu$  is a  $\sigma^2$ -sub-Gaussian distribution in  $\mathbb{R}^d$ . Let  $x_1, \ldots, x_n$  be iid random samples from law  $\mu$ . Then, with probability at least  $1 - \delta$ ,
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \| x _ {i} \| ^ {2} \leq \mathbb {E} _ {x \sim \mu} \big [ \| x \| ^ {2} \big ] + 8 \sigma^ {2} \max \left(\frac {\log (1 / \delta)}{n}, \sqrt {\frac {\log (1 / \delta)}{n}}\right).
$$
In particular, if  $\delta \geq e^{-n}$  then
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \| x _ {i} \| ^ {2} \leq \mathbb {E} \big [ \| x \| ^ {2} \big ] + 8 \sigma^ {2} \sqrt {\frac {\log (1 / \delta)}{n}}
$$
Proof. Firstly since  $\| x_{i} \|$  is  $\sigma^2$ -sub-Gaussian, from [Honorio and Jaakkola, 2014, Appendix B] we observe that  $\| x_{i} \|^{2}$  is  $(4\sqrt{2}\sigma^{2}, 4\sigma^{2})$ -sub-exponential. Next, by independence, for all  $\lambda > 0$  and for all  $|t| \leq \frac{1}{4\sigma^2}$  we have the following:
$$
\begin{array}{l} \mathbb {E} \left[ \right. \exp \left(\lambda \big (\sum_ {i = 1} ^ {n} \| x _ {i} \| ^ {2} - \mathbb {E} \big [ \sum_ {i = 1} ^ {n} \| x _ {i} \| ^ {2} \big ]\right)\left. \right)\left. \right] = \prod_ {i = 1} ^ {n} \mathbb {E} \left[ \right. \exp \left(\lambda \big (\| x _ {i} \| ^ {2} - \mathbb {E} \big [ \| x _ {i} \| ^ {2} \big ]\right)\left. \right)\left. \right] \\ \leq \prod_ {i = 1} ^ {n} \exp \left(\frac {3 2 \sigma^ {4} \lambda^ {2}}{2}\right) \\ = \exp \left(\frac {3 2 n \sigma^ {4} \lambda^ {2}}{2}\right) \\ \end{array}
$$
which implies that  $\sum_{i=1}^{n}\|x_i\|^2$  is  $(4\sqrt{2n}\sigma^2, 4\sigma^2)$ -sub-exponential. Thus the tail bound of subexponential [Adams, 2022, Proposition 2.38] applied to  $\sum_{i=1}^{n}\|x_i\|^2$  yields
$$
\mu^ {n} \left( \right.\left| \right. \sum_ {i = 1} ^ {n} \| x _ {i} \| ^ {2} - \mathbb {E} \left[ \sum_ {i = 1} ^ {n} \| x _ {i} \| ^ {2} \right] \geq s\left. \right) \leq \exp \left(- \frac {1}{2} \min \left(\frac {s ^ {2}}{3 2 n \sigma^ {4}}, \frac {s}{4 \sigma^ {2}}\right)\right)
$$
Plugging-in  $s = nt$ , we have the following:
$$
\mu^ {n} \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \| x _ {i} \| ^ {2} - \mathbb {E} _ {x \sim \mu} \| x \| ^ {2} \geq t\right) \leq \exp \left(- \frac {1}{2} n \min \left(\frac {t ^ {2}}{3 2 \sigma^ {4}}, \frac {t}{4 \sigma^ {2}}\right)\right)
$$
Lastly, take  $\delta = \exp \left(-\frac{1}{2} n\min \left(\frac{t^2}{32\sigma^4},\frac{t}{4\sigma^2}\right)\right)$ . By rearranging the  $t$  with respect to  $\delta$ , we have the following: