How many samples are needed to train a deep neural network?

## Abstract

Abstract Neural networks have become standard tools in many areas, yet many important statistical questions remain open. This paper studies the question of how much data are needed to train a ReLU feed-forward neural network. Our theoretical and empirical results suggest that the generalization error of ReLU feed-forward neural networks scales at the rate 1 / n 1 𝑛 1/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}%
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}} 1 / square-root start_ARG italic_n end_ARG in the sample size n 𝑛 {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}%
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n} italic_n rather than the usual “parametric rate” 1 / n 1 𝑛 1/{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}%
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n} 1 / italic_n . Thus, broadly speaking, our results underpin the common belief that neural networks need “many” training samples.

## 1 Introduction

Neural networks have ubiquitous applications in science and business (Goodfellow et al., ; Graves et al., ; LeCun et al., ; Badrinarayanan et al., ). However, our understanding of their statistical properties remains incomplete. For example, a basic yet very important open question is:
how many training samples are needed to train a (non-linear) neural network?

Over the past two decades, significant progress has been made deriving upper bounds for the generalization error of neural networks (Anthony & Bartlett, ; Arora et al., ; Yarotsky, ; Nagarajan & Kolter, ).
Works like Neyshabur et al. (); Bartlett et al. (); Neyshabur et al. (; ) first highlight the relationship between the complexity of the network space and the generalization error and then, they bound the corresponding complexity measure.
For example, Neyshabur et al. () study the growth of the Rademacher complexity of classes of $\ell_{1}$-constraint neural networks.
Additionally, generalization error bounds for networks with spectral norm constraints are proposed by Neyshabur et al. () (employing a PAC-Bayesian framework) and Bartlett et al. () (employing a margin-based framework) for Lipschitz activation functions.
But a common limitation between all mentioned above works is that their generalization bounds tend to exhibit a strong dependence, often exponential, on either the depth or the width of the network.
Golowich et al. () can do away with this direct dependence on the network’s depth for neural networks based on norm constraint, as their rates grow polynomially in the network depth.
More recently, Taheri et al. (); Mohades & Lederer (); Lederer () propose generalization bounds for regularized neural networks that exhibit a logarithmic
growth in the total number of parameters of the network and the potential
for decrease with more layers against exponential growths.
For example, Taheri et al. () provides an upper bound for the generalization error of $\ell_{1}$-regularized feed-forward neural networks for general activation functions growing by $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}/2)^{1/2-{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}L}}\sqrt{\log(P)}(\log({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n})/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}})$, where $P$ corresponds to the total number of parameters in the network, ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}$ is the number of hidden layers, and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$ is the number of training samples.

While works mentioned above target to upper bound the generalization error of neural networks, another line of research study lower bounds for the mini-max risk of neural networks in a parametric setting.
The central question here is if improving the rate $1/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}$ is possible for neural networks in general or not.
Klusowski & Barron () provide a mini-max lower bound for shallow neural networks with Sinusoidal activation, assuming input vectors are uniformly distributed across $[-1,1]^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}}$ that decays as $1/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}$. S. Du et al. () provide a mini-max lower bound for a CNN with linear activation, where the input domain is normally distributed with a mean of 0 0 and an identity covariance matrix. They also determined the sample complexity for a convolutional neural network with linear activation functions and asserted that tackling the challenges posed by non-linear activation functions, even without convolution structure, remains a difficult task.

To make it short, to the best of our knowledge, a mini-max lower bound for deep feed-forward neural networks with non-linear activation functions that match upper bounds have not been established.
In this paper, we use a technique from information theory known as Fano’s inequality to establish such a bound. Our bound scales as $\sqrt{\log({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d})/{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}$ (with ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$ as the number of training samples and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}$ as the input dimension) that
matches the above-mentioned upper bound rate in Taheri et al. ().
We also confirm this rate in several applications for both regression and classification tasks.
Thus, we demonstrate that the minimum number of samples needed to train a neural network to achieve a prediction error ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\epsilon}^{2}$ essentially scales as $1/{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\epsilon}^{4}$—rather than $1/{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\epsilon}^{2}$.
One of the main challenges in our proofs is establishing a sharp lower bound for the packing number of deep ReLU neural networks, which, to the best of our knowledge, is being extracted for the first time in our paper (see Lemma 3.2).

#### Our two main contributions are:

- 1.
We establish a mini-max lower bound for the risk of ReLU feed-forward networks that depends on the input dimensions only through a logarithmic factor and the level of sparsity of the parameter space. The bound decreases as $1/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}$ with the number of training samples ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$,
which matches recent upper bounds. It is also important to note that the rate of $1/2$ in the exponent remains the same across various input dimensions and network configurations (Theorem 2.2).
- 2.
We support this theoretical bound empirically,
also beyond feed-forward settings to include CNNs (Section 5).
- 3.
We establish a lower bound for the packing number of deep ReLU networks (Lemma 3.2).

The main broader implication is as follows:

- $\bullet$
Most practitioners are convinced that deep learning requires “more” data than classical methods.
It is well-known that the accuracy of classical methods, like the least-squares estimator in linear regression, scale like $1/n$ in the sample size (Wainwright, ; Guo1 et al., ).
Our mathematical proof along with the numerical illustrations showing that this is not the case for (non-linear) deep learning, are arguably the first confirmation and underpinning of the mentioned practitioners’ belief.

More generally, the paper provides a rigorous mathematical perspective on an important aspect of deep learning to complement the much more abundant “engineering perspective” in the field.
We will come back to the relevance of this perspective in the conclusion section (Section 6).

Other related works:
In addition to the aforementioned works, a growing body of research has emerged focused on employing deep ReLU networks to approximate non-parametric regression models (Suzuki, ; Parhi & Nowak, ; Raskutti et al., ; Schmidt-Hieber & Bos, ; Raskutti et al., ; Schmidt-Hieber, ; Zhang & Wang, ; Tsuji, ). These models are characterized by sparse additive structures and specific smoothness properties, such as Hölder, Besov, or Sobolev functions. These studies investigate the approximation capabilities of deep ReLU networks for non-parametric regression functions by establishing mini-max optimal rates. It is crucial to consider that their mini-max lower bounds for the function classes degrade either with the depth of the network or with the parameters of smoothness.

In this paper, our perspective differs slightly from that of others, such as Schmidt-Hieber (), as we view deep ReLU networks as fundamental functions of interest in their own right, rather than as approximation methods for other function classes. We explore the statistical properties of deep ReLU networks by establishing a mini-max lower bound for these networks. In contrast, they emphasize the distinction between function classes and approximation methods and aim to confirm the superiority of deep learning to other representative methods like wavelet transforms, kernel methods, and spline methods in non-parametric settings. Schmidt-Hieber () shows that sparse deep neural networks with ReLU activation function and a well-designed architecture achieve the mini-max rates of convergence (up to log n-factors) in multivariate non-parametric regression, under a broad composition assumption on the regression function. They focus on the regression setup and demonstrate that deep ReLU networks can achieve faster rates under a hierarchical composition assumption on the regression function. This assumption encompasses (generalized) additive models as well as the composition models considered in (Juditsky et al., ; Kohler & Krzyżak, ). Furthermore, they demonstrate that, given the composition assumption, wavelet estimators are only able to obtain sub-optimal rates. Theoretical analysis in Suzuki () reveals that deep learning with ReLU activation in Besov spaces exhibits superior adaptivity, demonstrating mini-max optimal rates and outperforming non-adaptive estimators such as kernel ridge regression. Furthermore, the study highlights deep learning’s ability to mitigate the curse of dimensionality in mixed smooth Besov spaces, emphasizing its high adaptivity and effectiveness as a feature extractor in capturing the spatial inhomogeneity of target functions. Imaizumi & Fukumizu () provides a comprehensive review of prior research related to function estimation using deep neural networks.

Thus, to the best of our knowledge, our work illustrates for the first time in both theory and practice that in absence of additional assumptions (some works such as Schmidt-Hieber () get the rate of $1/n$, under very strict assumptions. For example, to obtain a rate of $1/n$ from their results, we must assume that the parameters are bounded by one and the network functions are bounded),
$1/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}$ is the “correct rate” in deep learning.

Organisation: Section 2 provides the problem formulation and establishes a lower bound on the mini-max risk for ReLU feed-forward neural networks (Theorem 2.2).
Section 3 provides some technical results that form our main result’s foundation including,
a lower bound for the packing number of a ReLU network’s space (Lemma 3.2). We provide the proof of our main theorem and the proof of Lemma 3.2 in Section 4.
In Section 5, we shift our focus to the empirical findings to support our theories.
We conclude our paper in Section 6.
More technical results, empirical details, and detailed proofs are deferred to the Appendix.

## 2 Problem formulation and main result

This section provides an outline of the core elements of our study.
We introduce the problem setting before presenting our main result. To start, we consider the following regression model

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{y}}_{i}}={\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f^{*}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}_{i}})+{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}u}_{i}}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\text{for}~{}i\in\{1, \ldots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}\}\,,$ (1) $$

for an unknown neural network ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f^{*}}:~{}\mathbb{R}^{{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}}\to\mathbb{R}$ and independent and identically distributed noise ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}u}_{i}}\sim\mathcal{N}(0,{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\sigma}^{2})$ with ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\sigma}\in(0,\infty)$. We observe ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$ independent and identically distributed data samples
$({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}_{1},{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}_{1}),({\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}\bm{{x}}}_{2},{\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{y}}_{2}),\ldots,({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}_{{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}})\in\mathbb{R}^{{\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}d}}\times\mathbb{R}$ drawn independently from a joint distribution ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}},{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{y}}}}$ with a fixed marginal distribution ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}}}=\mathcal{N}(
\bm{0},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{I}_{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}d}}}})$. It is assumed that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}u}_{i}}$ and
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}}_{i}}$ are independent
and that the networks are of the form

$$ $\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}}\ :\ {\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\mathbb{R}}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}d}}$ $\displaystyle\to{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{R}}$ (2) $\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}$ $\displaystyle\mapsto{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})~{}\mathrel{\hbox to0.0pt{\raisebox{1.2916 7pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=~{}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}W}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}L}}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\phi}}^{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}\bigl{(}\ldots{\color [rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{1}{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\phi}}^{1}({\color[rgb]{0,0,0}\definecolor[named ]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}W}^{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{{x}}})\bigr{)}$ $$

indexed by ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}=({\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}},\ldots,{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}W}^{0})$
summarizing the
weight matrices ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}W}^{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}l}}}~{}\in~{}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{R}}^{0pt_{
{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}l}+1}\times 0pt_{{\color
[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}l}}}$ for ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}l}~{}\in~{}\{0,1,\ldots,
{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}\}$.
The number of hidden layers (the depth of the network) denotes as ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}\in\{1,2,\ldots\}$, and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}0pt_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}l}}}$ denotes the number of nodes in the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}l}$-th layer (the width of the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}l}$-th layer), where $0pt_{0}={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}$ and $0pt_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}+1}=1$. The function ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\phi}}^{l}:{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{R}}^{0pt_{l}}\to
{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{R}}^{0pt_{l}}$ is the ReLU activation function of the $l$-th layer, and for a vector $\bm{z}=~{}[z_{1},\ldots,z_{0pt_{l}}]\in\mathbb{R}^{0pt_{l}}$ is defined as

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\phi}}^{l}(\bm{z}) \mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt} {$\cdot$}}=\bigl{[}\max\{0,z_{1}\},\max\{0,z_{2}\},\ldots,\max\{0,z_{0pt_{l}} \}\bigr{]}\,.$ $$

We then consider a ($\ell_{1}$-type) sparse parameter space ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{
\operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}$ constraints on the parameters of the network.
We consider $\ell_{1}$-type constraints as opposed to $\ell_{0}$-type constraints, primarily because $\ell_{0}$-type constraints tend to make the problem hard to optimize and “combinatorial”, particularly in high-dimensional settings (Lederer, , Chapter 2).
Then, we define a function class

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named] {pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}L}}}}}}}\mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{ $\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=\Bigl{\{}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}}~{}|~{}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}} =({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}L}},\ldots,{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}W}^{0})\in{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{ \operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}\Bigr{\}}\,,$ $$

where ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{
\operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}$, denotes a sparse parameter space for ReLU feed-forward networks and defined as

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{ \operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}\mathrel{ \hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$} }=\biggl{\{}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}L}},\ldots,{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}W}^{0})~{}\big{|}~{}\sum_{l=1}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}L}}|\!|\!|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{l}|\!|\!|_{1}\leq{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}v_{\operatorname{s}}},~{ }|\!|\!|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{0}_{j,\cdot}|\!|\!|_ {1}\leq{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}v_{0}}~{}\text{for all}~ {}~{}j\in\{1,\ldots,0pt_{1}\}\biggr{\}}\,,$ $$

for ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}W}^{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}l}}}~{}\in~{}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{R}}^{0pt_{
{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}l}+1}\times 0pt_{{\color
[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}l}}}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}l}~{}\in~{}\{0,1,\ldots,
{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}\}$ and we define $|\!|\!|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{l}|\!|\!|_{1}
\mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}
{$\cdot$}}=\sum_{k=1}^{0pt_{l+1}}\sum_{j=1}^{0pt_{l}}|{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}W}^{l}_{kj}|$.

###### Assumption 2.1 ( ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}v_{0}}=1$ ) .

For simplicity in the proof of our technical results (Lemma 3.2), we assume ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}v_{0}}=1$.

This assumption is useful for constructing a subclass of space ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$ in the proof of Lemma 3.2 to establish a lower bound for the packing number of a deep ReLU network’s space. This assumption basically determines the structure of the weight between the input layer and the first hidden layer of a neural network. It also guarantees that the number of input dimensions ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}$, matches the width of the constructed subclass of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$.

The mini-max risk for the function class ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$ can be defined as (Wainwright, , Chapter 15)

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{R}_{({\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n},{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}d})}({{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{ \operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}};{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}\circ{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\rho}})}\mathrel{\hbox to0.0pt{\raisebox{1.29167 pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=\inf_{\widehat{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}f}}}\sup_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f^{*}}\in{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named] {pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}L}}}}}}}}\mathbb{E}_{({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{{x}}}_{i}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{y}}_{i}})^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}}_{i=1}}\Bigl{[}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}\bigl{(} {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}(\widehat{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}},{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f^{*}})\bigr{)}\Bigr{]}\,,$ (3) $$

where ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}~{}:~{}{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}\times{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{
\operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}}
\rightarrow[0,\infty)$ is a semi metric(^1^11A semi metric satisfies all properties of a metric, except that there may exist pairs ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}\neq{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}^{\prime}$ for which ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}({\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f},{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}^{\prime})=0.$) and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}~{}:~{}[0,\infty)
\rightarrow[0,\infty)$ is an increasing function. The expectation is taken with respect to the training data $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}}_{i}},{\color[rgb]{0,0,0}\definecolor[named
]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{y}}_{i}})^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{i=1}$ and the infimum runs over all possible estimators $\widehat{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}}$ (measurable functions) of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f^{*}}$ on the training data $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}}_{i}},{\color[rgb]{0,0,0}\definecolor[named
]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{y}}_{i}})^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{i=1}$. Hence, $\widehat{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}}({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})\equiv\widehat{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}}({\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{{x}}},\{({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}_{i},{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}_{i})\}_{i=1}^{{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}})$, where ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}$ is a new data point with the same distribution ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}}}$. We use the notation ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{R}_{({\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n},{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}d})}({{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{
\operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}};{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}\circ{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}\rho}})}$ to emphasize that the mini-max risk depends on the number of training samples ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$, the input dimension ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}$ and the space ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$.

In this paper, our focus is on the standard setting where ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}$ represents the $L_{2}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}}})$-norm, and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}(t)=~{}t^{2}$. Therefore, ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}({\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\rho}(\widehat{{\color[rgb]{0,0,0}\definecolor[named
]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}},{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f^{*}}))$ is the squared $L_{2}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}}})$-norm,
that is our mini-max risk

$$ $\inf_{\widehat{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}}}\sup_{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f^{*}}\in{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{ \operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}}}\mathbb{E }_{({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}_{i}},{\color[rgb]{0,0,0}\definecolor[named ]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{y}}_{i}})^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{i=1}}\bigl{[} |\!|\widehat{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}}-{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f^{*}}|\!|_{L_{2}}^{2}\bigr{]}\,.$ $$

We assume that the distribution ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}}}$ has a density ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{h({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})}}$ with respect to the Lebesgue measure ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}}}$ which, implies that

$$ $|\!|\widehat{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}}-{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f^{*}}|\!|_{L_{2}}\mathrel{\hbox to0.0pt{\raisebox{1 .29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=\biggl{(}{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\int_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{{x}}}\in{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{X}}}} \bigl{(}\widehat{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}}({\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\bm{{x}}})-{\color[rgb]{0,0,0}\definecolor[named] {pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f^{*}}({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{{x}}})\bigr{)}^{2}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{h({\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}}{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}}\biggr{)}^{1/2}\,.$ $$

We now present our mini-max risk lower bound for deep ReLU neural networks. Considering the regression model defined in Equation (1), where ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f^{*}}\in{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{
\operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}}$ (a ReLU neural network with ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}$ hidden layers and $\ell_{1}$-bounded weights), then we have:

###### Theorem 2.2 (Mini-max risk lower bound for ReLU feed-forward networks) .

Using the $L_{2}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}}})$-norm as our underlying semi metric ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}$, and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}_{1},\ldots,{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}_{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}n}}\sim\mathcal{N}(\bm{0},{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\bm{I}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}d}}}})$, then for ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}$ large enough and any increasing function ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}:~{}[0,\infty)\to~{
}[0,\infty)$, it holds that

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{R}_{({\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n},{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}d})}({{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{ \operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}};{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}\circ{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\rho}})}\geq\frac{1}{2}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\Phi}\biggl{[}{\color[rgb]{0,0,0}\definecolor[named] {pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}c}\sqrt{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}V_{\mathcal{F}}}}\Bigl{(}\frac{\log({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}d})}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}\Bigr{)}^{1/4}\biggr{ ]}\,,$ (4) $$

with ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c}\mathrel{\hbox to0.0pt
{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=\sqrt{{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\sigma}/40{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\kappa}}$, where ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\kappa}\in[1,\infty)$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}\in(0,\infty)$ are numerical constants, and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}\mathrel
{\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$
}}=~{}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}v_{\operatorname{s}}}/{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L})^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}$.
For ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}(\cdot)=(\cdot)^{2}$, we specifically obtain

$$ $\inf_{\widehat{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}}}\sup_{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f^{*}}\in{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{ \operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}}}\mathbb{E }_{({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}_{i}},{\color[rgb]{0,0,0}\definecolor[named ]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{y}}_{i}})^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{i=1}}\bigl{[} |\!|\widehat{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}}-{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f^{*}}|\!|_{L_{2}}^{2}\bigr{]}\geq\frac{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c}^{2}}{2}({\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}})\sqrt{\frac{\log({\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}d})}{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}}}\,.$ (5) $$

We consider ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\kappa}$ as a scaling factor inspired by “local packing” version of Fano’s inequality (Wainwright, , Section 15.3.3).
Also for the technical reasons, we assume that the input dimension is large enough, let say ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}\in[10,\infty)$.
Our mini-max lower bound above reveals essentially a $1/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}$-decrease in the sample size, and a $\log({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d})$-increase in the input dimension, that perfectly aligns with the upper bounds on the generalization error of deep neural networks with $\ell_{1}$-type regularization (compare to Taheri et al. ()).
Additionally, the factor ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}$ in our rates can be interpreted as a product over the $\ell_{1}$-norm bounds of different layers in ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$. This interpretation aligns perfectly with the established upper and lower bounds on the generalization error of regularized neural networks found in the literature (Taheri et al., ; Klusowski & Barron, ).
The presence of
${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}$ in our rate supports the sharpness of our result (compare with Taheri et al. ()).

Theorem 2.2 demonstrates that for all possible $\widehat{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}}$, risk (also called “generalization error”) scales at least as ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}\sqrt{
\log({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d})/{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}$. Then, by considering an upper bound for the risk

$$ $\mathbb{E}_{({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}_{i}},{\color[rgb]{0,0,0}\definecolor[named ]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{y}}_{i}})^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{i=1}}\bigl{[} |\!|\widehat{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}}-{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f^{*}}|\!|_{L_{2}}^{2}\bigr{]}\leq{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\epsilon}^{2}\,,$ $$

the result of Theorem 2.2 can be reformulated. It implies that one requires a minimum of

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}\geq{\Bigl{(}\frac{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c}}{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\epsilon}}\Bigr{)}}^{4}\frac{({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}V_{\mathcal{F}}})^{2}\log({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}d})}{4}\,,$ $$

samples to achieve an error of at most ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\epsilon}^{2}$ for
ReLU feed-forward neural networks.
This stands in strong contrast to classical methods (consider, for instance, linear regression with the least squares estimator as the quintessential example of a traditional pipeline),
which usually require of the order $1/\epsilon^{2}$ samples to reach the same accuracy.
However, of course, this does not mean that one should not apply deep learning! Deep learning is extremely flexible in approximating functions,
which makes it the preferred method in many modern applications.

###### Assumption 2.1 ( v 0 = 1 subscript 𝑣 0 1 {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}%
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}v_{0}}=1 italic_v start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT = 1 ) .

###### Theorem 2.2 (Mini-max risk lower bound for ReLU feed-forward networks) .

## 3 Technical results

Here, we provide a technical result essential in proving our main theorem 2.2.
Since our proof approach is based on a classical result from information theory known as Fano’s inequality (Lemma B.1), we need to provide a lower bound for the $\log$ of the packing number of our network space provided in Lemma 3.2.

The following notation will be used throughout the paper. For a vector $\bm{v}\in\mathbb{R}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}}$, $\ell_{0}$-norm is defined by
$\hskip 1.42262pt\!|\!|\bm{v}|\!|_{0}\mathrel{\hbox to0.0pt{\raisebox{1.29167pt
}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=\#\{i\in\{1,\ldots,{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}\}~{}:~{}v_{i}\neq 0\}$,
$\ell_{1}$-norm is defined by
$\hskip 1.42262pt\!|\!|\bm{v}|\!|_{1}\mathrel{\hbox to0.0pt{\raisebox{1.29167pt
}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=~{}\sum_{i=1}^{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}d}}|v_{i}|$ and the Euclidean norm is defined by $|\!|\bm{v}|\!|_{2}\mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}
\raisebox{-1.29167pt}{$\cdot$}}=~{}\sqrt{\sum_{i=1}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}d}}(v_{i})^{2}}$. We recall $|\!|\!|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{l}|\!|\!|_{1}
\mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}
{$\cdot$}}=\sum_{k=1}^{0pt_{l+1}}\sum_{j=1}^{0pt_{l}}|{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}W}_{kj}^{l}|$ and $|\!|\!|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}W}^{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}l}}}|\!|\!|_{\infty}\mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}
\hss}\raisebox{-1.29167pt}{$\cdot$}}=~{}\max\limits_{1\leq k\leq 0pt_{l+1}}
\sum_{j=1}^{0pt_{l}}|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}_{kj}^{l}|$ for a matrix ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{l}\in\mathbb{R}^{0pt
_{l+1}\times 0pt_{l}}$. The cardinality of the $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-packing of the corresponding
network’s space ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$ for ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}\in(0,\infty)$ and with respect to $L_{2}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}}})$-norm is defined as ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}\mathrel{
\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}
}=~{}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}(2{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta},{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{
\operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}},|\!|\cdot
|\!|_{L_{2}})$. We use the notation ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}}(
\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}||\mathbb{{\color[rgb
]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Q}})$ to denote the Kullback-Leibler (KL) divergence between two probability distributions $\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}$ and $\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Q}}$. We define $[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]\mathrel{
\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}
}=\{1,\ldots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0
}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}\}$ as the index set. And we define the notation ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}X}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}\mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{$
\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=~{}({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}}_{1},\ldots,{\color[rgb]{0,0,0}\definecolor
[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}})^{\top}\text{ and }{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}Y}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}\mathrel{\hbox to0.0
pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=~{}({
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}_{1},\ldots,{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}_{{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}})^{\top}$.

We now proceed to provide the definition of packing number (Vaart & Wellner, ), which is of great importance in our study.

###### Definition 3.1 (Packing number) .

Consider a metric space consisting of a set ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$ and a semi metric ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}$ as defined in Section 2 then, an $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-packing of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$ in the semi metric ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}$ is a collection $\{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{1}}},
\ldots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}}}}\}
\subseteq{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$ such that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}({\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}},{\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}})\geq~{}2{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\delta}$ for all $j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]$ and $j\neq k$. The $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-packing number ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}(2{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta},{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{
\operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}},{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho})$, is the cardinality of the largest $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-packing.

Our next lemma provides a lower bound for the $\log$ of the packing number of a ReLU network’s space.

###### Lemma 3.2 (Lower bounding the packing numbers of ReLU feed-forward networks’ spaces) .

For a sparse collection of deep ReLU feed-forward networks’ spaces ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$, there exist ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}\in(0,\infty)$ such that

$$ $\log{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}\bigl{(}2{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta},{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{ \operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}},|\!|\cdot |\!|_{L_{2}}\bigr{)}\geq\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{20{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}}\Bigr{)}^{2}\log ({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d})\,,$ $$

where ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}\in(0,\infty)$ is a numerical constant and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}=({
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}v_{\operatorname{s}}}/{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L})^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}$.

This lemma provides valuable insights into the capacity and potential complexity of ReLU feed-forward networks.
The key components of this bound are twofold: 1. the favorite $\log({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d})$ factor (and not the huge network size) and 2. the factor ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}$, that both are enhanced employing the $\ell_{1}$-norm control over the parameters of the network.
In particular, the logarithmic dependence on the input dimension ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}$ and the factor ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}$ offer a perspective on how complexity growth relates to the network size and the sparsity level.

###### Definition 3.1 (Packing number) .

###### Lemma 3.2 (Lower bounding the packing numbers of ReLU feed-forward networks’ spaces) .

## 4 PROOF OF THEOREM 2.2

Here, we provide the proof of our main theorem:

###### Proof.

The proof is based on employing a variants of Fano’s method well-known as “local packing” or “rescaling” (Wainwright, ; Yang & Barron, ) and our Lemma 3.2.

Let’s start the proof writing the mutual information in terms of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ (see also Section B): assuming a $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-packing is available with centers $j\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]$, then, using Wainwright (, Equation 15.30) we can obtain

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}Y}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}})}=\frac{1}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}}\sum_ {j=1}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}}{\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}}~{}| |~{}\bar{\mathbb{Q}})\leq\max_{j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{M}}],j\neq k}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{ \operatorname{KL}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}}~{}| |~{}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}})\,,$ $$

where $\bar{\mathbb{Q}}\mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}
\raisebox{-1.29167pt}{$\cdot$}}=\sum_{j=1}^{{\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{M}}}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\mathbb{P}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}}$ is the mixture distribution. The naive idea of “local packing” is considering a $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-packing within the space ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$ (in the semi metric ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}$) such that
$\max_{j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0
}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}],j\neq k}{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}}({
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}}~{}|
|~{}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}})
\leq{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}(2\kappa\delta)^{2}$ for a quantity $\kappa\in[1,\infty)$.
That implies the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ should be bounded by a multiple of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$ for all centers in the packing set.
In fact, instead of the whole space, we consider just a local area of that (specified as a ball with radius $2\kappa{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$).
Then, employing our earlier display implies an upper bound for the mutual information.

Following above technique, we focus on a local area of the network’s space and call it ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{S}$, consider a $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-packing set of that where for two distinct networks ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}},{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}\in{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{S}$ with $j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}^{\prime}]$ we have ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}({\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})},{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})})\geq 2{\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\delta}$.
Also, we suppose that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}({\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})},{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})})\leq 2\kappa{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\delta}$ for a constant $\kappa\in[1,\infty)$ for ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}},{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}\in{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{S}$ with $j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}^{\prime}]$ (note that we can always find such an area just by rescaling across our networks output; exact value of $\delta$ be specified later).
Then, employing Lemma B.3, we obtain

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}}~{}| |~{}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}}) \leq\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}{2{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\sigma}^{2}}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\rho}\bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})},{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{)}^{2} \leq\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}{2{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\sigma}^{2}}(2\kappa\delta)^{2}\,,$ $$

for all $j\neq k~{}\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}^{
\prime}]$, where the last inequality is reached using our assumption on ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{S}$, that is a ball with radius $\kappa{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$.
Collecting results above, we can obtain ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}Y}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}})}\leq n(2\kappa\delta)^{2}/2{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\sigma}^{2}$ (see also Lemma B.5).

In the second step and based on (local) Fano’s inequality, we need to specify $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-packing set with the largest cardinality to verify $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}Y}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}})}+\log 2)/\log{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}
\leq 1/2$.
So we assign the value of $\delta$
to ensure

$$ $\log{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}\bigl{(}2{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta},{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{S},|\!|\cdot|\!|_{L_{2}}\bigr{)} \geq\biggl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb} {0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}(2{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\kappa}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\delta})^{2}}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\sigma}^{2}}+2\log {2}\biggr{)}\geq\biggl{(}\frac{8{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\kappa}{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\delta})^{2}}{{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\sigma}^{2}}\biggr{)}.$ $$

Next step is lower bounding $\log{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}(2{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta},{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{S},|\!|\cdot|\!|_{L_{2}})$.
Let recall that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{S}$ is a local area of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$ (${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{S}\subset{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$).
Employing Yang & Barron (, Lemma 3), $\log{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}(2{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta},{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{S},|\!|\cdot|\!|_{L_{2}})$ that is a local-entropy can be lower-bounded on a high level by a fraction of $\log{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}(2{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta},{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{
\operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}},|\!|\cdot
|\!|_{L_{2}})$ that is the global entropy.
Accordingly, we employ
our lower bound on the packing number in Lemma 3.2 as a lower bounding (a fraction of) $\log{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}(2{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta},{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{
\operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}},|\!|\cdot
|\!|_{L_{2}})$ that implies

$$ $\frac{8}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\sigma}^{2}}{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\kappa}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\delta})^{2}=\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{20{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}}\Bigr{)}^{2}\log ({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d})\,,$ $$

and gives

$$ ${{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}^{2}}=\frac{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\sigma}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}V_{\mathcal{F}}}}{40{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\kappa}}\sqrt {\frac{\log({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d})}{2{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}n}}}\,.$ $$

Let note that our specified value for $\delta$ above is not exact, because for simplicity we just keep the impact of main factors like ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n},{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}d}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\sigma}$, while skipping some constants connecting the local and global entropy.
Substituting the obtained value of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$ into “local packing” version of Fano’s inequality (Wainwright, , Section 15.3.3) can yield

$$ $\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{R}_{({\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n},{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}d})}({{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{ \operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}};{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}\circ{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\rho}})}$ $\displaystyle\geq\frac{1}{2}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\Phi}\Biggl{[}\biggl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\sigma}{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{40{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\kappa}}\sqrt{\frac{\log({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}d})}{2{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}}}\biggr{)}^{1/2}\Biggr{]}$ $\displaystyle=\frac{1}{2}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}\Biggl {[}\biggl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\sigma}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}V_{\mathcal{F}}}}{40{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\kappa}} \biggr{)}^{1/2}\biggl{(}\frac{\log({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}d})}{2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}\biggr{)}^{1/4}\Biggr {]}\,.$ $$

We can plug the value of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c}$— in the view of Theorem 2.2— into this inequality and get

$$ $\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{R}_{({\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n},{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}d})}({{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{ \operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}};{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}\circ{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\rho}})}\geq\frac{1}{2}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\Phi}\biggl{[}{\color[rgb]{0,0,0}\definecolor[named] {pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}c}\sqrt{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}V_{\mathcal{F}}}}\Bigl{(}\frac{\log({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}d})}{2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}\Bigr{)}^{1/4}\biggr{ ]}\,,$ $$

which proves our first claim.

For the second claim, we simply use ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}(\cdot)=(\cdot)^{2}$ to obtain

$$ $\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{R}_{({\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n},{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}d})}({{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{ \operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}};{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}\circ{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\rho}})}\geq\frac{{\color[rgb]{0,0,0}\definecolor [named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}c}^{2}}{2}({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}V_{\mathcal{F}}})\sqrt{\frac{\log({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}d})}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}}\,.$ $$

Based on our mini-max risk setting (Section 2), the above expression can be presented as follows:

$$ $\displaystyle\inf_{\widehat{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}}}\sup_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f^{*}}\in{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{ \operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}}}\mathbb{E }_{({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}_{i}},{\color[rgb]{0,0,0}\definecolor[named ]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{y}}_{i}})^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{i=1}}\bigl{[} |\!|\widehat{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}}-{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f^{*}}|\!|_{L_{2}}^{2}\bigr{]}\geq\frac{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c}^{2}}{2}({\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}})\sqrt{\frac{\log({\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}d})}{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}}}\,,$ $$

as desired.
∎

###### Proof.

### 4.1 PROOF OF LEMMA 3.2

###### Proof.

The core of this proof involves two steps: first, the construction of a subclass of functions within ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$, and then finding a lower bound for $\log$ of the cardinality of the constructed class. Second, using the fact that a lower bound for the cardinality of a smaller space can serve as a lower bound for the cardinality of the larger space. Let us begin by discussing the construction of the subclass of the function class ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$.

###### Proof.

### Step 1: the construction a subclass of function class ℱ ℬ L subscript ℱ subscript ℬ L {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}%
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}%
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}%
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]%
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}%
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}%
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}%
\pgfsys@color@gray@fill{0}L}}}}}}} caligraphic_F start_POSTSUBSCRIPT caligraphic_B start_POSTSUBSCRIPT roman_L end_POSTSUBSCRIPT end_POSTSUBSCRIPT

Our first step is to construct a subclass of our defined function class ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$ and then to find a lower bound for the $\log$ of the packing number of the constructed class. To achieve this, we begin by defining a set of binary vectors ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{C}}\in\{0,1\}^{
{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}}$ for ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}\in[10,\infty)$ such that each element of this set has a Hamming weight of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}^{\prime}$, where ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}^{\prime}\in[1,{\color
[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}/10]$ and the cardinality of this set is denoted by ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}$. Recall that, we assume that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}v_{0}}=1$, so, we can choose ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}^{\prime}={\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}v_{0}}^{2}=1$. Then, anyone can readily conclude that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}={\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}d}$ and we can consider the vector ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{i}\in\{0,1\}^
{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}}$ as a vector with all the entries equal to zero except for the $i$th entry, which is set to one. It implies that for all $i\neq j\in\{1,\ldots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}\}$

$$ $\lvert{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{i}^{\top}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{j}\rvert=0\,.$ $$

We can also conclude that for each ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{i}$ with $i\in\{1,\ldots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}\}$, we have

$$ $|\!|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{i}|\!|_{2}= \sqrt{\bigl{(}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{i})_{1} \bigr{)}^{2}+\ldots+{\bigl{(}({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\bm{b}}}_{i})_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb} {0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}}}\bigr{)}^{2}} =1\,.$ $$

Following the same argument as above we have

$$ $\hskip 1.42262pt\!|\!|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{i} |\!|_{1}=\bigl{|}{({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb} {0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{i})_{1 }}\bigr{|}+\ldots+\bigl{|}{({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\bm{b}}}_{i})_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb} {0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}}}\bigr{|}=1\,.$ $$

Then, for an enumeration ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{1},\ldots,{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{S}}}$ of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{C}}$, we claim that the function space ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{0}$ as defined below can represent a subspace of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$:

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{0}\mathrel {\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$ }}=\Biggl{\{}f_{(\overline{\overline{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{w}}}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}})}({\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})\mathrel{\hbox to 0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=\frac{ {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\sum_{k=1}^{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}}\overline{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}}_{k}{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\phi}}^{1}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{ \top}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})~{}:\overline{ {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}\in\overline{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{A}}}\Biggl{\}}$ $$

with $~{}\overline{\overline{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}}
\mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}
{$\cdot$}}=({{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0
}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}/{{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}})\overline{{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}$ and $\overline{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{A}}}\mathrel{
\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}
}=\{\overline{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}\in\{0,{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}\}^{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{S}}}~{}~{}:\hskip 1.42262pt\!|\!|\overline{{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}|\!|_{1}={\color
[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\lambda}}\}$
for ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}},{\color
[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}\in(0,\infty)$ and an integer ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}\in[1,{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}/10]$ (exact values of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}$ be specified later in the proof, while the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}$ is a constant depending on ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}v_{\operatorname{s}}}$.)
We argue that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{0}$ is representing a subspace of deep neural networks ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$ with non-negative weight matrices ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}W}^{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}l}}}$ for $l\in\{1,\dots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}\}$ inspired by Hebiri & Lederer ().
To be more precise, we state that for a fixed pair ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}},{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}\tau}$ and $\forall~{}\overline{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb
}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}\in
\overline{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{A}}}$, there exists a tuple of non-negative weight matrices $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{1},\dots,{\color[rgb
]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}})$ with $\sum_{l=1}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}|\!|\!|{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}W}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}l}}}|\!|\!|_{1}\leq{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}v_{\operatorname{s}}}$ that their product verifies ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{L}{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}W}^{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}L}-1}\dots\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W^{1}=\overline{{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}$ (see Remark 4.1).
For example, for a large enough ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}v_{\operatorname{s}}}$, we can fix ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}=1$, since our network space ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$ can generate binary vectors $\overline{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}$ with cardinality ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}$ employing a suitable tuple of non-negative matrices $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{1},\dots,{\color[rgb
]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}})$.
In fact, we are using the property that for non-negative weight matrices, the ReLU activations in deep neural networks (for layers $l\in\{1,\dots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}\}$) can be ignored and so, deep network for the corresponding layers behaves like a simple matrix product (Hebiri & Lederer, ).
This property gives us the chance to write those
specific networks (with non-negative weight matrices) in the form of a shallow neural network as stated in ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{0}$.
Let’s also note that for the corresponding networks, we can 1. invoke compatible norms ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}$ times, 2. use the inequality of arithmetic and geometric means, 3. definition of our ReLU network’s space (defined in Section 2) and 4. invoke the definition of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}$— in the view of Theorem 2.2—, to get

$$ $\displaystyle\hskip 1.42262pt\!|\!|{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}W}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}W}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}L}-1}\cdots{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{1}|\!|_{1}$ $\displaystyle\leq\hskip 1.42262pt\!|\!|{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}W}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}|\!|_{1}|\!|\!|{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}L}-1}\cdots{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}W}^{1}|\!|\!|_{\infty}$ $\displaystyle\vdots$ $\displaystyle\leq\hskip 1.42262pt\!|\!|{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}W}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}|\!|_{1}|\!|\!|{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}L}-1}|\!|\!|_{1}|\!|\!|{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}W}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}L}-2}|\!|\!|_{1}\cdots|\!|\!|{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}W}^{1}|\!|\!|_{1}$ $\displaystyle\leq\Bigl{(}\frac{1}{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}L}}\sum_{i=1}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}|\!|\!|{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{i}|\!|\!|_{1}\Bigr{) }^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}$ $\displaystyle\leq\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}v_{\operatorname{s}}}}{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}L}}\Bigr{)}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}$ $\displaystyle={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}\,.$ $$

Some rewriting over the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{0}$ implies

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{0}=\Biggl{ \{}f_{({\overline{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}},{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}})}({\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})=\frac{{\color [rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\lambda}}}\sum_{k=1}^{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{S}}}{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{w}}}_{k}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\phi}}^{1}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})~{}:{{\color[rgb]{0,0,0}\definecolor[named ]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{w}}}\in{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{A}}}\Biggl{\}}\,,$ $$

where $\overline{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}\mathrel{\hbox to
0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=({
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}/{\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\lambda}}){\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{w}}$ and
${{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{A}}}\mathrel{
\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}
}=\{{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}\in\{0,1\}^{{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}}~{}~{}:\hskip 1.422
62pt\!|\!|{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}|\!|_{1}={\color
[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}\}$ is the set in Lemma B.8. ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}\in[1,{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}/10]$ is an integer and denotes as the Hamming weight of each element of this set (Klusowski & Barron, , Theorem 2).

According to the above definition of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{0}$, we have

$$ $\mathbb{E}\bigl{[}|\!|f_{(\overline{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{w}}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}})}({\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})-f_{(\overline {{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}^{\prime},{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}})}({\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})|\!|_{L_{2}}^{ 2}\bigr{]}=\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\Bigr{)}^{2} \mathbb{E}\biggl{[}\Bigl{(}\sum_{k=1}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{S}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}_{k}-{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k}){\color[rgb]{0,0,0}\definecolor [named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\phi}}^{1}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{ \top}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})\Bigr{)}^{2} \biggr{]}\,,$ $$

where ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}},{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}\bm{w}}^{\prime}\in{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{A}}$.

Note that, based on the structure of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}^{\prime}$, for all $k\in\{1,\ldots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}\}$, $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}_{k}-{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k})$ falls within the set $\{-1,0,1\}$. And if $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}_{k}-{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k})=0$, the value of the expected term on the right-hand side –for the corresponding $k$– is equal to 0; thus for the sake of convenience, we consider an integer value ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}<{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}$ in such a way that $\lvert{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}_{k}-{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k}\rvert=1$ for all $k\in\{1,\ldots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}\}$. Based on the structure of all pairs ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}},{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}\bm{w}}^{\prime}\in{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{A}}$ and Lemma B.8, we can conclude that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}\geq{\color
[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}/5$. We will use ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}$ for the remainder of the proof.

We then proceed with

$$ $\mathbb{E}\bigl{[}|\!|f_{(\overline{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{w}}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}})}({\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})-f_{(\overline {{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}^{\prime},{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}})}({\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})|\!|_{L_{2}}^{ 2}\bigr{]}=\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\Bigr{)}^{2} \mathbb{E}\biggl{[}\Bigl{(}\sum_{k=1}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{S}}^{\prime}}\bigl{(}({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{w}}_{k}-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k}){ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\phi}}^{1}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})\bigr{)}\Bigr{)}^{2}\biggr{]}\,.$ $$

Next, we are motivated to find a lower bound for $\mathbb{E}[|\!|f_{(\overline{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{w}}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}})}({\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})-f_{(\overline
{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}^{\prime},{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}})}({\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})|\!|_{L_{2}}^{
2}]$. We can 1. employ the last view, 2. expand the squared over the sum and invoke the linearity of expected value, 3. invoke the linearity of expected value, 4. apply the above assumption that $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}_{k}-{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k})^{2}=1$, 5. use the result of Lemma B.7, which shows $\mathbb{E}[({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\phi}}}}({\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}))^{
2}]=1/2$, and the properties of sum function and 6. use the fact that $\mathbb{E}[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\phi}}^{1}}}({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}}){\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\phi}}^{1}}}({\color
[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{j}^{\top}{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})]=1/2\pi$, and the properties of sum function to obtain

$$ $\displaystyle\mathbb{E}\bigl{[}|\!|f_{(\overline{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{w}}},{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\bm{b}}})}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})-f_{( \overline{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}^{\prime},{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}})}({\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})|\!|_{L_{2}}^{ 2}\bigr{]}$ $\displaystyle=\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\Bigr{)}^{2} \mathbb{E}\biggl{[}\Bigl{(}\sum_{k=1}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{S}}^{\prime}}\bigl{(}({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{w}}_{k}-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k}){ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\phi}}^{1}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})\bigr{)}\Bigr{)}^{2}\biggr{]}$ $\displaystyle=\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\Bigr{)}^{2} \Biggl{(}\sum_{k=1}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime} }\mathbb{E}\Bigl{[}\bigl{(}({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{w}}_{k}-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k}){ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\phi}}^{1}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})\bigr{)}^{2}\Bigr{]}$ $\displaystyle\hskip 50.00008pt+\mathbb{E}\biggl{[}\sum_{k=1}^{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{S}}^{\prime}}\sum_{\begin{subarray}{c}j=1\\ j\neq k\end{subarray}}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime }}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}_{k}-{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k}){\color[rgb]{0,0,0}\definecolor [named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\phi}}^{1}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{ \top}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})({\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{w}}_{j}-{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{w}}^{\prime}_{j}){\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\phi}}^{1}}}({\color [rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{j}^{\top}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})\biggr{]} \Biggr{)}$ $\displaystyle=\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\Bigr{)}^{2} \Biggl{(}\sum_{k=1}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime} }\mathbb{E}\Bigl{[}\bigl{(}({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{w}}_{k}-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k}){ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\phi}}^{1}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})\bigr{)}^{2}\Bigr{]}$ $\displaystyle\hskip 50.00008pt+\sum_{k=1}^{{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{S}}^{\prime}}\sum_{\begin{subarray}{c}j=1\\ j\neq k\end{subarray}}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime }}\mathbb{E}\bigl{[}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}_{k}-{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k}){ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\phi}}^{1}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{w}}_{j}-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{j}){ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\phi}}^{1}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{b}}}_{j}^{\top}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})\bigr{]}\Biggr{)}$ $\displaystyle=\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\Bigr{)}^{2} \biggl{(}\sum_{k=1}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime} }\mathbb{E}\Bigl{[}\bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\phi}}^{1}}}({\color [rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})\bigr{)}^{2} \Bigr{]}$ $\displaystyle\hskip 50.00008pt+\sum_{k=1}^{{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{S}}^{\prime}}\sum_{\begin{subarray}{c}j=1\\ j\neq k\end{subarray}}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime }}\mathbb{E}\bigl{[}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}_{k}-{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k}){ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\phi}}^{1}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{w}}_{j}-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{j}){ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\phi}}^{1}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{b}}}_{j}^{\top}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})\bigr{]}\biggr{)}$ $\displaystyle=\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\Bigr{)}^{2} \biggl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}}{2}+ \sum_{k=1}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}}\sum_{ \begin{subarray}{c}j=1\\ j\neq k\end{subarray}}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime }}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}_{k}-{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k})({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{w}}_{j}-{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{w}}^{\prime}_{j})\mathbb{E}\bigl{[}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\phi}}^{1}}}({\color [rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}){\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\phi}}^{1}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{j}^{ \top}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})\bigr{]}\biggr {)}$ $\displaystyle=\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\Bigr{)}^{2} \biggl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}}{2}+ \frac{1}{2\pi}\sum_{k=1}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{S}}^{\prime}}\sum_{\begin{subarray}{c}j=1\\ j\neq k\end{subarray}}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime }}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}_{k}-{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k})({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{w}}_{j}-{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{w}}^{\prime}_{j})\biggr{)}\,,$ $$

here, we need to analyse the second term. We know that $\lvert{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}_{k}-{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k}\rvert=1$. Thus,
$({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}_{k}-{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k}),({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{w}}_{j}-{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{w}}^{\prime}_{j})$ for $k,j\in\{1,\dots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}\}$ can have either the same signs or different signs that result in the corresponding positive or negative terms. The worst case happens when the number of $+1$ and $-1$ cases are equal.
There, the number of total terms is ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}({\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}-1)$, and we have ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}(({\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}/2)-1)$ number of positive terms and the remaining terms are negative (${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime 2}/2$). So, we can proceed the lower bounding by 1. employing the last view, 2. considering the worst case scenario as discussed, 3. performing some simplifications, 4. performing some arithmetic math, 5. using the fact that $(\pi-1)/2\pi)>2/10$, 6. applying the conclusion that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}\geq{\color
[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}/5$ and 7. performing some simplifications to obtain

$$ $\displaystyle\mathbb{E}\bigl{[}|\!|f_{(\overline{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{w}}},{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\bm{b}}})}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})-f_{( \overline{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}}^{\prime},{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}})}({\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})|\!|_{L_{2}}^{ 2}\bigr{]}$ $\displaystyle=\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\Bigr{)}^{2} \biggl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}}{2}+ \frac{1}{2\pi}\sum_{k=1}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{S}}^{\prime}}\sum_{\begin{subarray}{c}j=1\\ j\neq k\end{subarray}}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime }}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{w}}_{k}-{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{w}}^{\prime}_{k})({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{w}}_{j}-{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{w}}^{\prime}_{j})\biggr{)}$ $\displaystyle\geq\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\Bigr{)}^{2} \Biggl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}}{2}+ \frac{1}{2\pi}\biggl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime} \biggl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}}{2}- 1\biggr{)}-\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime 2}}{2 }\biggr{)}\Biggr{)}$ $\displaystyle=\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\Bigr{)}^{2} \biggl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}}{2}+ \frac{-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}}{2\pi} \biggr{)}$ $\displaystyle=\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\Bigr{)}^{2} \biggl{(}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime}}\Bigl{(} \frac{\pi-1}{2\pi}\Bigr{)}\biggr{)}$ $\displaystyle\geq\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\Bigr{)}^{2} \biggl{(}\frac{1}{5}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}^{\prime} }\biggr{)}$ $\displaystyle\geq\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\Bigr{)}^{2} \biggl{(}\frac{1}{5}\times\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\lambda}}}{5}\biggr{)}$ $\displaystyle=\Bigl{(}\frac{({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}})^{2}}{2 5{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}}\Bigr{)}\,.$ $$

So, a $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-separation implies

$$ $(2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta})^{2}=\frac{({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}V_{\mathcal{F}}})^{2}}{25{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\lambda}}}\Longrightarrow{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\lambda}}=\Bigl{(}\frac{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}V_{\mathcal{F}}}}{10{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}}\Bigr {)}^{2}\,.$ $$

Then, we can 1. use the result of Lemma B.8 that $\log(\#{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{0})$ denotes as the log of the cardinality of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{0}$ is at least $\log{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}\choose{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\lambda}}}\geq({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\lambda}}/4)\log({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{S}})$ and 2. plugin the value of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}$ that gives

$$ $\displaystyle\log(\#{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{0})$ $\displaystyle\geq\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{20{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}}\Bigr{)}^{2}\log ({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}})$ $\displaystyle=\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{20{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}}\Bigr{)}^{2}\log ({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d})\,.$ $$

Based on the formula ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}=({\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}/10{\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\delta})^{2}$, when ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}$ is fixed, it is evident that as ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$ decreases, ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}$ increases. Moreover, since ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}\leq{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}/10$, we need to assume that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}$ is large enough. Consequently, for small values of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$, a sufficiently wide network becomes necessary. This observation is particularly interesting as it provides valuable insights into selecting an appropriate width for the network based on the input dimension. The larger the input dimension ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}$, the wider the network should be.

### Step 2: Deriving a lower bound for log ⁡ ( # ⁢ ℱ ℬ L ) # subscript ℱ subscript ℬ L \log(\#{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}%
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}%
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}%
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]%
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}%
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}%
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}%
\pgfsys@color@gray@fill{0}L}}}}}}}) roman_log ( # caligraphic_F start_POSTSUBSCRIPT caligraphic_B start_POSTSUBSCRIPT roman_L end_POSTSUBSCRIPT end_POSTSUBSCRIPT )

For the second step, our aim is to lower bound the $\log$ of the cardinality of the function class ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$ using the result of the first step. Since we define ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{0}$ as a subclass of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$, we can conclude that the lower bound established for $\log(\#{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{0})$ in the first step also serves as a lower bound for $\log(\#{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}})$. We then can get

$$ $\displaystyle\log{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}\bigl{ (}2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta},{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{ \operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}},|\!|\cdot |\!|_{L_{2}}\bigr{)}\geq\Bigl{(}\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\tau}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}V_{\mathcal{F}}}}{20{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}}\Bigr{)}^{2}\log ({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d})\,,$ $$

as desired.

###### Remark 4.1 (Constructing a vector in ${{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{A}}}$ using non-negative weights) .

Consider an example where ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}=3$, width of the network is set to four, and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}=2$. Our aim is constructing a binary vector with the length four (where ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\lambda}}$ of those are ones) by a factor of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}\in(0,\infty)$. To get our desired final vector, the weights could be as follows:

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{3}=\begin{bmatrix}0& \bm{c}&0&\bm{c}\end{bmatrix},~{}~{}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}W}^{2}={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{1}=\begin{bmatrix}0& 0&0&0\\ 0&\bm{c}&0&0\\ 0&0&0&0\\ 0&0&0&\bm{c}\\ \end{bmatrix}\,.$ $$

Then

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}W}^{3}{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}W}^{2}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}W}^{1}=\begin{bmatrix}0&\bm{c}&0&\bm{c}\end{bmatrix}\begin{bmatrix}0&0&0&0 \\ 0&\bm{c}&0&0\\ 0&0&0&0\\ 0&0&0&\bm{c}\\ \end{bmatrix}\begin{bmatrix}0&0&0&0\\ 0&\bm{c}&0&0\\ 0&0&0&0\\ 0&0&0&\bm{c}\\ \end{bmatrix}=\tau\begin{bmatrix}0&\bm{1}&0&\bm{1}\end{bmatrix}\,.$ $$

Now, for example if we set $c=1/2$, then we get ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tau}=1/8$.

∎

###### Remark 4.1 (Constructing a vector in 𝒜 𝒜 {{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}%
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{A}}} caligraphic_A using non-negative weights) .

## 5 Empirical studies

This section supports our theoretical findings with simulations on benchmark datasets.
While our theoretical results were established based on feed-forward ReLU neural networks, in this empirical study, we extend our investigation to include both ReLU feed-forward networks and ReLU CNNs. The focus is on understanding whether the generalization error of ReLU neural networks scales more significantly with a $1/{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$-rate or a $1/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}$-rate. We use prediction error of our test data as an estimate of the “generalization error” (as defined in Section 2) for the trained ReLU networks.

For our experiments, we consider both classification and regression tasks. The datasets used include MNIST, Fashion-MNIST and CIFAR10 for classification, and the California Housing Prices (CHP) dataset for regression analysis. We use Cross-entropy (CE) and Mean-squared (MS) error as loss functions for classification and regression datasets, respectively. The implementation of these neural networks was carried out using the TensorFlow library (see Appendix D for further details). It is also important to note that for none of these real-world datasets do we consider any assumptions such as Gaussian distributions or other assumptions.

We conduct our experiments in two steps: in the first step, we compute the prediction error of our test data for a trained network. In the second step, we determine the appropriate curve (either $1/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}$ or $1/{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$ scales) that best fits the test error values. To address the impact of various factors like network depth $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L})$, the number of parameters ($P$) and the width of the hidden layers, we consider appropriate curves $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c_{1}+{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\alpha}/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}})$ and $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c_{2}+{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\beta}/{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}})$ with ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\alpha},{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}\beta},{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}c_{1}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c_{2}}~{}\in~{}(0,\infty)$.
Optimizing these parameters is achieved through the Sequential Least Squares Quadratic Programming (SLSQP) method (Kraft, ) and the minimize function from scipy.optimize is employed for SLSQP implementation. The objective function calculates the sum of squared differences between the generalization error of a neural network and two separate curves. The optimization process aims to find the values for the coefficients that minimize this sum of squared differences. We begin this investigation with the MNIST dataset.

#### MNIST

The MNIST dataset contains $60\,000$ training images and $10\,000$ testing images, both with dimensions of $28\times 28$ pixels.
In our experiments, we have increased the number of training samples by a factor of 500.

We explore various neural network architectures, including both shallow and deep ReLU feed-forward networks, along with a CNN ReLU. Figure 1(a) illustrates the result of our analysis for a shallow neural network with the width of 128, while Figure 1(b) illustrates the result for a four-hidden layer ReLU feed-forward neural network with a uniform width of 128. Additionally, Figure 1(c) illustrates the result for a CNN ReLU. Our results reveal that, across these different architectures, the curve $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c_{1}+{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\alpha}/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}})$ provides a better fit to model the generalization error behavior compared to the curve $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c_{2}+{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\beta}/{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}})$.

Figure: (a)
Refer to caption: extracted/5596775/Figures/FNN_MNIST_Shallow.png

#### Fashion-MNIST

The Fashion-MNIST dataset contains $60\,000$ training images and $10\,000$ testing images, both with dimensions of $28\times 28$ pixels.
In our experiments, we have increased the number of training samples by a factor of 500 and examined with three different architectures. Figure 2(a) illustrates the result of our analysis for a shallow neural network with the width of 100, while Figure 2(b) illustrates the result for a three-hidden layer ReLU feed-forward neural network with a uniform width of 100. Additionally, Figure 2(c) illustrates the result for a CNN ReLU. Our results reveal that, across these different architectures, the curve $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c_{1}+{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\alpha}/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}})$ provides a better fit to model the generalization error behavior compared to the curve $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c_{2}+{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\beta}/{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}})$.

Figure: (a)
Refer to caption: extracted/5596775/Figures/Fashinmnist_FNN_Shallow.png

#### CIFAR10

The CIFAR10 dataset contains $50\,000$ training images and $10\,000$ testing images, both with dimensions of $32\times 32$ pixels.

In our experiments, we have increased the number of training samples by a factor of 500.
We explore various neural network architectures, including both ReLU feed-forward networks and CNN ReLU. Figure 3(a) illustrates the result of our analysis for a shallow ReLU neural network with the width of 1000, and Figure 3(b) illustrates the result for a CNN ReLU network. Our results reveal that, across these different architectures, the curve $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c_{1}+{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\alpha}/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}})$ provides a better fit to model the generalization error behavior compared to the curve $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c_{2}+{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\beta}/{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}})$.

Figure: (a)
Refer to caption: extracted/5596775/Figures/Cifar_Shallow1000.png

#### California housing prices ( CHP )

The version considered in this study comprises 8 numeric input attributes and a dataset of $20\,640$ samples. These samples were randomly divided into $15\,000$ for the training data and the remaining for the test data. The batch size for the training samples is set to 20. For this dataset, we increased the number of training samples by a factor of $200$.
Figure 4 reveals that for a five-hidden layer ReLU feed-forward neural network with a uniform width of 300, the curve $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c_{1}+{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\alpha}/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}})$ provides a better fit to model the generalization error behavior compared to $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c_{2}+{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\beta}/{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}})$.

Figure: Figure 4: Comparative analysis of the strength of two curves $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c_{1}+{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\alpha}/\sqrt{{\color[rgb]{0,0,0}\definecolor[named] {pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}n}}})$ and $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c_{2}+{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\beta}/{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}})$ to model the generalization error of a five-hidden layer network with a uniform width of 300) for CHP dataset
Refer to caption: extracted/5596775/Figures/housing_FNN_Deep.png

All the results for both classification and regression suggest that the generalization error scales more significantly with a $1/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}$-rate rather than a $1/{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$-rate. This observations hold true for both ReLU feed-forward networks and ReLU CNNs.

## 6 Conclusion

This paper uses Fano’s inequality to
establish a mini-max risk lower bound for ReLU feed-forward neural networks.
The bound scales at the rate $\sqrt{\log({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d})/{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}$.
Our empirical findings support this conclusion and indicate that for both
regression and classification problems, the generalization error of ReLU neural networks scales at the rate $1/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}$.
More specifically, our theories and empirical results demonstrate that the generalization error of a ReLU feed-forward network cannot be improved beyond the rate $1/\sqrt{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}$ in general.
Our theory is also much closer to current practice than earlier works:
for example, we focus on deep networks rather than shallow ones (S. Du et al., ; Klusowski & Barron, ), and we employ ReLU activation functions instead of Sinusoidal (Klusowski & Barron, ) or linear (S. Du et al., ) activation functions.

The current discussions about large language models (LLMs) illustrate that our research is extremely relevant and timely.
The training data for LLMs have steadily grown over the last years:
for example, GPT-3 was trained on about 300 billion tokens (Brown et al., ),
Chinchilla on about 1.4 trillion tokens (Hoffmann et al., ),
GPT-4 on about 13 trillion tokens (OpenAI, )
and Llama 3 on over 15 trillion tokens (Meta, ).
However, it is believed that this trend will have a natural end soon:
there will simply not be enough fresh texts any more.
A trend is, therefore, the development of smaller models, such as Phi-3-Mini (Abdin et al., ),
which are trained on less data in general and on “real” data enriched with synthetic data.
But how much data are really needed for such minimal AI-solutions?
This paper is a small step toward answering this question.

## Appendix A A high-level idea for proving our Theorem 2.2 :

We first construct a finite collection of networks {${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{1}}},
\ldots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}}}}\}
\subseteq{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$ such that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}({\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}},{\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}})\geq~{}2{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\delta}$ for all $j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]$ and $j\neq k$. Then, we find an upper bound for the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ of the corresponding family of distributions $\{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{1}}}},
\ldots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}}}}}\}$ along with finding a lower bound for packing of the $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-packing sets. Finally, we employ a variants of Fano’s method well-known as “local packing” or “rescaling” to achieve the lower bound for the mini-max risk of ReLU feed-forward networks.

## Appendix B Further technical results

In this section, we present additional technical results from the work of others and our own, that are essential for the proof of Theorem 2.2’s components but might also be of interest by themselves. We divide the results into two main parts. The first part includes Lemma B.5 and a few other results that are contained in the proof of this lemma, and the second part includes a few results, both from our work and others’ to prove Lemma 3.2. Before providing those preliminary results, we provide the general formula for Fano’s inequality (Wainwright, , Proposition 15.12). Based on the concept of packing number, assume that $\{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{1}}}},
\ldots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}}}}}\}$ is a family of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$-product distributions (as defined in Part 1) for the corresponding neural networks ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{1}}},
\cdots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}}}}$ which satisfy ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}({\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})},{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})})\geq 2{\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\delta}$ for all $j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]$ and $j\neq k$. Then, assume that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}J}$ is uniformly distributed over the index set $[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]$ and the conditional distribution of $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Y}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}})$ given ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}J}$ defined by $(({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Y}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}})~{}|~{}{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}J}=j)\sim{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\mathbb{P}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}}$. Accordingly, Fano’s inequality can be formalized as:

###### Lemma B.1 (Fano’s Inequality) .

Let $\{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{1}}},
\ldots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}}}}\}
\subseteq{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$ be a $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-packing set with respect to ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}$. Then, for any increasing function ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}:~{}[0,\infty)
\rightarrow~{}[0,\infty)$, the mini-max risk is
lower bounded by

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{R}_{({\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n},{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}d})}({{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\mathcal{B}_{ \operatorname{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}L}}}}}}};{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\Phi}\circ{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\rho}})}\geq{\color[rgb]{0,0,0}\definecolor[named ]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\Phi}({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\delta})\biggl{(}1-\frac{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}I({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}Y}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}}|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}X}^{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}n}})}+\log 2}{\log{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\mathcal{M}}}\biggr{)}\,.$ $$

The symbol ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}Y}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}})}$ represents the mutual information between a random index ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}J}$, which is drawn uniformly from the index set $[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]$ and the samples $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Y}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}})$ drawn from the prior distribution ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}}$ corresponding to ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}
\mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}
{$\cdot$}}={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}J}}}$. The mutual information, measures how much information can be revealed about the index ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}J}$ of a $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-packing set by drawing the samples (${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Y}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}$).

###### Lemma B.1 (Fano’s Inequality) .

### Part 1: Preliminary results for upper bounding the mutual information

We present some auxiliary results that are contained in the proof of Lemma B.5. To follow these results more conveniently, we explain the necessary steps briefly. After defining the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ as a measure of distance between two probability measures, we calculate the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ between two multivariate normal distributions Hayakawa & Suzuki (, Lemma A.1). Then, we calculate ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$-product of two multivariate normal distributions and finally, we find the connection between the mutual information and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$.

The ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ between two different probability distributions $\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}$ and $\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Q}}$ on domain ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{X}}$ with densities ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}q({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})}$, respectively, can be defined as (Wainwright, , Equation 3.57)

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \bigl{(}\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}~{}||~{}\mathbb {{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Q}}\bigr{)}=\int_{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}\in{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{X}}}{\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}\log\frac{{\color[rgb]{0,0,0}\definecolor [named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}p({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{{x}}})}}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}q({\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\bm{{x}}})}}d{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}\,.$ $$

As we have ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$ data samples, we are interested to find the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ between two different ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$-product distributions. Assume that $(\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}^{1},\ldots,\mathbb{{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}})$ be a collection of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$ probability distributions, and define $\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}^{1:{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}n}}\mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{$
\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=~{}\bigotimes_{i=1}^{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}n}}\mathbb{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}P}}^{i}$ as the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$-product distributions. Define another ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$-product distribution $\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Q}}^{1:{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}n}}$ in a similar way. For the ease of notation, we define $\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}\mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{$
\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=\mathbb{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}P}}^{1:{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}}$ and $\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Q}}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}\mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{$
\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=\mathbb{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}Q}}^{1:{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}}$. Then, the connection between the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$-product distributions $\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}$ and $\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Q}}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}$ and the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ of the individual pairs (Wainwright, , Equation 15.11a), can be formalized as the following lemma:

###### Lemma B.2 (Decomposition of the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{} divergence}}}$ for ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$ -product distributions) .

For two ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$-product distributions $\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}\text{ and }\mathbb{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}Q}}^{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}}$, it holds that

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \bigl{(}\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}^{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}n}}~{}||~{}\mathbb{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}Q}}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}}\bigr{)}=\sum_{i=1}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \bigl{(}\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}^{i}~{}||~{} \mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Q}}^{i}\bigr{)}\,.$ $$

And in the case of i$\cdot$i$\cdot$d$\cdot$ product distributions — meaning that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}^{i}={\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}P}^{1}$ and $\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Q}}^{i}=\mathbb{{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Q}}^{1}$ for all $i\in~{}\{1,\ldots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}\}$— we have

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \bigl{(}\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}^{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}n}}~{}||~{}\mathbb{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}Q}}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}}\bigr{)}={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}\times{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \bigl{(}\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}^{1}~{}||~{} \mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Q}}^{1}\bigr{)}\,.$ $$

We consider short-hands $\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}$ and $\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Q}}$, for
$\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}^{1}$ and $\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Q}}^{1}$, respectively. So, the previous equation takes form

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \bigl{(}\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}^{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}n}}~{}||~{}\mathbb{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}Q}}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}}\bigr{)}={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}\times{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \bigl{(}\mathbb{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}P}}~{}||~{}\mathbb {{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Q}}\bigr{)}\,.$ $$

We then proceed to calculate the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ between two normal distributions. Consider the regression model defined in Equation (1) and the network model defined in Equation (2). We assume that the noise terms are independent and identically distributed and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}u}_{i}}\in\mathcal{N}(0,{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\sigma}^{2})$. Recall that the explanatory variables ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}}_{i}}$ follow a fixed distribution ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}}}$ and have the density ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{h({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})}}$.
Then, we define ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}}\mathrel{\hbox to
0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=({
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}},{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{y}})\in\mathbb{R}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}d}}\times\mathbb{R}$ as the joint variable of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}$. According to the conditional probability, the joint density can be written as follows:

$$ $\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}$ $\displaystyle={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}Y}|{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}X}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}|{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{h({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}}$ (6) $\displaystyle=\frac{1}{\sqrt{2\pi{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\sigma}^{2}}}e^{-\frac{({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{y}}-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})})^{2}}{2{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\sigma}^{2}}}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{h({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}}\,,$ $$

where $j\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}$ is the joint density of $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}},{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{y}})$ with regression function ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})}$. And consider ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}$ as another joint density of $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}},{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{y}})$ in the same manner with regression function ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})}$ and two distinct corresponding normal distributions ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{k}}}}$ such that have densities ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}$, respectively. Recall that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}$ are any two distinct neural networks of the neural network model defined in Section 2, which parameterized by ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}\text{ and }{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}$ ($j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]$ as any two distinct indices of the $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-packing set). Then, the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ between any two normal distributions ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{k}}}}$ can be calculated as the following lemma (Yang & Barron, ):

###### Lemma B.3 (The ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{} divergence}}}$ between two multivariate normal distributions) .

Assume any two normal distributions ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{k}}}}$ for all $j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]$ and $j\neq k$, then it holds that

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}}~{}||~{}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}})=\frac{1}{2{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\sigma}^{2}}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\int_{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}\in{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{X}}}}\bigl{(}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}-{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{)}^{2}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{h({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}d{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}}$ $$

And that

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}}~{}| |~{}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}})= \frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}{2{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\sigma}^{2}}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\int_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}\in{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{X}}}}\bigl{(}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}-{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{)}^{2}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{h({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}d{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}}\,.$ $$

To fulfill this part’s goal, we have just left to find a connection between the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ and the mutual information.

In the next lemma, we are interested to upper bounding the mutual information (Scarlett & Cevher, ) —which measures the dependence between the joint distributions and the product of the marginals of two random variables— by describing it’s connection with the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$.
Assume that under the Markov chain ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}J}\to{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{J}}\to({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}Y}^{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}}|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}X}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}})}$, a random index ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}J}$ is drawn uniformly from $\{1,\ldots,{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}\}$ and samples $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Y}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}})$ are drawn from the prior distributions ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}}$ corresponding to ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}
\mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}
{$\cdot$}}={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}J}}}$. Note that if one sample $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}Y}|{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}X})$ drawn, then we have ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}Y}|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}X})}$.

There are many tools to upper bounding the mutual information and the most straight forward tools is based on the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ (Wainwright, , Equation 15.34) as follows:

###### Lemma B.4 (The connection between the mutual information and the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{} divergence}}}$ ) .

For any two distinct normal probability distributions ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{k}}}}$ for all $j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]$, it holds that

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}Y}|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}X})}\leq\frac{1}{{\color [rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}^{2}}\sum_{ \begin{subarray}{c}j,k=1\\ j\neq k\end{subarray}}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}} }{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}}~{}||~{}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}}\bigr{)}\,.$ $$

For any two distinct ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}$-product normal probability distributions ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}}$, it holds that

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}Y}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}})}\leq\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\mathcal{M}}^{2}}\sum_{\begin{subarray}{c}j,k=1\\ j\neq k\end{subarray}}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}} }{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}}~{}||~{}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}}\bigr{)}\,.$ $$

###### Lemma B.5 (Upper bounding ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}Y}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}})}$ of the $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$ -packing of network’s space ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named] {pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}L}}}}}}}$ ) .

For all possible pairs of two distinct networks ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}},{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}\in{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$ satisfy ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}({\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})},{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})})\geq 2{\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\delta}$, the mutual information ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}Y}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}})}$ is upper bounded by

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}Y}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}})}\leq\frac{2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb} {0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}({\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\kappa}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\delta})^{2}}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\sigma}^{2}}\,,$ $$

for a suitable ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\kappa}\in[1,\infty)$, such that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}({\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})},{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})})\leq 2{\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\kappa}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\delta}$.

###### Lemma B.2 (Decomposition of the KL ⁢ divergence KL divergence {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}%
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}%
divergence}}} roman_KL roman_divergence for n 𝑛 {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}%
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n} italic_n -product distributions) .

###### Lemma B.3 (The KL ⁢ divergence KL divergence {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}%
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}%
divergence}}} roman_KL roman_divergence between two multivariate normal distributions) .

###### Lemma B.4 (The connection between the mutual information and the KL ⁢ divergence KL divergence {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}%
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}%
divergence}}} roman_KL roman_divergence ) .

###### Lemma B.5 (Upper bounding I ⁢ ( J ; Y n | X n ) 𝐼 𝐽 conditional superscript 𝑌 𝑛 superscript 𝑋 𝑛 {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}%
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0}%
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}%
\pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{%
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill%
{0}Y}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}%
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0}%
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}%
\pgfsys@color@gray@fill{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{%
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill%
{0}n}})} italic_I ( italic_J ; italic_Y start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT | italic_X start_POSTSUPERSCRIPT italic_n end_POSTSUPERSCRIPT ) of the 2 ⁢ δ 2 𝛿 2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}%
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta} 2 italic_δ -packing of network’s space ℱ ℬ L subscript ℱ subscript ℬ L {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}%
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}%
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}%
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]%
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}%
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}%
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}%
\pgfsys@color@gray@fill{0}L}}}}}}} caligraphic_F start_POSTSUBSCRIPT caligraphic_B start_POSTSUBSCRIPT roman_L end_POSTSUBSCRIPT end_POSTSUBSCRIPT ) .

### Part 2: Preliminary results for deriving a lower bound for packing number of ReLU networks

In this section, we present supporting lemmas that are included in the proof of Lemma 3.2 for deriving the lower bound for the packing number of our defined ReLU network’s space. We start by calculating the Gaussian integrals over a half-space. Assume that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}$ is a realization of random variable ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}X}$ that follows the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}$-dimensional Gaussian distribution, then we say that for $k\in\{1,\ldots{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}\}$, ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}^{\top}_{k}{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}>0$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}^{\top}_{k}{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}\leq 0$ are two half-spaces of hyperplane ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}^{\top}_{k}{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}=0$ for ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}\in\mathbb{
R}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}}$. We then can define the probability density function of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}$ with mean vector ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{\mu}}}$ and the covariance matrix ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{\Sigma}}}$ as follows:

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p}({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}},{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\bm{\mu}}},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{\Sigma}}})= \frac{1}{(2\pi)^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}/2}\sqrt{\lvert{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{\Sigma}}}\rvert}}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\int_{{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}\in{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{X}}}}e^{\frac{-({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{{x}}}-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{\mu}}})^{\top}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{\Sigma}}}^{-1}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}-{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\bm{\mu}}})}{2}}d{\color[rgb]{0,0,0}\definecolor [named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}\,,$ $$

where $\lvert{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{\Sigma}}}\rvert
\equiv\det({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{\Sigma}}})$, is the determinant of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{\Sigma}}}$.

If ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{\mu}}}=\bm{0}$, then we have

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p}({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}},\bm{0},{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{\Sigma}}})=\frac{1}{(2\pi)^{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}d}/2}\sqrt{\lvert{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{\Sigma}}}\rvert}}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\int_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{{x}}}\in{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{X}}}}e^{ \frac{-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}^{\top}{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{\Sigma}}}^{-1}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}}{2}}d{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}\,.$ $$

Accordingly, the probability density function of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}$ on either half-space ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}^{\top}_{k}{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}>0$ or ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}^{\top}_{k}{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}\leq 0$ takes the form

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p}({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{b}}}^{\top}_{k}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}>0,\bm{0},{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{\Sigma}}})=\frac{1}{(2\pi)^{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}d}/2}\sqrt{\lvert{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{\Sigma}}}\rvert}}\int_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{b}}}^{\top}_{k}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}>\bm{0}}e^{\frac{-{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}^{\top}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{\Sigma}}}^{-1}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}}{2}}d{\color[rgb]{0,0,0}\definecolor[named ]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}\,,$ $$

and can be calculated as the following lemma:

###### Lemma B.6 (Gaussian integrals over a half-space) .

Assume that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}},{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}\in\mathbb{R}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}d}}$ and for a fixed vector ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}$, we define a half-space ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}^{\top}_{k}{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}>0$. Then, for the corresponding probability density function it holds that

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p}({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{b}}}^{\top}_{k}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}>0,\bm{0},{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{\Sigma}}})=\frac{1}{2}\,.$ $$

In the next lemma we are motivated to employ the result of Lemma B.6 to calculate $\mathbb{E}[({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\phi}}}}({\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}))^{
2}]$ which is necessary for the proof of Lemma 3.2.

###### Lemma B.7 (The expectation of squared ReLU functions) .

Let ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}$ be a Gaussian random variable and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}^{\top}_{k}{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}>0$ is a half-space, then,

$$ $\mathbb{E}\Bigl{[}\bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\phi}}}}({\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})\bigr{)}^{2} \Bigr{]}=\frac{1}{2}\,.$ $$

In the following lemma, we present Klusowski & Barron (, Lemma1), which concerns the cardinality of a set and is integral to the proof of Lemma 3.2. This lemma helps us define our desired set with a predefined Hamming weight, and its elements can be interpreted as binary codes. Now, let state the lemma.

###### Lemma B.8 (Cardinality of a binary set) .

For integers ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}^{\prime}$ with ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}\in[10,\infty)$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}^{\prime}\in[1,{\color
[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}/10]$, define a set

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{S}}\mathrel{ \hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$} }=\bigl{\{}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}\in\{0,1\}^{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}}:\hskip 1.42262pt\!| \!|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{w}}|\!|_{1}={\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}^{\prime}\bigr{\}}\,.$ $$

Then, there exists a subset ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{A}}\subset{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{S}}$ with cardinality at least ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{S}}\mathrel{\hbox to0.0
pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=\sqrt{
\genfrac{(}{)}{0.0pt}{2}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor
}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}}{{\color
[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}^{\prime}}}$ such that each element has Hamming weight ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}^{\prime}$ and any pairs of elements have minimum Hamming distance ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}^{\prime}/5$.

###### Lemma B.6 (Gaussian integrals over a half-space) .

###### Lemma B.7 (The expectation of squared ReLU functions) .

###### Lemma B.8 (Cardinality of a binary set) .

## Appendix C Proofs

Here, we provide the proofs of the lemmas found in Appendix B.

### C.1 PROOF OF Lemma B.3

###### Proof.

To calculate the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ between two normal distributions ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{k}}}}$ of a continuous random variable, each with the corresponding densities ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}$, $\text{for all }j,k\in~{}[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor
}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}
}]\text{ where }j\neq k$, we can 1. use the definition of the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$, 2. plug the value of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}$ in, 3. perform some simplification, 4. apply the definition of expected value, 5. the linearity of expected value, 6. use ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}={\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}+{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}u}$, 7. perform further rewriting, 8. apply the linearity of expected value, assuming independence between each ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}u}_{i}}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}}_{i}}$, 9. cancel out the second term ( $\mathbb{E}[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}u}]=0$) and 10. recognize that only ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}$ values remain, to get

$$ $\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}}( {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}}~{}||~{}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}})$ $\displaystyle={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\int_{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{X}}\times{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{Y}}}}{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}\log \frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}}d{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}}$ $\displaystyle={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\int_{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{X}}\times{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{Y}}}}{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}\log \Biggl{(}\frac{\bigl{(}1/\sqrt{2\pi{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\sigma}^{2}}\bigr{)}e^{-\bigl{(}({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{y}}-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})})^{2}/2{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\sigma}^{2}\bigr{)}}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{h({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{{x}}})}}}{\bigl{(}1/\sqrt{2\pi{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\sigma}^{2}}\bigr{)}e^{-\bigl{(}({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{y}}-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})})^{2}/2{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\sigma}^{2}\bigr{)}}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{h({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{{x}}})}}}\Biggr{)}d{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{z}}$ $\displaystyle={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\int_{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{X}}\times{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{Y}}}}{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})} \frac{1}{2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\sigma}^{2}}\Bigl{(} \bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}-{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})} \bigr{)}^{2}-\bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}-{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{)}^{2}\Bigr{)}d{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{z}}$ $\displaystyle=\mathbb{E}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{z}}\sim{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{z}})}}\biggl{[}\frac{1}{2{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\sigma}^{2}}\Bigl{(}\bigl{(}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{y}}-{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{)}^{2}- \bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}-{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})} \bigr{)}^{2}\Bigr{)}\biggr{]}$ $\displaystyle=\frac{1}{2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\sigma}^{2}} \mathbb{E}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}}\sim{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}}\Bigl{[}\bigl{ (}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}-{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})} \bigr{)}^{2}-\bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}-{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{)}^{2}\Bigr{]}$ $\displaystyle=\frac{1}{2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\sigma}^{2}} \mathbb{E}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}}\sim{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}}\Bigl{[}\bigl{ (}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}+{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}u}-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{)}^{2}-({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}u})^{2}\Bigr{]}$ $\displaystyle=\frac{1}{2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\sigma}^{2}} \mathbb{E}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}}\sim{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}}\Bigl{[}\bigl{ (}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}-{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{)}^{2}- 2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}u}\bigl{(}{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}-{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{)}\Bigr{]}$ $\displaystyle=\frac{1}{2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\sigma}^{2}} \biggl{(}\mathbb{E}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}}\sim{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}} \Bigl{[}\bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}-{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{)}^{2} \Bigr{]}$ $\displaystyle\hskip 80.00012pt-2\mathbb{E}[{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}u}]\mathbb{E}_{{\color[rgb]{0,0,0}\definecolor[named ]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{z}}\sim{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}p_{f_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{z}})}}\bigl{[}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}-{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{]}\biggr{)}$ $\displaystyle=\frac{1}{2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\sigma}^{2}} \biggl{(}\mathbb{E}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}}\sim{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p_{f_{{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{z}})}} \Bigl{[}\bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}-{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{)}^{2} \Bigr{]}\biggr{)}$ $\displaystyle=\frac{1}{2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\sigma}^{2}} {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\int_{{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}\in{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{X}}}}\bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}-{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{)}^{2}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{h({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{{x}}})}}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\bm{{x}}}}\,.$ $$

Furthermore, by combining this result with Lemma B.2’s result, it holds that for all $j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]$ and $j\neq k$

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}}~{}| |~{}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}^{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}})= \frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}{2{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\sigma}^{2}}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\int_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}\in{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{X}}}}\bigl{(}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}-{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{)}^{2}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{h({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}d{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}}\,,$ $$

as desired.
∎

###### Proof.

### C.2 PROOF OF Lemma B.4

###### Proof.

Consider a family of distributions $\{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{1}}}},\ldots,{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}}}}}\}$,
then ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}Y}|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}X})}$ with respect to ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}J}\to{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{J}}\to({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}Y}|{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}X})}$, can be defined by using the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ —as the underlying measure of distance— Wainwright (, Equation 15.29)

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}Y}|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}X})}\mathrel{\hbox to0.0 pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}={\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \bigl{(}\mathbb{Q}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}({\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}X},{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}Y})},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}J}}~{}||~{}\mathbb{Q}_{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}X},{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}Y})}}\mathbb{Q}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}J}}\bigr{)}\,,$ $$

where $\mathbb{Q}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}X},{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}Y})},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}J}}$ is the joint distribution of the pair $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}X},{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}Y})},{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}J})$ and $\mathbb{Q}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}X},{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}Y})}}\mathbb{Q}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb
}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}J}}$ is the product of their marginals, and assume that $\overline{\mathbb{Q}}\equiv\mathbb{Q}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}X},{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}Y})}}\mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{$
\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=1/{\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{M}}\sum_{j=1}^{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{M}}}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}}$ is the mixture distribution. Then, ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}Y}|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}X})}$ can be written in terms of component distributions $\{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}},j\in[{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\mathcal{M}}]\}$ as follows:

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}Y}|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}X})}=\frac{1}{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}}\sum_{j=1}^ {{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}}{\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}}~{}||~{}\overline{\mathbb{Q}}\bigr{)}\,.$ $$

Intuitively, it means the mean the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ between ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}}$ and $\overline{\mathbb{Q}}$- averaged over the choice of index $j$- gives the mutual information. Furthermore, based on the definition of the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$, we can conclude that for $j=k$

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}}~{}||~{}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}}\bigr{)}=0\,.$ $$

Accordingly, we can 1. employ the mixture distribution formula in the above equation, 2. use the convexity of the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$ and apply Jensen inequality and 3. use the linearity property of sum to obtain

$$ $\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}Y}|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}X})}$ $\displaystyle=\frac{1}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}} \sum_{j=1}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}}{\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \biggl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}}~{}||~{}\frac{1}{{\color[rgb]{0,0,0}\definecolor[named] {pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\mathcal{M}}}\sum_{k=1}^{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\mathcal{M}}}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}}\biggr{)}$ $\displaystyle\leq\frac{1}{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{M}}}\Biggl{(}\sum_{j=1}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{M}}}\biggl{(}\frac{1}{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{M}}}\sum_{k=1}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{M}}}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{ KL}}}\bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}}~{}||~{}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}}\bigr{)}\biggr{)}\Biggr{)}$ $\displaystyle=\frac{1}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}^ {2}}\sum_{\begin{subarray}{c}j,k=1\\ j\neq k\end{subarray}}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}} }{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}}~{}||~{}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}}\bigr{)}\,.$ $$

Consequently, if we can construct a $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-packing set such that all two distinct pairs of distributions ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{k}}}}$
are close in average, then the mutual information can be controlled.

For the second claim, we employ the previous view with the result of Lemma B.2 to get

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}Y}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}})}\leq\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\mathcal{M}}^{2}}\sum_{\begin{subarray}{c}j,k=1\\ j\neq k\end{subarray}}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}} }{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}}~{}||~{}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}}\bigr{)}\,,$ $$

as desired.
∎

###### Proof.

### C.3 PROOF OF LEMMA B.5

###### Proof.

The aim of this proof is to establish an upper bound on the mutual information ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}Y}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}n}})}$, for the $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-packing within the neural network’s space ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathcal{F}}_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\mathcal{B}_{\operatorname{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}L}}}}}}}$. To achieve this, we invoke the result of Lemma B.4, which establishes a connection between the mutual information and the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\operatorname{KL~{}
divergence}}}$.
We then apply the result obtained from Lemma B.3. Finally, we employ the same re-scaling procedure as demonstrated in Wainwright (, Example 15.14) and Wainwright (, Example 15.16) to construct a $2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\delta}$-packing in such a way that, for a suitable constant ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\kappa}\in[1,\infty)$, we ensure that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}({\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})},{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})})\leq 2{\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\kappa}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\delta}$ holds for all pairs ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})}$ corresponding to $j\neq k\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]$.

We can 1. use the result provided by Lemma B.4, 2. use the fact that $\sum_{j,k=1}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}}{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}}({
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{j}}}}~{}||~{}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\bm{\Theta}}}^{k}}}})\leq{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}\mathcal{M}}\choose 2}\underset{k,j}{\sup}({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}}({\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}}~{}||~{}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}}))$ for all $j\neq~{}k\in~{}[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{
0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]$, 3. calculate the permutation, 4. some arithmetic calculation, 5. use the fact that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}\geq 1$, so $0\leq~{}{({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}-1)}/{{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}}<~{}1$, 6. use the view of Lemma B.3, 7. invoke the definition of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\rho}$ as $L_{2}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}}})$- norm, 8. employ the re-scaling procedure and 9. simplify the factor 2 to obtain

$$ $\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}I({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}J};{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}Y}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}|{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}X}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}n}})}$ $\displaystyle\leq\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}^{2}}\sum_{ \begin{subarray}{c}j,k=1\\ j\neq k\end{subarray}}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}} }{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}D_{\operatorname{KL}}} \bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathbb{P}_{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}}~{}||~{}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}}\bigr{)}$ $\displaystyle\leq\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}^{2}}{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}\choose 2} \sup_{\begin{subarray}{c}j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{M}}]\\ j\neq k\end{subarray}}\Bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}D_{\operatorname{KL}}}\bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}}~{}||~{}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}}\bigr{)}\Bigr{)}$ $\displaystyle=\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}{{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}^{2}}\frac{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}!}{({\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}-2)!}\sup_{ \begin{subarray}{c}j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{M}}]\\ j\neq k\end{subarray}}\Bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}D_{\operatorname{KL}}}\bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}}~{}||~{}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}}\bigr{)}\Bigr{)}$ $\displaystyle=\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}({\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\mathcal{M}}-1)}{{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\mathcal{M}}}\sup_{\begin{subarray}{c}j,k\in[{\color [rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]\\ j\neq k\end{subarray}}\Bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}D_{\operatorname{KL}}}\bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}}~{}||~{}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}}\bigr{)}\Bigr{)}$ $\displaystyle\leq{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}\sup_{\begin{ subarray}{c}j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb} {0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]\\ j\neq k\end{subarray}}\Bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}D_{\operatorname{KL}}}\bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{j}}}}~{}||~{}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathbb{P}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}^{k}}}}\bigr{)}\Bigr{)}$ $\displaystyle=\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}{2{\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\sigma}^{2}}\sup_{\begin {subarray}{c}j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]\\ j\neq k\end{subarray}}\biggl{(}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\int_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}\in{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{X}}}}\bigl{(}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}-{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{)}^{2}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{h({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}d{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}}\biggr{)}$ $\displaystyle=\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}{2{\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\sigma}^{2}}\sup_{\begin {subarray}{c}j,k\in[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{M}}]\\ j\neq k\end{subarray}}\Bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\rho}\bigl{(}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0 }\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}^{j}}}({\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})},{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f}_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}^{k}}}({ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})}\bigr{)}^{2} \Bigr{)}$ $\displaystyle\leq\frac{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}(2{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\kappa}{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\delta})^{2}}{2{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\sigma}^{2}}$ $\displaystyle=\frac{2{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}({\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\kappa}{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\delta})^{2}}{{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\sigma}^{2}}\,,$ $$

as desired.
∎

###### Proof.

### C.4 PROOF OF LEMMA B.6

###### Proof.

In this proof, we first define a rotation matrix ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}\in\text{SO}({
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d})$, which belongs to the special orthogonal group. Then, based on the fact that ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}^{\top}={\color
[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}^{-1}$, we can write ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}^{\top}_{k}{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}=~{}{\color[rgb
]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}^{\top}_{k}{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}^{-1}{\color[
rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}\bm{{x}}}=~{}({\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\bm{R}}}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\bm{b}}}_{k})^{\top}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}
{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}$. Accordingly, we can obtain

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p}({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{b}}}^{\top}_{k}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}>0,\bm{0},{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{\Sigma}}})=\frac{1}{(2\pi)^{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}d}/2}\sqrt{\lvert{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{\Sigma}}}\rvert}}\int_{({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{R}}}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\bm{b}}}_{k})^{\top}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}>\bm{0}}e^{ \frac{-({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\bm{{x}}})^{\top}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{R}}}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\bm{\Sigma}}}^{-1}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}^{ \top}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\bm{{x}}})}{2}}d{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}\,.$ $$

By defining ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{G}}}\mathrel{\hbox
to
0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}={
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}\bm{{x}}}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tilde{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\bm{b}}}}}_{k}\mathrel{\hbox to0.0pt{\raisebox{1
.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}={\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\bm{R}}}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\bm{b}}}_{k}$ and $d{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}=(d{\color[rgb]
{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}/d{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\bm{G}}})\times d{\color[rgb]{0,0,0}\definecolor
[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\bm{G}}}$, we get

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p}({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{b}}}^{\top}_{k}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}>0,\bm{0},{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{\Sigma}}})=\frac{1}{(2\pi)^{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}d}/2}\sqrt{\lvert{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{\Sigma}}}\rvert}}\int_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\tilde{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\bm{b}}}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{G}}}> \bm{0}}e^{\frac{-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{G}}}^{\top}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\bm{\Sigma}}}^{-1}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{R}}}^{\top}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{G}}}}{2}}\Bigl{(}\frac{d{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}}{d{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\bm{G}}}}\Bigr{)}\times d{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\bm{G}}}\,.$ $$

Then, 1. by setting ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tilde{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\bm{\Sigma}}}}}\mathrel{\hbox to0.0pt{\raisebox{
1.29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}={\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\bm{R}}}{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\bm{\Sigma}}}^{-1}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}^{\top}$, $(d{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}/d{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\bm{G}}})\mathrel{\hbox to0.0pt{\raisebox{1.2916
7pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=\lvert{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\bm{R}}}\rvert$ ($\lvert{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}\rvert\equiv
\det({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}})$) and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tilde{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}{\bm{b}}}}}_{k}=~{}(|\!|{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}|\!|_{2},0,\ldots,0)^{\top}$, 2. by factoring out the term $\lvert{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}\rvert$, 3. the fact that the probability density function of a Gaussian distribution for a random variable across its domain is 1 and 4. by noting that $\lvert{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{R}}}\rvert=1$, we can obtain

$$ $\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0 }\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}p}({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{b}}}^{\top}_{k}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}}>0,\bm{0},{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{\Sigma}}})$ $\displaystyle=\frac{1}{(2\pi)^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}d}/2}\sqrt{\lvert{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{\Sigma}}} \rvert}}\int_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tilde{{\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}}}_{k}^{\top}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{G}}}>\bm{0}}e^{ \frac{-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{G}}}^{\top}{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tilde{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\bm{\Sigma}}}}}^{-1}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{G}}}}{2}}\rvert{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{R}}}\lvert d{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{G}}}$ $\displaystyle=\frac{\rvert{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\bm{R}}}\lvert}{(2\pi)^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}d}/2}\sqrt{\lvert{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{\Sigma}}} \rvert}}\int_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tilde{{\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}}}_{k}^{\top}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{G}}}>\bm{0}}e^{ \frac{-{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{G}}}^{\top}{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\tilde{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\bm{\Sigma}}}}}^{-1}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{G}}}}{2}}d{\color[rgb]{0,0,0}\definecolor[named ]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{\bm{G}}}$ $\displaystyle=\frac{\rvert{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\bm{R}}}\lvert}{2}$ $\displaystyle=\frac{1}{2}\,,$ $$

as desired.
∎

###### Proof.

### C.5 PROOF OF LEMMA B.7

###### Proof.

In this proof, our objective is to compute $\mathbb{E}[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\phi}}}}({\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}){
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\phi}}}}({\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{
rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})]$, which is a crucial component of the proof presented in Lemma 3.2, helping us establish a lower bound for a ReLU neural network. To achieve this, we can 1. employ the definition of expected value, 2. apply the definition of ReLU function, 3. perform some rewriting, 4. take out ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}$ and ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}$, 5. exploit the symmetry of ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}$ (Lemma B.6), 6. employ the definition of expectation, 7. apply the fact that $\mathbb{E}[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}\bm{{x}}}^{\top}]=\bm{I}_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}d}}$, 8. apply ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}\bm{
I}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}}~{}{\color[rgb]{0,0,0
}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}={\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[
named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}$ and 9. use ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}=1$ to obtain

$$ $\displaystyle\mathbb{E}\bigl{[}{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\phi}}}}({\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}){\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\phi}}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{ \top}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})\bigr{]}$ $\displaystyle=\int_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\mathcal{X}}}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\phi}}}}({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}){ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\phi}}}}({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}){ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{h({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}}d{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{{x}}}$ $\displaystyle=\int_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}:{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}>0}({\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})^{2}{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{h({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}}d{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{{x}}}$ $\displaystyle=\int_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}:{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}>0}({\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})({\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})^{\top}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{h({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{{x}}})}}d{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}$ $\displaystyle={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{ \top}\biggl{(}\int_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb }{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}:{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{\top}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}>0}{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}\bm{{x}}}^{\top}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}{h({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{{x}}})}}d{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}\biggr{)} {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}$ $\displaystyle={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{ \top}\biggl{(}\frac{\int_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\mathcal{X}}}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}^{\top}{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{h({\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})}}d{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{{x}}}}{2}\biggr{)}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}$ $\displaystyle={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}^{ \top}\frac{\mathbb{E}[{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{ rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}^{\top}]}{2}~{} {\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}$ $\displaystyle=\frac{1}{2}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{ k}^{\top}\bm{I}_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}d}}~{}{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}$ $\displaystyle=\frac{1}{2}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor }{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{ k}^{\top}{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{\bm{b}}}_{k}$ $\displaystyle=\frac{1}{2}\,,$ $$

as desired.
∎

###### Proof.

## Appendix D Empirical details

Here, we first explain the two loss functions employed for training the networks in both classification and regression datasets and we then provide some more details about our implementations’ setting in more detail.

### D.1 Loss functions:

In the training process of ReLU networks, we utilize two distinct loss functions implemented in TensorFlow: “mean_squared_error” for regression and “categorical_crossentropy” for classification. The details of these loss functions are as follows:

#### Cross-entropy loss

For classification purpose, we use (categorical) Cross-entropy and define it as  (Murphy, ; Ho & Wooky, )

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\ell_{{CE}}}~{}({\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}})~{}\mathrel{\hbox to0.0pt{\raisebox{1 .29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=-\frac{1}{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}n}}\sum_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}k}=1}^{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}m}}\sum_{i=1}^{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}\Bigl{(}({\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}_{i})_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}k}}\log p\bigl{(}{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}f_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}}_{i}),{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}k}\bigr{)}\Bigr{)}\,,$ $$

where
$({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}_{i})_{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}k}}$ is the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}k}$-th element of the one-hot vector of the target label for the $i$-th data sample and

$$ $p({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}({\color[rgb]{0,0,0}\definecolor[named] {pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})},{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}k})~{}\mathrel{\hbox to0.0pt{\raisebox{1.29167pt}{$\cdot$}\hss}\raisebox{-1 .29167pt}{$\cdot$}}=~{}\frac{e^{({\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}f_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}}({\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})})_{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}k}}}}{\sum_{i=1}^{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}m}}e^{({\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}f_{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}\bm{\Theta}}}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{{x}}})})_{i}}}\,,$ $$

where $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}({\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}})})_{{\color[rgb]{0,0,0}\definecolor[named]
{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}k}}$ is the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}k}$-th output of a network indexed by ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}$.

#### Mean-squared error

For regression, we use Mean-squared and define it as (Botchkarev, )

$$ ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\ell_{{MS}}}~{}({\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}})~{}\mathrel{\hbox to0.0pt{\raisebox{1 .29167pt}{$\cdot$}\hss}\raisebox{-1.29167pt}{$\cdot$}}=~{}\frac{1}{{\color[rgb ]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}\sum_{i=1}^{{\color[ rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}n}}\sum_{{\color[rgb]{ 0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke {0}\pgfsys@color@gray@fill{0}k}=1}^{{\color[rgb]{0,0,0}\definecolor[named]{ pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill {0}m}}~{}\frac{\Bigl{(}({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}_{i})_{{ \color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}k}}-\bigl{(}{\color[rgb] {0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f_{{\color[rgb]{0,0,0} \definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{\Theta}}}}({\color[rgb]{0,0,0}\definecolor[named ]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}\bm{{x}}})\bigr{)}_{{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}k}}\Bigr{)}^{2}}{{\color[rgb]{0,0,0}\definecolor[ named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0} \pgfsys@color@gray@fill{0}m}}\,,$ $$

where
$({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}{y}}_{i})_{{\color[rgb]{
0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke
{0}\pgfsys@color@gray@fill{0}k}}$, is the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}k}$-th element of the target vector and $({\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f_{{\color[rgb]{0,0,0}
\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{\Theta}}}}({\color[rgb]{0,0,0}\definecolor[named
]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}
\pgfsys@color@gray@fill{0}\bm{{x}}}))_{{\color[rgb]{0,0,0}\definecolor[named]{
pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill
{0}k}}$ is the ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}k}$-th output of a network indexed by ${\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\bm{\Theta}}$.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: There is little discussion about the number $n$ of samples that are needed to train a NN (only at lines 233-238). What are typical values for $n$?

See above:
 * experiments on toy datasets matching the assumptions?
 * motivation for finding a lower bound?
 * relevance of the comparison with [Taheri et al., 2021]?
soundness: 3
strengths: # Originality

This paper attempts to find a lower bound for the minimax risk, which is not common in the literature.

# Clarity

The paper is easy to ready.

# Quality

The proof seems to be sound (I did not check everything).
summary: This paper proposes to bound by below the minimax risk of a sub-class of ReLU Multi-Layer Perceptrons (MLP). This sub-class, denoted by $\mathcal{B}\_{\mathrm{L}}$, is built as the $\mathcal{L}^1$-ball of a given radius $v\_{\mathrm{s}}$ of the full space of parameters. The lower bound is obtained by using the packing number of $\mathcal{B}\_{\mathrm{L}}$.
weaknesses: # Originality

The paper uses almost the same tools as in [1], including the $\mathcal{L}^1$ regularization (which becomes the $\mathcal{L}^1$ ball in the space of parameters). The covering number is replaced with the packing number.

# Clarity

The motivation of finding such a lower bound is unclear: why would it be useful to get a lower bound on the minimax risk? What kind of information does it provide on the dataset, the function to approximate and the NN architecture?

Except it fits the framework of [1], what is the motivation for studying a subspace of MLPs defined by the $\mathcal{L}^1$ ball in the space of parameters? Why not choosing another metric?

# Significance

Unclear motivation (see Clarity).

# Quality

## Sharpness of the bound

According to the authors (see paragraph starting line 216), the lower bound is sharp, since it contains the same factors as the upper bound in [1]. Please note that the similarity between the two bounds is easily explained by the similarity of tools used.

But, very importantly, the bound in [1] is obtained with a loss with a strong $\mathcal{L}^1$ regularization term. Why would it be relevant to compare the two bounds?

## Interpretation of $V\_{\mathcal{F}}$

In paragraph starting line 216, the authors interpret $V\_{\mathcal{F}} = (v\_{\mathrm{s}} / L)^L$ as "a product over the $\mathcal{L}^1$-norm bounds of different layers in $\mathcal{F}\_{\mathcal{B}\_{\mathrm{L}}}$". This bound is obtained by performing an AM-GM inequality, which is known to be tighter when all the weight matrices have a similar $\mathcal{L}^1$ norm. However, it may be very loose otherwise, and the proposed interpretation tends to become inaccurate.

## Experiments

The authors have chosen to focus on experiments related to real-world data. This is undeniably useful, but the conditions of Theorem 2.2 are not fulfilled with such datasets. It would have been very useful to have results with a toy dataset, whose data are distributed in accordance to the assumptions (Gaussian inputs, outputs generated by a MLP, Gaussian noise...).


# References

[1] *Statistical guarantees for regularized neural networks*, Taheri et al., 2021.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: * Is it possible to get similar minimax rates if the input $x_i$ are subgaussian random vectors and/or noise are subgaussian?

* In view of lemma 5 of [1] (an upper bound on the generalization error) and theorem 2.2 in the paper, could a similar minimax lower bound be established for a function class consisting of more general architecture (e.g. other lipschitz activations, containing convolutions, etc) where the norm of the layers are bounded and the entire model is lipschitz?

[1] Mahsa Taheri, Fang Xie, & Johannes Lederer. (2020). Statistical Guarantees for Regularized Neural Networks.
soundness: 3
strengths: The minimax lower bound for deep ReLU networks and the lower bound for the packing number of deep ReLU neural networks is novel and interesting.
summary: The paper establishes minimax lower bounds for deep ReLU feedforward neural networks in the context of regression. Of note, to establish such lower bounds, the paper establishes a lower bound on the packing number of a ReLU network function class.
weaknesses: *

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See weakness section above.
soundness: 2
strengths: -	The problem of understanding the sample complexity of learning neural networks is an Important research topic.
-	The research presents a minimax lower bound for deep neural networks, which seems to be new.
summary: The paper focuses on analyzing the sample complexity of learning deep ReLU feed-forward networks that have a bounded elementwise $\ell_1$ norm. The authors show that the minimax rate scales as $1/\sqrt{n}$, where $n$ is the sample size, and back up their theoretical claims with experimental results.
weaknesses: -	It is not very clear to me why one should compare the results with $1/n$ rate. Based on my understanding, to achieve $1/n$ rate one often needs more structural assumptions on the target function, such as the ground-truth is spare in spare linear regression setting. Given that there is no such structural assumption in the paper, I’m not sure why one would expect such $1/n$ rate to happen.
-	The considered function space is elementwise $\ell_1$ norm bounded, which is not a very commonly considered setting.
-	The lower bound has an exponential dependency on $L$, which is not ideal though it is understandable that there might be technical difficulties to remove this.
-	It is not very clear to me the experiment setup is the right way to verify the correctness of the bound. These bounds are asymptotic bounds, meaning that the rate are only meaningful when $n$ is large enough. 
First, in the current scale of $n$, the difference between $\sqrt{n}$ and $n$ are not very significant. 
Second, the way of fitting the curve seems to consider all the data points, including those relatively small $n$. I believe such curve could not correctly represent the correct decaying rate.
Moreover, I believe these bounds in fact also depend on the norm of weight matrices.  It is possible the norm increases when $n$ increases, which could also affect the rate.
-	The experiment results many also affect by the optimization-related factors, for example converging to a local minima. These factors related to optimization are not considered in the theory, so it seems hard to draw conclusions from experiments.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. Can you come up with some conditions under which the lower bound O(1/sqrt(n)) is actually not valid? For example, what is your intuition about the optimal rate for different architectures? What do you expect for the case when \sigma=0?
soundness: 4
strengths: Importantly the bound depends on the logarithm of the input dimension, d, and scales as 1/sqrt(n) matching comparable upper bounds. While this type of bound exists before, it had not been established for deep, non-linear networks. 

Great clarity, organization and writing.
summary: The authors establish a lower bound on the packing number for deep ReLU networks with L1 constraints on the weights. Then they use the packing number bound and Fano’s inequality to provide a minimax risk lower bound that scale as sqrt(log(d)/n). 

The rate of 1/sqrt(n) has been established in a few other settings for neural networks (shallow feedforward networks, linear networks), but not for deep neural networks with non-linear activations.

One of the main implications is the separation between this bound and the known minimax rates for linear methods which scale as 1/n. 

 The authors provide some empirical support to their analysis, by experimenting with small datasets and networks, fitting O(1/n) and O(1/sqrt(n)) curves and showins that the latter achieves smaller residuals.

The question remains open on whether with specific data-dependent and algorithm-dependent assumptions 1/n is feasible in practical applications of neural networks.
weaknesses: 1. Some of the writing is missing clarity and could be rewritten in a more straightforward fashion. 
- The sentences in lines 38-43
- The sentences in lines 120-123.

2. [Minor] In your proof sketch of Section 4, please properly introduce random variable J (it is only introduce in the appendix).

Meta-review: **(a) Summary**  
This paper investigates the sample complexity required to train deep ReLU feed-forward neural networks. By establishing minimax lower bounds and leveraging Fano's inequality, the authors demonstrate that the generalization error scales at a rate of \(O(1/\sqrt{n})\) with the number of samples \(n\). The study also provides new insights into the role of packing numbers in quantifying neural network complexity and highlights a separation between these rates and the \(O(1/n)\) rates typically associated with linear models. Empirical experiments support the theoretical findings, showcasing how residuals align more closely with \(O(1/\sqrt{n})\) than \(O(1/n)\) in practical scenarios.

**(b) Strengths**  
- Novel theoretical contributions, including a minimax lower bound for ReLU networks and new results on packing numbers.  
- Clear and well-structured presentation, making the theoretical insights accessible.  
- Empirical results that corroborate the theoretical claims, bridging the gap between abstract theory and practical observations.  
- The paper addresses an important open question in learning theory, particularly in understanding the statistical limits of deep learning.

**(c) Weaknesses**  
- The motivation for comparing with \(O(1/n)\) rates is not fully clarified; the structural assumptions needed for such rates are not directly addressed.  
- The considered setting of elementwise \(L_1\)-bounded weights, while theoretically sound, is less common in practical neural network configurations.  
- Some experimental choices (e.g., including small sample sizes and curve fitting) could be better justified or improved to reflect the theoretical asymptotics more robustly.  
- The exponential dependency on the network depth in the bounds is acknowledged but not mitigated, which could be a concern for scalability.

**(d) Reasons for Acceptance**  
The paper makes significant theoretical contributions to understanding the sample complexity of deep neural networks. Its findings on \(O(1/\sqrt{n})\) scaling are robust, novel, and well-supported by empirical evidence. The theoretical rigor combined with practical relevance justifies its acceptance despite minor weaknesses in experimental presentation and the scope of applicability of the \(L_1\)-bounded assumption. The work addresses a core issue in learning theory and contributes meaningfully to ongoing discussions about generalization in deep learning.

Decision: Accept (Poster)