SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
m _ {\mathcal {H} _ {w}} (\epsilon , \delta) = O \left(\frac {w \log \left(\frac {w T}{\epsilon \sigma} \log \left(\frac {w T}{\epsilon \sigma}\right)\right) + \log (1 / \delta)}{\epsilon^ {2}}\right) = \widetilde {O} \left(\frac {w \log \left(\frac {T}{\sigma}\right) + \log (1 / \delta)}{\epsilon^ {2}}\right),
$$
where  $\widetilde{O}$  hides logarithmic factors.
One feature of the above theorem is the mild logarithmic dependence on  $1 / \sigma$ . Therefore, we can take  $\sigma$  to be numerically negligible and still get a significantly smaller sample complexity compared to the deterministic case for large  $T$ . Note that adding such small values of noise would not change the empirical outcome of RNNs on finite precision computers.
The milder (logarithmic) dependency on  $T$  is achieved by a novel analysis that involves bounding the covering number of noisy recurrent networks with respect to the total variation distance. Also, instead of "unfolding" the network, we exploit the fact that the same function/hypothesis is being used recurrently. We also want to emphasize that the above bound does not depend on the norms of weights of the network. Achieving this is challenging, since a little bit of noise in a previous layer can change the output of the next layer drastically. The next few sections are dedicated to give a high-level proof of this theorem.
6 Covering numbers: the classical view
One of the main tools to derive sample complexity bounds for learning a class of functions is studying their covering numbers. In this section we formalize this classic tool.
Definition 16 (Covering Number). Let  $(\mathcal{X},\rho)$  be a metric space. A set  $A\subset \mathcal{X}$  is  $\epsilon$ -covered by a set  $C\subseteq A$  with respect to  $\rho$ , if for all  $a\in A$  there exists  $c\in C$  such that  $\rho (a,c)\leq \epsilon$ . We denote by  $N(\epsilon ,A,\rho)$  the cardinality of the smallest set  $C$  that  $\epsilon$ -covers  $A$  and we refer to it as the  $\epsilon$ -covering number of  $A$  with respect to metric  $\rho$ .
The notion of covering number is defined with respect to a metric  $\rho$ . We now give the definition of extended metrics, which we will use to define uniform covering numbers. The extended metrics can be seen as measures of distance between two hypotheses on a given input set.
Definition 17 (Extended Metrics). Let  $(\mathcal{X},\rho)$  be a metric space. Let  $u = (a_{1},\ldots ,a_{m}),v = (b_{1},\ldots ,b_{m})\in \mathcal{X}^{m}$  for  $m\in \mathbb{N}$ . The  $\infty$ -extended and  $\ell_2$ -extended metrics over  $\mathcal{X}^m$  are defined by  $\rho^{\infty,m}(u,v) = \sup_{1\leq i\leq m}\rho (a_i,b_i)$  and  $\rho^{\ell_2,m}(u,v) = \sqrt{\frac{1}{m}\sum_{i = 1}^{m}(\rho(a_i,b_i))^2}$ , respectively. We drop  $m$  and use  $\rho^\infty$  or  $\rho^{\ell_2}$  if it is clear from the context.
One can also define PAC learnability for a class of random hypotheses and get a similar result without taking the expectation. However, working with a deterministic class helps to contrast the result with that of Theorem 10.
A useful property about extended metrics is that the  $\infty$ -extended metric always upper bounds the  $\ell_2$ -extended metric, i.e.,  $\rho^{\ell_2}(u,v) \leq \rho^\infty(u,v)$  for all  $u,v \in \mathcal{X}$ . Based on the above definition of extended metrics, we define the uniform covering number of a hypothesis class with respect to  $\| \cdot \|_2$ .
Definition 18 (Uniform Covering Number with Respect to  $\| .\| _2$ ). Let  $\mathcal{F}$  be a hypothesis class of functions from  $\mathcal{X}$  to  $\mathcal{V}$ . For a set of inputs  $S = \{x_{1},x_{2},\ldots ,x_{m}\} \subseteq \mathcal{X}$ , we define the restriction of  $\mathcal{F}$  to  $S$  as  $\mathcal{F}_{|S} = \{(f(x_1),f(x_2),\dots ,f(x_m)):f\in \mathcal{F}\} \subseteq \mathcal{Y}^m$ . The uniform  $\epsilon$ -covering numbers of hypothesis class  $\mathcal{F}$  with respect to  $\| .\| _2^\infty ,\| .\| _2^{\ell_2}$  are denoted by  $N_U(\epsilon ,\mathcal{F},m,\| .\| _2^\infty)$  and  $N_U(\epsilon ,\mathcal{F},m,\| .\| _2^{\ell_2})$  and are the maximum values of  $N(\epsilon ,\mathcal{F}_{|S},\| .\| _2^{\infty ,m})$  and  $N(\epsilon ,\mathcal{F}_{|S},\| .\| _2^{\ell_2,m})$  over all  $S\subseteq \mathcal{X}$  with  $|S| = m$ , respectively.
The following theorem connects the notion of uniform covering number with PAC learning. It converts a bound on the  $\left\| .\right\| _2^{\ell_2}$  uniform covering number of a hypothesis class to a bound on the sample complexity of PAC learning the class; see Appendix ?? for a more detailed discussion.
Theorem 19. Let  $\mathcal{F}$  be a class of functions from  $\mathcal{X}$  to  $\mathbb{R}$ . Then there exists an algorithm  $\mathcal{A}$  with the following property: For every distribution  $\mathcal{D}$  over  $\mathcal{X} \times \{-1,1\}$  and every  $\epsilon, \delta \in (0,1)$ , if  $S$  is a set of  $m$  i.i.d. samples from  $\mathcal{D}$ , then with probability at least  $1 - \delta$  (over the randomness of  $S$ ),
$$
\begin{array}{l} \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ l _ {\gamma} \left(\mathcal {A} (S), x, y\right) \right] \\ \leq \inf _ {f \in \mathcal {F}} \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ l _ {\gamma} (f, x, y) \right] + 1 6 \epsilon + \frac {2 4}{\sqrt {m}} \sqrt {\ln N _ {U} (\gamma \epsilon , \mathcal {F} , m , \| . \| _ {2} ^ {\ell_ {2}})} + 6 \sqrt {\frac {\ln (2 / \delta)}{2 m}}. \\ \end{array}
$$
Moreover, the algorithm that returns the function with the minimum error on  $S$  satisfies the above property (i.e., Algorithm A such that  $\mathcal{A}(S) = \arg \min_{f\in \mathcal{F}}\frac{1}{|S|}\sum_{(x,y)\in S}l_{\gamma}(f,x,y)$ ).
7 Total variation covers for random hypotheses
One idea to prove a generalization bound for noisy neural networks is to bound their covering numbers. However, noisy neural networks are random functions, and therefore their behaviours on a sample set cannot be directly compared. Instead, one can compare the output distributions of a random function on two sample sets. We therefore use the recently developed tools from Fathollah Pour and Ashtiani (2022) to define and study covering numbers for random hypotheses. These covering numbers are defined based on metrics between distributions. Specifically, our analysis is based on the notion of uniform covering number with respect to total variation distance.
Definition 20 (Total Variation Distance). Let  $\mu$  and  $\nu$  denote two probability measures over  $\mathcal{X}$  and let  $\Omega$  be the Borel sigma-algebra over  $\mathcal{X}$ . The TV distance between  $\mu$  and  $\nu$  is defined by
$$
d _ {T V} (\mu , \nu) = \sup  _ {B \in \Omega} | \mu (B) - \nu (B) |.
$$
Furthermore, if  $\mu$  and  $\nu$  have densities  $f$  and  $g$  then
$$
d _ {T V} (\mu , \nu) = \sup  _ {B \in \Omega} \left| \int_ {B} (f (x) - g (x)) d x \right| = \frac {1}{2} \int_ {\mathcal {X}} | f (x) - g (x) | d x = \frac {1}{2} \| f - g \| _ {1}.
$$
For two random variables  $\overline{x}$  and  $\overline{y}$  with probability measures  $\mu$  and  $\nu$  we sometimes abuse the notation and write  $d_{TV}(\overline{x},\overline{y})$  instead of  $d_{TV}(\mu ,\nu)$ . For example, we write  $d_{TV}(\overline{f_1} (\overline{x}),\overline{f_2} (\overline{x}))$  in order to refer to the Total Variation (TV) distance between pushforwards of  $\overline{x}$  under mappings  $\overline{f}_1$  and  $\overline{f_2}$ . We also write  $d_{TV}^{\infty,m}\left((\overline{f_1} (\overline{x_1}),\ldots ,\overline{f_1} (\overline{x_m})),(\overline{f_2} (\overline{x_1}),\ldots ,\overline{f_2} (\overline{x_m})))\right)$  to refer to the extended TV distance between mappings of the set  $S = \{\overline{x_1},\dots,\overline{x_m}\}$  by  $\overline{f_1}$  and  $\overline{f_2}$ . We use the extended total variation distance to define the uniform covering number for classes of random hypotheses.
Definition 21 (Uniform Covering Number for Classes of Random Hypotheses). Let  $\overline{\mathcal{F}}$  be a class of random hypotheses from  $\overline{\mathcal{X}}$  to  $\overline{\mathcal{Y}}$ . For a set of random variables  $\overline{S} = \{\overline{x_1}, \overline{x_2}, \dots, \overline{x_m}\} \subseteq \overline{\mathcal{X}}$ , the restriction of  $\overline{\mathcal{F}}$  to  $\overline{S}$  is defined as  $\overline{\mathcal{F}}_{|\overline{S}} = \{(\overline{f}(\overline{x_1}), \overline{f}(\overline{x_2}), \dots, \overline{f}(\overline{x_m})): \overline{f} \in \overline{\mathcal{F}}\} \subseteq \overline{\mathcal{Y}}^m$ . Let  $\Gamma \subseteq \overline{\mathcal{X}}$ . The uniform  $\epsilon$ -covering numbers of  $\overline{\mathcal{F}}$  with respect to  $\Gamma$  and  $d_{TV}^{\infty}$  is defined by
$$
N_{U}(\epsilon ,\overline{\mathcal{F}},m,d^{\infty}_{TV},\Gamma) = \sup_{S\subseteq \Gamma ,|S| = m}N(\epsilon ,\overline{\mathcal{F}}_{|\overline{S}},d^{\infty ,m}_{TV}).
$$
Some hypothesis classes that we analyze (e.g., single-layer noisy neural networks) may have "global" total variation covers that do not depend on  $m$ . This will be addressed with the following notation:  $N_U(\epsilon, \overline{\mathcal{F}}, \infty, \rho^\infty, \Gamma) = \lim_{m \to \infty} N_U(\epsilon, \overline{\mathcal{F}}, m, \rho^\infty, \Gamma)$ . The set  $\Gamma$  in Definition 21 is used to define the input domain for which we want to find the covering number of a class of random hypotheses. For instance, some of the covers that we see are derived with respect to inputs with bounded domain or some need the input to be first smoothed by Gaussian noise. In this paper, we will be working with the following choices of  $\Gamma$
-  $\Gamma = \overline{\mathcal{X}_d}$  and  $\Gamma = \overline{\mathcal{X}_{B,d}}$ : the set of all random variables defined over  $\mathbb{R}^d$  and  $[-B,B]^d$ , respectively, that admit a generalized density function. For example, we use  $\overline{\mathcal{X}_{0.5,d}}$  to address the set of random variables in  $[-1/2,1/2]^d$ .
-  $\Gamma = \overline{\Delta_{p\times T}} = \{\overline{U}\mid \overline{U} = \left[\delta_{u^{(0)}}\quad \dots \quad \overline{\delta_{u^{(T - 1)}}}\right]^{\top},u^{(i)}\in \mathbb{R}^{p}\}$  and  $\Gamma = \overline{\Delta_{B,p\times T}} = \{\overline{U}\mid \overline{U} = \left[\overline{\delta_{u^{(0)}}}\quad \dots \quad \overline{\delta_{u^{(T - 1)}}}\right]^{\top},u^{(i)}\in [-B,B]^{p}\}$ , where  $\overline{\delta_{u^{(i)}}}$  is the random variable associated with Dirac delta measure on  $u^{(i)}$ . Note that  $\overline{\Delta_{B,p\times T}}\subset \overline{\Delta_{p\times T}}$
-  $\Gamma = \overline{\mathcal{G}_{\sigma,d}} \circ \overline{\mathcal{X}_{B,d}} = \{\overline{g_{\sigma,d}}(\overline{x}) \mid \overline{x} \in \overline{\mathcal{X}_{B,d}}\}$ : all members of  $\overline{\mathcal{X}_{B,d}}$  after being "smoothed" by adding (convolving the density with) Gaussian noise.
We mentioned in Section 6 that a bound on the  $\| .\| _2^{\ell_2}$  uniform covering number can be connected to a bound on sample complexity of PAC learning. We now show that a bound on  $d_{TV}^{\infty}$  covering number of a class of random hypotheses can be turned into a bound on the  $\| .\| _2^{\ell_2}$  covering number of its derandomized version and, thus, PAC learning it.
Theorem 22  $(\| .\|_{2}^{\ell_{2}}$  Cover of  $\mathcal{E}(\mathcal{F})$  From  $d_{TV}^{\infty}$  Cover of  $\mathcal{F}$  (Fathollah Pour and Ashtiani, 2022)). Let  $\overline{\mathcal{F}}$  be a class of functions from  $\mathbb{R}^{p\times T}$  to  $[-B,B]^q$ . Then for every  $\epsilon >0$  and  $m\in \mathbb{N}$  we have
$$
N _ {U} (2 B \epsilon \sqrt {q}, \mathcal {E} (\overline {{\mathcal {F}}}), m, \|. \| _ {2} ^ {\ell_ {2}}) \leq N _ {U} (\epsilon , \overline {{\mathcal {F}}}, m, d _ {T V} ^ {\infty}, \overline {{\Delta_ {p \times T}}}) \leq N _ {U} (\epsilon , \overline {{\mathcal {F}}}, \infty , d _ {T V} ^ {\infty}, \overline {{\Delta_ {p \times T}}}).
$$
8 Bounding the covering number of recurrent models
In Section 6, we mentioned that finding a bound on covering number of a hypothesis class is a standard approach to bound its sample complexity. In the previous section, we introduced a new notion of covering number with respect to total variation distance that was developed by Fathollah Pour and Ashtiani (2022). We showed how this notion can be related to PAC learning for classes of random hypotheses. In the following, we give an overview of the techniques used to find a bound on the  $d_{TV}^{\infty}$  covering number of the class of noisy recurrent models. We also discuss why this bound results in a sample complexity that has a milder logarithmic dependency on  $T$ , compared to bounds proved by "unfolding" the recurrence and replacing the recurrent model with the  $T$ -fold composition.
One advantage of analyzing the uniform covering number with respect to TV distance is that it comes with a useful composition tool. The following theorem basically states that when two classes of hypotheses have bounded TV covers, their composition class has a bounded cover too. Note that such a result does not hold for the usual definition of covering number (e.g., Definition 18); see Fathollah Pour and Ashtiani (2022) for details.
Theorem 23 (TV Cover for Composition of Random Classes, Lemma 18 of Fathollah Pour and Ashtiani (2022)). Let  $\overline{\mathcal{F}}$  be a class of random hypotheses from  $\mathbb{R}^d$  to  $\mathbb{R}^p$  and  $\overline{\mathcal{H}}$  be a class of random hypotheses from  $\mathbb{R}^p$  to  $\mathbb{R}^q$ . For any  $\epsilon_1, \epsilon_2 > 0$  and  $m \in \mathbb{N}$ , denote  $N_1 = N_U\left(\epsilon_1, \overline{\mathcal{F}}, m, d_{TV}^\infty, \overline{\mathcal{X}_d}\right)$ . Then we have,
$$
N _ {U} \left(\epsilon_ {1} + \epsilon_ {2}, \overline {{\mathcal {H}}} \circ \overline {{\mathcal {F}}}, m, d _ {T V} ^ {\infty}, \overline {{\mathcal {X} _ {d}}}\right) \leq N _ {U} \left(\epsilon_ {2}, \overline {{\mathcal {H}}}, m N _ {1}, d _ {T V} ^ {\infty}, \overline {{\mathcal {X} _ {p}}}\right). N _ {1}.
$$
An approach to bound the TV uniform covering number of a recurrent model  $\mathrm{REC}[\overline{\mathcal{F}},T]$  is to consider it as the  $T$ -fold composition  $\overline{\mathcal{F}}\circ \overline{\mathcal{F}}\ldots \circ \overline{\mathcal{F}}$ . One can then use a similar analysis to that of Fathollah Pour and Ashtiani (2022) to bound the covering number of the  $T$ -fold composition. Unfortunately, this approach fails to capture the fact that a fixed function  $\overline{f}\in \overline{\mathcal{F}}$  is applied recursively, and therefore results in a sample complexity bound that grows at least linearly with  $T$ .
Instead, we take another approach to bound the covering number of recurrent models. Intuitively, we notice that any function in the  $T$ -fold composite class  $\overline{\mathcal{F}} \circ \ldots \circ \overline{\mathcal{F}} = \{\overline{f_1} \circ \ldots \circ \overline{f_T} \mid \overline{f_1}, \ldots, \overline{f_T} \in$
$\overline{\mathcal{F}}\}$  is determined by  $T$  functions from  $\overline{\mathcal{F}}$ . On the other hand, any function in  $\mathrm{REC}[\overline{\mathcal{F}}, T] = \left\{\overline{h} \mid \overline{h}(U) = \operatorname{Last}\left(\overline{f}^R(U, T - 1)\right)\right\}$  is only defined by one function in  $\overline{\mathcal{F}}$  and the capacity of this class must not be as large as the capacity of  $\overline{\mathcal{F}} \circ \ldots \circ \overline{\mathcal{F}}$ . Interestingly, data processing inequality for total variation distance (Lemma ??) suggests that if two functions  $\overline{f}$  and  $\widehat{\overline{f}}$  are "globally" close to each other with respect to TV distance (i.e.,  $d_{TV}(\overline{f}(\overline{x}), \widehat{\overline{f}}(\overline{x})) \leq \epsilon$  for every  $\overline{x}$  in the domain), then  $d_{TV}(\overline{f}(\overline{f}(\overline{x})), \widehat{\overline{f}}(\widehat{\overline{f}}(\overline{x}))) \leq 2\epsilon$  (i.e.,  $\overline{f} \circ \overline{f}$  and  $\widehat{\overline{f}} \circ \widehat{\overline{f}}$  are also close to each other). By applying the data processing inequality recursively, we can see that for the  $T$ -fold composition we have  $d_{TV}(\overline{f} \circ \ldots \circ \overline{f}(\overline{x}), \widehat{\overline{f}} \circ \ldots \circ \widehat{\overline{f}}(\overline{x})) \leq \epsilon T$ . The above approach results in the following theorem which bounds the  $\epsilon$ -covering number of a noisy recurrent model with respect to TV distance by the  $(\epsilon / T)$ -covering number of its recurring class. Intuitively, this theorem helps us to bound the covering number of noisy recurrent models using the bounds obtained for their non-recurrent versions. Here, Gaussian noise is added to both the input of the model (i.e.,  $\overline{\mathcal{F}_{\sigma}} = \overline{\mathcal{F}} \circ \overline{\mathcal{G}_{\sigma}}$ ) and the output of the model (by composing with  $\overline{\mathcal{G}_{\sigma}}$ ).
Theorem 24 (TV Covering Number of  $\overline{\mathcal{G}_{\sigma}}$  o REC[  $\overline{\mathcal{F}_{\sigma}},T]$  From  $\overline{\mathcal{G}_{\sigma}}\circ \overline{\mathcal{F}_{\sigma}})$ . Let  $s,p,q\in \mathbb{N}$  such that  $s = p + q - 1$ . Let  $\overline{\mathcal{F}}$  be a class of functions from  $\overline{\mathcal{X}_{B,s}}$  to  $\overline{\mathcal{X}_{B,q}}$  and denote by  $\overline{\mathcal{F}_{\sigma}} = \overline{\mathcal{F}}\circ \overline{\mathcal{G}_{\sigma,s}}$  the class of its composition with noise. Then we have
$$
N _ {U} \left(\epsilon , \overline {{\mathcal {G} _ {\sigma}}} \circ R E C [ \overline {{\mathcal {F} _ {\sigma}}}, T ], \infty , d _ {T V} ^ {\infty}, \overline {{\Delta_ {B , p \times T}}}\right) \leq N _ {U} \left(\epsilon / T, \overline {{\mathcal {G} _ {\sigma , q}}} \circ \overline {{\mathcal {F} _ {\sigma}}}, \infty , d _ {T V} ^ {\infty}, \overline {{\mathcal {X} _ {B , s}}}\right).
$$
For using this theorem, one needs to have a finer  $\epsilon / T$ -cover for the recurring class. As we will see in the next section, this will translate into a mild logarithmic sample complexity dependence on  $T$ .
8.1 Covering noisy recurrent networks
An example of  $\overline{\mathcal{F}_{\sigma}}$  is the class  $\overline{\mathrm{MNET}_{\sigma}}[p_0, p_k, w]$  of well-defined noisy multi-layer networks (Definition 13). Theorem 24 suggests that a bound on the covering number of  $\overline{\mathcal{G}_{\sigma}} \circ \mathrm{REC}[\overline{\mathrm{MNET}_{\sigma}}[p_0, p_k, w], T]$  can be found from a bound for  $\overline{\mathcal{G}_{\sigma}} \circ \overline{\mathrm{MNET}_{\sigma}}[p_0, p_k, w]$ . We use the following theorem as a bound for the class of single-layer noisy sigmoid networks together with theorem 23 to bound the covering number of  $\overline{\mathcal{G}_{\sigma}} \circ \overline{\mathrm{MNET}_{\sigma}}[p_0, p_k, w]$  (see Appendix ??, Theorem ??).
Theorem 25 (A TV Cover for Single-Layer Noisy Neural Networks, Theorem 25 of Fathollah Pour and Ashtiani (2022)). For every  $p,d\in \mathbb{N},\epsilon >0,\sigma < 5d / \epsilon$  we have
$$
\log N _ {U} (\epsilon , \overline {{\mathcal {G} _ {\sigma , p}}} \circ N E T [ d, p ], \infty , d _ {T V} ^ {\infty}, \overline {{\mathcal {G} _ {\sigma , d}}} \circ \overline {{\mathcal {X} _ {0 . 5 , d}}}) \leq p (d + 1) \log \left(3 0 \frac {d ^ {5 / 2} \sqrt {\ln \left(\frac {5 d - \epsilon \sigma}{\epsilon \sigma}\right)}}{\epsilon^ {3 / 2} \sigma^ {2}} \ln \left(\frac {5 d}{\epsilon \sigma}\right)\right).
$$
Interestingly, the above bound (on the logarithm of the covering number) is logarithmic with respect to  $1 / \epsilon$ . We will extend this result to multi-layer noisy networks, and then apply Theorem 24 to obtain the following bound on the covering number noisy recurrent neural networks. Crucially, the dependency (of the logarithm of the covering number) on  $T$  is only logarithmic.
Theorem 26 (A TV Covering Number Bound for Noisy Sigmoid Recurrent Networks). Let  $T \in \mathbb{N}$ . For every  $\epsilon, \sigma \in (0,1)$  and every well-defined class  $\text{REC}[\overline{\text{MNET}}_{\sigma}[p_0,p_k,w],T]$  we have
$$
\begin{array}{l} \log N _ {U} \left(\epsilon , \overline {{\mathcal {G} _ {\sigma}}} \circ R E C [ \overline {{M N E T _ {\sigma}}} [ p _ {0}, p _ {k}, w ], T ], \infty , d _ {T V} ^ {\infty}, \overline {{\Delta_ {0 . 5 , p \times T}}}\right) \\ = O \left(w \log \left(\frac {w T}{\epsilon \sigma} \log \left(\frac {w T}{\epsilon \sigma}\right)\right)\right) = \widetilde {O} \left(w \log \left(\frac {T}{\epsilon \sigma}\right)\right). \\ \end{array}
$$
Finally, we turn the above bound into a  $\left\| .\right\| _2^{\ell_2}$  covering number bound for the derandomized function  $\mathcal{E}\left(\overline{\mathcal{G}_{\sigma}}\circ \mathrm{REC}[\overline{\mathrm{MNET}}_{\sigma}[p_0,p_k,w],T]\right)$  by an application of Theorem 22. We then upper bound the sample complexity by the logarithm of covering number (see Theorem 19) and conclude Theorem 15. Limitations and future work. Our results are derived for sigmoid (basically bounded, monotone, and Lipschitz) activation functions. It is open whether such results can be proved for unbounded activation functions such as RELU. Our results are theoretical and we leave empirical evaluations on the performance of noisy networks to future work.
9 More on related work
There is plethora of work on generalization in neural networks. There are a family of approaches that aim to bound the VC-dimension of neural networks. (Baum and Haussler, 1988; Maass, 1994; Goldberg and Jerrum, 1995; Vidyasagar, 1997; Sontag et al., 1998; Koiran and Sontag, 1998; Bartlett et al., 1998; Bartlett and Maass, 2003; Bartlett et al., 2019). These approaches result in generalization bounds that are dependent on the number of parameters. Another family of approaches are aimed at obtaining generalization bounds that are dependent on the norms of the weights and Lipschitz continuity properties of the network (Bartlett, 1996; Anthony et al., 1999; Zhang, 2002; Neyshabur et al., 2015; Bartlett et al., 2017; Neyshabur et al., 2018; Golowich et al., 2018; Arora et al., 2018; Nagarajan and Kolter, 2018; Long and Sedghi, 2020). It has been observed that these generalization bounds are usually vacuous in practice. One speculation is that the implicit bias of gradient descent (Gunasekar et al., 2017; Arora et al., 2019; Ji et al., 2020; Chizat and Bach, 2020; Ji and Telgarsky, 2021) can lead to benign overfitting (Belkin et al., 2018, 2019; Bartlett et al., 2020, 2021). It has also been conjectured that uniform convergence theory may not be able to fully capture the performance of neural networks in practice (Nagarajan and Kolter, 2019; Zhang et al., 2021). It has been shown that there are data-dependent approaches that can achieve non-vacuouys bounds (Dziugaite and Roy, 2017; Zhou et al., 2019; Negrea et al., 2019). There are also other approaches that are independent of data (Arora et al., 2018); see Fathollah Pour and Ashtiani (2022) for more details.
Adding different types of noise such as dropout noise (Srivastava et al., 2014), DropConnect (Wan et al., 2013), and Denoising AutoEncoders (Vincent et al., 2008) are shown to be helpful in training neural networks. Wang et al. (2019) and Gao and Zhou (2016) theoretically analyze the generalization under dropout noise. More recently, Fathollah Pour and Ashtiani (2022) developed a framework to study the generalization of classes of noisy hypotheses and show that adding noise to the output of neurons in a network can be helpful in generalization. Jim et al. (1996) show that additive and multiplicative noise can help speed up the convergence of RNNs on local minima surfaces. Recently, Lim et al. (2021) showed that noisy RNNs are more stable and robust to input perturbations by formalizing the regularization effects of noise.