SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alireza F. Pour
Cheriton School of Computer Science
a2fathol@uwaterloo.ca
Hassan Ashtiani
Department of Computing and Software
zokaeiam@mcmaster.ca
We consider the class of noisy multi-layered sigmoid recurrent neural networks with  $w$  (unbounded) weights for classification of sequences of length  $T$ , where independent noise distributed according to  $\mathcal{N}(0, \sigma^2)$  is added to the output of each neuron in the network. Our main result shows that the sample complexity of PAC learning this class can be bounded by  $O(w\log (T / \sigma))$ . For the non-noisy version of the same class (i.e.,  $\sigma = 0$ ), we prove a lower bound of  $\Omega (wT)$  for the sample complexity. Our results indicate an exponential gap in the dependence of sample complexity on  $T$  for noisy versus non-noisy networks. Moreover, given the mild logarithmic dependence of the upper bound on  $1 / \sigma$ , this gap still holds even for numerically negligible values of  $\sigma$ .<sup>1</sup>
Recurrent Neural Networks (RNNs) are effective tools for processing sequential data. They are used in numerous applications such as speech recognition (Graves et al., 2013), computer vision (Karpathy and Fei-Fei, 2015), translation (Sutskever et al., 2014), modeling dynamical systems (Hardt et al., 2018) and time series (Qin et al., 2017). Recurrent models allow us to design classes of predictors that can be applied to (i.e., take input values from) sequences of arbitrary length. For processing a sequence of  $T$  elements, a predictor  $f$  (e.g., a neural network) "consumes" the input elements one by one, generating an output at each step. This output is then used in the next step (as another input to  $f$  along with the next element in the input sequence). Defining recurrent models formally takes some effort, and we relegate it to the next sections. In short, the function  $f$  is (recursively) applied  $T$  times in order to generate the ultimate outcome.
Let us fix a base class  $\mathcal{F}_w$  of all multi-layered feed-forward sigmoid neural networks with  $w$  weights. We can create a recurrent version of this class, which we will denote by  $\mathrm{REC}[\mathcal{F}_w,T]$ , for classifying sequences of length  $T$ . One can study the sample complexity of PAC learning  $\mathrm{REC}[\mathcal{F}_w,T]$  with respect to different loss functions. Koiran and Sontag (1998) studied the binary-valued version of this class by applying a threshold function at the end, and proved a lower bound of  $\Omega(wT)$  for its VC dimension.
There has also been efforts for proving upper bounds on the sample complexity of PAC learning  $\mathrm{REC}[\mathcal{F},T]$  for various base classes  $\mathcal{F}$  and different loss functions. Given the above lower bound, a gold standard has been achieving a linear dependence on  $T$  in the upper bound. Koiran and Sontag (1998) proved an upper bound of  $O(w^{4}T^{2})$  on the VC dimension of  $\mathrm{REC}[\mathcal{F}_w,T]$  discussed above. More recent papers have considered the more realistic setting of classification with continuous-valued
For the full version of this paper see F. Pour and Ashtiani (2023).
RNNs, e.g., by removing the threshold function and using a bounded Lipschitz surrogate loss. In this setting, Zhang et al. (2018) proved an upper bound of  $\widetilde{O}(T^4 w\|W\|^{O(T)})$  on the sample complexity<sup>2</sup> where  $\|W\|$  is the spectral norm of the network. Chen et al. (2020) improved over this result by proving an upper bound of  $\widetilde{O}(Tw\|W\|^2\min\{\sqrt{w}, \|W\|^{O(T)}\})$ . These bounds get close to the gold standard when the spectral norm of the network satisfies  $\|W\| \leq 1$ .
The above upper bounds are proved by simply "unfolding" the recurrence, effectively substituting the recurrent class  $\mathrm{REC}[\mathcal{F}_w,T]$  with the (larger) class of  $T$ -fold compositions  $\mathcal{F}_w\circ \mathcal{F}_w\dots \circ \mathcal{F}_w$ . These unfolding techniques do not exploit the fact that the function  $f$  (that is applied recursively for  $T$  steps to compute the output of the network) is fixed across all the  $T$  steps. Consequently, the resulting sample complexity has (super-)linear dependence on  $T$ . Therefore, we would need a prohibitively large sample size for training recurrent models for classifying very long sequences. Nevertheless, this dependence is inevitable in light of the lower bound of Koiran and Sontag (1998). Or is it?
In this paper, we consider a related class of noisy recurrent neural networks,  $\mathrm{REC}[\widetilde{\mathcal{F}}_w^\sigma ,T]$ . The hypotheses in this class are similar to those in  $\mathrm{REC}[\mathcal{F}_w,T]$ , except that outputs of (sigmoid) activation functions are added with independent Gaussian random variables,  $\mathcal{N}(0,\sigma^2)$ . Our main result demonstrates that, remarkably, the noisy class can be learned with a number of samples that is only logarithmic with respect to  $T$ .
Theorem 1 (Informal version of Theorem 15). The sample complexity of PAC learning the class  $REC[\widetilde{\mathcal{F}}_w^\sigma, T]$  of noisy recurrent networks with respect to ramp loss is  $\widetilde{O}(w\log(T / \sigma))$ .
One challenge of proving the above theorem is that the analysis involves dealing with random hypotheses. Therefore, unlike the usual arguments that bound the covering number of a set of deterministic maps with respect to the  $\ell_2$  distance, we study the covering number of a class of random maps with respect to the total variation distance. We then invoke some of the recently developed tools in Fathollah Pour and Ashtiani (2022) for bounding these covering numbers. Another challenge is deviating from the usual "unfolding method" and exploiting the fact that in recurrent models a fixed function/network is applied recursively.
The mere fact that learning  $\mathrm{REC}[\widetilde{\mathcal{F}}_w^\sigma, T]$  requires less samples compared to its non-noisy counterpart is not entirely unexpected. For classification of long sequences, however, the sample complexity gap is quite drastic (i.e., exponential). We argue that a logarithmic dependency on  $T$  is actually more realistic in practical situations: for finite precision machines, one can effectively break the  $\Omega(T)$  barrier even for non-noisy networks. To see this, let us choose  $\sigma$  to be a numerically negligible number (e.g., smaller than the numerical precision of our computing device). In this case, the class of noisy and non-noisy networks become effectively the same when implemented on a device with finite numerical precision. But then our upper bound shows a mild logarithmic dependence on  $1 / \sigma$ .
One caveat in the above argument is that the lower bound of Koiran and Sontag (1998) is proved for the 0-1 loss and perhaps not directly comparable to the setting of the upper bound which uses a Lipschitz surrogate loss. We address this by showing a comparable lower bound in the same setting.
Theorem 2 (Informal version of Theorem 10). The sample complexity of PAC learning  $REC[\mathcal{F}_w, T]$  with ramp loss is  $\Omega(wT)$ .
In the next section we introduce our notations and define the PAC learning problem. We state the lower bound in Section 3, and the upper bound in Section 5. Sections 6, 7, and 8 provide a high-level proof of our upper bound.
Additional Related Work. Due to space constraints, we postpone the discussion of some additional related work to Appendix 9.
$\| x\| _1,\| x\| _2$  , and  $\| x\|_{\infty}$  denote the  $\ell_1,\ell_2$  , and  $\ell_{\infty}$  norms of a vector  $x\in \mathbb{R}^d$  respectively. We denote the cardinality of a set  $S$  by  $|S|$  . The set of natural numbers smaller or equal to  $m$  is represented by
2 Ignoring the dependence of the sample complexity on the accuracy and confidence parameters.
$[m]$ . A vector of all zeros is denoted by  $0_d = [0 \ldots 0]^\top \in \mathbb{R}^d$ . We use  $\mathcal{X} \subseteq \mathbb{R}^d$  as a domain set. We will study classes of vector-valued functions; a hypothesis is a Borel function  $f: \mathbb{R}^d \to \mathbb{R}^p$ , and a hypothesis class  $\mathcal{F}$  is a set of such hypotheses.
We find it useful to have an explicit notation—here an overline—for the random versions of the above definitions:  $\overline{\mathcal{X}}$  is the set of all random variables defined over  $\mathcal{X}$  that admit a generalized density function<sup>3</sup>.  $\overline{x} \in \overline{\mathcal{X}}$  is a random variable in this set. To simplify this notation, we sometimes just write  $\overline{x} \in \mathbb{R}^d$  rather than  $\overline{x} \in \overline{\mathbb{R}^d}$ .
$\overline{y} = f(\overline{x})$  is the random variable associated with pushforward of  $\overline{x}$  under Borel map  $f:\mathbb{R}^d\to \mathbb{R}^p$ . We use  $\overline{f}:\mathbb{R}^d\to \mathbb{R}^p$  to indicate that the mapping itself is random. Random hypotheses can be applied to both random and non-random inputs—e.g.,  $\overline{f} (\overline{x})$  and  $\overline{f} (x)^4$ . A class of random hypotheses is denoted by  $\overline{\mathcal{F}}$ .
Definition 3 (Composition of Two Hypothesis Classes). We denote by  $h \circ f$  the function  $h(f(x))$  (assuming the range of  $f$  and the domain of  $h$  are compatible). The composition of two hypothesis classes  $\mathcal{F}$  and  $\mathcal{H}$  is defined by  $\mathcal{H} \circ \mathcal{F} = \{h \circ f \mid h \in \mathcal{H}, f \in \mathcal{F}\}$ . Composition of classes of random hypotheses is defined similarly by  $\overline{\mathcal{H}} \circ \overline{\mathcal{F}} = \{\overline{h} \circ \overline{f} \mid \overline{h} \in \overline{\mathcal{H}}, \overline{f} \in \overline{\mathcal{F}}\}$ .
2.2 Feedforward neural networks
We will first define some classes associated with feedforward neural networks. Let  $\phi(x) = \frac{1}{1 + e^{-x}} - \frac{1}{2}$  be the centered sigmoid function.  $\Phi: \mathbb{R}^p \to [-1/2, 1/2]^p$  is the element-wise sigmoid activation function defined by  $\Phi((x^{(1)}, \ldots, x^{(p)})) = (\phi(x^{(1)}), \ldots, \phi(x^{(p)}))$ .
Definition 4 (Single-Layer Sigmoid Neural Networks). The class of single-layer sigmoid neural networks with  $d$  inputs and  $p$  outputs is defined by  $\mathsf{NET}[d,p] = \{f_W:\mathbb{R}^d\to [-1 / 2,1 / 2]^p\mid f_W(x) = \Phi (W^\top x),W\in \mathbb{R}^{d\times p}\}$ .
Based on Definition 4, we can define the class of multi-layer (feedforward) neural networks (with  $w$  weights) as a composition of several single-layer networks. Note that the number of hidden neurons can be arbitrary as long as the total number of weights/parameters is  $w$ .
Definition 5 (Multi-Layer Sigmoid Neural Networks). A class of multi-layer sigmoid networks with  $p_0$  inputs,  $p_k$  outputs, and  $w$  weights that take inputs in  $[-1/2, 1/2]^{p_0}$  is defined by
$$
M N E T [ p _ {0}, p _ {k}, w ] = \bigcup N E T [ p _ {k - 1}, p _ {k} ] \circ \dots \circ N E T [ p _ {0}, p _ {1} ]
$$
where union is taken over all choices of  $(p_1, p_2, \ldots, p_{k-1}) \in \mathbb{N}^{k-1}$  that satisfy  $\sum_{i=1}^{k} p_i \cdot p_{i-1} = w$ . We say MNET $[p_0, p_k, w]$  is well-defined if the union is not empty.
Well-definedness basically means that  $p_0, p_k$ , and  $w$  are compatible. For simplicity, in the above definition we restricted the input domain to  $[-1/2, 1/2]^d$ . This will help in defining the recurrent versions of these networks (since the input and output domains become compatible). However, our analysis can be easily extended to capture any bounded domain (e.g.,  $[-B, B]^d$ ).
2.3 Recursive application of a function and recurrent models
In this section we define  $\mathrm{REC}[\mathcal{F},T]$  which is the recurrent version of class  $\mathcal{F}$  for sequences of length  $T$ . Let  $v = (a_{1},\ldots ,a_{m})\in \mathcal{X}^{m}$  for  $m\in \mathbb{N}$ . We define First  $(v) = (a_{1},\dots,a_{m - 1})\in \mathcal{X}^{m - 1}$  and Last  $(v) = a_{m}\in \mathcal{X}$  as functions that return the first  $m - 1$  and the last dimensions of the vector  $v$ , respectively. Let  $u^{(0)},u^{(1)},\ldots ,u^{(T - 1)}$  be a sequence of inputs, where  $u^{(i)}\in \mathbb{R}^p$ , and let  $f:\mathbb{R}^s\to \mathbb{R}^q$  be a hypothesis/mapping. In the context of recurrent models, it is useful to define the recurrent application of  $f$  on this sequence. Note that out of the  $q$  dimensions of the range of  $f$ ,  $q - 1$  of them are recurrent and therefore are fed back to the model. Basically,  $f^{R}(U,t)$  will be the result of applying  $f$  on the first  $t$  elements of  $U$  (with recurrent feedback).
3Both discrete (by using Dirac delta function) and absolutely continuous random variables admit a generalized density function.
4Technically, we consider  $\overline{f} (x)$  to be  $\overline{f} (\overline{\delta_x})$  , where  $\overline{\delta_x}$  is a random variable with Dirac delta measure on  $x$
Figure 1: An example of a recurrent model in  $\mathrm{REC}[\mathcal{F},T]$ . The first  $q - 1$  dimensions of  $f^{R}(U,t - 1)$  is concatenated with  $u^{(t)}$  to form the input at time  $t$ . The last dimension of  $f^{R}(U,T - 1)$  is taken to be the final output of the recurrent model.
Definition 6 (Recurrent Application of a Function). Let  $U = \left[u^{(0)}\dots u^{(i)}\dots u^{(T - 1)}\right]\in \mathbb{R}^{p\times T}$  be a sequence of inputs of length  $T$ , where  $u^{(i)}\in \mathbb{R}^p$  denotes the  $i$ -th column of  $U$  for  $0\leq i\leq T - 1$ . Let  $f$  be a (random) function from  $\mathbb{R}^s$  to  $\mathbb{R}^q$ , where  $s = p + q - 1$ . Moreover, define  $f^R (U,0) = f\left(\left[0_{q - 1}\quad u^{(0)}\right]^\top\right)$ . Then, for any  $1\leq t\leq T - 1$ , the recursive application of  $f$  is denoted by  $f^R:\mathbb{R}^{p\times T}\times [T - 1]\to \mathbb{R}^q$  and is defined as  $f^R (U,t) = f\left(\left[First\left(f^R (U,t - 1)\right)\quad u^{(t)}\right]^\top\right)$ .
Now we are ready to define the (recurrent) hypothesis class  $\mathrm{REC}[\mathcal{F},T]$ . Each hypothesis in this class takes a sequence  $U$  of input vectors, and applies a function  $f\in \mathcal{F}$  recurrently on the elements of this sequence. The final output will be a real number. We give the formal definition in the following; also see Figure 1 for a visualization.
Definition 7 (Recurrent Class). Let  $s, p, q \in \mathbb{N}$  such that  $s = p + q - 1$ . Let  $\mathcal{F}$  be a class of functions from  $\mathbb{R}^s$  to  $\mathbb{R}^q$ . The class of recurrent models with length  $T$  that use functions in  $\mathcal{F}$  (which we denote by recurring class) as their recurring block is defined by
$$
R E C [ \mathcal {F}, T ] = \left\{h: \mathbb {R} ^ {p \times T} \rightarrow \mathbb {R} \mid h (U) = L a s t \left(f ^ {R} (U, T - 1)\right), f \in \mathcal {F} \right\}
$$
For example,  $\mathrm{REC}[\mathrm{MNET}[p_0,p_k,w],T]$  is the class of (real-valued) recurrent neural networks with length  $T$  that use  $\mathrm{MNET}[p_0,p_k,w]$  as their recurring block. We say that  $\mathrm{REC}[\mathrm{MNET}[p_0,p_k,w],T]$  is well-defined if  $\mathrm{MNET}[p_0,p_k,w]$  is well-defined and also the input/output dimensions are compatible (i.e.,  $p_0\geq p_k$ ).
2.4 PAC learning with ramp loss
In this section we formulate the PAC learning model for classification with respect to the ramp loss. The use of ramp loss is natural for classification (see e.g., Boucheron et al. (2005); Bartlett et al. (2006)) and the main features of the ramp loss that we are going to exploit are boundedness and Lipschitzness. We start by introducing the ramp loss.
Definition 8 (Ramp Loss). Let  $f: \mathcal{X} \to \mathbb{R}$  be a hypothesis and let  $\mathcal{D}$  be a distribution over  $\mathcal{X} \times \mathcal{Y}$ . Let  $(x, y) \in \mathcal{X} \times \mathcal{Y}$ , where  $\mathcal{Y} = \{-1, 1\}$ . The ramp loss of  $f$  with respect to margin parameter  $\gamma > 0$  is defined as  $l_{\gamma}(f, x, y) = r_{\gamma}(-f(x).y)$ , where  $r_{\gamma}$  is the ramp function defined by
$$
r _ {\gamma} (x) = \left\{ \begin{array}{l l} 0 & x <   - \gamma , \\ 1 + \frac {x}{\gamma} & - \gamma \leq x \leq 0 \\ 1 & x \geq 0. \end{array} \right.
$$
Definition 9 (Agnostic PAC Learning with Respect to Ramp Loss). We say that a hypothesis class  $\mathcal{F}$  of functions from  $\mathcal{X}$  to  $\mathbb{R}$  is agnostic PAC learnable with respect to ramp loss with margin parameter  $\gamma >0$  if there exists a learner  $\mathcal{A}$  and a function  $m:(0,1)^2\to \mathbb{N}$  with the following property: For every distribution  $\mathcal{D}$  over  $\mathcal{X}\times \{-1,1\}$  and every  $\epsilon ,\delta \in (0,1)$ , if  $S$  is a set of  $m(\epsilon ,\delta)$  i.i.d. samples from  $\mathcal{D}$ , then with probability at least  $1 - \delta$  (over the randomness of  $S$ ) we have
$$
\mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ l _ {\gamma} \left(\mathcal {A} (S), x, y\right) \right] \leq \inf  _ {f \in \mathcal {F}} \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ l _ {\gamma} \left(\mathcal {A} (S), x, y\right) \right] + \epsilon .
$$
The sample complexity of PAC learning  $\mathcal{F}$  with respect to ramp loss is denoted by  $m_{\mathcal{F}}(\epsilon, \delta)$ , which is the minimum number of samples required for learning  $\mathcal{F}$  (among all learners  $\mathcal{A}$ ). The definition of agnostic PAC learning with respect to ramp loss works for any value of  $\gamma$  and when we are analyzing the sample complexity we consider it to be a fixed constant.
3 A lower bound for sample complexity of learning recurrent neural networks
In this section, we consider the sample complexity of PAC learning sigmoid recurrent neural networks with respect to ramp loss. Particularly, we state a lower bound on the sample complexity of the class REC[MNET $[p_0, p_k, w], T]$  of all sigmoid recurrent neural networks with length  $T$  that use multi-layer neural networks with  $w$  weights as their recurring block. The main message is that this sample complexity grows at least linearly with  $T$ .
Theorem 10 (Sample Complexity Lower Bound for Recurrent Neural Networks). For every  $T \geq 3$  and  $w \geq 19$  there exists a well-defined class  $\mathcal{H}_w = \text{REC}[MNET[p_0, p_k, w], T]$  and a universal constant  $C > 0$  such that for every  $\epsilon, \delta \in (0, 1/40)$  we have
$$
m _ {\mathcal {H} _ {w}} (\epsilon , \delta) \geq C. \left(\frac {w T + \log (1 / \delta)}{\epsilon^ {2}}\right).
$$
The proof of the above lower bound is based on a similar result due to Sontag et al. (1998). However, the argument in Sontag et al. (1998) is for PAC learning with respect to 0-1 loss. To extend this result for the ramp loss, we construct a binary-valued class  $\mathcal{F}_w = \{f : f(U) = \mathrm{sign}(h(U)), h \in \mathcal{H}_w\}$  where  $\mathrm{sign}(x) = 1$  if  $x \geq 0$  and  $\mathrm{sign}(x) = -1$  if  $x < 0$ . We prove that every function  $f \in \mathcal{F}_w$  can be related to another function  $h \in \mathcal{H}_w$  such that the ramp loss of  $h$  is almost equal to the zero-one loss of  $f$ . This is formalized in the following lemma, which is a key result in proving Theorem 10. The proof of Theorem 10 and Lemma 11 can be found in Appendix ??
Lemma 11. Let  $\mathcal{H}_w = \text{REC}[MNET[p_0, p_k, w], T]$  be a well-defined class and let  $\mathcal{F}_w = \{f: [-1/2, 1/2]^{p \times T} \to \{-1, 1\} \mid f(U) = \text{sign}(h(U)), h \in \mathcal{H}_w\}$ . Then, for every distribution  $\mathcal{D}$  over  $[-1/2, 1/2]^{p \times T} \times \{-1, 1\}$ ,  $\eta > 0$ , and every function  $f \in \mathcal{F}_w$  there exists a function  $h \in \mathcal{H}_w$  such that  $\mathbb{E}_{(U,y) \sim \mathcal{D}}[l_{\gamma}(h,U,y)] \leq \mathbb{E}_{(U,y) \sim \mathcal{D}}[l^{0-1}(f,U,y)] + \eta$  where  $l^{0-1}(f,U,y) = 1$  ( $f(U) \neq y$ ).
4 Noisy recurrent neural networks
In this section, we will define classes of noisy recurrent neural networks. Let us first define the singleton Gaussian noise class, which contains a single additive Gaussian noise function.
Definition 12 (The Gaussian Noise Class). The  $d$ -dimensional noise class with scale  $\sigma \geq 0$  is denoted by  $\overline{\mathcal{G}_{\sigma,d}} = \{\overline{g_{\sigma,d}}\}$ . Here,  $\overline{g_{\sigma,d}}: \mathbb{R}^d \to \mathbb{R}^d$  is a random function defined by  $\overline{g_{\sigma,d}}(\overline{x}) = \overline{x} + \overline{z}$ , where  $\overline{z} \sim \mathcal{N}(0, \sigma^2 I_d)$ . When it is clear from the context we drop  $d$  and write  $\overline{\mathcal{G}_{\sigma}} = \{\overline{g_{\sigma}}\}$ .
The following is the noisy version of multi-layer networks in Definition 5. Basically, Gaussian noise is composed (Definition 3) before each layer.
Definition 13 (Noisy Multi-Layer Sigmoid Neural Networks). The class of all noisy multi-layer sigmoid networks with  $w$  weights that take values in  $[-1/2, 1/2]^{p_0}$  as input and output values in  $[-1/2, 1/2]^{p_k}$  is defined by
$$
\overline {{M N E T _ {\sigma}}} [ p _ {0}, p _ {k}, w ] = \bigcup N E T [ p _ {k - 1}, p _ {k} ] \circ \dots \circ \overline {{\mathcal {G} _ {\sigma}}} \circ N E T [ p _ {1}, p _ {2} ] \circ \overline {{\mathcal {G} _ {\sigma}}} \circ N E T [ p _ {0}, p _ {1} ] \circ \overline {{\mathcal {G} _ {\sigma}}},
$$
where  $\sigma \geq 0$  is scale of the Gaussian noise and the union is taken over all choices of  $(p_1, p_2, \ldots, p_{k-1}) \in \mathbb{N}^{k-1}$  that satisfy  $\sum_{i=1}^{k} p_i \cdot p_{i-1} = w$ .
Similar to the deterministic case,  $\overline{\mathrm{MNET}}_{\sigma}[p_0,p_k,w]$  is said to be well-defined if the union is not empty (i.e.,  $p_0,p_k$  and  $w$  are compatible). We can use Definition 7 to create recurrent versions of the above class. For example,  $\mathrm{REC}[\overline{\mathrm{MNET}}_{\sigma}[p_0,p_k,w],T]$  is a class of recurrent (and random) hypotheses for sequence of length  $T$  that use  $\overline{\mathrm{MNET}}_{\sigma}[p_0,p_k,w]$  as their recurring block. Again, similar to the deterministic case, we say  $\mathrm{REC}[\overline{\mathrm{MNET}}_{\sigma}[p_0,p_k,w],T]$  is well-defined if  $p_0,p_k$  and  $w$  are compatible and  $\overline{\mathrm{MNET}}_{\sigma}[p_0,p_k,w]$  is well-defined.
5 PAC learning noisy recurrent neural networks
In section 3, we established an  $\Omega (T)$  lower bound on the sample complexity of learning recurrent networks (i.e.,  $\mathrm{REC}[\mathrm{MNET}[p_0,p_k,w],T]$ ). In this section, we consider a related class (based on noisy recurrent neural networks) and show that the dependence of sample complexity on  $T$  is only  $O(\log T)$ . In particular,  $\overline{\mathcal{G}_{\sigma}}\circ \mathrm{REC}[\overline{\mathrm{MNET}_{\sigma}} [p_0,p_k,w],T]$  can be regarded as a (noisy) sibling of  $\mathrm{REC}[\mathrm{MNET}[p_0,p_k,w],T]$ . Since it is more standard to define PAC learnability for deterministic hypotheses, we define the deterministic version of the above class by derandomization<sup>5</sup>.
Definition 14 (Derandomization by Expectation). Let  $\mathcal{F}$  be a class of (random) functions from  $\mathbb{R}^{p\times T}$  to  $\mathbb{R}^q$ . The derandomization of a function class  $\overline{\mathcal{F}}$  by expectation is defined as  $\mathcal{E}(\overline{\mathcal{F}}) = \{h:\mathbb{R}^{p\times T}\to \mathbb{R}^q\mid h(u) = \mathbb{E}_{\overline{f}}\left[\overline{f} (u)\right],\overline{f}\in \overline{\mathcal{F}}\}$ .
We show that, contrary to Theorem 10, the sample complexity of PAC learning the (derandomized) class of noisy recurrent neural networks,  $\mathcal{E}(\overline{\mathcal{G}_{\sigma}}\circ \mathrm{REC}[\overline{\mathrm{MNET}}_{\sigma}[p_0,p_k,w],T])$  , grows at most logarithmically with  $T$  while it still enjoys the same linear dependence on  $w$  . This is formalized in the following theorem (see Appendix ?? for a proof).
Theorem 15 (Main Result). Let  $\overline{\mathcal{Q}_w} = \overline{\mathcal{G}_{\sigma}} \circ \text{REC}[\overline{\text{MNET}_{\sigma}}[p_0, p_k, w], T]$  be any well-defined class and assume  $T \in \mathbb{N}, 0 < \sigma < 1, \epsilon, \delta \in (0, 1)$ . Then the sample complexity of learning  $\mathcal{H}_w = \mathcal{E}(\overline{\mathcal{Q}_w})$  is upper bounded by