Title: On the Role of Noise in the Sample Complexity of Learning Recurrent Neural Networks: Exponential Gaps for Long Sequences
Venue: NeurIPS 2023
Accepted: yes
Keywords: PAC Learning

Recurrent Neural Networks

Noise

Sample Complexity
Abstract: We consider the class of noisy multi-layered sigmoid recurrent neural networks with $w$ (unbounded) weights for classification of sequences of length $T$, where independent noise distributed according to $\mathcal{N}(0,\sigma^2)$ is added to the output of each neuron in the network. Our main result shows that the sample complexity of PAC learning this class can be bounded by $O (w\log(T/\sigma))$. For the non-noisy version of the same class (i.e., $\sigma=0$), we prove a lower bound of $\Omega (wT)$ for the sample complexity. 
  Our results indicate an exponential gap in the dependence of sample complexity on $T$ for noisy versus non-noisy networks. Moreover, given the mild logarithmic dependence of the upper bound on $1/\sigma$, this gap still holds even for numerically negligible values of $\sigma$.

On the Role of Noise in the Sample Complexity of Learning Recurrent Neural Networks: Exponential Gaps for Long Sequences

Abstract

We consider the class of noisy multi-layered sigmoid recurrent neural networks with w (unbounded) weights for classification of sequences of length T , where independent noise distributed according to N (0 , σ 2 ) is added to the output of each neuron in the network. Our main result shows that the sample complexity of PAC learning this class can be bounded by O ( w log( T/σ )) . For the non-noisy version of the same class (i.e., σ = 0 ), we prove a lower bound of Ω( wT ) for the sample complexity. Our results indicate an exponential gap in the dependence of sample complexity on T for noisy versus non-noisy networks. Moreover, given the mild logarithmic dependence of the upper bound on 1 /σ , this gap still holds even for numerically negligible values of σ . 1

1 Introduction

Recurrent Neural Networks (RNNs) are effective tools for processing sequential data. They are used in numerous applications such as speech recognition (Graves et al., 2013), computer vision (Karpathy and Fei-Fei, 2015), translation (Sutskever et al., 2014), modeling dynamical systems (Hardt et al., 2018) and time series (Qin et al., 2017). Recurrent models allow us to design classes of predictors that can be applied to (i.e., take input values from) sequences of arbitrary length. For processing a sequence of T elements, a predictor f (e.g., a neural network) 'consumes' the input elements one by one, generating an output at each step. This output is then used in the next step (as another input to f along with the next element in the input sequence). Defining recurrent models formally takes some effort, and we relegate it to the next sections. In short, the function f is (recursively) applied T times in order to generate the ultimate outcome.

Let us fix a base class F w of all multi-layered feed-forward sigmoid neural networks with w weights. We can create a recurrent version of this class, which we will denote by REC[ F w , T ], for classifying sequences of length T . One can study the sample complexity of PAC learning REC[ F w , T ] with respect to different loss functions. Koiran and Sontag (1998) studied the binary-valued version of this class by applying a threshold function at the end, and proved a lower bound of Ω( wT ) for its VC dimension.

There has also been efforts for proving upper bounds on the sample complexity of PAC learning REC[ F , T ] for various base classes F and different loss functions. Given the above lower bound, a gold standard has been achieving a linear dependence on T in the upper bound. Koiran and Sontag (1998) proved an upper bound of O ( w 4 T 2 ) on the VC dimension of REC[ F w , T ] discussed above. More recent papers have considered the more realistic setting of classification with continuous-valued

RNNs, e.g., by removing the threshold function and using a bounded Lipschitz surrogate loss. In this setting, Zhang et al. (2018) proved an upper bound of ˜ O ( T 4 w ∥ W ∥ O ( T ) ) on the sample complexity 2 where ∥ W ∥ is the spectral norm of the network. Chen et al. (2020) improved over this result by proving an upper bound of ˜ O ( Tw ∥ W ∥ 2 min { √ w, ∥ W ∥ O ( T ) } ) . These bounds get close to the gold standard when the spectral norm of the network satisfies ∥ W ∥ ≤ 1 .

The above upper bounds are proved by simply 'unfolding' the recurrence, effectively substituting the recurrent class REC[ F w , T ] with the (larger) class of T -fold compositions F w ◦ F w . . . ◦ F w . These unfolding techniques do not exploit the fact that the function f (that is applied recursively for T steps to compute the output of the network) is fixed across all the T steps. Consequently, the resulting sample complexity has (super-)linear dependence on T . Therefore, we would need a prohibitively large sample size for training recurrent models for classifying very long sequences. Nevertheless, this dependence is inevitable in light of the of lower bound of Koiran and Sontag (1998). Or is it?

In this paper, we consider a related class of noisy recurrent neural networks, REC[ ˜ F σ w , T ]. The hypotheses in this class are similar to those in REC[ F w , T ], except that outputs of (sigmoid) activation functions are added with independent Gaussian random variables, N (0 , σ 2 ) . Our main result demonstrates that, remarkably, the noisy class can be learned with a number of samples that is only logarithmic with respect to T .

Theorem 1 (Informal version of Theorem 15) . The sample complexity of PAC learning the class REC[ ˜ F σ w , T ] of noisy recurrent networks with respect to ramp loss is ˜ O ( w log( T/σ )) .

One challenge of proving the above theorem is that the analysis involves dealing with random hypotheses. Therefore, unlike the usual arguments that bound the covering number of a set of deterministic maps with respect to the ℓ 2 distance, we study the covering number of a class of random maps with respect to the total variation distance. We then invoke some of the recently developed tools in Fathollah Pour and Ashtiani (2022) for bounding these covering numbers. Another challenge is deviating from the usual 'unfolding method' and exploiting the fact that in recurrent models a fixed function/network is applied recursively.

The mere fact that learning REC[ ˜ F σ w , T ] requires less samples compared to its non-noisy counterpart is not entirely unexpected. For classification of long sequences, however, the sample complexity gap is quite drastic (i.e., exponential). We argue that a logarithmic dependency on T is actually more realistic in practical situations: for finite precision machines, one can effectively break the Ω( T ) barrier even for non-noisy networks. To see this, let us choose σ to be a numerically negligible number (e.g., smaller than the numerical precision of our computing device). In this case, the class of noisy and non-noisy networks become effectively the same when implemented on a device with finite numerical precision. But then our upper bound shows a mild logarithmic dependence on 1 /σ .

One caveat in the above argument is that the lower bound of Koiran and Sontag (1998) is proved for the 0-1 loss and perhaps not directly comparable to the setting of the upper bound which uses a Lipcshitz surrogate loss. We address this by showing a comparable lower bound in the same setting.

Theorem 2 (Informal version of Theorem 10) . The sample complexity of PAC learning REC[ F w , T ] with ramp loss is Ω( wT ) .

In the next section we introduce our notations and define the PAC learning problem. We state the lower bound in Section 3, and the upper bound in Section 5. Sections 6, 7, and 8 provide a high-level proof of our upper bound.

Additional Related Work. Due to space constraints, we postpone the discussion of some additional related work to Appendix 9.

2 Preliminaries

2.1 Notations

∥ x ∥ 1 , ∥ x ∥ 2 , and ∥ x ∥ ∞ denote the ℓ 1 , ℓ 2 , and ℓ ∞ norms of a vector x ∈ R d respectively. We denote the cardinality of a set S by | S | . The set of natural numbers smaller or equal to m is represented by

[ m ] . A vector of all zeros is denoted by 0 d = [0 . . . 0] ⊤ ∈ R d . We use X ⊆ R d as a domain set. We will study classes of vector-valued functions; a hypothesis is a Borel function f : R d → R p , and a hypothesis class F is a set of such hypotheses.

We find it useful to have an explicit notation-here an overline-for the random versions of the above definitions: X is the set of all random variables defined over X that admit a generalized density function 3 . x ∈ X is a random variable in this set. To simplify this notation, we sometimes just write x ∈ R d rather than x ∈ R d .

y = f ( x ) is the random variable associated with pushforward of x under Borel map f : R d → R p . We use f : R d → R p to indicate that the mapping itself is random. Random hypotheses can be applied to both random and non-random inputs-e.g., f ( x ) and f ( x ) 4 . A class of random hypotheses is denoted by F .

Definition 3 (Composition of Two Hypothesis Classes) . We denote by h ◦ f the function h ( f ( x )) (assuming the range of f and the domain of h are compatible). The composition of two hypothesis classes F and H is defined by H◦F = { h ◦ f | h ∈ H , f ∈ F} . Composition of classes of random hypotheses is defined similarly by H◦F = { h ◦ f | h ∈ H , f ∈ F} .

2.2 Feedforward neural networks

We will first define some classes associated with feedforward neural networks. Let ϕ ( x ) = 1 1+ e -x -1 2 be the centered sigmoid function. Φ : R p → [ -1 / 2 , 1 / 2] p is the element-wise sigmoid activation function defined by Φ(( x (1) , . . . , x ( p ) )) = ( ϕ ( x (1) ) , . . . , ϕ ( x ( p ) )) .

Definition 4 (Single-Layer Sigmoid Neural Networks) . The class of single-layer sigmoid neural networks with d inputs and p outputs is defined by NET[ d, p ] = { f W : R d → [ -1 / 2 , 1 / 2] p | f W ( x ) = Φ( W ⊤ x ) , W ∈ R d × p } .

Based on Definition 4, we can define the class of multi-layer (feedforward) neural networks (with w weights) as a composition of several single-layer networks. Note that the number of hidden neurons can be arbitrary as long as the total number of weights/parameters is w .

Definition 5 (Multi-Layer Sigmoid Neural Networks) . A class of multi-layer sigmoid networks with p 0 inputs, p k outputs, and w weights that take inputs in [ -1 / 2 , 1 / 2] p 0 is defined by

where union is taken over all choices of ( p 1 , p 2 , . . . , p k -1 ) ∈ N k -1 that satisfy ∑ k i =1 p i .p i -1 = w . We say MNET[ p 0 , p k , w ] is well-defined if the union is not empty.

Well-definedness basically means that p 0 , p k , and w are compatible. For simplicity, in the above definition we restricted the input domain to [ -1 / 2 , 1 / 2] d . This will help in defining the recurrent versions of these networks (since the input and output domains become compatible). However, our analysis can be easily extended to capture any bounded domain (e.g., [ -B,B ] d ).

2.3 Recursive application of a function and recurrent models

In this section we define REC[ F , T ] which is the recurrent version of class F for sequences of length T . Let v = ( a 1 , . . . , a m ) ∈ X m for m ∈ N . We define First ( v ) = ( a 1 , . . . , a m -1 ) ∈ X m -1 and Last ( v ) = a m ∈ X as functions that return the first m -1 and the last dimensions of the vector v , respectively. Let u (0) , u (1) , . . . , u ( T -1) be a sequence of inputs, where u ( i ) ∈ R p , and let f : R s → R q be a hypothesis/mapping. In the context of recurrent models, it is useful to define the recurrent application of f on this sequence. Note that out of the q dimensions of the range of f , q -1 of them are recurrent and therefore are fed back to the model. Basically, f R ( U, t ) will be the result of applying f on the first t elements of U (with recurrent feedback).

f

0

q

-

1

u

(0)

First

(

f

R

(

U,

0)

)

f

First

(

f

R

(

U, t

-

1)

)

u

(

t

)

First

(

f

R

(

U, t

)

)

f

First

(

f

R

(

U, T

-

2)

)

u

(

T

-

1)

Last

(

f

R

(

U, T

-

1))

. . .

. . .

Definition 6 (Recurrent Application of a Function) . Let U = [ u (0) . . . u ( i ) . . . u ( T -1) ] ∈ R p × T be a sequence of inputs of length T , where u ( i ) ∈ R p denotes the i -th column of U for 0 ≤ i ≤ T -1 . Let f be a (random) function from R s to R q , where s = p + q -1 . Moreover, define f R ( U, 0) = f ( [ 0 q -1 u (0) ] ⊤ ) . Then, for any 1 ≤ t ≤ T -1 , the recursive application of f is denoted by f R : R p × T × [ T -1] → R q and is defined as f R ( U, t ) = f ( [ First ( f R ( U, t -1) ) u ( t ) ] ⊤ ) .

Now we are ready to define the (recurrent) hypothesis class REC[ F , T ]. Each hypothesis in this class takes a sequence U of input vectors, and applies a function f ∈ F recurrently on the elements of this sequence. The final output will be a real number. We give the formal definition in the following; also see Figure 1 for a visualization.

Definition 7 (Recurrent Class) . Let s, p, q ∈ N such that s = p + q -1 . Let F be a class of functions from R s to R q . The class of recurrent models with length T that use functions in F (which we denote by recurring class) as their recurring block is defined by

For example, REC[MNET[ p 0 , p k , w ] , T ] is the class of (real-valued) recurrent neural networks with length T that use MNET[ p 0 , p k , w ] as their recurring block. We say that REC[MNET[ p 0 , p k , w ] , T ] is well-defined if MNET[ p 0 , p k , w ] is well-defined and also the input/output dimensions are compatible (i.e., p 0 ≥ p k ).

2.4 PAC learning with ramp loss

In this section we formulate the PAC learning model for classification with respect to the ramp loss. The use of ramp loss is natural for classification (see e.g., Boucheron et al. (2005); Bartlett et al. (2006)) and the main features of the ramp loss that we are going to exploit are boundedness and Lipschitzness. We start by introducing the ramp loss.

Definition 8 (Ramp Loss) . Let f : X → R be a hypothesis and let D be a distribution over X × Y . Let ( x, y ) ∈ X × Y , where Y = {-1 , 1 } . The ramp loss of f with respect to margin parameter γ > 0 is defined as l γ ( f, x, y ) = r γ ( -f ( x ) .y ) , where r γ is the ramp function defined by

Definition 9 (Agnostic PAC Learning with Respect to Ramp Loss) . We say that a hypothesis class F of functions from X to R is agnostic PAC learnable with respect to ramp loss with margin parameter γ > 0 if there exists a learner A and a function m : (0 , 1) 2 → N with the following property: For every distribution D over X × {1 , 1 } and every ϵ, δ ∈ (0 , 1) , if S is a set of m ( ϵ, δ ) i.i.d. samples from D , then with probability at least 1 -δ (over the randomness of S ) we have

The sample complexity of PAC learning F with respect to ramp loss is denoted by m F ( ϵ, δ ) , which is the minimum number of samples required for learning F (among all learners A ). The definition of agnostic PAC learning with respect to ramp loss works for any value of γ and when we are analyzing the sample complexity we consider it to be a fixed constant.

3 Alower bound for sample complexity of learning recurrent neural networks

In this section, we consider the sample complexity of PAC learning sigmoid recurrent neural networks with respect to ramp loss. Particularly, we state a lower bound on the sample complexity of the class REC[MNET[ p 0 , p k , w ] , T ] of all sigmoid recurrent neural networks with length T that use multi-layer neural networks with w weights as their recurring block. The main message is that this sample complexity grows at least linearly with T .

Theorem 10 (Sample Complexity Lower Bound for Recurrent Neural Networks) . For every T ≥ 3 and w ≥ 19 there exists a well-defined class H w = REC[MNET[ p 0 , p k , w ] , T ] and a universal constant C > 0 such that for every ϵ, δ ∈ (0 , 1 / 40) we have

The proof of the above lower bound is based on a similar result due to Sontag et al. (1998). However, the argument in Sontag et al. (1998) is for PAC learning with respect to 0-1 loss. To extend this result for the ramp loss, we construct a binary-valued class F w = { f : f ( U ) = sign ( h ( U )) , h ∈ H w } where sign ( x ) = 1 if x ≥ 0 and sign ( x ) = -1 if x < 0 . We prove that every function f ∈ F w can be related to another function h ∈ H w such that the ramp loss of h is almost equal to the zero-one loss of f . This is formalized in the following lemma, which is a key result in proving Theorem 10. The proof of Theorem 10 and Lemma 11 can be found in Appendix ?? .

̸

Lemma 11. Let H w = REC[MNET[ p 0 , p k , w ] , T ] be a well-defined class and let F w = { f : [ -1 / 2 , 1 / 2] p × T →{-1 , 1 } | f ( U ) = sign ( h ( U )) , h ∈ H w } . Then, for every distribution D over [ -1 / 2 , 1 / 2] p × T ×{-1 , 1 } , η > 0 , and every function f ∈ F w there exists a function h ∈ H w such that E ( U,y ) ∼D [ l γ ( h, U, y )] ≤ E ( U,y ) ∼D [ l 0 -1 ( f, U, y ) ] + η where l 0 -1 ( f, U, y ) = 1 { f ( U ) = y } .

4 Noisy recurrent neural networks

In this section, we will define classes of noisy recurrent neural networks. Let us first define the singleton Gaussian noise class, which contains a single additive Gaussian noise function.

Definition 12 (The Gaussian Noise Class) . The d -dimensional noise class with scale σ ≥ 0 is denoted by G σ,d = { g σ,d } . Here, g σ,d : R d → R d is a random function defined by g σ,d ( x ) = x + z , where z ∼ N ( 0 , σ 2 I d ) . When it is clear from the context we drop d and write G σ = { g σ } .

The following is the noisy version of multi-layer networks in Definition 5. Basically, Gaussian noise is composed (Definition 3) before each layer.

Definition 13 (Noisy Multi-Layer Sigmoid Neural Networks) . The class of all noisy multi-layer sigmoid networks with w weights that take values in [ -1 / 2 , 1 / 2] p 0 as input and output values in [ -1 / 2 , 1 / 2] p k is defined by

where σ ≥ 0 is scale of the Gaussian noise and the union is taken over all choices of ( p 1 , p 2 , . . . , p k -1 ) ∈ N k -1 that satisfy ∑ k i =1 p i .p i -1 = w .

Similar to the deterministic case, MNET σ [ p 0 , p k , w ] is said to be well-defined if the union is not empty (i.e., p 0 , p k and w are compatible). We can use Definition 7 to create recurrent versions of the above class. For example, REC[MNET σ [ p 0 , p k , w ] , T ] is a class of recurrent (and random) hypotheses for sequence of length T that use MNET σ [ p 0 , p k , w ] as their recurring block. Again, similar to the deterministic case, we say REC[MNET σ [ p 0 , p k , w ] , T ] is well-defined if p 0 , p k and w are compatible and MNET σ [ p 0 , p k , w ] is well-defined.

5 PAC learning noisy recurrent neural networks

In section 3, we established an Ω( T ) lower bound on the sample complexity of learning recurrent networks (i.e., REC[MNET[ p 0 , p k , w ] , T ]). In this section, we consider a related class (based on noisy recurrent neural networks) and show that the dependence of sample complexity on T is only O (log T ) . In particular, G σ ◦ REC[MNET σ [ p 0 , p k , w ] , T ] can be regarded as a (noisy) sibling of REC[MNET[ p 0 , p k , w ] , T ]. Since it is more standard to define PAC learnability for deterministic hypotheses, we define the deterministic version of the above class by derandomization 5 .

Definition 14 (Derandomization by Expectation) . Let F be a class of (random) functions from R p × T to R q . The derandomization of a function class F by expectation is defined as E ( F ) = { h : R p × T → R q | h ( u ) = E f [ f ( u ) ] , f ∈ F} .

We show that, contrary to Theorem 10, the sample complexity of PAC learning the (derandomized) class of noisy recurrent neural networks, E ( G σ ◦ REC[MNET σ [ p 0 , p k , w ] , T ] ) , grows at most logarithmically with T while it still enjoys the same linear dependence on w . This is formalized in the following theorem (see Appendix ?? for a proof).

Theorem 15 (Main Result) . Let Q w = G σ ◦ REC[MNET σ [ p 0 , p k , w ] , T ] be any well-defined class and assume T ∈ N , 0 < σ < 1 , ϵ, δ ∈ (0 , 1) . Then the sample complexity of learning H w = E ( Q w ) is upper bounded by

where ˜ O hides logarithmic factors.

One feature of the above theorem is the mild logarithmic dependence on 1 /σ . Therefore, we can take σ to be numerically negligible and still get a significantly smaller sample complexity compared to the deterministic case for large T . Note that adding such small values of noise would not change the empirical outcome of RNNs on finite precision computers.

The milder (logarithmic) dependency on T is achieved by a novel analysis that involves bounding the covering number of noisy recurrent networks with respect to the total variation distance. Also, instead of 'unfolding' the network, we exploit the fact that the same function/hypothesis is being used recurrently. We also want to emphasize that the above bound does not depend on the norms of weights of the network. Achieving this is challenging, since a little bit of noise in a previous layer can change the output of the next layer drastically. The next few sections are dedicated to give a high-level proof of this theorem.

6 Covering numbers: the classical view

One of the main tools to derive sample complexity bounds for learning a class of functions is studying their covering numbers. In this section we formalize this classic tool.

Definition 16 (Covering Number) . Let ( X , ρ ) be a metric space. A set A ⊂ X is ϵ -covered by a set C ⊆ A with respect to ρ , if for all a ∈ A there exists c ∈ C such that ρ ( a, c ) ≤ ϵ . We denote by N ( ϵ, A, ρ ) the cardinality of the smallest set C that ϵ -covers A and we refer to is as the ϵ -covering number of A with respect to metric ρ .

The notion of covering number is defined with respect to a metric ρ . We now give the definition of extended metrics, which we will use to define uniform covering numbers. The extended metrics can be seen as measures of distance between two hypotheses on a given input set.

Definition 17 (Extended Metrics) . Let ( X , ρ ) be a metric space. Let u = ( a 1 , . . . , a m ) , v = ( b 1 , . . . , b m ) ∈ X m for m ∈ N . The ∞ -extended and ℓ 2 -extended metrics over X m are defined by ρ ∞ ,m ( u, v ) = sup 1 ≤ i ≤ m ρ ( a i , b i ) and ρ ℓ 2 ,m ( u, v ) = √ 1 m ∑ m i =1 ( ρ ( a i , b i )) 2 , respectively. We drop m and use ρ ∞ or ρ ℓ 2 if it is clear from the context.

A useful property about extended metrics is that the ∞ -extended metric always upper bounds the ℓ 2 -extended metric, i.e., ρ ℓ 2 ( u, v ) ≤ ρ ∞ ( u, v ) for all u, v ∈ X . Based on the above definition of extended metrics, we define the uniform covering number of a hypothesis class with respect to ∥ . ∥ 2 .

Definition 18 (Uniform Covering Number with Respect to ∥ . ∥ 2 ) . Let F be a hypothesis class of functions from X to Y . For a set of inputs S = { x 1 , x 2 , . . . , x m } ⊆ X , we define the restriction of F to S as F | S = { ( f ( x 1 ) , f ( x 2 ) , . . . , f ( x m )) : f ∈ F} ⊆ Y m . The uniform ϵ -covering numbers of hypothesis class F with respect to ∥ . ∥ ∞ 2 , ∥ . ∥ ℓ 2 2 are denoted by N U ( ϵ, F , m, ∥ . ∥ ∞ 2 ) and N U ( ϵ, F , m, ∥ . ∥ ℓ 2 2 ) and are the maximum values of N ( ϵ, F | S , ∥ . ∥ ∞ ,m 2 ) and N ( ϵ, F | S , ∥ . ∥ ℓ 2 ,m 2 ) over all S ⊆ X with | S | = m , respectively.

The following theorem connects the notion of uniform covering number with PAC learning. It converts a bound on the ∥ . ∥ ℓ 2 2 uniform covering number of a hypothesis class to a bound on the sample complexity of PAC learning the class; see Appendix ?? for a more detailed discussion.

Theorem 19. Let F be a class of functions from X to R . Then there exists an algorithm A with the following property: For every distribution D over X × {1 , 1 } and every ϵ, δ ∈ (0 , 1) , if S is a set of m i.i.d. samples from D , then with probability at least 1 -δ (over the randomness of S ),

Moreover, the algorithm that returns the function with the minimum error on S satisfies the above property (i.e., Algorithm A such that A ( S ) = arg min f ∈F 1 | S | ∑ ( x,y ) ∈ S l γ ( f, x, y ) ).

7 Total variation covers for random hypotheses

One idea to prove a generalization bound for noisy neural networks is to bound their covering numbers. However, noisy neural networks are random functions, and therefore their behaviours on a sample set cannot be directly compared. Instead, one can compare the output distributions of a random function on two sample sets. We therefore use the recently developed tools from Fathollah Pour and Ashtiani (2022) to define and study covering numbers for random hypotheses. These covering numbers are defined based on metrics between distributions. Specifically, our analysis is based on the notion of uniform covering number with respect to total variation distance.

Definition 20 (Total Variation Distance) . Let µ and ν denote two probability measures over X and let Ω be the Borel sigma-algebra over X . The TV distance between µ and ν is defined by

Furthermore, if µ and ν have densities f and g then

For two random variables x and y with probability measures µ and ν we sometimes abuse the notation and write d TV ( x, y ) instead of d TV ( µ, ν ) . For example, we write d TV ( f 1 ( x ) , f 2 ( x )) in order to refer to the Total Variation (TV) distance between pushforwards of x under mappings f 1 and f 2 . We also write d ∞ ,m TV (( f 1 ( x 1 ) , . . . , f 1 ( x m ) ) , ( f 2 ( x 1 ) , . . . , f 2 ( x m ) )) to refer to the extended TV distance between mappings of the set S = { x 1 , . . . , x m } by f 1 and f 2 . We use the extended total variation distance to define the uniform covering number for classes of random hypotheses.

Definition 21 (Uniform Covering Number for Classes of Random Hypotheses) . Let F be a class of random hypotheses from X to Y . For a set of random variables S = { x 1 , x 2 , . . . , x m } ⊆ X , the restriction of F to S is defined as F | S = { ( f ( x 1 ) , f ( x 2 ) , . . . , f ( x m )) : f ∈ F} ⊆ Y m . Let Γ ⊆ X . The uniform ϵ -covering numbers of F with respect to Γ and d ∞ is defined by

TV

Some hypothesis classes that we analyze (e.g., single-layer noisy neural networks) may have 'global' total variation covers that do not depend on m . This will be addressed with the following notation: N U ( ϵ, F , ∞ , ρ ∞ , Γ) = lim m →∞ N U ( ϵ, F , m, ρ ∞ , Γ) . The set Γ in Definition 21 is used to define the input domain for which we want to find the covering number of a class of random hypotheses. For instance, some of the covers that we see are derived with respect to inputs with bounded domain or some need the input to be first smoothed by Gaussian noise. In this paper, we will be working with the following choices of Γ

-Γ = X d and Γ = X B,d : the set of all random variables defined over R d and [ -B,B ] d , respectively, that admit a generalized density function. For example, we use X 0 . 5 ,d to address the set of random variables in [ -1 / 2 , 1 / 2] d .

-Γ = ∆ p × T = { U | U = [ δ u (0) . . . δ u ( T -1) ] ⊤ , u ( i ) ∈ R p } and Γ = ∆ B,p × T = { U | U = [ δ u (0) . . . δ u ( T -1) ] ⊤ , u ( i ) ∈ [ -B,B ] p } , where δ u ( i ) is the random variable associated with Dirac delta measure on u ( i ) . Note that ∆ B,p × T ⊂ ∆ p × T .

-Γ = G σ,d ◦ X B,d = { g σ,d ( x ) | x ∈ X B,d } : all members of X B,d after being 'smoothed' by adding (convolving the density with) Gaussian noise.

We mentioned in Section 6 that a bound on the ∥ . ∥ ℓ 2 2 uniform covering number can be connected to a bound on sample complexity of PAC learning. We now show that a bound on d ∞ TV covering number of a class of random hypotheses can be turned into a bound on the ∥ . ∥ ℓ 2 2 covering number of its derandomized version and, thus, PAC learning it.

Theorem 22 ( ∥ . ∥ ℓ 2 2 Cover of E ( F ) From d ∞ TV Cover of F (Fathollah Pour and Ashtiani, 2022)) . Let F be a class of functions from R p × T to [ -B,B ] q . Then for every ϵ > 0 and m ∈ N we have

8 Bounding the covering number of recurrent models

In Section 6, we mentioned that finding a bound on covering number of a hypothesis class is a standard approach to bound its sample complexity. In the previous section, we introduced a new notion of covering number with respect to total variation distance that was developed by Fathollah Pour and Ashtiani (2022). We showed how this notion can be related to PAC learning for classes of random hypotheses. In the following, we give an overview of the techniques used to find a bound on the d ∞ TV covering number of the class of noisy recurrent models. We also discuss why this bound results in a sample complexity that has a milder logarithmic dependency on T , compared to bounds proved by 'unfolding' the recurrence and replacing the recurrent model with the T -fold composition.

One advantage of analyzing the uniform covering number with respect to TV distance is that it comes with a useful composition tool. The following theorem basically states that when two classes of hypotheses have bounded TV covers, their composition class has a bounded cover too. Note that such a result does not hold for the usual definition of covering number (e.g., Definition 18); see Fathollah Pour and Ashtiani (2022) for details.

Theorem 23 (TV Cover for Composition of Random Classes, Lemma 18 of Fathollah Pour and Ashtiani (2022)) . Let F be a class of random hypotheses from R d to R p and H be a class of random hypotheses from R p to R q . For any ϵ 1 , ϵ 2 > 0 and m ∈ N , denote N 1 = N U ( ϵ 1 , F , m, d ∞ TV , X d ) . Then we have,

An approach to bound the TV uniform covering number of a recurrent model REC[ F , T ] is to consider it as the T -fold composition F ◦ F . . . ◦ F . One can then use a similar analysis to that of Fathollah Pour and Ashtiani (2022) to bound the covering number of the T -fold composition. Unfortunately, this approach fails to capture the fact that a fixed function f ∈ F is applied recursively, and therefore results in a sample complexity bound that grows at least linearly with T .

Instead, we take another approach to bound the covering number of recurrent models. Intuitively, we notice that any function in the T -fold composite class F ◦ . . . ◦ F = { f 1 ◦ . . . ◦ f T | f 1 , . . . , f T ∈

F} is determined by T functions from F . On the other hand, any function in REC[ F , T ] = { h | h ( U ) = Last ( f R ( U, T -1) )} is only defined by one function in F and the capacity of this class must not be as large as the capacity of F ◦ . . . ◦ F . Interestingly, data processing inequality for total variation distance (Lemma ?? ) suggests that if two functions f and ˆ f are 'globally' close to each other with respect to TV distance (i.e., d TV ( f ( x ) , ˆ f ( x )) ≤ ϵ for every x in the domain), then d TV ( f ( f ( x )) , ˆ f ( ˆ f ( x ))) ≤ 2 ϵ (i.e., f ◦ f and ˆ f ◦ ˆ f are also close to each other). By applying the data processing inequality recursively, we can see that for the T -fold composition we have d TV ( f ◦ . . . ◦ f ( x ) , ˆ f ◦ . . . ◦ ˆ f ( x )) ≤ ϵT . The above approach results in the following theorem which bounds the ϵ -covering number of a noisy recurrent model with respect to TV distance by the ( ϵ/T ) -covering number of its recurring class. Intuitively, this theorem helps us to bound the covering number of noisy recurrent models using the bounds obtained for their non-recurrent versions. Here, Gaussian noise is added to both the input of the model (i.e., F σ = F ◦ G σ ) and the output of the model (by composing with G σ ).

Theorem 24 (TV Covering Number of G σ ◦ REC[ F σ , T ] From G σ ◦ F σ ) . Let s, p, q ∈ N such that s = p + q -1 . Let F be a class of functions from X B,s to X B,q and denote by F σ = F ◦ G σ,s the class of its composition with noise. Then we have

For using this theorem, one needs to have a finer ϵ/T -cover for the recurring class. As we will see in the next section, this will translate into a mild logarithmic sample complexity dependence on T .

8.1 Covering noisy recurrent networks

An example of F σ is the class MNET σ [ p 0 , p k , w ] of well-defined noisy multi-layer networks (Definition 13). Theorem 24 suggests that a bound on the covering number of G σ ◦ REC[MNET σ [ p 0 , p k , w ] , T ] can be found from a bound for G σ ◦ MNET σ [ p 0 , p k , w ]. We use the following theorem as a bound for the class of single-layer noisy sigmoid networks together with theorem 23 to bound the covering number of G σ ◦ MNET σ [ p 0 , p k , w ] (see Appendix ?? , Theorem ?? ).

Theorem 25 (A TV Cover for Single-Layer Noisy Neural Networks, Theorem 25 of Fathollah Pour and Ashtiani (2022)) . For every p, d ∈ N , ϵ > 0 , σ < 5 d/ϵ we have

Interestingly, the above bound (on the logarithm of the covering number) is logarithmic with respect to 1 /ϵ . We will extend this result to multi-layer noisy networks, and then apply Theorem 24 to obtain the following bound on the covering number noisy recurrent neural networks. Crucially, the dependency (of the logarithm of the covering number) on T is only logarithmic.

Theorem 26 (A TV Covering Number Bound for Noisy Sigmoid Recurrent Networks) . Let T ∈ N . For every ϵ, σ ∈ (0 , 1) and every well-defined class REC[MNET σ [ p 0 , p k , w ] , T ] we have

Finally, we turn the above bound into a ∥ . ∥ ℓ 2 2 covering number bound for the derandomized function E ( G σ ◦ REC[MNET σ [ p 0 , p k , w ] , T ] ) by an application of Theorem 22. We then upper bound the sample complexity by the logarithm of covering number (see Theorem 19) and conclude Theorem 15. Limitations and future work. Our results are derived for sigmoid (basically bounded, monotone, and Lipschitz) activation functions. It is open whether such results can be proved for unbounded activation functions such as RELU. Our results are theoretical and we leave empirical evaluations on the performance of noisy networks to future work.

9 More on related work

There is plethora of work on generalization in neural networks. There are a family of approaches that aim to bound the VC-dimension of neural networks. (Baum and Haussler, 1988; Maass, 1994; Goldberg and Jerrum, 1995; Vidyasagar, 1997; Sontag et al., 1998; Koiran and Sontag, 1998; Bartlett et al., 1998; Bartlett and Maass, 2003; Bartlett et al., 2019). These approaches result in generalization bounds that are dependent on the number of parameters. Another family of approaches are aimed at obtaining generalization bounds that are dependent on the norms of the weights and Lipschitz continuity properties of the network (Bartlett, 1996; Anthony et al., 1999; Zhang, 2002; Neyshabur et al., 2015; Bartlett et al., 2017; Neyshabur et al., 2018; Golowich et al., 2018; Arora et al., 2018; Nagarajan and Kolter, 2018; Long and Sedghi, 2020). It has been observed that these generalization bounds are usually vacuous in practice. One speculation is that the implicit bias of gradient descent (Gunasekar et al., 2017; Arora et al., 2019; Ji et al., 2020; Chizat and Bach, 2020; Ji and Telgarsky, 2021) can lead to benign overfitting (Belkin et al., 2018, 2019; Bartlett et al., 2020, 2021). It has also been conjectured that uniform convergence theory may not be able to fully capture the performance of neural networks in practice (Nagarajan and Kolter, 2019; Zhang et al., 2021). It has been shown that there are data-dependent approaches that can achieve non-vacuouys bounds (Dziugaite and Roy, 2017; Zhou et al., 2019; Negrea et al., 2019). There are also other approaches that are independent of data (Arora et al., 2018); see Fathollah Pour and Ashtiani (2022) for more details.

Adding different types of noise such as dropout noise (Srivastava et al., 2014), DropConnect (Wan et al., 2013), and Denoising AutoEncoders (Vincent et al., 2008) are shown to be helpful in training neural networks. Wang et al. (2019) and Gao and Zhou (2016) theoretically analyze the generalization under dropout noise. More recently, Fathollah Pour and Ashtiani (2022) developed a framework to study the generalization of classes of noisy hypotheses and show that adding noise to the output of neurons in a network can be helpful in generalization. Jim et al. (1996) show that additive and multiplicative noise can help speed up the convergence of RNNs on local minima surfaces. Recently, Lim et al. (2021) showed that noisy RNNs are more stable and robust to input perturbations by formalizing the regularization effects of noise.

Another line of work focuses on the generalization of neural network that are trained with Stochastic Gradient Descent (SGD) or its noisy variant Stochastic Gradient Langevin Descent (SGLD) (Russo and Zou, 2016; Xu and Raginsky, 2017; Russo and Zou, 2019; Steinke and Zakynthinou, 2020; Raginsky et al., 2017; Haghifam et al., 2020; Neu et al., 2021). Zhao et al. (2020) analyze the memory properties of recurrent networks and how well they can remember the input sequence. Tu et al. (2020) study the generalization of RNN by analyzing the Fisher-Rao norm of weights, which they obtain from the gradients of the network. They offer generalization bounds that can potentially become polynomial in T . Allen-Zhu and Li (2019) analyze the change in output through the dynamics of training RNNs and prove generalization bounds for recurrent networks that are again polynomial in T .

References

Zeyuan Allen-Zhu and Yuanzhi Li. Can sgd learn recurrent neural networks with provable generalization? Advances in Neural Information Processing Systems , 32, 2019.

Martin Anthony, Peter L Bartlett, Peter L Bartlett, et al. Neural network learning: Theoretical foundations , volume 9. cambridge university press Cambridge, 1999.

Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In International Conference on Machine Learning , pages 254-263. PMLR, 2018.

Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems , 32, 2019.

Peter Bartlett. For valid generalization the size of the weights is more important than the size of the network. Advances in neural information processing systems , 9, 1996.

Peter Bartlett, Vitaly Maiorov, and Ron Meir. Almost linear vc dimension bounds for piecewise polynomial networks. Advances in neural information processing systems , 11, 1998.

Peter L Bartlett and Wolfgang Maass. Vapnik-chervonenkis dimension of neural nets. The handbook of brain theory and neural networks , pages 1188-1192, 2003.

Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association , 101(473):138-156, 2006.

Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems , 30, 2017.

Peter L Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight vc-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research , 20(1):2285-2301, 2019.

Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020.

Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta numerica , 30:87-201, 2021.

Eric Baum and David Haussler. What size net gives valid generalization? Advances in neural information processing systems , 1, 1988.

Mikhail Belkin, Daniel J Hsu, and Partha Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. Advances in neural information processing systems , 31, 2018.

Mikhail Belkin, Alexander Rakhlin, and Alexandre B Tsybakov. Does data interpolation contradict statistical optimality? In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1611-1619. PMLR, 2019.

Stéphane Boucheron, Olivier Bousquet, and Gábor Lugosi. Theory of classification: A survey of some recent advances. ESAIM: probability and statistics , 9:323-375, 2005.

Minshuo Chen, Xingguo Li, and Tuo Zhao. On generalization bounds of a family of recurrent neural networks. In International Conference on Artificial Intelligence and Statistics , pages 1233-1243. PMLR, 2020.

Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory , pages 1305-1338. PMLR, 2020.

Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Proceedings of the 33rd Annual Conference on Uncertainty in Artificial Intelligence (UAI) , 2017.

Alireza F. Pour and Hassan Ashtiani. On the role of noise in the sample complexity of learning recurrent neural networks: Exponential gaps for long sequences. arXiv preprint arXiv:2305.18423 , 2023.

Alireza Fathollah Pour and Hassan Ashtiani. Benefits of additive noise in composing classes with bounded capacity. Advances in Neural Information Processing Systems , 35:32709-32722, 2022.

Wei Gao and Zhi-Hua Zhou. Dropout rademacher complexity of deep neural networks. Science China Information Sciences , 59(7):1-12, 2016.

Paul W Goldberg and Mark R Jerrum. Bounding the vapnik-chervonenkis dimension of concept classes parameterized by real numbers. Machine Learning , 18(2):131-148, 1995.

Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory , pages 297-299. PMLR, 2018.

Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing , pages 6645-6649. Ieee, 2013.

Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems , 30, 2017.

Mahdi Haghifam, Jeffrey Negrea, Ashish Khisti, Daniel M Roy, and Gintare Karolina Dziugaite. Sharpened generalization bounds based on conditional mutual information and an application to noisy, iterative algorithms. Advances in Neural Information Processing Systems , 33:9925-9935, 2020.

Moritz Hardt, Tengyu Ma, and Benjamin Recht. Gradient descent learns linear dynamical systems. Journal of Machine Learning Research , 19:1-44, 2018.

Ziwei Ji and Matus Telgarsky. Characterizing the implicit bias via a primal-dual analysis. In Algorithmic Learning Theory , pages 772-804. PMLR, 2021.

Ziwei Ji, Miroslav Dudík, Robert E Schapire, and Matus Telgarsky. Gradient descent follows the regularization path for general losses. In Conference on Learning Theory , pages 2109-2136. PMLR, 2020.

Kam-Chuen Jim, C Lee Giles, and Bill G Horne. An analysis of noise in recurrent neural networks: convergence and generalization. IEEE Transactions on neural networks , 7(6):1424-1438, 1996.

Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 31283137, 2015.

Pascal Koiran and Eduardo D Sontag. Vapnik-chervonenkis dimension of recurrent neural networks. Discrete Applied Mathematics , 86(1):63-79, 1998.

Soon Hoe Lim, N Benjamin Erichson, Liam Hodgkinson, and Michael W Mahoney. Noisy recurrent neural networks. Advances in Neural Information Processing Systems , 34, 2021.

Philip M Long and Hanie Sedghi. Size-free generalization bounds for convolutional neural networks. In International Conference on Learning Representations , 2020.

Wolfgang Maass. Neural nets with superlinear vc-dimension. Neural Computation , 6(5):877-884,

1994.

Vaishnavh Nagarajan and J Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. Advances in Neural Information Processing Systems , 32, 2019.

Vaishnavh Nagarajan and Zico Kolter. Deterministic pac-bayesian generalization bounds for deep networks via generalizing noise-resilience. In International Conference on Learning Representations , 2018.

Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M Roy. Information-theoretic generalization bounds for sgld via data-dependent estimates. Advances in Neural Information Processing Systems , 32, 2019.

Gergely Neu, Gintare Karolina Dziugaite, Mahdi Haghifam, and Daniel M Roy. Information-theoretic generalization bounds for stochastic gradient descent. In Conference on Learning Theory , pages 3526-3545. PMLR, 2021.

Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory , pages 1376-1401. PMLR, 2015.

Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrallynormalized margin bounds for neural networks. In International Conference on Learning Representations , 2018.

Yao Qin, Dongjin Song, Haifeng Cheng, Wei Cheng, Guofei Jiang, and Garrison W Cottrell. A dual-stage attention-based recurrent neural network for time series prediction. In Proceedings of the 26th International Joint Conference on Artificial Intelligence , pages 2627-2633, 2017.

Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis. In Conference on Learning Theory , pages 1674-1703. PMLR, 2017.

Daniel Russo and James Zou. Controlling bias in adaptive data analysis using information theory. In Artificial Intelligence and Statistics , pages 1232-1240. PMLR, 2016.

Daniel Russo and James Zou. How much does your data exploration overfit? controlling bias via information usage. IEEE Transactions on Information Theory , 66(1):302-323, 2019.

Eduardo D Sontag et al. Vc dimension of neural networks. NATO ASI Series F Computer and Systems Sciences , 168:69-96, 1998.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research , 15(1):1929-1958, 2014.

Thomas Steinke and Lydia Zakynthinou. Reasoning about generalization via conditional mutual information. In Conference on Learning Theory , pages 3437-3452. PMLR, 2020.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems , 27, 2014.

Zhuozhuo Tu, Fengxiang He, and Dacheng Tao. Understanding generalization in recurrent neural networks. In International Conference on Learning Representations , 2020.

Mathukumalli Vidyasagar. A theory of learning and generalization: with applications to neural networks and control systems . Springer-Verlag, 1997.

Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning , pages 1096-1103, 2008.

Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning , pages 1058-1066. PMLR, 2013.

Haotian Wang, Wenjing Yang, Zhenyu Zhao, Tingjin Luo, Ji Wang, and Yuhua Tang. Rademacher dropout: An adaptive dropout for deep neural network via optimizing generalization gap. Neurocomputing , 357:177-187, 2019.

Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems , 30, 2017.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

Jiong Zhang, Qi Lei, and Inderjit Dhillon. Stabilizing gradients for deep neural networks via efficient svd parameterization. In International Conference on Machine Learning , pages 5806-5814. PMLR, 2018.

Tong Zhang. Covering number bounds of certain regularized linear function classes. Journal of Machine Learning Research , 2(Mar):527-550, 2002.

Jingyu Zhao, Feiqing Huang, Jia Lv, Yanjie Duan, Zhen Qin, Guodong Li, and Guangjian Tian. Do rnn and lstm have long memory? In International Conference on Machine Learning , pages 11365-11375. PMLR, 2020.

Wenda Zhou, Victor Veitch, Morgane Austern, Ryan P Adams, and Peter Orbanz. Non-vacuous generalization bounds at the imagenet scale: a pac-bayesian compression approach. In International Conference on Learning Representations (ICLR) , 2019.

Peer reviews:
comments: SUMMARY:
The authors consider the class of noisy multi-layered sigmoid recurrent neural networks, noisy meaning that noise is added to the output of each neuron in the network. They prove that the sample complexity of noisy is significantly better than non-noisy one with one clean upper bound and one clean lower bound.

STRENGTHS:
1. Clear writing
2. Well organised presentation
3. Reasonable assumptions on an important model
4. Studied multilayer neural networks, making a complete story.
5. Interesting technique TV Cover, handy for composition classes

WEAKNESSES:
Here are some minor weaknesses.

1. The authors could have discussed more about the practical implications. It occurs to me that adding noise might be one of the keys to next generation AI now that deep learning is currently struggling with sampling complexity.
2. Would be better if there could be discussion about expressive power.
3. sigmoid not RELU.
4. lack of experimental results

QUESTIONS:
How much does noise affect expressive power?

LIMITATIONS:
The authors adequately addressed most of the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the sample complexity of PAC learning noisy recurrent neural networks with respect to the ramp loss. Noisy recurrent networks are defined as multi-layered feed forward networks with sigmoid activation where independent mean-zero Gaussian noise is added to the activation.\n\nThe main result is that one can learn these noisy networks with sample complexity $\\tilde{O}(\\frac{w\\log(T/\\sigma) + \\log(1/\\delta)}{\\epsilon^2})$ (Theorem 15). Though different in terms of loss function and noise assumption with prior work, the bound notably avoids any dependence on the norms of the weights, which was common in upper bounds in prior work. These results are much more intriguing once contrasted to its non-noisy counterpart, i.e., the fact that non-noisy PAC learning has a lower bound of $\\Omega(\\frac{wT+\\log(1/\\delta)}{\\epsilon^2})$ (Theorem 10), which exhibits linear dependence on $T$ as opposed to $\\log(T)$ as in the noisy case.\n\nThe techniques used to prove the upper bound may be of independent interest to the PAC learning community. The authors use techniques derived from Fathollah Pour and Ashtiani (2022) to study covering numbers of random hypotheses and extend these tools so that their analysis is based on uniform covering number with respect to total variation distance.\n\nSTRENGTHS:\n- The paper shows an interesting contrast between learning noisy vs. clean recurrent neural networks. There is surprisingly an exponential gap in sample complexity w.r.t. $T$ and that noisy networks are easier to learn. This gives way to the suggestion that learning clean networks can \"bypass\" the existing hardness result by assuming there is an infinitesimally small noise $\\sigma$ close to 0 for the clean network, allowing the actual sample complexity upper bound be smaller in case of finite precision machines.\n- The proof techniques are interesting as the paper works with random hypotheses and establishing sample complexity via covering numbers under such randomness.\n\nWEAKNESSES:\n- The motivation to study noisy recurrent neural networks, as it is defined in this paper, is unclear to me. The formulation is similar to a noisy dynamical system but it is unclear why such noise assumption would be natural here for neural networks.\n\nQUESTIONS:\n- On one hand, learning noisy networks being easier than learning clean networks is surprising since it's noisier. Yet, it may be the case that, by adding noise (smaller signal-to-noise ratio), it becomes easier to learn as more hypotheses may seem to plausible (as if blurring an image). Is this why the noise is helping the learner?\n- What is the main reason for studying the ramp loss?\n\nLIMITATIONS:\nThe paper is clear with limitations and comparison to prior/future work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies learning recurrent neural networks with sigmoid activations, where a small amount of noise of magnitude sigma is added to each layer. This paper shows that there is a log(T/sigma) scaling of the sample complexity with the number of recurrent compositions.\n\n This is surprising because, without the noise, there is a linear in T scaling of the sample complexity. Furthermore, the quantity of noise can be very small, since the log(1/sigma) dependence is mild.\n\nSTRENGTHS:\nThe paper is notationally heavy, but is clearly written and was therefore easy to follow. The comparison with past works was good, and I felt that the main result was surprising.\n\nWEAKNESSES:\nFrom a technical point of view:\n* The proof of Theorem 10 (the sample complexity lower bound for learning with the Ramp Loss) is effectively the same as Koiran and Sontag (1998), who proved it for the 0-1 loss. The main observation is that neural networks with sigmoid activations can be forced to output approximately 0-1 valued functions by rescaling the final layer weights. And therefore the ramp loss is approximately equal to the 0-1 loss.\n\n* Most of the steps in the proof of Theorem 26 draw heavily on the work of \"Benefits of Additive Noise in Composing Classes with Bounded Capacity\" by Fathollah Pour and Ashtiani, 2022. The main technical novelty seems to be in Theorem 24, which gives an inequality for the TV covering number under composition. Did I understand correctly that this is the main new element?\n\nQUESTIONS:\n* Is the noise $\\mathcal{G}_{\\sigma}$ in Definition 13 reused, or fresh noise on each layer? \n\n\n* There seem to be some minor technical issues with the proof of Theorem 10 as stated. In the proof of Lemma 11 (restated as Lemma 32), the authors on line 604 define $z$. The authors write the definition of $z$ as\n$$z = \\min_b \\arg\\max_{0 < x < 1/2} P[|Last(b^R(U,T-1))| \\geq x] \\geq 1-\\eta.$$\nThis is nonstandard notation and it is unclear how to parse it. The authors state that \"intuitively, z is the largest possible value such that \"\n$$P[-z < Last(b^R(U,T-1)) < z] < \\eta$$ Considering this and the remainder of the proof, I am assuming that the authors meant\n$$z = \\max \\\\{x \\in (0,1/2) :  P[|Last(b^R(U,T-1))| \\geq x] \\geq 1-\\eta \\mbox{ for all } b\\\\}.$$\nHowever, $z$ may not be well-defined because the maximum might not exist. Consider the network $b$ that sends any input to $Last(b^R(U,T-1)) = 0$ almost surely. Then no $x \\in (0,1/2)$ is valid for the above problem -- we would have to take $x = 0$, which would lead to an issue with the proof since then $z = 0$.\n\n* How does the construction for Theorem 10 break if sigma=1/T^C magnitude noise is added for some large constant C? Your Theorem 26 would predict T*polylog(T) sample complexity, ao the construction in Theorem 10 should break, but I don't see exactly how. Could you please provide some intuition?\n\nTypos:\n* Appendix C, Proof of Theorem 10, have [l]^{\\gamma} and [l]^{0-1}, when it should be [l^{gamma}] and [l^{0-1}]\n* Statement of Theorem 23, display math between lines 311 and 312. Why is there \". N_1\" in the expression? Is it supposed to be times N_1?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors consider a class of noisy recurrent neural networks under the ramp loss setting, and prove that the noisy class can be learned with $O(w \\log (T/\\sigma))$, where $w$ is the width, $T$ is the length of the sequence, and $\\sigma$ is noise variance. \n\nThe derived results demonstrate the sample effiency when compared to the standard noiseless case. \nThe proof is based on the result of covering number of a class of random maps under the TV distance.\n\nSTRENGTHS:\n- the sample complexity $O(w \\log (T/\\sigma))$ is proved for such noise RNN under the ramp loss\n- the covering number under the TV distance is given\n\nWEAKNESSES:\n- The motivation of using ramp loss is unclear to me and quite weak. For example, ramp loss is non-convex and not commonly used in practice. I understand that the ramp loss will be quite close to the 0-1 loss, and thus the derivation would be relatively easier, e.g., Lemma 11. Nevertheless, it decreases the technical difficulty as well as the motivation.\n\n- The used techique is from (Fathollah Pour and Ashtiani 2022) on the covering number of a class of random maps under the TV distance. Their results have already applied to shallow and deeper NNs. The contribution appears not very significant under a somehow RNN with ramp loss. When checking the proof in a high level way, the key part, Lemma 11, aims to build the connection between the ramp loss and the 0-1 loss.\n\n- According to the derived sample complexity, it shows that, under a larger noise level, this noisy class can be learned with fewer samples. It makes sense because noise (gradually become the main component) can be easier to be learned than the data. However, this result somethimes is not enough in my view. Because a larger noise injection would lead to a worse performance. Solely adding noise result in nothing. I think this requires more detailed discussion.\n\nQUESTIONS:\nSee the above\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}