The Probabilistic Fault Tolerance of Neural Networks in the Continuous Limit

1 INTRODUCTION . Understanding the inner working of artificial neural networks ( NNs ) is currently one of the most pressing questions ( 20 ) in learning theory . As of now , neural networks are the backbone of the most successful machine learning solutions ( 37 ; 18 ) . They are deployed in safety-critical tasks in which there is little room for mistakes ( 10 ; 40 ) . Nevertheless , such issues are regularly reported since attention was brought to the NNs vulnerabilities over the past few years ( 37 ; 5 ; 24 ; 8 ) . Fault tolerance as a part of theoretical NNs research . Understanding complex systems requires understanding how they can tolerate failures of their components . This has been a particularly fruitful method in systems biology , where the mapping of the full network of metabolite molecules is a computationally quixotic venture . Instead of fully mapping the network , biologists improved their understanding of biological networks by studying the effect of deleting some of their components , one or a few perturbations at a time ( 7 ; 12 ) . Biological systems in general are found to be fault tolerant ( 28 ) , which is thus an important criterion for biological plausibility of mathematical models . Neuromorphic hardware ( NH ) . Current Machine Learning systems are bottlenecked by the underlying computational power ( 1 ) . One significant improvement over the now prevailing CPU/GPUs is neuromorphic hardware . In this paradigm of computation , each neuron is a physical entity ( 9 ) , and the forward pass is done ( theoretically ) at the speed of light . However , components of such hardware are small and unreliable , leading to small random perturbations of the weights of the model ( 41 ) . Thus , robustness to weight faults is an overlooked concrete Artificial Intelligence ( AI ) safety problem ( 2 ) . Since we ground the assumptions of our model in the properties of NH and of biological networks , our fundamental theoretical results can be directly applied in these computing paradigms . Research on NN fault tolerance . In the 2000s , the fault tolerance of NNs was a major motivation for studying them ( 14 ; 16 ; 4 ) . In the 1990s , the exploration of microscopic failures was fueled by the hopes of developing neuromorphic hardware ( NH ) ( 22 ; 6 ; 34 ) . Taylor expansion was one of the tools used for the study of fault tolerance ( 13 ; 26 ) . Another line of research proposes sufficient conditions for robustness ( 33 ) . However , most of these studies are either empirical or are limited to simple architectures ( 41 ) . In addition , those studies address the worst case ( 5 ) , which is known to be Under review as a conference paper at ICLR 2020 more severe than a random perturbation . Recently , fault tolerance was studied experimentally as well . DeepMind proposes to focus on neuron removal ( 25 ) to understand NNs . NVIDIA ( 21 ) studies error propagation caused by micro-failures in hardware ( 3 ) . In addition , mathematically similar problems are raised in the study of generalization ( 29 ; 30 ) and robustness ( 42 ) . The quest for guarantees . Existing NN approaches do not guarantee fault tolerance : they only provide heuristics and evaluate them experimentally . Theoretical papers , in turn , focus on the worst case and not on errors in a probabilistic sense . It is known that there exists a set of small worstcase perturbations , adversarial examples ( 5 ) , leading to pessimistic bounds not suitable for the average case of random failures , which is the most realistic case for hardware faults . Other branch of theoretical research studies robustness and arrives at error bounds which , unfortunately , scale exponentially with the depth of the network ( 29 ) . We define the goal of this paper to guarantee that the probability of loss exceeding a threshold is lower than a pre-determined small value . This condition is sensible . For example , self-driving cars are deemed to be safe once their probability of a crash is several orders of magnitude less than of human drivers ( 40 ; 15 ; 36 ) . In addition , current fault tolerant architectures use mean as the aggregation of copies of networks to achieve redundancy . This is known to require exponentially more redundancy compared to the median approach and , thus , hardware cost . In order to apply this powerful technique and reduce costs , certain conditions need to be satisfied which we will evaluate for neural networks . Contributions . Our main contribution is a theoretical bound on the error in the output of an NN in the case of random neuron crashes obtained in the continuous limit , where close-by neurons compute similar functions . We show that , while the general problem of fault tolerance is NP-hard , realistic assumptions with regard to neuromorphic hardware , and a probabilistic approach to the problem , allow us to apply a Taylor expansion for the vast majority of the cases , as the weight perturbation is small with high probability . In order for the Taylor expansion to work , we assume that a network is smooth enough , introducing the continuous limit ( 39 ) to prove the properties of NNs : it requires neighboring neurons at each layer to be similar . This makes the moments of the error linear-time computable . To our knowledge , the tightness of the bounds we obtain is a novel result . In turn , the bound allows us to build an algorithm that enhances fault tolerance of neural networks . Our algorithm uses median aggregation which results in only a logarithmic extra cost – a drastic improvement on the initial NP-hardness of the problem . Finally , we show how to apply the bounds to specific architectures and evaluate them experimentally on real-world networks , notably the widely used VGG ( 38 ) . Outline . In Sections 2-4 , we set the formalism , then state our bounds . In Section 5 , we present applications of our bounds on characterizing the fault tolerance of different architectures . In Section 6 we present our algorithm for certifying fault tolerance . In Section 7 , we present our experimental evaluation . Finally , in Section 8 , we discuss the consequences of our findings . Full proofs are available in the supplementary material . Code is provided at the anonymized repo github.com/iclr-2020-fault-tolerance/code . We abbreviate Assumption 1→ A1 , Proposition 1→ P1 , Theorem 1→ T1 , Definition 1→ D1 . 2 DEFINITIONS OF PROBABILISTIC FAULT TOLERANCE . In this section , we define a fully-connected network and fault tolerance formally . Notations . For any two vectors x , y ∈ Rn we use the notation ( x , y ) = ∑n i=1 xiyi for the standard scalar product . Matrix γ-norm for γ = ( 0 , +∞ ] is defined as ‖A‖γ = supx 6=0 ‖Ax‖γ/‖x‖γ . We use the infinity norm ‖x‖∞ = max |xi| and the corresponding operator matrix norm . We call a vector 0 6= x ∈ Rn q-balanced if min |xi| ≥ qmax |xi| . We denote [ n ] = { 1 , 2 , ... , n } . We define the Hessian Hij = ∂2y ( x ) /∂xi∂xj as a matrix of second derivatives . We write layer indices down and element indices up : W ijl . For the input , we write xi ≡ xi . If the layer is fixed , we omit its index . We use the element-wise Hadamard product ( x y ) i = xiyi . Definition 1 . ( Neural network ) A neural network with L layers is a function yL : Rn0 → RnL defined by a tuple ( L , W , B , ϕ ) with a tuple of weight matrices W = ( W1 , ... , WL ) ( or their distributions ) of size Wl : nl×nl−1 , biases B = ( b1 , ... , bL ) ( or their distributions ) of size bl ∈ Rnl by the expression yl = ϕ ( zl ) with pre-activations zl = Wlyl−1 + bl , l ∈ [ L ] , y0 = x and yL = zL . Note that the last layer is linear . We additionally require ϕ to be 1-Lipschitz 1 . We assume that the network was trained 11-Lipschitz ϕ s.t . |ϕ ( x ) − ϕ ( y ) | 6 |x− y| . If ϕ is K-Lipschitz , we rescale the weights to make K = 1 : W ijl →W ij l /K . This is the general case . Indeed , if we rescale ϕ ( x ) → Kϕ ( x ) , then , yl−1 → Ky ′ l−1 , and in the sum z′l = ∑ W ij/K ·Kyl−1 ≡ zl Under review as a conference paper at ICLR 2020 using input-output pairs x , y∗ ∼ X × Y using ERM2 for a loss ω . Loss layer for input x and the true label y∗ ( x ) is defined as yL+1 ( x ) = Ey∗∼Y |xω ( yL ( x ) , y∗ ) ) with ω ∈ [ −1 , 1 ] 3 Definition 2 . ( Weight failure ) Network ( L , W , B , ϕ ) with weight failures U of distribution U ∼ D| ( x , W ) is the network ( L , W + U , B , ϕ ) for U ∼ D| ( x , W ) . We denote a ( random ) output of this network as yW+U ( x ) = ŷL ( x ) with activations ŷl and pre-activations ẑl , as in D1 . Definition 3 . ( Bernoulli neuron failures ) Bernoulli neuron crash distribution is the distribution with i.i.d . ξil ∼ Be ( pl ) , U ij l = −ξil ·W ij l . For each possible crashing neuron i at layer l we define U il = ∑ j |U ij l | and W il = ∑ j |W ij l | , the crashed incoming weights and total incoming weights . We note that we see neuron failure as a sub-type of weight failure . This definition means that neurons crash independently , and they start to output 0 when they do . We use this model because it mimics essential properties of NH ( 41 ) . Components fail relatively independently , as we model faults as random ( 41 ) . In terms of ( 41 ) , we consider stuck-at-0 crashes , and passive fault tolerance in terms of reliability . Definition 4 . ( Output error for a weight distribution ) The error in case of weight failure with distribution D| ( x , W ) is ∆l ( x ) = yW+Ul ( x ) − yWl ( x ) for layers l ∈ [ L+ 1 ] We extend the definition of ε-fault tolerance from ( 23 ) to the probabilistic case : Definition 5 . ( Probabilistic fault tolerance ) A network ( L , W , B , ϕ ) is said to be ( ε , δ ) -fault tolerant over an input distribution ( x , y∗ ) ∼ X × Y and a crash distribution U ∼ D| ( x , W ) if P ( x , y∗ ) ∼X×Y , U∼D| ( x , W ) { ∆L+1 ( x ) ≥ ε } ≤ δ . For such network , we write ( W , B ) ∈ FT ( L , ϕ , p , ε , δ ) . Interpretation . To evaluate the fault tolerance of a network , we compute the first moments of ∆L+1 . Next , we use tail bounds to guarantee ( ε , δ ) -FT . This definition means that with high probability 1− δ additional loss due to faults does not exceed ε . Expectation over the crashes U ∼ D|x can be interpreted in two ways . First , for a large number of neural networks , each having permanent crashes , E∆ is the expectation over all instances of a network implemented in the hardware multiple times . For a single network with intermittent crashes , E∆ is the output of this one network over repetitions . The recent review study ( 41 ) identifies three types of faults : permanent , transient , and intermittent . Our definition 2 thus covers all these cases . Now that we have a definition of fault tolerance , we show in the next section that the task of certifying or even computing it is hard .

Summary: Review: This paper considers the problem of dropping neurons from a neural network.  In the case where this is done randomly, this corresponds to the widely studied dropout algorithm.  If the goal is to become robust to randomly dropped neurons during evaluation, then it seems sufficient to just train with dropout (there is also a gaussian approximation to dropout using the central limit theorem called "fast dropout").