Abstract:
The loss of a few neurons in a brain rarely results in any visible loss of function. However, the insight into what “few” means in this context is unclear. How many random neuron failures will it take to lead to a visible loss of function? In this paper, we address the fundamental question of the impact of the crash of a random subset of neurons on the overall computation of a neural network and the error in the output it produces. We study fault tolerance of neural networks subject to small random neuron/weight crash failures in a probabilistic setting. We give provable guarantees on the robustness of the network to these crashes. Our main contribution is a bound on the error in the output of a network under small random Bernoulli crashes proved by using a Taylor expansion in the continuous limit, where close-by neurons at a layer are similar. The failure mode we adopt in our model is characteristic of neuromorphic hardware, a promising technology to speed up artificial neural networks, as well as of biological networks. We show that our theoretical bounds can be used to compare the fault tolerance of different architectures and to design a regularizer improving the fault tolerance of a given architecture. We design an algorithm achieving fault tolerance using a reasonable number of neurons. In addition to the theoretical proof, we also provide experimental validation of our results and suggest a connection to the generalization capacity problem.

heading: 1 INTRODUCTION
text: Understanding the inner working of artificial neural networks (NNs) is currently one of the most pressing questions (20) in learning theory. As of now, neural networks are the backbone of the most successful machine learning solutions (37; 18). They are deployed in safety-critical tasks in which there is little room for mistakes (10; 40). Nevertheless, such issues are regularly reported since attention was brought to the NNs vulnerabilities over the past few years (37; 5; 24; 8). Fault tolerance as a part of theoretical NNs research. Understanding complex systems requires understanding how they can tolerate failures of their components. This has been a particularly fruitful method in systems biology, where the mapping of the full network of metabolite molecules is a computationally quixotic venture. Instead of fully mapping the network, biologists improved their understanding of biological networks by studying the effect of deleting some of their components, one or a few perturbations at a time (7; 12). Biological systems in general are found to be fault tolerant (28), which is thus an important criterion for biological plausibility of mathematical models. Neuromorphic hardware (NH). Current Machine Learning systems are bottlenecked by the underlying computational power (1). One significant improvement over the now prevailing CPU/GPUs is neuromorphic hardware. In this paradigm of computation, each neuron is a physical entity (9), and the forward pass is done (theoretically) at the speed of light. However, components of such hardware are small and unreliable, leading to small random perturbations of the weights of the model (41). Thus, robustness to weight faults is an overlooked concrete Artificial Intelligence (AI) safety problem (2). Since we ground the assumptions of our model in the properties of NH and of biological networks, our fundamental theoretical results can be directly applied in these computing paradigms. Research on NN fault tolerance. In the 2000s, the fault tolerance of NNs was a major motivation for studying them (14; 16; 4). In the 1990s, the exploration of microscopic failures was fueled by the hopes of developing neuromorphic hardware (NH) (22; 6; 34). Taylor expansion was one of the tools used for the study of fault tolerance (13; 26). Another line of research proposes sufficient conditions for robustness (33). However, most of these studies are either empirical or are limited to simple architectures (41). In addition, those studies address the worst case (5), which is known to be
Under review as a conference paper at ICLR 2020
more severe than a random perturbation. Recently, fault tolerance was studied experimentally as well. DeepMind proposes to focus on neuron removal (25) to understand NNs. NVIDIA (21) studies error propagation caused by micro-failures in hardware (3). In addition, mathematically similar problems are raised in the study of generalization (29; 30) and robustness (42). The quest for guarantees. Existing NN approaches do not guarantee fault tolerance: they only provide heuristics and evaluate them experimentally. Theoretical papers, in turn, focus on the worst case and not on errors in a probabilistic sense. It is known that there exists a set of small worstcase perturbations, adversarial examples (5), leading to pessimistic bounds not suitable for the average case of random failures, which is the most realistic case for hardware faults. Other branch of theoretical research studies robustness and arrives at error bounds which, unfortunately, scale exponentially with the depth of the network (29). We define the goal of this paper to guarantee that the probability of loss exceeding a threshold is lower than a pre-determined small value. This condition is sensible. For example, self-driving cars are deemed to be safe once their probability of a crash is several orders of magnitude less than of human drivers (40; 15; 36). In addition, current fault tolerant architectures use mean as the aggregation of copies of networks to achieve redundancy. This is known to require exponentially more redundancy compared to the median approach and, thus, hardware cost. In order to apply this powerful technique and reduce costs, certain conditions need to be satisfied which we will evaluate for neural networks. Contributions. Our main contribution is a theoretical bound on the error in the output of an NN in the case of random neuron crashes obtained in the continuous limit, where close-by neurons compute similar functions. We show that, while the general problem of fault tolerance is NP-hard, realistic assumptions with regard to neuromorphic hardware, and a probabilistic approach to the problem, allow us to apply a Taylor expansion for the vast majority of the cases, as the weight perturbation is small with high probability. In order for the Taylor expansion to work, we assume that a network is smooth enough, introducing the continuous limit (39) to prove the properties of NNs: it requires neighboring neurons at each layer to be similar. This makes the moments of the error linear-time computable. To our knowledge, the tightness of the bounds we obtain is a novel result. In turn, the bound allows us to build an algorithm that enhances fault tolerance of neural networks. Our algorithm uses median aggregation which results in only a logarithmic extra cost – a drastic improvement on the initial NP-hardness of the problem. Finally, we show how to apply the bounds to specific architectures and evaluate them experimentally on real-world networks, notably the widely used VGG (38). Outline. In Sections 2-4, we set the formalism, then state our bounds. In Section 5, we present applications of our bounds on characterizing the fault tolerance of different architectures. In Section 6 we present our algorithm for certifying fault tolerance. In Section 7, we present our experimental evaluation. Finally, in Section 8, we discuss the consequences of our findings. Full proofs are available in the supplementary material. Code is provided at the anonymized repo github.com/iclr-2020-fault-tolerance/code. We abbreviate Assumption 1→ A1, Proposition 1→ P1, Theorem 1→ T1, Definition 1→ D1.

heading: 2 DEFINITIONS OF PROBABILISTIC FAULT TOLERANCE
text: In this section, we define a fully-connected network and fault tolerance formally. Notations. For any two vectors x, y ∈ Rn we use the notation (x, y) = ∑n i=1 xiyi for the standard scalar product. Matrix γ-norm for γ = (0,+∞] is defined as ‖A‖γ = supx 6=0 ‖Ax‖γ/‖x‖γ . We use the infinity norm ‖x‖∞ = max |xi| and the corresponding operator matrix norm. We call a vector 0 6= x ∈ Rn q-balanced if min |xi| ≥ qmax |xi|. We denote [n] = {1, 2, ..., n}. We define the Hessian Hij = ∂2y(x)/∂xi∂xj as a matrix of second derivatives. We write layer indices down and element indices up: W ijl . For the input, we write xi ≡ xi. If the layer is fixed, we omit its index. We use the element-wise Hadamard product (x y)i = xiyi. Definition 1. (Neural network) A neural network with L layers is a function yL : Rn0 → RnL defined by a tuple (L,W,B, ϕ) with a tuple of weight matrices W = (W1, ...,WL) (or their distributions) of size Wl : nl×nl−1, biases B = (b1, ..., bL) (or their distributions) of size bl ∈ Rnl by the expression yl = ϕ(zl) with pre-activations zl = Wlyl−1 + bl, l ∈ [L], y0 = x and yL = zL. Note that the last layer is linear. We additionally require ϕ to be 1-Lipschitz 1. We assume that the network was trained
11-Lipschitz ϕ s.t. |ϕ(x)− ϕ(y)| 6 |x− y|. If ϕ is K-Lipschitz, we rescale the weights to make K = 1: W ijl →W ij l /K. This is the general case. Indeed, if we rescale ϕ(x)→ Kϕ(x), then, yl−1 → Ky ′ l−1, and in
the sum z′l = ∑ W ij/K ·Kyl−1 ≡ zl
Under review as a conference paper at ICLR 2020
using input-output pairs x, y∗ ∼ X × Y using ERM2 for a loss ω. Loss layer for input x and the true label y∗(x) is defined as yL+1(x) = Ey∗∼Y |xω(yL(x), y∗)) with ω ∈ [−1, 1]3 Definition 2. (Weight failure) Network (L,W,B, ϕ) with weight failures U of distribution U ∼ D|(x,W ) is the network (L,W + U,B, ϕ) for U ∼ D|(x,W ). We denote a (random) output of this network as yW+U (x) = ŷL(x) with activations ŷl and pre-activations ẑl, as in D1. Definition 3. (Bernoulli neuron failures) Bernoulli neuron crash distribution is the distribution with i.i.d. ξil ∼ Be(pl), U ij l = −ξil ·W ij l . For each possible crashing neuron i at layer l we define
U il = ∑ j |U ij l | and W il = ∑ j |W ij l |, the crashed incoming weights and total incoming weights. We note that we see neuron failure as a sub-type of weight failure.
This definition means that neurons crash independently, and they start to output 0 when they do. We use this model because it mimics essential properties of NH (41). Components fail relatively independently, as we model faults as random (41). In terms of (41), we consider stuck-at-0 crashes, and passive fault tolerance in terms of reliability. Definition 4. (Output error for a weight distribution) The error in case of weight failure with distribution D|(x,W ) is ∆l(x) = yW+Ul (x)− yWl (x) for layers l ∈ [L+ 1]
We extend the definition of ε-fault tolerance from (23) to the probabilistic case: Definition 5. (Probabilistic fault tolerance) A network (L,W,B, ϕ) is said to be (ε, δ)-fault tolerant over an input distribution (x, y∗) ∼ X × Y and a crash distribution U ∼ D|(x,W ) if P(x,y∗)∼X×Y, U∼D|(x,W ){∆L+1(x) ≥ ε} ≤ δ. For such network, we write (W,B) ∈ FT(L,ϕ, p, ε, δ).
Interpretation. To evaluate the fault tolerance of a network, we compute the first moments of ∆L+1. Next, we use tail bounds to guarantee (ε, δ)-FT. This definition means that with high probability 1− δ additional loss due to faults does not exceed ε. Expectation over the crashes U ∼ D|x can be interpreted in two ways. First, for a large number of neural networks, each having permanent crashes, E∆ is the expectation over all instances of a network implemented in the hardware multiple times. For a single network with intermittent crashes, E∆ is the output of this one network over repetitions. The recent review study (41) identifies three types of faults: permanent, transient, and intermittent. Our definition 2 thus covers all these cases. Now that we have a definition of fault tolerance, we show in the next section that the task of certifying or even computing it is hard.

heading: 3 THE HARDNESS OF FAULT TOLERANCE
text: In this section, we show why fault tolerance is a hard problem. Not only it is NP-hard in the most general setting but, also, even for small perturbations, the error of the output of can be unacceptable.

heading: 3.1 NP-HARDNESS
text: A precise assessment of an NN’s fault tolerance should ideally diagnose a network by looking at the outcome of every possible failure, i.e. at the Forward Propagated Error (23) resulting from removing every possible subset of neurons. This would lead to an exact assessment, but would be impractical in the face of an exponential explosion of possibilities as by Proposition 1 (proof in the supplementary material). Proposition 1. The task of evaluating E∆k for any k = 1, 2, ... with constant additive or multiplicative error for a neural network with ϕ ∈ C∞, Bernoulli neuron crashes and a constant number of layers is NP-hard.
We provide a theoretical alternative for the practical case of neuromorphic hardware. We overcome NP-hardness in Section 4 by providing an approximation dependent on the network, and not a constant factor one: for weights W we give ∆ and ∆ dependent on W such that ∆(W ) ≤ E∆ ≤ ∆(W ). In addition, we only consider some subclass of all networks.

heading: 3.2 PESSIMISTIC SPECTRAL BOUNDS
text: By Definition 4, the fault tolerance assessment requires to consider a weight perturbation W + U given current weightsW and the loss change yL+1(W+U)−yL+1(W ) caused by it. Mathematically, 2Empirical Risk Minimization – the standard task 1/k ∑m k=1 ω(yL(xk), y ∗ k)→ min
3The loss is bounded for the proof of Algorithm 1’s running time to work
Under review as a conference paper at ICLR 2020
this means calculating a local Lipschitz coefficientK (43) connecting |yL+1(W +U)−yL+1(W )| ≤ K|U |. In the literature, there are known spectral bounds on the Lipschitz coefficient for the case of input perturbations. These bounds use the spectral norm of the matrix ‖ · ‖2 and give a global result, valid for any input. This estimate is loose due to its exponential growth in the number of layers, as ‖W‖2 is rarely < 1. See Proposition 2 for the statement:
Proposition 2 (K using spectral properties). ‖yL(x2)− yL(x1)‖2 6 ‖x2 − x1‖2 · ∏L l=1 ‖Wl‖2
The proof can be found in (29) or in the supplementary material. It is also known that high perturbations under small input changes are attainable. Adversarial examples (5) are small changes to the input resulting in a high change in the output. This bound is equal to the one of (23), which is tight in case if the network has the fewest neurons. In contrast, in Section 4, we derive our bound in the limit n→∞. We have now shown that even evaluating fault tolerance of a given network can be a hard problem. In order to make the analysis practical, we use additional assumptions based on the properties of neuromorphic hardware.

heading: 4 REALISTIC SIMPLIFYING ASSUMPTIONS FOR NEUROMORPHIC HARDWARE
text: In this section, we introduce realistic simplifying assumptions grounded in neuromorphic hardware characteristics. We first show that if faults are not too frequent, the weight perturbation would be small. Inspired by this, we then apply a Taylor expansion to the study of the most probable case. 4
Assumption 1. The probability of failure p = max{pl ∣∣l ∈ [L]} is small: p . 10−4..10−3
This assumption is based on the properties of neuromorphic hardware (35). Next, we then use the internal structure of neural networks.
Assumption 2. The number of neurons at each layer nl is sufficiently big, nl & 102
This assumption comes from the properties of state-of-the-art networks (1). The best and the worst fault tolerance. Consider a 1-layer NN with n = n0 and nL = n1 = 1 at
input xi = 1: y(x) = ∑ xi/n. We must divide 1/n to preserve y(x) as n grows. This is the most robust network, as all neurons are interchangeable. Here E∆ = −p and Var∆ = p/n, variance decays with n. In contrast, the worst case y(x) = x1 has all but one neuron unused. Therefore E∆ = p and Var∆ = p, variance does not decay with n. The next proposition shows that under a mild additional regularity assumption on the network, Assumptions 1 and 2 are sufficient to show that the perturbation of the norm of the weights is small.
Proposition 3. Under A1,2 and if {W il } nl i=1 are q-balanced, for α > p, the norm of the weight perturbation U il at layer l is probabilistically bounded as: δ0 = P{‖U ·l‖1 ≥ α‖W ·l ‖} ≤ exp (−nl · q · dKL(α||pl)) with KL-divergence between numbers a, b ∈ (0, 1), dKL(a, b) = a log a/b+ (1− a) log (1− a)/(1− b) and W il from D3
Inspired by this result, next, we compute the error ∆ given a small weight perturbation U using a Taylor expansion. 5
4The inspiration for splitting the loss calculation into favorable and unfavorable cases comes from (27) 5In order to certify fault tolerance, we need a precise bounds on the remainder of the Taylor approximation. For example, for ReLU functions, Taylor approximation fails. The supplementary material contains another counter-example to the Taylor expansion of an NN. Instead, we give sufficient conditions for which the Taylor approximation indeed holds.
Under review as a conference paper at ICLR 2020
Assumption 3. As the width n increases, networksNNn have a continuous limit (39)NNn → NNc, where NNc is a continuous neural network (19), and n = min{nl}. That network NNc has globally bounded operator derivatives Dk for orders k = 1, 2. We define D12 = max{D1, D2}.6
See Figure 1 for a visualization of A3 and Table 1 for the description of A3. The assumption means that with the increase of n, the network uses the same internal structure which just becomes more fine-grained. The continuous limit holds in the case of explicit duplication, convolutional networks and corresponding explicit regularization. The supplementary material contains a more complete explanation. The derivative bound for order 2 is in contrast to the worse-case spectral bound which would be exponential in depth as in Proposition 2. This is consistent with experimental studies (11) and can be connected to generalization properties via minima sharpness (17). Proposition 4. Under A3, derivatives are equal the operator derivatives of the continuous limit:
∂kyL
∂yi 1 l ...∂y ik l
= 1
nkl δkyL δyl(i1)...δyl(ik) + o(1), nl →∞
For example 7, consider y(x) = 1/n1 ∑n1 i1=1 ϕ (∑n0 i0=1 xi0/n0 ) at xi ≡ 1. Factors 1/n0 and 1/n1
appear because the network must represent the same y∗ as n0, n1 →∞. Then, ∂y/∂xi = ϕ′(1)/n1 and ∂2y/∂xi∂xj = ϕ′′(1)/n21. Theorem 1. For crashes at layer l and output error ∆L at layer L under A1-3 with q = 1/nl and r = p+ q, the mean and variance of the error can be approximated as
E∆L = pl nl∑ i=1 ∂yL ∂ξi ∣∣∣ ξ=0 + Θ±(1)D2r 2, Var∆L = pl nl∑ i=1 ( ∂yL ∂ξi ∣∣∣ ξ=0 )2 + Θ±(1)D12r 3
By Θ±(1) we denote any function taking values in [−1, 1].8
The full proof of the theorem is in the supplementary material. The remainder terms are small as both p and q = 1/nl are small quantities under A1-2. In addition, P4 implies ∂yL/∂ξi ∼ 1/nl and thus, when nl → ∞, E∆ = O(1) remains constant, and Var∆L = O(1/nl). This is the standard rate in case if we estimate the mean of a random variable by averaging over nl independent samples, and our previous example in the beginning of the Section shows that it is the best possible rate. Our result shows sufficient conditions under which neural networks allow for such a simplification.9 In the next sections we use the obtained theoretical evaluation to develop a regularizer increasing fault tolerance, and say which architectures are more fault-tolerant.

heading: 5 PROBABILISTIC GUARANTEES ON FAULT TOLERANCE USING TAIL BOUNDS
text: In this section, we apply the results from the previous sections to obtain a probabilistic guarantee on fault tolerance. We identify which kinds of architectures are more fault-tolerant. 6A necessary condition for Dk to be bounded is to have a reasonable bound on the derivatives of the ground truth function y∗(x). We assume that this function is sufficiently smooth. 7The proposition is illustrated in proof-of-concept experiments with explicit regularization in the supplementary material. There are networks for which the conclusion of P4 would not hold, for example, a network with wij = 1. However, such a network does not approximate the same function as n increases since y(x)→∞, violating A3
8The derivative ∂yL/∂ξi(ξ) ≡ −∂yL(yl − ξ yl)/∂yil · yil is interpreted as if ξi was a real variable. 9However, the dependency Var∆ ∼ 1/nl is only valid if n < p−2 ∼ 108 to guarantee the first-order term to dominate, p/n > r3. In case if this is not true, we can still render the network more robust by aggregating multiple copies with a mean, instead of adding more neurons. Our current guarantees thus work in case if p2 ≤ n−1 ≤ p. In the supplementary material, we show that a more tight remainder, depending only on p/n, hence decreasing with n, is possible. However, it complicates the equation as it requires D3.
Under review as a conference paper at ICLR 2020
Under the assumptions of previous sections, the variance of the error decays as Var∆ ∼ ∑ Clpl/nl
as the error superposition is linear (see supplementary material for a proof), with Cl not dependent on nl. Given a fixed budget of neurons, the most fault-tolerant NN has its layers balanced: one layer with too few neurons becomes a single point of failure. Specifically, an optimal architecture with a fixed sum N = ∑ nl has nl ∼ √ plCl
Given the previous results, certifying (ε, δ)-fault tolerance is trivial via a Chebyshev tail bound (proof in the supplementary material):
Proposition 5. A neural network under assumptions 1-3 is (ε, δ)-fault tolerant for t = ε−E∆L > 0 with δ = t−2Var∆L for E∆ and Var∆ calculated by Theorem 1.
Evaluation of E∆ or Var∆ using Theorem 1 would take the same amount of time as one forward pass. However, the exact assessment would need O(2n) forward passes by Proposition 1. In order to make the networks more fault tolerant, we now want to solve the problem of loss minimization under fault tolerance rather than ERM (as previously formulated in (41)): inf(W,B)∈FT L(w,B) where FT = FT(L,ϕ, p, ε, δ) from Definition 5. Regularizing10 with Equation 1 can be seen as
an approximate solution to the problem above. Indeed, Var∆ ≈ pl ∑ i ( ∂L ∂yil · yil )2
(from T1) is connected to the target probability (P5). Moreover, the network is required to be continuous by A3, which is achieved by making nearby neurons’ weights close using a smoothing regularizing function smooth(W ) ≈ ∫ |W ′t (t, t′)|dtdt′. The µ term for q-balancedness comes from P3 as it is a necessary condition for A3. See the supplementary material for complete details. Here L̂ is the regularized loss, L the original one, and λ, µ, ν, ψ are the parameters:
L̂(W ) = L(W ) + λ nl∑ i=1 ( ∂L ∂yi · yi )2 + µ ( maxiW i l miniW il )2 + ψ · smooth(Wl) + ν‖W‖∞ (1)
We define the terms corresponding to λ, µ, ψ as R1 ≈ Var∆/pl, R2 = q2, R3 = smooth(Wl). If we have achieved δ < 1/3 by P5, we can apply the well-known median trick technique (31), drastically increasing fault tolerance. We only use R repetitions of the network with component-wise median aggregation to obtain (ε, δ · exp(−R))-fault tolerance guarantee. See supplementary material for the calculations. In addition, we show that after training, when Ex∇W yL+1(x) = 0, then ExEξ∆L+1 = 0 +O(r2) (proof in the supplementary material). This result sheds some light on why neural networks are inherently fault-tolerant in a sense that the mean ∆L+1 is 0. Convolutional networks of architecture Conv-Activation-Pool can be seen as a sub-type of fully connected ones, as they just have locally-connected matrices Wl, and therefore our techniques still apply. Using large kernel sizes (see supplementary material for discussion), smooth pooling and activations lead to a better approximation. We developed techniques to assess fault tolerance and to improve it. Now we combine all the results into a single algorithm to certify fault tolerance.

heading: 6 AN ALGORITHM FOR CERTIFYING FAULT TOLERANCE
text: We are now in the position to provide an algorithm (Algorithm 1) allowing to reach the desired (ε, δ)-fault tolerance via training with our regularizer and then physically duplicating the network a logarithmic amount of times in hardware, assuming independent faults. We note that our algorithm works for a single input x but is easily extensible if the expressions in Propositions are replaced with expectations over inputs (see supplementary material). In order to estimate the required number of neurons, we use bounds from T1 and P5 which require n ∼ p/ε2. However, using the median approach allows for a fast exponential decrease in failure probability. Once the threshold of failing with probability 1/3 is reached by P5, it becomes easy to reach any required guarantee. The time complexity (compared to the one of training) of the algorithm is O(D12 + Clpl/ε2) and space complexity is equal to that of one training call. See supplementary material for the proofs of resource requirements and correctness.
10We note that the gradient of Var∆ is linear time-computable since it is a Hessian-vector product. 11More neurons do not solve the problem, as E∆ stays constant with the growth of n by Theorem 1. Intuitively, this is due to the fact that if a mean of a random variable is too high, more repetitions do not make the estimate lower.
Under review as a conference paper at ICLR 2020
Data: Dataset D, input-output point (x, y∗), failure probabilities pl, depth L, activation function ϕ ∈ C∞, target ε and δ′, the error tolerance parameters from the Definition 5, maximal complexity guess C ≈ ∫ |y′l(t)|dt ≈ R guess 3
Result: An architecture with (ε, δ′)-fault-tolerance on x 1 Select initial width N = (n1, ..., nL−1); 2 while true do 3 Train a network to obtain W,B; 4 Compute q from Proposition 3; 5 If q < 10−2, increase regularization parameter µ from Eq. 1, continue; // go to line 3; 6 Compute δ0 from Proposition 3 using q; 7 If δ0 > 1/3, increase n by a constant amount, continue; 8 Compute R3 from Eq. 1; 9 If R3 > C, increase regularization parameter ψ from Eq. 1, continue;
10 Compute E∆ and Var∆ from Theorem 1; 11 If E∆ > ε, output infeasible; // cannot do better than the mean11; 12 Compute δ from Proposition 5; 13 If δ > 1/3, increase n by a constant amount and increase λ in Eq. 1, continue; 14 Compute R = O(log 1δ′ ); 15 Output number of repetitions R, layer widths N , parameters W,B; 16 end
Algorithm 1: Achieving fault tolerance after training. The numbers qmax = 10−2 and δmax = 1/3 are chosen for simplicity of proofs. The asymptotic behavior does not change with different numbers, as long as δmax < 1/2 and the constraints on q mentioned in the supplementary material are met

heading: 7 EXPERIMENTAL EVALUATION
text: In this section, we test the theory developed in previous sections in proof-of-concept experiments. We first show that we can correctly estimate the first moments of the fault tolerance using T1 for small (10-50 neurons) and larger networks (VGG). We test the predictions of our theory such as decay of Var∆, the effect of our regularizer and the guarantee from Algorithm 1. See the supplementary material for the technical details where we validate the assumption of derivative decay (A3) explicitly. Our code is provided at the anonymized repository github.com/iclr-2020-fault-tolerance/code. Increasing training dropout. We train sigmoid networks with N ∼ 100 on MNIST (see ComparisonIncreasingDropoutMNIST.ipynb). We use probabilities of failure at inference and training stages pi = 0.05 at the first layer and 10 values of pt ∈ [0, 1.2pi]. The experiment is repeated 10 times. When estimating the error experimentally, we choose 6 repetitions of the training dataset to ensure that the variance of the estimate is low. The results are in the Table 2. The experiments show that crashing MAE (Mean Absolute Error for the network with crashes at inference) is most dramatically affected by dropout. Specifically, training with pt ∼ pi makes network more robust at inference, which was well-established before. Moreover, the bound from T1 can correctly order which network is trained with bigger dropout parameter with only 4% rank loss, which is the fraction of incorrectly ordered pairs. All other quantities, including norms of the weights, are not able to order networks correctly. See supplementary material for a complete list of metrics in the experiment.
Under review as a conference paper at ICLR 2020
Quantity Train rank loss Test rank loss
Crashing, MAE 5.6% 5.6% Crashing, Accuracy 19.8% 17.7% Correct, MAE 23.3% 22.0% Correct, Accuracy 31.7% 39.9%
(a) Experimental metrics
Quantity Rank Loss
T1 Var∆ 3.6% P2 E∆ 24.8% P2 Var∆ 31.7% T1 E∆ 40.8%
(b) Theoretical bounds

heading: 8 CONCLUSION
text: Fault tolerance is an important overlooked concrete AI safety issue (2). This paper describes a probabilistic fault tolerance framework for NNs that allows to get around the NP-hardness of the problem. Since the crash probability in neuromorphic hardware is low, we can simplify the problem to allow for a polynomial computation time. We use the tail bounds to motivate the assumption that the weight perturbation is small. This allows us to use a Taylor expansion to compute the error. To bound the remainder, we require sufficient smoothness of the network, for which we use the continuous limit: nearby neurons compute similar things. After we transform the expansion into a tail bound to give a bound on the loss of the network. This gives a probabilistic guarantee of fault tolerance. Using the framework, we are able to guarantee sufficient fault tolerance of a neural network given parameters of the crash distribution. We then analyze the obtained expressions to compare fault tolerance between architectures and optimize for fault tolerance of one architecture. We test our findings experimentally on small networks (MNIST) as well as on larger ones (VGG-16, MobileNet). Using our framework, one is able to deploy safer networks into neuromorphic hardware. Mathematically, the problem that we consider is connected to the problem of generalization (29; 27) since the latter also considers the expected loss change under a small random perturbation EW+UL(W + U) − L(W ), except that these papers consider Gaussian noise and we consider Bernoulli noise. Evidence (32), however, shows that sometimes networks that generalize well are not necessarily fault-tolerant. Since the tools we develop for the study of fault tolerance could as well be applied in the context of generalization, they could be used to clarify this matter.
12Variance for P2 is derived in the supplementary material
Under review as a conference paper at ICLR 2020

Review summaries:
Review: This paper considers the problem of dropping neurons from a neural network.  In the case where this is done randomly, this corresponds to the widely studied dropout algorithm.  If the goal is to become robust to randomly dropped neurons during evaluation, then it seems sufficient to just train with dropout (there is also a gaussian approximation to dropout using the central limit theorem called "fast dropout").  

This contribution studies the impact of deletions of random neurons on prediction accuracy of trained architecture, with the application to failure analysis and the specific context of neuromorphic hardware. The manuscript shows that worst-case analysis of failure modes is NP hard and contributes a theoretical analysis of the average case impact of random perturbations with Bernouilli noise on prediction accuracy, as well as a training algorithm based on aggregation. The difficulty of tight bounds comes from the fact that with many layers a neural network can have a very large Lipschitz constant. The average case analysis is based on wide neural networks and an assumption of a form of smoothness in the values of hidden units as the width increases. The improve fitting procedure is done by adding a set of regularizing terms, including regularizing the spectral norm of the layers.