The Probabilistic Fault Tolerance of Neural Networks in the Continuous Limit
Authors: El-Mahdi El-Mhamdi
Rachid Guerraoui
Andrei Kucharavy
Sergei Volodin
Venue:  (2020)
Keywords: Robustness
theory of neural networks
fault tolerance
continuous limit
Taylor expansion
error bound
neuromorphic computing
continuous networks
functional derivative

Abstract: The loss of a few neurons in a brain rarely results in any visible loss of function. However, the insight into what “few” means in this context is unclear. How many random neuron failures will it take to lead to a visible loss of function? In this paper, we address the fundamental question of the impact of the crash of a random subset of neurons on the overall computation of a neural network and the error in the output it produces. We study fault tolerance of neural networks subject to small random neuron/weight crash failures in a probabilistic setting. We give provable guarantees on the robustness of the network to these crashes. Our main contribution is a bound on the error in the output of a network under small random Bernoulli crashes proved by using a Taylor expansion in the continuous limit, where close-by neurons at a layer are similar. The failure mode we adopt in our model is characteristic of neuromorphic hardware, a promising technology to speed up artificial neural networks, as well as of biological networks. We show that our theoretical bounds can be used to compare the fault tolerance of different architectures and to design a regularizer improving the fault tolerance of a given architecture. We design an algorithm achieving fault tolerance using a reasonable number of neurons. In addition to the theoretical proof, we also provide experimental validation of our results and suggest a connection to the generalization capacity problem.

Under review as a conference paper at ICLR 2020
THE PROBABILISTIC FAULT TOLERANCE OF NEURAL NETWORKS IN THE CONTINUOUS LIMIT
Anonymous authors Paper under double-blind review
ABSTRACT
The loss of a few neurons in brain rarely results in any visible loss of function. However, the insight into what "few" means in this context is unclear. How many random neuron failures will it take to lead to visible loss of function? In this paper, we address the fundamental question of the impact of the crash of a random subset of neurons on the overall computation of a neural network and the error in the output it produces. We study fault tolerance of neural networks subject to small random neuron/weight crash failures in probabilistic setting. We give provable guarantees on the robustness of the network to these crashes. Our main contribution is a bound on the error in the output of network under small random Bernoulli crashes proved by using Taylor expansion in the continuous limit, where close-by neurons at a layer are similar. The failure mode we adopt in our model is characteristic of neuromorphic hardware, promising technology to speed up artificial neural networks, as well as of biological networks. We show that our theoretical bounds can be used to compare the fault tolerance of different architectures and design a regularizer improving the fault tolerance of given architecture. We design algorithm achieving fault tolerance using a reasonable number of neurons. In addition to the theoretical proof, we also provide experimental validation of our results and suggest a connection the generalization capacity problem.
INTRODUCTION
Understanding the inner working of artificial neural networks (NNs) is currently one of the most pressing questions (20) in learning theory. As of now, neural networks are the backbone of the most successful machine learning solutions (37 18). They are deployed in cafety-crtical tasks in which there is little room for mistakes (I0 40). Nevertheless, such issues are regularly reported since
attention was brought to the NNs vulnerabilities over the past few years (37 5 24 g).
Fault tolerance as part of theoretical NNs research. Understanding complex systems requires understanding how they can tolerate failures of their components. This has been particularly fruitful method systems biology, where the mapping of the full network of metabolite molecules is a computationally quixotie venture. Instead of fully mapping the network, biologists improved their understanding of biological networks by studying the effect of deleting some of their components, one few perturbations at time 7 (2). Biological systems in general are found to be fault tolerant 28), which is thus an important criterion for biological plausibility of mathematical models.
Neuromorphic hardware (NH). Current Machine Learning systems bottlenecked by the under- lying computational power (T). One significant improvement over the now prevailing CPU/GPUs neuromorphic hardware. In this paradigm of computation, each neuron physical entity ( and the forward pass is done (theoretically) at the speed of light. However, components of such hardware are small and unreliable, leading to small random perturbations of the weights of the model (41) Thus, robustness to weight faults is an overlooked concrete Artificial Intelligence (AI) safety problem 2 Since we ground the assumptions our model the properties of NH and of biological networks, our fundamental theoretical results can be directly applied in these computing paradigms.
Research on NN fault tolerance. the 2000s, the fault tolerance of NNs was a major motivation for studying them (14 16, 4). In the 1990s, the exploration of microscopic failures was fueled by
the hopes of developing neuromorphic hardware (NH) 22 6 34). Taylor expansion was one of the tools used for the study of fault tolerance [3] 26). Another line of research proposes sufficient conditions for robustness (33). However, most of these studies are either empirical or are limited to simple architectures (41). In addition, those studies address the worst case (5). which is known be

---

Under review as a conference paper at ICLR 2020
more severe than random perturbation. Recently, fault tolerance was studied experimentally well. DeepMind proposes to focus on neuron removal (25) to understand NNs. NVIDIA (2I) studies error propagation caused by micro-failures in hardware (3). In addition, mathematically similar problems
are raised in the study of generalization (29, 30 and robustness (42)
The quest for guarantees. Existing NN approaches do not guarantee fault tolerance: they only provide heuristics and evaluate them experimentally. Theoretical papers, in turn, focus on the worst case and not on errors in a probabilistic sense. It is known that there exists set of small worst- case perturbations, adversarial examples (5). leading to pessimistic bounds not suitable for the average case of random failures, which the most realistic case for hardware faults. Other branch of theoretical research studies robustness and arrives at error bounds which, unfortunately, scale exponentially with the depth of the network (29). We define the goal of this paper to guarantee that the probability of loss exceeding threshold is lower than pre-determined small value. This condition sensible. For example, self-driving cars are deemed to be safe once their probability of crash is several orders of magnitude less than of human drivers (40) 15] 33). In addition. current fault tolerant architectures use mean as the aggregation of copies of networks to achieve redundancy. This is known to require exponentially more redundancy compared to the median approach and, thus, hardware cost. In order to apply this powerful technique and reduce costs, certain conditions need to be satisfied which we will evaluate for neural networks.
Contributions. Our main contribution is a theoretical bound on the error in the output of an NN in the case of random neuron crashes obtained in the continuous limit, where close-by neurons compute similar functions. We show that, while the general problem of fault tolerance is NP-hard, realistic assumptions with regard to neuromorphie hardware, and a probabilistic approach to the problem, allow us to apply a Taylor expansion for the vast majority of the cases, as the weight perturbation is small with high probability. In order for the Taylor expansion to work, we assume that a network is smooth enough, introducing the continuous limit (39) to prove the properties of NNs: it requires neighboring neurons at each layer to be similar. This makes the moments of the error linear-time computable. To our knowledge, the tightness of the bounds we obtain is novel result. In turn, the bound allows us to build an algorithm that enhances fault tolerance of neural networks. Our algorithm uses median aggregation which results in only a logarithmic extra cost drastic improvement on the initial -P-hhardness of the problem. Finally, we show how to apply the bounds to specific architectures and evaluate them experimentally on real-world networks, notably the widely used VGG (38).
we set the formalism, then state our bounds. In Section 5 we present applications of our bounds on characterizing the fault tolerance of different architectures. In Sec- tion 6 we present our algorithm for certifying fault tolerance. In Section 7 we present our ex- perimental evaluation. Finally, in Section g we discuss the consequences of our findings. Full proofs are available in the supplementary material. Code is provided at the anonymized repo We abbreviate Assumption - A1,
Outline. In Sections 2
github. cwm/ h////llre wwwo-oofall-oooeeaaaaaaa
Proposition 1 + P1, Theorem 1 > TI, Definition 1 D1.
2 DEFINITIONS OF PROBABILISTIC FAULT TOLERANCE
In this section, we define a fully-connected network and fault tolerance formally. Notations. For any two vectors x, y R² we use the notation (x, y) >m= Ti34 for the standard scalar product. Matrix y-norm for 7 = (0, +oo] is defined as ||||, = sup |Ax /|al>>/|[==|| || We use the infinity norm || loc = max [z,] and the corresponding operator matrix norm. We call a vector
O # Z E R q-balanced if min |2:| 2 max (zz]). We denote [n] {1, ..., n.. We define the Hessian HG = 82 ())) 0xx0 2xr as a matrix of second derivatives. We write layer indices down and
element indices up: W For the input, we write = x². If the layer is fixed, we omit its index. We use the element-wise Hadamard product (æ Q y); = J:Ali.
Definition 1. (Neural network) A neural network with layers is a function yr Rao + T. defined by a tuple (L, W,B, B. sp) with a tuple of weight matrices W = (W1.... WL) ) (or their distributions) of size Wi: ny x T1-11 biases B = (b..... by) (or their distributions) of size b E IT by the expression Vi = =(()) with pre-activations 2 Wiy b, E [T], Yo and yı = 21. Note that the last layer is linear. We additionally require 4p to be 1-Lipschirz M
We assume that the network was trained
$ 'l-ipschitz sp s.t. 11(2)) - +(v)l < |z v/. If is K- K-Lipschitz, we rescale the weights to make K = 1: W. + W/ /K. This the general case. Indeed, if we rescale ppxx) + K Ky((x), then, V1-1 + KyL-1- and in the sum Σ W /K. Ky-1 = 2
2

---

Under review as a conference paper at ICLR 2020
using input-ounn pairs 2, X x Y using ERM for loss Loss layer for input and the true label y (x) is defined as (L+1(z) = 1 ( yL (z), y yr)) with w E [
Definition 2. (Weight failure) Network (L, W. W.B. with weight failures U of distribution U oo DI W) is the nefwork (I., W + U, B) for U (x, W). We denote (random) oulput of this network as y W (2:) (z(z) with activations in and pre-activations 24, as in 131
Definition 3. (Bernoulli neuron faluuraillurss) Bernoulli neuron crash distribution is the distribution with i..d 5i ~ Be(pi), Uj" = Ei W For each possible crashing neuron i at layer we define Uf U E3 | and W/ = E, W the crashed incoming weights and total incoming weights. We note that we see neuron failure as sub-type of weight failure.
This definition means that neurons crash independently, and they start to output when they do. We use this model because it mimics essential properties of NH (41). Components fail relatively independently, as we model faults as random (4I). terms of (4), we consider stuck-at-O crashes, and passive fault tolerance in terms of reliability.
Definition 4. (Oulpul error for a weight distribution) The error in case of weight failure with distribution (x, W) is A((x)= = ui
W+U (x) yt
ui (x) for layers E [I+ 1]
We extend the definition of e-fault tolerance from (33) the probabilistic case:
Definition 5. (Probabilistic fault tolerance) A network (L, W, B, is said to be (E, 6.,88)faultt tol- erant over an input distribution (x. y²)
X x Y and a crash distribution U D (z. W)
if -Xx,)) U- D(a,,, (AL+1 FT(L, 6)..
c)
S 5. For such network, write (W, B)
Interpretation. To evaluate the fault tolerance of a network, we compute the first moments of A LL+1- Next, we use tail bounds guarantee (s, 8)-FT. This definition means that with high probability 1 1- additional loss due to faults does not exceed s. Expectation over the crashes U ~ D|æ can be interpreted in two ways. First, for large number of neural networks, each having permanent crashes, EA is the expectation over all instances of network implemented in the hardware multiple times. For single network with intermittent crashes, FA the output of this one network over repetitions. The recent review study (4I) identifies three types of faults: permanent, transient, and intermittent. Our definition
2 thus covers all these cases.
Now that we have definition of fault tolerance, we show in the next section that the task of certifying or even computing it is hard.
THE HARDNESS OF FAULT TOLERANCE
In this section, show why fault tolerance is hard problem. Not only it is NP-hard in the most general setting but, also, even for small perturbations, the error of the output of can be unacceptable.
3.1 NP-HARDNESS
A precise assessment of an NN's fault tolerance should ideally diagnose a network by looking at the outcome of every possible failure, i.e. at the Forward Propagated Error (23) resulting from removing every possible subset of neurons. This would lead to an exact assessment, but would be impractical in the face of an exponential explosion of possibilities by Proposition I (proof in the supplementary material).
Proposition 1. The task of evaluating & for any k = 1,2, with constant additive or multiplica- tive error for a neural network with yp E " neuron crashes and constant number of layers is NP-hard.
We provide a theoretical alternative for the practical case of neuromorphic hardware. We overcome NP-hardness in Section 4 by providing an approximation dependent on the network, and not constant factor one: for weights W we give A and A dependent on W such that A(W) < EA < (W). In addition, we only consider some subclass of all networks.
3.2 PESSIMISTIC SPECTRAL BOUNDS
By Definition 4 the fault tolerance assessment requires to consider weight perturbation +U given current weights W and the loss change yL+ (W ---U------
-W)) -W+
caused by it. Mathematically,
Empirical Risk Minimization - the standard task 1/k E (x.)) vio - min
"The loss is bounded for the proof of Algorithm I's running time work
3

---

Under review as a conference paper at ICLR 2020
Quantity
Diserete
Continuous
Input
æ: [no] - R
2: [0. 1] R
Weights
Wi: [ni] x [m-1 + R
Wi: [0. 1]² + R
Pre-activations
Ej W yl 0
(t) So Wi(t, t' 11 (t') + bt(t)
Table 1: Correspondence between discrete and continuous quantities. When an regular (discrete)
NN is function mapping vectors to vectors, a continuous NN is an operator mapping functions to functions
this means calculating a local Lipschitz coefficient K 44 connecting 1 (W -1))-++++(W)) I K K|U|. In the literature, there known spectral bounds the Lipschitz coefficient for the case of input perturbations. These bounds use the spectral norm of the matrix || |l2 and give global result, valid for any input. This estimate loose due to its exponential growth in the number of layers, as ||W]|2 222 is rarely < 1. See Proposition 2 for the statement:
Proposition 2 (K using spectral properties), (12) - yu (21 lla < ||2 - 10---1ll 112 IT IWill
The proof can be found in (29) or in the supplementary material. It is also known that high perturbations under small input changes are attainable. Adversarial examples (5) are small changes to the input resulting in a high change in the output. This bound is equal to the one of (23), which is tight in case if the network has the fewest neurons. In contrast, in Section we derive our bound in the limit n + 20.
We have now shown that even evaluating fault tolerance of given network can be hard problem. In order to make the analysis practical, we use additional assumptions based on the properties of neuromorphic hardware.
REALISTIC SIMPLIFYING ASSUMPTIONS FOR NEUROMORPHIC HARDWARE
In this section, we introduce realistic simplifying assumptions grounded in neuromorphic hardware characteristics. We first show that if faults are not too frequent, the weight perturbation would be small. Inspired by this, then apply Taylor expansion to the study of the most probable case. [
Assumption 1. The probability of failure p max(p:)3 E [L]} is small: p S 10-1010--3
This assumption is based on the properties of neuromorphic hardware 35 Next, we then use the internal structure of neural networks.
Assumption 2. The number of neurons at each layer ny is sufficiently big. n4 2 10
This assumption comes from the properties of state-of-the-art networks ①. The best and the worst fault tolerance. Consider I-layer NN with n = nn and nir, = n1 at input = 1: y(x) Σ s//n. We must divide 1 1/n to preserve y(æ) as grows. This is the most robust network, as all neurons are interchangeable. Here EA = -p and Var = p/n, variance decays with In contrast, the worst case y(x) .i has all but one neuron unused. Therefore EA = p and VarA- = p. variance does not decay with n.
The next proposition shows that under mild additional regularity assumption the network, Assumptions [ [nd are sufficient to show that the perturbation of the norm of the weights is small.
Proposition 3. Under 2722 and if (Wi) are q-balanced, for a > the norm of the weight perturbation U. at layer is probabilistically bounded as: óu = P( || 2 allW; |} S exp q (allp))) with KL-divergence between numbers a,b (0, (x. a bloga/b + - a) log (1 -1 a0laaa/( b) and Wi from 133
Inspired by this result, next, we compute the error A given small weight perturbation U uusing Taylor expansion.
'The inspiration for splitting the loss calculation into favorable and unfavorable cases comes from 27 'In order to certify fault tolerance, we need a precise bounds on the remainder of the Taylor approximation. For example, for ReLU functions. Taylor approximation fails. The supplementary material contains another counter-example to the Taylor expansion of an NN. Instead, we give sufficient conditions for which the Taylor approximation indeed holds.
4

---

Under review as a conference paper at ICLR 2020
1.0
Cont. z(t), [ 1]
0.5
Discrete z', [n]
0
(n
10)
1
0
(n 20)
1
(n = 30)
1
Figure 1: Discrete (standard) neural network approximating continuous network
As the width increases, networks N N' have continuous limit (39) N + NN where No a continuous neural network (79), and n min(n)). ). That network N. has globally bounded operator derivatives Dy for orders k 1.2. We define D12 max{D1, D2) }
Assumption 3.
See Figure for visualization of AB and Table forr the description A3 The assumption means that with the increase of n, the network uses the same internal structure which just becomes more fine-grained. The continuous limit holds in the case of explicit duplication, convolutional networks and corresponding explicit regularization. The supplementary material contains more complete explanation.
The derivative bound for order 2 is in contrast the worse-case spectral bound which would be exponential in depth as in Proposition [ This is consistent with experimental studies 111 and can be connected generalization properties via minima sharpness (17).
Proposition 4. Under derivatives are equal the operator derivatives of the contimuous limit:
1
8"yL
+ o(1), 14 00
0yj ...0y! ni "yr(
For example | consider y(x) = 1/n1 Em=1
9 1hi= 2/0 /mo) at = 1. Factors 1/no and 1/m1
appear because the network must represent the same y" as no, 12 Co. Then, 0y/0x; 41(1)/m and uuy 00,,00r,== (1)//n
Theorem 1. For crashes at layer and oulput error At. at layer L under A A7 with 1/mz and r == -=
+ g, the mean and variance of the error can upproximated as
11
OUL DE
dyL ag
2
EAr Pt
O1())2rr" VarAr. - Pt E- +
(1)
By O+ we denote any function taking values in |-1,
The full proof of the theorem is in the supplementary material. The remainder terms are small as both p and thus, when 14 - oo. EA- (1) remains constant, and Var =(ii/))) This the standard rate in case if we estimate the mean of a random variable by averaging over nu independent samples, and our previous example the beginning of the Section shows that it is the best possible rate. Our result shows sufficient conditions under which neural networks allow for such a simplification? ?9) In the next sections we use the obtained theoretical evaluation to develop a regularizer increasing fault tolerance, and say which architectures are more fault-tolerant.
1/n, are small quantities under 2 In addition, implics dyz/ / ~ 1/nz and
5 PROBABILISTIC GUARANTEES ON FAULT TOLERANCE USING TAIL BOUNDS
In this section, we apply the results from the previous sections to obtain a probabilistic guarantee on fault tolerance. We identify which kinds of architectures more fault-tolerant.
A necessary condition for D& to be bounded is to have reasonable bound on the derivatives of the ground truth function y (ar) We assume that this function is sufficiently smooth.
'The proposition is illustrated in praof-of-concepl experiments with explicit regularization in the supplemen- tary material. There are networks for which the conclusion of I would not hold, for example, network with w 1. However, such network does not approximate the same function increases since v(æ) oc. violating AB]
vi) /1y/ yf is interpreted if g was a real variable. "However, the dependency VarA 1/nı only valid < p 10 guarantee the first-order term to dominate, p/n= > y In case if this is not true, we can still render the network more robust by aggregating multiple copies with mean, instead of adding more neurons. Our current guarantees thus work in case if p² < n < p. In the supplementary material, we show that more tight remainder, depending only on p/n, hence decreasing with n, possible. However, complicates the equation requires D3-
"The derivative (g) = -0y (y
5

---

Under review as a conference paper at ICLR 2020
Under the assumptions of previous sections, the variance of the error decays as Var ~ E as the error superposition is linear (see supplementary material for a proof), with Cj not dependent on n/- Given fixed budget of neurons, the most fault-tolerant NN has its layers balanced: one layer with too few neurons becomes a single point of failure. Specifically, an optimal architecture with a fixed sum N = Σ ny has ny vpiCi
Given the previous results, certifying (e, 6)-fuul tolerance is trivial via a Chebyshev tail bound (proof in the supplementary material):
Proposition 5. A neural network under assumptions I 3 is (s, - for E with 8 1 VarAr, for EA and Var calculated by Theorem
EAg
Evaluation of EA or Var using Theorem ! would take the same amount of time as one forward pass. However, the exact assessment would need (22) forward passes by Proposition [
In order to make the networks more fault tolerant, we now want to solve the problem of loss minimiza- tion under fault tolerance rather than ERM (as previously formulated in (4I):: inf (w. [(w, B) where FT = FT P,,,,,
P, E, 5) from Definition Regularizing T0 with Equation can be seen as
an approximate solution to the problem above. Indeed, V pt E,
[aL Oyt
(from T1 is
connected to the target probability (F5). Moreover, the network is required to be continuous by A(] which is achieved by making nearby neurons' weights close using a smoothing regularizing function smooth(W) = [t, t') |dtdt'. The term for q- g-balancedness comes from 1(3 as it is necessary condition for A3 See the supplementary material for complete details. Here É is the regularized loss, L. the original one, and X, M4 U, 1/b are the parameters:
T
ac Oy
max; Wi min; W7
((W) C(W) +
Σ
+
+ (ob-smoooh()) + |||W
(1)
q², =22=4² = (W)). If we We define the terms corresponding to X, u4, 1b as Ri Pe VarA/pr, R2 R2= have achieved 5 < 1/3 by H5] we can apply the well-known median trick technique (3I), drastically increasing fault tolerance. We only use R repetitions of the network with component-wii -wise median aggregation to obtain (e, 8 exp(-R)) --aull tolerance guarantee. See supplementary material for the calculations.
In addition, we show that after training, when E,, Vwyr++ = O, then E, Eg4+1 = 0+ (r²)) (proof in the supplementary material). This result sheds some light on why neural networks are inherently fault-tolerant in a sense that the mean A L++1 is O. Convolutional networks of architecture Conv-Act vat Pon-Pol can be seen sub-type of fully connected ones, as they just have locally-commected matrices Wi, and therefore our techniques still apply. Using large kernel sizes (see supplementary material for diseussion), smooth pooling and activations lead to better approximation.
We developed techniques to assess fault tolerance and to improve it. Now we combine all the results into a single algorithm certify fault tolerance.
6 AN ALGORITHM FOR CERTIFYING FAULT TOLERANCE
We are now in the position to provide an algorithm (Algorithm D allowing to reach the desired (e, () ffult tolerance via training with our regularizer and then physically duplicating the network a logarithmic amount of times in hardware, assuming independent faults. We note that our algorithm works for a single input x but is easily extensible the expressions in Propositions are replaced with expectations over inputs (see supplementary material).
In order to estimate the required number of neurons, we use bounds from aad and which require p/E². However, using the median approach allows for fast exponential decrease in failure probability, Once the threshold of failing with probability 1/3 is reached by FSS it becomes easy to reach any required guarantee. The time complexity (compared to the one of training) of the algorithm is O(122 + Cipi/=?) and space complexity is equal to that of one training call. See supplementary material for the proofs of resource requirements and correctness.
We note that the gradient of Var is linear time-computable since it is Hessian-vector product. Il More neurons do not solve the problem, as EA stays constant with the growth of n by Theorem Intuitively. this is due to the fact that if mean of random variable is too high, more repetitions do not make the estimate lower.

---

Under review as a conference paper at ICLR 2020
Data: Dataset D, input-outpur point (x, "., failure probabilities pt, depth L, activation function E C target and 8', the error tolerance parameters from the Definition S maximal complexity guess C = (t)(t) R Rquess
Result: An architecture with 6,,0)) ( j'a-faulttolerancce on
Select initial width N (n1 0L-1);
while true do
Train a network to obtain W. B:
Compute q from Proposition 3
If < 10-2, increase regularization parameter from Eq. continue; // go to line 3;
Compute óg from Proposition using g:
If óo 1/3, increase by a constant amount, continue;
Compute R3 from Eq. T
If Rs C, increase regularization parameter from Eq. continue;
Compute EA and Var from Theorem [
10
11
IE E, output infeasible: cannot do better than the mear
12
Compute å from Proposition 5
13
I > 1/3, increase n by constant amount and increase X in Eq. continue;
Compute A = Oloo });
14
15
Output number of repetitions R, layer widths N, parameters W, B:
16 end
Algorithm I: Achieving fault tolerance after training. The numbers (max 10-2 and 6max 1 1/3 are chosen for simplicity of proofs. The asymptotic behavior does not change with different numbers, as long as mmax < 1/2 and the constraints on mentioned in the supplementary material are met
VarV slope- 06
-5.0
-7.5
log(n1): 4.0
4.5
5.0
5.5
6.0
6.5
7.0
7.5
Figure 2: The effect of the layer width n,, horizontal axis the variance of the fault tolerance error Var vertical axis
7 EXPERIMENTAL EVALUATION
In this section, we test the theory developed in previous sections proof-of-concept experi- ments. We first show that we correctly estimate the first moments of the fault tolerance using 10 for small (10-50 neurons) and larger networks (VGG). We test the predictions of theory such as decay of Var the effect of our regularizer and the guarantee from Al- gorithm See the supplementary material for the technical details where we validate the as- sumption of derivative decay (A3) explicitly. Our code provided at the anonymized repository tituub.cc wtt.:ccccm/ccccmmm1e11 feneeeeeeeettteeeeeee///eee code
Increasing training dropout. We train sigmoid networks with N 100 on MNIST (see Comparis son www.rrsnnnnnnaaaeem www.IInceeiinnnge wnw.aiinnnnneeennnnnoooooooooooo Doponngt www.IN...SPPP.. pynb). We use probabilities of failure at infer- ence and training stages p; 0.05 the first layer and values of Pt e [0, 1.2p,]. The experiment is repeated 10 times. When estimating the eттоr experimentally. we choose 6 repetitions of the training dataset to ensure that the variance of the estimate is low. The results are the Table The experiments show that crashing MAE (Mean Absolute Error for the network with crashes at inference) most dramatically affected by dropout. Specifically, training with Pt ~ Pt makes network more robust inference, which was well-established before. Moreover, the bound from T[ correctly order which network is trained with bigger dropout parameter with only 4% rank loss, which the fraction of incorrectly ordered pairs. All other quantities, including norms of the weights, not able to order networks correctly. See supplementary material for a complete list of metrics in the experiment.
7

---

Under review as a conference paper at ICLR 2020
Quantity
Train rank loss
Test rank loss
Quantity
Rank Loss
Crashing, MAE
5.6%
5.6 %
VarA
3.6%
Crashing, Accuracy
19.8%
17.7%
EA
24.8%
Correct, MAE
23.3%
22.0%
2 Var
31.7%
Correct, Accuracy 31.7%
39.9%
EA
40.8%
(a) Experimental metrics
(b) Theoretical bounds
Table 2: Comparison of networks trained with increased dropout. Rank loss between Ptrain and various metrics. Experiment shows crashing and correct networks, MAE or accuracy and test/train datasets. Theoretical bounds include P2 2ndd and 11
Regularization for fault tolerance. Previously, bound is demonstrated he able to correctly predict which network is more resilient. We therefore use it as regularization technique suggested by Eq. [ see Regularin zatiin. ipynb. We establish that the resilience of the network regularized with Dropout is similar to that of a network regularized with the bound
Testing the bound on larger networks. We test the bound on VGG16 and on smaller convnet, see ConvNet Test (NIST. ipynb and Test - 6. ipynb and verify that they correctly predict the magnitude of the error
Architecture and fault tolerance. Comparing different architectures a single image with p= = 0.01 (VGG16, VGG19, MobileNet) shows (and ConvNet Test-ft. ipyna) that the bigger the mean width of the layer (approximated by the number of parameters), the better is the fault tolerance, as predicted in Section 5 In addition, training networks on the MNIST dataset (see ipynb) shows decrease in variance with ny as predicted by Theorem ! see Figure 2 the variance decays as 1/ng. We regularize with & = (10 10 2) for derivatives and smoothing respectively (see supplementary material for explanation of coefficients) and À = 0.001.
FaultTolerance ConeeeContai miniiiitt y FC
Testing the algorithm. We test the Algorithm the MNIST dataset for 9- 10-3, 8 10 and obtain / = 20, n 500, X = 10-6, bound experimentally. Our experiment demonstrates the guarantee given by Proposition Jaanoonnnnnnnnn and be seen as an experimental confirmation of the algorithm's 's correctness. See TheAlgori thm. ipynb.
10-10 v = (10-1, 10-2). We evaluate the tail
We hence conclude that our proof-of-concept experiments show an overall validity of our assumptions and of our approach.
CONCLUSION
Fault tolerance an important overlooked concrete AI safety issue (2). This paper describes a probabilistic fault tolerance framework for NNs that allows to get around the NP-hardness of the problem. Since the crash probability in neuromorphic hardware is low, we can simplify the problem to allow for polynomial computation time. We use the tail bounds to motivate the assumption that the weight perturbation is small. This allows us to use Taylor expansion to compute the error. To bound the remainder, continuous limit: nearby neurons compute similar things. After we transform the expansion into a tail bound to give a bound on the loss of the network. This gives probabilistic guarantee of fault tolerance. Using the framework, we are able to guarantee sufficient fault tolerance of a neural network given parameters of the crash distribution. We then analyze the obtained expressions to compare fault tolerance between architectures and optimize for fault tolerance of one architecture. We test our findings experimentally on small networks (MNIST) as well as on larger ones (VGG-16. MobileNet). Using our framework, one is able to deploy safer networks into neuromorphic hardware.
require sufficient smoothness of the network, for which we the
Mathematically. the problem that we consider is connected to the problem of generalization 229 27) since the latter also considers the expected loss change under a small random perturbation Ew+uC(WW + U)= - L(W), except that these papers consider Gaussian noise and we consider Bernouli noise. Evidence (32), however, shows that sometimes networks that generalize well are not necessarily fault-tolerant. Since the tools we develop for the study of fault tolerance could as well be applied in the context of generalization, they could be used to clarify this matter.
Variance for +2) derived in the supplementary material
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
[T] D. Amodei and D. Hernandez. Al and compute. Downloaded from and and-compute, 2018.
[2] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in ai safety. arXiv preprint arXiv: 06065, 2016.
[3] A. P. Arechiga and A, J. Michaels. The robustness modern deep learning architectures against single event upset errors. In 2018 IEEE High Performance extreme Computing Conference (HPEC), pages 1-6. IEEE, 2018.
[4] Y. Bengio and R. De Mori. Use neural networks for the recognition of place of articulation. In Acoustics, Speech, and Signal Processing, 1988. ICASSP-88,, 1988 International Conference on, pages 103-106. IEEE, 1988.
[5] B. Biggio, I. Corona, Maiorea, Nelson, N. Smndić, Laskov, G. Giacinto, and F. Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, pages 387-402. Springer, 2013.
[6] C.-T. Chiu et al. Robustness of feedforward neural networks. In IEEE International Conference on Neural Networks, pages 783-788. IEEE, 1993.
[7] M. Costanzo, B. VanderSluis, E. N. Koch, A. Baryshnikova, C. Pons, G. Tan, W. Wang, M. Usaj, Hanchard, S. D. Lee, et al. A global genetic interaction network maps a wiring diagram of cellular function. Science, +53(6306):aaf1422) 2016.
[8] E. El Mhamdi, R. Guerraoui, and S. Rouault. The hidden vulnerability of distributed learning in Byzantium. In Proceedings of the 35th International Conference Machine Learning, vol- ume 80 of Proceedings of Machine Learning Research, pages 3521-3530, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
[9] S. K. Esser, R. Appuswamy, P. Merolla, J. V. Arthur, and D. S. Modha. Backpropagation for energy-efficient neuromorphic computing. In Advances in Neural Information Processing Systems, pages 1117-1225, 2015.
[10] A. Esteva, Kuprel, A. Novoa, Ko, S. M. Swetter, H. M. Blau, and S. Thrun. Dermatologist-leveel classification of skin cancer with deep neural networks. Nature, 15 2017.
[11] B. Ghorbani, S. Krishnan, and Y. Xiao. An investigation into neural net optimization via hessian 2019.
eigenvalue density. arXiv preprint
[12] J. I. Glass, N. Assad- Garcia, N. Alperovich, S. Yooseph, M. R. Lewis, M. Maruf, C. A. Hutchison, H. O. Smith, and J. C. Venter. Essential genes of a minimal bacterium. Proceedings of the National Academy Sciences, 103(2):425-330, 2006.
[13] N. C. Hammadi and H. Ito. A learning algorithm for fault tolerant feedforward neural networks. IEICE TRANSACTIONS on Information and Systems, 80(1):21-22 1997.
[14] S. Haykin. Neural networks and learning machines, volume 3. Pearson Education Upper Saddle River, 2009.
[15] N. Kalra and S. Paddock. How many miles of driving would it take to demonstrate autonomous vehicle reliability. Driving to Safety, 2016.
[16] P. Kerlirzin and F. Vallet. Robustness in multilayer perceptrons. Neural computation, 5(3):473- 482, 1993.
[17] N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On large- train- ing for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 100.0466, 2016.

---

Under review as a conference paper at ICLR 2020
[18] A. Krizhevsky, L. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, pages 1097-1109 2012.
[19] N. Le Roux and Y. Bengio. Continuous neural networks. In Artificial Intelligence and Statistics, pages 404-411, 2007.
[20] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521 2015.
[21] G. Li, S. K. S. Hari, M. Sullivan, T. Tsai, K. Pattabiraman, Emer, and S. W. Keckler. Under- standing eπrоr propagation in deep learning neural network (dnn) accelerators and applications. In Proceedings of the International Conference for High Performance Computing, Networking. Storage and Analysis, page 8. ACM, 2017.
[22] K. Mehrotra, C. K. Mohan, S. Ranka, and C.-t. Chiu. Fault tolerance of neural networks. Technical report, DTIC Document, 1994.
[23] E. M. E. Mhamdi and R. Guerraoui. When neurons fail. In 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 1028-1037, May 2017.
[24] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard. Universal adversarial perturba- tions. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 1765-1773, 2017. 2017.
[25] A. S. Morcos, D. G. Barrett, N. C. Rabinowitz, and M. Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations, 2018 18.
[26] A. Murray and J. Edwards. Enhanced mlp performance and fault tolerance resulting from synaptic weight noise during training. IEEE Transactions on neural networks, 5(5):792-802, 1994.
[27] V. Nagarajan. Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience. coRR, pages 1-36, 2019.
[28] S. Navlakha and Z. Bar-Joseph. Distributed information processing in biological and computa- tional systems. Communications of the ACM, 58(1):44-102, 2015.
[29] B. Neyshabur, S. Bhojanapalli, D. McAllester, and N. Srebro. Exploring Generalization in Deep Learning. coRR, 2017.
[30] B. Neyshabur, S. Bhojanapalli, and N. Srebro. A pac-bayesian approach to spectrally-normalizze margin bounds for neural networks. arXiv preprint arXiv: 2017.
[31] W. Niemiro and P. Pokarowski. Fixed precision meme estimation by median of products of averages. Journal of Applied Probability, 46(2):309-329, 2009.
[32] D. Phatak. Relationship between fault tolerance, generalization and the vapnik-chervonenkis (vc) dimension of feedforward anns. In Neural Networks, 1999. JJNN'99,, International Joint Conference on, volume 1, pages 705-709. IEEE, 1999.
[33] D. S. Phatak and I. Koren. Complete and partial fault tolerance of feedforward neural nets. IEEE Transactions on Neural Networks, 6(2):46-456, 1995.
[34] V. Piuri. Analysis of fault tolerance in artificial neural networks. Journal of Parallel and Distributed Computing, 61(1):18-48, 2001.
[35] C. D. Schuman, T. E. Potok, R. M. Patton, J. D. Birdwell, M. E. Dean, G. S. Rose, and J. S. Plank. A survey of neuromorphic computing and neural networks in hardware. arXiv preprint arXiv:1705.06663 2017.
[36] W. Schwarting, J. Alonso-Moral and D. Rus. Planning and decision decision-making for autonomous vehicles. Annual Review of Control, Robotics, and Autonomous Systems, 2018.
10

---

Under review as a conference paper at ICLR 2020
[37] D. Silver, A. Huang, J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 42947587)484, 2016.
[38] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1536, 2014.
[39] S. Sonoda and N. Murata. Double continuum limit of deep neural networks. In ICML Workshop Principled Approaches to Deep Learning. 2017.
[40] J. Stilgoe. Machine learning, social learning and the governance of self-driving cars. Social studies science, 48(1):25-56, 2018.
[41] C. Huitzzl and B. Girau. Fault and error tolerance in neural networks: A review. IEEE Access, 5:17322-17341, 2017.
[42] T.W. Weng, P.-Y. Chen, L. M. Nguyen, M. S. Squillante, I. Oseledets, and L. Daniel. Proven: Certifying robustness of neural networks with a probabilistic approach. arXiv preprini arXiv:1812.088 1812.0832, 2018.
[43] D. Zou, R. Balan, and M. Singh. On lipschitz bounds of general convolutional neural networks. arXiv preprint arXiv: 1808.01115 2018.

---

Probabilistic Fault Tolcrance of Neural Networks in the Continuous Limit. Supplementary material
Anonymous authors September 2019
1
Introduction
First. we prove all the propositions (labeled Proposition 1, 2, ...) from the main paper. Their names and sections match the main paper. We also give additional results, they are labelled Additional Proposition 1, 2, ctc. If they are formal statements of the results referred to in the main paper, they are in the samne section as the reference. We abbreviate Assumption 1 + A1, Proposition 1 + Pl, Theorem + T1, Definition + D1.
Less precise statements with possible future research directions fundamental questions required to make the guarantee even more strong are flushed right.
2 Definition of Probabilistic Fault Tolerance
10 Notations. For any vectors 2, E R we use the notation (x, = Σn J,yi for the standard scalar product. Matrix y-norm for 7 (0, +oo] is defined as ||||,, sup ||Ar||//|x=] We use the infinity norm || max [æy and the corresponding operator matrix norm. We call a vector 0 # IR q-balanced if min [æ;] 2 q max [as]- We denote [n] = 2,.,,... n2.. We define the Hessian Hij (yy())/Or'0 Ox as matrix of second derivatives. We write layer indices down and element indices up: W For the input, write x; = xf. If the layer is fixed, we omit its index. We use the element-wise Hadamard product (x ④ y) 2.3/.
Definition 1. (Neural network) A neural network with L layers is function yz 18 - R defined by tuple (L, W. B) with tuple of weight matrices W (Wi,.., Wt.) (or their distributions) of size Wi: mi-1, biases B = (bI....,LL) (or their distributions) of size by E IR by the expression Vi = (((2) with pre- pre-activations 21 = Witt-1 + bi, E [Z], yo = x and yL. = 21- Note that the last layer is linear. We additionally require yp to 21 1-Lipschitz D We assume that the network was trained using input-output pairs x, x,y" 20 for a loss w. Loss layer for impul a and the true label y² (x) is defined as (141(x) ( with
Y using ERM
23 w 1-111
24
Definition 2. (Weight failure) Network (L, W, B, with weight failures U of distribution D (x, W) is the network (I, W + U, B, sp) for U ~ DI(2, W)- We denote a (random) oulput of this nelwork as y WWU (x) = (x(x)) with activations ÿ and pre-activations êt, as in 47
27
Lipiischit s0 (e(x) p(v)] & Jar y/. If yp K- K-Lipschitz, rescale the weights make W w /K. This the general case. Indeed, we rescale s(z) -> Ky((x,, then, Wt-1 Kyf_1= and in the sum z > W /K Ky- = 21 2 Empirical Risk Minimisation the standard task 1/k mm w(y (xx). vg) min The loss is bounded for the proof of Algorithm I's running time to work

---

Definition 3. (Bernoulli neuron Jailures) Bernoulli neuron crash distribulion is the distribution with i.i.d. 5 Be(pi); Up --;-w,, For each possible crashing neuron i at layer define = [Up] and Wi -3 1W72) the croshed incoming weights and total incoming weights. We note that neuron faiture sub-type of weight failure.
Definition 4. (Oulput error for a weight distribution) The error in case of weight foilure with distribution D2 W)
25 is (((x) = u/ W (x)- yt (a) for layers E +1
8
We extend the definition of e-fault tolerance from
18 to the probabilistic case:
Definition 5. (Probabilistic fault tolerance) A network (L, W. (. is said to be (e, (),8)-fault tolerant over an input
distribution (xr ~ X x Y and crash distribution U ~ D) W) if (a."""))xxx U- (w, (A (x) 2 e) S 8.
For such network, we write (W,B) E (T,,,,,,,,E,,)
3 The Hardness of The Fault Tolerance
3.1 NP-Hardness
Proposition 1. The task of evaluating EA for any k = 2, with constant additive mulliplicative ervor for newral network with up E 0 Bernoulli neuron crashes and constant rumber of layers is NP-hard.
Proof. To prove that a problem is NP-hard, it suffices to take another NP-hard problem, and reduce any instance a of that problem to our problem, meaning that solving our problem would solve the original one.
We take the NP-hard Subset. Sum problem. states: given a finite set of integers as E Z, E [M], determine if 90 there exist non-empty subset S _ [M] such that Σ =;=00 #ES
51 We take subset sum instance 2; Z and feed it an input to a neural network with two first. layer neurons with being piecewise-linear function from 0 to at points -5 and e for some fixed (0, 1). Note that in this proof we only compute io at integer points, and therefore it is possible re-define y O such that it has same values in natural points.
Note that inputs to it are integers, therefore the outputs are 1 if and only (iff) the sum is greater than zero. " First neuron has coefficients all is and second all Is with no bias. The next neuron has coefficients 1 and 1 for 51 both inputs, bias -1.5 and threshold activation function, outputting 1 only both inputs are 1. Again, since s8 inputs to this neuron are integers, we can re-define o to be C The final neuron is a (linear) identity function 5 satisfy Definition 1 It takes the previous neuron its only input. Now, see that y(x) if and only if the sum of inputs to a network is 0.
We now feed the entire set S to the network as its input In case if y(z) = (which is easy to check), we have arrived at a solution, as the whole set has zero sum. In the following we consider the case when y(x) = 0. Suppose that there exist an algorithm calculating answer to the question EA* > 2 for any finite-procision 2. (1-p))-1oly(
a Now, the expectation has terms inside EΛ* =
Suppose that one of the terms is non-zero.
#ES
Then y( O s) A 0, which means that the threshold neuron outputs a value > 0 which means that sum of its inputs 6 is greater than 1.5. This can only happen they are both 1, which means that sum for a particular subset. both we solve the subset sum problem using one call an algorithm determining if EAR 0. Indeed, in case if the original problem has solution, the algorithm will output EΛ* < 0 as there are no non-zero terms inside the sum. In case if there is solution, there will be a non-zero term in the sum and thus EA & > 0 (all terms are non non-tvegtiee).
6
67 and < 0. By setting
Now, suppose there exist an amgorithm which always outputs an additive approximation to EA* giving numbers a i and & such that EA & E (p - E, μ =) for some constant E. Take a network from previous example with additional p.. Take subset sum instance d;. Output YES iff 0 & (n - E, + This is correct because if 0 & (μ e. + =), then it must be that EA* and vice versa. Multiplicative approximation has an analogous proof where we scale the outputs even more.
n layer scaling output by

---

We note that computing the distribution of A for one neuron, binary inpul and weights and threshold activation function is known as noise sensitivriyy in theoretical Computer Science. There exists an exact assessment for W w²= however for the case w¹ + 1 the exact distribution unknown G
79
3.2 Pessimistic Spectral Bounds
Additional Proposition 1 (Norm bound or bound b1). For any norm ||-|| the ervor A at the last layer on inpul z and failures in input can be upper-bounded as (for & @ being the crashed input)
| & I ..... |Willlls o xl
Proof. We assume faulty input (crashes at layer 0). By Definition the error at the last layer is 45 = ÚL VL.
83 By definition 1 y/.. = WL 1 + by, and by Definition 3 Ut. WIiirr-1 + by. Thus, ||Ar.]| = | + br.
84 W WLyL-11 br. = (ū.-1 yr--1)l - |l = - Vr-
Next, since Út-1 p(W +b-)) and yL-1 L- bL-1), we use the I-Lipschitzness of (.. In particular, use that for two vectors and y and (zx), (p() applicd clement- wisc, we have [1p(z) ( 3 || --
85
because the absolute difference in each component is less on the left-hand side. We thus get 1 yr. < 1 - WL-191-2 b--1ll= [aaal=|[W--11 WWW-1(0 previous equation, we have || A < [|, - -i yル-22
vL-22)l < |WW--llllr-2 y2-2ll- Plugging it in to the
Now, since the inner layer act in a similar manner, we inductively apply the same argument for underlying layers and get
| [W |W =||
Moreover have failing input, thus à & @ which completes the proof.
Proposition 2 (K using spectral properties 21
(1)( (xr)))l & | x1ll -I [Wll
03
4 Proof. Application of A for |- || ll2 and x2 = o
We see, there bounds considering different norms of matrices || (AF1) using the triangle inequality and absolute values (AH2). They still lead to pessimistic estimates. All these bounds are known to be loose [9] due to ||W|| = supayto W =|l/ |||||| being larger than the input-specific ||>l|//|<|| We will circunvent this issue by considering the average case instead of the worst case.
Additional Corollary 1 (Infinity norm, connecting to norm bounds). For input with C' ||2|100 for failures at the input with O(1) dependent only on layer size, but not on the weights,
| < PCC|IWllb..... |Wll 0(1)++(()))
Here |00 max [as]
201
Proof. First examine the expression (4) from the other paper g and show that it is equivalent to the result we are proving now:
Erf = E & Σ C (L)
II (Nv - fr ft |=++1
1=1
here we have Cj the maximal value at layer 1, K the Lipschitz constant, wm is the maximal over output neurons (rows of W) and mean absolute value over input neurons (columns W) weight, fi is the number of crashed neurons.
103
104
Now we set f1 = pN1 and f, 0 for and moreover we assumee K as in the main paper.
105
As only have neuron, the index is one-dimensional

---

Therefore the bound is rewritlen as:
B/ I CiNi (L)
II New
Now we notice that the quantity New, = || and therefore
N1
B S p
W : ...lIWEllo
Now assume that the network has more layer that the bound from works for faulty input in the original network:
No
E| Bl S pC
|
NL
Here C = max((x;]) = ||0/ Next we prove that result independently using Additional Proposition for || 1looo
A S 1l .... |Willoolle @ x pe-xxi x||o
Now we calculate Eląç @ x0 We write the definition of the expectation with f(p, m. p² p) k p + #() being the probability that a binary string of length n has a particular configuration with h ones, if its entries are ii.d. Bernoulli Be(p). Here S is the set of all possible network crash configurations at layer Each configuration 81 E St describes which neurons are crashed and which are working. We have |S|| = n
Elie & al1oo Σ (p, n, (s|) max ( ox
Now since for |s| the max(s @ x - = max(# - x) = 0, we consider cases |s = 1 and |s| > 1. For |s| > the quantity f(p n. |s|) O(p²) and therefore, in the first order:
O - 2/100 = Σ |ad| + S pNol + O
Next plug that back into the expression for E Allo:
E All No S W .... || || O(p²)
206
Now we note that this expression and the expression from differ only in numerical constant front of the bound: No instead of No, but the bounds behave in the same way with respect the weights.
NL
Additional Proposition 2 (Absolute value bound bound b2). The error on input x can be upper-bounded as:
E p|WLl .... |
For [W] being the matrix of absolute vatues (W]));; = [W |ar means component-wise absolute values of the
110
111
vector.
Proof. This expression involves absolute value the matrices multiplied together as matrices and then multiplied
113
by an absolute value of the column vector. The absolute value of column vector a vector of element-wiss
114
115 absolute values.
We assume faulty input. By Definition 4 the error at the last layer is AL OL VL By definition i WI.yr.11 + br. and by Definition 3 Sr. Wrir. by. Thus for the i'th component of the error,
A = Wi + b4 Wi - b = (0 92-1))= 11==)>)=>i
MiWEWwin ( -v2-1)

---

By the triangle inequality,
A -
10
v = (Weill - -2-11)*
Next we go one level deeper according to Definition 1
= (p - +b- b/-1)1
And then apply the 1-L.pschitznesss property of yp:
- +/ 1 - - 1 1 92-2)
This brings us to the previous case and thus analogously have
| S -WT-11 10r.-2 y/-22
Inductively repeating these steps, we obtain:
| S [WZ].... ..... Will
Now we take the expectation and move it inside the matrix product by lincarity of expectation:
E A E ... [i - x = [ ..... - -
The last expression involves E| - z/. This component-wist wise expectation of a vector and we examine a single component. Since g is Bernoulli random variable,
E| Clags' - xi|
b + - (1-p).0
Plugging it into the last expression for E A proves the proposition.
117
4 Realistic Simplifying Assumptions for Neuromorphic Hardware
4.1 When is The Weight Perturbation Small?
In this section give sufficient conditions for which case the probability of large weight perturbation under crash distribution is small. First, we define the properties of neuromorphic hardware.
Assumption 1. The probability of failure p max(pi e [L]} is small: P.S 10 10-4..10
123
Assumption 2. The number of neurons at each layer nn is sufficiently big, ni 2 10a
125
Additional Assumption 1. Vectors Wi E3 | are q-balanced for q O for each layer 1
A similar assumption on an "even distribution" of weights was made in 20

---

Toy examples. Naturally, we expect the error to decay with an increase in number of neurons n, because of redundancy. We show that this might not be always the case. First, consider a 1-layer NN with = no and 131 RL = n1 = 1 at input z; = 1: y(z) = Σ si/n. This is most robust network, as all ncurons are interchangeable as 132 they are computing the same function each. Essentially, this is an estimale of the mean of &, given n samples. Here - and Var 1/n Σ Varg, = p(1 - p)/n ~ p/n, variance decays with n. In contrast, the 134 worst case y(x) = Φ1 has all but one neuron unused. Therefore EA = x,EE,p and Var x Vare, p(1 p) ~ p.
133 EA - -E,
135 variance does not decay with n.
This proposition gives sufficient conditions under which the weight perturbation is small and it is less and less as n. increases and decreases:
Proposition 3. Under A and AAT for a > p, the norm of the weight perturbation Ui at layer I is probabilistically bounded as: 5u P( P[ a allWi S exp (-14 q. (xx(allpi)) with KI-divergence between numbers a. b E (0, 1), (R, b) = a log a/b + (1 a) log (1 a)/(1 b)
138
141 Proof. See 5 for the proof details as this is a standard technique based on Chernoff bounds for Binomial distribution
142 with p + O. These are a quantified version of the Law of Large Numbers (which states that an average of identical and independent random variables tends to its mean). Specifically, Chernoff bounds show that the tail of distribution is exponentially small: P[X 2 EX +eEX} S exp(-ce")
Specifically, if we consider a case X = Bin(n, p) with + 0, for which q= 1, we have 5 1 for = k//n > p:
k (1))))
P(X 2 k) S exp
-nDKI
In case if we rewrite k = an, this gives us the result. Specifically, if we consider ||U]|| = Σ = k/Wil. Therefore, this probability is P[X 2 k] = PillU] >
148
This shows that the probability that a fraction of Bernoulli successes is more and more concentrated around its mean, np. Therefore, it is less and less probable that the this fraction is 2 a p. Factor q appears because in the analysis of the sum
147
148
Σ W
n W2
min nq
n n2 max
W21
4.2
Taylor Expansion for a Small Weight Perturbation
In this section, we develop a more precise expression for EA and Vara. Previously, we have seen that the perfect fault-tolerant network has VarA = Opp/n). In this section, we give sufficient conditions when complex neural networks behave as the toy example from the previous section as well.
153
We would like to obtain a Taylor expansion of the expectation of random variable EA = Ti + T2 in terms of p and q= L/n with p+ + q where Ti = (p) and T2- 0(1²). For the variance, we want to have VarA = T3 + + with T3- Op//n) and T4- O(7)). Our goal here is to make this expression decay as n + oo as in the toy example. We will show in Theorem D the first-order terms indeed behave as we expect. However, the expansion also contains remainder (terms T2 and T.). In order for the remainder to be small, we need additional assumptions. Tt is easy to come up with an example for which the first term Ti is zero, but the error is still non-zero, illustrating that the remainder is important. Consider one neuron working and the rest. having zero weights. Consider a surmaation of outputs of the neurons, each with a quadratic activation. Then y = (x; - a)?, and VaY = 0 at H; = a. In addition, 161
154
155
150
157
180 EA= = Op)), however, the first term in Taylor expansion is Ti O. The remainder here is T2 = O(p) and not O(r²), 103 no matter how many neurons we have. The problem with this example is discontinuity: one neuron with non-zero weights is not at all like its neighbors. We thus show that discontinuity can lead to lack of fault tolerance. Next, we generalize this sketch and show that some form of continuity is sufficient for the network to be fault-tolerant. First, we reiterate on the toy motivating example from the previous section. Consider 1-layer neural network
n
y(
W

---

We assumne that all neurons and inpuls are used. Specifically, for the q-factor g(x)) = max /n:/ min [aral, we have (xx) = q(w) 1. We are interested in how the network behaves as + oo (the infinite width limit). For the input, 108 we want the magnitude of individual entries to stay constant in this limit. Thus, |2,|| = ((1). Now we look at 10s the function y(œ) that the network computes. Since the number of terms grows, each of them must decay as 1/m:
in ii 1/n. The simplest example has I (L,..., 1) and w = (1/n, 1/n), which results in y(x) = 1 for all n. Now we consider fault tolerance of such network. We take A = Σ wr - (() 10-11111 1)2; for f ~ Be(p) being the indicator [=1
that the i'th input neuron has failed. Therefore, EA -p/n Σ kim; -P and Var -=aumpp p) p/n. We see that the expectation does not change when n grows, but variance decays as 1/n. These are the values 114 that we will try to obtain from real networks. Intuitively, we expeet that the fault tolerance increases when width 17s increases, because there are more neurons. This the case for the simple example above. However, it is not the in8 case if all but one neuron are unused. Then, the probability of failure always p, no matter how many neurons in there are. Thus, the variance does not decrease with n. We thus are interested in utilizing the neurons we have to their maximal capacity to increase the fault tolerance.
1.0
Cont. z(t), [0[0, 1]
0.5
Discrete EE
0
(n
10)
1 0
(n 20)
1
0
(n
30) 1
Figure 1: Discrete (standard) neural network approximating continuous network
Overall, we have the following plan. We give expansion of Var in terms of and q 1/n, and give sufficient conditions for which the remainder terms are small. We use the first term explicitly when regularizing the network. In order for the expansion to work, we formalize the difference between "all neurons doing the same lask" and "all but one are unused". Specifically, we define a class of "good" networks for which fault tolerance is sufficient, via the continuous limit
101
102
Functions, functionals and operators. We call maps from numbers to numbers as functions, maps from functions to numbers as functionals and maps from functions to functions as operators.
We consider a subset of the space of real-valued functions with domain T: F(T) = BPO for T = [o, 1]. This is a space of bounded piccewise-continuous functions f E F, f:T T + such that there is a finite set of points {a;} C T such that ((L..0)1) (T) infiiitlly-iiffernnttbbe We note that F C L which means that her (dt < co. In addition, [1) < cc. Moreover, for two f.g E F, (f g)(() = f(t) g(t) EF.
106
187
188
100
Additoonal Definition 1. (Continuous neural network) A continnous neural network 15 of function class H with L loyers& is an operator y H + H defined by tuple (L, W. B, ()) with tuple of weights functions W = (W1,.., WL) with Wi E H(T2) (or their distributionss, bias functions B = (bi..., by) (or their distributions) with by E F by the expression (t) = (()((t)), 2x() = PET
Wi(t, ()) + bt),, 1 L, 20 = yo = x and (t = (t).
103
We note that a regular neural networks has T. [nu]. Continuous networks were re-introduced in 15 The classes of discrete and continuous networks can be generalized as Deep Function Machines [2] as special case when Lornors of imputs and activations of discrete networks approximating continuous ones increase linearly with n. In 23 depth of network is also considered to be continuous. Continuous networks are also called the integral representation 23 Previous work considers other limits rather than Lı-norm growing lincarly, but here we only consider this case.
144
105
Next, we connect continuous and regular (discrete) networks. To do that, we define how each quantity transforms when n grows.
200
201
Additional Definition 2. (Contimuons-discred. corvespondence with || = (ma)) Consider continuous NN, (L, We, Be, sp) and a discrete one N = (L, Wa, F). We consider a distribution of continuous impuls F 9 x ~ Pa and a distribution of discrete inputs IR" 3 x ~ Pa. These distributions are linked: each element from Pa corresponds exactly one from Pa- We define the approrimation error as
204
7m
A
sup (Ze XP
sup UUIILL Li3iSm
sup
(vf)) (vf)
11

---

We define = min| (,o,,..,,, vr. the minimal width of the discrele nelwork. If a series of discrele nelworks N N,, has An + 0, + oc for some NNe, we say that NN, - NN..
The summary of correspondences between discrete and continuous quantities is shown in Table 1
Quantity
Discrete (L1)
Continuous (AD1))
Relation (AT2)
AB4
Input
x: [no]] + IR
x: [0, 1] + R
Ij RJ I
Norm
| = ES |zو
|| = -J [x())]dt
|
Bias
by: [ny]] - R
by: [o, 1] + R
8f = b
Weights
Wi: [nr] x [m4-1] -
Wi: [0, 1]3 - l
W7
Wi
m111
n4- n
Pre-activations
z E3 w bj Ca Eig -
a1( So Wi(t, (0), (1) (t') bi(l)
2 2
Number of changes
C. S dt(tt)
Co
R Ce
Table 1: Correspondence between discrete and continuous quantities
Additional Proposition 3. Continnous networks with H. = F are universal approrimators
Proof Based on the proof of 15 By the property of discrete networks, they are universal approximators 15 Take a discrete network y with a sufficiently low error, and define a continuous network NN by using a piecewise- 212 constant W and b from y. Then, NN,- = NNa. The weights and biases are bounded and piccewise-contimoons with 21 Ni discontinuities at each layer. In addition, all functions are bounded since the weights are finile.
In the following we will always use H = F, as it is expressive enough (AF3). and it is useful for us. We give a sufficient condition for which An + 0 as n + co.
Additional Proposition 4. If discrete network N Nr is defoned from continuous network NN, in the following way, then NN,- + NN with error A = O(1/n). See (Figure 7
1 - bj bi
x² = x
w
Wi 11-1
Y2o
1-1 In the following we write à = - for an index i = 1.,ni, as the range for each of the indices is known.
Proof. For layer O, the error is 0 by definition of x'. Also, I and its derivatives are bounded. Suppose we 218 have shown that the error for layers 1.1 is I E, and that ye is bounded with globally bounded deriva-
219 tives. Consider (0//)' (if) (+=) |bj - of + E, Wovl [(t, (t')d(t")dtt) I 11/14-1 E, ((,,)) + & J [dtt'' + IS f(t)dt - 1/n IGI for f(t) = (()))) The second term 221 is a Riemann sum remainder and can be upper-bounded as 2n sup [f We note that [[0,1] = 1 and that [((()) [W;( (0)-(()))))))))))))))())) + ( ())())((())) S sup s (ul sup || sup |y'l < co and does not depend on or the imput, as the bound is global. Therefore, the error at layer I is S (s+ 1/n)C < D/n for the initial choice = = 1/n. Thus, the full error is decaying as 1/n: A = (1/n).
:20 e) - (W(,t')vl-1('
IT
Using that result, proximating y. First, by [5), continuous networks are universal approximators. Taking one and creating discrete networks from it (as in AF gives the desired result as sup ||ye - yalloo I AA= = O(1/n) S and ll -yll S E by the
conclude that for any sufficiently smooth operator y, there exist a sequence NN, ap-
x
228
approximation theorem for continuous nets.
Assumption 3. We assume that N NN, - NNa. We assume that there are global reasonable bounds on the derivatives of the function we want to opprorimate
229
230
(tz)
1
sup P (..()))..(((() (i,,..., Ett!
S DR. Dus = max Day Das Da+t...., Do}
n! Here for ig E [n] and 9j = li1e- = jil we define (io...,ūkx) = gil...u! ! a maltinomial coefficient. In the paper, we only need k E (1,2}.
231

---

200 to fit the same function as n incrcases. In addition, the magnitude of pre-activations must stay in the same range The limit means that close-by neurons inside discrete network compute similar functions. This helps fault tolerance because neurons become redundant. There can be many of such sequences NN,, but for the sake of clarity, one might consider that gradient descent always converges to the "same way" to represent a function, regardless of how many ueurons the network has. This can be made more realistic we instead consider the distribution of continuous networks to which the gradient descent converges to. In this limit, the distribution of activations each layer (including inputs and outputs) stays the same as n grows, and only the number of nodes changes. Note that this limit makes neurons ordered: their order is not random anymore. The limit works when ny are sufficiently large. After that threshold, the network stops learning new features and only refines the ones that it has created already.
236
230
237
238
230
241
The derivative bound part of AB can be enforced if there is a very good fit with the ground truth function ((z). Indeed, if f(x)= = (z)), then the derivative of the network depends on the function being approximated. For discrete-contimoous quantities we write X R X, meaning that X - Xel S e for sufficiently large n.
243
204
245
We note that our new assumptions extend the previous ones:
Additional Proposition 5. AB results in A with
(S ʃ (t, +1),, ny 00
q
((, dt
Proof. A simple calculation:
- - /IW(tt))ddtde Elwip-nn - / (fw(tr)aan) 2
Iwwl= = Σ w
di
Dividing these two gives the result.
240
Can we make the imput norm bounded? Sometimes, input data vectors are normalized: ||>||| = 1 [4. In 248 our analysis, it is better to do the opposite: ||~/|| = O(no). First, we want to preserve the magnitude of outputs
200 to prevent vanishing or exploding gradients. We extend that condition to the input layer as well for convenience.
751 Another approach would be to, for example, keep the norm I constant. as grows. In that case, guarantee
the same output magnitude, input weights must be larger. This is just choice of scaling for the convenience of notation. We note that inputs can be still normalized component-wise.
Less weight decay leads to no fault tolcrance. Consider the simplest case with Gaussian weights: y y= w a, Wi o0), Then y N(0, o(0,022|<<13)) In order to avoid vanishing gradients, we set 0f11:112 const. If ||-1|3 (n)) then we must set o² ~ 1/n for the variance of the output to stay constant. This is a well-known initialization technique preserving the variance of pre-activations and thus climinating vanishing or exploding gradicnts. In contrast to this, by A3 we preserve the meam activation at each neuron instead, like for stochastic neural networks
257
258
24. In the Gaussian case, the error A - E grr,w EA = and Var Σ (?E = a'pl|=13 = p const.
789
200
This does not decay with n. A more formal statement can be found in AH6
The NTK limit. In the NTK limit 14. there a decay of the variance Var since both ||>||| = and 0² ~ 1/n, but that happens because variance docreases in any case, not related to fault tolerance, since o² I² 1/n. The "lazy" regime 77] of the NTK limit implies that every hidden neuron is close to its initialization, which is random. Thus, there is continuity in the network: close-by neurons do not compute similar functions. Thus, the NTK limit is incompatible with our continuous limit.
201
282
261
Below we formalize the claim that a network with constant pre-activations variance is not fault-tolerant. This means that an untrained network with standard initialization is not fault tolerant.
287
Additional Proposition 6. The fault tolerance error variance for a network y = w² x with Ew; = and Varly(=)) = const does not decay with n
208
200
Proof Consider y(x)= = Σ w,J; and A Ewf = const. Then, VarA- = Euf
w.zee. Since Ew; = 0, By(x)= = EA- O. In addition, Vary(=)
= parr((=) = p const, not decaying with n
O
Next, we want to harness the benefits of having a continuous limit by AB We want to bound the Taylor remainder, and for that, we bound higher-order derivatives. We bound them via the operator derivative of the continuous limit.
272

---

Operator derivatives. The derivalives for functionals and operators are more cumbersome to handle because of the many arguments that they have (the function the operator is taken at, the function the resulting derivative operator acts on, and the arguments for these functions), so we explicitly define all of them below. We consider operators of the following form, where a E F and [[r] E F are functions:
[[=l(t)= = (ʃ ((
This is one layer from ADD We define the operator derivative of Y|c) point-wisc with the RHS being functional derivalive:
[
(()
(1)
8.
8x
For functionals F[x] we use the standard functional derivative:
8F1z]
F|x + svp
F[a]
[+]
lim 8-0
dx
For our case F[x] F([x]] [x) have
8F(z] 8x
lim 11 (=(F(L (o(()))) - ((W(.Daataa))) -dd(t)))) / W(1, (')())) 5-30
[p]
Next, we consider the functional derivative at a point. This quantity is similar to just one component of the gradient of an ordinary function, with the complete gradient being similar to the functional derivative defined above. We define the derivative at a point (s) via the Euler-Lagrange equation, since we only consider functionals of the form Fx (t) for some fixed t which are then given by an integral expression
F( Y[=](() = ( at'
Lie L(t)))))))
We define the inner part G[æ] = f ( (t) thus, F[z] (G[=)) In this case, since the integral only depends on the function x explicitly, but not on its derivatives, the functional derivative at point is
8F (x)(s)
QL d OL dr
OL dr
d'((())
o' (G[x))
(t,,
ds dr
The definition of functional derivative point 5F(x) (F(x)/6x(s) (" ("component of the gradient") can be reconciled with the definition of the functional derivative (x) ("full gradient") if we consider the Dirac delta-function:
6F 10 8(1' - = =)]==o((G[a) /
8f
W((,)))( - = (L))
dx
5x((s)
We define the operator derivative at point a point-wise manner via the functional derivative at a point:
[a
(1)=
80((s) (t) = 8z(s)
Having that definition, we compute for our case (0Y (f Now we see that the rules for differentiating operators in our case are the same the well-known rules for 577 the derivatives of standard vector-funccions. Indeed, we consider yı(x) = ={E, Wayry) with the inner part
= = ('G
278 (x) = Σ, Wija, giving (x(x) = (gg())), then 0y./0x = ('( This looks exactly like the expression By induction, this correspondence holds for higher derivatives as well. However, this does not 200 imply that these quantities are equal. In fact, we will show that they differ by a factor of 1 1/nf where k is the order
279 for OY [c]](]])a((()
of the derivative.
# We characterize the derivatives of a discrete NN 8 y ...vvi in terms of operator derivatives of a continuous NN 8t vr./ōy (1) (if).. We only assume the continuous limit:
Proposition 4. For sequence NN,, + No with E C the derivatives decay as:
ghy 1
8hy1
st n 8n 6 + o()) n - 0o
Oyl # ...y!
10

---

Intuitively, this means that the more neurons we have, the less is each of them important. First, consider a simple example y = o of w.r.). Here the weight function is w, = 1/n, w(i) = 1, 2, = 1, and A = 0. Then 206 0y/Or; = o'())ux ~ 1 1/n and / = o"mww ~ 1/n2. We note that the expression inside the sigmoid has limit and it's close to the integral by continuity of and o".
207
Proof Now we prove H Consider the first and second derivatives. Note that the operator derivatives only depend on the number of layers and the dataset. It does not depend on any n anymore.
Oy'i" Ors
Σ Wheh- "
By definition of a neural net
2 fd (o / wh Wo(ll,(t-11) (21--1(((t-))) continuous limit
(yr(ir) no Ox(io)
operator ordinary
Crucially, the factor 1/no appears because we do not sum over io, as it fixed, but we have a weight vector Wi 1/no nevertheless. For all other indices, we have weight matrix W ~ 1/n-- well as summation over
200
4-1-
Joy'' 0
Σ Winte- ...Wio
L Σ o"
II s'ys
W W ...W (2
Σ
From previous
in.-1...h
2 (i)(
contimouus+operaaoo ordinary
ni (x(())))((
Here, the factor 1/ng appears because we never sum over io, but the weight matrix W1 ~ 1/no appears twice.
When does the limit hold? Now, we have assumed that network has a continuous limit in A3 However, this might not be the case: the NTK limit [4] is an example of that, as weights there stay close to their random initialization 17, thus, they are extremely dissimilar.
1. Explicit duplication. We copy each neuron multiple times and reduce the outgoing weights. If we set t') to be piecewise-constant, then the approximation error is zero A = 0, and it does not depend on the degree of duplication. This is the obvious case where the network is becoming more fault-tolerant using duplication, and our framework confirms that. The problem with explicit duplication of non-regularized networks is that their fault- is suboptimal. Not all neurons are equally important to duplicate. Thus, it's more efficient to utilize all the neurons in the best way for fault tolerance by duplicating only the important ones.
248
209
500
301
2. Explicit regularization. We make adjacent neurons compute similar functions, thus, allowing for rodun- 304
dancy. We first consider the local "number of changes" metric (Table D. Specifically, for some function
305
=(t) E [0, 1), f' (t) represents how many times the function goes fully from 0 to 1 or vice versa. Our idea is to use that for the weights to quantify their discontinuity:
C = Σ W
W
Σ
((.3) - f(1+1,5)
no
i
Σ
i-1
j }-1 -
i
j
W
Y4n
111
ni
-
(i - -1
71o
ij
i-1
j-
W
m
T' n1

---

The term above, if small, guaranlees that, for each impul neuron, neighboring output neurons will use it in similar ways. The same is applied for W as well: C2=no/n Eij | - -WiJ+1| z f (t, (t, t')]dtt'' which guarantees that, for each output neuron, neighboring input neurons are used similarly by it.
310
In addition to making adjacent ncurons computing similar functions, we add another term C3 by Gaussian smoothing: a Gaussian kernel is convolved with the weights, and the weights are subtracted from the result. The difference shows how much the current. value differs from an aggregated local " "mean" value.
311
312
313
We explicitly enforce the continuous limit by adding a regularization term of smooth(W) := C1+ Here smooth consists of three parts, see FilterPlayground ippybb The first part C2 C2 computes the numerical derivative with 2, 4, 10, 14 points in the array. The second part convolves the input with a Gaussian kernel and subtracts the original, resulting in measure of discontimuity. All metrics are normalized to work with any size, so that scaling the network up does not change the magnitude of C2 significantly. The implementation can be found in continuity-pp py.
314
315
316
We check the derivative decay prediction (F4) which must follow from the continuity assumption A A3 exper- imentally. We run experiments for the MNIST dataset and architecture (784, N. 100, 10) 2 hidden layers with sigmoid activations, unit Lipshitz coefficient, batch size of 1000, and N from 50 to a few thousand. We mcasure || = max; Σ, | z max, Jr. (') which should stay constant (we mcasure the product of these over layers 1, as only the total norm is expected to stay constant, whereas individual layers can change the magnitude of W). We also mcasure the components of the first derivative (D.)) and the components of the Hessian avg:((Hxx)), (1aa)) which we expect to decay with n by H4 We show that without regularization, the weights || increase, see sians do decay but with a smaller rate, insufficient for T to work, see -C We repeat each experiment 10 times and report mean and standard deviation, see Figure 2 In contrast, our proposed regularization results a smooth transition between neurons at each layer. They are grouped by their similarity. This can be visually seen in
191
322
523
304
328
and the derivatives and Hes-
329
329
300
or in Figure 2 Contrary to networks without regularization, first layer weight profiles seem to have meaningful image, and these images are similar for close-by neurons. There is also an increase in continuity of activations for a fixed input. For this regularization technique, we show that the product of the weights stabilizes, and the derivatives decay with proper slopes of -1 and -2. The accuracy, however, drops from 98% to 90%. We note that this does not demonstrate that our approach necessarily leads to a decrease in accuracy, as continuous networks are general by AFB
331
s50
396
337
We test the result on Fashion MNIST as well (Figure 3 notebook names are the same but with Fashion) and report similar behavior. On the Boston Housing datasct, non-regularized networks already have the desired characteristies (Figure
331
339
340
Note: the condition above of C + C2 + C3 being small is, strictly speaking, only a necessary condition for A3] but not a sufficient one. Even if networks are smooth enough, they might not have limit NN,, + NN.. they could implement the function in drastically different ways: for example, the networks NN,, can be all smooth, but approximate different continuous functions. However, this condition is sufficient for the derivatives Da to stay constant (second part of A3), which is the only requirement for To work. Thus, our approach can give a formal guarantee of fault tolerance. An attempt to give a truly sufficient condition for A3 would be to train the bigger network given duplicated weights of smaller network, and penalizing a bigger network from having weights different from their initialization.
Different smoothing techniques. Currently, C2 unused in our implementation, it was sufficient use C C33 to achieve the fault tolerance required in our experiments. Another method to make close-by by neurons similar could be to use some clustering loss in the space of weights accounting for locality, like the Kohonen self-organizing map. One more idea is to regularize the weights with a likelihood of a Gaussian process, enforcing smoothness and a required range of dependencies. Another idea is to use the second derivative, which is connected to the curvature of a curve (a, u(x)) in the 2D space: k == ==y" /1 (2/)))3/2 The interpretation here is that к = 1/R for R being the radius of curvature, geometrical property of a curve showing how much it deviates from being a line.
"A discussion of why Da stay constant is given in the analysis of the correctness for Algorithm 1.
3. Convolutional networks. For images, this limit naturally corresponds to scaling the images I2 with intermediary pixels being just added between the original ones. Images are piecewise-contimmons because
12

---

adjacent pixels are likely to belong to the same object having the same color, and there are only finitely many objects on the image. Convolutional networks, due to their locality, fit our assumption on one condition. We need to have a large enough kernel size, as otherwisc the non-smoothness is high. Specifically, for CNNs, C1 is small, as neighboring neurons have similar receplive fields. In contrast, C2 can be large in case if the kernel size is small: for example, the kernel (-1,1) 1) in the ID case will always result in a high discontimuity; the coefficients are vastly different and require more values in between them to allow for a continuous limit and redundancy.
347
348
340
390
The notebook ConvNetTeat-VGGIs ManYGii--aa ipynb investigates into this. We note that do not present this result in the main paper and make it future research direction instead. In this paper, we give qualitative description of applicability of our techniques to convolutional networks (big kernel size for small C2 with respect to kernel size, smooth activation and pooling to allow for T to work), as it would be out of the scope of single paper to go into details.
352
356
4. Theoretical considerations for the general case.
How does the network behave when the number of neurons increases, and it is trained with gradient descent from scratch? First, there are permutations of neurons, which we ignore. Secondly, there could be many ways to represent the same function. One constraint is that the magnitude of outputs in the output layer is preserved. Intermediate layers need to have a non-vanishing pre-activation values to overcome /onishing//xploddiinn gradients. In addition, input limit might be enforced such that ~ z(i). Now, gradient descent results a discrete network NN, which can be seen as a discretization of some continuous network No. Since NN's derivatives are globally bounded, GD converges to a critical point. Each critical point determines the range of initializations which lead to it, partitioning the whole space into regions. Each fixed point with sufficiently low loss thus corresponds to set of continuous networks passsin through" resulting discrete network. Each of the continuous networks can have different implementations in discrete networks of larger size. We choose a path in networks of different sizes and denote the probability 8, over initializations to choose that particular continuous network. Therefore, on that path, derivatives decay as we want since NN,, - NN.. The problem might arise if a particular continuous limit NN,, has an extremely small probability (over initializations) of gradient descent giving NN; if sn - O, this particular network N N is unlikely appear. We leave the study of this as a future research direction.
Now we use the derivative decay from H to show fault tolerance using a Taylor expansion. We write q 1/m and r p- + q. In the following will use Assumption 3 only by its consoquence Proposition 4 We note that the conclusion of it can hold in other cases well. We just give sufficient conditions for which it holds.
300
Theorem 1. For crashes at layer and output of layer L under assumption the mean and variance of the error be approzimated as
dyL
dyL a
2
EA = PL
+01(1)D2r?, VarA, = pt OE
+ +1)
By O+(1) we denote any function taking values in [-1, 1]. The derivative dyz/DE((E) is interpreted as if E) was a real variable.
-dy - - v))0vl -00))/Ou vi
308
Proof We consider crashes layer 1 as crashes in the input to the rest of the layers of the network. Thus, without loss of generality, we set = 0.
Consider A(E) = ((11-) - {) o x) y()). Then we explicitly compute
a(E)
AA(E)
Oy(+(()) 8 P)((((E) Oz, Tn DE dr,02, xz3
aci
Now, consider A(E) (1)) + (A)(0). 155 (t(E) by the Taylor theorem with Lagrange remainder.
13

---

logiol 30occ 0.56
logil dog) Sippe 1.36
H. disgg
acc
prod
0.96
50203
legp ai
-10
0.34
20000
500
1000
1303
2000
login)
20
25
25
25
0
25
25
25
lug slop--o--000 0.57
-6
H. Hl
loght dugi 20p00000-1110 1.55
H. H_diag
acc
W prod
-8
2000
-10
750
-12
0.89
500
1000
15:00
250
300
1000
1500
login)
login!
lagin!
20
25
25
25 25
25
25
25
25
25
25
25
Figure Non-regularized MNIST (first two rows), regularized MNIST (second two rows). Odd rows show the charts of the decay of first derivatives (D) and second derivatives (average over all i, Hessian's 's components Hyyl and only the diagonal elements [Hq]). Next, validation accuracy is shown as well is the product of the infinity norms of the weights !... -.. WoW Even rows show first-layer weights as images
We assume | S (otherwise rescale W1). We group the terms into distinct cases i and + j:
EA -0) E(s") (t(())) Σ (("A"((E)))"
The second term is |F1] < 2D2/DD no = OppD2/mo) = (1y)2)
The third term is EE' s D2p² /nj Opp² D2).
dy(x)
Therefore, we have an expansion
EA = VeA(0) I (q+ pD2 = --P Σ
Iss
+ +(DDpp+ + D2r2).
Or;
373
The expectation just decays with p. but not with n2.
Now, consider the variance VarA Var(A (O)) + (U(E)) EA2-
374
Vi
V/2
VarV + VarV + 2Coo V1, 22
Consider VarVi {
dv(x) 22. O1p This is the leading term, the rest smaller.
dr,
And the second term VarV < EV, = Σ D Here we consider various cases for indices i,j, k. L, kyki
377
based the partition of 4 0+4 2 1+3. If all indices are different, get O(p²) from Eg. If all the same, get Oppnz²). If we have 2 groups of 2, we get Opp² /n²), if 3+ get O(p? /n²). Thus, VarV, O(D3r4) Consider the final term Cov(V, < VVVV
E E I O(pDi), E|[a S Dzp/n. EV Σ D2/2nnBBgeh All different indices give p² all same indices give p/nă, and 2+1 give p² /n. Thus, Cov Vz)
D2r
382
(y((x)
Finally,
VarA = PV'A(0)= 0(r + = Σ
x7 12+0 = - D2r3) Ox;
394
A better remainder. It. is possible to obtain a remainder of ((p2/n) + O(pn2²) for the variance instead of a generic 3) This makes the remainder decay with n as well. A more fineggrained analysis of the remainder could
396
14

---

Nggio 3000- 0.73
an 1.30
H,
light dagp stoge-11
diag
acc
prodd
50000
40000
30000
20000
0.84
-11
10000
500
1000
1030
login)
Ing(n)
20
0
25
0
25
0
25
0
25
25
25
log slpp- 0:96
H. Hl
loght dugi 20p0- 1.90
H. H_dlag
acc
1000
W prod
800
-10
-10
500
1900
15:00
1020
1500
login)
login!
lagini
20
25
0
25 0
25
25
0
25
25
25
25
25
25
25
Figure Non-regularized Fashion MNIST (first two rows), regularized Fashion MNIST (second rows). De- scription is the same in Fig.
all
togt cla)) skpa- 2,00
diag
55
val loss
W poood
login stppe-1.37
1.0
-2
50
10200
0.5
8800
4.
-2.5
35
6003
-1.0 Ingin
5.5
100
100
200
300
450
Figure 4: Non-regularized Boston Housing dataset
consider the expression for the variance and explicitly compute correlation between ('Ai(+(E)) and (T))(
388 However, we were unsuccessful in doing so. Another approach is take one more term in the expansion: O(19) it will make the previous term with pd go away, leaving only terms p² /nn and p/n? as the difference expressions in the
300 variance will cancel out. Another idea is to use a Taylor series VarA = Σ Tz (existing in principle). Next, we k=
explicitly bound the remainder since we have a global bound on D₁ S D. Finally, if we expand around the mean / Ex: Ex(x)) = (() + (1)(%) Varz) [ we get remainder p/n right away, but need to compute f()) which is the network at modified input (I pz).
5 Probabilistic Guarantees on The Fault Tolerance Using Tail Bounds
Under our assumptions, VarA Ei Cipt The constant C2 comes from Theorem (or its more computationally ni
tractable form in Add. Coroll. D Since we know that the error superposition lincar (Additional Proposition we the individual layer terms, hence Σ,- Proposition motivates the inverse dependency on m4-
306
Median trick. Suppose that we have a random variable X satisfying P(X 2 $} < 1/3. Then we create R independent. copies of X and calculate X (eeeiaamXX)]a Then PP 2 2) < (1/3)8/2 R/2 because order for the median be larger than value e, at least half of its arguments must be larger than e, and all X, are independent. Thus R = (log 1/8) in order to guarantee (1/3) R/2 < 5. This is standard technique.
399
Proposition A neural network under assumptions is (e,5)-fault tolerant for = EA, > 0 with 6p + VarrA, for EA and VarA calculated by Theorem and 86 from Proposilion
403
15

---

Proof We apply the Taylor expansion from Theorem 1
We directly apply the Chebyshev's inequality for X == A:
VarX
P(X 1+
E
f
5.1 Properties of The Fault Tolerance
Proposition 6. Suppose that C network is at stationary point after training: E. v L(x) 0. first order on P, 9, EgE+1 =
Then in the
Proof. Consider the quantity in question, Additional Proposition 9 for it and apply Eg both sides:
EAA4-1 = Ex OT. (ay ( = - Ex
(( aw
ØL W
Now since we know that Eg BW
AL
0, the linear term is 0
Additional Proposition 7. (Linearity of error superposition, for small limil) Consider network (I, W, B, with crashes at each layer with probability Pt, E O. Then in the first order p, the total mean or variance of the error at the last layer is equal to vector sum of errors in crashes were at a single layer
EAP Pa +.... + EAP PL L Var P DL Var A Ph + +... Var P
EA P PL
Proof. Consider each layer i having a probability of failure p. E [0, for i O, In this proof we utilize Assumption 1 We write the definition of the expectation with f(p, n,, p²11 (1-1p) p)"- being the probability that a binary string length n has particular configuration with ones, its entries are i.i.d Bernoulli Be(p). Here Sj is the of all possible network crash configurations layer 1. Fach configuration st St describes which neurons are crashed and which are working. We have |S|| 2m
EA Pa PL
Σ Ep
Σ #LESt.
f(po, 1sol, No) -.. 1o.1,1,,,..... NL) VL)
where
...) E1-1 ...) 41 We utilize the fact that the quantity f(p, n, k)) = pt p)" * = pt + ((ght)) Only those sets of crashing neurons 412 (So, 10.....,,81.. 81) matter the first order, which have one crash in total (in all layers). We denote it as st E St. Therefore
#z = (ppp....(PWW(( x x Eo)
+h1) (1)
+bz) @ &n.
yL = Wep(... (v( I
+b1)
+bg
we write
Σ 11 UL) +...+ Σ f(p,, N,) UL) SgE
s1.88
This is equivalent to a sum individual layer crashes up to the first order on
EA PL
EAPo +... EA?
The proof for the variance is analogous.
417 6 Algorithm for Certifying Fault Tolerance
In case we consider the RHS quantities Theorem I averaggdd over all data examples (x,, y²), then the Algorithum from the main paper would give guarantee for every erample: it will guarantee that Pr,ç A EA + f] small. Indeed, if we know that Ex, Var small, we know that the total variance Var,,e E., Varę A + Var, A (by the law of total variance) is small well. Indeed, the second term Var,EşA bounded by sup, E& which we assume to be small. In case if it is not small, it is unsafe to use the network, as even the expectation of the error is too high.
<18
419

---

Given a small total variance Vara,e we apply, as in Proposition 5 Chebyshev's inequality to the random variable A over the joint probability distribution D/a. This will give Pa 2 EA + t] S &t &. This probability
42 indicates how likely it is that a network with crashes and random inputs will encounter a too high crror.
420 Median aggregation of R copies works for the input distribution as well. Denole "w;(2) is bad" as the event 427 that the loss exceeds E for the i'th copy. We denote [True] = 1 and [False] = 0 (Iverson brackets). Now,
428 ô- Pa.e (M.e(Med(yssta) (x)) bad) = Ea least half of yi are bad]. Since the inner probability can be bounded expectation over Eg results in the quantity discussed before, Eg VarA.
I t (z) exp(-R), taking
We note that it would not be possible to consider yL+1 to be the total loss, as it makes quantities such as 0L/OW5(z,, (z, W) ill-defined, as they depend on z as well.
6.1 Analysis of the Algorithm 1
The algorithm consists of the main loop which is executed until we can guarantee the desired (c, (f,aatt tolerance. It trains networks, and upon obtaining a good enough network with d < 1/3, it repeats the network a logarithmic number of times. We note that the part on 9 and 6n is not strictly required to guarantee fault tolerance. Rather, satisfying these conditions is a natural necessary conditions to satisfy the more strict ones (on R3, EA and 6). These conditions are necessary bocause, by AP5 contimuous limit implics that the q is reasonable.
434
435
438
431
Space requirements. After cach itcration of the main loop of the algorithm, the previous network can be deleted it is no longer used. Therefore, each new iteration does not require any additional space. We only need to store the network itself and the computational graph for Var which depends network's gradients. The total space complexity is then O (Ei=" Th x m--1) which is just the space to store the network's weights.
438
Time and neurons requirements. Each iteration trains the network and then performs computations which involve only 1 forward pass, so the bottleneck is still the training stage which we assume can be done in time ((T). Now let us calculate the number of iterations. New iteration is requested in case if one of the conditions is not satisfied. We assume the loss to be bounded in [-1,1] by D. We analyze each of the possible non-satisfied conditions:
443
464
408
408
1. q < 10-2. In that case the target value of max |Wal/ min [W;] 100 which is many times greater than the loss itself (by assumption E [-), In this case the algorithm increases the regularization parameter μ to make 9 smaller. We assume that we double cach time. So by setting n² = 1 the network will definitely achieve q > 10 Therefore the number of doublings of μ is at most log L/p². Since μm is represented as a float, log 1/120 = (1)
451
2. 6u > 1/3. Since &g = esp(-niddz it will decay exponentially with the decay of ny In order to make 200
50- < 1/3, we need to set nigdxı > 2 which leads to n 2 axı.(a/pi) Since both a and p are small, we take only the first-order term in (kL(alp) = log toga/p + (1 a) log(1 a)/(1 p). The last term here is ~ p a 2 -a. In case if a 2 e log a/p 2 2 and (ap( 2 a. Now, a needs to be sufficiently small in order to guarantee the second-order term in the Taylor expansion a² D12 < :0 Therefore, a² ~ 1/2 and n 200 If the algorithm increases n by a constant amount, it will need to take O(D12) operations
453
3. Rs > C. In this case, we increase v.. Since this directly influences R3 via regularization, doubling se every lime results in a similar ((1) performance as in the analysis for and μ. Since R3 Rs C is the number of changes that the weights make, which is > 1, and the loss is bounded by 1.
480
4. 8 = E VarA R - Cipp > 1/3 (see Section 5 for C dependent on the function approximated by the NN and the continuous limit. Therefore, n ~ O(Cipt/=").
402
The total number of iterations is therefore 11D22+CCmm/ for Ct = ny /arA/pp for some ny (this is now a property of the function being approximated and the continuous limit). We note that the constants 1/3 and 10-2 are chosen for the simplicity of the proof. The asymptotic behavior of the algorithm does not depend on them, as long as they are constant.
408
For e: 1oge=1
Technically, here we silently implied that epp & 1/vDi2 in order to make a 2 cpp, which means that we cannot implement function with too high second derivatives in neuromorphic hardware with constant not matter how many neurons we take. Intuitively, this happens because for such a function, even failure of c²p fraction of neurons, which is a reasonable expectation, is too large to begin with. We assume that are fitting function which is not like that. If we encounter this, we will see that by having too high RA and the algorithm will output infeasible
17

---

Correctness: guarantee of robustness. Now we analyze correctness. We note that the algorithm is not guaranteed to find a good trade-off between accuracy and fault-tolerance. Up to this point, there is no complete 469 theory explaining the generalization behavior of ncural networks or their capacity. Therefore, we cannot give a proof for a sulficient trade-off without discovering first the complete properties of NNs capacily. We only show that the algorithm can achieve fault tolerance.
470
471
First, the condition on EA and Var implies that the first-order terms in the expansion from I are small enough. Now argue that the remainder is small as well.
472
The condition R3 < C' implies that discrete function is smooth enough to apply J [I l')dldi') x C1 < R3 < C 475 as well as (t, (tdd''t ss C2 < R3 < C. This means that the integral is small, which allows to bound the Riemann remainder Ra/ni from the proof of AF4 This implies that there exist a continuous network NN. such 470
477 that the approximation error from 2 is small. Here, the right metric is Rz/ng s C/ni < 1. The mumber of 4m changes C' must be less than a number of neurons at a layer ny.
Now; the remainder depends on the operator derivative bound D12. By the Assumption 3 (part on Dx), they are small. Below we describe why this part holds.
Experimental evidence for D2 being small. First, since we explicitly have a term R₁ for Var 2 (Qy/Oyl) vt ( in the regularizer, the first-order derivatives D2 would be small for each layer by design of our algorithm. 403 Next, socond-order derivatives yyy//ww /ow are found be small in experimental studies of the Hessian I
484 Since derivatives w.r.t. yr can be expressed via the derivatives w.r.t. weights by AHB and the continuous limit holds, D2 is small well.
General considerations for Dr being small. Note, we can always renormalize [x(t) < 1 and ly(() S 1 by rescaling the input and output weights. For the input layer, D12 can be bounded via the properties of the ground truth function y", if y approximates y well. However, is known that it can be false: the network can have a sufficient accuracy on the training dataset, but have much larger output-input derivatives (d[out) /d(in) than the ground truth function. Specifically, in the task of image recognition, we (humans) assume the problem to be quite smooth: picture of cat with missing ear whiskers is still picture of a cat. However, it was shown [3] that modern CNNs use non-robust features. This implies that CNNs are much more sensitive to small changes in input æ, in contrast to the smooth ground truth function y" that we want it to learn, making the bound for the derivative D, large.
For the hidden layers, it is possible that the continuous limit has large derivatives. For example, we can first apply an injective transformation FF with high derivatives, and then apply the inverse transform implemented as another neural network JF1 existing by APEL Then, we have an overall smooth (identity) operator yL which, however, consists of two very non-smooth parts.
We list the following approaches that potentially could resolve this issue theoretically rather than experimentally, First, we can consider the infinite depth limit [23]. This would allow to have regularities throughout the network' layers. Another approach is to study mathematically the ways that an operator can be decomposed into a hierarchical composition of operators. For example, for images, natural decomposition of the image classification operator could first detect edges, then simple shapes, then groups them into elements pertinent to specific classes and then detects the most probable class [6]. At each stage, only robust features are used. Thus, for such decomposition, output-hidden derivatives would be reasonable as well as the output-input ones: indeed, the decision to recognize cat would not change significantly if some internal hidden layer features (ears whiskers) are not present. Interestingly, just enforcing the continuous limit seems to make features more robust in our experiments, see hidden layer weights on Figures 2 and 3 Without regularization, the weights seem noisy, there are a lot of unused neurons, and even the used ones contain noisy patterns. In contrast, continuity-regularized networks seem to have first-layer weights similar to the input images. This could be an interesting research direction to continue.
We note, however, that such a study is not connected the fault tolerance anymore, as it is a fundamental investigation into the properties of the hierarchical functions that want to approximate, and into the properties of the neural networks which we can find by gradient descent.
T'hee idea to construct = FoF-1 with F-1 implementable by neural network is taken from
487
18

---

Relative error, %
Relative error. %
200
Rank loss
200
Rank loss
0.25
0.25
0.00
0.00
b3
b2
b1 ___infforrm
bi 1.0rmorm
bi 2_22rrmm
b1 infnerm
surt wntoor
01 200
sum Zono
1u01 su 1100
b4
b3
b2
b1 inffoom
b1. 1__nnom
b1 2022nomm
b4
b1 infnorm Tuuno 51 (1tto Co
suu 2ooooe
sum 20om0
sum
Relative error, %
Relative error, %
Rank loss
200
200
0.2
0.25
Rank
b3: *i
{
0.00
0
b4
b3
b2
b4
b3
b2
b4
b2
b4
b3
b2
Figure 5: Boston-traince networks. Rank loss (worst value 0.5) of ordering networks using bounds (left 4) and
relative error in predicting the value of A (right 4). Mean of the error EA (top) and standard deviation (bottom).
Variable input on a fixed network (Ist and 3rd columns) and variable network with fixed input (2nd and 4th columns).
7 Experimental Evaluation
Experiments were performed a single machine with 12 cores, 60GB of RAM and 2 NVIDIA GTX 1080 cards run- ning Ubuntu 16.04 LTS and concept: the Boston Housing dataset (https://wwv cs. toronto
We test the proposed bounds on two datasets as a proof of (regression) and the MNIST dataset (ortps wew.ototeeotetteeeeeeeeeeee//eee
.ttl)) In addition, we use We use
403
404 the Fashion MNIST dataset httpsfttii...ttpss.//////aagaaaa
standard pre-trained networks from Keras (VGG, MobileNet,
For the Dropout experiment 0.03 is used threshold after which visible change is happening. We use the unscaled version of dropouts]
In the experiments, we use computationally tractable evaluations of the result given by I1 The "bl bound" is the Spectral bound from F2 The "b2 bound" corresponds to AF The "b3 bound" corresponds to first-order terms from 1D or the Additional Corollary - and the "b4 bound" corresponds to an exact evaluation of single-neuron crashes (taking Onn) forward passos).
500
sou
502
7.1 Boston-trained networks with different initialization: rank loss with experimen- tal error (additional)
503
We compare sigmoid networks with N ~ 50 trained on Boston Housing dataset (see ErrorCanparisonBostonn ipynb), We use different inputs on a single network and single imput on different networks. We compare the bounds using rank loss which is the average number of incorrectly ordered pairs. The motivation that even if the bound does not predict the error exactly, it could be still useful is able tell which network more resilient crashes. The second quantity is the relative error of A prediction, which is harder to obtain. Experimental error computed random subset S' of all possible crashed configurations S with [S] 2 10t is used ground truth, with | big enough to make the expectation and variance results not change from one launch to another. The results are shown in Figure Even on this simple dataset we have only b3 and b4 giving meaningful results both for rank loss and relative error of error. The results show that is always better than b3, as expected. The same is done s14 for random networks.
506
107
508
308
515
8 note that the same as the scaled version up to the first order

---

7.2 Comparison on random networks (additional)
The same setup as for the Different Dropout Comparison experiment in the main paper is done for random networks
with N ~ 20, ReLU activation function, see BrrorComparisonkande ipynb. The results are shown in Figure 6 and they are qualitatively the same as for Boston dataset.
Relative error %
0.5
Relative error. %
Rank loss
Rank loss
0.5
200
b3 (1
0.0
0.0
62
b1 2norm
sump Intnormn
DI infnorm
sum rrnnomm
sum D1 2norm.
i.nomm
sum inorr
b4 (10
b2 a
b1 2nomm
Tronom
sum 1O00
DL 10-10000 m
sum 1rom
b4
b3
62
b1 2norm
b1 ___infformm
bl 11_lnorm
b1 2norm
b1 iinnnorrrm
b1 1norm
bi In
sum T
sum
Relative error. %
Rank loss
Rank loss
0.2
Relative error,
200
100
b2 +0- @
b3 (
b2 HH
0.0
0.0
b4 4
64
b3
b2
b4
b3
b2
b44
b3
Figure 6: Random networks. Rank loss (worst value 0.5) of ordering networks using bounds (left 4) and relative
error in predicting the value of A (right 4). Mean of the error EA (lop) and standard deviation (bottom). Variable input on a fixed network (Ist and 3rd columns) and variable network with fixed input (2nd and 4th columns).
7.3
Additional charts
Table presents the results of Conv:NetTest-ff.ipppe
Figure presents the results from Comperiioolcccccaannnrrooouottss
523
Figure 10 shows results from Regularization.
594
Figure shows how the algorithm's (Main paper, Algorithm TheAlgorithm. (it.gorrthmmmpyyn)) state evolves over itera-
525
tions. Figure shows the resulting distribution of the error A at the output layer for the final network given by
the algorithm.
Target threshold
4000
Experimental error
Frequency
2000
0
0.02
-0.01
0.00
0.01
A
Figure 7: The distribution A of the error in the output for the network obtained using Algorithm 1 from the main paper
20

---

0.3
4
N1, 1T
Experiment
0.0004
Theory
Empirical 6
0.2
Vara
0.00022
0.1
0.0
0.0000
-1 0 2 3 5 6 8
Algo iteration and stage (color)
Figure 8: Evolution of the experimental error probability and the variance the error over algorithm's iterations. There are two stages. In the first stage (red, iterations I 4) the algorithm increases continuity the network via increasing sb. In the sccond stage (green, itcrations 2 5) the algorithm increascs the number of neurons nı and the regularization parameter A. First, it be seen that at first stages the nelwork is not continuous enough, the algorithm makes it more continuous. This leads to increase in the empirical probability 8 of the network outputting a loss > and in the increase in the gap between Thcoretical Var and Experimental Var A. This happens because there are not enough neurons in the network. Later, as the number of neurons increases, the gap becomes smaller and the empirical probability decreases. Note that the first network (at iteration 0) empirically satisfies our fault tolerance guarantee. Nevertheless, we do not have a proof for such a network because is not continuous enough. Therefore, in order guarantee robustness, we need to proceed with the iterations.
Model
log VarA
log Parameters Layers]
log[Parameters)
Layers
VGG16
-18.7
15.6
18.7
23
VGG19
-18.4
15.5
18.8
26
MobileNet
-9.1
10.7
15.3
93
Table 2: Comparison of bigger convolutional networks when there are faults at every layer with 10
7.4 Error superposition (testing AP7 additional)
:20
590
We random network with computed subsets of failing layers a, b. Then all pairs of disjoint subsets considered, and the relative error of A estimation using linearity is computed as || A. A The results (see Figure 11 show that this relative error is only few percent both for mean and variance, with better results for the mean.
on random input (see ErrorAddittivityyRanoo ipynb). The error is
7.5 Testing A AP9 (additional)
AL to decay with the decay of the gradient the loss. We tested The result predicts the expected error E, By that experimentally on the Boston dataset using sigmoid networks (see ErrorOnTraining ipynb) and note that a similar result holds for ReLu The results are shown in Figure 12 The chart shows first that the experiment corresponds to b4, and b3 is close to b4. b3 is also equal to the result of AF both of which decay, as predicted.
s96
542
7.6
Practical bounds
541
9ReLUg C1. However only small percentage of neurons are near thus approximation works for most of them.
21

---

0.97
0.002 0.004
Accuracy
0.96
Eiror A
0.95
Train
Test
0.94
0.00
0.0S
Prrain
Figure 9: Comparison of networks trained with different dropout. Shown: accuracy and Var plots for different train dropout probability. Green curve shows accuracy of correct network, red shows accuracy for the crashing network. Dashed line show train dataset and solid represent test dataset. Variance of estimated by b4 shown orange, by b3 blue. Error bars are standard deviations in 10 repetitions of the experiment.
Accuracy
0.96-
correct acc acc_test
0.0025
Emro
crashing acc test
0.94
0.0000
10 Regularization
100
102
Figure 10: Horizontal axis represents increasing regularization parameter. Left vertical axis represents accuracy and has test crashing network accuracy (red) and correct network accuracy (green). Right axis shows variance of the error Var estimated by bound b3 used regularizer. Error bars are standard deviations in repetitions of the experiment.
Additional Proposition 8. For neural network have for any weight matrix Wi and input to the I I'th layer W1-1
dy w = Σ awl
dy vi-
Σ
0y
The equality also holds for particular 8Í-1:
Dy
W 8y
Σ
OW :vi 1n-1
Proof. Fix some layer I. The output of the network y depends on the weight matrix Wt and on the input to the 544 l'th layer yt-1- However we note that it only depends on their product and not on these quantities separately, Therefore we write y = (W,,, vt.), = (WW(r()) and denote x = yr-1, Wi and Wx:
= (Wx)
Now we take one
0y
On 0z,
Or, Fzg dz,
Frq /25
mean
variance
2
i Aa Aol. %
Figure 11: Distribution of relative enror in the predicted A in percent for 25 subset pairs. Distribution for the mean is shown in blue, for the variance in orange.

---

0.02
b3/Th8
b4
0.01
10
Experiment
50 100
150
200
250
300
Epochs
Figure 12: Decay with training of the loss (blue line, left vertical axis) and the error Eg By Eg (red, right axis) together with bounds b3 and b4 predictions (red and brown curves) and experimental values (dots). "Th8" mcans A
dz Since 2, 3 Wa TR: dz,
Wi.. Therefore we plug that in:
Oy
En Wij Ōz,
E
dx,
And multiply with zy:
Oy dxj
Ōn Wiy
Σ
azs
Now we compute aw,, using that in vector 2 only depends on Wiy and also that 2 Wx:
Oy On 0z: On
Oz. Ijj
aw, Dz, aW,)
Then we multiply it by Wij and sum over i:
Oy Wij
On
Σ
Σ
Oz. I. Wij
aw
By And now we note that the expressions for dz, Ty and Σ Differentiating the above expression gives a connection between second derivatives w.r.t weights and activations.
By Wi are exactly the same.
aw,
548
341
Additional Corollary 2. The error expressed weights in the first order is as in T
Oy aw
E = - Σ
w
dy
Oy w wh
Vara = ΣΣ
j.k Oh oW
Where A is the ervor in case of neurons failing al layer I. Layer means failing inpul. By
549
In other words, both mean and variance are defined by tensor X1 W1 O oW, zi
E X²,
EA = =p> 2). VarA = >
Proof. According to Theorem the expression for mean and variance are:
Oy Oz,
EA ~ (VVV)(2). = Σ
24
Oy It Ox:
VarA ~ ((V)(z)) = Σ
23

---

Then by Additional Proposition
the expression used can be rewritten as (note the index swap i 4+ f):
dy
By oW jz
34
Σ
W
Oz;
Then for the mean it is:
By Waa OW jz
EA - -P
ΣΣ
and the indices can be swapped again for the sake of notation. For the variance we write the square as the inner sum repeated twice with different indices and j:
dy Var P EE
Wj dy Wki aWk:
This gives the first statements. Then we notice that both expressions depend on the values of tensor aw W dy but not on W or aw alone. We therefore define X = W @ Oy and 2 = rewrite
Oy
= - Xij (sum over the first index) and
oW
EA - Σ E
X = = Σ
VarA =
X = EX,xX
Σ
Σ Xji = Σ - 21)²
ijk
Additional Proposition For neural network with C activation function have for porticular input x in the first order as in 7
153
Proof. Take the expression A 1E
AL
AE and consider E&A. By Additional Corollary have By
554
EA - J.. Wij
Therefore,
OL Oy
Oy
[LL By Ty
A= -p
Wij
=
> 25
Wij
ow, i3
By the chain rule for a single input we have
0L
Oy
aW,;
Oy
aW;
We plug that back in:
=
Σ
0L
W
ow,
This is the exact statement from the proposition.
8
Unused extra results
558

---

Skip-connections. If we consider a model with skip-connections (which do not fit Definition 1 with faults at every node, we expect that an assumption similar to AB would lead to a result similar to However, we did not test our thcory on models with skip-connections.
Another idea for fault tolerance. In case if we know p exactly, we can compensate for EA by multiplying each neuron's output by (1-p) p) In this way, the mean input would be preserved, and the network output will be unbiased. However, this only works in case if we know exactly of the hardware.
A Additional Proposition 10 (Variance for bound bl, not useful since the bound is not light. Done in similar manner as the mean bound in 9) The variance of the error VarA is upper- upper-bounded bounded as
12
Var <
Σ ai II
bj
j=i+1
for
al Orplla + +CCppLL
bg = (1 - PL) + (1 (1))))
C max(lyus1
al max((w w (1(1))))3
Br. max max(|1oo(4+2)11 |1|110000 |1000 (1ttt))))
(w(L+1), 10 (L+1)))
N₂ (L+1)
(1) (2) (L) - -)00) )I EAL & (1 C1
Proof. In the notation of 9 consider
|Az] |Fneu= Fiall
m
K| (|] P1) where (1) is the corrupted output from layer T. and K is the activation function &
yj
I. II (1 pu) Σ p
4
107 Lipschitz constant. Thus, EA, < Σ p||l
π K(1
==1+1
==111
p=]] w lh
1. Variance: Var B(A?) (EA)² < B(A?), here we will calculate EA?:
360
2. E = E
Σ
N
Σ
N1
(1)
(1- - yu Uiz (1 - < SiC
(1)
C 8-h Σ Wj;Wij N
j1=1
12
81
s- Ni
p².
Σ
W W/ = || - 1|001
N?
12
Therefore, <
dpplwil 12
Chp² ( 110))
3. EA =
Σ
Σ
w Ujj
(1)
(1 - ( )w (1) (1- & + +C fi- Σ Wÿ W/
+ N
31
32
$1
pCp (w, w') + p² CP (11 (|/w||allw/1
(w, w'))
575
4. EAZ- <
No.
Σ [wi
(1+1) 22(y) (1) 01 c()))
+
N₁. (L+1) (1+1) (1( -1("0 (1) - (10)
Σ wi
wi 1
ifj
Pellu| + Qu (1/1111 2
598
5. EA,A, w (1+)) (E (1) 5/ c!
w.
Ny. +
Σ m
_ ( " [w0 o..0 (0, - s30051001)
ifj
PL(W, w') + QL([ww//al11'1 (w,w')
6. P1 = =pCf, QI = =²²07
579
7. Pt- = Ely - 0E)² = EypPpEE = = = = 13 &
PPL).
8. Q Ely 2V - 0'E') C2 B1 40-- -|--) + (R
EP
25

---

9. Consider recurrence 21 = d1, In = On bnIn-11 Then Zn Σ 43 II bj
j=it1
10. Define az = max((u()t)), and 3r = max(|| (1 /1
(1+11) - (w(L+1), w (o(tt)))]. Then
EA EAL & PLac + Q1181
11. Thus, EAi & aL bb where
a = Cipi(as + + -1C by = k² - -p) + pr)8t).
(1)
384
(2)
501
586
Therefore,
BAL L
(Crnlan p(3) (1)))AAAA))))
II k² pr) + +11 pr 3v) |=1+1
387
The goal this proposition was to give an expression for the variance in a similar manner as is done for the mean in g However this proposition did not make to the article because bound bl was not showing any good experimental results.
509
590
9 Introduction into fault tolerance for neuromorphic hardware
Neuromorphic hardware (NH). Given the amount of processing power required for modern Machine Learning applications, emerging hardware technologies are nowadays reviving the newromorphic project and its promise to cut off energy consumption of machine learning by several orders of magnitude [T0]. Neuromorphic implementation of an NN is a physical device where cach neuron corresponds to piece hardware, and ncurons are physically linked forming weights. Thus, the computation is done (theoretically) at the speed of light, compared to many CPU/GPP cycles. The surge in performance could arguably even exceed the one that followed the switch from training on CPUs 8 Recent results on neuromorphic computing report on concrete successes such 10 or basic vowel recognition using only four coupled nano-oscillators 22 Since the
596
547
508
to training on GPUs and TPUS
milliwatt image recognition
components of neuromorphic network are small
25 and unreliable [rr there are crashes in individual neurons 17. They lead to performance degradation. A failure within neuromorphic
or weights inside the network 25 architecture involved in mission-critical application could be disastrous. Hence, fault tolerance in neural networks is an important concrete Artificial Intelligence (AI) safety problem 4. In terms of fault tolerance, the unit of failure in these architectures fine-grained, i.c., an individual neuron or a single synapse, with failure mode frequently being complete crash. This is in contrast with the now classical case of neural network a software deployed single machine where the unit of failure is coarse-grained, i.e., the whole machine holding the entire neural network. For instance, in the popular distributed setting of ML, the so-called parameter-server scheme [6], the unit of failure is worker or a server, but never a neuron or synapse. Whilst very important, fino-grained fault tolerance in neural networks has been overlooked as a concrete Al safety problem.
603
404
005
References
610
[1] Wikipodia. binomial distribution. scction tail bounds. distribution Accessod: 2019-09-19.
https://en http:://en..wikipdii....
61x
612
[2] agronskiy. https://stats..sackkest httpsssssssstaccccccaan..g.om http://cccccccccooooo//tttto//// nonrstioons/0490///te wtte:/itnne--eeeeetttteeeeeeeeee hatll:/pppeaaiiioooaaaaaaaaeeees remainder, 2013.
Taking the expectation of taylor
series
(especially the
remainder).
614
615
3 D. Amodei and D. Hernandez. AT and compute. Downloaded from hilps:/ Allps://blog. coolog.opemmi...om wtteemai.coo//am www./aiaaddcooommpeeeee 2018.
26

---

[4] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. Concrete problems in ai safety. orXiv preprint ar arXiv:1606.06560 2016.
5 R. Arratia and L. Gordon. Tutorial on large deviations for the binomial distribution. Bulletin of mathemalical biology, 51(():122 (1):122-111,, 1989.
6 Benjamini, G. Kalai, and O. Schramm. Noise sensitivity of boolean functions and applications to percolation. Publications Mathématiques de l'Institut des Hautes Etudes Scientifiques, 90(1):5 43, 1999.
7 L. Chizat, E. Oyallon, and F. Bach. On lazy training in differentiable programming. 2019.
624
8 A. Doerig, A. Schurger, K. Hess, and M. H. Herzog. The unfolding argument: Why it and other causal structure theories cannot explain consciousness. Consciousness and cogmition, 72:49 59, 2019.
9 F. EI Mhamdi, R. Guerraoui, and S. Rouault. On the robustness of neural network. In Reliable Distributed Systems (SRDS), 2017 IEEE 36th Symposium on, pages 84 93. IEEE, 2017.
10 S. K. Essor, R. Appuswamy, P. Merolla, J. V. Arthur, and D. S. Modha. Backpropagation for energy-efficient neuromorphic computing, In Advances in Neural Information Processing Systems, pages 1117 1125, 2015.
699
630
[1] B. Ghorbani, S. Krishnan, and Y. Xiao. An investigation into neural net optimization via hessian eigenvalue density. arXiv preprint 19t::901.11111 2019.
631
632
12] W. H. Guss. Deep function machinos: Generalized neural networks for topological layer expression. arXiv preprinl arXiv:1612.04790 2016.
[13 A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry. Adversarial examples are not bugs, they are features. Xie preprinl (oXxxx1905.02177 2019.
[14] A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neurul Information Processing Systems 31, pages 8580 8589. Curran Associates, Inc., 2018.
637
[15] N. Le Roux and Y. Bengio. Continuous neural networks. In Artificial Intelligence and Statistics, pages 404 411, 2007.
640
641
[16] M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su. Scaling distributed machine learning with the parameter server. In OSDI, volume 1, page 3, 2014.
642
643
[17] M. Liu, L. Xia, Y. Wang, and K. Chakrabarty. Fault tolerance in neuromorphic computing systems. In Proceedings of the 24th Asia and South Pacific Design Automation Conference, pages 216-223. ACM, 2019.
6m8
[88 E. M. E. Mhamdi and R. Guerraoui. When neurons fail. In 2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 1028-1037, May 2017.
19 V. Nagarajan. Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise- resilience. coRR, pages 1-36, 2019.
20 Naihong Wei, Shiyuan Yang, and Shibai Tong. A modified learning algorithm for improving the fault tolerance of bp networks. In Proceedings of International Conference on Neural Networks (ICNN 96), volume 1, pages 247-252 1.l.1, June 1996.
000
881
21] B. Neyshabur, S. Bhojanapalli, D. McAllester, and N. Srebro. Exploring Generalization in Deep Learning. CoRR, 2017.
653
654
[22] M. Romera, P. Talatchian, S. Tsunegi, F. A. Araujo, V. Cros, P. Bortolotti, J. Trastoy, K. Yakushiji, A. Fukushima, H. Kubota, et al. Vowel recognition with four coupled spin-torque nano-oscillators. Nature, :63(773)):230,, 2018.
856
857
[3 S. Sonoda and N. Murata. Double continuum limit of deep neural networks. In ICML Workshop Principled Approaches to Deep Learning, 2017.
24 Y. Tang and R. R. Salakhutdinov. Learning stochastic feedforward neural networks. In Advonces in Neural Information Processing Systems, pages 530-538, 2013.
27

---

[25] A. Tran, S. Yanushkevich, S. Lyshevski, and V. Shmerko. Design of neuromorphic logic networks and fault- tolerant computing. In 2011 11th IEEE International Conference on Nanotechnology, pages 457-462, IEEE, 2011.
664
[66] M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pages 818 833. Springer, 2014.
28


Reviews:
[{"review_id": "SyehYmu1qH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This contribution studies the impact of deletions of random neurons on prediction accuracy of trained architecture, with the application to failure analysis and the specific context of neuromorphic hardware. The manuscript shows that worst-case analysis of failure modes is NP hard and contributes a theoretical analysis of the average case impact of random perturbations with Bernouilli noise on prediction accuracy, as well as a training algorithm based on aggregation. The difficulty of tight bounds comes from the fact that with many layers a neural network can have a very large Lipschitz constant. The average case analysis is based on wide neural networks and an assumption of a form of smoothness in the values of hidden units as the width increases. The improve fitting procedure is done by adding a set of regularizing terms, including regularizing the spectral norm of the layers.\n\nThe robustness properties can be interesting to a wider community than that of neuromorphic hardware. In this sense, the manuscript provides interesting content, although I do fear that it is not framed properly. Indeed, the introduction and conclusion mention robustness as a central concern, which it is indeed, but the neuron failures are quite minor in this respect. More relevant questions would be: is the approach introduced here useful to limit the impact of adversarial examples? Does it provide good regularizations that improve generalization? I would believe that the regularization provided are interesting by looking at their expression; in particular the regularization of the operator norm of layers makes a lot of sense. That said, there is some empirical evidence that batch norm achieves similar effects, but with a significantly reduced cost.\n\nAnother limitation of the work is that it pushes towards very wide networks and ensemble predictions. These significantly increase the prediction cost and are often frowned upon by applications.\n\nIt seems to me that the manuscript has readability issues: the procedure introduced is quite unclear and could not be reimplemented from reading the manuscript (including the supplementary materials). Also, the results in the main part of the manuscript are presented to tersely: I do not understand where in table 2 dropout is varied.\n\nThe contributed algorithm has many clauses to tune dynamically the behavior of the regularizations and the architecture. These are very hard to control in theory. They would need strong empirical validation on many different datasets.\n\nIt is also very costly, as it involves repeatedly training from scratch a neural network.\n\nThe manuscript discusses in several places a median aggregation, which gives robustness properties to the predictor. I must admit that I have not been available to see it in the algorithm. This worries me, because it reveals that I do not understand the approach. The beginning of section 6.1, in the appendix, suggests details that are not understandable from the algorithm description.\n\nFinally, a discussion of the links to dropout would be interesting: both in practice, as dropout can be seen as simulating neuron failure during training, as well as from the point of view of theory, as there has been many attempts to analyze theoretically dropout (starting with Wager NIPS 2013, but more advanced work is found in Gal ICML 2015, Helmbold JMLR 2015, Mianjy ICML 2018).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Skx2F789YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\nReview: This paper considers the problem of dropping neurons from a neural network.  In the case where this is done randomly, this corresponds to the widely studied dropout algorithm.  If the goal is to become robust to randomly dropped neurons during evaluation, then it seems sufficient to just train with dropout (there is also a gaussian approximation to dropout using the central limit theorem called \"fast dropout\").  \n\nI think there are two directions I find interesting that are touched on by this paper.  One is the idea of dropping neurons as an adversarial attack, which I think has been studied empirically but not theoretically (to my knowledge).  However then it would be important to specify the budget of the attack - how many neurons can they remove and how precisely can they pick which neuron to remove?  Another would be studying the conditions for dropout to be useful as a regularizer (and not underfit), which are again somewhat understood experimentally but could deserve a more theoretical treatment.  \n\nHowever I don't think this paper solved a sufficiently clear problem and the motivation is somewhat confusing to me, especially when it seems like an analysis of dropout, and that isn't even mentioned until the 7th page.  \n\nNotes: \n  -Paper considers loss of function from loss of a few neurons.  \n\n  -Idea is to study more rigorously the fault tolerance of neural networks to losing a subset of the neurons.  \n\n  -In terms of impact of the work, one thing to consider is that even if a normally or arbitrarily trained network doesn't have perfect fault tolerance to dropping neurons, a neural network *trained* with dropping networks could learn hidden states which become more fault tolerant.  \n\n\nMinor Comments: \n  -I'm a bit unhappy with the argument about the brain losing neurons unless it has better referencing from neuroscience.  I imagine it's true in general but I wouldn't be surprised if some neurons were really essential.  For example squid have a few giant neurons that control propulsion.  It's just the first sentence so maybe I'm nitpicking.  \n\n  -  It also seems weird that the opening of the paper doesn't give more attention to dropout, since it's a well known regularizer and seems rather closely related conceptually.  \n\n  - In the intro it says neuromorphic hardware would pass information at the speed of light.  Is this really true?  My understanding is neuromorphic hardware would still generally use electrical or chemical signals but not pass things at the speed of light.  \n\n  - The citation format is not valid for ICLR. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Bkg_YfTZ9B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the problem of fault telorance\non NN: basically how the predictions are affected by\nfailure of certain neurons at prediction time. The analysis\nis theoretical under the classical assumption of lipschitz\nbounded non-linearities and looking at the limit case\nof an infinite number of neurons. The paper is well\nwritten and comes with public code that allows to replicate\nthe experiments illustrating the theoretical derivations.\n\nThe paper is well motivated and addresses a timely matter.\n\nTypos\n\n- \"the error of the output of\" -> \"the error of the output\"\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review:  This paper focuses on the problem of robustness in the network with random loss of neurons.  However, reviewers had issues with insufficient clarity of the presentation, and lack of discussion about closely related dropout approach.

 
 
Decision: Reject