Numerical influence of ReLU’(0) on backpropagation
Authors: David Bertoin
Jerome Bolte
Sébastien Gerchinovitz
Edouard Pauwels
Venue: NeurIPS 2021 Poster (2021)
Keywords: Nonsmooth algorithmic differentiation
numerical precision
neural network training
conservative fields

Abstract: In theory, the choice of ReLU(0) in [0, 1] for a neural network has a negligible influence both on backpropagation and training. Yet, in the real world, 32 bits default precision combined with the size of deep learning problems makes it a hyperparameter of training methods. We investigate the importance of the value of ReLU'(0) for several precision levels (16, 32, 64 bits), on various networks (fully connected, VGG, ResNet) and datasets (MNIST, CIFAR10, SVHN, ImageNet). We observe considerable variations of backpropagation outputs which occur around half of the time in 32 bits precision. The effect disappears with double precision, while it is systematic at 16 bits. For vanilla SGD training, the choice ReLU'(0) = 0 seems to be the most efficient. For our experiments on ImageNet the gain in test accuracy over ReLU'(0) = 1 was more than 10 points (two runs). We also evidence that reconditioning approaches as batch-norm or ADAM tend to buffer the influence of ReLU'(0)’s value. Overall, the message we convey is that algorithmic differentiation of nonsmooth problems potentially hides parameters that could be tuned advantageously.

Numerical influence of ReLU'(()) on backpropagation
David Bertoin IRT Saint Exupéry ISAE-SUPAERO ANITI
Jérôme Bolte
Toulouse School Economics
Université Toulouse Capitole
ANITI
Toulouse, France david. parrtonnnirt--snttxuupee com
Toulouse, France jbolte@ut-capitole
Sébastien Gerchinovitz
Edouard Pauwels
IRT Saint Exupéry
CNRS
Institut de Mathématiques de Toulouse
IRIT, Université Paul Sabatier
ANITI
ANITI
Toulouse, France
Toulouse, France edouard. pawels@irit.
sebastien argproor....coootttttrrrrreeeeee com
Abstract
In theory, the choice of Re)U'(0) in [0, 1] for a neural network has a negligible influence both backpropagation and training. Yet, in the real world, 32 bits default precision combined with the size of deep learning problems makes it a hyperparameter of training methods. We investigate the importance of the value of ReLU'(0) for several precision levels (16, 32, 64 bits), on various networks (fully connected, VGG, ResNet) and datasets (MNIST, CIFARIO, SVHN, Ima- geNet). We observe considerable variations of backpropagation outputs which occur around half of the time in 32 bits precision. The effect disappears with dou- ble precision, while it is systematic at 16 bits. For vanilla SGD training, the choice ReLU (0) the gain in test accuracy over ReLU'(() We also evidence that reconditioning approaches as batch-norm or ADAM tend buffer the influence of ReLU' (0)'s value. Overall, the message we convey is that algorithmic differentiation of nonsmooth problems potentially hides parameters that could be tuned advantageously.
seems be the most efficient. For our experiments on ImageNet was more than 10 points (two runs),
Introduction
The training phase neural networks relies on first- order methods such as Stochastic Gradient Descent (SGD) [14 9 and crucially algorithmic dif- ferentiation [15]. The fast "differentiator" used in practice to compute mini-batch descent directions is the backpropagation algorithm [29 4 Although designed initially for differentiable problems, is applied indifferently to smooth or nonsmooth networks. In the nonsmooth case this requires surrogate derivatives at the regularity points. We focus the famous ReLU max(0, -), for which a value for s = ReLU (O) has to be chosen. priori, any value in [0, 1] bears a variational sense it corresponds to subgradient (27). Yet in most libraries 0 chosen; it is the case for TensorFlow 2 PyTorch (66 or Jax 10]. Why this choice? What would be the impact of a different value of s? How this interacts with other training strategies? We will use the notation backprop, to denote backpropagation implemented with ReLU' (0) for any given real number s
Nonsmooth algorithmic differentiation:
Definition in Section This can be coded explicitly or cheaply emulated by backpropp- Indeed, consid- ering fa: x + (eLU(x) s(ReUU(-2) - ReLU((:) + æ) = ReLU(x), we have (ackpropp[/a((0)
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Weight difference 32bits
Minimaal activation 32bis
10-0
10
10-7
0-
10
-2
10-t
10-8
-4
25 50
75
100
-5.0
-2.5
0.0
2.5
Iteration counter
Iteration counter
1e-6
Figure 1: Left: Difference between network parameters (L' norm), 100 iterations within epoch.
"O vs O" indicates |00,,- 0R,0|l1 where Ō is second run for sanity check, "O vs I" indicates ||0:.0 - 0x. ||1- Center: minimal absolute activation of the hidden layers within the k-th mini-batch, before ReLUU At iteration 65, the jump on the left coincides with the drop in the center, and an exact evaluation of (0). Right: illustration of the bifurcation zone at iteration = 65 in a randomly generated network weight parameter plane (iteration 65 in the center). The quantity represented is the absolute value of the neuron of the first hidden layer which is found to be exactly zero within the mini-batch before application of ReLU (exact zeros are represented in white).
What does backpropagation compute? A popular thinking is that the impact of s = ReLU' should be limited concerns the value at single point. This happens be exact in theory but for surprisingly complex reasons related to Whitney stratification (see Section2lan references therein):
For given neural network architecture, backprop, outputs gradient almost everywhere, independently of the choice of 8 E R, see 77 and (5) for detailed treatment of ReLU networks.
Under proper randomization of the initialization and the step-size of SGD, with probability the value of 8 has no impact during training with backprop, as a gradient surrogate, see [8] 6).
In particular, the set of network parameters such that backprop, # backprop, is negligible and the vast majority of SGD sequences produced by training neural networks are not impacted by changing the value of = ReLU (0). These results should in principle close the question about the role of ReLU'(() (0). Yet, this is not what we observe practice with the default settings of usual libraries.
A surprising experiment and the bifurcation zone: An empirical observation on MNIST trig- gered our investigations: consider a fully connected ReLU network, and let (0s,))keN and (0x. )seN be two training weights sequences obtained using SGD, with the same random initialization and the same mini-batch sequence but choosing respectively in Figure[ (left), the two sequences differ. A closer look shows that the sudden divergence is related to what we call the bifurcation zone, i.e., the set So.1 of weights such that backprop, * backpropp- As mentioned previously this contradicts theory which predicts that the bifurcation zone is met with null probability during training. This contradiction is due to the precision tions and, to a lesser extent, to the size of deep learning problems. Indeed, rounding schemes used for inexact arithmetics over the reals (which set to zero all values below a certain threshold), may "thicken" negligible sets. This is precisely what happens in our experiments (Figure I.
and = 1 in PyTorch [26]. As depicted
floating point opera-
The role of numerical precision: Contrary to numerical linear algebra libraries such as numpy. which operates by default under 64 bits precision, the default choice in PyTorch is 32 bits precision (as in TensorFlow or Jax). We thus modulated machine precision to evaluate the importance of the bifurcation zone in Section 3 In 32 bits, observed that empirically the zone occupies from about 10% to 90% of the network weight space. It becomes invisible in 64 bits precision even for quite large architectures, while, in 16 bits, it systematically fills up the whole space. Although numerical precision is the primary cause of the apparition of the zone, we identify other factors such as network size, sample size. Let us mention that low precision neural network training a topic of active research [3 20 11 [6], see also 28 for an overview. Our investigations are complementary as we focus on the interplay between nonsmoothness and numerical precision.
Impact on learning: The next natural question is to measure the impact of the choice of (0) in machine learning terms. In Section 4 we conduct extensive experiments combin- ing different architectures (fully connected, VGG, ResNet), datasets (MNIST, SVHN, CIFAR10,

---

ImageNet) and other learning factors (Adam optimizer, batch normalization, dropout). In 32 bits numerical precision (default PyTorch or Tensorflow), we consistently observe an effect of choos- ing # O. We observe significant decrease in terms of test accuracy as |s| increases; this can be explained by chaotic oscillatory behaviors induced during training. In some cases gradients even ex- plode and learning cannot be achieved. The sensitivity to this effect highly depends the problem at hand, in particular, on the network structure and the dataset. On the other hand the choice seems to be the most stable. We also observe that both batch normalization [21] and-to lesser degree-the Adam optimizer [22] considerably mitigate this effect. All our experiments are done using PyTorch 26];; provide the code to generate all figures presented in this manuscript.
One important message that, even if the default choice 0 seems to be the most stable, our experiments show counter-intuitie phenomenon that illustrates the interplay between numerical precision and nonsmoothness, and calls for caution when learning nonsmooth networks.
Outline of the paper: In Section 2 we recall elements of nonsmooth algorithmic differentiation which are key to understand the mathematics underlying our experiments. Most results were pub- lished in 7 []; we provide more detailed pointers to this literature in Appendix A. In Section the bifurcation zone and factors influencing its importance using fully connected networks on the MNIST dataset. Neural network training experiments are detailed in Section with further experimental details and additional experiments reported Appendix [
we describe investigations
2 On the mathematics of backpropagation for ReLU networks
This section recalls recent advances on the mathematical properties of backpropagation, with in particular the almost sure absence of impact of ReI.U' (0) on the learning phase (assuming exact arithmetic over the reals). The main mathematical tools are conservative fields developed in [7] we provide a simplified overview which applicable wide class of neural networks.
2.1 Empirical risk minimization and backpropagation
Given a training set { j=m... the supervised training of a neural network ʃ consists in minimizing the empirical risk:
N N
min BER
((0)
(ff(z,,0..v.
where 0 E Rº' are the network's weight parameters and l is loss function. The problem can be rewritten abstractly, for each i = I...., N and 0 E RR, E( ((f(x, 1)) = (0))) where the function
4: IRP IR is a composition of the form
= 9t 9t.M
gt.1
where for each j = 1,.... M, the function J. is locally Lipschitz with appropriate input and output dimensions. A concrete example of what the functions gr.t look like is given in Appendix [A.2 in the special case of fully connected ReLU networks. Furthermore, we associate with each Ji.j generalized Jacobian J4, which such that (i,((w) Jac (w) whenever Ji.j is differentiable at w and Jac denotes the usual Jacobian. The value of Ji, at the nondifferentiabiiiiy loci of gi,j can be arbitrary for the moment. The backpropagation algorithm is an automatized implementation of the rules of differential calculus: for each i = 1...., N, we have
backprop (0) Ji.M (-i.M-1 .... It,1 (0)) x JI.M-1 (0L,M-00... gi.1 (0) ... Jr,1 (0).
Famous autograd libraries such as PyTorch 26 TensorFlow 1 implement dictionaries of func- tions with their corresponding generalized Jacobians J, as well as efficient numerical implemen- tation of the quantities defined in B
2.2 ReLU networks training
Our main example is based on the function ReLU: - R defined by ReLU(z) max(x,0) 03. It differentiable save at the origin and satisfies ReLU' (z) 0 for 0 and ReLU'(x) 1 for > 0.

---

The value of the derivative at x = 0 could be arbitrary in [o, 1] as we have (R))U(() [0, 1], where Ô denotes the subgradient from convex analysis. Let us insist on the fact that any value within [0. 1] has a variational meaning. For example Py PyTorch and TensorFlow use ReLU (0) = 0.
Throughout the paper, and following the lines of (8) we say that function g RR R2 is elementary log-exp if it can be described by a finite compositional expression involving the basic operations -, / as well as the exponential and logarithm functions, inside their domains of definition. Examples include the logistic loss log(1 + +) on IR, the multivariate Gaussian density (2x)- K/2 -
RK, and the softmax function /e":/Exi -===e"")) ISisk on R The expressions /²/2² and exp(-1/a²) do not fit this definition because evaluation at a = 0 cannot be defined by the formula. Roughly speaking computer evaluating an elementary log-exp expression should not output any NaN error for any input.
Definition (ReLU network training). Assume that in (T. the function is elementary log-exp, the network f has arbitrary structure and the functions Ji.j in (2) are either elementary log-exp or consist in applying the ReLU function to some coordinates of their input. We then call the problem in ① a ReLU network training problem. Furthermore, for any R. we denote by backprop, the quantity defined in 3 when ReLU'(0) = s for all ReLU functions involved in the composition 2
Other nonsmooth activation functions: The ReLU operation actually allows express many other types of nonsmoothness such as absolute values, maxima, minima, quantiles (med for me- dian) or soft-thresholding (st). For any I, z, E R, we indeed have |a| = ReLr((2) 3, 2 max(x, v) = (|= yl + x + y), min(r, y) min (z. y), max(a, 2), max(y, 2)), st(x)= ReLU(x - 1) ReLU((= 11-xx-1) 1).
max( xx, v). med(2. 2) =
Definition is thus much more general than it may seem since allows, for example, to express convolutional neural networks with max-pooling layers such as VGG or ResNet architectures which correspond to the models considered in the experimental section (although we do not re-program pooling using ReLUU). The following theorem is due to 77. with alternative version in (8.
Theorem 1 (Backprop returns gradient a.e.). Consider a ReLU network training problem as in Definition 1 and T 2 1. Define S c R as the complement of the set
{0 E PR, 1, differentiable at 6, backprop,1.()) = (O),
Vi (1....,N), SET,,T)
Then S is contained in a finite union of embedded differentiable manifolds of dimension at most P 1 (and in particular has Lebesgue measure zero).
Although this theorem looks natural, this is a nontrivial result about the backpropagation algorithm that led to the introduction of conservative fields 77. It implies that all choices for s = ReL. (O) in [0, = @ReLU(O) are essentially equivalent modulo negligible set S. Perhaps more surprisingly, s can be chosen arbitrarily in IR without breaking this essential property of backprop. The set is called the bifurcation zone throughout the manuscript. For ReLU network training problems, the bifurcation zone a Lebesgue zero set and is actually contained locally in a finite union of smooth objects of dimension strictly smaller than the ambient dimension. This geometric result reveals surprising link between backpropagation and Whitney stratifications, as described in 7 8). In any case the bifurcation zone is completely negligible. Note that the same result holds if we allow each different call to the ReLU function to use different values for ReLU' (0).
2.3 ReLU network training with SGD
Let (Be)ken denote sequence of mini-batcher with sizes |Bll f1,..., NN for all k and ag 0 the learning rate. Given initial weights 8g E R.' and any parameter s e R, the SGD training procedure of f consists in applying the recursion
0x+1. Ōk,e 7
ak
Σ nE B
[(( Ok.0), yn)).
(4)
BB
where 7 >> 0 is a step-size parameter. Note that we explicitly wrote the dependency of the sequence in = (0). According to Theorem | the initialization Op is chosen randomly, say, uniformly in a ball, hypercube, or with iid Gaussian entries, then with probability 1, 0g does not fall in the bifurcation zone S. Intuitively, since S is negligible, the odds that one of the iterates produced by the

---

algorithm fall on S are very low. As a consequence, varying in the recursion 4 does not modify the sequence, This rationale is actually true for almost all values of y. This provides a rigorous statement of the idea that the choice of (e) "does not affect" neural network training. The following result is based on arguments developed in [ see also [6] in a probabilistic context.
Theorem 2 (0) does not impact training with probability 1). Consider a ReLU network training problem as in Definition [ Let (Bx)keN be a sequence of mini-batckes with |Bgl с {l..... N for all k and ag > 0 the associated learning rate parameter. Choose 9o uniformly at random hypercube and y uniformly in a bounded interval T Let E R. set fo.s tn. and consider the recursion given in 4 Then, with probability one, for all k E N, Ok.s 0g.0-
3 Surprising experiments on a simple feedforward network
3.1 ReLU'(0) has an impact
Even though the ReLU activation function is non-differentiable at O. autograd libraries such as PyTorch [26] TensorFlow [1 implement its derivative with ReLU' (0) 0. What happens if one chooses ReLU (0) = 1? The popular answer to this question is that it should have no effect.
Theorems T and provide formal justification which is far from being trivial.
32 bits MNIST experiment We ran a simple experiment confirm this answer. We initialized two fully connected neural networks fo and fi of size 784 x 2000 x 128 x 10 with the same weights 0o.0- = 00,1 R which are chosen at random. Using the MNIST dataset [24], we trained fo and fi with the same sequence of mini- (B₂) kex) (minibatch size 128), using the recursion in (4) for s = 0 and s = and with a fixed a₃ = 1, and y chosen uniformly at random in [0.01, 0.012]. At each iteration k, we computed the sum ||0x,0 1k.11la of the absolute differences between the coordinates of 0x,0 and 0x.1. As a sanity check, we actually computed θ,,0 a second time, denoting this by 6x.0, using third network to control for sources of divergence in our implementation. Results are reported in Figure [] The experiment was run using PyTorch| (26 on a CPU.
Re)U'(() has impact First observe difference between θk.0 and dk.o. which shows that we have controlled all possible sources of divergence in PyTorch. Second, while no differences between 6%.0 and 0o, is expected (Theorem 2). we observe sudden deviation of |l0x,o 0x. ll at iteration 65 which then increases in smooth way. The deviation is sudden as the norm is exactly zero before iteration 65 and jumps above one after. Therefore this cannot be explained by an accu- mulation of small rounding errors throughout iterations, as this would produce a smooth divergence starting at the first iteration. So this suggests that there a special event at iteration 65.
The center part of Figure I displays the minimal absolute value of neurons of the first hidden layer evaluated on the current mini-batch, before the application of ReLU. It turns out that iteration 65, this minimal value is exactly 0, resulting in a drop in the center of Figure [ This means that the divergence is actually due an iterate of the sequence falling exactly on the bifurcation zone. According to Theorem this event is so exceptional that it should never been seen.
Practice and Theory: Numerical precision vs Genericity This contradiction can be solved as follows: the minimal absolute value Figure [ between 10-6 and 10-8 which roughly the value of machine precision in 32 bits float arithmetic. This machine precision value is of the 16 in 64 bits floating arithmetic which is orders of magnitude smaller than the typical value represented in Figure [ And indeed, performing the same experiment in 64 bits precision, the divergence of ||0s,0 0l. disappears and the algorithm can actually be run for many epochs without any divergence between the two sequences. This is represented in Figure in Appendix B We also report similar results using ReLU6 [19 in place of ReLU on a similar network.
order 10
10-16
3.2 Relative volume of the bifurcation zone and relative gradient variation
The previous experiment suggests that mini-batch SGD algorithm
4 crossed the bifurcation zone: So1 {0 e N 3i E {I...., N}. backprop, (4](0) + backprop,[4]0
c S. (5) This unlikely phenomenon due to finite numerical precision arithmetic which thickens the neg- ligible set Sp1 in proportion to the machine precision threshold. This is illustrated on the right of
5

---

Figure [ which represents the bifurcation zone at iteration 65 in a randomly generated hyperplane (uniformly among dimensional hyperplanes) centered around the weight parameters which gener- ated the bifurcation in the previous experiment (evaluation on the same mini-batch). The white area corresponds to some entries being exactly zero, i.e., below the 32 bits machine precision threshold, before application of ReLUU On the other hand, in 64 bits precision, the same representation is much smoother and does not contain exact zeros (see Figure 7 in Appendix BB. This confirms that the level of floating point arithmetic precision explains the observed divergence. We now estimate the relative volume of this bifurcation zone by Monte Carlo sampling (see Appendix All experiments are performed using Py PyTorch [26] on GPU.
C for details).
Experimental procedure - weight randomization: We randomly generate a set of parameters {0, 1 with M = 1000, for fully connected network architecture composed of L hidden lay- ers. Given two consecutive layers, respectively composed of m and m' neurons, the weights of the corresponding affine transform are drawn independently. uniformly in [-or a) where a = /6/ Vm. This the default weight initialization scheme in PyToorch (Kaiming-Unifool sample of parameters, iterating on the whole MNIST dataset, we approximate the proportion of 0z for which backprop, (4))(,) * backprop, (0,) for some i, for different networks and under different conditions (see Appendix M for details).
17). Given this
Impact of the noating-point precision: Using a fixed architecture of three hidden layers of 256 neurons each, we empirically measured the relative volume of Sor using the above experimental procedure, varying the numerical precision. Table reports the obtained estimates. As shown in Table line 1, at 16 bits noating-point precision, all drawn weights ] fall within S01- In sharp contrast, when using a 64 bits precision, none of the sampled weights belong to So1- This proportion is 40% in 32 bits precision. For the rare impacted mini-batches (Table line 2), the relative change in norm is above a factor 20, higher in 16 bits precision ( (Table[]]ine 3). These results confirm that the floating-poing arithmetic precision is key in explaining the impact of ReLU (0) during backpropagation, impacting both frequency and magnitude of the differences.
tonating-poine precision
16 bits
32 bits
64 bits
Proportion of (0:),² in S (CI = 5%)
100%
40%
0%
Proportion of impacted mini- mini-batches (CI 13%)
0.05%
0.0002%
0%
Relative L difference for impacted mini-batches (Ist quartile, median, 3rd quartile)
(98, 117, 137)
(19,25,47)
00)
Table 1: Impact of S according the floating-point precision on fully connected neural network (784 x 256 x 256 x 256 x 10) on MNIST. First line: proportion of drawn weights 0; such that at least one mini-batch results in difference between backpropo and backprop,. CI stands for Confidence Interval with 5% risk (Hoeffding CI, see Appendix C.. Second line: overall proportion of mini- batch-weight vector pairs causing a difference between backprop, and backprop, (McDiarmid CI, see Appendix C Third line: distribution of ||backprop, backprop, //2/ |bbckpropol/ for the affected mini-batch--eipe vector pairs.
Impact of sample and mini-batch size: Given a training set {I In Bn)]a J==I..N and a random variable 0 e !R with distribution iP., we estimate the probability that E So1 C S. Intuitively, this probability should increase with sample size N. We perform this estimation for fixed architecture of hidden layers composed of 256 neurons each while varying the sample size. Results are reported in Figure For both the 16 and the 32 bits floating- with the sample size while we do not find any sampled weights in Sm in 64 bits precision. We also found that the influence of mini-batch size is not significative.
precisions, our estimation indeed increases
Impact of network size: To evaluate the impact of the network size, we carried out similar Monte Carlo experiment, varying the depth and width of the network. Firstly, we fixed the number of hidden layers to 3. Following the same experimental procedure, we empirically estimated the relative volume of Su1, varying the width of the hidden layers. The results, reported in Figure 2 show that increasing the number of neurons by layer increases the probability to find a random 0 Po in Sm for both 16 and 32 floating-pon point precision. In 64 bits, even with the largest width tested (1024 neurons), no sampled weight parameter is found in Smn Similarly we repeated the experiment varying the network depth and fixing, this time, the width of the layers to 256 neurons. Anew, the

---

100
P Precision
Proportion
60
16
40
0
S Sample size (1000s)
Number neurons
Number of layers
Batch size
1
10
20
30
40
50
16
64
256
5i2 1024
64 256 5i2
1024
Figure 2: Influence of different size parameters on the proportion of drawn weight vector 0; such that least mini-batch results in difference between backpropp and backprop, MNIST dataset. Confidence interval at risk level 5% results in a variation of +5% for all represented proportions. First: 4 hidden layers with 256 neurons and batch size 256, varying the size of the training set. Second: 3 hidden layer network with mini- size 256, varying the number of neuron per layer. Third: 256 neurons per layer and mini-batch size 256, varying the number of layers. Fourth: hidden layers with 256 neurons per layer, varying mini-batch size.
results, reported in Figure 2 show that increasing the network depth increases the probability that 0 ~ Pp belongs to So1 for both 16 and 32 bits floating- precision while this probability is in 64 bits precision. This shows that the size of the network, both in terms of number of layers and size of layers is positively related to the effect of the choice of in backpropagation. On the other hand, the fact that neither the network depth, width, or the number of samples impact the 64 bits case suggests that, within our framework, numerical precision is the primary factor of deviation.
Consequences for learning
4.1 Benchmarks and implementation
Datasets and networks We further investigate the effect of the phenomenon described in Section
terms of learning using the CIFARIO dataset [23] and the VGG11 architecture [1I. To confirm our findings in alternative settings, we also use the MNIST 124), SVHN (25) and ImageNet [12 datasets, fully connected networks (3 hidden layers of size 2048), and the ResNet18 and ResNe150 architectures [8]. Additional details on the different architectures and datasets are found in Ap- pendix D.I By default, unless stated otherwise, we use the SGD optimizer. We also investigated the effect of batch normalization (2I), dropout (32, the Adam optimizer
[22] well numerical precision. All the experiments in this section are run in PyTorch [26] on GPU. For each training experiment presented in this section (except ImageNet experiments), to tune learning rates for each experimental condition; see also Appendix D.2
the optuna library B
4.2 Effect on training and test errors
We first consider training VGGI architecture on CIFARIO using the SGD optimizer. For differ- ent values of ReLU" (0), we train this network ten times with random initializations under 32 bits arithmetic precision. The results are depicted in Figure 3 Without batch normalization, varying the value of ReL.U' (0) beyond a magnitude of 0. has a detrimental effect on the test accuracy, resulting concave shaped curve with maximum around ReLU' (0) 0. On the other hand, the decrease of the training loss with the number of epochs suggests that choosing ReLU' U'( * 0 induces jiggling behaviors with possible sudden jumps during training. Note that the choice ReLU'(0)
0 leads to smooth decrease and that the magnitude of the jumps for other values is related to the magnitude of the chosen value. This is consistent with the interpretation that changing the value of RcLU' (0) has an excitatory effect on training. We observed qualitatively similar behaviors for fully connected network on MNIST and a ResNet18 on CIFARIO (Appendix D). Sensitivity to the magnitude of ReLU'(() depends the network architecture: our fully connected network on MNIST is less sen- sitive this value than VGG11 and ResNet18. The latter shows very high sensitivity since for values above 0.2, training becomes very unstable and almost impossible.

---

0.92
norm False
1.0
accuracy
ReLU'00
True
0.8
0.4
0.88
Train loss
0.6
0.2
0.86
0.4
0.2
0.0
0.84
0.2
0.4
0.6
0.82
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
50
100
150
200
0.8
ReLU
opoch
1.0
Figure 3: Training VGG11 network CIFARIO with SGD. Left: test accuracy with and without batch normalization. Right: training loss without batchnorm. For each experiment we performed 10 random initializations, depicted by the boxplots the left and the filled contours the right (standard deviation).
These experiments complement the preliminary results obtained in Section 3 In particular choosing different values for (0) has an effect, it induces a chaotic behavior during training which affects test accuracy. The default choice ReLU'(0) (0) = O seems to provide the best performances.
To conclude, conducted four training experiments for ResNer50 the ImageNet dataset 12 using the SGD optimizer. These were conducted with fixed learning rate, contrary to results reported results in massive drop from around
above. We observe that switching from ReLU'(0) (0)
75% to 63% or 55% for two runs.
4.3 Mitigating factors: numerical precision, batch-normalization and Adam
We analyze below the combined effects of the variations of Re) with numerical precision or classical reconditioning methods: Adam, batch-normalization and dropout.
As represented in Figure batch normalization 21 only allows to attain higher test accuracy, but it also completely filters out the effect of the choice of the value of ReLUU(()), resulting in flat shaped curve for test accuracy. This is consistent with what observed on the training loss (Figure 12 in Appendix D.3} for which different choices of ReLU' (0) lead to indistinguishable training loss curves. This experiment suggests that batch normalization has significant impact in reducing the effect of the choice of ReLU' Re). This observation was confirmed with a very similar behavior on the MNIST dataset with fully connected network (Figure g in Appendix D.2 We could observe a similar effect on CIFAR 10 using a ResNet18 architecture (see Appendix D.5p. however in this case the value of ReLU (0) still has sigmificative impact on test error, the ResNet18 architecture being much more sensitive.
Batch-normalization:
Using the Adam optimizer: The Adam optimizer 22 is among the most popular algorithms for neural network training; it combines adaptive step-size strategies with momentum. Adaptive step- size acts diagonal preconditioners for gradient steps 22 B and therefore be an effect on the loss landscape of neural network training problems. We trained a VGG1 network on both CIFAR 10 and SVHN using the Adam optimizer. The results presented in Figure 4 We observe a qualitatively similar behavior as in the experiments of Section |.2 but a much lower sensitivity the magnitude of ReLU'(0). In other words, the of the Adam optimizer mitigates the effect of this choice, both in terms test errors and by buffering the magnitude of the sometimes chaotic effect induced training loss optimization (Figure T3 in Appendix D.3)
having
Increasing numerical precision: As shown in Appendix VGG11 with CIFARIO cancels out the effect of RcLU'(0) I, in coherence with Section 3 More specifically (eLU'(0) = 1 in 64 bits precision obtains similar performances as ReLU' 32 bits precision. Furthermore, the numerical precision has barely any effect when ReLU'()) = 0. Finally, we remark that bits with ReI. ReI.U'(0) (0) training is extremely unstable so that we were not able to train the network in this setting.
D.3
using 64 bits floating precision on
0 in

---

097
Test aestaccuroo accuracy
Iest accuracy
095
Batch norm
Batch norm
0.9
False
True
Tue
093
4.0
10.0
ReLU/(0)
ReLU)(0)
Figure 4: Similar to Fig
with VGG1 and Adam optimizer, on CIFAR 10 (left) and SVHN (right).
CIFAR-VGG11
LFAR-VGG11-Batchhee
SVHN-VGG11 1IGGG11BatchNorm
1.0
1.0
P Sml
P0E
Precision
0.5
Precision
64
32b
32
16b
0.0
0.0
5000
10000
5000
10000
200000
400000
sample size
sample size
sample size
Figure 5: Monte Carlo estimation of relative volume on CIFARIO and SVHN with VGG1 network
32 another algorithmic process to regularize deep neu- ral networks. We investigated its effect when combined with varying choices of (e)U'(0). We used a fully connected neural network trained on the MNIST dataset with different values of dropout probability. The results reported in Figure II in Appendix D.2 We did not observe any joint effect of dropout probability and magnitude of ReLU'(0) in this context.
Combination with dropout: Dropout
4.4 Back to the bifurcation zone: more neural nets and the effects of batch-norm
The training experiments are complemented by similar Monte Carlo estimation of the relative volume of the bifurcation zone as performed in Section 3 (same experimental setting). To avoid random outputs we force the GPU to compute convolutions deterministically. Examples of results are given in Figure Additional results on fully connected network with MNIST and ResNet 88 on CIFAR10 10 are shown in Section D We consistently observe high probability of finding an example on the bifurcation zone for large sample sizes. Several comments are in order.
Numerical precision: Numerical precision is the major factor in the thickening of the bifurcation zone. comparison to 32 bits experiments, 16 bits precision dramatically increases its relative importance. We also considered 64 bits precision on SVHN, rather large dataset. Due to the com- putational cost, we only drew random weights and observed no bifurcation on any of terms of the loss whatsoever. This is consistent with the experiments conducted in Section 3 and suggests that, within our framework, 64 bit precision the main mitigating factor for our observations.
Batch normalization: In all our relative volume estimation experiments, observe that batch normalization has very significant effect on the proportion of examples found in the bifurcation zone. In 32 bits precision, the relative size of this zone increases with the addition of batch nor- malization, similar observations were made all experiments presented in Appendix [ This is a counter-intuitii behavior we have observed that batch normalization increases test accuracy and mitigates the effect of ReLU' (00. Similarly in 16 bits precision, the addition of batch normalization seems actually decrease the size of the bifurcation zone. Batch normalization does not result in the same qualitative effect depending on arithmetic precision. These observations open many questions which will be the topic of future research.
4.5 Total number of ReLU calls during training
We consider the MNIST dataset with a fully connected ReLU network, varying the number of layers in 1...., 6 and neuron per layers in 16, 64, 256, 512. For 16 and 32 bits precisions, we perform 100

---

Ratio of zeros by Relu calls
Ratio Relutt) calls
preci sion
10-1
16
32
10
500D0 100000 150000 200D00 250000 300000
Figure 6: Proportion of (0) calls by total number of ReLU calls after 100 epochs for ReLU networks MNIST with number of layers in 1...., 6 and neuron per layers in 16, 64, 256, 512.
epochs and proportion of ()) over all ReLU calls during training. Figure 6 suggests that, for a given precision, the number of times the bifurcation zone is met is roughly proportional to the total number of ReLU calls during training. independently of the architecture used (number of layers, neurons per layers). This suggests that the total number of ReLU calls during training an important factor in understanding the bifurcation phenomenon. Broader investigation of this conjecture will be a matter of future work.
5 Conclusions and future work
The starting point of our work was to determine if the choice of the value ReLU'(O) affects neural network training. Theory tells that this choice should have negligible effect. Performing a simple learning experiment, we discovered that this is false in the real world and the first purpose of this paper is to account for this empirical evidence. This contradiction between theory and practice due to finite floating point arithmetic precision while idealized networks are analyzed theoreti- cally within a model of exact arithmetic on the field of real numbers. Owing to the size of deep learning problems, rounding errors due to numerical precision occur at a relatively high frequency, and virtually all the time for large architectures and datasets under 32 bit arithmetic precision (the default choice for TensorFlow and PyTorch libraries).
Our second goal was to investigate the impact of the choice of s = ReLU' (0) in machine learning terms. In 32 bits precision has an effect on test accuracy which seems be the result of inducing a chaotic behavior in the course of empirical risk minimization. This was observed consistently in all our experiments. However we could not identify systematic quantitative description of this effect; it highly depends on the dataset at hand, the network structure as well as other learning parameters such as the presence of batch normalization and the use of different optimizers. Our experiments illustrate this diversity. We observe interesting robustness property of batch normalization and the Adam optimizer, as well as a high sensitivity to the network structure.
Overall, the goal of this work is to draw attention to an overlooked factor in machine learning and neural networks: nonsmoothness. The ReLU activation probably the most widely used nonlinear- ity in this context, yet its nondifferentiabiiity is mostly ignored. We highlight the fact that the default choice ReLU'(0) (0) 0 seems be the most robust, while different choices could potentially lead to instabilities. For a general nonsmooth nonlincarity, it is not clear priori which choice would be the most robust, if any, and our investigation underlines the potential importance of this question. Our research opens new directions regarding the impact of numerical precision on neural network training. its interplay with nonsmoothness and its combined effect with other learning factors, such as network architecture, batch normalization or optimizers. The main idea is that mathematically negligible factors are not necessarily computationally negligible.
Acknowledgments and Disclosure of Funding
The authors thank anonymous referees for constructive suggestions which greatly improved the pa- per. The authors acknowledge the support of the DEEL project, the AI Interdisciplinary Institute ANITI funding, through the French "Investing for the Future PIA3" program under the Grant agreement ANR- 19-19--313-0004, Air Force Office of Scientific Research, Air Force Material Com- mand, USAF, under grant numbers FA9550-19-1-7026 19-CE:230001--01. J. Bolte also acknowledges the support of ANR Chess, grant ANR-I7-EURE- 0010 and TSE-P.
119550-18-1-0221 and ANR MaSDOL
10

---

References
[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Joze- fowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasude- van, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. Ten- sorFlow: Large-scale machine learning on heterogencous systems, 2015. Software available from tensorflow.org.
[2] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis. J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, al. Tensorflow: A system for large-scale machine learning. In 12th {USENIX) symposium on operating systems design and implementation ({OSDI) 16), pages 265-283, 2016.
[3] T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, Optuna: A next-generation hyper- paramcter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623-2631, 2019.
[4] A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind. Automatic differentiation in machine learning: survey. Journal of machine learning research, 18, 2018.
[5] J. Berner, D. Elbrächter, P. Grohs, and A. Jentzen. Towards a regularity theory for relu cehao rule and global error estimates. 2019 13th International conference on Sampling Theory and Applications (Samp'lAA) pages 1-5. IEFE, 2019.
[6] Bianchi, W. Hachem, and S. Schechtman. Convergence of constant step stochastic gradient descent for non-smooth non-convex functions. arXiv preprint arXiv:2005.08513, 2020.
[7] J. Bolte and E. Pauwels. Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning. Mathematical Programming, pages 1-33, 2020.
[8] J. Bolte and E. Pauwels. A mathematical model for automatic differentiation in machine learn-
ing. In Advances in Neurul Information Processing Systems, volume 33, 2020.
[9] L. Bottou, F. E. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning.
Siam Review, 60(2):223-33
2018.
[10] Bradbury,
Frostig, P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transfor- mations of Python+NumPy programs, 2018.
[11] M. Courbariaux, Y. Bengio, and J.-P. David. Training deep neural networks with low precision multiplications. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
[12] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierar- chical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[13] Duchi, E. Hazan, and Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
[14] I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press Cambridge, 2016.
[15] A. Griewank and A. Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM, 2008.
[16] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan. Deep learning with limited nu- merical precision. In International conference machine learning, pages 1737-1746. PMLR, 2015.
[17] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[18] K. He. X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, 2016.
[19] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.048), 2017.
11

---

[20] K. Hwang and W. Sung. Fixed-poin feedforward deep neural network design using weights+
O, and- 1. In 2014 IEEE Workshop on Signal Processing Systems (SiPS), pages 1-6. IEEE, 2014.
[21] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pages 448-456.
PMLR, 2015.
[22] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, Conference Track Proceedings, 2015.
[23] A. Krizhevsky, V. Nair, and G. Hinton. Cifar-10 (canadian institute for advanced research).
URL. http://www. cs. toronto. edu/kriz/cffar. himl, 5, 2010.
[24] Y. LeCun, L.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(11). 1998.
[25] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B.
Wu, and A. Y. Ng. Reading digits in nat- ural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning 20/ 1, 2011.
[26] A. Paszke, S. Gross, F. Massa,
Lerer, J. Bradbury, G. Chanan, T. Killeen, 7. Lin, N. Gimelshein, I.. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. DeVito, M. Raison, A. Te- jani, S. Chilamkurthy, B. Steiner, L. Fang. Bai, and S. Chintala. Pytorch: An imperative style, high-perffrmance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[27] R. Rockafellar and R. J.-B. Wets. Variational analysis, volume 317. Springer Science & Business Media, 2009.
[28] A. Rodriguez, E. Segal, E. Meiri, E. Fomenko, Y. Kim, H. Shen, and B. Ziv. Lower numerical precision deep learning inference and training. Intel White Paper, 3:1-19, 2018.
[29] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back- propagating errors. nature, 323(6069) 1986.
[30] S. Scholtes. Introduction to piecewise differentiable equations. Springer Science & Business Media, 2012.
[31] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recog- nition. In International Conference on Learning Representations, 2015.
[32] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, ((11):992-11553) 2014.
[33] V. Vanhoucke, A. Senior, and M. Z. Mao. Improving the speed of neural networks cpus, In Deep Learning and Unsupervised Feature Learning NIPS Workshop., 2011.
12


Reviews:
[{"review_id": "pvDQF4pts9", "reviewer": "Reviewer_coEY", "summary": "The ReLU function is not differentiable at 0 which has a mathematical impact on ReLU neural networks. Nevertheless they are optimized using gradient based methods. The folk wisdom is that singular points only occur with zero probability and therefore the issue is irrelevant. This paper challenges this folk wisdom and identifies that the choice of the derivative at 0 does have a tangible impact on the optimization and classification performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this is a good paper and such a study has been overdue. \n\nThe authors may considering the paper https://arxiv.org/pdf/1905.04992.pdf which performs a mathematical analysis of related issues.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "_iDjFadQl7V", "reviewer": "Reviewer_7FU8", "summary": "This paper studies a long-neglected problem: the gradient of the non-smooth activation function in training a neural network. Specifically, the authors study the numerical influence of the ReLU derivative at zero on gradient descent. First, a theory is presented to illustrates the trivialness of ReLU'(0), then, the authors give counter-theory experiments to show that ReLU'(0) can affect training results.  In the last two sections, authors demonstrate that low-precision training, SGD optimizer, no batch norm training can magnify the impact of ReLU'(0).", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this is a very interesting topic that draws little attention in the community before. The authors empirically prove that the numerical choice of the non-smooth activation function's gradient can affect the training behavior. I guess this is not surprising for half-precision training since 16 bit has a much smaller representation range than single-precision values.\nThe paper is well-organized and well-written with a clear structure. \n\n\n**Pros**:\n\n+ The topic of this paper is quite interesting and can inspire more fellow research. \n\n+ The experiments evaluate relevant aspects, e.g. precision bit-with, optimizer, BN, etc.\n\n+ Paper is neat and well-written.\n\n**Cons**: \n\n- No large-scale experiments like ImageNet are conducted. \n\n- I'd like to see the impact of network architectures. For example, will ResNet outperform VGG-Net in terms of this stability of the non-smooth activation function? \n\n- Can the authors provide more examples of non-smooth activation and compare the results. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "My major concerns are the potential lack of more experiments, I hope authors can add more results to it. Then, I'll increase my score to accept. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "Nj110P1DY-i", "reviewer": "Reviewer_J8rk", "summary": "The paper demonstrates empirically that on a variety of architectures and datasets, the derivative of ReLU at 0 is evaluated a significant number of times during training in 16 and 32 bit precision. These events are frequent enough such that defining ReLU'(0) to be anything other than 0 (default value in common DNN frameworks) hurts generalization, and sometimes training as well.\n\nThe authors perform a variety of investigations around this effect, such as the influence of training set size, network width, depth, batch size, BatchNorm, Dropout.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Post-rebuttal Update\n\nThe authors have addressed my main concern on significance, namely that learning rate was not tuned for non-zero derivative settings. Conditioned on having the learning rate and number of training epochs tuned for each setting in section 4 (and other changes discussed in the rebuttal, e.g. the one on expected number of evaluations), I have raised the score from 5 to 7 and recommend accept, since I find it novel and potentially impactful that this setting has a strong influence not only on trainability, but also on generalization.\n\n----\n## Initial Review\n\nOverall, I found this to be an interesting investigation with a surprising result. However, I am leaning to reject this paper due to (1) minimal qualitative / theoretical analysis or explanations of the observed phenomena (see section \"Quality\"), and (2) potentially flawed experimental setting to evaluate the impact of ReLU'(0) on training and test performance (see section \"Significance\"). I can see how the former could be improved at least a bit, and the latter can be clarified / fixed with additional experiments during rebuttal, so I am open to changing my score.\n\n## Originality\n\nThe investigation and findings are to my knowledge original, although I am not an expert in autodiff literature.\n\n## Quality\n\nMixed impressions on quality.\n\nOn one hand, I find the present experiments very thorough, well-designed, and broad, convincingly backing the claims of the paper.\n\nOn the other hand, the empirical results aren't analyzed that much, and the questions of\n\n1. Which hyper-parameters influence the expected number of ReLU'(0) evaluations, and how?\n2. Why does setting ReLU'(0) != 0 hurt performance?\n\nremain unanswered. \n\nFor question (1), the authors could, with the data that they already have, for instance consider a conjecture that \"number of ReLU'(0) evaluations is proportional to the total number of all  ReLU'(.) evaluations during training\", and make the respective plot. This would explain why the number of such events grows with many hyperparameters, but not batch size. \n\nFor question (2) I don't have any immediate suggestions, apart from my comments on the learning rate in the \"Significance\" section.\n\n## Clarity\n\nGood, I found it easy to read the paper.\n\n\n\n## Significance\n\nSignificance is at least average, since I find the fact that ReLU'(0) gets evaluated so often as to influence training / generalization quite surprising. This could make researchers more careful when reasoning about 0-probability events in floating point precision.\n\nOn the other hand, \n\n* These results may not be of practical utility, since most frameworks already use ReLU'(0) = 0, which is empirically the best setting.\n* How did you tune your learning rate, when evaluating test / training performance of networks? If the learning rate was tuned once on ReLU'(0) = 0 architectures, then it is not surprising that altering the gradients while keeping the learning rate fixed leads to a gradual decrease in performance in the magnitude of the change. In this case, I would recommend the authors to tune the learning rate separately for each value of ReLU'(0), and compare the results then. If test / train performance still varies substantially (or if that's what the authors are already doing - please clarify), then I would consider this finding quite significant, and warranting follow-up research into why this happens.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main finding of the paper (that ReLU'(0) influences test / train loss) is potentially of limited impact, if the authors did not tune the learning rate separately for each setting, as I describe in section \"Significance\" above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "5haw4F5RPSD", "reviewer": "Reviewer_kt6J", "summary": "This paper analyzes the impact of the choice of ReLU'[0] on the fidelity of neural networks. Even though one would expect that the value of ReLU'[0] should not have any noticeable impact, the authors notice the divergence in learned parametric values that depend on this choice. They also notice that this effect is more pronounced at lower precision (due to rounding effects that are dominant at fp16 and fp32 vs. fp64). They also study this effect for neural network sizes, topologies (including networks with and without batch normalization) and datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe author does a good job explaining the theory and experiments associated with the study of the impact of ReLU'[0]. The challenge in my view is the following:\n[1] The studies seemed to indicate that the default values of ReLU'[0] = 0 seem to yield the best accuracy results! Therefore, its hard to see how the outcome of this work would be in general useful to neural network (and framework) designers. \n[2] The authors describe the proportion difference % between the different precisions. Can the authors comment on the final model accuracy difference for the different precisions for the different datasets. If there is no substantial difference in accuracy between the different precisions (as is indeed expected), then can the authors comment on why this result is important?\n[3] In general, it'd be good to study this effect on larger models and datasets (e.g. ResNet50 or EfficientNetBx on ImageNet) - to see if this effect results in any noticeable difference in accuracy and to see if there are better optimal values for ReLU'[0]?\n[4] Do the authors have results on other discontinuities besides ReLU'[0]? For e.g. one would expect ReLU[6] (used in MobileNets) to have a discontinuity at 0 and 6?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No comments here. ", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns here - beyond the use of Artificial Intelligence (AI) in general. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The ReLU nonlinearity is popular in deep learning. It's derivative at 0, ReLU’(0), is undefined. This paper presents the surprising observation that the chosen value of the derivative of ReLU at 0 has a substantial effect of the performance achieved in neural network training at typical numerical precision. After the initial round of reviews some concerns surfaced about how robust this observation is to the choice of learning rate and other hyperparameters. The authors addresses these concerns in their response and the reviewers now seem convinced the observed effect is real and meaningful. Most of the reviewers engaged in discussion with the authors and with the rest of the committee: They reached a consensus recommendation to accept the paper.

Reviewer kt6J is the sole remaining negative reviewer: Unfortunately they did not engage in discussion with the committee or the authors. Their review did not point out any serious problems that would invalidate the contribution made by the paper. I do not believe the review of kt6J provides enough reason to not accept the paper, given that the paper presents a clearly surprising and interesting result that is appreciated by the other reviewers.

Authors, please integrate all new results and discussion from the author response phase into the camera ready version of the paper.
Decision: Accept (Poster)