Numerical influence of ReLU’(0) on backpropagation

1 Introduction . Nonsmooth algorithmic differentiation : The training phase of neural networks relies on firstorder methods such as Stochastic Gradient Descent ( SGD ) [ 14 , 9 ] and crucially on algorithmic differentiation [ 15 ] . The fast “ differentiator ” used in practice to compute mini-batch descent directions is the backpropagation algorithm [ 29 , 4 ] . Although designed initially for differentiable problems , it is applied indifferently to smooth or nonsmooth networks . In the nonsmooth case this requires surrogate derivatives at the non regularity points . We focus on the famous ReLU : = max ( 0 , · ) , for which a value for s = ReLU′ ( 0 ) has to be chosen . A priori , any value in [ 0 , 1 ] bears a variational sense as it corresponds to a subgradient [ 27 ] . Yet in most libraries s = 0 is chosen ; it is the case for TensorFlow [ 2 ] , PyTorch [ 26 ] or Jax [ 10 ] . Why this choice ? What would be the impact of a different value of s ? How this interacts with other training strategies ? We will use the notation backprops to denote backpropagation implemented with ReLU′ ( 0 ) = s for any given real number s.1 1Definition in Section 2 . This can be coded explicitly or cheaply emulated by backprop0 . Indeed , considering fs : x 7→ ReLU ( x ) + s ( ReLU ( −x ) − ReLU ( x ) + x ) = ReLU ( x ) , we have backprop0 [ fs ] ( 0 ) = s. 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . What does backpropagation compute ? A popular thinking is that the impact of s = ReLU′ ( 0 ) should be limited as it concerns the value at a single point . This happens to be exact in theory but for surprisingly complex reasons related to Whitney stratification ( see Section 2 and references therein ) : — For a given neural network architecture , backprops outputs a gradient almost everywhere , independently of the choice of s ∈ R , see [ 7 ] and [ 5 ] for a detailed treatment of ReLU networks . — Under proper randomization of the initialization and the step-size of SGD , with probability 1 the value of s has no impact during training with backprops as a gradient surrogate , see [ 8 , 6 ] . In particular , the set of network parameters such that backprop0 6= backprop1 is negligible and the vast majority of SGD sequences produced by training neural networks are not impacted by changing the value of s = ReLU′ ( 0 ) . These results should in principle close the question about the role of ReLU′ ( 0 ) . Yet , this is not what we observe in practice with the default settings of usual libraries . A surprising experiment and the bifurcation zone : An empirical observation on MNIST triggered our investigations : consider a fully connected ReLU network , and let ( θk,0 ) k∈N and ( θk,1 ) k∈N be two training weights sequences obtained using SGD , with the same random initialization and the same mini-batch sequence but choosing respectively s = 0 and s = 1 in PyTorch [ 26 ] . As depicted in Figure 1 ( left ) , the two sequences differ . A closer look shows that the sudden divergence is related to what we call the bifurcation zone , i.e. , the set S0,1 of weights such that backprop0 6= backprop1 . As mentioned previously this contradicts theory which predicts that the bifurcation zone is met with null probability during training . This contradiction is due to the precision of floating point operations and , to a lesser extent , to the size of deep learning problems . Indeed , rounding schemes used for inexact arithmetics over the reals ( which set to zero all values below a certain threshold ) , may “ thicken ” negligible sets . This is precisely what happens in our experiments ( Figure 1 ) . The role of numerical precision : Contrary to numerical linear algebra libraries such as numpy , which operates by default under 64 bits precision , the default choice in PyTorch is 32 bits precision ( as in TensorFlow or Jax ) . We thus modulated machine precision to evaluate the importance of the bifurcation zone in Section 3 . In 32 bits , we observed that empirically the zone occupies from about 10 % to 90 % of the network weight space . It becomes invisible in 64 bits precision even for quite large architectures , while , in 16 bits , it systematically fills up the whole space . Although numerical precision is the primary cause of the apparition of the zone , we identify other factors such as network size , sample size . Let us mention that low precision neural network training is a topic of active research [ 33 , 20 , 11 , 16 ] , see also [ 28 ] for an overview . Our investigations are complementary as we focus on the interplay between nonsmoothness and numerical precision . Impact on learning : The next natural question is to measure the impact of the choice of s = ReLU′ ( 0 ) in machine learning terms . In Section 4 , we conduct extensive experiments combining different architectures ( fully connected , VGG , ResNet ) , datasets ( MNIST , SVHN , CIFAR10 , ImageNet ) and other learning factors ( Adam optimizer , batch normalization , dropout ) . In 32 bits numerical precision ( default in PyTorch or Tensorflow ) , we consistently observe an effect of choosing s 6= 0 . We observe a significant decrease in terms of test accuracy as |s| increases ; this can be explained by chaotic oscillatory behaviors induced during training . In some cases gradients even explode and learning can not be achieved . The sensitivity to this effect highly depends on the problem at hand , in particular , on the network structure and the dataset . On the other hand the choice s = 0 seems to be the most stable . We also observe that both batch normalization [ 21 ] and—to a lesser degree—the Adam optimizer [ 22 ] considerably mitigate this effect . All our experiments are done using PyTorch [ 26 ] ; we provide the code to generate all figures presented in this manuscript . One important message is that , even if the default choice s = 0 seems to be the most stable , our experiments show a counter-intuitive phenomenon that illustrates the interplay between numerical precision and nonsmoothness , and calls for caution when learning nonsmooth networks . Outline of the paper : In Section 2 we recall elements of nonsmooth algorithmic differentiation which are key to understand the mathematics underlying our experiments . Most results were published in [ 7 , 8 ] ; we provide more detailed pointers to this literature in Appendix A.1 . In Section 3 we describe investigations of the bifurcation zone and factors influencing its importance using fully connected networks on the MNIST dataset . Neural network training experiments are detailed in Section 4 with further experimental details and additional experiments reported in Appendix D . 2 On the mathematics of backpropagation for ReLU networks . This section recalls recent advances on the mathematical properties of backpropagation , with in particular the almost sure absence of impact of ReLU′ ( 0 ) on the learning phase ( assuming exact arithmetic over the reals ) . The main mathematical tools are conservative fields developed in [ 7 ] ; we provide a simplified overview which is applicable to a wide class of neural networks . 2.1 Empirical risk minimization and backpropagation . Given a training set { ( xi , yi ) } i=1 ... N , the supervised training of a neural network f consists in minimizing the empirical risk : min θ∈RP J ( θ ) : = 1 N N∑ i=1 ` ( f ( xi , θ ) , yi ) ( 1 ) where θ ∈ RP are the network ’ s weight parameters and ` is a loss function . The problem can be rewritten abstractly , for each i = 1 , . . . , N and θ ∈ RP , ` ( f ( xi , θ ) , yi ) = li ( θ ) where the function li : RP → R is a composition of the form li = gi , M ◦ gi , M−1 ◦ . . . ◦ gi,1 ( 2 ) where for each j = 1 , . . . , M , the function gi , j is locally Lipschitz with appropriate input and output dimensions . A concrete example of what the functions gi , j look like is given in Appendix A.2 in the special case of fully connected ReLU networks . Furthermore , we associate with each gi , j a generalized Jacobian Ji , j which is such that Ji , j ( w ) = Jacgi , j ( w ) whenever gi , j is differentiable at w and Jac denotes the usual Jacobian . The value of Ji , j at the nondifferentiability loci of gi , j can be arbitrary for the moment . The backpropagation algorithm is an automatized implementation of the rules of differential calculus : for each i = 1 , . . . , N , we have backprop li ( θ ) = Ji , M ( gi , M−1 ◦ . . . ◦ gi,1 ( θ ) ) × Ji , M−1 ( gi , M−2 ◦ . . . ◦ gi,1 ( θ ) ) × . . .× Ji,1 ( θ ) . ( 3 ) Famous autograd libraries such as PyTorch [ 26 ] or TensorFlow [ 1 ] implement dictionaries of functions g with their corresponding generalized Jacobians J , as well as efficient numerical implementation of the quantities defined in ( 3 ) . 2.2 ReLU networks training . Our main example is based on the function ReLU : R→ R defined by ReLU ( x ) = max { x , 0 } . It is differentiable save at the origin and satisfies ReLU′ ( x ) = 0 for x < 0 and ReLU′ ( x ) = 1 for x > 0 . The value of the derivative at x = 0 could be arbitrary in [ 0 , 1 ] as we have ∂ReLU ( 0 ) = [ 0 , 1 ] , where ∂ denotes the subgradient from convex analysis . Let us insist on the fact that any value within [ 0 , 1 ] has a variational meaning . For example PyTorch and TensorFlow use ReLU′ ( 0 ) = 0 . Throughout the paper , and following the lines of [ 8 ] , we say that a function g : Rp → Rq is elementary log-exp if it can be described by a finite compositional expression involving the basic operations + , − , × , / as well as the exponential and logarithm functions , inside their domains of definition . Examples include the logistic loss log ( 1 + e−x ) on R , the multivariate Gaussian density ( 2π ) −K/2 exp ( − ∑K k=1 x 2 k/2 ) on RK , and the softmax function ( exi/ ∑K k=1 e xk ) 1≤i≤K on R K . The expressions x3/x2 and exp ( −1/x2 ) do not fit this definition because evaluation at x = 0 can not be defined by the formula . Roughly speaking a computer evaluating an elementary log-exp expression should not output any NaN error for any input . Definition 1 ( ReLU network training ) . Assume that in ( 1 ) , the function ` is elementary log-exp , the network f has an arbitrary structure and the functions gi , j in ( 2 ) are either elementary log-exp or consist in applying the ReLU function to some coordinates of their input . We then call the problem in ( 1 ) a ReLU network training problem . Furthermore , for any s ∈ R , we denote by backprops the quantity defined in ( 3 ) when ReLU′ ( 0 ) = s for all ReLU functions involved in the composition ( 2 ) . Other nonsmooth activation functions : The ReLU operation actually allows to express many other types of nonsmoothness such as absolute values , maxima , minima , quantiles ( med for median ) or soft-thresholding ( st ) . For any x , y , z ∈ R , we indeed have |x| = ReLU ( 2x ) − x , 2 max ( x , y ) = ( |x − y| + x + y ) , min ( x , y ) = −max ( −x , −y ) , med ( x , y , z ) = min ( max ( x , y ) , max ( x , z ) , max ( y , z ) ) , st ( x ) = ReLU ( x− 1 ) − ReLU ( −x− 1 ) . Definition 1 is thus much more general than it may seem since it allows , for example , to express convolutional neural networks with max-pooling layers such as VGG or ResNet architectures which correspond to the models considered in the experimental section ( although we do not re-program pooling using ReLU ) . The following theorem is due to [ 7 ] , with an alternative version in [ 8 ] . Theorem 1 ( Backprop returns a gradient a.e. ) . Consider a ReLU network training problem ( 1 ) as in Definition 1 and T ≥ 1 . Define S ⊂ RP as the complement of the set { θ ∈ RP , li differentiable at θ , backpropsli ( θ ) = ∇li ( θ ) , ∀i ∈ { 1 , . . . , N } , s ∈ [ −T , T ] } . Then S is contained in a finite union of embedded differentiable manifolds of dimension at most

Summary: The paper demonstrates empirically that on a variety of architectures and datasets, the derivative of ReLU at 0 is evaluated a significant number of times during training in 16 and 32 bit precision. These events are frequent enough such that defining ReLU'(0) to be anything other than 0 (default value in common DNN frameworks) hurts generalization, and sometimes training as well.  The authors perform a variety of investigations around this effect, such as the influence of training set size, network width, depth, batch size, BatchNorm, Dropout.