Numerical influence of ReLU’(0) on backpropagation

1 Introduction . Nonsmooth algorithmic differentiation : The training phase of neural networks relies on firstorder methods such as Stochastic Gradient Descent ( SGD ) [ 14 , 9 ] and crucially on algorithmic differentiation [ 15 ] . The fast “ differentiator ” used in practice to compute mini-batch descent directions is the backpropagation algorithm [ 29 , 4 ] . Although designed initially for differentiable problems , it is applied indifferently to smooth or nonsmooth networks . In the nonsmooth case this requires surrogate derivatives at the non regularity points . We focus on the famous ReLU : = max ( 0 , · ) , for which a value for s = ReLU′ ( 0 ) has to be chosen . A priori , any value in [ 0 , 1 ] bears a variational sense as it corresponds to a subgradient [ 27 ] . Yet in most libraries s = 0 is chosen ; it is the case for TensorFlow [ 2 ] , PyTorch [ 26 ] or Jax [ 10 ] . Why this choice ? What would be the impact of a different value of s ? How this interacts with other training strategies ? We will use the notation backprops to denote backpropagation implemented with ReLU′ ( 0 ) = s for any given real number s.1 1Definition in Section 2 . This can be coded explicitly or cheaply emulated by backprop0 . Indeed , considering fs : x 7→ ReLU ( x ) + s ( ReLU ( −x ) − ReLU ( x ) + x ) = ReLU ( x ) , we have backprop0 [ fs ] ( 0 ) = s. 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . What does backpropagation compute ? A popular thinking is that the impact of s = ReLU′ ( 0 ) should be limited as it concerns the value at a single point . This happens to be exact in theory but for surprisingly complex reasons related to Whitney stratification ( see Section 2 and references therein ) : — For a given neural network architecture , backprops outputs a gradient almost everywhere , independently of the choice of s ∈ R , see [ 7 ] and [ 5 ] for a detailed treatment of ReLU networks . — Under proper randomization of the initialization and the step-size of SGD , with probability 1 the value of s has no impact during training with backprops as a gradient surrogate , see [ 8 , 6 ] . In particular , the set of network parameters such that backprop0 6= backprop1 is negligible and the vast majority of SGD sequences produced by training neural networks are not impacted by changing the value of s = ReLU′ ( 0 ) . These results should in principle close the question about the role of ReLU′ ( 0 ) . Yet , this is not what we observe in practice with the default settings of usual libraries . A surprising experiment and the bifurcation zone : An empirical observation on MNIST triggered our investigations : consider a fully connected ReLU network , and let ( θk,0 ) k∈N and ( θk,1 ) k∈N be two training weights sequences obtained using SGD , with the same random initialization and the same mini-batch sequence but choosing respectively s = 0 and s = 1 in PyTorch [ 26 ] . As depicted in Figure 1 ( left ) , the two sequences differ . A closer look shows that the sudden divergence is related to what we call the bifurcation zone , i.e. , the set S0,1 of weights such that backprop0 6= backprop1 . As mentioned previously this contradicts theory which predicts that the bifurcation zone is met with null probability during training . This contradiction is due to the precision of floating point operations and , to a lesser extent , to the size of deep learning problems . Indeed , rounding schemes used for inexact arithmetics over the reals ( which set to zero all values below a certain threshold ) , may “ thicken ” negligible sets . This is precisely what happens in our experiments ( Figure 1 ) . The role of numerical precision : Contrary to numerical linear algebra libraries such as numpy , which operates by default under 64 bits precision , the default choice in PyTorch is 32 bits precision ( as in TensorFlow or Jax ) . We thus modulated machine precision to evaluate the importance of the bifurcation zone in Section 3 . In 32 bits , we observed that empirically the zone occupies from about 10 % to 90 % of the network weight space . It becomes invisible in 64 bits precision even for quite large architectures , while , in 16 bits , it systematically fills up the whole space . Although numerical precision is the primary cause of the apparition of the zone , we identify other factors such as network size , sample size . Let us mention that low precision neural network training is a topic of active research [ 33 , 20 , 11 , 16 ] , see also [ 28 ] for an overview . Our investigations are complementary as we focus on the interplay between nonsmoothness and numerical precision . Impact on learning : The next natural question is to measure the impact of the choice of s = ReLU′ ( 0 ) in machine learning terms . In Section 4 , we conduct extensive experiments combining different architectures ( fully connected , VGG , ResNet ) , datasets ( MNIST , SVHN , CIFAR10 , ImageNet ) and other learning factors ( Adam optimizer , batch normalization , dropout ) . In 32 bits numerical precision ( default in PyTorch or Tensorflow ) , we consistently observe an effect of choosing s 6= 0 . We observe a significant decrease in terms of test accuracy as |s| increases ; this can be explained by chaotic oscillatory behaviors induced during training . In some cases gradients even explode and learning can not be achieved . The sensitivity to this effect highly depends on the problem at hand , in particular , on the network structure and the dataset . On the other hand the choice s = 0 seems to be the most stable . We also observe that both batch normalization [ 21 ] and—to a lesser degree—the Adam optimizer [ 22 ] considerably mitigate this effect . All our experiments are done using PyTorch [ 26 ] ; we provide the code to generate all figures presented in this manuscript . One important message is that , even if the default choice s = 0 seems to be the most stable , our experiments show a counter-intuitive phenomenon that illustrates the interplay between numerical precision and nonsmoothness , and calls for caution when learning nonsmooth networks . Outline of the paper : In Section 2 we recall elements of nonsmooth algorithmic differentiation which are key to understand the mathematics underlying our experiments . Most results were published in [ 7 , 8 ] ; we provide more detailed pointers to this literature in Appendix A.1 . In Section 3 we describe investigations of the bifurcation zone and factors influencing its importance using fully connected networks on the MNIST dataset . Neural network training experiments are detailed in Section 4 with further experimental details and additional experiments reported in Appendix D . 2 On the mathematics of backpropagation for ReLU networks . This section recalls recent advances on the mathematical properties of backpropagation , with in particular the almost sure absence of impact of ReLU′ ( 0 ) on the learning phase ( assuming exact arithmetic over the reals ) . The main mathematical tools are conservative fields developed in [ 7 ] ; we provide a simplified overview which is applicable to a wide class of neural networks . 2.1 Empirical risk minimization and backpropagation . Given a training set { ( xi , yi ) } i=1 ... N , the supervised training of a neural network f consists in minimizing the empirical risk : min θ∈RP J ( θ ) : = 1 N N∑ i=1 ` ( f ( xi , θ ) , yi ) ( 1 ) where θ ∈ RP are the network ’ s weight parameters and ` is a loss function . The problem can be rewritten abstractly , for each i = 1 , . . . , N and θ ∈ RP , ` ( f ( xi , θ ) , yi ) = li ( θ ) where the function li : RP → R is a composition of the form li = gi , M ◦ gi , M−1 ◦ . . . ◦ gi,1 ( 2 ) where for each j = 1 , . . . , M , the function gi , j is locally Lipschitz with appropriate input and output dimensions . A concrete example of what the functions gi , j look like is given in Appendix A.2 in the special case of fully connected ReLU networks . Furthermore , we associate with each gi , j a generalized Jacobian Ji , j which is such that Ji , j ( w ) = Jacgi , j ( w ) whenever gi , j is differentiable at w and Jac denotes the usual Jacobian . The value of Ji , j at the nondifferentiability loci of gi , j can be arbitrary for the moment . The backpropagation algorithm is an automatized implementation of the rules of differential calculus : for each i = 1 , . . . , N , we have backprop li ( θ ) = Ji , M ( gi , M−1 ◦ . . . ◦ gi,1 ( θ ) ) × Ji , M−1 ( gi , M−2 ◦ . . . ◦ gi,1 ( θ ) ) × . . .× Ji,1 ( θ ) . ( 3 ) Famous autograd libraries such as PyTorch [ 26 ] or TensorFlow [ 1 ] implement dictionaries of functions g with their corresponding generalized Jacobians J , as well as efficient numerical implementation of the quantities defined in ( 3 ) . 2.2 ReLU networks training . Our main example is based on the function ReLU : R→ R defined by ReLU ( x ) = max { x , 0 } . It is differentiable save at the origin and satisfies ReLU′ ( x ) = 0 for x < 0 and ReLU′ ( x ) = 1 for x > 0 . The value of the derivative at x = 0 could be arbitrary in [ 0 , 1 ] as we have ∂ReLU ( 0 ) = [ 0 , 1 ] , where ∂ denotes the subgradient from convex analysis . Let us insist on the fact that any value within [ 0 , 1 ] has a variational meaning . For example PyTorch and TensorFlow use ReLU′ ( 0 ) = 0 . Throughout the paper , and following the lines of [ 8 ] , we say that a function g : Rp → Rq is elementary log-exp if it can be described by a finite compositional expression involving the basic operations + , − , × , / as well as the exponential and logarithm functions , inside their domains of definition . Examples include the logistic loss log ( 1 + e−x ) on R , the multivariate Gaussian density ( 2π ) −K/2 exp ( − ∑K k=1 x 2 k/2 ) on RK , and the softmax function ( exi/ ∑K k=1 e xk ) 1≤i≤K on R K . The expressions x3/x2 and exp ( −1/x2 ) do not fit this definition because evaluation at x = 0 can not be defined by the formula . Roughly speaking a computer evaluating an elementary log-exp expression should not output any NaN error for any input . Definition 1 ( ReLU network training ) . Assume that in ( 1 ) , the function ` is elementary log-exp , the network f has an arbitrary structure and the functions gi , j in ( 2 ) are either elementary log-exp or consist in applying the ReLU function to some coordinates of their input . We then call the problem in ( 1 ) a ReLU network training problem . Furthermore , for any s ∈ R , we denote by backprops the quantity defined in ( 3 ) when ReLU′ ( 0 ) = s for all ReLU functions involved in the composition ( 2 ) . Other nonsmooth activation functions : The ReLU operation actually allows to express many other types of nonsmoothness such as absolute values , maxima , minima , quantiles ( med for median ) or soft-thresholding ( st ) . For any x , y , z ∈ R , we indeed have |x| = ReLU ( 2x ) − x , 2 max ( x , y ) = ( |x − y| + x + y ) , min ( x , y ) = −max ( −x , −y ) , med ( x , y , z ) = min ( max ( x , y ) , max ( x , z ) , max ( y , z ) ) , st ( x ) = ReLU ( x− 1 ) − ReLU ( −x− 1 ) . Definition 1 is thus much more general than it may seem since it allows , for example , to express convolutional neural networks with max-pooling layers such as VGG or ResNet architectures which correspond to the models considered in the experimental section ( although we do not re-program pooling using ReLU ) . The following theorem is due to [ 7 ] , with an alternative version in [ 8 ] . Theorem 1 ( Backprop returns a gradient a.e. ) . Consider a ReLU network training problem ( 1 ) as in Definition 1 and T ≥ 1 . Define S ⊂ RP as the complement of the set { θ ∈ RP , li differentiable at θ , backpropsli ( θ ) = ∇li ( θ ) , ∀i ∈ { 1 , . . . , N } , s ∈ [ −T , T ] } . Then S is contained in a finite union of embedded differentiable manifolds of dimension at most

Summary: The ReLU function is not differentiable at 0 which has a mathematical impact on ReLU neural networks. Nevertheless they are optimized using gradient based methods. The folk wisdom is that singular points only occur with zero probability and therefore the issue is irrelevant. This paper challenges this folk wisdom and identifies that the choice of the derivative at 0 does have a tangible impact on the optimization and classification performance.