On the numerical reliability of nonsmooth autodiff: a MaxPool case study
Authors: Ryan Boustany
Venue: Accepted by TMLR ()

Abstract: This paper considers the reliability of automatic differentiation for neural networks involving the nonsmooth MaxPool operation across various precision levels (16, 32, 64 bits), architectures (LeNet, VGG, ResNet), and datasets (MNIST, CIFAR10, SVHN, ImageNet). Although AD can be incorrect, recent research has shown that it coincides with the derivative almost everywhere, even in the presence of nonsmooth operations. On the other hand, in practice, AD operates with floating-point numbers, and there is, therefore, a need to explore subsets on which AD can be {\em numerically} incorrect. Recently, \cite{bertoin2021numerical} empirically studied how the choice of $\ReLU'(0)$ changes the output of AD and define a numerical bifurcation zone where using $\ReLU('0) = 0$ differs from using $\ReLU'(0) = 1$. To extend this for a broader class of nonsmooth operations, we propose a new numerical bifurcation zone (where AD is incorrect over real numbers) and define a compensation zone (where AD is incorrect over floating-point numbers but correct over reals). Using SGD for training, we found that nonsmooth MaxPool Jacobians with lower norms maintain stable and efficient test accuracy, while higher norms can result in instability and decreased performance. We can use batch normalization, Adam-like optimizers, or increase precision to reduce MaxPool Jacobians influence.

Published in Transactions Machine Learning Research (06/2024)
On the numerical reliability of nonsmooth autodiff: a Max- Pool case study
Ryan Boustany
ryan. www..boustany@uut-ccpiioeee
Toulouse School of Economics
Université de Toulouse
Thales LAS France
Reviewed on OpenReview: https: // openreview. net/ forum? id= 142ms In InVfp
Abstract
This paper considers the reliability of automatic differentiation for neural networks involving the nonsmooth MaxPool operation across various prccision levels (16, 32, 64 bits), architectures (LeNet, VGG, ResNet), and datasets (MNIST, CTFARIO, SVHN, ImageNet). Although AD can be incorrect, recent rescarch has shown that it coincides with the derivative almost everywhere, even in the presence of nonsmooth operations. On the other hand, in practice, AD operates with floating-point numbers, and there is, therefore, need to explore subsets on which AD can be numerically incorrect. Recently, Bertoin et al. (2021) empirically studied how the choice of ReLU (0) changes the output of AD and define a numerical bifurcation zone where using RcLU'(0) 0 differs from using RcLU'(0) 1. To extend this for broader class of nonsmooth operations, we propose a new numerical bifurcation zone (where AD is incorrect over real numbers) and define compensation zone (where AD is incorrect over floating point numbers but correct over reals). Using SGD for training, we found that nonsmooth MaxPool Jacobians with lower norms maintain stable and efficient test accuracy, while higher norms result in instability and decreased performance. We can use batch normalization, Adam-like optimizers, or increase precision to reduce MaxPool Jacobians influence.
1 Introduction
Nonsmooth neural networks are trained using optimization algorithms (Bottou et al. 2018
Davis
ct al. 2018) based on backpropagation and automatic differentiation (AD) (Speelpenning
1980
Rumelhart et al.] 1986b Baydin et al. 2018). AD is crucial tool in contemporary learning
architectures as it allows for fast differentiation (Gricwank & Walther 2008 Bolte et al. 2022
It is implemented in popular machine learning libraries such as TensorFlom AAadi et al. 2016
Poroh (Paszke et al. 2019), and Jax (Bradbury et al. 2018). Although the validity domain
of AD is theoretically limited to smooth functions (Griewank & Walther 2008), it commonly
used for nonsmooth functions (Bolte et al.) 2022 2021b Bertoin et al. (021). The behavior of
nonsmooth AD has been investigated in previous studies (Griewank & Walther
2008 Griewank
2013 Griewank et al., 2016
Barton et al.) 2018 Kakade & Lee.
2018
Griewank Rojas 2019
Griewank & Walther 2020
Bolte & Pauwels 2020a Bolte et al.
2022

---

Published in Transactions Machine Learning Research (06/2024)
MaxPool: a nonsmooth operation The MaxPool operation, introduced by Yamaguchi et al. 1990p, is commonly used in convolutional ncural networks (CNN) for image classification (Krizhevsky & Hinton 2010 Krizhevsky et al. 2012 Zeiler & Fergus 2014 LeCun et al. 2015). MaxPool reduces the spatial dimensions of a feature map by selecting the maximum value within specific patches. However, when applied to uniform pixel values, MaxPool can cause nonsmoothness, especially at image edges where identical pixels can be chosen arbitrarily. In such cases, different choices of MaxPool's nonsmooth Jacobians bear a variational sense. See Appendix AA. for an illustration. In this paper, the term MaarPol--drrioe program refers to specific choice of a MaxPool nonsmooth Jacobian.
Various types of nonsmooth AD errors: We carry out a PyTorch (Paszke et al. 2019 experiment to investigate the autodiff behavior of the nonsmooth max function, defined as max: + maxisis4 D3 E R. We implement two max programs (max1 and max2) with different derivative implementattins. For example, the max function is not differentiable at 1, and autodiff returns 01,0,0,0 0) for maxı and (0.25, 0.25, 0.25, 0.25) for maxę (see Appendix A.1 for more dctails). Let zero be a program as follows: zero: t + maxi (t x x) max2(t x x). The AD output of zero is denoted by zero'. As mathematical functions, both max, and maxę output the same value and zero always outputs O. However, we observe an unexpected behavior when using AD and floating-pon point numbers: zero'(() (t) A 0 for some LE R.
Table 1: Overview of numerical AD errors for the zero program with 32 bits precision.
zero' (t)
-10
-10-2
-10-1
0
10
102
103
T1 = 21,2,3,4)
0.0
0.0
0.0
-1.5
0.0
0.0
0.0
x2 = (1,4,11,,14,,11,,,
10 7
10 7
10"
10
7
10
7
10
7
10
7
For 3i, observe significant error at t 0 where the computed derivative, zero'(O), is -1.5, deviating from the correct derivative value. In contrast, for x2, which often appears tasks such as image classification (refer to Appendix A.2), theoretical calculations predict zero'(t) (t) = 0 for any E R. However, discrepancies emerge when using moating-point arithmetic, as illustrated by AD results. Specifically, across all values listed in Table 1 zero'(l) approximates to 5.96 x 10- (rounded to 10-7 in the table), which is near the computational precision limit of 32-bit systems. This phenomenon occurs due to numerical arithmetic limits. Typically, t denotes a neural network parameter and x represents an input image pixel area where the pixel values are identical, such as in the MNIST dataset (see Appendix A.2 The behavior observed in Table are not due to the nonsmooth multivariate nature of the max function. Similar phenomena can also be seen when the univariate ReLU operation is used to compute max. In contrast, we observe no errors near machine precision using NormPool-a nonsmooth multivariate function that computes the Euclidean norm. Furthermore, reproducing Table with zero: + (tLU,(t) - ReLU2(t) where ReLU'(0) = 0 and ReLU' = 1, also shows no AD errors near machine precision. These observations suggest that minor AD errors may stem from the intrinsic properties of the max function. Thus, our study primarily examines max and MaxPool operations. For more details, plcase refer to Appendix A.3 and Appendix A.4
Reals vs doati numbers: Over the real numbers, AD computes derivatives for nondif- ferentiable functions, except Lebesgue measure-zero subset of inputs (Bolte & Pauwels 2020a b).
2

---

Published in Transactions Machine Learning Research (06/2024)
However, as indicated in Table | the use of loating-point arithmetic can expand the subsets where AD yiclds incorrect results. In Section 3 we propose two subsets of network parameters where AD numerically fails: new bifureation zone, characterized by significant AD amplitude variations, and a compensation zone, where minor amplitude variations occur near machine precision due to rounding schemes in inexact arithmetic over reals (e.g., non-associativity) Our experiments show that in 64-bit network using MaxPool, the compensation zone covers the entire parameter space. In a 32-bit network, both compensation and bifurcation zones exist, while a 16-bit setting, the bifurcation zone dominates the entire parameter space.
examine the influence of different nonsmooth MaxPool Jacobians on learning processes. We find that nonsmooth Jacobians with low norms produce test accuracies comparable. Conversely, Jacobians with high norms tend to reduce test accuracy, primarily due to training instability or backprop-related issues, In 16-bit precision setting, which is a significant focus of research (Vanhoucke et al.) 2011 Hwang & Sung 2014 Courbariaux et al. 2015 Gupta et al., 2015), the effects of these Jacobians are more marked and vary depending on the network architecture, dataset, and the precision level employed. Additionally, we observe that the inclusion of batch normalization (Toffe & Szegedy 2015) and the use of the Adam optimizer (Kingma & Ba. 2014) help to alleviate these negative impacts. All experiments were conducted using PyTTrch (Paszke al. 2019), and our source code is available publicly !
Implications for learning dynamics: In Section
Related works and contributions: Recent works indicate that for a broad class of programs employing nonsmooth functions, AD is incorrect primarily on a Lebesgue measure-zero subset of the program's 's input domain (Bolte & Pauwels 2020a Lee et al. 2020). However, practical inputs are typically machine-representaall.. In this context, Lee et a.l. (0023) investigated AD correctness in neural networks with machinne-representable parameters, specifically excluding networks with MaxPool. Additionally, Bertoin et al. (2021) explored the impact of the ReLU'(0) on AD and training, identifying a bifurcation zone in ReLU networks where AD is incorrect. These studies, however, do not address situations where AD is incorrect over floating-point numbers yet correct over real numbers, such as those noted the last line of Table [ To bridge this gap, we propose a new bifurcation zone and introduce the concept of a compensation zone. Our research evaluates the reliability of automatic differentiation in MaxPool neural networks across various precision levels, examining how the effects of the compensation zone vary with network architecture, independent of the nonsmooth functions' dimensionallity (whether univariate multivariate). We also delve into how nonsmooth MaxPool Jacobians influence the stability and performance of neural network training.
Organization of the paper: In Section 2 we discuss the elements of nonsmooth backpropagation and define three subsets of network parameters the bifurcation zone, compensation zone, and regular zone. We also examine the implications of nonsmooth MaxPool Jacobians for backpropagation, based on Bolte & Pauwels (2020a b). In Section 3. we define and discuss the numerical bifurcation and compensation zones, including factors that influence their significance. In Section 4 we present detailed experiments on neural network training. For additional findings and experiments, please refer to Appendix C
3

---

Published in Transactions Machine Learning Research (06/2024)
2 MaxPool neural networks and nonsmooth AD
2.1 Preliminaries and notations
In supervised training for neural networks, we work with a set of training data (r,, W:) where cach T; is an input and Y4 its matching label. A neural network, through its function f, uses parameters to generale predictions ÿ = f(x;, 0). The difference between these predictions and the actual labels is measured by a loss function L. The aim is to reduce this discrepancy across the training set by minimizing an empirical loss function L such as:
1
min BERP
Σ l(o ys). L(O) = N
(1)
For all [I,...,N] and 0 E RP, Equation can be expressed with ((gs, 4(0), where 4 RP + represents a composition of H elementary functions as follows:
(0) = H - 1...-.....9.1110))
(2)
Equation (2) models common neural network types, including feed-forward (Rumclhart et al. 1986a). convolutional (LeCun et al. 1998). and recurrent networks (Hochreiter & Schmidhuber 1997). For more concrete example, please refer to Appendix A.2 in Bertoin et al. (2021). We focus on elementary functions that are locally Lipschitz and semialgebraic, commonly found in nonsmooth neural networks (Bolte & Pauwels 2020a b). Functions gi,j include operations such as linear transformations, ReLU, MaxPool, convolution with filters, and softmax for classification.
2.2 Nonsmooth AD framework
Training nonsmooth neural networks (Bolte & Pauwels 2020a Bolte et al.) 2021b 2022 2021a
Davis et al. 2020) challenging due to the need to compute subgradients from Equation (1). Major machine learning tools such as TensorFlow (Abadi et al. 2016), PyTorch (Paszke et al.
2019
and Jax (Bradbury et al.) 20188) address this issuc using automatic differentiation, referred to here as backprop (Rumelhart et al. 1986b; Baydin et al., 2018). They apply differential calculus to nonsmooth items, often replacing derivatives with Clarke Jacobians (Clarke 1983). Given locally Lipschitz continuous function F RPP + R2, the Clarke Jacobian of F is defined as:
Jac F(x) conv { lim Jac F(xx) 2h diff Th +00 x +00
(3)
where diff represents the full measure set where is differentiable and Jac is the standard Jacobian of F. A selection v in Jac F is a function Rp + B2xa such that, for all x RP, ((x) E Jac F(x). If C²,, the only possible selection is v Jac
Definition (Calculus model, programs and nonsmooth AD) Let be a composition func- tion evaluated at A E IR,, as specified in Equation (2) A program P that executes can be described through a sequence of subprograms such as:
Elementary programs: {0, } such that 1(0) 9H 9H 10..... 1 (0.....1(())
Derived programs: { where each vz(w) E Jac (y(w) at point 9j-1 ..... .. (0).
4

---

Published in Transactions Machine Learning Research (06/2024)
Then, the backprop algorithm automates applying differential calculus rules follows:
(a)) uu (gH-1 o..-1 g1 wH-1 (gH-2 gi()))
(4)
In practice, AD librarics AAbadi al. 2016 Paszke et al. 2019
Bradbury et al. 2018 implement
dictionaries (see for e.g- Griewank & Walther (2008); Bolte et al. (2022) containing conjointly
elementary programs and derived programs which elficiently computes the quantities defined in Equation (4
Remark 1 As seen in Section 1 with the zero program, various programs can implement a unique composition function I. Each nonsmooth elementary program Oj in the composition (sce Definition I can be associated with different derived programs ly. Specifically, for any j = 1,..., ,....., H and -j-10-1100 ....0 gg gi (f), all selections vy(w) from the Clarke Jacobian of gj gj(w) bear variational sense.
Example The Clarke subdifferential of (t) = max(0, t) at is 0 for t f<0, 0, 1 for t > 0, and the interval [0, I] for t = 0. All ReI.U-drrvee program that implements ReLUU((0) with E [0, 1] can be used for backprop.
Definition 2 (Backprop set) Let denote a composition function evaluated at 0 R", as specified in Equation (2) We define ((0) as the function that encompasses the set of all possible backprop outputs through all programs implementing ((0) as in Definition
(() = ( P is program implementing ((0))
55)
Remark 2 For a composition function composed by C elementary programs {o;) ((0) is a singelton for all e RR. For locally Lipchitz semialgebraic (or definable) elementary programs {g:1/=1: Equation
M returns an element within the backprop set.
Remark 3 The chain rule, essential for AD, often fails with Clarke subgradients. Hence, the backprop set might differ from the Clarke subdifferential (Clarke 1983). For example, the Clarke subdifferential of 2ReLU(x) - ReL.U(-x:) at x 0 is 2], whereas backprop outputs (with (o)U'(0) = 0).
2.3 Network parameters subsets
Recently, Bertoin et al. (2021) studied the bifurcation zone in ReLU networks, characterized by network parameters at which the output of AD diverges between ReLU'(0) = 0 and ReLU'(0) = 1. However, their study did not cover the scenario where backprop theoretically computes singleton, but AD inaccurately computes it due to floating-pois ppint arithmetic. From our knowledge, this phenomenon appears when comparing the output of AD across different derivatives implementations of MaxPool. To address this gap, introduce the concept of the compensation zone.
Definition 3 (Compensation, bifurcation and regular zones) For cach 1, 1....,,: denote a composition function evaluated at Rp and J;( denote the backprop set associated detailed in Definition
N, let 4
We define the following network parameters subsets of IR?:
On 0 E R Vi, E ( (1,...,,] {1...., H3, Jac gi (w) is singleton
(6)
θc (0 I OR a Vic (1,...,N,, (() singlcton)
(7)
OB = ( E R. OR Bie (1,...,, such that (i( is not a singleton),
(8)
5

---

Published in Transactions Machine Learning Research (06/2024)
where 9i.j-1 ... ... gi,(0)), OR is the regular zone, Oc the compensation zone and OB the bifurcation zonc.
The mathematical tools of Proposition 1 are conservative fields developed in Bolte & Pauwels (2020a). This proposition implies that theoretically (assuming exact arithmetic over the reals), the backprop set is almost everywhere a singleton. The proof is given in Appendix B
Proposition 1 Given subsets OR OB, and Oc in R' defined in Definition 3 the following properties hold:
θ Rg O and Oc form a partition of R
θ is a Lebesgue mull measure subset.
Remark 4 (Backprop returns gradient a.e.) Let A E R2 and P be a program implementing Then backprop(P(() = VI(0) almost everywhere.
a composition function ((0) as in Definition [
2.4 MaxPool-derived programs
Definition (Clarke Jacobian of matrix's maximum function) Lot be a m x n rcal ma- trix and Fs be function such that (x maxisism, Xis E IR, where s == m X n denotes the size of X. The Clarke Jacobian of Fg at the point X is:
Jac Fx(X) = conv
U
Eij
(9)
(i.)EA(X)
where A(X) == {(t,) E (1,..., m] x {l....,
Fx(X) = Xij} is the active set and Eij is an m xn matrix with all entries equal to 0 except for the (i,j)-th entry which is 1.
n)
Definition 5 (MaxPool operation) Let X R2xq be real matrix, and s m mxn be the size of a pooling window such that p2 m and q 2 n. For each i E {O...., _ - 1} and j E {o..... 1}, we define a submatrix Xij of X, of size x as follows:
Xi. (XM m k mm (i 1), (x=+)
(10)
where and are the indices of the entrics in X, in the lexicographic order. The MaxPool operation output matrix Y x where Yi- Fx(Xi.) for all i (O...., L 1} and j E ,O,..., -1.. Finally, the MaxPool Clarke Jacobian at point X, denoted as Jace MaxPool(X). can be obtained by replacing each submatrix Xij in X with Jac Fx(X.,).
Definition 6 (MexPool-derried programs) Define Xi,j Raxx a submatrix of X (Defini- tion 5 from which we derive MaxPool programs based on the Clarke Jacobian:
Native: Chooses the first index (11,11) from the active set A(Xig) and outputs Enji
Autograd librarics use this implementation.
Takes all indices from A(X,,), averaging them as TA E(.NEA(()))) Pg.- We called it "minimal' as it yields the smallest norm element within Equation g
Minimal:

---

Published in Transactions Machine Learning Research (06/2024)
Hybrid: A blend of native and minimal, parameterized by B 0:
(IA(2.0) Σ
1
(1 8) Biys + 3. (A(()))) (k)))E((XL,)
Ekd
Remark 5 The hybrid MaxPool-derived program is a selection of the MaxPool Clarke Jacobian for B [0, 1] and a selection a conservative Jacobian approach for other 3 values, as outlined Bolte & Pauwels [2020a).
The chain rule essential for AD, but it often faces challenges with Clarke subgradients. The backprop set defined in Definition 2 does not consistently qualify as a Clarke subdifferential. Therefore, in the following sections of this paper, we will also investigate the impact of using a conservative Jacobian (3 > 1) on the learning and training processes. This investigation is important conservative Jacobians have variational interpretation within the framework of nonsmooth AD (see Bolte & Pauwels (2020a)).
3 A more general numerical bifurcation zone
In this section, we explore numerically subsets of network parameters for neural networks involving MaxPool operations across various floating-point precisions. We find that the numerical bifurcation zone identified by Bertoin et al. [2021) does not apply to scenarios involving MaxPool-derived programs. Specifically, both the max function and MaxPool contribute minor AD errors that appear with Moating-point numbers but not with real numbers. As a result, we suggest a new numerical bifurcation and define compensation zone using two distinct methods: one with nondeterministic GPU computations and the other with ReLU-derived programs. These methods are based on the notations and frameworks discussed in Scctions 2.1 and 2.2
3.1 A numerical criteria for the bifurcation and compensation zone
Numerical bifurcation zone for ReLU networks: Recently, Bertoin et al. 20021 investigated
numerical bifurcation zone So1 specific ReLU-derived programs. For each I,....N: N, two programs implement a same function 4: R (using RcLU'(0) = 0) and R! (using RcLU'(0) = 1).
The bifurcation zone So1 is defined as:
So1 = - E IRP Bie (1...., N), backprop(A.](()) + (ackprop|R/](()-
(11)
Definition 7 (Backprop variation) Let (By)GEN be a sequence of mini-batches. where each batch size [B,l falls within (1,...,N}. Consider p= [P and Q = (Qi) as two neural network implementations using different nonsmooth-derivee programs (e.g., ReLU MaxPool). Each P and Q; computes a composition function 4, The backprop variation between and Q over M experiments with random parameters 10m)Me is defined as:
Dm.g(P.Q)
(P,Q) = backprop
P.(Om) sc By
Σ
backprop
Σ Q:(Om))
(12)
ic

---

Published in Transactions Machine Learning Research (06/2024)
A 32 bits MNIST experiment: Let P and Q be programs for a LeNet-5 network on MNIST, using native and minimal MaxPool programs, respectively. For a sanity check, let P be a copy of P. We compute the backprop variation (as in Definition 7 between P and P and between P and Q. We control all sources of divergence in our implementation using deterministic computation. Results are reported in Figure and the experiment was run on CPU under 32 bits precision.
10a
Proportion (96)
Proportion
10-4
10
loggolmagnitude)
[mgg[mmagnt
(a) D (P.Q(
(b) Dmng(Pp P)
Figure 1: Histogram of backprop variation Dm.g for LeNet-5 on MNIST (128 32-bit precision, comparing with P and P with Q over M = 1000 experiments.
size) at
We observe no variation in backpropagation between P and P, indicating controlled sources of divergence. This observation contrasts with the expectations from Proposition I which predicts variation between P and Q; we find (m, Q) > across all m. q. We identify two types of variations: minor ones, comprising 98. 78% of parameters, which align with machine precision 32 bits (between 10-8 and 10-"), and major ones, peaking at 10-3 and accounting for 1.22% of parameters. Notably, these findings differ significantly from the backprop variations observed with ReLU-derived programs as shown in Figure 3 where we either see significant divergences or none. Consequently, this section focuses on analyzing backprop variations in MaxPool-derived programs propose a new bifurcation zone.
An heuristic for the numerical bifurcation zone: In Figure ! we identify two types of backpropagation variations: one potentially arising from numerical bifurcations and another due to floating-point arithmetic errors, which we refer to as compensation errors. To establish critcria for proposing new numerical bifurcation zone for nonsmooth-derived programs, compare these observed backpropagation variations with known sources, such as GPU nondeterminism and variations from U-LUrved programs at 16 and 32-bit precision. This method allows us to differentiate between numerical bifurcations and compensation errors without presuming distinct zones. We denote floating-por point precision by w and consider various neural networks like LeNet-5, VGG, ResNet for our analysis.
threshold with nondeterministic GPU computations: We set a threshold The for the maximum backprop variation due to nondeterministic GPU computations (refer to Appendix A.6. for more dctails):
feo = max 15m5M,15g
(P,((,P)
(13)
where P and P compute a neural network f using the same nonsmooth-derived program, for example, ReLU'(0) = 0 or minimal MaxPool. See Figure 2 for an illustration. We observe no variation at 16, with PyTorch's nondeterministic GPU operations disabled. Additionally, !
8

---

Published in Transactions Machine Learning Research (06/2024)
can be interpreted as an upper bound on the expected backprop variation when repeatedly running the same program under nondeterministic conditions.
12
troportinn (%6
Proportion (%)
Propoctior (96)
131
16
kggalmagekus))
(a) f.16 =
(b) f.32 = 1.11 x 10-7
(c)
7 1.55 10-16
Figure Histogram of backprop variation under nondeterministic GPU operations, where f is a LeNet-5 network on MNIST with batch size 128 for M = 1000 experiments.
A threshold with ReLU-derived programs: For ReLU-derived programs, we define R (with ReLU 0) and R (with ReLU = 1) as two programs implementing same network f under deterministic GPU operations. We introduce threshold 7j. for backprop variation:
T).w min
(Dmg(R), R'): Pma(RR) RR) 0].
(14)
I5mSMI15
72 can be interpreted as lower bound on the error we expect to make when running two different ReLU-derived programs of the same function.
Remark 6 We do not consider 72 in the context of MaxPool-derived programs, as it is anticipated that 72 will approximate machine precision values in Figure
Properton 190
15Q)
Proportion 15Q)
(a) 1,
3.39 x 10-6
(b) 77.92 4.81 10-5
(c) 77,64
Figure 3: Histogram of backprop variation with ReLU-derived programs, where S is LeNet-5 network on MNIST with batch size 128 for M 1000 experiments.
Figure 3 illustrates two types of backpropagation variations with ReLU-derived programs: significant divergences or none at all, contrasting with phenomena observed with MaxPool-derived programs. These divergences may suggest the presence of numerical bifurcation zone. Additionally, variations from nondeterministic GPU computations, as shown in Figure 2 correspond to minor variations near machine precision, similar to those seen in Figure [ We propose establishing numerical bifurcation zone, applying different thresholds for various precisions to accommodate hardware constraints.

---

Published in Transactions Machine Learning Research (06/2024)
Criteria 1 (Numerical bifurcation zone) For neural network f and floating-point precision w, let Tfw be a fixed threshold (for c.g tww k/w). The numerical bifurcation zonc is defined as:
S(TFw) = { E RR lF E N), (P) > c OB. (15)
Here, Pj and Qi are programs implementing f using different nonsmooth-derived programs.
Table in Appendix A.6 lists threshold values for various networks and datasets across 16-bit, 32-bit, and 64-bit precisions. These thresholds are numerical guides and fluctuate based on the initial network parameters, datasets, and architecture. The characteristics of the compensation zone depend on the neural network's structure rather than the nature (univariate or multivariate) of the nonsmooth elementary programs defined in Definition | For example, computing MaxPool using ReLU programs can lead to compensation errors, as detailed in Table | (see Appendix (A.3) In convolutional networks like VGG or ResNet, computing MaxPool with ReLU functions using the formula max(x, y) = (x + y) + (ReLU(x) Reni--ReLU((pp)) y)) )))) (ReLU(y) ReLU(-x) does not align with the bifurcation zone proposed by Bertoin et al. (2021) Conversely, using NormPool-4
nonsmooth multivariate function calculating the Euclidean norm avvodd such compensation errors. Further details can be found in Appendix A.4
3.2 Volume of the numerical bifurcation zone
We employed Monte Carlo sampling to estimate the volume of the numerical bifurcation zone for various networks, adhering to Criteria [ Thresholds T/,16 T/32, and 7,64 were consistently applied across all networks, detailed in Appendix A.6.3 with reference to Equations (13) and 14..
Experimental Setup: We generated a set of network parameters {0m) randomly using Saiming-Uniform initialization (He et al.) 2015), with M 1000 experiments conducted. Sub- sequently, we iterated over the entire CIFARIO dataset to estimate the proportion of 0m within the numerical bifurcation zone S as defined in Criteria [ (referenced in Equation 188 and the
proportion of affected mini-batches (detailed in Equation (19
Impact of Roating-point precision: Using the VGG11 model on the CIFAR10 dataset, we evaluated the volume of S across different precision levels. The results revealed that at 16-bit and 32-bit precision, all parameters resided within S, whereas at 64-bit precision, none did. This variance demonstrates the significant role of precision in the effects of backprop with MaxPool- derived programs. Notably, the impact on mini-batches was substantial, with 46% at 32 bits and 100% at 16 bits, underscoring the influence of precision on the computational outcomes.
Table 2: Impact of S according to floating foating-point point precision using VGG11, on CIFAR10 dataset. and M = 1000 experiments. The first line represents network parameters 0m in S, while the second measured the proportion of affected mini-batches falling in S.
Floating-point precision
16 bits
32 bits
64 bits
Proportion [0m]
100%
100%
0%
Proportion of impacted mini-batches
100%
46.67%
0%
We also investigated the influence of mini-batch size on the proportion of affected mini-batches within the numerical bifurcation zone S using the VGG11 model on the CIFAR10 dataset. Our
10

---

Published in Transactions Machine Learning Research (06/2024)
findings indicate that larger mini-batch sizes correlate with an increased proportion of impacted mini-batchos at 32-bit precision. However, at 64-bit precision, no paramcters were observed to fall into S (as shown in Figure 4 Additionally, variations in network Sopth-examined across VGG variants 11, 13, 16, and 19 1iddid not significantly alter the impact on mini-batches at 16-bit and 32-bit precisions. Notably, the introduction of batch normalization markedly increased the number of affected mini-batches at 32-bit precision.
Precision
100
Proportion (%)
80
40
0 VGG11 VGG13 VGG16 VGG19
64
256
512
1024
False Batch-normalization
True
Networks
Mini-batch size
Figure 4: Impact of different size parameters on the proportion of affected mini-batches (see Equation 19 using CIFAR10 dataset. First: Different VGG network sizes. Second: VGG11 with varying mini-batch sizes. Third: VGG11 with and without batch normalization.
4 Impact on learning
4.1 Benchmarks and implementation
Datasets and architectures:
We train neural networks to investigate the impact of numerical
effeecs outlined in Section 3 Our experiments used CIFAR10 (Krizhevsky & Hinton 2010), MNIST
LeCunn ct al. 1998) and ImageNet (Deng et al.
2009) datasets. We test various network architectures
including VGG11 (Sionnyan & Zisserman
2014h. ResNet (He et al. 2016). and LeNet (LeCun
ct
1998
Details are available in Appendix C.1
Training settings: The default optimizer is SGD. Conducted on PyTorch and Nvidia V100 GPUs, we define mini-batch sequences (Bg)gEN with sizes [Bg C f1,...,NY N}, where ag > is the learning rate for each mini-batch q. Each program P, in P= ( ( implements a function 4 (as in Definition I.. The SGD algorithm updates network parameters 0g by:
Qg |B
0 -
0g.P
backprop(PP[(0.pp
(16)
ie
with > indicating the step-size parameter.
4.2 Effect on training and test errors
We further investigate the effect of the phenomenon described in Section in terms of learning using the CIFAR10 dataset (Krizhevsky & Hinton 2010) and the VGG11 architecture (Simonyan & Zisserman, 2014h. This was performed at 16-bit and 32-bit precisions with various values, repeating each configuration ten times with random initialization. The results are depicted in Figure
5. To confirm our findings in alternative settings, we also use the MNIST (LeCun et al.) 1998) and
ImageNet (Krizhevsky et al. 2012) datasets and the ResNet and ResNet50 architectures
(He
11

---

Published in Transactions Machine Learning Research (06/ (06/2024)
et al. 2016). Additional details on the different architectures and datasets experiments are found in Appendix
Training effect with 16-bit: For 8 values greater than 103, we observe training instability characterized by exploding gradients, unaffected by the presence batch normalization. Conversely, stable and efficient test accuracies are maintained for 3 values within the set {0, 1, 10, 100).
Training effect with 32-bit: When 3 values are large, such as 10 the training process may become unstable, exhibiting oscillations and sudden fluctuations in the learning curve. This instability can oceur if batch normalization is not used. However, incorporating batch normalization with high 3 values helps stabilize the training dynamics, enhances test data accuracy, and prevents gradient explosion.
1.0
Batch norm
B
Test accuracy
0.92
True
False
Train loss
0.8
0
0.6
1
0.91
0.4
100
0.2
1000
0.90
10000
10
100
1000
10000
0
50
100
150
200
B
epoch
(a) 16-bit
(b) 16-bit, with batch normalization
0.925
Batch norm False
Test Test accuracy
0.920
1.5
0
True
Train loss
1
1.0
10
100
0.910
1000
10000
0.905
0.0
100
1000
10000
0
50
100
150
200
epoch
(c) 32-bit
(d) 32-bit, without batch normalization
Figure 5: Training VGG network on CIFAR10 with SGD. We performed ten random initializations for each experiment, depicted by the boxplots and the filled contours (standard dcviation).
Training and weight differences: We trained seven VGG11 networks {P:) at 32-bit precision on CIFAR10 for 200 epochs, using 128-size mini-batches, fixed lcarning rate for cach mini-batch g ag 1.0, and step-size parameter 7 E [0.01, 0.012). All networks, starting with the same parameters, varied with hybrid MaxPool { Using nondeterministic GPU computation, we measured epsch-wise backpropagation differences between Po and the others, observing parameter variations and test accuracies. Variations and accuracies for 3 < 10 were consistent, showing I's minimal impact for practical scenarios. At 8 104, significant divergences and test accuracy drop were noted, indicating that high 3 values could destabilize training due to exploding gradients.
Recommendation for practitioners: We found that the choice of B significantly influences neural network training. Specifically, large 3 values destabilize training and reduce test accuracy.
12

---

Published in Transactions Machine Learning Research (06/2022)
0.9
10
0.7
10a
Tect accuracy
8 =
8
0.5
==10
10a
P 102
Weightt
p 10
101
0s 1
B
vs
10a
0.2
10
10*
0
0.0
0
25
50
75
100
125
150
175
200
25
50
75
100 Epoch
125
150
175
200
Epoch
Figure 6: Left: Difference between network parameters (L norm) at each epoch. "0 vs 0" indicates
|10kPto - 0x.pyll where Py is a sccond run of Po for sanity check, "0 vs indicates ||0k,Po 0g 111 Right: test accuracy of each {Pi) during 200 epochs.
It is noteworthy that such large values are impractical in real-world applications. For more realistic 3 values, we observed no impact on training loss or test accuracy. We recommend using low-norm Jacobians to ensure robust training; therefore, propose minimal MaxPool Jacobian (8 results the minimal norm). Furthermore, employing the Adam optimizer at 32-bit precision, as
noted by Bertoin et al. (2021), effectively counteracts the adverse effects of large / values, thereby stabilizing training (see Appendix C.2
Connexion with the choice of ReLU'(O): Initially, Bertoin et al. (2021 reported that the choice of (e) significantly impacts learning, with vanilla SGD training showing ReLU'(0) = 0 the most efficient option. A recent erratum published by the same authors (Bertoin et al. (2023)) revises this finding, indicating that the impact of (o) on learning outcomes is considerably less pronounced than initially stated. This is in line with our investigation of the MaxPool operation.
5 Conclusion
In our study, we evaluate the reliability of automatic differentiation in neural networks involving MaxPool. Testing across a variety of models and datascts revcaled that AD may inaccuratcly process MaxPool operations when using floating-point numbers. This observation suggests that the AD correctness findings by Lee et al. (2023) may not fully apply convolutional neural networks incorporating MaxPool. Our analysis identifies two critical zones: bifurcation zones, where AD inaccuracies manifest in both real and floating-point computations, and compensation zones, which are accurate in real numbers but may exhibit errors in Soating-poin representations. Although bifurcation zones are uncommon, they lead to significant AD discrepancies. In contrast, compensation zones more frequently display minor shifts in amplitude, related to machine precision.
Furthermore, employing lower-norn MaxPool Jacobians tends to enhance training stability and test accuracy, while higher-norm Jacobians increase the risk of training instability, especially in lower-precision environments. Factors such as dataset characteristics, network architecture, and learning parameters including batch normalization and the Adam optimizer significantly influence AD's numerical behavior.
13

---

Published in Transactions Machine Learning Research (06/2024)
Acknowledgments and Disclosure of Funding
The author acknowledges the support of the AT Interdisciplinary Institute ANITI funding under the grant agreement ANR-19-P13A-0004. The author acknowledges the help of the Association Nationale de la Recherche et de la Technologie (ANRT) and Thales LAS France, which contributed to Ryan B's grant. This work was performed using HPC resources from CALMIP (Grant 2023-P233440). The author would like to thank Jérôme Bolte and Edouard Pauwels for their precious feedback. The author would like to thank the collaborators at Thales LAS France, in particular Beatrice Pesquet-Popescu and Andrei Purica, for their help. comments.
References
Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation erence/
(OSDI 16), pp. 265 283, 2016.
URL httts:///wwi hwoppi//www..senix...m
usenix weaenix..gg/seeee
Paul I. Barton, Kamil A. Khan, Peter Stechlinski, and Harry A.J. Watson. Computationally relevant generalized derivatives: theory, evaluation and applications. Optimization Methods and Software, 33 (101 1072, 2018. doi: 10.100//1033078..017...33 2017.337188. (rcei.org/1001000//0555777 2017 1374385.
URL https:
Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of Morchine Learning Research, 2018.
David Bertoin, Jérôme Bolte, Sébastien Gerchinovitz, and Edouard Pauwels. Numerical influence of relu' (0) on backpropagation. Advances in Neural Information Processing Systems, 14:4684479 2021.
David Bertoin, Jérôme Bolte, Sébastien Gerchinovitz, and Edouard Pauwels. Erratum: Numerical influence of relu'(0) on backpropagation, 2023.
Jérôme Bolte and Edouard Pauwels. Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning. Mathematical Programming, pp. 1 1-33, 2020a.
Jérôme Bolte, Tam Le, Edouard Pauwels, and Antonio Silveti-Falls. Nonsmooth implicit dif- ferentiation for machine learning and optimization. CoRR, Abs/2106.04350, 2021a. URL
https://arxiv.orz wttp:///www//ab//1.......
Jérôme Bolte, Tam Le, Edouard Pauwels, and Tony Silveti-Falls. Nonsmooth implicit differentiation for machine-learning and optimization. Advances in Neural Information Processing Systems, 31, 2021b.
Jérôme Bolte, Ryan Boustany, Edouard Pauwels, and Béatrice of nonsmooth automatic differentiation. In The Eleventh International Conference on Learning Representations, 2022.
On the complexity
14

---

Published in Transactions Machine Learning Research (06/2024)
Jérôme Bolte and Edouard Pauwels. A mathematical model for automatic differentiation in machine lcarning. In Conference on Neural Information Processing Systems, 2020b.
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Oplimization methods for large-scale machine learning. Siam Review, 60(2):223 311, 2018.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnn and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. wtt. com/googee/jax
Frank H Clarke. Optimization and nonsmooth analysis. SIAM, 1983.
Matthicu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications. In Proceedings of the International Conference on Learning Representations, 2015.
D. Davis, D. Drusvyatskiy, S. Kakade, and J. D. Lee. Stochastic subgradient method couverges on tame functions. Foundations of Computational Mathematics., 2018.
Damek Davis, Dmitriy Drusvyatskiy, Sham Kakade, and Jason D Lee. Stochastic subgradient method converges on tame functions. Foundations of compulational mathematics, 20(1):119 154, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fci. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 148-255,22 2009.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
A. Griewank and A. Rojas. Treating artificial neural net training as a nonsmooth global optimization problem. In International Conference Machine Learning, Optimization, and Data Science (pp. 759-7700 (70). Springer, Cham., 2019.
A. Griewank and A. Walther. Beyond the oracle: Opportunities piecewise differentiation. In Numerical Nonsmooth Oplimization (pp. 331-361). Springer, Cham., 2020.
Andreas Griewank. On stable piecewise linearization and generalized algorithmic differentiation. Optimization Methods and Software, 28, 07 2013. doi: 10.1080/10556888.000 10111/000067881...70000
Andreas Griewank and Andrea Walther. Evaluating derivatives: principles and techniques of
algorithmic differentiation. SIAM, 2008.
Andreas Griewank, Andrea Walther, Sabrina Fiege, and Torsten Bosse. On lipschitz optimization
based on gray-box piecewise linearization. Mathemalical Programmingg 158:383 415, 2016.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International conference on machine learning, pp. 1737-1746. PMLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference computer vision, pp. 1026-100000000 1026-1034, 2015.
15

---

Published in Transactions Machine Learning Research (06/2024)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9: 1735 1735-1780, 1997.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 1700-1708, 2017.
Kyuycon Hwang and Wonyong Sung. Fixed-point quantization of deep convolutional networks. In Proceedings of the International Conference on Machine Learning, 2014.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint wrXiv:150200116 2015.
Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, et al. Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes. arXiv preprint 1rXi::1807.11206 2018.
Sham M Kakade and Jason D Lee. Provably correct automatic sub-differentiatiio for qualified programs. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1422.6000 6980, 2014.
Alex Krizhevsky and Geoff Hinton. Convolational deep belief networks on cifar-10. Unpublished manuscript, 40(7):1-9) 2010.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-25 222-2224,,, 1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521 (2555)))) 044,, 2015.
Wonyeol Lee, Hangyeol Yu, Xavier Rival, and Hongseok Yang. On correctness of automatic differentiation for non-differentiable functions. In NewrIPS 2020-34th Conference on Neural Information Processing Systems, 2020.
Wonycol Lee, Sejun Park, and Alex Aiken. On the correctness of automatic differentiation for neural networks with machine-representable parameters, 2023.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03744 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised fcature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 5, 2011.
16

---

Published in Transactions Machine Learning Research (06/2024)
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeon, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjic Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL http::/papers..nurrpps.cco perf www.eeperrreceeeeeeeeeeee oornaace--dee----aa-----------e-
D. E. Rumelhart, Geoffrey E. Hinton, and R. J. Williams. Learning representations by back- propagating errors. Nature, 323:533-555 -355,,6,,,,,, 1986a.
David E Rumelhart, Gcoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature, 323(88))) 536, 1986b.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint ar arXiv:1409. 1556, 2014.
Bert Speclpenning. Compiling fast partial derivatives of functions given by algorithms. University of Illinois at Jrbana-Champaign, 1980.
Vincent Vanhoucke, Andrew Senior, and Mark Z Mao. Improving the speed of neural networks on cpus. In Proceedings of the Deep Learning and Unsupervised Feature Learning Workshop, 2011.
Kouichi Yamaguchi, Kenji Sakamoto, Toshio Akabanc, and Yoshiji Fujimoto. neural network for ipedeeerindeeeedentt isolated word recognition. In ICSLP, 1990.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on compuler vision, pp. 818 833, 2014.
17

---

Published in Transactions Machine Learning Research (06/2024)
This is the appendix for "On the numerical reliability of nonsmooth autodiff: MaxPool case study".
Contents
A Further comments, discussion, and technical elements
18
B Proof related to Section 2.3
22
Complements on experiments
23
D Complementary information
26
A Further comments, discussion, and technical elements
A.1 Implementation of the zero program
The implementation of the zero function used in Table I is given in Figure i Programs max, and max2 correspond to an equivalent implementation of the same function max, but the computed derivatives are different.
def max1 (x) res x[0]
def mxx2(x)=
return torch, max (x)
for in range(1, 4)
x[i] res:
def zero(t):
res x [ii]
2 = t
return res
return max1 - max2(z)
Figure 7: Implementation of programs max1, max2 and zero using Pytorch. Programs maxi and max2 are an equivalent implementation of max, but with different derivatives due to the implementation.
A.2 Challenges posed by MaxPool in image processing
In Convolutional Neural Networks (CNNs), the MaxPool operation is frequently used for reducing dimensions and downsampling. This function is especially crucial in image contexts, where uniform intensity regions are common, especially around the edges of objects and flat surfaces. One common situation is encountering identical pixel values within pooling window, shown in Figure 8 MaxPool must choose among these equivalent values, creating a point of non-diffeeentiabiiity During training, this affects gradient calculation in backpropagation, affecting the updates to convolutional filters [Goodfellow et al. 2016).
A.3 AD errors with ReLU-derived programs
We conduct small PyTorch experiment using the nonsmooth function ReLU: + max(æ, 0). Consider two programs maxı and max2 implementing the max: + maxi function using different ReLU-derived programs. Note that max(x, y) = (x + v) (ReI.U(y) ReI.U-=z)). Let zerog: + max (t x æ)) max2 (t x x) be a program implementing
y)) +
18

---

Published in Transactions Machine Learning Research (06/2022)
3
Figure 8: Image segment post-convolation, spotlighting equal pixel values (marked in red) within a 2x2 MaxPool window.
the null function as described in Figure 9 Let zeroh denote the backward AD algorithm for the zero program. As mathematical functions, max, and maxę are equal and the program zero outputs constantly 0. However, for some E R, AD can return (t) # O. Results are reported in Table 3 and similar to Table
def relu (x):
return torch (xlu(x)
def relu2(x):
return torch. where >= O, x, torch. tensor (0.0))
def max01(x):
return (x[0] + x[1]) / 2 + relu((x -x[1]) / 2) relu((x[1] x[o]) / 2)
def mxx02(x):
return (x[0] + x[1]) / 2 relu2((x[0] - x[1]) / 2) + relu((x[1] x[o]) / 2)
def max1(x):
return max01 (torch. stack [[axx01 (x (x]]))) max01 (x(:4)]))
def max2(x):
return eax02(torch) stack ([max0(xx0:22), ( max02(((2:4))))
def pero_2(t):
2 x return max1 (z) max2(z)
Figure 9: Implementation of max1, maxę and zero2 using Pytorch. Programs max, and max2 are an equivalent implementation of max, but implemented using different ReLU-derived programs.
A.4 NormPool nonsmooth multivariate operation without compensation errors
We conducted an experiment to show that compensation errors are not caused by the multivariate nature of nonsmooth elementary functions when using floating-point arithmetic. In this experiment, used the NormPool operation, which is similar to the MaxPool operation but replaces the
19

---

Published in Transactions Machine Learning Research (06/2024)
2ero2
10
10
2
-10
0
10
102
103
1.0
2.0
3.0
1.0
0.0
0.0
0.0
1.5
0.0
0.0
0.0
x 1.4
1.4
1.4
1.4
10-7
10-7
10-7
10-7
10-7
10-7
10-7
Table 3: Summary of varions types of AD errors with zero2 program using PyTorch for different combinations of & and x.
maximum with the Euclidian norm. Two programs, P and Q, were used to implement a LeNet-5 network on the MNIST dataset with two different NormPool-derivee programs. We computed the backprop variation (see Definition between P and Q, while controlling all sources of divergence in our implementation using deterministic computation. The results are presented in Figure 10 The experiment was conducted CPU with 16-bit floating-poit precision.
107
Proportion
10a
10
4
lgwummagnitude)
Figure 10: Histogram of backprop variation between P and Q for LeNet-5 network on MNIST (128 mini-batch size) with 16-bit. We run M 1000 experiments.
In contrast to our findings with MaxPool, we obtained similar results to those reported in Bertoin et al. (2021) with ReLU-based programs. Specifically, for NormPool-based programs, we observed either significant divergence of backprop none.
A.5 Bifurcation zone: practical example
This section presents an example that demonstrates cases where AD can be incorrect. Calculating the accurate derivative for all inputs might be impossible, particularly when the function is nondifferentiable. This is because the derivative does not exist for inputs where the function is nondifferentaable.
A.5.1 Network configuration
Consider an input matrix X of size 4 x 4 given by:
1
0
1
0
0
0
0
X
(Input)
0
20

---

Published in Transactions Machine Learning Research (06/2024)
Let k be a positive number and W be convolution kernel of size 3 3 given by:
1
1
1
W = k
1
1
1
(Convolution kernel)
1 1
1/
Let's consider a composition function I such that:
I(W) -aaxPool (X W) = k
(17)
where the convolution operation X * W produces an output matrix Z of size 2 x 2, followed by the application of MaxPool with a pooling window of size 2 2.
A.5.2 Backprop computation: native vs minimal
Let P (resp. Q) be a program implementing the composition function in Equation equation 17 using the native (resp. minimal) MaxPool-derived program. Then, we have:
1
0
0
0.5
0
0.5
back)
0
0
0
backprop(Q[(w)
0
0
0
0
0
0
0
The convolutional kernel W falls within the bifurcation zone defined in Definition 3
A.6 Comments on Section 3
A.6.1
Non-determinism in GPU computation
Graphics Processing Units (GPUs) are designed for parallel processing, which can result in unpre- dictable behaviors.
Floating-point operations: The non-associative nature of Roating-point arithmetic can lead to discrepancics. These differences might become significant as they accumulate across operations.
Reduction operations: Functions like sum or maximum, espocially in GPUs, can cxhibit variability between runs. This variability can result in divergent accumulated rounding errors.
A.6.2 Threshold values for various networks in Section 3.1
Table presents threshold values for various neural networks on different datasets, computed under different floating-poin precisions (16-bit, 32-bit, and 64-bit). For simplicity, thresholds are approximated as powers of 10.
A.6.3 Details on Monte Carlo sampling in Section 3.2
Recall that, for a neural network f and a floating-pon point precision w, we want to estimate the volume of the set
S(TFW)== CRRERP: 3i (1,..., N), (ac))rop(P.](() backprop(Q:](0)1 > Tfw C
21

---

Published in Transactions Machine Learning Research (06/2024)
Network f
Dataset
2
2
2
Tf.22
7,,32
7,64
LeNet-5
MNIST
0
10
10
10
VGG-11
CIFAR-10
0
10-1
10-8
10-7
10-14
0
VGG-11
SVHN
0
10-1
10 10-8
10-7
10
10
0
VGG-13
CIFAR-10
0
10-1
10-9
10-9
10-14
0
VGG-16
CIFAR-10
0
10-2
10-10
10-9
10-15 15
0
VGG-19
CIFAR-10
0
10-3
10-11
10-10
10-15
ResNet-18
CIFAR-10
10-2
10
2
1
10
10
10 10
13
DenseNet-121
CIFAR-100
0
10-2
10-6
10
10-14
0
Table 4: Threshold values of various neural networks f across different datasets.
Our experiments divide a dataset into R mini-batches. Each r-th mini-batch is represented by the index set B, C f N}. The programs P, and Qr are associated with the neural network / and implement a composition function for cach r. Specifically, P. uses the native MaxPool-derived program, whereas Qr uses the minimal one. For every precision level E {16, 32, 64], we establish a threshold Tfw as in Section 3 Using the Kaiming- Uniform (He al., 2015 initialization in PyTorch, we randomly generate a parameter set {0}])=1 the formula
with M 1000. The first line of Table 2 is given by
||
Ts ): (18)
Σ 25,00) JCB,
M Σ Br (1...., R},
backprop Σ P3(0) JCB,
backprop
where 1 represents the indicator function, returning either or depending on the truth value of its argument's condition. Similarly, the second line of Table 2 is given by the formula
M
R 11
(Dackprop
Σ Pj(0) JCB,
- backprop
Σ 21(0) JCB,
> TTW
(19)
MR j=l
Using the formula
In 2
2n
and setting a = 0.05, we compute the error margin of the Hoeffding confidence interval as n = M for Table 2|s first line and .M R for its second. The first line adheres to 95% confidence interval under the iid assumption due to Hoeffding's inequality.
Using McDiarmid's inequality at risk level = 0.05, we compute the error margin of the second line in Table 2 by the formula.
In
2
M
R
B Proof related Section 2.3
Proof 1 (of Proposition
22

---

Published in Transactions Machine Learning Research (06/2024)
1. The three subsets have unique definitions, indicating that they are separate. For instance, a paramcter cannot belong to the regular and bifurcation zones since the regular zone is defined as the area where each program Jijj is assessed differentiable points. On the other hand, the bifurcation zone defined as the region where the set of all possible backprop outputs is not a singleton, indicating von-differnntiabiiiyy at some points. Additionally, the union of these zones covers the entire parameter space θ as every parameter must be assigned to one of the three subsets: resulting in differentiable points when evaluated, resulting in nondifferentiable points but having singleton backprop set, or resulting in nondifferentiable points with non-singleton backprop set. Therefore, OR U U U O.
2. As we consider locally Lipchitz semialgebraic (or dcfinable) functions, see [Thcorem 1, Bolte
& Pauwels (2020a)) for the proof arguments.
C Complements on experiments
C.1 Benchmark datasets and architectures
In this work, we utilized various well-known image classification benchmarks. Below are the datasets, including their characteristics and original references.
Datasets:
Dataset
Dimensionality
Training set
Test set
MNIST
28 x 28 (grayscale)
60K
10K
CIFAR10
32 x 32 (RGB)
60K
10K
SVHN
32 x 32 (RGB)
600K
26K
ImageNet
221 221 (RGB)
1.3M
50K
The corresponding references for these datascts are LeCun et al. (1998);
Krizhevsky & Hinton
2010); Netzer et al. (2011).
Neural network architectures:
We evaluated various CNN neural network architectures, with
details as follows:
Name
Layers
Loss function
LeNet-5
5
Cross-entropy
VGG11
11
Cross-entropy
VGG13
13
Cross-entropy
VGG16
16
Cross-entropy
VGG19
19
Cross-entropy
ResNet18
18
Cross-entropy
ResNet50
50
Cross-entropy
DenseNet121
125
Cross-entropy
The corresponding references for these architectures are Simonyan Zisserman (004): He et al.
2016); Huang et al. (2017) LeCun et al. (1998)
LeNet-5:
The implementation for LeNet-5 was sourced from the following GitHub repository:
23

---

Published in Transactions Machine Learning Research (06/2024)
VGG: We used the PyTorch repository's implementation for the VGG models. It can be accessed at the following link: https///....oooooooo//o////ll/lt
vgg. py-
For ResNet models, we utilized the PyTorch repository's impleeentttnnnnnnnnnnntto available at: . We made minor adjustments to the output layer's size (changing from 1000 to 10 classes) and the kernel size in the primary convolutional, varying from 7 to 3). When batch normalization was not used, we replaced the batch normalization layers with identity mappings.
ResNet:
DenseNet: The implementation for DenseNet was taken from the PyTorch repository, available at: http::/ttww...poooooo/o////////t
C.2
Mitigating factor: Adam optimizer
After training VGG11 network CIFAR-10 using the Adam optimizer, obtained results shown in Figure 11 Our findings are consistent with those presented in Section 3 but the network exhibits reduced sensitivity to 3, resulting in improved stability of both test errors and training loss.
0.5
0.92
Train loss
100
Batch norm
0.1
10000
1000
0.86
True
0.0
50
100
150
200
10
100
1000
10000
epoch
(a) 32-bit without batch normalization
(b) 32-bit
Figure 11: Training losses on CIFAR10 (left) and test accuracy (right) on VGG network trained with Adam optimizer and without batch normalization.
C.3 Additional experiments with MNIST and LeNet-5 networks
We repeated the experiments in Section 4.2 using a LeNet-5 network on the MNIST dataset. The results are depicted in Figure 12 We found that for 16 bits, the test accuracies were similar when training possible, but 8 {103, 104) caused chaotic training behavior. For 32 bits, the test accuracies were mostly similar, except for 3 10 We noticed that the chaotic oscillations had completely disappeared.
C.4 Additional experiments with ResNet18
We performed the same experiments described in Section 1.2 using ResNet18 architecture trained on CIFAR 10. Figure 13 represents the test errors with or without batch normalization. For 16 bits, test accuracies are similar, but 8 = 10 induces chaotic training behavior. For 32 bits, test accuracies are identical, and the chaotic oscillations phenomena have entirely disappeared.
24

---

Published in Transactions Machine Learning Research (06/20224)
0.10
Batth
Test acurssy
0. 0.994
False True
0.08
loes
0.993
10
100
0.02
0.993
10000
0.00
30
100
1060
10000
0
20
40
60
80
100
opoch
(a) 16-bit
(b) 16-bit, with batch normalization
0.15
Test 3ccuso
0.994
Train loss
0.10
0.992
10
Batch norm
0.990
False
1000
10000
0.00
100
10000
20
40
60
100
g
epoch
(c) 32-bit
(d) 32-bit, without batch normalization
Figure 12: Training LeNet-5 network on MNIST with SGD. We performed ten random initializations for each experiment, depicted by the boxplots and the filled contours (standard deviation).
0.93
norm
8
Test accuracy
False
0.92
0.6
0.91
0.4
10
0.2
1000
10000
0.0
100
1000
50
100
150
200
(a) 16-bit
(b) 16-bit, with batch normalization
0.94
Batch norm
2.0
Test accuracy
1.5
0.93
1.0
0.92
0.5
1000
0.91
0.0
10000
150
200
opoch
(c) 32-bit
(d) 32-bit, with batch normalization
Figure 13: Training ResNet18 network on CIFARI0 with SGD. We performed ten random initializations for each experiment, depicted by the boxplots and the filled contours (standard deviation).
C.5
Additional experiments with ResNet50 on ImageNet
We performed the same experiments described in Section 4.2 using a ResNet50 architecture trained on
ImageNet. The test accuracy represented in Figure 14 We employ mixed precision (Micikevicius
et al. 2017 Jia et al.] 2018), utilizing 16 and 32 bits precision to balance computational speed and
25

---

Published in Transactions Machine Learning Research (06/2024)
information retention. Test accuracies are similar when training is possible, but 10 induces
chaotic training behavior.
80
Test accuracy
60
0
1
40
100
1000
0
20
40
60
80
epoch
Figure 14: Test accuracy during training a Resnet50 on ImageNet with SGD using mixed precision.
The shaded area represents three runs. We have chaotic test accuracy behavior for 8 = 10
D Complementary information
Computational Resources: All the experiments were conducted on four Nvidia V100 GPUs. This ensured consistent and reliable computation times across different experimental runs.
The code corresponding to the experiments, as well as the results of these experiments, are publicly available. The repository can be accessed at the following URL: htt.ta//ggggtab....orrrrraaaaasa
Code and Results Availability:
Licenses:
The datasets used in our experiments are released under various licenses. CIFAR10 is under the MIT license, MNIST and SVHN are under the GNU General Public License, and ImageNet is under the BSD license. The libraries we used, Numpy and PyTorch, are released under the BSD license, while Python is released under the Python Software Foundation License.
Dataset
Network
Optimizer
Batch Size
Epochs
Time Per Epoch
Repetitions
MNIST
LeNet-5
SGD
128
100
2 seconds
10
CIFARI0
VGG11
SGD
128
200
9 seconds
10
CIFAR10
ResNet18
SGD
128
200
13 seconds
10
SVHN
VGG11
SGD
128
100
70 seconds
10
ImageNet
Resnet50
SGD
512
90
15 minutes
3
Table 5: Detailed experimental setup, including the dataset, neural network architecture, optimizer used, batch size, number of epochs, average computation time per epoch, and repetitions for each experiment.
26


Reviews:
[{"review_id": "Vk4t3gWQFj", "reviewer": "Reviewer_7NMi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1) the manuscript is well-written, and the motivation of the study is clear\n2) the suggested concepts look interesting and promising for analysis of the instability in training NN \n3) the numerical experiments with different floating-point formats illustrate the introduced theoretical concepts\n\nWeaknesses\n1) the main weakness of the presented work is the artificially constructed experimental setup. In particular, the learning rate equals 1, which is quite a large value; the tested values of \\beta look unnatural since they are too large to be used in practice. So, the authors in experiments operate with the possible scenarios which, however, do not typically appear in the workflow\n2) as far as I know, there are no issues in training the discussed CNNs for standard benchmarks via the most popular frameworks, e.g., PyTorch or Jax. So, the instability phenomenon of the gradient computations for the MaxPool layer is important yet does not significantly affect the final performance in the recommended experimental setup. At least, the authors do not demonstrate the possible improvement in the experiments (see section below)\n3) the authors focus only on the VGG-type CNN in the main text, although ResNet and DenseNet also include MaxPool layers. Thus, the universality of the discussed issues and suggested remedies is not convincing.\n4) the impact of the manuscript on using the MaxPool layer is limited since the authors do not present any recipes for modifying the standard implementation of the MaxPool layer in frameworks. \n\nAlso, there are multiple typesetting issues; see the list below.\n1) captions to the tables should be placed above, not below\n2) double the \"equation\" word in the caption to Figure 4 and refer to the equation from the appendix, which seems strange. Why not just move this equation into the main text?\n3) in most plots, lines for different betas are indistinguishable\n4) probably, \\cdot command is more appropriate rather than \\times in the Hybrid MaxPool program\n5) Figure 4 shows the 64-bit precision in legend, but the corresponding color is absent on the histogram", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present the analysis of the (sub)gradient computation for the MaxPool layer in the classical convolutional neural networks. The main contribution of the presented manuscript is the introduction of two separate groups of parameters corresponding to bifurcation and compensation zones. These zones indicate the large and small (of the machine precision order) deviations of gradient elements from the expected values, respectively. Extensive numerical experiments with CNNs demonstrate the changes in gradient computation if 16-, 32-, and 64-bit floating point numbers are used to store the CNNs' parameters. Also, the authors highlight the importance of the well-known BatchNorm layer in mitigating instabilities obtained from the large-norm Jacobian in the MaxPool layer.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) focus on the suggestions on how to modify the MaxPool backward step to improve stability (improve the test accuracy) or make this step adaptive for different floating point formats. \n2) add the same analysis for bfloat16 af tensorfloat32 formats since they are specially developed for training neural networks \n3) improve the quality of the plots and make the conclusions from them explicit\n4) revise the section about the experiments' results to make it consistent, and introduce modifications to the training setup incrementally to observe the effect of a particular ingredient. The explicit conclusions on the impact of Adam, the BatchNorm layer, and the proper computing of the MaxPool gradient make the results more convincing and easy to follow.", "broader_impact_concerns": "No concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8mnJRH5E2x", "reviewer": "Reviewer_rh1s", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n## Importance of the problem\n\nThis paper explores the influence of the floating-point precision and the practical implementation of MaxPool, and, to some extent, ReLU, on the results of AutoDiff. This kind of study is necessary when an entire community (Deep Learning) relies heavily on AutoDiff. Therefore, this paper tackles a important question.\n\n\n## Difference between compensation and bifurcation zones\n\nThe authors propose to distinguish two kinds of practical limitations: floating-point precision and practical implementation of non-smooth functions. This distinction is fundamental to tackle the initial problem, and is an important contribution of the paper.\n\n\n# Weaknesses\n\n## Scope of the paper\n\nOverall, the scope of the paper is unclear: it is a focus on MaxPool, but the analysis seems to be feasible with ReLU (and it is partially done), which is more natural to study given the past works.\n\n### Abstract\n\nIn the abstract, the distinction between existing works, known results and contributions of the paper is unclear. For instance: are the \"bifurcation zone\" and the \"compensation zone\" invented in this paper?\n\n### Intro: Various types of nonsmooth AD errors\n\nIn the Introduction, the authors show interesting preliminary results in Table 1. They are presented as the main motivation of the paper: using different implementations of the same function (here, MaxPool) may result in very different gradients.\n\nHowever, it is explained that these results, obtained with MaxPool, can be replicated with ReLU (see Appendix A.3). Therefore, why do the authors focus on MaxPool (as the title of the paper suggests), and do not claim to do a study involving both MaxPool and ReLU?\n\n### Intro: Related works and contributions\n\nMore clearly, the authors explain that they propose the concept of \"compensation zone\", while the concept of \"bifurcation zone\" already exists. However, why do they limit their study to MaxPool, while an extension of the preceding works with ReLU would be more natural?\n\n### Section 3.1: $\\tau^1$ and $\\tau^2$\n\nAs far as I understand, informally:\n * $\\tau^1$ is the upper bound on the error we expect to make when running several times the same program;\n * $\\tau^2$ is the lower bound on the error we expect to make when running two different implementations of the same function.\n \nIt is very strange that the authors computed $\\tau^1$ only with MaxPool and $\\tau^2$ only with ReLU. One may conversely use ReLU in a non-deterministic setting, and test two different implementations of MaxPool.\n\nIt is crucial to compute $\\tau^1$ and $\\tau^2$ with both ReLU and MaxPool: it would help to evaluate the relevance of $\\tau^1$ and $\\tau^2$ to distinguish the compensation zone and the bifurcation zone in each case.\n\nBesides, within the scope of the paper, it seems very important to be able to run experiments in a \"non-deterministic\" setting. The importance of this setting should be mentioned in the introduction (if not in the abstract itself).\n\n\n## 32 bits MNIST experiment (section 3.1)\n\nThe authors claim that the results plotted in Figure 1 are significantly different to the ones plotted in [1] (Figure 1). \n\nFirst, if I understand correctly, results of this section have been obtained with parameters at initialization (i.e., random parameters). However, the results showed in [1] (Figure 1) have been obtained with trained parameters (training experiments). So, these results are not comparable. \n\nSecond, even when comparing these plots to the ones presented in the Erratum of [1] (Figure 1), one should note that the weight difference is between $0$ and $10^{-3}$ in both cases. So, the \"contrast\" emphasized by the authors with the results of [1] is not very striking.\n\nI recommend to run experiments in the same condition to test the numerical stability of AD with ReLU and MaxPool: either both with random weights, or both throughout training.\n\n\n## $\\mathrm{P}$ versus $\\tilde{\\mathrm{P}}$ in a deterministic setup\n\nSince $\\tilde{\\mathrm{P}}$ is a copy of $\\mathrm{P}$ and all the computations are deterministic, I do not understand how experiments with $\\mathrm{P}$ and with $\\tilde{\\mathrm{P}}$ could lead to different results. Thus, Figure 1.b is superfluous (or could be, at most, regarded as a sanity check).\n\n\n## Clarity\n\n### Representation of a neural network: Eqn. (2)\n\nEven though it looks familiar, the representation of a neural network (NNs) proposed in Eqn. (2) is very uncommon. This kind of representation is also used in [1], but it comes with a short appendix explaining how it works. I recommend the authors to add a short description (possibly in the appendix) explaining how to represent usual NNs in this form.\n\n### Notation $M$ used twice\n\nThe letter $M$ is used as a number of layers (Section 2) and as a number of experiments (Section 3). For clarity, I recommend to choose a different letter.\n\nBesides, the letter $K$ is also used as a number of experiments (Figure 2).\n\n### $\\beta$: what is the meaning of $\\beta > 1$?\n\n$\\beta$ is used as a weight in the \"hybrid\" MaxPool (Definition 6). The authors do not explain in-text why we should be insterested in values $\\beta > 1$. Since most of experiments are made with $\\beta > 1$, the reason of this choice should be explained in this paper.\n\n### Definition of $\\tau^1$ and $\\tau^2$\n\nAs far as I understand, informally:\n * $\\tau^1$ is the upper bound on the error we expect to make when running several times the same program;\n * $\\tau^2$ is the lower bound on the error we expect to make when running two different implementations of the same function.\n \nThey are two distinct ways to delimit the compensation set (\"by above\") and the bifurcation set (\"by below\").\n\nBut this is an inference on my part (and I am not sure to understand correctly). To improve clarity, I recommend to:\n * provide an intuition of what the authors intend to measure with $\\tau^1$ and $\\tau^2$;\n * define first $\\tau^1$ and $\\tau^2$ on *only one* function (ReLU or MaxPool, it does not matter), then extend it to the other; as such, one may think that $\\tau^1$ is defined with MaxPool and $\\tau^2$ with ReLU.\n\n\n## Important typos\n\n * Figure 4, middle, label of x-axis: write \"Mini-batch size\" instead of \"Number of mini-batch size\" (which is unclear);\n * Appendix A.6.3: the indicator function is badly typeset; \"$\\mathbb{1}$\" should be used;\n * Section 4.2: \"$\\beta \\in \\\\{ 1, 10, 100\\\\}$\" instead of \"$\\beta \\in 1, 10, 100$\".\n\n\n## Reference to *Numerical influence of ReLU'(0) on backpropagation*\n\nApparently, [1] has been accepted as a NeurIPS poster in 2021. Why is this information missing in the References section (with 2023 as year of publication)? If the authors want to refer to the *Erratum* of [1], this should be mentioned in the information about [1] in the References section (by adding \"Erratum\" in the title of the paper, for instance). Otherwise, this reference may refer to two distinct works. \n\nBesides, even if it is worth mentioning the Erratum of [1], it would be better to refer to [1] directly for results that are not impacted by the error present in [1].\n\n[1] *Numerical influence of $\\mathrm{ReLU}'(0)$ on backpropagation*, Bertoin et al., 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a study of the influence of the practical implementation of MaxPool on the computed gradients when using AutoGrad, in the context of Deep Learning. The authors propose to distinguish two kinds of error: 1) the floating-point errors are caused by the limited precision when representing reals on computers; 2) the errors related to the choice of implementation of non-smooth functions. \n\nThe authors provide an empirical study involving MaxPool and ReLU, with random weights and during training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, the scope and the clarity of the paper need to be improved.\n\n# Scope\n\nThe scope of the paper should be improved: is it about MaxPool? compensation zones? the specific influence of non-deterministic computation? I recommend to deal with MaxPool and ReLU equally, to insist more on the importance of the compensation zones (which seem to be the core of the paper), and thus emphasize the influence of non-deterministic computation.\n\nN.B.: if it is technically irrelevant to study the compensation zones with ReLU, it should be discussed somewhere.\n\n# Clarity\n\nMany definitions lack clarity: $\\tau^1$ and $\\tau^2$ (which may be both defined with MaxPool or ReLU); use of letters $M$ and $K$; discussion on $\\beta > 1$.\n\nOne should discuss the relevance of training experiments (which rely on $\\beta > 1$). If there is no significant difference between experiments made with any $\\beta \\in [0, 1]$, it is interesting for the community and should be reported somewhere. I would prefer a discussion about a negative result in a standard setting than about not-so-clear results in an artificial setting (i.e., $\\beta > 1$).", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JWEp9mlcXl", "reviewer": "Reviewer_112e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nOverall, the paper presents a comprehensive analysis of the differentiability of the MaxPool operation, both from a theoretical and empirical standpoint. Experiments are well-conducted and the presentation of concepts is well-structured. Theoretical results appear correct.\n\n**Weaknesses**\n\nWhile generally well-written, a few changes could improve the clarity of the manuscript. The proposed changes are outlined below in the requested changes section.\n\nAnother weakness is that the practical value and significance of the paper's observations could be made more apparent. The paper analyzes test accuracy for networks with different implementations of MaxPool, but it would be helpful for the paper to make a set of concrete recommendations for practitioners.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies inaccuracies in the automatic differentiation of the MaxPool operations. The paper identifies two types of errors: bifurcation errors, occurring in non-differentiable regions of the function, and compensation zones, which are errors that occur purely due to machine imprecision. The authors demonstrate theoretically that the bifurcation zones occur in a set of measure 0. Next, the authors conduct experiments demonstrating when these errors occur and find, among other observations, that higher numerical precision can mitigate these errors somewhat.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Would strengthen**\n\nExplaining the implementations of max_1 and max_2 on page 2 instead of in the Appendix would be helpful since this is a key introductory piece of the paper\n\nIncreasing the size of some of the figures (especially 5 and 6) for readability\n\nImproving the table format by removing the vertical column dividers\n\nExplaining more clearly the practical significance of the paper's results", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies two types of numerical errors that can occur when training a neural network involving ReLU and MaxPool operators, i.e., floating-point errors by the limited precision on computers and the errors on the implementation of non-smooth functions. The paper introduces theoretical concepts on the differentiability of the MaxPool operation, which are also verified by extensive experiments. The paper is clearly written. The authors have made the changes suggested by the reviewers. All the reviewers are satisfied with the revision.
Decision: Accept (poster)