Smooth Activations and Reproducibility in Deep Networks
Authors: Gil I Shamir
Dong Lin
Lorenzo Coviello
Venue:  (2021)
Keywords: Deep networks
activation functions
reproducibility

Abstract: Deep networks are gradually penetrating almost every domain in our lives due to their amazing success.  However, with substantive performance accuracy improvements comes the price of irreproducibility.   Two identical models, trained on the exact same training dataset may exhibit large differences in predictions on individual examples even when average accuracy is similar, especially when trained on highly distributed parallel systems.  The popular Rectified Linear Unit (ReLU) activation has been key to recent success of deep networks.  We demonstrate, however, that ReLU is also a catalyzer to irreproducibility in deep networks.  We show that not only can activations smoother than ReLU provide better accuracy, but they can also provide better accuracy-reproducibility tradeoffs.  We propose a new family of activations; Smooth ReLU (SmeLU), designed to give such better tradeoffs, while also keeping the mathematical expression simple, and thus implementation cheap.  SmeLU is monotonic, mimics ReLU, while providing continuous gradients, yielding better reproducibility.  We generalize SmeLU to give even more flexibility and then demonstrate that SmeLU and its generalized form are special cases of a more general methodology of  REctified Smooth Continuous Unit (RESCU) activations. 
Empirical results demonstrate the superior accuracy-reproducibility tradeoffs with smooth activations, SmeLU in particular.

Under review as a conference paper at ICLR 2021
SMOOTH ACTIVATIONS AND REPRODUCIBILITY IN DEEP NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep networks are gradually penetrating almost every domain in our lives due their amazing success. However, with substantive performance accuracy improve- ments comes the price of irreproducibiliyy Two identical models, trained on the exact same training dataset may exhibit large differences in predictions on indi- vidual examples even when average accuracy is similar, especially when trained on highly distributed parallel systems. The popular Rectified Linear Unit (ReI.U) activation has been key to recent success of deep networks. We demonstrate, how- ever, that ReLU is also catalyzer to irreproducibility in deep networks. We show that not only can activations smoother than ReLU provide better accuracy, but they fam-
tradeoffs. We propose ily of activations; Smooth ReLU (SmeLU), designed to give such better tradeoffs, while also keeping the mathematical expression simple, and thus implementation cheap. SmeLU is monotonic, mimics ReLU, while providing continuous gradi- ents. yielding better reproducibility. We generalize SmeLU to give even more flex- ibility and then demonstrate that SmeLU and its generalized form are special cases of more general methodology of REctified Smooth Continuous Unit (RESCU) activations. Empirical results demonstrate the superior securacyreeroouucbblig tradeoffs with smooth activations, SmeLU particular.
can also provide better eecuracy-reproducible
INTRODUCTION
Recent developments in deep learning leave no question about the advantages of deep networks over classical methods, which relied heavily on linear convex optimization solutions. With their as- tonishing unprecedented success, deep models are providing solutions to a continuously increasing number of domains in our lives. These solutions, however, while much more accurate than their con- vex counterparts, are usually irreproducible in the predictions they provide. While average accuracy of deep models some validation dataset is usually much higher than that of linear convex models, predictions on individual examples of two models, that were trained to be identical, may diverge substantially, exhibiting Prediction Differences that may be high non-negligible fractions of the actual predictions (see, c.g.. [Cen et al. (2020); Dusenberry et al. (2020)). Deep networks ex- press (only) what they learned. Like humans, they may establish different beliefs as function of the order in which they had seen training data (Achille et af.) [201) [Beniio et al. 2009). Due to the huge amounts of data required to train such models, enforcing determinism (Nagarajan al.) (0018) may not be an option. Deep networks may be trained on highly distributed, parallelized systems. Thus two supposedly identical models, with the same architecture, parameters, training algorithm and training hyper-parameter that are trained on the same training dataset, even if they are initialized identically, will exhibit some randomness in the order in which they see the training set and apply updates. Due to the highly non-convex objective, such models may converge to different optima, which may exhibit equal average objective, but can provide very different predictions to individual examples. Inteproducibility in deep models is not the classical type of epistemic uncertainty, widely studied in the literature, nor it overfitting. It differs from these phenomena in several ways: It does not diminish with more training examples like classical epistemic uncertainty, and it does not cause degradation to test accuracy by overfitting unseen data to the training examples.
While irreproducibilty may be acceptable for some applications, it can be very detrimental in appli- cations, such as medical ones, where two different diagnoses to the same symptoms may be unac- ceptable. Furthermore, in online and/or re-enforcement systems, which rely on their predictions to

---

Under review as a conference paper at ICLR 2021
determine actions, that in turn, determine the remaining training examples, even small initial irre- producibility can cause large divergence of models that are supposed to be identical. One example is sponsored advertisement online effect of irreproducibility in CTR prediction can go far beyond changing the predicted CTR of an example, it may affect actions that take place downstream in a complex system. Reproducibility is a problem even if one trains only single model, as it may be impossible to determine whether the trained model provides acceptable solutions for applications that cannot tolerate unacceptable ones.
Brrck-Throuhh-Ree -ate (CTR) prediction (McMahan et al. 2013). The
A major factor to the unprecedented success of deep networks in recent years has been the Rec- rified Linear Unit (ReLU) activation, (Nair & Hinton| 2010p. ReLU nonlinearity together with propppropaatiion give simple updates, accompanied with superior accuracy. ReLU thus became the undisputed activation used in deep learning. However, is ReLU really the best to use? While gives better optima than those achieved with simple convex models, it imposes an extremely non- convex objective surface with many such optima. The direction of a gradient update with a gradient based optimizer is determined by the specific example which generates the update. Thus the order of seeing examples or applying updates can determine which optimum is reached. Many such optima, as imposed by ReLU, thus provide a recipe for irreproducibility. In recent years, different works started challenging the dominance of ReLU, exploring alternatives. Overviews of various activations were reported in Nwankpa al. [2018): Pedamonti (2018). Variations on ReLU were studied in Jin
iet al. (2015). Activations like SoftPlus (Zheng et al. 2015). Exponential Linear Unit (ELJU) (Clevert
al.
2015), Scaled Exponential Linear Unit (SELU) (Klambauer et al. [017 Sakketou & Ammazis
2019
Wang et al. 2017). or Continuously differentiable Exponential Linear Unit (CELU) (Barron
2017
were proposed, as well as the Gaussian Error Linear Unit (GELU) (Hendryeks & Gimpel
2016
Specifically, the Swish activation (Ramachandran et al. (017) (that can approximate GELU)
was found through automated search achieve superior accuracy to ReLU. Further activations with similarity to GELU were proposed recently; Mish (Misral 2019), and TanhExp (Liu & Dij 20020) Unlike ReLU, many of these activations are smooth with continuous gradient. Good properties of smooth activations were studied as early as |Mhaskar (1997) (see also Du (2019); Lokhande et al. (2020)) These series of papers started suggesting that smooth activations, if configured properly, may be superior to ReLU in accuracy. Recent work by Xie al. (2020), that was done subsequently to our work, and was inspired from our results that we report in this paper, (Lin & Shamir 2019). demonstrated also the advantage of smooth activations for adversarial training.
Our Contributions: In this paper, we first demonstrate the advantages of smooth activations to reproducibility in deep networks. We show that not only can smooth activations improve accuracy of deep networks, they can also achieve superior tradeoff's between reproducibility and accuracy, by attaining a lower average Prediction Difference (PD) for the same or better accuracy.
Smooth activations, like Swish, GELU, Mish, and TanhExp, all have a very similar non-monotonic form, that does not provide clear stop region (with strict 0; not only approaching 0), and slope region. While these activations approximate the mathematical form of ReLU, they lack these prop- erties of ReLU. All these activations including SoftPlus also require more expensive mathematical expressions, involving exponents, and, in some, logarithms, or even numerically computed values (e.g., GELU).. This can make deployment harder, especially with certain simplified hardware that supports only limited number of operations, as well as can slow down training due to the heavier computations. Unlike ReLU, which can be transformed into Leaky ReLU, the smooth activations described cannot be casily transformed into more general forms. In this work, we propose Smooth ReLU (SmeLUU, which is mathematically simple and based only on linear and quadratic expressions. It can be more easily deployed with limited hardware, and can provide faster training when hard- ware is limited. SmeLU provides a clearly defined 0 activation region, as well as a slope 1 region, is monotonie, and is also extendable ties of smooth activations providing better reproducibility as well as better accuracy-reproducice tradeoffs. Its generalized form allows even further accuracy improvements. The methodology to construct SmeLU is shown to be even more general, allowing for more complex smooth activations, which are all clustered under the category of Rectified Smooth Continuous Units (RESCUs).
leaky or more general form. SmeLU gives the good proper-
Related Work: Ensembles (Dietterich) 2000) have been used to reduce uncertainty (Lakshmi- narayanan et al.) (017). They are useful also for reducing irreproducibility. However, they make models more complex, and can trade off accuracy in favor of reproducibility if one attempts to keep constant computation costs (which require reducing capacity of each component of the ensemble). Compression of deep networks into smaller networks that attempt to describe the same information
2

---

Under review as a conference paper at ICLR 2021
is the emerging area of distillation (Hinton et al. 2013). Predictions of a strong teacher train weaker student model. The student is then deployed. This approach is very common if there are ample training resources, but deployment is limited, for mobile network devices. Co-distillation, proposed by Anil et al. (018) (see also also Zhang et (2018)), took advantage of distillation to address inreroduuibiility Instead of unidirectional transfer of knowledge, several models distill information between each other. They attempt to converge to the same solution. The method requires more training resources co-train models, but deployment only requires a single model. somewhat op- posite approach; anti-Disiillation. to address irreproducibility was proposed by Shamir & Coviello (2020), embracing ensembles, with an additional loss that forces their components away from one another. Each component is forced to capture (more) different part of the objective space, and as a whole, the predictions of the ensemble are more reproducible. To the best of our knowledge, all previously reported techniques to address irreproduciiility in deep networks required some form of ensembles. In this work, we leverage smooth activations, and do not require ensembles.
Outline:
Section 2 proposes several PD metrics, which we use to measure irreproducibility. Next, we overview our setup and smooth activations Section 3 and describe SmeLU and its generaliza- tion in Section Experimental results are shown in Section 5
2 PREDICTION DIFFERENCE
Average individual per-example Prediction Difference (PD) over a set of models that are configured, trained, and supposed to be identical over some validation dataset be defined in various ways. We refer to & Coviello (220)) for a more detailed discussion. We opt to the same definitions they used, where describe the classification case, in which measure the PD using some Lp norm on the actual label predictions. Following their notation, denote the number of models by M, and the number of validation examples by N. Let Pr.m be the distribution over labels predicted by model m for example n. (Pr.m (f) the probability predicted for label f.) Let P, A be the expected distribution over labels over all M models. Then, the pth norm PD, Ap. is given by
Em Pr.m/M /
N
M
N
M
1/p
A
Σ
Pm.m
Pnllp
Σ
Σ
Pm (x)))))
(1)
N
x=1
m=1
n=1
m=1
With practical large scale systems, with very high training costs, we can use M 2, where for binary labels, A 1/N En, (1) - (1), where 1 is the positive label. As in Shamir & Coviello 2020), we can consider relative PD, A", normalizing the innermost summand in 1 by (() which for binary problems can be tweaked into Âς, normalizing by (1) instead. PD, A can be computed only the observed true label, by replacing the innermost sum l in (I) by (Pn.m(eruu (ftrree)) for the true label, normalizing by (ftruee)). Its computation, however,
requires knowledge of the true label. Finally, in classification, one can use Hamming PD; A specifying the average fraction of labels predicted differently between model pairs.
3 SMOOTH ACTIVATIONS
We consider standard Neural Networks. In addition to numerical features, inputs can also be embed- ding vectors representing unmeasurable features, learned together with weights and biases of hidden layers. Networks can be deep or shallow (a few or even a single hidden layer). The output of the ma- trix multiplication in any of the networks' hidden layers is activated by nonlinear activation. Our results apply in a standard setup, where training passes over a training dataset for multiple epochs, allowing example revisits. We also consider the online setting, where training consists of a single pass over the data. Inference is applied to unseen examples before updates. Normally, updates are applied in mini-batches. Some form of normalization may be applied in the hidden layers to limit signal range, guarding from vanishing or exploding gradients, and providing some scale invariance. Without normalization, activations can start drifting to become larger. As we show, smooth activa- tions are controlled by parameters that dictate their behavior. Inputs that scale up generate an effect of scaling down widths of the activation regions. As we will observe, slower smoother gradient changes, manifested wider input regions for the same output change, are more reproducible. With such drifts, they gradually appear to be narrower, slowly losing reproducibility benefits. Normaliza- tions, as weight normalization (Salimans & Kingma 2016), layer normalization (Ba et al.) 2016).
3

---

Under review as a conference paper at ICLR 2021
or batch normalization (loffe & Szegedy 2015) can be considered. In this work, however, a slightly different form of weight normalization, where some Lp norm normalizes all matrix link weights incoming to an output neuron of layer. We specifically normalize the L2 norm of the incoming matrix weights into the neuron to some norm v without centering around the mean.
Let W be the weight matrix incoming into layer E from layer L - 1, use wf to denote the jth row of W Then, the pre-activation neuron vector of layer É is given by a = W f(af-1), where f()) denotes the activation non-linearity (except for the input layer & = 0, where it is the identity). Then,
with L2 weight normalization, we replace w by f e w.w.) [w]] 22. Norms, other than I.2, as well as other normalization regimes can also be used. As we demonstrate below, for smooth activations, the norm can trade off with the activation parameters, tuning the accuracyireroduuciiliy tradeoffs, but not changing the good effects of smooth activations. As demonstrated in Appendix [A] lack of normalization with nonlinearity in the form of clipping values can diminish the benefits of smooth activations. Note that recent work by Liu et al. 2020) combined normalization with activations.
As described, several forms of smooth activations have been recently proposed in the literature. They all attempt to mimic the shape of ReLU but with a smoother function. They also try to avoid similarity to the smooth Sigmoid, (x) = 1/(1 + exp(-x)), which was considered several decades ago, but had limited success due to diminishing gradients. Many of them can be parameterized by one parameter or more. While some are smooth for all parameter values, others, like SELU, are only smooth with specific values of the parameters. Let 3 be the main activation parameter (SELU will also have another parameter A). Let erf(.) denote the standard error function and (--) the standard normal CDF. Then, activation functions for some smooth activations are given by
BSELU
=
2.
{
if > 0 x. 3c - B,
(2)
1
USoftPlus
3
log [1 + exp (B-x))
U/wissh
= x. a (B.x)
(4)
YCELU
x [ + erf (3.zf x x/v2) =x. c
(5)
/Mish
tanh [log (1 + exp(B x))
(6)
tanh (em*).
VTanhExp
ELU is SELU with X = 1, and CELU is a differentially continuous variant of SELU, shown in Ap- pendix C We generalized SoftPlus, GELU, Mish, and TanhExp by adding the 3 parameter. GELU can be approximated by Swish with G = zo ( (v8/ 32). Fig. shows the different activa- tions and their gradients for different values of the parameter 3. (More detailed figures are shown in Appendix ) All the activations described thus far share a region in which the activation function limits changes in the signal with a gradient close to 0 (on the left), and a region in which the gradient approaches on the right. The width of the middle transition region a function of the parameter 3, where it is wider with smaller 3 and narrower with greater B3. For 3 + co, the activation resembles ReLU, and for 3 + 0, it becomes closer to linear.
Figure 1: Smooth activations (top) and their gradients (bottom) for different 3 parameter values.
4

---

Under review as a conference paper at ICLR 2021
The discontinuous gradient of the ReLU partitions the parameter space into regions, each of which has its own local optimum. Many of the local optima may be identical in the overall total objective, but not on how they predict for individual examples. Randomness in training leads the model into one of the regions, and locks it in a region of some local optima. Specifically, sudden jumps in the gradient, like those of ReL.U, may lock some hidden units at some state, requiring other units to compensate and move towards their nearest optima. If this happens early in training (see, c.g., Achille et al. (2017). it determines the trajectory of the optimization based on the examples seen and the updates applied. Different example and update orders thus lead to different trajectories.
Smoother activations give a training model less opportunities to diverge. Wider transition regions allow gradients to change more slowly, and units in the network to gradually adjust to updates and changes. Wider transition regions are closer shape to more reproducible linear functions. However, good activation must also retain good accuracy. The Sigmoid was unable to do that. Unfortunately, while reproducibility mostly tends to improve with widening the transition region of the activation, accuracy moves mostly in the opposite direction. As we lower 3 and widen the region, we generally trade accuracy for reproducibility. This, however, is not the full picture. While moving towards ReLU improves aceuracy on datasets observed, the optimal point is not at the ReLU point of B + oc but for some finite 3. after which accuracy degrades. This shows that properly designed smooth activations for the specific dataset can be actually superior ReLU in both accuracy and reproducibility. They also provide a knob to trade off between the two. Similar behavior also observed in the opposite direction, where the best reproducibility for some datasets may be achieved for some 8 > O, worsening for smaller values. The overall tradcoff's are dataset, model architecture and model configuration dependent.
4 SMELU TO THE RESCU
Swish, GELU, Mish, and TanhExp, all have a non-monotonic region on the left of the transition region, neighboring the signal suppression region. This may be vulnerable to input irregularities, possibly less interpretable, and perhaps suboptimal. SoftPlus is monotonic, but the wider the tran- sition region, the farther it is from the origin, degrading its accuracy. In addition, for all these activations, there are clean stop and pass regions like in ReLU, except for very large 3. Asymp- totically, for 3 - oc, all tend to look like ReLU, but for wider transition regions, which are good for reproducibility, they do not reach the ReLU extremes on the left and right. Furthermore, all the activations considered require complex mathematical functions, such as exponents. Such func- tions require more complex hardware implementations, that could be very costly in huge scale sys- tems. The computationally heavy functions could also slow down training and inference. We now show a very simple smooth ReLU activation, SmeLU, that benefits from smoothness, monotonicity, clean stop and pass regions, very simple mathematical implementation, and comparable or better reroouulyyyyo tradeoffs. We show that the same design is extendible to generalization of this activation as well as to more sophisticated extensions.
SmeL.U: In some literature, SoftPlus referred to as Smoothed ReLU.. Recent work (Bresler & Nagaraj 2020) also considered smoothing ReLU using initial coefficients of its Fourier series. ReLU
is piecewise continuous linear, but with a gradient discontinuity. As shown in Chen & Ho| (011)); Montufar et al. (2014). the concept of piecewise linear can be extended to multiple pieces. We take a further step here and construct a piecewise continuous activation but also with a continuous gradient, by simply enforcing these conditions. The general idea is to define the activation piecewise linear and quadratic (resembling Huber loss (Huber 1992) on one side). On both sides, match ReLU and its gradients, and we fit quadratic middle region between. We define / to be the half-width of a symmetric transition region around x O. Note that this definition of SmeLU uses a parameter 3 that reciprocal the in Swish and other activations like in the sense that larger 3 here gives a wider transition region, and smaller 3 a narrower one. To the left of the transition region, y = O. To its right, y = x. Then, we enforce the gradients to be continuous by
(dy/ 1 0, (dy/dx) ==B 1.
(8)
Applying the conditions gives the SmeLU activation
O; < -3 (a++-9)2 4
ySmel.U
|0| < B
x;;
x 2 B.
5

---

Under review as a conference paper at ICLR 2021
Fig. (also Fig. 12 shows SmeLU and its continuous gradient as function of 3. It matches ReLU on both sides, giving a gentle middle transition region, wider with greater 3 and narrower with smaller 3. SmeLU is a convolution of ReLU with box of magnitude 1/(23) in 1-3, 3]. Its gradient is hard Sigmoid. As with other smooth activations, we observe that with greater 9, reproducibility is improved possibly at the expense of accuracy. Optimal accuracy on datasets we experimented with is for some 3 > O. For 3 - O, SmeLU becomes ReLU. Fig. 2 demonstrates the surface of the output of a network with two inputs as function of the values of these inputs, with ReLU, SmeLU with different B, and no activation (top). Surfaces are obtained with weight normalization. The bottom row shows SmeLU with 3 0.5 for different L2 weight norms. Examples of effects of normalization, clipping, and other activations on this surface are shown in Figs. 5 7 in Appendix A Fig. 5 demonstrates the importance of normalization with clipping activations. Fig. shows similar surfaces of other smooth activations. Fig. 7 shows effects of (other) normalizations. In Fig 2 we observe that ReLU exhibits multiple valleys (that lead to multiple optima in higher dimensions). Smoothing the activation by increasing / smoothes these vallcys. Thus with smooth activation, there are fewer optima that the model can find while training. each spanning a larger region around it. However, activations that are too smooth resemble linear models with worse aceuracy.
The parameter of SmeLU (as well as of other smooth activations) interacts with the normalization applied, as demonstrated in the second row of Fig. 2 Smoothing of the surface can be achieved by either tuning 33 or the norm of the normalization applied. This, however, is only possible with smooth activations. Tuning weight norm with ReLU does not smoothen the surface. Thus the smooth activation allows for smoother surfaces whose smoothness (and good reproducibility behavior) can be tuned by either the parameters of the activation or by normalization. For consistency, one can keep the normalization to norm of y = 1, and study the tradeoff as function of 3, or apply SmeLU with no normalization (which may affect accuracy and reproducibility) and no clipping.
An approach that may improve the overall PD/accuracy tradeoff is to use wider values of 3 for layers closer to the inputs and narrower /9 values for layers closer to the output. Reproducibility seems to be dominated by rich parameter sets, and applying wider | for layers that control such sets could improve it. Then, layers closer to the output can focus on better prediction accuracy. Alternatively, one can learn the values of [ for the whole model, per layer, or even for individual units, part of the network training. Some insights on how this can be done are given in Appendix
relu, woight norm
amelu, 9-0.5. weight norm
smelu, 8=10 weight norm
linear, weight norm
smelu, p-0.5 weight norm -10
smelu, 9-0.5, weight norn +3
smela, B-0.5, weight norn =1
smelu, 9-055 weight norm -0.5
Figure 2: Three dimensional surfaces showing network outputs as function of 2-dimensional inputs in 1-, 6), with 5 hidden layers of dimensions [256, 128, 64, 32, 16) activated by ReLU, SmeLU with different 9 parameters, and no activation (linear). Layers apply weight normalization, and there is no clipping of large activations. Matrices of all hidden layers are equal for all figures and are randomly drawn from a standard normal A (0, 1) distribution. Top: Different activations and SmeLU ,3 values. Bottom: SmeL. with B 0.5 for different Lgg weight norms.
Generalized SmeLU: The constraints in 8 can be generalized to wider family of activations, including ones such as a leaky SmeLU, by allowing different gradients g- # O to the left and 9+ 9- to the right of the quadratic transition region, and allowing for an asymmetric transition region, and for a bias shift, The conditions are then
(dy/ -=-a = (dy/ds)) = It. y( -a) =6 L.
(10)
6

---

Under review as a conference paper at ICLR 2021
The five parameters {a, 9., t} define generalized SmeLU. Typically (but not necessarily),
a, 3 > 0, and t S possibly to allow the activation to cross through the origin. For a leaky activation, 9- 0, but can also have negative slope the left (where then we relax the monotonicity requirement). Enforcing (10) and continuity of the function at I = -a and x = 8, come to the definition of the generalized SmeLU activation
--r --
I I aa - 5x 3
V/gSmeLU
ax² + bx +
(11)
g + t 2 2 + 8-8
9+:
I x2 B
where
9+ 9- 2(a + 8)
b b= 09+ + 8g
c=1+ a + g-) 20,8g
(12)
a a 8
2(a (3)
The fifth pair of graphs in Fig. (also Fig. 12) shows examples of generalized SmeLU with = 0 and asymmetrie transition regions, including leaky versions with g- 0 and also versions with
g- S 0. SmeLU is a special case of the generalized version, with g- = 0, g+ = 1, a B, and
t = 0. Some additional special cases are reviewed in Appendix [D]
The hyper-parameter parameters of generalized SmeLU can be learned in training of a deep network together with the model parameters, either for the whole model, per layer, and even per neuron, as noted for SmeL.U and discussed in Appendix B Experiments demonstrate that for some datasets, layers closer to inputs may learn a negative g-, implying that they learn some sparsification of the input, for better model accuracy. Layers closer to the output learned, in similar cases, slopes closer to O.
Further Generalization RESCU: The general idea to derive the generalized SmeLU can be ex- tended in various ways. Generally, we define REctified Smooth Continuous Unit (RESCU) any function that consists of multiple smooth pieces, and is continuously differentiable. At any point, (dy/ /x la. We can extend the generalized SmeLU
a, y(a_)= v(a+), well as (dy/dx)la by additional linear pieces and quadratic pieces connecting between any pair of linear pieces with different slopes. For example, the non-monotonicity of Swish (and alike) can be achieved by adding a concave piece to the left of the convex quadratic piece. At the point the gradient becomes 0 it can be joined by a linear piece to its left with 0 gradient.
Alternatively, the same concept can be used by joining pieces of more complex mathematical fune- tions, ensuring smoothness at the connection points between the pieces. This makes activations like Swish special cases of RESCU with a single piece. SELU with / = 1 is another example with two pieces. Another example is the REctified Sigmoid Continuous Unit, which consists of a Sigmoid on the left, but linear on the right
2 28-0 o
(2(=-0) 3
x S 3 x;
VSiS-RESCU
(13)
x 2
We can apply the concept with polynomials of higher degree well.
EXPERIMENTS
Criteo: We evaluated SmeLU and Swish on the benchmark Critco Display Advertising Challenge d The dataset provides a binary classification task with 45M examples, which are split to 37M training examples, and validation and test sets of 4M examples each. Each example has 13 integer features and 26 categorical features. Models have fully connected hidden layers of dimension 2572, 1454, 1596, respectively. Each categorical feature 3&& is hashed into N& buckets. 110 the hash bin is encoded as a one-hot vector, otherwise it is embedded as vector
If N₂ < of dimension dy. Values of N₂ and dę are taken from Ovadia et al. (2019) (see Appendix E Each model is trained for one epoch using the Adam optimizer. Unlike Ovadia et al. (2019), input features are not fed into batch-norm layer. Integer features are log-square square-transformect with ReLU, SmeL. and Swish activations. Each experiment consisted of 40 independent runs. For each run, models are initialized to different random values, and a different random shuffle is applied to the data. Training mini-batches of 1024 examples were used. Table shows AUC and PDs; A
We trained models
Criteo data in https: http://wwwwa.c -tm
http://wwaaalleeec////rttttttoot ay-aapppaaayaaeo ad-callhhal 1enge

---

Under review as a conference paper at ICLR 2021
Table Smooth activations Criteo: AUC, and PDs; A Ai.
AUC
A
A
AT
Aj
Model ReLU
AUC
stdev
stdev
%
stdev %
0.781
0.0144
0.053
0.033
36.3
19.2
SmeLU
B
0.783
0.0116
0.044
0.032
30.5
17.8
3 = 1.5
0.785
0.0095
0.037
0.028
28.6
15.0
8 2
0.786
0.0068
0.032
0.021
25.2
11.8
B = 2.5
0.787
0.0012
0.029
0.004
22.5
4.5
8 3
0.787
0.0012
0.029
0.005
23.5
5.7
3 = 4
0.787
0.0012
0.029
0.004
22.8
4.9
Swish
3
0.765
0.0208
0.088
0.006
45.1
29.4
B 0.5
0.774
0.0197
0.078
0.025
45.3
25.1
8 = 1
0.766
0.0211
0.087
0.009
45.3
26.5
B 2
0.757
0.0208
0.078
0.028
40.0
29.1
3 = 4
0.760
0.0170
0.076
0.022
39.5
27.1
(absolute) and Ai (relative), on the test data. ReLU exhibits huge PDs over 36%. While only slightly improving AUC, around 40% decrease in both PD measurements well substantial order of magnitude reductions in AUC and PD standard deviations, are observed with SmeL.U. Tables P and 3 Appendix [ show that similar less extreme improvements are attained when training data is not shuffled, or models are identically initialized. Baseline ReLU PDs are lower but still over 20%. SmeLU still improves PD (about 12% without shuffling) and AUC/PD standard deviations over ReLU. With Swish, on this dataset, we were unable to obtain comparable AUC and PD values.
ReLU
ReLU
SmeLU
SmeLU
Tanhexp
11
Tanhexp
Relative PD M
10
Swish
Relative PD i
10
Swish
Mish
Mish
GeLU
GeLU
SoftPlus
SoftPlus
8
8
6
-0.0 -0.05 0.00
0.05
0.10
0.15
-0.2
0.4 Rank Loss Change % vs. ReLU
Log Loss Change % vs. ReLU
Figure 3: Relative PD; Â7, expressed in [%] of the positive label prediction as function of loss change [%] from ReLU baseline for different activations on real validation data. The X-axis is: left: logarithmic cross entorpy loss, right: ranking loss.
Real Data: We tested various smooth activations large private dataset for ad Click-Through Rate (CTR) prediction for sponsored advertisement. We constructed simple 6 layer fully connected network with dimensions [1024, 512, 256, 128, 64, 16] with 5 informative features that are used as leamed embedding vectors of different dimensions, providing a total of 240 inputs. Models are identically initialized, trained with data-shuffling on hundreds billions examples, using online mini-batch, single pass over the data, optimizing with the Adagrad optimizer (Duchi et al. 201 Resources limit the quantity of models trained, so we use M Inference applied to examples prior to training them, and progressive validation (Blum et al. 1999). commonly used for this problem (MMMahan et al. 2013). applied to measure validation logarithmic and ranking losses on predictions. Fig. 3 reports relative PD; Åς, (as defined for binary labels) as function of both average validation logarithmic (cross-entropy) loss (left) and ranking loss (right). Losses expressed as percentage the baseline ReLU model. Relative PD expressed as percentage of the actual positive label prediction. On the graph, moving left implies better accuracy, and moving down
8

---

Under review as a conference paper at ICLR 2021
implies better reproducibility. For smooth activations, the different points the result of different 3. Moving down at least before reaching an optimum implies larger 3 for SmeLU and smaller 3 for the other activations. In such large scale system, we observe major relative PD values of over 10% with ReLU (12% when models initialized equally, and 13% when initialized differently). Smooth activations are substantially better PD and better the tradeoffs than ReLUU. Among themselves, the tradeoff's are more comparable, where SmeLU, GELU and Swish appear to be slightly better on this data than Mish and Tanhexp. SoftPlus appears to push towards better PD, but at the expense of accuracy with unacceptable trade-offs (in such a system even a fraction of percent can make a difference). Continuously differential SELU appears inferior accuracy and PD. SmeLU also seems to be stronger on PD, whereas Swish on accuracy. Results shown use weight normalization with norm without clipping. We observe similar relative behavior without normalization (no clipping). However, PD's of smooth activations increase, but are still better than the ReLU baseline.
ReLU
0.016
ReLU
Prediction Difference A
0.0068
0.25
SmeLU
Prediction Difference A
SmeLU
Swish
Swish
Softplus
0.015
0.25
Softplus
0.0066
1.5
0.014
0.0064
0.5
0.5
0.013
3.5
4
0.0062
2.5
0.012
0.9
0.0060
0.9
1.0
1.0
1.2
1.8
Error Rate %
Error Rate %
Figure 4: PD A as function of error rate on MNIST dataset for different activations with different parameters. Left: with Adagrad optimizer, right: with SGD optimizer.
Benchmark MNIST Dataset: We tested PD and accuracy on the MNIST dataset (LeCun) 1998). Models were implemented with Tensorflow Keras. Networks were simple layer networks, with layers of width 1200. We regularized with dropout (input 0.2, layers 0.5) and image aug- mentation in training (starting from epoch 2, two dimensional independent random shifts by (-3.-2.-1, -, 1, 2,2,3) pixels take place with probability 0.5.) Hidden weights were identically initialized with the default Keras uniform random, and biases to O. For AdaGrad, used learning rate 0.02, accumulator initialization 0.1, and 150 epochs without weight normalization. For SGD, used learning rate 0.01, momentum 0.9, and 50 epochs. Batch size is 32. For each experiment, we trained M = 12 models. Fig. 4 shows PD A as function of error rate for both configura- tions, AdaGrad (left) and SGD (right). Graphs for other PD metrics, with similar behavior, are in Appendix FF PD values may not be as large as they are in large scale data, but we still see PD change substantially relative its lowest values (over 10% for AdaGrad, over 30% for SGD). As s in large scale, ReLU's PD is much worse than those achievable by smooth activations, and among themselves, smooth activations are comparable. With SGD, however, Swish appears inferior on PD.
CONCLUSIONS
We studied irreproducibility in deep networks, showing that the popular ReLU activation is major factor in exacerbating prediction differences. Smooth activations allow substantial improvement, giving better recuuucyyyyyoduucbbilttt tradeoffs. We introduced SmeLU, and its generalizations, specifically demonstrating that superior recuracy-eprooduciblitt tradeoffs are achievable even with very simple inexpensive to implement activations, whose performance on both accuracy and PD do not fall from and can be better than those of more complex expensive to implement, and non- monotonic, smooth activations. We empircially demonstrated these tradeoffs, showing that proper smooth activations configuration, one smooth activation may be better than the other. We have shown, however, that the much simpler SmeL.U be comparable superior the more complex activations.
superior ReLU. For specific dataset. model architccture, and model

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep neural networks. arXiv preprint arXiv:1711.08856 2017.
Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George Dahl, and Geoffrey Hinton. Large scale distributed neural network training through online distillation. arXiv preprint arXiv: 1804.03235 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 167.006007004500 2016.
Jonathan T Barron. Continuously differentiable exponential linear units. arXiv, pp. arXiv-1704, 2017.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning. pp- 41 41-48, 2009,
Avrim Blum, Adam Kalai, and John Langford. Beating the hold-out: Bounds for k-fold and progres- sive cross- volssvvaliitioon. In Proceedings of the twelfth annual conference on Computational learning theory, pp. 203-208, 1999.
Guy Bresler and Dheeraj Nagaraj. A corrective view of neural networks: Representation, memo- rization and learning. In Jacob Abernethy and Shivani Agarwal (eds.), COLT 2020, volume 125 of Proceedings of Machine Learning Research, pp. 848-901. PMLR, 09-12 Jul 2020.
Zhe Chen, Yuyan Wang, Dong Lin, Derek Cheng, Lichan Hong, Ed Chi, and Claire Cui. Beyond point estimate: Inferring ensemble prediction variation from neuron activation strength in recom- mender systems. arXiv preprint arXiv:2008. 2020.
Zhi Chen and Pin-han Ho. Deep global-connctted net with the generalized multi- multi-piccewise relu activation in deep learning. arXiv preprint arXiv:1807.03/1 2018
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 1xXi:1511.07288 2015.
T. G. Dietterich. Ensemble methods in machine learning. Lecture Notes in Computer Science, pp- 1-15, 2000.
Simon Du. Gradient Descent for Non-convex Problems in Modern Machine Learning. PhD thesis, Camegie Mellon University, 2019.
J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159, Feb. 2011.
Michael W Dusenberry, Dustin Tran, Edward Choi, Jonas Kemp, Jeremy Nixon, Ghassen Jerfel, Katherine Heller, and Andrew M Dai. Analyzing the role of model uncertainty for electronic health records. In Proceedings of the ACM Conference on Health, Inference, and Learning, pp- 204 204, 2020.
Dan Hendrycks and Kevin Gimpel. arXiv:1606.084/35 2016.
Gaussian error linear units (gelus). arXiv preprint
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.025.: 2015.
Peter Huber. Robust estimation of location parameter. In Breakthroughs in statistics, pp. 492- 518. Springer, 1992.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint 1rXiv:1502.033 2015.
Xiaojie Jin, Chunyan Xu, Jiashi Feng, Yunchao Wei, Junjun Xiong, and Shuicheng Yan. Deep learning with s-shaped rectified linear activation units. arXiv preprint arXiv: arXiv: 15/2000000 2015.
10

---

Under review as a conference paper at ICLR 2021
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In Advances in neural information processing systems, pp. 971-980, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. In Advances in neural information processing systems, pp. 6402-6413, 2017.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmist// 1998.
Dong Lin and Gil I. Shamir. Private communication to Quoe Le. Unpublished, Aug. 2019.
Hanxiao Liu, Andrew Brock, Karen Simonyan, and Quoc V Le. Evolving normalization-activall
layers. arXiv preprint arXiv:2004.02967. 2020.
Xinyu Liu and Xiaoguang Di. Tanhexp: A smooth activation function with high convergence speed for lightweight neural networks. arXiv preprint arXiv:2003.0966 00955, 2020.
Vishnu Suresh Lokhande, Songwong Tasneeyapant, Abhay Venkatesh, Sathya N Ravi, and Vikas Singh. Generating accurate pseudo-labels in semi-supervised learning and avoiding overconfident predictions via hermite polynomial activations. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. [1435-1143,, 2020.
H Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, et al. Ad click prediction: a view from the trenches. In Proceedings of the 19th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 1222-1230, 2013.
HN Mhaskar. On smooth activation functions. In Mathematics of Neural Networks, pp. 275-279. Springer, 1997.
Diganta Misra. Mish: A self regularized non-monotonic neural activation function. arXiv preprint arXiv:1908.0 08681, 2019.
Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. the number of linear regions of deep neural networks. In Advances in neural information processing systems, pp. 2924-2932, 2014.
Prabhat Nagarajan, Garrett Warnell, and Peter Stone. Deterministic implementations for repro- ducibility in deep reinforcement learning. arXiv preprint arXiv:1809.05676, 2018.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Chigozie Nwankpa, Winifred Ijomah, Anthony Gachagan, and Stephen Marshall. Activation functions: Comparison of trends in practice and research for deep learning. arXiv preprint arXiv:1811.03378, 2018
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snock. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances Neural Information Process- ing Systems, pp- 13991-14000 2019.
Dabal Pedamonti. Comparison of non-linear activation functions for deep neural networks mnist classification task. arXiv preprint 1804v:100..0777 2018
Prajit Ramachandran, Barret Zoph, and Quoc Le. Searching for activation functions. arXiv preprint arXiv: 1110.0:71000594/1 2017.
Flora Sakketou and Nicholas Ampazis. On the invariance of the selu activation function algorithm and hyperparameter selection in neural network recommenders. In IFIP International Conference Artificial Intelligence Applications and Innovations, pp- 673-685. Springer, 2019.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization accelerate training of deep neural networks. In Advances neural information processing systems, pp. 901- 909, 2016.
11

---

Under review as a conference paper at ICLR 2021
Gil I Shamir and Lorenzo Coviello. Anti-iistillation Improving reproducibility of deep networks. Preprint, 2020.
Tianyang Wang, Zhengrui Qin, and Michelle Zhu, An elu network with total variation for image denoising. In International Conference on Neural Information Processing, pp. 227-237. Springer, 2017.
Cihang Xie, Mingxing Tan, Boqing Gong, Alan Yuille, and Quoe V Le. Smooth adversarial training. arXiv preprint arXiv:2006. 112006.145550 14536, 2020.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4320- 4328, 2018
Hao Zheng, Zhanlei Yang, Wenju Liu, Jizhong Liang, and Yanpeng Li. Improving deep neural net- works using softplus units. In 2015 International Joint Conference on Neural Networks (IJCNN) pp. 1-4. IEEE, 2015.
A SMOOTH ACTIVATIONS AND NORMALIZATION INDUCED SURFACES
In this appendix, we demonstrate effects of smooth activations and normalization on the objective surfaces, enhancing on the description in Sections 3 and on the illustrations of Fig. 2 Fig. 5 ex- pands on Fig. 2 showing objective surfaces for different activations, parameters and normalizations. shows the transition from a non-smooth ReLU to a linear activation through different parameters of SmeLU. As we increase B, the objective surface becomes smoother. It also demonstrates the importance normalization, especially when activations are clipped to some cap. The first row duplicates that of Fig. for completeness. The second row shows the transition of a weight normal- ized clipped activation from ReLU to linear through widening SmeLUs. With weight normalization there are slight deviations from the top row, but the behavior appears rather similar. The third row shows the same transition without weight normalization and clipping. In order to be as smooth, a larger |3 must be used than in the first row. This result is in line with the experiments reported in Section 5 on real data, where PD values increased in all cases with the same 3 parameters if weight normalization was not applied. Furthermore, the objective variance is now much larger than in the first row. When clipping is added, as shown in the last row, the effect of the smoothness disappears, where numerous peaks appear all activations. With large 3, the effectiveness
the activation altogether also appears to diminish. This happens even with the linear activation. As observed in the second row, weight normalization diminishes the effect of clipping.
Fig. 6 shows surfaces for Swish, SoftPlus, GELU and SELU with weight normalization, without clipping. For this simple two dimensional input example, the smooth activations appear rather sim- ilar to the surfaces of SmeLU. As mentioned carlier, the smoothness is reversed as a function of 33 from SmeLU, with the larger 3 values being less smooth. Since GELU can be approximated by Swish with a larger parameter, for the same B, Swish is smoother. Very similar surfaces are observed for Mish and Tanhexp. SELU is smooth for 3 1, but is clearly not smooth for other values of 3. This is observed in Fig. 6 for 10.
Fig. demonstrates the effect of normalization (weight and layer) on no activation and on ReI.U, and the effects of layer normalization (Ba al. 2016)) on SmeLU. Unlike weight normalization, layer normalization changes the shape of the surface from the shape obtained when there is no normaliza- tion. While the type of norm changes the shape of the surface obtained, unlike smooth activations, the norm value v has no effect on the shape of the surface observed for either no activation or ReLU. (It does affect the magnitude of the objective, though.) For smooth activations, on the other hand, as observed in Fig. 2 the magnitude of the norm does affect the smoothness of the surface also with layer normalization, as demonstrated for SmeLU on the bottom row, with smoother objective as the norm is smaller.
B LEARNING SMELU PARAMETERS
As we observed, 3 in smooth activations roughly serves as a tuning knob between accuracy and reproducibility. However, the optimal accuracy is attained away from the end point (0 for SmeL.U,
12

---

Under review as a conference paper at ICLR 2021
relu, woight norm
smelu, woight norm
smelu, weight norm
weight norm
relu, weight norm, clip-6
smelu, B-0.5, weight norm, clip-6
omelu, B- 10, weight norm, clip-G
linear, weight norm, clip-5
rolu
smelu, Br5
smelu, B=20
licorr
relu, clip=8
smelu,
smelu, 8:20
linear. dlips
Figure 5: Three dimensional surfaces showing network outputs as function of 2-dimensional inputs in [- 6). with 5 hidden layers of dimensions [256, 128, 64, 32, 16] with no activation (linear), and activated by ReLU, and SmeLU with different 33 parameters. Matrices of all hidden layers are equal for all figures and are randomly drawn from standard normal N(0,, 1) distribution. Top: ReLU, SmeLU different 3 and linear, no clipping of pre-activation values with L2 norm 1 weight normalization. Second Row: weight normalization with activation clipping to [-6,6] 6). Third Row: no clipping, no weight normalization. Bottom Row: clipping, no weight normalization.
swish, 9-10, weight norm
swish, B-t, we neight norm
softpius, B-1, weight norm
sottplus, 9e0.01, weight norm
galu, 8:10, weight
gelu, 8:1. weight norm
selu, B=10, weight norm
selu, Rct. weight norm
Figure 6: Three dimensional surfaces with identical configuration Fig. 5 for Swish, Softplus, GELU, and SELU, with different 3 parameters, all with weight normalization and no clipping. Top: Swish and Softplus. Bottom: GELU and SELU.
13

---

Under review as a conference paper at ICLR 2021
linear, weight norm =50
linear, weight norm
linear, layer norm =50
Inear, layer norm
relu, weight norm :50
relu, weight norm =0.15
retu, layer norm =50
retu, layer norm =0.15
ameks, 9-0.5, norm =50
amelu, 9:0.5. leyer norm =3
smelu, B-0.5, norm
smetu, B-0.5 layer norm -0
Figure 7: Three dimensional surfaces with identical configuration to Fig. 5 for no activation, ReLU and Smelu (3 = 0.5) with weight and layer normalizations, with no clipping- Top: Linear with different weight and layer norms. Middle: ReLU with different weight and layer norms. Bottom: SmeLU with different layer norms.
0o for Swish and other smooth activations). Similarly, there are no guarantees that the wider is the activation the more reproducible it is, at the other extreme. It is expected that the optimal points will be properties of both the actual dataset and the model architecture. For the generalized SmeLU and wider RESCU generalization, more parameters should be tuned. While tuning can optimize various tasks, we focus here on tuning for accuracy and reproducibility.
The parameter or the parameters of the generalized SmeLU 9...3.9994.. t) can be trained and learned as part of the training process of deep network. Learning can optimize accuracy, repro- ducibility or any other objective. Training can learn single parameter set for the whole network, a single parameter set per layer, or single parameter set per neuron. Depending on the task and num- Initialization
ber of updates, learning rates can be tuned for each of the learned can initialize to known good parameters. For the generalized SmeLU, we can initialize values to those of a reasonable SmeLU, e.g., with g- = 0, g+ = 1, t = O, and a = 3 = 0.5. Empirically, we observe, when optimizing for accuracy, that each layer learns a different set of parameters. Specifi- cally, we observed that layers closer to the input tend to learn a negative g-. The first hidden layer that is connected to the input (especially in systems where the input consists of learned embeddings) feeds input that is usually more symmetric around the origin, while layers farther from the input see biased inputs. With the learned g-, appears that the layer closest the input attempts learn sparsification, forcing its inputs either to an extreme or to the origin.
Learning hyper-parameters for accuracy can be done simultaneously with training the parameters of the networks. Alternatively, this can be done in one training session. Then, the can be fixed, and used for training the parameters. The procedure can be further applied to learning functional forms of the activation as done by Raaachandran et al. (2017). One can apply such a methodology on a RESCU activation, where the pieces are learned under the constraints of contin- uous function and gradient between the pieces.
Optimization of the parameters can combine an objective for both accuracy and reproducibility. as a mean or weighted sum of both objectives. However, this must be done carefully, as these objectives may be conflicting. Optimizing the activation for reprodueibility may be more involved, as gradient methods with the objective loss used for training neural networks tend to try to improve the model's ' accuracy. There are several options for training the activation parameters to improve reproducibility. We can include reproducibility optimization of the parameters in training the full network, or train
14

---

Under review as a conference paper at ICLR 2021
the model offline while learning the parameters that are better for reproducibility first, and then peat training to learn the model itself using these parameters. In either case, ensembles can be used as proxy for reproducibility, where impose loss that minimizes the prediction difference (either in probability or on log-odds score) between components of the ensemble. This approach essentially applies co-distillation (Anil al., 2018) but for the purpose of learning the paremmeraseterss that improve reproducibility instead of training the actual model. Co-trained ensemble may not mimic the full effect causing divergence of models they may have less randomness in the order of exam- ples they see and updates they apply, but empirical results show that even with that components of ensemble tend to diverge from one another, even when initialized identically, and especially when initialized differently. Unlike co-distillation, to learn the hyper-pale pprrmppareeetess for better reproducibilityy, the objective loss in training the ensemble is applied only on the parameters, but the similarity loss is applied only on the hyper-parameters.
There are disadvantages to learning the hyper-parameters while training. This is because of po- tentially conflicting objectives to accuracy and reproducibility. Furthermore, in many cases, the preferred deployed model should not be an ensemble. We can use an ensemble to optimize the hyper-parametert for reproducibility offline, and then train and deploy model that retrains with the reproducibility optimized activation hyper- pppee--rrametes....
If we deploy ensemble, training with identically initialized components may be valuable for opti- mizing activation nyper-parameters for better reproducibility. but will be suboptimal for deployment, at least for better reproducibility. As shown by (Shamir & Coviello (020), ensembles advanta- geous for reproducibility by leveraging and encouraging the differences between their components, so that as whole, the ensemble captures the objective space better. Initializing components iden- tically for optimizing the hyper-parameters defeats this idea, and thus it is more desirable to apply such initialization only when learning the activation papee-parametes, but then retrain the models with these parameters with different initializations of the components, for better deployed repro- ducibility.
C GRAPHS OF DIFFERENT SMOOTH ACTIVATIONS
In this appendix, we provide activation function and gradient curves for the different (smooth) acti- vations discussed in this paper as function of their parameter. This set of curves completes the ones shown in Fig. I Fig 8 shows curves for the different activations with 8 = 1. Figs. T0 II and
12| show SELU and CELU, SoftPlus, Swish, GELU, Mish, Tanhexp, SmeLU, generalized SmeLU, and Sigmoid-RESCU respectively. SELU is shown to be smooth only for B 1. The equations for CELU are given by
x, cxp 1)
if 2
VCELU
B
if
(14)
We observe the close similarity between Swish, GELU, Mish, and Tanhexp, and the relation between Sggmoid-RESCU and SmeL.U. Subsequently to our work, and inspired by SmeLU, (Lin & Shamir 2019). a different SmoothRelu activation was given in Xie et al. (2020).
log (8x + 1), if x 0,
ySmoothRelu
(15)
if <.
This activation is also RESCU activation. Graphs for this activation are included in Fig. 12] As observed, this activation does not achieve full slope 1, unless >>> 1, in which case, it already approaches ReLU. Thus it may lack the smoothness tuning flcxibilities of SmeLU.
D GENERALIZED SMELU REFORMULATION AND SOME SPECIAL CASES
In this appendix we show some special cases of the generalized SmeLU defined in and also address the asymptotic relation between SmeL.U and SoftPlus.
D.1 ASYMMETRIC SMELU
SmeLU is a special case of the generalized SmeLU, The definition of the generalized SmeLU in can be used to obtain version of SmeL.U that is asymmetric around the origin. Like SmeL.U,
15

---

Under review as a conference paper at ICLR 2021
Figure 8: Different activations and their gradients for B = 1.
selu
selu
gradient
1401
celu
calu
gradient
Figure 9: SELU (top) and CELU (bottom) activations and gradients with different 3 parameter values.
16

---

Under review as a conference paper at ICLR 2021
softplus
softplus
gradient
Figure 10: SoftPlus activations and gradients with different 3 parameter values.
this activation can guarantee 0 output on the left, and slope on the right, but a transition region in [-a, B8. where # 3.
0:
x 8
-a
JAsymmetric-Seel.U
2
(o+a)²
-a
B
(16)
0-8
2 3.
Setting 3 gives SmeLU.
D.2 LEAKY SMELU
Setting 9- O. but g+ = 9-, 3 and t O, gives the SmeLU version of Leaky ReLU,
referred to as Leaky SmeLU, and given by
9- (x + 3);
S -B
LLeaky-SmeLU
43 x g- 3
1-9 x +
1+9_
x [1 3g-);
|>| S 8
(17)
I 2
Apart for the middle region, this activation resembles Leaky ReLU.
D.3 SHIFTED SMELU
The constraint used generate the generalized SmeLU constrains the g- slope region end the point = -a, t. Constraints can be specified differently, where this region ends at the origin, and then shifted horizontally and/or vertically. Vertical shift given by t, whereas we can define the activation xxx) = y( s) for horizontal shift by s.
D.4 ORIGIN CROSSING SMELU
SmeLU defined in (9) does not cross the origin. We can similarly define version of generalized SmeLU that is constrained to cross the origin instead of a point (-a, t). We can apply the general- ized SmeLU with the desired a, 3 and gradients, and then shift it vertically cross the origin (or we can solve directly with the new constraints). A zero crossing version keeps the sign of the input hand, but does suppress negative values to O. The form is given by
g
2at
(g- g+).
I S -a
Veero-cross-SmeLL
01 01-9- 2(a+
x² + ag +.g
-a S S
(18)
L -91
9+ +
> 3.
2a+
17

---

Under review as a conference paper at ICLR 2021
swish
swish
gradient
gelu
gelu
gradient
mish
mish
0.2
gradient
tanhexp
tanhexp
gradient
Figure 11: Swish, GELU, Mish, and TanhExp activations and gradients with different 3 parameter values.
18

---

Under review as a conference paper at ICLR 2021
smelu
smelu
12
gradient
25
gsmelu
gsmelu
gradient
16
gradient
smooth relu
smooth relu
82
gradient
Figure 12: SmeLU, Generalized SmeLU, Sigmoid-RESCU and SmoothRelu activations and gradi- ents with different a, and gradient parameter values.
19

---

Under review as a conference paper at ICLR 2021
D.5 ASYMPTOTIC RELATION OF SMELU AND SOFTPLUS
The gradient of the SmeLU function a hard Sigmoid. This already shows a connection to SoftPlus, whose gradient is Sigmoid. To further demonstrate the relation between SmeLU and SoftPlus, express SoftPlus using a reciprocal parameter 7 to the original definition in (3)
y/oftPlus = 7 In (1 + exp(x/7)).
(19)
Asymptotically, for x -0, we have y 0. For > 0. y 2 x. For |2| < 1, we can approximate the SoftPlus using Taylor series expansion by
x- x² In
M/ontPlls
(20)
2 87
Assigning the SmeLU parameter 3 = 2 we obtain for |r|
<< 1,
3 4 2
3
1
VSonPlus (x) 1/SmeLu (x) ~ y In
1n
4 h
(21)
2
Thus the middle region around x: 0 of SoftPlus is a vertical shift by 8/41n(4/e of the middle region of SmeLU with factor of 2 wider parameter, but the extremes on both sides meet those of SmeLU. Therefore, SoftPlus opens wider, away from the origin, making it even smoother for the same parameter, on one hand, but also potentially less accurate on the other, due to the distance of the middle region from the origin.
E EXPERIMENTS ON THE CRITEO DATASET
We describe the set up of the experiments the Criteo benchmark dataset and provide additional results.
Model architecture: Models have 3 fully connected hidden layers of dimension 2572, 1454, 1596 respectively. Each categorical feature I& is hashed into Nu buckets. N₂ < 110 the hash bin is encoded one-hot vector, otherwise it is embedded and dg are taken from Ovadia et al. (2019) and are the following (where dg = means that the hash bin is encoded as a one-hot vector):
vector of dimension d.. Values of N₂
Nk [1373, 2148, 4847, 9781, 306, 3591, 2798, 14, 7403, 2511, 5598, 9501,
46, 4753, 4056, 23, 3828, 5856, 12, 4226, 23, 3098, 494, 5087]
(22)
dk = [,,9, 29, 11, 17,0, 14, 0, 12, 19, 24, 29, 0, 13, 25, 0, 29, 0, 22, 0,0,01,0,,, 29] (23)
Unlike the set up in Ovadia al. (019), input features are not fed into batch-norm layer, while integer features are ng-squuare-transformmeld
Table Table shows results for ReLU, SmeLU and Swish when the training data shuffled, As a consequence, all models visit the data in the same order of batches, but are initialized differently. (Note, however, that due randomness in the platform, within a batch, there could still be some randomness in the order examples are being processed.) Table shhows results for ReLU, SmeLU and Swish when all models have the same initialization of the hidden layer weights. both cases, models exhibit irreproducibility. Without shuffling, as result of initialization (and random effect in the platform that applies the updates). With identical initialization, irreprodueibilityy is due to the training data shuffling. In both case, SmeLU improves A and A by about 12%, and exhibits a factor reduction of the standard deviations of both AUC and A
Additional results:
shows
F ADDITIONAL RESULTS ON MNIST
In this appendix, we show matching graphs to those in Fig. for the other PD metrics; Az. Af, and A in Figs 13 14 and 15 respectively. The behavior of the different activations shown in these figures for all the metrics is very similar to the behavior w.r.t. A1.
20

---

Under review as a conference paper at ICLR 2021
Table 2: Smooth activations on Criteo: AUC, A1, Ai.
Training data is shuffled.
AUC
4
A stdev
A
f
Aj
Model
AUC
stdev
%
stdev
ReLU
0.786
0.0068
0.028
0.014
21.5
8.3
SmeLU
B 1
0.787
0.0008
0.025
0.003
19.3
3.4
8 = 1.5
0.787
0.0007
0.025
0.003
19.0
2.9
B 2
0.787
0.0011
0.025
0.004
19.1
3.7
B = 2.5
0.786
0.0011
0.025
0.003
19.2
2.8
0.787
0.0011
0.025
0.003
18.4
2.8
B 4
0.787
0.0007
0.025
0.003
20.5
4.6
Swish
B = 0.25
0.769
0.0203
0.087
0.018
47.2
28.3
3 0.5
0.765
0.0216
0.087
0.006
44.3
32.8
3 0.768
0.0215
0.085
0.012
46.2
30.1
8 2
0.756
0.0180
0.070
0.030
37.4
29.9
B =
0.764
0.0180
0.082
0.010
46.6
23.1
Table 3: Smooth activations Criteo: AUC, A1, Aj. Models identically initialized.
AUC
A1
A1
A
A
Model
AUC
stdev
stdev
%
stdev
ReLU
0.787
0.0023
0.031
0.007
25.3
7.2
SmeLU
8 1
0.787
0.0004
0.028
0.004
22.2
5.0
8 = 1.5
0.787
0.0004
0.029
0.004
23.5
5.8
8 2
0.787
0.0003
0.029
0.004
23.6
6.4
B = 2.5
0.787
0.0005
0.028
0.004
22.5
5.6
B
0.787
0.0004
0.027
0.004
22.4
5.6
3 4
0.787
0.0003
0.027
0.004
22.4
5.7
Swish
3 25
0.767
0.0210
0.088
0.012
47.5
28.3
8 0.770
8 0.5
0.0207
0.083
0.015
45.9
27.5
0.769
0.0204
0.084
0.013
46.7
26.1
8 2
0.763
0.0204
0.085
0.012
43.4
27.9
B =
0.762
0.0181
0.083
0.014
45.3
24.3
0.0046
ReLU
ReLU SmeLU
0.0045
SmeLU
0.0100
Prediction Difference An
Prediction Difference A2
0.25
Swish
Swish
0.0044
Saftplus
0.0095
0.25
Softplus
0.0043
0.0090
0.0042
0.5
0.5
0.0041
1.5
0.8
0.0080
0.0040
0.9
2.5
0.8
0.9
1.0
1.1
1.2
1.0
1.2
1.4
1.6
1.8
Error Rate
Error Rate %
Figure 13: PD A2 as function of error rate on MNIST dataset for different activations with different parameters. Left: with Adagrad optimizer, right: with an SGD optimizer.
21

---

Under review as a conference paper at ICLR 2021
0.0072
ReLU SmeLU
0.0140
Prediction Difference 4
0.0070
Prediction Difference d
Swish
0.0135
0.0068
0.25
Softplus
0.0130
0.0066
0.25
0.0125
0.0064
0.5
0.0120
0.5
ReLU SmeLU
0.0062
1.5
1.5
0.0115
0.0060
066
Swish
0.9
Softplus
0.0058
0.8
0.9
1.0
1.1
1.2
1.0
1.4
1.6
1.8
Error Rate
Error Rate %
Figure 14: PD Af function of error rate MNIST dataset for different activations with different parameters. Left: with Adagrad optimizer, right: with an SGD optimizer.
0.0040
0.0080
ReLU SmeLU
Prediction Difference AH
0.0038
Swish
1.5
Prediction Difference
0.0075
Softplus
0.0036
2.5
0.0034
0.0070
2.0.55
0.25
0.0032
0.5
ReLU
0.0030
SmeLU
0.0065
Swish Softplus
0.0028
0.0060
0.8
0.9
1.0
11
1.0
1.2
1.4
1.6
1.8
Error Rate %
Error Rate %
Figure 15: PD A function of error rate MNIST dataset for different activations with different 3 parameters. Left: with Adagrad optimizer, right: with an SGD optimizer.
22


Reviews:
[{"review_id": "trGkmnNKawH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper claims that smooth activations are more reproducible than ReLU. The accuracy gain claims seem marginal and not carefully carried out, further ablation studies are needed to strengthen the conclusion on accuracy. However, the main point of the paper is reproducibility where the feature is measured by the ‘Prediction Difference’. PD (introduced in section 2) is a measure over a set of models where the PD score is low if the models output consistent estimates for the same validation samples. \n\nDo models with the same initialization, and same randomness seed for the shuffles of SGD have high PD? If so then there would be numerical issues in the way models are trained. Otherwise, where does the difference in PD come from? Why such fluctuations are considered irreproducible? Why would PD be identified with reproducibility?\n\nOverall the paper has a consistent story to tell. It’s a fresh perspective and focuses on the main problem at hand. However, I fail to understand some of the key measurements and their connection with reproducibility. It is undeniable that the shape of the landscape and the consistency of the models are linked. However, the variation found by ReLU’s could help increase ensemble accuracy. In other words, such variation can be a feature depending on the context. And, in principle, given the proper seeds and versions of the software, the results should be fully reproducible. Am I missing something? I am looking forward to the responses from the authors on this point. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "MDI9qFsgOVf", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper addresses the problem that deep neural networks (DNNs) can lead to different predictions (even when they are initialized the same way) due to the stochasticity of samples selected in mini-batch SGD and update procedures from different optimizers, which leads to convergence to different regions along the loss surface.  They attribute this problem to the complicated loss surface that arises from the discontinuity in relu activations. They show that smooth activations can help remedy this issue, by tuning the activation to become more relu-like, which leads to a better tradeoff between prediction differences (i.e. consistency) and model accuracy.  \n\nThe pitch of the problem is motivated by fields like healthcare where more consistent predictions are important. However, it’s unclear why one would expect reproducible predictions when two network (with same initialization) are trained with a different sampling of mini-batch shuffles. This stochasticity should lead to different solutions for complex surfaces, just as a different initialization would. Smoothing the loss surface can help here and this paper only explores the extent that the smoothness of the activation plays a role. It’s unclear to me whether the activation function alone is sufficient for solving this problem. Other options, such as regularization, are not explored.\n\nComments:\n* It is nice that traditional activations like swish and softplus are parameterized with a beta so that they can be modulated to become more relu-like. Given the stochasticity of mini-batching, it’s not surprising that you can start from the same initialization and navigate to a different region of the loss surface. Since the stochasiticity is due to the size of the mini-batch, this paper could benefit from exploring how the effect size of PDs changes with batch size. \n* Overall, the empirical evidence is very light — they compare different activation functions for a private dataset for ad Click-through-rate and MNIST. To make general claims, there should be various networks and various optimizers on various datasets. The paper can be more convincing if more datasets are explored, preferably not on a private dataset for which no one can validate their results. \n* Also, the optimizer plays a major role in navigating the loss surface, but only a single optimizer is compared in the first task, while only 2 are explored in task 2 (i.e. adagrad and sgd). Adding more optimizers, especially popular ones like adam, could help clarify whether the smooth activations is robust across optimizers or whether this is a special case for adagrad. I suspect the former, but experiments must be performed to prove any point. \n* On a smaller note, what does each dot in Fig 3 represent. Assuming it represents different beta values, there should be more consistency -- softplus has many points while tanhexp only has 4 points. Also, how do these betas alter the activation functions -- the  main/appendix show plots for 5 values of beta. Also, in Fig 4, why aren’t the same betas used for adagrad and sgd?  The choices of beta for a given base activation function should be consistent throughout. \n* Fig 4 should also show the standard deviation across the 12 models. This can provide a sense for the statistical significance of the results. \n* RESCU, a generalized activation function, is introduced but they never used in any comparisons. What is the benefit of such a formulation? This paper mentions that smelu is less expensive compared to the other smooth activations. Any direct statement like this should be followed up with quantitative comparisons — in this case the time per epoch. \n* This paper shows how weight normalization also influences the loss surface, but was not explored empirically. I think adding additional experiments taht scan different weight norms for a fixed beta would only strengthen their claims.\n* It is unclear whether their initialization strategy for the DNNs explored here is fixed or whether they simply sampled different weights from the same initialization distribution. The wording of the motivation was that PD happens for the same initialization but the language in the text was ambiguous of whether they enforced the same random number seed for the initializer across each experiment. \n* As a control experiment, the effect size of prediction differences between training a model with different initializations should be compared with their main results that explore training with different mini-batch order starting from the same initialization. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Clvpch4dzB2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission studies how the choice of activation function impacts the reproducibility of experiments involving deep networks. It proposes a new activation function with the goal of designing a smoothed ReLU, and provide experiments comparing it against other activations in terms of irreproducibility (measured via PD) and performance.\n\nThe problem of understanding how model design choices can have negative impacts on experimental reproducibility is interesting and timely, but I believe the paper does not provide a strong enough case for their approach and contributions.\n\nFirst, the adopted metric to measure irreproducibility, 'Prediction Difference (PD)', is never evaluated in terms of how sensible of a metric it is to capture reproducibility -- this also seems to be lacking in [1]. Actually , one can argue that it is not a sensible metric at all (except for its Hamming form), as it is not invariant to how the models are calibrated, as discussed below.\n\nFor example, take any binary classifier and consider two copies of it with different calibrations (i.e. scaling the output layer weights by positive scalars, one for each model): even though the models always agree on their predicted labels regardless of their calibrations, the PD can be made arbitrarily close to 0.5 by calibrating the models appropriately. Even more worrying is that the same can be done by taking a binary classifier and a copy of it with flipped predictions: the PD between the two can be made arbitrarily close to 0 by scaling their weights down. Note that this problem also happens with the relative PD.\n\nTo see how this is connected to the choice of activation functions (especially ReLU x SmeLU), note that for normally-distributed inputs (centered around the origin), gradient's variance of ReLU is 1/4 while for SmeLU it is approximately sigma^2 / (4 beta^2) for large enough beta (sigma^2 being the variance of the input distribution): this discrepancy can have a non-trivial impact on the model's calibration and cause differences in PD to be artifacts.\n\nSince this doesn't happen with the Hamming form of PD I believe Figure 15 in the Appendix to be the most informative one. However, it seems that different activations result in less than 1% prediction discrepancy across models, which is fairly insignificant and hence it is hard to argue that activations actually matter for reproducibility (at least from the presented experiments).\n\nLastly, it is hard to draw any conclusions from the presented experiments: the CTR results are based on a private dataset while the MNIST ones are extremely small-scale, with both the dataset and the model being arguably toy problems. There are numerous tasks where reproducibility is a prominent issue e.g. deep reinforcement learning, generative modelling (especially GANs), making training a 2-layer network on MNIST a poor choice to evaluate reproducibility problems.\n\nAs an additional note, the authors seem to rely heavily on the work of Shamir & Coviello '20 [1] which introduced the PD metric, even though the paper was only made publicly available on arXiv a week --after-- the reviewing period for this submission started. When citing papers which are yet to be made available it would be helpful to introduce and discuss the relevant content in a self-contained way -- while the authors avoided much of my confusion by presenting the full definition of the PD metric, the referred paper has useful information which was not discussed (such as which summand is normalized in its relative form and how the different variants compare).\n\nSince I have major concerns with the paper -- particularly on the reliability of PD as a metric and the unconvincing empirical results -- I am voting for rejection.\n\n[1] Shamir & Coviello, Anti-Distillation: Improving reproducibility of deep networks,\n\n\n------------\n\n\nUpdate after rebuttal:\n\n\"It appears that the comment made by the reviewer may stem from an assumption that two models which are compared for PD can be different in the operations they perform to generate the predictions.\"\n\nThis is incorrect, my review does not mention such assumption and my statements hold without it. As stated in my review, I consider models with different weight magnitudes, making no assumptions on the underlying cause.\n\n\"PD, as we defined in Section 2, is aimed explicitly at measuring differences between predictions of a set of models that are supposed to be identical in all their components\"\n\nIndeed, and my point is that comparing the PD of two sets of models that are not identical is also problematic **even if all models within each set are identical**, except for the PD in its Hamming form. More details below.\n\n\"Changing calibration between such models violates this assumption.\"\n\nPlease check the celebrated work of Guo et al., \"On Calibration of Modern Neural Networks\": calibration does not necessarily consist of an explicit, additional component that modifies the model, and the same model trained in different ways can present distinct calibrations. More specifically, two sets of models can have not only the same accuracy, but the exact same predictions (i.e. there is a 1-1 mapping from each model in one set to a model in the other set that has the exact same predictions for all data points) but vastly different internal calibrations, which will result in vastly different PDs (to be overly specific, the scalar PD of a set will be different from the scalar PD of the other set) even though the two sets agree \"point-wise\" in terms of predictions.\n\n\"If one changes something about one of the models (including how calibration is done), one would expect them to predict differently, and have different accuracies.\"\n\nThis is incorrect. First, I'm not assuming models are explicitly calibrated, only that they have distinct internal calibrations (confidences in terms of predicted probabilities, which depend mostly on the parameters' magnitudes). Second, \"scaling the output layer weights by positive scalars\" (quoting from my review) will not change a model's accuracy: while it changes the class-wise predicted probabilities, the rank of the logits is preserved. If the authors remain skeptical of this fact, let $\\phi(x)$ denote the activations of the previous to last layer of a model, and let $\\langle w_i, \\phi(x) \\rangle > \\langle w_j, \\phi(x) \\rangle$, where $w_i$ and $w_j$ are the weight vectors of output units respective to classes $i$ and $j$ (i.e. $p(y_i | x) > p(y_j | x)$ for probabilities produced by a softmax over logits). Then for any $\\alpha \\in \\mathbb R_+$, we have trivially that $\\langle \\alpha w_i, \\phi(x) \\rangle > \\langle \\alpha w_j, \\phi(x) \\rangle$ (hence $p'(y_i | x) > p'(y_j | x)$, for probabilities $p'$ computed from the new logits). Again, note that it is --not-- necessary for an external, explicit calibration factor $\\alpha$ to be employed: training the network differently, or even adopting a different activation function -- just consider $\\max(0, 10x)$ for clarity, which will scale $\\phi(x)$ by a positive factor and yield the same observation as above.\n\n\"Specifically, if one flips the predictions of a binary classifier, the flipped model will have much worse accuracy from the actual model of interest, and measuring PD at this point is irrelevant.\"\n\nThe fact that two classifiers with vastly different accuracies can have zero PD is worrying and shows that PD is not a trustworthy metric: claiming that such evaluation is 'irrelevant' and should not be done does not address the issue.\n\n\nSince the authors remained unconvinced that the PD is sensible to positive scalings of a model's parameters, and hence comparing the PDs of two sets of models with different activations (one activation per set) is not sensible, here is a more detailed explanation of this fact.\n\nAssume a fairly trivial example for clarity: two 1-d data points, $x_1 = +1, x_2 = -1$, and binary classification models $f_1, f_2$, where $f_1(x) = \\sigma(w_1 \\cdot \\phi(x))$ and $f_2(x) = \\sigma(w_2 \\cdot \\phi(x))$ are the assigned probabilities for the positive label, and $\\phi: \\mathbb R \\to \\mathbb R$ captures some notion of activation function and/or scale of weights before the final classification layer. For simplicity, let $\\phi(x) = \\alpha x$, for some $\\alpha \\in \\mathbb R_+$, and feel free to think of $\\alpha$ as a 'magnitude' of an activation function instead of some notion of internal calibration.\n\nThen, we have $P_{1,1} = (\\sigma(\\alpha w_1), \\sigma(-\\alpha w_1))$, $P_{1,2} = (\\sigma(\\alpha w_2), \\sigma(-\\alpha w_2))$, $P_{2,1} = (\\sigma(-\\alpha w_1), \\sigma(\\alpha w_1))$, and $P_{2,2} = (\\sigma(-\\alpha w_2), \\sigma(\\alpha w_2))$. The PD of the set consisting of the two defined models, after simplifying the 8 relevant terms, ends up being simply $\\Delta_1 = |\\sigma(\\alpha w_1) - \\sigma(\\alpha w_2)|$. Let's pick some numbers to make this crystal clear: let $\\alpha = 1, w_1 = 1.0, w_2 = 0.1$, so we get $\\Delta_1 = \\sigma(1) - \\sigma(0.1) \\approx 0.2$ (note that w.l.o.g. we can assume that $y_1 = +1, y_2 = -1$ so that for these weights both models achieve 100% accuracy).\n\nNow, take ANOTHER set, consisting of models $g_1, g_2$, defined similarly to $f_1, f_2$, but with $g_1(x) = \\sigma(w'_1 \\cdot \\phi'(x)), g_2(x) = \\sigma(w'_2 \\cdot \\phi'(x))$, where $\\phi'$ (not the derivative of $\\phi$) captures the the activation function and/or weight magnitude of layers preceding the classification head. Let $\\phi'(x) = \\beta x$ for simplicity. Consider the case where $\\beta = 0.1, w_1 = 1.0, w_2 = 0.1$, i.e. the weights of $g_1, g_2$ are *exactly the same* as the weights of $f_1, f_2$, but $\\phi'$ is a 'scaled-down' $\\phi$ (e.g. a different activation function): in this case (note that both $g_1$ and $g_2$ achieve 100% accuracy as well), **for this new set of models, consisting of the pair $g_1, g_2$**, we get $\\Delta_1 = \\sigma(0.1) - \\sigma(0.01) \\approx 0.02$, a value around 10 times smaller than the PD of the first set of models, **even though the second set predicts the exact same labels for each data point**, and claiming that the set $\\{g_1, g_2\\}$ is 'more robust' than the set $\\{f_1, f_2\\}$ in terms of reproducibility is simply factually wrong. If the idea of having $\\beta \\neq \\alpha$ sounds a bit of a stretch since the proposed activations are not simply 'scaled down' ReLUs, consider instead the case $\\beta = 1.0, w_1 = 0.1, w_2 = 0.01$ and note that we again get $\\Delta_1 \\approx 0.02$ for this second set of models: the discrepancy in terms of magnitude of weights can be caused by different optimizers, different strength of $\\ell_2$ regularization, or, as my original review already mentioned, smaller variance of gradients w.r.t. activation function.\n\nTo reiterate, in the above example we did **not**, at any point, compute the PD of a set of models that had different components: both $\\{f_1, f_2\\}$ (the first set) had the same 'activation function' $\\phi$, while $\\{g_1, g_2\\}$ had $\\phi'$.\n\nGoing a step further, which shows how problematic the PD is as a metric, consider an arbitrary set of binary classifiers $S_1 = \\{f_1, f_2, \\dots, f_M\\}$, where $f_i(x) = \\sigma( \\langle w_i, \\phi(x) \\rangle)$ is the probability assigned by the $i$'th model of $x$ belonging to the positive class. Now, take *another* set of binary classifiers $S_2 = \\{g_1, g_2, \\dots, g_M\\}$, with $g_i(x) = \\sigma(\\langle w_i, \\phi'(x) \\rangle)$, where $w_i$ is the **same** weight vector that model $f_i$ has (i.e. except for $\\phi'$, the set $S_2$ is 'point-wise' identical to the set $S_1$). Finally, let $\\phi'(x) = \\beta \\phi(x)$, where $\\beta \\in \\mathbb R_+$, and feel free to check that for any $\\beta$, every model $g_i$ from $S_2$ will agree with the model $f_i$ from $S_1$ in terms of predicted class (i.e. although the class probabilities will change, the rank is be preserved for any $\\beta$). This means that $S_2$ produces the **exact same** predictions as $S_1$ for **any possible data point**. Taking $\\beta \\to 0$ yields in $g_i(x) \\to 0.5$ for any $i \\in [M]$ and possible $x$, hence **the PD of $S_2$ will go to zero, even though the PD of $S_1$ can be arbitrarily large and the two model sets $S_1, S_2$ agree point-wise in terms of predicted classes**. In other words, taking an arbitrary set of models with ReLU activations, copying its weights and replacing the ReLU by $\\phi(x) = \\max(0, \\frac{x}{10^{10}})$, will yield a second model set with PD close to zero. Hopefully the authors agree with me that this trivial replacement of activation functions does not 'solve' any reproducibility problem in machine learning.\n\nWith the above in mind, I urge the authors to re-evaluate PD as a metric. As mentioned in my review, the Hamming form does not suffer from this issue, but the reported numbers in this case seem to indicate that there is little to no reproducibility challenge for the adopted tasks.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4yjS4bj-xD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thanks for the efforts of the authors. Some of my concerns have been addressed. However, I still think that the experiments are not convincing enough for ICLR. So I keep my score.\n_____________________________________________\nSummary: \nThe paper argues that a smooth activation function may produce a smooth surface of the output of a network, which identifies a good reproducibility behavior. Based on this observation, the paper proposes the SmeLU activation function and its generalized version. Experiments with fully-connected neural networks are presented. \n\nStrengths: \n+ The idea that smooth activation functions may have better reproducibility of neural networks is very interesting. It provides a different way to understand the role of activation functions in neural networks.\n\nWeaknesses: \n- Figure 2 shows the motivation of the paper. It uses the surface of the output of a neural network w.r.t. its input to show the number of local minima. However, we usually use loss landscape to show the local minima of a neural network. Moreover, the paper needs to test with various layers (e.g., shallow and deep), initializations (e.g., Gaussian or Uniform), and architectures. It seems that Figure 2 only works with networks having two-dimension input and one-dimension output. For other networks, the visualization of the loss landscape of neural networks may be a good reference.\n- For SmeLU, the parameter \\beta is very important. It balances the accuracy and the reproducibility. However, the paper does not give a practical method to choose its value. The paper claims in Appendix B that \\beta can be learned with the weights of a neural network. However, the commonly used objective functions correspond to the accuracy. How to update \\beta to balance the accuracy and the reproducibility during training is unknown. \n-In Figure 4, different optimizers could produce different results. However, the paper only uses Adagrad. It is better to test with SGD, Adagrad, Adam, and AMSGrad.\n-The datasets used in the paper is very small. It is better to test with larger datasets.\n-The paper only tests with fully-connected neural networks. It is better to test with convolutional neural networks.\n\nOverall, since the paper is not a theoretical one, it needs extensive experiments to verify the claims. However, the experiments in the paper are not convincing enough.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviews were negative for this paper, due to various issues. I think the main issue was that the experimental results were too weak to be convincing. For example, the reviewers were not sure if the differences in performance between different activations are significant. The reviewers also required more datasets and more experiments. The authors added std to results, more experiments and argued that the current datasets are sufficient, but the reviewers seemed to remain unconvinced.
Decision: Reject