Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: This submission studies how the choice of activation function impacts the reproducibility of experiments involving deep networks. It proposes a new activation function with the goal of designing a smoothed ReLU, and provide experiments comparing it against other activations in terms of irreproducibility (measured via PD) and performance.

The problem of understanding how model design choices can have negative impacts on experimental reproducibility is interesting and timely, but I believe the paper does not provide a strong enough case for their approach and contributions.

First, the adopted metric to measure irreproducibility, 'Prediction Difference (PD)', is never evaluated in terms of how sensible of a metric it is to capture reproducibility -- this also seems to be lacking in [1]. Actually , one can argue that it is not a sensible metric at all (except for its Hamming form), as it is not invariant to how the models are calibrated, as discussed below.

For example, take any binary classifier and consider two copies of it with different calibrations (i.e. scaling the output layer weights by positive scalars, one for each model): even though the models always agree on their predicted labels regardless of their calibrations, the PD can be made arbitrarily close to 0.5 by calibrating the models appropriately. Even more worrying is that the same can be done by taking a binary classifier and a copy of it with flipped predictions: the PD between the two can be made arbitrarily close to 0 by scaling their weights down. Note that this problem also happens with the relative PD.

To see how this is connected to the choice of activation functions (especially ReLU x SmeLU), note that for normally-distributed inputs (centered around the origin), gradient's variance of ReLU is 1/4 while for SmeLU it is approximately sigma^2 / (4 beta^2) for large enough beta (sigma^2 being the variance of the input distribution): this discrepancy can have a non-trivial impact on the model's calibration and cause differences in PD to be artifacts.

Since this doesn't happen with the Hamming form of PD I believe Figure 15 in the Appendix to be the most informative one. However, it seems that different activations result in less than 1% prediction discrepancy across models, which is fairly insignificant and hence it is hard to argue that activations actually matter for reproducibility (at least from the presented experiments).

Lastly, it is hard to draw any conclusions from the presented experiments: the CTR results are based on a private dataset while the MNIST ones are extremely small-scale, with both the dataset and the model being arguably toy problems. There are numerous tasks where reproducibility is a prominent issue e.g. deep reinforcement learning, generative modelling (especially GANs), making training a 2-layer network on MNIST a poor choice to evaluate reproducibility problems.

As an additional note, the authors seem to rely heavily on the work of Shamir & Coviello '20 [1] which introduced the PD metric, even though the paper was only made publicly available on arXiv a week --after-- the reviewing period for this submission started. When citing papers which are yet to be made available it would be helpful to introduce and discuss the relevant content in a self-contained way -- while the authors avoided much of my confusion by presenting the full definition of the PD metric, the referred paper has useful information which was not discussed (such as which summand is normalized in its relative form and how the different variants compare).

Since I have major concerns with the paper -- particularly on the reliability of PD as a metric and the unconvincing empirical results -- I am voting for rejection.

[1] Shamir & Coviello, Anti-Distillation: Improving reproducibility of deep networks,


------------


Update after rebuttal:

"It appears that the comment made by the reviewer may stem from an assumption that two models which are compared for PD can be different in the operations they perform to generate the predictions."

This is incorrect, my review does not mention such assumption and my statements hold without it. As stated in my review, I consider models with different weight magnitudes, making no assumptions on the underlying cause.

"PD, as we defined in Section 2, is aimed explicitly at measuring differences between predictions of a set of models that are supposed to be identical in all their components"

Indeed, and my point is that comparing the PD of two sets of models that are not identical is also problematic **even if all models within each set are identical**, except for the PD in its Hamming form. More details below.

"Changing calibration between such models violates this assumption."

Please check the celebrated work of Guo et al., "On Calibration of Modern Neural Networks": calibration does not necessarily consist of an explicit, additional component that modifies the model, and the same model trained in different ways can present distinct calibrations. More specifically, two sets of models can have not only the same accuracy, but the exact same predictions (i.e. there is a 1-1 mapping from each model in one set to a model in the other set that has the exact same predictions for all data points) but vastly different internal calibrations, which will result in vastly different PDs (to be overly specific, the scalar PD of a set will be different from the scalar PD of the other set) even though the two sets agree "point-wise" in terms of predictions.

"If one changes something about one of the models (including how calibration is done), one would expect them to predict differently, and have different accuracies."

This is incorrect. First, I'm not assuming models are explicitly calibrated, only that they have distinct internal calibrations (confidences in terms of predicted probabilities, which depend mostly on the parameters' magnitudes). Second, "scaling the output layer weights by positive scalars" (quoting from my review) will not change a model's accuracy: while it changes the class-wise predicted probabilities, the rank of the logits is preserved. If the authors remain skeptical of this fact, let $\phi(x)$ denote the activations of the previous to last layer of a model, and let $\langle w_i, \phi(x) \rangle > \langle w_j, \phi(x) \rangle$, where $w_i$ and $w_j$ are the weight vectors of output units respective to classes $i$ and $j$ (i.e. $p(y_i | x) > p(y_j | x)$ for probabilities produced by a softmax over logits). Then for any $\alpha \in \mathbb R_+$, we have trivially that $\langle \alpha w_i, \phi(x) \rangle > \langle \alpha w_j, \phi(x) \rangle$ (hence $p'(y_i | x) > p'(y_j | x)$, for probabilities $p'$ computed from the new logits). Again, note that it is --not-- necessary for an external, explicit calibration factor $\alpha$ to be employed: training the network differently, or even adopting a different activation function -- just consider $\max(0, 10x)$ for clarity, which will scale $\phi(x)$ by a positive factor and yield the same observation as above.

"Specifically, if one flips the predictions of a binary classifier, the flipped model will have much worse accuracy from the actual model of interest, and measuring PD at this point is irrelevant."

The fact that two classifiers with vastly different accuracies can have zero PD is worrying and shows that PD is not a trustworthy metric: claiming that such evaluation is 'irrelevant' and should not be done does not address the issue.


Since the authors remained unconvinced that the PD is sensible to positive scalings of a model's parameters, and hence comparing the PDs of two sets of models with different activations (one activation per set) is not sensible, here is a more detailed explanation of this fact.

Assume a fairly trivial example for clarity: two 1-d data points, $x_1 = +1, x_2 = -1$, and binary classification models $f_1, f_2$, where $f_1(x) = \sigma(w_1 \cdot \phi(x))$ and $f_2(x) = \sigma(w_2 \cdot \phi(x))$ are the assigned probabilities for the positive label, and $\phi: \mathbb R \to \mathbb R$ captures some notion of activation function and/or scale of weights before the final classification layer. For simplicity, let $\phi(x) = \alpha x$, for some $\alpha \in \mathbb R_+$, and feel free to think of $\alpha$ as a 'magnitude' of an activation function instead of some notion of internal calibration.

Then, we have $P_{1,1} = (\sigma(\alpha w_1), \sigma(-\alpha w_1))$, $P_{1,2} = (\sigma(\alpha w_2), \sigma(-\alpha w_2))$, $P_{2,1} = (\sigma(-\alpha w_1), \sigma(\alpha w_1))$, and $P_{2,2} = (\sigma(-\alpha w_2), \sigma(\alpha w_2))$. The PD of the set consisting of the two defined models, after simplifying the 8 relevant terms, ends up being simply $\Delta_1 = |\sigma(\alpha w_1) - \sigma(\alpha w_2)|$. Let's pick some numbers to make this crystal clear: let $\alpha = 1, w_1 = 1.0, w_2 = 0.1$, so we get $\Delta_1 = \sigma(1) - \sigma(0.1) \approx 0.2$ (note that w.l.o.g. we can assume that $y_1 = +1, y_2 = -1$ so that for these weights both models achieve 100% accuracy).

Now, take ANOTHER set, consisting of models $g_1, g_2$, defined similarly to $f_1, f_2$, but with $g_1(x) = \sigma(w'_1 \cdot \phi'(x)), g_2(x) = \sigma(w'_2 \cdot \phi'(x))$, where $\phi'$ (not the derivative of $\phi$) captures the the activation function and/or weight magnitude of layers preceding the classification head. Let $\phi'(x) = \beta x$ for simplicity. Consider the case where $\beta = 0.1, w_1 = 1.0, w_2 = 0.1$, i.e. the weights of $g_1, g_2$ are *exactly the same* as the weights of $f_1, f_2$, but $\phi'$ is a 'scaled-down' $\phi$ (e.g. a different activation function): in this case (note that both $g_1$ and $g_2$ achieve 100% accuracy as well), **for this new set of models, consisting of the pair $g_1, g_2$**, we get $\Delta_1 = \sigma(0.1) - \sigma(0.01) \approx 0.02$, a value around 10 times smaller than the PD of the first set of models, **even though the second set predicts the exact same labels for each data point**, and claiming that the set $\{g_1, g_2\}$ is 'more robust' than the set $\{f_1, f_2\}$ in terms of reproducibility is simply factually wrong. If the idea of having $\beta \neq \alpha$ sounds a bit of a stretch since the proposed activations are not simply 'scaled down' ReLUs, consider instead the case $\beta = 1.0, w_1 = 0.1, w_2 = 0.01$ and note that we again get $\Delta_1 \approx 0.02$ for this second set of models: the discrepancy in terms of magnitude of weights can be caused by different optimizers, different strength of $\ell_2$ regularization, or, as my original review already mentioned, smaller variance of gradients w.r.t. activation function.

To reiterate, in the above example we did **not**, at any point, compute the PD of a set of models that had different components: both $\{f_1, f_2\}$ (the first set) had the same 'activation function' $\phi$, while $\{g_1, g_2\}$ had $\phi'$.

Going a step further, which shows how problematic the PD is as a metric, consider an arbitrary set of binary classifiers $S_1 = \{f_1, f_2, \dots, f_M\}$, where $f_i(x) = \sigma( \langle w_i, \phi(x) \rangle)$ is the probability assigned by the $i$'th model of $x$ belonging to the positive class. Now, take *another* set of binary classifiers $S_2 = \{g_1, g_2, \dots, g_M\}$, with $g_i(x) = \sigma(\langle w_i, \phi'(x) \rangle)$, where $w_i$ is the **same** weight vector that model $f_i$ has (i.e. except for $\phi'$, the set $S_2$ is 'point-wise' identical to the set $S_1$). Finally, let $\phi'(x) = \beta \phi(x)$, where $\beta \in \mathbb R_+$, and feel free to check that for any $\beta$, every model $g_i$ from $S_2$ will agree with the model $f_i$ from $S_1$ in terms of predicted class (i.e. although the class probabilities will change, the rank is be preserved for any $\beta$). This means that $S_2$ produces the **exact same** predictions as $S_1$ for **any possible data point**. Taking $\beta \to 0$ yields in $g_i(x) \to 0.5$ for any $i \in [M]$ and possible $x$, hence **the PD of $S_2$ will go to zero, even though the PD of $S_1$ can be arbitrarily large and the two model sets $S_1, S_2$ agree point-wise in terms of predicted classes**. In other words, taking an arbitrary set of models with ReLU activations, copying its weights and replacing the ReLU by $\phi(x) = \max(0, \frac{x}{10^{10}})$, will yield a second model set with PD close to zero. Hopefully the authors agree with me that this trivial replacement of activation functions does not 'solve' any reproducibility problem in machine learning.

With the above in mind, I urge the authors to re-evaluate PD as a metric. As mentioned in my review, the Hamming form does not suffer from this issue, but the reported numbers in this case seem to indicate that there is little to no reproducibility challenge for the adopted tasks.
review_title: Interesting problem, but metrics and experiments are not convincing

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper claims that smooth activations are more reproducible than ReLU. The accuracy gain claims seem marginal and not carefully carried out, further ablation studies are needed to strengthen the conclusion on accuracy. However, the main point of the paper is reproducibility where the feature is measured by the ‘Prediction Difference’. PD (introduced in section 2) is a measure over a set of models where the PD score is low if the models output consistent estimates for the same validation samples. 

Do models with the same initialization, and same randomness seed for the shuffles of SGD have high PD? If so then there would be numerical issues in the way models are trained. Otherwise, where does the difference in PD come from? Why such fluctuations are considered irreproducible? Why would PD be identified with reproducibility?

Overall the paper has a consistent story to tell. It’s a fresh perspective and focuses on the main problem at hand. However, I fail to understand some of the key measurements and their connection with reproducibility. It is undeniable that the shape of the landscape and the consistency of the models are linked. However, the variation found by ReLU’s could help increase ensemble accuracy. In other words, such variation can be a feature depending on the context. And, in principle, given the proper seeds and versions of the software, the results should be fully reproducible. Am I missing something? I am looking forward to the responses from the authors on this point. 
review_title: Stability of smooth activations as opposed to ReLU

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper addresses the problem that deep neural networks (DNNs) can lead to different predictions (even when they are initialized the same way) due to the stochasticity of samples selected in mini-batch SGD and update procedures from different optimizers, which leads to convergence to different regions along the loss surface.  They attribute this problem to the complicated loss surface that arises from the discontinuity in relu activations. They show that smooth activations can help remedy this issue, by tuning the activation to become more relu-like, which leads to a better tradeoff between prediction differences (i.e. consistency) and model accuracy.  

The pitch of the problem is motivated by fields like healthcare where more consistent predictions are important. However, it’s unclear why one would expect reproducible predictions when two network (with same initialization) are trained with a different sampling of mini-batch shuffles. This stochasticity should lead to different solutions for complex surfaces, just as a different initialization would. Smoothing the loss surface can help here and this paper only explores the extent that the smoothness of the activation plays a role. It’s unclear to me whether the activation function alone is sufficient for solving this problem. Other options, such as regularization, are not explored.

Comments:
* It is nice that traditional activations like swish and softplus are parameterized with a beta so that they can be modulated to become more relu-like. Given the stochasticity of mini-batching, it’s not surprising that you can start from the same initialization and navigate to a different region of the loss surface. Since the stochasiticity is due to the size of the mini-batch, this paper could benefit from exploring how the effect size of PDs changes with batch size. 
* Overall, the empirical evidence is very light — they compare different activation functions for a private dataset for ad Click-through-rate and MNIST. To make general claims, there should be various networks and various optimizers on various datasets. The paper can be more convincing if more datasets are explored, preferably not on a private dataset for which no one can validate their results. 
* Also, the optimizer plays a major role in navigating the loss surface, but only a single optimizer is compared in the first task, while only 2 are explored in task 2 (i.e. adagrad and sgd). Adding more optimizers, especially popular ones like adam, could help clarify whether the smooth activations is robust across optimizers or whether this is a special case for adagrad. I suspect the former, but experiments must be performed to prove any point. 
* On a smaller note, what does each dot in Fig 3 represent. Assuming it represents different beta values, there should be more consistency -- softplus has many points while tanhexp only has 4 points. Also, how do these betas alter the activation functions -- the  main/appendix show plots for 5 values of beta. Also, in Fig 4, why aren’t the same betas used for adagrad and sgd?  The choices of beta for a given base activation function should be consistent throughout. 
* Fig 4 should also show the standard deviation across the 12 models. This can provide a sense for the statistical significance of the results. 
* RESCU, a generalized activation function, is introduced but they never used in any comparisons. What is the benefit of such a formulation? This paper mentions that smelu is less expensive compared to the other smooth activations. Any direct statement like this should be followed up with quantitative comparisons — in this case the time per epoch. 
* This paper shows how weight normalization also influences the loss surface, but was not explored empirically. I think adding additional experiments taht scan different weight norms for a fixed beta would only strengthen their claims.
* It is unclear whether their initialization strategy for the DNNs explored here is fixed or whether they simply sampled different weights from the same initialization distribution. The wording of the motivation was that PD happens for the same initialization but the language in the text was ambiguous of whether they enforced the same random number seed for the initializer across each experiment. 
* As a control experiment, the effect size of prediction differences between training a model with different initializations should be compared with their main results that explore training with different mini-batch order starting from the same initialization. 

review_title: Improving reproducibility with smooth(er) activations that approach relu

final_score: rating: 4: Ok but not good enough - rejection
review_content: Thanks for the efforts of the authors. Some of my concerns have been addressed. However, I still think that the experiments are not convincing enough for ICLR. So I keep my score.
_____________________________________________
Summary: 
The paper argues that a smooth activation function may produce a smooth surface of the output of a network, which identifies a good reproducibility behavior. Based on this observation, the paper proposes the SmeLU activation function and its generalized version. Experiments with fully-connected neural networks are presented. 

Strengths: 
+ The idea that smooth activation functions may have better reproducibility of neural networks is very interesting. It provides a different way to understand the role of activation functions in neural networks.

Weaknesses: 
- Figure 2 shows the motivation of the paper. It uses the surface of the output of a neural network w.r.t. its input to show the number of local minima. However, we usually use loss landscape to show the local minima of a neural network. Moreover, the paper needs to test with various layers (e.g., shallow and deep), initializations (e.g., Gaussian or Uniform), and architectures. It seems that Figure 2 only works with networks having two-dimension input and one-dimension output. For other networks, the visualization of the loss landscape of neural networks may be a good reference.
- For SmeLU, the parameter \beta is very important. It balances the accuracy and the reproducibility. However, the paper does not give a practical method to choose its value. The paper claims in Appendix B that \beta can be learned with the weights of a neural network. However, the commonly used objective functions correspond to the accuracy. How to update \beta to balance the accuracy and the reproducibility during training is unknown. 
-In Figure 4, different optimizers could produce different results. However, the paper only uses Adagrad. It is better to test with SGD, Adagrad, Adam, and AMSGrad.
-The datasets used in the paper is very small. It is better to test with larger datasets.
-The paper only tests with fully-connected neural networks. It is better to test with convolutional neural networks.

Overall, since the paper is not a theoretical one, it needs extensive experiments to verify the claims. However, the experiments in the paper are not convincing enough.
review_title: The idea that smooth activation functions may have better reproducibility of neural networks is interesting. However, the experiments need more elaborations.

Meta-review: All reviews were negative for this paper, due to various issues. I think the main issue was that the experimental results were too weak to be convincing. For example, the reviewers were not sure if the differences in performance between different activations are significant. The reviewers also required more datasets and more experiments. The authors added std to results, more experiments and argued that the current datasets are sufficient, but the reviewers seemed to remain unconvinced.
Decision: Reject