Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary: This paper analyzes the impact of the choice of ReLU'[0] on the fidelity of neural networks. Even though one would expect that the value of ReLU'[0] should not have any noticeable impact, the authors notice the divergence in learned parametric values that depend on this choice. They also notice that this effect is more pronounced at lower precision (due to rounding effects that are dominant at fp16 and fp32 vs. fp64). They also study this effect for neural network sizes, topologies (including networks with and without batch normalization) and datasets.
main_review: 
The author does a good job explaining the theory and experiments associated with the study of the impact of ReLU'[0]. The challenge in my view is the following:
[1] The studies seemed to indicate that the default values of ReLU'[0] = 0 seem to yield the best accuracy results! Therefore, its hard to see how the outcome of this work would be in general useful to neural network (and framework) designers. 
[2] The authors describe the proportion difference % between the different precisions. Can the authors comment on the final model accuracy difference for the different precisions for the different datasets. If there is no substantial difference in accuracy between the different precisions (as is indeed expected), then can the authors comment on why this result is important?
[3] In general, it'd be good to study this effect on larger models and datasets (e.g. ResNet50 or EfficientNetBx on ImageNet) - to see if this effect results in any noticeable difference in accuracy and to see if there are better optimal values for ReLU'[0]?
[4] Do the authors have results on other discontinuities besides ReLU'[0]? For e.g. one would expect ReLU[6] (used in MobileNets) to have a discontinuity at 0 and 6?

limitations_and_societal_impact: No comments here.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper demonstrates empirically that on a variety of architectures and datasets, the derivative of ReLU at 0 is evaluated a significant number of times during training in 16 and 32 bit precision. These events are frequent enough such that defining ReLU'(0) to be anything other than 0 (default value in common DNN frameworks) hurts generalization, and sometimes training as well.

The authors perform a variety of investigations around this effect, such as the influence of training set size, network width, depth, batch size, BatchNorm, Dropout.

main_review: ## Post-rebuttal Update

The authors have addressed my main concern on significance, namely that learning rate was not tuned for non-zero derivative settings. Conditioned on having the learning rate and number of training epochs tuned for each setting in section 4 (and other changes discussed in the rebuttal, e.g. the one on expected number of evaluations), I have raised the score from 5 to 7 and recommend accept, since I find it novel and potentially impactful that this setting has a strong influence not only on trainability, but also on generalization.

----
## Initial Review

Overall, I found this to be an interesting investigation with a surprising result. However, I am leaning to reject this paper due to (1) minimal qualitative / theoretical analysis or explanations of the observed phenomena (see section "Quality"), and (2) potentially flawed experimental setting to evaluate the impact of ReLU'(0) on training and test performance (see section "Significance"). I can see how the former could be improved at least a bit, and the latter can be clarified / fixed with additional experiments during rebuttal, so I am open to changing my score.

## Originality

The investigation and findings are to my knowledge original, although I am not an expert in autodiff literature.

## Quality

Mixed impressions on quality.

On one hand, I find the present experiments very thorough, well-designed, and broad, convincingly backing the claims of the paper.

On the other hand, the empirical results aren't analyzed that much, and the questions of

1. Which hyper-parameters influence the expected number of ReLU'(0) evaluations, and how?
2. Why does setting ReLU'(0) != 0 hurt performance?

remain unanswered. 

For question (1), the authors could, with the data that they already have, for instance consider a conjecture that "number of ReLU'(0) evaluations is proportional to the total number of all  ReLU'(.) evaluations during training", and make the respective plot. This would explain why the number of such events grows with many hyperparameters, but not batch size. 

For question (2) I don't have any immediate suggestions, apart from my comments on the learning rate in the "Significance" section.

## Clarity

Good, I found it easy to read the paper.



## Significance

Significance is at least average, since I find the fact that ReLU'(0) gets evaluated so often as to influence training / generalization quite surprising. This could make researchers more careful when reasoning about 0-probability events in floating point precision.

On the other hand, 

* These results may not be of practical utility, since most frameworks already use ReLU'(0) = 0, which is empirically the best setting.
* How did you tune your learning rate, when evaluating test / training performance of networks? If the learning rate was tuned once on ReLU'(0) = 0 architectures, then it is not surprising that altering the gradients while keeping the learning rate fixed leads to a gradual decrease in performance in the magnitude of the change. In this case, I would recommend the authors to tune the learning rate separately for each value of ReLU'(0), and compare the results then. If test / train performance still varies substantially (or if that's what the authors are already doing - please clarify), then I would consider this finding quite significant, and warranting follow-up research into why this happens.

limitations_and_societal_impact: The main finding of the paper (that ReLU'(0) influences test / train loss) is potentially of limited impact, if the authors did not tune the learning rate separately for each setting, as I describe in section "Significance" above.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The ReLU function is not differentiable at 0 which has a mathematical impact on ReLU neural networks. Nevertheless they are optimized using gradient based methods. The folk wisdom is that singular points only occur with zero probability and therefore the issue is irrelevant. This paper challenges this folk wisdom and identifies that the choice of the derivative at 0 does have a tangible impact on the optimization and classification performance.
main_review: I think this is a good paper and such a study has been overdue. 

The authors may considering the paper https://arxiv.org/pdf/1905.04992.pdf which performs a mathematical analysis of related issues.
limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies a long-neglected problem: the gradient of the non-smooth activation function in training a neural network. Specifically, the authors study the numerical influence of the ReLU derivative at zero on gradient descent. First, a theory is presented to illustrates the trivialness of ReLU'(0), then, the authors give counter-theory experiments to show that ReLU'(0) can affect training results.  In the last two sections, authors demonstrate that low-precision training, SGD optimizer, no batch norm training can magnify the impact of ReLU'(0).
main_review: Overall, this is a very interesting topic that draws little attention in the community before. The authors empirically prove that the numerical choice of the non-smooth activation function's gradient can affect the training behavior. I guess this is not surprising for half-precision training since 16 bit has a much smaller representation range than single-precision values.
The paper is well-organized and well-written with a clear structure. 


**Pros**:

+ The topic of this paper is quite interesting and can inspire more fellow research. 

+ The experiments evaluate relevant aspects, e.g. precision bit-with, optimizer, BN, etc.

+ Paper is neat and well-written.

**Cons**: 

- No large-scale experiments like ImageNet are conducted. 

- I'd like to see the impact of network architectures. For example, will ResNet outperform VGG-Net in terms of this stability of the non-smooth activation function? 

- Can the authors provide more examples of non-smooth activation and compare the results. 

limitations_and_societal_impact: My major concerns are the potential lack of more experiments, I hope authors can add more results to it. Then, I'll increase my score to accept.
review_title: null

Meta-review: The ReLU nonlinearity is popular in deep learning. It's derivative at 0, ReLU’(0), is undefined. This paper presents the surprising observation that the chosen value of the derivative of ReLU at 0 has a substantial effect of the performance achieved in neural network training at typical numerical precision. After the initial round of reviews some concerns surfaced about how robust this observation is to the choice of learning rate and other hyperparameters. The authors addresses these concerns in their response and the reviewers now seem convinced the observed effect is real and meaningful. Most of the reviewers engaged in discussion with the authors and with the rest of the committee: They reached a consensus recommendation to accept the paper.

Reviewer kt6J is the sole remaining negative reviewer: Unfortunately they did not engage in discussion with the committee or the authors. Their review did not point out any serious problems that would invalidate the contribution made by the paper. I do not believe the review of kt6J provides enough reason to not accept the paper, given that the paper presents a clearly surprising and interesting result that is appreciated by the other reviewers.

Authors, please integrate all new results and discussion from the author response phase into the camera ready version of the paper.
Decision: Accept (Poster)