Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper "Stochastic Adaptive Activation Function" successfully establishes a link between the statistics of feature selection and the recently proposed Swish-activation function.
Building upon the theoretical connection, an ASH (Adaptive Swish) non-linearity appears in the theory part.
The experimental part tests the new activation function Image net 2012, CIFAR-10, CIFAR-100, MS-COCO, and PASCAL VOC. The paper reports improved performance on every dataset and every network.

strengths_and_weaknesses: Strengths:
- The paper couples the Swish activation function to probability theory.
- The authors find improved performance for their ASH activation function across the board.
The numerical results are in line with those reported in https://arxiv.org/pdf/1710.05941;%20http://arxiv.org/abs/1710.05941.pdf for CIFAR-10.
- To the best of my knowledge, the contribution is novel. But I haven't been working looking into activation functions for some time (see confidence score below).

Weaknesses:
- The paper is hard to read at times.
- For a statistically motivated paper, it would have been neat to see variance values alongside the reported mean accuracy in the experimental section.


questions: - Line 17: The source code repositrory seems to be offline?
- Line 8: Does ASH mean something?
        - I found the meaning in Line 218. ASH stands for Adaptive SwisH. Perhaps this definition happens a little late in the paper?
- Line 72: What does Swish mean? Are SiLU and Swish synonymous? Could the exact definition be briefly given here? 
- Equation 1: Is the ASH-function a real-valued variant of the modRelu introduced in
    [arjovski16]? The authors may be interested in adding a citation and discussing the differences.
    [arjovski16] Unitary Evolution Recurrent Neural Networks https://arxiv.org/pdf/1511.06464.pdf

- Line 172: Do we have a citation for a Z-table, for convenience?
- Table 1: What is the variance here?
- Tables 2 and 3: Is the mean accuracy shown here? What about the variance?
   


limitations: Limitations are not discussed at all. All numbers are perfect. Is the experimental section too good to be true?

A discussion of limitations could include:
- Potential problems that were observed during training or hyperparameter tuning. Perhaps some negative properties have been observed at some point while working with the ASH activation?
- Reasons for not reporting the variance on the mean experiment values for the Image-Net experiments?
- Perhaps some theoretical drawbacks exist that readers should be aware of?
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors present the Adaptive SwisH (ASH) activation function, which is a more generalized form of the SwisH activation function. They present the parametric and adaptive properties of the ASH function and prove the baseline threshold of the ASH function is trainable during the training phase and that the threshold value adaptively changes according to the contexts of inputs. Additionally, they prove that the ASH function does not require heavy computations compared to some other activation functions. In terms of experiments, they show that the ASH activation function outperforms other activation functions on classification, object detection, and segmentation tasks.
strengths_and_weaknesses: Strengths
- proved parametric and adaptive properties of ASH function
- proved baseline threshold of the ASH function is trainable during the training phase and that the threshold value adaptively changes according to the contexts of inputs without heavy calculation
- proved ASH function is a generalized version of the SwisH activation function

Weaknesses
    Overall
    - research in the paper should be presented as a novel extension of the SwisH activation function since it is the more generalized form
    Abstract
    - should have placed the full name for the ASH activation function (Adaptive SwisH) in the "abstract" section
    Method
    - equation 10 needs more steps
    Main Result
    - Mention mathematical proofs for accuracy, sparsity, training time, and localization property as part of potential future work

questions: Why were only 3 models considered for each experiment (classification, object detection, segmentation)?
limitations: The authors addressed the fact that they did not search for the best performing parameters. Instead, they justified this by stating that allowing the network to learn these parameters on its own is more similar to how human neurons work. For future work, it would be nice to see the comparison between setting the threshold for zk and letting the network learn these parameters naturally.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a new trainable activation function that unlike most activation functions that work element-wise, this one activates over complete layers. 

More precisely, the activation function behaves like a ReLU, either activating with the input value, or cancelling to zero when deactivating. However, the activation criteria is whether the element is in the top k% of the elements in the layer.

The threshold is then controlled by a learnable parameter, making it a trainable activation function.
strengths_and_weaknesses: The paper is very interesting and I believe it could have applications and connections to other areas in deep learning.

The paper is well written and the empirical evaluation shows that the proposed activation works well.

The major weakness I see, is that the authors compare their activation mostly with other non-trainable activations. However, it is known that trainable activation functions such as [1], etc., usually perform better and such comparisons are missing in the empirical evaluation.

[1] Molina, Alejandro, Patrick Schramowski, and Kristian Kersting. "Padé Activation Units: End-to-end Learning of Flexible Activation Functions in Deep Networks." International Conference on Learning Representations. 2019.
questions: The empirical evaluation shows that this activation function is valuable and it works. 
I have however a series of questions that could be interesting to analyze to better understand the proposed method.

I'm thinking of the similarities of this activation with dropout. In particular, I'm wondering if the cancelation of the elements, should lead to a scale adjustment? It would seem like that could be beneficial as in dropout.

Now, considering the cancelation induced by the activation function, there are big parts of the network that won't receive gradient updates if they are not in the top k%. Would it make sense to introduce something like a negative tail or a scaling factor for the elements below the top k%? Think of Leaky ReLU.

Furthermore, as one would normally use these activation functions through the network. I'm wondering if there is some effect of vanishing gradients happening, or signal reduction, due to the cancellation imposed after each layer. I could imagine that if the learned parameter would cancel 90% of the signal at each layer, this could introduce a strong loss of information.

I'm also wondering whether this is a new approach for signal compression?. Consider a fixed top k% parameter. This directly implies that the number of outputs after the activation is a fraction of the size of the output. This is potentially equivalent to a reduction layer.
Here you do not get a dimensionality reduction, as for every input still different outputs could activate. But there is a compression in the amount of data reaching the next layer. If you were to control the percentage of information dropped, you could construct something like an auto-encoder and get the benefit of signal compression, without the effort needed in compressing the dimensionality. 
limitations: no negative societal impact
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes an rectifier based activation function where the threshold is set to the upper k% of the input tensor values.
strengths_and_weaknesses: - Originality: The main originality in this paper comes from determining a threshold based on a percentile from the input entries.  To my knowledge, this has not been considered before.  

- Quality: The main issue with the paper is that there are no statistical significance tests for the final results.  This prevents drawing any substantial conclusions from the results, which hurts the paper's quality.

- Clarity: The writing is at times difficult to follow and understand, but the mathematics appear sound.

- Significance: The paper would be much more significant with repeated runs and statistical significance tests, but as-is, it is unsignificant.
questions: - How is k determined?  Is it fixed, or learnable?  Is it the same across a network, or different for each layer?  What values for k are used in the experiments?

- The approximation of heaviside step function relies on a large value of alpha.  How does this affect the analysis in Equation 10?
limitations: - Some claims are excessive and unsubstantiated.  For example: "our activation function exhibits outstanding performance in any deep learning applications," or "the experiments empirically discovered that the Swish activation function is the best performance activation function."

- The experiments do not show any case in which ASH is outperformed by any other function.  The implication is that it is universally the best activation function.  This implied claim is problematic because there are no repeated runs (with the exception of 3 runs in Table 1) nor any statistical significance tests to back up these results.  See checklist item 3c.

- Figure 3 has no numbers on the x or y axes.
review_title: null

Meta-review: metareview: Reviewers appreciated the novelty of the proposed activation function, the theoretical motivation and its connection to the SwisH activation.
In terms of presentation and soundness of the results, Reviewers pointed out some weaknesses in the initial reviews for this paper. In particular, the reviews voiced some concerns with the clarity and formatting of some figures, the lack of clarity of a mathematical derivation, and most of all, issues in the presentation of the empirical results that didn't report confidence intervals allowing for an assessment of the statistical significance of accuracy differences. These weaknesses were however addressed in ways that satisfied the Reviewers in the rebuttals and subsequent versions of the paper.
Thanks to these welcome changes the paper has now garnered unanimous consensus among Reviewers that it should be accepted.
confidence: Certain
Decision: Accept