Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a learnable piece-wise linear activation unit whose hinges are placed symmetrically. It gives a proof on the universality of the proposed unit on a certain condition. The superiority of the method is empirically shown. The change of the activation during training is analyzed and insight on the behavior is provided. The robustness to adversarial attacks is also empirically examined.

This paper discusses a very basic component of neural network models: activation function. Thus, it should be of interest to many researchers. The proposed method is simple and seems easy to use in real settings. A number of experiments are conducted to validate the method and the results look promising. The experiments in Section 5 is particularly interesting. It might give some hints for the following studies.

However, there are several things to be addressed for acceptance.

1) What is actually proposed is not very clear. 

S-APL is formulated in Equation 2. However, there are some discussion after that which changes or restricts the equation. For example, it seems that b_i^s^+ = b_i^s^- is assumed throughout the paper. In that case, it should be just reflected in Equation 2. In the third paragraph of Section 3.2, it is mentioned that h_i^s(x) = h_i^s(-x) with b_i^s^+ = b_i^s^-. However, it should also assume that a^s^+ = a^s^-. From the experiments. apparently, a^s^+ = a^s^- is not assumed. It seems that the method has symmetry only for the hinge locations. 

In the first paragraph of Section 3.2, it is implied that parameters are shared across layers. It is not very clear what is shared and what is not. Please make that part clear. It will make it easier to understand the experimental settings, too.

2) Theorem 3.1 does not seem to prove the approximation ability of S-APL.

It is clear that g(x, S) can represent arbitrary h(x, S), but I am not sure if it is clear that h(x, S) can represent arbitrary g(x, S). It should also depend on the conditions on a^s^+, a^s^-, b_i^s^+, b_i^s^-. I think it needs to prove that h(x, S) can approximate arbitrary piecewise linear function (i.e., g(x, S)) if you want to prove the approximation ability of h(x, S).

Equation 4 seems to assume that all intervals are the same (i.e., ∀i, B_i - A_i = (B-A) / S). It should be stated explicitly. This relates to the problem 1).

I may not understand some important aspect. I am happy to be corrected.

3) Experimental conditions are not clear.

Please cite the papers which describe the architecture of the models used in the experiments. The effectiveness of the proposed method should depend on the network architecture and it is importable to be able to see the details of the models.

4) On the sensitivity of optimization on the initial value.

It is interesting to see that "fixed trained S-APL" is not comparable with "S-APL positive". If the hypothesis in the paper is correct, it is natural to assume that "fixed trained S-APL" also has some issue on training. It would be interesting to see experimental results with "initialized with trained-S-APL" and "S-APL positive with non-zero initial value".  It is a bit weird to observe that "S-APL positive" never becomes non-zero for x < 0.

5) Comparison results with other activation units in Section 6.

The proposed method is compared only with ReLU. It is important to see comparisons with other activations such as the plain APL.


Some other minor comments:

It is quite interesting that objects are actually modified for adversarial attack for the proposed method in Figure 5. It would be interesting to have some consideration on it.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: In this paper, a new activation function, i.e. S-APL is proposed for deep neural networks. It is an extension of the APL activation, but is symmetric w.r.t. x-axis. It also has more linear pieces (actually S pieces, where S can be arbitrarily large) than the existing activation functions like ReLU. Experimental results show that S-APL can be on par with or slightly better than the existing activation functions on MNIST/CIFAR-10/CIFAR-100 datasets with various networks. The authors also show that neural networks with the proposed activation can be more robust to adversarial attacks.

First of all, the activation function is much more complicated than the existing ones, as it has to determine the parameter S and the hinge positions. However, the gain is marginal as shown in Table 1. Besides, the authors never tell how to choose S and the hinge positions.

Secondly, the neural networks used in the experiments are quite outdated. And the error rates shown in Table 1 are far away from state-of-the-art. Why don't you choose a latest network such as ResNet/DenseNet/EfficientNet and replace the activation with S-APL? The results could be more convincing.

I am not an expert in adversarial attack. But is there any intuition why a complicated activation function is more robust to adversarial attack? Again, most of the models used in Table 2 are quite old (Lenet5, Net in Net, CNN).

In a word, the proposed activation function is unnecessarily complicated and the gain is not justified with the latest models and not significant enough to convince people to adopt it.

review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper proposes the S-APL (Symmetric Adaptive Piecewise Linear) activation function, based on the APL activation function proposed by (Agostinelli et al, 2014). This activation function is constructed as a piecewise linear function that is learned concurrently with training, and, in the case of S-APL, the activation function is forced to be symmetric. S-APL is claimed to help both with trainability and robustness of neural networks to adversarial examples.

Overall, the idea is clearly presented, but appears to have many critical flaws, enumerated below:
1. It is unclear what the motivation for the symmetry is upon first reading of the paper, i.e., Section 3.1 starts by saying "To overcome the shortcomings of APL," but up to this point I cannot find any exposition that explains what these shortcomings are---the beginning of section 3 just presents the formulation of APL and does not discuss its advantages/disadvantages.

2. The experimental results concerning network training are very hard to interpret, as they lack error bars, confidence intervals, and many critical experimental details (e.g. how many networks were trained, the hyper parameters for training, etc.) For these results to be interpretable, the authors should include a detailed description of the environment under which the experiments were performed, and present confidence intervals which demonstrate the significance of the improvement attained by S-APL. (As it stands, it is hard to tell whether, e.g., a 0.1% improvement on CIFAR-10 should be considered significant.)

3. The adversarial evaluation section should be substantially revised to address several important flaws:
(a) The evaluation for black-box attacks is done in very non-standard threat models (e.g. label-only 3-pixel black-box attacks) that do not seem to be relevant to the black-box robustness of a system. Even under these threat models, the authors should also use more powerful label-based black-box attacks, such as the BoundaryAttack [1] or the label-only attacks in [2] or [3].
(b) The avg(|Z_{true} - Z_{adv}|) has an absolute value around Z_{true} - Z_{adv}, which means that failed adversarial attacks actually contribute positively to the score, which severely limits its usefulness for judging adversarial robustness. 
(c) The open-box (white-box) setting only uses FGSM to evaluate the robustness, which is known to be a weak attack [4] and is recommended against for evaluating adversarial robustness [5]. The methods should be evaluated with PGD or CW-attacks to better judge robustness.
(d) Overall, the results presented are not sufficient to evaluate the adversarial robustness of the S-APL activation. A first step towards remedying this would be to follow the evaluation protocol suggestions outlined in [5].

The idea does seem promising, however, and the paper could be substantially improved by including the necessary introduction, evaluation protocols, and experimental details. However, this would require a substantial change to the paper, and thus my recommendation for now is to reject.

[1] https://arxiv.org/abs/1712.04248
[2] https://arxiv.org/abs/1804.08598
[3] https://arxiv.org/abs/1807.04457
[4] https://arxiv.org/abs/1802.00420
[5] https://arxiv.org/abs/1902.06705
review_title: Official Blind Review #4

Meta-review: This work presents a learnable activation function based on adaptive piecewise linear (APL) units. Specifically, it extends APL to the symmetric form. The authors argue that S-APL activations can lead networks that are more robust to adversarial attacks. They present an empirical evaluation to prove the latter claim. However, the significance of these empirical results were not clear due to non-standard threat models used in black-box setting and the weak attacks used in open-box setting. The authors revised the submission and addressed some of the concerns the reviewers had. This effort was greatly appreciated by the reviewers. However, the issues related to the significance of robustness results remained unclear even after the revision. In particular, as pointed by R4, some of the revisions seem to be incomplete (Table 4). Also, the concern R4 had initially raised about non-standard black-box attacks was not addressed. Finally, some experimental details are still missing. While the revision indeed a great step, the adversarial experiments more clear and use more standard setup be convincing.

Decision: Reject