Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper is utilizing reinforcement learning to search new activation function. The search space is combination of a set of unary and binary functions. The search result is a new activation function named Swish function. The authors also run a number of ImageNet experiments, and one NTM experiment.

Comments:

1. The search function set and method is not novel. 
2. There is no theoretical depth in the searched activation about why it is better.
3. For leaky ReLU, use larger alpha will lead better result, eg, alpha = 0.3 or 0.5. I suggest to add experiment to leak ReLU with larger alpha. This result has been shown in previous work.

Overall, I think this paper is not meeting ICLR novelty standard. I recommend to submit this paper to ICLR workshop track. 


review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: Authors propose a reinforcement learning based approach for finding a non-linearity by searching through combinations from a set of unary and binary operators. The best one found is termed Swish unit; x * sigmoid(b*x). 

The properties of Swish like allowing information flow on the negative side and linear nature on the positive have been proven to be important for better optimization in the past by other functions like LReLU, PLReLU etc. As pointed out by the authors themselves for b=1 Swish is equivalent to SiL proposed in Elfwing et. al. (2017).

In terms of experimental validation, in most cases the increase is performance when using Swish as compared to other models are very small fractions. Again, the authors do state that "our results may not be directly comparable to the results in the corresponding works due to differences in our training steps."   

Based on the Figure 6 authors claim that the non-monotonic bump of Swish on the negative side is very important aspect. More explanation is required on why is it important and how does it help optimization. Distribution of learned b in Swish for different layers of a network can interesting to observe.
review_title: Another approach for arriving at proven concepts on activation functions

final_score: rating: 7: Good paper, accept
review_content: The author uses reinforcement learning to find new potential activation functions from a rich set of possible candidates. The search is performed by maximizing the validation performance on CIFAR-10 for a given network architecture. One candidate stood out and is thoroughly analyze in the reste of the paper. The analysis is conducted across images datasets and one translation dataset on different architectures and numerous baselines, including recent ones such as SELU. The improvement is marginal compared to some baselines but systematic. Signed test shows that the improvement is statistically significant.

Overall the paper is well written and the lack of theoretical grounding is compensated by a reliable and thorough benchmark. While a new activation function is not exiting, improving basic building blocks is still important for the community. 

Since the paper is fairly experimental, providing code for reproducibility would be appreciated.
review_title: Well written paper and well conducted experiments.

Meta-review: The author's propose to use swish and show that it performs significantly better than Relus on sota vision models. Reviewers and anonymous ones counter that PRelus should be doing quite well too. Unfortunately, the paper falls in the category where it is hard to prove the utility of the method through one paper alone, and broader consensus relies on reproduction by the community. As a results, I'm going to recommend publishing to a workshop for now.
Decision: Invite to Workshop Track