Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: - Table 1's right most column seems to indicate that low deviation from the central monotonic segment correlates strongly with good performance, can you plot a similar measure in Fig 5, to see what the average deviation is and how it evolves during training?
- Can you hypothesize an explanation why the sin models in Fig 5 seem to be exclusively using the non-central part (since accuracy drops to 0 when replacing with clipped sin), and why it learns faster?
- Could the positive result on the 2 simple tasks be an artifact very specific to these tasks or is there some general advantage there?
- What other tasks do you suspect Sine might be good at?
review_title: Details

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
In this paper, the authors explore the advantages/disadvantages of using a sin activation function.
They first demonstrate that even with simple tasks, using sin activations can result in complex to optimize loss functions.
They then compare networks trained with different activations on the MNIST dataset, and discover that the periodicity of the sin activation is not necessary for learning the task well.
They then try different algorithmic tasks, where the periodicity of the functions is helpful.

Pros:
The closed form derivations of the loss surface were interesting to see, and the clarity of tone on the advantages *and* disadvantages was educational.

Cons: 
Seems like more of a preliminary investigation of the potential benefits of sin, and more evidence (to support or in contrary) is needed to conclude anything significant -- the results on MNIST seem to indicate truncated sin is just as good, and while it is interesting that tanh maybe uses more of the saturated part, the two seem relatively interchangeable. The toy algorithmic tasks are hard to conclude something concrete from.
review_title: Nice preliminary theoretical results of using sin activations, but more evidence needed

final_score: rating: 4: Ok but not good enough - rejection
review_content: Authors propose using periodic activation functions (sin) instead of tanh for gradient descent training of neural networks.
This change goes against common sense and there would need to be strong evidence to show that it's a good idea in practice. 
The experiments show slight improvement (98.0 -> 98.1) for some MNIST configurations. They show strong improvement (almost 100% higher accuracy after 1500 iterations) on a toy algorithmic task. It's not clear that this activation function is good for a broad class of algorithmic tasks or just for the two they present. Hence evidence shown is insufficient to be convincing that this is a good idea for practical tasks.
review_title: Mainly theoretical idea with insufficient evidence of being practical

final_score: rating: 4: Ok but not good enough - rejection
review_content: An interesting study of using Sine as activation function showing successful training of models using Sine. However the scope of tasks this is applied to is a bit too limited to be convincing. Maybe showing good results on more important tasks in addition to current toy tasks would make a stronger case?
review_title: Review

Meta-review: The reviewers unanimously recommend rejecting the paper.
Decision: Reject