Title: Taming the waves: sine as activation function in deep neural networks
Authors: Giambattista Parascandolo
Heikki Huttunen
Tuomas Virtanen
Venue: ICLR 2017
Keywords: Theory
Deep learning
Optimization
Supervised Learning

Abstract: Most deep neural networks use non-periodic and monotonic—or at least
quasiconvex— activation functions. While sinusoidal activation functions have
been successfully used for specific applications, they remain largely ignored and
regarded as difficult to train. In this paper we formally characterize why these
networks can indeed often be difficult to train even in very simple scenarios, and
describe how the presence of infinitely many and shallow local minima emerges
from the architecture. We also provide an explanation to the good performance
achieved on a typical classification task, by showing that for several network architectures
the presence of the periodic cycles is largely ignored when the learning
is successful. Finally, we show that there are non-trivial tasks—such as learning
algorithms—where networks using sinusoidal activations can learn faster than
more established monotonic functions.

TL;DR: Why nets with sine as activation function are difficult to train in theory. Also, they often don't use the periodic part if not needed, but when it's beneficial they might learn faster

Review discussion:
- Table 1's right most column seems to indicate that low deviation from the central monotonic segment correlates strongly with good performance, can you plot a similar measure in Fig 5, to see what the average deviation is and how it evolves during training?
- Can you hypothesize an explanation why the sin models in Fig 5 seem to be exclusively using the non-central part (since accuracy drops to 0 when replacing with clipped sin), and why it learns faster?
- Could the positive result on the 2 simple tasks be an artifact very specific to these tasks or is there some general advantage there?
- What other tasks do you suspect Sine might be good at?
Thank you for your interesting questions!
1. After writing the code needed to compute what you suggest in question 1, we will re-run the experiments in figure 5 and report the results.
2. Our current hypothesis is that the model with sine might implement some sort of soft modulo operation. We might get a better intuition about this from the same code used to answer the point above. Note that since the accuracy for a given sequence is 1 only if the entire sequence is predicted correctly (and 0 otherwise), even always relying on a single neuron's non-central part might be necessary for good performance.
Also, it ḿight turn out that only a very small amount of neurons actually rely on the non-central part.
3. We are now running experiments on a few more algorithmic tasks with similar setting. We'll report the results once they are ready.
4. There are tasks that are challenging for quasiconvex functions and very simple for sine: computing parity (with a DNN) is probably the most well known example. More in general I suspect that it might help for tasks that would benefit from modulo operations with several divisors, but we definitely need more evidence to support this.
Update:
(1) The deviation grows for the first few iterations and then it remains stuck to the same value (an example from 8 Digit sum vanilla RNN, https://drive.google.com/file/d/0B0gSUcSu6RNEZDlWdGJBQWlKbUk/view?usp=sharing)
(3) Experiments on new tasks (such as sorting, repeating, reversing and max) gave mixed results (in some cases sin is better, in some tanh is better, in others they behave similarly). Finding more tasks where sin is beneficial is currently outside of the scope of the paper (see the general comment), even though we do agree that it would be good to have such results.
An interesting study of using Sine as activation function showing successful training of models using Sine.
However the scope of tasks this is applied to is a bit too limited to be convincing. Maybe showing good results on more important tasks in addition to current toy tasks would make a stronger case?
The other main points of the paper, that go beyond showing that there are tasks where sin is potentially beneficial, is evidence that sine as activation function makes for a difficult optimization problem and that when learning is successful the periodic part of the function is often largely ignored for typical datasets (as sine has already been successfully used in several works mentioned in Section 2). Please, see the Comments to all reviewers.
We do agree that having more evidence of sine working on larger tasks would make for a stronger case supporting its use.
However we didn't have this as a goal of the paper, and — instead of finding what tasks sin excels at — at this stage our work has been mostly focused on analyzing what the consequences are on the representation learned and in characterizing what makes these nets difficult to train. From the results presented in the paper (and the appendix, with the case of larger networks closing or reversing the gap between sin and tanh) it does seem that it might be difficult to find tasks and conditions where sin is more beneficial, so we would propose this for investigation in future work.
Authors propose using periodic activation functions (sin) instead of tanh for gradient descent training of neural networks.
This change goes against common sense and there would need to be strong evidence to show that it's a good idea in practice.
The experiments show slight improvement (98.0 -> 98.1) for some MNIST configurations. They show strong improvement (almost 100% higher accuracy after 1500 iterations) on a toy algorithmic task. It's not clear that this activation function is good for a broad class of algorithmic tasks or just for the two they present.
Hence evidence shown is insufficient to be convincing that this is a good idea for practical tasks.
There has probably been a misunderstanding. The scope of the paper is not to show that sine is overall better than tanh as an activation function.
The main point of the paper is to impartially analyze the effect of having sine as an activation function, how it affects the representation learned and what consequences it has on learning. Our findings actually indicate that sine is probably not a good choice for typical datasets, as the analysis conducted in Section 3 and the experiments on MNIST in Section 4 demonstrate. We also offer a simple and plausible explanation to its previous successful uses (presented Section 2), i.e. that when it works it’s acting very similarly to a tanh in its non-saturated region. The algorithmic experiments are only supposed to show that there might still be tasks where sine turns out to be helpful, see O3 for more details.
In order to make the main contributions of the paper more clear, we have more explicitly mentioned these in the introduction to the paper.
Summary:
In this paper, the authors explore the advantages/disadvantages of using a sin activation function.
They first demonstrate that even with simple tasks, using sin activations can result in complex to optimize loss functions.
They then compare networks trained with different activations on the MNIST dataset, and discover that the periodicity of the sin activation is not necessary for learning the task well.
They then try different algorithmic tasks, where the periodicity of the functions is helpful.
Pros:
The closed form derivations of the loss surface were interesting to see, and the clarity of tone on the advantages *and* disadvantages was educational.
Cons:
Seems like more of a preliminary investigation of the potential benefits of sin, and more evidence (to support or in contrary) is needed to conclude anything significant -- the results on MNIST seem to indicate truncated sin is just as good, and while it is interesting that tanh maybe uses more of the saturated part, the two seem relatively interchangeable. The toy algorithmic tasks are hard to conclude something concrete from.
Review: Summary:
In this paper, the authors explore the advantages/disadvantages of using a sin activation function.
They first demonstrate that even with simple tasks, using sin activations can result in complex to optimize loss functions.
They then compare networks trained with different activations on the MNIST dataset, and discover that the periodicity of the sin activation is not necessary for learning the task well.
They then try different algorithmic tasks, where the periodicity of the functions is helpful.
This is an accurate summary. More specifically concerning point 2: from our experiments it emerges that when the task is learned well the periodicity is not used, and when the task is not learned (i.e. the network just overfits to the training set) the periodicity is used heavily.
Pros:
The closed form derivations of the loss surface were interesting to see, and the clarity of tone on the advantages *and* disadvantages was educational.
Cons:
Seems like more of a preliminary investigation of the potential benefits of sin, and more evidence (to support or in contrary) is needed to conclude anything significant
We agree that more evidence would make for a stronger case. We are now considering a few options in this direction, in order to support the main contribution of the paper. O2 would benefit from more experiments similar to the MNIST experiment to validate/falsify it. We are planning to choose a couple more datasets to test the claim on, for now we added the Reuters dataset. What else would you recommend to investigate in order to further improve the impact of the paper?
-- the results on MNIST seem to indicate truncated sin is just as good, and while it is interesting that tanh maybe uses more of the saturated part, the two seem relatively interchangeable.
This is correct.
However, the main contribution of the results on MNIST is evidence showing that when training is successful for typical datasets (as it's the case for most previous works using sine presented in Section 2) the network is not relying on the periodicity of the function. The network relies almost exclusively on the central part of the function, that is monotonic and similar to a tanh.
-- The toy algorithmic tasks are hard to conclude something concrete from.
Our conclusion from the toy tasks is the following:
There might still be room for sine as activation function in certain artificial/algorithmic tasks where sine is intuitively beneficial, since there are at least two simple tasks (i.e. sum and diff) where sin can outperform a standard RNN baseline using tanh.
Thank you to all the reviewers for the helpful comments. Probably due to a lack of clarity of the paper, some details in the proposed summaries differ from the main points we tried to convey. Let us then attempt to clarify what the point of the paper is supposed to be, both here and in the manuscript.
The main point of the paper is to analyze the effect of sine as an activation function, how it affects the representation learned and what consequences it has on learning.
The main contributions of the paper are:
O1 - a proof that confirms the claim that networks with sine might easily get stuck into local minima, obtained by analytically characterizing the loss surface.
O2 - evidence showing that when training is successful for typical datasets (as it's the case for most previous works using sine presented in Section 2) the network is actually not relying on the periodicity of the function. The network relies almost exclusively on the central part of the function, that is monotonic and similar to a tanh. Evidence for this is presented on MNIST and in the new version on Reuters too.
O3 - there might still be room for sine as activation function in certain artificial/algorithmic tasks where sine is intuitively beneficial, since there are at least two simple tasks (i.e. sum and diff) where sin can outperform a standard RNN baseline using tanh.
We are trying to make these points more clear and explicit in the paper.

Meta-review:
role: PC
data: comment: The reviewers unanimously recommend rejecting the paper.

Decision: Reject