mnielsen/nnlsv — pull request #3 by colah (2013-11-22 20:00:21+00:00)
Title: Provide citations for ReLU and softplus.

Comments on ReLU and softplus:

The major candidates for citation seem to be:

ReLU:
Jarret (2009) -- Use "positive part" and "abs", but seem to use it as an additional layer rather than an activation function.
Glorot (2010) -- Use "positive part" activation function, observe that it allows one to train deeper networks withtout pretraining.
Nair & Hinton (2010) -- Introduce the term ReLU, use them for RBMs.
Krizhesky (2012) -- Use them as a feedbforward activation, observe that they train much faster
Dahl (2013) -- Activation function, train much faster, counteracts cost of dropout.

(Seems like UofT group focuses on ReLUs as a speed increase, Bengio's group has "positive part"/ReLU as a way to get deeper networks.)

softplus:
Dugas -- use softplus kind of as an activation function in a strange network architecture where they multiply activations together, inspired by Black-Scholes.
Glorot (2010) -- explores softplus in contrast to ReLUs

It seems most helpful to the reader to cite people who are using them as activation functions in networks like ours... Though I'm not completely sure if that is what we should be focusing on? I'm also citing Nair & Hinton because they introduce the term ReLU.