BVLC/caffe — pull request #1501 by jyegerlehner (2014-11-29 20:16:57+00:00)
Title: Thresholded ReLU

Extend the ReLU unit to allow thresholded ReLU (AKA TRec) as presented in [Memisevic et al](http://arxiv.org/pdf/1402.3337v2.pdf).

Adds an optional parameter "theta" parameter to the ReLUParameter. Specify its value in order to get the thresholded ReLU ("TRec") behavior. I think you can also get the "TLin" behavior by also specifying a negative_slope = 1.0. The goal is that by using this along with bias_term=false on the INNER_PRODUCT or CONVOLUTION layer, and no weight decay, you'd get the zero-bias autoencoder described in the paper. I'm guessing you'd want to have at least mean-subtracted image inputs, if not PCA and whitened inputs as in the paper, if you run with no biases.

There's not any higher math involved in the implementation, but my understanding of it is a bit tenuous so I'm hoping someone can look at this and the paper and see if it's implemented correctly.