What about adding Rectified Linear units? They should be much faster to compute than transcendental activation functions (and could be regularized with a piecewise linear regularizer as well: http://arxiv.org/abs/1301.3577 even if introduced for autencoders in this paper)