# Enhancing Logistic Regression Using Neural Networks for Classification in Actuarial Learning
## 2 Notation and Overview of Techniques
### Neural Networks
#### 2.7.4 Activation Functions  
When the activation function is nonlinear, a deep neural network computes complex nonlinear functions. In fact, the universal approximation theorem states that any continuous function can be approximated with an arbitrarily small error by a neural network with a single hidden layer and a nonlinear activation. However, such functions are unlikely to generalize well as the hidden layer might need to be very large, and thus they will not be useful for learning a model that generalizes well to new data. Instead, one prefers multilayer networks where each layer computes certain patterns which are then combined by the next layer in order to form more complex patterns. In the context of computer vision, one can think of the first layers of a neural network of detectors of basic image features such as edges and contours, while the latter layers combine these basic shapes into more complex patterns that represent object parts.  
Below, we list the most widely used activation functions.  
* Sigmoid. \[\sigma(z)=\frac{1}{1+e^{-z}}\]The sigmoid function is widely used for binary classification as it returns a value in \((0,1)\) which can be treated as the probability that a given example belongs to one of two classes. For multi-classification problems, the function is generalized to the softmax function which returns the probability for each of \(k\) classes. However, the derivative of the sigmoid function is \(\sigma(x)(1-\sigma(x))\). Therefore, the maximum possible value of the derivative is 0.25. It is not advisable to use the sigmoid as the activation function for several layers as it is likely to encounter the problem of _vanishing gradients_: by the chain rule, we multiply the derivatives at each layer and thus the gradient will converge to 0 and updating the weights by gradient descent will converge very slowly towards an optimal solution. For this reason, the sigmoid is usually used as an activation function only for the output layer.
* Hyperbolic functions. \[tanh(z)=\frac{2}{1-e^{-2z}}-1\] This is an alternative to the sigmoid function where the range of values is between \(-1\) and \(1\) which might yield more insights, especially regarding the impact that certain features have on the output. Observe that \(tanh(z)=2\sigma(2z)-1\), and thus the derivative is at most 1 and is close to 1 only for \(z\approx 0\). Thus, stacking several layers with a hyperbolic activation function might again lead to the vanishing gradient problem, and thus it is recommended to use it only for layers close to the output layer.
* ReLU. \[\phi(z)=\max(z,0)\] This is a nonlinear function that can be used to obtain a piecewise linear approximation of arbitrary functions. As long as the weights are positive, gradient descent quickly converges towards the optimal solution. However, an obvious problem of the ReLu activation function is that if the input to a neuron is negative, then the derivative will always be 0, which is the so-called \"dying\" ReLUs problem. Alternatives have been proposed, such as the LeakyReLU with \(\phi(z)=\max(z,\gamma z)\), where \(\gamma\in(0,1)\) is a small constant, or the eLU \(\phi(z)=\max(z,\alpha e^{z}-1)\) for some \(\alpha\in(0,1]\).  
In our setting, we found that the eLU activation function yields the best results, slightly outperforming the ReLU. We will comment more on this in the next section.