# Enhancing Logistic Regression Using Neural Networks for Classification in Actuarial Learning
## 2 Notation and Overview of Techniques
### Neural Networks
#### 2.7.2 Neural Network Training  
Given a dataset with examples \(x_{i}\in\mathbb{R}^{n}\) and a target variable \(y_{i}\in\mathbb{K}^{m}\) (where \(\mathbb{K}=\mathbb{R}\) for regression problems, and \(\mathbb{K}=\mathbb{L}\subset\mathbb{N}\) for classification), we want to learn the parameters or _the weights_ of the layers of a neural network that computes a function \(f(x_{i})=y_{i}\). This is usually achieved by choosing a particular neural network architecture and a loss function that helps us achieve a given objective, e.g., high accuracy for classification or a small absolute error for regression. In our setting, the desired objective is to reflect the probability for an insurance claim. The loss function should be well defined and differentiable such that an optimal solution can be found. As neural networks represent complex functions, an optimal solution will not exhibit a closed form in the general case. Instead, a local optimum is found using numerical optimization methods. It is standard to use the gradient descent method. The approach starts with randomly assigned values for the weights for all neurons. The values are random but sampled from some distribution. Empirically, it was observed that  
Figure 1: A multilayer perceptron neural network with three hidden layers. The last neuron in each layer is the intercept term. sampling from a 0-mean normal distribution where the variance is a function of the number of input and output neurons at the layer guarantees the fast convergence of training. The examples are processed in batches. For example, let \(f(x)\in\mathbb{R}\) be the output of a neural network for a given regression problem. For a batch with \(n\) examples \(x_{i}\in\mathbb{R}^{d}\), the loss is defined as \(\frac{1}{n}\mathcal{L}(f(x_{i}),y_{i})\) where \(y_{i}\) is the response we want to learn. For regression problems, a commonly used loss function is the mean squared error, thus we have to optimize the parameters of \(F\), i.e., the weights of the layers in the neural network, such that \(\frac{1}{n}\sum_{i=1}^{n}(f(x_{i})-y_{i})^{2}\) is minimized. In the binary classification setting, appropriate choices for the loss function are cross-entropy and deviance, as discussed in the previous section. The method of choice for neural networks is _gradient descent_, as described in Algorithm 2. Weights are updated such that the weight values \(\beta\) move in the negative direction of the gradient of \(\mathcal{L}(f(x),y)\):  
\[\beta_{t+1}=\beta_{t}-\alpha\
abla\mathcal{L}(F(x;\beta_{t}),y)\]  
where \(f(x;\beta)\) denotes neural network evaluated at \(x\) with weights \(\beta\) and \(\alpha>0\) is the _learning rate_. For sufficiently small \(\gamma\), gradient descent is guaranteed to converge to a local optimum. Thus, for a convex function, it will probably find an optimal solution. \[\beta_{t+1}=\beta_{t}-\gamma\
abla\mathcal{L}(f(x),y)\]  
As discussed, a neural network implements a composition of functions. Thus, the loss function can be written as  
\[\mathcal{L}(f(x),y)=\mathcal{L}(f^{(k)}(f^{(k-1)}(\dots(f^{1}((x;\beta_{t})))),y)\]  
Using the chain rule for computing the derivative of a composite function, gradient descent is applied to a neural network using back propagation. Remember that \(z^{(k)}=f(z^{(k-1)})=\phi^{(k)}(W^{(k)}z^{(k-1)})\) and \(W^{(k)}=\left(\beta_{1}^{(k)},\cdots,\beta_{q_{k}}^{(k)}\right)^{\prime}\). In order to compute the derivative of the loss function with respect to the weight of the neurons in the \(t\)-th layer, for \(1\leq t\leq k\), we apply the chain rule as  
\[\frac{\partial\mathcal{L}(f(x),y)}{\partial W^{(t)}}=\frac{\partial\mathcal{L }(f^{(k)},y)}{\partial z^{(k-1)}}\frac{\partial f^{(k-1)}}{\partial z^{(k-2)} }\dots\frac{\partial f^{(t+1)}}{\partial W^{(t)}}\]  
In each layer, we compute the derivative at each layer with respect to the output of the previous layer.