# Enhancing Logistic Regression Using Neural Networks for Classification in Actuarial Learning
## 2 Notation and Overview of Techniques
### Training a Logistic Regression Model  
Let \(\{(Y_{i},x_{i})\}_{i=1}^{n}\) be a set of examples where \(Y_{i}\) is the binary response and \(x_{i}\in\mathbb{R}^{d+1}\) are explanatory variables, or input examples, such that each example is represented by \(d\) real valued features and augmented with the constant \(1\) that represents the intercept component. Let \(\beta\in\mathbb{R}^{d+1}\) be the parameters that define the model and \(\mathcal{L}_{Y}(X;\beta)\) be a loss function that evaluates how well the model fits the response \(Y\) for input \(X\). We want to optimize the loss with respect to the parameters \(\beta\), and thus, in the following, for brevity, we will denote \(\mathcal{L}_{Y}(X;\beta)\) by \(\mathcal{L}_{Y}(\beta)\). We will present later appropriate choices for the loss function. Let us for now only assume that the first and second derivatives \(\frac{\partial}{\partial\beta}\mathcal{L}_{Y}(\beta)\) and \(\frac{\partial^{2}}{\partial\beta^{2}}\mathcal{L}_{Y}(\beta)\) with respect to the parameters \(\beta\) are well defined and nonzero.  
Unfortunately, the logistic regression model does not admit a closed form of the optimal solution (for a reasonable choice of the loss function) and we cannot directly compute an optimal solution. Instead, the solution has to be numerically estimated using an iterative process. Below we present two widely used approaches to finding an approximation of the optimal solution: namely the Newton-Raphson algorithm for the logistic regression model, which is outlined in Algorithm 1, and the gradient descent approach, which is presented in Algorithm 2.  
```
1. Input: A design matrix \(X\in\mathbb{R}^{n\times(d+1)}\), a response vector \(Y\in\{0,1\}^{n}\).
2. Set an initial value for \(\beta\in\mathbb{R}^{d+1}\) and name it \(\beta^{(0)}\), set \(\Delta^{(0)}=\infty\).
3. Define the maximum permitted number of steps, \(r_{\max}\) and threshold \(\varepsilon>0\). while\(\Delta^{(i)}\geq\varepsilon\)do
4. Update: \[\beta^{(r+1)} =\beta^{(r)}-\Big{(}\frac{\partial^{2}}{\partial\beta^{2}} \mathcal{L}_{Y}(\beta^{(r)})\Big{)}^{-1}\frac{\partial}{\partial\beta} \mathcal{L}_{Y}(\beta^{(r)}),\] (4a) \[\Delta^{(r+1)} =\|\beta^{(r+1)}-\beta^{(r)}\|.\] (4b) if\(\Delta^{(r+1)}\geq\varepsilon\)then
5 if\(r>r_{\max}\)then
6. Conclude that the algorithm does not converge. else
7. Update the index \(r=r+1\) and repeat the while loop
8. end if
9.else
10. Present \(\beta^{(r+1)}\) as the solution
11. end while
```  
**Algorithm 1** Newton-Raphson algorithm  
Both algorithms use the gradient descent approach that iteratively approximates a local optimum by a starting at a randomly selected point and moving in the opposite direction to the gradient. For a convex loss function, this is also the global optimum. The difference between the approaches is that, if we use a small predefined constant learning rate that moves in the direction of the (local) optimum, we use the second derivative of the loss function that guarantees the convergence with a smaller number of steps. The reduced number of iterations of the Newton-Raphson algorithm comes at the price that we need the second derivative of the loss function. We will later discuss why computing the second derivative can be a computational bottleneck for neural network optimization.