# Enhancing Logistic Regression Using Neural Networks for Classification in Actuarial Learning
## 2 Notation and Overview of Techniques
### Maximum Likelihood Estimation  
Assume that we have a set of data points \(\{(Y_{i},x_{i})\}_{i=1}^{n}\) and we want to calculate the ML estimate of the \((d+1)\)-dimensional parameter \(\beta\) of the logistic regression model in Equations (1) and (2).  
The log-likelihood function of the model is given by  
\[\mathcal{L}_{Y}(\beta)=\sum_{i=1}^{n}[y_{i}\log(\pi_{i})+(1-y_{i})\log(1-\pi_{ i})], \tag{6}\]  
for \(i=1,\ldots,n\).  
Thus, by substituting \(\pi_{i}\) from Equation (3) into Equation (6), we have  
\[\mathcal{L}_{Y}(\beta)=\sum_{i=1}^{n}\Bigl{[}y_{i}\log(\sigma(x_{i}^{T}\beta)) +(1-y_{i})\log(1-\sigma(x_{i}^{T}\beta))\Bigr{]}, \tag{7}\]  
where \(\sigma:X\to(0,1)\) is the sigmoid function.  
To find MLE of \(\beta\), one has to solve the score equation  
\[0\overset{!}{=}\frac{\partial}{\partial\beta}\mathcal{L}_{Y}(\beta)=\sum_{i= 1}^{n}\Bigl{[}y_{i}-\sigma(x_{i}^{T}\beta)\Bigr{]}x_{i}. \tag{8}\]  
The negative likelihood thus defines the _cross-entropy loss_:  
\[\mathcal{L}_{Y}(\beta)=\frac{1}{n}\sum_{i=1}^{n}-y_{i}\log(\sigma(x_{i}^{T} \beta))-(1-y_{i})\log(1-\sigma(x_{i}^{T}\beta)), \tag{9}\]  
However, since there is no analytic solution for Equation (8), the solution has to be numerically estimated using an iterative process as the one described in the previous section. Next, we show how to employ the Newton-Raphson method which is the most popular method for solving systems of nonlinear equations (note that the MLE of the logistic regression model can be obtained by calling the glm() function in programming language R whilst setting the family argument to \"binomial\").  
The Newton-Raphson algorithm for the logistic regression model can be described in matrix form as follows:  
* Let \(\beta^{(r)}\) be the estimate of \(\beta\) obtained after the \(r\)-th iteration. The matrix representations of the first and the second derivative of the log-likelihood in (6)-(9) are given by \[\frac{\partial}{\partial\beta}\mathcal{L}_{Y}(\beta^{(r)})=\sum_{i=1}^{n}(y_{ i}-\pi_{i}^{(r)})x_{i}=X^{T}(y-\pi^{(r)})\] (10) and \[\frac{\partial^{2}}{\partial\beta^{2}}\mathcal{L}_{Y}(\beta^{(r)})=-\sum_{i=1 }^{n}x_{i}x_{i}^{T}\pi_{i}^{(r)}(1-\pi_{i}^{(r)})=-X^{T}W^{(r)}X,\] (11) where \(X=(x_{1},...,x_{n})^{T}\) is the design matrix and where \(\pi^{(r)}=(\pi_{1}^{(r)},...,\pi_{n}^{(r)})^{T}\), with \(\pi_{i}^{(r)}=\sigma(x_{i}^{T}\beta^{(r)})\), where \(\pi^{(r)}(1-\pi^{(r)})\) is the pointwise product \((\pi_{1}^{(r)}(1-\pi_{1}^{(r)}),\pi_{2}^{(r)}(1-\pi_{2}^{(r)}),...,\pi_{n}^{(r )}(1-\pi_{n}^{(r)}))^{T}\). Furthermore, let \(W^{(r)}=\text{diag}(\pi^{(r)}(1-\pi^{(r)}))\) be an \(n\times n\) diagonal matrix where the \(i\)-th diagonal entry is given by \([W^{(r)}]_{ii}=\pi_{i}^{(r)}(1-\pi_{i}^{(r)})\).
* Then, the \(r\)-th update is given by \[\beta^{(r+1)}=\beta^{(r)}-\Big{(}\frac{\partial^{2}}{\partial\beta^{2}} \mathcal{L}_{Y}(\beta^{(r)})\Big{)}^{-1}\frac{\partial}{\partial\beta} \mathcal{L}_{Y}(\beta^{(r)})=\beta^{(r)}+(X^{T}W^{(r)}X)^{-1}X^{T}(y-\pi^{(r) }).\] (12)  
The description of a gradient descent algorithm is straightforward from the above discussion, the difference being that we need to replace the second derivative with a constant learning rate \(\alpha\). In order to obtain good quality, we will need to set a sufficiently small \(\alpha\), and thus the gradient descent will converge in more iterations than the Newton-Raphson method. However, we avoid the computation of the second derivatives in the Hessian matrix which has a computational complexity of \(O(nd^{2})\), whereas computing the first derivative can be performed in \(O(nd)\) steps The big \(O\) notation denotes asymptotic growth. Formally, it holds \(f(x)=O(g(x))\) iff \(|f(x)|\leq cg(x)\) for all \(x\geq x_{0}\) for some constants \(c>0\) and \(x_{0}\).