# Enhancing Logistic Regression Using Neural Networks for Classification in Actuarial Learning
## 2 Notation and Overview of Techniques
### Neural Networks  
In this section, we give an overview of neural networks and we discuss how they can be used in our setting for the prediction of insurance claims in the French MTPL data we consider for our numerical illustration in Section 4. The design of advanced neural network architectures is a very active research area, with applications ranging from speech recognition to drug design. We refer the reader to [47] for an overview of different neural architectures. A feed-forward neural network consists of several layers such that each layer consists of a number of neurons. The first layer is the input layer and the last layer computes the output of the function. The layers between the input and the output layers are called _hidden_ layers. In Figure 1, we show a neural network with three hidden layers. This is the network that we are using for our model, as more details are provided in the following sections. Each neuron computes a linear combination of all inputs from the previous layer. We can think of the output of a layer as a matrix multiplication with the input from the previous layer where the entries in the matrix should be learned by training the model. A neuron corresponds to a row in this matrix. In order to represent nonlinear functions, we apply a nonlinear activation function to the output of each layer. More precisely, a neural network computes a function \(f\) mapping the features \(x\in\mathbb{R}^{n}\) to the response \(y\in\mathbb{R}^{m}\):  
\[y=f(x)\]  
As the neural network consists of several hidden layers, and each layer can be expressed as a mapping, we see that \(f\) is composed of several functions. In particular, let \(z^{(k-1)}\in\mathbb{R}^{q_{k-1}}\) be the input to the \(k\)-th hidden layer with \(q_{k}\) neurons. Then, the function in the \(k\)-the layer is given by \(f^{(k)}:\mathbb{R}^{q_{k-1}}\to\mathbb{R}^{q_{k}}\) where \(f^{(k)}(z^{(k-1)})=\phi_{k}(W^{(k)}z^{(k-1)})\) for a matrix \(W^{(k)}\in\mathbb{R}^{q_{k-1}\times q_{k}}\) and a nonlinear activation function \(\phi:\mathbb{R}^{q_{k}}\to\mathbb{R}^{q_{k}}\) that is applied entrywise to the vector \(W^{(k)}z^{(k-1)}\in\mathbb{R}^{q_{k}}\). Thus, the \(i\)-th neuron is represented by the \(i\)-th row in the matrix \(W\). Summarizing the above, the \(k\)-th layer computes the function  
\[f^{(k)}:\mathbb{R}^{q_{k-1}}\to\mathbb{R}^{q_{k}},\qquad z^{(k-1)}\mapsto z^{( k)}=f^{(k)}(z^{(k-1)})=\Big{(}f^{(k)}_{1}(z^{(k-1)}),\cdots,f^{(k)}_{q_{k}}(z^{ (k-1)})\Big{)}^{\prime},\]  
The hidden neurons are given by  
\[f^{(k)}_{j}(z^{(k-1)})=\phi\Bigg{(}\beta^{(k)}_{j,0}+\sum_{m=1}^{q_{k-1}}\beta ^{(k)}_{j,m}z^{(k-1)}_{m}\Bigg{)}\equiv\phi\langle\beta^{(k)}_{j},z^{(k-1)} \rangle,\qquad\text{for }j=1,\cdots,q_{k}, \tag{18}\]  
with \(W^{(k)}=\Big{(}\beta^{(k)}_{1},\cdots,\beta^{(k)}_{q_{k}}\Big{)}^{\prime}\in \mathbb{R}^{q_{k}\times(1+q_{k-1})}\). For \(k=1\), the input \(z^{(0)}\) is simply the design matrix \(x\) and \(q_{0}\) is equal to the dimension of the feature space \(\mathcal{X}\). The last layer computes a predicted value for the problem at hand. In our setting, we consider a binary classification problem. Thus, to take into account the exposure of every insurance policy, just as in logistic regression, the output layer is given by  
\[\lambda:\mathbb{R}^{q_{K}}\to\mathbb{R}_{+},\qquad z^{(K)}\mapsto\lambda(z^{( K)})=\sigma\Big{\langle}\beta^{(K+1)},z^{(K)}\Big{\rangle}, \tag{19}\]  
where \(\sigma:\mathbb{R}\to(0,1)\) is the sigmoid function which computes the probability that a given example will be an insurance claim. That is equivalent to adding a layer with \(q_{K+1}=1\) and a sigmoid activation function. This basic architecture of the neural network then has a depth of \(K\) with \(r=\sum_{k=1}^{K}q_{k}(1+q_{k-1})+(1+q_{K})\) parameters to be learned. As a concrete example, we consider the neural network shown in Figure 1. The input comprises \(11\) variables provided in the insurance claim dataset. There are \(K=3\) hidden layers with \(q_{0}=11\), \(q_{1}=100\), \(q_{2}=50\) and \(q_{3}=25\) neurons.