# Enhancing Logistic Regression Using Neural Networks for Classification in Actuarial Learning
## 1 Introduction
* The logistic regression model is represented as a shallow dense neural network with one hidden layer which takes in any number of explanatory variables, or input examples in ML jargon, and constrains the output to be between 0 and 1. This approach enables us to consider several extensions of the logistic regression model and utilize methods for optimization and regularization that come as part of platforms for neural network training. Specifically, we consider the following variants of the logistic regression:  
1. The one-layer, or _shallow_, perceptron neural network (NN) model. This is the representation of the logistic regression model as a neural network with one hidden layer. 2. The one-layer perceptron NN model using embeddings for features. Embedding learning is a more advanced method than one-hot encoding for treating categorical feature components with many nominal levels and it is useful for dealing with categorical variables with many levels in the context of regression analysis. It works by replacing instances of categorical variables by real-valued vectors of fixed size such that the vectors reflect similarity between nominal levels, for different notions of similarity. The embedding technique has its origins in natural language processing (NLP), as can be seen in [39; 40; 41], but it has also been used in other fields, as can be seen, for example, in Ref. [42]. Regarding actuarial applications, the use of embedding learning has recently been proposed by [3; 4; 43]. The approach we adopt is novel and slightly different since we aim at receiving a better predictive model by utilizing precomputed embeddings. More precisely, we replace the categorical variables for the one-layer perceptron NN model representation of the logistic regression with a low-dimensional continuous vector representation that was trained by a deep neural network model in order to cluster labels that are more similar to each other with respect to the given classification problem. 3. The one-layer perceptron NN model with frozen NN weights on the next-to-last layer. Since the balance property holds true under the canonical link choice for the logistic regression model, see, for example, Ref. [37], a deep neural network can learn feature representations from data. Thus, we partition the training dataset into two sub-datasets of equal size. The deep neural network is trained on the first sub-dataset, and then we feed the second sub-dataset into the pretrained model and we extract the weights of the last hidden layer. In this way, we obtained a new, augmented dataset with new features of different dimensionality. These new features represent nonlinear interactions between the original features. Finally, the one-layer perceptron NN model representation of the logistic regression is fitted on the augmented dataset. 4. The one-layer perceptron NN model with transfer learning. The idea is to use embeddings for categorical variables that are obtained from a deep neural network which has been pretrained either on the same or another dataset for another problem which nonetheless shares certain characteristics with the current setting. There are several benefits of using transfer learning. Firstly, one can benefit from other neural network-based models without the need to have access to the data they were trained on. This way, we can convert any regression type problem into a classification type problem and vice versa. Secondly, one can improve the training time for larger datasets. Thirdly, the one-layer perceptron NN model representation of the logistic regression can be updated with new observations, for instance, we can have new insurance policies with new risk characteristics entering the portfolio. Finally, the one-layer perceptron NN model representation of the logistic regression becomes more robust since transfer learning can be seen as a form of regularization.