# Enhancing Logistic Regression Using Neural Networks for Classification in Actuarial Learning
## 2 Notation and Overview of Techniques
### Neural Networks
#### 2.7.3 More on Gradient Descent Optimization  
Gradient descent has become the ubiquitous approach to training neural networks, and is the method of choice in literally all platforms for neural network training. As discussed in Section 2.1, we can apply the Newton-Raphson algorithm which converges in fewer iterations. However, this would require the computation of the second derivative with respect to the parameters at each layer, i.e., we need the Jacobian matrix of the layer parameters \(\beta_{k}\). For larger networks with many parameters, this is unfeasible as the Jacobian will have \((q_{k}q_{k-1})^{2}\) parameters, where \(q_{k-1}\) and \(q_{k}\) is the number of input and output neurons at the \(k\)-th layer. Even for our relatively small neural network from Figure 1, this would mean a Jacobian matrix with 25 million entries.  
However, gradient descent comes with many challenges that need consideration:We can get stuck in a local optimum which might result in a solution which is not good enough. Furthermore, for a non-convex function there might exist a saddle point where, in one dimension, the function has a positive slope, and a negative slope in other dimension. The derivative of a function at a saddle point is 0 but the saddle point is not a local optimum.
2. A small learning rate would lead to very slow convergence, and a large learning rate might lead to oscillation around the optimum.
3. Computing the gradient with respect to the whole training dataset might result in very slow updates.
4. For certain loss functions, we might encounter the problem of vanishing gradients. Consider the sigmoid function \(\sigma(z)\), for large or small values of \(z\) the gradient will be close to 0, and thus we will need many iterations until we reach an optimum.
5. The learning rate needs to be adapted in order to address changes in the curvature for complex functions. In particular, the gradients in different directions might be of largely different magnitudes, and thus using a constant learning rate might lead to oscillations. Researchers have approached the above issues using various techniques such as
1. In particular, the choice of activation function is of critical importance. We discuss different activation functions and how we use them in our architecture in greater detail in the next subsection.
2. Mini-batch training. Compute the gradients not for the whole dataset but for a batch of examples.
3. Adaptive learning rate. The idea is that the learning rate is not a constant but a function of the gradients of the past iterations of the optimization process. A popular choice in practice is the Adam approach (adaptive moment estimation) where the learning rate is adjusted after each iteration by using an exponentially decreasing average of past gradients. We refer to [48] for more details on the approach.
4. Parameter initialization. We start to train the neural network model by initializing the parameters with random values. However, the magnitude of these random values has to be carefully selected as it might lead to slower training or even vanishing or exploding gradients.  
Note that the above list is not exhaustive since, unfortunately, neural network optimization relies on heuristic techniques and trial-and-error-based hyperparameter tuning, and is thus often characterized as being more of an art than science. Regarding the classification problem we consider herein, we provide more details in the next section wherein we discuss the results of our models.