## 3 Neural Network Based DPD Architecture
### Features of the NN Model
#### 3.1.2 Optimizers  
The process of optimizing loss functions improves the function of the model. The procedure to minimize/maximize a mathematical function is called optimization and utilizes optimizers. Optimizers work by changing the parameters of NNs such as the learning rates or weights. The types of optimizers are:  
Gradient Descent (GD)  
The GD algorithm works on the principle that the direction opposite to that of the calculated gradient will locate the lower point/surface of the considered function, i.e., the potential minima. It then repeatedly takes steps in that direction with every iteration. In the case of GD, we require large memory as we calculate the gradient over the entire dataset after which the updates are performed meaning that, for huge data, it may take years to converge. Stochastic Gradient Descent (SGD)  
To make up for the problem faced by the GD algorithm, SGD was developed. The SGD algorithm works by computing derivatives by considering only one point at once. This solves the large memory requirement problem; however, time is still an issue as one epoch of SGD takes more time when compared to the GD algorithm. Mini Batch Stochastic Gradient Descent (MB-SGD)  
This algorithm was developed to overcome time complexity problems. MB-SGD as the name suggests, works by considering a small batch of points or a small subset of the whole data at once to compute the derivatives. It takes longer to converge than GD does and additionally, its weight updates are noisy. SGD with momentum (momentum for short) calculates another parameter, momentum in addition to the gradient in every step. This momentum is the cumulative movement from all previous steps, overcoming the issue of noisy weight updates in MB-SGD by denoising gradients. It moves faster owing to the accumulated momentum which helps it prevent local minima and plateau regions. In addition, the algorithm Nesterov Accelerated Gradient (NAG) is similar to momentum except, it is a future considering method. The cost function is evaluated using future parameters rather than the current ones to prevent the chance of skipping the minima in case of high momentum value. Adaptive Gradient Descent (AdaGrad)  
AdaGrad, works by accumulating the sum of gradient'squared'. As the name suggests the learning rate is adaptive and doesn't need to be tuned manually. The adaptive learning rate property helps escape typical complexities of non-convex functions such as saddle points as they take a straight path compared to methods such as GD, SGD, MB-SGD and even momentum. There are chances that these methods become stuck at saddle points and never converge to minima. Unfortunately, AdaGrad is very slow as the sum it accumulates continues to grow resulting in the learning rate to become very small, this consequently leads to vanishing of the gradient. Adaptive Delta (AdaDelta)  
As we know, AdaGrad is a slow converging method due to its large value of the accumulated sum of gradient squared. AdaDelta was developed to resolve this by modifying certain aspects of AdaGrad. Here, rather than storing all the past gradients inefficiently, we recursively define the sum of gradients as a decaying average of all the past squared gradients. This allows AdaDelta to continue learning even after multiple updates. RMSprop (Root Mean Square Propagation)  
RMSprop, similar to AdaDelta, was developed to counter AdaGrad's problem of vanishing learning rates. RMSProp fixes the issue by adding a decay factor which emphasizes on recent gradients neglecting the older ones. An exponentially decaying average of squared gradients is used to divide the learning rate. AdaGrad might keep up with RMSProp initially, but the sum of gradients squared for AdaGrad accumulate and become huge, eventually AdaGrad practically stops moving, in contrast with RMSProp, smartly stored the squares due to the decay rate. This makes it faster than AdaGrad and it impedes search in direction of oscillations.