## 3 Neural Network Based DPD Architecture
### Features of the NN Model
#### 3.1.5 Different Methods of Regularization  
All datasets are divided into two sections, namely, the training data and the testing data. Sometimes, we come upon conditions where models do not result in good performance on testing data despite performing well on training data. These situations arise due to underfitting and overfitting in Figure 6. Regularization is the procedure in which weight matrices of nodes are penalized, hence overcoming these problems.  
L1 and L2 are simple and widely known types. A regularization term is considered while updating the cost function in this method. This regularization term causes a reduction in the value of the weight matrices as they result in a simpler model, significantly reducing the problem of overfitting.  
In L2, we have:  
\[Loss=error+\lambda\sum_{i=1}^{N}\left|w_{i}^{2}\right| \tag{3}\]  
where \(\lambda\) is the regularization parameter (which is updated for more precise outcomes) and \(w\) is weight. The L2 method is referred to as a weight decay procedure because it compels the weights to decay towards zero although never exactly becoming zero.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Hyperparameter** & **Definition** & **Explanation** \\ \hline \multirow{3}{*}{Batch size} & Number of samples processed & GD = Size of Training Set; SGD = 1; \\  & while updating the internal & B-SGD = 1 \(<\) Batch Size \(<\) Size of \\  & parameters of a model. & Training Set \\ \hline \multirow{3}{*}{Epoch} & Number of complete passes & For MB-SGD if batch size = 25, \\  & through the samples & then number of complete passes \\  & being trained. & through these 25 samples is \\  & & number of epochs. \\ \hline \multirow{3}{*}{Regularization} & Penalizes weight matrices & \multirow{3}{*}{Methods-L1, L2, Dropout.} \\  & to resolve overfitting & \\  & and underfitting & \\ \hline \hline \end{tabular} Note that there are no pre-defined fixed values for these hyperparameters. There are some conventionally used values, but they are not mandatory and attaining a value suitable to a function is a trial-and-error method.  
\end{table}
Table 4: Definition of Hyperparameters.  
Figure 6: Graphical representation of the training curve (black) learning the training data (purple, scattered) in three labelled cases.  
In L1, we have:  
\[Loss=error+\lambda\sum_{i=1}^{N}\lvert w_{i}\rvert \tag{4}\]  
As observed in Equation (4) here, the absolute weight value is considered. Contrary to the L2 method, the weights here can become zero which is beneficial for compression of models.  
Dropout  
Dropout is the most used type of regularization due to its efficient results. It performs several iterations on the NN and at every iteration, random nodes are selected and removed along with all of their input and output signals/connections as seen in Figure 7. This results in selection of distinct nodes and hence distinct outputs per iteration. More randomness is expressed by splitting one network into its subsets leading to better performance when compared to a single dense model. The method can be implemented on input as well as hidden layers making it an adequate choice for large NNs.