## 3 Deep Neural Network
### Building Neural Network for Classification Tasks
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Parameter** & & **Characteristic** & **Advantage** & **Disadvantage** \\ \hline \multirow{8}{*}{Sigmoid} & \multirow{4}{*}{ReLU} & \multirow{4}{*}{Sigmoid takes a real value as \(\bullet\)} & Smooth gradient & \multirow{4}{*}{\(\bullet\)} & Vanishing \\  & & input and outputs another & & & Good for a classifier \\  & & value between 0 and 1. It’s & \multirow{4}{*}{} & Have activations & \multirow{4}{*}{\(\bullet\)} & Not zero centered \\  & & easy to work with and has all & & bound in a range & & Signal saturates \\  & & the nice properties of & & & and kills gradients \\  & & activation functions [25] & & & & \\ \hline \multirow{8}{*}{Activation function} & \multirow{4}{*}{Tanh} & \multirow{4}{*}{Tanh is also Sigmoidal (s-shaped) [25]} & \multirow{4}{*}{\(\bullet\)} & Biological plausibility & \multirow{4}{*}{\(\bullet\)} & Non-differentiable \\  & & & linear unit. Mathematica & & & Better \\  & & form: \(y=\max(0,1)\)[26,27] & & & gradient propagation & \\ \cline{1-1}  & & & efficient computation & & \\ \cline{1-1}  & & & & & \\ \cline{1-1}  & & & & & \\ \cline{1-1}  & & & & & \\ \cline{1-1}  & & & & & \\ \hline \multirow{8}{*}{SELU} & \multirow{4}{*}{SELU} & \multirow{4}{*}{\(\bullet\)} & SELU stands for Scaled & \multirow{4}{*}{\(\bullet\)} & Network \\  & & Exponential Linear Unit. It is & & converges faster. & \multirow{4}{*}{\(\bullet\)} & Relatively new \\  & & self-normalizing the neural & & No problem of \\  & & network [27] & & Vanishing and \\ \cline{1-1}  & & & & & \\ \cline{1-1}  & & & & & \\ \hline \multirow{8}{*}{Adam} & \multirow{4}{*}{\(\bullet\)} & Using moving average of the & \multirow{4}{*}{\(\bullet\)} & Only requires \\  & & gradient instead of gradient & & first-order gradients \\ \cline{1-1}  & & itself [28] & & with little \\ \cline{1-1}  & & & & & \\ \cline{1-1}  & & & & & \\ \cline{1-1}  & & & & & \\ \cline{1-1}  & & & & & \\ \cline{1-1}  & & & & & \\ \cline{1-1}  & & & & & \\ \hline \multirow{8}{*}{Optimizer} & \multirow{4}{*}{RMSProp} & \multirow{4}{*}{\(\bullet\)} & A gradient-based & \multirow{4}{*}{\(\bullet\)} & Differentiation \\  & & optimization technique used & & between parameters \\  & & in training neural networks & is maintained and & \\  & & & prevent convergence & \\  & & to zero [31] & & \\ \hline \multirow{8}{*}{AdaGrad} & \multirow{4}{*}{\(\bullet\)} & An optimizer with & \multirow{4}{*}{Well-suited for} \\  & & parameter-specific learning & dealing with \\ \cline{1-1}  & & rules, which are adapted & & sparse data \\ \cline{1-1}  & & relative to how frequently a & \multirow{4}{*}{\(\bullet\)} & Lesser to \\ \cline{1-1}  & & parameter gets updated & & manually tune \\ \cline{1-1}  & & during training [3] & & learning rate \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Characteristics of activation functions and optimizers. Finally, model fitting is required to execute the model. The training process will run for a fixed number of iterations over the dataset called epochs. The number of dataset rows is also arranged before the model weights are updated within each epoch, called the batch size. The number of epochs means the number of times that the algorithm repeatedly learns the entire training dataset. One epoch means that each sample of the training dataset can be used to update internal model parameters. The size of epochs is usually large up to hundreds to thousands to minimize errors sufficiently. The batch size is the number of samples processed before the model is updated. The numbers of epochs and batch sizes can be determined experimentally through trial and error. The model must be sufficiently trained to well map the rows of input data to the output classification. There are always some errors in the model, but after a certain point in time for a given model configuration, the model converges, and the amount of error is reduced. In this study, 2000 epochs and 200 batch sizes were used. Figure 5 shows the learning process of the neural network.