# GPON PLOAMd Message Analysis Using Supervised Neural Networks
## 4 PLOAMd Analysis Using Various Recurrent Neural Networks and Results Discussion
### Evaluation of Models  
All of the models learn to distinguish correct and faulty samples of the training dataset, which is clearly visible in average loss (the model loss represents output of objective function used during learning process) and average accuracy (the model accuracy represents number of correctly classified samples compared to all samples) evolution over learning epochs by blue solid line in Figures 2-4. In the figures, there is also the standard deviation for the average values represented by transparent fill  
Figure 3: Average loss and accuracy of the models with LSTM over learning epochs.  
Figure 2: Average loss and accuracy of the models with an RNN over learning epochs.  
in the same color. All inspected models have similar performance according to model metrics based on training dataset. Considering the standard deviation, it is difficult to choose one as the best.  
The quality of learned capabilities is evaluated by classifying the validation dataset, which is disjunctive to the learning dataset. The evolution of accuracy and loss over the epochs is visible by orange dotted line in Figures 2-4. The model based on SimpleRNN layer finds patterns in the data much more slowly compared to LSTM- or GRU-based models. This behavior is expected due to RNN being the simplest recurrent architecture and having fewer controlling instruments. All SimpleRNN model metrics in validation dataset indicate that the model has the worst performance of the all inspected models.  
The LSTM-based model has better accuracy and the other metrics compared to SimpleRNN. This model finds and learns correct characteristic patterns. However, this model has the largest number of arguments, thus it is resource demanding and the learning process take longer to proceed.  
The GRU-based model complexity is in between LSTM and SimpleRNN layers, but the validation dataset accuracy and other metrics point out that this model is the best of the inspected models. It is able to find distinguishing patterns better than SimpleRNN- and LSTM-based models. The GRU-based model has fewer gates (and learning parameters as well) and the learning process takes less time, even though it has the best performance according to validation dataset metrics. The training results and a summary of all model metrics are shown in Table 2.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{2}{c}{**SimpleRNN**} & \multicolumn{2}{c}{**LSTM**} & \multicolumn{2}{c}{**GRU**} \\ \hline \multirow{4}{*}{Learning} & loss & \(0.04897\pm 0.00470\) & \(0.11183\pm 0.05380\) & \(0.06980\pm 0.01116\) \\  & accuracy & \(0.97088\pm 0.00318\) & \(0.95212\pm 0.01856\) & \(0.96655\pm 0.00392\) \\  & recall & \(0.97005\pm 0.00491\) & \(0.93242\pm 0.02067\) & \(0.95103\pm 0.00751\) \\  & precision & \(0.97176\pm 0.00380\) & \(0.97085\pm 0.01821\) & \(0.98162\pm 0.00295\) \\  & auc & \(0.99846\pm 0.00026\) & \(0.99060\pm 0.00957\) & \(0.99685\pm 0.00106\) \\ \hline \multirow{4}{*}{Validation} & loss & \(1.11480\pm 0.41351\) & \(0.47466\pm 0.12479\) & \(0.43730\pm 0.12738\) \\  & accuracy & \(0.74893\pm 0.07493\) & \(0.85820\pm 0.02028\) & \(0.87654\pm 0.02105\) \\ \cline{1-1}  & recall & \(0.70229\pm 0.09837\) & \(0.80995\pm 0.03961\) & \(0.84339\pm 0.04841\) \\ \cline{1-1}  & precision & \(0.76943\pm 0.08938\) & \(0.89800\pm 0.03948\) & \(0.90251\pm 0.01986\) \\ \cline{1-1}  & auc & \(0.80492\pm 0.08378\) & \(0.92984\pm 0.01851\) & \(0.93931\pm 0.01630\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Recurrent models mean metrics with standard deviation.  
Figure 4: Average loss and accuracy of the models with GRU over learning epochs.