# Deep Learning-Based Log Parsing for Monitoring Industrial ICT Systems
## 6 Evaluation and Results
### Performance of the Offline Parser
#### 6.2.1 General Performance  
Table 2 shows the log parsing accuracy based on IPLoM, Drain, POP, NuLog, and our offline parser on all selected datasets. It is obvious that our method significantly outperforms all comparatives on all datasets. Furthermore, we notice that the conventional methods induce a quite low performance in certain scenarios, e.g., IPLoM on Jul95. We believe that the reason for this is two-fold: First, such methods inherently require domain knowledge for accuracy improvement while only basic data preprocessing was conducted in our evaluations. Second, the classification is not conducted based on the structural features of the log text. According to Table 2, on Nginx, when the classification criterion is'request type' (i.e., Nginx-4 events), all methods manage to achieve accurate classification. However, when the criterion is changed to 'protocol version' (i.e., Nginx-2 events), unlike our offline parser, none of the comparatives demonstrates an acceptable performance. NuLog, which adopted machine learning techniques, achieved higher log parsing accuracy than other conventional methods. However, it still does not reach an acceptable performance (less than 0.70 precision) on some datasets, i.e., Nginx-2, Nginx-20, Thunderbird, Web log, and ThingWorx. We believe that this is due to the lack of task-specific data preprocessing. The self-supervised learning approach could not automatically adapt to the mining task requirement. Such results indicate that _our offline parser manages to support user-defined classification of log events that cannot be achieved by existing methods_.  
Then, we further analyzed the results on ThingWorx, which implied that there are two major reasons for the low accuracy based on comparatives: First, the comparatives can  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **IPLoM** & **Drain** & **POP** & **NuLog** & **Offline** \\ \hline HDPS & 0.88 & 0.88 & 0.88 & 0.99 & 1.00 \\ \hline BGL & 0.46 & 0.29 & 0.66 & 0.95 & 0.93 \\ \hline Spark & 0.57 & 0.63 & 0.63 & 0.99 & 0.99 \\ \hline Apache & 0.06 & 0.48 & 0.93 & 1.00 & 1.00 \\ \hline UofS & 0.11 & 0.01 & 0.28 & 0.89 & 0.91 \\ \hline Jul95 & 0.00 & 0.91 & 0.28 & 1.00 & 1.00 \\ \hline Nginx-4 events & 1.00 & 0.97 & 0.83 & 1.00 & 0.99 \\ \hline Nginx-2 events & 0.08 & 0.50 & 0.53 & 0.65 & 0.99 \\ \hline Nginx-20 events & 0.00 & 0.48 & 0.52 & 0.65 & 0.99 \\ \hline Openstack & 0.73 & 0.74 & 0.72 & 0.99 & 1.00 \\ \hline Security Log & 1.00 & 0.30 & 0.59 & 1.00 & 0.99 \\ \hline Thunderbird & 0.07 & 0.04 & 0.26 & 0.67 & 0.93 \\ \hline Big Brother & 0.18 & 0.54 & 0.55 & 0.73 & 0.95 \\ \hline Web log & 0.00 & 0.05 & 0.08 & 0.12 & 0.96 \\ \hline ThingWorx & 0.22 & 0.26 & 0.72 & 0.52 & 0.99 \\ \hline
4 SICS-151020 & 0.61 & 0.77 & 0.73 & 0.95 & 1.00 \\ \hline
4 SICS-151021 & 0.44 & 0.70 & 0.64 & 0.93 & 0.99 \\ \hline
4 SICS-151022 & 0.38 & 0.64 & 0.58 & 0.91 & 0.95 \\ \hline EMSD & 0.27 & 0.60 & 0.46 & 0.74 & 0.95 \\ \hline IoT sentinel & 0.54 & 0.65 & 0.54 & 0.88 & 0.95 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Offline parsing result among all evaluation datasets.  
barely recognize certain types of events, e.g., the type five event 'anonymous error message' cannot be recognized by IPLoM (i.e., with 0.0 precision). Then, all comparatives induce low recalls except for those from POP and NuLog, which are still significantly lower than that from our method. Such results indicate that, unlike our offline parser, _the existing methods cannot effectively differentiate events in certain scenarios._