# Deep Learning-Based Log Parsing for Monitoring Industrial ICT Systems
## 6 Evaluation and Results
### Performance of the Offline Parser
#### 6.2.3 Performance of Multi-Source Log Parsing  
In this set of experiments, we compared the performance of LogParser to those of all comparatives on logs from multiple sources. To achieve this, we randomly selected a subset of log records with sizes from 100 k to 300 k from different log sets in Table 1 and constructed a _heterogeneous multi-source log set_ (see Table 4) and a _homologous multi-source log set_ (see Table 5), respectively. Specifically, _heterogeneous_ represents logs generated by different components, and _homologous_ represents logs generated by similar components. The experimental results are demonstrated in Figure 6 and Figure 7, respectively. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Label** & **Source Dataset** & **Subset Size** & **Subset Events** \\ \hline heter\_s\({}_{1}\) & HDFS & 191469 & 14 \\ \hline heter\_s\({}_{2}\) & Spark & 336890 & 23 \\ \hline heter\_s\({}_{3}\) & Apache & 175383 & 14 \\ \hline heter\_s\({}_{4}\) & BGL & 355266 & 23 \\ \hline heter\_s\({}_{5}\) & Openstack & 113885 & 9 \\ \hline heter\_s\({}_{6}\) & Security Log & 67833 & 5 \\ \hline heter\_s\({}_{7}\) & Thunderbird & 140007 & 7 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Heterogeneous multi-source dataset. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **IPLoM** & **Drain** & **POP** & **NuLog** & **Offline** \\ \hline HDFS(5117) & 0.97 & 0.91 & 0.93 & 1.00 & 1.00 \\ \hline HDFS(48472) & 0.98 & 0.90 & 0.98 & 1.00 & 1.00 \\ \hline HDFS(513053) & 0.98 & 0.98 & 0.98 & 1.00 & 1.00 \\ \hline HDFS(4938022) & 0.77 & 0.94 & 0.94 & 1.00 & 1.00 \\ \hline HDFS(11197705) & 0.88 & 0.88 & 0.88 & 1.00 & 1.00 \\ \hline Spark(4703) & 0.80 & 0.89 & 0.90 & 1.00 & 0.95 \\ \hline Spark(53908) & 0.76 & 0.89 & 0.89 & 1.00 & 1.00 \\ \hline Spark(547260) & 0.66 & 0.89 & 0.87 & 1.00 & 1.00 \\ \hline Spark(4927590) & 0.58 & 0.87 & 0.83 & 1.00 & 0.99 \\ \hline Spark(11215954) & 0.56 & 0.85 & 0.82 & 0.99 & 0.99 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Offline parsing result over different log size subset from HDFS and Spark datasets. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Label** & **Source Dataset** & **Subset Size** & **Subset Events** \\ \hline heter\_s\({}_{1}\) & HDFS & 191469 & 14 \\ \hline heter\_s\({}_{2}\) & Spark & 336890 & 23 \\ \hline heter\_s\({}_{3}\) & Apache & 175383 & 14 \\ \hline heter\_s\({}_{4}\) & BGL & 355266 & 23 \\ \hline heter\_s\({}_{5}\) & Openstack & 113885 & 9 \\ \hline heter\_s\({}_{6}\) & Security Log & 67833 & 5 \\ \hline heter\_s\({}_{7}\) & Thunderbird & 140007 & 7 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Heterogeneous multi-source dataset. According to Figure 6, we can see that, as the number of heterogeneous log sources increases, our offline parser has a negligible accuracy loss (i.e., no more than 5%), and NuLog has a slight accuracy loss (i.e., no more than 10%), while the accuracy of all comparatives drops significantly (i.e., from 29% to 32%). The reason, we believe, is that integrating heterogeneous log records from new sources into the original log set changes the original pattern of word distribution drastically. NuLog is able to learn the new pattern according to its self-supervise mechanism. However, it was still less effective than our method, which learned from labels. All conventional comparatives, however, rely on a single set of parameters to identify the event type of heterogeneous logs from multiple sources, which is difficult to achieve and downgrades the performance significantly. According to Figure 7, the results on the homologous multi-source log set are similar to those on the heterogeneous set, and our offline parser (with an accuracy loss of no more than 8%) outperforms all comparatives (with accuracy losses from 31% to 99%) more obviously.