# Deep Learning-Based Log Parsing for Monitoring Industrial ICT Systems
## 6 Evaluation and Results
### Performance of the Online Parser
#### 6.3.1 General Performance  
According to our design, before the online parser is updated, each incoming log record may imply a new type of event, i.e., the training set of the loaded offline parser contains  
Figure 6: Offline parsing accuracy on heterogeneous multi-source datasets.  
Figure 7: Offline parsing accuracy on homologous multi-source datasets.  
no log record implying the same event type of such an incoming log record. We use \(r\) to represent the _ratio of log records implying new types of events to the number of all records from the stream within a certain period of time._ Theoretically, a higher \(r\) should induce a more obvious impact on the online parsing accuracy, and an ideal online log parser should be able to prevent the accuracy from downgrading when \(r\) increases. Therefore, in this set of experiments, we evaluated the performance of our online parser on log streams with different settings of \(r\).  
As streams, the log sets used by the evaluation of our offline parser (see Table 1) were fed into our online parser, our offline parser, Drain, and NuLog, respectively, for comparisons. Specifically, we fed complete log sets to our offline parser, Drain, and to NuLog as our baseline and comparative, respectively. Furthermore, log sets with \(r\in\{0.2,0.4,0.6,0.8,1.0\}\) were fed to our online parser, respectively, to study the impact of \(r\) (for example, \(r=0.2\) represents that log records implying 20% of all event types are not selected by the training set of the offline parser). Figure 8 shows the online parsing results on the _HDFS_ and _Spark_ log sets (due to the page limitation, we only present such results for illustration), which demonstrates the F1-scores of all methods when different numbers of log records (up to \(10^{7}\)) were fed.  
According to Figure 8, it is obvious that our online parser outperforms Drain in all scenarios (i.e., from 2% to 15% in terms of accuracy), except for when \(r>0.6\) and 4703 records were fed on the Spark log set. NuLog, on the other hand, performs evenly with our online parser since it already trained the whole dataset in advance. In cases of such small dataset sizes with limited trained log event types, learning cost leads to a reduction in the final accuracy.  
For our online parser, as we expected, a higher \(r\) induces a more obvious accuracy loss. However, the parsing accuracy increases as the number of log records fed increases. When more than 5 million records are fed, a 1.0 F1-score is achieved, even when \(r=1.0\) (i.e., all incoming logs are with new event types). Meanwhile, for all settings of \(r\), the accuracy difference between our online and offline parsers keeps reducing, which becomes negligible (i.e., up to 4%) when 50 k records were fed.  
For certain cases, e.g., 4703 records, \(r=0.2\), more than 5 million records, and \(r\in\{0.2,0.4,0.6,0.8,1.0\}\), our online parser achieves higher (up to 2%) accuracy compared to our offline parser. We believe that this comes from the difference between the offline evaluation procedure and the online evaluation procedure. The offline parser needs  
Figure 8: Online parsing accuracy on datasets of different sizes.  
to find a global template to identify all log events, while the online parser only needs to focus on the recent log records.