# Deep Learning-Based Log Parsing for Monitoring Industrial ICT Systems
## 6 Evaluation and Results
### Experimental Methodology  
**Selected Datasets**: For a comprehensive performance evaluation, we selected 18 different log sets collected from different practical systems, as shown in Table 1, where 'Log Size' indicates the number of log records, and 'Events' indicates the number of log event types. Specifically, _HDFS_[5] is generated in a private cloud environment using a benchmark workload. _BGL_[26] is a supercomputer log set collected at Lawrence Livermore National Labs (LLNL) in Livermore, CA. _Spark_ is collected from our three-node cluster server that hosts our lab's calculation tasks. _Apache_ is an open-source Apache server access log set. _Uof_S[27] is a trace set containing all HTTP requests to the WWW server at the University of Saskatchewan within seven months. _Jul95_[27] is a trace set containing all HTTP requests to the WWW server at NASA Kennedy Space Center in Florida, within two months. _Nginx_ is an open-source Nginx server access log set. _Openstack_[28] is generated in CloudLab. _Security Log_[29] contains connection logs collected and used by Security Data Analysis Labs. _Thunderbird_[26] is a supercomputer log set collected from Sandia National Labs (SNL) in Albuquerque, NM. _Big Brother_[30] is a diagnostic log set from IEEE VAST Challenge 2013. _Web Log_[6] is a publicly available web log set. _ThingWorx_[7] contains system runtime logs collected from the ThingWorx platform, an Industrial Internet of Things (IIoT) platform, and _4 SICS-151020_, _4 SICS-151021_, _and 4 SCIS-151022_[31] are network traffic data captured from industrial network equipment by the industrial cyber security conference 4 SICS. _EMSD_ includes 30 days of Energy Management Systems logs ([[https://sites.google.com/a/uah.edu/tommy-morris-uah/ics-data-sets](https://sites.google.com/a/uah.edu/tommy-morris-uah/ics-data-sets)]([https://sites.google.com/a/uah.edu/tommy-morris-uah/ics-data-sets](https://sites.google.com/a/uah.edu/tommy-morris-uah/ics-data-sets)) accessed on 4 August 2020). _IoT sentinel_[32] contains the traffic emitted during the setup of 31 smart home IoT devices.  
**Comparisons**: We compared the performance of LogParser with that of multiple log parsing methods, i.e., _IPLoM_[11], _Drain_[12], _POP_[13], and _NuLog_[16]. All comparatives are implemented in Python. For POP, we used the single-node version. All parameters were set first as recommended and then tuned according to the actual performance.  
**Evaluation Metric**: We used _the F1-score_, or the harmonic mean of the _precision_ and _recall_, to evaluate the log parsing performance. Specifically, for a certain type of event, precision is defined as the number of correctly predicted records divided by the number of all records identified as such a type of event, and recall is defined as the number of correctly predicted records divided by the number of all records actually labeled as such a type of event (i.e., the ground truth).  
**Implementation**: For the implementation of LogParser, to take advantage of state-of-the-art techniques, we used Tensorflow ([[https://www.tensorflow.org/](https://www.tensorflow.org/)]([https://www.tensorflow.org/](https://www.tensorflow.org/)) accessed on 4 August 2020) (v1.7.0), an open-source software library for high-performance numerical computation. More specifically, we used a high-level API of Tensorflow, Keras ([[https://keras.io/](https://keras.io/)]([https://keras.io/](https://keras.io/)) accessed on 4 August 2020) (v2.1.5), to implement LogParser in Python. All of our evaluations were conducted on a Linux server (with 64-bit Ubuntu 16.04.1, Linux kernel 4.13.0) with an Intel Xeon E5-2650v4 CPU and 512 GB RAM.