# Efficient Online Log Parsing with Log Punctuations Signature
## 4 Evaluation
### Accuracy  
As illustrated in Table 2, we compare the accuracy of LogPunk with eight other baseline log parsers on 16 log datasets. According to the row, we can compare the PA of different log parsers on the same dataset. Moreover, the column demonstrates the accuracy distribution of the same log parser across datasets. Following the prior work [15], PA values greater than 0.9 are highlighted in bold, and the best PA values of each dataset are marked with asterisk \"*\".  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Platform** & **Description** & **\#Templates(2k)** & **\#Templates (Total)** & **Length (Max, Average)** & **Size** \\ \hline Android & Android framework & 166 & 76,923 & 32, 13.31 & 183.37 MB \\ Apache & Apache web server error & 6 & 44 & 14, 12.28 & 4.90 MB \\ BGL & Blue Gene/L supercomputer & 120 & 619 & 84, 15.32 & 708.76 MB \\ Hadoop & Hadoop map reduce job & 114 & 298 & 50, 14.82 & 48.61 MB \\ HDFS & Hadoop distributed file system & 14 & 30 & 111, 12.45 & 1.47 GB \\ HealthApp & Health app & 75 & 220 & 14, 4.93 & 22.44 MB \\ HPC & High performance cluster & 46 & 104 & 47, 9.56 & 32.00 MB \\ Linux & Linux system & 118 & 488 & 24, 14.39 & 2.25 MB \\ Mac & Mac OS & 341 & 2214 & 249, 15.49 & 16.09 MB \\ OpenSSH & OpenSSH server & 27 & 62 & 19, 13.81 & 70.02 MB \\ OpenStack & OpenStack infrastructure & 43 & 51 & 31, 20.63 & 58.61 MB \\ Proxifier & Proxifier software & 8 & 9 & 27, 13.73 & 2.42 MB \\ Spark & Spark job & 36 & 456 & 22, 12.76 & 2.75 GB \\ Thunderbird & Thunderbird supercomputer & 149 & 4040 & 132, 17.52 & 29.60 GB \\ Windows & Windows event & 50 & 4833 & 42, 31.93 & 26.09 GB \\ Zookeeeper & ZooKeeper service & 50 & 95 & 26, 13.46 & 9.95 MB \\ \hline \hline \end{tabular}
\end{table}
Table 1: Datasets overview.  
From Table 2, we can observe that LogPunk achieves the best accuracy on ten datasets out of 16, which significantly outperforms other baseline methods. In addition to that, LogPunk achieves over 0.9 accuracy on 13 datasets. In the remaining datasets, LogPunk also has a comparable accuracy. On average, LogPunk has the best accuracy of 0.919, followed by Drain+ of 0.877.  
In addition, Spell+ and Drain+ are better than the original Spell and Drain. Their accuracy is improved by 0.055 and 0.011, respectively, which shows that our signature method is effective and can be applied to other log parsers as an enhancement.  
The average accuracy of the HDFS and Apache datasets almost reaches 100%. Because HDFS and Apache logs have relatively simple structures, and the number of templates is small. Meanwhile, some datasets could not be parsed accurately due to their complex structure and abundant event templates, such as Mac and Linux. All log parsers perform poorly on the Proxifier dataset because it is skewed. It only has eight templates in the sample set, while template E8 takes up 947 out of 2000, and it is not easy to parse.  
LogPunk has the best accuracy for the following reasons. First, our effective signature method with a collision index of 1.25 avoids template crowding in one signature group. Even though the whole template set is complex, the situation is much simpler in each signature group. Second, adequate delimiters ensure the separation of variables and templates to deal with some complex situations. Third, we provide two hyperparameters (cf. Section 3.3. _similarity threshold_ and _prefix threshold_), which can be set flexibly according to the data feature.