# Efficient Online Log Parsing with Log Punctuations Signature
## 7 Conclusions  
A qualified general-purpose online log parser should be robust and efficient, which meets the reliability requirements of modern software systems. At present, the state of the art online log parsers are tree-based. Such a structure brings low robustness and inefficiency. To overcome these limitations, we get inspiration from log punctuations and propose a hash-like method.  
This paper proposed LogPunk, a robust and efficient log parser based on our novel log punctuations signature method. The candidate signature group is quickly located according to the log signature, which improves the efficiency. To solve the problem of signature collision, we design the log similarity function to find the most similar template, which improves the robustness.  
Experiments are conducted on 16 public log datasets. Our experimental results show that LogPunk obtains the highest accuracy on ten datasets out of 16 datasets. Especially, LogPunk achieves the best average accuracy of 0.919 among the other five baseline log parsers. In addition, experiments also show that LogPunk is robust and efficient. Finally, some future works are analyzed as follows.  
1. Automated parameters tuning. Logpunk has two hyperparameters _similarity threshold_ and _prefix threshold_ (cf. Section 3.3). During the experiment, these two hyperparameters are fine-tuned manually. This process is time-consuming, and the obtained hyperparameters may not be optimal. A mechanism for automated parameters tuning can greatly improve this situation.
2. Punctuation table generation. The _punctuation table_ (cf. Section 3.2) determines the log signature and affects the whole log parsing process. We presented a punctuation table by eliminating the punctuations appearing in variables, and it performs well on the 16 evaluated datasets. For a new system, if we customize a punctuation table for it, we may get better log parsing results. It is desirable to find a way to generate a customized punctuation table automatically for an unknown system.
3. Variable type identification. Existing log parsers treat all variables in the parsing result as strings. However, obviously, each variable has its specific type information (e.g., number, IP, URL, file path, etc.) and it is useful to detect the variable-related anomaly. If log parsing not only identifies variable but also variable types, it will bring more initiative to downstream tasks.  
Conceptualization, S.Z. and G.W.; methodology, S.Z.; software, S.Z.; validation, S.Z.; investigation, S.Z.; resources, S.Z.; writing--original draft preparation, S.Z.; writing--review and editing, G.W.; visualization, S.Z.; supervision, G.W.; project administration, G.W.; funding acquisition, G.W. All authors have read and agreed to the published version of the manuscript.  
This research received no external funding.  
Publicly available datasets were analyzed in this study. This data can be found here: [[[https://github.com/logpai/loghub](https://github.com/logpai/loghub)]([https://github.com/logpai/loghub](https://github.com/logpai/loghub))].  
The authors declare no conflict of interest.  
The following abbreviations are used in this manuscript:  
\begin{tabular}{l l} AIOps & Artificial Intelligence for IT Operations \\ LCS & Longest Common Subsequence \\ PA & Parsing Accuracy \\ URL & Uniform Resource Locator \\
**Symbols** & \\ \(EV\) & Log Content \\ \(e_{i}\) & A Log Template \\ \(v_{i}\) & A List of Variables \\ \(E\) & The Set of All Log Templates \\ \(L\) & A Sequence of Log Messages \\ \(l_{m}\) & A Log Message \\ \(\mathbf{t_{i}}\) & A List of Tokens \\ \(t_{j}\) & The Token with Index \(j\) \\ \(T\) & The Set of All Tokens \\ \(n\) & Message Length \\ \end{tabular}