# Automatic Parsing and Utilization of System Log Features in Log Analysis: A Survey
## 2 Log Parsing
### Accuracy of Parsing  
#### 2.2.1 Evaluation Indicators  
There are many evaluation indicators for log parsers, such as accuracy, efficiency, etc. In addition, researchers have compared and verified the adaptability and incremental learning ability of the parsers [34]. For the log parser, the most important attribute is the accuracy of log parsing. It is no exaggeration to say that the accuracy of parsing is directly related to the completion of downstream tasks. As a machine learning task, traditional evaluation indicators such as _Precision_, _Recall_, and _F-measure_ are applied to log parsing.  
\[Recall=\frac{TP}{TP+FN} \tag{1}\]  
\[Precision=\frac{TP}{TP+FP} \tag{2}\]  
\[F_{1}=2\cdot\frac{Precision\cdot Recall}{Precision+Recall} \tag{3}\]  
_TP_ refers to the number of logs that have been parsed into the correct log template, _FN_ refers to the number of logs that have been parsed into the current log template by mistake, and _FP_ refers to the number of logs that have been parsed into other error log templates. However, in this field, the accuracy indicators most commonly used by researchers in recent years are Parsing Accuracy (PA) and the Rand Index (RI).  
In order to better quantify the accuracy of the log parser, researchers proposed the evaluation indicators of parsing accuracy [38]. They can be interpreted as the ratio of the number of accurately parsed log messages to all log messages, as shown in Formula (4). For example, for the log message sequence \([L_{1},L_{2},L_{3}]\), the parsed template sequence is \([T_{1},T_{2},T_{3}]\), but the actual template is \([T_{1},T_{5},T_{6}]\); in this case, \(PA=1/3\). Because the latter two log events match incorrectly, partially matched events are considered incorrect in the calculation of the parameter PA. There is no doubt that log parsing accuracy is more stringent than traditional parameters (such as _Recall_, _F-measure_, etc.) [38].  
\[PA=\frac{\#Correct\_Templates}{\#Total\_Templates} \tag{4}\]  
\[RI=\frac{TP+TN}{TP+TN+FP+FN} \tag{5}\]  
Some researchers have selected the Rand Index as the evaluation index [34; 35]. The Rand Index is a common method to evaluate the accuracy of two or more classification tasks. The Rand Index is calculated by Formula (5), which has similar calculation formula to precision and recall, but its specific meaning is different. _TP_ refers to the number of log pairs that are classified into the same category and have the same template; _TN_ refers to the number of log pairs that are divided into different categories and have different templates; _FP_ refers to the number of log pairs that are divided into different categories and have the same template; _FN_ refers to the number of log pairs that are classified into the same category and have different templates [34; 35]. Compared to the recall and other evaluation indicators, the Rand Index can more comprehensively describe the accuracy of the analytical algorithm in the datasets. In addition, there are some studies that use self-defined accuracy indicators. The researchers in [33] considered the limitations of the accuracy of the expression of existing indicators, and they also made requirements for the accuracy of the parsed dynamic tokens. Such indicators are not within the scope of our research.