## 3 CFTL Log Parsing Method
### Algorithmic Details of CFTL
Log messages with the same token are clustered into the same group. After the first token is counted by CFTL, the token at the second position is processed until the token at the last position is processed. In addition, to improve the processingspeed, CFTL first saves the common subsequence within the group, because the words in the common subsequence are constants and do not need to be rechecked. ```
0: a group of clustered log messages: logGroup
0: several separated groups
1: function SEPARATING (logGroup, st, pos, groups)
2: //st means separation threshold
3: //pos records separation position
4: //groups is the result
5: \(sDict\leftarrow\) Empty dictionary
6: logLen \(\leftarrow\) the length of log messages in logGroup
7: if pos \(\geq\) logLen then
8: groups \(\leftarrow\) add logGroup
9: return
10: end if
11: for each log message logMsg in logGroup do
12: \(tk\leftarrow\) the pos\({}^{\text{th}}\) token in logMsg
13: ifSDict doesn't have the same key as \(tk\) then
14: sDict[\(tk\)] \(\gets\) 1
15: else
16: sDict[\(tk\)] \(\leftarrow\) sDict[\(tk\)] + 1
17: end if
18: end for
19: sDlen \(\leftarrow\) the number of keys in sDict
20: pos \(\leftarrow\) pos + 1
21: if sDlen < st then
22: Separate log messages into different groups according to different keys in sDict
23: Calling function SEPARATING with separated groups
24: else
25: SEPARATING (logGroup, st, pos, groups)
26: end if
27: end function
```  
**Algorithm 1**Separating  
The main goal of the proposed CFTL method is to solve the problem of operation system log parsing in distributed systems, such as Linux, Windows, and Android. The training process of the log parser, including the separation mechanism, is performed on the server side, and the trained log template is then deployed on the terminal device to compensate for limited terminal equipment resources. In addition, there are other reasons for this design. During the communication between the end devices and servers, the original token is generated on the server side, which allows it to provide richer and more original first token information. The abnormal behaviors of the end devices are primarily reflected in the interaction with the servers and, thus, in the system logs of the servers because most intrusion behaviors targeting the end of distributed systems are aimed at the ultimate unauthorized access to the servers. The end device is only a springboard. Highly subtle and multistep abnormal behaviors may not be reflected in the end device system logs. End devices of the same type may have different exception logs. Thus, the log templates of different terminals may differ. Abnormal behaviors known to other end devices of the same type but unknown to the local end cannot be identified. All end devices interact with servers, and the servers are the convergence of all abnormal end device behaviors; thus, parsing on the server can extract richer abnormal information. Simplifying these rules without compromising accuracy can be challenging given the limited resources on the edge server side. In addition, overfitting can occur if the rules are too complex. CFTL simplifies the separation rules by considering constant token differences and using a simple threshold design, which is a modest way to establish rules. If constant token difference is not considered, logs with similar initial tokens or lengths but completely different meanings may be mistaken for the same type of logs, resulting in insufficient precision. However, if an excessive number of rules are considered, such as the mapping relationship between token pairs and the information before and after tokens, the rules will be too complicated. The simplification of rules and the accuracy of the algorithm are typically contradictory; there are no perfect simplification schemes, and we can only balance the accuracy and usability of the algorithm as much as possible. **Extracting template. ** In this step, we use LCS to extract the templates of each log message group; that is, the LCS of each group represents the template of the respective group.