# Log-Based Fault Localization with Unsupervised Log Segmentation
## 2 Materials and Methods
### Unsupervised Log Sequence Segmentation  
Log sequence, as a part of the execution path, conveys some information on the software system and its execution. It contains events that are visible to humans and allows to distinguish patterns and segments of lines as they reflect the architecture of the system. Following this observation, a few approaches have been proposed to extract this structure to ease manual examination of the log file (for example [23; 27]). However, the application of these methods in automated log analysis has not yet been demonstrated. One of our contributions is that they can be used not only to improve the manual inspection of logs. We used VotingExperts [23] to extract meaningful segments from long log sequences and calculate the rankings of localized suspicious lines. Let \(S\) represent the set of all sequences of log templates  
\[\mathcal{S}=\{\varrho^{0},\ldots,\varrho^{m}\}, \tag{2}\]  
\begin{table}
\begin{tabular}{c l l l} \hline \hline \# & **Log Content** & **Template ID** & **Pass(l)** \\ \hline
1 & INF, State change notif received [CONNECTED] & 1 & 7 \\
2 & INF, State change notif received [CONNECTED] & 1 & - \\
3 & INF, State change notif received [CONNECTED] & 1 & - \\
4 & INF, State change notif received [CONNECTED] & 1 & - \\
5 & INF, State change notif received [CONNECTED] & 1 & - \\
6 & INF, State change notif received [CONNECTED] & 1 & - \\
7 & INF, State change notif received [CONNECTED] & 1 & - \\
8 & INF, SW Version: xxxx & 2 & 1 \\
9 & INF, currentImageType = package.01 & 3 & 1 \\ \hline Length & 9 & 3 & \\ \hline \hline \end{tabular}
\end{table}
Table 1: Normal log example.  
\begin{table}
\begin{tabular}{c l l l l} \hline \hline \# & **Fail Log** & **Template ID** & **Fail(l)** \\ \hline
1 & WRN, BlackBoxLoggingService MemoryItem:update & 4 & 1 \\
2 & WRN, BlackBoxLoggingService get entry value & 5 & 2 \\
3 & WRN, BlackBoxLoggingService get entry value & 5 & - \\
4 & INF, State change notif received [FAILED] & 1 & 4 \\
5 & INF, State change notif received [CONNECTED] & 1 & - \\
6 & INF, Recovery action requested & 6 & 1 \\
7 & INF, State change notif received [CONNECTED] & 1 & - \\
8 & INF, State change notif received [CONNECTED] & 1 & - \\
9 & INF, Received reset request because of node2 failure. & 7 & 1 \\
10 & INF, SW Version: xxxx & 2 & 1 \\
11 & INF, currentImageType = package.01 & 3 & 1 \\ \hline Length & 11 & 7 & \\ \hline \hline \end{tabular}
\end{table}
Table 2: Failed log example.  
\begin{table}
\begin{tabular}{c l l l l l} \hline \hline \# & **Template ID** & **Fail(l)** & **Pass(l)** & **Susp** & **Rank** \\ \hline
1 & 1 & 4 & 7 & 0.36 & 3 \\
3 & 3 & 1 & 1 & 0.5 & 2 \\
4 & 4 & 1 & 0 & 1.0 & 1 \\
5 & 5 & 2 & 0 & 1.0 & 1 \\
6 & 6 & 1 & 0 & 1.0 & 1 \\
7 & 7 & 1 & 0 & 1.0 & 1 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Not normalized Log-based Fault Localization.  
where \(\hat{e}^{i}\) is the \(i\)-th sequence and \(m+1\) is the total number of sequences. The sequence is obtained by extracting all log lines belonging to one thread, block ID, or node ID. A single sequence \(\hat{e}^{i}\) from \(\mathcal{S}\) contains a sequence of log templates \(l_{i}\):  
\[\hat{e}^{i}=<l_{0},...,l_{n}>. \tag{3}\]  
Each \(l_{i}\) belongs to a set of log templates \(\mathcal{L}\) contained in the log file. Segment \(Segm_{k}\) is the sequence of \(n_{k}\) log templates from \(\hat{e}^{i}\).  
\[Segm_{k}=\Big{\langle}l_{i_{k}},\dots,l_{i_{k+1}}\Big{\rangle}, \tag{4}\]  
where \(i_{k}\geq 0\) and \(i_{k}<n\). For the sake of simplicity, for further details, please refer to the original work [23] or our previous work [24], where we showed how unsupervised word segmentation methods can be transferred to the log sequence segmentation domain. For our experiments, we used VotingExperts with a window size of seven and threshold of four.