## 3 Hierarchical Classification
### Log Analysis Using Thematic Hierarchical Codes  
The next phase involved analyzing log messages to identify common themes and eliminate duplicates. The process consisted of searching the words of each message in the catalog, extracting their hierarchical codes, and comparing them between messages. Codes that shared a common prefix up to a certain threshold indicated a similar theme. This threshold was defined as a percentage of matching digits of the shortest code, allowing the determination of thematic specificity and relationship between messages. If we consider \(L\) as the set of all lines in the log file, each line \(l_{i}\) is a vector of words or tokens \(w_{i}=w_{i1}w_{2}\ldots w_{in}\), and \(w_{ij}\) is the \(j\)-th word in the \(i\)-th line, and \(n_{i}\) is the number of words in line \(i\). We have a dictionary \(D\) that associates words \(w\) with codes \(c\) represented as a mapping \(D:W\to C\), where \(W\) is the set of all possible words and \(C\) is the set of all possible codes. The transformation of each line \(l_{i}\) is performed using the substitution function \(f\), which is applied to each word \(w_{ij}\) in \(w_{i}\). The function \(f(w)\) is defined as:  
\[f(w)= \begin{cases}D(w)&\text{if }w\in\text{dom}(D)\\ \emptyset&\text{if }w\
otin\text{dom}(D)\end{cases} \]  
where \(\text{dom}(D)\) represents the domain of definition of the dictionary \(D\), i.e., the set of words or tokens for which \(D\) has an associated code. We apply \(f\) to each word in \(w_{i}\) to obtain a new vector of codes \(c_{i}=f(w_{i1})f(w_{i2})\ldots f(w_{in})\), and we apply \(g\) to each line \(l_{i}\) of the original file, consisting of the vector \(w_{i}\), to obtain a new output file:  
\[g(w_{i})=w_{i}\,c_{i}=v_{i}\]  
The transformed log file is represented by a new set of vectors \(v_{i}\), where each line contains the vector \(w_{i}\) from the original file followed by the vector \(c_{i}\) resulting from applying the transformation to each line \(l_{i}\) of the original file. We can describe the complete transformation process of the log file as follows:  
\[v_{i}=w_{i}1w_{i}2\ldots w_{i}nf(w_{i}1)f(w_{i}2)\ldots f(w_{i}n)\,\forall\, w_{i}\in L\]  
Figure 3: Creation of the world catalog and hierarchical codes. To compensate for differences between the base dictionary and the words used in the logs, we adapted the words in the messages to identify themes rather than exact matches. The adaptations included treating the words separated by dots or that start with uppercase letters as different, even if they are joined. All uppercase letters were transformed to lowercase, and substrings such as URLs, domains, and multimedia file extensions were removed. Digits, control characters, punctuation marks (except the apostrophe that appears in the base dictionary), and functional words stored in a file called 'paldescart' were also discarded. Additionally, accents were removed to align with the base dictionary, which lacks accented words. If the hierarchical codes of a word with \"-ing\" or \"-ed\" endings were not found, they were replaced by \"-ion\", and if still not found, alternatives were tried by removing the last, penultimate, or antepenultimate character. If no matches were found, it was checked if the word was in plural, consulting the 'plurirreg' file for irregular plurals and adjusting the ending as necessary to search for the singular corresponding hierarchical code. Obtaining the Summary Log  
The 'datacod' file contains rows that include both the original log message and the hierarchical codes that reflect its meaning. We implemented an algorithm that processes this file to obtain a much more compact log than the original, representing each template of the original log with a single message. The process is divided into three main modules:  
First, the log entry is stripped of rows with repeated hierarchical codes. Each log row includes at the end the hierarchical codes of the message that encapsulate the themes of that message.