## 4 Experiments and Results
The analysis of the annotated data sets reveals the distribution of tokens across different classes. Notably, a higher percentage of tokens is associated with causal relations in FMEA documents compared to presentation slides. Furthermore, in both document types, the cause and effect categories comprise a larger number of tokens than the trigger category. Additionally, a closer examination shows that causes tend to consist of a larger number of tokens than effects. The annotated data set is split into two primary components: a testing set and a training set. The training set is further subdivided into training and validation sets using a five-fold cross-validation approach. Specifically, 10% of the data are used for testing, 10% of the data for validation, and the remaining 80% of the data are used for training. This division enables the model to be trained on the training set, while the validation set is utilized for early stopping, thereby preventing overfitting. The hyperparameters employed for training both the SST and MST methods are as follows: a learning rate of \(2.5\times 10^{-5}\), a maximum length of 512 tokens, a batch size of 16, and an early stop patience of 5 epochs. Both methods were undertaken using on-premises GPUs. Notably, both SST and MST methods are trained using the same training and validation folds and subsequently tested using the same testing set. This ensures a fair comparison between the two methods. The performance of the trained models using different language models is summarized in Table 2. \begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Annotation**} & \multicolumn{4}{c}{**Data Set**} & \multicolumn{4}{c}{**Micro Avg**} & \multicolumn{4}{c}{**Macro Avg**} \\  & \multicolumn{2}{c}{**FMEA**} & \multicolumn{4}{c}{**Slides**} & \multicolumn{4}{c}{**FMEA and Slides**} & \multicolumn{4}{c}{**FMEA and Slides**} \\  & \(\kappa\)\% & **F1\%** & **Oc**\% & \(\kappa\)\% & **F1\%** & **Oc**\% & \(\kappa\)\% & **F1\%** & **Oc**\% & \(\kappa\)\% & **F1\%** & **Oc**\% \\ \hline Trigger & 94 & 94 & 8 & 93 & 93 & 3 & 94 & 94 & 5 & 94 & 94 & 6 \\ Cause & 87 & 90 & 23 & 84 & 85 & 9 & 87 & 88 & 14 & 86 & 88 & 16 \\ Effect & 86 & 88 & 16 & 67 & 68 & 6 & 81 & 83 & 10 & 76 & 78 & 11 \\ Macro Avg & 89 & 91 & 16 & 81 & 82 & 6 & 87 & 88 & 10 & 85 & 86 & 11 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: **Inter-annotator agreement (IAA) results for different data sets. ** Two metrics were used to calculate the IAA, the _F1_score_ (_F1%_) and _Cohen_′s_kappa_ (_x_%). The percentage occurrences (_Oc_) of tokens associated with the different classes in the annotated data sets are provided as well. The IAA is higher for the data extracted from semi-structured documents, namely FMEAs, compared to the data extracted from presentation slides. Additionally, the IAA is higher for annotations of the ‘trigger’ type compared to annotations of the ‘cause’ and ‘effect’ types. These results suggest that the extraction of causal information from FMEA documents is less ambiguous and more consistent compared to the extraction from presentation slides. This finding is attributed to the fact that the text in FMEA documents is typically simpler and more concise. Additionally, results suggest that the ‘trigger’ type annotations are easier to identify and agree upon by annotators. We explain this findings with the lower number of tokens used as triggers compared to the cause and effects as well as the limited lexical options for the trigger. The table provides a comprehensive evaluation of the performance of each method, including the \(F1\) score of the multi-label classifier for the SST method. Additionally, the table shows the performance of the MST binary classifier for trigger detection, presented in the trigger \(F1\) score. It also includes the performance of the trigger grouping model and the multi-label classifier for detecting causes and effects in the MST method.