## 4 Experiments and Results
\begin{table}
\begin{tabular}{l l c c c c c c c c} \hline \hline  & & & \multicolumn{3}{c}{**Test Data Set**} & \multicolumn{3}{c}{**Micro Avg**} & \multicolumn{2}{c}{**Macro Avg**} \\
**Model** & **Annotation** & \multicolumn{2}{c}{**FMEA**} & \multicolumn{2}{c}{**Slides**} & \multicolumn{2}{c}{**FMEA and Slides**} & \multicolumn{2}{c}{**FMEA and Slides**} \\  & & **F1\%** & \multicolumn{2}{c}{**F1\%**} & \multicolumn{2}{c}{**F1\%**} & \multicolumn{2}{c}{**F1\%**} & \multicolumn{2}{c}{**F1\%**} \\  & & **SST** & **MST** & **SST** & **MST** & **SST** & **MST** & **SST** & **MST** \\ \hline \multirow{4}{*}{BERT} & Trigger & \(98\pm 1\) & \(98\pm 2\) & \(75\pm 4\) & \(81\pm 2\) & \(86\pm 2\) & \(89\pm 2\) & \(86\pm 2\) & \(90\pm 2\) \\  & Cause & \(88\pm 2\) & \(88\pm 4\) & \(61\pm 3\) & \(62\pm 6\) & \(74\pm 2\) & \(73\pm 5\) & \(74\pm 2\) & \(75\pm 5\) \\  & Effect & \(74\pm 9\) & \(85\pm 6\) & \(61\pm 6\) & \(58\pm 2\) & \(67\pm 7\) & \(69\pm 3\) & \(68\pm 8\) & \(72\pm 4\) \\  & Macro Avg & \(87\pm 4\) & \(90\pm 4\) & \(66\pm 4\) & \(67\pm 3\) & \(76\pm 4\) & \(77\pm 3\) & \(76\pm 4\) & \(79\pm 4\) \\  & Trigger Grouping & - & \(96\pm 3\) & - & \(97\pm 4\) & - & \(96\pm 4\) & - & \(96\pm 4\) \\  & Trigger & \(99\pm 1\) & \(98\pm 1\) & \(81\pm 4\) & \(81\pm 3\) & \(89\pm 2\) & \(89\pm 1\) & \(90\pm 2\) & \(90\pm 2\) \\  & Cause & \(92\pm 5\) & \(87\pm 7\) & \(63\pm 6\) & \(57\pm 6\) & \(76\pm 4\) & \(71\pm 4\) & \(78\pm 6\) & \(72\pm 6\) \\  & Effect & \(90\pm 4\) & \(89\pm 3\) & \(67\pm 6\) & \(57\pm 4\) & \(76\pm 4\) & \(70\pm 2\) & \(78\pm 5\) & \(73\pm 4\) \\  & Macro Avg & \(94\pm 3\) & \(91\pm 4\) & \(70\pm 5\) & \(65\pm 4\) & \(80\pm 3\) & \(77\pm 2\) & \(82\pm 4\) & \(78\pm 4\) \\  & Trigger Grouping & - & \(98\pm 3\) & - & \(100\pm 0\) & - & \(99\pm 1\) & - & \(99\pm 2\) \\  & Trigger & \(99\pm 1\) & \(99\pm 1\) & \(79\pm 3\) & \(82\pm 4\) & \(89\pm 2\) & \(90\pm 2\) & \(89\pm 2\) & \(90\pm 2\) \\  & Cause & \(89\pm 5\) & \(90\pm 3\) & \(65\pm 3\) & \(64\pm 6\) & \(76\pm 3\) & \(75\pm 4\) & \(77\pm 4\) & \(77\pm 4\) \\ BERT PMI & Effect & \(88\pm 7\) & \(91\pm 5\) & \(71\pm 4\) & \(59\pm 2\) & \(78\pm 5\) & \(73\pm 2\) & \(80\pm 6\) & \(75\pm 4\) \\  & Macro Avg & \(92\pm 4\) & \(93\pm 3\) & \(72\pm 3\) & \(68\pm 4\) & \(81\pm 3\) & \(79\pm 3\) & \(82\pm 4\) & \(81\pm 3\) \\  & Trigger Grouping & - & \(98\pm 1\) & - & \(100\pm 1\) & - & \(99\pm 0\) & - & \(99\pm 1\) \\  & Trigger & \(97\pm 2\) & \(98\pm 1\) & \(76\pm 3\) & \(80\pm 4\) & \(86\pm 2\) & \(89\pm 2\) & \(86\pm 2\) & \(89\pm 2\) \\  & Cause & \(87\pm 3\) & \(89\pm 2\) & \(64\pm 17\) & \(66\pm 7\) & \(75\pm 8\) & \(75\pm 3\) & \(76\pm 10\) & \(78\pm 5\) \\ MatBERT & Effect & \(82\pm 8\) & \(93\pm 4\) & \(60\pm 10\) & \(64\pm 5\) & \(71\pm 9\) & \(79\pm 2\) & \(71\pm 9\) & \(78\pm 4\) \\  & Macro Avg & \(89\pm 4\) & \(93\pm 2\) & \(67\pm 10\) & \(70\pm 5\) & \(77\pm 6\) & \(81\pm 2\) & \(78\pm 7\) & \(82\pm 4\) \\  & Trigger Grouping & - & \(95\pm 4\) & - & \(98\pm 3\) & - & \(96\pm 2\) & - & \(96\pm 4\) \\  & Trigger & \(97\pm 2\) & \(97\pm 1\) & \(78\pm 4\) & \(81\pm 4\) & \(87\pm 3\) & \(88\pm 2\) & \(88\pm 3\) & \(89\pm 2\) \\  & Cause & \(88\pm 4\) & \(86\pm 8\) & \(66\pm 11\) & \(69\pm 6\) & \(76\pm 5\) & \(76\pm 3\) & \(77\pm 8\) & \(77\pm 7\) \\ MatBERT UM & Effect & \(84\pm 9\) & \(89\pm 4\) & \(69\pm 6\) & \(63\pm 8\) & \(77\pm 6\) & \(75\pm 3\) & \(76\pm 6\) & \(76\pm 6\) \\  & Macro Avg & \(90\pm 4\) & \(91\pm 4\) & \(71\pm 7\) & \(71\pm 6\) & \(80\pm 5\) & \(80\pm 3\) & \(80\pm 6\) & \(81\pm 5\) \\  & Trigger Grouping & - & \(96\pm 2\) & - & \(98\pm 2\) & - & \(97\pm 2\) & - & \(97\pm 2\) \\  & Trigger & \(96\pm 1\) & \(98\pm 1\) & \(76\pm 5\) & \(82\pm 4\) & \(86\pm 3\) & \(90\pm 2\) & \(86\pm 3\) & \(90\pm 2\) \\  & Cause & \(83\pm 8\) & \(88\pm 4\) & \(67\pm 7\) & \(65\pm 4\) & \(74\pm 1\) & \(75\pm 3\) & \(75\pm 8\) & \(76\pm 4\) \\ MatBERT PMI & Effect & \(86\pm 10\) & \(94\pm 3\) & \(65\pm 10\) & \(72\pm 6\) & \(76\pm 10\) & \(82\pm 4\) & \(76\pm 10\) & \(83\pm 4\) \\  & Macro Avg & \(88\pm 6\) & \(93\pm 3\) & \(69\pm 7\) & \(73\pm 5\) & \(79\pm 5\) & \(82\pm 3\) & \(79\pm 7\) & \(\mathbf{83}\pm 3\) \\  & Trigger Grouping & - & \(93\pm 4\) & - & \(98\pm 2\) & - & \(96\pm 2\) & - & \(96\pm 3\) \\ \hline \hline \end{tabularThe results indicate that the MST method, which leverages MatBERT and in-domain pre-training using PMI, achieves an average \(F1\_score\) of 83 \(\pm\) 3% (macro average) across various document types (i.e., FMEA and slides) and annotation types (i.e., trigger, cause, and effect).