## 7 Experiments and Results
### Task 3: Observation Categorization  
The third task we include involved the categorization of observations. This is split into two subtasks, i.e., \"Issue Source\" and \"Categories\".The possible application of this task is the automatic organization of past observations into meaningful classes, for analysis by specialized experts and the devising of meaningful feedback for the involved businesses. Regarding the \"Issue source\" subtask, the goal of this task is to predict the actor or situation that caused a particular safety issue. For this task, observations were labeled under the \"Activities\", \"Spaces\", \"Equipment\", \"Regulations\" and \"Other\" categories. For example, if an observation is classified under the \"Activities\" label, the cause of the issue is the actions of people during the observation time (e.g., employees not wearing protective gear); whereas, if it is classified under the \"Spaces\" category, the cause of the issue is related to the space where it was observed (e.g., missing fire extinguishers). For the \"Category\" subtask, we employed a semantic labeling of the observations under high-level categories, e.g., \"Electrical safety\", \"Fire safety\", etc. The goal was to organize observations into semantic clusters to make the post-processing easier. For example, one observation may concern compliance to safety regulations and should be reviewed by an SO that specializes in the law and regulations of the country or city, while another observation may regard electrical safety and should be processed by an SO with an electrical engineering background.  
In Table 10, we can see the results for the issue categorization tasks. We can see that, for these tasks, in contrast to the observation assessment and priority classification, the dominant modality is the textual description of the issue, while the visual modality does not convey much information. This is evident, as most of the multimodal configurations fail to surpass their unimodal counterparts. Furthermore, we can observe that the only configurations that manage to gain a positive \"\(\Delta\uparrow\) unimodal\" involve the use of the CLSS fusion approach, while the feature concatenation approach fails to extract good multimodal representations.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Modality** & **Text Encoder** & **Visual Encoder** & **Fusion Method** & **Accuracy (\%)** & \(\Delta\uparrow\) **Unimodal** \\ \hline \(\mathcal{V}\) & - & ResNet18 & - & 63.37 & - \\ \(\mathcal{V}\) & - & ViT & - & 68.2 & - \\ \(\mathcal{T}\) & Greek BERT & - & - & 69.15 & - \\ \(\mathcal{T}\) & GR-Electra & - & - & 65.94 & - \\ \hline \(\mathcal{T}+\mathcal{V}\) & Greek BERT & ResNet18 & cat & 64.48 & \(-\)4.67 \\ \(\mathcal{T}+\mathcal{V}\) & Greek BERT & ResNet18 & C1 SS & **74.73** & 5.58 \\ \(\mathcal{T}+\mathcal{V}\) & Greek BERT & ViT & cat & 68.8 & \(-\)0.35 \\ \(\mathcal{T}+\mathcal{V}\) & Greek BERT & ViT & C1 SS & **72.9** & 3.75 \\ \(\mathcal{T}+\mathcal{V}\) & GR-Electra & ResNet18 & cat & 64.14 & \(-\)1.79 \\ \(\mathcal{T}+\mathcal{V}\) & GR-Electra & ResNet18 & C1 SS & **72.23** & 6.29 \\ \(\mathcal{T}+\mathcal{V}\) & GR-Electra & ViT & cat & 73.24 & 5.04 \\ \(\mathcal{T}+\mathcal{V}\) & GR-Electra & ViT & C1 SS & **73.48** & 5.28 \\ \hline \multicolumn{5}{c}{Random Baseline} & 46.34 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Classification accuracy for the effective SO experience level of observations on the Safety4 All test set. \(\mathcal{T}\): Text, \(\mathcal{V}\): Visual.