# Multimodal Sentiment Classifier Framework for Different Scene Contexts
## 5 Tests, Results, and Discussion
### Multimodal Sentiment Classification  
The MSC was trained in the [PERSON]'s platform (with 14.8 GB RAM e GPU T4 x2) using B-T4 SAmultimodal and a stratified K-fold cross-validation technique, focusing only on samples where the image and the text share the same sentiment (273 samples). Before entering more details about MSC results, characterizing the samples per discrepancy is important. From the B-T4 SAmultimodal data set (classified by humans), the MSC framework using the specification stated in Section 4.3 separates samples into the following three categories: those where the image and text represent the same sentiment (38.60%), those where there is uncertainty about their alignment (55.66%), and those where the sentiments are clearly different (see Table 10). This means that 38.60% (242 from 627 samples) plus 5.74% (36 from 627 samples) are already classified (no additional processing is required). The former because _isc_ equals _tsc_, and consequently _msc_ equals _tsc_ returning SCv = [_tsc_, _tsc_, _tsc_, 0]; the latter because there is no possible classification for _msc_, once the model is completely sure of the sentiment for ISC and for TSC, and they are different, meaning SCv = [isc, tsc, \(\times\), 100]. The framework only needs to do the inference for the samples that have 0 < _dis_. < 100 (marked in bold), in the present data set 55.66% (i.e., 349 from the 627 samples). Table 11 shows the parameters, hyperparameters, and accuracies per discrepancy of the MSC model. In more details, if _dis_. = 0 (ISC and TSC report the same sentiment), then the framework reports a 100% accuracy, i.e., these samples need no further computation, once \begin{table}
\begin{tabular}{c c c} \hline \hline
**Discrepancy** & **Number of Samples** & **Percentage of Samples** \\ \hline _dis._ = 0 & 242 & 38.60\% \\ \hline
**0 \textless{} _dis._ < 100** & **349** & **55.66\%** \\ \hline _dis._ = 100 & 36 & 5.74\% \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Number of samples per sentiment discrepancy. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Number Units/Estimators** & **Hyperparameters + Activation Function** & **Accuracy** \\ \hline TSC\#1 (RF) & \(k\) = 100 (est.) & — & — & 90.10\% \\ \hline \multirow{3}{*}{TSC\#2 (NN)} & \(n\) = 300 & \
\begin{tabular}{c} Opt: SGD (\(1\times 10^{-2}\)) \\ Epochs: 10 \\ Batch size: 8 \\ \end{tabular} & \(\begin{array}{c}\chi_{i}=\text{ReLU}\\ i=\left[1,\ldots,3\right]\end{array}\) & 88.55\% \\ \hline TSC\#3 (NLTK) & — & — & — & 84.34\% \\ \hline TSC–RFt & \(k\) = 100 (est.) & — & — & 91.95\% \\ \hline \multirow{3}{*}{**TSC–NNt**} & \(n\) = 100 &
\begin{tabular}{c} Opt: SGD (\(1\times 10^{-2}\)) \\ Epochs: 10 \\ Batch size: 2 \\ \end{tabular} & \(\begin{array}{c}\chi_{i}=\text{ReLU}\\ i=\left[1,\ldots,3\right]\end{array}\) & **92.10\%** \\ \hline \hline \end{tabular}
\end{table} 
Table 9: TSC parameters, hyperparameters, and accuracy of the models. _isc_ equals _tsc_, so it just checks the sentiments against the ground truth (no inference was carried out). For 0 < _dis._ < 100, the best result was achieved for the ensemble using random forest, with an accuracy of 64.18% (using NN, the result is similar with a difference ~4%). At this point, it is again important to stress how the accuracy of MSC is computed, because it aggregates results from the ISC model and TSC model, and checks these predictions against the ground truth, once the image and text reflect or could reflect different sentiments.