# Multimodal Sentiment Classifier Framework for Different Scene Contexts
## 5 Tests, Results, and Discussion
### Discussion  
One of the difficulties of this research is the nonexistence of a sentiment data set classified by humans that can validate the main research goals. Specifically, these objectives are (1) developing a single hyperparameter image classification sentiment model that performs well across different environments, including validating that the classification accuracy supported on four categories is better than using a single category (including all images); and (2) the development of a framework that combines image and text sentiment classification. The framework must return a multimodal sentiment classification along with the discrepancy metric, which includes the idea that text and image can only be combined if both return the same sentiment or if both return uncertain sentiments. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{1}{c}{\multirow{2}{*}{**Model MSC**}} & \multirow{2}{*}{**Number Units/Estimators**} & \multirow{2}{*}{**Hyperparameters + Activation Function**} & \multicolumn{1}{c}{\multirow{2}{*}{
\begin{tabular}{c} **Accuracy** \\ **(Samples)** \\ \end{tabular} }} \\ \hline \multirow{3}{*}{**RFe**} & _dis._ = 0 & & — & 100.00\% (242/242) \\ \cline{2-5}  & \multirow{3}{*}{0 < _dis._ < 100} & \(k\) = 150 (est.) & & \\  & \(l\) = 4 (samples) & — & — & 64.18\% (224/349) \\  & \(m\) = 5 (samples) & & & \\ \hline \multicolumn{5}{c}{\multirow{5}{*}{**NNe**}} & _dis._ = 0 & & — & 100.00\% (242/242) \\ \cline{2-5}  & \multirow{3}{*}{0 < _dis._ < 100} & \(n\) = 466 & Opt: Adam (7.88 \(\times\) 10\({}^{-4}\)) & X\({}_{i}\) = ReLU \\  & & \(m\) = 24 & Epochs: 20 & \(i\) = [1, \(\ldots\), 3] & 60.17\% (210/349) \\ \cline{3-5}  & & & _MSC Model (using NNe)_ & 76.48\% (452/591) \\ \hline N.A. & _dis._ = 100 & & — & Not applicable \\ \hline \hline \end{tabular}
\end{table} 
Table 11: MSC parameters, hyperparameters, and accuracy. In the latter case, one text/image can complement the other to achieve the final sentiment classification. When both return different sentiments, they should not be joined. That is, when conflicting sentiments occur, empirically, it is possible to say that the person posted the text with a sentiment and used the image only for illustration purposes, or the opposite, they posted the image with sentiment and the text is only to \"frame\" the image. This needs further research, including how these posts can/should be used by managers who receive this information to manage their companies or platforms. Returning to the nonexistence of a data set with ground-truth sentiment for text and images from posts, the solution adopted in this paper to mitigate this problem was to use multiple data sets and sub-data sets. While this is not an ideal solution, it effectively supports the scientific goals of the paper. For the TSC, a single data set was used that filled all the requirements. For the ISC, the initial data set, classified automatically, was divided into five classes. From these five classes, the one depicting in the foreground (semi-)frontal humans was discarded due to the following three main reasons: (1) this class is certainly the most analyzed in the literature, presenting excellent models that validate the sentiment, with most of the cases using the human face; (2) text and image sentiment usually, in this case, are very similar; and (3) in a post, indoor and outdoor scenes many times are used with different purposes, such as transmitting a sentiment, being ironic, for illustration, etc. Usually, images with faces transmit a specific emotion in posts. For the MSC, it was not possible to find a data set that had human-annotated ground-truth classification for the image, the text, and the combined image-text. When developing the authors' data set, it was validated that these three possibilities are very crucial. For example, a person might classify a text with a positive sentiment and an image with a negative sentiment. However, when viewing both the image and the text together, the overall sentiment might be different, such as neutral.