# Multimodal Sentiment Classifier Framework for Different Scene Contexts
## 6 Conclusions  
This work presented research on text and image sentiment classification, especially in social media posts related to \"indoor\", \"man-made outdoors\", \"non-man-made outdoors\", and \"indoor/outdoor with persons\" environments. The study and analysis demonstrated the effectiveness of the proposed class-specific and holistic image classifiers and text classifiers in predicting sentiments, highlighting their potential applications and implications. The preprocessing techniques, the incorporation of deep learning models, and the advanced feature extraction techniques used led the Multimodal Sentiment Classifier framework to obtain a high accuracy in sentiment classification from text-image posts, as well as a very consistent prediction of image and text representing the same sentiment and indeterminate sentiments. Experiments on detecting sentiments in images showed promising results, demonstrating that the system can classify sentiments based on objects, colors, and other aspects present in an image. Additionally, scene-specific image models obtained higher accuracies due to their ability to capture specific details in contexts, while the holistic model offers a lower accuracy but a higher versatility, once it is not needed to use preclassification techniques to classify the images (the last being class segmentation, which is out of the focus of this paper). Finally, the ensemble models allowed the system to leverage the complementary information provided by the textual and visual models, which leads to better performance. This is very significant when a multiple model approach is used, in this case in sentiment analysis tasks. In summary, this study contributed to the understanding of sentiment detection from data present in text and images. Although the accuracy of the system can be improved, the potential of the models has been demonstrated. More significantly, it introduced a framework that has not yet been published in the literature. The framework utilizes separate sentiment models for text and image; these models are only combined at the end if they convey the same sentiment or if there are uncertainties about the sentiment, allowing one to enhance the other. In the cases where the image and language clearly convey different sentiments, they should not be merged. An empirical conclusion is that the user only intended to illustrate the text or vice versa, that is, the text was only used to frame the image. This paper presents an initial approach to the discussion of this problem, that in future works needs to be deepened. Continuing to improve and refine benchmark sentiment classification can open new possibilities to enable more sophisticated and nuanced sentiment analysis in interfaces and/or robots. Looking ahead, there are several directions for future research. The focus should be on improving the sentiment detection model in images and obtaining more and better image sentiment classification data sets. We also intend to train and test models for images of scenes not mentioned during this research, to further increase the effectiveness of the final model that contains the responses from each environment-specific classifier. Conceptualization, N.S., P.J.S.C. and J.M.F.R. ; methodology, N.S. ; software, N.S. ; validation, N.S., P.J.S.C. and J.M.F.R. ; formal analysis, N.S., P.J.S.C. and J.M.F.R. ; investigation, N.S. ; resources, N.S., P.J.S.C. and J.M.F.R. ; data curation, N.S., P.J.S.C. and J.M.F.R. ; writing--original draft preparation, N.S. ; writing--review and editing, N.S., P.J.S.C. and J.M.F.R. ; visualization, N.S., P.J.S.C. and J.M.F.R. ; supervision, P.J.S.C. and J.M.F.R. ; project administration, J.M.F.R. ; funding acquisition, P.J.S.C. and J.M.F.R. All authors have read and agreed to the published version of the manuscript. This research was supported by NOVA LINCS ref. UIDB/04516/2020 ([[https://doi.org/10.5](https://doi.org/10.5)]([https://doi.org/10.5](https://doi.org/10.5)) 4499/UIDB/04516/2020) and ref.