# Multimodal Sentiment Classifier Framework for Different Scene Contexts
## 1 Introduction  
The goal of human-centered artificial intelligence (HCAI) is to create technologies that assist people in carrying out various daily tasks, while also advancing human values, such as rights, fairness, and dignity [1]. As an interdisciplinary area involving computer science, psychology, and neuroscience, HCAI aims to achieve a balance between human control and (complete) automation by improving people's autonomy, well-being, and influence over future technologies. Affective computing (AffC) is a related field that combines the fields of sentiment analysis and emotion recognition. AffC is backed by a variety of physical information types, including text, audio (speech), visual data (such as body posture, facial expression, or environment), and physiological signals (such as electroencephalography or electrocardiograms). AffC may be developed using either unimodal or multimodal data within this framework [2]. HCAI and AffC have a wide range of applications. For example, in late sensu, a machine needs to be designed to cooperate with or learn how to function in interpersonal relationships with people. Emotions and sentiments are fundamental to human-machine relationships, and any robot communicating with humans must incorporate them. Conversely, social media platforms are becoming increasingly significant in today's digital marketing landscape, because they influence individuals to travel, look for and purchase goods, alter their lives, and alter their perspectives on many topics. The sheer numberof daily posts on various platforms has made it necessary and simultaneously possible to monitor, evaluate, and comprehend the mood, sentiments, and emotions that such messages convey. Considerable progress has been achieved in the area of sentiment analysis [3; 4; 5; 6], including, but not limited to, text, image, and text-image posts (even in cases when the image and words together may convey a different meaning), music, and video analysis, or even robots that can read facial emotions and improve human-robot relations. The quickest and easiest kind of publications to get people to click, buy, and read about a specific topic or product include short texts, images, and/or videos. This explains why social media platforms like Instagram and X (formerly known as Twitter) have gained much traction in recent years. There are two main categories of posts where the pair text-image is connected as follows:  
(1) _Where the text is (clearly) complemented by an image(s) of a person or group of persons._ In those cases, the environment (scene) is the background and the person(s) are in the foreground, i.e., they are looking in a (semi-)frontal manner at the camera, so their facial expressions or body posture have a powerful influence on the sentiment carried [7]. (2) _Where the text may or may not be complemented by the image(s)._ In those cases, there are no persons in the scene, or if existing, they are the \"background\", and the scene is the \"foreground\". In those cases, color, texture, edges, line type, orientation, etc., are important features in the attraction and sentiment that the image carries [8]. This paper focuses on the second case, i.e., detecting sentiments in posts relating to pair text-image(s) without persons, or existing persons who are only in the \"background\". In this case, in the literature, there is currently no good sentiment classifier for this type of image and text combination (see next section). To the best of our knowledge, sentiment classifiers that deal with this type of image use a holistic approach, i.e., the models are trained with all types of images, without considering their specific characteristics. Here, the images are segmented into four categories as follows: \"non-man-made outdoors\" (ONMM), \"man-made outdoors\" (OMM), \"indoor\" (IND), and \"indoor/outdoor with persons only in the background\" (IOwPB). Each class has partial results that are combined, resulting in a final model, which replaces the holistic approach.