# Multimodal Sentiment Classifier Framework for Different Scene Contexts
## 4 Multimodal Sentiment Classification Framework
### Image Sentiment Classification  
The Image Sentiment Classification model combines several individual image sentiment classifier models into an ensemble that predicts the final sentiment. In this context (see Figure 4 top), the ISC is made up of four blocks that correspond to each class of sentiment classifier, i.e., the ISC is the result of the ensemble of the results of the ISC for non-man-made outdoors (ISC_ONMM), for man-made outdoors (ISC_OMM), for indoor (ISC_IND), and for indoor/outdoor with persons in the background (ISC_IOwPB). Each sentiment classifier, ISC_class, with _class_\(\in\) {ONMM; OMM; IND; IOwPB}, returns the probabilities of an image carrying a negative, neutral, or positive sentiment. The outputs of the four class sentiment classifier blocks are then ensembled using a random forest (ISC_RF) and a neural network (ISC_NN). Each of these individual blocks, ISC_class, return results for each category (_class_), again doing an ensemble of the three DL models that will be presented below, this is illustrated in Figure 4 bottom. This last step follows the author's previous work carried out for ONMM [3]. Figure 3: Multimodal Sentiment Classification Framework. In summary, ISC_RF or ISC_NN fuses the 36 probabilities given by the four blocks (nine answers for each block resulting from the three probabilities outputs by each of the three individual models) to finally decide the final sentiment of an image (ISC). The number of individual models is a hyperparameter that can be tuned. The abovementioned three models, for each ISC_class_, have DL architectures. The architectures are based on backbones from known architectures followed by a handcrafted network head, based on fully connected layers. In this scenario, three distinct backbones were used to extract different types of features, namely, EfficientNetB0 [34], Xception [36], and ResNet-50 [37]. It is worth noting that all backbones were trained using the well-known ImageNet data set. Different strategies of transfer learning were used for the models, including different numbers of fully connected layers at the networks' heads. Nevertheless, all ISC_class_ blocks have the same three individual models, with the same architecture, and hyperparameter, that were only optimized for the ONMM _class_. What differentiates an individual classifier from the others is the class category of the sub-data set images (Flickr_ONMM, Flickr_OMM, Flickr_IND and Flickr_IOwPB) that were used to train the individual and ensemble models. The deep learning models are as follows:  
**(a) Model DL#1:** the backbone is an EfficientNetB0 (237 layers). Furthermore, let L\({}_{i,j}\) represent a dense layer, where \(i\) is the layer number and \(j\) is the number of units. Then, in ISC_class_DL#1, the head has 5 layers. In the initial layer, L\({}_{in}\), the number of units equals the number of outputs of the backbone. The second dense layer has \(n\) units with X\({}_{2}\) activation function (L\({}_{2,n}\)), a dense layer L\({}_{3,m}\), and a dense layer with 24 units (L\({}_{4,24}\)), all with X\({}_{i}\) activation functions (see Table 7). The last layer has 3 units with a softmax activation function (Ls). **(b) Model DL#2:** the backbone is Xception (71 layers) and the head also has 5 layers. In the ISC_class_DL#2, the first dense layer has L\({}_{in}\) units, equal to the number of outputs of the backbone. Then, it has a dense layer with \(n\) units (L\({}_{2,n}\)). The third layer has also \(m\) units and L\({}_{3,m}\). Then a dense layer with 24 units (L\({}_{4,24}\)). All layers have X\({}_{i}\) activation functions. The last layer is the Ls. **(c) Model DL#3:** the backbone is ResNet-50 (50 Layers) and the head has 4 layers. In the ISC_class_DL#3, the first dense layer is L\({}_{in}\), the second layer L\({}_{2,n}\), the third layer L\({}_{3,24}\), all with X\({}_{i}\) activation functions, and the last layer is the Ls. These architectures were tuned using the sub-data set ONMM.