# Multimodal Sentiment Classifier Framework for Different Scene Contexts
## 4 Multimodal Sentiment Classification Framework
### Multimodal Sentiment Classification  
The Multimodal Sentiment Classifier block, as largely mentioned in the text, is based on classifications resulting from text and image. Going to the block diagram in Figure 6, from the ISC model there are 36 output probabilities (4 _classes_\(\times\) 3 individual models \(\times\) 3 sentiments), from which the 9 output probabilities (3 individual models \(\times\) 3 sentiments) are used from the individual models corresponding to the _class_ of the image that is being analyzed. From the TSC, the 9 output probabilities (3 individual models \(\times\) 3 sentiments) are also used. Also, _isc_ and _tsc_ sentiment classification are used to compute the _discrepancy_, which acts as a selector to compute the _msc_ based on the ISC and TSC, as we will see next.  
Figure 5: Block diagram of the Text Sentiment Classifier.  
Figure 6: Block diagram of the Multimodal Sentiment Classifier.  
The _discrepancy_ (_dis._) varies between 0 and 100, with 0 corresponding to the sentiment between the image and text being the same, and 100 for the sentiment between the image and text being different, and it is computed as follows:  
1. _Similar sentiments_, i.e., \(isc=tsc\), then \(msc=isc=tsc\), and SCv = [\(tsc\), \(tsc\), \(tsc\), \(dis.=0\)] is the resulting output.
2. _Hypothetic different sentiments_, i.e., \(isc\
eq tsc\), then considering \(\mu_{I}\) the average results (between 0-100) from all ISC models that returned the predicted sentiment, \(\mu_{T}\) the average of all TSC models that returned the predicted sentiment, and a threshold \(\mu_{t}=85\) (empirically calculated), two cases can occur: 1. _The image and text have clearly different sentiments:_ If \(\mu_{I}\geq\mu_{t}\) and \(\mu_{T}\geq\mu_{t}\) then both ISC and TSC are considered to be certain of the predicted sentiment and, in this case, most probably the person who posted the text-image \"intended\" different sentiments. The ensemble block is not computed, resulting SCv = [\(isc\), \(tsc\), \(\times\), \(dis.=100\)].
2. _The image and text have indeterminate sentiments_: The remaining cases. It means that the ensemble between text sentiments and image sentiments must be computed, once there is no certainty about what the person intended to post. Resulting in SCv = [\(isc\), \(tsc\), \(msc\), \(dis.\)], where the \(dis.=100-(\mu_{I}-\mu_{T})/2\) if \(tsc<50\) or \(isc<50\), or \(dis.=(\mu_{I}-\mu_{T})/2\) for the remaining situations.  
Because there is no balance in the sentiment classes data set and the number of samples is relatively small, once the B-T4 Smultimodal (sub-)data set is used, the Stratified K-Fold cross-validation technique is used to train the MSC model (K = 5). The ensemble block is computed following the same principles presented before, namely:  
1. _Random Forest (MSC_RFe):_ \(k\) estimators, \(l\) minimum samples in a leaf, \(m\) minimum samples required to split the internal node, and Gini as criteria function. The rest of the hyperparameters were set to the default values of the scikit-learn library (v. 1.5);
2. _Neural Network (MSC_NNe):_ four dense layers, where the first layer has 18 units (L\({}_{in}\)), a dense layer with \(n\) units (L\({}_{2,n}\)), a third layer with \(m\) units (L\({}_{3,m}\)), with \(X_{i}\) activation function, and the fourth layer has the Ls with softmax as the activation function.  
The following section details the tests conducted and their results, followed by a discussion of the work undertaken.