# Multimodal Sentiment Classifier Framework for Different Scene Contexts
## 3 Data Set
### Image Sentiment Data Sets  
In the image sentiment study, we used three image data sets, one being further divided into sub-data sets. In this context, from the (i) **Flickr data set** (Flickr data set available at: [[https://www.ee.columbia.edu/ln/dvmm/vso/download/flickr_dataset.html](https://www.ee.columbia.edu/ln/dvmm/vso/download/flickr_dataset.html)]([https://www.ee.columbia.edu/ln/dvmm/vso/download/flickr_dataset.html](https://www.ee.columbia.edu/ln/dvmm/vso/download/flickr_dataset.html)), accessed on 1 August 2024) [34], four sub-data sets are extracted. The reason is that the Flickr data set includes outdoor, indoor, man-made scenes, scenes with persons, etc., all mixed together, and it is intended to be used to train four distinct ISC models (see Section 4.1). To do this subdivision, a scene detection algorithm (code available at: [[https://github.com/AMANVerma28/Indoor-Outdoor-scene-classification](https://github.com/AMANVerma28/Indoor-Outdoor-scene-classification)]([https://github.com/AMANVerma28/Indoor-Outdoor-scene-classification](https://github.com/AMANVerma28/Indoor-Outdoor-scene-classification)), accessed on 1 August 2024) was used, which classifies whether an image is outdoor or indoor, and some other attributes, such as man-made, natural light, no horizon, enclosed area, etc. This algorithm, out of the focus of this paper, was used to speed up the segmentation of the initial data set into the four sub-data set. The resulting subsets of images were then visually validated by the authors. These sub-data sets were used to train and test the ISC models (see Section 4.1). Furthermore, although in Flickr's original data set, the sentiment polarity of the images ranges from \(-2\) to \(2\), the present models have only three sentiment classes: _positive_, _neutral_, and _negative_. This was achieved by defining a negative sentiment as between \(-2\) and \(-1\), a neutral sentiment as between \(-1\) and \(1\), and a positive sentiment as between \(1\) and \(2\). In more detail, the Flickr data set was created from the computed Columbia Multilingual Visual Sentiment Ontology (MVSO) framework. This database covers a visual sentiment ontology consisting of 3244 adjective-noun pairs and SentiBank, which is a set of 1200 trained visual concept detectors, providing a midlevel representation of sentiment, associated training images acquired from Flickr, and a benchmark containing 603 photo tweets covering a diverse set of 21 topics. With \(\sim\)461 k images, the Flickr data set is not balanced, so, \(\sim\)179 k balanced images were randomly selected from those 461 k images (see Table 2, column \"balanced\"). From those, the images were subdivided into four sub-data sets (as mentioned above), one for each of the four environment categories as follows: (i.1) **Flickr_ONMM**; (i.2) **Flickr_OMM**; (i.3) **Flickr_IND**; and (i.4) **Flickr_IOwPB**. Since those sub-data sets were not balanced, for training and testing, when needed, they were balanced using data augmentation, namely, small rotations (5 degrees left/right), horizontal flips and zooms in (10% to 30%) on the images. Table 2 shows the number of images of the data set as well as the sub-data sets and Figure 1 shows examples of images existing in the data set. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Sentiment**} & \multicolumn{3}{c}{_Flickr_} & \multicolumn{3}{c}{_Sub-Data Sets (Flickr)_} \\ \cline{2-7}  & **Original** & **Balanced** & **ONMM** & **OMM** & **IND** & **IOwPB** \\ \hline Positive (+) & 280,157 & **59,794** & 32,878 & 28,653 & 12,664 & 57,639 \\ \hline Negative (\(-\)) & 121,377 & **59,794** & 32,878 & 28,653 & 12,664 & 57,639 \\ \hline Neutral (=) & 59,794 & **59,794** & 32,878 & 28,653 & 12,664 & 57,639 \\ \hline _Total_ & 461,328 & **179,382** & 98,634 & 85,959 & 37,992 & 172,917 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Summary of the Flickr sub-data sets for development of the image sentiment classifiers.