Combining image databases for affective image classification

Affective image classification has attracted much attention in recent years. However, the production of more exact classifiers depends on the quality of the sample database. In this study, we analyzed various existing databases used for affective image classification and we tried to improve the quality of the learning data by combining existing databases in several different ways. We found that existing image databases cannot cover the overall range of the arousal-valence plane. Thus, to obtain a wider distribution of emotion labels from images, we conducted a crowd-sourcing-based user study with Amazon Mechanical Turk. We aimed to construct several different versions of affective image classifiers by using different combinations of existing databases, instead of using one. We used low-level features in our classification experiments to explore the discriminatory properties of emotion categories. We report the results of intermediate comparisons using different combinations of databases to evaluate the performance of this approach. Keywords–image emotion; emotion-based classification I. EMOTION-BASED CLASSIFICATION A. Image collections Recently, many researchers have reported studies of emotion extraction from images. Several key issues influence the affective classification of images. In particular, it is necessary to obtain ground-truth emotion labels for images. However, obtaining high quality emotion-based images is not easy because of human subjectivity and there are no standard models of emotions. In general, researchers have conducted largescale user studies to obtain emotion information with two types of emotion models: categorical and continuous models. Categorical models give a discrete value to an emotion using a word, such as happy, sad, or gloomy. By contrast, continuous models represent specific emotions as coordinates in a multidimensional space (a two-dimensional plane is usually preferred, which is called the arousal-valence plane) and we used this type of model in our experiments. International Affective Picture System (IAPS) is a database of pictures that are used to elicit a range of emotions, which Lang et al. [1] employed in experimental studies of affective image classification. Mikels et al. [2] introduced a subset of the IAPS database for the categorization of images, which we used in our research to obtain the arousal and valence values of the pictures. Geneva Affective PicturE Database (GAPED) contains 730 images with emotional values [3]. GAPED has four specific types of negative contents, including spiders, snakes, and negative scenes. The positive pictures mainly comprise images of human and animal babies, and nature scenes. The pictures are rated according to their arousal, valence, and congruence values. The Nencki Affective Picture System (NAPS) [4], is another affective image database, which comprises 1,356 realistic, high-quality photographs with five subject categories (people, faces, animals, objects, and landscapes). The images were given affective arousal and valence ratings by 204 participants, who were mostly European. Obtaining emotion information using crowd-sourcing Machajdik et al. [5] obtained emotion information based on categorical labels. Furthermore, the range of arousal-valence values is highly limited in other databases, as shown in Figure 1(a). Therefore, we collected arousal and valence values for the images in Machajdik et al.’s database based on a largescale user survey. A total of 199 subjects were recruited to participate in the survey using Amazon Mechanical Turk and the subjects provided 6787 responses. We collected at least six responses for each image and each subject provided an average of 33 responses. Figure 1(b) shows the distribution of the emotion labels obtained in the survey, which demonstrates that the combined database was more evenly distributed in the arousal-valence plane compared with the original database.