Evaluation and Prediction of Evoked Emotions Induced by Image Manipulations

Various image editing tools make our pictures more attractive, and at the same time, evoke different emotional responses. With powerful and easy-to-use imaging applications, capturing, editing and then sharing pictures have become daily life for many. This paper investigates the influence of several image manipulations on evoked emotions for different types of images. To do so, various types of images clustered in different categories, were collected from Instagram and subjective evaluations were conducted via crowdsourcing to gather the emotional responses on different manipulations as perceived by subjects. Evaluation results show that certain image manipulations can induce different evoked emotions on transformed pictures when compared to the original ones. However, such changes in image emotions due to manipulation are highly content dependent. Then, we conducted a machine learning based experiment, in attempt to predict the emotions of a manipulated image given its original version and the desired manipulation method. Experimental results present a promising performance of such a prediction model, which could pave the road to automatic selection or recommendation of image editing tools that can efficiently transform or emphasize desired emotions in pictures.  Thanks to wide spread popularity of smart mobile devices with high-resolution cameras, as well as user-friendly imaging and social networking applications, taking pictures, then editing and sharing, have become part of everyday life for many. Photo sharing has been used as a way to share not only stories but also current moods with friends, family and public at large. Modern photo sharing applications equipped with advanced and easy-touse image editing tools, such as Instagram, provide consumers with very convenient solutions to make their pictures more attractive, and more importantly, to arouse stronger emotional resonances. Different types of image content generate different emotions. Using different photographic techniques, visual filters or editing tools, pictures of the same scene can also evoke different emotions. Motivated by these facts, we attempt to change an original picture’s evoked emotion and transform it to new emotions (stronger, weaker, or completely different) by image manipulation. To achieve this goal, we first need to understand the emotional responses evoked by different image manipulations when applied to pictures. This paper investigates the influence of image manipulations on evoked emotions, and tries to find the potential pattern between image manipulation and generated emotions. To do so, we conducted subjective experiments based on online crowdsourcing. Different types of images were collected from Instagram, and manipulated by a number of typical image editing tools. Crowdsourcing subjects were then exposed to each, and questioned regarding the emotions pictures induced on them. Using the crowdsourced data as groundtruth, we trained and evaluated a model based on machine learning for predicting evoked emotions, taking an original image and desired manipulation as input. The rest of the paper is structured as follows. The next section introduces the related works by other researchers, followed by a section describing the data collection and user study. Then we analyze and interpret emotional responses obtained from subjects, and report the experiments of emotion prediction upon image manipulation in the followed two sections. Finally, the last section concludes the paper and discusses future work. Prior Work Image aesthetic quality estimation, emotion recognition and classification have been largely studied in the field of computer vision [1, 2, 3, 4, 5]. Most previous works use image features for affective image classification and emotion prediction [2, 3, 6, 7, 5]. Such features include color, texture, composition, edge and semantic information. A few researchers have worked on transforming image emotions by editing images. In , Wang et al. associate color themes with emotion keywords depending on art theory and transform the color theme of an input image to the desired one. However, in their work, only a few cartoon-like images are used. Peng et al.  propose a framework to change an image’s emotion by randomly sampling from a set of possible target images, but only show a few examples. Jun et al.  show that changing brightness and contrast of an image can affect the pleasure and excitement felt by observers. However, only a limited variation of an input image can be produced by changing the two features. Peng et al.  change the color tone and texture related features of an image to transfer the evoked emotion distribution, with experiments conducted on only limited types of image content. Evaluating image’s evoked emotions after image manipulation is not a trivial task. Many well-established image manipulation and editing tools have been widely used in online photo sharing and social networks, as ways for users to enhance their image content either to draw better attention or to evoke stronger emotions. Popular image editing tools include image enhancement , grayscale conversion, vintage processing, cartoonizing [13], and more recently addition of stickers1 . However, most image manipulation methods have been studied merely from the perspective of image processing and not so much on their emotional impact. 1https://www.facebook.com/help/1597631423793468 (a) Original (b) Cartoon (c) Emoji (d) Enhance (e) Halo (f) Gray (g) Grunge (h) Old paper Figure 1. Example image manipulated by different methods. Several affective image databases have been created in previous works, including artistic photos or abstract paintings used in , International Affective Picture System (IAPS) , The Geneva affective picture database (GAPED)  and Emotion6 . In our research, we are more interested in the emotions of everyday photographs, especially those images that are widely shared by online users. Unfortunately, most existing affective image datasets contain either extremely emotional images, or images without much natural high-level semantic features like human face. All those types of images do not fit our requirements. Therefore we decided to collect our own dataset using Instagram, one of the most popular online photo sharing services. To measure emotions, different types of models have been designed by psychologists. One of the most popular is the valence-arousal (VA) model (proposed by Russell [17]), characterizing emotions in two dimensions, where valence measures attractiveness in a scale from positive to negative, while arousal indicates the degree of excitement or stimulation. In terms of categorization of emotions, Ekman’s six basic emotions (anger, disgust, fear, joy, sadness and surprise) [18] are widely known. In our work, we used both models similar to works in [16, 11]. Image Dataset and User Study This section describes in detail the image dataset creation and crowdsourcing experiment. Image Collection and Processing We collected images from Instagram. According to a previous study by Hu et al. , images shared within Instagram can be classified into the following eight basic categories in terms of their content: Friends, Food, Gadget, Captioned photo, Pet, Activity, Selfie and Fashion. Therefore, we collected image dataset by searching for the eight category keywords or their synonyms via Instagram #tag. This was mainly motivated in order to have a wider variety of image content. At the end 13 color images were selected manually for each category resulting in 104 images in total. All selected images have the same size of 640×640 pixels. For each image, seven different manipulations were applied to create different visual effects. We will refer to these manipulations as the following names: • Cartoon: Applies a cartoon effect to an image. •  Adds an Emoji on top-right corner of an image.  Applies brightness/contrast/colorization enhancement on an image via LAB colorspace. •  Applies a circular halo effect to an image. • Gray: Converts an image to gray scale. • Grunge: Applies a classic vintage effect with a grunge background to an image. •  Applies another heritage style vintage effect with an old paper background to an image. The reason of selecting the seven particular manipulations is that the changes of an image caused by these operations cover different aspects of image information, e.g. color, texture, composition, and higher-level image semantics. The emoji sticker “Tear of Joy” was selected as it has been in the top 10 most popular emojis on Emojipedia for all of 20152, and the emotion it expresses is not that obvious. The seven manipulations were implemented by using ImageMagick software3. An example image processed by the 7 different manipulations is illustrated in Figure 1. Summing up, a grand total of 832 (104× 8) images were generated, including the original versions of each image. The image dataset is publicly accessible at http://mmspg.epfl.ch/ emotion-image-datasets. User Study We used Microworkers4 platform to collect emotional responses from subjects. A questionnaire was designed where four emotion-related questions are asked for each image. The first two questions are about the valence and arousal ratings respectively, where a 9-point scale was used, same as [11, 15]. For valence, 1, 5, and 9 mean very negative, neutral, and very positive emotions respectively, in terms of attractiveness. For arousal, 1 and 9 mean emotions with very low and very high stimulating effects respectively. In the questionnaire, instead of directly asking subjects to provide VA scores, questions were rephrased to be similar as in [11]. The third question is about the emotion distribution of the image, based on Ekman’s six basic emotions [18]. Similar to [11], 7 emotion keywords (Ekman’s six basic emotions and “Neutral”) were used and subjects wer