VIDEO AFFECTIVE IMPACT PREDICTION WITH MULTIMODAL FUSION AND LONG-SHORT TEMPORAL CONTEXT

1 INTRODUCTION . Affective video content analysis aims at predicting the videos ’ emotional impact on audiences . It plays important roles in understanding the videos ’ content , highlight detection , and of a fundamental support for several advanced applications such as multi-modal search with sentimental queries . Predicting the audiences ’ emotional evolvement when watching movies is also an important way to help both online media-server providers or filmmakers to invest movies , evaluate on-line effect as well as distribute them more efficiently . In the affective computing community , human emotions can be categorically or continuously defined ( Izard , 2007 ; Barrett et al. , 2007 ) . Emotion categories are happy , sad , angry , surprise , disgust , neutral , which are commonly used in emotion recognition or classification ( Cowie et al. , 2001 ) . Compared to the categorical definition , continuous definition describes the emotions continuously in two dimensions : Valence ( positive vs. negative ) and Arousal ( active vs. calm ) . Any human emotion can be located in the space spanned by the two dimensions , which is more fine-grained than the categorical definition . The goal of our task is to predict the audiences ’ emotional states based on the movie content , i.e . the valence and arousal values with the movie going on . Finding discriminative features from raw videos for predicting valence and arousal values is far away from an easy task . Video is the typical multi-modal media involving both audio and visual modalities . Even if in visual content , human facial expressions , pose behaviors , scenes , etc . can also be regarded as modalities . Audiences ’ emotions can be triggered by any modality such as the actors ’ expressions or actions , the movies ’ scenes ( environment , atmosphere ) as well as background music . Therefore , the mainstream of affective video content analysis is to extract multi-modal features and combine all those features . Feature fusion is another challenging step . Multi-modal features are always complementary and the importance of each modality dynamically changes over time . For example , some movie clips ’ emotional impact can be captured by audio content while others may rely on visual features . Current studies of affective video content analysis mainly adopt either decision-level fusion ( Dobrišek et al. , 2013 ) or feature-level fusion ( Wimmer et al. , 2008 ) . The former combines results from each modality through voting or weighted average methods . Each modality-specific model is trained independently which can ’ t exploit the complementary information between modalities . The latter concatenates multi-modal features and learn parameters for all modalities at the same time which can easily lead to overfitting . We design a progressive training algorithm where each modality is trained and fine-tuned stepwisely . Each modality is only responsible for completing the missing parts of features extracted from previous modalities , thus the most discriminative modalities can be dynamically selected for each movie clip and the complementarity of multi-modal features can be fully utilized . The overfitting risk can also be suppressed since fewer parameters are learned at each step . Besides all the above , we also investigate how to utilize the temporal context of videos for sentiment prediction , which is lacking in most of the related works , where they simply apply LSTMs ( Hochreiter & Schmidhuber , 1997 ) or GRUs ( Cho et al. , 2014 ) for temporal dependency . We propose two-time-scale model structures considering the video ’ s long-short temporal context . For the short-time context , LSTMS are used for each modality . For the long-time dependency of the valence task , a structure that is similar to the temporal segment network ( TSN ) ( Wang et al. , 2016 ) is used to capture the long temporal context . For the arousal task , a moving mean post-processing method is adopted to utilize the trend of previous emotions . The contributions of this paper are as follows : We propose an effective multi-modal fusion network and design a two-time-scale model structure considering the video ’ s long-short temporal context for affective impact prediction . Also , a residual-based progressive training strategy is used to train the fusion network to fully utilize the complementary and representative capability of each modality . Our model and training strategy achieve a new state-of-art on several related tasks . 2 RELATED WORKS . Video content involves both audio and visual elements . The basic part of affective video content analysis involves extracting audio and visual features to characterize the video content . In early stages , most works ( Xu et al. , 2013 ; Moreira et al. , 2015 ) extract handcrafted features such as Local Binary Patterns ( LBP ) , Histogram of oriented gradient ( HOG ) , etc , to represent visual features , and Linear Predictive Coding coefficients ( LPC ) , Mel Frequency Cepstral Coefficient ( MFCC ) to represent audio features . In recent years , with the development of deep learning , semantics-meaningful features extracted by deep models are becoming more and more popular in affective video content analysis and many works extract features with pre-trained CNN models . For example , A VGG-like model named VGGish ( Hershey et al. , 2017 ) is adopted for extracting audio features and CNN models trained on generic task datasets , such as ImageNet ( Deng et al. , 2009 ) , RAF ( Li et al. , 2017 ) are used for extracting visual features ( Liu et al. , 2018 ) . Another focus of the emotion prediction is multi-modal fusion ( Atrey et al. , 2010 ) . Multi-modal fusion methods can be divided into two categories : feature-level fusion , and decision-level fusion . The key difference between the above two categories is the stage when the fusion happens . Rosas et al . ( 2013 ) concatenate linguistic , audio and visual features into a common feature vector and seek to find the hyperplane that best separates positive examples from negative examples using SVMs ( Cortes & Vapnik , 1995 ) with linear kernels . Wang & Cheong ( 2006 ) characterized every scene through concatenating audio and visual features to a vector and adopt a specially adapted variant of SVM to recognize anger , sadness , surprise , happiness , disgust and neutral . Metallinou et al . ( 2010 ) model face , voice and head movement cues for emotion recognition and fuse the results of all classifiers using a Bayesian framework . While those models can outperform unimodal models , they fail to utilize the dependencies among different modalities . Pang et al . ( 2015 ) use Deep Boltzmann Machine ( DBM ) to learn the highly non-linear relationships that exist among low-level features across different modalities for emotion prediction . Gan et al . ( 2017 ) propose a multi-modal deep regression Bayesian network ( MMDRBN ) to capture the dependencies between visual elements and audio elements and a fast learning algorithm is designed to learn the regression Bayesian network ( RBN ) , Then the MMDRBN is transformed into an inference network by minimizing the KL-divergence . To utilize the temporal context , Kurpukdee et al . ( 2017 ) extract phoneme-based features from raw input speech signals using convolutional long short-term memory ( LSTM ) , recurrent neural network ( ConvLSTM-RNN ) and adopt support vector machines ( SVM ) or linear discriminant analysis ( LDA ) to classify four emotions ( anger , happiness , sadness and neutral ) . Fan et al . ( 2016 ) use RNNs to fuse features extracted by the convolutional neutral network ( CNN ) over individual video frames and use C3D ( Tran et al. , 2015 ) to encode appearance and motion information at the same time , then the predicted scores from different models are combined in a weighted-sum rule . 3 PROPOSED METHODS . The overall framework consists of three major steps as shown in Fig 1 . We divide the untrimmed movies to non-overlap short clips with same length and predict valence and arousal for each clip . First , we extract frame-level modality-specific features using commonly-used pre-trained audio and image feature extractors . Then , the intra-clip feature fusion and inter-clip context fusion are used to consider the short and long temporal context . For intra-clip feature fusion , we append LSTMs after the frame-level feature extractor to consider the short-time temporal relation within the clip for each modality . Then , all modality-specific features are summed into a vector , representing the clip-level multi-modal features . For the inter-clip context , we adopt LSTMs based on the clip-level features to capture the long-time dependency between clips for the valence task . For the arousal task , a more simple but effective exponential moving average with decay weights is utilized . A residual-based progressive training algorithm is designed to train the network . 3.1 MULTI-MODAL FEATURES . The modal scope we could use for valence and arousal tasks in movie affective analysis is based on the following observations . 1 ) The actors/Actress ’ actions , dialogues , and facial expressions are the key factors that affect the audience ’ s emotions . Scenes ( such as environment , atmosphere ) and background music also implicitly deliver emotions to audiences . 2 ) Valence and Arousal emotional states depend on different modalities . For instance , Facial expressions are more related to the valence task while actions are more related to the arousal task . The actors ’ actions can affect the intensity of emotions while they are less related to the direction of emotions ( Detenber & Reeves , 1996 ) . Therefore , for the valence task , audio , scene and facial expression modalities are finally used . For the arousal task , audio , scene , and action modalities are finally used . Related experiments in section 4.2.1 and 4.2.2 also support our observations . 3.1.1 AUDIO FEATURES . Audio information in a movie can be divided into two main parts : the physical characteristics of sound and the content of language . We only focus on language-independent audio features . Here we adopt VGGish to extract semantically meaningful features with all audio characteristics taken into consideration . The inputs are log Mel-spectrograms with shape 96x64 computed with 0.96s long audio clips . We adopt the first 960 ms from every second of the audio . We adopt the output of the 128-wide fully connected layer followed by a PCA transformation and quantization as the compact embedding features for audio . Thus , every second of audio is converted into a 128-dimension vector . 3.1.2 VISUAL FEATURES . Visual features consist of the human ’ actions , expressions as well as scenes in videos . A sparse sampling strategy , one frame per second , is adopted to sample frames from clips . Each feature will be extracted in frame-level . We adopt pose-related features to represent the human ’ actions . Specifically , we append two groups of convolution/maxpool layers and a 128-wide fully connected layer after the last convolution layer of the pre-trained OpenPose ’ s backbone ( Cao et al. , 2018 ) . The weights of the OpenPose ’ s backbone are fixed and the newly-added layers will be fine-tuned on the target dataset . To represent the actors ’ expressions , we pre-train an Xception network ( Chollet , 2017 ) with fully connected layers on RAF to extract emotional facial features . To extract facial expression feature , we first detect faces from frames by MTCNN ( Zhang et al. , 2016 ) . Then we crop the largest face detected and resize it to 160x160 . The face image is fed into the pre-trained model and is represented by a 3072-dimension vector extracted from the last fully connected layer . If no faces are detected , the average face across the whole training dataset is used . A pre-trained VGG16 network on Places365 is adopted to extract features of the movies ’ scenes . We resize the frames to 224x224 and extract 512-dimension features at the last pooling layer .

Summary: This paper tackles the problem of affective impact prediction from multimodal sequences. The authors achieve state-of-the-art performance by using (i) a two-time-scale temporal feature extractor, (ii) progressive training strategy for multi-modal feature fusion, and (iii) pretraining. They divide a long video sequence evenly into several clips. The idea of applying one LSTM for intra-clips (short-time) temporal feature extraction and another LSTM for inter-clips (long-time) temporal feature extraction, resulting in two-time-scale, looks reasonable choice but not novel. The proposed progressive training strategy is mainly used for the modality-specific LSTMs training, which is used during the intra-clip short-time modeling phrase. It seems working well in practice. However, from the explanation, it's not clear why this strategy is good for the *complementary* fusion. Each modality LSTM is trained sequentially with features extracted from other fixed modality LSTMs. Also, the authors seem not explaining why they set the training order for LSTMs at this stage to be the descending order of their performance in the previous stage. In the experiments section, two weak baseline models and several previous state-of-the-art models are compared. However, enough ablation studies on the two-time-scale structure and proposed training strategy are not provided to demonstrate that the proposed method does provide complementary multi-modal feature fusion, which is claimed as a contribution.