VIDEO AFFECTIVE IMPACT PREDICTION WITH MULTIMODAL FUSION AND LONG-SHORT TEMPORAL CONTEXT
Authors: Yin Zhao
Longjun Cai
Chaoping Tu
Jie Zhang
Wu Wei
Venue:  (2020)
Keywords: multi-modal fusion
affective computing
temporal context
residual-based training strategy

Abstract: Predicting the emotional impact of videos using machine learning is a challenging task. Feature extraction, multi-modal fusion and temporal context fusion are crucial stages for predicting valence and arousal values in the emotional impact, but
have not been successfully exploited. In this paper, we proposed a comprehensive framework with innovative designs of model structure and multi-modal fusion strategy. We select the most suitable modalities for valence and arousal tasks respectively and each modal feature is extracted using the modality-specific pre-trained deep model on large generic dataset. Two-time-scale structures, one for the intra-clip and the other for the inter-clip, are proposed to capture the temporal dependency of video content and emotional states. To combine the complementary information from multiple modalities, an effective and efficient residual-based progressive training strategy is proposed. Each modality is step-wisely combined into the
multi-modal model, responsible for completing the missing parts of features. With all those above, our proposed prediction framework achieves better performance with a large margin compared to the state-of-the-art.

Under review as a conference paper at ICLR 2020
VIDEO AFFECTIVE IMPACT PREDICTION WITH MULTI- MODAL FUSION AND LONG-SHORT TEMPORAL CON- TEXT
Anonymous authors Paper under double-blind review
ABSTRACT
Predicting the emotional impact of videos using machine learning is challenging task. Feature extraction, multi-modal fusion and temporal context fusion are cru- cial stages for predicting valence and arousal values in the emotional impact, but have not been successfully exploited. In this paper, we proposed a comprehen- sive framework with innovative designs of model structure and multi-modal fu- sion strategy. We select the most suitable modalities for valence and arousal tasks respectively and each modal feature is extracted using the mpoeaiity-peccffe pre- trained deep model on large generic dataset. Two-time-scale structures, one for the intra-clip and the other for the inter-clip, proposed capture the temporal dependency of video content and emotional states. To combine the complemen- tary information from multiple modalities, an effective and efficient residual-bai based progressive training strategy is proposed. Each modality is step-wisely combined model, responsible for completing the missing parts of fea- tures. With all those above, our proposed prediction framework achieves better performance with large margin compared the offte-of-thh-ars
the
INTRODUCTION
Affective video content analysis aims at predicting the videos' emotional impact on audiences. plays important roles in understanding the videos" content, highlight detection, and of a fundamen- tal support for several advanced applications such as multi-modal search with sentimental queries. Predicting the audiences' emotional evolvement when watching movies is also an important way to help both online media-server providers or filmmakers to invest movies, evaluate on-line effect as well as distribute them more efficiently.
In the affective computing community, human emotions can be categorically or continuously defined [lzard] 2007; Barrett et al.) (007). Emotion categories are happy, sad, angry, surprise, disgust, neutral, which are commonly used in emotion recognition or classification (Cowie et al.] [001]] Compared to the categorical definition, continuous definition describes the emotions continuously in two dimensions: Valence (positive vs. negative) and Arousal (active vs. calm). Any human emotion can be located the space spanned by the two dimensions, which is more fine-grained than the categorical definition. The goal of our task is to predict the audiences' emotional states based on the movie content, i.e. the valence and arousal values with the movie going on.
Finding discriminative features from raw videos for predicting valence and arousal values is far away from an easy task. Video is the typical multi-modal media involving both audio and visual modalities. Even if in visual content, human facial expressions, pose behaviors, scenes, etc. can also be regarded as modalities. Audiences' emotions can be triggered by any modality such as the actors' expressions actions, the movies" scenes (environment, atmosphere) as well as background music. Therefore, the mainstream of affective video content analysis is to extract multi-moaal features and combine all those features.
Feature fusion another challenging step. Multi-modal features are always complementary and the importance of each modality dynamically changes over time. For example, some movie clips' emotional impact can be captured by audio content while others may rely on visual features. Cur- rent studies of affective video content analysis mainly adopt either decision-level fusion (Dobrišek)

---

Under review as a conference paper at ICLR 2020
et al. 2013) feature-level fusion (Wimmer et al. 2008). The former combines results from each modality through voting or weighted average methods. Each modality-speciin model is trained independently which can't exploit the complementary information between modalities. The latter concatenates multi-modal features and learn parameters for all modalities at the same time which can easily lead to overfitting We design a progressive training algorithm where each modality is trained and fine-tuned stepwisely. Each modality is only responsible for completing the missing parts of features extracted from previous modalities, thus the most discriminative modalities can be dynamically selected for each movie clip and the complementarity of multi-moda features can be fully utilized. The overfitting risk can also be suppressed since fewer parameters are learned at each step.
Besides all the above, we also investigate how to utilize the temporal context of videos for senti- ment prediction, which is lacking in most of the related works, where they simply apply LSTMs (Hochreiter Schmidhuber, 1997) or GRUs (Cho et al. 2014) for temporal dependency. We pro- pose two-time-scale model structures considering the video's 's long-short temporal context. For the short-time context, LSTMS are used for each modality. For the long-time dependency of the valence task, a structure that is similar to the temporal segment network (TSN) (Wang et al.) 2016) is used to capture the long temporal context. For the arousal task, a moving mean post- post-processing method is adopted utilize the trend of previous emotions.
The contributions of this paper are as follows: We propose an effective multi-modal fusion network and design two-time-scale model structure considering the video's 's long-short temporal context for affective impact prediction. Also, a residual-based progressive training strategy is used to train the fusion network fully utilize the complementary and representative capability of each modality. Our model and training strategy achieve a new state-of-art on several related tasks.
RELATED WORKS
Video content involves both audio and visual elements. The basic part of affective video content analysis involves extracting audio and visual features characterize the video content. In early stages, most works (Xu et al. 2013; Moreira et al. 2015) extract handcrafted features such as Local Binary Patterns (LBP), Histogram of oriented gradient (HOG), etc, to represent visual features, and Linear Predictive Coding coefficients (LPC), Mel Frequency Cepstral Coefficient (MFCC) to repre- sent audio features. In recent years, with the development of deep learning, semanties-meaningfuu meaningful features extracted by deep models are becoming more and more popular in affective video content analysis and many works extract features with pre-trained CNN models. For example, VGG-like model named VGGish (Hershey et al.) (017) is adopted for extracting audio features and CNN mod- els trained on generic task datasets, such as ImageNet (Deng et al. 2009). RAF (L et al. 2017) are used for extracting visual features (Liu et al. 2018
Another focus of the emotion prediction is multi-modal fusion (Atrey et al., 2010). Multi-modal fusion methods can be divided into two categories: feature-level fusion, and decision-level fusion. The key difference between the above two categories is the stage when the fusion happens. Rosas et al. (203) concatenate linguistic, audio and visual features into common feature vector and seek to find the hyperplane that best separates positive examples from negative examples using SVMs (Cortes & Vapnik 1995) with linear kernels. Wang & Cheong (2006) characterized every scene through concatenating audio and visual features to a vector and adopt specially adapted variant of SVM recognize anger, sadness, surprise, happiness, disgust and neutral. Metallinou et al. (2010) model face, voice and head movement cues for emotion recognition and fuse the results of all clas- sifiers using Bayesian framework. While those models can outperform unimodal models, they fail to utilize the dependencies among different modalities. [ang et al. (2015) use Deep Boltzmann Ma- chine (DBM) to learn the highly non-linear relationships that exist among low-level features across different modalities for emotion prediction. Gan et al. (2017) propose multi-modal deep regres- sion Bayesian network (MMDRBN) to capture the dependencies between visual elements and audio elements and a fast learning algorithm is designed to learn the regression Bayesian network (RBN), Then the MMDREN is transformed into an inference network by minimizing the KL-ddvergence.
To utilize the temporal context, Kurpukdee et al. (2017) extract phoneme-based features from raw input speech signals using convolutional long short-term memory (LSTM), recurrent neural net- work MOowL.STM-RNT and adopt support vector machines (SVM) or linear discriminant analysis
2

---

Under review as a conference paper at ICLR 2020
(LDA) to classify four emotions (anger. happiness, sadness and neutral). Fan al. (2016) RNNS to fuse features extracted by the convolutional neutral network (CNN) over individual video frames and use C3D (Tran al. 2015) encode appearance and motion information at the same time, then the predicted scores from different models are combined in a weighted-sum rule.
3 PROPOSED METHODS
The overall framework consists of three major steps as shown in Fig I We divide the untrimmed movies to non-overlap short clips with same length and predict valence and arousal for each clip. First, extract frame-level modality-specifft features using commonly-used pre-trane pre-trained audio and image feature extractors. Then, the intra-clip feature fusion and inter-clip context fusion are used to consider the short and long temporal context. For intra-clip feature fusion, we append LSTMs after the frame-level feature extractor to consider the short- nootttimee temporal relation within the clip for each modality. Then, all modality-speciiff features are summed into vector, representing the clip-level multi-modal features. For the inter-clip context, we adopt LSTMs based on the clip-level features to capture the long-time dependency between clips for the valence task. For the arousal task, a more simple but effective exponential moving average with decay weights is utilized. A residual- progressive training algorithm is designed to train the network.
City sta
temppota
Deniled
Tetx-dia Saner Baike
Erbediing leauree
Scarc
Modal
Expecsiion
PC
(5)
Figure 1: Part (a) describes our overall framework for valence and arousal tasks separately, Features for multi-modal are extracted as detailed in part (b)
3.1 MULTI-MODAL FEATURES
The modal scope could use for valence and arousal tasks in movie affective analysis is based on the following observations. 1) The actors/Actress' actions, dialogues, and facial expressions are the key factors that affect the audience's emotions. Scenes (such environment, atmosphere) and back- ground music also implicitly deliver emotions to audiences. 2) Valence and Arousal emotional states depend on different modalities. For instance, Facial expressions are more related to the valence task while actions are more related to the arousal task. The actors' actions can affect the intensity of emo- tions while they are less related to the direction of emotions (Detenber & Reeves [996).. Therefore, for the valence task, audio, scene and facial expression modalities are finally used. For the arousal task, audio, scene, and action modalities are finally used. Related experiments in section 4.2.1 and 4.2.2 also support our observations.
3.1.1
AUDIO FEATURES
Audio information in movie can be divided into two main parts: the physical characteristics of sound and the content of language. We only focus on manguage-independeer audio features. Here we adopt VGGish extract semantically meaningful features with all audio characteristics taken into
3

---

Under review as a conference paper at ICLR 2020
consideration. The inputs are log Mel-spectrograms with shape 96x64 computed with 0.96s long audio clips. We adopt the first 960 ms from every second of the audio. We adopt the output of the 128-wide fully connected layer followed by PCA transformation and quantization the compact emhedding features for audio. Thus, every second of audio is converted into 128-dimension vector.
3.1.2 VISUAL FEATURES
Visual features consist of the human' actions, expressions as well as scenes in videos. A sparse sampling strategy, one frame per second, is adopted sample frames from clips. Each feature will be extracted in frame-level.
We adopt pose-related features to represent the human' actions. Specifically, we append two groups of convolution/maxpool layers and a 128-wide fully connected layer after the last convolution layer of the pre-trained OpenPose's backbone (Cao et al., 2018). The weights of the OpenPose's 's backbone are fixed and the newly-added layers will be fine-tuned on the target dataset. To represent the actors' expressions, we pre-train an Xception network (Chollet, 2017) with fully connected layers on RAF to extract emotional facial features. To extract facial expression feature, we first detect faces from frames by MTCNN (Zhang et al.) 2016). Then we crop the largest face detected and resize it to 160x 160. The face image fed into the pre-trained model and is represented by a 3072-dimension vector extracted from the last fully connected layer. If no faces are detected, the average face across the whole training dataset is used. A pre-trained VGG16 network Places365 is adopted to extract features of the movies' scenes. We resize the frames to 224x224 and extract 51 12-dimennion features at the last pooling layer.
3.2 FEATURE FUSION WITH LONG-SHORT TEMPORAL CONTEXT
3.2.1 INTRA-CLIP SHORT TEMPORAL FUSION
Since each modality has its own time-dependency, we first earry out short-temporal fusion for each modality. After the frame-level features for each modality are obtained, we adopt two layers of bidirectional I.STMs for each modality inside the clip to fuse the temporal information. The hidden state of the final step of the LSTMs is adopted as the clip-level modality-specifie features. Having obtained the modality-specifie features, we sum all the features together for each clip. Then a fully connected layer is followed, forming the final clip-level multi-moda features.
3.2.2 INTER-CLIP LONG TEMPORAL FUSTON
We adopt two different methods to utilize the temporal context among clips for valence and arousal tasks. For the valence task, we adopt a TSN-like structure to utilize the temporal context among con- tinuous clips. Specifically, we use two-layers bidirectional LSTMs to combine the temporal context among continuous clips. The input for each step is the clip-level multi-modal features obtained in the intra-clip short temporal fusion. The hidden state for each step the final embedding features combining long temporal context for the corresponding clip. A fully connected layer with Tanh activation is adopted to make final valence value prediction for each clip.
For arousal value prediction, we directly append one fully connected layer with Tanh activation as the raw arousal prediction after the intra-clip features. Then a simple but effective exponential moving average with decay weights is utilized to consider the temporal context. The reason we use moving average instead LSTMs like the valence task is that the arousal value represents the intensity of emotions, which cannot dramatically change in relative short time, while for valence, the emotion direction might be more context-related.
3.3 TRAINING STRATEGY
Training the whole network, which has multiple stages with several sub-structures, an end-to- end way is not easy task, which can easily lead to be over-fitting because of the large number parameters. Thus, we train the whole network part by part. For VGGish, facial expression model, VGG16 pre-trained on Places365, and the modified OpenPose model, the weights are always fixed in our tasks using the pre- pre-trained weights. For the LSTMs that and feature fusion network, we use a residual-base based progressive training strategy.
4

---

Under review as a conference paper at ICLR 2020
3.3.1 RESIDUAL-BASED PROGRESSIVE TRAINING STRATEGY
To learn the LSTMs for every modality short temporal fusion and effectively fuse multi-modal features, we design two-stage residual based progressive training strategy as shown in Fig
Rosadaak leased hwwpeps--leelllrggrrroooo exsive tralnsing slperiten for intra chip--cpplsttturrr Fusien
Chp
Mivic
stp
Moulify-am te
Saap
Stap-1
Sop-
Eabollka barn
Kxyeu
lapor
Figure 2: Residual-basee progressive training strategy for intra-clip feature fusion of valence
Stage 1: Modality-spedde speciffc pre-training
The main role of this step is to obtain the importance of each modality and determine the training order for each modality in the next stage. We append two fully connected layers for auxiliary training after the LSTMs of each modality to form the modality-specin model. Then we train each modalittysspeciff model by minimizing the Mean Square Error (MSE) loss. The loss function can be computed as follows:
m
L(y. G)
1ly, m
G
Where y/ is the prediction for clip j. Gj represents the ground truth for clip j. is the mean of labels for every second within the clip. denotes the batch size.
After the models are trained, we sort modalities in descending order according to their performance. Thus, modality means the modality with the ith high performance and the LSTMs for will be trained in sequence
Stage 2: Training intra-clip model
At the first step, we train the LSTMS for modality 1. At the second step, we sum the features extracted by the LSTMs for modality and modality 2 and only train the LSTMs for modality 2. The remaining modalities' LSTMs are learned similarly. At each step, two new fully connected layers are appended after the summed features make predictions. Note that all the fully connected layers are only used for auxiliary LSTMs pre-training. After we get the weights for all modalities' LSTMs, we add two fully connected layers after the LSTMs and fine-tune the entire model. For the valence task, the output of the first fully connected layer is used as the clip- embedding features. For the arousal task, the fully connected layers are used to make the prediction for each clip. The training process for the valence task is shown in Fig 2 as example.
With the residual-based progressive training strategy, the model can dynamically select important modalities. Formally, assuming that we are training the ith modality, the features combining the first - modalities are denoted as fi-1- Thus the LSTMs for modality fit the mapping H₃(x) := ft ft-1- If fi-1 is discriminative enough make correct predictions and modality is of low importance, would push the mapping to zero. modality plays an important role, it would push the mapping complete the fi-1 towards f, to get better performance.
5

---

Under review as a conference paper at ICLR 2020
3.3.2 LONG TEMPORAL FUSION
For arousal, there is no parameters train for long temporal fusion since we adopt an exponential moving average post-processing method over the predictions made for intra-clip part. Formally, the final prediction can be computed as follows:
ema, * ema, + (1 3) y/
where y/ is the prediction for clip i, ema; denotes the exponential moving average value for clip i which is used as the final prediction, / is the weight decay.
For valence, we keep the weights for the intra-clip part fixed and only train the LSTMs and fully connected layer for the inter-clip part. The loss function be computed follows:
L(Y,))= - - E
E - G
(3)
Where 3f and G, denote the prediction and ground truth for the ith clip in the jth example respec- tively. m represents the batch size. L is the number of clips in example.
EXPERIMENTS AND RESULTS
4.1 DATASET AND METRICS
The LIRIS-ACCEDE dataset is the largest dataset for affective video content analysis, which is used in the MediaEval 2018 emotional impact of movies task. The LIRIS-ACCEDE dataset contains videos from set of 160 professionally made and amateur movies. Several movie genres are repre- sented in this collection of movies such horror, comedy, drama, action and so on. Languages are mainly English with small set of Italian, Spanish, French, and others. A total of 54 movies (total duration
hours and 49 minutes) from the set of 160 movies are provided as the development set. 12 other movies (total duration of 8 hours and 56 minutes) consist the test set. The scores of valence and arousal which range from -1 to 1 are provided continuously (every second) along movies. Valence is defined on continuous scale from most negative to most positive emotions, while arousal is defined continuously from calmest to most active emotions. The official metric is the Mean Square Error (MSE), which is the common measure generally used to evaluate regression models. However, we also consider Pearson's 's Correlation Coefficient (PCC) for the emotional trend analysis of movies.
4.2 RESULTS
4.2.1
MODALITY-SPECINE PERFORMANCE
To evaluate the importance of every modality for each task, we train set of modality-ppecific models, i.c. only using one modality in the overall prediction framework. The performance for modality-specific model is shown in Table [ To better demonstrate each modality's 's effectiveness, we visualize the features representations each modality in the test set. T-SNE is adopted for dimensionality reduction.
As shown in Table the modality-specific model trained on audio features has the best performance for both valence and arousal tasks. This demonstrates that audio signals contain more emotional information. For the valence task, expressions and scenes play similar importance while actions have the worst performance. For the arousal task, actions and scenes have equal status while expressions are useless. The importance of each modality can also be revealed by the discriminative of features extracted through the codaalty-speccife models as shown in Fig This demonstrates our analysis in section 3.1
4.2.2 MODAI- PERFORMANCE FOR INTRA-CLIP PART
Here, we compare the performance of different multi-modes combinations and fusion strategies in intra-clip prediction. According the analysis and experiments, audio and scenes used for both

---

Under review as a conference paper at ICLR 2020
Table 1: Performance for each modality-ppecific models.
Valence
Arousal
Modality used
MSE
PCC
MSE
PCC
Audio
0.098
0.264
0.140
0.172
Scene
0.103
0.192
0.152
0.140
Expression
0.110
0.150
0.162
0.061
Action
0.132
0.057
0.156
0.158
Audio
Scene
Emotion
Pose
valence
Arousal
Figure 3: Features extracted from movic clips in the test set by the modality-speciiff models. T-SNE adopted for dimensionality reduction. The red points represent the features with label greater than 0 and the blue points represent the features with label less than O.
valence and arousal tasks while expressions are only used in the valence task and actions are only used in the arousal task. The training process described in section 3.3.I
To compare the different modal combinations, we carry out several experiments with different se- tups, i.c audio scene, audio action, audio scene human expression and audio action scene using the same training strategy. We also conduct two baseline experiments to investigate performance of different the modal fusion strategies: feature-level fusion with traditional training method, i.e. training the entire model at the same time, which is referred as Baseline 1, and decision- level fusion, i.c. learning modality-speciiff models independently and averaging the results, which is referred as Baseline 2. For the valence task, both baseline models audio, scenes, and expres- sions. For the arousal task, we use audio, scenes, and actions. The results of various methods are listed in Table 2
As shown in Table 2 our proposed modal selection and feature-level fusion with our residual- based progressive training algorithm get better performance compared with traditional training method and also outperforms the decision-level fusion.
4.2.3 PARAMETER SELECTION FOR LONG TEMPORAL FUSION
The methods to utilize the long temporal context are described section 3.2.2 and 3.3.2 in which, the number of clips for the valence task and the decay weights 3 for the arousal task the param- eters that should be determined beforehand. Here we numerically evaluate the performance under different parameter values.

---

Under review as a conference paper at ICLR 2020
Table 2: Performance of different modal combinations & feature fusions.
Valence
Arousal
MSE
PCC
MSE
PCC
Audio+tScene
0.091
0.348
Audio+Action
0.140
0.293
Audio++Seene++Expressii
0.089
0.358
Actio-encin
0.138
0.314
Baseline
0.104
0.284
0.148
0.257
Baseline 2
0.090
0.300
0.142
0.278
Table 3: Performance with long temporal context. The left part shows performance for the valence task with different number of clips; the right part shows performance for the arousal task with different decay weights.
Valence
Arousal
number of clips
decay weights B
MSE
PCC
MSE
PCC
0.073
0.406
0.96
0.136
0.400
0.071
0.444
0.97
0.136
0.409
0.072
0.419
0.98
0.137
0.419
0.077
0.380
0.99
0.140
0.427
For the valence task, using 4 continuous movie clips gets the best result, which is shown in the left part of Table 3 Using fewer clips can't 't get enough long time information while using more clips weakens the flow of information and may introduce noises by obvious emotional change between clips. For the arousal task, the best choice Table The larger decay weight will pull the arousal value of every second towards the mean value resulting the higher MSE.
decay weight 0.98 as shown in the right part of
Table 4: Performance compared with the state-of-the the-
Valence
Arousal
MSE
PCC
MSE
PCC
CERTH-ITI (Batziou al. 2018
0.117
0.098
0.138
0.054
THUHCSI (Ma et al. 2018
0.092
0.305
0.140
0.087
Quan et al. (2018
0.115
0.146
0.171
0.091
et al. 2018)
0.090
0.301
0.136
0.175
GLA (Sun et al. 2018
0.084
0.278
0.133
0.351
Ko et al. (2018)
0.102
0.114
0.149
0.083
Ours
0.071
0.444
0.137
0.419
4.2.4 COMPARISON WITH THE STATE-OF-THE-ARR
We present our whole emotion prediction result in Table 4 It shows that our method gets signifi- cantly better performance compared with the other works. To further prove the effectiveness of our model structure and residual- based training strategy, we also carry out experiments on classification task of the LIRIS-ACCEDE dataset which is used MediaEval 2015 (Sjöberg et al. 2015). We use the same backbone network and the same training strategy with only the last layer changed to the softmax layer for classification. The performance compared with the other works are shown in Table 5 It can be
that our backbone network and training strategy still outperform the state-of-art in this task, which further demonstrates the effectiveness of our extracted features and residual-based progressive training algorithm.
8

---

Under review as a conference paper at ICLR 2020
Table 5: Performance compared with MediaEval 2015 related works.
Valence (acc)
Arousal (acc)
MIC-TJU Yi el al. 2015
0,420
0.559
NII-UIT Vu Lam
0,430
0.559
ICL-TUM-PASSAAN Tngeorrii et
2015
0,415
0.557
Huawei (Dai et al. 2015
0,418
0.488
TCS-ILAB (Chakraborty et al. 2015
0.357
0,490
UMons (Seddati et al. 2015
0.373
0.524
RFA (Mironica et al. 2015
0.330
0.450
KIT (Marin Vlastelica al. 2015) Ours
0.385
0.519
0.459
0.575
5 CONCLUSION AND FUTURE WORK
In this work, propose comprchensive video emotion impact prediction framework. For Ience and arousal tasks, modalities are carefully selected through evaluating the importance of every modality reduce noise introduced by less task-related modalities, then the pre-trained models are used to extract semantically meaningful features. Two-time-scale structures for for valence and arousal tasks adopted capture the shot-long temporal context. An effective and effi- cient residual-based progressive training algorithm is proposed. The experimental results on LIRIS- ACCEDE dataset with several comparative studies demonstrate the effectiveness of our methods. Future work might involve how to incorporate more sophisticated time dependency models of human emotion states and explore more discriminative features such linguistic content, visual relation, etc.
REFERENCES
Pradeep K Atrey, M Anwar Hossain, Abdulmotaleb El Saddik, and Mohan S Kankanhalli. Multi- modal fusion for multimedia analysis: survey. Multimedia systems, 16(6):345-37,, 2010.
Lisa Feldman Barrett, Kristen A Lindquist, Eliza Bliss-Moreau, Seth Duncan, Maria Gendron, Jen- nifer Mize, and Lauren Brennan. Of mice and men: Natural kinds emotions in the mammalian brain? response panksepp and izard. Perspectives on Psychological Science, 2(3):297-312, 2007.
Elissavet Batziou, Emmanouil Michail, Konstantinos Avgerinakis, Stefanos Vrochidis, Ioannis Pa- tras, and Ioannis Kompatsiaris. Visual and audio analysis of movies video for emotion detection @ emotional impact of movies task mediaeval 2018. In MediaEval, 2018.
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. OpenPose: realtime multi- person 2D pose estimation using Part Affinity Fields. In arXiv preprint arXiv:18/2.08000 2018.
Rupayan Chakraborty, Avinash Kumar Maurya, Meghna Pandharipande, Ehtesham Hassan, Hiran- may Ghosh, and Sunil Kumar Kopparapu. Tes-ilab-mediaewy 2015: Affective impact of movies and violent scene detection. In MediaEval, 2015.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1406. 1078, 2014.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1251-1258, 2017.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273-29) 1995.

---

Under review as a conference paper at ICLR 2020
Roddy Cowie, Ellen Douglas-Cowiz Nicolas Tsapatsoulis, George Votsis, Stefanos Kollias, Win- fried Fellenz, and John Taylor. Emotion recognition in human-computer interaction. IEEE Signal processing magazine, 18(1):32 2001.
Qi Dai, Rui- Wei Zhao, Zuxuan Wu, Xi Wang, Zichen Gu, Wenhai Wu, and Yu-Gang Jiang. Fudan- huawei at mediaeval 2015: Detecting violent scenes and affective impact in movies with deep learning. In MediaEval, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Benjamin H Detenber and Byron Reeves. A bio-informational theory of emotion: Motion and image size effects on viewers. Journal of Communication, 1996.
Simon Dobrišek, Rok Gajšek, France Mihelič, Nikola Pavešić, and Vitomir Struc. Towards efficient multi-modal emotion recognition. International Journal of Advanced Robotic Systems, 10(1):53) 2013.
Yin Fan, Xiangju Lu, Dian Li, and Yuanliu Liu. Video-based emotion recognition using enn-rnn and c3d hybrid networks. In Proceedings of the 18th ACM International Conference on Multimodal Interaction. pp. 445-450. ACM, 2016.
Quan Gan, Shangfei Wang, Longfei Hao, and Qiang A multimodal deep regression bayesian net- work for affective video content analyses. In Proceedings of the IEEE International Conference on Computer Vision, pp. 511--512, 2017.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Cnn architectures for large-scale audio classification. In 2017 ieee international conference on acoustics, speech and signal processing (icassp), pp. 131-135 IEEE, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Carroll E Izard. Basic emotions, natural kinds, emotion schemas, and a new paradigm. Perspectives on psychological science, 2(3):260-280, 2007.
Tobey H. Ko, Zhonglei Gu, Tiantian He, and Yang Liu. Towards learning emotional subspace. In MediaEval, 2018.
Nattapong Kurpukdee, Tomoki Koriyama, Takao Kobayashi, Sawit Kasuriya, Chai Wutiwiwatchai, and Poonlap Lamsrichan. Speech emotion recognition using convolutional long short-term mem- ory neural network and support vector machines. In 2017 Asia-Pucific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 1744-1749. IEEE, 2017.
Shan Li, Weihong Deng, and JunPing Du. Reliable crowdsourcing and deep locality-preserving learning for expression recognition the wild. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2852-2861, 2017. 2017..
Chuanhe Liu, Tianhao Tang, Kui Lv, and Minghao Wang. Multi-feature based emotion recognition for video clips. In Proceedings of the 2018 on International Conference on Multimodal Interac- lion, pp. 630-634. ACM, 2018
Ye Ma, Xihao Liang, and Mingxing Xu. Thuhesi in mediaeval 2018 emotional impact of movies task. In MediaEval, 2018
Marin Vlastelica, Sergey Hayrapetyan, Makarand Tapaswi, and Rainer Stiefelhagen. Kit at medi- aeval 2015-evaluating visual cues for affective impact of movies task. In MediaEval, 2015.
Angeliki Metallinou, Sungbok Lee, and Shrikanth Narayanan. Decision level combination of multi- ple modalities for recognition and analysis of emotional expression. In 2010 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2462-2465, 2465. IEFF, 2010.
10

---

Under review as a conference paper at ICLR 2020
Ionut Mironica, Bogdan lonescu, Mats Sjöberg, Markus Schedl, and Marcin Skowron. Rfa at medi- aeval 2015 affective impact of movies task: multimodal approach. In MediaEval, 2015.
Daniel Moreira, Sandra Eliza Fontes de Avila, Mauricio Perez, Daniel Moraes, Vanessa Testoni, Eduardo Valle, Siome Goldenstein, and Anderson Rocha. Recod mediaeval 2015: Affective impact of movies task. In MediaEval, 2015.
Lei Pang, Shiai Zhu, and Chong- Whong--Wah Wah Ngo. Deep multimodal learning for affective analysis and retrieval. IEEE Transactions on Multimedia, 17(1 (7(11):00882022) 2015.
Khanh-An C. Quan, Vinh-Tiep Nguyen, and Minh-Triet Tran. Frame- evaluation with deep features to predict emotional impact of movies. In MediaEval, 2018.
Verónica Pérez Rosas, Rada Mihalcea, and Louis-Philippe Morency. Multimodal sentiment analysis of spanish online videos. IEEE Intelligent Systems, 28(3):38 45, 2013.
Omar Seddati, Emre Kulah, Gueorgui Pironkov, Stéphane Dupont, Said Mahmoudi, and Thierry Du- toit. Umons at mediaeval 2015 affective impact of movies task including violent scenes detection. In MediaEval, 2015.
Mats Sjöberg, Yoann Baveye, Hanli Wang. Vu Lam Quang, Bogdan lonescu, Emmanuel Dellandréa, Markus Schedl, Claire-Hélen Demarty, and Liming Chen. The mediaeval 2015 affective impact of movies task. In MediaEval, 2015.
Jennifer Sun, Ting Liu, and Gautam Prasad. Gla in mediaeval 2018 emotional impact of movies task. In MediaEval, 2018.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 4489 19999-449,,
George Trigeorgis, Eduardo Coutinho, Fabien Ringeval, Erik Marchi, Stefanos Zafeiriou, and Bjöm W Schuller. The icl-tum-passau passau approach for the mediaeval 2015" affective impact of movies" task. In MediaEval, 2015.
Sang Phan Le Vu Lam and Duy-Dinh Le. Shin'ichi satoh, and duc anh duong. 2015. nii-uit at mediaeval 2015 affective impact movies task. In Working Notes Proceedings of the MediaEval 2015 Workshop, Wurzen, Germany, September, pp. 14-15.
Hee Lin Wang and Loong-Fah Cheong- Affective understanding in film. IEEE Transactions on circuits and systems for video technology, 16(6):689-704, 2006.
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. European conference on computer vision, pp. 20-36. Springer, 2016.
Matthias Wimmer, Björn Schuller, Dejan Arsic, Bernd Radig, and Gerhard Rigoll. Low-level fu- sion of audio and video feature for molti- emotion recognition. Proc. 3rd Int. Conf. on Computer Vision Theory and Applicutions VISAPP, Funchal, Madeira, Portugal, pp. 145-151, 2008
Min Xu, Changsheng Xu, Xiangjian He, Jesse S Jin, Suhuai Luo, and Yong Rui. Hierarchical affective content analysis in arousal and valence dimensions. Signal Processing, 93(8):2140- 2150, 2013.
Yun Yi, Hanli Wang, Bowen Zhang, and Jian Yu. Mic-tju in mediaeval 2015 affective impact of movies task. In MediaEval, 2015.
Yun Yi, Hanli Wang, and Qinyu Li. Con features for emotional impact of movies task. In MediaEval, 2018.
Kaipeng Zhang. Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10): 1499-1503, 2016.
11

---

Under review as a conference paper at ICLR 2020
A
CASES ANALYSIS
02
0.75
predicted scores ground truth
0.50
Mo Dal o Payp da
-0.4
predicted ground
-0.8
+1.00
140
1000
2000
3000
4000
Figure 4: The predictions of movies in test set for the valence task. MEDIAEVALI8 54; the right one is for MEDIAEVALI8 62.
The left one is for
1.00
predicted
0.75
ground truth
M
0.5
8.00
0.3
0.50
0.0
-1.00
500
3000
1500
2000
2500
200
400
600
800
2000
1200
1400
Figure The predictions of movies in the test set for the arousal task. MEDIAEVAL18 60; The right one is for MEDIAEVALI8 63.
The left one is for
0.8
06
0:5
0.00
02
-0.25
-0.
predicted
-0.75
-1.00
truth
200
400
600
500
3500
2000
2500
time
time
Figure 6: Bad cases in the test set. The left one is the valence predictions for MEDIAEVALI8 65 from 3000s to 4000s; The right one is the arousal predictions for MEDIAEVALI8 60.
In this part, we show both good and bad cases of the result for LIRIS-ACCEDI dataset in detail. Fig 4 and Fig 5 show the good cases, where our model can precisely predict the trend of movies' impact on audiences. Figlsshows some bad cases. the left chart of Fig the ground truth for most of the time are positive while the corresponding predictions are negative. This movie is a comedy, where the fighting and quarrels have positivechilariou) emotional impact but they are difficult to identify just by the modals of our model. In the right chart, the prediction fluctuates much more than the ground truth. This is due to there are fighting and shouting in that movie clips, but the low arousal labels result from human understanding of the movie story. Those bad cases indicate the difficulty of this task since too many factors might be missing by the computer models and there is still large gap machine understand human emotions.
12


Reviews:
[{"review_id": "SygpRQ0sFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work design a framework to predict valence and arousal values in the emotional impact. Although its performance is much better than previous works, I have some questions about the current submission:\n-- The writing is not clear enough. I have to guess the technical details based on the context. For example, in Equation 1, how to estimate y_{i}? And why put 1/m behind \\sum? Is there any particular reason for this? It is a classification problem here, why use MSE loss rather than cross-entropy loss?\n\n-- \"If modality i plays an important role, it would push the mapping to complete the f_{i-1} towards f_i to get better performance.\" Are you suggesting that sometimes H() will be about zero?\n\n-- Figure 2 is not clear enough for me. I still can not fully understand what ``progressive\" means here. I will appreciate it if the authors can make this more clear. \n\n-- What concerns me the most is the novelty of this work. It seems solid and effective. However, based on the current content, I can not fully appreciate its novelty. Using deep learning models to extract features from different modalities has been used before. Using deep learning to consider inter-clip and intra-clip relations also has been conducted. Using LSTM for fusion is not new, either. The authors claimed they propose a progressive training strategy for effectively training and information fusion. However, given the current version, I can not fully appreciate it.  \n\nI can change my point if the authors would provide more details and explanations later, which can help me to understand the novelty of this work fully. Thanks.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SygCYdSRFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of affective impact prediction from multimodal sequences. The authors achieve state-of-the-art performance by using (i) a two-time-scale temporal feature extractor, (ii) progressive training strategy for multi-modal feature fusion, and (iii) pretraining. They divide a long video sequence evenly into several clips. The idea of applying one LSTM for intra-clips (short-time) temporal feature extraction and another LSTM for inter-clips (long-time) temporal feature extraction, resulting in two-time-scale, looks reasonable choice but not novel. The proposed progressive training strategy is mainly used for the modality-specific LSTMs training, which is used during the intra-clip short-time modeling phrase. It seems working well in practice. However, from the explanation, it's not clear why this strategy is good for the *complementary* fusion. Each modality LSTM is trained sequentially with features extracted from other fixed modality LSTMs. Also, the authors seem not explaining why they set the training order for LSTMs at this stage to be the descending order of their performance in the previous stage. In the experiments section, two weak baseline models and several previous state-of-the-art models are compared. However, enough ablation studies on the two-time-scale structure and proposed training strategy are not provided to demonstrate that the proposed method does provide complementary multi-modal feature fusion, which is claimed as a contribution. \n\nOverall, although it achieves state-of-the-art performance on the task, none of the claimed contributions is novel or significant enough. It is a combination of existing ideas (but giving good performance). The proposed training procedure is also weak to be considered as a scientific contribution.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hkxiqjr8qr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a framework to predict valence and arousal tasks in videos. The framework mainly employs LSTM in a two-time-scale-structure to take multimodal inputs. In general, the proposed framework groups well-studied techniques to solve a well-known task of multimodal learning. \n\nDNN based Multimodal learning has been heavily investigated for a long time. Numerous frameworks have been proposed with various success. Although the proposed framework is technically sound, the proposed \"residual-based training strategy\" and \"long temporal fusion\" are kind of trivial or lackluster. I can hardly identify any significant contributions that support a publication in top machine learning conferences such as ICLR.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is no author response for this paper. The paper addresses the affective analysis of video sequences in terms of continual emotions of valence and arousal. The authors propose a multi-modal approach (combining modalities such as audio, pose estimation, basic emotions and scene analysis) and a multi-scale temporal feature extractor (to capture short and long temporal context via LSTMs) to tackle the problem. All the reviewers and AC agreed that the paper lacks (1) novelty, as the proposed approach is a combination of the existing well-studied techniques without explanations why and when this could be advantageous beyond the considered task, (2) clarity and motivation -- see R2’s and R3’s concerns and suggestions on how to improve. We hope the reviews are useful for improving the paper.

Decision: Reject