SlowGuess/ACL_2020_OCR huggingface dataset:
CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotations of Modality
Wenmeng Yu $^{1}$ , Hua Xu $^{2*}$ , Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, Kaicheng Yang
State Key Laboratory of Intelligent Technology and Systems, Department of Computer Science and Technology, Tsinghua University, Beijing, China Beijing National Research Center for Information Science and Technology ywm18@mails.tsinghua.edu.cn<sup>1</sup>, xuhua@tsinghua.edu.cn<sup>2</sup>
Previous studies in multimodal sentiment analysis have used limited datasets, which only contain unified multimodal annotations. However, the unified annotations do not always reflect the independent sentiment of single modalities and limit the model to capture the difference between modalities. In this paper, we introduce a Chinese single- and multimodal sentiment analysis dataset, CH-SIMS, which contains 2,281 refined video segments in the wild with both multimodal and independent unimodal annotations. It allows researchers to study the interaction between modalities or use independent unimodal annotations for unimodal sentiment analysis. Furthermore, we propose a multi-task learning framework based on late fusion as the baseline. Extensive experiments on the CH-SIMS show that our methods achieve state-of-the-art performance and learn more distinctive unimodal representations. The full dataset and codes are available for use at https://github.com/thuiar/MMSA.
Sentiment analysis is an important research area in Natural Language Processing (NLP). It has wide applications for other NLP tasks, such as opinion mining, dialogue generation, and user behavior analysis. Previous study (Pang et al., 2008; Liu and Zhang, 2012) mainly focused on text sentiment analysis and achieved impressive results. However, using text alone is not sufficient to determine the speaker's sentimental state, and text can be misleading. With the booming of short video applications, nonverbal behaviors (vision and audio) are introduced to solve the above shortcomings (Zadeh et al., 2016; Poria et al., 2017).
In multimodal sentiment analysis, intra-modal representation and inter-modal fusion are two im
Figure 1: An example of the annotation difference between CH-SIMS and other datasets. For each multimodal clip, in addition to multimodal annotations, our proposed dataset has independent unimodal annotations. M: Multimodal, T: Text, A: Audio, V: Vision.
important and challenging subtasks (Baltrusaitis et al., 2018; Guo et al., 2019). For intra-modal representation, it is essential to consider the temporal or spatial characteristics in different modalities. The methods based on Convolutional Neural Network (CNN), Long Short-term Memory (LSTM) network and Deep Neural Network (DNN) are three representative approaches to extract unimodal features (Cambria et al., 2017; Zadeh et al., 2017, 2018a). For inter-modal fusion, numerous methods have been proposed in recent years. For example, concatenation (Cambria et al., 2017), Tensor Fusion Network (TFN) (Zadeh et al., 2017), Low-rank Multimodal Fusion (LMF) (Liu et al., 2018),
3718
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3718-3727 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Memory Fusion Network (MFN) (Zadeh et al., 2018a), Dynamic Fusion Graph (DFG) (Zadeh et al., 2018b), and others. In this paper, we mainly consider late-fusion methods that perform intramodal representation learning first and then employ inter-modal fusion. An intuitive idea is that the greater the difference between inter-modal representations, the better the complementarity of intermodal fusion. However, it is not easy for existing late-fusion models to learn the differences between different modalities, further limits the performance of fusion. The reason is that the existing multimodal sentiment datasets only contain a unified multimodal annotation for each multimodal segment, which is not always suitable for all modalities. In other words, all modalities share a standard annotation during intra-modal representation learning. Further, these unified supervisions will guide intra-modal representations to be more consistent and less distinctive.
To validate the above analysis, in this paper, we propose a Chinese multimodal sentiment analysis dataset with independent unimodal annotations, CH-SIMS. Figure 1 shows an example of the annotation difference between our proposed dataset and the other existing multimodal datasets. SIMS has 2,281 refined video clips collected from different movies, TV serials, and variety shows with spontaneous expressions, various head poses, occlusions, and illuminations. The CHEAVD (Li et al., 2017) is also a Chinese multimodal dataset, but it only contains two modalities (vision and audio) and one unified annotation. In contrast, SIMS has three modalities and unimodal annotations except for multimodal annotations for each clip. Therefore, researchers can use SIMS to do both unimodal and multimodal sentiment analysis tasks. Furthermore, researchers can develop new methods for multimodal sentiment analysis with these additional annotations.
Based on SIMS, we propose a multimodal multi-task learning framework using unimodal and multimodal annotations. In this framework, the unimodal and multimodal tasks share the feature representation sub-network in the bottom. It is suitable for all multimodal models based on late-fusion. Then, we introduce three late-fusion models, including TFN, LMF, and Late-Fusion DNN (LF-DNN), into our framework. With unimodal tasks, the performance of multimodal task is significantly increased. Furthermore, we make a detailed discus
sion on multimodal sentiment analysis, unimodal sentiment analysis and multi-task learning. Lastly, we verify that the introduction of unimodal annotations can effectively expand the difference between different modalities and obtain better performance in inter-modal fusion.
In this work, we provide a new perspective for multimodal sentiment analysis. Our main contributions in this paper can be summarized as follows:
- We propose a Chinese multimodal sentiment analysis dataset with more fine-grained annotations of modality, CH-SIMS. These additional annotations make our dataset available for both unimodal and multimodal sentiment analysis.
- We propose a multimodal multi-task learning framework, which is suitable for all late-fusion methods in multimodal sentiment analysis. Besides, we introduce three late-fusion models into this framework as strong baselines for SIMS.
- The benchmark experiments on the SIMS show that our methods learn more distinctive unimodal representations and achieve state-of-the-art performance.
In this section, we briefly review related work in multimodal datasets, multimodal sentiment analysis, and multi-task learning.
2.1 Multimodal Datasets
To meet the needs of multimodal sentiment analysis and emotion recognition, researchers have proposed various of multimodal datasets, including IEMOCAP (Busso et al., 2008), YouTube (Morency et al., 2011), MOUD (Pérez-Rosas et al., 2013), ICT-MMMO (Wöllmer et al., 2013), MOSI (Zadeh et al., 2016), CMU-MOSEI (Zadeh et al., 2018b) and so on. In addition, Li et al. (2017) proposed a Chinese emotional audio-visual dataset and Poria et al. (2018) proposed a multi-party emotional, conversational dataset containing more than two speakers per dialogue. However, these existing multimodal datasets only contain a unified multimodal annotation for each multimodal corpus. In contrast, SIMS contains both unimodal and multimodal annotations.
Item	#
Total number of videos	60
Total number of segments	2,281
- Male	1,500
- Female	781
Total number of distinct speakers	474
Average length of segments (s)	3.67
Average word count per segments	15
Table 1: Statistics of SIMS Dataset.
2.2 Multimodal Sentiment Analysis
Multimodal sentiment analysis has become a major research topic that integrates verbal and nonverbal behaviors. Cambria et al. (2017) proposed a general multimodal sentiment analysis framework that is composed of representation learning on intra-modality and feature concatenation on intermodality. Based on this framework, many studies focused on designing a new fusion network to capture better multimodal representations and achieve better performance. Zadeh et al. (2017) proposed a tensor fusion network, which obtains a new tensor representation by computing the outer product between unimodal representations. Liu et al. (2018) used a low-rank multimodal fusion method to decompose the weight tensor and decrease the computational complexity of tensor-based methods. Zadeh et al. (2018a) designed a memory fusion network with a special attention mechanism for cross-view interactions. Tsai et al. (2019) proposed crossmodal transformers to reinforce a target modality from another source modality by learning the attention across the two modalities' features. Tsai et al. (2018) learned meaningful multimodal representations by factorizing representations into two sets of independent factors: multimodal discriminative and modality-specific generative factors. Different from the above methods, we aim to learn more distinctive unimodal representations by introducing independent unimodal annotations.
2.3 Multi-task Learning
Multi-task learning aims to improve the generalization performance of multiple related tasks by utilizing useful information contained in these tasks (Zhang and Yang, 2017). A classical method is that different tasks share the first several layers and then have task-specific parameters in the subsequent layers (Liu et al., 2015; Zhang et al., 2016b). Based on this method, we design a multimodal multi-task
learning framework for verifying the practicality and feasibility of independent unimodal annotations.
3 CH-SIMS Dataset
In this section, we introduce a novel Chinese multimodal sentiment analysis dataset with independent unimodal annotations, CH-SIMS. In the following subsections, we will explain the data acquisition, annotation, and feature extraction in detail.
Comparing with unimodal datasets, the requirements of multimodal datasets are relatively high. A fundamental requirement is that the speaker's face and voice must appear in the picture at the same time and remain for a specific period of time. In this work, to acquire video clips as close to life as possible, we collect target fragments from movies, TV series, and variety shows. After getting raw videos, we use video editing tools, Adobe Premiere Pro<sup>1</sup>, to crop target segments at the frame level, which is very time-consuming but accurate enough. Moreover, during the data collection and cropping, we enforce the following constraints:
- We only consider mandarin and are cautious with the selection of materials with the accent.
- The length of clips is no less than one second and no more than ten seconds.
- For each video clip, no other faces appear except for the speaker's face.
Finally, we collect 60 raw videos and acquire 2,281 video segments. SIMS has rich character background, wide age range, and high quality. Table 1 shows the basic statistics for SIMS.
3.2 Annotation
We make one multimodal annotation and three unimodal annotations for each video clip. In addition to the increase in workload, the mutual interference between different modalities is more confused. To avoid this problem as much as possible, we claim every labeler can only see the information in the current modality when annotating. Besides, conducting four annotations at the same time is not
<sup>1</sup>https://www.adobe.com/products/premiere.html
2We consulted a legal office to verify that the academic usage and distribution of very short length videos fall under the fair use category.
Figure 2: Left: the distribution of sentiment over the entire dataset in one Multimodal annotation and three single-modal (Text, Audio, and Vision) annotations. Right: the confusion matrix shows the annotations difference between different modalities in CH-SIMS. The larger the value, the greater the difference.
permitted. More precisely, every labeler makes unimodal annotation first and then performs multimodal annotation, which of the order is text first, audio second, then silent video, and multimodal last.
For each clip, every annotator decides its sentimental state as -1 (negative), 0 (neutral) or 1 (positive). we have five independent students in this field making annotations. Then, in order to do both regression and multi-classifications tasks, we average the five labeled results. Therefore, the final labeling results are one of  $\{-1.0, -0.8, -0.6, -0.4, -0.2, 0.0, 0.2, 0.4, 0.6, 0.8, 1.0\}$ . We further divide these values into 5 classifications: negative  $\{-1.0, -0.8\}$ , weakly negative  $\{-0.6, -0.4, -0.2\}$ , neutral  $\{0.0\}$ , weakly positive  $\{0.2, 0.4, 0.6\}$  and positive  $\{0.8, 1.0\}$ .
The histogram in the left of Figure 2 shows the distribution of sentiment over the entire dataset in four annotations. We can see that negative segments are more than positive segments. The main reason is that actors in film and television dramas are more expressive in negative sentiments than positive ones. The confusion matrix in the right of Figure 2 indicates the annotations difference between different modalities, which is computed as:
$$
D _ {i j} = \frac {1}{N} \sum_ {n = 1} ^ {N} \left(A _ {i} ^ {n} - A _ {j} ^ {n}\right) ^ {2} \tag {1}
$$
where  $i,j\in \{m,t,a,v\}$  , N is the number of all samples,  $A_{i}^{n}$  means the  $n_{th}$  label value in modal  $i$
From the confusion matrix, we can see that the difference between A and M is minimal, and the
difference between V and T is maximal, which is in line with expectations. Because audio contains text information, closer to multimodal while the connection between video and text is sparse.
Furthermore, we provide the other attribute annotations, including speakers' age and gender. And we use sentimental annotations only in our following experiments.
3.3 Extracted Features
The extracted features for all modalities are as follows (we use the same basic features in all experiments):
Text: All videos have manual transcription, including the Chinese and English versions. We use Chinese transcriptions only. We add two unique tokens to indicate the beginning and the end for each transcript. And then, pre-trained Chinese BERT-base word embeddings are used to obtain word vectors from transcripts (Devlin et al., 2018). It is worth noting that we do not use word segmentation tools due to the characteristic of BERT. Eventually, each word is represented as a 768-dimensional word vector.
Audio: We use LibROSA (McFee et al., 2015) speech toolkit with default parameters to extract acoustic features at  $22050\mathrm{Hz}$ . Totally, 33-dimensional frame-level acoustic features are extracted, including 1-dimensional logarithmic fundamental frequency (log F0), 20-dimensional Melfrequency cepstral coefficients (MFCCs) and 12-dimensional Constant-Q chromatogram (CQT). These features are related to emotions and tone of speech according to (Li et al., 2018).
Figure 3: Multimodal multi-task learning framework.
Vision: Frames are extracted from the video segments at  $30\mathrm{Hz}$ . We use the MTCNN face detection algorithm (Zhang et al., 2016a) to extract aligned faces. Then, following Zadeh et al. (2018b), we use MultiComp OpenFace2.0 toolkit (Baltrusaitis et al., 2018) to extract the set of 68 facial landmarks, 17 facial action units, head pose, head orientation, and eye gaze. Lastly, 709-dimensional frame-level visual features are extracted in total.
4 Multimodal Multi-task Learning Framework
In this section, we describe our proposed multimodal multi-task learning framework. Shown as Figure 3, based on late-fusion multimodal learning framework (Cambria et al., 2017; Zadeh et al., 2017), we add independent output units for three unimodal representations: text, audio, and vision. Therefore, these unimodal representations not only participate in feature fusion but are used to generate their predictive outputs.
For the convenience in following introduction, in text, audio and vision, we assume that  $L^u,D_i^u,D_r^u$  where  $u\in \{t,a,v\}$  , represent the sequence length, initial feature dimension extracted by section 3.3 and representation dimension learned by unimodal feature extractor, respectively. The batch size is  $B$
4.1 Unimodal SubNets
Unimodal subNets aim to learn intra-modal representations from initial feature sequences. A universal feature extractor can be formalized as:
$$
R _ {u} = S _ {u} (I _ {u}) \tag {2}
$$
where  $I_{u}\in R^{B\times L^{u}\times D_{i}^{u}}$ $R_{u}\in R^{B\times D_{r}^{u}}$ $S_{u}(\bullet)$  is the feature extractor network for modal  $u$
In this work, following Zadeh et al. (2017); Liu et al. (2018), we use a Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) network, a deep neural network with three hidden layers of weights  $W_{a}$  and a deep neural network with three hidden layers of weights  $W_{v}$  to extract textual, acoustic and visual embeddings, respectively.
4.2 Feature Fusion Network
Feature fusion network aims to learn inter-modal representation with three unimodal representations, formulated as:
$$
R _ {m} = F \left(R _ {t}, R _ {a}, R _ {v}\right) \tag {3}
$$
where  $R_{t}, R_{a}, R_{v} \in R^{B \times D_{r}^{u}}$  are the unimodal representations.  $F(\bullet)$  is the feature fusion network and  $R_{m}$  is the fusion representation.
In this work, for full comparison with existing works, we try three fusion methods: LF-DNN, TFN (Zadeh et al., 2017) and LMF (Liu et al., 2018).
4.3 Optimization Objectives
Except for the training losses in different tasks, we sparse the sharing parameters via L2 norm, which aims to select intra-modal features. Therefore, our optimization objectives is:
$$
\min  \frac {1}{N _ {t}} \sum_ {n = 1} ^ {N _ {t}} \sum_ {i} \alpha_ {i} L \left(y _ {i} ^ {n}, \hat {y} _ {i} ^ {n}\right) + \sum_ {j} \beta_ {j} \left\| W _ {j} \right\| _ {2} ^ {2} \tag {4}
$$
where  $N_{t}$  is the number of training samples,  $i\in$ $\{m,t,a,v\}$ $j\in \{t,a,v\}$ $L(y_i^n,\hat{y}_i^n)$  means the training loss of  $n_{th}$  sample in modality  $i$  .  $W_{j}$  is the sharing parameters in modality  $j$  and multimodal tasks.  $\alpha_{i}$  is the hyperparameter to balance different tasks and  $\beta_{j}$  represents the step of weight decay of subNet  $j$  , respectively.
Lastly, we use a three-layer DNN to generate outputs of different tasks. In this work, we treat these tasks as regression models and use the L1 loss as training loss in Equation 4.
In this section, we mainly explore the following problems using SIMS:
(1) Multimodal Sentiment Analysis: We evaluate the performance of multimodal multi-task learning methods comparing with the other methods. The aim is to validate the advantages of multi-task
Model	Acc-2	Acc-3	Acc-5	F1	MAE	Corr
EF-LSTM	69.37 ± 0.0	51.73 ± 2.0	21.02 ± 0.2	81.91 ± 0.0	59.34 ± 0.3	-04.39 ± 2.8
MFN	77.86 ± 0.4	63.89 ± 1.9	39.39 ± 1.8	78.22 ± 0.4	45.19 ± 1.2	55.18 ± 2.0
MULT	77.94 ± 0.9	65.03 ± 2.1	35.34 ± 2.9	79.10 ± 0.9	48.45 ± 2.6	55.94 ± 0.6
LF-DNN	79.87 ± 0.6	66.91 ± 1.2	41.62 ± 1.4	80.20 ± 0.6	42.01 ± 0.9	61.23 ± 1.8
MLF-DNN*	82.28 ± 1.3	69.06 ± 3.1	38.03 ± 6.0	82.52 ± 1.3	40.64 ± 2.0	67.47 ± 1.8
\( \bigtriangledown \)	↑2.41	↑2.15	↓3.59	↑2.32	↓1.37	↑6.24
LMF	79.34 ± 0.4	64.38 ± 2.1	35.14 ± 4.6	79.96 ± 0.6	43.99 ± 1.6	60.00 ± 1.3
MLMF*	82.32 ± 0.5	67.70 ± 2.2	37.33 ± 2.5	82.66 ± 0.7	42.03 ± 0.9	63.13 ± 1.9
\( \bigtriangledown \)	↑2.98	↑3.32	↑2.19	↑2.70	↓1.96	↑3.13
TFN	80.66 ± 1.4	64.46 ± 1.7	38.38 ± 3.6	81.62 ± 1.1	42.52 ± 1.1	61.18 ± 1.2
MTFN*	82.45 ± 1.3	69.02 ± 0.3	37.20 ± 1.8	82.56 ± 1.2	40.66 ± 1.1	66.98 ± 1.3
\( \bigtriangledown \)	↑1.79	↑4.56	↓1.18	↑0.94	↓1.86	↑5.80
learning with unimodal annotations and set up multimodal baselines for SIMS.
(2) Unimodal Sentiment Analysis: We analyze the performance in unimodal tasks with unimodal or multimodal annotations only. The aim is to validate the necessary of multimodal analysis and set unimodal baselines for SIMS.
(3) Representations Differences: We use t-SNE to visualize the unimodal representations of models with or without independent unimodal annotations. The aim is to show that the learned unimodal representations are more distinctive after using unimodal annotations.
In this section, we briefly review our baselines used in the following experiments.
Early Fusion LSTM. The Early Fusion LSTM (EF-LSTM) (Williams et al., 2018) concatenates initial inputs of three modalities first and then use LSTM to capture long-distance dependencies in a sequence.
Later Fusion DNN. In contrast with EF-LSTM, the Later Fusion DNN (LF-DNN) learns unimodal features first and then concatenates these features before classification.
Memory Fusion Network. The Memory Fusion Network (MFN) (Zadeh et al., 2018a) accounts for view-specific and cross-view interactions and continuously models them through time with a special attention mechanism and summarized through time with a Multi-view Gated Memory. MFN needs
Table 2: (%) Results for sentiment analysis on the CH-SIMS dataset. The models with * are multi-task models, extended from single-task models by introducing independent unimodal annotations. For example, MLF-DNN* is the extension of LF-DNN. The rows with  $\bigtriangledown$  means the improvements or reductions of new models compared to original ones in the current evaluation metric.
Item	Total	NG	WN	NU	WP	PS
#Train	1,368	452	290	207	208	211
#Valid	456	151	97	69	69	70
#Test	457	151	97	69	69	71
Table 3: Dataset splits in SIMS. We split train, valid and test set in 6:2:2. NG: Negative, WN: Weakly Negative, NU: Neutral, WP: Weakly Positive, PS: Positive.
word-level alignment in three modalities. However, this is not easy for SIMS because we haven't found a reliable alignment tool of Chinese corpus. In this work, we follow Tsai et al. (2019) to use CTC (Graves et al., 2006) as an alternative.
Low-rank Multimodal Fusion. The Low-rank Multimodal Fusion (LMF) (Liu et al., 2018) model learns both modality-specific and cross-modal interactions by performing efficient multimodal fusion with modality-specific low-rank factors.
Tensor Fusion Network. The Tensor Fusion Network (TFN) (Zadeh et al., 2017) explicitly models view-specific and cross-view dynamics by creating a multi-dimensional tensor that captures unimodal, bimodal and trimodal interactions across three modalities.
Multimodal Transformer. The Multimodal Transformer (MULT) (Tsai et al., 2019) using the directional pairwise crossmodal attention to realize the interactions between multimodal sequences across distinct time steps and latently adapt streams from one modality to another.
Task	Label	Acc-2	F1	MAE	Corr
A	A	67.70	79.61	53.80	10.07
M	65.47	71.44	57.89	14.54
V	V	81.62	82.73	49.57	57.61
M	74.44	79.55	54.46	38.76
T	T	80.26	82.93	41.79	49.33
M	75.19	78.43	52.73	38.55
Table 4: (%) Results for unimodal sentiment analysis on the CH-SIMS dataset using MLF-DNN. The column of "Label" indicates which annotation we use in this task.
In this section, we introduce our experimental settings in detail, including dataset splits, hyperparameters selection, and our evaluation metrics.
Dataset Splits. We shuffle all video clips in random first and then divide train, valid and, test splits by multimodal annotations. The detailed split results are shown in Table 3.
Hyper-parameters Selection. Due to the different sequence lengths in different segments, it is necessary that fixing sequence length for the specific modality. Empirically, we choose the average length plus three times the standard deviation as the maximum length of the sequence. Besides, for all baselines and our methods, we adjust their hyperparameters using grid search with binary classification accuracy. For a fair comparison, in each experiment, we select five same random seeds (1, 12, 123, 1234, and 12345) and report the average performance of five times.