CMMA: Benchmarking Multi-Affection Detection in Chinese Multi-Modal Conversations
Authors: Yazhou Zhang
Yang Yu
Qing Guo
Benyou Wang
Dongming Zhao
Sagar Uprety
Dawei Song
Qiuchi Li
Jing Qin
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: multi-modal affection detection
conversation dataset
multi-task learning
deep learning
multi-modal fusion
TL;DR: We make the first attempt to annotate the relevance intensity across related affections.

Abstract: Human communication has a multi-modal and multi-affection nature. The inter-relatedness of different emotions and sentiments poses a challenge to jointly detect multiple human affections with multi-modal clues. Recent advances in this field employed multi-task learning paradigms to render the inter-relatedness across tasks, but the scarcity of publicly available resources sets a limit to the potential of works. To fill this gap, we build the first Chinese Multi-modal Multi-Affection conversation (CMMA) dataset, which contains 3,000 multi-party conversations and 21,795 multi-modal utterances collected from various styles of TV-series. CMMA contains a wide variety of affection labels, including sentiment, emotion, sarcasm and humor, as well as the novel inter-correlations values between certain pairs of tasks. Moreover, it provides the topic and speaker information in conversations, which promotes better modeling of conversational context. On the dataset, we empirically analyze the influence of different data modalities and conversational contexts on different affection analysis tasks, and exhibit the practical benefit of inter-task correlations. The full dataset will be publicly available for research\footnote{https://github.com/annoymity2022/Chinese-Dataset}

CMMA: Benchmarking Multi-Affection Detection in Chinese Multi-Modal Conversations
Yazhou Zhang
Yang Yu Zhengzhou University of Light Industry yuyangg198001Beett...coo com
The Hong Kong Polytechnic University, China Mobile Communication Group Tianjin Co., Zhengzhou University of Light Industry y wzwhouuhhangge ed
Qing Guo
Benyou Wang
IHPC and CFAR, Agency for Science, Technology and Research, Singapore s tsinggguofieee org
The Chinese University of Hong Kong, Shenzhen, Shenzhen Research Institute Big Data
wabykinggemaal com
Dongming Zhao
Sagar Uprety
China Mobile Communication Group Tianjin Co. wai maiitman840000
Bravura Solutions
supretyybrravurasoolltioos.....s
Dawei Song
School of Computer Science and Technology, Beijing Institute of Technology dwsongebit. edu.. en
Qiuchi Li"
Jing Qin
Copenhagen University qiuchi .li@di, ku. dk
The Hong Kong Polytechnic University harzy. qin@pol edu.hA hk
Abstract
Human communication has a multi-modal and multi-affect nature. The inter- relatedness of different emotions and sentiments poses challenge to jointly detect multiple human affects with multi-modal clucs. Recent advances this field employed multi-task learning paradigms render the inter-relatedness across tasks, but the scarcity publicly available resources sets limit to the potential of works. To fill this gap, build the first Chinese Multi-modal Multi-Affect conversation (CMMA) dataset, which contains 3,000 multi-party conversations and 21,795 multi-modal utterances collected from various styles of TV-eeries CMMA contains a wide variety of affect labels, including sentiment, emotion, sarcasm and humor, as well as the novel inter-correlations values between certain pairs of tasks. Moreover, it provides the topic and speaker information in conversations. which promotes better modeling of conversational context. On the dataset, we empirically analyze the influence of different data modalities and conversational contexts on different affect analysis tasks, and exhibit the practical benefit of inter-task correlations. The full dataset will be publicly available for researcl
"Corresponding author: Qiuchi Li thups://gtthu..coo htt.g//www...oooooomm/moo////eea
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Dapic
Text
Conteal
Video
Speakers
Video
Audio
Sar.
Andio
Sar.
Deteet
Deteet
Canteall
Eiteonnce
Outpel
Output
sople
Cencriational tapie
2
Affection kaee
2
Contest
Multi cfflitiffeettee dercction
Mudii medial otterance
Muli-ctrdal utterance
(b)
Figure 1: Previous works neglect topic, speaker knowledge and relatedness across affects (a). Our work can incorporate all of them into unified framework (b). Abbreviations: Sen.: Sentiment, Emo.: Emotion, Sar.: Sarcasm, Hum.: Humor.
Introduction
Human communication is multi-modal with textual (e.g., natural language), visual (e.g., facial gestures) and audio (e.g., tones) channels, and also multi-affective in that different types of affects, such as sentiment, sarcasm and humor, are often expressed in a mixed manner. The interactions of different modalities and inter-correlations between different affect types bring opportunities well as challenges for multi-modal affect detection, especially in a conversational context. Research works traditionally targeted at sentiment [ and emotion 21. the two most common affect types. More recently, increasing attention has been placed on sarcasm detection
3 and humor detection [4].
To set the scope, we target the above four affect detection tasks, which are inherently inter-related. Sentiment and emotion correspond long-term and short-term human subjective experience and mental attitude, where stable sentiments are rooted in and expressed through emotion. Sarcasm is subtle form of human figurative language that intends to express criticism, humor or anger emotions, while humor is treated as sentimental tendency to provoke laughter and provide amusement, often in a sarcastic way 55. The recognition of one affect type may potentially benefit the understanding of the other.
Related work. Despite the crucial role of inter-relatedness between different affect types, they were not sufficiently captured in existing multi-modal affect detection works, largely due to the lack of publicly available datasets for this purpose. Traditional works simply rendered the tasks as being mutually independent and ignored the mutual influence across them (6). The recent popularity of the multi-task learning paradigm led to an emerging focus on multi-affect joint detection, such as joint classification
B. and sarcasm- humor joint detection (8. However, the datasets they worked on, such as MELD 9 IEMOCAP [IO], two types of affect, and inter-relatedness between tasks is absent. Without an explicit annotation of cross-task correlations, the potential of multi-modal multi-affect joint detection could not be fully explored, neither deepen the understanding on human complicated affects.
sentiment-emotion joint recognition 77], sentiment-sarcan
UR-FUNNY [5], MUStARD [[]] etc., have annotations solely
We fill the gap by constructing large-scale benchmark multi- multi-affect conversational dataset. We manage to tackle the following main challenges for building such a dataset: (1) multi- affect joint judgment: the subjectivity and creativity of human language make it hard to judge different affects at the same time accurately; (2) multi-affect correlation: different affects can be indistinguishable at certain circumstances, and it is difficult to accurately measure their relatedness; (3) context effect: an utterance may express different affects in different conversational contexts.
To address above challenges, we build a novel Chinese Multi-modal Aalti-Affection conversation dataset, termed CMMA. It consists of 21,795 multi-modal utterances from 3,000 multi-party conver- sations collected from various TV series with many kinds of subjects and styles, e.g., comedy, thriller, drama, etc. Each utterance is manually annotated with sentiment (including pride and romantic love). emotion, sarcasm and humor labels, accompanied by coatiment-emotionn and sarcasm humoo inter-relatedness measures. Considering that the external knowledge implicitly influences the speaker's affective state, the speaker's 's background (i.e., name, profession, sex, personality) and the topic of each conversation are provided (c.f. Fig. in supplementary document). CMMA supports both single-task and multi-task learning paradigms for human affect understanding, More importantly,

---

Table 1: Comparison of CMMA with other datasets. x denotes that the dataset only provides the speaker's name.
Duttast
Type
Skar
Modality
Rescurce
Langrage
Amuutatio
Knter-Task Correiation
Speaker Information
Topic
Youliube
Vilao
4F
Test, Image: Spazch
YouTube
Eaglish
Scniimzr
Video
Speech
YouTube
English
Sentincs
X
x
X
MOSE
Vilno
Text. Image. Spazch
Bnglish
Sentimzre
CH-SIMS
Vadao
2,381
Test, Image. Spazch
Mosic, TV
Cliooc
Seniinera
X
RMMDCAP
Dislegue
100099
Test. Speech
Performance
lnglish
Friviio
X
Dialogue
13,708
Text, Inage. Speech
Show
Eaglish
Sentinerr Enocion
K
MESIO
Dislogue
20,000
Year. Inage. Speech
TV Show
Biglish
Sentimerr Broccoon
X
ScenarioSA
Diálogue
24,072
Taxt
Social Medaa
English
Sentimes
X
MUAIDD
Disleguc
Image. Spazch
English
Teetter
Teeeet
24,835
Year. Image
TV Show
Biglish
Soceaen
Inezagrum pow
N0
West. Imppt. Sparch
TV Shone
Peglish
Seesem
MHD
Dislogue
13633
Text,
Speech
Show
English
Hamor
Diálogue
39,769
Text, Image. Speech
Show
English
Hansu
K
UR FUNNY
TED ull
16.514
Yosr, Inage. Speech
TV Show
Eaglish
Hamer
MUMOR
Dialoguc
19,103
Test, Image: Spezch
TV Show
Engiah, Chincec
Scaiineri Erssars, Hame
Dislegso
Image, Sporch
HnglishHirdh
X
Sentimzre, Fineraon, Sarcson
Memetion
Interve Mome
3,571
Test. Image
Sucodl Maraa
Baglish
Hamaer Oifimenc: Mativational
X
x
Senlimenl, Emollim, Sareasm
CMMA (Ours)
Dialogue
21,745
Test, Image, Speech
TV Show
Chinese
Humorr Prite, Love
as illustrated in Figure D CMMA can support multi-task learning paradigms better than existing datasets, with richer multi-task, multi-modal clues and more external knowledge. We present an overview of related resources covering sentiment, emotion, sarcasm and humor with a comprehensive comparison in Table [ Please refer to App.B (in supplementary document) for more details.
We further conduct pilot study on the CMMA dataset by employing state-of-the-an (SOTA) multi- modal models for small range of tasks with all combinations input features. We find out that multi-affect detection is more dependent on conversational context, but the speaker and topic information is also beneficial. In addition, we empirically show the benefits of sentiment-emottee human-sarcasm inter-relatedness to multi-task joint detection frameworks. In summary, our major contributions are presented as follows:
and
We construct the first Chinese multi-moda multi-affect conversation dataset annotated with the sentiment, emotion, sarcasm, and humor labels. It provides a benchmark for multi-affect detection frameworks.
We make the first attempt to manually annotate the relevance intensity between sentiment and emotion, and between sarcasm and humor. This will help determine the meco-decy task and improve current multi-task learning frameworks. addition, the dataset is the first of its kind that includes the speaker's 's information (i.e., name, profession, sex, personality) and conversation topics.
We provide the annotations of sentiment, emotion, sarcasm and humor, along with well-illustratee quality control and agreement analysis.
We show a comprehensive statistics of the dataset, covering the distribution of TV sources, characters and affect types.
We propose six multi-modal affect detection tasks to evaluate CMMA. The results SOTA baselines using different feature combinations suggest the need for multi-task learning models.
2
CMMA Dataset
2.1 Data Acquisition
We aim to create a large-scale multi-modal multi-affect dataset support affect understanding. Following the rule of "tepresentativeness" in corpus linguistics' guidelines [12,, the samples is diverse across (1) domains: a broad range of domains should be included cover the expressions of sentiment, emotion, sarcasm, humor, pride and love; (2) speakers: different speakers have various means of expressing affect; (3) topics: topics may have sentimental tendencies. For example, crimes and disasters often indicate negative sentiment, while romance and love suggest the opposite; (4) modalities: affect communication multi-modal, covering natural language (text), facial expression (vision) and acoustic tonality (audio).
3

---

Table 2: Statistics of CMMA, (t,v.a) = (text, video, audio).
Item
Train
Dev
Test
#Modalities
(1,v,a)
(Lvv.a)
(L.v.)
#Conversations
1800
600
600
#Utterances
13788
4046
3961
#Speakers
299
78
119
#Words
115,434
35,487
34,521
#Unique words
2,677
1,842
1,988
#Video duration
9.2h
3.0h
3.0h
#Average utterances per conversation
7.7
6.8
6,6
#Average words per conversation
64.1
59.1
57.5
#Average words per utterance
8.4
8.8
8.7
Average duration of conversation
18.5s
18.4s
17.8s
#Average duration an utterance
2.4s
2.7s
2.8s
#Average turns per conversation
3.7
3.3
3.2
Table 3: The dataset format. Notations: C_ID= conversation ID, U_ID- =
utterance ID, Prd. Pride, Lov. Love, StartTime and EndTime are in hh:mm:ss, ms format, -> denotes the directional correlation across two tasks.
C.
U
Utterrance
Speaker
Start'
End
Eino.
Sar.
Hom.
Emo--Sen.
Sar.-Hum.
Pri.
Law.
Frone TEna mottee
CL12
U_4
owo bundeed youn
Wei Zhang
00.00:07, 9900
00.00L09,48D
Nea.
Neu.
Nose
Non.
0
Noo.
Non.
buy piece
C_12
U.5
sotu and kill yourself on
Yumo Qie
00.000 10,000
0000212,854
Asg
Neg.
Sar.
Hum.
2
Noo. Noo.
To fulfill the requirements, we choose eighteen famous Chinese TV series as our domain. They consist of multi-speaker conversations with utterances in the forms of text (t), video (v) and audio (a), and cover different genres (viz. comedy, metropolitan opera, drama, crime thriller) and styles (viz. costume, war, idol, history, romance, family, crime). Furthermore, the conversations are extracted from all the episodes of different seasons and therefore cover various topics from daily events to political conflicts (c.f. Iab. in supplementary document).
In order to filter out noisy and irrelevant samples, we partition episodes into short ([ss, 7s), medium (8s, 13s),, long ([1s,, 19s]) and super long ([20s, 25s]) conversations based on the time intervals of video segments, and randomly sample raw videos from each group. As a result, over 3,800 multi-modal videos gathered. They then filtered by the following rules: (1) the video should not contain low-resolution or blank frames; (2) speakers speak clearly in standard mandarin; (3) The speaker's 's face and voice must appear simultaneously and remain for certain period of time; (4) The whole conversation in the video should take place in a single scene; and (5) There should be no ambiguity human annotation (See Section 2.3
After this step, we obtain total number of 3,000 video conversations. We extract all the subtitles and transcripts for each conversation with their respective timestamps through Google cloud transcription service"] and utilize Adobe Premiere Pro") to crop the conversation at the utterance level according the starting and ending timestamps of each utterance. In line with previous studies p 13,, the heuristic constraints are proposed to accomplish the conversation segmentation: the utterances conversation should be in ascending order. (2) The utterances conversation should be from the same episode and scene only.
(I) the timestamps of
After segmentation, obtain utterance- leveel aligned text, audio and visual clips of each conversation. The CMMA dataset contains 3,000 multi-party conversations, 21,795 multi-modal utterances and 185,442 word occurrences (license terms please refer App. 1.3 in Supp.). A conversation has average number of 7.0 utterances and 60.2 words. Each utterance lasts an average 2.6 seconds. More statistics be found in Table
Data Format. Each utterance is uniquely identified by a conversation ID and an utterance ID. Its text, video and audio clips are stored in .CSV, MP4, WAV files. For examples, see Table
hhtn::/cclougggoollccom///ppcccp htnp:///ww..aooorccoooppppttttpt

---

2.2 Label Selection and Annotation
Multi-affect annotations. We annotate each utterance with sentiment, emotion, sareasm and humor labels. For sentiment, apart from the common 3-level annotations (positive, neutral, negative), we present two novel social needs-oriented fine-grained sentiments, pride and romantic love. In psychology and philosophy, pride is defined as a complex secondary sentiment that involves a feeling of deep pleasure or satisfaction derived from one's ' own importance [14.. We consider pride and non-pride as the labels. The triangular theory of love [I5 16 considers romantic love (love for short) as an intense feeling of deep attraction towards another person. identifies four forms of love, i.e., immediate love. growing love, empty love and non-love. We thus take them as love labels For emotion, Ekman's 's six universal emotions [13, viizjooy sadness, anger, disgust. surprise, plus the neutral emotion are selected. We perform 2-1evel annotation for for both sarcasm and humor, ie., sareastic versus non-sarcastic and humor versus non-humor.
Cross-task Relevance. We make the first attempt to annotate the relevance between different affects. Emotion and sentiment are direct expression forms of affect while sareasm and humor are below-the- surface affect types. The challenges for identifying the inter-relatedness between affects mainly reside in the relevance judgment between pairs of affects of the same level, i.e., emotion vs sentiment, and sar- casm us humor. Therefore, we perform human annotations on the relevance judgment of the two affect pairs. Essentially, a 5-level annotation in [-2, -1, O, 1. 2] is used, where the sign stands for whether an affect contributes to the other or the other way around and the absolute value indicates the strength of contribution. For instance, 2 means emotion plays a significant role in sentiment analysis. 0 means the two tasks are irrelevant. Such information can help determine the main and auxiliary tasks, which in- structs the design of the multi-task model structure. Its empirical benefits are validated in Section 4.2
2.3 Human Annotation Procedure
We developed a Java-based interface for human annotation. It contains an utterance's 's video and its speaker and transcript (c.f. Fig.2 in App-A in Supp.). Single-choice questions of affect judgments are presented to annotators. An annotation can click "click to view context" to start watching the video of the target utterance. The interface also contains buttons to submit the annotations, move to the next utterance, clear all annotations on the page.
The annotation procedure consists of two phases: annotation and re-annotation. Specially, recruit nine well-educated volunteers including seven undergraduate and two master's students to take part in data annotation and re-annotation. They all signed on the consent form before the study and were paid an equal $7. $7.5/hour in local currency. Prior to annotation, they received professional guidance" covering the use of the annotation system, the criteria for labeling, human affect-related knowledge, positive and negative examples, etc. We answered further questions from the volunteers regarding the guidance. Then, they were instructed to annotate 100 examples first to strengthen the inter-annotator agreement, which should reach 90% in principle.
Seven annotators were randomly assigned with annotation tasks. Since human affects are usually intertwined, each annotator was asked to perform the complete set of annotations, including six affect judgment tasks and two relevance judgment tasks described in Section 2.2 The gold standard labels of each utterance are determined by majority voting on all human annotations.
The re-annotation phase started when there was strong disagreement among seven annotators (e.g.. the voting result is 3:3:1 or 2:2:2:1). Two new annotators were asked to re-annotate the disputed samples. We added their voting results to the existing annotations to decide the final golden label. If their votes disagreed, we removed the conversation containing the utterance from the dataset.
2.4 Quality Control and Inner Agreement
To guarantee high- annotations, we calculate the average agreement rate among nine annotators via the percent agreement calculation approach [7. The average agreement rates for eight tasks are 88.8%, 71.5%, 86.8%, 94.5% 82.5%, 94.9%, 69.7% and 67. 67.4%. To increase the
'The explanation of motivation of pride and love annotation is provided in App.K in supplementary document. 6 'emotion - sentiment and sareasm - humorr are the main directions. For - B. a positive value means A contributes to B. The guidance and the informed consent available on httpss//iihhuu.ccmmmmmmmmyyyyyyy 2022/hh Dataset.
5

---

positive
sadncss 0.00
Toe abed
0.000
Saecastic
neutral
0.02
0.96
0.02
000
WOA.O00 00000000000
disgust
Soar 100000000001
The
negative
0.04
0.07
0.89
Don--arcaatte
nrutral
poltive-
Angunt
aunea
succusic mon-sarcastic Prodicted label
Prodicted
Prodcted label
(a) Sentiment annotation.
(b) Emotion annotation.
(c) Sarcasm annotation.
immediate
0.000000000.095
True label
humoo
0.92
0.08
True label
pride
growing 0.000 0.00000777
66
empty 0.0000.000000 U.D00.
64
none 0.00200020000
non-bumor
0.03
0.97
non-pride
0.04
0.96
m00
humor rro-humer Prodicted label
pride non-pride Predicted label
Predicted label
(d) Humor annotation.
(c) Pride annotation.
(1) Love annotation.
Figure 2: The confusion matrices show the annotations difference between different labels for six affects.
8050
20009
7050
6020
5000
1000
10003
3000
2000
2003
1000
Auger
Juy
Supese
Foar
Digus
Sadaew
Neutal Sanceatic Hanor
Pride
Nox secoetto
Nou-tauoo
Nae-peide
Non
Posttive
Neotal
Positive Neonrel Negatiie
Figure 3: Sentimental distribution of emotion, sarcasm, humor, pride and love samples.
credibility, computed the Fleiss' kappa score (18. The overall agreement scores of the annotation are x 0.85, x 0.69, x = 0.68, K = 0.85, x = 0.71, K = 0.83, k = 0.62, K = 0.60 respectively, which means the nine participators have reached substantial agreement eight annotation tasks. Compared with other related datasets, we have attained the highest inter-agreement scores on all tasks (e.g., the sentiment kappa of MOSI 0.77, the emotion kappa of MELD is 0.43, the sarcasm kappa of MUSIARD is 0.58). We have recruited the highest number of annotators to enhance the quality")
Moreover, the confusion matrices are calculated over seven annotators' annotations to present the differences between different affect labels. From Figure one could easily distinguish positive from negative sentiment, but it is relatively difficult to tell neutral sentiment apart from either of them. Similarly, Figure (b) supports the above argument with a small distance between neutral and other emotions, especially sadness, joy and surprise. Figure 2 (c) suggests minor confusion between sarcastic and non-sarcastic labels. From Figure 2 (d), (e) and (f), we see that the annotators are able to correctly identify humor, pride and love.
Analysis of CMMA Dataset
Sentiment distribution over other affects. 3 presents the distribution of positive, negutive and neural samples in other affect labels. (a) For emotion samples, negative sentiment takes up
App..B B in Supp. provides systematic comparison between CMMA and other datasets.

---

Strongly forward correlation (2)
11%
10%
13%
Weakly forward correlation (1)
16%
43%
Unrelated
5%
13%
Weakly reversed correlation (-1)
25%
36%
Strongly reversed correlation (-2)
()) Emotion Sentimet.
(b) Sareasm Humor
Figure 4:
The relevance intensity across tasks.
the majority of anger, fear, disgust and sadness samples, at ratio of 50,4%, 67.5%, 79.7% and 55.3%, respectively. 80.0% joy samples are of a positive sentiment, confirming with our intuition that joyful expression implies positive sentiment. Since surprise can be expressed with positive or negative sentiment, positive and negative sentiments have close proportions in surprise samples. Quite surprisingly, about 17.8% samples with neutral emotion have a positive and negative sentiment, which indicates neutral emotion does not necessarily mean neutral sentiment. The results have verified our previous arguments that sentiment and emotion are distinct (See Section []. (b) Positive (29.9% 9%) and negative (26.4%) sentiments take close proportions of sarcastic samples, but still more samples (43.7%) of neutral sentiment. (c) Out of humor samples, positive sentiment has the largest proportion (50.4%) than the other two sentiments. This shows that humor tends to provoke positive feelings. (d) Pride samples consist of almost equal numbers of positive, neutral and negative sentiments. One possible reason is that pride is a complicated affect type, and can appear under positive, negative or neutral sentiment. The distribution of love samples is not plotted due to too few samples in the dataset. To sum up, a close relationship between sentiment and other affects is shown in the distributions.
Affect reateesss Figure iillustrates the relevance intensity between emotion and sentiment, and sareasm and humor. We observe that emotion and sentiment are considered related in 95% samples. The annotators argue that the result of emotion annotation offers great or less help to sentiment judgment in 68% utterances. In the remaining 27% samples, the result of sentiment judgment will help emotion annotation. Similarly, sarcasm and humor are annotated as correlated in 87% samples. The annotators argue that sarcasm judgment tends to offers greater help to humor annotation than humor detection to sarcasm understanding (64% us 23%).. Such annotation helps determine the main and auxiliary tasks in the multi-task learning paradigm (See Section 4.1).
Topic distribution. We generate a word cloud via Jieba to visually present 50+ topics of multi-moda conversations in CMMA dataset, as shown in Figure 5 in App.E. We observe that the conversations cover a broad range of ordinary topics, such as daily life, party, work, family, war, idle chat, love, as well as few relatively specific topies, such salary and kidnapping. The coverage and distribution of topics illustrate that the CMMA dataset meets the rule of "representativeness""
Speaker distribution. CMMA dataset contains around 500 speakers in total. Therefore, we focus solely on leading characters and plot the distributions of different affect types for each of them in Figure 3 (c.f. App. E.2 in Supp.). We observe that positive, joy, sarcastic and humor affects are more likely to happen in the characters of sitcoms. These characters produce plenty of profession-rellee rellted punchlines. Anger and fear emotions thrillers. This implicates that the speaker's 's information is also valuable for affect judgment. The overall topic and speaker distribution and analysis are detailed in App.E in supplementary document.
more frequently the tragic figures of dramas and crime
Experiments and Evaluation
4.1 Experiment Settings
Dataset Split. We randomly split the utterances of CMMA dataset into train, validation and test subsets by 60%. 20%, 20%. The detailed statistics in App.G.
hnnps///iitubbooo///ssyyiiii

---

Context
Query
Text
plaze kill
Textual
Extration
Key
Binnodal Fentures
Context
Dense
Sca.
Video
Vaand Fosturs
Extmetion
Value
Trimodal Features
Dense
Emno
(T.
Attentive
Aonatii Festure Carpetonn
Fasion
Densc
Sar.
Context
(L.A)
Multi Head Atteative
Andio
Fusson
Multi-modal Encoder
Multi-Head Attentive
Dense
Pride
Fasion
Dense
Love
Figure 5: Multi-modal multi-affect joint detection model.
Model architecture. We build model that seeks to classify all the six affect types, i.e., sentiment, emotion, sarcasm, humor, pride and love, with different combinations of input features. Figure 5 presents the multi-task model architecture. The text, video and audio inputs each passed to encoders to extract their features. We consider four neural text encoders, i.e., bidirectional LSTT (BiLSTM), BERT 19 and GPT-2 20 or GPT-3 21 For video, two widely used visual encoders, i.c., EfficientNet [22] and ResNet are selected. For audio encoder, the pre-trained VGGish network are used. For each modality, the encoded utterance is concatenated with its encoded context, and the unimodal contextual features are combined by multi-modal fusion. Specific multi-moda fusion strategies include multi- head attentive fusion, concatenation, addition, element- teiee multiplication and maximum. The obtained multi-modal representation is then passed through task- dense layers for each affect detection task. The labels of all tasks are produced the forward pass, where we set different weights for different tasks.
On top of the multi-task structure, add connections between dense layers of different tasks in the form of weighted sum, base on the overall statistics of the relevance intensity value. For example, if the emotion - sentiment relevance intensity -2, the sentiment dense layer will be directly added to the emotion dense layer. If the emotion - sentiment relevance intensity is -1, the sentiment dense layer will be halved before adding with the emotion dense layer. We adopt precision (P), recall (R), macro-Fl FI (M,-F1) and balanced accuracy (acc) as the evaluation metrics.
4.2 Results and Discussion
We present the experimental results in Table For text encoder, BiL.STM underperforms BERT and GPT on all tasks but pride detection. Between the contextualized encoders, GPT-3 outperforms BERT for sarcasm detection task, and GPT-2 and GPT-3 slightly outperform BERT for love classification task, but are weaker than BERT for other four tasks. Therefore, BERT performs better for most classification tasks. For video encoder, ResNet beats EfficientNett by a large margin. Among the unimodal classifiers, the pre-trained VGGish acoustic encoder performs significantly worse than the best-performed encoders for the other modalities. possibly due the difficulty of extracting acoustic features. For the same reason, Text+ Video is the pess-oeeroreed bi-modal setup, where BERT+ResNet occupies the best performance on almost all tasks. In addition, Audio has an overall better performance than Video+AAdiio Finally, the trimodal setups significantly outperform bimodal setups, where BERT+ResNet++ VGGish performs the best.
The performance gaps between trimodal and bimodal features illustrate the complementary nature of all three modalities. Still, the BERT-encoded textual feature plays major contribution to the overall performance. We posit that the reasons are two fold. First, text tends to directly influence affect
"Snce GPT-3 is not open source, only use it to perform text sentiment analysis, instead of constructing the multi-modal fusion model.

---

Table 4: Comparison of different models.
Model
Video
Audio
Sentiment
B
Emutiion
F1
31.F1
M..FI
BILSTM
30.35
31:22
30174
62.12
56.45
43.05
Text
BERT
55.51
70.87
51.5
GPT2
18.01
48.37
44.81
64.39
GPT3
54.65
54.21
34.43
48.72
47.65
48.08
53.27
54.87
54.06
49.64
47.21
48.49
Valeo
EiiicienNNet
ResNet
42.86
48,92
51.53
45:12
49.40
42.84
96.05
47.65
47.89
61.58
42.16
47.66
57.66
66-77
01.06
57.84
57.75
53.19
58.06
41.84
55.69
52.8
47.78
44.25
Ausho
Viries
41.15
62.15
44.89
35.24
36.70
50.64
42.19
41.54
23.85
34.98
44:31
46.34
SILSTM BILSTM
49.6
92:35
50.33
201.1
19:80
40..10
63.70
37.11
50300
44.67
58.15
50.35
ReaNet
48.77
$1.27
49.30
36.63
48.85
57,49
$0,60
57.17
53.74
42.51
61.7
50.34
Vider
BERT
BERT
RenNer
65447
67.32
73.36
09:35
05:75
68.89
11:18
56.24
61.65
68.54
57.82
44:25
53.74
67:88
35.33
65.00
50.00
66.75
53.39
52.00
66.25
61.9
53.44
58.79
OPPE
FRirienNNT
1111
17117
16.11
01:35
02715
83:45
55.05
50.30
38:17
51.15
51.11
GPP-33
ReaNet
59.09
66.32
60.03
42.17
61.80
45.91
50.55
61.65
55.56
45.75
64.6
$3.56
VideorAndio
ReaNet
VGGish
Votus
40.23
52.47
53.52
30.21
51.62
-8.27
52.12
41:15
62.12
51.04
$1.44
14.30
43.02
$3.55
30.19
56.74
49.06
66.75
42.63
30.80
64.18
65.84
51.75
49.36
BALS
VGUMs
06.7
1935
46.35
05.13
06335
10:32
00735
06.35
30701
41:21
06.15
5036
Aexto
VGGHs
55.35
63:31
50.36
66:22
VGOlish
11.41
5348
66.08
ResNet
76.31
RaSNat
VGGia
74.87
69.34
Thimoob
Table 5: Comparison of different multi-modal fusion strategies.
Trimodal
Sentiment
Emotion
Sarcasm
Humor
Accuracy
Validation
Test
Validation
Test
Validation
Test
Validation
Test
Multi-head Attention
74.81
78.48
72.24
77.09
82.44
85.64
84.31
86.15
Concatenate
76.76
76.31
73.14
76.32
82.22
84.28
85.06
85.88
Add
71.62
77.39
73.33
76.36
82.37
84.86
85.06
82.93
Multiply
69.85
72.22
70.39
73.05
78.77
78.54
80.91
81.31
Maximum
75.95
76.38
74.11
72.47
81.25
83.13
81.66
79.42
understanding, while visual and acoustic signals are higher level of abstraction. Second, BERT can effectively capture the word dependencies and extract contextualized features. The comparison given in Table 5 where multi-head attention performs best for all
of different fusion strategies tasks. The reason is that the attention mechanism is a best fit in that it automatically learns to pay attentions to different modalities for different utterances.
Effect of Conversational Context. We examined the effect conversational context by choosing different context lengths of [0, 1, and construct speaker-aware and topic-aware models on top of the main model architecture. From Table [6 we observe that the model with two contexts yield the best performance, and both speaker- results demonstrate a crucial role of conversational context, which motivates improved conversational context modeling strategies for future work. By simply merging two contexts with the target utterance, the strong model has obtained significant improvement against the previous setup with zero context. We argue that our setup provides strong baseline for multi-affect detection, and may achieve better classification performance if superior attempts on context modeling have been done.
and topic-aware settings beat the vanilla settings. All the
Effect of Cross-task Relevance. We investigate the impact of the relevance between sentiment- emotion, sarcasm-humor for affect detection by jointly detecting the two affects in each pair. We compare our setup with single-task learning (STL.) framework and the standard multi-task learning (S-MTL) paradigm for jointly addressing both tasks. For our relevance-awan models (RaM), the emotion dense layer is added to sentiment layer, while the sarcasm dense layer is halved and added to the humor dense layer, based on the statistics in Figure 4 We show their classification performance in Table 7 We notice that both multi-task models outperform single-task learning framework. Furthermore, our model consistently outperforms the S-MTL setting, which indicates that the intensity value effectively captures the cross-task relevance and brings benefit the multi-task learning paradigm. The success of the simple strategy demonstrates the enormous potential of cross-task relevance annotations. We expect see greater performance gains to multi-affective joint detection by improved leverage of the relevance annotations on a finer-grained level.
Linguistic Insights. We make detailed linguistic analysis in App. L.

---

Table 6: Effect of context, speakers and topics.
Sentiment
Emotion
Sareasm
Humor
Model
Range
M,-FI
Acc
M,-FI
Acc
M,-FI
Acc
M,-FI
Acc
0
73.29
76.31
61.76
76.32
75.23
85.64
75.91
86.
Context
71.36
73.58
65.81
77.97
71.97
84.99
71.01
84,94
2
76.91
78.12
68.79
79.02
75.72
87.76
76.06
87.97
No Speaker
73.29
76.31
61.76
76.32
75.23
85.64
75.91
86.15
Speaker
Speaker Aware
74.39
75.24
64.23
76.39
75.27
87.33
72.06
85.22
No Topic
73.29
76.31
61.76
76.32
75.23
85.64
75.91
86.15
Topic
Topic Aware
76.62
76.71
63.31
78.14
72.63
85.37
76.02
87.63
Table Effect of the relevance between sentiment-cmotionn sarcasm-humee
Sentiment
Emotion
Sarcasm
Humor
Setup
M.-Fi
Acc
M,-Fi
Acc
Ma-FI
Acc
Ma-F1
Acc
STL
71.17
72.22
59.75
72.47
71.97
83.13
73.21
79.41
5 MIL: : Emo.
71.61
72.85
61.76
76.32
MIL Sen.
73.14
75.52
60.12
73.39
RaM
74.31
79.55
61.76
77.09
S MTL: Sar.
74.22
85.64
73.77
80.46
MTL : Hum.
72.27
83.84
74.51
85.42
RaM
75.23
85.64
75.91
86.15
5 Conclusions and Future work
Few works (including the recent large language models) have set foot in multi-affect joint detection in conversations, largely due to the lack of multi-modal conversation datasets with multi-affect anno- tations. We have filled this gap by proposing CMMA, the first multi-modal multi-affect conversation dataset. CMMA consists of 21,795 multi-modal utterances from 3,000 multi-part mulli-party conversations. Apart from rich affect labels including sentiment, emotion, sarcasm and humor, the dataset con- tains annotation relevance between affect types. We have performed comprehensive qualitative and quantitative studies for analyzing the dataset, and presented a range baselines to evaluate the potential of CMMA. The results demonstrate the quality of the dataset and indicate the need of novel investigations in models in multi-modal multi-affect joint detection in conversations.
Limitations and Potential Risks. (1) The created dataset does not reach the requirement of "bal- ance". The CMMA dataset also has its unique characteristics and limitations that may affect generalization of results. The dataset's focus Chinese conversations from TV shows may limit the direct application of our findings to other languages. We encourage future researchers to consider cross-linguistic validation. (2) The created dataset does not reach the requirement of "balance". The implicit bias (e.g., age-related bias, profession-relater bias) may be introduced. In addition, the potential biases that may arise when sourcing content from TV shows, where the emotion expressed by the actors may be exaggerated. We will attempt to design a bias mitigating approach to check and alleviate the impact of such biases in the future work. The researchers should understand these limitations and take them into careful consideration.
10

---

Acknowledgements
This work is partly supported by a Project of Strategic Importance grant of The Hong Kong Poly- technic University (project no. 1-ZE2Q). This paper is also partly supported by grant under Hong Kong RGC Theme-based Research Scheme (project no. T45-401/22-N), This work is also supported by National Science Foundation of China under grant No. 62006212, Fellowship from the China Postdoctoral Science Foundation (2023M733907). This work funded in part by Beijing Municipal Natural Science Foundation (grant no: 4222036).
References
[1] Xincheng Ju, Dong Zhang, Rong Xiao, Junhui Li, Shoushan Li, Min Zhang, and Guodong Zhou. Joint multi-modal aspect-sentiment analysis with auxiliary cross- modsl relation detection. In Proceedings of the 202 Conference on Empirical Methods in Natural Language Processing, pages 4395 4405, 2021.
[2] Lukas Stappen, Eva-Maria Meßner, Erik Cambria, Guoying Zhao, and Björn W Schuller. Muse 2021 challenge: multimodal emotion, sentiment, ahysiological-emotiie and stress detection. In Proceedings of the 29th ACM International Conference on Multimedia, pages 2021.
[3] Yaochen Liu, Yazhou Zhang. Qiuchi Li, Benyou Wang, and Dawei Song. What does your smile mean? jointly detecting multi-modal sarcasm and sentiment using quantum probability. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 871-880,
2021.
[4] Badri Patro, Mayank Lunayach, Deepankar Srivastava, Hunar Singh, Vinay Namboodiri, et al. Multimodal humor dataset: Predicting laughter tracks for sitcoms. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 576-585, 2021.
[5] Md Kamrul Hasan, Wasifur Rahman, Amir Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, et al. Ur-funny: A multimodal language dataset for understanding humor. arXiv preprint 1XXi::99040060 2019.
[6] Md Shad Akhtar, Dushyant Singh Chauhan, Deepanway Ghosal, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya. Multi-task learning for multi-modal emotion recognition and sentiment analysis. urXiv preprint arXiv: urxivv:9955.0.0 000.99955005222 2019.
[7] Yazhou Zhang, Dawei Song, Xiang Li, Peng Zhang, Panpan Wang, Lu Rong, Guangliang Yu, and Bo Wang- quantum-like multimodal network framework for modeling interaction dynamics multiparty conversational sentiment analysis. Information Fusion, 62: 12:14331,4 2020.
[8] Shraman Pramanick, Aniket Roy, and Vishal M Patel. Multimodal learning using optimal transport for sarcasm and humor detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3930-3940, 2022.
[9] Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversa- tions. arXiv preprint arXiv: aeeti:810.00000 2018.
[10] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan, Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4))335-359 2008.
[11] Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann. Rada Mihalcea, and Soujanya Poria. Towards multimodal sarcasm detection (an _obvioosll__ perfect paper). arXiv preprint arXiv:1906.01815 2019.
[12] John Sinclair. Corpus and text-basic principles. Developing linguistic corpora: A guide to good practice, 92: 1-11 2005.

---

[13] Mauajama Firdaus, Hardik Chauban, Asif Ekbal, and Pushpak Bhattacharyya. Meisd: multi- modal multi-label emotion, intensity and sentiment dialogue dataset for emotion recognition and sentiment analysis in conversations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4441-4453, 2020.
[14] Jessica Tracy and Richard W Robins. The nature of pride. The self-conscious emotions:
Theory and research, pages 263-282, 2007.
[15] Robert Sterbberg. A triangular theory of love. Psychological review, 93(2): 19, 1986.
[16] Harry Harlow. The nature of love. American psychologist, 13(12);673, 1958.
[17] Ronald Hunt. Percent agreement, pearson's correlation, and kappa as measures of inter- examiner reliability. Journal of Dental Research, 65(2):128-130, 1986.
[18] Joseph L Fleiss and Jacob Cohen. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613 --9,11 1973.
[19] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv18810008805, 2018.
[20] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[21] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:18777-1901, 2020.
[22] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International conference on machine learning, pages 1105-6114. PMLR, 2019.
12


Reviews:
[{"review_id": "QzCYyXTAWg", "reviewer": "Reviewer_GqNB", "summary": "", "questions": "", "limitations": "The limitations are well described", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "As mentioned above in summary and contributions", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "X5XGIzdsIB", "reviewer": "Reviewer_4J35", "summary": "", "questions": "", "limitations": "Sourcing content from television shows may introduce a number of biases, and it's not hard to imagine that the effects of actors portraying emotions may be either ineffective or exaggerated. This may be mitigated by the annotation process where trained annotators are marking the perceived emotions. However, it is important to acknowledge the artificiality of the entertainment medium and the potential for distributional shift that might make classifiers trained on this material less effective in other applications.\n\nIn addition, it is possible that there may be related effects concerning the coverage of topics due to both government or entertainment companies censorship. This would likely not invalidate the work here, but it is something to keep in mind. The authors do acknowledge the bias towards professionals and particular age which is a reflection of the biases in the entertainment industry.\n\n", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "This paper is well written and the methodology used to develop the resources is well explained. Significant labor was used to process the data with multiple annotations, and it with the wealth of material, it should be feasible for even groups with limited resources to perform effective research (for example. building text-only classifiers for teams that cannot effectively process speech or video should be quite easy.)\n\nThe initial experiments use a good sampling of current techniques.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Cy8BCIAwua", "reviewer": "Reviewer_hf9n", "summary": "", "questions": "", "limitations": "Limitations described in the paper are brief and could be expanded to include details on potential biases, limitations to generalizations, and others. One additional point would be that the data are collected from Chinese TV-shows, so they reflect scripted communicative exchanges. ", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Data are rich and multi-faceted. For example, both affect in valence and arousal terms, as well as individual discrete emotions, such as anger or joy, are captured, in addition to several other aspects of affect and emotion that are less commonly available. The strength of the dataset lies in providing these data together, for the same dataset. I can see this versatile dataset offering several exciting opportunities for further studies.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "tZGuhY8zwh", "reviewer": "Reviewer_LR2c", "summary": "", "questions": "", "limitations": "The authors mentioned a potential limitation of the dataset, which is the possibility of introducing bias. \nIn addition to that, there are other limitations worth considering, as mentioned above.\n", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The proposed dataset encompasses the most comprehensive aspects for conversational affection extraction.\n- The authors have taken care to provide a detailed description of the data collection and processing specifications, ensuring the quality and integrity of the data.\n- Notably, the annotation process involved seven annotators for each sample, and the satisfactory agreement score indicates a high level of consistency in the annotation.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper introduces a new dataset of Chinese TV series, annotated with a variety of affection categories. Reviewers agree on the usefulness of the dataset, however, multiple reviewers, as well as the authors themselves raise concerns about the biased data, due to compilation of acted speech. I agree with this concern, and suggest extending this data with natural (i.e., not acted) conversational speech as well.
Decision: Accept (Poster)