SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Speaker distribution. CMMA dataset contains around 500 speakers in total. Therefore, we focus solely on leading characters and plot the distributions of different affect types for each of them in Figure 3 (c.f. App. E.2 in Supp.). We observe that positive, joy, sarcastic and humor affects are more likely to happen in the characters of sitcoms. These characters produce plenty of profession-related punchlines. Anger and fear emotions occur more frequently on the tragic figures of dramas and crime thrillers. This implicates that the speaker's information is also valuable for affect judgment. The overall topic and speaker distribution and analysis are detailed in App.E in supplementary document.
4 Experiments and Evaluation
Dataset Split. We randomly split the utterances of CMMA dataset into train, validation and test subsets by  $60\%$ ,  $20\%$ ,  $20\%$ . The detailed statistics in App.G.
$^{9}$ https://github.com/fxsjy/jieba
Figure 5: Multi-modal multi-affect joint detection model.
Model architecture. We build a model that seeks to classify all the six affect types, i.e., sentiment, emotion, sarcasm, humor, pride and love, with different combinations of input features. Figure 5 presents the multi-task model architecture. The text, video and audio inputs are each passed to an encoders to extract their features. We consider four neural text encoders, i.e., bidirectional LSTM (BiLSTM), BERT [19] and GPT-2 [20] or GPT-3 $^{10}$  [21]. For video, two widely used visual encoders, i.e., EfficientNet [22] and ResNet are selected. For audio encoder, the pre-trained VGGish network are used. For each modality, the encoded utterance is concatenated with its encoded context, and the unimodal contextual features are combined by multi-modal fusion. Specific multi-modal fusion strategies include multi-head attentive fusion, concatenation, addition, element-wise multiplication and maximum. The obtained multi-modal representation is then passed through task-specific dense layers for each affect detection task. The labels of all tasks are produced in the forward pass, where we set different weights for different tasks.
On top of the multi-task structure, we add connections between dense layers of different tasks in the form of weighted sum, base on the overall statistics of the relevance intensity value. For example, if the emotion  $\rightarrow$  sentiment relevance intensity is -2, the sentiment dense layer will be directly added to the emotion dense layer. If the emotion  $\rightarrow$  sentiment relevance intensity is -1, the sentiment dense layer will be halved before adding with the emotion dense layer. We adopt precision (P), recall (R), macro-  $F1$  (M_a-F1) and balanced accuracy (acc) as the evaluation metrics.
We present the experimental results in Table 4. For text encoder, BiLSTM underperforms BERT and GPT on all tasks but pride detection. Between the contextualized encoders, GPT-3 outperforms BERT for sarcasm detection task, and GPT-2 and GPT-3 slightly outperform BERT for love classification task, but are weaker than BERT for other four tasks. Therefore, BERT performs better for most classification tasks. For video encoder, ResNet beats EfficientNet by a large margin. Among the unimodal classifiers, the pre-trained VGGish acoustic encoder performs significantly worse than the best-performed encoders for the other modalities, possibly due to the difficulty of extracting acoustic features. For the same reason, Text+Video is the best-performed bi-modal setup, where BERT+ResNet occupies the best performance on almost all tasks. In addition, Text+Audio has an overall better performance than Video+Audio. Finally, the trimodal setups significantly outperform bimodal setups, where BERT+ResNet+VGGish performs the best.
The performance gaps between trimodal and bimodal features illustrate the complementary nature of all three modalities. Still, the BERT-encoded textual feature plays a major contribution to the overall performance. We posit that the reasons are two fold. First, text tends to directly influence affect
<sup>10</sup>Since GPT-3 is not open source, we only use it to perform text sentiment analysis, instead of constructing the multi-modal fusion model.
Table 4: Comparison of different models.
Model	Text	Video	Audio	Sentiment	Emotion	Sarcasm	Humor
P	R	\(M_{n}-F1\)	P	R	\(M_{n}-F1\)	P	R	\(M_{n}-F1\)	P	R	\(M_{n}-F1\)
Text	BiLSTM	-	-	50.36	51.22	50.74	41.52	62.12	44.74	56.43	50.64	52.29	43.69	55.9	49.05
BERT	-	-	56.77	55.51	54.89	51.85	70.87	56.18	54.56	53.88	53.61	51.5	56.94	54.08
GPT-2	-	-	53.88	58.01	54.35	45.33	44.37	45.21	51.41	53.48	52.42	44.81	64.39	52.85
GPT-3	-	-	54.66	54.21	54.43	48.72	47.65	48.18	53.27	54.87	54.06	49.84	47.21	48.49
Video	-	EfficientNet	-	42.86	45.12	42.84	38.08	61.58	42.18	46.77	61.66	53.19	38.06	52.8	44.23
-	ResNet	-	48.92	51.53	49.40	47.65	47.89	47.66	57.66	57.84	57.75	41.84	55.69	47.78
Audio	-	-	VGGish	41.15	62.12	44.89	33.24	26.70	30.64	42.19	43.54	42.85	34.98	44.81	46.84
Text+Video	BiLSTM	EfficientNet	-	49.68	52.33	50.20	40.51	39.69	40.10	45.70	57.17	50.80	44.67	58.18	50.53
BiLSTM	ResNet	-	48.77	51.27	49.30	36.68	48.86	37.49	50.69	57.17	53.74	42.51	61.7	50.34
BERT	EfficientNet	-	65.47	69.88	66.75	41.16	61.68	44.29	55.74	58.35	57.02	53.59	61.9	58.44
BERT	ResNet	-	67.32	73.36	68.89	56.24	68.54	57.82	67.84	65.69	66.75	52.03	66.25	58.29
GPT-2	EfficientNet	-	58.13	64.24	59.17	38.08	61.58	42.18	45.45	56.05	50.20	46.02	63.35	53.31
GPT-2	ResNet	-	59.09	66.32	60.03	42.17	61.80	45.91	50.55	61.65	55.56	45.75	64.6	53.56
Video+Audio	-	EfficientNet	VGGish	49.22	50.21	48.27	41.15	62.12	44.89	38.59	59.19	46.73	40.89	64.18	49.96
-	ResNet	VGGish	52.47	53.52	51.62	52.12	51.04	51.44	42.12	58.74	49.06	42.63	65.84	51.75
Text+Audio	BiLSTM	-	VGGish	46.97	49.55	46.84	43.13	64.83	46.82	40.85	66.59	50.64	42.23	64.18	50.94
BERT	-	VGGish	54.41	55.25	55.74	46.93	63.31	50.36	43.57	68.39	53.22	48.99	65.22	55.95
GPT-2	-	VGGish	51.41	53.48	52.42	45.23	66.98	49.44	41.52	69.73	52.05	45.29	63.77	52.97
Text+Video+Audio	BERT	EfficientNet	VGGish	69.59	73.98	71.12	53.03	74.37	57.36	69.38	65.02	67.13	63.76	69.57	66.53
BERT	ResNet	VGGish	71.64	76.31	73.29	56.71	76.32	61.76	76.28	74.22	75.23	76.47	75.36	75.91
GPT-2	EfficientNet	VGGish	65.66	69.47	66.86	47.06	73.71	51.49	58.95	62.78	60.8	58.16	62.73	60.36
GPT-2	ResNet	VGGish	71.76	74.87	72.88	52.09	73.82	56.17	74.44	67.26	70.67	65.80	73.29	69.34
Trimodal vs Bimodal (%)	-	-	+6.6	+4.0	+6.4	+0.8	+11.3	+6.8	+12.4	+6.4	+12.6	+42.6	+13.7	+29.6
Table 5: Comparison of different multi-modal fusion strategies.
Trimodal Accuracy	Sentiment	Emotion	Sarcasm	Humor
Validation	Test	Validation	Test	Validation	Test	Validation	Test
Multi-head Attention	74.81	78.48	72.24	77.09	82.44	85.64	84.31	86.15
Concatenate	76.76	76.31	73.14	76.32	82.22	84.28	85.06	85.88
Add	71.62	77.39	73.33	76.36	82.37	84.86	85.06	82.93
Multiply	69.85	72.22	70.39	73.05	78.77	78.54	80.91	81.31
Maximum	75.95	76.38	74.11	72.47	81.25	83.13	81.66	79.42
understanding, while visual and acoustic signals are on a higher level of abstraction. Second, BERT can effectively capture the word dependencies and extract contextualized features. The comparison of different fusion strategies are given in Table 5, where multi-head attention performs best for all tasks. The reason is that the attention mechanism is a best fit in that it automatically learns to pay attentions to different modalities for different utterances.
Effect of Conversational Context. We examined the effect of conversational context by choosing different context lengths of  $[0,1,2]$  and construct speaker-aware and topic-aware models on top of the main model architecture. From Table 6, we observe that the model with two contexts yield the best performance, and both speaker-aware and topic-aware settings beat the vanilla settings. All the results demonstrate a crucial role of conversational context, which motivates improved conversational context modeling strategies for future work. By simply merging two contexts with the target utterance, the strong model has obtained significant improvement against the previous setup with zero context. We argue that our setup provides a strong baseline for multi-affect detection, and it may achieve better classification performance if superior attempts on context modeling have been done.
Effect of Cross-task Relevance. We investigate the impact of the relevance between sentiment-emotion, sarcasm-humor for affect detection by jointly detecting the two affects in each pair. We compare our setup with single-task learning (STL) framework and the standard multi-task learning (S-MTL) paradigm for jointly addressing both tasks. For our relevance-aware models (RaM), the emotion dense layer is added to sentiment layer, while the sarcasm dense layer is halved and added to the humor dense layer, based on the statistics in Figure 4. We show their classification performance in Table 7. We notice that both multi-task models outperform single-task learning framework. Furthermore, our model consistently outperforms the S-MTL setting, which indicates that the intensity value effectively captures the cross-task relevance and brings benefit to the multi-task learning paradigm. The success of the simple strategy demonstrates the enormous potential of cross-task relevance annotations. We expect to see greater performance gains to multi-affective joint detection by improved leverage of the relevance annotations on a finer-grained level.
Linguistic Insights. We make detailed linguistic analysis in App. L.
Table 6: Effect of context, speakers and topics.
Model	Range	Sentiment	Emotion	Sarcasm	Humor
Ma-F1	Acc	Ma-F1	Acc	Ma-F1	Acc	Ma-F1	Acc
Context	0	73.29	76.31	61.76	76.32	75.23	85.64	75.91	86.15
1	71.36	73.58	65.81	77.97	71.97	84.99	71.01	84.94
2	76.91	78.12	68.79	79.02	75.72	87.76	76.06	87.97
Speaker	No Speaker	73.29	76.31	61.76	76.32	75.23	85.64	75.91	86.15
Speaker Aware	74.39	75.24	64.23	76.39	75.27	87.33	72.06	85.22
Topic	No Topic	73.29	76.31	61.76	76.32	75.23	85.64	75.91	86.15
Topic Aware	76.62	76.71	63.31	78.14	72.63	85.37	76.02	87.63
Table 7: Effect of the relevance between sentiment-emotion / sarcasm-humor.
Setup	Sentiment	Emotion	Sarcasm	Humor
Ma-F1	Acc	Ma-F1	Acc	Ma-F1	Acc	Ma-F1	Acc
STL	71.17	72.22	59.75	72.47	71.97	83.13	73.21	79.41
S-MTL:Emo.	71.61	72.85	61.76	76.32	-	-	-	-
S-MTL:Sen.	73.14	75.52	60.12	73.39	-	-	-	-
RaM	74.31	79.55	61.76	77.09	-	-	-	-
S-MTL: Sar.	-	-	-	-	74.22	85.64	73.77	80.46
S-MTL: Hum.	-	-	-	-	72.27	83.84	74.51	85.42
RaM	-	-	-	-	75.23	85.64	75.91	86.15
5 Conclusions and Future work
Few works (including the recent large language models) have set foot in multi-affect joint detection in conversations, largely due to the lack of multi-modal conversation datasets with multi-affect annotations. We have filled this gap by proposing CMMA, the first multi-modal multi-affect conversation dataset. CMMA consists of 21,795 multi-modal utterances from 3,000 multi-party conversations. Apart from rich affect labels including sentiment, emotion, sarcasm and humor, the dataset contains annotation relevance between affect types. We have performed comprehensive qualitative and quantitative studies for analyzing the dataset, and presented a range of baselines to evaluate the potential of CMMA. The results demonstrate the quality of the dataset and indicate the need of novel investigations in models in multi-modal multi-affect joint detection in conversations.
Limitations and Potential Risks. (1) The created dataset does not reach the requirement of "balance". The CMMA dataset also has its unique characteristics and limitations that may affect the generalization of our results. The dataset's focus on Chinese conversations from TV shows may limit the direct application of our findings to other languages. We encourage future researchers to consider cross-linguistic validation. (2) The created dataset does not reach the requirement of "balance". The implicit bias (e.g., age-related bias, profession-related bias) may be introduced. In addition, the potential biases that may arise when sourcing content from TV shows, where the emotion expressed by the actors may be exaggerated. We will attempt to design a bias mitigating approach to check and alleviate the impact of such biases in the future work. The researchers should understand these limitations and take them into careful consideration.
This work is partly supported by a Project of Strategic Importance grant of The Hong Kong Polytechnic University (project no. 1-ZE2Q). This paper is also partly supported by a grant under Hong Kong RGC Theme-based Research Scheme (project no. T45-401/22-N). This work is also supported by National Science Foundation of China under grant No. 62006212, Fellowship from the China Postdoctoral Science Foundation (2023M733907). This work is funded in part by Beijing Municipal Natural Science Foundation (grant no: 4222036).
[1] Xincheng Ju, Dong Zhang, Rong Xiao, Junhui Li, Shoushan Li, Min Zhang, and Guodong Zhou. Joint multi-modal aspect-sentiment analysis with auxiliary cross-modal relation detection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4395-4405, 2021.
[2] Lukas Stappen, Eva-Maria Meßner, Erik Cambria, Guoying Zhao, and Björn W Schuller. Muse 2021 challenge: multimodal emotion, sentiment, physiological-emotion, and stress detection. In Proceedings of the 29th ACM International Conference on Multimedia, pages 5706-5707, 2021.
[3] Yaochen Liu, Yazhou Zhang, Qiuchi Li, Benyou Wang, and Dawei Song. What does your smile mean? jointly detecting multi-modal sarcasm and sentiment using quantum probability. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 871-880, 2021.
[4] Badri N Patro, Mayank Lunayach, Deepankar Srivastava, Hunar Singh, Vinay P Namboodiri, et al. Multimodal humor dataset: Predicting laughter tracks for sitcoms. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 576-585, 2021.
[5] Md Kamrul Hasan, Wasifur Rahman, Amir Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis-Philippe Morency, et al. Ur-funny: A multimodal language dataset for understanding humor. arXiv preprint arXiv:1904.06618, 2019.
[6] Md Shad Akhtar, Dushyant Singh Chauhan, Deepanway Ghosal, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya. Multi-task learning for multi-modal emotion recognition and sentiment analysis. arXiv preprint arXiv:1905.05812, 2019.
[7] Yazhou Zhang, Dawei Song, Xiang Li, Peng Zhang, Panpan Wang, Lu Rong, Guangliang Yu, and Bo Wang. A quantum-like multimodal network framework for modeling interaction dynamics in multiparty conversational sentiment analysis. Information Fusion, 62:14-31, 2020.
[8] Shraman Pramanick, Aniket Roy, and Vishal M Patel. Multimodal learning using optimal transport for sarcasm and humor detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3930-3940, 2022.
[9] Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508, 2018.
[10] Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335-359, 2008.
[11] Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. Towards multimodal sarcasm detection (an _obviously_ perfect paper). arXiv preprint arXiv:1906.01815, 2019.
[12] John Sinclair. Corpus and text-basic principles. Developing linguistic corpora: A guide to good practice, 92:1-16, 2005.
[13] Mauajama Firdaus, Hardik Chauhan, Asif Ekbal, and Pushpak Bhattacharyya. Meisd: a multimodal multi-label emotion, intensity and sentiment dialogue dataset for emotion recognition and sentiment analysis in conversations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4441-4453, 2020.
[14] Jessica L Tracy and Richard W Robins. The nature of pride. The self-conscious emotions: Theory and research, pages 263-282, 2007.
[15] Robert J Sternberg. A triangular theory of love. Psychological review, 93(2):119, 1986.
[16] Harry F Harlow. The nature of love. American psychologist, 13(12):673, 1958.
[17] Ronald J Hunt. Percent agreement, pearson's correlation, and kappa as measures of interexaminer reliability. Journal of Dental Research, 65(2):128-130, 1986.
[18] Joseph L Fleiss and Jacob Cohen. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613-619, 1973.
[19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[20] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
[21] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[22] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114. PMLR, 2019.