Friends-MMC: A Dataset for Multi-modal Multi-party Conversation
  Understanding
Authors: name: Yueqian Wang; name: Xiaojun Meng; name: Yuxuan Wang; name: Jianxin Liang; name: Qun Liu; name: Dongyan Zhao
Published: 2024-12-23 05:32:48

Abstract: Multi-modal multi-party conversation (MMC) is a less studied yet important
topic of research due to that it well fits real-world scenarios and thus
potentially has more widely-used applications. Compared with the traditional
multi-modal conversations, MMC requires stronger character-centered
understanding abilities as there are many interlocutors appearing in both the
visual and textual context. To facilitate the study of this problem, we present
Friends-MMC in this paper, an MMC dataset that contains 24,000+ unique
utterances paired with video context. To explore the character-centered
understanding of the dialogue, we also annotate the speaker of each utterance,
the names and bounding bboxes of faces that appear in the video. Based on this
Friends-MMC dataset, we further study two fundamental MMC tasks: conversation
speaker identification and conversation response prediction, both of which have
the multi-party nature with the video or image as visual context. For
conversation speaker identification, we demonstrate the inefficiencies of
existing methods such as pre-trained models, and propose a simple yet effective
baseline method that leverages an optimization solver to utilize the context of
two modalities to achieve better performance. For conversation response
prediction, we fine-tune generative dialogue models on Friend-MMC, and analyze
the benefits of speaker information. The code and dataset is publicly available
at https://github.com/yellow-binary-tree/Friends-MMC and thus we call for more
attention on modeling speaker information when understanding conversations.