MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation
Year: 2023

Abstract:
Responding with multi-modal content has been recognized as an essential capability for an intelligent conversational agent. In this paper, we introduce the MMDialog dataset to facilitate multi-modal conversation better. MMDialog is composed of a curated set of 1.08 million real-world dialogues with 1.53 million unique images across 4,184 topics. MMDialog has two main and unique advantages. First, it is the largest multi-modal conversation dataset by the number of dialogues by 88x. Second, it contains massive topics to generalize the open domain. To build an engaging dialogue system with this dataset, we propose and normalize two response prediction tasks based on retrieval and generative scenarios. In addition, we build two baselines for the above tasks with state-of-theart techniques and report their experimental performance. We also propose a novel evaluation metric MM-Relevance to measure the multi-modal responses. Our dataset is available in https://github.com/victorsungo/ MMDialog.

heading: 1 Introduction
text: Empowering machines to converse like humans is a long-cherished goal of AI community, and there is growing interest in developing open-domain conversational agents (Li et al., 2017a; Gao et al., 2018; Ghazvininejad et al., 2018; Zhou et al., 2018a; OpenAI, 2022; Thoppilan et al., 2022). To usher machines into the real visual physical world of human, it is a desirable trait of conversational agents to understand, perceive, and respond appropriately to multi-modality contexts beyond pure text (Das et al., 2017; Mostafazadeh et al., 2017; Shuster
∗ Work done during the internship at Microsoft Research Asia. Corresponding author: caxu@microsoft.com and puzhao@microsoft.com
et al., 2020; Sun et al., 2022), which is similar to communicating through messenger tools (e.g., Facebook, WhatsApp, and WeChat) in reality.
Yet existing approaches to building multi-modal dialogue systems are primarily data-driven, requiring the collection of a large-scale dataset first. To facilitate this line of research, the community emerges a few dialogue datasets incorporating visual information (Meng et al., 2020; Wang et al.,
7348
2021; Zang et al., 2021; Zheng et al., 2022). For example, Visual Dialog (Das et al., 2017) is set up for visual question answering involving image inputs. IGC (Mostafazadeh et al., 2017) and ImageChat (Shuster et al., 2020) are constructed in a crowd-sourcing method in which annotators are employed to chat about given images only. MMChat (Zheng et al., 2022) initiates the conversation with images but does not constrain the topic of subsequent responses. OpenViDial (Meng et al., 2020; Wang et al., 2021) cuts out the image frames and captions in the video as dialogue. PhotoChat (Zang et al., 2021) is also built via crowd-sourcing and contains photo sharing in conversations.
Despite the diversity of multi-modal dialogue corpora, these datasets still have limitations: i) Except for PhotoChat, the responses in the above datasets are all plain text. ii) Visual Dialog, IGC and Image-Chat are all defined as limited scenarios, focusing only on the QA or conversation based on the topics in given images. iii) OpenViDial is mined from movies and TV series. The paired frame-caption is far from real-world dialogue. iv) PhotoChat is the closest to real-world multi-modal dialogue. However, it is still limited by small scale, only having single-modal responses (text or image), and lack of domain diversity, impeding further explorations on multi-modal dialogue modeling.
To address the aforementioned issues, we present MMDialog, a large-scale multi-turn dialogue dataset containing multi-modal open-domain conversations derived from real-world humanhuman interactions in an online social media platform. Comprising 1.08M dialogue sessions and 1.53M associated images, MMDialog boasts an average of 2.59 images per dialogue session, which can appear anywhere at any conversation turn. Fig-
ure 1 depicts an example of human conversations in our MMDialog. To the best of our knowledge, this is the first million-scale open-domain multimodal dialogue corpus. We anticipate that the large amount of dialogues and images can shed light on this line of research. We summarize the main characteristics of multi-modal dialogue datasets in Table 1.
Furthermore, we define the multi-modal response generation and retrieval tasks based on MMDialog that are essential for building a more engaging multi-modal dialogue agent. We also build baseline models and conduct several analyses of their performance. For the generative task, we follow Sun et al. (2022) and implement the models for multi-modal response generation. For the retrieval task, we also propose a CLIP-based retrieval model inspired by Zang et al. (2021). Evaluation results on MMDialog demonstrate that our designed baselines can achieve considerable performance on generation and retrieval tasks of both modalities.
Since the modality orders and amounts of predicted responses may not be aligned with the ground-truth responses, it is difficult for us to evaluate the response quality comprehensively and reasonably using single-modal metrics (e.g., BLEU, PPL, FID, Recall@K, etc.). To tackle the above challenges, we propose a novel evaluation metric called MM-Relevance, which performs visuallanguage matching based on the large-scale pretrained multi-modal CLIP model (Radford et al., 2021). Experiments on both retrieval and generative multi-modal dialog systems indicate that MMRelevance outperforms existing single-modal metrics in terms of correlation with human judgments.
Contributions of this work are four-fold: 1. We construct a novel multi-turn dialogue
dataset MMDialog that contains 1.08M multimodal open-domain conversations and 1.53M associated images derived from social media. To the best of our knowledge, this is the first million-scale multi-turn open-domain multimodal dialogue corpus. 2. We propose two benchmark tasks including generative and retrieval scenarios on MMDialog that are essential for building more engaging multi-modal dialogue systems. 3. We design two baselines for corresponding tasks to promote future research on this dataset and achieve considerable performance. 4. We propose a novel evaluation metric MMRelevance measuring the relevance between generated multi-modal response and ground truth. It can specifically mitigate the modal misalignment issues.

heading: 2 Related Works

heading: 2.1 Multi-Modal Dialogue Datasets
text: Recently, there emerge several multi-modal dialogue datasets. Das et al. (2017) introduce the task of Visual Dialog, which requires an AI agent to hold a meaningful dialogue with humans in natural, conversational language about visual content. Mostafazadeh et al. (2017) propose IGC, which contains 4K dialogues where each includes an image with a textual description, along with the questions and responses around the image. However, IGC is usually used for evaluation due to its small scale. Shuster et al. (2020) release ImageChat that is larger than IGC and consists of 202K image-grounded dialogues. However, the above three datasets were created by asking the crowd workers to talk about a shared image to generate the conversation. Therefore, the utterances are often triggered and grounded by these images. In contrast, human daily communication utterances are not always image-related (Zheng et al., 2022), which retain gaps with open-domain multi-modal conversation scenarios. Then, other groups of works proposed to derive the images from the multiturn conversations: Meng et al. (2020); Wang et al. (2021) construct OpenViDial 1.0/2.0 by directly extracting dialogues and their visual contexts from movies and TV series. Lee et al. (2021) also build a multi-modal dialogue dataset by replacing the selected utterances with retrieved relevant images. However, although these corpora were constructed from open-domain conversations with images, they
did not originate from a real multi-modal conversation scenario. Therefore, recently some researchers begin to introduce real human-human conversations. Zang et al. (2021) create the first humanhuman dialogue dataset with photo-sharing acts via crowd-sourcing. Zheng et al. (2022) collect multimodal dialogues from real conversations on social media. Nevertheless, they were still limited by their small scale or lack of domain diversity, which may hinder further explorations on multi-modal dialogue modeling. To address the above issue, we make the first attempt to construct a million-scale multi-turn dialogue dataset, namely MMDialog, derived from social media and conduct data filtering and post-processing elaborately.

heading: 2.2 Open-Domain Conversation
text: Open-domain dialogue systems can converse on a broad range of topics. Users often do not have any specific goal when participating in open-domain interactions. Early on, several open-domain dialog datasets are constructed from social media such as Twitter, Weibo and Reddit. Advanced datasets including (Ritter et al., 2010; Wang et al., 2013; Sordoni et al., 2015; Shang et al., 2015; Wu et al., 2017; Li et al., 2017b; Zhang et al., 2018b; Adiwardana et al., 2020). Later, researchers begin to incorporate knowledge into conversations (Zhang et al., 2018a; Dinan et al., 2018; Zhou et al., 2018b; Gopalakrishnan et al., 2019; Qin et al., 2019; Moon et al., 2019; Rashkin et al., 2019; Tuan et al., 2019; Song et al., 2020). Meanwhile, there emerge several advanced models based on these corpora including generative ones (Zhang et al., 2020; Shuster et al., 2022; OpenAI, 2022) and retrieval-based ones (Wu et al., 2017; Whang et al., 2021; Han et al., 2021; Feng et al., 2022).

heading: 2.3 Multi-Modal Dialogue Modeling
text: Based on the multi-modal dialogue datasets, many advanced works have been proposed. Several works (Niu et al., 2019; Gan et al., 2019; Qi et al., 2020) investigate how to escalate the performance of conversational agents in image-grounded dialogue. Afterward, researchers (Yang et al., 2021; Liang et al., 2021) explore enriching textual expression of generated dialogue responses through associative vision scenes. Zang et al. (2021) propose two tasks, including photo-sharing intent prediction to predict whether model should intend to share the photo in the next dialogue turn and a dialogue-based image retrieval task to retrieve the
most proper photo given the dialogue context. They also propose a dual-encoder model that uses object labels to encode image features. However, the authors do not conduct textual response retrieval tasks. Zheng et al. (2022) propose a Seq2Seq based multi-modal dialogue generation model. However, it only generates plain textual responses, which is not in line with the open domain multi-modal response generation scenario. Recently, Sun et al. (2022) make the first attempt to build a multi-modal dialogue response generation model named Divter that can effectively understand multi-modal dialogue context and generate informative text and high-resolution images. In this paper, we adapt Divter to our multi-modal response generation settings and extend the dual-encoder (Zang et al., 2021) to the retrieval-based scenarios as baselines.

heading: 3 Dialogue Creation
text: MMDialog is a large-scale multi-turn dialogue dataset towards multi-modal open-domain conversations. It derives from a worldwide social media platform where users can converse with each other and share daily life experiences freely through various modalities, including plain text, photos, or even videos. We design the data collection process into three phases: In the first phase, we extensively manually collect the hashtags commonly used by users and cover as many domains as possible. The second phase starts from the seed hashtags collected in phase one and focuses on collecting all turns that contain at least one image, which we subsequently refer to as anchors. For each anchor, we retrieve all the turns that replied to it and the turn it replied to. We could obtain a complete dialogue session by performing the above process iteratively. In the final phase, we also elaborately design a series of data filtering and post-processing steps to eliminate invalid cases and improve the quality of multi-modal dialogues in MMDialog. To protect the privacy and security of the data, users, and platform, MMDialog is only available to academic researchers under strict terms.

heading: 3.1 Hashtag Collection
text: To collect MMDialog, we crawl one of the most influential online social platform utilizing its academic access API. To improve the data quality, we consider extracting dialogues with their hashtags (e.g., ‘#travel’, ‘#friends’, ‘#golf’), as hashtags tend to show the primary topic of the textual utter-
ances and accompanying visual media. Specifically, we manually screen out 4,184 popular hashtags, each with a minimum of 1,000 dialogues. In this way, our dataset can not only satisfy the properties of open-domain, but also ensure a large scale. We depict the most popular hashtags in Appendix A.1.

heading: 3.2 Multi-modal Conversations Construction
text: Then, we leverage these hashtags as initial seeds for constructing multi-turn dialogues. Initially, for each hashtag, we crawl the turns containing the corresponding hashtag and only retain those incorporating at least one image object (i.e., anchors). Obviously, dialogues containing the anchors are the multi-modal multi-turn dialogues we pursue. Next, within the same conversation, for each anchor, we could seek out all the other turns i) that replied to the anchor until they reach the leaf node (i.e., the end of conversation), and ii) that anchor replied to up to the root node (i.e., the start of conversation). Thus, we could recursively follow the chain of replies to recover the entire conversation.

heading: 3.3 Data Filtering and Post-processing
text: Since the style of messages posted on social media platforms is widely varied, the initial version of MMDialog contains a significant number of invalid, noisy, and even harmful conversations, which may impede research utilizing this dataset. To tackle the above issue, we design a series of sophisticated data filtering processes to filter out highquality multi-modal conversations: a) We remove dialogues containing toxic statements with explicit offensive words; b) We ignore and discard dialogues with GIFs and other modalities (such as videos) which cannot be downloaded immediately. We leave this part of research as future work; c) We remove irregular characters from the dialogue content. For example, we do not consider any urls and ‘@’ items (i.e., expression items for mentioning somebody); d) In particular, we convert emojis and hashtags into corresponding natural language forms to guarantee the coherence of the dialogues; e) We remove all self-talking cases (such as replying to themselves for two or more consecutive dialogue turns) to enhance the integrity of the conversations; f) We discard dialogues with incomplete or missing images; g) We only keep the conversations of no less than three dialogue turns. We believe that by adopting the above data-filtering and postprocessing procedure, the final large-scale multiturn dialogues can be better leveraged to develop
multi-modal open-domain conversation models.

heading: 4 Corpus Statistics
text: MMDialog consists of 1,079,117 unique dialogues and 1,531,341 images. The statistics of several multi-modal open-domain dialogue corpora are detailed in Table 2. On average, each dialogue session in MMDialog has 2.59 images and 4.56 turns, with the images being located arbitrarily within any conversation turns. This reflects the freedom with which individuals can choose any conversational modality at any conversation stage in daily life. Compared to the recently released multi-modal open domain dialogue dataset PhotoChat (Zang et al., 2021), MMDialog enjoys a significantly larger scale of dialogue data and more visual objects, especially that the volume of dialogue sessions has reached million-level. The dialogues in MMDialog, which originate from a wide range of hashtags representing broad domains, are open-domain and cover diverse topics, shedding light on research of multi-modal dialogue modeling. Besides, on average each dialogue turn in MMDialog contains more text tokens than PhotoChat, demonstrating that our proposed data may convey more semantic information in textual utterances.

heading: 5 Task Definition
text: Suppose that we have a multi-modal dialogue dataset D = {(Ui, Ri)}ni=1, where ∀i ∈ {1, ..., n}, Ui is the dialogue context, Ri is the response regarding to Ui. Ui and Ri could contain multi-modal components: textual elements (e.g., utterances) and visual elements (e.g., images). For any U and R, we denote Ui = {umk }Kk=1 and Ri = {rml }Ll=1 as sequence of multi-modal elements including textual utterances and visual images. K and L are
the number of elements in context and response respectively. m ∈ {t, v} indicates the modal type of elements where t represents textual utterances while v signifies visual images.
Since advanced works on pure-text open-domain dialogue systems mainly include retrieval and generative methods. We adapt them to multi-modal scenarios and define the following two tasks that are essential for building a multi-modal opendomain dialogue system:
Task-1: Multi-modal Response Generation To generate a multi-modal response R, one should learn a multi-modal generation model g(R|U) from D. Thus, given a new dialogue context U , following g(R|U), one can directly synthesize a multimodal response R̃ consisting of textual utterances or visual images, or both of them.
Task-2: Multi-modal Response Retrieval The goal of this task is to learn a multi-modal response matching model s(·, ·) from D. For any contextresponse elements (U,R) pair, s(U,R) gives a score that reflects the matching degree between U and R, and thus allows one to rank a set of response candidates C according to the scores for response matching, C consists of both textual utterances and visual images. Thus we can obtain a multi-modal response R̃ consisting of textual utterances or visual images, or both of them.

heading: 6 Evaluation of Multi-Modal Responses

heading: 6.1 General Metrics
text: In Task-1, we follow Sun et al. (2022) and evaluate the quality of the generated responses by using metrics from both text and visual modalities. For the text modality, we compare the generated text to the ground-truth responses using metrics like BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004). For the visual modality, we measure the quality of the generated images using metrics like Inception Score (Salimans et al., 2016) (IS) and CLIP Score (Radford et al., 2021) (CLIP-Sim). Suppose the model does not generate a corresponding textual or visual element, we assign it a score of zero in that case, indicating no relevance between the “empty” element and the ground truth. However, we could not directly adopt the same strategy for Perplexity (PPL, for text) and Frechet Inception Distance (Heusel et al., 2017) (FID, for image). As a lower PPL (or FID) implies a closer distance between generated text (or image) distribution and
real-world text (or image) distribution. Unfortunately, we could not assign an infinity value (similar with zero value for BLEU/IS metrics) standing for “empty” element for both metrics.
In Task-2, we could also compute the Recall@N scores for text and image in similar way. Specifically, for each retrieval step, if the predicted modality intent of next element mismatches with the ground-truth, we could also assign the recall score of this step to zero values. The final recall scores of an example is the average scores of all elements group by the same modality.
Inspired by Zang et al. (2021) which use F1 score to evaluate intent prediction, we extend it from single to multi elements: our F1 score could measure whether the models predict right order of elements modality in a response. Specifically, we first get the modality sequence of generated/retrieved and ground-truth responses as M̃ = {m̃j}Jj=1 and M = {ml}Ll=1 respectively. Then, the F1 score is formulated as:
F1Intent = 2PintentRintent
Pintent + Rintent
Pintent = Match(M, M̃)
J
Rintent = Match(M, M̃)
L
Match(M, M̃) = min{L,J}∑
i=1
1(mi = m̃i)
(1)
where 1 is an indicator function that has value 1 when mi = m̃i, otherwise 0. J and L are the number of elements in responses. More details are described in Appendix A.2.

heading: 6.2 MM-Relevance
text: However, there exist two shortcomings of above metrics: i) They can only reveal the model performance of a single modality, and can not com-
prehensively measure the overall quality of multimodal responses. ii) They address the modalmisalignment issue between prediction and ground truth via assigning zero value, which overlooks the relevance between image and text elements.
Thus we propose a novel evaluation metric, dubbed MM-Relevance, which performs visuallanguage matching based on the large-scale pretrained multi-modal CLIP model (Radford et al., 2021) for multi-modal dialogue response generation and retrieval tasks. In specific, suppose we obtain a multi-modal response R̃ = {r̃mj }Jj=1, and the corresponding ground-truth R = {rml }Ll=1. We first align the two sequences from the left. Then, the representation vector of each element is obtained by encoding it through CLIP-TextEncoder or CLIP-Image-Encoder respectively. We denote the encoded vectors of two responses as: Ẽ = {ẽmj }Jj=1 and E = {eml }Ll=1. Then, we compute the CLIP relevance of the two elements from left to right until they cannot be aligned:
MMRel(R, R̃) = min{L,J}∑
i=1
(emi ) T · ẽmi (2)
In order to penalize the R̃ that is too long or short, we further improve MM-Relevance as F1MM:
F1MM = 2PMMRMM
PMM + RMM
PMM = MMRel(R, R̃)
J
RMM = MMRel(R, R̃)
L
(3)
The relevance can be computed thoroughly between two modal-misaligned responses R and R̃.

heading: 6.3 Correlation With Human Annotation
text: To verify the effectiveness of MM-Relevance in measuring the relevance between a response and
the ground truth, we consider evaluating the correlation between metrics in Task-1/2, including MMRelevance and human annotation on MMDialog. Following (Tao et al., 2018; Gao et al., 2021; Ghazarian et al., 2022), we randomly selected 200 contexts from the test set and solicited ratings from three English-speaking volunteers on a 1-5 Likert scale, reflecting their satisfaction with the response (retrieved or generated) to the given context. We ensure that each response receives three valid ratings, where the average value is used as the final human judgment. We excluded reference-free metrics such as IS and Recall@K, as they do not consider the ground truth. Besides, as Intent-F1 only measures the order accuracy of elements’ modalities in response while ignoring the information of each modal element, we also get rid of this metric.

heading: 7 Baselines

heading: 7.1 Multi-modal Response Generation Model
text: We consider to implement the state-of-the-art multimodal response generation model Divter (Sun et al., 2022) (Figure 2a), which consists of two components: a textual dialogue response generator G and a description-to-image translator F .
Specifically, G takes the dialogue context U as input, then generates a textual sequence which may contains a textual response rt or a textual image description rc or both of them. Noting that in our settings on MMDialog, there may also be several images uv in multi-turn dialogue context, we thereby replace these images by their descriptions uc with the help of an image-to-description translation model. In this way, we could concatenate the textual utterances ut and descriptions uc into a sequence as the input of G. Then, for a generated description rc , F would take them as condition input, and generate a realistic and consistent high resolution image rv as the real response.

heading: 7.2 Multi-modal Response Retrieval Model
text: Inspired by Parekh et al. (2021) and Zang et al. (2021), we also build a retrieval model R named DE++ which consists of a modality intent prediction module I and a ranking module S . As shown in Figure 2b, before each ranking action, I firstly takes the dialogue context U and previous retrieved j−1 response elements R̃<j as input, then predicts the intent I([U, R̃<j ]) as: i) the response is completed and model should stop retrieving new elements. or ii) the modality of next elements. If ii), S will calculate the relevance score S([U, R̃<j ], rmz ). In which C = {rmz }Zz=1 is the candidates set. Eventually, S will select the one with highest relevance score as j-th element rmj of R̃.
Specifically, I and S have similar architecture, we adopt CLIP-Text-Encoder and CLIP-ImageEncoder to represent text and image respectively. In I, we concatenate all the context embeddings with a [CLS] embedding prepending at the first and feed them into a transformer module to predict the intent. In S, we prepend the [CLS] embedding to the context and candidate embeddings, then feed them into a transformer module separately. After that we can obtain the representation vectors of context and candidate, and compute relevance scores by conducting dot-product of them.

heading: 8 Experiments
text: Experiments are conducted on MMDialog to assess two baselines. We perform response/intent predictions for all turns except the first turn of each dialogue and consider all previous turns as context.

heading: 8.1 Experimental Setup
text: We first sample 10K and 10K dialogue sessions for validation and testing respectively. The statistics are shown in Table 3. For detailed experimental and computational setup, please refer to Appendix A.3.

heading: 8.2 Results of Multi-modal Baselines
text: Table 4 reports the evaluation results of the generative baseline. Firstly, Divter achieves relatively low textual response generation performance (9.44 on BLEU-1 and 11.19 on ROUGE-L) on MMDialog. This emphasizes the complexity of the multimodal response generation task and the need for constructing a large-scale multi-modal dialogue dataset to build data-driven models. Secondly, the model shows better performance in image generation, which may be attributed to the use of pre-
trained DALL-E. Thirdly, Divter achieves a 71.77 F1 score in modality intent prediction, indicating its ability to determine whether to generate text or an image during the conversation. Finally, we leverage the proposed MM-Relevance metric to evaluate the overall relevance degree between generated multi-modal responses and ground-truth responses, and Divter achieves a score of 61.85.
Table 5 shows the results of the retrieval baseline. DE++ achieves 18.23% R@1 and 23.07% R@1 on image and text retrieval respectively, demonstrating the capacity of multi-modal retrieval model. Furthermore, we can also find that DE++ obtains a better intent F1 and MM-Relevance than Divter, which may be attributed to a limited retrieval space of test set (1K), as we observe that above metrics would decrease when the retrieval space increases. And we found out that the alignment of the modality would considerably improve the CLIP scores, which benefits the MM-Relevance.

heading: 8.3 Case Study
text: To further investigate the quality of multi-modal responses predicted by our proposed baselines, we display an example on the MMDialog test set in Figure 3. As demonstrated, Divter produces a textual response coherent with the dialogue context, and also can generate a realistic high-resolution image about the “Power Station” in the last turn of context, exhibiting the multi-modal generative capability of our proposed generative baseline. For retrieval, our baseline also retrieved a textual response about “PinkFloyd” and an image on “Power Station” semantically related to the context, validating the effectiveness of retrieval baseline.

heading: 8.4 Metrics Correlation With Human
text: Table 6 illustrates the Pearson, Spearman, and Kendall Correlation between various metrics and human judgments. The Fleiss’ kappa (Fleiss, 1971) of the generative and retrieval labeling is 0.52 and 0.47 respectively, indicating a relatively high agreement among the three labelers. MM-Relevance outperforms other baselines across both generative and retrieval scenarios, indicating a stronger correlation with human annotation compared to other metrics. Besides, the metrics in generative task exhibit a higher correlation than those in retrieval task. This discrepancy may be attributed to the lower relevance between retrieved responses and ground-truth responses, resulting in a relatively low agreement and degrading the correlation.

heading: 9 Conclusion
text: We presented MMDialog, a large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation. By extracting turns associated with images and their surrounding contexts from more than 4K topics, MMDialog provides a diverse and open-domain dataset. To facilitate research on building a more engaging multi-modal dialogue system, we define multi-modal response generation and retrieval tasks, and the MM-Relevance metric based on MMDialog. We also build baseline models and conduct comprehensive performance analyses. We believe MMDialog can serve as a rich resource to propel research in the multi-modal conversation, for years to help the community propose better methods suited to more scenarios.
Limitations
Besides its merits, this work still has limitations that could be further explored. On the one hand, we collect source data of MMDialog with only English language. Thus the applicability to other languages would be restricted. On the other hand, we get rid of GIFs and video-modality elements in MMDialog. In the future, we hope to extend MMDialog to multilingual scenarios and include more modalities such as audio and video.
Ethics Statement
This paper presents a large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation MMDialog. Data in MMDialog are collected from a public worldwide social media platform on which users can converse with each other and share their daily lives messages freely in multiple modalities, including plain text, photos, or even videos. The data-collecting API is only available for academic purposes. And to protect the privacy and security of data, users and platform, MMDialog is just released under strict terms for academic people only. This action fully aligns with the data using and sharing regulations of source data providers. Thus, there will not be any ethical problems or negative social consequences from the research. The proposed method does not introduce ethical/social bias in the data.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their constructive comments and suggestions. This work was supported by the National Key R&D Program of China under Grant No. 2020AAA0106600.

heading: A Appendix
text: A.1 The Most Popular Hashtags
We also depict the most popular hashtags in Figure 4.
A.2 Response Modal Intent Prediction
In MMDialog, the textual utterances and visual images can be freely located anywhere in the response. Therefore, the generation or retrieval order of the modality of response elements is also of great importance for the multi-modal conversation. The intent prediction task aims to predict the order of different modalities in response R̃ given the dialogue context U . Therefore, the intent prediction can be formulated as a classification task:
∀j ∈ [1, J ], I(U, R̃<j) ∈ {0, 1, 2} (4)
where I(·, ·) is the intent prediction model which takes the dialogue context U and previously generated/retrieved response elements R̃<j before j-th step as inputs and provides the modality of next element. Specifically, the model should predict 0 when rj is a textual utterance, 1 when rj is a visual image, and 2 which indicates that the response R̃ is completed and the model should stop generating/retrieving new elements.
A.3 Detailed Experimental Setup
For retrieval task, we randomly sample 999 negative textual utterances and 999 negative visual images from the same split set for each dialogue, maintaining the total number of candidate elements at 1K. While in training phase, the negative ones are in-batch sampled similar to Radford et al. (2021). To be consistent with the Divter of Sun et al. (2022), we fine-tune DialoGPTmedium (345M) (Zhang et al., 2020) as the textual dialogue response generator with Hugging Face transformers1. For the description-to-image translator, we implement DALL-E (Ramesh et al., 2021) using the code of “mega” version in https:// github.com/borisdayma/dalle-mini. We finetune DALL-E for one epoch with initial learning rate 1e-8 and batch size of 64. We process all images into 256 × 256 RGB format for DALL-E. To obtain the description of images in MMDialog, we adopt OFA-huge (Wang et al., 2022) using the code https://github.com/OFA-Sys/OFA/tree/
1https://github.com/huggingface/transformers
feature/add_transformers for image captioning. The version of CLIP model is “openai/clipvit-base-patch32” 2 that has 151M model parameters. As for the retrieval baseline, the representation vectors for both modality obtained by CLIP are fixed during training. The transformers of DE++ consist of 4 Transformer layers with a hidden size of 512 and 8 heads. We train the retrieval model with an initial learning rate of 5e-7 and batch size of 512. For all baselines, early stopping on the validation set is adopted as a regularization strategy and the best model is selected based on the validation performance. The training of both tasks is conducted on 8 Nvidia Tesla A100 80G GPU cards for about 20 hours. The BLEU and ROUGE scores are computed by codes in https: //github.com/Maluuba/nlg-eval, while the IS is obtained by https://github.com/toshas/ torch-fidelity.
2https://huggingface.co/openai/ clip-vit-base-patch32
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section "Limitations"

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section "Ethics Statement"

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 8

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 8.1 and Appendix A.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 6, Section 8.1 and Appendix A.3

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 and Section 8

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 8.1 and Appendix A.3
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 6.3

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section 6.3

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 6.3 and they are volunteers.

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section 3, Section 4, Section 6.3 and Ethics Statement

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Section 3, Section 4, Section 6.3 and Ethics Statement

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Section 3, Section 4, Section 6.3 and Ethics Statement

heading: Future Work
text: In the future, we hope to extend MMDialog to multilingual scenarios and include more modalities such as audio and video