MPCHAT: Towards Multimodal Persona-Grounded Conversation
Year: 2023

Abstract:
In order to build self-consistent personalized dialogue agents, previous research has mostly focused on textual persona that delivers personal facts or personalities. However, to fully describe the multi-faceted nature of persona, image modality can help better reveal the speaker’s personal characteristics and experiences in episodic memory (Rubin et al., 2003; Conway, 2009). In this work, we extend persona-based dialogue to the multimodal domain and make two main contributions. First, we present the first multimodal persona-based dialogue dataset named MPCHAT, which extends persona with both text and images to contain episodic memories. Second, we empirically show that incorporating multimodal persona, as measured by three proposed multimodal persona-grounded dialogue tasks (i.e., next response prediction, grounding persona prediction, and speaker identification), leads to statistically significant performance improvements across all tasks. Thus, our work highlights that multimodal persona is crucial for improving multimodal dialogue comprehension, and our MPCHAT serves as a high-quality resource for this research.

heading: 1 Introduction
text: With the rapid advance of conversational AI systems in recent years, developing self-consistent dialogue agents has been studied much (Li et al., 2016; Zhang et al., 2018). Considerable research aims to endow dialogue agents with persona, which represents an individual’s personality (Zhong et al., 2022; Cao et al., 2022). In particular, researchers have exploited textual description of persona, for example, in the form of unstructured sentences (Mazaré et al., 2018), structured key-value attributes (e.g., age, gender, location) (Song et al., 2020) and personality types (e.g., Big-Five) (Mairesse and Walker, 2007). Therefore, dialogue agents with persona have been found to
(1) exhibit greater self-consistency (Welleck et al., 2019; Kim et al., 2020; Majumder et al., 2020), (2) demonstrate awareness of long-term memory (Xu et al., 2022a,b; Bae et al., 2022), and (3) generate engaging responses instead of non-specific ones (Zhang et al., 2018; Mazaré et al., 2018).
However, existing studies restrict the role of persona only to personal facts (Zhang et al., 2018) or personalities (Li et al., 2020a), while it should be explored in multi-faceted ways (Moore et al., 2017). More than factual information, episodic memory (Tulving, 1972), which is the memory of everyday events or personal experiences connected to the self and autonoetic consciousness (Tulving, 2002; Conway, 2005), should be included in persona component. Wilson and Ross (2003) further supports this assertion by arguing that episodic memory plays a significant role in shaping personal
3354
identity, which in turn can influence one’s persona. Since episodic memories are often represented in the form of visual images or history scenes (Rubin et al., 2003; Conway, 2009), we propose to study the multimodal persona, which consists of a set of image-sentence pairs describing memorable moments as shown in Figure 1. Furthermore, visual information can complement textual information, which often lacks an explicit description of appearance or measurable quantities (Jin et al., 2022; Zhang et al., 2022).
In this work, we contribute to the personabased dialogue research in two important ways. First, we introduce a new multimodally personalized dialogue dataset named Multimodal Persona Chat (MPCHAT), where personas reveal speakers’ episodic-memories using both text and images. To the best of our knowledge, MPCHAT is the first dataset that supports multimodal persona in dialogue. To collect episodic-memory-based multimodal personas, we source users’ posts from social media Reddit. We carefully design a pipeline to curate multimodal conversation data that are wellgrounded on multimodal personas1.
Second, based on MPCHAT, we propose three retrieval-based dialogue tasks as benchmarks for multimodal persona-grounded dialogue understanding: next response prediction, grounding persona prediction, and speaker identification. By incorporating our proposed multimodal persona, we observe statistically significant performance improvements across all tasks.
Consequently, our work illustrates the significance of multimodal persona in enhancing multimodal dialogue comprehension, and our MPCHAT provides a valuable resource for the research, given its well-grounded dialogues (especially responses) on multimodal personas.

heading: 2 Related Work
text: Personalized dialogue. Personalized dialogue agents have exploited persona in the form of unstructured sentences (Zhang et al., 2018; Zhong et al., 2020), structured key-value attributes (Qian et al., 2018; Zheng et al., 2019), and personality types (Mairesse and Walker, 2007; Wen et al., 2021). Persona in these works reveals only personal facts (e.g., age, gender, job, location, hobby)
1Note that our dataset pipelining approach is not restricted to Reddit and can be extended to other sources such as Twitter, Instagram, and more.
or personalities (e.g., Big-Five, MBTI) in the textual format. Instead, we focus on an episodicmemory-based persona describing diverse, memorable moments of personal experiences (Schacter et al., 2009) using both sentences and images.
Multimodal datasets. To fuse visual and textual modalities, various works have been conducted on building datasets of paired images and text (Ordonez et al., 2011; Lin et al., 2014; Krishna et al., 2017; Sharma et al., 2018; Shao et al., 2019; Kuznetsova et al., 2020) and multimodal models (Lu et al., 2019; Li et al., 2020b, 2021). In these datasets, text tends to explicitly describe the paired images (e.g., image captioning and visual question answering) in a short sentence. On the other hand, Desai et al. (2021) released RedCaps, whose image-sentence pairs are sourced from social media Reddit and whose text captions are more conversational and diverse than existing datasets. We use Reddit to source image-sentence pairs as multimodal persona, but we build a new multi-turn dialogue dataset, MPCHAT, to extend the role of persona to reflect episodic memories and further explore multimodal dialogue comprehension in personalized dialogue.
Multimodal dialogue. Research on multimodal (or image-grounded) dialogue has focused on understanding images and utterances in a contextaware manner (Mostafazadeh et al., 2017; Das et al., 2017; Shuster et al., 2020; Zheng et al., 2021; Zang et al., 2021; Lee et al., 2021). Simple retrieval dialogue agents (Shuster et al., 2020; Lee et al., 2021), which fuse textual and visual features, have been used to produce image-grounded responses. MPCHAT also consists of images and dialogues, but we utilize multimodal persona to produce both image-grounded and persona-grounded responses.

heading: 3 The MPCHAT Dataset
text: We collect a multimodal persona-grounded dialogue dataset named MPCHAT (Multimodal Persona Chat). The objective of MPCHAT is to help a conversational agent utilize its episodicmemory-based persona, consisting of both linguistic and visual information, to produce personagrounded responses. To cover a wide range of episodic-memory-based multimodal persona, we source posts from social media Reddit.
However, dialogue with a multimodal persona introduces two new challenges. First, it is harder to collect persona image-sentence pairs than to collect
personas sentences. Second, it is also difficult to collect dialogue instances grounded on speakers’ multimodal personas since each utterance should be grounded on not only persona sentences but also persona images, which may require more finegrained information with additional commonsense knowledge (Cui et al., 2020; Liu et al., 2022). To overcome these challenges, we design the process of data construction as follows.

heading: 3.1 Collecting Multimodal Persona
text: Following RedCaps (Desai et al., 2021), we manually curate a set of subreddits with a high proportion of image posts, where images are photographed by Reddit users themselves, and post titles are related to the image content. In total, we use 648 subreddits, whose full list can be found in Appendix E.1. We then download all image posts from the selected subreddits. We intend to define a user’s multimodal persona as m number of image-sentence pairs where m is the number of the user’s posts. Thus, we group the downloaded posts according to users, and transform each post into a pair of one image and one sentence using (1) a rule-based method and (2) a model-based method as follows.
Rule-based lexical method. We use the post title as the persona sentence. If the title consists of multiple sentences, we select only the first one as done in Mazaré et al. (2018).We then retain the sentences that satisfy all the following rules: (1) each sentence must contain between 4 and 20 words, (2) it contains either the word I or my, and it consists of (3) at least one verb, (4) at least one noun or adjective, and (5) at least one content word. With this method, we improve the fluency and expressiveness of the persona sentences.
Model-based semantic method. After obtaining image-sentence pairs, we ensure that the image is semantically relevant to its paired sentence. We leverage the pretrained CLIP-ViT-B/32 (Radford et al., 2021) to calculate semantic similarity between the image and the sentence, which is widely used in past research (Hessel et al., 2021; Cho et al., 2022; Frans et al., 2022). Then, we ignore the pair with a cosine similarity less than 0.
Finally, we follow Desai et al. (2021) to avoid potential ethical risks of curating Internet-scale image datasets. See Appendix A.4 for the details of our ethical considerations. As a result, about 10% of downloaded posts are used to make multimodal personas, and the others can be exploited for dia-
logue data.

heading: 3.2 Collecting Dialogues
text: Once we obtain a set of users’ multimodal personas, we collect dialogue data where the users participate in the conversation. Discussions on Reddit consist of threads, each with one post and multiple comments, as shown in Figure 1. From the curated subreddits in Appendix E.2, we collect threads containing the comments the users wrote with multimodal persona. We exclude the threads used to make multimodal personas in § 3.1 to ensure that the source of persona is disjoint with that of conversation. We iteratively trace the parent comment nodes in threads until the root node appears, finding the post and all its comments before the persona user’s comment that constitutes a single conversation data. Therefore, in each dialogue data, the last utterance spoken by the persona user becomes the response, and all previous comments and the image post become the context. We set the maximum number of turns in the context to 20.
We filter out dialogues where a user’s response is posted earlier than the user’s persona posts since the episodic-memory persona should chronologically precede the user’s response. We additionally filter dialogues as explained in Appendix A.1.

heading: 3.3 Grounding Persona on Dialogues
text: To ensure persona-consistency, the user’s response in dialogue should be well grounded on his or her multimodal persona. Otherwise, it is impossible for an algorithm (or even a human) to correctly predict the response based on the persona, which may undermine the usefulness of our dataset.
We automatically filter out the conversations whose responses have no persona-related information by employing (1) heuristic rules and (2) pretrained models (Reimers and Gurevych, 2019; Radford et al., 2021); see Appendix A.2 for details.
Despite the effectiveness of the automatic filtering process, we empirically find that some responses are still not grounded on persona since the pretrained models used for automatic filtering are not perfect. According to Welleck et al. (2019), identifying an utterance grounded on (i.e., consistent with) a persona sentence can be reduced to a natural language inference (NLI) task. Thus, we conduct additional human NLI annotation to make sure that the user’s response is grounded on the multimodal persona.
In our NLI setting, the premise p = (pi, pt) is a persona image-sentence pair among the speaker’s multimodal persona set P = {p1, ..., pm}, and the hypothesis r is the response in conversation from the same speaker. The goal is to perform a binary classification for a pair (r, p): (1) ENTAILED if there is enough evidence in p = (pi, pt) to conclude that r is most likely true. (2) NOT ENTAILED if (i) there is enough evidence in p to conclude that r is most likely false, or (ii) there is not enough evidence in p to draw a conclusion about r.
We annotate entailment labels from human workers via Amazon Mechanical Turk (Mturk). To reduce the label costs, we only collect entailment labels for at most two persona elements (among m elements) per response r. See Appendix A.3.2 on how to select the two persona elements.
Given a context c = (ct, ci), response r and a persona image-sentence pair p, we ask three annotators to categorize a pair (r, p) into the two classes. Following previous works (Bowman et al., 2015; Xie et al., 2019), we finalize labels according to the majority vote criterion (at least 2 out of 3). As a result, we obtain the labels for 16,327 pairs from human workers, and 50.4% of them are finally labeled as ENTAILED. We defer the annotations’ details to Appendix A.3.4. The inter-annotator agreement for entailment labels is measured using Krippendorff’s α (Krippendorff, 2011). It is 0.47, implying a good agreement despite the difficulty of the task (Chen et al., 2020; Zhang and de Marneffe, 2021).

heading: 3.4 Final Multi-turn Dialogue Data
text: In summary, one dialogue consists of the response as the last utterance spoken by the persona speaker and the context as all prior utterances from the Reddit post. We then construct a multi-turn dialogue by merging the dialogues sharing common threads (i.e., multiple responses by persona users exist in a single dialogue). Finally, we have 7,898 multiturn dialogue data whose responses are ENTAILED with (or grounded on) the persona (i.e., at least one persona element-response pair is labeled as ENTAILED). Also, we add a similar amount of dialogue data whose responses are grounded on no persona element, since the dataset should be able to evaluate whether the method can correctly identify no grounding. It also follows persona-sparse real-world conversations (Zheng et al., 2020) that contain a limited amount of dialogues grounded on speakers’ persona. By randomly selecting 7,102
such dialogues, eventually, MPCHAT consists of 15,000 multi-turn dialogues.

heading: 3.5 Analysis of MPCHAT Compared to Other Persona-Based Dialogue Datasets
text: The dataset consists of 15,000 multi-turn dialogues with 42,531 utterances by 25,877 users. We divide MPCHAT into train/valid/test split with 11,975/1,516/1,509 dialogues chronologically; the test set is the most recent dialogues so that they are disjoint with existing Reddit-sourced datasets.
Statistics and properties. Table 1 compares MPCHAT with other persona-based dialogue datasets. Only MPCHAT uses images for persona, and describes episodic-memory-based persona beyonds fact, thought, or personality. Moreover, MPCHAT provides additional persona entailment labels that indicate whether a response is grounded on a given image-sentence persona.
Frequent verbs in personas. Figure 2 compares the top-20 frequent verbs in persona sentences from MPCHAT and PersonaChat (Zhang et al., 2018). Thanks to Reddit’s abundant sources, the number of verbs from MPCHAT is much larger than those from PersonaChat. The persona sentences in our dataset also include past tense verbs such as made, found, and finished while persona sentences in PersonaChat do not. It is because our personas are based on episodic memory, which is the collection of personal experiences or memorable moments at particular times.
Lexical diversity of personas. Table 2 compares the lexical diversity of persona sen-
tences from MPCHAT with those from PersonaChat (Zhang et al., 2018) and PEC (Zhong et al., 2020). We count the number of N-grams from the fixed number (i.e., 6,737) of randomly sampled persona sentences from each dataset. Then, we measure lexical diversity using three metrics: MTLD, HD-D (McCarthy and Jarvis, 2010) and MATTR scores (Covington and McFall, 2010). Surprisingly, persona sentences from MPCHAT achieve the highest scores in all lexical diversity metrics. This result is also caused by the different properties of persona sentences: specific personal experiences of episodic memory in MPCHAT vs. permanent characteristics, repeated events, and emotions in PersonaChat and PEC.
We report more dataset analyses in Appendix B.

heading: 4 Task Definition
text: As benchmark tasks for MPCHAT, we consider three retrieval tasks as follows. (1) The next response prediction task is to predict the next response given a context and the speaker’s multimodal persona, which has been often regarded as a main task of persona-based dialogue (Humeau et al., 2020; Zhang et al., 2018). (2) The grounding persona prediction task is to predict speaker’s persona element, either based on the dialogue context
alone or based on both the dialogue context and the response. This task is derived from and symmetrical to the next response prediction task. Both the next response prediction and grounding persona prediction tasks are designed to ensure both multimodal context-awareness and multimodal personaconsistency. (3) The speaker identification task is to identify the speaker participating in a dialogue given a context and a response, which is crucial in personalized dialogues (Zhang et al., 2018; Sang et al., 2022). In this task, we design it as a ranking problem, considering that MPCHAT supports multi-party dialogues. Furthermore, we expand the existing task into the multimodal domain.
Specifically, the dialogue dataset D is a list of N dialogues, each of which consist of (c, r, P ), where a context c = (ci, ct) contains a context image ci and context text ct (i.e., context utterances), r is a response to context c, and a persona set P = {(pi1, pt1), ..., (pim, ptm)} is a set of m = 5 persona image-sentence pairs of the speaker who spoke the response r. We below describe each task setting.
Next response prediction. The goal of this task is to predict the next response r∗ based on Pr(r|c, P,Rc), from a response candidate set Rc = {r1, r2, ..., rCr}, as shown in Figure 3. The response candidate set Rc contains a correct response r∗ and Cr − 1 randomly sampled test responses.
Grounding persona prediction. This task aims at predicting the persona element p∗, which grounds r (i.e., labeled as ENTAILED in § 3.3) based on Pr(p|c, r, P̄ , Pc) or Pr(p|c, P̄ , Pc). Pc = {p1, p2, ..., pCp} is a persona (element) candidate set, which includes a correct persona element p∗ and Cp − 1 randomly sampled persona elements
from other speakers. P̄ is the speaker’s remainder persona set, a set of m− 1 persona image-sentence pairs in P except p∗. Note that we consider two cases of whether r is given or not. If r is not given (i.e., no-response case), then a model needs to retrieve the most likely persona element p∗ based on a given context c and a remainder persona set P̄ before producing a response r. If r is given (i.e., response case), a model predicts p∗ that grounds r, which is much easier than the former case.
Speaker identification. Finally, we predict the speaker (with his/her multimodal persona set) P ∗ who spoke the response r based on Pr(P |c, r,Pc), from a speaker candidate set Pc = {P1, P2, ..., PCP }. The speaker candidate set Pc includes a correct speaker P ∗ and CP − 1 randomly sampled speakers.
Following Humeau et al. (2020); Zhong et al. (2020); Shuster et al. (2020); Lee et al. (2021), we use Recall@1 and mean reciprocal rank (MRR) as evaluation metrics, and set the number of retrieval candidates Cr, Cp, and CP to 100.

heading: 5 Models
text: To solve the proposed retrieval-based dialogue tasks, we first define a set of unimodal encoders for the input of persona image and text (P i, P t), context image and text (ci, ct), and a response r. We then construct multimodal persona-aware models by combining these modules based on input components for each task. Note that we design our models to be simple and standard, to investigate the characteristics of our dataset.
Text encoder. We use a Transformer (Vaswani et al., 2017) as the text encoder for context text ct, persona sentences P t, and a response r. We test two initialized weights of SBERT2 (Reimers and Gurevych, 2019) and the CLIP-ViT-B/32 text model (Radford et al., 2021). For a persona input P t, we encode the concatenation of m persona sentences. The representation of each text input (hct , hP t , hr) is obtained by the mean-pooled output of the entire sequence for SBERT or the hidden state of the first token [CLS] (for CLIP), followed by a linear layer.
Image encoder. We encode a context image ci and a set of persona images P i using a single grid-based ViT-B/32 (Dosovitskiy et al., 2021) and CLIP-ViT-B/32 vision model (Radford et al., 2021)
2https://huggingface.co/sentence-transformers/ multi-qa-distilbert-cos-v1.
due to its zero-shot ability. We use the hidden states of the first patch of each image, followed by a linear layer, as a pooled representation following Dosovitskiy et al. (2021), which is mean-pooled to obtain a representation of persona images hP i .

heading: 5.1 Models for Three Dialogue Tasks
text: Figure 4 shows our model for the next response prediction task, from which models for the two other tasks can be easily inferred.
Next response prediction. After encoding each input separately, we first average hP i and hP t to produce the representation of a persona set hP . Then, we mean-pool hP , hct , hci as the final representation hout, which is used to compute the dotproduct score for a response r among candidate pool Rc using hout · hr.
Grounding persona prediction. We first meanpool hP̄ i and hP̄ t to obtain hP̄ . We then output hout by averaging all input embeddings of hP̄ , hct , hci for the no-response case and hr together for the response case. Lastly, hout is used to compute the dot-product score for an image-sentence pair p among candidate pool Pc by hout · hp, where hp = mean-pool(hpi , hpt).
Speaker identification. We mean-pool hct , hci , hr to produce hout, which is used to compute the dot-product for a speaker’s persona pairs P = (P i, P t) among candidate pool Pc using hout · hP , where hP = mean-pool(hP i , hP t).

heading: 5.2 Training and Inference
text: According to encoder types, we test three conversation models: SBERT+ViT, SBERT+CLIP, and CLIP+CLIP (i.e., original CLIP). During training of all three tasks, we consider the other labels in each batch as negatives and train with a cross entropy loss over the matching scores as in Humeau et al. (2020). We do not update the parameters
of image encoders (except CLIP+CLIP), which were common in previous studies (Shuster et al., 2020; Lee et al., 2021). At the inference stage, each model selects the response that maximizes the dot-product score with the candidate set, such as hout · hrj with rj ∈ Rc for next response prediction, the persona element pj ∈ Pc with hout · hpj for persona prediction, and the speaker’s persona Pj ∈ Pc with hout · hPj for speaker identification. We defer implementation details to Appendix C.1.

heading: 6 Experiments
text: The main goal of our experiments is to verify that multimodality from images and text indeed helps better understand persona-based dialogues, and our MPCHAT is properly collected for this purpose. Thus, we design our experiments as follows. (1) Our models are rather simple and standard, as discussed in §5. (2) We compare our models that take advantage of full inputs with several baselines that use only parts of them.

heading: 6.1 Next Response Prediction
text: Baselines. We compare with the following baselines. (1) Context text only (ct): This baseline outputs the matching score with the dot product between hct and hrj . In addition, we add a simple information retrieval baseline, where the response candidates are arranged in the order of their weighted similarity (i.e., TF-IDF score) to the context text ct. (2) Context image only (ci): It takes the dot product between hci and hrj as the matching score. (3) Context only (c): The matching score is the dot product between hc = mean-pool(hci , hct) and hrj . (4) Context + persona sentences (c, P
t): The matching score is the dot product between hc;P t = mean-pool(hci , hct , hP t) and hrj . (5) Context + persona images (c, P i): The matching score is the dot product between hc;P i = mean-pool(hci , hct , hP i) and hrj .
Evaluation metrics. We evaluate the performance using Recall@1 and MRR metrics as described in § 4. Statistical significance is computed using a two-sided t-test against the best competitor in all tasks, including grounding persona prediction (§ 6.2) and speaker identification (§ 6.3).

heading: 6.1.1 Results
text: Table 3 shows the results of next response prediction task. We observe the following findings.
Context image (ci) helps response prediction. In all models, conditioning on the context
image (ci) significantly improves models to predict next response: +7.34% recall@1 score for SBERT+ViT model and +9.05% recall@1 score for SBERT+CLIP model. These performance gaps show that dialogues in MPCHAT are well grounded on context images. CLIP zero-shot model outperforms SBERT zero-shot model, demonstrating CLIP’s ability to retrieve the correct text response from the context image only.
Persona images P i are important as well as persona sentences P t. In all models, conditioning on persona images (i.e., context + persona images) and on persona sentences (i.e., context + persona sentences) enhance next response prediction. In addition, conditioning on persona sentences shows better performance than conditioning on persona images, meaning that textual information in persona is more helpful than the image in persona to predict the textual response.
Using both persona images P i and sentences P t achieves the best performance. In all models, using multimodal persona leads to the best Recall@1 and MRR scores. It concludes that (1) MPCHAT is well grounded on multimodal persona,
and (2) the persona image and sentence can complement each other to improve performance.

heading: 6.2 Grounding Persona Prediction
text: Baselines. We use the following baselines. We set the no-response as a default case. (1) Context only (c): The matching score is the dot product between hpj and hc = mean-pool(hci , hct) (or hc;r = mean-pool(hci , hct , hr) for the response case). (2) Context + remainder persona sentences (c, P̄ t): The matching score is the dot product between hpj and hc;P̄ t = mean-pool(hci , hct , hP̄ t) (or hc;r;P̄ t = mean-pool(hci , hct , hr, hP̄ t)). (3) Context + remainder persona images (c, P̄ i): The matching score is the dot product between hpj and hc;P̄ i = mean-pool(hci , hct , hP̄ i) (or hc;r;P̄ i = mean-pool(hci , hct , hr, hP̄ i)).

heading: 6.2.1 Results
text: We present the results of grounding persona prediction in Table 4 for the no-response as well as response cases.
Providing response r drastically improves performance. Compared to no-response case, results at response case indicate that all models can predict the correct persona element based on the response with a 90% chance or more, meaning that persona entailment labels collected in § 3.3 are well annotated.
Remainder persona images P̄ i provide visual clues. While not true for all cases, the results demonstrate that P̄ i improves models better than P̄ t in the following scenarios: CLIP+CLIP in both no-response and response cases, as well as CLIP+ViT in the response case. Therefore, visual clues from P̄ i as well as textual clues from P̄ t are helpful in accurate persona prediction.
Again, using both remainder persona images P̄ i and sentences P̄ t maximizes the performance. In both cases, models equipped with full inputs attain the best Recall@1 and MRR scores. It verifies the usefulness of the multimodal remainder persona set P̄ = (P̄ i, P̄ t).

heading: 6.3 Speaker Identification
text: Baselines. (1) Text only dialogue (ct, r) + speaker’s persona sentences (P tj ): The matching score is the dot product between hct;r = mean-pool(hct , hr) and hP tj . (2) Dialogue (c, r) + speaker’s persona sentences (P tj ): The matching score is the dot product between hc;r = mean-pool(hci , hct , hr) and hP tj . (3) Dialogue
(c, r) + speaker’s persona images (P ij ): The matching score is the dot product between hc;r = mean-pool(hci , hct , hr) and hP ti .

heading: 6.3.1 Results
text: From Table 5, we can find several observations about the speaker identification task.
Persona sentences P tj are more important than persona images P ij . In all models, predicting the speaker based on his/her persona sentences P tj outperforms that on persona images P ti . It indicates that textual information plays a key role in retrieving the right speaker in this task.
Using multimodal information Pj still enhances speaker identification. In all models, identifying the speaker based on his/her persona imagesentence pairs Pj = (P ij , P t j ) shows the highest scores. That is, persona images can complement persona sentences, showing the necessity of multimodal persona for the speaker identification task.
Furthermore, we present additional analyses that go beyond the main experiments in Appendix D.

heading: 6.4 Error Analysis
text: We investigate error cases, specifically focusing on next response prediction and grounding persona prediction (no-response) tasks. We analyze missed retrieved responses/persona and discuss fac-
tors related to multimodal comprehension and understanding of both dialogue context and persona information.

heading: 6.4.1 Next Response Prediction
text: We randomly selected 30 examples from the 629 incorrect predictions made by the CLIP+CLIP (with full inputs) out of the test set. Among them, we observed the following patterns in errors:
Multimodal understanding. 19 instances (63%) failed in multimodal understanding, indicating challenges in effectively leveraging both visual and textual information. Specifically, 14 instances required multi-hop reasoning between the multimodal context (ci, ct) and multimodal persona components (P i, P t), such as cases involving visual coreference resolution. Additionally, 5 instances solely relied on context comprehension (c only) without considering persona information.
Text understanding. 9 instances (30%) struggled with text understanding, indicating persistent difficulties in comprehending complex textual clues. Out of these instances, 7 required multi-hop reasoning between the context ct and persona P t, while 2 instances required context comprehension (ct only) without considering persona information.
Task ambiguity. 2 instances (7%) failed due to the task ambiguity, where the next response r∗ is
not the only response given context c and a persona set P .
6.4.2 Grounding Persona Prediction (no-response)
We randomly selected 30 examples from the 123 incorrect predictions made by the CLIP+CLIP (with full inputs) out of the test set, and identified the following error patterns:
Multimodal understanding. Among the instances, 17 (57%) failed in multimodal understanding. 15 instances required multi-hop reasoning between the multimodal context (ci, ct) and multimodal persona components (P̄ i, P̄ t), while 2 instances required persona-consistency comprehension (P̄ only) without context information.
Text understanding. 9 instances (30%) failed in text understanding. Out of these, 7 required multihop reasoning between the context ct and persona P t. 2 instances required persona-consistency comprehension (P̄ t only) without considering context information.
Task ambiguity. In 4 instances (13%), errors were caused by task ambiguity, where the persona element p∗ is not the only answer given context c and a remainder persona set P̄ .
These results highlight the challenges in effectively leveraging multimodal information and emphasize that understanding both multimodal context and multimodal persona poses a greater challenge for dialogue models compared to understanding context or persona alone.

heading: 7 Conclusion
text: We studied episodic-memory-based multimodal persona-grounded dialogue, and introduced MPCHAT as the first multimodal personagrounded multi-turn dialogue dataset. We proposed three retrieval-based dialogue tasks to evaluate the effectiveness of multimodal persona. With the help of multimodal persona, all of the proposed models exhibited better dialogue comprehension abilities. Our empirical results showed that dialogues (especially responses) in MPCHAT are well grounded on multimodal personas as intended. One interesting future work would be to expand MPCHAT in both the size (e.g., scaling up the number of dialogues and personas) and the scope (e.g., adding audio/video modality).

heading: Limitations
text: Since MPCHAT sources the data from Reddit, it has the limitation that it may not be representative of the general population. First, all subreddits of MPCHAT are primarily written in English, and a significant percentage of Reddit users are from English-speaking countries. The four countries with the highest desktop traffic on Reddit are the US, UK, New Zealand, and Australia, accounting for 66% of the total user (Clement, 2022). Moreover, compared to the average US population, Barthel et al. (2016) reported that Reddit users are more likely to be male (67% vs. 49%), young (64% 18-29 years old vs. 22%), college-educated (42% vs. 28%), and politically liberal (43% vs. 24%). Therefore, MPCHAT may reflect such somewhat narrow interests, and the demographic group represented by our model may be biased toward personal conversations suitable for it.

heading: Ethics Statement
text: We put much effort into ensuring that our MPCHAT dataset includes no personal identifying information (PII): we only picked subreddits that were not aimed at people and filtered out faces, license plates, and email addresses. Also, we only selected subreddits without 18+ tags and filtered NSFW images, offensive words, etc. Note that we manually filtered out all images containing PII or NSFW content before publicly releasing MPCHAT. Human annotators earned an average wage of $16 per hour, above the minimum wage in their areas. We abided by the Reddit API Terms of Use and also informed our annotators about this. Finally, we specified all licenses of scientific artifacts and will include them when distributing our data. See Appendix A.4 and C.2 for the details.
However, potential risks still remain in our data. As mentioned in Limitations 7 and Appendix A.3.4, authors and annotators of MPCHAT are primarily in the US, UK, New Zealand, and Australia. These demographic and geographic biases mean that MPCHAT may not equally represent all groups. Meanwhile, Wang et al. (2021); Lee et al. (2022) reported that preprocessing data with CLIP can cause gender-bias issues. We use CLIP to measure image-text similarity in the pre-processing for data collection, so this problem may exist in our dataset.
Users of our dataset should be aware of these risks. To comply with the Reddit API Terms of Use and to protect the privacy of Reddit users, commer-
cial and for-profit use of our data is limited. It must be available for academic purposes only.

heading: Acknowledgements
text: First of all, we thank all our workers on MTurk for their dedication and enormous contribution to constructing MPCHAT through this project. We would also like to thank Hyunwoo Kim, Jiwan Chung, Soochan Lee, Jinseo Jeong, Insu Jeon, Jaekyeom Kim, Euihyun Tae, and the anonymous reviewers for their valuable comments. This work was supported by Samsung Research Funding Center of Samsung Electronics (No. SRFCIT210101) and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2021-0-01343, Artificial Intelligence Graduate School Program for Seoul National University, and No.2022-0-00156, Fundamental research on continual meta-learning for quality enhancement of casual videos and their 3D metaverse transformation). Gunhee Kim is the corresponding author.

heading: Appendix

heading: A More details on Dataset Collection

heading: A.1 Filtering Dialogue Data
text: We filter Reddit conversation data to ensure that (1) each post is between 2 and 100 words, and (2) each comment is between 2 and 60 words3. We remove dialogues whose images contain potential ethical risks; see Appendix A.4 for the ethical considerations in detail. We automatically filter out whose utterances contain words or phrases from a blocklist4 to prevent models from training offensive expressions. Also, we ignore dialogues that are written earlier than the user’s multimodal persona. This is because a multimodal persona represents episodic memory in history, and thus predicting responses in conversations that precede the persona may not be reasonable. Finally, we lowercase all text and remove emojis, special symbols, URLs, and email IDs (including “@”) from each sentence.

heading: A.2 Automatic Filtering of Persona Irrelevant Conversation
text: Given a dialogue context that consists of image ci and text ct parts and a response r, and a set of persona image-sentence pairs P = {(pi1, pt1), ..., (pij , ptj), ..., (pim, ptm)} of the speaker who wrote r, we filter the conversation as follows.
We first filter out the conversation if the length of the response (r) is shorter than five words because short responses usually do not contain personarelated information.
Next, we keep the conversation if any persona element (pij , p t j) in P is related to the response r as follows: we measure the text similarity (i.e., cosine similarity) score between the response and the persona sentence simSBERT (r, ptj) and again measure the text similarity score between the context text and the persona sentence simSBERT (ct, ptj) by employing a Sentence BERT (or SBERT) model5 (Reimers and Gurevych, 2019). After manually checking some data instances, we set a threshold of 0.5 to filter out instances in which r is not related to ptj . That is, if both simSBERT (r, p t j) and simSBERT (c t, ptj) are below the threshold, we filter out the persona element. We also measure the image-text similarity (i.e., cosine similarity) between the response and the per3This is because posts are usually longer than comments. 4https://github.com/rominf/profanity-filter 5https://huggingface.co/sentence-transformers/ all-MiniLM-L6-v2
sona image simCLIP (r, pij) and again measure the similarity between the context text and the persona image simCLIP (ct, pij) by employing a CLIP-ViTB/32 model (Radford et al., 2021). In this case, we set a threshold of 0 to filter out no personarelated conversations, and if either simCLIP (r, pij) or simCLIP (ct, pij) is below the threshold, we filter out the persona element.
After all, we keep the conversation if any of the persona elements are unfiltered.

heading: A.3 Details on Persona Entailment Labeling

heading: A.3.1 Two-Class Persona Entailment
text: Unlike previous works (Williams et al., 2018; Welleck et al., 2019) that use 3-way labels of {entailment, contradiction, neutral}, we modify it to 2-way labels of {ENTAILED, NOT ENTAILED } since we are interested in the detection of personaresponse grounding. Also, we find that the same speaker is unlikely to post contradictory sentences (or images), leading to merging contradicted and neutral labels into NOT ENTAILED label.

heading: A.3.2 Persona Selection for Entailment Labeling
text: Given a dialogue with a context image ci, context text ct and a response r, and a set of persona elements P = {(pi1, pt1), ..., (pij , ptj), ..., (pim, ptm)} of the speaker who wrote r, we select at most two persona elements per response r as follows. First, we apply the same method as in Appendix A.2 to filter out no persona-related response. We drop the whole dialogue and do not select any persona element if all elements are filtered out. If only one persona element is survived, then we select it. If multiple persona elements are survived, we select at most two persona elements based on text similarity scores: (1) an element with the best simSBERT (r, ptj) score and (2) one with the best score of the sum of simSBERT (r, ptj) + simSBERT (c
t, ptj). Then the persona elements selection is over, and the remaining data (i.e., a set of at most two persona element-dialogue pairs) moves on to the next step: human annotations for the persona entailment labeling task.

heading: A.3.3 UI design for Mturk
text: Figure 6 and Figure 7 show the annotation page for annotators labeling persona entailment labels. Note that we provide 3-way labels among entailed, contradicted, and irrelevant (i.e., neutral), and then
reduce them to 2-way labels by merging contradicted and irrelevant into NOT ENTAILED, while maintaining entailed label as ENTAILED.

heading: A.3.4 Quality Control for Human Annotators
text: We only allow annotators located at one of [AU, CA, NZ, US, GB]. We use a qualification test to discern annotators who do not fully understand the task (e.g., only selecting NOT ENTAILED regardless of the problem, or selecting ENTAILED just because r and pt seem to be lexically similar). Based on submitted answers in the qualification, we manually approve workers if they earn an acceptable score. We periodically block malicious annotators to maintain high approval rates, while providing a reasonable bonus to benevolent workers. Moreover, we steadily profile workers whose accuracy is lower than the average and re-educate them by showing examples with detailed explanations. As a result, a total of 65 workers participated in the annotation process.

heading: A.4 Ethical Considerations in Data Collection
text: In our data collection, we follow the overall ethical considerations proposed by RedCaps (Desai et al., 2021) to align with the Reddit API terms of use and avoid violating ethical principles. We perform additional efforts to protect user privacy, such as license plate detection.
Privacy. The foremost consideration for us is to protect the privacy of Reddit users. Although MPCHAT gathers ‘persona’ data of each speaker in the dialogues, we try not to involve private information. The details are as follows.
1. We manually select the subreddits that are not focused on describing people. The resulting subreddits are mainly about general photography, animals, plants, objects, food, scenery, or activities.
2. We perform automatic data filtering with RetinaFace (Deng et al., 2019) to remove any image with a human face with confidence ≥ 0.9.
3. We automatically detect license plates using an open source detector6 and filter out corresponding images with confidence ≥ 0.5.
4. From the dialogue text, we delete any URL and email address (detected by “@”) to avoid
6https://github.com/ThorPham/ License-plate-detection
mentioning any explicit references to SNS IDs or email addresses.
Harmful contents. We also filter out offensive, insulting, or threatening content with the following steps:
1. We manually select only non-NSFW(i.e., not safe for work) subreddits.
2. Within the curated subreddits, we do not include posts with over 18 tags.
3. We perform automatic data filtering through InceptionV3 (Szegedy et al., 2016) from an open source model7 with confidence ≥ 0.031. All data instances that include images classified into porn or hentai are discarded.
4. We automatically filter out persona imagesentence pairs and dialogues that contain offensive words, as introduced in Appendix A.1.
The above protection schemes can effectively reduce the probability of including personally identifiable information (PII) or NSFW in MPCHAT, but we cannot guarantee a zero possibility. Hence, we manually checked and excluded any images containing PII or NSFW content prior to the public release of MPCHAT. Out of 153K images, only 0.6% (938 images) were filtered out. To provide further details, 364 images contained face information, 8 images contained NSFW content, and 580 images contained license plate information. Note that our filtering process was thorough, going as far as excluding images with partially visible faces or reflections caused by glasses in the case of face detection. Similarly, we eliminated images with unidentifiable plates due to high vehicle speed or low image quality.
Consent. The consent of Reddit users to collect their data is achieved through the Reddit API Terms of Use, based on which users expect that their posts will be publicly available on Reddit and can be downloaded through Reddit API. However, they do not explicitly agree on data usage of MPCHAT and any related research. To mitigate this issue, we only distribute URLs instead of images. We also have an official request form that Reddit users can ask us for data removal. Furthermore, our data’s commercial and for-profit uses are restricted – it should be only available for academic purposes.
7https://github.com/GantMan/nsfw_model
Human annotation. During human annotation, all workers have agreed to the statement of consent prohibiting personal use of the data shown to them. Also, they have agreed to comply with the Reddit User Agreement and Privacy Policy and the Reddit API Terms of Use.
We ensured that our annotators were paid a fair wage of approximately $16/hour, which is higher than the minimum wage in the countries where we recruited annotators from. The time to complete each task was determined as 15 seconds by running multiple trials with researchers, and the payment per task was then calculated as $ 0.07 from this time. Overall the cost per datapoint was approximately $0.21.

heading: B Further Analyses on MPCHAT

heading: B.1 Comparing Persona in MPCHAT and PersonaChat
text: Figure 5 shows examples of persona of each dataset: MPCHAT and PersonaChat. Persona in ours reveal one’s episodic memory, such as a computer setup at Christmas or playing with a dog in the water. Furthermore, persona images provide visual information that complements textual information.

heading: B.2 Statistics of MPCHAT
text: Table 6 summarizes the statistics of MPCHAT. Thanks to Reddit’s abundant sources, the average number of persona image-sentence pairs per
user is more than 14. Table 7 compares MPCHAT with other image-grounded dialogue datasets. Only MPCHAT deals with multimodal persona consisting of both sentences and images. Despite the similar number of dialogues, the total number of unique images is larger in MPCHAT than in PhotoChat, IGC, MDD and VisualDialog. Furthermore, the average response length of MPCHAT is the largest among other image-grounded dialogue datasets.
   [Main HIT] Dialogue Consistency Labeling: Does the (image & text) PERSONAL MEMORY entail the RESPONSE?

heading: C Experiment Details
text: C.1 Implementation Details for Three Tasks In all experiments, we use AdamW optimizer with β1 = 0.9, β2 = 0.999, ϵ = 1e
−8. We use decoupled weight decay of 0.05 in all experiments. We do not use linear warmup steps. We search for the best hyperparameters by testing six different learning rate values (1e−6, 2e−6, 3e−6, 1e−5, 2e−5, 3e−5). Regardless of learning rate values, we use a linear scheduler that decreases the learning rate linearly to 0.
We conduct all finetuning experiments on a single NVIDIA Quadro RTX 6000 GPU. For all experiments, we utilize 13 different random seeds for repeated trials: we then report the average scores and standard deviations. The number of total parameters for SBERT+ViT, SBERT+CLIP, and CLIP+CLIP models are 376M, 376M, and 366M.

heading: C.1.1 Next Response Prediction
text: We train all models for 5 epochs (approximately 12K steps) with batch size 8. For SBERT+ViT
and SBERT+CLIP, we set learning rate to 1e−5. This takes approximately 2.5 GPU hours. For CLIP+CLIP, we set the learning rate to 3e−6. Training this model takes approximately 4 GPU hours. Note that it takes less time to train SBERT+ViT and SBERT+CLIP than to train CLIP+CLIP since the image encoder parameters are not updated during training for the former models, whereas they are updated for the latter.

heading: C.1.2 Grounding Persona Prediction
text: In both response and no-response cases, we train all models for 5 epochs (approximately 4K steps) with batch size 8. For SBERT+ViT and SBERT+CLIP, we set learning rate to 1e−5. It takes approximately 1 GPU hour. For CLIP+CLIP, we set learning rate to 3e−6, taking approximately 1.5 GPU hours. Note that the number of total parameters reduces at no-response case: 310M, 310M and 303M for SBERT+ViT, SBERT+CLIP and CLIP+CLIP.

heading: C.1.3 Speaker Identification
text: All models are trained over a period of 5 epochs, which is equivalent to approximately 7.5K steps, using a batch size of 8. For SBERT+ViT and SBERT+CLIP, we set learning rate to 1e−5 and 2e−5 each which takes approximately 4 GPU hour. As for the CLIP+CLIP, the learning rate is set at 3e−6, and it takes roughly 5 GPU hours to complete the training.

heading: C.2 Licenses
text: We state the licenses that we used, corresponding to the code and models used in this study. First, we used codes that are distributed under
1. MIT license: CLIP8, RetinaFace9 10 InceptionV311
2. Apache license 2.0: ViT, BERT 12
We could not find the license for the license plate detection code, but the code was from a public GitHub repository. Also, Yolo v3, used in license plate detection, has a GNU General Public
8https://github.com/openai/CLIP/blob/main/ LICENSE
9https://github.com/biubug6/Pytorch_ Retinaface/blob/master/LICENSE.MIT
10https://github.com/redcaps-dataset/ pytorch-retinaface/blob/master/LICENSE.MIT
11https://github.com/GantMan/nsfw_model/blob/ master/LICENSE.md
12https://github.com/huggingface/transformers/ blob/v4.17.0/LICENSE
License v3.0 13. Since all the licenses include permissions for commercial use, modification, distribution, patent use, and private use of the artifacts, we comply with the regulations of the above licenses.

heading: D Further Analyses on Experiments

heading: D.1 Ablation Study based on Textual Persona-Response Similarity
text: Previously, we observed that conditioning on persona sentences yielded better performance compared to conditioning on persona images in the next response prediction (§ 6.1) and the speaker identification (§ 6.3) tasks. We hypothesize that dialogue models tend to retrieve responses based on textual similarities, such as lexical or semantic similarity, between the response r and persona sentences P t. Conversely, we assume that dialogue models face challenges in retrieving responses (or speakers) when this textual similarity is low, where persona images P i may contain useful hints.
To investigate the importance of persona images in specific dialogue instances, we split the test set as follows: for each instance, we calculate F1 score between the response r and persona sentences P t = {pt1, ..., ptm}: F1rt1 ,...,F1rtm . We then identify the maximum F1 value and split them using a specific threshold (i.e., 0.3). We refer to dialogue instances with lower F1 scores as the low-f1 subset,
13https://github.com/ultralytics/yolov3/blob/ master/LICENSE
while the remaining instances form the high-f1 subset. In the next response prediction task (or the speaker identification task), the low-f1 subset contains 571 (or 284) instances, while the high-f1 subset consists of 1,750 (or 1,255) instances. For each subset, we measure the performance gap between dialogue models with full inputs and models without persona images, as shown in Table 8.
All models perform better in the high-f1 subsets compared to the low-f1 subsets. In both tasks, the models demonstrate improved performance in the high-f1 subsets compared to the low-f1 subsets, providing evidence that persona sentences P t are utilized as valuable cues for predicting the response or speaker.
The performance gaps are more pronounced in the low-f1 subsets than in the high-f1 subsets. The performance gaps between the models with full inputs and the models without persona images are larger in the low-f1 subsets. This indicates that textual information from persona sentences tends to be less helpful, while visual information from persona images P i becomes crucial for predicting the gold response or speaker in such cases.
In conclusion, persona images play a critical role, particularly when persona sentences fail to
provide useful cues for predicting the responses or speakers.

heading: D.2 Ablation Study on Persona-Consistency in Grounding Persona Prediction Task
text: Grounding persona prediction task is designed to ensure both multimodal context-awareness and multimodal persona-consistency, as mentioned in § 4. We focus on evaluating multimodal personaconsistency by excluding context information as shown in Table 9.
Omitting context information significantly lowers performance. Models without c perform worse compared to models with either c, P̄ or c, r, P̄ , highlighted in gray. This result highlights the crucial role of context information in the grounding persona prediction task. Nevertheless, models without c can still achieve a recall rate of over 50% in predicting the persona element p∗ at Recall@1, showing the task’s persona-consistent characteristics.
Still, using both remainder persona images P̄ i and persona sentences P̄ t maximizes performance. Models equipped with both P̄ i and P̄ t achieve the highest scores in terms of Recall@1 and MRR scores, indicating the importance of leveraging multimodal persona information to its full extent. In addition, note that the results indicate that P̄ i contributes more signifcantly to model improvement compared to P̄ t.
In summary, the results illustrate the grounding persona prediction task’s ability to capture personaconsistent traits. That is, the model exhibits the
capability to predict persona element p∗ by only leveraging the remainder persona set P̄ .

heading: E Coverage of Domains
text: For both the text and image data in MPCHAT, their coverage of domains is a subset of Reddit posts. To be more precise, the content of MPCHAT is derived from subreddits listed in Appendix E.1 and Appendix E.2.

heading: E.1 List of all subreddits for personas
text: We list all subreddits curated for multimodal persona collection. There are 648 subreddits for all multimodal personas, consisting of 140,658 imagesentence pairs, including 16,327 pairs used to obtain persona entailment labels.
pics (7274), cats (7172), aww (6785), succulents (5372),
houseplants (4957), gardening (4805), crochet (4135), baking
(3275), aquariums (3018), food (2489), sneakers (2069), some-
thingimade (2018), foodporn (1885), mildlyinteresting (1576),
breadit (1489), thriftstorehauls (1431), rabbits (1398), foun-
tainpens (1341), crafts (1293), guineapigs (1293), bicycling
(1204), woodworking (1171), embroidery (1142), blackcats
(1135), quilting (1118), cakedecorating (1107), dogpictures
(1097), bladesmith (1094), plantedtank (1016), bettafish (984),
knives (946), indoorgarden (875), knitting (828), crossstitch
(819), coins (810), blacksmith (806), trees (748), plantclinic
(744), cactus (737), squirrels (714), catpictures (680), rarepup-
pers (669), itookapicture (658), parrots (642), redditlaqueris-
tas (621), mechanicalkeyboards (604), earthporn (602), or-
chids (597), sewing (590), plants (577), castiron (570), corgi
(569), tea (565), proplifting (551), pitbulls (550), tonights-
dinner (550), snakes (549), fishing (543), sourdough (533),
photocritique (533), husky (515), eyebleach (498), beerporn
(487), horses (475), hotpeppers (470), spiders (465), reptiles
(453), mycology (445), knifeclub (439), shittyfoodporn (419),
beardeddragons (405), knifemaking (394), brochet (391), ger-
manshepherds (368), pizza (355), watches (353), silverbugs
(345), shrimptank (343), flyfishing (340), lookatmydog (328),
backyardchickens (327), bulldogs (324), casualknitting (318),
pottery (311), crystals (303), cakewin (298), cocktails (298),
birding (292), smoking (274), vinyl (266), vegetablegardening
(262), dachshund (258), hamsters (255), guns (246), hiking
(245), flowers (243), campingandhiking (241), cookiedecorat-
ing (241), bbq (238), savagegarden (237), equestrian (236),
vegan (232), chickens (226), bonsai (221), grilling (220), bird-
pics (219), airplants (218), supermodelcats (217), lego (213),
diy (209), tools (206), barista (205), tarantulas (205), reef-
tank (205), eatsandwiches (204), ceramics (199), trucks (196),
camping (193), duck (192), amigurumi (191), yarnaddicts
(191), drunk (188), pyrex_love (185), spaceporn (183), bul-
letjournal (182), spiderbro (180), carporn (178), spicy (177),
subaru (176), cozyplaces (176), 3dprinting (175), wirewrap-
ping (175), fixedgearbicycle (174), dessertporn (172), bat-
tlestations (170), bikecommuting (169), chihuahua (167), edc
(165), steak (163), cheesemaking (161), catloaf (160), na-
tureisfuckinglit (156), pugs (156), metaldetecting (156), floof
(155), interestingasfuck (154), gamecollecting (154), home-
stead (152), rats (151), zerowaste (151), haworthia (150), tuxe-
docats (149), mineralporn (149), kayaking (147), rainbowev-
erything (144), burgers (142), 1200isplenty (135), pomerani-
ans (135), miata (134), monstera (134), outdoors (134), model-
makers (134), insects (131), leathercraft (129), tuckedinkitties
(128), travel (128), flytying (128), jeep (127), goldenretrievers
(125), sailing (125), herpetology (124), cat (121), curledfeet-
sies (121), cakes (121), bassfishing (121), journaling (120),
chefknives (118), frogs (118), greatpyrenees (117), metalwork-
ing (115), delightfullychubby (115), turning (114), macarons
(113), leopardgeckos (113), microgrowery (112), marijuanaen-
thusiasts (111), kitting (110), penmanshipporn (110), christ-
mas (109), sneks (108), mid_century (108), plantidentifica-
tion (108), vans (107), autos (105), sonyalpha (103), hand-
writing (102), rockhounds (102), pens (100), fermentation
(100), mealprepsunday (97), exposureporn (96), ferrets (95),
hunting (95), veganfoodporn (95), terrariums (95), plantsand-
pots (95), hoyas (93), golf (91), astrophotography (91), tor-
ties (90), justrolledintotheshop (90), beginnerwoodworking
(90), watchescirclejerk (89), vintageaudio (89), mostbeauti-
ful (88), takeaplantleaveaplant (88), doggos (88), upcycling
(86), catbellies (86), entomology (85), wildlifephotography
(84), bostonterrier (83), ramen (83), astronomy (83), funkopop
(82), cockatiel (82), sushi (81), wicked_edge (81), woodcarv-
ing (81), 4runner (81), ballpython (80), randomactsofpolish
(80), longboarding (79), antiques (77), muglife (76), botan-
icalporn (76), chonkers (76), seniorkitties (75), awww (75),
aviation (75), gunpla (75), jigsawpuzzles (74), crestedgecko
(73), lithops (73), awwnverts (73), hotsauce (72), goldfish (72),
bmw (72), needlefelting (71), foraging (71), jewelrymaking
(71), canning (70), veganrecipes (70), classiccars (70), 4x4
(69), homebrewing (69), vegetarian (69), damnthatsinteresting
(69), jewelry (68), aquaticsnails (68), sousvide (68), ama-
teurphotography (68), bordercollie (68), weed (67), amateur-
roomporn (67), welding (67), dessert (67), crh (66), seriouse-
ats (65), vandwellers (65), whiskey (63), siberianhusky (63),
mustang (63), beagle (63), kayakfishing (62), plant_progress
(62), mead (62), covidcookery (61), drunkencookery (61),
budgies (61), skyporn (60), puppysmiles (59), snails (59),
catsareassholes (59), chinesefood (59), beforenafteradoption
(59), fishing_gear (59), australiancattledog (59), cottagecore
(59), panporn (58), roses (58), shiba (58), projectcar (58),
workbenches (58), labrador (57), turtle (57), oldmandog (56),
dumpsterdiving (56), charcuterie (55), analog (55), airsoft
(55), siamesecats (55), audiophile (54), ar15 (53), knifeporn
(53), swords (53), ntbdbiwdfta (53), jarrariums (53), geckos
(53), illegallysmolcats (52), bakingnoobs (52), cupcakes (52),
nails (52), vintage (52), australianshepherd (52), skiing (52),
breakfastfood (51), hotwheels (51), mushrooms (51), climb-
ing (51), birdsofprey (51), landscaping (51), pourpainting
(51), pothos (51), hedgehog (50), grilledcheese (50), cich-
lid (50), polymerclay (50), cheese (50), healthyfood (50),
dunksnotdead (50), kitchenconfidential (49), abandonedporn
(49), beekeeping (49), wildernessbackpacking (49), discgolf
(49), aquascape (49), superbowl (48), honda (47), propagation
(47), shrooms (47), origami (46), aquarium (46), multicopter
(46), malelivingspace (45), ford (45), macroporn (45), dvdcol-
lection (45), butterflies (44), xbiking (44), functionalprint (44),
flashlight (44), cityporn (43), volkswagen (43), bikesgonewild
(43), gshock (43), bushcraft (42), cricut (42), matureplants
(42), lockpicking (42), ketorecipes (42), gardenwild (42), bees
(41), animalporn (41), retrogaming (41), interiordesign (40),
stance (40), harley (40), aldi (40), volvo (40), guitarpedals
(40), drums (39), toyotatacoma (39), handtools (39), wine (38),
absoluteunits (38), cherokeexj (38), beadsprites (38), slow-
cooking (38), resincasting (38), vexillology (38), dog (37),
drunkknitting (37), foxes (37), pug (37), chameleons (37), vis-
iblemending (36), beerandpizza (36), wigglebutts (36), mini
(36), mountainbiking (36), headphones (35), whiskyporn (35),
bathandbodyworks (35), espresso (34), pelletgrills (34), soap-
making (34), velvethippos (34), salsasnobs (34), moths (34),
axolotls (34), wellworn (33), backpacking (33), cassetteculture
(33), waltdisneyworld (33), sanpedrocactus (33), mainecoons
(32), whiskeytribe (32), geology (31), blop (31), shihtzu (31),
shittyveganfoodporn (31), sharks (31), antkeeping (31), cute
(31), homedecorating (31), begonias (31), owls (31), wran-
gler (31), rolex (31), dobermanpinscher (30), mushroomgrow-
ers (30), greatdanes (30), actionfigures (30), paintball (29),
chinchilla (29), catsandplants (29), bookshelf (28), perfect-
fit (28), roastmycar (28), glocks (28), golfgti (28), porsche
(28), retrobattlestations (28), planetzoo (28), canadaguns (28),
catswithjobs (27), mazda3 (27), mazda (27), keto_food (27),
kombucha (27), disneyland (27), rccars (27), transformers
(27), guitars (27), greyhounds (26), weaving (25), craftbeer
(25), buyitforlife (25), budgetaudiophile (25), electricians (25),
osha (25), snowboarding (25), catsmirin (25), catsinsinks (25),
scotch (24), hometheater (24), composting (24), gunporn (24),
glassheads (24), ants (24), teaporn (24), breakfast (23), fish
(23), pokemontcg (23), toyota (23), dualsport (23), tastyfood
(22), nikon (22), bonecollecting (22), gravelcycling (22), trains
(22), bento (22), boxer (22), audi (22), waterporn (21), boating
(21), formula1 (21), nebelung (21), bookhaul (20), modeltrains
(20), femalelivingspace (20), techsupportgore (19), power-
washingporn (19), soup (19), guitarporn (19), reloading (19),
natureporn (19), poodles (19), philodendron (19), typewriters
(18), tinyanimalsonfingers (18), archery (18), mechanicalpen-
cils (18), firearms (18), gamingpc (18), carpentry (18), otters
(18), scooters (18), vintageapple (18), fordranger (17), tacos
(17), cameras (17), subaruforester (17), bernesemountaindogs
(17), amiibo (17), cartalk (17), toolporn (17), glutenfree (17),
tortoise (17), trailrunning (17), tequila (16), chefit (16), analog-
community (16), luthier (16), bmx (16), tacobell (16), mantids
(16), vhs (16), roomporn (15), fiddleleaffig (15), gameboy
(15), macrame (14), designmyroom (14), lizards (14), book-
porn (14), bengalcats (14), frenchbulldogs (14), sloths (14),
comicbookcollecting (14), hockeyjerseys (14), starwarscollect-
ing (14), instantpot (14), seiko (14), polaroid (14), machinists
(14), shroomid (14), coffeestations (13), geologyporn (13),
icecreamery (13), wrx (13), hvac (13), ender3 (13), carnivo-
rousplants (13), architectureporn (13), camaro (13), massef-
fect (13), balisong (13), tamagotchi (13), ft86 (13), farming
(12), urbanexploration (12), f150 (12), shroomers (12), per-
maculture (12), cabinporn (12), beerwithaview (12), ruralporn
(12), wewantplates (12), samoyeds (12), sigsauer (12), jdm
(12), cornsnakes (12), gold (11), photographs (11), crows
(11), nerf (11), rottweiler (11), blender (11), sffpc (11), supre-
meclothing (11), gemstones (10), homelab (10), pebble (10),
longrange (10), villageporn (10), ak47 (10), playingcards (10),
tfablineporn (10), mushroomporn (9), jellyfish (9), tiedye (9),
winterporn (9), corvette (9), volumeeating (9), liberalgunown-
ers (9), warhammer (8), goldendoodles (8), skateboarding (8),
animefigures (8), czfirearms (8), dirtbikes (8), simracing (8),
siberiancats (8), averagebattlestations (8), cubers (8), bassgui-
tar (8), budgetfood (7), fireporn (7), streetphotography (7),
birdphotography (7), legostarwars (7), vinyljerk (7), regular-
carreviews (7), petmice (7), homegym (7), synthesizers (7),
motorcycleporn (7), telescopes (6), cider (6), schnauzers (6),
fossilporn (6), birds (6), plantbaseddiet (5), tractors (5), awwd-
ucational (5), infrastructureporn (5), melts (5), helicopters (5),
lightsabers (5), mousereview (5), mercedes_benz (5), motor-
cycle (5), unclebens (5), liminalspace (5), seaporn (4), berries
(4), houseporn (4), microgreens (4), crtgaming (4), focusst
(4), machineporn (4), thedepthsbelow (3), pkmntcgcollections
(3), boatporn (3), autumnporn (3), f1porn (3), desksetup (3),
microporn (2), nfa (2), squishmallow (2), onewheel (2), bridge-
porn (1), desertporn (1), underwaterphotography (1), castles
(1), weatherporn (1), workspaces (1)

heading: E.2 List of all subreddits for dialogues
text: We list all subreddits curated for dialogue collection. There are 110 subreddits in total for the 15,000 dialogues.
pics (1287), cats (1075), cakedecorating (771), bladesmith
(472), houseplants (440), gardening (414), itookapicture (400),
breadit (363), tonightsdinner (313), crochet (312), succulents
(309), bicycling (275), guineapigs (256), aquariums (246), diy
(244), mildlyinteresting (226), sneakers (212), rabbits (210),
baking (198), crossstitch (186), burgers (182), casualknitting
(181), earthporn (180), fountainpens (178), embroidery (172),
grilling (171), rarepuppers (167), camping (166), ceramics
(163), cocktails (163), blackcats (162), bassfishing (158), tea
(152), dogpictures (148), husky (148), cakewin (144), hik-
ing (132), zerowaste (130), cookiedecorating (128), food
(125), brochet (118), parrots (113), cheesemaking (109), up-
cycling (109), plantedtank (109), bikecommuting (107), thrift-
storehauls (104), flyfishing (100), corgi (98), crystals (93),
snakes (91), mechanicalkeyboards (89), coins (85), horses
(77), pitbulls (77), eyebleach (77), chickens (76), squirrels
(75), dachshund (73), duck (69), beardeddragons (69), quilt-
ing (68), bulldogs (65), germanshepherds (61), foodporn (58),
barista (57), pomeranians (55), catpictures (55), reptiles (53),
castiron (53), blacksmith (51), kayaking (51), watches (51), in-
doorgarden (50), greatpyrenees (49), campingandhiking (47),
workbenches (47), lookatmydog (43), chinesefood (42), eques-
trian (40), battlestations (40), sewing (40), photocritique (40),
hotpeppers (40), pizza (39), sourdough (37), sailing (36), or-
chids (36), trucks (35), vinyl (34), plants (33), cozyplaces
(33), bettafish (32), cactus (32), beerandpizza (29), spiders
(29), charcuterie (24), pug (21), veganrecipes (19), knives
(18), doggos (18), amateurphotography (17), mycology (17),
fishing (17), villageporn (5), infrastructureporn (2), desertporn
(1), awwducational (1), seaporn (1), f1porn (1)
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations
3 A2. Did you discuss any potential risks of your work? Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract, Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3
3 B1. Did you cite the creators of artifacts you used? Section 3.1, Section 3.3, Section 3.5, Section 5, Section A.2, Section A.4
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section A.4, Section C.2
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Ethics Statement, Section A.4, Section C.2
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Ethics Statement, Section A.4
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Limitations, Section A.3.4, Section A.4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3.5, Section 6.4, Section B.2, Section D.1
C 3 Did you run computational experiments? Section 6, Section C, Section D
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section C.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section C.1
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 6, Section C.1
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section C
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3.3
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Section A.3
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 3.3, Section A.3, Section A.4
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Section A.4
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? We did not apply for approval from an ethics review board. However, our work does not include human subjects because we did not collect identifiable information nor directly interact with the authors of Reddit content. In addition, we have gone to great lengths to remove offensive or sensitive materials from the data before the annotation. Thus, we concluded that our data collection process caused no legal or ethical issues for the authors of the Reddit content or the annotators.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Limitations, Section A.3.4

heading: Future Work
text: One interesting future work would be to expand MPCHAT in both the size (e. g. , scaling up the number of dialogues and personas) and the scope (e. g. , adding audio/video modality)