Multimodal Persona Based Generation of Comic Dialogs
Year: 2023

Abstract:
We focus on the novel problem of persona based dialogue generation for comic strips. Dialogs in comic strips is a unique and unexplored area where every strip contains utterances from various characters with each one building upon the previous utterances and the associated visual scene. Previous works like DialoGPT, PersonaGPT and other dialog generation models encode two-party dialogues and do not account for the visual information. To the best of our knowledge we are the first to propose the paradigm of multimodal persona based dialogue generation. We contribute a novel dataset, COMSET, consisting of 54K strips, harvested from 13 popular comics available online. Further, we propose a multimodal persona-based architecture, MPDIALOG, to generate dialogues for the next panel in the strip which decreases the perplexity score by ∼10 points over strong dialogue generation baseline models. We demonstrate that there is still ample opportunity for improvement, highlighting the importance of building stronger dialogue systems that are able to generate persona-consistent dialogues and understand the context through various modalities.

heading: 1 Introduction
text: Multimodal conversational agents build dialog systems that engage with modalities beyond text, in constructing next responses. They open up a novel direction of text-vision multimodality, where the agent is part of the scene, rather than being a distant observer. This facilitates research and creation of support based multimodal agents. These agents could be critical for various applications such as assistants for visually impaired, conversations with robots in physical settings, instruction following by a digital agent that is manipulating images, clarification discussions during a presentation and so on. Such agents can help to promote literacy and language skills, as users engage with the generated dialogue to create their own stories. In all such cases, a natural conversational experience will be
emulated better if visual or other modal elements get incorporated in the AI models.
There is substantial recent research in building neural conversational AI systems for text-only taskoriented dialogues (Eric et al., 2017; Madotto et al., 2018; Wu et al., 2018, 2021; Hosseini-Asl et al., 2020; He et al., 2022) as well as open domain conversations (Gao et al., 2020; Zhang et al., 2020; Santra et al., 2021; Shuster et al., 2022). On the other hand, research on multimodal conversation is still in its early stages. A key exception is Visual Dialog (Das et al., 2017), where an agent answers multi-turn questions about a single static image. However, to the best of our knowledge, there is little work that builds dialog systems with multiple evolving images.
Our goal is to advance research in such multimodal dialog systems. A particular domain that enables us to study this is that of comic books. In contrast with Visual Dialog, a comic strip has several images with temporal progression and an aligned dialog. Building an effective comic dialog
14150
system necessitates understanding the visual narrative, in addition to the textual context, making it a good testbed for multimodal dialog.
In addition to multimodality, comics have several other unique characteristics that make the domain challenging for AI systems. For instance, comic conversations are often multiparty, whereas most existing dialog agents assume dyadic (two party) conversations. Moreover, each character in a comic has a distinctive persona and style, and the dialog agent has to learn to follow the right style for each speaker. Finally, many comics are humorous, necessitating the model to be funny in its responses.
To study dialog systems in the comics domain, we first curate a novel dataset, COMSET, which consists of ∼54K strips from 13 comics. Each strip is associated with the visual panel (with text masked), along with the text transcript. We harvest strips from a publicly available online collection, GoComics.1 Panel and dialogue segmentation on the visual scene data in these strips leads to a dataset with 200+ characters. To describe the distinctive persona of each lead character, we also curate a set of persona facts (inspired by Zhang et al. (2018)) from popular fandom websites.
We define the novel task of next utterance generation for comics conditioned on the textual dialog history, visual scene history, and the persona facts for the comic characters. Fig. 1 shows an example. Since existing dialogue generation models do not handle multi-image multimodal context along with persona, we implement a novel method (MPDIALOG) for the task, as illustrated in Fig. 4. Text utterances, persona facts, and visual scenes are passed into the MultiModal Embedding (MME) module which encodes them into tokens each of D=768 dimensions. These embeddings are then passed on to a language decoder to produce the output tokens. MME module (i) computes the text encodings using a text embedding (TE) layer, (ii) computes visual token embeddings of panel images using CLIP Vision encoder (VE), linearly projects (LP) each embedding of size D to n×D and reshaping it to n tokens each of size D, (iii) interleaves text and visual token embeddings. Interleaving occurs such that the dialogues of a panel are preceded by the respective panel embedding. Extensive comparisons show that MPDIALOG outperforms multiple text-only dialogue generation systems as well as those systems that do not use persona facts.
1https://gocomics.com/
Overall, we make the following main contributions in this work. (1) We contribute a novel multimodal comics dataset, COMSET, containing ∼54K strips and persona facts for 200+ characters. (2) We propose a multimodal persona-based dialog generation baseline, MPDIALOG, which incorporates both the modalities and generates the next utterances effectively. (3) We demonstrate empirically that multimodality and persona orientation leads to better dialogues. This paper adds interesting questions around multimodal persona-based dialogue generation modeling and we hope that our study motivates more work in this area. We make code and dataset publicly available.2

heading: 2 Related Work
text: Our work is related to the following three areas: dialogue generation, multimodal models, and multimodal datasets for dialogue generation. Dialogue Generation: Recently, several neural dialog generation models have been proposed (Gao et al., 2018; Ni et al., 2022); we discuss a few here. DialoGPT (Zhang et al., 2020) uses a GPT2 (Radford et al., 2019) decoder pretrained on Reddit conversations and can effectively capture the contextual information in dialogues, thereby generating interesting and human-like responses. However DialoGPT does not allow explicit style control over the generated responses. EDGE (Gupta et al., 2021) allows for controlled response generation by conditioning on semantic frames of exemplar responses. A particular kind of style control models are persona-based models which use “persona” information for personalized dialog generation. Bert-over-Bert (Song et al., 2021) disentangles persona-based dialogue generation into two tasks: dialogue generation and consistency understanding; the model uses a shared BERT encoder but has two task-specific decoders. PersonaGPT (Tang et al., 2021) uses GPT-2 finetuned on PersonaChat (Zhang et al., 2018) dataset, with added persona fact tokens for personalized generation and question control codes for controlled generation. None of these models capture the multimodal multi-party context which is the setting for comic dialogues. Multimodal Datasets for Dialogue Generation: The COMICS (Iyyer et al., 2017) dataset contains scanned images of comic strips but it does not contain manually extracted transcript information or
2https://github.com/dair-iitd/MPdialog
information about comic characters. Further, as the authors mention, the dataset is unsuitable for generation tasks due to OCR detection inaccuracies. PersonaChat (Zhang et al., 2018) has conversations between two agents and their corresponding persona facts but it has no images. Other multimodal datasets include ImageChat (Shuster et al., 2020), PhotoChat (Zang et al., 2021) and VisualDialog (Das et al., 2017) which have a conversation between speakers about a single reference image. They differ from our setting, where the speakers are themselves a part of the image, and we have multiple panels (images). Multimodal Models: Recently, several types of multimodal encoders and generators have been proposed for a variety of applications. Models like CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) are based on alignment of visual and textual embedding spaces. Frozen (Tsimpoukelli et al., 2021) and ClipCap (Mokady et al., 2021) also align text and visual embedding by projecting visual embeddings onto the textual embedding space. Textimage cross attention is used in VisualGPT (Chen et al., 2021), VC-GPT (Luo et al., 2022), CoCa (Yu et al., 2022). Perceiver-IO (Jaegle et al., 2021) is a fully attentional read-process-write architecture with variants like Uni-Perciever-MOE (Zhu et al., 2022) which use Mixture of Experts for response selection. In SimVLM (Wang et al., 2021) and VisualBERT (Li et al., 2019) the Visual and Textual Models are jointly trained on the task itself. Given its significant zero-shot image classification capabilities, we use CLIP as the image encoder for our MPDIALOG architecture.

heading: 3 The COMSET Dataset
text: We contribute a novel comics dataset, COMSET, containing 13 popular English comic strips, obtained from GoComics.1 Each comic strip contains transcription and an image. We remove duplicate strips (re-broadcasts with minor modifications) based on Levenshtein distance between transcripts. For each comic, we also obtained persona facts (representative personality traits) for each character by manually curating such information from websites like Fandom,3 Wikipedia,4 and TV Tropes,5 and paraphrasing all collected persona facts into first person English sentences. We describe data
3https://www.fandom.com/topics/comics 4https://www.wikipedia.org/ 5https://tvtropes.org/
pre-processing and analysis in this section.

heading: 3.1 Dataset Pre-processing
text: The raw dataset was pre-processed as follows.
Parsing Transcripts: Parsing transcripts involves parsing speaker (character) and utterance pairs from unstructured conversation transcripts. We first obtained a list of comic characters (for our 13 comics) from same websites that were used to gather character personas. We also added character aliases to this list. Further, we mined frequent proper nouns with PERSON entity tag from all transcripts to search for all potential speaker candidates. We reduced infrequent characters into a catch-all character OTHER. Around 17% utterances in our corpus are attributed to OTHER. Further, there were some frequent speakers which were not named entities, for example, Man, Woman, Stranger, Voice, Noise, Sound. We conflated Voice, Noise, Sound into a single speaker (Voice) and added all such characters to list of characters. Finally, for all comics except Doonesbury and Cleats, we used list lookup for extraction of mention spans for character named entities. Using basic heuristics like word followed by colon or quotation characters, we could also do a fuzzy character name match to handle spelling errors in transcripts.
Transcripts for Doonesbury and Cleats contain free-form text like Bucky is holding Smacky and says .... Typically each sentence contains four parts: character/speaker name (Bucky), action or attribute phrase (is holding Smacky and), speaking verb (says, replies, asks, proclaims, etc.6), and utterance. To obtain these parts from transcripts, we first perform part-of-speech tagging, named entity recognition, and dependency parsing using spaCy (Honnibal et al., 2020). Then we use heuristics like (a) speaker name should have the POS tag PROPN, must be the nominal subject (nsubj) and have the NER tag as PERSON, (b) The speaker should have a direct/indirect relation to the speaking verb. Panel Segmentation: Each strip image had several panels and utterances across panels. Classical vision methods like Hough Transform (Duda and Hart, 1972), polygon detection (Li et al., 2014), recursive cuts (Pang et al., 2014) and density gradients (Tanaka et al., 2007) led to poor panel segmentation due to their assumptions about uniform white background and clean gutters. Inspired
6https://archiewahwah.wordpress.com/ speech-verbs-list/
by Iyyer et al. (2017), we model the panel segmentation as an object detection problem. We used the 500 manually annotated panel bounding boxes out of comic strips provided by them to train a Faster-RCNN (Ren et al., 2015) architecture with a ResNet-50 (He et al., 2016) backbone, and used it to segment panels from our comic strips. Some segmentation results are shown in Fig. 2. Dialogue Text Detection and Masking: While predicting the next utterance for a character in the current panel, the ground truth utterance in the panel image could lead to a label leak. Hence, to eliminate redundancies and to avoid possibilities of label leak, we mask the utterance text from panel images. Iyyer et al. (2017) detect utterance text on images by training a Faster-RCNN model on 1500 manually annotated panels to detect text boxes. This approach led to poor results for our dataset since text box structure is not consistent across comics, and often there is no explicit text box or bubble to encapsulate the dialogue, also evident from Figs. 2 and 3. Hence, we used offthe-shelf OCR, specifically EasyOCR,7 to extract the text and bounding boxes from each segmented panel. We filled bounding boxes with random noise so as to not bias the model towards any color at utterance positions, as shown in Fig. 3. Multimodal Alignment: For each comic strip c, panel segmentation yields a sequence of nc panel images along with OCR text {Pj}ncj=1 for each panel j, and transcript parsing yields a sequence of mc utterances {Di}mci=1 along with speaker labels. For next utterance prediction, the model needs both text and visual context aligned with each other. For each (Di, Pj) pair, we calculate a
7https://github.com/JaidedAI/EasyOCR
string fuzzy Levenshtein distance-based similarity score Sij which determines the extent to which Di matches with text Pj . The panel index for the ith utterance is then calculated as σi = argmaxj Sij . The matched panel sequence can be written as Σ = {σ1, σ2, . . . , σm}. Due to inaccurate OCR, Σ may not be monotonically increasing. We handle this inconsistency by transforming Σ to a sorted sequence Σ = DP(Σ) where DP is a dynamic programming method to sort an input sequence with minimum number of edits. We found that the DP filter was needed for only 2% of all the utterances.

heading: 3.2 Dataset Statistics and Quality
text: Across 13 comics, COMSET contains 53,903 strips covering a total of 159,610 panels and 238,484 utterances. Thus, there are 2.96 images per strip. On average, a dialogue contains 16.09 tokens. Each strip has 2.98 characters on average. The dataset contains 6.66 persona facts per character on average across 202 characters. Each persona fact contains 12.23 tokens on average. Table 1 shows key statistics for COMSET. Table 2 shows distribution of number of strips, panels, utterances and characters across the 13 comics. We split the 13 comics into a seen set of 8 comics and unseen set of 5 comics. Seen set was further split randomly 70:10:20 into train:val:test stratified by comic name.
We manually inspected our dataset quality using 50 randomly chosen examples. We found that our scripts for parsing speaker from transcripts had an accuracy of ∼98%. Some comics had bad transcripts, and speaker information was completely missing (<1%). In ∼2% of utterances, there were some parts of the speaker overflowing into the previous utterance due to whitespace in speaker names
(ex. ‘Voice from television’, ‘Person on TV’). Text masking was evaluated on 1000 examples and we found ∼4% of all comics had italicized text, or font size too small, low character spacing, that made it difficult to detect and mask bounding boxes. In ∼3% comics, panel segmentation was challenging due to no clear demarcation between several frames, as depicted in Fig. 7 in the Appendix. In ∼5% of all utterances, the dialogue did not map correctly to its panel primarily due to OCR detection errors. We had also assumed that a dialogue can be mapped one-to-one to a panel which is not always true as a dialogue can sometimes overflow into multiple panels, in which case a panel with the most matching words was chosen. Overall, we find error percentages in each part of the dataset curation pipeline to be low. The end to end accuracy over 200 random datapoints from the test set came out to be 91.5% indicating that the resulting dataset is of high enough quality to study the comic generation task.

heading: 4 Methodology
text: In this section we formalize the next utterance prediction task in the multi-modal persona-based dialogue setting for benchmarking COMSET and propose a novel baseline architecture MPDIALOG.

heading: 4.1 Next Utterance Prediction Task
text: For a comic strip, consider a conversation history with utterances {Ci}ni=1 and an aligned sequence of images {Ij}mj=1. At any time step t, the objective is to generate Ct given the textual conversation history {Ci}t−1i=1 and the corresponding image history sequence {Ij}kj=1 where Ct is aligned with Ik, t ≤ n, and k ≤ m.
In practice, it may be useful to limit historical context to a history size h of past utterances and their corresponding panel images. While this problem formulation is generally applicable to any setting with multimodal conversation history, we
propose a model for next utterance prediction for comics in this work.

heading: 4.2 Baseline Methods
text: We first describe our adaptation to the existing language model (LM) only methods, as well as LM+persona based methods. LM only: LM only methods use only the text part of the conversations. We experiment with DialoGPT (Zhang et al., 2020) and EDGE (Gupta et al., 2021). DialoGPT is trained on a 147M multi-turn dialogue dataset from Reddit, and conditions response generation on the previous conversation context. EDGE (Gupta et al., 2021) allows controlling dialogue response generation based on semantic structure of exemplar responses. During inference, EDGE retrieves the exemplar responses of the test set context with train set dialogues as the candidate set using a ParlAI Polyencoder (Humeau et al., 2019) model8 pretrained on the ConvAI2 dataset. EDGE then uses the opensesame (Swayamdipta et al., 2017) frame extraction model, which is a frame-semantic parser for automatically detecting FrameNet (Baker et al., 1998) frames and their frame-elements from sentences. We adapt these models to COMSET by extracting the conversation history Ct′:t−1 and finetune the model to predict Ct, where t′ = max(0, t−h). We set the maximum history size h = 5. LM+Persona: These baselines utilize the conversation context along with persona facts for each character to generate persona consistent responses. Models evaluated include PersonaGPT (Tang et al., 2021) and BoB (Song et al., 2021). These models assume a dyadic conversation and require persona facts of both the speakers as input to generate responses. PersonaGPT is finetuned on the PersonaChat (Zhang et al., 2018) dataset, with added spe-
8https://parl.ai/projects/polyencoder
cial tokens ([p1], [p2]) to mark the persona facts. Since our problem is different and includes multi party conversations, we provide the persona facts of the speaker of utterance Ct as a prefix to the input (marked by a single special persona token [p]) with the objective of predicting Ct. Thus, we pre-train the models using the target character persona prefix on their respective datasets and later finetune it on COMSET.

heading: 4.3 MPDIALOG Architecture
text: The architecture for MPDIALOG is inspired by Frozen (Tsimpoukelli et al., 2021). It consists of a vision encoder and a language decoder with a linear projection module in between. The vision encoder encodes the visual comic panels into a single vector, which is then projected into visual tokens using the linear projection module and is fed into the language decoder along with text tokens for generation as shown in Fig. 4. We use a CLIP vision encoder (Radford et al., 2021), as it has been shown to be effective in aligning visual embeddings into the same semantic space as text embeddings. Similar to Frozen, we project the panel embedding of D dimensions into an n×D dimensional vector using a linear layer and reshape it into n visual tokens each with embedding size D. We use n=2 in our experiments as suggested in Tsimpoukelli et al. (2021).
As shown in Fig. 4, in the MME (MultiModal Embedding) module, we concatenate the dialog history tokens separated by the end of text separator token ([eot]) and insert visual token embeddings wherever the panel in the conversation changes (including the first) and feed the resulting sequence into the language decoder. When persona informa-
tion is available we prepend the persona tokens to the sequence input of the language decoder, along with the persona start token [p]. The multimodal MME output is fed as input to the GPT-2 language decoder. We train the model using causal language modelling, i.e., auto regressive loss over the target prediction tokens. We use PersonaGPT-base and a pre-trained CLIP vision encoder as the textual and visual components of MPDIALOG respectively, and finetune it on COMSET in an end-to-end fashion. The projection module is simply a linear layer. Unlike Frozen, we do not freeze any component and train the entire architecture endto-end. Once trained we generate responses using nucleus sampling (Holtzman et al., 2019) and set top-p=0.95, top-k=50. Other generation parameters are as follows; temperature=0.05, repetition penalty=1.2 (Keskar et al., 2019).

heading: 5 Experiments and Results

heading: 5.1 Hyper-parameters for Reproducibility
text: All results are computed on a GeForce GTX 1080 Ti (12 GB) cluster with 64 cores each of Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz. There are 12 layers each in both the vision encoder and the text decoder, with 8 multi attention heads in each transformer layer. The vision encoder weights are initialized from openai/clip-vit-base-patch32 available on HuggingFace9 and the text decoder weights are initialized from a PersonaGPT-base model trained on PersonaChat dataset. The model was trained for 3 epochs on 2 GPUs with a learning rate of 5e-5 and a linear decay schedule with an initial warmup of 500 steps using the AdamW (ϵ=1e-8) optimizer on a batch size of 12. Further details can be found in our repository.2

heading: 5.2 Evaluation Metrics
text: We report the results of various baselines and our proposed method on several natural language generation metrics. We report the perplexity score for each model which measures the uncertainty of a model to output the target sequence given the context words. Further, we evaluate the models on lexical metrics like unigram precision, recall and F1 scores as well as neural metrics like BLEURT (Sellam et al., 2020) and MaUde (Sinha et al., 2020). MaUde is a particularly relevant metric as it is curated specifically for dialogue generation and mea-
9https://huggingface.co/
sures the coherence of responses with the previous conversation context.

heading: 5.3 Main Results
text: We design extensive experiments to answer the following questions: (1) Can existing dialogue generation language models adapt their knowledge to the comic setting? (2) To what extent does persona orientation of language models help in generating comics? (3) Does adding multimodality help the language model in better understanding the context and thereby generating coherent responses? (4) How does the generalizability to unseen comics (zero-shot setting) vary across architectures. To answer these questions we finetune each of the baseline language-only models and those with persona alignment on only the textual component and later train MPDIALOG on the multimodal dataset. We generate response for each of the trained models using nucleus sampling. This was done for both the seen (finetuned) and unseen (zero-shot) splits of our dataset. The results for these experiments are shown in Table 3. Performance on Seen Dataset: We observe that the proposed model, MPDIALOG, outperforms both the language model only as well as personabased baselines. Language only models (like DialoGPT and EDGE) cannot generate coherent responses (high perplexity and low MaUde) in the
comic setting. This is expected as it is very hard to understand the context of a comic without any information about the characters or the visual scene. We observe that adding persona information of the characters significantly boosts performance as is evident from the perplexity scores, BLEURT and MaUde, of PersonaGPT-base. We conducted a Welch-T (Welch, 1947) test on results of MPDialog with other baselines for precision, recall, F1, MaUde, BLEURT and we got max(p) < 0.025 indicating statistical significance. Persona information delivers meaningful insights into the context and helps the model in understanding the conversation better. Moreover, adding visual scene information along with the persona also boosts performance as the model has now access to the actual scene of the comic in which the conversation is happening. As an ablation we also trained a model with language and visual components but without the persona information. Its perplexity came out to be 23.76 on the seen set, which is better than just the language part (DialoGPT) but worse than MPDialog (which also incorporates persona). For illustration, we show a cherry-picked example in Fig. 5 where PersonaGPT-base and MPDIALOG are able to align their responses with the character persona, whereas responses for other language only models are either too banal (EDGE and BoB) or completely nonsensical (DialoGPT).
Performance on Unseen Dataset: We also show results on unseen comics set, for various models in Table 3. Again, MPDIALOG outperforms baselines across most metrics. Perplexity scores of language only models (both DialoGPT and EDGE) are quite high. Adding persona information and evaluating PersonaGPT we find that it significantly boosts the models’ ability to generate responses with more certainty, as the perplexity score drops by a factor of ∼12 points. It is worthwhile to note that PersonaGPT performs much better on neural metrics like BertScore and MauDe as compared to language only baselines. Our multimodal model (MPDIALOG) further improves these neural metrics and shows that it is able to produce much more coherent responses by taking into account the associated visual information. Thus the trend (LM + persona + images > LM + persona > LM) can still be observed as in the “seen comics” setting.

heading: 5.4 Comic-wise Quantitative Analysis
text: Table 4 shows comic-level BLEURT scores for both the seen as well as unseen test sets. We also show MaUde and perplexity scores in Tables 5 and 6 respectively. For most comics across all the three metrics, MPDIALOG performs better than other models. Unlike most comic strips, Cleats comic focuses on the relationships between the characters, their sportsmanship and the challenges of being part of a team. We believe that images in Cleats do not contain much additional information and hence multi-modality of MPDIALOG does not lead to improved results.

heading: 5.5 Qualitative Analysis
text: The proposed method, MPDIALOG, is personabased. How well does it capture the persona style in the generations, compared to other persona-based baselines? To answer this question, we perform the following experiment. For every character c in the train set, we obtain its unigram vocabulary distribution Trainc. Given a model, over the entire test set, we also compute unigram vocabulary distribution Outputsc from combined text of all generations for character c. If the model has captured persona for character c well, the symmetric KL-divergence between Trainc and Outputsc should be small. Hence, we compare MPDIALOG with other persona-based baseline models (PersonaGPT and BoB) using the symmetric KL divergence metric. We observe that symmetric KL divergence is 4.41, 4.56 and 3.36 for PersonaGPT, BoB and MPDIALOG respectively. Thus, we infer that MPDIALOG is the best at capturing the persona information.
We also attempt to understand the image patch attribution for a generated dialogue by our model as applied on Fig. 5. We conducted a GradCAM (Selvaraju et al., 2017) analysis to check where the model “looks” while generating its utterances. Since generation is stochastic and dependent on nucleus sampling, we cannot attribute the model’s output to a particular attention map over the image. As a surrogate, we calculate the attention map over the visual panels when the model generates the last [eot] token. In Fig. 5, we were able to observe that the model does indeed look at Arlene’s face and Garfield’s face (as indicated in Fig. 6) and gives less relative importance to the background and the bubble above it. It helped us confirm that our model is able to contextualize within the images as well and generates tokens
based on meaningful and interpretable image features.

heading: 5.6 Human Evaluation Results
text: We obtain manual annotations for the utterances generated by various models on fluency, engagingness, dialog-consistency, scene-consistency and persona-detection. Four annotators performed judgments on a set of 65 examples, randomly sampled from the test set. We compute inter-annotator agreement as pairwise correlation between method rankings, averaged across the five criteria. It was found to be 0.318 (Kendall’s Tau ‘B’) which is considered as strong agreement. Detailed annotation guidelines are mentioned in the appendix. Specifically, we measure persona detection as follows. Given persona facts of two characters, and a response, the annotator is asked to guess which of the twi persona the response matches to. Table 7 shows that MPDIALOG performs best on all measures except for dialog consistency where EDGE performs the best. EDGE uses semantic frame examplars to guide a structure for the uttterance leading to better consistency. All the other models do not make use of this extra structural input, and amongst them, MPDIALOG performs best. On persona detection, MPDIALOG performs comparably to PersonaGPT. Overall, MPDIALOG performs quite well on human perceived quality of generated comic dialogues.
As an additional qualitative analysis for the proposed model, we performed the following experiment. We considered examples, where in the multimodal input context, we changed the last character prompt to some other character from the same or other comic. The goal was to check how would another character (say “Tyr”) respond in a situation in a comic (say “Garfield”). We found that the
generated responses often reflect the persona of the injected character. For example, in Garfield, we found for the same situation: (1) Mom’s response showing her down-to-earth, exasperated and sensitive nature who loves her son dearly, and (2) Susie’s response to be teasing Calvin, thereby showing her love-hate relationship with Calvin. Thus, our model seems to be capturing the persona behavior somewhat, but we feel there is much more work to be done to generate responses that are contextually more coherent, and at the level of human skill.

heading: 6 Conclusions and Future Work
text: We propose a novel problem of next utterance prediction for comics given historical multimodal context consisting of previous utterances and panel images. We contribute a novel dataset, COMSET, which contains 53,903 strips, 159,610 panels and 238,484 utterances from 13 comics. We also propose a multimodal persona-based baseline model, MPDIALOG, which performs better compared to strong language-only and persona-based dialogue generation models, both in the seen comic and the unseen comic settings. We make our code and dataset publicly available2. In the future we plan to (1) focus on generation of humor-focused text, and (2) explore generation of next utterances and panel images together.

heading: Acknowledgements
text: This work is supported by grants by Google, Verisk, and 1MG, an IBM SUR award, and the Jai Gupta chair fellowship by IIT Delhi. We also acknowledge travel support from Google and Yardi School of AI travel grants. We thank the IIT Delhi HPC facility for its computational resources. We also thank Rocktim Jyoti Das for his help with the code for MPDialog.
Limitations
In this paper, we focused on English comics only because of their ease of availability. Although we have not experimented with non-English text, we
expect the proposed model to work well in multilingual settings if we replace GPT-2 decoder with other decoders like BLOOM (Scao et al., 2022).
Ethics Statement
Most of our dataset has been obtained from GoComics (https://gocomics.com/). The website allows downloads of comic images for research purposes. However, they do not allow redistribution of images. Hence, in our dataset release, we have only provided links to images on GoComics website. Providing links to images or webpages is a common trend (e.g., Google Landmarks, GoogleConceptualCaptions, WIT datasets). That said, our code base provides all the scripts needed to (1) do pre-processing and modeling based on this images (2) gather transcripts and align with the panels in comic strips. Thus, overall, all steps in the paper are reproducable. Further, we have also provided character identification annotations that we perform on these images as part of the dataset.
Natural language generation is in general prone to issues like biased, offensive, harmful, misinformative text generation. Fortunately, in this work, we finetune our models using relatively clean comics dataset. Also, given that these generations are meant to be consumed in a humorous form, we do not foresee the bias (if at all) generated by our model to be hurtful. To the extent we browsed over the generations produced by our model, we did not observe any biased, offensive, harmful, misinformative text getting generated.

heading: A Panel Segmentation Errors
text: Our method produced errors where the demarcation between frames was not very clear as shown in a few examples in Fig. 7.

heading: B Annotation details
text: Human annotations were done by four undergraduate Computer Science students (3 male, 1 female) with an interest in comics in the age group 21-22 years. They were paid as per the rules of our institute for the task. The annotators were informed that this data will be used for research on dialogue generation for comics.
The following guidelines were provided to the annotators for evaluation.
• Fluency: How fluent is the response on it’s own? (1-5), where 1 is “not fluent at all”, 5 is “extremely fluent”. Fluency encompasses how easy to understand the response is.
• Engagingness: How much engaging is the response on its own? (1-5), where 1 is “not engaging at all” or “generic”, 5 is “extremely engaging” or “unique”. Engagingness is defined as how interesting and unique the response is. Repetition and generic responses are scored low and highly detailed and attention grabbing responses are scored high.
• Dialog Consistency: How consistent is the response to the dialogue history? (1-5) 1 is “totally unrelated” and 5 is “Fully consistent”.
• Scene Consistency: How much consistent is the response to the image history? (1-5) 1 is “totally unrelated” and 5 is “Fully consistent” and 3 is “OK”.
• Persona Detection: Given persona facts of two characters, which persona does the response match to?
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 8

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1 + Abstract
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 8

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 8

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 8

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 8

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3
C 3 Did you run computational experiments? Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 5
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 5

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix C

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Appendix C

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix C
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? There is no personally identifiable information in the dataset. Hence, no specific ethics review was needed.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix C

heading: Future Work
text: In the future we plan to (1) focus on generation of humor-focused text, and (2) explore generation of next utterances and panel images together