SlowGuess/NAACL_2022_OCR huggingface dataset:
Seungju Han† Beomsu Kim† Jin Yong Yoo† Seokjun Seo Sangbum Kim Enkhbayar Erdenee Buru Chang* Hyperconnect
{seungju.han,beomsu.kim,jeffrey,seokjun.seo,airdish,enkhbayar.erdenee, buru.chang}@hpcnt.com
In this paper, we consider mimicking fictional characters as a promising direction for building engaging conversation models. To this end, we present a new practical task where only a few utterances of each fictional character are available to generate responses mimicking them. Furthermore, we propose a new method named Pseudo Dialog Prompting (PDP) that generates responses by leveraging the power of large-scale language models with prompts containing the target character's utterances. To better reflect the style of the character, PDP builds the prompts in the form of dialog that includes the character's utterances as dialog history. Since only utterances of the characters are available in the proposed task, PDP matches each utterance with an appropriate pseudo-context from a predefined set of context candidates using a retrieval model. Through human and automatic evaluation, we show that PDP generates responses that better reflect the style of fictional characters than baseline methods.
How would you feel if you could talk to your favorite character?
In recent years, open-domain conversation models (Adiwardana et al., 2020; Roller et al., 2021) have achieved remarkable progress with the development of large-scale language models (Radford et al., 2019; Brown et al., 2020). Meanwhile, recent studies have suggested several directions reflecting desirable traits of real-life conversation to make open-domain conversation models more engaging beyond plain chit-chat. Style-controlling conversation models generate responses in the target styles such as emotion (Zhou et al., 2018; Demszky et al., 2020) and empathy (Rashkin et al., 2019). Personagrounded conversation models (Zhang et al., 2018a; Kim et al., 2020; Majumder et al., 2020) produce
Figure 1: Illustration of PDP. The retriever matches pseudo-context for utterances from the character, and utilizes them in a prompt while generating the response.
responses that preserve consistent personalities by leveraging personal descriptions (e.g., "I have two dogs"). In this paper, we consider mimicking fictional characters as a promising direction for building engaging conversation models.
When it comes to building conversation models that mimic fictional characters, two major challenges prevent us from directly applying previous models designed for conditional response generation: (1) It is difficult to define fictional characters with only a few descriptions, as in person-grounded conversation models. Furthermore, it is not expressive enough to represent characters' styles with discrete labels (e.g., angry, happy), as style-controlling conversation models do. (2) There lacks sufficient dialog data of fictional characters for training conversation models. It is inefficient to manually create dialog datasets of characters for training, especially considering that additional data is needed for each new character.
To address these two challenges, we propose a new task where only a few utterances of the fictional characters are available to generate responses mimicking the characters. Such setting is justified by the two following reasons: (1) Utterances of fictional characters provide useful clues for gen
Human Language Technologies, pages 5114 - 5132
erating responses mimicking the characters as the personal traits or styles of speakers are inherent in their utterances (Boyd et al., 2020; Li et al., 2020). (2) Collecting only a few utterances of target characters is a cost-effective scenario compared to constructing the full dialog data consisting of context and utterance pairs; this allows us to extend our method to a new character easily.
To perform the task, we introduce Pseudo Dialog Prompting (PDP), a method that builds prompts using a few numbers of target characters' utterances to leverage the power of pre-trained language models. We claim that designing the prompt in the form of dialog that includes the character's utterances as dialog history (as in Figure 1) is an effective method for reflecting the style of character. However, since only utterances of the characters are available in the proposed task, we match each utterance with an appropriate pseudo-context by using a retrieval model (Humeau et al., 2019) to select the relevant context from a predefined set of context candidates. Through human and automatic evaluation, we show that PDP generates responses that better reflect the style of fictional characters than existing baseline models.
We model a conversation agent that generates a response  $r$  corresponding to a given context  $x$  while mimicking an arbitrary character with  $k$  utterances  $\{u_1, u_2, \dots, u_k\}$  of the character. The simplest way to design the prompt with the character's utterances is to concatenate utterances as Madotto et al. (2021) does for PersonaChat (Zhang et al., 2018a). However, in our preliminary experiments, we observed that this method tends to generate dull responses that do not reflect the styles of the character (will be shown in Section 4). We hypothesize that the language model fails to utilize the utterances because such a format of the prompt is unlikely to have appeared naturally in the training set (Brown et al., 2020; Wei et al., 2021).
To address this issue, we propose PDP, which builds a dialog format prompt where character utterances are included in the dialog history, as depicted in Figure 1. Since a speaker tends to maintain a consistent style throughout the conversation, using such a prompt will induce the language model to generate responses that seamlessly reflect the style from the character's utterances. To build a dialog when only given the utterances of the character, we
require a pseudo-context  $c_i$  matching each utterance  $u_i$  to get a context-utterance pair  $(c_i, u_i)$ . We use a retriever  $R$  to acquire a pseudo-context  $c_i$ . Particularly, we employ Bi-encoder (Humeau et al., 2019) as our retriever  $R$ . We first define a fixed set of single-turn context candidates  $\mathcal{C}$  obtained from BST dataset (Smith et al., 2020b), which is the largest open-domain conversation dataset released to date. We then select a candidate as the pseudo-context  $c_i$  for the given utterance  $u_i$  using  $R$ . Bi-encoder maps the context  $c$  and the response  $r$  into the embedding space as  $e_{\mathrm{ctx}}(c)$  and  $e_{\mathrm{resp}}(r)$ , respectively. Bi-encoder is trained to represent the relevance score between a context  $c$  and response  $r$  with  $e_{\mathrm{ctx}}(c) \cdot e_{\mathrm{resp}}(r)$ . There are several variants to select the pseudo-context  $c_i$  as follows:
- Static Match selects a pseudo-context  $c_i$  that can coherently precede the given utterance  $u_i$  using the retrieval model  $R$ . Given  $u_i$ ,  $R$  calculates a score  $s_{\mathrm{stat}}$  for each  $c \in \mathcal{C}$  by  $s_{\mathrm{stat}}(c; u_i) = e_{\mathrm{ctx}}(c) \cdot e_{\mathrm{resp}}(u_i)$ . We set the pseudo-context  $c_i$  of  $u_i$  as  $c_i = \operatorname{argmax}_c s_{\mathrm{stat}}(c; u_i)$ . We name this variant static since the selected pseudo-context  $c_i$  depends only on the given utterance  $u_i$ .
- Dynamic Match selects a pseudo-context  $c_i$  relevant to the input context  $x$  in addition to  $u_i$ . Given  $x$  and  $u_i$ ,  $R$  calculates a score  $s_{\mathrm{dyn}}$  for each  $c \in \mathcal{C}$  by  $s_{\mathrm{dyn}}(c; x, u_i) = e_{\mathrm{ctx}}(c) \cdot e_{\mathrm{ctx}}(x) + s_{\mathrm{stat}}(c; u_i)$ . We set the pseudo-context  $c_i$  of  $u_i$  as  $c_i = \operatorname{argmax}_c s_{\mathrm{dyn}}(c; x, u_i)$ . Since language models quickly adapt to the context-response mapping of the given prompt via in-context learning, we believe providing pseudo-contexts that are semantically similar to the input context as in Dynamic Match facilitates the reflection of styles in corresponding utterances. We name this variant dynamic because the pseudo-context  $c_i$  depends on the varying input context  $x$ .
- Random Match selects a pseudo-context  $c_{i}$  randomly from the context candidates set  $\mathcal{C}$  without using  $R$ . This variant is used as a baseline to study the effect of the pseudo-context  $c_{i}$ .
Finally, all the  $k$  pairs  $(c_{i},u_{i})$  of the character are sorted by  $e_{\mathrm{ctx}}(x)\cdot e_{\mathrm{resp}}(u_i)$  in ascending order and are concatenated into a prompt in a dialog format.
3.1 Evaluation
We employ the HLA-Chat (Li et al., 2020) dataset to define the set of characters for evaluation. HLA-Chat consists of single-turn dialogs of characters
in various TV shows. We select ten characters among all the characters and manually curate eight utterances that best reveal each character's unique characteristics from their utterances in the dataset. Note that we consider that creating eight utterances is feasible even if new characters are given and we also empirically observed that language models adequately reflect each character's unique characteristics from the eight utterances.
In evaluating the performance of each method, we focus on two criteria: (1) Does the model's response reflect the style of a given character? (2) Does the model respond coherently to the given dialog context? To examine these two criteria, we run the model on fixed dialog contexts and calculate metrics that exhibit the style reflection and dialog coherency. We use the utterances of the test split of DailyDialog (Li et al., 2017) for dialog contexts.
Human Evaluation. We conduct a human evaluation to assess the quality of the generated responses. First, we select five characters which style can be distinguished apparently. We then randomly sample 50 contexts from the full fixed-context set of the characters. Using Amazon MTurk, we collect human annotations for the samples contexts. Human evaluators are asked to rate from 0 to 2 scale score how each model response (1) strongly reveals the style of a given character (Style Strength) and (2) whether a response is fluent and appropriate for a given dialog context (Appropriateness). To reduce annotator bias and inter-annotator variability, we apply Bayesian Calibration (Kulikov et al., 2019) to the human evaluation score.
Automatic Evaluation. Similar to the previous works on text style transfer (Li et al., 2018a; Riley et al., 2021; Smith et al., 2020a), we utilize a character classifier trained on the utterances in HLA-Chat to measure the style strength of the generated responses. We denote StyleProb as the classifier's average probability of predicting a target character. We use StyleProb instead of Style Accuracy since HLA-Chat has a class imbalance issue so that the performance on infrequent classes are hard to be measured by accuracy. For measuring coherency, we use MaUdE (Sinha et al., 2020), an automated dialog evaluation metric known to capture human judgment on the coherency of response.
3.2 Pre-trained Language Model
For all the methods, we use a decoder-only transformer of 3.8B parameters, denoted as Base-LM,
as a base language model. To make Base-LM acquire general language skills and better understand conversations, we train Base-LM on The Pile (Gao et al., 2020) and an additional corpus of public web documents.
Only Utterances. Instead of utilizing pseudo-context as suggested in our methods, we provide the set of character utterances as the "quotes of character during conversation" in the prompt. Comparing PDP with this method will verify the effect of pseudo-contexts.
Zero-shot Prompting. In this method, we only include the name of the character and the show in the prompt without using utterances of the character. The format of the prompt is similar to the prompt of Madotto et al. (2021) for controlled generation.
TextSETTR (Riley et al., 2021). We first construct a dialog prompt similar to Zero-shot Prompting (but without character information) and use it with Base-LM to generate plain responses. Then, we use TextSETTR, a few-shot text style transfer model that can transfer arbitrary styles without additional training, to transfer the style of plain responses to the target character's style.
GCC (Boyd et al., 2020). GCC is a method to control a user persona by utilizing the user's conversation history by concatenating users' previous utterances before input dialog context. Still, it has the drawback that it requires further training on a large-size character-conditioned dialog dataset.
3.4 Advantaged Methods
Unlike baseline methods that only have access to a few utterances of characters, advantaged methods also have access to additional data, which gives them an unfair advantage over other methods.
HLA-Chat Full-dataset Fine-tuning. We fine-tune Base-LM on the full HLA-chat dataset. In this method, character information is injected by concatenating the character's name and the show's name at the front of the dialog input.
Gold Match. Instead of using pseudo-context, this model uses the actual contexts corresponding to character example utterances annotated in the HLA-chat dataset.
Details for all methods and experiments are further described in Appendix.
Method Type	Methods	Human Evaluation (Raw)	Human Evaluation (Cali.)	Automatic Evaluation
Style.	Appr.	Sum	Style.	Appr.	Sum	StyleProb	MaUdE
Baselines	Only Utterances	1.200	1.263	2.463	1.147	1.124	2.271	0.2098	0.8887
Zero-shot Prompt	1.172	1.236	2.408	1.114	1.037	2.151	0.1432	0.8857
TextSETTR	1.155	1.317	2.472	1.060	1.059	2.119	0.1045	0.8435
GCC	1.193	1.256	2.449	1.136	1.069	2.205	0.2391	0.8835
Ours	Static Match	1.277	1.229	2.506	1.195	1.123	2.318	0.4192	0.8467
Dynamic Match	1.331	1.203	2.534	1.276	1.021	2.297	0.4789	0.8299
Random Match	1.297	1.195	2.492	1.199	1.035	2.234	0.4774	0.8358
Advantaged	Full-dataset Fine-tuning	1.260	1.329	2.589	1.164	1.130	2.295	0.2880	0.8793
Gold Match	1.287	1.245	2.532	1.157	1.097	2.254	0.4575	0.8350
Table 1: Evaluation results of style reflection metrics and response coherency metrics. Raw and Cali. indicate raw human evaluation results and Bayesian calibrated scores, respectively.
Input: We'll pick you up at your place at noon. Be there or be square!
Marge Simpson (from The Simpsons): Homer, you are my rock! I'm counting on you!
Bart Simpson (from The Simpsons): Lis, are you coming with us, too?
Michael (from The Office US): I will be there.
I will be there! I will be there, all right?
So, just... be ready. Don't be late. You know, I'm sorry.
Dwight (from The Office US): Micheal, are you listening to me? Are you even paying attention?!
Rachel (from Friends): Oh my god, Phoebe, I just
Spock (from Star Trek): Aye, Mister Scott. I'll be there.
Table 2: Responses (Other rows) generated from given input (Top row) by Dynamic Match for each character.
Table 1 shows the experimental results. Overall, our proposed PDP demonstrates far better style reflection scores on both human evaluation and automated metrics than all baseline methods – and even better than advantaged methods. In particular, PDP shows significantly higher style reflection scores compared to Only Utterances. Considering that the core difference between the prompt of PDP and that of Only Utterances is the presence of pseudo-contexts, this result demonstrates that providing a dialog-formatted prompt is highly effective at reflecting the styles of a character.
While PDP methods generally report better style reflection scores than baseline methods, we observe that the performance on style reflection and response coherency varies to some extent depending on how pseudo-context is selected. Static Match shows the highest response coherency scores among all variants of PDP while performing a little bit worse than Dynamic Match in terms of style reflection metrics. On the other hand, Dynamic
Match shows the best performance on style reflection metrics, where it losses some coherency. This observation confirms our hypothesis that using pseudo-context  $c_{i}$  that is semantically similar to the input context  $x$  is effective for utilizing styles from the character's utterances. Thus, the choice between Static Match and Dynamic Match depends on which of the two qualities – style and coherency – is more important. Lastly, Random Match, which is considered a simple ablation baseline, also shows reasonably high performance in terms of style reflection metrics. We plan to analyze the Random Match method in a follow-up study since it is unexpected that such a simple baseline shows high performance.
Discussion. Gold Match shows worse performance in style strength than PDP. We believe that the gold context-response pairs in the HLA-Chat are not always the most appropriate pairs for our experiments. Since the HLA-Chat originated from scripts of TV shows, there might be some additional contexts outside of a single-turn dialogue (e.g., the background of characters, events that happened before the dialogue, audio-visual information, etc.). Without understanding the context behind the scripts, even gold context-response pairs might seem irrelevant. Therefore, directly using the context-response pairs in HLA-Chat as in Gold Match could adversely affect the quality of subsequent responses in style strength and coherency.
PDP methods tend to have slightly lower response coherency scores compared to other baselines. Our speculations for this phenomenon are as follows. Pseudo-dialog pairs  $(c_i, u_i)$  created by PDP methods might have some degree of incoherency, and it might incur adverse effects in coherency via in-context learning in the language model. The fact that the response coherency score
Pre-trained LM	Method	StyleProb	MaUdE
GPT-J (6B)	Only Utterances	0.2200	0.8827
Static Match	0.3805	0.8638
Dynamic Match	0.4166	0.8535
Random Match	0.4045	0.8589
Gold Match	0.3860	0.8671
GPT-Neo (2.7B)	Only Utterances	0.1834	0.8901
Static Match	0.3561	0.8691
Dynamic Match	0.3940	0.8604
Random Match	0.3950	0.8683
Gold Match	0.3872	0.8732
GPT2-xl (1.5B)	Only Utterances	0.1831	0.8817
Static Match	0.3388	0.8736
Dynamic Match	0.3760	0.8728
Random Match	0.3515	0.8780
Gold Match	0.3579	0.8754
Table 3: Automatic evaluation results of style reflection metric and response coherency metric using different pre-trained language models.
of Static Match is higher compared to Dynamic Match, which finds a pseudo-context that is more similar to the input context, or Random Match, which finds a random pseudo context at all, supports this claim. Additionally, automated metrics like MaUdE are tuned to work with texts in standard dialog style. Since responses that strongly reflect character styles (e.g., "Yippie ki-yi-yay!" in Figure 1) are out-of-domain examples when put next to standard texts, there might be an unavoidable decrease in MaUdE scores. An interesting future work would be finding a method that does not reduce response coherency while also successfully reflecting the character styles.
Applicability of PDP to other language models. We further evaluate our method by leveraging different language models instead of Base-LM to verify that our method generally works well on any language model. We use three pre-trained language models, GPT-J 6B (Wang and Komatsuzaki, 2021), GPT-Neo 2.7B (Black et al., 2021), and GPT2-xl 1.5B (Radford et al., 2019), which are publicly available. Similar to our main experiments, we conduct the automatic evaluation with these language models.
The results are shown in Table 3. The overall trend of the results is similar to the results using Base-LM as a pre-trained language model (Table 1). This common trend shows that mimicking characters through the PDP method can be generally used not only with Base-LM but also with other pretrained language models.
In this paper, we introduce the task of mimicking a fictional character by using only a few utterances of the character. We propose a new method, Pseudo Dialog Prompting, which builds a prompt for a language model to solve this task by creating a pseudo dialog using the given utterance set with a retrieval model. Extensive experiments show that our method effectively generates responses that reflect the style of a given character better than baseline models and even advantaged models.
Like any conversation or generation model, we note that the quality of the models' responses depends on the quality of its training data. Our Base-LM model was trained on The Pile dataset (Gao et al., 2020) and Pushshift Reddit dataset (Baumgartner et al., 2020). Since the contents in these datasets were collected online, they may include underlying biases or potentially offensive words. These biases and toxicities can be projected into our models. Therefore, we highly recommend that additional steps are taken to filter out profanity and inappropriate responses when the model is deployed to the real world.
Furthermore, while we intend our method to be used to mimic fictional characters from movies, shows and stories to build more engaging conversation models, we also recognize that it is possible to use our method to mimic real-life individuals based on their utterances. Some potential risks include impersonating individuals, which can be harmful to the targeted individuals, and mimicking figures to generate content that can be harmful to groups of individuals. We hope that our method is deployed in a safe manner to avoid such malicious usage.
Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. arXiv preprint arXiv:2001.09977.
Reina Akama, Kazuaki Inada, Naoya Inoue, Sosuke Kobayashi, and Kentaro Inui. 2017. Generating stylistically consistent dialog responses with transfer learning. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 408-412.