Prompted LLMs as Chatbot Modules for Long Open-domain Conversation
Year: 2023

Abstract:
In this paper, we propose MPC (Modular Prompted Chatbot), a new approach for creating high-quality conversational agents without the need for fine-tuning. Our method utilizes pre-trained large language models (LLMs) as individual modules for long-term consistency and flexibility, by using techniques such as fewshot prompting, chain-of-thought (CoT), and external memory. Our human evaluation results show that MPC is on par with fine-tuned chatbot models in open-domain conversations, making it an effective solution for creating consistent and engaging chatbots. 1

heading: 1 Introduction
text: Language models with billions of parameters, such as GPT-3 (Brown et al., 2020) and PaLM (Chowdhery et al., 2022), have achieved state-of-the-art performance on many NLP tasks. To fine-tune these large language models (LLMs) for open-domain chatbot tasks, one could use a dataset of conversational data that is representative of the target domain. However, fine-tuning LLMs for opendomain chatbots can be challenging due to the computational burden of updating models with billions of parameters and the scarcity of data in the dialogue domain. Furthermore, fine-tuning can limit the model’s versatility by restricting it to a specific domain, and result in the loss of domainagnostic knowledge acquired during pre-training, as reported by Yang and Ma (2022). Multi-task training on different datasets, as proposed by Roller et al. (2021), can address the versatility issue but has limitations, such as the need for data to train each skill and the difficulty determining the necessary skills for an open-domain chatbot. In fact, the growing number of modules for chatbots, as in Blenderbot3 (BB3) (Shuster et al., 2022), points
*Equal contributions; randomized order 1Code is available in https://github.com/krafton-ai/MPC.
towards the increasing burden of data and computation when fine-tuning for each new chatbot model.
Interestingly, some LLMs have the ability to perform in-context learning (ICL) (Nye et al., 2022; Wei et al., 2022b; Lewkowycz et al., 2022; Wei et al., 2022a; Zhou et al., 2022; Dasgupta et al., 2022; Chung et al., 2022). This capability enables the model to rapidly adapt to and execute a specific task based on a brief instruction and a few examples, without requiring additional fine-tuning. This can be utilized to create an open-domain chatbot, where a prompt describing a task required for opendomain dialogue and a few examples of solving such task can be provided to the LLM, allowing it to generate information that is pertinent to the current conversation.
Our Contributions We present a novel approach for creating high-quality conversational agents without the need for fine-tuning. Our proposed chatbot, MPC (Modular Prompted Chatbot), utilizes open-sourced pre-trained language models to increase the flexibility of designing the modules of an open-domain chatbot. Our approach enhances multiple conversational capabilities by utilizing a modularized agent that incorporates LLMs with prompt techniques such as few-shot ICL and Chain-of-Thought (CoT). In the paper, we design MPC to achieve long-term consistency, a domain in which previous chatbots have struggled. Our human evaluation results show that MPC is on par with or even preferred over fine-tuned LLMs, such as Blenderbot, in an open-domain conversational setting. This approach highlights the potential of pre-trained LLMs to adapt to new tasks without fine-tuning, providing an efficient solution for creating open-domain conversational agents.

heading: 2 Related Work
text: Modular Prompting Well-crafted elicitive prompts can enhance reasoning abilities, resulting
4536
in improved performance across various benchmarks (Kojima et al., 2022; Wei et al., 2022b; Suzgun et al., 2022). For complex problems, Press et al. (2022) identified the compositionality gap which arises when an LM can solve sub-problems but not the overall solution and further showed that CoT narrows this gap. Since then, there has been a flurry of work that solves tasks by decomposing them into smaller tasks solved by different “prompt modules” (Zhou et al., 2022; Wang et al., 2022; Khot et al., 2022; Khattab et al., 2022).
Modular prompting has found use beyond benchmarks and in conversation generation. Kim et al. (2022) used an LLM to generate a socially diverse dialogue dataset that is more natural and detailed than existing crowdsourced datasets. Moreover, hierarchical prompt modules prove to help long-range coherence for generating narratives and plays (Yang et al., 2022; Mirowski et al., 2022). We refer to Mialon et al. (2023) for a detailed overview on such augmented uses of LLMs.
Open-domain Chatbots Many recent dialogue agents rely on dialogue-finetuned LLMs. In Thoppilan et al. (2022), LaMDa has been trained on large amounts of crawled conversational data and has used a fine-tuned classifier for model safety. More recently, similar to our modularization approach, BB3 fine-tunes Open Pre-trained Transformers (OPT) (Zhang et al., 2022; Shuster et al., 2022) on QA and dialogue datasets and uses one shared model weight as multiple modules.
On the other hand, Madotto et al. (2021) eliminate the need for fine-tuning on dialogue data by feeding retrieved dialogue samples as few-shot for GPT-J (Wang and Komatsuzaki, 2021). We find this work to be complementary to our work, as the few-shot dialogue can be seen as an approach to enhance the utterance generator module.
Long-term Memory The Multi-Session Chat dataset (Xu et al., 2022) allows for measuring how well conversational agents maintain a long-term memory of facts about the user and bot. Information is retrieved using Dense Passage Retriever (DPR) (Karpukhin et al., 2020), while BART compresses memories before storing them. In Shuster et al. (2022), a modular approach is used to incorporate long-term memory and factual grounding through internet search with an LLM. This work is closest to our work since it includes an ablation study in which prompt-based modules are com-
pared with fine-tuned modules. However, in our work, we argue that more reasoning-based prompting, as demonstrated in Wei et al. (2022b), is beneficial for better contextual understanding.

heading: 3 Modular Prompted Chatbot
text: We present a modular chatbot system (Fig. 1) that uses prompt-based LLMs to maintain persona and engagement throughout long-term conversations.
At the start of a conversation, a pre-defined persona is stored in the memory pool. When a user sends a message, the clarifier rephrases it to resolve any ambiguities and passes it to the DPR model which retrieves relevant memories from the memory pool. The retrieved memories and clarifier output are fed into the memory processor to get a single context-relevant memory, which is then passed to an utterance generator for producing a response from the chatbot. Every few turns, we call upon a summarizer module to extract important information from dialogue and store it in the memory pool for future use (see Appendices D and E).
Utterance Clarifier As conversations are often muddled with vague coreferences and contextual
cues, our clarifier module is an LM prompted with the recent dialogue to resolve any ambiguities. For instance, depending on prior context, the user input “Do you like working there?” would output “Does Sarah like working at ZYX company?”. By resolving contextual ambiguity, the clarifier assists the DPR model and memory processor module by providing an information-dense query to fetch and process relevant memories.
Memory Processor As demonstrated in Fig. 1, we formulate memory processing as an LLM reasoning task of finding the most relevant information given the dialogue. Following the footsteps in solving hard reasoning tasks (Suzgun et al., 2022), we provide CoT examples to show reasons for ignoring certain memories and synthesizing others. For models incapable of CoT, we simply provide the few-shot examples without the reasoning portion.
Since the memory pool accumulates as the conversation progresses, we use a pre-trained DPR with the output of the clarifier as the query to retrieve the top-k most relevant memories from the memory pool. The memory processor then condenses the top memories into one refined memory.
Utterance Generator The utterance generator module generates the final response of the chatbot given the recent dialogue history and memory provided by the memory processor. The prompt consists of the dialogue history, condensed memory, and the generation instruction (e.g., “Give a friendly response to the user.”). For some models, we find that inserting the generation instruction at the end was helpful as placing it before the dialogue minimizes the effect of the instruction.
Dialogue Summarizer We provide a few-shot prompt to ensure we record specific details of the conversation and the user.

heading: 4 Experimental Setup
text: We evaluate our chatbot’s performance by assessing core skills necessary for long-term conversations. We assess consistency by assigning one of five personas, each with 12 facts from PersonaChat (Zhang et al., 2018), and presenting these facts to evaluators. For each experiment, we collect 20 turns from each evaluator and at least 500 turns in total from two subgroups: Amazon Mechanical Turk and university students. See Appendix C for a detailed explanation of our data collection.
In our setup, there are four groups of models.
1. Fine-tuned chatbot models such as BB3.
2. Vanilla is an utterance generator that either prepends full persona or no persona to the dialogue history in the prompt. This represents the naive approach of using an LM as a chatbot.
3. MPC is as described in Section 3. Specifically, we only form one memory from the memory processor. Full persona is not explicitly prepended.
4. MPC+full persona is MPC that prepends the full persona. See examples in Appendix E.

heading: 4.1 Single Model Evaluation
text: We evaluate each model separately using Sensibleness, Consistency, and Engagingness metrics and collect a final rating (out of 5.0). The exact questions and evaluation forms are in Appendix A. We also report two types of combined score SCE (Sensible Consistent and Engaging): a "perfect" score SCE-p, where all metrics must be positive for a positive response, and the weighted score SCE-w, which is similar to SSI (Thoppilan et al., 2022) and reported in Appendix B.
We use OpenAI GPT-3 text-davinci-002 (td2), davinci, OPT 30B, 66B, GPT-JT-6B (Together, 2022), and BLOOM-176B (Scao et al., 2022) as base LMs for MPC. For fine-tuned group, we use BB3-30B with the same persona settings. For BB3-175B, we request crowdworkers to evaluate the online demo for 20 turns. We also report the additional results of the recent models such as GPT 3.5 (gpt-3.5-turbo-0301) (OpenAI, 2022), GPT-4 (OpenAI, 2023), and Alpaca (Taori et al., 2023) in Appendix B.

heading: 4.2 Pairwise Models Evaluation
text: We A/B test two chatbot models by providing the user with two randomized responses A and B. The user then evaluates them based on Sensibleness, Consistency, Interestingness, and Preference. The conversation then continues with the response chosen for Preference. This lets us to control for dialogue history when comparing two models.
Specifically, we conduct two main experiments: (1) MPCOPT-30B vs. BB3-30B, where internet search for BB3 is disabled as we focus on consistency. Our evaluation enables a direct comparison, as BB3-30B is a fine-tuned version of OPT-30B. (2) MPCtd2 vs. Vanilla td2 (full persona).
Implicit Persona In reality, we implicitly learn about someone through dialogue. In contrast, our
previous experiments show explicit persona to both evaluators and models. As such, we devise an experiment by providing a 10-turn pre-defined dialogue to the crowdworker and pairwise models, MPCtd2 and Vanilla td2 (no persona). We then ask workers to ask about the previous dialogue for 6 new turns. Here, we set a shorter maximum context length than the 10-turn dialogue, so that the setup represents long conversations where necessary information is beyond the LM context length.

heading: 4.3 Metrics
text: In our work, we present two modes of experiments: single and pairwise model evaluation. Our single model evaluation is similar to a hybrid of SM-Turn and SM-Dialogue evaluations and a pairwise model to PW-Turn from Smith et al. (2022). For each turn, we ask crowdworkers to evaluate the quality of the chatbot response based on the following metrics.

heading: Single Model Evaluation
text: • Sensibleness Whether the response makes sense.
“Does the response make sense?”
• Consistency Whether the response does not contradict the contextual information or the persona.
“Is the response consistent with the information based on the persona list and context of the conversation?”
• Engagingness Whether the user is engaged and would want to continue the conversation.
“Are you engaged by the response? Do you want to continue the conversation?”
• Final Rating
“How was your chat? From a scale of 1 (very bad) to 5 (very good), rate the quality of the overall conversation.”

heading: Pairwise Model Evaluation
text: • Sensibleness Which response makes more sense.
“Which response makes more sense?”
• Consistency Which response is more true to and consistent with the persona.
“If you had to say one of these speakers is more true to and consistent with the listed persona and one is not, who would you say is more consistent?”
• Interestingness Which is more interesting.
“If you had to say one of these responses is interesting and one is boring, which would you say is more interesting?”
• Preference Which is preferred for a long conversation.

heading: 5 Results
text: Pre-trained vs. Fine-tuned Our human evaluations show that MPC, which uses a pre-trained LLM, is better than the fine-tuned BB3-30B.
Most notably, with a 9% SCE-p gap, MPCOPT-30B scores higher on all metrics than BB3-30B. In fact, the majority of our MPC models in Table. 2 demonstrates superior performance to BB3-30B.
For BB3-30B, we have observed issues of consecutive utterance repetition. We report the evaluation results of only including dialogues without repetition in Table 5. Even without repetition, MPCOPT-30B is still on par with BB3-30B. Moreoever, MPCOPT-30B in Table 1 shows higher sensibleness and preference than BB3-30B, while scoring similarly in consistency and interestingness.
Modular vs. Non-modular MPC excels in consistent dialogue in comparison to the vanilla approach. The implicit persona experiment in Table 3 demonstrates that MPCtd2 scores significantly higher than a vanilla application of td2 in all metrics. In Table 2, for MPCtd2+Full persona, consistency is on par with that of td2 (full persona), while engagingness, SCE-p, and rating are the best overall. Nevertheless, when we do not include full persona in the prompt for MPCtd2, it shows lower consistency than td2 (full persona), albeit the high
Model Sens. Cons. Eng. SCE-p Rating
Fine-tuned
BB3-30B 71.3 77.8 73.7 54.3 2.9 BB3-175B 85.9 (88.7) 84.8 73.1 3.8
rates of ties in sensibleness and consistency (Table 4). In general, we find that users would ask primarily about the bot’s persona rather than having a two-sided conversation, leading to td2 (full persona) performing better in consistency.
Effect of Size When other variables are held the same, we observe model size is positively correlated with positive evaluations. The most compelling evidence can be seen in the superiority of MPCOPT-66B across all metrics when compared to MPCOPT-30B since the two base LMs are trained nearly identically. Needless to say, model size is not the only factor. MPCBLOOM, one of the largest models, scores the lowest in our experiments.
Effect of Instruction-tuning Instruction-tuning helps the creation of a modular dialogue system by enabling adaptation to various tasks. Not only does MPCtd2 perform better than MPCdavinci, but
also MPCGPT-JT shows high sensibleness and consistency, despite its smaller size. Though not included in the main body due to limited sample size, we additionally see that MPCAlpaca-7B achieves higher SCE and rating than its full-persona counterpart in Table 5. In general, we posit that finding good prompts for each module for instructiontuned LMs is simpler and more robust to variations. MPCdavinci is worse than davinci (full persona), presumably due to error propagation in the modular system, though we do not rule out that there are better prompts for MPCdavinci.

heading: 6 Conclusion
text: We demonstrated that a modular approach using LLMs, namely MPC, can be an effective solution for long-term open-domain chatbots without further finetuning. We compared MPC to fine-tuned and vanilla LM baselines and found that our approach achieved superior performance by human evaluation. Additionally, our modular system incorporated persona and information from dialogue history more effectively than the non-modular ones according to our consistency evaluation.

heading: Limitations
text: In this work, we investigate the use of pre-trained language models for long-term English conversations. While we expect a modular approach may be effective for other languages when given a capable language model, it should also be noted that further research is needed to confirm the applicability of our findings to other languages. For instance, though BLOOM is trained as a multilingual language model, we only implement MPCBLOOM in English and evaluate its English capability as a open-domain dialogue agent.
Meanwhile, a modular system can create additional inference overhead or error accumulation. The system performance would become much bet-
ter if we optimally choose the LM for each module. For example, we could use GPT-3 td2 for the memory processor, while we employ OPT-175B for the utterance generator. We would need to evaluate every module to find the best model for each, which we leave to future work.
In terms of evaluation methodology, our human evaluations of MPC and its analysis face the same challenges as previous studies on evaluating interactive conversational tasks. As demonstrated by Smith et al. (2022), there is currently no definitive evaluation method for determining the best chatbot model. Additionally, there are several factors that must be taken into account during data collection and interpretation, such as annotator subjectivity, instruction bias, and crowdworker working conditions. For a more in-depth discussion of human-LM interaction, we refer the reader to Lee et al. (2022).
As described in Appendix C, to gather a diverse range of evaluations, we have collected qualitative data from two groups: English-speaking annotators on Amazon Mechanical Turk (MTurk), and qualified university students who were capable of speaking English. To some extent, this evaluation setup reduces cultural bias and platform homogeneity compared to using MTurk workers alone. However, the limitations of this approach should be acknowledged and this may further complicate the analysis when controlling for MPC’s performance on different subgroups.
Lastly, we note that running MPC requires at least as much memory as its underlying language model, making MPC infeasible to even load on a single node for heavy models such as BLOOM176B.

heading: Ethics Statement
text: MPC utilizes publicly available pre-trained LMs for chatbot utterance generation. Language generation from these LMs is known to have concerns about toxicity and bias (Xu et al., 2020). Thus, ensuring safe deployment and interaction is a necessity.
Accordingly, we outline our data collection procedure in Appendix C. We allow crowdworkers to directly provide us with feedback and also manually check for any offensive or controversial outputs. To ensure the protection of personal information, all crowdworkers were instructed not to share any personally identifiable or private information. Additionally, they were asked to give their consent
for the collection of anonymous information for research purposes. Prior to participating, all workers were informed of the purpose of data collection and, after evaluation, were compensated with a competitive hourly rate, approximately $12-16 per hour.

heading: A Evaluation Details

heading: A.1 Models
text: For single model evaluation of MPC, We use OPT (30B, 66B), OpenAI GPT-3 (davinci, text-davinci-002) GPT-JT, and BLOOM176B as the base LM for the open-sourced pretrained LLM group. On the other hand, we evaluate Blenderbot3 30B (BB3-30B), the best publicly available open-domain chatbot model, for the fine-tuned model group. We also evaluate BB3175B which is only available through the web interface https://blenderbot.ai/. We evaluate BB3-175B for comparison purposes, though the comparison is not fair as we cannot instill a persona into BB3 and cannot control for further differences, such as internet search and user interface.
For pairwise model evaluation of MPC, we pairwise test MPCOPT-30B and BB3-30B to compare human evaluations of a pre-trained model and a fine-tuned model. This experiment controls for many variables as BB3-30B was initialized with OPT-30B before fine-tuning. For the module experiment, we tested with OPT-30B, and we compared the one with the whole pipeline, and the other only with an utterance generator with a fixed persona.

heading: A.2 Dense Passage Retriever
text: For the DPR model of the memory module, we use the model weights from the custom DPR model finetuned for MultiDoc2Dial (Feng et al., 2021), as we have observed that this model performs slightly better than the original DPR model from Karpukhin et al. (2020).

heading: B Experimental Results
text: In this section, we report all experimental results and miscellaneous analysis.

heading: B.1 Full Single Model Evaluation
text: In this subsection, we report all our model evaluations with additional details, such as average latency (the amount of time it took to compute the next utterance and show the user), and average utterance length, which is the number of tokens measured by the OPT tokenizer. We also additionally report SCE-w, which was not reported in the main body due to space restrictions in Table 5.
We compute a weighted score SCE-w similar to SSI (Thoppilan et al., 2022), in which a response can only be consistent if it is sensible and only
engaging if it is sensible and consistent. The final score is the average over all SCE-w-adjusted metrics.
We also note that OpenAI API calls have highly variable latency since December 2022 due to ChatGPT, so a measure of latency due to modularization cannot properly be quantified.
BB3-175B Evaluation Our results for BB3175B are not directly comparable to other models due to a different evaluation procedure. Since the model weights for BB3-175B are not publicly available, we referred evaluators to interact with the Blenderbot version available at https://blenderbot.ai/chat. Evaluators were then asked to copy and paste all user inputs and chatbot responses while annotating the chatbot responses similarly to other models. We also note that consistency can only be evaluated for persona consistency within the dialog context as we are unable to provide a specific persona for the online version of Blenderbot.

heading: B.2 Model Configurations
text: We include our code for running each model we evaluate in our repository at https://github.com/krafton-ai/MPC. Each model configuration can be found in the repository. The configuration includes decoding parameters (e.g., sampling method, temperature for each module) and slight variations of the prompts for the utterance generator module.

heading: B.3 Additional Analysis
text: Alpaca-7B Alpaca-7B (Taori et al., 2023) is an instruction-tuned language model, fine-tuned from the LLaMA 7B model (Touvron et al., 2023) on 52K instruction and demonstration pairs. We run small-scale dialogue evaluations for Alpaca-7B with full persona and MPCAlpaca-7B.
We find that, although we have only collected limited samples due to resource constraints, we see that MPCAlpaca-7B achieves slightly higher SCE and rating than its full-persona counterpart in Table 5. Moreover, MPCAlpaca-7B shows higher scores than MPCOPT-30B. We attribute this high achievement to a better base LM, as demonstrated in Touvron et al. (2023), and to instruction-tuning, as we have explained in the main body.
ChatGPT and GPT-4 Although MPC results for GPT-3.5-turbo (ChatGPT) are overwhelmingly positive, we highlight that a direct comparison to
other models is difficult since GPT-3.5-turbo is trained specifically in dialogue-style data, and its model architecture and training process are opaque. One observation is that MPCGPT-3.5 responses are significantly longer compared to all other models.
While it is tempting to attribute the positive results of MPCGPT-3.5 and MPCGPT-4 solely to dialogue-tuning, such a conclusion may not be entirely accurate. Based on our experimental results and comprehensive analysis, we cautiously hypothesize that the success of MPCGPT-3.5 and MPCGPT-4 primarily stems from the utilization of a more robust language model. Strong base LLMs play a crucial role in ensuring sensibleness and consistency in generated responses. Moreover, as we have demonstrated, instruction-tuning is effective in facilitating the development of modular components for open-domain chatbot design. Thus, without more information about these LLMs, we carefully posit that it is mainly the combination of a stronger LM and effective instruction-tuning that contribute to the positive results in our evaluation of MPCGPT-3.5 and MPCGPT-4.

heading: B.4 Subgroup Analysis
text: We measure the difference in ratings, for each metric and our combined metrics, between MTurk and student evaluators and find no significant difference for most experiments and metrics (Fig.2. In general, we observe the greatest rating difference for sensibleness.
BB3-30B is an outlier with a significant difference across all metrics, especially for sensibleness with a difference of about 20%. BB3-30B dialogs contain an increased ratio of repetition issues for students but even when excluding those dialogs, we still find a significant rating difference. Looking at the dialog histories, we observe that BB3-30B student evaluators were less patient than MTurk evaluators when BB3-30B would repeat the same questions or suddenly change the subject. It should be noted that this is a qualitative observation and should be taken with a grain of salt.

heading: C Data Collection Details
text: We attach screenshots of the evaluation forms, used for data collection, for single (Fig. 3) and pairwise
MTurk - Student
models (Fig. 4) respectively. We also attach the screenshot of the instruction, alert, and consent format that we gave to participants (Fig. 5 and 6).

heading: C.1 MTurk Setting
text: We gathered at least 15 MTurk workers for each task. We require them to have an MTurk Master’s qualification, a HIT approval rate of more than 99%, and a number of HITs approved more than 100 times, and to live in the US, UK, Canada, Australia, and New Zealand.
We found that data quality improves significantly when using MTurk Master workers. These workers tend to produce more fluent messages than nonMaster workers and are less likely to provide random or generic responses. The difference was more pronounced as non-Master crowdworkers would often provide a score of 99% on average despite the clear problems of repetition and contradiction from the dialogue agent.

heading: C.2 Student Participants
text: To cross-validate the result of MTurk, we gathered 49 English-proficient students from a several universities. They are either undergraduate or graduate students. The minimum requirement they should meet is English proficiency: IELTS >= 7.0; TOEFL IBT >= 95; and TOEIC >= 900. This criterion is similar or equivalent to the minimum required level of English proficiency for admission to the graduate schools of MIT. The average English scores of students we gathered are the following: IBT 108.4
out of 120; TOEIC 960.9 out of 990; and IELTS 7.83 out of 9.0.

heading: C.3 Crowdworker Instruction
text: The crowd workers are asked to continue the chat for 20 turns and evaluate each response by the metrics described in the section below. We instruct them to type more than 3 words on average, and not to repeat meaningless or generic messages. Also, we request them not to randomly choose between yes or no and to provide evaluations honestly. Lastly, we strongly emphasize the importance of maintaining confidentiality and request the crowd workers refrain from disclosing any private information about themselves or others during the evaluation process.

heading: D Few-shot Prompts
text: In this section, we show our full few-shot prompts, including the CoT prompt, for each module. The utterance generator prompts vary slightly between model experiments. These prompts can be seen in the model configurations in our repository. The second example for dialogue summary is modified from a sample from the dataset DialogSum (Chen et al., 2021).

heading: E Utterance Generator Prompt Examples
text: In this section, we provide example prompts for the utterance generator of MPC and Vanilla models. The persona given is one of the five personas we use and the dialogue context is provided as an example.

heading: Utterance Generator
text: Rephrase User’s question in third-person.
Sarah: I’ve been working at the coffee shop for about six months. User: I see. what did you do before that? # Specifically, What did Sarah do before working at a coffee shop for six months?#
Ashley: Do you know who Ronaldo is? User: I don’t know, who is he? # Specifically, Who is Ronaldo?#
Robert: Hey, how are you doing? User: Good. What’s your name? Do you know my name? # Specifically, What is Robert’s name? Does Robert know User’s name?#
Jennifer: I like reading about history and science. User: What kind of history? I like history too! # Specifically, What kind of history does Jennifer read about?#
John: Hi! How are you doing today? User: good. how about you # Specifically, How is John doing today?#

heading: Memory Processor w/o CoT
text: # This is the list of John’s knowledge. John’s full name is John Parker. User is a teacher at a local middle school. User teaches biology. John likes to go for a run. User enjoys watching movies, but User doesn’t like superhero movies. Q: What does User do for a living? A: John thinks User is a biology teacher at a local middle school.
# This is the list of Ashley’s knowledge. Ashley likes history documentaries. Ashley does not like Korean food. Ashley is a teacher at a local middle school. User likes biology and especially anatomy. Ashley likes French cuisine. Q: What is Ashley’s favorite dish? A: Ashley thinks Ashley likes French cuisine but does not like Korean food.

heading: Memory Processor w/ CoT
text: # This is the list of Ashley’s knowledge. (1) Ashley likes history documentaries. (2) Ashley does not like Korean food. (3) Ashley is a teacher at a local middle school. (4) User likes biology and especially anatomy. (5) Ashley likes French cuisine. Q: What is Ashley’s favorite dish? A: Let’s think step by step. (1) History documentaries are not related to Ashley’s favorite dish. (2) Ashley’s favorite dish would not be Korean because she does not like Korean food. (3) Ashley being a teacher does not tell us anything about her favorite dish. (4) This fact is about User, not Ashley. (5) Ashley’s favorite dish may be French since she likes French cuisine. Therefore, (2) and (5) can help answer the question. Answer: Ashley thinks Ashley likes French cuisine but does not like Korean food.
Dialogue Summarizer
#Dialogue User: Tell me about yourself Sally: I’m 26 years old and graduated from a college in Wisconsin. User: Were you a leader when you were in college? Sally: Yes. I was the head TA for a computer science course at our university. User: Were you involved in any club activities at your university? Sally: Yes. I was a member of the basketball Society. I like playing basketball. #Summary - Sally is 26 years old and graduated college in Wisconsin. - Sally was the head TA for a computer science course. - Sally played basketball in college.#
#Dialogue User: Out of every movie that you’ve seen, which one is your favorite? John: I’m going to have to say that Superbad is the best movie ever. User: You think so, how come? John: Well, Superbad is super funny. User: You’re not lying, I found that movie absolutely hilarious. John: I didn’t know that you saw Superbad before. User: I made sure to be in line to see it the first day it came out. #Summary - John’s favorite movie is Superbad. - John and User think Superbad is funny. - User watched Superbad the first day it came out.#

heading: Utterance Generator of MPC
text: Sarah is polite, fun, and engaging. The following is a conversation between Sarah and Person1. Person1: My Favorite part of the concert is making sure I get great seats and then I can catch guitar picks Sarah: That sounds like a lot of fun! Person1: It is a lot of fun, so what do you like to do for fun? Sarah: I love going to concerts, car shows, music festivals, and houses with fireplaces. Person1: I love going to car shows and looking at older cars, what is your favorite car? Sarah: I think my favorite car is a vintage mustang. Person1: OH man vintage Mustangs are great, I had a friend who owned a 69 mach 1, that was a dream car Sarah: Well, I hope you get to experience your dream car one day! Person1: Maybe someday I will, do you own a Mustang?
The following statements are true about Sarah. Sarah thinks Sarah owns a Mustang.
Give a friendly sensible response that is interesting and polite to Person1. End with a question. Sarah: Well , I hope you get to experience your dream car one day! Person1: Maybe someday I will, do you own a Mustang? Sarah:
Response from previous prompt
"Yes, I do own a Mustang. It’s a great car to drive. Do you like driving?"
Utterance Generator of MPC with full persona
The following are persona facts about Sarah. Sarah is 40 years old. Sarah currently lives in small town in Georgia. Sarah reads twenty books a year. Sarah is a stunt double as Sarah’s second job. Sarah only eats kosher. Sarah was poor growing up. Sarah was raised in a single parent household. Sarah has two dogs. Sarah likes to work on vintage cars. Sarah owns two vintage mustangs. Sarah’s favorite music is country. Sarah volunteers at a soup kitchen. Sarah thinks Sarah grew up in a small town and didn’t have a car.
Sarah is polite, fun, and engaging. The following is a conversation between Sarah and Person1. Person1: We at least had a playground in tow and my grandparents lived in right in front of it, so we went to both places a good bit. Sarah: Maybe it was difficult to find things to do in your town, but it sounds like you had a lot of family support, which is really important. It’s good that you had a place to go where you felt safe and loved. Person1: You ended up growing up poor too. Sarah: Yes, I did. I think it’s made me a lot more sympathetic to other people’s experiences. Person1: We were poor too and my family didn’t have a car. Sarah: I can imagine that it would be tough to not have a car. My family had a car, but we were poor too. Person1: It was. Almost every month we had barely any food and my parents had to deal with what we had. Give a friendly sensible response that is interesting and polite to Person1. End with a question. Sarah:
Response from previous prompt
"It sounds like it was really tough for you and your family. I’m sorry that you had to go through that. Do you think that growing up in a small town made it harder for you?"

heading: Utterance Generator of Vanilla (full persona)
text: The following statements are true about Sarah. Sarah is 24 years old. Sarah currently lives in Canada. Sarah is a swim coach at Sarah’s local pool. Sarah is studying to be a computer programmer. Sarah is also a graduate student. Sarah is now looking for a new job. Sarah’s mother is very traditional while Sarah prefers to be more free spirited. Sarah’s family and Sarah are from India. Sarah’s favorite music genre is death metal. Sarah is a famous twitch streamer. Sarah likes watching war documentaries. Sarah’s favorite food is mexican food.
Sarah is polite, fun, and engaging. The following is a conversation between Sarah and Person1. Person1: Have you thought about finding a job that allows you to travel? Sarah: I have, but I’m not sure what type of job would allow me to do that. Person1: What types of jobs have you looked into already? Sarah: I’ve looked into jobs in the travel industry, but I’m not sure if that’s what I really want to do. Person1: Since your a twitch streamer, could you maybe make that your career? Sarah:
Response from previous prompt
"I don’t know, I’ve never thought about that before. I’ll have to look into it."
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: For Grammar fixes and for Making our text more concise to fit into the page limit. Mostly we used GPT-3 text davinci 3 to generate them, but the results are mostly similar to what we originally wrote.
B 3 Did you use or create scientific artifacts? 1,4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 1,4

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: code will be publicly available under MIT license after review period

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Limitations

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Ethic Statements

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Limitations
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix B
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 4, Appendix B
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4, Appendix A,B,E

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5, appendix B

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 1,2,4, git repository
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? section 4; Appendix E

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Appendix E

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Ethics Statement, Appendix E

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Appendix E
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix E. But partly revealed because of anonymity, we hide the detailed nationality.

heading: Future Work
text: We would need to evaluate every module to find the best model for each, which we leave to future work. In terms of evaluation methodology, our human evaluations of MPC and its analysis face the same challenges as previous studies on evaluating interactive conversational tasks. As demonstrated by Smith et al. (2022), there is currently no definitive evaluation method for determining the best chatbot model. Additionally, there are several factors that must be taken into account during data collection and interpretation, such as annotator subjectivity, instruction bias, and crowdworker working conditions