CausalDialogue: Modeling Utterance-level Causality in Conversations
Year: 2023

Abstract:
Despite their widespread adoption, neural conversation models have yet to exhibit natural chat capabilities with humans. In this research, we examine user utterances as causes and generated responses as effects, recognizing that changes in a cause should produce a different effect. To further explore this concept, we have compiled and expanded upon a new dataset called CausalDialogue through crowdsourcing. This dataset includes multiple causeeffect pairs within a directed acyclic graph (DAG) structure. Our analysis reveals that traditional loss functions struggle to effectively incorporate the DAG structure, leading us to propose a causality-enhanced method called Exponential Maximum Average Treatment Effect (ExMATE) to enhance the impact of causality at the utterance level in training neural conversation models. To evaluate the needs of considering causality in dialogue generation, we built a comprehensive benchmark on CausalDialogue dataset using different models, inference, and training methods. Through experiments, we find that a causality-inspired loss like ExMATE can improve the diversity and agility of conventional loss function and there is still room for improvement to reach human-level quality on this new dataset. 1

heading: 1 Introduction
text: Over time, broadly-defined dialogue models have become increasingly prevalent in society and been integrated in a range of domains from speech assistants and customer service systems to entertainment products, such as video games, where the non-playable characters (NPCs) engage in conversation with players. A core goal of training chatbots is enabling them to interact with humans naturally (Vinyals and Le, 2015; Sordoni et al., 2015). This includes, but is not limited to: considering
1Our code and dataset are available at https://github. com/Pascalson/CausalDialogue
both the machine and addressee’s personalities (Li et al., 2016b), diversifying responses to be less generic (e.g., the same response “I don’t know.” is often produced in a traditional setting for different dialogues) (Li et al., 2016a), grounding on external knowledge to be informative (Ghazvininejad et al., 2018), and tailoring responses specific to nuanced differences in conversation.
To the best of our knowledge, no recent studies have prioritized the ability to tailor responses for minor differences in conversations. This problem is currently implicitly approached by training models with larger scale or cleaner conversation data (Zhang et al., 2020; Roller et al., 2021; Thoppilan et al., 2022) or involving human-in-the-loop (Li et al., 2016c; Jaques et al., 2020). However, the effectiveness of these methods is unclear, the online rewarding scheme can be expensive, and a suitable testbed for evaluating the solution to this problem has not yet been identified.
To this end, we propose a benchmark to foster research in tailoring responses for nuanced differences in conversations by answering the question “if all prior turns are the same, but the last turns in two conversations are semantically different, how should future turns differ?” We call this concept Agility and model it as the utterance-level causes and effects in dialogue response generation, where the causes are the slightly different prior turns and the effects are the resulting future turns.
We introduce CausalDialogue, a dataset seeded by expert-written dialogues containing branching dialogue paths, which we further expand in terms of scale and linguistic abundance with crowdsourcing. Each conversation is represented as a directed acyclic graph (DAG) for ease of storage and causal analysis (Pearl, 2009) as shown in Figure 1. As conversations progress, each utterance can elicit multiple responses, resulting in a split of the conversation (branch-splitting). Alternatively, multiple conversations that share a common starting point
12506
may sometimes lead to the same response, even if the middle exchanges differ (branch-colliding). Due to the DAG structure of CausalDialogue, it is ideal for aiding research on response generation that requires abundant IF-bases, for instance, causal inference and offline reinforcement learning, which may improve the response generation quality for nuanced differences in conversation.
To provide a benchmark for future work on the CausalDialogue dataset, we conduct experiments with various setups. We include both decoderonly and encoder-decoder transformer models pretrained on either common or dialogue-specific corpora, various inference methods, conventional training losses, and a newly proposed loss, Exponential Maximum Average Treatment Effect (ExMATE), inspired by Average Treatment Effect (Holland, 1986; Imai et al., 2008), which is a method commonly used to approximate the causal effect of a treatment and its outcome. In this benchmark, we show that existing methods are not sufficient in tackling the agility issue, and a simple causalityinspired loss demonstrates improvement.
Our key contributions are: • A novel dataset, CausalDialogue, including both
expert-written scripts and crowd-sourced utterances with a DAG structure. • A new training loss, ExMATE, for considering the utterances as causes and effects in a dialogue, inspired by the average treatment effect in research on causal inference. • A benchmark with experiments showing that existing methods need improvement on the agility problem, and a causality-inspired method can be a promising direction to improve it.

heading: 2 Related Work
text: Chit-Chat Dialogue Datasets. To boost the research of dialogue models, the community has collected dialogues based on scripts written by experts from movies (Danescu-Niculescu-Mizil and Lee, 2011; Banchs, 2012; Lison and Tiedemann, 2016), TV shows (Poria et al., 2019; Tuan et al., 2019; Yu et al., 2020; Rameshkumar and Bailey, 2020), and education purposes (Li et al., 2017b; Cui et al., 2020). For abundant diversity and real-life scenarios, Ritter et al. (2011); Wang et al. (2013); Lowe et al. (2015); Pasunuru and Bansal (2018) collected datasets based on the publicly available data from social media and forums. Additionally, previous work has explored the idea of collecting data through crowd-sourcing with added constraints to improve its quality or expand label types. For example, Zhang et al. (2018) constructed a dataset with workers imitating a given personal profile. Rashkin et al. (2019) built a dataset by explicitly asking workers to show their empathy during a conversation. Urbanek et al. (2019); Narayan-Chen et al. (2019); Ammanabrolu et al. (2021) created datasets with the assistance of game structures, so the purpose of the dialogue is to complete a mission or collaborate with other agents. Finally, recent work by Dou et al. (2021) collected branches of dialogues for 120 self-written prompts to create dialogue trees. Compared to previous studies, our dataset is a fusion of the scripts written by experts and responses created by crowd-sourcers with manual correction, granting it high quality, linguistic abundance, and extensive metadata. Additionally, our dataset includes both branch-splitting and
branch-colliding instances, which has led us to classify dialogues as directed acyclic graphs (DAGs) instead of just sequences or trees.
Dialogue Generation Training Objectives. To train a dialogue response generation model, methods have been developed from maximizing the likelihood between the hypothesis and the groundtruth (Vinyals and Le, 2015; Serban et al., 2016), guiding responses to match a higher reward in reinforcement learning (Li et al., 2016d), and allowing for extra latent variables to optimize divergence through variational autoencoder (Zhao et al., 2017) or generative adversarial networks (Li et al., 2017a; Tuan and Lee, 2019). Recent works have introduced the concept of causal inference (Holland, 1986; Imai et al., 2008; Pearl, 2009; Cunningham, 2021) into generative adversarial networkbased (Zhu et al., 2020) and multiple-stage inference based dialogue generation model (Tuan et al., 2020). Utterance-level offline reinforcement learning has also been explored to optimize response generation (Jaques et al., 2020; Verma et al., 2022). However, they were studied by expanding available sequence data with imaginations. Now by providing a chit-chat dialogue DAG structure that is enriched with multiple if-else cases, CausalDialogue can be studied for causal inference and offline reinforcement learning on response generation. We also propose a new method called ExMATE for better optimizing a response generation model on the DAG data structure.

heading: 3 CausalDialogue Dataset
text: In this section, we introduce CausalDialogue, a novel dataset that includes chit-chat conversations in a Conversational Directed Acyclic Graph (DAG) data structure. This structure allows for the natural inclusion of various dialogue flows, such as forks (branch-splitting) and colliders (branchcolliding) (Pearl, 2009). Our goal is to offer re-
searchers a valuable resource for studying the complexities of human conversation and advancing the understanding of causal inference in dialogue.
To create CausalDialogue, we sourced expertwritten dialogues from a role-playing game (Section 3.1) and expanded upon them with Amazon Mechanical Turk (MTurk)2 and manual correction (Section 3.2). By using our fused collection method, the dataset contains high-quality, engaging conversations with abundant linguistic usage that imitates daily life.

heading: 3.1 Data Collection
text: CausalDialogue is derived from the English scripts of the popular role-playing game (RPG) Fire Emblem: Three Houses, which we sourced from the fandom wikipedia3 under the GNU Free Documentation License(GFDL)4. This RPG is well-known for its diverse, story-driven conversations, which mix the interactions of approximately 40 main characters. In this game, players have the ability to shape the narrative by making choices that lead to different dialogue branches.
Table 2 lists the statistics of the two main types of the crawled data, which are already divided in the raw scripts. We name the first conversation type ORI.-2S, which are mostly dialogues between two speakers, and generally include conversations about interpersonal relationships. We name the second conversation type MULTI, which are dialogues between two or more speakers, and usually describe the current status of the story line. In the following sections, we will introduce the DAG structure to better describe the dataset, as well as how we obtained additional examples from crowd-sourcing to create the EXPANSION to these expert-written scripts.
2 https://www.mturk.com 3 https://fireemblem.fandom.com/ 4 https://fireemblem.fandom.com/wiki/Fire_
Emblem_Wiki:Copyrights
Dialogue DAGs. Conventional linear dialog data structures can be challenging to create when dealing with forks and colliders, as they can lead to ambiguity in the form of duplicated utterances and split responses. To address this issue, we propose using a conversational DAG to maintain the fidelity of the dialog. We convert each textual conversation into a DAG, as demonstrated in Figure 1. Formally, each node is a dictionary containing the text type (utterance/scene information), text, speaker, and its own id in the dialogue. A directed edge (i, j) then indicates that a node with id j is a possible response to the node with id i. Saving dialogues as DAGs may introduce some complexity, but it also offers numerous benefits. For example, it reduces the memory required to save each dialogue branch independently, enables a natural visualization of the multiple possible dialogues flows, and fosters the survey of causality on dialogue utterances.
Speaker Profiles. Prior work has shown the relationship between personality and language uses in conversations (Mairesse et al., 2007). To ensure consistent personality, as well as to diversify linguistic features across speakers, we leverage the speaker profiles during the data collection process. The resulting CausalDialogue dataset comprises 41 main speakers who have been thoughtfully crafted by the game’s developers. These speakers possess diverse backgrounds, perspectives, and interests, and their characteristics are both human-like and distinct. These speaker profiles are simplified for collecting the EXPANSION partition to reduce workers’ cognitive load, and a set of examples are provided in Appendix A.1. Compared with the speaker profiles in CausalDialogue, previous works
have provided limited information (e.g. “I have a dog.”) (Zhang et al., 2018; Urbanek et al., 2019), or have a significantly smaller number of speakers (Poria et al., 2019; Tuan et al., 2019)

heading: 3.2 Data Expansion
text: In order to increase the breadth and scope of our dataset, we propose utilizing a crowd-sourcing approach to add more diverse and current language as shown in Figure 2 (More details in Appendix A.2).
Initial Dialogue Selection. We first randomly select 1,200 partial dialogues from the ORI.-2S partition, which is of higher quality after our manual inspection. This can result in more stable quality when crowd-sourcing responses.
Expansion Collection. Each initial dialogue along with the continuing speaker profile is presented to 3 workers on MTurk to write the next utterance. A new branch of continued dialogue will then be presented to another 1-2 workers playing as another speaker to gather another round of responses. We repeated this process three times and collect a total of about 13,000 written utterances. Table 2 lists the detailed statistics of the expanded data in the column EXPANSION. Note that the statistics of EXPANSION in Table 2 include the initial dialogues. Figure 3 shows an DAG representation of an expanded example.
Quality Control. We adopt three strategies to control for dialogue quality. First, we asked the workers on MTurk to annotate if they regard a dialogue as already completed or having too specific of details to continue. The purpose of the first stage of quality control is to identify conversations which cannot be continued, either because the conversa-
tion has already concluded or because the workers are lacking enough information about the world to continue the conversation. Second, we used an off-the-shelf model5 to label potential ethical issues inside the collected utterances for reference in the next step. Finally, we invited real players of the game and machine learning researchers to manually check all the utterances by their fluency, coherence, and ethics as well as referring to the labels from the previous two steps to ensure the final EXPANSION partition is of high quality.

heading: 4 Task Definition
text: In this work we consider a conversation among two or more speakers. At each time step t, a speaker st takes their turn with an utterance ut. The goal, as in conventional response generation, is to train a model parameterized by θ that can predict a plausible next utterance given the speakers and utterances in prior turns as:
ut+1 ∼ Pθ(⋅∣s1u1, s2u2, ..., stut, st+1) . (1) Distinct from prior conversation datasets, CausalDialogue has multiple dialogue branches. If we consider each branch as an independent conversation (flatten the branches), many conversations will have large overlaps and thus bias the dataset. We consider this point and extract triples(DH,x, y) from CausalDialogue. To simplify notations for following sections, we denote stut as x, st+1ut+1 as y and DH is the dialogue history s1u1, s2u2, ..., st−1ut−1. The key idea is that for a DH , we will not extract duplicated pairs (x, y), but x or y itself can be shared.
5 https://github.com/unitaryai/detoxify
The CausalDialogue response generation task is therefore defined as finding a possible turn-taking speaker and their response given the dialogue history DH with an utterance cause x. y ∼ Pθ(⋅∣DH,x) . (2) The sequences x = x1x2...xi...x∣x∣ and y = y1y2...yj ...y∣y∣, where xi and yj are tokens, and∣x∣ and ∣y∣ are the length of the sequences x and y respectively.

heading: 4.1 Agility
text: While the above task definition resembles the standard dialogue generation setting with the exception of speaker prediction and conversation overlaps, our primary interest lies in tailoring responses to minor differences in conversation history. We refer to this concept as Agility, where a minor difference in conversations can be a shared DH with different continuation x.
To quantify the idea of agility, we propose a new metric with the following idea: If the predicted next utterance y and the previous turn x has causaleffect relationship (i.e., x1 → y1 and x2 → y2), we anticipate that it is less likely that y2 is caused by x1. The newly proposed metric, named confidence causal-effect (CCE) is formally defined as:
CCE =E(x,y)∈D,(x,y′)∉D,(x′,y′)∈D[PPLθ(y′∣DH,x) − PPLθ(y∣DH,x)] , (3)
where PPL refers to perplexity. Note that CCE is not a metric that stands by itself and needs to refer to PPL at the same time. That is, given a similar PPL score, a model with higher CCE score is better.
Additionally, it is important to acknowledge that the concept of agility has been indirectly incorporated into conventional dialogue generation models and evaluation metrics, but it has not been specifically examined in isolation. Our newly introduced dataset and CCE metric can be seen as an initial step towards addressing this aspect.

heading: 5 Methods
text: In this section, we describe how conventional generative models can be used and propose a simple yet effective approach to model causal effect.

heading: 5.1 Maximize Likelihood Estimation
text: An often used method to train a conditional sequence generation model is minimizing the negative log likelihood (Vinyals and Le, 2015; Serban et al., 2016). The loss function is as following:
LMLE = E(DH,x,y)∼PD ∣y∣ ∑ j=1
− logPθ(yj∣DH,x, y1...j−1) , (4)
where PD represents the data distribution. Since the duplication of dialogue history is already taken in to account in our task definition (Section 4), this MLE method can be seen as the recently proposed dialogue tree model (Dou et al., 2021). However, this function only models a part of the cause-effect relationship between the condition and the output sequence. This neglect may lead to a more vague predicted probability distribution of the output, thus generating less agile responses.

heading: 5.2 Maximize Average Treatment Effect
text: To explicitly model the causal effect in a conversation, we propose the Exponential Maximum Average Treatment Effect (ExMATE), taking into account the treatment effect in causal inference (Pearl, 2009). The treatment effect, denoted by δ, is defined as the difference between the outcome under treatment I = 1, represented by OI=1, and the outcome under treatment I = 0, represented by OI=0. This measures the variation in outcomes when an event I is present or absent. A higher value of δ indicates that the event I is more likely to be a true cause of the outcome. Conversely, a small value of δ suggests that the event I is unlikely to be a cause of the outcome and may only be correlated. We aim to utilize this characteristic in dialogue generation
modeling to ensure that a preceding utterance can be considered the genuine cause of the predicted response.
We consider the fork-like DAGs (as shown in Figure 4) existing in a dataset such as Figure 1 and Figure 3. Without loss of generality, in a binary case, this type of DAG involves two triples that share the same DH and can be simplified as having nodes DH , X1, X2, Y1, and Y2. Here we use (X1, Y1) and (X2, Y2) to denote two possibilities of (x,y) after DH . We take I = 1 as choosing the branch X1, and I = 0 as choosing an alternative branch X2. Therefore, a traditional definition of the treatment effect δi = ∣OI=1i − OI=0i ∣ for the i-th example in this type of DAG can be rewritten as:
δi ≜ E X1∼PD(⋅∣DHi), X2∼PD(⋅∣DHi),
X1≠X2 ∣OX1i −OX2i ∣ , (5) where OX1i or O X2 i is the outcome of an oracle given X1 or X2 as the input. Since the outcome of a dialogue model is hard to be mathematically described only by an input X , we instead utilize the uncertainty of predicting the pair (x, y) by a model θ. We abuse the notation Oi here and redefine it as,
OX1i,Y1 ≜ Pθ(Y1∣DH,X1) . (6) After formulating a dialogue generation problem as utterance-level causal analysis as above, we apply the Average Treatment Effects (ATE) (Holland, 1986) to conversational DAGs, which is defined as
ATE ≜ Ei[δi] = Ei[δi,Y1 + δi,Y2] = Ei[OX1i,Y1 −OX2i,Y1 +OX2i,Y2 −OX1i,Y2] . (7)
Recall that our goal is to strengthen the causeeffect relationship of each pair, (X1,Y1) and (X2,Y2) in the binary case. This can be taken as maximizing the defined ATE in Equation 7 with respect to the model parameters θ.
Therefore, we substitute the OXi,Y term in Equation 7 with its definition stated in Equation 6 and derive:
argmax θ
ATE =
argmax θ E(Xi,Yi)∼PD(⋅∣DH)Pθ(Yi∣DH,Xi) − E
Xi∼PD(⋅∣DH),Yj∼PD(⋅∣DH)(DH,Xi,Yj )∉D Pθ(Yj∣DH,Xi) .
(8)
To stabilize the training, we modify it with logarithmic and exponential terms and call it the ExMATE loss function. Formally, it is written as:
LExMATE = E(DH,x,y)∼PD, xc∼PD(⋅∣DH),(DH,xc,y)∉D − logPθ(y∣DH,x) + exp(logPθ(y∣DH,xc)) .
(9)
The intuition for this change is that without exp(⋅), the gradient of the second term will dominate the loss function, since log(u) has much larger gradient for u close to 0 than u close to 1 and an exp(⋅) term can linearize it.
Overall, the idea of ExMATE is to maximize the response generation model’s causal effects given a specific Xi (or (DH,x)) as the current cause. At the end, we found that this ATE-inspired approach turns out to be a combination of MLE and a subtraction of specific negative samples. This formulation shares a similar concept with negative sampling and contrastive learning (Goldberg and Levy, 2014; Chen et al., 2020), but has different example selection scheme and is not applied on the embedding space. With this method, we are
interested in the research question: Will a model trained on the CausalDialogue dataset be affected when using a causality-inspired loss?

heading: 6 Experiments
text: We provide a preliminary benchmark for CausalDialogue with often used methods and a naive causality-inspired loss. We fine-tuned two types of pretrained language models based on transformers (Vaswani et al., 2017): decoder-only architecture, DialoGPT (Zhang et al., 2020) and encoderdecoder architecture, T5 (Raffel et al., 2020), by the conventional MLE loss and the proposed ExMATE loss, and inferred by various sampling methods. We evaluate three aspects of the generated responses: Fluency (perplexity (PPL) and BLEU (Papineni et al., 2002)), Diversity (Distinct n-grams Dist1 and Dist2 (Li et al., 2016a)), and our proposed Agility (CCE) in Section 4.1. Furthermore, we use accuracy to evaluate if the speaker for a given turn is correctly predicted as the one in the human written responses (Identity Acc). More details are in Appendix A.4 and A.6.

heading: 6.1 Results
text: The test results of human written responses and models trained and inferred by different setups are listed in Table 3. We provide details of how we evaluate human written responses in Appendix A.5.
[Backbone Models] We observe that our trained T5 model is generally better than DialoGPT model, as evidenced by the significant difference in PPL and Identity Acc between them. [Inference Methods] We observe that Softmax and TopK can
achieve better results than greedy search in this dataset, as evidenced by their BLEU and DistinctN scores. The reason is similar to the conventional generic response problem in open-domain dialogue generation (Li et al., 2016a; Tuan and Lee, 2019), since in a DAG, a (DH , x) pair have multiple y as references, causing even an ideal probability distribution to have high entropy. [Loss Functions] We find that ExMATE improves MLE with better diversity, agility, and identity accuracy, while maintaining similar fluency scores. This meets our expectation that ExMATE should not deteriorate the MLE’s ability in training a model while maximizing the potential causal effect in response prediction. This result empirically shows that the causal effect can help to increase diversity and predict the turn-taking speaker as well. Finally, compared to the evaluation results of human written responses (a hard-to-reach upper bound), current methods still need improvement, except for diversity scores.

heading: 6.2 Human Evaluation
text: We randomly sample 100 dialogues, present each example to three workers on MTurk and ask them score the three dimensions, agility, coherence, and informativeness, scaling from 1 to 5. The evaluation form is provided in Appendix A.3. For each example, we present one shared dialogue history with two branches and the corresponding machine generated responses or a human written response. We randomly mix the human written ones to validate if the human evaluation is reliable to an extent, by anticipating human written ones will get higher
scores. We list the average ratings in Table 5. The model trained with ExMATE achieves a similar informativeness level as human written ones, and gets a higher agility rating, which is its main goal. However, ExMATE can compromises coherence due to the subtraction of a counter example, which is a natural sentence, in its objective function. The human evaluation demonstrates the challenge of models to meet human-level quality in CausalDialogue featured by conversational DAGs, a portion of the diversed types of flows in the real world.

heading: 6.3 Qualitative Analyses and Discussion
text: Table 4 shows an example of a shared dialogue history, four different continuations (case1-4), and responses generated by the same backbone model, T5, trained with different objectives and inferred with different sampling methods. We observe that responses produced by MLE+T (TopK), ExMATE+S (Softmax), ExMATE+T are generally coherent to the conversation, while ExMATE often produces more diverse and agile responses to different continuation cases (different x). It is notable that other than the improvements, we find
that all the models have three types of issues: mode collapse, semantic repetition, and identity misplacement. [Mode Collapse] The problem is often-seen when inferring a model by greedy search, specifically, the predicted responses often repeat the same phrase such as “I’m not sure”. While tacking the issue by adopting inference sampling, we conjecture the reason is that in a DAG, using a typical loss function can learn a probability distribution with higher entropy. This also demonstrates the need of a new loss function for training on a conversational DAG dataset. [Semantic Repetition] An example is the MLE+T response in Table 4 case 4, where “can’t help you ” and “help you out” have semantic overlaps. This issue can possibly be mitigated by repetition reduction techniques, such as unlikelihood training (Welleck et al., 2019) in future work. [Identity Misplacement] The problem happens when a model is confused about its position in a dialogue. For instance, the MLE+T response in Table 4 case 3 is more like an utterance of speaker Lysithea instead of Ignatz. This issue might be soothed by existing persona consistent techniques (Li et al., 2016b; Mazaré et al., 2018; Su et al., 2019) for building a overall good chatbot, while in this work, we focus on proposing a new dataset to benchmarking on the agility issue.

heading: 7 Conclusion
text: In this paper, we presented a new dataset, CausalDialogue, with novel conversational DAG structure. With experiments on various model setups with a newly proposed loss, ExMATE, we demonstrate that there is room for improvement to reach humanlevel quality, even though ExMATE improves the diversity, informativeness, and agility. This dataset serves as a testbed for future research that needs abundant conversation cases, like causal inference and offline reinforcement learning. Moreover, with the naturally paired metadata, future work can use this dataset for other tasks, such as speaker prediction in multi-speaker scenarioes and personalized dialogue generation.
Limitations
The introduced dataset has a moderate scale, as it is currently designed for fine-tuning instead of large model pretraining. Our proposed collection scheme can be futher applied to enlarge the dataset. Moreover, as we focus on English, the data source has multiple language versions written by
experts. Hence, extending CausalDialogue to multilingual is straightforward. With reward labeling, the dataset can be more intuitively used for offline RL. Meanwhile, the dataset includes personality descriptions that can be used for personalized dialogue generation, even though is not the focus in this paper. Finally, training a generative model on dialogue domain can require various computational costs, depending on the aspects such as lengths of input and output texts and number of model parameters, as well as special designs to prevent misuses.
Ethics Consideration
The dataset is based on RPG game in fantasy world with diverse scenarios, including wars. To match the story background, a model trained on this dataset might produce war-related words. We manually looked into each example to meanwhile keep each speaker’s personality and remove utterances that could potentially cause negative impact, such as violence, bias, and offensive words.
For the data annotation part and human evaluation part, we utilized the Amazon Mechanical Turk platform and required workers to have a HIT Approval Rate of greater than 95% and be located in CA or the US. We pay the annotators over 16 US dollars per hour on average, which is above the highest state minimum wage. Given our setting, the workers understood the scenarios and agreed that their annotations will be used for research. The data annotation part of the project is classified as exempt by Human Subject Committee via IRB protocols.

heading: Acknowledgement
text: This work was supported in part by the National Science Foundation under #2048122 and an unrestricted gift award from Google Deepmind. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the sponsors.

heading: A Appendix
text: A.1 Speaker Profiles
Table 6 provides a few examples of the speakers’ profiles and utterances.
A.2 Data Expansion Details
Initial Dialogue Selection. We first randomly select m dialogues with replacement from the ORI.2S partition, which is of higher quality after our manual inspection. This can result in more stable quality when doing crowd-sourcing. For each sampled dialogue, we randomly select a start time stamp t from Poisson(λ = 1). Next, we adjust the sampled time stamp t to make sure it lies in an appropriate point to continue the dialogue by t ∗ = max(min(t + 2, L), 2), where L is the maximum time stamp of this dialogue. For each time stamp, if the original dialogue has multiple possible nodes, we select one randomly from a uniform distribution. This process results in m initial dialogues D0 with various lengths (at least two utterances) for expansion.
Expansion Collection. Each initial dialogue D0 along with the continuing speaker profile is presented to n workers on MTurk to write the next utterance. The new continued dialogues D1 will then be presented to another 1-2 workers, decided by p%, playing as another speaker to gather another round of responses. This results in about mn((1+p)T −1)/p new utterances for data expansion, where T is the number of iterations. Our expansion data is set with m = 1200, n = 3, p = 0.2, and T = 3. This setting results in about 13,000 written utterances.
A.3 Human Annotations
Interface - Data Expansion. We design two user interfaces to launch on MTurk for the first stage and the remaining stages separately of the data expansion process. The interface used for the remaining stages is shown in Figure 5. We include detailed instructions about the step-by-step works, examples, and requirements to obey. We put some information into a button to reduce cognitive burden when writing for multiple hits.
Interface - Human Evaluation. Our used human evaluation form is shown in Figure 6.
Setup and Payments. We collect the expanded dataset and evaluate generated responses via
MTurk, a crowdsourcing platform. We obtained consent from workers by showing them the study purpose before they agree to do the annotations. We set additional restrictions of location to United States and Canada. We pay the annotators from 16- 18 US dollars per hour according to the difficulty of the collection stage (remaining stages are more difficult than the first stage). The payments are made to be higher than the law’s minimum wage 15 US dollars per hour in California in 2022 and 15.5 US dollars per hour in 2023, which are the highest among the US states.
A.4 Evaluation Metrics Here we discuss more about our selection of evaluation metrics.
Fluency. The predicted next utterance should be both coherent to the previous turn and consistent with the dialogue history. We evaluate the extent of coherence by perplexity and reference-based metric BLEU (Papineni et al., 2002). For nodes with multiple childs we use multiple references when computing BLEU metrics. Although that BLEU may not be well correlated with human intuition in conversation (Liu et al., 2016), we use it for reference as it is still widely used in dialogue generation. The perplexity (PPL) is considered to be the less the better, whereas BLEU is the higher the better.
Diversity. A dialog model can suffer from the generic issue that given different dialogue history and previous turn, the predicted utterance is similar, such as “I’m sorry”. We adopt distinct-N scores (Dist1 and Dist2) to evaluate this dimension by considering the percentage of distinct n-grams within the total number of n-grams in the corpus-level (Li et al., 2016a). However, the distinct-N scores are not always the higher the better. We can think about this in a intuitive example, if we randomly sample words from a uniform distribution, the distinct-N score can be high but meaningless. We anticipate a good distinct-N score is in a similar range as the score evaluated on human written responses.
A.5 Evaluate Human Responses The PPL on human written responses are evaluated by an oracle that will predict a uniform distribution over all human written responses y given the same (DH,x). The BLEU scores on human written responses are evaluated on data examples with multiple possible responses and the response to be evaluated is hold out from the reference set.
Otherwise, the BLEU scores will be 100 since the response to be evaluated is within the reference set.
A.6 Experiment Details Model architecture. We use DialoGPT-small with 117M parameters and T5-base with 250M parameters. DialoGPT model is based on the GPT model architecture (a transformer decoder) but pretrained on conversation-like dataset such as Reddits. T5 model uses the transformer encoder-decoder architecture and is pretrained on web-extracted text from Common Crawl, which is a publicly-available web archive of scraped HTML files. The maximum tokens allowed as the input are 256.
Hyperparameters. For hyperparameter search, we tried the learning rate from {5e-5,2e-5,1e-5} and the batch size times gradient accumulation steps from {32,64,128}. We found out that using a learning rate 1e-5 and batch size 64 can generally fine-tuning a model well with different learning algorithms in our experiments. We train each model with different combinations of setups for single run.
Data preprocessing. For data preprocessing, we have tried to utilize the original case and punctuation, transform all words into lower case, or meanwhile remove all punctuation.
Computation Resources. Each model is train on one Titan RTX or one RTX A6000 and costs around five hours.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: We have included a limitation section after the main content.

heading: 3 A2. Did you discuss any potential risks of your work?
text: We have included a limitation section and an ethical consideration section after the main content.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: In Abstrat and Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: In Section 3,4,5,6

heading: 3 B1. Did you cite the creators of artifacts you used?
text: In Section 3,5,6

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: In Section 3

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? In Section 3. We include the license and follow the intended use.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? In Section 3

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? In Section 3 and Appendix A.3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. In Section 3
C 3 Did you run computational experiments?

heading: In Section 6

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? In Appendix A.6
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? In Section 6 and Appendix A.6

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? In Appendix A.6

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? In Appendix A.6 and Supplementary Material.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: In Section 3.

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? In Appendix A.3

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? In Appendix A.3 and Ethics Consideration section

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? In Appendix A.3 and Ethics Consideration section
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? The data annotation part of the project is classified as exempt by Human Subject Committee via IRB protocols.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: In Appendix A.3 and Ethics Consideration section

heading: Future Work
text: This dataset serves as a testbed for future research that needs abundant conversation cases, like causal inference and offline reinforcement learning. Moreover, with the naturally paired metadata, future work can use this dataset for other tasks, such as speaker prediction in multi-speaker scenarioes and personalized dialogue generation. Limitations
The introduced dataset has a moderate scale, as it is currently designed for fine-tuning instead of large model pretraining. Our proposed collection scheme can be futher applied to enlarge the dataset. Moreover, as we focus on English, the data source has multiple language versions written by
experts. Hence, extending CausalDialogue to multilingual is straightforward. With reward labeling, the dataset can be more intuitively used for offline RL. Meanwhile, the dataset includes personality descriptions that can be used for personalized dialogue generation, even though is not the focus in this paper. Finally, training a generative model on dialogue domain can require various computational costs, depending on the aspects such as lengths of input and output texts and number of model parameters, as well as special designs to prevent misuses