Venue: EMNLP 2020

# Counterfactual Off-Policy Training for Neural Dialogue Generation

Qingfu Zhu\({}^{\ddagger}\), Weinan Zhang\({}^{\sharp}\)1 , Ting Liu\({}^{\sharp}\), William Yang Wang\({}^{\flat}\)

\({}^{\sharp}\)Harbin Institute of Technology, Harbin, China

\({}^{\flat}\)University of California, Santa Barbara, USA

{qfzhu, wnzhang, tliu}@ir.hit.edu.cn

william@cs.ucsb.edu

Footnote 1: Corresponding author.

###### Abstract

Open-domain dialogue generation suffers from the data insufficiency problem due to the vast size of potential responses. In this paper, we propose to explore potential responses by counterfactual reasoning. Given an observed response, the counterfactual reasoning model automatically infers the outcome of an alternative policy that could have been taken. The resulting counterfactual response synthesized in hindsight is of higher quality than the response synthesized from scratch. Training on the counterfactual responses under the adversarial learning framework helps to explore the high-reward area of the potential response space. An empirical study on the DailyDialog dataset shows that our approach significantly outperforms the HRED model as well as the conventional adversarial learning approaches.

## 1 Introduction

Open-domain dialogue generation Shang et al. (2015); Vinyals and Le (2015); Sordoni et al. (2015) intends to produce coherent responses given dialogue history. Nevertheless, it suffers from data insufficiency problem as there may exist many potential responses for a given dialogue history Li et al. (2016). An ideal way of exploring the potential responses is to train the model by chatting with real users, which is usually time-consuming and labor-intensive in practice. Although replacing a real user with a user simulator could address the issue, the simulator only roughly approximates real user statistics, and its development process is costly Su et al. (2016).

In contrast, humans could independently reason potential responses based on past experiences from the true environment. Having observed a response, one might naturally ask himself or herself: "_What would happen if I respond differently, while everything else in the environment remains the same._" Answering the question will result in a potential response (as an example in Figure 1), and it is beneficial for improving future decision making Roese (1997). The potential response inferred in hindsight is called a _counterfactual_ response, where the concept "counterfactual" describes the posterior process of reasoning the outcome of alternative actions (i.e., a different responding policy) that could have been taken while keeping everything else unchanged Buesing et al. (2019).

Motivated by this, we propose a counterfactual off-policy training (COPT) approach to explore potential responses. Building upon the adversarial learning framework, COPT casts a dialogue generator as a structural causal model (SCM), which describes a generation process with two ingredients: _scenarios_ and _causal mechanisms_Wright (1920); Buesing et al. (2019). The scenario is a random noise variable that captures all unobserved yet relevant aspects of the environment, i.e., user profiles. The causal mechanism is a deterministic function that takes a scenario and dialogue history as input and outputs a response. In this way, reasoning a counterfactual response in an observed response's environment can be achieved by feeding the scenario of the observed response into the

Figure 1: An example of a counterfactual response, which is a potential response inferred in hindsight from given observed response.



causal mechanism. After generating the counterfactual response, the generator will receive a reward from a discriminator and optimize itself accordingly.

Intuitively, a counterfactual response is synthesized by grounding the model in the scenario where an observed response occurs, rather than the scenario sampled from scratch as standard adversarial learning-based approaches. This improves the quality of the synthesized responses and subsequently benefits the model that learns from the synthesis. To verify the effectiveness of our approach, we conduct experiments on the public available DailyDialog dataset Li et al. (2017). Experimental results show that our approach significantly outperforms previous adversarial learning-based approaches in both automatic and human evaluations. The contributions of this paper are summarized as follows:

* We connect the concept of counterfactual reasoning with the dialogue generation by casting the dialogue generation model as a structural causal model.
* Our counterfactual response is of higher quality than the response synthesized from scratch in standard adversarial learning-based dialogue generation model.
* Our approach is model-agnostic and can be applied to any adversarial learning-based dialogue generation model.

## 2 Related Work

Dialogue GenerationData-driven dialogue systems can be roughly divided into two categories: retrieval-based Leuski et al. (2006); Ji et al. (2014); Yan et al. (2016) and generation-based Shang et al. (2015); Sordoni et al. (2015); Vinyals and Le (2015). Responses of retrieval-based methods come from a fixed candidate response set and thus are incapable of being customized. The generation-based methods can create new responses, but the vanilla sequence to sequence model tends to produce generic responses Li et al. (2016).

One way to address the generic response problem is by introducing external knowledge, such as keywords Mou et al. (2016); Zhu et al. (2019), topics Xing et al. (2017), persona information Zhang et al. (2019); Song et al. (2019), and retrieved candidate responses Song et al. (2018); Wu et al. (2019); Zhu et al. (2019). Another way is to optimize the architecture of networks. There are two architectures widely employed in this research line: the variational auto-encoder Bowman et al. (2016); Zhao et al. (2017) and the generative adversarial network Goodfellow et al. (2014); Li et al. (2017); Zhang et al. (2018); Xu et al. (2018); Tuan and Lee (2019). Our approach falls into the latter category. The differences between our approach and other adversarial learning-based approaches are as follows. First, we cast the dialogue generation model as an SCM to explore potential responses in the environment where observed responses occur. Second, we learn on counterfactual responses that inferred from the SCM. Third, a pre-trained behavior policy is involved during the generation process, making our approach an off-policy algorithm and benefits the exploration of potential responses.

Counterfactual ReasoningThe counterfactual reasoning is a concept derived from psychology. It describes the human capacity to learn from experience by reasoning the outcome of an alternative action that could have been taken Pearl and Mackenzie (2018). Combined with the SCM, counterfactual reasoning improves the performance of policy evaluation in reinforcement learning Buesing et al. (2019); Oberst and Sontag (2019). In the area of NLP, counterfactual reasoning in previous work is mainly used for data augmentation Qin et al. (2019); Fu et al. (2020); Kaushik et al. (2020), which rewrites the original data given a counterfactual label or condition. In this paper, we connect the concept of counterfactual reasoning with the dialogue generation and are the first to cast a generation model as an SCM under the adversarial learning framework.

## 3 Method

We cast a dialogue generation model as an SCM to explore potential responses by counterfactual reasoning during the training process. We will first review the concept of the SCM (Sec. 3.2), and then introduce our COPT approach (Sec. 3.3).

### Notation

We use capital letters for random variables (e.g., \(V\)), lowercase letters for instances of random variables (e.g., \(v\)), and bold letters for vectors (e.g., \({\mathbf{V}}=\left\{V_{1},...,V_{N}\right\}\)). During the training process, we denote the response generated by COPT as _counterfactual response_. In contrast, the response of standard adversarial learning-based dialogue 

generation (i.e., REGS Li et al., 2017a) is denoted as _standard response_.

### Background: Structural Causal Model

A structural causal model over random variables consists of independent noise random variables with distribution and deterministic functions , where the parents in a given DAG (Buesing et al., 2019). is called _scenarios_, and is called _causal mechanisms_. Figure 2 (Left) shows an example of an SCM. Each random variable is determined by its parents in , , , e.g., .

During the training process, we cast a dialogue generation model as an SCM over two random variables: dialogue history and response . This is achieved by converting the conditional distribution into (for more details see Sec. 3.3). The scenario is a random noise variable that captures all unobserved yet relevant properties, like user profiles. The causal mechanism is denoted as to highlight the role of the policy (parameters) of the model. The dialogue generation SCM makes it possible to sample counterfactual responses in the scenario where observed responses occur. This improves the quality of synthesized responses and subsequently helps the model to explore the high-reward area of the potential response space in the training process.

Intervention in SCMGiven an SCM, an intervention is defined as the replacement of some causal mechanisms. Figure 2 shows an example of intervention. The original causal mechanism in the left SCM is replaced with resulting in a new SCM in the right. Accordingly, intervention in our dialogue generation SCM denotes the update of the policy. For instance, the update from the behavior policy that generates observed responses to the target policy that we aim to learn is the intervention of replacing with .

Counterfactual Reasoning in SCMGiven an SCM and observed a variable , counterfactual reasoning answers the question: _"What the variable would have been if I take an intervention ? While remaining everything else unchanged"_. In this way, generating a counterfactual response can be seen as querying: _"Having observed a response , what the response would have been if I take an intervention by following the target policy , rather than the behavior policy that generates the observed responses"_.

Typically, counterfactual reasoning answers the question by the following steps (as Figure 3):

* Observed when , infer the scenario in hindsight from .
* Take an intervention by replacing the causal mechanism with .
* Reason a counterfactual response by the resulting new SCM.

In the following sections, we denote an observed response from the training set as and a model-generated response as .

### Counterfactual Off-Policy Training

Our COPT approach is model-agnostic and can be applied to any adversarial learning-based dialogue generation model. Without loss of generality, we take the combination of COPT and the reward for every generation step (REGS) model (Li et al., 2017a) as an example in this section. It consists of two main components: a generator and a discriminator .

GeneratorThe generator is a sequence to sequence (Seq2Seq) model (Sutskever et al., 2014) equipped with the attention mechanism (Bahdanau et al., 2015; Luong et al., 2015). During the encoding process, reads the dialogue history into hidden states using an encoder LSTM (Hochreiter and Schmidhuber, 1997):

(1)

where is the -th word of the dialogue history, and denotes the corresponding hidden state.

Figure 2: An example of an SCM and an intervention. **Left:** An SCM with random variables , scenarios and causal mechanisms represented by colored squares. **Right:** A new SCM after taking an intervention on the left SCM. The original causal mechanism (denoted by the orange square) is replaced by (denoted by (denoted by the purple square).

 

At the \(j\)-th decoding time step, the hidden states are summarized into a context vector \({\mathbf{C}_{j}}\) by the attention mechanism. Subsequently, \(G\) predicts the distribution of the next word over the vocabulary by a decoder LSTM:

\[{\mathbf{S}_{j}}=\text{LSTM}([\,{\mathbf{e}(\hat{Y}_{j-1})},{\mathbf{C}_{j}}],{ \mathbf{S}_{j-1}}), \tag{2}\] \[{\mathbf{P}^{\pi}_{j}}(\hat{Y}_{j}|{\mathbf{X}},\hat{\mathbf{Y}}_{1:j-1})= \text{softmax}({\mathbf{S}_{j}}\cdot{\mathbf{O}}), \tag{3}\]

where the bracket \([\cdot,\cdot]\) denotes concatenation, and \({\mathbf{e}(\cdot)}\) denotes the embedding of a word. \({\mathbf{S}_{j}}\) is the \(j\)-th hidden state of the decoder LSTM. \(\hat{Y}_{j-1}\) is the word generated in the previous time step. \({\mathbf{O}}\) is the output matrix. We use the superscript in \({\mathbf{P}^{\pi}_{j}}\) to highlight the role of the policy (parameters of \(G\)).

Adversarial learning-based dialogue generation model is optimized according to the reward of responses sampled from \({\mathbf{P}^{\pi}_{j}}(\hat{Y}_{j}|{\mathbf{X}},\hat{\mathbf{Y}}_{1:j-1})\in\mathbb{R}^{|V|}\) (abbreviated as \({\mathbf{P}^{\pi}_{j}}\) in the following), where \(|V|\) is the vocabulary size. Using the Gumbel-Max trick (Luce, 2012), the sampling process can be achieved by:

\[\hat{Y}_{j}= \arg\max_{\hat{Y}_{j}}(\log{\mathbf{P}^{\pi}_{j}}+{\mathbf{U}_{j}}), \tag{4}\]

where the element of \({\mathbf{U}_{j}}\) follows the standard Gumbel distribution. In this way, the generator turns into a Gumbel-Max SCM (Oberst and Sontag, 2019), whose scenarios and causal mechanisms are represented by \({\mathbf{U}_{j}}\) and Equation 4, respectively.

From the perspective of the SCM, each response is generated in a scenario. For instance, a standard response is produced in a scenario sampled from scratch. In contrast, the scenario for a counterfactual response is inferred from an observed response \({\mathbf{y}}=\{y_{j}|\,y_{j}=\arg\max_{y_{j}}(\log{\mathbf{p}^{*}_{j}}+{\mathbf{u}_{j}})\}\), where * is the user's policy that generates the observed response in the true environment. However, the user's policy is not available in practice, which hinders the posterior inference of the scenario. To this end, we introduce a behavior policy \(\mu\) instead and learn it by minimizing the MLE loss on observed responses. In this way, an observed response can be seen as generating in a scenario \({\mathbf{u}_{j}}\) while following the policy \(\mu\): \(y_{j}=\arg\max_{y_{j}}(\log{\mathbf{p}^{\mu}_{j}}+{\mathbf{u}_{j}})\).

According to Oberst and Sontag (2019), there are two ways to infer the scenario \({\mathbf{u}_{j}}\) in hindsight from \(y_{j}=\arg\max_{y_{j}}(\log{\mathbf{p}^{\mu}_{j}}+{\mathbf{u}_{j}})\) given \(y_{j}\) and \(\mu\). One way is the rejection sampling, which samples \({\mathbf{u}_{j}}\) from the standard Gumbel distribution and rejects those where \(y_{j}\neq\arg\max_{y_{j}}(\log{\mathbf{p}^{\mu}_{j}}+{\mathbf{u}_{j}})\). The other way of the posterior inference makes use of the properties of the shifted Gumbel \({\mathbf{g}}=\log{\mathbf{p}^{\mu}_{j}}+{\mathbf{u}_{j}}\): the maximum of \({\mathbf{g}}\) follows the standard Gumbel distribution and is independent with the argmax of \({\mathbf{g}}\)(Maddison et al., 2014). Therefore, \({\mathbf{g}}\) can be obtained by first sampling a maximum and then sampling the remaining elements truncated at the maximum. And \({\mathbf{u}_{j}}\) is subsequently computed by subtracting \(\log{\mathbf{p}^{\mu}_{j}}\) from \({\mathbf{g}}\). We employ the second method to infer the scenario in COPT because it is more time-efficient than rejection sampling1.

Footnote 1: In our experiments, using the second sampling method takes 0.79 seconds training on a batch when the batch size is 64. In contrast, the rejection sampling takes roughly 1.45 hours, making it hard to be used in practice.

Given the scenario inferred from the observed response, COPT reasons the counterfactual response by feeding the dialogue history and the scenario into the SCM (Equation 4). Then the discriminator evaluates the counterfactual response and returns

Figure 3: The architecture of our COPT approach. \(\pi\) is the target policy that we aim to learn. \(\mu\) is the behavior policy that generates observed responses. First, we infer the scenario \({\mathbf{u}}\) where the observed response occurs. Then we update the policy from \(\mu\) to \(\pi\), which can be seen as an intervention on the left SCM and results in the right SCM. Then, the counterfactual response is reasoned in the inferred scenario \({\mathbf{u}}\) by the causal mechanism \({\mathbf{Y}}=f_{\pi}({\mathbf{X}},{\mathbf{U}})\).

 

a reward to the generator. Note that the counterfactual response and the SCM are utilized for the training process. During the inference process, responses are generated in the same way as the standard adversarial learning-based dialogue generation (beam search or sampling from \({\mathbf{P}_{j}^{\pi}}\), we use the former in our approach) because the observed response is not available.

DiscriminatorThe discriminator \(D\) provides a reward for each generation step. It takes as input the dialogue history \({\mathbf{X}}\), the word \(\hat{Y}_{j}\) produced in the current generation step, and the prefix \(\tilde{{\mathbf{Y}}}_{1:j-1}\) in previous steps, where \(\tilde{{\mathbf{Y}}}\in\{{\mathbf{Y}},\tilde{{\mathbf{Y}}}\}\) can be either an observed response or a model-generated response. The output reward \(D(\hat{Y}_{j}|{\mathbf{X}},\tilde{{\mathbf{Y}}}_{1:j-1})\) is the probability that \(\hat{Y}_{j}\) is human-generated. Concretely, \(D\) first reads \({\mathbf{X}}\) and \(\tilde{{\mathbf{Y}}}_{1:j}\) with an encoder-decoder model. Then, it computes the reward by a Multi-Layer Perceptron (MLP), which takes as input the last hidden state of the decoder.

Adversarial LearningWe train \(G\) and \(D\) under the adversarial learning framework, where \(G\) tries to fool \(D\) by generating human-like responses while \(D\) aims to distinguish between model-generated and human-generated (the observed) responses. Since a response is a sequence of discrete tokens, we pass by the gradient of \(D\) to \(G\) using the policy gradient algorithm. In this way, \(G\) converts into an agent whose partially generated response and parameters define a state and a policy, respectively. At each generation step, the agent takes an action by producing a word and observes a reward from \(D\) to update its policy.

Note that there are two policies in COPT: the target policy that we aim to learn and the behavior policy used for the reasoning of scenarios. The behavior policy is pre-trained and then froze during adversarial learning because it aims to maximize the likelihood of a fixed set of observed responses. Introducing the behavior policy makes COPT an off-policy approach because the counterfactual response, from which the target policy learns, is not entirely based on the target policy itself.

```
1:Pre-train \(\pi\) and \(\mu\) with MLE loss;
2:Pre-train \(D\) on positive instances sampled from observed responses, and negative instances generated by pre-trained \(\pi\);
3:for epoch in number of epochs do
4:for\(g\) in g-steps do
5: Infer \({\mathbf{u}}\) from an observed response;
6: Generate a counterfactual response in \({\mathbf{u}}\);
7: Optimize \({\mathbf{\theta}}\) according to Equation 5;
8:endfor
9:for\(d\) in d-steps do
10: Sample positive instances from observed responses;
11: Sample negative instances from responses generated by \(\pi\);
12: Update \({\mathbf{\phi}}\) according to Equation 6;
13:endfor
14:endfor
```

**Algorithm 1** Counterfactual Off-Policy Training

The goal of the generator is to minimize the negative expected reward: \(J_{G}({\mathbf{\theta}})=-\mathbb{E}_{\hat{{\mathbf{Y}}}_{1:j}\sim G}D(\hat{Y}_{j}|{ \mathbf{X}},\hat{{\mathbf{Y}}}_{1:j-1})\), where \({\mathbf{\theta}}\) is the parameters of \(\pi\). The gradient of \({\mathbf{\theta}}\) can be derived by the likelihood ratio trick Williams (1992):

\[\bigtriangledown J_{G}({\mathbf{\theta}})= -\mathbb{E}_{\hat{{\mathbf{Y}}}_{1:j}\sim G}D(\hat{Y}_{j}|{\mathbf{X}}, \hat{{\mathbf{Y}}}_{1:j-1})\] \[\cdot\bigtriangledown\log G_{\pi}(\hat{Y}_{j}|{\mathbf{X}},\hat{{\mathbf{ Y}}}_{1:j-1}), \tag{5}\]

where \(G_{\pi}(\hat{Y}_{j}|{\mathbf{X}},\hat{{\mathbf{Y}}}_{1:j-1})\) is the probability of generating \(\hat{Y}_{j}\) with the policy \(\pi\) given \({\mathbf{X}}\) and \(\hat{{\mathbf{Y}}}_{1:j-1}\).

The discriminator distinguishes between observed responses and model-generated responses. This is achieved by minimizing the following loss:

\[J_{D}({\mathbf{\phi}})= -\mathbb{E}_{{\mathbf{Y}}_{1:j}\sim{\mathbf{S}}}\log D(Y_{j}|{\mathbf{X}},{ \mathbf{Y}}_{1:j-1}) \tag{6}\] \[-\mathbb{E}_{\hat{{\mathbf{Y}}}_{1:j}\sim G}\log(1-D(\hat{Y}_{j}|{ \mathbf{X}},\hat{{\mathbf{Y}}}_{1:j-1})),\]

where \({\mathbf{\phi}}\) is the parameters of \(D\). As a positive instance, \({\mathbf{Y}}_{1:j}\) is a prefix randomly sampled from observed response set \({\mathbf{S}}\). A negative instance \(\hat{{\mathbf{Y}}}_{1:j}\) for training \(D\) is a prefix of a standard response, rather than a counterfactual response. This is because the latter is of higher quality than the former (as shown in Sec. 4.7).

Pre-trainingInitialized with different parameters, \(\pi\) and \(\mu\) are pre-trained on the training set with MLE loss. The pre-training of \(D\) depends on the specific model that COPT applied to. For example, REGS pre-trains \(D\) on the prefix of a response. In contrast, the discriminator of StepGAN Tuan and Lee (2019) is randomly initialized during the adversarial learning process. The overall algorithm of COPT is summarized as Algorithm 1 



## 4 Experiments

### Data

The experiments are conducted on the DailyDialog dataset Li et al. (2017).2 It is a multi-turn dialogue dataset and covers various topics of daily life. The dataset has already been divided into training, validation, and test sets, as shown in Table 1. Given a dialogue that consists of \(K\) utterances, we divide it into \(K\)-1 instances. Each instance has at most three continuous utterances. The last utterance is the response, and the previous utterances are concatenated as the dialogue history.

Footnote 2: [http://yannan.li/dailydialog](http://yannan.li/dailydialog)

### Baselines

We compare COPT with the following dialogue generation models:

* HRED Serban et al. (2016): The hierarchical recurrent encoder-decoder. An implementation by Park et al. (2018) is available3. Footnote 3: [https://github.com/ctrdsi/A-Hierarchical-Latent-Structure-for-Variational-Conversation-Modeling](https://github.com/ctrdsi/A-Hierarchical-Latent-Structure-for-Variational-Conversation-Modeling)
* REGS Li et al. (2017): Reward for every generation step. Its discriminator is trained on partially generated responses to provide a reward for each generation step.
* DPGAN Xu et al. (2018): The diversity-promoting GAN introduces a language model based discriminator to encourage the generation of informative responses.4 Footnote 4: [https://github.com/lancopku/DPGAN](https://github.com/lancopku/DPGAN)
* StepGAN Tuan and Lee (2019): The stepwise GAN trains the discriminator by maximizing the average of state-action values of observed responses. During the adversarial learning process, the discriminator assigns scores for every generation step in the same way as REGS.

Distinct from previous approaches, COPT casts a dialogue generation model as an SCM and trains it on counterfactual responses. It is model-agnostic and can be applied to any adversarial learning-based dialogue generation model, such as REGS, DPGAN, and StepGAN.

### Training Details

We implement REGS, StepGAN, and their variants with COPT using OpenNMT Klein et al. (2017), an open-source framework for building sequence to sequence models. We manually tune the parameters according to the perplexity on the validation set. The vocabulary consists of the most frequent 10,000 words. Including more words (up to 17,438, the total number of DailyDialog vocabulary) observes no improvement but takes more time for training. We use 300 dimensional GloVe Pennington et al. (2014) vectors to initialize word embeddings. Both the encoder and the decoder are a two-layer LSTM in \(G\) and a single layer LSTM in \(D\). The number of hidden units is 500.

During the adversarial learning process, we use the ADAM algorithm to alternately optimize \(G\) and \(D\) for one batch and five batches. The batch size is 64. We have tested the learning rate from 1e-6 to 1e-3. REGS+COPT and StepGAN+COPT achieve the best performance on 1e-5. The number of parameters for all the baselines is in a range of 21M to 26M. Equipping an adversarial learning baseline with COPT will introduce extra parameters with the same amount of the generator's parameters. Contributed by the behavior policy, the parameters are learned by pre-training, and COPT will not increase the number of trainable parameters in adversarial learning. Table 2 shows the average training time. COPT may increase the training time due to the posterior inference of scenarios. But it facilitates the exploration of the high-reward area of the potential response space and subsequently improves the quality of responses.

\begin{table}
\begin{tabular}{c r} \hline \hline Training Dialogues & 11,118 \\ Validation Dialogues & 1,000 \\ Test Dialogues & 1,000 \\ Average Tokens Per Dialogue & 114.7 \\ Average Tokens Per Utterance & 14.6 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of the DailyDialog dataset.

\begin{table}
\begin{tabular}{l r} \hline \hline Model & Time (s/epoch) \\ \hline HRED & 84 \\ DPGAN & 608 \\ REGS & 912 \\ REGS+COPT & 1,215 \\ StepGAN & 951 \\ StepGAN+COPT & 1,244 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The average training time (in seconds per epoch) on a single GPU.





### Evaluation Metrics

Automatic EvaluationWe evaluate the diversity and the relevance of generated responses using _distinct_Li et al. (2016) and _BLEU_Papineni et al. (2002), respectively. The distinct-\(k\) is the number of distinct \(k\)-grams normalized by the number of words of responses. Since BLEU might correlate weakly with human judgments of quality in the single-reference setting Liu et al. (2016), we use the multi-reference DailyDialog test set Gupta et al. (2019), where each instance is augmented with four human-written diverse responses.5

Footnote 5: [https://github.com/prakharguptaz/multirefeval](https://github.com/prakharguptaz/multirefeval)

Human EvaluationThe human evaluation is conducted on 200 instances randomly sampled from the test set. We create a project on Amazon Mechanical Turk Buhrmester et al. (2016) (AMT) and employ five AMT workers to give a preference between two responses generated by our approach and a baseline.6 To maintain the quality of the evaluation, the task is visible to workers whose approve rate is greater than 95%, and the number of approved is greater than 500.

Footnote 6: [https://requester.mturk.com/](https://requester.mturk.com/)

### Results

Table 3 shows the results of automatic evaluation. Both REGS and StepGAN outperform HRED in distinct-1 and distinct-2, indicating that adversarial learning is beneficial for improving the diversity of responses. There is no increase in DPGAN compared with HRED in our experiments. We believe this is because the scale of the DailyDialog dataset is not large enough for sufficiently training the language model based discriminator. For the same reason, COPT is not added to DPGAN. After introducing COPT, both distinct-1 and distinct-2 in REGS and StepGAN further increase, and the improvement is significant (t-test, \(p<\)0.01). This suggests that COPT is model-agnostic to adversarial learning-based approaches and helps to promote the diversity. In terms of BLEU in Table 3, both REGS and StepGAN achieve higher BLEU scores with COPT, and the improvements of BLEU-1 and BLEU-2 are significant (\(p<\)0.05). This demonstrates the effectiveness of COPT in improving the relevance of responses. The less significant result of BLEU-3 and BLEU-4 is mainly due to the sparsity of tri-grams and four-grams, which are harder to be covered by references than uni-grams and bi-grams.

The human evaluation results are shown in Table 5. Our approach is clearly preferred as it

\begin{table}
\begin{tabular}{l l} \hline \hline Dialogue History & A: This is our counter-offer. \\  & B: We are sincere to make business with you, but the difference between your counter-offer and our price is too great. \\ \hline HRED & A: We can do that, but you can take a look at this one. \\ DPGAN & A: Think that the will be, but he said that they had a think we have a good job \\  & in a. Me, I am sure that I can be able to have a look at the first course. \\ REGS & A: If you are willing to withdraw money, we can offer you a private vehicle. \\ REGS+COPT & A: If you are willing to pay a commission fee, we can offer you $5 \% discount. \\ StepGAN & A: What do you mean by that? \\ StepGAN+COPT & A: In that case, we can offer you a discount. \\ \hline \hline \end{tabular}
\end{table}
Table 4: An example of generated responses given dialogue history between person A and B.

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c} \hline \hline Model & Dist-1 & Dist-2 & BLEU-1 & BLEU-2 & BLEU-3 & BLEU-4 \\ \hline HRED Serban et al. (2016) & 0.011 & 0.045 & 33.01 & 4.54 & 1.05 & 0.29 \\ DPGAN Xu et al. (2018) & 0.002 & 0.008 & 31.61 & 3.74 & 0.43 & 0.06 \\ \hline REGS Li et al. (2017a) & 0.021 & 0.097 & 38.39 & 6.85 & 2.00 & 0.68 \\ REGS+COPT & **0.026** & 0.116 & 39.80 & 7.67 & 2.34 & 0.77 \\ \hline StepGAN Tuan and Lee (2019) & 0.018 & 0.097 & 34.76 & 6.51 & 1.97 & 0.60 \\ StepGAN+COPT & **0.026** & **0.117** & **43.83** & **8.43** & **2.66** & **0.95** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Automatic evaluation results of distinct-1 (Dist-1), distinct-2 (Dist-2), and BLEU scores.



has more winning instances than losing instances (\(p<\)0.01). The results indicate that COPT helps improve the quality of responses. Following Zhou et al. (2018) and Ke et al. (2018), we measure the agreement of annotators using inter-rater consistency. The percentage of instances that at least three annotators have the same preference (3/5 agreement) is 84.18%. The percentage for 4/5 agreement is 46.89%.

### Case Study

Table 4 shows an example of responses generated by baselines and our approach. The response of DPGAN sometimes is not fluent and can be very long. We believe this is also because the scale of the DailyDialog dataset is not enough for the language model discriminator. The response of HRED is not as informative as that of our approach. Its first part is generic, and what the pronoun "that" refers to is not clear. The response of StepGAN is not informative enough as well. In contrast, the response of REGS is quite informative, but its content is not entirely relevant to the dialogue history. After introducing COPT, the responses of REGS+COPT and StepGAN+COPT propose offering a discount to address Person B's concern of the price, which is both informative and relevant.

### Analysis

To further analyze COPT's effectiveness in exploring the high-reward area of the potential response space during the training process, we compare the reward of a counterfactual response and a standard response on the same 10,000 randomly sampled training instances. However, the comparison between the two types of responses could be biased if their rewards are computed by different discriminators. Besides, the quality of responses is determined not only by the way they generated (with or without COPT) but also by the generator. To focus on the analysis of COPT and eliminate the bias between generators and discriminators, we generate and evaluate the two types of responses using an identical generator and its corresponding discriminator. Here, we use REGS+COPT and StepGAN+COPT as testbeds because they could generate both the two types of responses.

Figure 4 shows the distribution of rewards and

Figure 4: Reward distribution and the average reward of the counterfactual response (w/ COPT) and the standard response (w/o COPT). The y-axis in (a) and (b) is the percentage, and the x-axis corresponds to three reward intervals in different epochs, including: Low reward interval [0.00, 0.33], Middle reward interval (0.33, 0.66], and High reward interval (0.66, 1.00]. The y-axis in (c) and (d) is the reward, and the x-axis corresponds to epochs.

\begin{table}
\begin{tabular}{l|c c c} \hline \hline  & Win & Tie & Lose \\ \hline REGS+COPT vs. HRED & 63.15 & 13.20 & 23.65 \\ REGS+COPT vs. REGS & 32.28 & 43.81 & 23.92 \\ StepGAN+COPT vs. HRED & 65.95 & 15.38 & 18.67 \\ StepGAN+COPT vs. StepGAN & 45.69 & 20.91 & 33.40 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Wins, losses, and ties (in %) of our approach against baselines based on the human evaluation.

 

the average reward. The percentage of counterfactual responses in the high reward interval (0.66, 1.00] is higher than that of standard responses. Meanwhile, counterfactual responses generated with COPT achieve a higher average than standard responses. The results demonstrate the effectiveness of the counterfactual response in exploring the high-reward area of the potential response space during the training process. Note that the distribution and the average between different epochs are not comparable due to the update of the discriminator as the training processes.

## 5 Conclusion

We propose a model-agnostic approach, COPT, that can be applied to any adversarial learning-based dialogue generation models. In contrast to existing approaches, it learns on counterfactual responses inferred from the structural causal model, taking advantage of observed responses. This helps the model to explore the high-reward area of the potential response space. Experiments show that the COPT significantly improves the quality of the generated responses, which demonstrates the effectiveness of this approach.

The authors would like to thank all the anonymous reviewers for their insightful comments. This paper is supported by the National Natural Science Foundation of China under Grant No. 62076081, No. 61772153, and No. 61936010. The author from UCSB is not supported by any of the projects above.

## References

* D. Bahdanau, K. Cho, and Y. Bengio (2015)Neural machine translation by jointly learning to align and translate. In Proceedings of the Third International Conference on Learning Representations, Cited by: SS4.1.
* S. R. Bowman, L. Vilnis, O. Vinyals, A. Dai, R. Jozefowicz, and S. Bengio (2016)Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pp. 10-21. External Links: Document Cited by: SS1, SS2.1.
* L. Buesing, T. Weber, Y. Zwols, N. Heess, S. Racaniere, M. Guez, and J. Lespiau (2019)Woulda, coula, shoulda: counterfactually-guided policy search. In Proceedings of the Seventh International Conference on Learning Representations, Cited by: SS1, SS2.1.
* M. Buhrmester, T. Kwang, and S. D. Gosling (2016)Amazon's mechanical turk: a new source of inexpensive, yet high-quality data?. External Links: Link Cited by: SS1, SS2.1.
* T. Fu, X. Wang, M. Peterson, S. Grafton, M. Eckstein, and W. Yang Wang (2020)Counterfactual vision-and-language navigation via adversarial path sampling.. In Proceedings of the 16th European Conference on Computer Vision, Cited by: SS1, SS2.1.
* I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio (2014)Generative adversarial nets. In Proceedings of the Twenty-eighth Conference on Neural Information Processing Systems, pp. 2672-2680. Cited by: SS1, SS2.1.
* P. Gupta, S. Mehri, T. Zhao, A. Pavel, M. Eskenazi, and J. Bigham (2019)Investigating evaluation of open-domain dialogue systems with human generated multiple references. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, Stockholm, Sweden, pp. 379-391. External Links: ISBN 978-3-540-3861-1, Link, Document Cited by: SS1, SS2.1.
* S. Hochreiter and J. Schmidhuber (1997)Long short-term memory. Journal of Neural computation9 (8), pp. 1735-1780. External Links: ISSN 0037-9092, Link, Document Cited by: SS1, SS2.1.
* Z. Ji, Z. Lu, and H. Li (2014)An information retrieval approach to short text conversation. arXiv preprint arXiv:1408.6988. Cited by: SS1, SS2.1.
* D. Kaushik, E. Hovy, and Z. C. Lipton (2020)Learning the difference that makes a difference with counterfactually-augmented data. In Proceedings of the Eighth International Conference on Learning Representations, Cited by: SS1, SS2.1.
* P. Ke, J. Guan, M. Huang, and X. Zhu (2018)Generating informative responses with controlled sentence function. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 1499-1508. External Links: ISBN 978-3-540-3861-1, Link, Document Cited by: SS1, SS2.1.
* G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. Rush (2017)OpenNMT: open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pp. 67-72. Cited by: SS1, SS2.1.
* A. Leuski, R. Patel, D. Traum, and B. Kennedy (2006)Building effective question answering characters. In Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue, pp. 18-27.

 

Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 110-119.
* Li et al. (2017a) Jiwei Li, Will Monroe, Tianlin Shi, Sebastien Jean, Alan Ritter, and Dan Jurafsky. 2017a. Adversarial learning for neural dialogue generation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2157-2169.
* Li et al. (2017b) Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017b. Dailydialog: A manually labelled multi-turn dialogue dataset. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 986-995.
* Liu et al. (2016) Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2122-2132, Austin, Texas. Association for Computational Linguistics.
* Luce (2012) R Duncan Luce. 2012. _Individual choice behavior: A theoretical analysis_. Courier Corporation.
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1412-1421.
* Maddison et al. (2014) Chris J Maddison, Daniel Tarlow, and Tom Minka. 2014. A* sampling. In _Advances in Neural Information Processing Systems_, pages 3086-3094.
* Mou et al. (2016) Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang, and Zhi Jin. 2016. Sequence to backward and forward sequences: A content-introducing approach to generative short-text conversation. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_, pages 3349-3358.
* Oberst and Sontag (2019) Michael Oberst and David Sontag. 2019. Counterfactual off-policy evaluation with gumbel-max structural causal models. In _International Conference on Machine Learning_, pages 4881-4890.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
* Park et al. (2018) Yookoon Park, Jaemin Cho, and Gunhee Kim. 2018. A hierarchical latent structure for variational conversation modeling. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1792-1801, New Orleans, Louisiana. Association for Computational Linguistics.
* Pearl and Mackenzie (2018) Judea Pearl and Dana Mackenzie. 2018. _The book of why: the new science of cause and effect_. Basic Books.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
* Qin et al. (2019) Linahui Qin, Antoine Bosseltu, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. 2019. Counterfactual story reasoning and generation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5043-5053, Hong Kong, China. Association for Computational Linguistics.
* Roese (1997) Neal J Roese. 1997. Counterfactual thinking. _Psychological bulletin_, 121(1):133.
* Serban et al. (2016) Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_.
* Shang et al. (2015a) Lifeng Shang, Zhengdong Lu, and Hang Li. 2015a. Neural responding machine for short-text conversation. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1577-1586, Beijing, China. Association for Computational Linguistics.
* Shang et al. (2015b) Lifeng Shang, Zhengdong Lu, and Hang Li. 2015b. Neural responding machine for short-text conversation. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1)_, pages 1577-1586.
* Song et al. (2019) Haoyu Song, Wei-Nan Zhang, Yiming Cui, Dong Wang, and Ting Liu. 2019. Exploiting persona information for diverse generation of conversational responses. In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence_.
* Song et al. (2018) Yiping Song, Rui Yan, Cheng-Te Li, Jian-Yun Nie, Ming Zhang, and Dongyan Zhao. 2018. An ensemble of retrieval-based and generation-based human-computer conversation systems. In _Proceedings of 

the 27th International Joint Conference on Artificial Intelligence and the 23rd European Conference on Artificial Intelligence_.
* Sordoni et al. (2015a) Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015a. A neural network approach to context-sensitive generation of conversational responses. In _Proceedings of the Association for Computational Linguistics: Human Language Technologies_, pages 196-205, Denver, Colorado. Association for Computational Linguistics.
* Sordoni et al. (2015b) Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015b. A neural network approach to context-sensitive generation of conversational responses. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 196-205.
* Su et al. (2016) Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina M. Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Online active reward learning for policy optimisation in spoken dialogue systems. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2431-2441, Berlin, Germany. Association for Computational Linguistics.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Proceedings of the Twenty-Eighth Conference on Neural Information Processing Systems_, pages 3104-3112.
* Tuan and Lee (2019) Yi-Lin Tuan and Hung-Yi Lee. 2019. Improving conditional sequence generative adversarial networks by stepwise evaluation. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_, 27(4):788-798.
* Vinyals and Le (2015) Oriol Vinyals and Quoc Le. 2015. A neural conversational model. _arXiv preprint arXiv:1506.05869_.
* Williams (1992) Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. _Journal of Machine Learning_, 3(8):229-256.
* Wright (1920) Sewall Wright. 1920. The relative importance of heredity and environment in determining the piebald pattern of guinea-pigs. _Proceedings of the National Academy of Sciences of the United States of America_, 6(6):320.
* Wu et al. (2019) Yu Wu, Furu Wei, Shaohan Huang, Zhoujun Li, and Ming Zhou. 2019. Response generation by context-aware prototype editing. In _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence_.
* Xing et al. (2017) Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2017. Topic aware neural response generation. In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_, volume 17, pages 3351-3357.
* Xu et al. (2018) Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun. 2018. Diversity-promoting GAN: A cross-entropy based generative adversarial network for diversified text generation. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3940-3949.
* Yan et al. (2016) Rui Yan, Yiping Song, and Hua Wu. 2016. Learning to respond with deep neural networks for retrieval-based human-computer conversation system. In _Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 55-64.
* Zhang et al. (2019) Wei-Nan Zhang, Qingfu Zhu, Yifa Wang, Yanyan Zhao, and Ting Liu. 2019. Neural personalized response generation as domain adaptation. _World Wide Web_, 22(4):1427-1446.
* Zhang et al. (2018) Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, and Bill Dolan. 2018. Generating informative and diverse conversational responses via adversarial information maximization. In _Proceedings of the Thirty-Second Conference on Neural Information Processing Systems_, pages 1810-1820.
* Zhao et al. (2017) Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 654-664, Vancouver, Canada. Association for Computational Linguistics.
* Zhou et al. (2018) Hao Zhou, Tom Young, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. 2018. Commonsense knowledge aware conversation generation with graph attention. In _the 27th International Joint Conference on Artificial Intelligence and the 23rd European Conference on Artificial Intelligence_, pages 4623-4629.
* Zhu et al. (2019a) Qingfu Zhu, Lei Cui, Wei-Nan Zhang, Furu Wei, and Ting Liu. 2019a. Retrieval-enhanced adversarial training for neural response generation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 3763-3773, Florence, Italy. Association for Computational Linguistics.
* Zhu et al. (2019b) Qingfu Zhu, Weinan Zhang, Lei Cui, and Ting Liu. 2019b. Order-sensitive keywords based response generation in open-domain conversational systems. _ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP)_, 19(2):1-18