Introduction
Humans in general find it relatively easy to have chat-like conversations that are both coherent and engaging at the same time. While not all human chat is engaging, it is arguably coherent BIBREF0 , and it can cover large vocabularies across a wide range of conversational topics. In addition, each contribution by a partner conversant may exhibit multiple sentences, such as greeting+question or acknowledgement+statement+question. The topics raised in a conversation may go back and forth without losing coherence. All of these phenomena represent big challenges for current data-driven chatbots.
We present a novel approach for chatbot training based on the reinforcement learning BIBREF1 , unsupervised learning BIBREF2 and deep learning BIBREF3 paradigms. In contrast to other learning approaches for Deep Reinforcement Learning chatbots that rely on partially labelled dialogue data BIBREF4 , BIBREF5 , our approach assumes only unlabelled data. Our learning scenario is as follows: given a dataset of human-human dialogues in raw text (without any manually provided labels), an ensemble of Deep Reinforcement Learning (DRL) agents take the role of one of the two partner conversants in order to learn to select human-like sentences when exposed to both human-like and non-human-like sentences. In our learning scenario the agent-environment interactions consist of agent-data interactions – there is no user simulator as in task-oriented dialogue systems BIBREF6 , BIBREF7 . During each verbal contribution and during training, the DRL agents
This process—illustrated in Figure FIGREF6 —is carried out iteratively until the end of a dialogue for as many dialogues as necessary, i.e. until there is no further improvement in the agents' performance. During each verbal contribution at test time, the agent exhibiting the highest predictive dialogue reward is selected for human-agent interactions.
This article makes the following contributions to neural-based chatbots:
In the next two sections, 2 and 3, we review related work on neural-based chatbots and provide related background on deep reinforcement learning. Then we describe our proposed approach and methodology in section 4. This is followed by a comprehensive set of automatic and human evaluations in section 5, which use (i) a dataset of chitchat conversations, and (ii) human ratings of human-chatbot dialogues. Section 6 draws conclusions and discusses avenues for future research.
Background
A reinforcement learning agent induces its behaviour from interacting with an environment through trial and error, where situations (representations of sentences in a dialogue history) are mapped to actions (follow-up sentences) by maximising a long-term reward signal. Such an agent is typically characterised by: (i) a finite set of states INLINEFORM0 that describe all possible situations in the environment; (ii) a finite set of actions INLINEFORM1 to change in the environment from one situation to another; (iii) a state transition function INLINEFORM2 that specifies the next state INLINEFORM3 for having taken action INLINEFORM4 in the current state INLINEFORM5 ; (iv) a reward function INLINEFORM6 that specifies a numerical value given to the agent for taking action INLINEFORM7 in state INLINEFORM8 and transitioning to state INLINEFORM9 ; and (v) a policy INLINEFORM10 that defines a mapping from states to actions BIBREF1 , BIBREF29 . The goal of a reinforcement learning agent is to find an optimal policy by maximising its cumulative discounted reward defined as DISPLAYFORM0
where function INLINEFORM0 represents the maximum sum of rewards INLINEFORM1 discounted by factor INLINEFORM2 at each time step. While a reinforcement learning agent takes actions with probability INLINEFORM3 during training, it selects the best action at test time according to DISPLAYFORM0
A deep reinforcement learning agent approximates INLINEFORM0 using a multi-layer neural network BIBREF30 . The INLINEFORM1 function is parameterised as INLINEFORM2 , where INLINEFORM3 are the parameters or weights of the neural network (recurrent neural network in our case). Estimating these weights requires a dataset of learning experiences INLINEFORM4 (also referred to as `experience replay memory'), where every experience is described as a tuple INLINEFORM5 . Inducing a INLINEFORM6 function consists in applying Q-learning updates over minibatches of experience INLINEFORM7 drawn uniformly at random from the full dataset INLINEFORM8 . This process is implemented in learning algorithms using Deep Q-Networks (DQN) such as those described in BIBREF30 , BIBREF31 , BIBREF32 , and the following section describes a DQN-based algorithm for human-chatbot interaction.
Proposed Approach
This section explains the main components of Figure FIGREF6 as follows. Motivated by BIBREF33 , we first describe the ensemble of Deep Reinforcement Learning (DRL) agents, we then explain how to conceive a finite set of dialogue actions from raw text, and finally we describe how to assign dialogue rewards for training DRL-based chatbots.
Ensemble of DRL Chatbots
We assume that all deep reinforcement learning agents in our ensemble use the same neural network architecture and learning algorithm. They only differ in the portion of data used for training and consequently the weights in their trained models—see BIBREF34 , BIBREF35 for alternative approaches. Our agents aim to maximise their cumulative reward over time according to DISPLAYFORM0
where INLINEFORM0 is the numerical reward given at time step INLINEFORM1 for choosing action INLINEFORM2 in state INLINEFORM3 , INLINEFORM4 is a discounting factor, and INLINEFORM5 is the optimal action-value function using weights INLINEFORM6 in the neural network of chatbot INLINEFORM7 . During training, a DRL agent will choose actions in a probabilistic manner in order to explore new INLINEFORM8 pairs for discovering better rewards or to exploit already learnt values—with a reduced level of exploration over time and an increased level of exploitation over time. During testing, our ensemble-based DRL chatbot will choose the best actions INLINEFORM9 according to DISPLAYFORM0
where INLINEFORM0 is a trajectory of state-action pairs of chatbot INLINEFORM1 , and INLINEFORM2 is a function that predicts the dialogue reward of chatbot INLINEFORM3 as in BIBREF36 . Given the set of trajectories for all agents—where each agent takes its own decisions and updates its environment states accordingly—the agent with the highest predictive reward is selected, i.e. the one with the least amount of errors in the interaction.
Our DRL agents implement the procedure above using a generalisation of DQN-based methods BIBREF30 , BIBREF31 , BIBREF32 —see Algorithm SECREF15 , explained as follows.
After initialising replay memory INLINEFORM0 with learning experience INLINEFORM1 , dialogue history INLINEFORM2 with sentences INLINEFORM3 , action-value function INLINEFORM4 and target action-value function INLINEFORM5 , we sample a training dialogue from our data of human-human conversations (lines 1-4).
Once a conversation starts, it is mapped to its corresponding sentence embedding representation, i.e. `sentence vectors' as described in Section SECREF26 (lines 5-6).
Then a set of candidate responses is generated including (1) the true human response and (2) a set of randomly chosen responses (distractors). The candidate responses are clustered as described in the next section and the resulting actions are taken into account by the agent for action selection (lines 8-10).
Once an action is chosen, it is conveyed to the environment, a reward is observed as described at the end of this section, and the agent's partner response is observed in order to update the dialogue history INLINEFORM0 (lines 11-14).
In response to the update above, the new sentence embedding representation is extracted from INLINEFORM0 for updating the replay memory INLINEFORM1 with experience INLINEFORM2 (lines 15-16).
Then a minibatch of experiences INLINEFORM0 is sampled from INLINEFORM1 for updating weights INLINEFORM2 according to the error derived from the difference between the target value INLINEFORM3 and the predicted value INLINEFORM4 (see lines 18 and 20), which is based on the following weight updates: DISPLAYFORM0
where INLINEFORM0 and INLINEFORM1 is a learning rate hyperparameter.
The target action-value function INLINEFORM0 and environment state INLINEFORM1 are updated accordingly (lines 21-22), and this iterative procedure continues until convergence.
ChatDQN Learning [1] Initialise Deep Q-Networks with replay memory INLINEFORM0 , dialogue history INLINEFORM1 , action-value function INLINEFORM2 with random weights INLINEFORM3 , and target action-value functions INLINEFORM4 with INLINEFORM5 Initialise clustering model from training dialogue data Sample a training dialogue (human-human sentences) Append first sentence to dialogue history INLINEFORM6 INLINEFORM7 sentence embedding representation of INLINEFORM8 Generate noisy candidate response sentences INLINEFORM9 INLINEFORM10 Execute chosen clustered action INLINEFORM11 Observe human-likeness dialogue reward INLINEFORM12 Observe environment response (agent's partner) Append agent and environment responses to INLINEFORM13 INLINEFORM14 sentence embedding representation of INLINEFORM15 Append learning experience INLINEFORM16 to INLINEFORM17 Sample random minibatch INLINEFORM18 from INLINEFORM19 INLINEFORM20 Set INLINEFORM21 Gradient descent step on INLINEFORM22 with respect to INLINEFORM23 Reset INLINEFORM24 every INLINEFORM25 steps INLINEFORM26 INLINEFORM27 end of dialogue Reset dialogue history INLINEFORM28 convergence
Sentence and Dialogue Clustering
Actions in reinforcement learning chatbots correspond to sentences, and their size is infinite assuming all possible combinations of word sequences in a given language. This is especially true in the case of open-ended conversations that make use of large vocabularies, as opposed to task-oriented conversations that make use of smaller (restricted) vocabularies. A clustered action is a group of sentences sharing a similar or related meaning via sentence vectors derived from word embeddings BIBREF37 , BIBREF38 . We represent sentences via their mean word vectors—similarly as in Deep Averaging Networks BIBREF39 —denoted as INLINEFORM0 , where INLINEFORM1 is the vector of coefficients of word INLINEFORM2 , INLINEFORM3 is the number of words in sentence INLINEFORM4 , and INLINEFORM5 is the embedding vector of sentence INLINEFORM6 . Similarly, a clustered dialogue is a group of conversations sharing a similar or related topic(s) via their clustered actions. We represent dialogues via their clustered actions. Dialogue clustering in this way can be seen as a two-stage approach, where sentences are clustered in the first step and dialogues are clustered in the second step. In our proposed approach, each DRL agent is trained on a cluster of dialogues.
While there are multiple ways of selecting features for clustering and also multiple clustering algorithms, the following requirements arise for chatbots: (1) unlabelled data due to human-human dialogues in raw text (this makes it difficult to evaluate the goodness of clustering features and algorithms), and (2) scalability to clustering a large set of data points (especially in the case of sentences, which are substantially different between them due to their open-ended nature).
Given a set of data points INLINEFORM0 and a similarity metric INLINEFORM1 , the task is to find a set of INLINEFORM2 groups with a clustering algorithm. In our case each data point INLINEFORM3 corresponds to a dialogue or a sentence. For scalability purposes, we use the K-Means++ algorithm BIBREF40 and the Euclidean distance INLINEFORM4 with INLINEFORM5 dimensions, and consider INLINEFORM6 as a hyperparameter – though other clustering algorithms and distance metrics can be used with our approach. In this way, a trained sentence clustering model assigns a cluster ID INLINEFORM7 to features INLINEFORM8 , where the number of actions (in a DRL agent) refers to the number of sentence clusters, i.e. INLINEFORM9 .
Human-Likeness Rewards
Specifying reward functions in reinforcement learning dialogue agents is often a difficult aspect. We propose to derive rewards from human-human dialogues by assigning positive values to contextualised responses seen in the data, and negative values to randomly chosen responses due to lacking coherence (also referred to as `non-human-like responses') – see example in Tables TABREF29 and TABREF30 . An episode or dialogue reward can thus be computed as INLINEFORM0 , where index INLINEFORM1 refers to the dialogue in focus, index INLINEFORM2 to the dialogue turn in focus, and INLINEFORM3 is given according to DISPLAYFORM0
Table TABREF29 shows an example of a well rewarded dialogue (without distortions) and Table TABREF30 shows an example of a poorly rewarded dialogue (with distortions). Other dialogues can exhibit similar dialogue rewards or something in between (ranging between INLINEFORM0 and INLINEFORM1 ), depending on the amount of distortions—the higher the amount of distortions the lower the dialogue reward.
We employ the algorithm described in BIBREF36 for generating dialogues with varying amounts of distortions (i.e. different degrees of human-likeness), which we use for training and testing reward prediction models using supervised regression. Given our extended dataset INLINEFORM0 with (noisy) dialogue histories INLINEFORM1 represented with sequences of sentence vectors, the goal is to predict dialogue scores INLINEFORM2 as accurately as possible.
Alternative and automatically derived values between -1 and +1 are also possible but considered as future work. Section SECREF67 provides an evaluation of our reward function and its correlation with human judgement. We show that albeit simple, our reward function is highly correlated with our judges' ratings.
Methodology
Our proposed approach can be summarised through the following methodology:
Collect or adopt a dataset of human-human dialogues (as in SECREF39 )
Design or adopt a suitable reward function (as in SECREF27 )
Train a neural regressor for predicting dialogue rewards (as in BIBREF36 )
Perform sentence and dialogue clustering in order to define the action set and training datasets (as in SECREF26 )
Train a Deep Reinforcement Learning agent per dialogue cluster (as described in SECREF15 )
Test the ensemble of agents together with the predictor of dialogue rewards (as in SECREF51 and SECREF67 ), and iterate from Step 1 if needed
Deploy your trained chatbot subject to satisfactory results in Step 6
Data
We used the Persona-Chat dataset, stats are shown in Table TABREF41 .
Experimental Setting
Our agents' states model dialogue histories as sequences of sentence vectors—using GloVe-based BIBREF38 mean word vectors BIBREF39 —with pre-trained embeddings. All our experiments use a 2-layer Gated Recurrent Unit (GRU) neural network BIBREF42 . At each time step INLINEFORM1 in the dialogue history, the first hidden layer generates a hidden state INLINEFORM2 as follows: DISPLAYFORM0
where INLINEFORM0 refers to a set of sentence vectors of the dialogue history, INLINEFORM1 is a reset gate that decides how much of the previous state to forget, INLINEFORM2 is an update gate that decides how much to update its activation, INLINEFORM3 is an internal state, INLINEFORM4 and INLINEFORM5 are the Sigmoid and hyperbolic Tangent functions (respectively), INLINEFORM6 and INLINEFORM7 are learnt weights, and INLINEFORM8 refers to the element-wise multiplication. If the equations above are summarised as INLINEFORM9 we get the following output action taking into account both hidden layers in our neural net: INLINEFORM10 , where INLINEFORM11 and INLINEFORM12 .
While a small number of sentence clusters may result in actions being assigned to potentially the same cluster, a larger number of sentence clusters would mitigate the problem, but the larger the number of clusters the larger the computational cost—i.e. more parameters in the neural net. Table TABREF45 shows example outputs of our sentence clustering using 100 clusters on our training data. A manual inspection showed that while clustered sentences sometimes do not seem very similar, they made a lot of sense and they produced reasonable outputs. Our human evaluation (see Section SECREF67 ) confirms this. All our experiments use INLINEFORM0 due to a reasonable compromise between system performance and computational expense.
The purpose of our second clustering model is to split our original training data into a group of data subsets, one subset for each ChatDQN agent in our ensemble. We explored different numbers of clusters (20, 50, 100) and noted that the larger the number of clusters the (substantially) higher the computational expense . We chose 100 clusters for our experiments due to higher average episode rewards of cluster-based agents than non-cluster-based ones. Figure FIGREF46 shows visualisations of our sentence and dialogue clustering using 100 clusters on our training data of 17.8K data points. A manual inspection was not as straightforward as analysing sentences due to the large variation of open-ended sets of sentences—see next section for further results.
Automatic Evaluation
We compared three DQN-based algorithms (DQN BIBREF30 , Double DQN BIBREF31 and Dueling DQN BIBREF32 ) in order to choose a baseline single agent and the learning algorithm for our ensemble of agents. The goal of each agent is to choose the human-generated sentences (actions) out of a set of candidate responses (20 available at each dialogue turn). Figure FIGREF52 (left) shows learning curves for these three learning algorithms, where we can observe that all agents indeed improve their performance (in terms of average episode reward) over time. It can also be observed that DQN and Double DQN performed similarly, and that Dueling DQN was outperformed by its counterpart algorithms. Due to its simplicity, we thus opted for using DQN as our main algorithm for the remainder of our experiments.
Figure FIGREF52 (right) shows the performance of 100 ChatDQN agents (one per dialogue cluster), where we also observe that all agents improve their performance over time. It can be noted however that the achieved average episode reward of INLINEFORM0 -1 is much greater than that of the single agent corresponding to INLINEFORM1 -5.5. Additional experiments reported that the lower the number of clusters the lower the average episode reward during training. We thus opted for using 100 dialogue clusters in the remainder of our experiments.
We analysed the performance of our agents further by using the test set of 999 totally unseen dialogues during training. We clustered the test set using our trained dialogue clustering model in order to assess the goodness of each agent in dialogues that were similar but not the same. The box plots in Figure FIGREF55 report the performance of our DRL agents according to the following metrics while tested on training data and test data: Avg. Episode Reward, Avg. F1 score, Avg. Recall@1, and Average Recall@5. One can quickly observe the striking performance gap between testing on training data vs. testing on test data. This can be interpreted as ChatDQN agents being able to learn well how to select actions on training data, but not being able to replicate the same behaviour on test data. This may not be surprising given that only 720 sentences (out of 263,862 training sentences and 15,586 test sentences) are shared between both sets, and it is presumably a realistic scenario seen that even humans rarely use the exact same sentences in multiple conversations. On the one hand our results also suggest that our training dataset is rather modest, and that a larger dataset is needed for improved performance. On the other hand, our results help us to raise the question `Can chitchat chatbots with reasonable performance be trained on modest datasets— i.e. with thousands of dialogues instead of millions?' If so, the generalisation abilities of chatbots need to be improved in future work. If not, large (or very large) datasets should receive more attention in future work on neural-based chatbots.
Finally, we compared the performance of 5 dialogue agents on 999 dialogues with 20 candidate sentences at every dialogue turn:
Upper Bound, which corresponds to the true human sentences in the test dataset;
Lower Bound, which selects a sentence randomly from other dialogues than the one in focus;
Ensemble, which selects a sentence using 100 agents trained on clustered dialogues as described in section SECREF4 – the agent in focus is chosen using a regressor as predictor of dialogue reward INLINEFORM0 using a similar neural net as the ChatDQN agents except for the final layer having one node and using Batch Normalisation BIBREF44 between hidden layers as in BIBREF36 ;
Single Agent, which selects a sentence using a single ChatDQN agent trained on the whole training set; and
Seq2Seq, which selects a sentence using a 2-layer LSTM recurrent neural net with attention – from the Parlai framework (http://www.parl.ai) BIBREF21 , trained using the same data as the agents above.
Table TABREF66 shows the results of our automatic evaluation, where the ensemble of ChatDQN agents performed substantially better than the single agent and Seq2Seq model.
Human Evaluation
In addition to our results above, we carried out a human evaluation using 15 human judges. Each judge was given a form of consent for participating in the study, and was asked to rate 500 dialogues (100 core dialogues—from the test dataset—with 5 different agent responses, dialogues presented in random order) according to the following metrics: Fluency (Is the dialogue naturally articulated as written by a human?), Engagingness (Is the dialogue interesting and pleasant to read?), and Consistency (without contradictions across sentences). This resulted in INLINEFORM0 ratings from all judges. Figure FIGREF70 shows an example dialogue with ratings ranging from 1=strongly disagree to 5=strongly agree.
Figure FIGREF71 shows average ratings (and corresponding error bars) per conversational agent and per metric. As expected, the Upper Bound agent achieved the best scores and the Lower Bound agent the lowest scores. The ranking of our agents in Table TABREF66 is in agreement with the human evaluation, where the Ensemble agent outperforms the Seq2Seq agent, and the latter outperforms Single Agent. The difference in performance between the Ensemble agent and the Seq2Seq agent is significant at INLINEFORM0 for the Fluency metric and at INLINEFORM1 for the other metrics (Engagingness and Consistency)—based on a two-tailed Wilcoxon Signed Rank Test.
Furthermore, we analysed the predictive power of dialogue rewards, derived from our reward function, against human ratings on test data. This analysis revealed positive high correlations between them as shown in Figure FIGREF72 . These scatter plots show data points of test dialogues (the X-axes include Gaussian noise drawn from INLINEFORM0 for better visualisation), which obtained Pearson correlation scores between 0.90 and 0.91 for all metrics (Fluency, Engagingness and Consistency). This is in favour of our proposed reward function and supports its application to training open-ended dialogue agents.
Conclusions and Future Work
We present a novel approach for training Deep Reinforcement Learning (DRL) chatbots. It uses an ensemble of 100 DRL agents based on clustered dialogues, clustered actions, and rewards derived from human-human dialogues without any manual annotations. The task of the agents is to learn to choose human-like actions (sentences) out of candidate responses including human generated and randomly chosen sentences. Our ensemble trains specialised agents with particular dialogue strategies according to their dialogue clusters. At test time, the agent with the highest predicted reward is used during a dialogue. Experimental results using chitchat dialogue data report that DRL agents learn human-like dialogue policies when tested on training data, but their generalisation ability in a test set of unseen dialogues (with mostly unseen sentences, only 4.62% seen sentences to be precise) remains a key challenge for future research in this field. As part of our study, we found the following:
Future work can investigate further the proposed learning approach for improved generalisation in test dialogues. Some research avenues are as follows.

Question: How do they obtain human generated policies?
Answer: derive rewards from human-human dialogues by assigning positive values to contextualised responses seen in the data, and negative values to randomly chosen responses due to lacking coherence

Question: How many agents do they ensemble over?
Answer: 100