Venue: NAACL 2021

# Clipping Loops for Sample-Efficient Dialogue Policy Optimisation

 Yen-Chen Wu

University of Cambridge

ycw30@cam.ac.uk &Carl Edward Rasmussen

University of Cambridge

cer54@cam.ac.uk

###### Abstract

Training dialogue agents requires a large number of interactions with users: agents have no idea about which responses are bad among a lengthy dialogue. In this paper, we propose loop-clipping policy optimisation (LCPO) to eliminate useless responses. LCPO consists of two stages: loop clipping and advantage clipping. In loop clipping, we clip off useless responses (called loops) from dialogue history (called trajectories). The clipped trajectories are more succinct than the original ones, and the estimation of state-value is more accurate. Second, in advantage clipping, we estimate and clip the advantages of useless responses and normal ones separately. The clipped advantage distinguishes useless actions from others and reduces the probabilities of useless actions efficiently. In experiments on Cambridge Restaurant Dialogue System, LCPO uses only \(260\) training dialogues to achieve \(80\%\) success rate, while PPO baseline requires \(2160\) dialogues. Besides, LCPO receives \(3.7/5\) scores in human evaluation where the agent interactively collects \(100\) real-user dialogues in the training phase.

+
Footnote †: _Corresponding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3420–3428

June 6-11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Based on dialogue policies, task-oriented dialogue systems decide when and how to give or request information from users. Learning dialogue policies is often formulated as a reinforcement learning (RL) problem since we usually receive feedback from users for the whole dialogue but not the correct answer for a single response Young et al. (2013); Levin et al. (1997). With high-capacity of function approximation, deep reinforcement learning has been widely applied to dialogue policy optimisation Su et al. (2016); Li et al. (2016); Casanueva et al. (2017). Typically, when applying deep reinforcement learning for dialogue policy management, more than thousands of dialogues are required to reach convergence Casanueva et al. (2017). However, requiring thousands of human dialogues during training is quite impractical for most academic or real-life scenarios. Users might lose patience and exhibit different behaviour during training. Therefore, in most prior work, the agents are trained via simulated users instead of real ones Liu et al. (2018); Gao et al. (2018).

Model-based reinforcement learning (MBRL) is commonly applied to make dialogue policy optimisation sample-efficient. MBRL approaches for dialogue management build a user model to predict users' behaviour Wu et al. (2020, 2020); Peng et al. (2018); Su et al. (2018); Wu et al. (2019); Zhang et al. (2019). Using the user model, DDQ Peng et al. (2018) generates pseudo-data. The accuracy of the user model strongly affects the quality of generated pseudo-data. If the behaviour of pseudo-data is far from real users' behaviour, dialogue policies learnt from these data might not be optimal Su et al. (2018). Manipulating when to use how much data in experience buffers becomes critical in these approaches.

Trainable-action-mask (TAM) Wu et al. (2020) blocks useless actions by learning action-masks from data to explore the action space more efficiently. Instead of predicting the users' behaviour directly, TAM predicts only the termination and similarity of future dialogue states to ease the training difficulties. However, the wrong predictions of the user model block the wrong actions, which makes the policy performance unstable. Moreover, the wrong output of policy does not learn from the predictions of the user model since it is blocked. Wrong values in policy networks make the performance unstable.

In this work, we propose loop-clipping policy optimisation (LCPO), which clips off useless actions in trajectories, computes advantages of actions in/out of the loop separately and optimises policy based on proximal policy optimisation (PPO) Schulman et al. (2017). First, LCPO is a

model-free and parameter-free algorithm. There is no additional effort of tuning hyperparameters of the user model. Also, it takes almost no extra running time during testing. Second, instead of brutally blocking actions like TAM does, LCPO directly reduces the probabilities of useless actions which makes optimisation smoother and easier. In our experiment on the Cambridge Restaurant Dialogue System, LCPO uses only \(260\) dialogues in the training phase to reach an \(80\%\) success rate, while the PPO baseline requires \(2160\). In the human-in-the-loop experiment, LCPO that trained with only \(100\) dialogue receives 3.7/5 scores and high remarks of conciseness and fluency. Overall, our main contributions are two-fold:

* We propose LCPO, a parameter-free, sample efficient algorithm to optimise dialogue policies. This algorithm is easy to implement and has barely any overhead.
* We demonstrate that training dialogue systems with real users is feasible within \(100\) dialogues on Cambridge Restaurant Dialogue System.

## 2 Preliminaries

This section goes through the notations in this paper. We start with formulating dialogue management as an RL problem in section 2.1. In section 2.2, we explain how to optimise the policy through proximal policy optimisation (PPO). In section 2.4, we explain what is episodic memory.

### Reinforcement learning for dialogue systems

When applying reinforcement learning for dialogue management Levin et al. (1997); Young et al. (2013); Williams (2008), a state \(s\), or a belief state, is the belief distribution over users' requests. An action \(a\) is the summarised action taken by a system. A reward \(r\) and a termination \(t\) are given by simulated users or real users. An episode \(E\) is a dialogue. The goal of reinforcement learning is to learn a policy \(\pi(a_{i}|s_{i})\) that maximises the cumulative reward \(R=\sum_{i=0}^{L}\gamma^{i}r_{i}\), where \(L\) is the length of the dialogue.

### Proximal Policy Optimisation (PPO)

Policy gradient is a fundamental optimisation algorithm with the loss function:

\[L^{PG}(\theta)=\hat{\mathbb{E}}_{i}[\log\pi_{\theta}(a_{i}|s_{i})\hat{A}_{i}], \tag{1}\]

where \(\hat{A}_{i}\) is the estimated advantage at timestep \(i\).

In order to ensure new policy is not changing far from the old one, trust region policy optimisation (TRPO) is set to surrogate the KL-divergence between the old and the current policies. In a similar but much simpler way, proximal policy optimisation (PPO) Schulman et al. (2017) clips probability ratios \(r_{i}\) to mitigate the excessive updates in TRPO.

\[r_{i}=\frac{\pi_{\theta}(a_{i}|s_{i})}{\pi_{\theta_{old}}(a_{i}|s_{i})}. \tag{2}\]

\[L^{PPO}(\theta)=\hat{\mathbb{E}}[\min(r_{i}(\theta)\hat{A}_{i},r_{i}^{clip} \hat{A}_{i})], \tag{3}\]

Figure 1: Illustration of LCPO. (a) Loop clipping. Clip off loops in trajectories to make them succinct. (b) Advantage clipping. Set a threshold such that the advantages of loops are always less than other useful actions. (c) Policy optimisation. We adopt proximal policy optimisation (PPO) in this paper.

 

where

\[r_{i}^{clip}=clip(r_{i}(\theta),1-\epsilon,1+\epsilon) \tag{4}\]

Advantage \(\hat{A}_{i}\) and state-value \(\hat{V}_{i}\) are estimated by generalised advantage estimation (GAE) as follow:

\[\hat{A}_{i}=\delta_{i}+\gamma\lambda\hat{A}_{i+1}, \tag{5}\]

\[\hat{V}_{i}=\hat{A}_{i}+V_{i}, \tag{6}\]

where \(\gamma\) decays future state-value, which represents our confidence in state-value estimation. \(\lambda\) decays the future TD-error, which represents a trade-off between bias and variance of advantage estimation. \(V_{i}\) is the predicted state-value of \(s_{i}\), and \(\delta_{i}\) is the TD-error:

\[\delta_{i}=r_{i}+\gamma V_{i+1}-V_{i}. \tag{7}\]

### Trainable-action-mask (TAM)

Trainable-action-mask (TAM) Wu et al. (2020b) is a model-based baseline that blocks useless actions directly. TAM learns a user model during dialogue interaction. The user model predicts the termination, reward, and the similarity between the current and the next dialogue state, and the action mask is constructed based on these features.

Though TAM is simple and effective, it is not stable enough. The first reason is a common pitfall of model-based approaches: the user model is hard to train and usually leads to inaccurate predictions that harm the dialogue policy. Second, the policy and state-value approximator (i.e. the policy network and value network in PPO) do not learn from the predictions of the user model. The wrong values estimated by these networks can not be updated efficiently since these actions are blocked.

### Episodic memory

In most policy gradient algorithms, the history of interactions is recorded in a memory buffer \(M\), which contains several episodes \(E\). An episode \(E\) consists of \(N\) transitions \(\{T_{0},T_{1},..T_{N}\}\). A transition \(T_{i}\equiv\{s_{i},a_{i},s_{i+1},r_{i},t_{i}\}\), where \(s_{i}\) is the current state. \(a_{i}\) is the action taken on \(s_{i}\), which leads to the next state \(s_{i+1}\) with a reward \(r_{i}\). If the episode terminates after taking action \(a_{i}\), \(t_{i}\) is \(True\) or otherwise \(False\).

## 3 Loop-clipping Policy Optimisation (LCPO)

In this paper, we propose loop-clipping policy optimisation (LCPO) to improve sample efficiency. As illustrated in Figure 1, LCPO consists of three components: loop clipping, advantage clipping, and policy optimisation. We adopt proximal policy optimisation (PPO) Schulman et al. (2017) for the policy optimisation part in this work.

Firstly, we give definitions to loops in section 3.1, and illustrate how to get clean trajectories via loop clipping in section 3.2. In section 3.3, we demonstrate how to estimate and clip advantages and state-values of loops for policy optimisation. Note that in the following subsections, we utilise two domain knowledge in dialogue systems.

* **Prior 1**: Information gain is non-negative since by asking more questions, we know better about user needs.
* **Prior 2**: The last action of a failed dialogue and actions that loop over the same state are unwanted.

### Definition of loop

In this paper, loop means transitions that consist of useless or unwanted actions. As illustrated in Figure 2, we define two kinds of the loop: \(N\)-hop loop and termination loop corresponding to our prior 2.

**Definition 1**.: _A \(N\)-hop loop \(L_{i}^{N}\) is a sequence of \(N\) transitions \(\{T_{i},T_{i+1},..T_{i+N-1}\}\) where \(s_{i}=s_{i+N}\)._

Since in a loop that the starting state \(s_{i}\) becomes the same as final state \(s_{N}\), \(\{a_{i},a_{i+1},..,a_{i+N-1}\}\) is a useless action sequence on state \(s_{i}\). In dialogue systems, \(N\)-hop loop might result from repetitively asking the same questions or giving the same information. Compared with the definition of useless

Figure 2: Illustration of two kinds of loops. In this paper, loop means transitions consist of useless or unwanted actions. (a) N-hop loop \(L^{N}\), where \(N=5\) in this example. (b) Termination loop \(L^{T}\).

 

actions in TAM (Wu et al., 2020b) which only considers the similarity of the next state (i.e \(1\)-hop loop), \(N\)-hop loop is a more general definition and is able to detect more useless actions.

**Definition 2**.: _A termination loop \(L^{T}_{i}\) is a transition \(T_{i}\) state \(s_{i}\) where \(t_{i}=True\) and \(r_{i}\leq 0\)._

In dialogue systems, \(a_{i}\) is a useless action on state \(s_{i}\) since the dialogue is terminated and failed. For example, termination loops might result from saying goodbye before completing tasks or making users out of patience. Note that the definition of loops utilises domain knowledge and might not be suitable for other applications.

### Loop clipping

As illustrated in Figure 3, the original trajectory might contain several identical states. We search for the identical states pair-wisely and detect loops by definitions in section 3.1. 1 The detected loops are clipped off from the original trajectory. After clipping, the trajectory becomes succinct so that reward signals can be assigned to useful actions effectively (Figure 3b).

Footnote 1: When the loop structure is complex. The solution of N-hop loop clipping is not unique.

In dialogue systems, the information for each state in a loop is the same since there is no information gain after taking useless actions. Therefore, a \(N\)-hop loop can be viewed as multiple one-hop loops as illustrated in Figure 3c.

### Loop advantage estimation (LAE)

After loop clipping, the original trajectory is split into a clean trajectory and several loops. Then We estimate the advantages of the clean trajectory and loops separately. For the clean trajectory, standard generalised advantage estimation (GAE) (Schulman et al., 2015) is applied as shown in Eq. 5, 6. If we only update the policy based on clean trajectories, the clipped useless actions will not be treated as training data. Therefore, these useless actions will not be penalised, resulting in unwanted lengthy dialogue. We first illustrate how to estimate state-values and advantage for loops, noted as loop advantage estimation (LAE) to distinguish from GAE. Second, we propose an advantage clipping trick, which makes the policy optimisation much more sample-efficient.

State-value estimationAccording to prior 1, in dialogue systems, information gain \(V_{i+1}-V_{i}\geq 0\).

In a loop \(L^{N}_{i}\) with length \(N\), since the

\[\hat{V}_{i}\leq\hat{V}_{i+1}\leq..\leq\hat{V}_{i+N} \tag{8}\]

and the same states share the same value i.e.

\[\hat{V}_{i}=\hat{V}_{i+N}, \tag{9}\]

all the state-values in \(L^{N}_{i}\) are the same:

\[\hat{V}_{i}=\hat{V}_{i+1}=..=\hat{V}_{i+N}. \tag{10}\]

Advantage estimationThe loop advantage for action \(a_{i}\) is:

\[\hat{A}^{LAE}_{i}=\delta_{i}+\gamma\lambda\hat{A}^{GAE}, \tag{11}\]

where \(\delta_{i}=r_{i}+\gamma V_{i+1}-V_{i}\). Note that \(\hat{A}^{GAE}\) is the next advantage \(\hat{A}_{i+N}\) after the loop \(L^{n}_{i}\). No matter how long the loop is, the loop advantage is computed from the transition after loop.

When state-values converge, \(V_{i+1}\simeq V_{i}\) in loop

Figure 3: Loop clipping. The shaded circles are states and the arrows are actions taken between states. (a) The original trajectory. (b) The clean trajectory after loop clipping. In this example, a 1-hop loop and a 2-hop loop are detected. (c) In dialogue systems, the states in a loop are the same since there is no information gain.

 

\(L_{i}\) by eq. 10. We can see that

\[V_{i+1}\simeq V_{i} \implies \delta_{i}\simeq r_{i}+(\gamma-1)V_{i}\doteq R_{i} \tag{12}\] \[\implies \hat{A}^{LAE}_{i}\simeq R_{i}+\gamma\lambda\hat{A}^{GAE}\] \[\implies \hat{A}^{LAE}_{i}\simeq\hat{A}^{LAE}_{i+1},\]

where \(R_{i}=r_{i}+(\gamma-1)V_{i}\). It is straightforward that when values converge, the advantage of loop is the advantage of best actions \(A^{GAE}_{i}\) with a one-turn penalty for all useless actions on state \(s_{i}\) (since the agent wastes one more turn on the same state). When \(\hat{A}^{GAE}\) converges to zero, \(\hat{A}^{LAE}\) converges to \(R_{i}\).

Advantage clippingHowever, we found that the advantage estimation is still not very accurate in the early stage of training process. The advantages of looping actions sometimes are higher than others and these actions are not penalised.

To properly penalise the looping actions, we clip the advantages in both LAE and GAE. The clipping threshold \(R_{i}=r+(\gamma-1)V_{i}\) since \(\hat{A}^{LAE}\) converges to this value.

\[\hat{A}^{ClipGAE}_{i}=\max(R_{i},\hat{A}^{GAE}_{i}), \tag{13}\]

\[\hat{A}^{CliplLAE}_{i}=\min(R_{i},\hat{A}^{LAE}_{i}), \tag{14}\]

where \(R_{i}=r_{i}+(\gamma-1)V_{i}\), so that

\[\hat{A}^{ClipGAE}_{i}\geq R_{i}\geq\hat{A}^{CliplLAE}_{i}. \tag{15}\]

This trick distinguishes bad responses from good ones explicitly and makes policy converge faster.

The instruction of LCPO implementation is in Algorithm 1.

```
1Collect \(N\) transitions into Memory \(M\)for Episode E in \(M\)do// Loopclipping
2for\(T_{i}\)in \(E\)doif i<ptrthencontinue
3for\(T_{i}\)in \(E\)doif\(s_{i}==s_{j}\)then ptr = j
4if i<ptrthen\(T_{i}\in\mathbb{L}\) // Advantagestimationin reversedorder
5forTransition\(T_{i}\)inreversed(\(E\))do if\(T_{i}\in\mathbb{L}\)then Estimate \(\hat{A},\hat{V}\) via clipped LAE (Eq. 14, 10) else Estimate \(\hat{A},\hat{V}\) via clipped GAE (Eq. 13, 6)
7 Optimise the policy \(\pi\) via PPO (Eq. 3), with \(K\) epochs and mini-batch size \(B\)
```

**Algorithm 1**LCPO Algorithm

## 4 Experiments

Experiments are conducted on the Cambridge restaurant dialogue system using the PyDial toolkit Ultes et al. (2017). We evaluate the agents on both a simulated user and real users. From section 4.1 to 4.5, we illustrate the experiments with a simulated user. For human-in-the-loop experiment, see section 4.6.

### Settings

User simulatorWe use a goal-driven simulated user on the semantic level Schatzmann et al. (2007); Schatzmann and Young (2009). The maximum dialogue length is set to \(25\) turns and \(\gamma=0.99\). The reward is defined as \(20\) for a successful dialogue minus the number of turns in the dialogue. \(15\%\) semantic error rate (SER) is included in the user simulator to accommodate for automatic speech recognition (ASR) error.

Policy optimisationProximal policy optimisation (PPO) is applied. The state and action dimension of policy and value networks are \(268\) and \(16\). Dimensions of two hidden layers are \(130\) and \(50\). The agent collects a \(N=100\) transitions to update the policy \(\pi\) with \(K=10\) epochs and mini-batch size \(B=16\). After an update, the memory is flushed and becomes empty again. Optimiser is ADAM Kingma and Ba (2014) with a learning rate \(0.001\). Entropy coefficient is \(0.01\) and standardised advantages is applied. During testing, actions are sampled from the output distributions of the policy network.

Loops DetectionIn theory, the starting state and the ending state are identical in a loop. Yet, due to numerical uncertainty, we use cosine similarity with threshold \(\eta=0.99\) to justify whether two states are the same. Under this strict setting, two states are considered different if they have any dif 

ferent slot-values. In practice the optimal threshold depends on the noise level of state observation.

EvaluationIn the experiment with the simulated user, we evaluate each agent with \(500\) dialogues after every \(100\) training dialogues. The mean and standard deviation of performance is computed over \(10\) runs with different neural networks initialisation. The mean \(\pm\) standard deviation is depicted as the shaded area.

The x-axes of figures are in log-scale to emphasise both the early stage and the final performance of the training process.

### Baseline Comparisons

In figure 4, we compare the performance of PPO Schulman et al. (2017), TAM Wu et al. (2020b), and LCPO. The left part of the figure shows the learning curves of the success rate. We can see that LCPO is considerably stable and sample-efficient. Worth to note that LCPO has the best final performance. TAM learns slower, and PPO requires a large number of training dialogues.

The right part of the figure shows the average turns taken by the agent. The lower, the better. We can see that LCPO takes more turns in the beginning but becomes more concise than the baselines later.

In table 1, we can see the detail of performance at \(200\) and \(2000\) training dialogues respectively. In low resource scenario, where the dialogue policy is trained by \(200\) dialogues, LCPO outperforms other baselines with small variance. Yet the average number of loops in each dialogue is higher. That is because LCPO takes more turn than other agents. Other agents often give poor responses so that the users leave the dialogue out of patience with fewer turns.

Regarding final performance at \(2000\) dialogues, all of the agents perform similarly. We can note that LCPO takes the least number of turns since its algorithm prevents from doing useless actions. LCPO requires only 260 dialogues to reach \(80\%\) success rate while PPO takes \(2160\). In addition, LCPO is light-packed and does not consume a lot of additional training time like TAM.

Figure 4: Learning curves of different algorithms. _Left:_ Success rate. _Right:_ Average number of turns. The results are evaluated by \(10\) runs. The lines are averages and the shades represent standard deviations.

\begin{table}
\begin{tabular}{c c c c c c c c}  & Low resource experiment@200D & \multicolumn{3}{c}{Final performance @2000D} \\ _Algorithm_ & Suc. & Turns & \#Loops & \#Dialog & Suc. & Turns & \#Loops & Time \\ \hline \hline PPO & \(37.8\pm 17.2\%\) & **7.4**\(\pm 2.3\) & **3.5** & 2160 & \(95.16\pm 1.1\%\) & \(7.8\pm 0.4\) & 2.3 & **58 min** \\ \hline TAM & \(36.2\pm 6.8\%\) & \(8.6\pm 0.9\) & 4.1 & 1140 & \(93.58\pm 1.5\%\) & \(7.4\pm 0.5\) & 2.5 & 125 min \\ \hline LCPO & \(\textbf{76.0}\pm 2.9\%\) & \(11.5\pm 0.8\) & 4.7 & **260** & \(\textbf{95.7}\pm 1.1\%\) & \(\textbf{6.3}\pm 0.3\) & **0.9** & 68 min \\ \end{tabular}
\end{table}
Table 1: Baseline comparison. The highest performance in each column is highlighted. The success rate, number of turns, and number of loops are reported for \(200\) and \(2000\) training dialogues. The number of training dialogue required to reach \(80\%\) success rate and the training time usage are also listed in the table. #DIalogue means the average required number of dialogues to reach 80% success rate. Time means the average training time for \(2000\) training dialogues.

 



### Ablation study: termination loop

In the left part of figure 5, the red and brown lines are LCPO with and without clipping termination loop \(L^{T}\) respectively. We can see that without clipping \(L^{T}\), the learning curves become less stable and inhibit the cold-start problem at the beginning of training.

In a failed dialogue, some actions are good and should not be penalised for the failure of conversation. Therefore, we should clip off the last transition in failed dialogue, so that the rest transitions in the clean trajectories (not in loops) are not penalised for the failure. For example, if we clip off the last the action "bye" in a failed dialogue, only 'bye' is strongly penalised while other normal interactions are not.

### Ablation study: advantage clipping

We propose 4 agents for comparisons: 1) clip both GAE and LAE, 2) clip LAE, 3) clip GAE, 4) no advantage clipping. The success rates after training with \(100\), \(200\), \(2000\) dialogues are reported in Table 2.

In the low-resource scenario (less than 200 dialogues), clipping both GAE and LAE outperforms other methods considerably. And LCPO with no advantage clipping is the worst. Without clipping, inaccurate advantage estimation in the early stage of the training process cannot reduce the probabilities of useless actions efficiently.

Regarding the final performance after training agents with 2000 dialogues, all of the methods perform similarly. Yet, if we only clip the GAE, the final performance is slightly worse than others. That is because not all the actions in clean trajectories are useful. The 'clean' trajectories still contain several useless actions though not detected. Assigning larger advantages to all actions in clean trajectories makes performance unstable.

### Robustness to hyperparameters

In the right part of figure 5, policy update interval is set to \(50\) and \(100\) for PPO and LCPO. The red and brown lines are LCPO and the green and blue lines are PPO with different update intervals. We can see that the performance of PPO is strongly affected by the update interval. In contrast, LCPO still shows high stability and sample efficiency. Its robustness to hyperparameters makes tuning LCPO effortless.

### Human-in-the-loop Evaluation

General SettingsThe dialogue system uses a rule-based belief tracker, and an NLG model Wen et al. (2015). In each dialogue, one of the agents is randomly picked to talk with a user. The users have to interact with the agent according to a given instruction on the user goal sampled from the corpus. The users can decide to leave the dialogue session if they are out of patience.

Figure 5: _Left: Ablation study of termination loop. Right: Robustness to hyper-parameters._

Training SettingsWe experiment on two training algorithms: PPO and LCPO. The hyper-parameters of PPO and LCPO are the same as the simulated user experiment. A human user interacts with each agent for \(100\) dialogues. At the end of each dialogue, the user gives \(20\) scores to the agent for a successful dialogue and gives 0 scores for a failed one. A penalty of \(-1\) is also applied in each turn.

A successful dialogue means the restaurant given by the agent must fulfil all the constraints and the requested information like phone number or address must be provided. In other words, the agents only receive feedback on the aspect of task completion.

Evaluation SettingsEach human user interacts with each agent for \(5\) dialogue and gives his/her feedback on four aspects:

* **Task completion**: The agent finds a restaurant that meets the constrains. The requested information is also given.
* **Conciseness**: The agent is to the point and does not ask/provide the same information repetitively.
* **Fluency**: The agent does not interrupt the dialogue flow and answer the questions logically.
* **Overall score**: The overall score for chatting with this agent.

Each agent is evaluated on \(100\) dialogues, the mean and variance of each score are reported in Table 3. The scores are range from \(0\) to \(5\). We also evaluate the agents by a simulated user via \(500\) dialogues for each agent.

ResultsIn table 3, we can see that LCPO significantly outperform PPO in all aspects. The task completion is close to the success rate evaluated by the simulated user. Conciseness is the feature of this work, and the improvement is also the most considerable. Regarding fluency, the difference between PPO and LCPO is smaller. Sometimes a fluent conversation takes more turns. Sometimes a non-logical response can complete the task as well (e.g. inform a restaurant name in the beginning). However, LCPO is still better than PPO in terms of fluency since a non-logical response usually accompanies with no information gain.

## 5 Conclusion

Our contributions are:

* We propose LCPO to improve sample efficiency for dialogue policy optimisation. LCPO has two critical components: loop clipping and advantage clipping. Both of them are strongly effective in low resource scenario and easy to implement. LCPO also demonstrates strong robustness to hyperparameters.
* We train and evaluate dialogue agents with real users on the Cambridge Restaurant domain. We also demonstrate that human-in-the-loop training is feasible within \(100\) dialogues. The evaluation has four aspects to clarify what has been learnt by each agent. LCPO outperforms PPO in all aspects.

In this paper, LCPO integrates with PPO. In the future, we will generalise loop clipping method to other off-policy reinforcement learning approaches with episodic memory since off-policy approaches are considered more sample-efficient.

## References

* Casanueva et al. [2017] Inigo Casanueva, Pawel Budzianowski, Pei-Hao Su, Nikola Mrksic, Tsung-Hsien Wen, Stefan Ultes, Lina Rojas-Barahona, Steve Young, and Milica Gasic. 2017. A benchmarking environment for reinforcement learning based task oriented dialogue management. _arXiv preprint arXiv:1711.11023_.
* Gao et al. [2018] Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neural approaches to conversational ai. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, pages 1371-1374.

\begin{table}
\begin{tabular}{l c c}  & PPO & LCPO \\ \hline \hline Task Completion & \(2.0\pm 1.7\) & **3.2**\(\pm 1.5\) \\ Conciseness & \(1.8\pm 0.8\) & **3.9**\(\pm 1.1\) \\ Fluency & \(2.6\pm 0.5\) & **3.6**\(\pm 0.9\) \\ Overall score & \(2.1\pm 0.4\) & **3.7**\(\pm 0.9\) \\ \hline Success rate (SimUser) & \(41.7\%\) & **66.8**\(\%\) \\ \end{tabular}
\end{table}
Table 3: Human-in-the-loop experiment. Human users evaluate each agent in four aspects. Each agent is trained by interacting with a human for \(100\) dialogues. The highest success rate in each row is highlighted. The last row is the success rates over 500 dialogues evaluated by a simulated user.

 

* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Levin et al. (1997) Esther Levin, Roberto Pieraccini, and Wieland Eckert. 1997. Learning dialogue strategies within the markov decision process framework. In _1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings_, pages 72-79. IEEE.
* Li et al. (2016) Jiwei Li, Alexander H Miller, Sumit Chopra, Marc'Aurelio Ranzato, and Jason Weston. 2016. Dialogue learning with human-in-the-loop. _arXiv preprint arXiv:1611.09823_.
* Liu et al. (2018) Bing Liu, Gokhan Tur, Dilek Hakkani-Tur, Pararth Shah, and Larry Heck. 2018. Dialogue learning with human teaching and feedback in end-to-end trainable task-oriented dialogue systems. _arXiv preprint arXiv:1804.06512_.
* Peng et al. (2018) Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Kam-Fai Wong. 2018. Integrating planning for task-completion dialogue policy learning. _arXiv preprint arXiv:1801.06176_.
* Schatzmann et al. (2007) Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-based user simulation for bootstrapping a pomdp dialogue system. In _Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers_, pages 149-152. Association for Computational Linguistics.
* Schatzmann and Young (2009) Jost Schatzmann and Steve Young. 2009. The hidden agenda user simulation model. _IEEE transactions on audio, speech, and language processing_, 17(4):733-747.
* Schulman et al. (2015) John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015. High-dimensional continuous control using generalized advantage estimation. _arXiv preprint arXiv:1506.02438_.
* Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _arXiv preprint arXiv:1707.06347_.
* Su et al. (2016) Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. On-line active reward learning for policy optimisation in spoken dialogue systems. _arXiv preprint arXiv:1605.07669_.
* Su et al. (2018) Shang-Yu Su, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Yun-Nung Chen. 2018. Discriminative deep dyna-q: Robust planning for dialogue policy learning. _arXiv preprint arXiv:1808.09442_.
* Ultes et al. (2017) Stefan Ultes, Lina M Rojas Barahona, Pei-Hao Su, David Vandyke, Dongho Kim, Inigo Casanueva, Pawel Budzianowski, Nikola Mrksic, Tsung-Hsien Wen, Milica Gasic, et al. 2017. Pydial: A multi-domain statistical dialogue system toolkit. _Proceedings of ACL 2017, System Demonstrations_, pages 73-78.
* Wen et al. (2015) Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. _arXiv preprint arXiv:1508.01745_.
* Williams (2008) Jason D Williams. 2008. The best of both worlds: Unifying conventional dialog systems and pomdps. In _Ninth Annual Conference of the International Speech Communication Association_.
* Wu et al. (2020a) Yen-chen Wu, Bo-Hsiang Tseng, and Milica Gasic. 2020a. Actor-double-critic: Incorporating model-based critic for task-oriented dialogue systems. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_, pages 854-863.
* Wu et al. (2020b) Yen-Chen Wu, Bo-Hsiang Tseng, and Carl Edward Rasmussen. 2020b. Improving sample-efficiency in reinforcement learning for dialogue systems by using trainable-action-mask. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 8024-8028. IEEE.
* Wu et al. (2019) Yuexin Wu, Xiujun Li, Jingjing Liu, Jianfeng Gao, and Yiming Yang. 2019. Switch-based active deep dyna- q: Efficient adaptive planning for task-completion dialogue policy learning. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 7289-7296.
* Young et al. (2013) Steve Young, Milica Gasic, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. _Proceedings of the IEEE_, 101(5):1160-1179.
* Zhang et al. (2019) Zhirui Zhang, Xiujun Li, Jianfeng Gao, and Enhong Chen. 2019. Budgeted policy learning for task-oriented dialogue systems. _arXiv preprint arXiv:1906.00499_.