SlowGuess/NAACL_2022_OCR huggingface dataset:
Chang Tian† and Wenpeng Yin* and Marie-Francine Moens†
$\dagger$ Department of Computer Science, KU Leuven
*LanguageX Lab, Temple University
chang.tian@kuleuven.be
A dialogue policy module is an essential part of task-completion dialogue systems. Recently, increasing interest has focused on reinforcement learning (RL)-based dialogue policy. Its favorable performance and wise action decisions rely on an accurate estimation of action values. The overestimation problem is a widely known issue of RL since its estimate of the maximum action value is larger than the ground truth, which results in an unstable learning process and suboptimal policy. This problem is detrimental to RL-based dialogue policy learning. To mitigate this problem, this paper proposes a dynamic partial average estimator (DPAV) of the ground truth maximum action value. DPAV calculates the partial average between the predicted maximum action value and minimum action value, where the weights are dynamically adaptive and problem-dependent. We incorporate DPAV into a deep Q-network as the dialogue policy and. Our method can achieve better or comparable results compared to top baselines on three dialogue datasets of different domains with a lower computational load. In addition, we also theoretically prove the convergence and derive the upper and lower bounds of the bias compared with those of other methods.
Task-completion dialogue systems are commonly implemented in two schemes. One is by end-to-end training, such as (Zhang et al., 2020a). The other is a pipeline framework (Chen et al., 2017), which typically consists of four modules that are independently trained, as shown in Figure 1a: natural language understanding (NLU), dialogue state tracker (DST), dialogue policy learning (DPL) and natural language generation (NLG). For this pipeline-style dialogue system, the conversation text from a user is first fed to the NLU module, where the user utterance is parsed into semantic slots for DST. DST manages the inputs of each dialogue turn together
with the dialogue history. Then DST outputs the current dialogue state embedding to the DPL module, where a dialogue action is taken based on current dialogue state and knowledge base data. The NLG module maps the selected dialogue action into natural language to converse with the user.
Reinforcement learning (RL) algorithms, specifically Q-learning (Watkins and Dayan, 1992) based algorithms, have become a mainstream method for training the dialogue policy module (Peng et al., 2018; Zhang et al., 2020b). For each step, the policy agent updates its action value  ${}^{1}$  estimate as the sum of the observed reward and the estimated maximal action value in the next state. However, this update rule suffers from an overestimation problem (Hasselt, 2010): mostly the estimated maximal action value is larger than the ground truth. The overestimation problem causes that the dialogue policy module has inaccurate action values estimations after the training, which misleads the dialogue policy to choose the wrong dialogue action (see the wrong dialogue action in Figure 2). Some prior studies have tried to address this problem in domains like video game playing and multi-agent systems, but they either suffered from the underestimation problem (Hasselt, 2010; Lan et al., 2020) or required heavy computational load, such as those ensemble methods (Anschel et al., 2017; Lan et al., 2020; Lee et al., 2021).
In this work, we propose dynamic partial average (DPAV), a novel approach to mitigate the overestimation problem specifically for the task-completion dialogue policy. DPAV utilizes the partial average between the predicted maximal action value and the predicted minimal action value to estimate the ground truth maximum action value, where the weights are dynamically adaptive and problem-dependent. The rationale here is that
This value is the expected return for taking the action under a certain state, and it is represented as the Q value of Q-learning.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 565 - 577 July 10-15, 2022 ©2022 Association for Computational Linguistics
(a) Four modules in the pipeline framework
(b) The overestimation problem in RL-based dialogue policy
Figure 2: Wrong and correct dialogue actions
Figure 1: Task-completion dialogue system
DPAV learns the optimal trade-off between the predicted maximal action value and the predicted minimal action value so that the dialogue policy learning procedure will be more reasonable and stable. Our system not only yields a better dialogue process (see Figure 2), but also has much lower computational cost compared to ensemble models.
Overall, our main contributions are as follows: (i) This is the first work to investigate and handle the overestimation problem of the reinforcement learning framework for task-completion dialogue systems. (ii) We propose a novel and effective approach, the dynamic partial average DPAV, which can alleviate the overestimation problem with lower computational load. (iii) We theoretically prove the convergence and derive the upper and lower bounds of our method to claim its effectiveness.
Dialogue Policy. The dialogue policy module makes a dialogue decision given the current state (Zhang et al., 2019). Early methods are rule-based (Chen et al., 2017). Since handcrafted rules are non-extensible and resource-consuming (Zhao et al., 2021), deep reinforcement learning (DRL) has become a mainstream method for training dialogue policies (Wu et al., 2019; Wang et al., 2020; Zhao et al., 2021). Task-completion dialogue policy learning is often regarded as an RL problem (Zhang et al., 2021).
Overestimation Bias. The value-based algorithm Q-learning, a common unit of the dialogue policy module, suffers the overestimation bias (Thrun and Schwartz, 1993; Hasselt, 2010). Prior studies addressed the problem in multiple ways, including (1) bias compensation with additive pseudo costs and (2) a variety of estimators. Bias-corrected Q-Learning (Lee et al., 2013) subtracts a quantity from the target but this method cannot address the bias from the function approximation (Pentaliotis and Wiering, 2021). It is known that the bias compensation method is labor involved and time consuming (Anwar and Patnaik, 2008; Lee and Powell, 2012). Double Q-learning (Hasselt, 2010) trades overestimation bias for underestimation bias using the double estimator. Since underestimation bias is not preferable (Hasselt, 2010; Lan et al., 2020), Weighted Q-learning proposes (D'Eramo et al., 2016; Zhang et al., 2017) the weighted estimator for the maximal action value based on a weighted average of estimated actions values. However, the weights computation is only practical in a tabular setting (D'Eramo et al., 2017). Our work differs from the foregoing in that it proposes a new estimator which could be generalized into the deep Q-learning network setting.
Overestimation bias is more problematic in the deep Q-learning network (DQN) algorithm (Fan et al., 2020) due to the function approximation errors of DRL. Polishing estimation tricks of a single model and using ensemble models are two mainstream solutions. Double Q-learning is subsequently adapted to a neural network as Double DQN (Van Hasselt et al., 2016), and Duel DQN proposes a new action value estimation scheme (Wang et al., 2016). But the two methods still suffer the bias of double estimator and maximum estimator, respectively. Another approach against overestimation bias is based on the idea of ensembling. Averaged DQN controls the estimation bias by tak
ing the average over action values of multiple target networks (Anschel et al., 2017). Later, (Lan et al., 2020) claims that an average operation will never completely remove the overestimation bias, and they propose the Maxmin DQN which takes a minimum from multiple maximums of different ensemble units to estimate the maximum action value in a selective process. Then, (Kuznetsov et al., 2020) recognizes that Maxmin DQN also suffers underestimation bias and that the bias control is coarse. Recently, the SUNRISE method uses the uncertainty estimates of the ensemble. But it only down-weights the biased estimation (Lee et al., 2021). In this work, the model only uses a value function instead of a combination of multiple value functions and tailors the predicted maximum and minimum of a value function to approximate the optimal action value. Our work does not move towards underestimation and avoid the computational complexity of ensemble models.
Even though an unbiased estimator does not exist (D'Eramo et al., 2016), the maximum estimator (Watkins and Dayan, 1992) and double estimator (Hasselt, 2010; Van Hasselt et al., 2016) are the most representative among the relevant works.
Maximum estimator (ME). This method is used by deep Q-learning to approximate the ground truth maximum action value of the following state by maximizing over a set of action values  $Q(s_{t + 1},\cdot)$  It represents the target  $y^{DQN}$  for taking a possible action  $a$  under the state  $s_{t + 1}$  as:
$$
y ^ {D Q N} = r _ {t + 1} + \gamma \max  _ {a} Q (s _ {t + 1}, a; \boldsymbol {\theta} ^ {-}) \tag {1}
$$
where  $r_{t + 1}$  is the reward,  $\gamma$  is the discount value for future rewards and  $\theta^{-}$  is parameters of the target network. As Smith and Winkler (2006) found, the estimate of ME is larger than the ground truth (i.e., the estimated maximum value of the following state,  $\max_{a}Q\left(s_{t + 1},a;\theta^{-}\right)$  is overestimated), which results in the biased loss:
$$
\begin{array}{l} L (\boldsymbol {\theta}) = \\ \underset {\left\langle s _ {t}, a _ {t}, r _ {t}, s _ {t + 1} \right\rangle \sim m} {\mathbb {E}} \left[ \left(y ^ {D Q N} - Q \left(s _ {t}, a _ {t}; \boldsymbol {\theta}\right)\right) ^ {2} \right], \tag {2} \\ \end{array}
$$
where  $m$  is the RL experience replay pool and  $\pmb{\theta}$  is parameters of the DQN model. Thus, the  $Q(s_{t},\cdot)$  will not be perfectly accurate after training,
Double estimator (DE). This method (Hasselt, 2010; Van Hasselt et al., 2016) is used by deep Q-learning to solve the overestimation problem of ME in DQN. The Double DQN has two estimators, and one estimator decides the action index while the other estimator evaluates the action value of the selected action. Then Double DQN (DDQN) uses the evaluated action value to estimate the ground truth maximum action value of state  $s_{t+1}$ :
$$
\begin{array}{l} y ^ {D D Q N} = \\ r _ {t + 1} + \gamma Q \left(s _ {t + 1}, \underset {a} {\operatorname {a r g m a x}} Q \left(s _ {t + 1}, a; \boldsymbol {\theta} ^ {+}\right); \boldsymbol {\theta} ^ {-}\right). \tag {3} \\ \end{array}
$$
However, DE suffers from the underestimation problem and does not guarantee better estimation than ME (Lan et al., 2020).
3.2 Problem in Dialogue Policy
Q-learning is a common unit of RL-based dialogue policies. The overestimation bias of ME propagates into model action values  $Q(s_{t},\cdot)$ . In dialogue  $Q(s_{t},\cdot)$  represent the dialogue action values, which are the expected returns the dialogue system will receive after taking an action under the state  $s_{t}$ . Since  $Q(s_{t},\cdot)$  are biased, the dialogue policy cannot issue accurate actions accordingly. This hurts dialogue performances.
Example. We use a dialogue turn to show the negative effects of the overestimation bias. In Figure 1b, the dialog state tracker module outputs state embedding, dialogue policy processes state embedding and predicts the wrong dialogue action B instead of the correct action A based on the biased action values.
4.1 Dynamic Partial Average
Q-learning suffers from overestimation bias because of the ME (Hasselt, 2010). To reduce the bias, in this work, we propose the dynamic partial average (DPAV) estimator. DPAV utilizes the partial average between the predicted maximal action value and the minimal action value to estimate the ground truth maximal action value  $Q_{*}(s_{t + 1})$  of the target of Q-learning update. The mathematical formula of the DPAV estimator of Q-learning is as follows:
$$
\begin{array}{l} Q _ {*} (s _ {t + 1}) \approx (1 - \lambda_ {t}) * \max  _ {a ^ {\prime}} Q \left(s _ {t + 1}, a ^ {\prime}; \theta^ {\prime}\right) \\ + \lambda_ {t} * \min  _ {a ^ {\prime \prime}} Q \left(s _ {t + 1}, a ^ {\prime \prime}; \theta^ {\prime}\right) = Q _ {D P A V} \left(s _ {t + 1}\right), \tag {4} \\ \end{array}
$$
$\lambda_{t}$  is a float number between [0,1] that is dynamic in time and problem-dependent such that the DPAV can take the average between the maximum and minimum of the action values. The weights assigned to the maximum and minimum are not the same, so it is a partial average.
The DPAV estimator is deployed in Q-learning as DPAV Q-learning, so that we have the action value function Q update formula as:
$$
\begin{array}{l} Q \left(s _ {t}, a _ {t}\right) \leftarrow \left(1 - \alpha_ {t}\right) Q \left(s _ {t}, a _ {t}\right) + \alpha_ {t} y ^ {D P A V}, \\ y ^ {D P A V} = r _ {t + 1} + \gamma * Q _ {D P A V} \left(s _ {t + 1}\right). \tag {5} \\ \end{array}
$$
where  $\gamma$  is the discount factor for the future action value and  $\alpha_{t}$  is the step size.  $\lambda_{t}$  in  $Q_{DPAV}(s_{t + 1})$  decays according to a predefined rate as training progresses, the decay formula is as follows:
$$
\lambda_ {t + 1} = \lambda_ {t} * d, \tag {6}
$$
where  $d$  is the decay rate that is set to the fixed value in training. So  $1 - \lambda_{t}$  will give more weight to the maximal action value during the training.
To apply DPAV to the complex dialogue policy learning setting, this paper combines it with the deep Q-learning network (DQN) and proposes DPAV DQN. Its loss function is adapted from Equation 2 to the formula:
$$
\begin{array}{l} L _ {\theta} = \\ \underset {\left\langle s _ {t}, a _ {t}, r _ {t + 1}, s _ {t + 1} \right\rangle \sim m} {\mathbb {E}} \left[ \left(y ^ {D P A V} - Q \left(s _ {t}, a _ {t}; \theta\right)\right) ^ {2} \right]. \tag {7} \\ \end{array}
$$
The algorithm of the dynamic partial average deep Q-learning network is summarized in Algorithm 1.2
The intuition behind this approach is that the predicted maximal action value overestimates the ground truth, so DPAV uses the predicted minimal action value to shift the estimate towards the ground truth. Because the predicted action values accuracy is improved in training, DPAV assigns less weight to the predicted minimum to avoid shifting towards the small estimate too much. DPAV reduces the overestimation bias in the target of the training loss, so it is less biased. This improves the dialogue action values accuracy of the DPAV DQN dialogue policy, so this dialogue policy issues more accurate dialogue actions accordingly which improve dialogue performances.
Additionally, this method has a lower computational complexity compared to those of ensemble models. Even if the latter could trade time complexity for space complexity by parallel computing, they still have high computational complexity in general as shown in the Table 2. And this method achieves better or comparable performances according to the Figure 3. The upper and lower bounds of the DPAV DQN estimation bias are also reasonable compared with those of other methods. A detailed explanation is found in section 4.3.
Algorithm 1: DPAV DQN
Initialize replay memory  $\mathcal{D}$  to capacity  $N$ , action-value function  $Q$  with random weights, and decay rate  $d$
for episode  $= 1,\dots,M$  do
Initialise state  $s_1$
for  $j = 1,\dots,T$  do
With probability  $\epsilon$  select a random action  $a_{j}$ , otherwise select
$$
a _ {j} = \max  _ {a} Q ^ {*} (s _ {j}, a; \theta)
$$
Execute action  $a_{j}$  in environment, observe reward  $r_{j + 1}$  and come into state  $s_{j + 1}$ . Store transition
$(s_j,a_j,r_{j + 1},s_{j + 1})$  in  $\mathcal{D}$  , and set  $s_j = s_{j + 1}$
Sample random minibatch of  $(s_t, a_t, r_{t+1}, s_{t+1})$  from  $\mathcal{D}$
$$
\operatorname {S e t} y _ {t} = \left\{ \begin{array}{l} r _ {t + 1} \\ \text {i f t e r m i n a l s t a t e} s _ {t + 1} \\ y ^ {D P A V} \\ \text {n o n - t e r m i n a l s t a t e} s _ {t + 1} \end{array} \right.
$$
Perform a gradient descent step on  $L_{\theta} = (y_t - Q(s_t, a_t; \theta))^2$  to update  $\theta$
Replace target parameter  $\theta^{-}\longleftrightarrow \theta$  after every  $L$  iterations. Update average weight  $\lambda_{t + 1}\longleftarrow \lambda_t*d$  after every  $U$  iterations.
In this subsection we show in Theorem 1 that in the limit DPAV Q-learning converges to the optimal
2In the algorithm, if the state  $s_{t + 1}$  is a terminal state, it means the Markov decision process ends. And in the dialogue, it means the dialogue ends.
policy. The proof<sup>3</sup> of this result using the Lemma 1 (Singh et al., 2000) is in the Appendix A.2.
Theorem 1. In a Markov decision process, the approximate action value function  $Q$  as updated by DPAV Q-learning in Equation 5 converges to the optimal action value function  $q_{*}$  with probability one if an infinite number of experience tuples in the form of  $(s_t, a_t, r_{t+1}, s_{t+1})$  are given by a learning policy for each state action pair and if the following conditions are satisfied:
1. The Markov decision process is finite (i.e.  $|\mathcal{S} \times \mathcal{A} \times \mathcal{R}| < \infty$ ,  $\mathcal{S}$  means the set of states,  $\mathcal{A}$  means the set of actions, and  $\mathcal{R}$  is the set of rewards.).
2.  $\gamma \in [0,1)$
3.  $\alpha_{t}(s,a)\in [0,1],\sum_{t}\alpha_{t}(s,a) =$ $\infty ,\sum_{t}\alpha_{t}^{2}(s,a) <   \infty w.p.1,$  and  $\forall s,a\neq s_t,a_t:$ $\alpha_{t}(s,a) = 0.\alpha_{t}(s,a)$  is the step size of a Qlearning update.
4.3 Upper and Lower Bound
As shown in (D'Eramo et al., 2016; Hasselt, 2010), considering a set of  $M \geq 2$  independent random variables  $X = \{X_{1},\ldots ,X_{M}\}$ , each random variable  $X_{i}$  has a mean  $\mu_{i} = \mathbb{E}[X_{i}]$  and a variance  $\sigma_{i} = \operatorname {Var}[X_{i}]$ . In many problems, one is interested in the maximum expected value in such a set  $\mu_{*} = \max_{i}E\{X_{i}\}$ . Without knowledge of the functional form and parameters of the underlying distribution of each variable  $X_{i}$ , it is impossible to find  $\mu_{*}$  analytically. Given a set of a limited number of samples,  $S = \{S_1,\dots,S_M\}$ ,  $S_{i}$  corresponds to the subset of samples drawn from the unknown distribution of the random variable  $X_{i}$ . The maximum estimator (Watkins and Dayan, 1992) and double estimator (Hasselt, 2010) are the most representative methods to estimate  $\mu_{*}$ . ME estimation:  $\hat{\mu}_{*}^{ME}(S) = \max_{i}\hat{\mu}_{i}(S) = \max_{i}E\{S_{i}\} \approx \mu_{*}$ . DE splits the set  $S$  into  $S^A = \{S_1^A,\dots,S_M^A\}$  and  $S^B = \{S_1^B,\dots,S_M^B\}$ . DE estimation:  $\hat{\mu}_{*}^{DE}\left(S^{A},S^{B}\right) = \hat{\mu}_{a^{*}}\left(S^{B}\right) = E\left[S_{a^{*}}^{B}\right] \approx \mu_{*}$ , with  $a^* = \arg \max_i\hat{\mu}_i(S^A)$ .
4.3.1 Bias
We start with representing the main results about the bias of Maximum Estimator (ME) and Double Estimator (DE) reported in (Van Hasselt, 2013). As for the direction of the bias, ME is positively biased, while DE is negatively biased. ME is
bounded by: Bias  $\left(\hat{\mu}_{*}^{ME}\right) \leq \sqrt{\frac{M-1}{M} \sum_{i=1}^{M} \frac{\sigma_{i}^{2}}{|S_{i}|}}$ . For the bound of DE, (Van Hasselt, 2013) conjectures the following lower bound: Bias  $\left(\hat{\mu}_{*}^{DE}\right) \geq -\frac{1}{2} \left( \sqrt{\sum_{i=1}^{M} \frac{\sigma_{i}^{2}}{|S_{i}^{A}|}} + \sqrt{\sum_{i=1}^{M} \frac{\sigma_{i}^{2}}{|S_{i}^{B}|}} \right)$ .  $M$  means the number of sample means,  $\sigma_{i}$  means the variance of the  $i_{th}$  sample mean. For the bias of DPAV estimator, we have the following bounds.
Theorem 2. For any given set  $X$  of  $M$  random variables: Bias  $\left(\hat{\mu}_{*}^{DPAV}\right) \leq$  Bias  $\left(\hat{\mu}_{*}^{ME}\right)$ , and Bias  $\left(\hat{\mu}_{*}^{DPAV}\right) \geq$  Bias  $\left(\hat{\mu}_{*}^{DE}\right)$ .
Explanation. ME uses the maximum of sample means to estimate the ground truth maximal expected value (MEV), while DPAV takes the partial average over the maximum and minimum of sample means. The minimum will shift the DPAV estimation towards the ground truth, so the upper bound of the DPAV estimator bias will be smaller than that of ME. DE uses the minimum of sample means to estimate the ground truth in the worst case, however, the DPAV estimator mitigates this bias through importing the maximum into the partial average shifting the estimation towards the ground truth. So its lower bound is larger than that of DE.
4.3.2 Variance
Since the MSE loss of an estimator is the sum of its squared bias and its variance, so we should also consider its variance to evaluate its goodness. Van Hasselt (2013) proved that both the variance of ME and the one of DE could be upper bounded by the sum of variances of sample means:
$$
\operatorname {V a r} \left(\hat {\mu} _ {*} ^ {M E / D E}\right) \leq \sum_ {i = 1} ^ {M} \frac {\sigma_ {i} ^ {2}}{| S _ {i} |}.
$$
Theorem 3. The variance of  $DPAV$  estimator is upper bounded by:  $\operatorname{Var}\left(\hat{\mu}_{*}^{DPAV}\right) \leq \frac{\sigma_{Max / Min}^{2}}{|S_{Max / Min}|} \leq \sum_{i=1}^{M} \frac{\sigma_{i}^{2}}{|S_{i}|}$ .
Explanation. Because the DPAV estimator utilizes the partial average between the maximum and minimum of sample means to estimate the ground truth. The weights assigned to the maximum and minimum are in the range (0,1), and the sum of weights is 1. According to the variance math properties (Casella and Berger, 2021), the estimation variance is smaller than the larger one among the variances of maximum and minimum of sample means. Therefore, it is also smaller than the maximal variance of all sample means.
$^{3}$ Lemma 1 was also used to prove the convergence of SARSA (Rummery and Niranjan, 1994) and Double Q-learning (Van Hasselt et al., 2016)
Task	Intentss	Slots	User goals
Movie-ticket booking	11	16	128
Restaurant reservation	11	30	3525
Taxi ordering	11	29	2830
We evaluate the DPAV DQN method and baselines on three public task-completion dialogue datasets: movie-ticket booking (Li et al., 2016, 2017), restaurant reservation and taxi ordering (Li et al., 2018). The statistics of the datasets are given in Table 1 (see Appendix B.1 for details)
The evaluation metrics are success rate and averaged reward. Success rate is the ratio of the number of tasks successfully completed by the dialogue system in evaluation to the total number of dialogues in the test set. Averaged reward refers to the average of the cumulative rewards obtained by the dialogue system for completing each dialogue of the test set.
To benchmark our method performance, we use different DQN variants as baselines in dialogue policy module for comparison: (1) DQN policy is learned with standard DQN algorithm (Mnih et al., 2015). (2) Duel DQN policy is learned by the duel network structure (Wang et al., 2016).(3) Double DQN policy uses Double Estimator of Q-learning to train (D'Eramo et al., 2016). (4) Averaged DQN policy is trained by taking average over multiple action values of target networks (Anschel et al., 2017). (5) Maxmin DQN policy uses the minimum of multiple maximums from different ensemble units to estimate the ground truth maximal action value in a selective process (Lan et al., 2020). (6) SUNRISE policy trains with weighted Bellman backups from multiple networks (Lee et al., 2021). Our model DPAV uses a value function instead of a combination of multiple value functions to tailor the maximum and the minimum action value.
We conduct two  $\lambda$  searching schemes: neural network (NN) searching and heuristic searching. We also analyze the influence of different initial value  $\lambda_0$  in the heuristic searching. So we have following models in the experiment: (1) LambdaX is