SlowGuess/NAACL_2022_OCR huggingface dataset:
the heuristic searching version of the DPAV DQN. The floating number  $X$  is the initial value  $\lambda_0$  with the range  $(0, 1)$ . And LambdaX (e.g. Lambda0.5, Lambda0.6) searches different floating numbers  $X$  for initial  $\lambda_0$  in the heuristic searching. (2) LambdaNet is the neural network searching version of the DPAV DQN. It trains a NN to find a value for  $\lambda_t$  for each dialogue state  $s_k$  in the training process. Here,  $\lambda_t$  means the value  $\lambda$  in the training episode  $t$ , and  $s_k$  represents the dialogue state  $k$  sampled from the experience replay buffer of reinforcement learning.
5.3 Implementation Details
This work is implemented with PyTorch toolkit. Compared with the standard DQN algorithm, we change the loss with the one defined by DPAV DQN in Algorithm 1. For these RL-based dialogue policies, action value network  $Q(\cdot)$  is a MLP with one hidden layer of 80 hidden nodes. ReLU is the activation function. A greedy policy is used in the evaluation. All neural networks warm start 120 episodes using the same rule-based policy before training and are trained with the same hyper-parameters. We follow the default hyperparameters of the user simulator setting. The discount factor  $\gamma$  for future reward is 0.9. The batch size is 16, and the learning rate is 0.001. The test set size in the movie domain and other domains is set to 100 and 500, respectively. All baselines are based on DQN for a fair comparison. We set  $L = 40$  as the maximum of dialogue turns in all domains. The heuristically searched decay rate  $d$  and decay interval of the DPAV estimator in the movie domain and other domains are set to (0.75, 15 train iterations) and (0.9965, 30 train iterations), respectively. For specific parameters of each model and the user simulator, we refer to Appendix B.2.
The main simulation results are reported in Figure 3, we evaluate each dialogue policy performance in terms of success rate and averaged reward. The top two rows of Figure 3 show DPAV DQN consistently outperforms DQN. The overestimation error in target Q values gets propagated into the DQN Q values, while DPAV DQN reduces the overestimation error then its Q values will be less biased. So it more correctly creates dialogue utterances based on the Q values and achieves a better success rate and averaged reward.
Our DPAV DQN method performs better than
$^{4}$  (Zhao et al., 2021) argued that the three tasks have been widely used in the research of dialogue policy.
(a) Movie domain success rate
(b) Restaurant domain success rate
(c) Taxi domain success rate
(d) Movie domain average reward
(e) Restaurant domain average reward
(f) Taxi domain average reward
(g) Movie domain parameter study
(h) Restaurant domain parameter study
(i) Taxi domain parameter study
Figure 3: The top row shows the learning curves of dialogue policies. The X-axis is the number of training epochs and the Y-axis is the success rate of dialogue policies on the test dataset. The second row shows the averaged reward of each dialogue in the test dataset. The third row shows the influences of different initial  $\lambda$  values and value search schemes. The X-axis and Y-axis are the same as those of the top row. Each learning curve is averaged over 3 runs on the test dataset.
the baselines in terms of general performance. Since the training starts with the experience pool initialized by the the same rule-based dialogue policy, the models' performance in the very first few episodes is very similar. After that, the performance improved for all models, but much rapidly for DPAV DQN, which finally converges to a higher success rate and averaged reward. As we claimed above, the DPAV estimator reduces the overestimation error propagated into its model Q values and results in better action values estimation. Ensemble models performance relies on its number of networks. With a limited number of networks, as mentioned in the Related Work, Averaged DQN still suffers overestimation bias, Maxmin DQN has estimation bias from the coarse estimator and SUNRISE only down-weights the biased estimation. For
non-ensemble models, Duel DQN suffers overestimation with the Maximum Estimator, while Double DQN has underestimation bias (Anschel et al., 2017). These drawbacks of the baselines get their biased loss propagated into the policy model Q values and hurt the accuracy of the policy models. So their performance (i.e., success rate and averaged reward) cannot improve further after reaching a certain level. The training efficiency and performance of DPAV DQN in comparison validate the effectiveness of our model.
However, in the taxi domain, Duel DQN outperforms other dialogue policies. DPAV DQN only slightly improves the results compared to DQN but it converges faster than DQN. Because sometimes there is no explicitly preferable action for a state, so the action values of the state will be sim
Figure 4: The learning curves of the averaged maximal action value of the dialogue starting state when dialogue policies are evaluated on the movie test set during the training. The Y-axis means the averaged maximal action value of the starting state.
ilar (Thrun and Schwartz, 1993), and the DPAV estimator cannot notably reduce estimation bias through averaging between maximum and minimum action values. But the DPAV estimator still could estimate better than other baselines (except for Duel DQN) as shown in the results. Duel DQN uses the duel network structure to estimate action values (Wang et al., 2016), which is helpful for recognizing the correct action when confronted with confusing states.
5.5 Influence of Parameter  $\lambda$
Intuitively, the optimal  $\lambda$  should seek the best tradeoff between the estimated maximum and minimum that could be used to train the dialogue policy properly. It is a non-trivial optimization problem because the distribution of action values  $Q(s_{t},\cdot)$  at state  $s_t$  is constantly updated, and the optimal  $\lambda$  for  $s_t$  should be adjusted accordingly. The third row of Figure 3 shows that with the neural network (NN) searching almost each dialogue policy evaluation has zero success rate and can not converge. Since the distribution and the optimal  $\lambda$  are changing for the same state  $s_t$ , the fixed  $\lambda$  searched by the neural network does not work. This validates that the  $\lambda$  for  $s_t$  is dynamic, and a fixed  $\lambda$  leads to bad performances.
It is a difficult problem to calculate the exact ground truth maximal action value, so existing works use estimators to approximate it (Lan et al., 2020; Lee et al., 2021; Anschel et al., 2017). DPAV DQN uses the DPAV estimator for the approximation. In the heuristic searching version of DPAV
DQN,  $\lambda_0$  is a very important initial value for the DPAV estimator.  $\lambda_0$  is the initial weight of the minimum, because finally we will give the total confidence to our model, the weight of the maximum will be nearly 1. So the lower bound for  $\lambda_0$  is 0. Since model reduces the overestimation of the maximum through shifting the estimate towards the minimum action value. It is a trade-off, so the upper bound for  $\lambda_0$  is 1. It is problem dependent and should be set in a range (0,1).
As shown in the third row of Figure 3, among three dialogue datasets: movie, restaurant and taxi, we empirically find that the initial value  $\lambda_0$  around 0.5 results in good performances. And other heuristic values degrade the dialogue policy performances. Since shifting the estimate towards the minimum action value too much or too less both causes the estimation bias of the ground truth maximal action value. These validate that  $\lambda_0$  is problem dependent and  $\lambda_t$  should decay to proper values to balance the maximum and the minimum along the training.
5.6 Computational Complexity Comparison
All baselines and DPAV DQN use various estimators or estimation tricks to approximate the ground truth maximal action value  $Q_{*}(s_{t + 1})$ . Given the state  $s_{t + 1}$  as the input to all baselines and DPAV DQN, the time complexity of the forward propagation of every model unit has a similar time complexity besides the minor differences (e.g., addition). In order to facilitate the comparison, we denote the time complexity for the forward propagation of every model unit as  $O(N)$ , here,  $N$  means the dimension of the vector input for forward propagation. In this comparison, we suppose each ensemble model has  $K$  model units.
Combining the results of Figure 3 and Table 2, DPAV DQN achieves better or comparable performances with a lower time complexity. Although the time complexity of ensemble models can be reduced by parallel computing, but that increases the space complexity. So, the overall computational complexity is still high and resource consuming.
5.7 Results on Maximum Action Value
In reinforcement learning, the action value  $Q$  is the expectation of return  $R_{t}$  that is the sum of the discounted rewards:  $Q(s,a) = E\{R_{t} \mid s_{t} = s, a_{t} = a\} = E\{\sum_{k=0}^{\infty} \gamma^{k} r_{t+k+1} \mid s_{t} = s, a_{t} = a\}$ . Fig-
Model Name	Time Complexity
DQN	O(N)
Double DQN	O(N)
Duel DQN	O(N)
Averaged DQN	O(K*N)
Maxmin DQN	O(K*N)
SUNRISE	O(K*N)
DPAV DQN	O(N)
Table 2: Time complexity comparison among baselines and DPAV DQN. N refers to the dimension of the vector input for the forward propagation. K means the number of ensemble units. O measures the time complexity of models.
ure 4 shows learning curves<sup>5</sup> of the averaged maximum action value for the starting state on the test set, the value in dialogue context means how much return the dialogue policy assumes it could maximally receive from the starting state.
At the first few training epochs, we notice that the averaged maximum value of DPAV DQN is negative which is consistent with the averaged reward of its evaluation shown in Figure 3d, because at the early training stage, the policy quality is too low to finish most of the dialogues so the averaged reward is low and the averaged maximum action value should be low if the model Q values are accurate. But the values of other models are not consistent with and larger than the real averaged reward. Because the estimation bias of the loss makes that these models have inaccurate Q values, the maximum action value of these models is larger than the ground truth.
The policy training based on these inaccurate Q values will be negatively affected. Only using Maximum Estimator (ME) will cause overestimation bias and even lead to worse policy quality, it can be observed from the curves of DQN and Duel DQN in Figure 4. Averaged DQN and Maxmin DQN use ME in their single unit so the bias leads their Q functions to converge into inaccurate values, which prevents averaged maximum action values from improving further. SUNRISE down-weights the biased estimation and it is trained in such a way so that SUNRISE dialogue policy receives more rewards during the evaluation 3d. As shown in the Figure 4, the averaged maximal action value of DPAV DQN remains the highest among the three datasets because its model gets trained better with
the less biased loss and receives more return from successful dialogues during evaluation. This also coincides with the averaged reward from test dialogues in Figure 3d. This empirically validates that DPAV is a better estimator than others because of less estimation bias.
This paper is the first to investigate the negative effects of the overestimation problem in task-completion dialogue systems. We propose the DPAV estimator to mitigate this problem of Q-learning. We also theoretically prove convergence and derive the upper and lower bounds of the estimation bias compared with those of other methods. The resulting DPAV DQN model is empirically evaluated on three dialogue datasets and achieves better or comparable results with lower computational load compared to state-of-the-art baselines.
We would like to thank Xiumei Zhao, Dan Li, Wentao Huang and Pengjie Ren for reading the paper draft. This research was partially supported by the China Scholarship Council. All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors. Marie-Francine Moens is supported by the ERC Advanced Grant CALCULUS (788506).
Oron Anschel, Nir Baram, and Nahum Shimkin. 2017. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In International conference on machine learning, pages 176-185. PMLR.
Shamama Anwar and K Sridhar Patnaik. 2008. Actor critic learning: A near set approach. In International Conference on Rough Sets and Current Trends in Computing, pages 252-261. Springer.
George Casella and Roger L Berger. 2021. Statistical inference. Cengage Learning.
Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A survey on dialogue systems: Recent advances and new frontiers. Acm Sigkdd Explorations Newsletter, 19(2):25-35.
Carlo D'Eramo, Alessandro Nuara, Matteo Pirotta, and Marcello Restelli. 2017. Estimating the maximum expected value in continuous reinforcement learning problems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
5To save space, we only present the results on the movie dataset, and the results on other datasets are similar.
Carlo D'Eramo, Marcello Restelli, and Alessandro Nuara. 2016. Estimating maximum expected value through gaussian approximation. In International Conference on Machine Learning, pages 1032-1040. PMLR.
Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. 2020. A theoretical analysis of deep q-learning. In Learning for Dynamics and Control, pages 486-489. PMLR.
Hado Hasselt. 2010. Double q-learning. Advances in neural information processing systems, 23:2613-2621.
Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. 2020. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, pages 5556-5566. PMLR.
Qingfeng Lan, Yangchen Pan, Alona Fyshe, and Martha White. 2020. Maxmin q-learning: Controlling the estimation bias of q-learning. arXiv preprint arXiv:2002.06487.
Donghun Lee, Boris Defourny, and Warren B Powell. 2013. Bias-corrected q-learning to control max-operator bias in q-learning. In 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pages 93-99. IEEE.
Donghun Lee and Warren B Powell. 2012. An intelligent battery controller using bias-corrected q-learning. In Twenty-Sixth AAAI Conference on Artificial Intelligence.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. 2021. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning, pages 6131-6141. PMLR.
Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017. End-to-end task-completion neural dialogue systems. arXiv preprint arXiv:1703.01008.
Xiujun Li, Zachary C Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen. 2016. A user simulator for task-completion dialogues. arXiv preprint arXiv:1612.05688.
Xiujun Li, Yu Wang, Siqi Sun, Sarah Panda, Jingjing Liu, and Jianfeng Gao. 2018. Microsoft dialogue challenge: Building end-to-end task-completion dialogue systems. arXiv preprint arXiv:1807.11125.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. nature, 518(7540):529-533.
Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, Kam-Fai Wong, and Shang-Yu Su. 2018. Deep dynaq: Integrating planning for task-completion dialogue policy learning. arXiv preprint arXiv:1801.06176.
Andreas Pentaliotis and Marco A Wiering. 2021. Variation-resistant q-learning: Controlling and utilizing estimation bias in reinforcement learning for better performance. In ICAART (2), pages 17-28.
Gavin A Rummery and Mahesan Niranjan. 1994. Online Q-learning using connectionist systems, volume 37. CiteSeer.
Satinder Singh, Tommi Jaakkola, Michael L Littman, and Csaba Szepesvári. 2000. Convergence results for single-step on-policy reinforcement-learning algorithms. Machine learning, 38(3):287-308.
James E Smith and Robert L Winkler. 2006. The optimizer's curse: Skepticism and postdecision surprise in decision analysis. Management Science, 52(3):311-322.
Sebastian Thrun and Anton Schwartz. 1993. Issues in using function approximation for reinforcement learning. In Proceedings of the Fourth Connectionist Models Summer School, pages 255-263. Hillsdale, NJ.
Hado Van Hasselt. 2013. Estimating the maximum expected value: an analysis of (nested) cross validation and the maximum sample average. arXiv preprint arXiv:1302.7175.
Hado Van Hasselt, Arthur Guez, and David Silver. 2016. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30.
Sihan Wang, Kaijie Zhou, Kunfeng Lai, and Jianping Shen. 2020. Task-completion dialogue policy learning via monte carlo tree search with dueling network. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3461-3471.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. 2016. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995-2003. PMLR.
Christopher JCH Watkins and Peter Dayan. 1992. Q-learning. Machine learning, 8(3-4):279-292.
Yuexin Wu, Xiujun Li, Jingjing Liu, Jianfeng Gao, and Yiming Yang. 2019. Switch-based active deep dynaq: Efficient adaptive planning for task-completion dialogue policy learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7289-7296.
Rui Zhang, Zhenyu Wang, Mengdan Zheng, Yangyang Zhao, and Zhenhua Huang. 2021. Emotion-sensitive deep dyna-q learning for task-completion dialogue policy learning. Neurocomputing, 459:122-130.
Yichi Zhang, Zhijian Ou, Min Hu, and Junlan Feng. 2020a. A probabilistic end-to-end task-oriented dialog model with latent belief states towards semi-supervised learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9207-9219.
Zheng Zhang, Ryuichi Takanobu, Qi Zhu, MinLie Huang, and XiaoYan Zhu. 2020b. Recent advances and challenges in task-oriented dialog systems. Science China Technological Sciences, pages 1-17.
Zhirui Zhang, Xiujun Li, Jianfeng Gao, and Enhong Chen. 2019. Budgeted policy learning for task-oriented dialogue systems. arXiv preprint arXiv:1906.00499.
Zongzhang Zhang, Zhiyuan Pan, and Mykel J Kochenderfer. 2017. Weighted double q-learning. In *IJCAI*, pages 3455–3461.
Yangyang Zhao, Zhenyu Wang, Changxi Zhu, and Shihan Wang. 2021. Efficient dialogue complementary policy learning via deep q-network policy and episodic memory policy. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4311-4323.
A.1 Lemma
Lemma 1 (Hasselt, 2010). Let  $(\beta_{t},\Delta_{t},F_{t})$  be a stochastic process, where  $\beta_{t},\Delta_{t},F_{t}:X\mapsto \mathbb{R}$  satisfy,
$$
\Delta_ {t + 1} (x _ {t}) = \left(1 - \beta_ {t} (x _ {t})\right) \Delta_ {t} (x _ {t}) + \beta_ {t} (x _ {t}) F _ {t} (x _ {t})
$$
where  $x_{t} \in X$  and  $t = 0,1,2,\ldots$ . Let  $P_{t}$  be a sequence of increasing  $\sigma$ -fields such that  $\beta_{0}$  and  $\Delta_0$  are  $P_{0-}$  measurable and  $\beta_{t}, \Delta_{t}$ , and  $F_{t-1}$  are  $P_{t}$ -measurable, with  $t \geq 1$ . Assume that the following conditions are satisfied:
1. The set  $X$  is finite (i.e.  $|X| < \infty$ ).
2.  $\beta_{t}(x_{t})\in [0,1],\sum_{t}\beta_{t}(x_{t}) = \infty ,\sum_{t}\beta_{t}^{2}(x_{t}) < \infty$  w.p. 1, and  $\forall x\neq x_{t}:\beta_{t}(x) = 0.\beta_{t}(x_{t})$  is the step size of the update.
3.  $\| \mathbb{E}\left\{F_t\mid P_t\right\} \| \leq \kappa \| \Delta_t\| +c_t$  , where  $\kappa \in [0,1)$  and  $c_{t}\to 0$  w.p.1.
4.  $\mathbb{V}\left\{F_t(x_t)\mid P_t\right\} \leq C\left(1 + \kappa \left\| \Delta_t\right\|\right)^2$  where  $C$  is some constant.
where  $\mathbb{V}\{\cdot\}$  denotes the variance and  $\|\cdot\|$  denotes the maximum norm. Then  $\Delta_t$  converges to zero with probability one.
Proof. See (Singh et al., 2000).
A.2 DPAV Q-learning Convergence Proof
Proof. We apply Lemma 1 with  $X = \mathcal{S} \times \mathcal{A}$ ,  $\Delta_t = Q_t(s_t, a_t) - q_*(s_t, a_t)$ ,  $\beta_t = \alpha_t$ ,  $\beta_t$  is also the step size,  $P_t = \{Q_0, s_0, a_0, \alpha_0, r_1, s_1, \ldots, s_t, a_t\}$  and
$$
F _ {t} \left(s _ {t}, a _ {t}\right) = r _ {t + 1} + \gamma Q _ {d p a v} \left(s _ {t + 1}, \cdot\right) - q _ {*} \left(s _ {t}, a _ {t}\right), \tag {8}
$$
$$
\begin{array}{l} Q _ {d p a v} \left(s _ {t + 1}, \cdot\right) = \left(1 - \lambda_ {t}\right) Q _ {t} \left(s _ {t + 1}, a _ {\max }\right) \tag {9} \\ + \lambda_ {t} Q _ {t} \left(s _ {t + 1}, a _ {\min }\right). \\ \end{array}
$$
And  $a_{max} = \arg \max_{a'} Q_t(s_{t+1}, a')$  while  $a_{min} = \arg \min_{a''} Q_t(s_{t+1}, a'')$ . The first condition of the Lemma 1 is satisfied because  $|S \times \mathcal{A}| < \infty$ . The second condition of Lemma 1 is met by the third condition of Theorem 1. Because the absolute value of reward  $|r| < \infty \Rightarrow \forall t : \mathbb{V}\{r_{t+1} \mid P_t\} < \infty$ . Since  $Q_t$  is the expected cumulative reward in Q-learning and  $F_t(s_t, a_t)$  is composed of reward  $r$ , so  $\forall t : \mathbb{V}\{r_{t+1} \mid P_t\} < \infty \Rightarrow \forall t : \mathbb{V}\{F_t(s_t, a_t) \mid P_t\} < \infty$ , the fourth condition of the Lemma 1 is sufficed. This leaves to show that the third condition of the Lemma 1 on the expected contraction of  $F_t$  holds. We can write
$$
\begin{array}{l} F _ {t} \left(s _ {t}, a _ {t}\right) = r _ {t + 1} + \gamma \left(\left(1 - \lambda_ {t}\right) Q _ {t} \left(s _ {t + 1}, a _ {\max }\right) \right. \\ + \lambda_ {t} Q _ {t} \left(s _ {t + 1}, a _ {\min }\right)) - q _ {*} \left(s _ {t}, a _ {t}\right) \\ = r _ {t + 1} + \gamma \left(Q _ {t} \left(s _ {t + 1}, a _ {\max }\right) - \lambda_ {t} Q _ {t} \left(s _ {t + 1}, a _ {\max }\right) \right. \\ + \lambda_ {t} Q _ {t} \left(s _ {t + 1}, a _ {\min }\right)) - q _ {*} \left(s _ {t}, a _ {t}\right) \\ = r _ {t + 1} + \gamma Q _ {t} \left(s _ {t + 1}, a _ {\max }\right) - q _ {*} \left(s _ {t}, a _ {t}\right) \\ + \gamma \lambda_ {t} \left(Q _ {t} \left(s _ {t + 1}, a _ {\min }\right) - Q _ {t} \left(s _ {t + 1}, a _ {\max }\right)\right) \\ = r _ {t + 1} + \gamma Q _ {t} \left(s _ {t + 1}, a _ {\max }\right) - q _ {*} \left(s _ {t}, a _ {t}\right) \\ + \gamma \lambda_ {t} Q _ {s u b} \\ = F _ {t} ^ {\prime} \left(s _ {t}, a _ {t}\right) + \gamma \lambda_ {t} Q _ {s u b}, \\ \end{array}
$$
where  $F_{t}^{\prime}$  is the value of  $F_{t}$  if normal Qlearning would be under consideration, and  $Q_{sub} = Q_{t}(s_{t + 1},a_{\mathrm{min}}) - Q_{t}(s_{t + 1},a_{\mathrm{max}})$ . Since it is well known that  $\forall t:\| \mathbb{E}\{F_t'\mid P_t\} \| \leq$ $\gamma \| \Delta_t\|$  (Hasselt, 2010), it follows that,  $\| \mathbb{E}\{F_t\mid P_t\} \|$
$$
\begin{array}{l} = \left\| \mathbb {E} \left\{F _ {t} ^ {\prime} \mid P _ {t} \right\} \right\| + \gamma \lambda_ {t} \| \mathbb {E} \left\{Q _ {s u b} \mid P _ {t} \right\} \| \\ \leq \gamma \| \Delta_ {t} \| + \gamma \lambda_ {t} \| \mathbb {E} \left\{Q _ {\text {s u b}} \mid P _ {t} \right\} \| \\ \end{array}
$$
Since in DPAV Q-learning, the  $\lambda_{t}$  will decay as  $\lambda_{t + 1} = \lambda_t*d$ . When  $t\to \infty$ , given  $\varepsilon >0,\exists t_0:\forall t\geq t_0\Rightarrow \lambda_t < \varepsilon \Rightarrow \lim_{t\to \infty}\lambda_t = 0$ . Therefore, it suffices to show that  $c_{t} = \gamma \lambda_{t}Q_{sub}\rightarrow 0$  w.p.1. Since all the conditions of lemma 1 are satisfied, it holds that,  $\forall s,a:Q_{t}(s,a)\to q_{*}(s,a)$  w.p.1.