SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 2015.
[2] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
[3] John Tsitsiklis and Benjamin Van Roy. Analysis of temporal-difference learning with function approximation. Advances in neural information processing systems, 1996.
[4] Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning. Elsevier, 1995.
[5] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[6] Richard S Sutton. Learning to predict by the methods of temporal differences. Journal of Machine Learning Research, 1988.
[7] Geoffrey J Gordon. Stable function approximation in dynamic programming. In Machine learning. Elsevier, 1995.
[8] Donghwan Lee and Niao He. Target-based temporal-difference learning. In International Conference on Machine Learning, 2019.
[9] Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[10] Hamid Reza Maei. Gradient temporal-difference learning algorithms. PhD thesis, University of Alberta, 2011.
[11] Kavosh Asadi, Rasool Fakoor, and Shoham Sabach. Resetting the optimizer in deep RL: An empirical study. Advances in Neural Information Processing Systems, 2023.
[12] Joan Bas-Serrano, Sebastian Curi, Andreas Krause, and Gergely Neu. Logistic Q-learning. In International Conference on Artificial Intelligence and Statistics, 2021.
[13] Doina Precup and Richard S Sutton. Exponentiated gradient methods for reinforcement learning. In International Conference on Machine Learning, 1997.
[14] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 2005.
[15] Hamid Reza Maei, Csaba Szepesvári, Shalabh Bhatnagar, and Richard S Sutton. Toward off-policy learning control with function approximation. International Conference on Machine Learning, 2010.
[16] J Kolter. The fixed points of off-policy td. Advances in Neural Information Processing Systems, 2011.
[17] Carl D Meyer. Matrix analysis and applied linear algebra. SIAM, 2000.
[18] Amir Beck. First-order methods in optimization. SIAM, 2017.
[19] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society. Series B (Methodological), 1996.
[20] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[21] Brandon Amos, Lei Xu, and J Zico Kolter. Input convex neural networks. In International Conference on Machine Learning, 2017.
[22] Kavosh Asadi and Michael L Littman. An alternative softmax operator for reinforcement learning. In International Conference on Machine Learning, 2017.
[43] András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path. Journal of Machine Learning Research, 2008.
[44] Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. Sbeed: Convergent reinforcement learning with nonlinear function approximation. In International Conference on Machine Learning, 2018.
[45] Ehsan Saleh and Nan Jiang. Deterministic Bellman residual minimization. In Proceedings of Optimization Foundations for Reinforcement Learning Workshop at NeurIPS, 2019.
[46] Michail G Lagoudakis and Ronald Parr. Least-squares policy iteration. Journal of Machine Learning Research, 2003.
[47] Daniel Lizotte. Convergent fitted value iteration with linear function approximation. Advances in Neural Information Processing Systems, 2011.
[48] Jianqing Fan, Zhaoran Wang, Yuchen Xie, and Zhuoran Yang. A theoretical analysis of deep Q-learning. In Learning for Dynamics and Control, 2020.
[49] Rémi Munos. Performance bounds in 1-p-norm for approximate value iteration. SIAM journal on control and optimization, 2007.
[50] Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 2008.
[51] Amir-massoud Farahmand, Csaba Szepesvári, and Rémi Munos. Error propagation for approximate policy and value iteration. Advances in Neural Information Processing Systems, 2010.
[52] Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, 2020.
[53] Simon S Du, Yuping Luo, Ruosong Wang, and Hanrui Zhang. Provably efficient Q-learning with function approximation via distribution shift error checking oracle. Advances in Neural Information Processing Systems, 2019.
[54] Simon S Du, Jason D Lee, Gaurav Mahajan, and Ruosong Wang. Agnostic Q-learning with function approximation in deterministic systems: Near-optimal bounds on approximation error and sample complexity. Advances in Neural Information Processing Systems, 2020.
[55] Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural complexity and representation learning of low rank mdps. Advances in neural information processing systems, 2020.
[56] Shuhang Chen, Adithya M Devraj, Fan Lu, Ana Busic, and Sean Meyn. Zap q-learning with nonlinear function approximation. Advances in Neural Information Processing Systems, 2020.
[57] Richard S Sutton, Hamid Maei, and Csaba Szepesvári. A convergent  $o(n)$  temporal-difference algorithm for off-policy learning with linear function approximation. Advances in neural information processing systems, 2008.
10 Appendix
$$
\theta^ {t + 1} - \theta^ {\star} = \mathbf {M} _ {w} ^ {- 1} \mathbf {M} _ {\theta} (\theta^ {t} - \theta^ {\star}).
$$
Proof. Since  $\theta^{t + 1}\gets \arg \min_wH(\theta^t,w)$ , we have  $\nabla_wH(\theta^t,\theta^{t + 1}) = \mathbf{0}$ . Using (7) we have:
$$
\mathbf {M} _ {w} \theta^ {t + 1} = \Phi^ {\top} D R + \mathbf {M} _ {\theta} \theta^ {t} = (\mathbf {M} _ {w} - \mathbf {M} _ {\theta}) \theta^ {\star} + \mathbf {M} _ {\theta} \theta^ {t},
$$
where the last equality follows from  $\theta^{\star}$  being the fixed-point and therefore  $\nabla_w h(\theta^\star, \theta^\star) = \mathbf{0}$ , which in light of (7) translates to  $\Phi^\top DR = (\mathbf{M}_w - \mathbf{M}_\theta)\theta^\star$ . Multiplying both sides by  $\mathbf{M}_w^{-1}$ , we get:
$$
\theta^ {t + 1} = \left(I - \mathbf {M} _ {w} ^ {- 1} \mathbf {M} _ {\theta}\right) \theta^ {\star} + \mathbf {M} _ {w} ^ {- 1} \mathbf {M} _ {\theta} \theta^ {t}.
$$
Rearranging the equality leads into the desired result.
Proof. First notice that  $\theta^{t + 1}\gets \arg \min_wH(\theta^t,w)$ , so we have  $\nabla_wH(\theta^t,\theta^{t + 1}) = \mathbf{0}$ . Now, using the  $F_{w}$ -strong convexity of  $w\to H(\theta^t,w)$ , we get that
$$
\begin{array}{l} F _ {w} \| \theta^ {t + 1} - \theta^ {\star} \| ^ {2} \leq \langle \theta^ {\star} - \theta^ {t + 1}, \nabla_ {w} H (\theta^ {t}, \theta^ {\star}) - \nabla_ {w} H (\theta^ {t}, \theta^ {t + 1}) \rangle \\ = \langle \theta^ {\star} - \theta^ {t + 1}, \nabla_ {w} H (\theta^ {t}, \theta^ {\star}) \rangle \quad (\text {f r o m} \nabla_ {w} H (\theta^ {t}, \theta^ {t + 1}) = \mathbf {0}). \\ \end{array}
$$
Now, since  $\theta^{\star}$  is a fixed-point, it follows that  $\nabla_w H(\theta^\star, \theta^\star) = \mathbf{0}$ . Therefore, we have:
$$
\begin{array}{l} F _ {w} \left\| \theta^ {t + 1} - \theta^ {\star} \right\| ^ {2} \leq \left\langle \theta^ {\star} - \theta^ {t + 1}, \nabla_ {w} H \left(\theta^ {t}, \theta^ {\star}\right) - \nabla_ {w} H \left(\theta^ {\star}, \theta^ {\star}\right) \right\rangle \\ \leq \left\| \theta^ {t + 1} - \theta^ {\star} \right\| \cdot \left\| \nabla_ {w} H \left(\theta^ {t}, \theta^ {\star}\right) - \nabla_ {w} H \left(\theta^ {\star}, \theta^ {\star}\right) \right\| \\ \leq F _ {\theta} \| \theta^ {t + 1} - \theta^ {\star} \| \cdot \| \theta^ {t} - \theta^ {\star} \|, \\ \end{array}
$$
where in the second line we used to Cauchy-Schwartz inequality, and the last inequality follows from the  $F_{\theta}$ -Lipschitz property of  $\nabla_w H(\cdot, \theta^\star)$ . Since this inequality holds true for any  $t \in \mathbb{N}$ , we get that if  $\theta^t = \theta^\star$ , then we also have that  $\theta^{t+1} = \theta^\star$ . Thus, if  $\theta^T = \theta^\star$  for some  $T \in \mathbb{N}$ , then  $\bar{\theta}^t = \theta^\star$  for any  $t \geq T$  and so the algorithm has converged. On the other hand, if  $\theta^t \neq \theta^\star$  for all  $t \in \mathbb{N}$ , the desired result follows after dividing both sides by  $\| \theta^{t+1} - \theta^\star \|$ .
To prove Proposition 5 we need to provide two new Propositions to later make use of. We present these two proofs next, and then restate and prove Proposition 5.
Proposition 6. Let  $\{\theta^t\}_{t\in \mathbb{N}}$  be a sequence generated by Algorithm 2. Then, for all  $t\in \mathbb{N}$  and  $0\leq k\leq K - 1$ , we have
$$
H \left(\theta^ {t}, \theta^ {\star}\right) - H \left(\theta^ {t}, w ^ {t, k}\right) \leq \frac {F _ {\theta} ^ {2}}{2 F _ {w}} \| \theta^ {t} - \theta^ {\star} \| ^ {2} - \left(\frac {1}{\alpha} - \frac {L}{2}\right) \| w ^ {t, k + 1} - w ^ {t, k} \| ^ {2}.
$$
Proof. Let  $t \in \mathbb{N}$ . From the  $F_{w}$ -strong convexity of  $w \to H(\theta^t, w)$ , we obtain that
$$
H (\theta^ {t}, w ^ {t, k + 1}) \geq H (\theta^ {t}, \theta^ {\star}) + \langle \nabla_ {w} H (\theta^ {t}, \theta^ {\star}), w ^ {t, k + 1} - \theta^ {\star} \rangle + \frac {F _ {w}}{2} \| w ^ {t, k + 1} - \theta^ {\star} \| ^ {2},
$$
$$
H \left(\theta^ {t}, \theta^ {\star}\right) - H \left(\theta^ {t}, w ^ {t, k + 1}\right) \leq \left\langle \nabla_ {w} H \left(\theta^ {t}, \theta^ {\star}\right), \theta^ {\star} - w ^ {t, k + 1} \right\rangle - \frac {F _ {w}}{2} \| w ^ {t, k + 1} - \theta^ {\star} \| ^ {2}. \tag {8}
$$
Moreover, in light of the fixed-point characterization of TD, namely that  $\nabla_w H(\theta^\star, \theta^\star) = \mathbf{0}$ , we can write
$$
\begin{array}{l} \langle \nabla_ {w} H (\theta^ {t}, \theta^ {\star}), \theta^ {\star} - w ^ {t, k + 1} \rangle = \langle \nabla_ {w} H (\theta^ {t}, \theta^ {\star}) - \nabla_ {w} H (\theta^ {\star}, \theta^ {\star}), \theta^ {\star} - w ^ {t, k + 1} \rangle \\ = \left(\nabla_ {w} H \left(\theta^ {t}, \theta^ {\star}\right) - \nabla_ {w} H \left(\theta^ {\star}, \theta^ {\star}\right)\right) ^ {\top} \left(\theta^ {\star} - w ^ {t, k + 1}\right) \\ \leq \frac {1}{2 F _ {w}} \| \nabla_ {w} H (\theta^ {t}, \theta^ {\star}) - \nabla_ {w} H (\theta^ {\star}, \theta^ {\star}) \| ^ {2} + \frac {F _ {w}}{2} \| w ^ {t, k + 1} - \theta^ {\star} \| ^ {2} \\ \leq \frac {F _ {\theta} ^ {2}}{2 F _ {w}} \| \theta^ {t} - \theta^ {\star} \| ^ {2} + \frac {F _ {w}}{2} \| w ^ {t, k + 1} - \theta^ {\star} \| ^ {2}. \tag {9} \\ \end{array}
$$
Here, the first inequality follows from the fact that for any two vectors  $a$  and  $b$  we have  $a^\top b \leq (1/2d)\|a\|^2 + (d/2)\|b\|^2$  for any  $d > 0$ . In this case, we chose  $d = F_w$ . Also the last inequality follows from the  $F_\theta$ -Lipschitz property of  $\nabla_w H$ , which is our assumption.
Now, by combining (8) with (9) we obtain that
$$
\begin{array}{l} H (\theta^ {t}, \theta^ {\star}) - H (\theta^ {t}, w ^ {t, k + 1}) \leq \frac {F _ {\theta} ^ {2}}{2 F _ {w}} \| \theta^ {t} - \theta^ {\star} \| ^ {2} + \frac {F _ {w}}{2} \| w ^ {t, k + 1} - \theta^ {\star} \| ^ {2} - \frac {F _ {w}}{2} \| w ^ {t, k + 1} - \theta^ {\star} \| ^ {2} \\ = \frac {F _ {\theta} ^ {2}}{2 F _ {w}} \| \theta^ {t} - \theta^ {\star} \| ^ {2}. \tag {10} \\ \end{array}
$$
From the Lipschitz assumption we can write, due to the Descent Lemma [18] applied to the function  $w \to H(\theta^t, w)$ , that:
$$
H \left(\theta^ {t}, w ^ {t, k + 1}\right) - H \left(\theta^ {t}, w ^ {t, k}\right) \leq \langle \nabla_ {w} H \left(\theta^ {t}, w ^ {t, k}\right), w ^ {t, k + 1} - w ^ {t, k} \rangle + \frac {L}{2} \| w ^ {t, k + 1} - w ^ {t, k} \| ^ {2}
$$
Now, notice that according to Algorithm 2 we have  $w^{t,k + 1} = w^{t,k} - \alpha \nabla_w H(\theta^t,w^{t,k})$ , and so we can write:
$$
\begin{array}{l} H \left(\theta^ {t}, w ^ {t, k + 1}\right) - H \left(\theta^ {t}, w ^ {t, k}\right) \leq \frac {1}{\alpha} \langle w ^ {t, k} - w ^ {t, k + 1}, w ^ {t, k + 1} - w ^ {t, k} \rangle + \frac {L}{2} \| w ^ {t, k + 1} - w ^ {t, k} \| ^ {2} \\ = - \left(\frac {1}{\alpha} - \frac {L}{2}\right) \| w ^ {t, k + 1} - w ^ {t, k} \| ^ {2}. \tag {11} \\ \end{array}
$$
Adding both sides of (10) with (11) yields:
$$
H \left(\theta^ {t}, \theta^ {\star}\right) - H \left(\theta^ {t}, w ^ {t, k}\right) \leq \frac {F _ {\theta} ^ {2}}{2 F _ {w}} \| \theta^ {t} - \theta^ {\star} \| ^ {2} - \left(\frac {1}{\alpha} - \frac {L}{2}\right) \| w ^ {t, k + 1} - w ^ {t, k} \| ^ {2},
$$
Now, we can prove the following result.
Proposition 7. Let  $\{\theta^t\}_{t\in \mathbb{N}}$  be a sequence generated by Algorithm 2. Then, for all  $t\in \mathbb{N}$  and  $0\leq k\leq K - 1$ , we have
$$
\| w ^ {t, k + 1} - \theta^ {\star} \| ^ {2} \leq (1 - \alpha F _ {w}) \| w ^ {t, k} - \theta^ {\star} \| ^ {2} + \frac {\alpha F _ {\theta} ^ {2}}{F _ {w}} \| \theta^ {t} - \theta^ {\star} \| ^ {2} - (2 - \alpha L) \| w ^ {t, k + 1} - w ^ {t, k} \| ^ {2}.
$$
In particular, when  $\alpha = 1 / L$ , we have
$$
\| w ^ {t, k + 1} - \theta^ {\star} \| ^ {2} \leq \left(1 - \frac {F _ {w}}{L}\right) \| w ^ {t, k} - \theta^ {\star} \| ^ {2} + \frac {F _ {\theta} ^ {2}}{L F _ {w}} \| \theta^ {t} - \theta^ {\star} \| ^ {2}.
$$
Proof. Let  $t \in \mathbb{N}$ . From the definition of steps of Algorithm 2, that is,  $w^{t,k+1} = w^{t,k} - \alpha \nabla_w H(\theta^t, w^{t,k})$ , for any  $0 \leq k \leq K-1$ , we obtain that
$$
\begin{array}{l} \left\| w ^ {t, k + 1} - \theta^ {\star} \right\| ^ {2} = \left\| \left(w ^ {t, k} - \theta^ {*}\right) - \alpha \nabla_ {w} H \left(\theta^ {t}, w ^ {t, k}\right) \right\| ^ {2} \\ = \left\| w ^ {t, k} - \theta^ {\star} \right\| ^ {2} + 2 \alpha \left\langle \nabla_ {w} H \left(\theta^ {t}, w ^ {t, k}\right), \theta^ {\star} - w ^ {t, k} \right\rangle + \left\| \alpha \nabla_ {w} H \left(\theta^ {t}, w ^ {t, k}\right) \right\| ^ {2} \\ = \| w ^ {t, k} - \theta^ {\star} \| ^ {2} + 2 \alpha \left\langle \nabla_ {w} H \left(\theta^ {t}, w ^ {t, k}\right), \theta^ {\star} - w ^ {t, k} \right\rangle + \| w ^ {t, k + 1} - w ^ {t, k} \| ^ {2}. \tag {12} \\ \end{array}
$$
Using the  $F_{w}$ -strong convexity of  $w \to H(\theta^t, w)$ , we have
$$
H \left(\theta^ {t}, \theta^ {\star}\right) \geq H \left(\theta^ {t}, w ^ {t, k}\right) + \left\langle \nabla_ {w} H \left(\theta^ {t}, w ^ {t, k}\right), \theta^ {\star} - w ^ {t, k} \right\rangle + \frac {F _ {w}}{2} \| w ^ {t, k} - \theta^ {\star} \| ^ {2}. \tag {13}
$$
Combining (12) and (13), we get:
$$
\begin{array}{l} \left\| w ^ {t, k + 1} - \theta^ {\star} \right\| ^ {2} \leq \left\| w ^ {t, k} - \theta^ {\star} \right\| ^ {2} + 2 \alpha \left(H \left(\theta^ {t}, \theta^ {\star}\right) - H \left(\theta^ {t}, w ^ {t, k}\right) - \frac {F _ {w}}{2} \left\| w ^ {t, k} - \theta^ {\star} \right\| ^ {2}\right) \\ + \left\| w ^ {t, k + 1} - w ^ {t, k} \right\| ^ {2} \\ = \left(1 - \alpha F _ {w}\right) \| w ^ {t, k} - \theta^ {\star} \| ^ {2} + 2 \alpha \left(H \left(\theta^ {t}, \theta^ {\star}\right) - H \left(\theta^ {t}, w ^ {t, k}\right)\right) + \| w ^ {t, k + 1} - w ^ {t, k} \| ^ {2}. \\ \end{array}
$$
Hence, from Proposition 6, we obtain
$$
\begin{array}{l} \left\| w ^ {t, k + 1} - \theta^ {\star} \right\| ^ {2} \leq \left(1 - \alpha F _ {w}\right) \left\| w ^ {t, k} - \theta^ {\star} \right\| ^ {2} + 2 \alpha \left(\frac {F _ {\theta} ^ {2}}{2 F _ {w}} \| \theta^ {t} - \theta^ {\star} \| ^ {2} - \left(\frac {1}{\alpha} - \frac {L}{2}\right) \| w ^ {t, k + 1} - w ^ {t, k} \| ^ {2}\right) \\ + \left\| w ^ {t, k + 1} - w ^ {t, k} \right\| ^ {2} \\ = \left(1 - \alpha F _ {w}\right) \| w ^ {t, k} - \theta^ {\star} \| ^ {2} + \frac {\alpha F _ {\theta} ^ {2}}{F _ {w}} \| \theta^ {t} - \theta^ {\star} \| ^ {2} - \left(2 - \alpha L\right) \| w ^ {t, k + 1} - w ^ {t, k} \| ^ {2}, \\ \end{array}
$$
which completes the first desired result.
Moreover, by specifically choosing the step-size  $\alpha = 1 / L$  we obtain that:
$$
\begin{array}{l} \| w ^ {t, k + 1} - \theta^ {\star} \| ^ {2} \leq \left(1 - \frac {F _ {w}}{L}\right) \| w ^ {t, k} - \theta^ {\star} \| ^ {2} + \frac {F _ {\theta} ^ {2}}{L F _ {w}} \| \theta^ {t} - \theta^ {\star} \| ^ {2} - \| w ^ {t, k + 1} - w ^ {t, k} \| ^ {2} \\ \leq \left(1 - \frac {F _ {w}}{L}\right) \| w ^ {t, k} - \theta^ {\star} \| ^ {2} + \frac {F _ {\theta} ^ {2}}{L F _ {w}} \| \theta^ {t} - \theta^ {\star} \| ^ {2}. \\ \end{array}
$$
This concludes the proof of this proposition.
Using these two results, we are ready to present the proof of Proposition 5.
Proof. Let  $t \in \mathbb{N}$ . From Proposition 7 (recall that  $\alpha = 1 / L$ ) and the fact that  $\theta^{t + 1} = w^{t,K}$ , we have
$$
\begin{array}{l} \| \theta^ {t + 1} - \theta^ {\star} \| ^ {2} = \| w ^ {t, K} - \theta^ {\star} \| ^ {2} \\ \leq (1 - \kappa) \| w ^ {t, K - 1} - \theta^ {\star} \| ^ {2} + \eta^ {2} \kappa \| \theta^ {t} - \theta^ {\star} \| ^ {2} \\ \leq (1 - \kappa) \left[ (1 - \kappa) \| w ^ {t, K - 2} - \theta^ {\star} \| ^ {2} + \eta^ {2} \kappa \| \theta^ {t} - \theta^ {\star} \| ^ {2} \right] + \eta^ {2} \kappa \| \theta^ {t} - \theta^ {\star} \| ^ {2} \\ = (1 - \kappa) ^ {2} \| w ^ {t, K - 2} - \theta^ {\star} \| ^ {2} + \eta^ {2} \kappa (1 + (1 - \kappa)) \| \theta^ {t} - \theta^ {\star} \| ^ {2} \\ \leq \dots \\ \leq \left(1 - \kappa\right) ^ {K} \| w ^ {t, 0} - \theta^ {\star} \| ^ {2} + \eta^ {2} \kappa \sum_ {k = 0} ^ {K - 1} \left(1 - \kappa\right) ^ {k} \| \theta^ {t} - \theta^ {\star} \| ^ {2} \\ = (1 - \kappa) ^ {K} \| \theta^ {t} - \theta^ {\star} \| ^ {2} + \eta^ {2} \kappa \sum_ {k = 0} ^ {K - 1} (1 - \kappa) ^ {k} \| \theta^ {t} - \theta^ {\star} \| ^ {2}, \\ \end{array}
$$
where the last inequality follows from the fact that  $w^{t,0} = \theta^t$ . Because  $\kappa \in [0,1]$ , the geometric series on the right-hand side is convergent, and so we can write:
$$
\left(1 - \kappa\right) ^ {K} + \eta^ {2} \kappa \sum_ {k = 0} ^ {K - 1} \left(1 - \kappa\right) ^ {k} = \left(1 - \kappa\right) ^ {K} + \eta^ {2} \kappa \frac {1 - \left(1 - \kappa\right) ^ {K}}{1 - \left(1 - \kappa\right)} = \left(1 - \kappa\right) ^ {K} + \eta^ {2} \left(1 - \left(1 - \kappa\right) ^ {K}\right),
$$
which completes the desired result.
Now, following our discussion in Section 6.3 about examples of loss functions that satisfy our assumption, we recall that here we focus on the following loss
$$
H (\theta , w) = \frac {1}{2} \sum_ {s} d (s) \sum_ {a} \pi (a | s) \left(\mathbb {E} _ {s ^ {\prime}} \left[ r + \gamma \max _ {a ^ {\prime}} q (s ^ {\prime}, a ^ {\prime}, \theta) \right] - q (s, a, w)\right) ^ {2}.
$$
As mentioned in Section 6.3 all is left is to show that Lipschitzness of  $\nabla_w H(\theta, w)$  with respect to  $\theta$ .
Proposition 8. Assume that for any  $(s,a)$ , the function  $\theta \rightarrow q(s,a,\theta)$  is  $L_{q}(s,a)$ -Lipschitz. Then, there exists  $F_{\theta} > 0$  such that
Proof. First, we compute the gradient  $\nabla_w H$ :
$$
\nabla_ {w} H (\theta , w) = \sum_ {s} d (s) \sum_ {a} \pi (a | s) \left(q (s, a, w) - \mathbb {E} _ {s ^ {\prime}} \left[ r + \gamma \max _ {a ^ {\prime}} q (s ^ {\prime}, a ^ {\prime}, \theta) \right]\right) \nabla q (s, a, w).
$$
For the simplicity of the proof, we denote  $Q(\theta_1,\theta_2) = \max_{a'}q(s',a',\theta_2) - \max_{a'}q(s',a',\theta_1)$ . Hence
$$
\begin{array}{l} \| \nabla_ {w} H (\theta_ {1}, w) - \nabla_ {w} H (\theta_ {2}, w) \| = \gamma \left\| \sum_ {s} d (s) \sum_ {a} \pi (a | s) \left(\mathbb {E} _ {s ^ {\prime}} [ Q (\theta_ {1}, \theta_ {2}) ]\right) \nabla q (s, a, w) \right\| \\ \leq \gamma \sum_ {s} d (s) \sum_ {a} \pi (a | s) \left| \mathbb {E} _ {s ^ {\prime}} [ Q (\theta_ {1}, \theta_ {2}) ] \right| \| \nabla q (s, a, w) \|, \\ \end{array}
$$
where we first cancelled the common parts in both gradient terms and the inequality follows immediately from the Cauchy-Schwartz inequality. Now, by using Jensen's inequality on the expectation, we obtain that
$$
\left\| \nabla_ {w} H (\theta_ {1}, w) - \nabla_ {w} H (\theta_ {2}, w) \right\| \leq \gamma \sum_ {s} d (s) \sum_ {a} \pi (a | s) \mathbb {E} _ {s ^ {\prime}} \left[ | Q (\theta_ {1}, \theta_ {2}) | \right] \left\| \nabla q (s, a, w) \right\|.
$$
Moreover, without the loss of generality we can assume that  $\max_{a'} q(s', a', \theta_2) \geq \max_{a'} q(s', a', \theta_1)$ , which means that we can remove the absolute value and obtain the following
$$
\left\| \nabla_ {w} H (\theta_ {1}, w) - \nabla_ {w} H (\theta_ {2}, w) \right\| \leq \gamma \sum_ {s} d (s) \sum_ {a} \pi (a | s) \mathbb {E} _ {s ^ {\prime}} \left[ Q (\theta_ {1}, \theta_ {2}) \right] \left\| \nabla q (s, a, w) \right\|.
$$
Now, by using a simple property of the maximum operation, we obviously have that
$$
\max  _ {a ^ {\prime}} q \left(s ^ {\prime}, a ^ {\prime}, \theta_ {2}\right) - \max  _ {a ^ {\prime}} q \left(s ^ {\prime}, a ^ {\prime}, \theta_ {1}\right) \leq \max  _ {a ^ {\prime}} \left(q \left(s ^ {\prime}, a ^ {\prime}, \theta_ {2}\right) - q \left(s ^ {\prime}, a ^ {\prime}, \theta_ {1}\right)\right).
$$
Since the function  $\theta \to q(s', a', \theta)$  is Lipschitz, we have that
$$
q \left(s ^ {\prime}, a ^ {\prime}, \theta_ {2}\right) - q \left(s ^ {\prime}, a ^ {\prime}, \theta_ {1}\right) \leq L _ {q} \left(s ^ {\prime}, a ^ {\prime}\right) \| \theta_ {1} - \theta_ {2} \|.
$$
It is also well-known that for such functions the gradient is bounded and therefore  $\| \nabla q(s,a,w)\| \leq L_q(s,a)$ . Combining all these facts yield
$$
\begin{array}{l} \| \nabla_ {w} H (\theta_ {1}, w) - \nabla_ {w} H (\theta_ {2}, w) \| \leq \gamma \sum_ {s} d (s) \sum_ {a} \pi (a | s) \mathbb {E} _ {s ^ {\prime}} \left[ Q (\theta_ {1}, \theta_ {2}) \right] \| \nabla q (s, a, w) \| \\ \leq \gamma \sum_ {s} d (s) \sum_ {a} \pi (a | s) \mathbb {E} _ {s ^ {\prime}} \left[ \max _ {a ^ {\prime}} \left(q \left(s ^ {\prime}, a ^ {\prime}, \theta_ {2}\right) - q \left(s ^ {\prime}, a ^ {\prime}, \theta_ {1}\right)\right) \right] \| \nabla q (s, a, w) \| \\ \leq \gamma \sum_ {s} d (s) \sum_ {a} \pi (a | s) \mathbb {E} _ {s ^ {\prime}} \left[ \max  _ {a ^ {\prime}} L _ {q} (s ^ {\prime}, a ^ {\prime}) \| \theta_ {1} - \theta_ {2} \| \right] L _ {q} (s, a) \\ \leq \gamma \| \theta_ {1} - \theta_ {2} \| \mathbb {E} _ {s ^ {\prime}} \left[ \max  _ {a ^ {\prime}} L _ {q} (s ^ {\prime}, a ^ {\prime}) ^ {2} \right] \sum_ {s} d (s) \sum_ {a} \pi (a | s) \\ = \gamma \| \theta_ {1} - \theta_ {2} \| \mathbb {E} _ {s ^ {\prime}} \left[ \max  _ {a ^ {\prime}} L _ {q} (s ^ {\prime}, a ^ {\prime}) ^ {2} \right], \\ \end{array}
$$
where the last equality follows from the basic properties of the distribution matrix  $D$ . This proves the desired result.