SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Verification of Assumption 3. We now verify Assumption 3 that the gap between  $\pmb{\theta}_{0}^{*}$  and  $\pmb{\theta}^{*}$  can be bounded by the span of  $f_{0}(\pmb{\theta}_{0}^{*})$  and  $f(\pmb{\theta}^{*})$  with a constant  $c_{0}$ . In Figure 4, we show  $c_{0}$  as a function of the number of neurons in the hidden layer  $m$ .
It clearly indicates that constant  $c_{0}$  exists and decreases as the number of neurons grows larger.
Figure 4: Verification of Assumption 3 w.r.t the constant  $c_{0}$ .
We presented Neural-Q-Whittle, a Whittle index based Q-learning algorithm for RMAB with neural network function approximation. We proved that Neural-Q-Whittle achieves an  $\mathcal{O}(1 / k^{2 / 3})$  convergence rate, where  $k$  is the number of iterations when data are generated from a Markov chain and Q-function is approximated by a ReLU neural network. By viewing Neural-Q-Whittle as 2TSA and leveraging the Lyapunov drift method, we removed the projection step on parameter update of Q-learning with neural network function approximation. Extending the current framework to two-timescale Q-learning (i.e., the coupled iterates between Q-function values and Whittle indices) with general deep neural network approximation is our future work.
This work was supported in part by the National Science Foundation (NSF) grant RINGS-2148309, and was supported in part by funds from OUSD R&E, NIST, and industry partners as specified in the Resilient & Intelligent NextG Systems (RINGS) program. This work was also supported in part by the U.S. Army Research Office (ARO) grant W911NF-23-1-0072, and the U.S. Department of Energy (DOE) grant DE-EE0009341. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funding agencies.
[1] Jinane Abounadi, Dimitrib Bertsekas, and Vivek S Borkar. Learning algorithms for markov decision processes with average cost. SIAM Journal on Control and Optimization, 40(3):681-698, 2001.
[2] Joshua Achiam, Ethan Knight, and Pieter Abbeel. Towards characterizing divergence in deep q-learning. arXiv preprint arXiv:1903.08894, 2019.
[3] Konstantin E Avrachenkov and Vivek S Borkar. Whittle index based q-learning for restless bandits with average reward. Automatica, 139:110186, 2022.
[4] Dimitri Bertsekas and John N Tsitsiklis. Neuro-dynamic programming. Athena Scientific, 1996.
[5] Dimitris Bertsimas and José Niño-Mora. Restless Bandits, Linear Programming Relaxations, and A Primal-Dual Index Heuristic. Operations Research, 48(1):80-90, 2000.
[6] Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference on learning theory, pages 1691-1692. PMLR, 2018.
[7] Shalabh Bhatnagar, Richard S Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural Actor-Critic Algorithms. Automatica, 45(11):2471-2482, 2009.
[8] Arpita Biswas, Gaurav Aggarwal, Pradeep Varakantham, and Milind Tambe. Learn to intervene: An adaptive learning policy for restless bandits in application to preventive healthcare. In Proc. of IJCAI, 2021.
[9] Vivek S Borkar. Stochastic Approximation: A Dynamical Systems Viewpoint, volume 48. Springer, 2009.
[10] Vivek S Borkar and Karan Chadha. A reinforcement learning algorithm for restless bandits. In 2018 Indian Control Conference (ICC), pages 89-94. IEEE, 2018.
[11] Vivek S Borkar and Vijaymohan R Konda. The Actor-Critic Algorithm as Multi-Time-Scale Stochastic Approximation. Sadhana, 22(4):525-543, 1997.
[12] Vivek S Borkar, K Ravikumar, and Krishnakant Saboo. An index policy for dynamic pricing in cloud computing under price commitments. Applicationes Mathematicae, 44:215-245, 2017.
[13] Qi Cai, Zhuoran Yang, Jason D Lee, and Zhaoran Wang. Neural temporal difference and q learning provably converge to global optima. Mathematics of Operations Research, 2023.
[14] Zaiwei Chen, Siva Theja Maguluri, Sanjay Shakkottai, and Karthikeyan Shanmugam. Finite-Sample Analysis of Stochastic Approximation Using Smooth Convex Envelopes. arXiv e-prints, pages arXiv-2002, 2020.
[15] Zaiwei Chen, Sheng Zhang, Thinh T Doan, Siva Theja Maguluri, and John-Paul Clarke. Performance of Q-learning with Linear Function Approximation: Stability and Finite-Time Analysis. arXiv preprint arXiv:1905.11425, 2019.
[16] Wenhan Dai, Yi Gai, Bhaskar Krishnamachari, and Qing Zhao. The Non-Bayesian Restless Multi-Armed Bandit: A Case of Near-Logarithmic Regret. In Proc. of IEEE ICASSP, 2011.
[56] Peter Whittle. Restless Bandits: Activity Allocation in A Changing World. Journal of applied probability, pages 287-298, 1988.
[57] Guojun Xiong, Jian Li, and Rahul Singh. Reinforcement Learning Augmented Asymptotically Optimal Index Policies for Finite-Horizon Restless Bandits. In Proc. of AAAI, 2022.
[58] Guojun Xiong, Xudong Qin, Bin Li, Rahul Singh, and Jian Li. Index-aware Reinforcement Learning for Adaptive Video Streaming at the Wireless Edge. In Proc. of ACM MobiHoc, 2022.
[59] Guojun Xiong, Shufan Wang, and Jian Li. Learning infinite-horizon average-reward restless multi-action bandits via index awareness. Proc. of NeurIPS, 2022.
[60] Guojun Xiong, Shufan Wang, Jian Li, and Rahul Singh. Whittle index based q-learning for wireless edge caching with linear function approximation. arXiv preprint arXiv:2202.13187, 2022.
[61] Guojun Xiong, Shufan Wang, Gang Yan, and Jian Li. Reinforcement Learning for Dynamic Dimensioning of Cloud Caches: A Restless Bandit Approach. In Proc. of IEEE INFOCOM, 2022.
[62] Pan Xu and Quanquan Gu. A finite-time analysis of q-learning with neural network function approximation. In International Conference on Machine Learning, pages 10555-10565. PMLR, 2020.
[63] Zhuoran Yang, Yongxin Chen, Mingyi Hong, and Zhaoran Wang. Provably Global Convergence of Actor-Critic: A Case for Linear Quadratic Regulator with Ergodic Cost. In Proc. of NeurIPS, 2019.
[64] Zhe Yu, Yunjian Xu, and Lang Tong. Deadline Scheduling as Restless Bandits. IEEE Transactions on Automatic Control, 63(8):2343-2358, 2018.
[65] Shangtong Zhang, Yi Wan, Richard S Sutton, and Shimon Whiteson. Average-Reward Off-Policy Policy Evaluation with Function Approximation. arXiv preprint arXiv:2101.02808, 2021.
[66] Sheng Zhang, Zhe Zhang, and Siva Theja Maguluri. Finite Sample Analysis of Average-Reward TD Learning and  $Q$ -Learning. Proc. of NeurIPS, 2021.
[67] Shaofeng Zou, Tengyu Xu, and Yingbin Liang. Finite-sample analysis for sarsa with linear function approximation. Advances in neural information processing systems, 32, 2019.
Online Restless Bandits. The online RMAB setting, where the underlying MDPs are unknown, has been gaining attention, e.g., [16, 33, 34, 50, 42, 26]. However, these methods do not exploit the special structure available in the problem and contend directly with an extremely high dimensional state-action space yielding the algorithms to be too slow to be useful. Recently, RL based algorithms have been developed [10, 23, 53, 8, 28, 57, 59, 61, 58, 3], to explore the problem structure through index policies. For instance, [23] proposed a Q-learning algorithm for Whittle index under the discounted setting, which lacks of convergence guarantees. [8] approximated Whittle index using the difference of  $Q(s,1) - Q(s,0)$  for any state  $s$ , which is not guaranteed to converge to the true Whittle index in general scenarios. To our best knowledge, the Q-Whittle in (4)-(5) proposed by [3] is the first algorithm with a rigorous asymptotic analysis. Therefore, [23, 3, 8, 28] lacked finite-time performance analysis and multi-timescale stochastic approximation algorithms usually suffer from slow convergence.
[53, 57] designed model-based low-complexity policy but is constrained to either a specific Markovian model or depends on a simulator for a finite-horizon setting which cannot be directly applied here. Latter on, [60] showed the finite-time convergence performance under the Q-Whittle setting of [3] with linear function approximation. However, the underlying assumption in [3, 60] is that data samples are drawn i.i.d per iteration. This is often not the case in practice since data samples of Q-learning are drawn according to the underlying Markov decision process. Till now, the finite-time convergence rate of Q-Whittle under the more challenging Markovian setting remains to be an open problem. Though [43] proposed a novel DQN method and applied it to Whittle index learning, it lacks of theoretical convergence analysis. To our best knowledge, our work is the first to study low-complexity model-free Q-learning for RMAB with neural network function approximation and provide a finite-time performance guarantee.
Two-Timescale Stochastic Approximation. The theoretical understanding of average-reward reinforcement learning (RL) methods is limited. Most existing results focus on asymptotic convergence [51, 1, 52, 65], or finite-time performance guarantee for discounted Q-learning [15, 46, 14]. However, the analysis of average-reward RL algorithms is known to be more challenging than their discounted-reward counterparts [66, 55]. In particular, our Neural-Q-Whittle follows the 2TSA scheme [11, 30, 7]. The standard technique for analyzing 2TSA is via the ODE method to prove asymptotic convergence [9]. Building off the importance of asymptotic results, recent years have witnessed a focus shifted to non-asymptotic, finite-time analysis of 2TSA [24, 21, 19, 63]. The closest work is [19], which characterized the convergence rate for a general non-linear 2TSA with i.i.d. noise. We generalize this result to provide a finite-time analysis of our Neural-Q-Whittle with Markovian noise. In addition, existing finite-time analysis, e.g., sample complexity [66] and regret [55] of Q-learning with average reward focus on a single-timescale SA, and hence cannot be directly applied to our Neural-Q-Whittle. Finally, existing Q-learning with linear function approximation [37, 6, 67] and neural network function approximation [13, 62] requires an additional projection step onto a bounded set related to the unknown stationary distribution of the underlying MDPs, or focuses on a single-timescale SA [15].
B Review on Whittle Index Policy
Whittle index policy addresses the intractable issue of RMAB through decomposition. In each round  $t$ , it first calculates the Whittle index for each arm  $n$  independently only based on its current state  $s_n(t)$ , and then the Whittle index policy simply selects the  $K$  arms with the highest indices to activate. Following Whittle's approach[56], we can consider a system with only one arm due to the decomposition, and the Lagrangian is expressed as
$$
L (\pi , \lambda) = \lim  _ {T \rightarrow \infty} \inf  _ {\pi} \frac {1}{T} \mathbb {E} _ {\pi} \sum_ {t = 1} ^ {T} \left\{r (t) + \lambda (1 - a (t)) \right\}, \tag {25}
$$
where  $\lambda$  is the Lagrangian multiplier (or the subsidy for selecting passive action). For a particular  $\lambda$ , the optimal activation policy can be expressed by a set of states in which it would activate this arm, which is denoted  $D(\lambda)$ .
Definition 3 (Indexiability). We denote  $D(\lambda)$  as the set of states  $S$  for which the optimal action for the arm is to choose a passive action, i.e.,  $A = 0$ . Then the arm is said to be indexable if  $D(\lambda)$  increases with  $\lambda$ , i.e., if  $\lambda > \lambda'$ , then  $D(\lambda) \supseteq D(\lambda')$ .
Following the indexability property, the Whittle index in a particular state  $S$  is defined as follows.
Definition 4 (Whittle Index). The Whittle index in state  $S$  for the indexable arm is the smallest value of the Lagrangian multiplier  $\lambda$  such that the optimal policy at state  $S$  is indifferent towards actions  $A = 0$  and  $A = 1$ . We denote such a Whittle index as  $\lambda(S)$  satisfying  $\lambda(S) := \inf_{\lambda \geq 0} \{S \in D(\lambda)\}$ .
Definition 5 (Whittle index policy). Whittle index policy is a controlled policy which activates the  $K$  arms with the highest whittle index  $\lambda_{i}(S_{i}(t))$  at each time slot  $t$ .
C Proof of Lemmas for "Step 2: Convergence Rate of  $\hat{M}(\theta_k, \lambda_k)$  in (16)"
$$
f _ {0} (\boldsymbol {\theta}; \boldsymbol {\phi} (s, a)) = \frac {1}{\sqrt {m}} \sum_ {r = 1} ^ {m} b _ {r} \mathbb {1} \left\{\mathbf {w} _ {r, 0} ^ {\intercal} \boldsymbol {\phi} (s, a) > 0 \right\} \mathbf {w} _ {r} ^ {\intercal} \boldsymbol {\phi} (s, a).
$$
Thus we denote  $\nabla_{\pmb{\theta}}f_0(\pmb {\theta};\pmb {\phi}(s,a))$  as
$$
\begin{array}{l} \nabla_ {\boldsymbol {\theta}} f _ {0} (\boldsymbol {\theta}; \boldsymbol {\phi} (s, a)) := \left[ \frac {1}{\sqrt {m}} b _ {1} \mathbb {1} \{\mathbf {w} _ {1, 0} ^ {\intercal} \boldsymbol {\phi} (s, a) > 0 \} \boldsymbol {\phi} (s, a) ^ {\intercal}, \dots , \right. \\ \left. \frac {1}{\sqrt {m}} b _ {m} \mathbb {1} \left\{\mathbf {w} _ {m, 0} ^ {\top} \boldsymbol {\phi} (s, a) > 0 \right\} \boldsymbol {\phi} (s, a) ^ {\top} \right] ^ {\top}. \tag {26} \\ \end{array}
$$
Since  $\| \pmb {\phi}(s,a)\| \leq 1,\forall s\in \mathcal{S},a\in \mathcal{A}$  and the fact that  $b_{r},\forall r\in [m]$  is uniformly initialized as 1 and -1, we have  $\| \nabla_{\pmb{\theta}}f_0(\pmb {\theta};\pmb {\phi}(s,a))\| \leq 1$
Therefore, we have the following inequality for any parameter pairs  $(\pmb{\theta}_1, \lambda_1)$  and  $(\pmb{\theta}_2, \lambda_2)$  with  $X = (s, a, s') \in \mathcal{X}$ ,
$$
\begin{array}{l} \left\| h _ {0} \left(X, \boldsymbol {\theta} _ {1}, \lambda_ {1}\right) - h _ {0} \left(X, \boldsymbol {\theta} _ {2}, \lambda_ {2}\right) \right\| \\ = \left\| \nabla_ {\boldsymbol {\theta}} f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, a)\right) \left[ r (s, a) + (1 - a) \lambda_ {1} - I _ {0} \left(\boldsymbol {\theta} _ {1}\right) + \max  _ {a _ {1}} f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} \left(s ^ {\prime}, a _ {1}\right)\right) - f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, a)\right) \right] \right. \\ \left. - \nabla_ {\boldsymbol {\theta}} f _ {0} \left(\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} (s, a)\right) \left[ r (s, a) + (1 - a) \lambda_ {2} - I _ {0} \left(\boldsymbol {\theta} _ {2}\right) + \max  _ {a _ {2}} f _ {0} \left(\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} \left(s ^ {\prime}, a _ {2}\right)\right) - f _ {0} \left(\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} (s, a)\right) \right] \right\| \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a _ {1})} {=} \left\| \nabla_ {\boldsymbol {\theta}} f _ {0} (\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, a)) \left[ (1 - a) (\lambda_ {1} - \lambda_ {2}) + I _ {0} (\boldsymbol {\theta} _ {2}) - I _ {0} (\boldsymbol {\theta} _ {1}) + f _ {0} (\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} (s, a)) - f _ {0} (\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, a)) \right. \right. \\ \left. \left. + \max  _ {a _ {1}} \left(f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} \left(s ^ {\prime}, a _ {1}\right)\right) - \max  _ {a _ {2}} f _ {0} \left(\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} \left(s ^ {\prime}, a _ {2}\right)\right) \right] \right\rvert   \right\rvert \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a _ {2})} {\leq} \| (1 - a) (\lambda_ {1} - \lambda_ {2}) \| + \| f _ {0} (\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} (s, a)) - f _ {0} (\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, a)) \| \\ + \left\| \frac {1}{2 S} \sum_ {\tilde {s} \in S} f _ {0} (\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} (\tilde {s}, 0)) - f _ {0} (\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (\tilde {s}, 0)) + f _ {0} (\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} (\tilde {s}, 1)) - f _ {0} (\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (\tilde {s}, 1)) \right\| \\ + \left\| \right. \max  _ {a _ {1}} \left( \right.f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} \left(s ^ {\prime}, a _ {1}\right)\right) - \max  _ {a _ {2}} f _ {0} \left(\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} \left(s ^ {\prime}, a _ {2}\right)\right)\left. \right\| \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(a _ {3})} {\leq} \| (1 - a) (\lambda_ {1} - \lambda_ {2}) \| + \| \nabla_ {\boldsymbol {\theta}} f _ {0} (\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, a)) (\boldsymbol {\theta} _ {2} - \boldsymbol {\theta} _ {1}) \| \\ + \left\| \frac {1}{2 S} \sum_ {\tilde {s} \in \mathcal {S}} \nabla_ {\boldsymbol {\theta}} f _ {0} (\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (\tilde {s}, 0)) (\boldsymbol {\theta} _ {2} - \boldsymbol {\theta} _ {1}) + \nabla_ {\boldsymbol {\theta}} f _ {0} (\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (\tilde {s}, 1)) (\boldsymbol {\theta} _ {2} - \boldsymbol {\theta} _ {1}) \right\| \\ + \left\| \right. \max  _ {a _ {1}} \left( \right.f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} \left(s ^ {\prime}, a _ {1}\right)\right) - \max  _ {a _ {2}} f _ {0} \left(\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} \left(s ^ {\prime}, a _ {2}\right)\right)\left. \right\| \\ \end{array}
$$
$$
\stackrel {(a _ {4})} {\leq} \| (\lambda_ {1} - \lambda_ {2}) \| + 2 \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \| + \left\| \max  _ {a _ {1}} \left(f _ {0} (\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s ^ {\prime}, a _ {1})) - \max  _ {a _ {2}} f _ {0} (\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} (s ^ {\prime}, a _ {2})) \right\| \right.
$$
$$
\begin{array}{l} \stackrel {(a _ {5})} {\leq} \| (\lambda_ {1} - \lambda_ {2}) \| + 2 \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \| + \left\| \max  _ {a ^ {\prime}} f _ {0} (\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s ^ {\prime}, a ^ {\prime})) - f _ {0} (\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} (s ^ {\prime}, a ^ {\prime})) \right\| \\ \stackrel {(a _ {6})} {\leq} \| (\lambda_ {1} - \lambda_ {2}) \| + 3 \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \|. \\ \end{array}
$$
Specifically,  $(a_{1})$  holds due to the fact that  $\nabla_{\pmb{\theta}}f_0(\pmb {\theta}_1;\pmb {\phi}(s,a)) = \nabla_{\pmb{\theta}}f_0(\pmb {\theta}_2;\pmb {\phi}(s,a))$  as in (26). Since
$$
I _ {0} (\boldsymbol {\theta} _ {k}) = \frac {1}{2 S} \sum_ {\tilde {s} \in \mathcal {S}} \left[ f _ {0} (\boldsymbol {\theta} _ {k}; \boldsymbol {\phi} (\tilde {s}, 0)) + f _ {0} (\boldsymbol {\theta} _ {k}; \boldsymbol {\phi} (\tilde {s}, 1)) \right],
$$
$(a_{2})$  is due to the fact that  $\| \pmb {x} + \pmb {y}\| \leq \| \pmb {x}\| +\| \pmb {y}\|$ $\forall \pmb {x},\pmb {y}\in \mathbb{R}^{md}$  and  $\| \pmb {x}\cdot \pmb {y}\| \leq \| \pmb {x}\| \cdot \| \pmb {y}\|$ $\forall x,y\in \mathbb{R}^{md}$  and  $\| \phi (s,a)\| \leq 1,\forall s,a.$ $(a_3)$  holds since
$$
f _ {0} \left(\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} (s, a)\right) - f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, a)\right) = \nabla_ {\boldsymbol {\theta}} f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, a)\right) \left(\boldsymbol {\theta} _ {2} - \boldsymbol {\theta} _ {1}\right), \forall s \in \mathcal {S}, a \in \mathcal {A}. \tag {27}
$$
$(a_4)$  holds for the same reason as  $(a_2)$ .  $(a_5)$  is due to the fact that
$$
\begin{array}{l} \| \max  _ {a ^ {\prime}} f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} \left(s ^ {\prime}, a ^ {\prime}\right)\right) - f _ {0} \left(\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} \left(s ^ {\prime}, a ^ {\prime}\right)\right) \| \leq \max  \left(\left\| \max  _ {a ^ {\prime}} f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} \left(s ^ {\prime}, a ^ {\prime}\right)\right) - f _ {0} \left(\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} \left(s ^ {\prime}, a ^ {\prime}\right)\right) \right\|\right), \\ \left. \left\| \min  _ {a ^ {\prime}} f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} \left(s ^ {\prime}, a ^ {\prime}\right)\right) - f _ {0} \left(\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} \left(s ^ {\prime}, a ^ {\prime}\right)\right) \right\|\right). \tag {28} \\ \end{array}
$$
$(a_{6})$  holds for the same reason as  $(a_{3})$  and  $(a_{4})$
Proof. Since  $g_0(\cdot)$  is irrelevant with  $X$  and  $\lambda$ , in the following, we write  $g_0(X,\pmb{\theta},\lambda)$  with  $g_0(\lambda)$  interchangeably. For any  $\pmb{\theta}_1 \in \mathbb{R}^{md}$  and  $\pmb{\theta}_2 \in \mathbb{R}^{md}$ , we have
$$
\begin{array}{l} \left\| g _ {0} \left(\boldsymbol {\theta} _ {1}\right) - g _ {0} \left(\boldsymbol {\theta} _ {2}\right) \right\| \\ = \| f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, 1)\right) - f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, 0)\right) - f _ {0} \left(\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} (s, 1)\right) + f _ {0} \left(\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} (s, 0)\right) \| \\ \leq \left\| f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, 1)\right) - f _ {0} \left(\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} (s, 1)\right) \right\| + \left\| f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, 0)\right) - f _ {0} \left(\boldsymbol {\theta} _ {2}; \boldsymbol {\phi} (s, 0)\right) \right\| \\ = \| \nabla_ {\boldsymbol {\theta}} f _ {0} (\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, 1)) (\boldsymbol {\theta} _ {2} - \boldsymbol {\theta} _ {1}) \| + \| \nabla_ {\boldsymbol {\theta}} f _ {0} (\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, 0)) (\boldsymbol {\theta} _ {2} - \boldsymbol {\theta} _ {1}) \| \\ \leq \left\| \nabla_ {\boldsymbol {\theta}} f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, 1)\right) \right\| \cdot \left\| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \right\| + \left\| \nabla_ {\boldsymbol {\theta}} f _ {0} \left(\boldsymbol {\theta} _ {1}; \boldsymbol {\phi} (s, 0)\right) \right\| \cdot \left\| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \right\| \\ \leq 2 \| \boldsymbol {\theta} _ {1} - \boldsymbol {\theta} _ {2} \|, \\ \end{array}
$$
where the first inequality is due to the fact that  $\| \pmb{x} + \pmb{y}\| \leq \| \pmb{x}\| +\| \pmb{y}\|$ ,  $\forall \pmb{x},\pmb{y}\in \mathbb{R}^{md}$ , the second inequality holds due to  $\| \pmb{x}\cdot \pmb{y}\| \leq \| \pmb{x}\| \cdot \| \pmb{y}\|$ ,  $\forall \pmb{x},\pmb{y}\in \mathbb{R}^{md}$ , and the last inequality holds since  $\| \nabla_{\pmb{\theta}}f_0(\pmb{\theta}_1;\pmb {\phi}(s,a))\| \leq 1,\forall s\in \mathcal{S},a\in \mathcal{A}$ .
Proof. For any  $\pmb{\theta}_{1} \in \mathbb{R}^{md}$  and  $\pmb{\theta}_{2} \in \mathbb{R}^{md}$ , we have