SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The root  $(\pmb{\theta}^{*},\lambda^{*})$  of the nonlinear 2TSA (9) can be established by using the ODE method following the solution of suitably defined differential equations [9, 49, 3, 21, 19, 20], i.e.,  $\dot{\pmb{\theta}} = H(\pmb {\theta},\lambda)$ $\dot{\lambda} = \frac{\eta}{\alpha} G(\pmb {\theta},\lambda)$ , where a fixed stepsize is assumed for ease of expression at this moment.
As inspired by [20], the finite-time analysis of such a nonlinear 2TSA boils down to the choice of two step sizes  $\{\alpha_k, \eta_k, \forall k\}$  and a Lyapunov function that couples the two iterates in (9). To this end, we first define the following two error terms:
$$
\tilde {\boldsymbol {\theta}} _ {k} = \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} ^ {*}, \quad \tilde {\lambda} _ {k} = \lambda_ {k} - y (\boldsymbol {\theta} _ {k}), \tag {14}
$$
which characterize the coupling between  $\pmb{\theta}_k$  and  $\lambda_{k}$ . If  $\tilde{\pmb{\theta}}_k$  and  $\tilde{\lambda}_k$  go to zero simultaneously, the convergence of  $(\pmb{\theta}_k,\lambda_k)$  to  $(\pmb{\theta}^{*},\lambda^{*})$  can be established. Thus, to prove the convergence of  $(\pmb{\theta}_k,\lambda_k)$  of the nonlinear 2TSA in (9) to its true value  $(\pmb{\theta}^{*},\lambda^{*})$ , we can equivalently study the convergence of  $(\tilde{\pmb{\theta}}_k,\tilde{\lambda}_k)$  by providing the finite-time analysis for the mean squared error generated by (9). To couple the fast and slow iterates, we define the following weighted Lyapunov function
$$
M \left(\boldsymbol {\theta} _ {k}, \lambda_ {k}\right) := \frac {\eta_ {k}}{\alpha_ {k}} \| \tilde {\boldsymbol {\theta}} _ {k} \| ^ {2} + \| \tilde {\lambda} _ {k} \| ^ {2} = \frac {\eta_ {k}}{\alpha_ {k}} \| \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} ^ {*} \| ^ {2} + \| \lambda_ {k} - y \left(\boldsymbol {\theta} _ {k}\right) \| ^ {2}, \tag {15}
$$
where  $\|\cdot\|$  stands for the Euclidean norm for vectors throughout the paper. It is clear that the Lyapunov function  $M(\pmb{\theta}_k, \lambda_k)$  combines the updates of  $\pmb{\theta}$  and  $\lambda$  with respect to the true neural network function  $f(\pmb{\theta}; \pmb{\phi}(s, a))$  in (6).
To this end, our goal turns to characterize finite-time convergence of  $\mathbb{E}[M(\pmb{\theta}_k,\lambda_k)]$ . However, it is challenging to directly finding the global optimum of the corresponding nonlinear equations due to the nonlinear parameterization of Q-function in Neural-Q-Whittle. In addition, the operators  $h(\cdot),g(\cdot)$  and  $y(\cdot)$  in (10), (11) and (13) directly relate with the convoluted neural network function  $f(\pmb{\theta};\pmb{\phi}(s,a))$  in (6), which hinders us to characterize the smoothness properties of theses operators. Such properties are often required for the analysis of stochastic approximation [15, 19, 21].
To mitigate this, (Step 1) we instead approximate the true neural network function  $f(\pmb{\theta}, \pmb{\phi}(s, a))$  with a collection of local linearization  $f_0(\pmb{\theta}; \pmb{\phi}(s, a))$  at the initial point  $\pmb{\theta}_0$ . Based on the surrogate stationary point  $\pmb{\theta}_0^*$  of  $f_0(\pmb{\theta}; \pmb{\phi}(s, a))$ , we correspondingly define a modified Lyapunov function  $\hat{M}(\pmb{\theta}_k, \lambda_k)$  combining updates of  $\pmb{\theta}$  and  $\lambda$  with respect to such local linearization. Specifically, we have
$$
\hat {M} \left(\boldsymbol {\theta} _ {k}, \lambda_ {k}\right) := \frac {\eta_ {k}}{\alpha_ {k}} \left\| \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} _ {0} ^ {*} \right\| ^ {2} + \left\| \lambda_ {k} - y _ {0} \left(\boldsymbol {\theta} _ {k}\right) \right\| ^ {2}, \tag {16}
$$
where  $y_0(\cdot)$  is in the same expression as  $y(\cdot)$  in (13) by replacing  $f(\cdot)$  with  $f_0(\cdot)$ , and we will describe this in details below. (Step 2) We then study the convergence rate of the nonlinear 2TSA using this modified Lyapunov function under general conditions. (Step 3) Finally, since the two coupled parameters  $\theta$  and  $\lambda$  in (9) are updated with respect to the true neural network function  $f(\pmb{\theta};\pmb{\phi}(s,a))$  in (6) in Neural-Q-Whittle, while we characterize their convergence using the approximated neural network function in Step 2. Hence, this further requires us to characterize the approximation errors. We visualize the above three steps in Figure 1 and provide a proof sketch in Section 4.3. Combining them together gives rise to our main theoretical results on the finite-time performance of Neural-Q-Whittle, which is formally stated in the following theorem.
Theorem 1. Consider iterates  $\{\pmb{\theta}_k\}$  and  $\{\lambda_k\}$  generated by Neural-Q-Whittle in (7) and (8). Given  $\alpha_{k} = \frac{\alpha_{0}}{(k + 1)}$ $\eta_{k} = \frac{\eta_{0}}{(k + 1)^{4 / 3}}$  we have for  $\forall k\geq \tau$
$$
\begin{array}{l} \mathbb {E} [ M (\pmb {\theta} _ {k + 1}, \lambda_ {k + 1}) | \mathcal {F} _ {k - \tau} ] \leq \frac {2 \tau^ {2} \mathbb {E} [ \hat {M} (\pmb {\theta} _ {\tau} , \lambda_ {\tau}) ]}{(k + 1) ^ {2}} + \frac {1 2 0 0 \alpha_ {0} ^ {3}}{\eta_ {0}} \frac {(C _ {1} + \| \hat {\pmb {\theta}} _ {0} \|) ^ {2} + (2 C _ {1} + \| \hat {\lambda} _ {0} \|) ^ {2}}{(k + 1) ^ {2 / 3}} \\ + \frac {2 \eta_ {0} c _ {0} ^ {2}}{\alpha_ {0} (1 - \kappa) ^ {2}} \| s p a n \left(\Pi_ {\mathcal {F}} f \left(\boldsymbol {\theta} ^ {*}\right) - f \left(\boldsymbol {\theta} ^ {*}\right)\right) \| ^ {2} + \left(\frac {2}{(k + 1) ^ {2 / 3}} + 2\right) \mathcal {O} \left(\frac {c _ {1} ^ {3} \left(\| \boldsymbol {\theta} _ {0} \| + | \lambda_ {0} | + 1\right) ^ {3}}{m ^ {1 / 2}}\right), \tag {17} \\ \end{array}
$$
where  $C_1 \coloneqq c_1(\|\pmb{\theta}_0\| + \|\lambda_0\| + 1)$  with  $c_1$  being a proper chosen constant,  $c_0$  is a constant defined in Assumption 3,  $\tau$  is the mixing time defined in (22), span denotes for the span semi-norm [47], and  $\Pi_{\mathcal{F}}$  represents the projection to the set of  $\mathcal{F}$  containing all possible  $f_0(\pmb{\theta}; \pmb{\phi}(s, a))$  in (18).
The first term on the right hand side (17) corresponds to the bias compared to the Lyapunov function at the mixing time  $\tau$ , which goes to zero at a rate of  $\mathcal{O}(1 / k^2)$ . The second term corresponds to the accumulated estimation error of the nonlinear 2TSA due to Markovian noise, which vanishes at the rate  $\mathcal{O}(1 / k^{2 / 3})$ . Hence it dominates the overall convergence rate in (17). The third term captures the distance between the optimal solution  $(\pmb{\theta}^{*},\lambda^{*})$  to the true neural network function  $f(\pmb{\theta}_k;\pmb {\phi}(s,a))$  in
Figure 1: Neural-Q-Whittle operates w.r.t. true neural function  $f(\cdot)$  with its finite-time performance given in Theorem 1 (indicated in dashed lines). Our proofs operate in three steps: (i) Step 1: Obtain local linearization  $f_0(\cdot)$  and define Lyapunov function  $\hat{M}(\cdot)$  w.r.t.  $f_0(\cdot)$ . (ii) Step 2: Characterize the finite-time performance w.r.t.  $\hat{M}(\cdot)$  using Lyapunov drift method. Since Neural-Q-Whittle is updated w.r.t.  $f(\cdot)$ , we need to characterize the gap between  $f(\cdot)$  and  $f_0(\cdot)$ . (iii) Step 3: Similarly, we characterize the approximation errors between  $M(\cdot)$  and  $\hat{M}(\cdot)$ .
(6) and the optimal one  $(\pmb{\theta}_0^*, y_0(\pmb{\theta}_0^*))$  with local linearization  $f_0(\pmb{\theta}_k; \pmb{\phi}(s, a))$  in (18), which quantifies the error when  $f(\pmb{\theta}^*)$  does not fall into the function class  $\mathcal{F}$ . The last term characterizes the distance between  $f(\pmb{\theta}_k; \pmb{\phi}(s, a))$  and  $f_0(\pmb{\theta}_k; \pmb{\phi}(s, a))$  with any  $\pmb{\theta}_k$ . Both terms diminish as  $m \to \infty$ . Theorem 1 implies the convergence to the optimal value  $(\pmb{\theta}^*, \lambda^*)$  is bounded by the approximation error, which will diminish to zero as representation power of  $f_0(\pmb{\theta}_k; \pmb{\phi}(s, a))$  increases when  $m \to \infty$ . Finally, we note that the right hand side (17) ends up in  $\mathcal{O}(1 / k^2) + \mathcal{O}(1 / k^{2/3}) + c$ , where  $c$  is a constant and its value goes to 0 as  $m \to \infty$ . This indicates the error bounds of linearization with the original neural network functions are controlled by the overparameterization value of  $m$ . Need to mention that a constant step size will result in a non-vanishing accumulated error as in [15].
Remark 2. A finite-time analysis of nonlinear 2TSA was presented in [39]. However, [39] required a stability condition that  $\lim_{k\to \infty}(\pmb{\theta}_k,\lambda_k) = (\pmb{\theta}^*,\lambda^*)$ , and both  $h$  and  $g$  are locally approximated as linear functions. [19, 60] relaxed these conditions and provided a finite-time analysis under i.i.d. noise. These results were later extended to Markovian noise [20] under the assumption that  $H$  function is strongly monotone in  $\pmb{\theta}$  and  $G$  function is strongly monotone in  $\lambda$ . Since [20] leveraged the techniques in [19], it needed to explicitly characterize the covariance between the error caused by Markovian noise and the parameters' residual error in (14), leading to the convergence analysis much more intrinsic. [15] exploited the mixing time to avoid the covariance between the error caused by Markovian noise and the parameters' residual error; however, it only considered the single timescale  $Q$ -learning with linear function approximation. Though our Neural-Q-Whittle can be rewritten as a nonlinear 2TSA, the nonlinear parameterization of  $Q$ -function caused by the neural network function approximation makes the aforementioned analysis not directly applicable to ours and requires additional characterization as highlighted in Figure 1. The explicit characterization of approximation errors further distinguish our work.
In this section, we sketch the proofs of the three steps shown in Figure 1 as required for Theorem 1.
4.3.1 Step 1: Approximated Solution of Neural-Q-Whittle
We first approximate the optimal solution by projecting the Q-function in (6) to some function classes parameterized by  $\pmb{\theta}$ . The common choice of the projected function classes is the local linearization of  $f(\pmb{\theta};\pmb{\phi}(s,a))$  at the initial point  $\pmb{\theta}_0$  [13, 62], i.e.,  $\mathcal{F} \coloneqq \{f_0(\pmb{\theta};\pmb{\phi}(s,a)), \forall \pmb{\theta} \in \Theta\}$ , where
$$
f _ {0} (\boldsymbol {\theta}; \boldsymbol {\phi} (s, a)) = \frac {1}{\sqrt {m}} \sum_ {r = 1} ^ {m} b _ {r} \mathbb {1} \left\{\mathbf {w} _ {r, 0} ^ {\intercal} \boldsymbol {\phi} (s, a) > 0 \right\} \mathbf {w} _ {r} ^ {\intercal} \boldsymbol {\phi} (s, a). \tag {18}
$$
Then, we define the approximate stationary point  $\pmb{\theta}_0^*$  with respect to  $f_0(\pmb {\theta};\pmb {\phi}(s,a))$  as follows.
Definition 1. [[13, 62]] A point  $\pmb{\theta}_0^* \in \Theta$  is said to be the approximate stationary point of Algorithm 1 if for all feasible  $\pmb{\theta} \in \Theta$  it holds that  $\mathbb{E}_{\mu, \pi, \mathcal{P}}[(\Delta_0 \cdot \nabla_\pmb{\theta} f_0(\pmb{\theta}; \pmb{\phi}(s, a)))^\top (\pmb{\theta} - \pmb{\theta}_0^*)] \geq 0, \forall \pmb{\theta} \in \Theta$ , with  $\Delta_0 := [r(s, a) + (1 - a)\lambda^* - I_0(\pmb{\theta}) + \max_{a'} f_0(\pmb{\theta}; \pmb{\phi}(s', a)) - f_0(\pmb{\theta}; \pmb{\phi}(s, a))]$ , where  $I_0(\pmb{\theta}) = \frac{1}{2S}\sum_{s \in S}[f_0(\pmb{\theta}; \pmb{\phi}(s, 0)) + f_0(\pmb{\theta}; \pmb{\phi}(s, 1))]$ .
Though there is a gap between the true neural function (6) and the approximated local linearized function (18), the gap diminishes as the width of neural network i.e.,  $m$ , becomes large [13, 62].
With the approximated stationary point  $\theta_0^*$ , we can redefine the two error terms in (14) as
$$
\hat {\boldsymbol {\theta}} _ {k} = \boldsymbol {\theta} _ {k} - \boldsymbol {\theta} _ {0} ^ {*}, \quad \hat {\lambda} _ {k} = \lambda_ {k} - y _ {0} (\boldsymbol {\theta} _ {k}), \tag {19}
$$
using which we correspondingly define a modified Lyapunov function  $\hat{M} (\pmb {\theta}_k,\lambda_k)$  in (16), where
$$
y _ {0} (\boldsymbol {\theta}) = r (s, 1) + \sum_ {s ^ {\prime}} p \left(s ^ {\prime} \mid s, 1\right) \max  _ {a} f _ {0} (\boldsymbol {\theta}; \phi \left(s ^ {\prime}, a\right)) - r (s, 0) - \sum_ {s ^ {\prime}} p \left(s ^ {\prime} \mid s, 0\right) \max  _ {a} f _ {0} (\boldsymbol {\theta}; \phi \left(s ^ {\prime}, a\right)). \tag {20}
$$
4.3.2 Step 2: Convergence Rate of  $\hat{M} (\theta_k,\lambda_k)$  in (16)
Since we approximate the true neural network function  $f(\pmb{\theta}; \pmb{\phi}(s, a))$  in (6) with the local linearized function  $f_0(\pmb{\theta}; \pmb{\phi}(s, a))$  in (18), the operators  $h(\cdot)$  and  $g(\cdot)$  in (10)-(11) turn correspondingly to be
$$
h _ {0} \left(X _ {k}, \boldsymbol {\theta} _ {k}, \lambda_ {k}\right) = \nabla_ {\boldsymbol {\theta}} f _ {0} \left(\boldsymbol {\theta} _ {k}; \boldsymbol {\phi} \left(S _ {k}, A _ {k}\right)\right) \Delta_ {k, 0}, g _ {0} \left(\boldsymbol {\theta} _ {k}\right) := f _ {0} \left(\boldsymbol {\theta} _ {k}; \boldsymbol {\phi} (s, 1)\right) - f _ {0} \left(\boldsymbol {\theta} _ {k}; \boldsymbol {\phi} (s, 0)\right), \tag {21}
$$
with  $\Delta_{k,0} := r(S_k, A_k) + (1 - A_k)\lambda_k - I_0(\pmb{\theta}_k) + \max_a f_0(\pmb{\theta}_k; \pmb{\phi}(S_{k+1}, a)) - f_0(\pmb{\theta}_k; \pmb{\phi}(S_k, A_k))$ .
Before we present the finite-time error bound of the nonlinear 2TSA (9) under Markovian noise, we first discuss the mixing time of the Markov chain  $\{X_{k}\}$  and our assumptions.
Definition 2 (Mixing time [15]). For any  $\delta >0$ , define  $\tau_{\delta}$  as
$$
\tau_ {\delta} = \min  \left\{k \geq 1: \| \mathbb {E} \left[ h _ {0} \left(X _ {k}, \boldsymbol {\theta}, \lambda\right) \mid X _ {0} = x \right] - H _ {0} (\boldsymbol {\theta}, \lambda) \| \leq \delta \left(\| \boldsymbol {\theta} - \boldsymbol {\theta} _ {0} ^ {*} \| + \| \lambda - y _ {0} \left(\boldsymbol {\theta} _ {0} ^ {*}\right) \|\right) \right\}. \tag {22}
$$
Assumption 1. The Markov chain  $\{X_{k}\}$  is irreducible and aperiodic. Hence, there exists a unique stationary distribution  $\mu$  [32], and constants  $C > 0$  and  $\rho \in (0,1)$  such that  $d_{TV}(P(X_k|X_0 = x),\mu)\leq C\rho^k,\forall k\geq 0,x\in \mathcal{X},$  where  $d_{TV}(\cdot ,\cdot)$  is the total-variation (TV) distance [32].
Remark 3. Assumption 1 is often assumed to study the asymptotic convergence of stochastic approximation under Markovian noise [4, 9, 15].
Lemma 1. The function  $h_0(X, \pmb{\theta}, \lambda)$  defined in (21) is globally Lipschitz continuous w.r.t  $\pmb{\theta}$  and  $\lambda$  uniformly in  $X$ , i.e.,  $\| h_0(X, \pmb{\theta}_1, \lambda_1) - h_0(X, \pmb{\theta}_2, \lambda_2) \| \leq L_{h,1} \| \pmb{\theta}_1 - \pmb{\theta}_2 \| + L_{h,2} \| \lambda_1 - \lambda_2 \|, \forall X \in \mathcal{X}$ , and  $L_{h,1} = 3, h_{h,2} = 1$  are valid Lipschitz constants.
Lemma 2. The function  $g_0(\pmb{\theta})$  defined in (21) is linear and thus Lipschitz continuous in  $\pmb{\theta}$ , i.e.,  $\| g_0(\pmb{\theta}_1) - g_0(\pmb{\theta}_2) \| \leq L_g \| \pmb{\theta}_1 - \pmb{\theta}_2 \|$ , and  $L_g = 2$  is a valid Lipschitz constant.
Lemma 3. The function  $y_0(\pmb{\theta})$  defined in (20) is linear and thus Lipschitz continuous in  $\pmb{\theta}$ , i.e.,  $\| y_0(\pmb{\theta}_1) - y_0(\pmb{\theta}_2) \| \leq L_y \| \pmb{\theta}_1 - \pmb{\theta}_2 \|$ , and  $L_y = 2$  is a valid Lipschitz constant.
Remark 4. The Lipschitz continuity of  $h_0$  guarantees the existence of a solution  $\pmb{\theta}$  to the ODE  $\dot{\pmb{\theta}}$  for a fixed  $\lambda$ , while the Lipschitz continuity of  $g_0$  and  $y_0$  ensures the existence of a solution  $\lambda$  to the ODE  $\dot{\lambda}$  when  $\pmb{\theta}$  is fixed. These lemmas often serve as assumptions when proving the convergence rate for both linear and nonlinear 2TSA [31, 39, 18, 24, 19, 17, 27].
Lemma 4. For a fixed  $\lambda$ , there exists a constant  $\mu_1 > 0$  such that  $h_0(X,\pmb{\theta},\lambda)$  defined in (10) satisfies
$$
\mathbb {E} [ \hat {\boldsymbol {\theta}} ^ {\mathsf {T}} h _ {0} (X, \boldsymbol {\theta}, \lambda) ] \leq - \mu_ {1} \| \hat {\boldsymbol {\theta}} \| ^ {2}.
$$
For fixed  $\pmb{\theta}$ , there exists a constant  $\mu_2 > 0$  such that  $g_0(X,\pmb{\theta},\lambda)$  defined in (11) satisfies
$$
\mathbb {E} [ \hat {\lambda} g _ {0} (X, \boldsymbol {\theta}, \lambda) ] \leq - \mu_ {2} \| \hat {\lambda} \| ^ {2}.
$$
Remark 5. Lemma 4 guarantees the stability and uniqueness of the solution  $\pmb{\theta}$  to the ODE  $\dot{\pmb{\theta}}$  for a fixed  $\lambda$ , and the uniqueness of the solution  $\lambda$  to the ODE  $\dot{\lambda}$  for a fixed  $\pmb{\theta}$ . This assumption can be viewed as a relaxation of the stronger monotone property of nonlinear mappings [19, 15], since it is automatically satisfied if  $h$  and  $g$  are strong monotone as assumed in [19].
Lemma 5. Under Assumption 1 and Lemma 1, there exist constants  $C > 0$ ,  $\rho \in (0,1)$  and  $L = \max(3, \max_X h_0(X, \pmb{\theta}_0^*), y_0(\pmb{\theta}_0^*))$  such that
$$
\tau_ {\delta} \leq \frac {\log (1 / \delta) + \log (2 L C m d)}{\log (1 / \rho)}.
$$
Remark 6.  $\tau_{\delta}$  is equivalent to the mixing time of the underlying Markov chain satisfying  $\lim_{\delta \to 0} \delta \tau_{\delta} = 0$  [15]. For simplicity, we remove the subscript and denote it as  $\tau$ .
We now present the finite-time error bound for the Lyapunov function  $\hat{M} (\pmb {\theta}_k,\lambda_k)$  in (16).
Theorem 2. Consider iterates  $\{\pmb{\theta}_k\}$  and  $\{\lambda_k\}$  generated by Neural-Whittle in (7) and (8). Given Lemma 1-4,  $\alpha_{k} = \frac{\alpha_{0}}{(k + 1)}$ $\eta_{k} = \frac{\eta_{0}}{(k + 1)^{4 / 3}}$ $C_1\coloneqq c_1(\| \pmb {\theta}_0\| +\| \lambda_0\| +1)$  with a constant  $c_{1}$
$$
\begin{array}{l} \mathbb {E} [ \hat {M} (\boldsymbol {\theta} _ {k + 1}, \lambda_ {k + 1}) | \mathcal {F} _ {k - \tau} ] \leq \frac {\tau^ {2} \mathbb {E} [ \hat {M} (\boldsymbol {\theta} _ {\tau} , \lambda_ {\tau}) ]}{(k + 1) ^ {2}} + \frac {6 0 0 \alpha_ {0} ^ {3}}{\eta_ {0}} \frac {(C _ {1} + \| \hat {\boldsymbol {\theta}} _ {0} \|) ^ {2} + (2 C _ {1} + \| \hat {\lambda} _ {0} \|) ^ {2}}{(k + 1) ^ {2 / 3}} \\ + \frac {\mathcal {O} \left(c _ {1} ^ {3} \left(\| \boldsymbol {\theta} _ {0} \| + | \lambda_ {0} | + 1\right) ^ {3} m ^ {- 1 / 2}\right)}{(k + 1) ^ {2 / 3}}, \quad \forall k \geq \tau . \tag {23} \\ \end{array}
$$
4.3.3 Step 3: Approximation Error between  $M(\pmb{\theta}_k, \lambda_k)$  and  $\hat{M}(\pmb{\theta}_k, \lambda_k)$
Finally, we characterize the approximation error between Lyapunov functions  $M(\pmb{\theta}_k, \lambda_k)$  and  $\hat{M}(\pmb{\theta}_k, \lambda_k)$ . Since we are dealing with long-term average MDP, we assume that the total variation of the MDP is bounded [47].
Assumption 2. There exists  $0 < \kappa < 1$  such that  $\sup_{(s,a),(s',a')} \| p(\cdot | s, a) - p(\cdot | s', a') \|_{TV} = 2\kappa$ .
Hence, the Bellman operator is a span-contraction operator [47], i.e.,
$$
\operatorname {s p a n} \left(\mathcal {T} f _ {0} \left(\boldsymbol {\theta} _ {0} ^ {*}\right) - \mathcal {T} f \left(\boldsymbol {\theta} ^ {*}\right)\right) \leq \kappa \operatorname {s p a n} \left(f _ {0} \left(\boldsymbol {\theta} _ {0} ^ {*}\right) - f \left(\boldsymbol {\theta} ^ {*}\right)\right). \tag {24}
$$
Assumption 3.  $\| \pmb{\theta}_0^* - \pmb{\theta}^* \| \leq c_0 \| \text{span}(f_0(\pmb{\theta}_0^*) - f(\pmb{\theta}^*)) \|$ , with  $c_0$  being a positive constant.
Lemma 6. For  $M(\pmb{\theta}_k, \lambda_k)$  in (15) and  $\hat{M}(\pmb{\theta}_k, \lambda_k)$  in (16), with constants  $c_1$  and  $c_0$  (Assumption 3),
$$
M (\boldsymbol {\theta} _ {k}, \lambda_ {k}) \leq 2 \hat {M} (\boldsymbol {\theta} _ {k}, \lambda_ {k}) + \frac {2 \eta_ {k} c _ {0} ^ {2}}{\alpha_ {k} (1 - \kappa)} \| s p a n (\Pi_ {\mathcal {F}} f (\boldsymbol {\theta} ^ {*}) - f (\boldsymbol {\theta} ^ {*})) \| + 2 \mathcal {O} \Big (\frac {c _ {1} ^ {3} (\| \boldsymbol {\theta} _ {0} \| + | \lambda_ {0} | + 1) ^ {3}}{m ^ {1 / 2}} \Big).
$$
We numerically evaluate the performance of Neural-Q-Whittle using an example of circulant dynamics [23, 3, 8]. The state space is  $S = \{1,2,3,4\}$ . Rewards are  $r(1,a) = -1$ ,  $r(2,a) = r(3,a) = 0$ , and  $r(4,a) = 1$  for  $a \in \{0,1\}$ . The dynamics of states are circulant and defined as
$$
P ^ {1} = \left[ \begin{array}{c c c c} 0. 5 & 0. 5 & 0 & 0 \\ 0 & 0. 5 & 0. 5 & 0 \\ 0 & 0 & 0. 5 & 0. 5 \\ 0. 5 & 0 & 0 & 0. 5 \end{array} \right] \text {a n d} P ^ {0} = \left[ \begin{array}{c c c c} 0. 5 & 0 & 0 & 0. 5 \\ 0. 5 & 0. 5 & 0 & 0 \\ 0 & 0. 5 & 0. 5 & 0 \\ 0 & 0 & 0. 5 & 0. 5 \end{array} \right].
$$
This indicates that the process either remains in its current state or increments if it is active (i.e.,  $a = 1$ ), or it either remains the current state or decrements if it is passive (i.e.,  $a = 0$ ). The exact value of Whittle indices [23] are  $\lambda(1) = -0.5$ ,  $\lambda(2) = 0.5$ ,  $\lambda(3) = 1$ , and  $\lambda(4) = -1$ .
In our experiments, we set the learning rates as  $\alpha_{k} = 0.5 / (k + 1)$  and  $\eta_{k} = 0.1 / (k + 1)^{4 / 3}$ . We use  $\epsilon$ -greedy for the exploration and exploitation tradeoff with  $\epsilon = 0.5$ . We consider a two-layer neural network with the number of neurons in the hidden layer as  $m = 200$ . As described in Algorithm 1,  $b_{r}, \forall r$  are uniformly initialized in  $\{-1,1\}$  and  $w_{r}, \forall r$  are initialized as a zero mean Gaussian distribution according to  $\mathcal{N}(\mathbf{0}, \mathbf{I}_d / d)$ . These
(a) Neural-Q-Whittle vs. Q-Whittle [3].
Figure 2: Convergence of Neural-Q-Whittle.
(b) Convergence of Lyapunov function in (15).
results are carried out by Monte Carlo simulations with 100 independent trials.
Convergence to true Whittle index. First, we verify that Neural-Q-Whittle convergences to true Whittle indices, and compare to Q-Whittle, the first Whittle index based Q-learning algorithm. As illustrated in Figure 2a, Neural-Q-Whittle guarantees the convergence to true Whittle indices and outperforms Q-Whittle [3] in the convergence speed. This is due to the fact that Neural-Q-Whittle updates the Whittle index of a specific state even when the current visited state is not that state.
Second, we further compare with other other Whittle index learning algorithms, i.e., Q-Whittle-LFA [60], WIQL [8] and QWIC [23] in Figure 3. As we observe from Figure 3, only Neural-Q-Whittle and Q-Whittle-LFA in [60] can converge to the true Whittle indices for each state, while the other two benchmarks algorithms do not guarantee the convergence of true Whittle indices. Interestingly, the learning Whittle indices converge and maintain a correct relative order of magnitude, which is still able to be used in real world problems [60]. Moreover, we observe that Neural-Q-Whittle achieves similar convergence performance as Q-Whittle-LFA in the considered example, whereas the latter has been shown to achieve good performance in real world applications in
(a) Whittle index  $\lambda (1)$  for  $s = 1$  .(b) Whittle index  $\lambda (2)$  for  $s = 2$
(c) Whittle index  $\lambda (3)s = 3$  . (d) Whittle index  $\lambda (4)$  for  $s = 4$
Figure 3: Convergence comparison between Neural-Q-Whittle and benchmark algorithms.
[60]. Though this work focuses on the theoretical convergence analysis of Q-learning based whittle index under the neural network function approximation, it might be promising to implement it in real-world applications to fully leverage the strong representation ability of neural network functions, which serves as future investigation of this work.
Convergence of the Lyapunov function defined in (15). We also evaluate the convergence of the proposed Lyapunov function defined in (15), which is presented in Figure 2b. It depicts  $\mathbb{E}[M(\pmb{\theta}_k, \lambda_k)]$  vs. the number of iterations in logarithmic scale. For ease of presentation, we only take state  $s = 4$  as an illustrative example. It is clear that  $M(\pmb{\theta}_k, \lambda_k)$  converges to zero as the number of iterations increases, which is in alignment with our theoretical results in Theorem 1.