SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guojun Xiong, Jian Li
{guojun.xiong, jian.li.3}@stonybrook.edu
Whittle index policy is a heuristic to the intractable restless multi-armed bandits (RMAB) problem. Although it is provably asymptotically optimal, finding Whittle indices remains difficult. In this paper, we present Neural-Q-Whittle, a Whittle index based Q-learning algorithm for RMAB with neural network function approximation, which is an example of nonlinear two-timescale stochastic approximation with Q-function values updated on a faster timescale and Whittle indices on a slower timescale. Despite the empirical success of deep Q-learning, the non-asymptotic convergence rate of Neural-Q-Whittle, which couples neural networks with two-timescale Q-learning largely remains unclear. This paper provides a finite-time analysis of Neural-Q-Whittle, where data are generated from a Markov chain, and Q-function is approximated by a ReLU neural network. Our analysis leverages a Lyapunov drift approach to capture the evolution of two coupled parameters, and the nonlinearity in value function approximation further requires us to characterize the approximation error. Combining these provide Neural-Q-Whittle with  $\mathcal{O}(1 / k^{2 / 3})$  convergence rate, where  $k$  is the number of iterations.
We consider the restless multi-armed bandits (RMAB) problem [56], where the decision maker (DM) repeatedly activates  $K$  out of  $N$  arms at each decision epoch. Each arm is described by a Markov decision process (MDP) [45], and evolves stochastically according to two different transition kernels, depending on whether the arm is activated or not. Rewards are generated with each transition. Although RMAB has been widely used to study constrained sequential decision making problems [5, 38, 12, 64, 29, 36, 35, 25], it is notoriously intractable due to the explosion of state space [44]. A celebrated heuristic is the Whittle index policy [56], which computes the Whittle index for each arm given its current state as the cost to pull the arm. Whittle index policy then activates the  $K$  highest indexed arms at each decision epoch, and is provably asymptotically optimal [54].
However, the computation of Whittle index requires full knowledge of the underlying MDP associated with each arm, which is often unavailable in practice. To this end, many recent efforts have focused on learning Whittle indices for making decisions in an online manner. First, model-free reinforcement learning (RL) solutions have been proposed [10, 23, 53, 8, 28, 57, 59, 61, 58, 3], among which [3] developed a Whittle index based Q-learning algorithm, which we call Q-Whittle for ease of exposition, and provided the first-ever rigorous asymptotic analysis. However, Q-Whittle suffers from slow convergence since it only updates the Whittle index of a specific state when that state is visited. In addition, Q-Whittle needs to store the Q-function values for all state-action pairs, which limits its applicability only to problems with small state space. Second, deep RL methods have been leveraged to predict Whittle indices via training neural networks [40, 41]. Though
these methods are capable of dealing with large state space, there is no asymptotic or finite-time performance guarantee. Furthermore, training neural networks requires to tuning hyper-parameters. This introduces an additional layer of complexity to predict Whittle indices. Third, to address aforementioned deficiencies, [60] proposed Q-Whittle-LFA by coupling Q-Whittle with linear function approximation and provided a finite-time convergence analysis. One key limitation of Q-Whittle-LFA is the unrealistic assumption that all data used in Q-Whittle-LFA are sampled i.i.d. from a fixed stationary distribution.
To tackle the aforementioned limitations and inspired by the empirical success of deep Q-learning in numerous applications, we develop Neural-Q-Whittle, a Whittle index based Q-learning algorithm with neural network function approximation under Markovian observations. Like [3, 60], the updates of Q-function values and Whittle indices form a two-timescale stochastic approximation (2TSA) with the former operating on a faster timescale and the later on a slower timescale. Unlike [3, 60], our Neural-Q-Whittle uses a deep neural network with the ReLU activation function to approximate the Q-function. However, Q-learning with neural network function approximation can in general diverge [2], and the theoretical convergence of Q-learning with neural network function approximation has been limited to special cases such as fitted Q-iteration with i.i.d. observations [22], which fails to capture the practical setting of Q-learning with neural network function approximation.
In this paper, we study the non-asymptotic convergence of Neural-Q-Whittle with data generated from a Markov decision process. Compared with recent theoretical works for Q-learning with neural network function approximation [13, 22, 62], our Neural-Q-Whittle involves a two-timescale update between two coupled parameters, i.e., Q-function values and Whittle indices. This renders existing finite-time analysis in [13, 22, 62] not applicable to our Neural-Q-Whittle due to the fact that [13, 22, 62] only contains a single-timescale update on Q-function values. Furthermore, [13, 22, 62] required an additional projection step for the update of parameters of neural network function so as to guarantee the boundedness between the unknown parameter at any time step with the initialization. This in some cases is impractical. Hence, a natural question that arises is
Is it possible to provide a non-asymptotic convergence rate analysis of Neural-  $Q$ -Whittle with two coupled parameters updated in two timescales under Markovian observations without the extra projection step?
The theoretical convergence guarantee of two-timescale Q-learning with neural network function approximation under Markovian observations remains largely an open problem, and in this paper, we provide an affirmative answer to this question. Our main contributions are summarized as follows:
- We propose Neural-Q-Whittle, a novel Whittle index based Q-learning algorithm with neural network function approximation for RMAB. Inspired by recent work on TD learning [48] and Q-learning [15] with linear function approximation, our Neural-Q-Whittle removes the additional impractical projection step in the neural network function parameter update.
- We establish the first finite-time analysis of Neural-Q-Whittle under Markovian observations. Due to the two-timescale nature for the updates of two coupled parameters (i.e., Q-function values and Whittle indices) in Neural-Q-Whittle, we focus on the convergence rate of these parameters rather than the convergence rate of approximated Q-functions as in [13, 22, 62]. Our key technique is to view Neural-Q-Whittle as a 2TSA for finding the solution of suitable nonlinear equations. Different from recent works on finite-time analysis of a general 2TSA [20] or with linear function approximation [60], the nonlinear parameterization of Q-function in Neural-Q-Whittle under Markovian observations imposes significant difficulty in finding the global optimum of the corresponding nonlinear equations. To mitigate this, we first approximate the original neural network function with a collection of local linearization and focus on finding a surrogate Q-function in the neural network function class that well approximates the optimum. Our finite-time analysis then requires us to consider two Lyapunov functions that carefully characterize the coupling between iterates of Q-function values and Whittle indices, with one Lyapunov function defined with respect to the true neural network function, and the other defined with respect to the locally linearized neural network function. We then characterize the errors between these two Lyapunov functions. Putting them together, we prove that Neural-Q-Whittle achieves a convergence in expectation at a rate  $\mathcal{O}(1 / k^{2 / 3})$ , where  $k$  is the number of iterations.
- Finally, we conduct experiments to validate the convergence performance of Neural-Q-Whittle, and verify the sufficiency of our proposed condition for the stability of Neural-Q-Whittle.
RMAB. We consider an infinite-horizon average-reward RMAB with each arm  $n \in \mathcal{N}$  described by a unichain MDP [45]  $\mathcal{M}_n \coloneqq (\mathcal{S}, \mathcal{A}, P_n, r_n)$ , where  $\mathcal{S}$  is the state space with cardinality  $S < \infty$ ,  $\mathcal{A}$  is the action space with cardinality  $A$ ,  $P_n(s'|s, a)$  is the transition probability of reaching state  $s'$  by taking action  $a$  in state  $s$ , and  $r_n(s, a)$  is the reward associated with state-action pair  $(s, a)$ . At each time slot  $t$ , the DM activates  $K$  out of  $N$  arms. Arm  $n$  is "active" at time  $t$  when it is activated, i.e.,  $A_n(t) = 1$ ; otherwise, arm  $n$  is "passive", i.e.,  $A_n(t) = 0$ . Let  $\Pi$  be the set of all possible policies for RMAB, and  $\pi \in \Pi$  is a feasible policy, satisfying  $\pi : \mathcal{F}_t \mapsto \mathcal{A}^N$ , where  $\mathcal{F}_t$  is the sigma-algebra generated by random variables  $\{S_n(h), A_n(h) : \forall n \in \mathcal{N}, h \leq t\}$ . The objective of the DM is to maximize the expected long-term average reward subject to an instantaneous constraint that only  $K$  arms can be activated at each time slot, i.e.,
$$
\text {R M A B :} \quad \max  _ {\pi \in \Pi} \lim  _ {T \rightarrow \infty} \inf  _ {T \rightarrow \infty} \frac {1}{T} \mathbb {E} _ {\pi} \left(\sum_ {t = 0} ^ {T} \sum_ {n = 1} ^ {N} r _ {n} (t)\right), \quad \text {s . t .} \quad \sum_ {n = 1} ^ {N} A _ {n} (t) = K, \quad \forall t. \tag {1}
$$
Whittle Index Policy. It is well known that RMAB (1) suffers from the curse of dimensionality [44]. To address this challenge, Whittle [56] proposed an index policy through decomposition. Specifically, Whittle relaxed the constraint in (1) to be satisfied on average and obtained a unconstrained problem:  $\max_{\pi \in \Pi} \lim \inf_{T \to \infty} \frac{1}{T} \mathbb{E}_{\pi} \sum_{t=1}^{T} \sum_{n=1}^{N} \{r_n(t) + \lambda (1 - A_n(t))\}$ , where  $\lambda$  is the Lagrangian multiplier associated with the constraint. The key observation of Whittle is that this problem can be decomposed and its solution is obtained by combining solutions of  $N$  independent problems via solving the associated dynamic programming (DP):  $V_n(s) = \max_{a \in \{0,1\}} Q_n(s,a), \forall n \in \mathcal{N}$ , where
$$
Q _ {n} (s, a) + \beta = a \left(r _ {n} (s, a) + \sum_ {s ^ {\prime}} p _ {n} \left(s ^ {\prime} \mid s, 1\right) V _ {n} \left(s ^ {\prime}\right)\right) + (1 - a) \left(r _ {n} (s, a) + \lambda + \sum_ {s ^ {\prime}} p _ {n} \left(s ^ {\prime} \mid s, 0\right) V _ {n} \left(s ^ {\prime}\right)\right), \tag {2}
$$
where  $\beta$  is unique and equals to the maximal long-term average reward of the unichain MDP, and  $V_{n}(s)$  is unique up to an additive constant, both of which depend on the Lagrangian multiplier  $\lambda$ . The optimal decision  $a^{*}$  in state  $s$  then is the one which maximizes the right hand side of the above DP. The Whittle index associated with state  $s$  is defined as the value  $\lambda_{n}^{*}(s) \in \mathbb{R}$  such that actions 0 and 1 are equally favorable in state  $s$  for arm  $n$  [3, 23], satisfying
$$
\lambda_ {n} ^ {*} (s) := r _ {n} (s, 1) + \sum_ {s ^ {\prime}} p _ {n} \left(s ^ {\prime} \mid s, 1\right) V _ {n} \left(s ^ {\prime}\right) - r _ {n} (s, 0) - \sum_ {s ^ {\prime}} p _ {n} \left(s ^ {\prime} \mid s, 0\right) V _ {n} \left(s ^ {\prime}\right). \tag {3}
$$
Whittle index policy then activates  $K$  arms with the largest Whittle indices at each time slot. Additional discussions are provided in Section B in supplementary materials.
Q-Learning for Whittle Index. Since the underlying MDPs are often unknown, [3] proposed Q-whittle, a tabular Whittle index based Q-learning algorithm, where the updates of Q-function values and Whittle indices form a 2TSA, with the former operating on a faster timescale for a given  $\lambda_{n}$  and the later on a slower timescale. Specifically, the Q-function values for  $\forall n\in \mathcal{N}$  are updated as
$$
\begin{array}{l} Q _ {n, k + 1} (s, a) := Q _ {n, k} (s, a) + \alpha_ {n, k} \mathbb {1} _ {\left\{S _ {n, k} = s, A _ {n, k} = a \right\}} \left(r _ {n} (s, a) + (1 - a) \lambda_ {n, k} (s) \right. \\ \left. + \max _ {a} Q _ {n, k} (S _ {n, k + 1}, a) - I _ {n} (Q _ {k}) - Q _ {n, k} (s, a)\right), \quad (4) \\ \end{array}
$$
where  $I_{n}(Q_{k}) = \frac{1}{2S}\sum_{s\in S}(Q_{n,k}(s,0) + Q_{n,k}(s,1))$  is standard in the relative Q-learning for long-term average MDP setting [1], which differs significantly from the discounted reward setting [45, 1].  $\{\alpha_{n,k}\}$  is a step-size sequence satisfying  $\sum_{k}\alpha_{n,k} = \infty$  and  $\sum_{k}\alpha_{n,k}^{2} < \infty$ .
Accordingly, the Whittle index is updated as
$$
\lambda_ {n, k + 1} (s) = \lambda_ {n, k} (s) + \eta_ {n, k} \left(Q _ {n, k} (s, 1) - Q _ {n, k} (s, 0)\right), \tag {5}
$$
with the step-size sequence  $\{\eta_{n,k}\}$  satisfying  $\sum_k\eta_{n,k} = \infty$ ,  $\sum_k\eta_{n,k}^2 < \infty$  and  $\eta_{n,k} = o(\alpha_{n,k})$ . The coupled iterates (4) and (5) form a 2TSA, and [3] provided an asymptotic convergence analysis.
3 Neural Q-Learning for Whittle Index
A closer look at (5) reveals that Q-Whittle only updates the Whittle index of a specific state when that state is visited. This makes Q-Whittle suffers from slow convergence. In addition, Q-Whittle
Algorithm 1 Neural-Q-Whittle: Neural Q-Learning for Whittle Index
1: Input:  $\phi (s,a)$  for  $\forall s\in S,a\in A$  , and learning rates  $\{\alpha_{k}\}_{k = 1,\dots,T}$ $\{\eta_k\}_{k = 1,\dots,T}$    
2: Initialization:  $b_{r}\sim \mathrm{Unif}(\{-1,1\})$ $\mathbf{w}_{r,0}\sim \mathcal{N}(\mathbf{0},\mathbf{I}_d / d),\forall r\in [1,m]$  and  $\lambda (s) = 0,\forall s\in S$    
3: for  $s\in S$  do   
4: for  $k = 1,\ldots ,T$  do   
5: Sample  $(S_{k},A_{k},S_{k + 1})$  according to the  $\epsilon$  -greedy policy;   
6:  $\Delta_{k} = r(S_{k},A_{k}) + (1 - A_{k})\lambda_{k}(s) + \max_{a}f(\pmb{\theta}_{k};\pmb {\phi}(S_{k + 1},a)) - I(\pmb{\theta}_{k}) - f(\pmb{\theta}_{k};\pmb {\phi}(S_{k},A_{k}))$    
7:  $\pmb{\theta}_{k + 1} = \pmb{\theta}_k + \alpha_k\Delta_k\nabla_\pmb{\theta}f(\pmb{\theta}_k;\pmb {\phi}(S_k,A_k));$    
8:  $\lambda_{k + 1}(s) = \lambda_k(s) + \eta_k(f(\pmb {\theta}_k;\pmb {\phi}(s,1)) - f(\pmb {\theta}_k;\pmb {\phi}(s,0));$    
9: end for   
10: end for   
11: Return:  $\lambda (s),\forall s\in S.$
needs to store the Q-function values for all state-action pairs, which limits its applicability only to problems with small state space. To address this challenge and inspired by the empirical success of deep Q-learning, we develop Neural-Q-Whittle through coupling Q-Whittle with neural network function approximation by using low-dimensional feature mapping and leveraging the strong representation power of neural networks. For ease of presentation, we drop the subscript  $n$  in (4) and (5), and discussions in the rest of the paper apply to any arm  $n \in \mathcal{N}$ .
Specifically, given a set of basis functions  $\phi_{\ell} : S \times \mathcal{A} \mapsto \mathbb{R}, \forall \ell = 1, \dots, d$  with  $d \ll SA$ , the approximation of Q-function  $Q_{\pmb{\theta}}(s,a)$  parameterized by a unknown weight vector  $\pmb{\theta} \in \mathbb{R}^{md}$ , is given by  $Q_{\pmb{\theta}}(s,a) = f(\pmb{\theta}; \pmb{\phi}(s,a))$ ,  $\forall s \in S, a \in \mathcal{A}$ , where  $f$  is a nonlinear neural network function parameterized by  $\pmb{\theta}$  and  $\pmb{\phi}(s,a)$ , with  $\pmb{\phi}(s,a) = (\phi_1(s,a), \dots, \phi_d(s,a))^{\top}$ . The feature vectors are assumed to be linearly independent and are normalized so that  $\| \pmb{\phi}(s,a) \| \leq 1, \forall s \in S, a \in \mathcal{A}$ . In particular, we parameterize the Q-function by using a two-layer neural network [13, 62]
$$
f (\boldsymbol {\theta}; \boldsymbol {\phi} (s, a)) := \frac {1}{\sqrt {m}} \sum_ {r = 1} ^ {m} b _ {r} \sigma \left(\mathbf {w} _ {r} ^ {\intercal} \boldsymbol {\phi} (s, a)\right), \tag {6}
$$
where  $\pmb{\theta} = (b_{1},\dots,b_{m},\mathbf{w}_{1}^{\intercal},\dots,\mathbf{w}_{m}^{\intercal})^{\intercal}$  with  $b_{r}\in \mathbb{R}$  and  $\mathbf{w}_r\in \mathbb{R}^{d\times 1},\forall r\in [1,m]$ .  $b_{r},\forall r$  are uniformly initialized in  $\{-1,1\}$  and  $w_{r},\forall r$  are initialized as a zero mean Gaussian distribution according to  $\mathcal{N}(\mathbf{0},\mathbf{I}_d / d)$ . During training process, only  $\mathbf{w}_r,\forall r$  are updated while  $b_{r},\forall r$  are fixed as the random initialization. Hence, we use  $\pmb{\theta}$  and  $\mathbf{w}_r,\forall r$  interchangeably throughout this paper.  $\sigma (x) = \max (0,x)$  is the rectified linear unit (ReLU) activation function<sup>1</sup>.
Given (6), we can rewrite the Q-function value updates in (4) as
$$
\boldsymbol {\theta} _ {k + 1} = \boldsymbol {\theta} _ {k} + \alpha_ {k} \Delta_ {k} \nabla_ {\boldsymbol {\theta}} f (\boldsymbol {\theta} _ {k}; \boldsymbol {\phi} (S _ {k}, A _ {k})), \tag {7}
$$
with  $\Delta_{k}$  being the temporal difference (TD) error defined as  $\Delta_{k} := r(S_{k}, A_{k}) + (1 - A_{k})\lambda_{k}(s) - I(\pmb{\theta}_{k}) + \max_{a} f(\pmb{\theta}_{k}; \pmb{\phi}(S_{k+1}, a)) - f(\pmb{\theta}_{k}; \pmb{\phi}(S_{k}, A_{k}))$ , where  $I(\pmb{\theta}_{k}) = \frac{1}{2S} \sum_{s \in S} [f(\pmb{\theta}_{k}; \pmb{\phi}(s, 0)) + f(\pmb{\theta}_{k}; \pmb{\phi}(s, 1))]$ . Similarly, the Whittle index update (5) can be rewritten as
$$
\lambda_ {k + 1} (s) = \lambda_ {k} (s) + \eta_ {k} \left(f \left(\boldsymbol {\theta} _ {k}; \boldsymbol {\phi} (s, 1)\right) - f \left(\boldsymbol {\theta} _ {k}; \boldsymbol {\phi} (s, 0)\right)\right). \tag {8}
$$
The coupled iterates in (7) and (8) form Neural-Q-Whittle as summarized in Algorithm 1, which aims to learn the coupled parameters  $(\pmb{\theta}^{*},\lambda^{*}(s))$  such that  $f(\pmb{\theta}^{*},\pmb{\phi}(s,1)) = f(\pmb{\theta}^{*},\pmb{\phi}(s,0)),\forall s\in S.$
Remark 1. Unlike recent works for  $Q$ -learning with linear [6, 37, 67] or neural network function approximations [13, 22, 62], we do not assume an additional projection step of the updates of unknown parameters  $\pmb{\theta}_k$  in (7) to confine  $\pmb{\theta}_k$ ,  $\forall k$  into a bounded set. This projection step is often used to stabilize the iterates related to the unknown stationary distribution of the underlying Markov chain, which in some cases is impractical. More recently, [48] removed the extra projection step and established the finite-time convergence of TD learning, which is treated as a linear stochastic approximation algorithm. [15] extended it to the  $Q$ -learning with linear function approximation.
<sup>1</sup>The finite-time analysis of Deep Q-Networks (DQN) [13, 22, 62] and references therein focuses on the ReLU activation function, as it has certain properties that make the analysis tractable. ReLU is piecewise linear and non-saturating, which can simplify the mathematical analysis. Applying the same analysis to other activation functions like the hyperbolic tangent (tanh) could be more complex, which is out of the scope of this work.
Table 1: Comparison of settings in related works.
Algorithm	Noise	Approximation	Timescale	Whittle index
Q-Whittle [3]	i.i.d.	X	two-timescale	✓
Q-Whittle-LFA [60]	i.i.d.	linear	two-timescale	✓
Q-Learning-LFA [6, 37, 67]	Markovian	linear	single-timescale	X
Q-Learning-NFA [13, 15, 22, 62]	Markovian	neural network	single-timescale	X
TD-Learning-LFA [48]	Markovian	linear	single-timescale	X
2TSA-IID [19, 21]	i.i.d.	X	two-timescale	X
2TSA-Markovian [20]	Markovian	X	two-timescale	X
Neural-Q-Whittle (this work)	Markovian	neural network	two-timescale	✓
However, these state-of-the-art works only contained a single-timescale update on  $Q$ -function values, i.e., with the only unknown parameter  $\theta$ , while our Neural- $Q$ -Whittle involves a two-timescale update between two coupled unknown parameters  $\theta$  and  $\lambda$  as in (7) and (8). Our goal in this paper is to expand the frontier by providing a finite-time bound for Neural- $Q$ -Whittle under Markovian noise without requiring an additional projection step. We summarize the differences between our work and existing literature in Table 1.
4 Finite-Time Analysis of Neural-Q-Whittle
In this section, we present the finite-time analysis of Neural-Q-Whittle for learning Whittle index  $\lambda(s)$  of any state  $s \in S$  when data are generated from a MDP. To simplify notation, we abbreviate  $\lambda(s)$  as  $\lambda$  in the rest of the paper. We start by first rewriting the updates of Neural-Q-Whittle in (7) and (8) as a nonlinear two-timescale stochastic approximation (2TSA) in Section 4.1.
4.1 A Nonlinear 2TSA Formulation with Neural Network Function
We first show that Neural-Q-Whittle can be rewritten as a variant of the nonlinear 2TSA. For any fixed policy  $\pi$ , since the state of each arm  $\{S_k\}$  evolves according to a Markov chain, we can construct a new variable  $X_{k} = (S_{k},A_{k},S_{k + 1})$ , which also forms a Markov chain with state space  $\mathcal{X} := \{(s,a,s')|s\in S,\pi (a|s)\geq 0,p(s'|s,a) > 0\}$ . Therefore, the coupled updates (7) and (8) of Neural-Q-Whittle can be rewritten in the form of a nonlinear 2TSA [20]:
$$
\boldsymbol {\theta} _ {k + 1} = \boldsymbol {\theta} _ {k} + \alpha_ {k} h \left(X _ {k}, \boldsymbol {\theta} _ {k}, \lambda_ {k}\right), \quad \lambda_ {k + 1} = \lambda_ {k} + \eta_ {k} g \left(X _ {k}, \boldsymbol {\theta} _ {k}, \lambda_ {k}\right), \tag {9}
$$
where  $\theta_0$  and  $\lambda_0$  being arbitrarily initialized in  $\mathbb{R}^{md}$  and  $\mathbb{R}$ , respectively; and  $h(\cdot)$  and  $g(\cdot)$  satisfy
$$
h \left(X _ {k}, \boldsymbol {\theta} _ {k}, \lambda_ {k}\right) := \nabla_ {\boldsymbol {\theta}} f \left(\boldsymbol {\theta} _ {k}; \phi \left(S _ {k}, A _ {k}\right)\right) \Delta_ {k}, \quad \boldsymbol {\theta} _ {k} \in \mathbb {R} ^ {m d}, \lambda_ {k} \in \mathbb {R}, \tag {10}
$$
$$
g \left(X _ {k}, \boldsymbol {\theta} _ {k}, \lambda_ {k}\right) := f \left(\boldsymbol {\theta} _ {k}; \boldsymbol {\phi} (s, 1)\right) - f \left(\boldsymbol {\theta} _ {k}; \boldsymbol {\phi} (s, 0)\right), \quad \boldsymbol {\theta} _ {k} \in \mathbb {R} ^ {m d}. \tag {11}
$$
Since  $\eta_{k} \ll \alpha_{k}$ , the dynamics of  $\pmb{\theta}$  evolves much faster than those of  $\lambda$ . We aim to establish the finite-time performance of the nonlinear 2TSA in (9), where  $f(\cdot)$  is the neural network function defined in (6). This is equivalent to find the root $^{2}$ $(\pmb{\theta}^{*}, \lambda^{*})$  of a system with two coupled nonlinear equations  $h: \mathcal{X} \times \mathbb{R}^{md} \times \mathbb{R} \to \mathbb{R}^{md}$  and  $g: \mathcal{X} \times \mathbb{R}^{md} \times \mathbb{R} \to \mathbb{R}$  such that
$$
H (\boldsymbol {\theta}, \lambda) := \mathbb {E} _ {\mu} [ h (X, \boldsymbol {\theta}, \lambda) ] = 0, \quad G (\boldsymbol {\theta}, \lambda) := \mathbb {E} _ {\mu} [ g (X, \boldsymbol {\theta}, \lambda) ] = 0, \tag {12}
$$
where  $X$  is a random variable in finite state space  $\mathcal{X}$  with unknown distribution  $\mu$ . For a fixed  $\pmb{\theta}$ , to study the stability of  $\lambda$ , we assume the condition on the existence of a mapping such that  $\lambda = y(\pmb{\theta})$  is the unique solution of  $G(\pmb{\theta}, \lambda) = 0$ . In particular,  $y(\pmb{\theta})$  is given as
$$
y (\boldsymbol {\theta}) = r (s, 1) + \sum_ {s ^ {\prime}} p \left(s ^ {\prime} \mid s, 1\right) \max  _ {a} f \left(\boldsymbol {\theta}; \phi \left(s ^ {\prime}, a\right)\right) - r (s, 0) - \sum_ {s ^ {\prime}} p \left(s ^ {\prime} \mid s, 0\right) \max  _ {a} f \left(\boldsymbol {\theta}; \phi \left(s ^ {\prime}, a\right)\right). \tag {13}
$$