SlowGuess/NeurIPS_2023_OCR huggingface dataset:
On the Convergence and Sample Complexity Analysis of Deep Q-Networks with  $\varepsilon$ -Greedy Exploration
Hongkang Li
Meng Wang
Miao Liu
Pin-Yu Chen
Sijia Liu
Keerthiram Murugesan
Subhajit Chaudhury
This paper provides a theoretical understanding of Deep Q-Network (DQN) with the  $\varepsilon$ -greedy exploration in deep reinforcement learning. Despite the tremendous empirical achievement of the DQN, its theoretical characterization remains underexplored. First, the exploration strategy is either impractical or ignored in the existing analysis. Second, in contrast to conventional Q-learning algorithms, the DQN employs the target network and experience replay to acquire an unbiased estimation of the mean-square Bellman error (MSBE) utilized in training the Q-network. However, the existing theoretical analysis of DQNs lacks convergence analysis or bypasses the technical challenges by deploying a significantly overparameterized neural network, which is not computationally efficient. This paper provides the first theoretical convergence and sample complexity analysis of the practical setting of DQNs with  $\varepsilon$ -greedy policy. We prove an iterative procedure with decaying  $\varepsilon$  converges to the optimal Q-value function geometrically. Moreover, a higher level of  $\varepsilon$  values enlarges the region of convergence but slows down the convergence, while the opposite holds for a lower level of  $\varepsilon$  values. Experiments justify our established theoretical insights on DQNs.
Reinforcement learning (RL) is a sequential decision-making process for a learning agent taking actions in the environment. RL has found important applications in autonomous control [37, 36], healthcare [16], Internet of Things [40, 33], and natural language processing [70]. The environment of an RL problem is modeled as a Markov decision process (MDP) with an underlying state transition probability matrix. The goal of the problem is to find an optimal policy to select the best actions to maximize the immediate and future rewards. Q-learning [77] has been recognized as one of the most promising and efficient learning algorithms for seeking the optimal policy because it does not require the knowledge of the model of the environment (namely, "model free") and can learn from data generated from a non-optimal policy (namely, "off-policy"). Traditional Q-learning approaches are centered on tabular methods [82, 41] or linear function approximations [65, 13] to estimate the optimal action-value (Q-value) function. However, tabular methods require sample complexity scaling in the order of state space, which is impractical for modern RL problems involving large or even infinite state space [82]. Q-learning with linear function approximation is limited to applications only when the transition matrix admits a linear feature representation [23].
Table 1: Comparison among some representative works of Q-learning with function approximation.
Work	Neural Approximation	Convergence of MSBE	ε-greedy
Yang & Wang (2019)	X	✓	X
Xu & Gu (2020)	✓	✓	X
Fan et al. (2020)	✓	X	X
Liu et al. (2022)	✓	X	✓
This work	✓	✓	✓
Due to the remarkable advancements in deep neural networks (DNNs), the Deep Q-network (DQN) framework has emerged as a powerful approach that leverages the expressive power of non-linear functions and the ability to generalize to unknown states. DQN has proven to be a pioneering solution that addresses challenges encountered by traditional approaches. In the DQN framework [51], the Q-value function is approximated using a DNN, and the algorithm iteratively updates the Q-value function by collecting data following the  $\varepsilon$ -greedy policy. The  $\varepsilon$ -greedy policy is the simplest approach to balance exploration and exploitation. Namely, with a probability of  $\varepsilon$ , we select a random action (exploration), and with a probability of  $1 - \varepsilon$ , we choose the best action according to the current estimated Q-value function (exploitation).  $\varepsilon$ -greedy is myopic compared with other strategic explorations, e.g., Thompson sampling-based [71, 55, 25] and optimism in the face of the uncertainty (OFU)-based [30, 31] ones. However, implementing these strategic explorations is not computationally efficient in DQNs [17], while DQNs equipped with  $\varepsilon$ -greedy policy have shown empirical success in diversified applications, e.g., the game of Go [63], Atari games [51], robotics [34], and autonomous vehicles [61, 12, 60].
Despite the numerical success, the theoretical understanding of DQN remains elusive, which could prevent its applications in domains requiring reliable and safe decision-making. First, updating the Q-value function involves minimizing a mean-squared Bellman error (MSBE) function. However, existing convergence analysis and statistical properties are predominantly limited to linear models, failing to capture the complexities present in non-linear neural networks like DQN. Second, the sample complexity required for the convergence of MSBE is still not well comprehended. Achieving the desired accuracy in this context often demands a sample complexity that grows exponentially with the input dimension [46], rendering it impractical and inefficient in real-world scenarios. Third, the optimal selection of the  $\varepsilon$  value in DQN remains a gap in existing research. The hyperparameter tuning in algorithms involved with neural networks can be arduous and time-consuming. For instance, without a well-designed hyperparameter configuration, only a small fraction (e.g.,  $1\%$ ) of the possible combinations yield satisfactory results in neural network training [76].
Contributions. To the best of our knowledge, this paper presents the first theoretical study with convergence analysis for Deep Q-Networks (DQNs) utilizing the  $\varepsilon$ -greedy policy. A comparison with existing works can be found in Table 1. The paper focuses on the Q-value function approximated by a DNN. It offers a comprehensive analysis of the convergence of DQNs and provides insights into the estimation error of the learned Q-value function, accompanied by an analysis of the sample complexity. The key contributions of this study are as follows:
1. The convergence analysis of DQN with bounded estimation errors. Assuming the existence of a DNN with unknown weights  $W^{\star}$  that matches the optimal Q-value function, this paper proves that the learned model via DQNs equipped with  $\varepsilon$ -greedy policy through the (accelerated) gradient descent algorithm converges linearly to  $W^{\star}$  up to some characterizable estimation error.
2. The theoretical characterization for a wide selection range of  $\varepsilon$  over iterations. This paper provides lower and upper bounds of  $\varepsilon$  at each iteration for the convergence of DQNs and, in particular, characterizes the sample complexity, estimation error, and convergence rate of the DQN equipped with  $\varepsilon$ -greedy with decreasing  $\varepsilon$ . Moreover, this paper proves that a higher level of  $\varepsilon$  values leads to an enlarged region of convergence, which relaxes the requirement on the initialization, while a lower level of  $\varepsilon$  values leads to faster convergence.
3. The sample complexity analysis for learning a desired Q-value function. We quantify the required sample complexity, depending on the neural network parameters and distribution shift of the collected data, for the learned model to converge to the desired accuracy. Typically, the estimation error of the converged model scales in the order of  $1 / \sqrt{N_s}$ , where  $N_{s}$  is the number of samples.
2 Related Works.
Q-learning with linear function approximation. In the setting of linear function approximation, the Q-function is assumed to be a linear function of either the feature mapping [83, 32, 92] or a mixture of some basis kernels [91, 52]. Early works mainly focus on the algorithm design [6, 48, 2] and convergence analysis [38, 47, 67, 14, 69] but lacks theoretical guarantees with polynomial sample complexity. Assuming the underlying Q-function can be exactly represented as a linear function of the feature mapping with some unknown parameters, several sample-efficient algorithms are proposed to find the ground-truth mapping with finite-sample guarantee [11, 80], and the sample complexity depends linearly on the feature dimension [80].
Q-learning with non-linear function approximation. Recent approaches with non-linear function approximation mainly fall into the frameworks of the Bellman Eluder (BE) dimension [30, 18, 58, 27], Neural Tangent Kernel (NTK) [81, 10, 79, 10, 20, 53], and Besov regularity [68, 29, 54, 46, 24]. The Eluder dimension is at least in an exponential order even for a two-layer neural network [19], leading to uncharacterizable sample complexity. The NTK framework linearizes deep neural networks to tackle convergence in non-linear models. However, it requires using computationally inefficient extremely wide neural networks [81]. Moreover, the sample complexity can exponentially increase with the input feature dimension, necessitating a substantial number of samples for accurate estimation. Furthermore, the NTK approach fails to explain the advantages of non-linear neural networks over linear function approximation [46, 24]. Besov space requires the neural networks to be sparse and makes an unpractical assumption that the algorithm can find the global optimal of the non-convex objective function [29, 54, 46, 24]. To the best of our knowledge, only [46] considers  $\varepsilon$ -greedy policy with theoretical analysis applicable to DQNs. However, the model is limited to sparse neural networks, and it cannot characterize the case of varying  $\varepsilon$ .
Supervised learning with neural networks. Compared with Q-learning in RL, where the label of the sampled data depends on the currently estimated Q function, analyzing supervised learning is less challenging, where sampled data label is known and fixed across the training. Existing theoretical results for supervised learning are largely built upon NTK [28, 21, 39, 15, 43, 45], model recovery [90, 26, 3, 64, 88, 85, 87], and structured data [44, 62, 8, 1, 35, 78, 84, 42]. Due to the high non-convexity of neural networks [59], the one-hidden-layer neural network is still a state-of-the-art practice for convergence analysis and generalization guarantees. Additional assumptions, e.g., Gaussian distribution [89, 7], linear separable data [75, 9] on the data distribution, are needed for finite-sample analysis.
3 Problem Formulation: Notation, Background, and Algorithm
The Markov Decision Process and Q-learning. A discounted Markov decision process (MDP) is defined as  $(\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma)$ , where  $\mathcal{S}$  is the state space,  $\mathcal{A}$  is the action set.  $\mathcal{P}:\mathcal{S}\times \mathcal{A}\longrightarrow \Delta (\mathcal{S})$  is the transition operator, and  $p_{s,s'}^a \coloneqq \mathcal{P}(s'|s,a)$  denotes the transition probability from current state  $s$  and action  $a$  to the next state  $s'$ . In addition,  $r:\mathcal{S}\times \mathcal{A}\longrightarrow [-R_{\max},R_{\max}]$  is the reward function, and  $\gamma \in (0,1)$  is the discount factor.
At a state  $s \in S$ , the agent takes action  $a \in \mathcal{A}$  according to some behavior policy  $\pi$ , denoted as  $a \sim \pi(s)$  (or  $a = \pi(s)$  for deterministic policy). Then, the system moves to the next state  $s'$  following the transition probability  $p_{s,s'}^a$ . Meanwhile, the agent receives an immediate reward  $r(s,a)$  from the environment. Let  $\{s_i, a_i\}_{i=0}^{\infty}$  be the generated sequential data given the behavior policy  $\pi$  and transition probability  $\mathcal{P}$ . We define the state-value function  $V_{\pi}$  at state  $s$  as
$$
V ^ {\pi} (\boldsymbol {s}) = \mathbb {E} _ {\pi , \mathcal {P}} \left[ \sum_ {i = 0} ^ {\infty} \gamma^ {i} r \left(\boldsymbol {s} _ {i}, a _ {i}\right) \mid \boldsymbol {s} _ {0} = \boldsymbol {s} \right], \tag {1}
$$
which is the expected total discounted reward starting from the state  $s$ . For any state-action  $(s, a)$ , the corresponding Q-value (or action-value) function  $Q_{\pi}$  of a policy  $\pi$  is defined as
$$
Q ^ {\pi} (\pmb {s}, a) = \mathbb {E} _ {\pi , \mathcal {P}} \left[ \sum_ {i = 0} ^ {\infty} \gamma^ {i} r (\pmb {s} _ {i}, a _ {i}) \mid \pmb {s} _ {0} = \pmb {s}, a _ {0} = a \right]. (2)
$$
Then, the goal of the agent is to find an optimal policy  $\pi^{\star}$  that maximizes the state-value function in (1) for all states, which is equivalent to
$$
Q ^ {\star} (\boldsymbol {s}, a) := \max  _ {\pi} Q ^ {\pi} (\boldsymbol {s}, a) = r (\boldsymbol {s}, a) + \gamma \cdot \mathbb {E} _ {\boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a} \max  _ {a ^ {\prime}} Q ^ {\star} \left(\boldsymbol {s} ^ {\prime}, a ^ {\prime}\right), \tag {3}
$$
where (3) is known as the Bellman equation. With the optimal Q-value function  $Q^{\star}$ , the optimal policy can be derived via  $\pi^{\star}(\pmb {s}) = \mathrm{argmax}_aQ^{\star}(\pmb {s},a)$  [77, 66].
The Deep Neural Network Model. The DQN utilizes a deep neural network (DNN)  $H: \mathbb{R}^d \longrightarrow \mathbb{R}$  to approximate the optimal Q-value function  $Q^{\star}$  in (3). Specifically, given input  $\pmb{x} \in \mathbb{R}^d$ , the output of the  $L$ -hidden-layer DNN with  $K$  neurons in each hidden layer is defined as
$$
H (\boldsymbol {W}; \boldsymbol {x}) := \mathbf {1} ^ {\top} / K \cdot \phi \left(\boldsymbol {W} _ {L} ^ {\top} \dots \phi \left(\boldsymbol {W} _ {1} ^ {\top} \boldsymbol {x}\right)\right), \tag {4}
$$
where  $\mathbf{W}_1\in \mathbb{R}^{d\times K}$ ,  $\mathbf{W}_l\in \mathbb{R}^{K\times K}$  with  $l = 2,\dots ,L$ , and  $\mathbf{W} = [\mathrm{vec}(\mathbf{W}_1)^\top ,\dots ,\mathrm{vec}(\mathbf{W}_L)^\top ]^\top$  is the concatenation of the vectorization of all parameter matrices.  $\phi (\cdot)$  is the nonlinear activation function, and we consider the ReLU activation function, i.e.,  $\phi (z) = \max \{0,z\}$ . Then, the Q-value function  $Q(s,a)$  is parameterized using the DNN as
$$
Q (\boldsymbol {W}; \boldsymbol {s}, a) = H (\boldsymbol {W}; \boldsymbol {x} (\boldsymbol {s}, a)), \tag {5}
$$
where  $\pmb{x}:\mathcal{S}\times \mathcal{A}\longrightarrow \mathbb{R}^d$  is the feature mapping of the state-action pair. Without loss of generality, we assume  $|\pmb {x}(\pmb {s},a)|\leq 1$ . Then, the goal of DQN-based Q-learning is to minimize the mean squared Bellman error (MSBE) as
$$
\min  _ {\boldsymbol {W}}: f (\boldsymbol {W}) := \mathbb {E} _ {(\boldsymbol {s}, a) \sim \pi^ {\star}} \left(Q (\boldsymbol {W}; \boldsymbol {s}, a) - r (\boldsymbol {s}, a) - \gamma \cdot \mathbb {E} _ {\boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a} \max  _ {a ^ {\prime}} Q (\boldsymbol {W}; \boldsymbol {s} ^ {\prime}, a ^ {\prime})\right) ^ {2}, \tag {6}
$$
where  $\mu$  is the distribution of  $(s,a)$  following the optimal policy  $\pi^{\star}$ .
3.1 The Deep Q-Network Algorithm
Algorithm 1 Deep Q-Network
1: Input: Number of iterations  $T \times M$ , and experience replay buffer size  $N$ , exploration probability  $\{\varepsilon_t\}_{t=1}^T$ , step size  $\eta$ , and momentum parameter  $\beta$ .
2: Initialize the Q-network with weights  $\mathbf{W}^{(0,0)}$ .
3: for  $t = 0,1,2,\dots ,T - 1$  do
4: Select the initial weights  $\mathbf{W}^{(t,0)}$ .
5: for  $m = 0,1,2,\dots ,M - 1$  do
6: Sample data and store in the experience replay buffer  $\mathcal{D}_t$  following  $\varepsilon_{t}$ -greedy policy, namely, at state  $\pmb{s}_n$ , with probability  $\varepsilon_t$ , select a random action  $a_n$ , otherwise select  $a_n = \operatorname{argmax}_a Q(\pmb{W}^{(t,0)}; \pmb{s}_n, a)$ .
7: Sample random mini-batch of transition  $\mathcal{D}_t^{(m)}$  from the replay buffer  $\mathcal{D}_t$ .
8: Set  $y_{n} = r_{n} + \gamma \max_{a} Q(\pmb{W}^{(t,0)}; \pmb{s}_{n}', a)$  for  $n \in 1, 2, \dots, |\mathcal{D}_t^{(m)}|$ .
9: Perform a gradient descent step
$$
\boldsymbol {W} ^ {(t, m + 1)} = \boldsymbol {W} ^ {(t, m)} - \eta \cdot g _ {t} ^ {(m)} (\boldsymbol {W} ^ {(t, m)}) + \beta (\boldsymbol {W} ^ {(t, m)} - \boldsymbol {W} ^ {(t, m - 1)}).
$$
10: end for
11: Set  $\pmb{W}^{(t + 1,0)} = \pmb{W}^{(t,M)}$ .
The learning problem (6) is solved via the DQN equipped with  $\varepsilon$ -greedy exploration, as summarized in Algorithm 1. In  $t$ -th outer loop, we initialize the Q-value function using currently estimated weights  $\boldsymbol{W}^{(t,0)}$  as  $Q(\boldsymbol{W}^{(t,0)})$  (line 4). Then, for each inner loop, the agent selects and executes actions according to the  $\varepsilon$ -greedy policy (line 6), namely, with probability  $\varepsilon$ , we select a random action, and with probability  $1 - \varepsilon$ , we select the action based on the greedy policy with respect to  $Q(\boldsymbol{W}^{(t,0)})$ . The data are stored in a replay buffer with size  $N$  (line 7). Then, we sample a minibatch of independent samples from  $\mathcal{D}_t$ , denoted as  $\mathcal{D}_t^{(m)}$  for the  $m$ -th inner loop (line 8). Next, we update the current weights using a mini-batch (accelerated) gradient descent algorithm (line 9). The gradient descent (GD) direction in this step at point  $\boldsymbol{W}$  is represented as
$$
g _ {t} ^ {(m)} (\boldsymbol {W}) = \sum_ {n \in \mathcal {D} _ {t} ^ {(m)}} \left(Q (\boldsymbol {W}; \boldsymbol {s} _ {n}, a _ {n}) - y _ {n} ^ {(t)}\right) \cdot \nabla_ {\boldsymbol {W}} Q (\boldsymbol {W}; \boldsymbol {s} _ {n}, a _ {n}), \tag {7}
$$
$$
y _ {n} ^ {(t)} = r _ {n} + \gamma \cdot \max  _ {a ^ {\prime} \in \mathcal {A}} Q \left(\boldsymbol {W} ^ {(t, 0)}; \boldsymbol {s} _ {n} ^ {\prime}, a ^ {\prime}\right). \tag {8}
$$
Note that (7) can be viewed as the gradient of
$$
\mathbb {E} _ {(\boldsymbol {s}, a) \sim \mu_ {t}} \left(Q (\boldsymbol {W}; \boldsymbol {s}, a) - r - \mathbb {E} _ {\boldsymbol {s} ^ {\prime} | \boldsymbol {s}, a} \max  _ {a ^ {\prime}} Q \left(\boldsymbol {W} ^ {(t, 0)}; \boldsymbol {s} ^ {\prime}, a ^ {\prime}\right)\right) ^ {2}, \tag {9}
$$
which is the approximation to (6) via fixing the  $\max_{a} Q(W)$  as  $\max_{a} Q(W^{(t,0)})$ . After moving along the GD direction, accelerated gradient descent (AGD) adds a momentum term, denoted by  $\beta(W^{(t,m)} - W^{(t,m-1)})$  to accelerate the convergence rate [57]. Vanilla SGD can be viewed as a special case of AGD by letting  $\beta = 0$ . After updating neuron weights in the inner loop, we set the network as the currently estimated Q-value function  $Q(W^{(t,0)})$  (line 11) and repeat the steps above.
4.1 Takeaways of the Theoretical Findings
We consider the general setup of DQNs with  $\varepsilon$ -greedy under some commonly used assumptions. To the best of our knowledge, we provide the first theoretical characterization of both the convergence and sample complexity analysis for DQNs with  $\varepsilon$ -greedy. The major notations are summarized in Table 2. We first briefly introduce the key takeaways of our results, and the formal theoretical results are introduced in Section 4.3.
Table 2: Some Important Notations
K	Number of neurons in each hidden layer.	L	Number of the hidden layers.
d	Dimension of the feature mapping of (s,a).	Ns	The sample complexity for δ-optimal policy.
W*	The global optimal to (6).	et	The value of ||W(t,0) - W*||F.
cε	A small positive constant with a linear dependence on εt.	Ct	The fraction of actions with data at iteration t such that argmaxaQ(W(t,0);s,a) ≠ argmaxaQ(W*;s,a).
(T1) Theoretical characterizations of  $\{\varepsilon_t\}_{t=1}^T$  for convergence. We prove that for a wide selection of  $\{\varepsilon_t\}$  values that decrease over time, Algorithm 1 converges to  $Q^\star$  linearly up to some estimation error. Let  $c_\varepsilon$  measure the value level of  $\varepsilon_t$ 's. A higher level of  $\varepsilon$  values (i.e., a larger  $c_\varepsilon$ ) leads to an enlarged region of convergence (in the order of  $c_\varepsilon$ ), measured by the distance from the initialization  $\mathbf{W}^{(0,0)}$  to  $\mathbf{W}^*$ . Thus, larger  $\varepsilon$  values relax the requirements on  $\mathbf{W}^{(0,0)}$ . A lower level of  $\varepsilon$  values (i.e., a smaller  $c_\varepsilon$ ) leads to faster convergence with a rate in the order of  $c_\varepsilon$ . Our findings explain the intuition that the agent tends to explore more at the beginning and exploit more after gaining enough knowledge during the exploration.
(T2) Convergence to the optimal Q-value function  $Q^{\star}$  with geometric decay. The learned models converge to the ground truth model  $Q^{\star}$  with a geometric decay up to some bounded estimation error. The convergence rate is upper bounded by  $\gamma + c_{\varepsilon} \cdot (1 - \gamma)$ . When  $\gamma$  is close to one, the problem emphasizes long-term rewards. While the immediate reward can be observed directly, the future reward needs to be estimated by the learned Q function as shown in (7). Therefore, with a large  $\gamma$ , the learned Q function needs more iterations to converge, which leads to a slow convergence rate.
(T3) Sample complexity for achieving a desired estimation error of the optimal Q-value function. With the proper selection of  $\{\varepsilon_t\}_{t=1}^T$ , the estimation error of the learned model scales in the order of  $\frac{C_T}{(1 - \gamma)^2\sqrt{N_s}}$ .  $C_T$  is the fraction of actions following the current greedy policy that differ from the ones following the optimal policy.  $N_s$  is the number of samples. With a smaller discounted factor  $\gamma$ , the problem focuses more on the immediate reward, which can be observed directly, making  $Q^\star$  easier to learn. The learned model achieves a small estimation error given a small distribution shift, a large sample size, or a small discounted factor.
4.2 Assumptions
We propose assumptions commonly used in existing RL and neural network learning theories and notations to simplify the presentation. Assumption 1 assumes the existence of a good approximation of DNN to  $Q^{\star}$ , which guarantees (6) is solvable. The assumption is commonly used in both deep learning theories [90, 86] and reinforcement learning theories [46, 24]. Assumption 2 assumes the samples from experience replay are independent and identically distributed (i.i.d.), which follows the assumptions in existing theoretical analysis of DQN [24, 53] and matches the intuition of using experience replay to break temporal dependence among the samples. Specifically, we have
Assumption 1. There exists a DNN with weights  $\pmb{W}^{\star}$  such that minimizes (6) as  $f(\pmb{W}^{\star}) = 0$ .
Assumption 1 assumes that the Q-value function with some unknown ground truth  $W^{\star}$  can represent the optimal Q-value function.
Assumption 2. Suppose the mini-batch data are i.i.d. samples from the replay buffer following the distribution  $\mu_t$ , which is the stationary distribution of the behavior policy at  $t$ -th outer loop.
Assumption 2 assumes the mini-batch at the  $t$ -th outer loop are i.i.d. samples following  $\mu_t$ , where  $\mu_t$  is the stationary distribution of  $(s, a)$  generated by  $\varepsilon_t$ -greedy policy at  $t$ -th outer loop<sup>2</sup>. The minibatch samples are close to being independent given the experience size in practice is sufficiently large ( $\sim$  millions [51]).
In what follows, we define two quantities  $C_t$  and  $\rho$  to simplify the presentation of theoretical results.
Definition 1.  $C_t \in [0,1]$  is the fraction of non-optimal state-action pair  $(s,a)$  in the greedy policy with respect to  $Q(\pmb{W}^{(t,0)})$ , i.e., the fraction of  $(s,a)$  pairs that satisfy
$$
a = \underset {a ^ {\prime}} {\operatorname {a r g m a x}} Q \left(\boldsymbol {W} ^ {(t, 0)}; \boldsymbol {s}, a ^ {\prime}\right) \neq \pi^ {\star} (\boldsymbol {s}) \tag {10}
$$
among all  $(s, a)$  pairs following the greedy policy at  $t$ -th outer loop as  $a = \operatorname{argmax}_{a'} Q(\mathbf{W}^{(t,0)}; s, a')$ .