SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Value Divergence with Neural Network. In online reinforcement learning (RL), off-policy algorithms that employ value function approximation and bootstrapping can experience value divergence, a phenomenon known as the deadly triad [46; 4; 47; 48; 11]. Deep Q-Networks (DQN) typify this issue. As they employ neural networks for function approximation, they are particularly susceptible to Q-value divergence [19; 13; 48; 60]. Past research has sought to empirically address this divergence problem through various methods, such as the use of separate target networks [39] and Double-Q learning [19; 13]. Achiam et al. [1] analyze a linear approximation of Q-network to characterizes the diverge, while CrossNorm [6] uses a elaborated version of BatchNorm [20] to achieve stable learning. Value divergence becomes even more salient in offline RL, where the algorithm learns purely from a fixed dataset without additional environment interaction [14; 30]. Much of the focus in the field of offline RL has been on controlling the extent of off-policy learning, i.e., policy constraint [12; 40; 26; 14; 53; 51; 8; 42]. Several previous studies [41; 5; 25; 28] have empirically utilized LayerNorm to enhance performance in online and offline RL. These empirical results partially align with the experimental section of our work. However, our study makes a theoretical explanation for how LayerNorm mitigates divergence through the NTK analysis. Specifically, we empirically and theoretically illustrate how LayerNorm reduces SEEM. In addition to LayerNorm, our contribution extends to explaining divergence and proposing promising solutions from the perspective of reducing SEEM. Specially, we discover that WeightNorm can also be an effective tool and explain why other regularization techniques fall short. Finally, we perform comprehensive experiments to empirically verify the effectiveness of LayerNorm on the  $\% \mathrm{X}$  dataset, a practical setting not explored in previous work. Thus, our contributions are multifaceted and extend beyond the mere application of LayerNorm.
Offline RL. Traditional RL and bandit methods have a severe sample efficiency problem [32; 14; 36; 37]; offline RL aims to tackle this issue. However, offline RL presents significant challenges due to severe off-policy issues and extrapolation errors. Some existing methods focuses on designs explicit or implicit policy regularizations to minimize the discrepancy between the learned and behavior policies. For example, TD3+BC [13; 12; 55; 56] directly adds a behavior cloning loss to mimic the behavior policy, Diffusion-QL [51] further replace the BC loss with a diffusion loss and using diffusion models as the policy. AWR-style [42; 52; 7] and IQL [26] impose an implicit policy regularization by maximizing the weighted in-distribution action likelihood. Meanwhile, some other works try to alleviate the extrapolation errors by modifying the policy evaluation procedure. Specifically, CQL [31] penalizes out-of-distribution actions for having higher Q-values, while MISA [38] regularizes both policy improvement and evaluation with the mutual information between state-action pairs of the dataset. Alternatively, decision transformer (DT) [9] and trajectory transformer [22] cast offline RL as a sequence generation problem, which are beyond the scope of this paper. Despite the effectiveness of the above methods, they usually neglect the effect of function approximator and are thus sensitive to hyperparameters for trading off performance and training stability. Exploration into the function approximation aspect of the deadly triad is lacking in offline RL. Moreover, a theoretical analysis of divergence in offline RL that does not consider the function approximator would be inherently incomplete. We instead focus on this orthogonal perspective and provide both theoretical understanding and empirical solution to the offline RL problem.
B Proof of Main Theorems
Before proving our main theorem, we first state an important lemma.
Lemma 1. For  $L$ -layer ReLU-activate MLP and fixed input  $\pmb{x}, \pmb{x}'$ , assume the biases are relatively negligible compared to the values in the hidden layers. If we scale up every parameter of  $f_{\theta}$  to  $\lambda$  times, namely  $\pmb{\theta}' = \lambda \pmb{\theta}$ , then we have following equations hold for any  $\pmb{x}, \pmb{x}' \in S \times \mathcal{A}$
$$
\left\| f _ {\boldsymbol {\theta} ^ {\prime}} (\boldsymbol {x}) - \lambda^ {L} f _ {\boldsymbol {\theta}} (\boldsymbol {x}) \right\| = o (1) \cdot \| f _ {\boldsymbol {\theta} ^ {\prime}} (\boldsymbol {x}) \|
$$
$$
\left\| \nabla_ {\boldsymbol {\theta}} f _ {\boldsymbol {\theta} ^ {\prime}} (\boldsymbol {x}) - \lambda^ {L - 1} \nabla_ {\boldsymbol {\theta}} f _ {\boldsymbol {\theta}} (\boldsymbol {x}) \right\| = o (1) \cdot \| \nabla_ {\boldsymbol {\theta}} f _ {\boldsymbol {\theta} ^ {\prime}} (\boldsymbol {x}) \|,
$$
$$
\left| k _ {\boldsymbol {\theta} ^ {\prime}} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) - \lambda^ {2 (L - 1)} k _ {\boldsymbol {\theta}} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) \right| = o (1) \cdot k _ {\boldsymbol {\theta} ^ {\prime}} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}).
$$
Proof. Recursively define
$$
\boldsymbol {z} _ {\ell + 1} = \boldsymbol {W} _ {\ell} \tilde {\boldsymbol {z}} _ {\ell} + \boldsymbol {b} _ {\ell}, \quad \tilde {\boldsymbol {z}} _ {0} = \boldsymbol {x}
$$
$$
\tilde {z} _ {\ell} = \sigma (z _ {\ell}).
$$
Then it is easy to see that if we multiply each  $\mathbf{W}_{\ell}$  and  $\mathbf{b}_{\ell}$  by  $\lambda$ , denote the new corresponding value to be  $\mathbf{z}_{\ell}'$  and note that the bias term is negligible, we have
$$
\begin{array}{l} z _ {1} ^ {\prime} = \lambda z _ {1} \\ \boldsymbol {z} _ {2} ^ {\prime} = \lambda \boldsymbol {W} _ {2} \sigma \left(\boldsymbol {z} _ {1} ^ {\prime}\right) + \lambda \boldsymbol {b} _ {2} \\ = \lambda^ {2} \left(\boldsymbol {W} _ {2} \tilde {\boldsymbol {z}} _ {1} + \boldsymbol {b} _ {2}\right) + (\lambda - \lambda^ {2}) \boldsymbol {b} _ {2} \\ = \lambda^ {2} \boldsymbol {z} _ {2} + (\lambda - \lambda^ {2}) \boldsymbol {b} _ {2} \\ = \lambda^ {2} \boldsymbol {z} _ {2} + \epsilon_ {2} \quad (\text {n e g i l i b l e} \left\| \boldsymbol {\epsilon} _ {2} \right\| = o \left(\lambda^ {2}\right)) \\ \end{array}
$$
$$
\begin{array}{l} \boldsymbol {z} _ {\ell + 1} ^ {\prime} = \lambda \boldsymbol {W} _ {\ell} \sigma (\boldsymbol {z} _ {\ell} ^ {\prime}) + \lambda \boldsymbol {b} _ {\ell + 1} \\ = \lambda \boldsymbol {W} _ {\ell} \sigma \left(\lambda^ {\ell} \boldsymbol {z} _ {\ell} + \boldsymbol {\epsilon} _ {\ell}\right) + \lambda \boldsymbol {b} _ {\ell + 1} \\ = \lambda^ {\ell + 1} \boldsymbol {W} _ {\ell} \sigma \left(\boldsymbol {z} _ {\ell + 1} ^ {\prime}\right) + \lambda \boldsymbol {b} _ {\ell + 1} + \lambda \boldsymbol {W} _ {\ell} \delta_ {\ell} \quad (\delta_ {\ell} = \sigma \left(\lambda^ {\ell} \boldsymbol {z} _ {\ell} + \epsilon_ {\ell}\right) - \sigma \left(\boldsymbol {z} _ {\ell + 1} ^ {\prime}\right)) \\ = \lambda^ {\ell + 1} \boldsymbol {z} _ {\ell + 1} + (\lambda - \lambda^ {\ell + 1}) \boldsymbol {b} _ {\ell + 1} + \lambda \boldsymbol {W} _ {\ell} \delta_ {\ell} \\ \end{array}
$$
Note that  $\lambda \| \boldsymbol{w}_{\ell} \delta_{\ell} \| = \lambda \| \boldsymbol{W}_{\ell} (\sigma(\lambda^{\ell} \boldsymbol{z}_{\ell} + \epsilon_{\ell}) - \sigma(\boldsymbol{z}_{\ell+1}') \| \leq \lambda \| \boldsymbol{W}_{\ell} \| \cdot \| (\lambda^{\ell} \boldsymbol{z}_{\ell} + \epsilon_{\ell}) - \boldsymbol{z}_{\ell+1}'\|$ , the last step is because  $\sigma(\cdot)$  is 1-Lipshitz, and we have  $(\lambda^{\ell} \boldsymbol{z}_{\ell} + \epsilon_{\ell}) - \boldsymbol{z}_{\ell+1} = \boldsymbol{\epsilon}_{\ell}$  and  $\| \boldsymbol{\epsilon}_{\ell}\| = o(\lambda^{\ell})$ . When value divergence is about to occur, the dot product  $W_{i} z_{i}$  becomes relatively large compared to the scalar  $b_{i}$ , and the effect of bias on the output becomes negligible. Therefore, we can recursively deduce that the remaining error term
$$
(\lambda - \lambda^ {\ell + 1}) \boldsymbol {b} _ {\ell + 1} + \lambda \boldsymbol {W} _ {\ell} \delta_ {\ell}
$$
is always negligible compared with  $z_{\ell + 1}^{\prime}$ . Hence we know  $f_{\pmb{\theta}^{\prime}}(\pmb{x})$  is close to  $\lambda^{L}f_{\pmb{\theta}}(\pmb{x})$  with only  $o(1)$  relative error.
Taking gradient backwards, we know that  $\left\| \frac{\partial f}{\partial \tilde{W}_{\ell}} \right\|$  is proportional to both  $\| \tilde{z}_{\ell} \|$  and  $\left\| \frac{\partial f}{\partial z_{\ell + 1}} \right\|$ . Therefore we know
$$
\frac {\partial f _ {\boldsymbol {\theta} ^ {\prime}}}{\partial \boldsymbol {W} _ {\ell} ^ {\prime}} = \tilde {\boldsymbol {z}} _ {\ell} ^ {\prime} \frac {\partial f _ {\boldsymbol {\theta} ^ {\prime}}}{\partial \boldsymbol {z} _ {\ell + 1} ^ {\prime}} = \lambda^ {\ell} \tilde {\boldsymbol {z}} _ {\ell} \cdot \lambda^ {L - \ell - 1} \frac {\partial f _ {\boldsymbol {\theta}}}{\partial \boldsymbol {z} _ {\ell + 1}} = \lambda^ {L - 1} \frac {\partial f _ {\boldsymbol {\theta}}}{\partial \boldsymbol {W} _ {\ell}}.
$$
This suggests that all gradients with respect to the weights become scaled by a factor of  $\lambda^{L - 1}$ . The gradients with respect to the biases are proportional to  $\lambda^{L - l}$ . When  $\lambda$  is large enough to render the gradient of the bias term negligible, it follows that  $\left\| \nabla_{\theta}f_{\theta^{\prime}}(\boldsymbol {x}) - \lambda^{L - 1}\nabla_{\theta}f_{\theta}(\boldsymbol {x})\right\| = o(1)\cdot \| \nabla_{\theta}f_{\theta^{\prime}}(\boldsymbol {x})\|$ . This equation implies that the gradient updates for the model parameters are dominated by the weights, with negligible contribution from the bias terms. And since NTK is the inner product between gradients, we know  $G_{\theta^{\prime}}(\pmb {x},\pmb{x}^{\prime})$ ’s relative error to  $\lambda^{2(L - 1)}G_{\theta}(\pmb {x},\pmb{x}^{\prime})$  is  $o(1)$  negligible. We run experiments in Appendix F to validate the homogeneity. We leave analysis for Convolution Network [27] and its variants [43; 49; 50] for future work.
Theorem 1 Suppose that the network's parameter at iteration  $t$  is  $\pmb{\theta}_t$ . For each transition  $(s_i, a_i, s_{i+1}, r_i)$  in dataset, denote  $\pmb{r} = [r_1, \dots, r_M]^\top \in \mathbb{R}^M$ ,  $\hat{\pi}_{\pmb{\theta}_t}(s) = \arg \max_a \hat{Q}_{\pmb{\theta}_t}(s, a)$ . Denote  $\pmb{x}_{i,t}^* = (s_{i+1}, \hat{\pi}_{\pmb{\theta}_t}(s_{i+1}))$ . Concatenate all  $\pmb{x}_{i,t}^*$  to be  $\pmb{X}_t^*$ . Denote  $\pmb{u}_t = f_{\pmb{\theta}_t}(\pmb{X}) - (\pmb{r} + \gamma \cdot f_{\pmb{\theta}_t}(\pmb{X}_t^*))$  to be TD error vector at iteration  $t$ . The learning rate  $\eta$  is infinitesimal. When the maximal point of  $\hat{Q}_{\pmb{\theta}_t}$  is stable as  $t$  increases, we have the following evolving equation for  $\pmb{u}_{t+1}$
$$
\boldsymbol {u} _ {t + 1} = \left(\boldsymbol {I} + \eta \boldsymbol {A} _ {t}\right) \boldsymbol {u} _ {t}. \tag {3}
$$
where  $\mathbf{A} = \left(\gamma \phi_{\boldsymbol{\theta}_t}\bigl (\mathbf{X}_t^*\bigr) - \phi_{\boldsymbol{\theta}_t}\bigl (\mathbf{X}\bigr)\right)^\top \phi_{\boldsymbol{\theta}_t}\bigl (\mathbf{X}\bigr) = \gamma G_{\boldsymbol{\theta}_t}\bigl (\mathbf{X}_t^*,\mathbf{X}\bigr) - G_{\boldsymbol{\theta}_t}\bigl (\mathbf{X},\mathbf{X}\bigr).$
Proof. For the sake of simplicity, denote  $\mathbf{Z}_t = \nabla_\theta f_\theta(\mathbf{X})\big|_{\theta_t}, \mathbf{Z}_t^* = \nabla_\theta f_\theta(\mathbf{X}_t^*)\big|_{\theta_t}$ . The Q-value iteration minimizes loss function  $\mathcal{L}$  defined by  $\mathcal{L}(\boldsymbol{\theta}) = \frac{1}{2}\|f_\boldsymbol{\theta}(\mathbf{X}) - (\mathbf{r} + \gamma \cdot f_{\boldsymbol{\theta}_t}(\mathbf{X}_t^*))\|_2^2$ . Therefore we have the gradient as
$$
\frac {\partial \mathcal {L} (\boldsymbol {\theta})}{\partial \boldsymbol {\theta}} = \boldsymbol {Z} _ {t} \left(f _ {\boldsymbol {\theta}} (\boldsymbol {X}) - \left(\boldsymbol {r} + \gamma \cdot f _ {\boldsymbol {\theta} _ {t}} \left(\boldsymbol {X} _ {t} ^ {*}\right)\right)\right) = \boldsymbol {Z} _ {t} \boldsymbol {u} _ {t}. \tag {4}
$$
According to gradient descent, we know  $\pmb{\theta}_{t + 1} = \pmb{\theta}_t - \eta Z_t\pmb{u}_t$ . Since  $\eta$  is very small, we know  $\pmb{\theta}_{t + 1}$  stays within the neighborhood of  $\pmb{\theta}_t$ . We can Taylor-expand function  $f_{\pmb{\theta}}(\cdot)$  near  $\pmb{\theta}_t$  as
$$
f _ {\boldsymbol {\theta}} (\boldsymbol {X}) \approx \nabla_ {\boldsymbol {\theta}} ^ {\top} f _ {\boldsymbol {\theta}} (\boldsymbol {X}) \Big | _ {\boldsymbol {\theta} _ {t}} (\boldsymbol {\theta} - \boldsymbol {\theta} _ {t}) + f _ {\boldsymbol {\theta} _ {t}} (\boldsymbol {X}) = \boldsymbol {Z} _ {t} ^ {\top} (\boldsymbol {\theta} - \boldsymbol {\theta} _ {t}) + f _ {\boldsymbol {\theta} _ {t}} (\boldsymbol {X}). \tag {5}
$$
$$
f _ {\boldsymbol {\theta}} \left(\boldsymbol {X} _ {t} ^ {*}\right) \approx \left(\boldsymbol {Z} _ {t} ^ {*}\right) ^ {\top} \left(\boldsymbol {\theta} - \boldsymbol {\theta} _ {t}\right) + f _ {\boldsymbol {\theta} _ {t}} \left(\boldsymbol {X} _ {t} ^ {*}\right). \tag {6}
$$
When  $\eta$  is infinitesimally small, the equation holds. Plug in  $\theta_{t + 1}$ , we know
$$
f _ {\boldsymbol {\theta} _ {t + 1}} (\boldsymbol {X}) - f _ {\boldsymbol {\theta} _ {t}} (\boldsymbol {X}) = - \eta \boldsymbol {Z} _ {t} ^ {\top} \boldsymbol {Z} _ {t} \boldsymbol {u} _ {t} = - \eta \cdot \boldsymbol {G} _ {\boldsymbol {\theta} _ {t}} (\boldsymbol {X}, \boldsymbol {X}) \boldsymbol {u} _ {t}. \tag {7}
$$
$$
f _ {\boldsymbol {\theta} _ {t + 1}} \left(\boldsymbol {X} _ {t} ^ {*}\right) - f _ {\boldsymbol {\theta} _ {t}} \left(\boldsymbol {X} _ {t} ^ {*}\right) = - \eta \left(\boldsymbol {Z} _ {t} ^ {*}\right) ^ {\top} \boldsymbol {Z} _ {t} \boldsymbol {u} _ {t} = - \eta \cdot \boldsymbol {G} _ {\boldsymbol {\theta} _ {t}} \left(\boldsymbol {X} _ {t} ^ {*}, \boldsymbol {X}\right) \boldsymbol {u} _ {t}. \tag {8}
$$
According to Assumption 2, we know  $\| \pmb{X}_t^* - \pmb{X}_{t + 1}^*\| = o(\eta)$ . So we can substitute  $\pmb{X}_{t + 1}^{*}$  into  $\pmb{X}_t^*$  since it is high order infinitesimal. Therefore,  $\pmb{u}_{t + 1}$  boils down to
$$
\begin{array}{l} \boldsymbol {u} _ {t + 1} = f _ {\boldsymbol {\theta} _ {t + 1}} (\boldsymbol {X}) - \boldsymbol {r} - \gamma f _ {\boldsymbol {\theta} _ {t + 1}} \left(\boldsymbol {X} _ {t + 1} ^ {*}\right) (9) \\ = f _ {\boldsymbol {\theta} _ {t}} (\boldsymbol {X}) - \eta \cdot \boldsymbol {G} _ {\boldsymbol {\theta} _ {t}} (\boldsymbol {X}, \boldsymbol {X}) \boldsymbol {u} _ {t} - \boldsymbol {r} - \gamma \left(f _ {\boldsymbol {\theta} _ {t}} \left(\boldsymbol {X} _ {t} ^ {*}\right) - \eta \cdot \boldsymbol {G} _ {\boldsymbol {\theta} _ {t}} \left(\boldsymbol {X} _ {t} ^ {*}, \boldsymbol {X}\right) + o (\eta)\right) \boldsymbol {u} _ {t} (10) \\ = \underbrace {f _ {\boldsymbol {\theta} _ {t}} (\boldsymbol {X}) - \boldsymbol {r} - \gamma f _ {\boldsymbol {\theta} _ {t}} \left(\boldsymbol {X} _ {t} ^ {*}\right)} _ {\boldsymbol {u} _ {t}} + \eta \cdot \left(\gamma \boldsymbol {G} _ {\boldsymbol {\theta} _ {t}} \left(\boldsymbol {X} _ {t} ^ {*}, \boldsymbol {X}\right) - \boldsymbol {G} _ {\boldsymbol {\theta} _ {t}} (\boldsymbol {X}, \boldsymbol {X}) + o (\eta)\right) \boldsymbol {u} _ {t} (11) \\ = \left(\boldsymbol {I} + \eta \boldsymbol {A} _ {t}\right) \boldsymbol {u} _ {t}. (12) \\ \end{array}
$$
where  $\pmb {A} = \gamma \pmb{G}_{\pmb{\theta}_t}(\pmb {X}_t^*,\pmb {X}) - \pmb{G}_{\pmb{\theta}_t}(\pmb {X},\pmb {X}).$
Theorem 3 Suppose we use SGD optimizer for  $Q$ -value iteration with learning rate  $\eta$  to be infinitesimal. Given iteration  $t > t_0$ , and  $\pmb{A} = \gamma \bar{\pmb{G}} (\bar{\pmb{X}}^{*},\pmb {X}) - \bar{\pmb{G}} (\pmb {X},\pmb {X})$ , where  $\bar{\pmb{G}}$  is the Gram matrix under terminal kernel  $\bar{k}$ . The divergence of  $\pmb{u}_t$  is equivalent to whether there exists an eigenvalue  $\lambda$  of  $\pmb{A}$  such that  $\mathrm{Re}(\lambda) > 0$ . If converge, we have  $\pmb{u}_t = (\pmb {I} + \eta \pmb {A})^{t - t_0}\cdot \pmb{u}_{t_0}$ . Otherwise,  $\pmb{u}_t$  becomes parallel to the eigenvector of the largest eigenvalue  $\lambda$  of  $\pmb{A}$ , and its norm diverges to infinity at following order
$$
\left\| \boldsymbol {u} _ {t} \right\| _ {2} = O \left(\frac {1}{\left(1 - C ^ {\prime} \lambda \eta t\right) ^ {L / (2 L - 2)}}\right). \tag {13}
$$
Proof. According to Assumption 2, max action becomes stable after  $t_0$ . It implies  $\| \pmb{X}_t^* - \bar{\pmb{X}}^* \| = o(1)$ . The stability of the NTK direction implies that for some scalar  $k_t$  and the specific input  $\pmb{X}^*, \pmb{X}$ , we have  $G_{\theta_t}(\bar{\pmb{X}}^*, \pmb{X}) = k_t \bar{G}(\pmb{X}^*, \pmb{X}) + \epsilon_{t,1}$  and  $G_{\theta_t}(\pmb{X}, \pmb{X}) = k_t \bar{G}(\pmb{X}, \pmb{X}) + \epsilon_{t,2}$  where  $\| \epsilon_{t,i} \| = o(k_t)$ ,  $i = 1, 2$ . Therefore, we have
$$
\begin{array}{l} \boldsymbol {A} _ {t} = \gamma \boldsymbol {G} _ {\boldsymbol {\theta} _ {t}} \left(\boldsymbol {X} _ {t} ^ {*}, \boldsymbol {X}\right) - \boldsymbol {G} _ {\boldsymbol {\theta} _ {t}} (\boldsymbol {X}, \boldsymbol {X}) (14) \\ = \gamma \bar {G} \left(\boldsymbol {X} _ {t} ^ {*}, \boldsymbol {X}\right) - \bar {G} (\boldsymbol {X}, \boldsymbol {X}) + \left(\epsilon_ {t, 1} - \epsilon_ {t, 2}\right) (15) \\ = k _ {t} \left(\gamma \bar {\boldsymbol {G}} \left(\bar {\boldsymbol {X}} ^ {*}, \boldsymbol {X}\right) - \bar {\boldsymbol {G}} (\boldsymbol {X}, \boldsymbol {X})\right) + \left(\gamma \boldsymbol {\epsilon} _ {t, 3} + \boldsymbol {\epsilon} _ {t, 1} - \boldsymbol {\epsilon} _ {t, 2}\right) (16) \\ \left(\boldsymbol {\epsilon} _ {t, 3} = \bar {\boldsymbol {G}} \left(\bar {\boldsymbol {X}} _ {t} ^ {*}, \boldsymbol {X}\right) - \bar {\boldsymbol {G}} \left(\bar {\boldsymbol {X}} ^ {*}, \boldsymbol {X}\right)\right) \\ = k _ {t} \boldsymbol {A} + \boldsymbol {\delta} _ {t} (17) \\ \end{array}
$$
$k_{t}$  equals 1 if the training is convergent, but will float up if the model's predicted Q-value blows up. According to Assumption 2, the rest term  $\delta_{t} = \gamma \epsilon_{t,3} + \epsilon_{t,1} - \epsilon_{t,2}$  is of  $o(1)$  norm and thus negligible, we know all the eigenvalues of  $\pmb{I} + \eta \pmb{A}$  have form  $1 + \eta \lambda_{i}$ . Considering  $\eta$  is small enough, we have  $|1 + \eta \lambda_i|^2\approx 1 + 2\eta \operatorname {Re}(\lambda)$ . Now suppose if there does not exist eigenvalue  $\lambda$  of  $\pmb{A}$  satisfies  $\mathrm{Re}(\lambda) > 0$ , we have  $|1 + \eta \lambda_i|\leq 1$ . Therefore, the NTK will become perfectly stable so  $k_{t} = 1$  for  $t > t_0$ , and we have
$$
\begin{array}{l} \boldsymbol {u} _ {t} = (\boldsymbol {I} + \eta \boldsymbol {A} _ {t - 1}) \boldsymbol {u} _ {t - 1} = (\boldsymbol {I} + \eta \boldsymbol {A} _ {t - 1}) (\boldsymbol {I} + \eta \boldsymbol {A} _ {t - 2}) \boldsymbol {u} _ {t - 2} = \dots = \prod_ {s = t _ {0}} ^ {t - 1} (\boldsymbol {I} + \eta \boldsymbol {A} _ {s}) \boldsymbol {u} _ {t _ {0}} (18) \\ = \prod_ {s = t _ {0}} ^ {t - 1} (\boldsymbol {I} + \eta \boldsymbol {A}) \boldsymbol {u} _ {t _ {0}} = (\boldsymbol {I} + \eta \boldsymbol {A}) ^ {t - t _ {0}} \cdot \boldsymbol {u} _ {t _ {0}}. (19) \\ \end{array}
$$
Otherwise, there exists an eigenvalue for  $\mathbf{A}$  satisfying  $\mathrm{Re}(\lambda) > 0$ . Denote the one with the largest real part as  $\lambda$ , and  $\mathbf{v}$  to be the corresponding eigenvector. We know matrix  $\mathbf{I} + \eta \mathbf{A}$  also has left eigenvector  $\mathbf{v}$ , whose eigenvalue is  $1 + \eta \lambda$ . In this situation, we know after each iteration,  $\| \pmb{u}_{t+1} \|$  will become larger than  $\| \pmb{u}_t \|$ . Moreover, to achieve larger and larger prediction values, the model's parameter's norm  $\| \pmb{\theta}_t \|$  also starts to explode. We know  $\pmb{u}_t$  is homogeneous with respect to ReLU networks. The output  $f_{\pmb{\theta}_t}(\pmb{X})$  enlarges  $p^L$  times when  $\pmb{\theta}_t$  enlarges  $p$  times. When the reward values is small with respect to the divergent Q-value, TD error  $\pmb{u}_t = O(f_{\pmb{\theta}_t}(\pmb{X})) = O(\pmb{\theta}_t^L)$ . Besides, according to Lemma 1, we know  $k_t = O(\| \pmb{\theta}_t \|^{2(L-1)}) = O(\| \pmb{u}_t \|^{2(L-1)/L}) = O(\| \pmb{u}_t \|^{2-2/L})$ .
Denote  $g(\eta t) = \pmb{v}^{\top}\pmb{u}_t$ , left multiply  $\pmb{v}$  to equation  $\pmb{u}_{t + 1} = (\pmb {I} + \eta k_t\pmb {A})\pmb {u}_t$ . We have  $g(\eta t + \eta) = (1 + \eta \lambda k_{t})g(\eta t)$ . Since we know such iteration will let  $\pmb{u}_t$  to be dominated by  $\pmb{v}$  and align with  $\pmb{v}$ , we know  $g(\eta t) = O(\| \pmb{u}_t\|)$  for large  $t$ . Therefore  $k_{t} = O(\| \pmb{u}_{t}\|^{2(L - 1) / L}) = C\cdot g(\eta t)^{2 - 2 / L}$ . This boils down to  $g(\eta t + \eta) = g(\eta t) + C\eta \lambda g(\eta t)^2$ , which further becomes
$$
\frac {g (\eta t + \eta) - g (\eta t)}{\eta} = C \lambda g (\eta t) ^ {3 - 2 / L} \tag {20}
$$
Let  $\eta \rightarrow 0$ , we have an differential equation  $\frac{\mathrm{d}g}{\mathrm{d}t} = C\lambda g(t)^{3 - 2 / L}$ . When  $L = 1$ , the MLP network degenerates to a linear function. The solution of ODE is
$$
\left\| \boldsymbol {u} _ {t} \right\| = g (\eta t) = C ^ {\prime} e ^ {\lambda t}, \tag {21}
$$
reflecting the exponential growth under linear function that has been studied in previous works [47]. When  $L > 2$ , Solving this ODE gives
$$
g (t) = \frac {1}{\left(1 - C ^ {\prime} \lambda t\right) ^ {L / \left(2 L - 2\right)}}. \tag {22}
$$
So at an infinite limit, we know  $\| \pmb{u}_t\| = g(\eta t) = O\left(\frac{1}{(1 - C'\lambda\eta t)^{L / (2L - 2)}}\right)$ . Specially, for the experimental case we study in Figure 2 where  $L = 2$ , it reduces to  $O\left(\frac{1}{1 - C'\lambda\eta t}\right)$ . We conduct more experiments with  $L = 3$  in Appendix C.2 to verify our theoretical findings.
It is also worthwhile to point out another important mechanism that we found when the Q-value network is in a steady state of diverging. Although we assume that the learning rate  $\eta$  is infinitesimal, the arg max function is not  $L$ -Lipshitz. So why could both NTK function and optimal policy action  $\mathbf{X}_t^*$  converge in direction? We found that when the model enters such "terminal direction",  $(s_i, \hat{\pi}_{\theta_t}(s_i))$  not only becomes the max value state-action pair, but also satisfies
$$
\hat {\pi} _ {\boldsymbol {\theta} _ {t}} \left(s _ {i}\right) = \arg \max  _ {a \in \mathcal {A}} \sum_ {i = 1} ^ {M} k _ {\boldsymbol {\theta} _ {t}} \left(\boldsymbol {x} _ {a}, \boldsymbol {x} _ {i}\right), \quad \boldsymbol {x} _ {a} = \left(s _ {i}, a\right).
$$
This means each  $\boldsymbol{x}_{t,i}^{*} = (s_{i},\hat{\pi}_{\boldsymbol{\theta}_{t}}(s_{i}))$  also gain the maximum value increment among all the possible actions, maintaining its policy action property. Such stationary policy action for all state causes the model's parameter to evolve in static dynamics, which in turn reassures it to be policy action in the next iteration. Therefore, after the critical point  $t_0$ , model's parameter evolves linearly to infinity, its NTK and policy action becomes stationary, and all these factors contribute to the persistence of each other.
C More Observations and Deduction
C.1 Model Alignment
In addition to the findings presented in Theorem 1 and Theorem 3, we have noticed several intriguing phenomena. Notably, beyond the critical point, gradients tend to align along a particular direction, leading to an infinite growth of the model's parameters in that same direction. This phenomenon is supported by the observations presented in Figure 16, Figure 17, and Figure 18, where the cosine similarity between the current model parameters and the ones at the ending of training remains close to 1 after reaching a critical point, even as the norm of the parameters continually increases.
C.2 Terminal Time
Theorem 3 claims  $\| \pmb{u}_t\| = O(f_{\pmb{\theta}_t}(\pmb {X})) = O\left(\frac{1}{(1 - C'\lambda\eta t)^{L / (2L - 2)}}\right)$ , implying the relation
$$
1 / q ^ {(2 L - 2) / L} \propto 1 - C ^ {\prime} \lambda \eta t. \tag {23}
$$