SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Beides, it implies the existence of a "terminal time"  $\frac{1}{C' \eta \lambda}$  that the model must crash at a singular point. When the training approaches this singular point, the estimation value and the model's norm explode rapidly in very few steps. We have run an experiment with  $L = 2$  in Figure 2, from which we can see that Q-value's inverse proves to decay linearly and eventually becomes Nan at the designated time step. When  $L = 3$ , from our theoretical analysis, we have  $1 / q^{\frac{4}{3}} \propto 1 - C' \lambda \eta t$ . The experimental results in Figure 10 corroborate this theoretical prediction, where the inverse Q-value raised to the power of  $4/3$  is proportional to  $1 - C' \lambda \eta t$  after a critical point and it eventually reaches a NAN value at the terminal time step.
Figure 10: Linear decay with SGD and  $\mathrm{L} = 3$
C.3 Adam Case
In this section, we will prove that if the algorithm employs Adam as the optimizer, the model still suffers divergence. Moreover, we demonstrate that the norm of the network increases linearly, of which the slope is  $\eta \sqrt{P}$ , where  $P$  is the number of parameters and  $\eta$  is the learning rate. Also, the Q-value prediction will increase at  $L_{th}$ -polynomial's rate, where  $L$  is the number of layers of model  $f_{\theta}$ . Experimental results in Figure 4 verified our findings. Besides, we show that all runnings across D4RL environments represent the linear growth of the norm of the Q-network in Figure 16, Figure 17, and Figure 18.
Theorem 4. Suppose we use Adam optimizer for  $Q$ -value iteration and all other settings are the same as Theorem 3. After  $t > t_0$ , the model will diverge if and only if  $\lambda_{\max}(\mathbf{A}) > 0$ . If it diverges, we have  $\| \pmb{\theta}_t \| = \eta \sqrt{P} t + o(t)$  and  $\| \pmb{u}_t \| = \Theta (t^L)$  where  $P$  and  $L$  are the number of parameters and the number of layers for network  $f_{\theta}$ , respectively.
Proof. We only focus on the asymptotic behavior of Adam. So we only care about the dynamics for  $t > T$  for some large  $T$ . Also, at this regime, we know that the gradient has greatly aligned with the model parameters. So we assume that
$$
\nabla L \left(\theta_ {t}\right) = - C \cdot \theta_ {t}. \quad C > 0 \tag {24}
$$
Recall that each iteration of the Adam algorithm has the following steps.
$$
g _ {t} = \nabla L \left(\theta_ {t}\right), \tag {25}
$$
$$
m _ {t} = \beta_ {1} m _ {t - 1} + (1 - \beta_ {1}) g _ {t}, \tag {26}
$$
$$
v _ {t} = \beta_ {2} v _ {t - 1} + (1 - \beta_ {2}) g _ {t} ^ {2}, \tag {27}
$$
$$
\hat {m} _ {t} = \frac {m _ {t}}{1 - \beta_ {1} ^ {t}}, \tag {28}
$$
$$
\hat {v} _ {t} = \frac {v _ {t}}{1 - \beta_ {2} ^ {t}}, \tag {29}
$$
$$
\theta_ {t + 1} = \theta_ {t} - \frac {\eta}{\sqrt {\hat {v} _ {t}} + \epsilon} \hat {m} _ {t}. \tag {30}
$$
Instead of exactly solving this series, we can verify linear growth is indeed the terminal behavior of  $\theta_{t}$  since we only care about asymptotic order. Assume that  $\theta_{t} = kt$  for  $t > T$ , we can calculate  $m_{t}$  by dividing both sides of the definition of  $m_{t}$  by  $\beta_1^t$ , which gives
$$
\frac {m _ {t}}{\beta_ {1} ^ {t}} = \frac {m _ {t - 1}}{\beta_ {1} ^ {t - 1}} + \frac {1 - \beta_ {1}}{\beta_ {1} ^ {t}} g _ {t}. \tag {31}
$$
$$
\frac {m _ {t}}{\beta_ {1} ^ {t}} = \sum_ {s = 0} ^ {t} \frac {1 - \beta_ {1}}{\beta_ {1} ^ {s}} g _ {s}. \tag {32}
$$
$$
m _ {t} = - C \sum_ {s = 0} ^ {t} \left(1 - \beta_ {1}\right) \beta_ {1} ^ {t - s} k s = - k C t + o (t) \tag {33}
$$
, where  $g_{t}$  is given in Equation (24). Similarly, we have
$$
v _ {t} = k C ^ {2} t ^ {2} + o \left(t ^ {2}\right) \tag {34}
$$
Hence we verify that
$$
\theta_ {t + 1} - \theta_ {t} = - \eta \cdot \frac {m _ {t}}{1 - \beta_ {1} ^ {t}} \cdot \sqrt {\frac {1 - \beta_ {2} ^ {t}}{v _ {t}}} \rightarrow \eta \cdot \frac {k C t}{\sqrt {k ^ {2} C ^ {2} t ^ {2}}} = \eta
$$
therefore we know each iteration will increase each parameter by exactly constant  $\eta$ . This in turn verified our assumption that parameter  $\theta_{t}$  grows linearly. The slope for the overall parameter is thus  $\eta \sqrt{P}$ . This can also be verified in Figure 4. When we have  $\theta_{t} = \eta \sqrt{P}\bar{\theta}$ , where  $\bar{\theta}$  is the normalized parameter, we can further deduce the increasing order of the model's estimation. According to lemma 1, the Q-value estimation (also the training error) increase at speed  $O(t^{L})$ .
D LayerNorm's Effect on NTK
In this section, we demonstrate the effect of LayerNorm on SEEM. Our demonstration is just an intuitive explanation rather than a rigorous proof. We show that adding a LayerNorm can effectively reduce the NTK between any  $x_0$  and extreme input  $x$  down from linear to constant. Since each entry of Gram matrix  $G$  is an individual NTK value, we can informally expect that  $G(X_t^*, X)$ 's eigenvalue are greatly reduced when every individual NTK value between any  $x_0$  and extreme input  $x$  is reduced.
We consider a two-layer MLP. The input is  $\pmb{x} \in \mathbb{R}^{d_{in}}$ , and the hidden dimension is  $d$ . The parameters include  $\pmb{W} = [\pmb{w}_1, \dots, \pmb{w}_d]^\top \in \mathbb{R}^{d \times d_{in}}$ ,  $\pmb{b} \in \mathbb{R}^d$  and  $\pmb{a} \in \mathbb{R}^d$ . Since for the NTK value, the last layer's bias term has a constant gradient, we do not need to consider it. The forward function of the network is
$$
f _ {\boldsymbol {\theta}} (\boldsymbol {x}) = \sum_ {i = 1} ^ {d} a _ {i} \sigma \left(\boldsymbol {w} _ {i} ^ {\top} \boldsymbol {x} + b _ {i}\right).
$$
Proposition 1. For any input  $\pmb{x}$  and network parameter  $\theta$ , if  $\nabla_{\theta}f_{\theta}(\pmb{x}) \neq \mathbf{0}$ , then we have
$$
\lim  _ {\lambda \rightarrow \infty} k _ {\mathrm {N T K}} (\boldsymbol {x}, \lambda \boldsymbol {x}) = \Omega (\lambda) \rightarrow \infty . \tag {35}
$$
Proof. Denote  $z_{i} = \pmb{w}_{i}^{\top}\pmb{x} + b_{i}$ , according to condition  $\nabla_{\theta}f_{\theta}(\pmb{x})\neq \mathbf{0}$ , we know there must exist at least one  $i$  such that  $z_{i} > 0$ , denote this set as  $P$ . Now consider all the  $i\in [d]$  that satisfy  $z_{i} > 0$  and  $\pmb{w}_i^\top \pmb {x} > 0$  (otherwise take opposite sign of  $\lambda$ ), we have
$$
\left. \frac {\partial f}{\partial a _ {i}} \right| _ {\boldsymbol {x}} = \sigma \left(\boldsymbol {w} _ {i} ^ {\top} \boldsymbol {x} + b _ {i}\right) = \boldsymbol {w} _ {i} ^ {\top} \boldsymbol {x} + b _ {i}, \tag {36}
$$
$$
\left. \frac {\partial f}{\partial \boldsymbol {w} _ {i}} \right| _ {\boldsymbol {x}} = a _ {i} \boldsymbol {x}, \tag {37}
$$
$$
\left. \frac {\partial f}{\partial b _ {i}} \right| _ {\boldsymbol {x}} = a _ {i}. \tag {38}
$$
$$
\left. \frac {\partial f}{\partial a _ {i}} \right| _ {\lambda \boldsymbol {x}} = \sigma \left(\lambda \boldsymbol {w} _ {i} ^ {\top} \boldsymbol {x} + b _ {i}\right) = \lambda \boldsymbol {w} _ {i} ^ {\top} \boldsymbol {x} + b _ {i}, \tag {39}
$$
$$
\left. \frac {\partial f}{\partial \boldsymbol {w} _ {i}} \right| _ {\lambda \boldsymbol {x}} = \lambda a _ {i} \boldsymbol {x}, \tag {40}
$$
$$
\left. \frac {\partial f}{\partial b _ {i}} \right| _ {\lambda x} = a _ {i}. \tag {41}
$$
$$
\sum_ {i \in P} \left\langle \frac {\partial f (\boldsymbol {x})}{\partial \boldsymbol {\theta} _ {i}}, \frac {\partial f (\lambda \boldsymbol {x})}{\partial \boldsymbol {\theta} _ {i}} \right\rangle = \lambda \left(\left(\boldsymbol {w} _ {i} ^ {\top} \boldsymbol {x}\right) ^ {2} + b _ {i} \boldsymbol {w} _ {i} ^ {\top} \boldsymbol {x} + a _ {i} ^ {2} \| \boldsymbol {x} \| ^ {2}\right) + O (1) = \Theta (\lambda).
$$
Denote  $N = \{1, \ldots, d\} \setminus P$ . We know for every  $j \in N$  either  $\frac{\partial f(\boldsymbol{x})}{\partial a_j} = \frac{\partial f(\boldsymbol{x})}{\partial b_j} = \frac{\partial f(\boldsymbol{x})}{\partial \boldsymbol{w}_j} = 0$ , or  $\boldsymbol{w}_j^\top \boldsymbol{x} < 0$ . For the latter case, we know  $\lim_{\lambda \to \infty} \frac{\partial f(\lambda \boldsymbol{x})}{\partial a_j} = \frac{\partial f(\lambda \boldsymbol{x})}{\partial b_j} = \frac{\partial f(\lambda \boldsymbol{x})}{\partial \boldsymbol{w}_j} = 0$ . In both cases, we have
$$
\lim  _ {\lambda \rightarrow \infty} \sum_ {j \in N} \left\langle \frac {\partial f (\boldsymbol {x})}{\partial \boldsymbol {\theta} _ {j}}, \frac {\partial f (\lambda \boldsymbol {x})}{\partial \boldsymbol {\theta} _ {j}} \right\rangle = 0.
$$
Therefore, according to the definition of NTK, we have
$$
\lim  _ {\lambda \rightarrow \infty} k _ {\mathrm {N T K}} (\boldsymbol {x}, \lambda \boldsymbol {x}) = \lim  _ {\lambda \rightarrow \infty} \left\langle \frac {\partial f (\boldsymbol {x})}{\partial \boldsymbol {\theta} _ {i}}, \frac {\partial f (\lambda \boldsymbol {x})}{\partial \boldsymbol {\theta} _ {i}} \right\rangle = \Theta (\lambda) \rightarrow \infty .
$$
For the model equipped with LayerNorm, the forward function becomes
$$
f _ {\boldsymbol {\theta}} (\boldsymbol {x}) = \boldsymbol {a} ^ {\top} \sigma (\psi (\boldsymbol {W} \boldsymbol {x} + \boldsymbol {b})),
$$
where  $\psi (\cdot)$  is the layer normalization function defined as
$$
\psi (\boldsymbol {x}) = \sqrt {d} \cdot \frac {\boldsymbol {x} - \mathbf {1 1} ^ {\top} \boldsymbol {x} / d}{\left\| \boldsymbol {x} - \mathbf {1 1} ^ {\top} \boldsymbol {x} / d \right\|}.
$$
Denote  $P = I - 11^{\top} / d$ , note that the derivative of  $\psi (\cdot)$  is
$$
\dot {\psi} (\boldsymbol {x}) = \frac {\partial \psi (\boldsymbol {x})}{\partial \boldsymbol {x}} = \sqrt {d} \cdot \left(\frac {\boldsymbol {I}}{\| \boldsymbol {P} \boldsymbol {x} \|} - \frac {\boldsymbol {P} \boldsymbol {x} \boldsymbol {x} ^ {\top} \boldsymbol {P}}{\| \boldsymbol {P} \boldsymbol {x} \| ^ {3}}\right) \boldsymbol {P}. \tag {42}
$$
Specially, we have
$$
\psi (\lambda \boldsymbol {x}) = \sqrt {d} \cdot \frac {\lambda \boldsymbol {x} - \lambda \mathbf {1 1} ^ {\top} \boldsymbol {x} / d}{\lambda \| \boldsymbol {x} - \mathbf {1 1} ^ {\top} \boldsymbol {x} / d \|} = \psi (\boldsymbol {x}). \tag {43}
$$
Now we state the second proposition.
Proposition 2. For any input  $\pmb{x}$  and network parameter  $\pmb{\theta}$  and any direction  $\pmb{v} \in \mathbb{R}^{d_{in}}$ , if the network has LayerNorm, then we know there exists a universal constant  $C$ , such that for any  $\lambda \geq 0$ , we have
$$
k _ {\mathrm {N T K}} (\boldsymbol {x}, \boldsymbol {x} + \lambda \boldsymbol {v}) \leq C. \tag {44}
$$
Proof. Since for finite range, there always exists a constant upper bound, we just need to analyze the case for  $\lambda \to +\infty$  and shows that it is constant bounded. First compute  $\nabla_{\theta}f_{\theta}(x)$  and get
$$
\left. \frac {\partial f}{\partial \boldsymbol {a}} \right| _ {\boldsymbol {x}} = \sigma (\psi (\boldsymbol {W} \boldsymbol {x} + \boldsymbol {b})), \tag {45}
$$
$$
\left. \frac {\partial f}{\partial \boldsymbol {W}} \right| _ {\boldsymbol {x}} = \boldsymbol {a} ^ {\top} \sigma^ {\prime} (\psi (\boldsymbol {W} \boldsymbol {x} + \boldsymbol {b})) \dot {\psi} (\boldsymbol {W} \boldsymbol {x} + \boldsymbol {b}) \boldsymbol {x}, \tag {46}
$$
$$
\left. \frac {\partial f}{\partial \boldsymbol {b}} \right| _ {\boldsymbol {x}} = \boldsymbol {a} ^ {\top} \sigma^ {\prime} (\psi (\boldsymbol {W} \boldsymbol {x} + \boldsymbol {b})) \dot {\psi} (\boldsymbol {W} \boldsymbol {x} + \boldsymbol {b}). \tag {47}
$$
These quantities are all constant bounded. Next we compute  $\lim_{\lambda \to \infty} \nabla_{\pmb{\theta}} f_{\pmb{\theta}}(\pmb{x} + \lambda \pmb{v})$
$$
\left. \frac {\partial f}{\partial \boldsymbol {a}} \right| _ {\boldsymbol {x} + \lambda \boldsymbol {v}} = \sigma (\psi (\boldsymbol {W} (\boldsymbol {x} + \lambda \boldsymbol {v}) + \boldsymbol {b}))), \tag {48}
$$
$$
\left. \frac {\partial f}{\partial \boldsymbol {W}} \right| _ {\boldsymbol {x} + \lambda \boldsymbol {v}} = \boldsymbol {a} ^ {\top} \sigma^ {\prime} (\psi (\boldsymbol {W} (\boldsymbol {x} + \lambda \boldsymbol {v}) + \boldsymbol {b}))) \dot {\psi} (\boldsymbol {W} (\boldsymbol {x} + \lambda \boldsymbol {v}) + \boldsymbol {b}) (\boldsymbol {x} + \lambda \boldsymbol {v}), \tag {49}
$$
$$
\left. \frac {\partial f}{\partial \boldsymbol {b}} \right| _ {\boldsymbol {x} + \lambda \boldsymbol {v}} = \boldsymbol {a} ^ {\top} \sigma^ {\prime} (\psi (\boldsymbol {W} \boldsymbol {x} + \boldsymbol {b})) \dot {\psi} (\boldsymbol {W} (\boldsymbol {x} + \lambda \boldsymbol {v}) + \boldsymbol {b}). \tag {50}
$$
According to the property of LayerNorm in Equation (43), we have
$$
\begin{array}{l} \left. \varlimsup_ {\lambda \rightarrow \infty} \frac {\partial f}{\partial \boldsymbol {a}} \right| _ {\boldsymbol {x} + \lambda \boldsymbol {v}} = \varliminf_ {\lambda \rightarrow \infty} \sigma (\psi (\boldsymbol {W} (\boldsymbol {x} + \lambda \boldsymbol {v}) + \boldsymbol {b})) (51) \\ = \sigma (\psi (\boldsymbol {W} (\lambda \boldsymbol {v}))) (52) \\ = \sigma (\psi (\boldsymbol {W} \boldsymbol {v})) = \text {C o n s t a n t} (53) \\ \end{array}
$$
$$
\begin{array}{l} \left. \frac {\lim  _ {\lambda \rightarrow \infty} \frac {\partial f}{\partial \boldsymbol {W}}}{\partial \boldsymbol {W}} \right| _ {\boldsymbol {x} + \lambda \boldsymbol {v}} = \overline {{\lim  _ {\lambda \rightarrow \infty}}} \boldsymbol {a} ^ {\top} \sigma^ {\prime} (\psi (\boldsymbol {W} (\boldsymbol {x} + \lambda \boldsymbol {v}) + \boldsymbol {b}))) \dot {\psi} (\boldsymbol {W} (\boldsymbol {x} + \lambda \boldsymbol {v}) + \boldsymbol {b}) (\boldsymbol {x} + \lambda \boldsymbol {v}) (54) \\ = \varlimsup_ {\lambda \rightarrow \infty} \boldsymbol {a} ^ {\top} \sigma^ {\prime} (\psi (\boldsymbol {W} \boldsymbol {v}))) \dot {\psi} (\boldsymbol {W} (\boldsymbol {x} + \lambda \boldsymbol {v}) + \boldsymbol {b}) (\boldsymbol {x} + \lambda \boldsymbol {v}) (55) \\ = \varlimsup_ {\lambda \rightarrow \infty} \boldsymbol {a} ^ {\top} \sigma^ {\prime} (\psi (\boldsymbol {W} \boldsymbol {v}))) \sqrt {d} \cdot \left(\frac {\boldsymbol {I}}{\| \boldsymbol {P} \lambda \boldsymbol {W} \boldsymbol {v} \|} - \frac {\boldsymbol {P} (\lambda \boldsymbol {W} \boldsymbol {v}) (\lambda \boldsymbol {W} \boldsymbol {v}) ^ {\top} \boldsymbol {P}}{\| \boldsymbol {P} (\lambda \boldsymbol {W} \boldsymbol {v}) \| ^ {3}}\right) \boldsymbol {P} (\boldsymbol {x} + \lambda \boldsymbol {v}) (56) \\ = \varlimsup_ {\lambda \rightarrow \infty} \boldsymbol {a} ^ {\top} \sigma^ {\prime} (\psi (\boldsymbol {W} \boldsymbol {v}))) \sqrt {d} \cdot \left(\frac {\boldsymbol {P} (\boldsymbol {x} + \lambda \boldsymbol {v})}{\lambda \| \boldsymbol {P W} \boldsymbol {v} \|} - \frac {\boldsymbol {P W} \boldsymbol {v} \boldsymbol {v} ^ {\top} \boldsymbol {W} ^ {\top} \boldsymbol {P} (\boldsymbol {x} + \lambda \boldsymbol {v})}{\lambda \| \boldsymbol {P W} \boldsymbol {v} \| ^ {3}}\right) (57) \\ = \boldsymbol {a} ^ {\top} \sigma^ {\prime} (\psi (\boldsymbol {W} \boldsymbol {v}))) \sqrt {d} \cdot \left(\frac {\boldsymbol {P} \boldsymbol {v}}{\| \boldsymbol {P} \boldsymbol {W} \boldsymbol {v} \|} - \frac {\boldsymbol {P} \boldsymbol {W} \boldsymbol {v} \boldsymbol {v} ^ {\top} \boldsymbol {W} ^ {\top} \boldsymbol {P} \boldsymbol {v}}{\| \boldsymbol {P} \boldsymbol {W} \boldsymbol {v} \| ^ {3}}\right) (58) \\ = \text {C o n s t a n t}. (59) \\ \end{array}
$$
$$
\begin{array}{l} \left. \overline {{\lim  _ {\lambda \rightarrow \infty} \frac {\partial f}{\partial \boldsymbol {b}}} \right| _ {\boldsymbol {x} + \lambda \boldsymbol {v}} = \overline {{\lim  _ {\lambda \rightarrow \infty} \boldsymbol {a} ^ {\top} \sigma^ {\prime} (\psi (\boldsymbol {W} \boldsymbol {v})))}} \sqrt {d} \cdot \left(\frac {\boldsymbol {I}}{\lambda \| \boldsymbol {P W} \boldsymbol {v} \|} - \frac {\boldsymbol {P W} \boldsymbol {v} \boldsymbol {W} \boldsymbol {v}) ^ {\top} \boldsymbol {P}}{\lambda \| \boldsymbol {P} (\boldsymbol {W} \boldsymbol {v}) \| ^ {3}}\right) \boldsymbol {P} (60) \\ = 0. (61) \\ \end{array}
$$
Therefore we know its limit is also constant bounded. So we know there exists a universal constant with respect to  $\pmb{\theta},\pmb{x},\pmb{v}$  such that  $k_{\mathrm{NTK}}(\pmb {x},\pmb {x} + \lambda \pmb {v}) = \left\langle \frac{\partial f(\pmb{x})}{\partial\pmb{\theta}_i},\frac{\partial f(\pmb{x} + \lambda\pmb{v})}{\partial\pmb{\theta}_i}\right\rangle \leq C.$
SEEM Experiments For the experiments presented in Section Section 3.1, we adopted TD3 as our baseline, but with a modification: instead of using an exponential moving average (EMA), we directly copied the current Q-network as the target network. The Adam optimizer was used with a learning rate of  $0.0003$ ,  $\beta_{1} = 0.9$ , and  $\beta_{2} = 0.999$ . The discount factor,  $\gamma$ , was set to 0.99. Our code builds upon the existing TD3+BC framework, which can be found at https://github.com/sfujim/TD3_BC.
SEEM Reduction Experiments For the experiments discussed in Section Section 4, we maintained the same configuration as in the SEEM experiments, with the exception of adding regularizations and normalizations. LayerNorm was implemented between the linear and activation layers with learnable affine parameters, applied to all hidden layers excluding the output layer. WeightNorm was applied to the output layer weights.
Offline RL Algorithm Experiments For the X% Mujoco Locomotion offline dataset experiments presented in Section Section 5, we used true offline RL algorithms including TD3+BC, IQL, Diff-QL, and CQL as baselines. We implement our method on the top of official implementations of TD3+BC and IQL; for CQL, we use a reliable JAX implementation at https://github.com/sail-sg/offbench [25]; for Diffusion Q-learning (i.e., combining TD3+BC and diffusion policy), we use an efficient JAX implementation EDP [24]. LayerNorm was directly added to the critic network in these experiments. For the standard Antmaze experiment, we assign a reward  $+10$  to the successful transitions, otherwise 0. Considering the extreme suboptimality in Antmaze medium and large datasets, we adopt Return-based Offline Prioritized Replay (OPER-R) with  $p_{\mathrm{base}} = 0$  to rebalance these suboptimal datasets [58; 59; 18]. Specifically, instead of uniform sampling during policy improvement, we assign a higher probability to the transitions in the successful trajectories. Meanwhile, a uniform sampler is still used for policy evaluation. When removing OPER-R, diff-QL with LayerNorm can still achieve a stable performance with slightly lower scores (see Table 2).
Linear Decay of Inverse Q-value with SGD Given that the explosion in D4RL environments occurs very quickly in the order of  $\frac{1}{1 - C' \lambda \eta t}$  and is difficult to capture, we opted to use a simple toy task for these experiments. The task includes a continuous two-dimensional state space  $s = (x_1, x_2) \in S = R^2$ , where the agent can freely navigate the plane. The action space is discrete, with 8 possible actions representing combinations of forward or backward movement in two directions. Each action changes the state by a value of 0.01. All rewards are set to zero, meaning that the true Q-value should be zero for all state-action pairs. For this task, we randomly sampled 100 state-action pairs as our offline dataset. The Q-network was implemented as a two-layer MLP with a hidden size of 200. We used SGD with a learning rate of 0.01, and the discount factor,  $\gamma$  was set to 0.99.
Table 2: Ablate on Antmaze. The best score metrics is presented in the parenthesis.
Dataset	diff-QL	ours w.o. OPER-R	ours
antmaze-umaze-v0	95.6 (96.0)	94.3	94.3 ± 0.5 (97.0)
antmaze-umaze-diverse-v0	(84.0)	88.5	88.5 ± 6.1 (95.0)
antmaze-medium-play-v0	0.0 (79.8)	77.0	85.6 ± 1.7 (92.0)
antmaze-medium-diverse-v0	6.4 (82.0)	70.4	83.9 ± 1.6 (90.7)
antmaze-large-play-v0	1.6 (49.0)	47.8	65.4 ± 8.6 (74.0)
antmaze-large-diverse-v0	4.4 (61.7)	60.3	67.1 ± 1.8 (75.7)
average	29.6 (75.4)	73.1	80.8 (87.4)
Figure 11: SAC in the online setting.
F.1 Validate the homogeneity in Lemma 1.
We define a family of 3-layer ReLU-activated MLPs (with bias term) with different scaling  $\lambda$  of the same network parameter  $\theta$  (from D4RL training checkpoints when value divergence happens), and
feed these networks with the same input. We show one example in the below table, which confirms that the  $L$ -degree homogeneity of output, gradient and NTK is valid at high precision, the NTK is almost parallel, too. The larger the scaling factor is, the more accurate  $k^L$  increasing pattern is for output ( $k^{L-1}$  for gradient and  $k^{2(L-1)}$  for NTK). Further, we empirically validated the  $L$ -degree homogeneity of NTK holds for all checkpoints in D4RL experiments where divergence happens.
Table 3: Validate homogeneity in Lemma 1.
λ	5	10
fλθ(x)/fθ(x)	124.99 (53)	999.78 (103)
grad scale	25.001 (52)	100.001 (102)
NTK scale	624.99(54)	9999.84 (104)
NTK cos	1 - 10-6	1 - 10-6
F.2 Ablate the contributions of EMA, Double-Q, and LayerNorm.
We conducted an ablation study in the following sequence: "DDPG without EMA", "DDPG + EMA", "DDPG + EMA + DoubleQ", and "DDPG + EMA + DoubleQ + LayerNorm". Each successive step built upon the last. We carried out these experiments in settings of  $10\%$  and  $50\%$  Mujoco - environments notably susceptible to divergence. We report the total scores across nine tasks in Figure 12. It clearly reveals that the EMA and double-Q, which are used as common practice in popular offline RL algorithms such as CQL, TD3+BC contribute to preventing divergence and thus boosting performance. Nevertheless, built upon EMA and DoubleQ, our LayerNorm can further significantly boost the performance by surpassing divergence.
Figure 12: Ablate EMA, Double-Q, and LayerNorm on  $10\%$  and  $50\%$  Mujoco datasets.
F.3 Benchmarking Normalizations.
Previously, we have demonstrated that LayerNorm, BatchNorm, and WeightNorm can effectively maintain a low SEEM and stabilize Q convergence in Section 4. Our next goal is to identify the most suitable regularization method for the value network in offline RL. Prior research has shown that divergence is correlated with poor control performance[48; 19]. In this context, we evaluate the effectiveness of various regularization techniques based on their performance in two distinct settings - the Antmaze task and the X% Mujoco dataset we mentioned above. Previous offline RL algorithms have not performed particularly well in these challenging scenarios. As displayed in Figure 13, TD3+BC, when coupled with layer normalization or batch normalization, yields significant performance enhancement on the  $10\%$  Mujoco datasets. The inability of batch normalization to improve the performance might be attributed to the oscillation issue previously discussed in Section 4. In the case of Antmaze tasks, which contain numerous suboptimal trajectories, we select TD3 with a diffusion policy, namely Diff-QL [51], as our baseline. The diffusion policy is capable of capturing multi-modal behavior. As demonstrated in Figure 13 and Table 4, LayerNorm can markedly enhance performance on challenging Antmaze tasks. In summary, we empirically find LayerNorm to be a suitable normalization for the critic in offline RL.
Figure 13: Normalizations effect on  $10\%$  Mujoco Locomotion Datasets.
Table 4: Normalizations effect on two challenging Antmaze tasks.
Dataset	diff-QL	LN	BN	WN
antmaze-large-play-v0	1.6	72.7	1.0	35.0
antmaze-large-diverse-v0	4.4	66.5	2.1	42.5
F.4 How LayerNorm should be added.