SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption 2.8 (Markov chain mixing). There exists constants  $C > 0$  and  $\beta \in [0,1)$  with the following property: for all  $\theta$ , if we consider a Markov chain generated by  $a_{t} \sim \pi(\cdot | s_{t}, \theta)$ ,  $s_{t+1} \sim P_{\mathrm{env}}(\cdot | s_{t}, a_{t})$  starting from state  $s$ , then
$$
\left| \left| p _ {\tau} - \mu_ {\theta} ^ {\prime} \right| \right| _ {1} \leq C \beta^ {\tau}, \forall \tau \geq 0, \forall s \in S,
$$
where  $p_{\tau}$  is the probability distribution of the state of this Markov chain after  $\tau$  steps.
To assure AC explores every possible state, we make the following assumption:
Assumption 2.9. (Exploration) Suppose there exists some constant  $\mu_{\mathrm{min}} > 0$  such that, for all  $\theta$ ,  $\mu_{\theta}^{\prime}$  is uniformly bounded away from 0. In other words,
$$
\mu_ {\theta} ^ {\prime} \geq \mu_ {\min} > 0, \forall \theta .
$$
Recall that  $\mu_{\theta}$  was defined earlier to be the stationary distribution of the transition matrix associated with the policy  $\pi_{\theta}$ . A key point is that the constants  $C, \beta$  and  $\mu_{\mathrm{min}}$  in the above assumptions do not depend on  $\theta$ .
We note that there is some redundancy in our assumptions. As discussed above, we require  $\ln \pi_{\theta}(a|s)$  to have a smooth gradient for all  $s, a$ , which ensures that  $\pi_{\theta}$  assigns a strictly positive probability to every action. This implies Assumptions 2.8 and 2.9 which can therefore be made into propositions. Nevertheless, we explicitly make Assumptions 2.8 and 2.9 (even though both of them are actually implied by our earlier assumption) since the quantities appearing in them (specifically, the mixing time  $\beta$  and the constant  $\mu_{\mathrm{min}}$ ) appear in various bounds we will derive.
More precisely, we follow the earlier literature by setting  $C\beta^{\tau}$  to be proportional to  $T^{-0.5}$ , the typical of stepsize in Stochastic Gradient Descent. We call the smallest  $\tau$  such that  $C\beta^{\tau} \leq O\left(T^{-0.5}\right)$  the mixing time and denote it by  $\tau_{\mathrm{mix}}$ . It is easy to see that  $\tau_{\mathrm{mix}} = O\left((1 - \beta)^{-1}\log T\right)$ . The quantity  $\tau_{\mathrm{mix}}$  will appear throughout our paper.
2.6  $D$ -norm and Dirichlet Norm in MDPs
A key ingredient is our analysis is the choice of norm: we have found that a certain norm originally introduced in Ollivier (2018) significantly simplifies analysis of the problem. We next introduce this norm and state our assumptions about it.
Let  $D_{\theta} = \mathrm{diag}(\mu_{\theta}(s,a))$  be the diagonal matrix whose elements are given by the entries of the stationary distribution  $\mu_{\theta}$  associated with the policy  $\pi_{\theta}$ . Given a function  $f: S \times A \to \mathbb{R}$ , its  $D$ -norm is defined as
$$
\left\| f \right\| _ {D} ^ {2} = f ^ {T} D _ {\theta} f = \sum_ {(s, a) \in S \times A} \mu_ {\theta} (s, a) f (s, a) ^ {2}. \tag {10}
$$
The  $D$ -norm is similar to the Euclidean norm except each entry is weighted proportionally to the stationary distribution. We also define the Dirichlet semi-norm of  $f$ :
$$
\left| \left| f \right| \right| _ {\text {D i r}} ^ {2} = \frac {1}{2} \sum_ {(s, a), (s ^ {\prime}, a ^ {\prime}) \in S \times A} \mu_ {\theta} (s, a) P _ {\theta} \left(s ^ {\prime}, a ^ {\prime} \mid s, a\right) \left(f \left(s ^ {\prime}, a ^ {\prime}\right) - f (s, a)\right) ^ {2}. \tag {11}
$$
A semi-norm satisfies the axioms of a norm except that it may be equal to zero at a non-zero vector. Note that  $||f||_{\mathrm{Dir}}$  depends on the policy both through the stationary distribution  $\mu_{\theta}(s,a)$  as well as through the transition matrix  $P_{\theta}$ .
Finally, following Ollivier (2018), the weighted combination of the  $D$ -norm and the Dirichlet seminorm is denoted as  $\mathcal{N}_{\theta}(f)$  will be defined
$$
\mathcal {N} _ {\theta} (f) = (1 - \gamma) \| f \| _ {D} ^ {2} + \gamma \| f \| _ {\operatorname {D i r}} ^ {2}. \tag {12}
$$
Note that as long as  $\mu_{\theta}(s,a) > 0$ , which is stated in Assumption 2.9, for all  $s,a$ , we have that  $\sqrt{\mathcal{N}_{\theta}(f)}$  is a valid norm.
3 Our Main Results
To simplify the expression that follows, we will adopt the notations  $\Delta_V$  and  $\Delta_Q$  for the two losses that we want to bound in our paper:
$$
\Delta_ {V} = \frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \left| \left| \nabla V _ {\theta_ {t}} ^ {*} \right| \right| ^ {2} \right], \quad \Delta_ {Q} = \frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \mathcal {N} _ {\theta_ {t}} \left(Q \left(w _ {t}\right) - \hat {Q} _ {\theta_ {t}} ^ {*}\right) \right]. \tag {13}
$$
Intuitively,  $\Delta_V$  corresponds to the actor error: ideally, we want to reach a point where the gradient of the actor value function is zero. Note that, since the value function is not convex in general, the actor error is measured in terms of distance to a stationary point as above.
Similarly,  $\Delta_{Q}$  is a measure of the critic error: it equals zero precisely if  $Q(w_{t})$ , the approximator of  $Q$ -function, equals  $\hat{Q}_{\theta_t}$ . Of course, as discussed above, the critic neural network may not be able to perfectly represent the true  $Q$ -function. Now we are ready to state our main results.
Theorem 3.1. Consider the neural AC algorithm mentioned in Section 2.4. Suppose Assumptions 2.1-2.9 hold and the step-sizes  $\alpha^{\theta}$  and  $\alpha^{w}$  are both chosen to decay proportionally to  $O\left(T^{-0.5}\right)$ .
1. In the i.i.d. sampling case,
$$
\Delta_ {V} \leq O \left(\frac {1}{\sqrt {T}}\right) + O (\epsilon) + \tilde {O} \left(\frac {1}{\sqrt {m}}\right), \quad \Delta_ {Q} \leq O \left(\frac {1}{\sqrt {T}}\right) + O (\epsilon) + \tilde {O} \left(\frac {1}{\sqrt {m}}\right).
$$
2. In the Markov sampling case,
$$
\Delta_ {V} \leq O \left(\frac {(\log T) ^ {2}}{\sqrt {T}}\right) + O (\epsilon) + \tilde {O} \left(\frac {1}{\sqrt {m}}\right), \quad \Delta_ {Q} \leq O \left(\frac {(\log T) ^ {2}}{\sqrt {T}}\right) + O (\epsilon) + \tilde {O} \left(\frac {1}{\sqrt {m}}\right).
$$
In all  $O(\cdot)$  notations above, we treat factors that do not depend on  $T,\epsilon ,m$  as constants.
We next provide a more detailed comparison to the previous works of (Wang et al. (2019); Cayci et al. (2022)). Our discussion partially reprises the discussion in the Introduction, but can now be discussed at a greater level of detail:
- Arbitrary depth/single-timescale. The main contribution of this paper to provide an analysis that applies to neural networks of arbitrary depth. Moreover, we do so in a single-loop/single-timescale method where the critic and actor iterate simultaneously, which is matching what is typically done in practice. Such an analysis is inherently more technically challenging, since when the actor can wait for the critic to go through sufficiently many iterations, one could argue that the resulting  $Q$ -values are approximately accurate and the process resembles gradient descent.
- Representability. Both previous works for the single-layer case assume the  $Q$ -function lies in some function class, which, as discussed after Assumption 6 in Farahmand et al. (2016), is one kind of "no function approximation error" assumption. By contrast, we make no such assumption: rather we allow any approximation error for the critic  $\epsilon$ , and our final result is given in terms of  $\epsilon$ .
- Lower bound on  $m$ . Previous works require  $m$ , the width of neural network, to be sufficiently large. In Wang et al. (2019), given that  $m$  is sufficiently large, Section 3.1 and Corollary A.3 argue that the gradient, denoted by  $\bar{\phi}_{\theta}$  and  $\bar{\phi}_w$ , can be well approximated by the "centered feature mapping corresponding to the initialization", denoted by  $\bar{\phi}_0$ . In Cayci et al. (2022), this dependency is even more emphasised since the upper bound shown in Theorem 2 could diverge with small  $m$ .
- Relation to NTK theory. NTK theory (Jacot et al. (2018)) tells us that neural networks get more linear as  $m \to \infty$ . The classic analyses of this proceed by arguing that as  $m \to \infty$ , the neural network stays close to its initialization during training Chizat et al. (2019). In that sense, we should expect to get a convergence result for AC as  $m \to \infty$ , but if the critic neural network stays close to its initial condition, the algorithm will effectively be using random linear features at initialization. For this reason, it is desirable not to argue that the critic neural network always stays close to its initial condition. We do not use such an argument in this work, whereas both Wang et al. (2019) and Cayci et al. (2022) obtain their results by arguing that the critic neural network stays close to its initial condition. This theoretical distinction is shown in Tian et al. (2023) to match what happens in simulations, which shows empirically that even for projected neural TD, the critic neural network will move to the boundary of the projection ball.
- Linearization. Previous works assume some kind of linearization around the initial point. The objective is explicitly linearized in Wang et al. (2019). In Cayci et al. (2022), while the objective is not linearized, the neural networks weights are projected onto a radius of size  $O(1 / \sqrt{m})$  around the initial point.
4 Tools in Our Analysis
4.1 Choice of Norm and Gradient Splitting
A linear function  $h(\theta)$  is said to be a gradient splitting of a convex quadratic  $f(\theta)$  minimized at  $\theta = a$  if
$$
\frac {1}{2} \nabla f (\theta) ^ {T} (a - \theta) = h (\theta) ^ {T} (\theta - a). \tag {14}
$$
In other words, a splitting  $h(\theta)$  has exactly the same inner product with the "direction to the optimal solution" as the true gradient of  $f(\theta)$  (up to the factor of  $1/2$ ). The connection between this idea and RL was made in the following papers:
Figure 1: Key property of gradient splitting:  $h(\theta)$  has the same inner product with  $a - \theta$  as  $\nabla f(\theta)$  up to a factor of  $1/2$ .
- In Ollivier (2018) it was shown that in TD Learning, if the matrix  $P$  corresponds to a reversible Markov chain, then  $E[\bar{g}(\theta_t)] = \nabla_\theta \mathcal{N}(f)$  for some  $f$ . This makes Neural TD easy to analyze in the reversible case as it is exactly gradient descent.
- In Liu & Olshovsky (2021), it was shown how to further use the function  $\mathcal{N}(\cdot)$  to analyze TD learning with linear approximation when the policy is not necessarily reversible. In particular, it was shown that the mean update of  $TD$  with linear approximation is a gradient splitting of the function  $\mathcal{N}(\cdot)$ . This is one of the crucial ideas we build on in this paper.
4.2 Nonlinear Small-Gain Theorem
Inspired by Olshovsky & Gharesifard (2022), our second main tool is a nonlinear version of the small-gain theorem. Because the actor and critic update simultaneously, we need to rule out the possibility that errors in the actor compound with errors in the critic to create divergence. For example, it is conceivable that, when the policy is fixed, the critic converges to a reasonable approximation; when the critic is fixed, the actor converges to an approximate of the stationary point; but both updating simultaneously results in divergence.
The core idea of small-gain is to write these updates in such a way so that one can argue that if certain coefficients are small enough, this "interconnection" of the actor and critic systems converges. The
small-gain theorem we use is a nonlinear version of the textbook version Drazin (1992). This is a widely-used trick in control theory that avoids the necessity of explicitly finding a Lyapunov function.
We have provided an analysis of Neural AC using a convex combination of the  $D$ -norm and the Dirichlet semi-norm to describe the error. Our main result is an error rate of  $O\left(T^{-0.5} + \epsilon\right) + \tilde{O}\left(m^{-0.5}\right)$  under the i.i.d. sampling and  $O\left((\log T)^2 \cdot T^{-0.5} + \epsilon\right) + \tilde{O}\left(m^{-0.5}\right)$  under the Markov sampling for neural networks of arbitrary depth. Crucially, our proof does not make assumptions that force the neural networks to stay close to their initial conditions, relying instead on arguments that show that neural networks which are not "too nonlinear" will still converge to an approximate minimum.
This research was partially supported by the NSF under grants CCF-2200052, DMS-1664644, and IIS-1914792, by the ONR under grant N00014-19-1-2571, by the DOE under grant DE-AC02-05CH11231, by the NIH under grant UL54 TR004130, and by Boston University.
Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, (5):834-846, 1983.
Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference on learning theory, pp. 1691-1692. PMLR, 2018.
Qi Cai, Zhuoran Yang, Jason D Lee, and Zhaoran Wang. Neural temporal-difference learning converges to global optima. Advances in Neural Information Processing Systems, 32, 2019.
Semin Cayci, Niao He, and R Srikant. Finite-time analysis of entropy-regularized neural natural actor-critic algorithm. arXiv preprint arXiv:2206.00833, 2022.
Tianyi Chen, Yuejiao Sun, and Wotao Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. Advances in Neural Information Processing Systems, 34: 25294-25307, 2021.
Xuyang Chen and Lin Zhao. Finite-time analysis of single-timescale actor-critic. arXiv preprint arXiv:2210.09921, 2022.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
Philip G Drazin. Nonlinear systems. Number 10. Cambridge University Press, 1992.
Amir-massoud Farahmand, Mohammad Ghavamzadeh, Csaba Szepesvári, and Shie Mannor. Regularized policy iteration with nonparametric function spaces. The Journal of Machine Learning Research, 17(1):4809-4874, 2016.
FR Gantmacher. The theory of matrices. New York, 1964.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
Harshat Kumar, Alec Koppel, and Alejandro Ribeiro. On the sample complexity of actor-critic method for reinforcement learning with function approximation. Machine Learning, pp. 1-35, 2023.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-444, 2015.
Chaoyue Liu, Libin Zhu, and Misha Belkin. On the linearity of large non-linear models: when and why the tangent kernel is constant. Advances in Neural Information Processing Systems, 33: 15954-15964, 2020.
Rui Liu and Alex Olshovsky. Temporal difference learning as gradient splitting. In International Conference on Machine Learning, pp. 6905-6913. PMLR, 2021.
A Yu Mitrophanov. Sensitivity and convergence of uniformly ergodic markov chains. Journal of Applied Probability, 42(4):1003-1014, 2005.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Yann Ollivier. Approximate temporal difference learning is a gradient descent for reversible policies. arXiv preprint arXiv:1805.00869, 2018.
Alex Olshovsky and Bahman Gharesifard. A small gain analysis of single timescale actor critic. arXiv preprint arXiv:2203.02591, 2022.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-359, 2017.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Haoxing Tian, Ioannis Paschalidis, and Alex Olshovsky. On the performance of temporal difference learning with neural networks. In The Eleventh International Conference on Learning Representations, 2023.
Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. arXiv preprint arXiv:1909.01150, 2019.
Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems, 33:17617-17628, 2020.
Pan Xu and Quanquan Gu. A finite-time analysis of q-learning with neural network function approximation. In International Conference on Machine Learning, pp. 10555-10565. PMLR, 2020.
Shaofeng Zou, Tengyu Xu, and Yingbin Liang. Finite-sample analysis for sarsa with linear function approximation. Advances in neural information processing systems, 32, 2019.
Dan Zwillinger. CRC standard mathematical tables and formulas. chapman and hall/CRC, 2018.
A Sketch of Proof
In this section we give a basic idea of how we prove Theorem 3.1. Briefly speaking, our idea contains two directions: First, the Critic error (captured by  $\Delta_{Q}$ ) can be upper-bounded by the Actor error (captured by  $\Delta_{V}$ ); Next, the Actor error can also be upper-bounded by the Critic error. Therefore, both errors are bounded and converge to 0. Based on this idea, our proof can be divided into three steps:
Step 1: Analysis of Actor update.
In Appendix B, we first bound  $\Delta_V$  by  $\Delta_Q$  through Actor update. On one hand, by considering Actor update and comparing it with mean-path update (where we replace  $g$  by  $\bar{g}$ ), one would have
$$
\mathbb {E} \left[ \theta_ {t + 1} - \theta_ {t} \mid \mathcal {F} _ {t} \right] = - \alpha^ {\theta} \nabla V _ {\theta_ {t}} ^ {*} - \alpha^ {\theta} D _ {Q}.
$$
$$
\mathbb {E} \left[ | | \theta_ {t + 1} - \theta_ {t} | | ^ {2} | \mathcal {F} _ {t} \right] \leq | | \alpha^ {\theta} \nabla V _ {\theta_ {t}} ^ {*} + \alpha^ {\theta} D _ {Q} | | ^ {2} + \frac {4}{(1 - \gamma) ^ {2}} \alpha^ {\theta^ {2}} U _ {g} ^ {2}.
$$
where  $D_{Q} = \nabla \ln \pi (\theta_{t})^{T}\Phi_{\theta_{t}}\left[Q(w_{t}) - Q_{\theta_{t}}^{*}\right],\mathcal{F}_{t} = (w_{t},\theta_{t})$  and  $U_{g}$  is defined in Lemma C.13.
On the other hand, Lemma C.12 suggests  $V_{\theta}^{*}$  is smooth w.r.t.  $\theta$ . Hence,
$$
V _ {\theta_ {t + 1}} ^ {*} \leq V _ {\theta_ {t}} ^ {*} + \nabla V _ {\theta_ {t}} ^ {*} \left(\theta_ {t + 1} - \theta_ {t}\right) + \frac {H _ {V}}{2} \left\| \theta_ {t + 1} - \theta_ {t} \right\| ^ {2}.
$$
Our claim is a combination of the above facts and some simple calculations:
$$
\begin{array}{l} \left(\frac {\alpha_ {\theta}}{2} - \alpha_ {\theta} ^ {2} H _ {V}\right) \Delta_ {V} \leq \frac {\mathbb {E} \left[ V _ {\theta_ {1}} ^ {*} - V _ {\theta_ {T + 1}} ^ {*} \right]}{T} + c _ {1} \left(\alpha_ {\theta} ^ {2} H _ {V} + \frac {\alpha_ {\theta}}{2}\right) \Delta_ {Q} \\ + c _ {2} \left(\alpha_ {\theta} ^ {2} H _ {V} + \frac {\alpha_ {\theta}}{2}\right) \epsilon^ {2} + \frac {2 H _ {V}}{(1 - \gamma) ^ {2}} \alpha^ {\theta 2} U _ {g} ^ {2}. \\ \end{array}
$$
We successfully bound  $\Delta_V$  by  $\Delta_Q$ .
Step 2: Analysis of Critic update.
In Appendix C, we next bound  $\Delta_Q$  by  $\Delta_V$  through Critic update. Here we perform classical way of analysis, which begins with
$$
\begin{array}{l} \mathbb {E} \left[ | | w _ {t + 1} - \hat {w} _ {\theta_ {t + 1}} ^ {*} | | ^ {2} \right] \\ \leq \underbrace {\mathbb {E} \left[ \left| \left| w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} f (O _ {t} , w _ {t}) \right| \right| ^ {2} \right]} _ {I _ {1}} \\ + \underbrace {\mathbb {E} \left[ 2 (\hat {w} _ {\theta_ {t}} ^ {*} - \hat {w} _ {\theta_ {t + 1}} ^ {*}) ^ {T} (w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} f (O _ {t} , w _ {t})) \right]} _ {I _ {2}} + \underbrace {\mathbb {E} \left[ | | \hat {w} _ {\theta_ {t}} ^ {*} - \hat {w} _ {\theta_ {t + 1}} ^ {*} | | ^ {2} \right]} _ {I _ {3}}. \\ \end{array}
$$
We treat the above three terms respectively. To address  $I_{1}$ , by comparing with mean-path update:
$$
\begin{array}{l} \mathbb {E} \left[ | | w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} + \alpha^ {w} f (O _ {t}, w _ {t}) | | ^ {2} \right] \\ = \mathbb {E} \left[ \left| \left| w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*} \right| \right| ^ {2} \right] + \underbrace {\mathbb {E} \left[ 2 \alpha^ {w} \left(w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*}\right) ^ {T} \bar {f} \left(w _ {t} , \theta_ {t}\right) \right]} _ {I _ {1, 1}} \\ + \underbrace {\mathbb {E} \left[ \alpha^ {w ^ {2}} | | f (O _ {t} , w _ {t}) | | ^ {2} \right]} _ {I _ {1, 2}} + \underbrace {\mathbb {E} \left[ 2 \alpha^ {w} (w _ {t} - \hat {w} _ {\theta_ {t}} ^ {*}) ^ {T} \left[ f (O _ {t} , w _ {t}) - \bar {f} (w _ {t} , \theta_ {t}) \right] \right]} _ {I _ {1, 3}}. \\ \end{array}
$$
Now let us examine this equation carefully.  $I_{1,1}$  is the inner product between  $w_{t} - \hat{w}_{\theta_{t}}^{*}$  and the mean-path update  $\bar{f} (w_t,\theta_t)$ , which can be captured by gradient splitting;  $I_{1,2}$  decays as  $\alpha_{t}^{2}$ , so a loose bound on  $||f(O_t,w_t)||^2$  is enough (See Lemma C.15);  $I_{1,3}$  is Markov sampling noise, which is handled using the same procedure as in Bhandari et al. (2018).
To discuss more about how to address Markov sampling noise, the idea is to using Assumption 2.8 to show that, after  $\tau_{\mathrm{mix}}$  steps, the distance between distribution of agent and the stationary distribution
decaying geometrically, and thus  $I_{1,3}$  also decays geometrically. However, there is still a lot of difficulties to apply the same analysis in our work since TD(0) is considered in Bhandari et al. (2018) while Actor-Critic methods is considered here. The difficulties is induced by the constant changing of policy in every time steps during training. We introduce an auxiliary chain (See the definitions before Lemma C.10) to further address the changing of policy problem inspired by Zou et al. (2019); Wu et al. (2020); Chen & Zhao (2022).
Now we move on to  $I_{2}$ . We notice that the dominate term is  $\mathbb{E}\left[2(\hat{w}_{\theta_t}^* - \hat{w}_{\theta_{t + 1}}^*)^T (w_t - \hat{w}_{\theta_t}^*)\right]$  since the remaining term  $\mathbb{E}\left[2(\hat{w}_{\theta_t}^* - \hat{w}_{\theta_{t + 1}}^*)^T\alpha^w f(O_t, w_t)\right]$  decays as  $\alpha^\theta \alpha^w$  ( $\alpha^\theta$  comes from  $||\hat{w}_{\theta_t}^* - \hat{w}_{\theta_{t + 1}}^*\|$  which can be seen using Assumption 2.6). To handle the dominate term, we first view  $\hat{w}_{\theta}^{*}$  as a function of  $\theta$  and use a second order expansion as follows. Then the problem gets solved after noticing that we already derive relationships on  $\theta_{t + 1} - \theta_t$  in Step 1.
$$
\hat {w} _ {\theta_ {t + 1}} ^ {*} (i) = \hat {w} _ {\theta_ {t}} ^ {*} (i) + \nabla \hat {w} _ {\theta_ {t}} ^ {*} (i) ^ {T} (\theta_ {t + 1} - \theta_ {t}) + \frac {1}{2} (\theta_ {t + 1} - \theta_ {t}) ^ {T} \nabla^ {2} \hat {w} _ {\theta_ {t} ^ {\prime}} ^ {*} (i) (\theta_ {t + 1} - \theta_ {t}).
$$
To address  $I_{3}$ , we notice that it decays as  $\alpha^{w2}$  as a direct result of Assumption 2.6.
Combine all of the above result we can finally arrive at the relationship between  $\Delta_Q$  and  $\Delta_V$ .
Step 3: Combine result from Step 1 and 2 by small-gain theorem.
Now we are ready to use the Small Gain theorem. We fit the results from Step 1 and Step 2 by the following form:
$$
x \leq a _ {1} y + a _ {2}, \quad y \leq b _ {1} x + b _ {2} + b _ {3} \sqrt {y}.
$$
Then, Small Gain theorem implies that  $y$  can be upper bounded by the following inequality:
$$
y \leq \frac {2 b _ {2} + b _ {3} ^ {2} + 2 a _ {2} b _ {1}}{1 - 2 a _ {1} b _ {1}}.
$$