SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Assumption 3.3 (Lipschitz continuity of policy). Let  $\pi_{\theta}(a|s)$  be a policy parameterized by  $\pmb{\theta} \in \mathbb{R}^d$ . There exists positive constants  $B, L_l$  and  $L_{\pi}$  such that for any  $\pmb{\theta}, \pmb{\theta}_1, \pmb{\theta}_2 \in \mathbb{R}^d$ ,  $s \in S$ , and  $a \in A$ , it holds that:
(a)  $\| \nabla \log \pi_{\pmb{\theta}}(a|s)\| \leq B$
(b)  $||\nabla \log \pi_{\pmb{\theta}_1}(a|s) - \nabla \log \pi_{\pmb{\theta}_2}(a|s)||\leq L_l||\pmb{\theta}_1 - \pmb{\theta}_2||$
(c)  $|\pi_{\pmb{\theta}_1}(a|s) - \pi_{\pmb{\theta}_2}(a|s)| \leq L_\pi \| \pmb{\theta}_1 - \pmb{\theta}_2\|$
Assumption 3.3 is standard in the literature of policy gradient methods [22, 42, 38, 34, 31, 8, 21]. This assumption holds for many policy classes such as Gaussian policy [10], Boltzmann policy [15], and tabular softmax policy [1].
Assumption 3.4. For any  $\pmb{\theta},\pmb{\theta}^{\prime}\in \mathbb{R}^{d}$ , there exists constant  $L_{\mu}$  such that  $\| \nabla \mu_{\pmb{\theta}} - \nabla \mu_{\pmb{\theta}^{\prime}}\| \leq L_{\mu}\| \pmb{\theta} - \pmb{\theta}^{\prime}\|$ , where  $\mu_{\pmb{\theta}}(s)$  is the stationary distribution under the policy  $\pi_{\pmb{\theta}}$ .
Assumption 3.4 is introduced in [8] to show the smoothness of the optimal critic  $\omega^{*}(\pmb{\theta})$ , which is critical to guarantee the convergence of single-timescale AC. This assumption holds for the finite state-action space setting [18].
3.2 Finite-Time Analysis
We define the following uniform upper bound for the linear function approximation error of the critic:
$$
\epsilon_ {\mathrm {a p p}} := \sup _ {\pmb {\theta}} \sqrt {\mathbb {E} _ {s \sim \mu_ {\pmb {\theta}}} (\phi (s) ^ {\top} \pmb {\omega} ^ {*} (\pmb {\theta}) - V _ {\pmb {\theta}} (s)) ^ {2}}.
$$
The error  $\epsilon_{\mathrm{app}}$  is zero if  $V_{\theta}$  is indeed a linear function for any  $\theta$ . Naturally, it can be expected that the learning errors of Algorithm 1 depend on  $\epsilon_{\mathrm{app}}$ .
We define the following integer  $\tau_T$  that will be useful in the statement of the theorems, which depends on the number of total iterations  $T$ :
$$
\tau_ {T} := \min  \{i \geq 0 \mid m \rho^ {i - 1} \leq \frac {1}{\sqrt {T}} \},
$$
where  $m, \rho$  are constants defined in Assumption 3.2. Therefore, we choose  $\tau_T = \frac{\log m\rho^{-1}}{\log\rho^{-1}} + \frac{\log T}{2\log\rho^{-1}} = \mathcal{O}(\log T)$  such that  $m\rho^{\tau_T - 1} \leq \frac{1}{\sqrt{T}}$ . The integer  $\tau_T$  represents the mixing time of an ergodic Markov chain, which will be used to control the Markovian noise in the analysis.
We quantify the learning errors by defining  $y_{t} \coloneqq \eta_{t} - J(\pmb{\theta}_{t})$ , which is the difference between the reward estimator and the true time-average reward  $J(\pmb{\theta}_t)$  at time  $t$ . For the critic, we define,  $z_{t} \coloneqq \omega_{t} - \omega_{t}^{*}$  with  $\omega_{t}^{*} \coloneqq \omega^{*}(\pmb{\theta}_{t})$  to measure the error between the critic and its target value at iteration  $t$ . The following two theorems summarize our main results.
Theorem 3.5 (Markovian sampling). Consider Algorithm 1 with  $\alpha_{t} = \alpha = \frac{c}{\sqrt{T}}$ ,  $\beta_{t} = \beta = \frac{1}{\sqrt{T}}$ ,  $\gamma_{t} = \gamma = \frac{1}{\sqrt{T}}$ , where  $c$  is a constant depending on problem parameters. Suppose Assumptions 3.1-3.4 hold, we have for  $T \geq 2\tau_{T}$
$$
\frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} y _ {t} ^ {2} = \mathcal {O} (\frac {\log^ {2} T}{\sqrt {T}}) + \mathcal {O} (\epsilon_ {\mathrm {a p p}}),
$$
$$
\frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \boldsymbol {z} _ {t} \| ^ {2} = \mathcal {O} (\frac {\log^ {2} T}{\sqrt {T}}) + \mathcal {O} (\epsilon_ {\mathrm {a p p}}),
$$
$$
\frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \nabla J (\pmb {\theta} _ {t}) \| ^ {2} = \mathcal {O} (\frac {\log^ {2} T}{\sqrt {T}}) + \mathcal {O} (\epsilon_ {\mathrm {a p p}}).
$$
We defer the interpretation of the above results a bit and present below the analysis results under the i.i.d. sampling first for better comparison. The major difference of i.i.d. from Markovian sampling is that at the  $t$ -th iteration, the state  $s_t$  is sampled from the stationary distribution  $\mu_{\theta_t}$  instead of the evolving Markov chain (see Algorithm 2 in Appendix E). The i.i.d. sampling simplifies the analysis in the way that many Markovian noise terms reduce to zero effectively. This leads to a tighter sample complexity bound compared to the Markovian sampling by up to logarithmic factors.
Theorem 3.6 (i.i.d. sampling). Consider Algorithm 2 (see Appendix  $E$ ) with  $\alpha_{t} = \alpha = \frac{c}{\sqrt{T}}$ ,  $\beta_{t} = \beta = \frac{1}{\sqrt{T}}$ ,  $\gamma_{t} = \gamma = \frac{1}{\sqrt{T}}$ , where  $c$  is a constant depending on problem parameters. Suppose Assumptions 3.1-3.4 hold, we have for  $T \geq 2\tau_{T}$
$$
\frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} y _ {t} ^ {2} = \mathcal {O} (\frac {1}{\sqrt {T}}) + \mathcal {O} (\epsilon_ {\mathrm {a p p}}),
$$
$$
\frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \pmb {z} _ {t} \| ^ {2} = \mathcal {O} (\frac {1}{\sqrt {T}}) + \mathcal {O} (\epsilon_ {\mathrm {a p p}}),
$$
$$
\frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \nabla J (\pmb {\theta} _ {t}) \| ^ {2} = \mathcal {O} (\frac {1}{\sqrt {T}}) + \mathcal {O} (\epsilon_ {\mathrm {a p p}}).
$$
If the critic approximation error  $\epsilon_{\mathrm{app}}$  is zero, we see that the reward estimator, the critic, and the actor estimation errors all diminish at a sub-linear rate of  $\widetilde{\mathcal{O}}(T^{-\frac{1}{2}})$ . The additional logarithmic term hidden by  $\widetilde{\mathcal{O}}(\cdot)$  is incurred by the mixing time of the Markov chain, which can be get rid of under the i.i.d. sampling. It also hides the polynomials of all other problem parameters. They are explicitly characterized in the proofs up to the last step of analyzing the overall interconnected error propagation system. One can easily keep and get the dependence orders of all parameters if needed. Here we focus on the dependence of the iteration number for ease of presentation.
To put the results into perspective, note that  $\mathcal{O}(T^{-\frac{1}{2}})$  is the rate one would obtain from stochastic gradient descent (SGD) on general non-convex functions with unbiased gradient updates. In terms of sample complexity, to obtain an  $\epsilon$ -approximate stationary point, it takes a number of  $\tilde{\mathcal{O}}(\epsilon^{-2})$  samples for Markovian sampling (Algorithm 1) and  $\mathcal{O}(\epsilon^{-2})$  for i.i.d. sampling (Algorithm 2), which matches the state-of-the-art performance of SGD on non-convex optimization problems.
The obtained sample complexities are superior to those of other AC variants. Notably, [16] provided finite-time convergence for double-loop variant with a  $\mathcal{O}(\epsilon^{-4})$  sample complexity and [31] analysed two-timescale variant, yielding a  $\widetilde{\mathcal{O}} (\epsilon^{-2.5})$  sample complexity. The sample complexity gap is intrinsic to their inefficient usage of data. In the double-loop setting, the critic starts over to estimate the Q-value for an intermediate policy in the inner loop, ignoring the fact that the consecutive Q-values can be similar given a relatively minor policy update. The two-timescale setting artificially slows down the actor update by adopting an actor stepsize that decays faster than the critic. The single-timescale approach updates the critic and actor parallelly with proportional stepsizes and thus learns more efficiently.
Moreover, our result matches the  $\mathcal{O}(\epsilon^{-2})$  sample complexity of policy gradient methods such as REINFORCE [2, 22] under the i.i.d sampling. It is previously found in [31] that there is a sample complexity gap between Algorithm 1 adopting two-timescale step sizes and (variance-reduced) REINFORCE [22]. In this paper, we close this gap by providing a single-timescale analysis for Algorithm 1 which shows that this practical single-timescale AC can have the same sample complexity as REINFORCE.
The main challenge in the finite-time analysis lies in that the estimation errors of the time-average reward, the critic, and the policy gradient are strongly coupled. To overcome this difficulty, we view the propagation of these errors as an interconnected system and analyze them holistically. To better appreciate the advantage of our analysis framework over the decoupled methods that are traditionally adopted in analyzing double-loop and two-timescale variants, we sketch the main proof steps of Theorem 3.5 in the following. We also highlight the key challenges and techniques developed correspondingly. All supporting lemmas mentioned below can be found in Appendix.
We first derive implicit (coupled) upper bounds for the reward estimation error  $y_{t}$ , the critic error  $z_{t}$  and the policy gradient  $\nabla J(\pmb{\theta}_t)$ , respectively. Then, we solve a system of inequalities to establish finite-time convergence.
Step 1: Reward estimation error analysis. Using the reward estimator update rule (Line 7 of Algorithm 1), we decompose the reward estimation error into:
$$
\begin{array}{l} y _ {t + 1} ^ {2} = \left(1 - 2 \gamma_ {t}\right) y _ {t} ^ {2} + 2 \gamma_ {t} y _ {t} \left(r _ {t} - J \left(\boldsymbol {\theta} _ {t}\right)\right) \\ + 2 y _ {t} \left(J \left(\boldsymbol {\theta} _ {t}\right) - J \left(\boldsymbol {\theta} _ {t + 1}\right)\right) + \left(J \left(\boldsymbol {\theta} _ {t}\right) - J \left(\boldsymbol {\theta} _ {t + 1}\right) + \gamma_ {t} \left(r _ {t} - \eta_ {t}\right)\right) ^ {2}. \tag {7} \\ \end{array}
$$
The second term on the right-hand side of (7) is a bias term caused by the Markovian sample, which is characterized in Lemma C.1. As shown in Lemma E.1, this bias reduces to 0 under i.i.d. sampling after taking the expectation. The third term captures the variation of the moving targets  $J(\theta_t)$ . The double-loop variant of AC runs a policy evaluation sub-problem in the inner loop for each target  $J(\theta_t)$  to estimate the policy gradient accurately. This easily ensures the monotonic decreasing of  $J(\theta_t)$  and consequently the convergence. The two-timescale variant utilizes the additional property of  $\lim_{t\to \infty}\alpha_t / \beta_t = 0$  to annihilate this term and consequently can have a decoupled analysis. In the case of single-timescale AC, we do not have the aforementioned special algorithm designs and properties to ease the analysis. Instead, we utilize the smoothness of  $J(\theta)$  (see Lemma B.2) and derive an implicit upper bound for this term as a function of the norm of  $y_t$  and  $\nabla J(\theta_t)$ . This bound will be combined with the implicit bounds derived in Step 2 and Step 3 below to establish the non-asymptotic convergence altogether. The last term in (7) reflects the variance in reward estimation, which is bounded by  $\mathcal{O}(\gamma_t)$ .
Step 2: Critic error analysis. Using the critic update rule (Line 8 of Algorithm 1), we decompose the squared error by (we neglect the projection for the time being for the ease of comprehension. The complete analysis can be found in the appendix.)
$$
\begin{array}{l} \left\| \boldsymbol {z} _ {t + 1} \right\| ^ {2} = \left\| \boldsymbol {z} _ {t} \right\| ^ {2} + 2 \beta_ {t} \langle \boldsymbol {z} _ {t}, \bar {g} (\boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}) \rangle + 2 \beta_ {t} \Psi (O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}) + 2 \beta_ {t} \langle \boldsymbol {z} _ {t}, \Delta g (O _ {t}, \eta_ {t}, \boldsymbol {\theta} _ {t}) \rangle \\ + 2 \left\langle \boldsymbol {z} _ {t}, \boldsymbol {\omega} _ {t} ^ {*} - \boldsymbol {\omega} _ {t + 1} ^ {*} \right\rangle + \left\| \beta_ {t} \left(g \left(O _ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) + \Delta g \left(O _ {t}, \eta_ {t}, \boldsymbol {\theta} _ {t}\right)\right) + \boldsymbol {\omega} _ {t} ^ {*} - \boldsymbol {\omega} _ {t + 1} ^ {*} \right\| ^ {2}, \\ \end{array}
$$
where  $O_{t} \coloneqq (s_{t}, a_{t}, s_{t + 1})$  is a tuple of observations and the definitions of  $g, \bar{g}, \Delta g$ , and  $\Psi$  can be found in (12) and (13) in Appendix A. Without diving into the detailed definitions, here we focus on illustrating the high-level insights of our proof. First of all, the second term on the right-hand side of (8) can be bounded by  $-2\lambda \beta_{t}\| z_{t}\|^{2}$  under Assumption 3.1. It provides an explicit characterization of how sufficient exploration can help the convergence of learning. The third term is a Markovian noise, which is further bounded implicitly in Lemma C.3. For the i.i.d sampling case, as shown in Lemma E.1, this bias reduces to 0 after taking the expectation. The fourth term is caused by inaccurate reward and critic estimations, which can be bounded by the norm of  $y_{t}$  and  $z_{t}$ . The fifth term tracks both the critic estimation performance  $z_{t}$  and the difference between the drifting critic targets  $\omega_{t}^{*}$ . Similar to the case of Step 1, the double-loop approach bounds this term relying on the accurate policy evaluation sub-problem in the inner loop for each target  $\omega_{t}^{*}$ , whereas the two-timescale approach ensures its convergence by additionally requiring  $\lim_{t\to \infty}\alpha_t / \beta_t = 0$ . In contrast, we establish an implicit upper bound for this term as a function of  $y_{t}$  and  $z_{t}$  by utilizing the smoothness of the optimal critic proved in Lemma B.4. Finally, the last term reflects the variances of various estimations, which is bounded by  $\mathcal{O}(\beta_t)$ .
Step 3: Policy gradient norm analysis. Using the actor update rule (Line 9 of Algorithm 1) and the smoothness property of  $J(\pmb{\theta})$  (see Lemma B.2), we derive
$$
\begin{array}{l} \left\| \nabla J \left(\boldsymbol {\theta} _ {t}\right) \right\| ^ {2} \leq \frac {1}{\alpha_ {t}} \left(J \left(\boldsymbol {\theta} _ {t + 1}\right) - J \left(\boldsymbol {\theta} _ {t}\right)\right) + \Theta \left(O _ {t}, \boldsymbol {\theta} _ {t}\right) - \langle \nabla J \left(\boldsymbol {\theta} _ {t}\right), \Delta h \left(O _ {t}, \eta_ {t}, \boldsymbol {\omega} _ {t}, \boldsymbol {\theta} _ {t}\right) \rangle \tag {9} \\ - \left\langle \nabla J (\pmb {\theta} _ {t}), \mathbb {E} _ {O _ {t} ^ {\prime}} [ \Delta h ^ {\prime} (O _ {t} ^ {\prime}, \pmb {\theta} _ {t}) ] \right\rangle + \frac {L _ {J ^ {\prime}}}{2} \alpha_ {t} \| \delta_ {t} \nabla \log \pi_ {\pmb {\theta} _ {t}} (a _ {t} | s _ {t}) \| ^ {2}, \\ \end{array}
$$
where  $O_{t}^{\prime}$  is a shorthand for an independent sample from stationary distribution  $s\sim \mu_{\theta_t},a\sim$ $\pi_{\theta_t},s'\sim \mathcal{P}(\cdot |s,a)$ $\Theta$  is defined in (13), and  $L_{J'}$  is a constant. The first term on the right-hand side of (9) compares the actor's performances between consecutive updates, which can be bounded via Abel summation by parts. The second term is a noise term introduced by Markovian sampling, which is characterized in Lemma C.6. Again, as proven in Lemma E.1, this bias reduces to 0 under i.i.d. sampling after taking the expectation. The third term is an error introduced by the inaccurate estimations of both the time-average reward and the critic. This term was directly bounded to zero under both the double-loop setting and the two-timescale setting due to their particular algorithm design, to enable a decoupled analysis. We control this term by providing an implicit bound depending on  $y_{t},z_{t}$  , and  $\nabla J(\pmb {\theta}_t)$  . The fourth term comes from the linear function approximation error. The last term captures the variance of the stochastic gradient update, which is bounded by  $\mathcal{O}(\alpha_t)$
Step 4: Interconnected iteration system analysis. Taking the expectation of and summing (7), (8), and (9) from  $\tau_T$  to  $T - 1$ , respectively, we obtain the following system of inequalities in terms of  $Y_{T}$ ,  $Z_{T}$ ,  $G_{T}$ :
$$
Y _ {T} := \frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} y _ {t} ^ {2} \leq \mathcal {O} (\frac {\log^ {2} T}{\sqrt {T}}) + l _ {1} \sqrt {Y _ {T} G _ {T}},
$$
$$
Z _ {T} := \frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \pmb {z} _ {t} \| ^ {2} \leq \mathcal {O} (\frac {\log^ {2} T}{\sqrt {T}}) + \mathcal {O} (\epsilon_ {\mathrm {a p p}}) + l _ {2} \sqrt {Y _ {T} Z _ {T}} + l _ {3} \sqrt {Z _ {T} (2 Y _ {T} + 8 Z _ {T})},
$$
$$
G _ {T} := \frac {1}{T - \tau_ {T}} \sum_ {t = \tau_ {T}} ^ {T - 1} \mathbb {E} \| \nabla J (\pmb {\theta} _ {t}) \| ^ {2} \leq \mathcal {O} (\frac {\log^ {2} T}{\sqrt {T}}) + \mathcal {O} (\epsilon_ {\mathrm {a p p}}) + l _ {4} \sqrt {G _ {T} (2 Y _ {T} + 8 Z _ {T})},
$$
where  $l_{1}, l_{2}, l_{3}, l_{4}$  are positive constants. By solving the above system of inequalities, we further prove that if
$$
l _ {1} \left(1 + 2 l _ {4} ^ {2} + 8 l _ {4} ^ {2} \left(2 l _ {2} ^ {2} + l _ {3}\right)\right) \leq 1 \quad \text {a n d} \quad 1 6 l _ {3} \leq 1,
$$
then  $Y_{T}, Z_{T}, G_{T}$  converge at a rate of  $\mathcal{O}\left(\frac{\log^2 T}{\sqrt{T}}\right)$ . This condition can be easily satisfied by choosing the stepsize ratio  $c$  to be smaller than a threshold identified in Equation (28). Thus, it completes the proof.
The above proof applies to i.i.d sampling straightforwardly, with the corresponding terms pointed out in the above steps reducing to 0 in the analysis. The additional proof can be found in Lemma E.1.
4 Conclusion and Discussion
In this paper, we establish the finite-time analysis for single-timescale AC with Markovian sampling. Our work compares favorably to existing works in terms of analyzing online learning and considering the continuous state space. We developed a series of lemmas that characterize the propagation of errors, and establish their convergence simultaneously by solving a system of nonlinear inequalities. The proposed framework is general and can be applied to analyze other single-timescale stochastic approximation algorithms. Our future work includes further considering the continuous action space problems and developing new proof techniques that require fewer assumptions.
This work was supported by the Singapore Ministry of Education Tier 1 Academic Research Funds (A0009030-00-00, 22-5460-A0001). The authors would like to thank the timely help from Yue Wu and Quanquan Gu for clarifying the proof of their seminar work on finite-time analysis of two-timescale actor-critic.
[1] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. In Conference on Learning Theory, pages 64-66. PMLR, 2020.
[2] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-76, 2021.
[3] Jonathan Baxter and Peter L Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research, 15:319-350, 2001.
[4] Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference on learning theory, pages 1691-1692. PMLR, 2018.
[5] Shalabh Bhatnagar, Richard S Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor-critic algorithms. Automatica, 45(11):2471-2482, 2009.
[6] Vivek S Borkar. Stochastic approximation: a dynamical systems viewpoint, volume 48. Springer, 2009.
[7] Dotan Di Castro and Ron Meir. A convergent online single time scale actor critic algorithm. The Journal of Machine Learning Research, 11:367-410, 2010.
[8] Tianyi Chen, Yuejiao Sun, and Wotao Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. Advances in Neural Information Processing Systems, 34:25294-25307, 2021.
[9] Xuyang Chen, Jingliang Duan, Yingbin Liang, and Lin Zhao. Global convergence of two-timescale actor-critic for solving linear quadratic regulator. arXiv preprint arXiv:2208.08744, 2022.
[10] Kenji Doya. Reinforcement learning in continuous time and space. Neural computation, 12(1):219-245, 2000.
[11] Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In International Conference on Machine Learning, pages 1467-1476. PMLR, 2018.
[12] Zuyue Fu, Zhuoran Yang, and Zhaoran Wang. Single-timescale actor-critic provably finds globally optimal policy. arXiv preprint arXiv:2008.00483, 2020.
[31] Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems, 33:17617-17628, 2020.
[32] Huaqing Xiong, Lin Zhao, Yingbin Liang, and Wei Zhang. Finite-time analysis for double q-learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020 (spotlight), 2020.
[33] Pan Xu, Felicia Gao, and Quanquan Gu. Sample efficient policy gradient methods with recursive variance reduction. arXiv preprint arXiv:1909.08610, 2019.
[34] Pan Xu, Felicia Gao, and Quanquan Gu. An improved convergence analysis of stochastic variance-reduced policy gradient. In Uncertainty in Artificial Intelligence, pages 541-551. PMLR, 2020.
[35] Tengyu Xu, Zhe Wang, and Yingbin Liang. Improving sample complexity bounds for (natural) actor-critic algorithms. Advances in Neural Information Processing Systems, 33:4358-4369, 2020.
[36] Tengyu Xu, Zhe Wang, and Yingbin Liang. Non-asymptotic convergence analysis of two time-scale (natural) actor-critic algorithms. arXiv preprint arXiv:2005.03557, 2020.
[37] Zhuoran Yang, Yongxin Chen, Mingyi Hong, and Zhaoran Wang. Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost. Advances in neural information processing systems, 32, 2019.
[38] Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization, 58(6):3586-3612, 2020.
[39] Shangtong Zhang, Bo Liu, Hengshuai Yao, and Shimon Whiteson. Provably convergent two-timescale off-policy actor-critic with function approximation. In International Conference on Machine Learning, pages 11204-11213. PMLR, 2020.
[40] Lin Zhao, Huaqing Xiong, and Yingbin Liang. Faster non-asymptotic convergence for double q-learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021, 2021.
[41] Mo Zhou and Jianfeng Lu. Single timescale actor-critic method to solve the linear quadratic regulator with convergence guarantees. Journal of Machine Learning Research, 24(222):1-34, 2023.
[42] Shaofeng Zou, Tengyu Xu, and Yingbin Liang. Finite-sample analysis for sarsa with linear function approximation. Advances in neural information processing systems, 32, 2019.
A Notation 13
B Preliminary Lemmas 14
C Proof of Main Theorem 14
C.1 Step 1: Reward estimation error analysis 14
C.2 Step 2: Critic error analysis 17
C.3 Step 3: Policy gradient norm analysis 21
C.4 Step 4: Interconnected iteration system analysis 23
D Proof of Supporting Lemmas 25
E IID Sample Analysis 32
We make use of the following auxiliary Markov chain to deal with the Markovian noise.
Auxiliary Markov Chain:
$$
s _ {t - \tau} \xrightarrow {\theta_ {t - \tau}} a _ {t - \tau} \xrightarrow {\mathcal {P}} s _ {t - \tau + 1} \xrightarrow {\theta_ {t - \tau}} \widetilde {a} _ {t - \tau + 1} \xrightarrow {\mathcal {P}} \widetilde {s} _ {t - \tau + 2} \xrightarrow {\theta_ {t - \tau}} \widetilde {a} _ {t - \tau + 2} \dots \xrightarrow {\mathcal {P}} \widetilde {s} _ {t} \xrightarrow {\theta_ {t - \tau}} \widetilde {a} _ {t} \xrightarrow {\mathcal {P}} \widetilde {s} _ {t + 1}. \tag {10}
$$
For reference, we also show the original Markov chain.
Original Markov Chain: