SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xuyang Chen
chenxuyang@u.nus.edu
Lin Zhao*
elezhli@nus.edu.sg
Actor-critic methods have achieved significant success in many challenging applications. However, its finite-time convergence is still poorly understood in the most practical single-timescale form. Existing works on analyzing single-timescale actor-critic have been limited to i.i.d. sampling or tabular setting for simplicity. We investigate the more practical online single-timescale actor-critic algorithm on continuous state space, where the critic assumes linear function approximation and updates with a single Markovian sample per actor step. Previous analysis has been unable to establish the convergence for such a challenging scenario. We demonstrate that the online single-timescale actor-critic method provably finds an  $\epsilon$ -approximate stationary point with  $\widetilde{\mathcal{O}}(\epsilon^{-2})$  sample complexity under standard assumptions, which can be further improved to  $\mathcal{O}(\epsilon^{-2})$  under the i.i.d. sampling. Our novel framework systematically evaluates and controls the error propagation between the actor and critic. It offers a promising approach for analyzing other single-timescale reinforcement learning algorithms as well.
Actor-critic (AC) methods have achieved great success in solving many challenging reinforcement learning (RL) problems [17, 20, 24]. AC updates the actor (i.e., the policy) using the estimated policy gradient (PG), which is a function of the Q-value under the policy. Meanwhile, it employs a bootstrapping critic to estimate the Q-value, which often helps reduce variance and accelerates convergence in practice.
Despite the empirical success, the non-asymptotic convergence analysis of AC in the most practical single-timescale form remains underexplored. A large body of existing works consider the double-loop variants, where the critic runs many steps to accurately estimate the Q-value for a given actor [37, 16, 27, 2]. This leads to a decoupled convergence analysis of the critic and the actor, which involves a policy evaluation sub-problem in the inner loop and a perturbed gradient descent in the outer loop. Its finite-time convergence is relatively well understood [16, 37, 35]. Nevertheless, the double-loop setting is mainly for ease of analysis, which is barely adopted in practice. Since it requires an accurate critic estimation, it is typically sample inefficient. In fact, it is unclear whether an inner loop of accurate policy evaluation is really necessary since it only corresponds to one transient policy among many iterations.
Another body of works considers the (single-loop) two-timescale variants [31, 9, 36, 13], where the actor and the critic are updated simultaneously in each iteration with step sizes of different timescales. The actor stepsize is typically smaller than that of the critic, with their ratio converging to zero as the iteration number goes to infinity. Hence, the actor is updated much slower than the critic. The two-timescale allows the critic to approximate the desired Q-value in an asymptotic way, which enables a decoupled convergence analysis of the actor and the critic. This variant is occasionally
Table 1: Comparison with related single-timescale actor-critic algorithms
Reference	Setting	Sampling	Sample Complexity
State Space	Reward	Actor	Critic
[21]	Finite	Discounted	i.i.d.	i.i.d.	O(ε-2)
[8]	Infinite	Discounted	i.i.d.	i.i.d.	O(ε-2)
This Paper	Infinite	Average	Markovian	Markovian	O(ε-2)
adopted to improve learning stability. However, it is still considered inefficient as the actor update is artificially slowed down.
In this paper, we consider the more practical single-timescale AC algorithm, which is the one introduced in many works of literature as well as in [25] as a classic AC algorithm. In single-timescale AC, the stepsizes of the critic and the actor diminish at the same timescale. Unlike the aforementioned variants, which have specialized designs aimed at simplifying the convergence analysis, the analysis of the single-timescale AC presents a greater challenge. Due to the substantial errors in critic estimation and the close coupling between the parallel critic update and actor update, the algorithm is more prone to unstable error propagation. It remains unclear under what condition the errors will converge to zero. To study its finite-time convergence, we consider the challenging undiscounted time-average reward formulation [25, 31, 37], which consists of three parallel updates: the (time-average) reward estimator, the critic estimator, and the actor estimator. We keep track of the reward estimation error, the critic error, and the policy gradient norm (which measures the actor error) by deriving an implicit bound for each of them. They are then analyzed altogether as an interconnected system inspired by [21] to establish the convergence simultaneously. Specifically, we identify the (constant) ratio between the actor stepsize and the critic stepsize, below which all three errors will diminish to zero, despite the inaccurate estimation in all three updates (reward estimation, critic, actor).
We summarise our main contributions as follows:
- We provide a finite-time analysis for the single-timescale AC under the Markovian sampling and prove an  $\widetilde{\mathcal{O}} (\epsilon^{-2})$  sample complexity, where  $\widetilde{\mathcal{O}} (\cdot)$  hides additional logarithmic terms. We further show that this sample complexity can be improved to  $\mathcal{O}(\epsilon^{-2})$  under i.i.d. sampling, which matches the state-of-the-art performance of SGD on general non-convex optimization problems. Our proof clearly shows that the additional logarithmic term under the Markovian sampling is introduced by the mixing time of the underlying Markov chain.
- Our result compares favorably to existing works on single-timescale AC. To our knowledge, the only other results of single-timescale AC in the general MDP (Markov decision process) case are from [8] and [21], both of which obtain a sample complexity of  $O(\epsilon^{-2})$  under discounted reward setting. However, both [8] and [21] considered the i.i.d. sampling, where the transition tuples are independently sampled from stationary distribution and discounted state-action visitation distribution. In this paper, we consider the more practical Markovian sampling, where the transition tuples are generated from a single trajectory (see Table 1).
Furthermore, [8] follows an explicit Lyapunov analysis, where they leave a biased term in the critic and eliminated in the actor. Therefore, their proof framework cannot show the convergence of the critic. We instead give a neat proof framework to guarantee convergence for both the critic and the actor. Additionally, [21] only considered the tabular case (finite state-action space) where we allow the state space to be infinite (see Table 1). It is worth emphasizing that moving from a finite state space to an infinite state space takes significantly non-trivial effort in analysis. The analysis in [21] concatenates all state-action pairs to create a finite-dimensional feature matrix, which however becomes impossible in the infinite state space scenario. Consequently, their analysis technique and established results are not applicable in our context.
- Technically, we develop a new analysis framework that can establish the finite-time convergence for single-timescale AC under the general setting. The existing analysis for double-loop AC [37] and two-timescale AC [31] hinge on decoupling the analysis of actor and critic, which typically establishes the convergence of critic first and then actor [37, 31, 8]. We instead investigate the evolution of
the coupled estimation errors of the time-average reward, the critic, and the policy gradient norm altogether as an interconnected system in a much less conservative way. We emphasize that our analysis framework includes the discounted setting as a simple special case where the interconnected system is only two-dimensional without the time-average reward estimation.
Policy gradient methods. Policy gradient methods [26, 25] learn a parameterized policy, constituting a departure from the value-based approach [28, 32, 40]. The asymptotic convergence of policy gradient methods has been well established in [29, 26, 3, 14] via stochastic approximation methods [6]. Some recent works have shown that PG methods can find the global optimum of some particular class of problems, such as LQR [11, 19] and tabular case problem [1]. Under general function approximation setting, finite-time convergence of PG methods was analyzed in [1, 38, 33, 34]. Specifically, [1] established the finite-time convergence of PG methods under both tabular policy parameterizations and general parametric policy classes. [38] showed that a variant of PG methods can attain an  $\epsilon$ -accurate stationary point at a sample complexity of  $\mathcal{O}(\epsilon^{-2})$ , where they adopted Monte-Carlo sampling to find an unbiased estimation of policy gradient. [33, 34] studied the variance reduction PG and acceleration PG.
Actor-Critic methods. The AC algorithm was initially proposed by [15]. Later, [14] extended it to the natural AC algorithm. The asymptotic convergence of AC algorithms has been well established in [14, 5, 7, 39] under various settings. Many recent works focused on the finite-time convergence of AC methods. Under the double-loop setting, [37] established the global convergence of AC methods for solving linear quadratic regulator (LQR). [27] studied the global convergence of AC methods with both the actor and the critic parameterized by neural networks. [16] studied the finite-time local convergence of a few AC variants with linear function approximation.
Under the two-timescale AC setting, [31] established the finite-time local convergence to a stationary point at a sample complexity of  $\tilde{\mathcal{O}} (\epsilon^{-2.5})$  under the undiscounted time-average reward setting. [36] studied both local convergence and global convergence for two-timescale (natural) AC, with  $\tilde{\mathcal{O}} (\epsilon^{-2.5})$  and  $\tilde{\mathcal{O}} (\epsilon^{-4})$  sample complexity, respectively, under the discounted accumulated reward. The algorithm collects multiple samples to update the critic. [13] proposed a two-timescale stochastic approximation algorithm for bilevel optimization and the algorithm was subsequently employed in the context of two-timescale AC. [9] established the global convergence of two-timescale AC methods for solving LQR, where only a single sample is used to update the critic in each iteration.
Under the single-timescale setting, [12] considered the least-squares temporal difference (LSTD) update for the critic and obtained the optimal policy within the energy-based policy class for both linear function approximation and nonlinear function approximation using neural networks. [41] studied single-timescale AC on LQR. In addition, [8] and [21] considered the single-timescale AC in general MDP cases, which have been reviewed and compared in Section 1.1.
Notation. We use non- bold letters to denote scalars and use lower and upper case bold letters to denote vectors and matrices respectively. Without further specification, we write  $x_{n} = \mathcal{O}(y_{n})$  if there exists an absolute positive constant  $C$  such that  $x_{n}\leq Cy_{n}$  , for two sequences  $\{x_n\}$  and  $\{y_n\}$  .We use  $\tilde{\mathcal{O}} (\cdot)$  to hide logarithm factors. The total variation distance of two probability measure  $\mu$  and  $v$  is defined by  $d_{TV}(\mu ,v)\coloneqq \frac{1}{2}\int_{\mathcal{X}}|\mu (dx) - v(dx)|$  . In addition, we use  $\mathbb{P}$  to denote a generic probability of some random event.
In this section, we review the basics of the Markov decision process, policy gradient algorithm, and single-timescale AC with linear function approximation.
2.1 Markov decision process
We consider the standard Markov Decision Process (MDP) characterized by  $(\mathcal{S},\mathcal{A},\mathcal{P},r)$ , where  $\mathcal{S}$  is the state space and  $\mathcal{A}$  is the action space. We consider a finite action space  $|\mathcal{A}| < \infty$ , whereas the state space can be either a finite set or an (unbounded) real vector space  $\mathcal{S} \subset \mathbb{R}^n$ .  $\mathcal{P}(s_{t+1}|s_t,a_t) \in [0,1]$  denotes the transition kernel. We consider a bounded reward  $r: \mathcal{S} \times \mathcal{A} \to [-U_r,U_r]$ , which is a
function of the state  $s$  and action  $a$ . A policy  $\pi_{\theta}(\cdot | s) \in \mathbb{R}^{|A|}$  parameterized by  $\theta$  is defined as a mapping from a given state to a probability distribution over actions.
The RL problem of consideration aims to find a policy  $\pi_{\theta}$  that maximizes the infinite-horizon time-average reward [26, 25, 37, 31], which is given by
$$
J (\pmb {\theta}): = \lim _ {T \to \infty} \mathbb {E} _ {\pmb {\theta}} \frac {\sum_ {t = 0} ^ {T - 1} r (s _ {t} , a _ {t})}{T} = \underset {s \sim \mu_ {\pmb {\theta}}, a \sim \pi_ {\pmb {\theta}}} {\mathbb {E}} [ r (s, a) ],
$$
where the expectation  $\mathbb{E}_{\theta}$  is over the Markov chain under the policy  $\pi_{\theta}$ , and  $\mu_{\theta}$  denotes the stationary state distribution induced by  $\pi_{\theta}$ . The existence of the stationary distribution can be guaranteed by the uniform ergodicity of the underlying MDP, which is a common assumption. Hereafter, we refer to  $J(\theta)$  as the time-average reward (or exchangeably, performance function), which can be evaluated by the expected reward over the stationary distribution  $\mu_{\theta}$  and the policy  $\pi_{\theta}$ .
The state-value function is used to evaluate the overall rewards starting from a state  $s$  and following policy  $\pi_{\theta}$  thereafter, which is defined as
$$
V _ {\boldsymbol {\theta}} (s) := \mathbb {E} _ {\boldsymbol {\theta}} \left[ \sum_ {t = 0} ^ {\infty} \left(r \left(s _ {t}, a _ {t}\right) - J (\boldsymbol {\theta})\right) \mid s _ {0} = s \right],
$$
where the action follows the policy  $a_{t} \sim \pi_{\theta}(\cdot | s_{t})$  and the next state comes from the transition kernel  $s_{t + 1} \sim \mathcal{P}(\cdot | s_{t}, a_{t})$ . Similarly, we define the action-value (Q-value) function to evaluate the overall rewards starting from  $s$ , taking action  $a$ , and following policy  $\pi_{\theta}$  thereafter:
$$
\begin{array}{l} Q _ {\boldsymbol {\theta}} (s, a) = \mathbb {E} _ {\boldsymbol {\theta}} [ \sum_ {t = 0} ^ {\infty} (r (s _ {t}, a _ {t}) - J (\boldsymbol {\theta})) | s _ {0} = s, a _ {0} = a ] \\ \stackrel {\mathrm {(i)}} {=} r (s, a) - J (\boldsymbol {\theta}) + \mathbb {E} \left[ V _ {\boldsymbol {\theta}} \left(s ^ {\prime}\right) \right], \\ \end{array}
$$
where the expectation in (i) is taken over  $s' \sim \mathcal{P}(\cdot | s, a)$ .
2.2 Policy gradient theorem
The policy gradient theorem [26] provides an analytic expression for the gradient of the performance function  $J(\pmb{\theta})$  with respect to the policy parameter  $\pmb{\theta}$ , which is given by:
$$
\nabla_ {\pmb {\theta}} J (\pmb {\theta}) = \mathbb {E} _ {s \sim \mu_ {\pmb {\theta}}, a \sim \pi_ {\pmb {\theta}}} [ Q _ {\pmb {\theta}} (s, a) \nabla_ {\pmb {\theta}} \log \pi_ {\pmb {\theta}} (a | s) ]. \qquad (1)
$$
Evaluating this gradient requires the Q-value corresponding to the current policy  $\pi_{\theta}$ . The REINFORCE [29] is a Monte Carlo-based episodic algorithm, which uses all the rewards collected along the sample trajectory (that is, the differential return) as an approximation to the true Q-value.
Note that for any function  $b:\mathcal{S}\to \mathbb{R}$  that is independent of the action, we have
$$
\sum_ {a \in \mathcal {A}} b (s) \nabla \pi_ {\boldsymbol {\theta}} (a | s) = b (s) \nabla (\sum_ {a \in \mathcal {A}} \pi_ {\boldsymbol {\theta}} (a | s)) = b (s) \nabla 1 = 0.
$$
Therefore, the policy gradient theorem can be written equivalently as:
$$
\nabla J (\boldsymbol {\theta}) = \mathbb {E} _ {s \sim \mu_ {\boldsymbol {\theta}}, a \sim \pi_ {\boldsymbol {\theta}}} [ (Q _ {\boldsymbol {\theta}} (s, a) - b (s)) \nabla_ {\boldsymbol {\theta}} \log \pi_ {\boldsymbol {\theta}} (a | s) ],
$$
where  $b(s)$  is called the baseline function. A popular choice of baseline is the state-value function, which leads to the following advantage-based policy gradient
$$
\nabla_ {\boldsymbol {\theta}} J (\boldsymbol {\theta}) = \mathbb {E} _ {s \sim \mu_ {\boldsymbol {\theta}}, a \sim \pi_ {\boldsymbol {\theta}}} \left[ \Delta_ {\boldsymbol {\theta}} (s, a) \nabla_ {\boldsymbol {\theta}} \log \pi_ {\boldsymbol {\theta}} (a | s) \right],
$$
where  $\Delta_{\theta} = Q_{\theta}(s,a) - V_{\theta}(s)$  is known as the advantage function. This is the "REINFORCE with baseline" [29]. The baseline function can help reduce variance. However, like all Monte Carlo-based methods, it can still suffer from high variance and thus learns slowly. In addition, it is inconvenient to implement the algorithm online for continuing tasks [25].
AC algorithm instead employs a bootstrapping critic to estimate the Q-value. We describe the classic single-timescale AC in the next subsection.
2.3 The single-timescale actor-critic algorithm
We consider the classic single-sample single-timescale AC method, where the critic is bootstrapping and uses a single sampled reward to update in each iteration. This directly accommodates online learning for continuing tasks. We consider the following linear function approximation of the state-value function:
$$
\widehat {V} _ {\boldsymbol {\theta}} (s; \boldsymbol {\omega}) = \phi (s) ^ {\top} \boldsymbol {\omega},
$$
where  $\phi(\cdot): S \to \mathbb{R}^d$  is a known feature mapping, which satisfies  $\|\phi(\cdot)\| \leq 1$ . To drive  $\widehat{V}_{\theta}(s; w)$  towards its true value  $V_{\theta}(s)$ , the semi-gradient TD(0) update is applied to estimate the linear coefficient  $\omega$  (hereafter referred to as the critic):
$$
\boldsymbol {\omega} _ {t + 1} = \boldsymbol {\omega} _ {t} + \beta_ {t} \left[ \left(r _ {t} - J (\boldsymbol {\theta}) + \phi \left(s _ {t + 1}\right) ^ {\top} \boldsymbol {\omega} _ {t} - \phi \left(s _ {t}\right) ^ {\top} \boldsymbol {\omega} _ {t}\right) \right] \phi \left(s _ {t}\right), \tag {2}
$$
where  $\beta_{t}$  is the step size of the critic  $\omega$  and  $r_t\coloneqq r(s_t,a_t)$ . Since  $J(\pmb {\theta})$  is unknown, the time-average reward setting introduces an additional estimator  $\eta$  to estimate it. Hereafter, we simply refer to  $\eta$  as the reward estimator. The temporal difference error can be defined as
$$
\delta_ {t} := r _ {t} - \eta_ {t} + \phi (s _ {t + 1}) ^ {\top} \omega_ {t} - \phi (s _ {t}) ^ {\top} \omega_ {t}.
$$
Then, the update rule for the critic is given by
$$
\eta_ {t + 1} = \eta_ {t} + \gamma_ {t} \left(r _ {t} - \eta_ {t}\right),
$$
$$
\omega_ {t + 1} = \omega_ {t} + \beta_ {t} \delta_ {t} \phi (s _ {t}),
$$
where  $\gamma_{t}$  is the step size of the reward estimator  $\eta_t$ .
Since  $\delta_t$  is an approximation of the advantage function, similar to REINFORCE with baseline, the corresponding update rule for the actor can be written as:
$$
\boldsymbol {\theta} _ {t + 1} = \boldsymbol {\theta} _ {t} + \alpha_ {t} \delta_ {t} \nabla_ {\boldsymbol {\theta}} \log \pi_ {\boldsymbol {\theta} _ {t}} \left(a _ {t} | s _ {t}\right),
$$
where  $\alpha_{t}$  is the actor stepsize. The above-described AC is summarized in Algorithm 1, which is introduced in [25] as a classic online one-step AC algorithm. Algorithm 1 can be efficiently implemented for continuing tasks due to its online nature.
Algorithm 1 Single-timescale Actor-Critic
1: Input initial actor parameter  $\theta_0$ , initial critic parameter  $\omega_0$ , initial reward estimator  $\eta_0$ , stepsize  $\alpha_t$  for actor,  $\beta_t$  for critic, and  $\gamma_t$  for reward estimator.  
2: Draw  $s_0$  from some initial distribution  
3: for  $t = 0, 1, 2, \dots, T - 1$  do  
4: Take action  $a_t \sim \pi_{\theta_t}(\cdot | s_t)$   
5: Observe next state  $s_{t+1} \sim \mathcal{P}(\cdot | s_t, a_t)$  and reward  $r_t = r(s_t, a_t)$   
6:  $\delta_t = r_t - \eta_t + \phi(s_{t+1})^\top \omega_t - \phi(s_t)^\top \omega_t$   
7:  $\eta_{t+1} = \eta_t + \gamma_t (r_t - \eta_t)$   
8:  $\omega_{t+1} = \Pi_{U\omega} (\omega_t + \beta_t \delta_t \phi(s_t))$   
9:  $\theta_{t+1} = \theta_t + \alpha_t \delta_t \nabla_\theta \log \pi_{\theta_t}(a_t | s_t)$   
10: end for
Note that the "single-timescale" refers to the fact that the stepsizes  $\alpha_{t},\beta_{t},\gamma_{t}$  are only constantly proportional to each other. In addition, this is a "single-sample" algorithm, since only one sample is needed for the update in each iteration. We remark that Algorithm 1 is more common in practice than double loop variants. In Line 8 of Algorithm 1, a projection  $(\Pi_{U_{\omega}})$  is introduced to keep the critic norm-bounded by  $U_{\omega}$ , which is widely adopted in the literature [31, 37, 36, 8] for analysis. Note that the projection can be handled easily, which is relaxed using its non-expansive property in our analysis.
We first present several standard assumptions that are common in the literature of analyzing AC with linear function approximation [12, 35, 31, 8, 21]. Insights into these conditions and connections with relevant works are also discussed.
By taking the expectation of  $\omega_{t + 1}$  in (2) with respect to the stationary distribution, we have for any given  $\omega_{t}$
$$
\mathbb {E} _ {\boldsymbol {\theta}} \left[ \boldsymbol {\omega} _ {t + 1} \mid \boldsymbol {\omega} _ {t} \right] = \boldsymbol {\omega} _ {t} + \beta_ {t} \left(\boldsymbol {b} _ {\boldsymbol {\theta}} + \boldsymbol {A} _ {\boldsymbol {\theta}} \boldsymbol {\omega} _ {t}\right), \tag {3}
$$
$$
\boldsymbol {A} _ {\boldsymbol {\theta}} := \mathbb {E} _ {(s, a, s ^ {\prime})} [ \phi (s) (\phi (s ^ {\prime}) - \phi (s)) ^ {\top}) ], \tag {4}
$$
$$
\boldsymbol {b} _ {\boldsymbol {\theta}} := \mathbb {E} _ {(s, a)} [ (r (s, a) - J (\boldsymbol {\theta})) \phi (s) ], \tag {5}
$$
and  $s \sim \mu_{\pmb{\theta}}(\cdot), a \sim \pi_{\pmb{\theta}}(\cdot | s), s' \sim \mathcal{P}(\cdot | s, a)$  is the subsequent state of the  $(s, a)$ . It can be easily shown that [25] the TD limiting point  $\omega^{*}(\pmb{\theta})$  satisfies:
$$
\boldsymbol {b} _ {\theta} + \boldsymbol {A} _ {\theta} \omega^ {*} (\boldsymbol {\theta}) = 0. \tag {6}
$$
Note that  $A_{\theta}$  reflects the exploration of the policy. To see this, note that without sufficient exploration,  $A_{\theta}$  can be rank deficient and (6) can be unsolvable. Consequently, the critic update (2) will not converge. Hence, the following assumption is made to guarantee the problem's solvability.
Assumption 3.1 (Exploration). For any  $\theta$ , the matrix  $A_{\theta}$  defined in (4) is negative definite and its maximum eigenvalue can be upper bounded by  $-\lambda$ .
Assumption 3.1 is commonly adopted in analyzing TD learning with linear function approximation [4, 42, 31, 23, 8, 21]. In particular, Assumption 3.1 holds if the policy  $\pi_{\theta}$  can explore all state-action pairs in the tabular case [21]. In addition, with this assumption, we can choose  $U_{\omega} = \frac{2U_r}{\lambda}$  so that all  $\omega^{*}$  lie within the projection radius  $U_{\omega}$  because  $\| b_{\theta}\| \leq 2U_{r}$  and  $\| A_{\theta}^{-1}\| \leq \lambda^{-1}$ , which justifies the projection operator introduced in Line 8 of Algorithm 1.
Assumption 3.2 (Uniform ergodicity). For any  $\theta$ , denote  $\mu_{\theta}(\cdot)$  as the stationary distribution induced by the policy  $\pi_{\theta}(\cdot|s)$  and the transition probability measure  $\mathcal{P}(\cdot|s,a)$ . For a Markov chain generated by the policy  $\pi_{\theta}$  and transition kernel  $\mathcal{P}$ , there exists  $m > 0$  and  $\rho \in (0,1)$  such that
$$
d _ {T V} \left(\mathbb {P} (s _ {\tau} \in \cdot | s _ {0} = s), \mu_ {\boldsymbol {\theta}} (\cdot)\right) \leq m \rho^ {\tau}, \forall \tau \geq 0, \forall s \in \mathcal {S}.
$$
Assumption 3.2 assumes the Markov chain is geometrically mixing, which can be implied by the uniform ergodicity. It is commonly employed to characterize the noise induced by Markovian sampling in RL algorithms [4, 42, 31, 8, 21].