Title: Finite-Time Analysis of Single-Timescale Actor-Critic
Venue: NeurIPS 2023
Accepted: yes
Keywords: Finite-time analysis

single-timescale actor-critic
Abstract: Actor-critic methods have achieved significant success in many challenging applications. However, its finite-time convergence is still poorly understood in the most practical single-timescale form. Existing works on analyzing single-timescale actor-critic have been limited to i.i.d. sampling or tabular setting for simplicity. We investigate the more practical online single-timescale actor-critic algorithm on continuous state space, where the critic assumes linear function approximation and updates with a single Markovian sample per actor step.  Previous analysis has been unable to establish the convergence for such a challenging scenario. We demonstrate that the online single-timescale actor-critic method provably finds an $\epsilon$-approximate stationary point with $\widetilde{\mathcal{O}}(\epsilon^{-2})$ sample complexity under standard assumptions, which can be further improved to $\mathcal{O}(\epsilon^{-2})$ under the i.i.d. sampling. Our novel framework systematically evaluates and controls the error propagation between the actor and critic. It offers a promising approach for analyzing other single-timescale reinforcement learning algorithms as well.

Finite-Time Analysis of Single-Timescale Actor-Critic

Abstract

Actor-critic methods have achieved significant success in many challenging applications. However, its finite-time convergence is still poorly understood in the most practical single-timescale form. Existing works on analyzing single-timescale actor-critic have been limited to i.i.d. sampling or tabular setting for simplicity. We investigate the more practical online single-timescale actor-critic algorithm on continuous state space, where the critic assumes linear function approximation and updates with a single Markovian sample per actor step. Previous analysis has been unable to establish the convergence for such a challenging scenario. We demonstrate that the online single-timescale actor-critic method provably finds an ϵ -approximate stationary point with ˜ O ( ϵ -2 ) sample complexity under standard assumptions, which can be further improved to O ( ϵ -2 ) under the i.i.d. sampling. Our novel framework systematically evaluates and controls the error propagation between the actor and critic. It offers a promising approach for analyzing other single-timescale reinforcement learning algorithms as well.

1 Introduction

Actor-critic (AC) methods have achieved great success in solving many challenging reinforcement learning (RL) problems [17, 20, 24]. AC updates the actor (i.e., the policy) using the estimated policy gradient (PG), which is a function of the Q-value under the policy. Meanwhile, it employs a bootstrapping critic to estimate the Q-value, which often helps reduce variance and accelerates convergence in practice.

Despite the empirical success, the non-asymptotic convergence analysis of AC in the most practical single-timescale form remains underexplored. A large body of existing works consider the doubleloop variants, where the critic runs many steps to accurately estimate the Q-value for a given actor [37, 16, 27, 2]. This leads to a decoupled convergence analysis of the critic and the actor, which involves a policy evaluation sub-problem in the inner loop and a perturbed gradient descent in the outer loop. Its finite-time convergence is relatively well understood [16, 37, 35]. Nevertheless, the double-loop setting is mainly for ease of analysis, which is barely adopted in practice. Since it requires an accurate critic estimation, it is typically sample inefficient. In fact, it is unclear whether an inner loop of accurate policy evaluation is really necessary since it only corresponds to one transient policy among many iterations.

Another body of works considers the (single-loop) two-timescale variants [31, 9, 36, 13], where the actor and the critic are updated simultaneously in each iteration with stepsizes of different timescales. The actor stepsize is typically smaller than that of the critic, with their ratio converging to zero as the iteration number goes to infinity. Hence, the actor is updated much slower than the critic. The two-timescale allows the critic to approximate the desired Q-value in an asymptotic way, which enables a decoupled convergence analysis of the actor and the critic. This variant is occasionally

adopted to improve learning stability. However, it is still considered inefficient as the actor update is artificially slowed down.

In this paper, we consider the more practical single-timescale AC algorithm, which is the one introduced in many works of literature as well as in [25] as a classic AC algorithm. In singletimescale AC, the stepsizes of the critic and the actor diminish at the same timescale. Unlike the aforementioned variants, which have specialized designs aimed at simplifying the convergence analysis, the analysis of the single-timescale AC presents a greater challenge. Due to the substantial errors in critic estimation and the close coupling between the parallel critic update and actor update, the algorithm is more prone to unstable error propagation. It remains unclear under what condition the errors will converge to zero. To study its finite-time convergence, we consider the challenging undiscounted time-average reward formulation [25, 31, 37], which consists of three parallel updates: the (time-average) reward estimator, the critic estimator, and the actor estimator. We keep track of the reward estimation error, the critic error, and the policy gradient norm (which measures the actor error) by deriving an implicit bound for each of them. They are then analyzed altogether as an interconnected system inspired by [21] to establish the convergence simultaneously. Specifically, we identify the (constant) ratio between the actor stepsize and the critic stepsize, below which all three errors will diminish to zero, despite the inaccurate estimation in all three updates (reward estimation, critic, actor).

1.1 Main Contributions

We summarise our main contributions as follows:

· We provide a finite-time analysis for the single-timescale AC under the Markovian sampling and prove an ˜ O ( ϵ -2 ) sample complexity, where ˜ O ( · ) hides additional logarithmic terms. We further show that this sample complexity can be improved to O ( ϵ -2 ) under i.i.d. sampling, which matches the state-of-the-art performance of SGD on general non-convex optimization problems. Our proof clearly shows that the additional logarithmic term under the Markovian sampling is introduced by the mixing time of the underlying Markov chain.

· Our result compares favorably to existing works on single-timescale AC. To our knowledge, the only other results of single-timescale AC in the general MDP (Markov decision process) case are from [8] and [21], both of which obtain a sample complexity of O ( ϵ -2 ) under discounted reward setting. However, both [8] and [21] considered the i.i.d. sampling, where the transition tuples are independently sampled from stationary distribution and discounted state-action visitation distribution. In this paper, we consider the more practical Markovian sampling, where the transition tuples are generated from a single trajectory (see Table 1).

Furthermore, [8] follows an explicit Lyapunov analysis, where they leave a biased term in the critic and eliminated in the actor. Therefore, their proof framework cannot show the convergence of the critic. We instead give a neat proof framework to guarantee convergence for both the critic and the actor. Additionally, [21] only considered the tabular case (finite state-action space) where we allow the state space to be infinite (see Table 1). It is worth emphasizing that moving from a finite state space to an infinite state space takes significantly non-trivial effort in analysis. The analysis in [21] concatenates all state-action pairs to create a finite-dimensional feature matrix, which however becomes impossible in the infinite state space scenario. Consequently, their analysis technique and established results are not applicable in our context.

· Technically, we develop a new analysis framework that can establish the finite-time convergence for single-timescale AC under the general setting. The existing analysis for double-loop AC [37] and two-timescale AC [31] hinge on decoupling the analysis of actor and critic, which typically establishes the convergence of critic first and then actor [37, 31, 8]. We instead investigate the evolution of

the coupled estimation errors of the time-average reward, the critic, and the policy gradient norm altogether as an interconnected system in a much less conservative way. We emphasize that our analysis framework includes the discounted setting as a simple special case where the interconnected system is only two-dimensional without the time-average reward estimation.

1.2 Related Work

Policy gradient methods. Policy gradient methods [26, 25] learn a parameterized policy, constituting a departure from the value-based approach [28, 32, 40]. The asymptotic convergence of policy gradient methods has been well established in [29, 26, 3, 14] via stochastic approximation methods [6]. Some recent works have shown that PG methods can find the global optimum of some particular class of problems, such as LQR [11, 19] and tabular case problem [1]. Under general function approximation setting, finite-time convergence of PG methods was analyzed in [1, 38, 33, 34]. Specifically, [1] established the finite-time convergence of PG methods under both tabular policy parameterizations and general parametric policy classes. [38] showed that a variant of PG methods can attain an ϵ -accurate stationary point at a sample complexity of O ( ϵ -2 ) , where they adopted Monte-Carlo sampling to find an unbiased estimation of policy gradient. [33, 34] studied the variance reduction PG and acceleration PG.

Actor-Critic methods. The AC algorithm was initially proposed by [15]. Later, [14] extended it to the natural AC algorithm. The asymptotic convergence of AC algorithms has been well established in [14, 5, 7, 39] under various settings. Many recent works focused on the finite-time convergence of AC methods. Under the double-loop setting, [37] established the global convergence of AC methods for solving linear quadratic regulator (LQR). [27] studied the global convergence of AC methods with both the actor and the critic parameterized by neural networks. [16] studied the finite-time local convergence of a few AC variants with linear function approximation.

Under the two-timescale AC setting, [31] established the finite-time local convergence to a stationary point at a sample complexity of ˜ O ( ϵ -2 . 5 ) under the undiscounted time-average reward setting. [36] studied both local convergence and global convergence for two-timescale (natural) AC, with ˜ O ( ϵ -2 . 5 ) and ˜ O ( ϵ -4 ) sample complexity, respectively, under the discounted accumulated reward. The algorithm collects multiple samples to update the critic. [13] proposed a two-timescale stochastic approximation algorithm for bilevel optimization and the algorithm was subsequently employed in the context of two-timescale AC. [9] established the global convergence of two-timescale AC methods for solving LQR, where only a single sample is used to update the critic in each iteration.

Under the single-timescale setting, [12] considered the least-squares temporal difference (LSTD) update for the critic and obtained the optimal policy within the energy-based policy class for both linear function approximation and nonlinear function approximation using neural networks. [41] studied single-timescale AC on LQR. In addition, [8] and [21] considered the single-timescale AC in general MDP cases, which have been reviewed and compared in Section 1.1.

Notation. We use non-bold letters to denote scalars and use lower and upper case bold letters to denote vectors and matrices respectively. Without further specification, we write x n = O ( y n ) if there exists an absolute positive constant C such that x n ≤ Cy n , for two sequences { x n } and { y n } . We use ˜ O ( · ) to hide logarithm factors. The total variation distance of two probability measure µ and v is defined by d TV ( µ, v ) := 1 2 ∫ X | µ ( dx ) -v ( dx ) | . In addition, we use P to denote a generic probability of some random event.

2 Preliminaries

In this section, we review the basics of the Markov decision process, policy gradient algorithm, and single-timescale AC with linear function approximation.

2.1 Markov decision process

We consider the standard Markov Decision Process (MDP) characterized by ( S , A , P , r ) , where S is the state space and A is the action space. We consider a finite action space |A| < ∞ , whereas the state space can be either a finite set or an (unbounded) real vector space S ⊂ R n . P ( s t +1 | s t , a t ) ∈ [0 , 1] denotes the transition kernel. We consider a bounded reward r : S × A → [ -U r , U r ] , which is a

function of the state s and action a . A policy π θ ( ·| s ) ∈ R |A| parameterized by θ is defined as a mapping from a given state to a probability distribution over actions.

The RL problem of consideration aims to find a policy π θ that maximizes the infinite-horizon time-average reward [26, 25, 37, 31], which is given by

where the expectation E θ is over the Markov chain under the policy π θ , and µ θ denotes the stationary state distribution induced by π θ . The existence of the stationary distribution can be guaranteed by the uniform ergodicity of the underlying MDP, which is a common assumption. Hereafter, we refer to J ( θ ) as the time-average reward (or exchangeably, performance function), which can be evaluated by the expected reward over the stationary distribution µ θ and the policy π θ .

The state-value function is used to evaluate the overall rewards starting from a state s and following policy π θ thereafter, which is defined as

where the action follows the policy a t ∼ π θ ( ·| s t ) and the next state comes from the transition kernel s t +1 ∼ P ( ·| s t , a t ) . Similarly, we define the action-value (Q-value) function to evaluate the overall rewards starting from s , taking action a , and following policy π θ thereafter:

where the expectation in (i) is taken over s ′ ∼ P ( ·| s, a ) .

2.2 Policy gradient theorem

The policy gradient theorem [26] provides an analytic expression for the gradient of the performance function J ( θ ) with respect to the policy parameter θ , which is given by:

Evaluating this gradient requires the Q-value corresponding to the current policy π θ . The REINFORCE [29] is a Monte Carlo-based episodic algorithm, which uses all the rewards collected along the sample trajectory (that is, the differential return) as an approximation to the true Q-value.

Note that for any function b : S → R that is independent of the action, we have

Therefore, the policy gradient theorem can be written equivalently as:

where b ( s ) is called the baseline function. A popular choice of baseline is the state-value function, which leads to the following advantage-based policy gradient

where ∆ θ = Q θ ( s, a ) -V θ ( s ) is known as the advantage function. This is the 'REINFORCE with baseline' [29]. The baseline function can help reduce variance. However, like all Monte Carlo-based methods, it can still suffer from high variance and thus learns slowly. In addition, it is inconvenient to implement the algorithm online for continuing tasks [25].

AC algorithm instead employs a bootstrapping critic to estimate the Q-value. We describe the classic single-timescale AC in the next subsection.

2.3 The single-timescale actor-critic algorithm

We consider the classic single-sample single-timescale AC method, where the critic is bootstrapping and uses a single sampled reward to update in each iteration. This directly accommodates online learning for continuing tasks. We consider the following linear function approximation of the state-value function:

where ϕ ( · ) : S → R d is a known feature mapping, which satisfies ∥ ϕ ( · ) ∥ ≤ 1 . To drive ̂ V θ ( s ; w ) towards its true value V θ ( s ) , the semi-gradient TD(0) update is applied to estimate the linear coefficient ω (hereafter referred to as the critic):

where β t is the step size of the critic ω and r t := r ( s t , a t ) . Since J ( θ ) is unknown, the time-average reward setting introduces an additional estimator η to estimate it. Hereafter, we simply refer to η as the reward estimator. The temporal difference error can be defined as

Then, the update rule for the critic is given by

where γ t is the step size of the reward estimator η t .

Since δ t is an approximation of the advantage function, similar to REINFORCE with baseline, the corresponding update rule for the actor can be written as:

where α t is the actor stepsize. The above-described AC is summarized in Algorithm 1, which is introduced in [25] as a classic online one-step AC algorithm. Algorithm 1 can be efficiently implemented for continuing tasks due to its online nature.

Algorithm 1 Single-timescale Actor-Critic

1: Input initial actor parameter θ 0 , initial critic parameter ω 0 , initial reward estimator η 0 , stepsize α t for actor, β t for critic, and γ t for reward estimator.

2: Draw s 0 from some initial distribution

3: for t = 0 , 1 , 2 , · · · , T -1 do

4: Take action a t ∼ π θ t ( ·| s t )

5: Observe next state s t +1 ∼ P ( ·| s t , a t ) and reward r t = r ( s t , a t )

6: δ t = r t -η t + ϕ ( s t +1 ) ⊤ ω t -ϕ ( s t ) ⊤ ω t

7: η t +1 = η t + γ t ( r t -η t )

8: ω t +1 = Π U ω ( ω t + β t δ t ϕ ( s t ))

9: θ t +1 = θ t + α t δ t ∇ θ log π θ t ( a t | s t )

10: end for

Note that the 'single-timescale' refers to the fact that the stepsizes α t , β t , γ t are only constantly proportional to each other. In addition, this is a 'single-sample' algorithm, since only one sample is needed for the update in each iteration. We remark that Algorithm 1 is more common in practice than double loop variants. In Line 8 of Algorithm 1, a projection ( Π U ω ) is introduced to keep the critic norm-bounded by U ω , which is widely adopted in the literature [31, 37, 36, 8] for analysis. Note that the projection can be handled easily, which is relaxed using its non-expansive property in our analysis.

3 Main Results

We first present several standard assumptions that are common in the literature of analyzing AC with linear function approximation [12, 35, 31, 8, 21]. Insights into these conditions and connections with relevant works are also discussed.

3.1 Assumptions

By taking the expectation of ω t +1 in (2) with respect to the stationary distribution, we have for any given ω t

and s ∼ µ θ ( · ) , a ∼ π θ ( ·| s ) , s ′ ∼ P ( ·| s, a ) is the subsequent state of the ( s, a ) . It can be easily shown that [25] the TD limiting point ω ∗ ( θ ) satisfies:

Note that A θ reflects the exploration of the policy. To see this, note that without sufficient exploration, A θ can be rank deficient and (6) can be unsolvable. Consequently, the critic update (2) will not converge. Hence, the following assumption is made to guarantee the problem's solvability.

Assumption 3.1 (Exploration) . For any θ , the matrix A θ defined in (4) is negative definite and its maximum eigenvalue can be upper bounded by -λ .

Assumption 3.1 is commonly adopted in analyzing TD learning with linear function approximation [4, 42, 31, 23, 8, 21]. In particular, Assumption 3.1 holds if the policy π θ can explore all state-action pairs in the tabular case [21]. In addition, with this assumption, we can choose U ω = 2 U r λ so that all ω ∗ lie within the projection radius U ω because ∥ b θ ∥ ≤ 2 U r and ∥ A -1 θ ∥ ≤ λ -1 , which justifies the projection operator introduced in Line 8 of Algorithm 1.

Assumption 3.2 (Uniform ergodicity) . For any θ , denote µ θ ( · ) as the stationary distribution induced by the policy π θ ( ·| s ) and the transition probability measure P ( ·| s, a ) . For a Markov chain generated by the policy π θ and transition kernel P , there exists m> 0 and ρ ∈ (0 , 1) such that

Assumption 3.2 assumes the Markov chain is geometrically mixing, which can be implied by the uniform ergodicity. It is commonly employed to characterize the noise induced by Markovian sampling in RL algorithms [4, 42, 31, 8, 21].

Assumption 3.3 (Lipschitz continuity of policy) . Let π θ ( a | s ) be a policy parameterized by θ ∈ R d . There exists positive constants B,L l and L π such that for any θ , θ 1 , θ 2 ∈ R d , s ∈ S , and a ∈ A , it holds that:

(a) ∥∇ log π θ ( a | s ) ∥ ≤ B

(b) ∥∇ log π θ 1 ( a | s ) -∇ log π θ 2 ( a | s ) ∥ ≤ L l ∥ θ 1 -θ 2 ∥

Assumption 3.3 is standard in the literature of policy gradient methods [22, 42, 38, 34, 31, 8, 21]. This assumption holds for many policy classes such as Gaussian policy [10], Boltzmann policy [15], and tabular softmax policy [1].

Assumption 3.4. For any θ , θ ′ ∈ R d , there exists constant L µ such that ∥∇ µ θ -∇ µ θ ′ ∥ ≤ L µ ∥ θ -θ ′ ∥ , where µ θ ( s ) is the stationary distribution under the policy π θ .

Assumption 3.4 is introduced in [8] to show the smoothness of the optimal critic ω ∗ ( θ ) , which is critical to guarantee the convergence of single-timescale AC. This assumption holds for the finite state-action space setting [18].

3.2 Finite-Time Analysis

We define the following uniform upper bound for the linear function approximation error of the critic:

where

The error ϵ app is zero if V θ is indeed a linear function for any θ . Naturally, it can be expected that the learning errors of Algorithm 1 depend on ϵ app .

We define the following integer τ T that will be useful in the statement of the theorems, which depends on the number of total iterations T :

where m,ρ are constants defined in Assumption 3.2. Therefore, we choose τ T = log mρ -1 log ρ -1 + log T 2 log ρ -1 = O (log T ) such that mρ τ T -1 ≤ 1 √ T . The integer τ T represents the mixing time of an ergodic Markov chain, which will be used to control the Markovian noise in the analysis.

We quantify the learning errors by defining y t := η t -J ( θ t ) , which is the difference between the reward estimator and the true time-average reward J ( θ t ) at time t . For the critic, we define, z t := ω t -ω ∗ t with ω ∗ t := ω ∗ ( θ t ) to measure the error between the critic and its target value at iteration t . The following two theorems summarize our main results.

Theorem 3.5 (Markovian sampling) . Consider Algorithm 1 with α t = α = c √ T , β t = β = 1 √ T , γ t = γ = 1 √ T , where c is a constant depending on problem parameters. Suppose Assumptions 3.1-3.4 hold, we have for T ≥ 2 τ T ,

We defer the interpretation of the above results a bit and present below the analysis results under the i.i.d. sampling first for better comparison. The major difference of i.i.d. from Markovian sampling is that at the t -th iteration, the state s t is sampled from the stationary distribution µ θ t instead of the evolving Markov chain (see Algorithm 2 in Appendix E). The i.i.d. sampling simplifies the analysis in the way that many Markovian noise terms reduce to zero effectively. This leads to a tighter sample complexity bound compared to the Markovian sampling by up to logarithmic factors.

Theorem 3.6 (i.i.d. sampling) . Consider Algorithm 2 (see Appendix E) with α t = α = c √ T , β t = β = 1 √ T , γ t = γ = 1 √ T , where c is a constant depending on problem parameters. Suppose Assumptions 3.1-3.4 hold, we have for T ≥ 2 τ T ,

If the critic approximation error ϵ app is zero, we see that the reward estimator, the critic, and the actor estimation errors all diminish at a sub-linear rate of ˜ O ( T -1 2 ) . The additional logarithmic term hidden by ˜ O ( · ) is incurred by the mixing time of the Markov chain, which can be get rid of under the i.i.d. sampling. It also hides the polynomials of all other problem parameters. They are explicitly characterized in the proofs up to the last step of analyzing the overall interconnected error propagation system. One can easily keep and get the dependence orders of all parameters if needed. Here we focus on the dependence of the iteration number for ease of presentation.

To put the results into perspective, note that O ( T -1 2 ) is the rate one would obtain from stochastic gradient descent (SGD) on general non-convex functions with unbiased gradient updates. In terms of sample complexity, to obtain an ϵ -approximate stationary point, it takes a number of ˜ O ( ϵ -2 ) samples for Markovian sampling (Algorithm 1) and O ( ϵ -2 ) for i.i.d. sampling (Algorithm 2), which matches the state-of-the-art performance of SGD on non-convex optimization problems.

The obtained sample complexities are superior to those of other AC variants. Notably, [16] provided finite-time convergence for double-loop variant with a O ( ϵ -4 ) sample complexity and [31] analysed two-timescale variant, yielding a ˜ O ( ϵ -2 . 5 ) sample complexity. The sample complexity gap is intrinsic to their inefficient usage of data. In the double-loop setting, the critic starts over to estimate the Qvalue for an intermediate policy in the inner loop, ignoring the fact that the consecutive Q-values can be similar given a relatively minor policy update. The two-timescale setting artificially slows down the actor update by adopting an actor stepsize that decays faster than the critic. The single-timescale approach updates the critic and actor parallelly with proportional stepsizes and thus learns more efficiently.

Moreover, our result matches the O ( ϵ -2 ) sample complexity of policy gradient methods such as REINFORCE [2, 22] under the i.i.d sampling. It is previously found in [31] that there is a sample complexity gap between Algorithm 1 adopting two-timescale stepsizes and (variance-reduced) REINFORCE [22]. In this paper, we close this gap by providing a single-timescale analysis for Algorithm 1 which shows that this practical single-timescale AC can have the same sample complexity as REINFORCE.

3.3 Proof Sketch

The main challenge in the finite-time analysis lies in that the estimation errors of the time-average reward, the critic, and the policy gradient are strongly coupled. To overcome this difficulty, we view the propagation of these errors as an interconnected system and analyze them holistically. To better appreciate the advantage of our analysis framework over the decoupled methods that are traditionally adopted in analyzing double-loop and two-timescale variants, we sketch the main proof steps of Theorem 3.5 in the following. We also highlight the key challenges and techniques developed correspondingly. All supporting lemmas mentioned below can be found in Appendix.

We first derive implicit (coupled) upper bounds for the reward estimation error y t , the critic error z t , and the policy gradient ∇ J ( θ t ) , respectively. Then, we solve a system of inequalities to establish finite-time convergence.

Step 1: Reward estimation error analysis. Using the reward estimator update rule (Line 7 of Algorithm 1), we decompose the reward estimation error into:

The second term on the right-hand side of (7) is a bias term caused by the Markovian sample, which is characterized in Lemma C.1. As shown in Lemma E.1, this bias reduces to 0 under i.i.d. sampling after taking the expectation. The third term captures the variation of the moving targets J ( θ t ) . The double-loop variant of AC runs a policy evaluation sub-problem in the inner loop for each target J ( θ t ) to estimate the policy gradient accurately. This easily ensures the monotonic decreasing of J ( θ t ) and consequently the convergence. The two-timescale variant utilizes the additional property of lim t →∞ α t /β t = 0 to annihilate this term and consequently can have a decoupled analysis. In the case of single-timescale AC, we do not have the aforementioned special algorithm designs and properties to ease the analysis. Instead, we utilize the smoothness of J ( θ ) (see Lemma B.2) and derive an implicit upper bound for this term as a function of the norm of y t and ∇ J ( θ t ) . This bound will be combined with the implicit bounds derived in Step 2 and Step 3 below to establish the non-asymptotic convergence altogether. The last term in (7) reflects the variance in reward estimation, which is bounded by O ( γ t ) .

Step 2: Critic error analysis . Using the critic update rule (Line 8 of Algorithm 1), we decompose the squared error by (we neglect the projection for the time being for the ease of comprehension. The complete analysis can be found in the appendix.)

where O t := ( s t , a t , s t +1 ) is a tuple of observations and the definitions of g, ¯ g, ∆ g , and Ψ can be found in (12) and (13) in Appendix A. Without diving into the detailed definitions, here we focus on illustrating the high-level insights of our proof. First of all, the second term on the right-hand side of (8) can be bounded by -2 λβ t ∥ z t ∥ 2 under Assumption 3.1. It provides an explicit characterization of how sufficient exploration can help the convergence of learning. The third term is a Markovian noise, which is further bounded implicitly in Lemma C.3. For the i.i.d sampling case, as shown in Lemma E.1, this bias reduces to 0 after taking the expectation. The fourth term is caused by inaccurate reward and critic estimations, which can be bounded by the norm of y t and z t . The fifth term tracks both the critic estimation performance z t and the difference between the drifting critic targets ω ∗ t . Similar to the case of Step 1, the double-loop approach bounds this term relying on the accurate policy evaluation sub-problem in the inner loop for each target ω ∗ t , whereas the two-timescale approach ensures its convergence by additionally requiring lim t →∞ α t /β t = 0 . In contrast, we establish an implicit upper bound for this term as a function of y t and z t by utilizing the smoothness of the optimal critic proved in Lemma B.4. Finally, the last term reflects the variances of various estimations, which is bounded by O ( β t ) .

Step 3: Policy gradient norm analysis . Using the actor update rule (Line 9 of Algorithm 1) and the smoothness property of J ( θ ) (see Lemma B.2), we derive

where O ′ t is a shorthand for an independent sample from stationary distribution s ∼ µ θ t , a ∼ π θ t , s ′ ∼ P ( ·| s, a ) , Θ is defined in (13), and L J ′ is a constant. The first term on the right-hand side of (9) compares the actor's performances between consecutive updates, which can be bounded via Abel summation by parts. The second term is a noise term introduced by Markovian sampling, which is characterized in Lemma C.6. Again, as proven in Lemma E.1, this bias reduces to 0 under i.i.d. sampling after taking the expectation. The third term is an error introduced by the inaccurate estimations of both the time-average reward and the critic. This term was directly bounded to zero under both the double-loop setting and the two-timescale setting due to their particular algorithm design, to enable a decoupled analysis. We control this term by providing an implicit bound depending on y t , z t , and ∇ J ( θ t ) . The fourth term comes from the linear function approximation error. The last term captures the variance of the stochastic gradient update, which is bounded by O ( α t ) .

Step 4: Interconnected iteration system analysis. Taking the expectation of and summing (7), (8), and (9) from τ T to T -1 , respectively, we obtain the following system of inequalities in terms of Y T , Z T , G T :

where l 1 , l 2 , l 3 , l 4 are positive constants. By solving the above system of inequalities, we further prove that if

then Y T , Z T , G T converge at a rate of O ( log 2 T √ T ) . This condition can be easily satisfied by choosing the stepsize ratio c to be smaller than a threshold identified in Equation (28). Thus, it completes the proof.

The above proof applies to i.i.d sampling straightforwardly, with the corresponding terms pointed out in the above steps reducing to 0 in the analysis. The additional proof can be found in Lemma E.1.

4 Conclusion and Discussion

In this paper, we establish the finite-time analysis for single-timescale AC with Markovian sampling. Our work compares favorably to existing works in terms of analyzing online learning and considering the continuous state space. We developed a series of lemmas that characterize the propagation of errors, and establish their convergence simultaneously by solving a system of nonlinear inequalities. The proposed framework is general and can be applied to analyze other single-timescale stochastic approximation algorithms. Our future work includes further considering the continuous action space problems and developing new proof techniques that require fewer assumptions.

References

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. In Conference on Learning Theory , pages 64-66. PMLR, 2020.

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research , 22(98):1-76, 2021.

Jonathan Baxter and Peter L Bartlett. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research , 15:319-350, 2001.

Jalaj Bhandari, Daniel Russo, and Raghav Singal. A finite time analysis of temporal difference learning with linear function approximation. In Conference on learning theory , pages 16911692. PMLR, 2018.

Shalabh Bhatnagar, Richard S Sutton, Mohammad Ghavamzadeh, and Mark Lee. Natural actor-critic algorithms. Automatica , 45(11):2471-2482, 2009.

Vivek S Borkar. Stochastic approximation: a dynamical systems viewpoint , volume 48. Springer, 2009.

Dotan Di Castro and Ron Meir. A convergent online single time scale actor critic algorithm. The Journal of Machine Learning Research , 11:367-410, 2010.

Tianyi Chen, Yuejiao Sun, and Wotao Yin. Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems. Advances in Neural Information Processing Systems , 34:25294-25307, 2021.

Xuyang Chen, Jingliang Duan, Yingbin Liang, and Lin Zhao. Global convergence of twotimescale actor-critic for solving linear quadratic regulator. arXiv preprint arXiv:2208.08744 , 2022.

Kenji Doya. Reinforcement learning in continuous time and space. Neural computation , 12(1):219-245, 2000.

Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In International Conference on Machine Learning , pages 1467-1476. PMLR, 2018.

Zuyue Fu, Zhuoran Yang, and Zhaoran Wang. Single-timescale actor-critic provably finds globally optimal policy. arXiv preprint arXiv:2008.00483 , 2020.

Mingyi Hong, Hoi-To Wai, Zhaoran Wang, and Zhuoran Yang. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actorcritic. SIAM Journal on Optimization , 33(1):147-180, 2023.

Sham M Kakade. A natural policy gradient. Advances in neural information processing systems , 14, 2001.

Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems , 12, 1999.

Harshat Kumar, Alec Koppel, and Alejandro Ribeiro. On the sample complexity of actorcritic method for reinforcement learning with function approximation. arXiv preprint arXiv:1910.08412 , 2019.

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436-444, 2015.

Qijun Luo and Xiao Li. Finite-time analysis of fully decentralized single-timescale actor-critic. arXiv preprint arXiv:2206.05733 , 2022.

Dhruv Malik, Ashwin Pananjady, Kush Bhatia, Koulik Khamaru, Peter Bartlett, and Martin Wainwright. Derivative-free methods for policy optimization: Guarantees for linear quadratic systems. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2916-2925. PMLR, 2019.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928-1937. PMLR, 2016.

Alex Olshevsky and Bahman Gharesifard. A small gain analysis of single timescale actor critic. SIAM Journal on Control and Optimization , 61(2):980-1007, 2023.

Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, and Marcello Restelli. Stochastic variance-reduced policy gradient. In International conference on machine learning , pages 4026-4035. PMLR, 2018.

Shuang Qiu, Zhuoran Yang, Jieping Ye, and Zhaoran Wang. On finite-time convergence of actor-critic algorithm. IEEE Journal on Selected Areas in Information Theory , 2(2):652-664, 2021.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature , 550(7676):354-359, 2017.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999.

Lingxiao Wang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. arXiv preprint arXiv:1909.01150 , 2019.

Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning , 8(3):279-292, 1992.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3):229-256, 1992.

Yue Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite time analysis of two time-scale actor critic methods. arXiv preprint arXiv:2005.01350 , 2020.

Yue Frank Wu, Weitong Zhang, Pan Xu, and Quanquan Gu. A finite-time analysis of two time-scale actor-critic methods. Advances in Neural Information Processing Systems , 33:1761717628, 2020.

Huaqing Xiong, Lin Zhao, Yingbin Liang, and Wei Zhang. Finite-time analysis for double q-learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020 (spotlight) , 2020.

Pan Xu, Felicia Gao, and Quanquan Gu. Sample efficient policy gradient methods with recursive variance reduction. arXiv preprint arXiv:1909.08610 , 2019.

Pan Xu, Felicia Gao, and Quanquan Gu. An improved convergence analysis of stochastic variance-reduced policy gradient. In Uncertainty in Artificial Intelligence , pages 541-551. PMLR, 2020.

Tengyu Xu, Zhe Wang, and Yingbin Liang. Improving sample complexity bounds for (natural) actor-critic algorithms. Advances in Neural Information Processing Systems , 33:4358-4369, 2020.

Tengyu Xu, Zhe Wang, and Yingbin Liang. Non-asymptotic convergence analysis of two time-scale (natural) actor-critic algorithms. arXiv preprint arXiv:2005.03557 , 2020.

Zhuoran Yang, Yongxin Chen, Mingyi Hong, and Zhaoran Wang. Provably global convergence of actor-critic: A case for linear quadratic regulator with ergodic cost. Advances in neural information processing systems , 32, 2019.

Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization , 58(6):3586-3612, 2020.

Shangtong Zhang, Bo Liu, Hengshuai Yao, and Shimon Whiteson. Provably convergent twotimescale off-policy actor-critic with function approximation. In International Conference on Machine Learning , pages 11204-11213. PMLR, 2020.

Lin Zhao, Huaqing Xiong, and Yingbin Liang. Faster non-asymptotic convergence for double q-learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021 , 2021.

Mo Zhou and Jianfeng Lu. Single timescale actor-critic method to solve the linear quadratic regulator with convergence guarantees. Journal of Machine Learning Research , 24(222):1-34, 2023.

Shaofeng Zou, Tengyu Xu, and Yingbin Liang. Finite-sample analysis for sarsa with linear function approximation. Advances in neural information processing systems , 32, 2019.

Table of Contents

A Notation

We make use of the following auxiliary Markov chain to deal with the Markovian noise.

Auxiliary Markov Chain:

For reference, we also show the original Markov chain.

Original Markov Chain:

In the sequel, we denote by ˜ O t := ( ˜ s t , ˜ a t , ˜ s t +1 ) the tuple generated from the auxiliary Markov chain in (10) while O t := ( s t , a t , s t +1 ) denotes the tuple generated from the original Markov chain in (11).

We define the following functions, which will benefit to decompose the errors and simplify the presentation.

We also define the following functions, which characterize the Markovian noise.

where O ′ is a shorthand for an independent sample from stationary distribution s ∼ µ θ , a ∼ π θ , s ′ ∼ P . Define U δ := 2 U r +2 U ω so that we have | δ t | ≤ U δ , where δ t comes from Line 6 in Algorithm 1. Note that from Assumption 3.3, we have ∥ δ ∇ log π θ ∥ ≤ G := U δ B .

B Preliminary Lemmas

Lemma B.1 ([31], Lemma C.4) . For any θ 1 , θ 2 , we have

where L J = 2 U r |A| L π (1 + ⌈ log ρ m -1 ⌉ + 1 1 -ρ ) .

Lemma B.2 ([38], Lemma 3.2) . For the performance function J ( θ ) , there exists a constant L J ′ > 0 such that for all θ 1 , θ 2 ∈ R d , it holds that

which further implies

Lemma B.3 ([31], Proposition 4.4) . There exists a constant L ∗ > 0 such that

Lemma B.4 ([8], Proposition 8) . For any θ 1 , θ 2 ∈ R d , we have

where L s is a positive constant.

Lemma B.5 ([42],[31]) . For any θ 1 and θ 2 , it holds that

Lemma B.6 ([31], Lemma B.2) . Given time indexes t and τ such that t ≥ τ > 0 , consider the auxiliary Markov chain in (10) . Conditioning on s t -τ +1 and θ t -τ , we have

C Proof of Main Theorem

C.1 Step 1: Reward estimation error analysis

In this subsection, we will establish an implicit bound for estimator.

Lemma C.1. From any t ≥ τ > 0 , we have

Theorem C.2. Choose α t = c √ T , β t = γ t = 1 √ T , we have

Proof. From the update rule of reward estimator in Line 7 of Algorithm 1, we have

Then we have

Taking expectation up to s t +1 (the whole trajectory), rearranging and summing from τ T to T -1 , we have

For term I 1 , from Abel summation by parts, we have

For term I 2 , from Lemma C.1, we have

Choose τ = τ T , we have

For I 3 , if y t > 0 , from (15), we have

If y t ≤ 0 , from (16), we have

Overall, we get

For term I 4 , we have

For term I 5 , we have

Therefore, we get

Since τ T = O (log T ) , we have T T -τ T ≤ 2 √ for large T . Then we get

Thus we finish the proof.

C.2 Step 2: Critic error analysis

In this subsection, we will establish an implicit upper bound for critic.

Lemma C.3. For any t ≥ τ > 0 , we have

Lemma C.4. Given the definition of Ξ( O t , ω t , θ t ) , for any t ≥ τ > 0 , we have

where C 2 := 3 BU 2 δ |A| L π (1 + ⌈ log ρ m -1 ⌉ + 1 1 -ρ ) + 3 U 2 δ L l +8 U δ BL ∗ .

Theorem C.5. Choose α t = c √ T , β t = γ t = 1 √ T , we have

Proof. From the update rule of critic in Line 8 of Algorithm 1, we have

Therefore, we have

Note that we have

where the first equality is due to the fact that ¯ g ( ω ∗ t , θ t ) = 0 and the last inequality follows from Assumption 3.1.

Taking expectation up to s t +1 , we have

where (1) follows from the L s -smoothness of ω ∗ in Lemma B.4; (2) uses L s G 2 2 α 2 t ≤ λβ t for large T . For term E ⟨ z t , -( ∇ ω ∗ t ) ⊤ δ t ∇ log π θ t ( a t | s t ) ⟩ , we have

Note that from Cauchy-Schwartz inequality and L ∗ is the Lipschitz constant of ω ∗ in Lemma B.3, we have

Furthermore, it holds that

Therefore, we have

Substituting (21) and (22) into (20) yields

Plugging (23) into (19), we have

Rearranging and summing from τ T to T gives

In the sequel, we will tackle I 1 , I 2 , I 3 , I 4 , I 5 respectively.

For term I 1 , from Abel summation by parts, we have

where the inequality is due to E ∥ z t ∥ 2 ≤ U 2 δ and discard the last term.

For term I 2 , from Lemma C.3, choose τ = τ T , we have

Then we get

For term I 3 , from Lemma C.4, choose τ = τ T , we have

Therefore, we have

For term I 4 and I 5 , from Cauchy-Schwartz inequality, we have

Overall, we get

Therefore, we have

where (1) follows from τ T = O (log T ) so that T -τ T ≥ 1 2 T for large T . Therefore, we have

which completes the proof.

C.3 Step 3: Policy gradient norm analysis

In this subsection, we will establish an implicit upper bound for policy gradient norm.

Lemma C.6. For any t ≥ τ > 0 , it holds that

where D 1 = max { U δ BL J ′ +3 L J L h , 2 U δ BL J |A| L π } and D 2 = 4 U δ BL J Theorem C.7. We have

.

Proof. From the update rule of actor in Line 9 of Algorithm 1 and 15, we have

where the last equality is due to the fact

Rearranging the above inequality and taking expectation, we have

Note that from Cauchy-Schwartz inequality, we have

From Lemma C.6 and choosing τ = τ T , we have

It has been shown that

Therefore, we have

where we use ∥∇ J ( θ ) ∥ ≤ L J which comes from Lemma B.1. Plugging the three terms yields

Summing over t from τ T to T -1 gives

For term I 1 , from Abel summation by parts, we have

Overall, we have

Therefore, we get

which concludes the proof.

C.4 Step 4: Interconnected iteration system analysis

In this subsection, we perform an interconnected iteration system analysis to prove Theorem 3.5. Proof of Theorem 3.5.

Proof. Combining (17), (18), and (25), we have

Denote

Then we have

For Y T , we get

For G T , we get

For Z T , we have

If it satisfies 4 l 3 ≤ 1 4 , we further have

Plugging (26) and (27) into the above inequality gives

Therefore, if l 1 (1 + 2 l 2 4 +8 l 2 4 (2 l 2 2 + l 3 ) ≤ 1 , we have

According to the definition of l 1 , l 2 , l 3 , l 4 , we have

which satisfies the above two inequalities. Therefore, we have

Thus we conclude our proof.

D Proof of Supporting Lemmas

The following three lemmas only deal with the Markovian noise, which are originally proved in [31] and updated in [30]. We include the proof with slight modifications for proving Theorem 3.5.

Proof of Lemma C.1 .

Proof. We will divide the proof of this lemma into four steps.

Step 1: show that for any θ 1 , θ 2 , η, O = ( s, a, s ′ ) , we have

By the definition of Φ( O,η, θ ) in (13), we have

Step 2: show that for any θ , η 1 , η 2 , O , we have

Overall, we require

Thus we choose and consequently,

By definition, we have

Step 3: show that for original tuple O t and the auxiliary tuple ˜ O t , conditioned on s t -τ -1 and θ t -τ , we have

By definition, we have

By definition of total variation norm, we have

Repeat the above argument from t to t -τ +1 , we have

Plugging (33) into (32), we have

Step 4: show that conditioned on s t -τ +1 and θ t -τ , we have

Note that according to definition, we have

where O ′ t -τ = ( s ′ t -τ , a ′ t -τ , s ′ t -τ +1 ) is the tuple generated by s ′ t -τ ∼ µ t -τ , a ′ t -τ ∼ π θ t -τ , s ′ t -τ +1 ∼ P . From the uniform ergodicity in Assumption 3.2, it shows that

Then we have

Combing (29), (30), (31), and (34), we have

which concludes the proof.

Proof of Lemma C.3 .

Proof. We will divide the proof of this lemma into four steps.

Step 1: show that for any θ 1 , θ 2 , ω and tuple O = ( s, a, s ′ ) , we have

where C 1 = 2 U 2 δ |A| L π (1 + ⌈ log ρ m -1 ⌉ + 1 1 -ρ ) + 2 U δ L J +2 U δ L ∗ .

By definition of Ψ( O, ω , θ ) in (13), we have

For term I 1 , we have

where we use the fact that U δ = 2 U r +2 U ω and the last inequality comes from Lemma B.5. For term I 2 , from Cauchy-Schwartz inequality, we have

Combining the results from I 1 and I 2 , we get

where C 1 = 2 U 2 δ |A| L π (1 + ⌈ log ρ m -1 ⌉ + 1 1 -ρ ) + 2 U δ L J +2 U δ L ∗ .

Step 2: show that for any θ , ω 1 , ω 2 and tuple O ( s, a, s ′ ) , we have

By definition, we have

where the last inequality is due to ∥ g ( O, ω 1 , θ ) -g ( O, ω 2 , θ ) ∥ = | ( ϕ ( s ′ ) -ϕ ( s )) ⊤ ( ω 1 -ω 2 ) | ≤ 2 ∥ ω 1 -ω 2 ∥ , ∥ ¯ g ( ω 1 , θ ) -¯ g ( ω 2 , θ ) ∥ ≤ 2 ∥ ω 1 -ω 2 ∥ , and 2 U ω ≤ U δ .

Step 3: show that for tuples O t = ( s t , a t , s t +1 ) and ˜ O t = ( ˜ s t , ˜ a t , ˜ s t +1 ) . Conditioning on s t -τ +1 and θ t -τ , we have

By the definition of total variation norm, we have

where (1) follows from (33).

Step 4: show that conditioning on s t -τ +1 and θ t -τ ,

From the definition of Ψ( O, ω , θ ) , we have

where O ′ t -τ is the tuple generated by s ′ t -τ ∼ µ θ t -τ , a ′ t -τ ∼ π θ t -τ , s ′ t -τ +1 ∼ P . From Assumption 3.2, we have

Then, it holds that

Combining (35), (36), (37), and (38), we have

Proof of Lemma C.4 .

Proof. We will divide the proof of this lemma into four steps.

Step 1: show that

Since Ξ( O t , ω t , θ t ) = ⟨ ω t -ω ∗ t , E O ′ [ h ( O ′ , θ )] -h ( O, θ ) ⟩ , we define E θ [ h ( O ′ , θ )] := E O ′ [ h ( O ′ , θ )] , where E θ is the shorthand of E O ′ ∼ ( µ θ ,π θ , P ) . In the following, we will show that each term in Ξ( O t , ω t , θ t ) is Lipschitz.

Term ω t is not related to θ and term ω ∗ t := ω ∗ ( θ t ) is L ∗ -Lipschitz.

For term h ( O, θ ) , denote δ ( O, θ ) := r ( s, a ) -r ( θ ) + ( ϕ ( s ′ ) -ϕ ( s )) ⊤ ω ∗ , we have

Hence we have h ( O, θ ) is L h -Lipschitz, where L h denotes the above coefficient. For term E θ [ h ( O ′ , θ )] , we have

Then we have ω t -ω ∗ t is U δ -bounded and L ∗ -Lipschitz; h ( O, θ ) -E θ [ h ( O ′ , θ )] is 3 L h -Lipschitz and 2 U δ B -bounded. By the triangle inequality, we have

Step 2: show that

Actually, we have

Step 3: show that for tuples O t = ( s t , a t , s t +1 ) and ˜ O t = ( ˜ s t , ˜ a t , ˜ s t +1 ) . Conditioning on s t -τ +1 and θ t -τ , we have

By definition of Ξ( O, ω , θ ) , we have

where the inequality comes from the definition of total variation distance. The total variation norm between O t and ˜ O t has been computed in (33). Plugging (33) into (42), we get

Step 4: Show that conditioning on s t -τ +1 and θ t -τ , we have

It can be shown that

where (1) is due to the fact that O ′ t is from the stationary distribution which satisfies E [Ξ( O ′ t , ω t -τ , θ t -τ )] = 0 and (2) follows from the definition of total variation distance. From Assumption 3.2, we know that

We also have the fact that

Therefore, we have

Combining (39)-(43), we can decompose the Markovian bias as

Thus we conclude our proof.

Proof of Lemma C.6 .

Proof. We will divide the proof of this lemma into three steps.

Step 1: show that

Since Θ( O, θ ) = ⟨∇ J ( θ ) , E θ [ h ( O ′ , θ )] -h ( O, θ ) ⟩ , we will show that each term in Θ( O, θ ) is Lipschitz.

For the term ∇ J ( θ ) , by Lemma B.3 we know it's L J ′ -Lipschitz.

For term h ( O, θ ) , denote δ ( O, θ ) := r ( s, a ) -r ( θ ) + ( ϕ ( s ′ ) -ϕ ( s )) ⊤ ω ∗ , we have

Hence we have h ( O, θ ) is L h -Lipschitz, where L h denotes the above coefficient. For term E θ [ h ( O ′ , θ )] , we have

Then we have ∇ J ( θ ) is L J -bounded and L J ′ -Lipschitz; h ( O, θ ) -E θ [ h ( O ′ , θ )] is 3 L h -Lipschitz and 2 U δ B -bounded. By the triangle inequality, we have

Step 2: show that for t ≥ τ > 0 , we have

By definition of Θ( O, θ ) , we have

where the inequality comes from the definition of total variation distance. The total variation norm between O t and ˜ O t has been computed in (33). Plugging (33) into (46), we get

Step 3: show that for t ≥ τ > 0 , we have

From the definition of Θ( O, θ ) , we have

The inequality is due to the definition of total variation distance. From Assumption 3.2, we know that d TV ( P ( s t ∈ · ) , µ θ ) ≤ mρ τ -1 .

We also have the fact that

Therefore, we have

Combining (44), (45), and (47), we can decompose the Markovian bias as

where ˜ O t is from the auxiliary Markovian chain defined in (10) and O ′ t is from the stationary distribution which satisfies E [Θ( O ′ t , θ t -τ )] = 0 .

Then we have

where D 1 = max { U δ BL J ′ +3 L J L h , 2 U δ BL J |A| L π } and D 2 = 4 U δ BL J . Thus we conclude the proof.

E IID Sampling Analysis

Algorithm 2 Single-timescale Actor-Critic (i.i.d. sampling)

1: Input initial actor parameter θ 0 , initial critic parameter ω 0 , initial reward estimator η 0 , stepsize α t for actor, β t for critic and γ t for reward estimator.

2: for t = 0 , 1 , 2 , · · · , T -1 do

3: Sample s t ∼ µ θ t

4: Take the action a t ∼ π θ t ( ·| s t )

6: δ t = r t -η t + ϕ ( s ′ t ) ⊤ ω t -ϕ ( s t ) ⊤ ω t

5: Observe next state s ′ t ∼ P ( ·| s t , a t ) and the reward r t = r ( s t , a t )

7: η t +1 = η t + γ t ( r t -η t )

8: ω t +1 = Π U ω ( ω t + β t δ t ϕ ( s t ))

9: θ t +1 = θ t + α t δ t ∇ θ log π θ t ( a t | s t )

10: end for

Note that under i.i.d. sampling in Algorithm 2, we denote by s t the samples from the stationary distribution and s ′ t the subsequent state following transition kernel s ′ t ∼ P ( ·| s t , a t ) . Correspondingly, we redefine the observation tuple as O t = ( s t , a t , s ′ t ) (in contrast to O t = ( s t , a t , s t +1 ) in the Markovian sampling case). This modification implies the decoupling of O t and O t +1 since s t +1 in tuple O t +1 is a new state sampled from the stationary distribution rather than inherited from O t . This intuitively elucidates the vanishment of Markovian noise under i.i.d. sampling.

Lemma E.1. Under i.i.d sampling, we have

Proof. Note that the expectation is taken over all the random variables. We use the notation O t to denote the tuple ( s t , a t , s ′ t ) and v 0: t to denote the sequence ( s t , a t , s ′ t ) , ( s t , a t , s ′ t ) , · · · , ( s t , a t , s ′ t ) . By definition in (13), it can be shown that

where is second equality is due to law of total expectation. Once we know v 0: t -1 , η t and J ( θ t ) is not a random variable any more. It holds that

where the last equation is due to E O t [( r t -J ( θ t )) | v 0: t -1 ] = 0 under i.i.d. sampling.

By a similar argument, we have

where we use the fact that E O t [ g ( O t , ω t , θ t ) -¯ g ( ω t , θ t ) ⟩| v 0: t -1 ] = 0 .

Similarly, we have

where we use fact that O t = O ′ t under i.i.d. sampling.

The proof of E [Ξ( O t , ω t , θ t )] = 0 is the same as the above argument, which concludes the proof.

Proof of Theorem 3.6 .

Proof. The proof follows similarly to the Markovian sampling case. Specifically, all the Markovian noises (see the definitions in (13)) present in the former analysis reduce to zero after taking expectations. The detailed results and proof are presented in Lemma E.1. Then, replacing Lemma C.1, Lemma C.3, and Lemma C.6 with Lemma E.1, we will get the desired O ( T -1 2 ) convergence rate and thus an O ( ϵ -2 ) sample complexity accordingly.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies finding the optimal policy in an infinite-horizon average-reward MDP with an online, sample-based algorithm. The state-of-the-art algorithm in this setting uses two different timescales and is known to have a sample complexity of $\\widetilde{\\mathcal{O}}(\\epsilon^{-2.5})$. This paper shows that a single-timescale version of the actor-critic algorithm enjoys an improved complexity of $\\widetilde{\\mathcal{O}}(\\epsilon^{-2})$, up to a logarithm factor. If we were always able to sample according to the stationary distribution under the current policy, the logarithm factor can be removed.\n\nSTRENGTHS:\nBeing able to improve the convergence rate from $\\widetilde{\\mathcal{O}}(\\epsilon^{-2.5})$ to $\\widetilde{\\mathcal{O}}(\\epsilon^{-2})$ is a pretty significant contribution. As the authors noted, this rate already matches the rate of standard SGD for non-convex functions. The paper is also well-written.\n\nWEAKNESSES:\nWhile the presentation of the paper is mostly clear, more discussion how exactly the authors improved the convergence rate and why previous works failed to do that is needed for the audience to appreciate the technical contribution. In fact, many existing works on two-timescale AC do not start off trying to the make updates two-timescale. Making the step sizes for the actor and critic decay at the same rate is allowed, but doing so hurts their convergence rates. Do the authors take advantage of any other structure of the MDP beyond general non-convexity to make a single timescale more favorable?\n\nQUESTIONS:\n- Related work section is somewhat limited. The connection of the paper to existing works on single-loop AC algorithms including the following ones should be discussed.\n\nHong, M., Wai, H.T., Wang, Z. and Yang, Z., 2023. A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor-critic. SIAM Journal on Optimization, 33(1), pp.147-180.\n\nZeng, S., Doan, T.T. and Romberg, J., 2021. A two-time-scale stochastic optimization framework with applications in control and reinforcement learning. arXiv preprint arXiv:2109.14756.\n\nKhodadadian, S., Doan, T.T., Romberg, J. and Maguluri, S.T., 2022. Finite sample analysis of two-time-scale natural actor-critic algorithm. IEEE Transactions on Automatic Control.\n\nZhou, M. and Lu, J., 2022. Single Time-scale Actor-critic Method to Solve the Linear Quadratic Regulator with Convergence Guarantees. arXiv preprint arXiv:2202.00048.\n\n\n- I do not feel that the discussion of i.i.d. sampling case and the theorem adds much to the paper. The claimed contribution on Markovian sampling also seems a bit oversold. Many existing works have shown how Markovian samples can be handled at the cost of introducing a log factor, which does not appear if the samples are i.i.d.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work studies the actor-critic algorithm’s convergence under the single-timescale update where the step-size of actor and critic variables are only proportional by a  constant.  Authors suggest the epsilon-optimal solution with a sample complexity $\\tilde{O}(\\epsilon^{-2})$  under standard assumptions and $O(\\epsilon^{-2})$ under i.i.d. sampling.\n\nSTRENGTHS:\nThis paper tries to analyze the sample complexity of the AC method under more practical Markovian sampling, where the transition tuples are generated from a single trajectory. The theoretical problem is well-motivated.\n\nThe paper proposes a new analysis framework for the algorithm and establishes the sample complexity of  $O(\\epsilon^{-2})$ which matches the best existing sample complexity for the single-timescale AC algorithm, and a sample complexity of  $\\tilde{O}(\\epsilon^{-2})$ without i.i.d. assumption.\n\nWEAKNESSES:\nAfter removing the assumption of independence and identically distributed (i.i.d.) data, the author introduces a new assumption, namely that the Markov chain is geometrically mixing and the rollout state distribution rapidly approximates the stationary distribution (assumption 3.2). I am curious whether this assumption, although slightly weaker than the i.i.d. assumption, can still be considered strong, as it may not be commonly applicable in practical scenarios. Furthermore, this assumption appears to conflict with the statement \"the transition tuples are generated from a single trajectory.\"\n\nIn the final results, the author requires a condition of $T>2\\tau$ to ensure that the obtained samples are sufficiently close to samples from an i.i.d. stationary distribution.\n\nNumerous studies have already examined actor-critic methods and achieved the same sample complexity of $O(\\epsilon^{-2})$. The author claimed that their proposed methods can be utilized without relying on the i.i.d. assumption and achieve a slightly inferior complexity of $\\tilde{O}(\\epsilon^{-2})$ under alternative assumptions. It is hard to judge the contribution in this situation since it is natural to obtain a worse result under milder assumptions.\n\nQUESTIONS:\nThe author asserts that their methods are capable of addressing continuous state situations, even when the state space is infinite. However, it is unclear whether the analyses can also be extended to infinite action spaces. Could the author provide some clarification and analyses on this aspect? \n\nSince the paper obtain the same results of  $O(\\epsilon^{-2})$  with previous works, could the author discuss and compare their results more specifically? (e.g., the constant comparison or simple control experiments)\n\nLIMITATIONS:\nThe paper still needs many strong assumptions to analyze the problem and it is very understandable due to the theoretical nature of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper studies the actor-critic algorithm with linear function approximation. The authors provide a single-time-scale analysis for the AC and achieve $\epsilon^{-2}$ sample complexity.

STRENGTHS:
- The paper is overall well written and easy to follow
- The single-time-scale analysis with markovian noise for actor-critic is new in the literature, to my best knowledge.
- The policy gradient norm error analysis is interesting

WEAKNESSES:
- I'm confused about Assumption 3.4, I wonder how $L_\mu$ contributes to the final results.
- Assumption 3.1, though the authors mentioned a few paper using the same assumption, sounds pretty strong for me. It basically requires even the optimal policy should visit all state-action pairs, which is quite uncommon in practice.

QUESTIONS:
My concerns are listed in the weakness part.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a finite-time analysis of single-timescale, single-sample, average-reward actor-critic with a linear critic under Markovian sampling. It is established that the standard scheme achieves $\\epsilon$-approximate stationarity with $\\widetilde{O}(\\epsilon^{-2})$ sample complexity. The analysis is achieved by extending the small-gain analysis of [A. Oleshevsky & B. Gharesifard, _A small-gain analysis of single-timescale actor-critic_. SICON 2023], which was for the discounted setting, to the average reward setting. This is accomplished by leveraging a uniform ergodicity assumption (Assumption 3.2, line 202) and associated analysis (like, e.g., that of [Y. F. Wu, W. Zhang, P. Xu, Q. Gu, _A finite-time analysis of two time-scale actor-critic methods_. NeurIPS, 2020]) to handle the Markovian sampling that becomes particularly important in the average-reward setting. It is furthermore claimed that the results apply to problems with continuous state spaces. The sample complexity achieved matches the state of the art for finite-time analyses of actor-critic methods, yet applies to the single-timescale, single-sample, average-reward, Markovian sampling setting, which was previously an open question.\n\nSTRENGTHS:\nAs stated in the summary, this work establishes that single-timescale, single-sample, average-reward actor-critic under Markovian sampling achieves $\\epsilon$-stationarity with $\\widetilde{O}(\\epsilon^{-2})$ sample complexity. This shows that this version of actor-critic matches the state of the art, resolving what was a previously an open question. Despite the analysis presented in the appendix being notation-heavy and somewhat difficult to follow, and though there are issues in the exposition and assumptions that need to be addressed (see weaknesses and questions below), the main steps in the theoretical results appear to be sound. The analysis appears to build off of the small-gain analysis of [Oleshevsky & Gharesifard, 2023], combined with elements of previous two-timescale analyses for average-reward actor-critic with the uniform ergodicity assumption such as [Wu et al., 2020], as well as some algebraic manipulations of inequalities. Though of uncertain practical utility, this contribution is definitely of interest to the theoretical reinforcement learning community.\n\nWEAKNESSES:\nI have some concerns about: (i) clarity about the innovation required in the analysis; (ii) the satisfiability of the assumptions made; and (iii) the dependence of the main result presented in Step 4 in lines 331-339 on having access to potentially unknown problem-specific constants when designing stepsizes. Addressing these issues or being clear about any limitations will likely strengthen the paper.\n\n(i) As stated above, the analysis appears to build off of [Oleshevsky & Gharesifard, 2023] to handle the interconnectedness of the single-timescale analysis, elements of two-timescale analyses like [Wu et al., 2020] to handle Markovian sampling, as well as algebraic manipulations of inequalities to combine it all together. However, it is unclear from the main body whether the analysis is a straightforward (albeit complicated) combination of these existing methods, or if some critical new insight and innovation are required. If the former, the significance of the contribution is weakened. If the latter, it should be stated more clearly. Also, the abstract claims that the analysis applies to continuous state spaces, and on lines 73-74 it is stated that achieving this requires \"significantly non-trivial effort in the analysis\". It is not clear where in the analysis this non-trivial effort takes place, however. If serious additional effort was required, it should be clearly stated exactly where it occurs.\n\n(ii) Two of the assumptions call into doubt the applicability of the result to continuous state spaces. On lines 198-199, it is stated that Assumption 3.1 holds under certain conditions in the tabular case, but applicability to the continuous state space case is not mentioned. The reader wonders: does Assumption 3.1 hold when $|S| = \\infty$? In addition, lines 220-221 state that Assumption 3.4 holds for the finite state-action space setting. Again: what about when $|S| = \\infty$? Since the analysis relies on these assumptions, it must be shown that they hold in the continuous state space setting for the corresponding claims in the abstract and introduction to be true. Finally, though uniform ergodicity (Assumption 3.2) is commonly assumed in average-reward analyses, the recent two-timescale analysis of [W. A. Suttle, A. S. Bedi, B. Patel, B. M. Sadler, A. Koppel, D. Manocha, _Beyond exponentially fast mixing in average-reward reinforcement learning via multi-level monte carlo actor-critic_. ICML 2023] removes the need for this assumption. It would be helpful to see a discussion of why Assumption 3.2 is still required in the single-timescale analysis or how it might be eliminated.\n\n(iii) On lines 334-337, it is pointed out that, for Step 4 (cf. lines 331-339) to work, a certain set of inequalities (line 335) needs to be satisfied. It is stated that this condition can be satisfied by choosing the stepsize ratio $c$ to be smaller than some threshold. On line 556 in the appendix, this threshold is given, but it involves the problem-specific constants $\\lambda, L_*, G, B$. It thus appears necessary that we have oracle knowledge or that an additional estimation procedure is required for the specific stepsize scheme to be operable.\n\nQUESTIONS:\n* What is the key innovation required in the analysis?\n* What is the effort required in accommodating continuous state spaces?\n* Under what conditions do Assumptions 3.1 and 3.4 hold for continuous state spaces?\n* Why is Assumption 3.2 necessary? Or can it be eliminated?\n* How do we choose the stepsize ratio $c$ in practice?\n\nLIMITATIONS:\nAside from the weaknesses and questions described above, the authors have adequately addressed the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
This works provides finite sample analysis for single timescale actor critic.

STRENGTHS:
A finite sample analysis for single timescale actor critic under Markovian noise and infinite state space is definitely a notable contribution to the community.

WEAKNESSES:
My biggest concern is the correctness of this work. This work is a resubmission from ICML 2023. In the ICML round, all reviewers agree that this is a good paper until one expert reviewer that seems to really be in the field pointed out one critical technical error in the proof, leading to the rejection. But I unfortunately do not have access to the ICML reviews now so I will recommend "rejection" for now. If the authors provide the following information in the rebuttal, I am more than happy to evaluate the work again.

1. What errors did the reviewer pointed out in the ICML round (it would help a lot if a copy of the original review can be provided, assuming the authors still have access to it)?
2. Was the reviewer wrong or the authors were wrong?
3. If the authors were wrong, how is the bug fixed in this version? If the reviewer was wrong, what mistake did the reviewer make?


It would help a lot if the authors could answer the above questions in details and in a self-contained way.

QUESTIONS:
See above

LIMITATIONS:
See above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-18