Year: 2020

Abstract:
Asynchronous and parallel implementation of standard reinforcement learning (RL) algorithms is a key enabler of the tremendous success of modern RL. Among many asynchronous RL algorithms, arguably the most popular and effective one is the asynchronous advantage actor-critic (A3C) algorithm. Although A3C is becoming the workhorse of RL, its theoretical properties are still not well-understood, including the non-asymptotic analysis and the performance gain of parallelism (a.k.a. speedup). This paper revisits the A3C algorithm with TD(0) for the critic update, termed A3C-TD(0), with provable convergence guarantees. With linear value function approximation for the TD update, the convergence of A3C-TD(0) is established under both i.i.d. and Markovian sampling. Under i.i.d. sampling, A3CTD(0) obtains sample complexity ofO( −2.5/N) per worker to achieve accuracy, where N is the number of workers. Compared to the best-known sample complexity of O( −2.5) for two-timescale AC, A3C-TD(0) achieves linear speedup, which justifies the advantage of parallelism and asynchrony in AC algorithms theoretically for the first time. Numerical tests on synthetically generated instances and OpenAI Gym environments have been provided to verify our theoretical analysis.

Asynchronous and parallel implementation of standard reinforcement learning (RL) algorithms is a key enabler of the tremendous success of modern RL. Among many asynchronous RL algorithms, arguably the most popular and effective one is the asynchronous advantage actor-critic (A3C) algorithm. Although A3C is becoming the workhorse of RL, its theoretical properties are still not well-understood, including the non-asymptotic analysis and the performance gain of parallelism (a.k.a. speedup). This paper revisits the A3C algorithm with TD(0) for the critic update, termed A3C-TD(0), with provable convergence guarantees. With linear value function approximation for the TD update, the convergence of A3C-TD(0) is established under both i.i.d. and Markovian sampling. Under i.i.d. sampling, A3CTD(0) obtains sample complexity ofO( −2.5/N) per worker to achieve accuracy, where N is the number of workers. Compared to the best-known sample complexity of O( −2.5) for two-timescale AC, A3C-TD(0) achieves linear speedup, which justifies the advantage of parallelism and asynchrony in AC algorithms theoretically for the first time. Numerical tests on synthetically generated instances and OpenAI Gym environments have been provided to verify our theoretical analysis.
1 INTRODUCTION
Reinforcement learning (RL) has achieved impressive performance in many domains such as robotics [1, 2] and video games [3]. However, these empirical successes are often at the expense of significant computation. To unlock high computation capabilities, the state-of-the-art RL approaches rely on sampling data from massive parallel simulators on multiple machines [3, 4, 5]. Empirically, these approaches can stabilize the learning processes and reduce training time when they are implemented in an asynchronous manner. One popular RL method that often achieves the best empirical performance is the asynchronous variant of the actor-critic (AC) algorithm, which is referred to as A3C [3].
A3C builds on the original AC algorithm [6]. At a high level, AC simultaneously performs policy optimization (a.k.a. the actor step) using the policy gradient method [7] and policy evaluation (a.k.a. the critic step) using the temporal difference learning (TD) algorithm [8]. To ensure scalability, both actor and critic steps can combine with various function approximation techniques. To ensure stability, AC is often implemented in a two time-scale fashion, where the actor step runs in the slow timescale and the critic step runs in the fast timescale. Similar to other on-policy RL algorithms, AC uses samples generated from the target policy. Thus, data sampling is entangled with the learning procedure, which generates significant overhead. To speed up the sampling process of AC, A3C introduces multiple workers with a shared policy, and each learner has its own simulator to perform data sampling. The shared policy can be then updated using samples collected from multiple learners.
Despite the tremendous empirical success achieved by A3C, to the best of our knowledge, its theoretical property is not well-understood. The following theoretical questions remain unclear: Q1) Under what assumption does A3C converge? Q2) What is its convergence rate? Q3) Can A3C obtain benefit (or speedup) using parallelism and asynchrony?
For Q3), we are interested in the training time linear speedup with N workers, which is the ratio between the training time using a single worker and that using N workers. Since asynchronous parallelism mitigates the effect of stragglers and keeps all workers busy, the training time speedup
can be measured roughly by the sample (i.e., computational) complexity linear speedup [9], given by
Speedup(N) = sample complexity when using one worker
average sample complexity per worker when using N workers . (1)
If Speedup(N) = Θ(N), the speedup is linear, and the training time roughly reduces linearly as the number of workers increases. This paper aims to answer these questions, towards the goal of providing theoretical justification for the empirical successes of parallel and asynchronous RL.
1.1 RELATED WORKS
Analysis of actor critic algorithms. AC method was first proposed by [6, 10], with asymptotic convergence guarantees provided in [6, 10, 11]. It was not until recently that the non-asymptotic analyses of AC have been established. The finite-sample guarantee for the batch AC algorithm has been established in [12, 13] with i.i.d. sampling. Later, in [14], the finite-sample analysis was established for the double-loop nested AC algorithm under the Markovian setting. An improved analysis for the Markovian setting with minibatch updates has been presented in [15] for the nested AC method. More recently, [16, 17] have provided the first finite-time analyses for the two-timescale AC algorithms under Markov sampling, with both Õ( −2.5) sample complexity, which is the bestknown sample complexity for two-timescale AC. Through the lens of bi-level optimization, [18] has also provided finite-sample guarantees for this two-timescale Markov sampling setting, with global optimality guarantees when a natural policy gradient step is used in the actor. However, none of the existing works has analyzed the effect of the asynchronous and parallel updates in AC.
Empirical parallel and distributed AC. In [3], the original A3C method was proposed and became the workhorse in empirical RL. Later, [19] has provided a GPU-version of A3C which significantly decreases training time. Recently, the A3C algorithm is further optimized in modern computers by [20], where a large batch variant of A3C with improved efficiency is also proposed. In [21], an importance weighted distributed AC algorithm IMPALA has been developed to solve a collection of problems with one single set of parameters. Recently, a gossip-based distributed yet synchronous AC algorithm has been proposed in [5], which has achieved the performance competitive to A3C.
Asynchronous stochastic optimization. For solving general optimization problems, asynchronous stochastic methods have received much attention recently. The study of asynchronous stochastic methods can be traced back to 1980s [22]. With the batch size M , [23] analyzed asynchronous SGD (async-SGD) for convex functions, and derived a convergence rate of O(K− 12M− 12 ) if delay K0 is bounded by O(K 1 4M− 3 4 ). This result implies linear speedup. [24] extended the analysis of [23] to smooth convex with nonsmooth regularization and derived a similar rate. Recent studies by [25] improved upper bound of K0 to O(K 1 2M− 1 2 ). However, all these works have focused on the single-timescale SGD with a single variable, which cannot capture the stochastic recursion of the AC and A3C algorithms. To best of our knowledge, non-asymptotic analysis of asynchronous two-timescale SGD has remained unaddressed, and its speedup analysis is even an uncharted territory.
1.2 THIS WORK
In this context, we revisit A3C with TD(0) for the critic update, termed A3C-TD(0). The hope is to provide non-asymptotic guarantee and linear speedup justification for this popular algorithm.
Our contributions. Compared to the existing literature on both the AC algorithms and the asyncSGD, our contributions can be summarized as follows.
c1) We revisit two-timescale A3C-TD(0) and establish its convergence rates with both i.i.d. and Markovian sampling. To the best of our knowledge, this is the first non-asymptotic convergence result for asynchronous parallel AC algorithms.
c2) We characterize the sample complexity of A3C-TD(0). In i.i.d. setting, A3C-TD(0) achieves a sample complexity of O( −2.5/N) per worker, where N is the number of workers. Compared to the best-known complexity of O( −2.5) for i.i.d. two-timescale AC [18], A3C-TD(0) achieves linear speedup, thanks to the parallelism and asynchrony. In the Markovian setting, if delay is bounded, the sample complexity of A3C-TD(0) matches the order of the non-parallel AC algorithm [17].
c3) We test A3C-TD(0) on the synthetically generated environment to verify our theoretical guarantees with both i.i.d. and Markovian sampling. We also test A3C-TD(0) on the classic control tasks and Atari Games from OpenAI Gym. Code is available in the supplementary material.
Technical challenges. Compared to the recent convergence analysis of nonparallel two-timescale AC in [16, 17, 18], several new challenges arise due to the parallelism and asynchrony.
Markovian noise coupled with asynchrony and delay. The analysis of two-timescale AC algorithm is non-trivial because of the Markovian noise coupled with both the actor and critic steps. Different from the nonparallel AC that only involves a single Markov chain, asynchronous parallel AC introduces multiple Markov chains (one per worker) that mix at different speed. This is because at a given iteration, workers collect different number of samples and thus their chains mix to different degrees. As we will show later, the worker with the slowest mixing chain will determine the convergence.
Linear speedup for SGD with two coupled sequences. Parallel async-SGD has been shown to achieve linear speedup recently [9, 26]. Different from async-SGD, asynchronous AC is a two-timescale stochastic semi-gradient algorithm for solving the more challenging bilevel optimization problem (see [18]). The errors induced by asynchrony and delay are intertwined with both actor and critic updates via a nested structure, which makes the sharp analysis more challenging. Our linear speedup analysis should be also distinguished from that of mini-batch async-SGD [27], where the speedup is a result of variance reduction thanks to the larger batch size generated by parallel workers.
2 PRELIMINARIES
2.1 MARKOV DECISION PROCESS AND POLICY GRADIENT THEOREM
RL problems are often modeled as an MDP described byM = {S,A,P, r, γ}, where S is the state space, A is the action space, P(s′|s, a) is the probability of transitioning to s′ ∈ S given current state s ∈ S and action a ∈ A, and r(s, a, s′) is the reward associated with the transition (s, a, s′), and γ ∈ [0, 1) is a discount factor. Throughout the paper, we assume the reward r is upper-bounded by a constant rmax. A policy π : S → ∆(A) is defined as a mapping from the state space S to the probability distribution over the action space A. Considering discrete time t in an infinite horizon, a policy π can generate a trajectory of state-action pairs (s0, a0, s1, a1, . . .) with at ∼ π(·|st) and st+1 ∼ P(·|st, at). Given a policy π, we define the state and state action value functions as
Vπ(s) := E [ ∞∑ t=0 γtr(st, at, st+1) | s0 = s ] , Qπ(s, a) := E [ ∞∑ t=0 γtr(st, at, st+1) | s0 = s, a0 = a ] (2)
where E is taken over the trajectory (s0, a0, s1, a1, . . .) generated under policy π. With the above definitions, the advantage function is Aπ(s, a) := Qπ(s, a) − Vπ(s). With η denoting the initial state distribution, the discounted state visitation measure induced by policy π is defined as dπ(s) := (1 − γ) ∑∞ t=0 γ
tP(st = s | s0 ∼ η, π), and the discounted state action visitation measure is d′π(s, a) = (1− γ) ∑∞ t=0 γ
tP(st = s | s0 ∼ η, π)π(a|s). The goal of RL is to find a policy that maximizes the expected accumulative reward J(π) := Es∼η[Vπ(s)]. When the state and action spaces are large, finding the optimal policy π becomes computationally intractable. To overcome the inherent difficulty of learning a function, the policy gradient methods search the best performing policy over a class of parameterized policies. We parameterize the policy with parameter θ ∈ Rd, and solve the optimization problem as
max θ∈Rd J(θ) with J(θ) := E s∼η [Vπθ (s)]. (3)
To maximize J(θ) with respect to θ, one can update θ using the policy gradient direction given by [7]
∇J(θ) = E s,a∼d′
θ
[Aπθ (s, a)ψθ(s, a)] , (4)
where ψθ(s, a) := ∇ log πθ(a|s), and d′θ := (1 − γ) ∑∞ t=0 γ
tP(st = s | s0, πθ)πθ(a|s). Since computing E in (4) is expensive if not impossible, popular policy gradient-based algorithms iteratively update θ using stochastic estimate of (4) such as REINFORCE [28] and G(PO)MDP [29].
2.2 ACTOR-CRITIC ALGORITHM WITH VALUE FUNCTION APPROXIMATION
Both REINFORCE and G(PO)MDP-based policy gradient algorithms rely on a Monte-Carlo estimate of the value function Vπθ (s) and thus ∇J(θ) by generating a trajectory per iteration. However, policy gradient methods based on Monte-Carlo estimate typically suffer from high variance and large sampling cost. An alternative way is to recursively refine the estimate of Vπθ (s). For a policy πθ, it is known that Vπθ (s) satisfies the Bellman equation [30], that is
Vπθ (s) = E a∼πθ(·|s), s′∼P(·|s,a)
[ r(s, a, s′) + γVπθ (s ′) ] , ∀s ∈ S. (5)
In practice, when the state space S is prohibitively large, one cannot afford the computational and memory complexity of computing Vπθ (s) and Aπθ (s, a). To overcome this curse-of-dimensionality, a popular method is to approximate the value function using function approximation techniques. Given the state feature mapping φ(·) : S −→ Rd′ for some d′ > 0, we approximate the value function linearly as Vπθ (s) ≈ V̂ω(s) := φ(s)>ω, where ω ∈ Rd ′ is the critic parameter.
Given a policy πθ, the task of finding the best ω such that Vπθ (s) ≈ V̂ω(s) is usually addressed by TD learning [8]. Defining the kth transition as xk := (sk, ak, sk+1) and the corresponding TD-error as δ̂(xk, ωk) := r(sk, ak, sk+1) + γφ(sk+1)>ωk − φ(sk)>ωk, the parameter ω is updated via
ωk+1 = ΠRω ( ωk + βkg(xk, ωk) ) with g(xk, ωk) := δ̂(xk, ωk)∇ωk V̂ωk(sk) (6)
where βk is the critic stepsize, and ΠRω is the projection with Rω being a pre-defined constant. The projection step is often used to control the norm of the gradient. In AC, it prevents the actor and critic updates from going a too large step in the ‘wrong’ direction; see e.g., [6, 16, 17].
Using the definition of advantage function Aπθ (s, a) = Es′∼P [r(s, a, s′) + γVπθ (s′)]− Vπθ (s), we can also rewrite (4) as ∇J(θ) = Es,a∼d′θ,s′∼P [(r(s, a, s ′) + γVπθ (s ′)− Vπθ (s))ψθ(s, a)]. Leveraging the value function approximation, we can then approximate the policy gradient as
∇̂J(θ) = ( r(s, a, s′) + γV̂ω(s ′)− V̂ω(s) ) ψθ(s, a) = δ̂(x, ω)ψθ(s, a) (7)
which gives rise to the policy update
θk+1 = θk + αkv(xk, θk, ωk) with v(xk, θk, ωk) := δ̂(xk, ωk)ψθk(sk, ak) (8)
where αk is the stepsize for the actor update.
To ensure convergence when simultaneously performing critic and actor updates, the stepsizes αk and βk often decay at two different rates, which is referred to the two-timescale AC [17, 18].
3 ASYNCHRONOUS ADVANTAGE ACTOR CRITIC WITH TD(0)
To speed up the training process, we implement AC over N workers in a shared memory setting without coordinating among workers — a setting similar to that in A3C [3]. Each worker has its own simulator to perform sampling, and then collaboratively updates the shared policy πθ using AC updates. As there is no synchronization after each update, the policy used by workers to generate samples may be outdated, which introduces staleness.
Notations on transition (s, a, s′). Since each worker will maintain a separate Markov chain, we thereafter use subscription t in (st, at, st+1) to indicate the tth transition on a Markov chain. We use k to denote the global counter (or iteration), which increases by one whenever a worker finishes the actor and critic updates in the shared memory. We use subscription (k) in (s(k), a(k), s′(k)) to indicate the transition used in the kth update. Specifically, we initialize θ0, ω0 in the shared memory. Each worker will initialize the simulator with initial state s0. Without coordination, workers will read θ, ω in the shared memory. From each worker’s view, it then generates sample (st, at, st+1) by either sampling st from µθ(·), where µθ(·) is the stationary distribution of an artificial MDP with transition probability measure P̃(·|st, at) := γP(·|st, at) + (1 − γ)η(·), or, sampling st from a Markov chain under policy πθ. In both cases, each worker obtains at ∼ πθ(·|st) and st+1 ∼ P̃(·|st, at). Sampling st+1 from P̃(·|st, at) can be achieved by sampling st+1 from η(·) with probability 1− γ and from P(·|st, at) otherwise. Once
Algorithm 1 Asynchronous advantage AC with TD(0): each worker’s view. 1: Global initialize: Global counter k = 0, initial θ0, ω0 in the shared memory. 2: Worker initialize: Local counter t = 0. Obtain initial state s0. 3: for t = 0, 1, 2, · · · do 4: Read θ, ω in the shared memory. 5: Option 1 (i.i.d. sampling): 6: Sample st ∼ µθ(·), at ∼ πθ(·|s), st+1 ∼ P̃(·|st, at). 7: Option 2 (Markovian sampling): 8: Sample at ∼ πθ(·|st), st+1 ∼ P̃(·|st, at). 9: Compute δ̂(xt, ω) = r(st, at, st+1) + γV̂ω(st+1)− V̂ω(st). 10: Compute g(xt, ω) = δ̂(xt, ω)∇ωV̂ω(st). 11: Compute v(xt, θ, ω) = δ̂(xt, ω)ψθ(st, at). 12: In the shared memory, perform update (9). 13: end for
obtaining xt := (st, at, st+1), each worker locally computes the policy gradient v(xt, θ, ω) and the TD(0) update g(xt, ω), and then updates the parameters in shared memory asynchronously by
ωk+1 = ΠRω ( ωk + βkg(x(k), ωk−τk) ) , (9a)
θk+1 = θk + αkv(x(k), θk−τk , ωk−τk), (9b)
where τk is the delay in the kth actor and critic updates. See the A3C with TD(0) in Algorithm 1.
Sampling distributions. Since the transition kernel required by the actor and critic updates are different in the discounted MDP, it is difficult to design a two-timescale AC algorithm. To address this issue, we adopt the sampling method introduced in the seminal work [6, 31] and the recent work [15, 16], which inevitably introduces bias by sampling from the artificial transition P̃ instead of P . However, as we will mention later, this extra bias is small when the discount factor γ is close to 1.
Parallel sampling. The AC update (6) and (8) uses samples generated “on-the-fly” from the target policy πθ, which brings overhead. Compared with (6) and (8), the A3C-TD(0) update (9) allows parallel sampling from N workers, which is the key to linear speedup. We consider the case where only one worker can update parameters in the shared memory at the same time and the update cannot be interrupted. In practice, (9) can also be performed in a mini-batch fashion.
Minor differences from A3C [3]. The A3C-TD(0) algorithm resembles the popular A3C method [3]. With nmax denoting the horizon of steps, for n ∈ {1, ..., nmax}, A3C iteratively uses n-step TD errors to compute actor and critic gradients. In A3C-TD(0), we use the TD(0) method which is the 1-step TD method for actor and critic update. When nmax = 1, A3C method reduces to A3C-TD(0). The n-step TD method is a hybrid version of the TD(0) method and the Monte-Carlo method. The A3C method with Monte-Carlo sampling is essentially the delayed policy gradient method, and thus its convergence follows directly from the delayed SGD. Therefore, we believe that the convergence of the A3C method based on TD(0) in this paper can be easily extended to the convergence of the A3C method with n-step TD. We here focus on A3C with TD(0) just for ease of exposition.
4 CONVERGENCE ANALYSIS OF TWO-TIMESCALE A3C-TD(0)
In this section, we analyze the convergence of A3C-TD(0) in both i.i.d. and Markovian settings. Throughout this section, the notation O(·) contains constants that are independent of N and K0. To analyze the performance of A3C-TD(0), we make the following assumptions. Assumption 1. There exists K0 such that the delay at each iteration is bounded by τk ≤ K0,∀k.
Assumption 1 ensures the viability of analyzing the asynchronous update; see the same assumption in e.g., [5, 25]. In practice, the delay usually scales as the number of workers, that is K0 = Θ(N).
With P̃πθ (s′|s) = ∑ a∈A P̃(s′|s, a)πθ(a|s), we define that:
Aθ,φ := E s∼µθ,s′∼P̃πθ [φ(s)(γφ(s′)− φ(s))>], bθ,φ := E s∼µθ,a∼πθ,s′∼P̃ [r(s, a, s′)φ(s)]. (10)
It is known that for a given θ, the stationary point ω∗θ of the TD(0) update in Algorithm 1 satisfies
Aθ,φω ∗ θ + bθ,φ = 0. (11)
Assumption 2. For all s ∈ S, the feature vector φ(s) is normalized so that ‖φ(s)‖2 ≤ 1. For all γ ∈ [0, 1] and θ ∈ Rd, Aθ,φ is negative definite and its max eigenvalue is upper bounded by −λ.
Assumption 2 is common in analyzing TD with linear function approximation; see e.g., [17, 32, 33]. With this assumption, Aθ,φ is invertible, so we have ω∗θ = −A −1 θ,φbθ,φ. Define Rω := rmax/λ, then we have ‖ω∗θ‖2 ≤ Rω . It justifies the projection introduced in Algorithm 1. In practice, the projection radius Rω can be estimated online by methods proposed in [32, Section 8.2] or [34, Lemma 1]. Assumption 3. For any θ, θ′ ∈ Rd, s ∈ S and a ∈ A, there exist constants such that: i) ‖ψθ(s, a)‖2 ≤ Cψ; ii) ‖ψθ(s, a)− ψθ′(s, a)‖2 ≤ Lψ‖θ − θ′‖2; iii) |πθ(a|s)− πθ′(a|s)| ≤ Lπ‖θ − θ′‖2.
Assumption 3 is common in analyzing policy gradient-type algorithms which has also been made by e.g., [34, 35, 36]. This assumption holds for many policy parameterization methods such as tabular softmax policy [36], Gaussian policy [37] and Boltzmann policy [31]. Assumption 4. For any θ ∈ Rd, the Markov chain under policy πθ and transition kernel P(·|s, a) or P̃(·|s, a) is irreducible and aperiodic. Then there exist constants κ > 0 and ρ ∈ (0, 1) such that
sup s∈S
dTV (P(st ∈ ·|s0 = s, πθ), µθ) ≤ κρt, ∀t (12)
where µθ is the stationary state distribution under πθ, and st is the state of Markov chain at time t.
Assumption 4 assumes the Markov chain mixes at a geometric rate; see also [32, 33]. The stationary distribution µθ of an artificial Markov chain with transition P̃ is the same as the discounted visitation measure dθ of the Markov chain with transition P [6]. This means that if we sample according to at ∼ πθ(·|st), st+1 ∼ P̃(·|st, at), the marginal distribution of (st, at) will converge to the discounted state-action visitation measure d′θ(s, a), which allows us to control the gradient bias.
4.1 LINEAR SPEEDUP RESULT WITH I.I.D. SAMPLING
In this section, we consider A3C-TD(0) under the i.i.d. sampling setting, which is widely used for analyzing RL algorithms; see e.g., [13, 18, 38].
We first give the convergence result of critic update as follows. Theorem 1 (Critic convergence). Suppose Assumptions 1–4 hold. Consider Algorithm 1 with i.i.d. sampling and V̂ω(s) = φ(s)>ω. Select step size αk = c1(1+k)σ1 , βk = c2 (1+k)σ2 , where 0 < σ2 < σ1 < 1 and c1, c2 are positive constants. Then it holds that
1
K K∑ k=1 E ∥∥ωk − ω∗θk∥∥22=O( 1K1−σ2 ) +O ( 1 K2(σ1−σ2) ) +O ( K20 K2σ2 ) +O ( K0 Kσ1 ) +O ( 1 Kσ2 ) . (13)
Different from async-SGD (e.g., [9]), the optimal critic parameter ω∗θ is constantly drifting as θ changes at each iteration. This necessitates setting σ1 > σ2 to make the policy change slower than the critic, which can be observed in the second term in (13). If σ1 > σ2, then the policy is static relative to the critic in an asymptotic sense.
To introduce the convergence of actor update, we first define the critic approximation error as
app := max θ∈Rd
√ E
s∼µθ |Vπθ (s)− V̂ω∗θ (s)| 2 ≤ fa + sp, (14)
where µθ is the stationary distribution under πθ and P̃ . The error app captures the quality of the critic approximation under Algorithm 1. It can be further decomposed into the function approximation error fa, which is common in analyzing AC with function approximation [14, 15, 17], and the sampling error sp = O(1− γ), which is unique in analyzing two-timescale AC for a discounted MDP. The error app is small when the value function approximation is accurate and the discounting factor γ is close to 1; see the detailed derivations in Lemma 7 of supplementary material. Now we are ready to give the actor convergence.
Theorem 2 (Actor convergence). Under the same assumptions of Theorem 1, select step size αk = c1
(1+k)σ1 , βk = c2 (1+k)σ2 , where 0 < σ2 < σ1 < 1 and c1, c2 are positive constants. Then it holds that
1
K K∑ k=1 E ‖∇J(θk)‖22 = O ( 1 K1−σ1 ) +O ( K0 Kσ1 ) +O ( K20 K2σ2 ) +O ( 1 K K∑ k=1 E ‖ωk − ω∗θk‖ 2 2 ) +O( app).
(15)
Different from the analysis of async-SGD, in actor update, the stochastic gradient v(x, θ, ω) is biased because of inexact value function approximation. The bias introduced by the critic optimality gap and the function approximation error correspond to the last two terms in (15).
In Theorem 1 and Theorem 2, optimizing σ1 and σ2 gives the following convergence rate. Corollary 1 (Linear speedup). Given Theorem 1 and Theorem 2, select σ1 = 35 and σ2 = 2 5 . If we further assume K0 = O(K 1 5 ), then it holds that
1
K K∑ k=1 E ‖∇J(θk)‖22 = O ( K− 2 5 ) +O( app) (16)
where O(·) contains constants that are independent of N and K0.
By setting the first term in (16) to , we get the total iteration complexity to reach -accuracy is O( −2.5). Since each iteration only uses one sample (one transition), it also implies a total sample complexity of O( −2.5). Then the average sample complexity per worker is O( −2.5/N) which indicates linear speedup in (1). Intuitively, the negative effect of parameter staleness introduced by parallel asynchrony vanishes asymptotically, which implies linear speedup in terms of convergence.
4.2 CONVERGENCE RESULT WITH MARKOVIAN SAMPLING
Theorem 3 (Critic convergence). Suppose Assumptions 1–4 hold. Consider Algorithm 1 with Markovian sampling and V̂ω(s) = φ(s)>ω. Select step size αk = c1(1+k)σ1 , βk = c2 (1+k)σ2 , where 0 < σ2 < σ1 < 1 and c1, c2 are positive constants. Then it holds that
1
K K∑ k=1 E ∥∥ωk − ω∗θk∥∥22 = O( 1K1−σ2 ) +O ( 1 K2(σ1−σ2) ) +O ( K20 K2σ2 ) +O ( K20 log 2K Kσ1 ) +O ( K0 logK Kσ2 ) .
(17)
The following theorem gives the convergence rate of actor update in Algorithm 1. Theorem 4 (Actor convergence). Under the same assumptions of Theorem 3, select step size αk =
c1 (1+k)σ1 , βk = c2 (1+k)σ2 , where 0 < σ2 < σ1 < 1 and c1, c2 are positive constants. Then it holds that
1
K K∑ k=1 E ‖∇J(θk)‖22 =O ( 1 K1−σ1 ) +O ( K20 log 2K Kσ1 ) +O ( K20 K2σ2 ) +O ( 1 K K∑ k=1 E ‖ωk − ω∗θk‖ 2 2 ) +O( app). (18) Assume K0 = O(K 1 5 ). Given Theorem 3, select σ1 = 35 and σ2 = 2 5 , then it holds that
1
K K∑ k=1 E ‖∇J(θk)‖22 = Õ ( K0K − 2 5 ) +O( app), (19)
where Õ(·) hides constants and the logarithmic order of K.
With Markovian sampling, the stochastic gradients g(x, ω) and v(x, θ, ω) are biased, and the bias decreases as the Markov chain mixes. The mixing time corresponds to the logarithmic term logK in (17) and (18). Because of asynchrony, at a given iteration, workers collect different number of samples and their chains mix to different degrees. The worker with the slowest mixing chain will determine the rate of convergence. The product of K0 and logK in (17) and (18) appears due to the slowest mixing chain. As the last term in (17) dominates other terms asymptotically, the convergence rate reduces as the number of workers increases. While the theoretical linear speedup is difficult to establish in the Markovian setting, we will empirically demonstrate it in Section 5.2.
5 NUMERICAL EXPERIMENTS
We test the speedup performance of A3C-TD(0) on both synthetically generated and OpenAI Gym environments. The settings, parameters, and codes are provided in supplementary material.
5.1 A3C-TD(0) IN SYNTHETIC ENVIRONMENT
To verify the theoretical result, we tested A3C-TD(0) with linear value function approximation in a synthetic environment. We use tabular softmax policy parameterization [36], which satisfies Assumption 3. The MDP has a state space |S| = 100, an discrete action space of |A| = 5. Each state feature has a dimension of 10. Elements of the transition matrix, the reward and the state features are randomly sampled from a uniform distribution over (0, 1). We evaluate the convergence of actor and critic respectively with the running average of test reward and critic optimality gap ‖ωk − ω∗θk‖2. Figures 1 and 2 show the training time and sample complexity of running A3C-TD(0) with i.i.d. sampling and Markovian sampling respectively. The speedup plot is measured by the number of samples needed to achieve a target running average reward under different number of workers. All the results are average over 10 Monte-Carlo runs. Figure 1 shows that the sample complexity of A3C-TD(0) stays about the same with different number of workers under i.i.d. sampling. Also, it can be observed from the speedup plot of Figure 1 that the A3C-TD(0) achieves roughly linear speedup with i.i.d. sampling, which is consistent with Corollary 1. The speedup of A3C-TD(0) with Markovian sampling shown in Figure 2 is roughly linear when number of workers is small.
5.2 A3C-TD(0) IN OPENAI GYM ENVIRONMENTS
We have also tested A3C-TD(0) with neural network parametrization in the classic control (Carpole) environment and the Atari game (Seaquest and Beamrider) environments. In Figures 3-5, each curve is generated by averaging over 5 Monte-Carlo runs with 95% confidence interval. Figures 3–5 show the speedup of A3C-TD(0) under different number of workers, where the average reward is computed by taking the running average of test rewards. The speedup and runtime speedup plots are respectively measured by the number of samples and training time needed to achieve a target running average reward under different number of workers. Although not justified theoretically, Figures 3–5 suggest that the sample complexity speedup is roughly linear, and the runtime speedup slightly degrades when the number of workers increases. This is partially due to our hardware limit. Similar observation has also been obtained in async-SGD [9].
6 CONCLUSIONS
This paper revisits the A3C algorithm with TD(0) for the critic update, termed A3C-TD(0). With linear value function approximation, the convergence of the A3C-TD(0) algorithm has been established under both i.i.d. and Markovian sampling settings. Under i.i.d. sampling, A3C-TD(0) achieves linear speedup compared to the best-known sample complexity of two-timescale AC, theoretically justifying the benefit of parallelism and asynchrony for the first time. Under Markov sampling, such a linear speedup can be observed in most classic benchmark tasks.
REFERENCES
[1] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in Proc. of International Conference on Learning Representations, 2016.
[2] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, p. 529, 2015.
[3] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning.” in Proc. of International Conference on Machine Learning, 2016.
[4] A. Nair, P. Srinivasan, S. Blackwell, C. Alcicek, R. Fearon, A. De Maria, V. Panneershelvam, M. Suleyman, C. Beattie, S. Petersen et al., “Massively parallel methods for deep reinforcement learning,” arXiv preprint:1507.04296, 2015.
[5] M. Assran, J. Romoff, N. Ballas, J. Pineau, and M. Rabbat, “Gossip-based actor-learner architectures for deep reinforcement learning,” in Proc. of Advances in Neural Information Processing Systems, 2019.
[6] V. Konda, Actor-critic algorithms. PhD thesis, Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, 2002.
[7] R. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation.” in Proc. of Advances in Neural Information Processing Systems, 2000.
[8] R. Sutton, “Learning to predict by the methods of temporal differences,” Machine Learning, vol. 3, pp. 9–44, 1988.
[9] X. Lian, H. Zhang, C. Hsieh, Y. Yijun Huang, and J. Liu, “A comprehensive linear speedup analysis for asynchronous stochastic parallel optimization from zeroth-order to first-order,” in Proc. of the Advances in Neural Information Processing Systems, 2016.
[10] V. Borkar and V. Konda, “The actor-critic algorithm as multi-time-scale stochastic approximation,” Sadhana, vol. 22, no. 4, pp. 525–543, 1997.
[11] S. Bhatnagar, R. Sutton, M. Ghavamzadeh, and M. Lee, “Natural actor critic algorithms,” Automatica, vol. 45, pp. 2471–2482, 2009.
[12] Z. Yang, K. Zhang, M. Hong, and T. Başar, “A finite sample analysis of the actor-critic algorithm,” in Proc. of IEEE Conference on Decision and Control, 2018, pp. 2759–2764.
[13] H. Kumar, A. Koppel, and A. Ribeiro, “On the sample complexity of actor-critic method for reinforcement learning with function approximation,” arXiv preprint:1910.08412, 2019.
[14] S. Qiu, Z. Yang, J. Ye, and Z. Wang, “On the finite-time convergence of actor-critic algorithm,” in Optimization Foundations for Reinforcement Learning Workshop at Advances in Neural Information Processing Systems, 2019.
[15] T. Xu, Z. Wang, and Y. Liang, “Improving sample complexity bounds for (natural) actor-critic algorithms,” in Proc. of Advances in Neural Information Processing Systems, 2020.
[16] ——, “Non-asymptotic convergence analysis of two time-scale (natural) actor-critic algorithms,” arXiv preprint:2005.03557, 2020.
[17] Y. Wu, W. Zhang, P. Xu, and Q. Gu, “A finite time analysis of two time-scale actor critic methods,” in Proc. of Advances in Neural Information Processing Systems, 2020.
[18] M. Hong, H.-T. Wai, Z. Wang, and Z. Yang, “A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic,” arXiv preprint:2007.05170, 2020.
[19] M. Babaeizadeh, I. Frosio, S. Tyree, J. Clemons, and J. Kautz, “Reinforcement learning through asynchronous advantage actor-critic on a gpu,” in Proc. of International Conference on Learning Representations, 2017.
[20] A. Stooke and P. Abbeel, “Accelerated methods for deep reinforcement learning,” arXiv preprint:1803.02811, 2019.
[21] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu, “Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures,” arXiv preprint:1802.01561, 2018.
[22] D. Bertsekas and J. Tsitsiklis, Parallel and distributed computation: numerical methods. Prentice-Hall, 1989.
[23] A. Agarwal and J. Duchi, “Distributed delayed stochastic optimization,” in Proc. of Advances in Neural Information Processing Systems, 2011.
[24] H. Feyzmahdavian, A. Aytekin, and M. Johansson, “An asynchronous mini-batch algorithm for regularized stochastic optimization,” arXiv preprint:1505.04824, 2015.
[25] X. Lian, Y. Huang, Y. Li, and J. Liu, “Asynchronous parallel stochastic gradient for nonconvex optimization.” in Proc. of Advances in Neural Information Processing Systems, 2015.
[26] T. Sun, R. Hannah, and W. Yin, “Asynchronous coordinate descent under more realistic assumptions,” in Proc. of Advances in Neural Information Processing Systems, 2017.
[27] X. Lian, C. Zhang, H. Zhang, C.-J. Hsieh, W. Zhang, and J. Liu, “Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent,” in Proc. of Advances in Neural Information Processing Systems, 2017.
[28] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine Learning, vol. 8, no. 3-4, pp. 229–256, May 1992.
[29] J. Baxter and P. L. Bartlett, “Infinite-horizon policy-gradient estimation,” J. Artificial Intelligence Res., vol. 15, pp. 319–350, 2001.
[30] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT Press, 2018.
[31] V. Konda and V. Borkar, “Actor-critic–type learning algorithms for markov decision processes,” SIAM Journal on Control and Optimization, vol. 38, no. 1, pp. 94–123, 1999.
[32] J. Bhandari, D. Russo, and R. Singal, “A finite time analysis of temporal difference learning with linear function approximation.” in Proc. of Conference on Learning Theory, 2018.
[33] T. Xu, Z. Wang, Y. Zhou, and Y. Liang, “Reanalysis of variance reduced temporal difference learning,” in Proc. of International Conference on Learning Representations, 2020.
[34] S. Zou, T. Xu, and Y. Liang, “Finite-sample analysis for SARSA with linear function approximation,” in Proc. of Advances in Neural Information Processing Systems, 2019.
[35] K. Zhang, A. Koppel, H. Zhu, and T. Başar, “Global convergence of policy gradient methods to (almost) locally optimal policies,” arXiv preprint:1906.08383, 2019.
[36] A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan, “Optimality and approximation with policy gradient methods in markov decision processes.” in Proc. of Thirty Third Conference on Learning Theory, 2020.
[37] K. Doya, “Reinforcement learning in continuous time and space,” Neural Computation, vol. 12, no. 1, pp. 219–245, 2000.
[38] R. Sutton, H. Maei, D. Precup, S. Bhatnagar, D. Silver, and E. Szepesvári, C.and Wiewiora, “Fast gradient-descent methods for temporal-difference learning with linear function approximation,” in Proc. of International Conference on Machine Learning, 2009.
[39] A. Y. Mitrophanov, “Sensitivity and convergence of uniformly ergodic markov chains,” Journal of Applied Probability, vol. 42, no. 4, pp. 1003–1014, 2005.
[40] Dgriff, “Pytorch implementation of a3c,” https://github.com/dgriff777/rl_a3c_pytorch, 2018.
Supplementary Material
A PRELIMINARY LEMMAS
A.1 GEOMETRIC MIXING
The operation p⊗ q denotes the tensor product between two distributions p(x) and q(y), i.e. (p⊗ q)(x, y) = p(x) · q(y). Lemma 1. Suppose Assumption 4 holds for a Markov chain generated by the rule at ∼ πθ(·|st), st+1 ∼ P̃(·|st, at). For any θ ∈ Rd, we have
sup s0∈S dTV
( P((st, at, st+1) ∈ ·|s0, πθ), µθ ⊗ πθ ⊗ P̃ ) ≤ κρt. (20)
where µθ(·) is the stationary distribution with policy πθ and transition kernel P̃(·|s, a).
Proof. We start with
sup s0∈S dTV
( P((st, at, st+1) = ·|s0, πθ), µθ ⊗ πθ ⊗ P̃ ) = sup s0∈S dTV ( P(st = ·|s0, πθ)⊗ πθ ⊗ P̃, µθ ⊗ πθ ⊗ P̃
) = sup s0∈S 1 2 ∫ s∈S ∑ a∈A ∫ s′∈S |P(st = ds|s0, πθ)πθ(a|s)P̃(ds′|s, a)− µθ(ds)πθ(a|s)P̃(ds′|s, a) ∣∣∣
= sup s0∈S
1
2 ∫ s∈S |P(st = ds|s0, πθ)− µθ(ds)| ∑ a∈A πθ(a|s) ∫ s′∈S P̃(ds′|s, a)
= sup s0∈S
dTV (P(st ∈ ·|s0, πθ), µθ)
≤ κρt, which completes the proof.
For the use in the later proof, given K > 0, we first define mK as: mK := min { m ∈ N+ |κρm−1 ≤ min{αk, βk} } , (21)
where κ and ρ are constants defined in (4). mK is the minimum number of samples needed for the Markov chain to approach the stationary distribution so that the bias incurred by the Markovian sampling is small enough.
A.2 AUXILIARY MARKOV CHAIN
The auxiliary Markov chain is a virtual Markov chain with no policy drifting — a technique developed in [34] to analyze stochastic approximation algorithms in non-stationary settings. Lemma 2. Under Assumption 1 and Assumption 3, consider the update (9) in Algorithm 1 with Markovian sampling. For a given number of samples m, consider the Markov chain of the worker that contributes to the kth update:
st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−1−−−−−−→ at−m+1 · · · st−1 θk−d1−−−−→ at−1 P̃−→ st θk−d0−−−−→ at P̃−→ st+1, where (st, at, st+1) = (s(k), a(k), s′(k)), and {dj} m j=0 is some increasing sequence with d0 := τk.
Given (st−m, at−m, st−m+1) and θk−dm , we construct its auxiliary Markov chain by repeatedly applying πθk−dm :
st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−−−−→ ãt−m+1 · · · s̃t−1 θk−dm−−−−→ ãt−1 P̃−→ s̃t θk−dm−−−−→ ãt P̃−→ s̃t+1.
Define xt := (st, at, st+1), then we have: dTV (P(xt ∈ ·|θk−dm , st−m+1),P(x̃t ∈ ·|θk−dm , st−m+1))
≤ 1 2 |A|Lπ dm∑ i=τk E [‖θk−i − θk−dm‖2|θk−dm , st−m+1] . (22)
Proof. Throughout the lemma, all expectations and probabilities are conditioned on θk−dm and st−m+1. We omit this condition for convenience.
First we have
dTV (P(st+1 ∈ ·),P(s̃t+1 ∈ ·))
= 1
2 ∫ s′∈S |P(st+1 = ds′)− P(s̃t+1 = ds′)|
= 1
2 ∫ s′∈S ∣∣∣∣∣ ∫ s∈S ∑ a∈A P(st = ds, at = a, st+1 = ds′)− P(s̃t = ds, ãt = a, s̃t+1 = ds′) ∣∣∣∣∣ ≤ 1
2 ∫ s′∈S ∫ s∈S ∑ a∈A |P(st = ds, at = a, st+1 = ds′)− P(s̃t = ds, ãt = a, s̃t+1 = ds′)|
= 1
2 ∫ s∈S ∑ a∈A ∫ s′∈S |P(st = ds, at = a, st+1 = ds′)− P(s̃t = ds, ãt = a, s̃t+1 = ds′)|
= dTV (P(xt ∈ ·),P(x̃t ∈ ·)) , (23)
where the last second equality is due to Tonelli’s theorem. Next we have
dTV (P(xt ∈ ·),P(x̃t ∈ ·))
= 1
2 ∫ s∈S ∑ a∈A ∫ s′∈S |P(st = ds, at = a, st+1 = ds′)− P(s̃t = ds, ãt = a, s̃t+1 = ds′)|
= 1
2 ∫ s∈S ∑ a∈A |P(st = ds, at = a)− P(s̃t = ds, ãt = a)| ∫ s′∈S P̃(st+1 = ds′|st = ds, at = a)
= 1
2 ∫ s∈S ∑ a∈A |P(st = ds, at = a)− P(s̃t = ds, ãt = a)|
= dTV (P ((st, at) ∈ ·) ,P ((s̃t, ãt) ∈ ·)) . (24)
Due to the fact that θk−τk is dependent on st, we need to write P(st, at) as P(st, at) = ∫ θk−τk∈Rd P(st, θk−τk , at)
= ∫ θ∈Rd P(st)P(θk−τk = dθ|st)πθk−τk (at|st)
= P(st) ∫ θ∈Rd P(θk−τk = dθ|st)πθk−τk (at|st) = P(st)E[πθk−τk (at|st)|st].
Then we have
dTV (P ((st, at) ∈ ·) ,P ((s̃t, ãt) ∈ ·))
= 1
2 ∫ s∈S ∑ a∈A ∣∣∣P(st = ds)E[πθk−τk (at = a|st = ds)|st = ds]− P(s̃t = ds)πθk−dm (ãt = a|s̃t = ds)∣∣∣ ≤ 1
2 ∫ s∈S ∑ a∈A ∣∣∣P(st = ds)E[πθk−τk (at = a|st = ds)|st = ds]− P(st = ds)πθk−dm (at = a|st = ds)∣∣∣ + 1
2 ∫ s∈S ∑ a∈A ∣∣P(st = ds)πθk−dm (ãt = a|s̃t = ds)− P(s̃t = ds)πθk−dm (ãt = a|s̃t = ds)∣∣ = 1
2 ∫ s∈S P(st = ds) ∑ a∈A ∣∣∣E[πθk−τk (at = a|st = ds)|st = ds]− πθk−dm (at = a|st = ds)∣∣∣ + 1
2 ∫ s∈S |P(st = ds)− P(s̃t = ds)| . (25)
Using Jensen’s inequality, we have
dTV (P ((st, at) ∈ ·) ,P ((s̃t, ãt) ∈ ·))
≤ 1 2 ∫ s∈S P(st = ds) ∑ a∈A E [∣∣∣πθk−τk (at = a|st = ds)− πθk−dm (at = a|st = ds)∣∣∣∣∣∣ st = ds]
+ 1
2 ∫ s∈S |P(st = ds)− P(s̃t = ds)|
≤ 1 2 ∫ s∈S P(st = ds) ∑ a∈A E [‖θk−τk − θk−dm‖2| st = ds] + 1 2 ∫ s∈S |P(st = ds)− P(s̃t = ds)|
= 1
2 |A|Lπ E ‖θk−τk − θk−dm‖2 + dTV (P(st ∈ ·),P(s̃t ∈ ·)) (26)
where the last inequality follows Assumption 3.
Now we start to prove (22).
dTV (P(xt ∈ ·),P(x̃t ∈ ·)) (24) = dTV (P((st, at) ∈ ·),P((s̃t, ãt) ∈ ·))
(25) ≤ dTV (P(st ∈ ·),P(s̃t ∈ ·)) + 1
2 |A|Lπ E ‖θk−τk − θk−dm‖2
(23) ≤ dTV (P(xt−1 ∈ ·),P(x̃t−1 ∈ ·)) + 1
2 |A|Lπ E ‖θk−τk − θk−dm‖2.
Now we have
dTV (P(xt ∈ ·),P(x̃t ∈ ·)) ≤ dTV (P(xt−1 ∈ ·),P(x̃t−1 ∈ ·)) + 1
2 |A|Lπ E ‖θk−τk − θk−dm‖2.
(27)
Since dTV (P(xt−m ∈ ·),P(xt−m ∈ ·)) = 0, recursively applying (27) for {t− 1, ..., t−m} gives
dTV (P(xt ∈ ·),P(x̃t ∈ ·)) ≤ 1
2 |A|Lπ m∑ j=0 E ‖θk−dj − θk−dm‖2
≤ 1 2 |A|Lπ dm∑ i=τk E ‖θk−i − θk−dm‖2,
which completes the proof.
A.3 LIPSCHITZ CONTINUITY OF VALUE FUNCTION
Lemma 3. Suppose Assumption 3 holds. For any θ1, θ2 ∈ Rd and s ∈ S, we have
‖∇Vπθ1 (s)‖2 ≤ LV , (28a) |Vπθ1 (s)− Vπθ2 (s)| ≤ LV ‖θ1 − θ2‖2, (28b)
where the constant is LV := Cψrmax/(1− γ) with Cψ defined as in Assumption 3.
Proof. First we have
Qπ(s, a) = E [ ∞∑ t=0 γtr(st, at, st+1)|s0 = s, a0 = a ]
≤ ∞∑ t=0 γtrmax = rmax 1− γ .
By the policy gradient theorem [7], we have ‖∇Vπθ1 (s)‖2 = ∥∥E [Qπθ1 (s, a)ψθ1(s, a)]∥∥2
≤ E ∥∥Qπθ1 (s, a)ψθ1(s, a)∥∥2
≤ E [ |Qπθ1 (s, a)|‖ψθ1(s, a)‖2 ] ≤ rmax
1− γ Cψ,
where the first inequality is due to Jensen’s inequality, and the last inequality follows Assumption 3 and the fact that Qπ(s, a) ≤ rmax1−γ . By the mean value theorem, we immediately have
|Vπθ1 (s)− Vπθ2 (s)| ≤ sup θ1∈Rd ∥∥∇Vπθ1 (s)∥∥2 ‖θ1 − θ2‖2 = LV ‖θ1 − θ2‖2, which completes the proof.
A.4 LIPSCHITZ CONTINUITY OF POLICY GRADIENT
We give a proposition regarding the LJ -Lipschitz of the policy gradient under proper assumptions, which has been shown by [35]. Proposition 1. Suppose Assumption 3 and 4 hold. For any θ, θ′ ∈ Rd, we have ‖∇J(θ) − ∇J(θ′)‖2 ≤ LJ‖θ − θ′‖2, where LJ is a positive constant.
A.5 LIPSCHITZ CONTINUITY OF OPTIMAL CRITIC PARAMETER
We provide a justification for Lipschitz continuity of ω∗θ in the next proposition. Proposition 2. Suppose Assumption 3 and 4 hold. For any θ1, θ2 ∈ Rd, we have ‖ω∗θ1 − ω ∗ θ2‖2 ≤ Lω‖θ1 − θ2‖2, where Lω := 2rmax|A|Lπ(λ−1 + λ−2(1 + γ))(1 + logρ κ−1 + (1− ρ)−1).
Proof. We use A1, A2, b1 and b2 as shorthand notations of Aπθ1 , Aπθ2 , bπθ1 and bπθ2 respectively. By Assumption 2, Aθ,φ is invertible for any θ ∈ Rd, so we can write ω∗θ = −A −1 θ,φbθ,φ. Then we have
‖ω∗1 − ω∗2‖2 = ‖ −A−11 b1 +A −1 2 b2‖2
= ‖ −A−11 b1 −A −1 1 b2 +A −1 1 b2 +A −1 2 b2‖2 = ‖ −A−11 (b1 − b2)− (A −1 1 −A −1 2 )b2‖2 ≤ ‖A−11 (b1 − b2)‖2 + ‖(A −1 1 −A −1 2 )b2‖2 ≤ ‖A−11 ‖2‖b1 − b2‖2 + ‖A −1 1 −A −1 2 ‖2‖b2‖2 = ‖A−11 ‖2‖b1 − b2‖2 + ‖A −1 1 (A2 −A1)A −1 2 ‖2‖b2‖2 ≤ ‖A−11 ‖2‖b1 − b2‖2 + ‖A −1 1 ‖2‖A −1 2 ‖2‖b2‖2‖(A2 −A1)‖2 ≤ λ−1 ‖b1 − b2‖2 + λ −2rmax ‖A1 −A2‖2 , (29)
where the last inequality follows Assumption 2, and the fact that
‖b2‖2 = ‖E[r(s, a, s′)φ(s)]‖2 ≤ E ‖r(s, a, s ′)φ(s)‖2 ≤ E [|r(s, a, s ′)|‖φ(s)‖2] ≤ rmax. Denote (s1, a1, s′1) and (s2, a2, s′2) as samples drawn with θ1 and θ2 respectively, i.e. s1 ∼ µθ1 , a1 ∼ πθ1 , s′1 ∼ P̃ and s2 ∼ µθ2 , a2 ∼ πθ2 , s′2 ∼ P̃ . Then we have
‖b1 − b2‖2 = ∥∥E [r(s1, a1, s′1)φ(s1)]− E [r(s2, a2, s′2)φ(s2)]∥∥
2
≤ sup s,a,s′ ‖r(s, a, s′)φ(s)‖2‖P((s1, a1, s′1) ∈ ·)− P((s2, a2, s′2) ∈ ·)‖TV
≤ rmax‖P((s1, a1, s′1) ∈ ·)− P((s2, a2, s′2) ∈ ·)‖TV
= 2rmaxdTV ( µθ1 ⊗ πθ1 ⊗ P̃, µθ2 ⊗ πθ2 ⊗ P̃ ) ≤ 2rmax|A|Lπ(1 + logρ κ−1 + (1− ρ)−1)‖θ1 − θ2‖2, (30)
where the first inequality follows the definition of total variation (TV) norm, and the last inequality follows Lemma A.1. in [17]. Similarly we have:
‖A1 −A2‖2 ≤ 2(1 + γ)dTV (µθ1 ⊗ πθ1 , µθ2 ⊗ πθ2) = (1 + γ)|A|Lπ(1 + logρ κ−1 + (1− ρ)−1)‖θ1 − θ2‖2. (31)
Substituting (30) and (31) into (29) completes the proof.
B PROOF OF MAIN THEOREMS
B.1 PROOF OF THEOREM 1
For brevity, we first define the following notations: x := (s, a, s′),
δ̂(x, ω) := r(s, a, s′) + γφ(s′)>ω − φ(s)>ω, g(x, ω) := δ̂(x, ω)φ(s),
g(θ, ω) := E s∼µθ,a∼πθ,s′∼P̃ [g(x, ω)] .
We also define constant Cδ := rmax + (1 + γ) max{ rmax1−γ , Rω}, and we immediately have ‖g(x, ω)‖2 ≤ |r(x) + γφ(s′)>ω − φ(s)>ω| ≤ rmax + (1 + γ)Rω ≤ Cδ (32) and likewise, we have ‖g(x, ω)‖2 ≤ Cδ . The critic update in Algorithm 1 can be written compactly as:
ωk+1 = ΠRω ( ωk + βkg(x(k), ωk−τk) ) , (33)
where τk is the delay of the parameters used in evaluating the kth stochastic gradient, and x(k) := (s(k), a(k), s ′ (k)) is the sample used to evaluate the stochastic gradient at kth update.
Proof. Using ω∗k as shorthand notation of ω ∗ θk , we start with the optimality gap ‖ωk+1 − ω∗k+1‖22 = ‖ΠRω ( ωk + βkg(x(k), ωk−τk) ) − ω∗k+1‖22
≤ ‖ωk + βkg(x(k), ωk−τk)− ω∗k+1‖22 = ‖ωk − ω∗k‖ 2 2 + 2βk 〈 ωk − ω∗k, g(x(k), ωk−τk) 〉 + 2 〈 ωk − ω∗k, ω∗k − ω∗k+1 〉 + ∥∥ω∗k − ω∗k+1 + βkg(x(k), ωk−τk)∥∥22 = ‖ωk − ω∗k‖ 2 2 + 2βk 〈 ωk − ω∗k, g(x(k), ωk−τk)− g(x(k), ωk) 〉 + 2βk 〈 ωk − ω∗k, g(x(k), ωk)− g(θk, ωk)
〉 + 2βk 〈ωk − ω∗k, g(θk, ωk)〉+ 2 〈 ωk − ω∗k, ω∗k − ω∗k+1 〉 + ∥∥ω∗k − ω∗k+1 + βkg(x(k), ωk−τk)∥∥22
≤ ‖ωk − ω∗k‖ 2 2 + 2βk 〈 ωk − ω∗k, g(x(k), ωk−τk)− g(x(k), ωk) 〉 + 2βk 〈 ωk − ω∗k, g(x(k), ωk)− g(θk, ωk) 〉 + 2βk 〈ωk − ω∗k, g(θk, ωk)〉+ 2 〈 ωk − ω∗k, ω∗k − ω∗k+1 〉 + 2
∥∥ω∗k − ω∗k+1∥∥22 + 2C2δβ2k. (34) We first bound 〈ωk − ω∗k, g(θk, ωk)〉 in (34) as
〈ωk − ω∗k, g(θk, ωk)〉 = 〈ωk − ω∗k, g(θk, ωk)− g(θk, ω∗k)〉 = 〈 ωk − ω∗k,E [ (γφ(s′)− φ(s))> (ωk − ω∗k)φ(s) ]〉 = 〈 ωk − ω∗k,E [ φ(s) (γφ(s′)− φ(s))> ] (ωk − ω∗k)
〉 = 〈 ωk − ω∗k, Aπθk (ωk − ω ∗ k) 〉
≤ −λ‖ωk − ω∗k‖22, (35) where the first equality is due to g(θ, ω∗θ) = Aθ,φω ∗ θ + b = 0, and the last inequality follows Assumption 2. Substituting (35) into (34), then taking expectation on both sides of (34) yield
E ‖ωk+1 − ω∗k+1‖22 ≤ (1− 2λβk)E ‖ωk − ω∗k‖ 2 2 + 2βk E 〈 ωk − ω∗k, g(x(k), ωk−τk)− g(x(k), ωk) 〉 + 2βk E 〈 ωk − ω∗k, g(x(k), ωk)− g(θk, ωk) 〉 + 2E 〈 ωk − ω∗k, ω∗k − ω∗k+1
〉 + 2E ∥∥ω∗k − ω∗k+1∥∥22 + 2C2δβ2k. (36)
We then bound the term E 〈 ωk − ω∗k, g(x(k), ωk−τk)− g(x(k), ωk) 〉 in (36) as
E 〈 ωk − ω∗k, g(x(k), ωk−τk)− g(x(k), ωk) 〉 = E 〈 ωk − ω∗k, ( γφ(s′(k))− φ(s(k)) )> (ωk−τk − ωk)φ(s(k)) 〉 ≤ (1 + γ)E [‖ωk − ω∗k‖2‖ωk−τk − ωk‖2]
≤ (1 + γ)E ‖ωk − ω∗k‖2 ∥∥∥∥∥ k−1∑ i=k−τk (ωi+1 − ωi) ∥∥∥∥∥ 2  ≤ (1 + γ)E [ ‖ωk − ω∗k‖2
k−1∑ i=k−τk βi‖g(xi, ωi−τi)‖2
]
≤ (1 + γ)E [ ‖ωk − ω∗k‖2 k−1∑ i=k−τk βk−K0‖g(xi, ωi−τi)‖2 ] ≤ Cδ(1 + γ)K0βk−K0 E ‖ωk − ω∗k‖2, (37)
where the second last inequality is due to the monotonicity of step size, and the last inequality follows the definition of Cδ in (32).
Next we jointly bound the fourth and fifth term in (36) as 2E 〈 ωk − ω∗k, ω∗k − ω∗k+1 〉 + 2E ∥∥ω∗k − ω∗k+1∥∥22 ≤ 2E [ ‖ωk − ω∗k‖2
∥∥ω∗k − ω∗k+1∥∥2]+ 2E∥∥ω∗k − ω∗k+1∥∥22 ≤ 2Lω E [‖ωk − ω∗k‖2 ‖θk − θk+1‖2] + 2L 2 ω E ‖θk − θk+1‖ 2 2
= 2Lωαk E [ ‖ωk − ω∗k‖2 ∥∥∥δ̂(x(k), ωk−τk)ψθk−τk (s(k), a(k))∥∥∥2]+ 2L2ωα2k E∥∥∥δ̂(x(k), ωk−τk)ψθk−τk (s(k), a(k))∥∥∥22 ≤ 2LωCpαk E ‖ωk − ω∗k‖2 + 2L 2 ωC 2 pα 2 k, (38)
where constant Cp := CδCψ. The second inequality is due to the Lω-Lipschitz of ω∗θ shown in Proposition 2, and the last inequality follows the fact that
‖δ̂(x(k), ωk−τk)ψθk−τk (s(k), a(k))‖2 ≤ CδCψ = Cp. (39)
Substituting (37) and (38) into (36) yields
E ‖ωk+1 − ω∗k+1‖22 ≤ (1− 2λβk)E ‖ωk − ω∗k‖ 2 2 + 2βk(C1 αk βk + C2K0βk−K0)E ‖ωk − ω∗k‖2
+ 2βk E 〈 ωk − ω∗k, g(x(k), ωk)− g(θk, ωk) 〉 + Cqβ 2 k, (40)
where C1 := LωCp, C2 := Cδ(1 + γ) and Cq := 2C2δ + 2L 2 ωC 2 p max(k) α2k β2k = 2C2δ + 2L 2 ωC 2 p c21 c22 .
For brevity, we use x ∼ θ to denote s ∼ µθ, a ∼ πθ and s′ ∼ P̃ in this proof. Consider the third term in (40) conditioned on θk, ωk, θk−τk . We bound it as
E [〈 ωk − ω∗k, g(x(k), ωk)− g(θk, ωk) 〉 |θk, ωk, θk−τk ] = 〈 ωk − ω∗k, E
x(k)∼θk−τk
[ g(x(k), ωk)|ωk ] − g(θk, ωk) 〉 = 〈 ωk − ω∗k, g(θk−τk , ωk)− g(θk, ωk)
〉 ≤ ‖ωk − ω∗k‖2‖g(θk−τk , ωk)− g(θk, ωk)‖2
≤ 2Rω ∥∥∥∥ Ex∼θk−τk [g(x, ωk)]− Ex∼θk[g(x, ωk)] ∥∥∥∥ 2
≤ 2Rω sup x ‖g(x, ωk)‖2 ∥∥∥µθk−τk ⊗ πθk−τk ⊗ P̃ − µθk ⊗ πθk ⊗ P̃∥∥∥TV ≤ 4RωCδdTV (µθk−τk ⊗ πθk−τk ⊗ P̃, µθk ⊗ πθk ⊗ P̃), (41)
where second last inequality follows the definition of TV norm and the last inequality uses the definition of Cδ in (32).
Define constant C3 := 2RωCδ|A|Lπ(1 + logρ κ−1 + (1− ρ)−1). Then by following the third item in Lemma A.1 shown by [17], we can write (41) as
E [〈 ωk − ω∗k, g(x(k), ωk)− g(θk, ωk) 〉 |θk, ωk, θk−τk ] ≤ 4RωCδdTV (µθk−τk ⊗ πθk−τk ⊗ P̃, µθk ⊗ πθk ⊗ P̃) ≤ C3 ‖θk−τk − θk‖2
≤ C3 k−1∑
i=k−τk
αi‖g(xi, ωi−τi)‖2
≤ C3CδK0αk−K0 , (42)
where we used the monotonicity of αk and Assumption 1.
Taking total expectation on both sides of (42) and substituting it into (40) yield
E ‖ωk+1 − ω∗k+1‖22 ≤ (1− 2λβk)E ‖ωk − ω∗k‖ 2 2 + 2βk ( C1 αk βk + C2K0βk−K0 ) E ‖ωk − ω∗k‖2
+ 2C3CδK0βkαk−K0 + Cqβ 2 k. (43)
Taking summation on both sides of (43) and rearranging yield
2λ K∑ k=K0 E ‖ωk − ω∗k‖ 2 2 ≤ K∑ k=K0 1 βk ( E ‖ωk − ω∗k‖ 2 2 − E ∥∥ωk+1 − ω∗k+1∥∥22) I1 +Cq K∑ k=K0 βk I2
+ 2 K∑ k=K0 2C3CδK0αk−K0
I3
+2 K∑ k=K0 ( C1 αk βk + C2K0βk−K0 ) E ‖ωk − ω∗k‖2
I4
.
(44)
We bound I1 as
I1 = K∑ k=MK 1 βk ( E ‖ωk − ω∗k‖ 2 2 − E ∥∥ωk+1 − ω∗k+1∥∥22)
= K∑ k=MK ( 1 βk − 1 βk−1 ) E ‖ωk − ω∗k‖ 2 2 +
1 βMK−1 E ∥∥ωMK − ω∗MK∥∥22 − 1βk E ∥∥ωK+1 − ω∗K+1∥∥22
≤ K∑
k=MK
( 1
βk − 1 βk−1
) E ‖ωk − ω∗k‖ 2 2 +
1 βMK−1 E ∥∥ωMK − ω∗MK∥∥22
≤ 4R2ω
( K∑
k=MK
( 1
βk − 1 βk−1
) +
1
βMK−1
) =
4R2ω βk = O(Kσ2), (45)
where the last inequality is due to the fact that
‖ωk − ω∗θ‖2 ≤ ‖ωk‖2 + ‖ω∗θ‖2 ≤ 2Rω.
We bound I2 as
K∑ k=MK βk = K∑ k=MK c2 (1 + k)σ2 = O(K1−σ2) (46)
where the inequality follows from the integration rule ∑b k=a k −σ ≤ b 1−σ 1−σ .
We bound I3 as
I3 = K∑ k=K0 2C3CδK0αk−K0 = 2C3Cδc1K0 K−K0∑ k=0 (1 + k)−σ1 = O(K0K1−σ1). (47)
For the last term I4, we have
I4 = K∑ k=K0 ( C1 αk βk + C2K0βk−K0 ) E ‖ωk − ω∗k‖2
≤ √√√√ K∑ k=K0 ( C1 αk βk + C2K0βk−K0 )2√√√√ K∑ k=K0 ( E ‖ωk − ω∗k‖2
)2 ≤
√√√√ K∑ k=K0 ( C1 αk βk + C2K0βk−K0 )2√√√√ K∑ k=K0 E ‖ωk − ω∗k‖ 2 2 , (48)
where the first inequality follows Cauchy–Schwartz inequality, and the second inequality follows Jensen’s inequality. In (48), we have K∑ k=K0 ( C1 αk βk + C2K0βk−K0 )2 ≤ K−K0∑ k=0 ( C1 αk βk + C2K0βk )2
= C21 K−K0∑ k=0 α2k β2k + 2C1C2K0 K−K0∑ k=0 αk + C 2 2K 2 0 K−K0∑ k=0 β2k
= O ( K2(σ2−σ1)+1 ) +O ( K0K −σ1+1 ) +O ( K20K 1−2σ2 ) (49)
where the first inequality is due to the fact that αkβk and βk−K0 are monotonically decreasing.
Substituting (49) into (48) gives
I4 ≤ √ O ( K2(σ2−σ1)+1 ) +O (K0K−σ1+1) +O (K20K1−2σ2) √√√√ K∑ k=MK E ‖ωk − ω∗k‖ 2 2 . (50)
Substituting (45), (46), (47) and (50) into (44), and dividing both sides of (44) by K −K0 + 1 give
2λ 1
K −K0 + 1 K∑ k=K0 E ‖ωk − ω∗k‖ 2 2
≤
√ O ( K2(σ2−σ1)+1 ) +O (K0K−σ1+1) +O (K20K1−2σ2) K −K0 + 1 √√√√ K∑ k=K0 E ‖ωk − ω∗k‖ 2 2
+O ( 1
K1−σ2
) +O ( 1
Kσ2
) +O ( K0 Kσ1 ) . (51)
We define the following functions:
T1(K) := 1
K −K0 + 1 K∑ k=K0 E ‖ωk − ω∗k‖ 2 2 ,
T2(K) := O ( 1
K1−σ2
) +O ( 1
Kσ2
) +O ( K0 Kσ1 ) ,
T3(K) := O ( K2(σ2−σ1)+1 ) +O ( K0K −σ1+1 ) +O ( K20K 1−2σ2 )
K −K0 + 1 .
Then (51) can be written as:
T1(K)− 1
2λ
√ T1(K) √ T3(K) ≤ 1
2λ T2(K).
Solving this quadratic inequality in terms of T1(K), we obtain
T1(K) ≤ 1
λ T2(K) +
1
2λ2 T3(K), (52)
which implies
1
K −K0 + 1 K∑ k=K0 E ‖ωk − ω∗k‖ 2 2
= O ( 1
K1−σ2
) +O ( 1
K2(σ1−σ2)
) +O ( K20 K2σ2 ) +O ( K0 Kσ1 ) +O ( 1 Kσ2 ) .
We further have
1
K K∑ k=1 E ‖ωk − ω∗k‖22 ≤ 1 K ( K0−1∑ k=1 4R2ω + K∑ k=K0 E ‖ωk − ω∗k‖22 )
= K0 − 1 K 4R2ω + K −K0 + 1 K
1
K −K0 + 1 K∑ k=K0 E ‖ωk − ω∗k‖22
= O ( K0 K ) +O ( 1 K −K0 + 1 K∑ k=K0 E ‖ωk − ω∗k‖ 2 2 )
= O
( 1
K −K0 + 1 K∑ k=K0 E ‖ωk − ω∗k‖ 2 2
) (53)
which completes the proof.
B.2 PROOF OF THEOREM 2
We first clarify the notations:
x := (s, a, s′),
δ̂(x, ω) := r(s, a, s′) + γφ(s′)>ω − φ(s)>ω, δ(x, θ) := r(s, a, s′) + γVπθ (s ′)− Vπθ (s).
The update in Algorithm 1 can be written compactly as:
θk+1 = θk + αk δ̂(x(k), ωk−τk)ψθk−τk (s(k), a(k)). (54)
For brevity, we use ω∗k as shorthand notation of ω ∗ θk . Then we are ready to give the proof.
Proof. From LJ -Lipschitz of policy gradient shown in Proposition 1, we have:
J(θk+1) ≥ J(θk) + 〈∇J(θk), θk+1 − θk〉 − LJ 2 ‖θk+1 − θk‖22
= J(θk) + αk 〈 ∇J(θk), ( δ̂(x(k), ωk−τk)− δ̂(x(k), ω∗k) ) ψθk−τk (s(k), a(k)) 〉 + αk 〈 ∇J(θk), δ̂(x(k), ω∗k)ψθk−τk (s(k), a(k)) 〉 − LJ 2 α2k‖δ̂(x(k), ωk−τk)ψθk−τk (s(k), a(k))‖ 2 2
≥ J(θk) + αk 〈 ∇J(θk), ( δ̂(x(k), ωk−τk)− δ̂(x(k), ω∗k) ) ψθk−τk (s(k), a(k)) 〉 + αk 〈 ∇J(θk), δ̂(x(k), ω∗k)ψθk−τk (s(k), a(k)) 〉 − LJ 2 C2pα 2 k,
where the last inequality follows the definition of Cp in (39).
Taking expectation on both sides of the last inequality yields E[J(θk+1)] ≥ E[J(θk)] + αk E 〈 ∇J(θk), ( δ̂(x(k), ωk−τk)− δ̂(x(k), ω∗k) ) ψθk−τk (s(k), a(k)) 〉 I1
+ αk E 〈 ∇J(θk), δ̂(x(k), ω∗k)ψθk−τk (s(k), a(k)) 〉 I2 −LJ 2 C2pα 2 k. (55)
We first decompose I1 as I1 = E 〈 ∇J(θk), ( δ̂(x(k), ωk−τk)− δ̂(x(k), ω∗k) ) ψθk−τk (s(k), a(k)) 〉 = E 〈 ∇J(θk), ( δ̂(x(k), ωk−τk)− δ̂(x(k), ωk) ) ψθk−τk (s(k), a(k))
〉 I (1) 1
+ E 〈 ∇J(θk), ( δ̂(x(k), ωk)− δ̂(x(k), ω∗k) ) ψθk−τk (s(k), a(k)) 〉 I (2) 1 .
We bound I(1)1 as
I (1) 1 = E 〈 ∇J(θk), ( γφ(s′(k))− φ(s(k)) )> (ωk−τk − ωk)ψθk−τk (s(k), a(k)) 〉 ≥ −E [ ‖∇J(θk)‖2‖γφ(s′(k))− φ(s(k))‖2‖ωk − ωk−τk‖2‖ψθk−τk (s(k), a(k))‖2
] ≥ −2Cψ E [‖∇J(θk)‖2‖ωk − ωk−τk‖2] ≥ −2CψCδK0βk−1 E ‖∇J(θk)‖2,
where the last inequality follows
‖ωk − ωk−τk‖2 = ∥∥∥∥∥ k−1∑
i=k−τk
(ωi+1 − ωi) ∥∥∥∥∥ 2
≤ k−1∑
i=k−τk
‖βig(xi, ωi−τi)‖2
≤ βk−1 k−1∑
i=k−τk
‖g(xi, ωi−τi)‖2
≤ βk−1K0Cδ,
where the second inequality is due to the monotonicity of step size, and the third one follows (32).
Then we bound I(2)1 as
I (2) 1 = E 〈 ∇J(θk), ( δ̂(x(k), ωk)− δ̂(x(k), ω∗k) ) ψθk−τk (s(k), a(k)) 〉 = −E 〈 ∇J(θk), ( γφ(s′(k))− φ(s(k)) )> (ω∗k − ωk)ψθk−τk (s(k), a(k))
〉 ≥ −E [ ‖∇J(θk)‖2‖γφ(s′(k))− φ(s(k))‖2‖ωk − ω ∗ k‖2‖ψθk−τk (s(k), a(k))‖2
] ≥ −2Cψ E [‖∇J(θk)‖2‖ωk − ω∗k‖2] .
Collecting the lower bounds of I(1)1 and I (2) 1 gives
I1 ≥ −2Cψ E [‖∇J(θk)‖2 (CδK0βk−1 + ‖ωk − ω∗k‖2)] . (56)
Now we consider I2. We first decompose I2 as I2 = E 〈 ∇J(θk), δ̂(x(k), ω∗k)ψθk−τk (s(k), a(k)) 〉 = E 〈 ∇J(θk), ( δ̂(x(k), ω ∗ k)− δ̂(x(k), ω∗k−τk) ) ψθk−τk (s(k), a(k))
〉 I (1) 2
+ E 〈 ∇J(θk), ( δ̂(x(k), ω ∗ k−τk)− δ(x(k), θk−τk) ) ψθk−τk (s(k), a(k)) 〉 I (2) 2
+ E 〈 ∇J(θk), δ(x(k), θk−τk)ψθk−τk (s(k), a(k))−∇J(θk) 〉 I (3) 2 +‖∇J(θk)‖22.
We bound I(1)2 as
I (1) 2 = E 〈 ∇J(θk), ( δ̂(x(k), ω ∗ k)− δ̂(x(k), ω∗k−τk) ) ψθk−τk (s(k), a(k)) 〉 = E 〈 ∇J(θk), ( γφ(s′(k))− φ(s(k)) )> ( ω∗k − ω∗k−τk ) ψθk−τk (s(k), a(k))
〉 ≥ −E [ ‖∇J(θk)‖2‖ ( γφ(s′(k))− φ(s(k)) )> ‖2 ∥∥ω∗k − ω∗k−τk∥∥2 ‖ψθk−τk (s(k), a(k))‖2]
≥ −LV Cψ(1 + γ)E ∥∥ω∗k − ω∗k−τk∥∥2 ≥ −LV LωCψ(1 + γ)E ‖θk − θk−τk‖2 ≥ −LV LωCψCp(1 + γ)K0αk−K0 ,
where the second last inequality follows from Proposition 2 and the last inequality uses (39) as
‖θk − θk−τk‖2 ≤ k−1∑
i=k−τk
‖θi+1 − θi‖2
= k−1∑ i=k−τk αi‖δ̂(xi, ωi−τi)ψθi−τi (si, ai)‖2
≤ k−1∑
i=k−τk
αk−τkCp
≤ CpK0αk−K0 . (57)
We bound I(2)2 as
I (2) 2 = E 〈 ∇J(θk), ( δ̂(x(k), ω ∗ k−τk)− δ(x(k), θk−τk) ) ψθk−τk (s(k), a(k)) 〉 ≥ −E [ ‖∇J(θk)‖2
∣∣∣δ̂(x(k), ω∗k−τk)− δ(x(k), θk−τk)∣∣∣ ‖ψθk−τk (s(k), a(k))‖2] ≥ −Cψ E [ ‖∇J(θk)‖2
∣∣∣δ̂(x(k), ω∗k−τk)− δ(x(k), θk−τk)∣∣∣] = −Cψ E [ ‖∇J(θk)‖2
∣∣∣γ (φ(s′(k))>ω∗k−τk − Vπθk−τk (s′(k)))+ Vπθk−τk (s(k))− φ(s(k))>ω∗k−τk ∣∣∣] ≥ −Cψ E [ ‖∇J(θk)‖2 ( γ ∣∣∣φ(s′(k))>ω∗k−τk − Vπθk−τk (s′(k))∣∣∣+ ∣∣∣Vπθk−τk (s(k))− φ(s(k))>ω∗k−τk ∣∣∣)]
= −Cψ E [ ‖∇J(θk)‖2 E [ γ ∣∣∣φ(s′(k))>ω∗k−τk − Vπθk−τk (s′(k))∣∣∣+ ∣∣∣Vπθk−τk (s(k))− φ(s(k))>ω∗k−τk ∣∣∣∣∣∣ θk, θk−τk]] ≥ −2Cψ app E ‖∇J(θk)‖2 ≥ −2CψLV fa − 2Cψ sp E ‖∇J(θk)‖2 (58)
where the second last inequality follows from the fact that E [ γ ∣∣∣φ(s′(k))>ω∗k−τk − Vπθk−τk (s′(k))∣∣∣+ ∣∣∣Vπθk−τk (s(k))− φ(s(k))>ω∗k−τk ∣∣∣]
≤ γ √ E ∣∣∣φ(s′(k))>ω∗k−τk − Vπθk−τk (s′(k))∣∣∣2 + √ E ∣∣∣Vπθk−τk (s(k))− φ(s(k))>ω∗k−τk ∣∣∣2
≤ 2 app.
Define artificial transition x̄(k) := (s(k), a(k), s̄′(k) ∼ P), then I (3) 2 can be bounded as
I (3) 2 = E 〈 ∇J(θk), δ(x(k), θk−τk)ψθk−τk (s(k), a(k))−∇J(θk) 〉 = E [ E [〈 ∇J(θk), δ(x(k), θk−τk)ψθk−τk (s(k), a(k))−∇J(θk)
〉∣∣∣ θk−τk , θk]] = E 〈 ∇J(θk),E [( δ(x(k), θk−τk)− δ(x̄(k), θk−τk) ) ψθk−τk (s(k), a(k))
∣∣∣ θk−τk , θk]〉 + E 〈 ∇J(θk),E [ δ(x̄(k), θk−τk)ψθk−τk (s(k), a(k))
∣∣∣ θk−τk , θk]−∇J(θk)〉 ≥ −E [ ‖∇J(θk)‖2
∥∥∥E [(δ(x(k), θk−τk)− δ(x̄(k), θk−τk))ψθk−τk (s(k), a(k))∣∣∣ θk−τk , θk]∥∥∥2] − E [ ‖∇J(θk)‖2
∥∥∥E [δ(x̄(k), θk−τk)ψθk−τk (s(k), a(k))∣∣∣ θk−τk , θk]−∇J(θk)∥∥∥2] . (59) The first term in the last inequality can be bounded as
E [( δ(x(k), θk−τk)− δ(x̄(k), θk−τk) ) ψθk−τk (s(k), a(k)) ∣∣∣ θk−τk , θk] = E [( δ(x(k), θk−τk)− δ(x̄(k), θk−τk) ) ψθk−τk (s(k), a(k))
∣∣∣ θk−τk , θk] = E [( r(x(k)) + γ E[r(s′k, a′, s′′)]− ( r(x̄(k)) + γ E[r(s̄′k, a′, s′′)] )) ψθk−τk (s(k), a(k))
∣∣∣ θk−τk , θk] ≤ 2Cψrmax‖P̃ − P‖TV ≤ 8Cψrmax(1− γ), (60)
where the last inequality follows ‖P̃ − P‖TV = 2 ∫ s′∈S ∣∣∣P̃(s′|s, a)− P(s′|s, a)∣∣∣ = 2(1− γ)∫ s′∈S |P(s′|s, a)− η(s′)| ≤ 4(1− γ).
(61)
The second term in (59) can be rewritten as E [ δ(x̄(k), θk−τk)ψθk−τk (s(k), a(k)) ∣∣∣ θk−τk , θk] = E
s(k)∼µθk−τk a(k)∼πθk−τk s̄′(k)∼P
[( r(x̄(k)) + γVπθk−τk (s̄′(k))− Vπθk−τk (s(k)) ) ψθk−τk (s(k), a(k)) ∣∣∣∣θk−τk , θk]
= E s(k)∼µθk−τk a(k)∼πθk−τk
[( Qπθk−τk (s(k), a(k))− Vπθk−τk (s(k)) ) ψθk−τk (s(k), a(k)) ∣∣∣∣θk−τk , θk]
= E s(k)∼µθk−τk a(k)∼πθk−τk
[ Aπθk−τk (s(k), a(k))ψθk−τk (s(k), a(k)) ∣∣∣∣θk−τk , θk]
= E s(k)∼dθk−τk a(k)∼πθk−τk
[ Aπθk−τk (s(k), a(k))ψθk−τk (s(k), a(k)) ∣∣∣∣θk−τk , θk] = ∇J(θk−τk) (62) where the second last equality follows µθ(·) = dθ(·) with dθ being a shorthand notation of dπθ [6].
Substituting (60) and (62) into (59) yields
I (3) 2 ≥ −8Cψrmax(1− γ)E ‖∇J(θk)‖2 − E [‖∇J(θk)‖2‖∇J(θk−τk)−∇J(θk)‖2] ≥ −8Cψrmax(1− γ)E ‖∇J(θk)‖2 − LV LJ E ‖θk−τk − θk‖2 ≥ −8Cψrmax(1− γ)E ‖∇J(θk)‖2 − LV LJCpK0αk−K0 , (63)
where the second last inequality is due to LJ -Lipschitz of policy gradient shown in Proposition 1, and the last inequality follows (57).
Collecting lower bounds of I(1)2 , I (2) 2 and I (3) 2 gives
I2 ≥ −D1K0αk−K0 − (2Cψ sp + 8Cψrmax(1− γ))E ‖∇J(θk)‖2 − 2CψLV fa + ‖∇J(θk)‖ 2 2, (64) where the constant is D1 := LV LωCψCp(1 + γ) + LV LJCp.
Substituting (56) and (64) into (55) yields E[J(θk+1)] ≥ E[J(θk)]− 2αkCψ ( sp + 4rmax(1− γ) + CδK0βk−1 + ‖ωk − ω∗k‖2)E ‖∇J(θk)‖2
− αkD1K0αk−K0 − 2αkCψLV fa + αk‖∇J(θk)‖22 − LJ 2 C2pα 2 k. (65)
By following Cauchy-Schwarz inequality, the second term in (65) can be bounded as ( sp + 4rmax(1− γ) + CδK0βk−1 + ‖ωk − ω∗k‖2)E ‖∇J(θk)‖2
≤ √ E ‖∇J(θk)‖22 E [ ( sp + 4rmax(1− γ) + CδK0βk−1 + ‖ωk − ω∗k‖2) 2 ]
≤ √ E ‖∇J(θk)‖22 √ E [ 4C2δK 2 0β 2 k−1 + 4‖ωk − ω∗k‖22 + 4 2sp + 64r2max(1− γ)2 ] = 2 √ E ‖∇J(θk)‖22 √ C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O( 2sp), (66)
where the last inequality follows the order of sp in Lemma 7.
Collecting the upper bound gives E[J(θk+1)] ≥ E[J(θk)]− 4αkCψ √ E ‖∇J(θk)‖22 √ C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O( 2sp)
− αkD1K0αk−K0 − 2αkCψLV fa + αk‖∇J(θk)‖22 − LJ 2 C2pα 2 k. (67)
Dividing both sides of (67) by αk, then rearranging and taking summation on both sides give K∑
k=K0
E ‖∇J(θk)‖22 ≤ K∑
k=K0
1
αk (E[J(θk+1)]− E[J(θk)])
I3
+ K∑ k=K0 ( D1K0αk−K0 + LJ 2 C2pαk ) I4
+ 4Cψ K∑ k=K0 √ E ‖∇J(θk)‖22 √ C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O( 2sp)
I5
+ 2CψLV (K −K0 + 1) fa. (68)
We bound I3 as
I3 = K∑ k=K0 1 αk (E [J(θk+1)]− E [J(θk)])
= K∑ k=K0 ( 1 αk−1 − 1 αk ) E [J(θk)]−
1
αMK−1 E [J(θMK )] +
1
αK E [J(θK+1)]
≤ 1 αK E [J(θK+1)] ≤ rmax 1− γ 1 αK = O(Kσ1), (69)
where the first inequality is due to the αk is monotonic decreasing and positive, and last inequality is due to Vπθ (s) ≤ rmax1−γ for any s ∈ S and πθ.
We bound I4 as
I4 = K∑ k=K0 ( D1K0αk−K0 + LJ 2 C2pαk ) ≤ K−K0∑ k=0 ( D1K0αk + LJ 2 C2pαk ) = O(K0K1−σ1).
We bound I5 as
I5 = K∑ k=K0 √ E ‖∇J(θk)‖22 √ C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O( 2sp)
≤ √√√√ K∑ k=K0 E ‖∇J(θk)‖22 √√√√ K∑ k=K0 ( C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O( 2sp) )
= √√√√ K∑ k=K0 E ‖∇J(θk)‖22 √√√√C2δK20 K∑ k=K0 β2k−1 + K∑ k=K0 E ‖ωk − ω∗k‖22 +O(K 2sp), (70)
where the first inequality follows Cauchy-Schwartz inequality.
In (70), we have
K∑ k=K0 β2k−1 ≤ K−K0∑ k=0 β2k = K−K0∑ k=0 c22(1 + k) −2σ2 = O(K1−2σ2).
Substituting the last equality into (70) gives
I5 ≤ √√√√ K∑ k=MK E ‖∇J(θk)‖22 √√√√O(K20K1−2σ2) + K∑ k=MK E ‖ωk − ω∗k‖22 +O(K 2sp). (71)
Dividing both sides of (67) by K −K0 + 1 and collecting upper bounds of I3, I4 and I5 give
1
K −K0 + 1 K∑ k=K0 E ‖∇J(θk)‖22
≤ 4Cψ K −K0 + 1 √√√√ K∑ k=K0 E ‖∇J(θk)‖22 √√√√O(K20K1−2σ2) + K∑ k=K0 E ‖ωk − ω∗k‖22 +O(K 2sp)
+O ( 1
K1−σ1
) +O ( K0 Kσ1 ) +O( fa). (72)
Define the following functions
T4(K) := 1
K −K0 + 1 K∑ k=K0 E ‖∇J(θk)‖22,
T5(K) := 1
K −K0 + 1
( O(K20K1−2σ2) + K∑ k=K0 E ‖ωk − ω∗k‖22 +O(K 2sp) ) ,
T6(K) := O ( 1
K1−σ1
) +O ( K0 Kσ1 ) +O( fa).
Then (72) can be rewritten as
T4(K) ≤ T6(K) + √ 2(1 + γ)Cψ √ T4(K) √ T5(K).
Solving this quadratic inequality in terms of T4(K), we obtain
T4(K) ≤ 2T6(K) + 4(1 + γ)2C2ψT5(K), (73) which implies
1
K −K0 + 1 K∑ k=K0 E ‖∇J(θk)‖22
= O ( 1
K1−σ1
) +O ( K0 Kσ1 ) +O ( K20 K2σ2 ) +O ( 1 K −K0 + 1 K∑ k=K0 E ‖ωk − ω∗k‖22 ) +O( app).
We further have
1
K K∑ k=1 E ‖∇J(θk)‖22 ≤ 1 K ( K0−1∑ k=1 L2V + K∑ k=K0 E ‖∇J(θk)‖22 )
= K0 − 1 K L2V + K −K0 + 1 K
1
K −K0 + 1 K∑ k=K0 E ‖∇J(θk)‖22
= O ( K0 K ) +O ( 1 K −K0 + 1 K∑ k=K0 E ‖∇J(θk)‖22 )
= O
( 1
K −K0 + 1 K∑ k=K0 E ‖∇J(θk)‖22
) (74)
which completes the proof.
B.3 PROOF OF THEOREM 3
Given the definition in Section B.1, we now give the convergence proof of critic update in Algorithm 1 with linear function approximation and Markovian sampling.
By following the derivation of (40), we have
E ‖ωk+1 − ω∗k+1‖22 ≤ (1− 2λβk)E ‖ωk − ω∗k‖ 2 2 + 2βk(C1 αk βk + C2K0βk−K0)E ‖ωk − ω∗k‖2
+ 2βk E 〈 ωk − ω∗k, g(x(k), ωk)− g(θk, ωk) 〉 + Cqβ 2 k, (75)
where C1 := CpLω , C2 := Cδ(1 + γ) and Cq := 2C2δ + 2L 2 ωC 2 p max(k) α2k β2k = 2C2δ + 2L 2 ωC 2 p c21 c22 .
Now we consider the third item in the last inequality. For some m ∈ N+, we define M := (K0 + 1)m+K0. Following Lemma 4 (to be presented in Sec. C.1), for some dm ≤M and positive constants C4, C5, C6, C7, we have E 〈 ωk − ω∗k, g(x(k), ωk)− g(θk, ωk)
〉 ≤ C4 E ‖θk − θk−dm‖2 + C5
dm∑ i=τk E ‖θk−i − θk−dm‖2 + C6 E ‖ωk − ωk−dm‖2 + C7κρm−1
≤ C4 k−1∑
i=k−dm E ‖θi+1 − θi‖2 + C5 dm−1∑ i=τk k−i−1∑ j=k−dm E ‖θj+1 − θj‖2 + C6 k−1∑ i=k−dm E ‖ωi+1 − ωi‖2 + C7κρm−1
≤ C4 k−1∑
i=k−dm
αiCp + C5 dm−1∑ i=τk k−i−1∑ j=k−dm αjCp + C6 k−1∑ i=k−dm βiCδ + C7κρ m−1
≤ C4αk−dm k−1∑
i=k−dm
Cp + C5αk−dm dm−1∑ i=τk k−i−1∑ j=k−dm Cp + C6βk−dm k−1∑ i=k−dm Cδ + C7κρ m−1
≤ C4dmCpαk−dm + C5(dm − τk)2Cpαk−dm + C6dmCδβk−dm + C7κρm−1 ≤ ( C4M + C5M 2 ) Cpαk−M + C6MCδβk−M + C7κρ m−1, (76)
where the third last inequality is due to the monotonicity of step size, and the last inequality is due to τk ≥ 0 and dm ≤M . Further letting m = mK which is defined in (21) yields
E 〈 ωk − ω∗k, g(x(k), ωk)− g(θk, ωk) 〉 = ( C4MK + C5M 2 K ) Cpαk−MK + C6CδMKβk−MK + C7κρ mK−1
≤ ( C4MK + C5M 2 K ) Cpαk−MK + C6CδMKβk−MK + C7αK , (77)
where MK = (K0 + 1)mK +K0, and the last inequality follows the definition of mK .
Substituting (77) into (75), then rearranging and summing up both sides over k = MK , ...,K yield
2λ K∑ k=MK E ‖ωk − ω∗k‖ 2 2 ≤ K∑ k=MK 1 βk ( E ‖ωk − ω∗k‖ 2 2 − E ∥∥ωk+1 − ω∗k+1∥∥22) I1 +Cq K∑ k=MK βk I2
+ 2 K∑ k=MK (( C4MK + C5M 2 K ) Cpαk−MK + C6CδMKβk−MK + C7αK ) I3
+ 2 K∑ k=MK ( C1 αk βk + C2K0βk−K0 ) E ‖ωk − ω∗k‖2
I4
. (78)
where the order of I1, I2 and I4 have already been given by (45), (46) and (50) respectively.
We bound I3 as
I3 = ( C4MK + C5M 2 K ) Cp K∑ k=MK αk + C6CδMK K∑ k=MK βk + C7αK K∑ k=MK 1
≤ ( C4MK + C5M 2 K ) Cpc1 K1−σ1
1− σ1 + C6CδMKc2
K1−σ2 1− σ2 + C7c1K(1 +K) −σ1
= O ( (K20 log 2K)K1−σ1 ) +O ( (K0 logK)K 1−σ2 ) , (79)
where the last inequality follows from the integration rule ∑b k=a k −σ ≤ b 1−σ
1−σ , and the last equality is due to O(MK) = O(K0mK) = O(K0 logK). Collecting the bounds of I1, I2, I3 and I4, and dividing both sides of (78) by K −MK + 1 yield
2λ 1
K −MK + 1 K∑ k=MK E ‖ωk − ω∗k‖ 2 2
≤
√ O ( K2(σ2−σ1)+1 ) +O (K0K−σ1+1) +O (K20K1−2σ2) K −MK + 1 √√√√ K∑ k=MK E ‖ωk − ω∗k‖ 2 2
+O ( 1
K1−σ2
) +O ( K20 log 2K
Kσ1
) +O ( K0 logK
Kσ2
) . (80)
Similar to the derivation of (52), (80) implies
1
K −MK + 1 K∑ k=MK E ‖ωk − ω∗k‖ 2 2
= O ( 1
K1−σ2
) +O ( 1
K2(σ1−σ2)
) +O ( K20 K2σ2 ) +O ( K20 log 2K Kσ1 ) +O ( K0 logK Kσ2 ) .
Similar to (53), we have
1
K K∑ k=1 E ‖ωk − ω∗k‖22 = O ( K0 logK K ) +O ( 1 K −MK + 1 K∑ k=MK E ‖ωk − ω∗k‖ 2 2 )
= O
( 1
K −MK + 1 K∑ k=MK E ‖ωk − ω∗k‖ 2 2
) (81)
which completes the proof.
B.4 PROOF OF THEOREM 4
Given the definition in section B.2, we now give the convergence proof of actor update in Algorithm 1 with linear value function approximation and Markovian sampling method.
By following the derivation of (55), we have E[J(θk+1)] ≥ E[J(θk)] + αk E 〈 ∇J(θk), ( δ̂(x(k), ωk−τk)− δ̂(x(k), ω∗k) ) ψθk−τk (s(k), a(k)) 〉 I1
+ αk E 〈 ∇J(θk), δ̂(x(k), ω∗k)ψθk−τk (s(k), a(k)) 〉 I2 −LJ 2 C2pα 2 k. (82)
The item I1 can be bounded by following (56) as
I1 ≥ −2Cψ E [‖∇J(θk)‖2 (CδK0βk−1 + ‖ωk − ω∗k‖2)] . (83)
Next we consider I2. We first decompose it as I2 = E 〈 ∇J(θk), δ̂(x(k), ω∗k)ψθk−τk (s(k), a(k)) 〉 = E 〈 ∇J(θk), ( δ̂(x(k), ω ∗ k)− δ(x(k), θk) ) ψθk−τk (s(k), a(k))
〉 I (1) 2
+ E 〈 ∇J(θk), δ(x(k), θk)ψθk−τk (s(k), a(k))−∇J(θk) 〉 I (2) 2 +E ‖∇J(θk)‖22. (84)
For some m ∈ N+, define M := (K0 + 1)m + K0. Following Lemma 5, for some dm ≤ M and positive constants D2, D3, D4, D5, I (1) 2 can be bounded as
I (1) 2 = E 〈 ∇J(θk), ( δ̂(x(k), ω ∗ k)− δ(x(k), θk) ) ψθk−τk (s(k), a(k)) 〉 ≥ −D2 E ‖θk−τk − θk−dm‖2 −D3 E ‖θk − θk−dm‖2 −D4
k−τk∑ i=k−dm E ‖θi − θk−dm‖2
−D5κρm−1 − 2CψLV fa − 2Cψ sp E ‖∇J(θk)‖2 ≥ −D2(dm − τk)Cpαk−dm −D3dmCpαk−dm −D4(dm − τk)2Cpαk−dm −D5κρm−1 − 2CψLV fa − 2Cψ sp E ‖∇J(θk)‖2, (85)
where the derivation of the last inequality is similar to that of (76).
By setting m = mK in (85), and following the fact that dmK ≤MK and τk ≥ 0, we have
I (1) 2 ≥ −D2MKCpαk−MK −D3MKCpαk−MK −D4M2KCpαk−MK −D5κρmK−1
− 2CψLV fa − 2Cψ sp E ‖∇J(θ)‖2 = − ( (D2 +D3)CpMK +D4CpM 2 K ) αk−MK −D5κρmK−1 − 2CψLV fa − 2Cψ sp E ‖∇J(θk)‖2
≥ − ( (D2 +D3)CpMK +D4CpM 2 K ) αk−MK −D5αK − 2CψLV fa − 2Cψ sp E ‖∇J(θk)‖2,
(86)
where the last inequality is due to the definition of mK .
Following Lemma 6, for some positive constants D6, D7, D8 and D9, we bound I (2) 2 as
I (2) 2 = E 〈 ∇J(θk), δ(x(k), θk)ψθk−τk (s(k), a(k))−∇J(θk) 〉 ≥ −D6 E ‖θk−τk − θk−dm‖2 −D7 E ‖θk − θk−dm‖2 −D8
dm∑ i=τk E ‖θk−i − θk−dm‖2
−D9κρm−1 − 8Cψrmax(1− γ)E ‖∇J(θk)‖2 .
Similar to the derivation of (86), we have
I (2) 2 ≥ − (D6 +D7 +D8MK)CpMKαk−MK −D9αK − 8Cψrmax(1− γ)E ‖∇J(θk)‖2 . (87)
Collecting the lower bounds of I(1)2 and I (2) 2 yields
I2 ≥ −2CψLV fa − 2Cψ ( sp + 4rmax(1− γ))E ‖∇J(θk)‖2 + E ‖∇J(θk)‖22 −DKαk−MK − (D5 +D9)αK , (88)
where we define DK := (D4 +D8)CpM2K + (D2 +D3 +D6 +D7)CpMK for brevity.
Substituting (83) and (88) into (82) yields
E[J(θk+1)] ≥ E[J(θk)]− 2αkCψ E [‖∇J(θk)‖2 ( sp + 4rmax(1− γ) + CδK0βk−1 + ‖ωk − ω∗k‖2)]
− αk (DKαk−MK + (D5 +D9)αK)− 2CψLV faαk + αk E ‖∇J(θk)‖22 − LJ 2 C2pα 2 k.
Similar to the derivation of (67), the last inequality implies E[J(θk+1)] ≥ E[J(θk)]− 4αkCψ √ E ‖∇J(θk)‖22 √ C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O( 2sp)
− αk (DKαk−MK + (D5 +D9)αK)− 2CψLV faαk + αk E ‖∇J(θk)‖22 − LJ 2 C2pα 2 k.
Rearranging and dividing both sides by αk yield
E ‖∇J(θk)‖22 ≤ 1
αk (E[J(θk+1)]− E[J(θk)]) +DKαk−MK + (D5 +D9)αK + LJ 2 C2pαk
+ 4Cψ √ E ‖∇J(θk)‖22 √ C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O( 2sp) + 2CψLV fa.
Taking summation gives
K∑ k=MK E ‖∇J(θk)‖22 ≤ K∑ k=MK 1 αk (E[J(θk+1)]− E[J(θk)])
I3
+ K∑ k=MK ( DKαk−MK + LJ 2 C2pαk + (D5 +D9)αK ) I4
+ 4Cψ K∑ k=MK √ E ‖∇J(θk)‖22 √ C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O( 2sp)
I5
+ 2CψLV (K −MK + 1) fa. (89)
in which the upper bounds of I3 and I5 have already been given by (69) and (71) respectively.
We bound I4 as
I4 = K∑ k=MK ( DKαk−MK + LJ 2 C2pαk + (D5 +D9)αK )
≤ K∑
k=MK
( DKαk−MK +
LJ 2 C2pαk−MK + (D5 +D9)αK
)
= ( DK +
LJ 2 C2p ) K∑ k=MK αk−MK + (D5 +D9)(K −MK + 1)αK
= ( DK +
LJ 2 C2p )K−MK∑ k=0 αk + (D5 +D9)(K −MK + 1)αK
≤ ( DK +
LJ 2 C2p
) c1
1− σ1 K1−σ1 + c1(D5 +D9)(K + 1) 1−σ1
= O ( (K20 log 2K)K1−σ1 )
(90)
where the last inequality uses ∑b k=a k −σ ≤ b 1−σ 1−σ , and the last equality is due to the fact that
O(DK) = O(M2K +MK) = O((K0mK)2 +K0mK) = O(K20 log 2K).
Substituting the upper bounds of I3, I4 and I5 into (89), and dividing both sides by K−MK + 1 give
1
K −MK + 1 K∑ k=MK E ‖∇J(θk)‖22
≤ 4Cψ K −MK + 1 √√√√ K∑ k=MK E ‖∇J(θk)‖22 √√√√O(K20K1−2σ2) + K∑ k=MK E ‖ωk − ω∗k‖22 +O(K 2sp)
+O ( 1
K1−σ1
) +O ( K20 log 2K
Kσ1
) +O( fa). (91)
Following the similar steps of those in (73), (91) essentially implies
1
K −MK + 1 K∑ k=MK E ‖∇J(θk)‖22
= O ( 1
K1−σ1
) +O ( K20 log 2K
Kσ1
) +O ( K20 K2σ2 ) +O ( 1 K −MK + 1 K∑ k=MK E ‖ωk − ω∗θk‖ 2 2 ) +O( app).
Similar to (74), we have
1
K K∑ k=1 E ‖∇J(θk)‖22 = O ( K0 logK K ) +O ( 1 K −MK + 1 K∑ k=MK E ‖∇J(θk)‖22 )
= O
( 1
K −MK + 1 K∑ k=MK E ‖∇J(θk)‖22
)
which completes the proof.
C SUPPORTING LEMMAS
C.1 SUPPORTING LEMMAS FOR THEOREM 3
Lemma 4. For any m ≥ 1 and k ≥ (K0 + 1)m+K0 + 1, we have
E 〈 ωk − ω∗θk , g(x(k), ωk)− g(θk, ωk) 〉 ≤ C4 E ‖θk − θk−dm‖2 + C5 dm∑ i=τk E ‖θk−i − θk−dm‖2
+ C6 E ‖ωk − ωk−dm‖2 + C7κρm−1,
where dm ≤ (K0 + 1)m + K0, and C4 := 2CδLω + 4RωCδ|A|Lπ(1 + logρ κ−1 + (1 − ρ)−1), C5 := 4RωCδ|A|Lπ and C6 := 4(1 + γ)Rω + 2Cδ , C7 := 8RωCδ .
Proof. Consider the collection of random samples {x(k−K0−1), x(k−K0), ..., x(k)}. Suppose x(k) is sampled by worker n, then due to Assumption 1, {x(k−K0−1), x(k−K0), ..., x(k−1)} will contain at least another sample drawn by worker n. Therefore, {x(k−(K0+1)m), x(k−(K0+1)m+1), ..., x(k−1)} will contain at least m samples from worker n.
Consider the Markov chain formed by m+ 1 samples in {x(k−(K0+1)m), x(k−(K0+1)m+1), ..., x(k)}:
st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−1−−−−−−→ at−m+1 · · · st−1 θk−d1−−−−→ at−1 P̃−→ st θk−d0−−−−→ at P̃−→ st+1,
where (st, at, st+1) = (s(k), a(k), s′(k)), and {dj} m j=0 is some increasing sequence with d0 := τk.
Suppose θk−dm was used to do the kmth update, then we have xt−m = x(km). Following Assumption 1, we have τkm = km − (k − dm) ≤ K0. Since x(km) is in {x(k−(K0+1)m), ..., x(k)}, we have km ≥ k − (K0 + 1)m. Combining these two inequalities, we have
dm ≤ (K0 + 1)m+K0. (92) Given (st−m, at−m, st−m+1) and θk−dm , we construct an auxiliary Markov chain as that in Lemma 2:
st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−−−−→ ãt−m+1 · · · s̃t−1 θk−dm−−−−→ ãt−1 P̃−→ s̃t θk−dm−−−−→ ãt P̃−→ s̃t+1.
For brevity, we define
∆1(x, θ, ω) := 〈ω − ω∗θ , g(x, ω)− g(θ, ω)〉 .
Throughout this proof, we use θ, θ′, ω, ω′, x and x̃ as shorthand notations of θk, θk−dm , ωk, ωk−dm , xt and x̃t respectively.
First we decompose ∆1(x, θ, ω) as
∆1(x, θ, ω) = ∆1(x, θ, ω)−∆1(x, θ′, ω) I1 + ∆1(x, θ ′, ω)−∆1(x, θ′, ω′) I2
+ ∆1(x, θ ′, ω′)−∆1(x̃, θ′, ω′)
I3
+ ∆1(x̃, θ ′, ω′)
I4
. (93)
We bound I1 in (93) as
∆1(x, θ, ω)−∆1(x, θ′, ω) = 〈ω − ω∗θ , g(x, ω)− g(θ, ω)〉 − 〈ω − ω∗θ′ , g(x, ω)− g(θ′, ω)〉 ≤ |〈ω − ω∗θ , g(x, ω)− g(θ, ω)〉 − 〈ω − ω∗θ′ , g(x, ω)− g(θ, ω)〉|
+ |〈ω − ω∗θ′ , g(x, ω)− g(θ, ω)〉 − 〈ω − ω∗θ′ , g(x, ω)− g(θ′, ω)〉| . (94)
For the first term in (94), we have
|〈ω − ω∗θ , g(x, ω)− g(θ, ω)〉 − 〈ω − ω∗θ′ , g(x, ω)− g(θ, ω)〉| = |〈ω∗θ − ω∗θ′ , g(x, ω)− g(θ, ω)〉| ≤ ‖ω∗θ − ω∗θ′‖2‖g(x, ω)− g(θ, ω)‖ ≤ 2Cδ‖ω∗θ − ω∗θ′‖2 ≤ 2CδLω‖θ − θ′‖2,
where the last inequality is due to Proposition 2.
We use x ∼ θ′ as shorthand notations to represent that s ∼ µθ′ , a ∼ πθ′ , s′ ∼ P̃ . For the second term in (94), we have
|〈ω − ω∗θ′ , g(x, ω)− g(θ, ω)〉 − 〈ω − ω∗θ′ , g(x, ω)− g(θ′, ω)〉| = |〈ω − ω∗θ′ , g(θ′, ω)− g(θ, ω)〉| ≤ ‖ω − ω∗θ′‖2‖g(θ′, ω)− g(θ, ω)‖2 ≤ 2Rω‖g(θ′, ω)− g(θ, ω)‖2
= 2Rω ∥∥∥∥ Ex∼θ′ [g(x, ω)]− Ex∼θ [g(x, ω)] ∥∥∥∥
2
≤ 2Rω sup x ‖g(x, ω)‖2‖µθ′ ⊗ πθ′ ⊗ P̃ − µθ ⊗ πθ ⊗ P̃‖TV
≤ 2RωCδ‖µθ′ ⊗ πθ′ ⊗ P̃ − µθ ⊗ πθ ⊗ P̃‖TV
= 4RωCδdTV ( µθ′ ⊗ πθ′ ⊗ P̃, µθ ⊗ πθ ⊗ P̃ ) ≤ 4RωCδ|A|Lπ(1 + logρ κ−1 + (1− ρ)−1)‖θ − θ′‖2,
where the third inequality follows the definition of TV norm, the second last inequality follows (32), and the last inequality follows Lemma A.1. in [17].
Collecting the upper bounds of the two terms in (94) yields I1 ≤ [ 2CδLω + 4RωCδ|A|Lπ(1 + logρ κ−1 + (1− ρ)−1) ] ‖θ − θ′‖2.
Next we bound E[I2] in (93) as E[I2] = E[∆1(x, θ′, ω)−∆1(x, θ′, ω′)]
= E 〈ω − ω∗θ′ , g(x, ω)− g(θ′, ω)〉 − 〈ω′ − ω∗θ′ , g(x, ω′)− g(θ′, ω′)〉 ≤ E |〈ω − ω∗θ′ , g(x, ω)− g(θ′, ω)〉 − 〈ω − ω∗θ′ , g(x, ω′)− g(θ′, ω′)〉|
+ E |〈ω − ω∗θ′ , g(x, ω′)− g(θ′, ω′)〉 − 〈ω′ − ω∗θ′ , g(x, ω′)− g(θ′, ω′)〉| . (95) We bound the first term in (95) as
E |〈ω − ω∗θ′ , g(x, ω)− g(θ′, ω)〉 − 〈ω − ω∗θ′ , g(x, ω′)− g(θ′, ω′)〉| = E |〈ω − ω∗θ′ , g(x, ω)− g(x, ω′) + g(θ′, ω′)− g(θ′, ω)〉| ≤ 2Rω (E ‖g(x, ω)− g(x, ω′)‖2 + E ‖g(θ′, ω′)− g(θ′, ω)‖2)
≤ 2Rω ( E ‖g(x, ω)− g(x, ω′)‖2 + E ∥∥∥∥ Ex∼θ′[g(x, ω′)]− Ex∼θ′[g(x, ω)] ∥∥∥∥
2 ) = 2Rω ( E ‖(γφ(s′)− φ(s))>(ω − ω′)‖2 + E ∥∥∥∥ Ex∼θ′ [(γφ(s′)− φ(s))>] (ω′ − ω) ∥∥∥∥
2 ) ≤ 2Rω ((1 + γ)E ‖ω − ω′‖2 + (1 + γ)E ‖ω − ω′‖2) = 4Rω(1 + γ)E ‖ω − ω′‖2.
We bound the second term in (95) as E |〈ω − ω∗θ′ , g(x, ω′)− g(θ′, ω′)〉 − 〈ω′ − ω∗θ′ , g(x, ω′)− g(θ′, ω′)〉| = E |〈ω − ω′, g(x, ω′)− g(θ′, ω′)〉| ≤ 2Cδ E ‖ω − ω′‖2.
Collecting the upper bounds of the two terms in (95) yields E[I2] ≤ (4(1 + γ)Rω + 2Cδ)E ‖ω − ω′‖2.
We first bound I3 as E[I3|θ′, ω′, st−m+1] = E [∆1(x, θ′, ω′)−∆1(x̃, θ′, ω′)|θ′, ω′, st−m+1]
≤ |E [∆1(x, θ′, ω′)|θ′, ω′, st−m+1]− E [∆1(x̃, θ′, ω′)|θ′, ω′, st−m+1]| ≤ sup
x |∆1(x, θ′, ω′)| ‖P(x ∈ ·|θ′, ω′, st−m+1)− P(x̃ ∈ ·|θ′, ω′, st−m+1)‖TV
≤ 8RωCδdTV (P(x ∈ ·|θ′, st−m+1),P(x̃ ∈ ·|θ′, st−m+1)) , (96)
where the second last inequality follows the definition of TV norm, and the last inequality follows the fact that
|∆1(x, θ′, ω′)| ≤ ‖ω′ − ω∗θ′‖2‖g(x, ω′)− g(θ′, ω′)‖2 ≤ 4RωCδ.
By following (22) in Lemma 2, we have
dTV (P(x ∈ ·|θ′, st−m+1),P(x̃ ∈ ·|θ′, st−m+1)) ≤ 1
2 |A|Lπ dm∑ i=τk E [‖θk−i − θk−dm‖2| θ′, st−m+1] .
Substituting the last inequality into (96), then taking total expectation on both sides yield
E[I3] ≤ 4RωCδ|A|Lπ dm∑ i=τk E ‖θk−i − θk−dm‖2.
Next we bound I4. Define x := (s, a, s′) where s ∼ µθ′ , a ∼ πθ′ and s′ ∼ P̃ . It is immediate that E[∆1(x, θ′, ω′)|θ′, ω′, st−m+1] = 〈ω′ − ω∗θ′ ,E[g(x, ω′)|θ′, ω′, st−m+1]− g(θ′, ω′)〉 = 〈ω′ − ω∗θ′ , g(θ′, ω′)− g(θ′, ω′)〉 = 0. (97) Then we have
E[I4|θ′, ω′, st−m+1] = E [∆1(x̃, θ′, ω′)−∆1(x, θ′, ω′)|θ′, ω′, st−m+1] ≤ |E [∆1(x̃, θ′, ω′)|θ′, ω′, st−m+1]− E [∆1(x, θ′, ω′)|θ′, ω′, st−m+1]| ≤ sup
x |∆1(x, θ′, ω′)| ‖P(x̃ ∈ ·|θ′, st−m+1)− P(x ∈ ·|θ′, st−m+1)‖TV
≤ 8RωCδdTV (P(x̃ ∈ ·|θ′, st−m+1),P(x ∈ ·|θ′, st−m+1))
= 8RωCδdTV ( P(x̃ ∈ ·|θ′, st−m+1), µθ′ ⊗ πθ′ ⊗ P̃ ) , (98)
where the second inequality follows the definition of TV norm, and the third inequality follows (97).
The auxiliary Markov chain with policy πθ′ starts from initial state st−m+1, and s̃t is the (m− 1)th state on the chain. Following Lemma 1, we have:
dTV ( P(x̃ ∈ ·|θ′, st−m+1), µθ′ ⊗ πθ′ ⊗ P̃ ) = dTV ( P ((s̃t, ãt, s̃t+1) ∈ ·|θ′, st−m+1) , µθ′ ⊗ πθ′ ⊗ P̃ ) ≤ κρm−1.
Substituting the last inequality into (98) and taking total expectation on both sides yield
E[I4] ≤ 8RωCδκρm−1. Taking total expectation on (93) and collecting bounds of I1, I2, I3, I4 yield
E [∆1(x, θ, ω)] ≤ C4 E ‖θk − θk−dm‖2 + C5 dm∑ i=τk E ‖θk−i − θk−dm‖2
+ C6 E ‖ωk − ωk−dm‖2 + C7κρm−1, where C4 := 2CδLω + 4RωCδ|A|Lπ(1 + logρ κ−1 + (1 − ρ)−1), C5 := 4RωCδ|A|Lπ, C6 := 4(1 + γ)Rω + 2Cδ and C7 := 8RωCδ .
C.2 SUPPORTING LEMMAS FOR THEOREM 4
Lemma 5. For any m ≥ 1 and k ≥ (K0 + 1)m+K0 + 1, we have E 〈 ∇J(θk), ( δ̂(x(k), ω ∗ k)− δ(x(k), θk) ) ψθk−τk (s(k), a(k)) 〉 ≥ −D2 E ‖θk−τk − θk−dm‖2
−D3 E ‖θk − θk−dm‖2 −D4 dm∑ i=τk E ‖θk−i − θk−dm‖2 −D5κρm−1 − 2CψLV fa − 2Cψ sp E ‖∇J(θ)‖2,
where D2 := 2LV LψCδ, D3 := (2CδCψLJ + LV Cψ(Lω + LV )(1 + γ) + 2CψLJ app), D4 := 2LV CψCδ|A|Lπ and D5 := 4LV CψCδ .
Proof. For the worker that contributes to the kth update, we construct its Markov chain:
st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−1−−−−−−→ at−m+1 · · · st−1 θk−d1−−−−→ at−1 P̃−→ st θk−d0−−−−→ at P̃−→ st+1,
where (st, at, st+1) = (s(k), a(k), s′(k)), and {dj} m j=0 is some increasing sequence with d0 := τk. By (92) in Lemma 4, we have dm ≤ (K0 + 1)m+K0. Given (st−m, at−m, st−m+1) and θk−dm , we construct an auxiliary Markov chain:
st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−−−−→ ãt−m+1 · · · s̃t−1 θk−dm−−−−→ ãt−1 P̃−→ s̃t θk−dm−−−−→ ãt P̃−→ s̃t+1. First we have〈 ∇J(θk), ( δ̂(x(k), ω ∗ k)− δ(x(k), θk) ) ψθk−τk (s(k), a(k))
〉 = 〈 ∇J(θk), ( δ̂(x(k), ω ∗ k)− δ(x(k), θk) )( ψθk−τk (s(k), a(k))− ψθk−dm (s(k), a(k))
)〉 + 〈 ∇J(θk), ( δ̂(x(k), ω ∗ k)− δ(x(k), θk) ) ψθk−dm (s(k), a(k)) 〉 . (99)
We first bound the fist term in (99) as〈 ∇J(θk), ( δ̂(x(k), ω ∗ k)− δ(x(k), θk) )( ψθk−τk (s(k), a(k))− ψθk−dm (s(k), a(k)) )〉 ≥ −‖J(θk)‖2|δ̂(x(k), ω∗k)− δ(x(k), θk)|‖ψθk−τk (s(k), a(k))− ψθk−dm (s(k), a(k))‖2
≥ −‖J(θk)‖2 ( |δ̂(x(k), ω∗k)|+ |δ(x(k), θk)| ) ‖ψθk−τk (s(k), a(k))− ψθk−dm (s(k), a(k))‖2
≥ −LV ( |δ̂(x(k), ω∗k)|+ |δ(x(k), θk)| ) ‖ψθk−τk (s(k), a(k))− ψθk−dm (s(k), a(k))‖2
≥ −2LV Cδ‖ψθk−τk (s(k), a(k))− ψθk−dm (s(k), a(k))‖2 ≥ −2LV LψCδ‖θk−τk − θk−dm‖2, (100)
where the last inequality follows Assumption 3 and second last inequality follows
|δ̂(x, ω∗θ)| ≤ |r(x)|+ γ‖φ(s′)‖2‖ω∗θ‖2 + ‖φ(s)‖2‖ω∗θ‖2 ≤ rmax + (1 + γ)Rω ≤ Cδ,
|δ(x, θ)| ≤ |r(x)|+ γ|Vπθ (s′)|+ |Vπθ (s)| ≤ rmax + (1 + γ) rmax 1− γ ≤ Cδ.
Substituting (100) into (99) gives〈 ∇J(θk), ( δ̂(x(k), ω ∗ k)− δ(x(k), θk) ) ψθk−τk (s(k), a(k)) 〉 ≥ −2LV LψCδ‖θk−τk − θk−dm‖2 + 〈 ∇J(θk), ( δ̂(x(k), ω ∗ k)− δ(x(k), θk) ) ψθk−dm (s(k), a(k)) 〉 .
(101)
Then we start to bound the second term in (101). For brevity, we define ∆2(x, θ) := 〈 ∇J(θ), ( δ̂(x, ω∗θ)− δ(x, θ) ) ψθk−dm (s, a) 〉 .
In the following proof, we use θ, θ′, ω∗θ , ω ∗ θ′ , x and x̃ as shorthand notations for θk, θk−dm , ω ∗ k, ω∗k−dm , xt and x̃t respectively. We also define x := (s, a, s ′), where s ∼ µθ′ , a ∼ πθ′ and s′ ∼ P̃ .
We decompose the second term in (101) as
∆2(x, θ) = ∆2(x, θ)−∆2(x, θ′) I1 + ∆2(x, θ ′)−∆2(x̃, θ′) I2 + ∆2(x̃, θ ′)−∆2(x, θ′) I3 + ∆2(x, θ ′) I4 .
We bound the term I1 as I1 = 〈 ∇J(θ), ( δ̂(x, ω∗θ)− δ(x, θ) ) ψθ′(s, a) 〉 − 〈 ∇J(θ′), ( δ̂(x, ω∗θ′)− δ(x, θ′) ) ψθ′(s, a) 〉 = 〈 ∇J(θ), ( δ̂(x, ω∗θ)− δ(x, θ) ) ψθ′(s, a) 〉 − 〈 ∇J(θ′), ( δ̂(x, ω∗θ)− δ(x, θ) ) ψθ′(s, a)
〉 + 〈 ∇J(θ′), ( δ̂(x, ω∗θ)− δ(x, θ) ) ψθ′(s, a) 〉 − 〈 ∇J(θ′), ( δ̂(x, ω∗θ′)− δ(x, θ′) ) ψθ′(s, a) 〉 .
For the first term in I1, we have〈 ∇J(θ), ( δ̂(x, ω∗θ)− δ(x, θ) ) ψθ′(s, a) 〉 − 〈 ∇J(θ′), ( δ̂(x, ω∗θ)− δ(x, θ) ) ψθ′(s, a) 〉 = 〈 ∇J(θ)−∇J(θ′), ( δ̂(x, ω∗θ)− δ(x, θ) ) ψθ′(s, a)
〉 ≥ −‖∇J(θ)−∇J(θ′)‖2‖δ̂(x, ω∗θ)− δ(x, θ)‖2‖ψθ′(s, a)‖2 ≥ −2CδCψ‖∇J(θ)−∇J(θ′)‖2 ≥ −2CδCψLJ‖θ − θ′‖2,
where the last inequality is due to the LJ -Lipschitz of policy gradient shown in Proposition 1.
For the second term in I1, we have〈 ∇J(θ′), ( δ̂(x, ω∗θ)− δ(x, θ) ) ψθ′(s, a) 〉 − 〈 ∇J(θ′), ( δ̂(x, ω∗θ′)− δ(x, θ′) ) ψθ′(s, a) 〉 = 〈 ∇J(θ′), ( δ̂(x, ω∗θ)− δ̂(x, ω∗θ′) + δ(x, θ′)− δ(x, θ) ) ψθ′(s, a)
〉 ≥ −LV Cψ
∣∣∣δ̂(x, ω∗θ)− δ̂(x, ω∗θ′) + δ(x, θ′)− δ(x, θ)∣∣∣ ≥ −LV Cψ
∣∣γφ(s′)>(ω∗θ − ω∗θ′) + φ(s)>(ω∗θ′ − ω∗θ) + γVπθ′ (s′)− γVπθ (s′) + Vπθ (s)− Vπθ′ (s)∣∣ ≥ −LV Cψ ( γ‖ω∗θ − ω∗θ′‖2 + ‖ω∗θ′ − ω∗θ‖2 + γ|Vπθ′ (s ′)− Vπθ (s′)|+ |Vπθ (s)− Vπθ′ (s)| ) ≥ −LV Cψ (γLω‖θ − θ′‖2 + Lω‖θ − θ′‖2 + γLV ‖θ − θ′‖2 + LV ‖θ − θ′‖2) = −LV Cψ(Lω + LV )(1 + γ)‖θ − θ′‖2,
where the last inequality is due to the Lω-Lipschitz continuity of ω∗θ shown in Proposition 2 and LV -Lipschitz continuity of Vπθ (s) shown in Lemma 3. Collecting the upper bounds of I1 yields
I1 ≥ − (2CδCψLJ + LV Cψ(Lω + LV )(1 + γ)) ‖θ − θ′‖2.
First we bound I2 as
E[I2|θ′, st−m+1] = E [∆2(x, θ′)−∆2(x̃, θ′)|θ′, st−m+1] ≥ − |E [∆2(x, θ′)| θ′, st−m+1]− E [∆2(x̃, θ′)| θ′, st−m+1]| ≥ − sup
x |∆2(x, θ′)| ‖P(x ∈ ·|θ′, st−m+1)− P(x̃ ∈ ·|θ′, st−m+1)‖TV
≥ −4LV CψCδdTV (P(x ∈ ·|θ′, st−m+1),P(x̃ ∈ ·|θ′, st−m+1)) ≥ −2LV CψCδ|A|Lπ dm∑ i=τk E [‖θk−i − θk−dm‖2| θ′, st−m+1] , (102)
where the second inequality is due to the definition of TV norm, the last inequality follows (22) in Lemma 2, and the second last inequality follows the fact that
|∆2(x, θ′)| ≤ ‖∇J(θ′)‖2|δ̂(x, ω∗θ′)− δ(x, θ′)|‖ψθ′(s, a)‖2 ≤ 2LV CδCψ. (103)
Taking total expectation on both sides of (102) yields
E[I2] ≥ −2LV CψCδ|A|Lπ dm∑ i=τk E ‖θk−i − θk−dm‖2.
Next we bound I3 as
E[I3|θ′, st−m+1] = E [∆2(x̃, θ′)−∆2(x, θ′)| θ′, st−m+1] ≥ − |E [∆2(x̃, θ′)| θ′, st−m+1]− E [∆2(x, θ′)| θ′, st−m+1]| ≥ − sup
x |∆2(x, θ′)| ‖P(x̃ ∈ ·|θ′, st−m+1)− P(x ∈ ·|θ′, st−m+1)‖TV
≥ −4LV CψCδdTV ( P(x̃ ∈ ·|θ′, st−m+1), µθ′ ⊗ πθ′ ⊗ P̃ ) , (104)
where the second inequality is due to the definition of TV norm, and the last inequality follows (103).
The auxiliary Markov chain with policy πθ′ starts from initial state st−m+1, and s̃t is the (m− 1)th state on the chain. Following Lemma 1, we have:
dTV ( P(x̃ ∈ ·|θ′, st−m+1), µθ′ ⊗ πθ′ ⊗ P̃ ) =dTV ( P ((s̃t, ãt, s̃t+1) ∈ ·|θ′, st−m+1) , µθ′ ⊗ πθ′ ⊗ P̃ ) ≤ κρm−1.
Substituting the last inequality into (104) and taking total expectation on both sides yield
E[I3] ≥ −4LV CψCδκρm−1
We bound I4 as E[I4|θ′] = E [〈 ∇J(θ′), ( δ̂(x, ω∗θ′)− δ(x, θ′) ) ψθ′(s, a) 〉∣∣∣ θ′] ≥ −Cψ‖∇J(θ′)‖2 E
[∣∣∣δ̂(x, ω∗θ′)− δ(x, θ′)∣∣∣∣∣∣ θ′] = −Cψ‖∇J(θ′)‖2 E
[∣∣γ (φ(s′)>ω∗θ′ − Vπθ′ (s′))+ Vπθ′ (s)− φ(s)>ω∗θ′ ∣∣∣∣ θ′] ≥ −Cψ‖∇J(θ′)‖2 ( γ E [ |φ(s′)>ω∗θ′ − Vπθ′ (s ′)| ∣∣ θ′]+ E [|Vπθ′ (s)− φ(s)>ω∗θ′ |∣∣ θ′])
≥ −Cψ‖∇J(θ′)‖2 ( γ √ E [ |φ(s′)>ω∗θ′ − Vπθ′ (s ′)|2 ∣∣ θ′]+√E [|Vπθ′ (s)− φ(s)>ω∗θ′ |2∣∣ θ′])
= −Cψ‖∇J(θ′)‖2 ( γ √
E s′∼µθ′ |φ(s′)>ω∗θ′ − Vπθ′ (s ′)|2 +
√ E
s∼µθ′ |Vπθ′ (s)− φ(s)>ω∗θ′ |2 ) ≥ −2Cψ‖∇J(θ′)‖2 app,
where the second last inequality follows Jensen’s inequality.
The last inequality further implies
E[I4] ≥ −2Cψ E ‖∇J(θ′)−∇J(θ) +∇J(θ)‖2 app ≥ −2Cψ app E ‖∇J(θ′)−∇J(θ)‖2 − 2Cψ app E ‖∇J(θ)‖2 ≥ −2Cψ app E ‖∇J(θ′)−∇J(θ)‖2 − 2Cψ fa E ‖∇J(θ)‖2 − 2CψLV sp ≥ −2CψLJ app E ‖θ − θ′‖2 − 2Cψ fa E ‖∇J(θ)‖2 − 2CψLV sp,
where the last inequality follows Proposition 1.
Taking total expectation on both sides of (101), and collecting lower bounds of I1, I2, I3 and I4 yield E 〈 ∇J(θk), ( δ̂(x(k), ω ∗ k)− δ(x(k), θk) ) ψθk−τk (s(k), a(k)) 〉 ≥ −D2 E ‖θk−τk − θk−dm‖2 −D3 E ‖θk − θk−dm‖2 −D4
dm∑ i=τk E ‖θk−i − θk−dm‖2
−D5κρm−1 − 2CψLV fa − 2Cψ sp E ‖∇J(θk)‖2, where D2 := 2LV LψCδ, D3 := (2CδCψLJ + LV Cψ(Lω + LV )(1 + γ) + 2CψLJ app), D4 := 2LV CψCδ|A|Lπ and D5 := 4LV CψCδ . Lemma 6. For any m ≥ 1 and k ≥ (K0 + 1)m+K0 + 1, we have
E 〈 ∇J(θk), δ(x(k), θk)ψθk−τk (s(k), a(k))−∇J(θk) 〉 ≥ −D6 E ‖θk−τk − θk−dm‖2 −D7 E ‖θk − θk−dm‖2−D8 dm∑ i=τk E ‖θk−i − θk−dm‖2−D9κρm−1 − 8Cψrmax(1− γ)E ‖∇J(θk)‖2 ,
where D6 := LV CδLψ, D7 := CpLJ + (1 + γ)L2V Cψ + 2LV LJ + 8CψrmaxLJ(1 − γ), D8 := LV (Cp + LV )|A|Lπ , D9 := 2LV (Cp + LV ).
Proof. For the worker that contributes to the kth update, we construct its Markov chain:
st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−1−−−−−−→ at−m+1 · · · st−1 θk−d1−−−−→ at−1 P̃−→ st θk−d0−−−−→ at P̃−→ st+1,
where (st, at, st+1) = (s(k), a(k), s′(k)), and {dj} m j=0 is some increasing sequence with d0 := τk. By (92) in Lemma 4, we have dm ≤ (K0 + 1)m+K0. Given (st−m, at−m, st−m+1) and θk−dm , we construct an auxiliary Markov chain:
st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−−−−→ ãt−m+1 · · · s̃t−1 θk−dm−−−−→ ãt−1 P̃−→ s̃t θk−dm−−−−→ ãt P̃−→ s̃t+1.
First we have 〈 ∇J(θk), δ(x(k), θk)ψθk−τk (s(k), a(k))−∇J(θk) 〉 = 〈 ∇J(θk), δ(x(k), θk) ( ψθk−τk (s(k), a(k))− ψθk−dm (s(k), a(k))
)〉 + 〈 ∇J(θk), δ(x(k), θk)ψθk−dm (s(k), a(k))−∇J(θk) 〉 . (105)
We bound the first term in (105) as〈 ∇J(θk), δ(x(k), θk) ( ψθk−τk (s(k), a(k))− ψθk−dm (s(k), a(k)) )〉 ≥ −‖∇J(θk)‖2 ‖δ(x(k), θk)‖2‖ψθk−τk (s(k), a(k))− ψθk−dm (s(k), a(k))‖2 ≥ −LV ‖δ(x(k), θk)‖2‖ψθk−τk (s(k), a(k))− ψθk−dm (s(k), a(k))‖2 ≥ −LV Cδ‖ψθk−τk (s(k), a(k))− ψθk−dm (s(k), a(k))‖2 ≥ −LV CδLψ‖θk−τk − θk−dm‖2, (106)
where the last inequality follows Assumption 3, and the second last inequality follows the fact that
|δ(x, θ)| ≤ |r(x)|+ γ|Vπθ (s′)|+ |Vπθ (s)| ≤ rmax + (1 + γ) rmax 1− γ ≤ Cδ.
Substituting (106) into (105) gives〈 ∇J(θk), δ(x(k), θk)ψθk−τk (s(k), a(k))−∇J(θk) 〉 ≥ −LV CδLψ‖θk−τk − θk−dm‖2 + 〈 ∇J(θk), δ(x(k), θk)ψθk−dm (s(k), a(k))−∇J(θk) 〉 . (107)
Then we start to bound the second term in (107). For brevity, we define
∆3(x, θ) := 〈 ∇J(θ), δ(x, θ)ψθk−dm (s, a)−∇J(θ) 〉 .
Throughout the following proof, we use θ, θ′, x and x̃ as shorthand notations of θk, θk−dm , xt and x̃t respectively.
We decompose ∆3(x, θ) as
∆3(x, θ) = ∆3(x, θ)−∆3(x, θ′) I1 + ∆3(x, θ ′)−∆3(x̃, θ′) I2 + ∆3(x̃, θ ′) I3 .
We first bound I1 as
|I1| = |∆3(x, θ)−∆3(x, θ′)| = ∣∣〈∇J(θ), δ(x, θ)ψθ′(s, a)〉 − ‖∇J(θ)‖22 − 〈∇J(θ′), δ(x, θ′)ψθ′(s, a)〉+ ‖∇J(θ′)‖22∣∣
≤ |〈∇J(θ), δ(x, θ)ψθ′(s, a)〉 − 〈∇J(θ′), δ(x, θ′)ψθ′(s, a)〉|+ ∣∣‖∇J(θ′)‖22 − ‖∇J(θ)‖22∣∣ ≤ |〈∇J(θ), δ(x, θ)ψθ′(s, a)〉 − 〈∇J(θ′), δ(x, θ′)ψθ′(s, a)〉|+ ‖∇J(θ′) +∇J(θ)‖2‖∇J(θ′)−∇J(θ)‖2 ≤ |〈∇J(θ), δ(x, θ)ψθ′(s, a)〉 − 〈∇J(θ′), δ(x, θ′)ψθ′(s, a)〉|+ 2LV LJ‖θ − θ′‖2, (108)
where the last equality is due to LV -Lipschitz of value function and LJ -Lipschitz of policy gradient. We bound the first term in (108) as
|〈∇J(θ), δ(x, θ)ψθ′(s, a)〉 − 〈∇J(θ′), δ(x, θ′)ψθ′(s, a)〉| ≤ |〈∇J(θ), δ(x, θ)ψθ′(s, a)〉 − 〈∇J(θ), δ(x, θ′)ψθ′(s, a)〉|
+ |〈∇J(θ), δ(x, θ′)ψθ′(s, a)〉 − 〈∇J(θ′), δ(x, θ′)ψθ′(s, a)〉| = |〈∇J(θ), (δ(x, θ)− δ(x, θ′))ψθ′(s, a)〉|+ |〈∇J(θ)−∇J(θ′), δ(x, θ′)ψθ′(s, a)〉| ≤ LV Cψ |δ(x, θ)− δ(x, θ′)|+ Cp‖∇J(θ)−∇J(θ′)‖2 = LV Cψ
∣∣γ(Vπθ (s′)− Vπθ′ (s′)) + Vπθ′ (s)− Vπθ (s)∣∣+ Cp‖∇J(θ)−∇J(θ′)‖2 ≤ LV Cψ ( γ ∣∣Vπθ (s′)− Vπθ′ (s′)∣∣+ ∣∣Vπθ′ (s)− Vπθ (s)∣∣)+ Cp‖∇J(θ)−∇J(θ′)‖2 ≤ LV Cψ (γLV ‖θ − θ′‖2 + LV ‖θ′ − θ‖) + CpLJ‖θ − θ′‖2 = ( CpLJ + (1 + γ)L 2 V Cψ ) ‖θ − θ′‖2.
Substituting the above inequality into (108) gives the lower bound of I1: I1 ≥ − ( CpLJ + (1 + γ)L 2 V Cψ + 2LV LJ ) ‖θ − θ′‖2.
First we bound I2 as
E[I2|θ′, st−m+1] = E [∆3(x, θ′)−∆3(x̃, θ′)|θ′, st−m+1] ≥ − |E [∆3(x, θ′)|θ′, st−m+1]− E [∆3(x̃, θ′)|θ′, st−m+1]| ≥ − sup
x |∆3(x, θ′)| ‖P(x ∈ ·|θ′, st−m+1)− P(x̃ ∈ ·|θ′, st−m+1)‖TV
≥ −2LV (Cp + LV )dTV (P(x ∈ ·|θ′, st−m+1),P(x̃ ∈ ·|θ′, st−m+1)) ≥ −LV (Cp + LV )|A|Lπ dm∑ i=τk E [‖θk−i − θk−dm‖2|θ′, st−m+1] , (109)
where the second inequality is due to the definition of TV norm, the last inequality is due to (22) in Lemma 2, and thesecond last inequality follows the fact that
|∆3(x, θ′)| ≤ ‖∇J(θ)‖2 ( ‖δ(x, θ)ψθk−dm (s, a)‖2 + ‖∇J(θ)‖2 ) ≤ LV (Cp + LV ). (110)
Taking total expectation on both sides of (109) yields
E[I2] ≥ −LV (Cp + LV )|A|Lπ dm∑ i=τk E ‖θk−i − θk−dm‖2.
Define x := (s, a, s′), where s ∼ dθ′ , a ∼ πθ′ and s′ ∼ P̃ . Then we have
E[I3] = E [∆3(x̃, θ′)−∆3(x, θ′)] + E [∆3(x, θ′)] . (111)
We bound the first term in (111) as
E [∆3(x̃, θ′)−∆3(x, θ′)|θ′, st−m+1] ≥ − |E [∆3(x̃, θ′)|θ′, st−m+1]− E [∆3(x, θ′)|θ′, st−m+1]| ≥ − sup
x |∆3(x, θ′)| ‖P(x̃ ∈ ·|θ′, st−m+1)− P(x ∈ ·|θ′, st−m+1)‖TV
≥ −2LV (Cp + LV )dTV (P(x̃ ∈ ·|θ′, st−m+1),P(x ∈ ·|θ′, st−m+1)) = −2LV (Cp + LV )dTV ( P(x̃ ∈ ·|θ′, st−m+1), dθ′ ⊗ πθ′ ⊗ P̃ ) = −2LV (Cp + LV )dTV ( P(x̃ ∈ ·|θ′, st−m+1), µθ′ ⊗ πθ′ ⊗ P̃ ) (112)
where the second inequality follows the definition of total variation norm, and the third inequality follows (110). The last equality is due to the fact shown by [6] that µθ′(·) = dθ′(·), where µθ′ is the stationary distribution of an artificial MDP with transition kernel P̃(·|s, a) and policy πθ′ .
The auxiliary Markov chain with policy πθ′ starts from initial state st−m+1, and s̃t is the (m− 1)th state on the chain. Following Lemma 1, we have:
dTV ( P(x̃ ∈ ·|θ′, st−m+1), µθ′ ⊗ πθ′ ⊗ P̃ ) = dTV ( P ((s̃t, ãt, s̃t+1) ∈ ·|θ′, st−m+1) , µθ′ ⊗ πθ′ ⊗ P̃ ) ≤ κρm−1.
Substituting the last inequality into (112) and taking total expectation on both sides yield
E [∆3(x̃, θ′)−∆3(x, θ′)] ≥ −2LV (Cp + LV )κρm−1.
Consider the second term in (111). Note its form is similar to (59), so by following the derivation of (63), we directly have
E[∆3(x, θ′)] = E 〈∇J(θ′), δ(x, θ′)ψθ′(s, a)−∇J(θ′)〉 ≥ −8Cψrmax(1− γ)E ‖∇J(θ′)‖2 ,
which further implies
E[∆3(x, θ′)] ≥ −8Cψrmax(1− γ)E ‖∇J(θ′)‖2 ≥ −8Cψrmax(1− γ)E ‖∇J(θ′)−∇J(θ)‖2 − 8Cψrmax(1− γ)E ‖∇J(θ)‖2 ≥ −8CψrmaxLJ(1− γ)E ‖θ′ − θ‖2−8Cψrmax(1− γ)E ‖∇J(θ)‖2
where the last inequality follows from Proposition 1.
Collecting the lower bounds gives
E[I3] ≥ −2LV (Cp + LV )κρm−1 − 8Cψrmax(1− γ) (LJ E ‖θ′ − θ‖2 − E ‖∇J(θ)‖2) .
Taking total expectation on ∆3(x, θ) and collecting lower bounds of I1, I2, I3 yield E[∆3(x, θ)] ≥ − ( CpLJ + (1 + γ)L 2 V Cψ + 2LV LJ + 8CψrmaxLJ(1− γ) ) E ‖θk − θk−dm‖2
− LV (Cp + LV )|A|Lπ dm∑ i=τk E ‖θk−i − θk−dm‖2 − 2LV (Cp + LV )κρm−1 − 8Cψrmax(1− γ)E ‖∇J(θk)‖2 .
Taking total expectation on (107) and substituting the above inequality into it yield E 〈 ∇J(θk), δ(x(k), θk)ψθk−τk (s(k), a(k))−∇J(θk) 〉 ≥ −D6 E ‖θk−τk − θk−dm‖2 −D7 E ‖θk − θk−dm‖2 −D8 dm∑ i=τk E ‖θk−i − θk−dm‖2 −D9κρm−1 − 8Cψrmax(1− γ)E ‖∇J(θk)‖2 ,
where D6 := LV CδLψ, D7 := CpLJ + (1 + γ)L2V Cψ + 2LV LJ + 8CψrmaxLJ(1 − γ), D8 := LV (Cp + LV )|A|Lπ , D9 := 2LV (Cp + LV ).
C.3 EXPLANATION OF THE APPROXIMATION ERROR
In this section, we will provide a justification for the circumstances when the approximation error app defined in (14) is small. Lemma 7. Suppose Assumption 2 and 4 hold. Then it holds that
app ≤ max θ∈Rd
√ E
s∼µθ |Vπθ (s)− V̂ω̄∗θ (s)|2 + 4rmax(λ −1 + λ−2rmax)
( 1 + logρ κ −1 + 1
1− ρ
) (1− γ)
(113)
where ω̄∗θ the critic stationary point of original Markov chain with policy πθ and transition kernel P .
In (113), the first term captures the quality of critic function parameterization method which also appears in previous works [14, 15, 17]. When using linear critic function approximation, it becomes zero when the value function Vπθ belongs to the linear function space for any θ. The second term corresponds to the error introduced by sampling from the artificial transition kernel P̃(·|s, a) = (1− γ)P(·|s, a) + γη(·). For a large γ close to 1, the artificial Markov chain is close to the original
one. In this case, the second error term is therefore small. This fact also consists with practice where large γ is commonly used in two time-scale actor critic algorithms [3].
Before going into the proof, we first define that:
Āθ,φ := E s∼µ̄θ,s′∼Pπθ [φ(s)(γφ(s′)− φ(s))>], b̄θ,φ := E s∼µ̄θ,a∼πθ,s′∼P [r(s, a, s′)φ(s)],
where µ̄θ as the stationary distribution of the original Markov chain with πθ and transition kernel P . Proof. Recall the definition of the approximation error:
app = max θ∈Rd
√ E
s∼µθ |Vπθ (s)− V̂ω∗θ (s)|2,
where µθ is the stationary distribution of the artificial Markov chain with πθ and transition kernel P̃ , and ω∗θ is the stationary point of critic update under the artificial Markov chain.
We decompose app as
app = max θ∈Rd
√ E
s∼µθ |Vπθ (s)− V̂ω̄∗θ (s) + V̂ω̄∗θ (s)− V̂ω∗θ (s)|2
≤ max θ∈Rd
√ E
s∼µθ |Vπθ (s)− V̂ω̄∗θ (s)|2
fa
+ max θ∈Rd
√ E
s∼µθ |V̂ω̄∗θ (s)− V̂ω∗θ (s)|2
sp
, (114)
where the first term corresponds to the function approximation error fa, and second term corresponds to the sampling error sp.
With A, b and Ā, b̄ as shorthand notations for Aθ,ψ, bθ,ψ and Āθ,ψ, b̄θ,ψ respectively, we bound the second term in (114) as
|V̂ω̄∗θ (s)− V̂ω∗θ (s)| = ∣∣φ(s)>ω∗θ − φ(s)>ω̄∗θ ∣∣
≤ ∥∥A−1b− Ā−1b̄∥∥
2 = ∥∥A−1b−A−1b̄+A−1b̄− Ā−1b̄∥∥
2 ≤ ∥∥A−1(b− b̄)∥∥ 2 + ∥∥(A−1 − Ā−1)b̄∥∥ 2
≤ λ−1‖b− b̄‖2 + rmax ∥∥A−1 − Ā−1∥∥
2 = λ−1‖b− b̄‖2 + rmax ∥∥A−1(Ā−A)Ā−1∥∥
2 ≤ λ−1‖b− b̄‖2 + λ−2rmax ∥∥Ā−A∥∥
2 . (115)
We bound the first term in last inequality as
‖b− b̄‖2 = ∥∥∥∥∥ Es∼µθ,a∼πθ,s′∼P̃[r(s, a, s′)φ(s)]− Es∼µ̄θ,a∼πθ,s′∼P[r(s, a, s′)φ(s)] ∥∥∥∥∥
≤ sup ‖r(s, a, s′)φ(s)‖2‖µθ ⊗ πθ ⊗ P̃ − µ̄θ ⊗ πθ ⊗ P‖TV ≤ 2rmaxdTV (µθ ⊗ πθ ⊗ P̃, µ̄θ ⊗ πθ ⊗ P). (116)
We now bound the divergence term in the last inequality as
dTV (µθ ⊗ πθ ⊗ P̃, µ̄θ ⊗ πθ ⊗ P)
= ∫ s∈S ∑ a∈A ∫ s′∈S ∣∣∣µθ(s)πθ(a|s)P̃(s′|s, a)− µ̄θ(s)πθ(a|s)P(s′|s, a)∣∣∣ =
∫ s∈S ∑ a∈A ∫ s′∈S |µθ(s)πθ(a|s)P̃(s′|s, a)− µθ(s)πθ(a|s)P(s′|s, a) + µθ(s)πθ(a|s)P(s′|s, a)− µ̄θ(s)πθ(a|s)P(s′|s, a)|
≤ ∫ s∈S ∑ a∈A µθ(s)πθ(a|s) ∫ s′∈S ∣∣∣P̃(s′|s, a)− P(s′|s, a)∣∣∣+ ∫ s∈S |µθ(s)− µ̄θ(s)| . (117)
We bound the first term in (117) as∫ s′∈S ∣∣∣P̃(s′|s, a)− P(s′|s, a)∣∣∣ = (1− γ)∫ s′∈S |P(s′|s, a)− η(s′)| ≤ 2(1− γ). (118) Following [39, Theorem 3.1], the second term in (117) can be bounded as∫ s∈S |µθ(s)− µ̄θ(s)| ≤ ( logρ κ −1 + 1 1− ρ ) sup s ∫ s′∈S ∣∣∣∣∣∑ a πθ(a|s) ( P̃(s′|s, a)− P(s′|s, a)
)∣∣∣∣∣ ≤ (
logρ κ −1 +
1
1− ρ ) sup s ∑ a πθ(a|s) ∫ s′∈S ∣∣∣P̃(s′|s, a)− P(s′|s, a)∣∣∣ ≤ 2 ( logρ κ −1 + 1
1− ρ
) (1− γ), (119)
where the last inequality follows (118).
Substituting (118) and (119) into (117) gives dTV (µθ ⊗ πθ ⊗ P̃, µ̄θ ⊗ πθ ⊗ P) ≤ 2 ( 1 + logρ κ −1 + 1
1− ρ
) (1− γ).
Substituting the above inequality into (116) gives ‖b− b̄‖2 ≤ 4rmax ( 1 + logρ κ −1 + 1
1− ρ
) (1− γ). (120)
Similarly, we also have ‖A− Ā‖2 ≤ 4rmax ( 1 + logρ κ −1 + 1
1− ρ
) (1− γ). (121)
Substituting (120) and (121) into (115), then substituting (115) into (114) completes the proof.
D EXPERIMENT DETAILS
Hardware device. The tests on synthetic environment and CartPole was performed in a 16-core CPU computer. The test on Atari game was run in a 4 GPU computer.
Parameterization. For the synthetic environment, we used linear value function approximation and tabular softmax policy [36]. For CartPole, we used a 3-layer MLP with 128 neurons and sigmoid activation function in each layer. The first two layers are shared for both actor and critic network. For the Atari seaquest game, we used a convolution-LSTM network. For network details, see [40].
Hyper-parameters. For the synthetic environment tests, we run Algorithm 1 with actor step size αk = 0.05 (1+k)0.6 and critic step size βk = 0.05 (1+k)0.4 . In tests of CartPole, we run Algorithm 1 with a minibatch of 20 samples. We update the actor network with a step size of αk = 0.01(1+k)0.6 and critic network with a step size of βk = 0.01(1+k)0.4 . See Table 1 for hyper-parameters to generate the Atari game results in Figure 4.

Review summaries:
This paper studied the two time scale A3C in discounted MDP based on recent development in the finite sample analysis of A2C. The sample complexity result in this paper matches previous result in two time-scale A2C in terms of the dependence of \epsilon, and this paper further shows the benefit of "linear speed up" brough by the structure of A3C. Given the practical usefulness of A3C, the result established in this paper is meaninful.

This paper revisits the A3C algorithm with TD(0) for the critic update to provide better theoretical analysis of A3C. A3C-TD(0) achieves linear speedup and it also matches our intuition. To show the empirical results, the authors provide convergence results of A3C-TD(0) with Markovian sampling in synthetic environments and speedup of A3C-TD(0) in CartPole and Seaquest.