Asynchronous Advantage Actor Critic: Non-asymptotic Analysis and Linear Speedup

Asynchronous and parallel implementation of standard reinforcement learning ( RL ) algorithms is a key enabler of the tremendous success of modern RL . Among many asynchronous RL algorithms , arguably the most popular and effective one is the asynchronous advantage actor-critic ( A3C ) algorithm . Although A3C is becoming the workhorse of RL , its theoretical properties are still not well-understood , including the non-asymptotic analysis and the performance gain of parallelism ( a.k.a . speedup ) . This paper revisits the A3C algorithm with TD ( 0 ) for the critic update , termed A3C-TD ( 0 ) , with provable convergence guarantees . With linear value function approximation for the TD update , the convergence of A3C-TD ( 0 ) is established under both i.i.d . and Markovian sampling . Under i.i.d . sampling , A3CTD ( 0 ) obtains sample complexity ofO ( −2.5/N ) per worker to achieve accuracy , where N is the number of workers . Compared to the best-known sample complexity of O ( −2.5 ) for two-timescale AC , A3C-TD ( 0 ) achieves linear speedup , which justifies the advantage of parallelism and asynchrony in AC algorithms theoretically for the first time . Numerical tests on synthetically generated instances and OpenAI Gym environments have been provided to verify our theoretical analysis . 1 INTRODUCTION Reinforcement learning ( RL ) has achieved impressive performance in many domains such as robotics [ 1 , 2 ] and video games [ 3 ] . However , these empirical successes are often at the expense of significant computation . To unlock high computation capabilities , the state-of-the-art RL approaches rely on sampling data from massive parallel simulators on multiple machines [ 3 , 4 , 5 ] . Empirically , these approaches can stabilize the learning processes and reduce training time when they are implemented in an asynchronous manner . One popular RL method that often achieves the best empirical performance is the asynchronous variant of the actor-critic ( AC ) algorithm , which is referred to as A3C [ 3 ] . A3C builds on the original AC algorithm [ 6 ] . At a high level , AC simultaneously performs policy optimization ( a.k.a . the actor step ) using the policy gradient method [ 7 ] and policy evaluation ( a.k.a . the critic step ) using the temporal difference learning ( TD ) algorithm [ 8 ] . To ensure scalability , both actor and critic steps can combine with various function approximation techniques . To ensure stability , AC is often implemented in a two time-scale fashion , where the actor step runs in the slow timescale and the critic step runs in the fast timescale . Similar to other on-policy RL algorithms , AC uses samples generated from the target policy . Thus , data sampling is entangled with the learning procedure , which generates significant overhead . To speed up the sampling process of AC , A3C introduces multiple workers with a shared policy , and each learner has its own simulator to perform data sampling . The shared policy can be then updated using samples collected from multiple learners . Despite the tremendous empirical success achieved by A3C , to the best of our knowledge , its theoretical property is not well-understood . The following theoretical questions remain unclear : Q1 ) Under what assumption does A3C converge ? Q2 ) What is its convergence rate ? Q3 ) Can A3C obtain benefit ( or speedup ) using parallelism and asynchrony ? For Q3 ) , we are interested in the training time linear speedup with N workers , which is the ratio between the training time using a single worker and that using N workers . Since asynchronous parallelism mitigates the effect of stragglers and keeps all workers busy , the training time speedup can be measured roughly by the sample ( i.e. , computational ) complexity linear speedup [ 9 ] , given by Speedup ( N ) = sample complexity when using one worker average sample complexity per worker when using N workers . ( 1 ) If Speedup ( N ) = Θ ( N ) , the speedup is linear , and the training time roughly reduces linearly as the number of workers increases . This paper aims to answer these questions , towards the goal of providing theoretical justification for the empirical successes of parallel and asynchronous RL . 1.1 RELATED WORKS Analysis of actor critic algorithms . AC method was first proposed by [ 6 , 10 ] , with asymptotic convergence guarantees provided in [ 6 , 10 , 11 ] . It was not until recently that the non-asymptotic analyses of AC have been established . The finite-sample guarantee for the batch AC algorithm has been established in [ 12 , 13 ] with i.i.d . sampling . Later , in [ 14 ] , the finite-sample analysis was established for the double-loop nested AC algorithm under the Markovian setting . An improved analysis for the Markovian setting with minibatch updates has been presented in [ 15 ] for the nested AC method . More recently , [ 16 , 17 ] have provided the first finite-time analyses for the two-timescale AC algorithms under Markov sampling , with both Õ ( −2.5 ) sample complexity , which is the bestknown sample complexity for two-timescale AC . Through the lens of bi-level optimization , [ 18 ] has also provided finite-sample guarantees for this two-timescale Markov sampling setting , with global optimality guarantees when a natural policy gradient step is used in the actor . However , none of the existing works has analyzed the effect of the asynchronous and parallel updates in AC . Empirical parallel and distributed AC . In [ 3 ] , the original A3C method was proposed and became the workhorse in empirical RL . Later , [ 19 ] has provided a GPU-version of A3C which significantly decreases training time . Recently , the A3C algorithm is further optimized in modern computers by [ 20 ] , where a large batch variant of A3C with improved efficiency is also proposed . In [ 21 ] , an importance weighted distributed AC algorithm IMPALA has been developed to solve a collection of problems with one single set of parameters . Recently , a gossip-based distributed yet synchronous AC algorithm has been proposed in [ 5 ] , which has achieved the performance competitive to A3C . Asynchronous stochastic optimization . For solving general optimization problems , asynchronous stochastic methods have received much attention recently . The study of asynchronous stochastic methods can be traced back to 1980s [ 22 ] . With the batch size M , [ 23 ] analyzed asynchronous SGD ( async-SGD ) for convex functions , and derived a convergence rate of O ( K− 12M− 12 ) if delay K0 is bounded by O ( K 1 4M− 3 4 ) . This result implies linear speedup . [ 24 ] extended the analysis of [ 23 ] to smooth convex with nonsmooth regularization and derived a similar rate . Recent studies by [ 25 ] improved upper bound of K0 to O ( K 1 2M− 1 2 ) . However , all these works have focused on the single-timescale SGD with a single variable , which can not capture the stochastic recursion of the AC and A3C algorithms . To best of our knowledge , non-asymptotic analysis of asynchronous two-timescale SGD has remained unaddressed , and its speedup analysis is even an uncharted territory . 1.2 THIS WORK In this context , we revisit A3C with TD ( 0 ) for the critic update , termed A3C-TD ( 0 ) . The hope is to provide non-asymptotic guarantee and linear speedup justification for this popular algorithm . Our contributions . Compared to the existing literature on both the AC algorithms and the asyncSGD , our contributions can be summarized as follows . c1 ) We revisit two-timescale A3C-TD ( 0 ) and establish its convergence rates with both i.i.d . and Markovian sampling . To the best of our knowledge , this is the first non-asymptotic convergence result for asynchronous parallel AC algorithms . c2 ) We characterize the sample complexity of A3C-TD ( 0 ) . In i.i.d . setting , A3C-TD ( 0 ) achieves a sample complexity of O ( −2.5/N ) per worker , where N is the number of workers . Compared to the best-known complexity of O ( −2.5 ) for i.i.d . two-timescale AC [ 18 ] , A3C-TD ( 0 ) achieves linear speedup , thanks to the parallelism and asynchrony . In the Markovian setting , if delay is bounded , the sample complexity of A3C-TD ( 0 ) matches the order of the non-parallel AC algorithm [ 17 ] . c3 ) We test A3C-TD ( 0 ) on the synthetically generated environment to verify our theoretical guarantees with both i.i.d . and Markovian sampling . We also test A3C-TD ( 0 ) on the classic control tasks and Atari Games from OpenAI Gym . Code is available in the supplementary material . Technical challenges . Compared to the recent convergence analysis of nonparallel two-timescale AC in [ 16 , 17 , 18 ] , several new challenges arise due to the parallelism and asynchrony . Markovian noise coupled with asynchrony and delay . The analysis of two-timescale AC algorithm is non-trivial because of the Markovian noise coupled with both the actor and critic steps . Different from the nonparallel AC that only involves a single Markov chain , asynchronous parallel AC introduces multiple Markov chains ( one per worker ) that mix at different speed . This is because at a given iteration , workers collect different number of samples and thus their chains mix to different degrees . As we will show later , the worker with the slowest mixing chain will determine the convergence . Linear speedup for SGD with two coupled sequences . Parallel async-SGD has been shown to achieve linear speedup recently [ 9 , 26 ] . Different from async-SGD , asynchronous AC is a two-timescale stochastic semi-gradient algorithm for solving the more challenging bilevel optimization problem ( see [ 18 ] ) . The errors induced by asynchrony and delay are intertwined with both actor and critic updates via a nested structure , which makes the sharp analysis more challenging . Our linear speedup analysis should be also distinguished from that of mini-batch async-SGD [ 27 ] , where the speedup is a result of variance reduction thanks to the larger batch size generated by parallel workers . 2 PRELIMINARIES 2.1 MARKOV DECISION PROCESS AND POLICY GRADIENT THEOREM RL problems are often modeled as an MDP described byM = { S , A , P , r , γ } , where S is the state space , A is the action space , P ( s′|s , a ) is the probability of transitioning to s′ ∈ S given current state s ∈ S and action a ∈ A , and r ( s , a , s′ ) is the reward associated with the transition ( s , a , s′ ) , and γ ∈ [ 0 , 1 ) is a discount factor . Throughout the paper , we assume the reward r is upper-bounded by a constant rmax . A policy π : S → ∆ ( A ) is defined as a mapping from the state space S to the probability distribution over the action space A . Considering discrete time t in an infinite horizon , a policy π can generate a trajectory of state-action pairs ( s0 , a0 , s1 , a1 , . . . ) with at ∼ π ( ·|st ) and st+1 ∼ P ( ·|st , at ) . Given a policy π , we define the state and state action value functions as Vπ ( s ) : = E [ ∞∑ t=0 γtr ( st , at , st+1 ) | s0 = s ] , Qπ ( s , a ) : = E [ ∞∑ t=0 γtr ( st , at , st+1 ) | s0 = s , a0 = a ] ( 2 ) where E is taken over the trajectory ( s0 , a0 , s1 , a1 , . . . ) generated under policy π . With the above definitions , the advantage function is Aπ ( s , a ) : = Qπ ( s , a ) − Vπ ( s ) . With η denoting the initial state distribution , the discounted state visitation measure induced by policy π is defined as dπ ( s ) : = ( 1 − γ ) ∑∞ t=0 γ tP ( st = s | s0 ∼ η , π ) , and the discounted state action visitation measure is d′π ( s , a ) = ( 1− γ ) ∑∞ t=0 γ tP ( st = s | s0 ∼ η , π ) π ( a|s ) . The goal of RL is to find a policy that maximizes the expected accumulative reward J ( π ) : = Es∼η [ Vπ ( s ) ] . When the state and action spaces are large , finding the optimal policy π becomes computationally intractable . To overcome the inherent difficulty of learning a function , the policy gradient methods search the best performing policy over a class of parameterized policies . We parameterize the policy with parameter θ ∈ Rd , and solve the optimization problem as max θ∈Rd J ( θ ) with J ( θ ) : = E s∼η [ Vπθ ( s ) ] . ( 3 ) To maximize J ( θ ) with respect to θ , one can update θ using the policy gradient direction given by [ 7 ] ∇J ( θ ) = E s , a∼d′ θ [ Aπθ ( s , a ) ψθ ( s , a ) ] , ( 4 ) where ψθ ( s , a ) : = ∇ log πθ ( a|s ) , and d′θ : = ( 1 − γ ) ∑∞ t=0 γ tP ( st = s | s0 , πθ ) πθ ( a|s ) . Since computing E in ( 4 ) is expensive if not impossible , popular policy gradient-based algorithms iteratively update θ using stochastic estimate of ( 4 ) such as REINFORCE [ 28 ] and G ( PO ) MDP [ 29 ] . 2.2 ACTOR-CRITIC ALGORITHM WITH VALUE FUNCTION APPROXIMATION Both REINFORCE and G ( PO ) MDP-based policy gradient algorithms rely on a Monte-Carlo estimate of the value function Vπθ ( s ) and thus ∇J ( θ ) by generating a trajectory per iteration . However , policy gradient methods based on Monte-Carlo estimate typically suffer from high variance and large sampling cost . An alternative way is to recursively refine the estimate of Vπθ ( s ) . For a policy πθ , it is known that Vπθ ( s ) satisfies the Bellman equation [ 30 ] , that is Vπθ ( s ) = E a∼πθ ( ·|s ) , s′∼P ( ·|s , a ) [ r ( s , a , s′ ) + γVπθ ( s ′ ) ] , ∀s ∈ S. ( 5 ) In practice , when the state space S is prohibitively large , one can not afford the computational and memory complexity of computing Vπθ ( s ) and Aπθ ( s , a ) . To overcome this curse-of-dimensionality , a popular method is to approximate the value function using function approximation techniques . Given the state feature mapping φ ( · ) : S −→ Rd′ for some d′ > 0 , we approximate the value function linearly as Vπθ ( s ) ≈ V̂ω ( s ) : = φ ( s ) > ω , where ω ∈ Rd ′ is the critic parameter . Given a policy πθ , the task of finding the best ω such that Vπθ ( s ) ≈ V̂ω ( s ) is usually addressed by TD learning [ 8 ] . Defining the kth transition as xk : = ( sk , ak , sk+1 ) and the corresponding TD-error as δ̂ ( xk , ωk ) : = r ( sk , ak , sk+1 ) + γφ ( sk+1 ) > ωk − φ ( sk ) > ωk , the parameter ω is updated via ωk+1 = ΠRω ( ωk + βkg ( xk , ωk ) ) with g ( xk , ωk ) : = δ̂ ( xk , ωk ) ∇ωk V̂ωk ( sk ) ( 6 ) where βk is the critic stepsize , and ΠRω is the projection with Rω being a pre-defined constant . The projection step is often used to control the norm of the gradient . In AC , it prevents the actor and critic updates from going a too large step in the ‘ wrong ’ direction ; see e.g. , [ 6 , 16 , 17 ] . Using the definition of advantage function Aπθ ( s , a ) = Es′∼P [ r ( s , a , s′ ) + γVπθ ( s′ ) ] − Vπθ ( s ) , we can also rewrite ( 4 ) as ∇J ( θ ) = Es , a∼d′θ , s′∼P [ ( r ( s , a , s ′ ) + γVπθ ( s ′ ) − Vπθ ( s ) ) ψθ ( s , a ) ] . Leveraging the value function approximation , we can then approximate the policy gradient as ∇̂J ( θ ) = ( r ( s , a , s′ ) + γV̂ω ( s ′ ) − V̂ω ( s ) ) ψθ ( s , a ) = δ̂ ( x , ω ) ψθ ( s , a ) ( 7 ) which gives rise to the policy update θk+1 = θk + αkv ( xk , θk , ωk ) with v ( xk , θk , ωk ) : = δ̂ ( xk , ωk ) ψθk ( sk , ak ) ( 8 ) where αk is the stepsize for the actor update . To ensure convergence when simultaneously performing critic and actor updates , the stepsizes αk and βk often decay at two different rates , which is referred to the two-timescale AC [ 17 , 18 ] . 3 ASYNCHRONOUS ADVANTAGE ACTOR CRITIC WITH TD ( 0 ) To speed up the training process , we implement AC over N workers in a shared memory setting without coordinating among workers — a setting similar to that in A3C [ 3 ] . Each worker has its own simulator to perform sampling , and then collaboratively updates the shared policy πθ using AC updates . As there is no synchronization after each update , the policy used by workers to generate samples may be outdated , which introduces staleness . Notations on transition ( s , a , s′ ) . Since each worker will maintain a separate Markov chain , we thereafter use subscription t in ( st , at , st+1 ) to indicate the tth transition on a Markov chain . We use k to denote the global counter ( or iteration ) , which increases by one whenever a worker finishes the actor and critic updates in the shared memory . We use subscription ( k ) in ( s ( k ) , a ( k ) , s′ ( k ) ) to indicate the transition used in the kth update . Specifically , we initialize θ0 , ω0 in the shared memory . Each worker will initialize the simulator with initial state s0 . Without coordination , workers will read θ , ω in the shared memory . From each worker ’ s view , it then generates sample ( st , at , st+1 ) by either sampling st from µθ ( · ) , where µθ ( · ) is the stationary distribution of an artificial MDP with transition probability measure P̃ ( ·|st , at ) : = γP ( ·|st , at ) + ( 1 − γ ) η ( · ) , or , sampling st from a Markov chain under policy πθ . In both cases , each worker obtains at ∼ πθ ( ·|st ) and st+1 ∼ P̃ ( ·|st , at ) . Sampling st+1 from P̃ ( ·|st , at ) can be achieved by sampling st+1 from η ( · ) with probability 1− γ and from P ( ·|st , at ) otherwise . Once Algorithm 1 Asynchronous advantage AC with TD ( 0 ) : each worker ’ s view . 1 : Global initialize : Global counter k = 0 , initial θ0 , ω0 in the shared memory . 2 : Worker initialize : Local counter t = 0 . Obtain initial state s0 . 3 : for t = 0 , 1 , 2 , · · · do 4 : Read θ , ω in the shared memory . 5 : Option 1 ( i.i.d . sampling ) : 6 : Sample st ∼ µθ ( · ) , at ∼ πθ ( ·|s ) , st+1 ∼ P̃ ( ·|st , at ) . 7 : Option 2 ( Markovian sampling ) : 8 : Sample at ∼ πθ ( ·|st ) , st+1 ∼ P̃ ( ·|st , at ) . 9 : Compute δ̂ ( xt , ω ) = r ( st , at , st+1 ) + γV̂ω ( st+1 ) − V̂ω ( st ) . 10 : Compute g ( xt , ω ) = δ̂ ( xt , ω ) ∇ωV̂ω ( st ) . 11 : Compute v ( xt , θ , ω ) = δ̂ ( xt , ω ) ψθ ( st , at ) . 12 : In the shared memory , perform update ( 9 ) . 13 : end for obtaining xt : = ( st , at , st+1 ) , each worker locally computes the policy gradient v ( xt , θ , ω ) and the TD ( 0 ) update g ( xt , ω ) , and then updates the parameters in shared memory asynchronously by ωk+1 = ΠRω ( ωk + βkg ( x ( k ) , ωk−τk ) ) , ( 9a ) θk+1 = θk + αkv ( x ( k ) , θk−τk , ωk−τk ) , ( 9b ) where τk is the delay in the kth actor and critic updates . See the A3C with TD ( 0 ) in Algorithm 1 . Sampling distributions . Since the transition kernel required by the actor and critic updates are different in the discounted MDP , it is difficult to design a two-timescale AC algorithm . To address this issue , we adopt the sampling method introduced in the seminal work [ 6 , 31 ] and the recent work [ 15 , 16 ] , which inevitably introduces bias by sampling from the artificial transition P̃ instead of P . However , as we will mention later , this extra bias is small when the discount factor γ is close to 1 . Parallel sampling . The AC update ( 6 ) and ( 8 ) uses samples generated “ on-the-fly ” from the target policy πθ , which brings overhead . Compared with ( 6 ) and ( 8 ) , the A3C-TD ( 0 ) update ( 9 ) allows parallel sampling from N workers , which is the key to linear speedup . We consider the case where only one worker can update parameters in the shared memory at the same time and the update can not be interrupted . In practice , ( 9 ) can also be performed in a mini-batch fashion . Minor differences from A3C [ 3 ] . The A3C-TD ( 0 ) algorithm resembles the popular A3C method [ 3 ] . With nmax denoting the horizon of steps , for n ∈ { 1 , ... , nmax } , A3C iteratively uses n-step TD errors to compute actor and critic gradients . In A3C-TD ( 0 ) , we use the TD ( 0 ) method which is the 1-step TD method for actor and critic update . When nmax = 1 , A3C method reduces to A3C-TD ( 0 ) . The n-step TD method is a hybrid version of the TD ( 0 ) method and the Monte-Carlo method . The A3C method with Monte-Carlo sampling is essentially the delayed policy gradient method , and thus its convergence follows directly from the delayed SGD . Therefore , we believe that the convergence of the A3C method based on TD ( 0 ) in this paper can be easily extended to the convergence of the A3C method with n-step TD . We here focus on A3C with TD ( 0 ) just for ease of exposition . 4 CONVERGENCE ANALYSIS OF TWO-TIMESCALE A3C-TD ( 0 ) In this section , we analyze the convergence of A3C-TD ( 0 ) in both i.i.d . and Markovian settings . Throughout this section , the notation O ( · ) contains constants that are independent of N and K0 . To analyze the performance of A3C-TD ( 0 ) , we make the following assumptions . Assumption 1 . There exists K0 such that the delay at each iteration is bounded by τk ≤ K0 , ∀k . Assumption 1 ensures the viability of analyzing the asynchronous update ; see the same assumption in e.g. , [ 5 , 25 ] . In practice , the delay usually scales as the number of workers , that is K0 = Θ ( N ) . With P̃πθ ( s′|s ) = ∑ a∈A P̃ ( s′|s , a ) πθ ( a|s ) , we define that : Aθ , φ : = E s∼µθ , s′∼P̃πθ [ φ ( s ) ( γφ ( s′ ) − φ ( s ) ) > ] , bθ , φ : = E s∼µθ , a∼πθ , s′∼P̃ [ r ( s , a , s′ ) φ ( s ) ] . ( 10 ) It is known that for a given θ , the stationary point ω∗θ of the TD ( 0 ) update in Algorithm 1 satisfies Aθ , φω ∗ θ + bθ , φ = 0 . ( 11 ) Assumption 2 . For all s ∈ S , the feature vector φ ( s ) is normalized so that ‖φ ( s ) ‖2 ≤ 1 . For all γ ∈ [ 0 , 1 ] and θ ∈ Rd , Aθ , φ is negative definite and its max eigenvalue is upper bounded by −λ . Assumption 2 is common in analyzing TD with linear function approximation ; see e.g. , [ 17 , 32 , 33 ] . With this assumption , Aθ , φ is invertible , so we have ω∗θ = −A −1 θ , φbθ , φ . Define Rω : = rmax/λ , then we have ‖ω∗θ‖2 ≤ Rω . It justifies the projection introduced in Algorithm 1 . In practice , the projection radius Rω can be estimated online by methods proposed in [ 32 , Section 8.2 ] or [ 34 , Lemma 1 ] . Assumption 3 . For any θ , θ′ ∈ Rd , s ∈ S and a ∈ A , there exist constants such that : i ) ‖ψθ ( s , a ) ‖2 ≤ Cψ ; ii ) ‖ψθ ( s , a ) − ψθ′ ( s , a ) ‖2 ≤ Lψ‖θ − θ′‖2 ; iii ) |πθ ( a|s ) − πθ′ ( a|s ) | ≤ Lπ‖θ − θ′‖2 . Assumption 3 is common in analyzing policy gradient-type algorithms which has also been made by e.g. , [ 34 , 35 , 36 ] . This assumption holds for many policy parameterization methods such as tabular softmax policy [ 36 ] , Gaussian policy [ 37 ] and Boltzmann policy [ 31 ] . Assumption 4 . For any θ ∈ Rd , the Markov chain under policy πθ and transition kernel P ( ·|s , a ) or P̃ ( ·|s , a ) is irreducible and aperiodic . Then there exist constants κ > 0 and ρ ∈ ( 0 , 1 ) such that sup s∈S dTV ( P ( st ∈ ·|s0 = s , πθ ) , µθ ) ≤ κρt , ∀t ( 12 ) where µθ is the stationary state distribution under πθ , and st is the state of Markov chain at time t. Assumption 4 assumes the Markov chain mixes at a geometric rate ; see also [ 32 , 33 ] . The stationary distribution µθ of an artificial Markov chain with transition P̃ is the same as the discounted visitation measure dθ of the Markov chain with transition P [ 6 ] . This means that if we sample according to at ∼ πθ ( ·|st ) , st+1 ∼ P̃ ( ·|st , at ) , the marginal distribution of ( st , at ) will converge to the discounted state-action visitation measure d′θ ( s , a ) , which allows us to control the gradient bias . 4.1 LINEAR SPEEDUP RESULT WITH I.I.D . SAMPLING In this section , we consider A3C-TD ( 0 ) under the i.i.d . sampling setting , which is widely used for analyzing RL algorithms ; see e.g. , [ 13 , 18 , 38 ] . We first give the convergence result of critic update as follows . Theorem 1 ( Critic convergence ) . Suppose Assumptions 1–4 hold . Consider Algorithm 1 with i.i.d . sampling and V̂ω ( s ) = φ ( s ) > ω . Select step size αk = c1 ( 1+k ) σ1 , βk = c2 ( 1+k ) σ2 , where 0 < σ2 < σ1 < 1 and c1 , c2 are positive constants . Then it holds that 1 K K∑ k=1 E ∥∥ωk − ω∗θk∥∥22=O ( 1K1−σ2 ) +O ( 1 K2 ( σ1−σ2 ) ) +O ( K20 K2σ2 ) +O ( K0 Kσ1 ) +O ( 1 Kσ2 ) . ( 13 ) Different from async-SGD ( e.g. , [ 9 ] ) , the optimal critic parameter ω∗θ is constantly drifting as θ changes at each iteration . This necessitates setting σ1 > σ2 to make the policy change slower than the critic , which can be observed in the second term in ( 13 ) . If σ1 > σ2 , then the policy is static relative to the critic in an asymptotic sense . To introduce the convergence of actor update , we first define the critic approximation error as app : = max θ∈Rd √ E s∼µθ |Vπθ ( s ) − V̂ω∗θ ( s ) | 2 ≤ fa + sp , ( 14 ) where µθ is the stationary distribution under πθ and P̃ . The error app captures the quality of the critic approximation under Algorithm 1 . It can be further decomposed into the function approximation error fa , which is common in analyzing AC with function approximation [ 14 , 15 , 17 ] , and the sampling error sp = O ( 1− γ ) , which is unique in analyzing two-timescale AC for a discounted MDP . The error app is small when the value function approximation is accurate and the discounting factor γ is close to 1 ; see the detailed derivations in Lemma 7 of supplementary material . Now we are ready to give the actor convergence . Theorem 2 ( Actor convergence ) . Under the same assumptions of Theorem 1 , select step size αk = c1 ( 1+k ) σ1 , βk = c2 ( 1+k ) σ2 , where 0 < σ2 < σ1 < 1 and c1 , c2 are positive constants . Then it holds that 1 K K∑ k=1 E ‖∇J ( θk ) ‖22 = O ( 1 K1−σ1 ) +O ( K0 Kσ1 ) +O ( K20 K2σ2 ) +O ( 1 K K∑ k=1 E ‖ωk − ω∗θk‖ 2 2 ) +O ( app ) . ( 15 ) Different from the analysis of async-SGD , in actor update , the stochastic gradient v ( x , θ , ω ) is biased because of inexact value function approximation . The bias introduced by the critic optimality gap and the function approximation error correspond to the last two terms in ( 15 ) . In Theorem 1 and Theorem 2 , optimizing σ1 and σ2 gives the following convergence rate . Corollary 1 ( Linear speedup ) . Given Theorem 1 and Theorem 2 , select σ1 = 35 and σ2 = 2 5 . If we further assume K0 = O ( K 1 5 ) , then it holds that 1 K K∑ k=1 E ‖∇J ( θk ) ‖22 = O ( K− 2 5 ) +O ( app ) ( 16 ) where O ( · ) contains constants that are independent of N and K0 . By setting the first term in ( 16 ) to , we get the total iteration complexity to reach -accuracy is O ( −2.5 ) . Since each iteration only uses one sample ( one transition ) , it also implies a total sample complexity of O ( −2.5 ) . Then the average sample complexity per worker is O ( −2.5/N ) which indicates linear speedup in ( 1 ) . Intuitively , the negative effect of parameter staleness introduced by parallel asynchrony vanishes asymptotically , which implies linear speedup in terms of convergence . 4.2 CONVERGENCE RESULT WITH MARKOVIAN SAMPLING Theorem 3 ( Critic convergence ) . Suppose Assumptions 1–4 hold . Consider Algorithm 1 with Markovian sampling and V̂ω ( s ) = φ ( s ) > ω . Select step size αk = c1 ( 1+k ) σ1 , βk = c2 ( 1+k ) σ2 , where 0 < σ2 < σ1 < 1 and c1 , c2 are positive constants . Then it holds that 1 K K∑ k=1 E ∥∥ωk − ω∗θk∥∥22 = O ( 1K1−σ2 ) +O ( 1 K2 ( σ1−σ2 ) ) +O ( K20 K2σ2 ) +O ( K20 log 2K Kσ1 ) +O ( K0 logK Kσ2 ) . ( 17 ) The following theorem gives the convergence rate of actor update in Algorithm 1 . Theorem 4 ( Actor convergence ) . Under the same assumptions of Theorem 3 , select step size αk = c1 ( 1+k ) σ1 , βk = c2 ( 1+k ) σ2 , where 0 < σ2 < σ1 < 1 and c1 , c2 are positive constants . Then it holds that 1 K K∑ k=1 E ‖∇J ( θk ) ‖22 =O ( 1 K1−σ1 ) +O ( K20 log 2K Kσ1 ) +O ( K20 K2σ2 ) +O ( 1 K K∑ k=1 E ‖ωk − ω∗θk‖ 2 2 ) +O ( app ) . ( 18 ) Assume K0 = O ( K 1 5 ) . Given Theorem 3 , select σ1 = 35 and σ2 = 2 5 , then it holds that 1 K K∑ k=1 E ‖∇J ( θk ) ‖22 = Õ ( K0K − 2 5 ) +O ( app ) , ( 19 ) where Õ ( · ) hides constants and the logarithmic order of K. With Markovian sampling , the stochastic gradients g ( x , ω ) and v ( x , θ , ω ) are biased , and the bias decreases as the Markov chain mixes . The mixing time corresponds to the logarithmic term logK in ( 17 ) and ( 18 ) . Because of asynchrony , at a given iteration , workers collect different number of samples and their chains mix to different degrees . The worker with the slowest mixing chain will determine the rate of convergence . The product of K0 and logK in ( 17 ) and ( 18 ) appears due to the slowest mixing chain . As the last term in ( 17 ) dominates other terms asymptotically , the convergence rate reduces as the number of workers increases . While the theoretical linear speedup is difficult to establish in the Markovian setting , we will empirically demonstrate it in Section 5.2 . 5 NUMERICAL EXPERIMENTS We test the speedup performance of A3C-TD ( 0 ) on both synthetically generated and OpenAI Gym environments . The settings , parameters , and codes are provided in supplementary material . 5.1 A3C-TD ( 0 ) IN SYNTHETIC ENVIRONMENT To verify the theoretical result , we tested A3C-TD ( 0 ) with linear value function approximation in a synthetic environment . We use tabular softmax policy parameterization [ 36 ] , which satisfies Assumption 3 . The MDP has a state space |S| = 100 , an discrete action space of |A| = 5 . Each state feature has a dimension of 10 . Elements of the transition matrix , the reward and the state features are randomly sampled from a uniform distribution over ( 0 , 1 ) . We evaluate the convergence of actor and critic respectively with the running average of test reward and critic optimality gap ‖ωk − ω∗θk‖2 . Figures 1 and 2 show the training time and sample complexity of running A3C-TD ( 0 ) with i.i.d . sampling and Markovian sampling respectively . The speedup plot is measured by the number of samples needed to achieve a target running average reward under different number of workers . All the results are average over 10 Monte-Carlo runs . Figure 1 shows that the sample complexity of A3C-TD ( 0 ) stays about the same with different number of workers under i.i.d . sampling . Also , it can be observed from the speedup plot of Figure 1 that the A3C-TD ( 0 ) achieves roughly linear speedup with i.i.d . sampling , which is consistent with Corollary 1 . The speedup of A3C-TD ( 0 ) with Markovian sampling shown in Figure 2 is roughly linear when number of workers is small . 5.2 A3C-TD ( 0 ) IN OPENAI GYM ENVIRONMENTS We have also tested A3C-TD ( 0 ) with neural network parametrization in the classic control ( Carpole ) environment and the Atari game ( Seaquest and Beamrider ) environments . In Figures 3-5 , each curve is generated by averaging over 5 Monte-Carlo runs with 95 % confidence interval . Figures 3–5 show the speedup of A3C-TD ( 0 ) under different number of workers , where the average reward is computed by taking the running average of test rewards . The speedup and runtime speedup plots are respectively measured by the number of samples and training time needed to achieve a target running average reward under different number of workers . Although not justified theoretically , Figures 3–5 suggest that the sample complexity speedup is roughly linear , and the runtime speedup slightly degrades when the number of workers increases . This is partially due to our hardware limit . Similar observation has also been obtained in async-SGD [ 9 ] . 6 CONCLUSIONS This paper revisits the A3C algorithm with TD ( 0 ) for the critic update , termed A3C-TD ( 0 ) . With linear value function approximation , the convergence of the A3C-TD ( 0 ) algorithm has been established under both i.i.d . and Markovian sampling settings . Under i.i.d . sampling , A3C-TD ( 0 ) achieves linear speedup compared to the best-known sample complexity of two-timescale AC , theoretically justifying the benefit of parallelism and asynchrony for the first time . Under Markov sampling , such a linear speedup can be observed in most classic benchmark tasks . REFERENCES [ 1 ] T. P. Lillicrap , J. J . Hunt , A. Pritzel , N. Heess , T. Erez , Y. Tassa , D. Silver , and D. Wierstra , “ Continuous control with deep reinforcement learning , ” in Proc . of International Conference on Learning Representations , 2016 . [ 2 ] V. Mnih , K. Kavukcuoglu , D. Silver , A . A. Rusu , J. Veness , M. G. Bellemare , A. Graves , M. Riedmiller , A. K. Fidjeland , G. Ostrovski et al. , “ Human-level control through deep reinforcement learning , ” Nature , vol . 518 , no . 7540 , p. 529 , 2015 . [ 3 ] V. Mnih , A. P. Badia , M. Mirza , A. Graves , T. P. Lillicrap , T. Harley , D. Silver , and K. Kavukcuoglu , “ Asynchronous methods for deep reinforcement learning. ” in Proc . of International Conference on Machine Learning , 2016 . [ 4 ] A. Nair , P. Srinivasan , S. Blackwell , C. Alcicek , R. Fearon , A . De Maria , V. Panneershelvam , M. Suleyman , C. Beattie , S. Petersen et al. , “ Massively parallel methods for deep reinforcement learning , ” arXiv preprint:1507.04296 , 2015 . [ 5 ] M. Assran , J. Romoff , N. Ballas , J. Pineau , and M. Rabbat , “ Gossip-based actor-learner architectures for deep reinforcement learning , ” in Proc . of Advances in Neural Information Processing Systems , 2019 . [ 6 ] V. Konda , Actor-critic algorithms . PhD thesis , Department of Electrical Engineering and Computer Science , Massachusetts Institute of Technology , 2002 . [ 7 ] R. Sutton , D. McAllester , S. Singh , and Y. Mansour , “ Policy gradient methods for reinforcement learning with function approximation. ” in Proc . of Advances in Neural Information Processing Systems , 2000 . [ 8 ] R. Sutton , “ Learning to predict by the methods of temporal differences , ” Machine Learning , vol . 3 , pp . 9–44 , 1988 . [ 9 ] X. Lian , H. Zhang , C. Hsieh , Y. Yijun Huang , and J. Liu , “ A comprehensive linear speedup analysis for asynchronous stochastic parallel optimization from zeroth-order to first-order , ” in Proc . of the Advances in Neural Information Processing Systems , 2016 . [ 10 ] V. Borkar and V. Konda , “ The actor-critic algorithm as multi-time-scale stochastic approximation , ” Sadhana , vol . 22 , no . 4 , pp . 525–543 , 1997 . [ 11 ] S. Bhatnagar , R. Sutton , M. Ghavamzadeh , and M. Lee , “ Natural actor critic algorithms , ” Automatica , vol . 45 , pp . 2471–2482 , 2009 . [ 12 ] Z. Yang , K. Zhang , M. Hong , and T. Başar , “ A finite sample analysis of the actor-critic algorithm , ” in Proc . of IEEE Conference on Decision and Control , 2018 , pp . 2759–2764 . [ 13 ] H. Kumar , A. Koppel , and A. Ribeiro , “ On the sample complexity of actor-critic method for reinforcement learning with function approximation , ” arXiv preprint:1910.08412 , 2019 . [ 14 ] S. Qiu , Z. Yang , J. Ye , and Z. Wang , “ On the finite-time convergence of actor-critic algorithm , ” in Optimization Foundations for Reinforcement Learning Workshop at Advances in Neural Information Processing Systems , 2019 . [ 15 ] T. Xu , Z. Wang , and Y. Liang , “ Improving sample complexity bounds for ( natural ) actor-critic algorithms , ” in Proc . of Advances in Neural Information Processing Systems , 2020 . [ 16 ] —— , “ Non-asymptotic convergence analysis of two time-scale ( natural ) actor-critic algorithms , ” arXiv preprint:2005.03557 , 2020 . [ 17 ] Y. Wu , W. Zhang , P. Xu , and Q. Gu , “ A finite time analysis of two time-scale actor critic methods , ” in Proc . of Advances in Neural Information Processing Systems , 2020 . [ 18 ] M. Hong , H.-T. Wai , Z. Wang , and Z. Yang , “ A two-timescale framework for bilevel optimization : Complexity analysis and application to actor-critic , ” arXiv preprint:2007.05170 , 2020 . [ 19 ] M. Babaeizadeh , I. Frosio , S. Tyree , J. Clemons , and J. Kautz , “ Reinforcement learning through asynchronous advantage actor-critic on a gpu , ” in Proc . of International Conference on Learning Representations , 2017 . [ 20 ] A. Stooke and P. Abbeel , “ Accelerated methods for deep reinforcement learning , ” arXiv preprint:1803.02811 , 2019 . [ 21 ] L. Espeholt , H. Soyer , R. Munos , K. Simonyan , V. Mnih , T. Ward , Y. Doron , V. Firoiu , T. Harley , I. Dunning , S. Legg , and K. Kavukcuoglu , “ Impala : Scalable distributed deep-rl with importance weighted actor-learner architectures , ” arXiv preprint:1802.01561 , 2018 . [ 22 ] D. Bertsekas and J. Tsitsiklis , Parallel and distributed computation : numerical methods . Prentice-Hall , 1989 . [ 23 ] A. Agarwal and J. Duchi , “ Distributed delayed stochastic optimization , ” in Proc . of Advances in Neural Information Processing Systems , 2011 . [ 24 ] H. Feyzmahdavian , A. Aytekin , and M. Johansson , “ An asynchronous mini-batch algorithm for regularized stochastic optimization , ” arXiv preprint:1505.04824 , 2015 . [ 25 ] X. Lian , Y. Huang , Y. Li , and J. Liu , “ Asynchronous parallel stochastic gradient for nonconvex optimization. ” in Proc . of Advances in Neural Information Processing Systems , 2015 . [ 26 ] T. Sun , R. Hannah , and W. Yin , “ Asynchronous coordinate descent under more realistic assumptions , ” in Proc . of Advances in Neural Information Processing Systems , 2017 . [ 27 ] X. Lian , C. Zhang , H. Zhang , C.-J . Hsieh , W. Zhang , and J. Liu , “ Can decentralized algorithms outperform centralized algorithms ? a case study for decentralized parallel stochastic gradient descent , ” in Proc . of Advances in Neural Information Processing Systems , 2017 . [ 28 ] R. J. Williams , “ Simple statistical gradient-following algorithms for connectionist reinforcement learning , ” Machine Learning , vol . 8 , no . 3-4 , pp . 229–256 , May 1992 . [ 29 ] J. Baxter and P. L. Bartlett , “ Infinite-horizon policy-gradient estimation , ” J . Artificial Intelligence Res. , vol . 15 , pp . 319–350 , 2001 . [ 30 ] R. S. Sutton and A. G. Barto , Reinforcement learning : An introduction . MIT Press , 2018 . [ 31 ] V. Konda and V. Borkar , “ Actor-critic–type learning algorithms for markov decision processes , ” SIAM Journal on Control and Optimization , vol . 38 , no . 1 , pp . 94–123 , 1999 . [ 32 ] J. Bhandari , D. Russo , and R. Singal , “ A finite time analysis of temporal difference learning with linear function approximation. ” in Proc . of Conference on Learning Theory , 2018 . [ 33 ] T. Xu , Z. Wang , Y. Zhou , and Y. Liang , “ Reanalysis of variance reduced temporal difference learning , ” in Proc . of International Conference on Learning Representations , 2020 . [ 34 ] S. Zou , T. Xu , and Y. Liang , “ Finite-sample analysis for SARSA with linear function approximation , ” in Proc . of Advances in Neural Information Processing Systems , 2019 . [ 35 ] K. Zhang , A. Koppel , H. Zhu , and T. Başar , “ Global convergence of policy gradient methods to ( almost ) locally optimal policies , ” arXiv preprint:1906.08383 , 2019 . [ 36 ] A. Agarwal , S. M. Kakade , J. D. Lee , and G. Mahajan , “ Optimality and approximation with policy gradient methods in markov decision processes. ” in Proc . of Thirty Third Conference on Learning Theory , 2020 . [ 37 ] K. Doya , “ Reinforcement learning in continuous time and space , ” Neural Computation , vol . 12 , no . 1 , pp . 219–245 , 2000 . [ 38 ] R. Sutton , H. Maei , D. Precup , S. Bhatnagar , D. Silver , and E. Szepesvári , C.and Wiewiora , “ Fast gradient-descent methods for temporal-difference learning with linear function approximation , ” in Proc . of International Conference on Machine Learning , 2009 . [ 39 ] A. Y. Mitrophanov , “ Sensitivity and convergence of uniformly ergodic markov chains , ” Journal of Applied Probability , vol . 42 , no . 4 , pp . 1003–1014 , 2005 . [ 40 ] Dgriff , “ Pytorch implementation of a3c , ” https : //github.com/dgriff777/rl_a3c_pytorch , 2018 . Supplementary Material A PRELIMINARY LEMMAS A.1 GEOMETRIC MIXING The operation p⊗ q denotes the tensor product between two distributions p ( x ) and q ( y ) , i.e . ( p⊗ q ) ( x , y ) = p ( x ) · q ( y ) . Lemma 1 . Suppose Assumption 4 holds for a Markov chain generated by the rule at ∼ πθ ( ·|st ) , st+1 ∼ P̃ ( ·|st , at ) . For any θ ∈ Rd , we have sup s0∈S dTV ( P ( ( st , at , st+1 ) ∈ ·|s0 , πθ ) , µθ ⊗ πθ ⊗ P̃ ) ≤ κρt . ( 20 ) where µθ ( · ) is the stationary distribution with policy πθ and transition kernel P̃ ( ·|s , a ) . Proof . We start with sup s0∈S dTV ( P ( ( st , at , st+1 ) = ·|s0 , πθ ) , µθ ⊗ πθ ⊗ P̃ ) = sup s0∈S dTV ( P ( st = ·|s0 , πθ ) ⊗ πθ ⊗ P̃ , µθ ⊗ πθ ⊗ P̃ ) = sup s0∈S 1 2 ∫ s∈S ∑ a∈A ∫ s′∈S |P ( st = ds|s0 , πθ ) πθ ( a|s ) P̃ ( ds′|s , a ) − µθ ( ds ) πθ ( a|s ) P̃ ( ds′|s , a ) ∣∣∣ = sup s0∈S 1 2 ∫ s∈S |P ( st = ds|s0 , πθ ) − µθ ( ds ) | ∑ a∈A πθ ( a|s ) ∫ s′∈S P̃ ( ds′|s , a ) = sup s0∈S dTV ( P ( st ∈ ·|s0 , πθ ) , µθ ) ≤ κρt , which completes the proof . For the use in the later proof , given K > 0 , we first define mK as : mK : = min { m ∈ N+ |κρm−1 ≤ min { αk , βk } } , ( 21 ) where κ and ρ are constants defined in ( 4 ) . mK is the minimum number of samples needed for the Markov chain to approach the stationary distribution so that the bias incurred by the Markovian sampling is small enough . A.2 AUXILIARY MARKOV CHAIN The auxiliary Markov chain is a virtual Markov chain with no policy drifting — a technique developed in [ 34 ] to analyze stochastic approximation algorithms in non-stationary settings . Lemma 2 . Under Assumption 1 and Assumption 3 , consider the update ( 9 ) in Algorithm 1 with Markovian sampling . For a given number of samples m , consider the Markov chain of the worker that contributes to the kth update : st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−1−−−−−−→ at−m+1 · · · st−1 θk−d1−−−−→ at−1 P̃−→ st θk−d0−−−−→ at P̃−→ st+1 , where ( st , at , st+1 ) = ( s ( k ) , a ( k ) , s′ ( k ) ) , and { dj } m j=0 is some increasing sequence with d0 : = τk . Given ( st−m , at−m , st−m+1 ) and θk−dm , we construct its auxiliary Markov chain by repeatedly applying πθk−dm : st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−−−−→ ãt−m+1 · · · s̃t−1 θk−dm−−−−→ ãt−1 P̃−→ s̃t θk−dm−−−−→ ãt P̃−→ s̃t+1 . Define xt : = ( st , at , st+1 ) , then we have : dTV ( P ( xt ∈ ·|θk−dm , st−m+1 ) , P ( x̃t ∈ ·|θk−dm , st−m+1 ) ) ≤ 1 2 |A|Lπ dm∑ i=τk E [ ‖θk−i − θk−dm‖2|θk−dm , st−m+1 ] . ( 22 ) Proof . Throughout the lemma , all expectations and probabilities are conditioned on θk−dm and st−m+1 . We omit this condition for convenience . First we have dTV ( P ( st+1 ∈ · ) , P ( s̃t+1 ∈ · ) ) = 1 2 ∫ s′∈S |P ( st+1 = ds′ ) − P ( s̃t+1 = ds′ ) | = 1 2 ∫ s′∈S ∣∣∣∣∣ ∫ s∈S ∑ a∈A P ( st = ds , at = a , st+1 = ds′ ) − P ( s̃t = ds , ãt = a , s̃t+1 = ds′ ) ∣∣∣∣∣ ≤ 1 2 ∫ s′∈S ∫ s∈S ∑ a∈A |P ( st = ds , at = a , st+1 = ds′ ) − P ( s̃t = ds , ãt = a , s̃t+1 = ds′ ) | = 1 2 ∫ s∈S ∑ a∈A ∫ s′∈S |P ( st = ds , at = a , st+1 = ds′ ) − P ( s̃t = ds , ãt = a , s̃t+1 = ds′ ) | = dTV ( P ( xt ∈ · ) , P ( x̃t ∈ · ) ) , ( 23 ) where the last second equality is due to Tonelli ’ s theorem . Next we have dTV ( P ( xt ∈ · ) , P ( x̃t ∈ · ) ) = 1 2 ∫ s∈S ∑ a∈A ∫ s′∈S |P ( st = ds , at = a , st+1 = ds′ ) − P ( s̃t = ds , ãt = a , s̃t+1 = ds′ ) | = 1 2 ∫ s∈S ∑ a∈A |P ( st = ds , at = a ) − P ( s̃t = ds , ãt = a ) | ∫ s′∈S P̃ ( st+1 = ds′|st = ds , at = a ) = 1 2 ∫ s∈S ∑ a∈A |P ( st = ds , at = a ) − P ( s̃t = ds , ãt = a ) | = dTV ( P ( ( st , at ) ∈ · ) , P ( ( s̃t , ãt ) ∈ · ) ) . ( 24 ) Due to the fact that θk−τk is dependent on st , we need to write P ( st , at ) as P ( st , at ) = ∫ θk−τk∈Rd P ( st , θk−τk , at ) = ∫ θ∈Rd P ( st ) P ( θk−τk = dθ|st ) πθk−τk ( at|st ) = P ( st ) ∫ θ∈Rd P ( θk−τk = dθ|st ) πθk−τk ( at|st ) = P ( st ) E [ πθk−τk ( at|st ) |st ] . Then we have dTV ( P ( ( st , at ) ∈ · ) , P ( ( s̃t , ãt ) ∈ · ) ) = 1 2 ∫ s∈S ∑ a∈A ∣∣∣P ( st = ds ) E [ πθk−τk ( at = a|st = ds ) |st = ds ] − P ( s̃t = ds ) πθk−dm ( ãt = a|s̃t = ds ) ∣∣∣ ≤ 1 2 ∫ s∈S ∑ a∈A ∣∣∣P ( st = ds ) E [ πθk−τk ( at = a|st = ds ) |st = ds ] − P ( st = ds ) πθk−dm ( at = a|st = ds ) ∣∣∣ + 1 2 ∫ s∈S ∑ a∈A ∣∣P ( st = ds ) πθk−dm ( ãt = a|s̃t = ds ) − P ( s̃t = ds ) πθk−dm ( ãt = a|s̃t = ds ) ∣∣ = 1 2 ∫ s∈S P ( st = ds ) ∑ a∈A ∣∣∣E [ πθk−τk ( at = a|st = ds ) |st = ds ] − πθk−dm ( at = a|st = ds ) ∣∣∣ + 1 2 ∫ s∈S |P ( st = ds ) − P ( s̃t = ds ) | . ( 25 ) Using Jensen ’ s inequality , we have dTV ( P ( ( st , at ) ∈ · ) , P ( ( s̃t , ãt ) ∈ · ) ) ≤ 1 2 ∫ s∈S P ( st = ds ) ∑ a∈A E [ ∣∣∣πθk−τk ( at = a|st = ds ) − πθk−dm ( at = a|st = ds ) ∣∣∣∣∣∣ st = ds ] + 1 2 ∫ s∈S |P ( st = ds ) − P ( s̃t = ds ) | ≤ 1 2 ∫ s∈S P ( st = ds ) ∑ a∈A E [ ‖θk−τk − θk−dm‖2| st = ds ] + 1 2 ∫ s∈S |P ( st = ds ) − P ( s̃t = ds ) | = 1 2 |A|Lπ E ‖θk−τk − θk−dm‖2 + dTV ( P ( st ∈ · ) , P ( s̃t ∈ · ) ) ( 26 ) where the last inequality follows Assumption 3 . Now we start to prove ( 22 ) . dTV ( P ( xt ∈ · ) , P ( x̃t ∈ · ) ) ( 24 ) = dTV ( P ( ( st , at ) ∈ · ) , P ( ( s̃t , ãt ) ∈ · ) ) ( 25 ) ≤ dTV ( P ( st ∈ · ) , P ( s̃t ∈ · ) ) + 1 2 |A|Lπ E ‖θk−τk − θk−dm‖2 ( 23 ) ≤ dTV ( P ( xt−1 ∈ · ) , P ( x̃t−1 ∈ · ) ) + 1 2 |A|Lπ E ‖θk−τk − θk−dm‖2 . Now we have dTV ( P ( xt ∈ · ) , P ( x̃t ∈ · ) ) ≤ dTV ( P ( xt−1 ∈ · ) , P ( x̃t−1 ∈ · ) ) + 1 2 |A|Lπ E ‖θk−τk − θk−dm‖2 . ( 27 ) Since dTV ( P ( xt−m ∈ · ) , P ( xt−m ∈ · ) ) = 0 , recursively applying ( 27 ) for { t− 1 , ... , t−m } gives dTV ( P ( xt ∈ · ) , P ( x̃t ∈ · ) ) ≤ 1 2 |A|Lπ m∑ j=0 E ‖θk−dj − θk−dm‖2 ≤ 1 2 |A|Lπ dm∑ i=τk E ‖θk−i − θk−dm‖2 , which completes the proof . A.3 LIPSCHITZ CONTINUITY OF VALUE FUNCTION Lemma 3 . Suppose Assumption 3 holds . For any θ1 , θ2 ∈ Rd and s ∈ S , we have ‖∇Vπθ1 ( s ) ‖2 ≤ LV , ( 28a ) |Vπθ1 ( s ) − Vπθ2 ( s ) | ≤ LV ‖θ1 − θ2‖2 , ( 28b ) where the constant is LV : = Cψrmax/ ( 1− γ ) with Cψ defined as in Assumption 3 . Proof . First we have Qπ ( s , a ) = E [ ∞∑ t=0 γtr ( st , at , st+1 ) |s0 = s , a0 = a ] ≤ ∞∑ t=0 γtrmax = rmax 1− γ . By the policy gradient theorem [ 7 ] , we have ‖∇Vπθ1 ( s ) ‖2 = ∥∥E [ Qπθ1 ( s , a ) ψθ1 ( s , a ) ] ∥∥2 ≤ E ∥∥Qπθ1 ( s , a ) ψθ1 ( s , a ) ∥∥2 ≤ E [ |Qπθ1 ( s , a ) |‖ψθ1 ( s , a ) ‖2 ] ≤ rmax 1− γ Cψ , where the first inequality is due to Jensen ’ s inequality , and the last inequality follows Assumption 3 and the fact that Qπ ( s , a ) ≤ rmax1−γ . By the mean value theorem , we immediately have |Vπθ1 ( s ) − Vπθ2 ( s ) | ≤ sup θ1∈Rd ∥∥∇Vπθ1 ( s ) ∥∥2 ‖θ1 − θ2‖2 = LV ‖θ1 − θ2‖2 , which completes the proof . A.4 LIPSCHITZ CONTINUITY OF POLICY GRADIENT We give a proposition regarding the LJ -Lipschitz of the policy gradient under proper assumptions , which has been shown by [ 35 ] . Proposition 1 . Suppose Assumption 3 and 4 hold . For any θ , θ′ ∈ Rd , we have ‖∇J ( θ ) − ∇J ( θ′ ) ‖2 ≤ LJ‖θ − θ′‖2 , where LJ is a positive constant . A.5 LIPSCHITZ CONTINUITY OF OPTIMAL CRITIC PARAMETER We provide a justification for Lipschitz continuity of ω∗θ in the next proposition . Proposition 2 . Suppose Assumption 3 and 4 hold . For any θ1 , θ2 ∈ Rd , we have ‖ω∗θ1 − ω ∗ θ2‖2 ≤ Lω‖θ1 − θ2‖2 , where Lω : = 2rmax|A|Lπ ( λ−1 + λ−2 ( 1 + γ ) ) ( 1 + logρ κ−1 + ( 1− ρ ) −1 ) . Proof . We use A1 , A2 , b1 and b2 as shorthand notations of Aπθ1 , Aπθ2 , bπθ1 and bπθ2 respectively . By Assumption 2 , Aθ , φ is invertible for any θ ∈ Rd , so we can write ω∗θ = −A −1 θ , φbθ , φ . Then we have ‖ω∗1 − ω∗2‖2 = ‖ −A−11 b1 +A −1 2 b2‖2 = ‖ −A−11 b1 −A −1 1 b2 +A −1 1 b2 +A −1 2 b2‖2 = ‖ −A−11 ( b1 − b2 ) − ( A −1 1 −A −1 2 ) b2‖2 ≤ ‖A−11 ( b1 − b2 ) ‖2 + ‖ ( A −1 1 −A −1 2 ) b2‖2 ≤ ‖A−11 ‖2‖b1 − b2‖2 + ‖A −1 1 −A −1 2 ‖2‖b2‖2 = ‖A−11 ‖2‖b1 − b2‖2 + ‖A −1 1 ( A2 −A1 ) A −1 2 ‖2‖b2‖2 ≤ ‖A−11 ‖2‖b1 − b2‖2 + ‖A −1 1 ‖2‖A −1 2 ‖2‖b2‖2‖ ( A2 −A1 ) ‖2 ≤ λ−1 ‖b1 − b2‖2 + λ −2rmax ‖A1 −A2‖2 , ( 29 ) where the last inequality follows Assumption 2 , and the fact that ‖b2‖2 = ‖E [ r ( s , a , s′ ) φ ( s ) ] ‖2 ≤ E ‖r ( s , a , s ′ ) φ ( s ) ‖2 ≤ E [ |r ( s , a , s ′ ) |‖φ ( s ) ‖2 ] ≤ rmax . Denote ( s1 , a1 , s′1 ) and ( s2 , a2 , s′2 ) as samples drawn with θ1 and θ2 respectively , i.e . s1 ∼ µθ1 , a1 ∼ πθ1 , s′1 ∼ P̃ and s2 ∼ µθ2 , a2 ∼ πθ2 , s′2 ∼ P̃ . Then we have ‖b1 − b2‖2 = ∥∥E [ r ( s1 , a1 , s′1 ) φ ( s1 ) ] − E [ r ( s2 , a2 , s′2 ) φ ( s2 ) ] ∥∥ 2 ≤ sup s , a , s′ ‖r ( s , a , s′ ) φ ( s ) ‖2‖P ( ( s1 , a1 , s′1 ) ∈ · ) − P ( ( s2 , a2 , s′2 ) ∈ · ) ‖TV ≤ rmax‖P ( ( s1 , a1 , s′1 ) ∈ · ) − P ( ( s2 , a2 , s′2 ) ∈ · ) ‖TV = 2rmaxdTV ( µθ1 ⊗ πθ1 ⊗ P̃ , µθ2 ⊗ πθ2 ⊗ P̃ ) ≤ 2rmax|A|Lπ ( 1 + logρ κ−1 + ( 1− ρ ) −1 ) ‖θ1 − θ2‖2 , ( 30 ) where the first inequality follows the definition of total variation ( TV ) norm , and the last inequality follows Lemma A.1 . in [ 17 ] . Similarly we have : ‖A1 −A2‖2 ≤ 2 ( 1 + γ ) dTV ( µθ1 ⊗ πθ1 , µθ2 ⊗ πθ2 ) = ( 1 + γ ) |A|Lπ ( 1 + logρ κ−1 + ( 1− ρ ) −1 ) ‖θ1 − θ2‖2 . ( 31 ) Substituting ( 30 ) and ( 31 ) into ( 29 ) completes the proof . B PROOF OF MAIN THEOREMS B.1 PROOF OF THEOREM 1 For brevity , we first define the following notations : x : = ( s , a , s′ ) , δ̂ ( x , ω ) : = r ( s , a , s′ ) + γφ ( s′ ) > ω − φ ( s ) > ω , g ( x , ω ) : = δ̂ ( x , ω ) φ ( s ) , g ( θ , ω ) : = E s∼µθ , a∼πθ , s′∼P̃ [ g ( x , ω ) ] . We also define constant Cδ : = rmax + ( 1 + γ ) max { rmax1−γ , Rω } , and we immediately have ‖g ( x , ω ) ‖2 ≤ |r ( x ) + γφ ( s′ ) > ω − φ ( s ) > ω| ≤ rmax + ( 1 + γ ) Rω ≤ Cδ ( 32 ) and likewise , we have ‖g ( x , ω ) ‖2 ≤ Cδ . The critic update in Algorithm 1 can be written compactly as : ωk+1 = ΠRω ( ωk + βkg ( x ( k ) , ωk−τk ) ) , ( 33 ) where τk is the delay of the parameters used in evaluating the kth stochastic gradient , and x ( k ) : = ( s ( k ) , a ( k ) , s ′ ( k ) ) is the sample used to evaluate the stochastic gradient at kth update . Proof . Using ω∗k as shorthand notation of ω ∗ θk , we start with the optimality gap ‖ωk+1 − ω∗k+1‖22 = ‖ΠRω ( ωk + βkg ( x ( k ) , ωk−τk ) ) − ω∗k+1‖22 ≤ ‖ωk + βkg ( x ( k ) , ωk−τk ) − ω∗k+1‖22 = ‖ωk − ω∗k‖ 2 2 + 2βk 〈 ωk − ω∗k , g ( x ( k ) , ωk−τk ) 〉 + 2 〈 ωk − ω∗k , ω∗k − ω∗k+1 〉 + ∥∥ω∗k − ω∗k+1 + βkg ( x ( k ) , ωk−τk ) ∥∥22 = ‖ωk − ω∗k‖ 2 2 + 2βk 〈 ωk − ω∗k , g ( x ( k ) , ωk−τk ) − g ( x ( k ) , ωk ) 〉 + 2βk 〈 ωk − ω∗k , g ( x ( k ) , ωk ) − g ( θk , ωk ) 〉 + 2βk 〈ωk − ω∗k , g ( θk , ωk ) 〉+ 2 〈 ωk − ω∗k , ω∗k − ω∗k+1 〉 + ∥∥ω∗k − ω∗k+1 + βkg ( x ( k ) , ωk−τk ) ∥∥22 ≤ ‖ωk − ω∗k‖ 2 2 + 2βk 〈 ωk − ω∗k , g ( x ( k ) , ωk−τk ) − g ( x ( k ) , ωk ) 〉 + 2βk 〈 ωk − ω∗k , g ( x ( k ) , ωk ) − g ( θk , ωk ) 〉 + 2βk 〈ωk − ω∗k , g ( θk , ωk ) 〉+ 2 〈 ωk − ω∗k , ω∗k − ω∗k+1 〉 + 2 ∥∥ω∗k − ω∗k+1∥∥22 + 2C2δβ2k . ( 34 ) We first bound 〈ωk − ω∗k , g ( θk , ωk ) 〉 in ( 34 ) as 〈ωk − ω∗k , g ( θk , ωk ) 〉 = 〈ωk − ω∗k , g ( θk , ωk ) − g ( θk , ω∗k ) 〉 = 〈 ωk − ω∗k , E [ ( γφ ( s′ ) − φ ( s ) ) > ( ωk − ω∗k ) φ ( s ) ] 〉 = 〈 ωk − ω∗k , E [ φ ( s ) ( γφ ( s′ ) − φ ( s ) ) > ] ( ωk − ω∗k ) 〉 = 〈 ωk − ω∗k , Aπθk ( ωk − ω ∗ k ) 〉 ≤ −λ‖ωk − ω∗k‖22 , ( 35 ) where the first equality is due to g ( θ , ω∗θ ) = Aθ , φω ∗ θ + b = 0 , and the last inequality follows Assumption 2 . Substituting ( 35 ) into ( 34 ) , then taking expectation on both sides of ( 34 ) yield E ‖ωk+1 − ω∗k+1‖22 ≤ ( 1− 2λβk ) E ‖ωk − ω∗k‖ 2 2 + 2βk E 〈 ωk − ω∗k , g ( x ( k ) , ωk−τk ) − g ( x ( k ) , ωk ) 〉 + 2βk E 〈 ωk − ω∗k , g ( x ( k ) , ωk ) − g ( θk , ωk ) 〉 + 2E 〈 ωk − ω∗k , ω∗k − ω∗k+1 〉 + 2E ∥∥ω∗k − ω∗k+1∥∥22 + 2C2δβ2k . ( 36 ) We then bound the term E 〈 ωk − ω∗k , g ( x ( k ) , ωk−τk ) − g ( x ( k ) , ωk ) 〉 in ( 36 ) as E 〈 ωk − ω∗k , g ( x ( k ) , ωk−τk ) − g ( x ( k ) , ωk ) 〉 = E 〈 ωk − ω∗k , ( γφ ( s′ ( k ) ) − φ ( s ( k ) ) ) > ( ωk−τk − ωk ) φ ( s ( k ) ) 〉 ≤ ( 1 + γ ) E [ ‖ωk − ω∗k‖2‖ωk−τk − ωk‖2 ] ≤ ( 1 + γ ) E ‖ωk − ω∗k‖2 ∥∥∥∥∥ k−1∑ i=k−τk ( ωi+1 − ωi ) ∥∥∥∥∥ 2 ≤ ( 1 + γ ) E [ ‖ωk − ω∗k‖2 k−1∑ i=k−τk βi‖g ( xi , ωi−τi ) ‖2 ] ≤ ( 1 + γ ) E [ ‖ωk − ω∗k‖2 k−1∑ i=k−τk βk−K0‖g ( xi , ωi−τi ) ‖2 ] ≤ Cδ ( 1 + γ ) K0βk−K0 E ‖ωk − ω∗k‖2 , ( 37 ) where the second last inequality is due to the monotonicity of step size , and the last inequality follows the definition of Cδ in ( 32 ) . Next we jointly bound the fourth and fifth term in ( 36 ) as 2E 〈 ωk − ω∗k , ω∗k − ω∗k+1 〉 + 2E ∥∥ω∗k − ω∗k+1∥∥22 ≤ 2E [ ‖ωk − ω∗k‖2 ∥∥ω∗k − ω∗k+1∥∥2 ] + 2E∥∥ω∗k − ω∗k+1∥∥22 ≤ 2Lω E [ ‖ωk − ω∗k‖2 ‖θk − θk+1‖2 ] + 2L 2 ω E ‖θk − θk+1‖ 2 2 = 2Lωαk E [ ‖ωk − ω∗k‖2 ∥∥∥δ̂ ( x ( k ) , ωk−τk ) ψθk−τk ( s ( k ) , a ( k ) ) ∥∥∥2 ] + 2L2ωα2k E∥∥∥δ̂ ( x ( k ) , ωk−τk ) ψθk−τk ( s ( k ) , a ( k ) ) ∥∥∥22 ≤ 2LωCpαk E ‖ωk − ω∗k‖2 + 2L 2 ωC 2 pα 2 k , ( 38 ) where constant Cp : = CδCψ . The second inequality is due to the Lω-Lipschitz of ω∗θ shown in Proposition 2 , and the last inequality follows the fact that ‖δ̂ ( x ( k ) , ωk−τk ) ψθk−τk ( s ( k ) , a ( k ) ) ‖2 ≤ CδCψ = Cp . ( 39 ) Substituting ( 37 ) and ( 38 ) into ( 36 ) yields E ‖ωk+1 − ω∗k+1‖22 ≤ ( 1− 2λβk ) E ‖ωk − ω∗k‖ 2 2 + 2βk ( C1 αk βk + C2K0βk−K0 ) E ‖ωk − ω∗k‖2 + 2βk E 〈 ωk − ω∗k , g ( x ( k ) , ωk ) − g ( θk , ωk ) 〉 + Cqβ 2 k , ( 40 ) where C1 : = LωCp , C2 : = Cδ ( 1 + γ ) and Cq : = 2C2δ + 2L 2 ωC 2 p max ( k ) α2k β2k = 2C2δ + 2L 2 ωC 2 p c21 c22 . For brevity , we use x ∼ θ to denote s ∼ µθ , a ∼ πθ and s′ ∼ P̃ in this proof . Consider the third term in ( 40 ) conditioned on θk , ωk , θk−τk . We bound it as E [ 〈 ωk − ω∗k , g ( x ( k ) , ωk ) − g ( θk , ωk ) 〉 |θk , ωk , θk−τk ] = 〈 ωk − ω∗k , E x ( k ) ∼θk−τk [ g ( x ( k ) , ωk ) |ωk ] − g ( θk , ωk ) 〉 = 〈 ωk − ω∗k , g ( θk−τk , ωk ) − g ( θk , ωk ) 〉 ≤ ‖ωk − ω∗k‖2‖g ( θk−τk , ωk ) − g ( θk , ωk ) ‖2 ≤ 2Rω ∥∥∥∥ Ex∼θk−τk [ g ( x , ωk ) ] − Ex∼θk [ g ( x , ωk ) ] ∥∥∥∥ 2 ≤ 2Rω sup x ‖g ( x , ωk ) ‖2 ∥∥∥µθk−τk ⊗ πθk−τk ⊗ P̃ − µθk ⊗ πθk ⊗ P̃∥∥∥TV ≤ 4RωCδdTV ( µθk−τk ⊗ πθk−τk ⊗ P̃ , µθk ⊗ πθk ⊗ P̃ ) , ( 41 ) where second last inequality follows the definition of TV norm and the last inequality uses the definition of Cδ in ( 32 ) . Define constant C3 : = 2RωCδ|A|Lπ ( 1 + logρ κ−1 + ( 1− ρ ) −1 ) . Then by following the third item in Lemma A.1 shown by [ 17 ] , we can write ( 41 ) as E [ 〈 ωk − ω∗k , g ( x ( k ) , ωk ) − g ( θk , ωk ) 〉 |θk , ωk , θk−τk ] ≤ 4RωCδdTV ( µθk−τk ⊗ πθk−τk ⊗ P̃ , µθk ⊗ πθk ⊗ P̃ ) ≤ C3 ‖θk−τk − θk‖2 ≤ C3 k−1∑ i=k−τk αi‖g ( xi , ωi−τi ) ‖2 ≤ C3CδK0αk−K0 , ( 42 ) where we used the monotonicity of αk and Assumption 1 . Taking total expectation on both sides of ( 42 ) and substituting it into ( 40 ) yield E ‖ωk+1 − ω∗k+1‖22 ≤ ( 1− 2λβk ) E ‖ωk − ω∗k‖ 2 2 + 2βk ( C1 αk βk + C2K0βk−K0 ) E ‖ωk − ω∗k‖2 + 2C3CδK0βkαk−K0 + Cqβ 2 k. ( 43 ) Taking summation on both sides of ( 43 ) and rearranging yield 2λ K∑ k=K0 E ‖ωk − ω∗k‖ 2 2 ≤ K∑ k=K0 1 βk ( E ‖ωk − ω∗k‖ 2 2 − E ∥∥ωk+1 − ω∗k+1∥∥22 ) I1 +Cq K∑ k=K0 βk I2 + 2 K∑ k=K0 2C3CδK0αk−K0 I3 +2 K∑ k=K0 ( C1 αk βk + C2K0βk−K0 ) E ‖ωk − ω∗k‖2 I4 . ( 44 ) We bound I1 as I1 = K∑ k=MK 1 βk ( E ‖ωk − ω∗k‖ 2 2 − E ∥∥ωk+1 − ω∗k+1∥∥22 ) = K∑ k=MK ( 1 βk − 1 βk−1 ) E ‖ωk − ω∗k‖ 2 2 + 1 βMK−1 E ∥∥ωMK − ω∗MK∥∥22 − 1βk E ∥∥ωK+1 − ω∗K+1∥∥22 ≤ K∑ k=MK ( 1 βk − 1 βk−1 ) E ‖ωk − ω∗k‖ 2 2 + 1 βMK−1 E ∥∥ωMK − ω∗MK∥∥22 ≤ 4R2ω ( K∑ k=MK ( 1 βk − 1 βk−1 ) + 1 βMK−1 ) = 4R2ω βk = O ( Kσ2 ) , ( 45 ) where the last inequality is due to the fact that ‖ωk − ω∗θ‖2 ≤ ‖ωk‖2 + ‖ω∗θ‖2 ≤ 2Rω . We bound I2 as K∑ k=MK βk = K∑ k=MK c2 ( 1 + k ) σ2 = O ( K1−σ2 ) ( 46 ) where the inequality follows from the integration rule ∑b k=a k −σ ≤ b 1−σ 1−σ . We bound I3 as I3 = K∑ k=K0 2C3CδK0αk−K0 = 2C3Cδc1K0 K−K0∑ k=0 ( 1 + k ) −σ1 = O ( K0K1−σ1 ) . ( 47 ) For the last term I4 , we have I4 = K∑ k=K0 ( C1 αk βk + C2K0βk−K0 ) E ‖ωk − ω∗k‖2 ≤ √√√√ K∑ k=K0 ( C1 αk βk + C2K0βk−K0 ) 2√√√√ K∑ k=K0 ( E ‖ωk − ω∗k‖2 ) 2 ≤ √√√√ K∑ k=K0 ( C1 αk βk + C2K0βk−K0 ) 2√√√√ K∑ k=K0 E ‖ωk − ω∗k‖ 2 2 , ( 48 ) where the first inequality follows Cauchy–Schwartz inequality , and the second inequality follows Jensen ’ s inequality . In ( 48 ) , we have K∑ k=K0 ( C1 αk βk + C2K0βk−K0 ) 2 ≤ K−K0∑ k=0 ( C1 αk βk + C2K0βk ) 2 = C21 K−K0∑ k=0 α2k β2k + 2C1C2K0 K−K0∑ k=0 αk + C 2 2K 2 0 K−K0∑ k=0 β2k = O ( K2 ( σ2−σ1 ) +1 ) +O ( K0K −σ1+1 ) +O ( K20K 1−2σ2 ) ( 49 ) where the first inequality is due to the fact that αkβk and βk−K0 are monotonically decreasing . Substituting ( 49 ) into ( 48 ) gives I4 ≤ √ O ( K2 ( σ2−σ1 ) +1 ) +O ( K0K−σ1+1 ) +O ( K20K1−2σ2 ) √√√√ K∑ k=MK E ‖ωk − ω∗k‖ 2 2 . ( 50 ) Substituting ( 45 ) , ( 46 ) , ( 47 ) and ( 50 ) into ( 44 ) , and dividing both sides of ( 44 ) by K −K0 + 1 give 2λ 1 K −K0 + 1 K∑ k=K0 E ‖ωk − ω∗k‖ 2 2 ≤ √ O ( K2 ( σ2−σ1 ) +1 ) +O ( K0K−σ1+1 ) +O ( K20K1−2σ2 ) K −K0 + 1 √√√√ K∑ k=K0 E ‖ωk − ω∗k‖ 2 2 +O ( 1 K1−σ2 ) +O ( 1 Kσ2 ) +O ( K0 Kσ1 ) . ( 51 ) We define the following functions : T1 ( K ) : = 1 K −K0 + 1 K∑ k=K0 E ‖ωk − ω∗k‖ 2 2 , T2 ( K ) : = O ( 1 K1−σ2 ) +O ( 1 Kσ2 ) +O ( K0 Kσ1 ) , T3 ( K ) : = O ( K2 ( σ2−σ1 ) +1 ) +O ( K0K −σ1+1 ) +O ( K20K 1−2σ2 ) K −K0 + 1 . Then ( 51 ) can be written as : T1 ( K ) − 1 2λ √ T1 ( K ) √ T3 ( K ) ≤ 1 2λ T2 ( K ) . Solving this quadratic inequality in terms of T1 ( K ) , we obtain T1 ( K ) ≤ 1 λ T2 ( K ) + 1 2λ2 T3 ( K ) , ( 52 ) which implies 1 K −K0 + 1 K∑ k=K0 E ‖ωk − ω∗k‖ 2 2 = O ( 1 K1−σ2 ) +O ( 1 K2 ( σ1−σ2 ) ) +O ( K20 K2σ2 ) +O ( K0 Kσ1 ) +O ( 1 Kσ2 ) . We further have 1 K K∑ k=1 E ‖ωk − ω∗k‖22 ≤ 1 K ( K0−1∑ k=1 4R2ω + K∑ k=K0 E ‖ωk − ω∗k‖22 ) = K0 − 1 K 4R2ω + K −K0 + 1 K 1 K −K0 + 1 K∑ k=K0 E ‖ωk − ω∗k‖22 = O ( K0 K ) +O ( 1 K −K0 + 1 K∑ k=K0 E ‖ωk − ω∗k‖ 2 2 ) = O ( 1 K −K0 + 1 K∑ k=K0 E ‖ωk − ω∗k‖ 2 2 ) ( 53 ) which completes the proof . B.2 PROOF OF THEOREM 2 We first clarify the notations : x : = ( s , a , s′ ) , δ̂ ( x , ω ) : = r ( s , a , s′ ) + γφ ( s′ ) > ω − φ ( s ) > ω , δ ( x , θ ) : = r ( s , a , s′ ) + γVπθ ( s ′ ) − Vπθ ( s ) . The update in Algorithm 1 can be written compactly as : θk+1 = θk + αk δ̂ ( x ( k ) , ωk−τk ) ψθk−τk ( s ( k ) , a ( k ) ) . ( 54 ) For brevity , we use ω∗k as shorthand notation of ω ∗ θk . Then we are ready to give the proof . Proof . From LJ -Lipschitz of policy gradient shown in Proposition 1 , we have : J ( θk+1 ) ≥ J ( θk ) + 〈∇J ( θk ) , θk+1 − θk〉 − LJ 2 ‖θk+1 − θk‖22 = J ( θk ) + αk 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ωk−τk ) − δ̂ ( x ( k ) , ω∗k ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 + αk 〈 ∇J ( θk ) , δ̂ ( x ( k ) , ω∗k ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 − LJ 2 α2k‖δ̂ ( x ( k ) , ωk−τk ) ψθk−τk ( s ( k ) , a ( k ) ) ‖ 2 2 ≥ J ( θk ) + αk 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ωk−τk ) − δ̂ ( x ( k ) , ω∗k ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 + αk 〈 ∇J ( θk ) , δ̂ ( x ( k ) , ω∗k ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 − LJ 2 C2pα 2 k , where the last inequality follows the definition of Cp in ( 39 ) . Taking expectation on both sides of the last inequality yields E [ J ( θk+1 ) ] ≥ E [ J ( θk ) ] + αk E 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ωk−τk ) − δ̂ ( x ( k ) , ω∗k ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 I1 + αk E 〈 ∇J ( θk ) , δ̂ ( x ( k ) , ω∗k ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 I2 −LJ 2 C2pα 2 k. ( 55 ) We first decompose I1 as I1 = E 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ωk−τk ) − δ̂ ( x ( k ) , ω∗k ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 = E 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ωk−τk ) − δ̂ ( x ( k ) , ωk ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 I ( 1 ) 1 + E 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ωk ) − δ̂ ( x ( k ) , ω∗k ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 I ( 2 ) 1 . We bound I ( 1 ) 1 as I ( 1 ) 1 = E 〈 ∇J ( θk ) , ( γφ ( s′ ( k ) ) − φ ( s ( k ) ) ) > ( ωk−τk − ωk ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 ≥ −E [ ‖∇J ( θk ) ‖2‖γφ ( s′ ( k ) ) − φ ( s ( k ) ) ‖2‖ωk − ωk−τk‖2‖ψθk−τk ( s ( k ) , a ( k ) ) ‖2 ] ≥ −2Cψ E [ ‖∇J ( θk ) ‖2‖ωk − ωk−τk‖2 ] ≥ −2CψCδK0βk−1 E ‖∇J ( θk ) ‖2 , where the last inequality follows ‖ωk − ωk−τk‖2 = ∥∥∥∥∥ k−1∑ i=k−τk ( ωi+1 − ωi ) ∥∥∥∥∥ 2 ≤ k−1∑ i=k−τk ‖βig ( xi , ωi−τi ) ‖2 ≤ βk−1 k−1∑ i=k−τk ‖g ( xi , ωi−τi ) ‖2 ≤ βk−1K0Cδ , where the second inequality is due to the monotonicity of step size , and the third one follows ( 32 ) . Then we bound I ( 2 ) 1 as I ( 2 ) 1 = E 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ωk ) − δ̂ ( x ( k ) , ω∗k ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 = −E 〈 ∇J ( θk ) , ( γφ ( s′ ( k ) ) − φ ( s ( k ) ) ) > ( ω∗k − ωk ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 ≥ −E [ ‖∇J ( θk ) ‖2‖γφ ( s′ ( k ) ) − φ ( s ( k ) ) ‖2‖ωk − ω ∗ k‖2‖ψθk−τk ( s ( k ) , a ( k ) ) ‖2 ] ≥ −2Cψ E [ ‖∇J ( θk ) ‖2‖ωk − ω∗k‖2 ] . Collecting the lower bounds of I ( 1 ) 1 and I ( 2 ) 1 gives I1 ≥ −2Cψ E [ ‖∇J ( θk ) ‖2 ( CδK0βk−1 + ‖ωk − ω∗k‖2 ) ] . ( 56 ) Now we consider I2 . We first decompose I2 as I2 = E 〈 ∇J ( θk ) , δ̂ ( x ( k ) , ω∗k ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 = E 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ω ∗ k ) − δ̂ ( x ( k ) , ω∗k−τk ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 I ( 1 ) 2 + E 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ω ∗ k−τk ) − δ ( x ( k ) , θk−τk ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 I ( 2 ) 2 + E 〈 ∇J ( θk ) , δ ( x ( k ) , θk−τk ) ψθk−τk ( s ( k ) , a ( k ) ) −∇J ( θk ) 〉 I ( 3 ) 2 +‖∇J ( θk ) ‖22 . We bound I ( 1 ) 2 as I ( 1 ) 2 = E 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ω ∗ k ) − δ̂ ( x ( k ) , ω∗k−τk ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 = E 〈 ∇J ( θk ) , ( γφ ( s′ ( k ) ) − φ ( s ( k ) ) ) > ( ω∗k − ω∗k−τk ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 ≥ −E [ ‖∇J ( θk ) ‖2‖ ( γφ ( s′ ( k ) ) − φ ( s ( k ) ) ) > ‖2 ∥∥ω∗k − ω∗k−τk∥∥2 ‖ψθk−τk ( s ( k ) , a ( k ) ) ‖2 ] ≥ −LV Cψ ( 1 + γ ) E ∥∥ω∗k − ω∗k−τk∥∥2 ≥ −LV LωCψ ( 1 + γ ) E ‖θk − θk−τk‖2 ≥ −LV LωCψCp ( 1 + γ ) K0αk−K0 , where the second last inequality follows from Proposition 2 and the last inequality uses ( 39 ) as ‖θk − θk−τk‖2 ≤ k−1∑ i=k−τk ‖θi+1 − θi‖2 = k−1∑ i=k−τk αi‖δ̂ ( xi , ωi−τi ) ψθi−τi ( si , ai ) ‖2 ≤ k−1∑ i=k−τk αk−τkCp ≤ CpK0αk−K0 . ( 57 ) We bound I ( 2 ) 2 as I ( 2 ) 2 = E 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ω ∗ k−τk ) − δ ( x ( k ) , θk−τk ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 ≥ −E [ ‖∇J ( θk ) ‖2 ∣∣∣δ̂ ( x ( k ) , ω∗k−τk ) − δ ( x ( k ) , θk−τk ) ∣∣∣ ‖ψθk−τk ( s ( k ) , a ( k ) ) ‖2 ] ≥ −Cψ E [ ‖∇J ( θk ) ‖2 ∣∣∣δ̂ ( x ( k ) , ω∗k−τk ) − δ ( x ( k ) , θk−τk ) ∣∣∣ ] = −Cψ E [ ‖∇J ( θk ) ‖2 ∣∣∣γ ( φ ( s′ ( k ) ) > ω∗k−τk − Vπθk−τk ( s′ ( k ) ) ) + Vπθk−τk ( s ( k ) ) − φ ( s ( k ) ) > ω∗k−τk ∣∣∣ ] ≥ −Cψ E [ ‖∇J ( θk ) ‖2 ( γ ∣∣∣φ ( s′ ( k ) ) > ω∗k−τk − Vπθk−τk ( s′ ( k ) ) ∣∣∣+ ∣∣∣Vπθk−τk ( s ( k ) ) − φ ( s ( k ) ) > ω∗k−τk ∣∣∣ ) ] = −Cψ E [ ‖∇J ( θk ) ‖2 E [ γ ∣∣∣φ ( s′ ( k ) ) > ω∗k−τk − Vπθk−τk ( s′ ( k ) ) ∣∣∣+ ∣∣∣Vπθk−τk ( s ( k ) ) − φ ( s ( k ) ) > ω∗k−τk ∣∣∣∣∣∣ θk , θk−τk ] ] ≥ −2Cψ app E ‖∇J ( θk ) ‖2 ≥ −2CψLV fa − 2Cψ sp E ‖∇J ( θk ) ‖2 ( 58 ) where the second last inequality follows from the fact that E [ γ ∣∣∣φ ( s′ ( k ) ) > ω∗k−τk − Vπθk−τk ( s′ ( k ) ) ∣∣∣+ ∣∣∣Vπθk−τk ( s ( k ) ) − φ ( s ( k ) ) > ω∗k−τk ∣∣∣ ] ≤ γ √ E ∣∣∣φ ( s′ ( k ) ) > ω∗k−τk − Vπθk−τk ( s′ ( k ) ) ∣∣∣2 + √ E ∣∣∣Vπθk−τk ( s ( k ) ) − φ ( s ( k ) ) > ω∗k−τk ∣∣∣2 ≤ 2 app . Define artificial transition x̄ ( k ) : = ( s ( k ) , a ( k ) , s̄′ ( k ) ∼ P ) , then I ( 3 ) 2 can be bounded as I ( 3 ) 2 = E 〈 ∇J ( θk ) , δ ( x ( k ) , θk−τk ) ψθk−τk ( s ( k ) , a ( k ) ) −∇J ( θk ) 〉 = E [ E [ 〈 ∇J ( θk ) , δ ( x ( k ) , θk−τk ) ψθk−τk ( s ( k ) , a ( k ) ) −∇J ( θk ) 〉∣∣∣ θk−τk , θk ] ] = E 〈 ∇J ( θk ) , E [ ( δ ( x ( k ) , θk−τk ) − δ ( x̄ ( k ) , θk−τk ) ) ψθk−τk ( s ( k ) , a ( k ) ) ∣∣∣ θk−τk , θk ] 〉 + E 〈 ∇J ( θk ) , E [ δ ( x̄ ( k ) , θk−τk ) ψθk−τk ( s ( k ) , a ( k ) ) ∣∣∣ θk−τk , θk ] −∇J ( θk ) 〉 ≥ −E [ ‖∇J ( θk ) ‖2 ∥∥∥E [ ( δ ( x ( k ) , θk−τk ) − δ ( x̄ ( k ) , θk−τk ) ) ψθk−τk ( s ( k ) , a ( k ) ) ∣∣∣ θk−τk , θk ] ∥∥∥2 ] − E [ ‖∇J ( θk ) ‖2 ∥∥∥E [ δ ( x̄ ( k ) , θk−τk ) ψθk−τk ( s ( k ) , a ( k ) ) ∣∣∣ θk−τk , θk ] −∇J ( θk ) ∥∥∥2 ] . ( 59 ) The first term in the last inequality can be bounded as E [ ( δ ( x ( k ) , θk−τk ) − δ ( x̄ ( k ) , θk−τk ) ) ψθk−τk ( s ( k ) , a ( k ) ) ∣∣∣ θk−τk , θk ] = E [ ( δ ( x ( k ) , θk−τk ) − δ ( x̄ ( k ) , θk−τk ) ) ψθk−τk ( s ( k ) , a ( k ) ) ∣∣∣ θk−τk , θk ] = E [ ( r ( x ( k ) ) + γ E [ r ( s′k , a′ , s′′ ) ] − ( r ( x̄ ( k ) ) + γ E [ r ( s̄′k , a′ , s′′ ) ] ) ) ψθk−τk ( s ( k ) , a ( k ) ) ∣∣∣ θk−τk , θk ] ≤ 2Cψrmax‖P̃ − P‖TV ≤ 8Cψrmax ( 1− γ ) , ( 60 ) where the last inequality follows ‖P̃ − P‖TV = 2 ∫ s′∈S ∣∣∣P̃ ( s′|s , a ) − P ( s′|s , a ) ∣∣∣ = 2 ( 1− γ ) ∫ s′∈S |P ( s′|s , a ) − η ( s′ ) | ≤ 4 ( 1− γ ) . ( 61 ) The second term in ( 59 ) can be rewritten as E [ δ ( x̄ ( k ) , θk−τk ) ψθk−τk ( s ( k ) , a ( k ) ) ∣∣∣ θk−τk , θk ] = E s ( k ) ∼µθk−τk a ( k ) ∼πθk−τk s̄′ ( k ) ∼P [ ( r ( x̄ ( k ) ) + γVπθk−τk ( s̄′ ( k ) ) − Vπθk−τk ( s ( k ) ) ) ψθk−τk ( s ( k ) , a ( k ) ) ∣∣∣∣θk−τk , θk ] = E s ( k ) ∼µθk−τk a ( k ) ∼πθk−τk [ ( Qπθk−τk ( s ( k ) , a ( k ) ) − Vπθk−τk ( s ( k ) ) ) ψθk−τk ( s ( k ) , a ( k ) ) ∣∣∣∣θk−τk , θk ] = E s ( k ) ∼µθk−τk a ( k ) ∼πθk−τk [ Aπθk−τk ( s ( k ) , a ( k ) ) ψθk−τk ( s ( k ) , a ( k ) ) ∣∣∣∣θk−τk , θk ] = E s ( k ) ∼dθk−τk a ( k ) ∼πθk−τk [ Aπθk−τk ( s ( k ) , a ( k ) ) ψθk−τk ( s ( k ) , a ( k ) ) ∣∣∣∣θk−τk , θk ] = ∇J ( θk−τk ) ( 62 ) where the second last equality follows µθ ( · ) = dθ ( · ) with dθ being a shorthand notation of dπθ [ 6 ] . Substituting ( 60 ) and ( 62 ) into ( 59 ) yields I ( 3 ) 2 ≥ −8Cψrmax ( 1− γ ) E ‖∇J ( θk ) ‖2 − E [ ‖∇J ( θk ) ‖2‖∇J ( θk−τk ) −∇J ( θk ) ‖2 ] ≥ −8Cψrmax ( 1− γ ) E ‖∇J ( θk ) ‖2 − LV LJ E ‖θk−τk − θk‖2 ≥ −8Cψrmax ( 1− γ ) E ‖∇J ( θk ) ‖2 − LV LJCpK0αk−K0 , ( 63 ) where the second last inequality is due to LJ -Lipschitz of policy gradient shown in Proposition 1 , and the last inequality follows ( 57 ) . Collecting lower bounds of I ( 1 ) 2 , I ( 2 ) 2 and I ( 3 ) 2 gives I2 ≥ −D1K0αk−K0 − ( 2Cψ sp + 8Cψrmax ( 1− γ ) ) E ‖∇J ( θk ) ‖2 − 2CψLV fa + ‖∇J ( θk ) ‖ 2 2 , ( 64 ) where the constant is D1 : = LV LωCψCp ( 1 + γ ) + LV LJCp . Substituting ( 56 ) and ( 64 ) into ( 55 ) yields E [ J ( θk+1 ) ] ≥ E [ J ( θk ) ] − 2αkCψ ( sp + 4rmax ( 1− γ ) + CδK0βk−1 + ‖ωk − ω∗k‖2 ) E ‖∇J ( θk ) ‖2 − αkD1K0αk−K0 − 2αkCψLV fa + αk‖∇J ( θk ) ‖22 − LJ 2 C2pα 2 k. ( 65 ) By following Cauchy-Schwarz inequality , the second term in ( 65 ) can be bounded as ( sp + 4rmax ( 1− γ ) + CδK0βk−1 + ‖ωk − ω∗k‖2 ) E ‖∇J ( θk ) ‖2 ≤ √ E ‖∇J ( θk ) ‖22 E [ ( sp + 4rmax ( 1− γ ) + CδK0βk−1 + ‖ωk − ω∗k‖2 ) 2 ] ≤ √ E ‖∇J ( θk ) ‖22 √ E [ 4C2δK 2 0β 2 k−1 + 4‖ωk − ω∗k‖22 + 4 2sp + 64r2max ( 1− γ ) 2 ] = 2 √ E ‖∇J ( θk ) ‖22 √ C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O ( 2sp ) , ( 66 ) where the last inequality follows the order of sp in Lemma 7 . Collecting the upper bound gives E [ J ( θk+1 ) ] ≥ E [ J ( θk ) ] − 4αkCψ √ E ‖∇J ( θk ) ‖22 √ C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O ( 2sp ) − αkD1K0αk−K0 − 2αkCψLV fa + αk‖∇J ( θk ) ‖22 − LJ 2 C2pα 2 k. ( 67 ) Dividing both sides of ( 67 ) by αk , then rearranging and taking summation on both sides give K∑ k=K0 E ‖∇J ( θk ) ‖22 ≤ K∑ k=K0 1 αk ( E [ J ( θk+1 ) ] − E [ J ( θk ) ] ) I3 + K∑ k=K0 ( D1K0αk−K0 + LJ 2 C2pαk ) I4 + 4Cψ K∑ k=K0 √ E ‖∇J ( θk ) ‖22 √ C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O ( 2sp ) I5 + 2CψLV ( K −K0 + 1 ) fa . ( 68 ) We bound I3 as I3 = K∑ k=K0 1 αk ( E [ J ( θk+1 ) ] − E [ J ( θk ) ] ) = K∑ k=K0 ( 1 αk−1 − 1 αk ) E [ J ( θk ) ] − 1 αMK−1 E [ J ( θMK ) ] + 1 αK E [ J ( θK+1 ) ] ≤ 1 αK E [ J ( θK+1 ) ] ≤ rmax 1− γ 1 αK = O ( Kσ1 ) , ( 69 ) where the first inequality is due to the αk is monotonic decreasing and positive , and last inequality is due to Vπθ ( s ) ≤ rmax1−γ for any s ∈ S and πθ . We bound I4 as I4 = K∑ k=K0 ( D1K0αk−K0 + LJ 2 C2pαk ) ≤ K−K0∑ k=0 ( D1K0αk + LJ 2 C2pαk ) = O ( K0K1−σ1 ) . We bound I5 as I5 = K∑ k=K0 √ E ‖∇J ( θk ) ‖22 √ C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O ( 2sp ) ≤ √√√√ K∑ k=K0 E ‖∇J ( θk ) ‖22 √√√√ K∑ k=K0 ( C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O ( 2sp ) ) = √√√√ K∑ k=K0 E ‖∇J ( θk ) ‖22 √√√√C2δK20 K∑ k=K0 β2k−1 + K∑ k=K0 E ‖ωk − ω∗k‖22 +O ( K 2sp ) , ( 70 ) where the first inequality follows Cauchy-Schwartz inequality . In ( 70 ) , we have K∑ k=K0 β2k−1 ≤ K−K0∑ k=0 β2k = K−K0∑ k=0 c22 ( 1 + k ) −2σ2 = O ( K1−2σ2 ) . Substituting the last equality into ( 70 ) gives I5 ≤ √√√√ K∑ k=MK E ‖∇J ( θk ) ‖22 √√√√O ( K20K1−2σ2 ) + K∑ k=MK E ‖ωk − ω∗k‖22 +O ( K 2sp ) . ( 71 ) Dividing both sides of ( 67 ) by K −K0 + 1 and collecting upper bounds of I3 , I4 and I5 give 1 K −K0 + 1 K∑ k=K0 E ‖∇J ( θk ) ‖22 ≤ 4Cψ K −K0 + 1 √√√√ K∑ k=K0 E ‖∇J ( θk ) ‖22 √√√√O ( K20K1−2σ2 ) + K∑ k=K0 E ‖ωk − ω∗k‖22 +O ( K 2sp ) +O ( 1 K1−σ1 ) +O ( K0 Kσ1 ) +O ( fa ) . ( 72 ) Define the following functions T4 ( K ) : = 1 K −K0 + 1 K∑ k=K0 E ‖∇J ( θk ) ‖22 , T5 ( K ) : = 1 K −K0 + 1 ( O ( K20K1−2σ2 ) + K∑ k=K0 E ‖ωk − ω∗k‖22 +O ( K 2sp ) ) , T6 ( K ) : = O ( 1 K1−σ1 ) +O ( K0 Kσ1 ) +O ( fa ) . Then ( 72 ) can be rewritten as T4 ( K ) ≤ T6 ( K ) + √ 2 ( 1 + γ ) Cψ √ T4 ( K ) √ T5 ( K ) . Solving this quadratic inequality in terms of T4 ( K ) , we obtain T4 ( K ) ≤ 2T6 ( K ) + 4 ( 1 + γ ) 2C2ψT5 ( K ) , ( 73 ) which implies 1 K −K0 + 1 K∑ k=K0 E ‖∇J ( θk ) ‖22 = O ( 1 K1−σ1 ) +O ( K0 Kσ1 ) +O ( K20 K2σ2 ) +O ( 1 K −K0 + 1 K∑ k=K0 E ‖ωk − ω∗k‖22 ) +O ( app ) . We further have 1 K K∑ k=1 E ‖∇J ( θk ) ‖22 ≤ 1 K ( K0−1∑ k=1 L2V + K∑ k=K0 E ‖∇J ( θk ) ‖22 ) = K0 − 1 K L2V + K −K0 + 1 K 1 K −K0 + 1 K∑ k=K0 E ‖∇J ( θk ) ‖22 = O ( K0 K ) +O ( 1 K −K0 + 1 K∑ k=K0 E ‖∇J ( θk ) ‖22 ) = O ( 1 K −K0 + 1 K∑ k=K0 E ‖∇J ( θk ) ‖22 ) ( 74 ) which completes the proof . B.3 PROOF OF THEOREM 3 Given the definition in Section B.1 , we now give the convergence proof of critic update in Algorithm 1 with linear function approximation and Markovian sampling . By following the derivation of ( 40 ) , we have E ‖ωk+1 − ω∗k+1‖22 ≤ ( 1− 2λβk ) E ‖ωk − ω∗k‖ 2 2 + 2βk ( C1 αk βk + C2K0βk−K0 ) E ‖ωk − ω∗k‖2 + 2βk E 〈 ωk − ω∗k , g ( x ( k ) , ωk ) − g ( θk , ωk ) 〉 + Cqβ 2 k , ( 75 ) where C1 : = CpLω , C2 : = Cδ ( 1 + γ ) and Cq : = 2C2δ + 2L 2 ωC 2 p max ( k ) α2k β2k = 2C2δ + 2L 2 ωC 2 p c21 c22 . Now we consider the third item in the last inequality . For some m ∈ N+ , we define M : = ( K0 + 1 ) m+K0 . Following Lemma 4 ( to be presented in Sec . C.1 ) , for some dm ≤M and positive constants C4 , C5 , C6 , C7 , we have E 〈 ωk − ω∗k , g ( x ( k ) , ωk ) − g ( θk , ωk ) 〉 ≤ C4 E ‖θk − θk−dm‖2 + C5 dm∑ i=τk E ‖θk−i − θk−dm‖2 + C6 E ‖ωk − ωk−dm‖2 + C7κρm−1 ≤ C4 k−1∑ i=k−dm E ‖θi+1 − θi‖2 + C5 dm−1∑ i=τk k−i−1∑ j=k−dm E ‖θj+1 − θj‖2 + C6 k−1∑ i=k−dm E ‖ωi+1 − ωi‖2 + C7κρm−1 ≤ C4 k−1∑ i=k−dm αiCp + C5 dm−1∑ i=τk k−i−1∑ j=k−dm αjCp + C6 k−1∑ i=k−dm βiCδ + C7κρ m−1 ≤ C4αk−dm k−1∑ i=k−dm Cp + C5αk−dm dm−1∑ i=τk k−i−1∑ j=k−dm Cp + C6βk−dm k−1∑ i=k−dm Cδ + C7κρ m−1 ≤ C4dmCpαk−dm + C5 ( dm − τk ) 2Cpαk−dm + C6dmCδβk−dm + C7κρm−1 ≤ ( C4M + C5M 2 ) Cpαk−M + C6MCδβk−M + C7κρ m−1 , ( 76 ) where the third last inequality is due to the monotonicity of step size , and the last inequality is due to τk ≥ 0 and dm ≤M . Further letting m = mK which is defined in ( 21 ) yields E 〈 ωk − ω∗k , g ( x ( k ) , ωk ) − g ( θk , ωk ) 〉 = ( C4MK + C5M 2 K ) Cpαk−MK + C6CδMKβk−MK + C7κρ mK−1 ≤ ( C4MK + C5M 2 K ) Cpαk−MK + C6CδMKβk−MK + C7αK , ( 77 ) where MK = ( K0 + 1 ) mK +K0 , and the last inequality follows the definition of mK . Substituting ( 77 ) into ( 75 ) , then rearranging and summing up both sides over k = MK , ... , K yield 2λ K∑ k=MK E ‖ωk − ω∗k‖ 2 2 ≤ K∑ k=MK 1 βk ( E ‖ωk − ω∗k‖ 2 2 − E ∥∥ωk+1 − ω∗k+1∥∥22 ) I1 +Cq K∑ k=MK βk I2 + 2 K∑ k=MK ( ( C4MK + C5M 2 K ) Cpαk−MK + C6CδMKβk−MK + C7αK ) I3 + 2 K∑ k=MK ( C1 αk βk + C2K0βk−K0 ) E ‖ωk − ω∗k‖2 I4 . ( 78 ) where the order of I1 , I2 and I4 have already been given by ( 45 ) , ( 46 ) and ( 50 ) respectively . We bound I3 as I3 = ( C4MK + C5M 2 K ) Cp K∑ k=MK αk + C6CδMK K∑ k=MK βk + C7αK K∑ k=MK 1 ≤ ( C4MK + C5M 2 K ) Cpc1 K1−σ1 1− σ1 + C6CδMKc2 K1−σ2 1− σ2 + C7c1K ( 1 +K ) −σ1 = O ( ( K20 log 2K ) K1−σ1 ) +O ( ( K0 logK ) K 1−σ2 ) , ( 79 ) where the last inequality follows from the integration rule ∑b k=a k −σ ≤ b 1−σ 1−σ , and the last equality is due to O ( MK ) = O ( K0mK ) = O ( K0 logK ) . Collecting the bounds of I1 , I2 , I3 and I4 , and dividing both sides of ( 78 ) by K −MK + 1 yield 2λ 1 K −MK + 1 K∑ k=MK E ‖ωk − ω∗k‖ 2 2 ≤ √ O ( K2 ( σ2−σ1 ) +1 ) +O ( K0K−σ1+1 ) +O ( K20K1−2σ2 ) K −MK + 1 √√√√ K∑ k=MK E ‖ωk − ω∗k‖ 2 2 +O ( 1 K1−σ2 ) +O ( K20 log 2K Kσ1 ) +O ( K0 logK Kσ2 ) . ( 80 ) Similar to the derivation of ( 52 ) , ( 80 ) implies 1 K −MK + 1 K∑ k=MK E ‖ωk − ω∗k‖ 2 2 = O ( 1 K1−σ2 ) +O ( 1 K2 ( σ1−σ2 ) ) +O ( K20 K2σ2 ) +O ( K20 log 2K Kσ1 ) +O ( K0 logK Kσ2 ) . Similar to ( 53 ) , we have 1 K K∑ k=1 E ‖ωk − ω∗k‖22 = O ( K0 logK K ) +O ( 1 K −MK + 1 K∑ k=MK E ‖ωk − ω∗k‖ 2 2 ) = O ( 1 K −MK + 1 K∑ k=MK E ‖ωk − ω∗k‖ 2 2 ) ( 81 ) which completes the proof . B.4 PROOF OF THEOREM 4 Given the definition in section B.2 , we now give the convergence proof of actor update in Algorithm 1 with linear value function approximation and Markovian sampling method . By following the derivation of ( 55 ) , we have E [ J ( θk+1 ) ] ≥ E [ J ( θk ) ] + αk E 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ωk−τk ) − δ̂ ( x ( k ) , ω∗k ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 I1 + αk E 〈 ∇J ( θk ) , δ̂ ( x ( k ) , ω∗k ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 I2 −LJ 2 C2pα 2 k. ( 82 ) The item I1 can be bounded by following ( 56 ) as I1 ≥ −2Cψ E [ ‖∇J ( θk ) ‖2 ( CδK0βk−1 + ‖ωk − ω∗k‖2 ) ] . ( 83 ) Next we consider I2 . We first decompose it as I2 = E 〈 ∇J ( θk ) , δ̂ ( x ( k ) , ω∗k ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 = E 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ω ∗ k ) − δ ( x ( k ) , θk ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 I ( 1 ) 2 + E 〈 ∇J ( θk ) , δ ( x ( k ) , θk ) ψθk−τk ( s ( k ) , a ( k ) ) −∇J ( θk ) 〉 I ( 2 ) 2 +E ‖∇J ( θk ) ‖22 . ( 84 ) For some m ∈ N+ , define M : = ( K0 + 1 ) m + K0 . Following Lemma 5 , for some dm ≤ M and positive constants D2 , D3 , D4 , D5 , I ( 1 ) 2 can be bounded as I ( 1 ) 2 = E 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ω ∗ k ) − δ ( x ( k ) , θk ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 ≥ −D2 E ‖θk−τk − θk−dm‖2 −D3 E ‖θk − θk−dm‖2 −D4 k−τk∑ i=k−dm E ‖θi − θk−dm‖2 −D5κρm−1 − 2CψLV fa − 2Cψ sp E ‖∇J ( θk ) ‖2 ≥ −D2 ( dm − τk ) Cpαk−dm −D3dmCpαk−dm −D4 ( dm − τk ) 2Cpαk−dm −D5κρm−1 − 2CψLV fa − 2Cψ sp E ‖∇J ( θk ) ‖2 , ( 85 ) where the derivation of the last inequality is similar to that of ( 76 ) . By setting m = mK in ( 85 ) , and following the fact that dmK ≤MK and τk ≥ 0 , we have I ( 1 ) 2 ≥ −D2MKCpαk−MK −D3MKCpαk−MK −D4M2KCpαk−MK −D5κρmK−1 − 2CψLV fa − 2Cψ sp E ‖∇J ( θ ) ‖2 = − ( ( D2 +D3 ) CpMK +D4CpM 2 K ) αk−MK −D5κρmK−1 − 2CψLV fa − 2Cψ sp E ‖∇J ( θk ) ‖2 ≥ − ( ( D2 +D3 ) CpMK +D4CpM 2 K ) αk−MK −D5αK − 2CψLV fa − 2Cψ sp E ‖∇J ( θk ) ‖2 , ( 86 ) where the last inequality is due to the definition of mK . Following Lemma 6 , for some positive constants D6 , D7 , D8 and D9 , we bound I ( 2 ) 2 as I ( 2 ) 2 = E 〈 ∇J ( θk ) , δ ( x ( k ) , θk ) ψθk−τk ( s ( k ) , a ( k ) ) −∇J ( θk ) 〉 ≥ −D6 E ‖θk−τk − θk−dm‖2 −D7 E ‖θk − θk−dm‖2 −D8 dm∑ i=τk E ‖θk−i − θk−dm‖2 −D9κρm−1 − 8Cψrmax ( 1− γ ) E ‖∇J ( θk ) ‖2 . Similar to the derivation of ( 86 ) , we have I ( 2 ) 2 ≥ − ( D6 +D7 +D8MK ) CpMKαk−MK −D9αK − 8Cψrmax ( 1− γ ) E ‖∇J ( θk ) ‖2 . ( 87 ) Collecting the lower bounds of I ( 1 ) 2 and I ( 2 ) 2 yields I2 ≥ −2CψLV fa − 2Cψ ( sp + 4rmax ( 1− γ ) ) E ‖∇J ( θk ) ‖2 + E ‖∇J ( θk ) ‖22 −DKαk−MK − ( D5 +D9 ) αK , ( 88 ) where we define DK : = ( D4 +D8 ) CpM2K + ( D2 +D3 +D6 +D7 ) CpMK for brevity . Substituting ( 83 ) and ( 88 ) into ( 82 ) yields E [ J ( θk+1 ) ] ≥ E [ J ( θk ) ] − 2αkCψ E [ ‖∇J ( θk ) ‖2 ( sp + 4rmax ( 1− γ ) + CδK0βk−1 + ‖ωk − ω∗k‖2 ) ] − αk ( DKαk−MK + ( D5 +D9 ) αK ) − 2CψLV faαk + αk E ‖∇J ( θk ) ‖22 − LJ 2 C2pα 2 k. Similar to the derivation of ( 67 ) , the last inequality implies E [ J ( θk+1 ) ] ≥ E [ J ( θk ) ] − 4αkCψ √ E ‖∇J ( θk ) ‖22 √ C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O ( 2sp ) − αk ( DKαk−MK + ( D5 +D9 ) αK ) − 2CψLV faαk + αk E ‖∇J ( θk ) ‖22 − LJ 2 C2pα 2 k. Rearranging and dividing both sides by αk yield E ‖∇J ( θk ) ‖22 ≤ 1 αk ( E [ J ( θk+1 ) ] − E [ J ( θk ) ] ) +DKαk−MK + ( D5 +D9 ) αK + LJ 2 C2pαk + 4Cψ √ E ‖∇J ( θk ) ‖22 √ C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O ( 2sp ) + 2CψLV fa . Taking summation gives K∑ k=MK E ‖∇J ( θk ) ‖22 ≤ K∑ k=MK 1 αk ( E [ J ( θk+1 ) ] − E [ J ( θk ) ] ) I3 + K∑ k=MK ( DKαk−MK + LJ 2 C2pαk + ( D5 +D9 ) αK ) I4 + 4Cψ K∑ k=MK √ E ‖∇J ( θk ) ‖22 √ C2δK 2 0β 2 k−1 + E ‖ωk − ω∗k‖22 +O ( 2sp ) I5 + 2CψLV ( K −MK + 1 ) fa . ( 89 ) in which the upper bounds of I3 and I5 have already been given by ( 69 ) and ( 71 ) respectively . We bound I4 as I4 = K∑ k=MK ( DKαk−MK + LJ 2 C2pαk + ( D5 +D9 ) αK ) ≤ K∑ k=MK ( DKαk−MK + LJ 2 C2pαk−MK + ( D5 +D9 ) αK ) = ( DK + LJ 2 C2p ) K∑ k=MK αk−MK + ( D5 +D9 ) ( K −MK + 1 ) αK = ( DK + LJ 2 C2p ) K−MK∑ k=0 αk + ( D5 +D9 ) ( K −MK + 1 ) αK ≤ ( DK + LJ 2 C2p ) c1 1− σ1 K1−σ1 + c1 ( D5 +D9 ) ( K + 1 ) 1−σ1 = O ( ( K20 log 2K ) K1−σ1 ) ( 90 ) where the last inequality uses ∑b k=a k −σ ≤ b 1−σ 1−σ , and the last equality is due to the fact that O ( DK ) = O ( M2K +MK ) = O ( ( K0mK ) 2 +K0mK ) = O ( K20 log 2K ) . Substituting the upper bounds of I3 , I4 and I5 into ( 89 ) , and dividing both sides by K−MK + 1 give 1 K −MK + 1 K∑ k=MK E ‖∇J ( θk ) ‖22 ≤ 4Cψ K −MK + 1 √√√√ K∑ k=MK E ‖∇J ( θk ) ‖22 √√√√O ( K20K1−2σ2 ) + K∑ k=MK E ‖ωk − ω∗k‖22 +O ( K 2sp ) +O ( 1 K1−σ1 ) +O ( K20 log 2K Kσ1 ) +O ( fa ) . ( 91 ) Following the similar steps of those in ( 73 ) , ( 91 ) essentially implies 1 K −MK + 1 K∑ k=MK E ‖∇J ( θk ) ‖22 = O ( 1 K1−σ1 ) +O ( K20 log 2K Kσ1 ) +O ( K20 K2σ2 ) +O ( 1 K −MK + 1 K∑ k=MK E ‖ωk − ω∗θk‖ 2 2 ) +O ( app ) . Similar to ( 74 ) , we have 1 K K∑ k=1 E ‖∇J ( θk ) ‖22 = O ( K0 logK K ) +O ( 1 K −MK + 1 K∑ k=MK E ‖∇J ( θk ) ‖22 ) = O ( 1 K −MK + 1 K∑ k=MK E ‖∇J ( θk ) ‖22 ) which completes the proof . C SUPPORTING LEMMAS C.1 SUPPORTING LEMMAS FOR THEOREM 3 Lemma 4 . For any m ≥ 1 and k ≥ ( K0 + 1 ) m+K0 + 1 , we have E 〈 ωk − ω∗θk , g ( x ( k ) , ωk ) − g ( θk , ωk ) 〉 ≤ C4 E ‖θk − θk−dm‖2 + C5 dm∑ i=τk E ‖θk−i − θk−dm‖2 + C6 E ‖ωk − ωk−dm‖2 + C7κρm−1 , where dm ≤ ( K0 + 1 ) m + K0 , and C4 : = 2CδLω + 4RωCδ|A|Lπ ( 1 + logρ κ−1 + ( 1 − ρ ) −1 ) , C5 : = 4RωCδ|A|Lπ and C6 : = 4 ( 1 + γ ) Rω + 2Cδ , C7 : = 8RωCδ . Proof . Consider the collection of random samples { x ( k−K0−1 ) , x ( k−K0 ) , ... , x ( k ) } . Suppose x ( k ) is sampled by worker n , then due to Assumption 1 , { x ( k−K0−1 ) , x ( k−K0 ) , ... , x ( k−1 ) } will contain at least another sample drawn by worker n. Therefore , { x ( k− ( K0+1 ) m ) , x ( k− ( K0+1 ) m+1 ) , ... , x ( k−1 ) } will contain at least m samples from worker n. Consider the Markov chain formed by m+ 1 samples in { x ( k− ( K0+1 ) m ) , x ( k− ( K0+1 ) m+1 ) , ... , x ( k ) } : st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−1−−−−−−→ at−m+1 · · · st−1 θk−d1−−−−→ at−1 P̃−→ st θk−d0−−−−→ at P̃−→ st+1 , where ( st , at , st+1 ) = ( s ( k ) , a ( k ) , s′ ( k ) ) , and { dj } m j=0 is some increasing sequence with d0 : = τk . Suppose θk−dm was used to do the kmth update , then we have xt−m = x ( km ) . Following Assumption 1 , we have τkm = km − ( k − dm ) ≤ K0 . Since x ( km ) is in { x ( k− ( K0+1 ) m ) , ... , x ( k ) } , we have km ≥ k − ( K0 + 1 ) m. Combining these two inequalities , we have dm ≤ ( K0 + 1 ) m+K0 . ( 92 ) Given ( st−m , at−m , st−m+1 ) and θk−dm , we construct an auxiliary Markov chain as that in Lemma 2 : st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−−−−→ ãt−m+1 · · · s̃t−1 θk−dm−−−−→ ãt−1 P̃−→ s̃t θk−dm−−−−→ ãt P̃−→ s̃t+1 . For brevity , we define ∆1 ( x , θ , ω ) : = 〈ω − ω∗θ , g ( x , ω ) − g ( θ , ω ) 〉 . Throughout this proof , we use θ , θ′ , ω , ω′ , x and x̃ as shorthand notations of θk , θk−dm , ωk , ωk−dm , xt and x̃t respectively . First we decompose ∆1 ( x , θ , ω ) as ∆1 ( x , θ , ω ) = ∆1 ( x , θ , ω ) −∆1 ( x , θ′ , ω ) I1 + ∆1 ( x , θ ′ , ω ) −∆1 ( x , θ′ , ω′ ) I2 + ∆1 ( x , θ ′ , ω′ ) −∆1 ( x̃ , θ′ , ω′ ) I3 + ∆1 ( x̃ , θ ′ , ω′ ) I4 . ( 93 ) We bound I1 in ( 93 ) as ∆1 ( x , θ , ω ) −∆1 ( x , θ′ , ω ) = 〈ω − ω∗θ , g ( x , ω ) − g ( θ , ω ) 〉 − 〈ω − ω∗θ′ , g ( x , ω ) − g ( θ′ , ω ) 〉 ≤ |〈ω − ω∗θ , g ( x , ω ) − g ( θ , ω ) 〉 − 〈ω − ω∗θ′ , g ( x , ω ) − g ( θ , ω ) 〉| + |〈ω − ω∗θ′ , g ( x , ω ) − g ( θ , ω ) 〉 − 〈ω − ω∗θ′ , g ( x , ω ) − g ( θ′ , ω ) 〉| . ( 94 ) For the first term in ( 94 ) , we have |〈ω − ω∗θ , g ( x , ω ) − g ( θ , ω ) 〉 − 〈ω − ω∗θ′ , g ( x , ω ) − g ( θ , ω ) 〉| = |〈ω∗θ − ω∗θ′ , g ( x , ω ) − g ( θ , ω ) 〉| ≤ ‖ω∗θ − ω∗θ′‖2‖g ( x , ω ) − g ( θ , ω ) ‖ ≤ 2Cδ‖ω∗θ − ω∗θ′‖2 ≤ 2CδLω‖θ − θ′‖2 , where the last inequality is due to Proposition 2 . We use x ∼ θ′ as shorthand notations to represent that s ∼ µθ′ , a ∼ πθ′ , s′ ∼ P̃ . For the second term in ( 94 ) , we have |〈ω − ω∗θ′ , g ( x , ω ) − g ( θ , ω ) 〉 − 〈ω − ω∗θ′ , g ( x , ω ) − g ( θ′ , ω ) 〉| = |〈ω − ω∗θ′ , g ( θ′ , ω ) − g ( θ , ω ) 〉| ≤ ‖ω − ω∗θ′‖2‖g ( θ′ , ω ) − g ( θ , ω ) ‖2 ≤ 2Rω‖g ( θ′ , ω ) − g ( θ , ω ) ‖2 = 2Rω ∥∥∥∥ Ex∼θ′ [ g ( x , ω ) ] − Ex∼θ [ g ( x , ω ) ] ∥∥∥∥ 2 ≤ 2Rω sup x ‖g ( x , ω ) ‖2‖µθ′ ⊗ πθ′ ⊗ P̃ − µθ ⊗ πθ ⊗ P̃‖TV ≤ 2RωCδ‖µθ′ ⊗ πθ′ ⊗ P̃ − µθ ⊗ πθ ⊗ P̃‖TV = 4RωCδdTV ( µθ′ ⊗ πθ′ ⊗ P̃ , µθ ⊗ πθ ⊗ P̃ ) ≤ 4RωCδ|A|Lπ ( 1 + logρ κ−1 + ( 1− ρ ) −1 ) ‖θ − θ′‖2 , where the third inequality follows the definition of TV norm , the second last inequality follows ( 32 ) , and the last inequality follows Lemma A.1 . in [ 17 ] . Collecting the upper bounds of the two terms in ( 94 ) yields I1 ≤ [ 2CδLω + 4RωCδ|A|Lπ ( 1 + logρ κ−1 + ( 1− ρ ) −1 ) ] ‖θ − θ′‖2 . Next we bound E [ I2 ] in ( 93 ) as E [ I2 ] = E [ ∆1 ( x , θ′ , ω ) −∆1 ( x , θ′ , ω′ ) ] = E 〈ω − ω∗θ′ , g ( x , ω ) − g ( θ′ , ω ) 〉 − 〈ω′ − ω∗θ′ , g ( x , ω′ ) − g ( θ′ , ω′ ) 〉 ≤ E |〈ω − ω∗θ′ , g ( x , ω ) − g ( θ′ , ω ) 〉 − 〈ω − ω∗θ′ , g ( x , ω′ ) − g ( θ′ , ω′ ) 〉| + E |〈ω − ω∗θ′ , g ( x , ω′ ) − g ( θ′ , ω′ ) 〉 − 〈ω′ − ω∗θ′ , g ( x , ω′ ) − g ( θ′ , ω′ ) 〉| . ( 95 ) We bound the first term in ( 95 ) as E |〈ω − ω∗θ′ , g ( x , ω ) − g ( θ′ , ω ) 〉 − 〈ω − ω∗θ′ , g ( x , ω′ ) − g ( θ′ , ω′ ) 〉| = E |〈ω − ω∗θ′ , g ( x , ω ) − g ( x , ω′ ) + g ( θ′ , ω′ ) − g ( θ′ , ω ) 〉| ≤ 2Rω ( E ‖g ( x , ω ) − g ( x , ω′ ) ‖2 + E ‖g ( θ′ , ω′ ) − g ( θ′ , ω ) ‖2 ) ≤ 2Rω ( E ‖g ( x , ω ) − g ( x , ω′ ) ‖2 + E ∥∥∥∥ Ex∼θ′ [ g ( x , ω′ ) ] − Ex∼θ′ [ g ( x , ω ) ] ∥∥∥∥ 2 ) = 2Rω ( E ‖ ( γφ ( s′ ) − φ ( s ) ) > ( ω − ω′ ) ‖2 + E ∥∥∥∥ Ex∼θ′ [ ( γφ ( s′ ) − φ ( s ) ) > ] ( ω′ − ω ) ∥∥∥∥ 2 ) ≤ 2Rω ( ( 1 + γ ) E ‖ω − ω′‖2 + ( 1 + γ ) E ‖ω − ω′‖2 ) = 4Rω ( 1 + γ ) E ‖ω − ω′‖2 . We bound the second term in ( 95 ) as E |〈ω − ω∗θ′ , g ( x , ω′ ) − g ( θ′ , ω′ ) 〉 − 〈ω′ − ω∗θ′ , g ( x , ω′ ) − g ( θ′ , ω′ ) 〉| = E |〈ω − ω′ , g ( x , ω′ ) − g ( θ′ , ω′ ) 〉| ≤ 2Cδ E ‖ω − ω′‖2 . Collecting the upper bounds of the two terms in ( 95 ) yields E [ I2 ] ≤ ( 4 ( 1 + γ ) Rω + 2Cδ ) E ‖ω − ω′‖2 . We first bound I3 as E [ I3|θ′ , ω′ , st−m+1 ] = E [ ∆1 ( x , θ′ , ω′ ) −∆1 ( x̃ , θ′ , ω′ ) |θ′ , ω′ , st−m+1 ] ≤ |E [ ∆1 ( x , θ′ , ω′ ) |θ′ , ω′ , st−m+1 ] − E [ ∆1 ( x̃ , θ′ , ω′ ) |θ′ , ω′ , st−m+1 ] | ≤ sup x |∆1 ( x , θ′ , ω′ ) | ‖P ( x ∈ ·|θ′ , ω′ , st−m+1 ) − P ( x̃ ∈ ·|θ′ , ω′ , st−m+1 ) ‖TV ≤ 8RωCδdTV ( P ( x ∈ ·|θ′ , st−m+1 ) , P ( x̃ ∈ ·|θ′ , st−m+1 ) ) , ( 96 ) where the second last inequality follows the definition of TV norm , and the last inequality follows the fact that |∆1 ( x , θ′ , ω′ ) | ≤ ‖ω′ − ω∗θ′‖2‖g ( x , ω′ ) − g ( θ′ , ω′ ) ‖2 ≤ 4RωCδ . By following ( 22 ) in Lemma 2 , we have dTV ( P ( x ∈ ·|θ′ , st−m+1 ) , P ( x̃ ∈ ·|θ′ , st−m+1 ) ) ≤ 1 2 |A|Lπ dm∑ i=τk E [ ‖θk−i − θk−dm‖2| θ′ , st−m+1 ] . Substituting the last inequality into ( 96 ) , then taking total expectation on both sides yield E [ I3 ] ≤ 4RωCδ|A|Lπ dm∑ i=τk E ‖θk−i − θk−dm‖2 . Next we bound I4 . Define x : = ( s , a , s′ ) where s ∼ µθ′ , a ∼ πθ′ and s′ ∼ P̃ . It is immediate that E [ ∆1 ( x , θ′ , ω′ ) |θ′ , ω′ , st−m+1 ] = 〈ω′ − ω∗θ′ , E [ g ( x , ω′ ) |θ′ , ω′ , st−m+1 ] − g ( θ′ , ω′ ) 〉 = 〈ω′ − ω∗θ′ , g ( θ′ , ω′ ) − g ( θ′ , ω′ ) 〉 = 0 . ( 97 ) Then we have E [ I4|θ′ , ω′ , st−m+1 ] = E [ ∆1 ( x̃ , θ′ , ω′ ) −∆1 ( x , θ′ , ω′ ) |θ′ , ω′ , st−m+1 ] ≤ |E [ ∆1 ( x̃ , θ′ , ω′ ) |θ′ , ω′ , st−m+1 ] − E [ ∆1 ( x , θ′ , ω′ ) |θ′ , ω′ , st−m+1 ] | ≤ sup x |∆1 ( x , θ′ , ω′ ) | ‖P ( x̃ ∈ ·|θ′ , st−m+1 ) − P ( x ∈ ·|θ′ , st−m+1 ) ‖TV ≤ 8RωCδdTV ( P ( x̃ ∈ ·|θ′ , st−m+1 ) , P ( x ∈ ·|θ′ , st−m+1 ) ) = 8RωCδdTV ( P ( x̃ ∈ ·|θ′ , st−m+1 ) , µθ′ ⊗ πθ′ ⊗ P̃ ) , ( 98 ) where the second inequality follows the definition of TV norm , and the third inequality follows ( 97 ) . The auxiliary Markov chain with policy πθ′ starts from initial state st−m+1 , and s̃t is the ( m− 1 ) th state on the chain . Following Lemma 1 , we have : dTV ( P ( x̃ ∈ ·|θ′ , st−m+1 ) , µθ′ ⊗ πθ′ ⊗ P̃ ) = dTV ( P ( ( s̃t , ãt , s̃t+1 ) ∈ ·|θ′ , st−m+1 ) , µθ′ ⊗ πθ′ ⊗ P̃ ) ≤ κρm−1 . Substituting the last inequality into ( 98 ) and taking total expectation on both sides yield E [ I4 ] ≤ 8RωCδκρm−1 . Taking total expectation on ( 93 ) and collecting bounds of I1 , I2 , I3 , I4 yield E [ ∆1 ( x , θ , ω ) ] ≤ C4 E ‖θk − θk−dm‖2 + C5 dm∑ i=τk E ‖θk−i − θk−dm‖2 + C6 E ‖ωk − ωk−dm‖2 + C7κρm−1 , where C4 : = 2CδLω + 4RωCδ|A|Lπ ( 1 + logρ κ−1 + ( 1 − ρ ) −1 ) , C5 : = 4RωCδ|A|Lπ , C6 : = 4 ( 1 + γ ) Rω + 2Cδ and C7 : = 8RωCδ . C.2 SUPPORTING LEMMAS FOR THEOREM 4 Lemma 5 . For any m ≥ 1 and k ≥ ( K0 + 1 ) m+K0 + 1 , we have E 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ω ∗ k ) − δ ( x ( k ) , θk ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 ≥ −D2 E ‖θk−τk − θk−dm‖2 −D3 E ‖θk − θk−dm‖2 −D4 dm∑ i=τk E ‖θk−i − θk−dm‖2 −D5κρm−1 − 2CψLV fa − 2Cψ sp E ‖∇J ( θ ) ‖2 , where D2 : = 2LV LψCδ , D3 : = ( 2CδCψLJ + LV Cψ ( Lω + LV ) ( 1 + γ ) + 2CψLJ app ) , D4 : = 2LV CψCδ|A|Lπ and D5 : = 4LV CψCδ . Proof . For the worker that contributes to the kth update , we construct its Markov chain : st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−1−−−−−−→ at−m+1 · · · st−1 θk−d1−−−−→ at−1 P̃−→ st θk−d0−−−−→ at P̃−→ st+1 , where ( st , at , st+1 ) = ( s ( k ) , a ( k ) , s′ ( k ) ) , and { dj } m j=0 is some increasing sequence with d0 : = τk . By ( 92 ) in Lemma 4 , we have dm ≤ ( K0 + 1 ) m+K0 . Given ( st−m , at−m , st−m+1 ) and θk−dm , we construct an auxiliary Markov chain : st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−−−−→ ãt−m+1 · · · s̃t−1 θk−dm−−−−→ ãt−1 P̃−→ s̃t θk−dm−−−−→ ãt P̃−→ s̃t+1 . First we have〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ω ∗ k ) − δ ( x ( k ) , θk ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 = 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ω ∗ k ) − δ ( x ( k ) , θk ) ) ( ψθk−τk ( s ( k ) , a ( k ) ) − ψθk−dm ( s ( k ) , a ( k ) ) ) 〉 + 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ω ∗ k ) − δ ( x ( k ) , θk ) ) ψθk−dm ( s ( k ) , a ( k ) ) 〉 . ( 99 ) We first bound the fist term in ( 99 ) as〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ω ∗ k ) − δ ( x ( k ) , θk ) ) ( ψθk−τk ( s ( k ) , a ( k ) ) − ψθk−dm ( s ( k ) , a ( k ) ) ) 〉 ≥ −‖J ( θk ) ‖2|δ̂ ( x ( k ) , ω∗k ) − δ ( x ( k ) , θk ) |‖ψθk−τk ( s ( k ) , a ( k ) ) − ψθk−dm ( s ( k ) , a ( k ) ) ‖2 ≥ −‖J ( θk ) ‖2 ( |δ̂ ( x ( k ) , ω∗k ) |+ |δ ( x ( k ) , θk ) | ) ‖ψθk−τk ( s ( k ) , a ( k ) ) − ψθk−dm ( s ( k ) , a ( k ) ) ‖2 ≥ −LV ( |δ̂ ( x ( k ) , ω∗k ) |+ |δ ( x ( k ) , θk ) | ) ‖ψθk−τk ( s ( k ) , a ( k ) ) − ψθk−dm ( s ( k ) , a ( k ) ) ‖2 ≥ −2LV Cδ‖ψθk−τk ( s ( k ) , a ( k ) ) − ψθk−dm ( s ( k ) , a ( k ) ) ‖2 ≥ −2LV LψCδ‖θk−τk − θk−dm‖2 , ( 100 ) where the last inequality follows Assumption 3 and second last inequality follows |δ̂ ( x , ω∗θ ) | ≤ |r ( x ) |+ γ‖φ ( s′ ) ‖2‖ω∗θ‖2 + ‖φ ( s ) ‖2‖ω∗θ‖2 ≤ rmax + ( 1 + γ ) Rω ≤ Cδ , |δ ( x , θ ) | ≤ |r ( x ) |+ γ|Vπθ ( s′ ) |+ |Vπθ ( s ) | ≤ rmax + ( 1 + γ ) rmax 1− γ ≤ Cδ . Substituting ( 100 ) into ( 99 ) gives〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ω ∗ k ) − δ ( x ( k ) , θk ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 ≥ −2LV LψCδ‖θk−τk − θk−dm‖2 + 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ω ∗ k ) − δ ( x ( k ) , θk ) ) ψθk−dm ( s ( k ) , a ( k ) ) 〉 . ( 101 ) Then we start to bound the second term in ( 101 ) . For brevity , we define ∆2 ( x , θ ) : = 〈 ∇J ( θ ) , ( δ̂ ( x , ω∗θ ) − δ ( x , θ ) ) ψθk−dm ( s , a ) 〉 . In the following proof , we use θ , θ′ , ω∗θ , ω ∗ θ′ , x and x̃ as shorthand notations for θk , θk−dm , ω ∗ k , ω∗k−dm , xt and x̃t respectively . We also define x : = ( s , a , s ′ ) , where s ∼ µθ′ , a ∼ πθ′ and s′ ∼ P̃ . We decompose the second term in ( 101 ) as ∆2 ( x , θ ) = ∆2 ( x , θ ) −∆2 ( x , θ′ ) I1 + ∆2 ( x , θ ′ ) −∆2 ( x̃ , θ′ ) I2 + ∆2 ( x̃ , θ ′ ) −∆2 ( x , θ′ ) I3 + ∆2 ( x , θ ′ ) I4 . We bound the term I1 as I1 = 〈 ∇J ( θ ) , ( δ̂ ( x , ω∗θ ) − δ ( x , θ ) ) ψθ′ ( s , a ) 〉 − 〈 ∇J ( θ′ ) , ( δ̂ ( x , ω∗θ′ ) − δ ( x , θ′ ) ) ψθ′ ( s , a ) 〉 = 〈 ∇J ( θ ) , ( δ̂ ( x , ω∗θ ) − δ ( x , θ ) ) ψθ′ ( s , a ) 〉 − 〈 ∇J ( θ′ ) , ( δ̂ ( x , ω∗θ ) − δ ( x , θ ) ) ψθ′ ( s , a ) 〉 + 〈 ∇J ( θ′ ) , ( δ̂ ( x , ω∗θ ) − δ ( x , θ ) ) ψθ′ ( s , a ) 〉 − 〈 ∇J ( θ′ ) , ( δ̂ ( x , ω∗θ′ ) − δ ( x , θ′ ) ) ψθ′ ( s , a ) 〉 . For the first term in I1 , we have〈 ∇J ( θ ) , ( δ̂ ( x , ω∗θ ) − δ ( x , θ ) ) ψθ′ ( s , a ) 〉 − 〈 ∇J ( θ′ ) , ( δ̂ ( x , ω∗θ ) − δ ( x , θ ) ) ψθ′ ( s , a ) 〉 = 〈 ∇J ( θ ) −∇J ( θ′ ) , ( δ̂ ( x , ω∗θ ) − δ ( x , θ ) ) ψθ′ ( s , a ) 〉 ≥ −‖∇J ( θ ) −∇J ( θ′ ) ‖2‖δ̂ ( x , ω∗θ ) − δ ( x , θ ) ‖2‖ψθ′ ( s , a ) ‖2 ≥ −2CδCψ‖∇J ( θ ) −∇J ( θ′ ) ‖2 ≥ −2CδCψLJ‖θ − θ′‖2 , where the last inequality is due to the LJ -Lipschitz of policy gradient shown in Proposition 1 . For the second term in I1 , we have〈 ∇J ( θ′ ) , ( δ̂ ( x , ω∗θ ) − δ ( x , θ ) ) ψθ′ ( s , a ) 〉 − 〈 ∇J ( θ′ ) , ( δ̂ ( x , ω∗θ′ ) − δ ( x , θ′ ) ) ψθ′ ( s , a ) 〉 = 〈 ∇J ( θ′ ) , ( δ̂ ( x , ω∗θ ) − δ̂ ( x , ω∗θ′ ) + δ ( x , θ′ ) − δ ( x , θ ) ) ψθ′ ( s , a ) 〉 ≥ −LV Cψ ∣∣∣δ̂ ( x , ω∗θ ) − δ̂ ( x , ω∗θ′ ) + δ ( x , θ′ ) − δ ( x , θ ) ∣∣∣ ≥ −LV Cψ ∣∣γφ ( s′ ) > ( ω∗θ − ω∗θ′ ) + φ ( s ) > ( ω∗θ′ − ω∗θ ) + γVπθ′ ( s′ ) − γVπθ ( s′ ) + Vπθ ( s ) − Vπθ′ ( s ) ∣∣ ≥ −LV Cψ ( γ‖ω∗θ − ω∗θ′‖2 + ‖ω∗θ′ − ω∗θ‖2 + γ|Vπθ′ ( s ′ ) − Vπθ ( s′ ) |+ |Vπθ ( s ) − Vπθ′ ( s ) | ) ≥ −LV Cψ ( γLω‖θ − θ′‖2 + Lω‖θ − θ′‖2 + γLV ‖θ − θ′‖2 + LV ‖θ − θ′‖2 ) = −LV Cψ ( Lω + LV ) ( 1 + γ ) ‖θ − θ′‖2 , where the last inequality is due to the Lω-Lipschitz continuity of ω∗θ shown in Proposition 2 and LV -Lipschitz continuity of Vπθ ( s ) shown in Lemma 3 . Collecting the upper bounds of I1 yields I1 ≥ − ( 2CδCψLJ + LV Cψ ( Lω + LV ) ( 1 + γ ) ) ‖θ − θ′‖2 . First we bound I2 as E [ I2|θ′ , st−m+1 ] = E [ ∆2 ( x , θ′ ) −∆2 ( x̃ , θ′ ) |θ′ , st−m+1 ] ≥ − |E [ ∆2 ( x , θ′ ) | θ′ , st−m+1 ] − E [ ∆2 ( x̃ , θ′ ) | θ′ , st−m+1 ] | ≥ − sup x |∆2 ( x , θ′ ) | ‖P ( x ∈ ·|θ′ , st−m+1 ) − P ( x̃ ∈ ·|θ′ , st−m+1 ) ‖TV ≥ −4LV CψCδdTV ( P ( x ∈ ·|θ′ , st−m+1 ) , P ( x̃ ∈ ·|θ′ , st−m+1 ) ) ≥ −2LV CψCδ|A|Lπ dm∑ i=τk E [ ‖θk−i − θk−dm‖2| θ′ , st−m+1 ] , ( 102 ) where the second inequality is due to the definition of TV norm , the last inequality follows ( 22 ) in Lemma 2 , and the second last inequality follows the fact that |∆2 ( x , θ′ ) | ≤ ‖∇J ( θ′ ) ‖2|δ̂ ( x , ω∗θ′ ) − δ ( x , θ′ ) |‖ψθ′ ( s , a ) ‖2 ≤ 2LV CδCψ . ( 103 ) Taking total expectation on both sides of ( 102 ) yields E [ I2 ] ≥ −2LV CψCδ|A|Lπ dm∑ i=τk E ‖θk−i − θk−dm‖2 . Next we bound I3 as E [ I3|θ′ , st−m+1 ] = E [ ∆2 ( x̃ , θ′ ) −∆2 ( x , θ′ ) | θ′ , st−m+1 ] ≥ − |E [ ∆2 ( x̃ , θ′ ) | θ′ , st−m+1 ] − E [ ∆2 ( x , θ′ ) | θ′ , st−m+1 ] | ≥ − sup x |∆2 ( x , θ′ ) | ‖P ( x̃ ∈ ·|θ′ , st−m+1 ) − P ( x ∈ ·|θ′ , st−m+1 ) ‖TV ≥ −4LV CψCδdTV ( P ( x̃ ∈ ·|θ′ , st−m+1 ) , µθ′ ⊗ πθ′ ⊗ P̃ ) , ( 104 ) where the second inequality is due to the definition of TV norm , and the last inequality follows ( 103 ) . The auxiliary Markov chain with policy πθ′ starts from initial state st−m+1 , and s̃t is the ( m− 1 ) th state on the chain . Following Lemma 1 , we have : dTV ( P ( x̃ ∈ ·|θ′ , st−m+1 ) , µθ′ ⊗ πθ′ ⊗ P̃ ) =dTV ( P ( ( s̃t , ãt , s̃t+1 ) ∈ ·|θ′ , st−m+1 ) , µθ′ ⊗ πθ′ ⊗ P̃ ) ≤ κρm−1 . Substituting the last inequality into ( 104 ) and taking total expectation on both sides yield E [ I3 ] ≥ −4LV CψCδκρm−1 We bound I4 as E [ I4|θ′ ] = E [ 〈 ∇J ( θ′ ) , ( δ̂ ( x , ω∗θ′ ) − δ ( x , θ′ ) ) ψθ′ ( s , a ) 〉∣∣∣ θ′ ] ≥ −Cψ‖∇J ( θ′ ) ‖2 E [ ∣∣∣δ̂ ( x , ω∗θ′ ) − δ ( x , θ′ ) ∣∣∣∣∣∣ θ′ ] = −Cψ‖∇J ( θ′ ) ‖2 E [ ∣∣γ ( φ ( s′ ) > ω∗θ′ − Vπθ′ ( s′ ) ) + Vπθ′ ( s ) − φ ( s ) > ω∗θ′ ∣∣∣∣ θ′ ] ≥ −Cψ‖∇J ( θ′ ) ‖2 ( γ E [ |φ ( s′ ) > ω∗θ′ − Vπθ′ ( s ′ ) | ∣∣ θ′ ] + E [ |Vπθ′ ( s ) − φ ( s ) > ω∗θ′ |∣∣ θ′ ] ) ≥ −Cψ‖∇J ( θ′ ) ‖2 ( γ √ E [ |φ ( s′ ) > ω∗θ′ − Vπθ′ ( s ′ ) |2 ∣∣ θ′ ] +√E [ |Vπθ′ ( s ) − φ ( s ) > ω∗θ′ |2∣∣ θ′ ] ) = −Cψ‖∇J ( θ′ ) ‖2 ( γ √ E s′∼µθ′ |φ ( s′ ) > ω∗θ′ − Vπθ′ ( s ′ ) |2 + √ E s∼µθ′ |Vπθ′ ( s ) − φ ( s ) > ω∗θ′ |2 ) ≥ −2Cψ‖∇J ( θ′ ) ‖2 app , where the second last inequality follows Jensen ’ s inequality . The last inequality further implies E [ I4 ] ≥ −2Cψ E ‖∇J ( θ′ ) −∇J ( θ ) +∇J ( θ ) ‖2 app ≥ −2Cψ app E ‖∇J ( θ′ ) −∇J ( θ ) ‖2 − 2Cψ app E ‖∇J ( θ ) ‖2 ≥ −2Cψ app E ‖∇J ( θ′ ) −∇J ( θ ) ‖2 − 2Cψ fa E ‖∇J ( θ ) ‖2 − 2CψLV sp ≥ −2CψLJ app E ‖θ − θ′‖2 − 2Cψ fa E ‖∇J ( θ ) ‖2 − 2CψLV sp , where the last inequality follows Proposition 1 . Taking total expectation on both sides of ( 101 ) , and collecting lower bounds of I1 , I2 , I3 and I4 yield E 〈 ∇J ( θk ) , ( δ̂ ( x ( k ) , ω ∗ k ) − δ ( x ( k ) , θk ) ) ψθk−τk ( s ( k ) , a ( k ) ) 〉 ≥ −D2 E ‖θk−τk − θk−dm‖2 −D3 E ‖θk − θk−dm‖2 −D4 dm∑ i=τk E ‖θk−i − θk−dm‖2 −D5κρm−1 − 2CψLV fa − 2Cψ sp E ‖∇J ( θk ) ‖2 , where D2 : = 2LV LψCδ , D3 : = ( 2CδCψLJ + LV Cψ ( Lω + LV ) ( 1 + γ ) + 2CψLJ app ) , D4 : = 2LV CψCδ|A|Lπ and D5 : = 4LV CψCδ . Lemma 6 . For any m ≥ 1 and k ≥ ( K0 + 1 ) m+K0 + 1 , we have E 〈 ∇J ( θk ) , δ ( x ( k ) , θk ) ψθk−τk ( s ( k ) , a ( k ) ) −∇J ( θk ) 〉 ≥ −D6 E ‖θk−τk − θk−dm‖2 −D7 E ‖θk − θk−dm‖2−D8 dm∑ i=τk E ‖θk−i − θk−dm‖2−D9κρm−1 − 8Cψrmax ( 1− γ ) E ‖∇J ( θk ) ‖2 , where D6 : = LV CδLψ , D7 : = CpLJ + ( 1 + γ ) L2V Cψ + 2LV LJ + 8CψrmaxLJ ( 1 − γ ) , D8 : = LV ( Cp + LV ) |A|Lπ , D9 : = 2LV ( Cp + LV ) . Proof . For the worker that contributes to the kth update , we construct its Markov chain : st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−1−−−−−−→ at−m+1 · · · st−1 θk−d1−−−−→ at−1 P̃−→ st θk−d0−−−−→ at P̃−→ st+1 , where ( st , at , st+1 ) = ( s ( k ) , a ( k ) , s′ ( k ) ) , and { dj } m j=0 is some increasing sequence with d0 : = τk . By ( 92 ) in Lemma 4 , we have dm ≤ ( K0 + 1 ) m+K0 . Given ( st−m , at−m , st−m+1 ) and θk−dm , we construct an auxiliary Markov chain : st−m θk−dm−−−−→ at−m P̃−→ st−m+1 θk−dm−−−−→ ãt−m+1 · · · s̃t−1 θk−dm−−−−→ ãt−1 P̃−→ s̃t θk−dm−−−−→ ãt P̃−→ s̃t+1 . First we have 〈 ∇J ( θk ) , δ ( x ( k ) , θk ) ψθk−τk ( s ( k ) , a ( k ) ) −∇J ( θk ) 〉 = 〈 ∇J ( θk ) , δ ( x ( k ) , θk ) ( ψθk−τk ( s ( k ) , a ( k ) ) − ψθk−dm ( s ( k ) , a ( k ) ) ) 〉 + 〈 ∇J ( θk ) , δ ( x ( k ) , θk ) ψθk−dm ( s ( k ) , a ( k ) ) −∇J ( θk ) 〉 . ( 105 ) We bound the first term in ( 105 ) as〈 ∇J ( θk ) , δ ( x ( k ) , θk ) ( ψθk−τk ( s ( k ) , a ( k ) ) − ψθk−dm ( s ( k ) , a ( k ) ) ) 〉 ≥ −‖∇J ( θk ) ‖2 ‖δ ( x ( k ) , θk ) ‖2‖ψθk−τk ( s ( k ) , a ( k ) ) − ψθk−dm ( s ( k ) , a ( k ) ) ‖2 ≥ −LV ‖δ ( x ( k ) , θk ) ‖2‖ψθk−τk ( s ( k ) , a ( k ) ) − ψθk−dm ( s ( k ) , a ( k ) ) ‖2 ≥ −LV Cδ‖ψθk−τk ( s ( k ) , a ( k ) ) − ψθk−dm ( s ( k ) , a ( k ) ) ‖2 ≥ −LV CδLψ‖θk−τk − θk−dm‖2 , ( 106 ) where the last inequality follows Assumption 3 , and the second last inequality follows the fact that |δ ( x , θ ) | ≤ |r ( x ) |+ γ|Vπθ ( s′ ) |+ |Vπθ ( s ) | ≤ rmax + ( 1 + γ ) rmax 1− γ ≤ Cδ . Substituting ( 106 ) into ( 105 ) gives〈 ∇J ( θk ) , δ ( x ( k ) , θk ) ψθk−τk ( s ( k ) , a ( k ) ) −∇J ( θk ) 〉 ≥ −LV CδLψ‖θk−τk − θk−dm‖2 + 〈 ∇J ( θk ) , δ ( x ( k ) , θk ) ψθk−dm ( s ( k ) , a ( k ) ) −∇J ( θk ) 〉 . ( 107 ) Then we start to bound the second term in ( 107 ) . For brevity , we define ∆3 ( x , θ ) : = 〈 ∇J ( θ ) , δ ( x , θ ) ψθk−dm ( s , a ) −∇J ( θ ) 〉 . Throughout the following proof , we use θ , θ′ , x and x̃ as shorthand notations of θk , θk−dm , xt and x̃t respectively . We decompose ∆3 ( x , θ ) as ∆3 ( x , θ ) = ∆3 ( x , θ ) −∆3 ( x , θ′ ) I1 + ∆3 ( x , θ ′ ) −∆3 ( x̃ , θ′ ) I2 + ∆3 ( x̃ , θ ′ ) I3 . We first bound I1 as |I1| = |∆3 ( x , θ ) −∆3 ( x , θ′ ) | = ∣∣〈∇J ( θ ) , δ ( x , θ ) ψθ′ ( s , a ) 〉 − ‖∇J ( θ ) ‖22 − 〈∇J ( θ′ ) , δ ( x , θ′ ) ψθ′ ( s , a ) 〉+ ‖∇J ( θ′ ) ‖22∣∣ ≤ |〈∇J ( θ ) , δ ( x , θ ) ψθ′ ( s , a ) 〉 − 〈∇J ( θ′ ) , δ ( x , θ′ ) ψθ′ ( s , a ) 〉|+ ∣∣‖∇J ( θ′ ) ‖22 − ‖∇J ( θ ) ‖22∣∣ ≤ |〈∇J ( θ ) , δ ( x , θ ) ψθ′ ( s , a ) 〉 − 〈∇J ( θ′ ) , δ ( x , θ′ ) ψθ′ ( s , a ) 〉|+ ‖∇J ( θ′ ) +∇J ( θ ) ‖2‖∇J ( θ′ ) −∇J ( θ ) ‖2 ≤ |〈∇J ( θ ) , δ ( x , θ ) ψθ′ ( s , a ) 〉 − 〈∇J ( θ′ ) , δ ( x , θ′ ) ψθ′ ( s , a ) 〉|+ 2LV LJ‖θ − θ′‖2 , ( 108 ) where the last equality is due to LV -Lipschitz of value function and LJ -Lipschitz of policy gradient . We bound the first term in ( 108 ) as |〈∇J ( θ ) , δ ( x , θ ) ψθ′ ( s , a ) 〉 − 〈∇J ( θ′ ) , δ ( x , θ′ ) ψθ′ ( s , a ) 〉| ≤ |〈∇J ( θ ) , δ ( x , θ ) ψθ′ ( s , a ) 〉 − 〈∇J ( θ ) , δ ( x , θ′ ) ψθ′ ( s , a ) 〉| + |〈∇J ( θ ) , δ ( x , θ′ ) ψθ′ ( s , a ) 〉 − 〈∇J ( θ′ ) , δ ( x , θ′ ) ψθ′ ( s , a ) 〉| = |〈∇J ( θ ) , ( δ ( x , θ ) − δ ( x , θ′ ) ) ψθ′ ( s , a ) 〉|+ |〈∇J ( θ ) −∇J ( θ′ ) , δ ( x , θ′ ) ψθ′ ( s , a ) 〉| ≤ LV Cψ |δ ( x , θ ) − δ ( x , θ′ ) |+ Cp‖∇J ( θ ) −∇J ( θ′ ) ‖2 = LV Cψ ∣∣γ ( Vπθ ( s′ ) − Vπθ′ ( s′ ) ) + Vπθ′ ( s ) − Vπθ ( s ) ∣∣+ Cp‖∇J ( θ ) −∇J ( θ′ ) ‖2 ≤ LV Cψ ( γ ∣∣Vπθ ( s′ ) − Vπθ′ ( s′ ) ∣∣+ ∣∣Vπθ′ ( s ) − Vπθ ( s ) ∣∣ ) + Cp‖∇J ( θ ) −∇J ( θ′ ) ‖2 ≤ LV Cψ ( γLV ‖θ − θ′‖2 + LV ‖θ′ − θ‖ ) + CpLJ‖θ − θ′‖2 = ( CpLJ + ( 1 + γ ) L 2 V Cψ ) ‖θ − θ′‖2 . Substituting the above inequality into ( 108 ) gives the lower bound of I1 : I1 ≥ − ( CpLJ + ( 1 + γ ) L 2 V Cψ + 2LV LJ ) ‖θ − θ′‖2 . First we bound I2 as E [ I2|θ′ , st−m+1 ] = E [ ∆3 ( x , θ′ ) −∆3 ( x̃ , θ′ ) |θ′ , st−m+1 ] ≥ − |E [ ∆3 ( x , θ′ ) |θ′ , st−m+1 ] − E [ ∆3 ( x̃ , θ′ ) |θ′ , st−m+1 ] | ≥ − sup x |∆3 ( x , θ′ ) | ‖P ( x ∈ ·|θ′ , st−m+1 ) − P ( x̃ ∈ ·|θ′ , st−m+1 ) ‖TV ≥ −2LV ( Cp + LV ) dTV ( P ( x ∈ ·|θ′ , st−m+1 ) , P ( x̃ ∈ ·|θ′ , st−m+1 ) ) ≥ −LV ( Cp + LV ) |A|Lπ dm∑ i=τk E [ ‖θk−i − θk−dm‖2|θ′ , st−m+1 ] , ( 109 ) where the second inequality is due to the definition of TV norm , the last inequality is due to ( 22 ) in Lemma 2 , and thesecond last inequality follows the fact that |∆3 ( x , θ′ ) | ≤ ‖∇J ( θ ) ‖2 ( ‖δ ( x , θ ) ψθk−dm ( s , a ) ‖2 + ‖∇J ( θ ) ‖2 ) ≤ LV ( Cp + LV ) . ( 110 ) Taking total expectation on both sides of ( 109 ) yields E [ I2 ] ≥ −LV ( Cp + LV ) |A|Lπ dm∑ i=τk E ‖θk−i − θk−dm‖2 . Define x : = ( s , a , s′ ) , where s ∼ dθ′ , a ∼ πθ′ and s′ ∼ P̃ . Then we have E [ I3 ] = E [ ∆3 ( x̃ , θ′ ) −∆3 ( x , θ′ ) ] + E [ ∆3 ( x , θ′ ) ] . ( 111 ) We bound the first term in ( 111 ) as E [ ∆3 ( x̃ , θ′ ) −∆3 ( x , θ′ ) |θ′ , st−m+1 ] ≥ − |E [ ∆3 ( x̃ , θ′ ) |θ′ , st−m+1 ] − E [ ∆3 ( x , θ′ ) |θ′ , st−m+1 ] | ≥ − sup x |∆3 ( x , θ′ ) | ‖P ( x̃ ∈ ·|θ′ , st−m+1 ) − P ( x ∈ ·|θ′ , st−m+1 ) ‖TV ≥ −2LV ( Cp + LV ) dTV ( P ( x̃ ∈ ·|θ′ , st−m+1 ) , P ( x ∈ ·|θ′ , st−m+1 ) ) = −2LV ( Cp + LV ) dTV ( P ( x̃ ∈ ·|θ′ , st−m+1 ) , dθ′ ⊗ πθ′ ⊗ P̃ ) = −2LV ( Cp + LV ) dTV ( P ( x̃ ∈ ·|θ′ , st−m+1 ) , µθ′ ⊗ πθ′ ⊗ P̃ ) ( 112 ) where the second inequality follows the definition of total variation norm , and the third inequality follows ( 110 ) . The last equality is due to the fact shown by [ 6 ] that µθ′ ( · ) = dθ′ ( · ) , where µθ′ is the stationary distribution of an artificial MDP with transition kernel P̃ ( ·|s , a ) and policy πθ′ . The auxiliary Markov chain with policy πθ′ starts from initial state st−m+1 , and s̃t is the ( m− 1 ) th state on the chain . Following Lemma 1 , we have : dTV ( P ( x̃ ∈ ·|θ′ , st−m+1 ) , µθ′ ⊗ πθ′ ⊗ P̃ ) = dTV ( P ( ( s̃t , ãt , s̃t+1 ) ∈ ·|θ′ , st−m+1 ) , µθ′ ⊗ πθ′ ⊗ P̃ ) ≤ κρm−1 . Substituting the last inequality into ( 112 ) and taking total expectation on both sides yield E [ ∆3 ( x̃ , θ′ ) −∆3 ( x , θ′ ) ] ≥ −2LV ( Cp + LV ) κρm−1 . Consider the second term in ( 111 ) . Note its form is similar to ( 59 ) , so by following the derivation of ( 63 ) , we directly have E [ ∆3 ( x , θ′ ) ] = E 〈∇J ( θ′ ) , δ ( x , θ′ ) ψθ′ ( s , a ) −∇J ( θ′ ) 〉 ≥ −8Cψrmax ( 1− γ ) E ‖∇J ( θ′ ) ‖2 , which further implies E [ ∆3 ( x , θ′ ) ] ≥ −8Cψrmax ( 1− γ ) E ‖∇J ( θ′ ) ‖2 ≥ −8Cψrmax ( 1− γ ) E ‖∇J ( θ′ ) −∇J ( θ ) ‖2 − 8Cψrmax ( 1− γ ) E ‖∇J ( θ ) ‖2 ≥ −8CψrmaxLJ ( 1− γ ) E ‖θ′ − θ‖2−8Cψrmax ( 1− γ ) E ‖∇J ( θ ) ‖2 where the last inequality follows from Proposition 1 . Collecting the lower bounds gives E [ I3 ] ≥ −2LV ( Cp + LV ) κρm−1 − 8Cψrmax ( 1− γ ) ( LJ E ‖θ′ − θ‖2 − E ‖∇J ( θ ) ‖2 ) . Taking total expectation on ∆3 ( x , θ ) and collecting lower bounds of I1 , I2 , I3 yield E [ ∆3 ( x , θ ) ] ≥ − ( CpLJ + ( 1 + γ ) L 2 V Cψ + 2LV LJ + 8CψrmaxLJ ( 1− γ ) ) E ‖θk − θk−dm‖2 − LV ( Cp + LV ) |A|Lπ dm∑ i=τk E ‖θk−i − θk−dm‖2 − 2LV ( Cp + LV ) κρm−1 − 8Cψrmax ( 1− γ ) E ‖∇J ( θk ) ‖2 . Taking total expectation on ( 107 ) and substituting the above inequality into it yield E 〈 ∇J ( θk ) , δ ( x ( k ) , θk ) ψθk−τk ( s ( k ) , a ( k ) ) −∇J ( θk ) 〉 ≥ −D6 E ‖θk−τk − θk−dm‖2 −D7 E ‖θk − θk−dm‖2 −D8 dm∑ i=τk E ‖θk−i − θk−dm‖2 −D9κρm−1 − 8Cψrmax ( 1− γ ) E ‖∇J ( θk ) ‖2 , where D6 : = LV CδLψ , D7 : = CpLJ + ( 1 + γ ) L2V Cψ + 2LV LJ + 8CψrmaxLJ ( 1 − γ ) , D8 : = LV ( Cp + LV ) |A|Lπ , D9 : = 2LV ( Cp + LV ) . C.3 EXPLANATION OF THE APPROXIMATION ERROR In this section , we will provide a justification for the circumstances when the approximation error app defined in ( 14 ) is small . Lemma 7 . Suppose Assumption 2 and 4 hold . Then it holds that app ≤ max θ∈Rd √ E s∼µθ |Vπθ ( s ) − V̂ω̄∗θ ( s ) |2 + 4rmax ( λ −1 + λ−2rmax ) ( 1 + logρ κ −1 + 1 1− ρ ) ( 1− γ ) ( 113 ) where ω̄∗θ the critic stationary point of original Markov chain with policy πθ and transition kernel P . In ( 113 ) , the first term captures the quality of critic function parameterization method which also appears in previous works [ 14 , 15 , 17 ] . When using linear critic function approximation , it becomes zero when the value function Vπθ belongs to the linear function space for any θ . The second term corresponds to the error introduced by sampling from the artificial transition kernel P̃ ( ·|s , a ) = ( 1− γ ) P ( ·|s , a ) + γη ( · ) . For a large γ close to 1 , the artificial Markov chain is close to the original one . In this case , the second error term is therefore small . This fact also consists with practice where large γ is commonly used in two time-scale actor critic algorithms [ 3 ] . Before going into the proof , we first define that : Āθ , φ : = E s∼µ̄θ , s′∼Pπθ [ φ ( s ) ( γφ ( s′ ) − φ ( s ) ) > ] , b̄θ , φ : = E s∼µ̄θ , a∼πθ , s′∼P [ r ( s , a , s′ ) φ ( s ) ] , where µ̄θ as the stationary distribution of the original Markov chain with πθ and transition kernel P . Proof . Recall the definition of the approximation error : app = max θ∈Rd √ E s∼µθ |Vπθ ( s ) − V̂ω∗θ ( s ) |2 , where µθ is the stationary distribution of the artificial Markov chain with πθ and transition kernel P̃ , and ω∗θ is the stationary point of critic update under the artificial Markov chain . We decompose app as app = max θ∈Rd √ E s∼µθ |Vπθ ( s ) − V̂ω̄∗θ ( s ) + V̂ω̄∗θ ( s ) − V̂ω∗θ ( s ) |2 ≤ max θ∈Rd √ E s∼µθ |Vπθ ( s ) − V̂ω̄∗θ ( s ) |2 fa + max θ∈Rd √ E s∼µθ |V̂ω̄∗θ ( s ) − V̂ω∗θ ( s ) |2 sp , ( 114 ) where the first term corresponds to the function approximation error fa , and second term corresponds to the sampling error sp . With A , b and Ā , b̄ as shorthand notations for Aθ , ψ , bθ , ψ and Āθ , ψ , b̄θ , ψ respectively , we bound the second term in ( 114 ) as |V̂ω̄∗θ ( s ) − V̂ω∗θ ( s ) | = ∣∣φ ( s ) > ω∗θ − φ ( s ) > ω̄∗θ ∣∣ ≤ ∥∥A−1b− Ā−1b̄∥∥ 2 = ∥∥A−1b−A−1b̄+A−1b̄− Ā−1b̄∥∥ 2 ≤ ∥∥A−1 ( b− b̄ ) ∥∥ 2 + ∥∥ ( A−1 − Ā−1 ) b̄∥∥ 2 ≤ λ−1‖b− b̄‖2 + rmax ∥∥A−1 − Ā−1∥∥ 2 = λ−1‖b− b̄‖2 + rmax ∥∥A−1 ( Ā−A ) Ā−1∥∥ 2 ≤ λ−1‖b− b̄‖2 + λ−2rmax ∥∥Ā−A∥∥ 2 . ( 115 ) We bound the first term in last inequality as ‖b− b̄‖2 = ∥∥∥∥∥ Es∼µθ , a∼πθ , s′∼P̃ [ r ( s , a , s′ ) φ ( s ) ] − Es∼µ̄θ , a∼πθ , s′∼P [ r ( s , a , s′ ) φ ( s ) ] ∥∥∥∥∥ ≤ sup ‖r ( s , a , s′ ) φ ( s ) ‖2‖µθ ⊗ πθ ⊗ P̃ − µ̄θ ⊗ πθ ⊗ P‖TV ≤ 2rmaxdTV ( µθ ⊗ πθ ⊗ P̃ , µ̄θ ⊗ πθ ⊗ P ) . ( 116 ) We now bound the divergence term in the last inequality as dTV ( µθ ⊗ πθ ⊗ P̃ , µ̄θ ⊗ πθ ⊗ P ) = ∫ s∈S ∑ a∈A ∫ s′∈S ∣∣∣µθ ( s ) πθ ( a|s ) P̃ ( s′|s , a ) − µ̄θ ( s ) πθ ( a|s ) P ( s′|s , a ) ∣∣∣ = ∫ s∈S ∑ a∈A ∫ s′∈S |µθ ( s ) πθ ( a|s ) P̃ ( s′|s , a ) − µθ ( s ) πθ ( a|s ) P ( s′|s , a ) + µθ ( s ) πθ ( a|s ) P ( s′|s , a ) − µ̄θ ( s ) πθ ( a|s ) P ( s′|s , a ) | ≤ ∫ s∈S ∑ a∈A µθ ( s ) πθ ( a|s ) ∫ s′∈S ∣∣∣P̃ ( s′|s , a ) − P ( s′|s , a ) ∣∣∣+ ∫ s∈S |µθ ( s ) − µ̄θ ( s ) | . ( 117 ) We bound the first term in ( 117 ) as∫ s′∈S ∣∣∣P̃ ( s′|s , a ) − P ( s′|s , a ) ∣∣∣ = ( 1− γ ) ∫ s′∈S |P ( s′|s , a ) − η ( s′ ) | ≤ 2 ( 1− γ ) . ( 118 ) Following [ 39 , Theorem 3.1 ] , the second term in ( 117 ) can be bounded as∫ s∈S |µθ ( s ) − µ̄θ ( s ) | ≤ ( logρ κ −1 + 1 1− ρ ) sup s ∫ s′∈S ∣∣∣∣∣∑ a πθ ( a|s ) ( P̃ ( s′|s , a ) − P ( s′|s , a ) ) ∣∣∣∣∣ ≤ ( logρ κ −1 + 1 1− ρ ) sup s ∑ a πθ ( a|s ) ∫ s′∈S ∣∣∣P̃ ( s′|s , a ) − P ( s′|s , a ) ∣∣∣ ≤ 2 ( logρ κ −1 + 1 1− ρ ) ( 1− γ ) , ( 119 ) where the last inequality follows ( 118 ) . Substituting ( 118 ) and ( 119 ) into ( 117 ) gives dTV ( µθ ⊗ πθ ⊗ P̃ , µ̄θ ⊗ πθ ⊗ P ) ≤ 2 ( 1 + logρ κ −1 + 1 1− ρ ) ( 1− γ ) . Substituting the above inequality into ( 116 ) gives ‖b− b̄‖2 ≤ 4rmax ( 1 + logρ κ −1 + 1 1− ρ ) ( 1− γ ) . ( 120 ) Similarly , we also have ‖A− Ā‖2 ≤ 4rmax ( 1 + logρ κ −1 + 1 1− ρ ) ( 1− γ ) . ( 121 ) Substituting ( 120 ) and ( 121 ) into ( 115 ) , then substituting ( 115 ) into ( 114 ) completes the proof . D EXPERIMENT DETAILS Hardware device . The tests on synthetic environment and CartPole was performed in a 16-core CPU computer . The test on Atari game was run in a 4 GPU computer . Parameterization . For the synthetic environment , we used linear value function approximation and tabular softmax policy [ 36 ] . For CartPole , we used a 3-layer MLP with 128 neurons and sigmoid activation function in each layer . The first two layers are shared for both actor and critic network . For the Atari seaquest game , we used a convolution-LSTM network . For network details , see [ 40 ] . Hyper-parameters . For the synthetic environment tests , we run Algorithm 1 with actor step size αk = 0.05 ( 1+k ) 0.6 and critic step size βk = 0.05 ( 1+k ) 0.4 . In tests of CartPole , we run Algorithm 1 with a minibatch of 20 samples . We update the actor network with a step size of αk = 0.01 ( 1+k ) 0.6 and critic network with a step size of βk = 0.01 ( 1+k ) 0.4 . See Table 1 for hyper-parameters to generate the Atari game results in Figure 4 .

Summary: This paper revisits the A3C algorithm with TD(0) for the critic update to provide better theoretical analysis of A3C. A3C-TD(0) achieves linear speedup and it also matches our intuition. To show the empirical results, the authors provide convergence results of A3C-TD(0) with Markovian sampling in synthetic environments and speedup of A3C-TD(0) in CartPole and Seaquest.