Zeroth-Order Actor-Critic

1 INTRODUCTION . Reinforcement learning ( RL ) has achieved great success in a wide range of challenging domains , including video games ( Mnih et al. , 2015 ) , robotic control ( Schulman et al. , 2017 ) , autonomous driving ( Kendall et al. , 2019 ) , etc . The majority of RL methods formulate the environment as Markov decision process ( MDP ) and leverage the temporal structure to design learning algorithms such as Q-learning and policy gradient ( Sutton & Barto , 2018 ) . Actor-critic methods are among the most popular RL algorithms , which usually introduce two function approximators , one for value function estimation ( critic ) and another for optimal policy approximation ( actor ) , and optimize these two approximators by alternating between policy evaluation ( PEV ) and policy improvement ( PIM ) . Onpolicy actor-critic methods , e.g. , A3C ( Mnih et al. , 2016 ) and PPO ( Schulman et al. , 2017 ) , often use critics to construct advantage functions and substitute them for the Monte Carlo return used in vanilla policy gradient ( Williams , 1992 ) , which significantly reduces the variance of gradient estimation and improve learning speed and stability . Among existing actor-critic algorithms , a common choice is to use deep neural networks as the function approximators and conduct both PEV and PIM using first-order optimization techniques . An alternative approach for RL , though less popular , is to ignore the underlying MDP structures and regard RL problems as black-box optimization , and to directly search for the optimal policy in a zeroth-order way , i.e. , without using the first-order gradient information . Recent researches have shown that zeroth-order optimization ( ZOO ) methods , e.g. , ES ( Salimans et al. , 2017 ) , ARS ( Mania et al. , 2018 ) and GA ( Such et al. , 2017 ) , are competitive on common RL benchmarks , even when applied to deep neural network with millions of parameters . ZOO has several advantages compared to first-order MDP-based RL methods ( Sehnke et al. , 2010 ; Salimans et al. , 2017 ; Such et al. , 2017 ; Lehman et al. , 2018 ; Khadka & Tumer , 2018 ; Qian & Yu , 2021 ) : ( 1 ) ZOO is not restricted to differentiable policies ; ( 2 ) ZOO perturbs the policy in parameter space rather than in action space , which leads to state-dependent and temporally-extended exploration ; ( 3 ) Zeroth-order populationbased optimization possesses robustness-seeking property and diverse policy behaviors . Despite these attractive advantages , the main limitation of ZOO is its high sample complexity and high variance of the parameter update process , especially in high-dimensional problems . Recent researches have proposed various techniques to improve ZOO , e.g. , using orthogonal or antithetic sampling methods ( Sehnke et al. , 2010 ; Salimans et al. , 2017 ; Choromanski et al. , 2018 ; Mania et al. , 2018 ) , identifying a low-dimensional search subspace ( Maheswaranathan et al. , 2019 ; Choromanski et al. , 2019 ; Sener & Koltun , 2020 ) , or subtracting a baseline for variance reduction ( Sehnke et al. , 2010 ; Grathwohl et al. , 2018 ) . One of the major reasons for the sample inefficiency of ZOO is its ignorance of the MDP temporal structures . Many recent researches have tried to combine ZOO and first-order MDP-based RL into hybrid methods , e.g. , run evolutionary algorithms in parallel with off-policy RL algorithms and optimize the population of policies with information from both sides ( Khadka & Tumer , 2018 ; Pourchot & Sigaud , 2018 ; Bodnar et al. , 2020 ) , or inject parameter noise into existing RL algorithms for efficient exploration ( Plappert et al. , 2018 ; Fortunato et al. , 2018 ) . However , existing hybrid methods still conduct first-order gradient-based policy improvement ( at least as a part ) , which reimposes differentiable requirement on the policy . In this paper , we propose the Zeroth-Order Actor-Critic algorithm ( ZOAC ) , which unifies first-order and zeroth-order RL methods into an actor-critic architecture by conducting first-order PEV to update the critic and zeroth-order PIM to update the actor . In such a way , complementary advantages of both methods are preserved , e.g. , wide adaptability to policy parameterization , robustness seeking property , state-dependent and temporally-extended exploration . We modify the rollouts collection strategy from episode-wise perturbation as in traditional zeroth-order methods to timestep-wise perturbation , which results in higher sample efficiency and better exploration . We derive the zerothorder policy gradient under this setting and point out that a critic network can be introduced to estimate the state-value function and trade-off between bias and variance . We then propose a practical algorithm that utilizes several parallelized rollout workers and alternates between first-order PEV and zeroth-order PIM based on generated experiences in each iteration . We evaluate ZOAC on a range of challenging continuous control benchmarks from OpenAI gym ( Brockman et al. , 2016 ) , using two different types of policies , linear policies and neural networks . Experiment results show that ZOAC outperforms zeroth-order and first-order baseline algorithms in sample efficiency , final performance , and the robustness of the learned policies . We visualize the polices learned in an environment with sparse and delayed reward , which indicates sufficient exploration driven by parameter noise in ZOAC . Furthermore , we conduct ablation studies to demonstrate the indispensable contribution of the modified rollouts collection strategy and the introduced critic network to ZOAC . 2 PRELIMINARIES . 2.1 FROM POLICY GRADIENT TO ACTOR-CRITIC . In standard MDP-based RL settings , the environment is usually formulated as an MDP defined as ( S , A , P , r ) , where S is the state space , A is the action space , P : S ×A×S → R is the transition probability matrix , r : S × A → R is the reward function . The return is defined as the total discounted future reward Gt = ∑∞ i=0 γ ir ( st+i , at+i ) , where γ ∈ ( 0 , 1 ) is the discounting factor . The behavior of the agent is controlled by a policy π ( a|s ) : S × A → [ 0 , 1 ] , which maps states to a probability distribution over actions . The state-value function is defined as the expected return under policy π starting from a certain state : V π ( s ) = Ea∼π { Gt|st = s } . The goal of MDP-based RL is to find an optimal policy that maximizes the expectation of state-value function under a certain state distribution . Denoting a policy parameterized with θ as πθ , the objective function can be written as : JPG ( θ ) = Es∼d [ V πθ ( s ) ] , d = d0 or dπθ ( 1 ) where d0 is the initial state distribution and dπθ is the stationary state distribution of Markov chain under policy πθ . Generally , the former is used for episodic tasks with finite horizon and the latter is used for continuing tasks with infinite horizon . For any differentiable policy πθ , and for continuing or episodic tasks , the same form of policy gradient can be derived from the policy gradient theorem ( Sutton & Barto , 2018 ) . This vanilla policy gradient given by Williams ( 1992 ) is as follows : ∇θJPG ( θ ) = Est∼dπθ , at∼πθ [ Gt∇θ log πθ ( at|st ) ] ( 2 ) Vanilla policy gradient suffers from high variance since it directly uses Monte Carlo return from sampled trajectories . Actor-critic methods improved upon it , which usually introduce a critic network to estimate the value function and serve as a baseline to substitute the expected return Gt with a proper form of advantage function At , for example , TD residual ( Mnih et al. , 2016 ) , or generalized advantage estimation ( GAE ) ( Schulman et al. , 2015 ) . However , the above policy gradient based methods can only be applied to differentiable policies , and may be unavailable when a nondifferentiable controller needs to be optimized . 2.2 EVOLUTION STRATEGIES . Existing ZOO methods focus on episodic RL problems with finite horizon and treat them as blackbox optimization . In these cases , the length of trajectories is limited and the discounting factor γ is usually set as 1 . Evolution strategies ( ES ) is one of the most popular algorithms of ZOO , which optimizes a Gaussian smoothed objective function : JES ( θ ) = E ∼N ( 0 , I ) Es∼d0 [ V πθ+σ ( s ) ] ( 3 ) where d0 is the initial state distribution and σ is the standard deviation of the Gaussian noise added to the policy . The zeroth-order gradient can be derived using the log-likelihood ratio trick and the probability density function of Gaussian distribution ( Nesterov & Spokoiny , 2017 ) : ∇θJES ( θ ) = 1 σ E ∼N ( 0 , I ) Es∼d0 [ V πθ+σ ( s ) ] ( 4 ) In practice , the expectation over Gaussian distribution can be approximated by sampling n noise samples { i } i=1 , ... , n , and the corresponding state value V πθ+σ i can be approximated by the episodic return Gi = ∑T t=0 γ tr ( st , at ) of the sample trajectory of length T collected with policy πθ+σ i : ∇θJES ( θ ) ≈ 1 nσ n∑ i=1 Gi i ( 5 ) The zeroth-order gradient estimator in Equation ( 5 ) only relies on the episodic return of each evaluated random directions , so it is applicable to non-differentiable policies . Besides , each perturbed policy remains deterministic in one trajectory , which leads to state-dependent and temporally-extended exploration . Furthermore , the Gaussian smoothed objective also improves robustness of the learned policies in parameter space . 3 ZEROTH-ORDER ACTOR-CRITIC . 3.1 FROM ES TO ZOAC . In this section , we will derive an improved zeroth-order gradient combining the actor-critic architecture for policy improvement . We start from improving the sample efficiency and stability of ES . Most of the existing ES methods applied to RL optimize a deterministic policy ( Salimans et al. , 2017 ; Mania et al. , 2018 ) , where the exploration is driven by noise in parameter space . Without loss of generality , we follow them in the following derivations and algorithm design . A deterministic policy parameterized with θ is denoted as πθ : S → A , which directly maps states to actions . In ES , the policy is perturbed in parameter space at the beginning of an episode and remains unchanged throughout the trajectories . If a large number of random directions n is evaluated , the sample complexity will increase significantly . However , since the zeroth-order gradient is estimated as the weighted sum of several random directions , it exhibit excessively high variance when n is small ( Berahas et al. , 2021 ) , which may greatly harm the performance . Therefore , it is essential to trade-off this contradictory between sample efficiency and variance . To encourage sufficient exploration and low variance while maintaining high sample efficiency , here we consider perturbating the policy at every timestep , i.e. , the Gaussian noise is sampled identically and independently at every timesteps . We regard it as a stochastic exploration policy β = πθ+σ , where ∼ N ( 0 , I ) is Gaussian noise in parameter space and σ is the standard deviation . Our objective is to maximize the expectation of the state-value under stationary distribution dβ of the exploration policy β . We can leverage Bellman equation to estimate the state-value via bootstrapping , in which the one-step reward can be replaced with sampled experiences : JZOAC ( θ ) = Est∼dβ [ V β ( st ) ] = E ∼N ( 0 , I ) Est∼dβEst+1∼P [ r ( st , πθ+σ ( st ) ) + γV β ( st+1 ) ] ( 6 ) Since all the contents in the outer expectation E ∼N ( 0 , I ) [ · ] can be regarded as a function of , the zeroth-order gradient of this objective function can be derived in exactly the same way as in ES . Moreover , V β ( st ) can be subtracted as a baseline for variance reduction because of its uncorrelation to in the outer expectation and the zero mean property of the Gaussian noise : ∇θJZOAC ( θ ) = 1 σ E ∼N ( 0 , I ) Est∼dβEst+1∼P { [ r ( st , πθ+σ ( st ) ) + γV β ( st+1 ) − V β ( st ) ] } ( 7 ) Compared to ES which uses unbiased but high variance Monte Carlo return to evaluate each perturbed policy , the performance of each random direction here is estimated by one-step TD residual with low variance . In practice , a common approach is to introduce a critic network Vw ( s ) to estimate the state-value function V β , which may lead to high bias in this form of advantage estimation . To trade-off between bias and variance , we consider extending our derivation further to the case where each perturbed policies ( i.e. , each sampled random noise ) run forward N timesteps instead of one timestep only . Equation ( 7 ) can be extended to : ∇θJZOAC ( θ ) = 1 σ E ∼N ( 0 , I ) Est∼dβEP { [ N−1∑ i=0 γir ( st+i , πθ+σ ( st+i ) ) + γ NV β ( st+N ) − V β ( st ) ] } ( 8 ) where EP refers to expectation over N -step transition dynamics . Similar to one-step case , the cumulative reward within N step can be estimated from sampled experiences when st is the first state of the trajectory fragment collected with a certain perturbed policy πθ+σ . By introducing a critic network and choosing an appropriate length N , this N -step residual advantage function contributes to achieving a good trade-off between the bias and variance .

Summary: The paper proposes an actor-critic reinforcement learning algorithm, compatible with continuous actions, that combines a Monte-Carlo computation of the gradient of the actor (based on workers that perform rollouts from the current state) with a standard supervised-learning based critic update rule (using gradient descent for several epochs). The core contribution of the paper seems to be the method that combines the critic network V(s) with on-policy samples, using a form of eligibility traces. This combination is used to compute target V-Values for the critic, and the weighted average used to compute the gradient of the actor. Empirical results on MuJoCo show that the proposed algorithm outperforms PPO in sample-efficiency and final policy quality (extra experiments show that robustness is high too).