Zeroth-Order Actor-Critic

1 INTRODUCTION . Reinforcement learning ( RL ) has achieved great success in a wide range of challenging domains , including video games ( Mnih et al. , 2015 ) , robotic control ( Schulman et al. , 2017 ) , autonomous driving ( Kendall et al. , 2019 ) , etc . The majority of RL methods formulate the environment as Markov decision process ( MDP ) and leverage the temporal structure to design learning algorithms such as Q-learning and policy gradient ( Sutton & Barto , 2018 ) . Actor-critic methods are among the most popular RL algorithms , which usually introduce two function approximators , one for value function estimation ( critic ) and another for optimal policy approximation ( actor ) , and optimize these two approximators by alternating between policy evaluation ( PEV ) and policy improvement ( PIM ) . Onpolicy actor-critic methods , e.g. , A3C ( Mnih et al. , 2016 ) and PPO ( Schulman et al. , 2017 ) , often use critics to construct advantage functions and substitute them for the Monte Carlo return used in vanilla policy gradient ( Williams , 1992 ) , which significantly reduces the variance of gradient estimation and improve learning speed and stability . Among existing actor-critic algorithms , a common choice is to use deep neural networks as the function approximators and conduct both PEV and PIM using first-order optimization techniques . An alternative approach for RL , though less popular , is to ignore the underlying MDP structures and regard RL problems as black-box optimization , and to directly search for the optimal policy in a zeroth-order way , i.e. , without using the first-order gradient information . Recent researches have shown that zeroth-order optimization ( ZOO ) methods , e.g. , ES ( Salimans et al. , 2017 ) , ARS ( Mania et al. , 2018 ) and GA ( Such et al. , 2017 ) , are competitive on common RL benchmarks , even when applied to deep neural network with millions of parameters . ZOO has several advantages compared to first-order MDP-based RL methods ( Sehnke et al. , 2010 ; Salimans et al. , 2017 ; Such et al. , 2017 ; Lehman et al. , 2018 ; Khadka & Tumer , 2018 ; Qian & Yu , 2021 ) : ( 1 ) ZOO is not restricted to differentiable policies ; ( 2 ) ZOO perturbs the policy in parameter space rather than in action space , which leads to state-dependent and temporally-extended exploration ; ( 3 ) Zeroth-order populationbased optimization possesses robustness-seeking property and diverse policy behaviors . Despite these attractive advantages , the main limitation of ZOO is its high sample complexity and high variance of the parameter update process , especially in high-dimensional problems . Recent researches have proposed various techniques to improve ZOO , e.g. , using orthogonal or antithetic sampling methods ( Sehnke et al. , 2010 ; Salimans et al. , 2017 ; Choromanski et al. , 2018 ; Mania et al. , 2018 ) , identifying a low-dimensional search subspace ( Maheswaranathan et al. , 2019 ; Choromanski et al. , 2019 ; Sener & Koltun , 2020 ) , or subtracting a baseline for variance reduction ( Sehnke et al. , 2010 ; Grathwohl et al. , 2018 ) . One of the major reasons for the sample inefficiency of ZOO is its ignorance of the MDP temporal structures . Many recent researches have tried to combine ZOO and first-order MDP-based RL into hybrid methods , e.g. , run evolutionary algorithms in parallel with off-policy RL algorithms and optimize the population of policies with information from both sides ( Khadka & Tumer , 2018 ; Pourchot & Sigaud , 2018 ; Bodnar et al. , 2020 ) , or inject parameter noise into existing RL algorithms for efficient exploration ( Plappert et al. , 2018 ; Fortunato et al. , 2018 ) . However , existing hybrid methods still conduct first-order gradient-based policy improvement ( at least as a part ) , which reimposes differentiable requirement on the policy . In this paper , we propose the Zeroth-Order Actor-Critic algorithm ( ZOAC ) , which unifies first-order and zeroth-order RL methods into an actor-critic architecture by conducting first-order PEV to update the critic and zeroth-order PIM to update the actor . In such a way , complementary advantages of both methods are preserved , e.g. , wide adaptability to policy parameterization , robustness seeking property , state-dependent and temporally-extended exploration . We modify the rollouts collection strategy from episode-wise perturbation as in traditional zeroth-order methods to timestep-wise perturbation , which results in higher sample efficiency and better exploration . We derive the zerothorder policy gradient under this setting and point out that a critic network can be introduced to estimate the state-value function and trade-off between bias and variance . We then propose a practical algorithm that utilizes several parallelized rollout workers and alternates between first-order PEV and zeroth-order PIM based on generated experiences in each iteration . We evaluate ZOAC on a range of challenging continuous control benchmarks from OpenAI gym ( Brockman et al. , 2016 ) , using two different types of policies , linear policies and neural networks . Experiment results show that ZOAC outperforms zeroth-order and first-order baseline algorithms in sample efficiency , final performance , and the robustness of the learned policies . We visualize the polices learned in an environment with sparse and delayed reward , which indicates sufficient exploration driven by parameter noise in ZOAC . Furthermore , we conduct ablation studies to demonstrate the indispensable contribution of the modified rollouts collection strategy and the introduced critic network to ZOAC . 2 PRELIMINARIES . 2.1 FROM POLICY GRADIENT TO ACTOR-CRITIC . In standard MDP-based RL settings , the environment is usually formulated as an MDP defined as ( S , A , P , r ) , where S is the state space , A is the action space , P : S ×A×S → R is the transition probability matrix , r : S × A → R is the reward function . The return is defined as the total discounted future reward Gt = ∑∞ i=0 γ ir ( st+i , at+i ) , where γ ∈ ( 0 , 1 ) is the discounting factor . The behavior of the agent is controlled by a policy π ( a|s ) : S × A → [ 0 , 1 ] , which maps states to a probability distribution over actions . The state-value function is defined as the expected return under policy π starting from a certain state : V π ( s ) = Ea∼π { Gt|st = s } . The goal of MDP-based RL is to find an optimal policy that maximizes the expectation of state-value function under a certain state distribution . Denoting a policy parameterized with θ as πθ , the objective function can be written as : JPG ( θ ) = Es∼d [ V πθ ( s ) ] , d = d0 or dπθ ( 1 ) where d0 is the initial state distribution and dπθ is the stationary state distribution of Markov chain under policy πθ . Generally , the former is used for episodic tasks with finite horizon and the latter is used for continuing tasks with infinite horizon . For any differentiable policy πθ , and for continuing or episodic tasks , the same form of policy gradient can be derived from the policy gradient theorem ( Sutton & Barto , 2018 ) . This vanilla policy gradient given by Williams ( 1992 ) is as follows : ∇θJPG ( θ ) = Est∼dπθ , at∼πθ [ Gt∇θ log πθ ( at|st ) ] ( 2 ) Vanilla policy gradient suffers from high variance since it directly uses Monte Carlo return from sampled trajectories . Actor-critic methods improved upon it , which usually introduce a critic network to estimate the value function and serve as a baseline to substitute the expected return Gt with a proper form of advantage function At , for example , TD residual ( Mnih et al. , 2016 ) , or generalized advantage estimation ( GAE ) ( Schulman et al. , 2015 ) . However , the above policy gradient based methods can only be applied to differentiable policies , and may be unavailable when a nondifferentiable controller needs to be optimized . 2.2 EVOLUTION STRATEGIES . Existing ZOO methods focus on episodic RL problems with finite horizon and treat them as blackbox optimization . In these cases , the length of trajectories is limited and the discounting factor γ is usually set as 1 . Evolution strategies ( ES ) is one of the most popular algorithms of ZOO , which optimizes a Gaussian smoothed objective function : JES ( θ ) = E ∼N ( 0 , I ) Es∼d0 [ V πθ+σ ( s ) ] ( 3 ) where d0 is the initial state distribution and σ is the standard deviation of the Gaussian noise added to the policy . The zeroth-order gradient can be derived using the log-likelihood ratio trick and the probability density function of Gaussian distribution ( Nesterov & Spokoiny , 2017 ) : ∇θJES ( θ ) = 1 σ E ∼N ( 0 , I ) Es∼d0 [ V πθ+σ ( s ) ] ( 4 ) In practice , the expectation over Gaussian distribution can be approximated by sampling n noise samples { i } i=1 , ... , n , and the corresponding state value V πθ+σ i can be approximated by the episodic return Gi = ∑T t=0 γ tr ( st , at ) of the sample trajectory of length T collected with policy πθ+σ i : ∇θJES ( θ ) ≈ 1 nσ n∑ i=1 Gi i ( 5 ) The zeroth-order gradient estimator in Equation ( 5 ) only relies on the episodic return of each evaluated random directions , so it is applicable to non-differentiable policies . Besides , each perturbed policy remains deterministic in one trajectory , which leads to state-dependent and temporally-extended exploration . Furthermore , the Gaussian smoothed objective also improves robustness of the learned policies in parameter space . 3 ZEROTH-ORDER ACTOR-CRITIC . 3.1 FROM ES TO ZOAC . In this section , we will derive an improved zeroth-order gradient combining the actor-critic architecture for policy improvement . We start from improving the sample efficiency and stability of ES . Most of the existing ES methods applied to RL optimize a deterministic policy ( Salimans et al. , 2017 ; Mania et al. , 2018 ) , where the exploration is driven by noise in parameter space . Without loss of generality , we follow them in the following derivations and algorithm design . A deterministic policy parameterized with θ is denoted as πθ : S → A , which directly maps states to actions . In ES , the policy is perturbed in parameter space at the beginning of an episode and remains unchanged throughout the trajectories . If a large number of random directions n is evaluated , the sample complexity will increase significantly . However , since the zeroth-order gradient is estimated as the weighted sum of several random directions , it exhibit excessively high variance when n is small ( Berahas et al. , 2021 ) , which may greatly harm the performance . Therefore , it is essential to trade-off this contradictory between sample efficiency and variance . To encourage sufficient exploration and low variance while maintaining high sample efficiency , here we consider perturbating the policy at every timestep , i.e. , the Gaussian noise is sampled identically and independently at every timesteps . We regard it as a stochastic exploration policy β = πθ+σ , where ∼ N ( 0 , I ) is Gaussian noise in parameter space and σ is the standard deviation . Our objective is to maximize the expectation of the state-value under stationary distribution dβ of the exploration policy β . We can leverage Bellman equation to estimate the state-value via bootstrapping , in which the one-step reward can be replaced with sampled experiences : JZOAC ( θ ) = Est∼dβ [ V β ( st ) ] = E ∼N ( 0 , I ) Est∼dβEst+1∼P [ r ( st , πθ+σ ( st ) ) + γV β ( st+1 ) ] ( 6 ) Since all the contents in the outer expectation E ∼N ( 0 , I ) [ · ] can be regarded as a function of , the zeroth-order gradient of this objective function can be derived in exactly the same way as in ES . Moreover , V β ( st ) can be subtracted as a baseline for variance reduction because of its uncorrelation to in the outer expectation and the zero mean property of the Gaussian noise : ∇θJZOAC ( θ ) = 1 σ E ∼N ( 0 , I ) Est∼dβEst+1∼P { [ r ( st , πθ+σ ( st ) ) + γV β ( st+1 ) − V β ( st ) ] } ( 7 ) Compared to ES which uses unbiased but high variance Monte Carlo return to evaluate each perturbed policy , the performance of each random direction here is estimated by one-step TD residual with low variance . In practice , a common approach is to introduce a critic network Vw ( s ) to estimate the state-value function V β , which may lead to high bias in this form of advantage estimation . To trade-off between bias and variance , we consider extending our derivation further to the case where each perturbed policies ( i.e. , each sampled random noise ) run forward N timesteps instead of one timestep only . Equation ( 7 ) can be extended to : ∇θJZOAC ( θ ) = 1 σ E ∼N ( 0 , I ) Est∼dβEP { [ N−1∑ i=0 γir ( st+i , πθ+σ ( st+i ) ) + γ NV β ( st+N ) − V β ( st ) ] } ( 8 ) where EP refers to expectation over N -step transition dynamics . Similar to one-step case , the cumulative reward within N step can be estimated from sampled experiences when st is the first state of the trajectory fragment collected with a certain perturbed policy πθ+σ . By introducing a critic network and choosing an appropriate length N , this N -step residual advantage function contributes to achieving a good trade-off between the bias and variance .

Summary: This paper developed a very interesting zeroth-order actor-critic algorithm that nicely integrates gradient-based critic training with gradient-free actor training. The newly developed algorithm was also evaluated empirically on several benchmark datasets with promising outcomes. The literature review in the paper is very clear and comprehensive. The review adequately justified the technical innovation of the new algorithm.