Error Controlled Actor-Critic Method to Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) algorithms are combined with function approximation methods to adapt to the application scenarios whose state spaces are combinatorial , large , or even continuous . Many function approximation methods RL methods , including the Fourier basis ( Konidaris et al. , 2011 ) , kernel regression ( Xu , 2006 ; Barreto et al. , 2011 ; Bhat et al. , 2012 ) , and neural neworks ( Barto et al. , 1982 ; Tesauro , 1992 ; Boyan et al. , 1992 ; Gullapalli , 1992 ) have been used to learn value functions . In recent years , many deep reinforcement learning ( DRL ) methods were implemented by incorporating deep learning into RL methods . Deep Q-learning Network ( DQN ) ( Mnih et al. , 2013 ) reported by Mnih in 2013 is a typical work that uses a deep convolutional neural network ( CNN ) to represent a suitable action value function estimating future rewards ( returns ) ; it successfully learned end-to-end control policies for seven Atari 2600 games directly from large state spaces . Thereafter , deep RL methods , such as Deep Deterministic Policy Gradient ( DDPG ) ( Lillicrap et al. , 2016 ) , Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) , Twin Delayed Deep Deterministic policy gradient ( TD3 ) ( Fujimoto et al. , 2018 ) , and Soft Actor-Critic ( SAC ) ( Haarnoja et al. , 2018 ) , started to become mainstream in the field of RL . Althouth function approximation methods have assisted reinforcement learning ( RL ) algorithms to perform well in complex problems by providing great representation power ; however , they also cause an issue called overestimation phenomenon that jeopardize the optimization process of RL algorithms . Thrun & Schwartz ( 1993 ) presented a theoretical analysis of this systematic overestimation phenomenon in Q-learning methods that use function approximation methods . Similar problem persists in the actor-critic methods employed function approximation methods . Thomas ( 2014 ) reported that several natural actor-critic algorithms use biased estimates of policy gradient to update parameters when using function approximation to approximate the action value function . Fujimoto et al . ( 2018 ) proved that the value estimation in the deterministic policy gradient method also lead to overestimation problem . In brief , the approximation errors of value functions caused the inaccuracy of estimated values , and such inaccuracy induced the overestimation on value function ; so that poor performances might be assigned to high reward values . As a result , policies with poor performance might be obtained . Previous works attempted to find direct strategies to effectively reduce the overestimation . Hasselt ( 2010 ) proposed Double Q-learning , in which the samples are divided into two sets to train two independent Q-function estimators . To diminish the overestimation , one Q-function estimator is used to select actions , and the other one is applied to estimate its value . Fujimoto et al . ( 2018 ) proposed mechanisms , including clipped double Q-learning and delayed policy updates , to minimize the overestimation . In contrast to these methods , we focus on actor-critic setting and manage to reduce the approximation error of value function , which is the source of the overestimation , in an indirect but effective way . We use the concepts of domain adaptation ( Ben-David et al. , 2010 ) to derive an upper boundary of the approximation error in Q function approximator . Then , we find that the least upper bound of this error can be obtained by minimizing the Kullback-Leibler divergence ( KL divergence ) between new policy and its previous one . This means minimizing the KL divergence when traning policy can stabilize the critic and then confine the approximation error in Q function . Interestingly , we arrive at similar conclusion as two literatures Geist et al . ( 2019 ) ; Vieillard et al . ( 2020 ) by a somewhat different route . In their works , the authors directly studied the effect of KL and entropy regularization in RL and proved that a KL regularization indeed leads to averaging errors made at each iteration of value function update . While our idea is very different from theirs : It is impracticable to minimize the approximation error directly , so instead we try to minimize an upper bound of approximation error . This is similar to Expectation-Maximization Algorithm ( Bishop , 2006 ) which maximize a lower bound of log-likelihood instead of log-likelihood directly . We derive an upper boundary of approximation error for Q function approximatorin actor-critic methods , and arrive at a more general conclusion : approximation error can be reduced by keep new policy close to the previous one . Note that KL penalty is a effective way , but not the only way . Furthermore , the mentioned indirect operation ( i.e . the KL penalty ) can work together with the mentioned direct strategies for reducing overestimation , for example , clipped double Q-learning . Then , a new actor-critic method called Error Controlled Actor-critic ( ECAC ) is established by adopting an effective operation that minimizes the KL divergence to keep the upper bound as low as possible . In other words , this method ensures the similarity between every two consecutive polices in training process and reduces the optimization difficulty of value function , so that the error in Q function approximators can be decreased . Ablation studies were performed to examine the effectiveness of our proposed strategy for decreasing the approximation error , and comparative evaluations were conducted to verify that our method can outperform other mainstream RL algorithms . The main contributions of this paper are summarized as follow : ( 1 ) We presented an upper boundary of the approximation error in Q function approximator ; ( 2 ) We proposed a practical actor-critic method—ECAC which decreases the approximation error by restricting the KL divergence between every two consecutive policies and adopt a mechanism to automatically adjust the coefficient of KL term . 2 PRELIMINARIES . 2.1 REINFORCEMENT LEARNING . Reinforcement learning ( RL ) algorithms are modeled as a mathematical framework called Markov Decision Process ( MDP ) . In each time-step of MDP , an agent generates an action based on current state of its environment , then receives a reward and a new state from the environment . Environmental state and agent ’ s action at time t are denoted as st ∈ S and at ∈ A , respectively ; S and A denote the state and action spaces respectively , which may be either discrete or continuous . Environment is described by a reward function , r ( st , at ) , and a transition probability distribution , Pr ( st+1 = s′|st = s , at = a ) . Transition probability distribution specifies the probability that the environment will transition to next state . Initial state distribution is denoted as Pr0 ( s ) . Let π denotes a policy , η ( π ) denotes its expected discounted rewards : η ( π ) = Eπ [ R1 + γR2 + γ 2R3 + · · · ] = Eπ [ ∞∑ t=0 γkRt+1 ] , ( 1 ) where γ denotes a discount rate and 0 ≤ γ ≤ 1 . The goal of RL is to find a policy , π∗ , that maximizes a performance function over policy , J ( π ) , which measures the performance of policy : π∗ = argmax π J ( π ) . ( 2 ) A natural form of J ( π ) is η ( π ) . Different interpretations of this optimization goal lead to different routes to the their solutions . Almost all reinforcement learning algorithms involve estimating value functions , including statevalue and action-value functions . State-value function , V π ( s ) , gives the expected sum of discounted reward when starting in s and following a given policy , π. V π ( s ) specified by : V π ( s ) = Eπ [ ∞∑ k=0 γkRt+k+1|st = s ] . ( 3 ) Similarly , action-value function , Qπ ( s , a ) , is given by : Qπ ( s , a ) = Eπ [ ∞∑ k=0 γkRt+k+1|st = s , at = a ] . ( 4 ) 2.2 ACTOR-CRITIC ARCHITECTURE . To avoid confusion , by default , we discuss only RL methods with function approximation in this section . RL methods can be roughly divided into three categories : 1 ) value-based , 2 ) policy-based , and 3 ) actor-critic methods . Value-based method only learn value functions ( state-value or action-value functions ) , and have the advantage of fast convergence . Policy-based methods primarily learn parameterized policies . A parameterized policy ( with parameter vector , θ ) is either a distribution over actions given a state , πθ ( a|s ) , or a deterministic function , a = πθ ( s ) . Their basic update is θn+1 = θn+α∇J ( θn ) where is learning rate . Policy based methods show better convergence guarantees but have high variance in gradient estimates . Actor-critic methods learn both value functions and policies and use value functions to improve policies . In this way , they trade off small bias in gradient estimates to low variance in gradient estimates . Actor-critic architecture ( Peters & Schaal , 2008 ; Degris et al. , 2013 ; Sutton & Barto , 2018 ) consists of two components : actor and critic modules . Critic module learns learns state-value function , Vφ ( s ) , or action-value function , Qφ ( s , a ) or both of them , usually by temporal-difference ( TD ) methods . Actor module learns a stochastic policy , πθ ( a|s ) , or a deterministic policy , a = πθ ( s ) , and utilizes value function to improve the policy . For example , in actor module of DDPG ( Lillicrap et al. , 2016 ) , the policy is updated by using the following performance function J ( θ ) = Eπθ [ Qφ ( st , πθ ( st ) ) ] , ( 5 ) where πθ ( st ) is a deterministic policy . 2.3 DOMAIN ADAPTATION . Domain adaptation is a task which aims at adapting a well performing model from a source domain to a different target domain . It is used to describe the task of critic module in section 3.2 . The learning task of critic module is viewed as adapting a learned Q function approximator to next one , and the target error equates to the approximation error at current iteration of critic update . Here , we present some concepts in domain adaptation , including domain , source error , and target error . Domain is defined as a specific pair consisting of a distribution , P , on an input space , X , and a labeling function , f : X → R. In domain adaption , source and target domains are denoted as 〈PS , fS〉 and 〈PT , fT 〉 , respectively . A function , h : X → R , is defined as a hypothesis . Source error is the difference between a hypothesis , h ( x ) , and a labeling function of source domain , fS ( x ) , on a source distribution which is denoted as follow : eS ( h , fS ) = E x∼PS [ |h ( x ) − fS ( x ) | ] . ( 6 ) Target error is the difference between a hypothesis , h ( x ) , and a labeling function of target domain , fT ( x ) , on a target distribution which is denoted as follow : eT ( h , fT ) = E x∼PT [ |h ( x ) − fT ( x ) | ] . ( 7 ) For convenience , we use the shorthand eS ( h ) = eS ( h , fS ) and eT ( h ) = eT ( h , fT ) .

Summary: In this paper, the authors study the error introduced by the estimation of critic function in the Actor-Critic algorithm. Then the author proposed an algorithm that utilizes the idea of double Q learning and using a KL-divergence like regularization method to control this error. Experimentally the proposed algorithm achieves good results comparing to the vanilla Actor-Critic algorithm. This paper shows a successful routine from the theoretical analysis to a practical algorithm.