Learning  agents with prioritization and parameter noise in continuous state and action space

1 INTRODUCTION . Reinforcement learning ( RL ) is about an agent interacting with the environment , learning an optimal policy , by trail and error , for sequential decision making problems in a wide range of fields such that the agent learns to control a system and maximize a numerical performance measure that expresses a long-term objective ( Li ( 2017 ) ) . 1.1 RL IN CONTINUOUS STATE AND ACTION SPACE . Many interesting real-world control tasks , such as driving a car or riding a snowboard , require smooth continuous actions taken in response to high-dimensional , real-valued sensory input . In applying RL to continuous problems , the most common approach for a long time has been first to discretize state and action spaces and then to apply an RL algorithm for a discrete stochastic system ( Doya ( 2000 ) ) . However , this discretization approach has a number of drawbacks . Hence , the formulation of the reinforcement learning problem with continuous state and action space holds great value in solving more real-world problems . 1.2 DEEP REINFORCEMENT LEARNING . The advent of deep learning has had a significant impact on many areas in machine learning , dramatically improving the state-of-the-art tasks such as object detection , speech recognition and language translation LeCun et al . ( 2015 ) . The most important property of deep learning is that deep neural networks can automatically find low-dimensional representations of high-dimensional data.Therefore , deep learning enables RL to scale to problems which were previously intractable - setting with high dimensional/continuous state and action space Arulkumaran et al . ( 2017 ) . Few of the current state of the art methods in the area of Deep RL are : • Deep Q-learning Networks ( DQN ) - introduced novel concepts which helped in using neural networks as function approximators for reinforcement learning algorithms ( for continuous state space ) Mnih et al . ( 2013 ) . • Prioritized Experience Replay ( PER ) - builds upon DQN with some newer approaches to outperform DQN ( for continuous state space ) Schaul et al . ( 2015 ) . • Deep Deterministic Policy Gradients ( DDPG ) - follows a different paradigm as compared to the above methods . It uses DQN as the function approximator while building on the seminal work of Silver et al . ( 2014 ) ( for both continuous state and action space ) on deterministic policy gradients . 1.3 PRIORITIZED EXPERIENCE REPLAY IN DDPG ( PDDPG ) . We propose a new algorithm , Prioritized DDPG using the ideas proposed in DQN , prioritized experience replay and DDPG such that it outperforms the DDPG in the continuous state and action space . Prioritized DDPG uses the concept of prioritized sampling in the function approximator of DDPG . Our results show that prioritized DDPG outperforms DDPG in a majority of the continuous action space environments . We then use the concept of parameter space noise for exploration and show that this further improves the rewards achieved . 2 PREVIOUS WORK . 2.1 CRITIC METHODS OF RL . Critic methods rely exclusively on function approximation and aim at learning a “ good ” approximation of the value functionKonda & Tsitsiklis ( 2000 ) . We survey a few of the recent best known critic methods in RL . 2.1.1 DEEP Q-LEARNING NETWORKS . As in any value function-based approach , DQN method tries to find the value function for each state and then tries to find the optimal function . Min et al . Silver et al . ( 2014 ) also consider the approach of continuous state spaces . The novelty of their approach is that they use a non-linear function approximator efficiently . Until their work , non-linear function approximators were inefficient and also had convergence issues . This was due to the fact that there was a lot of correlation between the data being fed to the neural networks which resulted in them diverging ( neural networks assume that the data comes from independent and identically distributed source ) . To overcome this drawback , the novel ideas that were proposed which made it possible to use non-linear function approximator for reinforcement learning are the following : • Experience Replay Buffer : In this technique , the input given to the neural network is selected at random from a large buffer of stored observations , which ensures that there are no correlations in the data , which is a requirement for neural networks . • Periodic Target Network Updates : The authors propose that having two sets of parameters for the same neural networks can be beneficial . The two sets of parameters are used , one of the parameters is used to compute the target at any given iteration and the other , network parameters are used in the loss computation and are updated by the first network parameters periodically , this also ensures lesser co-relation . 2.1.2 PRIORITIZED EXPERIENCE REPLAY . The prioritized experience replay algorithm is a further improvement on the deep Q-learning methods and can be applied to both DQN and Double DQN . The idea proposed by the authors is as follows : instead of selecting the observations at random from the replay buffer , they can be chosen based on some criteria which will help in making the learning faster . Intuitively , what they are trying here is to replace those observations which do not contribute to learning or learning with more useful observations . To select these more useful observations , the criteria used was the error of that particular observation . This criterion helps in selecting those observations which help the agent most in learning and hence speeds up the learning process . The problem with this approach is that greedy prioritization focuses on a small subset of the experience and this lack of diversity might lead to over-fitting . Hence , the authors introduce a stochastic sampling method that interpolates between pure greedy and uniform random prioritization . Hence , the new probability if sampling a transition i is P ( j ) = pαj∑ k p α k ( 1 ) where pi is the priority of transition i and α determines how much prioritization is used . This approach , while it improves the results has a problem of changing the distribution of the expectation . This is resolved by the authors by using Importance Sampling ( IS ) weights wi = ( 1 N · 1 P ( i ) ) β ( 2 ) where if β = 1 , the non-uniform probabilities P ( i ) are fully compensated ( Schaul et al . ( 2015 ) ) . 2.2 ACTOR METHODS IN RL . Actor methods work with a parameterized family of policies . The gradient of the performance , with respect to the actor parameter is directly estimated by simulation , and the parameters are updated in the direction of improvement ( Konda & Tsitsiklis ( 2000 ) ) . 2.2.1 DETERMINISTIC POLICY GRADIENTS ( DPG ) . The most popular policy gradient method is the deterministic policy gradient ( DPG ) method and in this approach , instead of having a stochastic policy which we have seen till now , the authors make the policy deterministic and then determine the policy gradient . The deterministic policy gradient is the expected gradient of the action-value function , which integrates over the state space ; whereas in the stochastic case , the policy gradient integrates over both state and action spaces . What this leads to is that the deterministic policy gradient can be estimated more efficiently than the stochastic policy gradient . The DPG algorithm , presented by Silver et al . ( 2014 ) maintains a parameterized actor function µ ( s|θµ ) which is the current deterministic policy that maps a state to an action.They used the normal Bellman equation to update the critic Q ( s , a ) . They then went on to prove that the derivative expected return with respect to actor parameters is the gradient of the policy ’ s performance . 3 ACTOR-CRITIC METHODS . Actor critic models ( ACM ) are a class of RL models that separate the policy from the value approximation process by parameterizing the policy separately . The parameterization of the value function is called the critic and the parameterization of the policy is called the actor . The actor is updated based on the critic which can be done in different ways , while the critic is update based on the current policy provided by the actor ( Konda & Tsitsiklis ( 2000 ) Feinberg & Shwartz ( 2002 ) ) . 3.0.1 DEEP DETERMINISTIC POLICY GRADIENTS The DDPG algorithm tries to solve the reinforcement learning problem in the continuous action and state space setting . The authors of this approach extend the idea of deterministic policy gradients . What they add to the DPG approach is the use of a non-linear function approximator ( Lillicrap et al . ( 2015 ) ) . While using a deterministic policy , the action value function reduces from Qπ ( st , at ) = Ert , st+1∼E [ r ( st , at ) + γEat+1∼π [ Qπ ( st+1 , at+1 ) ] ] ( 3 ) to Qµ ( st , at ) = Ert , st+1∼E [ r ( st , at ) + γQµ ( st+1 , µ ( st+1 ) ) ] ( 4 ) as the inner expectation is no longer required . What this also tells us is that the expectation depends only on the environment and nothing else . Hence , we can learn off-policy , that is , we can train our reinforcement learning agent by using the observations made by some other agent . They then use the novel concepts used in DQN to construct their function approximator . These concepts could not be applied directly to continuous action space , as there is an optimization over the action space at every step which is in-feasible when there is a continuous action space ( Lillicrap et al . ( 2015 ) ) . Once we have both the actor and the critic networks with their respective gradients , we can then use the DQN concepts - replay buffer and target networks to train these two networks . They apply the replay buffer directly without any modifications but make small changes in the way target networks are used . Instead of directly copying the values from the temporary network to the target network , they use soft updates to the target networks . 4 PRIORITIZED DEEP DETERMINISTIC POLICY GRADIENTS . The proposed algorithm is primarily an adaptation of DQN and DDPG with ideas from the work of Schaul et al . ( 2015 ) on continuous control with deep reinforcement learning to design a RL scheme that improves on DDPG significantly . The intuition behind the idea is as follows : The DDPG algorithm uses the DQN method as a sub-algorithm and any improvement over the DQN algorithm should ideally result in the improvement of the DDPG algorithm . But from the above-described methods , not all algorithms which improve DQN can be used to improve DDPG . That is because some of them need the environment to have discrete action spaces . So , for our work , we will consider only prioritized experience replay method which does not have this constraint . 4.1 PRIORITIZED DDPG ALGORITHM . Now , the improvement to the DQN algorithm , the prioritized action replay method can be integrated into the DDPG algorithm in a very simple way . Instead of using just DQN as the function approximator , we can use DQN with prioritized action replay . That is , in the DDPG algorithm , instead of selecting observations randomly , we select the observations using the stochastic sampling method as defined in equation 1 . The pseudo-code for the prioritized action replay is given in Algorithm 1 . This algorithm is quite similar to the original DDPG algorithm with the only changes being the way the observations are selected for training in line 11 and the transition probabilities are being updated in line 16 . The first change ensures we are selecting the better set of observations which help in learning faster and the second change helps in avoiding over-fitting as it ensures all the observations have a non-zero probability of being selected to train the network and only a few high error observations are not used multiple times to train the network .

Summary: This paper combines elements of two existing reinforcement learning approaches, namely, Deep Q-learning Networks (DQN) with Prioritised Experience Replay (PER) and Deep Deterministic Policy Gradient (DDPG) to propose the Prioritized Deep Deterministic Policy Gradient (PDDPG) algorithm. The problem is interesting and there is a nice review of relevant work. The algorithm has a limited novelty with a simple modification of the DDPG algorithm to add the PER component. Experiment results show improvements in certain simulation environments. However, the paper lacks insight on how and why results are improved on some settings while performing worse than the others. Detailed comments are as follows: