The wisdom of the crowd: reliable deep reinforcement learning through ensembles of Q-functions

Reinforcement learning agents learn by exploring the environment and then exploiting what they have learned . This frees the human trainers from having to know the preferred action or intrinsic value of each encountered state . The cost of this freedom is reinforcement learning is slower and more unstable than supervised learning . We explore the possibility that ensemble methods can remedy these shortcomings and do so by investigating a novel technique which harnesses the wisdom of the crowds by bagging Q-function approximator estimates . Our results show that this proposed approach improves all three tasks and reinforcement learning approaches attempted . We are able to demonstrate that this is a direct result of the increased stability of the action portion of the state-actionvalue function used by Q-learning to select actions and by policy gradient methods to train the policy . 1 INTRODUCTION . In the reinforcement learning ( RL ) approach the agent learns by exploring its environment and the , sometimes , many approaches to solving a given problem . It frees the human trainers from having to know the preferred action or intrinsic value of each encountered state . There is no denying that RL has a grassroots feel : it is an important form of learning in the natural world and it is only natural that machine learning practitioners would want to mimic its success . This freedom comes at a price , however . The most common complaints about RL , especially when using function approximators to learn Q-functions , are that RL is too slow and unstable during learning . Learning by exploration of the environment results in a training signal which is less informative than a supervised training signal resulting in the requirement of a large number of training samples and repeated exposure to those samples . The proposed approach is an ensemble learning approach to RL similar to the well-known bagging approach , Breiman ( 1996 ) , which trains ensemble members using the experience replay memory and combines their action selections using voting or averaging . 1.1 RECENT ADVANCEMENTS IN RL . There has been tremendous advancements in RL in recent years spurred by the excitement surrounding deep Q-learning ( DQN ) , Mnih et al . ( 2015 ) . Impressively , RL has featured prominently in published work showing super-human performance in tasks which were previously considered untouchable by state of the art RL approaches just a few years ago . These include RL agents playing chess at a grand-master level , Lai ( 2015 ) , and the famous AlphaGo , Silver et al . ( 2016 ) . Through this work we have found just how far we can go when applying existing RL approaches with large amounts of computing power . A large effort has been devoted to addressing the RL obstacle of slow learning , Mnih et al . ( 2016 ) ; Schaul et al . ( 2016 ) , with an emphasis on speeding-up RL for the high-dimension inputs popularized by the DQN work – especially since DQN exacerbated this issue both in terms of the required computation and number of training time steps . There has also been work showing other RL approaches can be adapted to the DQN paradigm , Lillicrap et al . ( 2016 ) . For all of these recent works mentioned here the primary emphasis is achieving improved performance on simulated tasks in the shortest amount of wall-clock time . The issues of RL instability during training and of reducing the number of interactions with the training environment have received little attention . The crowd ensemble addresses the obstacle of training instability without sacrificing the number of interactions with the training environment which , in many real-world application , may be more expensive than the computational costs . Furthermore , the crowd ensemble method can be used alongside any of these recent advances in RL . 1.2 THE WISDOM OF CROWDS . Francis Galton , in 1906 , observed that a large group ’ s mean guess was able to come within one pound when guessing the weight of an Ox , Galton ( 1907 ) . This was surprising because the crowd , while containing a few potential experts , was presumably made of non-experts with no knowledge of estimating the weight of oxen . More recently Treynor ( 1987 ) emphasizes the need for independence of the individual to the success of the crowd and that allowing sources of shared error or bias will reduce the accuracy of a crowd ’ s prediction . He concludes that chasing an expert is folly and that a simple combination of crowd information is best . Larrick & Soll ( 2006 ) and Hastie & Kameda ( 2005 ) emphasize the power of averaging predictions . Appropriate use of averaging in group decisions begins with understanding that averaging is not a regression to the mean but , rather , an error-reduction technique Surowiecki ( 2005 ) synthesizes the many benefits of crowd-based decision-making . Although the crowd will not regularly out-perform the best individual , no field can provide a mechanism to predict which individual will out-perform the crowd . Fortunately , many of the obstacles to quality crowd decision-making are a result of the shortcomings and complexities of human group dynamics are not problems we encounter when training Q-function ensembles . 2 RELATED WORK . Most ensemble RL methods fit the mixture of experts paradigm described in Jacobs et al . ( 1991 ) . A Gaussian mixture model approach is utilized as the ensemble mechanism by Agostini & Celaya ( 2011 ) . This gives each ensemble member a region of expertise . They test their approach on the pendulum swing-up and cart-pole balancing tasks . An couple ensembles of ANN Q-function approximators are compared in a limited set of experiments on the pole-balancing task in Hans & Udluft ( 2010 ) . They conclude that a hard combination of ensemble members , that is only one expert is active at a time , is superior to the soft-combination of experts they initially attempted . In the multiple model reinforcement learning ( MMRL ) , Doya & Samejima ( 2002 ) , approach each expert has a forward model and a Q-function . The forward models determine how to combine member outputs and how to backpropogate the error signals . In Doya & Samejima ( 2002 ) MMRL is tested on the non-stationary pendulum swing-up task . For these experiments they are forced manually partition the state space . In Faußer & Schwenker ( 2015 ) a concept very similar to the crowd ensemble method is proposed . Their approach has an important difference : the ensemble members are trained in parallel with each other . This results in a factor of Ne more interactions with the training environment . Their results show that their approach with Ne > 3 performs better than a single Q-learner on a maze navigation task and a simplified tetris task ( SZ-tetris ) with a reduced set of pieces . No explanation is given for the improved performance of the ensemble . Recently Q-learning approaches have been developed which resemble ensemble methods and may share some of the same benefits . These approaches are not ensembles because they do not have a mechanism which combines ensemble member outputs . Mnih et al . ( 2016 ) and Nair & Silver ( 2015 ) present a RL approach which uses multiple , simultaneous simulations to speed-up DQN . In Mnih et al . ( 2016 ) this method is called asynchronous DQN . True to its name , the parameter updates are not synchronized allowing each simulation to periodically update the global parameter values using an accumulated gradient over several time steps . Asynchronous DQN is tested using a large number of arcade learning environment . Double Q-learning , Hasselt ( 2010 ) . is designed to address the issue of over estimation of Q-values . This is done by leveraging two Q-functions . In a later work the double Q-learning concept is adapted to the DQN framework van Hasselt et al . ( 2015 ) . The double Q-learning concept is taken a step further by investigating using any number of Q-learners which they call multi Q-learning , Duryea ( 2016 ) . 3 METHOD . Here we briefly describe the crowd ensemble approach to Q-learning and the experiments used to evaluated the proposed method . We begin with the crowd ensemble . 3.1 THE CROWD ENSEMBLE APPROACH TO Q-LEARNING . Q-learning , Sutton & Barto ( 1998 ) , learns a state-action function , Q ( s , a ) which is updated according to : Q ( s ( t ) , a ( t ) ) ← Q ( s ( t ) , a ( t ) ) + α [ r ( t + 1 ) + γmax a ′ ( Q ( s ( t+ 1 ) , a′ ) ) −Q ( s ( t ) , a ( t ) ) ] ( 1 ) where t is the current time step , Q ( s ( t ) , a ( t ) ) is the Q-function value for state/action pair ( s ( t ) , a ( t ) ) , s ( t + 1 ) is the state which occurs when taking action a ( t ) from state s ( t ) , r ( t ) is the reward at time t , α is a learning rate parameter , and γ is a parameter modulating the effect of future rewards . The value maxa′ ( Q ( s ( t + 1 ) , a ′ ) ) is the current estimate of future cumulative reward : it represents the current belief of the best possible action , a′ , taken at the next state , s ( t+ 1 ) . Using a feed-forward artificial neural network ( ANN ) to model a Q-function is a common function approximation method when applying Q-learning to tasks with continuous state spaces , Anderson et al . ( 2015 ) . This is typically done by having a single ANN output represent Q ( s , a ) and inputting s and a into the network . Updates are performed by backpropogating the error computed in ( 1 ) through the ANN . Alternatively , as is done in the DQN approach , the ANN can have a multiple outputs , one for each Q ( s|a ) , the value of Q ( s , a ) given an action , with only the state as inputs . Prior to learning , the Q-function approximator parameters for each expert should be randomly initialized . Learning in a crowd ensemble follows these steps . 1 . Vote : select an action for each expert . 2 . Tally : determine which action is selected by the most members . 3 . Act : take the action selected by the ensemble . 4 . Observe : store the new state and store it in the experience replay memory . 5 . Sample : independently sample from the experience replay memory for each member . 6 . Compute errors : compute a TD-error for each member . 7 . Update : update Q-function approximations using each member ’ s TD-error . Our implementation of a Q-learning crowd ensemble is described in greater detail in Algorithm 1 . In our experiments no new hyper-parameter search is conducted when applying a crowd ensemble : the parameters found via a manual search for the baseline approaches are also used for the crowd ensemble members . 4 EXPERIMENTS . The purpose of the cart-pole task is to start a trial with the agent ’ s pole in the down position and allow the agent to move the cart back and forth along a 2-D track in order to swing the pole up and balance it . The track is of finite length and each end of the track has a wall with which the cart interacts via elastic collisions . The cart-pole task has four state variables : cart position , cart velocity , pole angle , and pole angular velocity . The actions are discrete with a ∈ { −1 , 0 , 1 } which translate to push left , no push , and push right . The agent is rewarded a negative one when pointing downward , a positive one when pointing upward , and zero elsewhere . We approximate a Q-function using an ANN with a single hidden layer with 20 nodes . The ANN inputs are the four state dimensions and scalar action value . The ANN output is the associated Q ( s , a ) value . The parameters are updated using scaled conjugate gradient , Møller ( 1993 ) . We store all experiences and sample them in batches of 1000 for each update . We update the parameters using five batches every 1000 time steps during training . The training simulation is run continuously and is never reset . Evaluation is performed using a separate simulation which is reset for each evaluation . We also evaluate using a high-dimension state representation of the cart-pole task where the state is represented by two consecutive frames of an image of the cart-pole environment . The approach is similar to the DQN approach described in Mnih et al . ( 2015 ) . The ANN has two convolutional and two fully-connected layers . No batch normalization layers are used . The convolutional layers use ReLU activation functions while the fully-connected layers use tanh . The convolutional layers have 20 and 40 features , respectively . The window size of the first convolutional layer is 6 × 6 and the second is 4 × 4 . The strides for both layers is 2× 2 meaning a new window starts every two pixels in both directions leading to overlap of the windows . The fully-connected layers are of size 100 and 20 , respectively . The parameters are updated using ADAM , Kingma & Ba ( 2015 ) . The DQN approach requires a large number of parameters so we share the convolutional layers between the ensemble members . This is done by accumulating the gradient from the fully-connected layers of all ensemble members and using it update the shared convolutional layers . When applying the crowd ensemble via DQN the same cart-pole simulation is used by the inputs are two sequential frames of the simulation . These can be seen in Section 8.3 along with some example features extracted by the CNN . We also apply the crowd ensemble approach to the continuous state-action bipedal walker task , team ( 2016 ) , via the DDPG algorithm which is a actor-critic approach , ( Sutton & Barto , 1998 , 69 ) . The bipedal walker task objective is to train an agent to move a simple bipedal robot across a twodimension , set-width , gently sloping , plane . The robot consists of two legs and an oblong hull which sits on top of the legs . The action space is continuous in four dimensions : the actuations for the hip and knee in each leg of the robot . The state space is represented in 24 dimensions : hull angle , hull angular velocity , hull x velocity , hull y velocity , leg one hip angle , leg one hip speed , leg one knee angle , leg one knee velocity , leg one ground contact indicator ( boolean ) , leg two hip angle , leg two hip speed , leg two knee angle , leg two knee velocity , leg two ground contact indicator ( boolean ) , and ten lidar measurements measuring the distance to the ground from the center of the hull from ten different angles . The reward function is designed to reward forward motion with minimal motor actuation while encouraging the agent to keep the hull from pointing downward and severely penalizing the agent if the hull touches the ground . The task is considered solved if the entire course is traversed within the allocated amount of time with a total reward greater than 300 . The parameters used for training the actor and critic ANNs for the bipedal walker were taken from the supplementary material of Lillicrap et al . ( 2016 ) . DDPG views the actor outputs as defining the mean of a Normal distribution with unit variance . Combining the output of multiple actors using averaging will , most likely , result in a location of exceedingly small probability . Furthermore actor outputs will be multi-modal in four dimensions which will result in a challengingly large number of modes for which no straight-forward method to find the highest-probability location exists . Instead the crowd ensemble approach is applied to DDPG by training a single actor from the combined output of a crowd ensemble of critics .

Summary: This paper proposes the deep reinforcement learning with ensembles of Q-functions. Its main idea is updating multiple Q-functions, instead of one, with independently sampled experience replay memory, then take the action selected by the ensemble. Experimental results demonstrate that the proposed method can achieve better performance than non-ensemble one under the same training steps, and the decision space can also be stabilized.