Mixture of Step Returns in Bootstrapped DQN

1 INTRODUCTION . In recent value-based deep reinforcement learning ( DRL ) , a value function is usually utilized to evaluate state values , which stand for estimates of the expected long-term cumulative rewards that might be collected by an agent . In order to perform such an evaluation , a deep neural network ( DNN ) is employed by a number of contemporary value-based DRL methods ( Mnih et al. , 2015 ; Wang et al. , 2016 ; Hasselt et al. , 2016 ; Osband et al. , 2016 ; Hessel et al. , 2018 ) as the value function approximator , in which the network parameters are iteratively updated based on the agent ’ s experience of interactions with an environment . For many of these methods ( Mnih et al. , 2015 ; Wang et al. , 2016 ; Hasselt et al. , 2016 ; Osband et al. , 2016 ; Hessel et al. , 2018 ) , the update procedure is carried out by one-step temporal-difference ( TD ) learning ( Sutton & Barto , 1998 ) ( or simply “ one-step TD ” ) , which calculates the error between an estimated state value and a target differing by one timestep . One-step TD has been demonstrated effective in backing up immediate reward signals collected by an agent . Nevertheless , the long temporal horizon that the reward signals from farther states have to propagate through might lead to an extended learning period of the value function approximator . Learning from multi-step returns ( Sutton & Barto , 1998 ) is a way of propagating rewards newly observed by the agent faster to earlier visited states , and has been adopted in several previous works . Asynchronous advantage actor-critic ( A3C ) ( Mnih et al. , 2016 ) employs multi-step returns as targets to update the value functions of its asynchronous threads . Rainbow deep Q-network ( Rainbow DQN ) ( Hessel et al. , 2018 ) also utilizes multi-step returns during the backup procedure . The authors in ( Barth-Maron et al. , 2018 ) also modify the target value function of deep deterministic dolicy gradient ( DDPG ) ( Lillicrap et al. , 2016 ) to estimate TD errors using multi-step returns . Updating value functions with different backup lengths provides advantages in different aspects , including bias and variance of value estimates , convergence speed , and exploration behavior of the agent . Backing up reward signals through multi-step returns shifts the bias-variance tradeoff ( Hessel et al. , 2018 ) . Therefore , backing up with different step return lengths ( or simply ‘ backup length ’ hereafter ( Asis et al. , 2018 ) ) might lead to different target values in the Bellman equation , resulting in different exploration behaviors of the agent as well as different achievable performance of it . The authors in ( Amiranashvili et al. , 2018 ) have demonstrated that the performance of the agent varies with different backup lengths , and showed that both very short and very long backup lengths could cause performance drops . These insights suggest that identifying the best backup length for an environment is not straightforward . In addition , although learning based on multi-step returns enhances the immediate sensitivity to future rewards , it is at the expense of greater variance which may cause the value function approximator to require more data samples to converge to the true expectation . Moreover , relying on a single target value with any specific backup length constrains the exploration behaviors of the agent , and might limit the achievable performance of it . Based on the above observations , there have been several research works proposed to unify different target values with different backup lengths to leverages their respective advantages . The traditional TD ( λ ) ( Sutton & Barto , 1998 ) uses a target value equivalent to an exponential average of all n-step returns ( where n is a natural number ) , providing a faster empirical convergence by interpolating between low-variance TD returns and low-bias Monte Carlo returns . DQN ( λ ) ( Daley & Amato , 2019 ) further proposes an efficient implementation of TD ( λ ) for DRL by modifying the replay buffer memory such that λ-returns can be pre-computed . Although these methods benefit from combining multiple distinct backup lengths , they still rely on a single target value during the update procedure . Integrating step returns into a single target value , nevertheless , may sacrifice the diversity of the advantages provided by different step return targets . As a result , in this paper , we propose Mixture Bootstrapped DQN ( abbreviated as “ MB-DQN ” ) to address the above issues . MB-DQN is built on top of bootstrapped DQN ( Osband et al. , 2016 ) , which contains multiple bootstrapped heads with randomly initialized weights to learn a set of value functions . MB-DQN leverages the advantages of different step return targets by assigning a distinct backup length to each bootstrapped head . Each bootstrapped head maintains its own target value derived from the assigned backup length during the update procedure . Since the backup lengths of the bootstrapped heads are distinct from each other , MB-DQN provides heterogeneity in the target values as well as diversified exploration behaviors of the agent that is unavailable in approaches relying only on a single target value . To validate the proposed concept , in our experiments , we first provide motivational insights on the influence of different configurations of backup lengths in a simple maze environment . We then evaluate the proposed MB-DQN on the Atari 2600 ( Bellemare et al. , 2015 ) benchmark environments , and demonstrate its performance improvement over a number of baseline methods . We further provide a set of ablation studies to analyze the impacts of different design configurations of MB-DQN . In summary , the primary contributions of this paper include : ( 1 ) introducing an approach for maintaining the advantages from different backup lengths , ( 2 ) providing heterogeneity in the target values by utilizing multiple bootstrapped heads , and ( 3 ) enabling diversified exploration behaviors of the agent . The remainder of this paper is organized as the following . Section 2 provides the background material related to this work . Section 3 walks through the proposed MB-DQN methodology . Section 4 reports the experimental results , and presents a set of the ablation analyses . Section 5 concludes this paper . 2 BACKGROUND . In this section , we provide the background material related to this work . We first introduce the basic concepts of the Markov Decision Process ( MDP ) and one-step return , followed by an explanation of the concept of multi-step returns . Next , we provide a brief overview of the Deep Q-Network ( DQN ) . 2.1 MARKOV DECISION PROCESS AND ONE-STEP RETURN . In RL , an agent interacting with an environment E with state space S and action space A is often formulated as an MDP . At each timestep t , the agent perceives a state st ∈ S , takes an action at ∈ A according to its policy π ( a|s ) , receives a reward rt ∼ R ( st , at ) , and transits to next state st+1 ∼ p ( st+1|st , at ) , where R ( st , at ) and p ( st+1|st , at ) are the reward function and transition probability function , respectively . The main objective of the agent is to learn an optimal policy π∗ ( a|s ) that maximizes discounted cumulative return Gt = ∑T i=t γ i−trt , where γ ∈ ( 0 , 1 ] is the discount factor and T is the horizon . For a given policy π ( a|s ) , the state value function V π and state-action value function Qπ are defined as the expected discounted cumulative return Gt starting from a state s and a state-action pair ( s , a ) respectively , and can be represented as the following : V π ( s ) = E [ Gt|st = s , π ] , Qπ ( s , a ) = E [ Gt|st = s , at = a , π ] . ( 1 ) In order to maximize E [ Gt ] , conventional value-based RL methods often use one-step TD learning to iteratively update V π and Qπ . Take Qπ for example , the update rule is expressed as the following : Q ( st , at ) ← Q ( st , at ) + α [ rt + γQ ( st+1 , at+1 ) −Q ( st , at ) ] , ( 2 ) where α ∈ ( 0 , 1 ] is a step size parameter which controls the update speed . This update procedure only considers the immediate return rt and γQ ( st+1 , at+1 ) , which is together called one-step return . 2.2 MULTI-STEP RETURN . Multi-step return is a variant of one-step return presented in the previous section . Multi-step return modifies the target of one-step return through bootstrapping over longer time intervals . It replaces the single reward rt in Eq . ( 2 ) with the truncated multi-step return Rnt , which is represented as follows : Rnt = n∑ j=0 γjrt+j , ( 3 ) where n is the selected backup length . The update rule of Eq . ( 2 ) is then re-written as the following : Q ( st , at ) ← Q ( st , at ) + α [ Rnt + γn+1Q ( st+n+1 , at+n+1 ) −Q ( st , at ) ] . ( 4 ) A longer backup length n has been shown to increase the variance of the estimated Q ( st , at ) as well as decrease its bias ( Jaakkola et al. , 1994 ) . Despite of the high-variance and the increased computational cost , multi-step return enhances the immediate sensitivity of the value approximator to future rewards , and allows them to backup faster . As a result , in certain cases , it is possible to achieve a faster learning speed for the value approximator by using an appropriate backup length n larger than one ( Sutton & Barto , 1998 ; Hessel et al. , 2018 ; Amiranashvili et al. , 2018 ) . 2.3 DEEP Q-NETWORK . DQN is a DNN parameterized by θ for approximating the optimal Q-function . DQN is trained using samples drawn from an experience replay buffer Z , and is updated based on one-step TD learning with an objective to minimize a loss function LDQN , which is typically expressed as the following : LDQN = Es , a , r , s′∼U ( Z ) [ ( ys , a −Q ( s , a , θ ) ) 2 ] , ( 5 ) where ys , a = rt+γmaxaQ ( st+1 , a , θ− ) is the one-step target value , U ( Z ) is a uniform distribution over Z , and θ− is the parameters of the target network . θ− is updated by θ at predefined intervals . 3 METHODOLOGY . In this section , we first demonstrate the impacts of different backup lengths on the behaviors of an agent in a simple maze environment . Then , we walk through the details of the MB-DQN framework . 3.1 AGENT BEHAVIOR WITH DIFFERENT BACKUP LENGTHS IN DQN . To illustrate the impacts of different backup lengths on an agent ’ s behavior , we first consider a toy model in a two-dimensional maze environment containing a starting point and a goal , as depicted in Fig . 1 ( a ) . We use DQN as our default agent and perform our experiments on this maze environment ( denoted as Dense Maze ) with dense rewards . In this setting , the reward of a grid gradually decreases as its distance to the goal increases . We depict the states visited by the agents for 100k timesteps in the training phase in Figs . 1 ( b ) and ( c ) . It is observed that the agent trained with 5-step return reaches the goal through a shorter path than that of the 1-step return case . This is because the longer backup length allows the agent to adjust its value function estimation faster . On the other hand , although the agent trained with 1-step return might converge slower than that of the 5-step return case , it is observed that 1-step return enables the agent to visit and explore more states in the early stage . This is because the reward signal from a farther state has to propagate through a longer temporal horizon . Therefore , the agent explores more extensively before learning an effective policy to reach the goal .

Summary: This paper combines the idea of multi-step returns for value-based reinforcement learning with Bootstrapped DQN. Updating value functions with different step returns has potential benefits, and was elegantly integrated with the Bootstrapped DQN structure, where different heads have different backup lengths. This work provides empirical results on Atari game domains, and further provides ablation analysis.