Temporal Difference Weighted Ensemble For Reinforcement Learning

1 INTRODUCTION . Using ensemble methods that combine multiple function approximators can often achieve better performance than a single function by reducing the variance of estimation ( Dietterich ( 2000 ) ; Kuncheva ( 2014 ) ) . Ensemble methods are effective in supervised learning , and also reinforcement learning ( Wiering & Van Hasselt ( 2008 ) ) . There are two situations where multiple function approximators are combined : combining and learning multiple functions during training ( Freund & Schapire ( 1997 ) ) and combining individually trained functions to jointly decide actions during testing ( Breiman ( 1996 ) ) . In this paper , we focus on the second setting of reinforcement learning wherein each function is trained individually and then combined them to achieve better test performance . Though there is a body of research on ensemble algorithms in reinforcement learning , it is not as sizeable as the research devoted to ensemble methods for supervised learning . Wiering & Van Hasselt ( 2008 ) investigated many ensemble approaches combining several agents with different valuebased algorithms in Gridworld settings . Faußer & Schwenker ( 2011 ; 2015a ) have shown that combining value functions approximated by neural networks improves performance greater than using a single agent . Although previous work dealt with each agent equally contributing to the final output , weighting each contribution based on its accuracy is also a known and accepted approach in supervised learning ( Dietterich ( 2000 ) ) . However , unlike supervised learning , reinforcement learning agents learn from trajectories resulting from exploration , such that each agent learns from slightly different data . This characteristic is significant in tasks with high-dimensional state-space , where there are several possible optimal trajectories to maximize cumulative rewards . In such a situation , the final joint policy function resulting from simple averaging or majority voting is not always optimal if each agent learned different optimal trajectories . Furthermore , it is difficult to decide constant weights of each contribution as it is possible that agents with poor episode rewards have better performance in specific areas . In this paper , we propose the temporal difference weighted ( TDW ) algorithm , an ensemble method for reinforcement learning at test time . The most important point of this algorithm is that confident agents are prioritized to participate in action selection while contributions of agents unfamiliar with the current trajectory are reduced . To do so in the TDW algorithm , the weights of the contributions at each Q-function are calculated as softmax probabilities based on accumulated TD errors . Extending an averaging method and a majority voting method , actions are determined by weighted average or voting methods according to the weights . The advantage of the TDW algorithm is that arbitrary training algorithms can use this algorithm without any modifications , because the TDW algorithm only cares about the joint decision problem , which could be easily adopted in competitions and development works using reinforcement learning . In our experiment , we demonstrate that the TDW retains performance in tabular representation Gridworld tasks with multiple possible trajectories , where simple ensemble methods are significantly degraded . Second , to demonstrate the effectiveness of our TDW algorithm in high-dimensional state-space , we also show that our TDW algorithm can achieve better performance than baseline algorithms in Atari tasks ( Bellemare et al . ( 2013 ) ) . 2 RELATED WORK . Ensemble methods that combine multiple function approximators during training rather than evaluation have been studied in deep reinforcement learning . Bootstrapped deep Q-network ( DQN ) ( Osband et al . ( 2016 ) ) leverages multiple heads that are randomly initialized to improve exploration , because each head leads to slightly different states . Averaged-DQN ( Anschel et al . ( 2017 ) ) reduces the variance of a target approximation by calculating an average value of last several learned Qnetworks . Using multiple value functions to reduce variance of target estimation is also utilized in the policy gradients methods ( Fujimoto et al . ( 2018 ) ; Haarnoja et al . ( 2018 ) ) . In contrast , there has been research focused on joint decision making in reinforcement learning . Using multiple agents to jointly select an action achieves better performance than a single agent ( Wiering & Van Hasselt ( 2008 ) ; Faußer & Schwenker ( 2015a ; 2011 ) ) . However , such joint decision making has been limited to relatively small tasks such as Gridworld and Maze . Therefore , it is not known whether joint decision making with deep neural networks can improve performance in high-dimensional state-space tasks such as Atari 2600 ( Bellemare et al . ( 2013 ) ) . Doya et al . ( 2002 ) proposes a multiple mode-based reinforcement learning ( MMRL ) , a weighted ensemble method for model-based reinforcement learning , which determines each weight based by using prediction models . The MMRL gives larger weights to reinforcement learning controllers with small errors of special responsibility predictors . Unlike MMRL , our method does not require additional components to calculate weights . Our method is not the first one to use TD errors in combining multiple agents . Ring & Schaul ( 2011 ) proposes a module selection mechanism that chooses the module with smallest TD errors to learn current states , which will eventually assign each module to a small area of a large task . As a joint decision making method , a selective ensemble method is proposed to eliminate agents with less confidence at the current state by measuring TD errors ( Faußer & Schwenker ( 2015b ) ) , which is the closest approach to our method . This selection drops all outputs whose TD errors exceeds a threshold , which can be viewed as a hard version of our method that uses a softmax of all weighted outputs instead of elimination . The threshold is not intuitively determined . Because the range of TD errors varies by tasks and reward settings , setting the threshold requires sensitive tuning . 3 BACKGROUND . 3.1 REINFORCEMENT LEARNING . We formulate standard reinforcement learning setting as follows . At time t , an agent receives a state st ∈ S , and takes an action at ∈ A based on a policy function at = π ( st ) . The next state st+1 is given to the agent along with a reward rt+1 . The return is defined as a discounted cumulative reward Rt = ∑T i=t γ i−tr ( si , ai ) , where γ ∈ [ 1 , 0 ] is a discount factor . The true value of taking an action at at a state st is described as follows : Qπ ( st , at ) = E [ Rt|st , at ] where Qπ ( st , at ) is an action-value under the policy π . The optimal value is Q∗ ( st , at ) = maxπ Qπ ( st , at ) . With such an optimal Q-function , optimal actions can be determined based on the highest action-values at each state . DQN ( Mnih et al . ( 2015 ) ) is a deep reinforcement learning method that approximates an optimal Qfunction with deep neural networks . The Q-function Q ( st , at|θ ) with a parameter θ is approximated by a Q-learning style update ( Watkins & Dayan ( 1992 ) ) . The parameter θ is learned to minimize squared temporal difference errors . L ( θ ) = Est , at , rt+1 , st+1 [ ( Q ( st , at|θ ) − yt ) 2 ] ( 1 ) where yt = rt+1 + γmaxaQ ( st+1 , a|θ′ ) with a target network parameter θ′ . The target network parameter θ′ is synchronized to the parameter θ in a certain interval . DQN also introduces use of the experience replay ( Lin ( 1992 ) ) , which randomly samples past state transitions from the replay buffer to compute the squared TD error ( 1 ) . 3.2 ENSEMBLE METHODS . Assume there are N sets of trained Q-function Q ( s , a|θi ) where i denotes an index of the function . The final policy π ( st ) is determined by combining the N Q-functions . We formulate two baseline methods commonly used in ensemble algorithms : Average policy and Majority Voting ( MV ) policy ( Faußer & Schwenker ( 2011 ; 2015a ) ; Kuncheva ( 2014 ) ) . Majority Voting ( MV ) policy is an approach to decide the action based on greedy selection according to the formula : π ( st ) = argmax a N∑ i vi ( st , a ) ( 2 ) where vi ( s , a ) is a binary function that outputs 1 for the most valued action and 0 for others : vi ( s , a ) = { 1 ( a = argmaxa′ Q ( s , a ′|θi ) ) 0 ( otherwise ) ( 3 ) Contributions of each function to the final output are completely equal . Average policy is a method that averages all the outputs of the Q-functions , and the action is greedily determined : π ( st ) = argmax a 1 N N∑ i Q ( st , a|θi ) ( 4 ) Averaging outputs from multiple approximated functions reduces variance of prediction . Unlike MV policy , Average policy leverages all estimated values as well as the highest values . 4 TEMPORAL DIFFERENCE WEIGHTED ENSEMBLE . In this section , we explain the TDW ensemble algorithm that adjusts weights of contributions based on accumulated TD errors . The TDW algorithm is especially powerful in the complex situation such as high-dimensional state-space where it is difficult to cover whole state-space with a single agent . Section 4.1 describes the error accumulation mechanism . Section 4.2 introduces joint action selection using the weights computed with the accumulated errors . 4.1 ACCUMULATING TD ERRORS . We consider that a squared TD error δit = ( Q ( st , at|θi ) − rt+1 − γmaxaQ ( st+1 , a|θi ) ) 2 fundamentally consists of two kinds of errors : δit = δ i , p t + δ i , u t ( 5 ) where δp is a prediction error of approximated function , and δu is an error at states where the agent rarely experienced . In a tabular-based value function , δp will be near 0 at frequently visited states . In contrast , δu will be extremely large at less visited states with both a tabular-based value function and a function approximator because TD errors are not sufficiently propagated such a state . There are two causes of unfamiliar states : ( 1 ) states are difficult to visit due to hard exploration , and ( 2 ) states are not optimal to the agent according to learned state transitions . For combining multiple agents at a joint decision , the second case is noteworthy because each agent may be optimized at different optimal trajectories . Thus , some of the agents will produce larger δu when they face such states as a result of an ensemble , and contributions of less confident agents can be reduced based on the TD error δu . Algorithm 1 Temporal Difference Weighted Ensemble Require : α : a constant decay factor Require : θ1 ... θN : trained parameters Initialize u10 = 0 , ... , u N 0 = 0. for t = 1 , 2 , . . . , T do Receive state st and reward rt . for i = 1 , 2 , . . . , N do Calculate uit via ( 6 ) . end for for i = 1 , 2 , . . . , N do Calculate wit via ( 7 ) . end for Select action at via ( 8 ) or ( 9 ) . end for To measure uncertainty of less confident agents , we define uit as a uncertainty of an agent : uit = δ i t−1 + αu i t−1 ( 6 ) where α ∈ [ 0 , 1 ] is a constant factor decaying the uncertainty at a previous step . With a large α , the uncertainty ui is extremely large during unfamiliar trajectories , which makes it possible to easily distinguish confident agents from the others . However , a trade-off arises when prediction error δp is accumulated for a long horizon , which increases correlation between agents .

Summary: 1.	The paper suggests an ensemble method that leads to better performance over the execution of a single agent. The final (behavior) agent is a weighted average (or majority vote) of the members, where weights are determined by the accumulated TD error of the agent in the episode so far. The TD error serves as a confidence measure of the ensemble members in their predictions.