Temporal Difference Weighted Ensemble For Reinforcement Learning
Authors: Takuma Seno
Michita Imai
Venue:  (2020)
Keywords: reinforcement learning
ensemble
deep q-network

Abstract: Combining multiple function approximators in machine learning models typically leads to better performance and robustness compared with a single function. In reinforcement learning, ensemble algorithms such as an averaging method and a majority voting method are not always optimal, because each function can learn fundamentally different optimal trajectories from exploration. In this paper, we propose a Temporal Difference Weighted (TDW) algorithm, an ensemble method that adjusts weights of each contribution based on accumulated temporal difference errors. The advantage of this algorithm is that it improves ensemble performance by reducing weights of Q-functions unfamiliar with current trajectories. We provide experimental results for Gridworld tasks and Atari tasks that show significant performance improvements compared with baseline algorithms.

Under review as a conference paper at ICLR 2020
TEMPORAL DIFFERENCE WEIGHTED ENSEMBLE FOR REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Combining multiple function approximators in machine learning models typically leads to better performance and robustness compared with single function. reinforcement learning, ensemble algorithms such an averaging method and a majority voting method are not always optimal, because each function can learn fundamentally different optimal trajectories from exploration. In this paper, we propose a Temporal Difference Weighted (TDW) algorithm, an ensemble method that adjusts weights of each contribution based on accumulated temporal differ- ence errors. The advantage of this algorithm is that it improves ensemble per- formance by reducing weights of Q-functions unfamiliar with current trajectories. We provide experimental results for Gridworld tasks and Atari tasks that show significant performance improvements compared with baseline algorithms.
INTRODUCTION
Using ensemble methods that combine multiple function approximators often achieve bet- ter performance than a single function by reducing the variance of estimation (Dietterich (2000); Kuncheva (2014)). Ensemble methods are effective in supervised learning, and also reinforce- ment learning (Wiering & Van Hasselt (2008). There are two situations where multiple function approximators are combined: combining and learning multiple functions during training (Freund & Schapire (1997)) and combining individually trained functions to jointly decide actions during testing (Breiman (1996)). this paper, we focus on the second setting of reinforcement learning wherein each function is trained individually and then combined them to achieve better test perfor- mance.
Though there is body of research on ensemble algorithms in reinforcement learning. it is not as sizeable as the research devoted to ensemble methods for supervised learning. WWierin Van Has- seli (2008) investigated many ensemble approaches combining several agents with different value- based algorithms in Gridworld settings. Faußeer & Schwenker [2011] 2015a) have shown that com- bining value functions approximated by neural networks improves performance greater than using a single agent. Although previous work dealt with each agent equally contributing to the final out- put, weighting each contribution based on its accuracy is also known and accepted approach in supervised learning (Dictterich) (2000)).
However, unlike supervised learning, reinforcement learning agents learn from trajectories result- ing from exploration, such that each agent learns from slightly different data. This characteristic significant in tasks with high-dimensional state-space, where there are several possible optimal trajectories to maximize cumulative rewards. In such situation, the final joint policy function re- sulting from simple averaging or majority voting is not always optimal if each agent learned different optimal trajectories. Furthermore, it is difficult to decide constant weights of each contribution as it possible that agents with poor episode rewards have better performance specific areas.
In this paper, we propose the temporal difference weighted (TDW) algorithm, an ensemble method for reinforcement learning at test time. The most important point of this algorithm is that confident agents are prioritized to participate in action selection while contributions of agents unfamiliar with the current trajectory are reduced. To do so in the TDW algorithm, the weights of the contributions at each Q Q-function are calculated as softmax probabilities based accumulated TD errors. Extend- ing an averaging method and a majority voting method, actions are determined by weighted average voting methods according to the weights. The advantage of the TDW algorithm is that arbitrary

---

Under review as a conference paper at ICLR 2020
training algorithms use this algorithm without any modifications, because the TDW algorithm only cares about the joint decision problem, which could be easily adopted in competitions and de- velopment works using reinforcement learning. In experiment, we demonstrate that the TDW retains performance in tabular representation Gridworld tasks with multiple possible trajectories, where simple ensemble methods are significantly degraded. Second, demonstrate the effective- ness of our TDW algorithm in high-dimensional state-space, we also show that our TDW algorithm can achieve better performance than baseline algorithms in Atari tasks (Bellemare et al. (2013))
RELATED WORK
Ensemble methods that combine multiple function approximators during training rather than eval- uation have been studied in deep reinforcement learning. Bootstrapped deep Q-network (DQN) (Osband et al. (2016) leverages multiple heads that are randomly initialized to improve exploration, because each head leads to slightly different states. Averaged-DQN (Anschel et al. (2017) reduces the variance of a target approximation by calculating an average value of last several learned Q- networks. Using multiple value functions reduce variance of target estimation is also utilized in the policy gradients methods (Fujimoto et al. (2018): Haarnoja et al. (018)).
In contrast, there has been research focused joint decision making in reinforcement learning. Using multiple agents to jointly select an action achieves better performance than a single agent (Wiering & Van Hasselt (2008); Faußerr & Schwenker (2015a 2011)). However, such joint decision making has been limited to relatively small tasks such as Gridworld and Maze. Therefore, it is not known whether joint decision making with deep neural networks can improve performance in high-dimensional state-space tasks such as Atari 2600 (Bellemare et al. (2013)).
Doya al. (2002) proposes a multiple mode-based reinforcement learning (MMRL), weighted ensemble method for model-based reinforcement learning, which determines each weight based by using prediction models. The MMRL gives larger weights to reinforcement learning controllers with small errors of special responsibility predictors. Unlike MMRL, our method does not require additional components to calculate weights.
Our method is not the first one to use TD errors in combining multiple agents. Ring & Schaul (201T) proposes a module selection mechanism that chooses the module with smallest TD errors to learn current states, which will eventually assign each module to small area of a large task. As a joint decision making method, a selective ensemble method proposed to eliminate agents with less confidence at the current state by measuring TD errors (Faußer & Schwenker (2015))). which is the closest approach to our method. This selection drops all outputs whose TD errors exceeds a threshold, which can be viewed outputs instead of elimination. The threshold is not intuitively determined. Because the range of TT) error varies by tasks and reward settings, setting the threshold requires sensitive tuning.
hard version of method that uses softmax of all weighted
BACKGROUND
3.1 REINFORCEMENT LEARNING
We formulate standard reinforcement learning setting as follows. At time t, an agent receives a state
8, E S, and takes an action a, E A based on a policy function o; = T (8e). The next state 8:41 is given the agent along with reward r1+1- The return is defined as discounted cumulative reward R2 Σ. x (rs, a;), where 7 e [1, O] discount factor. The true value of taking an action
a, at a state is described as follows:
(st, 0x) = EE [e or]
where Qx(s, at) is an action-value under the policy л. The optimal value (+,, az) = max, (st at). With such an optimal Q-function, optimal actions can be determined based on the highest action-values at each state.
DQN (Mnih et al. (0155) is deep reinforcement learning method that approximates an optimal Q- function with deep neural networks. The Q-function ((s,, a, ()) with parameter is approximated by Q-learning style update (Watkins & Dayan (1992)). The parameter 0 is learned to minimize
2

---

Under review as a conference paper at ICLR 2020
squared temporal difference errors.
( Es, me ar)0) -3a)
(I)
where 1/ T1411 1 + 7 max, , aa0)) with target network parameter 0'. The target network parameter 0' is synchronized to the parameter 0 in certain interval. DQN also introduces use of the experience replay (Lin) (1992)), which randomly samples past state transitions from the replay buffer to compute the squared TD error
3.2 ENSEMBLE METHODS
Assume there are N sets of trained Q-function ((s, a/0;) where denotes an index of the function. The final policy π (s;) is determined by combining the N Q-functions. We formulate two baseline methods commonly used in ensemble algorithms: Average policy and Majority Voting (MV) policy Faulier Schwenker 1001 2015a); Kuncheva (2014)..
Majority Voting (MV) policy is an approach to decide the action based on greedy selection according the formula:
N
π (8))) = arg max Σ (s a)
(2)
where 02(8,a) a) is a binary function that outputs 1 for the most valued action and O for others:
(8 a) - { 1
(a arg max, a(s,))
(3)
(atherwise)
Contributions of each function to the final output are completely equal.
Average policy is a method that averages all the outputs of the Q-functions, and the action is greedily determined:
N
π (st) = arg max
N Σ ((s,
(4)
Averaging outputs from multiple approximated functions reduces variance of prediction. Unlike MV policy, Average policy leverages all estimated values well as the highest values.
4 TEMPORAL DIFFERENCE WEIGHTED ENSEMBLE
In this section, we explain the TDW ensemble algorithm that adjusts weights of contributions based on accumulated TD errors. The TDW algorithm especially powerful the complex situation such high-dimensiona state-space where it is difficult to cover whole state-space with a single agent. Section 4. describes the eтror accumulation mechanism. Section 4.2 introduces joint action selection using the weights computed with the accumulated errors.
4.1 ACCUMULATING TD ERRORS
We consider that a squared TD eπror 5 = (Q(st, ar(0;) - r141 max, Q(5441, alO,))² funda- mentally consists of two kinds of errors:
8 = 5 + 5:."
(5)
where 8" is prediction error of approximated function, and 8" is an error at states where the agent rarely experienced. In a tabular-based value function, 8 will be near 0 at frequently visited states. In contrast, 8 will be extremely large less visited states with both tabular-based value function and function approximator because TD errors are not sufficiently propagated such state. There are two causes of unfamiliar states: (1) states are difficult to visit due to hard exploration, and (2) states are not optimal to the agent according to learned state transitions. For combining multiple agents at a joint decision, the second case is noteworthy because each agent may be optimized at different optimal trajectories. Thus, some of the agents will produce larger 8 when they face such states as a result of an ensemble, and contributions less confident agents can be reduced based on the TD error 8".
3

---

Under review as a conference paper at ICLR 2020
Algorithm Temporal Difference Weighted Ensemble
Require: a: constant decay factor
Require: 0x..0xx: trained parameters
Initialize u/ .... u6 = O.
for 1, ........2 T do
Receive state 8, and reward Tz.
for i 1, ...... NN do
Calculate uj via (6). end for for i- 1, ........NN.................. . do
Calculate w via (7)
end for
Select action a, via (8) (9)
end for
To measure uncertainty of less confident agents, we define uf as uncertainty an agent:
(6) where a E [0, 1] is constant factor decaying the uncertainty at previous step. With a large a, the uncertainty u² is extremely large during unfamiliar trajectories, which makes it possible easily distinguish confident agents from the others. However, trade-off arises when prediction error 6p is accumulated for a long horizon, which increases correlation between agents.
= 0 +au
4.2 ACTION SELECTION
To reduce contributions of less confident agents, each contribution at joint decision is weighted based on uncertainty u,. Using the uncertainty u,, weight wi of each agent is calculated as a probability by the softmax function:
e
w/
(7)
Eje
When the agent has a small uncertainty value ut. the weight wi becomes large.
We consider weighted ensemble methods corresponding the Average policy and the MV policy based on the weights wf. As a counterpart the Average policy, our TDW Average policy is follows:
N
π ((:))
= arg max
Σ a(,,),,(0,))
(8)
For the MV policy, TDW Voting policy is as follows:
(st) arg max Σ (s
Unlike the averaging method, because TDW Voting policy directly uses probabilities calculated by 7 the correlation between agents can be increased significantly with large decay factor a, leading to worse performance. Although these weighted ensemble algorithms are simple enough to extend arbitrary ensemble methods, leave more advanced applications for future work that may demonstrate the effectiveness of our approach in a simpler setting. The complete TDW ensemble algorithm is described in Algorithm T
EXPERIMENTS
In this section, describe the experiments performed on the Gridworld tasks and Atari tasks (Belle- mare et al. (2033) in Section [.2] To build the trained O-functions, we used the table-based Q- learning algorithm Watkkins & Dayan (9922) and DQN (Mnih et al. (2015)) with standard model, respectively. In each experiment, we evaluated our algorithm to address performance improvements from there baselines well the effects of selecting the decay factor a.
4

---

Under review as a conference paper at ICLR 2020
G
(a) Two-slit Gridworld
(b) Four-slit Gridworld
Figure 1: Gridworld environments. denotes initial states. denotes goals with +100 reward.
Gray areas represents walls.
Table I: Results for Gridworld tasks. The algorithms of best mean episode rewards are highlighted bold font. The numbers in brackets show ranks based on significant difference computed with Welch's t-test (p 0.05). The results of Single (mean) are the episode reward averaged over all single Q-functions.
Method
Two-slit Gridworld
Four-slit Gridworld
Single (mean)
97.43
98.07
Average
77.72 (6)
-9.57 (6)
TDW Average (a 0.0)
89.04 (5)
73.77 (5)
TDW Average (a = 0.2)
96.90 (2-4)
85.73 (1-4)
TDW Average (a 0.4)
96.94 (2-4)
85.98 (1-4)
TDW Average (a 0.6)
96.94 (2-4)
86.22 (1-4)
TDW Average (a = 0.8)
97.00 (1)
86.14 (1-4)
Majority Voting
96.49 (6)
36.51 (6)
TDW Voting (a 0.0)
97.43 (1-5)
94.32 (1-4)
TDW Voting (a 0.2)
97.44 (1-5)
94,57 (1-4)
TDW Voting (a 0.4)
97.43 (1-5)
94.45 (1-4)
TDW Voting (a 0.6)
97.43 (1-5)
94,42 (1-4)
TDW Voting (a 0.8)
97.43 (1-5)
94.15 (5)
5.1 GRIDWORLD
5.1.1 EXPERIMENTAL SETUP
We first evaluated the TDW algorithms with tabular representation scenario show their effec- tiveness in the situation where is difficult to cover whole state-space with the single agent. We built two Gridworld environments as shown in Figure [ Each environment is designed bias of learned trajectories by setting multiple slits. As result of exploration, once an agent gets through one of the slits to the goal, the agent easily biased to aim for the same slit due the max operator of Q-learning.
induce
discrete index table with size of 13 x 13. There are four actions corresponding to steps of up, down, left and right. If a wall exists where the agent tries to move, the next state remains the same as the current state. The agent always starts from S depicted in Figure I At every timestep, the agent receives reward of -0.1 or +100 at goal states. The agent starts new episode if either the agent arrives at the goal states or the timestep reaches 100 steps.
The reteesttprrsnnntioonnn
We trained N 10 agents with different random seeds for e-greedy exploration with Each training continues until 1M steps have been simulated. We set the learning rate to 0.01 and y 0.95. After training, evaluated TDW ensemble algorithms for 20K episodes. As baselines, we also evaluate each single agent as well as ensemble methods of Average policy and MV policy for 20K episodes each.
0.3.
5

---

Under review as a conference paper at ICLR 2020
Table 2: Evaluation results for the Atari tasks. The results of algorithms with the best mean episode rewards are highlighted in boldface. The numbers in brackets show ranks based on significant dif- ference computed with Welch's t-esst (p 0.05). The results of Single (best) are the mean episode rewards of the best Q-functions.
Mathod
Asterix
Beamrider
Breakour
Enduro
MsPacman
Spacefnvaders
Single (mean))
3465.74
6616.89
830.07
Single (best)
4022.90
8046.62
366.18
817.73
1776.74
1103.93
Average
1835.13(1-3)
11695.67 (5-6)
404.21 (3-4)
1400
1734, 89 (3-4)
989. (5)
Weighted Average
3950.10 (7)
8126.58 (7)
364.22 (6)
793.06 (7)
1852.79 (1)
1092.32 (1-2)
TDW Average (or 0.0)
6691.90 (4)
10128.44 (5-6)
418.05 (2)
50
1815.54 (2)
1076.21 (3-4)
TDW Average (a 0.2)
6911.10 (1-2)
12878.37 (1-2)
403 43 (3-4)
1445.44 (1)
1711.20 (3-4)
1030.01 (3-4)
1DW Average (ax = 0.4)
64009.20 (5-6)
11516.25 (3-5)
362.21 (7)
1330.12 (5-6)
1698.05 (4-5)
986.95 (6)
TDW Average (cr
6907.55 (2-3)
11386.13 (3-4)
380.12
1415.18
1685.76 (6)
1124.90 (1-2)
TDW Average (ar = 0.8)
6472.22 (5-6)
12661.50 (1-2)
420.14 (1)
1316,30 (5-6)
1668.54 (7)
902.91 (7)
Majority Voting
4586.85 (6)
9945.25 (6)
344.30 (4)
1204,10
1542.78 (4)
889,38 (7)
Weighted Voting
4087.15 (7)
8064.10 (7)
364.91 (3)
799.96 (7)
1767.05 (1)
1097,84 (1)
TDW Voting
0.0)
(1-2)
11322.70 (1-2)
1149.08
(3)
(2)
TDW Voting (a 0.2)
5565.75 (5)
10982.07 (1-2)
370.18 (2)
1208.00 (1)
1457.43 (5)
928.29 (6)
TDW Voting (a 0.4)
6210.80 (2-3)
10790.61 (3)
298.14 (7)
1163.06 (3-4)
1693.65 (2)
1014,99 (3)
TDW Voting 0.6)
3972.80 (4)
10426.07 (4)
336.81
1172.65 (3-4)
1392.62 (7)
968.90 (4)
TDW Voting (ax 0.8)
6303.35 (1-3)
10352,76 (5)
410.29 (1)
1141.04 (6)
1410.55 (6)
934.24 (5)
5.1.2 RESULTS
The evaluation results the Gridworld environments are shown in Table Four-slit Gridworld is significantly more difficult than Two-slit Gridworld because each Q-function is not only horizontally biased, but also vertically biased. In both the Two-slit Gridworld and Four-slit Gridworld environ- ments, the TDW ensemble methods achieve better performance than their corresponding Average policy and the MV policy baselines. Additionally, the results of both of the Average policy and the MV policy were worse than the single models. It should be noted that Average policy degrades original performance more than MV policy.
For the selection of the decay factor a, a larger a tends to increase performance in TDW Average policy. In contrast, the larger a leads poor performance in TDW Voting policy especially Four- slit Gridworld. We believe that the large significantly reduces contributions of most Q-functions. which would ignore votes of actions that would be the best in equal voting. In contrast, TDW Average policy leverages values of all actions, exploiting all contributions select the best action.
5.2
ATART
5.2.1
EXPERIMENTAL SETUP
To demonstrate effectiveness in high-dimensional state-space, we evaluated TDW algorithm in Atari tasks. We trained DQN agents across 6 Atari tasks (Asterix, Beamrider, Breakout, Enduro, MsPac- man and SpaceInvaders) through OpenAl Gym (Brockman et al.] (2016)). At each task, N = 10 agents were trained with different random seeds for neural network initialization, exploration and environments in order to vary the learned Q-function. The training continued until 10M steps (40M game frames) with frame skipping and termination on loss of life enabled. The of exploration is linearly decayed from 1.0 to 0.1 through 1M steps. The hyperparameters of neural networks are same as (Mnih al. (2015)).
After training, evaluation was conducted with each Q-function, TDW Average policy, TDW Voting policy and the two baselines. We additionally evaluated weighted versions of the baselines whose Q-functions were weighted based on their evaluation performance. The evaluation continued for 1000 episodes with 0.05.
5.2.2 RESULTS
The experimental results are shown in Table Interestingly, both of Average policy and MV policy improved performance from mean performance of single agents, though the simple ensemble algo- rithms had not been investigated well in the domain of deep reinforcement learning. In the games of Asterix, Beamrider, Breakout and Enduro, the TDW algorithms achieve additional performance

---

Under review as a conference paper at ICLR 2020
A
B
A
-15
B
1
1590 135 2009
(a) Entropy
(b) Sample frames and weights
Figure 2: (a) Plots of entropies during a sample episode in Breakout. The r-axis represents episode steps. The y-axis represents negative entropies of the weights. (b) Bar graphs next to the game frames show the corresponding weights with the Q-function index on the r-axis. White arrows on the game frames show the ball positions and past trajectory.
6.3
63
14
(a) Asterix
(b) BeamRider
(c) Breakout
30
(d) Enduro
(e) MsPacman
(1) SpaceInvaders
Figure 3: Plots of the number of observations with certain entropy through 100K evaluation steps (TDW Average policy). x-axis represents entropy values with 20 bins. y-axis represents correspond- ing numbers in log scale.
improvements as compared with the non-weighted and weighted baselines. Even in MsPacman and SpaceInvaders, the TDW algorithms perform significantly better than non-weighted baselines and the best single models. In most of the cases, the globally weighted ensemble baselines performed worse than non-weighted versions. We believe this because these globally weighted ensemble methods will ignore local performance, which is significant in high-dimensional state-space because it is difficult to cover all possible states with single models. The TDW algorithms with small a tend to achieve better performance than those with large a, which suggests that significantly large can increase correlation between Q-fucctions and reduce contributions of less confident Q-functions.
To analyze changes of weights through episode, plot entropies during sample episode Breakout (TDW Average policy, a = 0.8) in Figure [(a] If an entropy is low (high in negative scale), some of Q-functions have large weights, while others have extremely small weights. Extreme

---

Under review as a conference paper at ICLR 2020
low entropies are observed when the ball comes closely to the pad shown in Figure 2(b) where the value should be close to for the non-optimal actions because missing the ball immediately ends its life. is easy for sufficiently learned Q Q-functions to estimate such terminal state that the entropy becomes low due to the gap between optimal Q-functions for the current states and the others. The entropies tend to be low during latter steps there many variations of the remaining blocks. We consider that the reason why the TDW algorithms with = 0.8 achieved the best performance in Breakout is that the large value reduces influence of the Q-functions which cannot correctly predict values with unseen variations of remaining blocks. In contrast to Breakout where living long leads to higher scores, in Spacelnvaders we observe that the low entropies appear at dodging beams rather than shooting invaders, because shooting beams requires long-term value prediction which does not induce large TD errors. Therefore, performance improvements on SpaceInvaders are not significantly better than weighted baselines.
To analyze correlation between the decay factor a and entropies, plots of the number of observations with a certain entropy are shown in Figure 3 In most games, higher decay factors increase the presence of low entropy states and decreases the presence of high entropy states. In the games with frequent reward occurences such as Enduro and MsPacman, there are more low-entropy observations than BeamRider and SpaceInvaders, where reward occurences are less frequent. Especially with regards MsPacman, believe that the TDW Average policy with larger values results in worse performance because the agent frequently receives positive rewards at almost every timestep, which often induces prediction error 8", and increases uncertainty in all Q-functions. Thus, globally weighted ensemble methods achieve better performance than TDW algorithms because it is difficult consistently accumulate uncertainties on MsPacman.
CONCLUSION
In this paper, we have introduced the TDW algorithm: an ensemble method that accumulates tempo- ral difference errors uncertainties in order to adjust weights of each Q-function, improving per- formance especially high-dimensional state-space or situations where there are multiple optimal trajectories. We have shown performance evaluations in Gridworld tasks and Atari tasks, wherein the TDW algorithms have achieved significantly better performance than non- weighte algorithms and globally weighted algorithms. However, it is difficult to correctly measure uncertainties with frequent reward occurrences because the intrinsic prediction errors are also accumulated. Thus, these types of games did not realize the same performance improvements.
In future work, we intend to investigate an extension of this work into continuous action-space tasks because only the joint decision problem of Q-functions is considered in this paper. We believe similar algorithm can extend a conventional ensemble method (Huang et al. (2017)) of Deep De- terministic Policy Gradients (Lillicrap al.) (2015) by measuring uncertainties of pairs of a policy function and a Q-function. We will also consider separate path, developing an algorithm that mea- sures uncertainties without rewards because reward information is not always available especially in the case of real world application.
REFERENCES
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqu: Variance reduction and stabilization for deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning Learning-Volume Volume 70, pp. 176- 185. JMLR. org, 2017.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-14)) 1996.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, arXiv preprint 1:X.v::606.0/5400 2016.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multi- ple classifier systems, pp. 1-15. Springer, 2000.
8

---

Under review as a conference paper at ICLR 2020
Kenji Doya, Kazuyuki Samejima, Ken-ichi Katagiri, and Mitsuo Kawato. Multiple model-based reinforcement learning. Neural computation, 14(6):1347-139, 2002.
Stefan FauBer and Friedhelm Schwenker. Ensemble methods for reinforcement learning with function approximation. In International Workshop on Multiple Classifier Systems, pp. 56-65. Springer, 2011.
Stefan Faußer and Friedhelm Schwenker. Neural network ensembles in reinforcement learning. Neural Processing Letters, 41(1):55-69 69, 2015a.
Stefan FauBer and Friedhelm Schwenker. Selective neural network ensembles in reinforcement learning: taking the advantage of many agents. Neurocomputiigg, 169:350-357, 2015b.
Yoav Freund and Robert E Schapire. A deeision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences, 55(1):119-135 1997.
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 1802.88477777 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv:1801.01299. 2018.
Zhewei Huang, Shuchang Zhou, BoEr Zhuang, and Xinyu Zhou. Learning to run with actor-critic ensemble. arXiv preprint arXiv:1712.08985 2017.
Ludmila Kuncheva. Combining pattern classifiers: methods and algorithms. John Wiley & Sons, 2014.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 1509.//500.0225 2015.
Long-Ji Lin. Self-improving reactive agents based reinforcement learning, planning and teaching. Machine learning, 8(3-4):293-321, 1992.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529, 2015.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp. 4026 4034, 2016.
Mark B Ring and Tom Schaul. Q-error as a selection mechanism in modular veinforreeeen--eariiigg systems. In Swenty-Second International Joint Conference on Artificial Intelligence, 2011.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279-22)) 1992.
Marco A Wiering and Hado Van Hasselt. Ensemble algorithms reinforcement learning. JEEE Transactions Systems, Man, and Cybernetics, Part B (Cybernetics), 38(4):930-99) 93699900-9999999 2008

---

Under review as a conference paper at ICLR 2020
A Q-FUNCTION TABLES OBTAINED ON GRIDWORLDS
Figure 4: Learned table- based Q values at Two-slit Grid World. Each cell corresponds to state and its color represents value of max, Q(s, a).
Figure 5: Learned table-based Q values at Four-slit Grid World. Each cell corresponds to state 8, and its color represents value of max, Q(8, a).
10


Reviews:
[{"review_id": "Ske-uTrptr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of combining separately learned action-value functions into a single, no-longer-learning, action selection algorithm. The result is not necessarily a combined policy, strictly speaking, because it need not take the same action (or produce the same action selection probabilities) each time it is in the same state. Such “ensemble methods” have been a big topic in supervised learning, where ensemble methods are often effective, but less so in reinforcement learning, where the evidence in support of the idea is, perhaps, weak. This paper just assumes it is a good idea, while citing prior work that i don’t think really shows that it is. A new method is proposed, called TDW, based on weighting the learner’s action values according to their recent squared temporal-difference errors. This is compared with simple ensemble method that have no memory other than their action values, on two gridworld problems and six Atari games. Comparison is also made with single (non-ensemble) methods that are given as much training data as one of the members of the ensemble. The results are consistent with TDW being better than the simpler ensemble methods on both gridworlds and Atari games, and with both ensemble methods being better than the single methods on Atari games but worse than the single methods on the gridworld problems. \n\nDoes this work make a contribution to the field of reinforcement learning? A lot depends on whether you think the problem of combining separate learners is an important problem. I am skeptical. First, this problem requires one to be able to identify separate learners that are operating in independent copies of the identical environment, and whose states and actions can be mapped one-to-one onto each another. (This is also a requirement of asynchronous methods, so this criticism applies to them as well.) Of course it is easy for us to arrange these things with our simulated test environments, and there may even be _a few_ real problems like this, but most of the time this is unrealistic. If a method requires this, then it is not a general method. Second, and I suppose relatedly, this new problem bears little relation to the original problem: Does our field benefit from the introduction of new problems and solution methods that apply only in special cases? Or are these special cases primarily distractions from the general case and thus ways to avoid coming to grips with the real problem?\n\nDifferent researchers might answer these rhetorical questions in different ways. It is a judgement call, but they reflect a real concern. A greater problem is that this work does not do a good job addressing the problem of ensembles. First, the introduction glosses over the question of whether ensembles are a good idea; it kinda suggests that their effectiveness has already been established, but does not commit itself to that statement; it does not make a clear claim that might be true or false about that prior work and that would validly support interest in this area. Second, this paper contains the wrong sort of comparisons of ensemble methods with single (non-ensemble) methods. It compares an ensemble of 10 learners with a single learner who only has as much data as one of the 10 learners in the ensemble. Thus, each single learner has only one-tenth as much data as has gone into the ensemble. Surely this is not the right comparison. It is a real problem that an early proponent of combining learners makes an unfair comparison with non-combined learners. It sets a poor standard and example for those who come later. The third way in which this work on ensemble RL agents is not very good it that its ensemble methods are not well suited to the task. That is, if one was going to combine learners, then there are many interesting natural ways to do this, and these are not done. It would be natural for each learner to keep track of how much experience or confidence it has in each part of the state space. The TDW algorithm can be seen as one way of doing this, but it is just one rather idiosyncratic way; there are many more natural ways. If we are going to explore this new problem, and new algorithms for solving it, then one really ought to do better than propose only methods that retain only their action-value function and nothing else that might facilitate the subsequent combination. In these three ways I feel this paper is not a good example of exploring the new problem of ensemble learning, even if you think that problem is a worthy one.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJeFubbPcH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1.\tThe paper suggests an ensemble method that leads to better performance over the execution of a single agent. The final (behavior) agent is a weighted average (or majority vote) of the members, where weights are determined by the accumulated TD error of the agent in the episode so far. The TD error serves as a confidence measure of the ensemble members in their predictions.\n2.\tAssume that one of the Q functions, Q_i is Q*. I.e., the optimal Q function. Let Q_ii = Q_i + B. I.e., a biased version of the optimal Q function. We know that both Q_i and Q_ii induce the same (optimal) policy. However, while delta_t^i is zero everywhere (the bellman error is zero for Q*), delta_t^ii is not zero (because of the discount factor). If B is large enough then delta_t^ii will be arbitrarily large to the point where the TDW algorithm will completely overlook Q_ii.\n3.\tFear of uncertainty: in stochastic settings the TD error will almost surely won’t be zero. How does the method work in the presence of stochasticity?\n4.\tIn my understanding, TDW favors policies that go to deterministic parts of the state space (where the TD error can be arbitrarily small), over policies that go to uncertain parts of the state space (where the TD error will never be zero). However, it is not difficult to construct an example where the optimal policy is to go to the uncertain parts of the state space. Therefore, TDW favors determinism over uncertainty. However, determinism is not necessarily linked to better performance.\n5.\tThe paper links between certainty (as reflected in the TD error) and performance. If the performance criterion would be risk-sensitive, e.g. CVAR, then I could agree more with the claim (maybe then performance is linked with certainty). However, certainty and performance do not go together, at least not when talking about the expected return criteria. \n6.\tOut of curiosity: since the method requires calculating the max over Q. How would it work in a continuous action space?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1xL3IS55r", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, this is a very interesting paper and, I think, would make a great addition to ICLR.  I find the ideas discussed in the paper to be stimulating and important to the community.\n\nThe authors are addressing a frequently overlooked and under-discussed aspect of RL: namely the instability of training that occurs when a Q-function becomes decreasingly familiar with areas of a state space while it focuses on a particular \"trajectory\" or solution to a task.\n\nThe authors add a measurement of uncertainty which is a simple heuristic.  It does require a couple additional hyperparameters.  One of which is not investigated in the experiments.\n\nThe experiments are ample and the approach is compared to a baseline and an alternative approach.  The experiments are informative and show the superiority of the proposed approach.\n\nComments:\n\nThe paper initially left me with the impression that the members were trained separately but Algorithm 1 looks like they are trained together.  This would have a significant impact on the number of training time steps required for training.  Please clarify.\n\nThe literature review is good but must add two relevant works.  One is MMRL by Doya et al.  I think their notion of a forward model is similar in spirit so it should be discussed.  Also, see the dissertation by DL Elliott entitled: THE WISDOM OF THE CROWD: RELIABLE DEEP REINFORCEMENT LEARNING THROUGH ENSEMBLES OF Q-FUNCTIONS.\n\nSection 3.3 didn't elucidate the topic for me.  It could be removed.  I think the description of the algorithm is sufficient.  Could be replaced by additional analysis of the uncertainty parameter from the experiments.  Also, what is the importance of the variable L in (5)?\n\nWhat does i denote in (6)?  I assume it's the index of the ensemble member.  Also, mention how the uncertainty values are initialized here.\n\nI feel that the paper is never really solidified in the mind until seeing (7).\n\nYour experiments indicate that the uncertainty value causes the \"preferred\" ensemble member to switch during the game of breakout.  That makes a lot of sense given the cyclical, for lack of a better term, nature of the game.  How about for tasks like the bipedal walker or the cart-pole task?  Would you expect, when training is completed, for a single member to dominant from the start position to the end of the trial?  If so, that's a limitation of the approach that doesn't diminish the contribution in my opinion.\n\nWould be interesting to see how the number of ensemble members used during a trial changes during training.  Does it increase/decrease?  Would love to see more analysis of this.  AGAIN, I don't think it's a negative thing if all are used but, eventually, one dominates.\n\nHow would using simpler Q-function approximators change the results?  Would it force them to decompose the task?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method to combine the decision of an ensemble of RL agents. It uses an uncertainty measure based on the TD error, and suggests a weighted average or weighted voting mechanism to combine their policy or value functions to come up with a joint decision.
The reviewers raised several concerns, including whether the method works in the stochastic setting, whether it favours deterministic parts of the state space, its sensitivity to bias, and unfair comparison to a single agent setting.
There is also a relevant PhD dissertation (Elliot, 2017), which the authors surprisingly refused to discuss and cite because apparently it was not published at any conference. A PhD dissertation is a citable reference, if it is relevant. If it is, a good scholarship requires proper citation.

Overall, even though the proposed method might potentially be useful, it requires further investigations. Two out of three reviewers are not positive about the paper in its current form. Therefore, I cannot recommend acceptance at this stage.

Elliott, Daniel L., The Wisdom of the crowd : reliable deep reinforcement learning through ensembles of Q-functions, PhD Dissertation, Colorado State University, 2017
Decision: Reject