# Ensemble Network Architecture for Deep Reinforcement Learning

Authors: Xi-liang Chen; Lei Cao; Chen-xi Li; Zhi-xiong Xu; Jun Lai
Journal: Mathematical Problems in Engineering
 (2018)

---

## Abstract

The popular deepQ learning algorithm is known to be instability because of the Q-value’s shake and overestimation action values under certain conditions. These issues tend to adversely affect their performance. In this paper, we develop the ensemble network architecture for deep reinforcement learning which is based on value function approximation. The temporal ensemble stabilizes the training process by reducing the variance of target approximation error and the ensemble of target values reduces the overestimate and makes better performance by estimating more accurate Q-value. Our results show that this architecture leads to statistically significant better value evaluation and more stable and better performance on several classical control tasks at OpenAI Gym environment.


---

## Body

## 1. Introduction

Reinforcement learning (RL) algorithms [1, 2] are very suitable for learning to control an agent by letting it interact with an environment. In recent years, deep neural networks (DNN) have been introduced into reinforcement learning, and they have achieved a great success on the value function approximation. The first deep Q-network (DQN) algorithm which successfully combines a powerful nonlinear function approximation technique known as DNN together with the Q-learning algorithm was proposed by Mnih et al. [3]. In this paper, experience replay mechanism was proposed. Following the DQN work, a variety of solutions have been proposed to stabilize the algorithms [3–9]. The deep Q-networks classes have achieved unprecedented success in challenging domains such as Atari 2600 and some other games.Although DQN algorithms have been successful in solving many problems because of their powerful function approximation ability and strong generalization between similar state inputs, they are still poor in solving some issues. Two reasons for this are as follows: (a) the randomness of the sampling is likely to lead to serious shock and (b) these systematic errors might cause instability, poor performance, and sometimes divergence of learning. In order to address these issues, the averaged target DQN (ADQN) [10] algorithm is implemented to construct target values by combining target Q-networks continuously with a single learning network, and the Bootstrapped DQN [11] algorithm is proposed to get more efficient exploration and better performance with the use of several Q-networks learning in parallel. Although these algorithms do reduce the overestimate, they do not evaluate the importance of the past learned networks. Besides, high variance in target values combined with the max operator still exists.There are some ensemble algorithms [4, 12] solving this issue in reinforcement learning, but these existing algorithms are not compatible with nonlinearly parameterized value functions.In this paper, we propose the ensemble algorithm as a solution to this problem. In order to enhance learning speed and final performance, we combine multiple reinforcement learning algorithms in a single agent with several ensemble algorithms to determine the actions or action probabilities. In supervised learning, ensemble algorithms such as bagging, boosting, and mixtures of experts [13] are often used for learning and combining multiple classifiers. But in RL, ensemble algorithms are used for representing and learning the value function.Based on an agent integrated with multiple reinforcement learning algorithms, multiple value functions are learned at the same time. The ensembles combine the policies derived from the value functions in a final policy for the agent. The majority voting (MV), the rank voting (RV), the Boltzmann multiplication (BM), and the Boltzmann addition (BA) are used to combine RL algorithms. While these methods are costly in deep reinforcement learning (DRL) algorithms, we combine different DRL algorithms that learn separate value functions and policies. Therefore in our ensemble approaches we combine the different policies derived from the update targets learned by deepQ-networks, deep Sarsa networks, double deep Q-networks, and other DRL algorithms. As a consequence, this leads to reduced overestimations, more stable learning process, and improved performance.


## 2. Related Work

### 2.1. Reinforcement Learning

Reinforcement learning is a machine learning method that allows the system to interact with and learn from the environment to maximize cumulative return rewards. Assume that the standard reinforcement learning setting where an agent interacts with the environmentε. We can describe this process with Markov Decision Processes (MDP) [2, 9]. It can be specified as a tuple (S,A,π,R,γ). At each step t, the agent receives a state st, and select an action at from the set of legal actions A according to the policy π, where π is a policy mapping sequences to actions. The action is passed to the environment E. In addition, the agent receives the next state st+1 and a reward signal rt. This process continues until the agent reaches a terminal state.The agent seeks to maximize the expected discounted return, where we define the future discounted return at timet as Rt=∑k=0∞γkrt+k with discount factor γ∈(0,1]. The goal of the RL agent is to learn a policy which makes the future discounted return maximize. For an agent behaving according to a stochastic policy π, the value of the state-action pair can be defined as follows: Qπs,a=ERt∣st=s,at=a,π. The optimal action-value function Q satisfies the Bellman equation Q∗s,a=Es′~εr+γmaxa′Q∗(s′,a′)∣s,a.The reinforcement learning algorithms estimate the action value function by iteratively updating the Bellman equationQ∗s,a=Es′~εr+γmaxa′Q∗(s′,a′)∣s,a. When t→∞, the algorithm makes Q-value function converge to the optimal action value function [1]. If the optimal Q-function Q∗ is known, the agent can select optimal actions by selecting the action with the maximal value in a state: π∗=argmaxaQ∗(s,a).


### 2.2. Target DeepQ Learning

RL agents update their model parameters while they observe a stream of transitions like(st,at,rt+1,st+1). They discard the incoming data after a single update. There are two issues with this method. The first one is that there are strong correlations among the incoming data, which may break the assumption of many popular stochastic gradient-based algorithms. Secondly, the minor changes in the Q function may result in a huge change in the policy, which makes the algorithm difficult to converge [7, 9, 14, 15].As for the deepQ-networks algorithms proposed in (Mnih et al., 2013), two aspects are improved. On the one hand, the action value function is approximated by the DNN, DQN uses the DNN with a parameter θ to approximate the value function, Q(s,a;θ)≈Q∗s,a;θ; on the other hand, the experience replay mechanism is adopted. The algorithm learns from sampled transitions from an experience buffer, rather than learning fully online. This mechanism makes it possible to break the temporal correlations by mixing more and less recent experience for updating and training. This model free reinforcement learning algorithm solves the problem of “model disaster” and uses the generalized approximation method of the value function to solve the problem of “dimension disaster."The convergence issue was mentioned in 2015 by Schaul et al. [14]. The above Q-learning update rules can be directly implemented in a neural network. DQN uses the DNN with parameters θ to approximate the value function°. The parameter θ updates from transition (st,at,rt+1,st+1) are given by the following [11]:(1)θt+1⟵θt+αytQ-Qst,at;θt∇θQst,at;θt,with yiDQN=rt+γmaxaQ~st+1,a;θt-The update targets for Sarsa can be described as follows:(2)yiSarsa=rt+γQ~st+1,at+1;θt-,where α is the scalar learning rate. θ- are target network parameters which are fixed to θ-=θt. In case the squared error is taken as a loss function Liθi=Es′~ε(yi-Q(s,a;θi))2.In general, experience replay can reduce the amount of experience required to learn and replace it with more computation and more memory, which are often cheaper resources than the RL agent’s interactions with its environment [14].


### 2.3. Double DeepQ Learning

InQ-learning and DQN, the max operator uses the same values to both select and evaluate an action. This can therefore lead to overoptimistic value estimates (van Hasselt, 2010). To mitigate this problem, the update targets value of double Q-learning error can then be written as follows:(3)yiDDQN=rt+γQ~st+1,arg⁡maxa′Q~st+1,a′;θt;θt-.DDQN is the same as for DQN [8], but with the target yiDQN replaced with yiDDQN.


## 2.1. Reinforcement Learning

Reinforcement learning is a machine learning method that allows the system to interact with and learn from the environment to maximize cumulative return rewards. Assume that the standard reinforcement learning setting where an agent interacts with the environmentε. We can describe this process with Markov Decision Processes (MDP) [2, 9]. It can be specified as a tuple (S,A,π,R,γ). At each step t, the agent receives a state st, and select an action at from the set of legal actions A according to the policy π, where π is a policy mapping sequences to actions. The action is passed to the environment E. In addition, the agent receives the next state st+1 and a reward signal rt. This process continues until the agent reaches a terminal state.The agent seeks to maximize the expected discounted return, where we define the future discounted return at timet as Rt=∑k=0∞γkrt+k with discount factor γ∈(0,1]. The goal of the RL agent is to learn a policy which makes the future discounted return maximize. For an agent behaving according to a stochastic policy π, the value of the state-action pair can be defined as follows: Qπs,a=ERt∣st=s,at=a,π. The optimal action-value function Q satisfies the Bellman equation Q∗s,a=Es′~εr+γmaxa′Q∗(s′,a′)∣s,a.The reinforcement learning algorithms estimate the action value function by iteratively updating the Bellman equationQ∗s,a=Es′~εr+γmaxa′Q∗(s′,a′)∣s,a. When t→∞, the algorithm makes Q-value function converge to the optimal action value function [1]. If the optimal Q-function Q∗ is known, the agent can select optimal actions by selecting the action with the maximal value in a state: π∗=argmaxaQ∗(s,a).


## 2.2. Target DeepQ Learning

RL agents update their model parameters while they observe a stream of transitions like(st,at,rt+1,st+1). They discard the incoming data after a single update. There are two issues with this method. The first one is that there are strong correlations among the incoming data, which may break the assumption of many popular stochastic gradient-based algorithms. Secondly, the minor changes in the Q function may result in a huge change in the policy, which makes the algorithm difficult to converge [7, 9, 14, 15].As for the deepQ-networks algorithms proposed in (Mnih et al., 2013), two aspects are improved. On the one hand, the action value function is approximated by the DNN, DQN uses the DNN with a parameter θ to approximate the value function, Q(s,a;θ)≈Q∗s,a;θ; on the other hand, the experience replay mechanism is adopted. The algorithm learns from sampled transitions from an experience buffer, rather than learning fully online. This mechanism makes it possible to break the temporal correlations by mixing more and less recent experience for updating and training. This model free reinforcement learning algorithm solves the problem of “model disaster” and uses the generalized approximation method of the value function to solve the problem of “dimension disaster."The convergence issue was mentioned in 2015 by Schaul et al. [14]. The above Q-learning update rules can be directly implemented in a neural network. DQN uses the DNN with parameters θ to approximate the value function°. The parameter θ updates from transition (st,at,rt+1,st+1) are given by the following [11]:(1)θt+1⟵θt+αytQ-Qst,at;θt∇θQst,at;θt,with yiDQN=rt+γmaxaQ~st+1,a;θt-The update targets for Sarsa can be described as follows:(2)yiSarsa=rt+γQ~st+1,at+1;θt-,where α is the scalar learning rate. θ- are target network parameters which are fixed to θ-=θt. In case the squared error is taken as a loss function Liθi=Es′~ε(yi-Q(s,a;θi))2.In general, experience replay can reduce the amount of experience required to learn and replace it with more computation and more memory, which are often cheaper resources than the RL agent’s interactions with its environment [14].


## 2.3. Double DeepQ Learning

InQ-learning and DQN, the max operator uses the same values to both select and evaluate an action. This can therefore lead to overoptimistic value estimates (van Hasselt, 2010). To mitigate this problem, the update targets value of double Q-learning error can then be written as follows:(3)yiDDQN=rt+γQ~st+1,arg⁡maxa′Q~st+1,a′;θt;θt-.DDQN is the same as for DQN [8], but with the target yiDQN replaced with yiDDQN.


## 3. Ensemble Methods for Deep Reinforcement Learning

As DQN classes use DNNs to approximate the value function, it has strong generalization ability between similar state inputs. The generalization can cause divergence in the case of repeated bootstrapped temporal difference updates. So we can solve this issue by integrating different versions of the target network.In contrast to a single classifier, ensemble algorithms in a system have been shown to be more effective. They can lead to a higher accuracy. Bagging, boosting, and Ada Boosting are methods to train multiple classifiers. But in RL, ensemble algorithms are used for representing and learning the value function. They are combined by major voting, Rank Voting, Boltzmann Multiplication, mixture model, and other ensemble methods. If the errors of the single classifiers are not strongly correlated, this can significantly improve the classification accuracy.


### 3.1. Temporal Ensemble

As described in Section2.2, the DQN classes of deep reinforcement learning algorithms use a target network with parameters θ- copied from θt every C steps. Temporal Ensemble method is suitable for the algorithms which use a target network for updating and training. Temporal ensemble uses the previous K learned networks to produce the value estimate and builds up K∈N complete networks with K distinct memory buffers. The recent Q-value function is trained according to its own target network Q(s,a;θt). So each one of Q-value functions Q1,Q2,…,Qk represents temporally extended estimate of Q-value function.Note that the more recent target network is likely to be more accurate at the beginning of the training and the accuracy of the target networks is increasing as the training goes on. So we denote a learning rate parameterλ∈0,1 here for target network. The weight of ith target network is wi=λi-1/∑i=1Nλi-1.So the learnedQ-value function by temporal ensemble can be described as follows:(4)QTs,a;θ=∑i=1Nλi-1Qis,a;θi∑i=1Nλi-1.As limλ⟶1⁡λi-1⁡/∑i=1Nλi-1=1/N, we can see that the target networks have the same weights when λ equals 1. This formula indicates that the closer the target networks are, the greater the target networks’ weight is. As target networks become more accurate, their weights become equal. The loss function remains the same as in DQN and so does the parameter update equation: (5)yiT=rt+γmaxa′⁡∑i=1NwiQTs′,a′;θT,θt+1⟵θt+αytT-Qst,at;θt∇θQst,at;θt.In every iteration, the parameters of the oldest ones are removed from the target network buffer and the newest ones are added to the buffer. Note that the Q-value functions are inaccurate at the beginning of training. So the parameter λ may be a function of time and even the state space.


### 3.2. Ensemble of Target Values

The traditional ensemble reinforcement learning algorithms maintain multiple tabular algorithms in memory space [4, 16], and majority voting, rank voting, Boltzmann addition, and so forth are used to combine these tabular algorithms. But deep reinforcement learning uses neutral networks as function approximators. The use of multiple neural networks is very computationally expensive and inefficient. In contrast to previous researches, we combine different DRL algorithms that learn separate value functions and policies. Therefore in our ensemble approaches we combine the different policies derived from the update targets learned by deep Q-networks, deep Sarsa networks, double deep Q-networks, and other DRL algorithms as follows: (6)yiDQN=rt+γmaxa⁡Q~st+1,a;θt-,yiSarsa=rt+γQ~st+1,at+1;θt-,yiDDQN=rt+γQ~st+1,argmaxa′⁡Q~st+1,at+1;θt;θt-.Besides these update targets formula, other algorithms based on value function approximators can be also used to combine. The update targets according to the algorithm k at time t will be denoted by yt=∑i=1kβiyti.The loss function remains the same as in DQN and so does the parameter update equation:(7)θt+1⟵θt+αytE-Qst,at;θt∇θQst,at;θt.


### 3.3. The Ensemble Network Architecture

The temporal and target values ensemble algorithm (TEDQN) is an integrated architecture of the value-based DRL algorithms. As shown in Sections3.1 and 3.2, the ensemble network architecture has two parts to avoid divergence and improve performance.The architecture of our ensemble algorithm is shown in Figure1; these two parts are combined together by evaluated network.Figure 1
The architecture of the ensemble algorithm.The temporal ensemble stabilizes the training process by reducing the variance of target approximation error [10]. Besides, the ensemble of target values reduces the overestimate and makes better performance by estimating more accurate Q-value. The temporal and target values ensemble algorithm are given by Algorithm 1.Algorithm 1:The temporal and target values ensemble algorithm.
( 1 ) Initialize action-value network Q with random weights θ ( 2 ) Initialize the target neural network buffer (Qi)i=1L ( 3 ) For episode 1, M do ( 4 )  For t=1, T do ( 5 )  With probability ε select a random action at, otherwise a t = a r g m a x a Q ( s t , a ; θ ) ( 6 )  Execute action at in environment and observe reward rt and next state st+1, and store transition (st,at,rt,st+1) in D ( 7 )  Sample random minibatch of transition (st,at,rt,st+1) from D ( 8 )  set wi=λi-1/∑i=1Nλi-1 ( 9 )  Ensemble Q-learner Q~s,a;θ=∑i=1NwiQi(s,a;θi) ( 10 )  set yiDQN=rt+γmaxaQ~st+1,a;θt- ( 11 )  set yiSarsa=rt+γQ~st+1,at+1;θt- ( 12 )  set yiDDQN=rt+γQ~st+1,argmaxa′Q~st+1,at+1;θt;θt- ( 13 )  Set yi=rj,if  episode  terminates  at  stepj+1;∑i=1kβiyti,otherwise ( 14 )  θ i = argmin θ ⁡ E y s , a i - Q s , a ; θ 2 ( 15 )  Every C steps reset Q~=Q ( 16 )  End for ( 17 ) End forAs the ensemble network architecture shares the same input-output interface with standardQ-networks and target networks, we can recycle all learning algorithms with Q-networks to train the ensemble architecture.


## 3.1. Temporal Ensemble

As described in Section2.2, the DQN classes of deep reinforcement learning algorithms use a target network with parameters θ- copied from θt every C steps. Temporal Ensemble method is suitable for the algorithms which use a target network for updating and training. Temporal ensemble uses the previous K learned networks to produce the value estimate and builds up K∈N complete networks with K distinct memory buffers. The recent Q-value function is trained according to its own target network Q(s,a;θt). So each one of Q-value functions Q1,Q2,…,Qk represents temporally extended estimate of Q-value function.Note that the more recent target network is likely to be more accurate at the beginning of the training and the accuracy of the target networks is increasing as the training goes on. So we denote a learning rate parameterλ∈0,1 here for target network. The weight of ith target network is wi=λi-1/∑i=1Nλi-1.So the learnedQ-value function by temporal ensemble can be described as follows:(4)QTs,a;θ=∑i=1Nλi-1Qis,a;θi∑i=1Nλi-1.As limλ⟶1⁡λi-1⁡/∑i=1Nλi-1=1/N, we can see that the target networks have the same weights when λ equals 1. This formula indicates that the closer the target networks are, the greater the target networks’ weight is. As target networks become more accurate, their weights become equal. The loss function remains the same as in DQN and so does the parameter update equation: (5)yiT=rt+γmaxa′⁡∑i=1NwiQTs′,a′;θT,θt+1⟵θt+αytT-Qst,at;θt∇θQst,at;θt.In every iteration, the parameters of the oldest ones are removed from the target network buffer and the newest ones are added to the buffer. Note that the Q-value functions are inaccurate at the beginning of training. So the parameter λ may be a function of time and even the state space.


## 3.2. Ensemble of Target Values

The traditional ensemble reinforcement learning algorithms maintain multiple tabular algorithms in memory space [4, 16], and majority voting, rank voting, Boltzmann addition, and so forth are used to combine these tabular algorithms. But deep reinforcement learning uses neutral networks as function approximators. The use of multiple neural networks is very computationally expensive and inefficient. In contrast to previous researches, we combine different DRL algorithms that learn separate value functions and policies. Therefore in our ensemble approaches we combine the different policies derived from the update targets learned by deep Q-networks, deep Sarsa networks, double deep Q-networks, and other DRL algorithms as follows: (6)yiDQN=rt+γmaxa⁡Q~st+1,a;θt-,yiSarsa=rt+γQ~st+1,at+1;θt-,yiDDQN=rt+γQ~st+1,argmaxa′⁡Q~st+1,at+1;θt;θt-.Besides these update targets formula, other algorithms based on value function approximators can be also used to combine. The update targets according to the algorithm k at time t will be denoted by yt=∑i=1kβiyti.The loss function remains the same as in DQN and so does the parameter update equation:(7)θt+1⟵θt+αytE-Qst,at;θt∇θQst,at;θt.


## 3.3. The Ensemble Network Architecture

The temporal and target values ensemble algorithm (TEDQN) is an integrated architecture of the value-based DRL algorithms. As shown in Sections3.1 and 3.2, the ensemble network architecture has two parts to avoid divergence and improve performance.The architecture of our ensemble algorithm is shown in Figure1; these two parts are combined together by evaluated network.Figure 1
The architecture of the ensemble algorithm.The temporal ensemble stabilizes the training process by reducing the variance of target approximation error [10]. Besides, the ensemble of target values reduces the overestimate and makes better performance by estimating more accurate Q-value. The temporal and target values ensemble algorithm are given by Algorithm 1.Algorithm 1:The temporal and target values ensemble algorithm.
( 1 ) Initialize action-value network Q with random weights θ ( 2 ) Initialize the target neural network buffer (Qi)i=1L ( 3 ) For episode 1, M do ( 4 )  For t=1, T do ( 5 )  With probability ε select a random action at, otherwise a t = a r g m a x a Q ( s t , a ; θ ) ( 6 )  Execute action at in environment and observe reward rt and next state st+1, and store transition (st,at,rt,st+1) in D ( 7 )  Sample random minibatch of transition (st,at,rt,st+1) from D ( 8 )  set wi=λi-1/∑i=1Nλi-1 ( 9 )  Ensemble Q-learner Q~s,a;θ=∑i=1NwiQi(s,a;θi) ( 10 )  set yiDQN=rt+γmaxaQ~st+1,a;θt- ( 11 )  set yiSarsa=rt+γQ~st+1,at+1;θt- ( 12 )  set yiDDQN=rt+γQ~st+1,argmaxa′Q~st+1,at+1;θt;θt- ( 13 )  Set yi=rj,if  episode  terminates  at  stepj+1;∑i=1kβiyti,otherwise ( 14 )  θ i = argmin θ ⁡ E y s , a i - Q s , a ; θ 2 ( 15 )  Every C steps reset Q~=Q ( 16 )  End for ( 17 ) End forAs the ensemble network architecture shares the same input-output interface with standardQ-networks and target networks, we can recycle all learning algorithms with Q-networks to train the ensemble architecture.


## 4. Experiments

### 4.1. Experimental Setup

So far, we have carried out our experiments on several classical control and Box2D environments on OpenAI Gym: CartPole-v0, MountainCar-v0, and LunarLander-v2 [15]. We use the same network architecture, learning algorithms, and hyperparameters for all these environments.We trained the algorithms using 10,000 episodes and used the Adaptive Moment Estimation (Adam) algorithm to minimize the loss with learning rateμ = 0.00001 and set the batch size to 32. The summary of the configuration is provided below. The target network updated each 300 steps. The behavior policy during training was ε-greedy with ε annealed linearly from 1 to 0.01 over the first five thousands steps and fixed at 0.01 thereafter. We used a replay memory of ten thousands most recent transitions.We independently executed each method 10 times, respectively, on every task. For each running time, the learned policy will be tested 100 times without exploration noise or prior knowledge by every 100 training episodes to calculate the average scores. We report the mean and standard deviation of the convergence episodes and the scores of the best policy.


### 4.2. Results and Analysis

We consider three baseline algorithms that use target network and value function approximation, namely, the version of the DQN algorithm from the Nature paper [8], DSN that reduce over estimation [17], and DDQN that substantially improved the state-of-the-art by reducing the overestimation bias with double Q-learning [9].Using this 10 no-ops performance measure, it is clear that the ensemble network does substantially better than a single network. For comparison we also show results for DQN, DSN, and DDQN. Figure2 shows the improvement of the ensemble network over the baseline single network of DQN, DSN, and DDQN. Again, we see that the improvements are often very dramatic.Figure 2
Training curves tracking the agent’s average score and average predicted action-value. (a) Performance comparison of all algorithms in terms of the average reward on each task. (b) Average predicted action-value on a held-out set of states on each task. Each point on the curve is the average of the action-valueQ computed over the held-out set of states. (c) The performance of DQN and TEDQN on each task. The darker line shows the average scores of each algorithm, and the orange shaded area shows the two extreme values of DQN and the green shaded area shows TE DQN.The results in Table1 show that algorithms we presented can successfully train neural network controllers on the classical control domain on OpenAI Gym. A detailed comparison shows that there are several games in which TE DQN greatly improves upon DQN, DSN, and DDQN. Noteworthy examples include CartPole-v0 (performance has been improved by 13.6%, 79.5%, and 7.8%, and variance has been reduced by 100%, 100%, and 100%), MountainCar-v0 (performance has been improved by 26.7%, 21.2%, and 24.8%, and variance has been reduced by 31.6%, 77.9%, and 8.4%), and LunarLander-v2 (performance has been improved by 28.3%, 32.8%, and 50.5%, and variance has been reduced by 19.2%, 46.4%, and 50.5%).Table 1
The columns present the average performance of DQN, DSN, DDQN, EDQN, and TE-DQN after 10000 episodes, usingε-greedy policy with ε = 0.0001 after 10000 steps. The standard variation represents the variability over seven independent trials. Average performance improved with the number of averaged networks.
Task(AVG score, Std.) CartPole-v0 MountainCar-v0 LunarLander-v2 DQN (264.9, 21.7) (−148.2, 17.4) (159.3, 16.7) DSN (167.1, 61.6) (−137.7, 53.9) (153.9, 25.2) Double DQN (278.2, 31.8) (−144.2, 16.8) (135.8, 11.8) TE DQNK=3 (299.1, 1.3) (−115.6, 21.4) (186.9, 19.1) TE DQNK=6 (300, 0) (−108.4, 11.9) (204.4, 13.5)


## 4.1. Experimental Setup

So far, we have carried out our experiments on several classical control and Box2D environments on OpenAI Gym: CartPole-v0, MountainCar-v0, and LunarLander-v2 [15]. We use the same network architecture, learning algorithms, and hyperparameters for all these environments.We trained the algorithms using 10,000 episodes and used the Adaptive Moment Estimation (Adam) algorithm to minimize the loss with learning rateμ = 0.00001 and set the batch size to 32. The summary of the configuration is provided below. The target network updated each 300 steps. The behavior policy during training was ε-greedy with ε annealed linearly from 1 to 0.01 over the first five thousands steps and fixed at 0.01 thereafter. We used a replay memory of ten thousands most recent transitions.We independently executed each method 10 times, respectively, on every task. For each running time, the learned policy will be tested 100 times without exploration noise or prior knowledge by every 100 training episodes to calculate the average scores. We report the mean and standard deviation of the convergence episodes and the scores of the best policy.


## 4.2. Results and Analysis

We consider three baseline algorithms that use target network and value function approximation, namely, the version of the DQN algorithm from the Nature paper [8], DSN that reduce over estimation [17], and DDQN that substantially improved the state-of-the-art by reducing the overestimation bias with double Q-learning [9].Using this 10 no-ops performance measure, it is clear that the ensemble network does substantially better than a single network. For comparison we also show results for DQN, DSN, and DDQN. Figure2 shows the improvement of the ensemble network over the baseline single network of DQN, DSN, and DDQN. Again, we see that the improvements are often very dramatic.Figure 2
Training curves tracking the agent’s average score and average predicted action-value. (a) Performance comparison of all algorithms in terms of the average reward on each task. (b) Average predicted action-value on a held-out set of states on each task. Each point on the curve is the average of the action-valueQ computed over the held-out set of states. (c) The performance of DQN and TEDQN on each task. The darker line shows the average scores of each algorithm, and the orange shaded area shows the two extreme values of DQN and the green shaded area shows TE DQN.The results in Table1 show that algorithms we presented can successfully train neural network controllers on the classical control domain on OpenAI Gym. A detailed comparison shows that there are several games in which TE DQN greatly improves upon DQN, DSN, and DDQN. Noteworthy examples include CartPole-v0 (performance has been improved by 13.6%, 79.5%, and 7.8%, and variance has been reduced by 100%, 100%, and 100%), MountainCar-v0 (performance has been improved by 26.7%, 21.2%, and 24.8%, and variance has been reduced by 31.6%, 77.9%, and 8.4%), and LunarLander-v2 (performance has been improved by 28.3%, 32.8%, and 50.5%, and variance has been reduced by 19.2%, 46.4%, and 50.5%).Table 1
The columns present the average performance of DQN, DSN, DDQN, EDQN, and TE-DQN after 10000 episodes, usingε-greedy policy with ε = 0.0001 after 10000 steps. The standard variation represents the variability over seven independent trials. Average performance improved with the number of averaged networks.
Task(AVG score, Std.) CartPole-v0 MountainCar-v0 LunarLander-v2 DQN (264.9, 21.7) (−148.2, 17.4) (159.3, 16.7) DSN (167.1, 61.6) (−137.7, 53.9) (153.9, 25.2) Double DQN (278.2, 31.8) (−144.2, 16.8) (135.8, 11.8) TE DQNK=3 (299.1, 1.3) (−115.6, 21.4) (186.9, 19.1) TE DQNK=6 (300, 0) (−108.4, 11.9) (204.4, 13.5)


## 5. Conclusion

We introduced a new learning architecture, making temporal extension and the ensemble of target values for deepQ learning algorithms, while sharing a common learning module. The new ensemble architecture, in combination with some algorithmic improvements, leads to dramatic improvements over existing approaches for deep RL in the challenging classical control issues. In practice, this ensemble architecture can be very convenient to integrate the RL methods based on the approximate value function.Although the ensemble algorithms are superior to a single reinforcement learning algorithm, it is noted that the computational complexity is higher. The experiments also show that the temporal ensemble makes the training process more stable, and the ensemble of a variety of algorithms makes the estimation of theQ-value more accurate. The combination of the two ways enables the training to achieve a stable convergence. This is due to the fact that ensembles improve independent algorithms most if the algorithms predictions are less correlated. So that the output of the Q-network based on the choice of action can achieve balance between exploration and exploitation.In fact, the independence of the ensemble algorithms and their elements is very important on the performance for ensemble algorithms. In further works, we want to analyze the role of each algorithm and eachQ-network in different stages, so as to further enhance the performance of the ensemble algorithm.



---

*Source: 2129393-2018-04-05.xml*