A Hierarchical Architecture for Sequential Decision-Making in Autonomous
Driving using Deep Reinforcement Learning

Majid Moghadam 1 Gabriel Hugh Elkaim 1

Abstract
Tactical decision making is a critical feature foradvanced driving systems, that incorporates several challenges such as complexity of the uncertain environment and reliability of the autonomous system. In this work, we develop amulti-modal architecture that includes the environmental modeling of ego surrounding and train
a deep reinforcement learning (DRL) agent thatyields consistent performance in stochastic highway driving scenarios. To this end, we feed theoccupancy grid of the ego surrounding into the
DRL agent and obtain the high-level sequentialcommands (i.e. lane change) to send them tolower-level controllers. We will show that dividing the autonomous driving problem into a multilayer control architecture enables us to leverage
the AI power to solve each layer separately andachieve an admissible reliability score. Comparing with end-to-end approaches, this architecture
enables us to end up with a more reliable system
which can be implemented in actual self-drivingcars.

1. Introduction
Advanced Driving Assistance Systems (ADAS) are developed to increase traffic safety by reducing the impact ofhuman errors. The evolution of various levels of driving autonomy has seen a significant speed up in last years aiming
to enhance comfort, safety and driving experience. For along time, with a limited amount of technological resources,automotive stakeholders were focusing on steady-state maneuvers to achieve some sort of autonomy. However, in
recent years the major focus of research in the field of autonomous driving is being directed to the transition maneuvers, most importantly tactical lane changing, required
1
Computer Science and Engineering Department, University of California, Santa Cruz, USA. Correspondence to: Majid
Moghadam <mamoghad@ucsc.edu>.

Real-world Sequential Decision Making workshop at ICML 2019,
Long Beach, California. Copyright 2019 by the author(s).

Figure 1. A sketch of our hierarchical approach for the autonomous
driving problem

for both fully and partially autonomous driving systems(Alizadeh et al., 2019).
Before AI, control and orientation of ground vehicles weretackled using feedback control techniques (Falcone et al.,2007; Broggi et al., 1999; Moghadam & Caliskan, 2015) that
attempt to stabilize the vehicle using the information collected from sensory measurements. The controller used togenerate input commands to the actuators like steering angle,
accelerator, and break to perform the driving tasks. Becauseof the extraordinary success of the neural networks (NN) in
the classification (Krizhevsky et al., 2012) and regression(Specht, 1991) problems, the researchers have decided toapply learning-based approaches to the control problems.
Leveraging Deep Reinforcement Learning (DRL) agents toplay Atari games (Mnih et al., 2015) was one of the earliest methods that gained considerable attention in the field.
After the success of DRL on video game plays, researchers
applied DRL on real-world problems like control and orientation of autonomous systems which have shown a great
potential to become a reliable alternative for the classicalcontrol approaches. There has been various deep learning(DL) techniques that applied to the problem of autonomousvehicles. One approach is to use an expert knowledge ofthe problem and train a neural network to learn the expert
policies in an end-to-end manner. This method is known as
imitation learning and has been applied to both self-drivingcars (Bojarski et al., 2016) and unmanned aerial vehicles(Bicer et al., 2019). Although the results are promising for

A Hierarchical Architecture for Decision-Making in Autonomous Driving using Deep RL

training an intelligent agent that shows off comparable performance as human, such agents may never outperform thehuman expert as the training dataset is being annotated using
solely expert demonstrations. Similar end-to-end architecture is also used to train DRL agents to drive autonomous
cars in simulation environments, like TORCS (Koutnı́k et al.,2013) and World Rally Championship (Perot et al., 2017).
Here the term end-to-end indicates that the raw pixel information of on-board camera on the vehicles are being used as
the state representation and continuous steering angle andacceleration is being calculated via DRL algorithm. Endto-end DRL approaches have the ability to outperform thehuman expert in various driving tasks. This can be verifies
by the results published in the mentioned articles. However,the safety and reliability of such algorithms become criticalproblems when applied to real-world problems. In addition,
the complexity of the environment in real-world applications may impeach the performance of these algorithms. In
other words, combining different layers of the ADAS systems which took long to reach the current performance inan end-to-end architecture which maps the observations directly to the actions, does not seem to guarantee the similar
performance in actual systems.
1.1. Our approach
As discussed above, in most of the studies in the literature
the raw sensory inputs like video frames of on-board camera
or RGB-D sensors on automobile have been used to train
a neural network in order to estimate the required action
to control the vehicle (see Fig.2). Most of the studies validated the performance of their methods in a number of
unreal simulations or video games. We believe that when it
comes to the real-world implementation of these methods,
the end-to-end learning techniques may suffer the lack ofreliability to control the agent as a result of huge number of
uncertainties while interacting in the actual world. To thisend, we divide the problem into different control level problems and try to solve each level separately. In other words,
the raw sensory inputs are going to be used to percept theenvironment and map it to a simpler two-dimensional world.
Accordingly, the AI agent in decision making layer will
generate high-level decision commands like left/right lanechange and send them to lower-level control layers which
deal with steering angle and gas pedal commands to stabilize the vehicle while following the generated high-level
actions. This approach is summarized in Fig.1. Also the
ADAS architecture is provided in Fig.2. Note that we use
the occupancy grid as the environment model around theego vehicle. We believe that, this approach is more reliable
in real-world implementations than end-to-end techniques
available in the literature because we do not perform a single
mapping from the highest level observations with considerable number of uncertainties to the lowest level actuator

control layer. This approach is much like the multi-loop
control techniques in control engineering which has provento show better performance in stabilizing complex systems(Slotine et al., 1991; Moghadam et al., 2018).

Figure 2. Hierarchical architecture of the general ADAS systems
vs. end-to-end approaches

In this study, we address the problem of high-level decision
making for an autonomous car using classical reinforcement
learning technique known as Q-learning. We implementthe −greedy algorithm to the problem defined in DeepCars simulation environment which is also designed andimplemented by the authors. After commenting on the performance of Q-learning algorithm, we implement the deep
reinforcement learning method for the same problem. Finally we present the results for both approaches and make acomparison between them.
1.2. Deep Q-Learning with real-time validation
The main idea in the reinforcement learning (RL) and dynamic programming (DP) is to control an agent or a process
while interacting with an environment using the observing
states and rewards received from the environment (Busoniu
et al., 2017). DP and RL are algorithmic methods for solving decision-making problem to achieve the desired goal
throughout the interaction with the world. DP methods require the model of the system’s behavior, whereas RL is amodel-free approach which improves the produced policy
while interacting with the environment. RL uses Markov
Decision Process (MDP) (Busoniu et al., 2017) to mathematically formalize discrete stochastic environment. As
MDP works in discrete time, states and actions in RL areusually discrete that leads to a sequential decision-making
problem. Rewards also provide an informative metric of
the agent’s performance, and the goal is to maximize theaccumulated long-term return over the track of the interaction with the environment. Although recent advances in RL(Lillicrap et al., 2015) enables us to apply to continuous
problems, here we leverage the discrete MDP to formalize
the decision-making problem in autonomous driving andtrain RL algorithms to achieve a favorable performance.

A Hierarchical Architecture for Decision-Making in Autonomous Driving using Deep RL

Q-learning evaluates how good taking an action might be ata particular state through learning the action-value function
Q(s, a). In Q-learning a memory table Q[s, a] is built tostore the Q-values for all the possible combinations of states
and actions. By sampling an action from the current state,
the reward R and the new states are found out, by which
the next action a that has the maximum Q(s0 , a0 ) from thememory table is taken. Taking an action in a particular state
has a Q − value which is depicted in eq. 1.
Q(s, a) = R(s, a, s0 ) + γmax
Qk (s0 , a0 )
0
a

(1)

Where s0 and a0 are next state and action respectively. However if the combinations of state and actions are too large orstates and actions are continuous, the memory and computation requirement for action-value function Q will be toohigh. To address this issue, Deep Q-Network (DQN) (Mnihet al., 2015) is utilized that approximates the action-valuefunction Q(s, a).
In this work, as shown in Algorithm 1, (Mnih et al., 2015;
Alizadeh et al., 2019), two networks, θ− and θ are created
and trained, one for retrieving Q values and one including
all updates in the training. Ultimately we synchronize θ−and θ to fix the Q − value targets temporarily, so it keeps
the target funciotn from changing abruptly. The loss will becalculated as in eq. 2.
!2
Li (θi ) =Es,a,s0 ,r∼D

r+γmaxQ(s , a ; θi− −Q(s, a; θi ))a000ables the agent to record the best trained model by validatingon unseen scenarios. Defining two various periods with different number of episodes helps the training to be faster andrecord more generalized model at the same time.

Algorithm 1 Deep Q-Network (DQN) with experience replay and real-time validation
Input: Initialize:
replay memory D to capacity M
action-value function Q with random weights θtarget action-value function Q̂ with wights θ− = θ
Output: Q∗
initialize sequence s1 = {x1 }initialize preprocesses sequence φ1 = φ(s1 )for t = 1, (..., T do
random action, with probability at =argmaxQ(φ(st ), a; θ) , otherwiseart , xt+1 : apply(at )st+1 = st , at , xt+1
φt+1 = φ(st+1 )D ← (φt , at , rt , φt+1 )(φj , aj , rj , φj+1 ) ← random(D)yj = rj + γmax
Q̂(φj+1 , a0 ; θ− )a0
∂ yj −Q(φj ,aj ,;θ)preform a gradient descent step on∂θ
if validation phase then
fix network parameters θfor t0 = 1, ...,val episodes dotake greedy actions: a = maxQ(φ(st0 ), a; θ)record episode rewards(2)
where the transitions (s, a, s0 , r) are retrieved from the experience replay D. Experience replay is used as a buffer,
which we sample a mini-batch of samples from it to train
the deep neural network. Since we are randomly samplingfrom the replay buffer, the data is closer to i.i.d and more
independent of each other that in turn makes the training
more stable. In addition, E indicates the expectation overthe probability distribution.
By utilizing the experience replay and target network, theinput and output of the model turns to be more stable to train
and the network behaves more like a supervised learningalgorithm.
We also appended the real-time validation phase to traditional DQN in order to record the best trained model during
the training. For this purpose, we define two periods, bywhich validation phase flag is activated and lates network
weights are being recorded and exploited during validation.
Depending on which period, the agent’s performance isevaluated for a number of episodes and the achieved mean
reward is compared to the latest maximum value. This en-

if mean validation reward is increased thenθbackup ← θevery C steps; Q̂ = Q

In order to improve the DQN performance, we also implemented Double DQN (DDQN) algorithm (Van Hasselt et al.,2016) which uses two Q−networks in order to deal with theoveroptimistic value estimations. Target value estimation in
Double Q-learning is performed using eq. 3

Q(s, a)DoubleQ = R(s, a, s0 )+γQ(s0 , argmaxQ(s0 , a, θ); θ0 )(3)
where, θ and θ0 represent different Q−network parameters,
one to determine the greedy policy and the other to decideon its value.

A Hierarchical Architecture for Decision-Making in Autonomous Driving using Deep RL

2. DeepCars Simulation Environment 2

3.1. Tabular Reinforcement Learning on DeepCars

As discussed before, we are planning to map a 3dimensional world into a much simpler 2D simulation environment and form the occupancy grid of the surrounding
actors around the ego vehicle. To make this possible, we
decided to design our own environment using the pygame(a free and open source python programming language library) to build our gaming environment. The source code isavailable in our GitHub repository (see footnote). We called
the environment DeepCars and a screen-shot from the gamescreen is shown in Fig.3.a.

As a starting point, we are going to implement the tabular
Q-learning algorithm (Fig.4) for the control of the agent in
DeepCars environment. The observed Markov state for theclassical RL is defined as the following:(5)
S = ego lane ID x0 x1 ... xn
in which the first element is the ego lane ID and xi indicatesthe distance to the closest car in lane i. Note that the lane
number and grid numbers start from 0. For instance thestate vector for the Fig.3.b is S = 2 6 3 8 0 8 .
Also note that the line-of-sight of the vehicle is 9 (from0 to 8). This means that the furthest car that the agent
can percept can be 9 grids (number of rectangles) away.
This constrain in fact simulates the sensor range that are
implemented on the vehicle.

Figure 4. Tabular Q-learning algorithm (Busoniu et al., 2017)
where x and u indicate the observed state and input respectively

Figure 3. A screen-shot from the DeepCars 2D environment andstate representation

The game receives high-level control commands as the input
vector and gives the game state and reward as the output. In
order to be able to formalize the problem in MDP to be usedin RL algorithm, the states and actions are discrete. In fact
the action space consist of three actions:A = lef t stay right(4)
switch to left/right lane or stay in the same lane. The observed Markov state for each RL is explained in corresponding sections.

The main objective is to train the agent to avoid
making collisions with other vehicles in the environment.
Thus, we define a simple reward function
+1 s0 6= sT
ρ(s, a, s0 ) =(6)−1 s0 = sT
Where sT indicates the terminal state that the agent makes
a collision. For the hyper-parameter tuning, we performeda simple grid search in the parameter space and found the
optimal setting:• discount factor: γ = 0.9• learning rate: α = 0.1

3. Results
We have implemented a tabular Q-learning, DQN, andDouble-DQN (DDQN) algorithms in DeepCars and compared the results. We will see that DDQN will demonstratebetter results in terms of the performance-training speed
trade-off in this problem.
2

https://github.com/MajidMoghadam2006/gym-deepcars• exploration:  = 0.2• exploitation: 1 −  = 0.8
We have trained the agent for 50000 steps. The resulting
Q-table were recorded in order to be used in the evaluation
phase. We evaluated the agent performance for 100’000
steps and recorded the results. In order to comment on

A Hierarchical Architecture for Decision-Making in Autonomous Driving using Deep RL

the performance of the agent we define the accuracy as theconsistent comparison metric between algorithms:
Accuracy =# of passed cars
× 100 (7)# of all cars (passed + collided)

The results were promising though imperfect. In fact for the
training and evaluation phases, the results were as follows:• Training: Accuracy = 98.28%• Evaluation: Accuracy = 99.14%
This shows that after 50000 frames the agent has learned
to control the vehicle by avoiding other vehicles. However,this is not the ideal performance, because there are a numberof collisions in the test set which are not appealing. In fact
our aim is to design an agent that can avoid collisions for
all cases. If we take a look at the recorded Q-table after
the training is finished in Fig.5 we can see that most of therows are still zeros which indicates a sparse Q-table that is
expected in tabular Q-learning. This situation arises because
not all of the state-action combinations were experienced
in the training phase. And, as the agent is not using any
function approximator, the similar situations cannot be approximated and collisions are unavoidable. We expect toovercome this problem by approximating Q-values using a
deep neural network architecture which indicates the DQN
algorithm. Note that the evaluation performance is better
than training because in evaluation phase we use the best
trained model. In addition, the actions are greedy in evaluation phase in contrast with −greedy strategy in training
phase.

In addition, we are planning to feed the occupancy gridof the environment into the agent which makes the useof classical RL impossible for this case. This leads us to
use the DQN and DDQN algorithms. The source code for
DQN and DDQN algorithms are provided in our GitHub
repository (see footnote). Here, we modified the original
DQN algorithm and added the real-time validation phase.
This helps us to perform the evaluation of the most recent
model during the training and select the best trained modelby far. This is possible by evaluating the latest model on
a number of game episodes, say 100, and recording theagent’s performance. Finally, we record the model that had
the best performance at the end of training.
Note that in DQN the observed MDP state is the occupancy grid while in tabular Q-learning, in order to reduce
the state space dimension, we defined the game state as ineq. 5. The occupancy grid incorporates the actors’ position
in a simple matrix with the same size as the environment
grid (Fig.3.a). Empty grids are filled with zeros, and ones
indicate the occupied grid with an actor. The flattened
version of this matrix forms the state vector. Finally, wefuse the ego lane number as a binary code of the lane ID
to create the MDP state. Including ego lane ID as a binarycode, enables us to have consistent binary values in the statevector, though it increases the observation space dimension.
In addition, in order to have a smooth graph for reward values we keep track of the mean accumulated rewards every
100 episodes as the performance metric. We implemented
our algorithm in a PC equipped with CPU: Intel Core i9
2.60GHz and GPU: GeForce GTX 1080 Ti. We trained theagent for 500’000 steps for three different architectures
for the neural network; shallow, medium, and deep. Eachdiffers in the number of layers of the neural network. We
have used {32} for shallow, {32, 64, 32} for medium, and{64, 128, 128, 64} for deep network architectures as thenumber of layers. The rest of the hyper-parameters wereequal for all cases. The performance comparison of these
networks are summarized in Fig.6. Obviously, the shallow

Figure 5. A crop of sparse Q-table in tabular RL (# of lanes: 3).
Redrectangle indicates the greedy optimal action where s =
1 6 3 03.2. Deep Reinforcement Learning on DeepCars3
As discussed in previous section, the tabular Q-learning
approach lacks the generalization property. In addition, thecourse of dimensionality is another problem while using
Q-learning techniques. In our case, the number of Markov
states ascend exponentially as the number of lanes increases.
3
https://github.com/MajidMoghadam2006/deepcarsreinforcement-learning

Figure 6. DQN performance for three network architectures

A Hierarchical Architecture for Decision-Making in Autonomous Driving using Deep RL

network has shown a better performance comparing with
two others. But the best comparison would be made in th
evaluation phase where we save the trained models andevaluate them in consistent sets of evaluation episodes
and keep track of the number of collisions. As shallow
network promised a better performance, we use the samearchitecture to compare DQN and DDQN. However, we
adjust the number of layers in network at the same timeto search for the best option. The performance of these
configurations in training phase are represented in Fig.7.
Obviously, the deep version of DDQN achieved a higher
mean 100 episode rewards
300

250

200

150

100

50

solve each layer separately with an admissible reliability
score.

4. Conclusion
In this study, we addressed the problem of autonomous driving in the highway scenario. We considered the hierarchicalarchitecture of the ADAS systems and focused on the decision making layer, where the sequential high-level decisioncommand are being made using a deep reinforcement learning algorithm. The agent receives the occupancy grid of theenvironment as the state representation and produces lanechange commands in order to avoid making collisions withother vehicles. The environment is stochastic in terms of thetraffic density and lane position of the other actors whichchallenges the agent’s performance and at the same time,
imitates the fusion layer output that may occur in actual
test scenarios. We have shown that it’s possible to leverage
the provided multi-layer architecture to generate high-levelcommands using DRL with an acceptable reliability score.

0
0

50000

100000150000200000250000300000

DDQN-16

DDQN-16x16

350000400000450000steps
DQN-16

DQN-16x16

Figure 7. DQN vs. DDQN with different network architectures.
Here 16x16 indicates two dense layers with 16 neurons at each.

reward peak value comparing with others, while shallow
DDQN converged much faster than its rivals. DQN has
shown a mediocre performance between these agents.
Following the real-time validation policy in algorithm
1, we recorded the best model in all training phases andevaluated their performance for 100’000 steps where
all of the agents achieved 100% accuracy (eq. 7). Thismeans that all agents are thoroughly trained and succeed toperform the evaluation phase without any collisions. This
accomplishment ascertains the generalization performanceof the DQN comparing with tabular Q-learning in previous
section. Among deep approaches, DDQN-16 has shown thebest performance-speed trade-off. Intuitively, the advanced
RL algorithm (DDQN) is much faster than its predecessor(DQN) which was expected. Also, a shallow networkwith one hidden layer and 16 neurons seems a reasonablechoice for this setting where the size of the occupancy gridas the input is 8 × 5 = 40. A rule of thumb to choose
the number of neurons in the hidden layer is to select anumber between the input and output vector sizes. These
facts may justify the choice of DDQN-16 agent as thebest performance-speed RL algorithm for the DeepCars
simulation environment.
Achieving 100% evaluation accuracy for the unseen
scenario in training phase strengthens the assertion that it’s
possible to decompose the autonomous driving problem
into a hierarchical architecture and leverage the AI power toFuture WorkWe are planning to implement the multi-layer architecturein Fig.2 and tackle each layer separately using the same
approach that we used in this study. We will exploit DeepCars as the 2D environmental model of the ego surrounding.
We summarized this scheme in Fig.1. We are also planning
to use Unreal Engine powered simulations like CARLA(Dosovitskiy et al., 2017) or Microsoft AirSim (Shah et al.,2018) to generate the occupancy grid of the ego surrounding from the ground truth information that we receive fromthe simulator. This way, we may bypass the fusion layer
in ADAS (Fig.2) and study the presented hierarchical approach (Fig.1) performance in more realistic and complex
situations. In addition, we will compare the original DQN
with its most recent extensions like dueling double DQN(Wang et al., 2015), prioritized experience replay (Schaul
et al., 2015), and rainbow agent (Hessel et al., 2018) which
is a combination of most recent improvements of the RL
in recent years. We will also perform a hyper-parameter
optimization to search for the best parameter setting in theparameter space.

A Hierarchical Architecture for Decision-Making in Autonomous Driving using Deep RL

References

through deep reinforcement learning. Nature, 518(7540):
529, 2015.

Alizadeh, A., Moghadam, M., Bicer, Y., Ure, N. K., Yavas,
U., and Kurtulus, C. Tactical lane changing with deep
reinforcement learning in dynamic and uncertain traffic
scenarios. In 22nd Intelligent Transportation Systems
Conference (ITSC2019-submitted), 2019.

Moghadam, M. and Caliskan, F. Actuator and sensor fault
detection and diagnosis of quadrotor based on two-stage
kalman filter. In 2015 5th Australian Control Conference(AUCC), pp. 182–187. IEEE, 2015.

Bicer, Y., Moghadam, M., Sahin, C., Eroglu, B., and Üre,N. K. Vision-based uav guidance for autonomous landing
with deep neural networks. In AIAA Scitech 2019 Forum,
pp. 0140, 2019.

Moghadam, M., Ure, N. K., and Inalhan, G. Autonomous
execution of aircraft supermaneuvers with switching nonlinear backstepping control. In 2018 AIAA Guidance,
Navigation, and Control Conference, pp. 1594, 2018.

Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B.,
Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller,
U., Zhang, J., et al. End to end learning for self-drivingcars. arXiv preprint arXiv:1604.07316, 2016.

Perot, E., Jaritz, M., Toromanoff, M., and De Charette,
R. End-to-end driving in a realistic racing game withdeep reinforcement learning. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition
Workshops, pp. 3–4, 2017.

Broggi, A., Bertozzi, M., Fascioli, A., Bianco, C. G. L.,
and Piazzi, A. The argo autonomous vehicles visionand control systems. International Journal of Intelligent
Control and Systems, 3(4):409–441, 1999.
Busoniu, L., Babuska, R., De Schutter, B., and Ernst, D.
Reinforcement learning and dynamic programming using
function approximators. CRC press, 2017.
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., andKoltun, V. Carla: An open urban driving simulator. arXiv
preprint arXiv:1711.03938, 2017.
Falcone, P., Borrelli, F., Asgari, J., Tseng, H. E., and Hrovat,
D. Predictive active steering control for autonomousvehicle systems. IEEE Transactions on control systems
technology, 15(3):566–580, 2007.
Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., andSilver, D. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conferenceon Artificial Intelligence, 2018.
Koutnı́k, J., Cuccu, G., Schmidhuber, J., and Gomez, F.
Evolving large-scale neural networks for vision-based
torcs. 2013.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet
classification with deep convolutional neural networks.
In Advances in neural information processing systems,
pp. 1097–1105, 2012.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez,
T., Tassa, Y., Silver, D., and Wierstra, D. Continuous
control with deep reinforcement learning. arXiv preprint
arXiv:1509.02971, 2015.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness,
J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control

Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. arXiv preprint arXiv:1511.05952,
2015.
Shah, S., Dey, D., Lovett, C., and Kapoor, A. Airsim: Highfidelity visual and physical simulation for autonomousvehicles. In Field and service robotics, pp. 621–635.
Springer, 2018.
Slotine, J.-J. E., Li, W., et al. Applied nonlinear control,
volume 199. Prentice hall Englewood Cliffs, NJ, 1991.
Specht, D. F. A general regression neural network. IEEE
transactions on neural networks, 2(6):568–576, 1991.
Van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double q-learning. In Thirtieth AAAI
Conference on Artificial Intelligence, 2016.
Wang, Z., Schaul, T., Hessel, M., Van Hasselt, H., Lanctot, M., and De Freitas, N. Dueling network architectures for deep reinforcement learning. arXiv preprint
arXiv:1511.06581, 2015.