# A Survey on Applications of Reinforcement Learning in Flying Ad-Hoc Networks
## 2 Fundamentals of Deep Reinforcement Learning
### Deep Reinforcement Learning  
The \(Q\)-learning algorithm is efficient in terms of its comparatively small action and state space. However, the system becomes more complicated for large action and state space. In this situation, the \(Q\)-learning algorithm may not be able to achieve an optimal policy owing to the complex and large \(Q\)-table. To overcome this problem, researchers replaced the \(Q\)-table with a deep neural network (DNN) and named it deep \(Q\)-learning (DQL) [15]. DQL is a deep reinforcement learning (DRL) that works with \(Q\)-values similar to \(Q\)-learning, except for the \(Q\)-table part as shown in Figure 3.  
The main goal of the DNN is to skip manual calculations each time by learning from the data. A DNN is a computational nonlinear model like the structure of the human brain, which can learn and perform tasks such as decision-making, prediction, classification, and visualization [20]. It is composed of neurons arranged in multiple layers. It typically has one input layer, two hidden layers, and on output layer, interconnected as depicted in Figure 4 [21]. The input layer accepts the inputs with the input neurons and sends them to the hidden layers. The hidden layer then sends the data to the output layer. Every neuron has a weighted input, an activation function, and an output. The activation function determines the output depending on the input of the neuron [22]. It acts as a trigger that depends on the weighted input.  
During the training phase, the weighted values of the inputs of the neurons are updated based on the outputs of the output layer using backpropagation by the agent. The agent takes the output of the policy DNN and compares it with a target DNN model and calculates error [23]. Then the agent updates the policy DNN using backpropagation. This process is generally referred to as optimization with gradient descent. After a certain time, the agent updates the target DNN using policy DNN. For a more stable  
Figure 4: Deep neural network.  
Figure 3: Simple deep \(Q\)-learning.  
convergence of the optimal policy, experience replay memory (ERM) is introduced into the DQL framework [24; 25]. The agent takes different actions and saves the present states, obtained rewards, next states, and actions taken in ERM [24; 25]. Then, the agent takes a mini-batch of data from the ERM and trains the policy DNN. Figure 5 and Algorithm 2 illustrate the framework and flow of the DQN better [26]. Thus, the agent can make decisions efficiently and in a timely manner using the learned DNN.  
```
0: Initialize policy and target DQL network with random \(w\) and \(w^{\prime}\), respectively.
0: Initialize experience replay memory (ERM).
0: Initialize \(\epsilon\). for\(t=1,2,\dots,T\)do  Select an action \(a_{t}\) for present state \(s_{t}\) based on probability \(\epsilon\).  Observe the immediate reward \(r_{t}\) and next state \(s_{t+1}\).  Insert \((s_{t},a_{t},r_{t},s_{t+1})\) in ERM.  Create a mini-batch with random sample of \((s_{t},a_{t},r_{t},s_{t+1})\) from ERM.  Optimize the weights \(w\) of the policy DNN with gradient descent via MDP. \(w^{\prime}\gets w\) after certain number of time steps. end
```  
**Algorithm 2** The Deep \(Q\)-learning Algorithm.