## 2 Related Work
### DQN Network Architecture  
The recommended network model architecture incorporates reinforcement learning and a recurrent neural network method. The recurrent network is effective for applying sequential circumstances and for forecasting environmental target attributes. Figure 3, given below, illustrates the network structure of the learning procedure. The DQN network model processing steps with targeted action and state values of the learning procedure are depicted in the diagram below. The taken action and state values will be stored for the next step of training operations as an initial input value. Additionally, preserving learned information allows drone agents to avoid unnecessarily surveying the same location twice. This action may appear to be a repeated procedure, but in a virtual environment, the action and state value will be identical to the preceding one. Figure 3 depicts a portion of Q-learning for updating the samples or minibatches for a given iteration. In this process, the updated Q-value determines the agent's actions, where the action with the highest targeted Q-learning value is chosen, and the update is accomplished through the use of a Bellman Equation (2) defined as  
\[\hat{Q}(s_{t},a_{t})\leftarrow\underbrace{Q(s_{t},a_{t})}_{old~{}value}+ \underbrace{\alpha}_{learning~{}rate}\cdot\underbrace{\underbrace{r_{t}}_{reward}+\underbrace{\gamma}_{ discount~{}factor}\cdot\underbrace{\max_{a}Q(s_{t+1},a)}_{optimum~{}future~{}value~{}estimation}-\underbrace{Q(s_{t},a_{t})}_{old~{}value}}_{intended~{}new~{}value} \tag{2}\]  
where \(\hat{Q}(s_{t},a_{t})\) is the result of the updated Q-network iteration value; \(Q(s_{t},~{}a_{t})\) in which the initial old value will be added; \(\alpha\) is the learning rate of the training network; \(r_{t}\) is the reward value; in this equation \(\gamma\) relates the rewards to the time domain; and \(\underset{a}{\max}Q(s_{t+1},a)\) is the optimum future value estimation. In the equation above, the calculation of the updated Q-Network iteration value has been presented. An updated iteration value can be formulated by adding the previous \(Q(s_{t},~{}a_{t})\) value and multiplying the learning rate with \(\alpha\), which represents the difference in computation time. The max in this equation represents the maximized actions that help agents to take in the upper arcs of the VCE itself. Suppose, our drone agent was in state \(s\) and it took some action \(a\). Because of that action, the environment might land our drone agent in any of the states \(s_{t+1}\), and from these states, it would maximize the action. From these values, the drone agent will choose the action with the maximum \(Q\) value: \(\underset{a}{\max}Q(s_{t+1},a)\). Intended new values can be calculated by multiplying the discount factor \(\gamma\) with the maximum \(Q\) value and adding to them the reward \(r_{t}\) value. In our DQN-network concept, the DQN agent contains a replay memory class unit that keeps track of the environment in dynamic mode. All state (\(s_{t}\)), action (\(a_{t}\)), new state (\(s_{t+1}\)),  
Figure 3: DQN network structure with targeted Q-learning outcomes in operation with states. reward (\(r_{t}\)), and done transitions are memorized. This replay memory approach allows us to effectively sample minibatches from preserved values and generate the accurate state representation. In order to obtain the best results from memorized values and keep track dynamically, a responding buffer memory is integrated with the VCE simulation platform. Figure 4 illustrates the construction of the replay memory unit, where the storage monitoring process is shown. The buffer memory is directly connected to the virtual environment, which adds the required transition to the memory unit. During the sampling process, a random number of map indices of varying sizes are produced in memory, and the returned indices may be retrieved using the \"get state\" function of the AirSim python client. Minibatches will be generated by using the last saved values from the training process.