# Deep Reinforcement Learning Based Resource Management in UAV-Assisted IoT Networks
## 5 Proposed Method Description
### Multi-Agent DRL for the Joint Resource Management Problem  
According to Figure 5, a multi-agent DRL is used to build a joint strategy for resource management. It is because the cumulative reward values of one user equipment are certainly influenced by other user equipment's actions in the resource management process. Hence, the initial state and UAV to user actions affect the value of the reward. Therefore, Equations (9)-(19) illustrate the relationship between the state, action, and reward for a multi-agent DRL-based scheme. 1. State: The A2G channel used by IoT users, the consumption of power rate, and the effects of interference are considered as a state. Then, the values of each state \((S_{t})\) with the state-action pairs at a time slot as \(Q_{t}(S_{t},a_{t})\) for the next state values as \(Q_{t}(S_{t}^{\prime},a_{t})\). Thus, for Equation (9), \(S_{t}\) and \(S_{t}^{\prime}\) are the values of a state and the next state,  
Figure 5: An illustration of multi-agent framework for proposed DRL-based RM algorithm. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Resources** & **Duration** & **Queue** & **Arrival Time** \\ \hline R1 & 3 & 0 & 0 \\ R2 & 4 & 1 & 0 \\ R3 & 3 & 2 & 0 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Illustration of record of resources with a round robin scheduling-algorithm. respectively. Moreover, \(r_{t}\), is the value of the reward; \(a_{t}\) is the value of action at a time slot; \(\gamma\) is used to represent and control the changing situations. The value of \(\pi_{t}^{*}\) represents the policy. Next, the agent will have access to the next state of \(S_{t}^{\prime}\). \[Value(S_{t}\ )=maxa_{t}(r_{t}(S_{t},a_{t})\ \gamma\sum S_{t}^{\prime}P\big{(}S_{t },a_{t},S_{t}^{\prime}\big{)}, \tag{9}\]  
\[Q_{t}(S_{t}\,a_{t})=r_{t}\big{(}Q_{t}(S_{t},a_{t})+\sum S_{t}^{\prime}\ \pi_{t}^{*} \big{(}S_{t},a_{t},S_{t}^{\prime}\big{)}V\text{alue}\big{(}S_{t}^{\prime} \big{)}\big{)}Q_{t}, \tag{10}\]  
Then, the expected values of the next state, as in Equation (10), where \(Q_{t}\) is the target value; \((S_{t}^{\prime})\) used for the next state and action as in \(Q_{t}(S_{t}^{\prime},a_{t}{}^{\prime})\). Then, the time slot calculation for the state to action communicates, as in Equations (11) and (12). Then, the time slot (\(T_{s}\)) is calculated as in Equation (13):  
\[Q_{t}(S_{t}\,a_{t})=r_{t}(S_{t}\,a_{t})+\sum S_{t}^{\prime}\ \pi_{t}^{*} \big{(}S_{t},a_{t},S_{t}^{\prime}\big{)}maxa_{t}{}^{\prime}Q_{t}\big{(}S_{t}^{ \prime},a_{t}{}^{\prime}\big{)}), \tag{11}\]  
\[Q_{t}(S_{t},a_{t})=r_{t}(S_{t},a_{t})+\gamma maxa_{t}{}^{\prime}Q_{t}\big{(}S_ {t}^{\prime},a_{t}{}^{\prime}\big{)}, \tag{12}\]  
\[T_{s}(a_{t},\ S_{t}\ )=r_{t}(S_{t},\ a_{t})+\gamma maxa_{t}{}^{\prime}Q\big{(}S_{t} ^{\prime},a_{t}\big{)}-Q_{t}-S_{t}\,a_{t}), \tag{13}\]
2. Joint action selection strategy: The allocation of resources such as channel allocation of bandwidth, throughput, and power is the action taken by the agent. This is used to equate action1 with next action vs. action1 to target1, and action n to target n. Where, to control the randomness of the environment, add the time slot as in Equation (14). Based on Equation (13) of \(Q_{t}-1(S_{t},a_{t})\) and the time slot of \(Q_{t}(S_{t}\,a_{t})\). Then, as in Equation (15), this shows how our target (\(Q_{t}\)) values are learned from the state and updated over time with \(T_{s}(a_{t},S_{t})\) as in Equation (16): \[Q_{t}(S_{t},a_{t})=\ Q_{t}-1(S_{t}\,a_{t})+a_{t}\ T_{st}(a_{t},\ S_{t}\ )\] (14) \[Q_{t}(S_{t},a_{t})=Q_{t}-1(S_{t},a_{t})+a_{t}(r_{t}(S_{t},a_{t})+\gamma max \ a_{t}{}^{\prime}Q_{t}(S_{t}^{\prime},a_{t}{}^{\prime})-Q_{t}-1(S_{t},a_{t}))\] (15) \[T_{s}(a_{t},S_{t}\ )=r_{t}(S_{t}\,a_{t})+\gamma max\ a_{t}{}^{\prime}Q_{t} \big{(}S_{t}^{\prime},a_{t}{}^{\prime}\big{)}-Q_{t}-1(S_{t},a_{t})\] (16)  
Next, the loss function is computed as in Equation (17), with the sum of the Q-values and their target differences as follows:  
\[Loss=\sum\big{(}Q_{t}-Target-Q_{t}\big{)}^{2} \tag{17}\]
3.