## 5 DRL-Based Online Resource Allocation
### Process of DRL-Based Online Resource Allocation
To solve this problem, a deep neural network is implemented as the mapping function from state to value \(V^{\pi}(s)\) or \(Q^{\pi}(s,a)\). RL methods that approximate value function through a deep neural network are called 'deep RL' (DRL), which are still based on the classical theory of RL, but facilitate deep learning to replace the original tabular value function estimation module. Deep Q-network (DQN) [24] is a popular method of DRL, which improves the Q-learning method by introducing a neural network to estimate the state action value \(Q^{\pi}(s,a)\) instead of a \(Q\)-value table. The deep neural network in DQN is called the \(Q\) network, whose parameter of neurons is denoted by \(\theta\). The input of the \(Q\) network is the state \(s_{t}\), while the output of the network is the \(Q\) value of each action under the state \(s_{t}\). More specifically, the \(Q\) value of each state-action pair \(\langle s_{t},a_{t}\rangle\) denotes the long-term reward of choosing the action \(a_{t}\) in state \(s_{t}\). The essential idea of DQN is the same with Q learning, which is to determine the \(Q\) value of each action under a specific state when the state transition probability \(P\) is unknown, so as to obtain the optimal decision. The optimal strategy is to choose the action with the largest \(Q\) value. \[\pi^{*}(s_{t})=\underset{a\in\mathcal{A}}{\operatorname{argmax}}Q^{*}(s_{t},a) \tag{18}\]  
With this strategy, the state value of \(s_{t}\) can be denoted as  
\[V^{\pi}(s_{t})=\max_{a\in\mathcal{A}}Q^{*}(s_{t},a) \tag{19}\]The optimal \(Q\) value can be obtained by iteratively calculating Equations (17) and (19). More specifically, the estimated value of \(Q\) network can be calculated by  
\[y_{t}=r_{t}+\gamma\max_{a_{t+1}\in\mathcal{A}}\hat{Q}\big{(}s_{t+1},a_{t+1}\big{|} \theta^{-}\big{)} \tag{20}\]  
Therefore, the difference between the estimated and prediction value of \(Q\) network can be denoted as  
\[L(\theta)=\mathrm{E}\Big{[}(y_{t}-Q(s_{t},a_{t}|\theta))^{2}\Big{]} \tag{21}\]  
This section adopts a gradient decent method to train the \(Q\) network, with \(L(\theta)\) employed as a loss function. The procedures of the DRL-based online resource allocation algorithm (DRL-QoS-RA) are elaborated in Algorithm 2. ```
Parameter initialization. Initialize network \(Q\) and \(Q^{\prime}\) with random weight. Initialize DRL training parameters, such as target network update step \(N_{Q^{\prime}}\) greedy exploration probability \(\varepsilon\), Buffer size \(N_{B}\), replay start size \(N_{IS}\), and batch size \(N_{BS}\). Initialize the S-IoT environment parameter. ``` For time_stick \(t=1,T\)do  Collect new data transmission request into set \(\mathcal{U}_{new}^{t}\)  Collect the suspended data transmission into set \(\mathcal{U}_{On}^{t}\) and release the resource occupied by them. If \(\mathcal{U}_{new}^{t}\
eq\varnothing\):  For each data transmission request \(u_{k}^{t}\),  Reformulate state information \(s_{t}=\left\{CSI_{t},\inf_{0k,i}\inf_{0k,i}\inf_{0k}\right\}\) into tensor manner\(Feature_{k,t}\). Calculate a candidate action set \(\Phi_{satisfy}^{t}\)  Adopt the tensor feature \(Feature_{k,t}\) as the input of the action-value network \(Q\) and obtain the output of network \(Q\) as \(\mathbf{v}_{a_{t}}=Q(s_{t}^{t}|\theta)\)  With probability \(1-\varepsilon\) to decide \(a_{t}=\mathrm{argmax}_{a_{t}}\), otherwise randomly choose \(a_{t}\). If \(a\in\Phi_{satisfy}^{t}\) : allocate resource for \(u_{k}^{t}\) according to \(a_{t}\). Collect environmental information \(s_{t+1}\), then formulate to \(Feature_{k,t+1}\) and calculate \(r_{t}\)  Store this experience \(\big{(}Feature_{k,t},a_{t},r_{t},Feature_{k,t+1}\big{)}\) into experience pool \(\mathcal{D}\)  If the sample of \(\mathcal{D}\) is more than this, replay start size \(N_{IS}\):  Sample \(N_{BS}\) experience \(\big{(}Feature_{k,t},a_{t},r_{t},Feature_{k,t+1}\big{)}\) from \(\mathcal{D}\)  Calculate \(y_{t}\) and \(L(\theta)\) by Equation (20) and Equation (21) respectively.