## 5 DRL-Based Online Resource Allocation
### Mechanism of DRL
#### 5.2.2 State Space  
In addition to the action space, another essential issue in DRL is constructing appropriate state space, which can be divided into the following two steps, choosing the related information and constructing these elements to feature the input of the DNN. Based on the analysis in Section 3, in the S-IoT scenario, the state \(s_{t}\) should include the information of channel quality, current arrival terminal \(u_{k,t}\), and the terminal set \(\mathcal{U}_{t}^{On}\), which contains terminals that have been allocated resource and are still transmitting data to the satellite.  
The information of the current arrival request \(u_{k,t}\) consists of its location \(ord_{k}\), QoS requirement \(Q_{k_{t}}\), and its maximum transmission power \(P_{k}\), which can be denoted as \(\inf_{Q_{k_{t}}}=\left\{ord_{k},P_{k},Q_{k_{t}}\right\}\).  
For terminals that are still transmitting data to the satellite, namely \(k_{t}^{On}\in\mathcal{U}_{t}^{On}\), the involved information includes their location \(ord_{k_{t}^{On}}\), QoS requirement \(Q_{k_{t}^{On}}\), and the occupied channel \(c_{k_{t}^{On}}\) and transmission power \(p_{k_{t}^{On}}\). Therefore, for services \(k_{t}^{On}\in\mathcal{U}_{t}^{On}\), the state information is denoted as \(\inf_{Q_{k}^{On}}=\left\{ord_{k_{t}^{On}},c_{k_{t}^{On}},p_{k_{t}^{On}},Q_{k _{t}^{On}}\right\}\).  
Based on the above analysis, the state at time stick \(t\) can be expressed as \(s_{t}=\left\{CSI_{t},\inf_{\text{O}_{k_{t}}},\inf_{\text{O}_{t}^{\text{O}_{m}}}\right\}\). Note that although the agent's observation of the environment includes CSI, there can be noise and error without the assumption that global CSI is perfectly known.  
Then, to make the feature better observed by the DNN, we reformulate the elements of state information from structural representation to tensor manner, as shown in Algorithm 1.  
```
Divide all the in-service terminals into \(M\) groups as the interference merely occurred between co-channel terminals, as \(\mathcal{U}_{\text{cs}}^{t}\). For \(m=\)0, \(1,\ \ldots\ldots M\): Rank the terminals using channel \(c_{m}\) according to the distance between them and the current arrival request\({}^{\prime}\)s terminal \(u_{k}^{t}\).Get the top \(K_{N}\) nearest terminals as \(\mathcal{U}_{\text{cs}}^{t}\). If \(\mathcal{U}_{\text{cs}}^{t}\
eq\varnothing\):  For each terminal \(u_{i}\in\mathcal{U}_{\text{cs}_{i}}^{t}\):  Construct \(u_{j}\)'s contextual information \(D_{u_{j}-u_{k}^{t}},p_{j,S}^{t-1},D_{K_{t}}CsI_{j,S}^{t-1},C_{j}^{t-1},EE_{j}^{ t-1},Rc_{j}^{t},Rd_{j}^{t},Rr_{j}^{t}\) as feature vector \(s_{j}^{t}\), where \(Dis_{u_{j}-u_{k}^{t}}\) denotes the distance between \(u_{k}\) and \(u_{j},\ p_{j,S}\) denotes the transmissionpower of \(u_{j},\ D_{k}\)  denotes the data transmission amount of \(u_{j},\ \mathcal{C}_{\text{ss}}^{t-1}\) denotes the data transmission of last time stick, \(EE_{j}^{t-1}\)  denotes the energy efficiency of lasttime stick, \(RC_{j}^{t},\ Rd_{j}^{t}\), and \(Rr_{j}^{t}\) denotes the QoS requirement on transmission rate, delay and reliability of terminal \(u_{j}\) respectively.  Arrange the state information vector \(s_{j}^{t}\) of the \(K_{N}\) terminals as a matrix \(s_{\text{cs}_{\text{cs}}}^{t}\in\mathbb{R}^{K_{N}\times 9}\), which contains the contextual information on channel \(c_{m}\) at time stick t. Construct the state information matrix \(s_{\text{cs}_{\text{cs}}}^{t}\) of each channel to tensor manner and \(\text{gets}^{t}_{\text{Is}}\in\mathbb{R}^{K_{N}\times 9\times M}\) as the input of DQN
```  
**Algorithm 1**Formulate State Information  
Through the above process, the state information \(s_{t}=\left\{CSI_{t},\inf_{\text{O}_{k_{t}}},\inf_{\text{O}_{t}\mathcal{U}_{ \text{cs}}^{m}}\right\}\) is formulated as a \(K_{N}\times 9\times M\) tensor.