## 4 Throughput Optimization Strategy of Hybrid Service
### Throughput Optimization Algorithm for Hybrid Service  
In reinforcement learning, each agent chooses a strategy \(\pi\) that maximizes the cumulative reward. Among them, the strategy \(\pi\) refers to the probability distribution of the agent mapped to the action \(a\) when the agent is in the current state \(s\). The cumulative discount function is usually used to represent the expected return in the strategy \(\pi\), calculated as follows:  
\[V_{\pi}(s,a)=\sum_{t}\xi_{t}r_{t}(s_{t},a_{t}) \tag{10}\]  
where \(\xi_{t}\) is the impairment rate. When \(\xi_{t}\) tends to 1, it means that the agents in the network will obtain greater cumulative returns, and the transmission rate of the mMTC link will be improved. Therefore, when the remaining capacity in the communication link is not zero, the value is increased during the training process, encouraging each agent to access as many mMTC services as possible. Therefore, our model aims to find an optimization strategy, and its definition is as follows:  
\[V_{\pi}^{*}(s,a)=\max_{\pi}V_{\pi}(s,a) \tag{11}\]  
When the model finds \(V_{\pi}^{*}(s,a)\), it means that the optimization strategy in the current state can be obtained as follows:  
\[\pi^{*}(s_{t})=\arg\max_{a_{t}\in A}\sum_{t}r_{t}(s_{t},a_{t})+\sum_{s_{t+1}}P( s_{t+1}|s_{t},a_{t})V_{\pi}^{*}(s_{t+1},a_{t+1}) \tag{12}\]  
To find the optimal strategy \(V_{\pi}^{*}(s,a)\), an iterative algorithm can be used. However, in the actual environment, it is difficult to obtain the state transition probability BB due to the lack of prior knowledge \(P(s_{t+1}|s_{t},a_{t})\). Therefore, we utilize the Deep Q-Network to deal with the lack of experience in unknown environments. Specifically, each agent has one DQN and takes the state space \(S\) as input. Then, it outputs the value function corresponding to all actions \(A\). The DQN is trained through multiple iterations, and, in each iteration step, all agents adopt some soft policy to select the action with the largest estimate in the state-action space with probability \(1-\varepsilon\) and random action with probability \(\varepsilon\)[23]. The channel state and environment change when an agent takes an action. Each agent will collect and store the current state-action space, reward value, and state space in the experience pool \(L\) for the next step. In each iteration, part of the information will be extracted from the experience pool to update the parameter \(\theta\) in the stochastic gradient descent method. Then, a fixed set of parameters will be obtained to reduce the error value. When the impairment factor is \(\gamma\), the error function is calculated as follows:  
\[\psi(\theta_{t})=\left[r_{t+1}(s_{t},a_{t})+\gamma\max_{a\in A}Q_{t}(s_{t+1},a _{t+1},\theta_{t}^{{}^{\prime}})-Q_{t}(s_{t},a_{t},\theta_{t})\right]^{2} \tag{13}\]  
In summary, the proposed HSCRS algorithm involves building each AIoT communication link into an agent, exploring the reward value of different strategies in unknown environments during iterative training, and learning to make up for any lack of experience. This is illustrated in Algorithm 1.  
```
0: DQN Architecture, Environment Model, Parameters of Agents.
0: The trained DQN model.
1:for each iteration do
2: Initialize all agent parameters, \(Q(s,a)\), \(\pi(s,a)\), \(\theta\), \(L\)
3:for each step \(t\) in each iteration do
4: Each agent observes its state \(s_{t}\)
5: Select a random action \(a_{t}\) with probability \(\varepsilon\)
6: Execute the action and obtain the reward value according to (9), then get the new state \(s_{t+1}\)
7: Save information to experience pool \(L\)
8:endfor
9:for each agent do
10: Pull part of the information from the experience pool
11: Update the error function according to (13)
12:endfor
13:endfor
```  
**Algorithm 1** HSCRS.