## 4 Throughput Optimization Strategy of Hybrid Service  
### Multi-Agent Modeling in Hybrid Service  
In a hybrid service coexistence system, both mMTC and URLLC services can be transmitted within the same bandwidth. However, due to the different transmission power and delay requirements of these services, the resource scheduling problem for different services can be modeled as a multi-agent reinforcement learning problem [23]. Reinforcement learning can be used to iteratively find better resource scheduling strategies in different environments and meet the transmission requirements of more services as much as possible. Each communication link between the URLLC service and the mMTC service is considered an agent. By collecting experience from different decisions made in various environments, an appropriate scheduling strategy can be selected for execution in the current environment state. When multiple services need to be transmitted simultaneously, multiple communication links form a multi-agent cluster to jointly explore the sub-bandwidth scheduling strategy and power control strategy in the current environment. In the multi-agent reinforcement learning model, as shown in Figure 2, the decision-making process of each agent can be defined as \((S,A,P,r,d)\), where \(S\) is the state space set. \(A\) denotes the action space set. \(P\) represents the state transition probability, which means that in the current state \(s_{t}\), if the action \(a_{t}\) is taken, the probability of obtaining a new state is \(s_{t+1}\). \(r\) is the reward value, because it is necessary to connect as many mMCs as possible while trying to complete the URLLC service. \(d\) is the loss factor. It can obtain the corresponding reward value after each agent takes an action, then update the reinforcement learning model and gradually find better decisions. When the training of the model is completed, each agent selects an action based on the historical experience obtained from the trained model that can bring greater benefits. The details of the multi-agent model in the hybrid system of URLLC and mMTC are shown in Figure 2.  
**Agent**: Each communication link in the network. **State**: The network performance parameters mainly considered in this scenario include the status of sub-channels, remaining capacity of communication links, service tolerance for delay, and channel conditions in the sub-bandwidth. Therefore, the network state can be defined as \(s=\{s_{state},s_{capacity},s_{time},s_{SNR}\}\in S\). \(s_{state}\) is the sub-bandwidth occupation state. \(s_{capacity}\) denotes the remaining capacity of the bandwidth. \(s_{time}\) represents the service delay tolerance. \(s_{SNR}\) is the channel signal-to-noise ratio. **Action**: For the resource scheduling problem in the mixed service system, each agent can decide to utilize any sub-bandwidth for transmission. For the downlink transmission power of F-AP, we define the transmission power control of the mMTC service as \(P_{mMTC}=\{50,100,150,200\}\) mW. Therefore, the dimension of the action space is \(4\times H\), and each action corresponds to a sub-bandwidth and transmission power. When the number of communication links is large, the overall action space will become larger. **Reward**: To optimize the resource scheduling in the network, a reward function must be established to obtain the optimal strategy. Each agent can adjust its scheduling strategy by receiving reward values from each decision, allowing it to approach the optimal decision. The goal of this chapter is to enhance the network throughput performance and the transmission success rate of mMTC services in mixed service scenarios within a given time interval. To achieve this, network performance parameters, such as the sub-channel status, remaining capacity of communication links, service tolerance delay, and channel conditions in the sub-bandwidth are taken into consideration. Therefore, the reward function for the URLLC link in each iteration can be calculated by \(\sum\limits_{h\in H}C_{URLLC}\).