## 4 Problem Transformation and DRL-Based Solution  
The key challenge for solving the problem \(\mathbf{P}\) is that \(q(\lambda,\mathbf{w},\delta,\mathbf{g})\) cannot be expressed by a closed-form mathematical model [26]. The difficulties in solving the problem \(\mathbf{P}\) are reflected in the following aspects. * _Diverse traffic models and QoS requirements:_ Different slices have different traffic models and different QoS requirements. The resource allocation policy of a single slice cannot be directly applied to other slices. * _Highly dynamic environment with user mobility:_ Due to the varying network environment caused by the mobility of UEs and the uncertainty of task arrival, it is difficult to use concrete mathematical methods to allocate resources quickly. * _Markovian characteristics of resource slicing:_ The inter-slice resource allocation exhibits Markovian characteristics, where problems whose probability of the problem entering the next state depends only on the current state and action selected. In this paper, the allocation policy affects not only the current QoS performance and slice load balance, but also the future network state and utility, e.g., the queue of packets in the buffer. Such dynamic characteristics make DRL naturally become a good choice for this problem-solving. Reinforcement learning is a class of machine learning techniques where an agent is trained for decision-making mechanisms by interacting with an environment through action, with the objective of maximizing its cumulative reward. The environment is in a certain state and is changed to the next state by an action from the agent, and the agent receives the corresponding reward for that transition [27]. Therefore, we proposed a well-designed DDPG-Agent to solve the problem. We denote the state, action, state change, and return of the system as the MDP architecture. Given the system state \(s_{t}=s\) and the action \(a_{t}=a\) at time \(t\), the next state changes to \(s_{t+1}=s^{\prime}\) with transition probabilities \(p^{a}_{ss^{\prime}}\) as follows,  
\[p^{a}_{ss^{\prime}}=P\{s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a\}, \tag{13}\]  
where \(p^{a}_{ss^{\prime}}\) is influenced by the uncertainty of the previous state, the UEs' mobility, the traffic demands, and the performed action, while part of this, called prior knowledge, is unknown. A well-designed DRL-based solution of DDPG is proposed to solve the problem dynamically. DDPG can be used to maximize the throughput in wireless communication with a large dimensional continuous action space [28]. It is assumed that the DDPG-Agent is deployed at the BS and the BS can perfectly provide the required state of the system. The design of the DDPG-Agent mainly includes the architecture of the neural network, the observation of the system, and the reward function, as follows. **State:** The current system state is determined by the serving state of the BS. The proposed DDPG-Agent in this paper takes action at the slice level; therefore, the DDPG-Agent needs to collect information about the state at the slice level and generate resource allocation decisions. The state is defined as \(s=Q\cup E\cup A\). \(Q=\{q_{m}^{d}|m\in\mathcal{M},d\in\mathcal{D}\}\) represents a tuple of QoS satisfaction rate of each slice. \(E=\{\eta_{m}^{d,nr}|m\in\mathcal{M},d\in\mathcal{D}\}\cup\{\eta^{sul}\}\) represents a tuple of the load of each slice, as well as the SUL band. \(A=\{\alpha_{m}^{d}|m\in\mathcal{M},d\in\mathcal{D}\}\) represents a tuple of the weight of each slice. **Action:** The action is defined as a tuple \(a=\{\delta,\lambda,\mathbf{w}\}\). The well-designed architecture of the actor network and the critic network are shown in Figure 3. \(N_{state}/N_{action}/N_{experience}\) represents the dimension of the state/action/experience. The action mapping for taking the output of the actor network as the input of the BS includes action exploring with \(e\)-greedy and action clipping. To avoid severe performance degradation due to some extreme actions, we clip the actions to within reasonable limits.