# AI-Based Resource Allocation in E2E Network Slicing with Both Public and Non-Public Slices
## 3 Algorithm Design
### The RB Allocation Problem (30)  
We first model the problem (30) as a Markov Decision Process (MDP). Then, we employ the multi-agent reinforcement learning IPPO algorithm to solve the MDP.  
MDP Formulation for RB Allocation  
The MDP of the problem is given by the triplet \((\mathcal{S},\mathcal{A},\mathcal{R})\), where \(\mathcal{S}\) represents the state space, \(\mathcal{A}\) denotes the action space, and \(\mathcal{R}\) is the reward function. In this paper, we decide to use multi-agent reinforcement learning in order to reduce the size of the action space. The details of the MDP are defined as follows:  
**Agent:**: slice \(s,\ \forall s\in\mathcal{S}\)  
**State space \(\mathcal{S}\)**: For agent \(s\), the state space \(\mathcal{S}^{s}\) is given using the following tuple:  
\[\mathcal{S}^{s}=(\mathcal{U}_{s},G_{s},\mathcal{Q}_{s},N_{s}) \tag{32}\]where \(\mathcal{U}_{s}\) represents the set of users in slice \(s\), \(G_{s}\) denotes the set of channel gain of users in slice \(s\), \(\mathcal{Q}_{s}\) represents the PRU in slice \(s\), and \(N_{s}\) denotes the set of RBs assigned to each user in slice \(s\). Thus, the whole state space can be expressed as:  
\[\mathcal{S}=\{\mathcal{S}^{1},\mathcal{S}^{2},\dots,\mathcal{S}^{S}\} \tag{33}\]  
**Action space \(\mathcal{A}\) :** For agent \(s\), the action space \(\mathcal{A}^{s}\) is defined as follows:  
\[\mathcal{A}^{s}=\{0,1\}^{K\times[\mathcal{U}_{s}]} \tag{34}\]  
where \(K\) denotes the total number of RBs and \(\mathcal{U}_{s}\) denotes the set of users in the slice \(s\). The above action space \(\mathcal{A}^{s}\) represents the set of specific schemes for the resource allocation, which can be represented as a row vector:  
\[\mathcal{A}^{s}=\{x_{s,1}^{1},\dots,x_{s,|\mathcal{U}_{s}|}^{1},x_{s,1}^{2}, \dots,x_{s,|\mathcal{U}_{s}|}^{2},\dots,x_{s,1}^{K},\dots,x_{s,|\mathcal{U}_{ s}|}^{K}\}\]  
where \(x_{s,u}^{k}\) is defined in (1). Each slice is responsible for the RB assignment for all the users within the slice. Thus, the whole action space can be expressed as the Cartesian product of all \(\mathcal{A}^{s}\), i.e.,  
\[\mathcal{A}=\mathcal{A}^{1}\times\mathcal{A}^{2}\times\dots\times\mathcal{A} ^{S}\]  
**Reward function \(\mathcal{R}\) :**  
Each action corresponds to an RB allocation scheme. When the RB allocation scheme satisfies all the constraints of the problem (30), the reward function is related to the corresponding profit, which is the optimization objective of (30). Otherwise, a negative reward is given. More specifically,  
\[\mathcal{R}=\begin{cases}e^{(\psi-\bar{\psi})/10},&\text{If all constraints are met}\\ -1,&\text{Otherwise}\end{cases} \tag{35}\]  
where \(\psi\) represents the achieved profit of the RB allocation scheme, which is given in (19), and \(\bar{\psi}\) is the approximation of the average between the minimum and maximum profit over all RB allocation schemes. A relatively accurate approximation will aid in the convergence of the reinforcement learning.  
It should be noted that, to check whether all constraints are met in (35), the RF algorithm introduced in Section 3.1 is used to determine whether the last constraint of problem (30) is satisfied. The remaining constraints of the problem (30) can be easily judged using the given RB allocation scheme.  
Agent \(s\) observes its own current state \(\mathcal{S}^{s}\) and chooses an action \(a^{s}\) from the action space \(\mathcal{A}^{s}\). The actions of all slices form a joint action \(a=\{a^{1},a^{2},\dots,a^{S}\}\). The joint action updates the current state to the next state and receives the reward \(\mathcal{R}\). The reward will be given to all agents as feedback for the agents to keep adjusting their actions. The whole MDP process is shown in Figure 4.