# Reinforcement Learning Based Resource Management for Network Slicing
## 4 Resource Trading System Using a Markov Decision Process
### State Space  
In our system, the tenant plays a role as an agent that interacts with the environment and exhibits strategic behavior to maximize its reward. The state space at time slot \(t\) can be defined as  
\[s_{t}=[\varphi,D_{n,t},\rho,\hat{t}]\in S, \tag{4}\]  
Figure 2: QoS with respect to the resource demand.  
where \(\varphi\), \(D_{n,t},\rho\), and \(\hat{t}\) are the flow ratio of the slice, total resource demand, current market price ratio, and time zone index of the current time slot, respectively. The information of these four state elements is considered to be known to the tenant at each time slot \(t\).  
To characterize the type of services supported by each slice, we define the flow ratio, denoted as \(\varphi\), which is computed by dividing the number of inelastic flows by the total number of flows as follows:  
\[\varphi=\frac{|I|}{|I|+|E|}, \tag{5}\]  
where \(I\) and \(E\) denote the set of inelastic and elastic flows, respectively. We expect \(\varphi\) to be used as a parameter to determine how much bandwidth the tenant purchases in advance. It is a very important approach for our dynamic resource trading system because the system will violate QoS and get a penalty if there is not enough immediate bandwidth. The effect of \(\varphi\) on the system performance is analyzed in detail in Section 6.  
Let \(D_{n,t}^{i}\) be the sum of the minimum demands of all inelastic flows belonging to slice \(n\) at time slot \(t\) and \(D_{n,t}^{e}\) denote the sum of the average demands of all elastic flows served by slice \(n\) at time slot \(t\), respectively, as follows:  
\[D_{n,t}^{i}=\sum_{i\in I}d_{n}^{min},\ D_{n,t}^{e}=\sum_{e\in E}d_{e}^{avg}. \tag{6}\]  
Then, we can define the total resource demand at time slot \(t\), which is the sum of the minimum demands of all inelastic flows and average demands of all elastic flows, as follows:  
\[D_{n,t}=D_{n,t}^{i}+D_{n,t}^{e}. \tag{7}\]  
If the total bandwidth demand exceeds the total purchased bandwidth at time slot \(t\), i.e., \(D_{n,t}>R_{n,t}\), resource utilization is 1 because the used bandwidth cannot exceed the total purchased bandwidth.  
The price ratio, denoted as \(\rho\), represents the ratio of the unit price of the bandwidth that tenants purchase from the provider to the unit price of the bandwidth that they sell to the end-user, i.e., \(\rho=p^{buy}/p^{sell}\). \(\rho\) is included in the state since it highly affects the profit of the slice tenant. We assume that \(p^{buy}\) is determined by the provider according to the amount of the total resource remained in the physical network. For example, if the amount of remaining resource in the physical network is decreased, the provider increases \(p^{buy}\). This implies that \(p^{buy}\) can play an important role in the resource balancing in the physical network through regulating the amount of resource being purchased by adjusting \(p^{buy}\).  
Time slot needs to be included in the state because the tenant may act differently at different time slots. As an effort to avoid a situation that the number of states goes infinitely high, we apply the concept of the time zone; i.e., we use the index of the time zone rather than the index of the actual time slot. Let \(\tau\) and \(T\) denote the number of time zones in the slice and the lifetime of the slice, respectively. Then, the time zone index of the current time slot can be defined as follows:  
\[\hat{t}=\left\lceil\frac{t\cdot\tau}{T}\right\rceil. \tag{8}\]