# Holistic Utility Satisfaction in Cloud Data Centre Network Using Reinforcement Learning
## 4 Reinforcement-Learning-Based Cloud Resource Allocation
### Q-Learning Model  
In principle, in Q-learning, as a typical reinforcement learning method, the main objective is that a learning agent finds an optimum strategy for maximising a weighted and cumulative reward function by some repeated actions imposed on an unknown system [63; 64]. In Q-learning, four essential elements exist, which are the \(agent,state,action\), and \(reward\), respectively. In Figure 4, the interactions between the agent (CDCN manager or CP in this case) and the environment (cloud CDCN system) are depicted. In each period/epoch, the Q-learning algorithm executes itself in the following manner. First, the learning agent observes the state \(\mathbf{S}_{t}^{\ell}\) of the environment at iteration \(\ell\) of time \(t\). Then, the learning agent chooses a corresponding action \(\mathbf{Z}_{t}^{\ell}\) among all possible actions. The environment gives a corresponding reward \(\mathbf{R}_{t}^{\ell}\) for the action \(\mathbf{Z}_{t}^{\ell}\). The learning agent works in two distinct phases, which are \(exploration\) and \(exploitation\). In the exploitation phase, the learning agent selects an action with the largest reward value based on the past experiences. In the exploration phase, a random action in action set \(\mathbf{Z}\) is normally selected. The ultimate goal is to identify the optimal action-state pair that can maximise the long-term reward. Hereafter, four elements of the proposed Q-learning methodology in the cloud DCN environment are described in more detail. **State and environment:** In the cloud data centre network system, the environment is composed of different virtualised data centres with the associated resource pools and cloud users. The state is a vector that consists of the current available and free virtualised resources (CPU core, RAM, storage capacity, and connection bandwidth) of each data centre. The state can be represented as follows:  
\(\mathbf{S}_{t}^{\ell}\) is a \(1\times(6M(t))\) vector and is defined as \(\mathbf{S}_{t}^{\ell}=[\mathcal{C}_{\ell}^{(1)}(t),\mathcal{D}_{\ell}^{(1)}(t),\mathcal{G}_{\ell}^{(1)}(t),\mathcal{B}_{\ell}^{(1)}(t),\)  
\(\mathbf{E}[\mathcal{L}_{\ell}^{(1)}(t)],\mathbf{E}[\mathcal{H}_{\ell}^{(1)}(t )],\cdots,\mathcal{C}_{\ell}^{(M(t))}(t),\mathcal{D}_{\ell}^{(M(t))}(t), \mathcal{G}_{\ell}^{(M(t))}(t),\mathcal{B}_{\ell}^{(M(t))}(t),\mathbf{E}[ \mathcal{L}_{\ell}^{(M(t))}(t)],\)  
\(\mathbf{E}[\mathcal{H}_{\ell}^{(M(t))}(t)]\), in iteration \(\ell\) in time slot \(t\). Figure 4: Mapping the elements of the proposed Q-learning algorithm to the CDCN. Based on Equation (10), we also have the following four capacity constraints:  
\[\sum_{k=1}^{M(t)}\mathcal{C}_{\ell}^{(k)}(t)\leq C^{CP}(t),(\mathbf{ C}_{1}); \sum_{k=1}^{M(t)}\mathcal{D}_{\ell}^{(k)}(t)\leq D^{CP}(t),(\mathbf{C}_{2})\] \[\sum_{k=1}^{M(t)}\mathcal{G}_{\ell}^{(k)}(t)\leq G^{CP}(t),( \mathbf{C}_{3}); \sum_{k=1}^{M(t)}\mathcal{B}_{\ell}^{(k)}(t)\leq B^{CP}(t),(\mathbf{C}_{4}) \tag{10}\]  
where \(C^{CP}(t),D^{CP}(t)\), \(G^{CP}(t)\), and \(B^{CP}(t)\) are the existing total CPU amount, storage capacity, RAM, and connection bandwidth for the cloud provider for each time frame \(t\). The state space \(\mathcal{S}(t)\) is a convex polytope, as described in Equation (11):  
\[\mathcal{S}(t)=\prod_{k=1}^{6\times M(t)}[0,U_{k}(t)]\bigcap_{j=1}^{4} \mathbf{C}_{j} \tag{11}\]  
where the upper limit for each state variable \(k\) is denoted by \(U_{k}(t)\) for each time frame \(t\) and \(\mathbf{C}_{1},\cdots,\mathbf{C}_{4}\) are the corresponding constraint sets in Equation (10), respectively. It must be mentioned that the state expression in Equation (11) has the potential for increasing the number of states exponentially. In addition, in a real data centre setting, we have four variables, which are the VM-based variables (CPU, RAM, storage) and connection bandwidth. Furthermore, two other independent variables are associated with each data centre user (expected RTT and expected packet loss), which results in six variable for each user.