# Holistic Utility Satisfaction in Cloud Data Centre Network Using Reinforcement Learning
## 4 Reinforcement-Learning-Based Cloud Resource Allocation
### The Proposed QLRA algorithm  
In this subsection, we introduce the proposed Q-learning-based resource allocation (QLRA) algorithm based on the Q-learning methodology. As we described earlier, in the Q-learning algorithm, two distinct exploration and exploitation phases exist. To avoid local optima, the Q-learning method enters the exploration phase by a pre-defined strategy. Different strategies exist, which allows the Q-learning algorithm to switch between these two phases. Among these strategies, the \(\epsilon\)-greedy policy is the one that is most frequently used [65]. In this policy, a parameter \(\epsilon(0\leq\epsilon\leq 1)\) is used to trim the exploration part. In other words, in the current training epoch, the agent explores a randomly chosen action with probability \(\epsilon\) and, then, exploits this selected action with probability \((1-\epsilon)\)[63]. To balance exploration and exploitation, we used a function introduced by the authors in [64]. This function enforces a high probability of exploration for the beginning of training and a high probability of exploitation in the last stages of the training periods, as described in Equation (12):  
\[\epsilon=\frac{0.5}{1+\text{exp}\big{(}\frac{10\times(\ell-0.4\times L)}{L} \big{)}} \tag{12}\]  
in which \(L\) is the total number of training episodes and \(\ell\) represents the current and remaining number of training episodes. The Q-value\(/\)-score is normally initialised to 0 at the very beginning of the learning procedure. Then, the Q-value is updated according to the Q-learning update rule given by Equation (13). The Q-value is used for evaluating the performance of the specific actions and states. \[Q(\mathbf{S}_{t}^{\ell},\mathbf{Z}_{t}^{\ell})\gets Q(\mathbf{S}_{t}^{ \ell},\mathbf{Z}_{t}^{\ell})+\gamma(\ell)\bigg{(}\mathbf{R}_{t}^{\ell}+\zeta \max_{Z^{\ell}}Q(\mathbf{S}_{t}^{\ell+1},\mathbf{Z}_{t}^{\ell})-Q(\mathbf{S}_ {t}^{\ell},\mathbf{Z}_{t}^{\ell})\bigg{)} \tag{13}\]  
where \(\mathbf{S}_{t}^{\ell}\) demonstrates the environment state in time slot \(t\) in training period\(/\)episode \(\ell\). \(\mathbf{Z}_{t}^{\ell}\) denotes the action of the agent in time slot \(t\) of training episode \(\ell\). \(Z^{\ell}\) means the action space. \(\zeta\) (\(0\leq\zeta\leq 1\)) is the so-called discount proportion. \(\gamma\) is the learning rate (\(0\leq\gamma(\ell)<1\)), and \(\mathbf{R}_{t}^{\ell}\) is the reward value in time slot \(t\) of training episode \(\ell\). \(\max_{Z^{\ell}}Q(\mathbf{S}_{t}^{\ell+1},\mathbf{Z}_{t}^{\ell})\) is the maximum reward that can be obtained from state \(\mathbf{S}_{t}^{\ell+1}\). Algorithm 1 represents the detailed QLRA algorithm. Figure 5: Two-dimensional representation of the state space and a sample action in iteration \(\ell\) of time slot \(t\). ```
1 Initialise \(Q\)-score and time slot, \(Q=0\), \(t=0\)
2\(t=t+1\)
3 Generate state set \(\mathcal{S}(t)\)
4 for\(\ell=1:L\)do:
5 Generate action set \(\mathcal{Z}^{\ell}(t)\) and \(\epsilon\) according to [64]:
6\(\epsilon=\frac{0.5}{1+exp^{(\frac{10.5(t-0.4\cdot\ell))}{L}}}\)
7 Choose an action according to:
8 \[\mathbf{Z}_{t}^{\ell}=\left\{ \begin{array}{ll}\mathbf{Z}_{t}^{\ell}\in\max_{ \mathcal{Z}^{\ell}}Q(\mathbf{S}_{t}^{\ell},\mathbf{Z}_{t}^{\ell})&\text{with probability}(1-\epsilon)\\ \text{a uniformly random action in }\mathcal{Z}^{\ell}&\text{with probability}( \epsilon)\end{array}