# Holistic Utility Satisfaction in Cloud Data Centre Network Using Reinforcement Learning
## 2 Related Work
In [3], the authors used the cooperative co-evolution evolutionary algorithm, which is a multi-objective optimisation methodology, for solving the joint user-provider utility satisfaction problem in cloud data centre networks. Multi-objective optimisation methods have been used by researchers such as [49] and [50] for addressing some cloud challenges such as high power consumption, virtual machine consolidation, and under-utilisation of resources. In [51], [PERSON] et al. used reinforcement learning techniques for automating energy-efficient virtual machine consolidation in cloud data centres. In [52], [PERSON] et al. proposed a time-driven data placement strategy for a scientific workflow combining edge computing and cloud computing. In [53], [PERSON] et al. developed a shallow neural network with kernel approximation for prediction problems in data communication networks. Policy gradient methods, proximal policy optimisations, and imitation learning have also been used for cloud resource management [54; 55; 56]. In [54], the authors used reinforcement learning heuristics for efficiently scheduling data processing jobs on distributed compute clusters, which require complex algorithms. In [55], the authors proposed a framework for optimising the trade-off between QoS and energy consumption using reinforcement learning approaches. [PERSON] et al. in [56] used a convolutional neural network to capture the cloud resource management model and utilise imitation learning in the reinforcement process to reduce the training time of the optimal policy. In contrast with all of the mentioned work, in the current paper, we have a holistic view of the cloud resource allocation optimisation problem by taking into account the benefits of cloud users (in terms of required cloud resources, expected network packet loss, and expected round-trip time) and the cloud provider (in terms of allocated resource prices and energy consumption costs) simultaneously. On the other hand, the Q-learning methodology was selected in the current work because it is an online and low-complexity algorithm, which can be tailoredin a proper manner to tackle the time-varying aspects of the resource allocation in cloud data centre network systems. In comparison with [3], in the current work, a more accurate user utility model was developed. This model incorporates the stochastic nature of packet loss and round-trip time (RTT) in calculating cloud user utility functions. Moreover, in all of the above papers (except [3]), cloud resource management was deployed by different strategies in order to satisfy multiple objectives, which target the cloud user side utility and cloud provider side utility in a non-holistic manner. In the current work, a joint resource scheduling strategy was developed that can enhance the resource usage in order to simultaneously satisfy the CDCN ecosystem and, in parallel, can consider energy saving constraints. In doing so, we selected online ML techniques in the form of RL-based optimisation. An agent (cloud provider) interacts with the environment (cloud resources and users) in an action-reward form in order to converge to an optimal resource assignment strategy in each time slot. According to our best knowledge, this is the first time that RL techniques have been employed in the resource allocation of cloud data centre networks for joint cloud provider/cloud user satisfaction considering energy constraints. In Table 1, we classify the mentioned cloud resource allocation strategies into different categories.