# Holistic Utility Satisfaction in Cloud Data Centre Network Using Reinforcement Learning
###### Abstract
RL is a machine learning sub-type, which is model-free. It uses some agents, which interact with an unknown environment. The objective of this type of algorithm is to optimise the cumulative and long-term reward [9]. In RL methods, given the input data, the algorithm learns to take actions that maximise a cumulative reward [10; 11]. All these types of learning empower the systems with the required intelligence to enhance their future performance based on current information/data. Communication networks may use ML for different use cases [15]. The major reasons for using ML are creating the required robustness or adaptivity in confronting the changing and normally unpredictable user, traffic, and network conditions [16; 17]. Based on [18], virtualisation technology is one of the most-critical elements in cloud computing, which allows a physical resource to be virtually shared amongst different network users. Virtualisation can speed up the IT operations, and by optimising the use of the networking infrastructure, it reduces the operational costs. In a cloud computing system, the job scheduling is an essential part. The job scheduling system allocates the required resources (in terms of virtual machines) to dynamically arriving workflows [19]. Normally, the jobs are performed in a priority-based manner. Jobs with a higher priority are scheduled to be performed prior to those with a lower priority level. Most of the cloud providers consider only the cloud provider benefits and constraints in satisfying the user SLAs [20]. As the users' demands are different, cloud providers normally use some admission control policies to allocate resources to higher-priority users. This makes the lower-priority users unsatisfied, and this leads to customer churn [21]. Cloud providers must try to guarantee quality of service (QoS) parameters (such as transfer delay, response time, etc.) for satisfying their cloud users [22; 23]. Cloud-based resource assignment in cloud computing has been addressed in many previous works such as [24; 25; 26; 27; 28; 29; 30]. In contrast to the work presented [30; 31; 32], in this paper, we focus on the problem of simultaneous SLA satisfaction for the whole CDCN ecosystem (including customers and provider). As different players may use different cloud-based applications (with different QoS requirements) and finding a solution strategy for joint SLA satisfaction for all of these users is very difficult due to the non-linear and time-dependent nature of the resulting utility functions, we used an intelligent reinforcement learning approach [33] for tackling the resulting complex and non-linear resource allocation/assignment problem. The most-important contributions of the current work are as follows:  
* Utility modelling was adopted for the overall CDCN system, in terms of virtualising the networking resources, which are: CPU, RAM, storage, and access/connection bandwidth. * The energy consumption model of the cloud data centres is described and used in the utility modelling of the cloud manager. * After modelling the utility/satisfaction functions associated with the cloud users/provider, a reinforcement learning sub-type (which is Q-learning) was adopted for optimised resource assignment to different cloud users, which also simultaneously satisfies the requirements of the CDCN manager/provider in terms of energy efficiency in a holistic manner. * Finally, the online and model-free property of the Q-learning algorithm results in converging to optimal utility levels for both the cloud users and cloud provider in different cloud user population scenarios in a fast and low-complexity manner. In this paper, an RL-based optimisation approach based on the Q-learning algorithm [34] was developed. The cloud provider of the CDNC system must deploy this algorithm and, based on that, assign optimal VM resources (CPU, storage, and RAM) and, also, the connection bandwidth to active cloud customers [33; 35] in order to simultaneously optimise cloud user/provider utilities. The rest of the paper is organised as follows.