A high penetration of renewable energy sources such as wind power generation and photo- voltaic generation causes some problems in power systems such as the duck curve and unreliability due to environmental variability. An e ective solution to this problem is Demand Response (DR). Electric Water Heaters (EWHs) are considered ideal candidates for DR due to their energy storage capability. Due to the bene ts, control strategies or techniques for EWHs have received considerable academic at- tention. The energy sector has recently tapped into the disruptive arti cial intelligence world to learn, among other related priorities, how to enhance operations, maintain energy resilience and improve con- sumer service. Consequently, this paper reviews the use of machine learning (ML) for optimization and scheduling of EWHs. The main contributions of this review paper are, rstly, to identify state of the art of energy optimization and scheduling of EWHs. Secondly, to review the current ML models for energy optimization and scheduling of EWHs in smart grids and smart building environment. While classical control strategies may deliver substantial improvements, optimum eciency may not be reached. ML has demonstrated clear advantages over classical control. Based on these conclusions, recommendations for further research topics are drawn. Keywords| electric water heaters, machine learning, supervised learning, unsupervised learning, reinforce- ment learning, renewable energy 1 Introduction Climate change and energy security are the pressing issues globally today. The buildings are one of the largest users of electricity. The sector accounts for more than 40% of the global energy use and accounts for 30% of greenhouse gases (GHG) emissions [1]. The e ort to minimize GHG emissions involves a substantial improvement in human activity in energy use, the manufacture of more environmentally sustainable goods, and the identi - cation and reduction of the sources of these undesirable emissions [2]. The convergence of ICT and Internet of Things (IoT) concepts is moving today's cities into the idea of smart cities, helping them to conserve energy and becoming more energy ecient [3]. The current energy mix in the developing countries is still dependent on conventional fossil fuels which accounts to more than 70% of the energy supply [4]. In 2014 Brazil, Russia, India, China and South Africa ∗Corresponding author 1 together were reported to contribute 38% of the total carbon emissions [5]. Moreover, renewable energy can signi cantly reduce CO2 emissions by replacing carbon-intensive energy sources [6]. The share of renewable energy accounts for 26% of global electricity generation in 2019. It also predicted that in 2050, under optimal scenario, 63% of the global electricity generation will come from renewable energy [7]. At the same time, high penetration of renewable energy brings about challenges in the electric power system because the availability of the energy is variable, and may therefore not align well with demand. For instance, there can be cloudy, rainy and windless days and this will cause a rapid change in frequency. The volatility and low generating inertia of these energy resources cause imbalance in the power system that can negatively a ect the stability the electric grid [8]. To maintain stability and reliability of power systems with high penetration of renewable energy resources the balance between generation and demand must be consistently maintained. Issues relating to the dynamics and control of electrical power systems with integration of renewable power are discussed in ref [9{14]. A promising solution to deal with these challenges is DR. In times of high stress on the electrical system, utilities have been utilizing conventional DR programmes for years to send information to customers to minimize energy usage. A new generation of communication and control technology will now facilitate "demand exibility" by allowing pre-selected loads to react continuously or regularly to changes in the level of electricity supply and other market signals. Energy for water heating is the largest source of households energy use [15]. In Europe, EWHs accounts for about 87 TWh of electric energy in 43.5 million households [16]. Water heating load accounted for 17% of energy consumption in European households in 2012 [17]. Furthermore, in China 15%{20% of the total residential electric energy usage for an average family is used for water heating [18]. Due to their energy storage characteristics, EWHs are considered ideal candidates for DR [19]. EWHs can therefore be used for frequency regulation and power balancing [20][21][22]. Consequently, it is valuable to predict energy usage of EWHs, and subsequently hot water usage, in residential sector to optimize their energy usage to achieve energy conservation to reduce environmental impacts and for power system reliability. Control strategies or techniques for EWHs have received considerable academic attention in recent years. Ripple relays have been used to control EWHs in South Africa for more than half a century. However, ripple relay technology uses a unidirectional controller with no consideration for the comfort of the user or customer. The energy sector has recently taken advantage of the creative eld of Arti cial Intelligence (AI) to learn, among other related objectives, how to improve processes, ensure energy resilience and enhance consumer service. One of the key requirements to incorporate AI in domestic electric appliances is Demand Response (DR). In short, DR can be de ned as the behaviour of individual electricity customers that reduce or adjust their usage of electricity over a given period of time, typically at peak hours, in response to a price signal or a grid reliability [23]. 1.1 Motivation of this review In the past, engineering and statistical methods were predominantly used to predict energy demand for energy optimization. The engineering methods, also known as white-box methods, use physical principles to predict energy behaviour [24]. These methods need detailed parameters and depend on complex physical principles to predict or forecast energy consumption. However, these parameters are not always available. Statistical methods, also known as black-box methods, on the other hand, establishes the relationship between energy usage and in uencing variables on the historical data through the empirical models. For non-linear relationships, statistical methods are appropriate and prevent the errors of engineering methods [25]. For example, Lomet et al. used a statistical model to develop an auto regressive moving average (ARMA) model to forecast the daily domestic hot water consumption. In [26], Fisher et al. proposed a stochastic behavioural model for domestic hot water (DHW) and space heating demands. A correlation of 92% and a mean relative error of 3% of the daily load pro le for DHW usage were recorded by the author against the calculated data for German 2 single-family houses. Rouleau et al.[27] developed a uni ed probabilistic model that forecasts the numbers of people in many households, the use of DHW and non-HVAC electricity usage in multiple residences. The authors recorded 97% of the DHW usage pro les and 92% of the electricity usage pro les as a correlation between the simulated and calculated results. Using consumption forecasting through statistical modelling, Denis et al. [28] evaluated the energy savings that can be achieved in DHW output. The authors reported the energy savings of 3.6% to 12.8%. Nevertheless, statistical methods are constrained by the fact that they require data, large amounts of high-quality historical data that need to be gathered over a long period and that large computing memory needs to be taken into account. To overcome the limitations of each previous method Machine Learning (ML) method was proposed and devel- oped. In recent years, numerous ML techniques have been proposed for predicting building energy consumption and to control heating, ventilation and air conditioning (HVAC) systems [29][30][31][32][33]. Nevertheless, the utilization of ML methods for energy conservation of thermostatically controlled loads (TCLs) remains in an infancy stage. We expect that ML methods could have a great contribution in this eld. 1.2 Organization The rest of this review report is organized as follows: Section 2 reviews the current ML models for energy usage optimization and scheduling of EWHs. Section 3 introduces the concept of ML and its models. Section 4 provides the novelty and the review approach. Section 5 highlights research and development challenges. Section 6 presents a summary of lessons learned and provides direction for future research. 2 State of the art of electric water heaters control techniques Over the recent years, a number of scienti c research has covered di erent DR strategies for EWHs. Control strategies can enable signi cant energy eciency improvement of the hot water production systems and generate cost savings [17][34]. In this section, two classes of control strategies for EWHs are discussed, namely, rule-based and model-based. 2.1 Rule-based control Rule-based (RB) control strategies use pre-de ned conditions to change the current state of a system. Concerning water-heating systems, these pre-de ned conditions can be the temperature of water inside the tank and the users' comfort. This control technique involves, in particular, state machine control, threshold control and power monitoring to ensure that the key components function in the most ecient environment [35]. This technique is bene cial because of its simplicity of execution and the reliability of temperature control [36]. This type of control can be classi ed according to exible objectives, namely, energy reduction, energy cost reduction, load shifting, and provision of ancillary services [37]. Application of RB control techniques on EWH are reviewed in this subsection. 2.1.1 Energy and cost reduction Using EWHs and binary particle swarm optimization, an optimum load demand plan approach was proposed by Sepulveda et al. [38] to minimize the peak load demand while optimizing the degree of consumer comfort. A 1-node model has been used to formulate the optimization by minimizing the peak load while maximizing the heaters temperature to ensure user comfort. However, in 1-node model the temperature of the EWH can be modelled precisely as long as no large water is drawn because large water draws cause strati cation between the warm body of water and the large volume of cold water entering the tank, which is not modelled by the 1-node 3 model [39]. Booysen et al. [40] used a 2-node hybrid model for horizontal EWHs to verify savings achieved in lab and eld experiments by scheduled operation. The scheduled control of EWHs reduced energy usage by 29% compared to thermostat control. However, this model is only valid for horizontally mounted EWHs. Laurent et al. [41] proposed an optimization based approach to produce heating schedules. The 1-node model was used to formulate a linear optimization problem for peak load minimization. The method of generation of columns was used to divide the problem into independent parts. This model is expensive to compute. In 1-node model the temperature of the EWH can be modelled precisely as long as no large water is drawn. Zimmerman et al. [42] proposed a linear programming model under which the client-side agent applies cost management techniques within the real-time pricing scheme. A range of acceptable water temperatures was proposed to de ne users' comfort. However, the model presumed that the inlet temperature is constant and tank heat loss per hour are set. Similarly, Mueller et al. [43] proposed a linear optimization scheduling technique based on real-time pricing. The authors successfully implemented an operation of a group of ve EWHs by ripple control in a eld test. In response to the time-of-use price, Shah et al. [44] proposed an optimization schedule of EWHs to reduce the electrical cost. However, their algorithm is complicated and complex, and could result in incorrect calculations of standby losses. Similarly, Safouri and Kapsalis [45] developed a heuristic algorithm that sets out the EWHs. To reduce energy costs without substantially reducing the perceived degree of comfort, the proposed algorithm takes into account the comfort needs of the consumer and forecasts hot water consumption and electricity prices. Unlike the model presented in [44], the bene t of this algorithm is that it is less complex. Moreover, to address the issue of complexity, Atikol [46] proposed a less complex approach for DSM scheduling technique of EWHs for regions with weak technical infrastructures. He analysed lab results on the thermal behaviour of EWHs and concluded the need to use accurately designed switching programs for successful reduction of peak load at high system eciency. Goh and Apt [47] recommended three separate methods aimed at generating optimum electricity cost reductions under dynamic pricing. The goal of these methods is to adjust the temperature between the minimum and the highest value, based on the price of energy. This technique does not consider the comfort factor and, thus, does not always guarantee an appropriate water tank temperature. In [48], Kapsalis and Hadellis suggested a dynamic pricing approach to set up Dijkstra's algorithm [49], which deals with the scheduling of EWH operations using a day-to-day real-time pricing tari . In their analysis, Dijkstra's algorithm is used to identify control behavior that resulted in savings of between 23% and 29%. The objective function aims to optimise the energy cost and the user's comfort. Vanthournout et al. [50] considered DLC based on the idea of an accurate estimate of the state of charge of the EWHs. A state of charge model is used to formulate the scheduling algorithm. This work formulates the DR problem by using a multi-node model to incorporate strati cation within the tank. Du and Lu [51] proposed a model for the lowest cost scheduling for a thermostatically controlled appliance, according to customer comfort settings. The authors have analysed the in uence of shifts in the de ned hot-water temperature levels on the optimum EWH schedule. Their model also take into account the uncertainties in price forecasting and hot water usage. However, the article does not explain how the expected demand for water is modi ed [52]. 2.1.2 Load shifting Lu and Katipamula [53] proposed strategies for power management of EWH loads, based on shifting energy usage from the high-cost to the low-cost period to reduce the peak-load and electricity cost, while maintaining tank water temperature beyond minimum level. The focus of their load shifting algorithm did not include analysis of the grid capacity. Wang et al. [54] proposed an optimization approach to solve the uncertainties in two fundamental factors, namely hot water demand and ambient temperature, for residential EWHs load scheduling. Due to high expense of commercial optimizer it makes it impossible to apply the approach in low-cost embedded controllers. Belov et al. [55] developed a two-stage deterministic optimization method of EWHs in a double-price 4 tari , in which both the energy-comfort and expense-comfort issues are considered. The authors analyzed the impact of economic bene ts on implementing EWHs load shifting. Ericson [56] investigated the e ects of direct load control (DLC) on 475 Norwegian households. His interpretation of the data found that shifting the load created a new, higher peak load. Klaasen et al. [57] have used a binary integer optimization to calculate the optimal scheduling of 30 000 water heaters regarding total load variations. By comparing results with classical xed schedules, the potential for dynamic scheduling was successfully demonstrated. Nehrir et al. [58] introduced a fuzzy logic method that can shift the domestic EWH load from peak to o -peak hours. This approach require a complex modelling process and signi cantly a ect the temperature of the EWHs output water, resulting in customer discomfort. 2.1.3 Provision of regulation services Diao et al. [59] proposed a centralized control for frequency regulation using EWHs. The evaluation was based on a grid simulation comprising multiple EWHs using 2-mass composite models. In the case of a hot water drawdown, the model is based on converting from a 1-node model to a 2-node model, resulting in a partial depletion stage of the tank. This model is only true for vertically aligned EWHs because it does not take into account the cross-sectional region of the EWH. The model accuracy is not validated with measured data. Kondoh et al. [60] proposed a control algorithm of EWH systems to regulate the aggregated power usage to stabilize a network supplied with renewable resources. The author used an additional thermostat in the control circuit of a EWH design with two heating elements to provide regulation service. A 2-node model has been used to simulate the uni- and bi-directional regulation signals. However, the cost of bidirectional devices with communication hardware may prevent implementation of this proposed method. Diduch et al. [61] proposed a system for clustering EWHs for aggregation in a virtual power plant to deliver ancillary services. A forecasted day-to-day model is used and control commands are transmitted based on the expected results. However, owing to an inaccurate predictive rate of 33.3%, this would restrict its utility in providing reliable ancillary services. Similarly, Gela zanskas and Gamage [62] proposed a method for scheduling residential EWHs to mitigate the inaccuracy in day-to-day wind generation prediction. Every 5 minutes, the control system schedules heating times for the next 12 hours to shift the demand to balance the supply. This study aimed to balance wind energy production with multiple EWH energy usage. The user's comfort was not taken into account with the optimization of EWH operation. Tammam et al. [63] proposed a multistage stochastic optimization model for the reduction of EWH loads in the presence of renewable energy connected to the grid. This model computes the optimal day-ahead energy usage of aggregated EWHs under various levels of wind power production. The authors claim that their day-ahead forecasted model is accurate and does facilitate the integration of renewable energy into the power system. Despite these claims, their model is expensive to implement. Nehrir et al. [64] implemented a load model that is appropriate for voltage regulation, reducing the power grid peak load while maintaining the user comfort. Authors nd that the peak of EWHs demand and the overall peak of residential demand are highly correlated. However, the modelling process was not described in detail. Similarly, Malik and Havel [65] developed a centralized DLC of EWHs to facilitate the load under high PV penetration in the Czech electricity market, to mitigate the imbalances between generation and demand within the controlled area. The authors showed that their proposed technique improved voltage pro le and reduced energy loss. 2.2 Model-based predictive control Model-based control strategies rely on a model of the system to project its behaviour in the future. This type of control strategy is mostly used to solve optimization problems [66]. There are several phases in the 5 implementation of mode based control, this includes the selection of accurate models, the estimation of model parameters, estimation of the state of the system, and the prediction of independent variables [67]. Compared to rule-based controls, model-based controls are more complex. These types of control methods have been applied to EWHs for di erent problems using di erent approaches. Applications of model-based control strategies are discussed in this subsection. Booysen et al. [68] suggested three distinct types of methods to control the EWHs. The three model-based controls decide the optimal schedule and the best temperature set point in the storage tank to accomplish speci c objectives. These objectives involve matching the delivery temperature, matching the delivered energy in the hot water, and ensuring Legionella sterility. The study evaluates 30 water heaters for 20 days. When the results were evaluated, the temperature matching method resulted in an energy saving of 7.9%, the energy matching method gave an energy saving of 17.8% and energy matching for daily Legionella sterilization gave an energy saving of 13.1%. Ritchie et al. [69] performed a similar study using predicted usage pro les for 77 water heaters in which strati cation was taken in place and found savings ranging from 2.2% and 9.6%. Kepplinger et al. [70] proposed the use of a dynamic programming strategy for optimizing the cost and energy usage of EWHs. The approach required an hourly optimization of EWHs to be implemented. This method has been demonstrated to be an alternative grid-balancing solution for domestic EWHs. Lin et al. [71] analyzed the minimization of the energy cost of the water heater thus satisfying the convenience needs of hot water consumers utilizing neural networks. A nonlinear autoregressive network with external input (NARX) using a neural network was used to achieve the comfort requirement with minimum energy usage. Bomela et al. [72] conducted a similar study where a general thermostatically controlled loads (TCL) model- based control was proposed to develop a continuous oscillator model of a TCL and compute its phase response to changes in temperature and applied power. In [73], Vanhoudt et al. applied a model-based control algorithm to maximize the pro t of a small scale district heating system heated by a combined heat and power plant (CHP). The authors reported that the presented control algorithm in uence the heat demand pro le of the connected buildings. Wei et al. [74] proposed a model-based real-time two-stage optimization model for multiple TCL groups to smooth the power uctuation in the distribution network. The authors showed that the proposed approach e ectively decrease the net exchange power uctuation as well as regulation costs. Zhang et al. [75] applied a Model Predictive Control (MPC) strategy to derive optimal switching times regarding costs and user comfort. The algorithm uses a nonlinear least-squares formulation. This model assumes the physical properties of the tank, which in most existing systems is typically an unreal assumption. Sossan et al. [76] proposed MPC technique for optimizing PV self-usage in the household setting, taking advantage of the exible demand of EWHs. Although their results are encouraging, the model is assumed to be perfect. The inaccuracies of their prediction model were not taking in to account. Liu and Shi [77] proposed a model in which MPC control is used to regulate the aggregated temperature of TCLs to provide frequency regulation services. The model aims to determine the TCL to turn on/o to monitor the frequency regulation whilst maintaining the temperatures within a certain range. Zong et al. [78] proposed MPC approach that utilizes aggregated EWHs as a grid-scale storage resource. The simulation results show that aggregated EWHs can eciently be operated with MPC to provide reserve services for renewable energy resources. Knudsen and Petersen [79] used the Economic Model Predictive Controller (EMPC) to control the water heater system that is used to increase the supply temperature in ultra-low temperature district heating since it can easily manage time-varying tari s. Awadelrahman et al. [80] also proposed an EMPC to maximize heating energy costs with varying electricity price signals in a residential building. It was shown that while the temperatures in the systems were kept within limits, EMPC shifted the electricity demand based on the price. Despite the mentioned successful implementations and the bene ts of MPC, this control technique has several disadvantages [81]. To list a few, in order for MPC to accurately predict the relevant variables namely, the model 6 Machine Learning Supervised Learning Classi cation RegressionUnsupervised Learning Association ClusteringReinforcement Learning Classi cation Control Figure 3.1: Machine Learning Methods parameters, the state of the system, and the exogenous variables the model has to be highly accurate and precise [82]. Moreover, MPC also requires a wealth of high quality input data during operation, as a result the system has to detect and lter out erroneous input data [67]. This makes MPC an expensive technique to implement [83]. 3 Background to Machine Learning 3.1 Overview ML evolved from Arti cial Intelligence (AI). ML methods are used to characterize algorithms that learn from existing data and these algorithms use a large amount of data for the learning process with a very limited number of input features [28]. There are three major ML methods available for behavioural analysis. Namely, the supervised learning, unsupervised learning and reinforcement learning. Figure 3.1 summarizes the learning algorithms of ML. 3.2 Supervised Machine Learning Supervised Learning (SL) is the development of algorithms that can create hypotheses with external instances to forecast the events of the future [84][85]. In other words, supervised algorithms use the labelled training data set to construct contingent functions for mapping new instances [86]. The main purpose of SL methods is to learn how to predict a random variable based on a set of random variables. SL consists of techniques for automatically building a predictive function that maps the predictor attributes to the target variable. SL method has two types of categories, namely, the classi cation and regression [87]. The key distinction between both types of de nitions is that the output variable for regression is empirical, while the output variable for classi cation is numerical. Both the classi cation and regression algorithms involves two steps in the pre- processing phase, namely, the feature extraction phase and the feature selection phase. In the feature extraction phase, the numerical features are extracted from the data set. In the feature selection phase, statistical approaches are used to nd correlations between the predictor attributes and the target variables to eliminate attributes with low predictive accuracy [88]. The SL algorithms comprise of Support Vector Machine (SVM), linear regression, logistic regression, neural networks, decision tree, random forest, naive Bayes, and K-nearest neighbour. 3.3 Unsupervised Machine Learning This ML approach has gained signi cant interest in the study of building energy since it has proved valuable in the implementation of energy benchmarking where evaluating baseline buildings is essential for measuring energy performance [89]. Unsupervised Learning (USL) is applied to unlabelled data such that it can be grouped based on function similarity. USL has two types of categories, namely, the clustering and association rule mining. However, clustering is the most popular algorithm used for the prediction of energy usage [90]. 7 To classify correlation between data elements that are unlabelled and uncategorized, the clustering algorithm is used. The purpose of clustering is to nd di erent classes in a collection of results. K-means, which iteratively searches for a local limit, is the most popular clustering algorithm employed in energy prediction. The algorithm starts with a random set of K-centroids (cluster centre), each of which is attributed to the closest centre point. So the mean of all data points in a category is used to recalculate all centroids. 3.4 Reinforcement Learning Reinforcement Learning (RL) is an AI algorithm focused on the agent in which agents learn the optimal set of actions to optimize a numerical reward signal, i.e. learn what to do. RL is also extended to topics concerning sequential dynamics and the optimization of a scalar goal of output. The main goal of RL is to train information and to nd a single input value that maximizes the total amount of rewards over the sequence of decisions [91][92]. RL algorithms operate in two distinct groups, namely model-based and model-free RL. The model is rst trained in the model-based RL and then used in a scheduling process, such as Least-Square Policy Iteration (LSPI) or tted Q-iteration. In the model-free method, the agent learns to associate optimal behaviour for each state without speci cally deciding the probability of transfer between states. Model-free RL interaction-based algorithms include techniques such as Q-learning and State-Action-Reward-State-Action (SARSA) [93]. Model- free RL control solutions are regarded as a valuable substitute or complement to model-based RL control solutions [94]. Due to its simplicity, regardless of the implementation area, Q-learning, introduced by Watkins and Dayan [95], is the most commonly used model-free RL technique [96]. 4 Novelty and review approach 4.1 Contribution To the best of our knowledge, this is the rst survey that covers the recent advancements made in energy consumption optimization and scheduling of individual EWH in smart homes and aggregated EWHs from the perspectives of data analytics, and ML. The main contributions of our review are to: (a) Identify state of the art models for energy optimization and scheduling of EWHs. (b) Review the current ML models for load management of EWHs in smart grids and smart building environ- ment. (c) Identify and discuss research challenges and directions for EWH energy usage optimization and scheduling using ML. 4.2 Literature Search To provide an overview of the existing research, a literature search has been conducted from Web of Science (WoS), IEEE Xplore, Google Scholar, and Scopus. The search strings de ned in Table 4.1 were used to identify relevant literature. The selection of the literature was carried out based on the consideration of energy consumption prediction of EWHs for optimization and scheduling. The focus is on ML based algorithms. Further, the approach used in the search was such that di erent papers published between 2010 and 2020 dealing with the use of ML models as methods of data analysis were included. The literature search was performed using the following notation, TITLE-ABS-KEY (A AND (B OR (C AND D)). Where the parameters A, B, C and D are to the search terms shown in Table 4.1. The results of the search are presented in Table 4.2. The reviewed papers were classi ed based on the learning method and associated algorithms. 8 Table 4.1: Search Queries A B C D Electrical water heater Machine Learning Energy consumption Prediction Hot water Supervised Learning Demand response Forecasting Unsupervised Learning Optimization Reinforcement Learning Scheduling Table 4.2: Summary of the papers reviewed Ref Year Learning Method Algorithm Objectives [97] 2008 SL ANN Energy, cost reduction [98] 2009 SL ANN Energy, cost reduction and comfort [99] 2014 SL ANN Peak reduction [100] 2016 SL SVR Energy, cost reduction [71] 2017 SL ANN Energy, cost reduction and comfort [101] 2019 SL SVM Energy, cost reduction and comfort [102] 2019 SL SVM Energy, cost reduction [103] 2019 SL ANN Energy cost reduction, comfort [104] 2019 SL FNN Energy, cost reduction and peak reduction [105] 2019 USL K-Means Energy, cost reduction [106] 2018 USL K-Means Energy conversation [107] 2019 USL RNN Energy, cost reduction and comfort [108] 2012 RL Q-Learning Ancillary service [109] 2014 RL Fuzzy Q-learning Energy reduction [110] 2014 RL BRL Cost reduction [111] 2016 RL BRL Energy, cost reduction and comfort [112] 2017 RL Actor-critic Q-learning Cost reduction [113] 2017 RL BRL Integration of renewable energy [32] 2017 RL Q-iteration Energy, cost reduction and comfort [114] 2018 RL Auto-encoder network Energy, cost reduction and tted Q-iteration 5 Machine learning based demand response of electric water heater in smart grid Learning-based methods, commonly known as model-free predictive models, are less complex in modelling and solving an optimal control problems than model-based predictive and rule-based controls. This is achieved using algorithms to derive a control trajectory to forecast the price and demand [115]. Model-free predictive control methods have been applied to EWHs for di erent problems using di erent approaches. Each one is summarized separately below. We will start rst discuss the application of neural network (NN) models in water heating systems and we will then place more focus on the application of machine learning algorithms/techniques for water heating systems. Bartecsko-Hibbert et al. [98] developed a nonlinear autoregressive network with external input (NARX) using arti cial neural network (ANN) for electric water heaters to predict the temperature characteristics of EWHs. The model was trained used a Generic Algorithm (GA). The main objective of their study was to minimise the cost of energy while maintaining the user's comfort. An optimization problem, based on the NARX water heater model, is formulated to optimize energy management of the water heater in a day-ahead, dynamic electricity price framework. However, their proposed models showed a relatively high error rate. The ndings indicate that 9 the models trained on three separate systems yield errors fewer than 11%. Sonnekalb and Lucia [103] also proposed ANN to optimize the heating schedule of water heaters to achieve signi cant energy savings. 18 months data sets from 17 di erent domestic water heaters (DWHs) in Ireland were simulated. The authors used Gaussian processes to train the model to learn individual human behaviour based on hot water usage data. The authors claim that their proposed method saved between 20% and 34% of the energy. However, this method requires a huge amount of historic data. The challenge is that in most of the cases this data is not available. Maltais and Gosselin [107] developed a predicting model for DWH demand using recurrent neural networks (RNN). They implemented their study to a 40-unit residential building in Quebec City, Canada, in order to predict the usage. The author claimed that the evaluated performance indices for the prediction of the next demand are satisfying with an R2 of 0.71. However, this proposed method does not tackle the prediction errors. To address the above mentioned challenges of NN, ML algorithms incorporated with NN are used. 5.1 Supervised Machine Learning NN is the widely employed supervised ML technique to predict energy performance due to its ability to handle complex and nonlinear problems [116]. Bakker et al. [97] used ANN with the heat demand pro le input from the previous day and previous week, as well as the weather data to forecast 24 hours heat demand. The model has been shown to have good estimates of daily hot water usage, but updates are required to provide additional variables relevant to user activity as inputs for the model. In [99], Shaad et al. developed a prediction module that uses NN to forecast the aggregated EWHs energy usage in real-time. The mean absolute error rate was recorded in the range of 6.5 kW to 7.8 kW when predicting aggregated data from 95 water heaters. Qu et al. [104] developed a control method using fuzzy neural network to manage thermostatically controlled loads (TCLs) in the smart grid to ensure a stable frequency of the grid. The results of the simulation showed that the developed model has signi cant advantages of monitoring accuracy without modelling the aggregated TCLs. However, the main disadvantage of NN is a large amount of computational time [117]. Support vector machines (SVM) have shown great potential in forecasting electricity prices and estimation of power usage over NN [118]. Aki et al. [100] proposed a bottom-up strategy using SVR for the prediction of EWH energy usage. The proposed model predicts the cumulative hot water usage for the next day by taking into account of recent trends of historical data. Cao et al. [101] used SVM to predict the shower behavior of occupants to forecast the hot water usage. The authors collected data from seven occupants and the results showed that, compared to a traditional control method, the proposal can reduce heat loss by up to 33%. Guo and Nariman [102] developed a ML methodology using SVMs to classify days into three groups (hot, cold and mild) based on the data from aggregated EWHs on substation level. The ndings of the simulation show an accuracy of over 88%. However, the drawback of supervised learning techniques is that they require a large amount of historical data, which could be hard to access. 5.2 Unsupervised Machine Learning Two main techniques that are widely used in the predicting energy consumption in buildings for unsupervised learning are K-means and hierarchical clustering methods [119][120]. In the K-means clustering process, the algorithm starts by randomly choosing K-centroids, and each data is allocated to the nearest centre point. All the centroids would be recalculated using the mean of all the data points in the group. This method proceeds until a stopping condition is reached [89]. In [106], Wu et al. proposed an energy usage diagnosed method based on ML algorithm to solve the energy diagnosed problem of boiler hot water heat supply. The authors used the clustering method, K-means algorithm, to lter the data that has better energy conservation performance from 10 all data. Gong et al. [105] proposed a bottom-up forecasting model using Markov-based error reduction approach to predict the power usage of aggregated EWHs. To group data samples of small aggregated EWHs, the K-means algorithm was used to predict the power usage of large aggregated EWHs. The authors claim that the method proposed improved the accuracy of the forecast by about 20% to 80%. However, the proposed approach does not x the cost and hot water usage estimation errors. Zuniga et al. [121] proposed an EWH control strategy based on the dynamic programming and power usage pro le classi cation using the K-means clustering algorithm. Based on the analysis and simulation, the authors suggest that this algorithm can be implemented to control aggregated EWH to reduce the peak demand and to meet the hot water demand. 5.3 Reinforcement Learning Q-learning is the most often employed RL methodology applied to a demand response [122]. After every interac- tion with the environment, the Q-learning system utilizes temporal di erence learning to update its state-action meaning feature [123]. Al-Jabery et al. [109] suggested a load control strategy of EWHs utilizing Q-learning and the action-dependent heuristic dynamic programming (ADHDP) for the demand-side management of EWHs. Their ndings reveal that their suggested algorithm of Q-learning o ers better convergence than the traditional Q-learning. In terms of cost savings, customer satisfaction and load peak reduction, the ADHDP approach ob- tained the best performance. In a di erent study, Al-Jabery et al. [112] extended their work by applying an actor-critic Q-learning approach network to control a water heater and demonstrated that it achieved a better performance than the Q-learning controller. The disadvantage of Q-learning is that after each state update the given observation is discarded. This will require more iteration for already known state space. In [108], Kara et al. employed Q-learning technique utilising thermostatically controlled load to o er short-term auxiliary services to the power grid. Kazmi et al. [111] proposed a general occupant-driven optimization model-based RL algorithm to improve the energy eciency for domestic hot water (DHW) production in residential buildings. The study applied to 32 homes showed a 20% reduction in the energy usage for water heating and no loss in user comfort. In [113], De Somer et al. implemented a Batch Reinforcement Learning (BRL) model-based technique that learns the behaviour of the occupants and forecasts the output of solar PV to monitor the heating cycle of a domestic water heater to optimize the use of energy from a local solar PV system. Using this algorithm, the authors conducted an experiment with six residential buildings and the ndings revealed that the self-consumption of solar PV production increased by 20% relative to the default thermostat control. In [110], Ruelens et al. implemented a model-free BRL algorithm to control a cluster of 100 water heaters. Their simulation ndings revealed that, relative to a hysteresis controller, the BRL technique can lower the energy cost during a learning span of 40-45 days. Inspired by the developments in BRL, Ruelens et al. [32] continued their work by evaluating the bene ts of Q iteration on a water heater in a laboratory and observed a 15% reduction in energy usage in as little as 40 days. In [114], the authors merged the auto-encoder network and the tted Q-iteration to minimize the cost of energy usage of EWHs. To test this proposed model, a simulation-based experiment using EWH with 50 temperature sensors was used. The authors claim that this approach has been able to reduce the average cost of EWH energy use by 15%. 6 Discussion In the previous sections, we have performed detailed reviews of control strategies used for energy optimization and scheduling of EWHs. In this section, we analyze and discuss those articles that involved the application of DR in EWHs, particularly those that provide ancillary services and facilitate the integration of renewable energy resources in the power system. These articles are summarized in Table 6.1. We also discuss the key opportunities and challenges of the various control methods that were identi ed by our study. 11 It is noted that the application of rule-based controls can yield good results in terms of providing ancillary services in the power system. This has been successfully applied in the