We investigate both approaches in this chapter. Singh et al. [217] produced a review of load forecasting techniques and methodologies and reported that hybrid methods, which combine two or more different techniques, are gaining traction, as well as soft computing approaches (AI) such as genetic algorithms. Our work presents a hybrid method which combines k-means clustering with multiple different learning algorithms. Dillon et al. [57] presented a neural network for short term load forecasting. Their neural network consisted of three-layers and used adaptive learning for training. They proposed the use of weather information to augment their electricity load data. They found better results with the adaptive neural network than with a linear model, or non-adaptive neural network. In contrast to Dillon our work focuses on a non-adaptive neural network and does not take into account weather information. Chen et al. [33] used an Artificial neural network to predict electricity demand of three substations in Taiwan. They integrated temperature data and reported that the best results when forecasting residential and commercial substations were during the week due to the influence of weather. In contrast to the work done by Chen et al., we focus on client-side prediction using smart meter data as opposed to substation data. We were, therefore, able to cluster the data based on load profile, as opposed to geographical location. Al-Musaylh et al. [6] proposed the use of Support Vector Regression (SVR), an Auto Regressive Integrated Moving Average (ARIMA) model and a Multivariate Adaptive Regression Spline (MARS) in their short term electricity demand forecasting system. They found that for a half, and one-hour forecasting horizons, the MARS model outperformed both the ARIMA and SVR. Taylor [98] evaluates different statistical methods including ARIMA, an adaptation of Holt-Winters' exponential smoothing [108], and an exponential smoothing method which focuses on the evolution of the intra-day cycle. He found that the double seasonal adaptation of the Holt-Winters' exponential smoothing method was the most accurate method for short lead times between 10 and 30 minutes. In contrast to Taylor, Fard et al. [67] proposed a novel hybrid forecasting method based on both artificial intelligence and classical time series approaches. They utilised the wavelet transform, ARIMA and ANNs for short term load forecasting. The ARIMA model is created by finding the appropriate order using the Akaike information criterion [5]. The ARIMA model models the linear component of the load time series, and the residuals contain the non-linear components. These residuals are then decomposed by the discrete wavelet transform into its sub-frequencies. ANNs are then applied to these sub-frequencies, and the outputs of both the Artificial Neural Network (ANN) and ARIMA models are summed to make the final prediction. They found that this hybrid technique outperformed traditional methods. Our work does not integrate artificial intelligence and classical time series techniques, due to time constraints. Multiple techniques have been proposed for the clustering of electricity load data prior to forecasting. Both Shu and Luonan [215], and Nagi et al [173] propose a hybrid approach in which self-organising maps are used to cluster the data, and Support Vector Regression is used to make predictions. This technique proved robust for different data types, and was able to tackle the non-stationarity of the data. Shu showed that this hybrid approach outperformed a single SVR technique, whilst Nagi showed superior results to a traditional ANN system. In contrast to both Nagi et al. and Shu and Luonan our work utilises k-means as the clustering algorithm. Quilumba et al. [67] also apply machine learning techniques to individual households' electricity consumption by aggregation. To achieve this aggregation, they use k-means clustering to aggregate the households to improve their forecasting ability. The authors also use a neural network-based model for forecasting, and show that the number of optimum clusters for forecasting is dependent on the data, with three clusters optimal for the Con Edison Launches Smart Grid Pilot Program in Queens [38], and four for the Commission for Energy Regulation (CER) dataset [39]. Wijaya et al. [248] demonstrated that implementing clusters improved load-forecasting accuracy up to a certain level. Whilst, a study by Ilić et al. [118] showed that increasing the number of clusters did not improve accuracy. Humeau et al. [113] compare MLPs, SVRs and linear regression at predicting smart meter data. They aggregate different households and observe which models work the best at each aggregate level. They find that linear regression outperforms both MLP and SVR when forecasting individual households. However, after aggregating over 32 households, SVR outperforms linear regression. Whilst multiple papers have looked at demand-side forecasting, as shown by the review paper by Singh et al. [217], to the best of our knowledge, the impact of online learning has been discussed with less frequency. In addition to this, our research models the impact of the performance of different algorithms on investments made, electricity sources dispatched and carbon emissions over a 17 year period. To model this, we use the model ElecSim. In our work, we trial a set of different algorithms to our problem. Due to time and compute constraints, we do not trial the additional techniques discussed in this literature review within this thesis. Johansson et al. [126] apply online machine learning algorithms for heat demand forecasting. They find that their demand predictions display robust behaviour within acceptable error margins. Finding that artificial neural networks (ANNs) provide the best forecasting ability of the standard algorithms and can handle data outside of the training set. Johansson et al., however, do not look at the long-term effects of different algorithms on their application. Baram et al. [15] combine an ensemble of active learners by developing an active-learning master algorithm. To achieve this, they propose a simple maximum entropy criterion that provides effective estimates in realistic settings. Their active-learning master algorithm is empirically shown to, in some cases, outperform the best algorithm in the ensemble on a range of classification problems. Schmitt et al. [212] also extends on existing algorithms through an extension of the FLORA algorithm [246].
The FLORA algorithm generates a rule-based model, which has the ability to make binary decisions. Their FLORA-MC enhances the FLORA algorithm for multiclassification and numerical input values. They use this algorithm for an ambient computing application. Ambient computing is where computing and communication merges into everyday life. They find that their model outperforms traditional offline learners by orders of magnitude. Similarly to our work, Pindoriya et al. [190] trial several different machine learning methods such as adaptive wavelet neural network (AWNN). They find that AWNN has good prediction properties when compared to other forecasting techniques such as wavelet-ARIMA, multilayer perceptron (MLP) and Radial Basis Function (RBF) neural networks as well as the fuzzy neural network (FNN). Goncalves Da Silva et al. [89] show the effect of prediction accuracy on local electricity markets. To this end, they compare forecasting of groups of consumers in comparison to single individuals. They trial the use of the Seasonal-Naïve and Holt-Winters algorithms and look at the effect that the errors have on trading in an intra-day electricity market of consumers and prosumers. They found that with a photovoltaic penetration of 50%, over 10% of the total generation capacity was uncapitalised and roughly 10, 25 and 28% of the total traded volume were unnecessary buys, demand imbalances and unnecessary sells respectively. This represents energy that the participant has no control. Uncapitalised generation capacity is where a participant could have produced energy; however, it was not sold on the market. Additionally, due to forecast errors, the participant might have sold less than it should have. Our work, however, focuses on a national electricity market, as opposed to a local market. In this section, we explore the principles behind the methods used in this chapter. The Mean Absolute Percentage Error (MAPE) is a measure of prediction accuracy which is used in this thesis. It can be defined as follows:

where y i is the actual value, ŷi is the forecast value and n is the number of points forecast [148]. The root mean squared error (RMSE) is a measure between the values predicted by a model and the observed values. The RMSE is the sample standard deviation of the differences between the predicted and observed values. The RMSE is defined as follows:

where ŷi are the predicted values, y i are the observed values, and n is the number of observations. The mean absolute scaled error (MASE) is a measure of the accuracy of forecasts [116]. It is defined as the mean absolute error of the forecast values, divided by the mean absolute error of the in-sample one-step naive forecast. MASE can be scaled across different scales, has symmetry for both positive and negative errors, is interpretable and has predictable behaviour for a value of zero. MASE can be defined as follows:

where e j is the forecast error for a given period, J is the number of forecasts. Where e j is defined as the actual value (Y j ) minus the forecast value (F j ) for that period. The denominator is the mean absolute error of the one-step naive forecast method on the training set. This naive forecast is the actual value from the prior period, or F t = Y t-1 . T is the total number of forecasts. Machine learning is a methodology for finding and describing structural patterns in data [249]. Offline learning models are trained with the data available at a single point in time. With nonstationary data -where underlying distributions change -the model must be retrained at periodic intervals, determined by how quickly the model goes out of step with the true data. With online learning, the model is able to retrain every time a new data point becomes available, without having to retrain the entire model. This makes these models good for time-series data which exhibit moderate to significant non-stationary properties, such as electricity demand profiles. Examples of online learning algorithms are Passive Aggressive (PA) Regressor [45], Linear Regression, Box-Cox Regressor [25], K-Neighbors Regressor [72] and Multilayer perceptron regressor [102]. For our work, we trial the stated algorithms, in addition to a host of offline learning techniques. The offline techniques trialled were Lasso regression [231], ridge regression [83], Elastic Net [79], Least Angle Regression [60], Extra Trees Regressor [60], Random Forest Regressor [27], AdaBoost Regressor [78], Gradient Boosting Regressor [80] and Support vector regression [42]. We chose the boosting and random forest techniques due to our previous successes of these algorithms when applied to electricity demand forecasting [131]. We trialled the additional algorithms due to the availability of these algorithms using the scikit-learn package and online learning package, Creme [94,186]. Linear regression is an approach to modelling linear relationships between a dependent variable and one or more independent variables. Linear regressions can be used for both online and offline learning. In this thesis, we use them for both online and offline learning. Linear regression models are often fitted using the least-squares approach. The least-squares approach minimises the sum of the squares of the residuals. Other methods for fitting linear regressions are by minimising a penalised version of the leastsquares cost function, such as in ridge and lasso regression [83,231]. Ridge regression is a useful approach for mitigating the problem of multicollinearity in linear regression. Multicollinearity is where one predictor variable can be linearly predicted from the others with a high degree of accuracy. This phenomenon often occurs in models with a large number of parameters. In ridge regression, the OLS loss function is augmented so that we not only minimize the sum of squared residuals but also penalized the size of parameter estimates, in order to shrink them towards zero:

we solve for β , where β are the OLS parameter estimates. β is a vector. n is the number of observations of the response variable, Y , Where Y = {y 0 , y 1 , . . . , y n }.
With a linear combination of m predictor variables, X, where x = {x 0 , x 1 , . . . , x n }. λ is the regularisation penalty which can be chosen through cross-validation, or the value that minimises the cross-validated sum of squared residuals, for instance. Lasso is a linear regression technique which performs both variable selection and regularisation. It is a type of regression that uses shrinkage. Shrinkage is where data values are shrunk towards a central point, such as the mean. The lasso model encourages models with fewer parameters. This enables the selection of models with fewer parameters, or automate the process of variable selection. Under lasso the loss is defined as:

The only difference between lasso and ridge regression is the penalty term. Elastic net is a regularization regression that linearly combines the penalties of the lasso and ridge methods. Specifically, Elastic Net aims to minimize the following loss function:

where α is the mixing parameter between ridge (α = 0) and lasso (α = 1). The two parameters λ and α can be tuned. Least Angle Regression (LARS) provides a means of producing an estimate of which variables to include in a linear regression, as well as their coefficients. The decision tree is a model which goes from observations to output using simple decision rules inferred from data features [201]. To build a decision tree, recursive binary splitting is used on the training data. Recursive binary splitting is a greedy top-down algorithm used to minimise the residual sum of squares. The RSS, in the case of a partitioned feature space with M partitions, is given by:

Where y is the value to be predicted, ŷ is the predicted value for partition R m . Beginning at the top of the tree, a split is done into two branches. This split is carried out multiple times and the split is chosen that minimises the current RSS. To obtain the best sequence of subtrees cost complexity, pruning is used as a function of α. α is a tuning parameter that balances the depth of the tree and the fit to the training data. This parameter is used to see how deep the tree should be. This parameter can be tuned using cross-validation. The AdaBoost training process selects only the features of a model known to improve the predictive power of the model [78]. By doing this, the dimensionality of the model is reduced and can improve compute time. This can be used in conjunction with multiple different models. In this thesis, we utilise the decision tree-based algorithm with AdaBoost. Random Forests are an ensemble learning method for classification and regression [27]. Ensemble learning methods use multiple learning algorithms to obtain better predictive performance. They work by constructing multiple decision trees at training time, and outputting the predicted value that is the mode of the predictions of the individual trees. To ensure that the individual decision trees within a Random Forest are not correlated, bagging is used to sample from the data. Bagging is the process of randomly sampling with replacement of the training set and fitting the trees. This has the benefit of reducing the variance of the model without increasing the bias. Random Forests differ in one way from this bagging procedure. Namely, using a modified tree learning algorithm that selects, at each candidate split in the learning process, a random subset of the features, known as feature bagging. Feature bagging is undertaken due to the fact that some predictors with a high predictive ability may be selected many times by the individual trees, leading to a highly correlated Random Forest. ExtraTrees adds one further step of randomisation [60]. ExtraTrees stands for extremely randomised trees. There are two main differences between ExtraTrees and Random Forests. Namely, each tree is trained using the whole learning sample (and not a bootstrap sample), and the top-down splitting in the tree learner is randomised. That is, instead of computing an optimal cut-point for each feature, a random cut-point is selected from a uniform distribution. The split that yields the highest score is then chosen to split the node. Gradient boosting is a machine learning technique which is used for both regression and classification problems [80]. Typically, decision trees are used to produce a prediction model in the form of an ensemble of weak prediction models. The model is built in a stage-wise fashion. These model types are generalisable by allowing for an optimisation of an arbitrary differentiable loss function. The outline of the algorithm is defined here. As discussed, in many supervised learning problems, one has an output variable y and a vector of input variables x. These two variables are described via a joint probability distribution P(x, y). The goal of supervised learning is to use a training set {(x 1 , y 1 ), . . . , (x n , y n ))} of known x and y values to find an approximation of F(x), F(x). F(x) is found by minimizing the expected value of some specified loss function L(y, F(x)):

(4.8)

The gradient boosting method assumes a real-valued y and seeks an approximation F(x), where F(x) is a weighted sum of function h i (x) from some class H. Where H are called weak leaners:

This method tries to find an approximation F(x) that minimizes the average value of the loss function on the training set. It does so, by starting with a constant function model. F 0 (x) and incrementally expands it in a greedy fashion:

where h m ∈ H is a weak learner function. Due to the fact that choosing the best function h is computationally infeasible, an approximation must be sought. This approximation is found using functional gradient descent to this minimization problem. If we consider the continuous case, i.e.
where H is the set of arbitrary differentiable functions on R, we would update the model in accordance with the following equations:

)

where the derivatives are taken with respect to the function F i for i ∈ {1, . . . , m} and γ m is the step length. Support vector regression is an algorithm which finds a hyperplane and decision boundary to map an input domain to an output [42]. The hyperplane is chosen by minimising the error within a certain tolerance. Suppose we have the training set: (x 1 , y 1 ), . . . , (x i , y i ), . . . , (x n , y n ), where x i is the input, and y i is the output value of x i . Support Vector Regression solves an optimization problem [32,215], under given parameters C > 0 and ε > 0, the form of support vector regression is [58]: subject to:

x i is mapped to a higher dimensional space using the function φ . The ε-insensitive tube (ω T φ (x i ) + b)y i ≤ ε is a range in which errors are permitted. ξ i and ξ * i are slack variables which allow errors for data points which fall outside of ε. This enables the optimisation to take into account the fact that data does not always fall within the ε range [219]. The constant C > 0 determines the trade-off between the flatness of the support vector function. ω is the model fit by the SVR. The parameters which control regression quality are the cost of error C, the width of the tube ε, and the mapping function φ [32,215]. k-Neighbours regression is a non-parametric method used for regression [72]. One such k-NN algorithm uses a weighted average of the k nearest neighbours, weighted by the inverse of their distance. The algorithm works as follows:

1. Compute the Euclidean or Mahalanobis distance from the test example to the labelled examples. 2. Order the labelled examples by increasing distance. 3. Find a heuristically optimal number k of nearest neighbours, based on cross-validation. 4. Calculate an inverse distance weighted average with the k-nearest multivariate neighbours. A neural network can be used in both offline and online cases. In this thesis, we used them for both online and offline. Artificial Neural Networks are a model which can model non-linear relationships between input and output data [5]. A popular neural network is a feed-forward multilayer perceptron. Fig. 4.3 shows a three-layer feed-forward neural network with a single output unit, k hidden units, n input units. w i j is the connection weight from the i th input unit to the j th hidden unit, and T j is the connecting weight from the j th hidden unit to the output unit [184]. These weights transform the input variables in the first layer to the output variable in the final layer using the training data. For a univariate time series forecasting problem, suppose we have N observations y 1 , y 2 , . . . , y N in the training set, and m observations in the test set, y N+1 , y N+2 , . . . , y N+m , and we are required to predict m periods ahead [184]. The training patterns are as follows:

y p+m+1 = f W (y p+1 , y p , . . . , y 2 ) (4.17) . . . where f W (•) represents the MLP network and W are the weights. For brevity we omit W . The training patterns use previous time-series points, for example, y p , y p-1 , . . . , y 1 as the time series is univariate. Where p is the total number of points in the univariate time-series. That is, we only have the time series in which we can draw inferences from. In addition, these time series points are correlated, and therefore provide information that can be used to predict the next time point. The m testing patterns are:

y N+2 = f W (y N+2-m , y N-m+1 , . . . , y N-m-p+3 ), (4.20) . . . The training objective is to minimise the overall predictive mean sum of squared estimate of errors (SSE) by adjusting the connection weights. For this network structure the SSE can be written as:

where ŷi is the prediction from the network. The number of input nodes corresponds to the number of lagged observations. Having too few or too many input nodes can affect the predictive ability of the neural network [184]. It is also possible to vary the hyperparameter or the number of input units. Typically, various different configurations of units are trialled, with the best configuration being used in production. The weights W in f W are trained using a process called backpropagation [207], which uses labelled data and gradient descent to update and optimise the weights. Long Short Term Memory Neural Network (LSTM) [104] is an artificial recurrent neural network (RNN) architecture, which is used in the field of deep learning. They are able to process entire sequences of time-series data. A common LSTM unit is composed of a cell, an input gate, an output gate and a forget gate. The cell is able to remember values over arbitrary time intervals and the three gates regulate the flow of information into and out of the cell. LSTMs are well suited to time-series data, due to the presence of lags between important events, with an unknown duration. LSTMs have an advantage over traditional RNNs by solving the vanishing gradient problem. This is where the gradients which are back-propagated can tend to zero or infinity. This is due to the computations involved in the process, which use finite-precision numbers. LSTMs partially solve this problem by allowing gradients to flow unchanged in the back-propagation process.
In this section we discuss the algorithms which were used exclusively for online learning in this thesis. In this subsection, we discuss the Box-Cox regressor. Ordinary least squares is a method for estimating the unknown parameters in a linear regression model. It estimates these unknown parameters by the principle of least squares. Specifically, it minimises the sum of the squares of the differences between the observed variables and those predicted by the linear function. The ordinary least squares regression assumes a normal distribution of residuals. However, when this is not the case, the Box-Cox Regression may be useful [25]. It transforms the dependent variable using the Box-Cox Transformation function and employs maximum likelihood estimation to determine the optimal level of the power parameter lambda. The Box-Cox Regression requires that no dependent variable has any negative values. Variable selection and ordinary least squares output dialogues are identical to that of linear regression. The Box-Cox regression will transform the dependent variable as follows:

Where λ is the power parameter, and the data vectors are y i = (y 1 , . . . , y n ). The optimal value of (λ ) is determined by maximising the following log-likelihood function:

where σ 2 (λ ) is the estimate of the least squares variance using the transformed y variable. The goal of the Passive-Aggressive (PA) algorithm is to change itself as little as possible to correct for any mistakes and low-confidence predictions it encounters [45]. Specifically, with each example PA solves the following optimisation [158]:

Where x t is the input data and y i the output data, and w t are the weights for the passive aggressive algorithm. Updates occur when the inner product does not exceed a fixed confidence margini.e., y i (w • x t ) ≥ 1. The closed-form update for all examples is as follows:

where

a t is derived from a derivation process which uses the Lagrange multiplier. For full details of the derivation see [45]. In this section we present the work undertaken for short-term demand forecasting. We forecast 30-minutes ahead using smart meter data. The Data

Smart meter data obtained from the Irish Social Science Data Archive (ISSDA) on the 28th of September 2017 was used in this study [71]. The Commission for Energy Regulation released a public dataset of anonymised smart meter data from the "Electricity Smart Metering Customer Behaviour Trials" [71]. This dataset is made up of over 5,000 Irish homes and businesses and is sampled at 30-minute intervals. The data was recorded between the 14th July 2009 and 31st December 2010, providing 17 months worth of data. For the purposes of cross-validation, the data was split into two partitions, the training set and the testing set. The training set made up the first 11 months of data and was used to parametrise the models, whereas the test set is made up of the remaining six months of data. This split was chosen to balance the amount of training data with the test data and to give the models a chance to learn the periodicity inherent in a twelve month period of electricity load. The test set was used for evaluation of the models proposed. Due to the long training times for these algorithms, we worked with a sub-sample of 709 individual Irish homes from the whole dataset. However, we believe that our results would hold over the full dataset. Figure 4.4 demonstrates the electricity consumption profile of a single week for a single user. Whilst it can be seen that electricity usage changes significantly between days, a pattern of behaviour is exhibited. There is a large peak displayed each day in the evening, as well as a peak earlier during the day. It can, therefore, be assumed that this customer has some form of habitual behavioural pattern. These figures display that electricity consumption changes per person per day. To capture this variability between customer types, these customers are clustered and then aggregated. Each  of the different aggregated electricity consumptions should provide a less stochastic load profile, and therefore increase the accuracy of the models. We propose that clustering similar customer load profiles and aggregating each cluster's electricity consumption improves the accuracy of the models. Figure 4.6 displays four different customers with similar load profiles. Each of the users display a strong peak in electricity consumption during the evening and less consumption during the day. These customers may potentially be clustered together by the k-means clustering algorithm. To cluster the load profiles, different options were considered. Hierarchical clustering using metrics such as Euclidean and wavelet distance metrics were evaluated [166], as was k-means [72]. K-means was demonstrated to be the most robust and best-performing clustering algorithm, and thus was chosen for use in this thesis. To select the optimum number of clusters (k) cross-validation was explored. This allowed us to compare the results of each of the models and select k with the highest MAPE accuracy. The cross-validation method proposed, worked by trying a different number of clusters per model, and testing for the resulting MAPE. The optimum number of clusters with a low MAPE is then chosen. In this thesis we varied k between 1 and 7; this range was chosen due to the fact that the error did not vary greatly past seven clusters. We fit multiple models per cluster and predicted six months of electricity consumption. With k-means clustering, it is possible that with the same initialisation number of clusters, different clusters are formed. This is due to the algorithm converging at a local minima. To overcome local minima, the k-means algorithm is run multiple times, and the partition with the smallest squared error is chosen [124]. In our case, the k-means clustering algorithm is run 1,000 times to reduce the chance of finding a local minima. The clustering technique utilised in our work was a scaled input approach. The daily load profile was averaged for each customer based on each day of the training data.
The data was then scaled so that households of different sizes, but with similar usage profiles were clustered together. This data, which is made up of a m-by-n matrix, where m is equal to the total number of meters and n is equal to 48 (two readings for each hour in the day). To find the optimum number of clusters it is recommended that the user selects a value of k that is high enough that distinct average load profiles are displayed, however, not so high that well-clustered customers are split. By doing this, the stochasticity of the load profiles in each of the clusters will be reduced, and thus lead to the best results. Once each customer is assigned to their respective cluster using k-means clustering, the total electricity consumed per cluster is aggregated. This is achieved by summing the electricity consumed at each time interval per cluster. This creates a partial system load. A different model is trained on each of the different partial system loads, and the resultant forecasts are aggregated to generate the total system load forecast. The total system load forecast is then used to evaluate the accuracy of each of the different models using MAPE. Random Forests, Support Vector Regression, Multilayer Perceptron neural networks and Long-Short Term Memory neural networks were evaluated, and a comparison between the different models were made. These models were chosen due to their ability to model multivariate non-linear relationships. They are data-driven methods and therefore suited to this type of problem. Each component of the training data is known as a feature. Features encode information from the data that may be useful in predicting electricity consumption. Due to the daily, weekly and annual periodicity of the electricity consumption daily calendar attributes may be useful to model the problem. The calendar attributes included are as follows:

• Hour of day These attributes enable the daily, weekly and annual periodicity to be taken into account by the model. We chose these due to the daily, monthly and weekly variations for electricity demand. It is also observed that electricity demand is changed on a public holiday. It is noted that electricity consumption changes on a public holiday such as Christmas or New Year's Eve. For example, a hotel may consume more electricity and an office space less. It is therefore proposed that public holidays in Ireland are input into the model as features. For testing purposes, two sets of models for Random Forests, Multilayer Perceptrons and Support Vector Regression were fit. One set omitted these calendar attributes whilst the other didn't. This is done to evaluate the importance of periodicity in electricity consumption prediction. As well as the calendar attributes, it is important to consider the historical load demand. This allows the time-series element to be modelled. To do this, a lagged input of the previous 3 hours, the equivalent three hours from the previous day, and the equivalent 3 hours from the previous week were used. For example, to predict the electricity consumed on the 21st December 2010 at 12:00 pm the electricity between 9:00 pm and 11:30 pm on the 21st of December are used as inputs, as are the times between 9:00 pm and 12:00 pm on the 20th and 14th of December. Long-Short Term Memory neural networks remember values over arbitrary time intervals. They can remember short-term memory over a long period of time; for this reason, five lagged inputs of the previous two and a half hours were used as features to the Long-Short Term Memory network. Once useful information is selected, we must encode the data for input into the models. To encode the day of the week, seven binaries are utilised in the form of one-hot encoding. Six of the binaries are for Monday through to Saturday. When all six binaries are equal to zero Sunday is encoded. A single binary for public holidays is included. Eleven binaries are used for month of the year, with the first eleven representing January to November, with December represented by all zeros in the calendar binaries. The current hour and date are input using a numerical attribute. The lagged data inputs, such as previous hour's electricity usage are also inputted using a numerical attribute for each entry, totalling 20 attributes (six half-hourly entries for each 3 hour period multiplied by three days plus 2 entries for the time to be predicted on the previous day and week). Table 4.1 displays these features. Overfitting can be a significant problem within the machine learning domain. Overfitting is where a model is fit too closely to the training data. Therefore, whilst the model is able to predict the original dataset well, it is unable to predict data it has not seen before to the same degree of accuracy. To avoid this problem, we used both a training and a test set. The training set is used to fit the data, however, to see the performance of the model, we tested the model using previously unseen data, which is called the test set. This therefore allowed us to verify that the models trained were not overfitting solely to the training data. This section explores the different methods used to select the model parameters, and the tests to evaluate our models. Once the parameters were chosen the models were trained on the different clusters of residential customers. Each model was run five times to explore the variance of the results. To implement a Support Vector Regression model, a variety of parameters must be chosen. These parameters influence the performance of the model. The parameters were evaluated using cross-validation. To do this, the data was split 75% into training data, and the remaining 25% into test data. This split was chosen to balance the trade-off between having enough training data so that the model can accurately learn the underlying form of the data, but also to have enough data to validate each model.
To choose the optimum support vector machine kernel cross-validation was also used. Again, with 75% acting as the training data and 25% as the test. The kernels compared were polynomial, radial basis function (RBF) and the linear kernel [30,230]. These were chosen due to their popularity, support and relative speed of computation. The parameter values selected are shown in Table 4.2. These parameter values were chosen from the results of cross-validation for each of the different kernels. the linear kernel was found to be the best performing. For this reason, the linear kernel was utilised for prediction of electricity consumption in this thesis. To initialise the Random Forest algorithm with the number of variables randomly sampled as candidates at each split, cross-validation was used. Once again, 75% of the data was used for training and the remaining 25% for testing due to the trade-off between training and testing. Figure 4.7 shows the results of tuning the parameter of the number of variables randomly sampled as candidates at each split. The optimum number was found to be 23. Either side of this value the RMSE increases. Therefore the value 23 was selected to be the number of variables randomly sampled as candidates at each split in the Random Forest model. It is proposed that the value 23 was found to be optimum due to the 20 lagged inputs, as this data is crucial for the Random Forest to learn the underlying nature of electricity load. A feed-forward Multilayer Perceptron is a common neural network architecture used for the prediction of time-series data, which has comparable, and in some cases better, results than statistical models [100]. The first step when designing a Multilayer Perceptron neural network is to design the architecture. For this case, the number of input neurons is set to 41 (see Table 4.1). Once an input for each neuron is entered, the output layer must be designed. Due to the fact that we are forecasting only one-time step ahead (30 minutes ahead) one output neuron is required. The next step is to design the architecture of the hidden layers. To accomplish this, crossvalidation is utilised as per the previous models. A maximum of 3 hidden layers were tested, and the results analysed. A similar method to Fan et al. [66] was evaluated to choose the number of neurons and hidden layers, a technique known as the Levenberg-Marquardt technique [171]. The Levenberg-Marquardt is a technique suitable for training medium-sized Artificial Neural Networks with a low mean-squared error. The fundamental rule is to select the minimum number of neurons in the hidden layer so as to capture the complexity of the model, but not too many as to introduce over-fitting, which results in a loss in the generalisation of the algorithm. The method begins by choosing a small number of neurons and gradually increasing the number each time the model is trained, and the forecast error obtained. The forecast error is monitored until an optimum value is found, to which no further improvement is noted. Once the optimum number of neurons in the layer is obtained, an additional layer is added, and the same technique is used. Using this technique, an optimal architecture with three layers is obtained. The first layer contained two neurons, the second contained five, and the third contained four. To initialise the LSTM, cross-validation was used to select the number of stacked layers and memory units. We trialled the number of stacks from the following list: [1,2,3,4] and the number of memory units from the following list: [5,6,7,8,9,10,20,30,40,50,75,100]. The optimum number of layers was found to be 2, with a total of 50 memory units. Different combinations of layers and memory units displayed worse results. To test the accuracy of the trained model, the data was split into a training and test set. The data between the 14th of July 2009 and the 15th of June 2010 was used as the training data, whilst the data between the 15th of June 2010 and 31st of December 2010 was used for testing purposes. The test set is separate from the training set and not used during training. We did not, however, consider seasonal issues for this work. 28 independent forecasting models are constructed for each of the Random Forests, Support Vector Regression, LSTMs and Multilayer Perceptron neural networks for each of the groups with k varying from 1 to 7. This was done to determine the optimal number of clusters. Each of   the 28 models are trained independently, five times each so that the standard deviation results of MAPE for each cluster could be displayed. We evaluated the MAPE of the overall prediction. Figure 4.10 demonstrates the impact of using calendar attributes such as month, day of the month, and day of the week on prediction accuracy. The results show an increase in prediction accuracy of 6% for neural networks, 4% for Random Forests and 1% for support vector regression when taking into account these variables. It is proposed that the ability for the models to take into account the yearly, monthly and weekly seasonal behaviour improves the results. Figure 4.8 displays the accuracy of the models trained at different numbers of clusters (k). The results demonstrate that introducing clusters to group similar customers improve results in all cases. The optimum value for k for Random Forests, Support Vector Regression and neural networks was shown to be four for our dataset. After this, the accuracy diminishes slightly. The error bars shown in Figure 4.8 show a small variance in MAPE in SVRs, ANNs and Random Forests. However, the MAPE of the LSTMs seem to vary by up to 11% in the five models run. Figure 4.9 shows the average load profiles of different clusters when k = 4. It is proposed that the optimum number of clusters is four due to the distinct load profiles that can be seen in Figure 4.9.
The four different distinct patterns seen are high night time use in cluster 1, a typical residential load profile is shown in cluster 2, a spread of usage in cluster 3, and high daytime usage in cluster 4. At k = 3 these distinct patterns are not adequately clustered, and at k = 5 one of the distinct clusters is split, leading to an increase in stochasticity. It is true that the optimum number of clusters will vary for different datasets. Whilst residential smart meter datasets may be similar, it is entirely possible that different geographies display different usage characteristics based on factors such as culture, temperature and economic reasons. It is therefore important to choose an optimal number of clusters for each dataset. The results demonstrate that SVR, Random Forests and the Multilayer Perceptrons have a similar overall accuracy. The LSTM shows a similar pattern in increasing accuracy with a number of clusters. However, the Random Forest seems to outperform each of the models at every point. This may be due, in part, to the internal operation of the Random Forest which undertakes its own cross-validation using out-of-bag samples and only having a few tuning parameters. This implies that neural networks and support vector regressions require more tuning. It has been shown that neural networks, SVR and Random Forests all perform within an adequate range of predicting electricity consumption. Whilst LSTMs perform poorly. This may be due to the features given to the LSTM, which only had previous two and a half hours of data as input. However, it is well known that the best machine learning technique for predicting energy consumption cannot be chosen a priori. Therefore it is necessary to compare different techniques to find the best solution to a particular regression problem [4]. For this work, the training time was tested by timing how long the models would take to fit to one cluster (single model trained on the training set). The Support Vector Regression took much less time than all of the other methods, whereas the LSTM took the longest. The Artificial Neural Network required 9 minutes and 5 seconds to run. The Support Vector Regression model required 3 minutes and 32 seconds to run. The Random Forest, on the same data, required 9 minutes and 44 seconds to run, whilst the LSTM took 12 minutes 55 seconds. In this section, we expand on the work undertaken in Section 4.4 by utilising further time-series prediction algorithms, including online machine learning methods. We take the error distributions and perturb the exogenous electricity demand of ElecSim, and observe the long-term impacts of poor error forecasts on the UK electricity market. It should be noted that this could be applied to any decentralised electricity market. Similarly to our previous work in Chapter 3 [131], we selected a number of calendar attributes and demand data from the GB National Grid Status dataset provided by the electricity market settlement company Elexon, and the University of Sheffield [62]. This dataset contained data between the years 2011-2018 for the United Kingdom. The calendar attributes used as predictors to the models were hour, month, day of the week, day of the month and year. These attributes allow us to account for the periodicity of the data within each day, month and year. It is also the case that electricity demand on a public holiday which falls on a weekday is dissimilar to load behaviours of ordinary weekdays [139]. We, therefore, marked each holiday day to allow the model to account for this. As demand data is highly correlated with historical demand, we lagged the input demand data. In this context, the lagged data is where we provide data of previous time steps at the input. For example, for predicting t+1, we use n inputs: t, t-1, t-2, . . ., t-n. This enabled us to take into account correlations on previous days, weeks and the previous month. Specifically, we used the previous 28 hours before the time step to be predicted for the previous 1st, 2nd, 7th and 30th day. We chose this as we believe that the previous two days were the most relevant to the day to be predicted, as well as the weekday of the previous week and the previous month. We chose the previous 28 hours to account for a full day, plus an additional 4 hours to account for the previous day's correlation with the day to be predicted. We could have increased the number of days provided to the algorithm, as well as marking holiday days. However, due to time and computational constraints, we used our previously described intuition for lagged data selection. The number of lagged inputs to trial increases exponentially with each additional day added, therefore making the problem intractable when also trialling such a high number of algorithms and hyperparameters. In addition to this, we marked each of the days with their respective seven seasons. These seasons were defined by the National Grid Short Term Operating Reserve (STOR) Market Information Report [177]. These differ from the traditional four seasons by splitting autumn into two further seasons, and winter into three seasons. Finally, to predict a full 24-hours ahead, we used 24 different models, 1 for each hour of the day. The data is standardised and normalised using min-max scaling between -1 and 1 before training and predicting with the model. This is due to the fact that the inputs such as day of the week, hour of day are significantly smaller than that of demand. Therefore, the demand will influence the result more due to its larger value; however, this does not necessarily mean that demand has greater predictive power. To find the optimum hyperparameters, cross-validation is used. As this time-series data was correlated in the time-domain, we took the first six years of data (2011-2017) for training and tested on the remaining year of data (2017-2018). Each machine learning algorithm has a different set of parameters to tune.
To tune the parameters in this thesis, we used a grid search method. Grid search is a brute force approach that trials each combination of parameters at our choosing; however, for our search space, this was small enough to make other approaches not worth the additional effort. Tables 4.3 and 4.4 display each of the models and respective parameters that were used in the grid search. Table 4.3 shows the offline machine learning methods, whereas Table 4.4 displays the online machine learning methods. Each of the parameters within the columns "Values" are trialled with every other parameter. Whilst there is room to increase the total number of parameters, due to the exponential nature of grid-search, we chose a smaller subset of hyperparameters, and a larger number of regressor types. Specifically, with neural networks, there is a possibility to extend the number of layers as well as the number of neurons, to use a technique called deep learning. Deep learning is a class of neural networks that use multiple layers to extract higher levels of features from the input. Each of the previously mentioned models trialled will have a certain degree of error. Prediction residuals are the difference between the estimated and actual values. We collect the prediction residuals to form a distribution for each of the models. We then trial 80 different closed-form distributions to see which of the distributions best fits the residuals from each of the models. These 80 distributions were chosen due to their implementation in scikit-learn [186]. Once each of the prediction residual distributions are fit with a sensible closed-form distribution, we sample from this new distribution and perturb the demand for the electricity market at each time step within ElecSim. By perturbing the market by the residuals, we can observe what the effects are of incorrect predictions of demand in an electricity market using the long-term electricity market model, ElecSim. We are able to understand the differences that prediction residuals have on long-term investment decisions as well as generators utilised. In this section, we detail the accuracy of the algorithms and statistical models to predict 24 hours ahead for the day-ahead market. In addition to this, we display the impact of the errors on electricity generation investment and electricity mix from the years 2018 to 2035 using the agent-based model ElecSim. To generate these results, we use a training set to train the data, and a test set to see how well each algorithm performs on the testing data. That is, how well the algorithm can predict data it is yet to see. In our case, the training data was from 2011 to 2017, and the testing data was from 2017 to 2018. Figure 4.11 displays the mean absolute error of each of the offline statistical and machine learning models on a log scale. It can be seen that the different models have varying degrees of success. The least accurate models were linear regression, the Multilayer Perceptron (MLP) model and the Least Angle Regression (LARS). These all have mean absolute errors over 10,000MWh. This error would be prohibitively high in practice; the max tendered national grid reserve is 6,000MWh, while the average tendered national grid reserve is 2,000MWh [177]. A number of models perform well, with a low mean absolute error. These include the lasso, gradient Boosting Regressor and K-neighbours regressor. The best model, similar to the work presented in the 30-minute ahead forecasting [131], was the decision tree-based model, Extra Trees Regressor, with a mean absolute error of 1, 604MWh. This level is well within the average national grid reserve of 2,000MWh. It can be seen that the max tendered national grid reserve falls well above the 5% and 95% percentiles. However, there are occasions where the errors are greater than the maximum tendered national grid reserve. In addition, the majority of the time, the model's predictions fall within the average available tendered national grid reserve. It can be seen from Figure 4.13 that the time to fit varies significantly between algorithms and parameter choices. The multilayer perceptron consistently takes a long time to fit, when compared to the other algorithms and performs relatively poorly in terms of MAE. There are many models, such as the random forest regressor, and extra trees regressors which perform well, however, take a long time to fit, especially when compared to the K-Nearest neighbours. For a small deterioration in MAE it is possible to decrease the time it takes to train the model significantly. For example, by using the K-Nearest neighbours or support vector regression (SVR). The scoring time, displayed in Figure 4.14, also displays a large variation between model types. For instance, the MLP regressor takes a shorter time to sample predictions when compared to the K-Neighbors algorithm and support vector regression. It is possible to have a cluster of algorithms with low sample times and low mean absolute errors. However, often a trade-off is required, with a fast prediction time requiring a longer training time and vice-versa. To see if we can improve on the predictions, we utilise an online machine learning approach. If we are successful, we should be able to reduce the national grid reserves, reducing cost and emissions. Figure 4.15 displays the comparison of mean absolute errors for the different trialled online regressor models. To produce this graph, we performed various hyperparameter trials. Where the hyperparameters had the same results, we removed them. For the multilayer perceptron (MLP), we aggregated all hyperparameters, due to the similar nature of the predictions. This is due to  the fact that no additional information would be gained from presenting this work, as the results are the same or very similar. It can be seen that the best performing model was the Box-Cox regressor, with an MAE of 1,100. This is an improvement of over 30% on the best offline model. The other models perform less well.
However, it can be seen that the linear regression model improves significantly for the online case when compared to the offline case. The passive aggressive (PA) model improve significantly with the varying parameters, and the MLP performs poorly in all cases. Figure 4.16 displays the best online model. We can see a significant improvement over the best online model distribution, shown in Figure 4.12. We remain within the max tendered national grid reserve for 98.9% of the time, and the average available tendered national grid reserve is close to the 5% and 95% percentiles. Figure 4.17 displays the residuals for a model with poor predictive ability, the passive aggressive regressor. It displays a large period of time of prediction errors at -20,000MWh, and often falls outside of the national grid reserve. These results demonstrate the importance of trying a multitude of different models and parameters to improve prediction accuracy. Figure 4.18 displays a comparison between the actual electricity consumption compared to the predictions. It can be seen that the Box-Cox model better predicts the actual electricity demand in most cases when compared to the best offline model, the Extra Trees regressor. The Extra Trees regressor often overestimates the demand, particularly during weekdays. Whilst the Box-Cox regressor more closely matches the actual results. During the weekend (between the hours of 120 and 168), the Extra Trees regressor performs better, particularly on the Saturday (between hours of 144 and 168). Figure 4.19 shows the differences between predicted values and actuals. Where the horizontal line y = 0 would be perfect predictions. This figure shows that the Extra Trees over predicts during the week. The Box-Cox however, struggles with the weekends. The best performing model is the Box-Cox model, which is also the fastest to both train and test. The linear regression, which performs worse in terms of predictive performance, is as quick to train and test as the Box-Cox model. Additionally, the multilayer perceptron (MLP) is relatively quick to train and test when compared to the PA models. It is noted that when compared to the offline models, the training time is a good indicator to the testing time. In other words, models that are fast to train are also fast to test and vice-versa. This also tells us that these algorithms are highly suited to this problem domain, as the training and testing time are much shorter than the available time to solve these problems. In this section we explore the effect of these residuals on investments made and the electricity generation mix. To generate these graphs, we perturbed the exogenous demand in ElecSim by sampling from the best-fitting distributions for the respective residuals of each of the online methods. We did this for all of the online learning algorithms displayed in Figure 4.15. We let the simulation run for 17 years from 2018 to 2035. Running this simulation enabled us to see the effect on carbon emissions on the electricity grid over a long time period. For instance, does underestimating electricity demand mean that peaker power plants, such as reciprocal gas engines, are over utilised when other, less polluting power plants could be used? In this section we display the mean electricity mix contributed by different electricity sources over the years 2018 to 2035. The reciprocal gas engine was expected to increase with times of high error. This is because, traditionally, reciprocal gas engines are peaker power plants. Peaker power plants provide power at times of peak demand, which cannot be covered by other plants due to them being at their maximum capacity level or out of service. It may also be the case, that with higher proportions of intermittent technologies, there is a larger need for these peaker power plants to fill in for times where there is a deficit in wind speed and solar irradiance. It is hypothesised that coal and nuclear output increase to cover the predicted increased demands of the service. As these generation types are dispatchable, meaning operators can choose when they generate electricity, they are more likely to be used in times of higher predicted demand. Photovoltaics may be used more with higher errors due to the times at which the errors were greatest. For example, during the day, where demand is higher, as is solar irradiance. In this Section, we detail the difference in total technologies invested in over the time period between 2018 to 2035, as predicted by ElecSim. CCGT, onshore, and reciprocal gas engines are invested in less over the time period, as shown by Figures 4.23a,4.23d,4.24a respectively. While coal, offshore, nuclear and photovoltaics all exhibit increasing investments. It is hypothesised that coal and nuclear increase in investment due to their dispatchable nature. While onshore, non-dispatchable by nature, become a less attractive investment when compared to the other technologies. CCGT and reciprocal gas engines may have decreased in capacity over this time, due to the increase in coal. This could be because of the large consistent errors in prediction accuracy that meant that reciprocal gas engines were perceived to be less valuable. Figure 4.24b shows an increase in relative mean carbon emitted with mean absolute error of the predictions residuals. The reason for an increase in relative carbon emitted could be due to the increased output of utility of the reciprocal gas engine, coal, and decrease in offshore output. Reciprocal gas engines are peaker plants and, along with coal, can be dispatched. By being dispatched, the errors in predictions of demand can be filled. It is therefore recommended that by improving the demand prediction algorithms, significant gains can be made in reducing carbon emissions. From our results, it can be seen that different algorithms yield differing prediction accuracies. Online models can result in a decrease in 30% of prediction error on the best offline models. We calculated this by comparing the MAE for Extra Trees to the MAE for the Box-Cox regressor. We, therefore, recommend the use of online machine learning for predicting electricity demand in a day-ahead market.
Similar to our assumptions, the online learning algorithms were able to outperform the offline models. This is due to the non-stationary nature of the data. An online method is able to use the most up-to-date knowledge of the complex system of energy demand. For instance, a certain year may have a particularly warm winter when compared to previous years, reducing the amount of electricity used for heating. However, contrary to our assumptions, the online linear regression techniques outperformed the online machine learning techniques. This may be due to their simpler nature and ability to learn from a smaller subset of new data as opposed to relying on a large historic subset. For the offline models, the best performing algorithms were the decision tree approaches such as extra trees and random forests. This is a similar outcome to our previous work, which showed that the best performing method for demand forecasting were random forests [131]. Contrary to our assumptions, however, the lasso and ridge regression outperformed the machine learning techniques support vector regression and multilayer perceptron. This may be due to the ability of feature selection by lasso and ridge regression, which only uses the most important features. To the best of our knowledge, more work has been done using offline learning to predict electricity demand. This may be due to the additional complexity of running online algorithms, and a smaller number of available models to run in an online fashion. In terms of computing power, finding the optimal input parameters, hyperparameters and models to use can be a large undertaking. This is due to the exponential growth of the number of choices that can be made. This can be an issue where accuracy is of importance, especially when the data changes over time, meaning it may be necessary to retest previous results. However, due to the financial and sustainability implications, we believe the trade-off between compute time and accuracy is balanced towards compute time. There are also large implications if the model were to break at a certain point in time. We, therefore, recommend the reliance on multiple well-performing models, as opposed to solely the best performing model at any one time. That is, qualitatively compare and contrast models. However, a quantitative approach could be undertaken here. For example, taking the mean or median of decisions of various outputs of models. In addition, the time it takes to make a prediction using the algorithm is much smaller than the time you have to make the decision. That is, where an algorithm can be run in seconds, we have hours to submit bids onto the day ahead market. This therefore gives the user the ability to use these predictions in a real setting. For training time and prediction time, there is often a trade-off between training and predicting. For instance, the k-nearest neighbours is fast to train, but slow to sample from. Therefore stakeholders must make a decision based upon accuracy, speed of training and sampling. Additionally, the impact on the broader electricity market has been shown to be significant. Principally, the investment behaviours of generation companies change as well as the dispatched electricity mix. The relative mean carbon emitted over this time period increases, due to an increase in the utilisation of coal and reciprocal gas engines, at the expense of offshore wind. This work shows the importance for generator companies to predict electricity demand with high accuracy. Without this work, it is possible to have a sub-optimal electricity mix over the long-term from the perspective of cost minimisation. The generator companies can use these results to improve their predictions, and policy makers can use these results to ensure that a framework is in place to provide companies with high accuracy forecasts. The limitations of this work, from the perspective of a generator company or policy maker, is that it does not provide a fool-proof way of improving accuracy of predictions with online learning. Rather, a methodology to follow to improve results. In addition, we do not model the impact of weather for solar and onshore renewable energy sources. This may have a long-term significant impact on the electricity mix. Particularly with climate change effects, which may change weather patterns. The availability of high granularity data produced by the smart grid enables network operators to gain greater insights into their customer behaviour and electricity usage. This enables them to improve customer experience, utility operations and power management. In the first piece of work, we demonstrated that implementing the k-means clustering algorithm to group similar customers improved the accuracy of every one of the different models tested. Distinct models were trained for each of the clusters and the individual forecasts aggregated for the total aggregated forecast. It was found that Random Forests outperformed the other models at all levels of clustering and that the optimum number of clusters was 4. Whilst the dataset used focused on residential data it is expected that applying a similar clustering technique on commercial properties would have a similar effect. Next, we evaluated 16 different machine learning and statistical models to predict electricity demand in the UK for the day-ahead market. Specifically, we used both online and offline algorithms to predict electricity demand 24 hours ahead. We compared the ability for the offline models: lasso regression, random forests, support vector regression, for both online and offline learning: linear regression, multilayer perceptron and for just online learning: the Box-Cox transformation and the passive aggressive regressor, amongst others. The Box-Cox, as well as the passive aggressive regressors, were used as online learning algorithms, the multilayer perceptron and linear regression were used as both, whereas the rest were used as offline learning algorithms. We measured the errors and compared these to each model as well as the national grid reserve. We found that through the use of an online learning approach, we were able to significantly reduce error by 30% on the best offline algorithm.
We were also able to reduce our errors to significantly below the national grid's mean and maximum tendered reserve, thus significantly reducing the chances of blackouts. In addition to this, we took these errors, or residuals, and perturbed the electricity demand within the market of the agent-based model ElecSim. This enabled us to see the impact of different error distributions on the long-term electricity market, both in terms of investment and in terms of the electricity mix. We observed that with an increase in prediction errors, we get a higher proportion of electricity generated by coal, offshore, nuclear, reciprocal gas engines and photovoltaics. This could be due to the fact that more peaker and dispatchable plants are required to fill in for unexpected demand. In addition, a higher proportion of intermittent renewable energy sources leads to a higher use of peaker power plants to fill in the gaps of intermittency of wind and solar irradiance. However, by reducing the mean absolute error, we are able to significantly reduce the amount of reciprocal gas engines and coal usage. In future work, we will look into the features that best aid in the forecasting of electricity consumption, try a wider variety of models in an ensemble manner and try different clustering techniques such as Self-Organizing Maps (SOM) to obtain better accuracy measures. We will also compare different prediction error measures. We would also like to trial a different selection of algorithms and statistical models and trial different inputs to the models, for instance, by providing the model with two months worth of historical data as dependent variables. Additionally, we would like to see the impact of predicting wind speed and solar irradiance to see how these impact the overall investment patterns and electricity mix. Carbon taxes have been shown to be an efficient way to aid in a transition to a low-carbon electricity grid. In this work, we demonstrate how to find optimal carbon tax policies through a genetic algorithm approach, using the ElecSim model. To achieve this, we use the NSGA-II genetic algorithm [47] to minimise average electricity price and relative carbon intensity of the electricity mix. We demonstrate that it is possible to find a range of carbon taxes to suit differing objectives. In previous chapters we investigated the building of a long-term agent-based model for electricity markets in the UK, and investigated the impact of short-term predictions on long-term investments. However, a policy maker may have specific objectives within the electricity market that they are overseeing. For example, two key objectives in the current climate are a reduction in carbon emissions, as well as a reduction in electricity prices. A common tool for this are carbon taxes. However, a policy maker is unable to trial all carbon taxes and know the interactions between carbon price, carbon emissions and electricity price. Therefore, in this chapter we propose a machine learning method that carries out this process automatically. This goes beyond the manual work of setting carbon taxes exhibited in Chapter 3. It fits within this thesis by exploring the ability for machine learning to have an impact on the wider electricity market over the long-term. Our results show that we are able to minimise electricity cost to below £10/MWh as well as carbon intensity to zero in every case. In terms of the optimal carbon tax strategy, we found that an increasing strategy between 2020 and 2035 was preferable. Each of the Pareto-front optimal tax strategies are at least above £81/tCO2 for every year. The mean carbon tax strategy was £240/tCO2. Whilst this work was undertaken for the UK, it could be applied elsewhere. This chapter is structured as follows: we introduce our work in Section 5.1. Section 5.2 covers examples of optimisations using genetic algorithms and different carbon strategies. Section 5.3 details the optimization techniques applied. We present our results in Section 5.4, and conclude in Section 5.5. The work in this chapter has been published in [135]. In this work, we use the electricity market agent-based model ElecSim to find an optimum carbon tax policy [129]. Specifically, we use a genetic algorithm to find a carbon tax policy to reduce both average electricity price and the relative carbon density by 2035 for the UK electricity market. We compare this optimal strategy to the carbon tax policy of the UK British government. Carbon taxes have been shown to quickly lower emissions and lower the costs to the public [250]. Carbon taxes are able to send clear price signals, as opposed to a cap-and-trade scheme, such as the EU Emissions Trading System (ETS), which has been shown to be unstable [250]. In this work, we use the reference scenario projected by the UK Government's Department for Business & Industrial Strategy (BEIS) with model parameters calibrated by Kell et al. [53,134]. This reference scenario projects energy and emissions until 2035. We undertake various carbon tax policy interventions to see how we can reduce relative carbon density whilst at the same time, reduce the average electricity price. We did not model the EU ETS within ElecSim due to the significant additional complexity required to model it in a simulation. However, the carbon tax strategy that we took allows for policy makers to understand, in principle, the range of carbon tax price that are required. This information is directly relevant to both an ETS scheme or a carbon tax approach, as these schemes can be designed with the appropriate range to achieve the goals of the policy makers. The parameter space we optimise over is the carbon tax price over a 17 year period from 2018 to 2035. The carbon price is used to influence the objectives of average electricity price and relative carbon intensity in 2035. Grid and random search are approaches which trial parameters at evenly distributed spaces and random spaces respectively. These approaches are often inefficient, however, and require an increased number of simulations due to their static nature.
They are inefficient due to the likelihood that they will continue to search a poor parameter space area, even with consistent negative feedback. Genetic Algorithms, in contrast, use an evolutionary computing approach to find global optimal solutions faster, however, this is not guaranteed, as there are some scenarios where grid search or random search may be quicker to find an optimal solution. This is of particular importance in cases with a large number of parameters or in simulations which require a long compute time, which is the case for ElecSim. In order to optimise over two potentially competing objectives, i.e. average electricity price and relative carbon intensity, we use the Non-Dominated Sorting Genetic Algorithm II (NSGA-II) [47]. The NSGA-II algorithm can approximate a Pareto frontier [185,222]. A Pareto frontier is a curve in which there is no solution which is better than another along the curve for different sets of parameters. In this context, better means that a solution is closer to the optimal for a particular combination of objectives. We find that the rewards of average electricity price and relative carbon intensity are not mutually exclusive. That is, it is possible to have both a lower average electricity price and a lower relative carbon price. This is due to the low short-run marginal cost of renewable technology, which has been shown to lower electricity prices [182]. The main contribution of this work is to explore carbon tax strategies using genetic algorithms for multi-objective optimisation. Multi-objective optimisation problems are commonplace. In this section, we review multiple applications that have used multi-objective optimisation, as well as explore the literature which focus on finding optimal carbon tax strategies. Similar to our work, Ascione et al. [9] use the NSGA-II algorithm to generate a Pareto front to optimise for two objectives: operating cost for space conditioning and thermal comfort. The aim of their work is to optimise the hourly set point temperatures with a day-ahead planning horizon. A Pareto front is generated, which allows a user to select a solution according to their comfort needs and economic constraints. This work showed a reduction in operating costs by up to 56% as well as improved thermal comfort. Gorzałczany et al. [90] also apply the NSGA-II algorithm. However, they apply it to the credit classification problem. The objectives optimised over were accuracy and interpretability when making financial decisions such as credit scoring and bankruptcy prediction. This technique was able to significantly outperform the alternative methods in terms of interpretability while remaining competitive or superior in terms of the accuracy and speed of decision making in comparison with the existing classification methods. Ma et al. [159] use the multi-objective artificial immune optimisation algorithm for land use allocation (MOAIM-LUA model). They balance land use supply and demand based on the future dynamic demands from different land-use stakeholders in the region at the macro-level in Anlu County, China. The objectives to optimise were economic and ecological benefits. They found that for this application, they were able to obtain better solution sets than the NSGA-II algorithm. In our previous work, not presented here, we use the NSGA-II algorithm to optimise average overhead and energy consumption of a condor system [132]. We use the genetic algorithm to trial different parameters of a Q-learning reinforcement learning algorithm, which acted as a job scheduler. We found that we were able to generate a Pareto-front which would allow stakeholders to select an optimum for their use case. Whilst these works use a similar approach to solving an optimisation problem using the NSGA-II algorithm, they optimise a different model and application. In this section, we explore different strategies employed in the literature to analyse the benefits and consequences of a carbon tax. To the best of our knowledge, we are the first to employ a multi-objective optimisation algorithm to minimise average electricity price and relative carbon density. Levin et al. [146] use an optimisation model to analyse market and investment impacts of several incentive mechanisms to support investment in renewable energy and carbon emission reductions. Carbon tax was found to be the most cost-efficient method of reducing emissions. Zhou et al. [256] construct a social welfare model based on a Stackelberg game. The differences and similarities between a flat carbon tax and an increasing block tariff carbon tax are analysed using a numerical simulation. This work shows that an increasing block tariff carbon tax policy can significantly reduce tax burdens for manufacturers and encourage low-carbon production. In contrast to Zhou et al. we trial multiple different carbon tax strategies using a machine learning approach. Li et al. [147] use a hierarchical carbon market scheduling model to reduce carbon emissions. Multi-objective optimisation was applied to discover optimal behaviours for policymakers, customers and generators to minimise the costs incurred by these actors. Our work, however, focuses on the different strategies of carbon tax as opposed to optimal actor behaviour. These works do not use optimisation to find a carbon tax directly, instead seeing the impact of several different options. Our work has the ability to trial a vastly greater number of carbon tax strategies, to hopefully find an optimum. Multi-objective optimisation allows practitioners to overcome the problems with optimising multiple objectives with classical optimisation techniques, such as the genetic algorithm used in Chapter 3. Multi-objective optimisation algorithms are able to generate Pareto-optimal solutions as opposed to converting the multiple objectives into a single-objective problem. A singleobjective problem assumes that there is only a single optimum, or combination of results, and that other combinations are inferior. This may not be the case, as different solutions are superior for a different set of circumstances. A Pareto frontier is made up of many Pareto-optimal solutions which can be displayed graphically. A user is then able to choose between various solutions and trade-offs according to their wishes. The NSGA-II algorithm, a multi-objective genetic optimisation algorithm, is able to generate a Pareto frontier in a single optimisation run. In the following sub-sections, we detail the NSGA-II algorithm.
For an overview of the standard Genetic Algorithm, please refer to subsection 3.3.4. NSGA-II [47] is efficient for multi-objective optimisation on a number of benchmark problems and finds a better spread of solutions than Pareto Archived Evolution Strategy (PAES) [142] and Strength Pareto EA (SPEA) [257] when approximating the true Pareto-optimal front. The majority of multi-objective optimisation algorithms use the concept of domination during population selection [28]. A non-dominated algorithm, however, seeks to achieve the Pareto-optimal solution. This is where no single solution should dominate another. An individual solution x 1 is said to dominate another x 2 , if and only if there is no objective of x 1 that is worse than the same objective of x 2 and at least one objective of x 1 is better than the same objective of x 2 [14]. Non-domination sorting is the process of finding a set of solutions which do not dominate each other and make up the Pareto front. See Figure 5.1a for a visual representation, where f 1 and f 2 are two objectives to minimise and L1, L2 and L3 are dominated layers. In this section, we define the processes used by the NSGA-II algorithm to determine which solutions to keep:

We assume that there are M objective functions to minimise, and that x 1 = {x 1 j } and x 2 are two solutions. x 1 j < x 2 j implies solution x 1 is better than solution x 2 on objective j. A solution x 1 is said to dominate the solution x 2 if the following conditions are true:

1. The solution x 1 is no worse than x 2 in every objective. I.e. x 1 j ≤ x 2 j ∀ j ∈ {1, 2, . . . , M}. 2. The solution x 1 is better than x 2 in at least one objective. i.e. ∃ j ∈ {1, 2, . . . , M} s.t. x 1 j < x 2 j . Next, each of the solutions are ranked according to their level of non-domination. An example of this ranking is shown in Figure 5.1a. Here, f 1 and f 2 are the objectives to be minimised. The Pareto front is the first front. All of the solutions in the Pareto front are not dominated by any other solution. The solutions in layer 1, L1, are dominated only by those in the Pareto front, and are non-dominated by those in L2 and L3. The solutions are then ranked according to their layer. For example, the solutions in the Pareto front are given a fitness rank (i rank ) of 1, solutions in L1 have an i rank of 2. (i distance ) is calculated for each solution. This is the average distance between the two closest points to the solution in question. (≺ n ) is used to ensure that the final frontier is an evenly spread out Pareto-optimal front. This is achieved by using the two attributes: (i rank ) and(i distance ). The partial order is then defined as: i ≺ n j if (i rank < j rank ) or ((i rank = j rank ) and (i distance > j distance )) [47]. This order prefers solutions with a lower rank i rank . For solutions with the same rank, the solution in the less dense area is preferred. Similarly to the standard GA presented in subsection 3.3.4, a population P 0 is created with random parameters. The solutions of P 0 are then sorted according to non-domination. The child population C ′ 1 of size N is then created by binary tournament selection, recombination and mutation operators. In this case, tournament selection is the process of evaluating and comparing the fitness of various individuals within a population. In binary tournament selection, two individuals are chosen at random, their fitnesses are evaluated, and the individual with the better fitness is selected [2]. Algorithm 2 NSGA-II algorithm [47] 1: R t = P t ∪C ′ t combine parent and child population 2: F = fast-non-dominated-sort (R t )

where F = (F 1 , F 2 , . . .) 3: P t+1 = / 0 4: while do |P t+1 < N| 5:

Calculate the crowding distance of (F i ))

P t+1 = P t+1 ∪ F i 7: end while 8: Sort(P t+1 , ≺ n ) sort in descending order using ≺ n 9: P t+1 = P t+1 [0 : N] select the first N elements of P t+1 10: Q t+1 = make-new-population(P t+1 ) using selection, crossover and mutation to create the new population Q t+1 11:

R t has a size of 2N. R t is then sorted according to non-domination. A new population is then formed P t+1 . Solutions are added from the sorted R t in order of non-domination. Solutions are added until the size of P t+1 exceeds N. The solutions from the last layer are prioritised based on having the largest crowding distance [47]. This process is shown in Figure 5.1b, which is repeated until the termination condition is met. A termination condition could be: no significant improvement over X iterations or a specified number of iterations have been performed. The full procedure is detailed formally by Algorithm 2.

electricity. This high degree of freedom enables a high number of strategies to be trialled due to its non-parametric nature. This, however, comes with a large search space requiring a large number of iterations. To reduce the search space for the carbon strategy, we also trial a linear carbon strategy, of the form:

where p c is the carbon price, y t is the year, a 1 is the gradient or first attribute and a 2 is the intercept or second attribute. The constraints of the optimisation problem are that a 1 is bound by -14 and 14, and a 2 by 0 and 250.
These bounds are chosen to ensure that the carbon price does not exceed ∼£500 in the year 18 (2035) and is greater than about -£250, as well as ensuring that the carbon tax in the first year is greater than £0 but smaller than £250. The bounds for a 1 was chosen to make the mathematics simpler, whilst remaining in range. In this section, we explore the results of the optimisations, the optimum carbon strategies and the resultant electricity mixes. Figure 5.2 displays the development of the genetic algorithm against the rewards, relative carbon density and average electricity price. Darker colours display higher generation numbers. The first generation shows a widespread in relative carbon density and average electricity price. However, over successive generations, the solutions converge to a relative carbon density of 0 and an average electricity price under £10MW/h. The rewards of relative carbon density and average electricity price are not mutually destructive. This could be seen as counter-intuitive, as historically low-carbon electricity generation was more expensive than fossil fuels. However, these results could be shown due to the low short-run marginal cost of renewable energy which reduces both electricity prices and carbon emissions [182]. To understand the optimum carbon strategies, we visualised the parameters that produced the lowest average electricity prices in Figure 5.3. Specifically, we filtered for electricity prices under £5/MWh and displayed the results using a heat map. The darker colours represent a higher density of points. Figure 5.3 displays a general trend, where carbon tax starts at ∼£100 until the year 2030, where it increases to ∼£200 by 2035. This may be due to the fact that a lower initial carbon tax of ∼£100 encourages investment in low-carbon technologies before the higher rate of ∼£200 comes into force. This higher rate of carbon tax would allow GenCos to outcompete higher carbon-emitting generators over the lifetime of the plants. .4 displays the development of the genetic algorithm against the rewards: relative carbon density and average electricity price. Similarly to the non-parametric carbon policy shown in Figure 5.2, the first generation shows a wide spread of results. However, the spread is smaller than that of the linear carbon policy. This may be due to the fact that it is easier for the GenCos to predict the carbon policy, which increases confidence in the NPV calculations. The linear carbon policy also converges to a relative carbon density of 0, and an average electricity price smaller than £10MW/h. Figure 5.5 compares the distributions of average electricity price for both techniques. Both methods show improvements as the number of generations of the genetic algorithm increase. The linear policy, however, is able to more quickly converge to a low average electricity price, with a mode of ∼£5.4MW/h. The non-parametric policy has a number of poorer performing parameters, and Generation Number 4 has a bimodal distribution, with a mode of ∼£6.3MW/h. Figure 5.6a displays the linear carbon policies which had an average electricity price under £4.5MW/h. There is no single 'optimum' carbon policy; a range of policies are able to achieve low carbon and a low average electricity price. We explore the electricity mix generated of three different strategies shown in Figure 5.6b. We selected the highest, lowest, and the lowest flat carbon strategy to show a range of possible strategies. This means that whilst a single carbon tax may be sought, in reality there are a number of different strategies which each perform equally well. It is therefore up to stakeholders and decision makers to select a strategy within the bounds presented here. For example, a government may decide to impose the lowest carbon tax policy, to avoid impacting different groups and stakeholders. Or, they may choose a flat carbon tax strategy to prevent an unwelcome increase. Figure 5.7 displays the generated electricity mixes for each of the selected strategies. To generate these figures, we ran 80 scenarios to capture the variability between scenarios. Whilst there does not seem to be a significant difference between scenarios, with solar providing ∼60% of the electricity mix by 2035, there is an observable difference with the other generator types. The 'highest' carbon strategy exhibits a higher uptake in nuclear, possibly due to the fact that nuclear becomes more competitive when compared to coal or gas. The 'lowest' carbon strategy shows a higher uptake in Combined Cycle Gas Turbines (CCGT) during the years of 2026 to 2031 as it outcompetes nuclear. The 'flat' carbon policy shows a higher percentage of solar energy than any of the other scenarios, albeit with a lower percentage of nuclear. Onshore wind is shown to be consistent for these scenarios. We compare our optimal carbon tax strategy to a scenario based on the UK Government's policy in this section from 2018 to 2034. Whilst it is not possible to know the future carbon tax strategy over such a long time horizon, we used a naive model approach to project a static carbon tax strategy. That is, the strategy is maintained at the current level of £18.08 throughout the projected horizon. We chose this level, due to the carbon price at the time of writing. We ran 40 simulations to capture the variance of the electricity mix. The work in this subsection is additional work to that which was published in [135]. Figure 5.8 displays the resultant electricity mix of this carbon tax strategy. A significant difference can be seen when compared to the optimal carbon strategy. 50% of the electricity mix is provided by nuclear energy by 2034. This is almost double that in the optimal carbon tax strategy. Solar, the second-largest source of electricity, provides 30% of electricity supply, with CCGT providing ∼15%. This is a marked difference to the electricity mix shown by Figure 5.7, where solar provides over 50% of electricity supply, followed by nuclear, which provides ∼26%. A larger variance can be seen in this scenario when compared to the optimal carbon tax.
This may be because of the less defined differences between generator costs due to the lower carbon price, which does not distinguish so strongly between carbon-emitting and non-carbon emitting generators. Table 5.1 displays the average electricity price and relative carbon density of this scenario. As is expected, both average electricity price and relative carbon density are above that of the optimal carbon price scenario. This is because of the high short-run marginal cost of nuclear, and the carbon intensity of CCGT. These results show the importance of carbon tax in deciding the electricity mix, where a difference is shown between high carbon taxes and low-carbon taxes. However, solar continues to do well without a carbon tax, as does nuclear, due to the high nuclear subsidy provided. These results can be used by policy makers who require a range of possible carbon tax levels to achieve certain objectives. These results do not prescribe a single solution which should be used, rather we present a range of different options and outcomes which could be chosen. The limitations of this work are the limited scenarios that were explored. For instance, different electricity demand trajectories could be explored. This would provide the policy makers with an increased level of confidence that the carbon tax policies would be robust in the face of uncertainty. In this work, we have demonstrated that it is possible to use the genetic algorithm technique NSGA-II to optimise carbon tax policy using an electricity market agent-based model. We trialled a non-parametric carbon policy by allowing the genetic algorithm to optimise a carbon price for each year. These results showed us that a linear carbon tax might be appropriate. We then used a linear model as a carbon tax policy to reduce the total number of parameters for the genetic algorithm to optimise. We were able to show that a range of linear carbon taxes were able to achieve both low average electricity price and a relative carbon intensity of zero in 2035. By exploring three different carbon tax policies, we saw that ∼60% of electricity consumption in the UK would be provided by solar. The difference between these 'optimal' carbon tax policies was largely shown by competition between CCGT, coal and nuclear. This was largely due to the low short-run marginal cost of solar and nuclear energy, which means that they are often dispatched ahead of the fossil-fuel based generators. CCGT and coal, however, are useful for filling demand when there is low solar irradiance. Additionally, we ran a carbon tax scenario based on that of the UK Government's. We found that an optimal carbon tax strategy had a much lower average electricity price due to the low short-run marginal cost of renewable energy. We also showed that we were able to achieve a low relative carbon density through a higher carbon tax. The majority of the electricity was provided by nuclear, which, in this scenario, had a high subsidy. Therefore, we believe that a lower carbon density, and lower average electricity price can be obtained without subsidies, but with a higher carbon tax. In future work, we would like to try additional scenarios with varying future generation costs and calculate a sensitivity analysis to carbon taxes. In addition to this, we would like to model the uncertain reactions by consumers and generation companies with regards to carbon taxes. The linear carbon tax approach is an introductory approach which can be expanded upon. In addition, we would like to trial this approach, as well as the future work to other countries, not just the United Kingdom. Additionally, we believe that this work could allow for optimisation of other policies, and factors within the simulation. For example, it would be possible to optimise for a range of possibilities for subsidies, market designs and R&D investment. Questions such as, should solar photovoltaics take investment priority for both R&D and installation when compared to offshore wind in parts of the US. We believe that we have created a framework in which different policy options can be explored, and the use of computational optimisation can trial many differing variants. Under perfectly competitive electricity markets, generator companies (GenCos) tend to bid their short-run marginal costs (SRMC) when bidding into the day-ahead electricity market. However, electricity markets are often oligopolistic, where a small subset of GenCos provide the majority of the capacity to the market. Under these conditions, it is possible that the assumption that GenCos are price-takers does not hold. That is, large GenCos artificially increase the price of electricity to gain an increased profit using their market power. If they were price-takers, they would have to accept the competitive price set by the market. Reduced competition within electricity markets can lead to higher prices to consumers, with no societal benefit. It is, therefore, within the interests of the consumer and that of government to maintain a competitive market. Low energy costs enable innovation in other industries reliant on electricity, and in turn, make for a more productive economy. In this work, we explore the effect of total control over capacity on electricity prices. Specifically, we model different sizes of GenCos and groups of colluding GenCos, to bid strategically to maximise their profit using a reinforcement learning algorithm. This is in contrast to the strategy of bidding using the SRMC of their respective power plants, which would occur under perfect market conditions. To model this, we use deep reinforcement learning (RL) to generate a bidding strategy for GenCos in a day-ahead market. These GenCos are modelled as agents within the agent-based model, ElecSim [129,134]. We use the UK electricity market instantiated on 2018 as a case study, similar to our work in [130]. That is, we model each GenCo with their respective power plants in the year 2018 to 2019. In total, we model 60 GenCos with 1,085 power plants between them. It is possible, however, to generalise this approach and model to any other decentralised electricity market.
We use the Deep Deterministic Policy Gradient (DDPG) deep RL algorithm, which allows for a continuous action space [150]. Conventional RL methods require discretisation of state or action spaces and therefore suffer from the curse of dimensionality [254]. As the number of discrete states and actions increases, the computational cost grows exponentially. However, too small a number of discrete states and actions will reduce the information available to the GenCos, leading to sub-optimal bidding strategies. Additionally, by using a continuous approach, we allow for GenCos to consider increasingly complex bidding strategies. Other works have considered a simplified model of an electricity market by modelling a small number of GenCos or plants [65,229]. We, however, model each GenCo as per the UK electricity market with their respective power plants in a day-ahead market. In addition, further work focuses on a bidding strategy to maximise profit for a GenCo. However, in our work, we focus on the impact that large GenCos, or colluding groups of GenCos, can have on electricity price. Our approach does not require GenCos to formulate any knowledge of the data which informs the market-clearing algorithm or rival GenCo bidding strategies, unlike in game-theoretic approaches [239]. This enables a more realistic simulation where the strategy of rival GenCos are unknown. This work fits into the wider scope of this thesis through using machine learning to have an impact on the wider electricity market. In addition, it utilises the work carried out in Chapter 3 for an additional application. It is true that the long-term components of ElecSim are not required for this work, but utilising every aspect of ElecSim was not the central aim of this thesis. Intelligent bidding strategies for day-ahead electricity markets can be divided into two broad categories: simulation and game-theoretic models. Agent-based models (ABMs) allow for the simulation of heterogenous irrational actors with imperfect information. Additionally, ABMs allow for learning and adaption within a dynamic environment [65]. Game-theoretic approaches may struggle in complex electricity markets where Nash equilibriums do not exist [239]. Here, we explore game-theoretic approaches. Kumar et al. [236] propose a Shuffled Frog Leaping Algorithm (SFLA) to find bidding strategies for GenCos in electricity markets. SFLA is a meta-heuristic that is based on the evolution of memes carried by active individuals, as well as a global exchange of information among the frog population. They test the effectiveness of the SFLA algorithm on an IEEE 30-bus system and a practical 75-bus Indian system. A bus in a power system is a vertical line which several components are connected in a power system. For example, generators, loads, and feeders can all be connected to a bus, an example is shown in Figure 6.1. Kumar et al. find superior results when compared to particle swarm optimisation and the genetic algorithm with respect to total profit and convergence with CPU time. They assume that each GenCo bids a linear supply function, and model the expectation of bids from rivals as a joint normal distribution. In contrast to their work, we do not require an estimation of the rivals bids. Wang et al. [239] propose an evolutionary imperfect information game approach to analysing bidding strategies with price-elastic demand. Their evolutionary approach allows for GenCos to adapt and update their beliefs about an opponents' bidding strategy during the simulation. They model a 2-bus system with three GenCos. Our work, however, models a simulation with 60 GenCos across the entire UK, which would require a 28-bus system model [19]. Our work differs from both of these approaches, in that their work investigates what each GenCo should do, but not what would happen to the entire market from a macro perspective. Strategic bidding within Electricity Markets Fig. 6.1 Example 30 bus system. Next, we explore simulation based approaches, which utilise reinforcement learning to make intelligent bidding decisions in electricity markets. RL is a suitable method for analysing the dynamic behaviour of complex systems with uncertainties. RL can, therefore, be used to identify optimal bidding strategies in energy markets [252]. Simulations are often used to provide an environment for the reinforcement learning algorithm, which is true for the following papers reviewed here. Aliabadi et al. [65] utilise an ABM and the Q-learning algorithm to study the impact of learning and risk aversion on GenCos in an oligopolistic electricity market with five GenCos. They find that some level of risk aversion is beneficial, however excessive risk degrades profits by causing an intense price competition. Our work focuses on the impact of the interaction of many GenCos within the UK electricity market. In addition, we extend the Q-learning algorithm to use the DDPG algorithm, which uses a continuous action space. This allows for a continuous action space for bids, providing a higher granularity to the possible actions taken by GenCos. This allows for a more nuanced, and precise understanding of the optimal behaviour. Bertrand et al. [20] use RL in an intraday market. An intraday market, in this context, allows for bids to be made during the day, to aid in balancing supply and demand. Specifically, they use the REINFORCE algorithm to optimise the choice of price thresholds. The REINFORCE algorithm is a gradient-based method. They demonstrate an ability to outperform the traditionally used method, the rolling intrinsic method, by increasing profit per day by 4.2%. The rolling intrinsic method accepts any trade, which gives a positive profit if the contracted quantity remains in the bounds of capacity. In our work, we model a day-ahead market and use a continuous action for price bids. Ye et al. [254] propose a novel deep RL based methodology which combines the DDPG algorithm [150] with a prioritised experience replay (PER) strategy [211]. The PER samples from previous experience, but samples from the "important" ones more often [211]. The PER is a modification of the often used experience buffer, which is a buffer which stores previous transitions and samples uniformly. This helps to reduce the correlations between recent experiences.
They use a day-ahead market with hourly resolution and show that they are able to achieve approximately 41%, 20% and 11% higher profit for the GenCo than the MPEC, Q-learning and DQN methods, respectively. In our work, we instead look at how to prevent GenCos (or sets of colluding GenCos) from forcing higher prices above market rates. Zhao et al. [255] propose a modified RL method, known as the gradient descent continuous Actor-Critic (GDCAC) algorithm. This algorithm is used in a double-sided day-ahead electricity market simulation. Where in this case, a double-sided day-ahead market refers to GenCos selling their supply to distribution companies, retailers or large consumers. Their approach performs better in terms of participant's profit or social welfare compared with traditional table-based RL methods, such as Q-Learning. Our work also looks at improving on table-based methods by using function approximators, however our work looks at the entire electricity market as a whole, instead of from the point of view of GenCos. In this section, we describe the RL methodology used for the intelligent bidding process as well as the simulation model used as the environment. In reinforcement learning (RL) an agent interacts with an environment to maximise its cumulative reward. RL can be described as a Markov Decision Process (MDP). An MDP includes a statespace S , action space A , a transition dynamics distribution p(s t+1 |s t , a t ) and a reward function, where r : S × A → R. At each time step, an agent receives an observation of the current state which is used to modify the agent's behaviour. An agent's behaviour is defined by a policy, π. π maps states to a probability distribution over the actions π : S → P(A ). The return from a state is defined as the sum of discounted future reward:

Where γ is a discounting factor γ ∈ [0, 1]. The return is dependent on the action chosen, which is dependent on the policy π. Where r(s i , a i ) is the reward for state s i and action a i . The goal in reinforcement learning is to learn a policy that maximizes the expected return from the start distribution J

The expected return after taking an action a t in state s t after following policy π can be found by the action-value function. The action-value function is used in many reinforcement learning algorithms and is defined in Equation 6.2. The action-value function defines the expected reward at time t, given a state s t and action a t when under the policy π. An optimal policy can be derived from the optimal Q-values Q * (s t , a t ) = max π Q π (s t , a t ) by selecting the action corresponding to the highest Q-value in each state. Many approaches in reinforcement learning use the recursive relationship known as the Bellman equation, as defined in Equation 6.3:

The Bellman equation is equal to the action which maximizes the reward plus the discount factor multiplied by the next state's value, by taking the action after following the policy in state s t+1 or π(s t+1 ). The Q-value can therefore be improved by bootstrapping. This is where the current value of the estimate of Q π is used to improve its future estimate, using the known r(s t , a t ) value. This is the foundation of Q-learning [242], a form of temporal difference (TD) learning [226], where the update of the Q-value after taking action a t in state s t and observing reward r t , which results in state s t+1 is:

where,

α ∈ [0, 1] is the step size, δ t represents the correction for the estimation of the Q-value function and r t + γ max a t+1 Q(s t+1 , a t+1 ) represents the target Q-value at time step t.

It has been proven that if the Q-value for each state action pair is visited infinitely often, the learning rate α decreases over time step t, then as t → ∞, Q(s, a) converges to the optimal Q * (s, a) for every state-action pair [242]. However, it is often the case that Q-learning suffers from the curse of dimensionality. This is because Q-learning stores the Q-value function in a look-up table. This therefore requires the action and state spaces to be discretised. As the number of discretised states and actions increases, the computational cost increases exponentially, making the problem intractable. Many problems are naturally discretised which are well suited to a Q-learning approach, however this is not always the case. It is not straightforward to apply Q-learning to continuous action spaces. This is because in continuous spaces, finding the greedy policy requires an optimisation of a t at every time step. Optimising for a t at every time step would be too slow to be practical with large, unconstrained function approximators and nontrivial action spaces [150]. To solve this, an actor-critic approach based on the deterministic policy gradient (DPG) algorithm is used [216]. The DPG algorithm maintains a parameterized actor function µ(s|θ µ ) which specifies the current policy by deterministically mapping states to a specific action. The critic Q(s, a) is learned using the Bellman equation as in Q-learning. The actor is updated by applying the chain rule to the expected return from the start distribution J with respect to the actor parameters:

This is the policy gradient. The policy gradient is the gradient of the policy's performance. The policy gradient method optimises the policy directly by updating the weights, θ , in such a way that an optimal policy is found within finite time. This is achieved by performing gradient ascent on the policy and its parameters π θ . Introducing non-linear function approximators, however, means that convergence is no longer guaranteed. However, these function approximators are required in order to learn and generalise on large state spaces.
The Deep Deterministic Gradient Policy (DDPG) modifies the DPG algorithm by using neural network function approximators to learn large state and action spaces online. A replay buffer is utilised in the DDPG algorithm to address the issue of ensuring that samples are independently and identically distributed. The replay buffer is a finite-sized cache, R. Transitions are sampled from the environment through the use of the exploration policy, and the tuple (s t , a t , r t , s t+1 ) is stored within this replay buffer. R discards older experiences as the replay buffer becomes full. The actor and critic are trained by sampling from R uniformly. A copy is made of the actor and critic networks, Q ′ (s, a|θ Q ′ ) and µ ′ (s|θ µ ′ ) respectively. These are used for calculating the target values. To ensure stability, the weights of these target networks are updated by slowly tracking the learned networks. Pseudo-code of the DDPG algorithm is presented in Algorithm 3. Algorithm 3 DDPG Algorithm [150] 1: Initialize critic network Q(s, a|θ Q ) and actor µ(s|θ µ ) with random weights θ Q and θ µ 2: Initialize target network Q ′ and µ ′ with weights θ Q ′ ← θ Q , θ µ ′ ← θ µ 3: Initialize replay buffer R 4: for episode=1,M do Select action a t = µ(s t |θ µ ) + N t according to the policy and exploration noise, N t

Execute action a t and observe reward r t and new state s t+1 10:

Update critic by minimizing the loss:

Update the actor policy using the sampled policy gradient:

Update the target networks:

end for 17: end for

We utilized the long-term electricity market agent-based model, ElecSim [129,134] discussed in Chapter 3. The model was run using a short term approach by only iterating through a single year (2018), composed of eight representative days, each of 24-time steps. In this work, we explore whether large GenCos, or group of GenCos, can manipulate the price of the electricity market through virtue of their size. We achieve this by allowing a subset of GenCos to bid away from their SRMC and allow them to learn an optimal bidding strategy for maximising their income. The GenCo agents adopt a DDPG RL algorithm to select their bids. This is to explore whether large GenCos, or a group of GenCos can manipulate the price of the electricity market through market power. The remaining GenCos, which fall outside of this group, maintain a bidding strategy based upon their SRMC. For the purpose of this work, we do not consider flow constraints within the electricity mix. This is because we model the entire UK with more than one thousand generators, and many nodes and buses. This would make the optimisation problem intractable for the purpose of our simulation, especially when considering the many episodes required for training. It takes ∼125 seconds to run a single year in the simulation, or episode with our current setup. By increasing the simulation time further, we would make the compute time intractable due to the many episodes required for reinforcement learning to learn an effective policy. Additionally, we must train several reinforcement learning policies to account for each GenCo and market cap. Therefore a simulation that takes ∼2 minutes to run, which needs to be executed one thousand times takes ∼33 hours to complete, multiplied by 12 different GenCos takes ∼16 days, and with two market cap scenarios it would take ∼1 month. To parametrise the simulation, we used data from the United Kingdom in 2018. This included 1,085 electricity generators and power plants with their respective GenCos. The data for this was taken from the BEIS DUKES dataset [49]. The electricity load data was modelled using data from Elexon portal and Sheffield University [62]; offshore and onshore wind and solar irradiance data from renewables.ninja by Pfenninger et al. [189]. It would be possible to adopt this approach to other decentralised markets in other countries. By modelling bidding decisions as an RL algorithm, we hoped to observe the ability for RL to find the point at which market power artificially inflates electricity prices. To achieve this, we chose the six largest GenCos in the UK, as well as a smaller GenCo as a control. Groups of GenCos are modelled as a single GenCo with a single RL strategy for the purpose of this work. Table 6.1 displays the groups of GenCos, as well as individual GenCos, with their respective capacity and number of plants. For the reinforcement learning problem we have the following tuple: (s t , a t , r t , s t+1 ), where (s t , s t+1 ) is the state at time t and t + 1 respectively, a t is the action at time t and r t is the reward at time t. For our problem the state space is given by the tuple shown in Equation 6.7: s t = (H i , D i , p gas , p coal , p C02 , p c ), (6.7) where H i is the segment hour to bid into at timestep i, D i is the demand of the segment hour at timestep t, p gas is the price of gas, p coal the price of coal, p C02 is the carbon tax price, and p c is the clearing price. We set the reward, r t to be the average electricity price of that time step, p avg . For the action space, a t , we modelled two scenarios. Where there was a price cap of £150/MWh and £600/MWh. Only these two values were chosen to reduce computational load. We chose £150/MWh as a reasonable price cap that may be introduced by a Government. This was roughly double the average accepted price in 2018, therefore allowed for higher prices in times of high demand or low supply.
The £600/MWh was chosen to simulate a realistic unbounded price cap. This enabled us to see the price that an equilibrium is reached within a market with agents with market power. For this work, we assume that the action space a t only bids price, and not how much capacity to bid on the market. We assume this to reduce the dimensionality of a t , and simplify the training process. In this work, we assume that the GenCo groups have no information about the generation capacity, marginal cost, bid prices or profits of other GenCos [65]. They learn the maximum profit that can be made through experience within a particular market. We assume this because in real-life GenCo groups have little information on their competitors sensitive bidding data. If they were to have perfect information on all their competitors, they would be able to devise a perfect strategy which would always maximise their profit. In this section, we detail the results of the RL algorithm, and the effect that capacity has on average electricity price within the UK. Our approach could be generalised to any other decentralised electricity market in other countries. Figures 6.2 and 6.3 show the rewards over a number of time steps for the unbounded and bounded cases respectively. Figure 6.2 shows a clear difference between agents which use the DDPG RL strategy and have a large capacity (green and yellow) compared to those which have a smaller capacity (dark purple). The axis in Figure 6.2 are much larger than those of 6.3, highlighting the effect of market power on an unbounded market. q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q 0 50 100 The average electricity price for a capacity below 10,000MW, or ∼ 35% of total capacity, remains stable between £70/MWh and £100/MWh. This range may be due to the stochasticity in calculating the weights for the DDPG algorithm. The average electricity price does not change over the time steps or training. We, therefore, hypothesise that there is no market power as long as an individual GenCo owns below ∼35% of total electrical capacity. On the other hand, once the capacity of a GenCo or groups of GenCos is above 30,000MW, there is a significant increase in the average price for capacity. The average electricity price for capacity falls between the range of £170/MWh and £220/MWh. Figure 6.4 displays the capacity controlled by the agents that use the RL strategy versus the average electricity price for the unbounded case. The colour displays the number of steps. The step-change, as shown in Figure 6.2 can be seen clearly here, with agents with a capacity larger than ∼25,000MW causing a step change in electricity price. Electricity prices seems to cluster below ∼10,000MW. However, after this point, the average electricity price begins to increase.
q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q 0 20 40 60    6.3 shows a cluster between ∼£60/MWh and ∼£80/MWh irrespective of the capacity of the agents. This is verified by Figure 6.5. This seems to suggest that setting a lower market cap reduces the ability for generators, irrespective of size, from influencing the electricity price. Figures 6.6 and 6.7 display the actual bids made at the end of training within the electricity market for all of their power plants. The number of bids made by each GenCo changes dependent on the number of plants that they own, with the Orsted GenCo only making eleven bids per segment, and the largest group making 216 bids per segment. Figure 6.6 displays the uncapped scenario (£600/MWh) and 6.7 displays the capped scenario (£150/MWh). Figure 6.6a shows the bids made by the largest group of GenCos as shown in Table 6.1. A bimodal distribution can be seen, where the group of GenCos tend to bid either the maximum or the minimum bid. We hypothesise that they bid the maximum amount as this ensures that the market price is artificially raised, and that they are able to utilise their market power. The minimum price is bid the rest of the time to ensure that generators bids are always accepted, regardless of whether the market price has been artificially raised or not in each respective clearing segment. Figure 6.6b displays the bids of the small company with a market cap of £600/MWh. The small company also seems to have a bimodal distribution; however, bids the higher price more often. This may be due to the fact that it is able to influence the price at certain market segments, and the reward of the higher accepted reward outweighs the times in which it is not accepted on the market segments. Again, bidding low seems to be the strategy in which to take if the GenCo does not believe it will be able to influence the final price. As the market simulated is a uniform pricing market, having a £0 bid accepted does not mean that the GenCo will be paid £0. Rather, the GenCo will be paid the market clearing segment, which is set by the most expensive power plant accepted onto that market segment. Figures 6.7a shows the bids made by the largest group of GenCos in each market segment. It seems to take a similar strategy to that of the large company in the uncapped scenario, as shown in Figure 6.6a. We believe, as similar in the uncapped scenario, that this is due to the market power that this group possesses. Being able to influence the market price enables the GenCo  group to inflate the prices. It also takes a conservative strategy to bid the minimum price allowed, to ensure that the bid is accepted regardless of price. Figure 6.7b displays the strategy of the smallest company in the capped scenario. Here, it seems that the GenCo is unable to influence the price at all, and therefore bids £0/MWh for the majority of the time. This is similar to the expected strategy of GenCos, who tend to bid their short-run marginal cost to ensure that they do not miss out on potential profit.
The short-run marginal cost can often change based upon fuel, carbon and generator type. However, for renewable energy, it is near £0, and for fossil-fuel based plants it is near the cost of fuel and carbon at that point in time. We ran a sensitivity analysis to observe the effects of the market cap on final average accepted bid price. The largest GenCo group used a strategy for this sensitivity analysis. Figure 6.8 displays the results. It seems that whilst the average accepted bid price increased with the capped bid level; there is a significant increase after a market cap of £190/MWh. This may be due to the case that the GenCos begin to outbid the SRMC bidding GenCos at this price point. These results demonstrate a methodology for policy makers to investigate the ability for generator companies to artificially increase the electricity price. It gives quantitive advice on the level of capacity that a generator company can control before it becomes a problem from the perspective of consumers of the electricity market. These results also provide a method to control excessive electricity prices with the provision of a market cap. The limitations from the perspective of the policy maker, is that we do not provide any information on how to prevent an oligopoly from occurring. We only provide advice on what to do once one has occurred. Our results demonstrate the ability for GenCos to artificially increase the electricity price through market power in an uncapped market. Our results have shown that in an uncapped market, any single agent or groups of agents who make bids using the same strategy and information, should have less than ∼10,000MW. This defines the optimal capacity of any single GenCo to have a fair level of competition. After this, the electricity price begins to rise with the same outcome and welfare. It is also worth noting that when the market is capped, the average accepted price does not simply become the capped price. However, if there is an electricity market with a few large or colluding players, it is possible to remove their advantage through the introduction of a price cap. Our results show that whilst average accepted bids increase with market cap level, the value does not increase significantly until £200 is reached. This information and approach can help to inform government policy to ensure fair competition within electricity markets, as well as run the model for their own scenario. It is hypothesised that the findings in this work are generalisable to other decentralised electricity markets in other geographies due to their similar market structures. Whilst the figures presented here may not be the same; we hypothesise that the region of interest will be similar. In this work, we investigated the ability for GenCos to make strategic bids within an electricity market. We did this using the deep deterministic policy gradient (DDPG) reinforcement learning method. We utilised the agent-based model ElecSim to model the UK electricity market. We utilised the DDPG algorithm only for a certain subset of agents, from small individual generation companies (GenCos) to large groups of GenCos. This enabled us to explore the ability for GenCos with a large capacity to artificially increase the price in the electricity market within the UK if they are in control of a sufficiently large generation capacity. Our results show that the optimum level of control of any single GenCo or groups of GenCo is below ∼10,000MW or ∼11% of the total capacity. Above this, prices begin to increase with no real additional benefit to the consumer. After ∼25,000MW, or ∼35% of the total capacity, the prices begin to increase substantially, to ∼£200, over triple the original cost without this market power. The introduction of a market cap of £150 reduces all market power and maintains electricity price at a reasonable level. We found through a sensitivity analysis that the average electricity price in the market over a year remains low with a price cap smaller than £190. However, after this level, the average electricity price begins to increase. Our work has shown the ability for reinforcement learning to learn an optimal bidding strategy to maximise a GenCo's profit within an electricity market. The ability for GenCos to use their market power is also highlighted, and is dependent on electricity generation capacity of the respective GenCo. In future work, we would like to enable GenCos to withhold the capacity on offer to the electricity market. This would enable further market power by reducing competition further. Additionally, we would like to assess the market power in different countries with different market structures and total electricity supply. An other option would be to assess the effect of two large competing GenCos on the market. In this thesis, we presented an open-sourced, long-term agent-based model for electricity markets called ElecSim. We were able to validate the model through two methods: cross-validation using historic data from 2013 and projecting forward, and comparing our scenario to that of the UK Government to 2035. We found positive results in both: we were able to accurately model the transition from coal to gas between 2013 and 2018, and closely match the scenario of the UK Government. For this validation, we used optimisation to find realistic parameters that would generate the desired scenarios. The primary parameter that was optimised was the predicted price duration curve. We found a predicted price duration curve which closely matched the price found in 2018 for the cross-validation using historic data. For the UK government forward scenario, we found a variety of predicted price duration curves, with a few boom and bust cycles in the price of electricity. In addition to designing and developing ElecSim, we used the forward scenario generated to optimise a carbon tax strategy. That is, to reduce both carbon emissions and electricity price from 2018 to 2035. To achieve this we used a multi-objective genetic algorithm.
We found that we were able to reduce both electricity price and carbon emissions through a combined use of solar, nuclear and onshore energy. Further, we predicted electricity demand consumption on two time scales: 30-minutes ahead and a day-ahead to use in ElecSim. We found that we were able to predict electricity demand at these time scales well. Through the use of an online learning method, we were able to reduce the required reserve capacity of the national grid in the UK. We also looked at the long-term impact of poor forecasting errors on the long-term market, and found that poor forecasts led to a high carbon density of electricity grid over the long-term. Our final chapter focuses on the use of deep reinforcement learning to model a bidding strategy of a single generation company or group of generation company using ElecSim. The rest of the generation companies bid based upon the respective generator's short run marginal cost. We found that after a capacity controlled by a generation company of 10,000MW, the generation company with the reinforcement learning bidding strategy was able to influence electricity prices in their favour; effectively demonstrating market power within the electricity market. After a controlled capacity of 30,000MW the average electricity price triples from the equilibrium price. We utilised agent-based models in this work for several reasons:

1. Ability to model scenarios in a way that emerges from the generator companies. 2. The process which emerge a not a black box, unlike in equilibrium models. 3. Optimisation based methods should be interpreted in a normative manner, for example, how policy choices should be carried out. Whereas agent-based models make no assumption on outcomes of scenarios. 4. Ability to model decentralised agents with different desires. 5. Ability to model outcomes which are not in equilibrium, for example boom and bust cycles. An additional reason was for the limited availability of an open-sourced agent-based model, and to ensure that this model is available to the community. Open-sourced models allow for transparency, and to garner greater acceptance and understanding within the wider community. Our contributions are the following:

1. Developed a novel, open-source agent-based model for the electricity market called ElecSim. ElecSim and all related code can be accessed at: https://github.com/alexanderkell/ elecsim. 2. Validated this model by the means of cross-validation, and compared our model to that of the UK Government. 3. Investigated the effect of online learning to improve electricity demand forecasting a day-ahead, and looked at the long-term impacts that this had on the electricity markets. 4. Predicted electricity consumption 30-minutes ahead. 5. Found a variety of optimal carbon tax strategies using genetic algorithm based optimisation and the ElecSim model. 6. Investigated the impact of collusion within a oligopolistic electricity market. In this section we explore the conclusions that arose from this thesis. We explore each of the chapters where we present our primary technical contributions. We place our work in the context of our research question: what challenges can AI and ML tackle, and how do these methods relate back to the wider energy system? Each of the chapters explore a research subquestion, which are highlighted in Chapter 1. We provide answers to these subquestions here. Can a simulation model an electricity market over the long-term? Through the recreation of the salient features of the UK electricity market into an agent-based model, we show that we are able to derive similar outputs to an established, governmental model. Therefore, even though the underlying methodology between these two models are different, we have shown that it is possible to reach similar conclusions. This has important ramifications for future work, as features that could not previously be modelled with traditional, optimisation based models, such as heterogenous generation companies, can be modelled. This finding adds a significantly different tool to the established models, and allows modellers and policy makers to make findings from an alternative kind of methodology. Whilst this does not provide a ground truth model to the energy modelling community, it does provide an additional tool to understand complexity within an electricity market. Is it possible to model the variability of an electricity system? The variability within a modern day electricity system is significant. With the increase in solar photovoltaics and wind turbines a modern day electricity model must be able to model fluctuations in solar irradiance and wind speeds. This variability can change in minutes and therefore adds significant complexity for electricity system models. This is because a high temporal resolution is required to model this variability. To take into account this variability, we used a k-means clustering approach to select representative days in Chapter 3. Through the use of these representative days, we found that we were able to model significant peaks and troughs in demand, wind speed and solar irradiance. This was shown by comparing three different metrics: the normalised root mean squared error, correlation and relative energy error. This was demonstrated in Chapter 3. We recommend that a similar approach is undertaken by other modellers for a realistic representation of the real-world, without compromising on runtime. Can we trust an electricity model's outputs? Fundamentally, the ability to trust an electricity model's outputs is through the verification of its outputs with observed data. For long-term models this becomes a significant challenge because of all the uncertainties inherent in long-term investments. Further, a long-term electricity model produces scenarios and not predictions. This makes this problem even more difficult, as an output scenario could be both possible and realistic, but it does not mean that it is a certainty that it will occur. To answer the subquestion of whether we can trust an electricity model's outputs, we investigated an observed shift in electricity mix from coal to gas from the previous five years. We calibrated our model to find parametric data that would generate the outputs required.
We found that the parametric data required to generate these outputs were realistic, and we could model the shift in electricity mix from coal to gas over this time period. Whilst this approach has limitations, in that it raises an additional question: can short-term validation translate to long-term verification? However, this is the best solution we found without trying to predict many years in the future which is both infeasible and unrealistic. From this work, we recommend that models undertake some form of validation using realworld, observed data. Without this, it is possible that some aspects of the real-world system is not accurately modelled. And so, an iterative process can be used to improve the functioning of the model, with this approach. Do poor short-term forecasts of electricity demand have a long-term impact? Previous work on short-term forecasts of electricity demand has focused on improving the accuracy of such models. However, the long-term impact of these predictions has been studied to a lesser degree. To explore this question, we used a variety of different statistical and machine learning techniques to predict electricity demand. We then took the errors of these predictions and investigated the impact that these errors have on the long-term electricity market by perturbing the electricity demand perceived by the generator companies. Our results show that with an increase in mean absolute error, there is an increase in CCGT, coal, nuclear, photovoltaics and reciprocal gas engines, with a decrease in onshore and offshore wind. Such a scenario changes the electricity mix landscape in the future, and is suboptimal if a low-carbon electricity mix is desired. We recommend that a high amount of research is placed into improving short-term forecasting for electricity demand. Whilst this can be expensive in the short-term, the long-term benefits are significant through the use of a more optimal and efficient system. Is it possible to use an algorithm to set carbon policy? A significant goal for policy makers is to achieve a low-cost and and low-carbon system. However, in practice, this can be a difficult system to achieve. Policy makers have limited levers in which to test, and those levers can have an uncertain impact on the market. To simplify the process of finding a cost and carbon optimal carbon tax in the UK electricity market, we used a multi-objective genetic algorithm, NSGA-II, to vary carbon tax between 2018 and 2035. We found that it is possible to use a genetic algorithm to find a variety of carbon policies to minimise both carbon and cost of the electricity system. These carbon policies can be presented to policy makers to choose from, depending on their requirements and preferences. The recommendations that arise from this work is that a full quantitative analysis should be undertaken for long-term policy options, and that the search space can be explored using algorithms. This enables policy makers to find a variety of different options, from which they can choose from. Whilst the algorithm may not tell you the precise strategy, it can give a range of feasible options, in which policy makers can use their judgement with additional information. Is it possible to limit the power of large generator companies? Electricity markets can often be dominated by a small subset of generator companies. If given enough power, these generator companies can then use strategic bidding to artificially inflate the electricity price for their own benefit. With the use of reinforcement learning, we were able to simulate the strategic bidding behaviour that generator companies may exhibit. We found that if a generator company controls more than 11% of total capacity, it is likely that price inflation will occur. However, we found that such price inflation can be limited through the means of a cap on bid levels. We found that a relatively high bid price cap of £190/MWh prevents price inflation for the UK market. We therefore recommend that policy makers place a price cap if they believe that an oligopoly or monopoly is beginning to form. This can limit the impact that these companies can have on consumers and the overall market. What challenges can AI and ML tackle, and how do these methods relate back to the wider energy system? In this thesis, we explored the different methods that AI and ML can tackle within the British electricity system. Whilst much work has focused on incremental improvements to targeted problems, such as short-term load forecasting or the possibility of using reinforcement learning for short-term bidding, the wider impact on the market has been explored to a lesser extent. We found that AI and ML can be used for a wide range of different application domains, which are not limited to solely improving the accuracy of these domains. The subquestions answered previously in this section attest to this, and demonstrate that large-scale problems and systems can be optimised, improved and modelled with the use of state-of-the-art methods. There remains significant research that can be undertaken as part of this work that we would like to carry out in the future. A major aspect of intermittent renewable energy sources such as wind and solar is their distributed nature. On a geographic scale as large as the UK, weather conditions change. Therefore, at any one time, capacity factors may change across the country. It would, therefore, be beneficial to model this, by integrating a higher temporal resolution into the model. This would increase the complexity of the model, and therefore compute time, however, the optimal placement of intermittent renewable energy sources could be modelled to ensure the maximum supply at times when it was required from such sources. Work could be done to find simplifications that could be made to other aspects of the model to maintain tractability. Additionally, the integration of a larger amount of countries could be carried out. For instance, coupling the UK market with that of Ireland, or with the EU.
Whilst this would increase the compute time, the results would be more inclusive of other markets and enable for a larger temporal resolution still. Another area of interest is predicting the price of electricity in the future. Scenarios could be run to predict this price and integrate it into the model. Whilst increasing compute time, this would allow us to explore a wider range of scenarios. Another aspect is the comparison to further models. The traditional approach of optimisation models offer a variety of "gold-standard" scenarios that we could explore. These comparisons could be undertaken to find weaknesses in both our model as well as other models. An interesting area of research is the area of investment using optimisation or control algorithms such as reinforcement learning. Reinforcement learning suffers from a requirement to have multiple training epochs, and therefore we found this a difficult task to achieve in this work. However, through either speed improvements or use of another algorithm, this could be integrated into our work. Finally, the integration of fuel supply-demand curves could help us in better modelling electricity markets on both a local and global scale. Firstly, land is a limited resource, especially in the United Kingdom, and therefore limits to the amount of solar or wind that can be produced a real. Secondly, on a global scale, as demand for gas or coal reduces there are large effects on price. Firstly this may reduce due to the supply and demand relationship. However, if demand reduces sufficiently, the price may increase, due to the expense of extraction. This may turn out to be a large inflection point, where gas and coal are used much less. 6.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Conclusion 7.1 Thesis summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Future research direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3  Model schema and presentation of various energy models

3

1 Power plant parameters. expensive power plants, or solar will displace fossil-fuel based plants at times of high solar irradiance.