I hereby declare that except where specific reference is made to the work of others, the contents of this dissertation are original and have not been submitted in whole or in part for consideration for any other degree or qualification in this, or any other university. This dissertation is my own work and contains nothing which is the outcome of work done in collaboration with others, except as specified in the text and Acknowledgements. This dissertation contains fewer than 65,000 words including appendices, bibliography, footnotes, tables and equations and has fewer than 150 figures. November 2021

The impacts of global warming on the earth may have profound effects on land and ocean ecosystems [120]. The release of carbon emissions into the atmosphere increases the likelihood of the most severe impacts and increases the likelihood that tipping points are reached, where runaway carbon emissions and average temperature rises are likely. Therefore a transition to a low-carbon energy supply is required to prevent the impacts of climate change. A low-carbon electricity supply is one which releases a lower amount of carbon dioxide over its lifetime than the current, fossil-fuel based system. However, such a transition is complex and contains multiple uncertainties. For instance, what carbon tax should the UK government set over the next 30 years? Are poor short-term electricity demand forecasts locking us in to higher emissions over the long-term? Can we limit the market power of generator companies? And can we rely on these models to make decisions of such importance? Whilst much work has been carried out investigating energy models in different electricity and energy markets, these models are not often fully validated against real-world data. This thesis seeks to validate a novel agent-based model, ElecSim, by calibrating with real-world data. Through this calibration, confidence can be gained in the underlying dynamics of the model and provide policy makers with the opportunity to better understand the system with which they are dealing with. Secondly, much work has been undertaken to understand certain aspects of electricity markets using agent-based models and machine learning. However, this work, often, does not place these findings into a wider context. For instance, whilst a high degree of focus is placed on the ability of reinforcement learning to bid strategically within an agent-based setting, how to limit this behaviour has not been investigated to the same extent. Similarly, machine learning has been used to predict electricity demand at various time intervals. However, the effect that different prediction methods have on the long-term electricity market has not been explored. Finally, machine learning and simulation has the ability to optimise an entire system. However, this ability received much research attention, instead the focus has been on smaller scale changes to models. In this thesis we aim to fill this research gap by first calibrating our model, and secondly reducing electricity price and carbon emissions from the United Kingdom's electricity mix by optimising carbon tax strategies. The central question of this thesis is: how can artificial intelligence (AI) and machine learning (ML) answer fundamental questions of the energy transition using an agent-based model of the electricity system? This thesis aims to go beyond small-scale improvements to agent-based models and answering scope-limited questions by understanding first: what challenges can AI and ML tackle, and secondly: how do these methods relate back to the wider energy system? By taking this approach, we answer multiple subquestions, which are explored below:

1. Can a simulation model an electrcity market over the long-term? Traditional electricity market models mimic the behaviour of centralised actors with perfect foresight and information. Other models which model actors as having imperfect foresight and information lack the ability to model multiple time-steps over a long time horizon. In Chapter 3, a novel open-source agent-based model called ElecSim is presented which challenges these issues. We show that it is possible to create an electricity model which can simulate multiple time-steps over a long-time horizon and generate realistic electricity mixes as model outputs. 2. Is it possible to model the variability of an electricity system? Intermittent renewable energy can produce electricity at both maximum capacity and at zero capacity in short time intervals. It, therefore, becomes important to model these variations in power output over a long-term horizon. Otherwise, the model may overestimate the production of energy from renewables and underestimate the variability of such technologies. This is achieved in Chapter 3 by showing that with representative days, we are able to accurately model an entire year with a reduced computational burden. Without this additional feature, an overestimation of intermittent renewable energy and underestimation of dispatchable generation is observed. markets occurs prior to the time in which the demand must be supplied. However, the long-term effect on the markets of poor forecasts has not been investigated. In Chapter 4 we investigate the long-term impact of poor short-term predictions. We find that poor short-term demand forecasts leads to increased investments in coal, gas and nuclear power, with a reduction in both onshore and offshore wind. 5. Is it possible to use an algorithm to set carbon policy? Setting carbon taxes has been proposed as a solution to reduce our reliance on fossil fuels. However, the impact of such carbon taxes are unknown, as are the optimal strategies from different perspectives. Such a problem can be solved using optimisation based techniques. Here, a solution to finding optimal strategies from the perspective of a benevolent government is presented in Chapter 5. We find that it is possible to find a variety of different carbon tax strategies to minimise both electricity price and carbon emissions, as long as a high carbon price is set (around £200). 6. Is it possible to limit the power of large generator companies? It is known that oligopolies have a negative effect on markets for consumers. However, what has been explored to a lesser degree, is the proportion of capacity that generation company must own before they have market power.
In addition, what would the effect be of a market cap on such electricity markets? Would a market cap reduce the ability for Generation Companies (GenCos) to inflate electricity prices artificially? Chapter 6 investigates this issue. We find that if a generator company, or group of colluding generator companies own over 11% of the total generation capacity, electricity prices start to increase. However, the impact of this market power can be limited through the setting of a market cap. In the case of the UK, a market cap of £190/MWh suffices. Through these questions we not only answer whether AI and ML methods can be used with electricity market agent-based models, but also what is the wider impact of the behaviours on the market. Primarily, in this work, simulation is used as a tool to better understand and make projections for electricity markets. Specifically, in this thesis, the agent-based modelling paradigm is used. This enables us to model generator companies as individual agents, with heterogeneous strategies and characteristics. These agents have access to imperfect information and imperfect foresight. This methodology differentiates this work from the traditional centralised optimisation approach. Agent-based models are critical to model the behaviour of individual actors within an electricity system. Without this distinction, the system must be modelled as a homogenous system, which does not accurately reflect the real world. Through this approach, we hope to learn that it is possible to accurately model the UK's national electricity system with an agent-based approach. Machine learning and statistical techniques are used to make short-term forecasts of electricity demand. We use both deep learning, offline learning and online learning to further improve our methods. Online learning is a machine learning approach which utilises new data to update model weights, and does not require the model to be completely retrained, which is the case for offline learning. In comparison, deep learning utilises neural networks with many different layers. We utilise these methods due to their data-driven approach and strong ability to forecast time-series data. Online learning is used as over the time-periods with which we are forecasting, the underlying time-series changes in structure. Online learning is able to continually use new data points to retrain the model. Deep learning, on the other hand, uses many layers to learn more complex patterns from the training set. Through taking this approach, we hope to learn that it is possible to improve predictions for energy demand data when compared to the traditional machine learning methods. Once our simulation model is built, we are able to answer different questions using several approaches. For example, we perturb the exogenous electricity demand by the error distribution generated by the aforementioned electricity demand forecasting methods. This provides an insight into how small errors can have large impacts on long-term electricity markets in terms of both investments made and generator utilisation. This approach was taken due to its ability to mimic the behaviour of generator companies in a simple manner. We hope to learn what the impact of short-term decisions are on the long-term market. Multi-objective genetic algorithms are used to explore carbon tax policies which will reduce both carbon emissions and average electricity price. We find that we are able to achieve both of these goals by setting a median carbon tax of ∼£200 per tonne of carbon dioxide. This methodology is chosen due to the genetic algorithm's distributed nature. We are able to run the algorithm in parallel and reduce training time significantly. From this, we hope to learn that there is an automatic method to reduce the search space for policy makers when coming up with complex policy in a high parameter space. Finally, we explore the ability for deep reinforcement learning (DRL) to make strategic bidding decisions within a day-ahead electricity market. This work enables us to see the proportion of capacity that must be controlled to artificially inflate the electricity price in the market using market power. Deep reinforcement learning was chosen due to its ability to quickly form a policy on the expected environment. The bidding environment with multiple competing agents is highly complex and difficult to solve through a rule-based approach, and so DRL is chosen to simplify the approach of forming a policy. Through this, we hope to learn the parameters which allow for the manipulation of the market, such as the size of generator companies and total capacity controlled, and how to reduce this impact of it occurs. The work in this thesis makes a number of key contributions:

1. Development of the open-source, generalised long-term agent-based model for decentralised electricity markets, ElecSim [129]. This can be accessed at: https://github.com/ alexanderkell/elecsim. This model is parametrised to the UK electricity market, and contains the major pertinent features to model this market. This answers research question 1 by modelling an electricity market over the long-term. 2. Validation of the aforementioned model through the use of cross-validation through five years and comparison with the established UK Government model until 2035 [134]. Through this validation we are able to answer research questions 2 and 3 by modelling the inter-year variability and verify the outputs of the model. 3. Forecasting of electricity demand using machine learning models and exploration of the impact of the prediction errors on the long-term electricity market [131]. This contribution answers research question 4 by showing that short-term errors do have a large impact on the final electricity mix. 4. Optimisation of a carbon tax policy to reduce electricity cost and carbon emissions for the UK electricity market using a multi-objective genetic algorithm, from the perspective of a benevolent government [135]. This answers research question 5 by showing that it is possible to come up with an optimal strategy for setting a carbon price to reduce carbon emissions and electricity price. 5. Exploration of the long-term impact of strategic bidding and collusion on decentralised electricity markets [133]. This answers research question 6 by showing that if generator companies control a large part of the market, market power occurs.
However, it is possible to limit these market powers significantly by setting a market cap on electricity price. This work directly addresses the aim of the research. We use AI and ML to answer fundamental questions of the energy transition through the use of an agent-based model of the electricity system. With the development of a novel agent-based model, we are able to answer targeted questions on how the electricity system behaves under different pressures. We do not simply stop at validating the ability of an algorithm to perform a specific task, but rather relate this back to the wider market. Chapter 1 describes the motivations behind this thesis and highlights the main contributions of the research. Finally, the peer-reviewed publications produced during this PhD are presented. Chapter 2 describes the technical background material that relates to the rest of this work and investigates the different types of solutions that have been used in the current literature and differentiate this from this work. Chapter 3 introduces the simulation framework developed within this work. This includes the technical details of the simulation tool, how this simulation is validated and the difficulties of validating such simulation models. Finally, a sensitivity analysis to show the impact of various variables is displayed, and some example future scenarios are produced. This chapter contains contributions 1 and 2. Chapter 4 explores the literature on electricity demand forecasting, how this can be improved with online learning, and what the long-term impact of errors are on decentralised electricity markets. This chapter contains contribution 3. Chapter 5 demonstrates the ability for the model to come up with optimal strategies and scenarios through the use of machine learning techniques. Specifically, a carbon tax strategy between 2018 and 2035 is optimised to reduce both electricity cost and carbon emissions. This chapter contains contribution 4. Chapter 6 demonstrates the ability for large or colluding generator companies to influence the price of electricity in their favour using deep reinforcement learning, as well as an approach to prevent this from occurring through the use of price caps. This chapter contains contribution 5. Chapter 7 summarises the conclusions of the work and motivates future directions for work in this area. During the course of my PhD I have authored the following peer-reviewed publications: This work introduces the agent-based model, ElecSim. The current state-of-the-art of agent-based models is reviewed, and the technical foundations of how ElecSim works is detailed. An initial validation method of comparing the price duration curve of the model to that observed in real life is displayed. Finally, some example scenarios are presented. This work forms the basis for Chapter 3. In this paper, further scenarios are explored by varying the carbon tax level. The effect of carbon tax on investments is demonstrated in the electricity market. This work augments the work done in Chapter 3. [134] Kell, A. J. M., Forshaw, M., & McGough, A. S. (2020). Long-Term Electricity Market Agent Based Model Validation using Genetic Algorithm based Optimisation. The Eleventh ACM International Conference on Future Energy Systems (e-Energy '20). In this paper, further improvements are made to the ElecSim model. Through the addition of representative days, the model is validated between 2013 through 2018 by optimising for long-term predicted electricity price. The results are compared to those of the UK Government, for both a long-term and short-term validation. The results are comparable to those of the UK Government. This work further extends Chapter 3. In this work, various machine learning and deep learning techniques are used to predict electricity demand 30 minutes ahead using smart meter data. Various households are clustered using a k-means clustering technique to further improve the accuracy. This paper forms the basis for Chapter 4. [136] Kell, A. J. M., McGough, A. S., & Forshaw, M. (2020). The impact of online machinelearning methods on long-term investment decisions and generator utilization in electricity markets. 11th International Green and Sustainable Computing Conference, IGSC 2020. This paper expands on the work carried out in [131]. However, instead of predicting 30 minutes ahead, electricity demand over the next day is predicted, over a 24-hour horizon. To improve results, online learning is used, which is able to update the parameters of the models as new data points become available. The results are significantly improved using this method. Finally, the errors of these predictions are taken and the long-term effects of these are shown on the electricity market using the ElecSim models, both in terms of generator utilisation and long-term investment decisions. [135] Kell, A. J. M., McGough, A. S., & Forshaw, M. (2020). Optimising carbon tax for decentralised electricity markets using an agent-based model. The Eleventh ACM International Conference on Future Energy Systems (e-Energy '20), 454-460. In this paper, different carbon tax strategies are trialled using a multi-objective genetic algorithm. With the aim to minimise both the electricity price and carbon emissions. It is found that it is possible to achieve both of these goals through different carbon tax strategies. This work builds on [132] by using a similar multi-objective genetic algorithm optimisation approach. This paper uses reinforcement learning to control the bidding behaviour of a single GenCo, or a group of colluding GenCos in the day-ahead market. It is found that if the other agents bid using their short run marginal costs, the GenCos which use the reinforcement learning algorithm are able to artificially inflate the market price using their market power. The work in this paper forms the work done in Chapter 6. Papers not forming part of this thesis [132] Kell, A. J. M., Forshaw, M., & McGough, A. S., (2019). Optimising energy and overhead for large parameter space simulations. 2019 10th International Green and Sustainable Computing Conference, IGSC 2019. In this work, a multi-objective genetic algorithm is used to reduce both overhead and energy consumption of a cluster of computers at Newcastle University. This is achieved by varying different parameters of a reinforcement learning algorithm.
The methods used in this paper influence much of the work presented in Chapters 3, 4 and 6. [137] Kell, A. J. M., McGough A. S., Forshaw, M., Mercure This paper modifies the FTT:Power model by using reinforcement learning as the electricity generator investment algorithm [169]. FTT:Power is a global power model which uses logistic differential equations to simulate competition between different electricity generating technologies. These logistic differential equations are replaced with the deep deterministic policy gradient reinforcement learning method. It is found find that, if the goal is to reduce both carbon emissions and electricity price, a transition to renewables occurs. Chapter 2

This chapter provides an overview of the relevant material which motivates and underpins the work carried out in this thesis and places the work in context with a literature review. Section 2.1.1, introduces electricity markets and how they are regulated. In Section 2.1.2, an introduction into how electricity markets are modelled is presented. We provide an introduction in Section 2.1.3 to simulation and in Section 2.1.4 to machine learning. Finally, we conclude this chapter in Section 2.1.4. We also review the literature by giving an introduction to the relevant energy models and provide a systematic review on how AI has been applied to agent-based models. Section 2.2.1 gives an introduction to the field of energy modelling. In Section 2.2.2 we present different types of energy models, as well as a detailed view on agent-based models, the focus of this thesis. We present a table in Section 2.2.4, which displays a high-level overview of the major models in the literature. Section 2.2.9 details a systematic review on how AI has been utilised with agent-based models in the electricity sector. We conclude this chapter in Section 2.2.18, where we discuss the limitations and benefits of modelling types and how AI can be integrated into agent-based models in the future. Electricity markets are complex. One of the principal reasons for this is the expense and difficulty of storing electricity. This is because electricity must be consumed the moment it is generated. Additionally, as electricity travels over high voltage transmission lines, electricity doesn't always follow simple or unique paths, especially when the transmission lines become congested. In addition there is power loss during transmission. Finally, electricity markets require technical overseers to ensure that the entire transmission system operates safely and reliably. Another aspect to consider is the fact that electricity is homogeneous. A single unit of electricity produced by a wind turbine is equivalent to a unit of electricity produced by a gas turbine. However, the functioning of different electricity producers, or generators, are not homogenous. Coal, gas and oil power plants can be dispatched at the will of a human operator. Their ramp rates are well understood, as is the amount of fuel that is available. Where the ramp rate is defined as the increase or decrease in output per minute and is usually expressed as megawatts per minute (MW/min). Intermittent Renewable Energy Sources (IRES), however, such as solar, wind and tidal are dependent on the supply of solar irradiance, wind speed and the tide at any moment. Whilst these can be predicted, predictions are often wrong, and perfect knowledge is impossible. Therefore, at times where there is too much supply from Intermittent Renewable Energy Sources (IRES) generators must be curtailed if all regulator generators are already offline or can not be shut down or ramped down. In the opposite case, where there is too little supply from IRES, supply must be made up from other sources, such as coal, gas or hydro. The environmental impact from different electricity generators differs significantly. Whilst gas and coal can be dispatched at a time convenient to the grid operators, they emit CO 2 along with other toxic substances. In this context, dispatched refers to electricity that can be generated on demand at the request of power grid operators, according to market needs. Wind and solar do not dispatch such gases and substances, and therefore can not be controlled as easily. Storage technologies can be used to fill these gaps; however, large-scale storage depends on large pumped reservoirs that can move water to a higher position when demand for electricity is low, and supply is high. Not all geographies have access to such reservoirs, and therefore would rely on battery technology made from chemicals. However, reaching such high storage capabilities are expensive and are yet to have been done in the real world [36]. Another option is converting electricity to hydrogen. However, this technology is also expensive and uncompetitive with traditional fossil fuels such as coal, gas and oil. Currently, peaker plant are used to fill these gaps. Peaker plants are plants which are used in times of high demand and low supply. However, these plants are expensive to operate, highly polluting and use fossil fuels [152]. It is expected that these peaker plants will be used increasingly due to the intermittent nature of renewable technologies [152]. The electricity grid must match supply with demand at all times. Failure to do so results in an imbalance of supply and demand, and affects the frequency of the electricity network. Large differences between supply and demand can lead to blackouts or oversupply and may damage equipment. A number of different markets exist to regulate the supply and demand, running from within seconds, to days-ahead and bilateral contracts which settle electricity for years ahead [40]. There are a number of different market mechanisms that can be used to balance the supply and demand of electricity. Largely these can be divided between ancillary services and wholesale transactions. Wholesale transactions can occur as bilateral trades or on a day-ahead market [40]. Bilateral trades can occur between two electricity suppliers and those that have an electricity demand [40]. In this case, suppliers and customers create contracts for electricity in advance. Typically, these agents must let the market operator know of their trades [40].
In a day-ahead market, the system price is, in principle, determined by matching offers from generators to bids from consumers at each node to develop a supply and demand equilibrium price [40]. Ancillary markets, on the other hand, provide a method to facilitate and support the continuous flow of electricity so that supply continually meets demand [40]. These include markets to regulate power and voltage control as well as frequency control. These markets make use of increasing supply or reducing demand at the times where this is required [40]. There exist many markets to better regulate the supply and demand at different time intervals. For example the intra-day market, capacity market. These markets ensure that electricity supply continues to meet demand as more intermittent renewable energy comes online. The intra-day market is able to dynamically meet fluctuations in supply and demand which were not matched using the day-ahead and bilateral markets. Whereas, the capacity market pays participants a per MW rate for the capacity they offer to the market. Other mechanisms exist, such as the contracts for difference and the carbon price floor to transform the UK's electricity system. Contracts for difference pays a fixed price to generators irrespective of the market price. The carbon price floor places a minimum tax on carbon. An imbalanced system can also be resolved through the means of demand response (the reduction of electricity demand), interconnection with other countries, and through storage for example with vehicle to grid or by other means. In this thesis, however, we focus primarily on the day-ahead market and carbon price floor due to these markets having the largest impact on the macro-scale of the electricity market. Energy models are useful tools for insight into the functioning of electricity markets. For example, modellers and analysts can develop an intuition of how such a system works, and through modelling, they can challenge untested hypothesise. The argument that these models are used for insight and not just informational data is as old as the models themselves [115,188], and is true for models in many different disciplines [84]. We argue that energy models should not be taken as truth, because as previously mentioned, no model can perfectly model the real world. However, the intuition that can be learned can prove to be a valuable resource. Energy modelling and energy policy as a distinct field began after the oil crisis in the 1970s, where long-term planning was deemed as important in the electricity field [44]. Models which make use of optimisation techniques have been used since these times for diverse applications, from the global energy market to small off-grid systems. Optimisation techniques are models in which a central actor invests in the cost-optimal (cheapest) energy system over a multi-year time period. The utilisation of energy models has lately been redirected to ensuring that there exists a security of supply, the resilience of the energy system, affordability and that there is a transition to a low-carbon supply. These models can also be used to investigate the impact that different technologies have on investments made in the future. However, since the traditional models have been established, various changes in the energy industry have occurred. Originally, electricity systems were built upon large-scale centralised electricity production based on fossil fuels [76]. Since then there has been a transition towards decentralised, distributed, intermittent renewable energy sources, such as solar and wind [117]. In addition, there has been an advent in flexible demand driven by new technologies such as smart meters [10]. This paradigm shift requires models which can work with higher temporal and spatial detail to account for fluctuations in demand, supply and distributed electricity generators. Modelling electricity markets is a complex task. There exist many variables, actors, services and behaviours within electricity markets which make it impossible to perfectly model the system. Often simplifications must be made, where models are designed for a specific task [188]. Large established models exist which model every possible detail; however, with the increase in temporal and spatial resolution required, the computational tractability of these models can be negatively impacted. Many of the large models used today have existed for a long time, before the advent of the Internet [188]. Therefore, these models and modellers risk becoming outdated, as their models are not updated. Energy and electricity models generally follow two approaches: bottom-up or a top-down approach [203]. Bottom-up models are often referred to as the engineering approach and are based on detailed technological representations of the energy system. Top-down models, on the other hand, follow an economic approach and consider the long-term changes and macroeconomic relationships [163]. It is possible to combine both the technological properties and long-term changes by creating a hybrid approach [73]. A computer simulation is a virtual model of a real-world system which is programmed into computer software. These models can be used to study how such a system works. One is able to change parameters in the system and make predictions as to how a system might behave. These simulations are particularly beneficial when the system one is analysing is difficult to experiment with [70]. For example, the system exhibits a high financial costs of experimentation, or negative consequences may have large impacts [170]. Additionally, for systems that operate on long timescales, such as energy markets, one may not have the ability to repeat experiments in a controlled environment. Digital twins are a particular instance of a simulation. Digital twins have often been instantiated to a particular system, as opposed to a general system. For example, a digital twin can be made of the UK electricity market, whilst a simulation can be generalised to any decentralised market. By having a digital twin of a particular system, we are able to remove the risks associated with interacting with a system and iterate many experiments within a short time to find an optimal set of parameters. In addition, a digital twin is able to behave more like the system in question.
Whilst simulations must be built with expert knowledge, and through a thorough understanding of the system that one would like to model, machine learning is a data-driven approach. Data-driven approaches need not require an understanding of the system in which they are trying to model. Rather, they infer properties of the system entirely from data. These models are desirable in cases where a system is too complicated to have a full understanding of how the system works. These models have been shown to generate accurate results in many different disciplines [43,241,247]. Simulations can be used to learn how a system may evolve, find optimal input parameters, learn the key features of a system and to extract data from the system. Another key use of simulations is to interact with a physical system. This list is non-exhaustive, as there exist many different uses for simulations. Machine learning (ML) is the study of computer algorithms that improve automatically with the use of data. By using training data, these algorithms are trained to make predictions or decisions without being explicitly programmed to do so. Machine learning methods can be split into three different categories: (1) supervised learning, (2) unsupervised learning and (3) reinforcement learning. Each of these methods can be used in the following cases:

1. Supervised learning is used where the data used has labelled data. Labelled data is where the true value that one is trying to predict is available in the data. 2. Unsupervised learning is where there are no labels associated with the data. The model must, therefore infer from distinct clusters in the data where the divides in the values may lie. 3. Reinforcement learning is concerned with how software agents must take actions in an environment in order to maximise a cumulative reward. In addition to the aforementioned machine learning methods, there exists an additional paradigm: optimisation methods. These methods explore a mathematical or software function to find a minimum or maximum value of an objective. These can be used to minimise an expected error, minimise total cost or maximise total return from a system, for example. In the following sections we explore an overview of the different machine learning techniques that are applicable to the problems addressed in this thesis:

Supervised learning uses training data, which contains both the inputs and desired outputs, to predict the output associated with new inputs. A functioning supervised learning model, will therefore be able to correctly determine the output for inputs that were not part of the original training data. Supervised learning can be used for both regression and classification. Regression is where a continuous output is returned, whereas classification is where a discrete value is returned. The model widely used supervised learning algorithms are:

• Support vector machines

• Linear regression

• Decision trees

• Neural networks Support vector machines, neural networks and decision trees can be used for both regression and classification, whereas linear regression and logistic regression are used for regression and classification respectively. Therefore, decisions must be made when choosing the appropriate algorithm for the respective task. Once a decision has been made with respect to whether classification or regression is required, it is often the case that a number of different algorithms are trialled to see which yield the best results. However, these algorithms have different characteristics which should be taken into account. For instance, neural networks and support vector machines are able to learn complex patterns in data to a higher level than linear regression and so are often chosen for more complex datasets and problems. Secondly decision trees are interpretable, in that, one is able to graphically visualise why a specific output has been returned by the model through a series of decision criteria. This is a feature which is lacking in neural networks, which can often yield good results, but remain a black-box. Linear and logistic regression can be used to understand the most important variables to influence the output variables. So whilst, they are unable to model complex problems as well as neural networks or support vector machines, they can be used for better model interpretation. In this thesis, we use a variety of different supervised learning techniques for specific problems, as previously discussed. As previously mentioned, unsupervised learning learns patterns from unlabelled data. Unsupervised learning can therefore exhibit self-organisation that can capture patterns in data that were previously unknown. Some of the most common algorithms include:

• Hierarchical clustering

• k-means clustering

• Self-organizing maps Hierarchical clustering, similar to decision trees, can be graphically visualised to observe the decisions made by the algorithm. This can be useful, particularly when it is required to see which groups are more similar to one another, and different parent groupings. k-means partitions observations into k clusters, in which each observation belongs to each cluster. k-means provides a cluster centre, which serves as a prototype of the cluster, and provides useful information about the mean of the cluster. A self-organising map produces a low-dimensional representation of higher dimensional data while preserving the topological structure of the data. This can make high dimension data easier to visualise. Self-organising maps are a type of neural network. Reinforcement learning (RL) is an algorithm which allows intelligent agents to take actions in an environment in order to maximise a cumulative reward. Reinforcement learning does not require labelled input/output data. The basis of RL is to find a balance between exploration and exploitation. A large amount of research has been dedicated in recent times to improving the performance of reinforcement learning algorithms [8,150]. Various different methodologies have been tried and tested, from the use of neural networks in deep reinforcement learning to updating a lookup table. Some of the algorithms used in the literature are:

A major difference between the algorithms presented are the action and state spaces. For example, Q-learning requires both a discrete action and state space. Whereas, DDPG and DQN operate with a continuous action and state space.
Q-learning updates a look-up table to map observations to actions, whereas DDPG and DQN use deep neural networks to learn a policy. The type of algorithm chosen can therefore be dependent on the different features required. Due to Q-learning working with a look-up table, they are more interpretable than DDPG and DQNs, as one is able to simply look at the lookup table to see which action is taken with different observations. However, due to the discrete nature of the action and state space, this methodology is less useful if one would like to have more precise actions. It is possible to discretise the action and state space of Q-learning, but the speed and efficiency of the algorithm is greatly reduced with increasing discretised steps. Deep reinforcement learning techniques, similar to neural networks, are able to learn complex patterns within data, which can lead to better results. Care must therefore be taken when making decisions on the type of algorithm used. In this section, we have introduced key concepts that have been used as part of this thesis: electricity markets and their modelling, simulation and digital twins, machine learning methods such as supervised learning and reinforcement learning as well as optimisation techniques. All of the mentioned methods have been used in this thesis for the purpose of increasing our understanding of electricity markets over both a long, and short time periods. We have motivated the need for novel approaches to be used when understanding and modelling energy systems due to the fundamental changes that have occurred since the 1970s. From a system with a centralised actor and power stations which run on fossil fuels to one built on decentralised generation capacity and many heterogeneous actors. The gap between energy policy challenges and model capabilities [209] Assesses the ability of energy systems models to answer major energy policy questions. Agent-based simulation of electricity markets: a literature review [213] An overview of the work applying agent-based models to the analysis of electricity markets. A review of modelling tools for energy and electricity systems with large shares of variable renewables [203] An aid for modellers to choose an appropriate model which can cater for large shares of variable renewables. Energy systems modeling for twenty-first century energy challenges [188] The issues of using existing models for twenty-first century challenges in energy. A review of energy systems models in the UK: Prevalent usage and categorisation [95] Provide a classification schema for energy models. A survey of stochastic modelling approaches for liberalised electricity markets [172] Overview and classification of stochastic models dealing with price risks in electricity markets. Table 2.1 Different reviews of energy system models Additionally, we have discussed the complexity of modelling a system such as electricity markets. However, the insight that can be gained is invaluable and can provide further understanding to those who need to make decisions under large uncertainties. The ramifications of such decisions go far beyond the energy sector, and therefore any help that can be given to decision-makers is of utmost importance. Further details presented in this chapter will be presented in future chapters. Energy modelling is a broad field, so there have been multiple reviews that attempt to separate these models into different classifications [203,209,213]. Examples of the metrics for classification are the mathematical underpinning, the underlying methodology, analytical approach or data requirements. This thesis focuses specifically on agent-based models and AI applied to electricity markets and not the wider literature of energy models. We, therefore, refer the reader to the papers presented in Table 2.1 for a more thorough investigation of this broader field. Large, detailed bottom-up optimisation models have long been used for energy system modelling. These optimisation models are typically based upon a detailed description of the technical components of the energy system. The ultimate goal of optimisation models is to optimise a given quantity, for example, the minimisation of cost or the maximisation of welfare. In this context, welfare can be designed as the material and physical well-being of people [128]. However, these models do not state how likely each of these scenarios is to develop. There are limitations to optimisation based models: traditional centralised optimisation models are not designed to describe a system that is out of equilibrium. Optimisation models assume perfect foresight and risk-neutral investments with no regulatory uncertainty [188]. The core dynamics which emerge from equilibrium remain a black-box. For example, the model assumes a target will be reached and does not provide information when this is not the case. Reasons for this could be investment cycles which move the model away from equilibrium [31]. Equilibrium models take an economic approach. They model the energy sector as a part of the whole economy and study how it relates to the rest of the economy [203]. POLES [220] is a global detailed econometric model developed by the European Commission. E3MG is an econometric simulation developed by Cambridge Econometrics [46]. MARKAL-MACRO is a hybrid model. Where MARKAL is bottom-up, and MACRO is top-down. Simulation models simulate an energy system based upon specified equations, characteristics and rules. These are often bottom-up models, and are designed with a high level of technological description [203]. Agent-based models are a specific case of simulation models, where actors are modelled explicitly as agents with heterogeneous strategies and behaviours. Additionally, there exist a set of power system models which can help with decisions such as investment planning or decisions about generator dispatch. An example of a large power systems model is WASP [125]. In this subsection, we outline current agent-based models available and motivate why the model, ElecSim, is required. Part of the literature review outlined here has been previously published in [129]. Electricity market liberalisation in many western democracies has changed the framework conditions [198]. Centralised, monopolistic, decision making entities have given way to multiple heterogeneous agents acting for their own best interest [172]. Policy options must, therefore, be used to encourage changes to attain a desired outcome.
It has been proposed that these complex agents are modelled using agent-based models (ABMs) due to their non-deterministic nature [129]. A number of ABM tools have emerged over the years to model electricity markets: SEPIA [96], EMCAS [41], NEMSIM [17], AMES [224], GAPEX [35], PowerACE [206], EMLab [31] and MACSEM [198]. Table 2.2 shows, however, that these do not suit the needs of an open source, long-term market model. Table 2.2 contains five columns: tool name, whether the tool is open source or not, whether they model long-term investment in electricity infrastructure and the markets they model and we determine how the stochasticity of real life is modelled. We chose these columns to compare the models due to their importance in the application of understanding long-term electricity market scenarios. Firstly, an open-source tool enables independent users to verify the code developed for these models. It is important that this can be achieved, so that the outputs are not obfuscated by a closed system. Long-term investment allows for the endogenous propagation of an electricity market. We believe that a model which models both day-ahead and futures markets is of importance. This is because these markets are interlinked and correlated. If we did not take into account the day-ahead market, we would not be able to determine the economic success of the GenCos, where profitable GenCos are able to invest further, and those which are less successful invest less. The futures market enables GenCos to invest for the long-term, which determines the trajectory of a scenario. Other aspects, such as intra-day markets are important for the short-term, but less important for the long-term. In addition, the ability to model stochasticity is of importance, due to the non-deterministic nature of the real-world. For instance, the price paid by GenCos is non-deterministic and can vary at any point in time between GenCos. In reality, a large number of features are stochastic, but we discuss those that, we believe, are most pertinent to long-term electricity markets. There have been several recent studies using ABMs which focus on electricity markets. However, they often utilise ad-hoc tools designed for a particular application [93,144,210]. In our work, we develop the model ElecSim, which has been built for re-use and reproducibility. The survey [243] cites that many of these tools do not release source code or parameters, which is a problem that ElecSim seeks to address by being open source and releasing parameters. SEPIA [96] is a discrete event ABM. SEPIA models plants as being always on. SEPIA does not model a spot market, instead focusing on bilateral contracts. As opposed to this, ElecSim has been designed with a merit-order, spot market in mind. As shown in Table 2.2, SEPIA does not include a long-term investment mechanism. EMCAS [41] is a closed source ABM. EMCAS investigates the interactions between physical infrastructures and economic behaviour of agents. However, ElecSim focuses on the dynamics of the market, and provides a simplified, transparent model of market operation, whilst maintaining the robustness of results. NEMSIM [92] is an ABM that represents Australia's National Electricity Market (NEM). Participants are able to grow and change over time using learning algorithms. NEMSIM is non-generalisable to other electricity markets, unlike ElecSim. AMES [224] is an ABM specific to the US Wholesale Power Market Platform. GAPEX [35] is an ABM framework for modelling and simulating power exchanges. However, neither of these model the long-term dynamics for which ElecSim is designed. PowerACE [206] is a closed source ABM of electricity markets that integrates short-term daily electricity trading and long-term investment decisions. PowerACE models the spot market, forward market and a carbon market. Similarly to ElecSim, PowerACE initialises GenCos with each of their power plants. However, as shown in Table 2.2, unlike ElecSim, PowerACE does not consider stochasticity of price risks in electricity markets [172]. EMLab [31] is an open-source ABM toolkit for the electricity market. Like PowerACE, EMLab models an endogenous carbon market; however, they both differ from ElecSim by not taking into account stochasticity in outages and operating costs. MACSEM [198] has been used to probe the effects of market rules and conditions by testing different bidding strategies. MACSEM does not model long term investments or stochastic inputs. As shown in Table 2.2, none of the tools fill each of the characteristics that we require. We therefore propose ElecSim to contribute an open-source, long-term, stochastic investment model. For this work, we decided that a novel agent-based simulation was required to fill all of these categories, as the use-cases developed for our work is specific towards the requirements of our model. Whilst it is true that no other model meets each of our defined characteristics, however, it is not the case that these models are not useful. In fact, they are useful for their own specific purposes. In this section, we present a high-level overview of the various models that are in existence in a table format:  [95] we classify these as either agent-based models (ABM) or non-ABM. The time horizon details what the main purpose of the model is, is it a long-term model. Finally, the time-step column details the time resolution granularity. This subsection covers the difficulties inherent in validating energy models and the approaches taken in the literature to validate these models. Beckman et al. state that questions frequently arise as to how much faith one can put in energy model results. This is due to the fact that the performance of these models as a whole are rarely checked against historical outcomes [18]. Under the definition by Hodges et al. [105] long-range energy forecasts are not validatable [44]. Under this definition, validatable models must be observable, exhibit constancy of structure in time, exhibit constancy across variations in conditions not specified in the model and it must be possible to collect ample data [105]. Whilst it is possible to collect data for energy models, the data covering important characteristics of energy markets are not always measured.
Furthermore, the behaviour of the human population and innovation are neither constant nor entirely predictable. This leads to the fact that static models cannot keep pace with long-term global evolution. Assumptions made by the modeller may be challenged in the form of unpredictable events, such as the oil shock of 1973 [44]. This, however, does not mean that energy-modelling is not useful for providing advice in the present. A model may fail at predicting the long-term future because it has forecast an undesirable event, which led to a pre-emptive change in human behaviour-thus avoiding the original scenario that was predicted. This could, therefore, be viewed as a success of the model. Schurr et al. argued against predicting too far ahead in energy modelling due to the uncertainties involved [233]. However, they specify that long-term energy forecasting is useful to provide basic information on energy consumption and availability, which is helpful in public debate and in guiding policymakers. Ascher concurs with this view and states that the most significant factor in model accuracy is the time horizon of the forecast; the more distant the forecast target, the less accurate the model. This can be due to unforeseen changes in society as a whole [86]. It is for these reasons that we focus on a shorter-term (5-year) horizon window when calibrating our model for validation. This enables us to have increased confidence that the dynamics of the model work without external shocks and can provide descriptive advice to stakeholders. However, it must be noted that the UK electricity market exhibited a fundamental transition from natural gas to coal electricity generation during this period, meaning that a simple data-driven modelling approach would not work. In addition to this short-term cross-validation, we compare our long-term projections to those of BEIS from 2018 to 2035. It is possible that our projections and those of BEIS could be wrong. However, this allows us to thoroughly test a particular scenario with different modelling approaches, and allow for the possibility to identify potential flaws in the models. In this Section, we explore a variety of approaches used in the literature for energy model validation. The model OSeMOSYS [110] is validated against the similar model MARKAL/TIMES through the use of a case study named UTOPIA. UTOPIA is a simple test energy system bundled with ANSWER [114], a graphical user interface packaged with the MARKAL model generator [179]. Hunter et al. use the same case study to validate their model Temoa [114]. In these cases, MARKAL/TIMES is seen as the "gold standard". In this work, however, we argue that the ultimate gold standard should be real-world observations, as opposed to a hypothetical scenario. The model PowerACE demonstrates that realistic prices are achieved by their modelling approach. However, they do not indicate success in modelling GenCo investment over a prolonged time period [204]. Barazza et al. validate their model, BRAIN-Energy, by comparing their results with a few years of historical data; however, they do not compare the simulated and observed electricity mix [16]. This reduces the confidence that one may have in the results of the produced electricity mix. Background and Literature Review Work by Koomey et al. expresses the importance of conducting retrospective studies to help improve models [143]. In this case, a model can be rerun using historical data in order to determine how much of the error in the original forecast resulted from structural problems in the model itself, or how much of the error was due to incorrect specification of the fundamental drivers of the forecast [143]. A retrospective study published in 2002 by Craig et al. focused on the ability of forecasters to accurately predict electricity demand from the 1970s [44]. They found that actual energy usage in 2000 was at the very lowest end of the forecasts, with only a single exception. They found that these forecasts underestimated unmodelled shocks such as the oil crises which led to an increase in energy efficiency. Hoffman et al. also developed a retrospective validation of a predecessor of the current MARKAL/TIMES model, named Reference Energy System [107], and the Brookhaven Energy System Optimization Model [63]. These were studies applied in the 70s and 80s to develop projections to the year 2000. This study found that the models had the ability to be descriptive but were not entirely accurate in terms of predictive ability. They found that emergent behaviours in response to policy had a strong impact on forecasting accuracy. The study concluded that forecasts must be expressed in highly conditioned terms [106]. In this section, we have introduced various electricity market models and the categories that they fall into. However, it can prove to be challenging to place models within a clear boundary, as many models fall within a continuous spectrum. We introduced the concept that traditional models may not have the ability to detail every single component of an electricity market without losing tractability. The need for a new paradigm in which decentralised agents act within an environment was discussed. So was the need for a model with high temporal resolution to more accurately model the intermittency of renewable energy. Traditional optimisation models work in a normative, prescriptive way. However, it is not possible to describe a system which is out of equilibrium. Another limitation of the traditional optimisation models is that they assume perfect foresight, with risk-neutral investments and no regulatory uncertainty. It assumes that certain scenarios are possible, but does not highlight the way a target may not be reached. It is for these reasons that in this thesis, we focus on agent-based models, which move away from the traditional optimisation approach, and allow for a more dynamic solution without rigid mathematical expressions. Additionally, we found that there was a gap in the literature for an open-source agent-based model that could model long-term investments, was generalisable to many countries and modelled stochastic inputs. It is for this reason that we developed the model ElecSim.
In this section, we review the literature that investigates how artificial intelligence and machine learning can be integrated into agent-based models for the electricity sector. To select the related articles to review, we conduct a systematic analysis of relevant research in the field. We limited our search to literature published in the five most recent years (2016-2021). As a result of this, we provide a comprehensive status of the applications of ML and AI in agent-based models for the electricity sector. For this purpose we used the Elsevier Scopus database. To find the articles, we used the following set of search terms to select our articles:

1. Machine Learning, Artificial Intelligence, Deep Learning, Neural Networks, Decision Tree, Support Vector Machine, Clustering, Bayesian Networks, Reinforcement Learning, Genetic Algorithm, Online Learning, Linear regression. 2. Agent-based modelling. We searched using each of the keywords in each of the bullet points. For instance, the first keyword search was: Machine Learning, Agent-Based Modelling and Electricity. The second was: Artificial Intelligence, Agent-based modelling and Electricity. We selected these search terms to focus this review on agent-based models applied to the electricity sector and machine learning, which is the focus of this thesis. These search terms resulted in 149 research articles. However, not all of these were related to our research focus. For instance, a number of electric vehicle, buildings and biological papers were returned. After a further manual review, these 149 papers were reduced to 55 papers which were specifically related to agent-based modelling, electricity, artificial intelligence and machine learning. Figure 2.1 shows the amount of articles published each year between 2016 and the present date. Whilst the number of articles published in this field has increased per year since 2018 to Background and Literature Review 2020, the number of papers published in 2017 was lower when compared to the other years, with a large number published in 2016. We reviewed these 55 papers systematically in the following sections. In this literature review, we make three different market type distinctions: international/national energy market, local energy market and a microgrid. The international/national energy market typically considers a country, multiple countries or the world. A local energy market is a smaller region than the international/national energy market, for instance, a city or region. Whereas a microgrid serves a discrete geographic footprint, such as a university campus, business centre or neighbourhood. Whilst there is some cross-over between a local energy market and microgrid, a microgrid can be disconnected from the traditional grid and operate autonomously. Tables 2.4, 2.5 and 2.6, 2.7 and 2.8 categorise each of the market types respectively. The papers have been displayed in chronological order and categorise the market type, machine learning (ML) type used, the application in which it was used and the algorithm used. These different criteria are explored in the following subsections. Within this work, we have covered five different type of artificial intelligence paradigms. These are: supervised learning, unsupervised learning, reinforcement learning, optimisation and game theory Each of these techniques have been utilised in the papers surveyed. However, a particular focus has been placed on reinforcement learning within the research community. As shown by Figure 2.2, 37 out of the 55 papers used a reinforcement learning algorithm. This greatly outweighs the other machine learning types. The second most used machine learning type was supervised learning, used by eight papers. The fact that reinforcement learning has been used so extensively within the agent-based modelling community for electricity highlights the usefulness of this technique within this field. Within each of the different machine learning types there exist many algorithms. The algorithms used in the papers surveyed are now presented. Within reinforcement learning the deep deterministic policy gradient (DDPG), Deterministic Policy Gradient (DPG) Deep Q-Network (DQN), Deep Q-Learning, Fitted Q-iteration (FQI), long short-term memory neural network (LSTM), Multi-Agent Deep Deterministic Policy Gradient (MADDPG), Markov Decision Process (MDP), Novel WoLF-PHC, Policy Iteration (PI), Probe and Adjust, Q-Learning, Roth-Erev, SARIMAX and Variant Roth-Erev are used. Within supervised learning, the following algorithms were used: Artificial Neural Network (ANN), Bayesian networks, Classification trees, Extreme Machine Learning, Lasso regression, Linear regression and Support Vector Machine (SVM) were used. Fewer algorithms were used for both unsupervised learning and optimisation. For supervised learning, the following algorithms were used: Bayesian classifier, K-Means Clustering, Naive Bayes classifier. For optimisation the following algorithms were trialled: Bi-level coordination optimisation, Genetic Algorithm, Iterative algorithm and Particle Swarm Optimisation. For the game theory method, a game theoretic algorithm was used. Within this work, we classified each paper by the problem domain which they are trying to solve, or the application. The applications are: agent behaviour, bidding strategies, bilateral trading, demand forecasting, demand response, electricity grid control, expansion planning, forecasting carbon emissions, load scheduling, market investigation, microgrid management, peer to peer trading, price forecasting, risk management, scheduling of flexibility, secure demand side management and tariff design. Figure 2.3 displays the number of applications used by each machine learning type. The most utilised application was bidding strategies, with price forecasting and tariff design following behind. However, the bidding strategies application was investigated 27 times, with price forecasting investigated only 8 times. This demonstrates a considerable research effort in this area. Figure 2.4 displays the number of applications per machine learning type area. We can see that bidding strategies is highly used within the reinforcement learning machine learning type. The reinforcement learning algorithm, however, is shown to be highly versatile, with different applications investigated, from demand response, flexibility scheduling to expansion planning. This is due to the ability for reinforcement learning to learn different policies based upon solely the reward and observations within an environment. Within supervised learning, a large amount of research effort has been put into price forecasting. This is likely due to the strong ability of supervised learning techniques at making predictions. However, outside of classification and making predictions, supervised learning is not so versatile, when compared to reinforcement learning. Optimisation is used for five different applications.
This is because optimisation requires a problem domain to be maximised or minimised. This may not be the case for all applications. Unsupervised learning and the game theoretic machine learning types are shown to be used less than the other machine learning types. This is because unsupervised learning is preferential when there is no labelled data. However, with labelled data, supervised learning can yield more accurate results. Within simulations it is often the case that data is available, and so supervised learning is used in preference to unsupervised learning. The game theoretic approach has been used in a single application in the papers surveyed: bidding strategies. The application of game theory is possible for the problem of bidding strategies, however, the assumptions of a Nash equilibrium and perfect information may not always exist in an electricity market. In this section we review the papers that utilised reinforcement learning for the applications shown in Figure 2.4. Firstly, we cover the papers which consider the bidding strategies problem. Liu et al. [155] establish non-cooperative and cooperative game models between thermal power companies. They show that, compared with other RL algorithms, the MADDPG algorithm is more accurate, with an increase in revenue of 5.2%. They show that thermal companies are more inclined to use physical retention methods to make profits in the medium and long-term power market. Liang et al. [149] use the DDPG algorithm to model the bidding strategies of GenCos. They show that the DDPG algorithm is able to converge to a Nash equilibrium even with imperfect information. They find that they are able to reflect collusion through adjusting the GenCos' patience parameter. Purushothaman et al. [199] model the learning capabilities of power generators through the use of the Roth-Erev RL algorithm. They find that the agents are able to exhibit market power through this approach. Kiran et al. [141] use a variant of the Roth-Erev algorithm to investigate the ability for a generator to bid strategically within a market. They find that agents have the ability to bid strategically, and increase their net earnings. However, the impact on the wider market and how to limit this ability was not explored in these papers. Sousa et al. [221] use an ABM to model the Iberian electricity market, with a focus on hydropower plants. They use Q-Learning to bid in the day-ahead market. Poplavskaya et al. [197] model the balancing services market, and investigate the effect of different market  structures on price. They find that in an oligopoly, prices can deviate from the competitive benchmark by a factor of 4-5. They conclude that changing market type would not solve this issue. Nunna et al. [181] use a simulated-annealing-based Q-learning algorithm to develop bidding strategies for energy storage systems. They observe that they can effectively reinforce the balance between supply and demand in the microgrids using a mixture of local and global energy storage systems. Lin et al. [151] investigate herding behaviours of electricity retailers on the monthly electricity market. Herding behaviours are where individuals act collectively as part of a group. They model herding behaviours mathematically based on the relationship network of electricity retailers which are imbedded in an ABM. They use the Roth-Erev RL to model these behaviours. They find that the herding behaviours might bring positive or negative effects to electricity retailers depending on the differing bidding strategies. Wang et al. [238] investigate the bidding behaviour of all players in the electricity market. They propose a hybrid simulation model and integrate RL to bid. They find that with the hybrid simulation model, the dynamics of the entire market remain stable, the market clearing prices converge, and the market share is relatively uniform. Machado et al. [160] investigate how the energy price is affected when a government intervention is observed through the increase in number of public companies participating in auctions. They find that they are able to model the impact of public companies on the overall electricity market. Ye et al. [253] propose a novel multi-agent deep RL algorithm, where they combine the DPG algorithm with LSTM for multi-agent intelligence. Their algorithm achieves a significantly higher profit than other RL methods for a GenCo. Pinto et al. [192] investigate the ability for collaborative RL models to optimise energy contract negotiation strategies. The results show that the collaborative learning process enables players' to correctly identify which negotiation strategy to apply in each moment, context and opponent. Feng et al. [68] explore the effect of a transition from a monopoly to a competitive electricity market. They simulate a day-ahead market and use RL to bid strategically. They find that they can characterise the risk characteristics and decision-making objectives of market participants. Calabria et al. [29] simulate the Brazilian power system, using 3 years worth of data which covers 98% of the total hydro capacity of Brazil. They use RL to simulate the behaviour of virtual reservoirs in this market. They find that through the management of these virtual reservoirs, they can save water, while maintaining current efficiency and security levels. Gaivoronskaia et al. [81] present a modification of the classical Roth-Erev algorithm to represent agents' learning in an ABM of the Russian wholesale electricity market, in a day-ahead market. Staudt et al. [223] investigate the effect of the number of energy suppliers needed for a competitive market. They find that several suppliers are required to ensure a welfare optimal pricing. Where welfare is defined as a benefit to the community. Esmaeili Aliabadi et al. [64] study the effect of risk aversion on GenCos' bidding behaviour in an oligopolistic electricity market. They find that the change in the risk aversion level of even one GenCo can significantly impact on all GenCo bids and profits. Rashedi et al. [202] use Q-Learning to learn optimal bidding strategies of suppliers in a day-ahead market. They compare the competitive behaviour of players in both the multi-agent and single-agent case. Chrysanthopoulos et al. [34] take a similar approach, by modelling the day-ahead market as a stochastic game.
They use an ABM to model GenCos to maximise their profit using RL. Skiba et al. [218] model the bidding behaviour on the day-ahead and control reserve power markets using an RL algorithm in an ABM. They investigate the resulting market prices. Tang et al. [227] investigate the bidding strategies of generators under three pricing mechanisms. They find they are able to achieve market equilibrium results through their novel RL approach. Bakhshandeh et al. [13] assess the ability for GenCos to withhold capacity to increase the market price using RL. The results demonstrate the emergence of capacity withholding by GenCos, which have an effect on the market price. Xu et al. [251] simulate a proactive residential demand response in a day-ahead market using RL. They use residential data in China, and test a case with 30,000 households. They find that a proactive residential demand response may yield significant benefits for both the supply and demand side. Deng et al. [48] use a DDPG algorithm to also model residential demand response. They find that the goal of peak load-cutting and valley filling can be achieved through this method. Shafie-Khah et al. [214] develop a novel decentralised demand response model. In their model, each agent submits bids according to the consumption urgency and a set of parameters by the RL algorithm, Q-Learning. Their studies show that their decentralised model drops the electricity cost dramatically, which was nearly as optimal as a centralised approach. Najafi et al. [175] also propose a decentralised demand response model. They use Q-Learning and consider small scale GenCos. Similarly to Shafie-Khah et al., they show the effectiveness of this technique. Tomin et al. [232] propose an RL method to interact with the electricity grid for active grid management. They demonstrate the effectiveness of this approach on a test 77-node scheme and a real 17-node network diagram of the Akademgorodok microdistrict (Irkutsk). Huang et al. [112] propose a generation investment planning model for GenCos. They use a genetic algorithm and Q-learning to improve their optimisation ability, and show that the model is effective and could provide support for plant expansion planning. Manabe et al. [164] also consider the generation expansion planning problem. They find through their simulation that their agents can generate higher profit using RL. Foruzan et al. [75] use an ABM to study distributed energy management in a microgrid. They use an RL algorithm to allow generation resources, distributed storages and customers to develop optimal strategies for energy management and load scheduling. They are able to reach a Nash equilibrium, where all agents benefit through this approach. Viehmann et al. [235] analyse the different markets: Uniform Pricing (UP) and Discriminatory Pricing. UP is where all agents pay the maximum accepted price, and DP pay their accepted bid. Through the use of RL, they find that UPs lead to higher prices in all analysed market settings. Pinto et al. [194] introduce a learning model to enable players to identify the expected prices of bilateral agreements. For this they use a Q-Learning algorithm, and they use real data from the Iberian electricity market. Mbuwir et al. [167] explore two model-free RL techniques: policy iteration (PI) and fitted Q-iteration (FQI) for scheduling the operation of flexibility providers -battery and heat pump in a residential microgrid. Their simulation results show that PI outperforms FQI with a 7.2% increase in photovoltaic self-consumption in the multi-agent setting and a 3.7% increase in the single-agent setting. Naseri et al. [176] propose an autonomous trading agent which aims to maximise profit by developing tariffs. They find that designing tariffs with usage-based charges and fixed periodic charges can help the agent segment the retail market resulting in lower peak demand and capacity charges. Bose et al. [23] simulate a local energy market as a multi-agent simulation of 100 households. Through the use of the Roth-Erev reinforcement learning algorithm to control trading, and demand response of electricity. They are able to achieve self-sufficiency of up to 30% with trading, and 41.4% with trading and demand response. Kim et al. [138] consider a prosumer that consumes and produces electric energy with an energy storage system. They use the DQN reinforcement learning algorithm to maximise profit in peer-to-peer energy trading. Liu et al. [153] also investigate the peer-to-peer trading problem in a local energy market. They find that the community modelled are able to increase profitability through the use of DQN RL. In this section we review the papers that used a supervised learning approach with their agentbased models, which focus on electricity. First we consider papers which focus on price forecasting. Fraunholz et al. [77] use ANNs to forecast electricity price endogenously within the long-term energy model, PowerACE. They find that the ANN method outperforms the linear regression method and that this endogenous method has a significant impact on simulated electricity prices. This is of importance since these are major results for electricity market models. Pinto et al. [193] uses SVMs and ANNs for price forecasting using real data from MIBEL, the Iberian market operator. They show an ability to return promising results in a fast execution time. Goncalves et al. [88] use multiple different methods to understand the main drivers of electricity prices. These include lasso and standard regression, and causal analysis such as Bayesian networks and classification trees. Their results are coherent and show the impact of different generators on final electricity price. Opalinski et al. [183] propose a hybrid prediction model, where the best results from a possible large set of different short-term load forecasting models are automatically selected Background and Literature Review Year First Author Market Type Application Algorithm Used Bose S. [23] Local energy market Peer to peer trading Roth-Erev Naseri N. [176] Local energy market Tariff design SARIMAX, MDP Tang C. [227] International/National Tariff design Novel WoLF-PHC Liu D. [153] International/National Bidding strategies, Peer-to-Peer MADDPG Deng C. [48] International/National Demand response DDPG Viehmann J. [235] International/National Market investigation Q-Learning Tomin N. [232] Microgrid Electricity grid control Q-Learning Liang Y.
[149] International/National Bidding strategies DDPG Kim J.-G. [138] International/National Peer to peer trading Deep Q-Learning Shafie-Khah M. [214] Local energy market Demand response Q-Learning Sousa J.C. [221] International/National Bidding strategies Q-Learning Poplavskaya K. [197] International/National Bidding strategies Q-Learning Liu Y. [155] Local energy market Bidding strategies DQN Nunna H.S.V.S.K. [181] Microgrid Bidding strategies Q-Learning Purushothaman K. [199] International/National Bidding strategies Roth-Erev Mbuwir B.V. [167] Microgrid Scheduling of flexibility Policy Iteration (PI), Fitted Q-iteration (FQI) Kiran P. [141] International/National Bidding strategies Variant Roth-Erev Lin F. [151] International/National Bidding strategies Roth-Erev Wang J. [238] International/National Bidding strategies Roth-Erev Machado M.R. [160] International/National Bidding strategies Q-Learning Ye Y. [253] International/National Bidding strategies DPG,LSTM Pinto T. [194] International/National Price forecasting Q-Learning Pinto T. [192] International/National Bidding strategies Q-Learning Feng H. [68] Local energy market Bidding strategies Variant Roth-Erev Foruzan E. [75] Microgrid Load scheduling Q-Learning Calabria F.A. [29] International/National Bidding strategies Q-Learning Gaivoronskaia E.A. [81] International/National Bidding strategies Roth-Erev Staudt P. [223] Microgrid Bidding strategies Probe and adjust Najafi S. [175] Local energy market Demand response, Bidding strategies

Esmaeili Aliabadi D. [64] International/National Bidding strategies Q-Learning Huang X. [112] International/National Expansion planning Q-Learning Rashedi N. [202] International/National Bidding strategies Q-Learning Skiba L. [218] International/National Bidding strategies Roth-Erev Bakhshandeh H. [13] International/National Bidding strategies Q-Learning Chrysanthopoulos N. [34] Local energy market Bidding strategies Q-Learning  [24] Local energy market Forecasting carbon emissions ANN 2020 Babar M. [11] International/National Secure demand side management Naive Bayes classifier 2019 Maqbool A.S. [165] International/National Tariff design Linear regression 2019 Goncalves C. [88] International/National Price forecasting Linear regression, Lasso regression, Bayesian networks, Classification trees 2019 Pinto T. [191] International/National Risk management ANN 2019 El Bourakadi D. [61] Microgrid Bidding strategies Extreme Machine Learning 2016 Opalinski A. [183] International/National Demand forecasting Linear regression 2016 Pinto T. [193] International/National Price forecasting ANN, SVM Table 2.5 Articles relating to supervised learning algorithm type. based on their past performance by a multi-agent system. They show an increase in prediction accuracy with their approach. Bouziane et al. [24] forecast carbon emissions using a hybrid ANN and ABM approach from different energy sources from a city. They forecast energy production using agents and calculate the benefits of using renewable energy as an alternative way of meeting electricity demand. They find they are able to reduce emissions by 3% per day using this approach. Pinto et al. [191] propose an approach to addressing the adaptation of players' behaviour according to participation risk. To do this, they combine the two most commonly used approaches of forecasting: internal data analysis and sectorial data analysis. They show that their proposed approach is able to outperform most market participation strategies and reach a higher accumulated profit by adapting players' actions according to the participation risk. Maqbool et al. [165] investigate the impact of feed-in tariffs and the installed capacity of wind power on electricity consumption from renewables. They use linear regression to understand the outputs from an agent-based model to achieve this. They find that the effect of increasing installed capacity of wind power is more significant than the effect of feed-in tariffs. El Bourakadi et al. [61] propose the use of an Extreme Machine Learning (EML) algorithm to make decisions about selling/purchasing electricity from the main grid and charging and discharging batteries from an ABM. The EML algorithm predicts wind and photovoltaic power output from weather data and then makes a classification decision on whether to buy or sell power. Babar et al. [11] propose a secure demand-side management engine to preserve the efficient utilisation of energy based on a set of priorities. A model is developed to control intrusions into the smart grid using a naive Bayes classifier. Simulation is used to test the efficiency of the system, and the result reveal that the engine is less vulnerable to intrusion. Market Type Application Algorithm Used 2021 Gomes I.L.R. [87] Microgrid Microgrid management K-Means Clustering 2020 Imran K. [119] International/National Bilateral trading Bayesian classifier 2017 Čaušević S. [1] International/National Load scheduling Novel clustering algorithm Table 2.6 Articles relating to unsupervised learning algorithm type. In this section we discuss the papers which use an unsupervised learning approach with agentbased models. Imran et al. [119] develop a novel strategy for bilateral negotiations. They enable each GenCo to estimate the reservation price of its opponent using Bayesian learning. They show that the agents which use Bayesian learning gain an advantage over non-learning agents. Čaušević et al. [1] propose a novel clustering algorithm to cluster agents into virtual cluster members for the application of load scheduling. They show that large-scale centralised energy systems can operate in a decentralised fashion when only local information is available. Gomes et al. [87] propose a management system for the operation of a microgrid by an electricity market agent. They use K-means clustering for scenario reduction and a stochastic mixed-integer linear programming problem to manage the system. In this subsection we review the papers which use optimisation as a basis for investigation. Bevilacqua et al. [21] compare three optimisation methods to implement agent rationality in the Italian electricity market with an ABM. Through this, they observe that the moddel exhibits a very good fit to real data. Duan et al. [59] propose a bi-level coordination optimisation integrated resource strategy to unify supply-side and demand side resources across China. They do this for mid-long term planning. Gao et al. [82] use a genetic algorithm to determine an optimal bidding strategy. They verify their approach by modelling a 30-bus system as an example. Meng et al. [168] also use a genetic algorithm optimisation approach to make dynamic pricing decisions. They model the day-ahead market and propose a two-level optimisation model. They model the price responsiveness of different customers using the optimisation algorithm, and then optimise the dynamic prices that the retailer sets to maximise its profit. They confirm the feasibility and effectiveness of their technique through simulation. Filho et al. [69] is the only paper in this review which takes a game theoretic approach.
They deal with a comparative analysis of individual strategies of generating units in auctions, using. noncooperative game theory approach. They find that their method is best suited for second-price  [59] International/National Expansion planning Bi-level coordination optimization 2018 Gao Y. [82] International/National Bidding strategies Genetic Algorithm 2018 Meng F. [168] Local energy market Tariff design Genetic Algorithm auctions and can be extended to more complicated networks with high precision. However, they find that it is not possible to use this methodology in some, more complex, systems. For example, bidding within a stochastic and uncertain environment, where a predictions are required for the behaviour of other actors. Artificial intelligence (AI) and machine learning (ML) has been integrated with agent-based models to model the electricity sector with increasing frequency over the last years. This is due to the ability of AI to optimise agent behaviour, system parameters and add functionality to agentbased models (ABMs). This study, therefore, reviewed recent papers regarding applications of AI and ML in this space. We categorised these papers into four different criteria: market type, application, algorithm type and algorithm used. To do this, we used different search terms on Scopus and reviewed all 55 articles in the field over the past five years. It was found that the majority of papers used reinforcement learning applied to bidding strategies. However, a range of applications were investigated through a wide variety of means. This included price forecasting, demand forecasting, microgrid management and risk management. We highlight the major findings from this study in the following sections. Table 2.9 displays the frequency of market types for the papers reviewed. Whilst Table 2.10 shows the frequency of different applications per market type. 1. 68.9% of the papers reviewed relate to an international/national electricity market. The availability of data and the relative importance of the subject of whole system transitions in current affairs may explain why such research effort has been dedicated to this. 2. 32.8% of the papers for the international/national focus on bidding strategies, as is shown by Table 2.10. This is due to the ability of reinforcement learning to make strategic Market Type Percentage (%)

International/National 68.9

Local energy market 18.0 Microgrid 13.1

Table 2.9 Frequency of market type for papers reviewed. decisions under uncertainty. In addition, the ability to model strategic bidding is of significance importance for international/national energy models due to the appearance of oligopolies in national energy markets. However, whilst many studies have explored on strategic bidding and a few have focused on oligopolistic bidding strategies, a study on the impact on the wider market does not exist. 3. Price forecasting is the second largest category which is explored, and is investigated in 13.1% of papers. This is due to the ability for supervised learning to make predictions in time-series data. It is also of importance for agents to make realistic predictions of future prices. However, there exists a gap in the literature on the long-term effects of the different accuracies of these forecasts. 4. 18% of papers focus on the local energy market. A significant reduction when compared to the international/national electricity market. This is because of the limited availability of publicly available data for these local energy markets. ABMs require a high amount of data to inform the behaviour of the agents and environment, and so data collection for local energy markets can be expensive and difficult to obtain. Microgrids are explored in 13.1% of papers. Similarly to local energy markets this is because of a smaller amount of publicly available data. It is also possible that researchers place an increasing focus on international/national electricity markets due to the availability of these models and perceived impact from a large system. 1. A significant proportion of papers have focused on bidding strategies, with 44.3% of papers investigating this. Bidding strategies is top for each of the market types, indicating the versatility and applicability of this technique in electricity markets. 2. The application of control, for instance grid control, load scheduling and demand response has seen a significant amount of promising research. Particular in the application of decentralised control [175,214,1]. With the advent of distributed data collection and internet of things, it has become possible to achieve various welfare objectives through a distributed control algorithm. That is, to optimise for individuals or distributed generators from the perspective of those respective individuals and generators. Studies have shown efficiencies close to centralised algorithms. However, centralised algorithms may reduce the agency of individuals and therefore must overcome an additional barrier in uptake if the individuals do not perceive that the algorithm is working in their own best interest. Table 2.10 Frequency of market type and application for papers reviewed. 3. Various applications have been explored within the research community. In total, 17 different applications were explored. This demonstrates the versatility of ML applied to ABMs. This, however, highlights a significant gap in the literature as the majority of applications have only been explored by one or two papers. For instance, the ability to optimise the electricity system parameters in question has not been explored to the same level of detail as forecasting or trading behaviour. 4. Whilst there are many studies which look at optimising certain aspects of an electricity market, there is no study which looks at optimising the whole electricity system as a whole. Table 2.11 displays the frequency of each of the algorithms used. Q-Learning is the most used algorithm, with 29% of papers using this algorithm. Second is the Roth-Erev algorithm, which is used by 9.7% of papers. Whilst this shows the versatility of these algorithms, further research could be placed into the use of deep reinforcement learning (DRL) to improve results. DRL uses deep neural networks to act as function approximators. DDPG, DQN and DPG are examples of DRL algorithms.
The majority of the algorithms have only been used in a single paper, and so, there remains a significant gap in the literature to apply these algorithms to the different applications to investigate with results can be improved through other algorithms. Significant and increasing research interest has been placed into agent-based models applied to the electricity sector. However, this work has shown that a lot of research has clustered around similar subjects. For instance, the top application investigated for each market type is bidding strategies, largely exploring whether it is possible to use reinforcement learning to bid into an electricity market. However, the impact of these bidding strategies on the wider market has been investigated to a lesser extent. This is a gap in the literature and something we attempt to address in Chapter 6. In this chapter, we look at the effect of strategic bidding on the wider electricity market, and how to reduce the effect of monopolies. Another aspect that has been explored to a lesser extent is the optimisation of the electricity system parameters as a whole as applied to policy. We attempt to fill this gap in Chapter and 3 and 5. In these chapter, we minimise electricity price and carbon emissions using a genetic algorithm to optimise a carbon tax policy and also to calibrate our model, ElecSim. Finally, a large amount of research has been placed into forecasting time series within agentbased models. However, the research typically stops here and does not consider the impact of these forecasts on the wider market. Therefore, we attempt to fill this gap in Chapter 4. This chapter provides a systematic review of AI applied to agent-based models in the electricity sector. Through this analysis, various research gaps have been found, as the majority of papers focus on similar themes and algorithms. This, therefore, highlights a wide range of areas in which the research community can focus on in future work. In this chapter, we motivate and introduce the agent-based model, ElecSim. The majority of the work presented here was published in [129] and [134]. The contribution of this chapter is a new open-source framework for the long-term modelling of electricity markets. We provide curated data for the simulation, to improve realism with Monte Carlo sampling, and calibrate our model using genetic algorithm-based optimisation. We validate our model after this calibration, using observed data between 2013 and 2018, as well as compare our model to the UK Government's baseline scenario up until 2035. We also provide a sensitivity analysis of important variables. For example, these variables include the Weighted Average Cost of Capital (WACC) as well as the percentage of down payment required for a particular investment. The sensitivity analysis, as well as additional scenarios shown in Section 3.5 and Subsection ? ? were previously unpublished in [129] and [134]. We show, in this work, that it is possible to validate an agent-based model which focuses on the long-term electricity market by extracting the pertinent features of the market in question. We introduce our work in Section 3.1, including why a simulation model is required to aid in a low carbon transition. In addition to the literature review on energy-based simulations presented in Chapter 2, we provide a literature review of work done to validate energy models in Section 2.2.5. The architecture of the model is presented in Section 3.2. The model is validated in Section 3.3. Various scenarios are presented in Section 3.4, where we vary demand until 2035. A sensitivity analysis is provided in Section 3.5, where we vary the weighted average cost of capital, as well as the down payment required for investments. Finally, we present the limitations of our model in Section 3.6 and conclude our work in Section 3.7. The ElecSim model, and all related code can be accessed at: https://github.com/alexanderkell/ elecsim. Global carbon emissions from fossil fuels have significantly increased since 1900 [22]. Fossilfuel based electricity generation sources such as coal and natural gas currently provide 65% of global electricity [26]. Low-carbon sources such as solar, wind, hydro and nuclear provide 35% . To halt this increase in CO 2 emissions, a transition of the energy system towards a renewable energy system is required. Due to the long construction times, operating periods and high costs of power plants, investment decisions can have long term impacts on future electricity supply [31]. Governments, therefore, have a role in ensuring that the negative externalities of emissions are priced into electricity generation. This is most likely to be achieved via a carbon tax and regulation to influence electricity market players such as Generation Companies (GenCos). Decisions made in electricity markets may have unintended consequences due to their complexity. A method to test hypothesise before they are implemented would, therefore, be useful. To aid in such a transition, energy modelling can be used by governments, industry and agencies to explore possible scenarios under different variants of government policy, future electricity generation costs and energy demand. These energy modelling tools aim to mimic the behaviour of energy systems through different sets of equations and data sets to determine the energy interactions between different actors and the economy [161]. Live experimentation of physical processes is not often practical. The costs of real-life experimentation can be prohibitively high, and can require a significant amount of time in order to fully ascertain the long-term trends. There is also a risk that changes can have detrimental impacts and lead to risk-averse behaviour. These factors are true for electricity markets, where decisions can have long term impacts. For instance, by investing in a coal power plant today, this will lock in the carbon emissions emitted by the coal power plant for the next 25+ years. Simulation, however, can be used for rapidly prototyping ideas. The simulation is parametrised by real-world data and phenomena. Through simulation, the user is able to assess the likelihoods of outcomes under certain scenarios and parameters [145].
Simulation is often used to increase understanding as well as to reduce risk and reduce uncertainty. Simulation allows practitioners to realise a physical system in a virtual model. In this context, a model is defined as an approximation of a system through the use of mathematical formulas and algorithms. Through simulation, it is possible to test a system where real-life experimentation would not be practical due to reasons such as prohibitively high costs, time constraints or risk of detrimental impacts. This has the dual benefit of minimising the risk of real decisions in the physical system, as well as allowing practitioners to test less risk-averse strategies. Agent Based Models (ABMs) are a class of computational simulation models composed of autonomous, interacting agents. These can be built to model systems with many heterogeneous agents. Due to the numerous and diverse actors involved in electricity markets, ABMs have been utilised in this field to address phenomena such as market power [205]. The work presented in this chapter develops ElecSim, an open-source ABM that simulates GenCos in a wholesale electricity market. ElecSim models each GenCo as an independent agent. An electricity market facilitates trades between the two. Figure 3.1 displays an overview of a wholesale electricity market. The GenCos invest in power generation facilities, such as coal power plants, renewables or gas turbines which produce electricity. Typically, a reseller will purchase the produced electricity through bilateral contracts, or on the day-ahead market who then sell the electricity to end-users. Where end-users could be residential users, industry or the automotive sector. GenCos make bids for each of their power plants. Their bids are based on the generators short run marginal cost [187], which excludes capital and fixed costs. The electricity market accepts bids in cost order, also known as merit-order dispatch. GenCos invest in power plants based on expected profitability. ElecSim is designed to provide quantitative advice to policymakers, allowing them to test policy outcomes under different scenarios. A scenario is a single future which could occur under different circumstances. To achieve this, policymakers are able to modify a file to realise a scenario of their choice. For example, by changing anticipated future energy costs, carbon taxes or electricity demand. It can also be used by energy market developers who can test new electricity sources or policies, enabling the modelling of changing market conditions. This model can be used by the following players:

• Policy experts to test policy outcomes under different scenarios and provide quantitative advice to policymakers. They can provide a simple script defining the policies they wish to use along with the parameters for these policies. • Energy market developers who can use the extensible framework to add elements such as new energy sources, policies, consumer profiles and storage types. Thus allowing ElecSim to adapt to a changing ecosystem. Optimisation based solutions are the dominant approach for analysing energy policy [31]. However, the results of these models should be interpreted in a normative manner. For example, how investment and policy choices should be carried out, under certain assumptions and scenarios. The processes which emerge from an equilibrium model remain a black-box, making it difficult to fully understand the underlying dynamics of the model [31]. In addition to this, optimisation models do not allow for endogenous behaviour to emerge from typical market movements, such as investment cycles [31,91]. By modelling these naturally occurring behaviours, policy can be designed that is robust against movements away from the optimum/equilibrium. Thus, helping policy to become more effective in the real world. Agent-based models differ from optimisation-based models by the fact that they are able to explore 'what-if ' questions regarding how a sector could develop under different prospective policies, as opposed to determining optimal trajectories. ABMs are particularly pertinent in decentralised electricity markets, where a decentralised actors dictate investments made within the electricity sector. ABMs have the ability to closely mimic the real world by, for example, modelling irrational agents, in this case, Generation Companies (GenCos) with incomplete information in an uncertain future [85]. By undertaking calculations for each day of the simulation in ElecSim, we would potentially gain accuracy. For instance, by modelling the variations within days of solar irradiance and wind speed. However, this would have the effect of slowing the simulation down significantly. Therefore, as a compromise between speed and accuracy, we use a set of representative days to model an annual time period. Similar to Nahmmacher et al. we demonstrate how clustering of multiple relevant time series such as electricity demand, solar irradiance and wind speed can reduce computational time by selecting representative days [174]. In this context, representative days are a subset of days that have been chosen due to their ability to approximate the weather and electricity demand in an entire year. Similar to Nahmacher et al. we use a Ward hierarchical clustering algorithm [127], However, we also try a k-means clustering approach [72]. We chose the k-means clustering approach due to the previous success of this technique in clustering time series [131]. There is a desire to validate the ability of energy-models to make long-term predictions. Validation increases confidence in the outputs of a model and leads to an increase in trust amongst the public and policymakers. Energy models, however, are frequently criticised for being insufficiently validated, with the performance of models rarely checked against historical outcomes [18]. In answer to this, we postulate that ABMs can provide accurate information to decisionmakers in the context of electricity markets by using cross-validation of observed electricity mix compared to simulated electricity mix. We increase the temporal granularity of the work in our work in this chapter and use genetic algorithms to tune the model to observed data enabling us to perform calibration of the model. This enables us to understand the parameters required to observe certain phenomena, as well as use these fitted parameters to make inferences about the future.
We use a genetic algorithm approach to find an optimal set of price curves predicted by generation companies (GenCos) that adequately model observed investment behaviour in the real-life electricity market in the United Kingdom. Similar techniques can be employed for other countries of various sizes [129]. We measure the accuracy of projections for our improved ABM with those of the UK Government's Department for Business, Energy and Industrial Strategy (BEIS) for the UK electricity market between 2013 and 2018. In addition to this, we compare our projections from 2018 to 2035 to those made by BEIS in 2018 [53]. We select five years for the calibration and validation period as we believe that the transition from coal to gas in this period reflects the kind of dynamics that ElecSim must capture over the long-term. Whilst it can be argued that a five year period is too short a time to adequately model a long-term energy model, it is argued here that long-term validation is unfeasible due to the large amount of difficult to capture features over the long-term. In other words, a long-term model provides scenarios and not forecasts, where each of the scenarios could feasibly occur, but it is not feasible to predict which of these scenarios will occur due to the stochastic nature of the real-world in which we are attempting to model. We use numerical validation in this case to reduce the complexity required for theoretical validation. Theoretical validation requires the validation of many complex processes, and even after this process, we would not know the complex whether the interaction of these processes are realistic. We argue in this thesis, that both numerical and theoretical validation are required. Through this validation process, we are able to adequately model the transitional dynamics of the electricity mix in the United Kingdom between 2013 and 2018. During this time there was an ∼88% drop in coal use, ∼44% increase in Combined Cycle Gas Turbine (CCGT), ∼111% increase in wind energy and increase in solar from near zero to ∼1250MW. We are therefore able to test our model in a transition of sufficient magnitude. We show in this chapter, that agent-based models are able to mimic the behaviour of the UK electricity market under the same specific scenario conditions. Concretely, we show that under an observed carbon tax strategy, fuel price and electricity demand scenario, the model, ElecSim, closely matches the observed electricity mix between 2013 and 2018. We achieve this by determining an exogenous predicted price duration curve using a genetic algorithm to minimise the error between observed and simulated electricity mix in 2018. The predicted price curve is an arrangement of all price levels in descending order of magnitude. The predicted price duration curve achieved is similar to that of the simulated price duration curve in 2018, increasing confidence in the underlying dynamics of our model. In addition, we compare our projections to those of the BEIS reference scenario from 2018 to 2035 [53]. To achieve this, we use the same genetic algorithm optimisation technique as during our validation stage, optimising for predicted price duration curves for calibration. Our model demonstrates that we are able to closely match the projections of BEIS by finding a set of realistic price duration curves which are subject to investment cycles. Our model, however, exhibits a more realistic step-change in nuclear output than that of BEIS. This is because, whilst BEIS projects a gradual increase in nuclear output, our model projects that nuclear output will grow instantaneously at a single point in time as a new nuclear power plant comes online. This allows us to verify the scenarios of other models, in this case, BEIS' reference scenario, by ascertaining whether the optimal parameters required to achieve such scenarios are realistic. In addition to this, we are able to use these parameters to analyse 'what-if ' questions with further accuracy. As part of this work, we contribute a validated open-source agent-based model called ElecSim. Whilst we have used the United Kingdom as a use-case for this thesis, ElecSim is able to model decentralised markets of various sizes, and can therefore be used to model other countries. To improve our results as well as making the execution time tractable, we increased the temporal granularity of the model using a k-means clustering approach to select a subset of representative days for wind speed, solar irradiance and electricity demand. This subset of representative days enabled us to approximate an entire year and only required a fraction of the total time-steps that would be necessary to model each day of a year independently. We show that we are able to provide an accurate framework, through this addition, to allow policymakers, decision-makers and the public to explore the effects of policy on investment in electricity generators. We demonstrate that with a genetic algorithm approach, we are able to optimise parameters to improve the accuracy of our model. Namely, we optimise the predicted electricity price, the uncertainty of this electricity price and nuclear subsidy. We validate our model using the observed electricity mix between 2013-2018 through a process of calibration. That is, we find the predicted electricity price, uncertainty of this electricity price and nuclear subsidy which match with the stated scenario. We then use these values to project further. A major contribution of this work is to demonstrate that it is possible for agent-based models to accurately model transitions in the UK electricity market. This was achieved by comparing our simulated electricity mix to the observed electricity mix between 2013 and 2018. In this time, a transition from coal to natural gas was observed. We demonstrate that a high temporal granularity is required to accurately model fluctuations in wind and solar irradiance for intermittent renewable energy sources. In this Section, we detail how the architecture of ElecSim has been designed. ElecSim is made up of six parts:

• GenCos which are made up of agents. • Power plants. • A power exchange, which controls an electricity spot market.
• The time-steps. • The data for parametrisation. This includes both the configuration file and data sources. A schematic of ElecSim is displayed in Figure 3.2. We have placed the items in bold which relate to each part of the figure. Next, we will describe each of the elements described in Figure 3.2 in further detail. ElecSim contains a configuration file and a collection of data sources for parametrisation. These data sources contain information such as historical fuel prices, historical plant availability, wind and solar capacity. The configuration file allows for rapid changes to test different hypothesis and scenarios, and points to the different data sources. The configuration file enables one to change the demand growth and shape, future fuel and carbon prices, capital costs, plant availability, investment costs and simulation time. The demand agent is a simplified representation of aggregated demand in a country. The demand is represented as a Load Duration Curve (LDC). An example load duration curve for a year is demonstrated in Figure 3.3. This Figure effectively shows the baseload level at 460 hours, and the peak load at 2400 hours and above, with the profile displayed between these points. An LDC is an arrangement of all load levels in descending order of magnitude. where the lowest segment demand demonstrates baseload, and the highest segment represents peak demand. Each year, the demand agent changes each of the LDC segments proportionally, by multiplying each of the levels by the factor to increase or decrease these load levels. In other words, whilst the magnitude of the LDC may change, the profile does not. The GenCos have two main functions. Investing in power plants and making bids to sell their generation capacity. We will first focus on the buying and selling of electricity, and then cover the investment algorithm. The power exchange (or spot market) runs every year, accepting the lowest bids until supply meets demand. Once this condition is met, the spot price or system marginal price (SMP) is paid to all generators regardless of their initial bid. Generators are motivated to bid their SRMC, to ensure that their generator is being utilised, and reduce the risk of overbidding. Power Plants Power plants are made up of various parameters which dictate their behaviour within the electricity market. For example, cheaper power plants will undercut the costs of more Costs form an important element of markets and investment, and publicly available data for power plant costs for individual countries can be scarce, and when available is presented in different formats. Thus, extrapolation and interpolation is required to estimate costs for power plants of differing sizes, types and years of construction. Users are able to initialise costs relevant to their particular country by providing detailed cost parameters. They can also provide an average cost per MWh produced over the lifetime of a plant, known as the Levelised Cost of Electricity (LCOE). They do this within the data sources file. The parameters used to initialise the power plants are detailed in Table 3.1. Periods have units of years and costs in unit of currency per MW unless otherwise stated. If specific parameters are not known, the LCOE can be used for parameter estimation, through the use of linear optimisation. Constraints can be set by the user, enabling, for example, varying operation and maintenance costs per country as a fraction of LCOE. This includes future costs. To fully parametrise power plants, availability and capacity factors are required. Availability is the percentage of time that a power plant can produce electricity. This can be reduced by forced or planned outages. We integrate historical data to model improvements in reliability over time. The capacity factor is the actual electrical energy produced over a given time period divided by the maximum possible electrical energy it could have produced. The capacity factor can be impacted by regulatory constraints, market forces and resource availability. For example, higher capacity factors are common for photovoltaics in the summer and lower in winter. To model the intermittency of wind and solar power, we allow them to contribute only a certain percentage of their total capacity (nameplate capacity) for each load segment. This percentage is based upon empirical wind and solar capacity factors. In this calculation, we consider the correlation between demand and renewable resources. When initialised, V C is selected from a uniform distribution, with the ability for the user to set a maximum percentage increase or decrease. A uniform distribution was chosen to capture the large deviations that can occur in V C , especially over a long time period. By doing this, the variance in costs between individual power plants for processes such as preventative and corrective maintenance, labour costs and skill, health and safety and chance are different per plant instant. Future work would include finding data to better inform the V C price. Fuel price is controlled by the user; however, there is inherent volatility in fuel price. To take into account this variability, an ARIMA [245] model was fit to historical gas and coal price data. The standard deviation of the residuals was used to model the variance in price that a GenCo will buy fuel in a given year. This considers differences in chance and hedging strategies. The spot market is run by a power exchange. The power exchange effectively ranks all power plant bids on the market from lowest to highest for each demand segment, as shown by Figure 3.3. All bids are accepted until supply meets demand. This dispatch method is known as merit order dispatch. This method effectively ranks ensures that power plants with the lowest marginal cost are brought online first, and those with the highest marginal cost brought on last. For example, expensive peaker plants will only be used at peak times, where all cheaper electricity generation supply is being used. NPV is a summation of the present value of a series of present and future cash flow.
NPV provides a method for evaluating and comparing investments with cash flows spread over many years, making it well suited for evaluating power plants which have a long lifetime. NPV is based upon the fact that current cash flow is worth more than future cash flow. This is due to the fact that money today can be invested and have a rate of return. This means that, for example, $50,000 today is worth more than $50,000 in 10 years time. The value in which future cash flow is worth less than present cash flow is denoted by the discount rate. Equation 3.1 is the calculation of NPV, where t is the year of the cash flow, i is the discount rate, N is total number of periods, or lifetime of power plant, and R t is the net cash flow at time t.

A discount rate set by a GenCo's weighted average cost of capital (WACC) is often used [140]. WACC is the rate that a company is expected to pay on average for its stock and debt. Therefore to achieve a positive NPV, an income larger than the WACC is required. However, a higher WACC is often selected to adjust for varying risk profiles, opportunity costs and rates of return [140]. To account for these differences we sample from a Gaussian distribution, giving us sufficient variance whilst deviating from the expected price. Gaussians are often observed within real-life distributions, and therefore chosen for this work. To calculate the NPV, future market conditions must be considered. For this, each GenCo forecasts N years into the future, which we assume is representative of the lifetime of the plant. As in the real world, GenCos have imperfect information, and therefore must forecast expected demand, fuel prices, carbon price and electricity sale price. This is achieved by fitting functions to historical data. Each GenCo is different in that they will use differing historical time periods of data for forecasting. This is a notoriously difficult task, and there are multiple different ways in which this can be achieved [228]. However for this work, we forecast fuel and carbon price using two different linear regression functions. Linear regression was chosen due to the simplicity of the function, and the absence of unrealistic growth seen in other functions. Demand, however, is forecast using an exponential function, which considers compounded growth, as seen in the real-world. Linear regression is used if an exponential function is found to be sub-optimal. We define sub-optimal as the case that a solution can not be found. We use the linear regression for this, due to its characteristics of reducing unrealistic growth. The forecasted electricity price N years ahead is difficult to ascertain accurately. We therefore use two methods for forecasting these. The first is to simulate a market N years ahead. The second is to optimise for the predicted Price Duration Curve (PDC) using a genetic algorithm. We describe this optimisation in Section 3.3. For the simulated market, the forecasted data is used to simulate a market N years into the future using the electricity market algorithm. We simulate a market based on the expected bidsbased on short run marginal cost (SRMC) -that every operating power plant will make. SRMC is the cost for a power plant to produce one unit of electricity, excluding fixed and capital costs. We include the removal of plants that will be past their operating period, and the introduction of plants that are in construction or pre-development stages. This effectively runs a simulation within a simulation here. Where the GenCos must use a simulation technique to understand future market conditions. There may be scenarios where demand is forecast to grow significantly, and limited investments have yet been made to meet that demand. The expected price would be that of lost load. Lost load is defined as the price customers would be willing to pay to avoid disruption in their electricity supply. To avoid GenCos from estimating large profits, and under the assumption that further power plant investments will be made, the lost load price is replaced with a predicted electricity price using linear regression based on prices at lower points of the demand curve. If zero segments of demand are met, then the lost load price is used to encourage investment. Once this data has been forecasted, the NPV can be calculated. GenCos must typically provide a certain percentage of upfront capital, with the rest coming from investors in the form of stock and shares or debt (WACC). The percentage of upfront capital can be customised by the user in the configuration file. The GenCos then invest in the power plants with the highest NPV. Time-steps Time-steps are the steps made in a simulation which simulate time. Each timestep moves the simulation forward a single iteration, and mimics time in our case. For the time-steps, two approaches were taken. For the first approach, as per Chappin et al. [31], we modelled the LDC of electricity demand with twenty segments. Twenty segments enabled us to capture the variation in demand throughout the year to a high degree of accuracy, whilst reducing computational complexity. However, as we show later in Section 3.4, this led to an overestimation of the supply of IRES. For the second approach, we used representative days to model a year. Representative days in this context are a subset of days which have characteristics, that when scaled proportionally can accurately model an entire year. To select these representative days, we used a k-means approach. We describe this in full detail in Section 3.2.1 Figure 3.4 demonstrates the simulation and how it co-ordinates runs. The world contains data and brings together GenCos, the Power Exchange and demand. The investment decisions are based on future demand and costs, which in turn influence bids made. To explain the simulation, we will start from the left side of Figure 3.4 and move to the right.
A database of power plants is parametrised by a database of parameters, as discussed in Table 3.1. This ensures that the plants have realistic properties. Generation companies (GenCos) own and invest in these power plants through the use of investment decisions. The GenCos than submit bids to dispatch these power plants. They do this using the parameters of the power plants, which for renewable energy inclues the renewable energy capacity factors. For instance, wind, solar and hydro have varying profiles throughout the day and year. These bids are then augmented with the CO 2 price (tax) and fuel price to create the actual bid. The yearly electricity spot market then takes the demand and matches this to the submitted bids in merit order. The demand grows (or falls) each year, depending on the scenario. Exogenous variables include fuel and CO 2 prices as well as demand growth. Once the data is initialised, the model calls on the Power Exchange to operate the yearly electricity spot market. The world also settles the accounts of the GenCos, by paying bids, and removing operating and capital costs as well as loans and dividends. In this Subsection, we describe the work which allowed us to determine the granularity of time-steps. In this problem, there is a trade-off between accuracy and compute time. We visualise this trade-off and choose a cut-off point, in which we receive diminishing returns for an increase in temporal granularity, in terms of accuracy. Specifically, for this, we use representative days. Representative days, in this context, are a subset of days which, when scaled up to 365 days can adequately represent a year. In this work, we initialised the model to a scenario of the United Kingdom as an example. However, the fundamental dynamics of the model remain the same for other decentralised electricity markets. Similar to findings of other authors, using a relatively low number of time-steps from the representative days leads to an overestimation of the uptake of intermittent renewable energy resources (IRES) and an underestimation of flexible technologies [97,157]. This is due to the fact that the full intermittent nature of renewable energy could not be accurately modelled in such a small number of time-steps. For instance, by modelling the entire year in only 20 time-steps, a reduction in the ability for ElecSim to model hourly lulls in wind speed, the absence of solar irradiance at night time and differences in electricity demand. This is critically important for a power system with high amounts of electricity generated from renewable electricity. To address this problem, whilst maintaining a tractable execution time, we approximated a single year as a subset of proportionally weighted, representative days. This enabled us to reduce computation time whilst maintaining accuracy. Each representative day consisted of 24 equally separated time-steps, which model hours in a day. Hourly data was chosen, as this was the highest resolution of the dataset available for offshore and onshore wind and solar irradiance [189]. A lower resolution would allow us to model more days; however, we would lose accuracy in terms of the variability of the renewable energy sources. Similar to Nahmmacher et al. [174] we used a clustering technique to split similar days of weather and electricity demand into separate groups. We then selected the historic day that were closest to the centre of the cluster, known as the medoid, as well as the average of the centre, known as the centroid [174]. Similar to Nahmmacher, we used Ward's clustering algorithm and selected the centroid [127]. However, we also used the k-means clustering algorithm [72]. This was due to the ability for the k-means algorithm to cluster time-series into relevant groups [131]. These days were scaled proportionally to the number of days within their respective cluster to approximate a total of 365 days. A representative day is one where solar and wind are chosen on the same day. This is because on a particular day, wind and solar are correlated. The Ward's clustering algorithm is an extension of the work published in [134]. Equation 3.2 shows the series for a medoid or centroid, selected by the clustering algorithms:

where P x,i h is the medoid for series x, where x ∈ X refers to offshore wind capacity factor, onshore wind capacity factor, solar capacity factor and electricity demand, h is the hour of the day and i is the respective cluster. {P 1 , P 2 , . . . , P 24 } refers to the capacity values at each hour of the representative day. In other words, x is a set of offshore wind capacity factor, onshore wind capacity factor, solar capacity fator and electricity demand. We then calculated the weight of each cluster. This gave us a method of assigning the relative importance of each representative day when scaling the representative days up to a year. The weight is calculated by the proportion of days in each cluster. This gives us a method of determining how many days within a year are similar to the selected medoid or centroid. The calculation for the weight of each cluster is shown by Equation 3.3:

where w i is the weight of cluster i, n i is the number of days in cluster i, and ||N|| is the set of days that have been used for clustering. The next step was to scale up the representative days to represent the duration curve of a full year. A duration curve is similar to the load duration curve shown in Figure 3.3, however can be defined for other processes such as wind speed and solar irradiance. We achieved this by using the weight of each cluster, w i , to increase the number of hours that each capacity factor contributed in a full year. Equation 3.4 details the scaling process to turn the medoid or centroid, shown in Equation 3.2, into a scaled day. Where P x,i h is the scaled day:

where W = {w 1 , w 2 , . . . , w i }.
W is the set of weights, w 1 is the weight for the first day, w 2 the weight for the second day, and w i the i th day. Equation 3.4 effectively extends the length of the day, proportional to the amount of days in the respective cluster. Finally, each of the scaled representative days were concatenated to create a single series used for the calculations in the simulation. This concatenated number contains information of the required capacity factors and the respective duration curve. Equation 3.5 displays the total time series of a series x, where each scaled medoid is concatenated to produce an approximated time series, P x :

the total number of hours in the approximated time series, P x , is equal to the number of hours in a day multiplied by the number of days in a year, which gives the total number of hours in a year (24 × 365 = 8760), as shown by Equation 3.6:

where w ∈ W the Set of clusters. To measure the validity of our approximation using representative days and also compare the optimum number of days, or clusters, we used a technique similar to Poncelet et al. [56,195]. We trialled the number of clusters against three different metrics: Correlation (CE) (CE av ), Normalised Root Mean Squared Error (NRMSE) and Relative Energy Error (REE) (REE av ). REE av is the average value over all the considered time series P x ∈ P compared to the observed average value of the set P x ∈ P. Where P x ∈ P are the observed time series and P x ∈ P are the scaled, approximated time series using representative days. REE av is shown formally by Equation 3.7:

where DC P x t is the duration curve for P x and DC P x t is the duration curve for P x . In this context, the duration curve can be constructed by sorting the capacity factor and electrical load data from high to low. The x-axis for the duration curve exhibits the proportion of time that each capacity factor represents, similar to the load duration curve shown in Figure 3.3. The approximation of the duration curve is represented in this text as DC p x . t ∈ T refers to a specific time step of the original time series. DC refers to the approximated duration curve for P x . Note that in this text |•| refers to the absolute value, and ||•|| refers to the cardinality of a set and ||P|| refers to the total number of of considered time series. Specifically, the sum of the observed values, P x , and approximated values, P x , for all of the time series are summed. The proportional difference is found, which is summed for each of the different series, x, and divided by the number of series, to give REE av . Another requirement is for the distribution of load and capacity factors for the approximated series to correspond to the observed time series. It is crucial that we can account for both high and low levels of demand and capacity factor for IRES generation. This enables us to model for times where flexible generation capacity is required. The distribution of values can be represented by the duration curve (DC) of the time series. Therefore, the average normalised root-mean-square error (NRMSE av ) between each DC is used as an additional metric. The NRMSE av is shown formally by Equation 3.8:

Specifically, the difference between the approximated and observed duration curves for each time-step t is calculated. The average value is then taken of these differences. This average value is then normalised for the respective time series P x . The average of these average normalised values for each time series are then taken to provide a single metric, NRMSE av . The final metric used is the correlation between the different time series. This is used due to the fact that wind and solar output influences the load within a single region, solar and wind output are correlated, as well as offshore and onshore wind levels within the UK. This is referred to as the average correlation error (CE av ) and shown formally by Equation 3.9:

where corr p1,p2 is the Pearson correlation coefficient between two time series p 1 , p 2 ∈ P, shown by Equation 3.10. Here, V p 1 ,t represents the value of time series p 1 at time step t, and Vp 1 ,t refers to the mean of the values of the time series p 1 at time t:

(3.10)

To integrate the additional temporal granularity of the model using the representative days, extra time-steps were taken per year. This did not require a significant change to the ElecSim code, simply stepping through each hour of a representative day to gauge renewable energy capacity factors and electricity demand. This meant that we had to take an additional number of time-steps when compared to the 20 LDC case. Specifically, as we modelled each time-step as an hour, and a day contains 24 hours, we ran 24 time-steps per representative day. The difference between using the hourly representative days and 20 segments, is that peaks and troughs can be modelled more accurately with representative days. An LDC with 20 load segments can accurately model yearly intervals where each interval has an average of 5%, or 18.25 days (365 days ×5% = 18.25 days). It is true that peaks and troughs occur for less than 18.25 days, and therefore this granularity is lost with a 20 segment LDC. This amount of granularity may have been acceptable with predictable generation technologies, but this becomes significantly more challenging when trying to model intermittent renewable electricity. It is true that we could have modelled more than 20 segments in an LDC to have a similar number time-steps as in a set of representative days.
However, this would have given us an average of a year, in regular intervals. This would not provide a diverse set of days which are significantly different enough from each other. We did not want to capture solely the averages, but also the extremes that can occur in a year, which would have been lost through a simple LDC averaging approach. The higher temporal granularity of the model enabled us to accurately model the hourly fluctuations in solar and wind which leads to more accurate expectations of the investment opportunities of these technologies [97,157]. GenCos make bids at the beginning of every time-step, and the Power Exchange matches demand with supply in merit-order dispatch using a uniform pricing market. An example of the electricity mix in a single representative day is shown in Figure 3.5. Figure 3.5 displays the high utilisation of low marginal-cost generators such as nuclear, wind and photovoltaics. At hour 19, an increase in offshore wind leads to a direct decrease in CCGT. In contrast to this, a decrease in offshore and onshore between the hours of 8 and 12 lead to an increase in dispatch of coal and CCGT. One would expect this behaviour to prevent blackouts and meet demand at all times. This process has enabled us to more closely match fluctuations in IRES. In this Section, we detail the validation approaches taken in our model. For this, we take two approaches. One is to compare the price duration curve of the actual vs our simulated price duration curve in 2018 using the 20 time-steps per year approach. The other is to use cross-validation between the years 2013 and 2018, using our representative days approach. Validation of models is important to ascertain that the output is accurate. However, it should be noted that these long-term simulations are not predictions of the future, rather possible outcomes based upon certain assumptions. Jager posits that a certain outcome or development path, captured by empirical data, might have developed in a completely different direction due to chance. However, the processes that emerge from a model should be realistic and in keeping with expected behaviour [123]. We begin by comparing the price duration curve in the year 2018 for the case with 20 time-steps. Figure 3.6 shows the N2EX Day Ahead Auction Prices of the UK [180], the Monte-Carlo simulated electricity prices, and the non-Monte-Carlo electricity price throughout the year 2018. Fuel prices varying throughout a year, as does V C and WACC. WACC is sampled from a Gaussian distribution with a standard deviation of ±3%. V C is sampled from a uniform distribution between 30% and 200% of the mean V C price, whilst fuel price is sampled from the residuals of an ARIMA model fit on historical data. The N2EX Day Ahead Market is a day ahead market run by Nord Pool AS. Nord Pool AS runs the largest market for electrical energy in Europe, measured in volume traded and in market share [180]. We ran the initialisation of the model 40 times to capture the price variance. Outliers were removed as on a small number of occasions large jumps in prices at peak demand occurred which deviated from the mean. We did this, as although this does occur in real life, it occurs at a smaller fraction of the time than 5% of the year (twenty time-steps per year). Therefore the results would be unreasonably skewed for the highest demand segment. Figure 3.6 demonstrates very little variance in the non-stochastic case. This is due to the fact that combined cycle gas turbines (CCGTs) set the spot price. These CCGTs have little variance between one another as they were calibrated using the same dataset. By adding stochasticity of fuel prices and operation and maintenance prices, a curve that more closely resembles the actual data occurs. The stochastic curve, however, does not perfectly fit the real data, which may be due to higher variance in fuel prices and historical differences in operation and maintenance costs between power plants. One method of improving this would be fitting the data used to parametrise to the curve. Figure 3.6 shows that up to 4,000 hours there is an overestimation. Following this, there is an underestimation. This is due to the fact that some power generators are more expensive than the average, and some are less expensive, by chance. In this Section, we detail the approach taken in this work to validate our model using representative days as time-steps. Figure 3.7 displays the error metrics versus number of clusters, and therefore days. This is because from each cluster, the most central day of the cluster is chosen and used. Therefore, from eight clusters, eight representative days are chosen. Centroids are the days at the centre of the cluster, medoids are the average day of the cluster and Ward's is another clustering technique. Both CE av and NRMSE av display similar behaviour for the k-means approach (centroids and medoids), namely the error improves significantly from a single cluster to eight clusters for both centroids and medoids. For the number of clusters greater than eight, there are diminishing returns. For REE av , however, the error metric is best at a single cluster, and gets worse with the number of clusters. The Wards approach [240], however, performs significantly worse for all metrics at every number of clusters. We chose eight clusters, for centroids and medoids, as a compromise between accuracy of the three error metrics and compute time of the simulation. This is because eight was the largest number of clusters that gave us the lowest score for CE av , NRMSE av and REE av without significantly increasing compute time. Whilst there was little significant difference between centroid and medoid, we chose to use the medoids due to the fact that the extreme high and low values would not be lost due to averaging [99]. Therefore, a total of eight representative days were chosen.
To achieve this, we use a Genetic Algorithm (GA) to find the predicted price duration curves, which lead to the smallest error between our simulated electricity mix and the scenarios tested. The scenarios examined here are the observed electricity mix of the UK between 2013 and 2018 and the UK Government Department for Business, Energy and Industrial Strategy (BEIS) reference scenario projected in 2018 till 2035. When projecting the BEIS reference scenario, we also optimise for nuclear subsidy and uncertainty in the price duration curves. As mentioned in Section 3.2, GenCos make investments based upon the net present value. As shown in Equation 3.1, an expectation of the net cash flow, R t is required. The net cash flow, R t , is calculated by subtracting both the operational and capital costs from revenue over the expected lifetime of the prospective plant. The revenue gained by each prospective plant is the expected price they will gain per expected quantity of MWh sold over the expected lifetime of the plant. This is shown formally in Equation 3.11:

where m h,t is the expected quantity of megawatts sold in hour h of year t. PPDC h,t is the Predicted Price Duration Curve (PPDC) at year t and hour h. C var h,t is the variable cost of the power plant, which is dependent on expected megawatts of electricity produced, C c is the capital cost. The predicted price duration curve (PPDC h,t ) is an expectation of future electricity prices over the lifetime of the plant. The PPDC h,t is a function of supply and demand. However, with renewable electricity generator costs falling [121], future prices are uncertain and largely dependent upon long-term scenarios of electricity generator costs, fuel prices, carbon taxes and investment decisions [121]. Due to the uncertainty of future electricity prices over the horizon of the lifetime of a power plant, we have set future electricity prices as an exogenous variable that can be set by the user in ElecSim. To gain an understanding of expected electricity prices that lead to particular scenarios, we use a genetic algorithm optimisation approach. This enables us to understand the range of future electricity prices that lead to certain scenarios developing. In addition, it allows us to understand whether the parameters required for certain scenarios to develop are realistic. This enables us to check the assumptions of our model and the likelihood of scenarios. Further, using these optimised parameters, we are better able to further explore 'what-if ' scenarios. To verify the accuracy of the underlying dynamics of ElecSim, the model was initialised with data available in 2013 and allowed to develop until 2018. We used a genetic algorithm to find the optimum price duration curve predicted (PPDC) by the GenCos ten years ahead of the year of the simulation. This PPDC was used to model expected rate of return of prospective generation types, as shown in Equations 3.1 and 3.11. The genetic algorithm's objective was to reduce the error of simulated and observed electricity mix in the year 2018 by finding a suitable PPDC used by each of the GenCos for investment evaluation. For this experiment, we initialised ElecSim with parameters known in 2013 for the UK. ElecSim was initialised with every power plant and respective GenCo that was in operation in 2013 using the BEIS DUKES dataset [49]. The funds available to each of the GenCos was taken from publicly released official company accounts at the end of 2012 [52]. To ensure that the development of the electricity market from 2013 to 2018 was representative of the actual scenario between these years, we set the exogenous variables, such as carbon and fuel prices, to those that were observed during this time period. In other words, the scenario modelled equated to the observed scenario. The data for the observed EU Emission Trading Scheme (ETS) price between 2013 and 2018 was taken from Ember Climate [208]. Fuel prices for each of the fuels were taken from the department for business, energy and industrial strategy [50]. The electricity load data was modelled using data from Elexon portal ad Sheffield University [62], offshore, and onshore wind and solar irradiance data was taken from Pfenninger et al. [189]. There were three known significant coal plant retirements in 2016. These were removed from the simulation at the beginning of 2016. The price duration curve was modelled linearly in the form y = mx + c, where y is the cost of electricity, m is the gradient and refers to the relative increase in price with respect to demand, x is the demand of the price duration curve, and c is the intercept, or the price of electricity as demand approaches zero. A linear price duration curve was chosen due to its simplicity, small amount of parameters required for optimisation and also the similarity to the actual price duration curve. Equation 3.12 details the optimisation problem formally:

In addition to verifying the ability for ElecSim to mimic observed investment behaviour over five years, we compared ElecSim's long-term behaviour to that of the UK Government's Department for Business, Energy and Industrial Strategy (BEIS) [53]. This scenario shows the projections of generation by technology for all power producers from 2018 to 2035 for the BEIS reference scenario. This is the same scenario as discussed in the next Section. We initialised the model to 2018 based on our previous work [129]. The scenario for the development of fuel prices and carbon prices were matched to that of the BEIS reference scenario [53]. The optimisation approach taken was a similar process to that discussed in Subsection 3.3.2, namely using a genetic algorithm to find the optimum expected price duration curve. However, instead of using a single expected price duration curve for each of the agents for the entire simulation, we used a different expected price duration curve for each year, leading to 17 different curves. This enabled us to model the non-static dynamics of the electricity market over this extended time period. The agents are dynamic, however, not heterogenous in this regard. This is due to the exponentially increasing computational time of calculating predicted price duration curves for each agent.
In addition to optimising for multiple expected price duration curves, we optimised for a nuclear subsidy, S n . Further, we optimised for the uncertainty in the expected price parameters m and c, named σ m and σ c respectively, where σ is the standard deviation in a normal distribution. m and c are the parameters for the predicted price duration curve, as previously defined, of the form y = mx + c. Effectively, σ m and σ c represent the distribution around the price duration curve, that is sampled in a monte-carlo manner. This enabled us to model the different expectations of future price curves between the independent GenCos. The addition of a nuclear subsidy as a parameter is due to the likely requirement for government to provide subsidies for new nuclear [225]. A modification was made to the reward algorithm for the long-term scenario case. Rather than using the discrepancy between observed and simulated electricity mix in the final year (2018) as the reward, a summation of the error metric for each simulated year was used. This is detailed formally in Equation 3.13:

where M and C are the sets of the 17 parameters of m y and c y respectively for each year, y. y ∈ Y refers to each year between 2018 and 2035. m y and c y refer to the parameters for the predicted price duration curve, of the form y = mx + c for the year y. A y o refers to the actual electricity select each individual for P t from C ′ t or P t-1 10: end while mix percentage for the year y and generation type o. Finally, f y o (m y , c y ) refers to the simulated electricity mix percentage with the input parameters to the simulation of m and c for the year y. Genetic Algorithms (GAs) are a type of evolutionary algorithm which can be used for optimisation. We chose the genetic algorithm for this application due to its ability to find good solutions with a limited number of simulation runs, the ability for parallel computation and its ability to find global optima. These characteristics are useful for our application, as a single simulation can take up to 36 hours. In this Section, we detail the genetic algorithm used in this work. Initially, a population P 0 is generated for generation 0. This population of individuals is used for the parameters to the simulation. The output of the simulations for each of the individuals are then evaluated. A subset of these individuals C t+1 ⊂ P t are chosen for mating. This subset is selected proportional to their fitness, with 'fitter' individuals having a higher chance of reproducing to create the offspring group C ′ t+1 . C ′ t+1 have characteristics dependent on the genetic operators: crossover and mutation. The genetic operators are an implementation decision [12]. Once the new population has been created, the new population P t+1 is created by merging individuals from C ′ t+1 and P t . See Algorithm 1 for detailed pseudocode. We used the DEAP evolutionary computation framework to create our genetic algorithm [74]. This framework gave us sufficient flexibility when designing our genetic algorithm. Specifically, it enabled us to persist the data of each generation after every iteration to allow us to verify and analyse our results in real-time. The parameters chosen for the problem explained in Section 3.3.3 was a population size of 120, a crossover probability of 50%, a mutation probability of 20% and the parameters, m and c, as per Equation 3.12, were given the bounds of [0.0, 0.004] and [-30, 100] respectively. The bounds for m and c were calculated to ensure a positive price duration curve, with a maximum price of £300 for 50,000MW. The population size was chosen to ensure a wide range of solutions could be explored, whilst limiting compute time to ∼1 day per generation to allow for sufficient verification of the results. The crossover and mutation probabilities were chosen due to suggestions from the DEAP evolutionary computation framework [74]. The parameters chosen for the genetic algorithm for the problem discussed in Section 3.3.3 are displayed here. The population size was 127, a crossover probability of 50%, a mutation probability of 20%. The parameters m y , c y were given the bounds [0.0, 0.003] and [-30, 50] respectively, whilst σ m and σ c were both given the bounds of [0, 0.001]. Effectively, we did not allow these parameters to be larger or smaller than these bounds. The population size was slightly increased, and the bounds reduced when compared to the parameters for Section 3.3.4. This was to increase the likelihood of convergence to a global optimum, which was more challenging to achieve due to the significantly higher number of parameters. Here we present the results of the problem formulation of Section 3.3.3. Specifically, we compare the ability of our model to that of BEIS in the context of a historical validation between 2013 and 2018 of the UK electricity market. We also compare our ability to generate scenarios up to 2035 with that of BEIS. Figure 3.8 displays the output of ElecSim under the validation scenario, BEIS' projections and the observed electricity mix between 2013 and 2018, as explained in Section 3.3.2. We calibrated our data with the BEIS projection to generate these results. The observed electricity mix changed significantly between 2013 and 2018. A continuous decrease of electricity production from coal throughout this period was observed. 2015 and 2016 saw a marked decrease of coal, which can be explained by the retirement of 3 major coal power plants. The decrease in coal between 2013 and 2016 was largely replaced by an increase in gas. After 2016, renewables play an increasingly large role in the electricity mix and displace gas.
Both ElecSim and BEIS were able to model the fundamental dynamics of this shift from coal to gas as well as the increase in renewables. Both models, however, underestimated the magnitude of the shift from coal to gas. This could be due to unmodelled behaviours such as consumer sentiment towards highly polluting coal plants, a prediction from industry that gas would become more economically attractive in the future or a reaction to The Energy Act 2013 which aimed to close a number of coal power stations over the following two decades [54]. ElecSim was able to closely model the increase in renewables throughout the period in question, specifically predicting a dramatic increase in 2017. This is in contrast to BEIS, who predicted that an increase in renewable energy would begin in 2016. However, both models were able to accurately predict the proportion of renewables in 2018. ElecSim was able to better model the observed fluctuation in nuclear power in 2016. BEIS, on the other hand, projected a more consistent nuclear energy output. This small increase in nuclear power is likely due to the decrease in coal during that year. BEIS consistently underestimated the share of nuclear power. We display the error metrics to evaluate our models 5-year projections in Table 3.3. Where MAE is mean absolute squared error, MASE is Mean Absolute Squared Error (MASE) and Root Mean Squared Error (RMSE) is the root mean squared error. We are able to improve the projections for all generation types when compared to the naive forecasting approach using ElecSim, as shown by the MASE. Where the naive approach is simply predicting the next time-step by using the last known time-step, in this case, the last known time-step is the electricity mix percentage for each generation type in 2013. q q q q q q q q q qq q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q qq q q q q q q q q q q q q q q q q q qq q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q qqq q qq data by BEIS. As shown in Figure 3.8, we are able to predict the rise in renewables and nuclear better than that of BEIS, however, perform slightly worse for coal and gas. Figure 3.9 displays the optimal predicted price duration curve (PPDC) found by the genetic algorithm. This price curve was used by the GenCos to achieve the results shown in Figure 3.11. The orange points show the simulated price duration curve for the first year of the simulation (2018). The red line (Simulated Fit 2018) is a linear regression that approximates the simulated price duration curve (PDC 2018). The blue line shows the price duration curve predicted (PPDC) by the GenCos to be representative of the expected prices over the lifetime of the plant. The optimal predicted price duration curve (PPDC) closely matches the simulated fit in 2018, shown by Figure 3.9. However, the PPDC has a slightly higher peak price and lower baseload price. This could be due to the fact that there is a predicted increase in the number of renewables with a low SRMC. However, due to the intermittency of renewables such as solar and wind, higher peak prices are required to generate in times of low wind and solar irradiance at the earth's surface. ElecSim was able to model the increase in renewables and stability of nuclear energy in this time. ElecSim was also able to model the transition from coal to gas, however, underestimated the magnitude of the transition. This was similar to the projections BEIS made in 2013 as previously discussed. In this Section, we discuss the results of the analysis of the BEIS reference scenario explained in Section 3.3.3. Specifically, we created a scenario that mimicked that of BEIS in ElecSim and optimised a number of parameters using a genetic algorithm to match this scenario. Through this, we are able to gain confidence in the underlying dynamics of ElecSim to simulate long-term behaviours. Further, this enables us to verify the likelihood of the scenario by analysing whether the parameters required to make such a scenario are realistic. Figure 3.11 displays the electricity mix projected by both ElecSim and BEIS. To generate this image, we ran 60 scenarios under the optimal collection of predicted price duration curves, nuclear subsidy and uncertainty in predicted price duration curves. The optimal parameters were chosen by choosing the parameter set with the lowest mean error per electricity generation type and per year throughout the simulation, as shown by Equation 3.13. Figure 3.12 displays the optimal predicted price duration curves (PPDCs) per year of the simulation, shown in blue. These are compared to the price duration curve simulated in 2018, as per Figure 3.9. The optimal nuclear subsidy, S n , was found to be ∼£120/MWh, the optimal σ m and σ c were found to be 0 and ∼0.0006 respectively. In this context, optimal means the values which produced the required electricity mix scenarios.
The BEIS scenario demonstrates a progressive increase in nuclear energy from 2025 to 2035, a consistent decrease in electricity produced by natural gas, an increase in renewables and decrease to almost 0% by 2026 of coal. ElecSim is largely able to mimic the scenario by BEIS. A large increase in renewables is projected, followed by a decrease in natural gas. A significant difference, however, is the step-change in nuclear power in 2033. This led to an almost equal reduction in natural gas during the same year. In contrast, BEIS project a continuously increasing share of nuclear. We argue that the ElecSim projection of nuclear power is more realistic than that of BEIS due to the instantaneous nature of large nuclear power plants coming on-line. This is largely due to ElecSim's discrete nature of power plants. Figure 3.12 exhibits the price curves required to generate the scenario shown in Figure 3.11. The majority of the price curves are similar to the simulated price duration curve of 2018 (Simulated Fit 2018). However, there are some price curves which are significantly higher and significantly lower than the predicted price curve of 2018. These cycles in predicted price duration curves may be explained by investment cycles typically exhibited in electricity markets [91]. In this context, investment cycles reflect a boom and bust cycle over long timescales. When electricity supply becomes tight relative to demand, prices rise to create an incentive to invest in new capacity. Price behaviour in competitive markets can lead to periods of several years of low prices (close to short-run marginal cost) [244]. As plants retire or demand increases, the market becomes tighter until average prices increase to a level above the threshold for investment in new power generators. At this point, investors may race to bring new plants on-line to make the most out of the higher prices. Once adequate investments have been made, the market returns to a period of low prices and low investment until the next price spike [91]. The nuclear subsidy, S n , of ∼£120/MWh in 2018 prices is high compared to similar subsidies, but this may reflect the difficulty of nuclear competing with renewable technology with a shortrun marginal cost that tends to £0. However, subsidising nuclear may be an important option to gain reliable, low-carbon, base load, irrespective of the price. The low values of σ m and σ c demonstrates that the expectation of prices does not necessarily have to differ significantly between GenCos. This may be due to the fact that GenCos have access to the same market information. However, it may be true that agent's portfolios and cash reserves affect their decisions. q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q qq q q q q q q q q qq q

In this section we display scenario runs of ElecSim using both 20 time-steps per year and using representative days. In this Section, we vary the carbon tax and grow or reduce total electricity demand. This enables us to observe the effects of a carbon tax on investment. In this work, we have presented scenarios where electricity demand decreases 1% per year, due to the recent trend in the UK. For the first scenario run displayed, we have approximated the predictions by the UK Government, where carbon tax increases linearly from £18 to £200 by 2050 [51]. Figure 3.13a demonstrates a significant increase in gas turbines in the first few years, followed by a decrease, with onshore wind increasing. Figure 3.13b displays a run with a £40 carbon tax. This run demonstrates a higher share of onshore wind than in the previous scenario. We experimented with the following levels of carbon tax: £10 ($13), £20 ($26) and £70 ($90) with demand decreasing 1% per year. This was chosen due to the increasing efficiency of homes, industry and technology, as this was the recent trend in the UK. We run each scenario eight times to capture the stochastic nature of the process. Via the observation of the emergent investment behaviour until 2050, an understanding of how real-life investors may behave emerges. Figure 3.14a shows that with a carbon tax of £10, whilst renewable technology does grow, gas power plants provide the majority of supply in each year. However, at a level of £20 the increase in wind turbines is enough to match gas turbines. A carbon tax of £70, however, shows a near 100% uptake of wind turbines. These results demonstrate the large impact that a carbon tax can have on the final electricity mix over the long term. Policy makers can effectively remove high proportions of carbon emitting technology lock-in by setting a carbon tax. The limitations to this work, from the perspective of policy makers, is the inability to know for certain what the final electricity mix will be as the results show a large amount of variance.
The reason that onshore wind and CCGT become dominant is due to their perceived low-cost over the lifetime of these projects when compared to the other electricity generation technologies. The capacity factor of onshore wind is lower than that of CCGT, and therefore a high investment in onshore wind is required to meet increasing shares of electricity demand. For example, if onshore wind has a capacity factor of 0.2, an installed capacity 5× the total electricity demand would be required to meet 100% of demand. This is more expensive than investing in CCGT, which has a higher capacity factor and can therefore meet more demand with a lower total installed capacity. This trade-off between CCGT and onshore wind becomes more nuanced with different sized carbon taxes. For instance, in the £70 carbon tax case, it becomes economically efficient to install large amounts of onshore wind. It is currently infeasible, however, for the power supply to be provided solely by wind turbines today. This overestimation, however, is due to the low time granularity of the model [37]. This scenario therefore assumes perfect storage capabilities. This is because the model assumes that onshore wind's capacity factor is spread evenly across each of the LDC segments. However, this is not the case in the real world. For instance, the capacity factor of wind may be 0 at some times, and 1 at other times. This means that simply increasing the number of wind turbines will never meet electricity demand at times when there is no wind. Therefore, at these times other technologies are required, leading to a mix of different electricity generators. More detail from the representative days can capture days of zero wind and high demand. These runs demonstrate that a consistent, but relatively low carbon tax can have a larger impact in the uptake of renewable energy than increasing carbon tax over a long time frame. We hypothesise that an early carbon tax affects the long-term dynamics of the market for many years. We, therefore, suggest early action on carbon tax to transition to a low-carbon energy supply. Agent-based models provide a method of simulating investor behaviour in an electricity market. We observed that an increase in carbon tax had a significant impact on investment. These findings enable policymakers to better understand the impact that their decisions may have. For a high uptake of renewable energy technology, rapid results can be seen after ten years with a carbon tax of £70 ($90). In this Section, we discuss various scenarios under the model which uses representative days as time-steps. This work builds upon the work in Section 3.3; we used the same predicted price duration curves as modelled on BEIS' scenario. We selected an optimal carbon tax level which would reduce both electricity price and carbon emissions, as shown later in Chapter 5. The optimal carbon tax strategy found in Chapter 5 is shown by Figure 3.15. Each of the scenarios were run ten times to display any variability in the results. We chose ten runs to limit both computation time and cost. Figures 3.16 and 3.17 show the electricity mixes of various demand scenarios. Figure 3.16 displays the scenarios in which demand either stays flat, or decreases by 1% and 2%. For these scenarios, it can be seen that solar is the dominant electricity supply, supplying ∼50%, with nuclear power second supplying between 20% and 30%. With a decreasing demand scenario of 1% per year, as shown by Figure 3.16b, nuclear provides a higher proportion by the year 2034, of ∼30%, however before the year 2033, provides a similar proportion to the other scenarios as shown by Figures 3.16a and 3.16c. For the scenarios shown in Figure 3.16, CCGT, coal and onshore provide around ∼10% each by 2034. Coal and CCGT, however, progress towards 0% whereas onshore wind increases. This is to be expected due to the high carbon price, as shown by Figure 3.15. Offshore does not exhibit a high amount of investment. We believe this is the case as offshore wind is more expensive than onshore wind, and in our scenario subsidies other than for nuclear are not modelled. Figure 3.17 shows scenarios where demand increases per year. Whilst electricity mix distribution is similar to the scenarios shown in Figure 3.16, solar plays a significantly increased role than nuclear. This may be down to the large expense of nuclear, and the long time of deployment of this type of technology. Solar power, on the other hand, is able to be installed much more quickly to meet the high demand. This is especially true for the scenarios shown in Figures 3.17b and 3.17c where demand rises by 2% and 2.5% per year respectively. These results show the impact that different demand scenarios can have on the long-term electricity mix. It demonstrates that the electricity mix may change significantly dependent on a few factors, such as the electricity demand profile. Policy makers should therefore be conscious of the different electricity mixes that could occur, and provide support to ensure that demand is met at all times. For instance, a high nuclear subsidy may be required to meet the base-load demand, or tax credits for solar to ensure that increasing demand is met. It is also possible for generator companies to understand which technologies will be useful in different demand scenarios. This would ensure that generator companies can make better decisions in the face of uncertainty. The limitations, however, are that these scenarios do not place a probability to any of these scenarios. As, these results present the 2% increase and 2% decrease scenarios as equally likely. This may not be the case, and are dependent on more exogenous variables. This prevents policy makers and generator companies from making informed decisions about how they should adapt for relative scenarios. In this Section, we investigate a sensitivity analysis of ElecSim, where we vary the weighted average cost of capital and the down payment required for investment.
We used the reference scenario discussed in Section 3.4, with the optimal carbon tax to reduce both emissions and electricity price. The work presented in this section is in addition to the work published in [134]. We ran ten iterations per weighted average cost of capital and down payment element. We did this due to the monte-carlo nature of the simulation. We chose ten runs to give us sufficient variance in results, but reduce compute power, to reduce both time and cost of calculation. We show the results in the year 2035 due to this being where the largest change in electricity mix is likely to be observed over the whole horizon. Including more years in the analysis is possible, but would lead to many graphs with limited additional information. Figure 3.19 displays the results of the sensitivity analysis for the Weighted average cost of capital for non-nuclear power generators. For this, we trialled nine different Weighted Average Cost of Capital (WACC) values, where a value of 5.9% is the reference case [140]. It can be seen that the WACC has an effect on the total investment in solar, nuclear and CCGT. With a WACC equal to or greater than 7.4%, nuclear increases significantly, whilst solar decreases. Nuclear has a WACC of 10%; therefore 7.4% may be the point where nuclear becomes more competitive than solar in an environment where a low-carbon electricity supply is elicited from an optimal carbon tax. Offshore and coal do not change significantly over different levels of WACC. This may be due to the fact that CCGT and onshore are more competitive than coal and offshore, respectively, without external subsidies. Onshore seems to play a larger role at the lowest WACC, 3.9%. This may be due to onshore wind's high competitiveness when compared to nuclear. Figure 3.20 displays the relative carbon emissions in 2035. With a low WACC of 3.9%, the relative carbon emissions falls, on average, to zero. This seems to be due to the high levels of solar, onshore and nuclear. As WACC increases, so does carbon emissions, until there is a WACC of 0.5, where it reduces slightly. This is seemingly due to the higher levels of CCGT and coal, which is able to displace solar. However, it must be noted that in this scenario with the optimal carbon tax, the relative carbon emissions remains low. Figure 3.21 displays the sensitivity analysis results for different levels of down payment required for all investments. We varied the down payment required between the values of 10% and 40%. As down payment required increases, so does nuclear and onshore, whilst solar decreases. CCGT also shows an increase with down payment required. The increase in down payment may help nuclear, due to the high costs of WACC. With a higher down payment, the total costs of the project will fall when compared to other, cheaper, generators. It is likely that nuclear displaces onshore solar nuclear offshore ccgt coal 3.9 4.4 4.9 5.4 5.9 6.4 6.9 7.4 7.9 3.9 4.4 4.9 5.4 5.9 6.4 6.9 7. 4  Weighted average cost of capital (%) Relative carbon emissions Fig. 3.20 Sensitivity analysis where weighted average cost of capital (WACC) was varied. Results compare relative carbon emissions in 2035. solar in this case. CCGT increases up until a 35% down payment required. This may be due to the increased use of onshore wind, where CCGT and coal is required to fill for times of low wind speeds. Figure 3.22 displays the relative carbon emissions versus down payment required for investors. As down payment increases, so does relative carbon emissions. This is down to the increasing role that CCGT and coal play in the electricity mix and decreasing solar capacity. A downpayment of 10% seems to have the lowest carbon emissions. This is due to the high investment in solar, and low investment in CCGT and coal. These results demonstrate the importance of parameters on the final electricity mix. For instance, the weighted average cost of capital can have a large impact on the long-term electricity mix. As government and policy makers have the ability to access capital at low-cost, it adds an important lever that policy makers can have to influence the final electricity mix. For instance, providing financial backing for solar, which are highly capital intensive, could significantly reduce the barriers to entry for this technology. Generator companies also have the ability to analyse the attractiveness of each technology based on their access to capital. However, these results do not show a sensitivity analysis on different costs. For example, capital expenditure may reduce in the future for solar. This would change the results significantly. And so, generator companies and policy makers should consider a wider range of future technology cost scenarios in addition to the sensitivity analysis presented here. As with all models, ElecSim features a number of limitations. That is, ElecSim can not model the real world perfectly, and thus should be relied on only with a spectrum of other quantitative and qualitative methods. A number of limitations exist of ElecSim, which have been listed in this Section. This list, however, is not exhaustive, due to the possibility for unknown unknowns. 1. Requirement of exogenous price prediction curve. The prediction of electricity prices in the long-term future is difficult to predict with certainty. In ElecSim, we are required to predict electricity prices over the lifetime of a plant. These plants often have lifetimes of 25 years or more. Two methods, which were utilised by the GenCos, were used by ElecSim: simulating a market ten years into the future and optimising for predicted price duration curve. The first method of simulating a market ten years into the future has two major limitations. Firstly, the investments made ten years into the future, the costs of fuel, tax all have a large effect. Predictions which may seem perfectly reasonable at the time, may be unreasonable by the time the market occurs in real life.
For instance, investments in another country may significantly reduce the cost of renewables in the country where the simulation is being run. Secondly, even if the simulation ten years into the future is correct, it does not mean that it will be representative of the entire 25 years. However, to fix this problem would require many more simulations, which may not make the model more accurate. To overcome this limitation, we removed the simulation to generate a predicted price curve and used optimisation to find an optimal predicted price duration curve to generate a scenario. We were then able to check whether the predicted price duration curve had reasonable values. Whilst this enabled us to remove the problem of simulating an uncertain market in the future, it removed the ability to rapidly and quickly trial significantly different scenarios. This is because the predicted price duration curves were optimised for a certain scenario. Whilst it remained possible to change multiple parameters, it became difficult to trial many different carbon prices, as these have a significant impact on the predicted price duration curve. Long-term validation, as mentioned in Section 3.3 is particularly difficult due to the uncertainties involved. We, therefore, do not propose that ElecSim is used as a ground truth. Instead, it should be used as a way to understand possibilities that could arise from scenarios under certain assumptions and conditions. It is only with this knowledge that the model should be used. However, we argue that, whilst limited, quantitative advice has a large role to play in understanding the future, and reducing the uncertainty through understanding complex systems, such as electricity markets. It is only through building and understanding such models that the uncertainty and complexity involved can be understood. Additionally, when modelling the future, many other models compare their outcomes to other models for a specific scenario. Whilst this has the benefit of checking that the underlying dynamics of the models are similar, or the same, it does not test whether the models fit correctly to real-life. It is for this reason that we used cross-validation as well as comparing our model to that of the UK Government, BEIS department. However, a significant problem with using historical data to validate a model, is that, we know more exogenous information about what happened during the past than we do about the future. Therefore the comparison between the past and future is not equal. For instance, we knew that certain coal power plants were taken out of service during the 2013-2018 period. We do not know for certain that this will happen in the future. It is therefore not possible to equate the past and forward scenarios. This fundamental truth will always remain a problem. However, the modelling process is stronger by using as much available data as possible. If we did not use historic data to calibrate and validate our model, we would be losing valuable information for verifying our model, as well as finding problems within our model. For instance, we may have introduced coding errors that would not have been found without using historic data. It is for this reason that we chose this approach. Due to the high granularity of ElecSim on a technical and temporal level, it becomes difficult to model electricity markets on a global scale. Modelling a single run of the United Kingdom can take ∼16 hours, therefore modelling the rest of the world would take an infeasible amount of time and computing power to function. We run these on a Microsoft Azure, Dsv3-series virtual machine, which are made up of Intel Xeon Platinum 8272CL (Cascade Lake), Intel Xeon 8171M 2.1GHz (Skylake), Intel Xeon E5-2673 v4 2.3 GHz (Broadwell), or the Intel Xeon E5-2673 v3 2.4 GHz (Haswell) processors in a hyper-threaded configuration. Within an electricity market, a component of space becomes important. This is because electricity is not created in the locations that it is consumed. For instance, offshore wind is typically clustered on banks which are shallow enough to install wind turbines, whilst demand is typically clustered on land in residential and urban areas. In addition to this, different areas may have different demand profiles, for instance with high income earners or industry, typically consuming more than low-income areas. By accounting for these variations in spatial resolution, one is able to optimise the location of power plants, to minimise electricity transfer losses. In addition, different distribution networks can be designed, to account for differences in demand. As ElecSim is a national model, and the UK is a relatively small country, we did not take into account the supply-demand curve for resources such as natural gas, coal and other fossil fuels. This is due to the fact that the fossil fuel market is much bigger than solely the United Kingdom. In other words, an increase in demand in the UK would not increase the price on a global scale. We did not take into account limits of land usage for solar panels and wind turbines in the model either. This is due to the fact that future technologies may overcome such challenges. For example, currently, offshore wind turbines must be placed in geographies with suitable floors. However, with the advent of floating wind turbines, these limitations could be reduced. Additionally, solar panels may become more effective at being placed offshore. Whilst it is possible to compare multiple model outputs that look at a horizon of 20+ years, it is not possible to know whether all or any of these models are correct, or incorrect. Therefore, by comparing multiple models, it may be the case that all of the models are wrong and that no correct inference can be made. This is a significant challenge of long term energy modelling. Other approaches include cross-validation, where historic data is used to compare model outputs. However, whilst all models are wrong, insights can be gained from the functioning of these models to provide additional information that would not be available without these quantitative models.
Whilst it is possible that a model can closely match a scenario, it is not possible to know which scenario will occur in the future. Therefore, whilst one is able to model many different scenarios, it is reasonable to expect that none of these modelled scenarios actually occur. This means that although a model accurately models the system it is designed to replicate, none of the model outputs reflect that of real-life. This is a significant challenge of scenario modelling, to which there is no perfect solution. Increasing the number of scenarios and placing likelihoods to each of these scenarios is one option, however, it is not perfect. In this work, we did not model subsidies such as the contracts for difference auction, which is an auction found in the UK, where generators are able to sell their electricity at a stable level, irrespective of the daily price of electricity. This is because we focused on marking ElecSim a generalisable model for different countries. To ensure this, we had to focus on global parameters and reduce complexity. We did, however, model a simple subsidy where an extra price is paid for every MWh for Nuclear. Liberalised electricity markets with many heterogeneous players are suited to be modelled with ABMs. ABMs incorporate imperfect information as well as heterogeneous actors. ElecSim models imperfect information through forecasting of electricity demand and future fuel and electricity prices. This leads to agents taking risk on their investments, and model market conditions more realistically. In this chapter, we have demonstrated that it is possible to use ABMs to simulate liberalised electricity markets. Through validation, we are able to show that our model, ElecSim, is able to accurately mimic the observed, real-life scenario in the UK between 2013 and 2018. This provides confidence in the underlying dynamics of ElecSim, especially as we are able to model the fundamental transition between coal and natural gas observed between 2013 and 2018 in the UK. In addition to this, we were able to compare our long-term scenario to that of the UK Government, Department for Business, Energy & Industrial strategy. We show that we are able to mimic their reference scenario; however, demonstrate a more realistic increase in nuclear power. The parameters that were gained from optimisation show that the BEIS scenario is realistic, however, a high nuclear subsidy may be required. To improve the accuracy of our model, we used eight representative days of solar irradiance, offshore and onshore wind speed and demand to approximate an entire year. The particular days were chosen using a k-means clustering technique, and selecting the medoids. This enabled us to accurately model the daily fluctuations of demand and renewable energy resources. In addition to this, a method of dealing with the non-validatable nature of electricity markets, as per the definition of Hodges et al. is to vary input parameters over many simulations and look for general trends [105]. This could be achieved using ElecSim through the analysis of a reference case, and a limited set of scenarios which include the most important uncertainties in the model structure, parameters, and data, i.e. alternative scenarios which have both high plausibility and major impacts on the outcomes. Additionally, we showed a number of scenarios, and shows that total demand has an effect on the electricity mix. An increasing demand, year-on-year, can lead to an increase in solar to accommodate this demand. However, if demand reduces, there is a higher investment in nuclear, which contributes to the electricity mix by 2035. We ran a sensitivity analysis of Weighted Average Cost of Capital (WACC) and down payment required. We showed that these two variables have a large effect on the total electricity mix by the year 2035, which in turn affects the total carbon emissions. We, therefore, show that the input assumptions have an effect on the simulation, which must be considered when analysing model outputs. In this chapter, we use several different machine learning and statistical methods to predict electricity demand 30 minutes ahead as well as a day ahead. By predicting electricity demand ahead, GenCos are able to better schedule their power plants to dispatch to meet this demand. The 30 minutes ahead methodology is used in the work on day-ahead work. We utilise the errors from the day ahead predictions to see what the impact of such errors are on the electricity market over the long-term using ElecSim. The work looks specifically at the difference in electricity mix with prediction error, as well as carbon emitted. Whilst much of the literature investigates the short-term benefits of using machine learning in making short-term forecasts, this work goes further in that it integrates the short-term characteristics of machine learning into the long-term model, ElecSim. This helps to identify how the long-term market can be changed through algorithms with differing accuracies. This directly relates to the aims of the thesis, which is to look beyond the short-term impact of using machine learning, but to look at the wider impacts on the long-term market. The work on 30-minute ahead forecasting was published in [131]. We introduce this work in Section 4.1. Section 4.2 provides a literature review on the topic of demand forecasting. We introduce the methods used in Section 4.3. Sections 4.4 and 4.5 look at 30-minute ahead predictions and day-ahead predictions respectively. Additionally, Section 4.5 integrates these day ahead projections into the ElecSim model. The need for accurate load forecasting is essential for control and planning of electricity generation in electrical grids due to the fact that supply must meet demand [156]. Short-term electricity demand forecasting has become increasingly important due to the introduction of competitive energy markets. Accurate estimates of demand are required so that the correct amount of electricity is purchased on the wholesale market [57]. Electricity is unique to other commodities in that it must be either consumed the moment that it is generated or stored. The difficulties in storing electricity arise from high installation and maintenance costs, inefficiencies and low capacity [196].
It is therefore important to match demand to supply, and thus regulate frequency. This is because, if there is a mismatch between supply and demand, frequency will either increase or decrease. Failure to accurately forecast electricity demand can lead to financial loss and/or system-wide blackouts [101]. We focus on electricity demand in this work and not supply, solar or wind due to limited computing power and time. However, the conclusions found here could be extended to these timeseries. The integration of higher proportions of intermittent renewable energy sources (IRES) in the electricity grid will mean that the forecasting of electricity demand will become increasingly important and challenging. Examples of IRES are solar panels and wind turbines, which fluctuate in terms of power output based on localised wind speed and solar irradiance. However, as supply must meet demand at all times and the fact that IRES are less predictable than dispatchable energy sources such as coal and combined-cycle gas turbines (CCGTs), this means that extra attention must be made in predicting future demand if we wish to keep, or better reduce, the current frequency of blackouts [156]. A dispatchable source is one that can be turned on and off by human control and therefore, able to adjust output just in time, at a moment convenient for the grid. Typically, peaker plants, such as reciprocal gas engines, are used to fill fluctuations in demand that had not been previously planned for. Specifically, peaker plants meet the peaks in demand where other cheaper options are at full capacity. These peaker plants are typically expensive to run and have higher greenhouse gas emissions than their non-peaker counterparts [162]. Whilst peaker plants are also dispatchable plants; not all dispatchable plants are peaker plants. Figure 4.1 shows the differences between baseload generation, peaking resources and those in between. Peaker power plants will match peak demand, as shown by the Figure, whereas baseload generation is run at all times. This includes some gas power plants and nuclear plants, which have a low short run marginal cost. It is unlikely that a nuclear power plant will be used as a peaker plant due to its inflexibility and slow ramp up and ramp down times, and low short run marginal cost. As can be seen, the electricity generation types with the lowest marginal costs, such as nuclear and water power are dispatched first. Next, natural gas is dispatched which has a lower cost than coal and pump power, and finally the top power refers to the expensive peaker power plants, which are used to match peak demand only due to their high costs. To reduce reliance on peaker plants, it is helpful to know how much electricity demand there will be in the future so that more efficient plants can be used to meet this expected demand. This is so that these more efficient plants can be brought up to speed at a time suitable to match the demand. Forecasting a day into the future is especially useful in decentralised electricity markets which have day-ahead markets. Decentralised electricity markets are ones where electricity is provided by multiple generation companies, as opposed to a centralised source, such as a government. To aid in this prediction, machine learning and statistical techniques have been used to accurately predict demand based on several different factors and data sources [131], such as weather [109], day of the week [6] and holidays [237]. The introduction of smart meters in many countries (USA, Europe, Canada and South Korea) has led to an influx of high granularity electricity consumption data that can be used for load forecasting [55]. Smart meters are digital devices that measure the electricity consumption of individual households at regular intervals (intervals of an hour or less) and offer two-way communication between the meter and utility company. Smart meters aid customers to understand precisely how much electricity they consume at different time intervals, and enable dynamic pricing [3]. Dynamic pricing allows utilities to charge varying prices at different times, for instance, charging a higher price when costly generation sources are used in times of peak demand, and lower prices at night time or weekends when demand is low [122,154]. In this chapter, we forecast both 30-minutes ahead using smart meter data, as well as 24-hours ahead to simulate the process made for a day-ahead market. Firstly, we explore short term load-forecasting at an interval of 30 minutes ahead and cluster similar users based on their electricity usage. A variety of different forecasting techniques were evaluated such as Random Forests [103], Long-Short Term Memory neural networks (LSTM) [104], Multilayer Perceptron neural networks [234] and Support Vector Regression (SVR) [58]. Random Forests are an ensemble-based learning method for classification and regression, and are made up of many decision trees. LSTM networks are recurrent neural networks which remember values over arbitrary time intervals. Multilayer Perceptrons are a popular type of neural network which are made up of a minimum of three layers and can be used to make nonlinear predictions. SVRs are supervised learning models which analyse data used for regression analysis. To improve forecasting results, the clustering of smart meter data was evaluated to identify similar customers, which may have a more similar, and therefore easier to predict, profile. The technique used for this was k-means clustering. An average 24-hour electricity load profile was calculated, and the result used for clustering. The clustered sub-system is then aggregated and separate models trained on this aggregate. The yearly, weekly and daily periodicity of electricity load is accounted for by input variables into the models. Once forecasts for each cluster are made using the individual models, the results are aggregated for the final predictions. These predictions are compared to the actual results, and the accuracy measured using mean absolute percentage error (MAPE). Whilst the work carried out for short-term forecasting does not relate directly to the longterm characteristics of ElecSim and the work presented previously, we used the methodology to inform the long-term characteristics of the electricity market.
This work therefore provided the foundation for looking at the long-term using ElecSim. Therefore, secondly, we introduce day-ahead forecasting and observe the impact errors have on the long-term dynamics of the market. Various studies have looked at predicting electricity demand at various horizons [7,111,217]. However, the impact of poor demand predictions on the long-term electricity mix has been studied to a lesser degree. We compare several machine learning and statistical techniques to predict the energy demand for each hour over the next 24-hour horizon. We chose to predict over the next 24 hours to simulate a day-ahead market, which is often seen in decentralised electricity markets. However, our approach could be utilised for differing time horizons. In addition to this, we use our long-term agent-based model, ElecSim [129,134], to simulate the impact of different forecasting methods on long-term investments, power plant usage and carbon emissions for the years 2018 through 2035 in the United Kingdom. Our approach, however, is generalisable to any country through parametrisation of the ElecSim model. As part of our work, we utilise online learning methods to improve the accuracy of our predictions. Online learning methods can learn from novel data while maintaining what was learnt from previous data. Online learning is useful for non-stationary datasets and time-series data where recalculation of a model would take a prohibitive amount of time. Offline learning methods, however, must be retrained every time new data is added. Online approaches are constantly updated and do not require significant pauses while the offline training is being re-run. By training, in an offline manner, on data that has already been used for training, the computational load and time required increases. We trial different algorithms and train different models for different times of the year. Specifically, we train different models for the different seasons. We also split weekdays and train both weekends and holidays together. This is due to the fact that holidays and weekend exhibit similar load profiles due to the reduction in industry electricity use and an increase in domestic. This enables a model to become good at a specific subset of the data which share similar patterns, as opposed to having to generalise to all of the data. Examples of the algorithms used are linear regression, lasso regression, random forests, support vector regression, multilayer perceptron neural network, box-cox transformation linear regression and the passive aggressive model. We expect a-priori that online algorithms will outperform the offline approach. This is due to the fact that the demand time-series is non-stationary, and thus changes sufficiently over time. In terms of the models, we presume that the machine learning algorithms, such as neural networks, support vector regression and random forests will outperform the statistical methods such as linear regression, lasso regression and box-cox transformation regression. We expect this due to the fact that machine learning has been shown to be able to learn more complex feature representations than statistical methods [217]. However, it should be noted, that such a-priori intuition, is no substitute for analytical evidence and can (and has) been shown to be wrong in the past, due to imperfect knowledge of the data and understanding of some of the black box models, such as neural networks. Using online and offline methods, we take the error distributions, or residuals, and fit a variety of distributions to these residuals. We choose the distribution with the lowest sum of squared estimate of errors (SSE). SSE was chosen as the metric to ensure that both positive and negative errors were treated equally, as well as ensuring that large errors were penalised more than smaller errors. We fit over 80 different distributions, which include the Johnson Bounded distribution, the uniform distribution and the gamma distribution. The distribution that best fits the respective residuals is then used and sampled from to adjust the demand in the ElecSim model. We then observe the differences in carbon emissions, and which types of power plants were both invested in and utilised, with each of the different statistical and machine learning methods. To the best of our knowledge, this is the most comprehensive evaluation of online learning techniques to the application of day-ahead load forecasting as well as assessing the impacts of the errors that these models produce on the long-term electricity market dynamics. We show that online learning has a significant impact on reducing the error for predicting electricity consumption a day ahead when compared to traditional offline learning techniques, such as multilayer artificial neural networks, linear regression, extra trees regression and support vector regression, which are models used in the literature [4,32,156]. For a full list of algorithms used in this thesis see Table 4.3. We show that the forecasting algorithm has a non-negligible impact on carbon emissions and use of coal, onshore, photovoltaics, reciprocal gas engines and CCGT. Specifically, the amount of coal, photovoltaics, and reciprocal gas used from 2018 to 2035 was proportional to the median absolute error, while both onshore and offshore wind are inversely proportional to the median absolute error. Total investments in coal, offshore and photovoltaics are proportional to the median absolute error, while investments in CCGT, onshore and reciprocal gas engines are inversely proportional. The contributions of this work are:

1. A methodology to forecast smart-meter electricity demand using the k-means clustering technique. 2. The evaluation of different online and offline learning models to forecast the electricity demand profile 24 hours ahead. 3. Evaluation of poor predictive ability on the long-term electricity market in the UK through the perturbation of demand in the ElecSim simulation. In this Section, we carry out a literature review on 30-minute ahead forecasting, day-ahead forecasting and online forecasting methods. In addition, we cover the literature on the impact of forecasting on electricity markets. The forecasting of aggregated and clustered electricity demand has been the focus of a considerable amount of research in recent years. The research can generally be classified into two classes, Artificial Intelligence (AI) methods [139,173,200] and classical time series approaches [111,178].