Representing value functions in power systems using parametric network series

We describe a novel architecture for modeling the cost-to-go function in approximate dynamic programming problems involving country-scale , real-life electrical power generation systems . Our particular scenario features a heterogeneous power grid including dozens of renewable energy plants as well as traditional ones ; the corresponding state space is in the order of thousands of variables of different types and ranges . While Artificial Neural Networks are a natural choice for modeling such complex cost functions , their effective use hinges on exploiting the particular structure of the problem which , in this case , involves seasonal patterns at many different levels ( day , week , year ) . Our proposed model consists of a series of neural networks whose parameters are themselves parametric functions of a time variable . The parameters of such functions are learned during training along with the network parameters themselves . The new method is shown to outperform the standard backward dynamic programming program currently in use , both in terms of the objective function ( total cost of operation over a period ) and computational cost . Last , but not least , the resulting model is readily interpretable in terms of the parameters of the learned functions , which capture general trends of the problem , providing useful insight for future improvements . Keywords : energy dispatch , approximate dynamic programming , cost function approximation , artificial neural networks , parametric network series , reinforcement learning . 1 INTRODUCTION . The operation of electrical energy systems involving a mixture of thermal and renewable sources is particularly challenging due to a number of factors . The value of water stored in the dams needs to be assessed constantly depending on the availability of other sources and short , middle and long term forecasts , which are highly variable and depend on the seasons in complex ways . Other renewable sources such as wind and solar power also depend on highly variable factors . Thermal power plants , on the other hand , need to be bootstrapped over relatively long periods before reaching full capacity , so that the effect of turning them on and off also has lasting implications . Last but not least , the demand of electricity is highly variable and exhibits complex seasonal patterns at different scales . The concrete case that we are dealing with is the country-wide energy dispatch of Uruguay . This paper presents the preliminary implementation of a novel method for computing the optimal operation method of the Uruguayan grid . The method has been implemented within the Electric Energy Systems Platform ( SimSEE ) .1 The SimSEE system is currently in use by the Electric Market Administration ( ADME ) of Uruguay for continuously programming the optimal operation of the Uruguayan electrical system at several different time scales . Besides Uruguay , the SimSEE system is currently in use in República Dominicana and Belize . Besides optimizing the schedule , SimSEE is also capable of producing detailed simulations ( through precise physical models ) of the evolution of the system in terms of the actual continuous state-space . This feature is key in the development of our proposed method , which can be seen as a plug-in replacement for the operation optimization component of SimSEE . 1 { SimSEE } https : //simsee.org Methods based on Approximate Dynamic Programming ( ADP ) ( Bertsekas & Tsitsiklis , 1996 ; Sutton , 1995 ) , are the usual choice for optimizing the operation of energy systems . These methods rely on estimating the cost of operation for any possible initial state of the system . Unfortunately , when the number of state variables is large , ADP methods suffer from the so called Bellman ’ s curse of dimensionality ( Bellman , 1957 ) , meaning that the number of states and actions on which the cost function needs to be evaluated grows prohibitively large and thus can not be reliably estimated . This is true even for sophisticated variants of ADP such as Stochastic Dual Dynamic Programming ( SDDP ) ( Pereira & Pinto , 1991 ) . Moreover , the SDDP method is particularly sensitive to inputs with high variability such as renewable-energy sources . The aforementioned problem arises when attempting to explicitly evaluate all possible values of the state variable , whose number grows exponentially with the dimension of the state space . This is obvious even if the state variables are discrete . When the variables are continuous , traditional ADP methods transform them into discrete variables via some form of quantization . In such scenarios , a common strategy is to employ dimensionality reduction techniques such as Principal Component Analysis ( PCA ) ( Jolliffe , 2005 ) . Of course , the price of quantization and dimensionality reduction is that the true value function can only be evaluated in an approximate way . The current method used by SimSEE is a traditional Backward-ADP recursion applied over a quantized , reduced state-space of the whole Uruguayan system . The recent diversification of the power generation matrix has exerted a significant stress on the aforementioned method . This is particularly so for the short term operation , which is re-computed hourly . 1.1 VALUE FUNCTION APPROXIMATION . A recent alternative approach to the above techniques is to construct an implicit representation of the value function ( Powell , 2011 ; Sutton & Barto , 2018 ) . Instead of reducing the dimensions and/or quantizing the state space , a continuous model is built in the original state space based on a set of values of the function evaluated at arbitrary positions . Here , a rich set of tools from Approximation Theory is available to choose , construct and evaluate the appropriate approximation method for a particular task . Kernel methods ( Xu et al. , 2014 ) are a popular choice which is backed by the elegant theory of Reproducing Kernel Hilbert Spaces ( Paulsen & Raghupathi , 2016 ) . Another natural family of methods , more flexible , but also harder to characterize , is that of artificial neural networks ( ANN ) , which are well known for being universal function approximators ( R.Barron , 1994 ) . The above methods , however , have their drawbacks too . For instance , it has been shown that , if no additional measures are taken , the number of samples of the value function required for the overall approximation-based ADP method to converge can be even larger than that required by using traditional , explicit evaluation methods ( Du et al. , 2020 ) . Luckily , such requirement can be significantly relaxed if appropriate variance reduction techniques are applied , e.g. , Common Random Numbers ( Christophe et al. , 2015 ) . In the particular case of ANN approximators applied to an heterogeneous energy system , the above measure might not be enough . The great flexibility of ANNs also implies a great sensitivity to the input data , which in our case is highly variable due to the random nature of renewable energy sources . In this challenging scenario , further measures need to be taken in order to obtain parsimonious approximations . 1.2 PARAMETRIC NETWORK SERIES . Luckily for us , the signals and processes involved in the planning of energy dispatch usually exhibit regular patterns . This can be exploited to impose parsimonious approximations which extrapolate reasonably to unseen states . Our proposed method combines the flexibility of ANNs with prior information about the problem . In a nutshell , the value function , which is a function of state and time , is approximated by one neural network per time slot . The architecture of the network is the same for all time slots , reflecting the fact that the structure of the system itself does not change . The parameters do vary across networks , albeit in a controlled fashion : for any given link in the architecture , the corresponding weight is a function of time . The general idea is depicted in Figure 1 . 1.3 WHY NOT A SINGLE NET ? . Naturally , one could use a single parametric function to model the whole value function across all time steps . The reason for us to opt for a sequence of smaller models is again computational performance : instead of computing the whole approximation from scratch at each time step ( usually an hour ) , we can quickly update the overall model by dropping the first model ( at t = 1 ) , adding a new one at the end , and running a few iterations . We call this a sliding window strategy , in reference to similar modeling patterns used in other fields . 1.4 RELATED WORKS . As mentioned , the use of approximation in value functions is an established technique described in classic textbooks such as ( Bertsekas & Tsitsiklis , 1996 ; Powell , 2011 ; Sutton & Barto , 2018 ) . Also , incorporating time dependence in approximation models and , in particular , artificial neural networks , is an active line of research . The main difference between the related works in this matter is in how the time dependence is imbued into the architecture . Recurrent Neural Networks ( RNNs ) are widely deployed to capture time-dependent patterns using their current widely adopted incarnation , the so called Long-Short Term Memory ( LSTM ) networks ( Hochreiter & Schmidhuber , 1997 ) . In these models , the hidden layer outputs depend not only on their current input , but also on their own previous output , in a way similar to a recursive filter . Such architectures are able to produce outputs which depend not only on the current input , but on past inputs as well . However , these kind of architectures are incapable of capturing periodic patterns , especially if such patterns have low frequencies , such as periodic seasonal patterns . Another widespread technique to represent time in ANNs is generally known as positional encoding ( see Zheng et al . ( 2021 ) for a recent review on the subject ) . The typical positional encoding method involves auxiliary sinusoidal inputs to the network , usually several of them , with different frequencies . Although their typical use is in the context of language translation ( see e.g . ( Gehring et al. , 2017 ) ) , where the frequencies are high ( letter , word , phrase , a few time steps ) , nothing prevents one from using such techniques in a scenario such as the one described in this paper . The main difference between the “ positional encoding ” approach and our proposed method lies in the way that this periodical information is fed into the system . Whereas the former uses these auxiliary signals as inputs to the whole network , and their influence on the weights is implicitly learned through standard backpropagation techniques , our method imposes a periodic pattern explicitly on the weights themselves , giving them some room to accomodate for the particularities not captured by the periodic model . 1.5 CONTRIBUTIONS . In summary , the main contributions of this work are two . First , we develop a novel architecture for learning value functions in approximate dynamic programming problems ; the novelty lies in the use of a series of neural networks of identical architecture , one per time slot , where the weights are functions of time rather whose parameters are learned at training time . Second , we demonstrate the use of the proposed model on a real-life , complex case , where we can evaluate the actual performance of the model in terms of real operation cost savings . 1.6 DOCUMENT ORGANIZATION . The rest of the document is organized as follows . In Section 2 we provide a formal introduction to the problem and the notation used throughout the paper . Section 3 introduces the problem of electric systems operation . The proposed method is described in detail in Section 4.4 . Section 5 shows a detailed comparison between the current model in production and the proposed method , and discusses the results . Concluding remarks are given in Section 6 .

Summary: This paper proposed using time-dependent parameters to model the value function for evaluating the cost-to-go function in power systems. This work can be regarded as an extension from the traditional adaptive dynamic programming (ADP) with the study on how to represent the value function that fits the uncertainty of the dynamics of power systems induced by the installation decentralized energy resources (DERs). This work shows the performance of the proposed method on a realistic simulation. The background knowledge is well written and clear.