Representing value functions in power systems using parametric network series

We describe a novel architecture for modeling the cost-to-go function in approximate dynamic programming problems involving country-scale , real-life electrical power generation systems . Our particular scenario features a heterogeneous power grid including dozens of renewable energy plants as well as traditional ones ; the corresponding state space is in the order of thousands of variables of different types and ranges . While Artificial Neural Networks are a natural choice for modeling such complex cost functions , their effective use hinges on exploiting the particular structure of the problem which , in this case , involves seasonal patterns at many different levels ( day , week , year ) . Our proposed model consists of a series of neural networks whose parameters are themselves parametric functions of a time variable . The parameters of such functions are learned during training along with the network parameters themselves . The new method is shown to outperform the standard backward dynamic programming program currently in use , both in terms of the objective function ( total cost of operation over a period ) and computational cost . Last , but not least , the resulting model is readily interpretable in terms of the parameters of the learned functions , which capture general trends of the problem , providing useful insight for future improvements . Keywords : energy dispatch , approximate dynamic programming , cost function approximation , artificial neural networks , parametric network series , reinforcement learning . 1 INTRODUCTION . The operation of electrical energy systems involving a mixture of thermal and renewable sources is particularly challenging due to a number of factors . The value of water stored in the dams needs to be assessed constantly depending on the availability of other sources and short , middle and long term forecasts , which are highly variable and depend on the seasons in complex ways . Other renewable sources such as wind and solar power also depend on highly variable factors . Thermal power plants , on the other hand , need to be bootstrapped over relatively long periods before reaching full capacity , so that the effect of turning them on and off also has lasting implications . Last but not least , the demand of electricity is highly variable and exhibits complex seasonal patterns at different scales . The concrete case that we are dealing with is the country-wide energy dispatch of Uruguay . This paper presents the preliminary implementation of a novel method for computing the optimal operation method of the Uruguayan grid . The method has been implemented within the Electric Energy Systems Platform ( SimSEE ) .1 The SimSEE system is currently in use by the Electric Market Administration ( ADME ) of Uruguay for continuously programming the optimal operation of the Uruguayan electrical system at several different time scales . Besides Uruguay , the SimSEE system is currently in use in República Dominicana and Belize . Besides optimizing the schedule , SimSEE is also capable of producing detailed simulations ( through precise physical models ) of the evolution of the system in terms of the actual continuous state-space . This feature is key in the development of our proposed method , which can be seen as a plug-in replacement for the operation optimization component of SimSEE . 1 { SimSEE } https : //simsee.org Methods based on Approximate Dynamic Programming ( ADP ) ( Bertsekas & Tsitsiklis , 1996 ; Sutton , 1995 ) , are the usual choice for optimizing the operation of energy systems . These methods rely on estimating the cost of operation for any possible initial state of the system . Unfortunately , when the number of state variables is large , ADP methods suffer from the so called Bellman ’ s curse of dimensionality ( Bellman , 1957 ) , meaning that the number of states and actions on which the cost function needs to be evaluated grows prohibitively large and thus can not be reliably estimated . This is true even for sophisticated variants of ADP such as Stochastic Dual Dynamic Programming ( SDDP ) ( Pereira & Pinto , 1991 ) . Moreover , the SDDP method is particularly sensitive to inputs with high variability such as renewable-energy sources . The aforementioned problem arises when attempting to explicitly evaluate all possible values of the state variable , whose number grows exponentially with the dimension of the state space . This is obvious even if the state variables are discrete . When the variables are continuous , traditional ADP methods transform them into discrete variables via some form of quantization . In such scenarios , a common strategy is to employ dimensionality reduction techniques such as Principal Component Analysis ( PCA ) ( Jolliffe , 2005 ) . Of course , the price of quantization and dimensionality reduction is that the true value function can only be evaluated in an approximate way . The current method used by SimSEE is a traditional Backward-ADP recursion applied over a quantized , reduced state-space of the whole Uruguayan system . The recent diversification of the power generation matrix has exerted a significant stress on the aforementioned method . This is particularly so for the short term operation , which is re-computed hourly . 1.1 VALUE FUNCTION APPROXIMATION . A recent alternative approach to the above techniques is to construct an implicit representation of the value function ( Powell , 2011 ; Sutton & Barto , 2018 ) . Instead of reducing the dimensions and/or quantizing the state space , a continuous model is built in the original state space based on a set of values of the function evaluated at arbitrary positions . Here , a rich set of tools from Approximation Theory is available to choose , construct and evaluate the appropriate approximation method for a particular task . Kernel methods ( Xu et al. , 2014 ) are a popular choice which is backed by the elegant theory of Reproducing Kernel Hilbert Spaces ( Paulsen & Raghupathi , 2016 ) . Another natural family of methods , more flexible , but also harder to characterize , is that of artificial neural networks ( ANN ) , which are well known for being universal function approximators ( R.Barron , 1994 ) . The above methods , however , have their drawbacks too . For instance , it has been shown that , if no additional measures are taken , the number of samples of the value function required for the overall approximation-based ADP method to converge can be even larger than that required by using traditional , explicit evaluation methods ( Du et al. , 2020 ) . Luckily , such requirement can be significantly relaxed if appropriate variance reduction techniques are applied , e.g. , Common Random Numbers ( Christophe et al. , 2015 ) . In the particular case of ANN approximators applied to an heterogeneous energy system , the above measure might not be enough . The great flexibility of ANNs also implies a great sensitivity to the input data , which in our case is highly variable due to the random nature of renewable energy sources . In this challenging scenario , further measures need to be taken in order to obtain parsimonious approximations . 1.2 PARAMETRIC NETWORK SERIES . Luckily for us , the signals and processes involved in the planning of energy dispatch usually exhibit regular patterns . This can be exploited to impose parsimonious approximations which extrapolate reasonably to unseen states . Our proposed method combines the flexibility of ANNs with prior information about the problem . In a nutshell , the value function , which is a function of state and time , is approximated by one neural network per time slot . The architecture of the network is the same for all time slots , reflecting the fact that the structure of the system itself does not change . The parameters do vary across networks , albeit in a controlled fashion : for any given link in the architecture , the corresponding weight is a function of time . The general idea is depicted in Figure 1 . 1.3 WHY NOT A SINGLE NET ? . Naturally , one could use a single parametric function to model the whole value function across all time steps . The reason for us to opt for a sequence of smaller models is again computational performance : instead of computing the whole approximation from scratch at each time step ( usually an hour ) , we can quickly update the overall model by dropping the first model ( at t = 1 ) , adding a new one at the end , and running a few iterations . We call this a sliding window strategy , in reference to similar modeling patterns used in other fields . 1.4 RELATED WORKS . As mentioned , the use of approximation in value functions is an established technique described in classic textbooks such as ( Bertsekas & Tsitsiklis , 1996 ; Powell , 2011 ; Sutton & Barto , 2018 ) . Also , incorporating time dependence in approximation models and , in particular , artificial neural networks , is an active line of research . The main difference between the related works in this matter is in how the time dependence is imbued into the architecture . Recurrent Neural Networks ( RNNs ) are widely deployed to capture time-dependent patterns using their current widely adopted incarnation , the so called Long-Short Term Memory ( LSTM ) networks ( Hochreiter & Schmidhuber , 1997 ) . In these models , the hidden layer outputs depend not only on their current input , but also on their own previous output , in a way similar to a recursive filter . Such architectures are able to produce outputs which depend not only on the current input , but on past inputs as well . However , these kind of architectures are incapable of capturing periodic patterns , especially if such patterns have low frequencies , such as periodic seasonal patterns . Another widespread technique to represent time in ANNs is generally known as positional encoding ( see Zheng et al . ( 2021 ) for a recent review on the subject ) . The typical positional encoding method involves auxiliary sinusoidal inputs to the network , usually several of them , with different frequencies . Although their typical use is in the context of language translation ( see e.g . ( Gehring et al. , 2017 ) ) , where the frequencies are high ( letter , word , phrase , a few time steps ) , nothing prevents one from using such techniques in a scenario such as the one described in this paper . The main difference between the “ positional encoding ” approach and our proposed method lies in the way that this periodical information is fed into the system . Whereas the former uses these auxiliary signals as inputs to the whole network , and their influence on the weights is implicitly learned through standard backpropagation techniques , our method imposes a periodic pattern explicitly on the weights themselves , giving them some room to accomodate for the particularities not captured by the periodic model . 1.5 CONTRIBUTIONS . In summary , the main contributions of this work are two . First , we develop a novel architecture for learning value functions in approximate dynamic programming problems ; the novelty lies in the use of a series of neural networks of identical architecture , one per time slot , where the weights are functions of time rather whose parameters are learned at training time . Second , we demonstrate the use of the proposed model on a real-life , complex case , where we can evaluate the actual performance of the model in terms of real operation cost savings . 1.6 DOCUMENT ORGANIZATION . The rest of the document is organized as follows . In Section 2 we provide a formal introduction to the problem and the notation used throughout the paper . Section 3 introduces the problem of electric systems operation . The proposed method is described in detail in Section 4.4 . Section 5 shows a detailed comparison between the current model in production and the proposed method , and discusses the results . Concluding remarks are given in Section 6 .

Summary: This paper provides a deep learning-based method to estimate value functions for multi-step dispatch on a power grid. This method entails training a different neural network to approximate the value function for each time step within a fixed horizon; supervision is provided via point-wise estimates of the states visited within rolled-out trajectories, as estimated via compound TD estimation. The method is demonstrated to have comparable performance to the "classic" Bellman iteration-based method that is currently used within the SimSEE dispatch system (currently employed by the Uruguayan electricity system operator), while running significantly (~150x) faster.