Representing value functions in power systems using parametric network series
Authors: Ruben Chaer
Ximena Caporale
Vanina Camacho
Ignacio Ramírez
Venue: ICLR 2022 Submitted (2022)
Keywords: approximate dynamic programming
cost function approximation
artificial neural networks
parametric network series

Abstract: We describe a novel architecture for modeling the cost-to-go function in approximate dynamic programming problems involving country-scale, real-life electrical power generation systems. Our particular scenario features a heterogeneous power grid including dozens of renewable energy plants as well as traditional ones; the corresponding state space is in the order of thousands of variables of different types and ranges. While Artificial Neural Networks are a natural choice for modeling such complex cost functions, their effective use hinges on exploiting the particular structure of the problem which, in this case, involves seasonal patterns at many different levels (day, week, year). Our proposed model consists of a series of neural networks whose parameters are themselves parametric functions of a time variable. The parameters of such functions are learned during training along with the network parameters themselves. The new method is shown to outperform the standard backward dynamic programming program currently in use, both in terms of the objective function (total cost of operation over a period) and computational cost. Last, but not least, the resulting model is readily interpretable  in terms of the parameters of the learned functions, which capture general  trends of the problem, providing useful insight for future improvements.

Under review as a conference paper at ICLR 2022
REPRESENTING VALUE FUNCTIONS IN POWER SYSTEMS USING PARAMETRIC NETWORK SERIES
Anonymous authors Paper under double-blind review
ABSTRACT
We describe novel architecture for modeling the cost-to-go funetion in approxi- mate dynamic programming problems involving country-scale, real-life electrical power generation systems. Our particular scenario features a heterogeneous power grid including dozens of renewable energy plants well traditional ones; the corresponding state space is in the order of thousands of variables of different types and ranges. While Artificial Neural Networks are natural choice for mod- eling such complex cost functions, their effective use hinges on exploiting the particular structure of the problem which, in this case, involves seasonal patterns at many different levels (day, week, year). Our proposed model consists of se- ries of neural networks whose parameters are themselves parametric functions of a time variable. The parameters of such functions are learned during training along with the network parameters themselves. The new method is shown to outperform the standard backward dynamic programming program currently in use, both in terms of the objective function (total cost of operation over a period) and com- putational cost. Last, but not least, the resulting model is readily interpretable in terms the parameters of the learned functions, which capture general trends the problem, providing useful insight for future improvements.
Keywords:
energy dispatch, approximate dynamic programming, cost function approximation, arti- ficial neural networks, parametric network series, reinforcement learning.
INTRODUCTION
The operation of electrical energy systems involving a mixture of thermal and renewable sources is particularly challenging due to number of factors. The value of water stored in the dams needs to be assessed constantly depending on the availability of other sources and short, middle and long term forecasts, which highly variable and depend on the seasons in complex ways. Other renewable sources such as wind and solar power also depend on highly variable factors. Thermal power plants, on the other hand, need be bootstrapped over relatively long periods before reaching full capacity, so that the effect of turning them on and off also has lasting implications. Last but not least, the demand of electricity highly variable and exhibits complex seasonal patterns at different scales.
The concrete case that we are dealing with is the country-wide energy dispatch of Uruguay. This pa- per presents the preliminary implementation method of the Uruguayan grid. The method has been implemented within the Electric Energy Sys- tems Platform (SimSEE) The SimSEE system is currently in use by the Electric Market Adminis- tration (ADME) of Uruguay for continuously programming the optimal operation of the Uruguayan electrical system at several different time scales. Besides Uruguay, the SimSEE system is currently in use in República Dominicana and Belize.
novel method for computing the optimal operation
Besides optimizing the schedule, SimSEE is also capable of producing detailed simulations (through precise physical models) of the evolution of the system in terms of the actual continuous state-space. This feature is key in the development of our proposed method, which can be seen as a plug-in replacement for the operation optimization component of SimSEE.
{ (tissss)ttts://////

---

Under review as a conference paper at ICLR 2022
Methods based Approximate Dynamic Programming (ADP) (Bertsekas & Tsitsiklis 1996 Sut- ton 1995). are the usual choice for optimizing the operation of energy systems. These methods rely on estimating the cost of operation for any possible initial state of the system. Unfortunately, when the number of state variables is large, ADP methods suffer from the so called Bellman's 's curse of dimensionality (Bellman 1957), meaning that the number of states and actions which the cost function needs to be evaluated grows prohibitively large and thus cannot be reliably estimated. This is true even for sophisticated variants of ADP such as Stochastic Dual Dynamic Programming (SDDP) (Pereira & Pinto [191). Moreover, the SDDP method is particularly sensitive to inputs with high variability such as renewable-energy sources.
The aforementioned problem arises when attempting to explicitly evaluate all possible values of the state variable, whose number grows exponentially with the dimension of the state space. This obvious even if the state variables are discrete. When the variables are continuous, traditional ADP methods transform them into discrete variables via some form of quantization. In such scenarios, common strategy is to employ dimensionality reduction techniques such as Principal Component Analysis (PCA) Jolliffe 2005). Of course, the price of quantization and dimensionality reduction is that the true value function can only be evaluated in approximate way.
The current method used by SimSEE is a traditional Backward- AD recursion applied over a quan- tized, reduced state-space of the whole Uruguayan system. The recent diversification of the power generation matrix has exerted a significant stress on the aforementioned method. This is particularly so for the short term operation, which is re-computed hourly.
1.1 VALUE FUNCTION APPROXIMATION
A recent alternative approach to the above techniques is to construct an implicit representation of the value function (Powell 2011, Sutton & Barto 2018). Instead of reducing the dimensions and/or quantizing the state space, continuous model is built in the original state space based on set of values of the function evaluated at arbitrary positions. Here, a rich set of tools from Approximation Theory is available to choose, construct and evaluate the appropriate approximation method for particular task. Kernel methods (Xu et al. 2014) are a popular choice which is backed by the elegant theory of Reproducing Kernel Hilbert Spaces (Paulsen & Raghupathi 2016). Another natural family of methods, more flexible, but also harder to characterize, is that of artificial neural networks (ANN), which are well known for being universal function approximators (arro 1994).
The above methods, however, have their drawbacks too. For instance, it has been shown that, if no additional measures are taken, the number of samples of the value function required for the overall approximation-bast
ADP method to converge can be even larger than that required by using traditional, explicit evaluation methods (Du et al.] (020).. Luckily, such requirement can be significantly relaxed if appropriate variance reduction techniques are applied, e.g., Common Random Numbers (Christophe et al.) 2015).
In the particular case of ANN approximators applied to an heterogeneous energy system, the above measure might not be enough. The great flexibility of ANNs also implies great sensitivity to the in- put data, which in our case is highly variable due to the random nature of renewable energy sources. In this challenging scenario, further measures need to be taken in order to obtain parsimonious approximations.
1.2 PARAMETRIC NETWORK SERIES
Luckily for us the signals and processes involved in the planning of energy dispatch usually exhibit regular patterns. This can be exploited to impose parsimonious approximations which extrapolate reasonably to unseen states. Our proposed method combines the flexibility of ANNs with prior information about the problem. In nutshell, the value function, which is function of state and time, is approximated by one neural network per time slot. The architecture of the network is the same for all time slots, reflecting the fact that the structure of the system itself does not change. The parameters do vary across networks, albeit in a controlled fashion: for any given link in the architecture, the corresponding weight is function of time. The general idea is depicted in Figure T]
2

---

Under review as a conference paper at ICLR 2022
W(
W(1)
W(T)
g(t.a)
Figure Parametric network series. When re-computing the hourly model, we can do warm- restart by droping the leftmost network and adding new one to the right.
1.3 WHY NOT A SINGLE NET?
Naturally, one could use a single parametric function to model the whole value function across all time steps. The reason for to opt for sequence of smaller models is again computational performance: instead of computing the whole approximation from scratch at each time step (usually an hour), we can quickly update the overall model by dropping the first model (at / 1), adding new one at the end, and running a few iterations. We call this a sliding window strategy, in reference similar modeling patterns used in other fields.
1.4 RELATED WORKS
As mentioned, the use of approximation in value functions is an established technique described
in classic textbooks such as (Bertsekas & Tsitsiklis 1996, Powell 2011; [Stton Barto 2018). Also, incorporating time dependence in approximation models and, in particular, artificial neural networks, is an active line of research. The main difference between the related works this matter is in how the time dependence is imbued into the architecture.
Recurrent Neural Networks (RNNs) are widely deployed to capture time-dependent patterns us- ing their current widely adopted incarnation, the called Long-Short Term Memory (LSTM) net- works (Hochreiter Schmidhuber 1997). these models, the hidden layer outputs depend not only on their current input, but also on their filter. Such architectures are able to produce outputs which depend not only on the current input, but on past inputs well. However, these kind of architectures incapable of capturing periodic patterns, especially if such patterns have low frequencies, such as periodic seasonal patterns.
previous output, in way similar to a recursive
Another widespread technique represent time in ANNs is generally known as positional encoding (see /Zeng et al. (2021) for a recent review on the subject). The typical positional encoding method involves auxiliary sinusoidal inputs the network, usually several of them, with different frequen- cies. Although their typical use is in the context of language translation (see e.g. (Gehring et al.) 2017 where the frequencies high (letter, word, phrase, few time steps), nothing prevents one from using such techniques in scenario such as the one described in this paper.
The main difference between the "positional encoding" approach and our proposed method lies in the way that this periodical information is fed into the system. Whereas the former uses these auxil- iary signals as inputs to the whole network, and their influence on the weights implicitly learned through standard backpropagation techniques, our method imposes a periodic pattern explicitly on the weights themselves, giving them some room to accomodate for the particularities not captured by the periodic model.
1.5 CONTRIBUTIONS
In summary, the main contributions of this work are two. First, we develop a novel architecture for learning value functions in approximate dynamic programming problems; the novelty lies in the use of series of neural networks of identical architecture, one per time slot, where the weights are func- tions of time rather whose parameters are learned training time. Second, we demonstrate the use
3

---

Under review as a conference paper at ICLR 2022
of the proposed model real-life, complex case, where we can evaluate the actual performance of the model in terms of real operation cost savings.
1.6 DOCUMENT ORGANIZATION
The rest of the document is organized as follows. In Section 2 we provide a formal introduction to the problem and the notation used throughout the paper. Section 3 introduces the problem of electric systems operation. The proposed method is described in detail in Section 4.4 Section shows detailed comparison between the current model in production and the proposed method, and discusses the results. Concluding remarks are given in Section
2
PRELIMINARIES
We follow the usual notation and conventions used in approximate dynamic programming. The task to operate a system over a time period in way that minimizes the total operation cost over that period. The time is observed and operated at regular time intervals tT. In energy systems, T may represent an hour, day, a week, etc. Here is the discrete time index. The state of the system at time I, represented by E X, evolves according to the following rule:
F++1 f(r,, ue, We,
The vector S4 represents the controllable variables (the action), we is vector of exogenous, random, uncontrollable variables, and f()) is the state evolution function. Note that, as the notation implies, the feasible set of actions 22 may vary through time. In the case of electric power systems, the function f(+) can modeled with great precision and thus can be assumed known. Then, possible to compute, the cost, at time t, of taking a particular action ti, while in state It, given the exogenous variables Wy. We call this stage cost function and write it as c(Fe, Uy We. t).
2.1 THE FUTURE COST-TO-GO, OR VALUE, COST FUNCTION
When we are about to take given action, it is not the stage cost that we are interested in, but its long term impact in the future cost of the system. Unfortunately, the system is subject to random fluctuations due to Wt, making the future cost random variable. The future cost function J(,, () is the expected value of the future cost, assuming that one can take the best action for every possible realization of wt. The value of J (z, t). in turn, can be decomposed as the stage cost at time t, and the future cost at time t + 1 and state Tt+1, leading to the well known Bellman recursion:
[ 1) Ew, [e(xe, Me, We, 1) + - J(f(xt, Ues We, 1), 1)],
where Ue is the best possible action given the current state Da and particular realization of the random inputs wy, which is represented by the random variable W., and g E (0, 1] is discount factor, generally defined by the compound interest rate of the eurrency (in this case, USD):
u: arg min minffe(xs, w,,t) + J(f wt, f),t 1)}. CER,
2.2
VALUE ITERATION
The above equation defines mapping between the current state 2t, realization of w,, and the action ue. We call this mapping an operation policy. Ideally, if J(x,,t) was perfectly known to us, the policy obtained above would be the optimal policy of the system. However, there is clear "chicken-and-egg" problem between u, and J J(x,, l), and so practical solutions need to provide starting point.
One of the well established methods in the theory of approximate dynamic programming (see (Bert-
sekas & Tsitsiklis 1996. Powell [011]; Sutton & Barto 2018)) is to transform the above recursive problem into an iterative one, by starting with an arbitrary future function approximation j(U) (x², t)
(here the superindex (0) refers to the initial iteration) and then estimating ((k) t) for ls 1,.. until convergence criterion is met.
4

---

Under review as a conference paper at ICLR 2022
Table 1: STM state space description.
variable
Description
cardinality
Volume lake Rincón de Bonete
10
2
Combined cycle boiler #1 blowdown (4h) and steam turbine #1 loading (2h) Combined cycle boiler #2 blowdown (4h) and steam turbine loading (2h)
4
Combined cycle boiler #1 cooling time (120h) h)
5
Combined cycle boiler #2 cooling time (120h)
Volume lake Palmar
7
Volume in lake Salto Grande
8
Runoff status of the Río Negro basin
9
Runoff status of the Uruguay River basin
SCENARIO
3.1
BACKGROUND ON ELECTRICAL POWER SYSTEMS
Traditional, thermal power generating plants (natural gas, combined cycles, coal-fired plants, nu- clear) have high variable cost (measured in USD per MWh), but very predictable. On the contrary, renewable sources such as hydroelectric, solar or wind have zero variable cost, but are highly unpredictable.
When renewable resources are abundant (sun, wind, etc.), the cost of supplying the demand may be On the other hand, when such resources are short, the system must rely on the highly costly thermal sources. Thus, the ever increasing addition of wind and solar energy in electrical power generation systems implies savings, but also greatly complicates planning, as the fluctuations in the stage costs become larger and more unpredictable. It is therefore very important to construct the best possible stochastic models for the resources involved. These models must capture the correlations between the different resources (e.g., wind at nearby solar plants) as well as their temporal depen- dencies. This subject is in itself a large area of active research. The interested reader is referred to (Flieller& & Chaer 2020).
The state transition function is also paramount importance. Naturally, these functions are specific to the physical characteristics of each energy power generation system. For the particular case of the Uruguayan system see (Chaer & Monzon 2008).
Last, but not least, the value function is modeled that it measures the economic impact of the different actions. Different time scales involve different actions, each with its own range of options, and thus different value functions and operational restrictions are applied in each case. We provide more information on this in appendixes B and C.
Traditional value function approximation methods either construct independent estimations Ja,, t) for each time I, or attempt to construct a single J(r,, 1) that works for all I. Our proposed method, which lies in between these two approaches, exploits the continuity in time of the value function to construct a series of approximations (J(r,, t), 1,..., nz) where the corresponding parameters (O,, t I,..., Te)) are themselves a function of the time t. The parameters of these guiding func- tions are themselves learned within the iterative approximation loop. The technical details of this method are described in the next section.
3.2 SHORT TERM (STM) OPERATION OF THE URUGUAYAN SYSTEM
In recent years, the Uruguayan system has undergone radical changes in generation matrix (Cor- nalino et al. 2018) with the widespread incorporation of wind and solar sources. These changes have exerted considerable pressure on the current optimization tools in use, thus motivating the development of newer approaches such as the presented in this work.
The STM contains 9 visible state variables (several other state variable exist, but are not shown since the optimal operation does not depend them). Table I shows detail of these variables; the last column shows the number of discrete values into which each variable is quantized when solving the problem using the classic Bellman recursion-based optimization method currently in production.
5

---

Under review as a conference paper at ICLR 2022
3
2.5
2
1.5
IB mm xixl
x
1
x1/2
0.5
0
1
6
Time step.
Figure 2: Depiction of the common random numbers in one-dimensional state space. The blue and red trajectories correspond to two different initial states. The dotted trajectories correspond to one random seed, and the solid ones to another.
PROPOSED METHOD
The overall algorithm begins with an initial continuous value function estimation j( (x, X, I. E [ne.. Here [nx] is an abbreviation for 2,......... n2), and X is the continous state space. The superscript indicates the initial iteration & 0. Each iteration i) simulation, ii) sampling of the value function (at discrete points), and iii) approximation (of the continuous value function) based on the aforementioned points. The following subsections describe each of these steps in detail, albeit for generic architecture. The last subsection deseribes the spe- cific architecture used in our case. diagram of the overall algorithm is also provided Appendix A.
O involves three steps:
4.1 SIMULATION USING COMMON RANDOM NUMBERS
In order construct approximation Jax t). the function J(x, t) needs to be sampled number of points state space. In order to obtain a set of interesting points on which to evaluate the function, well as their corresponding values, simulate the operation of the system subject to number of different realizations of the external random variables (e.g., rain, sunshine, wind).
Starting at state To x, the system is allowed to evolve up pre-specified time horizon n4 by drawing samples from we & W., computing the optimal action ues and updating F++1 < f(r,, ut, Wt, f). A well known issue with this scheme is that the variance of the estimated variable [xx 1) grows quickly as the horizon ni increases, thus limiting the ability of the overall method to evaluate the long term impact of the chosen actions.
The common random numbers technique, depicted in Figure 2 greatly reduces the variance of the estimates (at the cost of increasing their bias) by using, for every simulation stage, fixed set of n. pseudo-random ensembles (W1, W )..)) A given ensemble w,- (wro, Wrl. Wyn.) is called chronicle.
Since we always begin from the same known state the simulation results in trajectories, one per chronicle. Since the trajectories depend globally on the current approximation the value function, fx) t), a different set of trajectories will be obtained at each iteration
4.2 POINT-WISE EVALUATION OF THE VALUE FUNCTION
Once the n, trajectories computed, the sample values ( (x, need be estimated all the states visited by all the trajectories. For this, we use a variant of the method described in (Sutton &
Barto 2018
Chapter 12) compound TD estimation. Given trajectory x" (2)(k) t [n.]].

---

Under review as a conference paper at ICLR 2022
compute the corresponding sample values (J(K) (k) :4) 1. [n,]] as follows:
TD (k) (1))))
(3)
5=0
where
8 gd ((T)(((+))) uedd, wred t d) + q (ikk (2 d).
TD3S
(K) (1
Σ d=0
In order further enhance the overall stability of the method, each of the above estimates (w t) is mixed with the value obtained using the current continuous approximation of (2 (R) t) (note that, in general, will not have point-wise estimations of these values, as the trajectories from one iteration to the other are different). The mixture is computed as: (K) ,t.
j(m (I t) t (1 - a) (a (R) t) + J(k() (x
(4)
where 0 < a 1 is parameter
4.3 VALUE FUNCTION APPROXIMATION
Note: for the rest of this section, we will omit the iteration index k to clarify notation, as all related variables belong the current iteration.
After simulation, we have state-value training pairs available to approximate the value function each time slot > We will now describe the generic training procedure, leaving the details of the architecture for the next subsection.
The overall approximation these models is adapted using the value function sampled at all states visited at time during the simulation stage. We call this set of points, {zit r E [mr]), a constellation. The training data at time is the set of pairs (Irs, [x.,, t)).
given by n, parametric models, one for each [ny]. and each of
Given the training data, the approximation of the value function, and the time step t, Jxx t), E X, depends on a set of p parameters θ = (01e, On,t); )) we write this dependency explicitly as [zx f/Or). As usual, these parameters are adapted that the approximation error is minimized. This error given by:
The Σ [(a,,, - -(rret)) -((rrt))))
(5)
Besides minimizing 5 each of the Rp parameters of the approximation model is encouraged to follow a particular curve, which is itself parametric. Concretely, the n. values of the p-th parameter are modeled after a function g(t, ap).
of the architecture across time, (0p1, 0p...... 8p..)) On ~ ap),
where a, is the set of n. hyper-parameters associated to parameter p which are themselves adaptive. Correspondingly, an additional term is added to the cost function to enforce that relation:
Rp.
1
Bpt (Opt g(1, ap))
2
p=1.t11
Using different penalties Spe we can modulate the degree of fitness the guiding functions g(l, ay) depending on the time index t and/or the corresponding architecture parameter p. For example, we can ignore the imposition of a temporal pattern by setting the corresponding Stp 0 for particular p and all f.
Finally, the hyper-paramee parem-eareeterss ap. P- = 1,..., Up are regularized using an & (Ridge) penalty with parameter The overall cost function to be minimized during learning is:
I(Or, ap t [me].p (mp))
[ t -(.)))
1
Σ Spx 10pz ,(t,2a))) Σ 2
2

---

Under review as a conference paper at ICLR 2022
35
Classic TD(167)
25
15
0
20
40
60
80
100
Value approximation iteration
Figure 3: Difference between PDS for A 167 and the Classic algorithm as a function of the iteration. The PDS-based method surpasses the classic one after 30 iterations.
4.4 ARCHITECTURE
As mentioned in Section 3 electrical power systems exhibit extremely variable costs. Because of this, and despite the imposed regularity, the best architecture found to represent the value function at given state for this (simplified) STM model consists single hidden layer of neurons with tanh activations, and a linear output. Recall now that we have ensemble of ne 240 such networks, one per time step. The parameters of each of these networks are regularized by encouraging them to follow smooth deeendenndd function, described before. In the case of the STM, choose third order polynomial:
(t, 01ps a2p; 03p, asp) 01p + a2pt + azpt? + aspt".
For medium and long-term models, which are currently in development, we can use sinusoidal functions matching the seasonal patterns of both energy production and demand (day, week, year).
RESULTS
In this section we compare the total operation cost obtained with the new method and with the "Classic" Bellman iteration-based method currently in We evaluate the STM hourly models of both methods along the 10 day (240 hour) period that goes from 8/24/2021 9:00AM to 9/3/2021 9:00 AM.
We also set = 10 in 7 the discount factor = 1, the number of chronicles used in the simulation to n2,- 100, A 167 in (3) and
0.3
Figure 3 compares the expected 10 day cost-to-go value obtained with the proposed and the one obtained using the Classic one. The new PDS-based method is seen to surpass the performance of the classic method after 30 iterations (more details on the evaluation of these policies are given in Appendix F).
Figure 4 shows the evolution of the cost-to-go derivatives with respect to the state variables for given fixed state during the simulation horizon. can be seen, the parsimony imposed on the parameters of the set of networks forces an evolution with similar parsimony in the derivatives. From the point of view of the operation of the system, parsimony these derivatives is essential since these derivatives are transformed into operating instructions and it would not be feasible to abruptly change the dispatch the generation units.
In terms of execution time, 30 iterations using the PDS approximation requires just 12 minutes to complete, whereas the classic Bellman recursion requires 31 hours to reach the same performance over the same STM state space (see Appendix E for details the complexity of the CBS method for this scenario). It is important to remark that the optimal programming needs to recomputed on an hourly basis, thus rendering the Classic approach infeasible for this scenario (the Classic method
8

---

Under review as a conference paper at ICLR 2022
6.
-02
42
-44 we0ge
Palmar
Vol 5180, Grande
RRS Rin, Negm
4
RBB Rir, Uniguy
-10
$me-tap
Figure 4: Here we show the effect of g(t, a) on a few network weights (left), and the derivative of the cost-to-go function w.r.t the state variables (right), reflecting the parsimonious nature of the weights.
here was trained off-line). Thus, the new PDS-based method not only reduces times dramatically, but also allows to obtain a better operation by considering more complex state space.
CONCLUDING REMARKS
We have presented a new parametric value function approximation model capable computing the optimum programming of large and diverse electrical power systems. We have tested this new method on the short term model of the real Uruguayan system (the most challenging one), and confirmed to be superior to the one production using drastically less resources. Also, proposed parametric network series (PNS) provides both stable and interpretable models by capturing the time-dependent trends in the network parameters.
We are currently working on implementing the sliding window strategy for obtaining even faster updates, as described Section 1 and on deploying the new model on medium (MTM) and long term (LTM) scenarios.
REFERENCES
R. Bellman. Dynamic Programing. Princeton University Press, 1957. [
D. Bertsekas and J. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, 1996. 1.4
2.2
R. Chaer and P. Monzon. Stability conditions for stochastic dynamic optimizer for optimal dispatch policies in power systems with hydroelectrical generation. In 2008 IEEE/PES Transmission and Distribution Conference and Exposition: Latin America, pp. 1-5, 2008. doi: 10.1 10.1109/TDC-LA 4008.4641717
J.-J. Christophe, Decock, Liu, and O. Teytaud. Variance reduction in population- based opti- mization: Application to unit commitment. In Artificial Evolution (FA2015), Lyon, France, 2015.
E. Cornalino, P. Sobes, F. Palacios, M.C. Alvarez, Coppes, G. Casaravilla, and R. Chaer. Handling the intermittence of wind and solar energy resources, from planning operation. uruguay's 's cess. In 36th USAEE/IAEE North American Conference, Washington, DC, pp. 23-26, September 2018. 3.2
S. S. Du, S. M. Kakade, R. Wang, and L. F. Yang. Is good representation sufficient for sample ef- ficient reinforcement learning? In International Conference on Learning Representations (ICLR) 2020, 2020. 1.1

---

Under review as a conference paper at ICLR 2022
G. Flieller and R. Chaer. Introduction of ensemble based forecasts to the electricity dispatch simu- lator simsee. 2020 JEEE PES Transmission Distribution Conference and Exhibition Latin America (T&D LA), 2020. 3.1
Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin. Convolutional sequence sequence learning. In In International Conference on Machine Learning, pp. 1243-1252, 2017. 1.4
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9:1735-80. 12 1997. doi: 10.1162/neco. 1177..98.733. I.4
I. Jolliffe. Principal Component Analysis, chapter online, pp. - American Cancer Society,
2005. ISBN 9780470013199. doi: http:://ww..agg////0000004011112 bsa501. URL https: // /onlinelibrary.dle http://wwwwaapaac.ooo/////aaaaaa
V. I. Paulsen and M. Raghupathi. An Introduction to the Theory of Reproducing Kernel Hilbert Spaces. Cambridge University Press, 2016. ISBN 177140744 I.]
M. M.V.F Pereira and L.M. Pinto. Mtae stochastic optimization applied to energy planning. Mathematical Programming, 52:359-375, 1991. doi:
W. B. Powell. Approximate Dynamic Programming. Wiley, 2011.
A. R.Barron, Approximation and Estimations Bounds for Neural Networks. Machine Learning, pp- 115-133, 1994. 1.1
R. Sutton. TD models: Modeling the world at mixture of time scales. In Proc. the 12th Int. Conf. Machine Learning, pp. 531-539, 1995. 1
R. S. Sutton and G. Barto. Reinforcement Learning: An Introduction. MIT Press, Feb, 2018.
1.4
2.2 4.2
X. Xu, Lian, L. Zuo, and H. He. Kernel-based approximate dynamic programming for real-time online learning control: An experimental study. IEEE Trans. on Control Systems Technology, 22 (I), Jan. 2014. L.
Zheng, S. Ramasinghe, and Lucey. Rethinking Positional Encoding. preprint, 2021. URL http:/ //tp///axvvvv//a////a//eee/aawva wtt.//gg///bbaooo/.oooo 2207 02561
10

---

Under review as a conference paper at ICLR 2022
A
VALUE ITERATION LEARNING LOOP DIAGRAM
diagram-box of overall algorithm is shown Figure 5
Simulate a set of chronicles using
(.,
2a
Compute a new (.,) estimation from the collected information using compound time difference technique
2b
Sample: J(a (.,
at the visited states and apply the learning rate combining with the previos computed values:
(x
(xx, (k) ()==(1=))]]) (k) (x .t),tJ+JJ (x (k)
(k)
3
Train the network using (xx t) (K) t
and get the new (1, (1+1))
Figure 5: Value Iteration Learning loop.

---

Under review as a conference paper at ICLR 2022
B VALUE FUNCTION IN ENERGY DISPATCH PROGRAMMING
In the generation system we have several time horizons to solve. The value function weights different operation costs depending on the horizons. In the medium term (several months) the value function assigns value to the water stored in the lakes with the highest capacity. This is used to trigger purchases of ships with fuel, which must be made 90 days in advance so that the fuel is available when required.
On a weekly basis, the operating policy takes into considcration the information of the rain, wind, solar forecasts, and the demand. As result, the distributions of the exportable energy blocks (system surplus) and their recovery (minimum price to receive) are available one week in advance.
For daily simulations, the value function captures the effect of the forecasts of availability of water, solar and wind resources and allows determining the order of dispatch of the resources, the instruc- tions given the operators regarding the recovery of the dammed water in each of the lakes (derived from the value function with respect to using a unit of energy stored in each lake). This valuation allows the operators. in real time, to make decisions while maintaining a dispatch of minimum cost. The same short-term simulations with hourly detail, allow visualize the probabilities of dispatching thermal power plants and give advance notice to the operators of said units so that they are ready to dispatch when requested.
maintain the power balance of the system
C PENALTIES ON THE VIOLATION OF RESTRICTIONS
In the representation of the system used, if a restriction included through a penalty, the value of the penalty is calculated by evaluating the real effect on the system when said restriction is violated. Therefore, the value function includes both the costs associated with the use of fuels and imports as well as the implicit costs associated with the penalties the represented restrictions. As example, it is a common practice to represent the power balance constraint at each node of the system as the sum of the generator powers minus the sum of the demands equal zero, but this constraint is represented with penalties whose values are estimated as the cost of the country's 's economy for not supplying the demand.
D THE NEED TO RE-TRAIN THE MODEL AT EACH TIME STEP
Modern electric systems integrate several wind and solar energy sources. These sources have impor- tant fluctuations in terms of hours. It is the introduction of these short-term variabilities that makes the filtering elements (reservoirs, battery banks, etc.) of the system that were not important before, now do. The need to operate these new filtering elements, using the information from the forecasts, leads to an increase in the dimension of the state of the system to be considered. This change has been accentuated in the last 5 years and will continue in the same direction in the coming years as worldwide trend. Theoretically, if the representation of the state of the system would not be necessary to fit a model for each time step. But the representation of the state is never complete. It should be remembered that the structure of the system is variable over time. In the short term, the forecast information is represented based on time series of biases and attenuators that shift the distributions that model the associated stochastic processes. This leads the same state (represented) of the system, but in different time steps, the value function is different. In the long term, the structure of the system changes as the demand grows, the power plants age and are withdrawn and new generators enter. On an annual scale, the different stations impose shifts in the distributions associated with wind, solar and hydraulic energy and on the Demand of the system.
complete, it
E COMPUTATIONAL COST OF THE CLASSIC BELLMAN RECURSION
Table 1 shows the nine state variables considered in the model along with the corresponding number of discretized values. Solving a single time step in Classic Bellman's 's Recursion (CBR) requires one to compute the value function for every combination of the discrete values of all nine variables. In our case, there are 10 x 6 x 6 x x x x 5 x 4 x 4 = 1 296000 different states. This, in tur,, has to be carried out for each time step. In our case have 240 such stems, for total of
12

---

Under review as a conference paper at ICLR 2022
311040000 value function estimations. On the other hand, in the proposed method, simulations of 100 chronicles are being carried out in each simulation cycle. Each chronicle implies computing the value at just the 240 visited states and therefore each simulation eycle implies solving the dispatch in 100 x 240 = 24000 states. The computational effort of the classical Bellman resolution is 12960 times greater than that of information collection cycle of the proposed algorithm. For the example presented, the proposed algorithm converged to a slightly better Operation Policy than the classical one in just 30 iterations of the learning loop.
then 31 10400000/2444
F DETAILS IN THE EVALUATION OF THE POLICIES
First 100 iterations of the learning loop were performed (see Figure 5 with block diagram of the algorithm). Then, each of these iterations, the functions of value (., were taken and the operation of the system with these functions was simulated for set of 1000 new realizations of the stochastic processes (generated with new random seeds not used for training previous). Finally with these simulations, the estimate of the expected value of the future operation with each (., t) was obtained. And with the same random seeds, it was simulated using the value function obtained with the classical Bellman algorithm (currently in use for dispatch resolution). The obtained estimations are shown in Figure
G ON THE CHOICE OF THE PARSIMONY CRITERION
There are two aspects for which the imposition of temporal parsimony seems important to us. The first is that in the actual operation there is a certain temporary behavior that must be respected. They are restrictions imposed by the operation. As example, it would not be permissible to have operating instructions that imply dispatching a combined cycle power plant at one hour and shutting it down the next. These operating instructions arise directly from the derivatives of the value function with respect to the state variables and therefore, by giving temporary parsimony to the network parameters, indirectly the type of parsimony is being imposed the operation instructions.
The other aspect, less important, is that it is way of introducing expert knowledge, such as the trends and seasonality to be expected in the variables of the system (demand growth, rainy seasons, windy seasons, etc.). It would be possible to let the network learn these relationships from simulations, but remember that we are fighting the Bellman curse and that we intend to focus the neural network's 's representational capacity and computational effort efficiently. The implementation carried out allows to experience the effect of considering or not the temporal parsimony increasing or reducing the value of the penalties 8pt.
13


Reviews:
[{"review_id": "ywmbUXzPh-g", "reviewer": "Reviewer_dQUM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n* The problem addressed in the paper is well-motivated from the perspective of both theory and practice, with direct application to a real-world system.\n* The high-level approach and assumptions are clearly described.\n* The performance of the method seems very competitive with respect to the existing baseline, though more details on the evaluation should be provided (see below). \n\nWeaknesses/questions:\n* In general, I think the paper lacks some concreteness on the exact power systems setting, which make certain details of the paper harder to grasp. For instance, how is the value function actually used during planning? Do the cost functions represent only power generation constraints, or also e.g. constraint violations?\n* I was not able to understand why a different model would need to be fit to every time step, or why this would make sense in the power systems setting. Is the idea that T=24 hours and t=1 hour, and since load values follow a roughly daily fluctuation, similar networks are used at times of similar load? Again, without more concreteness on the specific setting being studied, this was hard to grasp.\n* Some critical details were missing, which made it hard to grasp the quality or soundess of the underlying method. For instance:\n  * Section 4.1: More details should be given on the common random numbers technique (either in the main body or in an appendix), as the way the ensembles/chronicles are generated presumably has large implications for performance.\n  * Section 4.3: I am not fully sure how the \"anchor values\" were employed.\n  * Section 4.4: Why does this particular choice of $g$ make sense in the power systems context?\n* Section 4.2: In the update equation, should the terms with/without tildes be switched? I am not sure the current update as written mathematically follows clearly from the verbal description/I am not sure where all the terms are coming from.\n* Isn't doing online forward simulation at each step potentially expensive? \n* In general, it was hard to fully understand how the different steps presented in different sections of the paper fit together. An algorithm box would greatly enhance clarity.\n* Additional details need to be provided in the experiments, in order to fully understand the efficacy of the proposed method. For instance:\n  * Section 5: Some more details of the classic Bellman iteration-based method that is currently used could be provided. E.g., why is it currently so expensive?\n  * Figure 3: More details on how the evaluation was conducted should be provided -- e.g. details of how $w_t$'s were generated in practice may have a large bearing on the results.\n* Figure 4: It could be more thoroughly explained why parsimony between these two quantities is necessary/useful. Right now, the explanation given seems somewhat hand-wavy. \n\nMinor points (not affecting my score):\n* Page 2, Section 1.2: \"planification\" -> \"planning\"\n* Section 4.2: Why was $\\alpha = 0.3$ chosen?", "summary_of_the_paper": "This paper provides a deep learning-based method to estimate value functions for multi-step dispatch on a power grid. This method entails training a different neural network to approximate the value function for each time step within a fixed horizon; supervision is provided via point-wise estimates of the states visited within rolled-out trajectories, as estimated via compound TD estimation. The method is demonstrated to have comparable performance to the \"classic\" Bellman iteration-based method that is currently used within the SimSEE dispatch system (currently employed by the Uruguayan electricity system operator), while running significantly (~150x) faster. ", "summary_of_the_review": "The problem studied in this paper is important and well-motivated, and at a high level, the proposed approach seems reasonable and to work well. However, descriptions of several critical components of the theory and evaluation are either vague or not provided, making it difficult to truly evaluate the soundness and performance of the method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ypVg6jV3tOA", "reviewer": "Reviewer_QHRg", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I would say the only strength of this paper may be the application results yet it is difficult to be sure given that the method being compared to is not expected to perform well on a large and complex system. It may be currently in use as the authors are saying yet that should qualify their approach as a good option for whoever is interested in using it for the Uruguayan system but not as a scientific argument, in my opinion. \n\nAs for weaknesses, the paper seem to be a concatenation of well known and simple techniques with some specifics that are not justified. For example, why is the third order polynomial chosen in Section 4.4 and why are betas as penalties chosen as in Section 5. There are no numerical characterizations of the performance of the algorithm, no bounds on the errors, no proofs of convergence, etc. In general, the paper provides no technical justifications for the claims. \n", "summary_of_the_paper": "The paper provides an approximation of the cost-to-go for a particular system. The particular system is the Uruguayan system. The results are compared to the approximate dynamic programming as provided in some classical books that are cited in the paper. ", "summary_of_the_review": "The paper does not provide technical justifications for the claims. The approach is compared to a classical technique that is not expected to be very efficient when applied to a large and complex system, in general. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "hEN4DNe_kRJ", "reviewer": "Reviewer_wkjh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n1. The background knowledge is well written.\n2. The presentation of the main contribution is clear enough.\n3. A good attempt on combining data-driven ADP on the real-world problem.\n4. Although the authors seem not a researcher in the ML community, the fundamental knowledge of ML described in the paper is almost correct and the logic is very clear.\n\n## Weaknesses\n1. The proposed method on representing the value function is actually not novel enough in the community of ML. It can be seen as a special case of using the hyper-networks to model the value function [1]. Nevertheless, the attempt to create innovations in the traditional fields should be encouraged.\n2. Although the motivation of proposing time-dependent value function is correct, the representation and resolutions of **time steps** should be discussed. The inefficient representation of time steps could lead to the learning problem such as gradient explosion. Although in the paper the authors use hour as the time resolution, I doubt whether this can be generalized to all scenarios since it depends on the initial time points. If this is not generalizable, then the modelling of time dependency is less effcient than the time-independent but with extra context representing the state. Can the authors give more discussion on this point?\n3. Can the authors give more details about the constructions of $g(\\cdot)$ and $\\beta\\_{pt}$? In other words, why and how are the constructions yielded?\n4. The authors lack many works on applying data-driven method to power system operations from IEEE transactions on smart grid and IEEE transactions on power systems. The comparisons with the related works in the same field is important. Can the authors give more related works in the discussion stage?\n\n\n## References\n[1] Ha, David, Andrew Dai, and Quoc V. Le. \"Hypernetworks.\" arXiv preprint arXiv:1609.09106 (2016).\n", "summary_of_the_paper": "This paper proposed using time-dependent parameters to model the value function for evaluating the cost-to-go function in power systems. This work can be regarded as an extension from the traditional adaptive dynamic programming (ADP) with the study on how to represent the value function that fits the uncertainty of the dynamics of power systems induced by the installation decentralized energy resources (DERs). This work shows the performance of the proposed method on a realistic simulation. The background knowledge is well written and clear.", "summary_of_the_review": "In summary, this work is a good attempt on extending the traditional method with ML techniques. However, due to multiple concerns I have (see weaknesses), I recommend reject. On the other hand, if the authors can improve the work that can satisfy the standard of ICLR, the community should welcome such works dealing with the realistic problems.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews are of good quality. The responses by the authors are commendable, but ICLR is selective and reviewers continue to feel that important choices in the research are not sufficiently clear and fully justified.
Decision: Reject