Generalization of backpropagation with application to a recurrent gas market model

-Backpropagation is often viewed as a method for adapting artificial neural networks to classify patterns. Based on parts of the book by Rumelhart and colleagues, many authors equate backpropagation with the generalized delta rule applied to fully-connected feedforward networks. This paper will summarize a more general Jbrmulation of backpropagation, developed in 1974, which does more justice to the roots of the method in numerical analysis and statistics, and also does more justice to creative approaches expressed by neural modelers in the past year or two. It will discuss applications of backpropagation to forecasting over time (where errors have been halved by using methods other than least squares), to optimization, to sensitivity analysis, and to brain research. This paper will go on to derive a generalization of backpropagation to recurrent systems (which input their own output), such as hybrids of perceptron-style networks and Grossberg/HopfieM networks. Unlike the proposal of Rumelhart, Hinton, and Williams, this generalization does not require the storage of intermediate iterations to deal with continuous recurrence. This generalization was applied in 1981 to a model of natural gas markets, where it located sources of forecast uncertainty related to the use o f least squares to estimate the model parameters in the first place. Keywords--Backpropagation, Recurrent, Continuous time, Reinforcement learning, Energy models, Prediction, Modelling, Cerebral cortex. 1. I N T R O D U C T I O N Backpropagation, as formulated by Rumelhart, Hinton, and Williams (1986) with acknowledgement of the prior work by David Parker (1985), may well be the most widely-used method to adapt artificial neural networks, for use in pattern classification. Nevertheless, the limitations of that formulation have been severely criticized by neuropsychologists and by classical computer scientists. The neuropsychologists have argued that simple feedforward networks cannot do justice to the structure and power of the brain. Neuropsychologists and computer scientists have argued that complex, interesting problems tend to require iterative procedures ( or networks) for their solution. Many other criticisms have been raised, which merit serious attention. Section 2 of this paper will review a different formulation of backpropagation, developed in the period The views expressed in this paper are those of the author, and do not necessarily reflect those of any component of the Federal governmeat. Requests for reprints should be sent to Paul J. Werbos, Neuroengineering Program, Room 1134, NSF, 1800 G Street N.W., Washington, DC 20550. 339 between 1968 and 1974, which can overcome many of these difficulties. This formulation deals with the general case of nonlinear systems of equations. It lacks the concrete, specialized appeal of Rumelhart 's discussion, but it can apply to neural networks, econometric models, and other systems as special cases. Applications to prediction, optimization and sensitivity analysis become possible; as an example, this paper will discuss an application to the sensitivity analysis of a natural gas market model developed by the Depar tment of Energy. Werbos (1987a) discussed at length a research strategy for brain research and factory automation based upon this formulation. Section 3 of this paper will show how derivatives may also be propagated through recurrent networks (such as those discussed by Grossberg, 1976 and Hopfield and Tank, 1986) without the expensive storage of information for each iteration (as required by the approach of Rumelhart et al., 1986). Our approach will require storage, however, to handle true external time lags; the significance of this will be discussed, along with ways to implement this storage and issues related to real-time adaptation. When external time lags are totally absent, our method is closely related to the 340 1~ .l. 14"~,rbo; method of Almeida (1987), though slightly more general. Finally, Section 4 will display a practical application of the methods given m Section 3 an analysis of the properties of a natural gas market model, actually used by the Depar tment of Energy several years ago. The conclusions of this analysis were double-checked by explicit numerical perturbations of the model. This analysis provided an insight into the limitations of the model, which are related to certain limitations of multiple regression, the method used to estimate (adapt) the model in the first place. Multiple regression is closely related to the generalized delta rule for network adaptation; however, alternative estimation (adaptation) rules exist which have overcome these limitations m simulation studies and in several practical examples (Werbos, 1974. 1983a, 1988a; Werbos &Titus. t978). Those alternative rules are consistent with the general framework proposed here. 2. G E N E R A L F R A M E W O R K : BACKGROUND, T E R M I N O L O G Y , AND APPLICATIONS Rumelhart, Hinton, and Williams (RHW) Debates about backpropagation have been confused. in part, by different definitions of the word. The index to Rumelhar t et al. (1986) defines the word backpropagation by pointing to three pages of text which discuss the generalized delta rule. The generalized delta rule. in turn, is defined as a set of three steps to be applied to feedforward networks. ( R H W also discuss recurrent networks, but that extension will not be discussed until Section 3. in order to simplify things here.) R H W specify feedforward networks as: