Optimal neurocontrol: practical benefits, new results and biological evidence

From late 1993 to March 1995, five groups have presented -for the first time -serious engineering accomplishments in simulations of control designs which meet four criteria for what is a "brain-like" intelligent system. In addition to demonstrated, intelligible engineering functionality, a "brain-like'' system should contain at least three major general-purpose adaptive components: (1) an Action or Motor system, capable of outputting effective control signals to the plant or environment; (2) an "Emotional" or "Evaluation" system or "Critic," used to assess the long-term costs and benefits of near-term alternative outcomes; (3) an "Expectations" or "System Identification" component, which serves as a model or emulator of the external environment or of the plant to be controlled. This paper will: (1) summarize the practical advantages of moving from more classical designs to more brain-like designs, in today's engineering applications; (2) summarize the recent accomplishments of the 5 groups mentioned above; (3) argue that these designs (with some enhancement) do constitute a worthwhile theory of how intelligence works in the brain, generating testable predictions and proposed experiments; (4) scope out some key areas of research needed to implement the required enhancements in engineering. 1 Background and Introduction From the Abstract, you can see that this paper will provide a highly condensed summary of a huge amount of material. Except for some of the recent material on brain linkages, the details are described at a more reasonable pace -with more complete logic -in other sources, which I will cite and stress. Back in 1981[1] and 1987[2], I published a simplified "straw man" model of the brain as an intelligent learning system, based on a form of reinforcement learning or optimization, with backpropagation (but not supervised learning! ) as a pervasive tool down in the lowest-level subsystems. This model was oversimplified (for reasons discussed at length in [2]), but it did promise to reproduce four basic attributes required in any credible model of intelligence in the brain: (1) It contained an "emotional system," a subsystem to estimate the values of observed objects or variables to the organism (Le., secondary reinforcement); (2) It contained an "expectations" or "prediction" subsystem, which -as Grossberg has often stressed -is essential to replicating basic experiments in "classical conditioning"; (3) It contained an "action" or "motor" system, a system to calculate the movements of muscles or other actuators required in order to implement the values calculated by the value system; (4) It made sense, purely as an engineering system, designed to solve very difficult, generic problems in optimal control based on learning. In retrospect, as I look back on that model and the other work which has appeared in the last few years, I now have a higher opinion of the value of that "simple" model. So far as I know, it is still the only model which anyone has ever formulated which meets all four basic conditions above. There are many biologically-inspired models which meet the first three conditions, or which come a little closer to a fifth desirable condition (to match all the specific cell types in the brain in a comprehensive, testable manner), but none of them really address the phenomenon of "intelligence" in a generic, functional way; typicallyhey provide some form of associative memory, some narrow task-specific competence or the ability to replicate some simple behavioral experiments (as in the work of Klopf). *The views herein are those of the author, not those of NSF, but are nevertheless in the government public domain. This paper was previously published in Proc. World Congress on Neural Networks, Erlbaum, July 1995 Since 1981, my primary goal here has been to enhance this "simple" model, to improve both its engineering capabilities and its detailed fit to the brain, in part by linking up with "bottoms up" biological information and models; however, this is clearly too large a task for one person working alone on personal time. The most serious difficulty here was that the 1981 model -however simplified it may be, compared with the brain -was still far more advanced than the rest of the community was ready to work with; thus to mobilize more talent, it was crucial to work on the engineeringhathematics side, to help it catch up -in real engineering practice -to the general concepts in the 1981 work. This has been a slow step-by-step process, but we finally do appear to be getting there, based on the engineering successes of this past year. Meanwhile, however, with guidance from people like Pribram, Houk, Pellionisz and Mishkin, I have gone quite a bit beyond the 1981 model in understanding the more complex kind of control system we see in the brain and in the mind. These strategic considerations lead directly to the four more focused topics listed in the Abstract, which I will address one at a time. 2 Practical Advantages of Brain-Like Control Increasingly, neurocontrol seems to be splitting into two groups -the "haves" and the "have-nots. " The "haves" -like Feldkamp and Prokhorov in this session, among others -have gone far beyond the early truck-backerupper[3] kind of design. They understood the truck-backerupper technology long ago, and have moved on to work which is much more sophisticated, more brain-like, and more useful in important real-world applications. Unfortunately, the "have-nots" have generally sunk to levels well below the truck-backer-upper, and have not even understood the key concepts or techniques [4] needed to ISBN# 0-7803-2636-9 580 achieve that level of capability. This seems to be mainly a communications problem, due to the great complexity of design options, societies, and personal networks. To reduce this problem, ideally, I should spell out all the specialized designs in use mday, their weaknesses, the specific benefits of moving just one step "up the ladder" to more brain-like designs, and the specific techniques and examples which make this now easier to do. It has taken me many years to develop this complex map[5,6]; here I will only give some highlights. Here I will only discuss neurocontrol proper (where the control signals themselves come from a neural net). Every useful example of neurocontrol to date rests on a generic capability to perform one or more of three basic tasks: (1) cloning of a human or other expert; (2) tracking a set-point or desired reference trajectory; (3) dynamic optimization, maximization of a performance measure over time, accounting for the impact of present actions on performance many periods into the future. Cloning is still quite useful as a way to initialize neural nets. (It is very popular in adaptive fuzzy control, but losing popularity in neurocontrol.) In practical applications, tracking error or performance is the real objective; it is better to use cloning designs as a starting point, and then adapt them further to do better, using tracking or optimization approaches. Often there are better ways to do the initialization. Tracking is now mainly done by using neural nets instead of matrices in classical model-based adaptive control designs. Narendra of Yale -who pioneered stability theorems for classical adaptive control -has proven similar theorems for the neural versions, and many others have followed him. (He has now begun -step by step, carefully -to move on to more brain-like approaches.) In essence, these designs use some form of backpropagation to train an Action network to output those actions, u(t), which maximize a measure of tracking error at time t+l. These designs have 3 major limitations. First, the requirements for stability are very restrictive, both for classical and for neural designs. (Actual stability, not theorems, is a major concern at decision-making levels in industry.) Minimizing error at time t+l will often cause overshoots or other problems, which result in more error at later times, and outright instability and explosions. (E.g., see the bioreactor test problem in [3].) Thus chemical plants typically will not consider classical or neural adaptive control; instead, they often use "Model-Predictive Control" (MPC) [7, ch.101, explicitly optimizing over many time periods into the future, accounting for process constraints as well. (Experts in adaptive control may note that stabilization is often possible by a very clever (very difficult) choice of error measure at time t+l : however, some forms of optimal neurocontrol may be interpreted as constructive methods to find such an error measure, which is normally quite difficult. [5]. ) A second problem is that generic real-time learning is a slow way to adapt to changes in familiar parameters like mass and friction: it results in unnecessarily long transient responses and unnecessarily weak performance during transients. A better approach is to "learn offline to be adaptive online" (as I proposed in [8]), so as to tune the adaptation parameters themselves, in effect, to the specific parameters. This requires the use of optimization over time (which could be done in real-time as well) applied to a timelagged recurrent network used as an Action network, exactly as described in the plenary talk by Feldkamp of Ford at this conference. As Feldkamp has stressed, it is critical to know how to calculate the required derivatives correctly here, and the literature is now pervaded by inaccurate shortcuts and unnecessarily expensive methods for doing this. (See [4] for a tutorial and [7,ch.10] for more advanced methods for exact derivative calculation.) Third, if we wish to directly optimize performance measures like fuel consumption, mass ratios and pollution over time in a highly dynamic system, we must move on to the designs for optimization over time. (Some people try to optimize performance by use of hand-tweaking here, or the equivalent, but this is not as effective as an aut