Abstract:
The present disclosure provides novel techniques for defining empirical models having control, prediction, and optimization modalities. The empirical models may include neural networks and support vector machines. The empirical models may include asymptotic analysis as part of the model definition as allow the models to achieve enhanced results, including enhanced high-order behaviors. The high-order behaviors may exhibit gains that are non-zero trending, which may be useful for controller modalities.

Description:
BACKGROUND 
     The present disclosure relates generally to control systems, predictive systems, and optimization systems that utilize empirical models. 
     Empirical models are typically used in the modeling of complex processes, including control systems, predictive systems and optimization systems. During empirical modeling, historical data may be used as training data that aids in defining the empirical model. The trained empirical model, for example, embodied in a controller, may then present relatively accurate results within the range of the training data. However, when the trained model encounters inputs outside of the training range, the extrapolated results may not be as accurate. Further, other properties of the training data, such as quality of the data fit, may not be sufficient to render the empirical model useful. 
     BRIEF DESCRIPTION 
     The present disclosure provides novel techniques for defining controllers, predictive systems, and/or optimization systems by utilizing empirical models that are capable of incorporating desired extrapolation properties, such as a candidate basis/kernel function φ b (•), as factors used to determine the structure of the empirical model. Once the model has been defined, the model may then be utilized in controller embodiments, model predictive control embodiments, environmental management embodiments, production performance management embodiments, plant operations optimization embodiments, industrial scheduling systems embodiments, and so forth. 
     An empirical model may be first defined using the following general equation: 
                     f   ⁡     (   x   )       =       ∑   b     N   B       ⁢           ⁢       φ   b     ⁡     (       w   b     ,   x     )                 (   1   )               
where xε             N     u    is the N u -dimensional input vector, ƒ(•):           N     u   →           N     y    is a linear or nonlinear mapping from the N u -dimensional input space to N y -dimensional output space, w b  is the parameters of the basis/kernel function φ b (•) that are determined in the course of the modeling process, and N B  is the number of the basis/kernel functions used for the approximation. Accordingly, the model is capable of utilizing a set of inputs, processing the inputs through a set of basis/kernel functions φ b (•), so as to arrive at a result. The model may be incorporated into various systems, such as control systems, predictive systems, optimization systems, or a combination thereof. The model results may thus be used, for example, as controller setpoints, predictive finish times of industrial processes, optimization of production line quality, and so forth.

     In one example, a method of empirical modeling includes determining the model parameters w b  given the obtainable data. The empirical model may then be further defined by the use of training data. That is, the obtainable data may be split into a training data subset and a test data subset. A training process may then be employed where the training data is presented as inputs to the model, and the model output compared to desired results. Any errors in the output may then be used to adjust the parameters (e.g., weights, biases, or tuning parameters) of the model until the model generates valid results within a certain margin of error. The test dataset may then be used as inputs to verify and validate the model. For example, the test dataset may be used as inputs into the model and the results then verified as suitable for use in production environment. Based on the quality and on the range of the training data, such an approach may be useful. However, in many instances it is difficult for the model to extrapolate outside of the training data. Accordingly, the extrapolation property of the model may be as important as the model&#39;s accuracy over the training dataset. Indeed, even if the resulting model exhibits a good quality of fit (i.e., high fidelity) over the training data set, this model property by itself may not be sufficient to render the model useful. 
     For example, when the model is used in industrial control, the gain of the model (e.g., the first order derivative of the output with respect to the input) may trend towards zero, especially in extrapolation regions. Zero model gains may result in infinite controller gains, which may not be useful in industrial applications. The extrapolation regions of the empirical model may also be susceptible to gain inversion. Model gain inversion may result, for example, in valves opening when they should be closing (and vice versa). Accordingly, in certain embodiments described in more detail herein, the empirical modeling process is altered by making the analysis of the extrapolation properties of the empirical model an integral part of the selection of the basis/kernel function φ b (•). That is, the basis/kernel function φ b (•) will be added to the model only if the asymptomatic behavior of the approximate function can emulate the known and/or the desired asymptomatic behavior of the actual system that is described by ƒ(x). More specifically, while the detailed behavior of the system ƒ(x) may not be known, the information on asymptotic behavior of the system may be available. By incorporating the system&#39;s asymptotic information, the resultant empirical model may be capable of a superior extrapolation behavior. Some example asymptotic behaviors may be as follows: 
                         lim     x   →   ∞       ⁢     f   ⁡     (   x   )         =       l   ∞     ⁡     (   x   )         ;           (   2   )                     lim     x   →     -   ∞         ⁢     f   ⁡     (   x   )         =       l     -   ∞       ⁡     (   x   )         ;           (   3   )                     lim     x   →   ∞       ⁢           δ   2     ⁢     f   l         δ   ⁢           ⁢     x   i     ⁢   δ   ⁢           ⁢     x   j         ⁢     (   x   )         =       l     ij   ,   l     2     ⁡     (   x   )         ;           (   4   )                   lim     x   →   ∞       ⁢         δ   ⁢           ⁢     f   j         δ   ⁢           ⁢     x   i         ⁢     (   x   )         =         l   ij   1     ⁡     (   x   )       .             (   5   )               
Where x→∞ is used as a shorthand notation to denote various ways by which the asymptotic behavior of the system may be manifested. For example, a trend towards infinity may be described as x i →∞ for all iε{1, . . . , N u }. Another example may be when all of the inputs of the input vector (or set) x are constant except for the j-th component such that x j →∞. For the examples cited in equations 2-5 above, the asymptotic behavior could assume the following forms. l ∞ (x) is the asymptotic behavior of f(x) as all components of the input vector x grow towards +∞. One example includes a constant, l ∞ (x)=C. A second example includes a linear function of inputs l ∞ (x)=a·x. In these examples, C and a are known constants. l −∞ (x) is the asymptotic behavior of f(x) as all elements of x go to −∞. l ij   1 (x) is the asymptotic behavior of the gain from the i-th input to the j-th output as the i-th component of the input vector x grows towards +∞ while all other components of the input vector are kept constant. l ij,l   2 (x) is the asymptotic behavior of the partial derivative of
 
               δ   ⁢           ⁢     f   l         δ   ⁢           ⁢     x   j             
with respect to the i-th input x i , as x i  and x j  go to ∞. It is to be understood that in other examples, other asymptotic behaviors could be used. Indeed, any number and type of asymptotic behaviors are capable of being used with the techniques described in further detail below.
 
    
    
     
       DRAWINGS 
       These and other features, aspects, and advantages of the present invention will become better understood when the following detailed description is read with reference to the accompanying drawings in which like characters represent like parts throughout the drawings, wherein: 
         FIG. 1  is a block diagram a plant control, prediction and optimization system, in accordance with one aspect of the disclosure; 
         FIG. 2  is a block diagram of an embodiment of an empirical model, in accordance with one aspect of the disclosure; 
         FIG. 3  is a flow diagram of an embodiment of logic, in accordance with one aspect of the disclosure; 
         FIG. 4  is a diagram of an exemplary embodiment of a neural network, in accordance with one aspect of the disclosure; 
         FIG. 5  includes graph diagrams of test and training results of a neural network, in accordance with one aspect of the disclosure; 
         FIG. 6  includes graph diagram of test and training results of the neural network of  FIG. 3 , in accordance with one aspect of the disclosure; and 
         FIG. 7  is a block diagram of an exemplary support vector machine, in accordance with one aspect of the disclosure. 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  illustrates an embodiment of an industrial controller  10  that is capable of controlling embodiments of a system  12 . The system  12  may include a plurality of industrial system embodiments such as a manufacturing plant, an oil refinery, a chemical plant, a power generation facility, and others. The industrial controller  10  may include embodiments of an empirical model  14  capable of managing all aspects of the system  12 , including control, prediction, and/or optimization. Indeed, the model  14  may be capable of control, prediction, and optimization of the system  12 . For example, the model  14  may be capable of process control, quality control, energy use optimization (e.g., electricity use optimization, fuel use optimization), product mix management, financial optimization, and so forth. 
     The system  12  receives a set of inputs  16  and produces a set of outputs  18 . The inputs  16  may include process inputs (e.g., types of materiel, quantities of materiel, product delivery schedules), financial inputs (e.g., accounting data, economic data), regulatory inputs (e.g., emission constraints, regulatory rules), and so forth. The outputs  18  may include manufactured products, refined products (chemicals, gasoline, coal), power (e.g., electricity), and so forth. Indeed, the system  12  is capable of receiving any number of inputs  16  and using the inputs  16  to produce a set of outputs  18 . 
     In certain embodiments, the industrial controller  10  includes a sensor interface circuit  20 , a actuator interface circuit  22 , and a monitor/control circuit  24 . The sensor interface circuit  20  is capable of communicating with a plurality of sensors  26 . The sensors  26  may be capable of sensing a number of inputs  16  as well as signals internal to the system  12 , such as temperature measurements, liquid levels, chemical composition, flow measurements, pressure measurements, electrical measurements, and so forth. Accordingly the sensors  26  may include temperature sensors, optical sensors, chemical sensors, pressure sensors, flow volume sensors, valve position sensors, speed sensors, vibration sensors, voltage sensors, amperage sensors, and so forth. Indeed, any type of sensing device may be used. The sensor interface circuit  20  may interface with the monitor/control circuit  24  so as to communicate measurements and other data based on the inputs  16  and on signals from the sensors  26 . The monitor/control circuit  24  may then transform the inputted data into control signals suitable for use by the actuator interface circuit  22 . The actuator interface circuit  22  may utilize a plurality of actuators  28  to perform any number of actions such as adjusting a valve position, moving a conveyor belt, controlling a robotic device, and so forth. 
     An operator interface  30  is communicatively connected with the monitor/control circuit  24  and used to aid an operator in interfacing with the monitor/control circuit  24 . The operator interface  30  may be capable of programming the monitor/control circuit  24 , modifying data, modifying the model  14 , and so forth. In certain embodiments, the operator interface  30  may remotely located and may communicate through a network such as a local area network (LAN), the internet, or a wide area network (WAN). The operator interface  30  may also be capable of interacting with the model  14  in order to modify certain aspects of the model  14  as described in more detail below with respect to  FIG. 2 . 
       FIG. 2  is a block diagram of the empirical model  14  capable of control, prediction, and/or optimization modalities. As mentioned above, the empirical model  14  is capable of modeling almost all aspects the systems  12 . Accordingly, the model  14  includes a plurality of basis/kernel equations φ b (w b ,x)  32  that may be used to describe a modeled system  34 . Indeed, by using techniques described herein, including the equations φ b (w b ,x)  32 , the modeled system  34  may be suitable for modeling any number of systems  10 . In certain embodiments, such as neural network embodiments, the equation φ b (w b ,x)  32  may be used as a basis/kernel equation as described with more detail below with respect to  FIG. 4 . In other embodiments, such as support vector machine embodiments, the equation φ b (w b ,x)  32  may be used as a kernel/basis function as described with more detail below with respect to  FIG. 7 . More generally, the equation φ b (w b ,x)  32  may be used to express the empirical model  14  in the form 
                 f   ⁡     (   x   )       =       ∑   b     N   B       ⁢           ⁢       φ   b     ⁡     (       w   b     ,   x     )           ,         
as mentioned above, where N B  is the number of the basis/kernel functions used for the approximation of the modeled system  34 . A set of inputs x  36  where xε             N     u    is the N u -dimensional input vector, may be used as inputs into the modeled system  34 . The modeled system  34  may then generate a plurality of outputs y  38  where yε           N     z    is the N y -dimensional output space.

     The equation φ b (w b ,x)  32  may include parameters w b  that are determined in the course of the modeling process. For example, parameters w b  may be incorporated corresponding to weights of links between nodes, biases of nodes, tuning factors, and so forth. In certain embodiments, a method for modeling includes determining the model parameters w b  given the available data and any knowledge about the modeled system  34 . Indeed, such embodiments are capable of making the analysis of the extrapolation properties of the empirical model  14  an integral part of the selection of the basis/kernel function φ b (•). In these embodiments, an asymptotic behavior of the system  10  may be known or desired. That is, the detailed behavior of the system may not be known, but the more general asymptotic behavior may be known, or if not known, a certain asymptotic behavior may be desired. Accordingly, a basis/kernel function φ b (•) may be selected for use as equation φ b (w b ,x)  32  such that the asymptotic behavior of the modeled system  34  matches that of the actual system  10 . Some example asymptotic behaviors and corresponding exemplary basis/kernel functions are described below. 
     For constant asymptotic behavior where φ b (x)˜c:
 
φ b ( x )=1/(1+exp(− x ));  (6)
 
φ b ( x )=tan  h ( x );  (7)
 
φ b ( x )= x /√{square root over ((1 +x   2 ))};  (8)
 
φ b ( x )= x /(1 +|x |).  (9)
 
     For linear asymptotic behavior where φ b (x)˜x:
 
φ b ( x )=log(1+exp( x )).  (10)
 
     For logarithmic asymptotic behavior where φ b (x)˜log(x):
 
φ b ( x )=log(1 +x   2 );  (11)
 
φ b ( x )=log(√{square root over (1 +x   2 )}).  (12)
 
     For quadratic asymptotic behavior where φ b (x)˜x 2 :
 
φ b ( x )=log(1+exp( x   2 ));  (13)
 
φ b ( x )=log 2 (1+exp( x )).  (14)
 
     For exponential asymptotic behavior where φ b (x)˜exp(x):
 
φ b ( x )=exp( x );  (15)
 
φ b ( x )=sin  h ( x ).  (16)
 
     For quasi-periodical asymptotic behavior where φ b (x)˜sin(x):
 
φ b ( x )=sin( x ).  (17)
 
     By incorporating the asymptotic behavior of the system  10 , the resulting modeled system  34  may be capable of a substantially improved extrapolation behavior, including the ability to more closely model the actual system  10 . 
       FIG. 3  is a flow chart depicting a logic  40  that may be used to define the model  14 . The logic  40  may collect (block  42 ) a data  44 . The data  44  may include sensor  26  data, actuator  28  data, system inputs  16 , and/or any data suitable for observing the behavior of the actual system  10 . The data collection (block  42 ) may be automated, that is, the data may be logged into a system through a data logger, extracted from existing databases, submitted from an external source, and so forth. The data collection (block  42 ) may also be manual, that is, an operator may enter certain data points of interest. Indeed, any number of data collection systems and methodologies may be used. 
     The collected data  44  may then be pre-processed (block  46 ) so as to render the collected data  44  more useful for empirical modeling. For example, a filter such as a low-pass filter may be utilized to filter noise out of the collected data. In another example, correlated data may be removed. Removing correlated data may be useful in facilitating clusterability and discrimination in the data. The process data  44  may also be pre-processed (block  44 ) so as to place the data  44  in a common time scale. The data  44  may also be normalized during pre-processing (block  46 ), for example, by scaling the data  44  into a numeric range appropriate for use by the model  14 . Accordingly, the data  44  may thus be transformed into a pre-processed data  48 . 
     The pre-processed data  48  may then be utilized as part of an analysis of the extrapolation behavior of the system  10 . The extrapolation behavior may use knowledge available about the system  10  to determine the extrapolation behavior of the system  10 . In one example, the pre-processed data  48  may be analyzed and used to determine any asymptotic tendencies of the system  10 . For example, techniques such as linear regression (e.g., least squares, adaptive estimation, and ridge regression), the method of dominant balance (MDB), and/or others may be used. Indeed, any number of methods useful in the asymptotic analysis of the system  10  may be employed. In cases where the asymptotic behavior of the system is not known, then a desired asymptotic behavior may be used. For example, a linear behavior may suitably control a heating process, while a constant behavior may suitably control a mixing process. Accordingly, an extrapolation behavior  50  may be found or defined. 
     The basis/kernel function φ b (•) capable of rendering the extrapolation behavior  50  may be selected (block  52 ). As mentioned above, any number of equations may be used to define the basis/kernel function φ b (•), including equations 6-17. Indeed, by incorporating the extrapolation behavior of the system  10  into the selection of the basis/kernel function φ b (•), the modeled system  34  may be capable of more closely modeling the actual system  10  (i.e., exhibiting higher fidelity). Once the basis/kernel function φ b (•) has been selected (block  52 ), the logic  40  may then formulate an optimization problem to determine the model&#39;s parameters (block  54 ). The optimization problem may be formulated based on for example, on the modeling goals (e.g., reducing the modeling error), system constraints (e.g., demonstrating a desired asymptotic behavior), and so forth. 
     The logic  40  may then solve the optimization process to determine the model&#39;s parameters (block  56 ). In certain embodiments, this training process may employ a gradient descent method. In gradient descent, one takes steps proportional to the negative of the gradient of the optimization function at its current point. The function may typically decrease fastest if one moves in a gradient towards a local minimum. Accordingly, the gradient descent may find the local minima through the stepwise moves. The set of model parameters w b  may thus be ascertained. The logic  40  may then analyze the resulting model  14  to determine if the resulting model  14  is acceptable (decision  58 ). In certain embodiments, the acceptability of the model may depend on the model&#39;s accuracy, the model&#39;s high-order behavior (e.g., gain or second order derivatives), and the model&#39;s extrapolation properties. In these embodiments, a test data may be used as inputs to the model  14 , and the outputs of the model may then be analyzed to see if they are adequate. For example, the outputs may be analyzed to see if control embodiments of the model  14  adequately respond to the test data by issuing appropriate control signals. Similarly, predictive embodiments of the model  14  may use the test data as inputs of, for example, a simulation, and the simulation results may be studied to verify the quality of the simulation. Likewise, a optimization modalities of the model  14  may use the test data as inputs to optimization embodiments of the model  14  and the outputs of the optimization embodiments may be analyzed to determine their suitability to accurately optimize certain aspects of the system  10  such as cost, production outputs, power generation, and so forth. If the model  14  is deemed not suitable for use, then the logic  40  may loop to block  42  to repeat the model&#39;s training process. Indeed, the model  14  may be iteratively trained so as to achieve an accuracy, a high-order behavior, and extrapolation properties suitable for modeling the system  10 . The model  14  may include neural network and/or support vector machine embodiments capable of employing the techniques described herein, including asymptotic analysis techniques, capable of superior extrapolation properties that may be especially useful in control, prediction, and optimization applications. 
       FIG. 4  depicts an embodiment of a neural network  60  capable of employing the techniques described herein. More specifically, the illustrated neural network may be trained by using the logic  40  as described above with respect to  FIG. 3  to incorporate asymptotic analysis. In the illustrated embodiment, the neural network includes a plurality of input nodes (i.e., input layer)  62 , a plurality of hidden nodes (i.e., hidden layer)  64 , and multiple output nodes (i.e., output layer)  66 . Accordingly, the neural network  60  is a multi-layer, feed-forward network with linear outputs having a single hidden layer  64 . It is to be understood that while the depicted embodiment illustrates a specific neural network architecture, other architectures having more or less nodes as well as having more than one hidden layer may be used. Indeed, the techniques herein may be incorporated in any number of neural network architectures. 
     In the illustrated embodiment, let i, h, and o denote the indexes of the nodes in the input layer  62 , hidden layer  64 , and output layer  66  respectively. Weights between output and hidden nodes are denoted by w oh  and weights between hidden and input nodes are denoted by w hi . Biases of the hidden and output nodes are b h  and b o  respectively. For the hidden nodes, u h  is the weighted sum input, ƒ(•) is the transfer function, and z h  is the output of the hidden nodes. The variable y o  is the o-th component of the multi-output process f(•). Accordingly, the equations below represent the inputs u h , hidden layer output z h , and the o-th component of f(•): 
                       u   h     =         ∑   i     ⁢           ⁢     (       w   hi     ⁢     x   i       )       +     b   h         ;           (   18   )                   z   h     =     f   ⁡     (     u   h     )         ;           (   19   )                 y   o     =         ∑   h     ⁢           ⁢     (       w   oh     ⁢     z   h       )       +       b   o     .               (   20   )               
The o-th component of the multi-output process f(•) may be expressed so as to incorporate the more generic φ(•) as follows:
 
                     y   o     =       ∑   h     ⁢               ⁢     [         w   oh     ·     f   (         ∑   i     ⁢           ⁢     (       w   hi     ⁢     x   i       )       +     b   h       )       +     b   o       ]         ︷       φ     o   ,   h       ⁡     (   ·   )                     (   21   )               
where φ o,h (•) is the o-th component of the basis/kernel function of the neural network  60 , and h is the index that varies over the nodes in the hidden layer  64 . As mentioned above, the higher-order behaviors, such as the sensitivity (i.e., gain) of the model  14  may be useful in certain embodiments such as control embodiments. Accordingly, the sensitivity
 
               ⅆ     y   o         ⅆ     x   i             
or the gain K oi  of the output y o  with respect to the input can be derived as follows:
 
                     K   oi     =         ⅆ     y   o         ⅆ     x   i         =         ∑   h     ⁢           ⁢       w   oh     ⁢     w   hi     ⁢       f   ′     ⁡     (     u   h     )           =       ∑   h     ⁢           ⁢       w   ohi     *         f   ′     ⁡     (     u   h     )       .                     (   22   )               
where ƒ′(u h ) is the derivative of the transfer function ƒ(u h ) and the product w oh w hi =w ohi  is used to emphasize that the gain of a network is a weighted sum of the derivative of its hidden nodes. Sigmoidal functions (e.g., S-shaped functions) such as ƒ(x)=1/(1+exp(−x)) or ƒ(x)=tan h(x) may be used as activation functions for the neural network  60 . However, the emphasis given has not been the asymptotic behavior of the neural network but rather on the quality of the neural network&#39;s  60  fit over the training data. Indeed, in other approaches, the architecture of the neural network  60  may have to be modified extensively, for example, by adding additional hidden layers  64 , in order to arrive at a neural network  60  that may exhibit improved extrapolation behaviors. By way of contrast, the techniques disclosed herein allow for the use of a single hidden layer  64 . By incorporating the extrapolation properties of a node as a deciding factor in the selection of the activation functions, and by using exemplary non-sigmoidal activation functions ƒ(x)=1/(1+exp(x)) or ƒ(x)=1/(1+exp(x 2 )), the neural network  60  may be trained by so as to result in embodiments that overcome the zero model gain and gain inversion problems. In certain embodiments, the neural network  60  may be trained by using the logic  40  of  FIG. 3 .
 
     Sigmoidal functions may be used in neural networks because of their ability to fit the training data well. However, sigmoidal functions are monotonic and bounded. The derivatives of the sigmoidal functions are also bounded, with the gain assuming its maximum when input is at zero and decreasing towards zero as the input moves away from zero. As a consequence, the neural network gain (i.e., the weighted sum of the sigmoidal function&#39;s derivative), also trends to zero as the input moves away from zero. While such behavior may work well in applications that require bounded outputs, other applications such as control applications, typically avoid zero gains. For applications where zero and small gains are to be avoided, a choice of transfer function (i.e., activation function) for the neural network  60  would be as follows: 
                     f   ⁡     (   u   )       =       log   ⁡     (     1   +     exp   ⁡     (   u   )         )       ⇒       {           u   →         -   ∞     ⁢           ⁢   then   ⁢           ⁢     f   ⁡     (   u   )         →   0                 u   →         +   ∞     ⁢           ⁢   then   ⁢           ⁢     f   ⁡     (   u   )         →   u             }     .               (   23   )               
The gain equations will be as follows:
 
     
       
         
           
             
               
                 
                   
                     
                       f 
                       ′ 
                     
                     ⁡ 
                     
                       ( 
                       u 
                       ) 
                     
                   
                   = 
                   
                     
                       1 
                       / 
                       
                         ( 
                         
                           1 
                           + 
                           
                             exp 
                             ⁡ 
                             
                               ( 
                               
                                 - 
                                 u 
                               
                               ) 
                             
                           
                         
                         ) 
                       
                     
                     ⇒ 
                     
                       
                         { 
                         
                           
                             
                               
                                 u 
                                 → 
                                 
                                   
                                     
                                       - 
                                       ∞ 
                                     
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     then 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     
                                       
                                         f 
                                         ′ 
                                       
                                       ⁡ 
                                       
                                         ( 
                                         u 
                                         ) 
                                       
                                     
                                   
                                   → 
                                   0 
                                 
                               
                             
                           
                           
                             
                               
                                 u 
                                 → 
                                 
                                   
                                     
                                       + 
                                       ∞ 
                                     
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     then 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     
                                       
                                         f 
                                         ′ 
                                       
                                       ⁡ 
                                       
                                         ( 
                                         u 
                                         ) 
                                       
                                     
                                   
                                   → 
                                   1 
                                 
                               
                             
                           
                         
                         } 
                       
                       . 
                     
                   
                 
               
               
                 
                   ( 
                   24 
                   ) 
                 
               
             
           
         
       
     
     The activation function in equation 23 is a monotonic but unbounded function whose derivative, equation 24 is also monotonic and bounded in [0, 1]. Consequently, the neural network  60  utilizing such functions will be monotonic with bounded overall gain: 
     
       
         
           
             
               
                 
                   
                     K 
                     oi 
                     min 
                   
                   ≤ 
                   
                     ( 
                     
                       
                         ∑ 
                         h 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         
                           w 
                           ohi 
                         
                         * 
                         
                           
                             f 
                             ′ 
                           
                           ⁡ 
                           
                             ( 
                             
                               u 
                               h 
                             
                             ) 
                           
                         
                       
                     
                     ) 
                   
                   ≤ 
                   
                     
                       K 
                       oi 
                       max 
                     
                     . 
                   
                 
               
               
                 
                   ( 
                   25 
                   ) 
                 
               
             
           
         
       
     
     K oi   min  and K oi   max  can be determined by the user and enforce during training by identifying the suitable gain bounds for the system  10  being modeled. Indeed, the bounded gain can be expressed in terms of the hidden node layer  64 . The hidden node layer  64  may be divided into two groups, h p  for hidden nodes with w h =w oh w hi ≧0 and h n  for the hidden nodes with w h =w oh w hi &lt;0. With 0≦ƒ′(•)≦1 for all input values, the gain bounds on K oi  may be obtained as follows: 
     
       
         
           
             
               
                 
                   
                     
                       max 
                       ⁡ 
                       
                         ( 
                         
                           K 
                           oi 
                         
                         ) 
                       
                     
                     ≤ 
                     
                       
                         ∑ 
                         hp 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         
                           w 
                           oh 
                         
                         ⁢ 
                         
                           w 
                           hi 
                         
                       
                     
                   
                   ; 
                 
               
               
                 
                   ( 
                   26 
                   ) 
                 
               
             
             
               
                 
                   
                     min 
                     ⁡ 
                     
                       ( 
                       
                         K 
                         oi 
                       
                       ) 
                     
                   
                   ≥ 
                   
                     
                       ∑ 
                       hn 
                     
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       
                         w 
                         oh 
                       
                       ⁢ 
                       
                         
                           w 
                           hi 
                         
                         . 
                       
                     
                   
                 
               
               
                 
                   ( 
                   27 
                   ) 
                 
               
             
           
         
       
     
     The upper and lower bounds of equations 26 and 27 are valid for all inputs. However, it may be possible for the gain bounds not to be tight, that is, some gain may be far from the bounds. However, by analyzing the asymptotic gain for the neural network  60 , one can derive K oi   +  for x i →∞ and K oi   −  for x i →∞ as follows: 
                       K   oi   +     =       ∑   hip     ⁢           ⁢       w   oh     ⁢     w   hi           ;           (   28   )                   K   oi   -     =       ∑   hin     ⁢           ⁢       w   oh     ⁢     w   hi           ;           (   29   )               
where hip are the nodes in the hidden layer  64  with w hi ≧0 and hin are the nodes in the hidden layer with w hi &lt;0. Accordingly, this asymptotic analysis allows the training of the neural network as described above with respect to  FIG. 3  where the following optimization problem may be solved:
 
                     {             min   ⁢           ⁢     w   hi       ,     b   h     ,     w   oh     ,     b   o               ∑   d     ⁢           ⁢     L   ⁡     (       y   d     -     y   o       )                   subject   ⁢           ⁢   to   ⁢     :             Eq   .           ⁢     (     18   ⁢     -     ⁢   20     )                               K   oi     m   ⁢           ⁢   i   ⁢           ⁢   n       ≤     (       K   oi     =       ∑   h     ⁢           ⁢       w   ohi     ⁢       f   ′     ⁡     (     u   h     )             )     ≤     K   oi     ma   ⁢           ⁢   x               }     .           (   30   )               
where w hi , b h , w ho , b o  are the neural network parameters, d indexes over the data points, L(•) is any appropriate loss function as a 1-norm, 2-norm, or ∞-norm, and K oi  is the neural network gain to be bounded between known and/or desired gain constraints K oi   min  and K oi   max . Indeed, the neural network  60  created and trained by using this techniques may be capable of improved extrapolation properties, including the avoidance of gain trending towards zero as well as gain inversion.
 
     Further, the neural network  60  may be capable of using unscaled input/output variables after training. As mentioned above, scaling may be used during training to pre-process the raw data. Once neural network  60  has been architected and trained, raw inputs are typically scaled to conform with the scaling performed during training. Similarly, outputs are scaled to conform, for example, to actual engineering units. In certain embodiments, it may be possible to avoid the scaling of inputs and outputs after training. Indeed, in these embodiments, the neural network  60  may be capable of transforming the trained weights and biases such that the neural network  60  may use raw variables in engineering units directly. 
     As described above, the neural network  60  is a multi-layer, feed-forward neural network with one hidden layer  64  and linear output nodes  66 . Let the subscripts i, h, and o index the nodes in the input layer  62 , hidden layer  64 , and output layer  66  respectively. If upper case names denote unscaled variables (e.g., raw or engineering units), while lower case names denote scaled variables, X i /x i  denotes the unscaled/scaled inputs, and Y o /y o  denotes the unscaled/scaled outputs. Weights between output and hidden nodes are denoted by w oh  and weights between hidden and input nodes are denoted by w hi . Biases of the hidden and output nodes are b h  and b o  respectively. For the hidden nodes, u h  is the weighted sum input, ƒ(•) is the transfer function, and z h  is the output of the hidden nodes. For training purposes, the input and output data may be scaled and translated as follows:
 
 x   i   =S   i   X   i   +T   i ;  (31)
 
 y   o   =S   o   X   o   +T   o .  (32)
 
     The network values are given by equations 14-16 above. After training, rather than incurring a time and resource penalty to scale and unscale data points during run time, it may be more efficient to rescale the neural network  60  and then use the raw data directly during run time. The exemplary equations used for scaling the weights and biases after training in terms of the data scaling used during training are as follows: 
                             u   h     =       ⁢         ∑   i     ⁢           ⁢     (       w   hi     ⁢     x   i       )       +     b   h         ;                 =       ⁢         ∑   i     ⁢           ⁢     [       w   hi     ⁡     (         S   i     ⁢     X   i       +     T   i       )       ]       +     b   h         ;                 =       ⁢         ∑   i     ⁢           ⁢     (       w   hi     ⁢     S   i     ⁢     X   i       )       +       ∑   i     ⁢           ⁢     (       w   hi     ⁢     T   i       )       +     b   h         ;                 =       ⁢         ∑   i     ⁢           ⁢     (       W   hi     ⁢     X   i       )       +     B   h         ;                       (   33   )                               (   34   )                               (   35   )                               (   36   )                   where                             W   hi     =       w   hi     ⁢     S   i         ;   and           (   37   )                 B   h     =         ∑   i     ⁢           ⁢     (       w   hi     ⁢     T   i       )       +       b   h     .               (   38   )               
Similarly,
 
                       y   o     =         ∑   h     ⁢           ⁢     (       w   oh     ⁢     z   h       )       +     b   o         ;           (   39   )                       S   o     *     Y   o       +     T   o       =         ∑   h     ⁢           ⁢     (       w   oh     ⁢     z   h       )       +     b   o         ;           (   40   )                   Y   o     =         ∑   h     ⁢           ⁢     [       (       w   oh     /     S   o       )     ⁢     z   h       ]       +       (       b   o     -     T   o       )     /     S   o           ;   and           (   41   )                   Y   o     =         ∑   h     ⁢           ⁢     (       W   oh     ⁢     z   h       )       +     B   o         ;           (   42   )             where                             W   oh     =       w   oh       S   o         ;           (   43   )                 B   o     =         (       b   o     -     T   o       )       S   o       .             (   44   )               
Indeed, while the original neural network having parameters w hi , b h , w ho , and, b o  maps the scaled inputs x i  to the scale outputs y o , the transformed neural network  60  having parameters W hi , B h , W ho , and B o  maps the raw inputs X i  to the raw outputs Y o . Such mapping may be capable of improved speed during run time of the model  14  as well as improved use of resources such as processing cycles and memory. By incorporating the techniques described herein, including activation functions that have been selected due to their asymptotic properties, the neural network  60  embodiments may be capable of enhanced extrapolation behaviors, such as those described in  FIG. 6 , and improved run time performance.
 
       FIGS. 5 and 6  illustrate the results of utilizing the activation functions ƒ(x)=tan h(x) and ƒ(x)=1/(1+exp(x)), respectively. In  FIGS. 5 and 6 , the data is generated by the equation data=0.01*x 2 . That is, the system  10  is being modeled as a polynomial so as to test the ability of the neural networks to predict the resulting output.  FIG. 5  shows the results of a neural network that does not incorporate the techniques described herein. Graph  68  of  FIG. 5  shows the output of the trained neural network over a training data where the training data has been restricted to an ordinate range [0.0, 1.0] and an abscissa range [−10, 10]. The neural network adequately predicts the outputs of the system  10  over the training data. Indeed, graph  68  shows that the neural network fits the trained data very precisely. That is, the predicted output  70  fits over the training output  72 . Neural networks are known to exhibit universal approximation properties over their training data. In other words, the neural network may be designed and/or trained to fit the quadratic training data to any desired degree of accuracy. However, traditional neural networks may not extrapolate well outside of the training data. Graph  74  shows the predictive deficiencies of the neural network over a test data set. The test data set includes an ordinate range [0, 100] and an abscissa range [−100, 100]. The neural network not utilizing the techniques described herein incorrectly predicts an asymptotically flat output  76  whereas the actual output should be the curve  78 . 
     The higher order behaviors of the neural network are illustrated in graph  80 . As expected, the gain shows a well defined, increasing gain profile  82  over the training data. By way of contrast, graph  84  shows a gain profile  86  that trends to zero gain outside of a narrow training band. Such zero trending behavior would be detrimental if used in, for example, controller embodiments. Indeed, such zero trending behavior may result in infinite controller gains. 
       FIG. 6  shows the results of a neural network  60  that incorporates the techniques described herein, including the use of asymptotic analysis to choose a non-sigmoidal activation function ƒ(x)=1/(1+exp(x)). Graph  88  depicts the output of the trained neural network  60  over the training data where, as described above, the training data has been restricted to an ordinate range [0.0, 1.0] and an abscissa range [−10, 10]. The neural network  60  also adequately predicts the outputs of the system  10  over the training data. Indeed, graph  88  shows that the predicted output  90  fits the actual output  92  very precisely. Moreover, graph  94  shows that behavior of the trained neural network  60  over the test data, where the test data also consist of the ordinate range [0, 100] and the abscissa range [−100, 100]. It is clear that the predicted output  96  is no longer flat and that the predictive output  96  shows asymptotic trending of the activation function. Further, the trained neural network  60  shows a much improved extrapolation behavior not only over the predicted output but also over higher order properties of the neural network  60 . 
     Graph  100  shows a gain profile  102  of the trained neural network  60  over the training data. The gain profile  102  is similar to the gain profile  82  in that the gain profile  102  is well defined, and not trending to zero. Further, graph  104  illustrates a much improved gain profile  106  over the test data. As illustrated, the gain profile  106  is also better defined and no longer trending towards zero. Accordingly, the neural network  60  incorporating techniques such as asymptotic analysis and use of unscaled data may be substantially more useful in, for example, control embodiments. Indeed, such techniques may be useful in other embodiments such as support vector machines. 
       FIG. 7  illustrates and embodiment of a support vector machine (SVM)  108 . SVMs, like neural networks, may be architected and/or trained on a set of data so as to achieve universal approximator properties. Accordingly, the SVM  108  may be included in the model  14 . A linear or nonlinear system f(x) may be approximated using the SVM  108  as follows:
 
 f ( x )= w   T φ( x )+ b.   (45)
 
     The equation 45 may be deemed the primal equation for the approximation of f(x), where w and b are the weight and basis coefficients, respectively, and φ(x) is the primal kernel function used for the approximation. Given N set of input/output measurements, the approximation error f(x)−(w T φ(x)+b) may be captured with various loss functions. In one embodiment, the loss function is as follows: 
                         min     [     w   ,   b   ,     ζ   +     ,     ζ   -       }       ⁢     ??   ⁡     (     w   ,   b   ,     ζ   +     ,     ζ   -       )         =         1   2     ⁢     w   T     ⁢   w     +     c   ⁢       ∑     k   =   1     N     ⁢           ⁢     {       ζ   k   +     +     ζ   k   -       }             ;           (   46   )               
subject to:
 
     
       
         
           
             
               
                 
                   
                     { 
                     
                       
                         
                           
                             
                               
                                 
                                   f 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       x 
                                       k 
                                     
                                     ) 
                                   
                                 
                                 - 
                               
                             
                           
                           
                             
                               
                                 
                                   - 
                                   
                                     f 
                                     ⁡ 
                                     
                                       ( 
                                       
                                         x 
                                         k 
                                       
                                       ) 
                                     
                                   
                                 
                                 + 
                               
                             
                           
                         
                         ⁡ 
                         
                           [ 
                           
                             
                               
                                 
                                   
                                     
                                       w 
                                       T 
                                     
                                     ⁢ 
                                     
                                       ϕ 
                                       ⁡ 
                                       
                                         ( 
                                         
                                           x 
                                           k 
                                         
                                         ) 
                                       
                                     
                                   
                                   + 
                                   b 
                                 
                               
                             
                             
                               
                                 
                                   
                                     
                                       w 
                                       T 
                                     
                                     ⁢ 
                                     
                                       ϕ 
                                       ⁡ 
                                       
                                         ( 
                                         
                                           u 
                                           k 
                                         
                                         ) 
                                       
                                     
                                   
                                   + 
                                   b 
                                 
                               
                             
                           
                           ] 
                         
                       
                       ⁢ 
                       
                         
                           
                             
                               
                                 ≤ 
                                 
                                   
                                     ζ 
                                     k 
                                     + 
                                   
                                   + 
                                   ɛ 
                                 
                               
                               , 
                               
                                 
                                   ζ 
                                   
                                     k 
                                     , 
                                   
                                   + 
                                 
                                 ≥ 
                                 0 
                               
                               , 
                               
                                 
                                   for 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   k 
                                 
                                 = 
                                 1 
                               
                               , 
                               
                                 … 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 N 
                               
                             
                           
                         
                         
                           
                             
                               
                                 ≤ 
                                 
                                   
                                     ζ 
                                     k 
                                     + 
                                   
                                   + 
                                   ɛ 
                                 
                               
                               , 
                               
                                 
                                   ζ 
                                   k 
                                   - 
                                 
                                 ≥ 
                                 0 
                               
                               , 
                               
                                 
                                   for 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   k 
                                 
                                 = 
                                 1 
                               
                               , 
                               
                                 … 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 N 
                               
                             
                           
                         
                       
                     
                     } 
                   
                   . 
                 
               
               
                 
                   ( 
                   47 
                   ) 
                 
               
             
           
         
       
     
     The constrained optimization problem described in equations 46 and 47 above may be deemed the primal problem. The primal problem may use a preselected basis/kernel function φ, to yield the w and b coefficients. Additionally, the constant c of equation 46 may be a positive real constant and may be used as a tuning parameter. ε is the desired accuracy level, and ζ k   + , ζ k   −  are slack variables. 
     In another embodiment, the optimization problem may be defined as follows: 
                               max     {     ∞   ,     ∞   *       }       ⁢     Q   ⁡     (     ∞   ,     ∞   *       )         =       -     1   2       ⁢       ∑     k   ,     l   =   1       N     ⁢           ⁢       (       α   k     -     α   k   *       )     ⁢     (       α   l     -     α   l   *       )     ⁢     K   ⁡     (       x   k     ,     x   l       )               ;                 =         -   ɛ     ⁢       ∑     k   =   1     N     ⁢           ⁢     (       α   k     +     α   k   *       )         +       ∑     k   =   1     N     ⁢           ⁢       f   ⁡     (     x   k     )       ⁢     (       α   k     -     α   k   *       )             ;                                   (   48   )                                                           (   49   )                     
subject to
 
                     {               ∑     k   =   1     N     ⁢           ⁢     (       α   k     -     α   k   *       )       =   0                 α   k     ,       α   k   *     ∈     [     0   ,   c     ]               }     ;           (   50   )               
where K(x k ,x l )=φ T (x k )φ(x l ) is the basis/kernel function for the optimization problem, and α k  and α* k  are Lagrange multipliers. Furthermore, the function f(x) may alternatively be defined as:
 
     
       
         
           
             
               
                 
                   
                     f 
                     ⁡ 
                     
                       ( 
                       x 
                       ) 
                     
                   
                   = 
                   
                     
                       
                         ∑ 
                         
                           k 
                           , 
                           
                             l 
                             = 
                             1 
                           
                         
                         N 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       
                         
                           ( 
                           
                             
                               α 
                               k 
                             
                             - 
                             
                               α 
                               k 
                               * 
                             
                           
                           ) 
                         
                         ⁢ 
                         
                           K 
                           ⁡ 
                           
                             ( 
                             
                               
                                 x 
                                 k 
                               
                               , 
                               
                                 x 
                                 l 
                               
                             
                             ) 
                           
                         
                       
                     
                     + 
                     
                       b 
                       . 
                     
                   
                 
               
               
                 
                   ( 
                   51 
                   ) 
                 
               
             
           
         
       
     
     An added advantage of the embodiment utilizing equations 48-51 is that the embodiment may be augmented with explicit constraints on the gain bounds on the SVM  108 . Indeed, the asymptotic behavior of the basis/kernel functions may now be explicitly analyzed and incorporated into the SVM  108 . The SVM  108  may then be optimized and the gain bounded by incorporating the final gains of the trained SVM  108  explicitly as a function of its inputs. The SVM  108  may be trained, for example, through the use of the logic  40  of  FIG. 3 . The gain&#39;s maximum and minimum may be found by solving an optimization problem with the gain as the objective function and the SVM  108  inputs as the decision variables. The inputs can be defined to a domain (e.g., train, test, unbounded) to obtain the precise maximum and minimum gains for the intended operating region. In one embodiment, the gain bounds may be formulated as hard constraints, as shown in equation 30. In another embodiment, the gain bounds may be formulated as penalty functions. Such enhanced capabilities allow the SVM  108  to exhibit superior extrapolation properties. It is to be noted that the techniques described with respect to the SVM  108  may also apply to other empirical modeling approaches such as neural networks. 
     While only certain features of the invention have been illustrated and described herein, many modifications and changes will occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.