Abstract:
Methods and apparatus are provided pertaining to a design of experiments. The method comprises generating a data set from historical data; identifying and removing any fault data points in the data set so as to create a revised data set; supplying the data points from the revised data set into a nonlinear neural network model; and deriving a simulator model characterizing a relationship between the input variables and the output variables. The apparatus comprises means for generating a data set from historical data; means for identifying and removing any fault data points in the data set so as to create a revised data set; means for supplying the data points from the revised data set into a nonlinear neural network model; and means for deriving a simulator model characterizing a relationship between the input variables and the output variables.

Description:
TECHNICAL FIELD  
       [0001]     The present invention generally relates to a method and apparatus for empirical designs of experiments, and more particularly relates to a particular design of experiments pertaining to a simulator model using historical data and a nonlinear neural network model.  
       BACKGROUND  
       [0002]     Designs of experiments are often used in studying the effects of multiple input variables upon one or more output variables, such as the quantifiable output of a particular process. For example, designs of experiments can be used in testing the effects of various environmental conditions upon the operation of a particular apparatus, such as a gas turbine engine. In such an example, the input variables can represent certain quantifiable conditions, such as altitude and inlet pressure, and the output variables can represent quantifiable measures representing the operation of an apparatus, such as the exhaust gas temperature of a gas turbine engine. Designs of experiments often use linear models to approximate the relationship between the input variables and the output variables.  
         [0003]     Often a design of experiments is conducted by running a series of experiments on an apparatus such as a gas turbine engine. In such experiments, the input variables representing the environmental conditions are systematically altered, and the corresponding effects on the output variables are recorded. However, in many circumstances the physical apparatus may be costly to obtain and/or not readily available. Moreover, it is often difficult, costly and time consuming to properly configure the testing so that the input variables represent the entire range of environmental conditions, and to perform the testing and collect the data from the results of all of the tests to obtain complete and accurate results in the experiments on the apparatus.  
         [0004]     An alternative approach, using an accurate model as a proxy for the apparatus, can save a significant amount of time and money with little loss of accuracy, depending on the accuracy of the baseline model. However, frequently the available models are too complex and/or cumbersome to run efficiently, often relying on thousands of data points, and taking weeks or months to run, for example in the case of available finite element models for gas turbine engines. Other available models, such as linear regression models, may not provide a very accurate fit for the data, particularly for nonlinear relationships among the variables.  
         [0005]     Accordingly, there is a need for an improved design of experiments for modeling relationships between input variables and output variables associated with the operation of an apparatus or other process, such as the operation of a gas turbine engine, that is more accurate, time effective and/or cost effective than existing models, that does not require running new tests on the apparatus or process, and that does not have the limitations of a linear regression model.  
       BRIEF SUMMARY OF THE INVENTION  
       [0006]     A method is provided for a design of experiments for modeling the effects of two or more input variables on one or more output variables. The method comprises a first step of generating a data set comprising data points from historical data for the input variables and the output variables, each data point comprising corresponding values for one or more input variables and one or more output variables. The method further comprises a second step of identifying any fault data points in the historical data, a fault data point being a data point in which an output variable value is determined to be caused by factors other than the input variables, and a third step of removing the identified fault data points from the data set, thereby generating a revised data set. The method further comprises a fourth step of supplying the data points from the revised data set into a nonlinear neural network model, and a fifth step of deriving a simulator model characterizing a relationship between the input variables and the output variables using the nonlinear neural network model with the supplied data.  
         [0007]     An apparatus is provided for modeling the effects of two or more input variables on one or more output variables. The apparatus comprises a means for generating a data set comprising data points from historical data for the input variables and the output variables, in which each data point comprises corresponding values for one or more input variables and one or more output variables. The apparatus further comprises means for identifying any fault data points from the historical data, and means of removing the identified fault data points from the data set, thereby generating a revised data set. The apparatus further comprises means for supplying the data points from the revised data set into a nonlinear neural network model, and means for deriving a simulator model characterizing a relationship between the input variables and the output variables using the nonlinear neural network model with the supplied data. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0008]     The present invention will hereinafter be described in conjunction with the following drawing figures, wherein like numerals denote like elements, and  
         [0009]      FIG. 1  depicts a flowchart illustrating a basic objective of a design of experiments, as known in the prior art;  
         [0010]      FIG. 2  depicts one embodiment of a method for a design of experiments;  
         [0011]      FIG. 3  depicts one embodiment of a method for generating a design of experiments and an enhanced algorithm;  
         [0012]      FIG. 4  depicts an example of a full factorial design from the method of  FIG. 3 ;  
         [0013]      FIG. 5  depicts an application of the full factorial design from the example of  FIG. 4  in connection with an APU simulator;  
         [0014]      FIG. 6  depicts use of statistical analysis of variance techniques in evaluating the input variables from the method of  FIG. 2  in connection with the example of  FIG. 4 ;  
         [0015]      FIG. 7  depicts an example of a linear regression model as applied to the example in  FIG. 4 ;  
         [0016]      FIG. 8  depicts results from the linear regression model from  FIG. 7 ;  
         [0017]      FIG. 9  depicts results from the nonlinear simulator model from  FIGS. 1-3  as applied to the example in  FIG. 4 ;  
         [0018]      FIG. 10  depicts a confusion matrix comparing the results from the linear regression model from  FIG. 8  with the results from the nonlinear simulator model from  FIG. 9 ; and  
         [0019]      FIG. 11  depicts an exemplary computer system for implementing the methods of  FIGS. 2-3 . 
     
    
     DETAILED DESCRIPTION OF THE INVENTION  
       [0020]     The following detailed description of the invention is merely exemplary in nature and is not intended to limit the invention or the application and uses of the invention. Furthermore, there is no intention to be bound by any theory presented in the preceding background of the invention or the following detailed description of the invention.  
         [0021]      FIG. 1  depicts a flowchart illustrating a basic objective of the invention, namely to create a simulator model  10  for modeling the effects of two or more input variables  12  on one or more output variables  14 . In a preferred embodiment, the simulator model  10  pertains to a gas turbine engine for an aircraft. In this preferred embodiment, the input variables  12  comprise various operating conditions affecting performance of a gas turbine engine, such as total air temperature, altitude, inlet pressure, bleed air flow and bleed air pressure, and the output variables  14  comprise one or more performance indicators for a gas turbine engine, such as exhaust gas temperature. However, it will be appreciated by one of skill in the art that the simulator model  10 , the input variables  12 , and the output variables  14  can pertain to operations or other characteristics of any one of a number of different types of apparatus, systems or processes.  
         [0022]     Turning now to  FIG. 2 , a design of experiments (“DOE”) process  16  is shown for generating a simulator model  10  using historical data  18  for the input variables  12  and the output variables  14 . The DOE process  16  comprises a first step  20 , in which data points  22  are gathered from the historical data  18 .  
         [0023]     Preferably each data point  22  comprises corresponding values for each input variable  12  and each output variable  14 , so that the data points  22  represent more accurate and meaningful relationships between the input variables  12  and the output variables  14 . For example, in the case of a gas turbine engine, each data point  22  preferably includes values for each of the input variable  12  environmental conditions affecting turbine engine performance during a particular time period, as well as values for each of the output variable  14  engine turbine performance measures resulting from this particular set of environmental conditions. By including values for each of the input variables  12  and each of the output variables  14  in each data point  22 , this preferred embodiment helps to prevent a situation in which the effects of a particular input variable  12  may otherwise be masked or incorrectly attributed to another input variable  12 , which could occur if the particular input variable  12  did not have a value represented in a particular data point. However, it will be appreciated that in some situations values may be unavailable for one or more of the input variables  12  or output variables  14  in a particular data point  22 , in which case the data point  22  may take a different configuration with less than all of the variable values.  
         [0024]     It will also be appreciated that the historical data  18  may be obtained in any one of a number of different manners, for example from sensor records of prior operations of an apparatus or system. Next, in step  24  a data set  26  is generated by assembling the various data points  22 . The data set  26  comprises the various data points  22  of the historical data  18 .  
         [0025]     Next, in step  28 , the data set  26  is analyzed so as to split fault data  30  from no fault data  32 . For the purposes of step  28 , the fault data  30  includes any data points  22  for which an output variable  14  value is determined to be caused by factors other than the input variables  12 . For example, in the example of a gas turbine engine, the fault data  30  may include data points  22  for which the output variable  14  values are determined to be caused in significant part by some problem in the gas turbine engine, or the operation thereof, rather than by any environmental conditions that may be represented in the input variables  12 . For the purposes of step  28 , the no fault data  32  includes any data points  22  that are not fault data  30 . In other words, the no fault data  32  includes data points  22  for which the output variable  14  values are determined to be caused predominantly by the input variables  12 . As shown, the fault data  30  is removed from the data set  26  in step  34 , the fault data  30  thereby becoming removed data  36 . Conversely, the no fault data  32  is retained in step  38 , resulting in a revised data set  40  comprising data points  22  of the no fault data  32 . The revised data set  40  allows for a more accurate modeling of the effects of the input variables  12  on the output variables  14 .  
         [0026]     Next, in step  42 , the data points  22  of the revised data set  40  are supplied to a neural network model  44  for the purposes of generating a simulator model  10 . In a preferred embodiment a feed-forward neural network model  44  is used; however, it will be appreciated that any one of a number of different types of nonlinear models can be used for the neural network model  44 . Regardless of the particular type of model used, in step  46  the neural network model  44  generates the simulator model  10 , which includes one or more formulas modeling the effects of the input variables  12  on the output variables  14 . For ease of reference, steps  42  and  46  will be collectively hereafter referenced as a single step  48 , “Build a simulator model”, as depicted in  FIG. 3 .  
         [0027]     The simulator model  10  can be a very useful tool in designing, monitoring, and analyzing the particular apparatus, systems or processes for which the simulator model  10  is used. For example, in the above-mentioned application of a gas turbine engine, the simulator model  10  can be used for designing a gas turbine engine or components or parts thereof, improving the engine, components or parts, and predicting performance of an engine, among various other uses. The simulator model  10  can save significant time and money, particularly when (i) the apparatus, system or process to be studied is expensive or difficult to obtain; (ii) it is difficult, expensive or time consuming to run comprehensive testing on the apparatus, system or process; and/or (iii) available models lack sufficient accuracy, precision, simplicity or speed in running.  
         [0028]      FIG. 3  depicts another embodiment of the invention, namely an extended process  50  for enhancing an original algorithm  52 . The extended process  50  includes steps  20 - 48  of the previously described DOE process  16 , as well as additional steps  56 ,  58 ,  60 , and  62 . It will be appreciated that, for clarity and ease of both depiction and description, not all of the steps  20 - 48  are shown in  FIG. 3 .  
         [0029]     The original algorithm  52  includes a subset of the input variables  12 , namely certain input variables  12  originally determined to have substantial effects on the output variables  14 . It will be understood that the original algorithm  52  may be commonly known in the industry, and/or may be the result of empirical testing, a theory or hypothesis, or any one of a number of different ways to generate an algorithm. Regardless of the origin of the original algorithm  52 , the extended process  50  uses a series of steps for enhancing the original algorithm  52 .  
         [0030]     The extended process  50 , similar to the DOE process  16 , begins with generating the data set  26 , comprising data points  22  from the historical data  18 . The data set  26  is split into fault data  30  and no fault data  32 , as with the DOE process  16 . Next, in step  48  the simulator model  10  is built, via the process set forth in greater detail in steps  42  and  46  of  FIG. 2 , utilizing the neural network model  44 .  
         [0031]     In addition, statistical measures  54  are selected for the revised data set  40 , and values for the statistical measures  54  are determined for each of the input variables  12 , based on the data points  22  in the revised data set  40 , for subsequent use with the simulator model  10 . As shown in  FIG. 3 , and in more detail in  FIG. 4 , in a preferred embodiment the statistical measures  54  include minimum, median and maximum values for each of the input variables  12  in the revised data set  40 . However, it will be appreciated that the statistical measures  54  may include any one of a number of other measures such as the mean, mode, lower quartile and/or upper quartile values. Regardless of the particular measures chosen, the use of the statistical measures  54  provides an easier, quicker and more cost effective method for testing the simulator model  10 , and for enhancing the original algorithm  52 , as compared with using the entire revised data set  40 .  
         [0032]     Next, in step  56 , the data points  22  corresponding to the values of the statistical measures  54  are used, in conjunction with the simulator model  10 , to predict values of the output variables  14  corresponding to the values of the statistical measures  54  for the input variables  12 . As shown in  FIGS. 4 and 5  in a preferred embodiment involving a gas turbine engine, the minimum, median, and maximum values of various input variables  12 , such as total air temperature (TAT), altitude (ALT), inlet pressure (P2A), generator load (GLA), bleed air flow (WB), bleed air pressure (PT), inlet guide vane angle (IGV), surge control valve position (SCV), and low oil temperature (LOT), are used to determine a number of corresponding data points  22  for use with the simulator model  10  in determining calculated values  66  of a particular output variable  14 , namely the calculated value  66  for exhaust gas temperature, represented as EGT*. As mentioned above, it will be appreciated that the simulator model  10 , the input variables  12 , the output variables  14 , and the statistical measures  54  can take any of a number of different forms.  
         [0033]     Returning now to  FIG. 3 , in step  58  the results of step  56  are analyzed to determine the dominant input variables  12 , specifically the input variables  12  having dominant effects on the output variables  14 . In a preferred embodiment, the analysis in step  58  is conducted using statistical analysis of variance (“ANOVA”) techniques.  FIG. 6  depicts use of such ANOVA techniques with respect to the above-mentioned application of a gas turbine engine.  
         [0034]     Specifically,  FIG. 6  displays results of ANOVA testing on the effects of specific input variables  12  on EGT*, which, as mentioned above, represents the calculated value  66  of the exhaust gas temperature (EGT) output variable  14  in this example.  FIG. 6  provides, among other information, a value for degrees of freedom (DF)  68  and an F-statistic  70  corresponding to each input variable  12 . For any given degrees of freedom  68 , a larger F-statistic  70  for a particular input variable  12  represents a larger measure of dominance for that particular input variable  12 . In this example,  FIG. 6  shows altitude (ALT), inlet pressure (P2A), bleed air flow (WB), generator load (GLA), total air temperature (TAT), bleed air pressure (PT), and inlet guide vane angle (IGV) having the highest measures of dominance on EGT*, with surge control valve position (SCV) and low oil temperature (LOT) having the smallest measures of dominance on EGT*. While  FIG. 6  depicts a particular application of ANOVA testing, it will be appreciated that any one of a number of different techniques can be used in the step  58  analysis, and that such analysis can be used in any one of a number of different applications.  
         [0035]     Returning again to  FIG. 3 , in step  60  the determination of the dominant input variables  12  from step  58  is compared with the input variables  12  from the original algorithm  52 , to identify any candidates for adding to or removal from the original algorithm  52 . For example, if certain input variables  12  were not in the original algorithm  52  but nonetheless were determined to be dominant input variables  12  in step  58 , such as altitude (ALT), bleed air flow (WB), and bleed air pressure (PT) in the example of  FIG. 6 , such input variables  12  are identified as candidates for adding to the original algorithm  52 . Conversely, if any input variables  12  were in the original algorithm  52  but nonetheless were not determined to be dominant input variables  12  in step  58  (not depicted in  FIG. 6 ), such input variables  12  would be candidates for removal from the original algorithm  52 . As mentioned above, while  FIG. 6  depicts one particular embodiment for algorithm selection with ANOVA testing, it will be appreciated that any one of numerous different techniques can be used in determining the candidates for addition to and removal from the original algorithm  52 .  
         [0036]     Regardless of the particular techniques used in steps  58  and  60 , the results of these steps are utilized in step  62  in generating an enhanced algorithm  64 , which represents the addition and/or removal of certain input variables  12  as determined in steps  58  and  60 . The enhanced algorithm  64  can be used for various purposes such as, for example, improved modeling and analysis of the effects of the input variables  12  on the output variables  14 . For example, step  42  of the DOE process  16  can be re-run using values from the revised data set  40  corresponding with the input variables  12  in the enhanced algorithm  64 , along with the output variables  14 , to generate a new simulator model  10  in step  46  corresponding with the enhanced algorithm  64 . As shown in  FIGS. 7-10  with respect to the above-mentioned gas turbine engine application, the enhanced algorithm  64  can also be used to test the simulator model  10  and to compare the simulator model  10  with other models such as those utilizing linear regression.  
         [0037]     For example,  FIG. 7  depicts a linear regression model  72  utilizing the enhanced algorithm  64  for the above-mentioned gas turbine engine example.  FIG. 8  depicts results  74  from the linear regression model  72  for this example, through a graph of the calculated values  66  of the output variable  14  (depicted as EGT*) versus actual values  76  of the output variable  14  (depicted as EGT). As shown in  FIG. 8 , the linear regression model  72  detected the fault data  30 , but also generated false positives  78  for the fault data  30 .  
         [0038]      FIG. 9  depicts, for comparison, results  80  of the simulator model  10  as applied to this particular example, using the enhanced algorithm  64 . As shown in  FIG. 9 , the simulator model  10  detected the fault data  30 , and had no false positives  78  for the fault data  30 .  
         [0039]     In  FIG. 10 , a confusion matrix  82  compares the respective results  74  from the linear regression model with the results  80  of the simulator  10 , and compares the respective results  74  and  80  with results of the original algorithm  52  (not shown in graphical form). As shown in the confusion matrix  82 , the simulator model  10  performed better than the linear regression model  72  in this application. For example, the simulator model  10  had zero false-positives  78  for fault data  30 , compared with six false-positives  78  for the linear regression model  72 , while still identifying all of the fault data  30 . The simulator model  10  and the linear regression model  72  both performed better than the original algorithm  52 , which had fifteen false-positives  78  for fault data  30 .  
         [0040]     It will be appreciated that the extended process  50  and the enhanced algorithm  64  can be used for various other types of testing, modeling, and analysis, and can be used in any one of a number of different applications.  
         [0041]     In addition, the DOE process  16  and the extended process  50  can be implemented in a wide variety of platforms including, for example, any one of numerous computer systems. Turning now to  FIG. 11 , an exemplary computer system  84  is illustrated by way of example. Computer system  84  illustrates the general features of a computer system that can be used to implement the DOE process  16  and the extended process  50 . Of course, these features are merely exemplary, and it should be understood that the processes  16 ,  50  can be implemented using different types of hardware that can include more or different features. It should be noted that the computer system  84  can be implemented in many different environments, such as within a particular apparatus or system, or remote from a particular apparatus or system. The exemplary computer system  84  includes a processor  86 , an interface  88 , a storage device  90 , a bus  92 , and a memory  94 .  
         [0042]     The processor  86  performs the computation and control functions of the computer system  84 . The processor  86  may comprise any type of processor, include single integrated circuits such as a microprocessor, or may comprise any suitable number of integrated circuit devices and/or circuit boards working in cooperation to accomplish the functions of a processing unit. In addition, the processor  86  may comprise multiple processors implemented on separate systems. In addition, the processor  86  may be part of an overall system for an apparatus or process. During operation, the processor  86  executes the programs contained within the memory  94  and as such, controls the general operation of the computer system  84 .  
         [0043]     The memory  94  can be any type of suitable memory. This would include the various types of dynamic random access memory (DRAM) such as SDRAM, the various types of static RAM (SRAM), and the various types of non-volatile memory (PROM, EPROM, and flash). It should be understood that the memory  94  may be a single type of memory component, or it may be composed of many different types of memory components. In addition, the memory  94  and the processor  86  may be distributed across several different computers that collectively comprise the computer system  84 . For example, a portion of the memory  94  may reside on a computer within a particular apparatus or process, and another portion may reside on a remote computer.  
         [0044]     The bus  92  serves to transmit programs, data, status and other information or signals between the various components of the computer system  84 . The bus  92  can be any suitable physical or logical means of connecting computer systems and components. This includes, but is not limited to, direct hard-wired connections, fiber optics, infrared and wireless bus technologies.  
         [0045]     The interface  88  allows communication to the computer system  84 , and can be implemented using any suitable method and apparatus. It can include one or more network interfaces to communicate to other systems, terminal interfaces to communicate with technicians, and storage interfaces to connect to storage apparatuses such as the storage device  90 . The storage device  90  can be any suitable type of storage apparatus, including direct access storage devices such as hard disk drives, flash systems, floppy disk drives and optical disk drives. As shown in  FIG. 11 , the storage device  90  can comprise a disc drive device that uses discs  96  to store data.  
         [0046]     In accordance with a preferred embodiment, the computer system  84  includes a program  98  for use in implementing the DOE process  16  and/or the extended process  50 . During operation, the program  98  is stored in the memory  94  and executed by the processor  86 . As one example implementation, the computer system  84  may also utilize an Internet website, for example for providing or maintaining data or performing operations thereon.  
         [0047]     It should be understood that while the embodiment is described here in the context of a fully functioning computer system, those skilled in the art will recognize that the mechanisms of the present invention are capable of being distributed as a program product in a variety of forms, and that the present invention applies equally regardless of the particular type of computer-readable signal bearing media used to carry out the distribution. Examples of signal bearing media include: recordable media such as floppy disks, hard drives, memory cards and optical disks (e.g., disk  96 ), and transmission media such as digital and analog communication links.  
         [0048]     While at least one exemplary embodiment has been presented in the foregoing detailed description of the invention, it should be appreciated that a vast number of variations exist. It should also be appreciated that the exemplary embodiment or exemplary embodiments are only examples, and are not intended to limit the scope, applicability, or configuration of the invention in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient road map for implementing an exemplary embodiment of the invention, it being understood that various changes may be made in the function and arrangement of elements described in an exemplary embodiment without departing from the scope of the invention as set forth in the appended claims and their legal equivalents.