Abstract:
The invention concerns a method for the computerized control and/or regulation of a technical system (T). Within the context of the method according to the invention, there is implemented in a preset regulating process (CO 1,  CO 2 ) an exploration rule (EP) by means of which new, as yet unknown, states (x) of the technical system (T) are started, a simulation model (SM) of the technical system (T) checking whether the actions (a 2 ) of the exploration rule (EP) lead to sequential states (x′) lying within predetermined thresholds. Only in that case is the corresponding action (a 2 ) performed according to the exploration rule (EP) on the technical system. The method according to the invention enables new states to be explored within the framework of the operation of a technical system, it being ensured through checking of appropriate thresholds that the exploration is carried out imperceptibly and does not lead to incorrect operation of the technical system. The method according to the invention is suitable in particular for controlling or regulating gas turbines or wind turbines.

Description:
[0001]    This application is the National Stage of International Application No. PCT/EP2014/051186, filed Jan. 22, 2014, which claims the benefit of German Patent Application No. DE 10 2013 205 356.8, filed Mar. 26, 2013. The entire contents of these documents are hereby incorporated herein by reference. 
     
    
     BACKGROUND 
       [0002]    The present embodiments relate to computer-assisted open-loop and/or closed-loop control of a technical system. 
         [0003]    Various computer-assisted methods for performing open-loop and/or closed-loop control of technical systems are known from the prior art. These methods specify, for a current state of the technical system, which action is to be carried out on the technical system. A state is described here by a number of state variables, and an action is described by a number of action variables. In addition to simple table-based regulators that assign corresponding actions to states of the technical system by a table, there are also regulators having an action selection rule that has been learnt with a machine learning method (e.g., based on a recurrent neural network). An application case of such regulators is gas turbines in order to optimize parameters of the turbine such as the efficiency, combustion chamber dynamics, emissions of pollutants. A further application case of these regulators is the control of a wind turbine. In this case, for example, the wear and the efficiency are optimized. 
         [0004]    In order to implement machine-learnt action selection rules, training data is provided. The training data specifies corresponding subsequent states for a number of states and actions that are carried out in these states. In order to generate new training data, the technical system is to be operated in still unknown states. However, these new states are to not disrupt or severely impair the operation of the technical system and, for example, do not bring about malfunctions of the technical system. 
       SUMMARY AND DESCRIPTION 
       [0005]    The scope of the present invention is defined solely by the appended claims and is not affected to any degree by the statements within this summary 
         [0006]    The present embodiments may obviate one or more of the drawbacks or limitations in the related art. For example, a method for performing open-loop and/or closed-loop control of a technical system with which new states of the technical system may be explored without the operation of the technical system being adversely affected is provided. 
         [0007]    The method according to one or more of the present embodiments serves to perform open-loop and/or closed-loop control of a technical system. The open-loop control or closed-loop control carries out, at respective times, actions including one or more action variables on the technical system as a function of one or more states of the technical system, including in each case one or more state variables. 
         [0008]    Within the scope of the method according to one or more of the present embodiments, in act a) a predefined regulating method determines, at respective times, first actions that are to be carried out on the technical system as a function of at least the state of the technical system at the respective time (e.g., if appropriate, also as a function of states at earlier times). 
         [0009]    According to act b), at least one exploration rule determines one or more second actions to be carried out on the technical system, as a function of at least the state of the technical system at the respective time. In the case of a plurality of exploration rules, each exploration rule determines a second action separately. The exploration rules may be defined as desired, and examples of such rules are given below. According to the exploration rule, second actions with which states of the technical system are entered, are defined according to specific criteria. 
         [0010]    In act c), a set of resulting subsequent states of the technical system is predicted based on a simulation model of the technical system for each second action. Such simulation models are known from the prior art. The term set of subsequent states is to be understood broadly and may, if appropriate, also include just one subsequent state at a time that directly follows the respective (current) time. However, the set of subsequent states may include a plurality of future subsequent states. For example, the subsequent states for a predefined future time interval may be predicted with the simulation model. 
         [0011]    In act d), it is checked for each set of the subsequent states predicted in act c) whether the respective set of subsequent states satisfies a predefined threshold value criterion that defines thresholds that are to be complied with for at least one portion of the state variables of the respective set of subsequent states. 
         [0012]    If in act e) one or more sets of subsequent states satisfy the threshold value criterion, a second action is supplied to the predefined regulating method, for which action the predicted set of subsequent states satisfies the threshold value criterion. If there are a plurality of such second actions, corresponding criteria, with which it is specified which of these second actions are supplied to the regulating method, may be defined. If the emphasis is on the exploration of the state space, such second actions having subsequent states that are close to the corresponding thresholds may be supplied to the regulating method. If the emphasis is on the safety of the operation of the technical system, such second actions with subsequent states that are far away from corresponding thresholds of the threshold value criterion are instead fed to the regulating method. The predefined regulating method carries out, as an action, on the technical system the supplied second action instead of the first action or the first action, determined in act a), in combination with the second action. 
         [0013]    The method according to one or more of the present embodiments permits the implementation of an exploration rule in the closed-loop-controlled or open-loop-controlled operation of a technical system. It is provided, through the definition of corresponding threshold values for state variables of subsequent states that are predicted by a simulation model, that the operation of the technical system is not adversely affected, and therefore, new states are entered imperceptibly within the scope of the operation of the technical system. New training data is generated based on the new states with the actions carried out in these states and the subsequent states, which may be used at a later time for machine learning of simulation methods or regulating methods. During the operation of the technical system, the actions that are carried out in act e) are accordingly stored and/or output in combination with the corresponding state and the subsequent state that occurs in reality. 
         [0014]    Depending on the refinement, it is possible, in the event of none of the sets of subsequent states in act e) satisfying the threshold value criterion, to initiate certain measures. In one case, the predefined regulating method carries out the first action determined in act a). One or more modified actions for which the acts c) to e) described above are executed may be determined. The criterion as to how modified second actions are determined may be defined as desired. For example, a percentage with which a second action that is originally determined in step b) is modified may be specified. In a further variant, the acts b) to e) of the method are executed once more. This is appropriate, for example, when the exploration rule is random, and the same second action is not always determined for the same state or if different exploration rules are used during the repeated execution of acts b) to e). 
         [0015]    In one embodiment, the second actions that are determined in act b) constitute a variation of the first actions that are determined in act a). In this context, in act e), the first action that is varied by the variation of the supplied second action is carried out as an action on the technical system. The exploration of the state space is thus realized in the manner of an offset regulating process that, if appropriate, changes corresponding first actions. Nevertheless, there is also the possibility of the second action being defined such that the second action replaces the first action when the second action is used. 
         [0016]    In a further refinement of the method according to one or more of the present embodiments, the threshold value criterion specifies, for at least a portion of the state variables of the subsequent states, respective value ranges within which the values of these state variables are to lie. The value ranges may be defined in absolute terms here but, if appropriate, may also depend on the values of the state variables of the subsequent states and, for example, define a percentage deviation from these values. 
         [0017]    As already mentioned above, the at least one exploration rule may be defined differently depending on the refinement of the method. For example, the at least one exploration rule may include one or more of the following rules: a rule that randomly selects the second action; a rule that is based on expert knowledge; and an ε greedy rule. The exploration rules that are described in documents [1] to [3] may be used. In a further refinement of the method, the simulation model is learnt with a machine learning method and is based, for example, on a learnt neural network (e.g., a recurrent neural network). Such simulation models are sufficient known from the prior art (see, for example, document [4]). 
         [0018]    The predefined regulating method that is used within the scope of one or more of the present embodiments may be configured as desired. For example, regulating methods that are known per se are used here. The predefined regulating method may be one of the following methods: a table-based regulating method in which respective states of the technical system are assigned to first actions by a table; an action selection rule that is learnt with a machine learning method (e.g., based on the method in document [4]); a combination of a table-based regulating method and an action selection rule that is learnt with a machine learning method, where the action selection rule may specify a variation of an action that is determined with the table-based regulating method, and the first action may be the action of the table-based regulating method that is varied by the variation. 
         [0019]    The method according to one or more of the present embodiments is suitable, for example, for performing open-loop or closed-loop control of a gas turbine and/or wind turbine. In this context, a state of the gas turbine may include one or more of the following state variables: the temperature and/or the pressure at one or more locations on the gas turbine (e.g., the ambient temperature and/or the ambient pressure and/or the compressor temperature and/or the compressor pressure and/or the temperature in the combustion chamber and/or the pressure in the combustion chamber); the air humidity at one or more locations on the gas turbine; the combustion chamber accelerations in the gas turbine; the emission of pollutants by the gas turbine (e.g., the emission of nitrogen oxide); and the power generated by the gas turbine. 
         [0020]    In the case of a gas turbine, the action variable or action variables of the first and/or second actions to be carried out at the gas turbine may include a change in the setting of one or more fuel injection valves and/or a change in the position of one or more vanes (e.g., inlet guide vanes) and/or a change in one or more cooling air streams in gas turbines. 
         [0021]    In order to provide undisrupted operation of a gas turbine within the scope of the exploration of the state space, the state variable or state variables in act d) for which thresholds that are to be complied with are defined may be the combustion chamber accelerations in the gas turbine and/or the emission of pollutants by the gas turbine. 
         [0022]    In the case of a technical system in the form of a wind turbine, a state of the wind turbine may include one or more of the following state variables: the temperature and/or the pressure and/or the air humidity at one or more locations on the wind turbine (e.g., the ambient temperature and/or the ambient pressure and/or the ambient air humidity); the strength of the wind at the wind turbine; the amplitude of the oscillations of the tower of the wind turbine; the mechanical loading of the rotor blades of the wind turbine; and the power generated by the wind turbine. 
         [0023]    In the case of a wind turbine, the action variable or action variables of the first and/or second actions to be carried out at the wind turbine include a change in the angles of attack of the rotor blades of the rotor of the wind turbine and/or a change in the orientation of the rotor of the wind turbine with respect to the wind. 
         [0024]    Undisrupted operation of the wind turbine within the scope of the method according to one or more of the present embodiments is provided, for example, if the state variable or state variables in act d), for which thresholds that are to be complied with are defined, include the amplitude of the oscillations of the tower of the wind turbine and/or the mechanical loading of the rotor blades of the wind turbine. 
         [0025]    In addition to the method described above, one or more of the present embodiments also relate to a computer program product having a program that is stored on a machine-readable carrier (e.g., a non-transitory computer-readable storage medium), for executing the method according to one or more of the present embodiments or one or more variants of the method according to one or more of the present embodiments when the program runs on a computer. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0026]      FIG. 1  shows a schematic illustration of the sequence of an embodiment of the method. 
       
    
    
     DETAILED DESCRIPTION 
       [0027]    One or more of the present embodiments are explained according to  FIG. 1  with reference to the control of a technical system T in the form of a gas turbine. In the left-hand part P 1  of  FIG. 1 , an embodiment of the regulating method that is executed online during the operation of the gas turbine is represented. In contrast, in the right-hand part P 2  of  FIG. 1 , method acts with which parts of the method in the section P 1  are generated are represented. The method acts in the section P 2  are executed offline here (e.g., not during the operation of the technical system) and constitute in this sense preparatory steps for the implementation of the method according to one or more of the present embodiments on a gas turbine. 
         [0028]    The control method of the turbine in the section P 1  includes a conventional regulating method based on a first regulator CO 1  and a second regulator CO 2 . The first regulator CO 1  is realized in a manner known as a table-based regulator and outputs, for a state x of the gas turbine at the current time, an action a 0  that is to be carried out at the turbine. In this context, the actions that are to be carried out for corresponding states are stored in the table for the states. The second regulator CO 2  is realized in the form of a learnt action selection rule. The states of the gas turbine include, for example, one or more of the state variables, described above, of a turbine, and the actions relate to the changes in one or more actuation variables of the gas turbine. These actuation variables may be the change in the setting of fuel ratios by one of more fuel injection valves, and the change in position of vanes in the gas turbine and, for example, of inlet guide vanes, as has also been described above. 
         [0029]    According to  FIG. 1 , the regulator CO 1  therefore determines an action a 0  based on the current state x of the gas turbine by accessing a table. The current state x and the action a 0  are supplied to the regulator CO 2  in the form of a learnt action selection rule. This rule includes a simulation model SM of the gas turbine that models the dynamic behavior of the turbine. This simulation model interacts with a control rule CP that determines an optimum first action a 1  as a function of the current state x of the gas turbine by taking into account the action a 0 . The degree to which an action is optimum is based on a suitable reward function. The simulation model SM was previously learnt by a neural network NN that accesses a database DB in which training data in the form of states x previously assumed by the technical system, actions a carried out in these states, and resulting subsequent states x′ are stored. The control rule CP was generated by a control generator CG that accesses the training data from the database DB. 
         [0030]    The regulator CO 2  therefore outputs, as a function of a current action x, an action a 1  that is to be executed on the technical system. Such types of regulators are known from the prior art and are therefore not described in detail. For example, the regulator CO 2  may be based on a recurrent neural network, as is described, for example, in document [4] below. In the embodiment of the regulator described, the regulator is realized as an offset regulator that specifies an offset or a variation that is added to the action a 0 . As a result of this, the first action a 1  is obtained. 
         [0031]    Conventionally, this action a 1  is supplied to the regulator CO 1 , which then carries out this action as action a on the technical system. The states x that are assumed by the gas turbine and the actions a that are carried out in these states, with resulting subsequent states x′, are stored in the database DB, which has already been mentioned above and which may be used to learn the neural network NN or the control generator CG. 
         [0032]    Within the scope of the method according to one or more of the present embodiments, an exploration module EM is provided as a further module. The exploration module EM may be connected as required and therefore used within the scope of the regulation of the gas turbine. When the exploration module is connected, the first action a 1  is no longer supplied directly to the regulator CO 1  but instead is transferred to the exploration module EM and to the action selection module AS there. In addition, the exploration module receives the current state x of the gas turbine. The exploration module EM serves to explore the space of the states of the gas turbine (e.g., to search for states that have not been assumed by the gas turbine). Since this module is used during the ongoing operation of the technical system, the exploration of new states is to occur in an imperceptible way. 
         [0033]    The new states are to be selected such that the operation of the gas turbine is not perceptively impaired. For example, the emissions of pollutants and/or the combustion chamber accelerations and therefore the humming of the combustion chamber should not increase too much. 
         [0034]    An exploration rule EP that may be defined differently depending on the refinement is stored in the exploration module EM. The exploration rule specifies here, based on the current state x of the gas turbine, a second action a 2 , which is in turn an offset or a variation that is added to the first action a 1  in the event of the exploration rule being applied. The exploration rule may be realized in different ways depending on the application case. For example, the exploration rule may randomly determine a second action a 2  based on the current state x. The exploration rule may be defined based on the methods in documents [1] to [3] below or may be an ε greedy exploration, which is known. The exploration rule may, if appropriate, be defined by expert knowledge. 
         [0035]    The second action a 2 , which is determined with the exploration rule, is supplied to the action selection module AS, which interacts with a simulation model SM of the gas turbine. This simulation model corresponds to the simulation model of the regulator CO 2  and was determined by the learning of the neural network NN. In the action selection module AS, the simulation model SM determines which subsequent states x′ of the technical system the application of the action a 1  varied with the action a 2  will lead to. In this context, the subsequent states are considered for a predetermined future time interval. 
         [0036]    In the action selection module AS, predetermined value ranges for specific state variables of the subsequent states of the gas turbine are stored. For example, in this context, an upper threshold value for the humming of the combustion chamber and/or the emissions of nitrogen oxide are stored. If this threshold value of the state variables is exceeded by at least one of the subsequent states, the second action a 2  is not permissible. In this case, the originally determined action a 1  is transmitted by the action selection module to the regulator CO 1 , which then carries out this action. If, in contrast, the corresponding threshold values of the state variables of the subsequent states are not exceeded, the second action a 2 , which is determined by the exploration rule EP, is fed to the regulator CO 1  in addition to the action a 1 . The regulator then carries out the sum of the two actions. In this way, it is provided that exploration of the state space of the gas turbine to the extent of new states is carried out only when specific limiting values of the emission of pollutants and of the humming of the combustion chamber are not exceeded. This results in the operation of the gas turbine not being adversely affected by the exploration. The exploration therefore takes place in an imperceptible way and is not perceived by the operator of the gas turbine. 
         [0037]    In modifications of the method described above, if appropriate, a plurality of exploration rules EP may also be stored in the exploration module EM. In this context, a plurality of second actions a 2  are determined by the exploration modules, and subsequently, all the second actions having subsequent states that lie within the corresponding limiting values of the state variables are determined. An action is then selected from these second actions in accordance with a predetermined criterion. This may be, for example, the action that gives rise to subsequent states having state variables that are furthest away from the threshold values if the priority is placed on safe operation of the technical system. If, in contrast, the priority is placed on good exploration of the state space, that second action a 2 , which gives rise to subsequent states having state that variables are closest to the corresponding threshold values, may also be used. 
         [0038]    The exploration module EM may, if appropriate, also be used in a regulating method that uses only the table-based regulator CO 1 . In this case, the first action corresponds to the action a 0  that is sent to the exploration module EM and processed there in a way analogous to that described above. 
         [0039]    The embodiment of the method according to one or more of the present embodiments that is described above has a series of advantages. For example, exploration of the state space of the technical system to the extent of new states is also achieved imperceptibly within the scope of regulated operation of a technical system. These new states together with the actions carried out therein, and the subsequent states therefore extend the set of data relating to the technical system. The data may be used as training data in corresponding methods for learning simulation models or action selection rules of the technical system. Within the scope of the exploration, the exploration does not give rise to undesired operating states of the technical system that, under certain circumstances, trigger warnings or alarms. Instead, the exploration is configured by defining corresponding threshold values such that the exploration takes place imperceptibly and does not adversely affect the operation of the technical system. 
         [0040]    The elements and features recited in the appended claims may be combined in different ways to produce new claims that likewise fall within the scope of the present invention. Thus, whereas the dependent claims appended below depend from only a single independent or dependent claim, it is to be understood that these dependent claims may, alternatively, be made to depend in the alternative from any preceding or following claim, whether independent or dependent. Such new combinations are to be understood as forming a part of the present specification. 
         [0041]    While the present invention has been described above by reference to various embodiments, it should be understood that many changes and modifications can be made to the described embodiments. It is therefore intended that the foregoing description be regarded as illustrative rather than limiting, and that it be understood that all equivalents and/or combinations of embodiments are intended to be included in this description. 
       LIST OF REFERENCES 
       [0042]    [1] Michael Kearns and Satinder Singh, “Near-optimal reinforcement learning in polynomial time”, in Proceedings of the 15th International Conference on Machine Learning, pp. 260-268 (1998). 
         [0043]    [2] Ronen I. Brafman and Moshe Tennenholtz, “R-max—a general polynomial time algorithm for near-optimal reinforcement learning”, Journal of Machine Learning Research, 3, 213-231, (2003). 
         [0044]    [3] Alexander L, Strehl and Michael L. Littman, “An analysis of model-based interval estimation for markov decision processes.”, Journal of Computer and System Sciences, 74(8), 1309-1331, (2008). 
         [0045]    [4] DE 10 2007 001 025 A1