Patent Description:
<CIT> describes that "a cycle in which a current state of an environment in which a learning target exists is observed, a predetermined action is executed in the current state, and some reward is given to the action is repeated in a trial-and-error manner, and a measure that maximizes a total sum of rewards is learned as an optimal solution". Further, <CIT> relates to a plant control system using machine learning to achieve said control.

The present invention is defined by the subject-mater of claim <NUM>. The dependent claims describe further preferred embodiments, and this description explains how the present invention may be carried out.

Hereinafter, (some) embodiment(s) of the present invention will be described. The embodiment(s) do(es) not limit the invention according to the claims,
and all the combinations of the features described in the embodiment(s) are not necessarily essential to means provided by aspects of the invention.

<FIG> illustrates an example of a block diagram of a learning device <NUM> according to the present embodiment together with equipment <NUM> provided with a control target <NUM>. The learning device <NUM> according to the present embodiment initializes a machine learning model to be used for the control of the control target <NUM> by performing preliminary learning before the start of the reinforcement learning of the machine learning model.

The equipment <NUM> is an installation, a device, or the like provided with the control target <NUM>. For example, the equipment <NUM> may be a plant or may be a composite device obtained by combining a plurality of instruments. Examples of the plant include an industrial plant such as a chemical plant or a bio plant, a plant for managing and controlling a wellhead such as a gas field and an oil field and surroundings thereof, a plant for managing and controlling power generation such as a hydraulic power, a thermal power, or a nuclear power, a plant for managing and controlling environmental power generation such as a solar power or a wind power, and a plant for managing and controlling vertical water, dams, or the like. As an example, the equipment <NUM> may be a three-stage water tank, a heat treatment furnace, or the like, which is one of process devices.

The equipment <NUM> is provided with the control target <NUM>. In the present drawing, a case where only one control target <NUM> is provided in the equipment <NUM> is illustrated as an example, but the present invention is not limited thereto. The equipment <NUM> may be provided with a plurality of the control targets <NUM>.

The equipment <NUM> may be provided with one or more sensors (not illustrated) for measuring various states (physical quantity) inside and outside the equipment <NUM>. The sensor outputs state data indicating the measured state. Such state data may include, for example, operation data, consumption amount data, external environment data, and the like.

Here, the operation data indicates an operation state as a result of controlling the control target <NUM>. The operation data includes a process variable PV (Process Variable) called a process value. As an example, when the equipment <NUM> is a three-stage water tank, the operation data may include data indicating the water level of the water tank. In addition, when the equipment <NUM> is a heat treatment furnace, the operation data may include data indicating the internal temperature (furnace temperature) of the furnace.

The operation data includes data indicating a manipulated variable (MV) given to the control target <NUM>. As an example, when the equipment <NUM> is a three-stage water tank, the operation data may include data indicating the opening degree of a valve as the control target <NUM>. In addition, when the equipment <NUM> is a heat treatment furnace, the operation data may include data indicating a current to a heating wire of a heater as the control target <NUM>.

The consumption amount data indicates the consumption amount of at least one of an energy and a raw material in the equipment <NUM>. For example, the consumption amount data may include the consumption amount of a power or a fuel, or the like.

The external environment data indicates a physical quantity which can act as a disturbance for the control of the control target <NUM>. For example, the external environment data may include the temperature, the humidity, the sunshine, the wind direction, the wind volume, and the precipitation of the outside air of the equipment <NUM>, various physical quantities which change with the control of another instrument provided in the equipment <NUM>, or the like.

The control target <NUM> is an instrument, a device, and the like to be controlled. For example, the control target <NUM> may be an actuator, which controls at least one physical quantity of the amount, the temperature, the pressure, the flow rate, the speed, the pH, or the like of an object in the process of the equipment <NUM>, such as a valve, a heater, a motor, a fan, or a switch and executes a required manipulation corresponding to the manipulated variable MV. As an example, when the equipment <NUM> is a three-stage water tank, the control target <NUM> may be a valve which controls the water level of the water tank. In addition, when the equipment <NUM> is a heat treatment furnace, the control target <NUM> may be a heater which controls a furnace temperature.

Such a control target <NUM> may be switchable, for example, between feedback (FB) control based on a manipulated variable MV (FB) given by an FB controller and artificial intelligence (Al) control based on a manipulated variable MV (Al) given by a machine learning model (also referred to as an AI model). In addition, such FB control may be, for example, control using at least one of proportional control (P control), integral control (I control), and differential control (D control), and may be PID control as an example.

The learning device <NUM> according to the present embodiment initializes the machine learning model to be used for the AI control of such a control target <NUM> by performing the preliminary learning before the start of the reinforcement learning of the machine learning model. That is, the learning device <NUM> according to the present embodiment initializes the machine learning model in order to start the reinforcement learning of the machine learning model from a state where prior knowledge is introduced by the preliminary learning instead of starting the reinforcement learning from a fresh state.

The learning device <NUM> may be a computer such as a personal computer (PC), a tablet computer, a smartphone, a workstation, a server computer, or a general-purpose computer, or may be a computer system in which a plurality of computers are connected. Such a computer system is also a computer in a broad sense. In addition, the learning device <NUM> may also be implemented by one or more virtual computer environments executable in a computer. Alternatively, the learning device <NUM> may be a dedicated computer designed for the preliminary learning of the machine learning model or may be dedicated hardware realized by a dedicated circuit. In addition, when the learning device <NUM> can be connected to the Internet, the learning device <NUM> may be realized by cloud computing.

The learning device <NUM> includes a data acquisition unit <NUM>, an extraction unit <NUM>, a preliminary learning unit <NUM>, and a model storage unit <NUM>. Note that these blocks are functional blocks which are functionally separated from each other, and may not necessarily coincide with an actual device configuration. That is, in the present drawing, even though the block is illustrated as one block, the block may not necessarily be configured by one device. In addition, in the present drawing, even if the blocks are illustrated as separate blocks, the blocks may not necessarily be configured by separate devices.

Before the control of the control target <NUM> provided in the equipment <NUM> by the machine learning model which outputs the action corresponding to the state of the equipment <NUM>, the data acquisition unit <NUM> acquires initialization data including state data indicating the state of the equipment <NUM> and action data indicating the action on the control target <NUM>. The data acquisition unit <NUM> supplies the acquired initialization data to the extraction unit <NUM>.

The extraction unit <NUM> extracts sample data to be used for the initialization of the machine learning model from the initialization data. More specifically, the extraction unit <NUM> includes a selection unit <NUM> and a definition unit <NUM>.

The selection unit <NUM> selects the initialization data acquired by the data acquisition unit <NUM>. As a result, the extraction unit <NUM> extracts the sample data from the selected initialization data. The selection unit <NUM> supplies the selected initialization data to the definition unit <NUM>.

The definition unit <NUM> defines options for the machine learning model to choose an action on the basis of the initialization data selected by the selection unit <NUM>. As a result, the extraction unit <NUM> extracts, as the sample data, a combination of the state data included in the initialization data and the action included in the option. The extraction unit <NUM> supplies the extracted sample data to the preliminary learning unit <NUM>.

Before the start of the reinforcement learning of the machine learning model, the preliminary learning unit <NUM> initializes the machine learning model by the preliminary learning on the basis of the initialization data. More specifically, the preliminary learning unit <NUM> performs the preliminary learning by using the sample data extracted by the extraction unit <NUM> from the initialization data acquired by the data acquisition unit <NUM>, thereby initializing the machine learning model.

The model storage unit <NUM> stores the machine learning model. When the preliminary learning unit <NUM> performs the preliminary learning on the basis of the initialization data, the model storage unit <NUM> stores the initialized machine learning model initialized by the preliminary learning unit <NUM>. In this manner, the learning device <NUM> initializes the machine learning model to be used for the AI control of the control target <NUM> by performing the preliminary learning before the start of the reinforcement learning of the machine learning model. This will be described in detail by exemplifying a case where the equipment <NUM> is a three-stage water tank.

<FIG> illustrates an example of the process variable PV and the manipulated variable MV which may be acquired as the state data by the learning device <NUM> according to the present embodiment. In the present drawing, a horizontal axis represents the time T. In addition, on the upper side in the present drawing, a vertical axis represents the process variable PV. Here, the process variable PV indicates the water level of the water tank. In addition, on the lower side in the present drawing, the vertical axis represents the manipulated variable MV. Here, the manipulated variable MV indicates the valve opening degree.

In the present drawing, a state is illustrated in which the process variable PV = <NUM> and the manipulated variable MV = <NUM> at the time TA. Then, at the time TB following the time TA, the state is illustrated to change to the state of the manipulated variable MV = <NUM>. The learning device <NUM> according to the present embodiment may acquire at least such process variable PV and manipulated variable MV as the state data.

<FIG> illustrates an example of a distribution of a manipulated variable change amount ΔMV which may be acquired as the action data by the learning device <NUM> according to the present embodiment. In the present drawing, a horizontal axis represents the manipulated variable change amount ΔMV. Here, the manipulated variable change amount ΔMV indicates a change amount in the manipulated variable MV, that is, a value obtained by subtracting a current value from a next value in the manipulated variable MV. As an example, the manipulated variable change amount ΔMV at the time TA is <NUM> - <NUM> = -<NUM>. The learning device <NUM> according to the present invention acquires such a manipulated variable change amount ΔMV as the action data. In addition, in the present drawing, a vertical axis indicates the number of times the corresponding manipulated variable change amount ΔMV appears. In this manner, as illustrated in the present drawing, the manipulated variable change amount ΔMV may be distributed such that some groups of the manipulated variable change amounts ΔMV concentrated to some extent exist instead of arbitrary manipulated variable change amounts ΔMV being randomly distributed.

<FIG> illustrates an example of a flow of the preliminary learning by the learning device <NUM> according to the present embodiment.

In step S410, the learning device <NUM> acquires the initialization data. For example, before the control of the control target <NUM> provided in the equipment <NUM> by the machine learning model which outputs the action corresponding to the state of the equipment <NUM>, the data acquisition unit <NUM> acquires the initialization data including the state data indicating the state of the equipment <NUM> and the action data indicating the action on the control target <NUM>.

The data acquisition unit <NUM> acquires the initialization data before the control (Al control) of the control target <NUM> by the machine learning model. At this time, for example, the data acquisition unit <NUM> may acquire the initialization data from data obtained when the control target <NUM> is subjected to the FB control (for example, PID control), may acquire the initialization data from data obtained when the control target <NUM> is manually operated by an operator, or may acquire the initialization data from data obtained from a step response of the control target <NUM>. Note that when there is no or insufficient actual data, the data acquisition unit <NUM> may acquire the initialization data from simulation data obtained by performing simulation on the basis of the physical model of the control target <NUM>. At this time, the data acquisition unit <NUM> may acquire the initialization data such that various data in various situations due to a large number of initial conditions and disturbances is included as well as limited data to be stabilized from one initial state to a target value.

For example, the data acquisition unit <NUM> receives the state data measured by the sensor provided in the equipment <NUM> in time series from the equipment <NUM> via the network. However, the present invention is not limited thereto. The data acquisition unit <NUM> may acquire such state data by receiving the state data from another device different from the equipment <NUM>, may acquire the state data via user input, or may acquire the state data by reading the state data from various memory devices.

As an example, the data acquisition unit <NUM> may acquire the state data including the process variable PV as the state <NUM> and the manipulated variable MV as the state <NUM> as illustrated in <FIG>, for example. As a result, the data acquisition unit <NUM> acquires, for example, the state data indicating that state (state <NUM>, state <NUM>) = (<NUM>, <NUM>) at the time TA.

The data acquisition unit <NUM> acquires data indicating the manipulated variable change amount ΔMV by subtracting the current value from the next value in the manipulated variable MV. As an example, the state is assumed to change to the state of the manipulated variable MV = <NUM> at the time TB following the time TA. In this case, the data acquisition unit <NUM> subtracts the manipulated variable MV = <NUM> at the time TA from the manipulated variable MV = <NUM> at the time TB to acquire the data indicating that the manipulated variable change amount ΔMV = -<NUM> at the time TA. The data acquisition unit <NUM> may acquire such a manipulated variable change amount ΔMV as the action data. As a result, the data acquisition unit <NUM> acquires, for example, the action data indicating that the action is (-<NUM>) at the time TA.

That is, the data acquisition unit <NUM> may acquire each of the state (<NUM>, <NUM>) as the state data and the action (-<NUM>) as the action data for the time TA. This means that in a state where the water level of the water tank is <NUM>, and the valve opening degree is <NUM>% at the time TA, the valve as the control target <NUM> is rotationally controlled by -<NUM>% (for example, <NUM>% in a clockwise direction which is a direction in which the valve is closed).

The data acquisition unit <NUM> may acquire the initialization data in this manner, for example. Note that, in the above description, a case where the data acquisition unit <NUM> receives the state data via the network and performs calculation by itself using the received state data to acquire the action data has been described as an example. However, the present invention is not limited thereto. The data acquisition unit <NUM> may receive the action data in addition to the state data via the network. The data acquisition unit <NUM> supplies the acquired initialization data to the extraction unit <NUM>.

In step S420, the learning device <NUM> selects the initialization data. For example, the selection unit <NUM> selects the initialization data acquired in step S410. That is, the selection unit <NUM> selects data to be used for the preliminary learning from the acquired initialization data. At this time, for example, the selection unit <NUM> may automatically calculate the width of an overshoot/undershoot or a hunting, an offset value, or the like as the evaluation value of a control performance, and select the initialization data such that each evaluation value is only data within a predetermined range. In addition, for example, the selection unit <NUM> may evaluate a similarity between the data on the basis of a kernel function and select the initialization data such that a large amount of data having a low similarity is included. The selection unit <NUM> supplies the selected initialization data to the definition unit <NUM>.

In step S430, the learning device <NUM> defines options. For example, the definition unit <NUM> defines options for the machine learning model to choose an action on the basis of the initialization data selected in step S420. As an example, the definition unit <NUM> defines the options by analyzing the manipulated variable change amount ΔMV included in the initialization data selected in step S420. At this time, the definition unit <NUM> may classify the manipulated variable change amount ΔMV by an existing cluster analysis technique such as an x-means method, and define, as the option, the manipulated variable change amount ΔMV (for example, a median value, an average value, or the like of the manipulated variable change amount ΔMV belonging to the same class) as a representative of each class. As an example, it is assumed that the manipulated variable change amounts ΔMV included in the selected initialization data are distributed as illustrated in <FIG>. In this case, the definition unit <NUM> may classify the manipulated variable change amounts ΔMV into seven classes, and define, as the options, a set of manipulated variable change amounts ΔMV including the representative values of the respective classes, here, manipulated variable change amounts ΔMV = -<NUM>, -<NUM>, -<NUM>, <NUM>, <NUM>, <NUM>, and <NUM>. In this manner, the definition unit <NUM> defines the options on the basis of the distribution of the actions indicated by the action data included in the initialization data.

In step S440, the learning device <NUM> extracts sample data. For example, the extraction unit <NUM> extracts the sample data from the initialization data selected in step S420. At this time, the extraction unit <NUM> does not use the actual data of the manipulated variable change amount ΔMV as it is, but replaces the actual data with a closest manipulated variable change amount ΔMV' among the options defined in step S430. Then, the extraction unit <NUM> extracts a combination of the state data at the same time point and the manipulated variable change amount ΔMV' replaced as sample data. As an example, when the action (-<NUM>) is acquired as the action data for the time TA, the extraction unit <NUM> replaces "-<NUM>" with the closest manipulated variable change amount ΔMV' among the options defined in step S430, here, "-<NUM>". Then, the extraction unit <NUM> extracts a combination of the state (<NUM>, <NUM>) and the action (-<NUM>) as the sample data for the time TA. In this manner, the extraction unit <NUM> extracts, as the sample data, a combination of the state data included in the initialization data (more specifically, the initialization data selected in step S420) and the action included in the option. The extraction unit <NUM> supplies the extracted sample data to the preliminary learning unit <NUM>.

In step S450, the learning device <NUM> performs the preliminary learning. For example, before the start of the reinforcement learning of the machine learning model, the preliminary learning unit <NUM> initializes the machine learning model by the preliminary learning on the basis of the initialization data. More specifically, the preliminary learning unit <NUM> performs the preliminary learning by using the sample data extracted in step S440 from the initialization data acquired in step S410, thereby initializing the machine learning model.

Here, the preliminary learning unit <NUM> stores, in the machine learning model, a policy for deciding an action for controlling the control target <NUM> according to the state of the equipment <NUM>. As an example, the preliminary learning unit <NUM> stores a plurality of pieces of sample data extracted in step S440 in the table of the machine learning model. Such a table includes a combination of the state (state <NUM>, state <NUM>), that is, the process variable PV and the manipulated variable MV, and the action, that is, the manipulated variable change amount ΔMV', and a weight representing evaluation for the combination. The preliminary learning unit <NUM> stores each combination of the state and the action in the sample data extracted in step S440 in the table, and sets a weight for each combination to an initial value (for example, all values are <NUM>).

Note that, in the above description, a case where the preliminary learning unit <NUM> temporarily sets the weight for each combination to a uniform value has been described as an example, but the present invention is not limited thereto. When an importance level is different for each combination, the preliminary learning unit <NUM> may set the weight for each combination to a value corresponding to the importance level.

In the above description, a case where the preliminary learning unit <NUM> stores, in the table, the state and the action in the sample data by using raw data has been described as an example, but the present invention is not limited thereto. The preliminary learning unit <NUM> may normalize and store at least one of the state and the action in the sample data in a predetermined range (for example, <NUM> to <NUM>).

In this manner, the preliminary learning unit <NUM> initializes the machine learning model to choose an action closer to the action data corresponding to the state data in response to the input of the state data on the basis of the initialization data.

In step S460, the learning device <NUM> stores the machine learning model. For example, the model storage unit <NUM> stores the initialized machine learning model initialized by the preliminary learning in step <NUM>.

<FIG> illustrates an example of a table of the initialized machine learning model initialized by the preliminary learning by the learning device <NUM> according to the present embodiment. As described above, the state <NUM> indicates the process variable PV, and here indicates the water level of the water tank. In addition, the state <NUM> indicates the manipulated variable MV, and here, indicates the valve opening degree. In addition, the action indicates the manipulated variable change amount ΔMV'.

In the present drawing, for example, the first row stores the sample data which is obtained by rotating the valve by +<NUM>% (<NUM>% in a counterclockwise direction) from a state where the water level of the water tank is <NUM> and the valve opening degree is <NUM>. Similarly, the second row stores the sample data which is obtained by rotating the valve by +<NUM>% from a state where the water level of the water tank is <NUM> and the valve opening degree is <NUM>. Then, in this table, the weights are all set to <NUM>, which is the initial value, for each combination of such a state and action.

Since the machine learning model decides an action by using the table initialized in this manner as the policy, the machine learning model outputs the action corresponding to the state of the equipment on the basis of each weight for the combinations of the state data included in the initialization data and the actions included in the options.

It should be noted that only one value of -<NUM>, -<NUM>, -<NUM>, <NUM>, <NUM>, <NUM>, and <NUM> is stored as the action. That is, in the table of the machine learning model, only actions included in the options defined by the definition unit <NUM> are stored. As a result, the action output by the machine learning model is limited to any action included in the options, that is, any one of the manipulated variable change amounts ΔMV = -<NUM>, -<NUM>, -<NUM>, <NUM>, <NUM>, <NUM>, and <NUM>.

Conventionally, the PID control has been used in process control such as temperature adjustment, liquid level adjustment, and flow rate adjustment. In the PID control, stable control can be performed, but an overshoot or an undershoot may occur at the time of rising. In particular, when the overshoot occurs in the temperature adjustment control, the temperature of the target does not decrease, and a problem such as the delay of production start occurs. Here, it is possible to adjust a PID gain so as not to cause the overshoot or the like. However, in that case, a settling time until the response is stabilized is lengthened. Therefore, in order to improve the control performance, currently, it takes much time and effort to adjust each coefficient of the PID to an optimum value.

In this regard, the AI control using the machine learning model has also been proposed. In the AI control, when a machine learning model is generated by performing machine learning such that a phenomenon such as the overshoot is suppressed toward the target value of a certain control target to be stabilized more quickly to the vicinity of the target value, expected control can be performed. One of methods for generating such a machine learning model is reinforcement learning. In general, in a reinforcement learning algorithm, at the initial stage of learning, the machine learning model takes an action of randomly changing the manipulated variable, and the machine learning model is updated by repeating a large number of trials and errors. In this case, it is a current problem that it takes an enormous learning time to complete a model with an excellent control performance. In addition, when the reinforcement learning is applied to an N-order delay system such as temperature control having a long response time, the randomness of action choice at the initial stage of learning and the setting of an inappropriate action width cause a problem that convergence to the target value cannot be achieved even when the learning is repeatedly executed, or a model with an excellent control performance cannot be obtained.

In this regard, the learning device <NUM> according to the present embodiment initializes the machine learning model to be used for the AI control of the control target <NUM> by performing the preliminary learning before the start of the reinforcement learning of the machine learning model. That is, the learning device <NUM> according to the present embodiment initializes the machine learning model in order to start the reinforcement learning of the machine learning model from a state where prior knowledge is introduced by the preliminary learning instead of starting the reinforcement learning from a fresh state. As a result, according to the learning device <NUM> according to the present embodiment, the prior knowledge of control is introduced into the machine learning model, and thus it is possible to shorten the learning time in the subsequent reinforcement learning and improve the accuracy of the model. That is, at the initial stage of learning of the reinforcement learning to be executed afterwards, the machine learning model does not choose the action of randomly changing the manipulated variable, but chooses an action on the basis of the initialization including the know-how of the PID control, manual operation, or the like. Thus, it is possible to obtain a model that achieves more excellent control performance with a small number of times of learning.

The learning device <NUM> according to the present embodiment selects the initialization data and extracts the sample data used for the preliminary learning from the selected initialization data. As a result, the learning device <NUM> according to the present embodiment does not use all the acquired initialization data in the preliminary learning, but actively uses, for example, data when the control performance is excellent or data with a low similarity, and thus it is possible to further shorten the learning time and improve the accuracy of the model.

The learning device <NUM> according to the present embodiment defines an option for the machine learning model to choose an action, and extracts a combination of the state data included in the initialization data and the action included in the option as the sample data used for the preliminary learning. As a result, according to the learning device <NUM> according to the present embodiment, the action output by the machine learning model can be limited to any action included in the options, and thus it is possible to suppress an adverse effect due to the randomness of action choice in the initial learning of the reinforcement learning and the setting of an inappropriate action width.

At this time, the learning device <NUM> according to the present embodiment defines options on the basis of a distribution of actions indicated by the action data included in the initialization data. As a result, according to the learning device <NUM> according to the present embodiment, for example, the initialization can be performed such that the machine learning model outputs an action with a high frequency taken under the PID control or the manual operation.

<FIG> illustrates an example of a block diagram of the learning device <NUM> according to a modification of the present embodiment. In <FIG>, members having the same functions and configurations as those in <FIG> are denoted by the same reference numerals, and description thereof will be omitted except for following differences. The learning device <NUM> according to the present modification further has a function of updating the machine learning model by the reinforcement learning in addition to the function of initializing the machine learning model by the preliminary learning. The learning device <NUM> according to the present modification further includes a reinforcement learning unit <NUM> in addition to the functional units included in the learning device <NUM> according to the above-described embodiment.

In the present modification, the data acquisition unit <NUM> acquires the state data in response to the control of the control target <NUM> by the machine learning model. That is, the data acquisition unit <NUM> acquires the state data under the AI control using the initialized machine learning model or the updated machine learning model obtained by updating the initialized machine learning model. The data acquisition unit <NUM> supplies the acquired state data to the reinforcement learning unit <NUM>. In addition, the data acquisition unit <NUM> inputs the acquired state data to the machine learning model stored in the model storage unit <NUM>.

The reinforcement learning unit <NUM> performs the reinforcement learning by using, as the learning data, the state data and the action data acquired from the machine learning model in response to input of the state data to the machine learning model and updates the machine learning model. For example, in response to the input of the state data acquired by the data acquisition unit <NUM> to the machine learning model (the initialized machine learning model or the updated machine learning model obtained by updating the initialized machine learning model) stored in the model storage unit <NUM>, the reinforcement learning unit <NUM> acquires, as the action data, the action output by the machine learning model.

Here, the machine learning model outputs the action corresponding to the state of the equipment <NUM> as follows, for example. The machine learning model performs kernel calculation with respect to each sample data stored in the table for a combination of the input state data and each action included in the option and calculates a distance to each sample data. Then, the machine learning model sequentially adds the result obtained by multiplying the distance calculated for each sample data by the corresponding weight and calculates an evaluation value for each combination. Then, the machine learning model outputs, as the next action, the action in the combination having the highest evaluation value. For example, the reinforcement learning unit <NUM> acquires, as the action data, the action output from the machine learning model in this manner. Then, the reinforcement learning unit <NUM> executes the reinforcement learning by using, as the learning data, the state data and the action data acquired in this manner under the AI control.

The reinforcement learning here may be similar to the conventional reinforcement learning except that the machine learning model is initialized. For example, the reinforcement learning unit <NUM> executes the reinforcement learning on the basis of each sample data in the learning data and a reward value for the sample data by a known algorithm such as kernel dynamic policy programming (KDPP). At this time, the reinforcement learning unit <NUM> evaluates the chosen action on the basis of the next state data of the manipulated control target <NUM> and calculates a reward value. In this case, as an example, the reinforcement learning unit <NUM> may set a reward function such that the reward value increases when the process variable PV approaches the target value. As a result, the reinforcement learning unit <NUM> overwrites the weight of each sample data in the initialized table and further adds new sample data which has not been stored so far to the table.

<FIG> illustrates an example of a calculation result when the learning device <NUM> according to the modification of the present embodiment outputs an action corresponding to a state by the machine learning model. In the present drawing, a case where the learning device <NUM> acquires the state (state <NUM>, state <NUM>) = (<NUM>, <NUM>) as the state data under the AI control is illustrated as an example. In addition, in the present drawing, a case where a set of the manipulated variable change amounts ΔMV including the manipulated variable change amounts ΔMV = -<NUM>, -<NUM>, -<NUM>, <NUM>, <NUM>, <NUM>, and <NUM> is defined as options is illustrated as an example. Therefore, in the present drawing, each row indicates a combination of the input state data and each action included in the options.

As an example, the first row means that the action (<NUM>) which is one of the options is chosen in the state (<NUM>, <NUM>). Similarly, the second row means that the action (<NUM>) which is one of the options is chosen in the state (<NUM>, <NUM>). The machine learning model calculates each evaluation value for such a combination of the state data and each action included in the options.

For example, the machine learning model performs kernel calculation with respect to each sample data stored in the table for the combination in the first row and calculates a distance to each sample data. Then, the machine learning model sequentially adds the result obtained by multiplying the distance calculated for each sample data by the corresponding weight and calculates the evaluation value S(<NUM>). The machine learning model repeatedly executes such calculation, and calculates the evaluation value S(<NUM>) when the action (<NUM>) is chosen, the evaluation value S(<NUM>) when the action (<NUM>) is chosen, the evaluation value S(<NUM>) when the action (<NUM>) is chosen, the evaluation value S(-<NUM>) when the action (-<NUM>) is chosen, the evaluation value S(-<NUM>) when the action (-<NUM>) is chosen, and the evaluation value S(-<NUM>) when the action (-<NUM>) is chosen. Then, the machine learning model outputs, as the next action, the action in the combination having the highest evaluation value. As an example, when the evaluation value S(-<NUM>) is the highest, the machine learning model outputs the action (-<NUM>) as the next action.

<FIG> illustrates an example of a table of a machine learning model obtained when the learning device <NUM> according to the modification of the present embodiment performs updating by reinforcement learning. As illustrated in the present drawing, the weight of each sample data initialized in the preliminary learning is updated from the initial value. In addition, as illustrated in the present drawing, new sample data which is not stored in the initial learning is added to the table. The reinforcement learning unit <NUM> evaluates the action output, for example, according to the evaluation result in <FIG> by the machine learning model on the basis of the next state data in the equipment <NUM> and calculates the reward value. Then, the reinforcement learning unit <NUM> updates the machine learning model to further increase the reward obtained by a series of actions. That is, the reinforcement learning unit <NUM> overwrites the weight of each sample data stored in the table in order to make it easier for the machine learning model to output an action for further increasing the reward. In addition, the reinforcement learning unit <NUM> can also add new sample data which has not been stored so far to the table. For example, in this manner, the reinforcement learning unit <NUM> updates the machine learning model to further increase the reward obtained by a series of actions.

In general reinforcement learning, the machine learning model chooses a random action at the initial stage of learning. However, in the learning device according to the present modification, the action based on the initialization including the know-how of the PID control, the manual operation, or the like is chosen, and thus it is possible to search for a control method capable of achieving a more excellent control performance with a small number of times of learning.

<FIG> illustrates an example of a block diagram of the control device <NUM> according to the present embodiment together with the equipment <NUM> provided with the control target <NUM>. In <FIG>, members having the same functions and configurations as those in <FIG> are denoted by the same reference numerals, and description thereof will be omitted except for following differences. The control device <NUM> according to the present embodiment further has a function of controlling the control target <NUM> by the machine learning model in addition to the function of the learning device <NUM> described above. The control device <NUM> further includes a control unit <NUM> in addition to the functional units included in the learning device <NUM> described above.

The control unit <NUM> controls the control target <NUM> by the machine learning model. For example, the control unit <NUM> gives the action output by the machine learning model to the control target <NUM> to control the control target <NUM>. That is, the control unit <NUM> may function as a so-called AI controller. In this manner, the control device <NUM> according to the present embodiment may include the above-described learning device <NUM> and the control unit <NUM> which controls the control target by the machine learning model. Note that, at this time, the control unit <NUM> and other functional units may be integrally configured, or may be configured separately (for example, another functional unit is executed in a cloud).

Such a control device <NUM> may be combined with an existing FB controller, for example, a PID controller, and the control of the control target <NUM> may be switched according to a situation. That is, the control device <NUM> may further include an FB controller, and may control the control target <NUM> by switching between the FB control by the FB controller and the AI control by the machine learning model according to various situations (for example, the progress status of learning, the control accuracy, or the like).

Heretofore, the above-described embodiment has been described by exemplifying one possible aspect. However, the above-described embodiment may be modified or applied in various forms. For example, in the above description, a case where the definition unit <NUM> defines common options regardless of the state of the equipment has been described as an example. That is, a case where the definition unit <NUM> defines the set of the manipulated variable change amounts ΔMV including the manipulated variable change amounts ΔMV = -<NUM>, -<NUM>, -<NUM>, <NUM>, <NUM>, <NUM>, and <NUM> as the only option regardless of the state of the equipment <NUM> has been described as an example. However, when the analysis is performed for each state of the equipment <NUM>, a different result can be obtained in the distribution of the manipulated variable change amounts ΔMV. For example, in a state where the water tank is close to empty (the process variable PV is close to <NUM>), it is conceivable that the number of times of appearance of the manipulated variable change amount ΔMV having a large absolute value and a sign of + increases. Conversely, in a state where the water level of the water tank is close to the target value, it is conceivable that the number of appearances of the manipulated variable change amount ΔMV having a small absolute value and a sign of + or - increases. In this manner, when the state of the equipment <NUM> can affect the number of occurrences of the manipulated variable change amount ΔMV, the definition unit <NUM> may define a plurality of options corresponding to the state of the equipment <NUM>.

Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, where the blocks may represent (<NUM>) a stage of a process in which an operation is performed or (<NUM>) a section of a device that is responsible for performing the operation. Specific steps and sections may be implemented by a dedicated circuit, a programmable circuit provided with a computer-readable instruction stored on a computer-readable medium, and/or a processor provided with the computer-readable instruction stored on the computer-readable medium. The dedicated circuit may include a digital and/or analog hardware circuit, and may include an integrated circuit (IC) and/or a discrete circuit. The programmable circuit may include a reconfigurable hardware circuit which includes memory elements such as logical AND, logical OR, logic XOR, logic NAND, logic NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA), and the like.

The computer-readable medium may include any tangible device capable of storing instructions for execution by a suitable device, so that the computer-readable medium having the instructions stored thereon includes a product including instructions that can be executed to create means for executing the operations designated in the flowcharts or block diagrams. Examples of the computer-readable medium may include an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, and the like. More specific examples of the computer-readable medium may include a floppy (registered trademark) disk, a diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an electrically erasable programmable read-only memory (EEPROM), a static random access memory (SRAM), a compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a Blu-Ray (registered trademark) disk, a memory stick, an integrated circuit card, and the like.

The computer-readable instruction may include any one of a source code or an object code written in any combination of one or more programming languages including assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, an object oriented programming language such as Smalltalk (registered trademark), JAVA (registered trademark), and C++, and a conventional procedural programming language such as the "C" programming language or similar programming languages.

The computer-readable instruction may be provided for a processor of a general-purpose computer, a special purpose computer, or another programmable data processing apparatus, or a programmable circuit locally or via a local area network (LAN) or a wide area network (WAN) such as the Internet, and the computer-readable instruction may be executed to create means for executing the operations designated in the flowcharts or block diagrams. Examples of the processor include a computer processor, a processing unit, a microprocessor, a digital signal processor, a controller, a microcontroller, and the like.

<FIG> illustrates an example of a computer <NUM> in which a plurality of aspects of the present invention may be embodied in whole or in part. A program installed in the computer <NUM> can cause the computer <NUM> to function as an operation associated with the device according to the embodiment of the present invention or as one or more sections of the devices, or can cause the operation or the one or more sections to be executed, and/or can cause the computer <NUM> to execute a process according to the embodiment of the present invention or a stage of the process. Such a program may be executed by a CPU <NUM> to cause the computer <NUM> to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described in the present specification.

The computer <NUM> according to the present embodiment includes the CPU <NUM>, a RAM <NUM>, a graphic controller <NUM>, and a display device <NUM>, which are interconnected by a host controller <NUM>. The computer <NUM> also includes input/output units such as a communication interface <NUM>, a hard disk drive <NUM>, a DVD drive <NUM>, and an IC card drive, which are connected to the host controller <NUM> via an input/output controller <NUM>. The computer also includes as a ROM <NUM> and legacy input/output units such as a keyboard <NUM>, which are connected to input/output controller <NUM> via an input/output chip <NUM>.

The CPU <NUM> operates according to the programs stored in the ROM <NUM> and the RAM <NUM>, thereby controlling each unit. The graphics controller <NUM> acquires the image data generated by the CPU <NUM> in a frame buffer or the like provided in the RAM <NUM> or in itself and causes the image data to be displayed on the display device <NUM>.

The communication interface <NUM> communicates with other electronic devices via a network. The hard disk drive <NUM> stores programs and data used by the CPU <NUM> in the computer <NUM>. The DVD drive <NUM> reads programs or data from the DVD-ROM <NUM> and provides the programs or data to the hard disk drive <NUM> via the RAM <NUM>. The IC card drive reads programs and data from the IC card, and/or writes programs and data to the IC card.

The ROM <NUM> stores therein a boot programs or the like executed by the computer <NUM> at the time of activation and/or a program depending on the hardware of the computer <NUM>. The input/output chip <NUM> may also connect various input/output units to the input/output controller <NUM> via parallel ports, serial ports, keyboard ports, mouse ports, or the like.

The program is provided by a computer-readable medium such as the DVD-ROM <NUM> or the IC card. The program is read from a computer-readable medium, installed in the hard disk drive <NUM>, the RAM <NUM>, or the ROM <NUM> which are also examples of the computer-readable medium, and executed by the CPU <NUM>. The information processing described in these programs is read by the computer <NUM> and provides cooperation between the programs and various types of hardware resources. The device or method may be configured by implementing operations or processing of information according to use of the computer <NUM>.

For example, when communication is performed between the computer <NUM> and an external device, the CPU <NUM> may execute a communication program loaded in the RAM <NUM> and instruct the communication interface <NUM> to perform communication processing on the basis of the processing described in the communication program. Under the control of the CPU <NUM>, the communication interface <NUM> reads transmission data stored in a transmission buffer processing area provided in a recording medium such as the RAM <NUM>, the hard disk drive <NUM>, the DVD-ROM <NUM>, or the IC card, transmits the read transmission data to the network, or writes reception data received from the network in a reception buffer processing area or the like provided on the recording medium.

The CPU <NUM> may cause the RAM <NUM> to read all or a necessary portion of a file or a database stored in an external recording medium such as the hard disk drive <NUM>, the DVD drive <NUM> (DVD-ROM <NUM>), or the IC card, and may execute various types of processing on data on the RAM <NUM>. Next, the CPU <NUM> writes back the processed data to the external recording medium.

Various types of information such as various types of programs, data, tables, and databases may be stored in a recording medium and subjected to information processing. The CPU <NUM> may execute various types of processing, which is described throughout the present disclosure and includes various types of operations designated by an instruction sequence of a program, information processing, condition determination, conditional branching, unconditional branching, information retrieval/replacement, and the like, on the data read from the RAM <NUM> and writes back the results to the RAM <NUM>. In addition, the CPU <NUM> may retrieve information in a file, a database, or the like in the recording medium. For example, when a plurality of entries each having the attribute value of a first attribute associated with the attribute value of a second attribute is stored in the recording medium, the CPU <NUM> may retrieve an entry matching a condition in which the attribute value of the first attribute is designated from the plurality of entries, read the attribute value of the second attribute stored in the entry, and thus acquire the attribute value of the second attribute associated with the first attribute satisfying a predetermined condition.

The programs or software modules described above may be stored in a computer-readable medium on the computer <NUM> or near the computer <NUM>. In addition, a recording medium such as a hard disk or a RAM provided in a server system connected to a dedicated communication network or the Internet can be used as the computer-readable medium, thereby providing a program to the computer <NUM> via the network.

While the embodiments of the present invention have been described, the technical scope of the invention is not limited to the above-described embodiments. It is apparent to persons skilled in the art that various alterations and improvements can be added to the above-described embodiments. It is also apparent from the scope of the claims that the embodiments added with such alterations or improvements can be included in the technical scope of the invention.

Claim 1:
A control device (<NUM>) comprising:
a learning device (<NUM>) comprising:
a data acquisition (<NUM>) unit configured to acquire, before control of a control target provided in equipment by a machine learning model that outputs an action corresponding to a state of the equipment, initialization data including state data indicating the state of the equipment and action data indicating an action on the control target; wherein the state data include operation data indicating a process variable of the control target measured using sensors of the equipment and a manipulated variable used to control the control target, and the action data is the manipulated variable change amount; and
a preliminary learning unit (<NUM>) configured to initialize the machine learning model by performing preliminary learning on a basis of the initialization data before start of reinforcement learning of the machine learning model, wherein
the learning device (<NUM>) further comprises an extraction unit (<NUM>) configured to extract sample data to be used for initialization of the machine learning model from the initialization data,
wherein the extraction unit (<NUM>) includes a definition unit (<NUM>) configured to define an option for the machine learning model to choose the action,
the extraction unit (<NUM>) is configured to extract, as the sample data, a combination of the state data included in the initialization data and the action included in the option, and
the definition unit (<NUM>) is configured to define the option on a basis of a distribution of actions indicated by the action data included in the initialization data that is acquired from data obtained when the control target is subjected to a feedback control or a manual control, and
the control device (<NUM>) further comprises a control unit (<NUM>) configured to control the control target, which is an actuator controlling a physical quantity of the equipment corresponding to the manipulated variable change amount, by the machine learning model.