Patent Description:
Model predictive control, which is generally applied in the fields of automobiles and plants (power and industrial), has the need to predict states of a control object and its surrounding environment in a more distant future.

A device and a method described below exist for predicting future states of an operation target and its surrounding environment.

PTL <NUM> discloses a method of predicting a future state using a model simulating a control object and its surrounding environment, and calculating an operation amount suitable for the future state.

PTL <NUM> discloses a method of predicting present and future states of an industrial system as a control object and optimizing a control law so as to maximize an objective function.

PTL <NUM> discloses a method in which a nonlinear and dynamic system such as a heat reaction furnace process is modeled by a regression method, and an optimal operation amount is calculated using a future state predicted by the model.

PTL <NUM> is a control parameter automatic adjustment apparatus that can automatically optimize a control parameter according to a purpose while satisfying a constraint condition in plant operation and also shorten a calculation time required for optimization of the control parameter. A method of calculating a control law considering a future state by using a plant model and a machine learning method such as reinforcement learning is disclosed.

PTL <NUM> describes the probabilistic prediction of future behaviour and/or the optimisation of productivity parameters of a vast number of remote local industrial systems. PTL <NUM> describes a system for modeling, simulating and analyzing chemical and biochemical reactions includes a modeling environment for constructing a model of a chemical or biochemical system comprising a plurality of chemical reactions.

PTLs <NUM>, <NUM>, <NUM>, and <NUM> predict a future state using a model simulating a control object and its surrounding environment, and calculate an optimal control method from the predicted future state. Although there is the need to predict a more distant future state, a method using iterative calculation requires a longer time for prediction calculation as time to a future state to be predicted is longer. In view of the above, up to a future state in a predictable finite time is generally calculated under the constraints of computer performance and a control period.

In view of the above, an object of the present invention is to provide a future state estimation method and a future state estimation device with which, within space in a finite state defined in advance, it is possible to rapidly estimate states of a control object and its surrounding environment in an infinite time ahead in a form of probability density distribution.

According to the present invention, it is possible to calculate future states of a control object and its surrounding environment in an infinite time ahead in a form of probability density distribution without depending on the time to a future state to be predicted.

Further, according to an embodiment of the present invention, by using this calculation result, it is possible to calculate an optimal control law in consideration of a future state in an infinite time ahead.

Further, according to an embodiment of the present invention, a route optimization method considering all routes that may exist in the field of automatic design, a pricing method considering a distant future state in the field of finance, and a metabolic pathway optimization method considering all routes within a modelable range in the field of bioengineering can be performed.

Hereinafter, an embodiment will be described with reference to the drawings.

<FIG> is a configuration diagram showing an example of a processing device <NUM> implemented with a high-speed estimation method for a long-term future state according to a first embodiment of the present invention. The processing device <NUM> includes an input device <NUM>, a data reading device <NUM>, an output device <NUM>, a storage device <NUM>, and an arithmetic device <NUM> as main elements.

Of these, the input device <NUM> is a part that receives an instruction from an operator, and includes a button, a touch panel, and the like.

The data reading device <NUM> is a part that receives data from the outside of the processing device <NUM>, and includes a CD drive, a USB terminal, a LAN cable terminal, a communication device, and the like.

The output device <NUM> is a device that outputs instruction information to an operator, a read image, a read result, and the like, and includes a display and a communication device.

The above configurations are standard ones, and any or all of the input device <NUM>, the data reading device <NUM>, and the output device <NUM> may be connected to the outside of the processing device <NUM>.

The storage device <NUM> is a part that stores various types of data, and includes a model storage unit <NUM> and a future state prediction result storage unit <NUM>. Of these, the model storage unit <NUM> is a part that stores a model that simulates the behavior of an object or a phenomenon that is a target of predicting a future state in the processing device <NUM>. Further, the future state prediction result storage unit <NUM> is a part that stores a calculation result of a future state prediction arithmetic unit <NUM> described later. Details of the storage device <NUM> will be described later, and only schematic functions are described here.

The arithmetic device <NUM> processes data input from the input device <NUM> and the data reading device <NUM> and data stored in the storage device <NUM>, outputs a result of the processing to the output device <NUM> or records the result in the storage device <NUM>, and includes processing units (an input control unit <NUM>, the future state prediction result storage unit <NUM>, and an output control unit <NUM>).

The input control unit <NUM> is a part that divides data input from the input device <NUM> or the data reading device <NUM> into commands, models, and the like, and transfers the data to each unit of the storage device <NUM> and the arithmetic device <NUM>.

The future state prediction arithmetic unit <NUM> calculates an attenuation type state transition matrix from model data stored in the model storage unit <NUM>, and records the matrix in the future state prediction result storage unit <NUM>.

The output control unit <NUM> is a part that outputs data stored in the storage device <NUM> to the output device <NUM>. When an output destination is a screen or the like, it is preferable that a result is output each time reading operation is performed. When an output destination is a communication destination or the like, output processing may be performed each time a state transition probability matrix is updated or calculation of the future state prediction arithmetic unit <NUM> is performed, or may be performed in such a manner that pieces of data of a plurality of times are collected, pieces of data are collected at predetermined time intervals, or the like.

Hereinafter, details of processing executed by using the processing device <NUM> of <FIG> will be described. Note that, in description hereinafter, an object or a phenomenon that is a target of predicting a future state will be referred to as a simulation target in the present invention. Examples of the simulation targets include behavior of machines and living things, natural and physical phenomena, chemical reactions, changes in money and prices, and changes in consumer demand. However, the simulation targets are not limited to these examples in the present invention.

Input of a model in the present invention is a state of a simulation target and elapse of time, influencing factors such as operation, disturbance, and the like, and output is a state of the simulation target after being influenced by the influencing factors. In the present invention, this model will be referred to as a state transition model. Models such as a state transition model are stored in the model storage unit <NUM> of <FIG>. Further, a state transition model expresses states of a simulation target and its surrounding environment in infinite time or at infinite step ahead in finite state space in a form of probability density distribution.

As an example of a storage format of a state transition model in the model storage unit <NUM>, for example, a state transition probability matrix, a neural network, a radial basis function network, or a matrix expressing a weight of a neural network or a radial basis function network can be considered. However, the present invention does not limit the model storage format of a simulation target to these examples.

<FIG> shows an example of a case where a format of a model stored in the model storage unit <NUM> is a state transition probability matrix T. <FIG> shows a state si (i = <NUM>, <NUM>,. , n) of a transition source and a state sj (j = <NUM>, <NUM>,. , n) of a transition destination in a vertical and horizontal matrix. State transition probability P (sj | si) is numerically displayed in the matrix. The transition probability matrix T is generally a kind of a model that simulates a motion characteristic and a physical phenomenon of a control object, and is a function or matrix that stores transition probability between all states. Here, the row of the table is the state si (i = <NUM>, <NUM>,. , n) of a transition source, the column is the state sj (j = <NUM>, <NUM>,. , n) of a transition destination, and the element Tij is the probability P (sj | si) that the state transits from the state si to the state sj when the set time interval Δt (or step) elapses.

Regarding a simulation target to which the present invention is applied, when states of the simulation target and its surrounding environment in the infinite time or an infinite step ahead in the form of probability density distribution, calculation time may be independent of any one or more of a distance to a future state to be estimated, time, and a step. In a case where the state transition probability P (sj | si) does not depend on time, a step τ indicating an amount and the number of times an influencing factor interferes with the simulation target may be used instead of time t.

<FIG> shows that, when a focus is placed on s1 of the state si of a transition source, the probability P (s1 | s1) of becoming s1 of the state sj of a transition destination after the elapsed time Δt is <NUM>, the probability P (s2 | s1) of becoming s2 is <NUM>, and the probability P (s3 | s1) of becoming s3 or more is <NUM>. Similarly, when a focus is placed on s2, the probability P (s1 | s2) of becoming s1 of the state sj of a transition destination after the elapsed time Δt is <NUM>, the probability P (s2 | s2) of becoming s2 is <NUM>, the probability P (s3 | s2) of becoming s3 is <NUM>, and the probability P (s4 | s1) of becoming s4 is <NUM>. Note that the table of <FIG> shows the states of a transition source and the probability of a moving destination of movement after transition, so this table can be regarded as a table of probability density distribution. The probability density distribution shows, for example, a mountain shape.

Note that, in the above description, for the state transition probability matrix T, the table Tij showing only one cross section before and after the elapsed time Δt is illustrated. However, in reality, tables at intervals of the elapsed time Δt are continuously present, and the state transition probability matrix T that is a model stored in the model storage unit <NUM> is formed. A table after the elapsed time Δt of the table Tij is Ti + <NUM>, j + <NUM>, and a table after the elapsed time Δt is Ti + <NUM>, j + <NUM>.

In the example of <FIG>, a state s is handled as discrete space in which the whole is divided by n into ranges. However, by using a neural network, a radial basis function network, and the like, the state s can be handled as continuous space. Further, in a case where a neural network, a radial basis function network, or the like is used, the state transition probability matrix T may be substituted with a matrix having a weight coefficient of an input signal entering a neuron or a weight coefficient of a basis function as an element value.

Returning to <FIG>, the future state prediction result storage unit <NUM> is a part that stores a calculation result of the future state prediction arithmetic unit <NUM>. In the present invention, data stored in the future state prediction result storage unit <NUM> will be referred to as an attenuation type state transition matrix. The attenuation type state transition matrix and its calculation method will be described later.

The future state prediction arithmetic unit <NUM> calculates an attenuation type state transition matrix from model data recorded in the model storage unit <NUM>, and records the matrix in the future state prediction result storage unit <NUM>. An example of a method of calculating the attenuation type state transition matrix is shown in Equation (<NUM>) below. Note that, in the example of Equation (<NUM>), the state transition probability matrix T is assumed as a storage format of a model in the model storage unit <NUM>. [Equation <NUM>] <MAT>.

In Equation (<NUM>), D is an attenuation type state transition matrix, and γ is a constant of <NUM> or more and less than <NUM> referred to as an attenuation rate. Further, Tk is a function (or matrix) that stores transition probabilities between all states when the time of Δt × k elapses.

<FIG> is a diagram schematically showing the processing of Equation (<NUM>), in which a plurality of state transition probability matrices Tij for the elapsed time Δt in <FIG> are multiplied by a weighting coefficient γ that attenuates at each of the elapsed time Δt, and the total of them is calculated. Note that, in <FIG>, the probability distribution showing the state si of a transition source and the state sj of a transition destination in a plurality of the state transition probability matrices Tij is grasped as, for example, a mountain-shaped characteristic group.

As described above, the attenuation type state transition matrix D is the sum of the state transition probability matrix T after time Δt elapses to the state transition probability matrix T∞ after time Δt × ∞ elapses, and is also a matrix that stores statistical proximity between all states. Further, in order to reduce a weight for a state transition to a distant future, a large amount of the attenuation rate γ is multiplied according to the elapsed time.

With Equation (<NUM>), which requires calculation from the state transition probability matrix T at a current time point to the state transition probability matrix T∞ after time ∞ elapses, calculation within real time is difficult. In view of the above, the present invention is characterized in that Equation (<NUM>) is converted into Equation (<NUM>) below. In short, Equation (<NUM>) is for performing calculation equivalent to the series of the state transition probability matrix when states of the simulation target and its surrounding environment in infinite time or an infinite step ahead are estimated in the form of probability density distribution. [Equation <NUM>] <MAT>.

In Equation (<NUM>), E is a unit matrix. Equation (<NUM>) is a calculation formula equivalent to Equation (<NUM>). By converting the calculation of the sum of the state transition probability matrix T in Equation (<NUM>) to the state transition probability matrix T∞ into an inverse matrix of (E-γT) in Equation (<NUM>), the same calculation result as Equation (<NUM>) is obtained in finite time. Here, in a case where the state transition probability matrix T is not linearly independent, a pseudo inverse matrix may be used. Further, instead of the attenuation type state transition matrix D, a matrix obtained by normalizing the attenuation type state transition matrix in each row may be used.

As described above, the present invention makes it possible to calculate the state transition probability after time Δt × k by calculating Tk by using a model that simulates the behavior of a simulation target as a state transition model. Further, the sum from the state transition probability matrix T after the lapse of time Δt to the state transition probability matrix T∞ after time Δt×∞ elapses is taken, and weighting is performed with the attenuation rate γ according to the elapsed time, so that the state transition probability in consideration of elapse of time Δt×∞ can be calculated within finite time.

<FIG> is a diagram showing a process of processing performed by the processing device <NUM>.

First, by processing of processing step S1201, data regarding a model of a simulation target is input from the data reading device <NUM> based on a command from the input control unit <NUM>, and the data is recorded in the model storage unit <NUM>.

Next, by processing of processing step S1202, the data regarding the model of a simulation target recorded in the model storage unit <NUM> is transferred to the future state prediction arithmetic unit <NUM>, the attenuation type state transition matrix D is calculated based on Equation (<NUM>), and its result is recorded in the future state prediction result storage unit <NUM>.

Finally, by processing of processing step S1203, the data recorded in a future state prediction result storage unit <NUM> is transferred to the output control unit <NUM> and output to the output device <NUM>.

<FIG> is a configuration diagram showing an example of the processing device <NUM> obtained by expanding the processing device <NUM> according to the first embodiment to optimize model-based control. A simulation target in the processing device <NUM> is the behavior of the control object and its surrounding environment, and a model stored in the model storage unit <NUM> also simulates the behavior of the control object and its surrounding environment. As described above, in a second embodiment, a case where a simulation target includes a control object is assumed.

The processing device <NUM> includes the input device <NUM>, the data reading device <NUM>, the output device <NUM>, the storage device <NUM>, and an arithmetic device <NUM> as main elements.

The output device <NUM> is a device that outputs instruction information to an operator, a read image, a read result, and the like, and includes a display, a CD drive, a USB terminal, a LAN cable terminal, a communication device, and the like.

The storage device <NUM> includes the model storage unit <NUM>, the future state prediction result storage unit <NUM>, a reward function storage unit <NUM>, and a control law storage unit <NUM>. Of these, the future state prediction result storage unit <NUM> has substantially the same function as that of the first embodiment.

There is a case where the model storage unit <NUM> has the same function as that of the first embodiment, and there is also a case where the behavior of a simulation target changes not only in a state but also in an operation amount in control. In a case where the behavior of a simulation target changes according to an operation amount, the attenuation type state transition matrix can be calculated as in the first embodiment by adding information of the operation amount to the model.

The reward function storage unit <NUM> is a part that stores control targets such as a target position and a target speed in the form of fa unction, a table, a vector, a matrix, and the like. In the present invention, a function, a table, a vector, a matrix, and the like having information of this control target will be referred to as a reward function R. <FIG> shows an example of a case where a reward function is in a vector format. In <FIG>, the reward function R is represented by a numerical value for each state ID of a transition source. According to this figure, the state s is handled as discrete space in which the whole is divided into n ranges, and the goal is make a transition from an initial state to a state s3. Here, an element value of a target vector is <NUM> in the state s3 and <NUM> in the other states. In the present invention, an element value of this vector and a value of the reward function R are referred to as rewards. As a reward for control, a desired value or an objective function at the time of reinforcement learning in AI is exemplified.

Returning to <FIG>, the control law storage unit <NUM> is a part that stores an optimal control law for a control target. An example of the control law stored in the control law storage unit <NUM> is shown in <FIG>. In <FIG>, an operation amount ID is represented by a numerical value for each state ID (si) of a transition source. According to this figure, the state si is handled as discrete space in which the whole is divided into n ranges, and an optimal operation amount ac (c = <NUM>, <NUM>,. , m) is stored for a range of each state. A method of calculating an optimal operation amount a will be described later.

Returning to <FIG>, the arithmetic device <NUM> processes data input from the input device <NUM> and the data reading device <NUM> and data stored in the storage device <NUM>, outputs a result of the processing to the output device <NUM> or records the result in the storage device <NUM>, and includes processing units described below.

An input control unit <NUM> is a part that divides data input from the input device <NUM> or the data reading device <NUM> into commands, models, and the like, and transfers the data to each unit of a storage device and an arithmetic device.

A future state prediction arithmetic unit <NUM> is equivalent to the future state prediction arithmetic unit <NUM> of the first embodiment. Further, an output control unit <NUM> is also equivalent to the output control unit <NUM> of the first embodiment.

A control law arithmetic unit <NUM> calculates an optimal control law (optimal operation amount a) from the attenuation type state transition matrix D recorded in the future state prediction result storage unit <NUM> and the reward function R recorded in the reward function storage unit <NUM>, and records the control law in the control law storage unit <NUM>.

An example of a method of calculating an optimal control law is shown below. In the present example, calculation is performed in three stages below in order to obtain an optimum control law.

Stage <NUM>: First, a function for storing closeness (or a statistical index indicating the ease of transition) between each of the states s and a state sgoal as a target in the reward function R is calculated. In the present invention, this function is referred to as a state value function V. Further, the state value function V may be stored in the form of a table, a vector, a matrix, or the like in addition to a function, and a storage format is not limited in the present invention. An example of a calculation method of the state value function V is shown in Equation (<NUM>) below. [Equation <NUM>] <MAT>.

As shown in Equation (<NUM>), the state value function V is the product of the attenuation type state transition matrix D and the reward function R. For example, as shown in <FIG> and <FIG>, in a case where the attenuation type state transition matrix D is an n-by-n matrix and the reward function R is an n-dimensional vector, the state value function V is an n-dimensional vector as shown in <FIG>. An element value of the state value function V is higher in a state where transition to the state sgoal as a target is more likely. In the present invention, this element value is referred to as a value. Further, the state value function V of the present invention is equivalent in value to the definition of a state value function in a reinforcement learning method.

Stage <NUM>: Next, using the state value function V, the state sj* that is most likely to make a transition to the state sgoal as a target among the states sj of a transition destination to which a transition can be made from the state si of a transition source is calculated. An example of the calculation method of the state sj* is shown in Equation (<NUM>) below. [Equation <NUM>]
<MAT>.

Here, T(si, sj) is an element value in the row si and the column sj in the state transition probability matrix T. <FIG> shows an example of a calculation result of Equation (<NUM>). In <FIG>, the state ID(sj) of a transition destination is represented for each state ID(si) of a transition source. According to <FIG>, in a case where a state of a transition source is the state s1, states that may be a transition destination may be two, the state s1 and the state s2, in the state transition probability matrix T (<FIG>). Of these two states, the state s2 has a higher value in the state value function V. For this reason, in the example of <FIG>, the state s2 is stored as a state of a transition destination of the state s1 of a transition source.

Stage <NUM>: In the final stage, the operation amount a required to make a transition from each of the states si of a transition source to the state sj* obtained in Stage <NUM> is calculated. The operation amount a can be calculated, for example, by obtaining an inverse model of the model storage unit <NUM> (a model in which the state si and the state sj* of a transition source are input and the corresponding operation amount a is output). As a calculation result of Stage <NUM>, for example, a control law as shown in <FIG> is obtained.

Calculation of a value with Equation (<NUM>) as described above enables evaluation of the likeliness of transition to sgoal of each state, Equation (<NUM>) enables identification of the state sj* that is most likely to make a transition to sgoal among the states to which transition can be made by elapse of time Δt, and the inverse model enables identification of the operation amount a for making a transition to the state sj*.

Returning to <FIG>, a model update unit <NUM> modifies model data based on update data when the update data of the model data recorded in the model storage unit <NUM> is input from the data reading device <NUM>, and records the modified model data in the model storage unit <NUM>.

First, in processing step S1301 of <FIG>, data regarding a model of a simulation target and data regarding the reward function R are input from the data reading device <NUM> based on a command from the input control unit <NUM>, and the pieces of the data are recorded in the model storage unit <NUM> and the reward function storage unit <NUM>.

Next, in processing step S1302, the data regarding the model of a simulation target recorded in the model storage unit <NUM> is transferred to the future state prediction arithmetic unit <NUM>, the attenuation type state transition matrix D is calculated based on Equation (<NUM>), and its result is recorded in the future state prediction result storage unit <NUM>.

Next, in processing step S1303, the attenuation type state transition matrix D recorded in the future state prediction result storage unit <NUM> and the reward function R recorded in the reward function storage unit <NUM> are transferred to the control law arithmetic unit <NUM>, an optimal control law is calculated, and its result is recorded in the control law storage unit <NUM>.

Next, in processing step S1304, pieces of the data recorded in the future state prediction result storage unit <NUM> and the control law storage unit <NUM> are transferred to the output control unit <NUM> and output to the output device <NUM>.

Next, in processing step S1305, a control object determines whether or not to finish the control. In a case where the control is to be continued, the processing proceeds to processing step S1306, and in a case where the control is to be finished, the process is also finished.

Next, in processing step S1306, the control object calculates the operation amount a based on the control law sent from the output device <NUM> to the control object, and executes operation.

Next, in processing step S1307, the control object transmits states of the control object and its surrounding environment measured before and after the operation is executed to the data reading device <NUM>.

Next, in processing step S1308, the input control unit <NUM> determines whether or not the data reading device <NUM> receives data of states of the control object and its surrounding environment measured before and after the execution of the operation. In a case where the data is received, the processing proceeds to processing step S1309, and in a case where the data is not received, the processing returns to processing step S1305.

In processing step S1309, in a case where the data reading device <NUM> receives data of states of the control object and its surrounding environment measured before and after the execution of the operation in the processing of processing step S1308, the received data and model data recorded in the model storage unit <NUM> are transferred to the model update unit <NUM>, and updated model data is recorded in the model storage unit <NUM>. After the above, the processing proceeds to processing step S1302.

<FIG>, and <FIG> are examples of a screen displayed on the output device <NUM> in the first and second embodiments.

<FIG> shows the state transition probability matrix T displayed on a screen as an example of model data recorded in the model storage unit <NUM>. In the figure, the state transition probability matrix T is displayed on a screen in a matrix format with the movement source state si and the movement destination state sj as an example of a storage format of a model, and element values of the matrix may be updated from the present screen through the input device <NUM>.

<FIG> shows an example of a case where the attenuation type state transition matrix D stored in the future state prediction result storage unit <NUM> is displayed on a screen. In the figure, the attenuation type state transition matrix D is displayed on a screen in a matrix format with the movement source state si and the movement destination state sj. Note that, instead of the attenuation type state transition matrix D, a matrix obtained by normalizing the attenuation type state transition matrix D in each row may be displayed on the screen.

<FIG> is an example of a case where transition probability distribution P is displayed as data obtained by processing model data stored in the model storage unit <NUM>. On a screen, the transition probability P is displayed with the state sj of a transition destination in the horizontal axis.

Further, a state setting section <NUM> of the transition source si, a graph output section <NUM>, and an elapsed time setting section <NUM> are formed on the screen. In the state setting section <NUM> of the transition source si, a state of a transition source is input through the input device <NUM>. Here, s3 is shown as an example and is assumed to be input. Specific examples of states of a transition source at the time of input s3 are a temperature, pressure, and a flow rate of a simulation target. Here, when a3 is input from a state ID button, the state ID is converted into values of a temperature, pressure, and a flow rate of the simulation target.

This conversion can be realized by creating in advance a correspondence table of an ID, a temperature, pressure, and a flow rate as shown in <FIG>. In this example, since s3 is selected, the temperature of <NUM> degrees, the pressure of <NUM> MPa, and the flow rate of <NUM> t/h of a simulation target are displayed.

Further, in the graph output section <NUM> of <FIG>, when a graph output button is pressed, a diagram in which an element value of the attenuation type state transition matrix of a row corresponding to a state ID set in advance is graphed is displayed on a screen.

Further, in the elapsed time setting section <NUM>, an appropriate time interval Δt can be set within a range of maximum and minimum time set in advance. By designating the time range, the attenuation type state transition matrix D within the designated time range is displayed. The attenuation type state transition matrix D in a case where the time range is limited is obtained, for example, by Equation (<NUM>) described below. [Equation <NUM>] <MAT>.

Here, tmin is a minimum value of the designated time range, tmax is a maximum value of the designated time range, and Δt is a time interval set in advance. Further, by adjusting the scrolling on the right side of the screen, it is possible to display the attenuation type state transition matrix D of when a specific time elapses on the screen. The attenuation type state transition matrix D of when a designated time elapses is obtained, for example, by Equation (<NUM>) described below. [Equation <NUM>] <MAT>.

Here, tp is a designated elapsed time. In the example of the screen of <FIG>, the attenuation type state transition matrix D in a case where tp is <NUM> seconds is displayed. With a graph displayed on the present screen, it is possible to check the probability p of transition from a state of a transition source to each state within the designated elapsed time tp or a time range from tmin to tmax.

<FIG>, <FIG>, <FIG>, and <FIG> are examples of screens displayed on the output device <NUM> in the second embodiment.

<FIG> is an example of a case where tables of the reward function R recorded in the reward function storage unit <NUM>, a control law recorded in the control law storage unit <NUM>, a state value function calculated in the control law arithmetic unit <NUM>, and the state sj* for the state si of a transition source calculated in the control law arithmetic unit <NUM> are displayed on the screen. An element value of the reward function may be updated from the present screen through the input device <NUM>. Further, by bringing a mouse cursor close to the state ID in the screen, values of a temperature, pressure, and a flow rate corresponding to the state ID may be displayed on the screen based on an example of the table as shown in <FIG>.

<FIG> is an example, in which a graph <NUM> that predicts how states of a control object and its surrounding environment change as time elapses is displayed on a screen by control using a control law recorded in the control law storage unit <NUM>. After an initial state is set on a screen, a behavior start button <NUM> is pressed, so that transition of a state with respect to elapse of time can be checked. As in the example, in a case where the state is associated with three of a temperature, pressure, and a flow rate, a button <NUM> for switching the display may be set to display one by one.

<FIG> is an example of a screen displayed on a screen when model data recorded in the model storage unit <NUM> is updated in processing step S1309 of <FIG> and the screen of <FIG>. On the present screen, update content of a model (before and after update), update content of a control law accompanying update of a model (before and after update), a change in a state transition prediction result due to elapse of time equivalent to that of <FIG> due to update of a control rule (before and after update), and buttons <NUM>, <NUM>, and <NUM> for designating whether or not model update is possible are displayed. When the model update permission button <NUM> is pressed, processing step S1309 in <FIG> is executed, and when the decline button <NUM> is pressed, <NUM> is not executed. Further, when a hold button processing step S is pressed, processing step S1309 is not executed. However, the same screen can be read again even after the hold button <NUM> is pressed.

<FIG> is an example of a screen different from that of <FIG> in a method of displaying update content (before and after update) of a model. In <FIG>, a state of a transition source as an update target, a state after transition, and its transition probability are displayed, whereas in <FIG>, state transition matrices before and after update are displayed.

Claim 1:
A future state estimation device comprising:
a model storage unit (<NUM>) that stores a model for simulating a simulation target and a surrounding environment of the simulation target;
a future state prediction result storage unit (<NUM>) that stores information obtained by estimating future states of the simulation target and a surrounding environment of the simulation target in infinite time or a time step ahead within finite space in a form of probability density distribution; and
a future state prediction arithmetic unit (<NUM>, <NUM>) that performs calculation equivalent to a series using a model for simulating future states of the simulation target and a surrounding environment of the simulation target in a form of probability density distribution,
wherein a format of the model stored in the model storage unit (<NUM>) is a state transition probability matrix T, and
wherein the sum from the state transition probability matrix T after a lapse of time Δt to a state transition probability matrix T∞ after time Δt×∞ elapses is taken, and weighting is performed with an attenuation rate γ according to the elapsed time Δt×∞, so that the state transition probability in consideration of elapse of time Δt×∞ is calculated within finite time,
wherein the simulation target includes a control object, the future state estimation device further comprising a control law arithmetic unit (<NUM>) that calculates an operation amount of the control object by using an estimation result of states of the simulation target and a surrounding environment of the simulation target in infinite time or an infinite step ahead estimated in the future state prediction arithmetic unit (<NUM>, <NUM>),
wherein the control object is a plant, or an automobile, and
wherein the states of the simulation target include a target position, speed, temperature, pressure and flow rate.