Patent ID: 12246482

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments will be described in detail with reference to the attached drawings. Note, the following embodiments are not intended to limit the scope of the claimed invention. Multiple features are described in the embodiments, but limitation is not made to an invention that requires all such features, and multiple such features may be combined as appropriate. Furthermore, in the attached drawings, the same reference numerals are given to the same or similar configurations, and redundant description thereof is omitted.

FIGS.1A and1Beach show the arrangement of a control apparatus1100according to the first embodiment.FIG.1Ashows a state before a processor1010comes into contact with a workpiece1001, andFIG.1Bshows a state in which the processor1010has come into contact with the workpiece1001. The control apparatus1100can be formed to process the workpiece1001by, for example, pressing the processor (object)1010against the workpiece (processing object)1001at a desired force. The control apparatus1100controls the force applied to the processor (object)1010. A driving direction of the processor1010will be described as the X-axis hereinafter.

The control apparatus1100can include, for example, the processor1010, a driver1015for driving the processor1010in a predetermined direction (X-axis direction), a measurement unit1025for measuring the displacement of the processor1010, and a controller1035for outputting a manipulation amount (command value) to the driver1015. In addition, the control apparatus1100can include a feeder (member)1050for supplying energy (for example, electric power) and a fluid (a gas and/or a liquid) necessary for the processing to the processor1010. The feeder1050is directly or indirectly connected to the processor1010, and a disturbance force can be applied to the processor1010via the feeder1050. The driver1015can be formed to apply a force to the processor1010in the X-axis direction. The driver1015can include, for example, an actuator such as a linear actuator or the like. For example, a linear encoder or the like can be used as the measurement unit1025. The controller1035can be formed by, for example, a PLD (the abbreviation of a Programmable Logic Device) such as an FPGA (the abbreviation of a Field Programmable Gate Array), an ASIC (the abbreviation of an Application Specific Integrated Circuit), a general-purpose computer embedded with a program, a dedicated computer, or a combination of all or some of these components.

The controller1035can have a mode for performing position feedback control on the processor1010set as a control object, and a mode for performing force feedback control on the processor1010set as the control object. The controller1035can perform position feedback control on the processor1010when the processor1010is to be moved near the workpiece1001, and perform force feedback control on the processor1010when the workpiece1001is to be processed by the processor1010.

FIGS.2A and2Bare block diagrams each showing the arrangement and the operation of the control apparatus1100when force feedback control is performed. Here,FIG.2Ashows a case in which a disturbance force Fd from the feeder1050is not applied to the processor1010, andFIG.2Bshows a case in which the disturbance force Fd from the feeder1050is applied to the processor1010.FIG.2Awill be described first. Reference symbol Fr indicates a target value (to be referred to as a force target value hereinafter) of a force applied to the processor1010, reference symbol Gc indicates a control compensator, reference symbol Ki indicates a current gain, and reference symbol Kt indicates a thrust constant of the driver1015. Reference symbol G0indicates a transfer function of the force of the processor1010to a displacement, and reference symbol X0indicates a displacement (a position measurement value obtained by the measurement unit1025) of the processor1010. An output F output by the driver1015is a sum of a driving force F0necessary for driving the processor1010and a processing force Fa necessary for processing the workpiece1001.

To control the processing force Fa in accordance with the force target value Fr, the processing force Fa needs feedback. However, due to the terms of processing, the control apparatus1100according to the first embodiment cannot directly measure the processing force Fa. Hence, inFIG.2A, the output F obtained by multiplying a manipulation voltage Vc with the thrust constant Kt and the current gain Ki is supplied as feedback. The manipulation voltage Vc can be understood to be a manipulation amount or a command value. Since the output F is obtained by adding the processing force Fa and the driving force F0, the target value of the driving force F0needs to be added to the force target value Fr. Hence, inFIG.2A, a target driving force F0ris added to the force target value Fr. The target driving force F0rmay be obtained by multiplying the displacement X0of the processor1010by 1/G0as shown inFIG.2Aor may be prepared in advance as a table corresponding to the displacement X0. This will allow the processing force Fa to be controlled according to the force target value Fr.

In reality, however, disturbance force Fd will be applied to the processor1010as shown inFIG.2Bdue to the influence of the feeder1050shown inFIGS.1A and1B, the processing force Fa will be Fa′=Fa+Fd, and the workpiece1001cannot be processed by the processing force Fa according to the force target value Fr. The disturbance force Fd depends on the displacement and the speed of the processor1010in the first embodiment. Hence, as shown inFIG.2B, letting Gd be a transfer function from the displacement of the processor1010to the disturbance force Fd and X0be the displacement of the processor1010, the disturbance force Fd will be expressed as Fd=Gd×X0.

Therefore, in the first embodiment, a disturbance estimator (estimator) Gde will be used as a disturbance observer as shown inFIG.3. The disturbance estimator Gde estimates an estimated disturbance force Fde, as an estimation value of the disturbance force Fd applied to the processor1010, based on information related to the state of the processor1010. The disturbance estimator Gde can estimate the estimated disturbance force Fde applied to the processor1010based on, for example, the displacement X0of the processor1010measured by the measurement unit1025. The processor1010will be able to output the processing force Fa according to the force target value Fr by subtracting a voltage corresponding to the estimated disturbance force Fde from the manipulation voltage Vc. A disturbance estimator1305uses a plurality of parameter values determined in advance to obtain the estimated disturbance force Fde based on the information related to the state of the processor1010. The plurality of parameter values are determined by the determiner1350. Note that reference numerals301to307denoted inFIG.3indicate a correspondence relationship with the components shown inFIG.13to be described below.

FIG.13shows an example of the physical arrangement of the control apparatus1100corresponding toFIG.3. The controller1035can include a target value generator1301, a compensator1302, a corrector1303, a dynamic stiffness multiplier1304, and the disturbance estimator1305. The target value generator1301generates the force target value Fr. The compensator1302obtains the manipulation voltage Vc from the sum of the force target value Fr and the target driving force F0r. The compensator1302corresponds to a block formed by the control compensator Gc (301), the current gain Ki (302), and the thrust constant Kt (303) ofFIG.3, and its gain is Gc/(1+GcKtKi). The corrector1303corresponds to a block formed by 1/Ki (304), 1/kt (305), and the operator (subtractor)306ofFIG.3. The dynamic stiffness multiplier1304corresponds to 1/G0(307) ofFIG.3. The corrector1303corrects, based on the estimated disturbance force Fde, the manipulation amount (command value) provided to the driver1015to reduce the influence of the disturbance force Fd. The disturbance estimator1305corresponds to the disturbance estimator Gde (306) ofFIG.3.

The disturbance estimator Gde (306) will be described next. The disturbance estimator Gde has been obtained by modeling the transfer function Gd of a disturbance shown inFIGS.2B and3. The transfer function Gd of a disturbance represents a disturbance generated by the feeder1050shown inFIGS.1A and1B. Since the disturbance generated by the feeder1050is generated based on the displacement and the speed of the processor1010, it can be modeled as a three element model using a spring (stiffness) and a damper (attenuation). Hence, the disturbance estimator Gde can also be modeled as a three element model.FIG.4shows a disturbance estimation model using one three element model. Letting X0be the displacement of the processor1010and X1be a displacement of a spring K2, the estimated disturbance force Fde can be expressed as
Fde=−K1·(X0−X1)  (1)
Fde=−K2·X1−C2·{dot over (X)}1  (2)
Equation (1) and equation (2) can be deformed as

X⁢1.=-(K⁢1+K⁢2)C⁢2·X⁢1+K⁢1C⁢2·X⁢0(3)
Fde=K1·X1−K1·X0  (4)

respectively.

Since equation (3) represents a state equation and equation (4) represents an output equation, the disturbance estimation model shown inFIG.4can be expressed as a state space shown inFIG.5. A state matrix A, an input matrix B, an output matrix C, and a direct matrix D ofFIG.5can be expressed as

A=[-(K⁢1+K⁢2)C⁢2](5)B=[K⁢1C⁢2](6)
C=[K1]  (7)
D=[−K1]  (8)
respectively.

Hence, the disturbance estimator Gde can be expressed by the state space shown inFIG.5, and the estimated disturbance force Fde can be obtained when the displacement X0of the processor1010is input to the disturbance estimator Gde.

In addition, solving a differential equation expressed by equations (1) and (2), the estimated disturbance force Fde can be expressed as a function of a time t by

Fde=-K⁢1·K⁢2K⁢1+K⁢2·X⁢0·(1-e-tT)(9)
where T is a time constant of the disturbance estimation model and the disturbance estimator Gde shown inFIG.4and is expressed by

T=C⁢2K⁢1+K⁢2(10)

In a case in which the feeder1050is formed by a plurality of elements with different time constants from each other, a plurality of three element models will need to be combined. In the first embodiment, a disturbance estimation model obtained by combining three three-element models will be used as shown inFIG.6. In the disturbance estimation model shown inFIG.6, the state matrix A, the input matrix B, the output matrix C, and the direct matrix D of the disturbance estimator Gde are expressed as

A=[-(K⁢11+K⁢12)C⁢12000-(K⁢21+K⁢22)C⁢22000-(K⁢31+K⁢32)C⁢32](11)B=[K⁢11C⁢12K⁢21C⁢22K⁢31C⁢32](12)
C=[K11K21K31]  (13)
D=[−K11+K21+K31]  (14)
respectively.

In addition, the disturbance force Fd applied to the processor1010can be calculated, in a state in which the processor1010is controlled by position feedback control, based on the displacement X0of the processor1010and the output F output from the driver1015at this time.FIG.7shows a block diagram showing the arrangement and the operation of the control apparatus1100in a state in which the position feedback control of the processor1010is performed. The control apparatus1100shown inFIG.7includes an acquirer1450for acquiring the disturbance force Fd. Reference symbol Xr represents a displacement target value (position command value) of the processor1010. Since the output F output by the driver1015will be F=F0−Fd when position feedback control is performed on the processor1010, the driving force F0will be applied to the processor1010even if the disturbance force Fd is applied. Hence, the processor1010can be positioned according to the displacement target value Xr.

Therefore, the disturbance force Fd can be obtained by subtracting the driving force F0from the output F (=F0−Fd) and multiplying the obtained result by −1. The driving force F0can be obtained by multiplying the displacement X0of the processor1010by 1/G0.

When acquiring the disturbance force Fd, the acquirer1450executes position feedback control of driving the driver1015based on the difference between the displacement target value Xr and the displacement X0so that the displacement X0(measured value) of the processor1010will match the displacement target value Xr for commanding the position of the processor1010. In this position feedback control, the acquirer1450acquires the disturbance force Fd based on the difference between the displacement target value Xr and the displacement X0and the displacement X0(measured value). In the example ofFIG.7, the output F is operated based on the difference between the displacement target value Xr and the displacement X0, the driving force F0is operated based on the displacement X0(measured value), and the disturbance force Fd is operated from the output F and the driving force F0. Note that reference numerals701to708denoted inFIG.7indicate a correspondence relationship with the components shown inFIG.14to be described below.

FIG.14shows an example of the physical arrangement of the control apparatus1100corresponding toFIG.7. The controller1035can include a target value generator1401, a compensator1402, a dynamic stiffness multiplier1403, and operators1404,1405, and1406. The target value generator1401generates the displacement target value Xr (position command value). The operator1406operates the difference (deviation of displacement) between the displacement target value Xr and the displacement X0. The compensator1402obtains the driving force F (=F0−Fd) from the difference (deviation of displacement) between the displacement target value Xr and the displacement X0. The compensator1402corresponds to a block formed by the control compensator Gc (701), the current gain Ki (702), and the thrust constant Kt (703) ofFIG.7, and its gain is GcKtKi. The operator1404corresponds to the operator706ofFIG.7. InFIG.7, the driving force F (=F0−Fd) calculated by the block formed by the control compensator Gc (701), the current gain Ki (704), and the thrust constant Kt (705) is input to the operator706. The operator1405corresponds to the operator707ofFIG.7. The dynamic stiffness multiplier1403corresponds to 1/G0(708) ofFIG.7. The compensator1402, the dynamic stiffness multiplier1403, and the operators1404,1405, and1406form the acquirer1450for acquiring the disturbance force Fd. The determiner1350shown inFIG.13determines, by machine learning based on the disturbance force Fd acquired by the acquirer1450, a plurality of parameter values to be used by the disturbance estimator1305. The plurality of parameter values here will be used to obtain the estimated disturbance force Fde based on the information related to the state of the processor1010.

In order for the estimated disturbance force Fde, which is generated by the disturbance estimator1305, to be a value that correctly estimates the disturbance force Fd, the plurality of parameter values of the disturbance estimator Gde need to be correctly determined. If the disturbance estimation model is the three three-element model shown inFIG.6, the total number of springs and damper elements will be 9, and it will be very difficult to determine 9 parameter values by calculating on paper. Hence, in the first embodiment, the determiner1350will determine the plurality of parameter values of the disturbance estimator Gde by machine learning. For example, unsupervised learning can be used as the machine learning method. A function for executing machine learning can be embedded in, for example, the controller1035.

First, to execute machine learning, the controller1035will acquire data to be used for training. The position feedback control of the processor1010will be performed to collect, as data, the disturbance forces Fd of various kinds of displacements X0and the displacement X0of the processor1010. The collected data of the displacements X0will be set as displacement data data_X0and the collected data of disturbance forces Fd will be set as the disturbance force data data_Fd. The controller1035will input the displacement X0in the disturbance estimator Gde and collect the estimated disturbance force Fde output from the disturbance estimator Gde as estimated disturbance force data data_Fde. The controller1035will use data collected in this manner as the training data.

In unsupervised learning, it is desirable to make the estimated disturbance force Fde match the disturbance force Fd as much as possible. Hence, the controller1035can be formed to set, as a loss function L, a function for obtaining a mean squared error of the estimated disturbance force Fde and the estimated disturbance force Fde, and optimize the plurality of parameter values of the disturbance estimator Gde so as to minimize the loss function L. The loss function L can be expressed as

L=∑i=1n{Fde⁡(i)-Fd⁡(i)}2(15)
where i is a data number of the training data, and n is a total number of the training data.

A function that obtains, other than the mean squared error, for example, an average mean squared error, a mean absolute error, or a mean squared logarithmic error can be employed as the loss function.

An example of an optimization method to be employed in machine learning will be described next. For example, a gradient method can be used as the optimization method in machine learning.FIG.8Ashows an example of the procedure of the gradient method. First, in step SS1, the controller1035selects training data for learning from the collected training data. The training data for learning may be selected randomly from the collected training data or training data which is well characterized by a feature desired to be learned may be selected. Next, in step SS2, to reduce the value of the loss function, the controller1035obtains the gradient of each parameter value. Letting L be the value of the loss function, the gradient of each parameter value can be expressed as, for example.

∂L∂K,∂C=[∂L∂K⁢11⋮∂L∂C⁢32](16)

Next, in step SS3, the controller1035updates each parameter value by an infinitesimal amount in the direction of the gradient. The updating of each parameter value can be performed in accordance by

K⁢11=K⁢11-η·∂L∂K⁢11⋮C⁢32=C⁢32-η·∂L∂C⁢32(17)
where η represents a learning rate and is a parameter that determines the degree to which the parameter value is to be updated in one learning operation. In general, the learning rate η can be about 0.001 to 0.1.

In step SS4, the controller1035calculates the loss function L of the updated parameter values. In step SS5, the controller1035determines whether the termination condition has been satisfied. If the termination condition has been satisfied, the learning will end. Otherwise, the process will return to step SS1, and the learning will be continued.

FIG.8Bshows another example of the procedure of a gradient method which is obtained by partially changing the procedure of the gradient method shown inFIG.8A. In the procedure of the gradient method shown inFIG.8B, a step SS6in which an amount ΔL of change of the loss function L with respect to a stored loss function Lold is calculated and whether the amount ΔL of change is a threshold or less is determined has been added as shown by
ΔL=|L−Lold|  (18)
If the amount ΔL of change is not the threshold or less, the process will return to step SS3to update the parameter values, and the loss function L will be calculated in step SS4. The procedure ofFIG.8Bis a method in which the parameter values are continuously changed in the direction of the gradient once the gradient has been obtained and the direction of advancement has been determined, and the parameter values are updated by obtaining a new gradient when the reduction of the loss function L has stopped. This method can obtain an optimal solution that will minimize the loss function L more quickly than the procedure shown inFIG.8A.

The learning rate η can be gradually decreased in accordance with the degree of progress of the learning operation. This will allow the parameter values to be changed greatly in a case in which the parameter values are far from an optimal solution, and the parameter values to be changed finely in a case in which the parameter values have become close to the optimal solution. As result, the optimal solution can be reached more quickly and more accurately.

Although the gradient method is used as the optimization method of machine learning in the first embodiment, another optimization method may also be employed.

The processing for determining the parameter values of the disturbance estimator Gde may be executed while the control apparatus1100is operated (while the workpiece1001is processed) or may be executed by driving the processor1010in accordance with a dedicated driving profile during a non-operating time. In a case in which the parameter values of the disturbance estimator Gde are to be determined during the non-operating time, the processing for determining the parameter values of the disturbance estimator Gde can be executed, for example, periodically, at an arbitrary timing, or when a predetermined condition has been satisfied. For example, in a case in which an error of the estimated disturbance force Fde with respect to the disturbance force Fd has exceeded a predetermined ratio (for example, 5%), the parameter values of the disturbance estimator Gde can be updated or redetermined.

FIG.9shows a block diagram showing the arrangement of a force feedback control system that performed disturbance force correction according to the second embodiment. In place of a disturbance estimator Gde of the first embodiment, a disturbance estimator Gden formed by a neural network can be used in the second embodiment. The disturbance estimator Gden can include a input layer, a plurality of intermediary layers, and an output layer as exemplified inFIG.10. The total number of intermediary layers and the number of nodes of each layer can be set appropriately. The optimization of the parameter values of the disturbance estimator Gden can be performed by using, for example, a method similar to the method for determining the parameter values of the disturbance estimator Gde according to the first embodiment. Disturbance estimation can be performed more accurately by using the disturbance estimator Gden formed by a neural network than by using the disturbance estimator Gde which uses a three element model.

Parameter values of a disturbance estimator Gde according to the first embodiment will be determined by using reinforcement learning in the third embodiment. Although values of various kinds of formats can be used as a reward to be a target during learning in reinforcement learning, an example of learning by “a behavior evaluation function Qπ(s, a)” will be exemplified. The behavior evaluation function Qπ(s, a) can be expressed as follows.

Qπ(s,a)=Eπ⁢{∑k⁢γk⁢rt+1+k⁢❘"\[LeftBracketingBar]"st=s∖⁢at=a}(19)
where t represents a time, s represents a state,arepresents an action, π represents a policy, Eπ{ } represents an expected value under the policy π, r represents a reward, γ represents a discount factor of a future reward, and k represents a time until a future reward.

An agent of action, that is, a controller1035in this case will behave to maximize the behavior evaluation function Qπ(s, a) of equation (19). At this time, the agent will select and execute “an optimal behavior based on past experience” and “a search for a new behavior” to pursue further reward obtainment in accordance with the predetermined policy π. Since the equation is set as an expected value with consideration to a future reward, it will be possible to deal with a state in which a large reward can be obtained in the long term although the reward will decrease in the short term. As a result, the agent can learn the state and behavior that can maximize the behavior evaluation function Qπ(s, a).

The reward of the behavior evaluation function Qπ(s, a) shown in equation (19) can be defined as a reciprocal of the evaluation function of equation (15) as follows.

r=1L=1∑i=1n⁢(Fde-Fd)2(20)

Equation (20) is an equation that expects the reward r to increase if the disturbance force Fde estimated by the disturbance estimator Gde is close to the actual disturbance force Fd.

For example, the actionacan be defined as follows. The number of actionsawill be 9×2=18 as the number of parameters.

aj={aj⁢1,…,aj⁢1⁢8}aj⁢1:K⁢11→K⁢11+Δ⁢K⁢11aj⁢2:K⁢11→K⁢11-Δ⁢K⁢11⋮aj⁢1⁢7:C32→C⁢3⁢2+Δ⁢C⁢3⁢2aj⁢18:C32→C⁢3⁢2-Δ⁢C⁢3⁢2(21)
where j represents a number of epochs. The actionais a behavior that can increase or decrease the attenuation coefficient and the spring constant of the disturbance estimation model by an infinitesimal amount ΔK and an infinitesimal amount ΔC, respectively, to find parameter values optimal for the disturbance estimator Gde. The sizes of ΔK and ΔC can be obtained by multiplying the spring constant and the attenuation coefficient, respectively, of the initial state by a predetermined ratio, and this ratio can fall within a range of, for example, about 0.1% to 10%. Note that the behavior is not limited to the actionaas long as it is a behavior that allows for optimal parameter values of the disturbance estimator Gde to be searched, and another behavior may be defined.

The policy π can be defined as follows.

Policy π: an action aiwill be randomly selected, and the parameter values of the disturbance estimator Gde will be updated if the behavior evaluation function Qπ(s, a) increases.

Here, without consideration to the future reward, the behavior evaluation function Qπ(s, a) may be made equal to the reward r by setting the time k to the future reward to 0 and the discount factor γ of the future reward as 0.01.

The parameter values of the disturbance estimator Gde are updated based on a randomly selected action ajh(h is one of 1 to 18). If the behavior evaluation function Qπ(s, a) at this time has increased more than the behavior evaluation function Qπ(s, a) of an initial state s0, the parameter values of the disturbance estimator Gde of a state s1at t=1 will be updated. On the other hand, if the behavior evaluation function Qπ(s, a) at this time has not increased more than the behavior evaluation function Qπ(s, a) of the initial state s0, the parameter values of the disturbance estimator Gde of the state s1at t=1 will not be updated. In this manner, the parameter values of the disturbance estimator Gde are updated when the behavior evaluation function Qπ(s, a) is better than the previous behavior evaluation function Qπ(s, a). As a result, the parameter values of the disturbance estimator Gde can be optimized so that the behavior evaluation function Qπ(s, a) will increases as the time t advances under the policy π Since the time t is equivalent to the number of epochs in this case, the progression of the time t by one degree can be referred to as “performing one cycle of learning”.

The definition of the policy π is not limited to the example described above. An arbitrary condition such as not selecting a behavior that will shift to a search completed state or the like may be added. The time k until a future reward may be set to a value equal to 1 or more, and the behavior evaluation function Qπ(s, a) may be maximized based on accumulated information. In such a case, the parameter values of the disturbance estimator Gde can be determined in a state in which the number of epochs until optimization will increase, but will not be stuck to a local solution.

In the fourth embodiment, parameter values of a disturbance estimator Gde according to the first embodiment are determined not by machine learning, but by an optimization method.FIG.15shows an example in which K11is optimized by the optimization method. A controller1035will calculate a loss function Lp of a case in which the parameter K11has been increased by ΔK11and a loss function Lm of a case in which K11has been decreased by ΔK11. Subsequently, the controller1035will compare Lp and Lm, and will change K11in a direction which can make a loss function L smaller. A case in which Lp is smaller will be exemplified here.

First, the controller1035will substitute Lp into L_old. Next, L of a case in which K11is further increased by ΔK11will be calculated. If L is smaller when L and L_old are compared, a loop in which L is calculated by substituting L into L_old to further increase K11by ΔK11and the resultant L and L_old are compared will be continued until the loss function L stops decreasing. If L is larger than L_old, the controller1035will exit the loop, substitute L_old into L_K11, and substitute K11_old into K11. Finally, K11and L_K11will be output.

A method of optimizing the parameter values of a plurality of parameters will be described with reference toFIG.16. In the example ofFIG.16, the parameter values of the parameters K11, K12, and C12will be optimized. However, the parameter values of many more parameters may be simultaneously optimized, and the parameter values may be optimized by targeting several parameters as in the fourth embodiment.

In the example ofFIG.16, first, the controller1035will substitute the current K11, K12, and C12into K11_old, K12_old, and C12_old, respectively. Next, the controller1035will optimize each of K11, K12, and C12, and output the optimized K11, K12, and C12and their respective loss functions L_K11, L_K12, and L_C12. Furthermore, the controller1035will compare L_K11, L_K12, and L_C12, and change the parameter value of a parameter that can minimize the loss function L the most. For example, if L_K11is smallest, the controller1035will change K11and further substitute L_K11into L. Next, the controller1035will compare L and L_old, substitute L into L_old if L is smaller than L_old, and further optimize each of K11, K12, and C12. A loop in which L_K11, L_K12, and L_C12are compared and L is updated by changing the parameter value of a parameter that can minimize the loss function L will be continued until the loss function L stops decreasing. If L is larger than L_old, the controller1035will exit the loop and substitute K11_old into K11, K12_old into K12, and C12_old into C12. Subsequently, the controller1035will finally output K11, K12, and C12. In this manner, the parameter values of the parameters of the disturbance estimator Gde can be determined not by machine learning, but by using the optimization method.

The fifth embodiment provides an example in which a disturbance estimator Gde or a control apparatus1100according to the first embodiment is applied to an imprint apparatus.FIGS.11A and11Bschematically show the arrangement of an imprint apparatus100according to the fifth embodiment.FIG.11Ashows a state before a mold10and an imprint material60on a substrate1are in contact with each other, andFIG.11Bshow a state in which the mold10and the imprint material60on the substrate1are in contact with each other. Assume hereinafter that two axes which are perpendicular to each other on a plane parallel to the surface of the substrate1are an X-axis and a Y-axis, and an axis perpendicular to the X-axis and the Y-axis is an Z-axis.

The imprint apparatus100can include a substrate manipulation unit23that holds the substrate1, a supplier18that supplies the imprint material60, a mold manipulation unit24that holds the mold10, a light source16, an alignment scope21, and a controller35. The imprint apparatus100brings the mold10and the imprint material60supplied onto the substrate1into contact with each other and applies curing energy to the imprint material60to form a pattern of a cured product in which a concave-convex pattern of the mold10has been transferred. The imprint apparatus100ofFIGS.11A and11Bcan be used for manufacturing an article such as a semiconductor device.

The substrate manipulation unit23can include a substrate chuck2, a θ stage3(rotation driver), and an XY stage4(XY driver). The substrate chuck2holds the substrate1by a vacuum suction force or a electrostatic suction force. InFIGS.11A and11B, the substrate1is held by the substrate chuck2. The θ stage3corrects the position of the substrate1in a θ direction (a rotation direction about the Z-axis), and is arranged on the XY stage4for positioning the substrate1in the X direction and the Y direction. The XY stage4can be driven in the X direction and the Y direction by a linear motor19. The θ stage3and the XY stage4hold the substrate chuck2and move the substrate1held by the substrate chuck2. The XY stage4is placed on a base5. A linear encoder6is attached on the base5in the X direction and the Y direction, and measures the position of the XY stage4. A support column8stands on the base5and supports a top plate9.

A single-crystal silicon substrate, an SOI (Silicon On Insulator) substrate, or the like can be used as the substrate1. Glass, a ceramic, a metal, a semiconductor, a resin, or the like may also be used as the substrate1, and a member or a layer made of a material different from a base material may be provided on the surface of the base material as needed. More specifically, the substrate can include a silicon wafer, a compound semiconductor wafer, or silica glass. The substrate1can include a plurality of shot regions, and the imprint material60can be supplied to the shot regions by the supplier18. The imprint apparatus100can form a pattern on the entire surface of the substrate1by repeatedly performing an imprint process for forming an imprint material pattern on each shot regions.

A curable composition (to be also referred to a resin in an uncured state) to be cured by receiving curing energy is used as the imprint material60. As the curing energy, an electromagnetic wave or heat can be used. The electromagnetic wave can be, for example, light selected from the wavelength range of 10 nm or more to 1 mm or less, for example, infrared light, a visible light beam, or ultraviolet light. The curable composition can be a composition cured by light irradiation or heating. Among compositions, a photo-curable composition cured by light irradiation contains at least a polymerizable compound and a photopolymerization initiator, and may further contain a nonpolymerizable compound or a solvent as needed. The nonpolymerizable compound is at least one material selected from the group consisting of a sensitizer, a hydrogen donor, an internal mold release agent, a surfactant, an antioxidant, and a polymer component. In this embodiment, as one example, a photocurable composition that has a property of being cured by ultraviolet light will be used as the imprint material60. The supplier18can arrange the imprint material60on the substrate1in the form of droplets or in the form of an island or film formed by connecting a plurality of droplets. The viscosity (the viscosity at 25° C.) of the imprint material can be, for example, 1 mPa·s or more to 100 mPa·s or less. The imprint material60may also be supplied onto the substrate in the form of a film by a spin coater or a slit coater.

The supplier18(dispenser) supplies the imprint material60on the substrate1. The supplier18includes, for example, discharge nozzles (not shown), and supplies, from the discharge nozzles, the imprint material60onto the substrate1. Note that in this embodiment, as one example, the supplier18will supply the imprint material onto the substrate1by discharging droplets of the liquid imprint material60onto the surface of the substrate1. The amount of the imprint material to be supplied by the supplier18can be determined based on the thickness of the imprint material to be needed, the density of the pattern to be formed, and the like. In addition, the supplier18need not always be arranged in the imprint apparatus100, and a supplier which is arranged outside the imprint apparatus100may supply the imprint material onto the substrate1.

The mold10is a mold for forming the imprint material on the substrate. The mold can also be called a template or an original. The mold10includes, for example, a pattern region P that has a rectangular peripheral portion and whose surface facing the substrate1has a three-dimensionally formed convex-convex pattern, which is to be transferred to the imprint material60supplied on the substrate1. The pattern region P is also referred to as a mesa portion. The pattern region P is formed as a convex portion of several 10 μm to several 100 μm so that the substrate1will not come into contact with a region (region surrounding the pattern region P) other than the pattern region P of the mold10. The mold10is made of a material, for example, quartz or the like, which can transmit light (ultraviolet light) for curing the imprint material on the substrate.

The mold manipulation unit24can include a mold chuck (mold holder)11, a mold stage22, and a linear actuator15(mold driver). The mold chuck11holds the mold10by a vacuum suction force or an electrostatic suction force. The mold chuck11is held by the mold stage22. The mold stage22has a function for adjusting the Z position of the mold10and a tilt function for correcting the tilt of the mold10. The linear actuator15drives the mold10held by the mold chuck11in the Z-axis direction, brings the mold10into contact with the imprint material60on the substrate1, and separates the mold10from the imprint material60. The linear actuator15is, for example, an air cylinder or a linear motor. Note that each of the mold chuck11and the mold stage22includes an opening (not shown) to allow the light emitted from the light source16to pass through to the mold10.

In the process for curing the imprint material60on the substrate1, the light source16irradiates the substrate1with light (ultraviolet light) for curing the imprint material60via a collimator lens17a. The light source16can be, for example, a light source that generates an i-ray (365 nm), but may also be a light source that generates light of another wavelength. A beam splitter20is arranged on an optical path between the light source16and the mold manipulation unit24, and separates the light for curing the imprint material60from the light to be used for observing the contact state of the mold10by the alignment scope21. The alignment scope21captures the pattern region P of the mold10via the beam splitter20.

The controller35controls the operation and the adjustment of each unit that forms the imprint apparatus100. The controller35can be formed by, for example, a computer or the like, be connected to each unit that forms the imprint apparatus100via a communication path, and execute control of each unit in accordance with a program or the like. The controller35may be arranged in the imprint apparatus100or may be installed in a separate location from the imprint apparatus100and remotely control the imprint apparatus100.

The imprint process performed by the imprint apparatus100formed in the above-described manner will be described next.FIG.12is a flowchart showing the imprint process performed in the imprint apparatus100according to the fifth embodiment. Each step can be executed based on the control of each unit of the imprint apparatus100by the controller35. First, various kinds of parameters necessary for the imprint process are set (step S0). Thereafter, the substrate chuck2on which the substrate1is placed is moved in the X direction and the Y direction by driving the XY stage4of the imprint apparatus100so that a shot region (target shot region) to be the target of the imprint process will be arranged bellow the supplier18(step S1). Subsequently, the predetermined amount of uncured imprint material60is supplied to the substrate1(step S2).

Next, the substrate chuck2is moved again by driving the XY stage4so that the target shot region will be arranged in a position that will face the pattern region P of the mold10, and the position of the substrate1in the θ direction is corrected by driving the θ stage3(step S3). Subsequently, the mold10is brought into contact with the uncured imprint material60on the substrate1by driving the linear actuator15to move the mold stage22in the −Z direction (contact process, step S4). In step S4, instead of moving the mold stage22, the substrate chuck2may be moved in the Z direction or each of the mold stage22and the substrate chuck2may be moved. The controller35determines whether a contact force generated when the mold10and the uncured imprint material60on the substrate1are brought into contact is optimal (step S5).

If it is determined that the contact force is not optimal (NO in step S5), the mold stage22will change the tilt of the mold chuck11so that the contact force between the mold10and the imprint material60will be a predetermined value. In addition, the force used to press the mold10against the imprint material is adjusted by changing the pressing amount of the linear actuator15(step S6).

If it is determined that the contact force is optimal (YES in step S5), the alignment scope21will detect alignment marks AM formed on the mold10and the substrate1, and positioning will be performed based on the detected measurement result. Positioning of the mold10and the substrate1is performed by obtaining a relative shift between the mold10and the substrate1from the measurement result and driving the XY stage4and the θ stage3(step S7).

After positioning the mold10and the substrate1, the light source16irradiates (exposes) the imprint material60on the substrate1with light (ultraviolet light) to cure the imprint material60(step S8). The irradiation region of this main exposure operation is the entire surface of the shot region. After light (ultraviolet light) irradiation is performed for a predetermined time and completed, a mold releasing step (step S9) of separating the mold10from the cured imprint material60on the substrate1is performed by driving the linear actuator15to move the mold stage22upward in the +Z direction. In step S9, instead of moving the mold stage22, the substrate chuck2may be moved in the Z direction or each of the mold stage22and the substrate chuck2may be moved.

Subsequently, whether pattern formation has been completed for all of the shot regions on the substrate1is determined (step S12). If a shot region in which an imprint material pattern is to be formed remains, the XY stage4will be driven to move the substrate1so that the imprint material60will be supplied to the next target shot region (step S1). These series of processes are repeated until pattern formation is completed for all of the shot regions on the substrate1. When pattern formation has been completed for all of the shot regions, the substrate1will be moved to a predetermined position by driving the XY stage4(step S13), and the imprint process of one substrate1is ended.

Although the force used to press the mold10against the imprint material is adjusted in step S6, an feeder50for supplying electric power and a fluid is connected to the mold manipulation unit24, and the feeder50can apply a disturbance force to the mold manipulation unit24. Hence, in the fifth embodiment, the controller35will include the disturbance estimator Gde, and a displacement ZO of the mold stage22measured by a position measurement device (not shown) of the mold manipulation unit24will be supplied to the disturbance estimator Gde. Subsequently, the disturbance estimator Gde can obtain an estimated disturbance force Fde, and a manipulation voltage Vc can be corrected based on the disturbance estimator Gde. The disturbance estimator Gde uses a plurality of three-element models, and unsupervised learning of machine learning can be used for the determination of the parameter values of the disturbance estimator Gde in a manner similar to the first embodiment.

In the sixth embodiment, a disturbance estimator Gden according to the second embodiment is applied to an imprint apparatus. The disturbance estimator Gden is formed by a neural network, and unsupervised learning of machine learning can be used for the determination of the parameter values of the disturbance estimator Gden in a manner similar to the second embodiment.

In the seventh embodiment, a disturbance estimator Gde according to the third embodiment is applied to an imprint apparatus. The disturbance estimator Gde uses a plurality of three-element models, and reinforcement learning can be used for the determination of the parameter values of the disturbance estimator Gde in a manner similar to the third embodiment.

In the first to seventh embodiments, a disturbance estimator can estimate the disturbance to be applied to the processor1010or the mold manipulation unit24and a manipulation amount can be corrected based on this estimation. However, since various kinds of disturbance causes influence each other in a complicated manner in an imprint apparatus, the overlay accuracy may not be improved by executing correction based only on the disturbance force of the feeder50. Hence, in the eighth embodiment, in order to estimate a disturbance force in the contact process for bringing the mold into contact with the imprint material on the substrate, the parameter values of the disturbance estimator are determined by using disturbance force data data_Fdimp which is calculated based on a measurement result obtained by an overlay inspection apparatus. A neural network or the like can be used as the disturbance estimator. As values to be input to the disturbance estimator, a manipulation voltage Vcimp to a linear actuator15for driving a mold chuck11can also be used in addition to a displacement ZO of the mold chuck11during the imprint process. Based on the displacement ZO and the manipulation voltage Vcimp, the disturbance estimator can output an estimated disturbance force Fdeimp. Since the disturbance force data data_Fdimp is a value output for each shot region, the estimated disturbance force Fdeimp at the time of the contact process can also be collected for each shot region to set estimated disturbance force data data_Fdeimp. The parameter values of the disturbance estimator can be determined by executing, in a manner similar to the second embodiment, unsupervised learning of machine learning by using the disturbance force data data_Fdimp at the time of the contact process and the estimated disturbance force data data_Fdeimp as training data. As a result, a disturbance at the time of an imprint process that can influence the overlay accuracy can be estimated by the disturbance estimator, and the overlay accuracy can be improved by correcting the manipulation amount based on the disturbance.

In the ninth embodiment, reinforcement learning is used in a manner similar to the third embodiment for the determination of parameter values of a disturbance estimator according to the eighth embodiment.

A method of manufacturing an article will be described as the 10th embodiment hereinafter. The pattern of a cured product formed using an imprint apparatus is used permanently for at least some of various kinds of articles or temporarily when manufacturing various kinds of articles. The articles are an electric circuit element, an optical element, a MEMS, a recording element, a sensor, a mold, and the like. Examples of the electric circuit element are volatile and nonvolatile semiconductor memories such as a DRAM, an SRAM, a flash memory, and an MRAM and semiconductor elements such as an LSI, a CCD, an image sensor, and an FPGA. The mold includes an imprint mold or the like.

The pattern of the cured product is directly used as at least some of the constituent members of the above-described articles or used temporarily as a resist mask. After etching or ion implantation is performed in the substrate processing step, the resist mask is removed.

A method of manufacturing an article in which the above-described imprint apparatus forms a pattern on a substrate, processes the substrate on which the pattern is formed, and manufactures an article from the processed substrate will be described next. As shownFIG.17A, a substrate1zsuch as a silicon wafer with a processed material2zsuch as an insulator formed on the surface is prepared. Next, an imprint material3zis applied to the surface of the processed material2zby an inkjet method or the like. A state in which the imprint material3zis applied as a plurality of droplets onto the substrate is shown here.

As shown inFIG.17B, a side of a mold4zfor imprint with a concave-convex pattern is directed toward and made to face the imprint material3zon the substrate. As shownFIG.17C, the substrate1zto which the imprint material3zis applied is brought into contact with the mold4z, and a pressure is applied. The gap between the mold4zand the processed material2zis filled with the imprint material3z. In this state, when the imprint material3zis irradiated with light as energy for curing via the mold4z, the imprint material3zis cured.

As shown inFIG.17D, after the imprint material3zis cured, the mold4zis separated from the substrate1z, and the pattern of the cured product of the imprint material3zis formed on the substrate1z. In the pattern of the cured product, the concave portion of the mold corresponds to the convex portion of the cured product, and the convex portion of the mold corresponds to the concave portion of the cured product. That is, the concave-convex pattern of the mold4zis transferred to the imprint material3z.

As shown inFIG.17E, when etching is performed using the pattern of the cured product as an etching resistant mask, a portion of the surface of the processed material2zwhere the cured product does not exist or remains thin is removed to form a groove5z. As shown inFIG.17F, when the pattern of the cured product is removed, an article with the grooves5zformed in the surface of the processed material2zcan be obtained. Here, the pattern of the cured product is removed. However, instead of removing the pattern of the cured product after the process, it may be used as, for example, an interlayer dielectric film included in a semiconductor element or the like, that is, a constituent member of an article.

Other Embodiments

Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

This application claims the benefit of Japanese Patent Application No. 2020-144872, filed Aug. 28, 2020, which is hereby incorporated by reference herein in its entirety.