Patent ID: 12260548

DETAILED DESCRIPTION

Reference will now be made in detail to various embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, systems, and components have not been described in detail so as not to unnecessarily obscure aspects of the various embodiments.

Hereinafter, embodiments of an abnormality detection apparatus and an abnormality detection method disclosed herein will be described in detail with reference to the accompanying drawings. The abnormality detection apparatus and abnormality detection method disclosed herein are not limited by the embodiments.

Abnormality detection using a learning model such as artificial intelligence (AI) has been studied. In the normality detection using the learning model, when the learning model is made to only learn normal image data when a monitoring object is operating normally, it may not be possible to accurately detect an abnormality in a narrow range. Therefore, the learning model may also learn abnormal image data in the case in which an abnormality occurs in the monitoring object. However, in reality, there are various places where an abnormality occurs in a monitoring object. In addition, it may be rare that an abnormality occurs in a monitoring object. For this reason, it may be difficult to acquire sufficient abnormal image data required for learning from abnormal image data in the case where an abnormality actually occurs in a monitoring object. As a result, an abnormality may not be detected accurately through abnormality detection using a learning model.

Therefore, a technique capable of detecting an abnormality with high accuracy is required.

Embodiment

[Monitoring Object]

An embodiment will be described. Hereinafter, a case in which the occurrence of an abnormality of a monitoring object is detected using an abnormality detection apparatus will be described as an example. Further, hereinafter, a case in which the monitoring object is a coater that ejects a liquid will be described as an example.FIG.1is a schematic view illustrating an exemplary monitoring object according to an embodiment. InFIG.1, a coater10that ejects a liquid is illustrated as a monitoring object. The coater10is exemplary equipment provided with a liquid supply. The coater10is provided in a liquid-processing apparatus such as a coating apparatus or a developing apparatus that uses a liquid such as a resist, pure water, or a solvent.

FIG.1illustrates a target surface S to which a liquid is ejected. The target surface S is, for example, a surface to be coated with a liquid on a top surface of a substrate such as a semiconductor wafer. The coater10is arranged around the target surface S. The coater10has an arm11and a support12. The arm11is supported by the support12. The arm11is arranged such that a tip end11aof the arm11is located above the target surface S. A nozzle13, which is a liquid ejection port, is provided at the tip end11aof the arm11. The nozzle13is connected to a supply system (not illustrated) including a liquid supply source, a valve, and a metering pump, via a pipe (not illustrated) provided inside the arm11and the support12. The coater10ejects, from the nozzle13, the liquid supplied from the supply system. The coater10may include a movement mechanism so that the arm11is movable in a horizontal direction and a vertical direction. The coater10may be configured to be movable to a coating position at which the tip end11aof the arm11is located above the target surface S and a standby position away from the target surface S by rotating the support12.

A camera20is arranged around the coater10. The camera20is arranged such that the arm11, the nozzle13, and the target surface S are located within an angle of view of the camera20, and is configured to be capable of photographing the arm11and the nozzle13to obtain an image viewed from a side of the arm11and the nozzle13. The camera20photographs the situation of the target surface S and the arm11having the nozzle13at a predetermined frame rate (e.g., 30 fps) during a coating process in which the coater10ejects the liquid onto the target surface S.

FIG.2is a view illustrating an exemplary image obtained by photographing a monitoring object according to an embodiment.FIG.2illustrates an exemplary image captured by the camera20. In the image illustrated inFIG.2, the images of the arm11, the nozzle13, and the target surface S are captured. InFIG.2, the liquid is being ejected from the nozzle13, and the target surface S is being coated with the liquid.

The image data of an image captured by the camera20is output to an abnormality detection apparatus, and abnormality detection is performed.

[Configuration of Abnormality Detection Apparatus]

Next, an abnormality detection apparatus will be described in detail.FIG.3is a block diagram schematically illustrating a configuration of an abnormality detection apparatus50according to an embodiment. The abnormality detection apparatus50is, for example, a computer such as a personal computer or a server computer. The abnormality detection apparatus50includes an external interface (VF)51, a display52, an input part53, a storage54, and/or a controller55. The abnormality detection apparatus50may have various functional parts of a known computer in addition to the functional parts illustrated inFIG.3.

The external VF51is an interface for inputting/outputting information to and from other devices. For example, the external VF51is a communication interface such as a universal serial bus (USB) port or a LAN port. Image data of an image captured by the camera20is input to the external VF51.

The display52is a display device that displays various types of information. The display52may be a display device such as a liquid crystal display (LCD) or a cathode ray tube (CRT). The display52displays various types of information.

The input part53is an input device for inputting various types of information. For example, the input part53may include an input device such as a mouse or a keyboard. The input part53receives an operation input from, for example, an administrator, and inputs operation information indicating received operation content to the controller55.

The storage54is a storage device that stores various types of data. For example, the storage54is a storage device such as a hard disc, a solid state drive (SSD), or an optical disc. The storage54may be a semiconductor memory that is capable of rewriting data, such as a random access memory (RAM), a flash memory, or a nonvolatile static random access memory (NVSRAM).

The storage54stores an operating system (OS) and/or various programs executed by the controller55. For example, the storage54stores various programs including a program that executes a generation process or an abnormality detection process to be described later. The storage54stores various data used in the programs executed by the controller55. For example, the storage54stores model generation data60and model data61. In addition to the data illustrated above, the storage54may also store other data.

The model generation data60is data used for generating a determination model to be described later. The model generation data60includes various data used for generating the determination model. For example, the model generation data60includes multiple pieces of normal image data60a, multiple pieces of actual abnormal image data60b, and multiple pieces of pseudo-abnormal image data60c.

The normal image data60ais data of an image obtained by photographing a monitoring object that operates normally. For example, the normal image data60ais image data of an image obtained in the state in which the coater10ejects a liquid without causing an abnormality such as dripping or dropping of a liquid. Here, the term “dripping” refers to a state in which liquid droplets hang from the nozzle13or the arm11. Further, the term “dropping” refers to a state in which liquid has flowed down from the nozzle13and the arm11. For example, multiple pieces of image data obtained by photographing a series of operations in which the coater10ejects the liquid without causing an abnormality by the camera20are stored as the normal image data60a.

The actual abnormality image data60bis image data obtained by photographing a monitoring object in which an abnormality has occurred. For example, the actual abnormality image data60bis image data when the coater10causes an abnormality such as dripping or dropping. For example, when an abnormality occurs in the coater10, image data captured by the camera20is stored as the actual abnormality image data60b.

The pseudo-abnormal image data60cis image data obtained in a state in which an image when an abnormality occurs is pseudo-generated. Details of the pseudo-abnormal image data60cwill be described later.

The model data61is data that stores a determination model generated using a learning model such as AL.

The controller55is a device that controls the abnormality detection apparatus50. As the controller55, an electronic circuit such as a central processing unit (CPU) or a micro-processing unit (MPU), or an integrated circuit such as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA) may be adopted. The controller55has an internal memory for storing programs and control data that define various processing procedures, and executes various processes using these programs or data. The controller55serves as various processing parts by executing various programs. For example, the controller55includes a first generation part70, a reception part71, a second generation part72, an acquisition part73, a detection part74, and an output part75.

In abnormality detection using a learning model, when the learning model is made to learn only the normal image data60a, it may not be possible to accurately detect an abnormality in a narrow range. Therefore, the learning model may also learn the actual abnormal image data60bin cases where an abnormality occurs in the monitoring object. However, in reality, there are various places where an abnormality occurs in a monitoring object, and there are cases where an abnormality rarely occurs in a monitoring object. For example, the coater10may cause an abnormality such as dripping or dropping from the nozzle13. In the coater10, liquid leakage may occur from a pipe inside the arm11, and an abnormality, such as dripping or dropping from the arm11, may occur. The coater10is designed so as to prevent the occurrence of an abnormality, such as dripping or dropping. Therefore, an abnormality rarely occurs in the coater10. For example, an abnormality rarely occurs in the arm11. For this reason, it may be difficult to acquire abnormal image data in the state in which an abnormality actually occurs in a monitoring object. For example, it is difficult to acquire image data in a state in which an abnormality, such as dripping or dropping, actually occurs at various places in the arm11. As a result, an abnormality may not be detected accurately through abnormality detection using a learning model.

Therefore, the first generation part70generates pseudo-abnormal image data60cobtained by partially changing the image of the normal image data60a. For example, the first generation part70generates the pseudo-abnormal image data60cby synthesizing a substantially circular image at a random position of the image of the normal image data60a. The substantially circular shape includes a circular shape and an elliptical shape. As an example, the first generation part70generates the pseudo-abnormal image data60cby performing random erasing on the image of the normal image data60aand synthesizing a substantially circular image at a random position of the image of the normal data60a. Here, random erasing is a technique of increasing an amount of teaching data as normal image data by generating images by masking random partial rectangular areas in an image to be teaching data, usually for the purpose of improving the robustness of determination. In the present embodiment, the pseudo-abnormal image data60c, which is pseudo-abnormal, is generated by synthesizing a substantially circular image having a shape similar to an abnormality such as dripping or dropping with an image of normal image data60ausing the random erasing.

The first generation part70generates multiple pieces of pseudo-abnormal image data by partially changing an image for each of multiple pieces of normal image data obtained by photographing a series of operations of a monitoring object that operates normally. For example, the first generation part70generates multiple pieces of pseudo-abnormal image data60cby partially changing an image for each of multiple pieces of normal image data60aobtained by photographing a series of operations in which the coater10ejects a liquid without causing an abnormality. For example, in the present embodiment, multiple pieces of image data obtained using the camera20by photographing a series of operations in which the coater10ejects a liquid without causing an abnormality ten times are each stored as normal image data60a. The first generation part70generates the pseudo-abnormal image data60cby synthesizing a substantially circular image at one random position on an image of one piece of normal image data60athat corresponds to one out of the ten times of the series of operations.

Here, in the monitoring object, the place where an abnormality occurs may be biased to a specific range. In addition, the size of an abnormal place in an image may be biased to a specific size or less. For example, the coater10causes an abnormality such as dripping or dropping in the nozzle13or arm11. In addition, the dripping or dripping occurs within a narrow range with a small size.

Therefore, the reception part71receives designation of a range within which an image is to be synthesized and a size of the image to be synthesized. For example, the reception part71causes the display unit52to display a screen for designating the range within which the image is to be synthesized and the size of the image to be synthesized, and receives, from the input part53, the designation of the range within which the image is to be synthesized and the size of the image to be synthesized.

The first generation part70generates pseudo-abnormal image data60cobtained by synthesizing a substantially circular image at a random position within the designated range with a size smaller than or equal to the size designated by the reception part71. For example, when detecting dripping or dropping from the nozzle13or arm11as an abnormality, an administrator designates a range including the nozzle13or arm11and the maximum size of the dripping or dropping droplets. The first generation part70generates the pseudo-abnormal image data60cobtained by synthesizing a substantially circular image having a size smaller than or equal to the designated size at random positions within the designated range including the nozzle13or the arm11.

FIG.4is a view illustrating exemplary pseudo-abnormal image data60caccording to an embodiment.FIG.4illustrates a pseudo-abnormal image obtained by synthesizing a white and substantially circular image80with the image illustrated inFIG.2. InFIG.4, the white and substantially circular image80is synthesized near the bottom surface of the arm11.

The color of the image80to be synthesized is not limited to white. The color of the image80to be synthesized may be predetermined. The color of the image80to be synthesized may be determined from the normal image data60a. For example, the first generation part70may generate pseudo-abnormal image data60cobtained by synthesizing a substantially circular image80having brightness set to a brightness value of a predetermined ratio (e.g., 20%) when the brightness values of the pixels of the image of the normal image data60aare arranged in a descending order from the top. By determining the color of the image80to be synthesized based on the normal image data60ain this manner, it is possible to synthesize the image80with a color similar to the color of the actual image. By synthesizing the image80with a color similar to the color of the actual image in this manner, an actual occurrence of an abnormality can be detected with high accuracy.

The second generation part72generates a determination model for determining whether a monitoring object is normal or abnormal by performing learning of multiple pieces of normal image data60a, multiple pieces of actual abnormal image data60b, and multiple pieces of pseudo abnormal image data60c. For example, the second generation part72adds additional information that the multiple pieces of normal image data60aare normal to the multiple pieces of normal image data60a, and adds additional information that the multiple pieces of actual abnormal image data60band the multiple pieces of pseudo-abnormal image data60care abnormal to the multiple pieces of actual abnormal image data60band the multiple pieces of pseudo-abnormal image data60c. For example, the determination model is a model that outputs image data when image data is input. The second generation part72performs learning such that the reproducibility of normal image data is high and the reproducibility of abnormal image data is low by, for example, opposite learning, and generates a determination model that outputs image data with high reproducibility for normal image data. By performing learning of the pseudo-abnormal image data60c, the determination model may learn an abnormality that rarely occurs in reality from the pseudo-abnormal image data60c. Therefore, the determination accuracy of the determination model is improved. The second generation part72may generate a determination model by performing learning of the multiple pieces of normal image data60aand the multiple pieces of pseudo-abnormal image data60cwithout using the multiple pieces of actual abnormal image data60b. However, by generating a determination model that has learned the actual abnormal image data60bin addition to the normal image data60aand the pseudo-abnormal image data60c, the determination model is able to increase the difference in reproducibility between the normal image data and the abnormal image data, thereby improving the determination accuracy.

The second generation part72stores data of the generated determination model in the storage54as model data61.

The acquisition part73acquires image data obtained by photographing a monitoring object. For example, the acquisition part73acquires image data output from the camera20via the external VF51.

The detection part74detects an abnormality of the monitoring object from the image data acquired by the acquisition part73using the determination model of the model data61.FIG.5is a view schematically illustrating exemplary abnormality detection according to an embodiment. For example, the detection part74inputs the acquired image data to the determination model. The determination model outputs image data. When the output image data is normal image data, the reproducibility of the input image data is high, and when it is abnormal image data, the reproducibility is low. The detection part74compares the output image data with the input image data so as to obtain a difference between the output image data and the input image data. For example, the detection part74obtains a change in the pixel value of each pixel as a difference. When the obtained difference is within a predetermined threshold value, the detection part74determines that the input image data is normal, and when the difference exceeds the threshold value, the detection part74determines that the input image data is abnormal, thereby detecting the abnormality.

The determination model may be the following model. For example, the second generation part72adds additional information that the multiple pieces of normal image data60aare normal to the multiple pieces of normal image data60a, and adds additional information that the multiple pieces of actual abnormal image data60band the multiple pieces of pseudo-abnormal image data60care abnormal to the multiple pieces of actual abnormal image data60band the multiple pieces of pseudo-abnormal image data60c. Then, the second generation part72generates a determination model for determining whether the input image data is normal or abnormal and for outputting a determination result by performing machine learning, such as deep learning, using the multiple pieces of normal image data60a, the multiple pieces of actual abnormal image data60b, and the multiple pieces of pseudo-abnormal image data60c. The detection part74inputs the acquired image data to the determination model. The determination model outputs whether the input image data is normal or abnormal. The detection part74detects an abnormality using the output result of normality and abnormality of the determination model. In this case as well, the second generation part72may generate a determination model by performing learning of the multiple pieces of normal image data60aand the multiple pieces of pseudo-abnormal image data60cwithout using the multiple pieces of actual abnormal image data60b. However, by generating a determination model that has learned the actual abnormal image data60bin addition to the normal image data60aand the pseudo-abnormal image data60c, the determination model is improved in determination accuracy of normality and abnormality.

In some cases, noise may be temporarily generated in image data captured by the camera20, and it may be detected that an abnormality has occurred in the coater10even though the coater10is normal. When the camera20photographs a series of operations in which the coater10applies a liquid, an abnormality such as dripping or dropping is detected in multiple pieces of image data. Therefore, the detection part74may detect that an abnormality has occurred in the monitoring object when it is determined that an abnormality has occurred in a predetermined number of pieces of continuously captured image data. For example, when it is determined that an abnormality has occurred in three continuously captured image data, the detection part74detects that an abnormality has occurred in the coater10. As a result, the accuracy of determining an abnormality is improved.

The output part75outputs the detection result of the detection part74. For example, when an abnormality is detected by the detection part74, the output part75outputs to the display52an indication that an abnormality has occurred. The output part75may output the data of the determination result of the detection part74to another device. For example, when the detection part74determines that an abnormality has occurred, the output part75may output data to the effect that an abnormality has occurred in a management device that manages the device on which the coater10is mounted.

[Flow of Process]

Next, a flow of various processes performed by the abnormality detection apparatus50according to an embodiment will be described. First, a flow of a generation process in which the abnormality detection apparatus50according to the embodiment generates a determination model will be described.FIG.6is a flowchart illustrating an exemplary flow of a generation process according to an embodiment.

The first generation part70generates pseudo-abnormal image data60cobtained by partially changing an image of normal image data60a(Step S10). For example, the first generation part70generates pseudo-abnormal image data60cobtained by synthesizing a substantially circular image at a random position of the image of the normal image data60a.

The second generation part72generates a determination model for determining whether a monitoring object is normal or abnormal by performing learning of the normal image data60a, the actual abnormal image data60b, and the pseudo abnormal image data60c(Step S11). The second generation part72stores data of the generated determination model in the storage54as the model data61(Step S12) and terminates the process.

Next, a flow of a generation process in which the abnormality detection apparatus50according to the embodiment detects an abnormality using the determination model will be described.FIG.7is a flowchart illustrating an exemplary flow of an abnormality detection process according to an embodiment.

The acquisition part73acquires image data obtained by photographing a monitoring object (Step S20). For example, the acquisition part73acquires the image data output from the camera20via the external VF51.

The detection part74detects an abnormality of the monitoring object from the image data acquired by the acquisition part73using the determination model of the model data61(Step S21).

The output part75outputs the detection result of the detection part74(Step S22) and terminates the process.

As described above, the abnormality detection apparatus50according to the embodiment includes the first generation part70, the second generation part72, the acquisition part73, and the detection part74. The first generation part70generates pseudo-abnormal image data60cobtained by synthesizing a substantially circular image at a random position of an image of normal image data60a, which is obtained by photographing equipment (the coater10) that supplies a liquid from a liquid supply (the nozzle13) normally (that is, without an abnormality). The second generation part72generates a determination model for determining whether the equipment is normal or abnormal by performing learning of the normal image data60aand the pseudo-abnormal image data60c. The acquisition part73acquires image data obtained by photographing the equipment. The detection part74detects an abnormality in the equipment from the image data acquired by the acquisition part73using the determination model. As a result, since the determination model is capable of learning an abnormality that rarely occurs in reality from the pseudo-abnormal image data60c, the abnormality detection apparatus50is capable of detecting an abnormality with high accuracy.

In addition to the normal image data60aand the pseudo-abnormal image data60c, the second generation part72generates the determination model by performing learning of actual abnormal image data60bobtained by photographing the equipment (the coater10) in which an abnormality has occurred. As a result, since the determination model is capable of learning an abnormality that actually occurred from the actual abnormal image data60b, the abnormality detection apparatus50is capable of detecting an abnormality with higher accuracy.

Further, the equipment (the coater10) may include the arm11provided with a pipe or a nozzle13configured to eject a liquid. The first generation part70generates pseudo-abnormal image data60cobtained by synthesizing a substantially circular image80at a random position of the image of the normal image data60a. As a result, the abnormality detection apparatus50is capable of detecting an abnormality such as dripping or dropping with higher accuracy.

In addition, the abnormality detection apparatus50further includes a reception part71. The reception part71receives designation of a range within which a substantially circular image is to be synthesized and/or a size of the substantially circular image to be synthesized. The first generation part70generates pseudo-abnormal image data60cobtained by synthesizing a substantially circular image at a random position within the designated range with a size smaller than or equal to the size designated by the reception part71. As a result, the abnormality detection apparatus50is capable of detecting an abnormality with high accuracy by appropriately designating the range in which an image is to be synthesized and the size of the image to be synthesized that correspond to an abnormality to be detected.

Further, when brightness values of pixels of the image of the normal image data60aare arranged in a descending order, the first generation part70generates pseudo-abnormal image data60cobtained by synthesizing a substantially circular image having brightness set to a brightness value of a predetermined ratio from the top. As a result, the abnormality detection apparatus50is capable of detecting an abnormality that actually occurred with high accuracy.

In addition, the first generation part70generates multiple pieces of pseudo-normal image data60cobtained by synthesizing a substantially circular image at a random position of an image for each of multiple pieces of normal image data60a, which is obtained by photographing a series of operations of the equipment (the coater10) that operates normally (that is, without an abnormality). The second generation part72generates a determination model for determining whether the equipment is normal or abnormal by performing learning of the multiple pieces of normal image data60aand the multiple pieces of pseudo-abnormal image data60c. The acquisition part73acquires multiple pieces of image data obtained by photographing a series of operations of the equipment. The detection part74determines whether the multiple pieces of image data acquired by the acquisition part73are normal or abnormal using the determination model, and when any of the multiple pieces of image data is determined to be abnormal, the detection part74detects that an abnormality has occurred in the equipment. As a result, since it is possible to cause the determination model to learn a series of operations of the equipment, the abnormality detection apparatus50is capable of detecting an abnormality that has occurred during the series of operations of the equipment with high accuracy.

Further, the detection part74detects that an abnormality has occurred in the equipment when a predetermined number of continuously captured pieces among the multiple pieces of image data is determined to be abnormal. As a result, the accuracy of the abnormality detection apparatus50in determining an abnormality is improved.

Although embodiments have been described above, it should be considered that the embodiments disclosed herein are illustrative and are not restrictive in all respects. Indeed, the embodiments described above can be implemented in various forms. In addition, the embodiments described above may be omitted, replaced, or modified in various forms without departing from the scope of the claims.

For example, in the embodiments described above, the cases in which the equipment to be monitored is the coater10have been described as an example. However, the present disclosure is not limited thereto. Any unit may correspond to the equipment as long as the unit is provided with a liquid supply that ejects (supplies) a liquid.

In the embodiments described above, the cases in which an abnormality such as dripping or dropping have been described as an example, but the present disclosure is not limited thereto. Any abnormality may be detected.

In the embodiments described above, the cases in which the abnormality detection apparatus50generates a determination model have been described as an example, but the present disclosure is not limited thereto. The determination model may be generated by another device, and may be stored in the storage54of the abnormality detection apparatus50. The abnormality detection apparatus50may detect an abnormality of the equipment from acquired image data using a determination model generated by another device and stored in the storage54.

According to the present disclosure, it is possible to detect an abnormality with high accuracy.

While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the disclosures. Indeed, the embodiments described herein may be embodied in a variety of other forms. Furthermore, various omissions, substitutions and changes in the form of the embodiments described herein may be made without departing from the scope of the disclosures. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope of the disclosures.