ANALYSIS APPARATUS, ANALYSIS SYSTEM, ANALYSIS PROGRAM, AND ANALYSIS METHOD

Provided is an analysis apparatus that can improve detection accuracy and versatility while suppressing an amount of training data in, for example, identifying a cause of a defect in a product using machine learning. An analysis apparatus includes: an acquisition section that acquires input data; an identification section that identifies any one of predetermined recognition classes by inputting the input data to a machine learning model trained in advance with training data; a database that stores a correspondence relationship between the recognition class and predetermined information; an analysis section that analyzes the correspondence relationship between the recognition class and the predetermined information by using the database; and an output section that outputs an analysis result.

TECHNICAL FIELD

The present invention relates to an analysis apparatus, an analysis system, an analysis program, and an analysis method.

BACKGROUND ART

In industrial product production sites and the like, a defective product is sorted by detecting a defect in a product through visual inspection or the like.

In addition, development of a technology for identifying, based on information on the defect detected by visual inspection or the like, an apparatus or the like that has caused the defect so as to be able to cope with the identified apparatus or the like is underway.

Patent Literature 1 discloses the following technologies. In a manufacturing line of a semiconductor device, a defect in a circuit pattern on a wafer is inspected by a visual inspection apparatus, and an apparatus type as a cause of the defect is identified from an inspection result by using a defect mode database indicating a correspondence relationship between a defect mode or the like and the apparatus type. Then, the apparatus causing the defect is identified from the identified apparatus type and the processing history.

CITATION LIST

Patent Literature

Patent Literature 1: JP 2005-197437 A

SUMMARY OF INVENTION

Technical Problem

However, in the above-described inspection, in a case where an erroneous identification is made in defect inspection, there is a problem that a wrong apparatus is identified by using the database. In a case where an attempt is made to identify an apparatus type or the like as a cause of a defect in End-to-End by machine learning by using a captured image or the like of an inspection target such as an industrial product, there is a problem that inspection accuracy decreases due to lack of training data for machine learning in a case where the type of the inspection target is expanded. The above-described conventional art cannot cope with such problems.

The present invention has been made to solve the above-mentioned problems. That is, an object of the present invention is to provide an analysis system, an analysis program, an analysis apparatus, and an analysis method that can improve detection accuracy and versatility while suppressing an amount of training data in, for example, identifying a cause of a defect in a product using machine learning.

Solution to Problem

The above-described object of the present invention is achieved by the following means.

(1) An analysis apparatus including: an acquisition section that acquires input data; an identification section that identifies any one of predetermined recognition classes by inputting the acquired input data to a machine learning model trained in advance with training data; a database that stores a correspondence relationship between the recognition class and predetermined information; an analysis section that analyzes the correspondence relationship between the recognition class and the predetermined information by using the database; and an output section that outputs an analysis result obtained by the analysis section.

(2) The analysis apparatus according to (1), wherein the input data is an image, the database stores a correspondence relationship between the recognition class of the defect and the detection result of an apparatus type or a manufacturing process, and the analysis section analyzes the correspondence relationship between the recognition class and the apparatus type or the manufacturing process by using the database.

(3) The analysis apparatus according to (1), wherein the input data is an image, the database stores a correspondence relationship between the recognition class of a behavior and a monitoring place, and the analysis section analyzes the correspondence relationship between the recognition class of the behavior and the monitoring place by using the database.

(4) The analysis apparatus according to (1), wherein the input data is an image, the database stores a correspondence relationship between the recognition class of a behavior and applicability to a detection target behavior, and the analysis section analyzes the correspondence relationship between the recognition class and the applicability to the detection target behavior by using the database.

(5) The analysis apparatus according to (1), wherein the machine learning model is a neural network, and the identification section inputs the input data to the neural network and converts the input data into any one of the recognition classes and a certainty factor of the recognition class.

(6) The analysis apparatus according to (1), wherein the input data is an image, and the analysis apparatus further includes a data control section that switches the database used for analysis by the analysis section or limits the database to a part of the database in accordance with information on a product included in the image or information on a place where a person included in the image is present.

(7) The analysis apparatus according to (5), wherein the analysis section does not perform analysis when the certainty factor of the recognition class converted by the identification section is less than a predetermined threshold value.

(8) The analysis apparatus according to (5), wherein the analysis section weights the certainty factor in accordance with a degree of importance set for each of the recognition classes converted by the identification section, and analyzes, using the database, a correspondence relationship between the recognition class having a highest weighted certainty factor and the predetermined information.

(9) The analysis apparatus according to (2), wherein the analysis section accumulates identification results by the identification section of respective images of different analysis targets of a same product, and detects the apparatus type through analysis based on a plurality of accumulated identification results.

(10) An analysis system including: the analysis apparatus according to any one of (1) to (9); and a storage section that stores the machine learning model and the database.

(11) An analysis program for causing a computer to execute a process including the procedures of: (a) acquiring input data detected; (b) inputting the input data acquired in the procedure (a) to a machine learning model trained in advance with training data to identify any one of predetermined recognition classes; (c) analyzing a correspondence relationship between the recognition class and predetermined information by using a database storing the correspondence relationship between the recognition class and the predetermined information; and (d) outputting an analysis result obtained in the procedure (c).

(12) The analysis program according to (11), wherein the input data is an image, the database stores a correspondence relationship between the recognition class of a defect and an apparatus type or a manufacturing process, and the procedure (c) includes analyzing the correspondence relationship between the recognition class and the apparatus type or the manufacturing process by using the database.

(13) The analysis program according to (11), wherein the input data is an image, the database stores a correspondence relationship between the recognition class of a behavior and a monitoring place, and the procedure (c) includes analyzing the correspondence relationship between the recognition class of the behavior and the monitoring place by using the database.

(14) The analysis program according to (11), wherein the input data is an image, the database stores a correspondence relationship between the recognition class of a behavior and applicability to a detection target behavior, and the procedure (c) includes analyzing the correspondence relationship between the recognition class and the applicability to the detection target behavior by using the database.

(15) The analysis program according to claim (11), wherein the machine learning model is a neural network, and the procedure (b) includes inputting the input data to the neural network, and converting the input data into any one of the recognition classes and a certainty factor of the recognition class.

(16) The analysis program according to (11), wherein the input data is an image, and the process further includes a procedure (d) of switching the database used for analysis in the procedure (c) or limiting the database to a part of the database in accordance with information on a product included in the image or information on a place where a person included in the image is present.

(17) The analysis program according to (15), wherein the procedure (c) includes not performing analysis when the certainty factor of the recognition class converted in the procedure (b) is less than a predetermined threshold value.

(18) The analysis program according to (15), wherein the procedure (c) includes weighting the certainty factor in accordance with a degree of importance set for each of the recognition classes converted in the procedure (b), and analyzing a correspondence relationship between the recognition class having a highest weighted certainty factor and the predetermined information by using the database.

(19) The analysis program according to (12), wherein the procedure (c) includes accumulating identification results in the procedure (b) of respective images of different analysis targets of a same product, and detecting the apparatus type through analysis based on a plurality of accumulated identification results.

(20) An analysis method including the steps of: (a) acquiring input data detected; (b) inputting the input data acquired in the step (a) to a machine learning model trained in advance with training data to identify any one of predetermined recognition classes; (c) analyzing a correspondence relationship between the recognition class and predetermined information by using a database storing the correspondence relationship between the recognition class and the predetermined information; and (d) outputting an analysis result obtained in the step (c).

Advantageous Effects of Invention

Any one of predetermined recognition classes is identified from input data using a trained machine learning model, and a correspondence relationship between the recognition class and predetermined information is analyzed using a database that stores the correspondence relationship between the recognition class and the predetermined information. Thus, in, for example, identifying a cause of a defect in a product using machine learning, it is possible to improve detection accuracy and versatility while suppressing an amount of training data.

DESCRIPTION OF EMBODIMENTS

Hereinafter, with reference to the drawings, an analysis apparatus, an analysis system, an analysis program, and an analysis method according to embodiments of the present invention will be described. Note that in the drawings, the same constituent elements are denoted by the same reference numerals, and redundant description thereof will be omitted. In addition, dimensional ratios in the drawings are exaggerated for convenience of description and may be different from actual ratios.

First Embodiment

FIG.1is a diagram illustrating a configuration of an analysis system10.FIG.2is a block diagram of an analysis apparatus100included in the analysis system10. The analysis apparatus100may be configured by a plurality of apparatuses.

The analysis system10may include the analysis apparatus100and an imaging apparatus200.

The imaging apparatus200captures an image201of an analysis target202(seeFIG.3) (hereinafter, also simply referred to as the “image201”). The image201constitutes input data to be input to the analysis apparatus100. The image201may be an image of all or a part of the analysis target202. The image201may be an image including an object other than the analysis target202. The imaging apparatus200includes, for example, a camera. The analysis target202is, for example, a product, and the product includes not only a finished product such as an automobile or a semiconductor chip but also components such as an automobile body, a bolt, a nut, and a flat plate. The imaging apparatus200can be replaced with a scanner, a microphone, a voice recognition apparatus that converts voice into text data, an odor sensor, a temperature sensor, and the like. Data detected (acquired) by these apparatuses also constitutes input data.

The image201may be, for example, a monochrome image or a color image, and may be a 128 pixel 128 pixel image. The imaging apparatus200transmits the image201to the analysis apparatus100.

The analysis apparatus100detects an abnormality of the analysis target by analyzing the image. The abnormality includes, for example, a flaw, a chip, a fold, a bend, dirt, discoloration (a change in color), an abnormal temperature, and a malodor. As will be described later, the analysis apparatus100can detect an abnormality (defect) of the analysis target as any one of predetermined recognition classes indicating types of abnormality.

Note that in a third embodiment described later, the analysis target202may be a person. In this case, the abnormality includes, for example, an abnormality of a behavior of a person in a manufacturing line of a product.

The analysis apparatus100further performs analysis to detect (identify), from the detected type of abnormality (recognition class of defect), a type of an apparatus that has caused the abnormality, and outputs an analysis result.

As illustrated inFIG.2, the analysis apparatus100includes a controller110, a storage section120, a communication section130, and an operation display section140. These constituent elements are connected to each other via a bus150. The analysis apparatus100is constituted of, for example, a computer terminal.

The controller110includes a central processing unit (CPU), and a memory such as a random access memory (RAM) and a read only memory (ROM).

The controller110performs control and arithmetic processing of the respective sections of the analysis apparatus100in accordance with a program. Details of functions of the controller110will be given later.

The storage section120includes a hard disc drive (HDD), a solid state drive (SSD), or the like, and stores various programs and various types of data.

The communication section130is an interface circuit (e.g., a local-area network (LAN) card) for communicating with an external apparatus through a network.

The operation display section140may be constituted of, for example, a touch panel. The operation display section140accepts various inputs from a user. The operation display section140displays various information including the analysis result of the inspection target.

Functions of the controller110will be described below.

FIG.3is a functional block diagram of the controller110. The controller110functions as an acquisition section111, a conversion section112, an analysis section113, and an output section114.FIG.3also illustrates a learned model121and a knowledge database122stored in the storage section120.

The acquisition section111acquires the image201by receiving it from the imaging apparatus200. In a case where the image is stored in the storage section120, the acquisition section111can acquire the image201by reading it from the storage section120.

InFIG.3, the analysis target202is a flat plate that is a component, and an image201in which dirt of a black circle is present at the lower right of the flat plate is illustrated.

The conversion section112inputs the image201to the learned model read out from the storage section120, and identifies any one of the predetermined recognition classes. The identification includes identification performed through conversion and classification. The conversion section112constitutes an identification section. In the following description, for the sake of convenience in the description, it is assumed that the identification is performed through conversion. Specifically, for example, the conversion section112converts the image201into a vector of the certainty factor of each recognition class (the certainty factor of each recognition class). The certainty factor is a likelihood indicating the probability that the image201corresponds to a corresponding recognition class. The predetermined recognition class may be a recognition class of the defect (abnormality). Converting the image201into the vector of the certainty factor of each recognition class by the conversion section112is to enable identification of a recognition class having a high certainty factor. Converting the image201into the vector of the certainty factor of each recognition class by the conversion section112is equivalent to converting the image201into a predetermined identification class. Note that the conversion section112may convert the image201into a recognition class having the highest certainty factor instead of the vector of the certainty factor. More specifically, the recognition class having the highest certainty factor is data for identifying the recognition class having the highest certainty factor.

The learned model121is an example of a machine learning model trained in advance, and may be a learned model of a neural network. The learned model121may be a learned model of a support vector machine or a random forest, which is a model other than the neural network. Hereinafter, for the sake of convenience in the description, description will be given on the assumption that the learned model121is a learned model of a neural network.

The learned model121is generated in advance by training a neural network using train data of a combination of the image201and a ground truth label corresponding to the image201. The learned model121is generated in advance by training a neural network using a relatively large amount of train data. The ground truth label is a recognition class. The train data is training data. The generated learned model121can be stored in the storage section120. Specifically, the neural network is trained so that the difference (loss) between the recognition class when the image201is input and the ground truth label becomes small. The neural network is trained by back propagation. Specifically, the recognition class is a vector of the certainty factor of each recognition class. Specifically, the ground truth label is a ground truth of a vector of the certainty factor of each recognition class.

It is preferable that the train data is a combination of the image201of the defective product and the ground truth label of the recognition class of the defect. The recognition class of the defect is the recognition class corresponding to the image201of the defective product. Thus, it is possible to efficiently improve accuracy in detection of an abnormality (defect) performed by the conversion section112.

The predetermined recognition classes are types of abnormality. The predetermined recognition classes include, for example, “a flaw having a length equal to or longer than a predetermined length”. “a flaw having a length less than a predetermined length”. “a flaw having a thickness equal to or greater than a predetermined thickness”, and “a flaw having a thickness less than a predetermined thickness”. The predetermined recognition classes include, for example, “dirt having an area equal to or larger than a predetermined area”. “dirt having an area less than a predetermined area”, and “discoloration”. In addition, for example, “dirt having an area equal to or larger than a predetermined area”, “dirt having an area equal to or smaller than a predetermined area”, and the like can be subdivided into a plurality of recognition classes for respective locations of dirt in the image201. That is, the recognition classes may include, for example, “upper right dirt having an area equal to or larger than a predetermined area” and “lower right dirt having an area equal to or larger than a predetermined area”. The recognition classes may include “upper left dirt having an area equal to or larger than a predetermined area” and “lower left dirt having an area equal to or larger than a predetermined area”. The recognition classes may include “upper right dirt having an area equal to or smaller than a predetermined area” and “lower right dirt having an area equal to or smaller than a predetermined area”. The recognition classes may include “upper left dirt having an area equal to or smaller than a predetermined area”, “lower left dirt having an area equal to or smaller than a predetermined area”, and the like. The recognition classes may be further subdivided as necessary. The predetermined length, the predetermined thickness, and the predetermined area can be appropriately set through experiments or the like from the viewpoint of analysis accuracy of analysis for detecting an apparatus type from the recognition classes. The recognition classes may include a class (e.g., “non-defective product”) other than the types of abnormality. The recognition classes may be recognition classes of a behavior when a person is included in the analysis target.

The analysis section113performs analysis to detect the type of the apparatus that has caused the abnormality, using conversion results obtained by the conversion section112and the knowledge database122stored in the storage section120. The conversion results obtained by the conversion section112are the recognition classes. That is, the analysis section113analyzes a correspondence relationship between the recognition classes and the predetermined information (detection results) by using the knowledge database122. Thus, the type of the apparatus that has caused the abnormality is detected. The apparatus type detected through the analysis performed by the analysis section113constitutes predetermined information. The predetermined information may be a manufacturing process, a monitoring place, or the like. The knowledge database122may be a data group (or apparatus) that indicates (stores) a correspondence relationship between the recognition classes and predetermined information. It is preferable that the predetermined information is information different from the recognition classes and is not obtained from the machine learning model, but may be information obtained from the machine learning model. The predetermined information may be detection results detected by a detection device or may be input information input to an input device such as the operation display section140. Specifically, the predetermined information is preferably information on a cause or a place of occurrence of a defect, an abnormality, or the like. It is more preferable that the predetermined information is information on an apparatus type, a manufacturing process, or a monitoring place. Hereinafter, for the sake of convenience in the description, description will be made on the assumption that the predetermined information is an apparatus type, unless otherwise specified. Specifically, for example, in a case where the certainty factor of the recognition class of “lower right dirt having an area equal to or smaller than a predetermined area” is the highest in the conversion results obtained by the conversion section112, the analysis section113identifies the apparatus type corresponding to “lower right dirt having an area equal to or smaller than a predetermined area”. Specifically, for example, the analysis section113detects the apparatus type corresponding to “lower right dirt having an area equal to or smaller than a predetermined area” by identifying it using the knowledge database.

The correspondence relationship between the recognition classes and the apparatus types included in the knowledge database122may be based on, for example, knowledge of a skilled worker or engineer in a production factory. This is because a worker or the like is highly likely to be able to more accurately identify the process (type of apparatus) that has caused the abnormality (defect), for example, simply by looking at an image of a flat plate including a lower right dirt having an area equal to or smaller than a predetermined area, based on his or her experience.

The correspondence relationship between the recognition classes and the apparatus types included in the knowledge database122may be set based on a past repair history of the apparatus.

In this way, even the type of abnormality is estimated by machine learning without End-to-End detection of the type of the apparatus that has caused the abnormality from the image201by machine learning. Then, using the knowledge database122, information (such as language information) on the type of apparatus that has caused the abnormality is identified from information (such as language information) on the type of abnormality. Thus, it is possible to improve detection accuracy and versatility while suppressing the amount of training data. Detection of an abnormality based on an image or the like is likely to be affected by an imaging condition (e.g., an imaging environment). Therefore, in a case of End-to-End detection of a type of an apparatus that has caused the abnormality from an image or the like by machine learning, retraining of a learned model is required every time a production factory changes. In the present embodiment, the information in the knowledge database122is knowledge of the correspondence relationship between the type of abnormality and the type of the apparatus that has caused the abnormality. For this reason, in the present embodiment, the learned model121can be reused in a different production factory, and retraining can be made unnecessary.

The analysis section113may detect the type of the apparatus that has caused the abnormality by using a plurality of recognition classes each having a certainty factor equal to or greater than a predetermined threshold value and the knowledge database122. For example, in a case where the recognition classes each having a certainty factor equal to or greater than a predetermined threshold value are “lower right dirt having an area equal to or smaller than a predetermined area” and “upper left dirt having an area equal to or larger than a predetermined area”, the apparatus type corresponding to these two recognition classes is detected. The apparatus type corresponding to these two recognition classes can be detected by being identified using the knowledge database. Therefore, the knowledge database122may include data indicating a correspondence relationship between a plurality of recognition classes and one apparatus type. The predetermined threshold value can be appropriately set by an experiment or the like from the viewpoint of the analysis accuracy of the analysis for identifying the apparatus type from the recognition classes.

In a case where the certainty factor of the recognition classes converted from the image201is less than a predetermined threshold value, the analysis section113may not perform the analysis for detecting the apparatus type. The certainty factor of the recognition class converted from the image201is, for example, the highest certainty factor among the certainty factors included in the vectors of the certainty factors of the respective recognition classes.

The degree of importance of the type of abnormality may be set to each recognition class. The analysis section113performs weighting by multiplying the certainty factor of the recognition class indicating the type of abnormality by the degree of importance. The analysis section113may use, as the analysis result, the apparatus type that corresponds to the recognition class having the highest certainty factor after weighting in the knowledge database122. The degree of importance can be appropriately set by an experiment or the like from an any viewpoint. For example, the degree of importance can be appropriately set from the viewpoint of the magnitude of the degree of influence of the type of abnormality on the entire manufacturing line.

The analysis section113accumulates, in the storage section120or the like, the conversion results by the conversion section112of the respective images201of the different analysis targets202of the same product. The analysis section113can detect the apparatus type by analysis based on the plurality of accumulated conversion results. Specifically, for example, the analysis section113acquires each of the images201of a plurality of same components flowing on the manufacturing line. The analysis section113converts the plurality of images201of the same component into respective recognition classes and accumulates the conversion results. The analysis section113calculates the average value of the vectors of the certainty factors of the recognition classes, which are the plurality of accumulated conversion results. The analysis section113may detect, using the knowledge database122, the apparatus type corresponding to the recognition class having the highest average value of the certainty factors.

The output section114outputs the analysis result obtained by the analysis section113. Specifically, the output section114outputs the apparatus type detected through the analysis performed by the analysis section113by displaying the apparatus type on the operation display section140. The output section114may output the apparatus type detected by the analysis section113by transmitting the apparatus type to an external apparatus via the communication section130.

FIG.4is a flowchart illustrating an operation of the analysis system10. This flowchart can be executed by the controller110of the analysis apparatus100in accordance with a program.

The controller110acquires the image201by receiving it from the imaging apparatus200(S101).

The controller110converts the image201into a vector of the certainty factor of each recognition class using the learned model121(S102).

The controller110performs analysis for detecting an apparatus type from the conversion results in step S102using the knowledge database122(S103).

The controller110outputs the apparatus type detected through the analysis by, for example, displaying the apparatus type on the operation display section140(S104).

Second Embodiment

A second embodiment will be described. The present embodiment is different from the first embodiment in the following points. In the present embodiment, the knowledge database122used for analysis by the analysis section113is switched based on information on a manufacturing process for a product that is the analysis target202included in the image201. Alternatively, in the present embodiment, the knowledge database122used for analysis by the analysis section113is limited to a part of the knowledge database122. In other respects, the present embodiment is the same as the first embodiment, and therefore redundant description will be omitted or simplified.

FIG.5is a functional block diagram of the controller110. The controller110functions as the acquisition section111, the conversion section112, the analysis section113, the output section114, and a data control section115.

The acquisition section111acquires the image201by, for example, receiving it from the imaging apparatus200.

The data control section115acquires information on the manufacturing process for the product included in the image201. The information on the manufacturing process for the product is, for example, information indicating a painting process for a flat plate that is a component. The information on the manufacturing process for the product can be registered by being stored in the storage section120in advance in association with the imaging apparatus200that captures the image201of the product in the manufacturing process. For example, the data control section115can acquire the information on the manufacturing process for the product associated with the imaging apparatus200that is a reception destination of the image201by reading the information from the storage section120.

The data control section115generates a control signal for the knowledge database122based on the information on the manufacturing process for the product included in the image201. The control signal for the knowledge database122is a signal for the analysis section113to perform switching or the like of the knowledge database122used for analysis.

The analysis section113switches the knowledge database used for analysis by the analysis section113to the knowledge database122corresponding to the manufacturing process for the product included in the image201. The analysis section113switches the knowledge database used for analysis in accordance with the control signal for the knowledge database122. For example, when the knowledge database122is a comprehensive database corresponding to all analysis-target manufacturing processes in a manufacturing line, the data control section115may limit the database used for analysis by the analysis section113. The data control section115may limit the database used for analysis by the analysis section113to a part of the knowledge database122. The analysis target is a product or a component. The data control section115may limit the database used for analysis to a part of the knowledge database122in accordance with the control signal for the knowledge database122.

The analysis section113performs analysis by using the knowledge database122after switching in accordance with the control signal for the knowledge database122. The analysis section113may perform analysis by using only a part of the knowledge database122in accordance with the control signal for the knowledge database122.

FIG.6is a flowchart illustrating an operation of the analysis system10. This flowchart can be executed by the controller110of the analysis apparatus100in accordance with a program.

The controller110acquires the image201and information on a manufacturing process for a product included in the image201, by receiving them from the imaging apparatus200(S201).

The controller110converts the image201into a vector of the certainty factor of each recognition class using the learned model121(S202).

Based on the information on the manufacturing process for the product, the controller110switches the knowledge database122used for the analysis for detecting the type of the apparatus that has caused the abnormality (S203).

The controller110performs analysis for detecting the apparatus type from the conversion results in step S202by using the knowledge database122afler the switching (S204).

The controller110outputs the apparatus type detected through the analysis by, for example, displaying the apparatus type on the operation display section140(S205).

Third Embodiment

A third embodiment will be described. The present embodiment is different from the first embodiment in the following points. In the first embodiment, the analysis target is a component or the like. In the first embodiment, the predetermined recognition class into which the image201is converted is the type of abnormality. In the first embodiment, the knowledge database122indicates the relationship between the type of abnormality and the apparatus type, and the apparatus type is detected through analysis. In the present embodiment, the analysis target202is a person. In the present embodiment, the predetermined recognition class into which the image201is converted is a person's behavior. In the present embodiment, the knowledge database122indicates a relationship between the person's behavior and the applicability to a detection target behavior, and the applicability to the detection target behavior is detected through analysis. The person's behavior and the applicability to the detection target behavior are, for example, relevance or irrelevance of an abnormal behavior. In other respects, the present embodiment is the same as the first embodiment, and therefore redundant description will be omitted or simplified.

FIG.7is a functional block diagram of the controller110. The controller110functions as the acquisition section111, the conversion section112, the analysis section113, and the output section114.

The acquisition section111acquires the image201by, for example, receiving it from the imaging apparatus200. The image201includes an image of a person (e.g., a worker on a manufacturing line) as the analysis target202. The image201may include a plurality of persons as the analysis targets202. The image201may include images of apparatuses, floors, chairs, and the like.

The conversion section112inputs a plurality of images201(e.g., moving images) to the learned model read out from the storage section120, and converts them into person's behaviors that are predetermined recognition classes. Specifically, the conversion section112converts the plurality of images201into vectors of the certainty factors of respective recognition classes. The reason why the plurality of images201are converted into the predetermined recognition classes is that the recognition class is the person's behavior. For this reason, it is considered that the temporal transition (temporal change) of the normal image201is necessary in order to identify the behavior. Note that the conversion section112may convert one image201into a recognition class. This is because, for example, a behavior of a worker such as hitting another worker can be identified from one image201.

The learned model121may be generated in advance by training a neural network using a relatively large amount of train data and stored in the storage section120. The train data is a combination of the plurality of images201and recognition classes that are ground truth labels corresponding to the plurality of images201.

The training of the neural network using the train data of the combination of the plurality of images201and the recognition classes which are the ground truth labels corresponding to the plurality of images201, and the conversion of the plurality of images201into vectors of the certainty factors of the respective recognition classes using the learned model121can use, for example, a long short-term memory (LSTM) which is a well-known technique.

The predetermined recognition classes are person's behaviors, and include, for example, “not moving in a lying position”, “not moving in a sitting position”, and “hitting another person”. It is preferable that the following train data is used in a case where the learned model121is generated. As the train data, it is preferable to use a combination of an image201of an abnormal behavior such as “not moving in a lying position” or “hitting another person” and a ground truth label of a recognition class of the abnormal behavior. Thus, it is possible to efficiently improve accuracy in detection of the abnormal behavior performed by the conversion section112. The predetermined recognition classes may include a “normal behavior”.

The analysis section113uses the conversion results obtained by the conversion section112and the knowledge database122stored in the storage section120to perform analysis for detecting the applicability to the detection target behavior. That is, the analysis section113analyzes a correspondence relationship between the recognition classes and the predetermined information (detection results) by using the knowledge database122. Thus, the applicability to the detection target behavior is detected. The applicability to the detection target behavior detected through the analysis performed by the analysis section113constitutes predetermined information. The applicability to the detection target behavior includes relevance or irrelevance of the abnormal behavior (whether it is an abnormal behavior). Hereinafter, for the sake of convenience in the description, description will be given on the assumption that the applicability to the detection target behavior is relevance or irrelevance of the abnormal behavior. The knowledge database122may be a data group (or apparatus) that indicates (stores) a correspondence relationship between the recognition classes and predetermined information. The predetermined information includes information on relevance or irrelevance of the abnormal behavior. Specifically, in a case where the certainty factor of the recognition class of “not moving in a lying position” is the highest in the conversion results obtained by the conversion section112, the analysis section113identifies and detects relevance or irrelevance of an abnormal behavior corresponding to “not moving in a lying position”. The relevance or irrelevance of the abnormal behavior corresponding to “not moving in a lying position” is whether the behavior corresponding to “not moving in a lying position” is an abnormal behavior. The analysis section113detects relevance or irrelevance of the abnormal behavior corresponding to “not moving in a lying position” by identifying it using the knowledge database.

The correspondence relationship between the recognition class and the relevance or irrelevance of the abnormal behavior included in the knowledge database122can be set as follows based on, for example, a past care record or a past monitoring record. The correspondence relationship between the recognition class and the relevance or irrelevance of the abnormal behavior can be appropriately set by determining whether it is appropriate to determine that the recognition class corresponds to the abnormal behavior and issue a warning or the like.

The correspondence relationship between the recognition class and the relevance or irrelevance of the abnormal behavior included in the knowledge database122can be set in accordance with a place where the image201is captured (hereinafter, referred to as an “analysis environment”). For example, in a case where the analysis environment is in each process of the production line, the correspondence relationship between the recognition class of “not moving in a lying position” and the detection result of “relevance of the abnormal behavior” can be set. Furthermore, for example, in a case where the analysis environment is on a bed in a hospital, the correspondence relationship between the recognition class of “not moving in a lying position” and the detection result of “irrelevance of the abnormal behavior (normal)” can be set.

The degree of importance of the type of abnormality may be set to each recognition class. The analysis section113performs weighting by multiplying the certainty factor of the recognition class indicating the type of abnormality by the degree of importance. The analysis section113may use, as an analysis result, relevance or irrelevance of an abnormal behavior that corresponds to the recognition class having the highest certainty factor after weighting in the knowledge database122. The degree of importance can be appropriately set by an experiment or the like from an any viewpoint. For example, the degree of importance can be appropriately set from the viewpoint of the urgency of care (response) for a behavior of the recognition class.

The output section114outputs the analysis result obtained by the analysis section113. Specifically, the output section114outputs relevance or irrelevance of the abnormal behavior detected through the analysis performed by the analysis section113by displaying it on the operation display section140. The output section114may output the relevance or irrelevance of the abnormal behavior detected by the analysis section113by transmitting it to an external apparatus via the communication section130.

FIG.8is a flowchart illustrating an operation of the analysis system10. This flowchart can be executed by the controller110of the analysis apparatus100in accordance with a program.

The controller110acquires the plurality of images201by, for example, receiving them from the imaging apparatus200(S301).

The controller110converts the plurality of images201into vectors of the certainty factors of respective recognition classes using the learned model121(S302).

The controller110performs analysis for detecting relevance or irrelevance of the abnormal behavior from the conversion results in step S302by using the knowledge database122(S303).

The controller110outputs the relevance or irrelevance of the abnormal behavior detected through the analysis by, for example, displaying it on the operation display section140(S304).

Fourth Embodiment

A fourth embodiment will be described. The present embodiment is different from the third embodiment in the following points. In the present embodiment, the knowledge database122used for analysis by the analysis section113is switched based on information on the analysis environment included in the image201. The information on the analysis environment is information on a place where a person who is the analysis target202is present. Alternatively, in the present embodiment, the knowledge database122used for analysis by the analysis section113is limited to a part of the knowledge database122. In other respects, the present embodiment is the same as the third embodiment, and therefore redundant description will be omitted or simplified.

FIG.9is a functional block diagram of the controller110. The controller110functions as the acquisition section111, the conversion section112, the analysis section113, the output section114, and the data control section115.

The acquisition section111acquires the plurality of images201by, for example, receiving them from the imaging apparatus200.

The data control section115acquires information on an analysis environment which is information on a place where the image201is captured and where the person who is the analysis target202is present. The information on the analysis environment may be, for example, a workplace for a painting process for a flat plate in a manufacturing line or place on a bed in a hospital. The information on the analysis environment can be registered by being stored in advance in the storage section120in association with the imaging apparatus200that captures the image201. The data control section115can acquire the information on the analysis environment associated with the imaging apparatus200that is the reception destination of the image201by reading the information from the storage section120.

The data control section115generates a control signal for the knowledge database122based on the information on the analysis environment. The control signal for the knowledge database122is a signal for the analysis section113to switch the knowledge database122used for analysis.

The analysis section113switches the knowledge database used for the analysis by the analysis section113in accordance with the control signal for the knowledge database122. The analysis section113switches the knowledge database used for analysis by the analysis section113to the knowledge database122corresponding to the analysis environment where the image201is captured. For example, when the knowledge database122is a comprehensive database corresponding to a plurality of analysis environments, the data control section115may limit the database used for analysis by the analysis section113. The data control section115may limit the database used for analysis to a part of the knowledge database122for use in the analysis environment. The imaging environment is an analysis environment corresponding to the imaging apparatus200at the place where the person to be analyzed is imaged. The data control section115limits the database used for analysis by the analysis section113in accordance with the control signal for the knowledge database122.

The analysis section113performs analysis by using the knowledge database122after switching in accordance with the control signal for the knowledge database122. The analysis section113may perform analysis by using only a part of the knowledge database122in accordance with the control signal for the knowledge database122.

The relevance or irrelevance of the abnormal behavior may change depending on the imaging place where the person who is the analysis target202is present. However, by switching, in accordance with the analysis environment, the knowledge database122indicating the correspondence relationship between the behavior and relevance or irrelevance of the abnormal behavior, various situations of the monitoring target can be handled.

FIG.10is a flowchart illustrating an operation of the analysis system10. This flowchart can be executed by the controller110of the analysis apparatus100in accordance with a program.

The controller110acquires the plurality of images201and information on an analysis environment which is a place where the plurality of images201are captured (S401).

The controller110converts the images201into vectors of the certainty factors of respective recognition classes using the learned model121(S402).

Based on the information on the analysis environment, the controller110switches the knowledge database122used for the analysis for detecting the abnormality of relevance or irrelevance of the abnormal behavior (S403).

The controller110performs the analysis for detecting the relevance or irrelevance of the abnormal behavior from the conversion results in step S402by using the knowledge database122after the switching (S404).

The controller110outputs the relevance or irrelevance of the abnormal behavior detected through the analysis by, for example, displaying it on the operation display section140(S405).

The above-described embodiments have the following effects.

Any one of predetermined recognition classes is identified from input data using a trained machine learning model. The correspondence relationship between the recognition class and the predetermined information is analyzed using a database that stores the correspondence relationship between the recognition class and the predetermined information. Thus, in, for example, identifying a cause of a defect in a product using machine learning, it is possible to improve detection accuracy and versatility while suppressing an amount of training data.

Furthermore, it is assumed that the input data is an image and the database stores a correspondence relationship between a recognition class of a defect and an apparatus type or a manufacturing process. Then, the correspondence relationship between the recognition class and the apparatus type or the manufacturing process is analyzed using the database. Thus, in identification of an apparatus or a manufacturing process that has caused a defect in a product, using machine learning, it is possible to improve detection accuracy and versatility while suppressing the amount of training data.

Furthermore, it is assumed that the input data is an image, and the database stores a correspondence relationship between a recognition class of a behavior and a monitoring place. Then, the correspondence relationship between the recognition class of the behavior and the monitoring place is analyzed using the database. Thus, in identification of a monitoring place where a defect in a product has occurred, using machine learning, it is possible to improve detection accuracy and versatility while suppressing the amount of training data.

Furthermore, it is assumed that the input data is an image and the database stores a correspondence relationship between a recognition class of a behavior and a detection result of applicability to a detection target behavior. Then, the correspondence relationship between the recognition class and the applicability to the detection target behavior is analyzed using the database. Thus, in the detection of an abnormal behavior using machine learning, it is possible to improve detection accuracy and versatility while suppressing the amount of training data.

Furthermore, the machine learning model is set as a neural network. Then, the input data is input to the neural network, and the input data is converted into any one of the recognition classes and the certainty factor of the recognition class. Thus, in, for example, identifying the cause of the defect in the product using machine learning, the detection accuracy and versatility can be more simply improved while suppressing the amount of training data.

Furthermore, the database used for analysis is switched or limited to a part of the database in accordance with the information on the product included in the image or the information on the place where the person is present included in the image. Thus, it is possible to improve detection accuracy in, for example, identifying a cause of a defect in a product using machine learning.

Furthermore, when the certainty factor of the converted recognition class is less than a predetermined threshold value, the analysis is not performed. Thus, it is possible to suppress the occurrence of a false report and a report failure when the conversion of the input data into the recognition class fails.

Furthermore, the certainty factor is weighted in accordance with the degree of importance set for each converted recognition class. Then, the correspondence relationship between the recognition class having the highest certainty factor after weighting and the predetermined information is analyzed using the database. This can further suppress the occurrence of a report failure.

Furthermore, the identification results by the identification section of the respective images of different analysis targets of the same product are accumulated, and the apparatus type is detected through the analysis based on the plurality of accumulated identification results. Thus, it is possible to further improve the detection accuracy in, for example, identifying the cause of the defect in the product using the machine learning.

The description given above on the analysis system, the analysis program, the analysis apparatus, and the analysis method is about main components for describing the features of the above-described embodiments. The analysis system, the analysis program, the analysis apparatus, and the analysis method are not limited to the above-described configurations. The analysis system, the analysis program, the analysis apparatus, and the analysis method can be variously modified within the scope of the claims. In addition, a configuration included in a general abnormality detection system or the like is not excluded.

For example, some of the steps in the flowcharts described above may be omitted, and other steps may be added. Furthermore, some of the steps may be executed at the same time, and one step may be divided into a plurality of steps and executed.

In the embodiment, the conversion section112inputs the image201to the learned model121of the neural network. The case where the image201is converted into any one of the predetermined recognition classes by the convolution operation has been described as an example. However, the conversion section112compares the image of the non-defective product reproduced from the image including the analysis target202using an autoencoder with the image including the analysis target202. The conversion section112may convert the comparison result (difference data) into any one of predetermined recognition classes by inputting the comparison result to the learned model121of the neural network.

In addition, the conversion section112may estimate a joint point of a person from the image201by using an hourglass network, and detect a person's behavior based on the estimated joint point.

When the imaging apparatus200is replaced with a microphone, the conversion section112converts voice data into a predetermined recognition class. When the imaging apparatus200is replaced with an odor sensor, the conversion section112converts odor data into a predetermined recognition class. When the imaging apparatus200is replaced with a temperature sensor, the conversion section112converts temperature data into a predetermined recognition class. In these cases, the predetermined recognition classes can be, for example, a high-pitched (equal to or higher than a certain frequency) abnormal sound, a low-pitched (lower than the certain frequency) abnormal sound, and a burned odor. The predetermined recognition classes may be a chemical odor, a high temperature equal to or higher than a certain threshold value, a low temperature lower than a certain threshold value, and the like.

Furthermore, the means and methods for performing various kinds of processing in the above-described system can be implemented by a dedicated hardware circuit. Alternatively, the means and methods for performing various kinds of processing in the above-described system can also be implemented by a programmed computer. The program may be provided by a computer-readable recording medium such as a universal serial bus (USB) memory or a digital versatile disc (DVD)-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is generally transferred to and stored in a storage section such as a hard disk. Furthermore, the program may be provided as a single piece of application software, or may be incorporated, as a function, into software of an apparatus such as an abnormality detection apparatus.

This application is based on Japanese Patent Application (Japanese Patent Application No. 2021-002033) filed on Jan. 8, 2021, the disclosure of which is incorporated herein by reference in its entirety.

REFERENCE SIGNS LIST