Patent Description:
The present disclosure relates to data classification models based on artificial intelligence in industrial automation. Data classification models are used to classify objects into various categories based on different features reflected in information associated with objects. One such application in relation to data classification, relates to quality inspection systems in production plants.

In production plants, quality inspection of the work in progress items is performed after manufacturing operations. Currently, this can be performed using virtual inspections (i.e., using a data classification model) and/or physical inspections (i.e., using an inspection or testing process such as X-ray inspection, ultrasound testing, manual inspection by an operator, etc.). Usually, physical inspections require specialized, mostly expensive machines and additional material handling. Moreover, the physical inspections are non-value-adding, add complexity and prolong the manufacturing process (in comparison to virtual inspections). Therefore, usually, virtual inspections are preferred on all work in progress pieces and physical inspections are performed only when virtual inspections are not possible, or on random work in progress pieces or on final pieces after all manufacturing processes have been completed.

Conventionally, prior to deployment, such data classification models are trained using historic data which includes data associated with known fault categories. However, in case where new fault categories are identified during production, the deployed classifier needs to be extended for consideration of these new categories. Especially with increasing flexibility in production plants, this can become a major challenge. Accordingly, there is a need for a method and device which addresses the abovementioned issue.

<CIT> describes a method for inspecting displays that are produced for mobile devices. The method includes using a neural network model which analyses trace data from the manufacturing process and retraining of the model using a topological analysis dataset.

Active learning for defect classifiers while using selected data points and corresponding labels is known e. from <CIT>.

In <CIT> a method for accelerated training of a machine learning based model for semiconductor applications by using non-nominal instances of specimens (i.e., defective specimens) is described.

The current disclosure describes a method accordingly to claim <NUM>, a virtual inspection device according to claim <NUM> and a non-transitory storage medium according to claim <NUM> which address the above-mentioned aspects.

Accordingly, the current disclosure describes a method of virtually inspecting a quality of a product in a production environment using one or more classifier models. The method comprises receiving production information associated with the product, wherein the production information is indicative of an operation performed on the product, in relation to a first process of the production environment, determining a label for the product using a first classifier model based on the received information associated with the product, comparing the determined label against inspection data, wherein the inspection data is associated with an inspection of the product, and storing the production information and the inspection data associated with the product in an extension buffer, based on the comparison of the determined label and the inspection data, for retraining the first classifier model. Accordingly, the current disclosure describes a method of storing new fault category information in an extension buffer which is then dynamically used to retrain the model. Accordingly, this allows for retraining of classifier models without human intervention. Additionally, this allows for effective and efficient inspection of work in progress items or products using the retrained classifier models.

In an example, the first classifier model is configured to be retrained using one or more samples present in the extension buffer for adding one or more new labels, upon detecting a predefined number of samples in the extension buffer, each sample from the one or more samples comprising production information and corresponding inspection data associated with a corresponding product.

In an example, the method further comprises determining a number of samples in the extension buffer and retraining the first classifier model using the one or more samples in the extension buffer when the number of samples is greater than a predefined threshold value, wherein each sample from the one or more samples comprising production information and corresponding inspection data associated with a corresponding product. Accordingly, the current disclosure allows for automatic retraining based on a predetermined number of samples in the extension buffer. Accordingly, this allows for quick retraining of models to identify and detect new fault categories.

In an example, the production environment includes one or more processes wherein each process from the one or more processes includes one or more operations to be performed. In another example, wherein the inspection data includes a new label indicative of a condition of product determined during the inspection of the product.

In another aspect, the current disclosure describes a virtual inspection device for virtually inspecting a quality of a product in a production environment using one or more classifier models. The virtual inspection device comprises a network interface for receiving production information associated with the product, wherein the production information is indicative of an operation performed on the product, in relation to a first process of the production environment and one or more processors connected to a memory module. The one or more processors are configured to determine a label for the product using a first classifier model based on the received information associated with the product, compare the determined label against inspection data, wherein the inspection data is associated with an inspection of the product, and store the production information and the inspection data associated with the product in an extension buffer of the memory module, based on the comparison of the determined label and the inspection data, for retraining the first classifier model.

In yet another aspect, the current disclosure describes a non-transitory storage medium for virtually inspecting a quality of a product in a production environment using one or more classifier models. The non transitory storage medium comprises a plurality of instructions, which when executed on one or more processors, cause the one or more processors to determine a label for the product using a first classifier model based on the received information associated with the product, compare the determined label against inspection data, wherein the inspection data is associated with an inspection of the product, and store the production information and the inspection data associated with the product in an extension buffer, based on the comparison of the determined label and the inspection data, for retraining the first classifier model. The advantages of the method apply to the device and the non-transitory storage medium described herein. These aspects are further described in relation <FIG>.

<FIG> illustrates an example production environment <NUM> including a plurality of manufacturing or industrial processes (shown in the figure as process <NUM>, process <NUM>, process <NUM>). Each process from the one or more processes (<NUM>, <NUM>, <NUM>) includes one or more operations to be performed on the work in progress item <NUM>. Production environment <NUM> herein refers to any environment where the one or more industrial processes such as manufacturing, refining, smelting, assembly of equipment may take place, and accordingly includes process plants, automobile factories, integrated circuit fabrication industries, warehouses, etc. The production environment <NUM> includes a production control system (also referred to as industrial automation system) responsible for monitoring and controlling the industrial processes. The production control system comprises a plurality of industrial devices and subsystems such as control devices, field devices, operator stations, process historians, etc. Control devices includes process controllers, programmable logic controllers, supervisory controllers, automated guided vehicles, robots, operator devices, etc. Control devices are connected to a plurality of field devices (not shown in figure) such as actuators and sensor devices for monitoring and controlling industrial processes in the industrial facility. These field devices can include flowmeters, value actuators, temperature sensors, pressure sensors, etc. The control devices may be connected to each other via a control network (realized via wired and/or wireless networks). Operator stations display the status of the industrial plant to an operator and for allowing the operator to define setpoints for the control of the industrial processes in the facility.

Additionally, the production control system includes one or more quality inspection stations (including quality inspection devices and equipment) for physically inspecting quality of the work in progress product or item after a corresponding process. Examples of inspection devices in such inspection stations include X-Ray devices, ultrasonic testing devices, etc. For example, as shown in the figure, inspection station <NUM> is for checking the quality of the work in progress product or item after the operations of the process <NUM> have been carried out on the work in progress product. Similarly, inspection station <NUM> is for checking the quality of the work in progress product or item after the operations of the process <NUM> have been carried out on the work in progress product. Similarly, inspection station <NUM> is for checking the quality of the work in progress product or item after the operations of the process <NUM> have been carried out on the work in progress product. Additionally, since process <NUM> is the last process to be carried out on the work in progress product, the inspection station <NUM> serves as the final inspection station and all the necessary quality checks for approval of the product is carried out here. In addition to the quality inspections stations, the production control system includes one or more virtual inspection devices (shown in the figure as virtual inspection devices <NUM> and <NUM>). The one or more virtual inspection devices are capable of determining or estimate a quality of the work in progress item or product after a corresponding process based on production information associated with the work in progress product. The virtual inspection devices include one or more AI based classifier models (shown in the figure as classifier <NUM> and classifier <NUM>) which are trained to determine a label or category associated with the work in progress product based on the production information. Based on the result from the physical inspection from the quality inspection station or the virtual inspection device, the work in progress product is moved or transferred from the preceding process to the next process. The result of the virtual inspection device along with the production information of the associated work in progress product is stored in a sample buffer <NUM>.

Additionally, the production control system includes a label extension management module <NUM> configured to compare the result from the final inspection, i.e., from the inspection station <NUM> against the results from the virtual inspection devices <NUM> and <NUM> and, if required, retrain the classifier models <NUM> and <NUM> based on the comparison. This is further explained in the description of the <FIG>.

<FIG> illustrates a method <NUM> of virtually inspecting a quality of a product <NUM> in the production environment <NUM> using one or more classifier models (<NUM>, <NUM>). For ease in explanation, the method is described in relation to a work in progress product (shown in <FIG> as <NUM>) after the operations of the first process <NUM> has been performed on work in progress product <NUM>. Accordingly, the work in progress product <NUM> is up for virtual inspection by the virtual inspection device <NUM>. Accordingly, the method <NUM> is performed by the virtual inspection device <NUM> and the label extension management module <NUM>.

At step <NUM>, the virtual inspection device <NUM> receives production information associated with the product <NUM>. The production information is indicative of an operation performed on the product <NUM>, in relation to the first process <NUM> of the production environment <NUM>. Examples of production information includes process data such as time series data in relation to one or more process parameters associated with the first process, visual data of the work in the progress product <NUM> after the completion of the operations of the first process, etc. In an example, the process data is transmitted to the virtual inspection device <NUM> by a control device associated with the first process <NUM>. In another example, the visual data of the work in the progress product <NUM> is transmitted to the virtual inspection device <NUM> from one or more image capturing devices capable of capturing a plurality of images of the product <NUM> in a plurality of spectrums.

At step <NUM>, the virtual inspection device <NUM> determines a label (also referred to as prediction) for the product <NUM> using a first classifier model <NUM> based on the received production information associated with the product <NUM>. As mentioned previously, the virtual inspection device <NUM> includes a first classifier model <NUM> which is trained to determine a label associated with a product based on the received production information. The label is indicative of a condition of the product/work in progress item. For example, the label indicates if the product meets the approved quality or a fault associated with the product.

Then, at step <NUM>, the label management extension module <NUM> compares the determined label against inspection data. The inspection data is associated with an inspection of the product <NUM> performed subsequent to a process from the one or more processes. For example, the inspection is the final inspection performed on the corresponding product <NUM> at the inspection station <NUM>. The inspection data from the inspection includes information regarding the condition of the work in progress product <NUM>. For example, the inspection data indicates that the work in progress product <NUM> meets quality standards specified in the inspection. In another example, the inspection data includes fault data indicative of one or more reasons (i.e., faults or deficiencies) due to which the work in progress product <NUM> failed inspection. In an example, the inspection data includes a label indicative of a condition of product <NUM> determined during the inspection <NUM>, <NUM> of the product <NUM>. Accordingly, the label management extension module <NUM> compares the label from the classifier associated with the work in progress against the inspection data associated with the work in progress product <NUM>, to check if the label matches the inspection data or not.

Then, at step <NUM>, the label extension management module <NUM> stores the production information and the inspection data associated with the product <NUM> in an extension buffer <NUM>, based on the comparison of the determined label and the inspection data as mentioned above. Accordingly, based on the comparison mentioned above to check if the label matches the inspection data, the extension management module <NUM> determines if the production information and the inspection data associated with the product <NUM> is to be stored in the extension buffer <NUM>. In cases where the determined label matches the inspection data (i.e. the condition of work in progress as determined by the classifier model matches the condition as determined during the inspection), the label extension management module <NUM> determines that the classifier model <NUM> is working properly and therefore the production information and inspection data is discarded and not stored in the extension buffer <NUM>. In cases where the determined label does not match the inspection data (i.e. the condition of work in progress as determined by the classifier model does not match the condition as determined during the inspection), the label extension management module <NUM> determines that the classifier model <NUM> is not working properly and therefore the production information and inspection data is stored in the extension buffer <NUM>, for retraining the first classifier model. In an example, the extension management module <NUM> additionally checks if the label in the inspection data is present in the set of labels associated with the first classifier model <NUM>. If the label in the inspection data is present within the set of labels, the production information and the inspection data is discarded. It may be noted by a person skilled in the art that while the inspection data is discarded in the context of the current method, the discarded inspection data may still be used for training untrained models. If the label in the inspection data is not present within the set of labels associated with the first classifier model, the production information and the inspection data is stored in the extension buffer <NUM>.

Accordingly, a plurality of such pairs of production information and inspection data related to a plurality of work in progress products is stored in the extension buffer <NUM>. The pair of production information and inspection data of an associated work in progress product is known as a sample. Accordingly, the extension buffer <NUM> include a plurality of samples. The label extension management module <NUM> utilizes the one or more samples in the extension buffer <NUM> to retrain the first classifier model <NUM> for adding one or more new labels, upon detecting a predefined number of samples in the extension buffer <NUM>.

In an example, the label extension management module <NUM> is configured to receive the inspection data from the inspection station <NUM> after the inspection of the work in progress product <NUM> is completed. In an example, the label extension management module <NUM> is configured to retrieve the production information associated with the product <NUM> along with the label determined by the first classifier model <NUM> from the sample buffer upon receiving the inspection data from the inspection station <NUM>.

In an example, the label extension management module <NUM> determines a number of samples in the extension buffer <NUM>. And based on the number of samples, the label extension management module <NUM> retrains the first classifier model <NUM> using the one or more samples in the extension buffer <NUM>. In an example, the retraining is performed only when the number of samples is greater than a predefined threshold value. In an example, each sample in the extension buffer is associated with a particular category, category indicative of a particular condition or fault. Accordingly, the label extension management module <NUM> is configured to retrain the classifier model based on samples associated with a particular category based on the number of samples associated with that particular category. In an example, the classifier model <NUM> is trained utilizing a few-shot learning (FSL) methodology that allows for dynamic extension of the classifier from the set of initial labels. Accordingly, in an example, the extension buffer <NUM> is connected to an extension pipeline <NUM> which is a FSL based pipeline for classifier extension and accordingly reduces computational complexity compared to regular training.

It is to be noted that while the above method is explained in relation to virtual inspection device <NUM> and classifier model <NUM>, the method <NUM> is applicable to virtual inspection device <NUM> and classifier model <NUM> as well. Additionally, while the above method is explained in relation to first process <NUM> and subsequent inspection <NUM> or <NUM>, the same is applicable to other processes and inspections in the production environment <NUM>. It is also to be noted that while the sample buffer <NUM> and extension buffer <NUM> are shown as two different modules, both the buffers can be implemented in the same buffer system. In this case a sample is not 'moved' between the buffers by the label extension management module <NUM> but rather annotated with the respective unknown label and, possibly, marked as relevant for extension (i.e., classifier retraining) by using flags.

In an example, after retraining the classifier model, the retrained model instance is tested prior to replacing the previous version of the classifier model in operation. A validation test is conducted first and the retrained model is deployed only performed in case that the overall accuracy of the retrained model exceeds a certain predefined threshold. It is to be noted that while the above method <NUM> is explained in relation to virtual inspection device <NUM> and label extension management module <NUM>, the above method may be realized in a single device or a plurality of devices. For example, the method <NUM> may be in a virtual inspection device <NUM> as shown in <FIG>. Accordingly, the present disclosure can take a form of a computer program product comprising program modules accessible from computer-usable or computer-readable medium storing program code for use by or in connection with one or more computers, processing units, or instruction execution system.

Accordingly, in another aspect, the current disclosure describes a virtual inspection device <NUM> as shown in <FIG>, for virtually inspecting a quality of the product <NUM> in the production environment <NUM> using one or more classifier models (<NUM>, <NUM>). The virtual inspection device <NUM> comprises a network interface <NUM> for receiving production information associated with the product <NUM>. As mentioned previously, the production information is indicative of an operation performed on the product <NUM>, in relation to a first process <NUM> of the production environment <NUM>. Additionally, the virtual inspection device <NUM> includes one or more processors <NUM> connected to a memory module <NUM> (also referred to as non-transitory memory medium <NUM>). The memory module <NUM> comprises a plurality of instructions, which when executed on the one or more processors <NUM>, cause the one or more processors <NUM> to determine a label for the product using a first classifier model <NUM> based on the received information associated with the product <NUM>, compare the determined label against inspection data, wherein the inspection data is associated with an inspection (<NUM>, <NUM>) of the product <NUM>, and store the production information and the inspection data associated with the product <NUM> in an extension buffer <NUM> of the memory module, based on the comparison of the determined label and the inspection data, for retraining the first classifier model.

Claim 1:
A method (<NUM>) of virtually inspecting a quality of a product (<NUM>) in a production environment (<NUM>) using one or more classifier models (<NUM>, <NUM>), the method (<NUM>) comprising:
a. receiving (<NUM>) production information associated with the product (<NUM>), wherein the information is indicative of an operation performed on the product (<NUM>), in relation to a first process (<NUM>) of the production environment (<NUM>);
b. determining (<NUM>) a label for the product using a first classifier model (<NUM>) based on the received information associated with the product (<NUM>);
c. comparing (<NUM>) the determined label against inspection data, wherein the inspection data is associated with an inspection (<NUM>, <NUM>) of the product (<NUM>); and
d. storing (<NUM>) the production information and the inspection data associated with the product (<NUM>) in an extension buffer (<NUM>), based on the comparison of the determined label and the inspection data, for retraining the first classifier model,
characterized in,
e. retraining the first classifier model (<NUM>) using one or samples present in the extension buffer (<NUM>) for adding one or more new labels, upon detecting a predefined number of samples present in the extension buffer (<NUM>),
wherein each sample from the one or more samples comprising production information and corresponding inspection data associated with a corresponding product, the corresponding inspection data including a new label indicative of a condition of the corresponding product (<NUM>) determined during the inspection (<NUM>, <NUM>) of the product (<NUM>).