Anomaly detection with predictive normalization

A method is provided for model training to detect defective products. The method includes sampling training images of a product to (i) extract image portions therefrom made of a center patch and its context and (ii) black-out the center patch. The method further includes performing unsupervised back-propagation training of a Contextual Auto-Encoder (CAE) model using (i) the image portions with the blacked-out center patch as an input and, (ii) the center patch as a target output and, (iii) an image-based loss function, to obtain a trained CAE model. The method also includes sampling positive and negative center-patch-sized portions from the training images. The method additionally includes normalizing, using the trained CAE model, the positive and negative center-patch-sized portions. The method further includes performing supervised training of a classifier model using the normalized positive and negative center-patch-sized portions to obtain a trained supervised classifier model for detecting the defective products.

BACKGROUND

Technical Field

The present invention relates to anomaly detection and more particularly to anomaly detection with predictive normalization.

Description of the Related Art

Detecting small and faint anomalies is a difficult problem for a human and one that benefits greatly from automation. For example, x-ray images of metallic objects may reveal tiny hidden micro-fractures that have very low contrast with the surrounding pixel intensities. Detection of such anomalies is difficult for humans and automated systems. Increasing the accuracy of such detection is very valuable for quality assurance and quality control.

SUMMARY

According to an aspect of the present invention, a computer-implemented method is provided for model training to detect defective products. The method includes sampling, by a hardware processor, training images of a product to (i) extract image portions therefrom made of a center patch and its context and (ii) black-out the center patch. The method further includes performing, by the hardware processor, unsupervised back-propagation training of a Contextual Auto-Encoder (CAE) model using (i) the image portions with the blacked-out center patch as an input and, (ii) the center patch as a target output and, (iii) an image-based loss function, to obtain a trained CAE model. The method also includes sampling, by the hardware processor, positive and negative center-patch-sized portions from the training images. The method additionally includes normalizing, by the hardware processor using the trained CAE model, the positive and negative center-patch-sized portions. The method further includes performing, by the hardware processor, supervised training of a classifier model using the normalized positive and negative center-patch-sized portions to obtain a trained supervised classifier model for detecting the defective products.

According to another aspect of the present invention, a computer program product is provided for model training to detect defective products. The computer program product includes a non-transitory computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a computer to cause the computer to perform a method. The method includes sampling, by a hardware processor of the computer, training images of a product to (i) extract image portions therefrom made of a center patch and its context and (ii) black-out the center patch. The method further includes performing, by the hardware processor, unsupervised back-propagation training of a Contextual Auto-Encoder (CAE) model using (i) the image portions with the blacked-out center patch as an input and, (ii) the center patch as a target output and, (iii) an image-based loss function, to obtain a trained CAE model. The method also includes sampling, by the hardware processor, positive and negative center-patch-sized portions from the training images. The method additionally includes normalizing, by the hardware processor using the trained CAE model, the positive and negative center-patch-sized portions. The method further includes performing, by the hardware processor, supervised training of a classifier model using the normalized positive and negative center-patch-sized portions to obtain a trained supervised classifier model for detecting the defective products.

According to yet another aspect of the present invention, a computer processing system is provided for model training to detect defective products. The computer processing system includes a memory device including program code stored thereon. The computer processing system further includes a hardware processor, operatively coupled to the memory device, and configured to run the program code stored on the memory device to sample training images of a product to (i) extract image portions therefrom made of a center patch and its context and (ii) black-out the center patch. The processor is further configured to run the program code to perform unsupervised back-propagation training of a Contextual Auto-Encoder (CAE) model using (i) the image portions with the blacked-out center patch as an input and, (ii) the center patch as a target output and, (iii) an image-based loss function, to obtain a trained CAE model. The processor is also configured to run the program code to sample positive and negative center-patch-sized portions from the training images. The processor is additionally configured to run the program code to normalize, using the trained CAE model, the positive and negative center-patch-sized portions. The processor is further configured to run the program code to perform supervised training of a classifier model using the normalized positive and negative center-patch-sized portions to obtain a trained supervised classifier model for detecting the defective products.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

In accordance with the present principles, systems and methods are provided for anomaly detection with predictive normalization. In particularly useful embodiments, systems and methods can be configured to correct defects in products, such as, e.g., manufactured products.

In one embodiment, a product is inspected for defects using a reconstructor and contrastor. The reconstructor can include an encoder-decoder arrangement employed to reconstruct images of a product containing defects. The encoder-decoder arrangement is trained from examples that contain no defects. Thus, the encoder-decoder arrangement will use an incomplete image of the product, and reconstruct the image according to the defectless training. As a result, a reconstructed image of the product can be produced where the reconstructed image depicts the product having no defects.

The contrastor can then determine a difference between the reconstructed image and an original image of the product. If there is a substantial difference between the original image and the reconstructed image, then the product is considered not defectless. In other words, by failing to match a defectless reconstructed image of the product, it can be determined that the product does contain some anomaly or defect.

Because the systems and methods employed determine if a product fails to match a defectless reconstruction, an encoder-decoder arrangement can be trained with only normal, defectless images of the product. Thus, training samples can be found in abundance, and thus training the encoder-decoder arrangement can be cheap and efficient.

Referring now in detail to the figures in which like numerals represent the same or similar elements and initially toFIG. 1, a high-level system/method for correcting defective products is illustratively depicted in accordance with one embodiment of the present principles.

In one embodiment, a manufacturing plant10manufactures products. The manufactured products may be any physical product. The manufactured products can sometimes be manufactured with defects. Thus, the manufactured products can be sent to a quality assurance system20.

The quality assurance system20inspects the manufactured products to ensure that the products live up to quality standards with respect to issues such as defects and anomalies in the products. Such defects can include visual defects, such as, e.g., cracks, holes, protrusions, discolorations, or any feature that is not part of the original design of the product. The quality assurance system20can include an automated system for performing the inspections, such as, e.g., an automated visual system including a camera. However, other inspection systems can be used, including, e.g., radar, infrared, ultrasound, or other detection methods. Based on results, the quality assurance system20can automatically take steps to correct the product defects. Such steps can include, e.g., discarding the defecting product, automatically listing the product as defective and requiring refurbishment, alerting an operator to the defect, or any other corrective action.

According to an aspect of the present invention, the corrective action can include listing the product as defective and sending the product back to the manufacturing plant10to be recycled or refurbished. However, if the product is not defective, the product can be forwarded to a shipping and packaging system30to be sent to a customer.

Referring now toFIG. 2, a system/method for detecting and correcting defects and anomalies is illustratively depicted in accordance with an embodiment of the present principles.

According to aspects of the present invention, defects and anomalies in an item are captured by an imaging device100. The imaging device100captures images of an item for analysis to facilitate recognizing defects and anomalies. For example, the imaging device100can include, e.g., a camera device or multiple camera devices for capturing images of either the entire item or a portion of the item. Accordingly, multiple imaging devices100can be used to concurrently capture images of the item from multiple perspectives. Thus, the entirety of the item, such as, e.g., an entire surface, or for suitable imaging devices100, an entirety of the depth (as can be the case for imaging devices100including, e.g., magnetic resonance imaging (MRI) or computer aided tomography, etc.) can be captured by the imaging device100. Alternatively, a single imaging device100can be used, and the item and the imaging device100can be moved relative to each other to capture images from multiple perspectives by, e.g., conveyor, gimbal, or moving stage. The imaging device100can also include, e.g., an infrared sensor, a radar sensor, an ultrasound sensor, a light detection and ranging (LIDAR) sensor, among others. Thus, the imaging device100can generate an image of the item for analysis.

The image can then be analyzed by an anomaly detection and tagging system200. The anomaly detection and tagging system200process the image to recognize the presence of any potential anomalies with the item according to the image from the imaging device100. Here, an anomaly is considered to be a feature of the item, as depicted in the image, that deviates from what is usually present at that location of the item. For example, the anomaly detection and tagging system200can compare the image of the item to defectless items. Therefore, when the anomaly detection and tagging system200analyzes an image of an item that is different from the a defectless item, the anomaly detection and tagging system200will determine where in the image and on the item the difference is from the defectless item. The anomaly detection and tagging system200can then tag the feature of the item that is different from the defectless item as a potential anomaly or defect.

Information regarding the tagged anomaly can then be provided to an anomaly correction system300to take corrective action. The corrective action can depend on the severity of the defects. For example, if the item is a manufactured product, the production line may be stopped, or the item may be discarded or otherwise removed for recycling or refurbishment. A threshold may be used as well, such that if a threshold number of anomalies are tagged, more drastic action may be taken. For example, if the item is a manufactured product, reaching a first threshold number of anomalies can result in the item being discarded, while reaching a second threshold can result in a halting of the entire production line. Alternatively, or in addition, the anomaly correction system300can include a notification device to notify an operator of the item having tagged anomalies.

Referring now toFIG. 3, a system/method for detecting and correcting defects and anomalies using a reconstructor and contrastor is illustratively depicted in accordance with the present principles.

According to aspects of the present invention, an image of an item taken by an imaging device100can be provided to both a reconstructor220and a contrastor230. The reconstructor220can receive the image of the item, e.g., in a memory or storage, and generate a reconstructed image of the item that does not include any defects or anomalies such that the reconstructed image can be contrasted with the original image at the contrastor230.

The reconstructor220will receive the image of the item and reconstruct the image to remove any defects or anomalies. For example, the reconstructor can divide the image in multiple smaller portions of the image, partially mask a region in each smaller portion, and reconstruct each portion. However, other methods of reconstructing the image are contemplated. The reconstructor220can include, e.g., a processor to perform reconstruction on the image stored in a memory, such as, e.g., a storage, a random access memory (RAM), a buffer, or a cache, among others. The reconstructor220will, therefore, generate with the reconstructed image, a representation of the item that has no defects or anomalies. The reconstructed image can be stored or cached in a storage device, such as, e.g. a RAM, a buffer or a cache.

The reconstructed image will then be provided to the contrastor230along with the original image from the imaging device100. The contrastor230can then compare the reconstructed image with the original image. Because the reconstructed image is a defectless representation of the item, any differences between the reconstructed image and the original image will be detected by the contrastor230. Similar to the reconstructor220, the contrastor230can include, e.g., a processor, to perform the contrasting, and a storage, such as, e.g., a RAM, buffer or cache, to temporarily or permanently store the original image and the reconstructed image.

The contrastor230will then provide data regarding the detected difference to an anomaly tagging device240. The anomaly tagging device240will use data regarding the locations of difference between the reconstructed image and the original image to identify anomalies. Thus, the anomaly tagging device240can generate a tagged image of the item with anomalies identified and tagged.

The tagged image can be used by the anomaly correction system300to take corrective action regarding the anomalies, as described above. For example, the anomaly correction system300can, e.g., automatically determine that an item should be discarded if it has a certain number of anomalies. Alternatively, the anomaly correction system300can automatically determine that an item should be discarded if it has any anomalies, or that the item can be sent back to manufacturing to be refurbished or recycled. The anomaly correction system300can even, e.g., automatically determine that an entire production line should be stopped if an item with anomalies is found, or if a certain number of items with anomalies is found. The anomaly correction system300can also, e.g., notify an operator of the anomalies and providing the operator with the tagged image.

Referring now toFIG. 4, a system/method for reconstructing an image for detecting and correcting defects and anomalies is illustratively depicted in accordance with the present principles.

The reconstructor220can reconstruct the original image221of the item to represent a defectless version of the item by extracting image patches (interchangeable referred to herein as “image patch portions” or “patch portions” in short) with an image patch extractor222. The image patch extractor222can identify portions of the original image221to be reconstructed, such as, e.g., using a grid superimposed on the image. The image patch extractor222can then, using a processor, extract a series of image patches222afrom the original image221, where each image patch222ain the series is a different portion of the original image221according to the identified portions of the original image221. To facilitate reconstruction of each image patch222a, a region222cin each image patch222ais blacked-out or otherwise masked to generate a partially masked image patch222b. In this way, data regarding features of the item in the original image221is removed from the image patches222aso that reconstruction can be performed independent of any features present in corresponding portions of the original image221.

For each image in the series, a reconstruction module223can reconstruct the masked region222cof each partially masked image patch222b. The reconstruction module223can, therefore, include, e.g., a processing device including a processor, and a storage device such as, e.g., a hard drive, solid state drive, a flash memory or a temporary memory, such as, e.g., a RAM, a buffer, or a cache. To reconstruct the masked region222c, the reconstruction module223can utilize, e.g., an encoder-decoder arrangement stored in the storage device, or other suitable neural network for reconstructing images.

According to aspects of the present invention, the reconstruction module223is trained using defectless items of the type being analyzed. Thus, when the reconstruction module223reconstructs a masked region222cfor each partially masked image patch222bof the original image221, it does so based on defectless training by predicting the contents of the masked region222c. As a result, the reconstructed portion will appear to be defectless in a reconstructed image patch. By training the reconstruction module223with defectless items, training images can be easily found. Thus, training of the reconstruction module223is quick and efficient, with a large training set facilitating improved accuracy. Moreover, by reconstructing images to be defectless, types of defects and anomalies do not need to be taken into account, thus reducing the complexity of the reconstruction, improving speed and efficiency of an anomaly detection and tagging system200.

In embodiments of the present invention, the reconstruction module223employs an encoder-decoder arrangement including an encoder223aand decoder223b. Accordingly, the partially masked image patch222bwith a masked region222cis provided to the encoder223a, which transform the partially masked image patch222bwith a hidden layer to a latent representation in a feature space, such as, e.g., a multidimensional feature space vector. The hidden layer can include an activation function and a weight matrix. The encoder223acan include one or more hidden layers to arrive at an encoded representation, such as, e.g., the multidimensional feature space vector. In addition to masking features of the image patches222a, the encoder423acan be configured to reduce the dimensionality of the representation to further obfuscate any features in unmasked regions of the image patches223, and thus reduce the risk of anomalies being present in a reconstructed representation of the image patches223.

The encoded representation can then be decoded by the decoder223bto generate a predicted image patch224a. Similar to the encoder223a, the decoder223bcan use one or more hidden layers to transform the encoded representation to a representation corresponding to an output image by using an activation function and a weight matrix. The activation function and weight matrix of the decoder223bcan be the same or different from the activation function and weight matrix of the encoder223a.

Because the partially masked image patch222bincludes a masked region222c, the encoder223aencodes the partially masked image patch222bwithout any data related to any features of the item in the masked region222c. Thus, the features of the corresponding portion of the item are not encoded in the multidimensional features space vector. As a result, the decoder223bcan then reconstruct the image patch222aby predicting the masked region222cwithout any influence from features of a corresponding portion of the original image221. Thus, a defectless item can be predicted corresponding to the masked region222cin an efficient manner.

Because the encoder223aand decoder223bhave been trained with defectless items, the decoder223bis trained to predict defectless features. Thus, the predicted image portion includes a reconstructed image patch224ahaving no defects or anomalies, even if the corresponding image patch222aof the original image221did have defects or anomalies.

The reconstructed image patch224acan then be merged back into the original image221with the image merging module224. The image merging module224can include, e.g., a processing device including a process, and a storage device such as, e.g., a hard drive, solid state drive, a flash memory or a temporary memory, such as, e.g., a RAM, a buffer, or a cache. The reconstructed image patch224awill replace the corresponding portion of the original image221such that the original image221becomes a reconstructed image225. Alternatively, the reconstructed image patch224acan be stitched with other previously reconstructed image patches, independent of the original image. Each extracted image patch222awill be reconstructed by the reconstruction module223and merged back into the image with the image merging module224. Thus, the reconstructed image225will be produced with every identified portion replaced with a reconstructed version of that portion. As a result, the reconstructed image225will depict a defectless item. The reconstructed image225can then be stored, e.g., in a storage device such as, e.g., a hard drive, solid state drive, a flash memory or a temporary memory, such as, e.g., a RAM, a buffer, or a cache.

Referring now toFIG. 5, a system/method for contrasting a reconstructed image and original image for detecting and correcting defects and anomalies is illustratively depicted in accordance with the present principles.

A reconstructed image225can include an image having a number of portions of an original image221reconstructed by a reconstructor220. Thus, as discussed above, the reconstructed image225will depict an item having any defects or anomalies removed by the reconstruction process. The reconstructed image225can be contrasted with the original image221by a contrastor230using, e.g., a processing device including a processor and a storage device, such as, e.g., a hard drive, a solid state drive, a flash memory, a RAM, a buffer or a cache. The contrastor230can, e.g., determine a pixel-by-pixel difference between the images to produce an anomaly map231. Areas of high contrast between the two images will result in a larger difference of pixels at that location. That difference can be mapped to a new image depicting the pixel-by-pixel difference, thus highlighting the anomalies in an anomaly map231.

Referring now toFIG. 6, a system/method for training a reconstructor for detecting and correcting defective items is illustratively depicted in accordance with the present principles.

A reconstructor can be trained to reconstruct images by training a reconstruction learning module423with training images421. The training images421can each include an image of an item of a type to be analyzed by an anomaly detection and tagging system, such as the anomaly detection and tagging system200discussed above. Each training image421will be defectless, or in other words, “normal”. Thus, the reconstruction learning module423is training to reconstruct defectless item images. By training the reconstruction training module423with defectless items, training images can be easily found. Thus, training of the reconstruction training module423is quick and efficient, with a large training set facilitating improved accuracy. Moreover, by reconstructing images to be defectless, types of defects and anomalies do not need to be taken into account, thus reducing the complexity of the reconstruction, improving speed and efficiency of an anomaly detection and tagging system200.

To train the reconstruction learning module423, an image patch extractor422will extract patches422aof the training image421. The image patch extractor422, including a processing device having a processor, can identify portions of the training image421to be reconstructed, such as, e.g., using a grid superimposed on the image. The image patch extractor422can then extract a series of image patches422afrom the training image421, where each image patch422ain the series is a different portion of the training image421according to the identified portions of the training image421and temporarily or permanently store the image patches422ain a cache or a buffer.

A portion of each image patch422acan be blacked-out or otherwise masked to form a masked region422cof the image patch422ato generate a partially masked image patch422b. This masked region422ccontains no data regarding any features of a corresponding portion of the training image421. Thus, reconstruction of each partially masked image patch422bcan be performed independent of any features of the training image421.

For each image in the series, a reconstruction learning module423can reconstruct the masked region422cof each partially masked image patch422b. The reconstruction learning module423can, therefore, include, e.g., a processing device including a processor, and a storage device such as, e.g., a hard drive, solid state drive, a flash memory or a temporary memory, such as, e.g., a RAM, a buffer, or a cache. To reconstruct the masked region422c, the reconstruction learning module423can utilize, e.g., an encoder-decoder arrangement stored in the storage device, or other suitable neural network for reconstructing images.

Accordingly, the partially masked image patch422bis provided to the encoder423a, which transforms the masked region422cwith a hidden layer to a latent representation in a feature space, such as, e.g., a multidimensional feature space vector. The hidden layer can include an activation function and a weight matrix. The encoder423acan include one or more hidden layers to arrive at an encoded representation, such as, e.g., the multidimensional feature space vector. Because the reconstruction learning module423is trained with partially masked images422b, it is unnecessary to reduce the dimensionality of the multidimensional features space vector to below that of an identity function. However, reducing the encoder423acan be configured to reduce the dimensionality of the representation to further obfuscate any features in unmasked regions of the image patches423, and thus reduce the risk of anomalies being present in a reconstructed representation of the image patches223.

The encoded representation can then be decoded by the decoder423bto generate a predicted image patch424a. Similar to the encoder423a, the decoder423bcan use one or more hidden layers to transform the encoded representation to a representation corresponding to an output image by using an activation function and a weight matrix. The activation function and weight matrix of the decoder423bcan be the same or different from the activation function and weight matrix of the encoder423a.

Because the reconstruction learning module423is trained with partially masked images423, the reconstruction learning module423learns reconstruction for image patches by predicting data independent from any preexisting data in the masked portion422c. Complexity is, therefore, reduced for encoding and decoding, improving the speed and efficiency of the image reconstruction.

The predicted image can then be compared with the input image. An error can be determined according to difference between the input image and predicted image using a loss function. The error can be backpropagate to each hidden layer of the encoder423aand decoder423bto update the weight matrices at each layer using a suitable backpropagation process, such as, e.g., a gradient descent method, or a conjugate gradient method, among other backpropagation methods. This process is repeated with multiple training images. The training images will correspond to portions of images of defectless items of the type to be reconstructed. For example, the reconstruction learning module423can be trained with manufactured products to reconstruct images of manufactured products. Thus, the encoder-decoder arrangement of the reconstruction learning module423will be trained to reconstruct images of defectless items.

While, the training can be performed as an independent process to train the reconstruction learning module423prior to implementing the reconstruction learning module423as a reconstructing module, such as the reconstruction module223discussed above, the reconstruction module can be trained concurrently as a reconstruction learning module423with implementing the reconstruction module to reconstruct product images.

Therefore, while reconstructing image portions of a product, a reconstruction module, such as the reconstruction learning module423can reconstruct the image patches422a. The reconstructed image patches424acan then be merged back into the training image421with the image merging module424. The reconstructed image patch424awill replace the corresponding image patch424aof the training image421such that the training image421becomes a predicted reconstruction425. Each extracted image patch422awill be reconstructed by the reconstruction learning module423and merged back into the image with the image merging module424. Thus, the predicted reconstruction425can be produced with every identified portion replaced with a reconstructed version of that portion.

The reconstruction version can be contrasted with the original image using a process and system, such as the contrastor230and anomaly tagging module240described above. If the original image is found to be defect-free, the reconstructed portions can be used to determine an error with, e.g., a loss function, and backpropagate that error to the hidden layers of the reconstruction learning module423, as discussed above. Thus, the reconstruction learning module423can continuously be trained with defect-free product images while concurrently determining if a product has defects.

Referring now toFIG. 7, a high-level system/method for diagnosing medical anomalies, in accordance with the present principles is illustratively depicted in accordance with the present principles.

In one embodiment, a medical scanning device710generates scans of a person's anatomy, such as, e.g., an X-ray sensor, a magnetic resonance imaging (MRI) device, a computed tomography (CT) scan, a positron emission tomography (PET) scan, optical image, or other scanning device suitable for medical diagnosis. The medical scanning device710can, therefore, generate an image of anatomy or physiology. A person's anatomy or physiology can sometimes contain an anomaly that may indicate a disease or condition needing treatment. Thus, the image can be provided to a medical anomaly detection system720.

The medical anomaly detection system720will inspect the anatomy scans to determine if there are any anomalies. Such anomalies can include, e.g., physical signs and symptoms in anatomy and physiology that indicate a medical abnormality including, e.g., a tumor, a blood clot, a broken bone, a dislocation, a fracture, among others. The medical anomaly detection system720can determine that an abnormality exists by comparing the scans to a scan that of normal anatomy and physiology. For example, e.g., the medical anomaly detection system720can include a machine learning system that is trained with images of medically normal patient anatomy and physiology. By training the system with normal patient scans, the medical anomaly detection system720can generate a reconstruction of the anatomy scans that does not contain any abnormalities, and compare the reconstructed version with the original scans to identify differences. The locations of the differences will indicate an anomaly in anatomy or physiology. These differences can, therefore, be identified with the medical anomaly detection system720.

The identified abnormalities can be communicated to a diagnosis system730, such as, e.g., a display, computer, or other notification device. Thus, the diagnosis system730can notify a doctor of the abnormalities. Therefore, the doctor can easily and quickly find anomalies in a patient's anatomy that may have otherwise gone unnoticed or undetected. Alternatively, the diagnosis system730can include a device for automatically administering a medication. For example, where the medical scanning device710is an X-ray device, an anomaly may correspond to a broken bone. Thus, the diagnosis system730can, e.g., automatically administer a pain killer in response to tagging a bone related anomaly.

Referring now toFIG. 8, a system/method for detecting defects and anomalies with a reconstructor and contrastor is illustratively depicted in accordance with the present principles.

At block801, extract an image patch from a location on an original image of an item and partially mask the image patch.

The original image can be captured with an imaging device, such as, e.g., a camera, CCD, infrared sensor, LIDAR sensor, or other device for capturing images. The original image will be an image of the item, such as, e.g., a manufactured product, or a part of anatomy, among other items. Occasionally, the item will include defects and anomalies that undesired, such as, e.g., cracks, burrs, protrusions, dents, etc. The image device will capture these anomalies and defects.

Portions of the original image can then be identified to be reconstructed, such as, e.g., using a grid superimposed on the image. The portions are then extracted as a series of image patches from the original image, where each image patch in the series is a different portion of the original image according to the identified portions of the original image. To hide an area of the portions so that the area can be reconstructed, the area can be masked by, e.g., blacking-out, hiding, or otherwise removing the area from subsequent processing.

At block802, encode the partially masked image patch by transforming the partially masked image patch to a feature space vector using one or more hidden layers of an encoder.

In embodiments of the present invention, the image patch is provided to an encoder, which transform the image patch with a hidden layer to a feature space vector. The hidden layer can include an activation function and a weight matrix. The encoder can include one or more hidden layers to arrive at the feature space vector.

At block803, reconstruct the partially masked image patch of the item by decoding the feature space vector into a reconstructed patch using one or more hidden layers of a decoder.

The encoded representation can then be decoded by a decoder to generate a predicted image patch that returns the encoded representation back to a representation having the original number of dimensions of the input image patch. Similar to the encoder, the decoder can use one or more hidden layers to transform the encoded representation to a representation corresponding to an output image patch by using an activation function and a weight matrix. The activation function and weight matrix of the decoder can be the same or different from the activation function and weight matrix of the encoder.

Because the encoder encodes a partially masked image portion, the decoder predicts a masked region of the partially masked image portion. Thus, contents, including, e.g., physical or visible features of image patch used as the input image patch, are predicted during decoding by the decoder without influence from any features of the original image of the item. As a result, the output representation including the predicted image patch reconstructs the image patch according to the weight matrix and activation function.

At block804, merge the reconstructed patch into the location on the original image to generate a reconstructed image.

The reconstructed image patch can then be merged back into the original image. The reconstructed image patch will replace the corresponding portion of the original image such that the original image becomes a reconstructed image. Alternatively, the reconstructed image patch can be stitched with other previously reconstructed image patches, independent of the original image.

Each extracted image portion will be reconstructed and merged back into the image. Thus, the reconstructed image will be produced with every extracted patch replaced with a corresponding reconstructed patch. As a result, the reconstructed image will depict a defectless item. The reconstructed image can then be stored, e.g., in a storage device such as, e.g., a hard drive, solid state drive, a flash memory or a temporary memory, such as, e.g., a RAM, a buffer, or a cache.

At block805, contrast the reconstructed image with the original to generate an anomaly map that indicates anomalies at locations of differences between the reconstructed image and the original image.

To determine anomalies and defects in the item, the reconstructed image can then be compared to the original image with a contrastor. The contrastor can compare the reconstructed image with the original image by, e.g., performing a pixel-by-pixel difference between the images. However, other contrasting methods are contemplated. Because of the contrasting, the differences between the reconstructed image and original image can be mapped in an anomaly map that represents a degree of difference between the reconstructed image and the original image at each location. The anomaly map can take the form of, e.g., a visual representation, such as an image, or a matrix representation, or by any other suitable representation. As a result, the anomaly map can take the form of separate representation of the item, or it can be overlaid onto the original image to provide both a representation of the actual item, as well as a representation of the anomalies.

At block806, tag anomalies on the anomaly map to indicate possible anomalies in the item corresponding to the differences between the reconstructed image and the original image.

The anomaly map can then be used to identify anomalies by tagging areas of greatest difference between the reconstructed image and the original image corresponding to the results in the anomaly map. Tagged areas can be determined according to, e.g., an anomaly threshold value that represents degree of difference between the reconstructed image and the original image. If an area has a difference as represented in the anomaly of greater than the anomaly threshold, then that area can be tagged as containing an anomaly on a corresponding location of the item. Thus, item defects and anomalies can be identified. The anomaly tags can be applied to, e.g., the anomaly map, the original image, or both, or as a separate representation, such as, e.g., a list with coordinates.

At block807, automatically correct the anomalies.

In response to the anomaly map and the anomaly tags, corrective action can be taken. For example, a product having anomalies indicating defects can be automatically removed from a production line, or the entire production line can be automatically stopped. As another possible corrective action, a notification can be provided to an operator via a notification system, such as, e.g., a display or audible alert, such that the operator can take an appropriate action.

A description will now be given regarding embodiments of the present invention directed to anomaly detection with predictive normalization.

In the field of anomaly detection on images, there is always a scarcity of positive (anomalous) examples. However, negative (normal) examples are generally widely available. This invention makes use of the large amount of negative examples to train a model to enhance defects with respect to its surroundings so that a subsequent traditional supervised model can be trained with higher accuracy. Assuming that the rough size of the defects are known, we train a contextual auto-encoder (CAE) model with the context size being larger than the prediction size which corresponds to the approximate size of a defect. The CAE is trained with negative examples only and then used to normalize images that are used for training the supervised classifier. This normalization step is key to increase accuracy of the model and we observe an increase of 15 percentage points on the area under Precision-Recall curve (AUPRC).

FIG. 9is a block/flow diagram illustrating a high-level system/method for anomaly detection with predictive normalization, in accordance with an embodiment of the present invention. During an offline phase, training image(s) are first sampled by cutting out image samples of the size defined by the CAE model. Positive labels912indicate the position and shape of ground-truth defects on the image. Negative samples921are extracted at valid positions on the image, avoiding areas of positive samples. The CAE model932is then trained using the negative samples. For the supervised training phase, both negative and positive samples are extracted from the training image(s) and CIP normalization is applied941. The normalized samples are then used to train the supervised model951.

For the online phase, samples are extracted from test image(s)902and CIP normalization is applied. The normalized samples are then sent to the defect detection module960, where the supervised model951is applied to classify samples as normal or defect and where corrective action may be taken.

The image sampling module911extracts image samples of a certain size from the image. The size of the image samples is dictated by the size of the input of the model. The location of the samples is decided by a sampling strategy. One simple strategy, for example is pure random. Other strategies may involve preferring samples that exhibit large variance in luminance (the goal being to avoid flat featureless areas). Invalid parts of the image may also be ignored by the sampling process. Such parts may lie outside the marking of the contour of an object, for example. For the CAE training phase, areas of labeled defects should also be avoided. For the supervised training phase, the sampling module911extracts positive and negative samples from defect areas and normal areas respectively. Given that the number of positive labels is usually small, it is often desirable to generate as many positive examples as possible from a given label. A strategy is to sample in a grid fashion around the label such that the labeled defect is contained fully or partially within the image sample. Further augmentations may be applied, such as horizontal or vertical axis flips, 90-degree rotations and combinations of both. Such operations preserve the pixel intensities within the image sample. Non-preserving operations may also be applied such as free rotations or warping.

The unsupervised training module931which produces the CAE classifier model932is a contextual auto-encoder module. It includes two parts. First an encoder takes as input an image sample and compresses it via a convolutional neural network into a lower-dimensional vector of number (a code). Second, a decoder takes as input the code vector produced by the encoder and expands it via a convolutional neural network into an image. The parameters of the network (a.k.a. weights) are then gradually adjusted using the backpropagation procedure so as to minimize a loss function. The loss function can be, for example, the mean of squared differences between the output of the decoder and the center patch of the input image.

FIG. 10is a diagram further showing the CIP normalization module941ofFIG. 9, in accordance with an embodiment of the present invention.

An input image sample1001includes a center patch1005and its surrounding context1002. The size of the patch is decided based on the knowledge of the rough size of defects. For example, a size of 32×32 pixels may be used for the center patch and a size of 64×64 pixels may be used for the context. The center patch is blacked-out1005A (its values set to 0) and the image sample is presented to the CAE model932for inference, which produces the center patch1004. The predicted center patch1004is then sent together with the center patch1005to the normalization module1006, which produces a normalized center patch1007. The normalization module1006can include any operation combining both input images into a normalized output. For example, the predicted center patch1004may be subtracted, component-wise, from the input center patch1005. Further operations may be applied by normalization module1006, such as remapping the pixel intensities so that their new range makes the subsequent supervised learning procedure more efficient. A pixel intensity range of [−1;+1] is usually optimal. The mapping parameters may be obtained from the center patch itself or from the entire training set of center patches.

Referring back toFIG. 9, the supervised training of final model module950performs the supervised training of the final classifier model951. For this model a standard deep convolutional network (CNN), or any other appropriate classifier such as a support vector machine (SVM) can be utilized. The model is trained with positive and negative examples that were both normalized by the CIP normalization module941described above. Once trained, the model is used by the defect detection module960to predict whether a given normalized example is normal or not. Further actions may be taken by the defect detection module960if necessary. For example, if the number of defective examples in an image is above a certain threshold, the image maybe flagged for review, a corresponding product may be discarded or repaired, and so forth.

FIG. 11is a flow diagram illustrating a method1100for training an anomaly detector with predictive normalization, in accordance with an embodiment of the present invention.

At block1110, sample training images of a product to (i) extract image portions made of a center patch and its context and (ii) black-out the center patch. In an embodiment, the training images are sampled at random positions. However, other sampling approaches can be used, while maintaining the spirit of the present invention.

In an embodiment, block1110can involve (i) avoiding invalid areas, (ii) avoiding areas that include positive labels indicative of an anomaly, and (iii) favoring the portions for extraction that exhibit a pixel intensity variance higher than other portions while maintaining a threshold amount of randomness.

At block1120, perform unsupervised training of a Contextual Auto-Encoder (CAE) model using the sampled image portions to obtain a trained CAE model. In an embodiment, the unsupervised training can be performed using a mean of squared pixel intensity differences as a loss function.

At block1130, sample positive and negative image portions from the training images. The size of these portions corresponds to the size of the center patch in the previous block.

In an embodiment, block1130can involve (i) avoiding invalid areas, (ii) augmenting the positive and negative patch portions by combinations of a horizontal axis flip, a vertical axis flip, and 90 degree rotations, and (iii) augmenting the positive and negative samples by combinations of free rotations and warping.

At block1140, normalize, using the trained CAE model, the positive and negative patch portions.

In an embodiment, the normalization step in block1140can involve subtracting a patch predicted by the CAE model from the corresponding input patch, and further involve applying pixel intensity remapping. In an embodiment, the pixel intensity remapping can be determined based on intensity statistics of the patch itself and/or based on intensity statistics of several patches from the training set. The statistics can include, but are not limited to, for example, mean, standard deviation, minimum, maximum, percentiles, and so forth.

At block1150, perform supervised training of a classifier model using the normalized positive and negative patch portions to obtain a trained supervised classifier model for detecting the defective products.

FIG. 12is a flow diagram illustrating a method1200for using a trained anomaly detector with predictive normalization, in accordance with an embodiment of the present invention.

At block1210, sample an input test image to extract patch portions therefrom.

At block1220, normalize the patch portions extracted from the input test image using the trained CAE model to obtain normalized patch portions.

At block1230, classify the normalized patch portions using the trained supervised classifier to obtain classified samples.

In an embodiment, the classifying block (1230) is performed using a Deep Convolutional Neural Network (DCNN) to output a classification score, and wherein a particular one of the patch portions is deemed defective responsive to the particular one of the patch portions having the classification score below a threshold.

In an embodiment, the classifying block (1230) aggregates classifications of all of the patch portions sampled from the input test image, classifies the input test image as including a defect responsive to a number of defective ones of the patch portions sampled from the input test image exceeding a threshold, and displays locations of the patch portions including the detect by overlaying identifying information on the input test image.

At block1240, detect a defect in the product based on the classified samples.

At block1250, perform an action responsive to a detection of a defect. For example, the product may be discarded, a corrective action performed (e.g., replace the defective portion), a location of a defect(s) indicated on an overlay image over an image of a defective product, and so forth.