Precision defect detection based on image difference with respect to templates

A computer-implemented method is provided for image-based defect detection. The method includes performing, by a processor device, template matching and subtraction on a set of training images and at least one template image to obtain a set of difference images. The difference images have defects, if any, highlighted therein. The method further includes generating, by the hardware processor applying a binary classification model to each of the training images in the set, activation heatmaps. The method also includes identifying, by the hardware processor, rough defect areas of interest in the activation heatmaps. The method additionally includes super-imposing, by the hardware processor, the activation heatmaps onto the difference images to obtain a set of super-imposed images, and highlight, as true defect areas, any areas in the super-imposed images having the defects from the difference images that overlap with the rough defect areas of interest from the activation heatmaps.

BACKGROUND

Technical Field

The present invention generally relates to defect detection, and more particularly to precision defect detection based on image difference with respect to templates.

Description of the Related Art

Often, in defect detection in manufacturing scenarios, there is a relatively small defect area in a large picture. Accordingly, the defect needs to be detected and extracted first for accurate classification. Accurate position determining for a defect can also often determine the severity of the defect and has an impact on the subsequent processing flow (e.g., repair, rework, ignore, disposal, etc.).

In many defect detection applications in the manufacturing industry, standard reference templates are available for comparison to inspection images in order to detect defects. However, the performance of current defect location detection systems are sensitive to noise, change of illumination, small image deformations (e.g., due to different camera calibrations, etc.), and so forth. Moreover, while traditional template matching based approaches are able detect actual defect areas, they are prone to many false alarms. State-of-art object detection or image segmentation approaches based on deep learning require expensive labeling of defect locations and are thus difficult to efficiently apply in production. Hence, there is a need for an precise template-based defect detection approach.

SUMMARY

According to an aspect of the present invention, a computer-implemented method is provided for image-based defect detection. The method includes performing, by a processor device, template matching and subtraction on a set of training images and at least one template image to obtain a set of difference images. The difference images have defects, if any, highlighted therein. The method further includes generating, by the hardware processor applying a binary classification model to each of the training images in the set, activation heatmaps. The method also includes identifying, by the hardware processor, rough defect areas of interest in the activation heatmaps. The method additionally includes super-imposing, by the hardware processor, the activation heatmaps onto the difference images to obtain a set of super-imposed images, and highlight, as true defect areas, any areas in the super-imposed images having the defects from the difference images that overlap with the rough defect areas of interest from the activation heatmaps.

According to another aspect of the present invention, a computer program product is provided for image-based defect detection. The computer program product includes a non-transitory computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a computer to cause the computer to perform a method. The method includes performing, by a processor device, template matching and subtraction on a set of training images and at least one template image to obtain a set of difference images. The difference images have defects, if any, highlighted therein. The method further includes generating, by the hardware processor applying a binary classification model to each of the training images in the set, activation heatmaps. The method also includes identifying, by the hardware processor, rough defect areas of interest in the activation heatmaps. The method additionally includes super-imposing, by the hardware processor, the activation heatmaps onto the difference images to obtain a set of super-imposed images, and highlight, as true defect areas, any areas in the super-imposed images having the defects from the difference images that overlap with the rough defect areas of interest from the activation heatmaps.

According to yet another aspect of the present invention, a computer processing system is provided for image-based defect detection. The computer processing system includes a memory for storing program code. The computer processing system further includes a processor device for running the program code to perform template matching and subtraction on a set of training images and at least one template image to obtain a set of difference images. The difference images have defects, if any, highlighted therein. The processor device further runs the program code to generate, by applying a binary classification model to each of the training images in the set, activation heatmaps. The processor device also runs the program code to identify rough defect areas of interest in the activation heatmaps. The processor device additionally runs the program code to super-imposing the activation heatmaps onto the difference images to obtain a set of super-imposed images, and highlight, as true defect areas, any areas in the super-imposed images having the defects from the difference images that overlap with the rough defect areas of interest from the activation heatmaps.

DETAILED DESCRIPTION

The present invention is directed to defect detection based on image difference with respect to templates.

In an embodiment, an accurate defect detection system is provided which includes the following features:

(1) Channel concatenation of the aligned inspection image and template image as an input to a defect detection or a defect segmentation model.

(2) The use of template matching and subtraction to obtain a difference image with potential defect areas that can be used as training images.

(3) The use of merely class-labeled training images to train a classification model and obtain the activation heatmap for each training image classified as “defective”.

(4) The use of a heatmap to screen out false defect areas in the difference image and mark the overlapped defect area as a final defect area. In this way, an accurate defect area is obtained for a training image.

(5) The use of training images to train the aforementioned defect detection or defect segmentation model.

One of the many advantages of the present invention relates to the fully automatic generation of accurate defect locations for merely class-labeled training images. The preceding provides a significant savings with respect to location labeling that is needed in standard object detection/image segmentation tasks. In an embodiment, the present invention performs accurate location detection via the channel concatenation from aligned inspection/template images.

FIG. 1is a block diagram showing an exemplary processing system100to which the present invention may be applied, in accordance with an embodiment of the present invention. The processing system100includes a set of processing units (e.g., CPUs)101, a set of GPUs102, a set of memory devices103, a set of communication devices104, and set of peripherals105. The CPUs101can be single or multi-core CPUs. The GPUs102can be single or multi-core GPUs. The one or more memory devices103can include caches, RAMs, ROMs, and other memories (flash, optical, magnetic, etc.). The communication devices104can include wireless and/or wired communication devices (e.g., network (e.g., WIFI, etc.) adapters, etc.). The peripherals105can include a display device, a user input device, a printer, an imaging device, and so forth. Elements of processing system100are connected by one or more buses or networks (collectively denoted by the figure reference numeral110).

Moreover, it is to be appreciated that various figures as described below with respect to various elements and steps relating to the present invention that may be implemented, in whole or in part, by one or more of the elements of system100.

A description will now be given regarding two exemplary environments200and300to which the present invention can be applied, in accordance with various embodiments of the present invention. The environments200and300are described below with respect toFIGS. 2 and 3, respectively. In further detail, the environment200includes defect detection system operatively coupled to a controlled system, while the environment300includes a defect detection system as part of a controlled system. Moreover, any of environments200and300can be part of a cloud-based environment (e.g., seeFIGS. 6 and 7). These and other environments to which the present invention can be applied are readily determined by one of ordinary skill in the art, given the teachings of the present invention provided herein, while maintaining the spirit of the present invention.

FIG. 2is a block diagram showing an exemplary environment200to which the present invention can be applied, in accordance with an embodiment of the present invention.

The environment200includes an object/defect detection system210and a controlled system220. Hence, system210is selectively configurable to perform object detection and/or defect detection (where the defect is the object being detected). The object/defect detection system210and the controlled system220are configured to enable communications therebetween. For example, transceivers and/or other types of communication devices including wireless, wired, and combinations thereof can be used. In an embodiment, communication between the object/defect detection system210and the controlled system220can be performed over one or more networks, collectively denoted by the figure reference numeral230. The communication can include, but is not limited to, inspection images (and possible template images as well) from the controlled system220, and defect detection results and action initiation control signals from the object/defect detection system210. The controlled system220can be any type of processor-based system such as, for example, but not limited to, a surveillance system, a manufacturing system (e.g., an assembly line), an Advanced Driver-Assistance System (ADAS), and so forth.

The controlled system220provides images to the object/defect detection system210which can use the images to determinations regarding defects and perform certain actions in response thereto.

The controlled system220can be controlled based on a detection result from the object/defect detection system210. For example, based on result that an integrated circuit is defective, the integrated circuit may be discarded, while a lack of such result, the integrated circuit may be further processed. As another example, based on a scenario involving object detection, one or more actions directed to avoiding contact with a detected object can be performed such as braking, steering, and/or accelerating. As yet another example, based on a detecting an intruder, a surveillance system being controlled could lock or unlock one or more doors in order to secure someone in a certain place (holding area) and/or guide them to a safe place (safe room) and/or restrict them from a restricted place and/or so forth. It is to be appreciated that the preceding actions are merely illustrative and, thus, other actions can also be performed depending upon the implementation, as readily appreciated by one of ordinary skill in the art given the teachings of the present invention provided herein, while maintaining the spirit of the present invention.

In an embodiment, the object/defect detection system210can be implemented as a node in a cloud-computing arrangement. In an embodiment, a single object/defect detection system210can be assigned to a single controlled system or to multiple controlled systems e.g., different robots in an assembly line, and so forth). These and other configurations of the elements of environment200are readily determined by one of ordinary skill in the art given the teachings of the present invention provided herein, while maintaining the spirit of the present invention.

FIG. 3is a block diagram showing another exemplary environment300to which the present invention can be applied, in accordance with an embodiment of the present invention.

The environment300includes a controlled system320that, in turn, includes an object/defect detection system310. One or more communication buses and/or other devices can be used to facilitate inter-system, as well as intra-system, communication. The controlled system320can be any type of processor-based system such as, for example, but not limited to a surveillance system, a manufacturing system (e.g., an assembly line), an Advanced Driver-Assistance System (ADAS), and so forth.

Other than system310being included in system320, operations of these elements in environments200and300are similar. Accordingly, elements310and320are not described in further detail relative toFIG. 3for the sake of brevity, with the reader respectively directed to the descriptions of elements210and220relative to environment200ofFIG. 2given the common functions of these elements in the two environments200and300.

It has been noted that classification deep learning tasks usually have much lower labeling efforts in comparison to object detection and image segmentation. It has been further noted that deep learning result interpretation reveals activation heatmaps that are able to illuminate rough relevant areas in the input image that most intensively affect the corresponding classification results. Accordingly, the present invention exploits the opportunity to leverage such rough areas of interest for defect detection/segmentation applications.

In a convolutional neural network, the activation heatmap is generated over the output (called feature maps) of the last convolutional layer. The feature maps can be regarded as a transformed image representation of the original input image, with however smaller size. Thus, the ROI area marked via the heatmap is of low resolution. The ROI area can be resized to fit the original image size but the area remains a rough location of the defect. How “rough” the marking is depends on the resolution gap between the original input image and the feature maps of the last convolutional layer, which varies case by case in the application of the CNN. Hence, the term “rough” refers to a resolution gap between the input image and the feature map from which the heatmap is generated, as the heatmap has the same resolution as the feature map from which the heatmap is generated and which is less than the original input image.

Various methods are hereinafter described with respect toFIGS. 4 and 5. These methods can be performed by computer processing system100and/or by the object/defect detection system210ofFIG. 2and/or by the object/defect detection system310ofFIG. 3.

FIG. 4is a high-level block/flow diagram showing an exemplary inference model400, in accordance with an embodiment of the present invention.

At block410, receive an inspection image410A and a template image410B.

At block420, perform template matching on the inspection image410A and the template image410B to output aligned inspection420A and template images420B. That is, the inspect image420A and the template image420B that are output from the template matching are in alignment with respect to each other. In the prior art, these two aligned images would then be subjected to a subtraction operation that is prone to the problem of having false alarms. This problem is overcome by the following steps of the present invention.

At block430, perform channel concatenation with respect to the aligned inspection420A and template images420B to obtain a super-imposed image430A. The super-imposed image430A is a single image with six channels. The six channels consists of 3 channels for each the two RGB (Red, Green, Blue) images that form the super-imposed image430A.

At block440, apply a deep network for image segmentation and/or object detection to the super-imposed image430A to obtain a resultant image440A that includes an actual defect area or an actual object and that is obtained with false alarm suppression.

At block450, perform an action(s) responsive to the actual defect area or the actual object in the resultant image. For example, in the case of a manufactured item (e.g., integrated circuit, chip, etc.) and a scenario involving defect identification, the item can be discarded in order to avoid the manufactured item being further processed (e.g., further manufactured, packaged, shipped, etc.) for selling to a consumer. As another example, in the case of an object and a scenario involving object (e.g., a deer, another vehicle, a pedestrian, etc.) detection, one or more vehicle functions (e.g., braking, steering, accelerating) can be controlled to avoid contacting the detected object. It is to be appreciated that the preceding actions are merely illustrative and, thus, other actions can also be performed depending upon the implementation, while maintaining the spirit of the present invention.

The deep network is trained to segment out the real defect area and suppress false alarms by using training image that have classification labels (i.e., “good” versus “defect”), but without defect location labels.

FIG. 5is a block diagram showing an exemplary method500for labeling of defect locations in images for training a deep network for image segmentation or object detection, in accordance with an embodiment of the present invention.

At block510, train the deep network as a binary classification model using training images.

In an embodiment, block510can include one or more of blocks510A,510B, and510C.

At block510A, for each training image, perform template matching and subtraction to obtain a different image wherein potential defect areas are illuminated and/or otherwise indicated.

At block510B, feed each training image into the binary classification model. For training images classified as having a “defect”, generate their activation heatmaps and extract the rough defect Area Of Interest (AOI). In an embodiment, a heatmap is not generated for images not classified as having a “defect”.

At block510C, super-impose the rough defect AOI on the corresponding difference image. Mark the illuminated potential defect areas that overlap with the rough defective AOI as a true defect area. In this way, the defect areas are literally marked.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Referring now toFIG. 6, illustrative cloud computing environment650is depicted. As shown, cloud computing environment650includes one or more cloud computing nodes610with which local computing devices used by cloud consumers, such as, for example, personal digital assistant (PDA) or cellular telephone654A, desktop computer654B, laptop computer654C, and/or automobile computer system654N may communicate. Nodes610may communicate with one another. They may be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof. This allows cloud computing environment650to offer infrastructure, platforms and/or software as services for which a cloud consumer does not need to maintain resources on a local computing device. It is understood that the types of computing devices654A-N shown inFIG. 6are intended to be illustrative only and that computing nodes610and cloud computing environment650can communicate with any type of computerized device over any type of network and/or network addressable connection (e.g., using a web browser).

Hardware and software layer760includes hardware and software components. Examples of hardware components include: mainframes761; RISC (Reduced Instruction Set Computer) architecture based servers762; servers763; blade servers764; storage devices765; and networks and networking components766. In some embodiments, software components include network application server software767and database software768.

Virtualization layer770provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers771; virtual storage772; virtual networks773, including virtual private networks; virtual applications and operating systems774; and virtual clients775.

Workloads layer790provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation791; software development and lifecycle management792; virtual classroom education delivery793; data analytics processing794; transaction processing795; and precise defect detection based on image different with respect to templates796.