Patent ID: 12205360

To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements disclosed in one embodiment may be beneficially utilized on other embodiments without specific recitation.

DETAILED DESCRIPTION

Training a machine learning model for object detection and classification that generalizes well may benefit from a diverse training data set. This training data set usually involves human annotation, where a person goes sequentially through a list of images, labelling a sufficiently high number of examples in each image until an imposed threshold is reached. This conventional process is time consuming and unnecessary, especially when many of similar examples may be repeated. For large data sets, labeling every example may be infeasible due to the intrinsic constraints of cost and time. For optimal processing, the underlying task is to prioritize images for human labeling based on contribution to the training of the deep learning algorithm.

Naively, most operators may sort and label images alphanumerically by the order that they may be saved or numbers in the hose directory. This may vary by collection and file system and may be rarely correlated to the acquired data. After a brief global examination and determination of underlying classes, the operator may proceed to label every image, sequentially, through the directory, in the allotted time. Much of the data may be excess repeats, lack the sensitivity needed, contain background irregularities, and other iterations that may make the labeled data ill-suited for optimal training. The feedback between the final trained model and the labelling may be typically informal. Commonly, the separation between labeler and data processing is so large, that the labeler receives no feedback as to the usefulness of the labeled data, or knowledge as to how to formally optimize the labeling process. Gross errors are usually reported, but those errors are easy to sort from the data and do not contribute negatively to the overall training model. Though an operator may be instructed to ignore or pass through certain examples, compliance may become an unsolved issue and difficult to track or effectively enforce, especially over very large groups and many labelers.

Operators may also form a habitual pattern and labeling and become very skilled at detecting certain classes, i.e., those with a high number of examples. This may create the illusion of efficiency, but instead propagates bias away from rare classes and those objects that are difficult to identify and categorize. Operators may be more likely to ignore rare examples as background than label them as defects, when subjected to repetitive exposure to a common class for many images. For certain architectures, such as deep learning architectures, this presents an extremely large problem as areas that are not classified into a labeled class are treated as background and cause the model to not detect those types of defects. Combating this problem through several labelers or a voting process can be implemented, but is costly and is not guaranteed to yield a huge improvement in results.

One or more techniques provided herein present a multi-staged approach that may be used to improve upon conventional processes by continuously updating an order of the images in the dataset to be manually labeled in order to maximize the uniqueness and diversity of the labels. In some embodiments, the multi-staged approach may employ a deep learning segmentation model configured to predict the areas in an image of previously unseen data. In some embodiments, the multi-staged approach may focus on data sets where the objects may be distributed randomly, may vary in morphology within class, and may appear in small or large numbers per class. For these data sets, the present approach can achieve high accuracy model labelling.

Further, one or more techniques provided herein provide an improvement over conventional automatic labeling techniques. For example, using conventional approaches, a conventional system would run a model trained on previous data to generate labels and bounding boxes on the desired data set. The labeler would simply modify the bounding boxes. In the semiconductor domain and similar proprietary material domains, conventional automatic labeling approaches fail for lack of exposure to historical data and access to material specific classes that are the subject of the inspection and classification task. To address this, one or more techniques provided herein prompt a user or operator to label a small subset of images and define the class focus for the overall classification task. The goal is not to enforce an industry wide, or multi-user class restriction; but rather, to focus manual labeling on the key influencers. The model implemented by the present system may provide a hierarchical ordered list of the operator to label. In some embodiments, the model may be trained in an active learning fashion. For example, the model may be trained on a small initial data set and then may use the model to assist labeling the next batch of data, and then may retrain the model including the new data.

Still further, unlike conventional approaches, one or more technique described herein may utilize a curated training data set that includes more rare class examples. This curated training data set may help the model achieve better class balance in the training data set.

FIG.1illustrates an exemplary computing environment100for inspection of a specimen supported on a stage, according to exemplary embodiments. As shown, computing environment100may include an apparatus102in communication with a computing system150, according to example embodiments. Apparatus102may be configured to illuminate a specimen101with one or more light sources104,106. One or more light sources104,106may be configured to direct oblique light108,110toward specimen101at an angle. The oblique illumination may be reflected from a surface of specimen101as reflected light112. Apparatus102may include a camera device114having an image sensor that is configured to capture the reflected light. In some embodiments, light sources104,106may be moved to different positions located circumferentially around the object, with images taken at each position.

In some embodiments, apparatus102may provide the images captured by camera device114to computing system150for processing. Computing system150may be in communication with apparatus102via one or more communication channels. In some embodiments, the one or more communication channels may be representative of individual connections via the Internet, such as cellular or Wi-Fi networks. In some embodiments, the one or more communication channels may connect terminals, services, and mobile devices using direct connections, such as radio frequency identification (RFID), near-field communication (NFC), Bluetooth™, low-energy Bluetooth™ (BLE), Wi-Fi™, ZigBee™, ambient backscatter communication (ABC) protocols, USB, WAN, or LAN. Computing system150may be configured to analyze the images captured by camera device114and generate a topography of specimen101.

As shown, computing system150may include a pre-processing engine152and a prediction model154. Each of pre-processing engine152and prediction model154may include one or more software modules. The one or more software modules may be collections of code or instructions stored on a media (e.g., memory of computing system150) that represent a series of machine instructions (e.g., program code) that implements one or more algorithmic steps. Such machine instructions may be the actual computer code the processor interprets to implement the instructions or, alternatively, may be a higher level of coding of the instructions that is interpreted to obtain the actual computer code. The one or more software modules may also include one or more hardware components. One or more aspects of an example algorithm may be performed by the hardware components (e.g., circuitry) itself, rather as a result of the instructions.

Pre-processing engine152may be configured to generate one or more training data sets to train prediction model154. As provided above, it is often a tedious task for users to manually label and develop training sets for training prediction model154. Pre-processing engine152may include deep learning model156. Deep learning model156may be trained to sort images for labeling. In some embodiments, deep learning model156may be trained on an initial small subset of images. All objects of interest in the initial small subset of images may be labeled with rectangular bounding boxes. In some embodiments, the pixels inside of the bounding boxes of all labeled objects may be grouped into a first category (e.g., “foreground”); the rest of the image pixels may be grouped into a second category (e.g., “background”). Using this classification, two input segmentation masks may be generated: a first input segmentation mask for the background and a second input segmentation mask for the foreground. In some embodiments, bounding boxes around the foreground objects may be enlarged to eliminate or reduce ambiguity of classifying pixels on the border of the bounding boxes.

In some embodiments, deep learning model156may be trained to produce two probability maps: a first probability map providing the probability of each pixel belonging to the foreground; and a second probability map providing the probability of each pixel belonging to the background. In some embodiments, the size of each probability map may be the same size as the input image. Using the first probability map and the second probability map, pre-processing engine152may compute a measure that a pixel has not been seen, e.g., S unseen Pixels that may belong to examples of a new class not yet labeled or to novel-looking examples of a previously labeled class will have a high unseen score. For example:
Sunseen(x,y)=1−P((x,y)∈bg)−P((x,y)∈fg)

Using the per-pixel unseen scores, pre-processing engine may compute an overall image metric and used for raking images. In some embodiments, a high image metric may correspond to a high confidence that the image contains a pixel that has not been seen and therefore deserves labeling priority. In some embodiments, it may be unclear which image should have higher priority: an image that has few high scoring pixels or an image that has plenty of low scoring pixels. To account for this, in some embodiments, pre-processing engine118may compute two metrics: a threshold metric Mthreshand an alpha metric MalphaIn some embodiments, threshold metric may be equal to a number of pixels that have an unseen score above some threshold. Under the threshold metric, low scoring pixels may have no influence on the metric. For example:

Mthresh=∑(x,y)Su⁢n⁢s⁢e⁢e⁢n(x,y)>t

In some embodiments, alpha metric may be equal to the sum of all unseen scores at power, α. Under alpha metric, all pixels may be accounted for but lower scoring pixels have a lesser influence on the score.

Malpha=∑(x,y)Su⁢n⁢s⁢e⁢e⁢n(x,y)α

After the images are ranked using one of these metrics, the next batch of images to be labeled may be produced. This process may iterate in an active learning fashion: the new images are labeled and deep learning model156may be re-trained using the newly available labeled images, and deep learning model156may be invoked to produce the next batch of images to be labeled.

After a sufficient amount of data is labeled, prediction model154may be trained to detect defects in specimens. In some embodiments, prediction model154may also be used to automatically label the rest of the dataset.

FIG.2illustrates an architecture200of deep learning model156, according to example embodiments. As shown, architecture200may be based off the U-net architecture. Architecture200is a modified U-net architecture that removes the final softmax layer that produces the segmentation masks in order to directly obtain the pixel probabilities per class. Architecture200also includes a different set of convolutional layer padding to match the input size and the number of feature maps used by the convolutional layers.

FIG.3is a flow diagram illustrating a method300of generating a training data set to train prediction model154, according to example embodiments. Method300may begin at step302.

At step302, computing system150may receive a set of images for training prediction model154. In some embodiments, computing system150may receive a set of images from a client device in communication with computing system150. In some embodiments, computing system150may receive a set of images from a database associated with computing system150. In some embodiments, computing system150may receive a set of images from a third party website or system. In some embodiments, the set of images may include a subset of labeled images. The subset of labeled images may be labeled by a person or operator. The subset of labeled images may include labels for all defects present in the image. In some embodiments, random selection was not chosen to prevent any selection bias by the operator and to force alphanumeric bias, as would be performed by an operator under the conventional paradigm.

At step304, computing system150may generate a second subset of images for subsequent labeling. For example, computing system150may provide the set of images to deep learning model156for processing. The set of images may include the subset of labeled images, along with the rest of the unlabeled images. Deep learning model156may generate two probability maps for each image: a first probability map providing the probability of each pixel belonging to the foreground; and a second probability map providing the probability of each pixel belonging to the background. Using the first probability map and the second probability map, pre-processing engine152may compute the Sunseenmetric. Based on the Sunseenmetric, pre-processing engine152may generate the Malphametric for each image. Once pre-processing engine152generate each Malphametric, pre-processing engine152may rank the images and select a set of highest scoring images to form the second subset of images.

At step306, computing system150may prompt a user to label the second subset of images. For example, computing system150may prompt a user to label all defects contained in the second subset of images. Because deep learning model156was able to identify a second subset of images that include pixels unseen compared to the subset of labeled images from the original set of images, deep learning model156has identified additional images that may be useful for subsequent training of prediction model154.

At step308, computing system150may generate a third subset of images for subsequent labeling. For example, computing system150may provide the set of images to deep learning model156for continual training. The set of images may include the subset of labeled images, the second subset of images (that are now also labeled), and the remaining unlabeled images. Deep learning model156may generate two probability maps for each image: a first probability map providing the probability of each pixel belonging to the foreground; and a second probability map providing the probability of each pixel belonging to the background. Using the first probability map and the second probability map, pre-processing engine152may compute the Sunseenmetric. Based on the Sunseenmetric, pre-processing engine152may generate the Malphametric for each image. Once pre-processing engine152generate each Malphametric, pre-processing engine152may rank the images and select a set of highest scoring images to form the third subset of images.

At step310, computing system150may prompt a user to label the third subset of images. For example, computing system150may prompt a user to label all defects contained in the third subset of images. Because deep learning model156was able to identify a third subset of images that include pixels unseen compared to the subset of labeled images from the original set of images and the second subset of images, deep learning model156has identified additional images that may be useful for subsequent training of prediction model154.

At step312, computing system150may determine if there are a threshold number of labeled images for training prediction model154as specified by MthreshFor example, based on steps304-310, computing system150may determine whether there are sufficient labeled images for training prediction model154. If, at step312, computing system150determines that there is not a threshold number of labeled images, then method300may revert to step308for continuing generation of image labels. In other words, if there are not a threshold amount of labeled images, computing system150may continue the process of providing a subset of labeled images, the second subset of labeled images, the third subset of labeled images, the nth subset of labeled images, and the remaining unlabeled images to deep learning model156for continued ranking of new unlabeled images for labeling.

If, however, at step312, computing system150determines that there is a threshold amount of labeled images, then method300may proceed to step314. At step314, computing system150may output a set of labeled images for training prediction model154.

FIG.4is a flow diagram illustrating a method400of identifying defects in a specimen, according to example embodiments. Method400may begin at step402.

At step402, computing system150may identify a training set for training prediction model154to identify defects present in a specimen. The training set may be representative of the set of labeled images generated by deep learning model156. For example, the training set may include a plurality of images of various specimens, each image including one or more artifacts labels.

At step404, computing system150may train prediction model154. For example, computing system150may train prediction model154to identify defects in an image of a specimen based on the training set. In some embodiments, prediction model154may be representative of a faster region based convolutional neural network (R-CNN). Prediction model154may be trained on sequentially labeled images up to the time it took to label the algorithmically selected images (i.e., the same number of bounding boxes labeled).

At step406, computing system150may receive, from apparatus102, an image of a specimen under examination. In some embodiments, the specimen under examination may be a semiconductor substrate that may or may not contain one or more defects. Although the discussion references semiconductor substrates as one particular example, those skilled in the art recognize that the techniques disclosed herein are not limited to semiconductor substrates. For example, the techniques disclosed herein may be extended to detection features or defects/anomalies in biological tissue.

At step408, computing system150may identify one or more defects present in the image of the specimen. For example, computing system150may provide the image, as input, to fully trained prediction model154. Prediction model154may analyze the image to identify one or more defects present in the image of the specimen. In some embodiments, prediction model154may identify one or more defects present in the image by generating bounding boxes around each defect. In some embodiments, prediction model154may identify one or more defects present in the image by generating a probability map associated with the device.

At step410, computing system150may generate a graphical representation of the one or more defects present in the image of the specimen. In some embodiments, prediction model154may generate a graphical representation that overlays one or more bounding boxes over each of the identified one or more defects in the image. In some embodiments, prediction model154may generate a graphical representation that overlays a heatmap over the image. The heat map may include verifying intensity based on where the defects are present. For example, areas of the image where defects are present will have a higher intensity that areas of the image where defects are not present.

FIG.5illustrates an exemplary graphical output500generated by prediction model154, according to example embodiments. As shown, graphical output500may correspond to a probability map502generated by prediction model154. Probability map502may be overlaid on the image of the specimen. The higher intensity areas of probability map502may indicate a high probability that defects are present in that location.

FIG.6Aillustrates a system bus computing system architecture600, according to example embodiments. One or more components of system600may be in electrical communication with each other using a bus605. System600may include a processor (e.g., one or more CPUs, GPUs or other types of processors)610and a system bus605that couples various system components including the system memory615, such as read only memory (ROM)620and random access memory (RAM)625, to processor610. System600can include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of processor610. System600can copy data from memory615and/or storage device630to cache612for quick access by processor610. In this way, cache612may provide a performance boost that avoids processor610delays while waiting for data. These and other modules can control or be configured to control processor610to perform various actions. Other system memory615may be available for use as well. Memory615may include multiple different types of memory with different performance characteristics. Processor610may be representative of a single processor or multiple processors. Processor610can include one or more of a general purpose processor or a hardware module or software module, such as service1632, service2634, and service3636stored in storage device630, configured to control processor610, as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor610may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

To enable user interaction with the computing device600, an input device645which can be any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. An output device635can also be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems can enable a user to provide multiple types of input to communicate with computing device600. Communications interface640can generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

Storage device630may be a non-volatile memory and can be a hard disk or other types of computer readable media that can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, random access memories (RAMs)625, read only memory (ROM)620, and hybrids thereof.

Storage device630can include services632,634, and636for controlling the processor610. Other hardware or software modules are contemplated. Storage device630can be connected to system bus605. In one aspect, a hardware module that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor610, bus605, display635, and so forth, to carry out the function.

FIG.6Billustrates a computer system650having a chipset architecture, according to example embodiments. Computer system650may be an example of computer hardware, software, and firmware that can be used to implement the disclosed technology. System650can include one or more processors655, representative of any number of physically and/or logically distinct resources capable of executing software, firmware, and hardware configured to perform identified computations. One or more processors655can communicate with a chipset660that can control input to and output from one or more processors655. In this example, chipset660outputs information to output665, such as a display, and can read and write information to storage device670, which can include magnetic media, and solid state media, for example. Chipset660can also read data from and write data to RAM675. A bridge680for interfacing with a variety of user interface components685can be provided for interfacing with chipset660. Such user interface components685can include a keyboard, a microphone, touch detection and processing circuitry, a pointing device, such as a mouse, and so on. In general, inputs to system650can come from any of a variety of sources, machine generated and/or human generated.

Chipset660can also interface with one or more communication interfaces690that can have different physical interfaces. Such communication interfaces can include interfaces for wired and wireless local area networks, for broadband wireless networks, as well as personal area networks. Some applications of the methods for generating, displaying, and using the GUI disclosed herein can include receiving ordered datasets over the physical interface or be generated by the machine itself by one or more processors655analyzing data stored in storage670or675. Further, the machine can receive inputs from a user through user interface components685and execute appropriate functions, such as browsing functions by interpreting these inputs using one or more processors655.

It can be appreciated that example systems600and650can have more than one processor610or be part of a group or cluster of computing devices networked together to provide greater processing capability.

While the foregoing is directed to embodiments described herein, other and further embodiments may be devised without departing from the basic scope thereof. For example, aspects of the present disclosure may be implemented in hardware or software or a combination of hardware and software. One embodiment described herein may be implemented as a program product for use with a computer system. The program(s) of the program product define functions of the embodiments (including the methods described herein) and can be contained on a variety of computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory (ROM) devices within a computer, such as CD-ROM disks readably by a CD-ROM drive, flash memory, ROM chips, or any type of solid-state non-volatile memory) on which information is permanently stored; and (ii) writable storage media (e.g., floppy disks within a diskette drive or hard-disk drive or any type of solid state random-access memory) on which alterable information is stored. Such computer-readable storage media, when carrying computer-readable instructions that direct the functions of the disclosed embodiments, are embodiments of the present disclosure.

It will be appreciated to those skilled in the art that the preceding examples are exemplary and not limiting. It is intended that all permutations, enhancements, equivalents, and improvements thereto are apparent to those skilled in the art upon a reading of the specification and a study of the drawings are included within the true spirit and scope of the present disclosure. It is therefore intended that the following appended claims include all such modifications, permutations, and equivalents as fall within the true spirit and scope of these teachings.