AUTONOMOUS ENRICHING REFERENCE INFORMATION

A method for autonomous enriching reference information, including (a) obtaining a current group of trusted reference images of non-anomalous instances of an item: (b) calculating reference pixel-wise distribution information: (c) obtaining multiple sets of item instance pixels from current acquired images of instances of the item, each set originated from an image of an instance of the item and comprises multiple item instance pixels: (d) determining item features of the item for each set, based on the multiple item pixels of the set and by a non-item specific neural network: (e) determining a pixel score for item pixels of the multiple item pixels: (f) calculating a distance between the pixel score and the reference pixel-wise distribution information; and (g) selecting at least one current acquired image to add to the trusted reference images, based on at least one distance of at least one pixel per current acquired image.

BACKGROUND

Defect detection is a process that involve acquiring images of evaluated objects and processing the images to detect defects. A common method for defect detection include comparing an image of an evaluated object to an image of a reference object.

It may be beneficial to compare the evaluated object to a reference object that is defect free—but generating an image of a defect free reference object may also be time and resource consuming. Comparing the inspected object to an arbitrary reference object may provide ambiguous results—as a different between the evaluated object and the reference object may result from defects of the evaluated object or the reference object.

There is a growing need to provide a cost-effective method for cluster-based and autonomous finding of reference information.

SUMMARY

There is provided a method, a system and/or a non-transitory computer readable medium for autonomous enriching reference information.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Any reference in the specification to a method should be applied mutatis mutandis to a device or system capable of executing the method and/or to a non-transitory computer readable medium that stores instructions for executing the method.

Any reference in the specification to a system or device should be applied mutatis mutandis to a method that may be executed by the system, and/or may be applied mutatis mutandis to non-transitory computer readable medium that stores instructions executable by the system.

Any reference in the specification to a non-transitory computer readable medium should be applied mutatis mutandis to a device or system capable of executing instructions stored in the non-transitory computer readable medium and/or may be applied mutatis mutandis to a method for executing the instructions.

Any combination of any module or unit listed in any of the figures, any part of the specification and/or any claims may be provided.

Any one of the units may be implemented in hardware and/or code, instructions and/or commands stored in a non-transitory computer readable medium, may be included in a vehicle, outside a vehicle, in a mobile device, in a server, and the like.

The vehicle may be any type of vehicle that a ground transportation vehicle, an airborne vehicle, and a water vessel.

The specification and/or drawings may refer to an image. An image is an example of a media unit. Any reference to an image may be applied mutatis mutandis to a media unit. A media unit may be an example of sensed information. Any reference to a media unit may be applied mutatis mutandis to any type of natural signal such as but not limited to signal generated by nature, signal representing human behavior, signal representing operations related to the stock market, a medical signal, financial series, geodetic signals, geophysical, chemical, molecular, textual and numerical signals, time series, and the like. Any reference to a media unit may be applied mutatis mutandis to sensed information. The sensed information may be of any kind and may be sensed by any type of sensors-such as a visual light camera, an audio sensor, a sensor that may sense infrared, radar imagery, ultrasound, electro-optics, radiography, LIDAR (light detection and ranging), unidimensional data (current, voltage, pressure) etc. The sensing may include generating samples (for example, pixel, audio signals) that represent the signal that was transmitted, or otherwise reach the sensor.

The specification and/or drawings may refer to a processor. The processor may be a processing circuitry. The processing circuitry may be implemented as a central processing unit (CPU), and/or one or more other integrated circuits such as application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), full-custom integrated circuits, etc., or a combination of such integrated circuits.

Any combination of any steps of any method illustrated in the specification and/or drawings may be provided.

Any combination of any subject matter of any of claims may be provided.

Any combinations of systems, units, components, processors, sensors, illustrated in the specification and/or drawings may be provided.

There may be provide a method, system and a non-transitory computer readable medium for autonomous enriching reference information.

The method, system and a non-transitory computer readable medium can process any amount of images of instances of an item-without needing to tag the images.

There may be any number of instances of the item. If the item is, for example, an object then the different instances may include different objects that should (if manufactured under ideal defect-free conditions) be the same.

FIG. 1 illustrates an example of method 100 for autonomous enriching reference information.

Method 100 may start by step 110 of receiving (for example—from a human in the loop) a certain number (for example 5-20) of non-faulty reference item instance images.

Step 110 may be followed by step 120 of generating cropped images from the non-faulty reference item instance images.

Step 120 may be followed by step 130 of running inference on the cropped images using a non-item specific neural network—for example a generic, industry standard state-of-the-art pre-trained neural network (such but not limited to wide ResNet50 pretrained on ImageNet, though other architectures and/or training datasets may be applied mutatis mutandis). The non-item specific neural network may be a feature extractor.

Step 130 may be followed by step 140 of interpolating the final-layer feature map (although other layers and/or a combination of concatenated layers may be applied mutatis mutandis) of the feature extractor to the dimensions of the segmented image. The interpolating may “match” the dimensions of the final layer feature map to the dimensions of the segments image. Various interpolations may be used—for example a Bi-linear interpolation.

Step 140 may be followed by step 150 of calculating, per each of the non-faulty reference item instance images, a pixel wise mean value of the feature map for every pixel i, j as well as a covariance matrix-which together shall here on forth be referred to as the distribution.

Step 150 may be followed by step 160 of inferring the non-item specific neural network on each acquired image of a dataset (the dataset may include images other than the non-faulty reference item instance images) and collecting a value (denoted max_score) of a pixel with the maximum distance (from the distribution) in the image. Any distance may be used—for example Mahalanobis distance. The collected value may differ from the maximal value—for example may be the y′th highest value, whereas y may differ from 1, may range between 2 to 10 or 20, and the like.

Step 160 may be followed by step 170 of adding a predefined number (or a predefined portion of the images to be mined—for example up to 20%, 25%, 30%, 35%, and the like) of images with the lowest max scores to the non-faulty reference item instance images.

Step 170 may be followed by step 180 of determining whether to perform another iteration of steps 160 and 170—for example checking whether the non-faulty reference item instance images (after the adding of step 170) reaches a predefined threshold.

After a completion of the required repetitions-step 180 may be followed by step 190 of responding to the outcome of step 170.

Step 190 may include calculating the distance between the final decided upon centroids and every non-centroid image. The number of X (value of X may be determined in any manner) images with the highest distance can also be believed to be faulty.

Step 190 may include storing the non-faulty reference item instance images (after the adding of step 170), transmitting the non-faulty reference item instance images (after the adding of step 170), using the non-faulty reference item instance images (after the adding of step 170) in future inspection processes, and the like.

Method 200 may start by step 210 of obtaining a current group of trusted reference images of non-anomalous instances of an item.

Step 210 may be followed by step 220 of calculating, based on the current group, reference pixel-wise distribution information.

Step 220 may be followed by multiple repetitions of steps 230-280. Steps 210-220 may repeated multiple times.

Step 230 may include obtaining multiple sets of item instance pixels from current acquired images of instances of the item, each set originated from an image of an instance of the item and comprises multiple item instance pixels of the additional instance of the item.

Step 230 may include obtaining of the multiple item pixels by receiving an image and generating a cropped image that comprises the multiple item pixels.

Step 230 may be followed by step 240 of determining item features of the item for each set, based on the multiple item pixels of the set and by a non-item specific neural network.

The non-item specific neural network may be pre-trained to perform feature extraction of objects, at least some of the objects differ from the item.

Step 240 may be followed by step 250 of determining, based on the item features, a pixel score for item pixels of the multiple item pixels.

Step 250 may be followed by step 260 of calculating, for each of the item pixels, a distance between the pixel score and the reference pixel-wise distribution information.

The reference pixel-wise distribution information may include a reference mean matrix and reference covariance matrix.

The group covariance information may be a covariance matrix and the mean value information is a group mean value matrix.

Step 260 may be followed by step 270 of selecting at least one current acquired image to add to the trusted reference images, based on at least one distance of at least one pixel per current acquired image.

Step 270 may be based on a maximal distance per current acquired image.

Step 270 may include searching for up to a predefined number of current acquired images of lowest maximal distance. The predefined number does not exceed a number of current group of trusted reference images of a first iteration.

Step 270 may be followed by step 280 of responding to the selecting. The responding may include storing the one or more images of non-anomalous items, storing the one or more selected clusters, using the selected one or more images during an inspection or an evaluation process, and the like

Steps 230-270 may be repeated multiple times until reaching a predefined overall number of trusted reference images.

Method 300 may start by step 310 of obtaining a current group of trusted reference images of non-anomalous instances of an item.

Step 310 may be followed by step 320 of calculating, based on the current group, reference pixel-wise distribution information.

Step 320 may be followed by multiple repetitions of steps 330-390. Steps 310-320 may repeated multiple times.

Step 330 may include obtaining multiple sets of item instance pixels from current acquired images of instances of the item, each set originated from an image of an instance of the item and comprises multiple item instance pixels of the additional instance of the item.

Step 330 may be followed by step 340 of determining item features of the item for each set, based on the multiple item pixels of the set and by a non-item specific neural network.

Step 340 may be followed by step 350 of determining, based on the item features, a pixel score for item pixels of the multiple item pixels.

Step 350 may be followed by step 360 of clustering the pixel scores to provide multiple clusters having corresponding number of members.

Step 360 may be followed by step 370 of selecting one or more of the multiple clusters, based on the number of members per cluster to provide one or more selected clusters.

Step 370 may be followed by step 380 of selecting one of more images associated with the one or more selected clusters as one or more images of non-anomalous items.

Step 380 may be followed by step 390 of responding to the selecting. The responding may include storing the one or more images of non-anomalous items, storing the one or more selected clusters, using the selected one or more images during an inspection or an evaluation process, and the like.

Step 380 may include selecting one of more clusters with a highest number of members. For example—the X′th clusters having the X′th highest number of members. The value of X may be determined in any manner.

The mentioned above methods provide highly accurate and effective (in terms of computational and/or storage resource utilizations.

For example—when applying the methods using an iterative process of adding only a relative small number (for example—in the range of 10-100, 50-300 and the like) of images to the trusted reference images bank-allows to gradually expand the variability of the accepted reference images rather that adding more images at a time (and hence variability) which could be detrimental to the accuracy of the distribution building. Hence, the methods enable a gradual build of the distribution in a controlled way that manifests its accuracy superiority.

Furthermore—the mentioned above methods managed to reduce the false alarm rate (percentage of not good images being erroneously accepted as trusted reference images) from around 40% (achieved by a single non-iterative approach) to 0% (achieved by the iterative method we are currently discussing).

FIG. 4 is an example of images and data structures such as non-faulty reference item instance images 402, cropped images 404, feature map 406 (of final layer or another layer), acquired images 408 of a dataset, distribution 410, predefined number of images with the lowest max scores 412, current group of trusted reference images of non-anomalous instances of an item 414, reference pixel-wise distribution information 416, sets (418) of item instance pixels from current acquired images of instances of the item, item features 420, pixel score 422, selected at least one current acquired image 424, clusters 426, selected one or more clusters 428, one or more images of non-anomalous items 430.

FIG. 5 illustrates an example of a computerized system 500 and a manufacturing process tool 520.

The computerized system 500 may execute method 100 and/or method 200 and/or method 300.

The computerized system 500 may or may not communicate with the manufacturing process tool 520—for example to provide feedback about the manufacturing process applied by the manufacturing process tool 520 (that manufactured the evaluated manufactured items) and/or for receiving images of the evaluated manufactured items, and the like. The computerized system 500 may be included in the manufacturing process tool 520.

The computerized system 500 may include a communication unit 504, memory 506, processor 508 and may optionally include a man machine interface 510. The processor 508 may implement a neural network such as but not limited to a non-item specific neural network 509.

It will be appreciated by persons skilled in the art that the embodiments of the disclosure are not limited by what has been particularly shown and described hereinabove. Rather the scope of the embodiments of the disclosure is defined by the appended claims and equivalents thereof.