Patent ID: 12205363

DETAILED DESCRIPTION

Reference throughout this specification to “one embodiment,” “an embodiment,” or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, appearances of the phrases “in one embodiment,” “in an embodiment,” and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment, but mean “one or more but not all embodiments” unless expressly specified otherwise. The terms “including,” “comprising,” “having,” and variations thereof mean “including but not limited to” unless expressly specified otherwise. An enumerated listing of items does not imply that any or all of the items are mutually exclusive and/or mutually inclusive, unless expressly specified otherwise. The terms “a,” “an,” and “the” also refer to “one or more” unless expressly specified otherwise. The term “and/or” indicates embodiments of one or more of the listed elements, with “A and/or B” indicating embodiments of element A alone, element B alone, or elements A and B taken together.

Furthermore, the described features, advantages, and characteristics of the embodiments may be combined in any suitable manner. One skilled in the relevant art will recognize that the embodiments may be practiced without one or more of the specific features or advantages of a particular embodiment. In other instances, additional features and advantages may be recognized in certain embodiments that may not be present in all embodiments.

These features and advantages of the embodiments will become more fully apparent from the following description and appended claims or may be learned by the practice of embodiments as set forth hereinafter. As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method, and/or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module,” or “system.” Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having program code embodied thereon.

The computer readable medium may be a tangible computer readable storage medium storing the program code. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, holographic, micromechanical, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

More specific examples of the computer readable storage medium may include but are not limited to a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), an optical storage device, a magnetic storage device, a holographic storage medium, a micromechanical storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, and/or store program code for use by and/or in connection with an instruction execution system, apparatus, or device.

Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object-oriented programming language such as Python, Ruby, R, Java, Java Script, Smalltalk, C++, C sharp, Lisp, Clojure, PHP or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). The computer program product may be shared, simultaneously serving multiple customers in a flexible, automated fashion.

The schematic flowchart diagrams and/or schematic block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. Although various arrow types and line types may be employed in the flowchart and/or block diagrams, they are understood not to limit the scope of the corresponding embodiments. Indeed, some arrows or other connectors may be used to indicate only an exemplary logical flow of the depicted embodiment.

The description of elements in each figure may refer to elements of proceeding figures. Like numbers refer to like elements in all figures, including alternate embodiments of like elements.

FIG.1Ais a schematic block diagram of computer vision system100. The computer vision system100may inspect automation equipment. In addition, the computer vision system100may inspect work in progress parts, finished parts, and the like. In the depicted embodiment, the computer vision system100includes a user interface101, the communication module105, a computer vision device103, and a computer vision model107.

The communication module105may facilitate communications between the user interface101, the computer vision device103, and the computer vision model107. The communication module105, may comprise networks including wireless networks and wired networks, servers, and the like. The communication module105may be configured as a network.

The computer vision device103may capture images of the automation equipment, the work in process parts, and/or the finished parts. The computer vision device103may be configured as a computer and/or an edge device in a network.

The computer vision model107may be trained to identify component wear, failure scenarios, process errors, part nonconformities, and the like from the images captured by the computer vision device103.

The user interface101may present the images and other information including image inferences from the computer vision model107to the user. Unfortunately, a presented image may include an inaccurate and/or suboptimal identification of features in the image and/or predictions for the image component. In the past, improving the training of the computer vision module107was difficult, and may have been delayed by the training process before presenting these inaccurate and/or suboptimal features to the user. As a result, needed updates to the computer vision model107were difficult to implement.

The embodiments described herein allow a user to update the computer vision module107by creating a user input drawing on a presented image. The user input drawing may consist of free-hand pen/brush and shape tools to add or subtract a pixel mask to a training image representation, either directly or through assistance via the AI models or traditional computer vision tools. The user input drawing consists of an image of the same size as the first input image, where each pixel is a single bit designating whether an instance of a particular classification is represented or not represented. The first image can contain no input or user input drawing image representations, designating multiple classifications and instances of these classifications. The embodiments may convert the user input image to a training format image in a training format for the computer vision model107. The embodiments may further generate a training-representation drawing from the training format image and receive user feedback for the training-representation drawing. The embodiments may update the computer vision model107based on the user feedback to rapidly improve the efficacy of the computer vision model107. As a result, the performance of the computer vision system100is improved.

FIG.1Bis a schematic block diagram of the computer vision model107. In the depicted embodiment, the computer vision model107employs a modular algorithm architecture for switching between different vision tasks. The computer vision model107may include a model backbone111, an anomaly detection head113, an anomaly classification head119, an anomaly segmentation head125, an image classification head127, an object detection head129, an image segmentation head131, and an instance segmentation head133. As used herein, a head is a top of a network that consists of neural networks and/or statistical models. The model backbone111may perform computer vision functions to extract numerical features from any given image201. The anomaly classification head119may detect the presence of a class in images201. The anomaly segmentation head125may be used to detect anomalies' pixel locations within images201. The image classification head127may be used to detect the presence of a class in images201. The object detection head129may be used to detect the presence of a class, the number of class instances, and the general location via position and size of a box encompassing each class instance. The image segmentation head131may be used to detect the presence of a class and specify all pixels in images201as belonging to one of many classifications. The instance segmentation head133may be used to detect the presence of a class, the number of instances per classification, and the exact pixels in the images201for each instance where the same pixels may belong to multiple instance.

The computer vision model107may use the model backbone111from either a transfer-learning model or a meta-learning model using network weights from pre-training (e.g., unsupervised, self-supervised, or supervised) or using network weights from open-sourced model-weights such as a residual neural network (RESNET) trained on IMAGENET® database images. In a certain embodiment, no neural networks are trained. Instead, output of networks trained on open-source weights to inform a statistical model used for predictions such as inference. Meta-learning approaches applied by some inference heads may be much more lightweight to train and so may be done in minutes or faster on an edge device such as a computer. In one embodiment, the user may have the option to disable automatic retraining on annotation-updates to favor more intense training pipelines that will take more computation to complete such as on-device or on an external training machine.

FIG.1Cis a drawing of images201. The images201may be presented to the user via a user interface. In the depicted embodiment, images201of a saw blade are presented. A first image201aincludes an image annotation123. The image annotation123may emphasize a feature, prediction, characteristic, or the like of the image201. In the depicted embodiment, the image annotation123indicates a potential crack.

FIG.1Dis a drawing of user input drawings203. In the depicted embodiment, the images201ofFIG.1Care annotated with user feedback entered by the user via the user interface101. The user feedback may be a clear indication253indicating that the user has no pixels of interest in image201. In addition, a user annotation121may be added to the user input drawing203. The user input drawing203may be converted to a training format image as will be described hereafter.

FIG.1Eis a drawing of training-representation drawings207. In the depicted embodiment, the user input drawings203ofFIG.1Dthat were converted to training format images205are generated as training-representation drawings207. The training-representation drawings207may include image inferences211and/or saliency maps213.

The image inference211may comprise a pixel map agreement metric that highlights commonly marked and differently marked pixels in the user annotation121and image annotations123, pixel percentage metrics that compare the commonly marked and differently marked pixels in the user annotation121and image annotations123, and a comparison heatmap metric that shows the sums of the commonly marked and differently marked pixels over regions.

For example, if the user annotation121and the image annotation123differ by a set of X pixels while having a set of Y pixels in common, the pixel percentage metric PPM may be calculated using Equations 1-3,

P⁢P⁢M=YX+YEquation⁢1P⁢P⁢M=XX+YEquation⁢2P⁢P⁢M=❘"\[LeftBracketingBar]"X-Y❘"\[RightBracketingBar]"X+YEquation⁢3

The pixel map agreement metric may highlight the set of X pixels with a first color and the set of Y pixels with a second color.

In one embodiment, the comparison heatmap metric CHM is calculated for each pixel using Equation 4, where X(x, y) and Y(x, y) are specific pixels in the sets of X and Y pixels respectively.
CHM=|X(x,y)−Y(x,y)|  Equation 4

The saliency maps213may indicate areas of an image201that had the greatest influence on the model inference211. For example, the saliency map213may alert a user to increased uncertainty and/or long-term trends. The saliency map213may also determine a type of uncertainty, such as novelty or noise. The image inference211and/or saliency maps213may be generated from the input of at least one user.

FIG.1Fis a drawing of user feedback251for the training-representation drawings207ofFIG.1E. in the depicted embodiment, the user feedback251includes annotation pixels257, annotation colors259, an annotation bounding box255, and annotation layers261. The user feedback251is generated from the user annotation121. The user annotation121may be analyzed to determine agreement and/or disagreement with the image inferences211and/or saliency maps213presented in the training-representation drawings207. The user feedback251may be used to train and/or update the computer vision model107as will be described hereafter.

FIG.2Ais a schematic block diagram of image data200for an image201. The image data200may be organized as a data structure in a memory. In the depicted embodiment, the image data200includes the image201, the user input drawing203, the training format image205, the training-representation drawing207, and a drawing format209.

The image201may record the output of the computer vision device103. The user input drawing203may include user annotations121and/or user feedback251added to the image201via a user interface101. The user input drawing203may be in a format specified by the drawing format209. The drawing format209consists of selected pixels via the user interface101, which provides affordances for drawing free-hand, lines, boxes, circles, and other shapes with or without AI assistance. The training format image205may represent the user input drawing203in a training format for the computer vision model107. The training-representation drawing207may be generated from the training format image205.

In one embodiment, only a singular drawing interface is provided. As a result, a user need only learn the singular user interface101. The embodiments may limit improper labeling via the user annotation121, increasing the accuracy of feedback to the computer vision model107. In a certain embodiment, high resolution and precise user annotation121is employed via a singular drawing interface.

In the past, when converting between inference/training tasks, like from object detection to segmentation, when labeling employed bounding boxes, the user would have to relabel the images from bounding boxes to a higher-precision pixel-wise mask for segmentation. Because the embodiment may force users to start using high-resolution pixel mask user annotations121, user input is more consistent.

The user annotation121and the image annotation123share the drawing format209. As a result, the embodiments can determine whether the computer vision model107is inferring classes and/or instances for the same reasons as the users are inferring classes and/or instances from the same images201.

FIG.2Bis a schematic block diagram of training-representation drawing data208for the training-representation drawing207. The training-representation drawing data208may be organized as a data structure in a memory. In the depicted embodiment, the training-representation drawing data208includes the image201, an image inference211, a saliency map213, and feedback information220. The feedback information220may comprise a classification of the image201, the characterization of the image201, and/or an anomaly in the image201.

The use of high precision user annotation121improves the training-representation drawing data208and/or model data. In the past, differences identified by bounding boxes resulted in lower accuracy data. For example, two identical and overlapping bounding boxes each enclosing a spiral that does not intersect with the other spiral may have no shared pixels in their respective masks used for image segmentation. The use of high precision user annotation121thus increases the accuracy of the training-representation drawing data208and/or model data.

FIG.2Cis a schematic block diagram of the feedback information220. In the depicted embodiment, the feedback information220includes at least one of a coverage agreement221, an intersection over union (IoU) coverage223, a saliency coverage225, a ground truth coverage227, a shared focus area229, a divergent focus area231, a confusion area233, and a suggestion235.

A coverage agreement221may indicate a percentage agreement and/or specific pixel agreement between at least two user annotations121. The IoU coverage IoU223may be calculated using Equation 5, where S are pixels for an area of interest in a comparison drawing (user- or model-generated) and G are pixels for an area of interest in the image201from a user annotation.

I⁢o⁢U=❘"\[LeftBracketingBar]"S⋂G❘"\[RightBracketingBar]"❘"\[LeftBracketingBar]"S⋃G❘"\[RightBracketingBar]"Equation⁢5

The saliency coverage SC225may be calculated using Equation 6.

S⁢C=❘"\[LeftBracketingBar]"S⋂G❘"\[RightBracketingBar]"❘"\[LeftBracketingBar]"S❘"\[RightBracketingBar]"Equation⁢6

The ground truth coverage GT227may be calculated using Equation 7.

G⁢T=❘"\[LeftBracketingBar]"S⋂G❘"\[RightBracketingBar]"❘"\[LeftBracketingBar]"G❘"\[RightBracketingBar]"Equation⁢7

The shared focus area229may indicate areas of specific pixel agreement between the at least two user annotations121. The shared focus area229may describe how user annotations121and image annotations123are similar or dissimilar and indicate what actions may be done by the user to improve the computer vision model107. The divergent focus area231may indicate areas of no pixel agreement between the at least the user annotations121. The confusion area233may indicate regions of some pixel agreement and no pixel agreement between the at least two user annotations121. The suggestion235may record suggestions for improving an image inference211and are automatically generated and/or provided by a user.

A saliency map213may show whether the computer vision model107is inferring target classes/instances for the same reasons as the user.

FIG.2Dis a schematic block diagram of the user feedback251illustrated inFIG.1F. In the depicted embodiment, the user feedback251includes the annotation pixels257, the annotation colors259, and the annotation layers261. The annotation layers261may group instances of user annotations121. In addition, the annotation layers261may group classifications of user annotations121. The groups may be of similar instances and/or classifications. In addition, the groups may be of different instances and/or classifications.

The clear indication253may be determined from a user annotation121. In one embodiment, the clear indication253is determined from no user annotation121and/or minimal user annotations121. The clear indication253may identify clear images201that help the computer vision model107to distinguish points of interest from background in images. The clear indication253may be calculated to indicate selected pixel agreement between at least two user annotations121.

FIG.2Eis a schematic block diagram of format data271. The format data271may be organized as a data structure in a memory. In the depicted embodiment, the format data271includes a plurality of format converters273and the training format275. The training format275may specify at least one format for training data that is used to train the computer vision model107. The format converters273may convert the user input drawing203to a training format image205. The format converters273may also convert the user feedback251to a training format image205. The format converters273may further convert the training format image205into the training-representation drawing207.

FIG.2Fis a schematic block diagram of training data291. The training data291is used to train the computer vision model107. The training data291may be organized as a data structure in a memory. In the depicted embodiment, the training data291as one or more instances of an image201, an image inference211, and/or user feedback251.

FIG.2Gis a schematic block diagram of process history293. The process history293may be used to update the training data291. In addition, the process history293may be used to train the computer vision model107. The process history293may be organized as a data structure in a memory. In the depicted embodiment, the process history293includes process information295, the image201, and/or the image inference211. The process information205may comprise operating times, operating intervals, operating conditions, part yields, failure rates, and the like for automation equipment.

In one embodiment, images are time stamped and properly aligned with other measurements available from a process (e.g., temperature, pressure, etc.). An algorithm (e.g., clustering algorithms) on process data can be used to identify a given state of the operation. The operation states (e.g., clusters) may be used to filter or sort images presented to the user for annotation. A distance measure that combines similarity between images and operation states can be utilized to help users filter and search through images to annotate, improving the computer vision model performance within pertinent operation states.

FIG.2His a schematic block diagram of model data290. The model data290may be used to evaluate and/or improve the computer vision model107. The model data290may be organized as a data structure in a memory. In the depicted embodiment, the model data290includes a model health237, a user consistency239, an agreement score241, a target model health range243, a comparison245, and a process condition247.

The comparison245may compare at least two sets of training-representation drawings207, user input drawings203, and/or corresponding feedback information220. The user consistency230may be identified from the comparison245. In one embodiment, the user consistency230comprises a percentage of consistent pixels and user annotations121between the at least two sets of training-representation drawings207, user input drawings203, and/or corresponding feedback information220. The agreement score241may indicate an agreement between the user input drawings203, image inferences211, and/or corresponding feedback information220.

The target model health range243may specify a target range for the model health237. The model health237may indicate the certainty and/or efficacy of the computer vision model107using at least one parameter. The process condition247may be a failure condition, a part characteristic, an operational characteristic, a part failure characteristic, and the like.

FIG.3Ais a drawing of a model user interface300. The model user interface300may interact with a user via displays and input devices. The model user interface300presents the image201and/or image annotation123. In addition, the model user interface300may present labels303, predictions305, agreement307, metadata306, an annotation color selector319, an annotation pixel selector321, and an annotation bounding box selector323. Selecting labels303may present and/or add labels on the image201. Predictions305may present predictions from the computer vision model107. Agreement307may compare training-representation drawings207, user input drawings203, and/or corresponding feedback information220. The annotation colors selector319may select annotation colors259for user input. The annotation pixels selector321may select annotation pixels227for user input121. The annotation bounding box selector323may select a bounding box input.

The model user interface300may further include a draw selector311, an erase selector313, a fill selector315, and a move selector317. A select class selector327allows a class to be assigned to the image201. A reset labels button329resets labels on the image201. A label field325accepts a label for the image201. A directions field331presents labeling advice, directions, and/or other information to the user.

FIG.3Bis a drawing of the model user interface300ofFIG.3Bwith the user annotation121added to form a user input drawing203. In the depicted embodiment, the directions field331presents a fracture confidence333.

FIG.3Cis a drawing of a model user interface300. In the depicted embodiment, the model user interface300includes a human selector333, a both selector335, and a model selector337. The human selector333, both selector335, and model selector337may be used to select presenting the user annotation121, both the user annotation121and the image annotation123, or the image annotation123respectively. In the depicted embodiment, a training-representation drawing207is presented to the user.

FIG.3Dis a drawing of the model user interface300ofFIG.3C. In the depicted embodiment, the user annotation121is added to the training-representation drawing207to form user feedback251.

FIG.3Eis a drawing of a comparison presentation351of the comparison245. In the depicted embodiment, user feedback251from two different users of one image201is presented along with coverage agreement221shown as a Venn diagram.

FIG.3Fis a drawing of a model health presentation. The model health237is indicated is a graph over time.

FIG.3Gis a drawing of a user consistency presentation. The user consistency239is presented for the comparison245shown as a Venn diagram.

FIG.3His a drawing of a model health user interface360. Details of the model health237are shown for snapshots over time369. In the depicted embodiment, each snapshot includes but is not limited to an image frequency361, an inference frequency363, and a label frequency365. In one embodiment, an overall health367is presented as a color gradient background.

For example, the model health user interface360may show the number of images and detections with a color-gradient background, where green is healthy and red is unhealthy. Blue may indicate where no health metric is calculated, because of a lack of images or because the needed information has not been generated such as agreement metrics that require user-input, or model-health metrics that require a computer vision model107and/or model-generated image-features from the model backbone111. The user may see an orange-red region and hovers over the points with their cursor to quickly view images from that time period versus other time periods. The user may see that the lighting difference is significance, indicating the user should label images201from the red region where this lighting is an issue. Images and/or animated gifs may be used to describe a gradient visualization.

FIGS.3I-3Kare drawings of a computer vision model update process. InFIG.3I, an image201may be presented to a user. The image201may have an image annotation123. The user may annotate371the image201using the user interface101with a user annotation121to form a user input drawing203. The user annotation121is a drawing created using a drawing interface in the user interface101.

The user input drawing is converted373to a training format image205. A training format image205may include a format suitable for training a target computer vision model107, an image inference211, or a clear indication253for marking when an image is void of any particular areas of interest. The training image205may be used to update375and/or train375at least one computer vision model107.

InFIG.3J, at least one computer vision model107generates377at least one training-representation drawing207. A training-representation drawing207may include an image inference211. The training-representation drawing207may be presented379to the user to receive user feedback251. The user feedback251may be a drawing on the training-representation drawing207.

In the depicted embodiment, an image inference211aindicates that a training-representation drawing belongs to a specified class. The specified class may be defined by a user and/or administrator. In one embodiment, the specified class is automatically defined based on the image inferences211.

InFIG.3K, the user draws381a user annotation121on the training-representation drawing207. The user annotation121may be user feedback251. An agreement score241is calculated383between the image inference211and the user annotation121. The agreement scores241may be used to generate385labeling advice332that is presented in the directions field331.

FIG.4Ais a schematic block diagram of computer400. The computer400may be an edge device, a network device a server, and the like. In the depicted embodiment, the computer400includes a processor405, memory410, and communication hardware415. The memory410stores code and data. The processor405executes the code and processes the data. The communication hardware415may communicate with other devices.

FIG.4Bis a schematic diagram illustrating one embodiment of a neural network475. In the depicted embodiment, the neural network475includes input neurons450, hidden neurons455, and output neurons460. For simplicity, only a small number of input neurons450, hidden neurons455, and output neurons460are shown. The neural network475may be organized as a convolutional neural network, a recurrent neural network, RESNET, long short-term memory (LSTM) network, transformer network, and the like.

The neural network475may be trained with the training data291, including images, labels, classifications, and the like. The neural network475may be trained using one or more learning functions while applying the training data291to the input neurons450and known result values for the output neurons460. Subsequently, the neural network475may receive actual data at the input neurons450and make predictions at the output neurons460based on the actual data. The actual data may include images201, and augmented forms of images. Augmented forms of images include but are not limited to rotation, perspective change, flipping along axis, change in brightness, change in contrast, adding blur, adding sections from multiple images to one image, and so forth. These image augmentations allow for a model more robust to changes in image-capturing environments.

FIGS.5A-Bis a schematic flow chart diagram of a computer vision model update method500. The method500may update the computer vision module107and generate an image inference211for an image201. The method500may be performed by the computer vision system100and/or portions thereof such as a processor405. For simplicity, the following steps will be described as performed by the processor405. The method500may be performed on a single edge device computer400and/or a network of multiple computers400. In one embodiment, the method500is only performed by an edge device.

The method500starts, and in one embodiment, the processor405receives501the training data291. The training data may be without image inferences211, user feedback251, user annotations121, and/or image annotations123.

The processor405and/or the user may determine503whether to pre-train the computer vision model107. If the computer vision model is pre-trained, the processor405trains505and/or pre-trains505the computer vision model107using the training data291with or without any annotation data. For example, the computer vision model107may be pre-trained without receiving user annotations121.

The processor405may sort and/or filter506the images201. In addition, the processor405may provide sorting and/or filtering features to the user via the user interface101.

After uploading training data291to the system100, the processor405may run a pre-training script to refine the computer vision model107to better match images201in many down-stream tasks such as anomaly detection, classification, object detection, segmentation. In addition, the processor405may present507images201and the user may start annotating the images201with user annotations121without the training/pre-training step505by using pre-configured model weights.

In one embodiment, after annotating just a few images201, such as 5-10 images201, the processor405may provide the image inference211and saliency maps213of the training-representation drawing data208. This allows users to more-effectively understand what the computer vision model107is thinking/predicting while labeling and interacting with the computer vision model107using user annotations121rather than interacting with the computer vision model107post-hoc after large batches of annotation. This interactive form of annotation helps the computer vision model107become more informed over shorter periods of time, by having the human and computer collaboratively clarifying points of confusion.

In one embodiment, unsupervised learning is used to train the computer vision model107and/or model backbone111. Transfer-learning may modify the network ‘head’ for inference (if larger number of annotated images201). A meta-learning head may be used to conduct inference without a traditional neural network head (if smaller number of annotated images201or performance from meta-learning is sufficient).

If any training505is conducted on external computers400, the model backbone111from that training505may be utilized to improve the on-device inference and training. As many of the tunable network parameters are in the model backbone111, it may minimize training that cannot be done on an edge device computer400. In one embodiment, no training is required off an edge device computer400, and training only needs to be done once on an external computer400and may be repeated when model health237drops significantly.

In one embodiment, unlabeled images201are uploaded to the system100. The model backbone111may be trained505by having a network and/or server distinguish unlabeled images201from augmented versions of the same images201. Existing neural network weights, like open-source weights, may be used via transfer-learning to speed this process up, such as up to 100-200 epochs.

In one embodiment, before images201are annotated, the system100and/or processor405runs a pre-trained model backbone111or open-source weight model backbone111against all images201on the system100, saving all activations from network.

The processor405may present507at least one image201to the user via the user interface300. The user may employ the model user interface300to add a user annotation121to the at least one image201in the drawing format209to form a user input drawing203. The processor405may receive509the user input drawing203.

The processor405may convert511the user input drawing203to the training format image205in the training format275for the computer vision model107and/or model backbone111. In one embodiment, after a user annotates a small number, such as in the range of 5-10, of images201such as by marking them all with activations such as a clear indication253and/or a user annotation121. The processor405may run all clear indications253and/or user annotations121against a statistical model creating an image inference211of each image201in relation to images201, such as feedback information220indicating dissimilarity from group of empty images201and additionally generate saliency maps213or segments of the image201to render in the future. In one embodiment, this process requires no graphics processor unit (GPU) or less significant GPU usage as the model backbone111features are pre-calculated.

The processor405may generate513, with the computer vision model107, a training-representation drawing207from the training format image205. The training-representation drawing207may comprise an image inference211for the first image201. Training-representation drawings207may be generated513for a plurality images201.

The processor405may sort and/or filter515at least two training-representation drawings207. The sorting515may be informed by the user input drawings203including during generation of saliency maps213. The sorting and filtering515may also be based on user feedback251and/or feedback information220. The sorting515may generate a sortable image index to explain image inferences with respect to a guided user annotation121of a second image201.

In one embodiment, after a portion of this sorting process is completed, a user may sort by feedback information220and apply other filters, then start drawing user annotations121on images201to annotate or debug the computer vision model107. As a user is drawing, image inferences211and other information are calculated and shown in near-immediate time as the image inferences211are pre-calculated. AI-assisting drawing tools using the pre-generated image inferences211will be made available to help more quickly draw, for example using a click-and-drag box selection tool which finds parts of the machine inference-image that include a segment within that box to draw an initial best-guess.

The processor405may present517the training-representation drawings207to the user via the model user interface300. The processor405may further receive519user feedback251for the training-representation drawing207. The user feedback to51may be received in the drawing format209.

The processor405may update521the computer vision module107based on the user feedback251. The computer vision model107may be updated with training data291comprising images201and corresponding image inferences211and user feedback251. In addition, the computer vision model107may be updated with training data291comprising images201and corresponding image inferences211that incorporate user feedback251.

In one embodiment, training data291are automatically added or changed based on the number of available user-annotated user input drawings203, but can be manually set per classification-type, such as drawing color. Users may have the option to bootstrap training of some computer vision models107by using auto-labeling approaches during this process. If the training data291for the computer vision models107to be trained on edge device computer400exceeds the capacity of the edge device computer400, users may send the training data291to an external training device such as a server for more high-performance training with optional auto-labeling. If this external training is conducted, and the results are verified by a user, the processor405may deploy the computer vision model107in one or more computer vision systems100.

The processor405may generate523an image inference211for a second image201based on the updated computer vision model107. In one embodiment, the processor405caches partial results from image inferences211(e.g. backbone-generated features) to afford quicker updating of computer vision models107. In addition, the processor405may modify525a machine component based on the image inference211for the second image201and the method500ends. For example, the processor405may change a cutter based on the image inference211.

FIG.5Cis a schematic flow chart diagram of a model health improvement method550. The method550may improve the model health237of the computer vision model107. The method550may be performed by the computer vision system100or portions thereof including a processor405. The method550may be performed on a single edge device computer400and/or a network of multiple computers400.

In one embodiment, the processor405compares551at least two sets of training-representation drawings207, user input drawings203, and/or corresponding feedback information220. The processor405may identify553the user consistency239from the comparison245.

The processor405may identify555the model health237from the comparison245. Model health237may be calculated by using statistical tests comparing the distribution of pre-computed activations/features from user input drawings203to distributions of pre-computed activations/features from a group of incoming images201. In one embodiment, this does not require a GPU. A GPU may only be needed for training the model backbone111, inferring with a model backbone111(once per image for the most-recent updated backbone), and during inference if there is a neural network475.

The processor405may present557the model health237via the model user interface300such as illustrated inFIG.3H. In one embodiment, the processor405reconciles559conflicts between the at least two sets of training-representation drawings207, user input drawings203, and/or corresponding feedback information220.

Comparisons245can be utilized to consolidate or reconcile conflicting information between users and/or computer vision modules107, including for user consistency239. End goals for both improving model health247/user consistency239, and enlightening users of emergent points of confusion among humans so that more formal and more consistent definitions of how to mark user annotations121can be created which in turn should also improve the computer vision model107.

In one embodiment, the reconciled conflicts are presented to the user. The reconciled conflicts may inform the user on techniques, tactics, and/or strategies for applying user annotations121to the user input drawings203. The processor405may improve561the model health237to the target model health range243by reconciling559the conflicts.

After the computer vision model107reaches satisfactory performance, the computer vision system100can be run for a longer period of time without much supervision. If model health237crosses a threshold, a user may label and/or annotate more images where model health237was poor using interactive tools mentioned above to debug why models made mistakes, retrain the existing computer vision model107using the last methodology, and then change training methodology for users if needed. During all of the above, a manager can view and review annotations from all users and make changes to the training methodology. The manager will be the main point of contact to change device customization and helping configure how model output relates/maps to programmable logic controller (PLC) values.

To verify model-performance, pruning and quantization may be verified by the user to make sure the speed-accuracy tradeoff is good. To verify the speed-accuracy performance, annotations may be used as this varies per application/dataset. This creates a chicken-and-egg problem, where pruning could make the process faster but to evaluate that the system still works, we need the output of the pruned system. Because of this, pruning and/or quantization could optionally be done around step561when the model is about at satisfactory performance. The method550could make the option available as soon as right before step555, but making it clear there are risks with pre-maturely optimizing speed before knowing the objective of the system.

FIG.5Dis a schematic flow chart diagram of an alternate computer vision model update method600. The method600may update the computer vision model107. The method600may be performed by the computer vision system100and/or portions thereof including the processor405. The method600may be performed on a single edge device computer400and/or a network of multiple computers400.

The method600starts, and in one embodiment, the processor405records601the process history293. The process history293may include process information295, images201, and/or corresponding image inferences211. The process history293may be recorded601for at least one user. In addition, the process history293may be shared between computer vision systems100.

In one embodiment, the processor405appends603the process history293to the training data291. The processor405may update605the computer vision model107with the training data291and the method600ends.

FIG.5Eis a schematic flow chart diagram of an alternate computer vision model update method according to an embodiment. The method600may be performed by the computer vision system100and/or portions thereof including the processor405. The method650may be performed on a single edge device computer400and/or a network of multiple computers400.

The method620starts, and in one embodiment, the processor405identifies621a process condition247. The process condition247may be a failure condition, a part characteristic, an operational characteristic, a part failure characteristic, and the like.

The processor405may present623a second image201and/or process information295corresponding to the process condition247. The second image201may have a similar failure condition, part characteristic, operational characteristic, part failure characteristic, or the like.

The processor405may receive a user input drawing203comprising the user annotation121of the second image201. The processor405may generate627a training-representation drawing207from a training format image205of the user input drawing203. In addition, the processor405may update the computer vision model107with the training-representation drawing207and the method620ends.

Problem/Solution

Computer vision models107can be difficult to understand and maintain. Many of the decisions made by the computer vision models107are implemented by neural networks450that are difficult to inspect or update in a timely manner without experienced computer vision practitioners. Additionally, tools made to help novice users to setup, label, train, evaluate, optimize, monitor, and maintain computer vision systems tend to be disconnected, requiring Information Technology specialists to setup and interconnect.

The embodiments described herein provide an intuitive and wholistic feedback/training mechanism for computer vision models107. The embodiments allow a user to provide feedback via a user input drawing203with a user annotation121. The user input drawing203is converted to a training format image205that may be used to train and/or update the computer vision model107. In addition, the embodiments generate a training-representation drawing207from the training format image205. The training-representation drawing207may be presented to the user and user feedback251received from the user. The computer vision model107may be updated based on the user feedback251. As a result, the computer vision model107may be easily maintained, and updated.

This description uses examples to disclose the invention and also to enable any person skilled in the art to practice the invention, including making and using any devices or systems and performing any incorporated methods. The patentable scope of the invention is defined by the claims and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal language of the claims.