Patent Description:
Computer vision techniques enable a system to gain insight into its environment based on digital images, videos, scans, and similar visual mechanisms. High-level vision systems are necessary for a machine to accurately acquire, process, and analyze data from the real world. Computer vision and machine learning techniques allow a machine to receive input and generate output based on the input. Some machine learning techniques utilize deep artificial neural networks having one or more hidden layers for performing a series of calculations leading to the output. In many present-day applications, convolutional neural networks are used for processing images as input and generating a form of output or making decisions based on the output.

Artificial neural networks, modeled loosely after the human brain, learn mapping functions from inputs to outputs and are designed to recognize patterns. A deep neural network comprises an input layer and an output layer, with one or more hidden layers in between. The layers are made up of nodes, in which computations take place. Various training methods are used to train an artificial neural network during which the neural network uses optimization to continually update weights at the various nodes based on failures until a satisfactory model is achieved. Many types of deep neural networks currently exist and are used for a broad variety of applications and industries including computer vision, series forecasting, automated driving, performing medical procedures, aerospace, and many more. One advantage of deep artificial neural networks is their ability to learn by example, rather than needing to be specifically programmed to perform a task, especially when the tasks would require an impossible amount of programming to perform the operations they are used for today.

<NPL> discloses a method of image segmentation were a set of slightly different segmentations is derived from the same input and the final result is based on the consensus among them. <NPL> discloses an image segmentation algorithm which combines the results of a hybrid clustering ensemble.

It is with respect to this general technical environment that aspects of the present technology disclosed herein have been contemplated. Furthermore, although a general environment has been discussed, it should be understood that the examples described herein should not be limited to the general environment identified in the background.

This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject.

Various embodiments of the present technology generally relate to robotic devices and artificial intelligence. More specifically, some embodiments relate to systems, devices, and methods for segmenting images and modeling their uncertainty. In an embodiment of the present technology, a method of operating an image segmentation system is disclosed in claim <NUM>.

In some embodiments, the method further comprises, for an additional identified object region, determining that a confidence associated with the additional identified object region does not exceed the confidence threshold. The image segmentation system may further comprise a computer-imaging system comprising one or more camera and a robotic device comprising at least one picking element. In certain embodiments, based on the final segmentation prediction, the segmentation system directs the robotic device to attempt to pick up an object of the plurality of distinct objects using the at least one picking element and determines that the robotic device successfully picked up the object using the picking element.

In an alternative embodiment of the present technology, a system is disclosed in claim <NUM>.

In yet another embodiment, one or more computer-readable storage media are disclosed in claim <NUM>.

The components in the drawings are not necessarily drawn to scale. While several embodiments are described in connection with these drawings, the disclosure is not limited to the embodiments disclosed herein. On the contrary, the intent is to cover all alternatives, modifications, and equivalents.

The drawings have not necessarily been drawn to scale. Similarly, some components or operations may not be separated into different blocks or combined into a single block for the purposes of discussion of some of the embodiments of the present technology. Moreover, while the technology is amendable to various modifications and alternative forms, specific embodiments have been shown by way of example in the drawings and are described in detail below. The intention, however, is not to limit the technology to the particular embodiments described. On the contrary, the technology is intended to cover all modifications, equivalents, and alternatives falling within the scope of the technology as defined by the appended claims.

The following description and associated figures teach the best mode of the invention. For the purpose of teaching inventive principles, some conventional aspects of the best mode may be simplified or omitted. The following claims specify the scope of the invention.

Various embodiments of the technology described herein generally relate to systems and methods for modeling uncertainty. More specifically, certain embodiments relate to neural network models for expressing uncertainty in various relevant dimensions and methods for utilizing knowledge related to uncertainty in a meaningful way. In some embodiments, a robotic device may work in collaboration with a computer vision system for collecting visual data. Based on the visual data, machine learning techniques are implemented for identifying and quantifying uncertainty related to one or more dimensions of the visual data. The system can then make decisions related to future actions performed by the robotic device based on the uncertainty and operate the robotic device accordingly. In some examples, the machine learning techniques comprise the utilization of one or more artificial neural networks.

Artificial neural networks, such as those that may be implemented within embodiments related to computer vision, uncertainty modeling, picking, segmentation, ranking, and depth perception models described herein, are used to learn mapping functions from inputs to outputs. Generating mapping functions is done through neural network training processes. Many various types of training and machine learning methods presently exist and are commonly used including supervised learning, unsupervised learning, reinforcement learning, imitation learning, and many more. During training, the weights in a neural network are continually updated in response to errors, failures, or mistakes. In order to create a robust, working model, training data is used to initially dial in the weights until a sufficiently strong model is found or the learning process gets stuck and is forced to stop. In some implementations, the weights may continue to update throughout use, even after the training period is over, while in other implementations, they may not be allowed to update after the training period.

Parameters of a neural network are found using optimization with many, or sometimes infinite, possible solutions. Modern deep learning models, especially for computer vision and image processing, are based on convolutional neural networks, although may also incorporate other deep generative models. As described herein, artificial neural networks for uncertainty modeling, computer vision, robotic picking, and other processes described herein first require training. A variety of different training methods may be used to train a neural network for modeling uncertainty, segmenting units, or picking and placing items in a bin in accordance with embodiments of the technology described herein.

There is inherently some degree of uncertainty that comes along with an output to a machine learning model. In many scenarios, the uncertainty goes unused or ignored. However, a neural network model able to express that uncertainty can enable decision-making based on risk tolerances or other confidence parameters associated with a task. Most computer vision based prediction models used today output a single, unimodal prediction. Single prediction models can be wrong in their results but have no way to know that they are wrong or why. The output distributions from these predictions are often not expressive enough to capture the full range of uncertainty. For critical applications, single predictions may be insufficient and provide no means for trading off confidence with other performance metrics. Thus, training models that have more expressive output distributions so that they can model uncertainty and allow decision making based on high-confidence predictions derived from predicted uncertainty distributions.

Having a representation of uncertainty can benefit machines and machine learning techniques in a wide variety of applications. One application contemplated herein is the application of uncertainty modeling to robotic picking because when a machine is going to interact with another object, it can be very useful to understand how confident a "best guess" is before attempting to interact with the object. In some examples, a neural network may express uncertainty in various relevant dimensions related to another object or an item that the robotic device intends to pick up and algorithms may then be used to operate the robot differently based on uncertainty.

Parameters of an object where it may be useful to understand how uncertain a machine is regarding the parameter may include an object's weight, physical shape, materials, size, edges, and similar parameters that a computer vision system may have uncertainty about. A traditional neural network may just produce a best guess for each of those parameters, but in scenarios where it is best to be risk averse, it can be dangerous or have consequences to take a shot in the dark if the best guess it still relative uncertain. Thus, in the present example, one or more neural networks may be used to determine the uncertainty and according affect the robot's behavior. Understanding the uncertainty associated with an action in this scenario may be important to get right because getting it wrong could cause a wide variety of negative results. For example, if a robot picks up an item to move it to another bin or conveyor belt, but accidentally picks up two items because it was incorrect regarding the shape and boundaries of the item, it can cause issues such as dropping, damaging equipment, incorrect order fulfilment, and the like.

Many scenes a robot and/or a computer vision system sees are inherently ambiguous. For example, when looking at a scene with tightly packed items, the exact boundaries of each item may not visible, or may be hard to decipher for a variety of reasons. A model that outputs only one segmentation prediction is unable to properly capture this uncertainty. Thus, embodiments of the present disclosure use latent codes and variant mask-regions with a convolutional neural network (Mask-R-CNN) to achieve high-level instance segmentation predictions, wherein the R-CNN of the present embodiments may take one or more images of a scene and identify at least one distinct object boundary. Masks may be used to label object boundaries corresponding to a given segmentation prediction. One benefit of the present approach is the ability to express uncertainty only is situations or dimensions where uncertainty exists. For example, if a scene comprises several objects that have properties making them visually distinct and easy to decipher, the system produces a segmentation prediction that has little to no uncertainty, and the plurality of guesses may be almost exactly the same. However, if a scene also comprises objects with boundaries that are difficult to decipher with computer vision, the system may advantageously use its knowledge of what is certain and what is uncertain to increase accuracy and efficiency in predictions or actions.

<FIG> illustrates an example of warehouse environment <NUM> having robotic arm <NUM> for picking items from a bin in accordance with some embodiments of the present technology. <FIG> includes robotic arm <NUM>, bin <NUM>, conveyor belt <NUM>, camera <NUM>, and camera <NUM>. Robotic arm comprises picking element <NUM>. Picking element <NUM> comprises a set of suction-based picking mechanisms, however different numbers and types of picking mechanisms may be utilized in accordance with the present embodiment. Bin <NUM> is holding boxes that may be found in a warehouse, commercial setting, or industrial setting. Many other types of items may be in a bin or similar container for picking in accordance with the present embodiment. In the present example, robotic arm <NUM> is a six-degree-of-freedom (6DOF) robotic arm. Picking element <NUM> is designed for picking items out of bin <NUM> and placing them onto compartments of conveyor belt <NUM>.

An autonomous robot may benefit from having a means for recognizing the environment around it and processing that information to come up with a way to perform a task. Thus, if a robot is picking items out of a bin, its ability to sense the location and position of a specific item and apply that to determine how to pick up the item and move it to a desired location is beneficial. A robot capable of sensing and applying that knowledge, even within highly repetitive settings, dramatically decreases the need for human intervention, manipulation, and assistance. Thus, human presence may no longer be required when items aren't perfectly stacked or when a robot gets stuck, as a few examples. If a robot regularly gets stuck, it may defeat the purpose of using a robot altogether, because humans may be required to frequently assist the robot.

In some examples, robotic arm <NUM> and picking element <NUM> may pick boxes from bin <NUM> one at a time according to orders received and place the items on the conveyor belt for packaging or place them into packages for shipment. Furthermore, robotic arm <NUM> and picking element <NUM> may be responsible for picking items from various locations in addition to bin <NUM>. For example, several bins comprising different merchandise may be located in proximity to robotic arm <NUM>, and robotic arm <NUM> may fulfill requests for the different pieces of merchandise by picking the correct type of merchandise and placing it onto conveyor belt <NUM>.

Picking element <NUM> may comprise one or more picking mechanisms for grabbing items in a bin. Picking mechanisms may include one or more suction mechanisms, gripping mechanisms, robotic hands, pinching mechanisms, magnets, or any other picking mechanisms that could be used in accordance with the present disclosure. In some examples, picking element <NUM> may be additionally used for perturbation, such as poking, touching, stirring, or otherwise moving any items in bin <NUM>, as just a few examples. In further examples, robotic arm <NUM> may comprise a perturbation element such as a pneumatic air valve connected to a pneumatic air supply, wherein the pneumatic air valve blows compressed air into bins in certain situations. A perturbation sequence may be used in situations where the DNN or another model determines that there is low probability that it will be able to pick up any items in bin <NUM> as they are presently arranged. In some examples, the robotic arm may have already tried and failed to pick every visible item in the bin, and therefore decides to initiate a perturbation sequence. Robotic arm <NUM> may move and position picking element <NUM> such that it is able to pick up an item in bin <NUM>. In certain embodiments, determining which item to pick up and how to pick it up is determined using at least one deep artificial neural network. The deep neural network (DNN) may be trained to guide item pick-up and determine which items have the greatest probabilities of pick-up success. In other embodiments, picking may be guided by a program that does not use a DNN for decision making.

A computer vision system in accordance with embodiments herein may comprise any number of visual instruments, such as cameras or scanners, in order to guide motion, picking, and uncertainty modeling a computer vision system receives visual information and provides it to a computing system for analysis. Based on the visual information provided by the computer vision system, the system can guide motions and actions taken by robotic arm <NUM>. A computer vision system may provide information that can be used to decipher geometries, material properties, distinct items (segmentation), bin boundaries, and other visual information related to picking items from a bin. Based on this information, the system may decide which item to attempt to pick up and can then use the computer vision system to guide robotic arm <NUM> to the item. A computer vision system may also be used to determine that items in the bin should be perturbed in order to provide a higher probability of picking success. A computer vision system may be in a variety of locations allowing it can properly view bin <NUM> from, either coupled to or separate from robotic arm <NUM>. In some examples, a computer vision system may be mounted to a component of robotic arm <NUM> from which it can view bin <NUM> or may be separate from the robotic device.

Camera <NUM> images the contents of bin <NUM> and camera <NUM> images a region of conveyor belt <NUM>. Each of camera <NUM> and camera <NUM> may comprise one or more cameras. In some examples, a camera in accordance with the present example such as camera <NUM> comprises an array of cameras for imaging a scene such as bin <NUM>. Camera <NUM> and camera <NUM> are part of a computer vision system associated with robotic arm <NUM> such as a computer vision system in accordance with the technology disclosed herein.

In the example of <FIG>, robotic arm <NUM> has successfully picked box <NUM> from bin <NUM> and is in the process of moving box <NUM> to conveyor belt <NUM>. The computer vision system including camera <NUM> may help guide robotic arm <NUM> when picking box <NUM>. In some examples, before picking box <NUM>, the computer vision system images the contents of bin <NUM>, performs a segmentation sequence using the images, and identifies box <NUM> for picking based on the segmentation results. Based on the uncertainty model and distribution of segmentation predictions, the system of <FIG> determined that the uncertainties associated with box <NUM> were not beyond a confidence tolerance and therefore that the robot may attempt to pick box <NUM>. Robotic arm <NUM> may subsequently place box <NUM> onto conveyor belt <NUM> for distribution.

Many scenes a computer vision system sees are inherently ambiguous, such as a scene with tightly packed boxes or a single-sku bin where the items are difficult to tell apart. In a scene with matching boxes, such as in the present example, some dimensions of each box do not have enough visual information available to accurately predict the size every time. A model that outputs a single prediction would not be able to properly capture this uncertainty. In the case of segmentation, there are often regions of an image which are more likely to be an object that others. To find regions that are confidently objects, samples may be taken from a model of the image and then used to efficiently find the largest regions with a minimum amount of overlap. In some examples, object boundaries may be chosen based on a minimum percentage of agreement that a region is contained in an object. The flexibility of an adjustable percentage enables a tradeoff between being conservative in estimates with other performance metrics. For example, in some applications, double picks may be more costly or consequential than others. In a scenario where it is important to be cautious and avoid double picks, a high amount of agreement can be required, such as <NUM>%, before a system attempts to interact with an object, to reduce that chance of picking two items on accident. In other settings, additional resources may be available such as a scale to help detect double picks, so a lower confidence tolerance may be set which allows for more pickable area.

The uncertainty modeling used in accordance with <FIG> advantageously uses expressive output distributions to capture the full range of uncertainty associated with segmenting an image and then uses the distributions to make tunable confidence predictions that are well calibrated with the real world. A calibrated confidence prediction may be, for example, that the system must be <NUM>% confident in the segmentation of an object before it attempts to pick the item and the prediction will be correct <NUM>% of the time. In some examples, the downside of being incorrect about the dimensions of an object is large, and therefore if the system is uncertain about the object, it may be better to leave the object rather than picking it and allow a human operator to interact with the object instead. Alternatively, if there is a low confidence prediction before going to pick an item, it may allow the machine to act with a larger margin of error. For example, if the item may be smaller than predicted and the system is relatively uncertain of its boundaries, it may pick the item from a region of greatest certainty to avoid the risk of accidentally picking two items.

Segmentation plays an important role in warehouse environments and robotic picking such as in the example of <FIG>. Different scenarios can be extremely difficult to segment, especially when everything in a scene looks almost identical, has the same pattern, has overlapping objects, or similar scenarios. When a computer vision system images a scene, it collects pixels with intensity and color, but fails to gather information related to an understanding of the situation such as observing distinct objects next to each other. A computer vision system guided by a neural network can make a best guess at how to segment multiple items in an image but may not be able to determine if it is correct. There are many ways for a computer vision system to differentiate pixels from each other in an attempt to segment and image and there are lots of ways to fail at correctly segmenting images. Some scenarios may have high consequences for being wrong and taking action anyway, so the ability to model uncertainty in segmentation predictions can be extremely useful when determining whether or not to attempt to pick up an item. An incorrect understanding of a scene and the objects within in can cause an accidental pick up of multiple objects, thereby providing more items than requested or purchased in an order or causing a failed pick up by identifying a single object as multiple objects.

<FIG> shows process <NUM> for uncertainty modeling as it applies to instance segmentation. In step <NUM>, a computer vision system collects at least one image of a scene, wherein the scene comprises a plurality of distinct object.

<FIG> is a flow chart illustrating a series of steps for modeling uncertainty during instance segmentation in accordance with some embodiments of the present technology. In step <NUM>, a computer vision system collects at least one image of a scene, wherein the scene comprises a plurality of distinct objects. In some examples, the computer vision system may be the computer vision system including one or both of cameras <NUM> and <NUM> from <FIG>. The plurality of distinct object may be items from a bin, such as the contents of box <NUM>. In some implementations, one or more RGB images are captured and fed into a depth model for mapping depth in the image using one or more neural networks.

In step <NUM>, the computer vision system generates a plurality of segmentation predictions having identified regions corresponding to distinct objects. In some examples, identified regions are represented as object masks. A mask may be any label or representation illustrating a region of a distinct item in the image. Using an object mask, an RGB map, and a depth map, the system may generate segmentation predictions by determining which pixels belong to distinct objects. In the present example, a trained autoregressive model is used to produce segmentation predictions based on a random input such as time. The system may recursively generate segmentation predictions based on the random input until a satisfactory number of predictions has been produced. Each prediction may be generated based on a random seed or based on time, in some implementations.

In step <NUM>, the computer vision system outputs a final segmentation prediction comprising one or more identified regions chosen based on a confidence associated with the regions and a confidence requirement. The identified regions represent predicted regions of an object. In some examples, the outputted regions may form a boundary around the object, while in other examples, the outputted regions may represent parts of an object that the system has identified as likely to correspond to a single object. The final segmentation may be different from any single segmentation prediction from step <NUM> in some examples. The final segmentation may be an aggregate of regions derived from the predictions in step <NUM>, where regions with frequent overlap, such as <NUM>% of the time for a <NUM>% confidence requirement, are shown. In scenes without much uncertainty, such as in the example of <FIG>, the predictions from step <NUM> may all be very similar or the same and the final segmentation output in step <NUM> may therefore be very similar or the same as the predictions from step <NUM>.

<FIG> shows process <NUM> for segmenting images and modeling uncertainty in accordance with some embodiments of the present technology. In step <NUM>, an image segmentation system receives one or more RGB images of a scene comprising a plurality of distinct objects. Segmentation may be applied to a variety of applications, especially in applications in which a quantification of uncertainty may play a useful role in image processing. In step <NUM>, the image segmentation system generates a plurality of segmentation predictions, wherein each prediction comprises one or more object masks. Most computer vision models output a single prediction. However, the autoregressive instance segmentation model disclosed herein uses latent codes with variant Mask-R-CNN to achieve advanced instance segmentation and perform post-processing to find a confidence associated with different items.

Variations of predictions may come from one or more neural networks that use random numbers every time they are run, wherein each run corresponds to a single prediction. Based on the random number, the neural network should have a different output each time, unless the scene has no uncertainty associated with it. Using a latent variable model, the model can be run many times to get a distribution of outputs. In each iteration, a trained neural net will output a single, reasonable segmentation of the scene. After many iterations, such as hundreds in some examples, elements of the predictions that are more likely will have commonly appeared and elements that are unlikely will appear less often. After many iterations, the distribution will become proportional to scenarios that the neural net was trained, allowing for the recovery of what is likely and what is unlikely.

In step <NUM>, the image segmentation system identifies a predefined confidence requirement, wherein the confidence requirement identifies a minimum amount of required agreement for a region. As discussed previously, there may be scenarios in which a system should not output regions that they are not confident about, because of the high consequences associated with the scenario or any other risk tolerance. In other scenarios, it may be okay for the system to use less confident guesses. In step <NUM>, the image segmentation system outputs one or more object masks based on the confidence requirement. For example, if the system should only output regions that it is <NUM>% confident about then only regions will be output that <NUM>% of the predictions agreed upon.

<FIG> shows process <NUM> for image segmentation and uncertainty modeling in accordance with some embodiments of the present technology. In step <NUM>, a modeling system receives one or more RGB images of a scene comprising a plurality of distinct objects. In step <NUM>, the one or more images are provided to a depth model and a depth map of the scene is returned. In step <NUM>, the modeling system provides the images to a latent variable model to enable expressive output distributions. Most computer vision models output a single prediction. However, the autoregressive instance segmentation model disclosed herein uses latent codes with variant Mask-R-CNN to achieve advanced instance segmentation and perform post-processing to find a confidence associated with different items. In step <NUM>, the modeling system samples the latent variable model and makes a consistent prediction of one or more object masks in the scene. Step <NUM> may repeat until a satisfactory number of predictions have been produced. In some embodiments, a set number of predictions may be produced while in other embodiments there may a varying number of predictions produced based on an uncertainty associated with the predictions or a similar factor.

In step <NUM>, after a full set of object mask predictions have been generated, the modeling system outputs object masks that are fully contained in an object in x% of the samples. In the present example x% represents any certainty tolerance associated with a given scenario. As previously discussed, the outputted object masks may not be identical to any of the predicted object masks in step <NUM>. The regions of high overlap may be output in step <NUM>. In an example, the output regions may be any region that x% of segmentation predictions agree on.

A neural network for segmentation predictions, in some examples, may be trained on simulated data wherein virtual objects with known sizes and locations are segmented. The present technology does not require prior knowledge about segmented objects, such as from inventory data or a database. When the system first observes a scene, it considers the variety of different options for how the scene could be segmented, in addition to other considerations. The model may predict a finite number of guesses, such as <NUM>, for example.

The technology described herein should not be limited to robotic picking applications. The present technology has many applications in which a means for modeling uncertainty related to the outputs of neural networks is useful.

<FIG> illustrates an example flow within segmentation environment <NUM> in accordance with some embodiments of the present technology. In the present example, images <NUM> are used as input to and processed in unified reasoning module <NUM>. Images <NUM> may be one or more images taken by an imaging or computer vision system. Images <NUM>, in the present example, are collected by at least one camera in communication with a robotic arm, as is exemplified in <FIG>. The images are processed in unified reasoning module <NUM>. In some examples, unified reasoning model comprises at least one deep neural network trained for analyzing images. In the present example, image analysis includes performing segmentation and masking, RGB mapping, and depth mapping. However, fewer or additional image analysis processes may be included in unified reasoning module <NUM> and are anticipated.

Masking includes deciphering one or more distinct objects in images <NUM>. Masking may include a variety of sub-processes to assist in finding distinct objects and their boundaries. Understanding depth and RGB properties for each object may assist in segmentation and masking and may also assist when it is time to approach and pick up an item with a robotic device. Although masking, RGB, and depth are illustrated as individual outputs for the purpose of illustration, the outputs of unified reasoning module <NUM> may be a single, unified model comprising information related to the various types of data discussed or a variation or combination of the outputted data.

The output or outputs of unified reasoning module <NUM> serves as input to segmentation prediction module <NUM>. Segmentation prediction module <NUM> may process the unified model provided by unified reasoning module <NUM> to produce a set of predicted masks and/or shapes representing objects. The masks represent a proposed area encompassing an object. Since many scenes a computer vision system sees are inherently ambiguous, a model that outputs a single prediction won't be able to properly capture uncertainty. In a scene with many of the exact same item with complex patterns it may be difficult to find boundaries between items. However, an autoregressive model can properly model the distribution of segmentation possibilities. Using samples from the autoregressive model, segmentation possibilities can be efficiently computed, wherein each segmentation possibility represents one reasonable prediction of where distinct objects exist in images <NUM>.

Once a set of segmentation outcomes has been generated representing a distribution of segmentation possibilities, a variety of different methods may be employed to give a probabilistic meaning to segmented regions in the predictions, such as an associated certainty. In one example, areas or regions of the images may be used to find areas where a certain percentage of the segmentation predictions agree or overlap. For example, if it determined that the system should avoid trying to pick an object if it is less than <NUM>% confident in a region, then it may be a requirement that <NUM>% of the predictions share a segmentation region before trying to pick from that area. Thus, in the final step, a segmentation prediction is output based on the defined confidence tolerance for a given situation.

<FIG> shows segmentation hypothesis array <NUM> comprising nine segmentation hypotheses for the same scene. In some scenarios, there is some inherent ambiguity, and a computer vision system may not be able to determine a segmentation of an image with certainty. However, using trained DNNs, that same system may be useful for outputting a large number of hypotheses, wherein the distribution of outputs can show a likelihood or a certainty about instances in the hypotheses. Masks are used to show the predicted shapes of distinct objects in each hypothesis of <FIG>. The masks are regions of color outlined in white, although the colors filling the masks are solely for purposes of illustration and are arbitrary in regard to the present example. As can be seen in <FIG>, there are some regions and shapes that appear in many or most of the hypotheses. <FIG> also illustrates that certain shapes may be combined into larger shapes in some hypotheses.

One benefit of the present technology is that in scenarios similar to <FIG>, there may be areas where nearly all hypotheses agree, and the system can therefore be certain about those regions. However, in regions with less agreement, a large distribution of proposed segmentations can be leveraged to determine what is more certain and what is less certain. Based on how much the images of <FIG> agree with one another, the scene can be described using a metric that represents uncertainty. As previously discussed, this metric may be compared to a confidence requirement for a given scenario to make a decision or arrive at a final segmentation proposal. By changing how much agreement is required (i.e., by changing the confidence requirement), the system may produce a more conservative segmentation proposal or a less conservative segmentation proposal.

<FIG> shows segmentation hypothesis array <NUM> comprising nine segmentation hypotheses of the same scene. However, contrary to the scene of <FIG>, the scene in <FIG> has very little associated uncertainty. The masked items from the hypotheses of <FIG> are distinct shapes that barely touch one another, making it simple for the system to identify their shapes and draw masks with great certainty. The masks drawn in the nine segmentation hypotheses of <FIG> change very little between each proposed segmentation, indicating that there is much less uncertainty associated with what the system can see as compared to <FIG>. There is relatively little ambiguous nature about how to segment the scene in <FIG> and the predictions are nearly identical for that reason.

<FIG> illustrates that the present technology is useful for understanding both uncertainty and certainty. Using a distribution of possible outcomes does not tradeoff performance in scenarios that are very certain. In fact, the distribution of outcomes enables the system to understand when it is certain and provide an outcome or perform accordingly. While a unimodal prediction system may produce a segmentation hypothesis that is similar to one of the hypotheses in <FIG>, it would have no means for understanding how certain it is about its prediction, or how straightforward the scene is for segmentation purposes.

<FIG> show bin segmentation examples across several different scenarios. <FIG> shows an image of a bin comprising individual bags of product (image <NUM>) along with five segmentation hypotheses found in post-processing procedures below the image (image array <NUM>). The scenario of <FIG> is relatively difficult because the bags of product blend together and are difficult to tell apart. The boundaries between objects may be difficult for a computer vision system to decipher, but some regions of the image may be relatively straightforward and easy to segment. The segmentation hypotheses (image array <NUM>) shown are in order from less conservative (farthest left) to more conservative (farthest right). It can be seen in <FIG> that as the segmentation hypotheses become more conservative, more items are filtered out and fewer object regions are shown. In addition to showing fewer object masks as the hypotheses get more conservative (move to the right), the object masks shrink, thereby moving away from the boundaries between objects. This shrinking away from the boundaries is a result of being less confident as to where the item boundaries are.

<FIG> includes image <NUM> showing a bin comprising individual bags of product in a different arrangement from those in image <NUM>. <FIG> further includes image array <NUM> showing a series of segmentation hypotheses ranging from less conservative predictions (farthest left) with an increase in conservativeness in images to the right, the farthest right image being the most conservative prediction of image array <NUM>. <FIG> includes image <NUM> showing a bin comprising individual bags of product. <FIG> further includes image array <NUM> showing a series of segmentation hypotheses ranging from less conservative predictions (farthest left) with an increase in conservativeness in images to the right, the farthest right image being the most conservative prediction of image array <NUM>. <FIG> includes image <NUM> showing a bin comprising individual bags of product in a different arrangement from those in image <NUM>. <FIG> further includes image array <NUM> showing a series of segmentation hypotheses ranging from less conservative predictions (farthest left) with an increase in conservativeness in images to the right, the farthest right image being the most conservative prediction of image array <NUM>.

<FIG> shows another segmentation scenario for items in a bin. <FIG> illustrates an image of the contents of the bin (image <NUM>) and five segmentation hypotheses below the image (image array <NUM>). Similar to <FIG>, <FIG> shows an example of a scenario with little associated uncertainty. The bin of the present example comprises multiple different products instead of a several of the same product, such as in <FIG>. The five segmentation hypotheses (image array <NUM>) shown in <FIG> also move from less conservative hypotheses (far left) to the more conservative hypotheses (far right). However, unlike <FIG>, as the hypotheses become more conservative, very few items are filtered out, indicating that there is relatively high confidence associated with the object segmentation predictions in the scene. Due to the nature of the distribution of hypotheses generated when a plurality of guesses are made, the model is able to illustrate that there is little uncertainty associated with an image if the results are all similar to one another and if object regions overlap a large percentage of the time.

<FIG> show bin segmentation examples across different scenarios, with an additional comparison of single-cameras versus multiple-camera predictions. The present example illustrates how the confidence-based post-processing used in technology of the present disclosure can be useful for fusing information from multiple views. Four different camera viewpoints are used to take pictures in the present example and then segment each of the individual images. The segmentations are then combined to find objects that are the most agreed upon. In <FIG>, the top row of four segmentation hypotheses (e.g., image <NUM>, image <NUM>, image <NUM>, and image <NUM>) illustrates an image with overlying segmentation predictions from each of four cameras based on original image <NUM>. Image <NUM> shows a fused segmentation prediction generated from the combination of image from image <NUM>, image <NUM>, image <NUM>, and image <NUM> based on agreements or similarities between all of the views. In the present example, the segmentations predictions of image <NUM>, image <NUM>, image <NUM>, and image <NUM> are noticeably less accurate than the agreement between all views, illustrating the benefits of using multiple views and fusing information to produce confidence-based predictions.

In <FIG>, the top row of four segmentation hypotheses (e.g., image <NUM>, image <NUM>, image <NUM>, and image <NUM>) illustrates an image with overlying segmentation predictions from each of four cameras based on original image <NUM>. Image <NUM> shows a fused segmentation prediction generated from the combination of image from image <NUM>, image <NUM>, image <NUM>, and image <NUM> based on agreements or similarities between all of the views. In the present example, the segmentations predictions of image <NUM>, image <NUM>, image <NUM>, and image <NUM> are noticeably less accurate than the agreement between all views, further illustrating the benefits of using multiple views and fusing information to produce confidence-based predictions. In <FIG>, the top row of four segmentation hypotheses (e.g., image <NUM>, image <NUM>, image <NUM>, and image <NUM>) illustrates an image with overlying segmentation predictions from each of four cameras based on original image <NUM>. Image <NUM> shows a fused segmentation prediction generated from the combination of image from image <NUM>, image <NUM>, image <NUM>, and image <NUM> based on agreements or similarities between all of the views. In the present example, the segmentations predictions of image <NUM>, image <NUM>, image <NUM>, and image <NUM> are noticeably less accurate than the agreement between all views, further illustrating the benefits of using multiple views and fusing information to produce confidence-based predictions.

<FIG>, the top row of four segmentation hypotheses (e.g., image <NUM>, image <NUM>, image <NUM>, and image <NUM>) illustrates an image with overlying segmentation predictions from each of four cameras based on original image <NUM>. Image <NUM> shows a fused segmentation prediction generated from the combination of image from image <NUM>, image <NUM>, image <NUM>, and image <NUM> based on agreements or similarities between all of the views. In the present example, the segmentations predictions of image <NUM>, image <NUM>, image <NUM>, and image <NUM> are noticeably less accurate than the agreement between all views, further illustrating the benefits of using multiple views and fusing information to produce confidence-based predictions. In <FIG>, the top row of four segmentation hypotheses (e.g., image <NUM>, image <NUM>, image <NUM>, and image <NUM>) illustrates an image with overlying segmentation predictions from each of four cameras based on original image <NUM>. Image <NUM> shows a fused segmentation prediction generated from the combination of image from image <NUM>, image <NUM>, image <NUM>, and image <NUM> based on agreements or similarities between all of the views. In the present example, the segmentations predictions of image <NUM>, image <NUM>, image <NUM>, and image <NUM> are noticeably less accurate than the agreement between all views, further illustrating the benefits of using multiple views and fusing information to produce confidence-based predictions.

<FIG> illustrate the use of segmentation and uncertainty modeling technology disclosed herein for purposes of detecting double picks in a robotic picking environment such as in the example of <FIG>. The latent segmentation model described in reference to previous Figures can be used to count a number of objects picked or held by a robotic device to prevent accidental picking of multiple items. As previously discussed, the accidental picking of multiple items can lead to incorrect order fulfilment, dropping, equipment damage, product damage, and many other negative results. In <FIG>, image <NUM> shows an original unsegmented image while image array <NUM> includes <NUM> images illustrating a set of <NUM> hypotheses or samples generated by the latent model. Image <NUM> shows the unified segmentation prediction from a unimodal prediction model based on the predictions shown in image array <NUM>.

In <FIG>, image <NUM> shows an original unsegmented image while image array <NUM> includes <NUM> images illustrating a set of <NUM> hypotheses or samples generated by the latent model. Image <NUM> shows the unified segmentation prediction from a unimodal prediction model based on the predictions shown in image array <NUM>. In <FIG>, image <NUM> shows an original unsegmented image while image array <NUM> includes <NUM> images illustrating a set of <NUM> hypotheses or samples generated by the latent model. Image <NUM> shows the unified segmentation prediction from a unimodal prediction model based on the predictions shown in image array <NUM>.

The present examples illustrate one of the disadvantages of a single prediction model as compared to the latent distribution model of the present disclosure. It can be seen in <FIG>, for example, that there is some uncertainty regarding the number of items picked by the robotic arm. Some of the <NUM> hypotheses predict two items while others predict a single item. The unimodal prediction predicts only a single item. However, the distribution of hypotheses suggests that it is more likely that two items have been picked by the robotic arm because a majority of the hypotheses predict that there are two items. The unimodal hypothesis predicts that there is only one item and has no means for determining that it is wrong or finding an associated confidence with its prediction. In other words, the examples of <FIG> illustrates the risk that a unimodal prediction model can be "confidently wrong.

The processes described herein may be implemented in several different variations of media including software, hardware, firmware, and variations or combinations thereof. For example, methods of uncertainty modeling and segmentation described herein may be implemented in software, while a computing vision system or robotic picking device may be implemented entirely in hardware or a combination. Similarly, embodiments of the technology may be implemented with a trained neural net entirely in software on an external computing system or may be implemented as a combination of the two across one or more devices. The computer vision systems and uncertainty modeling herein may be implemented on various types of components including entirely software-based implementations, entirely hardware-based aspects, such as trained computer vision systems, or variations and combinations thereof.

<FIG> illustrates computing system <NUM> that is representative of any system or collection of systems in which the various processes, programs, services, and scenarios disclosed herein may be implemented. Examples of computing system <NUM> include, but are not limited to, desktop computers, laptop computers, server computers, routers, web servers, cloud computing platforms, and data center equipment, as well as any other type of physical or virtual server machine, physical or virtual router, container, and any variation or combination thereof.

Computing system <NUM> may be implemented as a single apparatus, system, or device or may be implemented in a distributed manner as multiple apparatuses, systems, or devices. Computing system <NUM> may include, but is not limited to, storage system <NUM>, software <NUM>, communication interface system <NUM>, processing system <NUM>, and user interface system <NUM>. Components of computing system <NUM> may be optional or excluded in certain implementations. Processing system <NUM> is operatively coupled with storage system <NUM>, communication interface system <NUM>, and user interface system <NUM>, in the present example.

Processing system <NUM> loads and executes software <NUM> from storage system <NUM>. Software <NUM> includes and implements various uncertainty modeling processes described herein, which is representative of the methods discussed with respect to the preceding Figures. When executed by processing system <NUM>, software <NUM> directs processing system <NUM> to operate for purposes of uncertainty modeling as described herein for at least the various processes, operational scenarios, and sequences discussed in the foregoing implementations. Computing system <NUM> may optionally include additional devices, features, or functionality not discussed for purposes of brevity.

Referring still to <FIG>, processing system <NUM> may comprise a microprocessor and other circuitry that retrieves and executes software <NUM> from storage system <NUM>. Processing system <NUM> may be implemented within a single processing device but may also be distributed across multiple processing devices or sub-systems that cooperate in executing program instructions. Examples of processing system <NUM> include general purpose central processing units, graphical processing units, application specific processors, and logic devices, as well as any other type of processing device, combinations, or variations thereof.

Storage system <NUM> may comprise any computer readable storage media readable by processing system <NUM> and capable of storing software <NUM>. Storage system <NUM> may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. Examples of storage media include random access memory, read only memory, magnetic disks, optical disks, optical media, flash memory, virtual memory and non-virtual memory, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other suitable storage media. In no case is the computer readable storage media a propagated signal.

Software <NUM> may be implemented in program instructions and among other functions may, when executed by processing system <NUM>, direct processing system <NUM> to operate as described with respect to the various operational scenarios, sequences, and processes illustrated herein. For example, software <NUM> may include program instructions for implementing uncertainty modeling processes, computer vision processes, neural networks, decision making processes, segmentation processes, or any other reasoning or operational processes as described herein.

In particular, the program instructions may include various components or modules that cooperate or otherwise interact to carry out the various processes and operational scenarios described herein. The various components or modules may be embodied in compiled or interpreted instructions, or in some other variation or combination of instructions. The various components or modules may be executed in a synchronous or asynchronous manner, serially or in parallel, in a single threaded environment or multi-threaded, or in accordance with any other suitable execution paradigm, variation, or combination thereof. Software <NUM> may include additional processes, programs, or components, such as operating system software, modeling, robotic control software, computer vision software, virtualization software, or other application software. Software <NUM> may also comprise firmware or some other form of machine-readable processing instructions executable by processing system <NUM>.

In general, software <NUM> may, when loaded into processing system <NUM> and executed, transform a suitable apparatus, system, or device (of which computing system <NUM> is representative) overall from a general-purpose computing system into a special-purpose computing system customized for one or more of the various operations or processes described herein. Indeed, encoding software <NUM> on storage system <NUM> may transform the physical structure of storage system <NUM>. The specific transformation of the physical structure may depend on various factors in different implementations of this description. Examples of such factors may include, but are not limited to, the technology used to implement the storage media of storage system <NUM> and whether the computer-storage media are characterized as primary or secondary storage, as well as other factors.

For example, if the computer readable storage media are implemented as semiconductor-based memory, software <NUM> may transform the physical state of the semiconductor memory when the program instructions are encoded therein, such as by transforming the state of transistors, capacitors, or other discrete circuit elements constituting the semiconductor memory. A similar transformation may occur with respect to magnetic or optical media. Other transformations of physical media are possible without departing from the scope of the present description, with the foregoing examples provided only to facilitate the present discussion.

Communication interface system <NUM> may include communication connections and devices that allow for communication with other computing systems (not shown) over communication networks or connections (not shown). Examples of connections and devices that together allow for inter-system communication may include network interface cards, antennas, power amplifiers, radio-frequency circuitry, transceivers, and other communication circuitry. The connections and devices may communicate over communication media to exchange communications with other computing systems or networks of systems, such as metal, glass, air, or any other suitable communication media. The aforementioned media, connections, and devices are well known and need not be discussed at length here.

Communication between computing system <NUM> and other computing systems (not shown), may occur over a communication network or networks and in accordance with various communication protocols, combinations of protocols, or variations thereof. Examples include intranets, internets, the Internet, local area networks, wide area networks, wireless networks, wired networks, virtual networks, software defined networks, data center buses and backplanes, or any other type of network, combination of network, or variation thereof. The aforementioned communication networks and protocols are well known and need not be discussed at length here.

As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method, or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit," "module," or "system.

Unless the context clearly requires otherwise, throughout the description and the claims, the words "comprise," "comprising," and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of "including, but not limited to. " As used herein, the terms "connected," "coupled," or any variant thereof means any connection or coupling, either direct or indirect, between two or more elements; the coupling or connection between the elements can be physical, logical, or a combination thereof. Additionally, the words "herein," "above," "below," and words of similar import, when used in this application, refer to this application as a whole and not to any particular portions of this application. Where the context permits, words in the above Detailed Description using the singular or plural number may also include the plural or singular number respectively. The word "or," in reference to a list of two or more items, covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list.

The phrases "in some embodiments," "according to some embodiments," "in the embodiments shown," "in other embodiments," and the like generally mean the particular feature, structure, or characteristic following the phrase is included in at least one implementation of the present technology, and may be included in more than one implementation. In addition, such phrases do not necessarily refer to the same embodiments or different embodiments.

Claim 1:
A computer-implemented method of operating an image segmentation system, the method comprising:
collecting at least one image of a scene, wherein the scene comprises a plurality of distinct objects;
generating a plurality of segmentation predictions for the scene, wherein each segmentation prediction of the plurality of segmentation predictions includes a plurality of object regions each representing a region predicted to correspond to a single object and a confidence associated with the prediction of each region; and
outputting a final segmentation prediction comprising at least one identified object region, wherein the at least one identified object region is chosen based on the confidence associated with the at least one identified object region and a confidence threshold,
wherein the confidence associated with the at least one identified object region is greater than the confidence threshold,
wherein outputting the final segmentation prediction comprises:
identifying the confidence threshold, wherein the confidence threshold corresponds to
a minimum percentage; and
wherein the at least one identified object region is a region where the minimum percentage of segmentation predictions from the plurality of segmentation predictions overlap.