Robot interaction with objects based on semantic information associated with embedding spaces

Techniques described herein relate to using reduced-dimensionality embeddings generated from robot sensor data to identify predetermined semantic labels that guide robot interaction with objects. In various implementations, sensor data obtained from one or more sensors of a robot includes data indicative of an object observed in an environment in which the robot operates. The sensor data is processed utilizing a first trained machine learning model to generate a first embedded feature vector that maps the data indicative of the object to an embedding space. Nearest neighbor(s) of the first embedded feature vector is identified in the embedding space. Semantic label(s) are identified based on the nearest neighbor(s). A given grasp option is selected from enumerated grasp options previously associated with the semantic label(s). The robot is operated to interact with the object based on the pose and using the given grasp option.

BACKGROUND

Robot grasping is a challenging aspect of robotics. On the one hand, it is desirable that robots be flexible enough to interact with a variety of different types of objects, such as door knobs, dishes, glasses, tools, etc. On the other hand, different types of objects require different types of grasps. Moreover, a state of an object, such as a pose of the object, a configuration of the object (e.g., does a cup have a lid?), can also influence how, or even whether, the object should be grasped by a robot. A variety of different techniques exist for determining how a robot should grasp objects. Many of these, however, are computationally complex, often because they evaluate large search spaces and/or use brute force approaches to determine how to grasp objects. For example, given the myriad different varieties of objects with which a robot may interact, the search space for determining an appropriate grasp may be large, requiring extensive computing resources (e.g., processing cycles, memory, etc.) and/or time to select an appropriate grasp.

SUMMARY

The present disclosure is generally directed to methods, apparatus, and computer-readable media (transitory and non-transitory) for using reduced-dimensionality embeddings generated from sensor data captured by a robot to identify predetermined semantic labels that guide robot interaction with objects. Put another way, techniques described herein leverage object embedding spaces that are, at least in part, indexed with semantic information that is usable to make decisions regarding how robots should interact with objects. These predetermined semantic labels may be used to select, for instance, a manner in which a robot will grasp an object. By leveraging semantic labels previously-assigned to object types that are commonly interacted with by robots, it is possible to “fast track” grasp selection.

In various implementations, vision frame(s) captured by one or more vision sensors of a vision component of a robot may take various forms. Some vision sensors are configured to capture RGB-D data. Additionally or alternatively, some vision sensors are configured to captured “point cloud” data indicative of a three-dimensional environment. Examples of vision sensors include stereographic cameras, RGB-D cameras, laser scanners, etc. Of particular relevance to the present disclosure, each vision frame may include one or more depictions of one or more objects captured in the vision frame.

In various implementations, the vision frames may be processed as input using one or more trained machine learning models (or “classifiers”) to generate embedded feature vectors, which are also referred to herein as “embeddings.” In various implementations, the embedded feature vectors may map depictions of objects contained in the vision frame(s) to an embedding space. Consequently, it may be possible to determine various attributes of depicted objects based on their embeddings, as well as their embeddings' proximities to other embeddings (i.e. nearest neighbors) in the embedding space.

Suppose a machine learning model is trained, e.g., using training data that includes vision frames labeled with object type classifications, to generate an embedded feature vector in an embedding space that is associated with object types. If an object depicted in the input vision frame is a chair, then the generated embedded feature vector may be embedded in relatively close proximity to (e.g., measured using Euclidean distance, dot products, cosine similarities, etc.) other embeddings associated with the object type of “chair.” By contrast, the generated embedded feature vector may be relatively distant in the embedding space from embeddings associated with other, non-chair-like objects. By determining the embedded feature vector's nearest neighbors, for example, it is possible to determine an object type classification of the depicted object.

In some implementations, various attributes (or classifications) of a depicted object may be determined in addition to or instead of the object's type. For example, different types of objects can have different types of “poses,” or physical arrangements relative to their environment. A cup can be standing upright, upside down, or tipped over. A coffee mug may have similar poses as a handle-less cup, as well as an orientation of its handle (which may be a potential grasping point for a robot). In various implementations, these additional attributes may also be determined using one or more machine learning models. For example, in some implementations, the same machine learning model used to generate the embedded feature vector may be trained to also generate output indicative of the depicted object's pose, e.g., as part of the same embedding.

For example, a given embedding generated from a tipped over cup may be proximate a cluster of other cup embeddings in an embedding space. Further, within that cluster of cup embeddings, the given embedding may be most proximate to cup embeddings generated from other tipped over cups. Consequently, by evaluating one or more nearest neighbors (or averages of x nearest neighbors, wherein x>0) to the given embedding in the embedding space, it is possible to determine both that the depicted object is a cup and that the depicted cup has a tipped over pose.

Additionally or alternatively, in other implementations, a determination made from generating an embedded feature vector may lead to selection of another machine learning model that can be used to determine various attributes of an object. For example, in some implementations, a first embedded feature vector generated based on a first machine learning model may be used to identify an object type classification of an object, e.g., based on semantic labels previously assigned to its nearest neighbors in object type classification embedding space. Based on the object type classification, one or more additional machine learning models may be selected and applied to determine additional attributes of the depicted object besides its object type classification.

Suppose the depicted object is an upright cup filled with liquid and no top. A first machine learning model may be used to generate a first embedded feature vector in an object type classification embedding space. This first embedding feature vector may be most proximate to other embeddings semantically labeled as cups, thereby enabling the object to be classified generally as a cup. Next, based on the classification of “cup,” one or more additional trained machine learning models that are applicable in the context of cups may be selected from a library of candidate trained machine learning models and used, with the first embedded feature vector or even with the original vision frame as input, to determine additional attributes.

For example, one selected machine learning model may generate output in the form of a second feature vector embedding within a second embedding space associated with whether objects are full or empty. Thus, in this example, the second embedding feature vector may be most proximate to other embeddings in the second embedding space that are associated with previously-assigned semantic labels of “full.” Another selected machine learning model may generate output in the form of a third feature vector embedding within a third feature embedding space associated with whether objects (e.g., cups, bottles, drawers, boxes, etc.) are open at the top. Thus, in this example, the third embedding feature vector may be most proximate to other embeddings in the third embedding space that are associated with previously-assigned semantic labels of “open.”

In various implementations, various embedding spaces may effectively be “indexed” (i.e., organized, classified) with semantic labels assigned to specific embeddings and/or relationships between multiple embeddings (e.g., an average of multiple embedding points in a cluster). By identifying nearest neighbor(s) to a given embedding, it is therefore possible to associate information from the previously-assigned semantic labels to the object used to generate the given embedding. Semantic labels assigned to embedding spaces (e.g., to specific embeddings, to embedding averages such as centroids, to embedding regions, etc.) may include other information besides potential object attributes. For example, in some implementations, semantic labels may include candidate grasp options that may be employed by a robot to interact with a given object.

Continuing with the example described previously, one or more nearest neighbors to the second embedded feature vector in the second embedding space—which was used to identify the cup as being full—may have previously-assigned semantic labels that provide a first set of one or more grasp options compatible with objects that are full. For example, it may be preferable to grab a full cup from the side, rather than from top by grabbing across a full diameter of the cup or by pinching a lip of the cup, which might result in the robot dropping the potentially unbalanced and/or heavy full cup. Likewise, one or more nearest neighbors to the third embedded feature vector in the third embedding space—which was used to identify the cup as lacking a top—may have previously-assigned semantic labels that provide a second set of one or more grasp options compatible with objects that are open at the top. For example, a cup that is open at the top may be graspable from the top by pinching its lip, but also may be graspable from the side or from the top by grabbing across the top diameter of the cup. In some implementations, an intersection of the first and second sets of grasp options may be used by a robot to select a suitable grasp option for grasping the full cup with an open top. Thus, in this example, a robot may grasp the cup from the side, as that is the only grasping option identified by the various semantic labels as suitable for grasping both open cups and full cups.

Techniques described herein may give rise to a variety of technical advantages. Indexing embedding spaces with semantic labels (e.g., assigning semantic labels to specific embeddings, combinations of embeddings, and/or regions in the embedding space) may facilitate reliance on the semantic labels to expedite robot decisions, such as selecting from one or more grasp options for an object of interest. Relying on previously-assigned semantic labels may be computationally inexpensive relative to other techniques for determining robot grasp for an object of interest. This may be particularly beneficial when interacting with objects that are common and/or interacted with by robots often, as the semantic labels may be effectively used as “shortcuts” in place of computationally-expensive grasp determination techniques. For example, using techniques described herein, robots may be able to make quick decisions about interacting with well-known objects in well-known poses/configurations because of the semantic labels previously assigned to such well-known objects.

In addition, in implementations in which a pipeline of machine learning models are employed to make a plurality of classifications about an object of interest—e.g., object type→object pose→object configuration), the potentially enormous search space is reduced based on upstream classifications. Robots that take advantage of techniques described herein (either themselves or via delegation to a cloud-based classification system) are therefore able to more quickly make decisions as to how to interact with (e.g., grasp) objects than robots that employ conventional techniques.

In various implementations, a method may include the following operations: obtaining, from one or more sensors of a robot, sensor data that includes data indicative of an object observed in an environment in which the robot operates; processing the sensor data as input utilizing a first trained machine learning model to generate a first embedded feature vector, wherein the first embedded feature vector maps the data indicative of the object to an embedding space; identifying, for the first embedded feature vector, at least one nearest neighbor in the embedding space; identifying one or more semantic labels based at least in part on the at least one nearest neighbor, wherein the one or more semantic labels are associated with a pose; selecting a given grasp option from one or more enumerated grasp options previously associated with the pose; and operating the robot to interact with the object based on the pose and using the given grasp option.

In various implementations, the method may further include determining an object class of the object based on the at least one nearest neighbor. In various implementations, In various implementations, the object class may include an object type. In various implementations, the object class may include the pose of the object. In various implementations, identifying the one or more semantic labels may include processing the first embedded feature vector as input utilizing a second trained machine learning model to generate a second embedded feature vector, and identifying the one or more semantic labels based on the second embedded feature vector. In various implementations, the second trained machine learning model may be selected from a plurality of candidate trained machine learning models based on the object class of the object. In various implementations, the plurality of candidate trained machine learning models may include at least some trained machine learning models that are trained to generate output indicative of respective predetermined poses of objects.

In various implementations, identifying the at least one nearest neighbor may include identifying an average of a plurality of nearest neighbors. In various implementations, the one or more sensors of the robot may include a vision sensor, and the sensor data may include one or more vision frames, and the data indicative of the object in the environment may include a depiction of the object in the one or more vision frames.

Other implementations may include a non-transitory computer readable storage medium storing instructions executable by a processor to perform a method such as one or more of the methods described above. Yet another implementation may include a control system including memory and one or more processors operable to execute instructions, stored in the memory, to implement one or more modules or engines that, alone or collectively, perform a method such as one or more of the methods described above.

DETAILED DESCRIPTION

FIG. 1depicts one example of a process that may occur in accordance with various implementations. A one or more inputs102may be captured by one or more sensors (not depicted inFIG. 1) that may or may not be integral with a robot (not depicted inFIG. 1). Examples of sensors, such as vision sensors, will be described below. Inputs102may be captured by the robot, for instance, while the robot maneuvers around an area such as a home or office. In some implementations, each captured input102may take the form of a vision frame captured by a vision sensor of a robot, though this is not required. InFIG. 1, input102includes a vision frame that includes a depiction103of a cup. In various implementations, all or a portion of input102(e.g., only the portion depicting the cup, a vector of features extracted from input102, etc.) may be processed, e.g., by an object identification engine104, as input utilizing an object identification machine learning model106to generate output.

Machine learning models such as106and others depicted inFIG. 1may come in various forms. In some implementations, these machine learning models may take the form of artificial neural networks, such as one or more convolutional neural networks. For example, object identification machine learning model106may comprise a neural network that is trained using training data that includes vision frames that are labeled with object types (e.g., cat, chair, car, cup, drawer, plate, etc.) depicted in the vision frames. The training data may be processed as input using object identification machine learning model106to generate output. The output may be compared to the labels associated with the training examples to determine differences/errors. Based on these differences/errors, various training techniques, such as back propagation and/or gradient descent (batch and/or stochastic) may be employed to alter hidden layer weights of the neural network.

In some implementations, once object identification machine learning model106is trained, it may be “cropped” so that instead of generating classification output (e.g., probabilities of a depicted object being one of x different object types), it generates an embedded feature vector, also referred to herein as an “embedding,” that maps a depiction of an object in an input vision frame to an embedding space, such as embedding space110.

Accordingly, in some implementations, the output of object identification engine104that is generated using object identification machine learning model106may take the form of an embedded feature vector108that comprises semantically-rich information about the depiction103of the object. For example, embedded feature vector108may map depiction103of the cup to embedding space110. Embedding space110is depicted in two dimensions inFIG. 1for the sake of brevity and clarity. However, this is not meant to be limiting. It should be understood that any embedding space described herein may have any number of dimensions.

In some implementations, embedding space110may be an object type classification embedding space in which embeddings are grouped into clusters of similar object types. InFIG. 1, for instance, three clusters,112A-C, are depicted. Each embedding in each cluster112are represented by a white circle. A black circle represents a location of embedded feature vector108in object type classification embedding space110. It can be seen that the nearest neighbors to embedded feature vector108—which may be determined using Euclidean distances, dot products, cosine similarities, etc.—are found in third cluster112C. Thus, embedded feature vector108is “closer to,” and hence is more similar to, embeddings in third cluster112C than it is to embeddings of clusters112A and112B.

As noted previously, in some implementations, various semantic information, e.g., contained in a semantic label index124, may be mapped to embedding spaces described herein. This mapped semantic information may be used to make various classification decisions about objects, as well as decisions on how to interact with objects. As an example, inFIG. 1, one semantic label may be assigned to first cluster112A (and/or to individual embeddings within first cluster112A). A second semantic label may be assigned to second cluster112B (and/or to individual embeddings within second cluster112B). A third semantic label, e.g., of “cup,” may be assigned to third cluster112C (and/or to individual embeddings within third cluster112C). Based on the third semantic label assigned to third cluster112C in which embedded feature vector108is grouped, in various implementations, it may be determined that object depiction103should be classified as a cup. While three clusters112are depicted inFIG. 1, this is not meant to be limiting. It should be understood that an object type classification embedding space (e.g.,110) may (and indeed is likely to) include any number of clusters, depending on how many objects have been embedded into the space.

In various implementations, an object attribute engine114may be configured to determine, based on one or more semantic labels mapped to object type classification embedding space110, an object type classification of the depiction103from input. In the example ofFIG. 1, object attribute engine114may identify the semantic label “cup” based on the proximity of embedded feature vector108to embeddings of third cluster112C in embedding space110. Once object attribute engine114has identified this classification, in various implementations, object attribute engine114may select one or more trained machine learning models118from a library116of training machine learning models1181-Nin order to determine other attributes of the depicted object.

For example, a first attribute machine learning model1181may be trained so that when utilized by object attribute engine114to process input (e.g., embedded feature vector108, or the original vision frame102), output is generated that is indicative of an object attribute. In some implementations, this output may take the form of one or more probabilities of the object having one or more respective attributes. Additionally or alternatively, in some implementations, the output may take the form of another embedded feature vector that maps to another embedding space (not depicted inFIG. 1) that is indexed or mapped with semantic labels. The same goes for the other machine learning models118of the library116, with each being trained to generate embeddings that map to other embedding spaces that are indexed/mapped with semantic labels related to other object attributes.

Continuing with the cup example, suppose first attribute machine learning model1181is trained to generate output, such as an embedded feature vector, that is indicative of an object having wings. Suppose second attribute machine learning model1182is trained to generate output that is indicative of an object being full or empty (e.g., with fluid, solids, etc.). And suppose third attribute machine learning model1183is trained to generate output that is indicative of an object having a top or no top (e.g., as would be applicable to cups, bottles, some bowls, cans, etc.). Since object attribute engine114was able to determine that the depiction103was of a cup, object attribute engine114may select only those downstream machine learning models118that are applicable to cups. Cups are not typically known for having or not having wings, so first attribute machine learning model1181is not applicable. However, cups are known for being full/empty and for having or not having tops/lids. Accordingly, second attribute machine learning model1182and third attribute machine learning model1183are applicable.

Object attribute engine114may then select and process input using the selected one or more machine learning models118to generate output, e.g., in the form of one or more additional embeddings1201-1202. First embedding1201may be an embedded feature vector that maps to an embedding space, not depicted inFIG. 1, that is indexed and/or mapped with semantic labels or other semantic information indicative of whether an object is full or empty. Second embedding may be an embedded feature vector that maps to another embedding, also not depicted inFIG. 1, that is indexed and/or mapped with semantic labels or other semantic information indicative of whether an object has or lacks a top/lid.

In some implementations, semantic information such as semantic labels used to map and/or index these additional embedding spaces may also include other information. In some such implementations, this other information may include information about how a robot should interact with objects having various attributes. For example, the embedding space of first embedding1201may include one cluster of embeddings associated with objects that are full, and another cluster of embeddings associated with objects that are empty. Other clusters may also be present, such as clusters of embeddings associated with objects that are half full, a quarter full, etc. If the cup depicted in the input vision frame (102) is full, then first embedding1201may be more proximate to other embeddings associated with objects that are full, and less proximate embeddings associated with objects that are empty. This by itself allows for a determination that the depicted cup from input102is full.

However, as noted above, the embedding space to which first embedding1201maps may also include other semantic information associated with, for instance, how a robot should interact with an object having particular attributes. For example, one or more embeddings of the cluster of embeddings associated with objects that are full may be labeled with semantic labels, e.g., contained in semantic label index124, that include enumerated lists of grasp options that may be employed by a robot when interacting with a full object. In various implementations, a semantic label engine122may identify semantic labels from semantic label index124that are associated with one or more nearest neighbor embeddings to first embedding1201. Based on these identified semantic labels, semantic label engine122may compile a list of candidate grasp options for the robot to employ to interact with a full cup. For example, the grasp options for a full object may include grasping from the side.

Similarly, one or more embeddings of a cluster of embeddings associated with objects that have open tops may be labeled with semantic labels, e.g., contained in semantic label index124, that include enumerated lists of grasp options that may be employed by a robot when interacting with an object that with an open/closed top. In various implementations, semantic label engine122may identify semantic labels from semantic label index124that are associated with one or more nearest neighbor embeddings to second embedding1202. Based on these identified semantic labels, semantic label engine122may compile a list of candidate grasp options for the robot to employ to interact with a cup having no top. For example, the grasp options for an object without a top may include grasping from the side, grasping from the top across the diameter of the object, and grasping an exposed lip at the object's top perimeter.

In some implementations, semantic label engine122may assemble an intersection of the sets of grasp options determined from semantic labels based on first embedding1201and second embedding1202. In some implementations, this intersection set may also include grasp options determined from semantic labels (e.g., stored in semantic label index124) associated with embeddings in the object type embedding space110. For example, cups may be graspable from the side, from the top (by grabbing across the diameter and/or grabbing a lip), from the bottom, etc. One or more grasp options contained in this intersection set may be provided, e.g., by semantic label engine122, as output, e.g., to a robot controller.

The robot controller may then select one or more of the grasp options to employ when interacting with the cup. In this example, the only grasp option contained in both sets determined from embeddings1201-2is the grasping from the side option, so that is the option the robot may employ. Intuitively, grasping a full cup with no top is most safely done from the side, because grasping a full cup from the top or bottom, especially from the lip, may lead to instability and increase the likelihood that the robot will drop the cup or at least spill some of its contents.

FIG. 2depicts an alternative implementation to that ofFIG. 1. Many aspects ofFIG. 2are the same asFIG. 1. For example, an input202takes the form of a vision frame captured by a robot vision sensor that once again includes, for purposes of explanation and not limitation, a depiction203of a cup. As was the case inFIG. 1, input202is processed as input by object identification engine104using object identification machine learning model106to generate output in the form of an embedded feature vector208.

However, inFIG. 2, the embedded feature vector208maps depiction203of the cup to an object identification embedding space210that is semantically richer than110ofFIG. 1. Embedded feature vector208is once again usable to determine an object type classification of the depicted object. However, in addition to an object type, embedded feature vector208also indicates other attributes of the object, such as its pose (e.g., standing upright, tipped over, upside down, etc.) and/or configuration (e.g., top, no top, filled, empty, etc.), all by virtue of its embedding into the semantically rich embedding space210.

For example, a first “supercluster”212in embedding space210may be associated with a first semantic label identifying a first object type classification (e.g., “cup”), which as noted above may be stored in semantic label index124. Similarly, a second supercluster214may be associated with a different semantic label identifying a second object type classification, such as “chair.” Only two object type superclusters are depicted inFIG. 2, but it should be understood that any number of object type superclusters may (and likely will) be present, depending on the number and variety of object depictions that have been embedded into embedding space210.

Within first supercluster212there are three sub clusters,212A-C. These sub clusters212may correspond to groups of cups that have similar attributes. For example, first sub cluster212A of supercluster212may include embeddings generated from depictions of cups that are standing upright. Second sub cluster212B of supercluster212may include embeddings generated from depictions of cups that are full. Third sub cluster212C of supercluster212may include embeddings generated from depictions of cups that are tipped over. And of course, there may be any number of other sub clusters within the supercluster212that represent other potential attributes (e.g., poses, configurations) of cups, such as having open tops, closed tops, handles, various shapes, etc.

Within second supercluster214there are also three sub clusters,214A-C. These sub clusters214may correspond to groups of chairs that have similar attributes. For example, first sub cluster214A of supercluster212may include embeddings generated from depictions of chairs that are standing upright. Second sub cluster214B of supercluster214may include embeddings generated from depictions of chairs with wheels on the bottom (this information may be used by a robot to influence how it might move a chair). Third sub cluster214C of supercluster214may include embeddings generated from depictions of chairs that are tipped over. And once again there may be any number of other sub clusters within the supercluster214that represent other potential attributes (e.g., poses, configurations) of chairs, such as having arm rests, being recline-able, having cushions, etc.

In various implementations, individual embeddings and/or regions of embedding space210may be associated with semantic labels, e.g., stored in semantic label index124. For example, first sub cluster212A of supercluster212(e.g., as a whole or by way of its individual embeddings) may be associated with one or more semantic labels that indicate a cup pose of “standing upright.” Additionally, in some implementations, the one or more semantic labels may include information that is usable by a robot to interact with standing-upright cups. For example, the semantic label(s) may include grasp options that are exercisable by a robot when interacting with an upright cup. Similar semantic labels may be associated with other sub clusters of super clusters212and214.

When the depiction203of input102is processed using object identification machine learning model106, the generated embedded feature vector208may map the depiction to embedding space210, which as noted above is more semantically rich than that ofFIG. 1. The single black dot in first sub cluster212A of supercluster212represents a coordinate of embedded feature vector208in embedding space210. Based on its location and its nearest neighbors, it can be determined, e.g., from the aforementioned semantic labels used to index embedding space210, that the depicted object is an empty cup that is standing upright. If the embedding were positioned relatively equidistant to both first sub cluster212A (standing upright) and second sub cluster212B (full), then semantic labels from both clusters may be applied to determine that the depicted cup is both standing upright and filled with something.

Unlike inFIG. 1in which a pipeline of machine learning models was effectively applied at least in part in series to determine attributes of the depiction103, inFIG. 2, a single machine learning model (i.e. object identification machine learning model106) was applied to generate an embedded feature vector208that maps to a semantically rich embedding space210. Semantic labels associated with embedding space210may then be used to determine that the depiction203is a cup that is standing upright (as opposed to, for instance, tipped over), and that the cup is not full. Grasp options that might be semantically associated with an upright cup include, for instance, grasping from the side, grasping from the top by pinching a lip of the cup, grasping from the top across a diameter of the cup, grasping from the bottom, etc.

Semantic label engine122may once again be configured to identify one or more semantic labels, e.g., from semantic label index124, that are associated with one or more nearest neighbors of embedded feature vector208within embedding space210. As noted above, the semantic labels may be associated with individual embeddings, with clusters of embeddings (e.g.,212A,212B,212C), with an average or other statistic computed from multiple embeddings (e.g., a centroid), and so forth. From these identified semantic labels, semantic label engine122may provide one or more grasp options as output, e.g., to a robot controller (not depicted).

In some implementations, new semantic labels may be added to semantic label index124, e.g., in association with new embeddings generated for embedding space210. For example, in some implementations, if a robot is able to successfully interact with an object based on grasp options determined from semantic labels associated with nearest neighbor embeddings, the current embedding may be labeled, e.g., in semantic label index124, with similar semantic labels. Additionally or alternatively, if the robot was not successful, in some implementations, one or more grasp options associated with one or more semantic labels may be eliminated, or may have their associated “confidence measures” reduced. These confidence measures may be used, for instance, if multiple grasp options are output by semantic label engine122, so that semantic label engine122and/or the robot is able to select from multiple candidate grasp options.

FIG. 3illustrates an example environment in which various techniques disclosed herein may be implemented. A robot300is illustrated inFIG. 3. Robot300may take various forms, including but not limited to a telepresence robot, a robot arm, a humanoid, an animal, an insect, an aquatic creature, a wheeled device, a submersible vehicle, an unmanned aerial vehicle (“UAV”), and so forth. In various implementations, robot300may include one or more processors302. Processor(s)302may take various forms, such as one or more CPUs, one or more GPUs, one or more field-programmable gate arrays (“FPGA”), and/or one or more application-specific integrated circuits (“ASIC”). In some implementations, the processor(s)302may be operably coupled with memory303. Memory303may take various forms, such as random access memory (“RAM”), dynamic RAM (“DRAM”), read-only memory (“ROM”), Magnetoresistive RAM (“MRAM”), resistive RAM (“RRAM”), NAND flash memory, and so forth.

In some implementations, processor(s)302may be operably coupled with one or more actuators3041-n, at least one end effector306, and/or one or more sensors3081-m, e.g., via one or more buses310. The robot300may have multiple degrees of freedom and each of the actuators3041-nmay control actuation of the robot300within one or more of the degrees of freedom responsive to control commands. The control commands are generated by one or more of the processor(s)302and provided to the actuators3041-n(e.g., via one or more of the buses310) to control the robot300. As used herein, “actuator” encompasses a mechanical or electrical device that creates motion (e.g., a motor), in addition to any driver(s) that may be associated with the actuator and that translate received control commands into one or more signals for driving the actuator. Accordingly, providing a control command to an actuator may comprise providing the control command to a driver that translates the control command into appropriate signals for driving an electrical or mechanical device to create desired motion.

As used herein, “end effector” may refer to a variety of tools that may be operated by robot300in order to accomplish various tasks. For example, some robots may be equipped with an end effector306that takes the form of a claw with two opposing “fingers” or “digits.” Such a claw is one type of “gripper” known as an “impactive” gripper. Other types of grippers may include but are not limited to “ingressive” (e.g., physically penetrating an object using pins, needles, etc.), “astrictive” (e.g., using suction or vacuum to pick up an object), or “contigutive” (e.g., using surface tension, freezing or adhesive to pick up object). More generally, other types of end effectors may include but are not limited to drills, brushes, force-torque sensors, cutting tools, deburring tools, welding torches, containers, trays, and so forth. In some implementations, end effector306may be removable, and various types of modular end effectors may be installed onto robot300, depending on the circumstances.

Sensors3081-mmay take various forms, including but not limited to vision components (e.g., laser scanners, stereographic cameras, monographic cameras), force sensors, pressure sensors, pressure wave sensors (e.g., microphones), proximity sensors (also referred to as “distance sensors”), torque sensors, barcode readers, radio frequency identification (“RFD”) readers, accelerometers, gyroscopes, compasses, position sensors (e.g., odometer, a global positioning system), speedometers, edge detectors, and so forth. While sensors3081-mare depicted as being integral with robot300, this is not meant to be limiting. In some implementations, sensors3081-mmay be located external to, but may be in direct or indirect communication with, robot300.

Also illustrated inFIG. 3, is a robot300A, which is one non-limiting example of robot300. The robot300A includes robot arm305A with a grasping end effector306A, that takes the form of a gripper with two opposing actuable members. The robot300A also includes a base309A with wheels307A1and307A2provided on opposite sides thereof for locomotion of the robot300A. The base309A may include, for example, one or more motors for driving corresponding wheels307A1and307A2to achieve a desired direction, velocity, and/or acceleration of movement for the robot300A.

The robot300A also includes a vision component308A. The vision component308A includes one or more vision sensors and may be, for example, a stereographic camera, a monographic camera, or a laser scanner. Vision data (e.g., vision frames) described herein can be generated based on output from vision sensor(s) of the vision component308A. For example, the output can be raw output from the vision sensor(s), or processed output. In some implementations, a stereographic camera includes two or more sensors (e.g., charge-coupled devices (CCDs)), each at a different vantage point. Vision data can be generated based on sensor data generated by the two sensors at a given instance, such as vision data that is a two-and-a-half-dimensional (“2.5D”) (2D with depth) image, where each of the pixels of the 2.5D image defines an X, Y, and Z coordinate of a surface of a corresponding object, and optionally color values (e.g., R, G, B values) and/or other parameters for that coordinate of the surface. In some other implementations, a stereographic camera may include only a single sensor and one or more mirrors utilized to effectively capture sensor data from two different vantage points. A monographic camera can include a single sensor and captures two-dimensional (“2D”) vision data. A laser scanner includes one or more lasers that emit light and one or more sensors that generate vision sensor data related to reflections of the emitted light. The vision data generated based on sensor output from a laser scanner may be 2.5D point cloud data. A laser scanner may be, for example, a time-of-flight laser scanner or a triangulation based laser scanner and may include a position sensitive detector (PSD) or other optical position sensor. In some implementations, vision data can be a voxel map as described herein. In some of those implementations, the voxel map is generated by processing of multiple instances of vision data. For example, multiple 2.5D images and/or multiple 2.5D instances of point cloud data from multiple different vantages can be processed to generate a voxel map of at least a portion of an environment of a robot.

As described herein, robot300A may operate autonomously at least part of the time and control actuators thereof in performance of various actions. For example, in performing various actions, one or more processors of the robot300A may provide control commands to actuators associated with the wheels307A1and/or307A2, the robot arm305A and/or the end effector306A. Further, in various situations the control commands provided at a given instance can be generated based at least in part on object detection as described herein.

Control system350is also illustrated inFIG. 3. Control system350includes the aforementioned object identification engine104, the aforementioned object attribute engine114, and the aforementioned semantic label engine122. Although the control system350is illustrated separate from the robot300inFIG. 3, connection arrow345indicates that the control system350can be implemented on robot300and/or can be in network communication (e.g., via a local area network and/or a wide area network) with robot300. For example, in some implementations, one or more (e.g., all) of the engines of the control system350are implemented by hardware that is local to the robot300, such as one or more of the processors302. Such processor(s) that implement one or more of the engines may optionally be separate from the processor(s) that generate and/or provide control commands to actuators304of the robot300. For example, aspects of the control system350may be implemented by one or more processors that do not operate in a real-time domain of the robot300, whereas other processor(s) that do operate in the real-time domain generate and provide control commands to actuators3041-n. In some implementations, one or more (e.g., all) of the engines of the control system350are implemented by hardware that is separate from the robot300. For example, engine(s) may be implemented “in the cloud” by a remote cluster of high performance computing devices and/or by one or more computing devices that are separate from the robot300, but that are geographically proximal to the robot (e.g., in the same building). In implementations where robot300and one or more aspects of control system350are separate, they may communicate over one or more wired or wireless networks (not depicted) or using other wireless technology, such as radio, Bluetooth, infrared, etc.

In various implementations, the object identification engine104receives input (e.g.,102), e.g., vision frames, that is generated based on output from, for instance, one or more vision components of the sensors3081-m. The object identification engine104uses the aforementioned trained object identification model106to process the received vision data to generate output that is indicative of an object type classification for one or more objects captured by the vision data. For example, the received vision data can be a 2.5D vision frame and object identification engine104can process the vision frame using trained object identification model106to generate output in the form of an embedded feature vector (e.g.,108,208) that maps the depicted object to an object type embedding space (e.g.,110,210).

Object attribute engine114may receive input, which may include the embedded feature vector generated by object identification engine104and/or “raw” sensor data, e.g., vision frames, that is generated based on output from one or more of the sensors3081-m. The object attribute engine114uses the object type classification determined from the output of object identification engine104to select one or more attribute machine learning models118. The selected attribute machine learning model(s) are then used to process the input to generate output that, for instance, includes embeddings (i.e., embedded feature vectors) that are usable to determine various attributes (e.g., pose, configuration) of the depicted object, as described above.

Also illustrated in the environment ofFIG. 3is a training engine365and training instances380. The trained object identification model106and/or attribute machine learning model(s)118can be trained by the training engine365utilizing training instances380. With reference toFIG. 4, additional description is provided of the training engine365, the training instances380, and the trained object identification model106. Similar techniques may be used to train attribute machine learning network(s)118.

FIG. 4illustrates an example training instance467of the training instances380, and how the training engine365can utilize the training instance467to train the object identification model106, which in this example may be a neural network such as a convolutional neural network. The training instance467includes a first training instance input482that includes a label identifying an object type and a second training instance input484that includes sensor data indicative of (e.g., depicting) the object. As noted previously, the sensor data484of training instance input467can be, for example, a 2D vision frame, a 2.5D vision frame, or a 3D voxel map. For example, the vision data can be a 2.5D vision frame captured by a stereo camera and have, for example, 312×312 pixels with four channels. The four channels can define red, green, blue, and depth values for each of the pixels. As another example, the vision data can be a 3D voxel map generated based on sensor output captured by a stereo camera and/or laser scanner from multiple vantages.

In training the object identification model106, the training engine365applies the training instance input,482and484, to object identification model106, and processes the training instance input467, utilizing object identification model106and based on current parameters of object identification model106, to generate an output. The training engine365then compares the generated output to the training instance label482, and updates one or more parameters of object identification model106based on the comparison. For example, the training engine365can generate an error based on differences between the generated output and the training instance label482, and backpropagate the error through object identification model106to update the model. Although only a single training instance467is illustrated inFIG. 4, object identification model106will likely be trained based on a large quantity of training instances. Those training instances can collectively include training instance input with diverse vision data (e.g., multiple diverse objects, varied lighting/other environmental parameters, etc.). Moreover, although a non-batch training example is described, batch training may additionally or alternatively be utilized.

FIG. 4also illustrates, within dashed lines, example layers that may be included in object identification model106. In particular, an input layer440A, convolution layer(s)440B, other layer(s)440C, and an output layer440D are illustrated. As described herein, the dimensions/shape of the input layer440A can be dependent on the shape of the vision data to be applied. In some implementations, multiple convolution layer(s)440B are provided, and max pooling and/or other layers may optionally be interposed between one or more of the convolution layer(s)440B. The other layer(s)440C can include such max pooling layers and/or layers such as a softmax layer(s) and/or fully connected layer(s) that precede the output layer440D.

FIG. 5depicts an example method500for practicing selected aspects of the present disclosure, in accordance with various implementations. For convenience, the operations of the flow chart are described with reference to a system that performs the operations. This system may include various components of various computer systems described herein. Moreover, while operations of method500are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added.

At block502, the system may obtain, e.g., from one or more sensors (e.g.,308) of a robot (e.g.,300), sensor data that includes data indicative of an object observed in an environment in which the robot operates. For example, in some implementations, the one or more sensors may include a vision sensor that generates various types of vision data/frames as described above, such as a 2.5D image, a point cloud, etc. In other implementations, the one or more sensors used to generate the data may be separate from the robot, e.g., as standalone sensors deployed in the environment, integral with a control system350, etc.

At block504, the system, e.g., by way of object identification engine104, may process one or more of the inputs utilizing a first trained machine learning model, e.g., object identification model106, to generate a first embedded feature vector (e.g.,108,208). In various implementation, the first embedded feature vector maps the data indicative of the object, such as depiction103/203, to an embedding space, such as embedding spaces110or210.

At block506, the system may identify, for the first embedded feature vector, at least one nearest neighbor (embedding) in the embedding space. For example, the system may identify the x nearest neighbors, wherein x is a positive integer. In some implementations, identification of the one or more nearest neighbors may yield a cluster of embeddings that are all semantically similar to the first embedded feature vector.

At block508, the system, e.g., by way of object attribute engine114and/or semantic label engine122, may identify, e.g., from semantic label index124, one or more semantic labels based at least in part on the at least one nearest neighbor. For example, in some implementations, the system may determine an object class of the object based on the at least one nearest neighbor. In various implementations, the object class may include, for instance, an object type (e.g., cup, chair, scissors, plate, bowl, vase, bottle, wine glass, vacuum cleaner, broom, etc.), a pose of the object (standing upright, tipped over/laying down, upside down), and/or a configuration of the object (e.g., top or no top, handle or no handle, etc.). Additionally or alternatively, in some implementations, one or more of the semantic labels may be associated with one or more enumerated grasp options that can be used by a robot to interact with an object having an attribute associated with the semantic label.

In some implementations, identifying the one or more semantic labels may further include processing the first embedded feature vector (or the original input vison frame) as input utilizing a second trained machine learning model (or any number of additional trained attribute machine learning models118) to generate a second (and/or additional) embedded feature vector. In various implementations, the one or more semantic labels may be identified based on the second embedded feature vector (and/or based on addition embedded feature vectors), as was described with respect toFIG. 1.

At block510, the system may select a given grasp option from one or more enumerated grasp options previously associated with one or more semantic labels identified at block508. In some implementations, the one or more grasp options may be associated with a pose conveyed by the one or more semantic labels. Examples of grasp options that are suitable for interaction with cups (e.g., full, empty, open top) were described previously. In some implementations, multiple sets of grasp options may be identified based on multiple semantic labels, and an intersection of the grasp options may be selected as usable by a robot to interact with the object. At block512, the system may operate the robot to interact with the object based at least on the given grasp option.

FIG. 6is a block diagram of an example computer system610. Computer system610typically includes at least one processor614which communicates with a number of peripheral devices via bus subsystem612. These peripheral devices may include a storage subsystem624, including, for example, a memory subsystem625and a file storage subsystem626, user interface output devices620, user interface input devices622, and a network interface subsystem616. The input and output devices allow user interaction with computer system610. Network interface subsystem616provides an interface to outside networks and is coupled to corresponding interface devices in other computer systems.

Storage subsystem624stores programming and data constructs that provide the functionality of some or all of the modules and/or engines described herein. For example, the storage subsystem624may include the logic to perform selected aspects of method500, and/or to implement one or more components depicted in the various figures. Memory625used in the storage subsystem624can include a number of memories including a main random access memory (RAM)630for storage of instructions and data during program execution and a read only memory (ROM)632in which fixed instructions are stored. A file storage subsystem626can provide persistent storage for program and data files, and may include a hard disk drive, a CD-ROM drive, an optical drive, or removable media cartridges. Modules implementing the functionality of certain implementations may be stored by file storage subsystem626in the storage subsystem624, or in other machines accessible by the processor(s)614.

Bus subsystem612provides a mechanism for letting the various components and subsystems of computer system610communicate with each other as intended. Although bus subsystem612is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

Computer system610can be of varying types including a workstation, server, computing cluster, blade server, server farm, smart phone, smart watch, smart glasses, set top box, tablet computer, laptop, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computer system610depicted inFIG. 6is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computer system610are possible having more or fewer components than the computer system depicted inFIG. 6.