Systems, apparatuses and methods may provide for controlling one or more end effectors by generating a semantic labelled image based on image data, wherein the semantic labelled image is to identify a shape of an object and a semantic label of the object, associating a first set of actions with the object, and generating a plan based on an intersection of the first set of actions and a second set of actions to satisfy a command from a user through actuation of one or more end effectors, wherein the second set of actions are to be associated with the command.

TECHNICAL FIELD

Embodiments generally relate to end effectors. More particularly, embodiments relate to control of end effectors of robots in dynamic environments.

BACKGROUND

Robots may be able to autonomously execute tasks to complete certain goals. For example, a human may instruct a robot to execute a task, and the robot may then execute the task without supervision. As the tasks rise in complexity and environments increase in variability, the robots may have increasing difficulties in executing the tasks with reliability and dependability. Moreover, dynamic environments may provide a difficult analysis for a robot. For example, objects may move and have irregular shapes making interaction difficult.

DESCRIPTION OF EMBODIMENTS

Turning now toFIG.1, an enhanced environmental analysis and robotic end effector control process100is illustrated. A robot may include sensor array102, map and semantics generator104, robot modeler106, mission planner108, end effector controller128and robotic end effector132(e.g., multi-fingered robot end-effectors). The process100may be an integral perception-planning solution to enable the robot to grasp free-form objects in a reliable, flexible, efficient and affordance-compelling manner (e.g., based on specific characteristics of an object having a variable size, objects having different material properties, objects have different surface characteristics, etc.). Affordance-compelling may include generating robot grasps configurations with 6D poses which are coherent to the intended flow of actions and partially imitate implicit social and cultural behaviors during task execution. The robot may operate in semi-structured human-centric-spaces (e.g., household, health, retail, etc.) to accomplish tasks. For example, the robot may grasp elements that are free-forming in human-centric spaces. In detail, the mission planner108may efficiently, rapidly and adaptively determine suitable grasping configurations for a given robot manipulator to grasp (e.g., physically manipulate) free-form objects. The free-form objects may be previously unseen so that the robot has no previous interaction with the object (e.g., the object is “new” to the robot).

The sensor array102may include imaging sensors (e.g., a 2D camera, a 3D depth camera and 6D inertial measurement unit), auditory sensors, range sensors, location sensors and so forth. The sensor array102may provide data to the map and semantics generator104, robot modeler106and mission planner108. For example, the sensor array102may provide image data (e.g., a red, green, blue and depth (RGB-D) image data, 3D camera orientation, 3D point-cloud, etc.) and/or range data110to the map and semantics generator104.

The map and semantics generator104may generate one or more maps based on the image and/or range data. For example, the map and semantics generator104may generate an occupancy map to represent an environment of the robot such as an occupancy map (continuous or discrete) that maps occupied spaces. In some embodiments, the map and semantics generator104may further map unoccupied spaces and/or unknown spaces (spaces that cannot be identified as occupied or unoccupied) and store the unoccupied spaces and/or unknown spaces in the occupied map or another map.

The map and semantics generator104may further generate a surface map that identifies surfaces based on the sensor data and the occupancy map (e.g., classify the occupied spaces into various surfaces). For example, the surface map may be a structured point-cloud that includes a collection of 3D vertex points linked by edges on the surfaces.

The map and semantics generator104may further generate a semantic labelled map (e.g., connect labels to surfaces in the surface map) based on the surface map and the sensor data. For example, the map and semantics generator104may include a deep neural network that identifies each object in the surface map, identifies boundaries of the object, applies a label (e.g., cup, cube, bottle, table, etc.) to the object (e.g., surface segments) and assigns a unique value (e.g., an instance identifier) to the object for future reference.

The map and semantics generator104may further generate a part labelled semantic map (e.g., generation of semantic endowed surface regions which may be referred to as semantic patches or surface patches) based on the semantic labelled map and the sensor data. For example, the part labelled semantic map may identify the parts of each objects. As a more detailed example, if a motorcycle is identified, the parts may include a handle, frame, seat, tank and wheel. Each of the parts may be labelled in the part labelled semantic map.

In some embodiments, the map and semantics generator104may omit portions of the object from further analysis if the portions are smaller than a predetermined size (e.g., smaller than a contact area of the end effector). In some embodiments, may decrease the resolution of surface patches for more efficient storage and access of corresponding identification data (e.g., corresponding image data may be stored as an octree) and further associate actions (e.g., a physical manipulation of the object) that may be taken with each identified object (e.g., move cup, refill cup, clean cup, etc.) and based on the labelled parts. Thus, the map and semantics generator104may link verbs (e.g., actions) and noun names in the segmented surfaces and/or parts

The robot modeler106may receive location data112from the sensor array102. In some embodiments however, the robot modeler106may execute the functions below without the location data112.

The robot modeler106may identify a current location of the robot based on the location data112for example. The robot modeler106may generate a model of the static and dynamic geometry (e.g., kinematics) of a robot to enable planning of motions by the mission planner108. For example, the robot modeler106may define robot actuators as a set of link bodies (e.g., CAD models) and joints (e.g., axes and joint range limits). The robot modeler106may further generate a graspability map. The graspability map may be a discretization of a workspace where the robot may apply contacts with a minimal nominal force of the robotic end effector132. The robot modeler106may further penalize grasps according to a force and kinematic feasibility, and quickly reject unsuitable grasps. The robot modeler106may further rank actions according to feasibility and force to identify actions that have the highest probability of success and based on particular metrics (e.g., actions that have a highest probability of success given a particular object in a map and/or image and a particular available space around the object).

In some embodiments, the location data112may provide sensor information which is used by the robot modeler106in conjunction with the direct and inverse kinematics to precompute a 3d body-relative reachability map. By using that reachability map and the current state of the scene, the robot modeler106may identify a possible set of actions (e.g., actions to physically manipulate the object). In some embodiments the map and semantics generator104may provide the scene to the robot modeler106.

In some embodiments, robot modeler106may receive the one or more maps of map and semantics generator104and determine suitable grips for various objects based on the graspability map. For example, the robot modeler106may identify that certain grips would be ineffective (e.g., would be unable to manipulate an object, would not be able to hold a slipper object, etc.) for the objects and thus exclude such grips from being propagated to the mission planner108as ranked action.

The map and semantics generator104may provide the map and semantic data118to the mission planner108and the robot modeler106may also provide the ranked actions120to the mission planner108. The mission planner108may receive command data114from the sensor array102. The command data may take the form of various sensor data such as an audio data, imaging data, etc. In some embodiments, the command data may be provided through a graphical user interface or other device. In some embodiments, the command data may be received through a distributed system (e.g., a first device sends the command to a second device directly or through the cloud).

The mission planner108may identify a task from a high level directive (e.g., clean the kitchen). For example, the high level directive may be decomposed into a sequence of granular atomic-actions which may be referred to as macro-plans. The macro-plans may not only provide actions (e.g., physical actions to physically manipulate the object) that may be undertaken to complete the task, but further provide operational limits corresponding to the specific domain and task.

For example, the mission planner108may set a maximal speed of the robotic end effector132along a manipulation trajectory (e.g., for social space sharing) or maintain containers with orientation limits to avoid failure of the mission (e.g., spilling liquids in a container). The case-by-case operational limits may change in each step of the plan. Thus, identifying the operation limits may filter both affordances by attribute and prioritize affordances by range matching.

In some embodiments, the mission planner108may receive the one or more maps of the map and semantics generator104and/or an identification of ranked actions from the robot modeler106. The mission planner108may determine a resulting symbolic plan with attributes and active subsets of actions based on the one or more maps of the map and semantics generator104. The active subset of actions may be a resulting intersection of actions (e.g., verbs linked by noun names in the segmented parts) from the one or more maps from the map and semantics generator104, actions from the ranked actions from the robot modeler106to implement the plan and actions (e.g., granular atomic-actions) identified from the command data114.

The mission planner108may provide the decomposed commands and plans126to the end effector controller128(e.g., a processor on the end effector that controls actions). Additionally, the robot modeler106may provide the ranked action to the end effector controller128, and the map and semantics generator104may provide the map and semantic data to the end effector controller128. The end effector controller128controls the robotic end effector130,132to implement the decomposed commands and plans that include actions that are identified by the mission planner108(e.g., intersections of actions). The sensor array102may further provide sensor data134to the end effector controller128so the end effector controller128may control the end effector130based on updated sensor data (e.g., positional data).

Some embodiments may be dependent with respect to learned kinematic-specific grasping functions. Some embodiments may include a multiresolution foundation that allows the integration of semantic and language knowledge-cues while extracting appropriate grasping configurations based on i) spatial-context (environment occupancy and reconstructed object surfaces), ii) part-wise semantic instance segmentation and iii) kinematic description of the end-effectors. Inputs may be merged via multiple processes orchestrated for reconstruction, filtering and inference jointly driven by high-level task-specific directives.

In some embodiments, the process100may implement a spatial-programming paradigm delivering enhanced parallel computational capabilities through hardware accelerators such as FPGAs. For example, any of the elements, such as map and semantics generator104, robot modeler106, mission planner108, sensor array102and end effector controller128may be constituted in hardware accelerators. In addition, the low-energy consumption required by such computational devices may enhance power efficiency which may be beneficial when deploying battery-powered mobile robots. Together, the enhanced any-time multiresolution analysis, language-based affordances and the advantageous use of low-power parallel devices may result in a dependable component for sensor-driven grasp planning. The above process100may empower autonomous service robots to perform real-world physical-interaction tasks generating and capturing value in semi-structured environments.

FIG.2shows a method800of controlling an end effector. The method800may generally be implemented in a robotic process such as, for example, the process100(FIG.1), already discussed. In an embodiment, the method800is implemented in one or more modules as a set of logic instructions stored in a machine- or computer-readable storage medium such as random access memory (RAM), read only memory (ROM), programmable ROM (PROM), firmware, flash memory, etc., in configurable logic such as, for example, programmable logic arrays (PLAs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), in fixed-functionality logic hardware using circuit technology such as, for example, application specific integrated circuit (ASIC), complementary metal oxide semiconductor (CMOS) or transistor-transistor logic (TTL) technology, or any combination thereof.

Illustrated processing block802generates a semantic labelled image based on image data, where the semantic labelled image is to identify a shape of an object and a semantic label of the object. Illustrated processing block804associates a first set of actions with the object. Illustrated processing block806generates a plan based on an intersection of the first set of actions and a second set of actions to satisfy a command from a user through actuation of one or more end effectors, where the second set of actions are to be associated with the command.

In some embodiments, the method800may include applying a first label to a first portion of the object, and applying a second label to a second portion of the object, wherein the second label is to be different from the first label. In some embodiments, the method800may further include generating a surface patch from the semantic labelled image that is to represent the object, decreasing a resolution of the surface patch, and generating the plan based on the surface patch having the decreased resolution.

In some embodiments, the method800may further include identifying a contact force map that represents contact force outputs mapped to portions of the one or more end effectors, and generating the plan based on the contact force map. In some embodiments, the method800may further include generating an occupancy map based on the image data, wherein the occupancy map is to identify portions of the image that are to be occupied, and generating a surface map based on the occupancy map that is to identify surfaces of objects. In some embodiments, the method800may further include connecting the surfaces in the surface map to labels to generate the semantic labelled image.

The method800may enable a robot to grasp free-form objects in a reliable, flexible, efficient and affordance-compelling manner (e.g., based on specific characteristics of an object having a variable size) so that the robot may operate in dynamic and quickly changing environments. The method800may enable robots to operate in semi-structured human-centric-spaces (e.g., household, health, retail, etc.) to accomplish tasks. In detail, the method800may efficiently, rapidly and adaptively determine suitable grasping configurations for a given robot manipulator to grasp free-form objects. The free-form objects may be previously unseen so that the robot has no previous interaction with the object (e.g., the object is “new” to the robot).

FIGS.3A and3Billustrates a scene segmentation process300. The process300may include a scene semantic spatial context generator304. The scene semantic spatial context generator304may be readily substituted for the map and semantics generator104ofFIG.1.

Imaging and/or range sensors302may provide sensor data336to the scene semantic spatial context generator304. The sensor data may include imaging data (e.g., RGB-D data) and/or range data. Imaging sensors of the imaging and/or range sensors302may be devices contained within a composed sensor (e.g., RGB-D camera or camera module). For example, the imaging and/or range sensors302may provide three data streams capturing information regarding a content in a field-of view and the time-varying 6D pose of one or more objects.

For example, the field of view may be captured in terms of structure through a discrete depth image or another image from which depth may be derived (e.g., 3D projection via intrinsic sensor calibration) in the form of a structured point-cloud. The image may be infrared enhanced (e.g., associated infrared, reflectance intensity or absorption values) graph-like set of 3D points (e.g., a noise-prone front-wave composition of the scene's surfaces).

The imaging and/or range sensors302may also provide a digital RGB-color image of the scene with a high resolution. This passive camera captures the chromatic appearance of the objects with fine detail. Such images may lack explicit depth information. Thus, based on extrinsic calibration between the depth and color cameras, the scene semantic spatial context generator304may map 3D points from the structured point-cloud to the RGB image plane in order to associate color values per point. Some embodiments may also associate depth to most of the pixels of the digital image. In some embodiments, imaging and/or range sensors302may include LIDARs or other 3D range sensors.

The imaging and/or range sensors302may also include a built-in multiple-axis inertial measurement unit (IMU), which may provide IMU measurements in the sensor data336. Thus, the scene semantic spatial context generator304may obtain a fused state assertion from measured acceleration, linear and angular velocities of the imaging and/or range sensors302(e.g., a camera module).

Therefore, the scene semantic spatial context generator304may generate a 6D kinematic frame based on the fused assertion. The 6D kinematic frame may be 6D because of 3 degrees of freedom for position and 3 degrees of freedom for orientation. The set of 6 degrees of freedom may unambiguously define a pose in space. In some embodiments, the motion may further be defined with respect to speed in each dimension namely Vx, Vy, Vzas well as Vroll, Vpitchand VYaw, which will may correspond to another set of 6 degrees of freedom. The degrees of freedom of the robotic end effector132may not be limited by 6 degrees of freedom, but may depend on a robot structure associated with the robotic end effector132, motors and joint types. In some embodiments, a robot may need to have at least 6 degrees of freedom to grasp objects in a general position

The 6D kinematic frame may include the added features such as gravitational orientation. In doing so, it may be possible to discern the relative direction of the ground-floor based on the constant gravitational acceleration, and may be used to provide context to surfaces and object dynamics. This collection of signals may be exteroceptive sensor input which is processed by geometric, semantic and fusion processes as follows as described below.

The scene semantic spatial context generator304may include a free and occupied map generator316that generates free and occupied space maps306,308,324. The free and occupied map generator316may generate spatial maps and surface reconstructions on the spatial maps.

The free and occupied map generator316may generate sparse dual-space map that may capture and split the occupied and unfilled (free) spaces. This mapping may allow for: i) registering diverse 3D images while exploring various interaction (e.g., grasping) scenarios for a kinematic end effector, ii) determine possible collision-free manipulator 6D poses in the environment and iii) serve as an effective scaffolding data structure to store multiresolution local surface descriptors such as volumetric (e.g., with respect to voxels) semantic labels and other attributes.

The sparse dual-space maps be high-resolution octrees of the workspace that may further allow for explicit separation of graspable spaces from non-graspable spaces, and in particular categorize spaces into three categories: 1) unseen spaces, 2) occupied spaces and 3) free spaces. The octree may be considered fully observed (e.g., consequently outer-voxels can be labeled as occupied) or empty spaces (e.g., larger inner-voxels labeled as empty) respectively. An initial map, which may be generated based on the sensor data, may be formally expressed as a continuous occupancy mapping function F with logarithmic evaluation complexity as
Γ(x∈3)→{1,0}.   Equation I
Moreover, for each point xi∈3the free and occupied map generator316may determine a corresponding voxel Θicontainer of the point and an implicit graph-path containing the subspace and/or subgraph via a discretization function. The discretization function may be a voxelization process that may be executed while generating an octree. The voxelization process may include sequential insertion into a spatial partitioning data structure. The result may be a regular tree with a valence (e.g., degree 8) that corresponds to the subdivision of space octants in 3D space. For example, the following equation 2 may express β which may be an octree:
β(x∈3)→[Θi,Θi-1,Θi-2, . . . ,Θ0]   Equation 2
In Equation 2, the root voxel at Θ0may represent an entire captured scene subspace. Both the outer and inner nodes of the octree may contain diverse attributes depending on the implementation. The attributes may be expressed by “access functions” as indicated below in Equation 3:
λ(x∈3)→Ω.   Equation 2
Equation 3 may be a value-property mapping over a tailored attribute set Ω. The attribute set and/or Equation 3 may be utilized during fusion and scaffolding roles of the free and occupied space maps306,308.

The free and occupied map generator316may extract a single-category version of the initial map to generate free and occupied space maps306,308. The occupied space map308may be the set of points fulfilling the constraint Γ(x)=1 as
MP:={x∈3|Γ(x)=1}   Equation 4
Thus, based on Equation 4, the union of occupied voxels may lead to the occupied space map308which may be determined based on the following equation 5:

Further, the free and occupied map generator316may invert the occupied space map308or modify one or more of equations 1-5 (e.g., modify occupied Γ(xi)=1 to empty Γ(xi)=0) to generate the free space map306. For example, Equations 3 and/or Equation 5 may be modified in order to define a point-wise empty space map Npor a discrete version in an empty voxel map by Equation 4 producing an Nvmap that is stored as the free space map306. Thus, the free space map306may be an inverted octree scene relative to the occupied space map308.

The free and occupied map generator316may provide the free and occupied maps306,308,326to the surface reconstruction and representation generator318. In some embodiments, only the occupied space map308is provided to the surface reconstruction and representation generator318, and/or may be filtered based on criteria. The surface reconstruction and representation generator318may further identify image data (e.g., RGB data) from the sensor data336.

A structured point-cloud (which may be the same as the point-cloud described above, and may be based on a 2D RGB image) may include of a collection of 3D vertex points. The 3D vertex points may be provided by:
P:={xi∈3}   Equation 6
The vertex points may be linked by edges in an 8-connectivy pixel-like function defined by the following:
L(xi,xj){1,0}   Equation 7
In enhancement to generating a graph structure on the point-cloud may be the ability to extract an implicit set of simplexes defining surface primitives (e.g., 3D triangles).

In some embodiments, due to sensing-limitations, depth images might omit values which may produce undesirable surface holes on the graph structure (e.g., graph structure has a hole over a surface of an object rather than a vertex). Some embodiments may further include additional metrics when extracting the underlying primitives. For example, an edge Ei,jmay exist if Equation 7 above meets L(xi,xj)=1 with a length |Ei,j|+<dmwhich is upper limited as a function of the depth from the camera to the vertices dm˜ε·max(xi,xj). This connectivity criteria may be illustrated in various ways, such as valid edges with a first characteristic (e.g., yellow colored) or invalid edges with a second characteristic (e.g., red) in the structured point-cloud. For each vertex in P incident to at least one primitive Ti,j,k, the surface reconstruction and representation generator318may determine the associated normal vector according to the following:
N(xi){ni∈3,|ni|=1}   Equation 8
Equation 8 may be based on the normalized aggregation of connected triangle normals. Equation 8 may be a way to obtain a normal vector for each point over the surface. Doing so may aid in planning the contact points and margin distances along the approximation of the robotic end effector132. For example, equation 8 may express that the T{i,j,k} references a triangle passing through surface points i, j and k, thus for any point it is possible to determine the normal by averaging the Normals of the incident triangles.

In some embodiments, a surface map310, which is represented by ϕ in the equation below, of the scene is the union of all primitives in the scene:
ϕ:={Ui,j,kTi,j,k|i,j,k∈P∧i≠j,i≠j,k≠k∧L(xi,xi)=L(xj,xk)=L(xk,xi)=1}.   Equation 9
Further, a connected subset of triangles may define a surface component ωr⊂ϕ whose extraction simultaneously links each triangle to its containing voxel within the occupied discrete map Mvfrom Equation 5. A subjective triangle-to-voxel attribute function may also allow for subsequent fusion and semantics transfer processes. The triangle-to-voxel attribute function may be provided below:
λ(Ti,j,k∈ϕ)→β(xi)∪β(xj)∪β(xk)   Equation 10
Additionally, the surface map310may include attributes from the subjective triangle-to-voxel attribute functions. The surface reconstruction and representation generator318may thus generate the surface map310,328and store the surface map310.

The surface reconstruction and representation generator318may provide the surface map310,338to the semantic object segmenter320. The semantic object segmenter320may identify objects, segment the objects and label the objects as explained below.

The semantic object segmenter320may identify an RGB-image I(α∈2)3and from the sensor data336. The semantic object segmenter320may exploit a pixel-wise semantic instance segmentation via depth neural networks DNN (e.g., expressed as single function κ), to generate an instance-identified semantic labelled image Ψ(α∈2)Σ.

In the semantic labelled image, each pixel may contain: i) the probability distribution over the setwith ||=n predefined prototypical classes and ii) the corresponding instance identifier (denoted as “id” that may be a non-zero unique integer) where the network distinctively associated pixels-to-instances. For example, a dataset of labels of the semantically labelled image may include the following labelled dataset Σ:=[cup,ε0,h1], [bottle,ε1, hj],[plate,ε2,hk], [table,ε3,hl], . . . }. In some embodiments, a special “unknown” class for all unclassified pixels as [unknown, εn, 0] may also be included. The dataset may further subject to the following equation:
Σinεi=1 and 0≤εi≤1   Equation 11
In the dataset, hirepresents numeric identifiers associating each pixel to an object instance or “0” for those in the unknown class. The overall extraction process over all pixel locations a may be provided by equation 12:
κ(I(α∈2))Σα:=[cup,ε0,hi],[bottle,ε1,hj],[plate,ε2,hk],[table,ε3,hl], . . . }Equation 12

The semantic object segmenter320may also integrate the derived semantic information, labels, confidence and instance ids through a consensus in the 3D occupancy while connecting labels to surface segments as follows. For example, while simultaneously exploiting the mutual extrinsic calibrations between imaging and/or range sensors302(e.g., the depth camera and the color camera) the semantic object segmenter320may project spatial points into a camera image plane (e.g., RGB camera plane). Based on this mapping, the 2D semantic labels may transferred to each vertex point of the surface. The semantic surface map with the labels transferred to each vertex point may be stored as semantic surface map332.

In some embodiments, the DNN may generate the results described in equation 12. In order to mitigate errors or discrepancies at borders of the objects represented in the images, each point may be a connected to surface component, as determined from Equation 9, and provided below:
ωr⊂ϕ   Equation 13
A split-and-merge process may conducted via conditioned region growing using the object ids (e.g., hiin Equation 12) only for high confidence classification probabilities εi. In doing so, the 3D surface connected-components may be transformed (e.g., one or more of erodes and splits) into semantic labelled surface regions or semantic patches for short ωrs∈ωr.

In some embodiments, extracted semantic patches (e.g., a surface labelled as a one semantic label such as back of a chair, armrest of a chair, seat of a chair, leg of a chair, etc.) of an object that each have a size smaller than a contact surface of a robot manipulator may be excluded from further consideration or interaction with the robot manipulator. Further, such excluded semantic patches may be excluded from subsequent phases without concerns about collisions during motion due to the up-to-date occupancy maps. At this point, the surface segments ωrsmay describe the content of the semantic surface map312.

The semantic object segmenter320may further conduct a part classification on each semantic patch associated to an object instance. For example, the semantic object segmenter320may conduct a part classification r refinement process, where qQdescribes the set of possible part names induced over the object category. The part classification may stored in the semantic surface map312. The semantic object segmenter320may provide the semantic surface map312,340to a perceptual part generator322that generates perceptual grasping components314,334.

The perceptual part generator322may further identify 2D RGB image data from the sensor data336. The perceptual part generator322may generate perceptual grasping components (“PGC”). Each PGC may be composed as 4-tuples with i) a semantic surface patch ωrs, ii) the class of object to which the patch belongs q∈, iii) the part-name associated with the patch r∈and iv) a set of verbs that may be defined by the following equation:
FPercetion(r):={(fi∈Å,0<σi≤1∈)}   Equation 14
The associated likelihood σ1describing the possible actions applied to such object-part may be provided by the following equation:
τ:=[ωrs,q∈,r∈q,F(r)]   Equation 15
The associative set of actions “A” may be extracted from the object-to-action co-occurrences from a frequency analysis of a large text corpus describing application procedures structurally organized by domain ontologies. Equation 15 may define the 4-tuples of the PGC.

Moreover, these PGC may be transformed in terms of geometric resolutions (e.g., decimated so that the boundaries of the objects have less resolution). For example and turning toFIG.3B, for each tuple τ of Equation 15, the surface patch ωrsmay subsampled (e.g., decimated by half sequentially) to lower the resolution of the corresponding image. In some embodiments, the decimation may occur at least five times so that the corresponding resolution is lowered by at least five levels 0≤L≤4 in such a way that the remaining signal content is 2{circumflex over ( )}(−L) at each level where L refers to the level of detail. In doing so, a signal-to geometry occupancy alignment in the octree map (Mvfrom Equation 5) may be maintained. The following equation may represent various degraded tuples that may be stored in the PGCs314:
τL=[(ωrs,L,Mv),q∈,r∈q,F(r)]   Equation 16
The function l may be a dual-purpose operation of L-subsampling and surface registration in the occupancy map attribute Mv.

For example, an original ultra-high PGC314amay be reduced. The process300may reduce the resolution342of the ultra-high PGC314ato generate high resolution PGC314b. The process300may also reduce the resolution344of the high resolution PGC314bto generate medium resolution PGC314c. The process300may also reduce the resolution346of the medium resolution PGC314cto generate low resolution PGC314d. The process300may also reduce the resolution348of the low resolution PGC314dto generate ultra-low resolution PGC314e. The PGC314emay be stored as the PGC314.

The PGCs314may be accessed by a mission planner, such as mission planner108ofFIG.1, to generate plans for end effector control to manipulate objects. In some embodiments, the low resolution PGC314emay be accessed by the mission planner to generate the plans.

FIG.4illustrates a process350to model a kinematic end effector. For example, robot modeler354may be connected to semantic surface map358and a PGC360as described herein. The robot modeler354may include a volumetric description356and end effector kinematics362. The robot modeler354may be readily substituted for the robot modeler106forFIG.1.

The volumetric description356and end effector kinematics362may be at least partially programmed ahead of time. The end effector kinematics362may represent robot actuators as a set of link bodies and joints (e.g., axes and joint range limits, torque capability maps, link and joint trees and axis limits). Such end effector kinematics362may facilitate planning for motions while avoiding undesired collisions. The end effector kinematics362may represent the direct and inverse kinematic of the actuator as mappings expressed by the following:

In equation 17, for n motorized degrees of freedom to rigid frames may be represented by T∈6⊂SE3. This notation may consider the non-bijective nature of the transformation by adding a no-solution configuration “\0” in the equation 17 on both domain and co-domain.

The volumetric descriptions356may include volumes of segments of the kinematic end effector. Different segments may include different volumes.

The robot modeler354may generate graspability maps368. The graspability maps368may be a discretization of a workspace where the kinematic end effector may apply contacts with a minimal nominal force. The grapsability maps368may be attribute-container voxel maps based on the semantic surface map358and/or the PGC360. For example, the grapsability maps368comprehends not only occupancy values but also the strength of forces and range of orientation reachable at that position. For example, each of the grapsability maps368may be a color map that reflects a cross section that encodes the dexterity of each voxel (e.g., blue stand for maximal graspability). This discrete function or grasping index may expressed by the following equation:
λgraspability(x∈3)→[0,1]∈Equation 18
The robot modeler354may penalize grasps according to volumetric feasibility, force feasibility and kinematic feasibility. For example, grasps that exceed a contact force, and as identified from the graspability maps368, may be rejected to avoid damaging objects.

The robot modeler354may therefore efficiently and quickly reject unsuitable grasps while simultaneously providing a mathematical analysis to drive gradient estimations for best grips during, for example, neural network training. The robot modeler354may further select grasps from the graspability map368and generate grasp actions and rankings based on direct and inverse kinematics and grasping indexes366. Thus, the robot modeler354may generate actionable grasp models, and rank the models based on metrics364.

FIG.5illustrates a process380to plan a mission. The process380may include a mission planner390. The mission planner390may be readily substituted for the mission planner108ofFIG.1. The mission planner390may be connected to a semantic surface map384, actional grasp models and ranking based on metrics382, sensor array386and PGC398.

The mission planner390may capture and unfold high-level directives from sensor data provided by the sensor array386(e.g., “clean the kitchen”). The mission planner390may decompose the directive into a fine granular sequence of physical atomic-actions or tasks (e.g., primary task, secondary task, target object part assertion, affordance list, etc.) to accomplish the high level directive. The tasks may be stored in the task information392.

Furthermore, actions may be considered macro-plans that provide operational limits corresponding to the specific domain and task and action goals394. The mission planner390may set a maximal speed of an end-effector along a manipulation trajectory (for social space sharing) or keeping containers with orientation limits (e.g., smoothness or responsiveness) to avoid spilling liquids. The case-by-case operational limits may change in each step of the plan, and may be stored in the operation limits388. Thus, providing this information in the grasp planning may enable filtering both affordances by attribute and prioritizing affordances by range matching.

The resulting symbolic plan with attributes and an active subsets of actions may be stored as part of the online composition and description of the atomic task396. The active subset of actions may be the resulting intersection of actions (verbs linked by noun names in the segmented parts or patches) that may be derived from the PGC398, and verbs from the decomposition of the high-level directive while creating atomic tasks.

Further, a language-based formal representation of a contextualized atomic task may be action tuples composed by five elements: i) a place-label ιcdescribing general context or location where the action is happening such as “Kitchen”, “Living-room”, etc. ii) an object-label ιodescribing the target object, for instances “Mug”, “Pillow”, etc. iii) a part-label ιpdepicting a region induced from the target object, namely “Handle”, “Arm”, “Rest”, iv) a verb-segment-label ιsdescribing the segment action which may contain one or more atomic actions, for example “Arrange”, “Sort”, “Dispose”, etc. and finally v) an atomic-verb-label ιadenoting indivisible operation such as “Pick”, “Slide”, “Push”, “Pull”, etc. For learning and inference purposes, these labels are embedded into vector Π(ι∈String)grepresentations which enable a mathematical vector operation rendering semantic properties such as implicit analogies by vector subtraction or additions. The following equation may express the above:
γ:=[Π(ιc),Π(ιo),Π(ιp),Π(ιs),Π(ιa)]∈5gEquation 19
The dimensionality (e.g.,5g) may depend on a specific embedding (e.g., bag of words or skip grammars), or a more compact one when using recent stand-alone unsupervised methods which require no text-window.

FIG.6shows a method400of analyzing and representing an environment. The method400may generally be implemented in a robotic process such as, for example, the process100(FIG.1) and/or the process300(FIGS.3A and3B) already discussed. The method400may be implemented in one or more modules as a set of logic instructions stored in a machine- or computer-readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., in configurable logic such as, for example, PLAs, FPGAs, CPLDs, in fixed-functionality logic hardware using circuit technology such as, for example, ASIC, CMOS or TTL technology, or any combination thereof.

Illustrated processing block402may generate one or more of an occupied space map, an unoccupied space map and an unknown space map. Illustrated processing block404generates a surface map based on the one or more of the occupied space maps, the unoccupied space map and the unknown space map. Illustrated processing block406labels objects in the surface map. Illustrated processing block408label parts of the objects in the surface map. Method400may further include conducting a PGC process on the surface map.

FIG.7shows a method420of classifying an end effector. The method420may generally be implemented in a robotic process such as, for example, the process100(FIG.1) and/or the process350(FIG.4) already discussed. The method420may be implemented in one or more modules as a set of logic instructions stored in a machine- or computer-readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., in configurable logic such as, for example, PLAs, FPGAs, CPLDs, in fixed-functionality logic hardware using circuit technology such as, for example, ASIC, CMOS or TTL technology, or any combination thereof.

Illustrated processing block422identifies characteristics of an end effector. Illustrated processing block424identifies capabilities of the end effector. Illustrated processing block426determines direct and inverse kinematics associated with the end effector. Illustrated processing block428generates end effector indexes (e.g., graspability maps). Illustrated processing block430ranks the actions.

FIG.8shows a method440of mission planning. The method440may generally be implemented in a robotic process such as, for example, the process100(FIG.1) and/or the process380(FIG.5) already discussed. The method440may be implemented in one or more modules as a set of logic instructions stored in a machine- or computer-readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., in configurable logic such as, for example, PLAs, FPGAs, CPLDs, in fixed-functionality logic hardware using circuit technology such as, for example, ASIC, CMOS or TTL technology, or any combination thereof.

Illustrated processing block442identifies operational limits of a mission. Illustrated processing block444identifies task information. Illustrated processing block446determines operational bounds. Illustrated processing block448determines a goal of the mission. Illustrated processing block450generates a plan based on information provided in the above identified processing blocks.

FIG.9illustrates an octree480as a spatial map separating the occupied from the empty space based on a multi scan point-cloud. As illustrated the octree480represents a staircase. The octree480may be an occupied space map.

FIG.10illustrates a structured point-cloud with vertices and edges. As illustrated, surfaces470may be tessellated with primitives that are triangles472. Valid triangles472are illustrated with first characteristics (e.g., unbroken lines). In contrast, a triangle474may be considered an invalid primitive (based on criteria as described herein) and illustrated as dashed lines. The invalid triangle474may be discarded for future processing.

FIG.11illustrates a semantic segmented and labeled image498. As illustrated, a first object may be labeled as phone486while the remaining objects may be labeled as cubes482,484,496. The labels may be generated as described herein. The labels may be stored in tuples for each object.

FIG.12illustrates a part-wise semantic segmentation image500of a chair. The various parts of the chair may be labeled. For example, the backrest488, arm490, seat492and base494may be identified and labeled. The labels may be stored in tuples for the chair.

FIG.13illustrates a surface and occupancy subsampling process510to reduce resolution of a high quality image and generate PGCs. The process510may reduce the resolution512,514to decay the level of detail exponentially (e.g., decimated) allowing efficient summarization while retaining collision cues.

FIG.14illustrates a training method550to train a DNN for any of the embodiments described herein. The method550may generally be implemented in a robotic process such as, for example, the process100(FIG.1), the method800(FIG.2), the process300(FIGS.3A-3B), the process350(FIG.4), the process380(FIG.5), the method400(FIG.6), the method420(FIG.7), the method440(FIG.8) already discussed. The method440may be implemented in one or more modules as a set of logic instructions stored in a machine- or computer-readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., in configurable logic such as, for example, PLAs, FPGAs, CPLDs, in fixed-functionality logic hardware using circuit technology such as, for example, ASIC, CMOS or TTL technology, or any combination thereof.

Illustrated processing block552identifies resolutions (e.g., five) to train the DNNs. Illustrated processing block554may train the DNNs at the identified resolutions.

In order to train each of these networks via supervised corrective learning (e.g., stochastic gradient descent), the exploitation of simulation engines may enable the recreation of sensor signals comparable (up to noise effects) to those from the real sensors. Using these virtual signals from 3D scenarios (e.g., watertight models with semantic and functional knowledge), the execution of a sensing pipeline may extract the view dependent geometric primitives while semantics and part labels are obtained directly from the models. This allows efficient production of quasi-realistic annotated perception datasets.

Further, exploiting discrete closed-form (DCF) methods for model-based grasping sampled at fine granularity, it may be possible to obtain large collections of valid grasp configuration that may lack only the semantic understanding. Consequently, the process associating semantic-parts (as PGC) with DCF-grasp configurations having enough (with respect to dynamic stability) contact points in a single labeled part may define a mini-batch during training. Further, by growing this associations via language-based affordance in from of vectors arising in language embedding, it may then be possible to formulate the training of each of the neural networks at their native resolutions with all geometric, semantic part, affordance cues. The cost function employed as optimization target during training may be expressed as:

Ψ=∑O∑τL∑ωrs(1-∏i∈Grasp-setλgrasp-ability(Ti))2︸Epanechnikov⁢Grasp-ability⁢Weighting·℧Grasp-set⁢(Ti,Tj),︸Alignment-cost︷For⁢All⁢Perceptual⁢Grasping⁢Components︷For⁢all⁢objects⁢i⁢n⁢the⁢training⁢setEquation⁢20
In equation 20, “O” stands for an object in the training set and the alignment cost functionGrasp-set+maps the delta in rotation, translation and joint configuration between the grasp Tiand Tjwith minimal distance from the training generated grasping set. The graspability may act as a weighting factor for this distance in such a way that the error with lower support imply larger cost for the network optimizer.

Finally, once the networks are trained, illustrated processing block556may compose the weights and kernel values into a bitstream for the programmable accelerators.

Turning now toFIG.15, an enhanced object manipulation planning computing system150is shown. The system150may generally be part of an electronic device/platform having computing functionality (e.g., personal digital assistant/PDA, notebook computer, tablet computer, convertible tablet, server), communications functionality (e.g., smart phone), imaging functionality (e.g., camera, camcorder), media playing functionality (e.g., smart television/TV), wearable functionality (e.g., watch, eyewear, headwear, footwear, jewelry), vehicular functionality (e.g., car, truck, motorcycle), robotic functionality (e.g., autonomous robot), etc., or any combination thereof. In the illustrated example, the system150includes a host processor152(e.g., CPU) having an integrated memory controller (IMC)154that is coupled to a system memory156.

The illustrated system150also includes an input output (10) module158implemented together with the host processor152and a graphics processor160(e.g., GPU) on a semiconductor die162as a system on chip (SoC). The illustrated IO module158communicates with, for example, a display164(e.g., touch screen, liquid crystal display/LCD, light emitting diode/LED display), a network controller166(e.g., wired and/or wireless), and mass storage168(e.g., hard disk drive/HDD, optical disk, solid state drive/SSD, flash memory). In some embodiments, the system150may further include processors and/or AI accelerators148dedicated to artificial intelligence (AI) and/or neural network (NN) processing. For example, the system SoC162may include vision processing units (VPUs) and/or other AI/NN-specific processors such as AI accelerator148, etc. In some embodiments, any aspect of the embodiments described herein may be implemented in one or more of the processors and/or accelerators such as AI accelerator148dedicated to AI and/or NN processing, the graphics processor160and/or the host processor152.

The host processor152, the graphics processor160and/or the IO module158may execute instructions170retrieved from the system memory156and/or the mass storage168. In an embodiment, the computing system150is operated in an application development stage and the instructions170include executable program instructions to perform one or more aspects of the process100(FIG.1), the method800(FIG.2), the process300(FIGS.3A-3B), the process350(FIG.4), the process380(FIG.5), the method400(FIG.6), the method420(FIG.7) and the method440(FIG.8) already discussed. Thus, execution of the illustrated instructions170may cause the computing system150to generate a semantic labelled image based on image data from the sensor data, where the semantic labelled image is to identify a shape of an object and a semantic label of the object, associate a first set of actions with the object, decompose a command from a user into a second set of actions associated with the object, and generate a plan based on an intersection of the first set of actions and the second set of actions to satisfy the command through actuation of one or more end effectors. The kinematic end effector144may implement the plan to manipulate an object.

The system150may further include an imaging sensor142and microphone140to receive sensor data. For example, a user may issue a verbal command to the system150through the microphone140. In some embodiments, the network controller166may register a command issued from another device coupled and remote to the system150. The imaging sensor142may capture images that are analyzed to determine the image data.

The illustrated computing system150is therefore considered to be performance-enhanced at least to the extent that it enables the computing system150to take advantage of environmental data to generate an effective plan to manipulate the kinematic end effector144to manipulate the object. The object may be previously unseen by the system150and the system150may be execute in a dynamic and changing environment.

FIG.16shows a semiconductor apparatus172(e.g., chip, die, package). The illustrated apparatus172includes one or more substrates174(e.g., silicon, sapphire, gallium arsenide) and logic176(e.g., transistor array and other integrated circuit/IC components) coupled to the substrate(s)174. In an embodiment, the apparatus172is operated in an application development stage and the logic176performs one or more aspects of the process100(FIG.1), the method800(FIG.2), the process300(FIGS.3A-3B, the process350(FIG.4), the process380(FIG.5), the method400(FIG.6), the method420(FIG.7) and the method440(FIG.8) already discussed. Thus, the logic176may generate a semantic labelled image based on image data from the sensor data, where the semantic labelled image is to identify a shape of an object and a semantic label of the object, associate a first set of actions with the object, decompose a command from a user into a second set of actions associated with the object, and generate a plan based on an intersection of the first set of actions and the second set of actions to satisfy the command through actuation of one or more end effectors. Thus, the logic176may allow for dynamic adjustments of kinematic end effectors to grasp objects based on the environment.

The logic176may be implemented at least partly in configurable logic or fixed-functionality hardware logic. In one example, the logic176includes transistor channel regions that are positioned (e.g., embedded) within the substrate(s)174. Thus, the interface between the logic176and the substrate(s)174may not be an abrupt junction. The logic176may also be considered to include an epitaxial layer that is grown on an initial wafer of the substrate(s)174.

In some embodiments, the logic176may further include processors (not shown) and/or accelerators (not shown) dedicated to AI and/or NN processing. For example, the logic176may include VPUs, and/or other AI/NN-specific processors, etc. In some embodiments, any aspect of the embodiments described herein may be implemented in the processors and/or accelerators dedicated to AI and/or NN processing.

FIG.17also illustrates a memory270coupled to the processor core200. The memory270may be any of a wide variety of memories (including various layers of memory hierarchy) as are known or otherwise available to those of skill in the art. The memory270may include one or more code213instruction(s) to be executed by the processor core200, wherein the code213may implement the process100(FIG.1), the method800(FIG.2), the process300(FIGS.3A-3B, the process350(FIG.4), the process380(FIG.5), the method400(FIG.6), the method420(FIG.7) and the method440(FIG.8) already discussed. The processor core200follows a program sequence of instructions indicated by the code213. Each instruction may enter a front end portion210and be processed by one or more decoders220. The decoder220may generate as its output a micro operation such as a fixed width micro operation in a predefined format, or may generate other instructions, microinstructions, or control signals which reflect the original code instruction. The illustrated front end portion210also includes register renaming logic225and scheduling logic230, which generally allocate resources and queue the operation corresponding to the convert instruction for execution.

Although not illustrated inFIG.17, a processing element may include other elements on chip with the processor core200. For example, a processing element may include memory control logic along with the processor core200. The processing element may include I/O control logic and/or may include I/O control logic integrated with memory control logic. The processing element may also include one or more caches.

Referring now toFIG.18, shown is a block diagram of a computing system1000embodiment in accordance with an embodiment. Shown inFIG.18is a multiprocessor system1000that includes a first processing element1070and a second processing element1080. While two processing elements1070and1080are shown, it is to be understood that an embodiment of the system1000may also include only one such processing element.

Each processing element1070,1080may include at least one shared cache1896a,1896b. The shared cache1896a,1896bmay store data (e.g., instructions) that are utilized by one or more components of the processor, such as the cores1074a,1074band1084a,1084b, respectively. For example, the shared cache1896a,1896bmay locally cache data stored in a memory1032,1034for faster access by components of the processor. In one or more embodiments, the shared cache1896a,1896bmay include one or more mid-level caches, such as level 2 (L2), level 3 (L3), level 4 (L4), or other levels of cache, a last level cache (LLC), and/or combinations thereof.

As shown inFIG.18, various I/O devices1014(e.g., biometric scanners, speakers, cameras, sensors) may be coupled to the first bus1016, along with a bus bridge1018which may couple the first bus1016to a second bus1020. In one embodiment, the second bus1020may be a low pin count (LPC) bus. Various devices may be coupled to the second bus1020including, for example, a keyboard/mouse1012, communication device(s)1026, and a data storage unit1019such as a disk drive or other mass storage device which may include code1030, in one embodiment. The illustrated code1030may implement the one or more aspects of process100(FIG.1), the method800(FIG.2), the process300(FIGS.3A-3B, the process350(FIG.4), the process380(FIG.5), the method400(FIG.6), the method420(FIG.7), the method440(FIG.8) already discussed, and may be similar to the code213(FIG.17), already discussed. Further, an audio I/O1024may be coupled to second bus1020and a battery1010may supply power to the computing system1000.

Note that other embodiments are contemplated. For example, instead of the point-to-point architecture ofFIG.18, a system may implement a multi-drop bus or another such communication topology. Also, the elements ofFIG.18may alternatively be partitioned using more or fewer integrated chips than shown inFIG.18.

ADDITIONAL NOTES AND EXAMPLES

Example 1 includes a computing system comprising one or more sensors to generate sensor data, the sensor data to include image data, a processor coupled to the one or more sensors, and a memory including a set of executable program instructions, which when executed by the processor, cause the computing system to generate a semantic labelled image based on image data from the sensor data, wherein the semantic labelled image is to identify a shape of an object and a semantic label of the object, associate a first set of actions with the object and generate a plan based on an intersection of the first set of actions and a second set of actions to satisfy a command from a user through actuation of one or more end effectors, wherein the second set of actions are to be associated with the command.

Example 2 includes the computing system of Example 1, wherein the instructions, when executed, further cause the computing system to apply a first label to a first portion of the object, and apply a second label to a second portion of the object, wherein the second label is to be different from the first label.

Example 3 includes the computing system of Example 1, wherein the instructions, when executed, further cause the computing system to generate a surface patch from the semantic labelled image that is to represent the object, decrease a resolution of the surface patch, and generate the plan based on the surface patch having the decreased resolution.

Example 4 includes the computing system of Example 1, wherein the instructions, when executed, further cause the computing system to identify a contact force map that is to represent contact force outputs mapped to portions of the one or more end effectors, and generate the plan based on the contact force map.

Example 5 includes the computing system of any one of Examples 1-4, wherein the instructions, when executed, further cause the computing system to generate an occupancy map based on the image data, wherein the occupancy map is to identify portions of the image that are to be occupied, and generate a surface map based on the occupancy map that is to identify surfaces of objects.

Example 6 includes the computing system of Example 5, wherein the instructions, when executed, further cause the computing system to connect the surfaces in the surface map to labels to generate the semantic labelled image, and identify the first set of actions based on the labels, wherein the first set of actions is to include a first action to physically manipulate the object, wherein the second set of actions is to include the first action to physically manipulate the object; and wherein the plan is to include a physical manipulation of the object with the first action.

Example 7 includes a semiconductor apparatus comprising one or more substrates, and logic coupled to the one or more substrates, wherein the logic is implemented in one or more of configurable logic or fixed-functionality logic hardware, the logic coupled to the one or more substrates to generate a semantic labelled image based on image data, wherein the semantic labelled image is to identify a shape of an object and a semantic label of the object, associate a first set of actions with the object, and generate a plan based on an intersection of the first set of actions and a second set of actions to satisfy a command from a user through actuation of one or more end effectors, wherein the second set of actions are to be associated with the command.

Example 8 includes the apparatus of Example 7, wherein the logic coupled to the one or more substrates is to apply a first label to a first portion of the object, and apply a second label to a second portion of the object, wherein the second label is to be different from the first label.

Example 9 includes the apparatus of Example 7, wherein the logic coupled to the one or more substrates is to generate a surface patch from the semantic labelled image that is to represent the object, decrease a resolution of the surface patch, and generate the plan based on the surface patch having the decreased resolution.

Example 10 includes the apparatus of Example 7, wherein the logic coupled to the one or more substrates is to identify a contact force map that is to represent contact force outputs mapped to portions of the one or more end effectors, and generate the plan based on the contact force map.

Example 11 includes the apparatus of any one of Examples 7-10, wherein the logic coupled to the one or more substrates is to generate an occupancy map based on the image data, wherein the occupancy map is to identify portions of the image that are to be occupied, and generate a surface map based on the occupancy map that is to identify surfaces of objects.

Example 12 includes the apparatus of Example 11, wherein the logic coupled to the one or more substrates is to connect the surfaces in the surface map to labels to generate the semantic labelled image, and identify the first set of actions based on the labels, wherein the first set of actions is to include a first action to physically manipulate the object, wherein the second set of actions is to include the first action to physically manipulate the object, and wherein the plan is to include a physical manipulation of the object with the first action.

Example 13 includes the apparatus of Example 7, wherein the logic coupled to the one or more substrates includes transistor channel regions that are positioned within the one or more substrates.

Example 14 includes at least one computer readable storage medium comprising a set of executable program instructions, which when executed by a computing system, cause the computing system to generate a semantic labelled image based on image data, wherein the semantic labelled image is to identify a shape of an object and a semantic label of the object, associate a first set of actions with the object, and generate a plan based on an intersection of the first set of actions and a second set of actions to satisfy a command from a user through actuation of one or more end effectors, wherein the second set of actions are to be associated with the command.

Example 15 includes the at least one computer readable storage medium of Example 14, wherein the instructions, when executed, further cause the computing system to apply a first label to a first portion of the object, and apply a second label to a second portion of the object, wherein the second label is to be different from the first label.

Example 16 includes the at least one computer readable storage medium of Example 14, wherein the instructions, when executed, further cause the computing system to generate a surface patch from the semantic labelled image that is to represent the object, decrease a resolution of the surface patch, and generate the plan based on the surface patch having the decreased resolution.

Example 17 includes the at least one computer readable storage medium of Example 14, wherein the instructions, when executed, further cause the computing system to identify a contact force map that is to represent contact force outputs mapped to portions of the one or more end effectors, and generate the plan based on the contact force map.

Example 18 includes the at least one computer readable storage medium of any one of Examples 14-17, wherein the instructions, when executed, further cause the computing system to generate an occupancy map based on the image data, wherein the occupancy map is to identify portions of the image that are to be occupied, and generate a surface map based on the occupancy map that is to identify surfaces of objects.

Example 19 includes the at least one computer readable storage medium of Example 18, wherein the instructions, when executed, further cause the computing system to connect the surfaces in the surface map to labels to generate the semantic labelled image, and identify the first set of actions based on the labels, wherein the first set of actions is to include a first action to physically manipulate the object, wherein the second set of actions is to include the first action to physically manipulate the object, and wherein the plan is to include a physical manipulation of the object with the first action.

Example 20 includes a method of operating a computing system, the method comprising generating a semantic labelled image based on image data, wherein the semantic labelled image is to identify a shape of an object and a semantic label of the object, associating a first set of actions with the object, and generating a plan based on an intersection of the first set of actions and a second set of actions to satisfy a command from a user through actuation of one or more end effectors, wherein the second set of actions are to be associated with the command.

Example 21 includes the method of Example 20, further comprising applying a first label to a first portion of the object, and applying a second label to a second portion of the object, wherein the second label is to be different from the first label.

Example 22 includes the method of Example 20, further comprising generating a surface patch from the semantic labelled image that is to represent the object, decreasing a resolution of the surface patch, and generating the plan based on the surface patch having the decreased resolution.

Example 23 includes the method of Example 20, further comprising identifying a contact force map that represents contact force outputs mapped to portions of the one or more end effectors, and generating the plan based on the contact force map.

Example 24 includes the method of any one of Examples 20-23, further comprising generating an occupancy map based on the image data, wherein the occupancy map is to identify portions of the image that are to be occupied, and generating a surface map based on the occupancy map that is to identify surfaces of objects.

Example 25 includes the method of Example 24, further comprising connecting the surfaces in the surface map to labels to generate the semantic labelled image, and identifying the first set of actions based on the labels, wherein the first set of actions is to include a first action to physically manipulate the object, wherein the second set of actions is to include the first action to physically manipulate the object, and wherein the plan is to include a physical manipulation of the object with the first action.

Example 26 includes a semiconductor apparatus comprising means for generating a semantic labelled image based on image data, wherein the semantic labelled image is to identify a shape of an object and a semantic label of the object, means for associating a first set of actions with the object, and means for generating a plan based on an intersection of the first set of actions and a second set of actions to satisfy a command from a user through actuation of one or more end effectors, wherein the second set of actions are to be associated with the command.

Example 27 includes the apparatus of Example 26, further comprising means for applying a first label to a first portion of the object, and means for applying a second label to a second portion of the object, wherein the second label is to be different from the first label.

Example 28 includes the apparatus of Example 26, further comprising means for generating a surface patch from the semantic labelled image that is to represent the object, means for decreasing a resolution of the surface patch, and means for generating the plan based on the surface patch having the decreased resolution.

Example 29 includes the apparatus of Example 26, further comprising means for identifying a contact force map that represents contact force outputs mapped to portions of the one or more end effectors, and means for generating the plan based on the contact force map.

Example 30 includes the apparatus of any one of Examples 26-29, further comprising means for generating an occupancy map based on the image data, wherein the occupancy map is to identify portions of the image that are to be occupied, and means for generating a surface map based on the occupancy map that is to identify surfaces of objects.

Example 31 includes the apparatus of Example 26, further comprising means for connecting the surfaces in the surface map to labels to generate the semantic labelled image, and means for identifying the first set of actions based on the labels, wherein the first set of actions is to include a first action to physically manipulate the object, wherein the second set of actions is to include the first action to physically manipulate the object, and wherein the plan is to include a physical manipulation of the object with the first action

Thus, technology described herein may provide for an autonomous robot that dynamically adjusts kinematic end effectors to manipulate (e.g., grip) objects in quickly changing environments. The autonomous robot may also reduce memory footprints and latency with resolution reduced semantic patches and early vetting of and rejection of unsuitable grasps and semantic patches.