Systems and methods for generating three dimensional skeleton representations

Systems, robots, and methods for generating three-dimensional skeleton representations of people are disclosed. A method includes generating, from a two-dimensional image, a two-dimensional skeleton representation of a person present in the two-dimensional image. The two-dimensional skeleton representation includes a plurality of joints and a plurality of links between individual joints of the plurality of joints. The method further includes positioning a cone around one or more links of the plurality of links, and identifying points of a depth cloud that intersect with the cone positioned around the one or more links of the two-dimensional skeleton. The points of the depth cloud are generated by a depth sensor and each point provides depth information. The method also includes projecting the two-dimensional skeleton representation into three-dimensional space using the depth information of the points that intersect with the cone, thereby generating the three-dimensional skeleton representation of the person.

TECHNICAL FIELD

Embodiments described herein generally relate to object recognition and, more particularly, systems, robots and methods for generating three dimensional skeleton representations of people in an environment.

BACKGROUND

Computer vision may be used to determine the presence of a person in an image. For example, robots may use computer vision to determine the presence of a person in an environment so that the robot may co-habitat a space with people. That is, robots may rely on computer vision to determine a pose, orientation, or the like of a human so as to interact with the human. However, existing systems and methods may not adequately utilize computer vision to accurately estimate a pose, orientation, or the like. In addition, existing computer vision systems may not be able to authenticate a particular human if the human's face is obscured.

SUMMARY

In one embodiment, a method of generating a three-dimensional skeleton representation of a person includes generating, from a two-dimensional image, a two-dimensional skeleton representation of a person present in the two-dimensional image, wherein the two-dimensional skeleton representation comprises a plurality of joints and a plurality of links between individual joints of the plurality of joints. The method further includes positioning a cone around one or more links of the plurality of links, and identifying points of a depth cloud that intersect with the cone positioned around the one or more links of the two-dimensional skeleton, wherein the points of the depth cloud are generated by a depth sensor and each point provides depth information. The method also includes projecting the two-dimensional skeleton representation into three-dimensional space using the depth information of the points of the depth cloud that intersect with the cone positioned around one or more links of the plurality of links, thereby generating the three-dimensional skeleton representation of the person.

In another embodiment, a robot includes a processor and a non-transitory memory device storing machine-readable instructions that, when executed by the processor, cause the processor to generate, from a two-dimensional image, a two-dimensional skeleton representation of a person present in the two-dimensional image, wherein the two-dimensional skeleton representation comprises a plurality of joints and a plurality of links between individual joints of the plurality of joints. The machine-readable instructions further cause the processor to position a cone around one or more links of the plurality of links, and identify points of a depth cloud that intersect with the cone positioned around the one or more links of the two-dimensional skeleton, wherein the points of the depth cloud are generated by a depth sensor and each point provides depth information. The machine-readable instructions also cause the processor to project the two-dimensional skeleton representation into three-dimensional space using the depth information of the points of the depth cloud that intersect with the cone positioned around one or more links of the plurality of links, thereby generating the three-dimensional skeleton representation of the person.

In another embodiment, a system includes a processor and a non-transitory, processor readable storage device. The non-transitory, processor-readable storage device includes one or more machine-readable instructions thereon that, when executed by the processor, cause the processor to generate, from a two-dimensional image, a two-dimensional skeleton representation of a person present in the two-dimensional image, where the two-dimensional skeleton representation comprises a plurality of joints and a plurality of links between individual joints of the plurality of joints. The non-transitory, processor-readable storage device further includes one or more machine-readable instructions thereon that, when executed by the processor, cause the processor to position a cone around one or more links of the plurality of links and identify points of a depth cloud that intersect with the cone positioned around the one or more links of the two-dimensional skeleton, where the points of the depth cloud are generated by a depth sensor and each point provides depth information. The non-transitory, processor-readable storage device also includes one or more machine-readable instructions thereon that, when executed by the processor, cause the processor to project the two-dimensional skeleton representation into three-dimensional space using the depth information of the points of the depth cloud that intersect with the cone positioned around one or more links of the plurality of links, thereby generating the three-dimensional skeleton representation of the person.

DETAILED DESCRIPTION

Embodiments disclosed herein are directed to systems and methods for generating three dimensional (3D) skeleton representations of people that include depth information. A 3D skeleton representation may be utilized to determine where a person is located in a 3D space. Further, embodiments enable the detection of a 3D pose estimation of a person in the 3D space. Particularly, a two dimensional (2D) skeleton representation of a person is generated from red-green-blue (RGB) image data. The 2D skeleton representation is then merged with depth information, such as depth information obtained from a depth sensor. As an example and not a limitation, the RGB image data and the depth information may be obtained from an RGB-D camera that creates both 2D RGB images and depth information in a single data package. The result is a 3D skeleton representation of a person providing information regarding a 3D pose of the person as well as the location of the person in 3D space. As an example and not a limitation, the 3D skeleton may be generated using video in real time.

The 3D skeleton representations described herein may be utilized in a wide variety of applications. In one non-limiting application, a robot may use the 3D skeleton representation to determine a location and pose of a person in the environment for the purposes of assisting humans in a variety of tasks. In one example, a robot may be deployed in human occupied spaces, such as homes, special care facilities, and hospitals. These robots may share the same space as humans for purposes such as general assistance and companionship. For example, a robot may be deployed in the home of a person needing physical assistance, such as an elderly person, a handicapped person, or an injured person. The robot may be mobile and may have actuators usable to retrieve objects for the person, for example. Such robots may make the person feel more independent because he or she may utilize the robot to be less reliant on other people for support. Accordingly, embodiments of the present disclosure may assist robots in interacting with people in the environment by determining the location and pose of the people using 3D skeleton representations. It should be understood that, although embodiments are described herein in the context of human-assistive robot applications, embodiments are not limited thereto.

The embodiments described herein may generally be employed on specialized machinery (i.e., robots) that are particularly adapted for carrying out the various processes for imaging an environment and determining whether a human is present, as well as particular characteristics of the human (i.e., pose). However, the present disclosure is not limited to specialized machinery. That is, certain embodiments described herein may be employed on a general computing device communicatively coupled to one or more sensors. In such embodiments, the systems and methods described herein may improve the functionality of the general computing device by providing the general computing device with an ability to more accurately recognize whether a human is present in an environment, how the human is posed, and/or the like, as well as accurately determine an identity of the human, even in instances where a human's face cannot be accurately sensed by the sensors (i.e., because the human is not facing the sensors or the human's face is otherwise obscured).

Referring now toFIG. 1, an illustrative robot, general designated100, according to embodiments may be deployed in a human-occupied space, such as, for example, a home. The robot100has motorized wheels106(or other mobility components such as skis, wings, rotors, continuous tracks, or the like) that are capable of moving the robot throughout the environment, as well as one or more arms104having an end effector105(e.g., a gripper, a robotic hand, and/or the like) capable of grasping and/or manipulating objects. Additionally, the robot100has a plurality of sensors102capable of sensing the environment and producing sensor data as a result that assists the robot100in detecting objects, manipulating objects, and navigating the environment. The plurality of sensors102may include, but is not limited to, cameras (e.g., CCD cameras), infrared sensors, depth sensors, proximity sensors, tactile sensors, Lidar sensors, radar sensors, time of flight sensors, and the like. The plurality of sensors102may be capable of generating 2D RGB images as well as depth information. In one example, at least one of the sensors is a RGB-D sensor housed in one package. In another example, the 2D RGB image data and the depth information are generated by separate sensors. It should be understood that the robot100shown inFIG. 1is provided for illustrative purposes only, and that the configuration of the robot is not limited by the present disclosure.

As previously described herein, the robot100depicted inFIG. 1may be particularly configured to develop a 3D skeleton representation of a person. Referring now toFIGS. 1 and 2, a flowchart130of an illustrative process for developing a 3D skeleton representation of a person is graphically illustrated. As shown inFIG. 1, the robot100(or other computing device) may image a person10in the environment. Still referring toFIGS. 1 and 2, sensor(s)102produce data regarding the person. That is, an RGB sensor (e.g., a CCD camera) produces a 2D image10′ (FIG. 3) of the person10, at block131. In addition, a depth sensor also produces a depth cloud of points within the environment (not shown). The depth sensor and the RGB sensor may be provided in a single sensor102as shown inFIG. 1, or in separate sensors.

Generally referring toFIGS. 1-5, a 2D skeleton representation20of the person10is created, an example of which is shown inFIGS. 4 and 5(block132ofFIG. 2).FIG. 4shows a full view of the 2D skeleton representation20whileFIG. 5is a partial view depicting the arm22of the 2D skeleton representation20shown inFIG. 4. Still referring toFIGS. 1-5, the 2D skeleton representation20may include a plurality of joints, such as, for example, a shoulder joint24, an elbow joint23, and a wrist26. Other joints not specifically described herein are also contemplated within the 2D skeleton representation20. A link is provided between joints, such as link25between the shoulder joint24and the elbow joint23(e.g., representing the humerus) and link27between the elbow joint23and the wrist26(e.g., representing the ulna and radius). Any known or yet-to-be-developed method for generating the 2D skeleton representation20may be utilized. For example, the 2D skeleton representation may be developed using MATLAB (Mathworks, Natick Mass.) in combination with a Microsoft COCO (Microsoft Corp., Redmond Wash.) dataset, and/or the like.

Next, at block133of the flowchart130shown inFIG. 2, a cone C having a diameter d is located around each of the links in the 2D skeleton representation20as shown inFIGS. 4 and 5. The cone C includes one or more computer-generated ring structures that are positioned around the links such that the links pass through a center of the cone C. The diameter d is generally selected based on an estimated size of the human's body part that corresponds to the link. For example, if the link corresponds to a human's arm, the diameter d of the cone C may be selected to correspond to an estimated diameter of the human's arm. As such, the cone C should generally correspond in size to the respective body part. Accordingly, the diameter d is not limited by this disclosure. As an example and not a limitation, the diameter d may be about three (3) centimeters to about five (5) centimeters. As further described hereinbelow, the cone C is located for the purposes of creating a depth cloud pertaining to the detected person.

Referring now toFIG. 6, the 2D skeleton representation20having the cones C is merged with the depth information received from the depth sensor. For example, the depth sensor (e.g., sensor102) creates a depth cloud having a plurality of points30in a scene. As an example and not a limitation, the points30may be generated by infrared laser beams that are projected by the depth sensor that are projected onto the person10(FIG. 1). That is, the depth sensor may emit light (e.g., one or more laser beams) in a direction generally toward the person10(FIG. 1). At least a portion of the light is reflected by the person10(FIG. 1) and/or objects/people surrounding the person10. The reflected light is visible by the depth sensor as the points30.

Referring toFIGS. 2 and 6, depth points that intersect with the cones C are determined at block134.FIG. 6shows the projection of the points30on the cone C and the other areas surrounding the link27. More specifically, the points30include intersecting points30aand non-intersecting points30b. The intersecting points30aare generally points that intersect with the cone C and the non-intersecting points30bare generally points that do not intersect with the cone C (i.e., are located outside the cone C). Only the intersecting points30athat intersect the cones C are considered in the present analysis. For purposes of illustration,FIG. 6only shows the intersecting points30athat intersect with the cone C of the arm22of the person10and a small number of surrounding non-intersecting points30b. However, it should be understood that there are many additional non-intersecting points30boutside of the cones C of the 2D skeleton representation20. Moreover, such non-intersecting points30bmay only be non-intersecting with respect to a particular cone C. That is, a particular point may be a non-intersecting point30bwith respect to the cone C around link25(FIG. 5) but may be an intersecting point30awith respect to the cone C around link27(FIG. 5). When a particular point is a non-intersecting point30b, such points are not considered for the purposes of determining the 2D skeleton representation. Each intersecting point30aof the depth cloud provides depth information regarding the 2D skeleton representation20. Particularly, each intersecting point30aprovides a distance from the object it is incident upon to the depth sensor. That is, a distance between the depth sensor and each particular one of the intersecting points30acan be determined, as described hereinbelow.

The depth information from the intersecting points30ais used to determine how far away the 2D skeleton representation20is from the depth sensor (e.g., sensor102shown inFIG. 1). In one example, an overall average depth is taken from the depth information of all of the intersecting points30athat intersect with the cones C of the 2D skeleton representation20(block135ofFIG. 2). That is, a depth is calculated for each one of the intersecting points30aby any method of calculating a distance from a distance sensor, including an angular calculation, a time-of-flight calculation, and/or the like. All of the calculated depths are then averaged together to obtain an overall average depth.FIG. 7schematically depicts a 3D skeleton representation20′ at a depth D from a sensor102based on an overall average of all of the intersecting points30athat intersect with the cones C of the 2D skeleton representation20. In another example, the depth is determined individually for particular parts of the 2D skeleton representation20. For example, a determination may be made as to which of the intersecting points30aintersect with a person's left arm, then the depth may be determined for each of those intersecting points30aon the left arm and all of the depth information for the left arm may be averaged to find an average depth for the left arm. Similarly, all of the depth information for the right arm may be averaged to find an average depth for the right arm, all of the depth information for the right leg may be averaged to find an average depth for the right leg, and the like. Further, in some embodiments, the depth of the 2D skeleton representation20may be determined in even further granularity, such as small sections of each link of the 2D skeleton representation20, or even at the intersecting point30alevel depending on the application. It should be understood that other ways of determining depth from the intersecting points30athat intersect with the cones C of the 2D skeleton representation20may be utilized.

Thus, the RGB-D sensor may be utilized to determine a location of a skeleton representation in 3D space. Further, embodiments may also use the 3D skeleton representation20′ to determine a pose of a person (block136ofFIG. 2). For example, pose estimation may be used to determine which direction a person is facing, a person's posture, where a person's arms are located, how a person's arms are arranged, whether a person or certain body parts thereof are moving, and the like. In a non-limiting example, the pose of the person may generally be used by a robot100to coordinate a handoff of an object between the robot100and a person so as to ensure that the robot100accurately contacts the object, positions the object appropriately for the handoff, and releases the object once the person grasps the object. Using the 3D skeleton representation20′ described herein, the location of a person's hand in 3D space is stored in memory for the robot100to access when needed. The robot100may also be capable of object recognition such that the robot100can locate a particular object held by the person or located adjacent to a person. For example, if a person is holding an apple out to the robot100with her right hand, the robot can detect that the user is holding an apple, the location of the apple in 3D space, the orientation of the apple, and/or the like, and use the 3D skeleton representation20′ to locate the person's right hand. The robot100may then use this information to determine and execute particular movement commands to cause a natural grasp of the apple from the person's right hand.

Filtering of the 2D skeleton representation20or the 3D skeleton representation20′ (collectively “skeleton representations”) may also be performed to provide an accurate representation of the person viewed by the sensor102. For example, historical skeleton representations may be stored in a memory or the like, and rules may be developed that represent valid skeleton representations. For example, links representing arms on the same person may generally be within a certain proportion to one another (e.g., one arm link cannot be significantly larger than the other arm link), the links representing legs should be within a proportional range with respect to the arms, the links of the skeleton representation should provide for a pose that is capable of being performed by a human (e.g., human arms cannot be bent back in a certain way).

When a detected skeleton representation (either a 2D skeleton representation20or a 3D skeleton representation20′ including depth information) violates one of the rules based on the historical data (e.g., the arms do not correspond in size or respective location), corrective action may be taken. For example, another measurement may be taken and the incorrect measurement disregarded, or modifications to one or more links may be made to satisfy the one or more rules that were violated. In this manner, skeleton representations may be filtered by applying certain predetermined rules.

In some embodiments, the 3D skeleton representation20′ may also be used to identify a particular person. Facial recognition is a technique that may be used to detect a particular person. However, a person's face is not always clearly in view of a sensor, such as a camera. As such, in a robotics application, the robot100may not be programmed to recognize who a person is if the person is not facing the robot or otherwise facing imaging sensors that are accessible to the robot100. In some embodiments, a database containing information relating to registered users and their respective 3D skeleton representations20′ may be developed. The links and joints of the 3D skeleton representations20′ may provide for a unique identifier of a person, much like a fingerprint. A user may become a registered user by registering several 3D skeleton representations20′ for different poses. The robot100(or other computing device) may then develop an identification using various attributes of the of the 3D skeleton representations20′, such as, for example, a length of links between joints, a location of joints, a ratio of a length of one link to another link, and/or the like. Such attributes are generally unique to the registered user. As another example, the robot100(or other computing device) may record a user's gait by way of the 3D skeleton representation20′. That is, a moving image of the person (and thus the 3D skeleton representation20′ thereof) may be recorded so that information regarding gait can be determined and stored. A person's gait provides identifying information regarding that person. Therefore, a person's gait may also be stored in the database for identification purposes.

Accordingly, when imaging a person10, the robot100(or other computing device) may access the database to identify a user in any number of ways. Thus, a user may be identified even when his or her face is not visible. Additionally, known attributes of the identified user's 3D skeleton representation20′ may be applied in real time to correct for any errors that may have occurred with the 3D skeleton representation20′ that is currently being generated (e.g., correct for errors in length of any one link in the skeleton representation, correct for gait, or the like).

FIGS. 8A-8Care example images of 3D skeleton representations20′ that may be displayed or otherwise used by a robot or a computing device for any number of applications. More specifically,FIG. 8Adepicts the 3D skeleton representation20′ of a particular user in a 3D space70.FIGS. 8B and 8Cdepict the 3D skeleton representation20′ superimposed over the 3D image of the person10′ in the 3D space70. As shown inFIGS. 8B and 8C, the shape, size, and arrangement of the 3D skeleton representation20′ corresponds to the shape, size, and arrangement of the person10′. Other objects80(such as furniture, target objects, and/or the like) that are present in the 3D space70are determined to not be part of the person10′ and thus are ignored by the robot100(FIG. 1).

Referring now toFIG. 9, components of a robot100are schematically depicted. As noted above, the 3D skeleton representation20′ functionalities described herein are not limited to robotic applications, and may be performed using one or more sensors102and a computing device. It should also be understood that the robot100may include more components and/or alternative components than are illustrated byFIG. 9, and thatFIG. 9is provided for illustrative purposes only. The robot100generally includes a processor110, a communication path111, network interface hardware112, a plurality of sensors102, one or more memory modules114, a plurality of inputs and outputs115, a plurality of actuators116, and a location sensor117.

The communication path111may be formed from any medium that is capable of transmitting a signal such as, for example, conductive wires, conductive traces, optical waveguides, or the like. Moreover, the communication path111may be formed from a combination of mediums capable of transmitting signals. In one embodiment, the communication path111includes a combination of conductive traces, conductive wires, connectors, and buses that cooperate to permit the transmission of electrical data signals to components such as processors, memories, sensors, input devices, output devices, and communication devices. Accordingly, the communication path111may be a bus. Additionally, it is noted that the term “signal” means a waveform (e.g., electrical, optical, magnetic, mechanical or electromagnetic), such as DC, AC, sinusoidal-wave, triangular-wave, square-wave, vibration, and the like, capable of traveling through a medium. The communication path111communicatively couples the various components of the robot100. As used herein, the term “communicatively coupled” means that coupled components are capable of exchanging data signals with one another such as, for example, electrical signals via conductive medium, electromagnetic signals via air, optical signals via optical waveguides, and/or the like.

The processor110of the robot100may be any device capable of executing machine-readable instructions including, but not limited to, machine-readable instructions for generating 3D skeleton representations20′ of people as described herein. Accordingly, the processor110may be a controller, an integrated circuit, a microchip, a computer, or any other computing device. The processor110is communicatively coupled to the other components of the robot100by the communication path111. Accordingly, the communication path111may communicatively couple any number of processors with one another, and allow the components coupled to the communication path111to operate in a distributed computing environment. Specifically, each of the components may operate as a node that may send and/or receive data. While the embodiment depicted inFIG. 9includes a single processor110, other embodiments may include more than one processor, including a plurality of dedicated processors that are each configured to complete a particular task or set of tasks.

The network interface hardware112is coupled to the communication path111and communicatively coupled to the processor110. The network interface hardware112may be any device capable of transmitting and/or receiving data via a network. Accordingly, the network interface hardware112can include a wireless communication module configured as a communication transceiver for sending and/or receiving any wired or wireless communication. For example, the network interface hardware112may include an antenna, a modem, a LAN port, a Wi-Fi card, a WiMax card, an LTE card, mobile communications hardware, near-field communications hardware, satellite communications hardware, and/or any wired or wireless hardware for communicating with other networks and/or devices. In one embodiment, the network interface hardware112may include hardware configured to operate in accordance with a wireless communication protocol, such as, for example, Bluetooth, an 802.11 standard, Zigbee, Z-wave, and the like. For example, the network interface hardware112may include a Bluetooth send/receive module for sending and receiving Bluetooth communications to/from a portable electronic device. The network interface hardware112may also include a radio frequency identification (“RFID”) reader configured to interrogate and read RFID tags. The network interface hardware112may be configured to transmit the 3D skeleton representations20′ to other electronics devices, such as connected mobile devices, displays and other devices to display or otherwise utilize the 3D skeleton representations20′.

The plurality of sensors102may be communicatively coupled to the processor110. The plurality of sensors102may include the RGB and depth sensors described herein, as well as any type of sensors capable of providing the robot100) with information regarding the environment. The plurality of sensors may include, but is not limited to, cameras (e.g., RGB CCD cameras), infrared sensors, depth sensors, proximity sensors, tactile sensors, Lidar sensors, radar sensors, time of flight sensors, inertial measurement units (e.g., one or more accelerometers and gyroscopes), and/or the like. Data from the sensors102are used to develop 3D skeleton representations20′, as described herein.

The memory module114of the robot100is coupled to the communication path111and communicatively coupled to the processor110. The memory module114may comprise RAM, ROM, flash memories, hard drives, or any non-transitory memory device capable of storing machine-readable instructions such that the machine-readable instructions can be accessed and executed by the processor110. The machine-readable instructions may comprise logic or algorithm(s) written in any programming language of any generation (e.g., 1GL, 2GL, 3GL, 4GL, or 5GL) such as, for example, machine language that may be directly executed by the processor, or assembly language, object-oriented programming (OOP), scripting languages, microcode, and the like, that may be compiled or assembled into machine-readable instructions and stored in the memory module114. Alternatively, the machine-readable instructions may be written in a hardware description language (HDL), such as logic implemented via either a field-programmable gate array (FPGA) configuration or an application-specific integrated circuit (ASIC), or their equivalents. Accordingly, the functionality described herein may be implemented in any conventional computer programming language, as pre-programmed hardware elements, or as a combination of hardware and software components. While the embodiment depicted inFIG. 9includes a single memory module114, other embodiments may include more than one memory module. The memory module114may also store sensor data as described herein.

The memory module114stores the machine-readable instructions capable of being executed by the processor to perform the various functionalities described herein. The memory module114also may store the database of registered 3D skeleton representations20′ for user identification purposes as described herein. Other data for generating 3D skeleton representations20′ and other functionalities described herein may also be stored in the memory module114. Further, in some embodiments, data for generating and storing the 3D skeleton representations20′ may be stored remotely, such as on a remote server (not shown).

The input and output devices115may include any number of input devices and output devices. Illustrative input devices include, but are not limited to, keyboards, buttons, switches, knobs, touchpads, touch screens, microphones, infrared gesture sensors, mouse devices, and the like. Illustrative output devices include, but are not limited to, speakers, electronic displays, lights, light emitting diodes, buzzers, tactile displays, and the like.

The plurality of actuators116may include, for example, mechanical actuators that enable the robot to navigate a space and/or manipulate objects. In some embodiments, the actuators116may include motorized wheel assemblies and/or other mobility devices (wings, propellers, rotors, skis, continuous tracks, etc.) that cause the robot to move within a space. Actuators may also include motors or the like that are controllable to move the arms104and the end effectors105of the robot100to grasp and manipulate objects.

The location sensor117is coupled to the communication path111and communicatively coupled to the processor110. The location sensor117may be any device capable of generating an output indicative of a location. In some embodiments, the location sensor117includes a global positioning system (GPS) sensor, though embodiments are not limited thereto. In some embodiments, the location sensor117may be integrated within the network interface hardware112such that the location can be at least partially determined from signals sent and received with the network interface hardware (e.g., use of wifi signal strength to determine distance). Some embodiments may not include the location sensor117, such as embodiments in which the robot100does not determine its location or embodiments in which the location is determined in other ways (e.g., based on information received from other equipment). The location sensor117may also be configured as a wireless signal sensor capable of triangulating a location of the robot100) and the user by way of wireless signals received from one or more wireless signal antennas.

It should be understood that the robot100may include other components not depicted inFIG. 9. For example, the robot100may be powered by a battery. The battery may be any device capable of storing electric energy for later use by the robot100. In some embodiments, the battery is a rechargeable battery, such as a lithium-ion battery or a nickel-cadmium battery. In embodiments in which the battery is a rechargeable battery, the robot100may include the charging port, which may be used to charge the battery.

It should now be understood that embodiments of the present disclosure are configured to generate 3D skeleton representations20′ of people within an environment. In one example a robot includes one or more sensors to generate a 3D skeleton representation20′ of a person to understand where the person is located in 3D space, to assist in path planning and grasp pattern development, person identification, user authentication, and other functionalities. The 3D skeleton representations20′ described herein are created by generating a 2D skeleton representation20from a 2D RGB image. The 2D skeleton representation20is projected into 3D space using depth information from a depth sensor. The RGB sensor and the depth sensor may be separate sensors, or one sensor in a single package.

As a result of the embodiments of the present disclosure, the functionality of the systems that are used to execute the processes described herein is improved because the embodiments described herein allow such systems to more accurately sense the presence of humans in a space, as well as their movement, their poses, and the like. In addition, the systems described herein have improved functionality because such systems are able to authenticate humans without a view of a human's face.