Patent Description:
Pose estimation is a computer vision technique where a human figure is detected from image or video data. In addition to detecting the presence of a human figure, computer vision techniques may further determine the position and orientation of the limbs of the human figure (i.e., the pose).

<CIT> discloses a method that comprises receiving one or more depth images from a depth camera, the depth images indicating a depth of a surface imaged by each pixel of the depth images. The method may further comprise identifying a human subject imaged by the depth images. The collected data may take the form of virtually any suitable data structure(s). A human subject may be continuously observed and modeled. A depth camera and/or cooperating computing system optionally may further analyze the pixels of the depth map of a human subject in order to determine a part of that subject's body that each such pixel is likely to image. A variety of different bodypart assignment techniques may be used to assess which part of a human subject's body a particular pixel is likely to image. Machine-learning can be used to assign each pixel a body part index. The machine-learning approach analyzes a human subject using information learned from a prior-trained collection of known poses. In other words, during a supervised training phase, a variety of different people are observed in a variety of different poses, and human trainers provide ground truth annotations labeling different machine-learning classifiers in the observed data. The observed data and annotations are used to generate one or more machine-learning algorithms that map inputs (e.g., observation data from a tracking device) to desired outputs (e.g., body part indices for relevant pixels).

Pose estimation can be useful in many fields, including autonomous driving. For example, the pose of a person may be used to determine the attention and intention of a human (e.g., a pedestrian, traffic police officer, etc.). Autonomous driving applications for an automobile may use the predicted or inferred intention and attention of a person from the estimated pose to determine driving behaviors.

Systems and methods for pose estimation based on two-dimensional images are disclosed by e.g. <CIT> and <CIT>. Further state of the art is known from <CIT>.

In the examples described below, this application describes techniques and devices for estimating the pose of one or more persons from a point cloud produced by a LiDAR (Light Detection and Ranging) sensor or other similar sensor. In some examples, the estimated pose for the one or more persons may be used to make driving decisions for an autonomous vehicle. However, the techniques of this disclosure are not limited to autonomous driving applications and may be used to estimate the pose of persons for any number of applications where pose estimation may be useful. By using the output of a LiDAR sensor, e.g., as opposed to a camera sensor, pose estimation may be performed quickly in difficult environments, including low-light environments.

A computing system may be configured to receive point cloud data from a LiDAR sensor or other similar sensor. The computing system may be further configured to convert the point cloud data into a structured data format, such as a frame of voxels (volume pixels). The computing system may then process the voxelized frame using a deep neural network. The deep neural network may be configured with a model that determines whether or not a person is present. The deep neural network also may perform a regression to estimate a pose for each of the one or more persons that are detected. In some examples, the computing system makes the determination of a person and the pose estimation serially. That is, in some examples, first the computing system detects a person with the deep neural network and then the computing system estimates the pose of the person using the deep neural network. In other examples, the computing system performs the determination of a person and the pose estimation in parallel. That is, in some examples, the computing system determines the presence of a person and the person's corresponding pose for each voxel at the same time. If the deep neural network determines that a person is not present in the voxel, the computing system discards the estimated pose.

The deep neural network may be configured to process the voxelized frame using one or more three-dimensional (3D) convolutional layers followed by one or more two-dimensional convolutional layers. 3D convolutional layers generally provide for a more accurate determination of a person and pose estimation, while 2D convolutional layers generally provide for a quicker determination of a person and pose estimation. By using a combination of 3D and 2D convolutional layers in the deep neural network, person detection and pose estimation may be performed with a desirable level of accuracy while also maintaining the speed useful for autonomous driving applications.

In another example, this disclosure describes techniques for annotating point cloud data. In order to train a deep neural network to estimate a pose of a person in point cloud data, the deep neural network may be configured and modified through processing of a training set of point cloud data. The training set of point cloud data is previously-labeled with the exact location and poses of persons within the point cloud (e.g., through manual labeling). This previous labeling of poses in the point cloud data may be referred to as annotation. Techniques for annotating human pose in two-dimensional images exist. However, annotating point cloud data is considerably different. For one, point cloud data is three-dimensional. Furthermore, point cloud data is sparse in relation to two-dimensional image data.

This disclosure describes a method, apparatus, and software for annotating point cloud data. A user may use the techniques of this disclosure to annotate point clouds to label one or more poses found in the point cloud data. The annotated point cloud data may then be used to train a neural network to more accurately identify and label poses in point cloud data in real-time.

The invention provides a method and an apparatus as defined in the appended claims.

Pose estimation is a computer vision technique where a human figure is detected from an image or video. In addition to detecting the presence of a human figure, computer vision techniques may further determine the position and orientation of the limbs of the human figure (i.e., the pose). Pose estimation can be useful in many fields, including autonomous driving. For example, the pose of a person may be used to determine the attention and intention of a human (e.g., a pedestrian, traffic cop, etc.) or the needs of a human (e.g., a pedestrian raising an arm to hail a taxi). Autonomous driving applications of an automobile may use the predicted intention and attention of a person from the estimated pose to determine driving behaviors.

In some examples, pose estimation is performed on image data received from a camera sensor. Such data has several drawbacks. For example, if the output from the camera sensor does not include depth information, it may be difficult to discern the relative locations of persons in the image. Even if the output from the camera sensor does include depth information, performing pose estimation may be difficult or impossible in dark environments.

This disclosure describes techniques for performing pose estimation using point cloud data, such as point cloud data produced by a LiDAR sensor. The point cloud output from a LiDAR sensor provides a 3D map of objects in the vicinity of the sensor. As such, depth information is available. In addition, as opposed to a camera sensor, a LiDAR sensor may generate the point clouds in a dark environment. The techniques of this disclosure include processing the point cloud from a LiDAR sensor using a deep neural network to detect the presence of persons near the sensor and to estimate the pose of such persons in order to make autonomous driving decisions.

<FIG> is a conceptual diagram illustrating an example operating environment of the techniques of this disclosure. In one example of the disclosure, automobile <NUM> may include components configured to perform pose estimation. In this example, automobile <NUM> may include a LiDAR sensor <NUM>, a computing system <NUM>, and optionally, a camera(s) <NUM>.

The techniques of this disclosure are described with reference to automotive applications, including autonomous driving applications. However, it should be understood that the techniques of this disclosure for person detection and pose estimation may be used for other contexts.

Automobile <NUM> may be any type of passenger vehicle. LiDAR sensor <NUM> may be mounted to automobile <NUM> using bracket <NUM>. In other examples, LiDAR sensor <NUM> may be mounted to automobile <NUM> in other configurations, or integrated in or carried by structure of the automobile, such as bumpers, sides, windshields, or the like. Additionally, automobile <NUM> may be configured to use a plurality of LiDAR sensors. As will be explained in more detail below, computing system <NUM> may be configured to receive point cloud data from LiDAR sensor <NUM> and determine the location and poses of persons in the field of view of LiDAR sensor <NUM>.

LiDAR sensor <NUM> includes a laser that is configured to emit laser pulses. LiDAR sensor <NUM> further includes a receiver to receive laser light reflected off objects near LiDAR sensor <NUM>. LiDAR sensor <NUM> measures distance to an object by illuminating the object with pulsed laser light and measuring the reflected pulses. Differences in return times and wavelengths of the reflected pulses are used to determine a <NUM>-D representation of one or more objects (e.g., people).

LiDAR sensor <NUM> may further include a global positioning sensor (GPS) or similar sensors to determine the exact physical location of the sensor and objects sensed from the reflected laser light. LiDAR sensor <NUM> may be further configured to detect additional information, like intensity. The intensity of a point in the point cloud may indicate the reflectivity of the object detected by LiDAR sensor <NUM>. Typically, the <NUM>-D representation captured by LiDAR sensor <NUM> is stored in the form of a point cloud. Point clouds are a collection of points that represent a 3D shape or feature. Each point has its own set of X, Y and Z coordinates and in some cases additional attributes (e.g., GPS location and intensity). The resulting point clouds of the LiDAR collection method may be saved and/or transmitted to computing system <NUM>.

While LiDAR sensors are described in this disclosure, the techniques for pose estimation described herein may be used with the output of any sensor that outputs point cloud data. Additional sensor types that may be used with the techniques of this disclosure may include, for example, radar, ultrasonic, camera/imaging sensors, and/or sonar sensors.

Computing system <NUM> may be connected to LiDAR sensor through wired or wireless communication techniques. Computing system may include one or more processors that are configured to receive the point cloud from LiDAR sensor <NUM>. As will be explained in more detail below, computing system <NUM> may be configured to perform pose estimation. For example, computing system <NUM> may be configured to receive the point cloud from LiDAR sensor <NUM>, the point cloud including a plurality of points representing positions of objects relative to the LiDAR sensor, process the point cloud to produce a voxelized frame including a plurality of voxels, process the voxelized frame using a deep neural network to determine one or more persons relative to the LiDAR sensor and a pose for each of the one or more persons, and output a location of the determined one or more persons and the pose for each of the determined one or more persons. The techniques of this disclosure are not limited to the detection and pose estimation for persons (e.g., pedestrians, cyclists, etc.), but may also be used for pose detection of animals (e.g., dogs, cats, etc.).

Bracket <NUM> may include one or more cameras <NUM>. The use of a bracket is just one example. The cameras <NUM> may be positioned in any suitable place on automobile <NUM>. Automobile <NUM> may further include additional cameras not shown in <FIG>. Computing system <NUM> may be connected to cameras <NUM> to receive image data. In one example of the disclosure, computing system <NUM> may be further configured to perform pose estimation using camera-based techniques. In such examples, computing system <NUM> may be configured to estimate the poses of one or more persons using both camera-based techniques and the LiDAR-based techniques described in this disclosure. Computing system <NUM> may be configured to assign weights to each of the poses determined by the camera-based and LiDAR based techniques and determine a final pose of a person based on a weighted average of the determined poses. Computing system <NUM> may be configured to determine the weights based on a confidence level for each of the techniques. For example, LiDAR-based techniques may have a higher confidence of being accurate, compared to camera-based techniques, in low-light environments.

<FIG> is a block diagram illustrating an example apparatus configured to perform the techniques of this disclosure. In particular, <FIG> shows an example of computing system <NUM> of <FIG> in more detail. Again, in some examples, computing system <NUM> may be part of automobile <NUM>. However, in other examples, computing system <NUM> may be a stand-alone system or may be integrated into other devices for use in other applications which may benefit from pose estimation.

Computing system <NUM> includes microprocessor <NUM> in communication with memory <NUM>. In some examples, computing system <NUM> may include multiple microprocessors. Microprocessor <NUM> may be implemented as fixed-function processing circuits, programmable processing circuits, or a combination thereof. Fixed-function circuits refer to circuits that provide particular functionality and are preset on the operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For instance, programmable circuits may execute software or firmware that cause the programmable circuits to operate in the manner defined by instructions of the software or firmware. Fixed-function circuits may execute software instructions (e.g., to receive parameters or output parameters), but the types of operations that the fixed-function processing circuits perform are generally immutable. In some examples, the one or more of the units may be distinct circuit blocks (fixed-function or programmable), and in some examples, the one or more units may be integrated circuits.

In the example of <FIG>, microprocessor <NUM> may be configured to execute one or more sets of instructions in LiDAR-based pose estimation module <NUM> to perform pose estimation in accordance with the techniques of this disclosure. The instructions that define LiDAR-based pose estimation module <NUM> may be stored in memory <NUM>. In some examples, the instructions that define LiDAR-based pose estimation module <NUM> may be downloaded to the memory <NUM> over a wired or wireless network.

In some examples, memory <NUM> may be a temporary memory, meaning that a primary purpose of memory <NUM> is not long-term storage. Memory <NUM> may be configured for shortterm storage of information as volatile memory and therefore not retain stored contents if powered off. Examples of volatile memories include random access memories (RAM), dynamic random-access memories (DRAM), static random-access memories (SRAM), and other forms of volatile memories known in the art.

Memory <NUM> may include one or more non-transitory computer-readable storage mediums. Memory <NUM> may be configured to store larger amounts of information than typically stored by volatile memory. Memory <NUM> may further be configured for long-term storage of information as non-volatile memory space and retain information after power on/off cycles. Examples of non-volatile memories include magnetic hard discs, optical discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. Memory <NUM> may store program instructions (e.g., LiDAR-based pose estimation module <NUM>) and/or information (e.g., point cloud <NUM> and pose and location of detected persons <NUM>) that, when executed, cause microprocessor <NUM> to perform the techniques of this disclosure.

The following techniques of the disclosure will be described with reference to microprocessor <NUM> executing various software modules. However, it should be understood that each of the software modules described herein may also be implemented in dedicated hardware, firmware, software, or any combination of hardware, software, and firmware.

LiDAR-based pose estimation module <NUM> may include a pre-processing unit <NUM>, a deep neural network (DNN) <NUM>, and a post-processing unit <NUM>. LiDAR-based pose estimation module <NUM> is configured to receive point cloud <NUM> from a LiDAR sensor (e.g., LiDAR sensor <NUM> of <FIG>). Pre-processing unit <NUM> is configured to make the unstructured raw input (i.e., point cloud <NUM>) into structuralized frame (e.g., matrix data), so deep neural network <NUM> can process the input data.

Pre-processing unit <NUM> may be configured to process point cloud <NUM> into a structuralized frame in many ways. In one example, pre-processing <NUM> may be configured to convert the point cloud into voxels (volume pixels). Pre-processing unit <NUM> may be configured to perform such voxelization according to a pre-defined data structure for the voxels. For example, each of the voxels may be defined by a size of a three-dimensional (3D) bin (e.g., in terms of X, Y, and Z coordinates), as well as the type of data stored for a 3D bin. For example, each 3D bin (i.e., voxel) may include data indicating the number of points from point cloud <NUM> located in the bin, the location of the points from point cloud <NUM> in the bin, as well as the intensity of such points. Other examples of data that may be stored in the voxels include mean and variance of height, width, length (x, y, z coordinates), mean and variance of intensity/reflectivity, and other statistics of the point cloud within or even neighboring the voxel. In some examples, a voxel may include zero points from point cloud <NUM>, one point from point cloud <NUM>, or multiple points from point cloud <NUM>. Using pre-defined bins may be referred to as manual voxelization. In other examples, pre-processing unit <NUM> may be configured to voxelize point cloud <NUM> in an adaptive manner, e.g., by using a neural network that takes raw point cloud <NUM> as input and outputs a structured (voxelized) frame.

Deep neural network <NUM> receives the voxelized frame from pre-processing unit <NUM>. A deep neural network is a type of machine learning algorithm. Deep neural network <NUM> may be configured with multiple layers of processing layers, each layer configured for determining and/or extracting features from the input data (in this case the voxelized frame of point cloud <NUM>). Each successive layer of deep neural network <NUM> may be configured to use the output from the previous layer as input.

In some examples, deep neural network <NUM> may configured as a convolutional deep neural network. A convolutional deep neural network is a type of deep, feed-forward neural network. Each layer of a convolutional deep neural network may be referred to as a convolutional layer. Convolutional layers apply a convolution operation to the input (e.g., a voxel of the voxelized frame), passing the result to the next layer. Deep neural network <NUM> may be configured with both 3D and 2D convolutional layers. The 3D convolutional layers provide for a more accurate feature extraction (e.g., more accurate identification of persons and corresponding poses), while the 2D convolutional layers provide for a faster feature extraction, as compared to the 3D convolutional layers. Deep neural network <NUM> may be configured to first process the voxelized frame with one or more 3D convolutional layers, and then continue to process the voxelized frame with one or more 2D convolutional layers. The 2D convolutional layers may be configured to only process data from the voxelized frame in the X and Y direction (i.e., not in the Z direction). The number of 3D and 2D convolutional layers, and the division point between the layers determines the tradeoff between speed and accuracy of the pose estimation. By using a combination of 3D and 2D convolutional layers in deep neural network <NUM>, person detection and pose estimation may be performed with a desirable level of accuracy while also maintaining the speed useful for autonomous driving applications.

Deep neural network <NUM> is configured to analyze the voxelized frame and produce two outputs for each of the voxels. One output may be called a classification. The classification indicates whether or not a person is present in the voxel being analyzed. The other output may be called a pose estimation that is produced from a regression. The regression determines the pose of the person (or a key point of a person) if such a person is present in the voxel. As will be explained in more detail below, deep neural network <NUM> may be configured to perform the classification and regression techniques in serial or in parallel.

Deep neural network <NUM> may be configured to process each voxel through DNN model <NUM>. DNN model <NUM> defines the number of 3D and 2D convolutional layers as well as the function performed for each layer. DNN model <NUM> may be trained with a large number of data-label pairs. In the data label-pair, the data is the voxelized point cloud data, while the label is a possible 3D pose. DNN model <NUM> is trained by manually annotating (e.g., labeling) point cloud data, and then training deep neural network <NUM> with the labeled data. The output of deep neural network <NUM> is compared to the expected output given the labeled data. Technicians may then adjust DNN model <NUM> to find an optimal set of weights for the layers of deep neural network <NUM> so that given a pre-annotated point cloud, the desired label is predicted when processed by deep neural network <NUM>. DNN model <NUM> may be predefined and may be periodically updated.

Deep neural network <NUM> may be configured to produce a classification and regression results for each anchor position. In one example, deep neural network may be configured to consider the center of a voxel as an anchor position. For each anchor position, deep neural network <NUM> may be configured to compare the data stored in the voxel to one or more predefined anchor skeletons (also called a standard or canonical skeleton). The anchor skeleton may be defined by a plurality of key points. In one example, anchor skeletons are defined by fourteen joints and/or key points: head, neck, left shoulder, right shoulder, left elbow, right elbow, left hand, right hand, left waist, right waist, left knee, right knee, left foot, and right foot. In general, a key point may correspond to a feature or structure of the human anatomy (e.g., a point on the human body).

During processing by deep neural network <NUM>, an anchor skeleton is activated (i.e., classified as positive for the presence of a person) if the overlapping area between a bounding box of the anchor skeleton and that of any ground truth skeleton (i.e., the data present in the voxel) satisfies a threshold condition. For example, if the overlapping area of the bounding box of the anchor skeleton and the voxel is above a certain threshold (e.g., <NUM>), the anchor skeleton is activated for that voxel and the presence of a person is detected. The threshold may be a measurement of the amount of overlap (e.g., the intersection-over-union (IOU). Deep neural network <NUM> may make the classification based on comparison to one or more multiple different anchor skeletons. Deep neural network <NUM> may also be performed to perform a regression that encodes the difference between an anchor skeleton and the ground truth skeleton (i.e., the data in the actual voxel). Deep neural network <NUM> may be configured to encode this difference for each of a plurality of key points defined for the anchor skeleton. The difference between the key points of the anchor skeleton and the data in the voxel is indicative of the actual pose of the person detected during classification. Deep neural network may then be configured to provide the classification (e.g., a location of the determined one or more persons) and the pose for each of the determined one or more persons to post-processing unit <NUM>. When multiple persons are detected from the point cloud, multiple anchor skeletons will be activated, thus achieving multi-person pose estimation.

Post-processing unit <NUM> may be configured to turn the output of deep neural network <NUM> into final output. For example, post-processing unit <NUM> may be configured to perform non-maximum suppression on the classified and estimated poses produced by deep neural network <NUM> and produce a final location and pose of the persons detected. Non-maximum suppression is an edge thinning technique. In some cases, deep neural network <NUM> will classify persons and estimate poses for many closely spaced groups of voxels where only one person actually exists. That is, in some circumstances, deep neural network will detect overlapping duplicates of the same person. Post-processing unit <NUM> may use non-maximum suppression techniques to remove duplicate skeletons. Post-processing unit <NUM> outputs the pose and location of the detected persons data <NUM>. Pose and location of the detected persons data <NUM> may include the location of a person detected by LiDAR-based pose estimation module <NUM> (e.g., in terms of GPS coordinates) as well as a pose of a skeleton defining the person (e.g., the location of the key points). The pose and location of the detected persons data <NUM> may be stored in memory <NUM>, sent to autonomous driving application <NUM>, other applications <NUM>, camera-based pose estimation application <NUM>, or transmitted from computing system <NUM> to another computing system.

In one example, autonomous driving application <NUM> may be configured to receive pose and location of detected persons data <NUM> and predict or determine the intention and/or attention, or other behavioral cues of the identified persons to make autonomous driving decisions.

In other examples, camera-based pose estimation application <NUM> may receive pose and location of detected persons data <NUM>. Camera-based pose estimation application <NUM> may be configured to determine a pose of one or more persons using image data produced by cameras <NUM> (<FIG>). Camera-based pose estimation application <NUM> may be further configured to assign weights to each of the poses determined by the camera-based and LiDAR based techniques and determine a final pose of a person based on a weighted average of the determined poses. Camera-based pose estimation application <NUM> may be configured to determine the weights based on a confidence level for each of the techniques. For example, LiDAR-based techniques may have a higher confidence of being accurate, compared to camera-based techniques, in low-light environments.

Other applications <NUM> represent various other contexts in which pose and location of detected persons data <NUM> may be used in other contexts. For example, the poses and locations output by LiDAR-based pose estimation module <NUM> may be used in various applications for body language recognition, motion understanding (e.g., traffic, police officers, emergency services personnel, or other personnel signaling/directing traffic), attention and intention detection (e.g., pedestrians waiting/crossing streets), movies, animation, gaming, robotics, human-computer interaction, machine learning, virtual reality, alternative reality, surveillance, abnormal behavior detection, and public security.

<FIG> is a block diagram illustrating a process flow of one example of the disclosure. As shown in <FIG>, LiDAR sensor <NUM> may be configured to capture a point cloud <NUM> that is the raw input to LiDAR-based pose estimation module <NUM>. LiDAR-based pose estimation module <NUM> processes point cloud <NUM> with pre-processing unit <NUM> (voxelization) to produce a voxelized frame. Deep neural network <NUM> then processes the voxelized frame to produce classifications of one or more persons (e.g., the location of one or more persons) as well as the pose or poses for the classified one or more persons. The pose for a person is defined by the locations of a plurality of key points of a skeleton. The output of deep neural network <NUM> are preliminary 3D poses. Post-processing unit <NUM> processes the preliminary 3D poses with a non-maximum suppression algorithm to produce the output 3D poses.

<FIG> is a conceptual diagram illustrating a parallel process flow using a deep neural network according to one example of the disclosure. As shown in <FIG>, point cloud <NUM> is first converted to voxelized frame that includes a plurality of voxels. In this example, deep neural network <NUM> processes each voxel <NUM> of the voxelized frame. Deep neural network <NUM> processes voxel <NUM> using one or more 3D convolutional layers <NUM>. 3D convolutional layer <NUM> represents the last layer that operates on 3D voxel data. After 3D convolutional layer <NUM>, deep neural network <NUM> processes voxel <NUM> with one or more 2D convolutional layers <NUM>. The 2D convolutional layers <NUM> operate on only two dimensions of voxel data (e.g., XY data). 2D convolutional layer <NUM> represents the last 2D convolutional layer which outputs both a classification and a pose estimation. In the example of <FIG>, the layers of deep neural network <NUM> are configured to classify and estimate poses for each of the voxels in parallel. That is, layers of deep neural network <NUM> may be configured to classify and estimate poses for more than one voxel at the same time. If deep neural network <NUM> determines that the voxel is not to be classified as a person, any estimated pose may be discarded.

<FIG> is a conceptual diagram illustrating a sequential process flow using a deep neural network according to one example of the disclosure. In the example of <FIG>, 3D convolutional layers <NUM> and <NUM>, and 2D convolutional layers <NUM> and <NUM> are configured to classify an input voxel as being a person or not. If 2D convolutional layer <NUM> does not classify a person, the process is ended. If 2D convolutional layer <NUM> does classify a person, deep neural network <NUM> will then process the input voxel using 3D convolutional layers <NUM> and <NUM> and 2D convolutional layers <NUM> and <NUM> to estimate a pose of the classified person. That is, deep neural network <NUM> may be configured to use separate neural networks for classification and pose estimation. In this example, the classification and pose estimation processes are performed sequentially.

<FIG> is a conceptual diagram showing an example skeleton. Skeleton <NUM> may represent either a predefined anchor skeleton or the pose of a ground truth skeleton estimated using the techniques of the disclosure described above. In one example of the disclosure, skeleton <NUM> may be defined by a plurality of key points and/or joints. In the example of <FIG>, skeleton <NUM> comprises <NUM> key points. As shown in <FIG>, skeleton <NUM> is defined by head key point <NUM>, neck key point <NUM>, left shoulder key point <NUM>, right shoulder key point <NUM>, left elbow <NUM>, right elbow key point <NUM>, left hand key point <NUM>, right hand key point <NUM>, left waist key point <NUM>, right waist key point <NUM>, left knee key point <NUM>, right knee key point <NUM>, left foot key point <NUM>, and right foot key point <NUM>. To determine a pose, microprocessor <NUM> (see <FIG>) may be configured to determine a location (e.g., a location in 3D space) of each of the key points of skeleton <NUM>. That is, the locations of each of the key points of skeleton <NUM> relative to each other define the pose of the skeleton, and thus the pose of the person detected from the point cloud.

For other applications, more or fewer key points may be used. The more key points that are used to define skeleton <NUM>, the greater number of unique poses that may be estimated. However, more key points may also result in a longer processing time to estimate the pose.

<FIG> is a conceptual diagram showing an example point cloud <NUM> with multiple classified skeletons with estimated poses. As shown in <FIG>, point cloud <NUM> is shown with a visualization of three detected skeletons <NUM>, <NUM>, and <NUM>. The skeletons are shown with different poses that may be seen by the different locations of the <NUM> key points from <FIG>. Note that skeleton <NUM> shows one example of skeleton that has not been processed by a non-maximum suppression algorithm. Rather than showing a single skeleton, skeleton <NUM> is actually multiple overlapping skeletons. In some examples of the disclosure, rather than just outputting data indicating the location and poses of detected persons, computing system <NUM> may be further configured to produce a visualization of the detected skeletons, such as the visualization shown in <FIG>.

<FIG> is a flowchart illustrating an example operation of an apparatus configured to perform pose estimation in accordance with one example of the disclosure. One or more processors may be configured to perform the techniques shown in <FIG>, including microprocessor <NUM> of computing system <NUM>. As described above, in some examples, computing system <NUM> may be part of automobile <NUM>. In this example, automobile <NUM> may be configured to use the pose estimation produced by computing system <NUM> to make autonomous driving decisions. However, the techniques of this disclosure are not so limited. Any processor or processing circuitry may be configured to perform the techniques of <FIG> for pose estimation for any number of applications, including AR/VR, gaming, HCI, surveillance and monitoring, and the like.

In one example of the disclosure, computing system <NUM> may include memory <NUM> configured to receive a point cloud <NUM> (see <FIG>) from LiDAR sensor <NUM> (see <FIG>). Computing system <NUM> may further include one or more processors implemented in circuitry (e.g., microprocessor <NUM> of <FIG>), the one or more processors being in communication with the memory. Microprocessor <NUM> may be configured to receive the point cloud from LiDAR sensor <NUM> (<NUM>). The point cloud includes a plurality of points representing positions of objects relative to LiDAR sensor <NUM>. Microprocessor <NUM> may be further configured to process the point cloud to produce a voxelized frame including a plurality of voxels (<NUM>). In one example of the disclosure, each voxel of the voxelized frame includes a data structure that indicates the presence or absence of points from the point cloud in the voxel.

Microprocessor <NUM> may be further configured to process the voxelized frame using one or more 3D convolutional layers of a deep neural network (<NUM>), and to process the voxelized frame using one or more 2D convolutional layers of the deep neural network (<NUM>). Microprocessor <NUM> processes the voxelized frame using the 3D and 2D convolutional layers to determine one or more persons relative to the LiDAR sensor and a pose for each of the one or more persons. Microprocessor <NUM> may then output a location of the determined one or more persons and the pose for each of the determined one or more persons (<NUM>).

In one example, microprocessor <NUM> may be configured to determine, for a first voxel of the voxelized frame, if a person is present, and activate an anchor skeleton for the first voxel based on the determination, wherein the data represented in the first voxel is defined as a ground truth skeleton. Microprocessor <NUM> may be configured to determine the presence of persons and a pose of such persons either sequentially or in parallel. In one example, microprocessor <NUM> may be configured to determine, in parallel with determining if the person is present, a difference between the ground truth skeleton and the anchor skeleton, estimate a pose of the ground truth skeleton based on the difference, and output the pose in the case that the anchor skeleton is activated. In another example, microprocessor <NUM> may be configured to determine a difference between the ground truth skeleton and the anchor skeleton in the case that the anchor skeleton is activated, estimate a pose of the ground truth skeleton based on the difference, and output the pose.

The anchor skeleton is defined by a plurality of key points. To determine the difference between the ground truth skeleton and the anchor skeleton, microprocessor <NUM> may be configured to determine the difference between the ground truth skeleton and each of the key points of the anchor skeleton.

In another example of the disclosure, microprocessor <NUM> may be further configured to process the determined one or more persons relative to the LiDAR sensor and the pose for each of the one or more persons using a non-maximum-suppression technique to remove duplicates of the one or more persons.

In other examples of the disclosure, the pose estimation techniques of this disclosure may be extended over a series of frames to detect a sequence of poses that may make up a certain action (e.g., waving, walking, running, etc.). Such action recognition may use temporal information (e.g., LiDAR point cloud data from multiple time instances) to perform the action recognition. Accordingly, in one example, DNN <NUM> may be configured to process a plurality of voxelized frames determine at least one person relative to a LiDAR sensor and a sequence of poses for the at least one person. DNN <NUM> may then determine an action for the at least one person from the sequence of poses. Two examples ways of implementing action recognition are described below.

In a first example, DNN <NUM> may be configured to stack and/or concatenate a fixed number of outputs for each frame of point clouds <NUM> into a single data sample. DNN <NUM> may feed the single data sample into a classifier to classify the action category. At frame index t, DNN <NUM> may be configured to use a time window size of w to produce a single sample which is the combined w output from frame t-w+<NUM> to t. DNN <NUM> may be configured to include a classifier that is a (multi-class) deep neural network or any type of machine learning model, such as a support-vector machine (SVM).

In another example, DNN <NUM> may be configured to use a per frame output in a sequential manner. For example, DNN <NUM> may be configured to feed the per frame output into a Recursive Neural Network, and determine a prediction of the action either at every frame or after a certain number of frames.

So, instead of or in addition to per frame pose estimation (e.g., the skeleton output), DNN <NUM> may be configured to either stitch outputs as a batch or feed the outputs sequentially to obtain a higher level of action recognition. Some of the possible categories for actions to recognize may include standing, walking, running, biking, skateboarding, waving, etc..

Additional examples and combinations of techniques of the disclosure are described below.

According to an embodiment, the method for pose estimation may comprise: receiving a point cloud from a LiDAR sensor, the point cloud including a plurality of points representing positions of objects relative to the LiDAR sensor; processing the point cloud to produce a voxelized frame including a plurality of voxels; processing the voxelized frame using a deep neural network to determine one or more persons relative to the LiDAR sensor and a pose for each of the one or more persons; and outputting a location of the determined one or more persons and the pose for each of the determined one or more persons.

According to an embodiment, each voxel of the voxelized frame may include a data structure that indicates the presence or absence of points from the point cloud in the voxel.

According to an embodiment, processing the voxelized frame using the deep neural network may comprise: processing the voxelized frame using a convolutional deep neural network, wherein the convolutional deep neural network includes one or more three-dimensional convolutional layers followed by one or more two-dimensional convolutional layers.

According to an embodiment, processing the voxelized frame using the deep neural network may comprise: determining, for a first voxel of the voxelized frame, if a person is present; and activating an anchor skeleton for the first voxel based on the determination, wherein the data represented in the first voxel is defined as a ground truth skeleton.

According to an, the method may further comprise: determining, in parallel with determining if the person is present, a difference between the ground truth skeleton and the anchor skeleton; estimating a pose of the ground truth skeleton based on the difference; and outputting the pose in the case that the anchor skeleton is activated.

According to an embodiment, the method may further comprise: determining a difference between the ground truth skeleton and the anchor skeleton in the case that the anchor skeleton is activated; estimating a pose of the ground truth skeleton based on the difference; and outputting the pose.

According to an embodiment, the anchor skeleton may be defined by a plurality of key points.

According to an embodiment, determining the difference between the ground truth skeleton and the anchor skeleton may comprise: determining the difference between the ground truth skeleton and each of the key points of the anchor skeleton.

According to an embodiment, the method may further comprise: processing the determined one or more persons relative to the LiDAR sensor and the pose for each of the one or more persons using a non-maximum-suppression technique to remove duplicates of the one or more persons.

According to an embodiment, the apparatus configured to perform pose estimation may comprise: a memory configured to receive a point cloud from a LiDAR sensor; and one or more processors implemented in circuitry, the one or more processors in communication with the memory and configured to: receive the point cloud from the LiDAR sensor, the point cloud including a plurality of points representing positions of objects relative to the LiDAR sensor; process the point cloud to produce a voxelized frame including a plurality of voxels; process the voxelized frame using a deep neural network to determine one or more persons relative to the LiDAR sensor and a pose for each of the one or more persons; and output a location of the determined one or more persons and the pose for each of the determined one or more persons.

According to an embodiment, each voxel of the voxelized frame includes a data structure that indicates the presence or absence of points from the point cloud in the voxel.

According to an embodiment of the apparatus, to process the voxelized frame using the deep neural network, the one or more processors may be further configured to: process the voxelized frame using a convolutional deep neural network, wherein the convolutional deep neural network includes one or more three-dimensional convolutional layers followed by one or more two-dimensional convolutional layers.

According to an embodiment, to process the voxelized frame using the deep neural network, the one or more processors may be further configured to: determine, for a first voxel of the voxelized frame, if a person is present; and activate an anchor skeleton for the first voxel based on the determination, wherein the data represented in the first voxel is defined as a ground truth skeleton.

According to an embodiment, the one or more processors may be further configured to: determine, in parallel with determining if the person is present, a difference between the ground truth skeleton and the anchor skeleton; estimate a pose of the ground truth skeleton based on the difference; and output the pose in the case that the anchor skeleton is activated.

According to an embodiment, the one or more processors may be further configured to: determine a difference between the ground truth skeleton and the anchor skeleton in the case that the anchor skeleton is activated; estimate a pose of the ground truth skeleton based on the difference; and output the pose.

According to an embodiment, to determine the difference between the ground truth skeleton and the anchor skeleton, the one or more processors may be further configured to: determine the difference between the ground truth skeleton and each of the key points of the anchor skeleton.

According to an embodiment, the one or more processors may be further configured to: process the determined one or more persons relative to the LiDAR sensor and the pose for each of the one or more persons using a non-maximum-suppression technique to remove duplicates of the one or more persons.

According to an embodiment, the apparatus may comprise an automobile that includes the LiDAR sensor.

According to an embodiment, the apparatus configured to perform pose estimation, may comprise: means for receiving a point cloud from a LiDAR sensor, the point cloud including a plurality of points representing positions of objects relative to the LiDAR sensor; means for processing the point cloud to produce a voxelized frame including a plurality of voxels; means for processing the voxelized frame using a deep neural network to determine one or more persons relative to the LiDAR sensor and a pose for each of the one or more persons; and means for outputting a location of the determined one or more persons and the pose for each of the determined one or more persons.

According to an embodiment, the apparatus configured to perform pose estimation may comprise means for performing any combination of steps in the processes as described above.

According to an embodiment, a non-transitory computer-readable medium may be configured to store instructions that, when executed, causes one or more processors to receive a point cloud from a LiDAR sensor, the point cloud including a plurality of points representing positions of objects relative to the LiDAR sensor, process the point cloud to produce a voxelized frame including a plurality of voxels, process the voxelized frame using a deep neural network to determine one or more persons relative to the LiDAR sensor and a pose for each of the one or more persons, and output a location of the determined one or more persons and the pose for each of the determined one or more persons.

This disclosure describes a method, apparatus, and software tool for annotating point cloud data. A user may use the techniques of this disclosure to annotate point clouds to label one or more poses found in the point cloud data. The annotated point cloud data may then be used to train a neural network to more accurately identify and label poses in point cloud data in real-time.

<FIG> is a block diagram illustrating an example computing system <NUM> configured to perform the point cloud annotation techniques of this disclosure. Computing system <NUM> can be implemented with, for example, a desktop computer, notebook computer, tablet computer, or any type of computing device. Computing system <NUM> includes processor <NUM>, a memory <NUM>, and one or more input devices <NUM>. In some examples, computing system <NUM> may include multiple processors <NUM>.

Processor <NUM> may be implemented as fixed-function processing circuits, programmable processing circuits, or a combination thereof. Fixed-function circuits refer to circuits that provide particular functionality and are preset on the operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For instance, programmable circuits may execute software or firmware that cause the programmable circuits to operate in the manner defined by instructions of the software or firmware. Fixed-function circuits may execute software instructions (e.g., to receive parameters or output parameters), but the types of operations that the fixed-function processing circuits perform are generally immutable. In some examples, the one or more of the units may be distinct circuit blocks (fixed-function or programmable), and in some examples, the one or more units may be integrated circuits.

Computing system <NUM> may be configured to generate information for display on display device <NUM>. For example, as will be described in more detail below, computing system <NUM> may generate a graphical user interface (GUI) <NUM> and cause GUI <NUM> to be displayed on display device <NUM>. A user may interact with GUI <NUM>, e.g., through input device(s) <NUM> to annotate point cloud data. In some examples, display device <NUM> is part of computing system <NUM>, and in other examples, display device <NUM> may be separate from computing system <NUM>. Display device <NUM> can be implemented with any electronic display, for example a liquid crystal display (LCD), a light emitting diode (LED) display, or an organic light emitting diode (OLED) display.

Input devices <NUM> are devices configured to receive user commands or other information. In some examples, input devices <NUM> are part of computing system <NUM>, and in other examples, input devices <NUM> may be separate from computing system <NUM>. Input devices <NUM> may include any device for entering information or commands, for example a keyboard, microphone, cursor-control device, or touch screen.

In accordance with the techniques of this disclosure, processor <NUM> may be configured to execute a set of instructions of annotation tool <NUM> to perform point cloud annotation in accordance with the techniques of this disclosure. The instructions that define annotation tool <NUM> may be stored in memory <NUM>. In some examples, the instructions that define annotation tool <NUM> may be downloaded to the memory <NUM> over a wired or wireless network.

In some examples, memory <NUM> may be a temporary memory, meaning that a primary purpose of memory <NUM> is not long-term storage. Memory <NUM> may be configured for short-term storage of information as volatile memory and therefore not retain stored contents if powered off. Examples of volatile memories include random access memories (RAM), dynamic random-access memories (DRAM), static random-access memories (SRAM), and other forms of volatile memories known in the art.

Memory <NUM> may include one or more non-transitory computer-readable storage mediums. Memory <NUM> may be configured to store larger amounts of information than typically stored by volatile memory. Memory <NUM> may further be configured for long-term storage of information as non-volatile memory space and retain information after power on/off cycles. Examples of non-volatile memories include magnetic hard discs, optical discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. Memory <NUM> may store program instructions (e.g., annotation tool <NUM>) and/or information (e.g., point cloud training data <NUM> and annotated point clouds <NUM>) that, when executed, cause processor <NUM> to perform the techniques of this disclosure.

The following techniques of the disclosure will be described with reference to processor <NUM> executing various software modules. However, it should be understood that each of the software modules described herein may also be implemented in dedicated hardware, firmware, software, or any combination of hardware, software, and firmware.

In accordance with the techniques of this disclosure, processor <NUM>, through the execution of annotation tool <NUM> may be configured to load point cloud training data <NUM>. Point cloud training data <NUM> may include one or more frames of point cloud data, e.g., point cloud data captured by a LiDAR sensor or any other type of sensor that captures point cloud data. Annotation tool <NUM> may be configured to generate a GUI <NUM> that includes one or more frames of point cloud training data <NUM> and cause display device <NUM> to display GUI <NUM>. A user may then interact with GUI <NUM> to annotate human poses onto a frame of point cloud training data <NUM> to define a pose of human that may be present in the point cloud data. After annotation, annotation tool <NUM> may be configured to output annotated point clouds <NUM>. The annotated point clouds <NUM> may be stored in memory <NUM> and/or offloaded outside of computing system <NUM>. Annotated point clouds <NUM> may then be used to train a deep neural network configured to estimate human poses from point cloud data (e.g., deep neural network <NUM> of <FIG>). Deep neural network <NUM> of <FIG> represents a version of deep neural network <NUM> before and/or during the training process. <FIG> illustrate various examples of GUI <NUM> that may be generated by annotation tool <NUM>. The operation of annotation tool <NUM> relative to the generated GUI <NUM> and user inputs will be discussed in more detail below.

<FIG> is a conceptual user interface diagram showing an input point cloud for annotation. Annotation tool <NUM> may cause display device <NUM> to display a GUI <NUM> that includes point cloud frame <NUM>. For example, annotation tool <NUM> may generate a GUI <NUM> in response to a user input to load point cloud frame <NUM> of point cloud training data <NUM> (see <FIG>). Annotation tool <NUM> may be configured to load and display point cloud frame <NUM> in response to a user interaction with import data controls <NUM>. As shown in <FIG>, point cloud frame <NUM> may include points <NUM> that may correspond to one or more persons captured in the point cloud data.

Import data controls <NUM> include a load match button, a load cloud(s) button, and a load Skel(s) (skeletons) button. When a user selects the load match button, annotation tool <NUM> opens a file explorer dialog box and reads the user selected file which contains a matching pairs of images and a point clouds from their corresponding directories. In this example, annotation tool <NUM> is able to load files from directories that include both an image and a matching point cloud. In this way, a paired image and point cloud may be viewed at the same time. In other examples, annotation tool <NUM> may only load one of an image or a point cloud.

When a user selects the load cloud(s) button, annotation tool <NUM> opens a file explorer dialog box and populates a list with the available point cloud files in the user selected directory. The user can then select which point cloud to view from the populated dropdown list.

When a user selects the load skel(s), annotation tool <NUM> opens a file explorer dialog box and loads any previously-annotated skeletons from a user selected file. The user can then edit any previously-annotated skeletons.

In some examples, annotation tool <NUM> may be configured to load multiple frames of point cloud training data <NUM> in response to a user input. In accordance with the invention, annotation tool initially displays a single point cloud frame <NUM> of the multiple frames. Annotation tool <NUM> further generates one or more video controls <NUM> that cause annotation tool <NUM> to display each of the multiple frames of point cloud data in sequence (e.g., like a video). In the example of <FIG>, video control buttons <NUM> include, according to the invention, a play and stop button, though further controls are provided according to the invention (pause, rewind fast forward, and possibly others).

Annotation tool <NUM> may further include edit data controls <NUM> in GUI <NUM>. A user may interact with edit data controls <NUM> to change the amount or region of data of point cloud frame <NUM> that is displayed in GUI <NUM>. A user may specify the region of point cloud frame <NUM> to be displayed by specifying minimum (min) and maximum (max) dimensions in a horizontal direction (x-lims), in a vertical direction (y-lims), and in a depth direction (z-lims). Annotation tool <NUM> may be configured to crop the point cloud frame <NUM> in response to user inputs in edit data controls <NUM> and display the cropped region of point cloud frame <NUM>. Edit data controls <NUM> may further include a rotate (rot) button that changes the perspective angle at which point cloud frame <NUM> is viewed. The size button in edit data controls <NUM> changes the size of each point in the point cloud.

In the above example, a user, through manipulation of edit data controls <NUM>, may use annotation tool <NUM> to manually crop a region of point cloud frame <NUM> to display. Cropping point cloud frame <NUM>, e.g., around one or more potential persons in the data, may make annotating point cloud frame <NUM> easier. Rather than having a user manually crop point cloud frame <NUM>, in other examples, annotation tool <NUM> may be configured to automatically identify regions of interest in point cloud frame <NUM> and automatically crop point cloud frame <NUM> to show only the identified regions of interest. In one example, annotation tool <NUM> may be configured to identify regions of interest by detecting which regions of point cloud frame <NUM> include data indicative of persons on which a pose may be annotated. In one example, annotation tool <NUM> may provide point cloud frame <NUM> to deep neural network <NUM> (<FIG>) in order to identify the regions of interest. Deep neural network <NUM> may be a deep neural network that is configured to identify persons and estimate poses from point cloud data in the same manner as deep neural network <NUM> (see <FIG>) described above. Deep neural network <NUM> may be a completed deep neural network configured for pose estimation or may be a deep neural network that is being trained with annotated point clouds <NUM> (see <FIG>) produced by annotation tool <NUM>. Deep neural network <NUM> may provide an indication of a region of interest to annotation tool <NUM> and annotation tool <NUM> may crop point cloud frame <NUM> of or around the indicated region of interest.

<FIG> is a conceptual user interface diagram showing a cropped point cloud for annotation. In <FIG>, annotation tool <NUM> displays a cropped region of point cloud frame <NUM> in top view window <NUM>, side view window <NUM>, and front view window <NUM>, via GUI <NUM>. In top view window <NUM>, annotation tool <NUM> displays the cropped region of point cloud frame <NUM> from directly overhead. In side view window <NUM>, annotation tool <NUM> displays the cropped region of point cloud frame <NUM> from an angle (e.g., <NUM> degrees) from a pre-defined front angle. In front view window <NUM>, annotation tool <NUM> displays the cropped region of point cloud frame <NUM> from the pre-defined front angle. In other examples, annotation tool <NUM> may display the cropped region of point cloud frame <NUM> from more or fewer angles and/or perspectives, including isometric perspectives. Additionally, annotation tool <NUM> is not limited to displaying cropped regions of point cloud frame <NUM> at different angles. Annotation tool <NUM> may also display the entirety of cropped frame <NUM> at various angles.

Returning to <FIG>, annotation tool <NUM> may further include annotate controls <NUM> (e.g., the button labeled "Skeletons"). When a user selects the "Skeletons" button of annotate controls <NUM>, annotation tool <NUM>, in accordance with the invention, labels points in point cloud frame <NUM> with a plurality of annotation points. That is, annotation tool <NUM> overlays a skeleton that is defined by a plurality of annotation points, where the plurality of annotation points corresponds to points on a human body.

<FIG> is a conceptual diagram showing an example skeleton for annotation. Skeleton <NUM> represents an example skeleton that annotation tool <NUM> may use to label points of point cloud frame <NUM>. In the example of <FIG>, skeleton <NUM> is defined by <NUM> annotation points, each of which corresponds to a human joint or other human anatomy. In other examples, skeleton <NUM> may include more or fewer annotation points. As will be explained below, a user may manipulate and move one or more of the annotation points of skeleton <NUM> to define a pose of a person represented by the points in point cloud frame <NUM>.

In <FIG>, skeleton <NUM> is facing toward the viewer. As such, the "right-side" limbs are shown on the left side of <FIG>. Skeleton <NUM> is defined by top of head annotation point <NUM> (<NUM>), center of neck annotation point <NUM> (<NUM>), left shoulder annotation point <NUM> (<NUM>), right shoulder annotation point <NUM> (<NUM>), left elbow annotation point <NUM> (<NUM>), right elbow annotation point <NUM> (<NUM>), left hand annotation point <NUM> (<NUM>), right hand annotation point <NUM> (<NUM>), left hip annotation point <NUM> (<NUM>), right hip annotation point <NUM> (<NUM>), left knee annotation point <NUM> (<NUM>), right knee annotation point <NUM> (<NUM>), left foot annotation point <NUM> (<NUM>), and right foot annotation point <NUM> (<NUM>). The numbers in parentheses next to reference numerals of the annotation points relate to selection buttons of annotation tool <NUM> that will be shown in <FIG> and <FIG>.

In addition to displaying the various annotation points of skeleton <NUM>, annotation tool <NUM> may also display lines between the annotation points to define limbs and/or major body parts of skeleton <NUM>. For example, annotation tool <NUM> may display a line between top of head annotation point <NUM> and center of neck annotation point <NUM> to define a head. Annotation tool <NUM> may display a line extending from center of neck annotation point <NUM>, through right shoulder annotation point <NUM> and right elbow annotation point <NUM>, and ending at right hand annotation point <NUM> to define a right arm. Annotation tool <NUM> may display a line extending from center of neck annotation point <NUM>, through left shoulder annotation point <NUM> and left elbow annotation point <NUM>, and ending at left hand annotation point <NUM> to define a left arm. Annotation tool <NUM> may display a line from center of neck annotation point <NUM>, to left hip annotation point <NUM>, to right hip annotation pint <NUM>, and back to center of neck annotation point <NUM> to define a body. Annotation tool <NUM> may display a line from right hip annotation point <NUM>, to right knee annotation point <NUM>, and ending at right foot annotation point <NUM> to define a right leg. Annotation tool <NUM> may display a line from left hip annotation point <NUM>, to left knee annotation point <NUM>, and ending at left foot annotation point <NUM> to define a right leg.

As shown in <FIG>, annotation tool <NUM> may use different line weights and/or dash types for the different limbs. In this way, a user may more easily distinguish the limbs of skeleton <NUM> from each other in order to select the appropriate annotation point for manipulation. In other examples, rather than use line weights or dash types, annotation tool <NUM> may use different colors to distinguish the different limbs. For example, the lines between annotation points defining the head may be blue, the lines between annotation points defining the right arm may be green, the lines between annotation points defining the left arm may be red, the lines between annotation points defining the body may be yellow, the lines between annotation points defining the right leg may be magenta, and the lines between annotation points defining the left leg may be cyan. Of course, other colors may be used.

<FIG> is a conceptual user interface diagram showing an estimated annotation of a point cloud. In <FIG>, in accordance with the invention annotation tool <NUM> labels points in point cloud frame <NUM> with the annotation points of skeleton <NUM>. Annotation tool <NUM> shows skeleton <NUM> in each of top view <NUM>, side view <NUM>, and front view <NUM>, in GUI <NUM>. Initially, annotation tool <NUM> may display skeleton <NUM> in a default position and in a default pose. As can be seen in <FIG>, the default position of skeleton <NUM> does not match the actual pose of the person depicted in the point cloud data of point cloud frame <NUM>. A user may manipulate and/or move one or more of the annotation points of skeleton <NUM> to match the actual pose of one or more persons present in the data of point cloud frame <NUM> to produce an annotated point cloud.

In the example of <FIG>, annotation tool <NUM> displays a single skeleton <NUM>. In other examples, annotation tool <NUM> may generate and display multiple skeletons if there are multiple poses to be annotated in point cloud frame <NUM>. In one example, annotation tool <NUM> may position skeleton <NUM> in a default position, e.g., in the center of each view. In other examples, a user may manually position skeleton <NUM> (e.g., through moving all annotation points together) into a position. In other examples, annotation tool <NUM> may automatically determine a location of one or more persons in point cloud frame <NUM> and position skeleton <NUM> at the automatically determined location. In one example, annotation tool <NUM> may provide point cloud frame <NUM> to deep neural network <NUM>. Deep neural network <NUM> may be a deep neural network that is configured to estimate poses in the same manner as deep neural network <NUM> (see <FIG>) described above. Deep neural network <NUM> may be a completed deep neural network configured for pose estimation or may be a deep neural network that is being trained with annotated point clouds <NUM> (see <FIG>) produced by annotation tool <NUM>. Deep neural network <NUM> may determine the locations of persons in point cloud frame <NUM> and indicate the position or positions of such person(s) to annotation tool <NUM>. Annotation tool <NUM> may then display the default skeleton <NUM> at the position indicated by deep neural network <NUM>.

Annotation tool <NUM> may also display skeleton <NUM> having a default pose. That is, annotation tool <NUM> may display the annotation points of skeleton in a default orientation relative to each other. <FIG> shows one example of a default pose of skeleton <NUM>. Of course, annotation tool <NUM> may generate other default poses. In other examples, rather than using a default pose, annotation tool <NUM> may estimate a position and pose of a person in point cloud frame <NUM>. In one example, annotation tool <NUM> may provide point cloud frame <NUM> to deep neural network <NUM>. Deep neural network <NUM> may be a deep neural network that is configured to estimate poses in the same manner as deep neural network <NUM> (see <FIG>) described above. Deep neural network <NUM> may be a completed deep neural network configured for pose estimation or may be a deep neural network that is being trained with annotated point clouds <NUM> (see <FIG>) produced by annotation tool <NUM>. Deep neural network <NUM> may determine an estimated position and pose of persons found in point cloud frame <NUM> and indicate the position and pose of such persons to annotation tool <NUM>.

Annotation tool <NUM> may then display the default skeleton <NUM> at the position and pose indicated by deep neural network <NUM>. Such estimated poses will often not be completely accurate. However, the estimated pose produced by deep neural network <NUM> may be closer to the actual pose found in the point cloud data than a default pose. Accordingly, a user of annotation tool <NUM> may start with a pose that is closer to the actual pose to be annotated, thus making the manual process for annotating the pose easier and faster.

Annotation tool <NUM> may provide several different tools for a user to manipulate skeleton <NUM> into a position representing the actual pose the person found in point cloud frame <NUM>. Annotation tool <NUM> may generate a select region button <NUM>. A user may activate select region button <NUM> and then select which of top view <NUM>, side view <NUM>, or front view <NUM> that the user will interact with. Manipulation of skeleton <NUM> may be easier in different views depending on the position and orientation of persons captured in point cloud frame <NUM>. Annotation point controls <NUM> allow a user to select particular annotation points to control. The numbers <NUM>-<NUM> of the checkboxes in annotation point controls <NUM> correspond to the numbers in parenthesis of the annotation points shown in <FIG>. A user may select one or more of the annotation points by selecting a corresponding checkbox. Annotation tool <NUM> then moves, in response to a user input, one or more of the selected annotation points to define a human pose and create annotated point cloud data <NUM> (see <FIG>).

In one example, annotation tool <NUM> may move a selected annotation point in response to a user interaction with a mouse (e.g., click and drag). In other examples, annotation tool <NUM> may move one or more of the annotation points in response to user interaction with rotation control <NUM> and/or position control <NUM>. In the example of <FIG>, rotation control <NUM> is a slider control that causes annotation tool <NUM> to rotate all annotation points of skeleton <NUM> around the top of head annotation point. Position control <NUM> includes individual sliders for each of the horizontal (X), vertical (Y), and depth (Z) dimensions of point cloud frame <NUM>. In response to a user moving the sliders, annotation tool <NUM> moves the selected annotation points in each of the specified dimensions within point cloud frame <NUM>. While the example of <FIG> shows slider controls, other control types may be used, including text entry of specific coordinates (e.g., (X,Y,Z,) coordinates) within point cloud frame <NUM>.

In one example of the disclosure, annotation tool <NUM> may assign a unique identifier to each annotation point in a skeleton. In this way, the pose and annotation points of a particular skeleton may be tracked across multiple frames. Furthermore, in conjunction with the action recognition techniques discussed above, annotation tool <NUM> may also include an action category label per frame and per skeleton (e.g., human pose).

In one example of the disclosure, to increase precision of positioning annotation points, annotation tool <NUM> may be configured to only allow the selection of a single annotation point at a time. Once the single annotation point is selected, a user may only cause annotation tool <NUM> to move the single selected annotation point. For example, once one of the checkboxes annotation points controls <NUM> is selected, annotation tool <NUM> may make the other checkboxes unavailable for selection. To move other annotation points, a user may first deselect the selected annotation point. While moving a single selected annotation point, annotation tool <NUM> leaves all other annotation points stationary. In other examples, annotation tool <NUM> may allow for the selection of multiple annotation points. In this example, annotation tool <NUM> will only move the selected annotation points (e.g., one or multiple selected annotation points) in response to user input. The non-selected annotation points will remain stationary.

That is, in one example, annotation tool <NUM> receives a selection of a single annotation point of the plurality of annotation points, and moves, in response to the user input, only the single selected annotation point to define a portion of the human pose. In other examples, annotation tool <NUM> receives a selection of two or more annotation points of the plurality of annotation points, and moves, in response to the user input, only the two or more selected annotation points to define a portion of the human pose. A user may type notes regarding the annotated point cloud frame <NUM> in annotation window <NUM>. Once annotation is completed, a user may active save controls <NUM> to save the final position of each annotation point. The annotated point cloud <NUM> (see <FIG>) may then be saved in memory (e.g., in a. json file) and then used for training a neural network (e.g., neural network <NUM>).

<FIG> is a conceptual user interface diagram showing an annotated point cloud. <FIG> shows the pose of skeleton <NUM> from <FIG> after manipulation by a user. As can be seen in <FIG>, the annotation points of skeleton <NUM> have been moved into a position (i.e., a pose) that more closely matches the actual pose of the person capture in point cloud <NUM>.

<FIG> is a flowchart illustrating an example operation an annotation tool in accordance with one example of the disclosure. The techniques in <FIG> may be performed by processor <NUM> executing instructions for annotation tool <NUM> (see <FIG>). The techniques of <FIG> depict one example process for annotating a single frame of point cloud data. The process of <FIG> may be repeated for multiple frames of point cloud data.

Annotation tool <NUM> may load and display point cloud data (<NUM>). For example, annotation tool <NUM> may load point cloud data from a point cloud file (e.g., point cloud training data <NUM> of <FIG>), and causes display device <NUM> to display a GUI <NUM> that includes the loaded point cloud data. Annotation tool <NUM> may then determine, either automatically or through a user input, whether or not to crop the point cloud (<NUM>). If yes, annotation tool <NUM> displays the cropped point cloud at one or more perspectives (<NUM>). Annotation tool <NUM> labels points in the displayed point cloud (e.g., the cropped point cloud) with annotation points (<NUM>). If the point cloud data is not cropped (<NUM>), annotation tool <NUM> also labels points in the displayed point cloud (e.g., the entire point cloud) with annotation points (<NUM>).

Annotation tool <NUM> then waits until an annotation point is selected (<NUM>). Once an annotation point is selected, annotation tool <NUM> will then move the selected annotation point in response to a user input (<NUM>). Annotation tool <NUM> will then check for a user input indicating that the annotation is complete (<NUM>). If yes, annotation tool <NUM> outputs the annotated point cloud (<NUM>). If no, annotation tool <NUM> will wait for another annotation point to be selected (<NUM>) and then repeat the move process (<NUM>) in response to user inputs.

It is to be recognized that acts or events of any of the techniques described herein may be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the techniques). Moreover, acts or events may be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially.

By way of example, and not limitation, such computer-readable data storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), or other equivalent integrated or discrete logic circuitry.

Claim 1:
A method comprising:
causing, by one or more processors (<NUM>), display (<NUM>) of a single frame (<NUM>) of multiple frames of point cloud data (<NUM>), and of one or more video control buttons (<NUM>) for causing a video display of the multiple frames of point cloud data in sequence, wherein the video control buttons include buttons to control play, stop, pause, rewind and fast forward functions;
labeling (<NUM>), by the one or more processors, points in the frame of the point cloud data with a plurality of annotation points (<NUM>-<NUM>),
the plurality of annotation points corresponding to points on a human body and defining a skeleton (<NUM>) overlaid on the point cloud data;
moving (<NUM>), by the one or more processors, and in response to a user input, one or more of the annotation points to define a human pose and create annotated point cloud data (<NUM>), wherein the human pose is the position and orientation of the limbs of a human; and
outputting (<NUM>), by the one or more processors, the annotated point cloud data (<NUM>) as data for training a neural network (<NUM>, <NUM>).