Patent Description:
The background description herein is provided for the purpose of generally presenting the context of the disclosure.

Multi-object pose tracking is the task of estimating human keypoints of all object instances in a set of images forming a video sequence and assigning unique ids for each keypoint at instance-level across frames in the video sequence. Multi-object pose tracking is one of the most important components in various computer vision applications, such as surveillance, sports video analysis.

Existing approaches for pose estimation of multiple objects can be classified into top-down and bottom-up approaches. The top-down approach [<NUM>, <NUM>] relies on a detection module to obtain human candidates and then apply a single-person human pose estimator to locate human keypoints. The bottom-up approach [<NUM>, <NUM>,<NUM>,<NUM>], on the other hand, detects human keypoints from all potential human candidates and then assembles these keypoints into human limbs for each individual based on various data association techniques. The advantage of bottom-up approaches is their excellent trade-off between estimation accuracy and computational cost because their computational cost is invariant to the number of human candidates in the image. In contrast, the main advantage of top-down approaches is their capability in disassembling the task into multiple comparatively easier tasks, i.e., object detection and single-person pose estimation. The object detector is expert in detecting hard (usually small) candidates, so that the pose estimator will perform better with a focused regression space.

For the task of multi-object pose tracking, accurate trajectory estimation of human keypoints is useful in human action recognition and human interaction understanding. Kocabas et al. [<NUM>] disclose MultiPoseNet, a bottom-up multi-person pose estimation architecture that combines a multi-task model with an assignment method. MultiPoseNet can jointly handle person detection, keypoint detection, person segmentation and pose estimation problems. The assignment method is implemented by the Pose Residual Network (PRN) which receives keypoint and person detections, and produces poses by assigning keypoints to person instances. PoseTrack [<NUM>] and ArtTrack [<NUM>] primarily introduce multi-object pose tracking challenge and propose a graph partitioning formulation, which transforms the pose tracking problem into a minimum cost multi-cut problem. However, hand-crafted graphical models are not scalable for long and unseen clips. Another line of research explores top-down approach [<NUM>, <NUM>, <NUM>] by operating multi-object pose estimation on each frame and linking them based on appearance similarities and temporal adjacencies. A solution is to apply multi-target object tracking on human detection candidates across frames and then estimate human poses for each human tubelet. While this is a feasible method, it neglects unique attributes of keypoints. Compared to the tracked bounding boxes, keypoints can potentially be helpful cues for both the bounding boxes and the keypoints tracking.

Therefore, there is a need for an improved approach to multi-object pose tracking.

In some aspects, the present disclosure relates to a method of tracking poses of multiple objects. The method includes detecting, by one or more processors, objects from each of a plurality of consecutive frames in a video sequence; estimating, by the one or more processors, a pose of each of the objects within each of the plurality of consecutive frames; and tracking, by the one or more processors, the poses of each of the objects across the plurality of consecutive frames.

The step of detecting, by one or more processors, objects from each of a plurality of consecutive frames in a video sequence includes determining, by the one or more processors, a bounding box for each of the objects from each of the plurality of consecutive frames; and determining, by the one or more processors, a confidence score of the bounding box for the one of the objects within the one of the plurality of consecutive frames.

The step of estimating, by the one or more processors, a pose of each of the objects within each of the plurality of consecutive frames includes determining, by the one or more processors, from a region defined by the bounding box for the one of the objects within the one of the plurality of consecutive frames, keypoints of the one of the objects within the one of the plurality of consecutive frames; and determining, by the one or more processors, confidence scores of the keypoints of the one of the objects within the one of the plurality of consecutive frames.

In some embodiments, the step of estimating, by the one or more processors, a pose of each of the objects within each of the plurality of consecutive frames may include: regressing, by the one or more processors, a region defined by the bounding box for the one of the objects within the one of the plurality of consecutive frames into heatmaps of the one of the objects within the one of the plurality of consecutive frames; suppressing, by the one or more processors, the heatmaps of the one of the objects within the one of the plurality of consecutive frames into keypoints of the one of the objects within the one of the plurality of consecutive frames with cross-heatmap pose non-maximum suppression (NMS); and determining, by the one or more processors, confidence scores of the keypoints of the one of the objects within the one of the plurality of consecutive frames.

The step of estimating, by the one or more processors, a pose of each of the objects within each of the plurality of consecutive frames includes applying, by the one or more processors, a model ensemble mode combining two or more pose estimation models to determine the keypoints of the one of the objects within the one of the plurality of consecutive frames, wherein the two or more pose estimation models include at least a first model and a second model.

In some embodiments, the model ensemble mode may include an expert mode in which, for each of the keypoints of each of the objects within each of the plurality of consecutive frames, a weight of one is assigned to one of pose estimation result for the first model and pose estimation result for the second model that has a higher average precision (AP) for the one of the keypoints, and a weight of zero is assigned to the other of the pose estimation result for the first model and the pose estimation result for the second model.

In some embodiments, the model ensemble mode may include an average mode in which, for all of the keypoints of each of the objects within each of the plurality of consecutive frames, a weight of <NUM> is assigned to pose estimation result for the first model and a weight of <NUM> is assigned to pose estimation result for the second model.

In some embodiments, one of the first model and the second model may be epoch <NUM>, and the other of the first model and the second model may be epoch <NUM>.

In some embodiments, the step of detecting, by one or more processors, objects from each of a plurality of consecutive frames in a video sequence may be implemented by a deformable feature pyramid network (FPN) that is determined based on an object detector selecting mechanism.

In some embodiments, the step of tracking, by the one or more processors, the poses of each of the objects across the plurality of consecutive frames may include: assigning, by the one or more processors, a same identifier to the bounding boxes indicating a same one of the objects across the plurality of consecutive frames, and associating, by the one or more processors, the keypoints within the bounding boxes having the same identifier across the plurality of consecutive frames, so as to build a pose flow for each of the objects across the plurality of consecutive frames.

In some embodiments, the step of tracking, by the one or more processors, the poses of each of the objects across the plurality of consecutive frames may further include: performing, by the one or more processors, adaptive keypoint pruning on the keypoints of each of the objects within each of the plurality of consecutive frames with an adaptive keypoint pruner.

In some embodiments, the step of performing, by the one or more processors, adaptive keypoint pruning on the keypoints of each of the objects within each of the plurality of consecutive frames with an adaptive keypoint pruner may include: determining, by the one or more processors, a keypoint drop threshold for each of predefined categories of keypoint, wherein each of the keypoints of each of the objects within each of the plurality of consecutive frames belongs to one of the predefined categories of keypoint; dropping, by the one or more processors, a keypoint having a confidence score lower than a keypoint drop threshold for one of the predefined categories of keypoint to which the keypoint belongs; and retaining, by the one or more processors, a keypoint having a confidence score greater than or equal to a keypoint drop threshold for one of the predefined categories of keypoint to which the keypoint belongs.

In some embodiments, the step of detecting, by one or more processors, objects from each of a plurality of consecutive frames in a video sequence may further include: converting, by the one or more processors, the result of the detection for each of the objects within each of the plurality of consecutive frames into an openSVAI standardized data format, so as to generate a standardized detection result, wherein openSVAI is a library that provides modular implementation of components and wherein the inputs and outputs of each component are JavaScript Object Notation files with a standardized data structure.

In some embodiments, the step of estimating, by one or more processors, a pose of each of the objects within each of the plurality of consecutive frames may further include: converting, by the one or more processors, the result of the estimation for each of the objects within each of the plurality of consecutive frames into an openSVAI standardized data format, so as to generate a standardized estimation result.

In some aspects, the present disclosure relates to a device for tracking poses of multiple objects. The device includes a processor; and a memory storing instructions which, when executed by the processor, cause the processor to: detect objects from each of a plurality of consecutive frames in a video sequence; estimate a pose of each of the objects within each of the plurality of consecutive frames; and track the poses of each of the objects across the plurality of consecutive frames.

In some embodiments, the instructions which, when executed by the processor, may further cause the processor to: determine a bounding box for each of the objects from each of the plurality of consecutive frames; and determine a confidence score of the bounding box for the one of the objects within the one of the plurality of consecutive frames.

In some embodiments, the instructions which, when executed by the processor, may further cause the processor to: determine from a region defined by the bounding box for the one of the objects within the one of the plurality of consecutive frames, keypoints of the one of the objects within the one of the plurality of consecutive frames; and determine confidence scores of the keypoints of the one of the objects within the one of the plurality of consecutive frames.

In some embodiments, the instructions which, when executed by the processor, may further cause the processor to: regress a region defined by the bounding box for the one of the objects within the one of the plurality of consecutive frames into heatmaps of the one of the objects within the one of the plurality of consecutive frames; suppress the heatmaps of the one of the objects within the one of the plurality of consecutive frames into keypoints of the one of the objects within the one of the plurality of consecutive frames with cross-heatmap pose non-maximum suppression (NMS); and determine confidence scores of the keypoints of the one of the objects within the one of the plurality of consecutive frames.

In some embodiments, the instructions which, when executed by the processor, may further cause the processor to: apply a model ensemble mode combining two or more pose estimation models to determine the keypoints of the one of the objects within the one of the plurality of consecutive frames, wherein the two or more pose estimation models include at least a first model and a second model.

In some embodiments, the instructions which, when executed by the processor, may further cause the processor to: assign a same identifier to the bounding boxes indicating a same one of the objects across the plurality of consecutive frames, and associate the keypoints within the bounding boxes having the same identifier across the plurality of consecutive frames, so as to build a pose flow for each of the objects across the plurality of consecutive frames.

In some embodiments, the instructions which, when executed by the processor, may further cause the processor to: perform adaptive keypoint pruning on the keypoints of each of the objects within each of the plurality of consecutive frames with an adaptive keypoint pruner.

In some embodiments, the instructions which, when executed by the processor, may further cause the processor to: determine a keypoint drop threshold for each of predefined categories of keypoint, wherein each of the keypoints of each of the objects within each of the plurality of consecutive frames belongs to one of the predefined categories of keypoint; drop a keypoint having a confidence score lower than a keypoint drop threshold for one of the predefined categories of keypoint to which the keypoint belongs; and retain a keypoint having a confidence score greater than or equal to a keypoint drop threshold for one of the predefined categories of keypoint to which the keypoint belongs.

In some embodiments, the instructions which, when executed by the processor, may further cause the processor to: convert the result of the detection for each of the objects within each of the plurality of consecutive frames into an openSVAI standardized data format, so as to generate a standardized detection result.

In some embodiments, the instructions which, when executed by the processor, may further cause the processor to: convert the result of the estimation for each of the objects within each of the plurality of consecutive frames into an openSVAI standardized data format, so as to generate a standardized estimation result.

In some aspects, the present disclosure relates to a non-transitory computer readable medium storing computer executable instructions which, when executed by a processor of a computing device, causes the processor to perform any of the aforementioned methods.

The accompanying drawings illustrate one or more embodiments of the disclosure and together with the written description, serve to explain the principles of the disclosure. Wherever possible, the same reference numbers are used throughout the drawings to refer to the same or like elements of an embodiment.

Various embodiments of the disclosure are now described in detail. Referring to the drawings, like numbers indicate like components throughout the views. As used in the description herein and throughout the claims that follow, the meaning of "a", "an", and "the" includes plural reference unless the context clearly dictates otherwise. Also, as used in the description herein and throughout the claims that follow, the meaning of "in" includes "in" and "on" unless the context clearly dictates otherwise. Moreover, titles or subtitles may be used in the specification for the convenience of a reader, which shall have no influence on the scope of the present disclosure. Additionally, some terms used in this specification are more specifically defined below.

Certain terms that are used to describe the disclosure are discussed below, or elsewhere in the specification, to provide additional guidance to the practitioner regarding the description of the disclosure. It will be appreciated that same thing can be said in more than one way. Consequently, alternative language and synonyms may be used for any one or more of the terms discussed herein, nor is any special significance to be placed upon whether or not a term is elaborated or discussed herein. Synonyms for certain terms are provided. The use of examples anywhere in this specification including examples of any terms discussed herein is illustrative only, and in no way limits the scope and meaning of the disclosure or of any exemplified term.

As used herein, "around", "about", "substantially" or "approximately" shall generally mean within <NUM> percent, preferably within <NUM> percent, and more preferably within <NUM> percent of a given value or range. Numerical quantities given herein are approximate, meaning that the term "around", "about", "substantially" or "approximately" can be inferred if not expressly stated.

As used herein, "plurality" means two or more.

As used herein, the terms "comprising", "including", "having", "containing", "involving", and the like are to be understood to be open-ended, i.e., to mean including but not limited to.

As used herein, the phrase "at least one of A, B, and C" should be construed to mean a logical (A or B or C), using a non-exclusive logical OR. It should be understood that one or more steps within a method may be executed in different order (or concurrently) without altering the principles of the present disclosure.

As used herein, the term "module" may refer to, be part of, or include an Application Specific Integrated Circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or group) that executes code; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip. The term "module" may include memory (shared, dedicated, or group) that stores code executed by the processor.

The term "code", as used herein, may include software, firmware, and/or microcode, and may refer to programs, routines, functions, classes, and/or objects. The term "shared", as used above, means that some or all code from multiple modules may be executed using a single (shared) processor. In addition, some or all code from multiple modules may be stored by a single (shared) memory. The term "group", as used above, means that some or all code from a single module may be executed using a group of processors. In addition, some or all code from a single module may be stored using a group of memories.

The present disclosure relates to computer systems. As depicted in the drawings, computer components may include physical hardware components, which are shown as solid line blocks, and virtual software components, which are shown as dashed line blocks. One of ordinary skill in the art would appreciate that, unless otherwise indicated, these computer components may be implemented in, but not limited to, the forms of software, firmware or hardware components, or a combination thereof.

The devices, systems, and methods described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions that are stored on a non-transitory tangible computer readable medium. The computer programs may also include stored data. Nonlimiting examples of the non-transitory tangible computer readable medium are nonvolatile memory, magnetic storage, and optical storage.

The present disclosure will now be described more fully with reference to the accompanying drawings, in which some embodiments of the present disclosure are shown. This disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

Hereinafter, a specific example of a device for tracking poses of multiple objects (or persons) based on a plurality of single-object pose estimators according to some embodiments of the present disclosure will be described in detail with reference to <FIG>, <FIG> and <FIG>.

<FIG> is a block diagram schematically illustrating an example of the configuration of a computing device <NUM> for tracking poses of multiple objects based on a plurality of single-object pose estimators according to some embodiments of the present disclosure. In some embodiments, the computing device <NUM> may be a server computer, a cluster, a cloud computer, a general-purpose computer, a mobile device, a tablet, or a specialized computer, which detect objects from each of a plurality of consecutive frames in a video sequence, estimate a pose of each of the objects within each of the plurality of consecutive frames, and track the poses of each of the objects across the plurality of consecutive frames. In some embodiments, the computing device <NUM> may communicate with other computing devices or services. In some embodiments, the communication is performed via a network, which may be a wired or wireless network, and may be of various forms, such as a public network and a private network.

As shown in <FIG>, the computing device <NUM> may include, but is not limited to, a processor <NUM> and a memory <NUM>. Further, the computing device <NUM> may also include an optional storage device <NUM>. In some embodiments, the computing device <NUM> may include other hardware components and software components (not shown) to perform its corresponding tasks. Examples of these hardware and software components may include, but not limited to, other required memory, interfaces, buses, Input/Output (I/O) modules or devices, network interfaces, and peripheral devices or the like.

The processor <NUM> may be a central processing unit (CPU) which is configured to control operation of the computing device <NUM>. The processor <NUM> can execute an operating system (OS) or other applications for the computing device <NUM>. In some embodiments, the computing device <NUM> may have more than one CPU serving as the processor, such as two CPUs, four CPUs, eight CPUs, or any suitable number of CPUs. The memory <NUM> may be a volatile memory, such as a random-access memory (RAM), for storing data and instructions for the operation of the computing device <NUM>. In some embodiments, the memory <NUM> may be a volatile memory array. In some embodiments, the computing device <NUM> may run on more than one memory <NUM>. Further, the memory <NUM> may also be a non-volatile memory, such as a Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically EPROM (EEPROM), etc..

The storage device <NUM> may be a non-volatile data storage media for storing the OS (not shown) and other applications for the computing device <NUM>. Examples of the storage device <NUM> may include non-volatile memory such as flash memory, memory cards, USB drives, hard drives, floppy disks, optical drives, solid-state drive (SSD) or any other types of data storage devices. In some embodiments, the storage device <NUM> may be a local storage, a remote storage, or a cloud storage. In some embodiments, the computing device <NUM> may have multiple storage devices <NUM>, which may be identical storage devices or different types of storage devices, and the applications for the computing device <NUM> may be stored in one or more of the storage devices <NUM> of the computing device <NUM>. In some embodiments, the computing device <NUM> may be a cloud computer, and the processor <NUM>, the memory <NUM>, and the storage device <NUM> are shared resources provided over the Internet on-demand. In some other embodiments, the computing device <NUM> may be a workstation communicatively connected to a mainframe server which stores all the data and instructions for operating the computing device <NUM>, and in such a case, the storage device <NUM> may be optionally omitted.

As shown in <FIG>, the storage device <NUM> may store thereon a multi-object pose tracking application <NUM>, and at least one of an image database <NUM> and training data <NUM>. However, some or all of them may be stored in other locations and the present disclosure is not limited thereto.

In some embodiments, the multi-object pose tracking application <NUM> includes, among other things, a human candidate detection module <NUM>, a human pose estimation module <NUM>, and a human pose tracking module <NUM>. In some embodiments, the multi-object pose tracking application <NUM> may include other applications or modules necessary for the operation thereof. It should be noted that the modules are each implemented by computer executable codes or instructions, or data table or databases, which collectively forms one application. In some embodiments, each of the modules may further include sub-modules. In some other embodiments, some of the modules may be combined as one stack. In yet some other embodiments, certain modules may be implemented as a circuit instead of executable code. In some further embodiments, some or all of the modules of the multi-object pose tracking application <NUM> may be located at a remote computing device or distributed in a cloud.

In some embodiments, the human candidate detection module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, detect objects from each of a plurality of consecutive frames in a video sequence. In some embodiments, the human candidate detection module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to determine a bounding box for each of the objects from each of the plurality of consecutive frames, and determine a confidence score of the bounding box for the one of the objects within the one of the plurality of consecutive frames. Here, the objects within each of the plurality of consecutive frames may be persons to be tracked therein. The video sequence may be stored in the image database <NUM>. However, the present disclosure is not limited thereto.

The human candidate detection module <NUM> may be implemented by a convolutional neural network (CNN). As an example, the human candidate detection module <NUM> may be implemented by a deformable feature pyramid network (FPN) (see, for example, <NPL>). In some embodiments, the human candidate detection module <NUM> may be implemented by a FPN network with ResNet <NUM> backbone. In further embodiments, the human candidate detection module <NUM> may be implemented by a FPN network with ResNet <NUM> backbone. In some embodiments, the human candidate detection module <NUM> may be implemented by a FPN network that is trained with ImageNet and COCO datasets. In further embodiments, the human candidate detection module <NUM> may be implemented by a FPN network that is trained with the CrowdHuman [<NUM>] dataset.

The aforementioned deformable FPN network may be determined based on an object detector selecting mechanism. Specifically, pre-trained models from deformable convolutional neural networks (ConvNets) [<NUM>] may be used. In order to increase the recall rate of human candidates, experiments are conducted on validation sets of both PoseTrack <NUM> [<NUM>] and PoseTrack <NUM> so as to choose the best object detector. Firstly, ground truth bounding boxes for human candidates are inferred from the annotated keypoints, because in PoseTrack <NUM> dataset, the bounding box position is not provided in the annotations. In a specific exemplary implementation, the present disclosure locate a bounding box from the minimum and maximum coordinates of the <NUM> keypoints, and then enlarge the bounding box by <NUM>% both horizontally and vertically. Even though ground truth bounding boxes are given in PoseTrack <NUM> dataset, a more consistent version is inferred based on ground truth locations of keypoints. Those inferred ground truth bounding boxes are utilized to train the pose estimator.

For the object detectors, the deformable convolution versions of the Region-based Fully Convolutional Network (R-FCN) [<NUM>] and of the Feature Pyramid Network (FPN) [<NUM>], both with Residual Net (ResNet) <NUM> backbone [<NUM>] are compared. The FPN feature extractor is attached to the Fast Region-based Convolutional Network (Fast R-CNN) [<NUM>] head for detection. The detection results are compared with the ground truth based on the precision and recall rate on PoseTrack <NUM> validation set. In order to eliminate redundant candidates, candidate(s) with lower likelihood or confidence score(s) are dropped. As shown in Table <NUM> illustrating Precision-Recall on PoseTrack <NUM> validation set, for various drop thresholds of bounding boxes, the precision and recall rate of the detectors are given. A bounding box is correct if its intersection-over-union (IoU) with GT is above certain threshold, which is set to <NUM> for all experiments. In that case, the deformable FPN achieves a precision of <NUM> and a recall rate of <NUM>, while the deformable R-FCN achieves a precision of <NUM> and a recall rate of <NUM>, explicitly showing that the deformable FPN network performs better in terms of both the precision and the recall rate. For PoseTrack <NUM> validation set, the deformable FPN network performs better as well. Therefore, with the above-described object detector selecting mechanism, the deformable FPN network may be selected as a human candidate detector to implement the human candidate detection module <NUM>.

In addition, in order to measure the gap between ideal detection results and detection results generated with the selected deformable FPN network, further experiments are conducted. Specifically, based on the fact that the upper bound for detection is the ground truth bounding box location, the ground truth bounding boxes are fed to the subsequent human pose estimation module <NUM> and human pose tracking module <NUM>, which will be described in detail below, and its performance is compared with that of the selected object detector (i.e., deformable FPN network) on the validation set. As shown in Table <NUM> illustrating comparison of single-frame pose estimation results using various detectors on PoseTrack <NUM> validation set, for various predefined keypoints, average precision (AP) achieved with the ground truth detections, deformable FPN and deformable R-FCN are given. It can be seen that the pose estimation performs around <NUM>% better with ground truth detections (a total precision of <NUM> with ground truth detections, and a total precision of <NUM> with the deformable FPN). Additionally, the pose estimation performs <NUM>% better with the deformable FPN network than with the deformable R-FCN network (a total precision of <NUM> with the deformable FPN network, and a total precision of <NUM> with the deformable R-FCN network), which further validates the proper selection of the deformable FPN network, instead of the deformable R-FCN network, as the object detector.

Also, as shown in Table <NUM> illustrating comparison of multi-frame pose tracking results using various detectors on PoseTrack <NUM> validation set, for various predefined keypoints, Multi-Object Tracking Accuracy (MOTA) results achieved with the ground truth detections, deformable FPN and deformable R-FCN are given. The term "MOTA" may refer to a criterion for evaluating performance of human pose tracking systems and/or apparatuses. The higher the MOTA result or value, the better the human pose tracking system or apparatus is. It can be seen that the pose tracking performs around <NUM>% better with ground truth detections (a MOTA value of <NUM> with ground truth detections, and a MOTA score of <NUM> with the deformable FPN). Additionally, the pose tracking performs <NUM>% better with the deformable FPN network than with the deformable R-FCN network (a MOTA value of <NUM> with the deformable FPN network, and a MOTA score of <NUM> with the deformable R-FCN network), which also further validates the proper selection of the deformable FPN network, instead of the deformable R-FCN network, as the object detector.

After determining the deformable FPN network as an object detector based on the object detector selecting mechanism and training it with datasets from the training data <NUM>, a video sequence consisting of a plurality of consecutive frames stored in the image database <NUM> may be acquired/retrieved and fed to the human candidate detection module <NUM>, so as to detect objects from each of the plurality of consecutive frames. As an example, <FIG> exemplarily illustrates output produced by the human candidate detection module <NUM> according to some embodiments of the present disclosure. Specifically, a plurality of bounding boxes each of which locates a respective one of a plurality of objects within a frame are detected and shown in <FIG>.

Then the result of the detection for each of the objects within each of the plurality of consecutive frames is converted into an openSVAI standardized data format (https://doc-opensvai. readthedocs. io/en/latest/ and https://media. readthedocs. org/pdf/doc-opensvai/latest/doc-opensvai. pdf, so as to generate a standardized detection result for each of the objects within each of the plurality of consecutive frames.

OpenSVAI is a library that provides modular implementation of components such as object detection, single-person pose estimation, pose tracking, semantic segmentation, etc. Existing open-sourced projects are usually not compatible. In order to build a complex system that consists of multiple components (each component is a self-sufficient research topic), we provide such a library that also standardizes the inputs and outputs of each component. Thus, each component's method is conveniently replaceable, making the build of a system much easier. The inputs and outputs are JavaScript Object Notation (JSON) files with a standardized data structure, which covers the information needed in a vision system: image ID, image path, image name, candidate, candidate ID, candidate's pose, candidate's mask, etc..

In some embodiments, the human candidate detection module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may further cause the processor <NUM> to, after generating the standardized detection result for each of the objects within each of the plurality of consecutive frames, send the standardized detection result to the human pose estimation module <NUM>.

In some embodiments, the human pose estimation module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, upon receiving the standardized detection result for each of the objects within each of the plurality of consecutive frames, estimating a pose of each of the objects within each of the plurality of consecutive frames. In some embodiments, the human pose estimation module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to determine, from a region defined by the bounding box for the one of the objects within the one of the plurality of consecutive frames, keypoints of the one of the objects within the one of the plurality of consecutive frames, and determine confidence scores of the keypoints of the one of the objects within the one of the plurality of consecutive frames.

Here, the term "keypoint" may refer to a body keypoint satisfying a set of geometric constraints and inter-dependency imposed by the human body model. In some embodiments, there may be defined multiple categories of keypoint. Hereinafter, details of human pose estimation will be described and explained by taking <NUM> categories of keypoint (such as, a neck keypoint, a left shoulder keypoint, a right shoulder keypoint, a left elbow keypoint, a right elbow keypoint, a left wrist keypoint, a right wrist keypoint, a left knee keypoint, a right knee keypoint, a left ankle keypoint, a right ankle keypoint, etc.) as an example. However, it is to be noted that the number of categories of keypoint is not limited thereto, and there may be more or fewer categories of keypoint than described in example embodiments.

In some embodiments, the human pose estimation module <NUM> may be implemented by a convolutional neural network (CNN). In some embodiments, the human pose estimation module <NUM> may be implemented by a Cascaded Pyramid Networks (CPN) (see, for example,<NPL>).

Specifically, the aforementioned CPN network is trained with the merged dataset of PoseTrack <NUM> and COCO for <NUM> epochs, and then fine-tuned solely on PoseTrack <NUM> training set for <NUM> epochs in order to mitigate the regression on head. For COCO dataset, bottom-head and top-head positions are not given. These keypoints are inferred through rough interpolation on the annotated keypoints. It is found that by finetuning on the PoseTrack dataset, the prediction on head keypoints (including a neck keypoint, etc.) will be refined. During finetuning, the technique of online hard keypoint mining, only focusing on losses from the <NUM> hardest keypoints out of the total <NUM> keypoints, is utilized.

Thereafter, the CPN network is tested on both PoseTrack <NUM> and PoseTrack <NUM> validation sets. That is, feed both PoseTrack <NUM> and PoseTrack <NUM> validation sets to the trained CPN network and generate a set of predicted joint coordinates (or keypoint coordinates) in the image coordinate system, and compare the set of predicted joint coordinates with ground truth joint coordinates, so as to test the pose estimation performance of the trained model. The first two rows of Table <NUM>, which illustrates comparison of single-frame pose estimation results with different ensemble modes on PoseTrack <NUM> validation set, show that different models (for example, epoch <NUM> and <NUM>, which are two trained models having relatively better performance in terms of average precision (AP)) perform differently on different categories of keypoint. As an example, for epoch <NUM>, the prediction of shoulders (an AP of <NUM>) and hips (an AP of <NUM>) renders better results than epoch <NUM> (an AP of <NUM> for shoulders, an AP of <NUM> for hips) on validation sets of both PoseTrack <NUM> and PoseTrack <NUM>. However, epoch <NUM> performs better on end limbs such as wrists (an AP of <NUM>) and ankles (an AP of <NUM>) than epoch <NUM> (an AP of <NUM> for wrists, an AP of <NUM> for ankles). Thus, in some embodiments, in order to render more accurate estimation, a manifold of two models from epoch <NUM> and <NUM> may be used during the testing.

Two manifold modes: (<NUM>) an average mode in which, for all predefined categories of keypoint, a weight of <NUM> is assigned to epoch <NUM> and a weight of <NUM> is assigned to epoch <NUM>; and (<NUM>) an expert mode in which, for each of the predefined categories of keypoint, a weight of one is assigned to one of two models from epoch <NUM> and epoch <NUM> having a higher AP for the respective category of keypoint and a weight of zero is assigned to the other of the two models, are tested, and results of the testing are illustrated in the last two rows in Table <NUM>. As can be seem from Table <NUM>, the average mode has a total AP for all categories of keypoint of <NUM>, and the expert mode takes shoulder/hip predictions from the previous model and end-limb predictions from the latter and has a total AP of <NUM>, both of which are higher than that of either of epoch <NUM> (<NUM>) and <NUM> (<NUM>), demonstrating that both modes perform better than plain testing on the pose estimation task. However, it is to be noted that although only pose estimation results on PoseTrack <NUM> validation set is shown in Table <NUM>, both the average mode and the expert mode perform better consistently on both PoseTrack <NUM> and PoseTrack <NUM> validation sets.

Therefore, in some embodiments, a model ensemble mode may be applied to determine keypoints of each of objects within each of a plurality of consecutive frames, which may boost performance of the human pose estimation module <NUM>. As an example, an average mode in which, for all predefined categories of keypoint, a weight of <NUM> is assigned to a first model and a weight of <NUM> is assigned to a second model, may be applied to determine keypoints of each of objects within each of a plurality of consecutive frames. As another example, an expert mode in which, for each of the predefined categories of keypoint, a weight of one is assigned to one of a first model and a second model having a higher AP for the respective category of keypoint and a weight of zero is assigned to the other of the first model and the second model, may be applied to determine keypoints of each of objects within each of a plurality of consecutive frames. Here, it is to be noted that the model ensemble mode is not limited thereto, and there may be more than two models for forming model ensemble mode. In some embodiments, one of the first model and the second model may be epoch <NUM>, and the other thereof may be epoch <NUM>. In some embodiments, the average mode and the expert mode are combined, where each category is assigned with a <NUM>:<NUM> or <NUM>:<NUM> or <NUM>:<NUM> weight for the first and second models. The assignment of the weight is based on the respective AP. In one example, the category of "head" is assigned with <NUM>:<NUM> weight for epoch <NUM> and epoch <NUM>, the category of "Hip" is assigned with epoch <NUM>, the categories of "Elbow," "Wrist," "Knee," and "Ankle" are assigned with epoch <NUM>, and the category of "shoulder" is assigned with epoch <NUM>, or an average of epoch <NUM> and <NUM>.

After training the CPN network with datasets from the training data <NUM> and determining to use a manifold of two models from epoch <NUM> and <NUM> to implement human pose estimation, the standardized detection result for each of the objects within each of the plurality of consecutive frames generated by the human candidate detection module <NUM> may be fed to the human pose estimation module <NUM>, so as to estimate a pose of each of the objects within each of the plurality of consecutive frames. The pose estimation is performed on all bounding boxes (i.e., candidates) from the human candidate detection module <NUM>. Specifically, a region defined by the bounding box for each of the objects within each of the plurality of consecutive frames may be regressed into heatmaps of the one of the objects within the one of the plurality of consecutive frames. Then, in order to render more accurate keypoint locations, for each bounding box, post-process may be performed on the predicted heatmaps with cross-heatmap pose non-maximum suppression (NMS) (see, for example, <NPL>). In particular, the heatmaps of each of the objects within each of the plurality of consecutive frames may be suppressed into the keypoints of the one of the objects within the one of the plurality of consecutive frames with a cross-heatmap pose NMS module <NUM>.

As an example, <FIG> is a block diagram schematically illustrating an example of the configuration of a cross-heatmap pose NMS module <NUM> according to some embodiments of the present disclosure. In some embodiments, the cross-heatmap pose NMS module <NUM> may be included in the human pose estimation module <NUM>. As shown in <FIG>, the cross-heatmap pose NMS module <NUM> may include, among other things, a heatmap obtaining module <NUM>, a blob gathering module <NUM>, a blob suppressing module <NUM>, and a heatmap number comparing module <NUM>. In some embodiments, the heatmap obtaining module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to receive a set of (for example, N) heatmaps regressed from an image of an object, wherein N is a positive integer. In some embodiments, the blob gathering module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, gather blobs from all of the N heatmaps for suppression. In some embodiments, the blob suppressing module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, determine the blob with the maximum response, and suppress other blobs from the same heatmap and blobs from other heatmaps very close to the determined blob in image coordinate system, and remove the heatmap from the set of heatmaps and let N=N-<NUM>. In some embodiments, the heatmap number comparing module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, compare current heatmap number N=N-<NUM> with zero. The process may repeat until current heatmap number N=N-<NUM> equals to zero. As a result, keypoints of each of the objects within each of the plurality of consecutive frames and respective confidence scores of respective keypoints may be determined.

In some embodiments, the heatmap NMS procedure is as follows:.

The implementation is open-sourced here: https://github. com/Guanghan/GNet-pose/blob/master/testing/utils/utils_nms py.

Kindly note: Keypoint corresponds to a single pixel in the heatmap, while a blob is obtained by blob detection from the heatmap. A blob is a collection of adjacent pixels; it can be of any shape; it can be regarded as a connected component. In a heatmap, there can be more than one blobs detected.

After heatmap NMS, each heatmap channel has no more than one blob. The keypoint will be the peak position of the blob from its corresponding channel map. For some keypoints, their corresponding heatmaps may not have remaining blobs. In this case, the keypoints for these channels will be the peak positions.

As an example, <FIG> exemplarily illustrate pose estimation process performed by the human pose estimation module <NUM> according to some embodiments of the present disclosure, wherein <FIG> exemplarily illustrates an object among a plurality of objects within a frame located by a respective bounding box according to some embodiments of the present disclosure, and <FIG> exemplarily illustrate part regression heatmap results of the image region defined by the bounding box for the object according to some embodiments of the present disclosure, and <FIG> exemplarily illustrates pose estimation result for the object with the cross-heatmap NMS suppression according to some embodiments of the present disclosure. The keypoints as well as lines connecting two adjacent keypoints may represent pose of an object. In some embodiments, an RGB-channel image of an object may be regressed into a set of <NUM> heatmaps, each of which may correspond to a keypoint. Also, it is to be noted that the number of heatmaps regressed from an image of an object is not limited thereto.

As an example, <FIG> exemplarily illustrates output produced by the human pose estimation module <NUM> according to some embodiments of the present disclosure. Specifically, part keypoints of the plurality of objects shown in <FIG> are determined from regions defined by the plurality of bounding boxes shown in <FIG>, and shown in <FIG>, wherein each of the plurality of objects corresponds to the same number of (for example, <NUM>) predefined part keypoints.

Then the result of the estimation for each of the objects within each of the plurality of consecutive frames is converted into an openSVAI standardized data format, so as to generate a standardized estimation result for each of the objects within each of the plurality of consecutive frames.

The human pose estimation module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may further cause the processor <NUM> to, after generating the standardized estimation result for each of the objects within each of the plurality of consecutive frames, send the standardized estimation result to the human pose tracking module <NUM>.

In some embodiments, the human pose tracking module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, upon receiving the standardized estimation result for each of the objects within each of the plurality of consecutive frames, track the poses of each of the objects across the plurality of consecutive frames. In some embodiments, the human pose tracking module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to assign a same identifier to the bounding boxes indicating a same one of the objects across the plurality of consecutive frames, and associate the keypoints within the bounding boxes having the same identifier across the plurality of consecutive frames, so as to build a pose flow for each of the objects across the plurality of consecutive frames. Here, the term "pose flow" may refer to a set of pose indicating the same person instance (i.e., object) within a plurality of consecutive frames.

In some embodiments, the human pose tracking module <NUM> may be implemented by a flow-based pose tracker (see, for example, <NPL>). Specifically, the human pose tracking module <NUM> may build the association of cross-frame poses determined by the human pose estimation module <NUM> and form pose flows based thereon, and then reduce redundant pose flows and relink temporal disjoint ones with pose flow non-maximum suppression (NMS). Details of the process may be referred to the above reference.

In order to improve the performance of the human pose tracking module <NUM>, extensive experiments are conducted. The performance of a pose tracker may be evaluated via MOTA, which is very strict. It penalizes mis-matches, false positives and misses. In order to get higher MOTA results, keypoints with lower confidence scores need to be dropped, sacrificing the recall rate of correct keypoints. Table <NUM> illustrates sensitivity analysis on how the drop thresholds of keypoints affect the performance in AP and MOTA (performed on Pose Track <NUM> validation set). As can be seen from Table <NUM>, with the drop threshold increases from <NUM> to <NUM>, the AP of pose estimation decrease from <NUM> to <NUM> while the MOTA result of pose tracking increases significantly from <NUM> to <NUM>, showing that the MOTA evaluation criterion quite sensitive to the drop rate of keypoints.

Considering the distinct difficulties of keypoints, e.g., shoulders are easier than ankles to localize, the confidence distribution for each joint may be not uniform. Dropping keypoints solely based on the keypoint confidence estimated by the pose estimator (for example, implemented by the human pose estimation module <NUM>) may not be an ideal strategy for pose tracking. In some embodiments, statistics on the drop rate of keypoints are collected from different joints, as shown in Table <NUM>, which illustrates statistics analysis on the drop rates of keypoints with different drop thresholds (the analysis is performed on PoseTrack <NUM> validation set, and the numbers indicate the percentage of keypoints maintained after pruning). It can be seen that from left to right, the keypoints become more and more difficult to estimate, as reflected by their respective preservation rate. The least and most difficult joints are the shoulders and ankles, respectively. In other words, the pose estimator is most confident on the shoulders but least confident on ankles. Therefore, an adaptive keypoint pruner may help to increase the MOTA performance while maintaining high recall rates.

In some embodiments, the adaptive keypoint pruner may be implemented by a convolutional neural network (CNN). In some embodiments, an adaptive keypoint pruner outperforms pruning keypoints solely based on a fixed threshold, which is sensitive to the test data and the pose estimator.

<FIG> is a block diagram schematically illustrating an example of the configuration of an adaptive keypoint pruner <NUM> according to some embodiments of the present disclosure. In some embodiments, the adaptive keypoint pruner <NUM> may be included in the human pose tracking module <NUM>.

As shown in <FIG>, the adaptive keypoint pruner <NUM> may include, among other things, a keypoint drop threshold determining module <NUM>, and a confidence score comparing module <NUM>. In some embodiments, the keypoint drop threshold determining module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to determine a keypoint drop threshold for each of the predefined categories of keypoint. In some embodiments, the confidence score comparing module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to compare confidence score of each keypoint of each object within each frame with a respective keypoint drop threshold, drop the keypoint with a confidence score lower than the respective keypoint drop threshold, and retain the keypoint with a confidence score greater than or equal to the respective keypoint drop threshold. In this way, keypoints with lower confidence score may be adaptively dropped based on respective drop thresholds of respective categories of keypoint.

After performing adaptive keypoint pruning on the keypoints of each of the objects within each of the plurality of consecutive frames received from the human pose estimation module <NUM>, retained keypoints of each of the objects within each of the plurality of consecutive frames may be fed to the human pose tracking module <NUM>. The pose tracking process may start from the first frame where human candidates are detected. The human pose tracking module <NUM> may assign an identifier for each object across a plurality of frames. To prevent assignments of IDs for persons (i.e., objects) which have already left the visible image area, IDs are only kept for a limited number of frames, afterwards they are discarded. As an example, <FIG> exemplarily illustrates output produced by the human pose tracking module <NUM> according to some embodiments of the present disclosure. As shown in <FIG>, each of the plurality of objects shown in <FIG> are assigned a unique identifier, and the same object within the plurality of consecutive frames has the same identifier.

It is to be noted that although <FIG> show apparatuses and modules of the computing device <NUM>, in other embodiments, the computing device <NUM> may include fewer, different, differently arranged, or additional apparatuses and/or modules than depicted in <FIG>. Additionally, or alternatively, one or more apparatuses and/or modules of the computing device <NUM> may perform one or more of the tasks described as being performed by one or more other apparatuses and/or modules of the computing device <NUM>.

Hereinafter, methods of tracking poses of multiple objects according to some embodiments of the present disclosure will be described in detail with reference to <FIG>. It should be particularly noted that, unless otherwise stated in the present disclosure, the steps of any of the methods may be arranged in a different order, and are thus not limited to the sequential order as shown in the drawings. In addition, the same description as that of the aforementioned device for tracking poses of multiple objects will be omitted.

<FIG> is a flow chart schematically illustrating a method of tracking poses of multiple objects based on a plurality of single-object pose estimators according to some embodiments of the present disclosure. In some embodiments, the method may be implemented by the computing device <NUM> shown in <FIG>. In some embodiments, the method shown in <FIG> may correspond to part or all of the function of the multi-object pose tracking application <NUM>. The method of tracking poses of multiple objects may include the following steps.

At step <NUM>, the human candidate detection module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to detect objects from each of a plurality of consecutive frames in a video sequence and send the detected result to the executed human pose estimation module <NUM>. In some embodiments, the result includes bounding boxes in each of the frames. In some embodiments, the result is described in openSVAI format.

At step <NUM>, the human pose estimation module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, in response to receiving the detected result from the human candidate detection module <NUM>, estimate a pose of each of the objects within each of the plurality of consecutive frames using the detection result, and send the estimated poses to the executed human pose tracking module <NUM>. In some embodiments, the estimated poses are defined in openSVAI format.

At step <NUM>, the human pose tracking module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, in response to receiving the estimated poses of the objects, track the poses of each of the objects across the plurality of consecutive frames.

<FIG> is a flow chart schematically illustrating a method of tracking poses of multiple objects based on a plurality of single-object pose estimators according to further embodiments of the present disclosure. In some embodiments, the method may be implemented by the computing device <NUM> shown in <FIG>. In some embodiments, the method shown in <FIG> may correspond to part or all of the function of the multi-object pose tracking application <NUM>. The method of tracking poses of multiple objects may include the following steps.

At step <NUM>, the human candidate detection module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to determine a bounding box for each of objects within each of a plurality of consecutive frames in a video sequence and a confidence score thereof, and send the determined bounding boxes and confidence scores to the executed human pose estimation module <NUM>.

At step <NUM>, the human pose estimation module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, in response to receiving the bounding boxes and confidence scores, determine, from a region defined by the bounding box for the one of the objects within the one of the plurality of consecutive frames, keypoints of the one of the objects within the one of the plurality of consecutive frames, and confidence scores thereof, and send the results to the executed human pose tracking module <NUM>.

In some embodiments, the step of determining, from a region defined by the bounding box for the one of the objects within the one of the plurality of consecutive frames, keypoints of the one of the objects within the one of the plurality of consecutive frames, and confidence scores thereof may include: regressing a region defined by the bounding box for the one of the objects within the one of the plurality of consecutive frames into heatmaps thereof; suppressing the heatmaps of the one of the objects within the one of the plurality of consecutive frames into keypoints thereof with cross-heatmap pose non-maximum suppression (NMS); and determining confidence scores of the keypoints of the one of the objects within the one of the plurality of consecutive frames. Details of the cross-heatmap pose NMS will be described below with reference to <FIG>.

In some embodiments, the step of determining, from a region defined by the bounding box for the one of the objects within the one of the plurality of consecutive frames, keypoints of the one of the objects within the one of the plurality of consecutive frames and confidence scores thereof may further include: applying a model ensemble mode combining two or more pose estimation models to determine the keypoints of the one of the objects within the one of the plurality of consecutive frames, wherein the two or more pose estimation models include at least a first model and a second model. In some embodiments, the model ensemble mode may include an average mode in which, for all of the keypoints of each of the objects within each of the plurality of consecutive frames, a weight of <NUM> is assigned to pose estimation result for the first model and a weight of <NUM> is assigned to pose estimation result for the second model. In other embodiments, the model ensemble mode may include an expert mode in which, for each of the keypoints of each of the objects within each of the plurality of consecutive frames, a weight of one is assigned to one of pose estimation result for the first model and pose estimation result for the second model that has a higher average precision (AP) for the one of the keypoints, and a weight of zero is assigned to the other of the pose estimation result for the first model and the pose estimation result for the second model. In some embodiments, one of the first model and the second model is epoch <NUM>, and the other of the first model and the second model is epoch <NUM>.

At step <NUM>, the human pose tracking module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, in response to receiving the pose estimation result from the pose estimation module <NUM>, perform adaptive keypoint pruning on the keypoints of each of the objects within each of the plurality of consecutive frames with an adaptive keypoint pruner. Details of the process performed by the adaptive keypoint pruner <NUM> will be described below with reference to <FIG>.

At step <NUM>, the human pose tracking module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, after performing adaptive keypoint pruning, assign a same identifier to the bounding boxes indicating a same one of the objects across the plurality of consecutive frames.

At step <NUM>, the human pose tracking module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, after assigning the same identifier to the bounding boxes indicating the same one of the objects in the consecutive frames, associate the keypoints within the bounding boxes having the same identifier across the plurality of consecutive frames, so as to build a pose flow for each of the objects across the plurality of consecutive frames.

<FIG> is a flow chart exemplarily illustrating a method of suppressing heatmaps of an object into keypoints with cross-heatmap NMS according to some embodiments of the present disclosure. In some embodiments, the method may be implemented by the cross-heatmap pose NMS module <NUM> shown in <FIG>. The method of cross-heatmap NMS suppression may include the following steps.

At step <NUM>, the heatmap obtaining module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to receive a set of (for example, N) heatmaps regressed from an image of an object, where N is a positive integer, and send the heatmaps to the executed blob gathering module <NUM>.

At step <NUM>, the blob gathering module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, in response to receiving the heatmaps, gather blobs from all of the N heatmaps for suppression, and send the gathered blobs to the executed blob suppressing module <NUM>.

At step <NUM>, the blob suppressing module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, in response to receiving the gathered blobs, determine the blob with the maximum response, and suppress other blobs from the same heatmap and blobs from other heatmaps very close to the determined blob in image coordinate system, and remove the heatmap from the set of heatmaps (N=N-<NUM>).

At step <NUM>, the heatmap number comparing module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, compare current heatmap number N=N-<NUM> with zero. If N=N-<NUM> is greater than zero, then the process returns to step <NUM>, while if not, then the process terminates. Here, it is to be noted that the term "terminate" means the suppression process with respect to the present object terminates. And other objects may follow the same suppression process. With the above-described cross-heatmap NMS method, keypoints of each of the objects within each of the plurality of consecutive frames may be determined.

<FIG> is a flow chart exemplarily illustrating a method of pruning keypoints according to some embodiments of the present disclosure. In some embodiments, the method may be implemented by the adaptive keypoint pruner <NUM> shown in <FIG>. The method of pruning keypoints may include the following steps.

At step <NUM>, the keypoint drop threshold determining module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to determine a keypoint drop threshold for each of the predefined categories of keypoint, and send the keypoint drop thresholds to the confidence score comparing module <NUM>.

At step <NUM>, the confidence score comparing module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to, in response to receiving the keypoint drop thresholds, determine if confidence score of each keypoint of each object within each frame is greater than or equal to a respective keypoint drop threshold. If YES, the process proceeds to step <NUM>, while if NO, the process proceeds to step <NUM>.

At step <NUM>, the confidence score comparing module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to retain the keypoint, in response to confidence score of the keypoint being greater than or equal to a respective keypoint drop threshold.

At step <NUM>, the confidence score comparing module <NUM> which, when loaded into the memory <NUM> and executed by the processor <NUM>, may cause the processor <NUM> to discard the keypoint, in response to confidence score of the keypoint being less than a respective keypoint drop threshold. With the above-described keypoint pruning method, keypoints with lower confidence score may be adaptively dropped based on respective drop thresholds of respective categories of keypoint.

As an example, final performance on the partial test set of PoseTrack <NUM> implemented with the above-described apparatuses and methods is given in Table <NUM> illustrating single-frame pose estimation results on PoseTrack <NUM> partial test set and Table <NUM> illustrating multi-frame pose tracking results on PoseTrack <NUM> partial test set.

In summary, some embodiments of the present disclosure provide an accurate and robust multi-object pose tracking system comprising a state-of-the-art object detector, single-person pose estimators, and human pose tracker. In some embodiments, the multi-object pose tracking system is configured to choose a generic object detector among state-of-the-art methods to detect human candidates, utilize cascaded pyramid network to estimate the corresponding human pose, and utilize a flow-based pose tracker to render keypoint-association across frames, i.e., assigning each human candidate a unique and temporally-consistent id, for the multi-target pose tracking purpose. Extensive ablative experiments are conducted to validate various choices of models and configurations and achieve better performance.

It is to be noted that, the multi-object pose tracking system based on a plurality of single-object pose estimators may be generic in its sub-modules, e.g., the human pose estimation module <NUM> and the human pose tracking module <NUM>, which are replaceable and can be upgraded.

The advantages of the present disclosure, among other things, include: (<NUM>) an end-to-end human pose estimation and tracking system is provided, which has sequentially an object detection module detecting objects in a series frames, a single-person pose estimation module estimating poses of objects in the series frames, and a pose tracking module tracking the objects in the series frames based on the estimated poses; (<NUM>) an pose estimation module that combines advantages of different models, where each keypoint may have different weights of using the different models; (<NUM>) an openSVAI standardized data format is provided for the whole top-down approach.

The embodiments were chosen and described in order to explain the principles of the disclosure and their practical application so as to enable others skilled in the art to utilize the disclosure and various embodiments and with various modifications as are suited to the particular use contemplated.

Claim 1:
A method of tracking poses of multiple objects, comprising:
detecting, by one or more processors, objects from each of a plurality of consecutive frames in a video sequence;
estimating, by the one or more processors, a pose of each of the objects within each of the plurality of consecutive frames; and
tracking, by the one or more processors, the poses of each of the objects across the plurality of consecutive frames,
wherein the step of detecting the objects from each of the plurality of consecutive frames in the video sequence comprises:
determining, by the one or more processors, a bounding box for each of the objects from each of the plurality of consecutive frames; and
determining, by the one or more processors, a confidence score of the bounding box for the one of the objects within the one of the plurality of consecutive frames,
wherein the step of estimating the pose of each of the objects within each of the plurality of consecutive frames comprises:
determining, by the one or more processors, from a region defined by the bounding box for the one of the objects within the one of the plurality of consecutive frames, keypoints of the one of the objects within the one of the plurality of consecutive frames;
determining, by the one or more processors, confidence scores of the keypoints of the one of the objects within the one of the plurality of consecutive frames;
characterised in that estimating the pose of each of the objects within each of the plurality of consecutive frames further comprises:
applying, by the one or more processors, a model ensemble mode combining two or more pose estimation models to determine the keypoints of the one of the objects within the one of the plurality of consecutive frames,
wherein the two or more pose estimation models comprise at least a first model and a second model.