Learning Local-Global Contextual Adaptation for Fully End-to-End Bottom-Up Human Pose Estimation

This paper presents a method of learning LOcal-GlObal Contextual Adaptation for1 fully end-to-end and fast bottom-up human Pose estimation , dubbed as LOGO-2 CAP . It is built on the conceptually simple center-offset formulation that lacks3 inaccuracy for pose estimation . When revisiting the bottom-up human pose es-4 timation with the thought of “ thinking , fast and slow ” by D. Kahneman , we in-5 troduce a “ slow keypointer ” to remedy the lack of sufficient accuracy of the “ fast6 keypointer ” . In learning the “ slow keypointer ” , the proposed LOGO-CAP lifts the7 initial “ fast ” keypoints by offset predictions to keypoint expansion maps ( KEMs ) 8 to counter their uncertainty in two modules . Firstly , the local KEMs ( e.g . 11×11 ) 9 are extracted from a low-dimensional feature map . A proposed convolutional mes-10 sage passing module learns to “ re-focus ” the local KEMs to the keypoint attraction11 maps ( KAMs ) by accounting for the structured output prediction nature of human12 pose estimation , which is directly supervised by the object keypoint similarity13 ( OKS ) loss in training . Secondly , the global KEMs are extracted , with a suffi-14 ciently large region-of-interest ( e.g. , 97 × 97 ) , from the keypoint heatmaps that15 are computed by a direct map-to-map regression . Then , a local-global contextual16 adaptation module is proposed to convolve the global KEMs using the learned17 KAMs as the kernels . This convolution can be understood as the learnable offsets18 guided deformable and dynamic convolution in a pose-sensitive way . The pro-19 posed method is end-to-end trainable with near real-time inference speed , obtain-20 ing state-of-the-art performance on the COCO keypoint benchmark for bottom-up21 human pose estimation . With the COCO trained model , our LOGO-CAP also22 outperforms prior arts by a large margin on the challenging OCHuman dataset.23 1 Introduction24 1.1 Motivation and Objective25 Human pose is highly articulated with large structural and appearance variations . 2D human pose26 estimation in images is a classic structured output prediction problem , and remains a challenging one27 in computer vision and machine learning . Human pose estimation has numerous applications such28 as people-centered image understanding , autonomous driving and Augmented Reality ( AR ) . With29 the recent resurgence of deep neural networks ( DNNs ) , the performance of human pose estimation30 has witnessed remarkable improvement [ 12 , 3 , 15 , 22 , 11 ] . This paper focuses on the deep learning31 based problem formulation.32 There are two deep learning based paradigms for human pose estimation in the literature . The top-33 down paradigm consists of human detection and single human pose estimation in each detected34 human bounding box [ 12 ] . The bottom-up paradigm also includes two components : human pose35 keypoint detection and keypoint grouping [ 3 ] . The top-down paradigm often obtains better accuracy36 performance , but suffers from its inferior efficiency since the computational cost of the single human37 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . pose estimation component is linearly scaled with respect to the number of detected human bounding38 boxes in an image . It is also largely affected by the performance of the human detection component39 ( e.g. , not handling occlusion very well ) . Thanks to its efficiency , especially in real-time applications,40 the bottom-up paradigm becomes more and more attractive . For both paradigms , state-of-the-art41 methods often are not fully end-to-end trained and utilize different post-hoc processing modules42 to improve the performance . This paper is interested in developing a fully end-to-end bottom-up43 paradigm and aims at bridging its performance gap with the top-down paradigm.44 For the bottom-up paradigm , the recently proposed center-offset approach [ 6 , 28 , 26 , 11 ] is a con-45 ceptually simple formulation ( see the left of Fig . 1 for an illustrative example and Fig . 3 for the46 detailed workflow ) . It alleviates the need of sophisticated keypoint grouping . When introducing47 human keypoints centers ( i.e. , anchors ) by treating objects as points [ 35 ] , it encodes a human pose48 as a star structure using the offset vectors of human keypoints relative to the anchors . The main49 challenge of the center-offset regression paradigm lies in the difficulty of accurately learning offset50 vectors with large structural variations , especially the long-range ones , which also leads to inferior51 performance . This paper builds on the center-offset approach and addresses its drawback.52 1.2 Method Overview53 To address the drawback of the center-offset formulation , we build the intuitive idea of54 “ Keypointing , fast and slow ” , by analogy to the modes of thought suggested by Daniel Kahne-55 man in “ Thinking , fast and slow ” [ 14 ] : ( i ) Fast Keypointer : We treat the vanilla center-offset based56 estimation [ 35 ] as the Fast Initializer of pose estimation . ( ii ) Slow Keypointer : The lack of localiza-57 tion accuracy in the Fast Initializer entails a Slow Solver that learns to refine the “ fast ” keypoints . By58 slow , it is only relatively speaking . The Slow Keypointer is actually fast with near real-time speed.59 Baseline Emp . Bound LOGO-CAP AP 60.1 88.9 70.0 AP50 85.2 93.1 88.2 AP75 66.7 90.6 76.4 APM 53.7 87.7 64.4 APL 71.5 90.2 78.4 clearly shows that the pose keypoints center and the offset vectors can be learned reasonably well.73 Instead of directly utilizing the learned offset vectors for human pose estimation , we treat them as74 human pose keypoint initialization and do a local window search to compute the empirical upper-75 bound of performance . More detailed , based on the predicted human poses , by introducing a local76 window ( e.g. , 11 × 11 ) centered at each detected key point and by computing the single keypoint77 similarity with the ground-truth keypoint , an empirical upper-bound of 88.9 AP is obtained , which78 is significantly higher than the state of the art and shows the potential of improving the vanilla79 center-offset regression paradigm.80 Motivated by the above observation , a straightforward way is just to learn a local heatmap ( e.g.,81 11 × 11 ) for each human pose keypoint based on the learned center and offset vectors , and then to82 compute the refined keypoints by taking arg max within the local heatmap . Although appealing,83 this does not work as observed during our development of the LOGO-CAP . The underlying reason84 is easy to understand : if this can work , the original offset vector regression should work at the85 first place since no additional information is introduced through learning the local heatmap . We86 hypothesize that on the one hand , on top of the local heatmap , the structural relationship between87 different keypoints of a human pose needs to be taken into account , and on the other hand , the88 intrinsic uncertainty of the local information in a local heatmap needs to be resolved . The former89 is the key challenge of structured output prediction problems . Many message passing algorithms90 have been developed in the literature . The latter can not be addressed by simply increasing the local91 window size . It entails learning stronger local-global information interaction and adaptation , .92 Along with the two hypotheses , the proposed LOGO-93 CAP lifts the initial keypoints via the center-offset pre-94 diction to keypoint expansion maps ( KEMs ) to counter95 their lack of localization accuracy in two modules ( Sec-96 tion 3.2 ) . The KEMs extend the star-structured repre-97 sentation of the center-offset formulation to the pictorial98 structure representation [ 10 , 8 ] . The first module com-99 putes local KEMs and learns to account for the struc-100 tured output prediction nature of the human pose esti-101 mation problem , leading to the keypoint attraction maps102 ( KAMs ) . The second computes global KEMs and learns103 to refine the global KEMs by leveraging the KAMs.104 Our LOGO-CAP is a fully end-to-end bottom-up hu-105 man pose estimation method with near real-time infer-106 ence speed . It obtains 70.0 AP in the fully-annotated sub-107 set of the COCO val-2017 dataset , which is an absolute108 increase of 9.9 AP compared to the vanilla center-offset109 method , making a significant step forward . Fig . 1 shows a110 pose estimation example . Fig . 2 shows the advantage of the proposed LOGO-CAP in terms of over-111 all speed-accuracy comparisons between our LOGO-CAP and prior arts . Meanwhile , we should112 notice that there is also a significant gap compared to the empirical upper bound ( Table 1 ) , which113 encourages more work to be investigated.114 2 Related Works and Our Contributions115 There is a vast body of literature for human pose estimation . Many elegant representation schema116 have been developed for modeling articulated human pose in the traditional approaches such as the117 well-known pictorial structure model [ 10 , 8 ] and its many variants [ 24 , 1 , 23 , 33 , 25 ] . Most of them118 focused on single person pose estimation . They perform inference over a combination of local ob-119 servations on body parts ( i.e. , the data term ) and the spatial dependencies between them ( i.e. , the120 spring or clique term ) . The spatial dependencies are captured either using directed and acyclic struc-121 tures that facilitate the global optimization by dynamic programming [ 2 , 9 ] , or using structures with122 loop introduced ( for high-order part relationship modeling ) which resort to approximate inference123 by loopy belief propagation [ 19 ] . The bottleneck of the traditional methods lies in the data term124 which is often based on hand-crafted features . With the resurgence of DNNs and the end-to-end125 learning , the data term has been largely improved . We briefly review the recent deep learning based126 approaches for bottom-up human pose estimation.127 Limb-based Grouping Approaches have been extensively developed due to the naturalness of128 modeling limbs based on keypoints . Given a predefined limb configuration ( e.g. , the COCO person129 skeleton template consisting of 19 limbs based on 17 keypoints ) , the grouping can be addressed by130 Part affinity field ( PAF ) [ 4 , 3 ] , Associative Embedding ( AE ) [ 20 ] , mid-range offset fields in Person-131 Lab [ 22 ] and the fields of Part Intensity and Association [ 15 ] . Typically , sophisticated designs are132 entailed to achieve good performance . For example , a bipartite graph matching is used in Open-133 Pose [ 3 ] . In addition to be computationally expensive , another drawback of these methods is not134 fully end-to-end trainable . More recently , the differentiability issue was studied by the Hierarchical135 Graph Clustering ( HGG ) method [ 13 ] , which utilizes graph convolution networks to repeatedly de-136 lineate pose parameters of multiple persons from a keypoint graph . HGG improves the performance137 compared to its baseline , the Associative Embedding method [ 20 ] at the expense of significantly138 increased computational cost . In contrast to thoses approaches , our proposed LOGO-CAP is fully139 end-to-end trainable and achieves near real-time inference speed.140 Direct Regression based Approaches have attracted much attention due to their conceptually sim-141 ple formulation [ 6 , 28 , 26 , 11 , 30 ] . These center-offset based formulation are inspired by the re-142 cent remarkable success of direct bounding box regression in object detection such as the FCOS143 method [ 29 ] and CenterNets [ 35 , 6 ] . As aforementioned , one main challenge is the difficulty of ac-144 curately regress the offset vectors , especially for the long-range keypoints with respect to the center.145 Sophisticated post-processing schema are often entailed to improve the performance . For example,146 a method of matching the directly regressed poses to the nearest keypoints that are extracted from147 the global keypiont heatmaps is used in [ 35 ] . Although being simple , the performance of this line of148 work is usually inferior to the limb-based approaches . The mixture regression network [ 30 ] allevi-149 ated the issue of regression quality to some extent , but still remained an indispensable performance150 gap comparing with the grouping-based approaches . Most recently , Geng et al . presented the first151 competitive direct method , DEKR [ 11 ] with a novel pose-specific neural architecture for disentan-152 gled keypoint regression . To improve the performance , the DEKR method utilizes a lightweight153 rescoring network to recalibrate the pose scores that are computed based on the keypoint heatmaps.154 Despite good performance , the DEKR method entails the additional rescoring stage in both training155 and testing , and thus is not fully end-to-end . The proposed LOGO-CAP retains the simplicity of the156 vanilla center-offset formulation and enjoys fully end-to-end training and fast inference speed.157 Our Contributions . The proposed LOGO-CAP makes three main contributions to the field of158 bottom-up human pose estimation : ( i ) It addresses the drawback of the vanilla center-offset for-159 mulation while retaining its efficiency . It proposes the key idea of lifting a keypoint to a keypoint160 expansion map to counter the lack of localization accuracy . To our knowledge , it is the first fully161 end-to-end trainable method that achieves state-of-the-art performance . ( ii ) It presents a novel local-162 global contextual adaptation formulation that accounts for the nature of structured output predic-163 tion in human pose estimation and harnesses local-global structural information integration . ( iii ) It164 obtains state-of-the-art performance in the COCO val-2017 and test-2017 datasets . It also shows165 state-of-the-art transferability performance in the OCHuman dataset.166 3 Approach167 3.1 Problem Formulation168 We follow the COCO protocol of defining the human pose . It consists of 17 human pose keypoints:169 8 pairs of symmetric keypoints ( hips , ankles , knees , shoulders , elbows , wrists , ears and eyes ) and170 the nose keypoint . Let P = { 1 , · · · 17 } be the set of keypoint indexes using a predefined order.171 Let Λ be an image lattice of the spatial size H ×W ( e.g. , 512 × 512 ) , and I be an image defined172 on Λ . Let PnI be the set of keypoint indexes for a human pose instance n in an image I and we173 have PnI ⊆ P . For example , in COCO , we typically have 1 ≤ n ≤ 30 , and different human pose174 instances have different number of visible keypoints due to occlusion and/or truncation . Denote by175 LnI = { ( xi , yi ) ; i ∈ PnI } the keypoint locations of a human pose instance n in an image I , where176 ( xi , yi ) ∈ Λ . In the center-offset formulation , we introduce the keypoints center ( i.e. , the anchor ) ,177 ( xc , yc ) based on a given LnI and we have,178 xc = 1/|LnI | · ∑ i∈PnI xi , yc = 1/|LnI | · ∑ i∈PnI yi . ( 1 ) With the anchor , a keypoint ( xi , yi ) is equivalently defined by its offset/displacement , denoted by179 ( ∆xi , ∆yi ) with ∆xi = xi − xc and ∆yi = yi − yc . So , LnI can also be equivalently expressed as180 LnI = { ( xc , yc ) , ( ∆xi , ∆yi ) ; i ∈ PnI } .181 The objective of human pose estimation is to recover LnI = { ( xi , yi ) ; i ∈ PnI } for all human pose182 instances in an image . Denote by L̂nI = { ( x̂i , ŷi ) ; i ∈ PnI } the estimated human pose . Following183 the COCO protocol , the object keypoint similarity ( OKS ) is used to evaluate the accuracy,184 ` OKS ( L̂ n I , L n I ) = 1/|PnI | · ∑ i∈PnI exp ( −d2i /2s2κ2i ) , ( 2 ) where di is the Euclidian distance between the ground-truth keypoint ( xi , yi ) and the predicted one185 ( x̂i , ŷi ) . s is the square root of the human segment area , and κ per-keypoint constant that controls186 fall-off in evaluation . We have ` OKS ( L̂nI , L n I ) ∈ [ 0 , 1 ] . The OKS metric is to evaluate the distance187 between predicted keypoints and ground-truth keypoints normalized by the scale of the person with188 the importance of keypoints equalized . In benchmarking different methods , the average precision189

Summary: This paper proposed a local-global contextual adaptation method for the bottom-up human pose estimation.  In the proposed method, local keypoint expansion maps (KEMs) and global KEMs are both extracted. The pose-sensitive kernels (keypoint attraction maps, KAMs) are learned from the local KEMs and then performed to the global KEMs. The final prediction is generated from the refined global KEMs. The proposed method is end-to-end trainable and achieves state-of-the-art performance on COCO and OCHuman datasets.