Learning Local-Global Contextual Adaptation for Fully End-to-End Bottom-Up Human Pose Estimation
Year: 2021

Abstract:
This paper presents a method of learning LOcal-GlObal Contextual Adaptation for 1 fully end-to-end and fast bottom-up human Pose estimation, dubbed as LOGO2 CAP. It is built on the conceptually simple center-offset formulation that lacks 3 inaccuracy for pose estimation. When revisiting the bottom-up human pose es4 timation with the thought of “thinking, fast and slow” by D. Kahneman, we in5 troduce a “slow keypointer” to remedy the lack of sufficient accuracy of the “fast 6 keypointer”. In learning the “slow keypointer”, the proposed LOGO-CAP lifts the 7 initial “fast” keypoints by offset predictions to keypoint expansion maps (KEMs) 8 to counter their uncertainty in two modules. Firstly, the local KEMs (e.g. 11×11) 9 are extracted from a low-dimensional feature map. A proposed convolutional mes10 sage passing module learns to “re-focus” the local KEMs to the keypoint attraction 11 maps (KAMs) by accounting for the structured output prediction nature of human 12 pose estimation, which is directly supervised by the object keypoint similarity 13 (OKS) loss in training. Secondly, the global KEMs are extracted, with a suffi14 ciently large region-of-interest (e.g., 97 × 97), from the keypoint heatmaps that 15 are computed by a direct map-to-map regression. Then, a local-global contextual 16 adaptation module is proposed to convolve the global KEMs using the learned 17 KAMs as the kernels. This convolution can be understood as the learnable offsets 18 guided deformable and dynamic convolution in a pose-sensitive way. The pro19 posed method is end-to-end trainable with near real-time inference speed, obtain20 ing state-of-the-art performance on the COCO keypoint benchmark for bottom-up 21 human pose estimation. With the COCO trained model, our LOGO-CAP also 22 outperforms prior arts by a large margin on the challenging OCHuman dataset. 23

text: This paper presents a method of learning LOcal-GlObal Contextual Adaptation for1 fully end-to-end and fast bottom-up human Pose estimation, dubbed as LOGO-2 CAP. It is built on the conceptually simple center-offset formulation that lacks3 inaccuracy for pose estimation. When revisiting the bottom-up human pose es-4 timation with the thought of “thinking, fast and slow” by D. Kahneman, we in-5 troduce a “slow keypointer” to remedy the lack of sufficient accuracy of the “fast6 keypointer”. In learning the “slow keypointer”, the proposed LOGO-CAP lifts the7 initial “fast” keypoints by offset predictions to keypoint expansion maps (KEMs)8 to counter their uncertainty in two modules. Firstly, the local KEMs (e.g. 11×11)9 are extracted from a low-dimensional feature map. A proposed convolutional mes-10 sage passing module learns to “re-focus” the local KEMs to the keypoint attraction11 maps (KAMs) by accounting for the structured output prediction nature of human12 pose estimation, which is directly supervised by the object keypoint similarity13 (OKS) loss in training. Secondly, the global KEMs are extracted, with a suffi-14 ciently large region-of-interest (e.g., 97 × 97), from the keypoint heatmaps that15 are computed by a direct map-to-map regression. Then, a local-global contextual16 adaptation module is proposed to convolve the global KEMs using the learned17 KAMs as the kernels. This convolution can be understood as the learnable offsets18 guided deformable and dynamic convolution in a pose-sensitive way. The pro-19 posed method is end-to-end trainable with near real-time inference speed, obtain-20 ing state-of-the-art performance on the COCO keypoint benchmark for bottom-up21 human pose estimation. With the COCO trained model, our LOGO-CAP also22 outperforms prior arts by a large margin on the challenging OCHuman dataset.23
1 Introduction24 1.1 Motivation and Objective25 Human pose is highly articulated with large structural and appearance variations. 2D human pose26 estimation in images is a classic structured output prediction problem, and remains a challenging one27 in computer vision and machine learning. Human pose estimation has numerous applications such28 as people-centered image understanding, autonomous driving and Augmented Reality (AR). With29 the recent resurgence of deep neural networks (DNNs), the performance of human pose estimation30 has witnessed remarkable improvement [12, 3, 15, 22, 11]. This paper focuses on the deep learning31 based problem formulation.32
There are two deep learning based paradigms for human pose estimation in the literature. The top-33 down paradigm consists of human detection and single human pose estimation in each detected34 human bounding box [12]. The bottom-up paradigm also includes two components: human pose35 keypoint detection and keypoint grouping [3]. The top-down paradigm often obtains better accuracy36 performance, but suffers from its inferior efficiency since the computational cost of the single human37
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
pose estimation component is linearly scaled with respect to the number of detected human bounding38 boxes in an image. It is also largely affected by the performance of the human detection component39 (e.g., not handling occlusion very well). Thanks to its efficiency, especially in real-time applications,40 the bottom-up paradigm becomes more and more attractive. For both paradigms, state-of-the-art41 methods often are not fully end-to-end trained and utilize different post-hoc processing modules42 to improve the performance. This paper is interested in developing a fully end-to-end bottom-up43 paradigm and aims at bridging its performance gap with the top-down paradigm.44
For the bottom-up paradigm, the recently proposed center-offset approach [6, 28, 26, 11] is a con-45 ceptually simple formulation (see the left of Fig. 1 for an illustrative example and Fig. 3 for the46 detailed workflow). It alleviates the need of sophisticated keypoint grouping. When introducing47 human keypoints centers (i.e., anchors) by treating objects as points [35], it encodes a human pose48 as a star structure using the offset vectors of human keypoints relative to the anchors. The main49 challenge of the center-offset regression paradigm lies in the difficulty of accurately learning offset50 vectors with large structural variations, especially the long-range ones, which also leads to inferior51 performance. This paper builds on the center-offset approach and addresses its drawback.52
1.2 Method Overview53 To address the drawback of the center-offset formulation, we build the intuitive idea of54 “Keypointing, fast and slow”, by analogy to the modes of thought suggested by Daniel Kahne-55 man in “Thinking, fast and slow” [14]: (i) Fast Keypointer: We treat the vanilla center-offset based56 estimation [35] as the Fast Initializer of pose estimation. (ii) Slow Keypointer: The lack of localiza-57 tion accuracy in the Fast Initializer entails a Slow Solver that learns to refine the “fast” keypoints. By58 slow, it is only relatively speaking. The Slow Keypointer is actually fast with near real-time speed.59
Baseline Emp. Bound LOGO-CAP
AP 60.1 88.9 70.0 AP50 85.2 93.1 88.2 AP75 66.7 90.6 76.4 APM 53.7 87.7 64.4 APL 71.5 90.2 78.4
clearly shows that the pose keypoints center and the offset vectors can be learned reasonably well.73 Instead of directly utilizing the learned offset vectors for human pose estimation, we treat them as74 human pose keypoint initialization and do a local window search to compute the empirical upper-75 bound of performance. More detailed, based on the predicted human poses, by introducing a local76 window (e.g., 11 × 11) centered at each detected key point and by computing the single keypoint77 similarity with the ground-truth keypoint, an empirical upper-bound of 88.9 AP is obtained, which78 is significantly higher than the state of the art and shows the potential of improving the vanilla79 center-offset regression paradigm.80
Motivated by the above observation, a straightforward way is just to learn a local heatmap (e.g.,81 11 × 11) for each human pose keypoint based on the learned center and offset vectors, and then to82
compute the refined keypoints by taking arg max within the local heatmap. Although appealing,83 this does not work as observed during our development of the LOGO-CAP. The underlying reason84 is easy to understand: if this can work, the original offset vector regression should work at the85 first place since no additional information is introduced through learning the local heatmap. We86 hypothesize that on the one hand, on top of the local heatmap, the structural relationship between87 different keypoints of a human pose needs to be taken into account, and on the other hand, the88 intrinsic uncertainty of the local information in a local heatmap needs to be resolved. The former89 is the key challenge of structured output prediction problems. Many message passing algorithms90 have been developed in the literature. The latter can not be addressed by simply increasing the local91 window size. It entails learning stronger local-global information interaction and adaptation,.92
Along with the two hypotheses, the proposed LOGO-93 CAP lifts the initial keypoints via the center-offset pre-94 diction to keypoint expansion maps (KEMs) to counter95 their lack of localization accuracy in two modules (Sec-96 tion 3.2). The KEMs extend the star-structured repre-97 sentation of the center-offset formulation to the pictorial98 structure representation [10, 8]. The first module com-99 putes local KEMs and learns to account for the struc-100 tured output prediction nature of the human pose esti-101 mation problem, leading to the keypoint attraction maps102 (KAMs). The second computes global KEMs and learns103 to refine the global KEMs by leveraging the KAMs.104
Our LOGO-CAP is a fully end-to-end bottom-up hu-105 man pose estimation method with near real-time infer-106 ence speed. It obtains 70.0 AP in the fully-annotated sub-107 set of the COCO val-2017 dataset, which is an absolute108 increase of 9.9 AP compared to the vanilla center-offset109 method, making a significant step forward. Fig. 1 shows a110
pose estimation example. Fig. 2 shows the advantage of the proposed LOGO-CAP in terms of over-111 all speed-accuracy comparisons between our LOGO-CAP and prior arts. Meanwhile, we should112 notice that there is also a significant gap compared to the empirical upper bound (Table 1), which113 encourages more work to be investigated.114
2 Related Works and Our Contributions115 There is a vast body of literature for human pose estimation. Many elegant representation schema116 have been developed for modeling articulated human pose in the traditional approaches such as the117 well-known pictorial structure model [10, 8] and its many variants [24, 1, 23, 33, 25]. Most of them118 focused on single person pose estimation. They perform inference over a combination of local ob-119 servations on body parts (i.e., the data term) and the spatial dependencies between them (i.e., the120 spring or clique term). The spatial dependencies are captured either using directed and acyclic struc-121 tures that facilitate the global optimization by dynamic programming [2, 9], or using structures with122 loop introduced (for high-order part relationship modeling) which resort to approximate inference123 by loopy belief propagation [19]. The bottleneck of the traditional methods lies in the data term124 which is often based on hand-crafted features. With the resurgence of DNNs and the end-to-end125 learning, the data term has been largely improved. We briefly review the recent deep learning based126 approaches for bottom-up human pose estimation.127
Limb-based Grouping Approaches have been extensively developed due to the naturalness of128 modeling limbs based on keypoints. Given a predefined limb configuration (e.g., the COCO person129 skeleton template consisting of 19 limbs based on 17 keypoints), the grouping can be addressed by130 Part affinity field (PAF) [4, 3], Associative Embedding (AE) [20], mid-range offset fields in Person-131 Lab [22] and the fields of Part Intensity and Association [15]. Typically, sophisticated designs are132 entailed to achieve good performance. For example, a bipartite graph matching is used in Open-133 Pose [3]. In addition to be computationally expensive, another drawback of these methods is not134 fully end-to-end trainable. More recently, the differentiability issue was studied by the Hierarchical135 Graph Clustering (HGG) method [13], which utilizes graph convolution networks to repeatedly de-136 lineate pose parameters of multiple persons from a keypoint graph. HGG improves the performance137 compared to its baseline, the Associative Embedding method [20] at the expense of significantly138
increased computational cost. In contrast to thoses approaches, our proposed LOGO-CAP is fully139 end-to-end trainable and achieves near real-time inference speed.140
Direct Regression based Approaches have attracted much attention due to their conceptually sim-141 ple formulation [6, 28, 26, 11, 30]. These center-offset based formulation are inspired by the re-142 cent remarkable success of direct bounding box regression in object detection such as the FCOS143 method [29] and CenterNets [35, 6]. As aforementioned, one main challenge is the difficulty of ac-144 curately regress the offset vectors, especially for the long-range keypoints with respect to the center.145 Sophisticated post-processing schema are often entailed to improve the performance. For example,146 a method of matching the directly regressed poses to the nearest keypoints that are extracted from147 the global keypiont heatmaps is used in [35]. Although being simple, the performance of this line of148 work is usually inferior to the limb-based approaches. The mixture regression network [30] allevi-149 ated the issue of regression quality to some extent, but still remained an indispensable performance150 gap comparing with the grouping-based approaches. Most recently, Geng et al. presented the first151 competitive direct method, DEKR [11] with a novel pose-specific neural architecture for disentan-152 gled keypoint regression. To improve the performance, the DEKR method utilizes a lightweight153 rescoring network to recalibrate the pose scores that are computed based on the keypoint heatmaps.154 Despite good performance, the DEKR method entails the additional rescoring stage in both training155 and testing, and thus is not fully end-to-end. The proposed LOGO-CAP retains the simplicity of the156 vanilla center-offset formulation and enjoys fully end-to-end training and fast inference speed.157
Our Contributions. The proposed LOGO-CAP makes three main contributions to the field of158 bottom-up human pose estimation: (i) It addresses the drawback of the vanilla center-offset for-159 mulation while retaining its efficiency. It proposes the key idea of lifting a keypoint to a keypoint160 expansion map to counter the lack of localization accuracy. To our knowledge, it is the first fully161 end-to-end trainable method that achieves state-of-the-art performance. (ii) It presents a novel local-162 global contextual adaptation formulation that accounts for the nature of structured output predic-163 tion in human pose estimation and harnesses local-global structural information integration. (iii) It164 obtains state-of-the-art performance in the COCO val-2017 and test-2017 datasets. It also shows165 state-of-the-art transferability performance in the OCHuman dataset.166
3 Approach167 3.1 Problem Formulation168 We follow the COCO protocol of defining the human pose. It consists of 17 human pose keypoints:169 8 pairs of symmetric keypoints (hips, ankles, knees, shoulders, elbows, wrists, ears and eyes) and170 the nose keypoint. Let P = {1, · · · 17} be the set of keypoint indexes using a predefined order.171 Let Λ be an image lattice of the spatial size H ×W (e.g., 512 × 512), and I be an image defined172 on Λ. Let PnI be the set of keypoint indexes for a human pose instance n in an image I and we173 have PnI ⊆ P . For example, in COCO, we typically have 1 ≤ n ≤ 30, and different human pose174 instances have different number of visible keypoints due to occlusion and/or truncation. Denote by175 LnI = {(xi, yi); i ∈ PnI } the keypoint locations of a human pose instance n in an image I , where176 (xi, yi) ∈ Λ. In the center-offset formulation, we introduce the keypoints center (i.e., the anchor),177 (xc, yc) based on a given LnI and we have,178
xc = 1/|LnI | · ∑ i∈PnI xi, yc = 1/|LnI | · ∑ i∈PnI yi. (1)
With the anchor, a keypoint (xi, yi) is equivalently defined by its offset/displacement, denoted by179 (∆xi,∆yi) with ∆xi = xi − xc and ∆yi = yi − yc. So, LnI can also be equivalently expressed as180 LnI = {(xc, yc), (∆xi,∆yi); i ∈ PnI }.181 The objective of human pose estimation is to recover LnI = {(xi, yi); i ∈ PnI } for all human pose182 instances in an image. Denote by L̂nI = {(x̂i, ŷi); i ∈ PnI } the estimated human pose. Following183 the COCO protocol, the object keypoint similarity (OKS) is used to evaluate the accuracy,184
`OKS(L̂ n I , L n I ) = 1/|PnI | · ∑ i∈PnI exp (−d2i /2s2κ2i ), (2)
where di is the Euclidian distance between the ground-truth keypoint (xi, yi) and the predicted one185 (x̂i, ŷi). s is the square root of the human segment area, and κ per-keypoint constant that controls186 fall-off in evaluation. We have `OKS(L̂nI , L n I ) ∈ [0, 1]. The OKS metric is to evaluate the distance187 between predicted keypoints and ground-truth keypoints normalized by the scale of the person with188 the importance of keypoints equalized. In benchmarking different methods, the average precision189

heading: Keypoint offset field
text: (AP) at OKS= 0.50 : 0.05 : 0.95 is used as the primary metric, together with AP 50 at OKS= 0.50,190 AP 75 at OKS= 0.75, and AP across medium and large scales, APM and APL respectively.191
3.2 The Proposed LOGO-CAP192 We first present the network and the inference of LOGO-CAP, and then give details of the train-193 ing. We keep different modules of the proposed LOGO-CAP simple, which in turn highlights the194 effectiveness of the proposed representation and algorithmic flow.195
3.2.1 The Network and the Inference196 As illustrated in Fig. 3, the proposed LOGO-CAP consists of four components as follows.197
i) A convolution neural network feature backbone. Given an input image I , the output of the fea-198 ture backbone is a C-dim feature map, denoted by F ∈ RC×h×w, where C is the feature dimension199 of the last convolutional layer in the feature backbone, and the spatial size h × w depends on the200 total stride in the feature backbone. We use off-the-shelf HRNets [27] in our experiments.201
ii) A parallel keypoint-offset regression module. Given the feature map F , the output of keypoint202 regression is an 18-dim feature map (i.e., heatmaps) for the 17 keypoints and the keypoints center203 respectively. Denote by H ∈ R18×h×w the heatmaps, and by H↑ ∈ R18×H×W the up-sampled204 heatmaps (using bi-linear interpolation in our experiments). The output of offset regression is a205 34-dim feature map (i.e., the offset vector fields) for the 17 keypoints. Denote by O ∈ R34×h×w206 the offset fields. We adopt a minimally-simple design in realizing the regression modules using a207 channel-wise multi-layer perceptron (MLP). In implementation, we first apply dimension reduction208 to the feature map F using a 1× 1 convolution followed by a Batch Normalization (BN) and a Rec-209 tified Linear Unit (ReLU). Then, the output is computed by a 1× 1 convolution. More specifically,210 we have the two parallel branches as follows,211
FC×h×w Conv+BN+ReLU ============⇒ C×1×1×C1 FHC1×h×w Conv =======⇒ C1×1×1×18 H18×h×w UpSampling ========⇒ bi-linear H↑18×H×W , (3)
FC×h×w Conv+BN+ReLU ============⇒ C×1×1×C2 FOC2×h×w Conv =======⇒ C2×1×1×34 O34×h×w, (4)
where C1 and C2 are predefined (e.g., C1 = 32 and C2 = 256 are typically used).212
Initial pose estimation via the center-offset approach. Based on the computed keypoints center213 heatmap H↑(18) and offset fields O, a predefined maximum number of pose candidates is computed214 as done in the vanilla center-offset approach. A non-maximum suppression (NMS) with a 3 × 3215 window is applied in H↑(18) and then the top-N keypoints centers are selected (e.g., N = 30 in our216 experiments). The N pose instances are computed by retrieving their offset vectors in O based on217 the selected N keypoints centers. The N pose instances are further pruned by thresholding their218 confidence scores inH↑(18) with a predefined threshold (e.g., 0.01 used in our experiments). Without219 confusion in the context, we still use N to denote the number of poses instances by this initial pose220 estimation step. We obtain the set of estimated keypoints centers, denoted by CN×3 each row of221 which represents the position coordinates and the confidence score.222
Lifting a keypoint to a keypoint expansion map (KEM) by imposing a mesh. For each of the N223 pose instances, each of the 17 keypoints are placed in a local geometric mesh (e.g., 11×11) with the224 estimated location as the mesh center, capturing the uncertainty of the center-offset pose estimation225 as aforementioned in the introduction. This mesh can thus be interpreted as keypoint expansion226 map (KEM), accounting for competency-aware representations. The entire mesh is denoted by227 MN×17×11×11×2, which is used in computing the empirical upper bound in Table 1. We have,228
{H↑(18),O34×h×w} initial pose estimation ===========⇒
center-offset {CN×3,MN×17×11×11×2} (5)
iii) A convolution message passing module. We first encode the geometric meshMN×17×11×11×2229 in a latent space with the dimensionality C3 (e.g., 64 in our experiments), computed based on the230 feature backbone output. Then, a keypoint is represented by a C3 × 11 × 11 local feature map. A231 pose instance is represented by concatenating all the 17 keypoints. We have,232
FC×h×w Conv+BN+ReLU ============⇒ C×1×1×C3 FMC3×h×w MN×17×11×11×2 ===========⇒ bi-linear KN×(17×C3)×11×11, (6)
where the bi-linear interpolation is used due to the sub-pixel based locations in the mesh and for233 better feature alignment.234
To facilitate the structural information flow between different latent codes of the keypoints of a pose235 instance, we propose a simple convolutional message passing (CMP) module with three layers of236 Conv+BN+ReLU operations,237
KN×(17×C3)×11×11 ⇒ [ Conv+BN+ReLU ============⇒ Cin×3×3×Cout ]×3 ⇒ · Conv =======⇒ C6×1×1×17 KN×17×11×11, (7)
where Cin ∈ {(17× C3), C4, C5} and Cout ∈ {C4, C5, C6} (e.g., C4 = 512, C5 = 256, C6 = 128238 in our experiments). The resulting KN×17×11×11 can be interpreted as keypoint attraction maps239 (KAMs) which are “re-focused” based on the KEMs by the CMP. To account for the specificity of240 different pose instances in the CMP, we adopt the Attention Normalization [17] to replace the BN in241 the second Conv+BN+ReLU layer, which further improves the performance in our experiments.242
Through the CMP, we obtain the dynamic (a.k.a., data-driven) kernels for the 17 keypoints in a pose243 instance-sensitive way, which are used to refine the global heatmapsH↑ for the 17 keypoints.244 iv) A local-global contextual adaptation module. We first compute another geometric mesh with245 enlarged mesh window a×a (e.g., a = 97) for each keypoint of theN pose instances, and the entire246 mesh is denoted byMLN×17×a×a×2, as done in Eqn. 5. The mesh can be interpreted as the global247 KEM. It is then instantiated with appearance features extracted from the global heatmaps H↑(1:17),248 similar to Eqn. 6, and we have,249
H↑(1:17) MLN×17×a×a×2 =========⇒
bi-linear HN×17×a×a
Ga×a(0,σ) ======⇒
reweighing H̄N×17×a×a. (8)
where to encode the Gaussian prior of keypoint heatmaps, the resulting pose-guided heatmaps H is250 reweighed by a Gaussian kernel Ga×a(0, σ = a−12×3 ) (e.g., σ = 16 when a = 97) in an element-wise251 way. By doing so, it means that the enlarged mesh follows the 3σ principle.252
Then, we apply the learned keypoint 11 × 11 kernels Kn,i’s (Eqn. 7) to convolve the reweighed253 a × a heatmap H̄n,i (Eqn. 8) in a pose instance-sensitive and keypoint-specific way, leading to254 LOcal-GlObal Contextual Adaptation,255
H̄N×17×a×a KN×17×11×11 ========⇒
LOGO-CA H̃N×17×a×a, (9)
which represents the refined heatmaps for the 17 human pose keypoints.256
The Pose Estimation Output. With the local-global contextually adpated heatmaps H̃N×17×a×a,257 we maintain the top-2 locations for each keypoint within the a×a heatmap, and then utilize a convex258 average of the top-2 locations as the final predicted offset vectors (i.e. (∆x′i,∆y ′ i)’s in Fig. 3), and259 of their confidence scores as the prediction score, with a predefined weight λ for the top-1 location260 (0.75 in our experiments). Together with the predicted keypoints centers CN×3 (Eqn. 5), the final261 prediction score for each keypoint is the product between the convex average confidence score and262 the center confidence score. We keep the keypoints whose final scores are greater than 0. We have,263
{CN×3, H̃N×17×a×a} Output
=========⇒ Score thresholding {L̂nI ;n = 1, · · ·N ′}, (10)
where N ′ is the number of the final predicted pose instances in an image I .264
3.2.2 Loss Functions in Training265 In the fully end-to-end training, we need to define loss functions for the global heatmapH (Eqn. 3),266 the refined local heatmap H̃ (Eqn. 9), the offset field O (Eqn. 4), and the keypoint kernels (Eqn. 7).267 The Heatmap Loss. The widely adopted mean squared error (MSE) loss is used. Denoted by268 HGT18×h×w the ground truth heatmaps in which each keypoint (including the center) is modeled by a269 2-D Gaussian with dataset-provided mean and variance. Let p = (i,x) be the index of the domain270 D of dimensions 18× h× w. For the predicted heatmapsH18×h×w, the MSE loss is defined by,271
LH = 1/|D| · ∑ p∈D ‖w(x)(H(p)− Ĥ(p))‖22, (11)
where w(x) represents the weight for the foreground and the background pixels. The foreground272 mask is provided by the dataset annotation. In our experiment, we set w(x) = 1 for a foreground273 pixel and w(x) = 0.1 for a background pixel.274
In defining the loss function LH̃ for the refined local heatmap H̃ (Eqn. 9), the ground-truth heatmap275 H̃GT is generated on-the-fly based on the mesh MLN×17×a×a (Eqn. 8) and the ground-truth key-276 points using a Gaussian model with mean being the displacement between the current predicted277 keypoints and the ground-truth ones, and variance σ (i.e., the standard deviation of the reweighing278 Gaussion prior model in Eqn. 8).279
The Offset Field Loss. The widely adopted SmoothL1 loss [] is used. LetOGT34×h×w be the ground-280 truth offset field, and CGT be the non-empty set of ground-truth keypoints centers (Eqn. 1). For the281 predicted offset field O34×h×w (Eqn. 4), we have,282
LO = 1/|CGT | · ∑
p∈CGT A(p) · SmoothL1
( O(·,p),OGT (·,p);β ) , (12)
whereA(p) is the area of the person centered at the pixel p, and β the cutting-off threshold (e.g., 19 in283 our experiments), and SmoothL1(a, b;β) = 0.5×|a−b|2/β if |a−b| ≤ β, otherwise |a−b|−0.5×β.284 The OKS Loss for the Keyoint Kernels. Consider a single predicted pose instance, learning the285 keypoint kernels, K17×11×11 (Eqn. 7) is the key to facilitate the local-global contextual adaptation.286 To that end, the figure of merits of the KEF,M17×11×11×2 (Eqn. 5) needs to directly reflect the task287 loss, i.e., the OKS loss (Eqn. 2). With respect to the NGT ground-truth pose instances in an image,288 we can compute the similarity score per keypoint candidate in the KEF, and obtain the score tensor289 S17×11×11×NGT . The score tensor is further clamped with a threshold 0.5, i.e., S17×11×11×NGT =290 max(S17×11×11×NGT , 0.5). A mean reduction is applied to the first three dimensions of the clamped291 score tensor to compute the matching score for each of the NGT pose instance. Then, the best292 ground-truth pose instance indexed by n∗ is selected in terms of the matching score, and its matching293 score is denoted by sn∗ . Based on the selected ground-truth pose instance, we compute the per-294 keypoint similarity score for the predicted pose instance at hand, denoted by sk (k ∈ [1, 17]). Then,295 the loss function fo the keypoint kernels are defined by,296
LK = sn∗ · ∑ k,i,j sk · |Kk,i,j − Sk,i,j,n∗ |2. (13)
The Total Loss is then defined by L = LH + LH̃ + λ · (LO + LK), where the trade-off parameter297 λ is used to balance the different loss items (λ = 0.01 in our experiments).298
4 Experiments299 In this section, we present detailed experimental results and analyses of the proposed LOGO-CAP.300 Our PyTorch source code will be released for reproducibility.301
Datasets. We use two datasets in our experiments: The COCO dataset [18] is the most popu-302 lar testbed for human pose estimation. It consists of 65k, 5k and 20k images with human pose303 well-annotated in the training, validation and testing datasets respectively. In all experiments, the304 proposed LOGO-CAP is trained using the 65k training images. The OCHuman dataset [34] is one305 popular testing-only dataset for evaluating human pose estimation under the occlusion scenarios. It306 consists of a total number of 4713 images with 8110 detailed annotated human pose instances using307 the COCO keypoint configuration. All the annotated 8110 human pose instances have occlusions308 with the maxIOU≥ 0.5. Furthermore, 32% instances are more challenging with the maxIOU≥ 0.75.309
4.1 Results on the COCO dataset310 Fig. 4 shows some qualitative examples of human pose estimation by the proposed LOGO-CAP.311 More examples will be provided in the supplementary material.312
The proposed LOGO-CAP is compared with prior arts including OpenPose [3], PifPaf [15], Person-313 Lab [22], AE [20] and DEKR [11]. As reported in Table 2, the proposed LOGO-CAP outperforms314 all of them on both both validation and test-dev datasets.315
In comparisons to the best-performing grouping approach, AE [20] with a larger backbone316 HrHRNet-W48 [5], our LOGO-CAP obtains competitive performance with a smaller HRNet-32317 backbone, and improves the AP score with HRNet-W48 backbone on the validation and testdev318 datasets by 2.3 and 2.5 points, respectively. For the fully differentiable grouping approach319 HGG [13], our LOGO-CAP achieves better performance by a significantly large margin, more than320 9.2 points on the validation set under the single-scale testing. Although the performance of HGG321 is improved by the multi-scale testing on the test-dev set, the performance of our LOGO-CAP is still322 significantly better without using the multi-scale testing scheme.323
In comparisons to the direct regression based approaches, our LOGO-CAP obtains the best results324 without incurring either the matching scheme used in CenterNet [35] or the additional rescoring325 network used in DEKR [11]. When we disable the rescoring network for DEKR [11] for fair com-326 parisons, our LOGO-CAP significantly improves the AP on the validation and testdev datasets by327 2.4 points and 1.6 points respectively when HRNet-W32 is used as backbone. The larger back-328 bone is beneficial for both DEKR and our method, which further improves the AP score of our329 LOGO-CAP to 72.2 and 70.8 on the validation and test-dev dataset respectively, outperforming330 DEKR by 1.9 and 1.5 respectively.331
4.2 Results on the OCHuman dataset332 Table 3 shows that our LOGO-CAP achieves the best AP performance on both the validation and333 testing datasets by significant margins of 2.4 and 2.2 points in comparing with the bottom-up334 approaches. For the top-down approaches, although they obtain strong AP scores on the validation335 split, there exists a large performance gap between the validation and testing sets. In comparisons336 to DEKR [11] (with the rescoring network), our LOGO-CAP improves the performance from 37.9337 to 39.0 and from 36.5 to 38.1 on the validation and testing splits with the same backbone HRNet-338 W32, respectively. The similar improvement is observed when the HRNet-W48 backbone is used,339 outperforming both bottom-up and top-down approaches.340
4.3 Inference Speed341 In comparing the inference speed, we test all the models on a single TITAN RTX GPU for its342 popularity in practice. The average inference speed, FPS (frames per second), over the 5000 images343 in COCO-val-2017 is used for the comparison. For DEKR [11], we re-implement their inference344 code with better speed obtained for fair comparisons at the algorithm level. For methods that have345 post-processing schema on CPU, only one thread is used. As shown in Table 4, our LOGO-CAP346 runs significantly faster than PifPaf [15] and AE [20]. The CenterNet [35] runs slower than DEKR347 and our LOCO-CAP as it requires a post-processing scheme to match the predicted offsets to the348 keypoints obtained from heatmaps. Comparing with DEKR, the speed improvement of our LOGO-349 CAP is from the lightweight design of head modules since the same backbones are used. For the350 comparisons in Table 2, we run the models with different resolutions of testing images.351
4.4 Potentials and Limitations of the Proposed LOGO-CAP352 Consider the generic applicability of the center-offset formulation to many computer vision tasks as353 demonstrated in [35], we hypothesize that the proposed LOGO-CAP has a great potential to remedy354 the lack of sufficient accuracy using the vanilla center-offset method in those tasks. We also notice355 that the minimally-simple design in learning the “Slow Keypointer” can be relaxed for different356 accuracy-speed trade-offs in practice. For example, for the convolutional message passing module,357 an alternative method could be the Transformer model [31], which potentially will further improve358 the performance at the expense of inference speed. We leave these for future work.359
5 Conclusion360 This paper focuses on deep learning based formulation for bottom-up human pose estimation. It361 presents a method of learning LOcal-GlObal Contextual Adaptation for Pose estimation, dubbed as362 LOGO-CAP. The proposed LOGO-CAP is built on the conceptually simple center-offset paradigm363 and addresses its drawback of lacking the capability of accurately localizing human pose keypoints.364 The key idea of our LOG-CAP is to lift the center-offset predicted keypoints to keypoint expansion365 maps (KEMs),which counters the inaccuracy and uncertainty of the initial keypoints. Two types of366 KEMs are introduced in two parallel modules on top of the feature backbone. Local KEMs are used367 to learn keypoint attraction maps (KAMs) via a convolutional message passing module that accounts368 for the structured output prediction nature of human pose estimation. Global KEMs are used to369 learn local-global contextual adaptation which convolves global KEMs using the KAMs as kernels.370 The refined global KEMs are used in computing the final human pose estimation. The proposed371 LOGO-CAP obtains state-of-the-art performance in COCO val-2017 and test-dev 2017 datasets for372 bottom-up human pose estimation. It also achieves state-of-the-art transferability performance in373 the OCHuman dataset with the COCO trained models.374
References375 [1] Mykhaylo Andriluka, Stefan Roth, and Bernt Schiele. Pictorial structures revisited: People detection and376 articulated pose estimation. In 2009 IEEE conference on computer vision and pattern recognition, pages377 1014–1021. IEEE, 2009. 3378
[2] Richard Bellman. Dynamic programming. Science, 153(3731):34–37, 1966. 3379
[3] Zhe Cao, Gines Hidalgo Martinez, Tomas Simon, Shih-En Wei, and Yaser A. Sheikh. Openpose: Realtime380 multi-person 2d pose estimation using part affinity fields. IEEE Trans. on Pattern Analysis and Machine381 Intelligence (PAMI), 2019. 1, 3, 8382
[4] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using383 part affinity fields. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages384 1302–1310, 2017. 3385
[5] Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S. Huang, and Lei Zhang. Higherhrnet:386 Scale-aware representation learning for bottom-up human pose estimation. In IEEE/CVF Conference on387 Computer Vision and Pattern Recognition (CVPR), pages 5385–5394. IEEE, 2020. 3, 8, 9388
[6] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint389 triplets for object detection. In IEEE/CVF International Conference on Computer Vision (ICCV), pages390 6568–6577, 2019. 2, 4391
[7] Haoshu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. RMPE: regional multi-person pose estimation. In392 IEEE International Conference on Computer Vision (ICCV), pages 2353–2362, 2017. 9393
[8] Pedro F Felzenszwalb and Daniel P Huttenlocher. Pictorial structures for object recognition. International394 journal of computer vision, 61(1):55–79, 2005. 3395
[9] Pedro F Felzenszwalb and Ramin Zabih. Dynamic programming and graph algorithms in computer vision.396 IEEE transactions on pattern analysis and machine intelligence, 33(4):721–740, 2010. 3397
[10] Martin A Fischler and Robert A Elschlager. The representation and matching of pictorial structures. IEEE398 Transactions on computers, 100(1):67–92, 1973. 3399
[11] Zigang Geng, Ke Sun, Bin Xiao, Zhaoxiang Zhang, and Jingdong Wang. Bottom-up human pose es-400 timation via disentangled keypoint regression. In IEEE Conference on Computer Vision and Pattern401 Recognition (CVPR), 2021. 1, 2, 3, 4, 8, 9402
[12] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In IEEE International403 Conference on Computer Vision (ICCV), pages 2980–2988, 2017. 1404
[13] Sheng Jin, Wentao Liu, Enze Xie, Wenhai Wang, Chen Qian, Wanli Ouyang, and Ping Luo. Differentiable405 hierarchical graph grouping for multi-person pose estimation. In European Conference on Computer406 Vision (ECCV), volume 12352, pages 718–734, 2020. 3, 8407
[14] Daniel Kahneman. Thinking, fast and slow. Macmillan, 2011. 2408
[15] Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi. Pifpaf: Composite fields for human pose estimation.409 In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11977–11986, 2019. 1,410 3, 8, 9411
[16] Jia Li, Wen Su, and Zengfu Wang. Simple pose: Rethinking and improving a bottom-up approach for412 multi-person pose estimation. In AAAI Conference on Artificial Intelligence (AAAI), pages 11354–11361,413 2020. 8414
[17] Xilai Li, Wei Sun, and Tianfu Wu. Attentive normalization. In Andrea Vedaldi, Horst Bischof, Thomas415 Brox, and Jan-Michael Frahm, editors, European Conference on Computer Vision (ECCV), volume 12362,416 pages 70–87, 2020. 6417
[18] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár,418 and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In David J. Fleet, Tomás Pajdla,419 Bernt Schiele, and Tinne Tuytelaars, editors, European Conference on Computer Vision (ECCV), volume420 8693, pages 740–755, 2014. 7, 12421
[19] Kevin Murphy, Yair Weiss, and Michael I Jordan. Loopy belief propagation for approximate inference:422 An empirical study. arXiv preprint arXiv:1301.6725, 2013. 3423
[20] Alejandro Newell, Zhiao Huang, and Jia Deng. Associative embedding: End-to-end learning for joint424 detection and grouping. In Advances in Neural Information Processing Systems 30 (NeurIPS), pages425 2277–2287, 2017. 3, 8, 9426
[21] Xuecheng Nie, Jiashi Feng, Jianfeng Zhang, and Shuicheng Yan. Single-stage multi-person pose ma-427 chines. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 6950–6959, 2019.428 8429
[22] George Papandreou, Tyler Zhu, Liang-Chieh Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Mur-430 phy. Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geomet-431 ric embedding model. In European Conference on Computer Vision (ECCV), pages 282–299, 2018. 1, 3,432 8433
[23] Leonid Pishchulin, Mykhaylo Andriluka, Peter Gehler, and Bernt Schiele. Poselet conditioned pictorial434 structures. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages435 588–595, 2013. 3436
[24] Deva Ramanan, David A Forsyth, and Andrew Zisserman. Strike a pose: Tracking people by finding437 stylized poses. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition438 (CVPR’05), volume 1, pages 271–278. IEEE, 2005. 3439
[25] Brandon Rothrock, Seyoung Park, and Song-Chun Zhu. Integrating grammar and segmentation for human440 pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,441 pages 3214–3221, 2013. 3442
[26] Ke Sun, Zigang Geng, Depu Meng, Bin Xiao, Dong Liu, Zhaoxiang Zhang, and Jingdong Wang.443 Bottom-up human pose estimation by ranking heatmap-guided adaptive keypoint estimates. CoRR,444 abs/2006.15480, 2020. 2, 4445
[27] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for446 human pose estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages447 5693–5703, 2019. 2, 5448
[28] Zhi Tian, Hao Chen, and Chunhua Shen. Directpose: Direct end-to-end multi-person pose estimation.449 CoRR, abs/1911.07451, 2019. 2, 4450
[29] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. FCOS: fully convolutional one-stage object detection.451 In IEEE/CVF International Conference on Computer Vision (ICCV), pages 9626–9635, 2019. 4452
[30] Ali Varamesh and Tinne Tuytelaars. Mixture dense regression for object detection and human pose esti-453 mation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13083–454 13092, 2020. 4455
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz456 Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017. 9457
[32] Bin Xiao, Haiping Wu, and Yichen Wei. Simple Baselines for Human Pose Estimation and Tracking.458 Computer Vision and Pattern Recognition, 2018. 9459
[33] Yi Yang and Deva Ramanan. Articulated human detection with flexible mixtures of parts. IEEE transac-460 tions on pattern analysis and machine intelligence, 35(12):2878–2890, 2012. 3461
[34] Song-Hai Zhang, Ruilong Li, Xin Dong, Paul L. Rosin, Zixi Cai, Xi Han, Dingcheng Yang, Haozhi462 Huang, and Shi-Min Hu. Pose2seg: Detection free human instance segmentation. In IEEE Conference463 on Computer Vision and Pattern Recognition (CVPR), pages 889–898, 2019. 7, 9, 12464
[35] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. CoRR, abs/1904.07850, 2019.465 2, 3, 4, 8, 9466
Checklist467
1. For all authors...468 (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s469 contributions and scope? [Yes]470 (b) Did you describe the limitations of your work? [Yes] See Section 4.4.471 (c) Did you discuss any potential negative societal impacts of your work? [No]472 (d) Have you read the ethics review guidelines and ensured that your paper conforms to473 them? [Yes]474 2. If you are including theoretical results...475
(a) Did you state the full set of assumptions of all theoretical results? [N/A]476 (b) Did you include complete proofs of all theoretical results? [N/A]477
3. If you ran experiments...478 (a) Did you include the code, data, and instructions needed to reproduce the main experi-479 mental results (either in the supplemental material or as a URL)? [Yes] See Section 4.480 (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they481 were chosen)? [Yes] See Section 4.482 (c) Did you report error bars (e.g., with respect to the random seed after running experi-483 ments multiple times)? [No]484 (d) Did you include the total amount of compute and the type of resources used (e.g., type485 of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4 and the supplemen-486 tary material.487
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...488 (a) If your work uses existing assets, did you cite the creators? [Yes] We cited the COCO489 dataset [18] and the OCHuman dataset [34].490 (b) Did you mention the license of the assets? [Yes] We mention the licenses in our source491 code.492 (c) Did you include any new assets either in the supplemental material or as a URL? [N/A]493
494
(d) Did you discuss whether and how consent was obtained from people whose data495 you’re using/curating? [Yes] We briefly discussed it in Section 4.496
(e) Did you discuss whether the data you are using/curating contains personally identifi-497 able information or offensive content? [No]498
5. If you used crowdsourcing or conducted research with human subjects...499 (a) Did you include the full text of instructions given to participants and screenshots, if500 applicable? [N/A]501 (b) Did you describe any potential participant risks, with links to Institutional Review502 Board (IRB) approvals, if applicable? [N/A]503 (c) Did you include the estimated hourly wage paid to participants and the total amount504 spent on participant compensation? [N/A]505

Review summaries:
This paper proposed a local-global contextual adaptation method for the bottom-up human pose estimation.  In the proposed method, local keypoint expansion maps (KEMs) and global KEMs are both extracted. The pose-sensitive kernels (keypoint attraction maps, KAMs) are learned from the local KEMs and then performed to the global KEMs. The final prediction is generated from the refined global KEMs. The proposed method is end-to-end trainable and achieves state-of-the-art performance on COCO and OCHuman datasets. 

This paper introduces an end-to-end human pose estimation method, termed LOGO-CAP. It proposes a local-global contextual adaptation formulation and applies KAMs and the global KEMs to refine the estimated center-offset poses. It achieves 72.2 mAP on COCO val-2017 dataset. 

This paper proposes a center-offset based bottom-up human pose estimation architecture. The proposed method first estimates the centers and offsets of keypoints to get the initial poses. Then for each keypoint, estimate the parameters of an 11x11 convolution kernel in an instance-aware way. Finally, for each keypoint, a larger window is extracted from the global heatmap, and the learned 11x11 kernels are applied to convolve the extracted heatmap.