Learning Local-Global Contextual Adaptation for Fully End-to-End Bottom-Up Human Pose Estimation
Authors: Nan Xue
Tianfu Wu
Zhen Zhang
Gui-Song Xia
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Human Pose Estimation
Structured Prediction
Deep Learning
Computer Vision

Abstract: This paper presents a method of learning LOcal-GlObal Contextual Adaptation for fully end-to-end and fast bottom-up human Pose estimation, dubbed as LOGOCAP. It is built on the conceptually simple center-offset formulation that lacks inaccuracy for pose estimation. When revisiting the bottom-up human pose estimation with the thought of “thinking, fast and slow” by D. Kahneman, we introduce a “slow keypointer” to remedy the lack of sufﬁcient accuracy of the “fast keypointer”. In learning the “slow keypointer”, the proposed LOGO-CAP lifts the initial “fast” keypoints by offset predictions to keypoint expansion maps (KEMs) to counter their uncertainty in two modules. Firstly, the local KEMs (e.g. 11×11) are extracted from a low-dimensional feature map. A proposed convolutional message passing module learns to “re-focus” the local KEMs to the keypoint attraction maps (KAMs) by accounting for the structured output prediction nature of human pose estimation, which is directly supervised by the object keypoint similarity (OKS) loss in training. Secondly, the global KEMs are extracted, with a sufﬁciently large region-of-interest (e.g., 97 × 97), from the keypoint heatmaps that are computed by a direct map-to-map regression. Then, a local-global contextual adaptation module is proposed to convolve the global KEMs using the learned KAMs as the kernels. This convolution can be understood as the learnable offsets guided deformable and dynamic convolution in a pose-sensitive way. The proposed method is end-to-end trainable with near real-time inference speed, obtaining state-of-the-art performance on the COCO keypoint benchmark for bottom-up human pose estimation. With the COCO trained model, our LOGO-CAP also outperforms prior arts by a large margin on the challenging OCHuman dataset.

Human pose is highly articulated with large structural and appearance variations. 2D human pose estimation in images is a classic structured output prediction problem, and remains challenging one 28 computer vision and machine learning. Human pose estimation has numerous applications such as people-centered image understanding, autonomous driving and Augmented Reality (AR). With Learning Local-Global Contextual Adaptation for Fully End-to-End Bottom-Up Human Pose Estimation
Anonymous Author(s)
Affiliation Address email
Abstract
This paper presents a method learning L.OcaI-G1Obal Contextual Adaptation for fully end-to-end and fast bottom-up human Pose estimation, dubbed LOGO- CAP. It is built on the conceptually simple center-offset formulation that lacks inaccuracy for pose estimation. When revisiting the bottom-up human pose es- timation with the thought of "thinking, fast and slow" by D. Kahneman, we in- troduce "slow keypointer" keypointer". In learning the "slow keypointer", the proposed LOGO-CAP lifts the initial "fast" keypoints by offset predictions keypoint expansion maps (KEMs) to counter their uncertainty in two modules. Firstly, the local KEMS (e.g. 11 x 11) extracted from low-dimensional feature map. A proposed convolutional mes- sage passing module learns to "re-focus" the local KEMs to the keypoint attraction maps (KAMs)) by accounting for the structured output prediction nature of human pose estimation, which is directly supervised by the object keypoint similarity (OKS) loss in training. Secondly, the global KEMs are extracted, with suffi- ciently large region eegion-of-interess (e.g., 97 x 97), from the keypoint heatmaps that are computed by a direct map-to-map regression. Then, local gloabll contextual adaptation module is proposed to convolve the global KEMs using the learned KAMs as the kernels. This convolution can be understood as the learnable offsets guided deformable and dynamic convolution in pose-sensitive way. The pro- posed method is end-to-end trainable with near real-time inference speed, obtain- ing
3
5
remedy the lack of sufficient accuracy of the "fast
7
11
12
13
14
15
16
17
18
20
21
performance on the COCO keypoint benchmark for bottom-up human pose estimation. With the COCO trained model, our LOGO-CAP also outperforms prior arts by large margin on the challenging OCHuman dataset.
22
Introduction
24
25
1.1 Motivation and Objective
26
27
30 the recent resurgence of deep neural networks (DNNs), the performance of human pose estimation has witnessed remarkable improvement [12, 3, 15, 22, 11]. This paper focuses on the deep learning
32 based problem formulation.
33 There are two deep learning based paradigms for human pose estimation in the literature. The top-
34 down paradigm consists of human detection and single human pose estimation in each detected
35 human bounding box [12]. The bottom-up paradigm also includes two components: human pose [3]. The top-down paradigm often obtains better accuracy performance, but suffers from its inferior efficiency since the computational cost of the single human
36 keypoint detection and keypoint grouping
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

To realize the Slow Keypointer, as illustrated in Fig. 61 and Fig. 3, this paper presents method of learning 1.0cal-GObal Contextual Adaptation for fully end-to- end and fast nottom-up human Pose estimation, dubbed Feature Sampling
OKS
Frmn: Map Passteg
Hestmape
(Tlcased
The Predicted Offset Fields
Local-Global Contextual Adapiation
Resolts of our LOGO-CAP
Figure I: Illustration of the proposed LOGO-CAP for bottom-up human pose estimation. It built on the center-offset representation. See text for detail.
pose estimation component is linearly scaled with respect to the number of detected human bounding 39 boxes in an image. also largely affected by the performance of the human detection component (e.g., not handling occlusion very well). Thanks its efficiency, especially in real-time applications, 41 the bottom-up up paradigm becomes more and more attractive. For both paradigms, sfate-of-the-ae art methods often not fully end-to-end trained and utilize different post-hoe processing modules to improve the performance. This paper is interested in developing a fully end-to-end bottom-up paradigm and aims at bridging its performance gap with the top-down paradigm.
43
45
For the bottom-up paradigm, the recently proposed center-offset approach [6, 28, 26, 11] is a con- ceptually simple formulation (see the left of Fig. 1 for illustrative example and Fig. 3 for the detailed workflow). It alleviates the need of sophisticated keypoint grouping. When introducing human keypoints centers (i.e., anchors) by treating objects points [35], it encodes human pose as a star structure using the offset vectors of human keypoints relative to the anchors. The main challenge of the center-offset regression paradigm lies in the difficulty of accurately learning offset vectors with large structural variations, especially the long-range ones, which also leads to inferior performance. This paper builds on the center-offset approach and addresses its drawback.
46
47
49
51
52
53
1.2 Method Overview
54
To address the drawback of the center-offset formulation, we build the intuitive idea of 55 "Keypointing, fast and slow", by analogy the modes of thought suggested by Daniel Kahne- man in "Thinking, fast and slow" [14]: (i) Fast Keypointer: We treat the vanilla center-offset based estimation [35] as the Fast Initializer of pose estimation.
56
(ii) Slow Keypointer: The lack of localiza- 58 tion accuracy in the Fast Initializer entails a Slow Solver that learns to refine the "fast" keypoints. By slow, it only relatively speaking. The Slow Keypointer actually fast with near real teal-time speed.
57
59
60
The performance of a vanilla center-offset regression approach, its empirical upper bound, and the per- formance of our proposed LOGO-CAP using HRNet-W33 [27] as the feature backbone. See text for detail.
Table 1:
82
63
64 as LOGO-CAP. To quantitatively motivate the proposed 65 method, we first present a surprisingly strong observa- 66 tion for a vanilla center-off'et regression method (Ta-
Baseline
Emp. Bound
LOGO-CAP
67 ble in the fully-annotated subset of the COCO val-2017 the vanilla regression method utilizes the HRNet- RR3Net-W3221 [27] as the feature backbone to directly predict keypoints center heatmap and the offset vectors. This vanilla center-offset model obtains 60. average pre- cision (AP), which is not great, but reasonably good. It
AP
60.1
88.9
70.0
68
Dattase.SSpecffice
AP30
85,2
93.1
88.2
69
Ap7s
66.7
90.6
76.4
APM
53.7
87.7
64.4
APL
71.5
71
73
clearly shows that the pose keypoints center and the offset vectors can be learned reasonably well. Instead of directly utilizing the learned offset vectors for human pose estimation, treat them as human pose keypoint initialization and do a local window search to compute the empirical upper- bound of performance. More detailed, based on the predicted human poses, by introducing local window (e.g.. 11 x 11) centered at each detected key point and by computing the single keypoint similarity with the ground-truth keypoint, empirical upper- bound of 88.9 AP is obtained, which significantly higher than the state of the art and shows the potential of improving the vanilla center-offset regression paradigm.
75
76
77
79
80
81
Motivated by the above observation, straightforward way is just to learn local heatmap (e.g., 11 x 11) for each human pose keypoint based on the learned center and offset vectors, and then to

---

83
compute the refined keypoints by taking arg max within the local heatmap. Although appealing, this does not work as observed during our development of the LOGO-CAP. The underlying reason is easy to understand: if this can work, the original offset vector regression should work at the first place since no additional information is introduced through learning the local heatmap. We hypothesize that on the one hand, on top of the local heatmap, the structural relationship between different keypoints of a human pose needs to be taken into account, and on the other hand, the intrinsic uncertainty of the local information in local heatmap needs to be resolved. The former is the key challenge of structured output prediction problems. Many message passing algorithms have been developed in the literature. The latter can not be addressed by simply increasing the local window size. It entails learning stronger -ololgll information interaction and adaptation.
84
85
86
87
88
89
90
91
92
93
Along with the two hypotheses. the proposed LOGO- CAP lifts the initial keypoints via the center-offset pre- diction to keypoint expansion maps (KEMs) to counter
LOGED-CP (Cano
94
DEMR [+]]
95
00O0 Kaypolmts AP
70
Conterler [SS]
PiPur
96
their lack of localization accuracy in two modules (Sec- tion 3.2). The KEMs extend the star-structured repre- sentation of the center-offset formulation to the pictorial structure representation [10, 8J. The first module com-
60
Al-t wo2 P))
86
98
99
100
putes local KEMs and learns to account for the struc- tured output prediction nature of the human pose esti- mation problem, leading to the keypoint attraction maps (KAMs). The second computes global KEMs and Iearns to refine the global KEMS by leveraging the KAMs.
102
Inference time ms)
Speed-accuracy comparisons between our LOGO-CAP and prior arts on the COCO val-2017 dataset. Wr- Y (e.g. W32-384) means that a model uses the backbone HRNet- FRRee--Wr (HRNet- W32) and is tested with the image reso- lution Y in the short side.
Figure 2:
104
105
Our LOGO-CAP is fully end-to-end bottom-up hu- man pose estimation method with near real-time infer- ence speed. It obtains 70.0 AP in the fully-annotated sub- set the COCO va1-2017 dataset, which is an absolute increase of 9.9 AP compared to the vanilla center-offset method, making a significant step forward. Fig. shows a
106
108
109
110
pose estimation example. Fig. 2 shows the advantage of the proposed LOGO-CAP in terms of over- all speed- apeed-accuracy comparisons between our LOGO-CAP and prior arts Meanwhile, we should notice that there is also significant gap compared to the empirical upper bound (Table 1), which encourages more work to be investigated.
112
113
114
115
2
Related Works and Our Contributions
116
There is a vast body of literature for human pose estimation. Many elegant representation schema have been developed for modeling articulated human pose in the traditional approaches such as the well-known pictorial structure model [10, 8] and its many variants [24, 1, 23, 33, 25]. Most of them focused on single person pose estimation. They perform inference over a combination of local ob- servations on body parts (i.e.. the data term) and the spatial dependencies between them (i.e., the spring or clique term). The spatial dependencies are captured either using directed and acyclic struc- tures that facilitate the global optimization by dynamic programming [2, 9], or using structures with loop introduced (for high-order part relationship modeling) which resort to approximate inference by loopy belief propagation [19]. The bottleneck of the traditional methods lies in the data term which often based on hand-crafted features. With the resurgence of DNNs and the end-to-end learning, the data term has been largely improved. We briefly review the recent deep learning based approaches for bottom-up human pose estimation.
117
118
119
120
121
122
123
124
125
126
127
128 I Limb-based Grouping Approaches have been extensively developed due to the naturalness of modeling limbs based on keypoints. Given a predefined limb configuration (c.g., the COCO person skeleton template consisting of 19 limbs based on 17 keypoints), the grouping can be addressed by Part affinity field (PAF) [4, 31, Associative Embedding (AE) [20], mid-range offiset fields in Person- Lab [22] and the fields of Part Intensity and Association [15]. Typically, sophisticated designs are entailed to achieve good performance. For example, bipartite graph matching is used in Open-
129
130
131
132
133
134
Pose [3]. In addition to be computationally expensive, another drawback of these methods is not fully end-to-end trainable. More recently, the differentiability issue was studied by the Hierarchical Graph Clustering (HGG) method [13], which utilizes graph convolution networks to repeatedly de-
135
136
137
lineate pose parameters of multiple persons from keypoint graph. HGG improves the performance compared to its baseline, the Associative Embedding method [20] at the expense of significantly
138
3

---

139
increased computational cost. In contrast to thoses approaches, our proposed LOGO-CAP is fully end-to-end trainable and achieves near real-time inference speed.
140
141
Direct Regression based Approaches have attracted much attention due to their conceptually sim- ple formulation [6, 28, 26, 11, 30]. These center-offset based formulation are inspired by the cent remarkable success of direct bounding box regression in object detection such as the FCOS method [29] and CenterNets [35, 6]. As aforementioned, one main challenge is the difficulty of ac- curately regress the offset vectors, especially for the long-range keypoints with respect to the center. Sophisticated post-processing schema are often entailed to improve the performance. For example, a method of matching the directly regressed poses to the nearest keypoints that are extracted from the global keypiont heatmaps is used [35]. Although being simple, the performance of this line of work is usually inferior to the limb-based approaches. The mixture regression network [30] allevi- ated the issue of regression quality to some extent, but still remained an indispensable performance gap comparing with the grouping-base basedd approaches. Most recently, Geng et al. presented the first competitive direct method, DEKR [11] with novel pose-specific neural architecture for disentan- gled keypoint regression. To improve the performance, the DEKR method utilizes a lightweight rescoring network to recalibrate the pose scores that are computed based on the keypoint heatmaps. Despite good performance, the DEKR method entails the additional rescoring stage in both training and testing, and thus is not fully end-to-end. The proposed LOGO-CAP retains the simplicity of the vanilla center-offset formulation and enjoys fully end-to-end training and fast inference speed.
142
143
144
145
146
147
148
149
150
152
154
156
158
Our Contributions. The proposed LOGO-CAP makes three main contributions to the field of bottom-up human pose estimation: (i) It addresses the drawback of the vanilla center-offset for- mulation while retaining its efficiency. It proposes the key idea of lifting keypoint to keypoint expansion map to counter the lack of localization accuracy. To our knowledge, it is the first fully end-to-end trainable method that achieves state-of-the-art performance. (ii) It presents a novel local- global contextual adaptation formulation that accounts for the nature of structured output predic- tion in human pose estimation and harnesses local-global structural information integration. (iii) obtains state-of-the-art performance in the COCO val-2017 and test-2017 datasets. It also shows state-of-ther the-rrr transfcrability performance in the OCHuman dataset.
159
160
162
164
186
3
Approach
187
168
3.1
Problem Formulation
189
We follow the COCO protocol of defining the human pose. It consists of 17 human pose keypoints: 8 pairs of symmetric keypoints (hips, ankles, knees, shoulders, elbows, wrists, ears and eyes) and the nose keypoint. Let P {1,...11 17} be the set of keypoint indexes using predefined order. Let A be an image lattice of the spatial size H W (e.g.. 512 x 512), and be an image defined on A. Let Ph be the set of keypoint indexes for human pose instance in an image I and we For example, in COCO, we typically have 1 < n I 30, and different human pose instances have different number of visible keypoints due to occlusion and/or truncation. Denote by Li = (xx, y:); E PP] the keypoint locations of a human pose instance n in an image I, where (I;, Mi) E A. In the center-offset formulation, we introduce the keypoints center (i.e., the anchor),
170
171
172
173
have PT C P. P..
174
176
177
178
(Iez ye) based on a given Li and we have,
Ne = 1/1
Σ Js. Ye - 1/1L
Σ yi- REP"
REP,
179
With the anchor, a keypoint (zt, y:) is equivalently defined by its offset/displaceemet,, denoted by (Aa,, Ay:) with Aa; Ti Ir and Ay y) ye- So, LT can also be equivalently expressed as Ly ((res ye), (Ar,, Ay;); E p.).
181
182
The objective of human pose estimation is to recover Li = {( v): E PP} for all human pose instances in an image. Denote by ب; (ā., i E P the estimated human pose. Following the COCO protocol, the object keypoint similarity (OKS) is used to evaluate the accuracy,
184
(oks(Lp, L7) = 1/1PP1- Σ exp PEPT
185
where d; is the Euclidian distance between the ground-truth keypoint (2t, 9:) and the predicted one (f,, Dt). s is the square root of the human segment area, and per-keypoint constant that controls
186
fall-off in evaluation. We have (oxs(i; LT) [0. 1]. The OKS metric is to evaluate the distance between predicted keypoints and ground-truth keypoints normalized by the scale of the person with
187
188
189
the importance of keypoints equalized. In benchmarking different methods, the average precision

---

27
"Feut" Keypolnter
Fizad Ponc Excination
5
(output)
CONTLY
dasting
Lak
La Fonii pere
Nosaje
Ingut Image
"Sler" Rapointer
Figure 3: Illustration of the network and algorithmic flow of the proposed LOGO-CAP for bottom- up human pose estimation. See text for detail.
(AP) at OKS= 0.50 : 0.05 0.95 is used as the primary metric, together with Ap50 at OKS= 0.50, A at OKS= 0.75, and AP across medium and large scales, APM and AP4 respectively.
190
3.2
The Proposed LOGO-CAP
We first present the network and the inference of LOGO-CAP, and then give details of the train- ing. We keep different modules of the proposed LOGO-CAP simple, which in turn highlights the effectiveness of the proposed representation and algorithmic flow.
194
195
3.2.1
The Network and the Inference
197
As illustrated in Fig. 3, the proposed LOGO-CAP consists of four components as follows.
i) A convolution neural network feature backhone. Given input image J, the output of the fea- ture backbone a C-dim feature map, denoted by F E RCxhxx where is the feature dimension of the last convolutional layer in the feature backbone, and the spatial size h x depends the total stride in the feature backbone. We use off-the-shelf HRNets [27] in our experiments.
199
201
ii A parallel keypoint-offse regression module. Given the feature map F. the output of keypoint regression is an 18-dim feature map (i.e., heatmaps) for the 17 keypoints and the keypoints center respectively. Denote by H. E R18x the heatmaps, and by H. R18x ff XW the up-sampled heatmaps (using bi-linear interpolation in our experiments). The output offset regression is a 34-dim feature map (i.e., the offset vector fields) for the 17 keypoints. Denote by O R34xhx the offset fields. We adopt a minimally-simple design in realizing the regression modules using a first apply dimension reduction to the feature map F using a x convolution followed by a Batch Normalization (BN) and a Rec- tified Linear Unit (ReLU). Then, the output computed by 1 convolution. More specifically,
203
205
207
channel-wise multi-layer perceptron (MLP). In implementation,
209
210
211
we have the two parallel branches as follows,
Foxhxw Cxix1xC:
Cone BNERRLU
F
Cix1xix18 Hi8xhxm
Cone
UpSampling
H H18x
bi-linear
Foxhxw
Cono+BN+ReLD
Fe C2x1x1x31 O34xhxw>
Cone
(4)
CxIxIxC2
212
where Ci and C2 are predefined (e.g., Ci = 32 and C2 256 typically used).
213
Initial pose estimation via center-offset approach. Based on the computed keypoints center heatmap Hl1s) and offset fields O, a predefined maximum number of pose candidates is computed as done in the vanilla center-offie offset approach. A non non-maximum suppression (NMS) with a 3 x 3 window is applied in Hi and then the top-N keypoints centers are selected (e.g.. N = 30 in our experiments). The N pose instances
214
215
216
computed by retrieving their offset vectors in O based on the selected N keypoints centers. The N pose instances are further pruned by thresholding their confidence scores in H(is) with predefined threshold (e.g., 0.01 used in our experiments). Without confusion in the context, we still use N to denote the number of poses instances by this initial pose estimation step. We obtain the set of estimated keypoints centers, denoted by CN each row of which represents the position coordinates and the confidence score.
218
219
221
5

---

223
Lifting keypoint to a keypoint expansion map (KEM) by imposing a mesh. For each of the N pose instances, each of the 17 keypoints are placed in a local geometric mesh (e.g.. 11 11) with the estimated location as the mesh center, capturing the uncertainty of the center-offset pose estimation as aforementioned in the introduction. This mesh can thus be interpreted as keypoint expansion map (KEM), accounting for competency-awarc representations. The entire mesh is denoted by
224
225
226
227
228
M Mxx1xxxxxxxxx which is used in computing the empirical upper bound in Table 1. We have,
(H(ss)> O34xhx1
initial pose estimation
{CNx3, MNXIXXXXXXXxXXIIxxxx
(5)
center-oliset
229
iii) A convolution message passing module. We first encode the geometric mesh MMxx M1Nx17x11x11x in latent space with the dimensionality C3 (e.g.. 64 in our experiments), computed based on the feature backbone output. Then, a keypoint is represented by C3 x 11 x 11 local feature map. A pose instance represented by concatenating all the 17 keypoints. We have,
230
231
232
Conv ReLLE, F.M C2
MaxIIxHXXX XNx(ITxCaxxIIII
Foxhxw Cx1x1xC,
(6)
bi-linear
233
where the bi-linear interpolation is used due to the sub-pixel based locations in the mesh and for better feature alignment.
234
To facilitate the structural information flow between different latent codes of the keypoints of a pose instance, we propose a simple convolutional message passing (CMP) module with three layers of
236
237
Conl+BN+ operations,
XXx(ITCG)XUXXI Conv+
Conv
Can X3x3xxxuu x3 Cxx1x1x17 KNxI7x11x11,
where Cin E {117 x Ca), Ca, and Cout E {C4, Cb. Co) (e.g.. C4 = 512, C3 256, Co 128 in our experiments). The resulting XNxI7xLIXXI can be interpreted as keypoint attraction maps (KAMs) which are "re-focused" based on the KEMs by the CMP. To account for the specificity of different pose instances in the CMP, we adopt the Attention Normalization [17] to replace the BN in the second Con++BN+ReLU layer, which further improves the performance in our experiments.
239
241
242
243
Through the CMP, we obtain the dynamic (a.k.a., data-driven) kernels for the 17 keypoints in a pose instance-sensinene way, which are used to refine the global heatmaps 'I8 for the 17 keypoints.
244
iv) A local-global contextual adaptation module. We first compute another geometric mesh with enlarged mesh window a x (e.g., a = 97) for each keypoint of the N pose instances, and the entire done in Eqn. 5. The mesh can be interpreted as the global KEM. It is then instantiated with appearance features extracted from the global heatmaps H. (1:77)
246
247
mesh is denoted by M x17xxxxx2
248
similar to Eqn. 6, and we have,
H ((117) MKNITNOKOX3
(0xa(0,0) H x1Nxxax
H N x17xaxa
(8)
bi-linear
reweighing
250
where to encode the Gaussian prior of keypoint heatmaps, the resulting pose-guided heatmaps H is reweighed by Gaussian kernel Gaxa(0, a
251
a-1 (e.g., a = 16 when a = 97) in an element-wise 2x3
252
way. By doing so, means that the enlarged mesh follows the 30 principle.
Then, we apply the learned keypoint 11 x 11 kernels Kn,i's (Eqn. 7) to convolve the reweighed 253
254
a x a heatmap Hn,s (Eqn. 8) in a pose instance-sensitive and seypoint-specifie way, leading to
1Dcal-GlObal Contextual Adaptation,
MxxI7xxxa KNNITXII11 HNX17xaxa:
LOGO-CA
256
which represents the refined heatmaps for the 17 human pose keypoints.
The Pose Estimation Output. With the local-global contextually adpated heatmaps x we maintain the top-2 locations for each keypoint within the a heatmap, and then utilize a convex average of the top-2 locations as the final predicted offset vectors (i.e. (Av', Ay/)'s in Fig. 3), and of their confidence scores as the prediction score, with predefined weight À for the top-1 location (0.75 in our experiments). Together with the predicted keypoints centers CNx3 (Eqn. 5), the final 261
257
259
260
prediction score for each keypoint is the product between the convex average confidence score and the center confidence score. We keep the keypoints whose final scores are greater than 0. We have, Output
263
{CNx3, H (7xaxa)
{L n = 1,-.- N'},
(10)
Score thresholding
264
where N' is the number of the final predicted pose instances in an image T.

---

285
3.2.2
Loss Functions in Training
266
In the fully end-to-end training. we need define loss functions for the global heatmap H (Eqn. 3), the refined local heatmap Hi (Eqn. 9), the offset field O (Eqn. 4), and the keypoint kernels (Eqn. 7).
267
268
The Heatmap Loss. The widely adopted mean squared error (MSE) loss is used. Denoted by HGT xx the ground truth heatmaps in which each keypoint (including the center) is modeled by a
289
270
2-D Gaussian with dataset-providde mean and variance. Let p (i.x) be the index of the domain
271
D of dimensions 18 x h x w. For the predicted heatmaps HisxAxw: the MSE loss is defined by,
Lx = 1/
Σ ([(())(H(p)) - (
(11)
PED
272
where w(x) represents the weight for the foreground and the background pixels. The foreground mask is provided by the dataset annotation. In our experiment, we set w(x) = 1 for a foreground pixel and w(x) = 0.1 for background pixel.
273
274
In defining the loss function C3 for the refined local heatmap H (Eqn. 9), the ground-truth heatmap HGTT is generated on-the-fly based on the mesh M x (Eqn. 8) and the ground-truth key- points using Gaussian model with mean being the displacement between the current predicted keypoints and the ground-truth ones, and variance a (i.e., the standard deviation of the reweighing Gaussion prior model in Eqn. 8).
276
277
278
279
The Offset Field Loss. The widely adopted SmoothL.1 loss [] is used. Let OCT 34x be the ground- truth offset field, and CGT be the non- con-empty set of ground-truth keypoints centers (Eqn. 1). For the predicted offset field O24xhxw (Eqn. 4), we have,
280
281
282
Lo = 1/
Σ
A(p)
SmoothL1 (O(-p),. OCT (+))):3)
(12)
DECGT
where A(p) is the area of the person centered at the pixel and 3 the cutting-off threshold (e.g.. 1 in our experiments), and SmoothL (a, b, B) = 0.5x //3 if ba < B, otherwise |a- 0
283
9
285
The OKS Loss for the Keyoint Kernels. Consider single predicted pose instance, learning the keypoint kernels, K K1xx11x11 (Eqn. 7) is the key to facilitate the local-global contextual adaptation.
286
287
To that end, the figure of merits of the KEF, MMTx11xx1x (Eqn. 5) needs to directly reflect the task loss. i.c., the OKS loss (Eqn. 2). With respect to the NGT ground-truth pose instances in an image,
288
289
we can compute the similarity score per keypoint candidate in the KEF, and obtain the score tensor IIxxIIIIxx The score tensor is further clamped with threshold 0.5, i.e., SI7x 11 11 Ner = max (S xxxxxxXXxxxx NGT, 0.5). A mean reduction is applied to the first three dimensions of the clamped
290
291
score tensor to compute the matching score for each of the NGT pose instance. Then, the best ground-trus pose instance indexed by n² is selected in terms of the matching score, and its matching
292
293
score denoted by s,,. Based on the selected ground truth pose instance, we compute the per- 294
295
keypoint similarity score for the predicted pose instance at hand, denoted by 8k: (I: E [1, 17)). Then, the loss function fo the keypoint kernels are defined by.
296
LK
Sn
8k" Σ R.3.J
|KKa Sk.
(13)
297
The Total Loss is then defined by L - Cu + Lg + A (Co + Lk), where the trade-off parameter
A is used to balance the different loss items (A = 0.01 in our experiments).
4 Experiments
299
300
In this section, we present detailed experimental results and analyses of the proposed LOGO-CAP. Our PyTorch source code will be released for reproducibility.
302
Datasets.
We use two datasets in our experiments: The COCO dataset [18] is the most popu- lar testbed for human pose estimation. It consists of 65k, 5k and 20k images with human pose well annotated in the training, validation and testing datasets respectively. In all experiments, the proposed LOGO-CAP is trained using the 65k training images. The OCHuman dataset [34] is one popular testing-only dataset for evaluating human pose estimation under the occlusion scenarios. consists of total number of 4713 images with 81 10 detailed annotated human pose instances using
303
304
305
306
308
the COCO keypoint configuration. All the annotated 1010 human pose instances have occlusions
309
with the max1OU> 0.5. Furthermore, 32% instances are more challenging with the max1OU> 0.75.
7

---

OK
OKSHO B6G
OKS-0460
CKS-0395
OKS 0.756
OKS-9885
Ahats
Figure 4: Examples of human pose estimation in the COCO val-2017 dataset by the proposed LOGO-CAP with the HRNet-W32 backbone. Top: The COCO skeleton template based visual- ization. Botton: The close-up visualization and OKS comparisons between the initial center-offset estimation and the refined keypoints.
Table 2: Evaluation results on the coco-val-2011 and coco-testdev-2211 dataset. For
HGG [13] and SimplePose [16]. the multi-scale inferencet is applied the testdev-2017 dataset. For DEKR [11] that uses an rescoring network to get the final predictions, we report both the per-
formance with and without rescoring (which is the fair baseline for LOGO-CAP). The numbers
of SPM [21] and HGG [13] are extracted from their papers.
COCO-aal-0017 APP)(S)
COCO-asdew-301
Method
Backbone
AP[S]
AP [S]
AP [S]
AP [%]
AP[S]
AP [R)
AP"[[R]
AP! [R]
AP4 [S]
OpenPose [SS]
vGG
61.0
34.9
67.5
56.3
69.3
61.8
$4.9
67.5
57.1
68.2
PifPaf
67.4
63.1
74.1
66.7
81.8
73.6
62.4
72.9
Personl [2a]
75.2
59.0
72.6
62.4
[30,5]
weara was
69.9
87.2
76.1
65.4
76.1
76.4
68.4
53.3
57.5
72.8
75.1
64,4
74.2
HGG [13)
Ikeedao
60.4
$3.0
66.2
67.61
85.11
23,71
62.71
74.6"
SimplePose
76.2
08.51
86.71
74.91
66.4
[2]]]
Ikeedao
CenserNet [35]
Hiegim
70.2
72.1
63.0
96.8
69.6
5K.9
DNrstt
DEKK [11]
IR3e 932
68.0
86.7
74.5
62.1
77.7
67.3
87.9
74.1
61.5
26.1
(w. Rescoring)
71.0
88.3
77.4
66.7
78.5
70.0
89.4
77.3
65.7
76.9
DEKK [1:]
IBNa 992
67.2
36.3
23.8
61.7
77.1
66.6
83.6
73.5
61.2
75.6
Resoorinei
70.3
66.3
78.0
69.3
89.1
76.7
65.3
76.4
LOGO CAP
78.0
68.2
88.7
74.9
62.8
76.0
(Ours)
75.9
70.8
77.8
310
4.1
Results on the COCO dataset
Fig. shows some qualitative examples of human pose estimation by the proposed LOGO-CAP. More examples will be provided in the supplementary material.
312
The proposed LOGO-CAP is compared with prior arts including OpenPose [3], PifPal [15], Person- Lab [22], AE [20] and DEKR [11]. As reported in Table 2, the proposed LOGO-CAP outperforms
314
all of them on both both validation and test-dev datasets.
316
In comparisons to the best-performing grouping approach, AE [20] with larger backbone HrHRNet- W48 [5], our LOGO-CAL obtains competitive performance with smaller HRNet-32 backbone, and improves the AP score with HRNet-Was W48 backbone on the validation and testdev datasets by 2. and 2.5 points, respectively. For the fully differentiable grouping approach HGG [13], our LOGO-CAP achieves better performance by a significantly large margin, more than 9.2 points the validation set under the single-scale testing. Although the performance of HGG improved by the multi-scale testing on the test-dev set, the performance of our LOGO-CAP is still significantly better without using the multi-scale testing scheme.
317
318
320
322
324
In comparisons the direct regression based approaches, our LOGO-CAP obtains the best results without incurring either the matching scheme used in CenterNet [35] or the additional rescoring network used in DEKR [1]]. When disable the rescoring network for DEKR [11] for fair com-
325
326
327
parisons, our LOGO-CAP significantly improves the AP on the validation and testdev datasets by
328
2. points and 1. points respectively when TIRNet-W33 is used as backbone. The larger back-
bone is beneficial for both DEKR and our method, which further improves the AP score of our
330
LOGO-CAP to 72. and 70.8 8 on the validation and test-dev dataset respectively, outperforming
DEKR by 1.9 and 1 respectively.
8

---

Table 3: Results on the OCHuman valida- tion and testing datasets [34].
Table 4: The single image inference speed com- parison for bottom-up human pose estimation ap- proaches.
Val. [S+]
Test
Methods
Backbone
AP [S]]
Time + [ms]
1op-drmn
RMPE [7]
Hourglass
38.8
30.7
Method
AP [%]]
Backbone
FPS
SBL [32]
SBL [32]
R8sNet-182
50
41.0
37.8
33.3
30.4
PilPal [15]
67.4
ResNet 152
213
4.68
AE [20, 5]
67.1
HoHRNe-W92
560
1.78
AE [20]
Hourglass
32.1
29.5
CenterNet [35]
64.0
Hourglass
147
6.80
Heaemrg
HGG [20)
Hourglass
35.6
34.8
DEKR [1]]
68.0
HRNes-W32
63
15.8
DEKR [11]
HRNet-W32 W32
37.9
36.5
DEKR [11]
71.0
HRNe:
139
7.21
HRNer-W44
38.8
38.2
LOGO CAP
HRNet- W32
39.0
38.1
LOGO-CAP
69.6
HRNe--W32
48
20.7
(Ours)
HRNet-W48
41.2
40.4
LOGO-CAP
72.2
HRNes-W48
112
8.95
332
4.2
Results on the OCHuman dataset
335
Table shows that our LOGO-CAP achieves the best AP performance on both the validation and testing datasets by significant margins of 2 and 2 points in comparing with the bottom-up approaches. For the top-down approaches, although they obtain strong AP scores on the validation split, there exists a large performance gap between the validation and testing sets. In comparisons
335
337
to DEKR [11] (with the rescoring network), our LOGO-CAP improves the performance from 37.9 to 39.0 and from 36.5 to 38.1 the validation and testing splits with the same backbone HRNet- W32. respectively. The similar improvement is observed when the HRNet- W48 backbone is used, outperforming both bottom-up and top-down approaches.
339
4.3
Inference Speed
342
In comparing the inference speed, we test all the models on single TITAN RTX GPU for its popularity practice. The average inference speed, FPS (frames per second), over the 5000 images in COCO-val-2017 is used for the comparison. For DEKR [11], we re-implement their inference code with better speed obtained for fair comparisons at the algorithm level. For methods that have post- processing schema on CPU, only one thread is used. As shown in Table 4, our LOGO-CAP runs significantly faster than PifPaf [i5] and AE [20]. The CenterNet [35] runs slower than DEKR and our LOCO-CAP as it requires post- post-processing scheme to match the predicted offsets to the keypoints obtained from heatmaps. Comparing with DEKR, the speed improvement of our LOGO- CAP is from the lightweight design of head modules since the same backbones are used. For the comparisons in Table 2, we run the models with different resolutions of testing images.
343
344
345
346
347
349
351
352
Potentials and Limitations of the Proposed LOGO-CAP
Consider the generic applicability of the center-offset formulation to many computer vision tasks as demonstrated in [35], we hypothesize that the proposed LOGO-CAP has a great potential to remedy the lack of sufficient accuracy using the vanilla center-offset method in those tasks. We also notice that the minimally-simple design in learning the "Slow Keypointer" can be relaxed for different accuracy-speed trade-offs in practice. For example, for the convolutional message passing module, an alternative method could be the Transformer model [31], which potentially will further improve the performance the expense of inference speed. We leave these for future work.
354
355
356
357
358
359
Conclusion
360
361
This paper focuses on deep learning based formulation for bottom-up human pose estimation. It presents a method of learning LOcal-GIObal Contextual Adaptation for Pose estimation, dubbed as L LOGO-CAP. The proposed LOGO-CAP is built on the conceptually simple center-offset paradigm and addresses its drawback of lacking the capability of accurately localizing human pose keypoints.
352
363
364
385
The key idea of our LOG-CAP is lift the center-offset predicted keypoints to keypoint expansion maps (KEMs), which counters the inaccuracy and uncertainty of the initial keypoints. Two types of KEMs are introduced in two parallel modules on top of the feature backbone. Local KEMs are used
386
387
368
to learn keypoint attraction maps (KAMss) via a convolutional message passing module that accounts for the structured output prediction nature of human pose estimation. Global KEMs are used to
369
370
leam local-global contextual adaptation which convolves global KEMs using the KAMs as kernels.
371
The refined global KEMs are used computing the final human pose estimation. The proposed
LOGO-CAP obtains state-of-the-art performance in COCO val-2017 and test-dev 2017 datasets for bottom-up human pose estimation. It also achieves state-of-the-ar transferability performance in
373
374
the OCHuman dataset with the COCO trained models.

---

References
375
[1] Mykhaylo Andriluka, Stefan Roth, and Bernt Schiele. Pictorial structures revisited: People detection and articulated pose estimation. In 2009 IEEE conference on computer vision and pattern recognition, pages 1014-1021. IEEE, 2009.
377
378
379
[2] Richard Bellman. Dynamic programming. Science, 153(3731):34 37, 1966.
380
[3] Zhe Cao, Gines Hidalgo Martinez, Tomas Simon, Shih-En Wei, and Yaser A. Sheikh. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Trans. on Pattern Analysis and Machine Intelligence ( (PAMI), 2019. 3,
381
383
[4] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages
385
1302 1310, 3
386
[5] Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas Huang, and Lei Zhang, Higherhrnet: Scalc-aware representation learning for bottom-up human pose estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5385-538 5394,,39, IEBE, 2020. 3, 3,8.9
387
388
389
[6] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In JEEF/CVF International Conference on Computer Vision (ICCV), pages 6568-6577, 2019. 2,4
Haoshu Fang, Shugin Xie, Yu-Wing Tai, and Cewu La. RMPE: regional multi-person pose estimation. In IEEE International Conference on Computer Vision (ICCV), pages 2353-2362, 2017."
393
394
[8] Pedro F Felzenszwalb and Daniel P Huttenlocher. Pictorial structures recognition International journal of computer vision, 61(1):55-79) (0.5579,2200 2005. 3
395
396
19|
Pedro F Felzenszwalb and Ramin Zabih. Dynamic programming and graph algorithms in computer vision. IEEE transactions on pattern analysis and machine intelligence, 33(4):721-744 2010. 3
387
398
10]
Martin A Fischler and Robert A Elschlager. The representation and matching of pictorial structures. IEEF Transactions on computers, 100(1 (MK1),67-92, 1973. 3
400
[1I] Zigang Geng, Ke Sun, Bin Xiao, Zhaoxiang Zhang, and Jingdong Wang. Bottom-up human pose es- timation via disentangled keypoint regression. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 1, 3, 8,9
402
403
[12] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In IEEE International Conference on Computer Vision (ICCV), pages 2980-2988 2017. 1
404
405
[13] Sheng Jin, Wentao Liu, Enze Xie, Wenhai Wang, Chen Qian, Wanli Ouyang, and Ping Luo. Diffcrentiable hierarchical graph grouping for Vision (ECCV), volume 12352, pages 718-734, 2020. 3,
406
pose estimation. In European Conference on Computer
407
[14] Daniel Kahneman. Thinking, fast and slow. Macmillan, 2011. 2
409
[15] Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi Pifpaf: Composite fields for human pose estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11977-11988 2019.
411
3, 8,9
412
[16] Li, Wen Su, and Zengfu Wang. Simple pose: Rethinking and improving a bottom-up approach for
413
multi-person pose estimation. In AAAI Conference on Artificial Intelligence (AAAI), pages 11354-11361, 2020. 8
414
415
[17] Xilai Li, Wei Sun, and Tianfu Wu. Attentive normalization. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michall Frahm, editors, European Conference Computer Vision (ECCV), volume 12362, pages 70-87, 2020. 6
416
[18] Tsung- Yi Lin, Michael Maire. Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsofi COCO: common objects context. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Eurupean Conference on Computer Vision (ECCV), volume 8693, pages 740-755 755, 2014. 7, 12
419
420
421
422
[19] Kevin Murphy, Yair Weiss, and Michael I Jordan. Loopy belief propagation for approximate inference: An empirical study. arXiv preprint /.01.:133..677 2013. 3
10

---

424
[20] Alejandro Newell, Zhiao Huang, and Jia Deng. Associative embedding: End-to-end learning for joint detection and grouping. In Advances in Neural Information Processing Systems 30 (NeurlPS), pages 2277-2287, 2017. 8,
426
427
[21] Xuecheng Nie, Jiashi Feng, Jianteng Zhang, and Shuicheng Yan. Single-stage multi-person pose ma- chines. In JEEE/CVF International Conference on Computer Vision (ICCV), pages 6950-6959, 2019.
428
429
430
[22] George Papandreou, Tyler Zhu, Liang-Chieh Chich Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Mur- phy. Personlab: Person pose estimation and instance segmentation with bottom-up, part-based, geomet- ric embedding model. In European Conference Computer Vision (ECCV), pages 282-299, 2018. 1,
433
[23] Leonid Pishchulin, Mykhaylo Andriluka, Peter Gehler, and Bernt Schicle. Poselet conditioned pictorial structures. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 588-595, 2013. 3
435
436
437
[24] Deva Ramanan, David A Forsyth, and Andrew Zisserman. Strike a pose: Tracking people by finding stylized poses. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 05), volume 1, pages 271-278. IEFF, 2005.
438
440
[25] Brandon Rothrock, Seyoung Park, and Song-Chure Zhu. Integrating grammar and segmentation for human pose estimation. In Proceedingy of the IEEE Conference Computer Vision and Pattern Recognition, pages
442
2013. 3
443
[26] Sun, Zigang Geng, Depu Meng, Bin Xiao, Dong Liu, Zhaoxiang Zhang, Jingdong Wang. Bottom-up human pose estimation by ranking heatmap-guided adaptive keypoint estimates. CoRR, abs/2006.154m 15480, 2020. 2,
444
445
446
[27] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In IEEE Conference Computer Vision and Puttern Recognition (CVPR), pages 5693-5703, 2019. 5
447
449
[28] Zhi Tian, Hao Chen, and Chunhua Shen. Directpose: Direct end-to-end multi-person pose estimation. CoRR, abs/191 1,07451, 2019. 2,4
[29] Zhi Tian. Chunhua Shen, Hao Chen, and Tong He. FCOS: fully convolutional one-stage object detection. In JEEE/CVF International Conference on Computer Vision (ICCV), pages 9626 9635,963,, 2019.
452
[30] Ali Varamesh and Tinne Tuytelaars. Mixture dense regression for object detection and human pose esti- mation. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13083- 13092, 2020. 4
454
455
456
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint 1rXiv:1706.0333 03762, 2017, 9
457
458
[32] Bin Xiao, Haiping Wu, and Yichen Wei. Simple Baselines for Human Pose Estimation and Tracking. Computer Vision and Pattern Recognition, 2018.
460
[33] Yi Yang and Deva Ramanan. Articulated human detection with flexible mixtures of parts. IEEE transac- tions pattern analysis and machine intelligence, 35(12):2878-2890 2012.
[34] Song-Ha Hai Zhang, Ruilong Li, Xin Dong, Paul L. Rosin, Zixi Cai, Xi Han, Dingcheng Yang, Haozhi Huang, and Shi-Min Hu. Pose2seg: Detection free human instance segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 889-898, 2019. 7,
463
464
485
[35] Xingyi Zhou, Dequan Wang, Philipp Krähenbühl. Objects as points. CoRR, abs/ 111199944975575 2019.
486
2, 4,

---

Checklist
487
468
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
470
contributions and scope? [Yes]
471
(b) Did you descrihe the limitations of your work? [Yes] See Section 4.4.
472
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
474
475
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
477
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See Section 4.
480
481
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.
482
483
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
484
(d) Did you include the total amount of compute and the type of resources used (c.g., type
486
of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4 and the supplemen- tary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
489
(a) If your work uses existing assets, did you cite the creators? [Yes] We cited the COCO dataset [18] and the OCHuman dataset [34].
490
491
(b) Did you mention the license of the assets? [Yes] We mention the licenses in our source
code.
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
494
495
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes] We briefly discussed it in Section 4.
496
497
(c) Did you discuss whether the data you are using/curating contains personally identifi-
able information or offensive content? [No]
489
5. If you used crowdsourcing or conducted research with human subjects...
500
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
502
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
503
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
505
12


Reviews:
[{"review_id": "_dD9bAWWdIv", "reviewer": "Reviewer_MSva", "summary": "This paper proposed a local-global contextual adaptation method for the bottom-up human pose estimation.  In the proposed method, local keypoint expansion maps (KEMs) and global KEMs are both extracted. The pose-sensitive kernels (keypoint attraction maps, KAMs) are learned from the local KEMs and then performed to the global KEMs. The final prediction is generated from the refined global KEMs. The proposed method is end-to-end trainable and achieves state-of-the-art performance on COCO and OCHuman datasets.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- This paper proposed an interesting way to refine the results of center-offset-based pose estimation. \n- The proposed method outperforms the state of the art on COCO and OCHuman datasets.\n- The details of the proposed method are well described. \n\nCons:\n-The experiment is not sufficient to prove the effectiveness of the proposed method. More ablation studies and analyses should be provided.\n1)  The KAM is the key component of the proposed method, so it is better to investigate different ways for KAM generation. For example, different sizes of local KEMs, different designs of convolutional message passing module, use or do not use KAM to refine the global KAMs. \n2) Some proposed modules should be evaluated in the ablation studies, e.g. the CMP, the attention normalization and the effectiveness of performing KAM to global KEMs, etc. \n3)  It is important to show the inference speed comparison between the baseline and the proposed LOGO-CAP. \n4) Further analysis should be provided. What kinds of KAMs kernels are generated adaptively and how do the KAMs refine the global maps? What kind of hard cases are refined by the proposed method.\n\n-This paper claims that ‘this is the first fully end-to-end trainable method that achieves the state-of-the-art performance’ (in L. 161). However, many end-to-end trainable pose estimation methods have been proposed, e.g. Mask R-CNN, ICCV 2017,  HGG[13]. \n\n-The paper should be further polished before being published.\n1) More explanations should be presented in the caption of each figure, e.g. Figure 1, 3, 4. \n2) In Figure 4, the major difference between the initial offsets and the refined pose should be highlighted. It is hard to find the improvements. \n3) Typos:\nIn L.45, ‘approach is ’ -> ‘approaches are’\nIn L. 280, the citation is missed.\nIn L. 319, 321, 328, ‘points’ should be ‘%’.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues found.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "98569tsbgzz", "reviewer": "Reviewer_1GBc", "summary": "This paper proposes a center-offset based bottom-up human pose estimation architecture. The proposed method first estimates the centers and offsets of keypoints to get the initial poses. Then for each keypoint, estimate the parameters of an 11x11 convolution kernel in an instance-aware way. Finally, for each keypoint, a larger window is extracted from the global heatmap, and the learned 11x11 kernels are applied to convolve the extracted heatmap.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method proposed by this paper is novel and technically sound. \nThe idea of combining keypoint features that belong to the same person to obtain the convolution kernel of a single keypoint is interesting. Extracting a 97x97 pose-guided heatmap for each keypoint introduces high resolution to the bottom-up architecture, what is exactly the bottom-up architecture needed.\nOf course, this paper also has some problems. First, the \"Fast Keypointer\" and \"Slow Keypointer\" mentioned at the beginning of this paper do not actually relate to speed. The \"Slow Keypointer\" proposed is actually a refinement of the coarse estimated pose. This statement affects readers' understanding to a certain extent. Second, there are many tricks used in this paper, such as Attention Normalization, reweigh heatmaps using gaussian kernel and OKS loss. But ablation study has not been carried out in the experiment to show the influence of these tricks. Adding an experiment like this would be more convincing. Last, since each keypoint of each person has to be dealt with separately, the speed of the method proposed in this paper should be greatly affected by the number of people in an image, and the memory usage should also be large, but this paper does not mention it. I don’t know if some techniques are used to alleviate these two problems.\nAll in all, the conclusions of this paper are significant. After DEKR, it proves again that the center-offset architecture can perform high-precision pose estimation.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Since each keypoint of each person has to be dealt with separately, the speed of the method proposed in this paper should be greatly affected by the number of people in an image, and the memory usage should also be large, but this paper does not mention it. If so, the author can add some relevant descriptions to the paper. If not, the author can mention it in the paper and leave it for follow-up work.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "5x7LUvEQQI-", "reviewer": "Reviewer_k2Ew", "summary": "This paper introduces an end-to-end human pose estimation method, termed LOGO-CAP. It proposes a local-global contextual adaptation formulation and applies KAMs and the global KEMs to refine the estimated center-offset poses. It achieves 72.2 mAP on COCO val-2017 dataset. ", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\nThis work proposes a novel local-global contextual adaptation formation for accurate pose estimation. Comparing with bottom-up center-offset methods, LOGO-CAP achieves a better trade-off between accuracy and efficiency, which is important in the task of human pose estimation.\n\nCons:\n1.\tThis paper argues that “it is the first fully end-to-end trainable method that achieves state-of-the-art performance”, which is over-claimed. 1) There are several end-to-end methods for human pose estimation (e.g., Mask RCNN, bottom-up methods [35, 20] and bottom-up regression methods [28, 11]); 2) LOGO-CAP does not achieve the state-of-the-art performance (SimpleBaseline[32], HRNet[27]).\n2.\tThe “Slow” keypointer, which introduces KAMs and the global KEMs to refine the poses estimated by center-offset formulation, is applied at person instance level (or even at joint level). It is more like a top-down approach rather than a bottom-up approach. It is likely that the center-offset formulation only provides the coarse position of persons or joints, then they are processed individually. Their inference time increases with the number of people. The reviewer believes that both top-down approaches and bottom-up approaches should be compared in Table2.\n3.\tThis paper lacks of ablation study to show the effect of proposed modules. It should clearly introduce where the improvement over the baseline come from. The following experiment results should be given: a) the effect of local-global contextual adaptation; b) the effect of Gaussian reweighting; c) the effect of convolution message passing module; d) The reviewer is also curious about the result of applying the local window search (the argmax) on the KAMs, which is similar to Table.1 Emp. Bound. \n4.\tThe inference time of the proposed method increases with the number of persons, as mentioned above. The details of how to test the inference speed for multi-person images and the reasons why LGO-CAP show superiority over the bottom up methods (DEKR[11]) on inference speed should be given.\n5.\tThe inaccuracy of global KEMs comes from the global keypoint heatmaps, while the KAMs measure the error of center-offset results. Here is the mismatch between heatmaps and center-offsets. How can KAMs help refine global KEMs? \n6.\tThe writing can be improved. Notations in this paper are not well defined. There are also too many similar concepts with different names, e.g. Local heatmaps, local KEMs, KAMs. It would be better if they use a consistent name throughout the paper. Some important details of the proposed approach are missing (e.g., how to compute the geometric mesh of the global KEMs). ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. The authors adequately described the limitations of the work. \nThis paper does not have obvious potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received 3 strongly diverging final ratings: 4, 3, 8.
On the positive side, reviewers appreciated the central idea and strong empirical performance.
At the same time, most reviewers felt like ablations were insufficient, some of the claims were inaccurate, and overall the submission needed another major revision and a round of peer reviews before it can be considered for publication.
After an extensive discussion between the authors and the reviewers during the rebuttal period, the ratings remained unchanged.
Overall, the AC agrees with the skeptical reviewers that the weaknesses of this submission at this point outweigh its strengths, so the final recommendation is to reject.
Decision: Reject