Robust Pose Estimation in Crowded Scenes with Direct Pose-Level Inference
Authors: Dongkai Wang
Shiliang Zhang
Gang Hua
Venue: NeurIPS 2021 Poster (2021)
Keywords: Multi-Person Pose Estimation
Crowded Scenes Pose Estimation
Human Pose Estimation

Abstract: Multi-person pose estimation in crowded scenes is challenging because overlapping and occlusions make it difficult to detect person bounding boxes and infer pose cues from individual keypoints. To address those issues, this paper proposes a direct pose-level inference strategy that is free of bounding box detection and keypoint grouping. Instead of inferring individual keypoints, the Pose-level Inference Network (PINet) directly infers the complete pose cues for a person from his/her visible body parts. PINet first applies the Part-based Pose Generation (PPG) to infer multiple coarse poses for each person from his/her body parts. Those coarse poses are refined by the Pose Refinement module through incorporating pose priors, and finally are fused in the Pose Fusion module. PINet relies on discriminative body parts to differentiate overlapped persons, and applies visual body cues to infer the global pose cues.  Experiments on several crowded scenes pose estimation benchmarks demonstrate the superiority of PINet. For instance, it achieves 59.8% AP on the OCHuman dataset, outperforming the recent works by a large margin.

Robust Pose Estimation in Crowded Scenes with Direct Pose-Level Inference
Dongkai Wang Peking University dongkai wang@pku. edu. en
Shiliang Zhang Peking University
Gang Hua Wormpex Al Research
slzhang, jd10pku edu.. edu.cn cn
ganghua@gmail com
Abstract
Multi-person pose estimation in crowded scenes is challenging because overlapping and occlusions make it difficult to detect person bounding boxes and infer pose cues from individual keypoints. direct pose-level inference strategy that is free of bounding box detection and key- point grouping. Instead of inferring individual keypoints, the Pose-level Inference Network (PINet) directly infers the complete pose cues for person from his/her visible body parts. PINet first applies the Part-based Pose Generation (PPG) to infer multiple coarse poses for each person from his/her body parts. Those coarse poses are refined by the Pose Refinement module through incorporating pose priors, and finally are fused in the Pose Fusion module. PINet relies discriminative body parts to differentiate overlapped persons, and applies visual body cues to infer the global pose cues. Experiments several crowded scenes pose estimation benchmarks demonstrate the superiority of PINet. For instance, achieves 59.8% AP the OCHuman dataset, outperforming the recent works by large margin
address those issues, this paper proposes a
Introduction
Multi-Person Pose Estimation (MPPE) performs both detection and pose keypoint localization for all persons appearing an image. Due to importance in human activity understanding. human-object interaction, human parsing, et al., MPPE has attracted increasing attention in recent years. Current MPPE research can be summarized into two categories according to their pipelines, i.e. i) top-down methods [5] 18
29 225 which detect person bounding boxes and perform pose estimation for each bounding box, and ii) bottom-up methods [I] 20 17 that first detect body keypoints, then assign them into corresponding persons. A more detailed review will be presented in Sec. 2
The above two lines of research have significantly boosted the performance of MPPE. However, performing MPPE is still challenging in crowded scenarios, where occlusions and person overlapping commonly occur. is difficult to detect accurate bounding boxes due to overlapped persons. The commonly used Non-maximum Suppression (NMS) in detection frameworks also tends to suppress dense bounding boxes, leading to many undetected persons. Those issues degrade the accuracy of top-down methods. Person overlapping and occlusions lead to invisible keypoints, making it difficult for bottom-up methods to infer accurate pose cues from individual keypoints. As shown in many works, applying existing MPPE methods in crowded scenes gets inferior performance [10] 22
As illustated in Fig. (a) and (b), existing works generally suffer from difficulties of person detection and keypoint grouping in crowded scenarios. This paper targets to perform MPPE with new pipeline, which is free of bounding box detection and keypoint grouping. We regard the human pose as an inference objective, and directly infer the complete pose cues, i.e., locations of all body keypoints, for a person from his/her visible body parts. In other words, we rely on discriminative visible body
*ooee is available at: httppp///githubb..oo///eeeeeeeee http:///iccuuu..cooee/meeettetee http://///.eeeetec///Iiieee.ee
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

in
pose
refine
group
fusc
(8)
(b)
Figure 1: (a) Top-down methods suffer from the difficulty of detecting overlapped person in red bounding box; (b) Bottom-up methods are sensitive keypoint occlusions in red circles, thus can not infer accurate pose cues; (c) The proposed DPLI infers initial pose cues from body parts, e.g., three parts in the figure. Initial part cues are refined and fused to get the pose cues for the occluded person.
parts, e.g., heads, to differentiate overlapped persons. We apply visible part cues and contexts to infer the global pose, including both visible and invisible keypoints. As this pipeline directly infers complete pose cues, we name it as the Direct Pose-Level Inference (DPLI). This idea is also inspired
by works that infer invisible cues according to visible ones [2 22 30). For instance, PRNet [24
detects occluded persons according to visible body parts and person bounding box ratio prior.
We implement DPLI with the Pose-level Inference Network (PINet) that progressively infers the occluded person pose from coarse to fine. Due to the lack of visible part annotations, we divide each person into several parts according to settings in Fig. 3 to acquire and exploit visible parts. The Part-based Pose Generation (PPG) generates initial pose cues from each part. The subsequent Pose Refinement model employs dynamic graph convolution leverage contexts and pose structure prior for the pose refinement. Finally, PINet uses Pose Fusion model to identify pose cues belonging to the same person, and fuse them as the final pose estimation.
We test our method on crowded scenes pose estimation benckmarks including OCHuman and CrowdPose. Experiments results show that, our PINet achieves superior performance compared with recent works. For instance, we achieve 59.8% AP on OCHuman, outperforming the recent OPEC-Net 22 and DEKR 4 by 30.7% and 7.8%. On CrowdPose, PINet achieves 62.2% AP on the hard test set mostly composed of crowded scenes. On this challenging test set, PINet outperforms the recent DEKR by 4.7% in AP.
Some recent works also detect occluded keypoints from visible ones for pose estimation. OPEC- Net [22] detects occluded keypoints by applying Graph Convolution Network (GCN) 32 visible keypoints and contexts. However, OPEC-Net falls into the top-down category because it requires bounding boxes to discriminate persons, thus suffers from the difficulty of person detection. To the best of our knowledge, this is an original contribution on the Direct Pose-Level Inference (DPLI) for pose estimation in crowded scenarios, which differs from the previous top-down and bottom-up pipelines. The superior performance shown by extensive experiments validates the effectiveness of our proposed PINet and illustrates potentials of the DPLI pipeline.
Related Work
This work related with multi-person pose estimation and crowded scenes pose estimation. This section briefly reviews related works in these categories respectively.
Multi-Person Pose Estimation aims to estimate the pose of every person in the image. Most of related works can be summarized into two categories, i.e., top-down methods and bottom-up methods, respectively. Top-down methods first detect person bounding box by an object detector like YOLO 23 then perform single-person person pose estimation in the cropped image or feature map. Mask R-CNN 1 a typical method that directly adds a keypoint detection branch on Faster R-CNN to use the Rol pooling feature. G-RMI [18] and following methods break top-down methods into two steps and use separate models for person detection and pose estimation, respectively. SimpleBaseline 29
introduces strong pose estimation baseline by adding deconvolution to ResNet [6. Hourglass [14
and HRNet (55) propose new convolution network architectures to generate high quality and high resolution feature map for heatmap estimation, leading to superior performance.

---

Bottom-up methods first detect identity-free body keypoints for all persons in an image, then group those keypoints into individual persons. Heatmap widely adopted for keypoint detection and most existing bottom-up methods focus on identifying and grouping keypoints belonging to the same person. DeepCut [20] is an early work that formulates grouping as an integer linear program. OpenPose [1 proposes the part affinity field approach to learn a vector field linking two keypoints. Grouping is conducted by calculating line integral between two keypoints and grouping the pair with the largest integral. Associative Embedding [13 learns each keypoint with tag embedding and conducts the grouping by clustering the tag. PersonLab [7 groups keypoints by directly learning 2D offset field for each pair of keypoints. PifPaf [9] extends OpenPose and PersonLab by using part intensity field to localize body parts, then applies part association field to associate body parts to infer the human pose. HGG 7 introduces differentiable graph clustering to replace traditional
grouping operation and obtains superior performance. Several recent works [16] 33 15 28 densely
regress person poses from the person center point. Those methods suffer from issues of center point
occlusion in crowded scenes, and difficulty of long distance and dense regression [16 26
Crowded Scenes Pose Estimation has not been fully exploited by the research community. There are few works to solve multi-person pose estimation when persons are highly overlapped. CrowdPose [[O] constructs large crowded pose estimation benchmark and proposes apply global association to deal with crowded scenes. OPEC-Net [22) refines initial estimated pose by introducing the Graph Convolution Network. It requires bounding boxes to discriminate persons, thus suffers from the difficulty of person detection. The proposed PINet follows a novel Direct Pose-Level Inference pipeline, and is more robust difficulties of person detection and keypoint grouping in crowded scenes.
3 Methodology
3.1
Overview
Given image I with multiple persons, the goal of crowded scenes pose estimation is estimate locations of pose keypoints for each person, including both visible and invisible keypoints,
(fAC(), 1) MPPE(I)
where {K(() and (E()) denote visible and invisible keypoints for i-th person, respectively. m
represents the number of persons in image I. We use n = (()()) + {COO) J to denote the number of pose kcypoints for each person, e.g., 17 for OCHuman 31 and
14 for CrowdPose (IO].
Unlike previous works that infer individual keypoint K., PINet directly infers the complete pose cues ((C6)) J. [K(() }} for each person. To achieve this goal, we adopt the Structure Pose Representation (SPR) [6 to represent the pose cue as a point p and a collection of offsets i.e.,
(1) where (x, y) represents the coordinates of point p and (ōæ,, dy::) denotes the offset of k-th pose keypoint to p. According to Eq. 1 pose estimation can be conducted by detecting p and estimating offsets o.
P= {p. o) {(x, y), (ōark, dy)))==)),
should be carefully selected during the training stage, e.g., locate on discriminate parts to infer high quality pose cues and avoid occlusions. SPR (16) uses the center point as p. which could suffer from occlusions. We divide each person into s parts and use each part center as More detailed discussions on the body partition and the selection of p are presented in Sec. 3.2 and Fig.
To locate (p, 12-1 for each person and acquire corresponding (o, ) we propose the PPG to infer confidence map and offset map for each part, i.e.,
(2) where Py represents the confidence map indicating the location of the j-th part, and O, denotes the offset map, respectively, (-)) the backbone for feature extraction. Initial pose cues are acquired by applying the NMS on P, to get multiple local maximum points P, E P, and sampling corresponding offsets 0₂ E Oj. Each pair of {p., 0,} composes an initial pose cue denoted by Pj.
[P,, , = PPG F ((),
The coarse pose P/ can be refined by another PR module according to pose structure prior and contextual information included in the feature map F, iee.
P, PRPPPP F. G).
(3)
3

---

ConNNet
PPG
PR
PF
Part- based Poee Generatin
Pose Refinement
Par-1 Inder
DGUN
Port 2 Infer
Port-3 Intor
Figure 2: Framework of the proposed Pose-level Inference Network (PINet). We use a CNN to extract the feature map. Part-based Pose Generation (PPG) infers the whole person pose from different body parts, as denoted by circles in different colors. Pose Refinement (PR) refines the coarse person pose with dynamic pose keypoint graph. PF denotes the pose fusion operation that fuses the poses from different parts and output the final pose. We only show the pose of occluded person.
where Pj denotes the pose refined by PR.
Because each part predicts pose cue, there exist duplicate poses for the same person. We hence post-processing module PF to identify duplicate pose cues and fuse them as the final pose estimation,
(P(0) M PF PF((P)))=
where p(() denotes the final pose estimation for the i-th person, is the number of poses generated by PPG and PR. PF can be implemented by measuring similarities between pose cues and merging similar poses.
We learn PINet by training the backbone (()), PPG(-), and PR(-) modules in end-to-end manner. The overall training objective is denoted by
L LPPG + ALPR.
(5)
where LPPG and LPR denote the loss for two modules and À a weight to balance two losses. Fig. illustrates the pipeline of proposed PINet. With the PPG, PR and PF, pose even when they are highly overlapping, and are free of missed keypoint and person detection in previous methods. In the following, we will present details of PPG, PR and PF.
2
infer high quality person
3.2 Part-based Pose Generation
As stated above, we divide each person into s body parts according to body skeleton to exploit the visible information. Generally speaking, the finer the body partition, the more likely there exists visible parts. However, small part is hard to regress full person pose offset due to the lack of global contexts. Moreover, too large also leads the difficulty of dense regression optimization, which is discussed in [26].
To investigate prior body partition strategy, we refer to previous work 11 and body parts and design several typical body partition strategies, as shown in Fig. 3 The partition strategies are listed from coarse fine. For instance, center denotes that we divide person into one part, while keypoint indicates using 13 parts. To test the robustness of different partition strategies to occlusions, we show the occlusion ratio on OCHuman in Fig, Note that, partition without any visible parts is regard as an occlusion. As shown in the figure, finer body partition is more robust to occlusion. However, using too many parts is also harder for model optimization, which is verified in Sec. 4 We empirically find that part-large partition strategy achieves a good tradeoff. We hence divide each person into parts, i.e., upper, middle and bottom parts.
21 pose keypoint
According above analysis, PPG detects upper, middle and bottom parts each person and regresses corresponding offsets. Following keypoint-based detection work (33), detect each part

---

conder 5
head s
part Met-large = part-smalll s
keypoint, s = 13
occlusion ratio
025
015
011
0
conter
head
part- part-lagee
part-amall
keypoint
Figure Illustration of different strategies of body partition in PPG: (a) Five different partition strategies, rectangles denote the body parts and brown circles denote part centers, and (b) their corresponding occlusion statisties from OCHuman dataset.
by estimating the part center with heatmap P. Specifically, given the feature map F = (()) from backbone, we pass it through the keypoint detection head and offset regression head,
P, =
R2(F), {u, fu.mm.r m. b}
(6)
where p, Oj represent the confidence map of j-th part and corresponding offset map, respectively. u, m, denote three parts. We adopt the tradeoff 12 regression loss learn P, and smooth I loss to learn the offset map Oj.
Lpoint
M @ (P PT)
(7)
Laffset
smoothh, (o; - -;;,
(8)
S
IEN
where superscript indicates the groundtruth label. P= P.: mi Po] denotes the point maps, M is a mask to down weight background region, the weight of positions not lying in part center region is 0.1, and others are Ω denotes the index set of part center and its neighbor points on feature map, S; is the area of i-th person. We generate the ground truth following (16). The loss of PPG is the weight sum of above two losses, ie., LPPG Lpoint + Xoh fset. Ao is a weight to balance to losses.
3.3 Pose Refinement
Although PPG generate complete person pose, it still confronts several issues. Firstly, the part-based generation strategy enhance the robustness of PINet to occlusion, but eliminate the person global information for each part infer the whole person pose. i.e., the upper body part feature is hard to encode foot due to limited receptive field. Secondly, the occluded pose keypoint should be inferred by considering keypoint connections, namely pose structure, not only visual information.
Thus we are motivated to propose PR module to address above issues. For pose P/ generated by point feature fp in PPG, we supply it with keypoint features fk to encode the person global information, which can obtained by bilinear sampling on the feature map with k-th keypoint location of pose P To fully utilize pose structure pose prior, we propose dynamic graph convolution network (DGCN) that considers keypoint connection information to refine and aggregate these keypoint features,
X++ x++ =GX='
where W is the parameters of DGCN, denotes ReLU, X and X is the original and updated features from 1-th layer, and is the dynamic graph, which initialized according to the human body skeleton and updated the model parameters. We adopt 3 layers DGCN with residual connection to refine these features. Each row the initial feature matrix X represents one keypoint. Following (27), we apply max pooling and average pooling to the refined keypoint features to obtain the global features of the pose, which is concatenated with fp and passed to two fully connected layers predict the offset AP,, of initial pose P/ and the ground truth. Thus
obtain the refined pose by P, = P/J + APJ. Similar to the offsetmap loss, we calculate the refinement loss by,
1
L
smoothy, (P)+ + AP', P?),
(10)
T
ET S,
5

---

where T is the sampled pose set for PR refinement, S; is the area of i-th person.
Discussions. Previous OPEC-Net [22] also utilize GCN. However, PR is different from it in several aspects. Firstly, OPEC-Net focuses on inferring occluded keypoints and takes keypoint coordinates input, thus require extra feature adaptation module to extract context information. Our PR focuses on extracting better person global information with pose structure prior and does not require extra module. Secondly, PR adopts dynamic graph refine keypoint features and can capture more keypoint relationship. For example, the skeleton contains no connections between left and right body, However, when the right ankle is occluded, we could refer left ankle and knee for inference. The dynamic graph is more flexible and can capture these properties with model learning. Fig. 2 illustrates an example of the dynamic graph during training.
3.4 Pose Fusion
PPG generates duplicate poses and low-quality poses from invisible body parts, thus a pose- proseesrcsssing operation is required to eliminate them. However, duplicate poses from different body parts perform well on different keypoints due to long distance regression [16 33 thus we can fuse them to get better results. A pose regressed from upper body parts performs well on locating keypoints nose, while the pose inferred from the bottom body part gets poor performance on locating those keypoints.
According to above analysis, we propose PF operation to fuse duplicate poses and remove low- quality poses. Given the poses inferred by PPG and PR, i.e., {P, j= PF firsts find the pose set that belong to the same person. Inspired by OKS-NMS [88, we adopt a OKS based matching strategy to identify the poses belonging to the same person. For two poses P1 and Ps,, we calculate their object keypoint similarity by,
|P [P] k - PAIAIE (Si + S2 S2)
OKS(P,, P2)
_ exp(
(1I)
where P[k] denotes the k-th keypoint, S1 and S2 represent the area of two poses, 0g is a per- keypoint constant that controls falloff. If the OKS of two poses is larger than 7, they are regraded as duplicate poses. According above matching strategy and NMS procedure,
divide (?))5-1 into m groups, where each group {P));=1 represent i-th person pose set. Then PF traverse each pose set (P,}%% and select n keypoints with maximum score to construct the final person pose p(1), The detailed process can be seen in the supplementary material.
The k-th keypoint score is formed by r = Ty Tkk where rp is the confidence score of point from P. Th is the keypoint confidence score which can be obtained by learning an extra score map H for each pose keypoints, i.e., the heatmap in most MPPE methods. A low-quality pose generated by PPG has small rp. thus will not appear in the final results. Moreover, r'k is helpful in selecting well estimated keypoints and reliefing the long regression problem. During implementation, we learn the score map with the point map P and change Eq. to,
Lpoint
|| @ ([P: PP;
(12)
where [: denotes concatenation operation.
Discussions. PF is a model -agelsig operation and can also be applied to other regression methods, i.e., [16]. However, it is more effective our method when duplicate predictions are from different parts, not merely center points. We also validate the effectiveness of PF on other methods in Sec.4.
4 Experiments
4.1 Datasets and Evaluation Protocol
OCHuman 31 is a recently proposed benchmark to examine the limitations of MPPE in highly challenging scenarios. It consists of 4731 images in total, including 2500 images for validation and 2231 images for testing. The average IoU of the bounding boxes is 67%. Following [22,, we train models on val set and report the performance on test set.
CrowdPose [10] is another dataset that includes many crowded scenes. It contains 10K, 2K and 8K images for train, val and test set. Following 2 4. train models on the train and val

---

Table 1: Ablation studies of our proposed PINet on individual components.
(a) Effectiveness of each component in PINet.
(b) Test different body partition strategies in PPG.
PPG
PR
PF
AP
Ap30
Ap75
Part partition
AP
Apau
Ap
47.6
67.5
53.2
Center
47.6
67.5
53.2
53.1
70.5
60.4
Head
49.8
70.9
55.6
58.8
74.3
65.5
Part-large
53.1
70.5
60.4
55.8
72.7
62.3
Part-small
50.3
69.4
57.2
59.8
74.9
65.9
Keypoint
40.8
59.0
45.7
(c) Test different refine strategies in PR.
(d) Test PF in different methods.
Method
AP
A
Ap76
Method
w/o PF
w/ PF
No GCN
56.8
74.1
63.0
SPM [6
47.6
48.1 1
Static graph
57.7
74.7
64.3
DEKR
52.2
53.0
Dynamic graph
59.8
74.9
65.9
PINet(w/o PR)
53.1
55,8
sets and report the results the test set. Since CrowdPose contains both crowded and non-crowded scenes, also report the performance crowded test set, namely metric AP"
COCO 12 is the popular MPPE benchmark that contains over 200,000 images. COCO keypoint benchmakr contains train, val and test-dev set. The train set includes 57K images and 150K person instances annotated with 17 keypoints, the val set contains 5K images, and the test-dev set consists of 20K images. However, it includes few crowded scenes, and we use it to valid the generation ability of our method to non-occluded scenes. We train models on the train set and report the results on the val and test-dev sets. Moreover, following 77, We also directly test the model trained on COCO in OCHuman val and test set.
Evaluation Metrics We follow the standard evaluation metric and use OKS-based metrics for pose estimation. We report average precision scores with different thresholds: AP, AP and AP76 On CrowdPose, we also report AP". AP and AP" which refers to the performance under different crowd index.
4.2 Implementation Details
All experiments are implemented PyTorch [9
We adopt ImageNet
K pre-trained HRNet-
W32 [25] as backbone for all experiments and follow the most configuration of 2
4.. We set À and
X, to 0.03 and 0.7 for all experiments. For PR, sample 40 poses from PPG for each part.
Training. The input image is resized to 512*512. We use Adam [8] to optimize the model, the learning rate is set to 0.001 for all layers. We train the model for 140 epochs OCHuman and COCO, with learning rate dividing by 10 at 90th, 120th epoch. For CrowdPose, we train model with 300 epochs and divide learning rate by 10 200th, 260th epoch. The batch size is set to 20 OCHuman and 40 for CrowdPose and COCO. We adopt data augmentation strategies including random rotation (-30,30), scale ((0.75.1.55), translation ([-40,40) and flipping (0.5).
Testing. We resize the short side of the images to 512 and keep the aspect ratio between height and width. We adopt single and multi-scale (.0.5,1.0,2.0) test with flipping following 2
Comparison methods. To make fair comparison, we report the performance SPM B3
(with SPR),
HigherHRNet [2] and DEKR 4 in the same setting, i.e., backbone, training schedule and test setting.
4.3 Ablation Study
Components analysis. We analyze the effectiveness of each proposed components OCHuman.
The results are shown Table PINet with only PPG achieves 53.1 % AP, which already higher than previous center-based regression work. This indicates that inferring complete pose from visible part is important for occluded pose estimation. incorporate pose structure prior refine the coarse, it remarkably enhances the AP by 5.7%. We also evaluate the propose pose fusion operation, and
7

---

ILm FESTIV
ILm FESTIV
ILM FESTIV
1
Figure 4: Qualitative results OCHuman test set. Each image shows the prediction results of PPG, PPG+PR and PPG+PR+PF. can be observed that both PR and PF can improve the accuracy of keypoint localization.
Table 2: Comparison with state-of-the-ar methods on OCHuman test set. denotes our implemen- tation based on the author's public code.
Method
Reference
AP
A
Ap
Mask RCNN 15
ICCV2017
20.2
33.2
24.5
SimpleBaseline 29
ECCV2018
24.1
37.4
26.8
SPPE+ 110
CVPR2019
27.6
40.8
29,9
OPEC-Net [2
ECCV2020
29.1
41.3
31.4
SPM* [16]
ICCV2019
47.6
67.5
53.2
HigherHIRNet=
2
CVPR2020
27.7
66.9
15.9
DEKR*
CVPR2021
52.2
69.9
56.6
Ours
this paper
59.8
74.9
65.9
boosts the performance by 1.0%, which validates the effectiveness of fusing duplicate predictions. As shown in Fig. 4 both PR and PF improve the accuracy of keypoint localization.
Analysis on PPG. An important factor of PPG the choice body partition strategy. We compare different partition strategies in Fig. Table (b) summarizes the results. We observe that center part achieves 47.6% .6% AP, while head part obtains 49.8% AP. This is because that center part usually occluded by other person, thus PPG cannot generate high quality pose. While head and face discriminative regions, thus is more likely to be visible. The proposed part-large partition achieves the best performance, obtaining 53.1% AP, outperforming center and head points by 5.5% and 3.3% AP, respectively. We also investigate finer point selection, i.e., part-small and keypoint. However, they work badly due the hard optimization with dense regression.
Analysis on PR. We also analyze the component PR. As shown in Table (c). We first test to directly refine estimated pose by PPG without incorporating pose structure prior, and it obtains 56.8% AP. Incorporating pose keypoint graph to refine feature achieves 57.7% AP. It indicates that pose structure prior is useful for occluded pose inference. Moreover, when adding dynamic graph into PR. we can further boost AP to 59.8% AP, which verify the effectiveness of proposed methods.
Analysis on PF. To verify that PF is a model-agnostic operation, we apply it to other regression- methods, i.e., SPM and DEKR. As shown in T (d), PF can constantly boost these methods about 0.6-0.8 AP. which demonstrates the effectiveness of PF regression methods. Moreover, PF is more useful when we regress pose from different parts, and it boost the performance of PINet from 53.1 to 55.8. Note that PR can also relieve the regression problem, thus we omit here to verify PF.

---

Table 3: GFLOPs and #parameters of the representative top competitors and our approaches.
denotes our implementation.
SPMs [66
Nigh 2
DEKR [4]
PINet
Input size
512
512
512
512
#param.
29.3
28.6
29.6
29.4
GFLOPs
41.8
47.7
45.4
42.4
Table 4: Comparison with state-of-the-ari methods on CrowdPose test set. tation. i denotes multi-scale test.
denotes our implemen-
Method
Reference
AP
AP50
AP
AP
APM
AP
Mask RCNN [5]
ICCV2017
57.2
83.5
60.3
69.4
57.9
45.8
SimpleBaseline 229
ECCV2018
60.8
81.4
65.7
71.4
61.2
51.2
SPPE+ 10
CVPR2019
66.0
84.2
71.5
75.5
66.3
57.4
OpenPose 1
CVPR2017
62.7
48.7
32.3
SPM* [6]
ICCV2019
63.7
85.9
68.7
70.3
64.5
55.7
HigherHIRNet
2
CVPR2020
65.9
86.4
70.6
73.3
66.5
57.9
HigherHRNetj
CVPR2020
67.6
87.4
72.6
75.8
58.9
DEKR 4
CVPR2021
65.7
85.7
70.4
73.0
66.4
57.5
DEKR [4]
CVPR2021
67.0
85.4
72.4
75.5
68.0
56.9
Ours
this paper
68.9
88.7
74.7
75.4
69.6
61.5
Ourst
this paper
69.8
89.1
75.6
76.4
70.5
62.2
OCHuman
CrowdPose
70
65
AP(%
AP(%)
60
60
55
55
0.5-0.6
0.6-0.7
0.7-0.8
0.8-0.9
0.9-1.0
0.5-0.6
0.6-0.7
0.7-0.8
0.8-0.9
0.9-1.
Average loU
Average IoU
PINel
DEKR
SPM
Figure 5: Comparison with DEKR, SPM and our proposed PINet under different IoU threshold. Our method constantly outperforms other methods, especially at highly crowded scenes (high loU).
4.4 Comparison with Recent Works
compare PINet with recent works on OCHuman [3], CrowdPose 10] and COCO [2]. The
results are shown in Table 2 and 5 Table 2 reports comparisons OCHuman. We compare top-down methods: Mask RCNN [5], SimpleBaseline (29), SPPE+ [10] and OPEC-Net [22), and
bottom-up methods SPM [16), HigherHRNet 2 and recent DEKR 4 On OCHuman, top-down
methods achieve relatively low AP. This is due to the issues mentioned in Sec T Our PINet adopts a pose-level inference, thus achieves the best 59.8% AP OCHuman in single-scale test, which is higher than all bottom-up methods under the same setting. For instance, DEKR obtains 52.2% AP, which is lower than PINet by 7.6%.
CrowdPose dataset, our method also obtains superior performance. As shown Table PINet achieves 68.9% and 69.8% AP in single-scale and multi-scale test, which is higher than HigherHRNee and DEKR. Moreover, crowded scenes, PINet achieves 61.5% and 62.2% A which is higher

---

Table Comparison with State-of-the-ee methods on COCO and OCHuman, Models are trained on COCO train set.
COCO
OCHuman test
Method
Reference
val
test-dev
val
OpenPose
CVPR2017
61.8
AE 13
NeurIPS2017
56.6
32.1
29.5
LersonLab
ECCV2018
66.5
66.5
HGG 7
ECCV2020
60.4
35.6
34.8
HigherH 2
CVPR2020
67.1
66.4
DEKR
4
CVPR2021
68.0
67.3
37.8
36.4
ours
this paper
67.4
66.7
38.3
37.2
than DEKR by 4.0% and 5.3%, respectively. Results two occluded datasets demonstrate that PINet effective to deal with crowded scenes pose estimation.
To further investigate the robustness of our method to crowded scenes, we follow previous work [2 to split datasets according to the average IoU of bounding box. Since there are few or no images with IoU smaller than 0.5 on OCHuman, CrowdPose into parts and report the performance at each IoU. As shown Fig. 5 our method constantly outperforms DEKR and SPM, especially at high IoU. We also report the parameters and Our method requires less or similar
start from 0.5 and divide the test set of OCHuman and
GFLOPs of existing bottom-up methods, as shown in Table 3 parameters than other methods but shows better performance.
We also report the performance on COCO dataset, the results are shown Table We train PINet on COCO train set, and directly test it on COCO and OCHuman. PINet achieves 67.4% AP on COCO val set, which comparable recent methods, i.e., HigherHRNet 2 with 67. AP and PersonLab [17] with 66.5% AP, which indicates that PINet also works well on non-occluded scenes. Moreover, following 77. We also directly test the model trained on COCO in OCHuman val set and test set, and obtained 38.3% AP, which is higher than HGG by 2.7% and DEKR by 0.5%. This result demonstrates the generalization ability of proposed methods and its effectiveness handle crowded scenes.
Discussion and Conclusion
In this paper we address the multi person pose estimation crowded scenes. Our proposed PINet has shown considerable improvements in crowded scenes pose estimation. This is achieved by the direct pose-level inference pipeline that infers pose by leveraging both body part cues and pose structure prior. Experiments several crowded scenes pose estimation benchmarks demonstrate the effectiveness of proposed methods to handle occlusion. However, the performance is still lower than non-crowded scenes, and PINet directly regresses the pose, which is not accurate as the heatmap-based methods. Further studies are required to address these problems.
multi-person
crowded scenes
Acknowledgements
This work is supported in part by Peng Cheng Laboratory, in part by The National Key Research and Development Program of China under Grant No. 2018YFB01 18400, in part by Natural Science Foundation of China under Grant No. U20B2052, 61936011, 61620106009, in part by Beijing Natural Science Foundation under Grant No. JQ18012.
References
[1] Z. Cao, T. Simon, S.-E. Wei, and Y. Sheikh. Realtime multi person 2d pose estimation using part affinity fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2017.
multi-person
10

---

[2] B. Cheng, B. Xiao, J. Wang, H. Shi, T. Huang, and L. Zhang. Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5386-5395, 2020.
[3] Deng, W. Dong, Socher, L.-J. Li, Li, and L. Fei-Fei. Imagenet: large-scale hierarchical image database. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 248-255. Ioee, 2009.
[4] Z. Geng, K. Sun, B. Xiao, Z. Zhang, and J. Wang. Bottom-up human pose estimation via disentangled keypoint regression. arXiv preprint arXiv:2104002300 2021.
[5] K. He. G. Gkioxari, P. Dollár. and R. Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference on Computer Vision, pages 2961-2969, 2017.
[6] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 770-778, 2016.
[7] S. Jin, W. Liu, E. Xie, W. Wang, C. Qian, W. Ouyang, and P. Luo. Differentiable hierarchical graph grouping for multi-person pose estimation. Proceedings of the European Conference on Computer Vision, pages 718-734, 2020.
[8] D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
[9] S. Kreiss, L. Bertoni, and A. Alahi. Pifpal: Composite fields for human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 11977-11986, 2019.
[10] J. Li, C. Wang, H. Zhu, Y. Mao, H.-S, Fang, and C. Lu. Crowdpose: Efficient crowded scenes pose estimation and new benchmark. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10863-10872, 2019.
[11] J. Li, S. Zhang, Q Tian, M. Wang. and W. Gao. Pose-guided representation learning for person re- identification. IEEE Transactions on Pattern Analyxis and Machine Intelligence, 2019.
[12] T.-Y. Lin, M. Maire, S. Belongie, Hays, P. Perona, Ramanan, P. Dollár, and L. Zitmick. Microsoft coco: Common objects in context. In Proceedings of the European Conference on Computer Vision, pages 740-755. Springer, 2014.
[13] A. Newell, Z. Huang, and J. Deng. Associative embedding: End-to-end learning for joint detection and grouping. In Advances in neural information processing systems, pages 2277-2287, 2017.
[14] A. Newell, K. Yang, and J. Deng. Stacked hourglass networks for human pose estimation. In Proceedings of the European conference on computer vision, pages 483-499. Springer, 2016.
[15] X. Nie, J. Feng, J. Xing, and S. Yan. Pose partition networks for multi-person person pose estimation. In Proceedings of the European Conference on Computer Vision, pages 684-699, 2018
[16] X. Nie, J. Feng, J. Zhang, and S. Yan. Single-stage multi-person pose machines. In Proceedings of the IEEE International Conference on Computer Vision, pages 6951-6960, 2019.
[17] G. Papandreou, Zhu, L.-C. Chen, S. Gidaris, Tompson, and Murphy. Personlab: Person pose estimation and instance sogmentation with bottom-up part- based, geometric embedding model. In Proceedings of the European conference on computer vision, pages (69-286677 2018
[18] G. Papandreou, T. Zhu, N. Kanazawa, A. Toshev, J. Tompson, C. Bregler, and K. Murphy. Towards accurate multi-person pose estimation in the wild. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition, pages 4903-4911, 2017.
[19] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein. L. Antiga, et al. Pytorch: imperative style, pirh-peeffrmance deep learning library. In Advances in neural information processing systems, pages 6026-8037, 2019.
[20] L. Pishchulin, E. Insafutdinov, S. Tang. B. Andres, Andriluka, P. V. Gehler. and B. Schiele. Deepout: Joint subset partition and labeling for multi person pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4929-4931 2016.
[21] R. Pytel, O. S. Kayhan, and J. C. van Gemert, Tilting at windmills: Data augmentation for deep pose estimation does not help with occlusions. arXiv preprint arXiv:2010.1045, 2020.
11

---

[22] L. Qiu, X. Zhang, Y. Li, G. Li, X. Wu, Z. Xiong, X. Han, and S. Cui. Peeking into occluded joints: A novel framework for crowd pose estimation. In Proceedings of the European Conference Computer Vision, pages 488-504, 2020.
[23] J. Redmon, S. Divvala. R. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 779-788, 2016.
[24] X. Song, K. Zhao, W.S. C. H. Zhang, and Guo. Progressive refinement network occluded pedestrian detection. Proceedings of the European Conference on Computer Vision, volume 7. page 9, 2020.
[25] Sun. Xiao, Liu, and J. Wang. Deep high-resolution representation learning human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5693-5703, 2019.
[26] A. Varamesh and T. Tuytelaars. Mixture dense regression for object detection and human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13086- 13095, 2020.
[27] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon. Dynamic graph cnn for learning point clouds. Acm Transactions On Graphics, 38(5):1-12) 2019.
[28] F. Wei, X. Sun, H. Li. J. Wang, and S. Lin. Point-set anchors for object detection, instance segmentation and pose estimation. In Proceedings of the European Conference Computer Vision, pages 527-544, 2020.
[29] B. Xiao, H. Wu, and Y. Wei. Simple baselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision, pages 466-454 2018.
[30] S. Zhang, L. Wen, X. Bian, Z. Lei, and S. Z. Li. Occlusion-awee aware r-cnn: detecting pedestrians in crowd.
In Proceedings of the European Conference on Computer Vision, pages 637 653, 2018.
[31] S.-H. Zhang, R. Li, X. Dong, P. Rosin, Z. Cai, X. Han, D. Yang, H. Huang, and S.-M. Hu. Poselseg: Detection free human instance segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 889-898, 2019.
[32] L. Zhao, Peng. Y. Tian, M. Kapadia, N. Metaxas. Semantic graph convolutional networks for 3d human pose regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pages 3425-3435, 2019.
[33] X. Zhou, D. Wang, and Krähenbühh.. Objects as points, arXiv preprint 4xXiv:1944.0785 2019.
12


Reviews:
[{"review_id": "oIYUpRQRIsc", "reviewer": "Reviewer_wMdR", "summary": "The paper presents a bottom-up method for 2-D human pose estimation specifically targeting crowded scenes. The main idea is to predict heatmaps around a few anchor candidates (e.g., head, torso, legs), then regress the full body pose from them, followed by pose fusion to get rid of duplicates. It shows good evaluation results on crowded scene datasets.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper carefully combines ideas from existing bottom-up methods and designs a system specifically targeting scenes. One key limitation of some existing single-shot systems that operate by only setting an anchor on the center keypoint is that the center may be occluded or overlapping in crowded scenes. Having multiple anchors allows the proposed method to be more robust against such occlusions.\n\nThe paper also proposes a carefully crafted pose fusion mechanism which allows mixing and matching keypoint estimates emanating from different anchors, which improves performance.\n\nThe reported results on datasets featuring heavily crowded or occluded persons are very good. Results on the more standard COCO dataset (reported in the supplementary material) are also reasonably good but not state of art.\n\nOverall I find that the paper presents a well-engineered human pose estimation system tuned for crowded scenes, which would be more appropriate for a computer vision conference. I think that the paper's methodological advancements and overall novelty is not sufficient enough for publication in NeurIPS.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "hOO4x8W9N-", "reviewer": "Reviewer_8zdn", "summary": "In this paper the authors study the problem of the multi-person pose estimation in crowded scenes. They propose a new method which directly infers poses by leveraging both body part cues and pose structure prior. In the experiments, the authors could show some promising results on crowded scenes.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this paper is well written and easy to follow. The author studies a challenging problem of pose estimation for crowded people and the proposed approach works well compared with existing approaches. \n- The idea of direct pose inference is interesting and could be quite efficient by getting rid of person detection in the pipeline.\n- The proposed framework consists of part-based pose generation, pose refinement and pose fusion. Pose refinement module and pose fusion module are adapted from existing methods and method novelty might be incremental.\n- The experiments are conducted on three benchmarks: OCHuman, CrowdPose and COCO. The comparisons with state-of-the-art are quite convincing.\n\nI have a few questions and comments for this paper.\n- How would the proposed approach handle keypoints that are out of view? Would it be possible to further adapt the part-based pose generation module to different modes (e.g., head only, upper-body only) instead of always predicting the whole body?\n- The design for body partition seems a little bit ad-hoc. Why the authors discard arms for part-large in the body partition illustrated in Fig. 3? If this is determined empirically, could you explain why a variant of this strategy which include arms in the torso part perform worse? Is this design optimal for all the datasets?\n- In Fig.4, it is hard for me to tell whether PPG+PR or PPG+PR+PF performs better. Could you provide more example results to illustrate under what scenarios do PF improve?\n- The experiments on COCO are quite important to validate whether the proposed approach obtains better performance for occluded people poses with the tradeoff of regular poses. The authors should consider moving this experiment from supplementary to the main text in the final version.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Overall this paper studies an under-explored problem of pose estimation for crowded people. Overall I feel this paper is well prepared and I incline to accept this paper. The authors are encouraged to respond my questions in the rebuttal.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "dXVcVzLPpQM", "reviewer": "Reviewer_AzWE", "summary": "This paper presents a deep network method to address occlusions for multi-person pose estimation in a single image. The key idea is to partition each person into several part groups. While the whole body may be occluded, at least one of the part groups may be visible. The proposed network predicts the center of each part group and the offsets towards all the joints for the whole body. The predicted joints from multiple part centers are further refined with dynamic graph convolution network and then fused to make the final prediction. Results are reported on two pose datasets targeted at occluded scenarios with ablation study and comparisons to related works. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1) This paper looks at a critical problem of occlusion for multi-person pose estimation in crowed scenes with inter-person or person-object interactions. The proposed part-based multiple pose hypothesis generation is an interesting idea that extends the previous work [15] by X. Nie et al. in ICCV 2019. The idea was first verified in occlusion statistical study (Figure 3) and then the effect of the number of part groups is ablated in Table 1[b]. \n2) Overall, this paper is well-written. Most of the tech components are explained clearly with a proper amount of math. \n3) Results on MSCOCO in supp helps verify the generalization ability of the proposed method. \n\nCons:\n1) The presentation of dynamic graph convolution network is lack of clarity and sufficient details, like how many iterations of updates and how the graph is constructed. The equation (9) is not self-explained. It'd be better to explain its meanings and effects in the context of pose estimation. \n2) The proposed network was trained on OCHuman validation set with 2500 images. In Table 2, this paper compared the proposed method with related works on OCHuman test set, but it is hard to tell which compared methods were trained on its validation set. Training on OCHuman validation set seems to be critical for its accuracy. In Table 1(b), the Center-based prediction is pretty similar to [15] without dedicated efforts for handling occlusions but still achieves good results. It'd be good to clarify the training information for Table 2. If the compared methods were not trained or fine-tuned on OCHuman validation, it'd be also interesting to do so and then compare. \n3) Table 4 missed an entry from OPEC-Net [21] which achieves 70.6 mAP higher than the proposed method. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It'd be good to present failure cases. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "HLvs8UsBCKM", "reviewer": "Reviewer_ciZS", "summary": "The paper introduces a novel method that is able to extract human pose information in crowded scenes. The authors present a bottom-up  method that does not require detection or keypoint grouping. Their method infers initial pose cues from body part. The initial part cues are refined and fused to get the pose cues for the occluded people. They evaluate their work on 2 datasets. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Hereafter, I will share the strengths and weaknesses of the paper:\n\n[Strengths]\n- The problem is relevant.\n- Good ablation study. \n\n[Weaknesses]\n-\tThe statement in the intro about bottom up methods is not necessarily true (Line 28). Bottom-up methods do have a receptive fields that can infer from all the information in the scene and can still predict invisible keypoints. \n-\tSeveral parts of the methodology are not clear. \n\t\t- PPG outputs a complete pose relative to every part’s center. Thus O_{up} should contain the offset for every keypoint with respect to the center of the upper part. In Eq.2 of the supplementary material, it seems that O_{up} is trained to output the offset for the keypoints that are not farther than a distance \\textit{r) to the center of corresponding part. How are the groundtruths actually built? If it is the latter, how can the network parts responsible for each part predict all the keypoints of the pose.\n\t\t- Line 179, what did the authors mean by saying that the fully connected layers predict the ground-truth in addition to the offsets?\n\t\t- Is \\delta P_{j} a single offset for the center of that part or it contains distinct offsets for every keypoint?\n\t\t- In Section 3.3, how is G built using the human skeleton? It is better to describe the size and elements of G. Also, add the dimensions of G,X, and W to better understand what DGCN is doing. \n-\tExperiment can be improved:\n\t\t- For instance, the bottom-up method [9] has reported results on crowdpose dataset outperforming all methods in Table 4 with a ResNet-50 (including the paper one). It will be nice to include it in the tables\n\t\t- It will be nice to evaluate the performance of their method on the standard MS coco dataset to see if there is a drop in performance in easy (non occluded) settings. \n\t\t- No study of inference time. Since this is a pose estimation method that is direct and does not require detection or keypoint grouping, it is worth to compare its inference speed to previous top-down and bottom-up pose estimation method.\n\t\t- Can we visualize G, the dynamic graph, as it changes through DGCN? It might give an insight on what the network used to predict keypoints, especially the invisible ones.\n\n[Minor comments]\n- \tIn Algorithm 1 line 8 in Suppl Material, did the authors mean Eq 11 instead of Eq.4?\n-\tFig1 and Fig2 in supplementary are the same\n-\tSpelling Mistake line 93: It it requires…\n-\tWhat does ‘… updated as model parameters’ mean in line 176\n-\tDo the authors mean Equation 7 in line 212?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have talked about limitations in Section 5 and have mentioned that there are not negative societal impacts.  ", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "8VhcMZtXclF", "reviewer": "Reviewer_1ttQ", "summary": " This paper presents a one-stage pose estimation approach that applies PPG to infer multiple poses from multiple body parts (head, upper body, and lower body). These coarse poses are then refined by a GCN module. Finally, the Pose Fusion module is applied to get the final result.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The overall idea of the paper is easy to follow.\n- The paper reads smooth.\n - The proposed method achieves the state-of-the-art performance on two recently proposed datasets, i.e. crowdpose and ochuman datasets. \n- The ablation study clearly shows the improvement of different modules.\n\nCons:\n- The novelty of this paper is limited. Using graph networks to refine the initial pose is not new. [Ref1] construct dynamic graphs to tolerate large variations in the human pose. And [21] propose an Image-Guided Progressive GCN to estimate the invisible joints to deal with the occlusions. [Ref2] propose a two-stage graph-based method that adopts a graph pose refinement module to model the human structure. Line58 “this is an original contribution on the Direct Pose-Level Inference (DPLI) for pose estimation in crowded scenarios, which differs from the previous top-down and bottom-up pipelines.” is over-claimed. As the GCN is simply applied to each initial pose for pose refinement, there is no much difference, and cannot be considered novel.\n[Ref1]  Z. Qiu, K. Qiu, J. Fu, and D. Fu, “Dgcn: Dynamic graph convolutional network for efficient multi-person pose estimation,” National Conference on Artificial Intelligence, 2020. \n[Ref2] J. Wang, X. Long, Y. Gao, E. Ding, and S. Wen, “Graph-pcnn: Two stage human pose estimation with graph pose refinement,” In European Conference on Computer Vision. Springer, 2020. \n- As stated above, some related works about GCN are missing [Ref1, Ref2]. Line86-87, detailed comparisons with previous works [15,32,14,27] are required. As most of them follow a similar one-stage pose estimation paradigm. \n-The proposed approach does not achieve the state-of-the-art performance on COCO (as reported in supplementary, mAP 67.2 this paper vs 68.0 DEKR). COCO and MPII are the most popular and standard benchmarks for evaluating human pose estimation, while OCHuman is a recently proposed dataset and there are only a few officially reported results. The readers will more care about the results on COCO and MPII. \n-The limitations are not adequately stated. Especially, why the performance on COCO is lower than other approaches. \n-If I understand correctly, the Pose Fusion module is a post-processing algorithm, which makes the whole pipeline not fully end-to-end trainable. The matching/fusion loss (or the loss of the final output pose) cannot be back-propagated to the backbone network.\n-Please also report the runtime performance (inference speed).\n\nMinor:\n-Why do you highlight AP^H in red in Table4? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are not adequately stated. Especially, why the performance on COCO is lower than other approaches. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues found.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "7or4OIEMdvp", "reviewer": "Reviewer_umsY", "summary": "This paper proposes a method for robustly estimating human poses in crowded scenes. The main idea is to first detect visible body parts, then estimate human poses from the body parts. Based on these estimated human poses, refinement and fusion are performed. Experiments on OCHuman and CrowdPose show the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is very well-written and the figures are carefully designed to clearly present the idea. It is a pleasure to read the paper.\n- The idea is novel: it starts from part detection instead of purely top-down or bottom-up like most previous methods.\n- One of the modules (pose fusion) in the proposed method is general and can be applied to other pose estimation frameworks. \n- Experiments are well designed and the results are promising.\n\nOverall I think the paper is interesting to the audiences in NeurIPS. Some comments rather than weaknesses:\n- It would be very interesting to know how accurate are the poses predicted from different parts. For example, how accurate are the poses predicted from the head alone?\n- How are the different parts contributing to the final results? Ablation studies can be done to see what the results are if some parts are ablated. \n- Would it be better if different pose refinement networks are learned for different parts? It might be useful, but it is also possible that the effect is very limited.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It doesn't seem to me that there's a potential negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "2KAPqh1Nqo1", "reviewer": "Reviewer_M2eL", "summary": "This paper proposes a three-step approach to conduct multi-person pose estimation with occlusion. Firstly, their framework starts with a part-based pose generation (PPG) module aiming at providing a set of body pose candidates, where the initial prediction could be less considering the holistic body pose information. Then, a dynamic graph convolutional network (DGCN) is applied to incorporate the holistic pose structure information for the whole set's pose refinement (PR). Finally, a pose fusion (PF) module is proposed to merge and ensemble from the multiple outputs of the previous two steps. \n\nThe paper builds a sophisticated framework to conduct the key point detection task, where each of the module is with careful design. The contribution of this paper is combining the pose regression (or termed PPG in the paper), graph based pose refinement and pose fusion together to achieve some good performance, where the architecture design and training could be non-trivial. There are some informative discussions such as, which body partition would be better for the optimization (Fig. 3), the importance of each of their proposed module (Table 1). Further, the consistent advantageous results achieved on two pose estimation datasets, OCHuman and CrowdPose.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Originality:\n\nThe overall framework is consists of three major components, PPG, PR and PF. Such combination is not seen in any of the related work. However, if looking into each of the component itself, the part-based pose generation essentially is a pose regression problem to regress both the center heat map P and the offset heat map O, which is a general representation as appeared in [15]. PR leverages dynamic graph convolutional network to conduct the key point refinement, where similar work [21] has applied the GCN (progressive GCN) in the framework to solve the same problem. The authors provide some discussion regarding their DGCN difference from the GCN in [21]. It is not very clear how their proposed G is dynamic, and how such dynamic graph is determined based on each pose map input. Finally, the PF step leverages the OKS proposed from [17] as the rule to determine the duplicate pose detections. Overall, I think the originality is okay but not strong enough.\n\n* quality:\n\nThe paper provides a good structure with sufficient problem and background introduction. The connection of their idea to the related work is also well addressed. The figures are clear, meaningful and easy for the readers to understand. There are sufficient implementation details for the readers to try and replicate their method. The experiment is towards thorough to cover most of their proposed modules, where consistent advantageous performance is achieved. Overall, I think the presentation quality of the paper is good and above average.\n\n* clarity:\n\nFor large part of their method, it is clear. However, there are also some other places that the authors need to take care of. \n\n  ** It is not straight-forward why the proposed PPG can deal with occlusion. As generally, PPG is based on the appearance input to regress the key point heat map, each part detects its own key points. There is no extra setting for the key points to indicate its occlusion likelihood. \n\nFurther, the occlusion can be inferred because of the holistic pose structure, where the GCN based pose refinement is to deal with the occlusion. It is also evidence in [21], where it states in Sec. 3.2 \"estimated position of invisible joints from the base module is sometimes far from their correct locations and this makes it challenging to directly regress their displacements Therefore, we design an intuitive coarse-to-fine learning mechanism in the coordinate-based module, which builds a progressive GCN architecture and leverages the performance steadily by enforcing multi-scale image features in a progressive manner\". \n\nIf the above is correct, the dealing of occlusion is from GCN, not from PPG. It'll require the authors to carefully think about their design and rephrase their motivation towards each of the proposed module.\n\n  ** Line 207, r_k is obtained from H: extra score map, what is the difference between r_k and r_p?\n\nThe authors mention to obtain r_k from ''heat map'' H. However, usually, the heat map is mostly regarding to the pose heat map P. It is not clear why there are several ''heat maps''. And why need this extra heat map H to infer the r_k for the confidence score? Would r_p be sufficient to tell the confidence here?\n\n  ** The method emphasis the capability of handling occlusion. There is dataset OCHuman which is the occluded human body dataset. From the experiments, there is average precision (AP) to indicate the performance, i.e., Table 1 (a) to indicate the effectiveness of each module. However, it is not clear which part is to handle the occlusion. It would be good to separate the testing set into occlusion-free subsets and occlusion subsets, which at least would further tell under occlusion heavy situation, what module is critical in handling it.\n\nOverall, I think the clarify of the paper needs further improvement, where there are several key parts need to be clarified.\n\n* Significance\n\nThe overall framework provides a reasonable solution to achieve the multi-body pose estimation. There are several comments that need the authors to further clarify.\n\n**  for part-based pose generation (PPG), according to the loss design, (Eqn. 7 and 8), the optimization are sort of deterministic, whereas it is not likely to have many optimal P and O.\n\nIt needs further clarification under what constraint with the objective of Eqn. 7 and 8, the optimization will lead to “duplicate poses and low-quality poses from invisible body parts”.\n\n** consider the order of pose refinement (PR) and pose fusion (PF), given that PPG is likely to generate multiple pose hypothesis, what would be the better order, i.e., PPG->PF->PR or PPG-PR-PF? \n\nit is likely from the inaccurate PPG, pose refinement further lead to noise amplified results, where the pose fusion is hard to vote the scattered key points.  While after PPG, firstly conduct pose fusion would provide a relatively more confident intermediate pose heat maps. The further pose refinement (PR) would refine to reduce the local distortions.\n\nIt would be good to demonstrate the effectiveness of the order of conducting PR and PF.\n\n** it lacks the ablation with or without the M for the loss. Usually the human body crop is already with little background, which is needed to verify how different by adding this background mask for the loss design.\n\n** it lacks cross dataset evaluation. For the shared joints, it would be good to see the performance. As network is likely to be overfitting to the specific dataset. Further, if the distribution of the training set (validation set) and the testing set is exactly the same, it is actually not convincing by showing high performance on the testing set.\n    \n-minor:\n* Would non-maximum suppression (NMS) achieving similar effect as the proposed PF? If not, please provide some analysis or experimental results for that.\n\n* Inconsistent naming of the proposed method, PINet appeared in most of the places, where DPLI appeared in most of the introduction part, and in Figure 1 caption. It needs to be unified. \n\nBy combining all the above aspects together, I think the paper provides a relatively original solution for a mature field of human pose estimation. However, there are several parts that need the authors to carefully identify, such as the occlusion handling, the design of PPG->PR->PF, whether the PPG will generate many P and O for the later PF stage and so on.  I would like to further hear from the authors for their analysis.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, for the human images they used in the paper, the authors all masked out the face area for the privacy consideration.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6hrs"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Initially, two of the reviewers expressed concerns about the paper (lack of clarity and limited novelty) and ranked the paper marginally below acceptance. As the ensuing rebuttal managed to successfully address most of reviewer’s concerns, ACs and the majority of the reviewers agreed that this is a strong paper that deserves acceptance. Authors are highly encouraged to address the key comments reported by reviewers as well as to implement all the improvements (as indicated by authors in the rebuttal) in the final camera-ready version.
Decision: Accept (Poster)