Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a deep network method to address occlusions for multi-person pose estimation in a single image. The key idea is to partition each person into several part groups. While the whole body may be occluded, at least one of the part groups may be visible. The proposed network predicts the center of each part group and the offsets towards all the joints for the whole body. The predicted joints from multiple part centers are further refined with dynamic graph convolution network and then fused to make the final prediction. Results are reported on two pose datasets targeted at occluded scenarios with ablation study and comparisons to related works. 
main_review: Pros:
1) This paper looks at a critical problem of occlusion for multi-person pose estimation in crowed scenes with inter-person or person-object interactions. The proposed part-based multiple pose hypothesis generation is an interesting idea that extends the previous work [15] by X. Nie et al. in ICCV 2019. The idea was first verified in occlusion statistical study (Figure 3) and then the effect of the number of part groups is ablated in Table 1[b]. 
2) Overall, this paper is well-written. Most of the tech components are explained clearly with a proper amount of math. 
3) Results on MSCOCO in supp helps verify the generalization ability of the proposed method. 

Cons:
1) The presentation of dynamic graph convolution network is lack of clarity and sufficient details, like how many iterations of updates and how the graph is constructed. The equation (9) is not self-explained. It'd be better to explain its meanings and effects in the context of pose estimation. 
2) The proposed network was trained on OCHuman validation set with 2500 images. In Table 2, this paper compared the proposed method with related works on OCHuman test set, but it is hard to tell which compared methods were trained on its validation set. Training on OCHuman validation set seems to be critical for its accuracy. In Table 1(b), the Center-based prediction is pretty similar to [15] without dedicated efforts for handling occlusions but still achieves good results. It'd be good to clarify the training information for Table 2. If the compared methods were not trained or fine-tuned on OCHuman validation, it'd be also interesting to do so and then compare. 
3) Table 4 missed an entry from OPEC-Net [21] which achieves 70.6 mAP higher than the proposed method. 

limitations_and_societal_impact: It'd be good to present failure cases.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper presents a bottom-up method for 2-D human pose estimation specifically targeting crowded scenes. The main idea is to predict heatmaps around a few anchor candidates (e.g., head, torso, legs), then regress the full body pose from them, followed by pose fusion to get rid of duplicates. It shows good evaluation results on crowded scene datasets.
main_review: The paper carefully combines ideas from existing bottom-up methods and designs a system specifically targeting scenes. One key limitation of some existing single-shot systems that operate by only setting an anchor on the center keypoint is that the center may be occluded or overlapping in crowded scenes. Having multiple anchors allows the proposed method to be more robust against such occlusions.

The paper also proposes a carefully crafted pose fusion mechanism which allows mixing and matching keypoint estimates emanating from different anchors, which improves performance.

The reported results on datasets featuring heavily crowded or occluded persons are very good. Results on the more standard COCO dataset (reported in the supplementary material) are also reasonably good but not state of art.

Overall I find that the paper presents a well-engineered human pose estimation system tuned for crowded scenes, which would be more appropriate for a computer vision conference. I think that the paper's methodological advancements and overall novelty is not sufficient enough for publication in NeurIPS.
limitations_and_societal_impact: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a three-step approach to conduct multi-person pose estimation with occlusion. Firstly, their framework starts with a part-based pose generation (PPG) module aiming at providing a set of body pose candidates, where the initial prediction could be less considering the holistic body pose information. Then, a dynamic graph convolutional network (DGCN) is applied to incorporate the holistic pose structure information for the whole set's pose refinement (PR). Finally, a pose fusion (PF) module is proposed to merge and ensemble from the multiple outputs of the previous two steps. 

The paper builds a sophisticated framework to conduct the key point detection task, where each of the module is with careful design. The contribution of this paper is combining the pose regression (or termed PPG in the paper), graph based pose refinement and pose fusion together to achieve some good performance, where the architecture design and training could be non-trivial. There are some informative discussions such as, which body partition would be better for the optimization (Fig. 3), the importance of each of their proposed module (Table 1). Further, the consistent advantageous results achieved on two pose estimation datasets, OCHuman and CrowdPose.
main_review: * Originality:

The overall framework is consists of three major components, PPG, PR and PF. Such combination is not seen in any of the related work. However, if looking into each of the component itself, the part-based pose generation essentially is a pose regression problem to regress both the center heat map P and the offset heat map O, which is a general representation as appeared in [15]. PR leverages dynamic graph convolutional network to conduct the key point refinement, where similar work [21] has applied the GCN (progressive GCN) in the framework to solve the same problem. The authors provide some discussion regarding their DGCN difference from the GCN in [21]. It is not very clear how their proposed G is dynamic, and how such dynamic graph is determined based on each pose map input. Finally, the PF step leverages the OKS proposed from [17] as the rule to determine the duplicate pose detections. Overall, I think the originality is okay but not strong enough.

* quality:

The paper provides a good structure with sufficient problem and background introduction. The connection of their idea to the related work is also well addressed. The figures are clear, meaningful and easy for the readers to understand. There are sufficient implementation details for the readers to try and replicate their method. The experiment is towards thorough to cover most of their proposed modules, where consistent advantageous performance is achieved. Overall, I think the presentation quality of the paper is good and above average.

* clarity:

For large part of their method, it is clear. However, there are also some other places that the authors need to take care of. 

  ** It is not straight-forward why the proposed PPG can deal with occlusion. As generally, PPG is based on the appearance input to regress the key point heat map, each part detects its own key points. There is no extra setting for the key points to indicate its occlusion likelihood. 

Further, the occlusion can be inferred because of the holistic pose structure, where the GCN based pose refinement is to deal with the occlusion. It is also evidence in [21], where it states in Sec. 3.2 "estimated position of invisible joints from the base module is sometimes far from their correct locations and this makes it challenging to directly regress their displacements Therefore, we design an intuitive coarse-to-fine learning mechanism in the coordinate-based module, which builds a progressive GCN architecture and leverages the performance steadily by enforcing multi-scale image features in a progressive manner". 

If the above is correct, the dealing of occlusion is from GCN, not from PPG. It'll require the authors to carefully think about their design and rephrase their motivation towards each of the proposed module.

  ** Line 207, r_k is obtained from H: extra score map, what is the difference between r_k and r_p?

The authors mention to obtain r_k from ''heat map'' H. However, usually, the heat map is mostly regarding to the pose heat map P. It is not clear why there are several ''heat maps''. And why need this extra heat map H to infer the r_k for the confidence score? Would r_p be sufficient to tell the confidence here?

  ** The method emphasis the capability of handling occlusion. There is dataset OCHuman which is the occluded human body dataset. From the experiments, there is average precision (AP) to indicate the performance, i.e., Table 1 (a) to indicate the effectiveness of each module. However, it is not clear which part is to handle the occlusion. It would be good to separate the testing set into occlusion-free subsets and occlusion subsets, which at least would further tell under occlusion heavy situation, what module is critical in handling it.

Overall, I think the clarify of the paper needs further improvement, where there are several key parts need to be clarified.

* Significance

The overall framework provides a reasonable solution to achieve the multi-body pose estimation. There are several comments that need the authors to further clarify.

**  for part-based pose generation (PPG), according to the loss design, (Eqn. 7 and 8), the optimization are sort of deterministic, whereas it is not likely to have many optimal P and O.

It needs further clarification under what constraint with the objective of Eqn. 7 and 8, the optimization will lead to “duplicate poses and low-quality poses from invisible body parts”.

** consider the order of pose refinement (PR) and pose fusion (PF), given that PPG is likely to generate multiple pose hypothesis, what would be the better order, i.e., PPG->PF->PR or PPG-PR-PF? 

it is likely from the inaccurate PPG, pose refinement further lead to noise amplified results, where the pose fusion is hard to vote the scattered key points.  While after PPG, firstly conduct pose fusion would provide a relatively more confident intermediate pose heat maps. The further pose refinement (PR) would refine to reduce the local distortions.

It would be good to demonstrate the effectiveness of the order of conducting PR and PF.

** it lacks the ablation with or without the M for the loss. Usually the human body crop is already with little background, which is needed to verify how different by adding this background mask for the loss design.

** it lacks cross dataset evaluation. For the shared joints, it would be good to see the performance. As network is likely to be overfitting to the specific dataset. Further, if the distribution of the training set (validation set) and the testing set is exactly the same, it is actually not convincing by showing high performance on the testing set.
    
-minor:
* Would non-maximum suppression (NMS) achieving similar effect as the proposed PF? If not, please provide some analysis or experimental results for that.

* Inconsistent naming of the proposed method, PINet appeared in most of the places, where DPLI appeared in most of the introduction part, and in Figure 1 caption. It needs to be unified. 

By combining all the above aspects together, I think the paper provides a relatively original solution for a mature field of human pose estimation. However, there are several parts that need the authors to carefully identify, such as the occlusion handling, the design of PPG->PR->PF, whether the PPG will generate many P and O for the later PF stage and so on.  I would like to further hear from the authors for their analysis.

limitations_and_societal_impact: Yes, for the human images they used in the paper, the authors all masked out the face area for the privacy consideration.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper the authors study the problem of the multi-person pose estimation in crowded scenes. They propose a new method which directly infers poses by leveraging both body part cues and pose structure prior. In the experiments, the authors could show some promising results on crowded scenes.
main_review: Overall this paper is well written and easy to follow. The author studies a challenging problem of pose estimation for crowded people and the proposed approach works well compared with existing approaches. 
- The idea of direct pose inference is interesting and could be quite efficient by getting rid of person detection in the pipeline.
- The proposed framework consists of part-based pose generation, pose refinement and pose fusion. Pose refinement module and pose fusion module are adapted from existing methods and method novelty might be incremental.
- The experiments are conducted on three benchmarks: OCHuman, CrowdPose and COCO. The comparisons with state-of-the-art are quite convincing.

I have a few questions and comments for this paper.
- How would the proposed approach handle keypoints that are out of view? Would it be possible to further adapt the part-based pose generation module to different modes (e.g., head only, upper-body only) instead of always predicting the whole body?
- The design for body partition seems a little bit ad-hoc. Why the authors discard arms for part-large in the body partition illustrated in Fig. 3? If this is determined empirically, could you explain why a variant of this strategy which include arms in the torso part perform worse? Is this design optimal for all the datasets?
- In Fig.4, it is hard for me to tell whether PPG+PR or PPG+PR+PF performs better. Could you provide more example results to illustrate under what scenarios do PF improve?
- The experiments on COCO are quite important to validate whether the proposed approach obtains better performance for occluded people poses with the tradeoff of regular poses. The authors should consider moving this experiment from supplementary to the main text in the final version.



limitations_and_societal_impact: Overall this paper studies an under-explored problem of pose estimation for crowded people. Overall I feel this paper is well prepared and I incline to accept this paper. The authors are encouraged to respond my questions in the rebuttal.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper introduces a novel method that is able to extract human pose information in crowded scenes. The authors present a bottom-up  method that does not require detection or keypoint grouping. Their method infers initial pose cues from body part. The initial part cues are refined and fused to get the pose cues for the occluded people. They evaluate their work on 2 datasets. 
main_review: Hereafter, I will share the strengths and weaknesses of the paper:

[Strengths]
- The problem is relevant.
- Good ablation study. 

[Weaknesses]
-	The statement in the intro about bottom up methods is not necessarily true (Line 28). Bottom-up methods do have a receptive fields that can infer from all the information in the scene and can still predict invisible keypoints. 
-	Several parts of the methodology are not clear. 
		- PPG outputs a complete pose relative to every part’s center. Thus O_{up} should contain the offset for every keypoint with respect to the center of the upper part. In Eq.2 of the supplementary material, it seems that O_{up} is trained to output the offset for the keypoints that are not farther than a distance \textit{r) to the center of corresponding part. How are the groundtruths actually built? If it is the latter, how can the network parts responsible for each part predict all the keypoints of the pose.
		- Line 179, what did the authors mean by saying that the fully connected layers predict the ground-truth in addition to the offsets?
		- Is \delta P_{j} a single offset for the center of that part or it contains distinct offsets for every keypoint?
		- In Section 3.3, how is G built using the human skeleton? It is better to describe the size and elements of G. Also, add the dimensions of G,X, and W to better understand what DGCN is doing. 
-	Experiment can be improved:
		- For instance, the bottom-up method [9] has reported results on crowdpose dataset outperforming all methods in Table 4 with a ResNet-50 (including the paper one). It will be nice to include it in the tables
		- It will be nice to evaluate the performance of their method on the standard MS coco dataset to see if there is a drop in performance in easy (non occluded) settings. 
		- No study of inference time. Since this is a pose estimation method that is direct and does not require detection or keypoint grouping, it is worth to compare its inference speed to previous top-down and bottom-up pose estimation method.
		- Can we visualize G, the dynamic graph, as it changes through DGCN? It might give an insight on what the network used to predict keypoints, especially the invisible ones.

[Minor comments]
- 	In Algorithm 1 line 8 in Suppl Material, did the authors mean Eq 11 instead of Eq.4?
-	Fig1 and Fig2 in supplementary are the same
-	Spelling Mistake line 93: It it requires…
-	What does ‘… updated as model parameters’ mean in line 176
-	Do the authors mean Equation 7 in line 212?

limitations_and_societal_impact: The authors have talked about limitations in Section 5 and have mentioned that there are not negative societal impacts.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary:  This paper presents a one-stage pose estimation approach that applies PPG to infer multiple poses from multiple body parts (head, upper body, and lower body). These coarse poses are then refined by a GCN module. Finally, the Pose Fusion module is applied to get the final result.

main_review: Pros:
- The overall idea of the paper is easy to follow.
- The paper reads smooth.
 - The proposed method achieves the state-of-the-art performance on two recently proposed datasets, i.e. crowdpose and ochuman datasets. 
- The ablation study clearly shows the improvement of different modules.

Cons:
- The novelty of this paper is limited. Using graph networks to refine the initial pose is not new. [Ref1] construct dynamic graphs to tolerate large variations in the human pose. And [21] propose an Image-Guided Progressive GCN to estimate the invisible joints to deal with the occlusions. [Ref2] propose a two-stage graph-based method that adopts a graph pose refinement module to model the human structure. Line58 “this is an original contribution on the Direct Pose-Level Inference (DPLI) for pose estimation in crowded scenarios, which differs from the previous top-down and bottom-up pipelines.” is over-claimed. As the GCN is simply applied to each initial pose for pose refinement, there is no much difference, and cannot be considered novel.
[Ref1]  Z. Qiu, K. Qiu, J. Fu, and D. Fu, “Dgcn: Dynamic graph convolutional network for efficient multi-person pose estimation,” National Conference on Artificial Intelligence, 2020. 
[Ref2] J. Wang, X. Long, Y. Gao, E. Ding, and S. Wen, “Graph-pcnn: Two stage human pose estimation with graph pose refinement,” In European Conference on Computer Vision. Springer, 2020. 
- As stated above, some related works about GCN are missing [Ref1, Ref2]. Line86-87, detailed comparisons with previous works [15,32,14,27] are required. As most of them follow a similar one-stage pose estimation paradigm. 
-The proposed approach does not achieve the state-of-the-art performance on COCO (as reported in supplementary, mAP 67.2 this paper vs 68.0 DEKR). COCO and MPII are the most popular and standard benchmarks for evaluating human pose estimation, while OCHuman is a recently proposed dataset and there are only a few officially reported results. The readers will more care about the results on COCO and MPII. 
-The limitations are not adequately stated. Especially, why the performance on COCO is lower than other approaches. 
-If I understand correctly, the Pose Fusion module is a post-processing algorithm, which makes the whole pipeline not fully end-to-end trainable. The matching/fusion loss (or the loss of the final output pose) cannot be back-propagated to the backbone network.
-Please also report the runtime performance (inference speed).

Minor:
-Why do you highlight AP^H in red in Table4? 
limitations_and_societal_impact: The limitations are not adequately stated. Especially, why the performance on COCO is lower than other approaches.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method for robustly estimating human poses in crowded scenes. The main idea is to first detect visible body parts, then estimate human poses from the body parts. Based on these estimated human poses, refinement and fusion are performed. Experiments on OCHuman and CrowdPose show the effectiveness of the proposed method.
main_review: Strengths:
- The paper is very well-written and the figures are carefully designed to clearly present the idea. It is a pleasure to read the paper.
- The idea is novel: it starts from part detection instead of purely top-down or bottom-up like most previous methods.
- One of the modules (pose fusion) in the proposed method is general and can be applied to other pose estimation frameworks. 
- Experiments are well designed and the results are promising.

Overall I think the paper is interesting to the audiences in NeurIPS. Some comments rather than weaknesses:
- It would be very interesting to know how accurate are the poses predicted from different parts. For example, how accurate are the poses predicted from the head alone?
- How are the different parts contributing to the final results? Ablation studies can be done to see what the results are if some parts are ablated. 
- Would it be better if different pose refinement networks are learned for different parts? It might be useful, but it is also possible that the effect is very limited.
limitations_and_societal_impact: It doesn't seem to me that there's a potential negative societal impact of this work.
review_title: null

Meta-review: Initially, two of the reviewers expressed concerns about the paper (lack of clarity and limited novelty) and ranked the paper marginally below acceptance. As the ensuing rebuttal managed to successfully address most of reviewer’s concerns, ACs and the majority of the reviewers agreed that this is a strong paper that deserves acceptance. Authors are highly encouraged to address the key comments reported by reviewers as well as to implement all the improvements (as indicated by authors in the rebuttal) in the final camera-ready version.
Decision: Accept (Poster)