Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed an end-to-end multi-person pose regression framework to address the problem of high complexity and redundant post-processes existing in the previous work. To achieve this, the authors introduced the Spatial Part Embedding Generation Module (SPEGM) and the Selective Iteration Module (SIM) to encode human instances by a set of learnable part-level queries together with instance-level queries. Different from previous works, the part-level queries can capture the spatial details and structural information of human poses. Experimental results show that the proposed method achieved the state-of-the-art performance on MS COCO and CrowdPose datasets.
strengths_and_weaknesses: __Strengths__

1. This paper is overall well-structured and easy-to-follow.

2. I think the problem this paper tackled is important for the community. Multi-person pose estimation is definitely an important problem as it is prohibitively expensive to collect ground truth with great human efforts for in-the-wild images.

3. The proposed method is technically sound and novel.

4. The results show that the performance of the proposed method is strong and even on par with two-stage methods.

__Weaknesses__

1. The authors suggest in the abstract and introduction of the paper that one of the main benefits of the proposed method is to reduce the complexity of the inference. However, there is no evidence to support this point. Detailed comparisons of runtime performance such as FLOPs and inference time should be reported.

2. The box detection pipeline can be replaced by a DETR-like architecture using object queries to further reduce the complexity of the model. I wonder if the authors have considered this design. If so, what’s the performance and why the current FPN-like architecture is preferred?

3. It is interesting to see in the supplementary that the proposed method outperforms two-stage models on CrowdPose by a quite notable margin. I’d like to see a more detailed justification/discussion on the results.

4. Some comments to improve the clarity of the paper: 1. In Figure 2, it is unclear to see how the results from the previous stage are used as the input of the current stage. 2. The results on CrowdPose are important and should be put into the main paper.

questions: See "Weaknesses".
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a method for multi-person pose estimation that builds off Sparse R-CNN with additional functionality to achieve accurate pose estimates. In Sparse R-CNN, there are reference boxes/queries that are iteratively updated to converge on a final set of detections. In this work, there are additionally query embeddings (Q_p) that are decoded to produce keypoint estimates. The key contributions of this work are the “Spatial Part Embedding Generation Module” (SPEGM) and the “Selective Iteration Module” (SIM) which are used to update Q_p.

- SPEGM: Given a bounding box for a detected person, produce ROI aligned features from the box and use convolutions to predict a set of attention maps (each map corresponding to different body parts). Use these attention maps to perform a weighted pooling of the ROI aligned features to yield a set of embeddings E_p.
- SIM: Given the current embeddings, predict weights to perform a weighted sum of Q_p and E_p providing a new Q_p which is then decoded for the current pose estimate.

This process is repeated multiple times as the bounding boxes and part queries are iteratively improved. When benchmarking on COCO this method outperforms prior end-to-end methods.


strengths_and_weaknesses: Strengths:

- There are a number of different pieces that come together in the proposed method, and the authors provide ablations showing that they all play a role (using part queries rather than predicting keypoints directly from the instance-level features, the SPEGM, the weighted sum for the SIM)
- The results are strong, outperforming prior work on very competitive benchmarks notably without using test-time augmentation (TTA)

Weaknesses:

- This method seems as though it must be fairly expensive, the authors report splitting a batchsize of 16 across 8 A100s, it would be helpful to see comparisons in terms of memory use and training and inference time compared to other methods.
- While many ablations are provided to show the benefits of the proposed approach, it is worth pointing out that some simpler baselines are close in their performance (e.g. a simple summation instead of the SIM (Table 2a), or a dynamic MLP instead of the SPEGM (Table 1)). I don’t know if the most compelling case has been made yet that the full pipeline in all its complexity is necessary. It feels as though a simpler set of layers and update rule could replace the SPEGM and SIM achieving the same level of performance. This criticism is perhaps a bit unfair, as the authors do clearly show that performance would be slightly worse with the simpler alternatives.
- The communication and organization of the paper could be improved. The method itself is interesting, but it can be challenging to work through some of the unwieldy names and acronyms. Also, I found the overview figure (Fig. 2) difficult to make sense of.

Overall:

Altogether I think the method is interesting enough and the benchmark results speak for themselves to lean towards acceptance, but the quality and organization of writing could be improved to better communicate the proposed ideas and justify the full pipeline.

Post rebuttal: Having read the other reviews and author response I lean towards accepting and raise my score.
questions: - For the SIM, instead of the proposed weighted sum were any standard recurrent updates considered (e.g. a GRU update)?
limitations: no discussion included by authors
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a query-based human pose estimation approach. It builds upon Sparse-RCNN and design Spatial Part Embedding Generation Module (SPEGM) and Selective Iteration Module (SIM) to further improve the performance. Experiments on COCO and CrowdPose validates the effectiveness of the proposed method.

strengths_and_weaknesses: Pros:
1.	The proposed method is interesting. It adopts the query-based detection framework for one-stage pose estimation and designs heads and interaction between instance query and pose query. 
2.	It achieves the state-of-the-art performance among the one-stage approaches.

Cons:
1.	Two-stage vs one-stage, computational complexity
a)	From my point of view, the proposed method is not a typical “one-stage” approach. One major drawback of the two-stage top-down approaches is that their computational cost increases with the number of people (This is also pointed out by the authors in Introduction). But the proposed method (QueryPose) also suffers from such problem. It first locates the bounding box and then estimates the human poses for each bounding box. Isn’t the whole pipeline similar to Mask-RCNN? So I think it is more like a top-down approach, and its performance should be compared with other top-down approaches.
b)	The total computational complexity and runtime performance are concerning, but not reported in the paper. 
2.	The paper writing can be improved. Especially, Figure2 is not clear. 

questions: See above.
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new method for multiperson-pose estimation in an end-to-end learnable scheme. It brings together ideas from Mask-RCNN/Roi-Align and Transformers/Attention in a cascade scheme in which the box and keypoint predictions are refined in several stages. The paper contains ablation studies which examine the effectiveness of the different modules and reports strong results on the COCO keypoints and CrowdPose pose estimation benchmarks.
strengths_and_weaknesses: Strengths:

+ Good experimental results on well-known competitive benchmarks.
+ Although the method is complex, the paper does a good job explaining what each of the several modules does. Figure 2 is particularly helpful.
+ Authors promise to share code. This is very helpful because the method is quite complex and it would be very hard to reproduce the reported results using the description provided in the paper.

Weaknesses:
- The proposed method is quite complex. There are several modules involved.
- No runtime information is provided. It is hard to tell how fast/slow the method is. Both training and inference times need to be provided in the updated manuscript.

Some minor typos:

Typo in Fig 2(e), Normalizing
Line 166: Multilayer Perceptron
questions: I think the key missing information is how fast/slow the proposed method is during training and inference times.

The proposed system is quite complex and hard to reproduce given the information provided in the paper or supplementary material. However sharing the code as the authors intend to do based on the abstract will be sufficient to address this point.
limitations: Not very much applicable to this paper.
review_title: null

Meta-review: metareview: The authors propose a novel framework for end-to-end multi-person pose estimation by employing a set of learnable part-level queries along with instance-level queries. Promising results are demonstrated on the challenging COCO and CrowdPose datasets. The provided author rebuttal successfully addressed all reviewer concerns. As a result, all four reviewers recommend accepting the papers. The AC has read the paper, reviewer comments, author rebuttal, and all the discussions. The AC agrees with the reviewer recommendations. The authors are encouraged to include the rebuttal results (e.g., runtime analysis) to their camera-ready.
confidence: Certain
Decision: Accept