QueryPose: Sparse Multi-Person Pose Regression via Spatial-Aware Part-Level Query
Authors: Yabo Xiao
Kai Su
Xiaojuan Wang
Dongdong Yu
Lei Jin
Mingshu He
Zehuan Yuan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Sparse
End-to-end
Spatial-aware part-level query.

Abstract: We propose a sparse end-to-end multi-person pose regression framework, termed QueryPose, which can directly predict multi-person keypoint sequences from the input image. The existing end-to-end methods rely on dense representations to preserve the spatial detail and structure for precise keypoint localization. However, the dense paradigm introduces complex and redundant post-processes during inference. In our framework, each human instance is encoded by several learnable spatial-aware part-level queries associated with an instance-level query. First, we propose the Spatial Part Embedding Generation Module (SPEGM) that considers the local spatial attention mechanism to generate several spatial-sensitive part embeddings, which contain spatial details and structural information for enhancing the part-level queries. Second, we introduce the Selective Iteration Module (SIM) to adaptively update the sparse part-level queries via the generated spatial-sensitive part embeddings stage-by-stage. Based on the two proposed modules, the part-level queries are able to fully encode the spatial details and structural information for precise keypoint regression. With the bipartite matching, QueryPose avoids the hand-designed post-processes. Without bells and whistles, QueryPose surpasses the existing dense end-to-end methods with 73.6 AP on MS COCO mini-val set and 72.7 AP on CrowdPose test set. Code is available at https://github.com/buptxyb666/QueryPose.

QueryPose: Sparse Multi-Person Pose Regression via Spatial-Aware Part-Level Query
Yabo Xiao
Kai Su ByteDance Inc. Shanghai, China
Xioajuan Wang" BUPT Beijing, China wj27 112718@butt.eer edu.
BUPT
Beijing, China xiaoyabo@bupt.com edu.
sukai wwwaeiaeneeedcccoom
Dongdong Yu OPPO Research Institute
Lei Jin BUPT
Beijing, China
Beijing, China j jinlei@bupt.ed edu. cn
yudongdongdoppo.com com
Mingshu He BUPT
Zehuan Yuan
ByteDance Inc. Beijing, China yuanzehuan@bytedanee.. com
Beijing, China hemingshu@bupt. edu. en
Abstract
We propose a sparse end-to-end multi-person pose regression framework, termed QueryPose, which can directly predict multi-person keypoint sequences from the input image. The existing end-to-end methods rely on dense representations to preserve the spatial detail and structure for precise keypoint localization. However, the dense paradigm introduces complex and redundant post-processes during in- ference. In our framework, each human instance is encoded by several learnable spatial-aware part-level queries associated with an instance-level query. First, we propose the Spatial Part Embedding Generation Module (SPEGM) that considers the local spatial attention mechanism to generate several spatial-sensite part embeddings. which contain spatial details and structural information for enhancing the part-level queries. Second, we introduce the Selective Iteration Module (SIM) to adaptively update the sparse part-level queries via the generated spatial- -enssivvee part embeddings stage-by-stage. Based on the two proposed modules, the pert-level queries are able to fully encode the spatial details and structural information for precise keypoint regression. With the bipartite matching, Query QueryPose avoids the hand-designed post- post-processes and surpasses the existing dense end-to-end methods with 73.6 AP on MS COCO mini-val set and 72.7 7 AP on CrowdPose test set. Code is available at https://github.e http:/hbwb...ooooooo/oooooee//se
Introduction
Multi- Multi-person pose estimation (MPPE) aims to locate all person keypoints from the input image, which is fundamental yet challenging task in computer vision. With the prevalence of deep learning techniques 2 3. MPPE has achieved remarkable progress and played important role in many other vision tasks, such as activity recognition
and pose tracking g [9.
"Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

quary
Roypoitt
NMSS
Grouping
mult mult--pessnn keypolnts
mull-pprsoon
mulli-panson
Spatisl-naree
kaypoint
keypaints
pen ploel
keypalints
heatmap
healmap
offert
[8) battom-up
(b) pixel- regressiion
(c) QueryPase
Dense End EndEnd MPPE
Sparse End-to-End MPPE
Figure I: Comparison with the dense end-to-end solutions, including (a) bottom-up and (b) pixel-wise regression paradigms. (c) QueryPose is a sparse end-to-end solution.
In general, the existing multi-person pose estimation solutions can be summarized top-down and bottom-up paradigms. The top-down strategy [10 12 [S] 14 [3] uses the human detector to predict person boxes and then leverages the single-person pose estimation model to localize the keypoints on each cropped human image. The two independent models lead to the non-end-to-en pipeline, called two-stage pipeline. Moreover, the human detector involves
memory as 19 uses the keypoint heatmap
well as computational cost. The bottom-up strategy [6 to locate all person keypoints first and then assigns them individuals via heuristic grouping process, as shown in Figure aa). Some researches attempt to leverage densely pixel-wise regression 20 21 23 24 (5). as shown in Figure bb), which predicts the center heatmap and pixel-wise keypoint offset in parallel. It can be regarded as a special case of the bottom-up paradigm. Both of them can achieve end-to-end optimization thus more efficient than the two-stage pipeline. However, in order to maintain the local details and spatial information, ones leverage the dense representations (e.g., keypoint or center heatmap). The dense manner requires the hand-crafted post-processes to suppress duplicate predictions or perform keypoint grouping during inference. The post-pry post-processes are non-differentiable and always involve many hand-designed parameters to tune. In this paper, we aim to build a sparse end-to-end MPPE solution to eliminate the complex and redundant post-processes.
Recently, query-based paradigm 26 27 28
30 has attracted much attention due to its purely
are em- ployed for several computer vision tasks and achieve promising performance. The success inspires us construct sparse end-to-end MPPE method via the learnable query. Nevertheless. how to effectively apply the query-based paradigm to the MPPE task is rarely explored. Intuitively, keypoints are similar to the corner points of the bounding box, and the instance-level object query can be used to regress the keypoint coordinates. However, we observe that it only achieves inferior performance. We argue that the instance-level query loses the local details and destroys the spatial structure, which are both critical for pose estimation task.
sparse and end-to-end differentiable training/inference pipeline. The variants (30] 31 29
Towards the aforementioned issue, we introduce the learnable part-level queries learn spatial- aware features and further construct a sparse end-to-end multi-person pose regression framework, termed QueryPose. First, divide the human region into several local parts and present the Spatial Part Embedding Generation Module (SPEGM), which uses the local spatial attention mechanism to generate the spatial-sensitive part embeddings for enhancing part-level queries. Second, we introduce the Selective Iteration Module (SIM) to adaptively update the part-level queries via the generated spatial-sensitin part embeddings stage-by-stage. The local details and structural features in the learnable part-level queries will be enhanced, and the noise will be filtered. Based on the bipartite matching, we eliminate all hand-crafted post-processes and directly output the multi multi-person keypoint coordinate sequences for the input image. Without bells and whistles, our sparse framework outperforms dense end-to-end methods on MS COCO 32 and CrowdPose 33
The main contributions can be summarized into three aspects as follows:
We propose to leverage the sparse learnable spatial-aware part queries to encode local spatial features. Specifically, we present the Spatial Part Embedding Generation Module (SPEGM) to generate the spatial-sensitive part embeddings via the local spatial attention mechanism, which preserves the local spatial features for enhancing the part-level queries.

---

We further introduce the Selective Iteration Module (SIM) to adaptively update the learnable part-level queries by using the newly generated spatial sensitive part embedding stage-by' stage. Accordingly, the encoded local details and spatial structure are strengthened, and the distractive information is ignored.
With two proposed modules and one-to-one bipartite matching, we construct purely sparse query-based MPPE framework, termed QueryPose, which removes the complex post- processes and directly outputs the multi-person keypoint coordinates. QueryPose achieves state-of-the-art performance and surpasses the most existing end-to-end MPPE methods on both MS COCO and CrowdPose.
2 Related Work
In this section, we review three relevant parts to our method including top-down paradigm, bottom-up paradigm and query-based paradigm.
Top-down paradigm. The top-down paradigm conducts single-person pose estimation on each cropped human image. Most top-down methods [13
12 10 g concentrate on learning the reliable
high-quality feature representations for predicting keypoint heatmap. few of researches [34] 35 try to improve the post-process and some others [36 37 attempt to exploit regression methods to bypass the keypoint heatmap. However, top-down methods consist of the independent human detector and Bongle-pesoon pose estimation model, leading to the non-end-to-en optimization pipeline.
Bottom-up paradigm. The bottom-up cuptom-up paradigm 38 18 39
16
40) formulates this task as per-pixel
keypoint positioning and grouping process. Moreover, densely pixel-wise regression (20] 21 22 23 24 can be considered special case of the bottom-up paradigm. One decomposes the MPPE into wixel-wise center localization and keypoint offset regression. However, both of them leverage the dense representations, thus introducing the non-differentiable post-processes during inference.
Query-based paradigm. The query-based paradigm 227
28
26
29
31 30 [4] uses the learnable
object query and one-to-one label assignment achieve the purely sparse pipeline, which is applied
to several vision tasks (e.g., object detection [27] 28 (26] and segmentation 29 31 30). However,
rare research employs the query-based paradigm for MPPE in end-to-end manner. PRTR [1 trivially
uses the keypoint query with multi head self afffaatention to probe the image feature, while losing the
spatial local structural information, resulting in inferior performance. In our framework, we propose to leverage several spatial-awars part-level queries to learn local details and spatial structures.
3 Method
3.1
Overall Architecture
The overall framework is shown in Figure The network is divided into three parts, including backbone, box decoder and keypoint decoder. The keypoint decoder consists of the proposed Spatial Part Embedding Generation Module and Selective Information Module. The box decoder and keypoint decoder are repeated S times to build the cascade framework. S is set to 6 by default, We only take stage as example for illustration.
following previous methods 27 28
Backbone, Given an input image 7, we extract the multiple level features P2 ~ P3 via the feature pyramid network built upon the backhone, where P; indicates the feature with 1 1 resolution of input and 256 channels. We feed multi-level features P2 ~ P3 to box decoder and only high-resolution Pz to keypoint decoder.
Box decoder. The design follows Sparse R-CNN 28] we adopt small set of learnable proposal boxes B E Nx4 as dynamic anchors to provide the prior information of object position. Each learnable instance-level query. Concretely, we utilize the 7x7 RolAlign
proposal box corresponds operation to extract features ROI box according to the learnable boxes B. The extract features are one-to-one correspondence with learnable instance-level queries Qr RNxxd After exploring the self-attention across Q, by multi-head self-attention (MHSA) layer, we aggregate the extracted feature ROT boz into instance-level query via the dynamic channel MI.P (DyMLP channel Eq. [T]. and feed the enhanced instance-level query into regression head Headpox in Eq. to predict the
3

---

SPEGM
Soatial send part enteddings
SIM
ROI
Selrcc Heerston
MHSA
Ep
A
Qu
Keypoint decoder
Spatial Attension
Part level Query
A
Ep
Refined
ROI
Seleectow Iteratinn
MYSA
FPN
Box
RO! pass
Box ROWeR
ROL
box
cts
Narmalzing Flow
it
decoder
Backoone
proposel bones
Dyname VIP
xx
Repeat
Pa ROI
(a) BackBone
Instence-levee Query
(b) Box decoder
(e) RLE
times
Figure Overview of QueryPose. (a) Backbone for extracting multi-level features. (b) Box
decoder. (c) Spatial Part Embedding Generation Module (SPEGM) employs the local spatial attention mechanism produce spatial-sensitive part embeddings Ep. (d) Selective Iteration Module (SIM) adaptively updates the learnable part-level queries Qr by using the spatial-sensitive part embeddings. (e) The apatial-aware part-level quires are used to regress the keypoint coordinates via residual log- lngllikelhhood estimation(RI.F) (37]. The box decoder is following Sparse RCNN. The keypoint decoder composed of the proposed SPEGM and SIM.
offset for refining the input proposal box. The above process is summarized as follows:
RO1 box RoLAlign P2 Po. Ba-1), Qf MHSA Q7-
Qi DyMLP channel Q7 ROI boan, B A (Headbox ( (Q)), B²-1).
Keypoint decoder. After the above process, the instance-level query Qr has encoded the information of the human instance, thus can be used to regress the keypoints. However, it only obtains unsatisfac- tory performance. We argue that the keypoint localization requires more inherent spatial structure and local details than box regression, while Qı loses both spatial structure and local details.
In the light of this, we present to utilize several learnable spatial-aware part-level queries Qp encode the human pose with spatial structure and local details maintained. Qr (Qr, 12r,, TR. TR 1 In=1- where n refers to n-th instance and M is the number of part-level query for each instance. First, by using the refined boxes, we perform the 14x 14 RolAlign operation only on high-resolution feature P2 to extract the more detailed features ROI pose. which more suitable for pose estimation. Second, the proposed Spatial Part Embedding Generation Module (SPEGM) applied for producing the spatial-sensitive part embeddings Ep = {E,, Ep, R Xd, J.N mechanism. Each spasial-sensitive part embedding focuses on the specific local part. Thus the local spatial details and structural information
J==1 via the local spatial attention
be preserved. Third, we utilize the Selective Iteration Module (SIM) to adaptively update the learnable spatial-aware pert-level queries via the spatial-sensitive part embeddings generated in the current stage. Consequently, the local details and spatial structure information of part-level queries are enhanced, and the noise is filtered. Then, the spatial-aware part queries sent into the multi-head self-attention (denoted MHSA) layer to explore the spatial relationships across the different local parts. Accordingly, the global structure of each person instance is also encoded into the part-level queries. Finally, we leverage the non-shared linear layers (denoted as Linear in Eq. ( to regress the keypoint coordinates respectively. The computing procedure of keypoint decoder be formulated follows:
ROI 8 pose RolAlign B*). Eş SPEGM ROII pose
(2)
e SIM EP), er MHSA Pose Linear(@$)..
As shown in Figure 3 b), the instance-level query Qr is iterated across the box decoder and keypoint decoder serially, and the part-level queries Op are iterated across keypoint decoder. The box decoder

---

Q6
Deconeer
Keypoint
Keyporn Decoder
Kaypont Dacader
Kepport Decoder
Q5
Q!
Q
B
Qf
Box
8
B" Qi
Decoder
Qf
Decodder 5
B
Decocer' $
8
Decoder
8'
(a)
Figure 3: The iteration pipeline of the instance-level query Q₁ and part-level queries Qp (a) The instance-level query Q1 is only iterated across the box decoder. (b) The instance-level query Q1 is iterated across the box decoder and keypoint decoder serially.
of next stage adopts the predicted box of current stage as proposal box. The keypoint coordinates are predicted independently for each stage.
3.2 Spatial Part Embedding Generation Module
Considering that the instance-level query Qr loses the local details and spatial structure information compared with dense representations, we propose the Spatial Part Embedding Generation Module to generate the spatial-sensitive part embeddings Ep, which are able to maintain more spatial local features conducive to keypoint localization. The spatial-sensitive part embeddings E, are used to enhance the learnable spatial-aware part-level queries.
Given the single ROI feature ROI E dx as example, We firstly use the stacked 3 convolutional layer and a deconvolutional layer to recover the higher resolution representation ROI pose E pdx22tx2 x2W Afterward, we take the spatial attention into considcration on the extracted ROI features ROT pase. In particular, we utilize a 3x3 convolutional layer to squeeze the features along the channel dimension and produce the M Mcchannels attention map A E IR 2 2 Each channel corresponds a spatial attention map for specific local parts, M refers to the number of local parts. We flatten the attention map A along the spatial dimension and normalize the sum of attention weight 1.0 via the softmax function. Then, by manipulating the spatial attention maps of the specific local parts, we assemble the spatial feature to generate M spatial-sensitive part embeddings. The process is formulated as: Ep Linear(A @ ROIT pose) IR Mxd, where & indicates dot product. The apatial-sensttive part embeddings focus on the different local parts, shown in Figure 5) Thus the local details and structural information are sufficiently preserved compared with only single instance-level query.
3.3 Selective Iteration Module
For enabling the learnable part-level queries 2p learn the spatial-aware features, we propose the Spatial Part Embedding Generation Module (SPEGM) to generate several spatial-sensitive part embeddings Ep. However, we consider that due to the inaccurate bounding boxes and inconsistent optimization goals in the early stage, the spatial-sensitive part embeddings contain disturbance information. Motivated by this, we present the Selective Iteration Module to adaptively update and refine the part-level queries Qp via the spatial-sensitive part embeddings Ep.
First, we directly sum the learnable pert-level queries 0 Mxd, output from previous stage and the spatial-sensitive part embeddings Eア, E Mx4, generated in current stage. Second, we utilize a Multi-Layer Perceptron together with sigmoid operation to output two weight vectors W and W which serve as two gates to control the contributions of the Ep, and O' Finally, We leverage the weighted summation fuse the Ej. and e-1 via WE, and W The above process formulated as:
W W = Sigmoid (MLP + 1), Φ = W * E
W eF-
(3)
By using the Selective Iteration Module, the informative local details and spatial structure part-level queries Qp will be refined, and the noise will be ignored.
5

---

3.4 Training and Testing Details
Training. In our framework, we leverage the instance-level query Qr to conduct the binary clas- sification and bounding box regression, then use several spatial-aware part-level queries Op to regress corresponding keypoints. The ground truth of each human instance contains the class label, box corners and keypoints coordinates, which are denoted as C. [(x -y("²), (x200 _y2=²), )), and
[( -1k } respectively. Following previous methods [27 28 we adopt bipartite matching to perform one-to-one label assignment. The set-based loss function for instance-wise query is formulated Lor Acts Lelss ÀLI LL1 + Agion Lyiou. where Lets refers to focal loss to perform binary classfication, LL1 and Lgions denote the LI loss and generalized IoU loss between the predicted box and the corresponding label, the loss weight Acts ALI, Agiou are set 2.0, 5.0 and 2.0 respectively,
For keypoint regression, we leverage malizing flow to capture the latent keypoint distribution Pa.a [x] pose), and model the keypoint regression loss from the per- spective of maximum likelihood estimation following RLE (37] The density distribu- tion Pe. (x ROI pose) reflects the probabil- ity of the keypoint annotation at the position ROI pose, where and are the train- able parameters of our regression network and flow model respectively. For easier op- timization, the flow model Fs. is used to map the simple distribution deformed one P (8) Our regression network outputs two values ğ and ē for shifting and rescaling the
Table I: Ablation studies for exploring the query type and feature interaction method. IQ and PQ indicate the instance-level query and part-level query respectively.
Query
Interaction
AP
APM
APr
1Q
MHSA
60.7
56.4
68.1
PQ
MHSA
62.6
58.2
69.9
IQ
DyMLP channel
60.4
56.3
67.4
PQ
DyMLI P channel
63.0
58.7
70.4
1Q
DyMLP spotial
60.5
56.5
67.5
PQ
DyMLP qpatial
62.8
58.6
70.3
IQ
Spatial Attn.
60.2
56.0
67.1
PQ
SPEGM
63.8
59.4
71.3
distribution P Pa (X), which is formulated as how the output deviates from annotations.
X + ē + f.. The process enables the network to learn
The different spatial-aware part queries are sent into the non-shared fully-connected layers to regress f I and & E X22 where is the corresponding prediction of Mgg {(* .V& on the matched pairs, the set-based keypoint regression loss for the spatial- part-level queries formulated as:
ky kp K ) Based
Lor log Po. (x) ROT pose))xxx4
(4)
It is noteworthy that M4 is normalized by the height and width of the predicted bounding box and shifted to (-0.5, 0.5) for stabilizing the training process. The keypoint out of the predicted bounding box will be masked. The total loss of our framework can be written as: Lioial LQ: + LQr-
Testing, â can be regarded as the uncertainty score of keypoint localization, thus we define the final pose score as & Σx_ (1 - ōx) = C, where C' the predicted instance score. During inference, our post-procedures (e.g., keypoint NMS, center NMS and grouping)
method climinates and only need to select the pose candidates with high pose score.
4 Experiments and Analysis
In this section, we briefly illustrate the dataset, evaluation metric as well as implementation details in subsection 4.1] Next, delve into the proposed modules and conduct the comprehensive ablation experiments to verify the effectiveness in subsection 4.2) Finally, we compare QueryPose with the previous methods both MS COCO and CrowdPose in subsection 4.3
4.1 Experimental Setup
Dataset. MS COCO 32 is a large-scale 2D human pose benchmark, which is split into train2017,
mini-val, test-dev2017 7 sets. We train our model on COCO train2017 set with 57k images, conduct the evaluation the mini-val set with 5k images and test-dev2017 set with 20K images respectively. CrowdPose [K contains 20,000 images with 80,000 annotated human instances. The train, validation and test set partitioned in proportional to 5:1:4.

---

Table Ablation experiments.
(a) Ablative studics for the iteration methods learnable part-level queries.
(b) The contributions of two proposed modules in overall framework.
Iteration AP
APMs
APr.
Method
AP
APM
APr.
None
63.8
59.4
71.3
Bascline
59.2
55.1
66.0
Summation
64.2
60.1
71.7
SPEGM
63.8
59.4
71.3
Concatenation
64.1
59.6
71.6
SIM
64.9
60.4
72.3
Selective Iter.
64.9
60.4
72.3
(d) Ablative studies for the different part di-
(c) The influence for the dimension of spatial sensitive part-level query.
vision schemes. Scheme (a). (b), (c), (d)
corresponding to the Figure H (a). (b), (c), (d).
Dimension
AP
APM
APL
Scheme
Part
AP
APM
AP2
32
64.1
59.8
71.3
(a)
17
64.7
60.4
72.1
64
64.4
60.0
71.8
(b)
13
64.6
60.1
72.0
128
64.9
60.4
72.3
(c)
7
64.9
60.4
72.3
256
64.7
60.5
72.1
(d)
5
64.5
60.5
71.7
Evaluation Metric. We leverage average precision and average recall based on different Object
Keypoint Similarity (OKS) (32) thresholds to evaluate performance. For COCO dataset, AP and A indicate AP over medium and large-sized instances respectively. For CrowdPose, APE. APM. A refer to AP scores over easy, medium and hard instances according to dataset annotations.
Implementation Details. During training, we random horizontal flip and random crop with probability 0.5 to augment the training samples. The input images are randomly resized, and the short side is randomly sampled from 480 to 800 pixels with the aspect ratio kept. The number of the sparse instance-level query is set to 100 by default, it can even be reduced to 50 without significant performance degradation due to the foreground only containing the person. The model is trained by using Adam 43 optimizer with a mini-batch size of 16 (2 per GPU) on eight Tesla A100 GPUs. The initial learning rate is linearly scaled 2.5e-5 and dropped 10x after 220k, 260k iterations, and terminated at 280k iterations (2x training schedule). The learning rate of the flow model is 10x of the regression model. All codes are implemented based Detectron2 44 During inference, keep the aspect ratio and resize the short side of the images to 800 pixels.
4.2 Ablation Study
We conduct the ablative studies on MS COCO mini-val set to explore the contributions of the proposed components and delve into the superiority of its design. All experiments adopt ResNet-50-FPN and 1 x training scheme. We first define the baseline as employing the stacked 3x3 conv-relu global average pooling on ROT pose to directly regress keypoint coordinates.
Analysis of the part-level query. As shown in Table [ to verify the superiority of the part-level query over the instance-level query, we leverage four different interaction methods with image feature R01pose: (1) MHSA refers to multi-head self-attention layer. (2) DyMLP DyMLPchunnel channel indicates the dynamic MLP acting on the channel dimension, where the weight of MLP is generated by the refers to the dynamic MLP acting on the spatial dimension.
corresponding query. (3) DyMLP
(4) Spatial Att. denotes using global spatial attention generate the instance embedding to enhance the instance- level query. We observe that part-level query can consis- tently achieve the notable improvements than instance-level query in case of utilizing the different ways to interact with the image features ROI pose. This phenomenon proves that part-level query is able to preserve more details of local area than instance-level query.
Table 3:
Ablation studies for the itera- tion pipeline of the instance-level query.
Iter. pipeline
A
A Appps
pose.
Figure a)
55.3
64.1
Figure b)
56.9
64.9
Analysis of Spatial Part Embedding Generation Module. To study the superiority of the Spatial Part Embedding Generation Module (SPEGM). we compare the performance with different interaction methods in Table Notably, using the SPEGM to generate spatial-sensitive part embeddings for boosting the learnable part-level queries achieves the superior performance (63.8 AP) than the other interaction ways. Furthermore, SPEGM improves the baseline by 4.6 AP as reported Table 2(b) We suggest that the local spatial attention is able to provide the spatial prior information and focus on
7

---

Sizal
slouudls
hip
luy
shoulder
A111
1etl Icp
right lcg
Figure 5: The visualizations of the local spatial attention maps A, which correspond to the local parts of each person. The local salient region visualized by bright color. The backgrounds of person regions are rendered to red. Best viewed after zooming in.
the informative local regions, shown in Figure Furthermore, compared with dynamic MLP with over 10M parameters, SPEGM is more effective and efficient with only 1.1M parameters.
Analysis of Selective Iteration Module. Selective Iteration Module (SIM) is presented to adaptively update the learnable spatial- aware part queries output from the pre- vious stage. We conduct the controlled experiments to investigate the superior- ity of the proposed SIM compared with the other two iteration methods, includ- ing summation and concatenation. As reported in Table 2(a) the Selective Iter- ation Module achieves 64.9 AP and im- proves the non-iteration structure by 1.1 AP. Moreover, SIM outperforms the sum- mation and concatenation by 0.7 and 0.8 AP, respectively. We consider that the Se- lective Iteration Module can adaptively boost the informative spatial details and filter the noise.
(a)
(b)
Figure 4: The different part division schemes. (a) Each keypoint corresponding to a part-level query. (b) All keypoints on head are corresponding to a part-level query. (c) Each divided part contains rigid structure and corre- sponds to a part-level query. (d) The head and four limbs are corresponding to a part-level query respectively.
Analysis of the dimension for the part-level query. The part-level queries are used to encode the local spatial detail and structure. Based the overall network, explore the dimension of the spatial-aware part query. As reported in Table we observed that setting the dimension to 128 is sufficient, and the larger dimension can not bring additional gains.
Analysis of the part division scheme. As shown in Figure 4 and Table 2(d) we probe four different part division schemes to design an efficient and effective partition strategy. The scheme (c) achieves the optimal performance. Compared with the scheme (a) and (b), the part-level query in scheme (c) is capable of obtaining local structural information of adjacent keypoints. Moreover, each part in scheme (c) is rigid without articulated structure, while the part in scheme (d) is deformable, thus obtaining slightly worse result.
Iteration pipeline of the instance-level query. As shown Figure Figure Table we explore the two different iteration pipeline of the instance-level query Q,. We observe that the instance-level query iterated across box decoder and keypoint decoder serially can improve 1.6 Apbor and 0.8 Aphps than only iterated across box decoder.
4.3 Results
COCO Mini-val set. Table reports the results of end-to-end methods on COCO mini-val set. Compared with dense end-to-end methods, QueryPose with ResNet-50 outperforms Mask R-CNN

---

Table Comparisons with the end-to-end MPPE methods on the COCO mini-val set. Note that all results are reported for single-scale testing. Flip refers to using the flip testing to boost performance. Methods
Backbone
AP
AP30
AP
APır
APr.
Time [ms]
Dense End-to-Eni to------d MPPE
Mask R-CNN
45
ResNet101
66.1
87.7
71.7
60.5
75.0
128
HrHRNet-War
[6
HrHRNet-We W48
66.6
85.3
72.8
61.7
74.4
233
ArHRRNet-WAR
(Flip) 16
HrHRNet- W48
69.8
87.2
76.1
65.4
76.4
317
SWAHR- W48
HrHRNet W48
67.3
87.1
62.1
75.0
242
SWAHR-W 48 (Flip)
400
HrHRNet--V
70.8
88.5
76.8
66.3
77.4
339
CenterCroup- W48
[46
HrHRNet- W48
69.1
CenterNet-HG (Flip)
12
HG-104
64.0
135
Mask R-CNN RLE
37
ResNet101
66.7
86.7
72.6
DHKR-W48
24
HRNet-W44
67.1
87.7
73.9
61.5
77.1
197
DEKR-WAS -AR (Flip)
HRNet-W48
71.0
88.3
77.4
66.7
78.5
284
AdaptivePose
HRNet-WAi
70.0
87.5
76.1
65.4
77.1
110
LQR (Flip)
HRNet-W48
72.4
89.1
79.0
67.3
80.4
297
Sparse End-to-End MPPE
PRTR 14
W488
66.2
85.9
72.1
61.3
74.4
QueryPose
ResNetSO
68.7
88.6
74.4
63.8
76.5
QueryPose
Swin-L
73.3
91.3
79.5
68.5
81.2
117
QueryPuse
HRNet-W32
72.4
89.8
78.6
67.9
79.7
100
QueryPose
HRNet- W48
ERRNet-W48
73.6
90.3
79.7
69.3
80.8
105
Table 5: Comprehensive comparisons on COCO test-dev set. indicates the refinement by
well-trained single-person pose estimation model. j refers to test-time augmentation (c.g., flip or multi-scale testing).
Methods
AP
APso
APrs
APM
APz
Two-Stage Methods
G-RMI 17
64.9
85.5
71.3
62.3
70.0
Integral Pose
67.8
88.2
74.8
63.9
74.0
CPN'
100
72.1
91.4
80.0
68.7
77.2
SimpleBaseline
8
73.7
91.9
81.1
70.3
80.0
HRNet
75.5
92.5
83.3
71.9
81.5
End-to- End Methods
CMU-Pose
39
61.8
84.9
67.5
57.1
68.2
AE
19
65.5
86.8
72.3
60.6
72.6
CenterNet-HG
63.0
86.8
69.6
58.9
70.4
SPM 22
66.9
88.5
72.9
62.6
73.1
PointSetNet
42
68.7
89.9
76.3
64.8
75.3
HrHRNet
16
70.5
89.3
77.2
75.8
DEKR
24
71.0
89.2
78,0
67.1
76.9
CenterGroup
46
71.1
90.5
77.5
66.9
76.7
TQQR
41
71.7
90.4
78.7
67.3
78.5
AdaptivePose! 200
71.3
90.0
78.3
67.1
77.2
QueryPose- W48
72.3
91.5
78.7
67.8
79.0
QueryPose-Swinl
72.2
92.0
78.8
67.3
79.4
45 with ResNet-101 by 2.6 AP. By only using HRNet-W32, we achieve 72.4 AP and markedly outperform all dense counterparts with larger backbone. QueryPose with HRNet-W44 obtains 73.6 AP and significantly surpasses the previous most competitive HrHRNet-Way, (Flip) [66] SWAHR-WAS (Flip) 440]. CenterGroup W48 46) by 3.8 AP, 2.8 AP. 4.5 AP respectively. Furthermore, QueryPose outperforms the regression- based AdaptivePose -W4 W48 20 and DEKR W48(Flip) (24] by 3.6 AP and 2.6 AP. QueryPose obtains 7.4 AP improvements over the existing sparse end-to-end method PRTR 14.. Finally, we use the transformer-based backbone Swin-Large 47 and also achieve 73.3 AP, which proves that our method is compatible for various network architecture. QueryPose eliminates the time-consuming post-processe and achieves the faster inference speed compared with the above ofte-oof-the-rr dense competitors.

---

Table Comparisons between the previous two-stage and end-to-end MPPE methods on CrowdPose test set. indicates multi-scale testing.
Methods
AP
APso
APTs
APR
APM
APи
Two- Two-Stage MPPE
Rmpe 11
61.0
81.3
66.0
71.2
61.4
51.1
SimpleBaseline
8
60.8
84.2
71.3
71.4
61.2
51.2
CrowdPose 33
66.0
84.2
71.5
75.5
66.3
57.4
End-to Mod-to-EBn Bad-to--En MPPE
CMU-Pose
39
62.7
48.7
32.3
Mask-RCNN 45
57.2
83.5
60.3
69.4
57.9
45.8
SWAHR-WA 40
71.6
88.5
77.6
78.9
72.4
63.0
DEKR-WAR
24
68.0
85.5
73.4
76.6
68.8
58.4
AdaptivePose-War
W48 20
69.2
87.3
75.0
76.7
70.0
60.9
HigherHRNet- W48
16
67.6
87.4
72.6
75.8
68.1
58,9
CenterGroup-W45
46
70.0
88.9
75.1
76.8
70.7
62.2
QueryPose-W43
72.1
89.4
78.0
79.8
73.3
63.0
QueryPose-SwinL.
72.7
91.7
78.1
79.5
73.4
65.4
COCO test-dev2017 set. As shown in Table we achieve 72.3 AP with HRNet-W48 and 72.2 AP with Swin-Large without flip and multi-scale testing. QueryPose exceeds all end-to-end methods and narrows the gap with two-stage methods. The results verify the superior keypoint positioning ability of QueryPose in single-forwand pass.
CrowdPose test set. We further evaluate the QueryPose CrowdPose. The model is trained on the train and val set, then evaluated on test set as previous methods [16] 40) 24]. We list the
comparisons in Table QueryPose achieves 72.1 1.1 AP with HRNet-WWa and 72.7 AP with Swin-large on CrowdPose test set. The results outperform the most existing MPPE methods.
Discussion. We leverage the sparse spatial-aware part-level queries to encode the local detail and spatial structure instead dense feature representation. Our sparse method achieves the better performance, proving that dense heatmap is not all you need for pose estimation. In crowded scenes, the sparse paradigm probably avoids the issues caused by the NMS process (e.g., the heavily overlapped instances keypoints may be removed after center keypoint NMS). Accordingly, the sparse paradigm may be more compatible for crowded scenes.
Furthermore, we observe that the typical single-stage regression based methods (without ROI) require the online auxiliary keypoint heatmap learning during training stage, which can bring 1.5-2.5 AP improvements for different approaches (e.g., AdaptivePosse [0], PointSetNet (22). In contrast, The online auxiliary keypoint heatmap learning is useless for QueryPose, while the heatmap pretrain is more suitable for QueryPose. We only leverage HRNet to verify the phenomenons and bring the 1.5 AP improvements.
Conclusion
In this paper, we present a sparse end-to-end multi-person pose regression framework termed as QueryPose. With two proposed modules, i.e., Spatial Part Embedding Generation Module and Selective Iteration Module, QueryPose is capable of utilizing the sparse spatial-aware part-level queries to capture the local spatial features instead of previous dense representations and achieving the better performance than most dense MPPE methods. Our method proves the potential of the sparse regression method for the multi-person pose estimation task. We believe that our core insight able to benefit some other methods and inspire future research.
Acknowledgements
This work is supported by the National Natural Science Foundation of China No. 6..66000656 No.6187 1046 and No.62102039.
10

---

References
[1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[2] Alejandro Newell, Kaiyu Yang. and Jia Deng. Stacked hourglass networks for human pose estimation. In European conference computer vision, pages 483-4994 Springer. 2016.
[3] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. lece, 2009.
[4] Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Actional-structtn graph convolutional networks for skeleton- based action recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 3595-3603, 2019.
[5] Lei Shi, Yifan Zhang. Jian Cheng. and Hanqing Lu. Two-stream adaptive graph convolutional networks for skeleton- bason action recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 12026-1 12035, 2019.
[6] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton- based action recognition. In Thirty-second second AAAI conference artificial intelligence, 2018.
[7] Haodong Duan, Yue Zhao, Kai Chen, Dian Shao, Dahua Lin, and Bo Dai. Revisiting skeleton-based action recognition. arXiv preprint arXiv:2 13586, 2021.
[8] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision (ECCV), pages 466-481, 2018.
[9] Dongdong Yu, Kai Su, Sun, and Changhu Wang. Multi-person pose estimation for pose tracking with enhanced cascaded pyramid network. In European Conference on Computer Vision, 2018.
[10] Yilun Chen, Zhicheng Wang, Yuxiang Peng. Zhiqiang Zhang. Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 7103-7112 12, 2018.
[11] Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. Rmpe: Regional multi multi-person person pose estimation. In Proceedings of the IEEE International Conference on Computer Vision, pages 2334 2343, 2017.
[12] Kai Su. Dongdong Yu, Zhenqi Xu, Xin Geng, and Changhu Wang. Multi-person pose estimation with enhanced channel-wist wise and spatial information. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5674-5682, 2019.
[13] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. Proceedings s of the IEFF conference on computer vision and pattern recognition, pages 5693-5703, 2019.
[14] Li. Shijie Wang. Xiang Zhang, Yifan Xu. Weijian Xu, and Zhuowen Tu. Pose recognition with cascade transformers. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1944-1953, 2021.
[15] Yanjie Li. Shoukui Zhang, Zhicheng Wang, Sen Yang. Wankou Yang Shu-Tao Xia, and Erjin Zhou. Tokenpose: Learning keypoint tokens for human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11313 11122,11222000 2021.
[16] Bowen Cheng, Bin Xiao, Jingdong Wang. Honghui Shi, Thomas S Huang, and Lei Zhang, Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5386-5395, 2020.
[17] George Papandreou, Tyler Zhu, Nori Kanazawa, Alexander Toshey, Jonathan Tompson, Chris Bregler, and Kevin Murphy. Towards accurate multi-person pose estimation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4903-4911, 2017.
[18] George Papandreou, Tyler 7hu, Liang-Chice Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Murphy. Personlab: Person pose estimation and instance segmentation with a bottom-up, part-hased, geometric embedding model. In Proceedings of the European Conference Computer Vision (ECCV), pages 269-286, 2018.
[19] Alejandro Newell, Zhiao Huang, and Jia Deng. Associative embedding: End-to-end learning for joint detection and grouping. In Advances in neural information processing systems, pages 2277-2287, 2017.
11

---

[20] Yabo Xiao, Xiao Juan Wang. Dongdong Yu, Guoli Wang. Qian Zhang, and HE Mingshu. Adaptivepose: Human parts adaptive points. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[21] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv preprint 1rXiv:1904.078:5 2019.
[22] Xuecheng Nie, Jiashi Feng, Jianfeng Zhang, and Shuicheng Yan. Single-stage multi-persoo pose machines. In Proceedings of the IEEE International Conference on Computer Vision, pages 6951-6960, 2019.
[23] Zhi Tian, Hao Chen, and Chunhua Shen. Directpose: Direct end-to-end multi-person person pose estimation. arXiv preprint 191445111..7751, 2011. 2019.
[24] Zigang Geng, Ke Sun, Bin Xiao. Zhaoxiang Zhang, and Jingdong Wang. Bottom-up human pose estimation via disentangled keypoint regression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1666614686, 2021.
[25] Yabo Xiao, Xiaojuan Wang. Dongdong Yu, Kai Su, Lei Jin, Mei Song, Shuicheng Yan, and Jian Zhao Adaptivepose++:: A powerful single-stage network for multi-person pose regression. arXiv preprint arXiv:2210.04014,22 2022.
[26] Xizhou Zhu, Weijie Su, Lewei La, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint urXiv:2010.04159, 2020.
[27] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[28] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, al. Sparse r-cmm: End-to-end object detection with learnable proposals. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 14454-
14463, 2021.
[29] Bowen Cheng, Alex Schwing, and Alexander Kirillov, Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34, 2021.
[30] Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unitied image segmentation. Advances in Neural Information Processing Systems, 34, 2021.
[31] Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. 2022.
[32] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[33] Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang, and Cewu Lu. Crowdpose: Efficient crowded scenes pose estimation and a new benchmark. Proceedings of the IEEE/CVF cenffreeeenferencee on computer vision and puttern recognition, pages 10863-10872, 2019.
[34] Junjie Huang, Zheng Zhu, Feng Guo, and Guan Huang. The devil is in the details: Delving into unbiased data processing for human pose estimation. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 5700-5709, 2020.
[35] Feng Zhang, Xiatian Zhu, Hanbin Dai, Mao Ye, and Ce Zhu. Distribation -wware aware coordinate representation for human pose estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7093-7102, 2020.
[36] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In Proceedings of the European Conference Computer Vision (ECCV), pages 529-545, 2018.
[37] Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu, and Cewu Lu. Human pose regres- sion with residual log- likeliioooodd estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1 1025-110344 2021.
[38] Sven Kreiss, Lorenzo Bertoni, and Alexandre Alahi. Pifpaf: Composite fields for human pose estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11977 11986. 2019.
12

---

[39] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime person 2d pose estimation using part aflinity fields. In Proceedings the IEEE conference on computer vision and patiern recognition. pages 7291-7299, 2017.
[40] Zhengxiong Luo, Zhicheng Wang. Yan Huang, Liang Wang. Tieniu Tan, and Erjin Zhou. Rethinking the heatmap regression for hottom-up human pose estimation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13264-13273, 2021.
[41] Yabo Xiao, Dongdong Yu, Xiao Juan Wang, Lei Jin, Guoli Wang, and Qian Zhang. Learning quality-aware representation for multi-person pose regression. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[42] Fangyun Wei, Xiao Sun, Hongyang Li, Jingdong Wang, and Stephen Lin. Point-set anchors for object detection, instance segmentation and pose estimation. In European Conference on Computer Vision, pages 227-544. Springer, 2020.
[43] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018.
[44] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2. https 2019.
//github. www.////cccbbboeeeeeer///etteeee
[45] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[46] Guillem Brasó, Nikita Kister, and Laura Leal-Taixé. The center of attention: Center-keypoint grouping via attention mulli-person pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11853-11866 2021.
[47] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10020 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] The code and weights will be released. We provided the detailed instructions in paper to reproduce the results.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No] The reported results the average values after running twice.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
13

---

(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you' ro using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "ymKHlXYAZAg", "reviewer": "Reviewer_ve12", "summary": "The paper proposes a new method for multiperson-pose estimation in an end-to-end learnable scheme. It brings together ideas from Mask-RCNN/Roi-Align and Transformers/Attention in a cascade scheme in which the box and keypoint predictions are refined in several stages. The paper contains ablation studies which examine the effectiveness of the different modules and reports strong results on the COCO keypoints and CrowdPose pose estimation benchmarks.", "questions": "I think the key missing information is how fast/slow the proposed method is during training and inference times.\n\nThe proposed system is quite complex and hard to reproduce given the information provided in the paper or supplementary material. However sharing the code as the authors intend to do based on the abstract will be sufficient to address this point.", "limitations": "Not very much applicable to this paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ Good experimental results on well-known competitive benchmarks.\n+ Although the method is complex, the paper does a good job explaining what each of the several modules does. Figure 2 is particularly helpful.\n+ Authors promise to share code. This is very helpful because the method is quite complex and it would be very hard to reproduce the reported results using the description provided in the paper.\n\nWeaknesses:\n- The proposed method is quite complex. There are several modules involved.\n- No runtime information is provided. It is hard to tell how fast/slow the method is. Both training and inference times need to be provided in the updated manuscript.\n\nSome minor typos:\n\nTypo in Fig 2(e), Normalizing\nLine 166: Multilayer Perceptron", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9ljvollRits", "reviewer": "Reviewer_YZdd", "summary": "The paper presents a query-based human pose estimation approach. It builds upon Sparse-RCNN and design Spatial Part Embedding Generation Module (SPEGM) and Selective Iteration Module (SIM) to further improve the performance. Experiments on COCO and CrowdPose validates the effectiveness of the proposed method.\n", "questions": "See above.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1.\tThe proposed method is interesting. It adopts the query-based detection framework for one-stage pose estimation and designs heads and interaction between instance query and pose query. \n2.\tIt achieves the state-of-the-art performance among the one-stage approaches.\n\nCons:\n1.\tTwo-stage vs one-stage, computational complexity\na)\tFrom my point of view, the proposed method is not a typical “one-stage” approach. One major drawback of the two-stage top-down approaches is that their computational cost increases with the number of people (This is also pointed out by the authors in Introduction). But the proposed method (QueryPose) also suffers from such problem. It first locates the bounding box and then estimates the human poses for each bounding box. Isn’t the whole pipeline similar to Mask-RCNN? So I think it is more like a top-down approach, and its performance should be compared with other top-down approaches.\nb)\tThe total computational complexity and runtime performance are concerning, but not reported in the paper. \n2.\tThe paper writing can be improved. Especially, Figure2 is not clear. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "9Zm4EN68RJ-", "reviewer": "Reviewer_2Ux2", "summary": "The authors propose a method for multi-person pose estimation that builds off Sparse R-CNN with additional functionality to achieve accurate pose estimates. In Sparse R-CNN, there are reference boxes/queries that are iteratively updated to converge on a final set of detections. In this work, there are additionally query embeddings (Q_p) that are decoded to produce keypoint estimates. The key contributions of this work are the “Spatial Part Embedding Generation Module” (SPEGM) and the “Selective Iteration Module” (SIM) which are used to update Q_p.\n\n- SPEGM: Given a bounding box for a detected person, produce ROI aligned features from the box and use convolutions to predict a set of attention maps (each map corresponding to different body parts). Use these attention maps to perform a weighted pooling of the ROI aligned features to yield a set of embeddings E_p.\n- SIM: Given the current embeddings, predict weights to perform a weighted sum of Q_p and E_p providing a new Q_p which is then decoded for the current pose estimate.\n\nThis process is repeated multiple times as the bounding boxes and part queries are iteratively improved. When benchmarking on COCO this method outperforms prior end-to-end methods.\n\n", "questions": "- For the SIM, instead of the proposed weighted sum were any standard recurrent updates considered (e.g. a GRU update)?", "limitations": "no discussion included by authors", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- There are a number of different pieces that come together in the proposed method, and the authors provide ablations showing that they all play a role (using part queries rather than predicting keypoints directly from the instance-level features, the SPEGM, the weighted sum for the SIM)\n- The results are strong, outperforming prior work on very competitive benchmarks notably without using test-time augmentation (TTA)\n\nWeaknesses:\n\n- This method seems as though it must be fairly expensive, the authors report splitting a batchsize of 16 across 8 A100s, it would be helpful to see comparisons in terms of memory use and training and inference time compared to other methods.\n- While many ablations are provided to show the benefits of the proposed approach, it is worth pointing out that some simpler baselines are close in their performance (e.g. a simple summation instead of the SIM (Table 2a), or a dynamic MLP instead of the SPEGM (Table 1)). I don’t know if the most compelling case has been made yet that the full pipeline in all its complexity is necessary. It feels as though a simpler set of layers and update rule could replace the SPEGM and SIM achieving the same level of performance. This criticism is perhaps a bit unfair, as the authors do clearly show that performance would be slightly worse with the simpler alternatives.\n- The communication and organization of the paper could be improved. The method itself is interesting, but it can be challenging to work through some of the unwieldy names and acronyms. Also, I found the overview figure (Fig. 2) difficult to make sense of.\n\nOverall:\n\nAltogether I think the method is interesting enough and the benchmark results speak for themselves to lean towards acceptance, but the quality and organization of writing could be improved to better communicate the proposed ideas and justify the full pipeline.\n\nPost rebuttal: Having read the other reviews and author response I lean towards accepting and raise my score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8exCq9F5UO", "reviewer": "Reviewer_f8E1", "summary": "This paper proposed an end-to-end multi-person pose regression framework to address the problem of high complexity and redundant post-processes existing in the previous work. To achieve this, the authors introduced the Spatial Part Embedding Generation Module (SPEGM) and the Selective Iteration Module (SIM) to encode human instances by a set of learnable part-level queries together with instance-level queries. Different from previous works, the part-level queries can capture the spatial details and structural information of human poses. Experimental results show that the proposed method achieved the state-of-the-art performance on MS COCO and CrowdPose datasets.", "questions": "See \"Weaknesses\".", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__\n\n1. This paper is overall well-structured and easy-to-follow.\n\n2. I think the problem this paper tackled is important for the community. Multi-person pose estimation is definitely an important problem as it is prohibitively expensive to collect ground truth with great human efforts for in-the-wild images.\n\n3. The proposed method is technically sound and novel.\n\n4. The results show that the performance of the proposed method is strong and even on par with two-stage methods.\n\n__Weaknesses__\n\n1. The authors suggest in the abstract and introduction of the paper that one of the main benefits of the proposed method is to reduce the complexity of the inference. However, there is no evidence to support this point. Detailed comparisons of runtime performance such as FLOPs and inference time should be reported.\n\n2. The box detection pipeline can be replaced by a DETR-like architecture using object queries to further reduce the complexity of the model. I wonder if the authors have considered this design. If so, what’s the performance and why the current FPN-like architecture is preferred?\n\n3. It is interesting to see in the supplementary that the proposed method outperforms two-stage models on CrowdPose by a quite notable margin. I’d like to see a more detailed justification/discussion on the results.\n\n4. Some comments to improve the clarity of the paper: 1. In Figure 2, it is unclear to see how the results from the previous stage are used as the input of the current stage. 2. The results on CrowdPose are important and should be put into the main paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept