Referring Human Pose and Mask Estimation In the Wild
Authors: Bo Miao
Mingtao Feng
Zijie Wu
Mohammed Bennamoun
Yongsheng Gao
Ajmal Saeed Mian
Venue: NeurIPS 2024 poster (2024)
Keywords: Referring Human Pose and Mask Estimation
Object Segmentation
Human-AI Interaction
Vision-and-Language
Multimodal Reasoning
Primary area: machine_vision
TL;DR: A new prompt reasoning task that predicts poses and masks of specified individuals using text or positional prompts to advance human-AI interaction, supported by a new large-scale dataset and a unified promptable approach.

Abstract: We introduce Referring Human Pose and Mask Estimation (R-HPM) in the wild, where either a text or positional prompt specifies the person of interest in an image. This new task holds significant potential for human-centric applications such as assistive robotics and sports analysis. In contrast to previous works, R-HPM (i) ensures high-quality, identity-aware results corresponding to the referred person, and (ii) simultaneously predicts human pose and mask for a comprehensive representation. To achieve this, we introduce a large-scale dataset named RefHuman, which substantially extends the MS COCO dataset with additional text and positional prompt annotations. RefHuman includes over 50,000 annotated instances in the wild, each equipped with keypoint, mask, and prompt annotations. To enable prompt-conditioned estimation, we propose the first end-to-end promptable approach named UniPHD for R-HPM. UniPHD extracts multimodal representations and employs a proposed pose-centric hierarchical decoder to process (text or positional) instance queries and keypoint queries, producing results specific to the referred person. Extensive experiments demonstrate that UniPHD produces quality results based on user-friendly prompts and achieves top-tier performance on RefHuman val and MS COCO val2017.

Referring Human Pose and Mask Estimation in the Wild
Bo Miao'
Mingtao Feng" Zijie Wu
Mohammed Bennamoun'
Yongsheng Gao Ajmal Mian
University of Western Australia Xidian University Hunan University *Griffith University
Abstract
We introduce Referring Human Pose and Mask Estimation (R-HPM) in the wild, where either a text or positional prompt specifies the person of interest in an image. This task holds significant potential for human-centric applications such assistive robotics and sports analysis. In contrast to previous works, R-HPM (i) ensures high-quality, identity-aware results corresponding the referred person, and (ii) simultancously predicts human pose and mask for a comprehensive repre- sentation. To achieve this, we introduce large-scale dataset named RefHuman, which substantially extends the MS COCO dataset with additional text and posi- tional prompt annotations. RefHuman includes over 50,000 annotated instances in the wild, each equipped with keypoint, mask, and prompt annotations. To en- able compt-conditionee estimation, we propose the first end-to-end promptable approach named UniPHD for R-HPM. UniPHD extracts multimodal representa- tions and employs proposed pose-centric hierarchical decoder to process (text positional) instance queries and keypoint queries, producing results specific to the referred person. Extensive experiments demonstrate that UniPHD produces quality results based on user-friendly prompts and achieves top-tier performance on RefHuman val and MS COCO va12017.
Selection, eg. NMS
Model
Model
Proeppts Seibbble OR PoSEt
Mask
Tex in
Mask
(8) Multi-person Pose Estimation
(b) Referring Human Pose Estimation
Figure 1: Task illustration of (a) multi-person pose estimation predicts numerous outcomes and requires selection strategies during deployment, potentially leading to false negatives or suboptimal target results, and (b) our referring human pose and mask estimation requires unified promptable model simultaneously predict accurate pose and mask for the person of interest, providing compre- hensive and identity-aware human representations to benefit human-Al interaction.
Introduction
Human pose estimation in the wild is fundamental yet challenging problem in the vision community, fueling advancements various applications like human-Al interaction, activity analysis, video surveillance, assistive robotics, and sports analysis. This task aims to locate keypoints (joint locations) of humans within images in unconstrained environments.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Previous multi- pose estimation techniques typically follow a two-stage paradigm, separating the problem into person detection and local keypoint regression. reypointregression. These techniques can be summarized as top-down and bottom-up approaches. Top-down approaches [9, 15, 63, 79, 84] use a detection model to identify human bounding boxes and separate pose estimation model to predict keypoints on the cropped single-human human image. The independent models in these methods lead to non-end- to-end pipeline with substantial computational costs. Bottom-up approaches [3, 10, 31, 62] usually predict keypoint heatmaps to obtain instance-agnostie keypoints and assign them to individuals using heuristic grouping algorithms. The intricate grouping algorithms in these approaches introduce hand-designed parameters and face challenges in handling complex scenarios such as occlusions.
With udvancements in attention mechanisms [2, 56, 57, 59, 77] and transformer architectures [4, 95], many recent approaches regard human pose estimation as direct set prediction problem and design end-to-end differentiable transformers, leading to notable improvements. They employ bipartite matching to establish one-to-one instance correspondence for the set of predictions, avoiding the need for post-processing in the training stage. Among them, PETR [70] proposes transformer that progressively predicts keypoint positions through pose and keypoint decoders. QueryPose [85], ED-Pose [87]. and GroupPose [44] further incorporate a human detection stage for instance feature extraction query initialization to improve performance and expedite model convergence. They all use keypoint-level queries to capture local details for accurate pose estimation.
Despite demonstrating favorable performance, these methods still require designed strategies to select the best match for target individual during deployment, which can result suboptimal outcomes or false negatives, and lack exploration of human-Al interaction to directly predict expected results based on natural prompts. Additionally, they overlook joint human pose and mask estimation, which provides comprehensive human representations to facilitate applications like assistive robotics and sports analysis. For example, accurate joint human pose and mask estimation in unconstrained environments enables robots to locate, analyze, and interact with target individuals, enhancing user experience and assistive tasks.
In this paper, we propose the new task of Referring Human Pose and Mask Estimation (R-HPM) in the wild. As illustrated in Figure unlike multi-person pose estimation [3. 15, 24, 66], R-HPM is a multimodal reasoning task that requires a unified model to predict both pose and mask of referred individual using user-friendly text or positional prompts, enabling comprehensive and identity- aware human representations without post-pre post-processing. To achieve this, we introduce RefHuman, a large-scale dataset that substantially extends MS COCO [40] with additional text, scribble, and point prompt annotations. Our dataset accommodates diverse task settings to enhance human-AI interaction. Manually annotating text descriptions and scribbles is expensive. To reduce annotation costs, we design a human-in-the-loow text generation strategy using powerful large language models and employ bezier curves to automate scribble generation.
To benchmark R-HPM, we propose an end-to-end promptable approach called UniPHD. Our approach directly adopts prompts as instance queries and coordinates them with keypoint queries to jointly predict identity-aware keypoint positions and masks for the target. Unlike previous works [44, 70, 85, 87], UniPHD introduces a pose-centric hierarchical decoder (PHD) that employs deformable and graph attention to effectively model local details and global dependencies, ensuring aargetsswareness and instance coherence. Furthermore, our approach follows a general paradigm that enables seamless integration with the decoders from recent waasformer-basee methods such as GroupPose [44] and ED-Pose [87] for R-HPM. We conduct extensive experiments on the RefHuman dataset. Integrating our approach with GroupPose and ED-Pose yields promising results in an end-to-end fashion. Moreover, our UniPHD approach achieves top-tier performance on both RefHuman val and MS COCO va12017, demonstrating the effectiveness of our approach and the significance of our task.
Our main contributions are summarized below:
We propose Referring Human Pose and Mask Estimation (R-HPM) in the wild, new task that simultaneously predicts the pose and mask of a specified individual using natural, user-friendly text or positional prompts. This task enhances models with identity and produces comprehensive human representations to benefit human-Al interaction.
We introduce RefHuman, a large-scale dataset that substantially extends COCO for R-HPM. RefHuman contains pose and mask annotations for individuals in diverse, unconstrained environments and is enriched with corresponding text and positional prompts.

---

We propose an end-to-end promptable UniPHD approach for R-HPM. Our approach per- forms pose-centric hierarchical decoding, achieving top-ranked performance and establishing a solid benchmark for future advancements in this field.
2 Related Work
2.1 2D Human Pose Estimation in the Wild
Human pose estimation aims to localize keypoints of individuals in unconstrained environments. For long period, two-stage approaches have dominated this field, generally divided into top-down and bottom-up methods. Top-down methods [9, 15, 34, 63, 79, 84] first detect and crop each person using object detector, then perform pose estimation using a separate model. Although effective, the redundancy from the extra detection step, region of interest operations, and separate training makes them suboptimal. Bottom-Up methods [3, 10, 31, 62] first detect abundant instance-agnostic keypoints and then group them into individual poses. While generally efficient, their intricate grouping algorithms pose challenges in handling complex scenarios, resulting in inferior performance. Furthermore, these two-stage approaches suffer from non-differentiable hand-crafted post-prooe processing steps that challenge optimization. Inspired by the one-stage object detectors [21, 76], pixel-wise regression methods [49, 53, 55, 64, 73, 75, 78, 81, 93] densely predict pose candidates in an end-to-end fashion and apply mon-maximum Suppression (NMS) to obtain poses for different individuals. However, these methods produce redundant results, challenging the removal of duplicates.
these cropped instance-level images
Recent human pose estimation methods [51, 52, 72] explore transformer-based architectures [2, 4, 77, 95] due to their sparse, end-to-end design and promising performance. These methods treat human pose estimation as a direct set prediction problem and use bipartite matching to establish one-to-one instance correspondence during training. Among these, [70] proposes transformer with pose decoder to predict keypoints and joint decoder to refine them. [85] performs estimation on extracted object features reduce noisy context. [85, 87] incorporate human detection stage for query initialization to enhance performance. These methods achieve favorable performance but require complex strategies to identify the best match for a specified person during deployment due to the lack of exploration of prompt reasoning. In this work, we propose multimodal reasoning task to directly and jointly predict the identity-awand aware pose and mask for a referred person, resulting in comprehensive human body representations and facilitating applications in human-Al interaction.
2.2 Referring Image Segmentation
Referring image segmentation (RIS) aims to segment target objects images based on natural
linguistic descriptions [18, 30,90,. 91]. It related tasks include interactive image segmentation [7, 8, 25, 37, 41, 86), which segments targets based on user clicks. Early RIS methods [6, 17, 18, 35, 43, 54, 71, 90] employ convolution and recurrent neural networks for multimodal encoding and segmentation. Their intrinsic constraints in capturing long-range dependencies and handling free-form features often lead to suboptimal performance. To improve multimodal representation and alignment, attention- based bidirectional [19, 20] and progressive [16, 22, 23, 88, 89] cross-modal interaction modules are proposed. Additionally, [80, 92] leverage the strong cross cross-modal alignment capabilities of pre-trained large language models [67, 69].
With advancements in transformer architectures [4, 61, 77, 95], [11, 13, 29, 38, 42, 45, 58, 60, 83, 94] design end-to-end transformers for language-conditioned segmentation and achieve favorable performance. In this work, we investigate unified promptable transformer that effectively processes both text and positional prompts, thereby broadening its generalizability and application scope. Moreover, our model coordinates prompts with keypoint queries, enabling the joint prediction of keypoint positions and segmentation masks for specified individuals.
RefHuman Dataset
We substantially extend COCO [40] to construct the RefHuman dataset. It contains pose and mask annotations for humans along with text and positional prompts to facilitate the new task of R-HPM.
3

---

Describe the person within the red bbox.
Combine the details in two descriptions.
Opt3: person yellow the far right
GPT-4
OptI: A person on far right
ChatGPT PT
Describe the appearance person.
Opt1
Human Correction
GPT-4
Opt2: A person in yellow coat
Opt2
Opt3
(a) Text Generation
Human Correction
Figure 2: Human-in-the-loop text prompt generation. We use GPT to generate descriptions with complementary local details and global context, then manually review/correct the descriptions.
3.1 Data Annotation
Pioneer works on 2D human pose estimation datasets in RGB images include [5, 12, 14, 26, 28]. Recent efforts focus human pose estimation 'in the wild', establishing datasets such MPII [1], MS COCO [40], CrowdPose [33]. COCO-WholeBody [27], and AI Challenger [82]. Despite their prevalence, these datasets lack crucial text and seribble prompt annotations needed for human-Al interaction. The large-scale RefHuman dataset extends the widely-used MS COCO with additional text and positional (scribbles and points) prompt annotations, facilitating promptable models to enhance buman-Al interaction.
Text Prompt Annotation. Manually annotating text descriptions is costly. To mitigate this, design a human-in-the-loos text generation strategy using the large language model, ChatGPT [65]. As shown in Figure 2 (a), first provide the entire image with bounding box indication to GPT-4 to generate a text description [Opt]] ] with global context. However, large language models often misidentify targets in complex scenes. To handle this, we crop the image focus the target person and let GPT-4 generate a description [Opt2] with correct local details. Given that individuals may have similar appearances, descriptions generated based on instance-level images are often not distinguishable. Therefore, we combine the global and local descriptions, integrating them into comprehensive description [Opt3] of the target person. Finally, we manually select and revise these automatically generated descriptions to create accurate text prompts.
Our strategy successfully generates acceptable descriptions for images with simple scenarios. These correspond to over 35% of the cases. In the remaining cases, unsatisfactory descriptions are generated. Common issues include misidentification of individuals, incorrect orientation (e.g., facing left right), and indistinguishable context. Nevertheless, these descriptions often provide valuable context, expediting the annotation process. To further scale up the RefHuman training set, integrate text annotations from RefCOCO/+/g [30, 50], doubling the number of referred instances.
Positional Prompt Annotation. Bézier curves are parametric curves based Bernstein Polynomials and widely used in computer graphics. this work, we employ cubic Bézier curves to simulate scribbles and generate clicks. Given four control points Po- P1. P2, and ps, the cubic Bézier curve starts at Po moving toward P1 and reaches py from the direction of p2. The general equation for a Bézier curve c(t, n) of degree is:
c(t,n) n) = Σ (t)(t)pss
0st51
(1)
1=0
12 1)m- i 0=0,... O...,
(2)
where (t) represents the Bernstein basis polynomials and (*) is the binomial coefficient. To generate scribble prompts, we randomly sample four control points within the foreground mask to fit a cubic Bézier curve, represented
ordered set of points {(11 v1), (z2, y2),... (æn, Bn)).
The curve is then discretized by uniformly sampling twelve points to form a scribble prompt s= kn | ( k 1,2...., 12). For point prompts, randomly select single point
p= v) from where and are the horizontal and vertical coordinates.

---

Table 1: Data statistics of numan--rlated images in RefCOCO [30], RefCOCO+ [30], RefCOCOg [50].
their combined dataset RefCOCO/+/g. and our RefHuman.
Dataset
Prompt
Target
Image
Instance
Expression
RefCOCO
Text
Mask
9209
22819
65550
RefCOCO+
Text
Mask
9209
22804
66463
RefCOCOg
Text
Mask
9141
16435
32299
RefCOCO/+/g
Text
Mask
13468
29470
149278
RefHuman (Ours)
Text, Scribble, Point
Pose, Mask
21634
50210
170017
3.2
Data Statistics
As shown in Table 1, RefHuman includes 50,210 human instances across 21,634 images, larger than RefCOCO [30], RefCOCO+ [30], and RefCOCOg [50], and with additional positional prompts. To construct RefHuman train set, we annotate prompts for all humans in MS COCO train2017 17 set with at least three surrounding people, minimum of eight visible keypoints, and area ratio of at least 2%. For the RefHuman val set, we annotate humans in MS COCO va12017 set, excluding those with non-visible keypoints often not visually clear and difficult to describe accurately. Note that each instance may have multiple text, scribble, and point annotations, with each amage-prompy pair treated as separate sample.
ratio below 1 %, instances below this threshold
4 Method
Viaxal Broder
FFN
Mask
Miltmoda
Errooeer
Poep cene-omtricc
Hiararchical
Denxeer
FFN
Box/Score
K eypoints
Point
(x.9)
Embed Retcoval
Switch
FW
Q
Pooling
Scribble
Embed Retrieval
Text skler kneeling on
Textual Encoder
the
Pose-centric Hierarchical Decoder
Soft
A Ajjccntt Matrix
X Conditional Sogmenlation
Global Dependency Modeling
Multi-scale Visual Feature
instance
Multimodal Feature psi
Sampling Offsets
FW
Word/Pixel-level Feature
Local Detail
2009
A Aggregation
Instance Query
Attention Weights
Instance Keypoint query
hip
Xnee
Figure 3: Detailed architecture of our UniPHD, which contains a multimodal encoder that imbues visual features with prompt awareness and a pose-centric hierarchical decoder that enables prompt- conditioned queries to effectively capture local details and global dependencies within targets. Our unified model is end-to-end and accepts text deseriptions, scribbles, or points prompts predict the keypoint positions and segmentation mask of the target person.
Given an image and a text prompt T or positional prompt P. our task aims to simultaneously predict the keypoint positions K E and binary segmentation mask M E #> of the referred person, where N the number of keypoints, and Н and W are the spatial dimensions. propose fully end-to-end promptable UniPHD approach, as illustrated in Figure 3. UniPHD consists of four key components: Backbone, Multimodal Encoder, Pose-centric Hierarchical Decoder, and Task fask-specific Prediction Heads. A small set of prompt-conditioned queries is employed to identify
this end, we
5

---

the referred person and estimate results. During inference. we directly output expected predictions from the sccorigg query group without any post-processing.
4.1 Backbone
Visual Encoder. We start by using a visual encoder to extract multi-scale visual features and apply a point-wise convolution reduce the channel dimension of features D =256 for efficient multimodal interactions. Feature maps with downsampling rates of {8, 16, 32, 64) are then flattened and concatenated into tokenized visual representations F". We adopt Swin Transformer [47] as the visual encoder in this work.
Prompt Encoder. For a linguistic description with words prompt, we RoBERTa [46] to extract word-level features F" E RLXD and generate sentence-level feature as the instance query Q' E RixD by pooling F", For a discretized scribble or a single point prompt, we retrieve embeddings directly from visual features at stride 16 based on their positions to obtain pixel-level prompt features F", which are then pooled to form the instance query Q'.
4.2 Multimodal Encoder
To generate target-related multimodal representations F visual tokens F" are first enhanced through a modality-speciff cross-attention, which integrate information from prompt features:
F F + (Q F", F"
F")
where i = 0 for text prompt and i = 1 for positional prompts, i.e, we adopt separate parameters for cross-modal fusion of different prompt types. After that, we follow previous works [44, 58, 87] to use memory-efficient deformable transformer encoder to encode the multimodal features. The output of the transformer encoder Fol is then forwarded to the decoder to update queries.
4.3 Pose-centric Hierarchical Decoder
We employ groups of queries for the referred human and generate n groups of results. Each group consists of a prompt-conditioned instance query and & learnable keypoint queries. The keypoint queries regard each keypoint as a target and aim to regress their respective positions, while each instance query, conditioned on the prompt, predicts a confidence score, dynamic segmentation filters, and bounding box to locate the referred human. The scores supervised by the losses of each query group to indicate result quality. During deployment, we directly generate results for the target using the highest-scoring scoring query group without any post-processing.
Prompt-conditioned Query Initialization. We first construct a query group template Q E ((+1))xD by concatenating the text/positional prompt embedding Q' with (e.g., 17) randomly initialized learnable embeddings Q To generate n query groups, we apply linear layers to the prompt-aware multimodal features Fui to identify the top-n highest-scoring positions p and estimate their corresponding keypoint positions and centers as reference points, denoted as c E R (++1)xx2 The template Q/R: is then repeated n times and enhanced with reference points c, their associated positional embeddings, and the top-n highest-scoring multimodal embeddings F to form Q'P Finally, the instance query in each copy incorporates the prompt embedding and positional embed- ding derived from the keypoints center of each candidate for pose-centric decoding. During decoding process, we employ graph-attention to model global dependencies for each query group, enabling all queries to utilize the prompt as guidance to identify the referred person only.
Hierarchical Local Context and Global Dependency Modeling. We introduce pose-centric hierarchical decoder to effectively model complementary local details and global dependencies for target-aware decoding. As illustrated in Figure 3, local details are first aggregated using the efficient deformable attention [95], similar to [44, 58, 87]. Each instance and keypoint query in Q/P independently predicts its sampling offsets and corresponding attention weights on the multimodal features F We then aggregate the sampled features accordingly for each query to capture local details. However, after local detail aggregation, the keypoint queries struggle to perceive the prompt information and lacks interactions with each other, challenging the target-awareness and instance coherence. Additionally, the instance query lacks sufficient relevant global context to accurately locate the referred person at pixel-level.

---

To address these issues, we model each query group as a bipartite graph - E, E, A, with nodes V representing different queries and edges E denoting relations between nodes, constrained by a learnable soft adjacent matrix A E [[(++1)x(*+1), which simulates inherent keypoint-to-kypponn keypoin--oo-nntance, and instance-to-keypoin relations. The edge E, from the i-th node to the j-th node can be formulated as:
Eij- (W*v)) A.j
where W and W are learnable projection matrices. Through our graph attention, both instance and keypoint queries capture global dependencies of the target and thus ensure instance coherence. Ultimately, the instance query generates dynamic filters for mask prediction, while the keypoint queries regress target keypoint positions.
4.4 Task-specific Prediction Heads
As illustrate in Figure 3, four lightweight heads are built top of the decoded queries to predict bounding boxes, class scores, masks, and keypoint positions for the target. The box head predicts the bounding box location of the target to aid the learning process. The class head outputs confidence scores, supervised by losses from each query group, to indicate the prediction quality of each query group. The mask head produces dynamic filters for conditional segmentation on the multimodal features Fol, The keypoint head predicts keypoint positions and their visibility for the referred person in images. Detailed training loss functions for these outputs are supplied in the Appendix.
Conditional Segmentation. After extracting semantically-rin multimodal features F" we address pixel-level target localization using dynamic filters generated by instance queries, which capture both local details and global dependencies. Similar to [58], we standardize the resolution of the multi-scale multimodal features Fel H/8 W/8 and combine them into single feature map F through efficient element-wise addition. We then reshape the prompt-conditioned dynamic filters to form two point-wise convolutions that apply to Fm to obtain the segmentation mask for the referred person.
Experiments
Metrics. We standard metrics to evaluate our task. For pose estimation, adopt OKS-based average precision (AP) and PCKh at a 0.5 threshold (PCKh@0.5) [1]. For segmentation, we report mask AP and overall Intersection-over- Uviinnnnn (IoU). The AP metries are evaluated across all query groups, consistent with prior works [44, 87], while PCKh 0.5 and oloU are measured using only the highest-scoring query group better reflect real-world deployment.
Implementation Details. We train our model on the RefHuman train, which contains approximately 46K person instances with 17 keypoints per instance, and evaluate the RefHuman val. We also evaluate our model on MS COCO by generating positional prompts for all images in the dataset. Due to the page limit, we leave the further implementation details in the Appendix.
5.1 Main Results
In Table 2, we evaluate our models and the recent advances [44, 87] on the RefHuman val set. For fairness, all models are evaluated with inputs resized to a maximum of 640 pixels on the longer side.
Effectiveness of Our Promptable Paradigm. Recent human pose estimation methods like Group- Pose [87] and ED-Pose [44] predict poses for multiple humans but require hand engineered selection strategies to identify the best match for a specified person, which can result in suboptimal outcomes or false negatives. By integrating their decoders into our end-to-end promptable paradigm, we directly generate poses and masks in one go for the referred person, achieving up to 73.0 pose AP, 89.6 PCKh 0.55, and 85.6 oloU with scribble prompts. This performance rivals that of previous models which use 3x more training data and hand engineered result selection strategies. Intersection- based result selection is chosen in Table because distance- and IoU- strategies lead to inferior perfor- mance, as discussed in the ablation study. Furthermore, our paradigm achieves advanced performance even with single point prompt, while previous models struggle with such simple guidance, as the random positive points/clicks can be near poses of unintended humans in crowds. These results demonstrate the effectiveness our end-to-end promptable paradigm and the significance of our proposed R-HPM task, i.e., accurate joint pose and mask estimation based on user- user- friendly prompts.
7

---

Table 2: Results RefHuman val split. Uni-ED-Pose and Uni- Uni-GroupPose integrate ED-Pose [87] and GroupPose [44] into our end-to-end paradigm. Intersection based result selection: selects results covering least 30% of the ground truth box. t: trains models using complete images in COCO train2017. FPS is measured on RTX 3090 with a batch size of 24. Uni-ED-Pose and Uni-GroupPose are trained with less data but rival the performance of vanilla models, which perform post-processing with ground truth boxes. Our UniPHD approach achieves top-tier performance.
Prompt
Backbone
Pose Estimation
Segmentation
AP
PCKh@0.5
oloU
AP
Params
FPS
with intersection-busee result selection using ground truth boxes
GroupPose' [44]
BBox
Swin-T
72.0
ED-Pose [87]
BBox
Swin-L
72.8
GroupPose' [44]
BBox
Swin-L
73.4
our R-HPM methods supporting various prompts, without result selection strategy
Uni-ED-Pose
Text
Swin-T
65.3
78.3
74.5
61.5
Uni-Grit Pro-Grou
Text
Swin-T
65.0
78.0
74.7
61.4
UniPHD
Text
Swin-T
66.7
79.0
75.0
62.2
Uni-ED-Pose
Point
Swin-T
71.9
88.9
82.8
67.7
Uni-GroupPose
Point
Swin-T
71.5
88.5
82.2
67.6
UniPHD
Point
Swin-T
73.5
88.7
83.1
68.9
Uni-ED-Pose
Scribble
Swin-T
72.9
84.9
69.2
175.7M
35.4
Uni-Ground GroupPospP
Scribble
Swin-T
73.0
89.3
85.6
69.0
177.7M
35.6
UniPHD
Scribble
Swin-T
74.7
90.4
85.7
70.0
184.0M
35.3
Table 3: Comparison with state-of-the-ar methods on MS COCO va12017. Our method achieves leading performance in pose estimation while also offering segmentation capabilities.
Max Res.
Backbone
Pose AP
APs0
AP76
APM
APz
PETR [70]
640
Swin-L
66.4
88.1
73.0
56.5
80.0
ED-Pose [87]
640
Swin-L
68.9
89.8
75.3
60.1
81.1
GroupPose [44]
640
Swin-T
68.5
88.9
75.4
60.5
79.9
UniPHD w/ Point
640
Swin-T
73.5
93.7
81.1
68.0
81.8
UniPHD w/ Scribble
640
Swin-T
73.9
93.9
81.9
82.2
Effectiveness of Our UniPHD Approach. In Table 2, our UniPHD approach sets new benchmark in overall performance for R-HPM through pose-centric hierarchical decoding, outperforming the nearest competitor, Uni-GroupPose w/ Scribble, by 1.7 Pose AP and 1.0 Mask AP, with comparable FPS. Intuitively, the interdependent influence of human keypoints makes graph networks suitable for modeling their dependencies. After applying deformable attention to capture local details, employ graph attention with a learnable soft adjacent matrix to simulate keypoint-ter soint-to-keypoinn keypoint- to-instance, and instance-to to-eeetoo--eetnn relations. This matrix guides edge construction to effectively model global dependencies and ensure instance-awarenes and coherence.
Results for Different Prompts. Table 2 shows that scribble prompts outperform point and text prompts in R-HPM. Scribble prompts offer more explicit positional guidance, enhancing instance query robustness. Point prompts, while slightly less accurate. offer greater convenience in human-AI interaction due to their single-click simplicity. Text prompts face challenges in multimodal alignment, especially for crowded scenarios, but provide linguistic flexibility and enable non-phi non-physical interactions. Overall, our unified model effectively handles various prompts, showing great potential to facilitate human-A interaction.
Evaluation on MS COCO va12017. We further evaluate UniPHD on MS COCO by aggregating results from all instances in each image. As shown in Table 3, all models are evaluated using inputs resized a maximum of 640 pixels on the longest side for comparison. Trained solely with positional prompts, UniPHD achieves state-of-the-art performance COCO va12017, even when compared

---

persen shorts smilling with hards OIL bire
An older man in dark adire toasts with wine glass
a.com ac
douuth hy
ng LIS mww
mon stamaling hows
woman doing lunge slance
player namber
Figure 4: Qualitative results of our UniPHD with text prompts in various challenging scenarios.
Table 4: Comparison with state-of- off-the-rrt art text- based segmentation methods on RefHuman.
Table 3: Ablation of result selection strategies for
GroupPose [44] with Swin-T.
Backbone
oloU
Selection Strategy
AP
AP,
AP;
LAVT [88]
Swin-T
74.5
None
33.5
30.3
36,8
CGFormer [74]
Swin-T
75.3
w/ L1
47.4
45.9
48.8
ReLA [42]
Swin-T
75.9
w/ IoU (7=0.3)
69.1
65.8
72.3
SgMg [58]
Swin-T
75.9
w/ Intersection (r=0.5)
70.9
68.7
73.1
Ours
Swin-T
76.3
w/ Intersection (r=0.3)
72.0
73.6
72.7
Table 6: Ablation of multi-task learning.
Table 7: Ablation of global dependency modeling.
Text Prompt
Scribble Prompt
Text Prompt
Scribble Prompt
Pose
Mask
Pose
Mask
Pose
Mask
Pose
Mask
w/o Pose Head
61.8
68.7
w/o Global Dep.
53.6
54.4
63.5
w/o Mask Head
63.3
72.7
Self-Attention
64.9
61.1
73.8
69.4
Ours
66.7
62.2
74.7
70.0
Ours
66.7
62.2
74.7
70.0
competitors using higher-resolution inputs, further demonstrating the efficacy of our approach and the significance of the proposed task.
To further validate the effectiveness of UniPHD segmentation, we compare it with popular open-source text-based segmentation methods trained RefHuman, shown in Table 4. Using only text prompts for training, our model achieves superior performance compared to competitive methods like SgMg [58] and ReLA [42].
Comparison with Text-based Segmentation Methods.
Qualitative Results. In Figure 4, we present qualitative results of UniPHD in various challenging scenarios such as crowded scenes, occlusions, similar appearances, dim lighting, and viewpoint changes. UniPHD effectively captures appearance, location, action, and context information to generate high-quality outputs. More visualizations are supplied in the Appendix.
5.2 Ablation Study
Result Selection Strategies. Table 5 shows the performance of recent human pose estimation model [44] using various result selection strategies on ReiHuman val. Without carefully designed strategies, the model struggles to accurately predict target individuals. The distance- strategy fails to effectively filter out high-scoring irrelevant results, while the intersection based strategy delivers the best performance.
Multi-task Learning. Table 6, we evaluate the impact of multi-task learning in our approach on RefHuman val using the AP metric. Removing either head adversely affects the performance of the other. This validates the effectiveness of our decoder, which enables bidirectional information flow between keypoint and instance queries to enhance both predictions.

---

Table 8: Ablation of query initialization.
Table 9: Training UniPHD with extra data.
Text Prompt
Scribble Prompt
Prompt
Pose Estimation
Segmentation
Posc
Mask
Pose
Mask
AP
P.
oloU
AP
w/o Initialization
65.5
61.0
74.0
69.6
Point
81.0
94.3
88.3
72.3
Ours
66.7
62.2
74.7
70.0
Scribble 81.2
94.9
89.4
73.2
Pose-centric Hierarchical Decoder. In Table 7, we analyze the impact of global dependency modeling in decoder. Without global dependencies, keypoint queries struggle perceive the prompts for predicting target-aware results. Our graph attention clearly outperforms self-attention because it not only models dynamic node relations but also simulates inherent keypoint-to-keypoinn, keypoint-to-instance, and instance-to-keypiin relations via soft adjacent matrix.
Query Initialization. Similar to recent transformer-basee pose estimation methods [44, 87], prompt-conditioned query initialization to enrich queries with dynamic spatial priors. Table 8 reveals that omitting these dynamic priors results in performance decrease of 0.4-1.2% AP. Nonetheless, our model maintains robust performance without query initialization, demonstrating its efficacy.
Training with Extra Data. To unleash the capability of UniPHD for positional crompt-baset predic- tion, we expand our training data beyond RefHuman to encompass the entire MS COCO train2017 images by generating additional point and scribble prompt annotations based on Bézier curves. As shown in Table 9, UniPHID, trained exclusively with positional prompts on the expanded dataset, achieves impressive performance on RefHuman val, with up to 94.9 PCKh @0.5 and 89.4 oloU, demonstrating its scalability.
Conclusion
We introduced Referring Human Pose and Mask Estimation in the wild, a new task aimed at simultaneously predicting the poses and masks of specified individuals using natural, user-friendly text or positional prompts. This task holds significant potential for enhancing human-Al interactions in fields such assistive robotics and sports analysis. To achieve this, introduced the RefHuman dataset that substantially extends MS COCO with additional text and positional prompt annotations. We also proposed the first end-to-end promptable approach named UniPHD, which employs pose- centric hierarchical decoder to model local details and global dependencies for R-HPM. UniPHD achieves top-tier performance, establishing solid benchmark for future advancements in this field. We hope our new task, dataset, and approach could foster advancements in human-A interaction and related areas.
Broader Impacts. Malicious use of R-HPM models can bring potential negative societal impacts, such as unauthorized mass surveillance. We believe that the model itself is neutral with positive human-centric applications, including assistive robotics and sports analysis.
Limitations. The proposed UniPHD approach produces high-quality results with various prompts. However, text prompts perform worse than positional prompts due to misidentification. Future research could enhance vision-language alignment reduce this performance disparity.
Acknowledgments. This work was supported by the Australian Research Council Industrial Trans- formation Research Hub IH180 100002. Professor Ajmal Mian is the recipient of an Australian Research Council Future Fellowship Award (project number FT210100268) funded by the Australian Government.
References
[1] Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2d human pose estimation: New bench- mark and state of the art analysis. In: CVPR. pp. 3686-3693 (2014)
[2] Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning align and translate. In: ICLR (2015)
10

---

[3] Cao, Z., Simon, T. Wei, S.E., Sheikh, Y.: Realtime multi affinity fields. In: CVPR. pp. 7291-7299 (2017)
multi-person person 2d pose estimation using part
[4] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: ECCV. pp. 213-229. Springer (2020)
[5] Charles, J., Pfister, T., Magee, D., Hogg, D., Zisserman, A.: Personalizing human video pose estimation. In: CVPR. pp. 3063-3072 (2016)
[6] Chen, D.J., Jia, S., Lo, Y.C.,, Chen, H.T., Liu, T.L.: See-through-text grouping for referring image segmentation. In: ICCV, pp. 7454-7463 (2019)
[7] Chen, X., Cheung. Y.S.J,, Lim, S.N., Zhao, H.: Scribbleseg: Scribble-based interactive image segmentation. arXiv preprint arXiv:2303.11322 (2023)
[8] Chen, X., Zhao, Z., Zhang, Y., Duan, M., Qi, D., Zhao, H.: Focalclick: Towards practical interactive image segmentation. In: CVPR. pp. 1300-1309 (2022)
[9] Chen, Y., Wang, Z., Peng. Y., Zhang. Z., Yu, G., Sun, J.: Cascaded pyramid network for multi-person pose estimation. In: CVPR. pp. 7103-7112 (2018)
[10] Cheng, B., Xiao, B., Wang, J., Shi, H., Huang, T.S., Zhang, L.: Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation. In: CVPR. pp. 5386-5395 (2020)
[11] Chng, Y.X., Zheng, H., Han, Y., Qiu, X., Huang, G.: Mask grounding for referring image segmentation. arxiv preprint arXiv:2312.12198 (2023)
[12] Dantone, M., Gall, J., Leistner, C., Van Gool, L.: Human pose estimation using body parts dependent joint regressors. In: CVPR. pp. 3041-3048 (2013)
[13] Ding, H., Liu, C., Wang. S., Jiang, X.: Vlt: Vision-language transformer and query generation for referring segmentation. TPAMI (2022)
[14] Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. International journal of computer vision 88, 303-338 (2010)
[15] Fang, H.S., Xie, S., Tai, Y.W., Lu, C.: Rmpe: Regional multi-person pose estimation. In: ICCV. pp. 2334-2343 (2017)
[16] Feng, G., Hu, Z., Zhang, L.,, Lu, H.: Encoder fusion network with co-attention embedding for referring image segmentation. In: CVPR. pp. 15506-15515 (2021)
[17] Hu, R., Rohrbach, M., Andreas, J., Darrell, T., Saenko, K.: Modeling relationships in referential expressions with compositional modular networks. In: CVPR. pp. 1115-1124 (2017)
[18] Hu, R., Rohrbach, M., Darrell, T.: Segmentation from natural language expressions. In: ECCV. pp. 108-124, Springer (2016)
[19] Hu, Y., Wang, Q. Shao, W., Xie, E., Li, Z., Han, J., Luo, P.: Beyond one-to-one: Rethinking the referring image segmentation. In: ICCV, pp. 4067-4077 (2023)
[20] Hu, Z., Feng, G., Sun, J., Zhang, L., Lu, H.: Bi-directional relationship inferring network for referring image segmentation. In: CVPR. pp. 4424 4433 (2020)
[21] Huang, L., Yang, Y., Deng, Y., Yu, Y.: Densebox: Unifying landmark localization with end to end object detection. arXiv preprint arXiv:1 0094444744444 (2015)
[22] Huang, S., Hui, T., Liu, S., Li, G., Wei, Y., Han, J., Liu, I.., Li, B.: Referring image segmentation via cross-modal progressive comprehension. In: CVPR. pp. 10488-10497 (2020)
[23] Hui, T., Liu, S., Huang, S., Li, G., Yu, S., Zhang, F., Han, J.: Linguistic structure guided context modeling for referring image segmentation. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16. pp. 59-75. Springer (2020)

---

[24] Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., Schiele, B.: Deepereut: A deeper. stronger, and faster multi-person pose estimation model. In: Computer Vision-ECCY ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1 1114, 2016, Proceedings, Part VI 14. pp- 34-50. Springer (2016)
[25] Jang, W.D., Kim, C.S.: Interactive image segmentation via backpropagating refinement scheme. In: CVPR. pp. 5297-5306 (2019)
[26] Jhuang, H., Gall, J., Zuffi, S., Schmid, C., Black, M.J.: Towards understanding action recogni- tion. In: ICCV. pp. 3192-3199 (2013)
[27] Jin, S., Xu, L., Xu, J., Wang, C., Liu, W., Qian, C., Ouyang, W., Luo, P.: Wholc-body human pose estimation in the wild. In: Computer Vision Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 23-28, 2020, Proceedings, Part IX 16. pp. 196-214. Springer (2020)
[28] Johnson, S., Everingham, M.: Clustered pose and nonlinear appearance models for human pose estimation. In: bmve. vol. 2, p. 5. Aherystwyth, UK (2010)
[29] Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., Carion, N.: Mdetr-modulated detection for end-to-end multi-modal understanding. In: ICCV, pp. 1780-1790 (2021)
[30] Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T.: Referitgame: Referring to objects in photographs of natural scenes. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787-798 (2014)
[31] Kreiss, S., Bertoni, L., Alahi, A.: Pifpaf: Composite fields for human pose estimation. In: CVPR. pp. 11977-11986 (2019)
[32] Kuhn, H.W.: The hungarian method for the assignment problem. Naval research logistics quarterly 2(1-2), 83-97 (1955)
[33] Li, J., Wang, C., Zhu, H., Mao, Y., Fang, H.S., Lu, C.: Crowdpose: Efficient crowded scenes pose estimation and new benchmark. In: CVPR. pp. 10863-10872 (2019)
[34] Li, K., Wang, S., Zhang. X., Xu, Y., Xu, W., Tu, Z.: Pose recognition with cascade transformers. In: CVPR. pp. 1944-1953 (2021)
[35] Li, R., Li, K., Kuo, Y.C., Shu, M., Qi, X., Shen, X., Jia, J.: Referring image segmentation via recurrent refinement networks. In: CVPR. pp. 5745-5753 (2018)
[36] Li, X., Sun, X., Meng, Y., Liang, J., Wu, F., L.; J.: Dice loss for data intellnmbaaeed nlp tasks. In: ACL (2020)
[37] Li, Z., Chen, Q., Koltun, V.: Interactive image segmentation with latent diversity. In: CVPR. pp. 577-585 (2018)
[38] Li, Z., Wang, M., Mei, J., Liu, Y.: Mail: A unified mask-image-lagguag trimodal network for referring image segmentation. arXiv preprint arXiv:21 10.10777 (2021)
[39] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: ICCV. pp. 2980-2988 (2017)
[40] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740-755. Springer (2014)
[41] Lin, Z., Zhang, Z., Chen, L.Z., Cheng, M.M., Lu, S.P.: Interactive image segmentation with first click attention. In: CVPR. pp. 93339-13344 (2020)
[42] Liu, C., Ding, H., Jiang, X.: Gres: Generalized referring expression segmentation. In: CVPR. pp. 12592-23300 (2023)
[43] Liu, C., Lin, 7.,, Shen, X., Yang, J., Lu, X., Yuille, A.: Recurrent multimodal interaction for referring image segmentation. In: ICCV, pp. 1271 2-71-1280 (2017)
12

---

[44] Liu. H., Chen, Q. Tan, Z., Liu, J..., Wang, J., Su, X., Li, X., Yao, K., Han, J., Ding, E., et al.: Group pose: A simple baseline for end-to-end multi-person pose estimation, In: ICCV. pp. 15029-15038 (2023)
[45] J., Ding, H., Cai, Z.., Zhang, Y., Satzoda, R.K., Mahadevan, V., Manmatha, R.: Polyformer: Referring image segmentation as sequential polygon generation. In: CVPR. pp. 18653-18663 (2023)
[46] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V: Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv:1907. 1692 (2019)
[47] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: ICCV. pp. 10012-10022 (2021)
[48] Loshchilov, I. Hutter, R: Decoupled weight decay regularization. arXiv preprint 1xXiv:7111.05111 (2017)
[49] Lu, P., Jiang, T., Li, Y., Li, X., Chen, K., Yang, W.: Rtmo: Towards high-performance one-stage real-time multi-person pose estimation. arXiv preprint arXiv:2312.07526 (2023)
[50] Mao, J., Huang, J., Toshev, A., Camburu, O., Yuille, A.L., Murphy, K.: Generation and comprehension of unambiguous object descriptions, In: CVPR. pp. 11-20 (2016)
[51] Mao, W., Ge. Y. Shen, C.. Tian, Z., Wang, X.. Wang. Z.: Típose: Direct human pose estimation with transformers. arXiv preprint atXiv:2103.15322 (2021)
[52] Mao, W., Ge, Y., Shen, C., Tian, Z., Wang, X., Wang, Z., den Hengel, A.v.: Poseur: Direct human pose regression with transformers. In: ECCV. pp. 72-88. Springer (2022)
[53] Mao, W., Tian, Z., Wang, X., Shen, C.: Fepose: Fully convolutional multi-person pose estima- tion with dynamic instance- aware convolutions. In: CVPR. pp. 9034-9043 (2021)
[54] Margffoy- Tureff E., Pérez, J.C., Botero, E., Arbeláez, P.: Dynamic multimodal instance segmen- tation guided by natural language queries. In: ECCV. pp. 630-645 (2018)
[55] McNally, W., Vats, K., Wong, A., McPhee, J.: Rethinking keypoint representations: Modeling keypoints and poses objects for multi-person human pose estimation. In: ECCV. pp. 37-54. Springer (2022)
[56] Miao, B., Bennamoun, M., Gao, Y., Mian, A.: Regional video object segmentation by efficient motion-awand mask propagation. In: DICTA. pp. 1-6 (2022)
[57] Miao, B., Bennamoun, M., Gao, Y., Mian, A.: supervupe video object segmentation by motion-aware mask propagation. In: ICME (2022)
[58] Miao, B., Bennamoun, M., Gao, Y., Mian, A.: Spectrum-guided multi-granularity referring video object segmentation. In: ICCV. pp. 920-930 (2023)
[59] Miao, B., Bennamoun, M., Gao, Y., Mian, A.: Region aware video object segmentation with deep motion modeling. IEEE Transactions on Image Processing (2024)
[60] Miao, B., Bennamoun, M., Gao, Y., Shah, M., Mian, A.: Temporally consistent referring video object segmentation with hybrid memory. IEEE Transactions on Circuits and Systems for Video Technology (2024)
[61] Miao, B., Zhou, L., Mian, A.S., Lam, T.L., Xu, Y.: Object-to-scene Learning to transfer object knowledge indoor scene recognition. In: IROS. pp. 2069-20776 (2021)
[62] Newell, A., Huang, 7.,, Deng. J.: Associative embedding: End-to-end learning for joint detection and grouping. Advances in neural information processing systems 30 (2017)
[63] Newell, A., Yang. K., Deng. J.: Stacked hourglass networks for human pose estimation. In: Computer Vision-ECCV 2016: 14th Furopean Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14. pp. 483-499. Springer (2016)
13

---

[64] Nie. X.. Feng, J., Zhang, J., Yan, S.: Single-stage multi person pose machines. In: ICCV. pp. 6951-6960 (2019)
[65] OpenAl: Gpt-4 technical report (2023)
[66] Papandreou, G., Zhu, T., Kanazawa, N., Toshev, A., Tompson, J., Bregler, C., Murphy, K.: Towards accurate multi pertso pose estimation in the wild. In: CVPR. pp. 4903 0903-4911200 (2017)
[67] Radford, A., Kim, J. W. Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al:: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748-88 8763. PMLR (2021)
[68] Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A.. Reid, L,, Savarese, S.: Generalized inter- section over union: metric and loss for bounding box regression. In: CVPR. pp. 658-666 (2019)
[69] Rombach, R.. Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR. pp. 10684-10695 (2022)
[70] Shi, D., Wei, X., Li. L., Ren, Y., Tan, W.: End-to-end multi-person pose estimation with transformers, In: CVPR. pp. 11069-11078 (2022)
[71] Shi, H., Li, H., Meng, F., Wu, Q.: Key-word-aware network for referring expression image segmentation. In: ECCV. pp. 38 54 (2018)
[72] Stoffl, L., Vidal, M., Mathis, A.: End-to-end trainable multi-instance pose estimation with transformers. arXiv preprint arXiv:2103. 12115 (2021)
[73] Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation. In: CVPR. pp. 5693-5703 (2019)
[74] Tang, J., Zheng, G., Shi, C., Yang. S.: Contrastive grouping with transformer for referring image segmentation. In: CVPR. pp. 23570-23580 (2023)
[75] Tian, Z., Chen, H., Shen, C.: Directpose: Direct end-to-end multi-person pose estimation. arXiv preprint 1cXiv:1911.07451 (2019)
[76] Tian, Z., Chu, X., Wang, X., Wei, X., Shen, C.: Fully convolutional one-stage 3d object detection on lidar range images. Advances in Neural Information Processing Systems 35, 34899-3491 (2022)
[77] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, E., Polosukhin, I.: Attention is all you need. In: NeurlPS (2017)
[78] Wang, D., Zhang, S.: Contextual instance decoupling for robust multi-person pose estimation. In: CVPR. pp. 11000-11666 (2022)
[79] Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., et al.: Deep high-resolution representation learning for visual recognition. IEEE transactions on pattern analysis and machine intelligence 43(10), 3349-3364 (2020)
[80] Wang, Z., LU, Y., Li, Q., Tao, X., Guo, Y., Gong, M., Liu, T.: Cris: Clip-driven referring image segmentation. In: CVPR. pp. 11686-1 1695 (2022)
[81] Wei, F. Sun, X., Li, H., Wang. J., Lin, S.: Point-set anchors for object detection, instance seg- mentation and pose estimation. In: Computer Mision-ECCv 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16. pp. 527-544. Springer (2020)
[82] Wu, J., Zheng, H., Zhao, B., Li, Y., Yan, B., Liang, R., Wang, W. Zhou, S., Lin, G., Fu, Y., et al.: Ai challenger: large-scale dataset for going deeper in image understanding. arXiv preprint arXiv:1711.06477 (2017)
[83] Wu, J., Jiang, Y., Sun, P., Yuan, 7., Luo, P.: Language as queries for referring video object segmentation. In: CVPR. pp. 4974-4984 (2022)
14

---

[84] Xiao, B., Wu, H.. Wei, Y:: Simple baselines for human pose estimation and tracking. In: ECCV. pp. 466 481 (2018)
[85] Xiao, Y. Su, K., Wang, X., Yu, D., Jin, L., He, M., Yuan, Z.: Querypose: Sparse multi-person pose regression via spatial-aware part-level query. Advances in Neural Information Processing Systems 35, 12464-12477 (2022)
[86] Xu, N., Price, B., Cohen, S., Yang, J., Huang, T.S.: Deep interactive object selection. In: CVPR. pp. 373-381 (2016)
[87] Yang. J., Zeng, A., Liu, S., Li, Zhang, R., Zhang, L.: Explicit box detection unifies end- to-end multi-person pose estimation. In: The Eleventh International Conference on Learning Representations (2022)
[88] Yang. Z.., Wang, J., Tang, Y., Chen, K., Zhao, H., Torr, P.H.: Lavt: Language-aware vision transformer for referring image segmentation. In: CVPR. pp. 18155-18166 (2022)
[89] Ye, L.., Rochan, M., Liu, Z., Wang. Y.: Cross- modal self-attention network for referring image segmentation. In: CVPR. pp. 10502-10511 (2019)
[90] Yu, L., Lin, Z., Shen, X., Yang, J., Lu, X., Bansal, M., Berg, T.L.: Matmet: Modular attention network for referring expression comprehension. In: CVPR. pp. 1307-1315 (2018)
[91] Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. pp. 69-85. Springer (2016)
[92] Zhao, W., Rao, Y., Liu, Z., Liu, B., Zhou, J., Lu, J.: Unleashing text-to-image diffusion models for visual perception. In: ICCV. pp. 5729-5739 (2023)
[93] Zhou, X., Wang, D., Krähenbühl, P.: Objects points. arXiv preprint arXiv: 190..078000088500 (2019)
[94] Zhu, C., Zhou, Y., Shen, Y., Luo, G., Pan, X., Lin, M., Chen, C.,, Cao, L., Sun, X., Ji, R.: Seqtr: A simple yet universal network for visual grounding. In: ECCV. pp. 598-615. Springer (2022)
[95] Zhu, X., Su, W., Lu, Li, B., Wang, X., Dai. J.: Deformable detr: Deformable transformers for end-to-end object detection, In: ICLR (2021)
15

---

A Appendix
A.1 Training Loss Functions
The overall loss function for UniPHD consists of four parts:
Ltrain = Lboz + Letoss + Loose + Lmasks
(5)
where Looz is for human box regression that contains L1 loss and GIOU [68] loss, Coiass is for classification includes focal loss [39] with a = 0.25 and ? = 2, Lpose is for keypoint regression and visibility prediction that includes L1 loss, the constrained L1 loss-OKS loss [70], and cross entropy loss, Lmakk is for segmentation that includes Dice [36] loss and focal loss. The loss coefficients A A Focal X OKSS ACE A X are 5, 2, 2, 10, 4, 4, 5, 2, following [44, 58, 87]. class ppose: "oose
The Hungarian algorithm [32] used to identify the optimal assignment (query group) with the highest similarity to the ground truth for training. The class label for the optimal instance query with minimum loss from the ground truth is set to one, while all others are set to zero.
A.2 Implementation Details
We follow the common optimization strategies in [44, 58, 83, 87] train the models. During training, we augment input images through random flip, random crop, and random resize with the shorter sides within the range of 360 to 640 pixels and the longer sides up to 640 pixels. For each iteration, randomly select either a text or positional prompt with equal probability. We use the Adam [48] optimizer with weight decay of x 100 and train our models on 24GB RTX 3090 GPUs with batch size 16 for 20 epochs. The initial learning rates are set to x 10 for the visual encoder and 1 10 for other components, with rate decay at the 18th epoch by a factor of 10. Both the multimodal encoder and pose-centric hierarchical decoder consist of 6 layers, and we use 20 query groups for our models, During testing, we resize the input images with their longer sides up 640 pixels.
A.3 Different Query Group Numbers
In Table 10, our approach perform well across different numbers of query groups, measured by the AP metric. This hyperparameter primarily affects pose estimation, with 20 query groups identifying more candidates achieve the best performance.
Table 10: Ablation of different number of query groups.
Num.
Text Prompt
Scribble Prompt
Pose
Mask
Pose
Mask
5
62.6
60.1
72.1
69.1
10
64.4
61.4
72.4
20
66.7
62.2
74.7
70.0
A.4 Increasing Model Capacity
In Table 11, we enhance model capacity by adding multimodal encoder layers and increasing the feature dimensions from 256 to 384. The results indicate our current settings are already highly effective.
Table 11: Ablation of increasing model capacity.
Text Prompt
Scribble Prompt
Pose
Mask
Pose
Mask
Baseline
66.7
62.2
74.7
70.0
w/ more layers
66.3
62.0
74.9
70.4
w/ higher dimensions
66.6
62.8
74.5
70.2
16

---

A.5
Additional Visualizations
In Figure we present additional qualitative results of UniPHD with text and positional prompts. UniPHD effectively processes these prompts to produce high-quality results.
Image
UniPHD w/ Text
UniPHD w/ Point
UniPHD w/ Scribble!
spectator oobess
Mocreded d locaing
iin Bock
STOP
STOP
STOP
STOP
holds surbbant
SE A Siiaaiiii Sai haindss crevardd
Figure 5: Qualitative results of our UniPHD with different prompts in various challenging scenarios.
Licenses of MS COCO and Re/COCO/+/g. The annotations in MS COCO belong the COCO Consortium and are licensed under CC BY 4.0. The use the images complies with Flickr Terms of Use. See https: https://cooooatasstt. org/ #termsofuse for more details. The RefCOCO/+ datasets are licensed under Apache License 2.0 and the RefCOCOg dataset is licensed under CC BY 4.0.
Terms of Use and License of RelHuman. RefHuman is licensed under CC BY 4.0.
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the claims, which are supported by our experimental results. See Abstract and Sec. 1.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
The paper does not contain any theoretical assumptions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We report the dataset, model, and training details in Sec. 3, Sec. 4, Sec. 5, and the Appendix. The introduced dataset and code publicly available.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
The introduced dataset and code are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaecccccccccccccclt Sondei/eeideeeeeeeuubiiiiiii ssisssolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
See Sec. 5 and the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the usual format used in previous related works to report and compare the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http///trpp...ppppiiip/epp// wtt.ehhhhhaiieeeeneeeeeea ines?
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Sec. 6.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We cite the creators or original papers of all the related code, data, and models used in this paper. All the assets are free for research study and widely used in previous related works. We also provide the licenses of the used datasets in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwoee com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
22

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We report the dataset, model, and training details in Sec. 3, Sec. 4, Sec. 5, and the Appendix. The introduced dataset and code are publicly available.
Guidelines:
The answer NA means that the paper does release assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We did use crowdsourcing conduct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did not crowdsourcing conduct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
23


Reviews:
[{"review_id": "Perb8QAM8C", "reviewer": "Reviewer_TbW6", "summary": "This paper proposes a new task called Referring Human Pose and Mask Estimation and introduces the corresponding RefHuman dataset, which is beneficial for research on human behavior comprehension. Additionally, the authors present a model that leverages three types of prompts for this task. The proposed UniPHD model achieves promising performance in this area.", "questions": "See the weaknesses", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed R-HPM task and the RefHuman dataset are beneficial to related research.\n\n2. The proposed method shows promising performance on both the new task and traditional human pose estimation tasks.", "weaknesses": "1. The descriptions of the method, especially in Section 4.3, is confusing. For example, what is the relationship between F^(vl) and P’? How do you enhance the template based on P’? And why do “the keypoint queries struggle to perceive the prompt information and lacks interactions with each other, challenging the target-awareness and instance coherence” ? It seems that these keypoint queries can interact with each other in existing decoders. More detailed descriptions and explanations are needed to understand this work.\n\n2. The author proposed that existing research overlooks joint human pose and mask estimation, which provides comprehensive human representations. However, the paper lacks comparisons of the proposed model with other variants, such as UniPHD without the mask head and UniPHD without the pose head. Such comparisons could help clarify the relationship and benefit of the pose estimation and mask estimation tasks.\n\n3. The proposed method needs further validation. Providing an ablation study of the proposed query initialization method and comparing model parameters and computational complexity would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer TbW6 for their comments and appreciation of our work. In response to the concerns expressed in Weaknesses and Questions, we provide the following answers:\n\n> What is the relationship between $\\mathbf{F}^{vl}$ and $\\mathbf{P}^{'}$, and how do you enhance the template based on $\\mathbf{P}^{'}$? \n\nThank you for the comments. We extract the multimodal embedding $\\mathbf{F}^{vl}_{\\mathbf{c}}$ at the highest-scoring position $\\mathbf{c}$ (evaluated using linear layers) to estimate a set of $k$=17 keypoint positions $\\mathbf{P}^{'}$ for better query initialization. We enhance the keypoint query template $\\mathbf{Q}^{P} \\in{\\mathbb{R} ^{k \\times D}}$ by adding semantically rich multimodal embeddings $\\mathbf{F}^{vl} _{\\mathbf{P}^{'}}$ extracted at positions $\\mathbf{P}^{'}$, which also serve as reference points for deformable attention in the decoder. We will explain this in more detail in the revised paper.\n\n> Why do \"the keypoint queries struggle to .....\"? It seems that these keypoint queries can interact with each other in existing decoders. \n\nPerhaps there is some misunderstanding which can be removed by rephrasing the sentence as \"However, *after local detail aggregation,* the keypoint queries struggle to perceive .......\".\nWe first update each query embedding separately by capturing local details, which lacks query interactions. Therefore, we mention this in L222 to introduce our global dependency modeling, which captures keypoint-to-keypoint, keypoint-to-instance, and instance-to-keypoint relations to enhance target awareness and instance coherence. We did not claim that other works' decoders cannot perform keypoint query interactions.\n\n>  Comparisons of UniPHD with its variants, such as UniPHD without the mask head and UniPHD without the\npose head.\n\nThanks for the valuable suggestion. We performed the comparisons in Table A, which reveal that removing either head adversely affects the performance. This confirms the effectiveness of our synergistic decoder, which facilitates keypoint-instance interactions, enabling bidirectional information flow and enhancing both predictions. We will add this ablation study to the paper.\n\n*Table A: Ablation of prediction heads.*\n| | Pose AP (Text) | Mask AP (Text) | Pose AP (Scribble) | Mask AP (Scribble) | \n|:-|:-:|:-:| :-:|:-:|\n| w/o pose head | - | 61.8 |  - | 68.7 | \n| w/o mask head | 63.3 | - | 72.7 | - | \n| Ours | **66.6** | **62.1** | **74.6** | **70.0** | \n\n> Provide an ablation study of the proposed query initialization. \n\nTable B shows the ablation study on query initialization by removing the query enhancement discussed in our response to the first comment. Results show a minor performance decrease due to the absence of dynamic spatial priors.\n\n*Table B: Ablation of query initialization.*\n| | Pose AP (Text) | Mask AP (Text) | Pose AP (Scribble) | Mask AP (Scribble) | \n|:-|:-:|:-:| :-:|:-:|\n| w/o enhancement | 65.5 | 61.0 |  74.0 | 69.6 | \n| Ours | **66.6** | **62.1** | **74.6** | **70.0** | \n\n\n> Compare model parameters and computational complexity.\n\nThank you for this suggestion. Table C compares the model parameters and FPS (measured on a single RTX4090 GPU). Our method demonstrates better performance at comparable FPS while using slightly more parameters. We will include this comparison in the revised paper.\n\n*Table C: Comparison with competitors using the scribble prompt for inference.*\n| | Pose AP | Mask AP | FPS | Params |\n|:-|:-:|:-:| :-:|:-:|\n| Uni-ED-Pose | 72.9 | 69.2 | 50 | **175.7M** | \n| Uni-GroupPose | 73.0 | 69.0 | **52** | 177.7M | \n| Ours | **74.6** | **70.0** |  50 | 184.0M |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hzFMP9iVQE", "reviewer": "Reviewer_apkP", "summary": "In this paper the authors tackle the problem of in-the-wild human pose estimation in a “referring” setting where the goal is to determine the pose of a person referred to using either a text prompt or positional prompt. To achieve this, the authors annotate MS COCO dataset with over 50K annotated instances for 2D keypoints, mask  and prompt (either as text, points or scribbles). They use this dataset to train a model called UniPHD which consists of several submodules. The results show the paper is able to train a strong baseline model which will be useful for future research.", "questions": "Q1. How does Eq. 4 fit into mask prediction task?  Is that applied only to the pose prediction branch?", "limitations": "The authors have briefly acknowledged limitations of their method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "S1. This paper introduces a new task of referring human pose estimation by releasing a large dataset of 50K annotations (as an extension to MS COCO) enabling researchers to train models that can interact with the model using text and points/scribble.\n\nS2. The paper also releases a baseline method for the same task with an aim to learn end-to-end R-HPM.", "weaknesses": "W1. The motivation for this obtaining pose in a “referring” manner is unclear. For instance, it might be more valuable to focus a text based human detection as opposed to pose estimation because the former can open up many avenues such as human tracking for research. \n\nW2. Multimodal encoder is shallow potentially limiting the interaction of the different encoded features. It might be helpful increase the capacity of the the multimodal encoder model.\n\nW3. Parts of the paper not are fully clear. I understand that the paper aims to be “single-stage” but on L205-L214, there seem to be multiple stages where candidates are detected and then used for subsequent pose estimation. (Which makes it two-stage?)\n\nMinor comments\n\n* Table. 1 does not require last column\n* Table. 3 typo “Scibble\" —> “Scribble”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer apkP for their comments and appreciation of our work. In response to the concerns expressed in Weaknesses and Questions, we provide the following answers:\n\n> The motivation for obtaining pose in a \"referring\" manner is unclear. It might be more valuable to focus on text-based human detection, which can open up many avenues.\n\nOur task predicts both **mask** and pose of the referred person simultaneously using common prompts to provide detailed, identity-aware human representations that benefit human-AI interaction.\nThis has the following advantages over text-based human detection:\n\n(1) Our segmentation provides finer-grained, less noisy information than detection, crucial for robotics and precision tasks.\n\n(2) Our pose estimation adds essential semantics for behavior understanding, complementing mask/box information.\n\n(3) Support for diverse prompts (text, point, scribble), broaden the scope of human-AI interaction.\n\n(4) Our proposed dataset supports various settings, including text-based human detection mentioned by the reviewer. \n\nReviewer DJJG also thinks the motivation of this work is pretty good.\n\n\n> It might be helpful to increase its capacity.\n\nThanks for this valuable suggestion. We enhanced the capacity of the multimodal encoder by adding three layers or increasing feature dimensions from 256 to 384. As shown in Table A, both strategies slightly improved overall performance, demonstrating that our current settings are highly effective. We will incorporate the results in the revised paper.\n\n\n*Table A: Increasing the capacity of the multimodal encoder.*\n| | Pose AP (Text) | Mask AP (Text) | Pose AP (Scribble) | Mask AP (Scribble) | \n|:-|:-:|:-:| :-:|:-:|\n| Baseline | **66.6** | 62.1 | 74.6 | 70.0 | \n| w/ more layers | 66.3 | 62.0 | **74.9** | **70.4** |\n| w/ higher dimensions | **66.6** | **62.8** |74.5 | 70.2 | \n\n\n> The method seems to be two-stage, according to L205-214.\n\nTo address the reviewer's concern, we will remove the term \"one-stage\" since it does not affect our contribution. Previous papers like ED-Pose described \"two-stage\" methods that first perform detection, then pose estimation on cropped single-human images, or use heuristic grouping to process numerous detected instance-agnostic keypoints. \nOur method uses linear layers to identify a high-scoring point $\\mathbf{c}$ on multimodal features for initializing keypoint queries (L205-214), which directly regress target keypoint positions on the *entire* image. \nOur query initialization is similar to the human query selection in ED-Pose, which claims to be one stage.\nHence, we call our method one stage because it is end-to-end and avoids cropping, pose estimation on cropped single-human images, or heuristic grouping. \n\nMoreover, the initialization on L205-214 does not even play a significant role in our method, evidenced by an ablation study using only learnable keypoint queries. As shown in Table B, this results in only a minor performance decrease, demonstrating the effectiveness of our method. We will include this ablation study in the paper.\n\n*Table B: Ablation of query initialization.*\n| | Pose AP (Text) | Mask AP (Text) | Pose AP (Scribble) | Mask AP (Scribble) | \n|:-|:-:|:-:| :-:|:-:|\n| w/o initialization | 65.5 | 61.0 | 74.0 | 69.6 | \n| Ours | **66.6** | **62.1** | **74.6** | **70.0** | \n\n>  Table 1 does not require last column and typo “Scibble\" —> “Scribble”.\n\nThank you for pointing this out. We will remove the last column from Table 1, correct the pointed typo and carefully check the complete paper.\n\n\n> How does Eq. 4 fit into mask prediction task? Is that applied only to the pose prediction branch?\n\n\nOur graph attention, including edge construction (Eq. 4), is applied concurrently to both mask and pose prediction branches. Our query set $\\mathbf{Q}^{IP} \\in{\\mathbb{R} ^{(k+1) \\times D}}$ comprises one instance (prompt) query and $k$=17 keypoint queries. The graph attention treats all queries in $\\mathbf{Q}^{IP}$ as nodes and models keypoint-to-keypoint, keypoint-to-instance, and instance-to-keypoint relations, enhancing all queries simultaneously. Ultimately, the instance query generates dynamic filters for mask prediction, while the keypoint queries estimate keypoint positions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rxHY1xwIVe", "reviewer": "Reviewer_DJJG", "summary": "This paper introduces a new task named as Referring Human Pose and Mask Estimation (R-HPM), which adopt text/point/scribble to represent a specific person and estimate its pose and segmentation mask. To achieve this goal, this paper proposes a new R-HPM dataset named RefHuman and a new method UniPHD to perform R-HPM. Experiments on the proposed dataset and MSCOCO demonstrate the effectiveness of the proposed method.", "questions": "See Weakness.\n\nOverall, I think the motivation and contribution of this work is pretty good. But there are still some questions that should be clarified in the revised paper.", "limitations": "This paper already discusses limitation in Sec.6.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe proposed R-HPM task is useful and complementary to existing HPM tasks, which can bring new insights into this area.\n2.\tThe proposed dataset RefHuman is large and can support the following research in R-HPM. The provided text/point/scribble annotation is complete and can accurately describe a specific person.\n3.\tExperiments on the proposed dataset and MSCOCO demonstrate the effectiveness of the proposed method in R-HPM task.", "weaknesses": "1.\tThe definition of point and scribble in Sec.3.1 is not clear. Does point prompt only contains only one point and scribble contains 12 points? How to define the point in point prompt? This paper should give a form definition, not textual description.\n2.\tSome confusion in experiments. First, what is the meaning of dagger in Table 2. If I understand correct, dagger denotes to adopt all images from MSCOCO (~60K) into training, but RefHuman only contains 20K images, how UniHPD can utilize the rest 40K images without ref annotation? Second, I think * should be the default evaluation configuration (namely choose the top-1), so what is the evaluation configuration without * and why adopt it as default configuration?\n3.\tHow about adopting multiple types of prompts to perform R-HPM? For example, using both text and point simultaneously to refer a specific person.\n4.\tEvaluation metric is not suitable to R-HPM. This paper adopts AP to evaluate performance, which is designed to evaluate multiple objects in one image. However, in referring setting, only single instance is involved, therefore single instance evaluation metric such as PCKh@0.5 is more suitable for R-HPM, or just evaluating the keypoint error/segmentation IoU is ok.\n5.\tTo establish a comprehensive benchmark, the authors are expected to test some existing methods so that the following work can take as reference. None of the compared methods in Table 2 are referring-based methods, so this paper should reimplement some referring-based methods to test their performance on RefHuman, e.g.,  some referring segmentation methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer DJJG for their comments and appreciation of our work. In response to the concerns expressed in Weaknesses and Questions, we provide the following answers:\n\n> Does a point prompt contain only one point, while a scribble contains 12 points? This paper should provide a formal definition for both point and scribble.\n\nYes, the point prompt contains only one point, while the scribble prompt contains 12 points uniformly sampled from the curve. We will add the following formal definitions as suggested:\n\n**Point prompt**: A single point $\\mathbf{p}$=$(x,y)$ at any position in the target area, where $x$ and $y$ are the horizontal and vertical coordinates.\n\n**Scribble prompt**: A scribble can be a continuous, free-form curve represented by an ordered set of $n$ points {$(x_{1},y_{1}), (x_{2},y_{2}), ..., (x_{n},y_{n})$} anywhere in the target area. In this work, we discretize the curve by uniformly sampling 12 points to form the scribble prompt $\\mathbf{s}$ = \\{$(x_{\\left\\lfloor kn/12 \\right\\rfloor},y_{\\left\\lfloor kn/12 \\right\\rfloor}) \\mid k = 1, 2, \\ldots, 12$\\}.\n\n> Dagger denotes to adopt all images from MSCOCO into training? How UniHPD can utilize the rest 40K MS COCO images without ref annotation?\n\nYes, dagger means that all images from MS COCO have been used in training, consistent with previous pose estimation works. Besides the images in RefHuman, we generated point and scribble prompt annotations for the remaining MS COCO images to support the training of UniPHD$^{\\dagger}$. We will release these annotations as well.\n\n>  I think * (namely choose the top-1) should be the default evaluation configuration. What is the evaluation configuration without * and why adopt it as default configuration?\n\nWe agree with the reviewer and will adopt $^*$ (top-1 ranked query group) as the default evaluation configuration in the final paper, as it reflects practical deployment where a model outputs a single result per input (L289). Some results are shown in Table A, our method still demonstrates top-tier performance under the suggested PCKh\\@0.5 and oIoU metrics.\n\nOur method without $^*$ uses 20 query groups for Pose AP and Mask AP evaluation (oIoU always uses only top-1), consistent with previous query-based pose estimation methods. For instance, GroupPose and ED-Pose use over 100 queries to enhance recall, despite most MSCOCO images contain fewer than three people. \n\n*Table A: Comparison on RefHuman using the top-1 query group* ($^*$).\n| | Pose PCKh\\@0.5 (Text) | Mask oIoU (Text) | Pose PCKh\\@0.5 (Scribble) | Mask oIoU (Scribble)  |\n|:-|:-:|:-:| :-:|:-:|\n| Uni-ED-Pose | 78.3  | 74.5 | 89.6  | 84.9  | \n| Uni-GroupPose  | 78.0 | 74.7 | 89.3  | 85.6  | \n| Ours  | **79.2** | **75.3** | **90.3** | **86.0** | \n\n> How about adopting multiple types of prompts to perform R-HPM? For example, using both text and point simultaneously to refer a specific person.\n\nInteresting idea! We sequentially cross-attend visual features with point and text prompts to enhance multimodal representations for R-HPM, achieving significantly improved results due to the complementary information from both prompts, as shown in Table B. Future research could use our introduced RefHuman dataset to develop more advanced models.\n\n*Table B: Ablation of using both text and point prompts.*\n| Prompts | Pose PCKh\\@0.5 | Mask oIoU  |\n|:-|:-:|:-:|\n| Text | 79.2 | 75.3 | \n| Point | 88.7 | 82.5 | \n| Text+Point | **91.4** | **86.6** | \n\n> PCKh\\@0.5 and IoU are more suitable as R-HPM metrics.\n\nThank you for this insightful advice. We have reported the overall IoU for segmentation evaluation, consistent with the referring segmentation task, and will adopt PCKh\\@0.5 as the primary pose estimation metric in the revised paper. Please kindly refer to Table A in the previous response. Our model still demonstrates top-tier performance under PCKh\\@0.5 and oIoU metrics.\n\n\n> Re-implement some referring segmentation methods to test their performance on RefHuman.\n\nThanks for the suggestion. We trained popular open-sourced referring segmentation models on our dataset, with results in Table C. Using only text prompt to train our model, we achieve top-tier performance compared to competitors like SgMg and GRES. These results will be included in the revised paper.\n\n*Table C: Comparison to language-conditioned segmentation models.*\n| Method | Backbone | oIoU  |\n|:-|:-:|:-:|\n| LAVT [1] | Swin-T | 74.5 | \n| CGFormer [2] | Swin-T | 75.3 | \n| GRES [3] | Swin-T | 75.9 | \n| SgMg [4] | Swin-T | 75.9 | \n| Ours | Swin-T | **76.3** |\n\n[1] LAVT: Language-Aware Vision Transformer for Referring Image Segmentation, CVPR, 2022.\n\n[2] Contrastive Grouping with Transformer for Referring Image Segmentation, CVPR, 2023.\n\n[3] GRES: Generalized Referring Expression Segmentation, CVPR, 2023.\n\n[4] Spectrum-guided Multi-granularity Referring Video Object Segmentation, ICCV, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewers for their thoughtful and constructive feedback.

We are glad that **all** the reviewers recognize the importance of our newly proposed task of Referring Human Pose and Mask Estimation and appreciate the significance of our new dataset, RefHuman. The reviewers also agree that the proposed UniPHD method achieves top-tier performance on the RefHuman and MS COCO datasets.

We have conducted all additional ablations and experiments suggested by the reviewers. We will revise our manuscript according to all the comments and remain committed to continuous improvement. We eagerly await your final decision.

Meta-review: Following the rebuttal, this work received positive feedback, with two weak accepts and one borderline accept. The authors effectively addressed most of the reviewers' queries, leading two reviewers to raise their scores. The contribution of the paper, particularly the introduction of the R-HPM task and the RefHuman dataset—a substantial extension to MS COCO with 50K annotations—was well appreciated. This dataset is expected to enable further research in related research area of human-AI interactions. The authors are encouraged to incorporate the suggested ablations and enhance the presentation in line with the reviewers' comments.
Decision: Accept (poster)