Referring Human Pose and Mask Estimation In the Wild

Authors: bo  miao
mingtao  feng
zijie  wu
mohammed  bennamoun
yongsheng  gao
ajmal  mian
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 13:01:30

Abstract: We introduce Referring Human Pose and Mask Estimation (R-HPM) in the wild, where either a text or positional prompt specifies the person of interest in an image. This new task holds significant potential for human-centric applications such as assistive robotics and sports analysis. In contrast to previous works, R-HPM (i) ensures high-quality, identity-aware results corresponding to the referred person, and (ii) simultaneously predicts human pose and mask for a comprehensive representation. To achieve this, we introduce a large-scale dataset named RefHuman, which substantially extends the MS COCO dataset with additional text and positional prompt annotations. RefHuman includes over 50,000 annotated instances in the wild, each equipped with keypoint, mask, and prompt annotations. To enable prompt-conditioned estimation, we propose the first end-to-end promptable approach named UniPHD for R-HPM. UniPHD extracts multimodal representations and employs a proposed pose-centric hierarchical decoder to process (text or positional) instance queries and keypoint queries, producing results specific to the referred person. Extensive experiments demonstrate that UniPHD produces quality results based on user-friendly prompts and achieves top-tier performance on RefHuman val and MS COCO val2017.

Main paper: 
 2d human pose estimation in the wild: 
 Human pose estimation aims to localize keypoints of individuals in unconstrained environments. For a long period, two-stage approaches have dominated this field, generally divided into top-down and bottom-up methods. Top-down methods [9, 15, 34, 63, 79, 84] first detect and crop each person using an object detector, then perform pose estimation on these cropped instance-level images using a separate model. Although effective, the redundancy from the extra detection step, region of interest operations, and separate training makes them suboptimal. Bottom-Up methods [3, 10, 31, 62] first detect abundant instance-agnostic keypoints and then group them into individual poses.While generally efficient, their intricate grouping algorithms pose challenges in handling complex scenarios, resulting in inferior performance. Furthermore, these two-stage approaches suffer from non-differentiable, hand-crafted post-processing steps that challenge optimization. Inspired by the one-stage object detectors [21, 76] , pixel-wise regression methods [49, 53, 55, 64, 73, 75, 78, 81, 93] densely predict pose candidates in an end-to-end fashion and apply Non-maximum Suppression (NMS) to obtain poses for different individuals. However, these methods produce redundant results, challenging the removal of duplicates.Recent human pose estimation methods [51, 52, 72] explore transformer-based architectures [2, 4, 77, 95] due to their sparse, end-to-end design and promising performance. These methods treat human pose estimation as a direct set prediction problem and use bipartite matching to establish one-to-one instance correspondence during training. Among these, [70] proposes a transformer with a pose decoder to predict keypoints and a joint decoder to refine them. [85] performs estimation on extracted object features to reduce noisy context. [85, 87] incorporate a human detection stage for query initialization to enhance performance. These methods achieve favorable performance but require complex strategies to identify the best match for a specified person during deployment due to the lack of exploration of prompt reasoning. In this work, we propose a multimodal reasoning task to directly and jointly predict the identity-aware pose and mask for a referred person, resulting in comprehensive human body representations and facilitating applications in human-AI interaction.Referring image segmentation (RIS) aims to segment target objects in images based on natural linguistic descriptions [18, 30, 91] . It related tasks include interactive image segmentation [7, 8, 25, 37, 41, 86] , which segments targets based on user clicks. Early RIS methods [6, 17, 18, 35, 43, 54, 71, 90] employ convolution and recurrent neural networks for multimodal encoding and segmentation. Their intrinsic constraints in capturing long-range dependencies and handling free-form features often lead to suboptimal performance. To improve multimodal representation and alignment, attentionbased bidirectional [19, 20] and progressive [16, 22, 23, 88, 89] cross-modal interaction modules are proposed. Additionally, [80, 92] leverage the strong cross-modal alignment capabilities of pre-trained large language models [67, 69] .With advancements in transformer architectures [4, 61, 77, 95] , [11, 13, 29, 38, 42, 45, 58, 60, 83, 94] design end-to-end transformers for language-conditioned segmentation and achieve favorable performance. In this work, we investigate a unified promptable transformer that effectively processes both text and positional prompts, thereby broadening its generalizability and application scope. Moreover, our model coordinates prompts with keypoint queries, enabling the joint prediction of keypoint positions and segmentation masks for specified individuals. 
refhuman dataset: 
 We substantially extend COCO [40] to construct the RefHuman dataset. It contains pose and mask annotations for humans along with text and positional prompts to facilitate the new task of R-HPM.Describe the person within the red bbox.Opt1: A person on far rightDescribe the appearance of the person. Pioneer works on 2D human pose estimation datasets in RGB images include [5, 12, 14, 26, 28] .Recent efforts focus on human pose estimation 'in the wild', establishing datasets such as MPII [1] , MS COCO [40] , CrowdPose [33] , COCO-WholeBody [27] , and AI Challenger [82] . Despite their prevalence, these datasets lack crucial text and scribble prompt annotations needed for human-AI interaction. The large-scale RefHuman dataset extends the widely-used MS COCO with additional text and positional (scribbles and points) prompt annotations, facilitating promptable models to enhance human-AI interaction.Text Prompt Annotation. Manually annotating text descriptions is costly. To mitigate this, we design a human-in-the-loop text generation strategy using the large language model, ChatGPT [65] . As shown in Figure 2 (a), we first provide the entire image with a bounding box indication to GPT-4 to generate a text description [Opt1] with global context. However, large language models often misidentify targets in complex scenes. To handle this, we crop the image to focus on the target person and let GPT-4 generate a description [Opt2] with correct local details. Given that individuals may have similar appearances, descriptions generated based on instance-level images are often not distinguishable. Therefore, we combine the global and local descriptions, integrating them into a comprehensive description [Opt3] of the target person. Finally, we manually select and revise these automatically generated descriptions to create accurate text prompts.Our strategy successfully generates acceptable descriptions for images with simple scenarios. These correspond to over 35% of the cases. In the remaining cases, unsatisfactory descriptions are generated. EQUATIONwhere b i,n (t) represents the Bernstein basis polynomials and n i is the binomial coefficient. To generate scribble prompts, we randomly sample four control points within the foreground mask to fit a cubic Bézier curve, represented as an ordered set of points {(x 1 , y 1 ), (x 2 , y 2 ), ..., (x n , y n )}. The curve is then discretized by uniformly sampling twelve points to form a scribble prompt s={(x ⌊ kn 12 ⌋ , y ⌊ kn 12 ⌋ ) | k = 1, 2, . . . , 12}. For point prompts, we randomly select a single point p=(x, y) from s, where x and y are the horizontal and vertical coordinates.Table 1 : Data statistics of human-related images in RefCOCO [30] , RefCOCO+ [30] , RefCOCOg [50] , their combined dataset RefCOCO/+/g, and our RefHuman. As shown in Table 1 , RefHuman includes 50,210 human instances across 21,634 images, larger than RefCOCO [30] , RefCOCO+ [30] , and RefCOCOg [50] , and with additional positional prompts. To construct RefHuman train set, we annotate prompts for all humans in MS COCO train2017 set with at least three surrounding people, a minimum of eight visible keypoints, and an area ratio of at least 2%. For the RefHuman val set, we annotate humans in MS COCO val2017 set, excluding those with non-visible keypoints or an area ratio below 1%, as instances below this threshold are often not visually clear and difficult to describe accurately. Note that each instance may have multiple text, scribble, and point annotations, with each image-prompt pair treated as a separate sample. Given an image I and a text prompt T or positional prompt P, our task aims to simultaneously predict the keypoint positions K ∈ R N ×2 and binary segmentation mask M ∈ R H×W of the referred person, where N is the number of keypoints, and H and W are the spatial dimensions. To this end, we propose a fully end-to-end promptable UniPHD approach, as illustrated in Figure 3 . UniPHD consists of four key components: Backbone, Multimodal Encoder, Pose-centric Hierarchical Decoder, and Task-specific Prediction Heads. A small set of prompt-conditioned queries is employed to identify the referred person and estimate results. During inference, we directly output expected predictions from the highest-scoring query group without any post-processing.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: The introduced dataset and code are publicly available.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
backbone: 
 Visual Encoder. We start by using a visual encoder to extract multi-scale visual features and apply a point-wise convolution to reduce the channel dimension of features to D =256 for efficient multimodal interactions. Feature maps with downsampling rates of {8, 16, 32, 64} are then flattened and concatenated into tokenized visual representations F v . We adopt Swin Transformer [47] as the visual encoder in this work.Prompt Encoder. For a linguistic description with L words as a prompt, we use RoBERTa [46] to extract word-level features F w ∈ R L×D and generate sentence-level feature as the instance query Q I ∈ R 1×D by pooling F w . For a discretized scribble or a single point prompt, we retrieve embeddings directly from visual features at stride 16 based on their positions to obtain pixel-level prompt features F w , which are then pooled to form the instance query Q I .To generate target-related multimodal representations F vl , visual tokens F v are first enhanced through a modality-specific cross-attention, which integrate information from prompt features:EQUATIONwhere i = 0 for text prompt and i = 1 for positional prompts, i.e., we adopt separate parameters for cross-modal fusion of different prompt types. After that, we follow previous works [44, 58, 87] to use a memory-efficient deformable transformer encoder to encode the multimodal features. The output of the transformer encoder F vl is then forwarded to the decoder to update queries.We employ n groups of queries for the referred human and generate n groups of results. Each group consists of a prompt-conditioned instance query and k learnable keypoint queries. The keypoint queries regard each keypoint as a target and aim to regress their respective positions, while each instance query, conditioned on the prompt, predicts a confidence score, dynamic segmentation filters, and a bounding box to locate the referred human. The scores are supervised by the losses of each query group to indicate result quality. During deployment, we directly generate results for the target using the highest-scoring query group without any post-processing.Prompt-conditioned Query Initialization. We first construct a query group template Q IP Init ∈ R (k+1)×D by concatenating the text/positional prompt embedding Q I with k (e.g., 17) randomly initialized learnable embeddings Q P . To generate n query groups, we apply linear layers to the prompt-aware multimodal features F vl to identify the top-n highest-scoring positions p and estimate their corresponding keypoint positions and centers as reference points, denoted as c ∈ R n×(k+1)×2 . The template Q IP Init is then repeated n times and enhanced with reference points c, their associated positional embeddings, and the top-n highest-scoring multimodal embeddings F vl p , to form Q IP . Finally, the instance query in each copy incorporates the prompt embedding and a positional embedding derived from the keypoints center of each candidate for pose-centric decoding. During decoding process, we employ graph-attention to model global dependencies for each query group, enabling all queries to utilize the prompt as guidance to identify the referred person only.Hierarchical Local Context and Global Dependency Modeling. We introduce a pose-centric hierarchical decoder to effectively model complementary local details and global dependencies for target-aware decoding. As illustrated in Figure 3 , local details are first aggregated using the efficient deformable attention [95] , similar to [44, 58, 87] . Each instance and keypoint query in Q IP independently predicts its sampling offsets and corresponding attention weights on the multimodal features F vl . We then aggregate the sampled features accordingly for each query to capture local details. However, after local detail aggregation, the keypoint queries struggle to perceive the prompt information and lacks interactions with each other, challenging the target-awareness and instance coherence. Additionally, the instance query lacks sufficient relevant global context to accurately locate the referred person at pixel-level.To address these issues, we model each query group as a bipartite graph G = {V, E, A}, with nodes V representing different queries and edges E denoting relations between nodes, constrained by a learnable soft adjacent matrix A ∈ R (k+1)×(k+1) , which simulates inherent keypoint-to-keypoint, keypoint-to-instance, and instance-to-keypoint relations. The edge E ij from the i-th node to the j-th node can be formulated as:EQUATIONwhere W q and W k are learnable projection matrices. Through our graph attention, both instance and keypoint queries capture global dependencies of the target and thus ensure instance coherence.Ultimately, the instance query generates dynamic filters for mask prediction, while the keypoint queries regress target keypoint positions.As illustrate in Figure 3 , four lightweight heads are built on top of the decoded queries to predict bounding boxes, class scores, masks, and keypoint positions for the target. The box head predicts the bounding box location of the target to aid the learning process. The class head outputs confidence scores, supervised by losses from each query group, to indicate the prediction quality of each query group. The mask head produces dynamic filters for conditional segmentation on the multimodal features F vl . The keypoint head predicts keypoint positions and their visibility for the referred person in images. Detailed training loss functions for these outputs are supplied in the Appendix.Conditional Segmentation. After extracting semantically-rich multimodal features F vl , we address pixel-level target localization using dynamic filters generated by instance queries, which capture both local details and global dependencies. Similar to [58] , we standardize the resolution of the multi-scale multimodal features F vl to H/8 × W/8 and combine them into a single feature map F m through efficient element-wise addition. We then reshape the prompt-conditioned dynamic filters to form two point-wise convolutions that apply to F m to obtain the segmentation mask for the referred person. 
experiments: 
 Metrics. We use standard metrics to evaluate our task. For pose estimation, we adopt OKS-based average precision (AP) and PCKh at a 0.5 threshold (PCKh@0.5) [1] . For segmentation, we report mask AP and overall Intersection-over-Union (IoU). The AP metrics are evaluated across all query groups, consistent with prior works [44, 87] , while PCKh@0.5 and oIoU are measured using only the highest-scoring query group to better reflect real-world deployment.Implementation Details. We train our model on the RefHuman train, which contains approximately 46K person instances with 17 keypoints per instance, and evaluate on the RefHuman val. We also evaluate our model on MS COCO by generating positional prompts for all images in the dataset. Due to the page limit, we leave the further implementation details in the Appendix.In Table 2 , we evaluate our models and the recent advances [44, 87] on the RefHuman val set. For fairness, all models are evaluated with inputs resized to a maximum of 640 pixels on the longer side.Effectiveness of Our Promptable Paradigm. Recent human pose estimation methods like Group-Pose [87] and ED-Pose [44] predict poses for multiple humans but require hand engineered selection strategies to identify the best match for a specified person, which can result in suboptimal outcomes or false negatives. By integrating their decoders into our end-to-end promptable paradigm, we directly generate poses and masks in one go for the referred person, achieving up to 73.0 pose AP, 89.6 PCKh@0.5, and 85.6 oIoU with scribble prompts. This performance rivals that of previous models which use 3× more training data and hand engineered result selection strategies. Intersection-based result selection is chosen in Table 2 because distance-and IoU-based strategies lead to inferior performance, as discussed in the ablation study. Furthermore, our paradigm achieves advanced performance even with a single point prompt, while previous models struggle with such simple guidance, as the random positive points/clicks can be near poses of unintended humans in crowds. These results demonstrate the effectiveness of our end-to-end promptable paradigm and the significance of our proposed R-HPM task, i.e., accurate joint pose and mask estimation based on user-friendly prompts. Effectiveness of Our UniPHD Approach. In Table 2 , our UniPHD approach sets a new benchmark in overall performance for R-HPM through pose-centric hierarchical decoding, outperforming the nearest competitor, Uni-GroupPose w/ Scribble, by 1.7 Pose AP and 1.0 Mask AP, with comparable FPS. Intuitively, the interdependent influence of human keypoints makes graph networks suitable for modeling their dependencies. After applying deformable attention to capture local details, we employ graph attention with a learnable soft adjacent matrix to simulate keypoint-to-keypoint, keypointto-instance, and instance-to-keypoint relations. This matrix guides edge construction to effectively model global dependencies and ensure instance-awareness and coherence.Results for Different Prompts. Table 2 shows that scribble prompts outperform point and text prompts in R-HPM. Scribble prompts offer more explicit positional guidance, enhancing instance query robustness. Point prompts, while slightly less accurate, offer greater user convenience in human-AI interaction due to their single-click simplicity. Text prompts face challenges in multimodal alignment, especially for crowded scenarios, but provide linguistic flexibility and enable non-physical interactions. Overall, our unified model effectively handles various prompts, showing great potential to facilitate human-AI interaction.Evaluation on MS COCO val2017. We further evaluate UniPHD on MS COCO by aggregating results from all instances in each image. As shown in Table 3 , all models are evaluated using inputs resized to a maximum of 640 pixels on the longest side for comparison. Trained solely with positional prompts, UniPHD achieves state-of-the-art performance on COCO val2017, even when compared Swin-T 74.5 CGFormer [74] Swin-T 75.3 ReLA [42] Swin-T 75.9 SgMg [58] Swin-T 75.9 Ours Swin-T 76.3Table 5 : Ablation of result selection strategies for GroupPose [44] with Swin-T. to competitors using higher-resolution inputs, further demonstrating the efficacy of our approach and the significance of the proposed task.Comparison with Text-based Segmentation Methods. To further validate the effectiveness of UniPHD in segmentation, we compare it with popular open-source text-based segmentation methods trained on RefHuman, as shown in Table 4 . Using only text prompts for training, our model achieves superior performance compared to competitive methods like SgMg [58] and ReLA [42] .Qualitative Results. In Figure 4 , we present qualitative results of UniPHD in various challenging scenarios such as crowded scenes, occlusions, similar appearances, dim lighting, and viewpoint changes. UniPHD effectively captures appearance, location, action, and context information to generate high-quality outputs. More visualizations are supplied in the Appendix.Result Selection Strategies. Table 5 shows the performance of a recent human pose estimation model [44] using various result selection strategies on RefHuman val. Without carefully designed strategies, the model struggles to accurately predict target individuals. The distance-based strategy fails to effectively filter out high-scoring irrelevant results, while the intersection-based strategy delivers the best performance.Multi-task Learning. In Table 6 , we evaluate the impact of multi-task learning in our approach on RefHuman val using the AP metric. Removing either head adversely affects the performance of the other. This validates the effectiveness of our decoder, which enables bidirectional information flow between keypoint and instance queries to enhance both predictions. Pose-centric Hierarchical Decoder. In Table 7 , we analyze the impact of global dependency modeling in our decoder. Without global dependencies, keypoint queries struggle to perceive the prompts for predicting target-aware results. Our graph attention clearly outperforms self-attention because it not only models dynamic node relations but also simulates inherent keypoint-to-keypoint, keypoint-to-instance, and instance-to-keypoint relations via soft adjacent matrix.Query Initialization. Similar to recent transformer-based pose estimation methods [44, 87] , we use prompt-conditioned query initialization to enrich queries with dynamic spatial priors. Table 8 reveals that omitting these dynamic priors results in a performance decrease of 0.4-1.2% AP. Nonetheless, our model maintains robust performance without query initialization, demonstrating its efficacy.Training with Extra Data. To unleash the capability of UniPHD for positional prompt-based prediction, we expand our training data beyond RefHuman to encompass the entire MS COCO train2017 images by generating additional point and scribble prompt annotations based on Bézier curves. As shown in Table 9 , UniPHD, trained exclusively with positional prompts on the expanded dataset, achieves impressive performance on RefHuman val, with up to 94.9 PCKh@0.5 and 89.4 oIoU, demonstrating its scalability. 
conclusion: 
 We introduced Referring Human Pose and Mask Estimation in the wild, a new task aimed at simultaneously predicting the poses and masks of specified individuals using natural, user-friendly text or positional prompts. This task holds significant potential for enhancing human-AI interactions in fields such as assistive robotics and sports analysis. To achieve this, we introduced the RefHuman dataset that substantially extends MS COCO with additional text and positional prompt annotations. We also proposed the first end-to-end promptable approach named UniPHD, which employs posecentric hierarchical decoder to model local details and global dependencies for R-HPM. UniPHD achieves top-tier performance, establishing a solid benchmark for future advancements in this field. We hope our new task, dataset, and approach could foster advancements in human-AI interaction and related areas.Broader Impacts. Malicious use of R-HPM models can bring potential negative societal impacts, such as unauthorized mass surveillance. We believe that the model itself is neutral with positive human-centric applications, including assistive robotics and sports analysis.Limitations. The proposed UniPHD approach produces high-quality results with various prompts.However, text prompts perform worse than positional prompts due to misidentification. Future research could enhance vision-language alignment to reduce this performance disparity.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: See Sec. 5 and the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We follow the usual format used in previous related works to report and compare the results.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: See Sec. 6.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the creators or original papers of all the related code, data, and models used in this paper. All the assets are free for research study and widely used in previous related works. We also provide the licenses of the used datasets in the Appendix.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
: 
 1 IntroductionHuman pose estimation in the wild is a fundamental yet challenging problem in the vision community, fueling advancements in various applications like human-AI interaction, activity analysis, video surveillance, assistive robotics, and sports analysis. This task aims to locate keypoints (joint locations) of humans within images in unconstrained environments.Previous multi-person pose estimation techniques typically follow a two-stage paradigm, separating the problem into person detection and local keypoint regression. These techniques can be summarized as top-down and bottom-up approaches. Top-down approaches [9, 15, 63, 79, 84 ] use a detection model to identify human bounding boxes and a separate pose estimation model to predict keypoints on the cropped single-human image. The independent models in these methods lead to a non-endto-end pipeline with substantial computational costs. Bottom-up approaches [3, 10, 31, 62] usually predict keypoint heatmaps to obtain instance-agnostic keypoints and assign them to individuals using heuristic grouping algorithms. The intricate grouping algorithms in these approaches introduce hand-designed parameters and face challenges in handling complex scenarios such as occlusions.With advancements in attention mechanisms [2, 56, 57, 59, 77] and transformer architectures [4, 95] , many recent approaches regard human pose estimation as a direct set prediction problem and design end-to-end differentiable transformers, leading to notable improvements. They employ bipartite matching to establish one-to-one instance correspondence for the set of predictions, avoiding the need for post-processing in the training stage. Among them, PETR [70] proposes a transformer that progressively predicts keypoint positions through pose and keypoint decoders. QueryPose [85] , ED-Pose [87] , and GroupPose [44] further incorporate a human detection stage for instance feature extraction or query initialization to improve performance and expedite model convergence. They all use keypoint-level queries to capture local details for accurate pose estimation.Despite demonstrating favorable performance, these methods still require designed strategies to select the best match for a target individual during deployment, which can result in suboptimal outcomes or false negatives, and lack exploration of human-AI interaction to directly predict expected results based on natural prompts. Additionally, they overlook joint human pose and mask estimation, which provides comprehensive human representations to facilitate applications like assistive robotics and sports analysis. For example, accurate joint human pose and mask estimation in unconstrained environments enables robots to locate, analyze, and interact with target individuals, enhancing user experience and assistive tasks.In this paper, we propose the new task of Referring Human Pose and Mask Estimation (R-HPM) in the wild. As illustrated in Figure 1 , unlike multi-person pose estimation [3, 15, 24, 66] , R-HPM is a multimodal reasoning task that requires a unified model to predict both pose and mask of a referred individual using user-friendly text or positional prompts, enabling comprehensive and identityaware human representations without post-processing. To achieve this, we introduce RefHuman, a large-scale dataset that substantially extends MS COCO [40] with additional text, scribble, and point prompt annotations. Our dataset accommodates diverse task settings to enhance human-AI interaction. Manually annotating text descriptions and scribbles is expensive. To reduce annotation costs, we design a human-in-the-loop text generation strategy using powerful large language models and employ bezier curves to automate scribble generation.To benchmark R-HPM, we propose an end-to-end promptable approach called UniPHD. Our approach directly adopts prompts as instance queries and coordinates them with keypoint queries to jointly predict identity-aware keypoint positions and masks for the target. Unlike previous works [44, 70, 85, 87] , UniPHD introduces a pose-centric hierarchical decoder (PHD) that employs deformable and graph attention to effectively model local details and global dependencies, ensuring target-awareness and instance coherence. Furthermore, our approach follows a general paradigm that enables seamless integration with the decoders from recent transformer-based methods such as GroupPose [44] and ED-Pose [87] for R-HPM. We conduct extensive experiments on the RefHuman dataset. Integrating our approach with GroupPose and ED-Pose yields promising results in an end-to-end fashion. Moreover, our UniPHD approach achieves top-tier performance on both RefHuman val and MS COCO val2017, demonstrating the effectiveness of our approach and the significance of our task.Our main contributions are summarized below:• We propose Referring Human Pose and Mask Estimation (R-HPM) in the wild, a new task that simultaneously predicts the pose and mask of a specified individual using natural, user-friendly text or positional prompts. This task enhances models with identity-awareness and produces comprehensive human representations to benefit human-AI interaction.• We introduce RefHuman, a large-scale dataset that substantially extends COCO for R-HPM. RefHuman contains pose and mask annotations for individuals in diverse, unconstrained environments and is enriched with corresponding text and positional prompts.• We propose an end-to-end promptable UniPHD approach for R-HPM. Our approach performs pose-centric hierarchical decoding, achieving top-ranked performance and establishing a solid benchmark for future advancements in this field.2 Related Work 
 
 Appendix: The overall loss function for UniPHD consists of four parts:EQUATIONwhere L box is for human box regression that contains L1 loss and GIOU [68] loss, L class is for classification includes focal loss [39] with α = 0.25 and γ = 2, L pose is for keypoint regression and visibility prediction that includes L1 loss, the constrained L1 loss-OKS loss [70] , and cross entropy loss, L mask is for segmentation that includes Dice [36] loss and focal loss. The loss coefficients [44, 58, 87] . The Hungarian algorithm [32] is used to identify the optimal assignment (query group) with the highest similarity to the ground truth for training. The class label for the optimal instance query with the minimum loss from the ground truth is set to one, while all others are set to zero.EQUATIONWe follow the common optimization strategies in [44, 58, 83, 87] to train the models. During training, we augment input images through random flip, random crop, and random resize with the shorter sides within the range of 360 to 640 pixels and the longer sides up to 640 pixels. For each iteration, we randomly select either a text or positional prompt with equal probability. We use the AdamW [48] optimizer with a weight decay of 1×10 -4 and train our models on 24GB RTX 3090 GPUs with batch size 16 for 20 epochs. The initial learning rates are set to 1×10 -5 for the visual encoder and 1×10 -4 for other components, with a rate decay at the 18th epoch by a factor of 10. Both the multimodal encoder and pose-centric hierarchical decoder consist of 6 layers, and we use 20 query groups for our models. During testing, we resize the input images with their longer sides up to 640 pixels.In Table 10 , our approach perform well across different numbers of query groups, measured by the AP metric. This hyperparameter primarily affects pose estimation, with 20 query groups identifying more candidates to achieve the best performance. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: Following the rebuttal, this work received positive feedback, with two weak accepts and one borderline accept. The authors effectively addressed most of the reviewers' queries, leading two reviewers to raise their scores. The contribution of the paper, particularly the introduction of the R-HPM task and the RefHuman dataset—a substantial extension to MS COCO with 50K annotations—was well appreciated. This dataset is expected to enable further research in related research area of human-AI interactions. The authors are encouraged to incorporate the suggested ablations and enhance the presentation in line with the reviewers' comments.

Reviews: review: paper_summary: paper_summary: This paper proposes a new task called Referring Human Pose and Mask Estimation and introduces the corresponding RefHuman dataset, which is beneficial for research on human behavior comprehension. Additionally, the authors present a model that leverages three types of prompts for this task. The proposed UniPHD model achieves promising performance in this area.
strength_weakness: strength_weakness: 1. The descriptions of the method, especially in Section 4.3, is confusing. For example, what is the relationship between F^(vl) and P’? How do you enhance the template based on P’? And why do “the keypoint queries struggle to perceive the prompt information and lacks interactions with each other, challenging the target-awareness and instance coherence” ? It seems that these keypoint queries can interact with each other in existing decoders. More detailed descriptions and explanations are needed to understand this work.

2. The author proposed that existing research overlooks joint human pose and mask estimation, which provides comprehensive human representations. However, the paper lacks comparisons of the proposed model with other variants, such as UniPHD without the mask head and UniPHD without the pose head. Such comparisons could help clarify the relationship and benefit of the pose estimation and mask estimation tasks.

3. The proposed method needs further validation. Providing an ablation study of the proposed query initialization method and comparing model parameters and computational complexity would be helpful.
questions: questions: See the weaknesses
limitations: limitations: Yes
paper_summary: paper_summary: In this paper the authors tackle the problem of in-the-wild human pose estimation in a “referring” setting where the goal is to determine the pose of a person referred to using either a text prompt or positional prompt. To achieve this, the authors annotate MS COCO dataset with over 50K annotated instances for 2D keypoints, mask  and prompt (either as text, points or scribbles). They use this dataset to train a model called UniPHD which consists of several submodules. The results show the paper is able to train a strong baseline model which will be useful for future research.
strength_weakness: strength_weakness: W1. The motivation for this obtaining pose in a “referring” manner is unclear. For instance, it might be more valuable to focus a text based human detection as opposed to pose estimation because the former can open up many avenues such as human tracking for research. 

W2. Multimodal encoder is shallow potentially limiting the interaction of the different encoded features. It might be helpful increase the capacity of the the multimodal encoder model.

W3. Parts of the paper not are fully clear. I understand that the paper aims to be “single-stage” but on L205-L214, there seem to be multiple stages where candidates are detected and then used for subsequent pose estimation. (Which makes it two-stage?)

Minor comments

* Table. 1 does not require last column
* Table. 3 typo “Scibble" —> “Scribble”
questions: questions: Q1. How does Eq. 4 fit into mask prediction task?  Is that applied only to the pose prediction branch?
limitations: limitations: The authors have briefly acknowledged limitations of their method.
paper_summary: paper_summary: This paper introduces a new task named as Referring Human Pose and Mask Estimation (R-HPM), which adopt text/point/scribble to represent a specific person and estimate its pose and segmentation mask. To achieve this goal, this paper proposes a new R-HPM dataset named RefHuman and a new method UniPHD to perform R-HPM. Experiments on the proposed dataset and MSCOCO demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1.	The definition of point and scribble in Sec.3.1 is not clear. Does point prompt only contains only one point and scribble contains 12 points? How to define the point in point prompt? This paper should give a form definition, not textual description.
2.	Some confusion in experiments. First, what is the meaning of dagger in Table 2. If I understand correct, dagger denotes to adopt all images from MSCOCO (~60K) into training, but RefHuman only contains 20K images, how UniHPD can utilize the rest 40K images without ref annotation? Second, I think * should be the default evaluation configuration (namely choose the top-1), so what is the evaluation configuration without * and why adopt it as default configuration?
3.	How about adopting multiple types of prompts to perform R-HPM? For example, using both text and point simultaneously to refer a specific person.
4.	Evaluation metric is not suitable to R-HPM. This paper adopts AP to evaluate performance, which is designed to evaluate multiple objects in one image. However, in referring setting, only single instance is involved, therefore single instance evaluation metric such as PCKh@0.5 is more suitable for R-HPM, or just evaluating the keypoint error/segmentation IoU is ok.
5.	To establish a comprehensive benchmark, the authors are expected to test some existing methods so that the following work can take as reference. None of the compared methods in Table 2 are referring-based methods, so this paper should reimplement some referring-based methods to test their performance on RefHuman, e.g.,  some referring segmentation methods.
questions: questions: See Weakness.

Overall, I think the motivation and contribution of this work is pretty good. But there are still some questions that should be clarified in the revised paper.
limitations: limitations: This paper already discusses limitation in Sec.6.

Comments: title: Post-rebuttal
comment: Dear Reviewer apkP,

We greatly appreciate your recognition of our work's contributions and the upgraded score. Your valuable comments have been crucial in refining our paper. As suggested, we will further improve the presentation and soundness in the final version.
The rebuttal has addressed some of the concerns. Hence, I increase the score to a weak accept. I continue to think that there is room to improve the presentation and soundness of the paper, but the contributions are valuable.
Dear Reviewer TbW6,

Thank you for your positive feedback! We are glad that our rebuttal has addressed your concerns. Your constructive comments have been very helpful in refining our work, and we will incorporate these additional results in the final paper.
I appreciate the authors' response. The additional explanations and ablation studies improve the quality of the paper and effectively demonstrate the design's effectiveness. As a result, I have raised the score.
Dear Reviewer DJJG,

Thank you for your diligent review of our submission. We have carefully addressed each of your concerns and provided our responses. We would greatly appreciate any additional comments, as your feedback is crucial in strengthening our work.

Your time and consideration are invaluable to us.
Dear Reviewer apkP,

Thank you for your diligent review of our submission. We have carefully addressed each of your concerns and provided our responses. We would greatly appreciate any additional comments, as your feedback is crucial in strengthening our work.

Your time and consideration are invaluable to us.
Dear Reviewer TbW6,

Thank you for your diligent review of our submission. We have carefully addressed each of your concerns and provided our responses. We would greatly appreciate any additional comments, as your feedback is crucial in strengthening our work. 

Your time and consideration are invaluable to us.
Dear Reviewers,

Thank you for your diligent effort in reviewing our submission! We have carefully addressed the concerns raised and conducted the requested experiments. We would greatly appreciate any additional comments and are ready to engage in further discussion. If there are no further issues, we respectfully request that you consider revising the score for our paper in light of the improvements we have made.

Thank you again for your time and consideration.