Seeing Beyond the Crop: Using Language Priors for Out-of-Bounding Box Keypoint Prediction

Authors: bavesh  balaji
jerrin  bright
yuhao  chen
sirisha  rambhatla
john s zelek
david anthony clausi
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 16:20:42

Abstract: Accurate estimation of human pose and the pose of interacting objects, like a hockey stick, is crucial for action recognition and performance analysis, particularly in sports. Existing methods capture the object along with the human in the bounding boxes, assuming all keypoints are visible within the bounding box. This necessitates larger bounding boxes to capture the object, introducing unnecessary visual features and hindering performance in real-world cluttered environments. We propose a simple image and text-based multimodal solution TokenCLIPose that addresses this limitation. Our approach focuses solely on human keypoints within the bounding box, treating objects as unseen. TokenCLIPose leverages the rich semantic representations endowed by language for inducing keypoint-specific context, even for occluded keypoints. We evaluate the performance of TokenCLIPose on a real-world Ice-Hockey dataset, and demonstrate its generalizability through zero-shot transfer to a smaller Lacrosse dataset. Additionally, we showcase its flexibility on CrowdPose, a popular occlusion benchmark with keypoints within the bounding box. Our method significantly improves over state-of-the-art approaches on all three datasets, with gains of 4.36\%, 2.35\%, and 3.8\%, respectively.

Main paper: 
 introduction: 
 The goal of 2D human pose estimation is to localize the human anatomical keypoints from an image, which is essential for scene understanding, action recognition [1, 2] , and human-object interaction detection [3, 4] . This is particularly challenging in cluttered real-world scenarios due to occlusions and other non-idealities [5, 6] . With the emerging applications in Virtual Reality (VR), and Augmented Reality (AR), and real-time sports analysis [7] , there is a fundamental need to understand how objects are manipulated via human-object interaction [8] . Often, in such scenarios, the objects that humans hold and interact with, which we define as extensions, can provide crucial information that aids in accurately estimating the human pose and the actions being performed [1] .Contemporary SOTA deep learning-based pose estimation methods predominantly follow a top-down approach: cropping each person in an image using bounding boxes before estimating their pose individually [9] [10] [11] [12] [13] [14] [15] [16] . While using existing top-down pose estimators seems intuitive for joint prediction of the humans and their extensions, this approach suffers from limitations, yielding suboptimal results as shown in Fig. 1(a) .Our key observation is: capturing the extension in the bounding box expands the field-of-view and introduces unnecessary visual features, which can be confusing to the model. A simple yet powerful fix is to confine the bounding box to capture the human body, treating the extension's keypoints as unseen. This approach reduces background interference, as we do not explicitly capture the extension. However, it leads to the loss of important visual information about the extension, making them unseen. From the figure, it is evident that while these embeddings maintain positional structure within the upper-body joints (the shoulders are placed above the elbows which are placed above the wrists, following the pose of a human standing normally) and the lower-body joints (knees placed above ankles), they fail to maintain positional structure between the upper-body and lower-body joints (elbows and wrists are placed below knees and ankles in the plot which does not follow the pose of a human being).Here, a crucial question arises: How can we effectively represent the spatial relationships of these unseen keypoints for accurate pose estimation?To answer this, we turn to other ways of informing the model about the unseen keypoints. Recent works have shown that using language to induce semantic context of keypoints can lead to effective feature representations [17] [18] [19] [20] . Specifically, existing works [18] on human pose estimation align the image features with keypoint-specific text embeddings generated from Vision Language Models (VLMs) using a contrastive loss. However, these text embeddings primarily capture local details, neglecting the crucial global relationship between lower-body and upper-body joints; In Fig. 1(b ) language models encode similar joints together, while losing the global structure. Hence, explicitly imposing the image features to be close to text embeddings could be suboptimal.Based on the observations, we present a simple yet effective solution to significantly improve reliability under dynamic real-world scenarios by leveraging language to see beyond the bounding box. Specifically, We utilize these text embeddings as priors and initialize our learnable keypoint tokens (referred to as text tokens) using these text embeddings. By integrating the rich semantic representations of keypoint-specific text embeddings with image features, and employing a transformer to capture global dependencies, we extract superior fine-grained representations which significantly boosts the performance across the board.We evaluate TokenCLIPose's performance on three real-world datasets containing a lot of occlusions and noise: an ice hockey dataset, a Lacrosse Dataset and the CrowdPose dataset [6] . The ice hockey and Lacrosse datasets are first-of-their-kind datasets that we curated for predicting the pose of human extensions (the sticks). Furthermore, in order to demonstrate the flexibility of TokenCLIPose in predicting unseen keypoints that are present within the bounding box, we evaluate it on the CrowdPose dataset. TokenCLIPose outperforms existing top-down approaches by 4.36% and 3.8% on the ice hockey and CrowdPose datasets respectively. Furthermore, TokenCLIPose demonstrates superior zero-shot capabilities in predicting extension keypoints when tested on the Lacrosse dataset, outperforming prior works by 2.35%. Our experiments highlight TokenCLIPose's ability to reliably predict unseen keypoints. 
related work: 
 2D Human Pose Estimation: Top-down approaches in 2D pose estimation can be broadly classified into two categories: Heatmap-based pose estimation has been the de-facto standard approach since the introduction of stacked hourglass networks [14] . These methods represent discrete (x, y) coordinates as continuous heatmaps where each pixel indicates the likelihood of a specific joint being located at that position. Most methods [9, 12, 14, 13] rely on powerful convolutional networks to extract highlevel multi-scale feature maps. While most research focuses on network architectures, a few works investigate the coordinate representation and the heatmap encoding and decoding process [21, 15] . Recently, researchers have begun exploring transformer-based architectures for pose estimation [10, 22, 11, 23] . Xu et al. [10] adopt the original vision transformer [24] and build baselines for pose estimation, showcasing the efficacy of vision transformers. Methods including [22, 11] use CNNs as feature extractors, and utilize transformers to model the relationship between different scale features and the keypoint features respectively.Regression-based pose estimation, in contrast to the dominance of heatmap-based approaches offers an alternative paradigm. Sun et al. [25] leverages a convolutional backbone to extract feature maps and then utilize an integral operation to directly regress keypoint coordinates. Li et al. [26] developed a novel pose regression model which aims at minimizing the distance between the predicted and underlying distribution. However, the global pooling operation used in [26] results in loss of spatial information that is crucial for reliable pose estimation. More recent works [27, 16, 28] include transformer-based architectures: [27, 16] use an encoder-decoder strategy and are based on DETR [29] and Deformable-DETR [30] respectively. While heatmap-based methods have achieved high accuracy, they have some limitations: 1) These methods have a non-differentiable heatmap decoding method; 2) Heatmap representation often leads to quantization error; and 3) They are not designed to predict out-of-bounding-box keypoints. Therefore, we adopt the regression-based approach for directly estimating the coordinates of all joints, both inside and outside the bounding box.Occlusion-Aware Pose Estimation: Several approaches tackle the problem of occlusions in crowded scenarios. Various methods tackle the problem by studying the relationships between multiple humans present in an image [31] [32] [33] . However, we do not compare with their works as we are interested in single-instance approaches. Park et al. [34] addresses the problem of out-of-bounding box keypoints by refining the bounding box before pose prediction, but this adds computational cost. Our approach, on the other hand, provides a parameter-free technique to induce spatial context for the out-of-bounding box keypoints by leveraging the knowledge of VLMs.Vision-Language Models: Language supervision has been shown to improve feature representations in various vision tasks, such as image classification, semantic segmentation, and pose estimation. Radford et al. [35] proposed the contrastive pretraining paradigm CLIP that leverages contrastive learning to optimize a text and image encoder jointly. This work also showcased the significance of large-scale vision-language pretraining by demonstrating accurate zero-shot classification. Following CLIP, various works such as [36] [37] [38] focused on improving image classification using CLIP. Zhang et al. [39] transfer the 2D pre-trained knowledge to 3D domains, thereby improving zero-shot pointcloud recognition. Rao et al. [40] showcased the efficacy of CLIP pretraining on dense prediction tasks by converting the image-text matching problem to a pixel-text matching problem.Recently, Tevet et al. [20] leverage text to inpaint missing poses in a sequence in a spatiotemporally consistent manner. Guo et al. [19] use pose-aware prompts to predict 3D hand meshes from images. Recent works on 2D pose estimation [18, 17] utilize joint-specific keypoints to learn richer representations. Particularly, they use a contrastive loss to align the image features extracted from a backbone to their keypoint-specific text embeddings. This, however, is suboptimal as the text embeddings do not capture global structure between human keypoints. Hence, aligning the image features to text embeddings might lead to loss of spatial information and inaccurate localization of joints. We improve this by using a transformer-based network to capture the global spatial dependencies between image and text features, thereby guiding our image features using text supervision rather than biasing them to the text prompts themselves. These multimodal tokens are fed to a transformer decoder to capture spatial dependencies between them and predict all the 2D keypoints μd . The coarse human keypoint predictions and the final keypoint predictions are supervised using the RLE loss.x = f ϕ (z) G ϕ (x) Q(x) P ϕ (x) Residual Log-Likelihood x = x • σ + μ Fvis Floc Ftext3 Method 
problem formulation: 
 Given a cropped image I ∈ R h×w of a human, generated from bounding boxes obtained through a detection network, we predict K out keypoints μd ∈ R Kout×2 that represent the 2D poses of extensions and/or humans, along with the scale parameter σd for each keypoint.We propose an encoder-decoder architecture to estimate 2D keypoints that are not captured in the bounding box. Firstly, an image encoder extracts multi-scale image features from the cropped image, which are then passed through a Multi-layer Perceptron (MLP) to generate coarse human keypoint proposals. These image features and locations are then projected onto the multimodal embedding space to form image and location tokens. The text-based keypoint encoder leverages text prompts to generate keypoint-specific text tokens. Finally, all these multimodal tokens are concatenated and passed through a transformer decoder to capture global relationships between these tokens and predict final 2D keypoints. The network is exemplified in Fig. 2 .Image Encoder. The cropped input image is initially passed through a pretrained CNN to extract multi-scale dense feature maps. These multi-scale feature maps are fused and projected onto a joint multimodal embedding space to form the image tokens F vis ∈ R N ×C emb , where N is the number of tokens and C emb is the joint multimodal space dimension. Furthermore, they are processed through a MLP to generate coarse human keypoint predictions μf ∈ R K h ×2 , and a scale parameter σf ∈ R K h ×1 , which are optimized using the RLE process as detailed in Section 3.3.Text-based Keypoint Encoder. Following image feature extraction, the keypoint encoder tackles the challenge of unseen keypoints in the image. We employ language-guided keypoint representations, where text prompts encode class-related information of each keypoint. By leveraging CLIP's pretrained text encoder, we generate the text tokens F text ∈ R Kout×C emb which are then concatenated with the extracted image tokens. This process indirectly injects visual context of the missing keypoints into the model, even when it's not directly visible in the cropped input image. Additionally, we incorporate the coarse human keypoint predictions μf and convert them to location tokens by projecting them onto the joint mulitmodal space F loc ∈ R K h ×C emb , and concatenate them with the image and text tokens. The final set of tokens fed to the decoder are denoted asF = {F vis , F text , F loc }.Transformer Decoder. To predict all the keypoints precisely, the relationships between these multimodal tokens (F) must be captured accurately. Therefore, we leverage the transformer decoder to understand the inherent correlations between different text, image and location tokens. Instead of treating each modality separately and utilizing a cross-attention mechanism to understand the associations between them, we treat all tokens together as a homogeneous entity and employ the standard self-attention mechanism. This approach offers a simpler way to gain a holistic view of the relationships between all keypoints, locations, and image patches. The transformer layers are followed by MLPs to estimate the final keypoint predictions μd ∈ R Kout×2 and the scale parameter σd ∈ R Kout×1 . To reduce artifacts in feature maps and understand richer relationships, we employ additional register tokens as [41] .Distribution Learning. Following [26, 16] , we formulate the regression task as a distribution learning problem and adopt Maximum Likelihood Estimation (MLE) to effectively predict the output coordinates. We use normalizing flows to estimate the deviation in predicted and ground truth keypoint distributions. In mathematical terms, given an input image I, our network estimates a distribution P Θ,Φ (x|I) representing the probability of the ground truth keypoint appearing at the location x. Here, Θ and Φ denote the parameters of our pose network and the flow model f ϕ , respectively. The flow model f ϕ acts as a refinement step, iteratively transforming a preset Gaussian distribution z ∼ N (0, Id) to capture the deviation of the predicted and ground truth distributions using the network's prediction ( μ and σ).Equation ( 1) depicts the mathematical formulation of the RLE loss, where Q( μg ) is the preset Gaussian distribution, G ϕ ( μg ) is the learned distribution by the flow model, μg = (µ g -μ)/ σ represents the normalized difference between ground truth and predicted keypoints, and s is a constant term.EQUATIONSimilar to [16] , we supervise both the coarse predictions ( μf , σf ) and the final predictions from the decoder ( μd , σd ) using RLE. Hence, our final loss function isL = L f RLE + L d RLE (2)where, L f RLE = -log P Θ f ,Φ f (x|I) 
experiments: 
 A critical challenge in evaluating pose networks for human extensions is the lack of publicly available datasets containing both the human and extension pose annotations. To address this limitation, we curate two new sports datasets, ice hockey and Lacrosse, featuring pose annotations for humans and their corresponding extensions. Experiments on these datasets demonstrate the performance of our model in estimating the pose of extension keypoints, which are not captured within the bounding box.We further evaluate the model's ability to predict unseen keypoints (due to occlusion/self-occlusion) within the bounding box using the benchmarked multi-person cluttered dataset, Crowdpose [6] , validating the generalizability of our model.For human pose estimation, the total keypoints (K out ) depend on extensions. Without extensions, K out equals the number of human keypoints (K h ). When extensions are present, K out increases to K h plus the number of extension keypoints (K e ).Input Image HRNet-W48 [12] Ours Input Image HRNet-W48 [12] Ours Dataset. The ice hockey dataset, captured from real-world National Hockey League (NHL) videos, presents a challenging environment for pose estimation due to the inherent fast pace of the sport. Players' rapid movements often result in motion blur within frames, making it difficult to distinguish body parts. Furthermore, the nature of the game leads to frequent occlusions, especially when players obstruct each other's bodies. Adding to the complexity is the bulky equipment worn by hockey players, which can obscure keypoints. These combined challenges-motion blur, occlusion, and bulky equipment-make the ice hockey dataset a valuable resource for evaluating the robustness of the pose estimation task.Our dataset consists of 10 video clips (30-45 seconds each, sampled at 30 fps) from various broadcast NHL videos. We utilize the CVAT tool to annotate the keypoints for multiple players and their hockey sticks in each frame. We followed the standard COCO format to annotate 17 keypoints for the human pose. Additionally, we annotate 3 keypoints for the hockey stick (butt end, heel, and toe). Finally, for each annotated frame, we create two versions of the input image: one cropped to include only the player (standard bounding box) and another incorporating both the player and their hockey stick (extended bounding box). In total, we generate 11.66K pose annotations, with 9.13K images from 9 clips used for training and 2.53K images from the tenth clip used for testing our model's performance.Evaluation Metric. We evaluate different State-Of-The-Art (SOTA) pose estimation models on our dataset using the Percentage of Correct Keypoints with head-normalization (PCKh) metric. Due to our method's use of smaller bounding box compared to the other models, achieving the same level of accuracy would result in a higher PCKh threshold for our model. To ensure a fair and direct comparison across all models, we employ the same threshold for evaluation.Training. All networks are trained for 200 epochs with a batch size of 64 on a single NVIDIA GeForce RTX 4090 GPU. We employ the Adam optimizer with a learning rate of 6 × 10 -4 for all CNN-based architectures while the transformer-based architectures are trained using the AdamW optimizer with an initial learning rate of 3 × 10 -4 . The weight decay is set to 10 -5 for all models.A stepLR scheduler was used to linearly reduce the learning rate from the initial value of 10 -5 .Consistent with [26] , we utilize RealNVP [42] as the flow model within our model.Results. The results, presented in Table 1 , emphasize our method's SOTA performance over existing methods in robustly predicting the ice hockey player and the hockey stick keypoints, outperforming existing works by 4.36%. This is further validated by the qualitative comparisons in Fig. 5 , where our model demonstrates the ability to predict semantic poses even in challenging scenarios with extreme motion blur and occlusion. These results support our hypothesis that re-considering the extension pose estimation task as an unseen keypoint problem without explicitly capturing it in the bounding box reduces background noise leading to robust poses.Dataset. In order to study the efficacy of TokenCLIPose's generalization capabilities in predicting the pose of extensions, we curate another small-scale Lacrosse dataset. Similar to ice hockey, it is characterized by motion blur and occlusions, but there are domain differences between the 2 datasets. This dataset consists of 300 pose annotations from one video sampled at 30 fps. We use the same 12 human keypoints used in the ice hockey dataset to denote the human's pose. However, we use only 2 keypoints to represent the Lacrosse stick's pose, as the blade of a Lacrosse stick is circular in nature. Hence, we disregard the 15th keypoint (stick toe) and use the other 14 keypoints to represent the pose of a Lacrosse player along with their stick.Zero-shot Results. We evaluate TokenCLIPose's generalization capability by performing zero-shot transfer from our pretrained model on ice hockey to the Lacrosse dataset. Due to the difference in the shape of the head of a lacrosse and hockey stick, we predict the two keypoints corresponding to the shaft of a lacrosse stick. Furthermore, the text prompts for the two extension keypoints are also changed while keeping the model frozen. The results presented in Table 2 showcase that our model outperforms the established baselines by 2.35%, thereby demonstrating the efficacy of our proposed model for generalizable pose estimation.Dataset. The CrowdPose dataset is a large-scale benchmark dataset for human pose estimation, containing 12K images and 43.4K labeled people in their trainval set, and 8K images with 29K labeled people in the test set. Following [32, 18, 31] , we use the trainval set for training and test set for evaluation.Evaluation Metric. We adopt standard Average Precision (AP) as our evaluation metric on the CrowdPose dataset. AP is calculated based on Object Keypoint Similarity (OKS) denoted by m OKS ∈ R, which is defined asEQUATION)where di is the Euclidean distance between the i-th predicted keypoint coordinate and the corresponding ground truth, v i is the visibility flag of the keypoint, s is the object scale, and k i is a keypoint-specific constant.Training. We employ the AdamW optimizer with an initial learning rate of 6 × 10 -4 and weight decay of 0.1. Following ViTPose [10] , we apply linear warmup for the first 2000 iterations with a warmup factor of 10 -3 . Furthermore, we perform gradient clipping to prevent overfitting. Results. Table 3 presents a comparison between established pose estimation techniques and our method. As shown in the table, TokenCLIPose outperforms all the top-down approaches by 3.8% demonstrating its efficacy in predicting unseen keypoints robustly. We also depict the qualitative results of TokenCLIPose to depict its robustness to occlusion. It is notable that even in scenarios where we only see the side-view of humans, TokenCLIPPose estimates the pose reliably.We conduct comprehensive ablations to study the effect of our design choices and verify the impact of each module on our proposed TokenCLIPose. For consistency, all the ablations are conducted on our ice hockey dataset unless specified otherwise.Gains from Each Modality. The influence of each modality on the proposed model's performance is shown in Table 4 . As evidenced by a performance improvement of 3.35%, the inclusion of text tokens plays a significant role in enhancing the pose estimation accuracy. On the other hand, we find that including the location tokens do not improve the performance significantly, only by a small margin of 0.45%. 5 , utilizing self-attention directly on all the tokens produces similar results to performing self-attention separately on image, location and text tokens, and then applying cross-attention. Therefore, it is not necessary to treat the tokens from each modality separately. We hypothesize that this could be due to all the tokens being projected to the same joint embedding space, thereby eliminating the need to process them differently.Impact of Text Prompts. We investigate the influence of language on our model's efficacy on the ice hockey dataset by varying the degree and complexity of text prompts that we use. Starting from randomly initializing the text tokens instead of using CLIP embeddings, we study the effects of using single prompts and the prompt ensemble technique proposed in [35] for ImageNet classification. It is evident from Table 6 that incorporating text instead of randomly initializing text tokens results in a significant improvement of 2.31% in the overall accuracy. Furthermore, using prompt ensemble techniques improve the accuracy of the model by 1.04%, showcasing the importance of the quality of text prompts.Influence of Bounding Boxes. We evaluate the influence of bounding box predictions on our model's performance by using ground truth bounding boxes obtained from ground truth human poses and the bounding boxes from the FasterRCNN object detector. The results are showcased in Table 7 . The above table illustrates our model's robustness to the quality of bounding boxes, as using ground truth bounding boxes improves the model's performance by a small margin of 2.47%.Table 7 : Impact of Bounding Boxes on the overall accuracy.Faster-RCNN [47] 76.28 Ground Truth 78.75 ↑ 2.47% 
conclusion: 
 In this work, we proposed TokenCLIPose, an innovative solution for robustly predicting human and extension poses. Instead of explicitly modeling extensions within the bounding box, we reformulated the extension pose estimation as an unseen keypoint prediction problem. We leverage the power of large pre-trained VLMs in augmenting the spatial information of unseen keypoints. We showcased that capturing relationships between multimodal tokens is more effective than aligning image features to text tokens. To evaluate the effectiveness of TokenCLIPose in predicting extension keypoints, we curated real-world datasets for ice hockey and Lacrosse. We significantly outperform existing topdown methods on these datasets (by 4.36% and 2.35%, respectively). Additionally, we achieve a 3.8% improvement over prior top-down networks on the CrowdPose dataset. This shows TokenCLIPose's flexibility to predict unseen keypoints within the bounding box as well. Future work will focus on curating and training more extensive and diverse datasets for human and extension pose estimation tasks.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We do work with an institution which has full access to our work. However, with their permission, we could release the dataset and models soon.Guidelines:• The answer NA means that paper does not include experiments requiring code. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
acknowledgement: 
 This work was supported in part by Stathletes, Compute Canada, the Natural Sciences and Engineering Research Council of Canada and MITACS. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: We conform to the NeurIPS code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Our work is strictly restricted to 2D pose estimation which is mainly used for scene understanding, activity recognition and sports analysis. Hence, we do not see any negative societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks. 
 
 Appendix: In this appendix, we provide more qualitative results on our ice hockey and Lacrosse datasets. Furthermore, we provide a quantitative comparison with a multimodal network ( [18] ).To investigate the effectiveness of our technique of leveraging text with existing multimodal methods, we compare TokenCLIPose's performance with the multimodal counterpart LAMP [18] . Though LAMP does not follow the top-down approach, it is the only network that uses language for 2D human pose estimation. Thus, we compare and highlight the results in Table 8 . As shown in the table, it is evident that TokenCLIPose outperforms LAMP on the CrowdPose dataset by 4.8%. This corroborates our claim that leveraging text as supervisory signals to guide image features provides better performance than aligning image features to the text tokens. We showcase additional qualitative results on our ice hockey dataset visualizing the effectiveness of our approach. Furthermore, we also demonstrate the generalization capabilities of our model in transferring to a Lacrosse dataset. The impact of language is visible in our model's robustness to domain changes.To further illustrate the effectiveness of our approach, we present additional qualitative results on the ice hockey dataset in Figure 5 . Furthermore, we demonstrate the model's generalization capabilities by achieving strong performance on a Lacrosse dataset, highlighting the impact of language on the model's robustness to domain changes.(a) ice hockey (b) Lacrosse Our work provides a new way to reconsider the pose estimation task to predict out-of-bounding box keypoints. This improves our understanding of why joint hockey stick pose prediction along with player pose is difficult for existing top-down networks and presents one way to mitigate the issues. This understanding should enable the application of top-down solutions for multi-instance pose estimation which encounters similar issues. Furthermore, accurate stick pose prediction implies that hockey teams can make more data-driven decisions to improve their teams' performance without incurring additional overhead expenses. This also implies that the use of invasive technology such as infrared sensors can be avoided to a large extent to robustly analyze players and teams.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.This can be leveraged in various domains where humans closely interact with objects, such as shoveling, where it could be used to study fatigue and biomechanics. Furthermore, as shown in our experiments on the CrowdPose dataset, TokenCLIPose also reliably predicts the pose of a human when there is high human-human interaction. Hence, this can be used for pose estimation in highly crowded scenarios, such as surveillance and monitoring people in crowds. 


Decision: The reviewers agree that the proposed method is clear and that the experiments show reasonable improvement over the state of the art.  There were some questions about whether the range of datasets evaluated was sufficient, but after the rebuttal and discussion, the reviewer closest to the area and the area chair both found this sufficient.  The reviewers make some comments about explaining the context and advantage of the proposed method that should, if possible, be utilized in cleaning up the paper.

Reviews: review: paper_summary: paper_summary: This paper introduces the problem of estimating the keypoints of humans together with a stick-like object that the human interacts with, i.e. hockey stick or lacrosse stick (which is referred to as "extension" in this paper). The paper claims that prior works on 2D human pose estimation cannot be naively extended to this task by simply using a larger cropped image containing the human and the object together, because it would introduce unnecessary features and hurt performance. To resolve this issue, the paper proposes the TokenCLIPose method, which focuses solely on human using a tight crop bounding box and utilizes keypoint-specific text embeddings from CLIP to predict out-of-bounding-box keypoints of the stick objects. Specifically, it uses a CNN to extract features from images and obtain a coarse estimation of human keypoints in the bounding box with an MLP. Subsequently, it feeds the image features, coarse keypoints features and the keypoint-specific text embeddings of the stick keypoints from a CLIP text encoder to a transformer to obtain a final keypoint estimation result. The experimental results indicate their method outperforms previous 2D human keypoint estimation methods in their newly curated datasets for this new setting, as well as on the CrowdPose dataset with multi person scenarios that have severe occlusion.
strength_weakness: strength_weakness: 1. The proposed problem of detecting keypoint of cropped out objects seems artificial, and it remains unclear if predicting the keypoints of cropped objects is actually necessary. One important ablation study that needs to be shown is to train TokenCLIPose on images that are not artificially cropped, and instead contain both the human and the stick together. It seems that if the text embedding is helpful for locating keypoints of unseen objects, it should also help when the in the bounding box.
2. The description of adopting the keypoint-specific text embeddings for predicting the keypoints of unseen keypoints is unclear. Why should the text embeddings contain any information of the location of unseen objects in not explained and not studied sufficiently in the experiments.
3. How are the baseline methods in the experiments adapted to the newly proposed task? This is not discussed at all. The number of keypoints for the extensions seems to vary between experiments. How do you adapt human pose estimation methods for this new task and why do these methods fail in this setting? This should be explained in more detail.
4. The proposed method should be compared to more recent works like ViTPose++ (TPAMI 2023) and BUCTD (ICCV 2023). The latest works used for comparison in the submission are ViTPose (NeurIPS 2022) and HRFormer (NeurIPS 2021). As this paper works on a different setting than classical human pose estimation, it is important to discuss why existing 2D human pose estimation methods are limited in dealing with this setting. 
5. What are the text prompts used for the Text-based Keypoint Encoder? There is no explanation regarding this essential part of the method.
6. This method is only verified with simple objects like hockey sticks and lacrosse sticks. A core open question is how the method would generalize to more complicated structures like chairs?
questions: questions: In addition to the questions contained in the previous section:
1. How is the visualization of Figure 1(b) obtained?
2. For the ablation study in Table 4, what is the result of only combining text token and image token?
limitations: limitations: The paper shows failure cases, but only limited quantitative results are shown and there is no discussion about these failure cases.
paper_summary: paper_summary: The paper argues that using images including both humans and interacting objects introduces unnecessary visual features that hinder pose estimation performance. To address this, the paper claims that treating objects as unseen to predict interacting object poses can achieve better results and proposes a TokenCLIPose solution. The method is evaluated on the Hockey and Lacrosse datasets, which contain images from a total of 11 video clips.
strength_weakness: strength_weakness: The experiments and comparisons are not convincing and the experiments are weak.

(1) The paper conducts its main experiments on the Hockey and Lacrosse datasets. However, these datasets are too small, comprising only 11 video clips.

(2) The paper claims that predicting object poses using human-only images performs better than using human-and-object images. However, the paper lacks in-depth analysis, and the experiments are insufficient to support this claim. The reviewer disagrees, arguing that observation is more accurate than imagination when an appropriate framework is used.

(3) The major contribution of the paper is not clearly defined.

(4) The paper did not discuss and compare with bottom-up methods, which are popular for human pose estimation.

(5) In Line 136, the text encoder simply encodes class-related information/names of each keypoint, which the reviewer argues cannot tackle the challenge of unseen keypoints in the image.

(6) Typos, such as "we curate two new sports dataset" and "Experimentations".
questions: questions: See Weaknesses.
limitations: limitations: No discussion on limitations is provided.
paper_summary: paper_summary: This paper proposes a text-guided keypoint localization method that can detect keypoint that is out-of-input image. The proposed method only crops person area and abandons the object area, then adopts language prior to predict keypoint that is out-of-input image. To verify the effectiveness of the proposed method, this paper also introduces a Ice-Hockey keypoint detection dataset. Experiments on CrowdPose demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: The proposed method is not well suited to the introduced tasks such as  Ice-Hockey. Abandoning the visible object is not a good solution because the model cannot make perfect prediction on out-side objects. For occluded keypoints, it is not bad because we can only guess the position, it is acceptable that the prediction is not so accurate. But for visible keypoints of hockey stick in Fig. 1(a), the predicted results are not acceptable. A better way to handle unnecessary visual features is to separate the object and person and perform keypoint localization on each individually. 

I think the proposed method can be used to perform the occluded/out-of-box keypoint localization, but not to solve the Human-Object Keypoint Localization tasks such as Ice-Hockey. The introduced task and dataset cannot fully demonstrate the effectiveness of the proposed method. It is better to introduce the method from the perspective of occluded or out-of-box keypoint localization, abandoning the Ice-Hockey tasks and background and conduct more experiments on occluded  keypoint localization benchmarks such as OCHuman.
questions: questions: In Weakness
limitations: limitations: The authors say they show the failure cases of the prooposed method, but I do not find them in the manuscript.
paper_summary: paper_summary: This paper focuses on an interesting and inherent problem in top-down human pose estimation; that is out-of-box prediction in a top-down paradigm. The core of the solution is to utilize the semantic context by giving proper text prompts to CLIP. Specifically, a cropped image is first given to a pre-trained CNN model to extract initial features and obtain a coarse prediction. Then the visual features from CNN, the coarse prediction of the person, and the semantic information from CLIP are concatenated and input to a transformer decoder to obtain the final prediction. The method is capable of solving the occluded points in the crowded scene and breaking the limits from the bounding box in the top-down model.
strength_weakness: strength_weakness: 1. Poor visualisation. Qualitative results are only compared with HRNet. More visualization should be shown to validate the effectiveness of the proposed method.  

2. A few images are very ambiguous. Especially, the Figure 1. (b). There is no notation for x, y, and colors. The caption said it is the text embedding and such embedding loses the global structure. Why did it lose the global structure?

3. In terms of performance, there is indeed an improvement. However, how many of them are coming from the unseen points? i.e. Is the source of improvement from unseen points or a more accurate estimation of the seen points? After all, the whole pipeline is a coarse-to-fine prediction with additional text information. Therefore, the seen points should also benefit. If the major improvement does not come from the unseen points, then the major problem proposed by this paper still remains unsolved.
questions: questions: 1. One statement in the introduction that I do not agree with is "Contemporary deep learning-based pose estimation methods predominantly follow a top-down approach" between line 27 and line 28. Bottom-up and one-stage models are also important parts of human pose methods. In fact, bottom-up models have their own merits such as real-time inference and bounding-box independent predictions. I suggest revising this statement.

2. It would be better if the authors could make a further explanation on using normalizing flow to estimate the human poses. I think it is quite interesting as this method was published in 2021 and is quite rare in recent work. Many other papers like PETR directly regress pose with the decoder. Then it is natural to ask Why use normalizing flow? What is the impact if we directly regress the pose? These two points may need further discussion.

3. What is the exact text prompt used in the given method? 

4. The authors also claimed that the previous method ignored the global spatial relation between points. Yet it is still unclear to me how the author's method solves this problem. In the introduction, it is claimed that transformers are used to solve the problem. But it is too general to convince me.
limitations: limitations: No negative societal impact are found.

Comments: comment: Thank you for the authors' responses. However, like Reviewer 9Npr, many of my concerns remain unaddressed. The experiments do not convincingly support the main claim.
**Dataset Size**: As mentioned in W1(a) of our response to reviewer F7X8, to the best of our knowledge, this work is the first to address human extensions and their pose estimation. Given this, there is no public dataset available for this specific task. Pose estimation requires precise annotations, which cannot be effectively generated by large language models (LLMs) or vision-language models (VLMs). Therefore, manual annotation was necessary, and this process is extremely time-consuming.

We believe it is important to recognize the significant effort required to manually create this dataset for the first time. As such, it would be unjust to criticize our dataset size given these constraints. Nonetheless, we really appreciate your comments and will try to collect more data to showcase the impact of our approach.
Unfortunately, my questions and comments from the previous post were not addressed. To me, the proposed task (predicting keypoints of sticks held by humans outside a cropped bounding box) is not sufficiently well motivated (which is also stated by reviewers F7X8 and AetR). I agree with reviewer F7X8 that the problem is very likely caused by the small amount of training data (in total only 11 videos, which is very few data in the context of keypoint detection). Hence, the overall relevance of this work is reduced to detecting keypoints of sticks held by humans from a small set of videos, which seems a rather niche area of research. While the proposed method does lead to some improvements in performance, I am not convinced of the general relevance of the contribution, which would need to be shown on a larger variety of object classes, not just sticks.
Thank you for taking the time to review our rebuttals. We believe we have conducted comprehensive experiments that sufficiently illustrate the rationale behind our approach of cropping out sticks. Specifically:

1. **Noise Influence:** We have demonstrated the influence of noise by comparing our model with ViTPose on two different data splits—one with predominantly visible keypoints and another with a mix of visible and noisy keypoints, as presented in Table 1 of our rebuttal document.

2. **Training on Human+Stick Bounding Boxes:** We have further validated our approach by training our model on datasets containing human+stick bounding boxes, as shown in Table 2 of our rebuttal document, which underscores the efficacy of our work.

3. **Qualitative Performance:** Finally, we have provided qualitative evidence of our model's superior performance in noisy conditions in Figure 1, where noisy regions are clearly marked.

Given these extensive evaluations, we are unsure what additional experiments would be necessary to further validate our hypothesis. If there are specific experiments or analyses you would like to see, we would greatly appreciate your guidance in this matter.
I appreciate the effort of the authors, however, it remains unclear to me why cropping out the object of interest from the image actually improves the performance. The argument made is that this introduces irrelevant features, but this should only be a problem in a limited data setting where the model starts overfitting on the background, is that correct? This hypothesis could be even verified with synthetic data rendered with computer graphics in cluttered backgrounds. To me, and it seems also several other reviewers, the paper does not sufficiently study when and why the model fails when the stick is included in the crop. Demonstrating a performance decrease in the results is a necessary initial step, but since the result is very unintuitive, I would expect a more in-depth study of the reasons for this problem.
We would like to thank the reviewer for their invaluable comments and suggestions.

1. **Standard deviation as confidence score**: Thanks for noting such minute details and raising this point! The results that we reported are using the standard deviation directly as a confidence measure.

2. **OCHuman Evaluation**: We appreciate the reviewer’s suggestion of evaluating our model on OCHuman. However, since the OCHuman dataset does not contain any train set, the general scheme followed by the research community is to train the model on COCO dataset and test it on OCHuman. This is difficult for us due to memory constraints as one complete training run on the COCO dataset takes approximately 4-5 days. Hence, this would not be possible before the rebuttal deadline. Moreover, in order to showcase robustness under occlusions, we did conduct experiments on CrowdPose, the largest occlusion benchmark.

3. **Impact of text prompts**: Thanks for the suggestion. We would like to point out that apart from citing other relevant research, we have also conducted quantitative experiments (Tables 4 and 6 of the main paper) showcasing the benefit of using text prompts on the accuracy of our model. Hence, we believe we have successfully demonstrated the positive impact of text prompts. Moreover, we agree that adding visualisations will further bolster our motivation behind utilizing text and will include them in our camera-ready version.
Thank you for your detailed response. Most of my concerns have been properly addressed, though I have a few additional points I’d like to discuss:

1.  As part of the ongoing discussion regarding the use of normalizing flow, I believe the method provides an explicit probability distribution for keypoint estimation, correct? If so, I’m curious whether the standard deviation of each prediction can be directly used as an uncertainty measurement or confidence score. If the method already employs this approach, could you please share the performance results? If not, I suggest testing this on the CrowdPose dataset, as its evaluation protocol (based on COCO) is highly sensitive to accurate confidence estimation of poses. Demonstrating improved performance in this area could potentially strengthen the paper.

2. Several reviewers have noted that the test dataset appears limited and relatively simple in terms of occlusion challenges. I recommend evaluating the method on the OCHuman [1], which has gained popularity recently for methods addressing occlusion. This could provide more robust evidence of the method’s effectiveness under challenging conditions.

3. Regarding the concern (C3) on how the text guides the model in predicting the exact location of unseen points, it would be helpful if the authors could visualize the attention map between the text prompt and the visual features (e.g., between F_text and F_vis or between F_text and F_loc). My assumption is that the attention between F_vis and F_loc (both visual features) struggles to focus on out-of-box regions because these regions (e.g., white regions if I am correct) lack visual similarity to in-box visual features. Thus, there is no hope to recover any useful information through the these visual features (F_vis and F_loc). However, text features might not rely on visual similarity and could direct attention to broader regions, including out-of-box areas.  In my point of view, this visualization could be more persuasive to other reviewers than simply citing other papers to support the intuition behind this idea.

I understand that addressing these points may require additional experiments, which would take time. Therefore, I won’t factor the response to these discussions into my final rating, even if no further experimental results are provided.

Finally, I noticed that most reviewers initially gave a negative rating, and I agree with many of their proper and valid points. I would like to see their feedback on the rebuttal before making my final decision. Also hope it will help to convince other reviewers through address a few of my concerns.

[1] Pose2Seg: Detection Free Human Instance Segmentation
**Note about surveillance**: We thank the reviewer for taking out the time to look into our paper. We will add a note about surveillance and potential safeguards in our  updated manuscript. If there are any other questions regarding our work, please do not hesitate to ask us.
We thank the reviewer for taking out the time to look into our paper. We are glad that the reviewer does not have any ethical concerns with our paper. If there are any other questions regarding our work, please do not hesitate to ask us.