Seeing Beyond the Crop: Using Language Priors for Out-of-Bounding Box Keypoint Prediction
Authors: Bavesh Balaji
Jerrin Bright
Yuhao Chen
Sirisha Rambhatla
John S. Zelek
David Anthony Clausi
Venue: NeurIPS 2024 poster (2024)
Keywords: 2D pose estimation
out-of-image keypoint prediction
multimodal pose estimation
CLIP
Primary area: machine_vision
TL;DR: We re-conceptualize pose estimation as an out-of-image keypoint prediction task to robustly predict extension keypoints from human pose.

Abstract: Accurate estimation of human pose and the pose of interacting objects, like a hockey stick, is crucial for action recognition and performance analysis, particularly in sports. Existing methods capture the object along with the human in the bounding boxes, assuming all keypoints are visible within the bounding box. This necessitates larger bounding boxes to capture the object, introducing unnecessary visual features and hindering performance in real-world cluttered environments. We propose a simple image and text-based multimodal solution TokenCLIPose that addresses this limitation. Our approach focuses solely on human keypoints within the bounding box, treating objects as unseen. TokenCLIPose leverages the rich semantic representations endowed by language for inducing keypoint-specific context, even for occluded keypoints. We evaluate the performance of TokenCLIPose on a real-world Ice-Hockey dataset, and demonstrate its generalizability through zero-shot transfer to a smaller Lacrosse dataset. Additionally, we showcase its flexibility on CrowdPose, a popular occlusion benchmark with keypoints within the bounding box. Our method significantly improves over state-of-the-art approaches on all three datasets, with gains of 4.36\%, 2.35\%, and 3.8\%, respectively.

Seeing Beyond the Crop: Using Language Priors for Out-of-Bounding Box Keypoint Prediction
Bavesh Balaji Jerrin 3 Yuhao Chen Sirisha Rambhatla 1.3,
John S. Zeleka.3, David Anthony Clausi?.3
CCriticll ML Lab, Vision and Image Processing Lab, SUniversity of Waterloo,
Abstract
Accurate estimation of human pose and the pose of interacting objects, like hockey stick, is crucial for action recognition and performance analysis, particu- larly in sports. Existing methods capture the object along with the human in the bounding boxes, assuming all keypoints are visible within the bounding box. This necessitates larger bounding boxes to capture the object, introducing unnecessary visual features and hindering performance real-world cluttered environments. We propose a simple image and text-based multimodal solution TokenCLIPose that addresses this limitation. Our approach focuses solely on human keypoints within the bounding box, treating objects as unseen. TokenCLIPose leverages the rich semantic representations endowed by language for inducing keypoint-specific con- text, even for occluded keypoints. We evaluate the performance of TokenCLIPose on a real-world ice hockey dataset, and demonstrate its generalizability through pero-shot transfer smaller Lacrosse dataset. Additionally, we showcase its flexibility CrowdPose, popular occlusion benchmark with keypoints within the bounding box. Our method significantly improves over sfate-of-the-art approaches on ice hockey, Lacrosse, and CrowdPose datasets, with gains of 4.36%, 2.35%, and 3,8%, respectively.
Introduction
The goal of 2D human pose estimation to localize the human anatomical keypoints from an image, which is essential for scene understanding, action recognition (12,, and human-object interaction detection 4 This particularly challenging in cluttered real-world scenarios due to occlusions and other non-idean non-idealities Reality (AR), and
[5] 6). With the emerging applications in Virtual Reality (VR), and Augmented sports analysis 77] there is a fundamental need understand how objects are manipulated via naman-objec interaction [8] Often, in such scenarios, the objects that humans hold and interact with, which we define as extensions, can provide crucial information that aids in accurately estimating the human pose and the actions being performed [
Contemporary SOTA deep learning-based pose estimation methods predominantly follow a top-down approach: cropping each person in an image using bounding boxes before estimating their pose individually While using existing top-down pose estimators seems intuitive for joint prediction of the humans and their extensions, this approach suffers from limitations, yielding suboptimal results as shown in Fig. I(a)
Our key observation is: capturing the extension in the bounding box expands the held-of-view and introduces unnecessary visual features, which can be confusing the model. A simple yet powerful fix is to confine the bounding box to capture the human body, treating the extension's keypoints as unseen. This approach reduces background interference, However, it leads to the loss of important visual information about the extension, making them unseen.
do not explicitly capture the extension.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Right Hap
Right Shouider
Left Hip
Right
Left Shoulder
Lef' Elhow
Right Ankle
Left Ankle
Right Elbow
Right Left
(a)
(b)
Figure Difference between existing networks and our network. (a) Qualitative comparisons between HRNet (left) and our pose estimator (right). (b) t-SNE Visualization of keypoint-speciffe prompt embeddings. The different colours represent whether they are upper-body or lower-body joints, with red representing upper-body joints, blue representing lower-body joints and green representing the hip joints. From the figure, it evident that while these embeddings maintain positional structure within the upper-body joints (the shoulders wrists, following the pose a human standing normally) and the lower-body joints (knees placed above ankles), they fail to maintain positional structure between the upper-body and lower-body joints (elbows and wrists are placed below knees and ankles in the plot which does not follow the pose of human being).
placed above the elbows which are placed above the
Here, crucial question arises: How effectively represent the spatial relationships of these unseen keypoints for accurate pose estimation?
To answer this, we turn to other ways of informing the model about the unseen keypoints. Recent works have shown that using language to induce semantic context of keypoints can lead to effective feature representations 17H20). Specifically, existing works [18] on human pose estimation align image features with keypoint-specific text embeddings generated from Vision Language Models (VLMs) using contrastive loss. However, these text embeddings primarily capture local details, neglecting the crucial global relationship between bower-body and upper-body joints; Fig. [(b) language models encode similar joints together, while losing the global structure. Hence, explicitly imposing the image features to close to text embeddings could be suboptimal.
Based on the observations, we present simple yet effective solution to significantly improve relia- bility under dynamic real-world scenarios by leveraging language to see beyond the bounding box. Specifically, We utilize these text embeddings as priors and initialize our learnable keypoint tokens (referred to as text tokens) using these text embeddings. By integrating the rich semantic represen- tations of keypoint-speciffe text embeddings with image features, and employing a transformer to capture global dependencies, we extract superior ane-grained representations which significantly boosts the performance across the board.
We evaluate TokenCLIPose's performance three real-world datasets containing lot of occlusions and noise: an ice hockey dataset, a Lacrosse Dataset and the CrowdPose dataset [6]. The ice hockey and Lacrosse datasets are ffrst-of-their-King datasets that we curated for predicting the pose of human extensions (the sticks). Furthermore, in order to demonstrate the flexibility of TokenCLIPoss predicting unseen keypoints that are present within the bounding box, we evaluate it on CrowdPose dataset. TokenCLIPose outperforms existing top-down approaches by 4.36% and 3.8% on the ice hockey and CrowdPose datasets respectively. Furthermore, TokenCLIPose demonstrates superior zero-shot capabilities in predicting extension keypoints when tested on the Lacrosse dataset,

---

outperforming prior works by 2.35%. Our experiments highlight TokenCLIPose's ability to reliably predict unseen keypoints.
2
Related Work
2D Human Pose Estimation:
Top-down approaches in 2D pose estimation can be broadly classified into two categories: Heatmap-based pose estimation has been the de-facto standard approach since the introduction of stacked hourglass networks [14. These methods represent discrete (x, v) coordinates as continuous heatmaps where each pixel indicates the likelihood a specific joint being located at
that position. Most methods ( 2 [3] rely on powerful convolutional networks to extract high- level multi-scale feature maps. While most research focuses on network architectures, a few works investigate the coordinate representation and the heatmap encoding and decoding process [21 [5]. Recently, researchers have begun exploring transformer-base
architectures for pose estimation [10 [22 II 23 Xu et al. [10] adopt the original vision transformer (24) and build baselines for pose estimation, showcasing the efficacy of vision transformers. Methods including [22 [1 use CNNs as feature extractors, and utilize transformers model the relationship between different scale features
and the keypoint features respectively.
Regression based pose estimation, in contrast to the dominance of heatmap- based approaches offers an alternative paradigm. Sun el al. [25] leverages a convolutional backbone to extract feature maps and then utilize an integral operation to directly regress keypoint coordinates. Li et al. [26] developed a novel pose regression model which aims at minimizing the distance between the predicted and underlying distribution. However, the global pooling operation used in 26 results in loss of spatial
information that is crucial for reliable pose estimation. More recent works 27 16 28 include
transformer-basee architectures:
27 16
use encoder-decoder strategy and are based DETR
29 and Deformable-D DETR 30 respectively.
While heatmap-based methods have achieved high accuracy, they have some limitations: 1) These methods have a non-differentiable heatmap decoding method: 2) Heatmap representation often leads to quantization error; and 3) They are not designed to predict out-of-bounding-bo keypoints. Therefore, we adopt the regression- bassed approach for directly estimating the coordinates of all joints, both inside and outside the bounding box.
Several approaches tackle the problem of occlusions in crowded scenarios. Various methods tackle the problem by studying the relationships between multiple humans present an image (31) However, we do not compare with their works as we are interested in single- inntaneisttanceeee approaches. Park et al. [34] addresses the problem of out of bounding box keypoints by refining the bounding box before pose prediction, but this adds computational cost. Our approach, on the other hand, provides a parameter-free technique to induce spatial context for the oul-of-bounding box keypoints by leveraging the knowledge of VLMs.
Occlusion-Aware Pose Estimation:
Vision-Languaal Models: Language supervision has been shown to improve feature representations in various vision tasks, such as image classification, semantic segmentation, and pose estimation. Radford et al. (35) proposed the contrastive pretraining paradigm CLIP that leverages contrastive learning to optimize a text and image encoder jointly. This work also showcased the significance of large-scale rision-languag CLIP, various works such as [36 38 focused on improving image classification using CLIP. Zhang el al. [39 transfer the 2D pre-trained knowledge to 3D domains, thereby improving zero-shot point- cloud recognition. Rao el al. [40] showeased the efficacy of CLIP pretraining on dense prediction tasks by converting the image-text matching problem to pixel-text matching problem.
pretraining by demonstrating accurate zero-shot classification. Following
Recently, Tevet et al. (20] leverage text to inpaint missing poses in a sequence in spatiotemporally consistent manner. Guo et al. [19] use pose-aware prompts to predict 3D hand meshes from images. Recent works on 2D pose estimation [T8] utilize joint-specific keypoints learn richer represen- tations. Particularly, they use contrastive loss to align the image features extracted from a backbone to their syypoint-speciffi text embeddings. This, however, is suboptimal as the text embeddings do not capture global structure between human keypoints. Hence, aligning the image features to text embeddings might lead to loss of spatial information and inaccurate localization of joints. We improve this by using a transformer-basee image and text features, thereby guiding our image features using text supervision rather than biasing them to the text prompts themselves.
network to capture the global spatial dependencies between
3

---

Image Encoder
29
MLP Layer
Backhome
Pa O
Aftestion
MLP
Layer
Tepul Irage
Plnal Keypoint Prodiction
00000 1
(b) Transformer Decoder
Torrafommer Dozader
00000000000000000000000000000000 00000000000000000000000000000000 0 10
x=x-+t
FLoc
CT C1.1P
RI Losk
L (2)
(x(
Cours Harran Koyper Prodiction
Text Promyys
Rosidhal Log Likelibood
(R))
(a) Proposed TokenCLIPose Architecture
(c) RLEI Loss
Figure 2: TokenCLIPose Architecture: We first incorporate an image encoder to extract multi-scale image features and coarse human keypoint locations få,, and project them onto a joint multimodal embedding space obtaining image tokens Fvia and location tokens F Foc respectively. Then, we leverage a text-based keypoint encoder extract keypoint-specific These multimodal tokens are fed to a transformer decoder to capture spatial dependencies between them and predict all the 2D keypoints fid. The coarse human keypoint predictions and the final keypoint predictions are supervised using the RLE loss.
tokens Fraxt from VLMs.
Method
3.1 Problem Formulation
Given a cropped image I k2xx of human, generated from bounding boxes obtained through a detection network, we predict Kout keypoints fåa E I2 Rowss that represent the 2D poses of extensions and/or humans, along with the scale parameter oa for each keypoint.
3.2 Network Architecture
We propose an encoder-decoder architecture to estimate 2D keypoints that are not captured in the bounding box. Firstly, an image encoder extracts multi-scale image features from the cropped image, which are then passed through a Multi-layer Perceptron (MLP) to generate coarse human keypoint proposals. These image features and locations are then projected onto the multimodal embedding space to form image and location tokens. The text-based keypoint encoder leverages text prompts generate keypoint-specifie text tokens. Finally, all these multimodal tokens concatenated and passed through a transformer decoder to capture global relationships between these tokens and predict final 2D keypoints. The network is exemplified in Fig. 2
Image Encoder. The cropped input image is initially passed through a pretrained CNN to extract multi-scale dense feature maps. These multi-scale feature maps fused and projected onto joint multimodal embedding space to form the image tokens Fyis E N XCCmb, where is the number of tokens and Cemb is the joint multimodal space dimension. Furthermore, they processed through a MLP to generate coarse human keypoint predictions Ú E RR K, x2 and a scale parameter o E K,XI,, x1, which are optimized using the RLE process as detailed in Section 3.3
Text-based Keypoint Encoder. Following image feature extraction, the keypoint encoder tackles the challenge of unseen keypoints in the image. We employ Iaguage-guides keypoint representations, where text prompts encode class-related information of each keypoint. By leveraging CLIP's pre- trained text encoder, we generate the text tokens F Ftart= R Kout XCeb which are then concatenated with the extracted image tokens. This process indirectly injects visual context of the missing keypoints into the model, even when it's not directly visible in the cropped input image. Additionally, we incor- porate the coarse human keypoint predictions pi and convert them to location tokens by projecting

---

them onto the joint mulitmodal space Fisc E IR XCems and concatenate them with the image and text tokens. The final set of tokens fed to the decoder are denoted F (Fria. Frett Fiee).
Transformer Decoder. To predict all the keypoints precisely, the relationships between these multimodal tokens (F) must be captured accurately. Therefore, we leverage the transformer decoder to understand the inherent correlations between different text, image and location tokens. Instead of treating each modality separately and utilizing cross-altention mechanism to understand the associations between them, we treat all tokens together as a homogeneous entity and employ the standard self-attention mechanism. This approach offers a simpler way to gain a holistic view of the relationships between all keypoints, locations, and image patches. The transformer layers are followed by MLPs estimate the final keypoint predictions jud E R.Kos x2 and the scale parameter dd E I Kout x1 To reduce artifacts in feature maps and understand richer relationships, we employ additional register tokens as
3.3 Loss Function
Distribution Learning. Following 226 [6], we formulate the regression task as distribution learning problem and adopt Maximum Likelihood Estimation (MLE) to effectively predict the output coordinates. We use normalizing flows to estimate the deviation in predicted and ground truth keypoint distributions. In mathematical terms, given an input image I, our network estimates distribution Pe. (x(Z) representing the probability of the ground truth keypoint appearing at the location x. Here, & and & denote the parameters of our pose network and the flow model fo- respectively. The flow model fe acts refinement step, iteratively transforming preset Gaussian distribution z (O, Id) capture the deviation of the predicted and ground truth distributions using the network's 's prediction (ji and σ).
Equation (1) depicts the mathematical formulation of the RLE loss, where Q(1g) is the preset Gaussian distribution, G((g)) is the learned distribution by the flow model, 14g (Mg p represents the normalized difference between ground truth and predicted keypoints, and is constant term
LALE = log ( - logg Go((ag) - llog 8 + log ō
Similar to [16,, we supervise both the coarse predictions (Hi,, oy) and the final predictions from the decoder (pia: da) using RLE. Hence, our final loss function is
L = =hEE + CLLE
(2)
where,
C = log Pay o (x
and C -lg Pos (x(Z)
x=
x=
Here, , and & denote the parameters of the backbone network and the flow model of the coarse keypoint predictions, and O₂ and h denote the parameters of our decoder regression model and flow model of the final keypoint predictions.
Experiments
A critical challenge in evaluating pose networks for human extensions is the lack of publicly available datasets containing both the human and extension pose annotations. To address this limitation, we curate two sports datasets, ice hockey and Lacrosse, featuring pose annotations for humans and their corresponding extensions. Experiments on these datasets demonstrate the performance of our model in estimating the pose of extension keypoints, which are not captured within the bounding box. We further evaluate the model's ability to predict unseen keypoints (due to cellusion/sel--occlussin within the bounding box using the benchmarked multi-person cluttered dataset, Crowdpose (6), validating the generalizability of our model.
For human pose estimation, the total keypoints (K depend extensions. Without extensions, Kont equals the number of human keypoints (K,). When extensions are present, Kost increases to KA plus the number of extension keypoints (K.).
5

---

Input Image
HRNet- W48
12
Ours
Input Image
HRNet-Wae
T2
Ours
PPER
<c
meifter
neisTer
Figure 3: Qualitative Comparison of TokenCLIP IPose with HRNet- W48 on our ice hockey dataset.
4.1 Ice Hockey Dataset
Dataset. The ice hockey dataset, captured from real-world National Hockey League (NHL) videos, presents a challenging environment for pose estimation due to the inherent fast pace of the sport. Players' rapid movements often result in motion blur within frames, making it difficult distinguish body parts. Furthermore, the nature of the game leads to frequent occlusions, especially when players obstruct each other's bodies. Adding to the complexity the bulky equipment worn by hockey players, which can obscure keypoints. These combined challenges- motion blur, occlusion, and bulky equipment- make the ice hockey dataset a valuable resource for evaluating the robustness of the pose estimation task.
Our dataset consists of 10 video clips (30-45 seconds each, sampled at 30 fps) from various broadcast NHL videos. We utilize the CVAT tool to annotate the keypoints for multiple players and their hockey sticks in each frame. We followed the standard COCO format to annotate 17 keypoints for the human pose. Additionally, we annotate 3 keypoints for the hockey stick (butt end, heel, and toe). Finally, for each annotated frame, we create only the player (standard bounding box) and another incorporating both the player and their hockey stick (extended bounding box). In total, we generate 11.66K pose annotations, with 9.13K images from 9 clips used for training and 2.53K images from the tenth clip used for testing our model's s performance.
versions of the input image: one cropped to include
Evaluation Metric. We evaluate different State-Of-The-Att (SOTA) pose estimation models on our dataset using the Percentage of Correct Keypoints with hemd-normalizztion (PCKh) metric. Due to our method's use of smaller bounding box compared to the other models, achieving the same level of accuracy would result in higher PCKh threshold for our model. To ensure fair and direct comparison across all models, we employ the same threshold for evaluation.
Training. All networks are trained for 200 epochs with batch size of 64 on single NVIDIA GeForce RTX 4090 GPU. We employ the Adam optimizer with a learning rate of 6 x 10-4 for all CNN-based architectures while the transformer- basee architectures are trained using the AdamW optimizer with an initial learning rate 3 x 10-4. The weight decay is set to 10 for all models. used linearly reduce the learning rate from the initial value of 10-
A stepLR scheduler Consistent with 266, we utilize RealNVP 42) as the flow model within our model.
Results. The results, presented in Table emphasize our method's SOTA performance over existing methods in robustly predicting the ice hockey player and the hockey stick keypoints, outperforming existing works by 4.36%. This is further validated by the qualitative comparisons in Fig. where model demonstrates the ability to predict semantic poses even in challenging scenarios with extreme motion blur and occlusion. These results support hypothesis that re-considering the extension pose estimation task as an unseen keypoint problem without explicitly capturing it in the bounding box reduces background noise leading to robust poses.
4.2 Lacrosse Dataset
Dataset. In order to study the efficacy of TokenCLIPose's generalization capabilities in predicting the pose of extensions, we curate another small-scale Lacrosse dataset. Similar to ice hockey, it is

---

Table 1: Comparison with SOTA Methods our real-world ice hockey dataser PPKh@@0.)
BoldFace represents the best score. Underline represents the top score in existing works.
Method
Backbone
Input Resolution
Body
Butt End
Stick Heel
Stick Toe
Mean
SimpleBaseline 13
ResNet-50
256x
93.59
69.57
57.19
52.76
68. 68.83
MSPN 9
256x 192
93.61
70.30
59.21
55.69
69.70
HR-
12
HRNet-W48
256x
94.90
71.48
60.29
55.36
70.44
TokenPosc-L/DD
HRNet-W49
256x192
95.13
70.96
60.93
56.27
70.82
ViTPose 10
ViT-B
256x
95.61
71.94
61.33
$8.80
71.92
TokenCLIPoae
ResNet-50
256x192
95.81
74.86
65.79
65.08
74.92
TokenCLIPose
MSPN
256x192
97.17
75.41
66.70
66.34
75.53
TokenCLIPose
HRNet-W48
256x192
97.37
75.94
67.82
66.15
76.28
Improvement
1.76%
4.00
6.49%
7.35%
4.36
Table Zero-shot Comparison with SOTA Methods on our real-world Lacrosse dataset
(PCKh@0.5), BoldFace represents the best score. Underline represents the second-best score.
Method
Backbone
Body
Butt End
Stick Heel
Mean
SimpleBaseline
ResNet-50
94.73
67.28
53.99
72.00
MSPN 9
95.84
70.68
57.40
74.64
HR-Net [12
HRNet- W488 VIT-B
95.92
71.35
58.41
75.22
ViTPose 10
95.77
72.85
60.18
76.26
TokenCLIPose
HRNet-Wal W48
97.24
76.60
65.01
78.61
Improvement
1.47%
3.75%
4.83%
2.35% 1
characterized by motion blur and occlusions, but there are domain differences between the datasets. This dataset consists of 300 pose annotations from one video sampled at 30 fps. We use the same 12 human keypoints used in the ice hockey dataset denote the human's pose. However, we use only keypoints to represent the Lacrosse stick's 's pose, as the blade of a Lacrosse stick is circular in nature. Hence, we disregard the 15th keypoint (stick toe) and use the other 14 keypoints represent the pose of Lacrosse player along with their stick.
Zero-shot Results. We evaluate TokenCLIPose's generalization capability by performing zero-shot transfer from our pretrained model on ice hockey to the Lacrosse dataset. Due to the difference in the shape of the head of lacrosse and hockey stick, predict the two keypoints corresponding to the shaft of a lacrosse stick. Furthermore, the text prompts for the two extension keypoints are also changed while keeping the model frozen. The results presented in Table showcase that model outperforms the established baselines by 2.35%, thereby demonstrating the efficacy of our proposed model for generalizable pose estimation.
4.3 CrowdPose Dataset
Dataset. The CrowdPose dataset is a large-scale benchmark dataset for human pose estimation, containing 12K images and 43.4K labeled people in their trainval set, and 8K images with 29K labeled people in the test set. Following 33 18 31, teat set for evaluation.
we use the trainval set for training and
Evaluation Metric. We adopt standard Average Precision (AP) as our evaluation metric on the CrowdPose dataset. AP is caleulated based on Object Keypoint Similarity (OKS) denoted by moks E I, which is defined as
mOks = Σ,
0)
E, 0)
where d, is the Euclidean distance between the i-th predicted keypoint coordinate and the corre- sponding ground truth, is the visibility flag of the keypoint, is the object scale, and k; is keypoint-speciff constant.
Training. We employ the AdamW optimizer with an initial learning rate of 6 x 10-1 and weight decay of 0.1. Following ViTPose [10], we apply linear warmup for the first 2000 iterations with warmup factor of 10-3, Furthermore, perform gradient clipping to prevent overfitting.
7

---

Table 3: Comparison with SOTA Methods on CrowdPose dataset. BoldFace represents the best score. Underline represents the second-best score.
Method
Input Resolution
AP
Also
Al75
APE
APM
APμ
Mask-RCNN 4
256 x 192
57.2
83.5
60.3
69.4
57.9
45.8
AlphaPose
256 x 192
61.0
81.3
66.0
71.2
61.4
51.1
SimpleBaseline
13
256 x 192
60.8
81.4
65.7
71.4
61.2
51.2
CrowdPose 6
256 x
66.0
84.2
71.5
75.5
66.3
57.4
Hourglass- 104
384 x 288
65.2
85.9
69.5
KAPAO-L 45
384 x 288
68.9
89.4
75.6
76.6
69.9
59.5
HRNet- 448 12
384 x 288
69.3
89.7
75.6
77.7
70.6
57.8
Transpose-H 22
384 X 288
71.8
91.5
77.8
79.5
72.9
62.2
HRFormer-B 23
384 x 288
72.4
91.5
77.9
80.0
73.5
62.4
TokenCLIPose
384 x 288
76.2
93.9
82.4
83.3
77.4
66.1
Improvement
3.8%
2.4%
4.5%
3.3%
3.9%
3.7%
Table 5: Effect of Attention Mechanisms on the overall accuracy.
Table Effect Text Prompts on the stick accuracy.
Attention-Mechanise
Mean
Prompt type
Stick Accuracy
Mean
Intention 46
75.14
No text 63.37
72.93
Self-attention
76.28
Single Prompt 67.41
75.24
Cross-attention 76.31 f
Prompt Ensemble
69.97 2.56%
76.28 1
1.04%
Results. Table presents a comparison between established pose estimation techniques and our method. As shown in the table, TokenCLIPose outperforms all the top-down approaches by 3.8% demonstrating its efficacy predicting unseen keypoints robustly. We also depict the qualitative results of TokenCLIPose to depict its robustness to occlusion. It is notable that even in scenarios where we only see the side- view of humans, TokenCLIPPose estimates the pose reliably.
4.4 Ablation Studies
We conduct comprehensive ablations to study the effect of our design choices and verify the impact of each module on our proposed TokenCLIPose, For consistency, all the ablations are conducted on ice hockey dataset unless specified otherwise.
Gains from Each Modality. The influence of each modality on the proposed model's performance is shown in Table As evidenced by performance improvement of 3.35%, the inclusion of text tokens plays significant role in enhancing the pose estimation accuracy. On the other hand, we find that including the location tokens do not improve the performance significantly, only by a small margin of 0.45%.
Table 4: Effect of Different Modalities on the overall accuracy.
Text tokens (F Fext))
Location tokens (F 10c) Image tokens (F via))
Mean
X
72.48
X
72.93
X
75.72
V
/
76.28
Do we need to treat each modality heterogeneously? We probe whether multimodal tokens need to treated heterogeneously by testing out different attention mechanisms for our transformer decoder. As shown in Table 5 utilizing self-attention directly on all the tokens produces similar results to performing aelf-artention separately on image, location and text tokens, and then applying cross-attention. Therefore, is not necessary to treat the tokens from each modality separately. We

---

Figure 4: Qualitative Results of our TokenCLIPose the CrowdPose dataset.
hypothesize that this could be due to all the tokens being projected to the same joint embedding space, thereby eliminating the need to process them differently.
Impact of Text Prompts. We investigate the influence of language on our model's 's efficacy on the ice hockey dataset by varying the degree and complexity of text prompts that we use. Starting from randomly initializing the text tokens instead of using CLIP embeddings. we study the effects of using single prompts and the prompt ensemble technique proposed 335 ImageNet classification. evident from Table [6) that incorporating text instead of randomly initializing text tokens results in significant improvement of 2.31% % in the overall accuracy. Furthermore, using prompt ensemble techniques improve the accuracy of the model by 1.04%, showcasing the importance of the quality of text prompts.
Influence of Bounding Boxes. We evaluate the influence of bounding box predictions on our model's performance by using ground truth bounding boxes obtained from ground truth human poses and the bounding boxes from the FasterRCNN object detector. The results are showcased in Table The above table illustrates our model's 's robustness to the quality of bounding boxes, as using ground truth bounding boxes improves the model's performance by a small margin of 2.47%.
Table 7: Impact Bounding Boxes on the overall accuracy.
Bounding boxes
Mean
Faster-RCNN [47]
76.28
Ground Truth 78.75 2.4%
Conclusion
In this work, we proposed TokenCLIPose, an innovative solution for robustly predicting human and extension poses. Instead of explicitly modeling extensions within the bounding box, we reformulated the extension pose estimation as an unseen keypoint prediction problem. We leverage the power of large pre- pre-trained VLMs in augmenting the spatial information of unseen keypoints. We showcased that capturing relationships between multimodal tokens is more effective than aligning image features to text tokens. To evaluate the effectiveness of TokenCLIPose in predicting extension keypoints, we curated real-world datasets for ice hockey and Lacrosse. We significantly outperform existing top- down methods on these datasets (by 4.36% and 2.35%, respectively). Additionally, we achieve 3.8%

---

improvement over prior top-down networks on the CrowdPose dataset. This shows TokenCLIPose''s flexibility to predict unseen keypoints within the bounding box as well. Future work will focus on curating and training more extensive and diverse datasets for human and extension pose estimation tasks.
Acknowledgement
This work was supported in part by Stathletes, Compute Canada, the Natural Sciences and Engineering Research Council of Canada and MITACS.
References
[1] H. Neher, K. Vats, A. Wong, and D. A. Clausi, "Hyperstacknet: hyper stacked hourglass deep convolutional neural network architecture for joint player and stick pose estimation in hockey." 2018 15th Conference on Computer and Robot Vision (CRV), pp. 313-320, 2018. [Online]. Available:
[2] D. C. Luvizon, D. Picard, and H. Tabia, "2d/3d pose estimation and action recognition using multitask deep learning," " in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 5137-5146.
[3] B. Wan, D. Zhou, Y. Liu, R. Li, and X. He, "Pose-aware multi-level feature network for human object interaction detection," " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 9469-9478.
[4] Z. Liang, J. Liu, Y. Guan, and J. Rojas, "Pose-based modular network for human-object interaction detection," arXiv preprint arXiv:2008 02042, 2020.
[5] S.-H. Zhang, R. Li, X. Dong, P. Rosin, Z. Cai, X. Han, D. Yang, H. Huang, and S.-M. Hu, "Pose2seg: Detection free human instance segmentation." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[6] Li, C. Wang. H. Zhu, Y. Mao, H.-S. Fang, and C. Lu, "Crowdpose: Efficient crowded scenes pose estimation and new benchmark," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 10863-10872.
[7] K. Ludwig, P. Harzig, and Lienhart, "Detecting arbitrary intermediate keypoints for human pose estimation with vision transformers." 2022 IEEE/CVF Winter Conference on Applicationn of Computer Vision Workshops (WACWW), pp. 663-671, 2022. [Online]. Available:
[8] B. Yao and L. Fei-Fei, "Modeling mutual context of object and human pose in human-object interaction activities," in 2010 JEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, 2010, pp. 17-24.
[9] W. Li, Z. Wang, B. Yin, Q. Peng, Y. Du, T. Xiao, G. Yu, Lu, Y. Wei, and J. Sun, "Rethinking on stage networks for human pose estimation," ArXiv, vol. abs/1901.00148, 2019. [Online]. Available: httpss//app.emaatticcchhe orw.ch/oa..roooooooosD/////7777
[10] Y. Xu, J. Zhang, Q. Zhang, and D. Tao, "Vitpose: Simple vision transformer baselines for human pose estimation." " ArXiv, vol. 1bs/2204.12484, 2022. [Online]. Available:
[11] Y. Li. S. Zhang, Z. Wang. S. Yang, W. Yang, S. Xia, and E. Zhou, "Tokenpose: Learning keypoint tokens for human pose estimation." 2021 IEEF/CVE International Conference Computer Vision (ICCV), pp. 11 293-11 302, 2021. [Online]. Available: htts://ap.emaatticchaaaaroooo/oo
[12] K. Sun, B. Xiao, Liu, and J. Wang. "Deep high-resolution representation learning for human pose estimation," 2019 IEEF/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 5686-5696, 2019. [Online]. Available:
10

---

[13] B. Xiao, H. Wu, and Y. Wei, "Simple baselines for human pose estimation and tracking," ArXiv, vol. abs/ 0bss/884.062000 504.06208, 2018. [Online]. Available: https://pp..emmatiicchho...too oatpaaatiiiiccoooooooooooollllll
4934594
[14] A. Newell, K. Yang. and J. Deng, "Stacked hourglass networks for human pose estimation," " European Conference on Computer Vision, 2016. [Online]. Available:
[15] Zhang. X. Zhu, H. Dai, M. Ye, and C. Zhu, "Distribution-wware coordinate representation for human pose estimation," 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7091-7100, 2019. [Online]. Available: http:://p..nmaallscccccaa/////pp
[16] W. Mao, Y. Ge, C. Shen, Z. Tian, X. Wang, Z. Wang, and A. van den Hengel, "Poseur: Direct human pose regression with transformers." " in European Conference on Computer Vision, 2022. [Online]. Available:
[17] X. Zhang, W. Wang, Z. Chen, Y. Xu, Zhang, and D. Tao, "Clamp: Prompt-based contrastive learning for connecting language and animal pose," in 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Jun. 2023. [Online]. Available:
http://dx.doi.org/ 10. 001100////V1155722222.2222222
[18] S. Hu, C. Zheng, Z. Zhou, C. Chen, and G. R. Sukthankar, "Lamp: Leveraging language prompts for multi-person pose estimation," 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3759-3766, 2023. [Online]. Available:
[19] S. Guo, Q Cai, L. Qi, and J. Dong, "Clip-hand3d: Exploiting 3d hand pose estimation via context-aware prompting," " MM '23. New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: http://doi.org//O0 11/501111145811888311122222
[20] G. Tevet, S. Raab, B. Gordon, Y. Shafir, D. Cohen-or, and A. H. Bermano, "Human motion diffusion model." The Eleventh International Conference on Learning Representations, 2023. [Online]. Available:
[21] J. Huang, Z. Zhu, F. Guo, and G. Huang, "The devil is in the details: Delving into unbiased data processing for human pose estimation," 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5699-5708, 2019. [Online]. Available: wwweaaannticcoolllrrrrroorrroppo
138553
[22] S. Yang, Z. Quan, M. Nie, and W. Yang. "Transpose: Keypoint localization via transformer," 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 782-1121111 792, 2020. [Online]. Available: http://pp.eemaaatiscccooar.rrgo wtt.aar.oorrrooooooD023222222100
[23] Y. Yuan, R. Fu, L. Huang, W. Lin, C. Zhang, X. Chen, and J. Wang, "Hrformer: High-resolution transformer for dense prediction," " ArXiv, vol. abs/21 10.09408, 2021. [Online]. Available:
[24] A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010.1 /999,, 2020.
[25] X. Sun, B. Xiao, S. Liang, and Y. Wei, "Integral human pose regression," " ArXiv, vol. abs/1711.00 1.7119,,, 2017. [Online]. Available:
[26] Li, S. Bian, A. Zeng, C. Wang, B. Pang. W. Liu, and C. Lu, "Human pose regression with residual log-likelihood estimation," 2021 IEFF/CVF International Conference on Computer Vision (ICCV), pp. 11 0100-11014 014, 2021. [Online]. Available: http::/www.aaaissooooooooooo..aa
[27] K. Li, S. Wang, X. Zhang, Y. Xu, W. Xu, and Tu, "Pose recognition with cascade transformers." in Proceedings of the JEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2021, pp. 1944-1953.

---

[28] P. Panteleris and A. Argyros, "Pe-former: Pose estimation transformer." in International Conference on Pattern Recognition and Artificial Intelligence. Springer, 2022, pp. 3-14.
[29] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, FEnd-to-ena object detection with transformers," " ArXiv, vol. abs/2005. 12872, 2020. [Online]. Available:
[30] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, "Deformable detr: Deformable transformers for end-to-end object detection." " ArXiv, vol. abs/2010.041599, 2020. [Online]. Available:
[31] R. Khirodkar, V. Chari, A. Agrawal, and Tyagi, "Multi-instance pose networks: Rethinking top-down pose estimation," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 3122-3131.
[32] M. Zhou, L. Stoffl, M. W. Mathis, and A. Mathis, "Rethinking pose estimation in crowds: Over- coming the detection information bottleneck and ambiguity," in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023, pp. 14 689-14 699.
[33] D. Wang and S. Zhang, "Contextual instance decoupling for robust multi-person pose estimation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 11 060-11 068.
[34] S. Park and J. Park, "Localizing human keypoints beyond the bounding box," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1602-1611.
[35] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark et al., "Learning transferable visual models from natural language supervi- sion," in International conference on machine learning. PMLR, 2021, pp. 8748-8763.
[36] Gao, Geng, R. Zhang. T. Ma, R. Fang. Y. Zhang, H. Li, and Y. Qiao, "Clip-adapter: Better vision-language models with feature adapters," International Journal of Computer Vision, vol. 132, no. 2. pp. 581-595, 2024.
[37] Y. Wei, Y. Cao, Z. Zhang, H. Peng, Z. Yao, Z. Xie, H. Hu, and B. Guo, "iclip: Bridging image classification and contrastive language-image pre-training for visual recognition," 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2776-2786,
2023. [Online]. Available: https//app.eemattischolaaa http:///w.oaaaaooooooooooo/oa/aa ConpusID::55266752
[38] R. Zhang, Z. Wei, R. Fang, P. Gao, K. Li, Dai, Y. J. Qiao, and H. Li, "Tip-adapter: Training-free free adaption of clip for few-shot classification." ArXiv, vol. http//207009191 2022. [Online]. Available:
[39] R. Zhang, Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y. Qiao, P. Gao, and H. Li, "Pointelip: Point cloud understanding by clip," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 8552-8562.
[40] Y. Rao, W. Zhao, G. Chen, Y. Tang, Zhu, G. Huang. J. Zhou, and J. Lu, "Denseclip: Language- guided dense prediction with context-aware prompting," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 18 0802-18809..
[41] T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski, "Vision transformers need registers," arXiv preprint arXiv: 16588, 2023.
[42] L. Dinh, J. N. Sohl-Dickstein, and S. Bengio, "Density estimation using real nvp." ArXiv, vol. nad/160550803. 2016. [Online]. Available: https//pppeeeaaansscccoooorrrroo
[43] K. He, G. Gkioxari, and P. Dollár, "Girshick ross. mask r-cnn," in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961-2969.
[44] J. Li, W. Su, and Z. Wang, "Simple pose: Rethinking and improving bottom-up approach for multi-person pose estimation," Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 07, 2020, pp. 11 354-11145 361.
12

---

[45] W. McNally, K. Vats, A. Wong, and J. McPhee, "Rethinking keypoint representations: Mod- eling keypoints and poses as objects for multi mulli-pprsonn human pose estimation." arXiv preprint 1.Xiv2211.00857, 2021.
[46] M. Garnelo and W. M. Czarnecki, "Exploring the space of key-value-query models with intention," arXiv preprint vrXiv:2305.10203 2023.
[47] S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks," Advances in neural information processing systems, vol. 28, 2015.
13

---

A
Appendix
In this appendix, provide more qualitative results our ice hockey and Lacrosse datasets. Furthermore, we provide a quantitative comparison with multimodal network (118).
A.1 Comparison with Multimodal Pose Estimators
To investigate the effectiveness of our technique of leveraging text with existing multimodal methods, we compare TokenCLI Cakse''' performance with multimodal counterpart LAMP [8]. Though LAMP does not follow the top-down approach, it is the only network that uses language for 2D human pose estimation. Thus, we compare and highlight the results Table g As shown in the table, it is evident that TokenCLIPose outperforms LAMP on the CrowdPose dataset by 4.8%. This corroborates our claim that leveraging text as supervisory signals to guide image features provides better performance than aligning image features to the text tokens.
Table 8: Comparison with LAMP on CrowdPose dataset
Method
Input Resolution
AP
AP'o
AP7
Al'g
APM
APи
LAMP
[8
512 512
71.4
90.3
77.1
77.9
72.1
64.2
TokenCLIPose
384 x 288
76.2
93.9
82.4
83.3
77.4
66.1
A.2 Qualitative Results
We showcase additional qualitative results our ice hockey dataset visualizing the effectiveness of our approach. Furthermore, we also demonstrate generalization capabilities of our model in transferring to Lacrosse dataset. The impact of language visible in our model's robustness to domain changes.
To further illustrate the effectiveness of our approach, we present additional qualitative results on the ice hockey dataset in Figure Furthermore, we demonstrate the model's generalization capabilities by achieving strong performance on Lacrosse dataset, highlighting the impact of language on the model's 's robustness to domain changes.
(a) ice hockey
(b) Lacrosse
Figure 5: Qualitative Results of TokenCLIPose for the extension pose estimation task on our ice hockey and Lacrosse datasets. We plot the pose of extensions outside the cropped image understand how TokenCLIPose works.
14

---

B Broader Impact Statement
Our work provides new way to reconsider the pose estimation task predict -ut-of-bounding box keypoints. This improves our understanding of why joint hockey stick pose prediction along with player pose is difficult for existing top-down networks and presents one way mitigate the issues. This understanding should enable the application of top-down solutions for multi- muntiiinstannee pose estimation which encounters similar issues. Furthermore, accurate stick pose prediction implies that hockey teams can make more data-driven decisions to improve their teams' performance without incurring additional overhead expenses. This also implies that the use of invasive technology such as infrared sensors can be avoided to large extent to robustly analyze players and teams.
This can be leveraged in various domains where humans closely interact with objects, such as shoveling, where it could be used to study fatigue and biomechanics. Furthermore, as shown in our experiments on the CrowdPose dataset, TokenCLIPose also reliably predicts the pose of a human when there is high human-human interaction. Hence, this can be used for pose estimation in highly crowded scenarios, such as surveillance and monitoring people crowds.
15

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We demonstrate and reason out the claims that we make in the abstract by rigorous quantitative and qualitative experimentation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [No]
16

---

Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
Our paper contains theoretical proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We specify the training and model details explicitly.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We do work with an institution which has full access our work. However, with their permission, we could release the dataset and models soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/public/eee. www.ddddeeeooeuuummmcecom ssi ssionyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We present all the experimental details necessary for reproduction.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Following prior works in 2D pose estimation, do not report standard deviations but clearly mention the evaluation metrics and report our performance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the GPU that we run the experiments on and all other details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
19

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htte////ih/////////////////////a
Answer:
[Yes]
Justification:
We conform to the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work strictly restricted to 2D pose estimation which is mainly used for scene understanding, activity recognition and sports analysis. Hence, do not see any negative socictal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeett of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
20

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The datasets that utilize, apart from the ones we created are all benchmark datasets who are cited properly.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pww.::s...aaseiiiia...... thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
We clearly mention how curate our datasets and create our models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
21

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "6cBTs5E0TY", "reviewer": "Reviewer_9Npr", "summary": "This paper introduces the problem of estimating the keypoints of humans together with a stick-like object that the human interacts with, i.e. hockey stick or lacrosse stick (which is referred to as \"extension\" in this paper). The paper claims that prior works on 2D human pose estimation cannot be naively extended to this task by simply using a larger cropped image containing the human and the object together, because it would introduce unnecessary features and hurt performance. To resolve this issue, the paper proposes the TokenCLIPose method, which focuses solely on human using a tight crop bounding box and utilizes keypoint-specific text embeddings from CLIP to predict out-of-bounding-box keypoints of the stick objects. Specifically, it uses a CNN to extract features from images and obtain a coarse estimation of human keypoints in the bounding box with an MLP. Subsequently, it feeds the image features, coarse keypoints features and the keypoint-specific text embeddings of the stick keypoints from a CLIP text encoder to a transformer to obtain a final keypoint estimation result. The experimental results indicate their method outperforms previous 2D human keypoint estimation methods in their newly curated datasets for this new setting, as well as on the CrowdPose dataset with multi person scenarios that have severe occlusion.", "questions": "In addition to the questions contained in the previous section:\n1. How is the visualization of Figure 1(b) obtained?\n2. For the ablation study in Table 4, what is the result of only combining text token and image token?", "limitations": "The paper shows failure cases, but only limited quantitative results are shown and there is no discussion about these failure cases.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is reasonably well-structured and the motivation and approach are described well.\n2. The integration of features from foundation models, like CLIP, for keypoint detection is original in general, but I am unsure why it would be beneficial in the particular application scenario proposed in this paper.\n3. The experiments on various dataset show that the proposed method outperforms related works.", "weaknesses": "1. The proposed problem of detecting keypoint of cropped out objects seems artificial, and it remains unclear if predicting the keypoints of cropped objects is actually necessary. One important ablation study that needs to be shown is to train TokenCLIPose on images that are not artificially cropped, and instead contain both the human and the stick together. It seems that if the text embedding is helpful for locating keypoints of unseen objects, it should also help when the in the bounding box.\n2. The description of adopting the keypoint-specific text embeddings for predicting the keypoints of unseen keypoints is unclear. Why should the text embeddings contain any information of the location of unseen objects in not explained and not studied sufficiently in the experiments.\n3. How are the baseline methods in the experiments adapted to the newly proposed task? This is not discussed at all. The number of keypoints for the extensions seems to vary between experiments. How do you adapt human pose estimation methods for this new task and why do these methods fail in this setting? This should be explained in more detail.\n4. The proposed method should be compared to more recent works like ViTPose++ (TPAMI 2023) and BUCTD (ICCV 2023). The latest works used for comparison in the submission are ViTPose (NeurIPS 2022) and HRFormer (NeurIPS 2021). As this paper works on a different setting than classical human pose estimation, it is important to discuss why existing 2D human pose estimation methods are limited in dealing with this setting. \n5. What are the text prompts used for the Text-based Keypoint Encoder? There is no explanation regarding this essential part of the method.\n6. This method is only verified with simple objects like hockey sticks and lacrosse sticks. A core open question is how the method would generalize to more complicated structures like chairs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1. **Training TokenCLIPose with sticks included:** To assess TokenCLIPose's performance on uncropped images, we extended our experiments on the uncropped images (images that contain both players and their sticks). As shown in Table 2 of the rebuttal document, the performance drops by 4.51\\% resulting in a mean accuracy of 71.77\\%. These results reinforce our hypothesis that expanding the field-of-view introduces unnecessary visual features which hinder model accuracy.\n\n---\n\nW2. **Impact of text prompts:** This is addressed in C3 of the author rebuttal.\n\n--- \n\nW3. (a) **Baseline training details:** Baseline methods are trained from scratch on the extended bounding boxes (boxes that contain human and stick). All models are trained on the Ice hockey dataset to predict 15 keypoints (12 human + 3 stick). For the zero-shot setting, we directly use the ice hockey trained models on our Lacrosse dataset and infer predictions. Specifically, we discard the 15th keypoint predicted by the model, since the Lacrosse dataset has only 14 points (12 human + 2 stick). The human keypoints are the same for both the Lacrosse and Ice hockey datasets.  We understand that we have failed to include this detail and will update this in the camera-ready version.\n\n(b) **Why do Baselines fail?** These methods fail in this setting primarily due to the expansion of the bounding boxes, which increases the field-of-view. This introduces unnecessary visual features as depicted in Figure 3 of the rebuttal document. which confuses the model and leads to suboptimal predictions.\n\n---\n\nW4. (a) **Why not ViTPose++ and BUCTD?** ViTPose++  deals with heterogeneous body keypoint categories thereby adopting task-agnostic and task-specific feed-forward networks in transformers. BUCTD uses a hybrid approach that combines the strengths of bottom-up and top-down methods. Thus, since we focused on improving over top-down approaches with task-specific fixed keypoints we avoided both these frameworks.\n\n(b) **ViTPose++ results:** However, for completeness, we trained ViTPose++ on the ice hockey dataset which resulted in a mean PCKh of  72.09\\% underperforming by 4.19\\% when compared to TokenCLIPose as shown in Table 3 of the rebuttal document.\n\n---\n\nW5. **What are the text prompts?** This has been explained in C2 of the author rebuttal.\n\n---\n\nW6. **Experiments on “simpler” objects:** To the best of our knowledge, this is the first work to address the problem of extension pose estimation and create a dataset for the task. Since pose estimation is a task that is heavily dependent on labels, we spent a lot of time collecting precise manual annotations. Also, the setting in which the objects are considered is highly dynamic, making the task extremely challenging. This can be seen in Tables 1 and 2 of the main paper as existing SOTA methods struggle to estimate the object pose. Hence, we politely disagree with the reviewer that we consider simple objects and find it unfair to criticize us on the data front as we are the first to address the problem.\n\n---\n\nQ1. **Figure 1(b) explanation:** The visualization in Figure 1(b) of the main paper is a t-SNE plot of each of the keypoint prompts used to describe the human keypoints.\n\n---\n\nQ2. **Text and image tokens ablation:** Combining text and image tokens resulted in a mean PCKh of  75.72\\%. Thank you for spotting the missed ablation. We will add it in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RlFPTxrKAE", "reviewer": "Reviewer_F7X8", "summary": "The paper argues that using images including both humans and interacting objects introduces unnecessary visual features that hinder pose estimation performance. To address this, the paper claims that treating objects as unseen to predict interacting object poses can achieve better results and proposes a TokenCLIPose solution. The method is evaluated on the Hockey and Lacrosse datasets, which contain images from a total of 11 video clips.", "questions": "See Weaknesses.", "limitations": "No discussion on limitations is provided.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is generally clear and the method uses language information to improve pose estimation.", "weaknesses": "The experiments and comparisons are not convincing and the experiments are weak.\n\n(1) The paper conducts its main experiments on the Hockey and Lacrosse datasets. However, these datasets are too small, comprising only 11 video clips.\n\n(2) The paper claims that predicting object poses using human-only images performs better than using human-and-object images. However, the paper lacks in-depth analysis, and the experiments are insufficient to support this claim. The reviewer disagrees, arguing that observation is more accurate than imagination when an appropriate framework is used.\n\n(3) The major contribution of the paper is not clearly defined.\n\n(4) The paper did not discuss and compare with bottom-up methods, which are popular for human pose estimation.\n\n(5) In Line 136, the text encoder simply encodes class-related information/names of each keypoint, which the reviewer argues cannot tackle the challenge of unseen keypoints in the image.\n\n(6) Typos, such as \"we curate two new sports dataset\" and \"Experimentations\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1. (a) **Custom dataset size:** To the best of our knowledge, this is the first work to address the problem of extension pose estimation and create a dataset for the task. Since pose estimation is a task that is heavily dependent on labels, we spent a lot of time collecting precise manual annotations. Though the reviewer might feel 11 video clips might be less, the number of annotations that we collected still sum up to a sizable 11630 images for ice hockey and around 300 for Lacrosse (used for zero-shot pose estimation). \n\n(b) SOTA results on large dataset (CrowdPose): Moreover, one of our main experiments is conducted on the CrowdPose dataset (Table 3 of the main paper) is a benchmark dataset from the literature that has atleast four times more human images when compared to the Ice hockey dataset.  The results show that TokenCLIPose outperforms existing state-of-the-art architectures on this CrowdPose dataset as well. This shows that irrespective of the dataset size, our network performs better when keypoints are unseen. \n\n---\n\nW2. **Exclusion of stick from the bounding box:** This is addressed in C1 of the author rebuttal. \n\n---\n\nW3. **Main contributions:** Our main contributions are as follows:-\n- We reformulate the extension pose estimation problem as an out-of-bounding box keypoint detection problem by explicitly leaving out the extensions, and leverage the human features and keypoint-specific text prompts to robustly estimate the human and extension pose.\n- Existing multimodal pose estimators try to align the image features with the keypoint-specific text embeddings using a contrastive loss. However, upon visualizing these text embeddings, we find out that this is not desirable as the embeddings do not maintain global structure (as shown in Fig. 1(b) of the main paper). Hence, we use them as additional priors and initialize our learnable keypoint tokens (also referred to as text tokens) with these text embeddings rather than biasing our model completely towards these text embeddings.\n- Realizing the lack of datasets for extension pose estimation, we introduce 2 datasets: an ice hockey and a Lacrosse dataset. Furthermore, we show that our model outperforms the existing SOTA approaches by 4.36\\% and 2.35\\% respectively. \n- We further demonstrate our model’s robustness to occlusions by conducting experiments on the CrowdPose dataset, where we outperform the SOTA top-down approaches by 3.8\\%.\n\n---\n  \nW4. **Why not bottom-up?** Most of the SOTA techniques [2-5] implicitly follow a top-down approach. This is mainly because they lead to more accurate predictions [1]. Hence, we compare with top-down approaches. \n\n---\n\nW5. **How do text prompts contain exact location?** This is addressed in C3 of the author rebuttal.\n\n---\n\n\nW6. Thanks for spotting the typos. We will update it for the camera-ready submission.\n\n---\n\nReferences:-\n\n[1]: Jiang et al., RTMW: Real-Time Multi-Person 2D and 3D Whole-body Pose Estimation, arXiv 2024. \n\n[2]: Xu et al., ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation, NeurIPS 2022.\n\n[3]: Sun et al., Deep High-Resolution Representation Learning for Visual Recognition, CVPR 2019.\n\n[4]: Li et al., TokenPose: Learning Keypoint Tokens for Human Pose Estimation, ICCV 2021.\n\n[5]: Yang et al., TransPose: Keypoint Localization via Transformer, ICCV 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "50uIWO9NEj", "reviewer": "Reviewer_AetR", "summary": "This paper proposes a text-guided keypoint localization method that can detect keypoint that is out-of-input image. The proposed method only crops person area and abandons the object area, then adopts language prior to predict keypoint that is out-of-input image. To verify the effectiveness of the proposed method, this paper also introduces a Ice-Hockey keypoint detection dataset. Experiments on CrowdPose demonstrate the effectiveness of the proposed method.", "questions": "In Weakness", "limitations": "The authors say they show the failure cases of the prooposed method, but I do not find them in the manuscript.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tUtilizing text to represent the spatial relationships of keypoints is reasonable and straightforward. The proposed TokenCLIPose can effectively model the keypoint relationship and estimate unseen keypoints accurately.\n2.\tThis paper is well-written and easy to understand.", "weaknesses": "The proposed method is not well suited to the introduced tasks such as  Ice-Hockey. Abandoning the visible object is not a good solution because the model cannot make perfect prediction on out-side objects. For occluded keypoints, it is not bad because we can only guess the position, it is acceptable that the prediction is not so accurate. But for visible keypoints of hockey stick in Fig. 1(a), the predicted results are not acceptable. A better way to handle unnecessary visual features is to separate the object and person and perform keypoint localization on each individually. \n\nI think the proposed method can be used to perform the occluded/out-of-box keypoint localization, but not to solve the Human-Object Keypoint Localization tasks such as Ice-Hockey. The introduced task and dataset cannot fully demonstrate the effectiveness of the proposed method. It is better to introduce the method from the perspective of occluded or out-of-box keypoint localization, abandoning the Ice-Hockey tasks and background and conduct more experiments on occluded  keypoint localization benchmarks such as OCHuman.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1. (a) **Excluding the sticks:** The concern of abandoning the visible object is addressed in the author's rebuttal. \n\n(b) **Individual localization of human and stick:** We would like to thank the reviewer for raising an insightful question of estimating the pose of human and their extension separately. While this might seem interesting, works on human-object interaction [1-3] and 2D pose estimation [4] showcase that the pose of the human and the extension are closely related, and can be used to refine each other’s pose. Our experiment on individual and joint pose estimation shown in Table 4 of the rebuttal document also demonstrates an improvement in the accuracy of human pose when jointly trained with the stick. \n \n(c) **Efficacy of our method to solve human-object Keypoint Localization:** We also appreciate the reviewer's suggestion of introducing this paper from the perspective of occluded keypoint localization. However, we disagree with the reviewer’s comment that our method cannot solve human-object keypoint localization. Through our extensive qualitative and quantitative experiments, we have demonstrated the superior performance of our model in comparison to SOTA approaches.\n\n---\n\nReferences:-\n\n[1]: Yao et al., Modeling mutual context of object and human pose in human-object interaction activities, CVPR 2010.\n\n[2]: Gupta et al., Observing Human-Object Interactions: Using Spatial and Functional Compatibility for Recognition, TPAMI 2009.\n\n[3]: Delaitre et al., Learning person-object interactions for action recognition in still images, NeurIPS 2011.\n\n[4]: Neher et al., HyperStackNet: A Hyper Stacked Hourglass Deep Convolutional Neural Network Architecture for Joint Player and Stick Pose Estimation in Hockey, CRV 2018", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QsFUKlypcM", "reviewer": "Reviewer_9aCe", "summary": "This paper focuses on an interesting and inherent problem in top-down human pose estimation; that is out-of-box prediction in a top-down paradigm. The core of the solution is to utilize the semantic context by giving proper text prompts to CLIP. Specifically, a cropped image is first given to a pre-trained CNN model to extract initial features and obtain a coarse prediction. Then the visual features from CNN, the coarse prediction of the person, and the semantic information from CLIP are concatenated and input to a transformer decoder to obtain the final prediction. The method is capable of solving the occluded points in the crowded scene and breaking the limits from the bounding box in the top-down model.", "questions": "1. One statement in the introduction that I do not agree with is \"Contemporary deep learning-based pose estimation methods predominantly follow a top-down approach\" between line 27 and line 28. Bottom-up and one-stage models are also important parts of human pose methods. In fact, bottom-up models have their own merits such as real-time inference and bounding-box independent predictions. I suggest revising this statement.\n\n2. It would be better if the authors could make a further explanation on using normalizing flow to estimate the human poses. I think it is quite interesting as this method was published in 2021 and is quite rare in recent work. Many other papers like PETR directly regress pose with the decoder. Then it is natural to ask Why use normalizing flow? What is the impact if we directly regress the pose? These two points may need further discussion.\n\n3. What is the exact text prompt used in the given method? \n\n4. The authors also claimed that the previous method ignored the global spatial relation between points. Yet it is still unclear to me how the author's method solves this problem. In the introduction, it is claimed that transformers are used to solve the problem. But it is too general to convince me.", "limitations": "No negative societal impact are found.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper focuses on a very interesting topic that indeed requires a deeper investigation.\n\n2. Having semantic information from text prompts is a quite novel and effective way of solving the occlusion and out-of-box points.\n\n3. The paper is in general well-written and easy to follow.", "weaknesses": "1. Poor visualisation. Qualitative results are only compared with HRNet. More visualization should be shown to validate the effectiveness of the proposed method.  \n\n2. A few images are very ambiguous. Especially, the Figure 1. (b). There is no notation for x, y, and colors. The caption said it is the text embedding and such embedding loses the global structure. Why did it lose the global structure?\n\n3. In terms of performance, there is indeed an improvement. However, how many of them are coming from the unseen points? i.e. Is the source of improvement from unseen points or a more accurate estimation of the seen points? After all, the whole pipeline is a coarse-to-fine prediction with additional text information. Therefore, the seen points should also benefit. If the major improvement does not come from the unseen points, then the major problem proposed by this paper still remains unsolved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "W1. **Qualitative Results:** More qualitative comparisons with ViTPose and HRNet are shown in Figure 1 of the rebuttal document. From the figure, it is evident that excluding the stick from the bounding box is much more beneficial as it reduces the noise present in the image. \n\n---\n\n\nW2. **t-SNE visualization of text embeddings:** Figure 1(b) of the main paper depicts the t-SNE visualization of the text embeddings generated from keypoint-specific text prompts. The two axes x and y represent the two reduced dimensions dimension1 and dimension 2 respectively. The different colours represent whether they are upper-body or lower-body joints, with red representing upper-body joints, blue representing lower-body joints and green representing the hip joints. In the paper, we mention that from the figure, it is evident that while these embeddings maintain positional structure within the upper-body joints (the shoulders are placed above the elbows which are placed above the wrists, following the pose of a human standing normally) and the lower-body joints (knees placed above ankles), they fail to maintain positional structure between the upper-body and lower-body joints (elbows and wrists are placed below knees and ankles in the plot which does not follow the pose of a human being). This has been explained in lines 41 to 46 in the introduction of the paper.\n\n---\n\n\nW3. **Improvement on unseen keypoints:** As demonstrated in Table 1, the unseen keypoints (stick top, stick heel, and stick toe) contribute significantly to the overall improvement of TokenCLIPose. Specifically, an  improvement of 4%, 6.4%, and 7.35% is observed on the three keypoints respectively when compared to the second-best model (ViTPose). On the other hand, the seen keypoints (human) only improve by 1.76\\%. Similar trends are observed when experimentation was done on the Lacrosse dataset in Table 2. Hence, it is evident that major improvements come from the unseen keypoints.\n\n---\n\nQ1. We intended to say that most of the state-of-the-art works [1-4] follow a top-down approach. However, we will revisit this statement and update it in our camera-ready version.\n\n---\n\n\nQ2. Thank you for noting our use of normalizing flows and asking an insightful question! We would like to explain this choice of ours by first stating the limitations of standard regression losses and then explain the reason behind using normalizing flows. \n**Limitations of standard loss functions:** Directly regressing poses using a standard $\\ell_2$ or $\\ell_1$  loss is not desirable as it is vulnerable to ambiguous and noisy labels [5] which is prevalent  in our setting due to high motion-blur. This is primarily because applying a standard distance function as our loss function is equivalent to assuming the distribution of the output probability distribution. For example, the loss function when we assume the output probability distribution to be a Gaussian with constant variance is equivalent to the standard $\\ell_2$ loss. Similarly, the loss function when we assume the output probability distribution to be a Laplacean with constant variance is equivalent to the standard $\\ell_1$ loss. \n\n**Why normalizing flows?** On the other hand, we learn the deviation between the predicted probability distribution and ground-truth probability distribution, which is estimated by a flow model, and minimize the deviation using MLE. By doing so, we get more robust predictions. We validate the impact of this Residual log likelihood (RLE) loss through an ablation study by replacing it with the standard $\\ell_2$ loss. Upon doing so, we get a mean PCKh accuracy of 75.22\\%, resulting in a 1.06\\% performance drop. \n\n---\n\n\nQ3. Addressed in C2 of the author rebuttal\n\n---\n\n\nQ4. **Our difference over existing multimodal methods:** Existing multimodal 2D pose estimators [6] align the image features around each keypoint to their corresponding text embeddings using a contrastive loss. This pushes the image features to be similar to the text embeddings. However, this is not desirable as the text embeddings do not capture global positional structure (showcased in figure 1(b) of the main paper and discussed in W2). This is what we explain in lines 41-46 of the main paper. In order to overcome these limitations, we use these text embeddings as priors and initialize our learnable keypoint tokens (also referred to as text tokens) using these text embeddings. This provides our model with the inductive priors required for robust prediction without biasing them completely towards the text embeddings. This is mentioned in lines 47-51 in the main paper.\n\n---\n\nReferences:-\n\n[1]: Xu et al., ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation, NeurIPS 2022.\n\n[2]: Sun et al., Deep High-Resolution Representation Learning for Visual Recognition, CVPR 2019.\n\n[3]: Li et al., TokenPose: Learning Keypoint Tokens for Human Pose Estimation, ICCV 2021.\n\n[4]: Yang et al., TransPose: Keypoint Localization via Transformer, ICCV 2021.\n\n[5]: Li et al., Human pose regression with residual log-likelihood estimation, ICCV 2021.\n\n[6]: Hu et al., LAMP: Leveraging Language Prompts for Multi-person Pose Estimation, IROS 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We would like to thank all the reviewers for providing constructive feedback that helped us improve the paper. We are delighted that the reviewers recognized the originality of our work in incorporating language to predict out-of-bounding box keypoints (R1, R4), significance of our focus on the interesting topic (R4), the effectiveness of our method in modeling spatial relationship between keypoints (R3), and the substantial improvement over the existing SOTA (R1, R4). Throughout the rebuttal, we refer to the reviewers in the following manner: {Reviewer 9Npr: R1, Reviewer F7X8: R2, Reviewer AetR: R3, Reviewer 9aCe: R4}.

We address some of the common concerns here. The concerns have been paraphrased as multiple reviewers had similar concerns and questions.

---

### C1. Counter-intuitiveness of excluding the extension from the bounding box (R2 and R3) ###
**Training TokenCLIPose with sticks included:** To showcase the effectiveness of cropping out the hockey stick, we train our model on the extended bounding boxes (containing both the human and stick) and report the results in Table 2 of the rebuttal document. From the table, we can see that including the stick in the bounding box deteriorates our model’s performance by 4.51\%, resulting in 71.77\% mean accuracy. 

**Impact of noise:** Furthermore, in order to investigate the impact of noise on the final predictions, we record the performance of ViTPose and TokenCLIPose under two different data split-ups in Table 1 of the rebuttal document. Specifically,  the ice hockey dataset is divided into 2 sets, one with images that predominantly contain visible keypoints (referred to as A) and the other with images that contain both visible and noisy (partially visible) keypoints (referred to as B). The results demonstrate that ViTPose outperforms TokenCLIPose (ours) by 2.49\% when only split-up A was used but underperforms by 4.36\% when split-up B is used. While our method might seem counter-intuitive, these experiments explicitly demonstrate that existing models aren't robust enough to noise necessitating the need for smaller bounding boxes. The distribution of the visibility of keypoints across frames is shown in Figure 2 of the rebuttal document.

---

### C2. What are the text prompts for each keypoint? (R1 and R4) ###
The text prompts corresponding to each keypoint used in TokenCLIPose are as follows:-

| Joint          | Text Prompt                                                                       |
|----------------|-----------------------------------------------------------------------------------|
| Left Shoulder  | left shoulder of a person                                                         |
| Right Shoulder | right shoulder of a person                                                        |
| Left elbow     | left elbow of a person                                                            |
| Right elbow    | right elbow of a person                                                           |
| Left wrist     | left wrist of a person                                                            |
| Right wrist    | right wrist of a person                                                           |
| Left hip       | left hip of a person                                                              |
| Right hip      | right hip of a person                                                             |
| Left knee      | left knee of a person                                                             |
| Right knee     | right knee of a person                                                            |
| Left ankle     | left ankle of a person                                                            |
| Right ankle    | right ankle of a person                                                           |
| Stick Top      | top portion of the stick opposite to the blade of a hockey stick held by a person |
| Stick Heel     | intersection of the blade and the shaft of a hockey stick                         |
| Stick Toe      | end portion of the blade that is farthest away from the shaft of a hockey stick   |

---

### C3. How do text embeddings contain any information about the exact location of the unseen keypoints? (R1 and R2) ###
Our choice of using text prompts to tackle the challenge of unseen keypoints is guided by recent works on 3D pose estimation [1] which use language to interpolate missing poses temporally, and image inpainting [2-3]. Furthermore, the authors have demonstrated in Tables 4 and 6 of the main paper that the proposed network drops in performance significantly without text prompts (by 3.35\%).

---

**Summary:** We address a critical yet overlooked problem: joint pose estimation of humans and extensions (objects that humans hold and interact with), and overcome the limitations of existing SOTA top-down approaches on this task. Our approach is counter-intuitive yet impactful: we propose excluding the extension from the bounding box and predicting keypoints beyond the bounding box, thereby challenging the standard practice of including all keypoints within a bounding box. To achieve this, we utilize language priors along with bounding box features through keypoint-specific text prompts. To evaluate our method, we introduce the first dataset with labeled human and extension poses from broadcast ice hockey videos. We also perform zero-shot evaluations on a self-curated Lacrosse dataset to showcase our model’s generalizability. Finally, to showcase our model’s robustness to occlusions, we conduct experiments on the CrowdPose dataset.

Please see our reviewer-specific feedback for more information.

References:-

[1]:  Tevet et al., Human Motion Diffusion Model, ICLR 2023.

[2]:  Ni et al., NUWA-LIP: Language-guided Image Inpainting with Defect-free VQGAN, CVPR 2023.

[3]: Zhang et al., Text-Guided Image Inpainting, MM 2020.

Meta-review: The reviewers agree that the proposed method is clear and that the experiments show reasonable improvement over the state of the art.  There were some questions about whether the range of datasets evaluated was sufficient, but after the rebuttal and discussion, the reviewer closest to the area and the area chair both found this sufficient.  The reviewers make some comments about explaining the context and advantage of the proposed method that should, if possible, be utilized in cleaning up the paper.
Decision: Accept (poster)