SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our method, Context-Aware PoseFormer, is referred to as "CA-PF-backbone name" in tables. We conduct experiments on two benchmarks (Human3.6M [21] and MPI-INF-3DHP [39]). Human3.6M [21] is the most popular benchmark for 3D human pose estimation, where more than 3.6 million video frames are captured indoors from 4 different camera views. MPI-INF-3DHP [39] includes videos collected from indoor scenes and challenging outdoor environments. Both datasets provide subjects performing various actions with multiple cameras. On Human3.6M, we report MPJPE (Mean Per Joint Position Error, where the Euclidean distance is computed between estimation and ground truth) and PA-MPJPE (the aligned MPJPE). For MPI-INF-3DHP, we report MPJPE, Percentage of Correct Keypoint (PCK) within the  $150\mathrm{mm}$  range, and Area Under Curve (AUC). Settings follow previous works [74, 29, 67, 50]. Due to page limitations, we place details about our implementations in Appendix B.
4.1 Comparison on Human3.6M
Accuracy. We compare our single-frame method with both single-frame and multi-frame methods on Human3.6M [21] (Table 1). Previous methods receive CPN-detected [8] 2D joints as input. We also follow this setting to ensure a fair comparison. Benefiting from spatial contextual clues encoded by the CPN pose detector, our approach outperforms state-of-the-art single-frame methods by a huge margin. For instance, Zeng et al. [65] obtain  $47.9\mathrm{mm}$  MPJPE, whereas our CA-PF-CPN achieves  $41.6\mathrm{mm}$  (a  $13.2\%$  error reduction). This result has already surpassed the performance of most modern multi-frame methods. To show the flexibility of our method and to further investigate spatial context
Table 1: Comparison with both single-frame (top) and multi-frame (middle) methods on Human3.6M. MPJPE is reported in millimeters. The best results are in bold, and the second-best ones are underlined.
Method	Venue	2D Pose Detector	Frame	FLOPs (G) for the Lifting Module	MPJPE	PA-MPJPE
Single Frame	GraphSH [60]	CVPR'21	CPN [8]	1	-	51.9	-
HCSF et al. [65]	ICCV'21	CPN [8]	1	-	47.9	39.0
GraFormer [70]	CVPR'22	CPN [8]	1	-	51.8	-
GraphMLP [28]	arXiv'22	CPN [8]	1	0.3	49.2	38.6
Multi-frame	Pavillo et al. [45]	CVPR'19	CPN [8]	243	0.03	46.8	36.5
Liu et al. [32]	CVPR'20	CPN [8]	243	-	45.1	35.6
Wang et al. [56]	ECCV'20	CPN [8]	96	-	45.6	35.5
Zeng et al. [64]	ECCV'20	CPN [8]	243	-	44.8	34.9
PoseFormer [74]	ICCV'21	CPN [8]	81	1.4	44.3	34.6
StridedTrans. [27]	TMM'22	CPN [8]	351	2.1	43.7	35.2
MHFormer [29]	CVPR'22	CPN [8]	351	14.2	43.0	34.4
MixSTE [67]	CVPR'22	CPN [8]	243	277.4	40.9	32.6
P-STMO [50]	ECCV'22	CPN [8]	243	1.5	43.0	34.4
Einfalt et al. [14]	WACV'23	CPN [8]	18	1.0	45.0	36.3
CA-PF-CPN (ours)	CPN [8]	1	0.6	41.6	33.9
CA-PF-HRNet-32 (ours)	HRNet-32 [52]	1	0.6	41.4	33.5
CA-PF-HRNet-48 (ours)	HRNet-48 [52]	1	0.6	39.8	32.7
from various backbones, we change the backbones that generate feature maps: our CA-PF-HRNet-32 [52] achieves  $41.4\mathrm{mm}$  MPJPE; with a larger backbone HRNet-48, the error reduces to  $39.8\mathrm{mm}$ , which is superior to all previous multi-frame methods. For example, this result outperforms MixSTE [67] with 243 frames (39.8 v.s.  $40.9\mathrm{mm}$ , a  $2.7\%$  error reduction) and MHFormer [29] with even 351 frames (39.8 v.s.  $43.0\mathrm{mm}$ , a  $7.4\%$  error reduction). Our method presents highly competitive accuracy in the absence of dense temporal modeling. Moreover, our method consistently gains improvements from advanced 2D pose detectors. We expect our approach's performance to increase further with powerful large visual foundation models, as such foundation models have been attracting increasing attention and interest.
Computational cost. Our method requires far less computation than state-of-the-art methods (see also Table 1). For example, with similar accuracy, MHFormer [29] takes 14.2 GFLOPs, whereas our CA-PF-CPN [8] demands only 0.6 GFLOPs (about  $24 \times$  reduction). Two reasons account for the lightweight property of the proposed approach: (1) Non-temporal input removes large computation for dense temporal modeling. Modeling temporal dependencies using e.g., transformers [74, 67] is computationally heavy especially when the frame number is increased for improved accuracy. (2) We utilize multi-scale feature maps (from pose detectors) in a sparse manner (i.e., with deformable attention [75]). We avoid applying global operations on the whole feature maps, which may introduce unnecessary computational budgets on uninformative backgrounds.
4.2 Comparison on MPI-INF-3DHP
In Table 2, the comparison is also conducted with both single-frame (top) and multi-frame (middle) methods on the MPI-INF-3DHP dataset [39]. We use ground truth 2D pose detection as input, following [74, 29, 50], and HRNet-32, HRNet-48 [52] are adopted as the backbone network to generate multi-resolution feature maps. Our approach outperforms other methods, including the state-of-the-art multi-frame one, P-STMO [50]. P-STMO uses 81 video frames as input, with additional masked joint self-supervised pre-training. In con
Table 2: Comparison on MPI-INF-3DHP.  $T$  indicates the number of video frames used by models.
Method	PCK ↑	AUC ↑	MPJPE ↓
Xu et al. [60] (T=1)	80.1	45.8	-
Li et al. [26] (T=1)	81.2	46.1	99.7
Zeng et al. [65] (T=1)	82.1	46.2	-
PoseFormer [74] (T=9)	95.4	63.2	57.7
MHFormer [29] (T=9)	93.8	63.3	58.0
MixSTE [67] (T=27)	94.4	66.5	54.9
P-STMO [50] (T=81)	97.9	75.8	32.2
CA-PF-HRNet-32 (ours)	98.0	75.4	32.7
CA-PF-HRNet-48 (ours)	98.2	76.3	31.4
trast, our approach has no access to temporal information and does not need an extra pre-training stage. The results verify the generalization ability of our method to different datasets, particularly in challenging outdoor environments.
Table 3: Ablation study on spatial context and each component of our method. Experiments are conducted on Human3.6M with HRNet-32 as the backbone. MPJPE is reported in millimeters.
Step	Context-aware	Inter-joint Modeling	Pose-Context Feature Fusion	Deform. Cont. Extraction	FLOPs (M)	MPJPE ↓
(0)	X	✓	X	X	446.0	51.2
(1)	✓	✓	X	X	448.0	43.5 (7.7↓)
(2)	✓	✓	✓	X	537.3	42.8 (8.4↓)
(3)	✓	✓	✓	✓	609.9	41.4 (9.8↓)
Gains from spatial context and ablations on model components. In this section, we first investigate the gains purely brought by spatial contextual clues from 2D pose detectors and then the effectiveness of each component in our method (please refer to Sec. 3.2 for more details). Specifically, we show how the context-agnostic counterpart of our method, (single-frame) PoseFormer [74] is converted to Context-Aware PoseFormer step by step (step-wise improvements are shown in Table 3).
- Step 0, Context-Agnostic: For PoseFormer, 2D joint coordinates detected by a pose detector are linearly embedded to dimension  $C$  as per-joint representation (i.e., joint embedding), and Inter-joint Modeling is subsequently performed to build up correlations across the joint embeddings. Plain 2D coordinates encode no spatial context for joints that helps reduce ambiguity, PoseFormer (and other lifting-based methods) is thus referred to as "context-agnostic". This serves as the starting point of our method.
- Step 1, Context-Aware but simple concatenation: The first step to make PoseFormer "context-aware" is to incorporate joint-context features into its per-joint representation. We retrieve the multi-scale feature maps produced by the 2D pose detector, simply sample feature vectors on the detected joint locations from each feature map, and project them to dimension  $C$ . For each joint, the sampled context feature vectors that encode joint-centric visual cues are concatenated together with the coordinate embedding that encodes positional information. Even naive involvement (as simple as concatenation) of spatial context clues brings a huge improvement as shown in Table 3: a  $15.0\%$  error reduction (from 51.2 to  $43.5\mathrm{mm}$ ), which demonstrates that leveraging the readily available visual representations learned by upstream backbones is effective and promising.
- Step 2, Context-Aware with Pose-Context Feature Fusion: We promote cross-modality feature fusion by applying transformers to joint embeddings and the sampled multi-scale context features before concatenation, which brings another  $1.6\%$  error reduction.
- Step 3, Final version of our Context-Aware PoseFormer: The context features in step 1 are obtained by directly sampling at the detected joints, which leads to suboptimal performance since 2D pose detectors inevitably introduce noise. Thus, the sampled feature vectors can not well represent the context associated with the joint of interest. We introduce Deformable Context Extraction to mitigate this issue, which adaptively attends to the detected joint along with its surrounding regions using learnable sampling points, which further reduces the error by  $3.3\%$ .
We also offer visualization of learned sampling points (brighter colors denote larger attention weights) in Fig. 4, where they try to approach the ground truth despite the noisy detected input. Note that in training we do not supervise them using ground truth 2D pose. More visualization is in Appendix E.
What kind of spatial context are we looking for? In this part, we go a step further to explore what context is desirable to improve 3D pose estimation accuracy. Our method benefits from multi-resolution feature maps produced by 2D detectors [52, 11, 63], which encode spatial contextual clues of various levels for human joints. For example, HRNet [52] outputs four feature maps in the last stage, which are  $4 \times$ ,  $8 \times$ ,  $16 \times$ ,  $32 \times$  downsampled compared to the input image size, respectively. We investigate which one of the four is most informative for our task via ablation experiments (in Table 4, MPJPE is reported on Human3.6M). Removing  $4 \times$  and  $16 \times$  downsampled feature
Figure 4: In the proposed Deformable Context Extraction module, sampling points attempt to approach the ground truth despite unreliable 2D joint detection.
maps brings the most significant performance drop (a  $3.6\%$  and  $4.1\%$  error increase for each). In contrast, only a  $1.0\%$  error increase is observed when the  $32\times$  downsampled feature map is eliminated.
Generally, high-resolution feature maps contribute more to 3D pose estimation than low-resolution ones. The reason is that low-resolution feature maps tend to keep the high-level semantics of joints, such as joint type, which may not be relevant to our task - to localize human joints in 3D, where 2D-to-3D joint correspondence is pre-defined. However, we can still gain from low-resolution feature maps as they encode most wide-range spatial dependencies, promoting an understanding
Table 4: Ablation study on feature resolutions.
4×	8×	16×	32×	MPJPE ↓
✓	✓	✓	✓	41.4
×	✓	✓	✓	42.9△1.5
✓	×	✓	✓	42.4△1.0
✓	✓	×	✓	43.1△1.7
✓	✓	✓	×	41.8△0.4
What pre-trained features are more desirable?
ImageNet [48] pre-trained backbones (e.g., ResNet [20]) profit a series of downstream tasks [30, 10, 58], yet this seems not applicable to 3D human pose estimation. In Table 5, we replace COCO pretrained backbones in our method with ImageNet classification pre-trained ones, showing a remarkable performance drop. This should be attributed to the large gap between the pre-training task (im
age classification) and the downstream task (3D human pose estimation). Exploring more suitable pre-trained features that can transfer their effectiveness into the 3D HPE task is a promising direction. We place more ablations on backbones and pre-training datasets in Appendix D.
Table 5: Ablation study on pre-training tasks.
Backbone	Pre-training	MPJPE ↓
ResNet-50	2D Pose	45.0
Image Class.	51.4△6.4
HRNet-32	2D Pose	41.4
Image Class.	45.8△4.4
HRNet-48	2D Pose	39.8
Image Class.	43.9△4.1
4.4 Visualization
Figure 5: Frame-wise comparison with MHFormer [29] and our context-agnostic counterpart (Sec. 4.3) on Human3.6M test set (S9 Discussion).  $T$  indicates the frame number used by models.
Spatial context promotes temporal stability. We show a frame-wise comparison on the Human3.6M [21] test set in Fig. 5. Our single-frame approach outperforms competitive MHFormer [29] using 351 frames. Moreover, our context-agnostic counterpart (Sec. 4.3) suffers from temporal instability (i.e., zigzags and error peaks), whereas our method presents a more stable trend. We also provide a qualitative comparison with both methods in Fig. 6. We include more visual comparison on both Human3.6M and MPI-INF-3DHP in Appendix F.
Spatial context improves the robustness against occlusions. We provide a comparison with PoseFormer [74] on in-the-wild videos in Fig. 7. We choose two sets of consecutive video frames where the 2D joint detection fails due to strong self-occlusion. Since PoseFormer only accepts 2D joints as input, its estimation is sensitive to the noise of input 2D poses. On the contrary, our method leverages spatial context from images in addition to 2D joint coordinates to localize joints in 3D. Thus, our method shows more reasonable and robust results despite noisy or even outlier 2D joints.
Figure 6: Visual comparison in hard cases, e.g., self-occlusion and noisy 2D detection. We include more qualitative results in Appendix F.
Figure 7: Comparison with PoseFormer [74] on in-the-wild videos. The 2D pose detector fails to localize 2D joints, given confusing clothing (left) and severe self-occlusion (right). Our method is more robust in such hard cases. False joint detection is indicated by yellow arrows, and the corresponding 3D joint estimation is indicated by orange arrows.
This work presents a new framework that leverages readily available visual representations learned by off-the-shelf 2D pose detectors. Our approach removes the need for extensive long-term temporal modeling excessively employed by existing 3D human pose estimation methods. We further propose a straightforward baseline to showcase its effectiveness: our single-frame model significantly outperforms other single-frame and even multi-frame methods using up to 351 frames while being far more efficient. Our work opens up opportunities for future research to utilize visual representations from well-trained upstream backbones in an out-of-the-box manner.
Limitations. We observed in Sec. 4.4 that incorporating the spatial context of joints improves temporal stability, enhancing consistency and smoothness in the estimated results, even without access to explicit temporal clues. However, we acknowledge for all single-frame methods, including ours, mitigating jitters remains a challenge compared to multi-frame methods that leverage temporal clues. This is primarily due to the non-temporal nature of single-frame methods. A potential solution is to extend our method to model short-term temporal dependencies, which should not introduce unacceptably high costs. We include some preliminary results in Appendix C. Compared to standard lifting-based methods, the memory cost of our method can be higher as we additionally process image features in the lifting stage. Leveraging the visual context of joints in a more lightweight way is our future research direction.
IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13232-13242, 2022. 1, 3, 4, 6, 7, 16
[68] Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N. Metaxas. Semantic graph convolutional networks for 3d human pose regression. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3425-3435, 2019. 4
[69] Qitao Zhao, Ce Zheng, Mengyuan Liu, Pichao Wang, and Chen Chen. Poseformerv2: Exploring frequency domain for efficient and robust 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2023. 3, 4
[70] Weixi Zhao, Weiqiang Wang, and Yunjie Tian. Graformer: Graph-oriented transformer for 3d pose estimation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20406-20415, 2022. 3, 7
[71] Ce Zheng, Xianpeng Liu, Guo-Jun Qi, and Chen Chen. Potter: Pooling attention transformer for efficient human mesh recovery. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 6
[72] Ce Zheng, Matias Mendieta, Taojiannan Yang, Guo-Jun Qi, and Chen Chen. Feater: An efficient network for human reconstruction via feature map-based transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13945–13954, 2023. 3
[73] Ce Zheng, Wenhan Wu, Chen Chen, Taojiannan Yang, Sijie Zhu, Ju Shen, Nasser Kehtarnavaz, and Mubarak Shah. Deep learning-based human pose estimation: A survey. ACM Computing Surveys, 56(1):1-37, 2023. 3
[74] Ce Zheng, Sijie Zhu, Matias Mendieta, Taojiannan Yang, Chen Chen, and Zhengming Ding. 3d human pose estimation with spatial and temporal transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11656-11665, October 2021. 1, 2, 3, 4, 6, 7, 8, 9, 10, 16, 17
[75] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020. 2, 3, 5, 7, 18
A Additional Discussion on Related Work
Involving 2D joint heatmaps in image-based 3D human pose estimation has also been explored in some works [62, 18]. Yin et al. [62] split the 3D human pose estimation into two sub-tasks: 2D pose estimation and depth regression. Habibie et al. [18] disentangle features for the 2D pose from global image features to exploit large-scale 2D pose datasets. These approaches do not explicitly learn features from 2D poses and lift them to 3D.
Keypoint-based feature extraction. IVT [46] formulates video-based multi-person 3D pose estimation as an end-to-end framework. The features of each instance in the video are gathered by learning joint offsets from the body center. Compared to IVT, our work follows the standard two-stage pipeline of 3D HPE, and the estimated 2D joints serve as reference points in feature extraction. Pixel-aligned features [49, 66] in the context of 3D human reconstruction use point-wise image features similar to our approach. Their feature extraction is based on the re-projection of estimated 3D meshes [66] or query 3D points [49] instead of 2D human joints from off-the-shelf 2D pose detectors. Pixel-aligned features primarily aim at reducing the misalignment between estimated 3D representations and input 2D images. In contrast, we propose joint-context feature extraction to reduce ambiguities in lifting-based HPE and alleviate current methods' reliance on temporal information.
2D pose detector settings. Our method's overall pipeline includes an off-the-self 2D pose detector and a lifting model. For the first stage, the 2D pose detector is pre-trained on the COCO [31] dataset, without finetuning on the 2D poses from 3D pose estimation datasets, i.e., Human3.6M [21] and MPI-INF-3DHP [39]. We use  $256 \times 192$  resolution for input images, and we found higher image resolution improves performance. For 2D-to-3D lifting, the weights of pre-trained 2D detectors are frozen, i.e., no finetuning on the 3D task is needed. This approach makes our method preferably flexible – our method is compatible with a wide range of off-the-shelf (pre-trained) 2D pose detectors. We show in Sec. 4 that our method gains consistent improvements by increasing the capability of 2D pose detectors. In the future, we may leverage more advanced 2D pose detectors to further improve the performance upper bound.
Lifting model settings. Our lifting model includes three basic modules: Deformable Context Extraction, Pose-Context Feature Fusion and Spatial Inter-joint Modeling. The layer number of each module is set to 4, following PoseFormer [74]. The hidden dimension (a shared projection dimension  $C$ ) of the model is 128. We use 8 heads in self-attention for transformer layers.
Training details. The experiments are conducted on a single NVIDIA RTX 3090 GPU. Our lifting model is trained using the AdamW [34] optimizer for 50 epochs with a batch size of 512. The initial learning rate is  $6.4\mathrm{e} - 3$  with an exponential learning rate decay schedule, and the decay factor is 0.99.
C A Simple Temporal Extension of Our Method
This section shows that our single-frame method can naturally extend to model temporal dependencies (an overview is in Fig. 8). As illustrated in Sec. 3.2, the Spatial Inter-joint Modeling module outputs a feature vector of dimension  $(L + 1)\times C$  for each joint, where  $L$  is the number of multi-scale feature maps and  $C$  is a shared projection dimension of the model. We use a Temporal Transformer to model temporal correlations of each joint independently. Specifically, for Temporal Transformer, the input token number is the total frame number, and each token is of dimension  $(L + 1)\times C$ . Using transformers to build up temporal dependencies is straightforward, and this approach has been adopted by PoseFormer [74], MixSTE [67], etc. The output of the temporal transformer encoder can be denoted by  $\mathbf{Z}_{temp}\in \mathbb{R}^{J\times F\times [(L + 1)\cdot C]}$ , where  $J$  is the joint number and  $F$  is the frame number. Following PoseFormer, we use 1D convolution to reduce its temporal dimension (gather temporal information) and a linear layer to obtain the final estimated 3D pose  $\boldsymbol {y}\in \mathbb{R}^{J\times 3}$ .
Quantitative results. Due to limited computational resources, we prepare contextual feature vectors (that are sampled at detected joints from multi-level feature maps) in advance and remove the
Figure 8: Architecture of our method's simple temporal extension (3 frames as input).
Table 6: We compare the (short-term) temporal extension of our small model (CA-PF-S) with PoseFormer. MPJPE: Mean Per Joint Position Error, the precision metric. MPJVE: Velocity Error, the temporal smoothness metric. The results are reported on Human3.6M (in millimeters).
Model	Frame	MPJPE ↓	MPJVE ↓
PoseFormer	1	53.2	13.7
3	51.0	7.1
9	49.9	4.8
81	44.3	3.1
CA-PF-S	1	44.7	8.5
3	44.2↓0.5	4.8↓3.7
9	43.4↓1.3	3.4↓5.1
27	40.2↓4.5	2.1↓6.4
Deformable Context Extraction module. Therefore, the input to our model is discrete feature vectors instead of image sequences. This approach largely reduces the memory cost. We also reduce the shared projection dimension  $C$  in our model to speed up training. This small variant of our model is referred to as "CA-PF-S", which has fewer FLOPs compared to our full model "CA-PF" presented in Sec. 4. To show the gains in precision and temporal smoothness from temporal modeling, we report two metrics, MPJPE (Position Error, the precision metric) and MPJVE (Velocity Error, the temporal smoothness metric), on Human3.6M [21]. We compare it with PoseFormer [74] and the results are in Table 6: First, increasing the number of input frames brings consistent improvements in both precision and temporal smoothness. For example, by using only 3 video frames, the MPJVE of our method decreases from 8.5 to  $4.8\mathrm{mm}$  (a  $43.5\%$  reduction), and the MPJPE is reduced by  $1.1\%$ . This indicates that even short-term temporal modeling largely mitigates jitters in estimated results and further improves precision upper bound. Second, considering the same number of input frames, our CA-PF-S consistently outperforms PoseFormer in terms of both MPJPE and MPJVE. Moreover, our 3-frame CA-PF-S has already achieved superior MPJPE to 81-frame PoseFormer, and our 9-frame CA-PF-S achieves comparable MPJVE with 81-frame PoseFormer. The results verify the two benefits of spatial contextual clues from 2D pose detectors - accuracy and temporal stability.
Based on the results above, we expect that our method can be successfully extended to model even longer-term temporal dependencies (e.g., 81 video frames) to further boost precision and temporal smoothness. However, modeling such long input sequences with acceptable computational costs is non-trivial and is out of the scope of this paper, which is left as our future direction.
Table 7: Ablation study on the ViT backbone and pre-training datasets.
Index	Backbone	Pre-training Datasets	Multi-scale Design	mAP on COCO ↑	MPJPE on Human3.6M ↓
1	CPN	COCO	✓	68.6	41.6
2	HRNet-32	COCO	✓	74.4	41.4
3	ViT	COCO	✗	75.8	44.5
4	ViT	COCO+AIC+MPII	✗	77.1	41.9