A Single 2D Pose with Context is Worth Hundreds for 3D Human Pose Estimation
Authors: Qitao Zhao
Ce Zheng
Mengyuan Liu
Chen Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: Human Pose Estimation; 2D-to-3D Lifting; Context-Aware

Abstract: The dominant paradigm in 3D human pose estimation that lifts a 2D pose sequence to 3D heavily relies on long-term temporal clues (i.e., using a daunting number of video frames) for improved accuracy, which incurs performance saturation, intractable computation and the non-causal problem. This can be attributed to their inherent inability to perceive spatial context as plain 2D joint coordinates carry no visual cues. To address this issue, we propose a straightforward yet powerful solution: leveraging the $\textit{readily available}$ intermediate visual representations produced by off-the-shelf (pre-trained) 2D pose detectors -- no finetuning on the 3D task is even needed. The key observation is that, while the pose detector learns to localize 2D joints, such representations (e.g., feature maps) implicitly encode the joint-centric spatial context thanks to the regional operations in backbone networks. We design a simple baseline named $\textbf{Context-Aware PoseFormer}$ to showcase its effectiveness. $\textit{Without access to any temporal information}$, the proposed method significantly outperforms its context-agnostic counterpart, PoseFormer, and other state-of-the-art methods using up to $\textit{hundreds of}$ video frames regarding both speed and precision. $\textit{Project page:}$ https://qitaozhao.github.io/ContextAware-PoseFormer

A Single 2D Pose with Context is Worth Hundreds for 3D Human Pose Estimation
Qitao Zhao Ce Zheng²
Mengyuan Liu?
Chen Chen²
'Robotics Institute Carnegie Mellon University
2 Center for Research in Computer Vision, University of Central Florida Key Laboratory of Machine Perception, Peking University, Shenzhen Graduate School
Abstract
The dominant paradigm in 3D human pose estimation that lifts 2D pose sequence to 3D heavily relies on long-term temporal clues (i.e., using a daunting number of video frames) for improved accuracy, which incurs performance saturation, intractable computation and the non-causal problem. This can be attributed to their inherent inability to perceive spatial context as plain 2D joint coordinates carry no visual cues. To address this issue, we propose straightforward yet powerful solution: leveraging the readily available intermediate visual representations pro- duced by off-the-shelf (pre-trained) 2D pose detectors task is even needed. The key observation is that, while the pose detector learns to localize 2D joints, such representations (e.g., feature maps) implicitly encode the joint-centric spatial context thanks to the regional operations in backbone networks. We design simple baseline named Context-Aware PoseFormer to showcase its effectiveness. Without access to any temporal information, the proposed method sig- nificantly outperforms its context-agnostic counterpart, PoseFormer [74]. and other sfate-of-the-arr methods using up to hundreds of video frames regarding both speed and precision. Project page: sitaozhaa.github... www.ghhhhooooootttttaaaaaaa ContextAAarrre Paeee-Poeeooraee Former
finetuning the 3D
Introduction
3D human pose estimation (HPE) aims to localize human joints in 3D, which has wide range of applications, including motion prediction [37], action recognition [9], and tracking [47]. Recently, with the large availability of 2D human pose detectors [8, 52]. lifting 2D pose sequences to 3D (referred to as lifting-based methods) has been the de facto paradigm in the literature. Compared to raw RGB images, 2D human poses (as intermediate representation) have two essential advantages. On the one hand, domain gaps exist between images (input) and 3D joint locations (output), whereas this is not the case for 2D and 3D joints. Primarily, 2D joint coordinates provide highly task-relevant positional information for localizing joints 3D. On the other, 2D coordinate representation is exceptionally lightweight in terms of memory cost. For example, in typical COCO [31] setting, 2D human pose requires only 17 x 2 floating points, whereas an RGB image requires 256 x 192 x (or even more with higher resolution). This property enables state-of-the-ann art lifting- based methods to leverage extremely long-temn tergteemm temporal clues for advanced accuracy, e.g., 243 video frames for VideoPose3D [45], MixSTE [67], and P-STMO [50]; large as 351 frames for MHFormer [29].
Scaling up the input has brought consistent improvements so far, yet some concerns naturally arise. First, the performance gains seem level off when excessively long sequences are used input (e.g., improvements are marginal when the frame number increases from 243 to 351 [29]). Second, long input sequences bring non-trivial computational costs. For instance, temporal processing using transformers (a popular choice) is expensive, especially with large frame numbers. Third, the use of
Work was done while Qitao was an intern mentored by Chen Chen.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Stage 1
Stage 2
Pose Detector
2D Pose
Context-Agnostic Lifting Model
(a)
Video
3D Pose (for the central frame)
Sequence (the only input)
(b)
Image
Pose Detector
2D Pose
Context-Aware Lifting Model
3D Pose
Readily Available
Visual Representations
pre-trained and with weights frozen. For our method, no finetuning needed for 2D detectons (i..,, readily available).
Intermediate visual representations are discarded, while 2D joint coordinates are the only input (to Stage 2).
reuse representations (chat sparial
cues) Stuge resolve ambiguiries.
Figure 1: A comparison of existing lifting-based methods with methods (vidco-based): time-intensive and context- context-agnostic. (b) Ours (image-based);; time-free and context-aware. We leverage intermediate visual representations from Stage 1. Notably, do not finetune the 2D detectors for the lifting task, thus easing training and bringing no extra costs.
framework level. (a) Existing
future video frames renders real-world online applications impossible. Concretely, existing works adopt a 2D pose sequence to estimate the 3D human pose for the central frame where half of the frames in the sequence are inaccessible for the current time step, which is known the non-causal problem [45]. While gains from long input sequences come at a cost, is there any alternative approach take easily boost performance? We attempt answer this question in work.
We start by revisiting the whole pipeline of lifting- bbsed methods, which involves two stages as 2D pose detector estimates human joints for each image from
shown in Fig. (a). In Stage video clip, with a set of intermediate representations as byproducts, e.g., feature maps of varying resolutions. In Stage 2, the detected 2D pose sequence (output of Stage 1) is lifted to 3D, while such representations are discarded. This can be problematic: the (multi-scale) joint-centric spatial context encoded by these feature maps is lost. We argue: the omission of spatial contextual information is the primary factor contributing to the time-intensive nature of existing lifting- tffting-bssed methods.
So, what makes spatial context important? As an inverse problem, 3D pose estimation inherently suffers from ambiguities [29, 36, 57] such as depth ambiguity and self-occlusion, yet they can be mitigated by utilizing spatial context from the monocular image. For depth ambiguity, the shading difference is an indicator depth disparity [57] whereas the occluded body parts can be inferred from visible ones with human skeleton constraints [36, 57J. Psychological studies [3, 36] provide evidence that appropriate context helps reduce ambiguities and promote visual recognition for the human perception system. Since 2D keypoints alone unable to encode such spatial context, existing lifting-baser approaches, in turn, resort to long-term temporal clues to alleviate ambiguitics.
We offer surprisingly straightforward solution to "fix" the established lifting framework as depicted in Fig. (b): retrieving the lost intermediate visual representations learned by 2D pose detectors and engaging them in the lifting process. As previous works limit their research scope to the design of lifting models (Stage 2), the cooperation of both stages largely under-explored. This approach yields two major benefits: First, such representations encode joint-centric spatial context that promotes reducing ambiguity- the core difficulty in 3D human pose estimation. Second, we illustrate this work that these representations can be used out of the box, i.e., no finetuning on the 3D task or extra training techniques are needed.
To showcase the effectiveness of our solution, we design a simple transformer-basee baseline named Context-Aware PoseFormer. The proposed method leverages multi-resolution feature maps pro- duced by 2D detectors in sparse manner. Specifically, we extract informative contextual features from these feature maps using deformable operations [12, 75] where the detected 2D joints serve as reference points. This helps mitigate the noise brought by imperfect pose detectors while avoiding heavy computation. Furthermore, we design pose-contexi interaction module to exchange informa- tion between the extracted multi-level contextual features and the 2D joint embedding that encodes positional clues. Finally, a spatial transformer module is adopted to model spatial dependencies between human joints, which follows PoseFormer [74]. Our approach shows encouragingly strong

---

performance (see Fig. 2): our single-frame frame model outperforms 351-frame MHFormer [29] and other of-te-of-the-art lifting- based methods, highlighting the potential of leveraging contextual information in improving pose estimation accuracy.
Our contributions are summarized follows: 1) We propose a novel framework that addresses the time-intensive issue present in existing meth- ods by incorporating context awareness. This is achieved by leveraging readily available visual representations learned by 2D pose detectors. 2) We introduce a simple baseline that effi- ciently extracts informative context features from learned representations and subsequently fuses them with 2D) joint embedding that pro- vides positional clues.
frame
GraphMLP/28] arxiv/22
framm
MHioorme((2)) CVPR/22
P. P-STMOySO) ECCV22
27 finmes
Zeing et al.165] ICCV22
MnjrEmmmn
46-
13.2%
frames
reduction
44
81 frames
243 fraaes
351 frames
3) On two benchmarks (Human3.61) 66 [21] and MPI-INE-DDHP [39J), our single-frame model demonstrates significant performance improve- ments over both non-temporal and temporal methods that use up to hundreds of video frames. 4) Our work opens up opportunities for more skeleton- daeseton-based methods to utilize visual repre- sentations learned by well-trained backbones from upstream tasks in an out-of-the-box manner.
42
frame
243 fraenes
Ours
100
150
200
250
350
Number of Frames
Figure 2: Our single-frame method outperforms both non-temporal and temporal methods that use up to 351 frames on Human3.6M.
2 Related Work
Early works [54, 17, 44, 53] estimate the 3D human pose from monocular images without explicitly using the corresponding 2D pose as an intermediate representation. With the rapid progress in 2D pose estimation [42, 8, 52], lifting 2D poses to 3D has been dominant in the literature. As our work aims at improving the lifting-basee framework, we primarily introduce works of this line. We refer readers the recent HPE survey [73] for more comprehensive background and detailed information.
Single-frame methods. Before the surge of deep learning, Jiang [22] performs 3D pose estimation based on nearest neighbors with a large 3D human pose library. and this approach is recently revisited by Chen er al. [5] with a closed-forn form algorithm to project 3D pose exemplars to 2D. Martinez er al. [38] design a simple linear-layer model to illustrate that the difficulty of 3D human pose estimation mainly lies in precise 2D joint localization. Moreno-Noguer [41] formulates the task as 2D-to-3D distance matrix regression. To mitigate the demand for 3D annotations, Pavlakos el al. [43] leverage ordinal depths of human joints as an extra training signal. Given the human skeleton topology, recent works (e.g., GraphSH [60], HCSF [65], GraFormer [70]. GraphMLP [28]) use graph neural networks to reason spatial constraints of human joints, and lift them to 3D.
Multi-frame methods. As initial attempts, some works explore temporal cues for robust pose estimation in crowded scenes [1] or temporally consistent results [40]. More recently, a number of works model
dependencies for 2D pose sequences, e.g., using LSTM [24, 59], CNN [45, 6] and GNN [56, 65]. They have achieved superior performance than non- non-temporal methods. Driven by the great success of vision transformers in image classification [13, 33], object detection 75] and etc, transformer-ber based methods [74, 29, 67, 50, 69, 72] are currently the center of research interest. PoseFormer [74] is the first transformer-basee
model in the community, which builds up inter-joint correlations within each video frame and human dynamics across frames with transformer encoders, respectively. Due to its straightforward architecture, PoseFormer is highly malleable and rapidly gains a series of follow-ups [67, 69]. Our work presents a simple single-frame baseline model where the inter-joint modeling module is adapted from the spatial encoder of PoseFormer.
Fusing image features with positional joint clues. Compared to previous lifting lifting-based works, our approach actively engages visual representations from 2D pose detectors in the lifting process, thus being context-aware and achieving promising results. Notably, the fusion of image features with positional information about human joints has been explored in some works but within differing contexts or with distinct motivations. Tekin el al. [55] propose to fuse image features with 2D joint heatmaps, where 2D joints are not explicitly regressed as in the lifting-barc based pipeline. Closer to
3

---

Input Image!
Stage
Stage 2
Target Pose
Reuse
Intermediate
Feature Maps
x N,
Context Features
Regression Head
H1
F
Spatial Inter-joint
N/
Modeling
Backbone (egg, HRNet)
H2
F2
Deformable Extraction
Pose-Context Feature Fusion
X
H3
F3
As Ref. Points
& Coordinate Embedding
P
F2
Fg
Output 2D Pose
P80
Add Positional Emieeddings
An overview of Context-Aware PoseFormer. Stage 1 (left): The 2D pose detector estimates the 2D pose, with set of feature maps as byproducts. In Stage 2 (right), we extract informative join-context features from such feature maps and subsequently fuse them with 2D pose features.
Figure 3:
our work, Zhao el al. [68] extend their graph network (SemGCN) by incorporating image features. Specifically, image features are pooled at the detected joints and then concatenated together with joint coordinates, serving as the input to SemGCN. Though this approach is similar to ours at first glance, our work differs in that our method needs no multi-stage training, 2) we consider the uncertainty associated with 2D joints in joint-context extraction, 3) we have designed dedicated modules to handle the domain gap between image features and joint coordinates, and 4) our performance surpasses that of SemGCN by significant margin. In addition to technical details and performance comparisons, our underlying motivation for this research also differs. More discussion about relevant literature can be found in Appendix A.
Method
We propose a new framework solve the time-intensive problem of previous lifting-based methods. This is achieved by incorporating the spatial context (encoded by intermediate feature maps from 2D pose detectors) into the 2D-to-3D lifting process (see also Fig. (b)). We design a simple baseline model named Context-Aware PoseFormer to exemplify the gains brought by this approach (an overview is in Fig. 3). In the following, we first briefly introduce the context-agnostic counterpart of method, PoseFormer [74]. Then, provide detailed descriptions of our approach.
3.1 Context-agnostic Counterpart: PoseFormer
PoseFormer [74] is of the first transformer-based models for 3D human pose estimation, which mainly consists of two modules: the spatial encoder and the temporal encoder. PoseFormer processes an input 2D pose sequence follows. First, the sequence Rxxx2 is projected to latent space indicated by E R FxXxC where F, and C denote the sequence length, joint number, and embedding dimension respectively. Learnable (zero-initialized) spatial positional embedding Expo R IxXxC is summed with S to encode joint-specific information. Second, the spatial encoder models inter-joint dependencies at each video frame (time step) with cascaded transformers. Third, the output of the spatial encoder Z E R Fx(JxC) is flattened as the input the temporal encoder. Similarly, temporal positional embedding Exemp E Fx(JXC) is added Z encode the information related frame indices. The temporal encoder builds up frame-level correlations with transformer layers in this stage. Finally, ID convolution layer is adopted to gather temporal information, and a linear layer outputs the target 3D pose R [x((33) for the central video frame. PoseFormer and its follow-ups [67, 50, 69] are inherently limited deal with ambiguities since they barely receive 2D joint coordinates input. Due to the simplicity, scalability, and effectiveness of PoseFormer, we choose PoseFormer as the

---

basis of our method, boosting it to be context-aware and free of temporal information. Detailed descriptions of our method are presented below.
3.2 Context-Aware PoseFormer
Our core insight is retrieving the spatial context encoded by readily available feature maps from 2D human pose detectors. As the multi-stage design promotes 2D human pose estimation [52, 11, 63], our method also benefits from multi-resolution feature maps. An illustration of model input is on the left of Fig. 3. For given RGB image of size H x W 3, an off-the-shelf 2D pose detector produces the corresponding 2D human pose E R/x22 and a set of intermediate feature maps |=s (where L the total number of feature
with varying resolutions as byproducts. {H, E R. maps). Since stacked network layers and down- down-sampling operations increase the receptive field (and correspondingly the scale of context) of each unit on feature maps [35], high-resolution feature maps encode fine-grained visual cues (e.g., joint existence) while low-resolution ones tend to keep high-level semantics (e.g. human skeleton structures). During the process that 2D pose detectors are trained to localize human joints, multi-level feature maps implicitly encode information about the spatial configurations of joints.
How to effectively utilize these feature maps is a non-trivial question. Blindly processing them in global way using, e.g., convolutions [20], vision transformers [13], may bring unnecessary computational overhead since the background that takes up a large proportion of the image does not provide task-relevant features related to human joints. The most straightforward approach to treat feature maps in a preferably sparse manner is to sample feature vectors at the detected joints. However, in real scenarios, 2D pose detectors inevitably introduce errors. Therefore, the context feature obtained by such an approach may not well represent the spatial context associated with the joint of our interest, leading to suboptimal performance. We propose a Deformable Context Extraction module to deal with this issue, which not only attends to the detected joints but also the regions around them. As the detected 2D joints are not the only source of contextual features, the errors brought by 2D pose detectors can be mitigated. Moreover, we design Feature Fusion module to fuse the features of two modalities: spatial contextual features and 2D joint embedding that encodes positional clues about human joints. Finally, Spatial Inter-joint Modeling is adopted to build up dependencies across human parts. They are organized hicrarchically (on the right, Fig, 3). The effectiveness of each element is verified in Sec. 4.3.
Deformable Context Extraction. This module uses deformable attention [75] to extract informative spatial contextual cues from feature maps. Specifically, for each detected joint, we produce a set of sampling points on multi-scale feature maps whose offsets and weights are learned based on the features of reference points (i.e., the detected joint of interest), denoted by [F We make learned offsets and weights "reference- weteee-oooiiitwwaaaaaeee source features (after they are projected to a shared dimension C). Let / index a feature level and j index a human joint, and Deformable Context Extraction formulated as:
by adding the embedding of 2D joint coordinates P to
F DeformAttn(F)) 1j + +P FTj
F P, PEER n 1==1..... I...N Fi F. ERE RC, n I...Ni
F = MLP(F()) + FFJ
(I)
DeformAttn(F)) (FT, + P,) =
Aimk Wi.,H + Apma)),
Hi(p, + A E R
m==
where m iterates over the attention heads, over the sampled points around the detected joint py. and K is the total sampling point number. For the 1hh feature map, Apink represents the sampling offset of the 4h sampling point in the mh attention head, while Ajmz denotes its corresponding attention weight. We provide visualization of sampling points learned by deformable attention in Sec. 4.3, where the learned sampling points attempt to discover ground truth 2D joints despite the noise brought by pose detectors. More details and visualization about this module are available in Appendix E.
Pose-Context Feature Fusion. This module aims at exchanging information between two modalities (i.e., pose embedding and the extracted spatial context) and between different levels of context features at the same time. Cross-modality learning gives rise to powerful languaee models [23, 2]. while the multi-scale design has proved beneficial to large set of computer vision tasks
5

---

[15, 16, 71]. Inspired by multi-modalliy models [2] that feed visual tokens and language tokens into unified transformer encoder, we naturally adapt this design to model interactions between pose embedding and multi-level context features, where transformers learn a joint representation for both modalities [51, 7]. The implementation of this module is:
x) Stack([P)) F Fi...... FY,,, dim 0),
Pj PRE F Fij..... FF E ER x RUAID = 1..... N2
X" = SelfAumn(X) ) X7-
X = M) + XX",
x ERE n1 lu.....N;
where j J) indicates that feature fusion is performed for each joint. Transformer layers reduce domain gaps for both modalities in shared latent space [25] and promote message-passing where joint-position information and multi-level contextual cues complement each other. In addition, we introduce Unified Positional Embedding E [X(+++xXXXC (L for joint-context feature vectors and for 2D pose embedding) to encode modality- related and joint-specific information simultaneously.
Spatial Inter-joint Modeling, With the two modules above, elegant local representations learned for each joint individually. To understand the human skeleton system and its corresponding spatial context with global view, inter-joint dependencies are modeled based the learned per-joint features using a spatial transformer encoder following PoseFormer [74]. The spatial encoder in PoseFormer simply receives the linear projection of (context-agnostic) 2D joint coordinates as input, whereas our joint-level representations are enhanced with spatial contextual cues (thus being context- aware). The output of Pose-Context Feature Fusion module (X / is flattened and stacked as input to the Ny-layer transformer encoder. Each joint token (J in total) is of (L + 1) x C dimensions, encoding both positional and contextual information for the related joint.
J=1
Output and loss function. A simple linear layer is adopted to obtain the final 3D pose y E /x3,, We use L2 loss PoseFormer [74] to supervise the estimated result.
Discussion. Our model follows local-to- to-glbbal hierarchy. Specifically, local representations are first learned for each joint, on which inter-joint modeling is subsequently performed. This design provides our model with preferable interpretability and scalability: modules are inherently disentangled according to the dependencies they attempt to learn and can be easily extended or replaced. Plus, the local-to-global organization avoids the heavy computation brought by globally modeling all elements at once. These virtues make our method desirable stepping stone for future research.
4 Experiments
Our method, Context-Aware PoseFormer, is referred to "CA-PT-backbone name" in tables. We conduct experiments on two benchmarks (Human3.66 6M [21] and MPI-INE-3DHP [39]). Human3.6M [21] is the most popular benchmark for 3D human pose estimation, where more than 3.6 million video frames are captured indoors from different camera views. MPI-INF-3DHP [39] includes videos collected from indoor scenes and challenging outdoor environments. Both datasets provide subjects performing various actions with multiple cameras. On Human3 6., we report MPJPE (Mean Per Joint Position Error, where the Euclidean distance is computed between estimation and ground truth) and PA-MP.JPE (the aligned MPJPE). For MPI-INF-3DIS we report MPJPE, Percentage of Correct Keypoint (PCK) within the 150mm range, and Area Under Curve (AUC). Settings follow previous works [74, 29, 67, 50J. Due to page limitations, we place details about our implementations in Appendix B.
4.1 Comparison on Human3.6M
Accuracy. We compare our fingle-ramee method with both single-frame frame and multi- framt--raameeee methods on Human3.6M, [21] (Table 1). Previous methods receive CPN-detected [8] 2D joints as input. We also follow this setting to ensure a fair comparison. Benefiting from spatial contextual clues encoded by the CPN pose detector, our approach outperforms state-of-the-art single-frame methods by a huge margin. For instance, Zeng el al. [65] obtain 47. .mm MPJPE, whereas our CA-PF-CPN achieves 41 66mm (a 13.2% error reduction). This result has already surpassed the performance of most modern multi- frame methods. To show the flexibility of our method and to further investigate spatial context

---

Table Comparison with both single-framn (top) and mulltifraame (middle) methods on Human3.
MPJPE is reported in millimeters. The best results are in bold, and the second- best ones are underlined.
FLOPS (G) the
Method
Venue
2D Pose Detector
Frame
Lifting Module
MPJPE
PA-MPIPE
GraphSH [60]
CVPR'21
CPN
Single
HCSF al. [65]
ICCV'21
CPN [8]
47.9
39.0
Frame
GraFormer [70]
CVPR 22
CPN [S]
1
51.8
GraphMLP [28]
arXiv'22
CPN [8]
49.2
38.6
Pavllo
CVPR 19
CPN
243
0.03
46.8
36.5
Liu et al. [32]
CVPR 20
CPN [8]
243
45.1
35.6
Wang et al. [56]
ECCV
CPN [S]
96
45.6
35.5
Zeng [64]
BCCV'20
[8]
243
Multi-
PoseFormer [74]
ICCV'21
CPN [S]
81
1.4
44.3
34.6
frame
Strided [27]
TMM
CPN [S]
351
2.1
43.7
35.2
MHFormer [29]
CVPR'22
CPN [S]
351
43.0
34.4
MixSTE [67]
CVPR'22
CPN
243
277.4
40.9
32.6
P-STMO [50]
ECCV'22
CPN [S]
243
1.5
43.0
34.4
Linfalt et al. [14]
WACV'23
CPN [8]
18
1.0
45.0
36.3
CA-PF-CPN (ours)
CPN
41.6
33.9
CA-PF-HRNet-32 (ours)
HRNet-32 [52]
41.4
33.5
CA-PT-HRNet-46 (ours)
HRNet-48 [52]
0.6
39.8
32.7
from various backbones, we change the backbones that generate feature maps: our CA-PF-HRNet-332
[52] achieves 41 ..4m MPJPE; with larger backbone HRNet-488 the error reduces to 39.8mm, which is superior to all previous multi-frame methods. For example, this result outperforms MiXSTE [67] with 243 frames (39.8 v.s. 40.9mm, 2.7% error reduction) and MHFormer [29] with even 351 frames (39.8 v.s. 43.0mm, a 7.4% enror reduction). Our method presents highly competitive accuracy in the absence of dense temporal modeling. Moreover, our method consistently gains improvements from advanced 2D pose detectors. We expect our approach's s performance to increase further with powerful large visual foundation models, as such foundation models have been attracting increasing attention and interest.
Computational cost. Our method requires far less computation than state-of-the-ant methods (see also Table 1). For example, with similar accuracy, MHFormer [29] takes 14.2 GFLOPs, whereas our CA-PF-CPN [8] demands only 0.6 GFLOPs (about 24x reduction). Two reasons account for the lightweight property of the proposed approach: (I) Non-temporal input removes large computation for dense temporal modeling. Modeling temporal dependencies using e.g., transformers [74, 67] is computationally heavy especially when the frame number increased for improved accuracy. (2) We utilize multi-scale feature maps (from pose detectors) in sparse manner (i.e., with deformable attention [75]). We avoid applying global operations on the whole feature maps, which may introduce unnecessary computational budgets uninformative backgrounds.
4.2 Comparison on MPI-INF-3DHP
In Table 2, the comparison is also conducted with both single-frame (top) and multi-frame (middle) meth- ods on the MPI-INF-3DHP dataset
Table 2: Comparison on MPI-INF-DHHP, T indicates the
number of video frames used by models.
Method
PCK
AUC1
MPJPE
[39]. We ground truth 2D pose detection as input, following [74, 29, 50], and HRNet-32, HRNet-48 [52] are adopted as the backbone network to generate multi-resolution feature maps. Our approach outperforms other methods, including the state- of-the-art multi-frame one, P-STMO [50]. P-STMO uses 81 video frames as input, with additional masked joint self-supervised pre-training. In con- trast, our approach has no access to temporal information and does not need an extra pre-training stage. The results verify the generalization ability of our method to different datasets, particularly in challenging outdoor environments.
Xu al. [60] (T=l)
80.1
45.8
Li et al. [26] (T=1)
81.2
46.1
99.7
Zeng et al. (T=1)
82.1
46.2
PoseFormer [74] (7'=9)
95.4
63.2
57.7
MHFormer [29] (T=9)
93.8
63.3
58.0
MixSTE [67] (T=27)
94.4
66.5
54.9
P-STMO [50] (T=81)
97.9
75.8
32.2
Net33- (ours)
98.0
75.4
32.7
CA-PF-HRNet-48 (ours)
98.2
76.3
31.4

---

Table 3: Ablation study spatial context and each component of method. Experiments conducted on Human3.6M with HRNet-32 as the backbone. MPJPE is reported in millimeters.
Context-
Inter-joint
Pose-Context
Deform. Cont.
Step
FLOPs (M)
MPJPE
aware
Modeling
Feature Fusion
Extraction
(0)
X
X
X
446.0
51.2
X
X
448.0
43.5 (7.7L)
(2)
X
537.3
42.8 (8.41)
(3)
609.9
(9.8()
4.3
Ablation Study
Gains from spatial context and ablations model components. In this section, we first investigate the gains purely brought by spatial contextual clues from 2D pose detectors and then the effectiveness of each component in our method (please refer Sec. 3.2 for more details). Specifically, we show how the context-agnostic counterpart of our method, (single-frame) PoseFormer [74] is converted to Context-Aware PoseFormer step by step (step-wise improvements are shown in Table 3).
Step 0, Context-Agnostic For PoseFrrmer, 2D joint coordinates detected by pose detector are linearly embedded to dimension C as per-joint representation (i.e. joint embedding), and Inter-joint Modeling is subsequently performed to build up correlations across the joint embeddings. Plain 2D coordinates encode no spatial context for joints that helps reduce ambiguity, PoseFormer (and other lifting-based methods) is thus referred to as "context-agnostic" This serves as the starting point of our method.
Step 1, Context-Aware but simple concatenation: The first step to make PoseFormer "context- aware" is to incorporate joint-context features into its per-joint representation. We retrieve the multi-scale feature maps produced by the 2D pose detector, simply sample feature vectors on the detected joint locations from each feature map, and project them to dimension C. For each joint, the sampled context feature vectors that encode joint-centric visual cues are concatenated together with the coordinate embedding that encodes positional information. Even naive involvement (as simple concatenation) of spatial context clues brings huge improvement as shown in Table 3: a 15.0% error reduction (from 51.2 2 to 43.5mm), which demonstrates that leveraging the readily available visual representations learned by upstream backbones is effective and promising.
Step 2, Context-Aware with Pose-Context Feature Fusion: We promote cross-modality feature fusion by applying transformers joint embeddings and the sampled multi-scale context features before concatenation, which brings another 1.6% error reduction.
Step 3, Final version of our Context-Aware PoseFormer: The context features in step obtained by directly sampling at the detected joints, which leads to suboptimal performance since 2D pose detectors inevitably introduce noise. Thus, the sampled feature vectors can not well represent the context associated with the joint of interest. We introduce Deformable Context Extraction mitigate this issue, which adaptively attends the detected joint along with its surrounding regions using learnable sampling points, which further reduces the error by 3.3%.
We also offer visualization of learned sampling points (brighter colors denote larger attention weights) in Fig. 4, where they try to approach the ground truth despite the noisy detected input. Note that in training we do not supervise them using ground truth 2D pose. More visualization is in Appendix E.
What kind of spatial context are we looking for? In this part, we go step further to explore what context desirable to improve 3D pose estimation accuracy. Our method benefits from multi-resolution feature maps produced by 2D detectors [52, 11, 63], which encode spatial contextual clues of various levels for human joints. For example, HRNet [52] outputs four feature maps in the last stage, which are 4x, 8x, 16x, 32x downsampled compared the input image size, respectively. We investigate which one of the four is most informative for our task via ablation experiments (in Table 4, MPJPE reported on Human3.0 6M). Removing 4x and 16x downsampled feature
Blue: detected joints, Creens ground Gray: Jeanned sampling points.
Figure 4: In the proposed De- formable Context Extraction mod- ule, sampling points attempt to ap- proach the ground truth despite reliable 2D joint detection.
8

---

maps brings the most significant performance drop (a 3.6% and 4.1% error increase for each). In contrast, only a 1.0% error increase observed when the 32x downsampled feature map is eliminated.
Generally, high-resolution feature maps contribute more to 31D pose estimation than low-resolution ones. The reason is that low-resolution feature maps tend to keep the high-level semantics of joints, such join type, which may not be relevant to our task - to localize human joints in 3D, where 2D-to-3D joint correspon- dence is pre-defined. However, we can still gain from low-resolution feature maps as they encode most wide- range spatial dependencies, promoting an understanding of the overall human skeleton structure.
Table Ablation study on feature resolutions.
4x
16x
32x
MPJPE J
/
41.4
x
/
/
/
42.9
/
x
/
42.4
/
x
/
43.1
X
41.8
What pre-trained features more desirable?
Table 5: Ablation study on pre-training tasks.
ImageNet [48] pre-trained backbones (e-g., ResNet [20]) profit series of downstream tasks [30, 10, 58]. yet this seems not applicable to 3D human pose estimation. In Table 5, we replace COCO pre- trained backbones in our method with ImageNet classification pre-trained ones, showing remark- able performance drop. This should be attributed the large gap between the pre-traue pre-training task (im- age classification) and the downstream task (3I) human pose estimation). Exploring more suitable pre-trained features that can transfer their effectiveness into the 3D HPE task is a promising direction. We place more ablations on backbones and pre-training datasets in Appendix D.
Backbone
Pre-training
MPJPE 1
2D Pose
45.0
ResNet-50
Image Class.
51.446.
2D Pose
41.4
HRNet-32
Image Class.
45.8
2D Pose
39.8
HRNet-48
Image Class.
43.9
4.4
Visualization
MHFormer(T-351 CVPR/22
140
Context-agnostic Counterpart(T-1)
Context-Aware PoseFomen(T-1)
120
H010
1.0
Figure 5: Frame-wise comparison with MHFormer [29] and our context-agnostic counterpart
(Sec. 4.3) on Human3.6M test set (S9 Discussion). T indicates the frame number used by models.
Spatial context promotes temporal stability. We show frame-wise comparison on the Human3.6 66M [21] test set in Fig. 5. Our single-frame approach outperforms competitive MHFormer [29] using 351 frames. Moreover, our context-agnostic counterpart (Sec. 4.3) suffers from temporal instability (i.e., zigzags and error peaks), whereas method presents more stable trend. We also provide a qualitative comparison with both methods in Fig. 6. We include more visual comparison on both Human3.6M, and MPI-INF-3DHP in Appendix F.
Spatial context improves the robustness against occlusions. We provide a comparison with PoseFormer [74] on in-the-wild videos in Fig. 7. We choose two sets of consecutive video frames where the 2D joint detection fails due to strong self-occlusion. Since PoseFormer only accepts 2D joints as input, its estimation is sensitive to the noise of input 2D poses. On the contrary, our method leverages spatial context from images addition to 2D joint coordinates to localize joints in 3D. Thus, our method shows more reasonable and robust results despite noisy even outlier 2D joints.

---

Imput
MifFormer
Content-agnociic
Ours
Ground Truth
Impue
MitFormer
Contest-ugneeic
Ours
Ground Trath
Figure 6: Visual comparison in hard cases, e.g. self- cuff-oclluiio and noisy 2D detection. We include more qualitative results Appendix F.
This a hat, not an arm!
The right arm is missed by the pose detector.
2212
Input
PoseFormer
Ours
Input
PoseFormer
Ours
Figure 7: Comparison with PoseFormer [74] on in-the-wild videos. The 2D pose detector fails to localize 2D joints, given confusing clothing (left) and severe self-occlusion (right). Our method is more robust in such hard cases. False joint detection is indicated by yellow arrows, and the corresponding 3D joint estimation indicated by orange arrows.
5 Conclusion and Discussion
This work presents new framework that leverages readily available visual representations learned by off-the-shelf 2D pose detectors. Our approach removes the need for extensive long-term temporal modeling excessively employed by existing 3D human pose estimation methods. We further propose a straightforward baseline showease its effectiveness: our Longle-frame model significantly outper- forms other single-frame and even multi-frame methods using up to 351 frames while being far more efficient. Our work opens up opportunities for future research to utilize visual representations from well-trained upstream backbones in out-of-the-hee manner.
Limitations. We observed in Sec. 4.4 that incorporating the spatial context of joints improves temporal stability, enhancing consistency and smoothness in the estimated results, even without access explicit temporal clues. However, we acknowledge for all single-frame methods, including ours, mitigating jitters remains a challenge compared to multi-frama methods that leverage temporal clues. This is primarily due the non-temporal nature of single-frame frame methods. potential solution is to extend our method to model short-term temporal dependencies. which should not introduce unacceptably high costs. We include some preliminary results in Appendix C. Compared to standard lifting-based methods, the memory cost of our method can be higher we additionally process image features in the lifting stage. Leveraging the visual context of joints in a more lightweight way our future research direction.
10

---

References
[1] Mykhaylo Andriluka, Stefan Roth, and Bernt Schiele. Monocular 3d pose estimation and tracking by detection. In 2010 IEEE Computer Society Conference on Computer Vision and Pattem Recognition, pages 623 630. Icee, 2010.
[2] Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with Advances in Neural Information Processing Systems, 35:32897-
mixtuue-of-moodaltt-eexps
32912, 2022. 5,
[3] Irving Biederman, Robert Mezzanotte, and Jan Rabinowitz. Scene perception: Detecting and judging objects undergoing relational violations. Cognitive psychology, 14(2):143-177,
1982.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pages 213-229. Springer, 2020.
[5] Ching- Chen and Deva Ramanan. 3d human pose estimation 2d pose estimation + matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7035-7043, 2017. 3
[6] Tianlang Chen, Chen Fang, Xiaohui Shen, Yiheng Zhu, Zhili Chen, and Jiebo Luo. Anatomy- aware 3d human pose estimation with bone-based pose decomposition. IEEE Transactions on Circuits and Systems for Video Technology, 32(1):198 209, 3021.
[7] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, pages 104-120. Springer, 2020. 6
[8] Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 7103-7112, 2018. 1, 6,
[9] Yuxin Chen, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, and Weiming Hu. Channel-wise topology refinement graph convolution for skeleton-basee action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13359-13368 2021. 1
[10] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification not all you need for semantic segmentation. Advances in Neural Information P Processing Systems, 34:17864
17875, 2021. 9
[11] Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S Huang, and Lei Zhang. Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 5386-5390 5395, 2020. 5,8
[12] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages
2017. 2
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021. 5. 18
[14] Moritz Einfalt, Katja Ludwig, and Rainer Lienhart. Uplift and upsample: Efficient 3d human pose estimation with uplifting transformers. In Proceedings of the JEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2023.
[15] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE|CVF International Conference Computer Vision, pages 6824-6835, 2021.
[16] Golnaz Ghiasi, Tsung- Yi Lin, and Quoc V Le. Nas-fpn: Learning scalable feature pyramid architecture for object detection. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 7036-7045, 2019.

---

[17] Agne Grinciunaite, Amogh Gudi, Emrah Tasli, and Marten Den Uyl. Human pose estimation in space and time using 3d enn. In Computer Asion-ECCY 2016 Workshops: Amsterdam, The Netherlands, October 8- 10 and 15-16, 2016, Proceedings, Part III, pages 32-39. Springer, 2016.
3
[18] Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Gerard Pons-Moll, and Christian Theobalt. In the wild human pose estimation using explicit 2d features and intermediate 3d representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10905-10914, 2019. 16
[19] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pages 16000-16009, 2022. 18
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. 5,
the
[21] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3.6m 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE TPAMI, 2014. 3, 6, 9, 16, 17
[22] Hao Jiang. 3d human pose reconstruction using millions of exemplars. In 2010 20th Interna- tional Conference on Pattern Recognition, pages 1674-1677. IEEE, 2010. 3
[23] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning, pages
5583 5594.594. PMLR, 2 5
[24] Kyoungoh Lee, Inwoong Lee, and Sanghoon Lee. Propagating Istm: 3d pose estimation based on joint interdependency. In Proceedings of the European conference on computer vision (ECCV), pages 119-135, 2018. 3
[25] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai- Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557. 2019. 6
[26] Shichao Li, Lei Ke, Kevin Pratama, Yu- Yung Tai, Chi-Keung Tang, and Kwang- Ting Cheng. Cascaded deep monocular 3d human pose estimation with evolutionary training data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
June 2020. 7
[27] Wenhao Li, Hong Liu, Runwei Ding, Mengyuan Liu, Pichao Wang, and Wenming Yang. Exploiting temporal contexts with strided transformer for 3d human pose estimation. IEEE Transactions on Multimedia, 2022. 7
[28] Wenhao Li, Hong Liu, Tianyu Guo, Hao Tang, and Runwei Ding. Graphmlp: A graph mlp-like architecture for 3d human pose estimation. arXiv preprint arXiv:2206.0644 2022. 7,7
[29] Wenhao Li, Hong Liu, Hao Tang, Pichao Wang, and Lue Van Gool. Mhformer: hyttthhypsssesis transformer for 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13147- June 2022. 1, 6,
7,9, 19
[30] Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125. 2017.
[31] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitick.. Microsoft coco: Common objects in context. In Computer ViCion-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014. 1, 16
[32] Ruixu Liu, Ju Shen, He Wang, Chen Chen, Sen-ching Cheung, and Vijayan Asari. Attention mechanism exploits temporal contexts: Real-time 3d human pose reconstruction. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 5064-5073,
2020.
12

---

[33] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10012-10022, 2021. 3
[34] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05101, 2017. 16
[35] Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Advances in neural information processing systems,
29, 2016. 5
[36] Xiaoxuan Ma, Jiajun Su, Chunyu Wang, Hai Ci, and Yizhou Wang. Context modeling in 3d human pose estimation: A unified perspective. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 6238-6288 6247,, 2021. 2
[37] Wei Mao, Richard I Hartley, Mathieu Salzmann, et al. Contact-aware human motion forecasting. Advances in Neural Information Processing Systems, 35:7356-7367, 2022. 1
[38] Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little. simple yet effective baseline for 3d human pose estimation. In Proceedings of the IEEE international conference on computer vision, pages 2640-2649, 2017. 3
[39] Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation in the wild using improved cnn supervision. In 2017 international conference on 3D vision (3DV), pages 506-516 016511.. IEEE, 2017.
3, 6, 7, 16
[40] Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Petee Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. Vnect: Real-time 3d human pose estimation with a single rgb camera. Acm transactions on graphics (tog), 36(4):1-14, 2017.
3
[41] Francesc Moreno-Noguer. 3d human pose estimation from a single image via distance matrix regression. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2823-2832, 2017.
[42] Alejandro Newell, Kaiyu Yang. and Jia Deng. Stacked hourglass networks for human pose estimation. In European conference on computer vision, pages 483-499. Springer, 2016.
[43] Georgios Pavlakos, Xiaowei Zhou, and Kostas Danilidii. Ordinal depth supervision for 3d human pose estimation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7307-7316, 2018. 3
[44] Georgios Pavlakos, Xiaowei Zhou, Konstantinos G Derpanis, and Kostas Daniilidis. Coarse- to-fine volumetric prediction for single-image 3d human pose. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7025-7034, 2017. 3
[45] Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 3d human pose estimation in video with temporal convolutions and semi-supervised training. In CVPR, 2019.
1, 3,7
[46] Zhongwei Qiu, Qiansheng Yang, Jian Wang, and Dongmei Fu. Ivt: An end-to-end instance- guided video transformer for 3d pose estimation. In Proceedings of the 30th ACM International Conference on Multimedia, pages 6174-61 6182, 2022. 16
[47] Jathushan Rajasegaran, Georgios Pavlakos, Angjoo Kanazawa, and Jitendra Malik. Tracking people by predicting 3d appearance, location and pose. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2740-2749, 2022.
[48] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 15:211-252, 2015. 9
[49] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resoll resh-resoluiion clothed human digitization. In Proceedings of the IEEEJCVF international conference on computer vision, pages 2304-2314,
2019. 16
[50] Wenkang Shan, Zhenhua Liu, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao. stmo: Pre-trained spatial temporal many-to-one model for 3d human pose estimation. In
13

---

Computer ECcV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27,
2022, Proceedings, Part V. pages 461-478. Springer, 2022. 1, 3, 4, 6, 7, 19, 20
[51] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings the international conference on computer vision, pages 7464-7473, 2019.
the IEEE/CVF
[52] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the /EEE/CVE conference on computer vision and pattern recognition, pages 5693-5703 2019. 7,
[53] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In Proceedings of the European conference on computer vision (ECCV), pages 529-545, 2018.
[54] Bugra Tekin, Isinsu Katircioglu, Mathicu Salzmann, Vincent Lepetit, and Pascal Fua. Structured prediction of 3d human pose with deep neural networks. In Proceedings of the British Machine Vision Conference (BMVC), pages 130-1, 2016. 3
[55] Bugra Tekin, Pablo Márquez-Neila, Mathieu Salzmann, and Pascal Fua. Learning to fuse 2d and 3d image cues for monocular body pose estimation. In Proceedings of the IEEE international conference computer vision, pages 3941-3950, 2017. 3
[56] Jingbo Wang, Sijie Yan, Yuanjun Xiong, and Dahua Lin. Motion guided 3d pose estimation from videos. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pages 764-780. Springer, 2020. 3,7
[57] Tom Wehrbein, Marco Rudolph, Bodo Rosenhahn, and Bastian Wandt. Probabilistic monocular 3d human pose estimation with normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11 1199-11209 2021. 2
[58] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision (ECCV), pages 466-481 2018. 9
[59] Jingwei Xu, Zhenbo Yu, Bingbing Ni, Jiancheng Yang, Xiaokang Yang, and Wenjun Zhang. Deep kinematies analysis for monocular 3d human pose estimation. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 3
[60] Tianhan Xu and Wataru Takano. Graph stacked hourglass networks for 3d human pose estima- tion. In Proceedings of the IEEE/CVV Conference on Computer Vision and Pattern Recognition (CVPR), pages 16105-16114, June 2021. 3,7
[61] Yufei Xu, Jing Zhang. Qiming Zhang, and Dacheng Tao. ViTPose: Simple vision transformer baselines for human pose estimation. In Advances in Neural Information Processing Systems,
2022. 18
[62] Binyi Yin, Dongho Zhang, Shuai Li, Aimin Hao, and Hong Qin. Context-aware network for 3d human pose estimation from monocular rgb image. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2019. 16
[63] Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang. Hrformer: High-resolution vision transformer for dense predict. Advances in Neural Information Processing Systems, 34:7281-7293, 2021. 5,8
[64] Ailing Zeng, Xiao Sun, Fuyang Huang, Minhao Liu, Qiang Xu, and Stephen Lin. Srnet: Improving generalization in 3d human pose estimation with split-and-recombine approach. In ECCV, 2020. 7
[65] Ailing Zeng, Xiao Sun, Lei Yang, Nanxuan Zhao, Minhao Liu, and Qiang Xu. Learning skeletal graph neural networks for hard 3d pose estimation. In Proceedings of the IEFEJCVF International Conference on Computer Vision, pages 11436-11445 2021. 3, 6,7
[66] Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun. Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. In Proceedings of the IEEEJCVF International Conference on Computer Vision, pages 11446-11456 2021. 16
(67] Jinlu Zhang, Zhigang Tu, Jianyu Yang, Yujin Chen, and Junsong Yuan. Mixste: Seq2seq mixed spatio-temporal encoder for 3d human pose estimation in video. In Proceedings of the
14

---

IEEEJCVF Conference on Computer Vision and Pattern Recognition, pages 13232-132444 2022. 1,3.4. 6, 7, 16
[68] Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N. Metaxas. Semantic graph convolutional networks for 3d human pose regression. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3425-3435, 2019. 4
[69] Qitao Zhao, Ce Zheng. Mengyuan Liu, Pichao Wang, and Chen Chen. Poseformerv2: Exploring frequency domain for efficient and robust 3d human pose estimation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, June 2023. 3,4
[70] Weixi Zhao, Weiqiang Wang, and Yunjie Tian. Graformer: Graph-oriented transformer for 3d pose estimation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20406-20444 2022. 3,7
[71] Ce Zheng, Xianpeng Liu, Guo-Jun Qi, and Chen Chen. Potter: Pooling attention transformer for efficient human mesh recovery. IEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), 2023. 6
[72] Ce Zheng, Matias Mendieta, Taojiannan Yang, Guo-Jun Qi, and Chen Chen. Feater: An efficient network for human reconstruction via feature map-based transformer. In Proceedings of the IEEEJCVE Conference on Computer Vision and Pattern Recognition, pages 13945-13954, 2023. 3
[73] Ce Zheng, Wenhan Wu, Chen Chen, Taojiannan Yang, Sijie Zhu, Ju Shen, Nasser Kehtarnavaz, and Mubarak Shah. Deep learning-based human pose estimation: A survey, ACM Computing Surveys, 56(1):1-37, 2023. 3
[74] Ce Zheng, Sijie Zhu, Matias Mendieta, Taojiannan Yang, Chen Chen, and Zhengming Ding. 3d human pose estimation with spatial and temporal transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 11656-11665, October 2021. 1, 2.
3, 4, 6, 7, 9, 10, 16, 17
[75] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-ena object detection. arXiv preprint arXi::001.004159, 2020. 2. 3, 7, 18
15

---

Appendix
A Additional Discussion on Related Work
Involving 2D joint heatmaps in image-based 3D human pose estimation has also been explored in some works [62, 18]. Yin et al. [62] split the 3D human pose estimation into two sub-tasks: 2D pose estimation and depth regression. Habibie et al. [18] disentangle features for the 2D pose from global image features to exploit large-scale 2D pose datasets. These approaches do not explicitly learn features from 2DD poses and lift them to 3D.
Keypoint-basee feature extraction, IVT [46] formulates video-based multi-person 3D pose estima- tion an end-to-end framework. The features of each instance in the video are gathered by learning joint offsets from the body center. Compared to IVT, our work follows the standard two-stage pipeline of 3D HPE, and the estimated 2D joints serve as reference points in feature extraction. Pixel-aligned features [49, 66] in the context of 3D human reconstruction use point-wise image features similar to our approach. Their feature extraction based on the re- projecction of estimated 3D meshes [66] or query 3D points [49] instead of 2D human joints from off-the-she 2D pose detectors. Pixel-aligned features primarily aim reducing the misalignment between estimated 3D representations and input 2D images. In contrast, we propose joint-context feature extraction to reduce ambiguities in lifting-based -base HPE and alleviate current methods' reliance on temporal information.
B Implementation Details
2D pose detector settings. Our method's overall pipeline includes an off-the-self 2D pose detector and lifting model. For the first stage, the 2D pose detector is pre-trained on the COCO [31] dataset, without finetuning on the 2D poses from 3D pose estimation datasets, i.e., Human3.6 66 [21] and MPI-INF-3DHP [39]. We 256 192 resolution for input images, and we found higher image resolution improves performance. For DD-to-3D lifting, the weights of pre- pre-trained 2D detectors are frozen, i.e. no finetuning on the 3D task is needed. This approach makes our method preferably flexible - our method is compatible with a wide range of off-the-sheeff (pre-trained) 2D pose detectors. We show in Sec. 4 that our method gains consistent improvements by increasing the capability of 2D) pose detectors. In the future, we may leverage more advanced 2D pose detectors to further improve the performance upper bound.
Lifting model settings. Our lifting model includes three basic modules: Deformable Context Extraction, Pose-Contexi Feature Fusion and Spatial Inter-joint Modeling. The layer number of each module is set to 4, following PoseFormer [74]. The hidden dimension (a shared projection dimension C) of the model is 128. We use 8 heads in self-attention for transformer layers.
Training details. The experiments are conducted on a single NVIDIA RTX 3090 GPU. Our lifting model is trained using the AdamW [34] optimizer for 50 epochs with batch size of 512. The initial learning rate is 6.4e-3 with an exponential learning rate decay schedule, and the decay factor is 0.99.
C A Simple Temporal Extension of Our Method
This section shows that our single-frame method can naturally extend to model temporal dependencies (an overview is in Fig. 8). As illustrated Sec. 3.2, the Sputial Inter-joint Modeling module outputs a feature vector of dimension (I. + 1) x C for each joint, where I. is the number of multi-scale feature maps and C is a shared projection dimension of the model. We use a Temporal Transformer to model temporal correlations of cach joint independently. Specifically, for Temporal Transformer, the input token number is the total frame number, and each token is of dimension (L + 1) x C. Using transformers to build up temporal dependencies is straightforward, and this approach has been adopted by PoseFormer [74], MixSTE [67], etc. The output of the temporal transformer encoder can be denoted by Zuemp R [xFX(((+)))) where J is the joint number and F is the frame number. Following PoseFormer, we use 1D convolution to reduce its temporal dimension (gather temporal information) and linear layer to obtain the final estimated 3D pose y E R/3
Quantitative results. Due to limited computational resources, we prepare contextual feature vectors (that are sampled at detected joints from multi-level feature maps) in advance and remove the
16

---

Target 3D Pose
Regression Head
Temporal Transformer
Spatial Inter-joint Modeling
Spatial Inter-joint Modeling
Spatial Inter-joint Modeling
N,
Pose-Context Feature Fusion
Pose-Contexs Feature Fusion
Pose-Contert Feature Fusion
x N2
x N2
x No
F1
p2
FP
F2
F3
F
F2
PE6
First frame
Second frame
Third frume
Figure Architecture of our method's simple temporal extension (3 frames as input).
Table 6: We compare the (short-term) temporal extension of our small model (CA-PF-S) with PoseFormer. MPJPE: Mean Per Joint Position Error, the precision metric. MPJVE: Velocity Error, the temporal smoothness metric. The results reported Human3.6% (in millimeters).
Model
Frame
MPJPE J
MPJVE 1
53.2
13.7
51.0
7.1
PoseFormer
49.9
4.8
81
44.3
3.1
44.7
8.5
44.2
4.8,37
CA-PF-S
9
43.4
3.4
40.2
2.1 2.1
Deformable Context Extraction module. Therefore, the input to our model is discrete feature vectors instead of image sequences. This approach largely reduces the memory cost. We also reduce the shared projection dimension C in our model to speed up training. This small variant of model is referred to as "CA-PF-S", which has fewer FLOPs compared to our full model "CA-PF" presented in Sec. To show the gains precision and temporal smoothness from temporal modeling, we report two metrics, MPJPE (Position Error, the precision metric) and MPJVE (Velocity Error, the temporal smoothness metric), on Human3.66 [21]. We compare with PoseFormer [74] and the results are in Table 6: First, increasing the number of input frames brings consistent improvements both precision and temporal smoothness. For example, by using only video frames, the MPJVE of our method decreases from 8.5 to 4.8mm (a 43.5% reduction). and the MPJPE is reduced by 1.1%. This indicates that even short-term temporal modeling largely mitigates jitters in estimated results and further improves precision upper bound. Second, considering the same number of input frames, our CA-PF-S consistently outperforms PoseFormer in terms both MPJPE and MPJVE. Moreover, our 3-frame CA-PF-S has already achieved superior MPJPE to 8 31-frame PoseFormer, and our 9-frame CA-PF-S achieves comparable MPJVE with 81-frame PoseFormer. The results verify the two benefits of spatial contextual clues from 2D pose detectors accuracy and temporal stability.
Based on the results above, we expect that our method can successfully extended to model even longer-term temporal dependencies (e.g., 81 video frames) to further boost precision and temporal smoothness. However, modeling such long input sequences with acceptable computational costs non-trivial and is out of the scope of this paper, which is left as our future direction.
17

---

Table 7: Ablation study on the ViT backbone and pre-training datasets.
Pre-training
Multi-scale Design
mAP
MPJPE on
Index
Backbone
Datasets
COCO 1
Human3 66M 1
CPN
COCO
68.6
41.6
HRNet-32
COCO
/
74.4
41.4
ViT
COCO
X
75.8
44.5
4
VIT
COCO+AIC+MPII
X
77.1
41.9
D More Ablations on Backbones and Pre-training Datasets
We additionally conduct experiments on ViTPose [61], the recent state of the art on 2D HPE. The weights of ViTs [13] in ViTPose initialized with MAE [19] pre-training, and then the model is further trained on 2D HPE datasets. The results are in Table 7, and observations are as follows.
Backbone design outweighs the results on 2D HPE. The first three rows in Table 7 demonstrate that better results on 2D HPE not necessarily translate to better performance on 3D HPE: Although ViTPose performs best COCO 2D HPE, it achieves worst results Human3.66 M.MM. We attribute this result to the lack of multi-scale network design. ViTPose gains from powerful MAE pre-training with modern transformer architecture, while its network design is arguably simplified for 2D HPE. Specifically, ViTPose processes on tokenized image patches with transformers and finally increases the resolution of feature maps using 2D deconvolution layers. They use multi-scale designs, e.g. high-resolution feature branches or multi-scale feature fusion, as in CPN and HRNet. However, such techniques may help our approach learn more task-relevant information (i.e. joint-context features) to localize joints in 3D.
More pre-training datasets on 2D HPE improve the performance on the 3D Task. A comparison between row 3 and row 4 in Table shows that multi-datase pre- pre-training improves performance on both 2D HPE and 3D HPE. Plus, the gains 3D HPE (5. 15.8% error reduction) significant than those on 2D HPE (1.3 points improvement on AP). We hypothesize that multi-dataset pre-training improves the generalization ability of learned backbone features. Therefore, the best. performance on 2D HPE ViTPose better transfers to the 3D task.
more
Figure 9: Visualization of consecutive frames the Human3.cr M.MM test set where severe self-occlusion occurs. Deformable Context Extraction learns sampling points that attempt to discover ground truth joints given false 2D joint detection reference.
E
More Details and Visualization for Deformable Context Extraction
More details. The Deformable Context Extraction module extracts informative joint-centric context features from feature maps using deformable attention [75]. The sampling points of each attention head are initialized in different directions (w.nt. the reference joint) to promote Icarning diverse
18

---

contextual clues from images. To prevent overly aggressive updates on sampling offsets (e.g., they may go outside the image), the learning rate of linear layers that generate sampling offsets set to 6.4e-4 (1/10 of that for other layers in the lifting model). We use 4 heads for deformable attention.
Visualization of learned sampling points on consecutive video frames. In Fig. 9, subject in the Human3.6M 666 test set raises his right arm where severe self-occlusion occurs, and the 2D pose detector fails to localize the right wrist (blue dots are detected results, and green dots are ground truth). We find that most sampling points (gray dots) are concentrated on the upper body of the subject. More interestingly, despite the unreliable 2D joint detection (reference points), some learned sampling points attempt to approach the ground truth. We indicate the sampling points that gain larger attention weights with higher brightness (we decrease the brightness of images for better visual effects). Note that we do not train sampling points using ground truth. This indicates that our adaptive context extraction strategy can learn informative contextual features based on the visual cues of images despite bad sampling references (i.e., false joint detection), which helps reduce uncertainty brought by imperfect 2D pose detectors.
F More Visualization
We provide more qualitative results Human3.6N 6. (Fig. 10) and MPI-INF-3DHP (Fig. 11). Our single-frame method obtains reliable estimation in hard cases, e.g., self-occlusion, and rare poses, compared to kate-of-the of-the-art art multi-free callt-framm methods such as 351-framd MHFormer [29] and 81-mre P-STMO [50].
Irput
MRFermer
Context agnostic
Ours
Ground Truth
Irput
MHFormer
Contestegnowiic
Ours
Groud Trurh
Figure 10: Qualitative comparison with MHFormer (351 frames) [29] and our context-agnostic counterpart (please refer Sec. 4.3 for more details) Human3.6M. Our method obtains reliable results despite severe self-occlusion, which may cause false 2D joint detection. Notable parts are indicated by arrows.
19

---

Input
P-STMO
Ours
Ground Truth
Input
P-STMO
Ours
Ground Truth
Figure 11: Qualitative comparison with P-STMO (81 frames) [50] on MPI-INF-3DHP Our method infers correct results given rare poses (e.g., the subject is lying the ground and relaxing the couch). Notable parts are indicated by arows or circles.
20


Reviews:
[{"review_id": "AFSyXcW3fg", "reviewer": "Reviewer_THhA", "summary": "This paper introduces Context-Aware PoseFormer, a new approach that leverages intermediate visual representations from pre-trained 2D pose detectors to implicitly encode spatial context for 3D human pose estimation. Despite the simple network structure, the proposed method outperforms existing state-of-the-art single- and multi-frame methods by a large margin without relying on temporal information.", "questions": "Please see the weakness", "limitations": "Please see the weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-This paper provides an interesting insight that leverages the spatial contextual information of pre-trained 2D pose detectors, referred to as \"context-aware\" information, to improve the accuracy of 3D human pose estimation. \n\n- The proposed approach provides considerable benefits by elevating the accuracy of single-frame methods to a level on par with state-of-the-art multi-frame methods", "weaknesses": "-Incremental novelty in the idea of context aware features and lack of discussion on the existing context aware features.\n   - Modeling context-aware operations or networks has been studied in many other computer vision topics. For example, pixel-aligned features [1, 2] have been extensively used in 3D human/hand reconstruction from single/multi images. This pixel-aligned feature is demonstrated to be the most useful module in Table 3 while other modules only improve the performance marginally.\n\n- Incomplete evaluation regarding the pretrained context-aware feature network.\n    - For the pretraining, what's the impact of different pre-training datasets (e.g. number of training data) on the estimation? \n    -  For the pretraining backbone, what's the impact of latest backbone on the final results?  e.g. ViT-based [3] pre-trained backbones MAE [4].\n\nReference\n- [1] Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. ICCV 2021\n- [2] Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. ICCV2019\n- [3] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale ICLR\n- [4] Masked autoencoders are scalable vision learners. CVPR 2022\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author Response to Reviewer THhA\n## 1. Clarifications on Novelty\nWe disagree that the novelty of our idea is incremental. We hope the reviewer could check our `General Response` for more comprehensive clarifications on the novelty of our work.\n\nWe thank the reviewer for sharing related works that leverage pixel-aligned features [1,2], and we will discuss them in our `Related Work (Sec. 2)` in the final version! However, our method can be distinctively differentiated from them in the following aspects: \n\n**1) Motivation:** Pixel-aligned features [1,2] were primarily proposed to **reduce the misalignment** between estimated 3D representations and input 2D images. However, we propose to leverage joint-context features to **reduce ambiguities** in lifting-based 3D HPE, e.g., depth ambiguity, thus removing the need for long-term temporal processing. \n\n**2) Source of Feature Maps and the Way to Extract Features of Interest:** Even though our works use feature maps in a point-wise manner similar to pixel-aligned features [1,2], technical details differ significantly. First, while PyMAF [1] and PIFu [2] introduce an **independent** image encoder to produce image features, we **reuse** the features maps from 2D pose detectors, which are *inherently* part of the two-stage 3D HPE pipeline. Plus, we extract joint-context features based only on **estimated 2D joints**. On the contrary, pixel-aligned features [1,2] are extracted by **projecting estimated 3D** meshes [1] or query 3D points [2] on the image plane with camera parameters to enforce the consistency between 2D input and 3D estimation. We do not use such 3D-to-2D projection. \n\n**3) Different Research Task and Pipeline:** Given input images, PyMAF [1] reconstructs human *meshes*, and PIFu [2] learns an implicit function for human *surfaces* and *textures*. In contrast, our work focuses on lifting-based 3D HPE, namely inferring 3D *joint coordinates* from images where 2D human joints are intermediate representations. PyMAF [1] and PIFu [2] do *not* use 2D human joints.\n\n[1] Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. ICCV 2021. \n\n[2] Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. ICCV 2019.\n\n## 2. Importance of Different Modules\nWe agree in Tab.3 joint-context features bring the most significant improvements over other modules. However, *Pose-Context Feature Fusion* and *Deformable Context Extraction* also provide *non-trivial* gains (**1.6%** and **3.3%** respectively). We want to highlight the **difficulty** of such gains from the two modules:\n1. The two modules build on an **already-competitive** baseline (43.5mm on Human3.6M, lower is better), which is comparable to many state-of-the-art methods, e.g., 351-frame MHFormer (43.0mm, CVPR'22) and 243-frame P-STMO (43.0mm, ECCV'22);\n2. They bring an error reduction of  2.1mm (from 43.5 to 41.4mm) together, while MHFormer (CVPR'22) and P-STMO (ECCV'22) only outperform the prior art PoseFormer (44.3mm, ICCV'21) by 1.3mm.\n\n## 3. More Evaluation\nWe also thank the reviewer for the advice on more evaluation regarding the pre-trained backbone networks. **We have already conducted experiments with commonly used 2D pose detectors**, e.g., SimpleBaseline (ResNet-50), CPN, and HRNet, on two pre-training tasks, i.e., (the most popular) COCO 2D HPE and ImageNet classification. Such comparisons can be found in the `main paper (Sec. 4.1)` and `supplementary material (Sec. 6)`.\n\nWe additionally conduct experiments on ViTPose [3], the recent SOTA on 2D HPE. The weights of ViTs in ViTPose were initialized with MAE [4] pre-training, then the model was further trained on 2D HPE datasets. The results and observations are summarized below.\n\n[3] ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation. NeurIPS 2022.\n\n[4] Masked autoencoders are scalable vision learners. CVPR 2022.\n\n| Index | Backbone | Pre-training Datasets | Multi-scale Design | mAP on COCO$\\uparrow$ | MPJPE on Human3.6M$\\downarrow$ (mm) |\n|:--:|:--:|:--:|:--:|:--:|:--:|\n|1|CPN|COCO|Y|68.6|41.6|\n|2| HRNet-32 |COCO|Y|74.4|41.4|\n|3|ViT [3]|COCO|N|75.8|44.5|\n|4|ViT [3]| COCO+AIC+MPII |N|77.1|41.9|\n\n**Note:** CPN and HRNet do *not* provide official multi-dataset pre-trained weights, so we apply this setting to ViTPose.\n\n### 3.1 The Impact of Backbone Design\n**Backbone Design Outweighs the Results on 2D HPE.** The first three rows in the table above demonstrate that better results on 2D HPE do *not* necessarily translate to better performance on 3D HPE: Although ViTPose performs best on COCO 2D HPE, it achieves worst results on Human3.6M. We attribute this result to the lack of multi-scale network design. ViTPose gains from powerful MAE pre-training with modern transformer architecture, while its network design is arguably simplified for 2D HPE. Specifically, ViTPose processes on tokenized image patches with transformers and finally increases the resolution of feature maps using 2D deconvolution layers. They use *no* multi-scale designs, e.g., high-resolution feature branches or multi-scale feature fusion, as in CPN and HRNet. However, such techniques may help our approach learn more task-relevant information (i.e., joint-context features) to localize joints in 3D.\n\n### 3.2 The Impact of Pre-training Datasets\n**More Pre-training Datasets on 2D HPE Improve the Performance on the 3D Task.** A comparison between row 3 and row 4 in the table above shows that multi-dataset pre-training improves performance on both 2D HPE and 3D HPE. Plus, the gains on 3D HPE (5.8% error reduction) are even more significant than those on 2D HPE (1.3 points improvement on AP). We hypothesize that multi-dataset pre-training improves the generalization ability of learned backbone features. Therefore, the best performance on 2D HPE of ViTPose better transfers to the 3D task.\n\nWe find these results inspiring and will incorporate them in `Sec. 4` of the final version."}, {"review_id": "B9eOzqMsYU", "reviewer": "Reviewer_ez57", "summary": "This paper leverages the readily available intermediate visual representations for 3d human pose estimation. The method discards temporal information to solve the time-intensive issue of existing lifting-based methods. The authors design a simple pipline, named Context-Aware PoseFormer, to extract informative context features and fuse these features to learn more positional clues.\n\nAfter reading authors' rebuttal, my previous concerns were all addressed.", "questions": "1. Does the resolution of input images influence the performance?\n\n2. It would be interesting to apply the proposed method to other related tasks like hand pose estimation to verify its generalization capability.", "limitations": "The authors should describe the limitation of the proposed method.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.The paper is generally well written. The problem to address and the shortcomings of the existing approaches are discussed well. \n\n2. The motivation and observation are clear and meaningful, and the network is well designed to solve these problems.\n\n3.The experiments demonstrate well the superiority of the proposed method over the prior art. The authors also provide adequate ablation studies.", "weaknesses": "More STOA methods in 2023 should be chosen for comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author Response to Reviewer ez57\n## 1. More Comparisons with SOTA in 2023\nWe thank the reviewer for the advice on performance comparison! We agree that comparing with more papers would make our work more complete. We include recently released CVPR'23 and ICCV'23 papers in the table below and will incorporate the results in our final version. \n\n|         Method        |   Venue  | 2D Pose Detector | Frame Number | MPJPE on Human3.6M$\\downarrow$ (mm) |\n|:---------------------:|:--------:|:----------------:|:------------:|:-----------------------------------:|\n| MPM [1]               | arXiv'23 |        CPN       |      243     |                 42.3                |\n| STCFormer [2]         |  CVPR'23 |        CPN       |      243     |                 41.0                |\n| GLA-GCN [3]           |  ICCV'23 |        CPN       |      243     |                 44.4                |\n| CA-PF-CPN (ours)      |          |        CPN       |       1      |                 41.6                |\n\n[1] MPM: A Unified 2D-3D Human Pose Representation via Masked Pose Modeling. arXiv 2023.\n\n[2] 3D Human Pose Estimation with Spatio-Temporal Criss-cross Attention. CVPR 2023.\n\n[3] GLA-GCN: Global-local Adaptive Graph Convolutional Network for 3D Human Pose Estimation from Monocular Video. ICCV 2023.\n\nAll approaches listed above (except ours) use **243** video frames, while we only use a **single** frame. Compared to the latest SOTAs in 2023, our approach is still highly competitive: Our CA-PF-CPN outperforms MPM [1] (arXiv'23) and GLA-GCN [3] (ICCV'23) by 1.7% and 6.8% respectively and achieves comparable performance with STCFormer [2] (CVPR'23). Our performance can be further improved using larger backbones (`Tab. 1, main paper`) or temporal information (`Sec. 3, supplementary`).\n\n## 2. The Impact of Image Resolution\nThanks for such an insightful question! We think it is exciting and meaningful to explore the impact of image resolution on our method. Our paper uses images of size 256 $\\times$ 192, a standard setting in COCO 2D HPE. As 384 $\\times$ 288 is also widely used for input image size in 2D HPE, we experiment on our CA-PF-HRNet-32 with such resolution, and the results are shown below.\n\n|         Method        |    Image Size    | GFLOPs of Backbone | MPJPE$\\downarrow$ (mm) |\n|:---------------------:|:----------------:|:------------------:|:----------------------:|\n| CA-PF-HRNet-32 (ours) | 256 $\\times$ 192 |         7.1        |          41.4          |\n|           /           | 384 $\\times$ 288 |        16.0        |          39.3          |\n\nIncreasing the image size from 256 $\\times$ 192 to 384 $\\times$ 288 brings a 5.1% error reduction. While the model architecture and hidden dimensions are unchanged for the backbone network, it produces larger feature maps. Larger feature maps encode more fine-grained joint-context features and therefore improving performance. This result is consistent with our finding in `Tab. 4 of the main paper`, showing that high-resolution features contribute more than low-resolution ones. We will incorporate the result and analysis in our `Abation Study (Sec. 4.3, main paper)`.\n\n## 3. Generalization Ability to Related Fields\nWe apologize for being unable to show the results at present, as our computational resources are limited. Exploring the generalization ability of our approach to other domains will definitely be our future direction! For example, we expect our method to work on human mesh reconstruction, hand pose estimation, and, more generally, other 3D tasks where the 2D skeleton representation of the target 3D object is readily available. We believe leveraging the well-learned 2D features that produce such skeletons will reduce ambiguities in 2D-to-3D lifting and significantly improve the performance of the 3D task. We will open a `Future Work` section in the final version to discuss potential directions.\n\n## 4. Discussion on Limitation\nIn the `supplementary`, we have included the discussion on the limitation of our method in `Sec. 2` and our corresponding solution in `Sec. 3`. Please check!\n\n"}, {"review_id": "YcYqPvilpE", "reviewer": "Reviewer_4XvM", "summary": "The paper targets the challenging 3D pose estimation problem based on a new context-aware lifting algorithm. The proposed approach is simple to implement and reproduce. Attractive experimental results have been reported on the challenging benchmarks. Also, the detailed ablations well validate the design of the proposed algorithm. ", "questions": "Please address the questions in the weakness section. ", "limitations": "The paper does not discuss the potential limitations of the proposed algorithm.  I would suggest the authors to have a discussion of the limitations (like the cases with high self-occlusion which temporal information may be helpful).", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of lifting 2d image features for 3D human pose estimation is interesing. \n2. The proposed algorithm report state-of-art performance on the 3d pose estimation benchmark. \n3. It also provides sufficient ablation studies to validate the algorithm design of the proposed approach. ", "weaknesses": "1. In stage 2, it involves a Deformable context extraction module to extract the context features (F1, F2, F3, P). How is the feature difference against H1-H3? Also, is it possible to add H1-H3 with positional embeding to the pose-context feature fusion module?\n\n2. It would be reasonable that 2D pose information can be helpful for the 3d pose estimation. But lifting from 2d to 3d is not-trivial. How could the proposed approach guarantee the consistency with the 3d location of the pose? \n\n3. If there are more temporal frames available, is it possible to further boost the performance of the proposed approach? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author Response to Reviewer 4XvM\n## 1. Details about Deformable Context Extraction\n$H_1$-$H_3$ refer to the *raw feature maps* produced by 2D pose detectors, while $F_1$-$F_3$ are *extracted feature vectors* named \"Context Features\" from the corresponding feature maps using *Deformable Context Extraction* (DCE). Specifically, we initialize $F_1$-$F_3$ as the feature vectors directly sampled at the detected 2D joints from $H_1$-$H_3$. As the detected joints inevitably introduce noise, e.g., due to occlusions, we use deformable attention [1] to produce a small set of sampling points around each detected joint and fuse their features. We provide a visualization of sampled points in `Fig. 4 of the main paper`. Such sampling-and-feature-mixture process happens for N1 (the layer number of DCE) times, and the final output of DCE that represents the context features for each joint from each feature map is denoted by $F_1$-$F_3$. $P$ is the linear embedding of 2D joint coordinates. Please check `Fig. 3` and `Sec. 3 (L173-200) in the main paper` and `Sec. 5 in the supplementary` for more details and visualization! \n\n[1] Deformable DETR: Deformable Transformers for End-to-End Object Detection. ICLR 2021.\n\n## 2. Add H1-H3 to the Pose-Context Feature Fusion module\n*Pose-Context Feature Fusion* primarily aims at fusing image cues (joint-context features, $F_1$-$F_3$) and position cues (joint-coordinate embedding, $P$) for each joint using transformers. Adding $H_1$-$H_3$ to *Pose-Context Feature Fusion* is possible and may provide more improvements as they intuitively encode rich context features. However, some concerns should be addressed:\n1. Processing feature maps ($H_1$-$H_3$) generally costs more GPU memory and computation compared to vectorized features ($F_1$-$F_3$) since $H_1$-$H_3$ also contain features for uninformative background;\n2. While it is convenient for transformers to process feature vectors ($P$, $F_1$-$F_3$) with the same dimension, it is not straightforward to process feature vectors ($P$) and feature maps ($H_1$-$H_3$) simultaneously as they have different shapes. We may need extra model design to make such a process possible.\n\nGiven the concerns above, we prefer to use vectorized features ($F_1$-$F_3$) rather than feature maps ($H_1$-$H_3$). Since we extract $F_1$-$F_3$ from $H_1$-$H_3$ based on detected 2D joints, they are reasonably informative to provide desirable joint-context features. We thank the reviewer for such an enlightening question, and we will incorporate the analysis and explanations about module design in our `Method` section in the final version.\n\n## 3. How to ensure 2D-to-3D consistency?\nThis is **not** a weakness of our method! We follow the *common practice* in the domain: the format (e.g., joint number and type) of 2D joints and 3D joints (input-output pairs) are pre-defined and aligned before training. Then our network learns to lift 2D joints to 3D via supervised learning using paired 2D and 3D data.\n\n## 4. Multi-frame Extension\n\n|  Method | Frame Number | MPJPE$\\downarrow$ | MPJVE$\\downarrow$ |\n|:-------:|:------------:|:-----------------:|:-----------------:|\n| CA-PF-S |       1      |        44.7       |        8.5        |\n|    /    |       3      |        44.2       |        4.8        |\n|    /    |       9      |        43.4       |        3.4        |\n|    /    |      27      |        40.2       |        2.1        |\n\n**Note:** CA-PF-S is a small model variant since temporal processing requires more computation and memory. The last row (27 frames) is our new result.\n\nIn `Sec.3 of the supplementary`, we have described a simple way to extend our single-frame approach to model temporal dependencies. Here we move some results from `Supp. Tab. 1` with a **new** result using 27 frames, showing that by using more video frames, our method *gains consistently* in terms of *performance* (position error, MPJPE) and *temporal smoothness* (velocity error, MPJVE). Given the results above, we expect our model to improve further if we increase the frame number.\n\n## 5. Discussion on Limitation\nIndeed, we **have included** the discussion on the limitation of our method in `Supp. Sec. 2`. Please check! Besides, we agree that temporal information helps to resolve self-occlusion by providing features from unoccluded video frames. Interestingly, we find that *contextual features may also improve the robustness of the model to self-occlusion*. In `Fig. 5 of the supplementary`, we provide two video clips where strong self-occlusion (due to pose or clothing) makes the 2D human pose unreliable. Despite inaccurate 2D input, our method gives robust results since we also leverage spatial contextual clues from images to localize joints in 3D in addition to noisy 2D joints. \n\nWe think \"how our model deals with self-occlusion and how temporal information helps model\" would provide exciting insights, and we will include more analysis in the final version. Plus, we will move the `Limitation` section to the main paper. We thank the reviewer for such an insightful suggestion. "}, {"review_id": "8Vo5tEn3CL", "reviewer": "Reviewer_jUmb", "summary": "This paper proposes Context-Aware PoseFormer, the key idea is to extract multi-scale spatial features for 3D pose lifting from 2D pose.\nThe authors claim that the single-frame beats hundreds of frames and demonstrate that their single-frame methods achieve comparable or better results than multi-frame methods, such as poseformer.\n\n ", "questions": "see weakness", "limitations": "No", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The motivation of this article is clear, and the writing is simple and easy to understand.\nThe experiments in this paper verify the effect of the proposed method, verify that the spatial context information is very important, and the effect of the proposed context aware poseformer.\nThe ablation study in this article is detailed and verifies the effect of each module in the method.", "weaknesses": "I am skeptical of the claim in this paper that the single frame is better than the multiple frame method. This article only verifies that his single-frame method is better than some multi-frame methods such as poseformer, but there are 3 problems below：\n1) The comparison of the methods in table 1 is unfair. For example, other methods use the 2D pose result of CPN as the output of the lifting network, while the method in this paper uses HRNet. It is well known that HRNet can perform better 2D detection than CPN.\n2) In addition, the method in this paper also inputs more visual features from images, so this paper should also be compared with other methods that use visual features, such as IVT.\n3) If the single-frame method in this paper works well under the same 2D network, it cannot be verified that it is definitely better than the multi-frame method. In the method of this article, is it possible to achieve better results by adding timing information? This is the consensus in the field, and this paper should also verify this.\n\nIn addition, the core contribution of the method in this paper is the multi-scale visual features of the image. The author uses a deformable method to extract multi-scale features to improve the performance of the network.\nIn the field of 3D posture, it is a consensus that extracting better spatial context features can improve performance.\nIn terms of context information extraction, this article has no additional contribution, only using other people's methods.\nTherefore, in general, the method in this paper is a bit incremental in terms of innovation, and the contribution is not enough to be accepted by NeurIPS。", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author Response to Reviewer jUmb\n## 1. Clarifications on Our Claim\nIn this paper, we enable a single-frame approach to outperform multi-frame ones (e.g., that uses 351 frames) for the first time. We believe that strong experimental results (`Tab. 1` and `Tab. 2 in the main paper`) well verify the effectiveness of our method, and the reviewers (**THhA**,**ez57**,**4XvM**,**dcLQ**) also appreciate our results. We also include more recent papers for comparison. Please check the `Performance Comparison` section below!\n\nAs new works are emerging in this field, we agree that we can *never* beat all multi-frame methods at any time, and therefore **we have *not* reached such a highly definitive conclusion**. However, we do **open up the *possibility* of significantly improving the performance of 3D HPE *in a highly disadvantaged position*** (i.e., with *no* access to temporal information), and the results we achieve are already encouraging and attractive (**4XvM**). We are open to suggestions from the reviewer to help us remove such confusion in the final version.\n\n## 2. Performance Comparison \n### 2.1 Fairness: \nWe believe **comparing our CA-PF-CPN model variant with other state-of-the-art methods is fair**, as CPN is consistently used as the 2D pose detector (in `Tab. 1` we use one column to show the 2D pose detector used by different methods). In such a fair setting, our method (41.6mm, lower is better) outperforms a series of works, e.g., 81-frame PoseFormer [1] (44.3mm), 351-frame MHFormer [2] (43.0mm), and 243-frame P-STMO [3] (43.0mm), and achieves comparable results with 243-frame MixSTE [4] (40.9mm). We experiment with other backbones (e.g., HRNet) to primarily show the flexibility of our method to incorporate different 2D pose detectors. We will add more clarifications about the fairness of our experiments in `Sec. 4 of the main paper`. \n### 2.2 Completeness:\nWe had tried our best to gather both single-frame and multi-frame methods for comparison before CVPR 2023 papers were released (`Tab. 1 in the main paper`). Following the advice from review **ez57**, we compare with more recent SOTAs, including several CVPR 2023 and ICCV 2023 papers, and we will incorporate them in our final version. Please check our response to review **ez57**!\n\n[1] 3D Human Pose Estimation with Spatial and Temporal Transformers. ICCV 2021.\n\n[2] MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation. CVPR 2022.\n\n[3] P-STMO: Pre-Trained Spatial Temporal Many-to-One Model for 3D Human Pose Estimation. ECCV 2022.\n\n[4] MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video. CVPR 2022.\n\n## 3. Comparison with IVT [5] \nWe thank the reviewer for sharing such great work, and *we will include this paper in our `Related Work` section*. Our approach **principally differs** from IVT in the following aspects, although both methods use image features: \n\n**1) Motivation:** Our work aims at removing heavy temporal dependencies of existing lifting-based 3D HPE methods; IVT targets formulating video-based 3D HPE as an end-to-end learnable framework. \n\n**2) Task Pipeline:** We follow the dominant two-stage pipeline, which **explicitly uses 2D human poses as an intermediate representation**. We do **not** use any temporal information; IVT **directly** estimates 3D human poses from **videos** (2D human poses are **not** involved as an intermediate output, and **they use temporal information**). \n\n**3) Implementation Details:** We use input images of size 256 $\\times$ 192, while IVT uses much larger images of size 512 $\\times$ 512. \n\nSince IVT is **not open-sourced**, we are unable to conduct a direct comparison. However, more analysis and discussions regarding this work will be placed in the final version to make our work more complete!\n\n[5] IVT: An End-to-End Instance-guided Video Transformer for 3D Pose Estimation. ACM MM 2022.\n\n## 4. Multi-frame Extension\n\n|  Method | Frame Number | MPJPE$\\downarrow$ | MPJVE$\\downarrow$ |\n|:-------:|:------------:|:-----------------:|:-----------------:|\n| CA-PF-S |       1      |        44.7       |        8.5        |\n|    /    |       3      |        44.2       |        4.8        |\n|    /    |       9      |        43.4       |        3.4        |\n|    /    |      27      |        40.2       |        2.1        |\n\n**Note:** CA-PF-S is a *smaller* model variant (than our full model) since temporal processing requires more computation and memory. The last row (27 frames) is our new result.\n\nThis is an insightful question! We have already verified the flexibility of our method to extend to multi-frame settings in `Sec. 3 of the supplementary`! We move some results here with a new 27-frame result. The table above shows that our approach **gains consistently** in terms of **performance** (MPJPE, position error) and **temporal smoothness** (MPJVE, velocity error) using more video frames. Please check the supplementary for more details!\n\n## 5. Concerns about Contribution \nWe disagree that \"the core contribution of our work method in this paper is the multi-scale visual features of the image.\" Leveraging visual representations from 2D pose detectors itself is a **non-trivial** and **novel idea** in the context of lifting-based 3D HPE, while our **novel framework** (**dcLQ**) to extract multi-scale joint-context features and fuse context features and joint embedding is **only one of our contributions**. Plus, we are the first to make a single-frame method outperform state-of-the-art multi-frame methods. We show our first attempt to tackle the heavy temporal reliance of existing lifting-based methods and establish strong single-frame baselines for future research.\n\nWe hope the reviewer could kindly check our `General Response` to all reviewers, where we comprehensively elucidate the **novelty** and **contributions** of our method. We are open to discussions if the review has further questions or concerns."}, {"review_id": "p17ltK0C2d", "reviewer": "Reviewer_dcLQ", "summary": "This paper proposes a context-aware method that combines the 2D image feature and the detected 2D key points for 3D human pose estimation. The joint-centric spatial context represented by the intermediate image feature is able to reduce the amguity in 3D lifting. Under this motivation, a novel framework that consists of a defrmable context extraction module, a pose-context feature fusion module and a spatial inter-joint modeling module is proposed. Experiment validates the effectiveness of the proposed modules in reducing calculation time and achieving better performance. On two standard benchmarks, the proposed method shows its effectiveness and stability without using any temporal information compared to other single-frame methods and multi-frame methods.", "questions": "1. According to the study in Sec. 3 in the supplementary material, temporal information brings less improvement in the proposed method. Why the result when using 81 frames as PoseFormer is not compared? Could the authors give more explaination about this?", "limitations": "The authors have discussed their limitation of motion jitter without temporal information in the supplementary material. But the limitation of memory consumption should also be discussed as mentioned in the weakness.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well presented and organized. The idea is clear and the method is somewhat novel.\n2. New SOTA results are achieved with a significante improvement even compared to most multi-frame methods.\n3. Experiments are well organized and the ablation study is insightful to support the claim of contributions.", "weaknesses": "1. Leveraging 2D intermediate feature is not a new idea. Some previous works also tried this route, such as [a][b][c]. Compared to other skeleton-based methods, either single-frame-based or multi-frame-based, utilizting 2D image feature will lead to more memory consumption. Such a kind of limitation needs to be considered.\n\n[a] Yin, Binyi, et al. \"Context-Aware Network for 3D Human Pose Estimation from Monocular RGB Image.\" 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 2019.\n\n[b] Habibie, Ikhsanul, et al. \"In the wild human pose estimation using explicit 2d features and intermediate 3d representations.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n\n[c] Sun, Xiao, et al. \"Integral human pose regression.\" Proceedings of the European conference on computer vision (ECCV). 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author Response to Reviewer dcLQ\n## 1. Clarifications on Novelty\nWe thank the reviewer for sharing related works, and we will discuss them in the `Related Work` section of the final version to make our work more complete. To address the reviewer's concern regarding novelty, we refer the reviewer to our `General Response`, where we comprehensively reaffirm the **novelty** and **contributions** of our approach. In addition, we would like to offer more clarifications on the difference between our method and the works [1,2,3] mentioned by the reviewer.\n\nWhile these works [1,2,3] also use image features, our work is *clearly different* for the reasons below:\n\n**1) Different Pipeline:** Our method follows the *two-stage* pipeline (2D pose detection then lifting, the dominant paradigm in recent years); Those works [1,2,3] estimate 3D human pose *end-to-end* from images *without explicitly introducing 2D human poses*. While using image features is **natural** even **stereotypical** for direct estimation (end-to-end) approaches, it is **non-trivial** to retrieve feature maps for lifting-based methods.\n\n**2) Different Feature Sources and Different Ways to Use Image Features:** In our work, we **reuse** feature maps from 2D pose detectors, which are **inherently** part of a lifting-based 3D HPE pipeline. Plus, we leverage feature maps **sparsely**, i.e., we extract and aggregate **point-wise feature vectors** based on the estimated 2D joints; Those works [1,2,3] introduce **independent** networks to extract image features and then use feature maps **as a whole** instead of in a sparse manner.\n\n**3) Different Research Focus:** We propose to leverage image features to remove the dependency of existing (two-stage) lifting-based methods on temporal information; Yin et al. [1] use image features to learn more accurate depth information; Habibie et al. [2] disentangle features for 2D pose from global image features to exploit large scale 2D pose datasets; Sun et al. [3] unify heatmap-based and regression-based methods, and learn 3D heatmaps from image features.\n\n[1] Context-Aware Network for 3D Human Pose Estimation from Monocular RGB Image. IJCNN 2019.\n\n[2] In the Wild Human Pose Estimation Using Explicit 2D Features and Intermediate 3D Representations. CVPR 2019.\n\n[3] Integral Human Pose Regression. ECCV 2018.\n\n## 2. Memory Issue\n\n|      Method      |  Venue  | Frame Number | MPJPE$\\downarrow$ (mm) | GPU Memory (MB) |\n|:----------------:|:-------:|:------------:|:----------------------:|:---------------:|\n|   MHFormer    | CVPR'22 |351|43.0|1651|\n|    MixSTE     | CVPR'22 |243|40.9|17305|\n| CA-PF-CPN (ours) | |1|41.6|721|\n\nWe agree our method would consume more GPU memory than other lifting-based methods *when the frame number is kept the same*. However, we would like to mention that our approach does *not* cost more GPU memory compared to state-of-the-art multi-frame approaches that use heavy temporal modeling.\n\nWe test the memory consumption of different methods during *stable training* using the same PyTorch code base on a single RTX 3090 GPU (24GB) with a batch size of 8. The results in the table above are from the command \"torch.cuda.max_memory_allocated()\" provided by PyTorch to return the peak allocated memory since the beginning of this program. The results show that our method costs less GPU memory than MHFormer and MixSTE by 2.3$\\times$ and 24.0$\\times$ respectively while achieving comparable performance.\n\nWe thank the reviewer for mentioning this issue and will incorporate the results and analysis in our `Limitation` section.\n\n## 3. Temporal Modeling\n\n|  Method | Frame Number | MPJPE$\\downarrow$ | MPJVE$\\downarrow$ |\n|:--:|:--:|:--:|:--:|\n| CA-PF-S |1|44.7|8.5|\n|    /    |3|44.2|4.8|\n|    /    |9|43.4|3.4|\n|    /    |27|40.2|2.1|\n\n**Note:** CA-PF-S is a *small* model variant. The last row (27 frames) is our new result.\n\n### 3.1 Less Improvement?\nWe additionally experiment on 27 frames, and the result is presented above (with previous results). We agree our temporal model gains less accuracy when using small frame numbers (e.g., 3 frames). However, more video frames bring significant improvements, e.g., our 27-frame model improves MPJPE and MPJVE by **7.8%** and  **38.2%** respectively over the 9-frame variant. We think the reason is that: Large human motions in long videos provide meaningful spatial varieties, and such temporal cues help reduce ambiguities in 2D-to-3D lifting. On the contrary, short video clips only include small motions while our joint-context features already encode such spatial changes (around joints). Therefore, improvements are minor when using small frame numbers.\n\n### 3.2 Clarifications on Not Using 81 Frames\nIn `Sec. 3 of the supplementary`, we extend our model to multi-frame settings **primarily aiming at solving the temporal consistency** issue (we discussed in `Supp. Sec. 2`), **not to pursue higher accuracy with more frames**. Important results from our experiments are listed below:\n1. Even **short-term** temporal information helps reduce jitters well (only 3 video frames bring a 43.5% reduction in velocity error);\n2. Our video-based model demonstrates better temporal consistency (and accuracy) than PoseFormer, e.g., our 9-frame model achieves less velocity error (MPJVE) than 9-frame PoseFormer (3.4 v.s. 4.8mm) and comparable results with 81-frame PoseFormer (3.4 v.s. 3.1mm);\n3. Current experiments (up to 27 frames) demonstrate **consistent** improvements in both **accuracy** and **temporal smoothness**. We expect our model to further gain improvements using more input video frames.\n\nGiven these results, the issue regarding temporal consistency should be well resolved by **short-term** temporal modeling, we think it **unnecessary** to further scale up the sequence length, which indeed **runs against our research motivation** (i.e., to reduce the reliance on temporal information). \n\nWe are open to discussions if the reviewer has further concerns or questions!"}]

Author rebuttal:
# General Response
We thank the reviewers for their careful reading and considerate feedback, and we are thrilled to receive the rating of 4 5 5 7 7!

We are glad that reviewers unanimously agree that *Context-Aware PoseFormer* is a **simple but effective approach** demonstrated by **strong experimental results** ("considerable benefits" (THhA), "superiority over the prior art" (ez57), "attractive experimental results" (4XvM), "experiments verify the effect of the proposed method" (jUmb), "new SOTA results" (dcLQ)). We’re further glad that reviewers agree that our **idea is interesting** ("interesting insight" (THhA), "idea is interesting" (4XvM)) and that our **motivation is well illustrated** with **clear writing** ("motivation and observation are clear and meaningful" (ez57), "motivation is clear" and "easy to understand" (jUmb), "paper is well presented and organized," and "idea is clear" (dcLQ)). The reviewers also agree that our ablations are "adequate" (ez57), "detailed" (jUmb), "insightful" (dcLQ), and "well validate the design of the proposed algorithm" (4XvM).

However, some reviewers raised their concerns regarding the novelty of our approach. To address those concerns, we first reaffirm the **novelty** and **contribution** of the proposed method, then answer specific questions for each reviewer in the corresponding rebuttal space, and we will incorporate all feedback in the final version.

## 1. We Identify and Tackle A **New Research Problem** 
The dominant paradigm in 3D human pose estimation (HPE) literature is lifting a 2D joint sequence to 3D (dubbed as *lifting-based* methods, in contrast to direct estimation from images). Heavily using temporal information (with up to 351 video frames) has been a *default* setting in the field and has proved to boost performance. However, we point out in this paper that such reliance on temporal information brings several problems: high computation, performance saturation, and the non-causal issue. Despite the problems, **reducing the heavy reliance on temporal modeling in lifting-based 3D HPE has not yet been visited. We for the first time enable a single-frame method to outperform state-of-the-art multi-frame methods that even use hundreds of video frames.** 

### Broader Impacts on the Community
Learning more powerful spatial-temporal representations with ever-increasing video frames (up to 351) has long been the research focus in lifting-based 3D HPE. However, **our approach breaks through such common practice (i.e., using no temporal information) while achieving encouragingly strong performance**. We establish solid baselines for the community and hope our work inspires more research to **think out of the box**: large-scale spatial-temporal modeling is not the only way to improve 3D HPE. We also believe that a research community should embrace different approaches, and our method can be such a starting point.

## 2. We Provide **Fresh Insights** regarding the Problem Cause 
We discover the fundamental cause of existing methods' heavy temporal reliance by revisiting the dominant two-stage pipeline in the literature (L42-56). We point out that the 2D skeleton alone is insufficient to deal with ambiguities in 2D-to-3D lifting, making previous works resort to long-term temporal clues to mitigate ambiguities. To address this problem, we propose to retrieve the *discarded* visual representations (intermediate feature maps) learned in the 2D pose detection stage, as such representations encode visual cues from images that potentially help to reduce ambiguities. These insights are **fresh** in the domain and appreciated by reviewer **THhA**.

## 3. We Design a **Novel Framework** to Solve the Problem
"How to leverage the visual representations learned by 2D pose detectors" is a **non-trivial** question. Naively incorporating the global image features into the lifting process may introduce unnecessary memory and computational costs on the background. We comprehensively consider the task pipeline and design a novel approach to extract most task-relevant information, named *joint-context features* from the feature maps with detected 2D joints as a reference. This approach enables us to attend to the most informative regions, i.e., the joints of our interest. Then we fuse the extracted context features with joint coordinate embedding using transformers. **The collaboration of two stages (2D HPE and 2D-to-3D lifting) and the approach to extract joint-context features from images are novel in lifting-based 3D HPE** ("novel framework" (**dcLQ**)). 

### While Using Image Features is Not New, What Makes Our Approach Different?
Although image features may have been explored in related fields, it does not indicate a lack of novelty in our work. Leveraging feature maps from 2D pose detectors is **non-trivial** in the context of lifting-based 3D HPE, as lifting only 2D joints to 3D is the *default* pipeline. **Our idea's uniqueness shines through its thorough understanding of the task setting.** We derive solutions directly from the task pipeline (i.e., leveraging readily available feature maps based on detected 2D joints, all elements are *off-the-shelf* in the pipeline), eliminating the need for external interventions (e.g., without introducing an extra network to extract image features). We believe it is important to evaluate the novelty of a research idea based on the domain challenge, as different domains may face different challenges. 

## 4. Other Merits: Our Method is **Easy** and **Simple**

1) We *reuse* 2D pose detector feature maps instead of introducing extra networks, avoiding potentially high computational overheads; 

2) 2D pose detectors are only pre-trained with 2D HPE (i.e., such backbones are largely available), requiring no finetuning on the 3D task or multi-stage training, which eases the training pipeline; 

3) Our method is compatible with different 2D pose detectors;

4) Our framework is simple (**THhA**,**ez57**,**4XvM**).


Meta-review: This paper received four positive ratings and one negative rating. The main concerns raised by the negative reviewer is that the paper mis-claimed the advantages over video-based methods. I agree with the reviewer that this should be revised in the camera ready. However, this does not seem a serious issue and as promised by the authors that they will revise the claims. Other than that, the paper presents an interesting approach which combines image features with 2D joints for 3D pose estimation. Promising results have also been achieved. So, I recommend accept for this paper.
Decision: Accept (poster)