Title: A Single 2D Pose with Context is Worth Hundreds for 3D Human Pose Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Human Pose Estimation; 2D-to-3D Lifting; Context-Aware
Abstract: The dominant paradigm in 3D human pose estimation that lifts a 2D pose sequence to 3D heavily relies on long-term temporal clues (i.e., using a daunting number of video frames) for improved accuracy, which incurs performance saturation, intractable computation and the non-causal problem. This can be attributed to their inherent inability to perceive spatial context as plain 2D joint coordinates carry no visual cues. To address this issue, we propose a straightforward yet powerful solution: leveraging the $\textit{readily available}$ intermediate visual representations produced by off-the-shelf (pre-trained) 2D pose detectors -- no finetuning on the 3D task is even needed. The key observation is that, while the pose detector learns to localize 2D joints, such representations (e.g., feature maps) implicitly encode the joint-centric spatial context thanks to the regional operations in backbone networks. We design a simple baseline named $\textbf{Context-Aware PoseFormer}$ to showcase its effectiveness. $\textit{Without access to any temporal information}$, the proposed method significantly outperforms its context-agnostic counterpart, PoseFormer, and other state-of-the-art methods using up to $\textit{hundreds of}$ video frames regarding both speed and precision. $\textit{Project page:}$ https://qitaozhao.github.io/ContextAware-PoseFormer

A Single 2D Pose with Context is Worth Hundreds for 3D Human Pose Estimation

Abstract

The dominant paradigm in 3D human pose estimation that lifts a 2D pose sequence to 3D heavily relies on long-term temporal clues ( i.e. , using a daunting number of video frames) for improved accuracy, which incurs performance saturation, intractable computation and the non-causal problem. This can be attributed to their inherent inability to perceive spatial context as plain 2D joint coordinates carry no visual cues. To address this issue, we propose a straightforward yet powerful solution: leveraging the readily available intermediate visual representations produced by o ff -the-shelf (pre-trained) 2D pose detectors - no finetuning on the 3D task is even needed. The key observation is that, while the pose detector learns to localize 2D joints, such representations ( e.g. , feature maps) implicitly encode the joint-centric spatial context thanks to the regional operations in backbone networks. We design a simple baseline named Context-Aware PoseFormer to showcase its e ff ectiveness. Without access to any temporal information , the proposed method significantly outperforms its context-agnostic counterpart, PoseFormer [74], and other state-of-the-art methods using up to hundreds of video frames regarding both speed and precision. Project page: qitaozhao.github.io / ContextAware-PoseFormer

1 Introduction

3D human pose estimation (HPE) aims to localize human joints in 3D, which has a wide range of applications, including motion prediction [37], action recognition [9], and tracking [47]. Recently, with the large availability of 2D human pose detectors [8, 52], lifting 2D pose sequences to 3D (referred to as lifting-based methods) has been the de facto paradigm in the literature. Compared to raw RGB images, 2D human poses (as an intermediate representation) have two essential advantages. On the one hand, domain gaps exist between images (input) and 3D joint locations (output), whereas this is not the case for 2D and 3D joints. Primarily, 2D joint coordinates provide highly task-relevant positional information for localizing joints in 3D. On the other, 2D coordinate representation is exceptionally lightweight in terms of memory cost. For example, in a typical COCO [31] setting, a 2D human pose requires only 17 × 2 floating points, whereas an RGB image requires 256 × 192 × 3 (or even more with higher resolution). This property enables state-of-the-art lifting-based methods to leverage extremely long-term temporal clues for advanced accuracy, e.g. , 243 video frames for VideoPose3D [45], MixSTE [67], and P-STMO [50]; large as 351 frames for MHFormer [29].

Scaling up the input has brought consistent improvements so far, yet some concerns naturally arise. First, the performance gains seem to level o ff when excessively long sequences are used as input ( e.g. , improvements are marginal when the frame number increases from 243 to 351 [29]). Second, long input sequences bring non-trivial computational costs. For instance, temporal processing using transformers (a popular choice) is expensive, especially with large frame numbers. Third, the use of

Detector

Video

Sequence

Context-Agnostic

(a)

Image

(b)

Readily Available

Stage 1

Stage 2

2D Pose

Detector

Visual Representations

: pre-trained and with weights frozen. For our method,

no finetuning is needed

for 2D detectors (i.e.,

readily available

).

: Intermediate visual representations are discarded, while 2D joint coordinates are the

only

input (to Stage 2).

: We reuse visual representations (that encode spatial contextual cues) from Stage 1 to resolve ambiguities.

3D Pose

(for the

central frame

)

(the

only

input)

Context-Aware

Lifting Model

future video frames renders real-world online applications impossible. Concretely, existing works adopt a 2D pose sequence to estimate the 3D human pose for the central frame where half of the frames in the sequence are inaccessible for the current time step, which is known as the non-causal problem [45]. While gains from long input sequences come at a cost, is there any alternative approach we can take to easily boost performance? We attempt to answer this question in our work.

We start by revisiting the whole pipeline of lifting-based methods, which involves two stages as shown in Fig. 1 (a). In Stage 1, a 2D pose detector estimates human joints for each image from a video clip, with a set of intermediate representations as byproducts, e.g. , feature maps of varying resolutions. In Stage 2, the detected 2D pose sequence (output of Stage 1) is lifted to 3D, while such representations are discarded . This can be problematic: the (multi-scale) joint-centric spatial context encoded by these feature maps is lost. We argue: the omission of spatial contextual information is the primary factor contributing to the time-intensive nature of existing lifting-based methods .

So, what makes spatial context important? As an inverse problem, 3D pose estimation inherently su ff ers from ambiguities [29, 36, 57] such as depth ambiguity and self-occlusion, yet they can be mitigated by utilizing spatial context from the monocular image. For depth ambiguity, the shading di ff erence is an indicator of depth disparity [57] whereas the occluded body parts can be inferred from visible ones with human skeleton constraints [36, 57]. Psychological studies [3, 36] provide evidence that appropriate context helps reduce ambiguities and promote visual recognition for the human perception system. Since 2D keypoints alone are unable to encode such spatial context, existing lifting-based approaches, in turn, resort to long-term temporal clues to alleviate ambiguities.

We o ff er a surprisingly straightforward solution to 'fix' the established lifting framework as depicted in Fig. 1 (b): retrieving the lost intermediate visual representations learned by 2D pose detectors and engaging them in the lifting process. As previous works limit their research scope to the design of lifting models (Stage 2), the cooperation of both stages is largely under-explored. This approach yields two major benefits: First, such representations encode joint-centric spatial context that promotes reducing ambiguity - the core di ffi culty in 3D human pose estimation. Second, we illustrate in this work that these representations can be used out of the box, i.e. , no finetuning on the 3D task or extra training techniques are needed.

To showcase the e ff ectiveness of our solution, we design a simple transformer-based baseline named Context-Aware PoseFormer . The proposed method leverages multi-resolution feature maps produced by 2D detectors in a sparse manner. Specifically, we extract informative contextual features from these feature maps using deformable operations [12, 75] where the detected 2D joints serve as reference points. This helps mitigate the noise brought by imperfect pose detectors while avoiding heavy computation. Furthermore, we design a pose-context interaction module to exchange information between the extracted multi-level contextual features and the 2D joint embedding that encodes positional clues. Finally, a spatial transformer module is adopted to model spatial dependencies between human joints, which follows PoseFormer [74]. Our approach shows encouragingly strong

performance (see Fig. 2): our single-frame model outperforms 351-frame MHFormer [29] and other state-of-the-art lifting-based methods, highlighting the potential of leveraging contextual information in improving pose estimation accuracy.

Our contributions are summarized as follows: 1) We propose a novel framework that addresses the time-intensive issue present in existing methods by incorporating context awareness. This is achieved by leveraging readily available visual representations learned by 2D pose detectors.

We introduce a simple baseline that e ffi -ciently extracts informative context features from learned representations and subsequently fuses them with 2D joint embedding that provides positional clues.

On two benchmarks (Human3.6M [21] and MPI-INF-3DHP [39]), our single-frame model demonstrates significant performance improvements over both non-temporal and temporal methods that use up to hundreds of video frames. 4) Our work opens up opportunities for more skeleton-based methods to utilize visual repre-

1 frame

1 frame

1 frame

27 frames

81 frames

243 frames

351 frames

81 frames

243 frames

Ours

13.2

%

reduction

49

Ours

GraphMLP[28]arXiv'22

48

MHFormer[29]CVPR'22

P-STMO[50]ECCV'22

47

Zengetal.[65]IcCV'21

MPJPE(mm)

46

45

44

43

42

0

50

100

150

200

250

300

350

NumberofFrames

sentations learned by well-trained backbones from upstream tasks in an out-of-the-box manner.

2 Related Work

Early works [54, 17, 44, 53] estimate the 3D human pose from monocular images without explicitly using the corresponding 2D pose as an intermediate representation. With the rapid progress in 2D pose estimation [42, 8, 52], lifting 2D poses to 3D has been dominant in the literature. As our work aims at improving the lifting-based framework, we primarily introduce works of this line. We refer readers to the recent HPE survey [73] for a more comprehensive background and detailed information.

Single-frame methods. Before the surge of deep learning, Jiang [22] performs 3D pose estimation based on nearest neighbors with a large 3D human pose library, and this approach is recently revisited by Chen et al. [5] with a closed-form algorithm to project 3D pose exemplars to 2D. Martinez et al. [38] design a simple linear-layer model to illustrate that the di ffi culty of 3D human pose estimation mainly lies in precise 2D joint localization. Moreno-Noguer [41] formulates the task as 2D-to-3D distance matrix regression. To mitigate the demand for 3D annotations, Pavlakos et al. [43] leverage ordinal depths of human joints as an extra training signal. Given the human skeleton topology, recent works ( e.g. , GraphSH [60], HCSF [65], GraFormer [70], GraphMLP [28]) use graph neural networks to reason spatial constraints of human joints, and lift them to 3D.

Multi-frame methods. As initial attempts, some works explore temporal cues for robust pose estimation in crowded scenes [1] or temporally consistent results [40]. More recently, a number of works model spatial-temporal dependencies for 2D pose sequences, e.g. , using LSTM [24, 59], CNN [45, 6] and GNN [56, 65]. They have achieved superior performance than non-temporal methods. Driven by the great success of vision transformers in image classification [13, 33], object detection [4, 75] and etc , transformer-based methods [74, 29, 67, 50, 69, 72] are currently the center of research interest. PoseFormer [74] is the first transformer-based model in the community, which builds up inter-joint correlations within each video frame and human dynamics across frames with transformer encoders, respectively. Due to its straightforward architecture, PoseFormer is highly malleable and rapidly gains a series of follow-ups [67, 69]. Our work presents a simple single-frame baseline model where the inter-joint modeling module is adapted from the spatial encoder of PoseFormer.

Fusing image features with positional joint clues. Compared to previous lifting-based works, our approach actively engages visual representations from 2D pose detectors in the lifting process, thus being context-aware and achieving promising results. Notably, the fusion of image features with positional information about human joints has been explored in some works but within di ff ering contexts or with distinct motivations. Tekin et al. [55] propose to fuse image features with 2D joint heatmaps, where 2D joints are not explicitly regressed as in the lifting-based pipeline. Closer to

As Ref. Points

Reuse

Intermediate

Feature Maps

Stage 1

Output 2D Pose

"

!

!

!

"

!

#

!

Backbone

(e.g., HRNet)

"

!

"

"

"

#

Pose-Context

Feature Fusion

Spatial Inter-joint

Modeling

Regression Head

×

N

2

×

N

3

Add Positional Embeddings (PE)

"#$

"#%

"#&

"#'

#

!

#

"

#

#

$

& Coordinate

Embedding

Target 3D Pose

$

Deformable

Context

Extraction

×

N

1

Stage 2

Context Features

our work, Zhao et al. [68] extend their graph network (SemGCN) by incorporating image features. Specifically, image features are pooled at the detected joints and then concatenated together with joint coordinates, serving as the input to SemGCN. Though this approach is similar to ours at first glance, our work di ff ers in that 1) our method needs no multi-stage training, 2) we consider the uncertainty associated with 2D joints in joint-context extraction, 3) we have designed dedicated modules to handle the domain gap between image features and joint coordinates, and 4) our performance surpasses that of SemGCN by a significant margin. In addition to technical details and performance comparisons, our underlying motivation for this research also di ff ers. More discussion about relevant literature can be found in Appendix A.

3 Method

We propose a new framework to solve the time-intensive problem of previous lifting-based methods. This is achieved by incorporating the spatial context (encoded by intermediate feature maps from 2D pose detectors) into the 2D-to-3D lifting process (see also Fig. 1 (b)). We design a simple baseline model named Context-Aware PoseFormer to exemplify the gains brought by this approach (an overview is in Fig. 3). In the following, we first briefly introduce the context-agnostic counterpart of our method, PoseFormer [74]. Then, we provide detailed descriptions of our approach.

3.1 A Context-agnostic Counterpart: PoseFormer

PoseFormer [74] is one of the first transformer-based models for 3D human pose estimation, which mainly consists of two modules: the spatial encoder and the temporal encoder. PoseFormer processes an input 2D pose sequence as follows. First, the sequence s ∈ R F × J × 2 is projected to a latent space indicated by S ∈ R F × J × C , where F , J and C denote the sequence length, joint number, and embedding dimension respectively. Learnable (zero-initialized) spatial positional embedding E spa ∈ R 1 × J × C is summed with S to encode joint-specific information. Second, the spatial encoder models inter-joint dependencies at each video frame (time step) with cascaded transformers. Third, the output of the spatial encoder Z ∈ R F × ( J × C ) is flattened as the input to the temporal encoder. Similarly, temporal positional embedding E temp ∈ R F × ( J × C ) is added to Z to encode the information related to frame indices. The temporal encoder builds up frame-level correlations with transformer layers in this stage. Finally, a 1D convolution layer is adopted to gather temporal information, and a linear layer outputs the target 3D pose y ∈ R 1 × ( J × 3) for the central video frame. PoseFormer and its follow-ups [67, 50, 69] are inherently limited to deal with ambiguities since they barely receive 2D joint coordinates as input. Due to the simplicity, scalability, and e ff ectiveness of PoseFormer, we choose PoseFormer as the

basis of our method, boosting it to be context-aware and free of temporal information. Detailed descriptions of our method are presented below.

3.2 Context-Aware PoseFormer

Our core insight is retrieving the spatial context encoded by readily available feature maps from 2D human pose detectors. As the multi-stage design promotes 2D human pose estimation [52, 11, 63], our method also benefits from multi-resolution feature maps. An illustration of model input is on the left of Fig. 3. For a given RGB image I of size H × W × 3, an o ff -the-shelf 2D pose detector produces the corresponding 2D human pose p ∈ R J × 2 , and a set of intermediate feature maps with varying resolutions as byproducts, { H l ∈ R Hl × Wl × Cl } L l = 1 (where L is the total number of feature maps). Since stacked network layers and down-sampling operations increase the receptive field (and correspondingly the scale of context) of each unit on feature maps [35], high-resolution feature maps encode fine-grained visual cues ( e.g. , joint existence) while low-resolution ones tend to keep high-level semantics ( e.g. , human skeleton structures). During the process that 2D pose detectors are trained to localize human joints, multi-level feature maps implicitly encode information about the spatial configurations of joints.

How to e ff ectively utilize these feature maps is a non-trivial question. Blindly processing them in a global way using, e.g. , convolutions [20], or vision transformers [13], may bring unnecessary computational overhead since the background that takes up a large proportion of the image does not provide task-relevant features related to human joints. The most straightforward approach to treat feature maps in a preferably sparse manner is to sample feature vectors at the detected joints. However, in real scenarios, 2D pose detectors inevitably introduce errors. Therefore, the context feature obtained by such an approach may not well represent the spatial context associated with the joint of our interest, leading to suboptimal performance. We propose a Deformable Context Extraction module to deal with this issue, which not only attends to the detected joints but also the regions around them. As the detected 2D joints are not the only source of contextual features, the errors brought by 2D pose detectors can be mitigated. Moreover, we design a Pose-Context Feature Fusion module to fuse the features of two modalities: spatial contextual features and 2D joint embedding that encodes positional clues about human joints. Finally, Spatial Inter-joint Modeling is adopted to build up dependencies across human parts. They are organized hierarchically (on the right, Fig. 3). The e ff ectiveness of each element is verified in Sec. 4.3.

Deformable Context Extraction. This module uses deformable attention [75] to extract informative spatial contextual cues from feature maps. Specifically, for each detected joint, we produce a set of sampling points on multi-scale feature maps whose o ff sets and weights are learned based on the features of reference points ( i.e. , the detected joint of interest), denoted by { F l } L l = 1 . We make learned o ff sets and weights 'reference-position-aware' by adding the embedding of 2D joint coordinates P to source features (after they are projected to a shared dimension C ). Let l index a feature level and j index a human joint, and Deformable Context Extraction is formulated as:

where m iterates over the attention heads, k over the sampled points around the detected joint p j , and K is the total sampling point number. For the l th feature map, ∆ p lmk represents the sampling o ff set of the k th sampling point in the m th attention head, while Almk denotes its corresponding attention weight. We provide visualization of sampling points learned by deformable attention in Sec. 4.3, where the learned sampling points attempt to discover ground truth 2D joints despite the noise brought by pose detectors. More details and visualization about this module are available in Appendix E.

Pose-Context Feature Fusion. This module aims at exchanging information between two modalities ( i.e. , pose embedding and the extracted spatial context) and between di ff erent levels of context features at the same time. Cross-modality learning gives rise to powerful vision-language models [23, 2], while the multi-scale design has proved beneficial to a large set of computer vision tasks

[15, 16, 71]. Inspired by multi-modality models [2] that feed visual tokens and language tokens into a unified transformer encoder, we naturally adapt this design to model interactions between pose embedding and multi-level context features, where transformers learn a joint representation for both modalities [51, 7]. The implementation of this module is:

where j (1 . . . J ) indicates that feature fusion is performed for each joint. Transformer layers reduce domain gaps for both modalities in shared latent space [25] and promote message-passing where joint-position information and multi-level contextual cues complement each other. In addition, we introduce Unified Positional Embedding E uni ∈ R ( L + 1) × J × C ( L for joint-context feature vectors and 1 for 2D pose embedding) to encode modality-related and joint-specific information simultaneously.

Spatial Inter-joint Modeling. With the two modules above, elegant local representations are learned for each joint individually. To understand the human skeleton system and its corresponding spatial context with a global view, inter-joint dependencies are modeled based on the learned per-joint features using a spatial transformer encoder following PoseFormer [74]. The spatial encoder in PoseFormer simply receives the linear projection of (context-agnostic) 2D joint coordinates as input, whereas our joint-level representations are enhanced with spatial contextual cues (thus being contextaware). The output of Pose-Context Feature Fusion module { X N 2 j } J j = 1 is flattened and stacked as input to the N 3-layer transformer encoder. Each joint token ( J in total) is of ( L + 1) × C dimensions, encoding both positional and contextual information for the related joint.

Output and loss function. A simple linear layer is adopted to obtain the final 3D pose y ∈ R J × 3 . We use L 2 loss as PoseFormer [74] to supervise the estimated result.

Discussion. Our model follows a local-to-global hierarchy. Specifically, local representations are first learned for each joint, on which inter-joint modeling is subsequently performed. This design provides our model with preferable interpretability and scalability: modules are inherently disentangled according to the dependencies they attempt to learn and can be easily extended or replaced. Plus, the local-to-global organization avoids the heavy computation brought by globally modeling all elements at once. These virtues make our method a desirable stepping stone for future research.

4 Experiments

Our method, Context-Aware PoseFormer, is referred to as 'CA-PFbackbone name ' in tables. We conduct experiments on two benchmarks (Human3.6M [21] and MPI-INF-3DHP [39]). Human3.6M [21] is the most popular benchmark for 3D human pose estimation, where more than 3.6 million video frames are captured indoors from 4 di ff erent camera views. MPI-INF-3DHP [39] includes videos collected from indoor scenes and challenging outdoor environments. Both datasets provide subjects performing various actions with multiple cameras. On Human3.6M, we report MPJPE (Mean Per Joint Position Error, where the Euclidean distance is computed between estimation and ground truth) and PA-MPJPE (the aligned MPJPE). For MPI-INF-3DHP, we report MPJPE, Percentage of Correct Keypoint (PCK) within the 150mm range, and Area Under Curve (AUC). Settings follow previous works [74, 29, 67, 50]. Due to page limitations, we place details about our implementations in Appendix B.

4.1 Comparison on Human3.6M

Accuracy . We compare our single-frame method with both single-frame and multi-frame methods on Human3.6M [21] (Table 1). Previous methods receive CPN-detected [8] 2D joints as input. We also follow this setting to ensure a fair comparison. Benefiting from spatial contextual clues encoded by the CPN pose detector, our approach outperforms state-of-the-art single-frame methods by a huge margin. For instance, Zeng et al. [65] obtain 47.9mm MPJPE, whereas our CA-PF-CPN achieves 41.6mm (a 13.2 %error reduction). This result has already surpassed the performance of most modern multi-frame methods. To show the flexibility of our method and to further investigate spatial context

from various backbones, we change the backbones that generate feature maps: our CA-PF-HRNet-32 [52] achieves 41.4mm MPJPE; with a larger backbone HRNet-48, the error reduces to 39.8mm, which is superior to all previous multi-frame methods. For example, this result outperforms MixSTE [67] with 243 frames (39.8 v.s. 40.9mm, a 2.7% error reduction) and MHFormer [29] with even 351 frames (39.8 v.s. 43.0mm, a 7.4 %error reduction). Our method presents highly competitive accuracy in the absence of dense temporal modeling. Moreover, our method consistently gains improvements from advanced 2D pose detectors. We expect our approach's performance to increase further with powerful large visual foundation models, as such foundation models have been attracting increasing attention and interest.

Computational cost . Our method requires far less computation than state-of-the-art methods (see also Table 1). For example, with similar accuracy, MHFormer [29] takes 14.2 GFLOPs, whereas our CA-PF-CPN [8] demands only 0.6 GFLOPs (about 24 × reduction). Two reasons account for the lightweight property of the proposed approach: (1) Non-temporal input removes large computation for dense temporal modeling. Modeling temporal dependencies using e.g. , transformers [74, 67] is computationally heavy especially when the frame number is increased for improved accuracy. (2) We utilize multi-scale feature maps (from pose detectors) in a sparse manner ( i.e. , with deformable attention [75]). We avoid applying global operations on the whole feature maps, which may introduce unnecessary computational budgets on uninformative backgrounds.

4.2 Comparison on MPI-INF-3DHP

In Table 2, the comparison is also conducted with both single-frame (top) and multi-frame (middle) methods on the MPI-INF-3DHP dataset [39]. We use ground truth 2D pose detection as input, following [74, 29, 50], and HRNet-32, HRNet-48 [52] are adopted as the backbone network to generate multi-resolution feature maps. Our approach outperforms other methods, including the stateof-the-art multi-frame one, P-STMO [50]. P-STMO uses 81 video frames as input, with additional masked joint self-supervised pre-training. In con-

trast, our approach has no access to temporal information and does not need an extra pre-training stage. The results verify the generalization ability of our method to di ff erent datasets, particularly in challenging outdoor environments.

4.3 Ablation Study

Gains from spatial context and ablations on model components. In this section, we first investigate the gains purely brought by spatial contextual clues from 2D pose detectors and then the e ff ectiveness of each component in our method (please refer to Sec. 3.2 for more details). Specifically, we show how the context-agnostic counterpart of our method, (single-frame) PoseFormer [74] is converted to Context-Aware PoseFormer step by step (step-wise improvements are shown in Table 3).

Step 0, Context-Agnostic: For PoseFormer, 2D joint coordinates detected by a pose detector are linearly embedded to dimension C as per-joint representation ( i.e. , joint embedding), and Inter-joint Modeling is subsequently performed to build up correlations across the joint embeddings. Plain 2D coordinates encode no spatial context for joints that helps reduce ambiguity, PoseFormer (and other lifting-based methods) is thus referred to as 'context-agnostic'. This serves as the starting point of our method.

Step 2, Context-Aware with Pose-Context Feature Fusion: We promote cross-modality feature fusion by applying transformers to joint embeddings and the sampled multi-scale context features before concatenation, which brings another 1.6% error reduction.

Step 1, Context-Aware but simple concatenation: The first step to make PoseFormer 'contextaware' is to incorporate joint-context features into its per-joint representation. We retrieve the multi-scale feature maps produced by the 2D pose detector, simply sample feature vectors on the detected joint locations from each feature map, and project them to dimension C . For each joint, the sampled context feature vectors that encode joint-centric visual cues are concatenated together with the coordinate embedding that encodes positional information. Even naive involvement (as simple as concatenation) of spatial context clues brings a huge improvement as shown in Table 3: a 15.0 %error reduction (from 51.2 to 43.5mm), which demonstrates that leveraging the readily available visual representations learned by upstream backbones is e ff ective and promising .

Step 3, Final version of our Context-Aware PoseFormer: The context features in step 1 are obtained by directly sampling at the detected joints, which leads to suboptimal performance since 2D pose detectors inevitably introduce noise. Thus, the sampled feature vectors can not well represent the context associated with the joint of interest. We introduce Deformable Context Extraction to mitigate this issue, which adaptively attends to the detected joint along with its surrounding regions using learnable sampling points, which further reduces the error by 3.3%.

We also o ff er visualization of learned sampling points (brighter colors denote larger attention weights) in Fig. 4, where they try to approach the ground truth despite the noisy detected input. Note that in training we do not supervise them using ground truth 2D pose. More visualization is in Appendix E.

What kind of spatial context are we looking for? In this part, we go a step further to explore what context is desirable to improve 3D pose estimation accuracy. Our method benefits from multi-resolution feature maps produced by 2D detectors [52, 11, 63], which encode spatial contextual clues of various levels for human joints. For example, HRNet [52] outputs four feature maps in the last stage, which are 4 × , 8 × , 16 × , 32 × downsampled compared to the input image size, respectively. We investigate which one of the four is most informative for our task via ablation experiments (in Table 4, MPJPE is reported on Human3.6M). Removing 4 × and 16 × downsampled feature

Blue: detected joints, Green: ground

truth, Gray: learned sampling points.

maps brings the most significant performance drop (a 3.6% and 4.1% error increase for each). In contrast, only a 1.0% error increase is observed when the 32 × downsampled feature map is eliminated.

Generally, high-resolution feature maps contribute more to 3D pose estimation than low-resolution ones. The reason is that low-resolution feature maps tend to keep the high-level semantics of joints, such as join type, which may not be relevant to our task - to localize human joints in 3D, where 2D-to-3D joint correspondence is pre-defined. However, we can still gain from low-resolution feature maps as they encode most wide-

4

×

8

×

16

×

32

×

↓

✓

✓

✓

✓

41.4

✗

✓

✓

✓

42.9

∆

1

.

5

✓

✗

✓

✓

42.4

∆

1

.

0

✓

✓

✗

✓

43.1

∆

1

.

7

✓

✓

✓

✗

41.8

∆

0

.

4

range spatial dependencies, promoting an understanding of the overall human skeleton structure.

What pre-trained features are more desirable? ImageNet [48] pre-trained backbones ( e.g. , ResNet [20]) profit a series of downstream tasks [30, 10, 58], yet this seems not applicable to 3D human pose estimation. In Table 5, we replace COCO pretrained backbones in our method with ImageNet classification pre-trained ones, showing a remarkable performance drop. This should be attributed to the large gap between the pre-training task (im-

age classification) and the downstream task (3D human pose estimation). Exploring more suitable pre-trained features that can transfer their e ff ectiveness into the 3D HPE task is a promising direction. We place more ablations on backbones and pre-training datasets in Appendix D.

4.4 Visualization

0

100

200

300

400

500

20

40

60

80

100

120

140

160

MPJPE(mm)

MHFormer(T=351) CVPR'22

Context-agnostic Counterpart(T=1)

Context-Aware PoseFormer(T=1)

Frame Index

Spatial context promotes temporal stability. Weshow a frame-wise comparison on the Human3.6M [21] test set in Fig. 5. Our single-frame approach outperforms competitive MHFormer [29] using 351 frames. Moreover, our context-agnostic counterpart (Sec. 4.3) su ff ers from temporal instability ( i.e. , zigzags and error peaks), whereas our method presents a more stable trend. We also provide a qualitative comparison with both methods in Fig. 6. We include more visual comparison on both Human3.6M and MPI-INF-3DHP in Appendix F.

Spatial context improves the robustness against occlusions . We provide a comparison with PoseFormer [74] on in-the-wild videos in Fig. 7. We choose two sets of consecutive video frames where the 2D joint detection fails due to strong self-occlusion. Since PoseFormer only accepts 2D joints as input, its estimation is sensitive to the noise of input 2D poses. On the contrary, our method leverages spatial context from images in addition to 2D joint coordinates to localize joints in 3D. Thus, our method shows more reasonable and robust results despite noisy or even outlier 2D joints.

Input

MHFormer

Ours

Context-agnostic

Ground Truth

Input

MHFormer

Ours

Context-agnostic

Ground Truth

Input

PoseFormer

Ours

Input

PoseFormer

Ours

This is a hat,

not an arm!

The right arm

is missed by the

pose detector.

5 Conclusion and Discussion

This work presents a new framework that leverages readily available visual representations learned by o ff -the-shelf 2D pose detectors. Our approach removes the need for extensive long-term temporal modeling excessively employed by existing 3D human pose estimation methods. We further propose a straightforward baseline to showcase its e ff ectiveness: our single-frame model significantly outperforms other single-frame and even multi-frame methods using up to 351 frames while being far more e ffi cient. Our work opens up opportunities for future research to utilize visual representations from well-trained upstream backbones in an out-of-the-box manner.

Limitations. We observed in Sec. 4.4 that incorporating the spatial context of joints improves temporal stability, enhancing consistency and smoothness in the estimated results, even without access to explicit temporal clues. However, we acknowledge for all single-frame methods, including ours, mitigating jitters remains a challenge compared to multi-frame methods that leverage temporal clues. This is primarily due to the non-temporal nature of single-frame methods. A potential solution is to extend our method to model short-term temporal dependencies, which should not introduce unacceptably high costs. We include some preliminary results in Appendix C. Compared to standard lifting-based methods, the memory cost of our method can be higher as we additionally process image features in the lifting stage. Leveraging the visual context of joints in a more lightweight way is our future research direction.

References

Mykhaylo Andriluka, Stefan Roth, and Bernt Schiele. Monocular 3d pose estimation and tracking by detection. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition , pages 623-630. Ieee, 2010. 3

Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. Advances in Neural Information Processing Systems , 35:3289732912, 2022. 5, 6

Irving Biederman, Robert J Mezzanotte, and Jan C Rabinowitz. Scene perception: Detecting and judging objects undergoing relational violations. Cognitive psychology , 14(2):143-177, 1982. 2

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision , pages 213-229. Springer, 2020. 3

Ching-Hang Chen and Deva Ramanan. 3d human pose estimation = 2d pose estimation + matching. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7035-7043, 2017. 3

Tianlang Chen, Chen Fang, Xiaohui Shen, Yiheng Zhu, Zhili Chen, and Jiebo Luo. Anatomyaware 3d human pose estimation with bone-based pose decomposition. IEEE Transactions on Circuits and Systems for Video Technology , 32(1):198-209, 2021. 3

Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX , pages 104-120. Springer, 2020. 6

Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. Cascaded pyramid network for multi-person pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7103-7112, 2018. 1, 3, 6, 7

Yuxin Chen, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, and Weiming Hu. Channel-wise topology refinement graph convolution for skeleton-based action recognition. In Proceedings of the IEEE / CVF International Conference on Computer Vision , pages 13359-13368, 2021. 1

Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems , 34:1786417875, 2021. 9

Bowen Cheng, Bin Xiao, Jingdong Wang, Honghui Shi, Thomas S Huang, and Lei Zhang. Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition , pages 5386-5395, 2020. 5, 8

Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision , pages 764-773, 2017. 2

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR , 2021. 3, 5, 18

Moritz Einfalt, Katja Ludwig, and Rainer Lienhart. Uplift and upsample: E ffi cient 3d human pose estimation with uplifting transformers. In Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision (WACV) , January 2023. 7

Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE / CVF International Conference on Computer Vision , pages 6824-6835, 2021. 6

Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le. Nas-fpn: Learning scalable feature pyramid architecture for object detection. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition , pages 7036-7045, 2019. 6

Agne Grinciunaite, Amogh Gudi, Emrah Tasli, and Marten Den Uyl. Human pose estimation in space and time using 3d cnn. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III , pages 32-39. Springer, 2016. 3

Ikhsanul Habibie, Weipeng Xu, Dushyant Mehta, Gerard Pons-Moll, and Christian Theobalt. In the wild human pose estimation using explicit 2d features and intermediate 3d representations. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition , pages 10905-10914, 2019. 16

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition , pages 16000-16009, 2022. 18

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016. 5, 9

C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE TPAMI , 2014. 3, 6, 9, 16, 17

Hao Jiang. 3d human pose reconstruction using millions of exemplars. In 2010 20th International Conference on Pattern Recognition , pages 1674-1677. IEEE, 2010. 3

Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning , pages 5583-5594. PMLR, 2021. 5

Kyoungoh Lee, Inwoong Lee, and Sanghoon Lee. Propagating lstm: 3d pose estimation based on joint interdependency. In Proceedings of the European conference on computer vision (ECCV) , pages 119-135, 2018. 3

Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 , 2019. 6

Shichao Li, Lei Ke, Kevin Pratama, Yu-Wing Tai, Chi-Keung Tang, and Kwang-Ting Cheng. Cascaded deep monocular 3d human pose estimation with evolutionary training data. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020. 7

Wenhao Li, Hong Liu, Runwei Ding, Mengyuan Liu, Pichao Wang, and Wenming Yang. Exploiting temporal contexts with strided transformer for 3d human pose estimation. IEEE Transactions on Multimedia , 2022. 7

Wenhao Li, Hong Liu, Tianyu Guo, Hao Tang, and Runwei Ding. Graphmlp: A graph mlp-like architecture for 3d human pose estimation. arXiv preprint arXiv:2206.06420 , 2022. 3, 7

Wenhao Li, Hong Liu, Hao Tang, Pichao Wang, and Luc Van Gool. Mhformer: Multi-hypothesis transformer for 3d human pose estimation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 13147-13156, June 2022. 1, 2, 3, 6, 7, 9, 19

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2117-2125, 2017. 9

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014. 1, 16

Ruixu Liu, Ju Shen, He Wang, Chen Chen, Sen-ching Cheung, and Vijayan Asari. Attention mechanism exploits temporal contexts: Real-time 3d human pose reconstruction. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition , pages 5064-5073, 2020. 7

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE / CVF International Conference on Computer Vision , pages 10012-10022, 2021. 3

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017. 16

Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the e ff ective receptive field in deep convolutional neural networks. Advances in neural information processing systems , 29, 2016. 5

Xiaoxuan Ma, Jiajun Su, Chunyu Wang, Hai Ci, and Yizhou Wang. Context modeling in 3d human pose estimation: A unified perspective. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition , pages 6238-6247, 2021. 2

Wei Mao, Richard I Hartley, Mathieu Salzmann, et al. Contact-aware human motion forecasting. Advances in Neural Information Processing Systems , 35:7356-7367, 2022. 1

Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little. A simple yet e ff ective baseline for 3d human pose estimation. In Proceedings of the IEEE international conference on computer vision , pages 2640-2649, 2017. 3

Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation in the wild using improved cnn supervision. In 2017 international conference on 3D vision (3DV) , pages 506-516. IEEE, 2017. 3, 6, 7, 16

Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei, Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. Vnect: Real-time 3d human pose estimation with a single rgb camera. Acm transactions on graphics (tog) , 36(4):1-14, 2017. 3

Francesc Moreno-Noguer. 3d human pose estimation from a single image via distance matrix regression. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2823-2832, 2017. 3

Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In European conference on computer vision , pages 483-499. Springer, 2016. 3

Georgios Pavlakos, Xiaowei Zhou, and Kostas Daniilidis. Ordinal depth supervision for 3d human pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7307-7316, 2018. 3

Georgios Pavlakos, Xiaowei Zhou, Konstantinos G Derpanis, and Kostas Daniilidis. Coarseto-fine volumetric prediction for single-image 3d human pose. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7025-7034, 2017. 3

Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 3d human pose estimation in video with temporal convolutions and semi-supervised training. In CVPR , 2019. 1, 2, 3, 7

Zhongwei Qiu, Qiansheng Yang, Jian Wang, and Dongmei Fu. Ivt: An end-to-end instanceguided video transformer for 3d pose estimation. In Proceedings of the 30th ACM International Conference on Multimedia , pages 6174-6182, 2022. 16

Jathushan Rajasegaran, Georgios Pavlakos, Angjoo Kanazawa, and Jitendra Malik. Tracking people by predicting 3d appearance, location and pose. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition , pages 2740-2749, 2022. 1

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015. 9

Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proceedings of the IEEE / CVF international conference on computer vision , pages 2304-2314, 2019. 16

Wenkang Shan, Zhenhua Liu, Xinfeng Zhang, Shanshe Wang, Siwei Ma, and Wen Gao. Pstmo: Pre-trained spatial temporal many-to-one model for 3d human pose estimation. In

Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part V , pages 461-478. Springer, 2022. 1, 3, 4, 6, 7, 19, 20

Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE / CVF international conference on computer vision , pages 7464-7473, 2019. 6

Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition , pages 5693-5703, 2019. 1, 3, 5, 7, 8

Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In Proceedings of the European conference on computer vision (ECCV) , pages 529-545, 2018. 3

Bugra Tekin, Isinsu Katircioglu, Mathieu Salzmann, Vincent Lepetit, and Pascal Fua. Structured prediction of 3d human pose with deep neural networks. In Proceedings of the British Machine Vision Conference (BMVC) , pages 130-1, 2016. 3

Bugra Tekin, Pablo Márquez-Neila, Mathieu Salzmann, and Pascal Fua. Learning to fuse 2d and 3d image cues for monocular body pose estimation. In Proceedings of the IEEE international conference on computer vision , pages 3941-3950, 2017. 3

Jingbo Wang, Sijie Yan, Yuanjun Xiong, and Dahua Lin. Motion guided 3d pose estimation from videos. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16 , pages 764-780. Springer, 2020. 3, 7

Tom Wehrbein, Marco Rudolph, Bodo Rosenhahn, and Bastian Wandt. Probabilistic monocular 3d human pose estimation with normalizing flows. In Proceedings of the IEEE / CVFinternational conference on computer vision , pages 11199-11208, 2021. 2

Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimation and tracking. In Proceedings of the European conference on computer vision (ECCV) , pages 466-481, 2018. 9

Jingwei Xu, Zhenbo Yu, Bingbing Ni, Jiancheng Yang, Xiaokang Yang, and Wenjun Zhang. Deep kinematics analysis for monocular 3d human pose estimation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020. 3

Tianhan Xu and Wataru Takano. Graph stacked hourglass networks for 3d human pose estimation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16105-16114, June 2021. 3, 7

Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. ViTPose: Simple vision transformer baselines for human pose estimation. In Advances in Neural Information Processing Systems , 2022. 18

Binyi Yin, Dongbo Zhang, Shuai Li, Aimin Hao, and Hong Qin. Context-aware network for 3d human pose estimation from monocular rgb image. In 2019 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2019. 16

Yuhui Yuan, Rao Fu, Lang Huang, Weihong Lin, Chao Zhang, Xilin Chen, and Jingdong Wang. Hrformer: High-resolution vision transformer for dense predict. Advances in Neural Information Processing Systems , 34:7281-7293, 2021. 5, 8

Ailing Zeng, Xiao Sun, Fuyang Huang, Minhao Liu, Qiang Xu, and Stephen Lin. Srnet: Improving generalization in 3d human pose estimation with a split-and-recombine approach. In ECCV , 2020. 7

Ailing Zeng, Xiao Sun, Lei Yang, Nanxuan Zhao, Minhao Liu, and Qiang Xu. Learning skeletal graph neural networks for hard 3d pose estimation. In Proceedings of the IEEE / CVF International Conference on Computer Vision , pages 11436-11445, 2021. 3, 6, 7

Hongwen Zhang, Yating Tian, Xinchi Zhou, Wanli Ouyang, Yebin Liu, Limin Wang, and Zhenan Sun. Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. In Proceedings of the IEEE / CVF International Conference on Computer Vision , pages 11446-11456, 2021. 16

Jinlu Zhang, Zhigang Tu, Jianyu Yang, Yujin Chen, and Junsong Yuan. Mixste: Seq2seq mixed spatio-temporal encoder for 3d human pose estimation in video. In Proceedings of the

IEEE / CVF Conference on Computer Vision and Pattern Recognition , pages 13232-13242, 2022. 1, 3, 4, 6, 7, 16

Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N. Metaxas. Semantic graph convolutional networks for 3d human pose regression. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3425-3435, 2019. 4

Qitao Zhao, Ce Zheng, Mengyuan Liu, Pichao Wang, and Chen Chen. Poseformerv2: Exploring frequency domain for e ffi cient and robust 3d human pose estimation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition , June 2023. 3, 4

Weixi Zhao, Weiqiang Wang, and Yunjie Tian. Graformer: Graph-oriented transformer for 3d pose estimation. In 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 20406-20415, 2022. 3, 7

Ce Zheng, Xianpeng Liu, Guo-Jun Qi, and Chen Chen. Potter: Pooling attention transformer for e ffi cient human mesh recovery. In IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 6

Ce Zheng, Matias Mendieta, Taojiannan Yang, Guo-Jun Qi, and Chen Chen. Feater: An e ffi cient network for human reconstruction via feature map-based transformer. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition , pages 13945-13954, 2023. 3

Ce Zheng, Wenhan Wu, Chen Chen, Taojiannan Yang, Sijie Zhu, Ju Shen, Nasser Kehtarnavaz, and Mubarak Shah. Deep learning-based human pose estimation: A survey. ACM Computing Surveys , 56(1):1-37, 2023. 3

Ce Zheng, Sijie Zhu, Matias Mendieta, Taojiannan Yang, Chen Chen, and Zhengming Ding. 3d human pose estimation with spatial and temporal transformers. In Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV) , pages 11656-11665, October 2021. 1, 2, 3, 4, 6, 7, 8, 9, 10, 16, 17

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 , 2020. 2, 3, 5, 7, 18

Appendix

A Additional Discussion on Related Work

Involving 2D joint heatmaps in image-based 3D human pose estimation has also been explored in some works [62, 18]. Yin et al. [62] split the 3D human pose estimation into two sub-tasks: 2D pose estimation and depth regression. Habibie et al. [18] disentangle features for the 2D pose from global image features to exploit large-scale 2D pose datasets. These approaches do not explicitly learn features from 2D poses and lift them to 3D.

Keypoint-based feature extraction. IVT [46] formulates video-based multi-person 3D pose estimation as an end-to-end framework. The features of each instance in the video are gathered by learning joint o ff sets from the body center. Compared to IVT, our work follows the standard two-stage pipeline of 3D HPE, and the estimated 2D joints serve as reference points in feature extraction. Pixel-aligned features [49, 66] in the context of 3D human reconstruction use point-wise image features similar to our approach. Their feature extraction is based on the re-projection of estimated 3D meshes [66] or query 3D points [49] instead of 2D human joints from o ff -the-shelf 2D pose detectors. Pixel-aligned features primarily aim at reducing the misalignment between estimated 3D representations and input 2D images. In contrast, we propose joint-context feature extraction to reduce ambiguities in lifting-based HPE and alleviate current methods' reliance on temporal information.

B Implementation Details

2D pose detector settings. Our method's overall pipeline includes an o ff -the-self 2D pose detector and a lifting model. For the first stage, the 2D pose detector is pre-trained on the COCO [31] dataset, without finetuning on the 2D poses from 3D pose estimation datasets, i.e. , Human3.6M [21] and MPI-INF-3DHP [39]. We use 256 × 192 resolution for input images, and we found higher image resolution improves performance. For 2D-to-3D lifting, the weights of pre-trained 2D detectors are frozen, i.e. , no finetuning on the 3D task is needed. This approach makes our method preferably flexible - our method is compatible with a wide range of o ff -the-shelf (pre-trained) 2D pose detectors. We show in Sec. 4 that our method gains consistent improvements by increasing the capability of 2D pose detectors. In the future, we may leverage more advanced 2D pose detectors to further improve the performance upper bound.

Lifting model settings. Our lifting model includes three basic modules: Deformable Context Extraction , Pose-Context Feature Fusion and Spatial Inter-joint Modeling . The layer number of each module is set to 4, following PoseFormer [74]. The hidden dimension (a shared projection dimension C ) of the model is 128. We use 8 heads in self-attention for transformer layers.

Training details. The experiments are conducted on a single NVIDIA RTX 3090 GPU. Our lifting model is trained using the AdamW [34] optimizer for 50 epochs with a batch size of 512. The initial learning rate is 6.4e-3 with an exponential learning rate decay schedule, and the decay factor is 0.99.

C A Simple Temporal Extension of Our Method

This section shows that our single-frame method can naturally extend to model temporal dependencies (an overview is in Fig. 8). As illustrated in Sec. 3.2, the Spatial Inter-joint Modeling module outputs a feature vector of dimension ( L + 1) × C for each joint, where L is the number of multi-scale feature maps and C is a shared projection dimension of the model. We use a Temporal Transformer to model temporal correlations of each joint independently. Specifically, for Temporal Transformer , the input token number is the total frame number, and each token is of dimension ( L + 1) × C . Using transformers to build up temporal dependencies is straightforward, and this approach has been adopted by PoseFormer [74], MixSTE [67], etc . The output of the temporal transformer encoder can be denoted by Z temp ∈ R J × F × [( L + 1) · C ] , where J is the joint number and F is the frame number. Following PoseFormer, we use 1D convolution to reduce its temporal dimension (gather temporal information) and a linear layer to obtain the final estimated 3D pose y ∈ R J × 3 .

Quantitative results . Due to limited computational resources, we prepare contextual feature vectors (that are sampled at detected joints from multi-level feature maps) in advance and remove the

Regression Head

Target 3D Pose

Pose-Context

Feature Fusion

Spatial Inter-joint

Modeling

×

N

2

×

N

3

𝐏𝐄𝟎

𝐏𝐄𝟏

𝐏𝐄𝟐

𝐏𝐄𝟑

𝐹

!

!

𝑷

𝟏

𝐹

#

!

𝐹

$

!

First frame

Pose-Context

Feature Fusion

Spatial Inter-joint

Modeling

×

N

2

×

N

3

𝐏𝐄𝟒

𝐏𝐄𝟓

𝐏𝐄𝟔

𝐏𝐄𝟕

𝐹

!

#

𝑷

𝟐

𝐹

#

#

𝐹

$

#

Second frame

Pose-Context

Feature Fusion

Spatial Inter-joint

Modeling

×

N

2

×

N

3

𝐏𝐄𝟖

𝐏𝐄𝟗

𝐏𝐄𝟏𝟎

𝐏𝐄𝟏𝟏

𝐹

!

$

𝑷

𝟑

𝐹

#

$

𝐹

$

$

Third frame

Temporal Transformer

Table 6: We compare the (short-term) temporal extension of our small model (CA-PF-S) with PoseFormer. MPJPE: Mean Per Joint Position Error, the precision metric. MPJVE: Velocity Error, the temporal smoothness metric. The results are reported on Human3.6M (in millimeters).

Deformable Context Extraction module. Therefore, the input to our model is discrete feature vectors instead of image sequences. This approach largely reduces the memory cost. We also reduce the shared projection dimension C in our model to speed up training. This small variant of our model is referred to as 'CA-PF-S', which has fewer FLOPs compared to our full model 'CA-PF' presented in Sec. 4. To show the gains in precision and temporal smoothness from temporal modeling, we report two metrics, MPJPE (Position Error, the precision metric) and MPJVE (Velocity Error, the temporal smoothness metric), on Human3.6M [21]. We compare it with PoseFormer [74] and the results are in Table 6: First, increasing the number of input frames brings consistent improvements in both precision and temporal smoothness. For example, by using only 3 video frames, the MPJVE of our method decreases from 8.5 to 4.8mm (a 43.5 % reduction), and the MPJPE is reduced by 1.1%. This indicates that even short-term temporal modeling largely mitigates jitters in estimated results and further improves precision upper bound. Second, considering the same number of input frames, our CA-PF-S consistently outperforms PoseFormer in terms of both MPJPE and MPJVE. Moreover, our 3-frame CA-PF-S has already achieved superior MPJPE to 81-frame PoseFormer, and our 9-frame CA-PF-S achieves comparable MPJVE with 81-frame PoseFormer. The results verify the two benefits of spatial contextual clues from 2D pose detectors - accuracy and temporal stability.

Based on the results above, we expect that our method can be successfully extended to model even longer-term temporal dependencies ( e.g. , 81 video frames) to further boost precision and temporal smoothness. However, modeling such long input sequences with acceptable computational costs is non-trivial and is out of the scope of this paper, which is left as our future direction.

D More Ablations on Backbones and Pre-training Datasets

We additionally conduct experiments on ViTPose [61], the recent state of the art on 2D HPE. The weights of ViTs [13] in ViTPose are initialized with MAE [19] pre-training, and then the model is further trained on 2D HPE datasets. The results are in Table 7, and observations are as follows.

Backbone design outweighs the results on 2D HPE. The first three rows in Table 7 demonstrate that better results on 2D HPE do not necessarily translate to better performance on 3D HPE: Although ViTPose performs best on COCO 2D HPE, it achieves worst results on Human3.6M. We attribute this result to the lack of multi-scale network design. ViTPose gains from powerful MAE pre-training with modern transformer architecture, while its network design is arguably simplified for 2D HPE. Specifically, ViTPose processes on tokenized image patches with transformers and finally increases the resolution of feature maps using 2D deconvolution layers. They use no multi-scale designs, e.g. , high-resolution feature branches or multi-scale feature fusion, as in CPN and HRNet. However, such techniques may help our approach learn more task-relevant information ( i.e. , joint-context features) to localize joints in 3D.

More pre-training datasets on 2D HPE improve the performance on the 3D Task. A comparison between row 3 and row 4 in Table 7 shows that multi-dataset pre-training improves performance on both 2D HPE and 3D HPE. Plus, the gains on 3D HPE (5.8% error reduction) are even more significant than those on 2D HPE (1.3 points improvement on AP). We hypothesize that multi-dataset pre-training improves the generalization ability of learned backbone features. Therefore, the best performance on 2D HPE of ViTPose better transfers to the 3D task.

E More Details and Visualization for Deformable Context Extraction

More details. The Deformable Context Extraction module extracts informative joint-centric context features from feature maps using deformable attention [75]. The sampling points of each attention head are initialized in di ff erent directions (w.r.t. the reference joint) to promote learning diverse

contextual clues from images. To prevent overly aggressive updates on sampling o ff sets ( e.g. , they may go outside the image), the learning rate of linear layers that generate sampling o ff sets is set to 6.4e-4 ( 1 / 10 of that for other layers in the lifting model). We use 4 heads for deformable attention.

Visualization of learned sampling points on consecutive video frames. In Fig. 9, a subject in the Human3.6M test set raises his right arm where severe self-occlusion occurs, and the 2D pose detector fails to localize the right wrist (blue dots are detected results, and green dots are ground truth). We find that most sampling points (gray dots) are concentrated on the upper body of the subject. More interestingly, despite the unreliable 2D joint detection (reference points), some learned sampling points attempt to approach the ground truth. We indicate the sampling points that gain larger attention weights with higher brightness (we decrease the brightness of images for better visual e ff ects). Note that we do not train sampling points using ground truth. This indicates that our adaptive context extraction strategy can learn informative contextual features based on the visual cues of images despite bad sampling references ( i.e. , false joint detection), which helps reduce uncertainty brought by imperfect 2D pose detectors.

F More Visualization

We provide more qualitative results on Human3.6M (Fig. 10) and MPI-INF-3DHP (Fig. 11). Our single-frame method obtains reliable estimation in hard cases, e.g. , self-occlusion, and rare poses, compared to state-of-the-art multi-frame methods such as 351-frame MHFormer [29] and 81-frame P-STMO [50].

Input

MHFormer

Ours

Context-agnostic

Ground Truth

Input

MHFormer

Ours

Context-agnostic

Ground Truth

Input

P-STMO

Ours

Ground Truth

Input

P-STMO

Ours

Ground Truth

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a context-aware method that combines the 2D image feature and the detected 2D key points for 3D human pose estimation. The joint-centric spatial context represented by the intermediate image feature is able to reduce the amguity in 3D lifting. Under this motivation, a novel framework that consists of a defrmable context extraction module, a pose-context feature fusion module and a spatial inter-joint modeling module is proposed. Experiment validates the effectiveness of the proposed modules in reducing calculation time and achieving better performance. On two standard benchmarks, the proposed method shows its effectiveness and stability without using any temporal information compared to other single-frame methods and multi-frame methods.\n\nSTRENGTHS:\n1. This paper is well presented and organized. The idea is clear and the method is somewhat novel.\n2. New SOTA results are achieved with a significante improvement even compared to most multi-frame methods.\n3. Experiments are well organized and the ablation study is insightful to support the claim of contributions.\n\nWEAKNESSES:\n1. Leveraging 2D intermediate feature is not a new idea. Some previous works also tried this route, such as [a][b][c]. Compared to other skeleton-based methods, either single-frame-based or multi-frame-based, utilizting 2D image feature will lead to more memory consumption. Such a kind of limitation needs to be considered.\n\n[a] Yin, Binyi, et al. \"Context-Aware Network for 3D Human Pose Estimation from Monocular RGB Image.\" 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 2019.\n\n[b] Habibie, Ikhsanul, et al. \"In the wild human pose estimation using explicit 2d features and intermediate 3d representations.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.\n\n[c] Sun, Xiao, et al. \"Integral human pose regression.\" Proceedings of the European conference on computer vision (ECCV). 2018.\n\nQUESTIONS:\n1. According to the study in Sec. 3 in the supplementary material, temporal information brings less improvement in the proposed method. Why the result when using 81 frames as PoseFormer is not compared? Could the authors give more explaination about this?\n\nLIMITATIONS:\nThe authors have discussed their limitation of motion jitter without temporal information in the supplementary material. But the limitation of memory consumption should also be discussed as mentioned in the weakness.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Context-Aware PoseFormer, the key idea is to extract multi-scale spatial features for 3D pose lifting from 2D pose.\nThe authors claim that the single-frame beats hundreds of frames and demonstrate that their single-frame methods achieve comparable or better results than multi-frame methods, such as poseformer.\n\nSTRENGTHS:\nThe motivation of this article is clear, and the writing is simple and easy to understand.\nThe experiments in this paper verify the effect of the proposed method, verify that the spatial context information is very important, and the effect of the proposed context aware poseformer.\nThe ablation study in this article is detailed and verifies the effect of each module in the method.\n\nWEAKNESSES:\nI am skeptical of the claim in this paper that the single frame is better than the multiple frame method. This article only verifies that his single-frame method is better than some multi-frame methods such as poseformer, but there are 3 problems below：\n1) The comparison of the methods in table 1 is unfair. For example, other methods use the 2D pose result of CPN as the output of the lifting network, while the method in this paper uses HRNet. It is well known that HRNet can perform better 2D detection than CPN.\n2) In addition, the method in this paper also inputs more visual features from images, so this paper should also be compared with other methods that use visual features, such as IVT.\n3) If the single-frame method in this paper works well under the same 2D network, it cannot be verified that it is definitely better than the multi-frame method. In the method of this article, is it possible to achieve better results by adding timing information? This is the consensus in the field, and this paper should also verify this.\n\nIn addition, the core contribution of the method in this paper is the multi-scale visual features of the image. The author uses a deformable method to extract multi-scale features to improve the performance of the network.\nIn the field of 3D posture, it is a consensus that extracting better spatial context features can improve performance.\nIn terms of context information extraction, this article has no additional contribution, only using other people's methods.\nTherefore, in general, the method in this paper is a bit incremental in terms of innovation, and the contribution is not enough to be accepted by NeurIPS。\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper targets the challenging 3D pose estimation problem based on a new context-aware lifting algorithm. The proposed approach is simple to implement and reproduce. Attractive experimental results have been reported on the challenging benchmarks. Also, the detailed ablations well validate the design of the proposed algorithm.

STRENGTHS:
1. The idea of lifting 2d image features for 3D human pose estimation is interesing. 
2. The proposed algorithm report state-of-art performance on the 3d pose estimation benchmark. 
3. It also provides sufficient ablation studies to validate the algorithm design of the proposed approach.

WEAKNESSES:
1. In stage 2, it involves a Deformable context extraction module to extract the context features (F1, F2, F3, P). How is the feature difference against H1-H3? Also, is it possible to add H1-H3 with positional embeding to the pose-context feature fusion module?

2. It would be reasonable that 2D pose information can be helpful for the 3d pose estimation. But lifting from 2d to 3d is not-trivial. How could the proposed approach guarantee the consistency with the 3d location of the pose? 

3. If there are more temporal frames available, is it possible to further boost the performance of the proposed approach?

QUESTIONS:
Please address the questions in the weakness section.

LIMITATIONS:
The paper does not discuss the potential limitations of the proposed algorithm.  I would suggest the authors to have a discussion of the limitations (like the cases with high self-occlusion which temporal information may be helpful).
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper leverages the readily available intermediate visual representations for 3d human pose estimation. The method discards temporal information to solve the time-intensive issue of existing lifting-based methods. The authors design a simple pipline, named Context-Aware PoseFormer, to extract informative context features and fuse these features to learn more positional clues.

After reading authors' rebuttal, my previous concerns were all addressed.

STRENGTHS:
1.The paper is generally well written. The problem to address and the shortcomings of the existing approaches are discussed well. 

2. The motivation and observation are clear and meaningful, and the network is well designed to solve these problems.

3.The experiments demonstrate well the superiority of the proposed method over the prior art. The authors also provide adequate ablation studies.

WEAKNESSES:
More STOA methods in 2023 should be chosen for comparison.

QUESTIONS:
1. Does the resolution of input images influence the performance?

2. It would be interesting to apply the proposed method to other related tasks like hand pose estimation to verify its generalization capability.

LIMITATIONS:
The authors should describe the limitation of the proposed method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Context-Aware PoseFormer, a new approach that leverages intermediate visual representations from pre-trained 2D pose detectors to implicitly encode spatial context for 3D human pose estimation. Despite the simple network structure, the proposed method outperforms existing state-of-the-art single- and multi-frame methods by a large margin without relying on temporal information.\n\nSTRENGTHS:\n-This paper provides an interesting insight that leverages the spatial contextual information of pre-trained 2D pose detectors, referred to as \"context-aware\" information, to improve the accuracy of 3D human pose estimation. \n\n- The proposed approach provides considerable benefits by elevating the accuracy of single-frame methods to a level on par with state-of-the-art multi-frame methods\n\nWEAKNESSES:\n-Incremental novelty in the idea of context aware features and lack of discussion on the existing context aware features.\n   - Modeling context-aware operations or networks has been studied in many other computer vision topics. For example, pixel-aligned features [1, 2] have been extensively used in 3D human/hand reconstruction from single/multi images. This pixel-aligned feature is demonstrated to be the most useful module in Table 3 while other modules only improve the performance marginally.\n\n- Incomplete evaluation regarding the pretrained context-aware feature network.\n    - For the pretraining, what's the impact of different pre-training datasets (e.g. number of training data) on the estimation? \n    -  For the pretraining backbone, what's the impact of latest backbone on the final results?  e.g. ViT-based [3] pre-trained backbones MAE [4].\n\nReference\n- [1] Pymaf: 3d human pose and shape regression with pyramidal mesh alignment feedback loop. ICCV 2021\n- [2] Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. ICCV2019\n- [3] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale ICLR\n- [4] Masked autoencoders are scalable vision learners. CVPR 2022\n\nQUESTIONS:\nPlease see the weakness\n\nLIMITATIONS:\nPlease see the weakness", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}