SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qitao Zhao $^{1*}$  Ce Zheng $^{2}$  Mengyuan Liu $^{3}$  Chen Chen $^{2}$
$^{2}$ Center for Research in Computer Vision, University of Central Florida
<sup>3</sup>Key Laboratory of Machine Perception, Peking University, Shenzhen Graduate School
The dominant paradigm in 3D human pose estimation that lifts a 2D pose sequence to 3D heavily relies on long-term temporal clues (i.e., using a daunting number of video frames) for improved accuracy, which incurs performance saturation, intractable computation and the non-causal problem. This can be attributed to their inherent inability to perceive spatial context as plain 2D joint coordinates carry no visual cues. To address this issue, we propose a straightforward yet powerful solution: leveraging the readily available intermediate visual representations produced by off-the-shelf (pre-trained) 2D pose detectors – no finetuning on the 3D task is even needed. The key observation is that, while the pose detector learns to localize 2D joints, such representations (e.g., feature maps) implicitly encode the joint-centric spatial context thanks to the regional operations in backbone networks. We design a simple baseline named Context-Aware PoseFormer to showcase its effectiveness. Without access to any temporal information, the proposed method significantly outperforms its context-agnostic counterpart, PoseFormer [74], and other state-of-the-art methods using up to hundreds of video frames regarding both speed and precision. Project page: qitaozhao.github.io/ContextAware-PoseFormer
3D human pose estimation (HPE) aims to localize human joints in 3D, which has a wide range of applications, including motion prediction [37], action recognition [9], and tracking [47]. Recently, with the large availability of 2D human pose detectors [8, 52], lifting 2D pose sequences to 3D (referred to as lifting-based methods) has been the de facto paradigm in the literature. Compared to raw RGB images, 2D human poses (as an intermediate representation) have two essential advantages. On the one hand, domain gaps exist between images (input) and 3D joint locations (output), whereas this is not the case for 2D and 3D joints. Primarily, 2D joint coordinates provide highly task-relevant positional information for localizing joints in 3D. On the other, 2D coordinate representation is exceptionally lightweight in terms of memory cost. For example, in a typical COCO [31] setting, a 2D human pose requires only  $17 \times 2$  floating points, whereas an RGB image requires  $256 \times 192 \times 3$  (or even more with higher resolution). This property enables state-of-the-art lifting-based methods to leverage extremely long-term temporal clues for advanced accuracy, e.g., 243 video frames for VideoPose3D [45], MixSTE [67], and P-STMO [50]; large as 351 frames for MHFormer [29].
Scaling up the input has brought consistent improvements so far, yet some concerns naturally arise. First, the performance gains seem to level off when excessively long sequences are used as input (e.g., improvements are marginal when the frame number increases from 243 to 351 [29]). Second, long input sequences bring non-trivial computational costs. For instance, temporal processing using transformers (a popular choice) is expensive, especially with large frame numbers. Third, the use of
*Work was done while Qitao was an intern mentored by Chen Chen.
(a)
Figure 1: A comparison of existing lifting-based methods with ours at a framework level. (a) Existing methods (video-based): time-intensive and context-agnostic. (b) Ours (image-based): time-free and context-aware. We leverage intermediate visual representations from Stage 1. Notably, we do not finetune the 2D detectors for the lifting task, thus easing training and bringing no extra costs.
future video frames renders real-world online applications impossible. Concretely, existing works adopt a 2D pose sequence to estimate the 3D human pose for the central frame where half of the frames in the sequence are inaccessible for the current time step, which is known as the non-causal problem [45]. While gains from long input sequences come at a cost, is there any alternative approach we can take to easily boost performance? We attempt to answer this question in our work.
We start by revisiting the whole pipeline of lifting-based methods, which involves two stages as shown in Fig. 1 (a). In Stage 1, a 2D pose detector estimates human joints for each image from a video clip, with a set of intermediate representations as byproducts, e.g., feature maps of varying resolutions. In Stage 2, the detected 2D pose sequence (output of Stage 1) is lifted to 3D, while such representations are discarded. This can be problematic: the (multi-scale) joint-centric spatial context encoded by these feature maps is lost. We argue: the omission of spatial contextual information is the primary factor contributing to the time-intensive nature of existing lifting-based methods.
So, what makes spatial context important? As an inverse problem, 3D pose estimation inherently suffers from ambiguities [29, 36, 57] such as depth ambiguity and self-occlusion, yet they can be mitigated by utilizing spatial context from the monocular image. For depth ambiguity, the shading difference is an indicator of depth disparity [57] whereas the occluded body parts can be inferred from visible ones with human skeleton constraints [36, 57]. Psychological studies [3, 36] provide evidence that appropriate context helps reduce ambiguities and promote visual recognition for the human perception system. Since 2D keypoints alone are unable to encode such spatial context, existing lifting-based approaches, in turn, resort to long-term temporal clues to alleviate ambiguities.
We offer a surprisingly straightforward solution to "fix" the established lifting framework as depicted in Fig. 1 (b): retrieving the lost intermediate visual representations learned by 2D pose detectors and engaging them in the lifting process. As previous works limit their research scope to the design of lifting models (Stage 2), the cooperation of both stages is largely under-explored. This approach yields two major benefits: First, such representations encode joint-centric spatial context that promotes reducing ambiguity – the core difficulty in 3D human pose estimation. Second, we illustrate in this work that these representations can be used out of the box, i.e., no finetuning on the 3D task or extra training techniques are needed.
To showcase the effectiveness of our solution, we design a simple transformer-based baseline named Context-Aware PoseFormer. The proposed method leverages multi-resolution feature maps produced by 2D detectors in a sparse manner. Specifically, we extract informative contextual features from these feature maps using deformable operations [12, 75] where the detected 2D joints serve as reference points. This helps mitigate the noise brought by imperfect pose detectors while avoiding heavy computation. Furthermore, we design a pose-context interaction module to exchange information between the extracted multi-level contextual features and the 2D joint embedding that encodes positional clues. Finally, a spatial transformer module is adopted to model spatial dependencies between human joints, which follows PoseFormer [74]. Our approach shows encouragingly strong
performance (see Fig. 2): our single-frame model outperforms 351-frame MHFormer [29] and other state-of-the-art lifting-based methods, highlighting the potential of leveraging contextual information in improving pose estimation accuracy.
Our contributions are summarized as follows:
1) We propose a novel framework that addresses the time-intensive issue present in existing methods by incorporating context awareness. This is achieved by leveraging readily available visual representations learned by 2D pose detectors.
2) We introduce a simple baseline that efficiently extracts informative context features from learned representations and subsequently fuses them with 2D joint embedding that provides positional clues.
3) On two benchmarks (Human3.6M [21] and MPI-INF-3DHP [39]), our single-frame model demonstrates significant performance improvements over both non-temporal and temporal methods that use up to hundreds of video frames. 4) Our work opens up opportunities for more skeleton-based methods to utilize visual repre
Figure 2: Our single-frame method outperforms both non-temporal and temporal methods that use up to 351 frames on Human3.6M.
sentations learned by well-trained backbones from upstream tasks in an out-of-the-box manner.
Early works [54, 17, 44, 53] estimate the 3D human pose from monocular images without explicitly using the corresponding 2D pose as an intermediate representation. With the rapid progress in 2D pose estimation [42, 8, 52], lifting 2D poses to 3D has been dominant in the literature. As our work aims at improving the lifting-based framework, we primarily introduce works of this line. We refer readers to the recent HPE survey [73] for a more comprehensive background and detailed information.
Single-frame methods. Before the surge of deep learning, Jiang [22] performs 3D pose estimation based on nearest neighbors with a large 3D human pose library, and this approach is recently revisited by Chen et al. [5] with a closed-form algorithm to project 3D pose exemplars to 2D. Martinez et al. [38] design a simple linear-layer model to illustrate that the difficulty of 3D human pose estimation mainly lies in precise 2D joint localization. Moreno-Noguer [41] formulates the task as 2D-to-3D distance matrix regression. To mitigate the demand for 3D annotations, Pavlakos et al. [43] leverage ordinal depths of human joints as an extra training signal. Given the human skeleton topology, recent works (e.g., GraphSH [60], HCSF [65], GraFormer [70], GraphMLP [28]) use graph neural networks to reason spatial constraints of human joints, and lift them to 3D.
Multi-frame methods. As initial attempts, some works explore temporal cues for robust pose estimation in crowded scenes [1] or temporally consistent results [40]. More recently, a number of works model spatial-temporal dependencies for 2D pose sequences, e.g., using LSTM [24, 59], CNN [45, 6] and GNN [56, 65]. They have achieved superior performance than non-temporal methods. Driven by the great success of vision transformers in image classification [13, 33], object detection [4, 75] and etc, transformer-based methods [74, 29, 67, 50, 69, 72] are currently the center of research interest. PoseFormer [74] is the first transformer-based model in the community, which builds up inter-joint correlations within each video frame and human dynamics across frames with transformer encoders, respectively. Due to its straightforward architecture, PoseFormer is highly malleable and rapidly gains a series of follow-ups [67, 69]. Our work presents a simple single-frame baseline model where the inter-joint modeling module is adapted from the spatial encoder of PoseFormer.
Fusing image features with positional joint clues. Compared to previous lifting-based works, our approach actively engages visual representations from 2D pose detectors in the lifting process, thus being context-aware and achieving promising results. Notably, the fusion of image features with positional information about human joints has been explored in some works but within differing contexts or with distinct motivations. Tekin et al. [55] propose to fuse image features with 2D joint heatmaps, where 2D joints are not explicitly regressed as in the lifting-based pipeline. Closer to
Figure 3: An overview of Context-Aware PoseFormer. Stage 1 (left): The 2D pose detector estimates the 2D pose, with a set of feature maps as byproducts. In Stage 2 (right), we extract informative join-context features from such feature maps and subsequently fuse them with 2D pose features.
our work, Zhao et al. [68] extend their graph network (SemGCN) by incorporating image features. Specifically, image features are pooled at the detected joints and then concatenated together with joint coordinates, serving as the input to SemGCN. Though this approach is similar to ours at first glance, our work differs in that 1) our method needs no multi-stage training, 2) we consider the uncertainty associated with 2D joints in joint-context extraction, 3) we have designed dedicated modules to handle the domain gap between image features and joint coordinates, and 4) our performance surpasses that of SemGCN by a significant margin. In addition to technical details and performance comparisons, our underlying motivation for this research also differs. More discussion about relevant literature can be found in Appendix A.
We propose a new framework to solve the time-intensive problem of previous lifting-based methods. This is achieved by incorporating the spatial context (encoded by intermediate feature maps from 2D pose detectors) into the 2D-to-3D lifting process (see also Fig. 1 (b)). We design a simple baseline model named Context-Aware PoseFormer to exemplify the gains brought by this approach (an overview is in Fig. 3). In the following, we first briefly introduce the context-agnostic counterpart of our method, PoseFormer [74]. Then, we provide detailed descriptions of our approach.
3.1 A Context-agnostic Counterpart: PoseFormer
PoseFormer [74] is one of the first transformer-based models for 3D human pose estimation, which mainly consists of two modules: the spatial encoder and the temporal encoder. PoseFormer processes an input 2D pose sequence as follows. First, the sequence  $s \in \mathbb{R}^{F \times J \times 2}$  is projected to a latent space indicated by  $S \in \mathbb{R}^{F \times J \times C}$ , where  $F$ ,  $J$  and  $C$  denote the sequence length, joint number, and embedding dimension respectively. Learnable (zero-initialized) spatial positional embedding  $E_{spa} \in \mathbb{R}^{1 \times J \times C}$  is summed with  $S$  to encode joint-specific information. Second, the spatial encoder models inter-joint dependencies at each video frame (time step) with cascaded transformers. Third, the output of the spatial encoder  $Z \in \mathbb{R}^{F \times (J \times C)}$  is flattened as the input to the temporal encoder. Similarly, temporal positional embedding  $E_{temp} \in \mathbb{R}^{F \times (J \times C)}$  is added to  $Z$  to encode the information related to frame indices. The temporal encoder builds up frame-level correlations with transformer layers in this stage. Finally, a 1D convolution layer is adopted to gather temporal information, and a linear layer outputs the target 3D pose  $y \in \mathbb{R}^{1 \times (J \times 3)}$  for the central video frame. PoseFormer and its follow-ups [67, 50, 69] are inherently limited to deal with ambiguities since they barely receive 2D joint coordinates as input. Due to the simplicity, scalability, and effectiveness of PoseFormer, we choose PoseFormer as the
basis of our method, boosting it to be context-aware and free of temporal information. Detailed descriptions of our method are presented below.
3.2 Context-Aware PoseFormer
Our core insight is retrieving the spatial context encoded by readily available feature maps from 2D human pose detectors. As the multi-stage design promotes 2D human pose estimation [52, 11, 63], our method also benefits from multi-resolution feature maps. An illustration of model input is on the left of Fig. 3. For a given RGB image  $\pmb{I}$  of size  $H \times W \times 3$ , an off-the-shelf 2D pose detector produces the corresponding 2D human pose  $\pmb{p} \in \mathbb{R}^{J \times 2}$ , and a set of intermediate feature maps with varying resolutions as byproducts,  $\{\pmb{H}_l \in \mathbb{R}^{H_l \times W_l \times C_l}\}_{l=1}^L$  (where  $L$  is the total number of feature maps). Since stacked network layers and down-sampling operations increase the receptive field (and correspondingly the scale of context) of each unit on feature maps [35], high-resolution feature maps encode fine-grained visual cues (e.g., joint existence) while low-resolution ones tend to keep high-level semantics (e.g., human skeleton structures). During the process that 2D pose detectors are trained to localize human joints, multi-level feature maps implicitly encode information about the spatial configurations of joints.
How to effectively utilize these feature maps is a non-trivial question. Blindly processing them in a global way using, e.g., convolutions [20], or vision transformers [13], may bring unnecessary computational overhead since the background that takes up a large proportion of the image does not provide task-relevant features related to human joints. The most straightforward approach to treat feature maps in a preferably sparse manner is to sample feature vectors at the detected joints. However, in real scenarios, 2D pose detectors inevitably introduce errors. Therefore, the context feature obtained by such an approach may not well represent the spatial context associated with the joint of our interest, leading to suboptimal performance. We propose a Deformable Context Extraction module to deal with this issue, which not only attends to the detected joints but also the regions around them. As the detected 2D joints are not the only source of contextual features, the errors brought by 2D pose detectors can be mitigated. Moreover, we design a Pose-Context Feature Fusion module to fuse the features of two modalities: spatial contextual features and 2D joint embedding that encodes positional clues about human joints. Finally, Spatial Inter-joint Modeling is adopted to build up dependencies across human parts. They are organized hierarchically (on the right, Fig. 3). The effectiveness of each element is verified in Sec. 4.3.
Deformable Context Extraction. This module uses deformable attention [75] to extract informative spatial contextual cues from feature maps. Specifically, for each detected joint, we produce a set of sampling points on multi-scale feature maps whose offsets and weights are learned based on the features of reference points (i.e., the detected joint of interest), denoted by  $\{F_l\}_{l=1}^L$ . We make learned offsets and weights "reference-position-aware" by adding the embedding of 2D joint coordinates  $P$  to source features (after they are projected to a shared dimension  $C$ ). Let  $l$  index a feature level and  $j$  index a human joint, and Deformable Context Extraction is formulated as:
$$
\boldsymbol {F} _ {l j} ^ {\prime n} = \operatorname {D e f o r m A t t n} (\boldsymbol {F} _ {l j} ^ {n - 1} + \boldsymbol {P} _ {j}) + \boldsymbol {F} _ {l j} ^ {n - 1}, \quad \boldsymbol {F} _ {l j} ^ {n - 1}, \boldsymbol {P} _ {j} \in \mathbb {R} ^ {C}, n = 1 \dots N _ {1}
$$
$$
\boldsymbol {F} _ {l j} ^ {n} = \operatorname {M L P} \left(\boldsymbol {F} _ {l j} ^ {\prime n}\right) + \boldsymbol {F} _ {l j} ^ {\prime n}, \quad \boldsymbol {F} _ {l j} ^ {n}, \boldsymbol {F} _ {l j} ^ {\prime n} \in \mathbb {R} ^ {C}, n = 1 \dots N _ {1} \tag {1}
$$
$$
\operatorname {D e f o r m A t t n} \left(\boldsymbol {F} _ {l j} ^ {n - 1} + \boldsymbol {P} _ {j}\right) = \sum_ {m = 1} ^ {M} \left[ \sum_ {k = 1} ^ {K} A _ {l m k} \cdot \boldsymbol {W} _ {l m} \boldsymbol {H} _ {l} \left(\boldsymbol {p} _ {j} + \Delta \boldsymbol {p} _ {l m k}\right) \right], \quad \boldsymbol {H} _ {l} \left(\boldsymbol {p} _ {j} + \Delta \boldsymbol {p} _ {l m k}\right) \in \mathbb {R} ^ {C _ {l}}
$$
where  $m$  iterates over the attention heads,  $k$  over the sampled points around the detected joint  $\pmb{p}_j$ , and  $K$  is the total sampling point number. For the  $l^{\mathrm{th}}$  feature map,  $\Delta p_{lmk}$  represents the sampling offset of the  $k^{\mathrm{th}}$  sampling point in the  $m^{\mathrm{th}}$  attention head, while  $A_{lmk}$  denotes its corresponding attention weight. We provide visualization of sampling points learned by deformable attention in Sec. 4.3, where the learned sampling points attempt to discover ground truth 2D joints despite the noise brought by pose detectors. More details and visualization about this module are available in Appendix E.
Pose-Context Feature Fusion. This module aims at exchanging information between two modalities (i.e., pose embedding and the extracted spatial context) and between different levels of context features at the same time. Cross-modality learning gives rise to powerful vision-language models [23, 2], while the multi-scale design has proved beneficial to a large set of computer vision tasks
[15, 16, 71]. Inspired by multi-modality models [2] that feed visual tokens and language tokens into a unified transformer encoder, we naturally adapt this design to model interactions between pose embedding and multi-level context features, where transformers learn a joint representation for both modalities [51, 7]. The implementation of this module is:
$$
\boldsymbol {X} _ {j} ^ {0} = \operatorname {S t a c k} \left(\left[ \boldsymbol {P} _ {j}, \boldsymbol {F} _ {1 j} ^ {N _ {1}}, \dots , \boldsymbol {F} _ {L j} ^ {N _ {1}} \right], d i m = 0\right), \quad \boldsymbol {P} _ {j} \in \mathbb {R} ^ {C}, \boldsymbol {F} _ {1 j} ^ {N _ {1}}, \dots , \boldsymbol {F} _ {L j} ^ {N _ {1}} \in \mathbb {R} ^ {C}
$$
$$
\boldsymbol {X} _ {j} ^ {\prime n} = \operatorname {S e l f A t t n} \left(\boldsymbol {X} _ {j} ^ {n - 1}\right) + \boldsymbol {X} _ {j} ^ {n - 1}, \quad \boldsymbol {X} _ {j} ^ {n - 1} \in \mathbb {R} ^ {(L + 1) \times C}, n = 1 \dots N _ {2} \tag {2}
$$
$$
\boldsymbol {X} _ {j} ^ {n} = \operatorname {M L P} \left(\boldsymbol {X} _ {j} ^ {\prime n}\right) + \boldsymbol {X} _ {j} ^ {\prime n}, \quad \boldsymbol {X} _ {j} ^ {\prime n} \in \mathbb {R} ^ {(L + 1) \times C}, n = 1 \dots N _ {2}
$$
where  $j(1 \dots J)$  indicates that feature fusion is performed for each joint. Transformer layers reduce domain gaps for both modalities in shared latent space [25] and promote message-passing where joint-position information and multi-level contextual cues complement each other. In addition, we introduce Unified Positional Embedding  $E_{uni} \in \mathbb{R}^{(L + 1) \times J \times C}$  ( $L$  for joint-context feature vectors and 1 for 2D pose embedding) to encode modality-related and joint-specific information simultaneously.
Spatial Inter-joint Modeling. With the two modules above, elegant local representations are learned for each joint individually. To understand the human skeleton system and its corresponding spatial context with a global view, inter-joint dependencies are modeled based on the learned per-joint features using a spatial transformer encoder following PoseFormer [74]. The spatial encoder in PoseFormer simply receives the linear projection of (context-agnostic) 2D joint coordinates as input, whereas our joint-level representations are enhanced with spatial contextual cues (thus being context-aware). The output of Pose-Context Feature Fusion module  $\{X_j^{N_2}\}_{j=1}^J$  is flattened and stacked as input to the  $N_3$ -layer transformer encoder. Each joint token ( $J$  in total) is of  $(L+1) \times C$  dimensions, encoding both positional and contextual information for the related joint.
Output and loss function. A simple linear layer is adopted to obtain the final 3D pose  $\mathbf{y} \in \mathbb{R}^{J \times 3}$ . We use  $L_{2}$  loss as PoseFormer [74] to supervise the estimated result.
Discussion. Our model follows a local-to-global hierarchy. Specifically, local representations are first learned for each joint, on which inter-joint modeling is subsequently performed. This design provides our model with preferable interpretability and scalability: modules are inherently disentangled according to the dependencies they attempt to learn and can be easily extended or replaced. Plus, the local-to-global organization avoids the heavy computation brought by globally modeling all elements at once. These virtues make our method a desirable stepping stone for future research.