ExtPose: Robust and Coherent Pose Estimation by Extending ViTs
Authors: Rongyu Chen
Li'an Zhuo
Linlin Yang
Qi WANG
Liefeng Bo
Bang Zhang
Angela Yao
Venue: ICML 2025 poster (2025)
Keywords: 3D Pose Estimation
Transformer
Primary area: applications->computer_vision

Abstract: Vision Transformers (ViT) are remarkable at 3D pose estimation, yet they still encounter certain challenges. One issue is that the popular ViT architecture for pose estimation is limited to images and lacks temporal information. Another challenge is that the prediction often fails to maintain pixel alignment with the original images. To address these issues, we propose a systematic framework for 3D pose estimation, called ExtPose. ExtPose extends image ViT to the challenging scenario and video setting by taking in additional 2D pose evidence and capturing temporal information in a full attention-based manner. We use 2D human skeleton images to integrate structured 2D pose information. By sharing parameters and attending across modalities and frames, we enhance the consistency between 3D poses and 2D videos without introducing additional parameters. We achieve state-of-the-art (SOTA) performance on multiple human and hand pose estimation benchmarks with substantial improvements to 34.0mm (-23%) on 3DPW and 4.9mm (-18%) on FreiHAND in PA-MPJPE over the other ViT-based methods respectively.

EXTPOSE
Robust and Coherent Pose Estimation by Extending ViTs
Rongyu Chen *1
Lian Zhuo 22 Linlin Yang Qi Wang² Liefeng Bo Bang Zhang? Angela Yao
https://gloryyrolg hatheolllgggtttttte github io/ extpoee
Abstract
Vision Transformers (ViT) are remarkable at 3D pose estimation, yet they still encounter certain challenges. One issue is that the popular VIT ar- chitecture for pose estimation is limited to images and lacks temporal information. Another chal- lenge is that the prediction often fails to main- tain pixel alignment with the original images. To address these issues, propose a system- atic framework for 3D pose estimation, called EXTPOSE. ExTPoSE extends image ViT to the challenging scenario and video setting by taking in additional 2D pose evidence and captur- ing temporal information manner. We use 2D human skeleton images integrate structured 2D pose information. By shar- ing parameters and attending across modalities and frames, we enhance the consistency between 3D poses and 2D videos without introducing ad- ditional parameters. We achieve state-of-the-ane (SOTA) performance on multiple human and hand pose estimation benchmarks with substantial im- provements 34.0mm (-23%) on 3DPW and 4.9mm (-18%) on FreiHAND in PA-MPJPE over the other ViT-based methods respectively.
fatiross
hand
VirPaGE whole body
Badly
todyy
Figure 1: EXTPOSE extends and aids SOTA ViT-based works in generalization with auxiliary 2D detection (1 column). Our method (3nd column) excels in robust and pixel-aligned mesh reconstruction even in extremely compli- cated parkour while ViT-based works (2nd column) predict a completely flipped hand and body.
full iirentionbbaede
human-object interaction, AI generation, etc. With the ex- tensive use of vision transformers (ViT) (Dosovitskiy et al., 2021), there has been growing trend towards leveraging ViT architectures for 2D HPE! (Xu al., 2022) and 3D HPE (Goel et al., 2023; Pavlakos et al., 2024). These meth- ods boast remarkable results after training large-scale datasets, but still face some challenges with robust general- ization and temporal coherence. Specifically, they struggle to align with 2D image evidence; orthogonally, when en- countering data in more common video formats, they cannot incorporate temporal information, instead merely guessing jittering poses frame by frame under depth and occlusion ambiguity.
"You can enjoy grander sight, By climbing to greater height."
Tang Poems
1. Introduction
First, predictions are often inconsistent with input im- ages, especially under challenging settings such as occlu- sions (Chen et al., 2023), motion blur (Oh et al., 2023; Chen et al., 2025b), poor lighting, and other imaging degrada- tions (Dwivedi al., 2024a). For instance, in the middle column of Fig. 1, the body estimate is oriented wrongly in the camera. In the case of hand pose estimation, the issue becomes more severe, as the hand tends to occupy only
Human and hand pose estimation a foundation for higher- level applications roboties, action recognition, animation,
Work done during internship at Alibaba Group 'Equal con- tribution 'Computer Vision & Machine Learning Group, Na- tional University of Singapore "Tongyi Lab, Alibaba Group ! "Communication University of China. Correspondence to: Bang Zhang <zhangbang. zb @ alibaba-inc.com/ inm..
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
For simplicity, HPE and PE in this work are used interchange- ably and denote Human and Hand Pose Estimation unless other- wise specified.

---

EXTPOSE: Robust and Coherent Pose Estimation by Extending ViTs
small region in the capture and is flexible move in space. Additionally, the palm and back of the hand have similar textures, leading to incorrect wrist rotations that globally flip the hand (Fig. 1). The challenging alignment issue of 3D pose estimation is not resolved by simply upgrading to the ViT architecture but also related to the task itself. Our method improvement inspired by the observation that 2D HPE models show better 2D alignment than those trained on 3D HPE tasks with the same architecture.
reuses pre-trained weights as much as possible.
Our unified EXTPOSE achieves remarkable improvement over previous ViT-based methods. For instance, with the aid of 2D hand pose estimates, the model has a better sense of hand global orientation (Fig. 1). Using video frames as input with temporal information alleviates prediction flickers and consistently gains in PA-MPJPP With the thought of unify- ing representation and reusing pre-trained knowledge, the model is enhanced at a rapid convergence. Additionally, we evaluate thoroughly the effectiveness of key design compo- nents, including 2D pose representations, fusion strategies, and learning capabilities. The framework is effective on the human body and hands for both image- and video- based set- tings. It binds these relevant settings with the foundational ViT-based methods; thus, they can benefit from advances made in ViT-based methods.
Second, the majority of ViT-based works focus on single- frame inference. With video inputs, however, temporal information is highly beneficial for handling complex occlu- sions and improving prediction stability. Adding the tem- poral dimension into ViT architectures incurs substantial computational overhead and demands 3D labeled video for training, thus impeding the development of video ViT-based methods in this domain. As result, existing video-based methods (Shen et al., 2023; Shin et al., 2024) rely on static frame features and additional temporal modules, thereby not fully maximizing the spatiotemporal feature interactions at each layer. Our approach to extend the well-developed image-based ViT to the video version along with advance- ments in large model training (OpenAI, 2024; Rasley al., 2020).
We summarize our contributions as follows:
we propose the innovative and systematic EXTPOSE framework to extend and enhance attention and ViT- based HPE backbones. The effectiveness is demon- strated by addressing two concrete issues in image alignment and temporal coherence in this work.
for effective fusion and utilization of the auxiliary 2D pose, its representation is unified and interacts with the image via the proposed dual-stream cross-modal attention of shared VIT.
This work enhances the image alignment and temporal co- herence of the ViT-based HPE features through modular attention extension and a unified framework. We draw in- spiration from the use of 2D skeletons as control signals in conditional human image and video generation (Zhang et al., 2023a; Hu et al., 2024). Moreover, the use specif- ically of 2D skeleton images, which precisely depict the human kinematic chain. Furthermore, the 2D skeleton im- ages exhibit strong pixel-wise spatial consistency with the original image. This enables joint processing with the input image with a shared backbone and can facilitate the use of pre-trained models. Extending this idea the temporal dimension, each frame of a video can also be processed by shared ViT backbone in parallel, with temporal relationships captured by frame interactions at each layer.
a new method is introduced for video-based HPE, where temporal context is seamlessly integrated through an extension of 2D image attention to 3D video attention, obviating the need for additional temporal modules.
ExTPoSE effectively remedies the current SOTA ViT- base method and promisingly shows consistent state-of- the-art results on several hand and human benchmarks including both monocular image and video settings.
2. Related Work
To that end, we propose EXTPOSE, short for Extending Poses, which extends a pre-trained VIT backbone with an extended attention mechanism. EXTPOSE elegantly lever- ages the power of ViT attention to model arbitrary and long-range relationships and flexible pairwise computations, inherently suitable for video tasks. It also allows for exten- sions to attend across different image and 2D pose modal- ities. Specifically, the attention mechanism is applied in a unified and hierarchical manner, depending on the availabil- ity of input information, and integrates auxiliary 2D pose evidence and video context to promote features for HPE. Finally, the framework does not introduce extra parameters or modules, as existing works do (Shin et al., 2024), and
2.1. Monocular Human & Hand Pose Estimation
Image-based HPE. 3D pose and shape estimation methods commonly use parametric 3D mesh models such as SMPL (Loper et al., 2015). SMPL-based HPE methods (Kanazawa et al., 2018; Kolotouros et al., 2019; Lin et al., 2021; Li et al., 2022: Wang et al., 2023) use a CNN backbone (e.g.. ResNet, HRNet) to extract image features and then regress pose and shape coefficients. More recent human (Goel et al., 2023; Dwivedi et al., 2024b; Zhuo et al., 2023) and hand (Pavlakos et al., 2024; Potamias et al., 2025) pose estimation methods have adopted a ViT architecture (Doso- vitskiy et al., 2021). MultiHPE (Baradel et al., 2024) and
2

---

EXTPOSE: Robust and Coherent Pose Estimation by Extending ViTs
AiOS (Sun et al., 2024) build ViT for multi-task learn- ing in a bottom-up paradigm, including human detection and pose estimation. Given ViT's potential for simplicity, scalability, and superior performance in HPE, it provides strong foundation for our work.
grating it with the image through structured representation and extended attention.
3. Preliminaries
3.1. Hand & Human Parametric Models
Video-based HPE extracts static features with CNN- based models (Kolotouros et al., 2019) and then use CNNs (Kanazawa et al., 2019), RNNS (Kocabas et al., 2020: Choi et al., 2021; Tekin et al., 2019), transformers Wan et al. (2021); Shen et al. (2023); Fu et al. (2023) to model temporal relationships. For example, TCMR (Choi et al., 2021) segments sequences into past, future, and whole seg- ments to ensure motion consistency. Wei et al. (2022) incor- porate non-local attention and hierarchical feature fusion, while Wan et al. (2021); Shen al. (2023); Fu et al. (2023) move to standard transformers (Vaswani et al., 2017). De- spite temporal integration, recent works find video-based methods often "surprisingly" underperform image-based methods due to an over-smoothing problem (Shin et al., 2024; Shen et al., 2023). Recently, Shin et al. (2024) inte- grates motion cues like 2D ViTPose (Xu et al., 2022) and SLAM camera data improve video-based HPE. In con- trast, our work enhances image-based ViT HPE with 3D attention. It better interacts and fuses spatial-temporal in- formation in a unified space and scalable manner (OpenAI, 2023; 2024), thereby eliminating the need for separate tem- poral modules and prerequisites, and achieving improved temporal coherence.
The parametric human body model SMPL. (Loper et al., 2015) and hand model MANO (Romero et al., 2017) is parameterized by the pose 0 [I00 x3 (0) = 72, 48 for human and hand respectively) and shape 3 E IR It uses Linear Blend Skinning function M to estimate the 3D mesh with vertices V E R RVIx
V.J= = M(0 B).
(I)
The pose consists of the global rotation of the root joint and the |J| 1 local rotations of other joints relative to their parents along the kinematic tree. The 3D joints JäD are based on a linear combination of the vertices V; with camera projection parameters R3 the 3D joints can be projected onto the 2D plane with projection IT:
22D =1[[JD c).
(2)
3.2. Self-Attention & Vision Transformer
Image feature extractors have evolved from ResNet (He et al., 2016), HRNet (Sun et al., 2019) to ViT (Dosovit- skiy et al., 2021). The ViT is scalable and stacks several of the same blocks to process information. It treats im- ages as a series of M non-overlapping d x d patch units X; E [gdxdx3, ; 1, 2. M, M a x W. first linearly projected into em-
2.2. HPE with Auxillary 2D Information
Different from fitting and adaptation with additional 2D (Kolotouros et al.. 2019; Lin et al., 2025), efforts to im- prove pixel aligmment HPE regression during generaliza- tion have focused on incorporating 2D inference alongside the image. Boukhayma et al. (2019) concatenate a hand keypoint heatmap with the RGB image along the chan- nel. More recently, Shin et al. (2024) follow the lifting paradigm (App. B) to integrate motion context from 2D joint coordinates via an MLP. Another approach learns spa- tially aligned intermediate representations. For example, Iqbal et al. (2018) use 2.51D representation, combining 2D heatmaps and depth maps to predict 3D coordinates. Zhang et al. (2021); Kocabas et al. (2021); Potamias et al. (2025) employ multi-task learning, supervising 2D auxiliary tasks (e.g., segmentation) to guide intermediate representations. These aligned representations often serve as spatial attention weights (Kocabas et al., 2021) or location features (Zhang et al., 2021: Potamias et al., 2025) during aggregation. As intermediate constraints tighten, they are incorporated into cascading two-stage paradigm (2D task followed by lifting) (Sengupta et al., 2021), where various 2D representations act as proxies (Pavlakos et al., 2018). This work incorporates auxiliary 2D information as model input, thoroughly inte-
d (d 16 as
default). The patches beddings by F; PatchEmb(XX) (X..)) E IR_,, where D is the embedding dimension. To encode the patch's 's position in the original image, H x W learnable positional embeddings PosEmb(i) E with equal dimensions, are then added to the tokens F., i.e., F, = F. + PosEmb(i). In each sub- sequent block of the encoder, each token fully exchanges information with others with core Self-Attention (SA) mechanism²:
1 J
(3) Formally, token F, computes attention weights A; with all tokens including itself by first obtaining the query, key, and value with respective projections Q: = Wº F,, K; W F,, V22 WVF E RD. The weight A, E RM is calculated by normalized scaled dot-product operation based on pairwise similarity:
F= + Attn(F, {Fi)M).
(K.i.K.........
KM KwI"
A = Softmax
(4)
DD
The index i in (F.)
M is omitted for simplicity.
3

---

EXTPOSE: Robust and Coherent Pose Estimation by Extending ViTs
Image
Feature
4.4 Cross Modality
Time
tuning
Attn
Modality
Emb
Block
Attn
4.5 Cross Frame
Pode
Attn
Fusa
Shape Query
AKv
4.3 2D Post Repy:
Canrrrn Query
Patchify
Backbone
Figure 2: The overview of EXTPOSE, extension of SOTA self-attention and ViT-based backbone. The skeleton image is used to represent 2D poses estimated by off-the-shelf ViTPose (Sec. 4.3). All VIT blocks are trainable and parameters are shared across the sequence of the image and 2D pose modality (Sec. 4.6). The image and 2D pose interact thoroughly with each other through cross- cross-modal modal attention (Sec. 4.4). Besides, cross-frame attention facilitates different frames to communicate and propagate video temporal context (Sec. 4.5). The enhanced feature is decoded into the mesh supervised by the HPE losses in Eqs. (12) to (15).
The feature residual for token is then derived from aggre- gating values (ViJM of all tokens with the attention weight A, i.., A O [V, V2...., V.rl.
4.2. Framework Overview
EXTPOSE refines ViT-based HPE by improving image align- ment and temporal coherence through a streamlined yet effective design. It first represents images, 2D poses, and video as unified visual token sequences {[F)) JM Jr within VIT (Sec. 4.3). This allows ViT to extract 2D pose features that encapsulate both spatial location and skeleton struc- ture. enhance localization, we introduce a cross-modal extension of self-attention, enabling thorough image-2D integration within a dual-stream design (Sec. 4.4). Orthogo- nally, Sec. 4.5 extends VIT's temporal scope by arranging tokens to attend across frames. Leveraging pre-trained 3D VIT pose estimator, EXTPoSE ensures efficient learning with rapid convergence (Sec. 4.6).
The above computation shows a core advantage of attention. Each token can collect information from tokens in any other position across the image; this facilitates capturing relation- ships between keypoints which are otherwise far away, e.g. between two wrists. Additionally, FeedForward Net (FFN) usually follows to transform feature space the end of the block. After stacked blocks, the feature of all patch tokens {F.) is extracted and could be applied to downstream tasks.
4. EXTPOSE
4.1. Problem Formulation
4.3. Unified 2D Pose Feature Extraction
Consider sequence of T' images of size H W (I+r XT*HxWx x3 Additionally, consider sequence of corre- sponding 2D poses {P* hr, and it becomes an image-based task when = 1. The dimension of 2D poses p which is annotated by an off-the-shelf 2D keypoint detector or hu- man, varies from representations discussed in Sec. 4.3. We aim to learn HPE model that predicts the pose parameters (0'}z Ir T for each frame and a video-shared shape parameter 3 E I0 The 3D mesh { ITT |xx and joint coordinate (}*)r E Tx|] could be obtained by Eq. (1). Camera projection parameters {cr) R Tx3 are also predicted to re-project the 3D mesh back onto the 2D plane (Eq. (2)), expected to align with human and hand in the image.
Various representations exist for 2D poses beyond tradi- tional joint coordinates used in the HPE work (Zhang et al., 2022), including heatmaps (Sun et al., 2019) and rendered skeleton images (Zhang al., 2023a) (Fig. 2). Heatmap representations H X place Gaussians joint locations spatially but are inconsistent in channels within diverse datasets and lack explicit structural relationships. While the skeleton images
E RHx x3 depict stick- figure poses on blank background, naturally aligning with human perception like RGB images. Joints are rendered as distinctively colored circles, connected by gradient-colored bones reflecting kinematic structure, with confidence (Gu et al., 2024) levels encoded via alpha blending. Confidence helps threshold unreliable 2D pose detection.

---

EXTPOSE: Robust and Coherent Pose Estimation by Extending ViTs
To integrate 2D poses into ViT's visual domain, inspired by (Zhang et al., 2023b), we employ skeleton images as alternative 2D pose representation. These images undergo the same patch embedding, positional encoding, and SA encoder processing as described in Sec. 3.2.
4.5. Attention from Images to Videos
Without requiring additional temporal modules, we extend EXTPoSE with its inherent attention mechanism to han- dle the temporal dimension. Unlike current video-based HPE methods, which use static frame features from image- based backbones (Kolotouros et al., 2019) and learn mo- tion on top. we directly leverage unified ViT's patchi- fied tokens, enabling access to richer information, i.e. 1
Training with skeleton images achieves rapid convergence and strong results without modifying architecture or train- ing settings (Sec. 5.4). This underscores ViT's generaliz- ability and scalability while demonstrating that 2D pose features extracted this way align well with image features. By employing a unified token-basee based framework, EXTPOSE maintains flexibility for future HPE model advancements.
1,2 T, where denotes the number of frames. Furthermore, while CNNs and RNNs struggle to capture long-term spatiotemporal dependencies, self-attention (SA) excels by allowing computation across frames, not just within them. This enables batch processing of frames, allowing information to propagate across distant frames, thereby enhancing temporal coherence and reducing monocular ambiguity in features and predictions.
4.4. Dual-stream Image-2D Pose Fusion
Figure 2 illustrates the dual-stream EXTPoSE-dua architec- ture, where a shared VIT backbone processes parallel fea- ture streams: image features {F/) yur and 2D pose features ) Both Image and 2D pose modality attend to both intra- and cross- moos- features. ensuring comprehensive fusion. For image feature F// the attention weights for self and mooal information collection are computed as:
Specifically, beyond inter-frame attention (Eq. (3)) between P and {PP)) JM, token E containing joint could at- tend to the same spatial location F in a different frame te and new location j at the different time F .j & i to capture motion. Note that the same location at t2 may cover the background.
[S!-T: S
The 3D spatiotemporal attention on inflated features (fFF)ar is formulated as:
A Softmax
(5)
VD
where S. - k'Q. - KPQ RR
(6)
= (LPP))))
(10)
K, K,Q.V are computed following Sec. 3.2. Similarly, the 2D pose feature also attends to both image and pose streams:
Above, F7 without superscript "I" and "p" implies the 3D attention is only performed within respective modal- ity streams, ie., orthogonal to the cross-modal attention in Sec. 4.4, as aims to focus on the time dimension, which can be implemented by simply collapsing the modality axis into the batch axis versatile attention mask (Vaswani et al., 2017). At each attention block, for two stream fea- ture {{F Jas hr, ([FF""
[K' KPI'P"
Aº = Softmax
(7)
VD
The final features enhanced by the other new branch infor- mation are:
JaM }r E T concatenated along the modality axis, the attention mask matrix has the following structure:
AF/ = A ④ V'P, AFT = A の V'P,
(8)
where 1'p [V'; VrJ. The whole operation is denoted as:
17 = | Aun([F; F411, (E PP))), (9)
-o0xx
M =
(1I)
00TXM
OrxM
and can be implemented by concatenating the two streams before self-attention.
where M E RTTMXMM is added into correlation within the Softmax operation Eq. (4) so that ooo masks out inactive interaction across modalities after the Softmax operation.
Note that the bimodal SA in a shared space benefits from our unified image-2D pose feature representation. Dot product- based attention assembles information anywhere in space without requiring perfectly aligned 2D poses. It maximizes the interaction between image and 2D pose features, allow- ing the network to refine features of both modalities dynam- ically and adaptively. As ExTPoSE- justifies easy model adaptation to 2D pose, learning more robust predictions is expected to converge fast.
This zero-shot "free lunch" for enhancing temporal coher- ence even comes with directly performing 3D attention on a pre-trained image-based model. Fusion in the face of disagreement benefits from further training to advance per- formance, with additional temporal positional embeddings that encode frame location and motion ordering.
5

---

EXTPOSE: Robust and Coherent Pose Estimation by Extending ViTs
4.6. Training Losses& & Implementation Details
Table I: Comparison with SOTA HPE methods on the 3DPW dataset. Our method trained based on HMR2.0 (Goel al., 2023). PJ is short for PA-MPJPE. i indicates the upper bound of using GT 2D poses. The best and second best scores are highlighted, respectively. The num- hers brackets are improvements w.r.. SOTA. Note that WHAM (Shin et al., 2024) also uses 2D poses, and ours outperforms it by 1.9mm PA-MPJPE.
We follow Kolotouros et al. (2019); Goel et al. (2023) and train on both 3D and 2D data jointly. The total loss C is weighted sum of the losses in Eqs. (12) to (15). These in- clude 3D joint loss Ljuint. parameter loss Cparam when SMPL parameters are available, a 2D reprojection loss Creproj, and an adversarial loss with a discriminator net to impose pose and shape reasibility (Kanazawa et al., 2018):
Gjintt -| - 11: param = 110 - 0112 + || B
(12)
METHOD
MPYPE,
MPJPEL
PJL
(13)
SPIN (ICCV'19)
112.8
96,9
59.2
12L-MEsHNer 20)
100.0
60.0
Cregrej 1| - 120|
(14)
HYBRIK (CVPR 21)
82.3
71.6
41.8
PYMAF (ICCV'21)
110.1
92.8
58.9
fadv [1, 3) 11.2
(15)
IMAGE
PARE (ICCV'21)
88.6
74.5
46.5
CLIFF (BCCVV222
81.2
69.0
43,0
BEDLAM-CLIFF (CVPR)))))
85.0
72.0
46,6
REFMICCC
75.1
65.3
40.5
During the training of all backbone parameters, besides standard affine and color data augmentation (Goel et al., 2023), each modality is masked out as a whole with a prob- ability of 50% cultivate EXTPoSE's ability to extract features in each input modality individually. We use the Py implementation of scaled dot product attention with the mask and accelerated flash attention to speed com- putation and save GPU memory. An Adam V optimizer (Loshchilov & Hutter, 2019) is deployed with a learning rate le-5, 31 = 0.9, 32- = 0.999, and weight decay of le- 3. Training lasts for 50K iterations with a batch size of 32 on 8 A100 GPUs For other details of datasets and settings, please refer to the human and hand experiment Secs. 5.2 and 5.3.
TOKFNHMR (CVPR'24)
84.6
71.0
44.3
I1MR2.0 (ICCV'23)
82.2
69.8
44.4
EXTPOSE (T' ===1)
68.9
55.6
35,5
(-8.3%
(-14,9%)
(-12.3%)
ExTPose (T
36.7
25,4
VIBE (CVPR 20)
98.4
82.9
$1.9
(CVPR'21) 21)
101.4
86.5
52.7
MAED (ICCV"21)
92.6
79.1
45.7
MPS-NET (CVPR'22)
99.0
84.3
52.1
GLAMR (CVPR'22)
51.1
VIDEO
D&D (ECCV'22)
13.7
42.7
SLAHMR (CVPR'23)
55.9
TRACE (CVPR 23)
95.4
79.1
50.9
GLOT (ICCV'23)
96.3
80.7
50.6
WHAM (CVPR 244)
68.7
57.8
35.9
EXTPoSE (T 16)
67.5
54.2
34.0
(-1.7%)
(-6.2%)
(-5.3%)
based settings (T = 1). Given complicated light condition- ing, occlusion, and truncation in 3DPW (Von Marcard et al., 2018), state-of-the-art (SOTA) HPE methods (Goel et al., 2023; Dwivedi et al., 2024b) struggle to predict accurate meshes based on the image's global context. In contrast, 2D pose estimators still provide more valuable localized cues, though SOTA is trained from ViTPose. Combining both modalities improves PA-MPJPE from 44.4mm to 35.5mm, as shown in Tab. 1. With GT 2D poses, EXTPOSE matches the annotation errors of the dataset.
5. Experiments
5.1. Evaluation Metrics
For 3D accuracy evaluation, the reconstruction performance usually measured in terms of Mean Per Joint and Vertex Error (MPJPE, MPVPE, in mmm and the ones after Pro- crustes Alignment (PA-MPJPE, PA-MPVPE). Addition- ally. the F-score of correct poses with errors less than 5mm and 15mm (F@5, @@ 15) is evaluated for hand benchmarks (Hampali et al., 2020). For 2D pose alignment evaluation, the commonly used Percentage of Correct Keypoint (PCK) metric is computed at different error tolerance thresholds including 0.05, 0.1, and 0.15.
Video benchmark results for video-based HPE are also presented in Tab. 1. Existing methods typically rely on static image features, whereas our approach directly trains on image- ViT methods, without additional modules. Contrary prior trends where video-based methods (Shen et al., 2023) underperform amage-based methods in frame- wise accuracy, our model achieves a 1.5mm improvement in PA-MPJPE by utilizing available video temporal context. PXSE also outperforms WHAM by 5.3%, reflecting that our extended cross- modsstyy and -frame attention derives a unified and superior framework.
5.2. Human Pose Estimation
Settings. Following standard practice (Shin al., 2024:
Goel et al., 2023), ExTPoSE initialized from HMR2.0 is trained on mixed 3D datasets including 3DPW (Von Mar- card et al., 2018), 8), Human3.6N 3.M (lonescu et al., 2013), MPI- INF-3DHP (Mehta et al., 2017), and COCO (Lin et al., 2014).
5.3. Hand Pose Estimation
As our framework is widely applicable for both hand and human tasks, experiments are also conducted on the standard
Image benchmark. We first evaluate our method in image-
6

---

EXTPOSE: Robust and Coherent Pose Estimation by Extending ViTs
Table 2: SOTA 3D accuracy results on the FreiHAND dataset. HaMeR (Pavlakos et al., 2024) is the ViT baseline. PJ and PV are short for PA-MPJPE and PA-MPVPE. The best and second best scores highlighted respectively. The numbers in brackets are improvements w.r.t. SOTA.
Table 3: SOTA 3D metric results on the (0) dataset. HaMeR (Pavlakos et al., 2024) is the ViT base- line. PJ and PV are short for PA-MPJPE and PA-MPVPE. The best and second best scores are highlighted respec- tively. The numbers in brackets are improvements w.r.t. SOTA.
METHOD
PJU
PVL
F@St
POSE2MESH (0)
7.7
7.8
0.674
0.969
12L MESSONTT (FCCV
7.4
7.6
0.681
0.973
METROD
AUXC
P1L
AUCC1
PV]
F-8.1
FGIST
METRO (CVPR'21) 21)
HAUV-HANDNET (ICCV"2I)
HANDAR (ICCV'21)
6.7
6.7
6.9
6.7
0.724
0.707
0.977
0,981
METRO
POGETMESM (ROCV-20)
MISHNNT (ECCV'20)
0.754
11.2
0.722
0.749
17.7
0.429
0.441
0.909
MESHIGRAPHIOREE (ICCV"21)
5.9
0.764
0.986
(CVPR
0.503
9.9
0,510
0.528
0.556
HAMER (CVPR'24) 24)
AMVUR (CVPR 23)
MOBRECCON (CVPR
6.2
6.0
5.7
5.8
6.1
0,785
0.767
0.784
0,990
0.987
0.986
DHACE
AatiBooss (CVPR 22)
KovIoINTT TRANS (CVPR 22)
E2UY HANDNer (ICCY'21)
MonRscox (CVPR
6.773
0.504
0.786
11.4
10.3
9.9
9.2
0.759
0.782
10.1
10.9
0.500
0.538
0.483
0.543
0.944
EXTPOSE
4,9
5.1
0.823
0.993
AMVUR (CVPR'
0.608
(-14.0%)
(-10.5%)
(+1.8%
(+0,3%)
HAMER (CVPS' 24)
0.946
2.3
0.841
2.9
0.635
0.900
ExTPosa (2" -
0,858
7.8
0,850
7.5
0.668
0,985
(+1.4%)
(4.1%)
(+1.1%)
(-5.1%
(+39%)
[+6.%]
ViTPose
HaMeR
Ours
(CVPR
Front
Side
Front
Side
V10800
TCMR (CVPR'21)
TEMPCLR (3DV 22)
11.4
10.6
10.9
10.6
0.481
0.463
0.933
0.937
DESORMER (IOCV-23)
M.4
911
0.546
0.263
Motion
EXTPosE (7 16)
6.9
0.856
0.667
6,901
(-266%)
(-1989)
(+22.3%)
(+3.9%)
blur
Table 4: SOTA 2D PCK results at different thresholds on the Hlnt dataset. HaMeR is the ViT baseline. f indicates using the upper bound of GT 2D poses. The best and second best scores are highlighted respectively. The numbers in brackets are improvements w.r.t. SOTA.
HOI
& ego3D
Merson
9:0.55:
NEw DAYS
640.1
90.151
(00.051
VISOR 660.1
600 151
FRANKMEOCAP (CCVW(22)
15.1
16.8
66.2
MEsHIGI SHORRII (CCCV13))
15.3
43.0
50.2
19.1
67.4
HomndeeNE (CVPR 223
13.9
19.1
50.3
13.6
58.7
68.8
Figure 3: Qualitative comparison. The reconstructions are consistently enhanced, particularly more challenging scenarios, e.g., motion blur and (object and self-) occlusion.
HAMER (CVPR 24)
45.0
78.01
55.8
43.0
76.9
59.3
VETPose (NoukiPs 22)
65.5
70.8
90.6
59.
4566
(-242%)
(48.7%)
(+..4%
(+42.15)
(-15.1%)
(+7.1%
61.0
hand benchmark to validate the method follows.
Video benchmark. HO3D is hand-object interaction video dataset with occasional occlusions, where the space of poses is narrower than FreiHAND (Zimmermann et al., 2019). In addition to PA-MPJPE, HO3D benchmarks (Ham- pali et al., 2020) evaluate the Area Under the Curve for correct 3D joints and vertices (AUC), AUCy). As shown in Tab. 3, method improves PA-MPJPE by 9.1% through better keypoint localization during object grasping, while further gains are achieved by leveraging information from other visual frames. Previous video-based HPE remains much less accurate than image- methods, aligning with trends observed human benchmarks.
Settings. For evaluation on hand video and image bench- marks below, we follow and also train HaMeR (Pavlakos et al., 2024) on multiple 3D datasets, FreiHAND (Zimmer- mann et al., 2019), HO3D (Hampali et al., 2020), MTC (Xiang al., 2019), RHD (Zimmermann & Brox, 2017), InterHand2.6M (Moon et al., 2020), H203D (Hampali et al., 2020), DeB (Chao et al., 2021), and 2D datasets, COCO WholeBody (Jin et al., 2020). Halpe (Fang et al., 2022) and MPII NZSL (Simon et al., 2017).
Image benchmark. We evaluate our method with T on standard hand image datasets. FreiHAND (Zimmer- mann et al., 2019) includes diverse postures and severe self-occlusion. Despite the significant misalignment issues typical in generalization, few studies have explored 2D evi- dence for improving hand pose estimation (Figs. I and 3). Our results on EXTPOSE in Tab. demonstrate the bene- fit of utilizing 2D grounding to enhance fair 3D accuracy benchmarks.
2D pose accuracy. To assess real-world generalization, Pavlakos et al. (2024) introduced the challenging HInt dataset, capturing hands in complex daily activities from both exocentric and egocentric perspectives. While state-of- the-art HPE methods struggle, 2D pose estimation performs reasonably well (Tab. 4 and Fig. 3). This supports the idea that structured 2D pose localization can indeed improve
7

---

EXTPOSE: Robust and Coherent Pose Estimation by Extending ViTs
Table 5: Ablations of branches and 2D pose representa- tions on the FreiHAND dataset. "Skel." " is short for the skeleton.
Table 6: Ablations of different fusion and training strate- gies on the HInt dataset. denotes adding extra parameters. "Only Q, K" indicates only training Q and K projectors of attention while "I"f half" means only the first half of ViT blocks are trained. ExTPoSE features full attention and shares and trains full parameters.
IMG
2D POSE
PA-MPIPE]
PA-MPYPEL
F0.5t
FØ 151
ID
6.5
6.6
0.724
0.983
HENIMA
6.3
6.3
0.747
0,984
Sket. IMAGE
6.2
0.742
0.985
SKEL. IMAGE
6.0
4,9
5.7
5.1
0.783
0.823
0.991
0,993
METHOD
@0.05
NEW DAYS
00.1
80.15
00.05
VISOR
@0.15
HAMER
18,0
78.0
88.8
43.0
76.9
89.3
FUSION
LATE PUSION
50.5
82.4
92.5
52.5
87.1
95.6
HPE generalization in challenging scenarios. The accuracy drop from 2D projection, compared to ViTPose, is likely due the generalization gap (similar to the GT 2D case), with model- HPE as the primary focus, trend also observed in (Chen et al., 2025a).
CHANNEL CONCAT"
56.3
83.6
92.2
55.9
87.3
95.3
CONTROLNET
55.6
83.5
92.3
57.7
87.5
95.5
TRAINING
FROM VITPOSE
49.9
82.2
92.2
46.4
85.33
95.2
ONLY Q.
50.0
81.9
92.3
49.1
85,3
95.1
1 HALF
50.8
82.2
92.3
50.2
85,8
95.2
EXTPOSE
59.6
84.8
92.7
61.1
88.5
95.6
Image/ /
2D pose p
5.4. Ablations& & Analysis
ling I
The ablation study is conducted for each component design one by one, on the in-distribution generalization of the Frei- HAND dataset (Zimmermann et al., 2019) and more chal- lenging generalization on the Hint dataset (Pavlakos et al., 2024) and the 3DPW dataset (Von Marcard et al., 2018). Results across several hand and human benchmarks demon- strate the effectiveness and generality of our EXTPOSE framework.
177 pos p
Image / w/ 2D pose p Attn maps
Attention weights
Figure 4: Visualization of cross-modal attention (Hertz et al., 2022). The attention weight is not solely concentrated along the diagonal, and the attention maps depiet that each branch attends distinct areas of the other branch, not just the center. Reference highlights the corner and boundary indicate spatial awareness for 3D pose estimation.
Optional 2D pose & quality. We first assess the impact of 2D poses by training a variant with only the image and setting the 2D pose input to zero. As the ViT backbone is well-trained, this variant shows minimal performance improvement (4th row in Tab. 5), highlighting the value of precise 2D pose localization. Next, using ground-truth 2D poses instead of those estimated by Xu et al. (2022) further improves performance to an impressive 3.7mm on the FreiHAND dataset.
Image-2D pose fusions. Apart from SOTA comparison with different fusion in Tab. 1, classic channel concatenation, late- layer fusion, and ControlNet fusion (Zhang et al., 2023a) are implemented under the same framework for a comprehen- sive comparison. Our attention mechanism enables more effective information exchange between the two streams at each layer, faster converging to improved performance (2nd block in Tab. 6). As shown in Fig. 4, cross-modal at- tention allows the branches to mutually enrich each other's s representations. The image stream enhances alignment by querying clear keypoint location and structure information provided by the 2D hand skeleton image (red box) while the 2D pose stream extensively attends to missing RGB in- formation across the hand and background (e.g., wrist the green box), aiding depth estimation.
We also evaluate a lifting counterpart with only 2D pose input (3nd row in Tab. 5). In this case, 2D-to-3D lifting fails to leverage RGB image texture in situations of depth ambiguity missing keypoint detection, underperforming image- based HPE. This reinforces the need for combining both modalities in HPE.
2D pose representations. Our key innovation is adopting an image-based representation of 2D poses, aligning more closely with RGB image features, and leveraging knowl- edge from pre-trained HPE models. For comparison with conventional ID arrays, we employ a transformer encoder to extract features from joint coordinates, which are then fused with image features using cross- attentionnn Our image representation not only converges faster (see convergence study in App. C), but also achieves 4.6% lower PA-MPJPE (first three rows in Tab. 5). Spatial heatmaps outperform ID arrays but are less preferred than skeleton images.
frter-frame attention. EXTPoSE with T =1 improves pixel alignment with 2D pose assistance but causes jittering in video due to the lack of temporal frame relationships. Instead of retraining a video-based HPE model, video-based EXTPOSE leverages well-trained image- models and extends attention across frames to reconstruct a smoother
8

---

EXTPOSE: Robust and Coherent Pose Estimation by Extending ViTs
References
Table Ablations of the sequence length on the 3DPW dataset (Von Marcard et al., 2018).
Baradel, F., Armando, M., Galaaoui, S., Brégier, R., Wein- zaepfel, P., Rogez, G., and Lucas, T. Multi-HMRR Multi- person whole-body human mesh recovery in single shot. In
L
MVEI
MPJPES
PA-MPJPE]
68.9
55.6
35.5
68.2
55.1
34.7
16
67.5
54.2
34.0
Boukhayma, A., Bem, d., and Torr, P. H. 3D hand shape and pose from images in the wild. In CVPR, 2019.
32
67.9
54.8
34.3
4D mesh at minimal cost. For a better assessment, we provide rendered videos in the Sup. Mat. The sequence length hyperparameter is studied in Tab. 7, with 16 yielding reasonable results for the 3DPW dataset.
Chao, Y.-.,, Yang, W., Xiang, Y., Molchanov, P., Handa, A., Tremblay, J., Narang, Y. S., Van Wyk, K., Iqbal, U. Birchfield, S., et al. Dex YCB: A benchmark for capturing hand grasping of objects. In CVPR, 2021.
Training strategies. A key advantage of EXTPOSE is that eliminates the need to train a model from scratch on large datasets. Table ablates network initialization and learnable layers (Sec. 4.6), showing that initialization from ViT-based HPE methods and training the whole backbone is crucial for enhancing the model.
Chen, R., Yang, L., and Yao, A. MHEntropy: Entropy meets multiple hypotheses for pose and shape recovery. In ICCV, 2023.
Chen, X., Song, Z.,, Jiang, X., Hu, Y., Yu, J., and Zhang, L. HandOS: 3D hand reconstruction in one stage. In CVPR, 2025a.
6. Conclusion & Limitations
Chen, Y., Chen, R., Zhang, Z., Cheng, Y., and Chang, H. J. Multi-hypothesis 3D hand mesh recovering from single blurry image. In ICME, 2025b.
VIT has been witnessed to be prevalent and dominant in HPE; yet, it is still not satisfactory with 2D image alignment and temporal coherence in real-world applications. In this work, flexible framework EXTPoSE is proposed to extend the attention of SOTA image-based VIT HPE to encompass information of multiple additional dimensions. It benefits from elaborate representation unification, parameter reusing and sharing, and full hierarchical attention fusion. Extensive experiments demonstrate its wide effectiveness in rectifying visual alignment and capturing temporal relationships. In the future, reconstruction could be extended cover the whole body. Additionally, it is also valuable to the framework to take advantage of more visual modalities, such as SAM prior (Ravi et al., 2025) and multiple views (Gao et al., 2024), as well as incorporating the latest Transformer efficiency studies.
Choi, H., Moon, G., and Lee, K. M. Pose2mesh: Graph con- volutional network for 3D human pose and mesh recovery from a 2D human pose. In ECCV, 2020.
Choi, H., Moon, G., Chang. J. Y., and Lee, K. M. Beyond static features for temporally consistent 3d human pose and shape from video. In CVPR, 2021.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLRR 2021.
Dwivedi, S. K., Schmid, C., Yi, H., Black. M. J., and Tzionas, D. POCO: 3D pose and shape estimation with confidence. In 3DV, 2024a.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Dwivedi, S. K., Sun, Y., Patel, P., Feng, Y., and Black, M. J. TokenHMR: Advancing human mesh recovery with tokenized pose representation. In CVPR, 2024b.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J.,
Acknowledgments
Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.
This research was also funded in part by Google South Asia & Southeast Asia Research Award given to A. Yao and the National Natural Science Foundation of China NNo.62406298). We would also like to thank the ACS and re- viewers for their valuable suggestions. Chen and Rongrong contributed professional guidance in art design.
Fang, H.-S., Li, J., Tang. H., Xu, C., Zhu, H., Xiu, Y., Li, Y.L.., and Lu, C. AlphaPose: Whole-body regional multi-person pose estimation and tracking in real-time. TPAMI, 2022.

---

EXTPOSE: Robust and Coherent Pose Estimation by Extending ViTs
Fu, Q., Liu, X., Xu, R., Niebles, J. C., and Kitani, K. M. Deformer: Dynamic fusion transformer for robust hand pose estimation. In ICCV, 2023.
Kolotouros, N., Pavlakos, G., Black, M. J., and Daniilidis, K. Learning reconstruct 3D human pose and shape via model-fitting in the loop. In ICCV, 2019.
Gao, R., Holynski, A., Henzler, P., Brussee, A., Martin- Brualla, R., Srinivasan, P. Barron, J. T., and Poole, B. Cat3D: Create anything in 3D with multi-view diffusion models. In NeurIPS, 2024.
Li, J., Xu, C., Chen, Z., Bian, S., Yang. L., and Lu, C. HybrlK: A hybrid analytical-neural inverse kinematics solution for 3D human pose and shape estimation. In CVPR, 2021.
Goel, S., Pavlakos, G., Rajasegaran, J., Kanazawa, A., and Malik, Humans 4D: Reconstructing and tracking humans with transformers. In ICCV, 2023.
Li, W., Liu, M., Liu, H., Wang, P., Cai, J., and Sebe, N. Hourglass tokenizer for efficient transformer-based 3D human pose estimation. In CVPR, 2024.
Gu, K., Chen, R., Yu, X., and Yao, A. On the calibration of human pose estimation. ICML,, 2024.
Li, Z., Liu, J., Zhang, Z., Xu, S., and Yan, Y. CLIFF: Carrying location information in full frames into human pose and shape estimation. In ECCV, 2022.
Hampali, S., Rad, M., Oberweger, M., and Lepetit, V. Hon- notate: A method for 3D annotation of hand and object poses. In CVPR, 2020.
Lin, K., Wang, L., and Liu, Z. Mesh graphormer. In ICCV, 2021.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In CVPR, 2016.
Lin, Q.. Chen, R., Gu, K., and Yao, A. Semantics-awan test-time adaptation for 3D human pose estimation. In ICML, 2025.
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D. Prompt-to-prone to-mmmpp image editing with cross attention control. In arXiv, 2022.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft COCO: Common objects in context. ECCV, 2014.
Hu, L., Gao, X., Zhang, P., Sun, K., Zhang, B., and Bo,
L. Animate anyone: Consistent and controllable image- video synthesis for character animation. In CVPR, 2024.
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., and Black, M. SMPL: A skinned multi-person linear model. In SIGGRAPH Asia, 2015.
lonescu, C., Papava, D., Olaru, V., and Sminchisescu, C. Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments. TPAMI, 2013.
Loshchilov, and Hutter, F. Decoupled weight decay regu- larization. In ICLR, 2019.
Martinez, J., Hossain, R., Romero, J., and Little, J. A sim- ple yet effective bascline for 3D human pose estimation. ICCV, 2017.
Iqbal, U., Molchanov, P., Gall, B. and Kautz, J. Hand pose estimation via latent 2.5D heatmap regression. In ECCV, 2018.
Mehta, D., Rhodin, H., Casas, D., Fua, P., Sotnychenko, O., Xu, W., and Theobalt, C. Monocular 3D human pose estimation the wild using improved CNN supervision. In 3DV, 2017.
Jin, S., Xu, L., Xu, J.. Wang, C., Liu, W., Qian, C., Ouyang, W., and Luo, P. Whole-body human pose estimation in the wild. In ECCV, 2020.
Kanazawa, A., Black, M. J., Jacobs, D. W., and Malik, J. End-to-end recovery of human shape and pose. In CVPR, 2018.
Moon, G., Yu, S.-I., Wen, H., Shiratori, T., and Lee, K. M. Interhand2.66 2:: A dataset and baseline for 3D interacting hand pose estimation from single RGB image. In ECCV, 2020.
Kanazawa, A., Zhang, Y., Felsen, P., and Malik, J. Lear- ing 3D human dynamics from video. In CVPR, 2019.
Oh, Y., Park, J., Kim, J., Moon, G., and Lee, K. M. Recov- ering 3D hand mesh sequence from a single blurry image: A new dataset and temporal unfolding. CVPR, 2023.
Kocabas, M., Athanasiou, N., and Black, M. VIBE: Video inference for human body pose and shape estimation. In CVPR, 2020.
OpenAl. GPT-4 technical report. In OpenAl, 2023.
Kocabas, M., Huang, C.-H. P., Hilliges, O., and Black, J. J. PARE: Part attention regressor for 3D human body estimation. In ICCV, 2021.
OpenAL. Video generation models as world simulators. In OpenAI, 2024.
10

---

EXTPOSE: Robust and Coherent Pose Estimation by Extending ViTs
Pavlakos, G., Zhu, L.., Zhou, X., and Daniilidis, K. Learning to estimate 3D human pose and shape from single color image. In CVPR, 2018.
Sun, Q.. Wang, Y., Zeng, A., Yin, W. Wei, C., Wang, W., Mei, H., Leung, C.-S., Liu, Z., Yang, I.., et al. AiOS: All- in-one-stage expressive human pose and shape estimation. In CVPR, 2024.
Pavlakos, G., Shan, D., Radosavovic, I, Kanazawa, A., Fouhey, D., and Malik, J. Reconstructing hands in 3D with transformers. In CVPR, 2024.
Tekin, B.,, Bogo, F., and Pollefeys, M. H+O: Unified egocen- tric recognition of 3D hand-object poses and interactions. In CVPR, 2019.
Potamias, R. A., Zhang, J., Deng, J., and Zafeiriou, S. WiLoR: End-to-end 3D hand localization and reconstruc- tion wn-the-wild. In CVPR, 2025.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A., Kaiser, L., and Polosukhin, I. Attention is all you need. In NIPS, 2017.
Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deep- Speed: System optimizations enable training deep learn ing models with over 100 billion parameters. In KDD, 2020.
Von Marcard, T., Henschel, R., Black, M. J., Rosenhahn, B., and Pons-Moll, G. Recovering accurate 3D human pose in the wild using imus and a moving camera. ECCV, 2018.
Ravi, N., Gabeur, V., Hu, Y.-T., Hu, R., Ryali. C., Ma, T., Khedr, H., Rädle, R., Rolland, C., Gustafson, I.., Mintun, E., Pan, J., Alwala, K. V., Carion, N., Wu, C.-Y., Girshick, R., Dollár, P., and Feichtenhofer, C. SAM 2: Segment anything in images and videos. In ICLR, 2025.
Wan, Z., Li, Z., Tian, M., Liu, J., Yi, S., and Li, H. Encoder- decoder with multi-level attention for 3D human shape and pose estimation. In ICCV, 2021.
Wang. W., Ge, Y., Mci, H., Cai, Z., Sun, Q., Wang, Y., Shen, C., Yang, L.,, and Komura, T. Zolly: Zoom focal length correctly for perspective-distoree human mesh reconstruction. In ICCV, 2023.
Romero, J., Tzionas, D., and Black, M. J. Embodied hands: Modeling and capturing hands and bodies together. In SIGGRAPH Asia, 2017.
Sengupta, A., Budvytís, I., and Cipolla, R. Hierarchical kinematic probability distributions for 3D human shape and pose estimation from images in the wild. In ICCV, 2021.
Wang. Y., Wang, Z., Liu, L.,, and Daniilidis, K. TRAM: Global trajectory and motion of 3D humans from in-the- wild videos. In ECCV, 2024.
Wei, W.-L., Lin, J.-C., Liu, T.-L., and Liao, H.-Y. M. Cap- turing humans in motion: Temporal-attentive 3D human pose and shape estimation from monocular video. In CVPR, 2022.
Shan, W., Liu, Z., Zhang, X., Wang, S., Ma, S., and Gao, W. P-STMO: Pre-trained spatial temporal many-to-one model for 3D human pose estimation. In ECCV, 2022.
Shan, W., Liu, Z., Zhang. X., Wang, Z., Han, K., Wang, S., Ma, S., and Gao, W. Diffusion-based 3D human pose estimation with multi nulti-hyppothssi aggregation. In ICCV, 2023.
Xiang, D., Joo, H., and Sheikh, Y. Monocular total capture: Posing face, body, and hands in the wild. In CVPR, 2019,
Xu, Y., Zhang, J., Zhang, Q., and Tao, D. ViTPose: Simple vision transformer basclines for human pose estimation. In NeurlPS, 2022.
Shen, X., Yang, Z., Wang, X., Ma, J., Zhou, C., and Yang, Y. Global-to-local modeling for video-based 3D human pose and shape estimation. In CVPR, 2023.
Yu, Z., Li, C., Yang, L., Zheng, X., Mi. M. B., Lee, G. H., and Yao, A. Overcoming the trade-off between accuracy and plausibility in 3D hand shape reconstruction. In CVPR, 2023.
Shin, S., Kim, J., Halilaj, E., and Black, M. J. WHAM: Reconstructing world-grounded humans with accurate 3D motion. In CVPR, 2024.
Zhang, H., Tian, Y., Zhou, X., Ouyang, W., Liu, Y., Wang, L., and Sun, Z. PyMAF: 3D human pose and shape regression with pyramidal mesh alignment feedback loop. ICCV, 2021.
Simon, T., Joo, H., Matthews, I., and Sheikh, Y. Hand keypoint detection in single images using multiview boot- strapping. In CVPR, 2017.
Sun, K., Xiao, B., Liu, D., and Wang, J. Deep high- resolution representation learning for human pose estima- tion. In CVPR, 2019.
Zhang, J., Tu, Z., Yang. J., Chen, Y., and Yuan, J. MixSTE: Seq2seq mixed spatio-temporal encoder for 3D human pose estimation in video. In CVPR, 2022.

---

EXTPOSE: Robust and Coherent Pose Estimation by Extending ViTs
Zhang, L., Rao, A., and Agrawala, M. Adding condi- tional control text-to-image diffusion models. In ICCV, 2023a.
In addition the self-contained manuscript, the appendix contains framework concept (App. A), more background (App. B), and experimental results (App. C).
Zhang, L., Rao, A., and Agrawala, M. Adding condi- tional control text-to-image diffusion models. In ICCV, 2023b.
A. Extension Framework of Image-based Pose Estimation
Zhu, W., Ma, X., Liu, Z., Liu, L., Wu, W., and Wang. Y. MotionBERT:: unified perspective on learning human motion representations. In ICCV, 2023.
As existing ViT-based methods typically require costly train- ing, to make model upgrade more efficient and easier, a new paradigm EXTPOSE is proposed: extension of pre-trained image-based VIT HPE. Specifically, no additional parameter or retraining must be performed, and new 2D and tempo- ral information are processed by a backbone shared with the image. It leverages flexible attention computation of ViT, which is not constrained to be identical to that in the pre-training. Therefore, the model easily extends to refer to newly provided information and adaptively captures mutual relationships. The new paradigm of our problem formula- tion also bridges relevant lifting and video-based HPE with image-based HPE and unifies them within a single cohesive framework. Notably, the extension paradigm is naturally driven by improvements in image-based HPE. This idea of clarity and practicality is inspired by other fields such as generation extending from the image (Esser et al., 2024) to current video (OpenAI, 2024) and 3D domain (Gao et al., 2024).
Zhuo, L., Cao, J., Wang, Q., Zhang, B., and Bo, L. Towards stable human pose estimation via -iew fusion and foot stabilization. In CVPR, 2023.
Zimmermann, C. and Brox, T. Learning to estimate 3D hand pose from single RGB images. In ICCV, 2017.
Zimmermann, C., Ceylan, D., Yang, J., Russell, B., Argus, M.. and Brox, T. FreiHAND: A dataset for markerless capture of hand pose and shape from single RGB images. In ICCV, 2019.
B. 2D-to-3D Lifting
As mentioned in the related work, some works focus on the second stage of HPE, relying on 2D inference and totally ig- noring the image. Among them, simple 2D joint coordinates are the most prevalent representation to work with, from which the 3D skeleton is lifted holistically, Martinez et al. (2017) establish preliminary benchmark with residual MLP in indoor human environments (lonescu al., 2013; Mehta al., 2017). Followup works (Shan et al., 2022; Zhang et al., 2022: Zhu et al., 2023; Shan et al., 2023; Li et al., 2024) extend to explore the vidco setting and Trans- former solution. Instead of outputting 3D joint coordinates, Choi et al. (2020) directly regresses the vertex coordinates of the mesh eliminating an additional Inverse Kinematies task (Li et al., 2021).
Comparison with our paradigm. Conventional lifting and our work share similarities supplementing models with 2D poses; both are evaluated with JPE-type errors. However, some critical protocol differences make the two lines of work difficult compare directly. Firstly, pose representation, the target outputs in lifting are only skele- ton joint coordinates, which favors the MPJPE metric (Yu et al., 2023). Secondly, lifting- -iase methods significantly benefit from using a larger window size (number of frames), typically 243, due to their operation lower-dimensional
12

---

EXTPOSE: Robust and Coherent Pose Estimation by Extending ViTs
Table SOTA 3D accuracy results on the Human3.6M dataset (Ionescu al., 2013). Our method improves HMR2.0 (Goel et al., 2023) with 16 frames and achieves comparable results to those using 243 frames. i denotes using GT 3D keypoints for final aggregation of multiple predictions. The best and second best scores are high- lighted for lifting and model-based methods respectively. Note that methods are usually compared within their respec- tive paradigms on this benchmark.
2.2
ID
Channel concat
ControlNet
1.6
Ours
Loss
METHOD
T
MPJPEL
PA-MPIPEL
1.0
TCN (CVPR'19) 19)
243
46.8
36.5
POSEFORMER (ICCV'21)
81
44.3
ANATOMY (1))
243
35.0
STRIDED (TMM 22)
351
43.7
(CVPR'22 22)
LIFTING
MIXSTE (CVPR'22)
P-STMO (ECCV'22)
243
39.8
34.4
30.6
0.4 0
2.5K
SK
7.5K
10K
POSEFORMERV2 (CVPR 23)
243
45.2
35.6
Steps
DUrPoSE (CVPR'23) 23)
243
36.9
28.7
MOTIONBERT (ICCV'23)
243
37.5
Figure 5: Faster and more stable convergence compared to different 2D pose representations and fusion of hand experiments (Sec. 5.3). Solid lines and filled regions repre- sent the mean and standard deviation respectively. Control- Net plug-in (Zhang et al., 2023b) is zero-initialized, result- ing in a lower starting point.
GLA GCN (ICCV'23)
243
44.4
34.8
D3DP' (ICCV'23)
28.7
HOT (CVPR'24) '24)
243
39.0
FINEPOSE (CVPR'24)
243
40.2
32.8
FINEPOSET (CVPR'24)
25.0
VIBE (CVPR'20)
16
65.6
41.4
MEVA (ACCV)20)
90
76.0
$3.2
MODEL-BASER
TCMR (CVRR22))
16
73.6
$2.0
MPS-NET (CVPR'S 22)
16
69.4
47.4
(2CCV'22)
35.5
GLoT (ICCV"23)
16
67.0
46.3
(ICCV'23)
33.6
ExTPoSE
16
43.5
27.2
(-2.9%)
(-19.0%)
44
space. Despite its simplicity, qualitatively tested it and found that the lifting-baser based method falls short in the mesh reconstruction application and robustness on in-the-wild data. For respective quantitative results, please see App. C.
HMRZ.0 .0 (T-1)
PAIMP (mm)
TokanHMR (T-1)
ExtPose (T-1)
40
(T-4)
ExtPose (T-6)
C. More Experimental Results
38
VIMO (T=16)
We present additional experiments with detailed captions, including quantitative results: SOTA comparison on the Human3.6M dataset (Tonescu et al., 2013) (App. C), con- vergence plot for hand experiments (Fig. 5), inference latency with extended attention (Fig. 6), and qualitative
WHAM (T=
34
16
Latency (ms)
results: with imperfect 2D poses (Fig. 7) and challenging failures for future work (Fig. 8).
Figure 6: The Efficiency-Acuuracy tradeoff curve on the 3DPW dataset, i.e. Latency (ms)-PA-MPJJPE (mm). The per-frame computation time (running time) of core modules is measured with batch frames of 1024 on one A GPU. For the VIMO (Wang et al., 2024) result, time- consuming camera estimation excluded. Please kindly note that the horizontal "Latency" axis itself does not scale proportionally.
Accuracy-efficiency tradeoff. Fig. 6 that the introduction of the 2D pose branch will roughly double the amount of data that needs to be pro- cessed; as keeping the total number of frames (batch size x sequence length) unchanged, the latency load brought by is not as obvious as it seems. Despite increasing computation, the overall efficiency is still acceptable to real-time.
worth mentioning in
13

---

EXTPOSE: Robust and Coherent Pose Estimation by Extending ViTs
ViTPose
Ground truth
Ours
ViTPose
Ground truth
Ours
Front
Side
Front
Side
Front
Side
Front
Side
Domino
omino
Figure 7: Extended cross-modal attention can correct errors in ViTPose input.
GT 2D
ViTPose Ours front Ours side
GT 2D
ViTPose
Ours front
Ours side
Missing
2D detection
Ambiguous
occlusion
Blur
Figure 8: Failures on include missing 2D keypoint detection, ambiguous occlusion, and blurred image. These issues may be mitigated by leveraging more 2D prior knowledge, e.g. SAM segmentation (Ravi al., 2025).
14


Reviews:
[{"review_id": "ea65xsRRRk", "reviewer": "Reviewer_uud2", "summary": "This work presents ExtPose, a ViT-based framework that extends a pre-trained ViT backbone to better handle image alignment and temporal coherence by introducing the following: 2D skeleton images as additional input, cross-modality interaction, and cross-frame interaction. Even though the proposed methods can be implemented efficiently, validation in multiple downstream tasks shows that ExtPose can outperform the baseline.\n\n\n## Update after Rebuttal\n\nAfter the rebuttal, the contribution of the paper seems solid and the performance of the proposed method is strong. Therefore, I have raised my score to 3, and I lean towards accepting this paper. However, I strongly recommend that the authors incorporate the feedbacks from the reviewers.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable feedback and address the concerns below:\n\n**[Method 1] Pose-pose attention mask in Eq. (11)?**\n\n**A:** Really thanks for your careful inspection! There is indeed a typo in Eq. (11). As described in the text Ls. 223-229 right above Eq. (11), the lower right quadrant pose-pose attention should also be a full 0 matrix. This is to focus on temporal modeling only within the respective modal branch.\n\n**[Question 1.1/Exp. 1] In Fig. 4, the highlight in the left corner indicates positional bias?**\n\n**A:** This is really an insightful observation! The phenomenon is also observed in other ViT-based methods like HMR2.0. Indeed, we speculate this is expected as the image corner and boundary may serve as helpful references for global/absolute positioning and final SMPL parameter regression. Thus, there are highlights indicating spatial awareness and being crucial for accurate pose estimation. \n\n**[Question 1.2/Exp. 2] How to interpret mutual attention of two branches from Fig. 4?**\n\n**A:** Thank you for highlighting the unclear points!\n\nColumn meanings: In the $3^{rd}$ column “Attention weights,” row $x$ inside the red box shows the flattened attention between point $x$ in the image and the entire 2D pose image. The red box in the $2^{nd}$ column “Attn maps” averages across the row (_i.e._, all points in the image branch) and reshapes to display the image branch's attention to the 2D pose for information gathering. The green boxes depict the reverse process: where the 2D pose focuses on the image.\n\nThus, the claim is supported by **keypoint** emphasis in the upper figure and **extensive** attention across the lower figure. Specifically, the upper figure in the middle column highlights localized 2D keypoint locations, explaining improved alignment; the lower figure reveals that 2D pose tokens focus on **not only the central foreground hand but also the background** (_e.g._, wrist in the green box, in Ls. 399-401 right), collecting RGB info and aiding 3D lifting. This broader focus helps estimate depth using the additional contextual info.\n\nWe will supply Fig. 4 with more details in our revision.\n\n**[Question 2] Comparison to lifting-based methods, specifically FinePOSE the only one better?**\n\n**A:** Thanks for raising this point! Indeed, lifting and our work share similarities of supplementing models with 2D poses; both are evaluated with JPE-type errors.  However, some critical protocol differences make the two lines of work difficult to compare directly.\n- Firstly, the target outputs in lifting are only skeleton joint coordinates.  In model-based human mesh reconstruction (HMR) which also estimates shape, the output is SMPL parameters or joint rotations.  The paradigm of direct coordinate regression has more advantage in body position prediction than SMPL-based methods due to the bottleneck of SMPL representation (Yu et al., 2023), thus tending to have a lower MPJPE (Tab. 8).\nThe _MPJPE_ metric has different preferences of different **pose representations**. \n- Secondly, FinePOSE (and other SOTA lifting methods like D3DP) tend to use **GT 3D** keypoints for final aggregation of its multiple predictions; this is the case of their SOTA performance of 25.0mm in _PA-MPJPE_.  Without GT 3D keypoints, their performance drops to 32.8mm (vs. our 27.2mm shown in Tab. a $2^{nd}$ row). Please find more details of the J-Best and J-Agg metric in the D3DP paper.\n- Thirdly, lifting-based methods significantly benefit from using a larger **window size** (number of frames), specifically 243, due to their operation in a lower-dimensional space. This advantage is evident as shown in the 3$^{rd}$ row of Table a, where a reduction in window size to 16, typical of model-based methods constrained by available GPU memory, results in a noticeable increase in PA-MPJPE (Pavllo et al., 2019; Zhang et al., 2022).\n\nTo make a more equal comparison, we add ablation studies in Tab. 5 and Ls. 382-384 right & 412-414 left: our ExtPose using both the image and 2D pose outperforms either the image or 2D pose branch alone (_i.e._ 2D-to-SMPL lifting) in our setting. We will add the context and description in the revision.\n\n**Table a. Comparisons with the SOTA lifting-based method on the Human3.6M dataset.** “Agg.” stands for aggregation. With **2)** no GT poses and **3)** a same window size of 16, ours outperforms FinePOSE.\n \n| Method | #Frames | MPJPE | PA-MPJPE |\n| --- | :---: | :---: | :---: |\n| FinePOSE* (GT Agg.) \t| 243 | 31.9 | 25.0 |\n| FinePOSE \t\t\t| 243 | 40.2 | 32.8 |\n| FinePOSE\t\t\t| 16 | 50.4 | 41.0 |\n| **ExtPose** \t\t\t| 16 | 43.5 | 27.2 |\n\n**References:**\n- Yu et al. Overcoming the trade-off between accuracy and plausibility in 3D hand shape reconstruction. CVPR 2023.\n\n**Other writing suggestions** will be revised accordingly, and thanks for bringing these to our attention.\n\nPlease feel free to point out any unclear issues, as we value your feedback highly and are always ready to discuss them further.", "claims_and_evidence": "The effectiveness of the presented approach has been validated by the evaluation of multiple downstream tasks, improving the performance over the baseline.", "methods_and_evaluation_criteria": "For most of the parts, the explanation of the method is easy to understand, however, I am quite confused of how the attention mask operates for the operation \"Cross-Frame Attention\" in Section 5.4.\nSpecifically, why is the pose-pose feature attention part added with a value of one whereas the image-image feature attention part is added with zero? I would like an extra explanation or justification for this choice.", "theoretical_claims": "There do not seem to be any theoretical claims in this work.", "experimental_designs_or_analyses": "The visualization and analysis of the cross-modal attention do not seem to be very convincing. Specifically, the following are some of the questions I would like the authors to clarify:\n\n1. Why is the attention map of the first element (the top left position) always show dominant weights in all four quadrants? This seems to show some kind of positional bias.\n2. The patterns in the attention weights do not seem very informative. I do not understand how this attention weight visualization can be connected to the authors' claim that \"The image stream queries clear location and structure information provided by the 2D hand skeleton image (red box) while the 2D pose stream attends to missing RGB information across the hand and background\", as it simply seems to be attending to most of the parts in the middle of the image, where the 2D pose skeleton will mostly appear.", "relation_to_broader_scientific_literature": "This work directly builds upon a recent framework and as the proposed framework is easy to implement, I believe that a similar approach can be applied to future improvements.", "essential_references_not_discussed": "I did not find any crucial references not discussed in the manuscript.", "other_strengths_and_weaknesses": "Although the technical contribution is not novel or significant, I believe that introducing the concept of 3D attention together with the 2D pose map is interesting and effective for improving the overall performance.", "other_comments_or_suggestions": "1. There is a missed period on L203, \"This underscores ViT’s generalizability and scalability while demonstrating that 2D pose features \n extracted this way align well with image features\".\n\n2. I would like to encourage the authors to simply describe Human Pose Estimation and Hand Pose Estimation in their full names. Although they have provided a subscript that HPE will be used interchangeably, I do not find a significant reason to do this, and it only increases confusion.", "questions_for_authors": "1. I would like the clarification of the visualization of the attention maps in Figure 4, as I have explained in \"Experimental Designs Or Analyses*\n\".\n\n2. I have a question regarding the evaluation shown in Table 8, which was my primary justification for giving a \"Weak Reject\" as my initial rating for this paper. It seems that the authors are considering Lifting-based methods not as their primary competitors but I did not understand why this is the case. As this framework also utilizes 2D poses, I am not fully sure why methods such as FinePose should not be discussed, even though they perform much better than the proposed method ExtPose. If the authors can justify why these methods should not be compared directly (i.e., additional ground truths, and different training data), I would like to raise my score.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lFBdS59AeA", "reviewer": "Reviewer_Sqcd", "summary": "The paper proposes a 3D pose estimation algorithm that simultaneously considers 2D pose information and temporal information. Through parameter sharing and multimodal data alignment strategies, the algorithm is able to accurately estimate 3D pose. The authors validate their proposed approach through extensive experiments, achieving state-of-the-art (SOTA) results across nearly all evaluation datasets and metrics.\n\n## update after rebuttal\nThanks for the clarification. After reviewing all the feedback and rebuttals, I’ll be keeping my score as is. However, I hope the authors can address the concerns raised by others and include the necessary details in the revised version.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's constructive comments and address the concerns below:\n\n**[Strengths & Weaknesses 1/Exp. 1] The effect of noise in 2D pose detection?**\n\n**A:** Thanks for your great insight in studying the effect of 2D pose quality on performance!\n- Firstly, in Fig. 6, we **qualitatively** highlight when erroneous 2D poses occur (_e.g._ left middle and right ring fingers), the original image branch assists in maintaining accurate predictions, _i.e._ correcting the 2D pose inaccuracy as the review rightly inferred. But indeed, there are failures where the prediction still adheres to the wrong 2D pose guess, particularly when the image information is hard to perceive in challenging cases, _e.g._ occlusion and blur in Fig. 7.\n- Secondly, to **quantitatively** assess the effect of 2D pose noise, we introduce Gaussian noise with increasing standard deviations to the 2D pose while following Gu et al. (2024) to adjust the confidence score accordingly to reflect the increasing prediction uncertainty. It is worth mentioning that not only the coordinate prediction but also confidence as additional information are drawn in the 2D pose image. Table b shows the benefit from auxiliary 2D poses diminishes progressively with the increase in noise levels. When the confidence becomes low and the keypoint is displayed as transparent in the 2D pose image, the model does not use much of 2D poses and degenerates to the original ViT model (instead of collapsing), indicating the robustness to 2D pose noise of varying levels.\n\n**Table b. The effect of synthetic 2D pose noise on the 3DPW image dataset.** ExtPose makes use of high-quality 2D pose detection while maintaining robustness to pose noise.\n| Noise (pix) | PA-MPJPE | MPJPE |\n| --- | :---: | :---: |\n| std = 0\t| 35.5 | 55.6 |\n| std = 4\t| 38.5 | 62.3 |\n| std = 8\t| 43.3 | 69.0 |\n| std = 12\t| 44.2 | 69.5 |\n| HMR2.0 \t| 44.4 | 69.8 |\n\n**References:**\n- Gu et al. On the calibration of human pose estimation. ICML 2024.\n\n**[S & W 2/Sup. 1] Explanation for the failure in the Sup. video?**\n\n**A:** Thanks for the insightful observations. Yes, although our method shows significant enhancement in robustly aligned and temporally coherent pose estimation, challenges remain. For instance, as mentioned earlier, Fig. 7 showcases some really challenging and long-tail scenarios like ambiguous occlusion and blur. Analogously, the hand misalignment in the Sup. video frame, we speculate, likely stems from 2D pose estimation difficulties and, therefore, are not confident to provide any useful cues (not depicted in the pose image). Such out-of-distribution cases can lead to diverse error predictions across different models. A thought of future work to mitigate the issue may be to leverage more 2D prior knowledge, _e.g._ SAM2 segmentation.\n\n**Other writing suggestions** will be fixed accordingly. Thanks!\n\nWe are confident that your input will greatly improve our manuscript. Should there be any unresolved issues in our feedback, please do not hesitate to inform us, as we are committed to continuous improvement and welcome further dialogue.", "claims_and_evidence": "The authors identify three key design factors that significantly impact the final results: consistency in the representation space, the use of parameter sharing, and the hierarchical attention fusion strategy. These factors are validated in the subsequent experiments.", "methods_and_evaluation_criteria": "The metrics used in the paper are standard, and when comparing with related work, reasonable common metrics are employed.", "theoretical_claims": "The theoretical explanations in the paper are reasonable, and the presented results align with the theoretical background.", "experimental_designs_or_analyses": "The experimental design in the paper is quite reasonable, and the algorithm's effectiveness is validated from different dimensions. Since I am not an expert in this field, I have one question: the method requires additional 2D pose as input. If the 2D pose detected by the algorithm contains significant noise, can it be corrected in the subsequent stages of the algorithm?", "relation_to_broader_scientific_literature": "The paper observes the limitations of previous work in addressing temporal images and proposes corresponding strategies to resolve these issues. It also introduces 2D pose as a prior, further improving accuracy. This is a contribution to the research direction in this area.", "essential_references_not_discussed": "Yes, this paper has discussed enough related references.", "other_strengths_and_weaknesses": "The contributions presented in the paper are validated in subsequent experiments, and compared to previous work, the results achieve the best performance across nearly all metrics. I believe this is a meaningful contribution. However, I have two concerns: first, the reliance on the output 2D pose, could you clarify how much this dependence impacts the results? Second, the discussion on failure cases and limitations, some frames in the supplementary materials do not produce a reasonable mesh, with a significant deviation from the image. Could you provide an explanation for this?", "other_comments_or_suggestions": "A small issue is that the paper repeatedly emphasizes \"extending\" ViT or poses. Could the title also use \"extending\" instead of \"expanding\" to maintain consistency?", "questions_for_authors": "As discussed above, I hope the authors can address the issues I have mentioned above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KRMvXFiFtD", "reviewer": "Reviewer_JotU", "summary": "The authors propose ExtPose: a robust and Coherent pose estimation by refining a ViT-based HPE. Several contributions are proposed in this paper: 1) 2D pose and image information are combined are combined in a ViT model, 2) Temporal context is integrated. The resulting model is compared to the SOTA models using 3DPW dataset. The proposed model outperforms the SOTA models with a large margin for some metrics. The ablation study shows that each component of the proposed model is effective. The paper is well written and the experiments are well conducted. This paper is a good contribution to my point of view.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewers' valuable efforts and their recognition of our work! If you have any other questions, we are always here, and you are welcome to discuss with us.", "claims_and_evidence": "- The authors propose to use a skeleton-based image representation in a ViT model (inspired from Zhang et al.). This is a good idea to combine 2D pose and image information in a ViT model.\n- Taking into account the temporal context is done by spatiotemporal attention on features between frames.\n- Experiments have been conducted on several datasets and the proposed model outperforms the SOTA models.\n- The experiments show that the proposed model outperforms the SOTA models with a large margin for some metrics.\n- The ablation study shows that each proposed contribution is effective and improves the performance of the model.", "methods_and_evaluation_criteria": "/", "theoretical_claims": "/", "experimental_designs_or_analyses": "/", "relation_to_broader_scientific_literature": "/", "essential_references_not_discussed": "/", "other_strengths_and_weaknesses": "/", "other_comments_or_suggestions": "/", "questions_for_authors": "/", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dnn84XNO7z", "reviewer": "Reviewer_AWPd", "summary": "This paper proposes a ViT-based model, ExtPose, for 3D single-human pose estimation. It can handle both image inputs and video inputs. Besides, it can utilize strong 2D HPE models (e.g. ViTPose) to enhance the 3D meshes. The overall performance is great - it achieves remarkable error reduction on existing benchmarks. ##update after rebuttal", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere appreciation for the helpful reviews and tackle the concerns below:\n\n**[Design 1] Accuracy-efficiency tradeoff curve?**\n\n**A:** Thanks for this suggestion. Table 9 in our submission already follows Xu et al. (2022) and Wang et al. (2024) to demonstrate computational efficiency. For clarity, we plot them along with accuracy in Fig. a ([anonymous link](https://anonymous.4open.science/r/ExtPose_ICML25/icml2025_extpose_2283.pdf)).\nAs the amount of attention computation increases with the addition of 2D pose modality and the number of frames, the error does decrease steadily, but the accuracy gain brought by the increase in computation gradually diminishes until the $T=16$ plateau, so it is necessary to weigh the actual accuracy against the efficiency required. It is worth mentioning that the introduction of the 2D pose branch will roughly double the amount of data that needs to be processed; as keeping the total number of frames (batch size$\\times$sequence length) unchanged, the latency load brought by $T$ is not as obvious as it seems. Despite of increasing computation, the overall efficiency is still acceptable to real-time. In this work, our primary goal is to achieve high accuracy, and we also mentioned that efficiency could be improved in future work.\n\n**References:**\n- Wang et al. YOLOv10: Real-time end-to-end object detection. NeurIPS 2024.\n- (VIMO) Wang et al. TRAM: Global trajectory and motion of 3D humans from in-the-wild videos. ECCV 2024.\n\n**[Design 2] ExtPose underperforms lifting-based methods?**\n\n**A:** Thanks for your careful review. A short answer is, model- and SOTA lifting-based paradigms are usually not fairly compared due to **1.** different representations, **2.** the use of 3D poses, and **3.** window sizes. We outperform them in a similar setting (R1 uud2-Tab. a) and our ablation study (Tab. 5). We kindly refer the reviewer to check our response to **R1 uud2-Question 2** in detail. We will also make the point clearer in the revision.\n\n**[Design. 3] Cross-frame novelty.**\n\n**A:** Thanks for your recommendation and comments. Our contribution lies in inspecting and effectively solving the challenges of 2D misalignment and temporal inconsistency of current ViT human pose estimation models with a simple and elegant solution. We devise a unified 2D pose representation with attention extended across the modality and frame to derive a robust video ViT model.  The attention is thus implemented “simply and straightforwardly”, which aligns with techniques applied in other fields such as Multiview 3D generation, as the reviewer kindly mentioned. We will also include it in the reference.\n\nWe believe your contribution will definitely help enhance our work. Please feel free to let us know if there are still concerns not addressed in our feedback, and we are more than happy to communicate.", "claims_and_evidence": "The claims are well-supported.", "methods_and_evaluation_criteria": "The benchmarks and metrics used are standard.", "theoretical_claims": "I do not see a problem.", "experimental_designs_or_analyses": "The experiments are well-designed and persuasive. ExtPose achieves a remarkable error reduction compared to existing state-of-the-art methods. However, I think the reviewers should show the accuracy-efficiency trade-off curve in the paper - from existing tables, I cannot tell if the performance mainly comes from more computation. Also, I noticed ExtPose still cannot outperform lifting-based methods in Table 8. I wonder the reason for that. Another minor novelty issue could be that cross-frame attention is a common technique in other areas [1].\n\n[1] Gao, Ruiqi, et al. \"Cat3d: Create anything in 3d with multi-view diffusion models.\" arXiv preprint arXiv:2405.10314 (2024).", "relation_to_broader_scientific_literature": "Human Pose Estimation/AR/VR.", "essential_references_not_discussed": "I do not see a problem.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This work introduces a 3D pose estimation method considering 2D pose information together with temporal information. The proposed method can estimate 3D pose reliably, by sharing parameters and aligning multi-modal data. The authors have done extensive experiments showing the efficacy of the method. Overall, the method is interesting and reasonable. All the reviewers recommend acceptance. AC agrees with reviewers.
Decision: Accept (poster)