Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
Authors: Qiuxia Lin
Rongyu Chen
Kerui Gu
Angela Yao
Venue: ICML 2025 poster (2025)
Keywords: Test-time Adaptation
3D Human Pose Estimation
Primary area: applications->computer_vision

Abstract: This work highlights a semantics misalignment in 3D human pose estimation. For the task of test-time adaptation, the misalignment manifests as overly smoothed and unguided predictions. The smoothing settles predictions towards some average pose. Furthermore, when there are occlusions or truncations, the adaptation becomes fully unguided. To this end, we pioneer the integration of a semantics-aware motion prior for the test-time adaptation of 3D pose estimation. We leverage video understanding and a well-structured motion-text space to adapt the model motion prediction to adhere to video semantics during test time. Additionally, we incorporate a missing 2D pose completion based on the motion-text similarity. The pose completion strengthens the motion prior's guidance for occlusions and truncations. Our method significantly improves state-of-the-art 3D human pose estimation TTA techniques, with more than 12% decrease in PA-MPJPE on 3DPW and 3DHP.

Semantics-aware
Test-time Adaptation for 3D Human Pose Estimation
Qiuxia Lin Rongyu Chen
Kerui Gu
Angela Yao
Abstract
This work highlights semanties misalignment in 3D human pose estimation. For the task of test- time adaptation, the misalignment manifests as overly smoothed and unguided predictions. The smoothing settles predictions towards some aver- age pose. Furthermore, when there are occlusions truncations, the adaptation becomes fully un- guided. To this end, pioneer the integration of semantics-avee aware motion prior for the test- time adaptation of 3D pose estimation. We lever- age video understanding and a well-structured motion-text space to adapt the model motion pre- diction adhere video semantics during test time. Additionally, we incorporate missing 2D pose completion based on the motion-text similar- ity. The pose completion strengthens the motion prior's s guidance for occlusions and truncations. Our method significantly improves state-of-the- art 3D human pose estimation TTA techniques, with more than 12% decrease PA-MPJPE on 3DPW and 3DHP.
Ours CycleAdapt
Ours CycleAdapt
Figure 1: We provide two examples to illustrate the issues in TTA methods. Compared to CycleAdapt (Nam et al., 2023), our method enables accurate bending of the legs in the "climbing-the-stae scenario and "walking" in the occluded scenario, demonstrating the effectiveness of incorporating semantics-awee aware motion information for both visible and occluded keypoints.
1. Introduction
3D human pose estimation from images and videos is ap- plicable for many scenarios, including human-come nomanr interactions (Zheng et al., 2023), robots (Gong al., 2022), and digital human assets (Moon et al., 2024). is challeng- ing task because 3D ground truth labels are hard to acquire for arbitrary visual data. Common methods are trained on MoCap datasets (Mahmood et al., 2019), which provide accurate 3D labels but are limited pose and appearance. As such, these methods may easily fail on in-the-wild im- ages during inference (Guan et al., 2021: Zhang al., 2020; Weng et al., 2022; Nam et al., 2023).
tion models to in-the-wild videos. A practical alternative improve predictions is Test-Time Adaptation (TTA). TTA directly uses the (unlabeled) target sequences to fine-tune the pre-trained model during inference. Often, pre-trained models exhibit a misalignment between their projected key- points and 2D image evidence (Chen et al., 2025a). Based on this result, previous TTA methods (Guan et al., 2022: Nam et al., 2023) usually penalize the distance between the 2D- projections and 2D evidence provided by either ground truth or estimated 2D poses. However, such scheme has two main challenges (Chen et al., 2023): 1) Depth ambi- guity, which arises because numerous 3D pose solutions correspond to the same 2D pose. This ambiguity may im- prove 2D alignment but worsen 3D accuracy and plausibility.
This paper tackles the adaptation of 3D human pose estima-
'Department of Computer Science, National University of Singapore, Singapore. Correspondence to: Qiuxia Lin <qi- wxia@comp.mus.edu.s.
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
2) Missing 2D keypoints, e.g,, under occlusions and trunca- tions. These keypoints have no guidance for improvement.
At the heart of our method is semantics-aware motion prior that supports test-time adaptation for 3D human pose estimation. The prior effectively complements 2D projected supervision and remedies werrsmoothingg When 2D key- points are available, motion-text alignment greatly reduces the ambiguous 3D solution space; when they are unavailable, text-aligned motion prediction provides useful 2D projected poses to complete the missing information. In summary, we
To alleviate the above issues, several works (Guan et al., 2022; Nam et al., 2023) leverage temporal information from the target videos. One strategy is to limit the prediction velocity, as a weak regularizer to narrow the feasible solu- tion space (Guan et al., 2022). Another is motion denoising, which enforces prior that predictions should remain close to the original or neighboring predictions (Nam et al., 2023). These improvements are rooted in the fact that denoising and smoothing the sequential predictions can filter out poor results. However, temporal smoothing may settle the se- quence towards an average pose. Consider the upper panel of Fig. 1. The state-of-the-an CycleAdapt (Nam et al.,, 2023) aligns the 3D mesh to the visual 2D evidence, but results in the wrong pose where the legs are straight. With some semantic knowledge of the man's ss activity (climbing stairs), it is clear that his knees should be bent.
Highlight the interesting and significant problem of mo- tion semanties, which existing TTA literature ignores exacerbates;
Propose novel high-level motion semantics prior for TTA, by leveraging a motion- longoanlanguuge model;
Propose a text-aligned motion predictions to complete 2D poses in occlusion or truncation cases, significantly improving performance up to 12.5% % PA-MPJPE im- provement on these challenging cases.
With this motivation in mind, we develop a semantics-aware motion prior to guide 3D predictions. We leverage vision- language models to describe the action of a given segment. This action text is projected with the predicted motion into motion and text aligned representation space. Specifi- cally, we use MotionCI.IP (Tevet et al., 2022), with mo- tion auto-encoder that aligns motion manifold with the semantically-strrcuured CLIP space (Radford et al., 2021). In our TTA, we align the estimated 3D pose sequence with the semantically identified action through a dedicated regu- larizer. In the "climbing -th--stairs"" example, the predicted motion originally resembles "sliding", but after introducing our motion-text alignment, the resulting poses are adapted to have bent knees and match the true motion of stair climbing.
2. Related Work
2.1. Test-time Adaptation in 3D Pose Estimation
Generalizing 3D human pose models to in th-the-wild data is challenging due to distribution shifts, driving interest in test-time adaptation methods with 2D information. Ini- tially, the methods (Zhang et al., 2020; Guan et al., 2021; 2022; Weng et al., 2022) rely on ground truth 2D poses to rectify 3D predictions along with the proposed adaptation techniques. For instance, ISO (Zhang al., 2020) lever- ages invariant geometric knowledge through self-supervised learning. BOA (Guan et al., 2021) introduces bilevel opti- mization to better integrate temporal information with 2D weak supervision. DynaBOA (Guan et al., 2022) extends BOA by using feature distance as a signal for domain shift with dynamic updates. DAPA (Weng et al., 2022) employs data augmentation by rendering estimated poses onto test images, guiding the model toward the target domain.
TTA can easily be influenced by challenging samples. For example, when there are occlusions, or truncation, there is no 2D evidence available for adaptation. Under such circum- stances, the model lacks guidance on what to predict. The lower panel of Fig. 1 shows an example of a person walk- ing, with the lower body occluded for the entire segment. CycleAdapt predicts a motion with a static lower body. In such cases, where 2D evidence is fully lacking, we observe that previous alignment in the motion-text space alone is insufficient for adaptation. This motivates us to strengthen the supervision by storing 2D predictions with high quality as exemplars and regard them 2D pseudo labels in the subsequent optimization. Specifically, for the keypoints not provided with 2D evidence, we examine the similarity be- tween the predicted motion and its text label in the feature space; if it exceeds a certain threshold, these keypoints will be completed by the 2D projection of model predictions. With 2D missing keypoints filled by text-aligned motion, we can offer stronger supervision on occluded or truncated body parts.
However, ground truth 2D poses are not always available in many real-world scenarios. Similar to our method, Cy- cleAdapt (Nam et al., 2023) operates on estimated 2D data and introduces a cyclic framework, where the motion denois- ing module smooths 3D pseudo labels to mitigate the impact of noisy 2D inputs. Nonetheless, the inherent 2D-to-3D am- biguity still significantly degrades model performance, and unguided predictions often occur when 2D inputs are un- reliable. Our method addresses this by leveraging explicit semantic understanding, i.e., actions, to guide the model in predicting specific motion patterns. This is the first ap- proach to emphasize high-level semantic information to
2

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
reduce the 2D-to-3D solution space in test-time 3D human poses jAD:
pose estimation.
jnD = I[) = ((MMPL.
(2)
2.2. Human Motion Priors
Leveraging motion prior as supervision is commonly ap- plied in video-based human pose estimation due to the lack of 3D annotations with RGB input. Trained on large and diverse human motion database AMASS (Mahmood et al., 2019; lonescu et al., 2013), the human prior provides a regularized motion space. Previous works mainly drag the predicted or generated motion to the regularized mo- tion space to achieve the refinement (Kocabas et al., 2020; Shin al., 2024; Rempe et al., 2021), infilling (Yuan al., 2022; Zhang et al., 2021), denoising (Nam et al., 2023) goals. Specifically, VIBE (Kocabas et al., 2020) extends VAE (Pavlakos et al., 2019) and adversarial (Kanazawa et al., 2018) feasible prior from pose into motion space to improve temporal continuity of movement. GLAMR (Yuan et al., 2022) learns a VAE motion infiller to infill motion where the moving human is occluded. CycleAdapt utilizes the prior to denoise 3D meshes and improves human pose estimation during test-time with noisy 2D poses. However, those applications of human motion prior only optimize the predicted meshes of the segment to be valid motion without consideration of the motion semantics. Compared to other works, we are the first to split regularized motion space according to text labels extracted from the video segment to benefit motion prediction.
where II is the camera projection function. Because method does not involve direct operations on 3D poses, we will use J to denote 2D poses in the following descriptions.
3.2. Motion-language Model
A motion-language model integrates human motion and natural language modality. It utilizes pre-trained language models (Radford et al., 2018; Kenton & Toutanova, 2019), to incorporate rich semantic space for improved motion rep- resentation. This is achieved by training with motion-text data, which enables tasks like text-conditioned motion gener- ation (Zhang et al., 2024; 2023) and motion captioning (Guo et al., 2022; Jiang et al., 2023). Our focus is leveraging mnotion-lnnguee
model to guide motion semantics for un- labeled test videos in TTA. We choose MotionCL.IP (Tevet et al., 2022) for its text-aligned motion representation, which is optimally suited to our task.
MotionCLIP maps human motion sequences onto the CLIP space (Radford et al., 2021). It positions semantically simi- lar motions closer together while maintaining clearer sepa- ration for distinct motions in the manifold. The motion-tes text affinity is measured using cosine similarity between the text CLIP feature from fCLIP and the motion feature from the motion encoder MMOTIO;
3. Preliminaries
3.1. Problem Setup
sim (s, m) = cos (FCLIP(s), AOOTOON (m)).
(3)
Given an arbitrary video, for each RGB frame X, we aim to predict the 3D human poses in joint coordinates J3D E k where K is the number of keypoints. Following previous works, we utilize SMPL model (Loper al., 2023) which takes as input the joint rotation 0 E ?2 and shape 3 E R 10 parameters along with translation t E IR3 w.r.t.
The motion-text pairs (s. m) are obtained from labeled mo- tion dataset (Punnakkal et al., 2021), where s is text label and m = {PI. ... . is motion segment of T = 60 frames. Each pose p; 24x6 is represented in 6D for- mat (Shi et al., 2019), including global and local rotations.
the camera to produce 3D poses, formally
JaD = SM,L(,,,
(1)
4. Method
In our method, the predicted 0, 3 and are given by a hu- man mesh recovery network (Kanazawa et al., 2018) which has been trained on the Human3.6m 3.6M (Ionescu et al., 2013) dataset following the setting of (Guan al., 2022: Zhang et al., 2020; Nam et al., 2023). We denote this network as our 3D human pose estimation model ННРЕ.
Fig. 2 shows the overall framework. It consists of semantics- aware motion prior (left panel) and 2D pose update module (right panel). Using semantics-aware motion prior (See. 4.1), predicted motion and text label are aligned in a shared space of motion-language model, supplementing other losses to enhance semantics awareness. In the 2D pose update module (Sec. 4.2), 2D poses are updated by EMA (right top panel) while the missing 2D poses are filled with text- aligned motion prediction (right bottom panel). This module strengthens the semantics-awarc motion prior on occlusion. Details of each module are introduced as follows.
During inference, the model is adapted with the help of 2D pose J2D from an off-tff estimator (Cao et al., 2017)
to improve the prediction of 3D poses. The predicted 2D
poses j2D can be obtained by projection of predicted 3D
3

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
1)
3D HPE Model
BEteT Beser
Motion Encoder
Test Image X
3D HPE Model
(0,.0,
Predicted 1
EMA
2D Pose J
2D Pose
OLIOT BEL-T
Predicted Fit-r 20 Label Picer
2)
Teat Video Segment X
3D
Model
12 8 BB
Motion Encoder
Zm
AzD
Ssmorth
Lalign
Cijmbing- she-ssairs
Prodicted
Emttadding Space
J
cos x;)
J² R
Text Label
CLIP
Motion
Text Label
CLIP
2
Text features
Figure 2: Overall framework: we perform model fine-tuning and 2D pose update in each epoch. Model Fine-tuning (Left); The extracted video segment and its text label will be used to update μHPE with 2D projection, alignment, and smooth losses. 2D Pose Update (Right): After model fine-tuning in each epoch, all images in the current test video are processed through (HPP to obtain predicted 2D and update the 2D keypoints using EMA. Subsequently, all video segments checked to fill in any missing 2D keypoints by verifying conditions of motion-text alignment.
4.1. Semantics-aware Motion Prior
in each image is obtained, we aim to align the the projected motion and text features within the shared space Motion- CLIP. To process video segments for training, we divide the video into segments and only keep those where all images share the same text label. This to avoid misalignment caused by mixed motions much possible. To further improve alignment, we adopt weighted sampling and pri- oritize those segments whose motion predictions deviate from the true semantics. Specifically, the sampling weight is defined as I minus cosine similarity between predicted motion and its text label in MotionCLIP.
Since the 2D-to-3D space is highly ambiguous, the model supervised by 2D projection loss still suffers from depth ambiguity. However, when video segments are available, contextual semantic information about human motion is evi- dent. We therefore can leverage it to reduce the 2D-to-3D ambiguity by specifying a pose space with clear semantic meaning. So aside 2D projection and temporal losses, we also use motion-text-alignee representation space learned by MotionCLIP (Tevet et al., 2022) to ensure the predic- tions grounded on the video semantics. This is achieved by aligning the features of predicted motion and text labels of motion semantics in the shared embedding space.
Motion-text feature alignment. Given batch of video segments, we pass them through our 3D human pose estima- tion model fHPEE to obtain pose motion predictions. For one single video segment X we first retrieve the exemplar motion segment in the motion dictionary (Punnakkal et al., 2021) based on the text label 8;. Next, we change pose prediction 0w.+ outputted from yHPF to 6D format PIG+T and replace the global rotation with that of the retrieved motion segment obtain P This focuses on updating only the local poses that are more text-relevant while avoid- ing the optimization towards specific global orientations present in the motion training data (Punnakkal et al., 2021). As shown in Fig. 2 (left), the PlEsTT is forwarded into the motion encoder /OOTOON, while text label s; is forwarded into the CLIP model fCLiP Semantics-aware motion prior is applied by motion-text feature alignment the shared embedding space:
Text labeling. We leverage Vision-Language Model (VLM) (Achiam et al., 2023) for human motion descrip- tion and matching with motion dictionary defined in (Pun- nakkal et al., 2021). In general, the inputs contain prompts specifying requirements, a video segment, and motion dic- tionary. The outputs are text labels, like running, sitting with feet crossed, or walking upstairs, which can be assigned to each frame within the video segment. We provide some examples of used text labels in our method as follows:
wa knnn
sitting with crossed standing with knees bent throw something the right hand and Malking posa with bent log and transition and walkking
Calign sim(s; P(s44)
(4)
More details about the VLM prompts and label verification
can be found in Appendix Sec. B.
where sim(..) )) is introduced in Eq. 3.
Video segment processing, After the text label for motion
The gradient is backpropagated to update the parameters of

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
the pose estimation model fHPE, Both the motion encoder not to affect the already structured feature space. With the motion feature better aligned with the text feature, the motion pre- diction can be performed based on a smaller semantic space, thus reducing the 2D-to-3D ambiguity.
Finally, we can summarize the updates of all cases as:
and CLIP model are frozen in our framework
a + J +
Jr. (1
if J * 0, if J = & sim > a,
J
Jr.
0,
if J & sim S a.
(8)
Specifically, in epoch 1, we rely on the 2D pose estimates
with uncertain values excluded. By the end of epoch 1, EMA update is applied to all estimates with predicted 2D, followed by a fill-in for some missing 2D estimates. This process is repeated the end of each subsequent epoch. We provide an illustration of 2D pose update in Fig. (right).
4.2. 2D Pose Update
In our work, we leverage a 2D estimator (Cao et al., 2017) to provide 2D pose estimates J and its corresponding visi- bility v. typically setting a threshold to exclude uncertain estimates (Guan al., 2022; Nam et al., 2023). Therefore, for the keypoint & on the test frame, the projection loss can be applied with LI distance as:
4.3. Overall Training
For each new video, we fine-tune the pre-trained model over several epochs for adaptation. The overall framework is shown in Fig. 2. The 2D pose updates (Eq. 8) and model fine-tuning are conducted separately, with the 2D poses remaining fixed during the fine-tuning process. And in each epoch, similar with (Nam et al., 2023), we have a smoothing loss with the motion denoise module:
LID Σ ok 11.x Julli Uk 1[Jx + 0}- k
(5)
Note that the projection loss is held for each frame in the video segment and the frame subscript is omitted for clarity.
Clearly, the 2D projection loss mentioned above cannot provide supervision for missing 2D keypoints, specifically 222 = 0. However, the semantic meaning of human motions can be highly evident even under occlusion or truncation by inferring from the video context. Therefore, we want to utilize the semantics-enhanced motion prediction to suggest possible 2D positions for model training. To this end, we manage fill in the missing 2D keypoinnss with text-aligned motion prediction if the motion feature has high cosine similarity with its text feature:
Smooch = 0 0|1 +
where the @ comes from the motion denoise module, and 3 is the averaged shape estimation with a sliding window operation. Together with motion-text feature alignment loss, 2D projection loss, and loss weight A, our training objective in each epoch can be summarized as:
Loverall A2D + de Caliin + Samooth
(10)
5. Experiments
JotL = J if Jr = and ssm p') > a.
5.1. Setup
where e subscript represents the epoch number. The k-tth joint of the 2D prediction Jj is assigned to the estimated 2D Je in the next epoch if the motion aligns with the text and the estimated keypoint is excluded, i.e., J = 0.
Datasets. We follow the adaptation tasks from previ- ous work (Zhang et al., 2020; Nam et al., 2023), us- ing Human3.6M (lonescu et al., 2013) as the labeled training dataset and 3DPW (Von Marcard et al., 2018) and 3DHP (Mehta et al., 2017) as the unlabeled test datasets. Human3.66 6.6 is widely used indoor dataset comprising 3.6 million images annotated with 2D and 3D labels. Despite its extensive pose diversity, the HMR network (Kanazawa et al., 2018) pre-trained on this dataset often struggles to generalize to in in-the-wild settings due to its limited variation in appearance and environmental factors. The 3DPW test set comprises 37 videos and presents a significant challenge as is an outdoor dataset featuring diverse scenarios, including subject occlusion and varied environments. The 3DHP test set includes 6 videos with both indoor and outdoor scenarios, and it presents an additional challenge by featuring some poses not seen in Human .6M. Furthermore, we validate our method on a egocentric dataset EgoBody (Zhang et al., 2022), which presents severe body truncation with missing 2D evidence.
However, the individually predicted 2D poses can still be noisy. The text-aligned motion predictions are based on each video segment and require model feedback to update temporally consistent predictions by incorporating all sug- gested 2D positions. To this end, we aim to update the estimated 2D alongside the predicted 2D keypoint, using an EMA (Exponential Moving Average) update with update factor a, given by:
jeti= - a
+ (1-
(7)
The EMA update is performed at the end of each epoch, and the updated pose Je+1 will be used as supervision for prediction jetz following Eq. 5 in the next epoch e + 1. Here, we not only update the fill-in 2D but also the original estimated 2D, as both can boost the performance.
5

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
Test-time adaptation comparison results on 3DPW (Von Marcard et al., 2018) and 3DHP (Mehta et al., 2017) based on HMR network pre-trained on Human3. 6M (lonescu et al., 2013). Our method achieves the best performance across all metries on 3DPW (Von Marcard et al., 2018) and 3DHP (Mehta et al., 2017) datasets, surpassing state-of-the-art methods.
Table 1:
3DPW
3DHР
Method
MPJPEL
PA-MPJPEL
MPVPE
MPJPEL
PA-MPJPER
Pre-trained HMR
230.3
123.4
253.4
218.5
119.6
BOA (Guan et al., 2021)
137.6
76.2
171.8
135.3
88.5
DynaBOA (Guan et al., 2022)
135.1
73.0
168.2
130.7
81.8
DAPA (Weng et al., 2022)
108.0
67.5
129.8
CycleAdapt (Nam et al., 2023)
87.7
53.8
105.7
110.3
74.4
Ours
76.4
47.2
94.0
101.3
65.1
Table 2: Test-time adaptation comparison results on Ego- Body (Zhang et al., 2022) and 3DPW (Von Marcard al., 2018) 1) based on HMR2.0 network (Goel et al., 2023). Our method achieves better performance than CycleAdapt (Nam et al., 2023).
Table
Ablation study of method components on the 3DPW dataset (Von Marcard et al., 2018) based HMR network, demonstrating the incremental improvements achieved by each component.
Alignment
EMA
Fill-in
MPJPEJ
PA-MPJPEJ
Method
EgoBody
3DPW
x
87.7
53.8
MPJPE
PA-MPIPEL
MPJPEL
PA-MPJPEJ
x
79.3
49.6
HMR2.0
118.2
69.6
80.2
53.3
/
78.0
48.7
76.4
47.2
CycleAdapt
108.7
60.4
73.7
46.8
Ours
104.6
56.1
71.2
44.3
methods: BOA (Guan et al., 2021), DynaBOA (Guan et al., 2022), DAPA (Weng et al., 2022) and CycleAdapt (Nam et al., 2023).
Evaluation metrics. We report three evaluation metries: Mean Per Joint Position Error (MPJPE) measuring the Fu- clidean distance between the predicted joints and the ground truth joints with aligned root joint; Procrustes- Aligned MPJPE (PA-MPJPE) measuring MPJPE after aligning pre- dictions with ground truth in 3D space; Mean Per Vertex Position Error (MPVPE) measuring the average error across all vertices. All results are reported in millimeters. The re- sults 3DPW contain all evaluation metrics. Since 3DHP does not provide ground truth SMPL parameters, we report only MPJPE and PA-MPJPE for this dataset.
5.3. Quantitative Results
As shown in Tab. 1, the pre-trained HMR network performs poorly on the 3DPW dataset, highlighting the limited gen- eralization ability of pre-training ios. While previous methods (BOA (Guan et al., 2021). DynaBOA (Guan et al., 2022), DAPA (Weng et al., 2022)) show improvements, they heavily rely on ground truth 2D labels and are prone to failure with noisy 2D inputs. In contrast, CycleAdapt (Nam et al., 2023) employs cyclic adaptation between the motion denoise module and the HMR network, effectively mitigating the impact of noisy 2D poses. Our method outperforms CycleAdapt by 12.9 % in MPJPE and 12.3 % in PA-MPJPE, demonstrating the effectiveness of inducing semantics-aware motion prior to improve 3D predictions. Similar promising results are ob- served on 3DHP. Specifically, our method significantly sur- passes others, achieving improvements of 8.2% % of MPJPE and 12.5 % of PA-MPJPE over CycleAdapt.
cow-variability scenar-
5.2. Implementation Details
At the start of test-time adaptation for each test video, the model parameters are initialized with the pre-trained values, following (Nam et al., 2023). We employed the Adam op- timizer (Kingma & Ba, 2014) with parameters set to betal 0.5, bela2 = 0.9, and a learning rate of 5.0e-5. cosine scheduler is used with minimum learning rate of 1.0c-6. The input images are resized to 224x224, and the frame number of each video segment is 60. We use batch size of 4 and the total training epoch is 6. The hyperparameters are A = 0.1, X2 = 0.2, o = 0.75, = 0.9. We use Openpose (Cao et al., 2017) to provide 2D poses with a confidence threshold of 0.3 (Gu et al., 2024). The compar- ison methods include oate-of-ther thet-rr test-time adaptation
We further validate our method by upgrading the backbone to the state-of-the-art HMR2.0 (Goel et al., 2023), which is trained with multiple in-the-wild datasets, on the more chal- lenging EgoBody dataset. As shown in Tab. 2, our method continues outperform Aycte with a stronger back-
6

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
Table 4: Analysis of semantics-incorroraaee strategies on the 3DPW dataset (Von Marcard et al., 2018) based on HMR network. Building on CycleAdapt (Nam et al., 2023), we design two methods to enhance pose semantics with generated motion sequences from (Jiang et al., 2023). wwwpwtoww_rampAndeee The former involves depth ambiguity when viewing people ascending stairs from behind, while the latter includes obstacles in low-lighting bar environment. Our method consistently improves across videos while others do not.
and "downtown_bar highlight different challenges.
www.town.ranpAndee
downtown. bbar
All videos
Method
MPJPEL
PA-MPJPEL
MPJPEL
PA-MPJPEJ
MPJPE
PA-MPJPEL
CycleAdapt (Nam et al., 2023)
94.5
60.4
97.3
65.0
87.7
53.8
+Motion discriminator (Kocabas et al., 2020)
85.7
52.9
117.9
72.2
115.0
69.1
+Unpaired local poses (Jiang al., 2023)
82.3
50.6
102.6
71.7
88.2
54.5
Ours
77.8
49.2
83.8
58.2
76.4
47.2
Average Implovement (e989
20.0
replacing our proposed motion-text alignment with each: motion discriminator (Kocabas et al., 2020) (second row) and unpaired local poses (third row). Through adversarial loss or L11 minimization, the predicted motion is enforced to align with the generated motion from (Jiang al., 2023) by sharing the same semantics as the video segment.
17.5
15.0
12.5
10.0
25
0.0
Cimbing the Sagent
walking
Walking dowmaukk
Scanding with knnes best
We implement the first comparison using motion discrimina- tor. It guides motion prediction by distinguishing between predicted and generated motions of the same semantic mean- ing. As shown in Tab. 4, despite improvement in some cases, the overall performance is worse than CycleAdapt. One rea- son is the difficulty of stabilizing adversarial training; the other is that the discriminator, pre-trained to refine unrealis- tic motion rather than semantic differentiation.
Semantics
Figure 3: The improvement distribution on 3DPW dataset based on HMR network. We only show the top 7 semantics for brevity. The improved motions include both common and rare semantics found in action datasets.
bone. This indicates that the loss of semantic information still happens in the state-of-the-arr backbone.
We next implement the second comparison by using the local poses from the generated motion to supervise the pre- dicted motion. Note that the local poses do not align with the video segment but share the same semantics. To our surprise, it shows effectiveness in some cases and achieves overall performance comparable to CycleAdapt. This may be because strong unpaired local pose supervision pulls predictions into specific semantic space, while 2D evi- dence helps prevent misalignment. However, in the case of "downtown_ar" many 2D evidences are unavailable due occlusion or poor lighting. Accordingly, the method predicts results identical the generated motion, ignoring the actual motion in the test video.
5.4. Analysis Experiments
Ablations on the method components. As shown in Tab. 3, we conduct an ablation study on 3DPW (Von Marcard et al., 2018) to evaluate the impact of each component in our method. Firstly, by adding semantics-aware motion prior, the performance can be improved from MPJPE 87.7mm to 79.3mm, which verifies the effectiveness of integrating semantic information in 3D predictions. We then intro- duce EMA and fill-in incrementally, progressively refining 2D poses and reinforcing the guidance of the motion prior. Each addition can further enhance model performance. The above experiments clearly validate the contribution of each component to the overall performance.
Compared to the above two methods, our method avoids the need for additional motion data generation while simultane- ously delivering consistent improvements.
irrantics-icooporatter strategies analysis. To the best of our knowledge, no existing works explicitly use semantic information to improve 3D motion in TTA. Therefore, we verify our method through comparison to the most relevant components from the literature, as shown in Tab, 4. Specifi- cally, we use CycleAdapt (Nam et al., 2023) as a baseline, which lacks explicit semantic guidance. We then evalu- ate two alternative strategies for incorporating semantics,
Improvement distribution. In Fig. 3, we illustrate the average MPJPE improvement for the top semanties. Our method enhances both common and less frequently observed motions in human action datasets, such as bending down and standing with bent knees.
Runtime analysis. We follow the same runtime evaluation setup as CycleAdapt (Nam et al., 2023) to ensure fair com-
7

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
Table 5: Runtime analysis (ms) on the 3DPW dataset. Our method introduces a modest runtime increase compared to Adale, but still much faster than other TTA methods.
Table 6: Comparison of fill-in threshold influence on model performance (MPJPE) on the 3DPW dataset. Balancing 2D keypoint quantity and quality can boost model performance.
Method
BOA
DynaBOA
DAPA
CycleAdapt
Ours
Threshold 0.55
0.60
0.65
0.70
0.75
0.80
Rumtime
840.3
1162.8
431.0
74.1
107.6
MPJPE
78.1
76.9
77.2
77.0
76.4
77.2
The ratio of fill-in to missing 2D
Table 7: Comparison on VLM failure cases: initial per- formance without adaptation, performance after adaptation with incorrect VLM label and correct (oracle) label, respec- tively. Adaptation with incorrect VLM label can still reduce MPJPE; accurate label yield greater improvement.
0.8
0.65
0.6
0.60
0.4
0.55
Method
MPJPE
0.50
0.55
0.60
0.85
0.70
0.75
0.80
Initial
213.8
The threshold of fill-in
Incorrect VLM label
92.2
Figure 4: The relationship between the fill-in threshold, number of keypoints, and 2D keypoint PCK. Increasing the threshold leads to decrease in the ratio of filled-in keypoints, but an improvement in their quality, as indicated by a higher PCK.
Correct (oracle) label
83.7
tively simple and VLM is reliable. For the remaining 3.6%, the VLM hallucinates actions based on background, e.g. person picking up foil is mistaken as "golfing" as the foil re- sembles golf club, or "eating something using both hands" and "cast fishing pole" scenes with fruit stands and rivers, respectively. The visualization of failure cases can be found in Appendix Fig. 11.
parison. As shown in Tab. 5, methods like BOA (Guan et al., 2021), DynaBOA (Guan et al., 2022), and DAPA (Weng et al., 2022) have high per-frame runtime due to complex optimization or extra synthesis rendering. Compared to Cy- cleAdapt, our method introduces an additional 33. 33.5ms run- time per frame, 85.4% of which comes from the VLM, and 14.6% from our proposed framework components, However, is still much faster than other TTA methods.
Despite these VLM failures, our method maintains robust- ness. As our semantics-awarr alignment is regularizer, its impact is limited if it conflicts with 2D projection loss, Tab. 7 shows that the adaptation with the wrong label still reduces the MPJPE, but not as much as the correct (oracle) action label were applied.
2D fill-in analysis. We try different fill-in thresholds in Eq. 8 (from 0.55 to 0.80) evaluate the quality and quantity of 2D keypoints. Fig. 4 shows as we increase the fill-in threshold, fewer new 2D keypoints are added (blue bars), but their PCK improves (green bars). Increasing the thresh- old helps use reliable but fewer predicted 2D keypoints in training, which have high feature similarity with their text labels.
5.6. Qualitative Results
We perform a qualitative comparison to visually assess the effectiveness of our method. In Fig. 5, we first illustrate that without 2D pose updates, the model predictions for the truncated part revert average poses. Specifically, in epoch 2, frames highlighted with a green square show some walk- ing poses. However, by epoch 4, these walking poses are back to standing poses (red squares). With 2D) pose updates ("Ours"), more frames accurately reflect the intended action. This result verifies the necessity of using 2D pose update to preserve motion semantics.
Tab. illustrates the overall performances on diffeernnt thresholds. We find out that using 0.75 threshold adds only 36% more 2D keypoints, yet the high quality (PCK=0.58) helps achieve 76.4mm MPJPE. Even with a moderate qual- ity (PCK=0.53), adequate new keypoints can still positively impact the model when the threshold is 0.60. Overall, the ex- periment demonstrates that balancing the quality and quan- tity of 2D keypoints can enhance model performance.
Furthermore, demonstrate that CycleAdapt (Nam et al., 2023) often predicts standing poses where 2D pose estimates are missing. Our method consistently estimates motions closer to the ground truth, verifying the feasibility of using semantic information to refine predicted motions. More visualizations can be found in Appendix Sec. C.
5.5. VLM Failure Cases
We manually examined 5,000 frames from the 3DPW dataset, where 96.4% is accurate, as the actions are rela-
8

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
Video
Epocy
Epoch 4
Ours
CycleAdapt
GT
video sequence from 3DPW dataset (Von Marcard et al., 2018) with text label "Walking". From epoch 2 to epoch 4, results show that motion-ter text alignment alone is insufficient to preserve motion semantics with missing 2D data. We use green and red squares to highlight the transition from walking to standing poses. Incorporating 2D pose updates (Ours) helps mitigate this problem. Compared to CycleAdapt (Nam et al., 2023), which shows static standing foot under image truncation, ours captures the natural alternation of foot and knee bending during walking, aligning more closely with the actual walking sequences.
Figure 5: Qualitative comparison
6. Conclusion and Limitations
of the subject, with discrepancies in step frequency du- ration; ambiguities like motion blur (Chen et al., 2025b) remains an open direction; and reliance on MotionCLIP limits adaptability to new motion descriptions. Addressing these challenges calls for improved adaptation strategies and more advanced motion- -anigage models.
We propose a novel text-guided test-time adaptation frame- work for solving 3D human pose estimation on unseen videos. Previous methods employ temporal smoothing to mitigate the impact of noisy 2D labels and depth ambiguity, but this often leads to average results and even stationary poses when 2D evidence is lacking. We address them by incorporating motion semantics from a well-trained motion- text space, refining 3D pose predictions in correct se- mantic solution space. Moreover, the text-aligned motion predictions will be used to update 2D) labels, enhancing the model understanding of the true motion. Our method sures semantically consistent pose predictions across video sequences, outperforming state-of-the-art methods in both quantitative and qualitative evaluations.
A Acknowledgments
This research was funded in part by Google South Asia & Southeast Asia Research Award given to A. Yao.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
However, our work has certain limitations: the refined mo- tion under occlusion may not fully capture the actual pattern

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
References
Kanazawa, A., Black, M. J., Jacobs, D. W. and Malik, J. End-to-end recovery of human shape and pose. In CVPR, pp. 7122-7131, 2018.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774: 2023.
Kenton, D. M.-W. C. and Toutanova, L. K. Bert: Pre- training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLI, pp. 4171- 4186, 2019.
Cao, Z., Simon, T., Wei, S.-E., and Sheikh, Y. Realtime multi-persee person 2d pose estimation using part affinity fields. In CVPR, pp. 7291-7299, 2017.
Kingma. D. P. and Ba, Adam: A method for stochastic optimization. In ICLR, 2014.
Chen, R., Yang, L.,, and Yao, A. MHEntropy: Entropy meets multiple hypotheses for pose and shape recovery. In ICCV, 2023.
Kocabas, M., Athanasiou, N., and Black, M. J. Vibe: Video inference for human body pose and shape estimation. In CVPR, June 2020.
Chen, R., Zhuo, I.., Yang, L., Wang. Q. Bo, I.., Zhang, B., and Yao, A. ExtPose: Robust and coherent pose estimation by extending ViTs. In ICML, 2025a.
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., and Black, M. Smpl: skinned multi- pertiipprsonn linear model. In ACM TOG, pp. 851-866. 2023.
Chen, Y., Chen, R., Zhang, Z., Cheng, Y., and Chang, H. Multi-hypothesis 3D hand mesh recovering from a single blurry image. In ICME, 2025b.
Mahmood, N., Ghorbani, N., Troje, N. F., Pons-Moll, G., and Black, M. J. AMASS: Archive motion capture as surface shapes. In ICCV, pp. 5442 5451, 2019.
Goel, S., Pavlakos, G., Rajasegaran, J., Kanazawa, A., and Malik, J. Humans 4D: Reconstructing and tracking humans with transformers. ICCV, 2023.
Mehta, D., Rhodin, H., Casas, D., Fua, P., Sotnychenko, O., Xu, W., and Theobalt, C. Monocular 3d human pose estimation in the wild using improved enn supervision. In 3DV, pp. 506-516. IEEE, 2017.
Gong, K., Li, B., Zhang, J., Wang, T., Huang, J., Mi, M. B., Feng, J., and Wang, X. PoseTriplet: Co-evolving 3D hu- man pose estimation, imitation, and hallucination under self-supervision. In CVPR, 2022.
Moon, G., Shiratori, T., and Saito, S. Expressive whole- body 3D gaussian avatar. In ECCV, 2024.
Nam, H., Jung, D. S., Oh, Y., and Lee, K. M. Cyclic test- time adaptation on monocular video for 3d human mesh reconstruction. In ICCV, pp. 14829-1499 2023.
Gu, K., Chen, R., Yu, X., and Yao, A. On the calibration of human pose estimation. In ICML, 2024.
Guan, S., Xu, J.. Wang. Y. Ni, B., and Yang, X. Bilevel human mesh recon-
Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A. A., Tzionas, D., and Black, M. J. Expressive body capture: 3d hands, face, and body from a single image. CVPR, pp. 10975-10985, 2019.
online adaptation for struction. In CVPR, pp. 10472-10481, 2021.
Guan, S., Xu, J., He, M. Z., Wang, Y., Ni, B., and Yang, X. Out-of-domaa human mesh reconstruction via dy- namic bilevel online adaptation. TPAMI, 45(4):0070- 5086, 2022.
Punnakkal, A. R., Chandrasekaran, A., Athanasiou, N., Quiros-Raminee A., and Black, M. J. Babel: Bodies, action and behavior with english labels. CVPR, pp. 722-731, 2021.
Guo, C., Zuo, X., Wang. S., and Cheng, L. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In ECCV, pp. 580-597. Springer, 2022.
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al. Improving language understanding by generative pre-training. 2018.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. ICML, 2021.
Jonescu, C., Papava, D., Olaru, V., and Sminchisescu, C. Hu- man3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. TPAMI, (6(7):1325-133)) 2013.
Rempe D., Birdal, T., Hertzmann, A., Yang, J., Sridhar, S., and Guibas, L. J. HuMoR: 3D human motion model for robust pose estimation. In ICCV, pp. 11488-11444 2021.
Jiang, B., Chen, X., Liu, W., Yu, J., Yu, G., and Chen, T. Motiongpt: Human motion as a foreign language. NeurlPS, 36:20067-20079 2023.
10

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
Shi, L., Zhang, Y., Cheng, J., and Lu, H. Two-stream adaptive graph convolutional networks for skeleton-based action recognition. In CVPR. pp. 12026-12035, 2019.
Shin, S., Kim, J., Halilaj, E., and Black, M. Wham: Reconstructing world-grounded humans with accurate 3d motion. In CVPR, 2024.
Tevet, G., Gordon, B., Hertz, A., Bermano, A. H., and Cohen-Or, Motionclip: Exposing human motion gen- eration to clip space. In ECCV, pp. 358-374. Springer, 2022.
Von Marcard, T., Henschel, R., Black, M. J., Rosenhahn, B., and Pons-Moll, G. Recovering accurate 3d human pose in the wild using imus and a moving camera. In ECCV, 2018.
Weng, Z., Wang, K.-C., Kanazawa, A., and Yeung, S. Do- main adaptive pose augmentation for in-the-wild hu- man mesh recovery. In 3DV, pp. 261-270. IEEE, 2022.
Yuan, Y.. Iqbal, U., Molchanov, P., Kitani, K., and Kautz, GLAMR: Global occlusion-aware human mesh recov- cry with dynamic cameras. In CVPR, pp. 1008--10049, 2022.
Zhang. J., Nie, X., and Feng, J. Inference stage optimization for cross-scenario 3d human pose estimation. NeurlPS, 33:2408-2419, 2020.
Zhang, J., Zhang, Y., Cun, X., Zhang. Y., Zhao, H., Lu, H., Shen, X., and Shan, Y. Generating human motion from textual descriptions with discrete representations. In CVPR, pp. 14730-14740, 2023.
Zhang, M., Cai, Z., Pan, L.,, Hong, F., Guo, X., Yang, L.,, and Liu, Z. Motiondiffuse: Text-driven human motion generation with diffusion model. TPAMI, 46(6):411-- 4128, 2024.
Zhang, S., Zhang, Y., Bogo, F., Pollefeys, M., and Tang, S. Learning motion priors for 4d human body capture in 3d scenes. In ICCV, pp. 11343-11353, 2021.
Zhang, S., Ma, Q., Zhang, Y., Qian, 7., Kwon, T., Pollefeys, M., Bogo, F., and Tang, S. Egobody: Human body shape and motion of interacting people from head-mounted devices. In ECCV, October 2022.
Zheng, X., Su, Z.,, Wen, C., Xue, Z., and Jin, X. Realistic full-body tracking from sparse observations via joint-level modeling, In ICCV, 2023.

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
Table 8: Comparison with different EMA hyperparameter a on the 3DPW dataset. a = 0.9 yields best performance, balancing estimated 2D pose and adapted model feedback.
Figure 6: A video segment example used as input for text labeling.
0.75
0.80
0.85
0.90
0.95
MPJPE 78.6
77.9
76.8
76.4
77.5
Here are the results based on the uploaded motion dictionary:
This appendix includes four sections: A. EMA Hyperpa- rameter, B. Text Labeling, C. More Qualitative Results, as referred in the manuscript.
The number of labels related walking: 191.
Most relevant text label: lying stretch.
Let me know if you need further analysis or clarifi- cation!
A. EMA Hyperparameter
We conduct grid search on the exponential moving average determine its optimal value. As shown in the Tab. 8, performance steadily improves increases up to 0.90, where the lowest MPJPE of 76.4mm is achieved. This indicates that most estimated 2D poses are reliable for guiding model adaptation. However, noisy estimated 2D poses still require refinement from the model prediction for better guidance (e.g., at a 0.95).
(EMA) hyperparameter
By modifying "walking" and motion descriptions, we will repeat four additional similar processes to verify GPT-4o's ability retrieve the text label relevant Specifically, modify "walking" to "stand up", "talking", "go up stairs" and "pick up". which results in 72, 13, 18, and 67 identified labels, respectively. The identified labels include the motion string, or its interchangeable variations, such as replacing "go up" with "walk up" or "talking" with "argue".
description.
B. Text Labeling
More descriptions provided by us and the retrieved text labels by GPT-4o are as follows:
In this section, we explain our text labeling process and its accuracy evaluation. Text labeling is conducted using GPT-4o (Achiam et al., 2023) as the VLM model. The evaluation is carried out through a CLIP (Radford et al., 2021) consistency check and manual correction.
a person bending down to the ground, using both hands pick up an object carefully.
As MotionCLIP (Tevet et al., 2022) has limited generaliza- tion on unseen text, GPT-4o is required to retrieve the text label from the motion dictionary used in (Tevet et al., 2022). We use the following prompt to ensure GPT-4o interprets the dictionary accurately:
picking up something using both hand.
a person drinking from bottle, head tilted back, then lowering the bottle and placing it gently on the table.
drink from bottle and put down bottle.
a person rising to a standing position, then lowering into squat with knees bent and arms extended for- ward.
Hi, the attached dictionary motion dict.csv contains text labels of various human motions. Please help confirm your understanding of the dictionary by answering the following:
rise up and squat.
a person hopping forward the left foot, with the right leg bent slightly and arms raised for balance.
How many labels are related to walking?
Identify the most relevant text label from the dictionary for a description like: "a person ly- ing on the ground, with limbs outstretched".
hopping on left foot.
We observe from the experiment results that GPT-4o can identify semantically similar labels and match text descrip- tions with an appropriate text label. We then use the follow- ing prompts to create formatted video segment label:
We can obtain the following output:
12

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
You are reliable assistant who will follow ALL rules and directions entirely and precisely. need your assistance in creating a text label that describes the motion in video segment. You need to first gen- erate descriptions for the video segment. Then, you need to assign the most appropriate text label for the segment using the motion dictionary provided in the motion_dict.cm file. Below are the three examples of your answers:
person running
0.16
Video segment description: The person in blue jeans remained seated and talked to the person next to him throughout the video segment. Text label: sitting.
person going up stairs
0.23
0.35
0.13
0.22
0.12
a person talking on phone
0.15
0.17
0.32
0.21
0.22
Video segment description: The person in the video throws his hands upwards as if throwing something, while also walking forward. Text label: throw something with the right hand and walking.
a person sitting and stand up
0.21
0.18
0.26
person bending over
0.17
0.23
0.19
0.22
0.30
Figure 7: The cosine similarity between video segments and text labels from 3DPW dataset (Von Marcard et al., 2018). Higher similarity indicates a closer semantic alignment be- tween text labels and video segments. This result shows that the cosine similarity of CLIP can help label verification.
Video segment description: The person per- forming figure skating on the ice, specifically executing a double Axel jump. Text label: None.
Your answers must be grounded in the provided video segment and follow this format. Limit seg- ment descriptions a maximum of 20 words. The text label must be selected from the dictionary; if suitable label exists, return "None" the text label.
is applied. Ultimately, we ensure that all video segments achieve high alignment between text labels and video semantics, with an average cosine similarity of 0.32.
C. More Qualitative Results
In this section, we provide more qualitative results 3DPW (Von Marcard et al., 2018), 3DHP (Mehta et al., 2017) and EgoBody (Zhang et al., 2022) datasets. As the 3DHP dataset lacks ground truth SMPL parameters, the qualitative results on 3DHP are limited to comparisons with other methods. In 3DPW (shown in Fig. 8), our method can provide more semantics-aligned predictions for cases where all 2D are visible (running case) some 2D invisible (walking case). In 3DHP (shown in Fig. 9), our method captures motion details more accurately in local poses, including hand positions, leg configurations, and arm movements. On the more challenging EgoBody dataset (shown in Fig. 10) with severe body truncation, our method excels in aligning with motion semantics.
Video segment description: The person blue jeans is rumning forward while adjusting his posture and avoiding obstacles.
Text label: running.
For each video, we first downsample both images and se- quences. Then, we divide the videos into 60-frame segments, which are subsequently fed as input to GPT-4o along with the above prompts. Below is an example of output for video segment as shown in Fig. 6.
For evaluation, our goal is to verify whether the retrieved text label correctly reflects human motion depicted in the video segment. We utilize CLIP (Radford et al., 2021) measure the alignment between the text label and video segment by calculating the cosine similarity between their respective feature representations and averaging over the segment. Empirically, we observe that a text label aligning well with video semantics usually has a cosine similarity greater than 0.3, as shown in Fig. 7. For cases where the similarity is lower than this threshold, manual correction
13

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
CycleAdapt (Front Video
Ours (Front
CycleAdapt (Sids
Qurs (Side View)
GT (Side
Video
CycleAdapt (Front View)
Ours (Front View)
CycleAdapt (Side View)
Ours (Side (Side
GT
Figure 8: Qualitative comparison on two video segments from 3DPW dataset (Von Mareard al., 2018). Our method demonstrates more camantics-alignel predictions for the "Running" and "Walking" motions. The red and green squares highlight the examples where CycleAdapt fails, but method performs better.
14

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
Video
CycleAdapt (Front View)
(Front. View)
DynaB (Side View)
CycleAdapt (Side View)
(Side View)
Video
CyceAdapt (Front View)
(Front View) View
DyneBOA (Side View)
CycleAdapt (iide View)
(Siee View)
Figure 9: Qualitative comparison on two video segments from 3DHP dataset (Mehta et al., 2017) with text labels of "Sitting" and "Spin around with right foot". Our method provides more accurate pose predictions in the motion details compared CycleAdapt (Nam al., 2023) and DynaBOA (Guan et al., 2022). For example, we can better capture hand positions while sitting (Ist segment), crossing legs while sitting (Ist segment), and movements while spinning around (2nd segment).
15

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
Video
CycleAdapt (Front
Ours View)
CycloAdapt (Side
Ours View)
GT Viow)
Video
CycloAdapt (Front
Ours (Front View)
(Full Front View) (Ful Front View) (Full Frant Frant View)
CycleAdapt
Figure 10: Qualitative comparison two video segments from EgoBody dataset (Zhang et al., 2022). We enhance the pose predictions to be aligned with "Standing" and "Sitting" text labels. The red and green squares highlight the examples where CycleAdapt fails, but our method performs better.
16

---

Semantics-aware Test-time Adaptation for 3D Human Pose Estimation
Video
(Front View)
(Side View)
(Side View)
(Front |FrontVied View)
(S:0e Viaw)
(Sde View
Video
(Front View)
(Side View)
(Sde View)
Figure 11: VLM failure cases: "golfing", "eating something using both hands". and "cast fishing pole". Despite VLM hallucination causing some text labels to misalign with action semantics, our method remains robust. Additionally, some VLM predictions and GT exhibit different motion directions (e,g., depth ambiguity of picking up from the front vs. right front in the first example), which may be interesting consider in future work.
17


Reviews:
[{"review_id": "zGktCizaIM", "reviewer": "Reviewer_c9x6", "summary": "This paper proposes a novel semantics-aware test-time adaptation (TTA) framework for 3D human pose estimation, addressing the issue of overly smoothed or unguided predictions, especially under occlusion or truncation. The key innovation is integrating motion semantics into the TTA process by leveraging MotionCLIP to align pose sequences with textual action labels inferred via a vision-language model (VLM). Additionally, missing 2D keypoints are completed based on motion-text similarity to improve supervision where 2D evidence is lacking. The method significantly outperforms prior approaches such as CycleAdapt on datasets including 3DPW, 3DHP, and EgoBody. Supplementary results show high semantic consistency, effective fill-in mechanisms, and robust improvements in both common and rare motion types.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere appreciation for the helpful reviews and welcome further discussion!", "claims_and_evidence": "The authors' claims are well supported:\n1. The semantics-aware motion prior reduces depth ambiguity and guides the adaptation process.\n2. 2D pose completion using text-aligned motion enhances adaptation under occlusion.", "methods_and_evaluation_criteria": "The methods are carefully designed and appropriate:\n\n1. MotionCLIP is used effectively as a semantics-aware regularizer.\n2. A VLM (GPT-4o) assigns action labels to video segments, with verification via CLIP cosine similarity.\n3. EMA and fill-in strategies help refine 2D poses over adaptation epochs.\n\nThe evaluation uses standard benchmarks (3DPW, 3DHP, EgoBody) and metrics (MPJPE, PA-MPJPE, MPVPE).", "theoretical_claims": "There are no deep theoretical claims in this work.", "experimental_designs_or_analyses": "Strong design and analysis. The paper includes:\n\n1. Solid baselines (BOA, DynaBOA, DAPA, CycleAdapt).\n2. Ablation studies isolating the contribution of alignment, EMA, and fill-in components.\n3. Per-semantic motion improvement analysis.\n4. Supplementary evaluations on fill-in threshold impact and text-labeling accuracy.", "relation_to_broader_scientific_literature": "This paper advances the state-of-the-art in semantics-aware adaptation for 3D human pose estimation, bridging two previously disjoint areas: vision-language modeling and test-time pose refinement. While MotionCLIP and CycleAdapt are relevant foundations, this work introduces a novel integration of motion semantics during adaptation. It complements prior efforts in temporal smoothing and motion priors by adding a high-level semantic guidance layer.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "N.A.", "other_comments_or_suggestions": "N.A.", "questions_for_authors": "N.A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6uFdBKSWuT", "reviewer": "Reviewer_qP6M", "summary": "The paper introduces a TTA HMR method from videos using semantic information to address challenges caused by a lack of information when a large portion of the body is occluded. The paper suggests using MotionCLIP to align predicted motions with the CLIP representation of the textual description of the actions as extracted by GPT-4o. The 3D HPE model is then adapted during test time to match OpenPose keypoints and achieve semantic alignment while being temporally regularized. Next, the fine-tuned HMR model obtains initial predictions, which are further improved using a semantic-aware EMA operation. The main finding is that incorporating this semantics-aware motion prior significantly improves performance over state-of-the-art TTA methods, especially in cases with occlusions or truncations.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere appreciation for the helpful reviews and tackle the concerns below:\n\n**[Method 1] Fairness in Table 1**\n\n**R:** Our paper highlights the problem of motion semantics and proposes a method to incorporate semantics as the core contribution.  No existing works use semantics, so it is difficult to compare directly. We verify our method by comparing it with the closest components from the literature, as shown in Table 3. Specifically, we use CycleAdapt as a baseline (top row) and then add semantics through a motion discriminator (row 2) or unpaired local poses (row 3). In this way, we enforce the predicted motion to be aligned with the generated motion from [1] by sharing the same semantics as the video segment. Table 3 in the paper shows that our method better integrates semantics without additional motion data generation.\n\n**[References 1] Diffusion models**\n\n**R:** Thank you for your suggestion. We will add these references and also discuss diffusion models in our related works section.\n\n\n**[Weakness 1] Method alternatives**\n\n**R:** Thanks for pointing this out. We explore different strategies of integrating semantic information; two alternatives to our method are shown in Table 3. We will adjust the labels and captions of Table 3 and Sec. 5.4 (‘Analysis on semantics-incorporated techniques’) in the text to clarify this point.\n\n**[Weakness 2] Long periods of occlusion or jitters**\n\n**R:** Thank you for pointing this out. \n\nFor long periods of full-body occlusion, motion can be generated to maintain semantic consistency with the predicted activity. However, this aligns more with motion generation, which is evaluated by plausibility (e.g., FID [2]), unlike reconstruction, which relies on ground truth. We plan to merge reconstruction and generation into a single framework and develop suitable evaluation metrics in future work.\n\nWe evaluate jitter on the 3DPW dataset, comparing our method with ground truth and CycleAdapt. As shown in the table below, our jitter is significantly lower than the image-based DynaBOA but 11.1% higher than CycleAdapt. However, CycleAdapt often predicts static motion, even in dynamic motion scenes, resulting in low jitter but deviating from the ground truth. Take the walking motion from Fig. 1 of the paper as an example, we report the jitter values for this case in the last row of the table below, where ours is higher than CycleAdapt but closely aligns with the ground truth.\n\n| Jitter$(10m/s^3)$ | GT | CycleAdapt | DynaBOA | Ours |\n| --- | :---: | :---: | :---: | :---: |\n| All sequences | 22.3 | 27.9 | 172.1 | 31.0 |\n| Walking | 46.7 | 16.8 | - | 48.3 |\n\n\n**[Other Comments 1] Typo**\n\n**R:** Thank you for pointing this out. We will correct it in the revision.\n\n**[Question 1] Runtime analysis**\n\n**R:** Thank you for your suggestion. We perform runtime analysis following the environment settings described in Table 7 in CycleAdapt. The table below shows that compared to CycleAdapt, our method introduces an additional 33.5ms runtime per frame, 85.4% of which is caused by VLM, and 14.6% by our proposed framework components. However, it is still much faster than other TTA methods. We will add runtime analysis in the revision.\n\n| Method | BOA | DynaBOA | DAPA | CycleAdapt | Ours |\n| --- | :---: | :---: | :---: | :---: | :---: |\n| Runtime(ms)  | 840.3 | 1162.8 | 431.0 | **74.1** | 107.6 (+33.5) |\n\n\n**[Question 2] Hyperparameters and EMA**\n\n**R:** We choose hyperparameters based on a grid search. The EMA update factor ($\\alpha$) values used for the search are provided in the table below. For the fill-in threshold ($\\sigma$) of missing 2D detection, please refer to Table 5 in the paper. \n\nThe EMA and $\\alpha$ do not cause jitters. This is because the EMA is performed over 2D poses for each frame and each frame has one fixed action label.\n\n| $\\alpha$| 0.75 | 0.80 | 0.85 | 0.90 | 0.95 |\n| --- | :---: | :---: | :---: | :---: | :---: |\n| MPJPE(mm)| 78.6 | 77.9 | 76.8 | **76.4** | 77.5 |\n\n\n**[Question 3] GT 2D**\n\n**R:** Thank you for your suggestion. We add results in the table below. Our method still performs better than CycleAdapt. The improvement becomes smaller because GT 2D keypoints provide very strong indications for occluded keypoints, which is unrealistic in real-world scenarios. Our method especially takes effect when 2D keypoints are noisy or missing.  \n\n| Method |      |3DPW  |          | 3DHP      |          |\n| --- | :---: | :---: | :---: | :---: | :---: |\n|       |    | MPJPE     | PA-MPJPE | MPJPE     | PA-MPJPE |\n| OpenPose | CycleAdapt | 87.7     | 53.8      | 110.3    | 74.4     |\n|        | Ours      | 76.4     | 47.2      | 101.3    | 65.1     |\n| GT     | CycleAdapt | 64.7     | 39.9      | 100.9    | 64.6     |\n|        | Ours      | **64.1** | **39.4** | **98.5** | **63.7** |\n\n\n**References**\n\n[1] Jiang et al. MotionGPT: Human motion as a foreign language. NeurIPS'2023.\n\n[2] Guo et al. Generating diverse and natural 3d human motions from text. CVPR'2022.", "claims_and_evidence": "The paper claims improved performance over SOTA, which is supported in the experiments by showing the effectiveness of adding semantic information. The paper also propose an interesting approach to incorporate the new modality and through ablations, shows the effectiveness of fill-ins and EMA.", "methods_and_evaluation_criteria": "The datasets and evaluation metrics are commonly used in this field. Using multiple datasets and ablations is appropriate to show the effectiveness of the proposed method. The experiments might be unfair in Table 1, since none of the other methods use semantic information, but it's acceptable given the context of the paper.", "theoretical_claims": "The paper does not have any theoretical proof.", "experimental_designs_or_analyses": "The experimental design and experiments seem sound. The paper claims that it follows the same evaluation protocol as prior research and uses recent papers for comparison.", "relation_to_broader_scientific_literature": "Motion priors trained on large motion capture datasets like AMASS have been used to regularize 3D human pose and motion estimation, with early examples like VPoser for pose regularization and later works like HuMoR employing VAEs for motion priors. Recent Test-Time Adaptation (TTA) approaches have demonstrated that fine-tuning 3D Human Mesh Recovery (HMR) models on test video sequences improves speed and accuracy. This paper builds upon TTA, proposing incorporating semantic information via a pre-trained motion-language model into fine-tuning. While the paper leverages existing components from the literature, it presents a well-justified strategy for addressing occlusions through semantic consistency during TTA.", "essential_references_not_discussed": "The paper understands related literature well, citing adequate and recent references from the last two years. The model lacks a reference to diffusion models, like RoHM and MDM, which also use trajectory information. However, this lack of reference is acceptable since that line of research focuses on motion generation.", "other_strengths_and_weaknesses": "### Strengths\n1. The paper is well-written, clear, and concise. It is easy to follow and provides sufficient details to understand the paper. Although some components are too reliant on the prior works, the paper has done a good job summarizing their efforts.\n2. The supplementary materials provides adequate information to replicate the experiments. \n3. The paper has a good understanding of the literature, citing and comparing with them when needed.\n4. The paper provides a compelling approach to incorporate sematic information into TTA, improving upon previous works.\n\n### Weaknesses\n1. The paper does not provide any additional alternatives other than previous works.\n2. As the authors mentioned, this method cannot perfectly capture the correct movements of the subjects while they are occluded. However, it could generate a plausible motion based on semantic consistency. There is no information about long periods of occlusion (whole body) or jitters (due to thresholding).", "other_comments_or_suggestions": "I did not find any major issues. Please address my questions below if possible. Thank you.\n\nTypo:\n- L302, right column, \"Egobody\" should be \"EgoBody\"", "questions_for_authors": "1. Could you please provide a runtime analysis, like Table 7 of the CycleAdapt paper? It is crucial to understand the computational requirements of training/testing of your method compared to others.\n2. How did you choose the hyperparameters? Wouldn't the EMA and $\\alpha$ cause a jitter in motion during different actions? \n3. Could you provide the results of using GT 2D human keypoints instead of OpenPose for table 1? It would be interesting to understand the noise robustness of your approach compared to others when working with GT/OpenPose.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P161PQiXl2", "reviewer": "Reviewer_qh14", "summary": "This paper presents a 3D human pose estimation that supports test-time optimization with semantics. The authors leverage video understanding and a well-structured motion-text space to adapt the model motion predictions. In addition, they incorporate a missing 2D pose completion with the motion-text similarity. Experimental results demonstrate the effectiveness of the proposed method.\n\n## update after rebuttal\n\nThe authors address my questions well. I keep my original rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere appreciation for the helpful reviews and tackle the concerns below:\n\n**[Weakness 1] Failure cases from the VLM**\n\n**R:** Thank you for your suggestions. We manually examined 5,000 frames from the 3DPW dataset, where 96.4% is accurate, as the actions are relatively simple and VLM is reliable. For the remaining 3.6%, the VLM hallucinates actions based on background, e.g. a person picking up foil is mistaken as \"golfing\" as the foil resembles a golf club, or \"eating something using both hands\" and \"cast fishing pole\" in scenes with fruit stands and rivers, respectively. Visualizations of these VLM's failure cases can be found in Fig.I ([anonymous GitHub link](https://anonymous.4open.science/r/visualization-D59C/anonymous_pdf.pdf)).\n\nDespite these VLM failures, our method maintains robustness. As our semantics-aware alignment is a regularizer, its impact is limited if it conflicts with 2D projection loss.  The table below shows that the adaptation with the wrong VLM label still reduces the MPJPE, but not as much as when the (oracle) correct action labels were applied.\n\n| Method | Initial | Incorrect VLM action output | Correct (oracle) action output |\n| --- | :---: | :---: | :---: |\n| MPJPE (mm) \t| 213.8 | 92.2 | 83.7 |\n\n\n\n**[Weakness 2] No GT mesh in Figure 1**\n\n**R:**  Thanks for pointing this out. As shown in Fig.IV ([anonymous GitHub link](https://anonymous.4open.science/r/visualization-D59C/anonymous_pdf.pdf)), we include GT mesh for a better comparison, where our predictions preserve the same motion semantics as the GT.\n\n\n**[Weakness 3] Runtime analysis**\n\n**R:** Thanks for pointing this out.  Yes, our runtime (107.6ms) is longer than CycleAdapt (74.1ms); the extra overhead is primarily from the VLM (85.4%).\n\nDetails are given in our response to **[Question 1] (Runtime analysis)** for the reviewer **qP6M**.\n\n**[Weakness 4] Include challenging poses** \n\n**R:** Thank you for your suggestions. We provide more challenging poses, i.e., \"open a door and sit\", \"hugging\", \"squats series\" and \"spin around with right foot\". Please refer to Fig.II and Fig.III in the [anonymous GitHub link](https://anonymous.4open.science/r/visualization-D59C/anonymous_pdf.pdf).\n\n\n**[Question 1] Direct optimization without fine-tuning** \n\n**R:**  Yes, it is faster since it does not require model fine-tuning. However, the success of direct optimization is linked heavily to the accuracy of the initial estimate.\n\nWe compare with ScoreHMR [3], a state-of-the-art direct optimization approach on the 3DPW dataset using OpenPose 2D poses. As shown in Fig.V ([anonymous GitHub link](https://anonymous.4open.science/r/visualization-D59C/anonymous_pdf.pdf)), our method is robust across different initial prediction qualities. ScoreHMR is more sensitive and when the initial MPJPE exceeds 200mm, ScoreHMR reduces error only by 26.6%, whereas we can reduce it by 64.5%.\n\n**References** \n\n[3] Stathopoulos et al. Score-guided diffusion for 3D human recovery. CVPR'2024.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "It is related to many human-centric computer vision tasks.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strengths\n\nAs the authors argued, existing TTA methods lack semantics, which can suffer from many failure cases, including the top row of Figure 1. Adding semantics with a vision-language model is a good step toward semantics-aware 3D human pose estimation. The effectiveness of such incorporation is well-shown in the experiments section.\n\nWeaknesses\n1. It is not clear how robust the system is to the wrong outputs from VLM. It could be interesting if the authors could show some failure cases of the proposed system including the failure cases from the VLM. \n2. The results in Figure 1 below are not appealing. Due to the occlusion, it is not clear what is the correct one between CycleAdapt and the proposed method’s outputs.\n3. The running time should be much slower than CycleAdapt due to the VLM. What is the exact running time in the TTA?.\n4. Overall, qualitative results only include easy poses, such as simply walking or standing. More challenging poses should be included.", "other_comments_or_suggestions": "Please see weaknesses.", "questions_for_authors": "1. What if we directly optimize the output 3D poses without fine-tuning the 3D HPE model? I think that should make the TTA much faster.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper achieved unanimously positive reviews among reviewers and has made solid contributions in 3D human pose estimation. The initial concerns are mostly about evaluation and the authors have addressed them well during rebuttal.

Therefore, I recommend acceptance of this paper and encourage the authors to integrate the rebuttal in their revision.
Decision: Accept (poster)