Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data
Authors: Seunggeun Chi
Pin-Hao Huang
Enna Sachdeva
Hengbo Ma
Karthik Ramani
Kwonjoon Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: pose estimation
egocentric pose estimation.
Primary area: machine_vision

Abstract: We study the problem of estimating the body movements of a camera wearer from egocentric videos. Current methods for ego-body pose estimation rely on temporally dense sensor data, such as IMU measurements from spatially sparse body parts like the head and hands. However, we propose that even temporally sparse observations, such as hand poses captured intermittently from egocentric videos during natural or periodic hand movements, can effectively constrain overall body motion. Naively applying diffusion models to generate full-body pose from head pose and sparse hand pose leads to suboptimal results. To overcome this, we develop a two-stage approach that decomposes the problem into temporal completion and spatial completion. First, our method employs masked autoencoders to impute hand trajectories by leveraging the spatiotemporal correlations between the head pose sequence and intermittent hand poses, providing uncertainty estimates. Subsequently, we employ conditional diffusion models to generate plausible full-body motions based on these temporally dense trajectories of the head and hands, guided by the uncertainty estimates from the imputation. The effectiveness of our methods was rigorously tested and validated through comprehensive experiments conducted on various HMD setup with AMASS and Ego-Exo4D datasets. Project page: https://sgchi.github.io/dsposer

Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data
Seunggeun Chit Purdue University chi chi650purdue.. edu
Pin-Hao Huang", Enna Sachdeva'
Honda Research Institute USA
{pin hhthhah_hhhhaaaaa hangh hetp__sschevaaahonnccee com
Hengbo Ma Honda Research Institute USA
Karthik Ramani Purdue University ramani @purdue edu
Kwonjoon Lee
Honda Research Institute USA
hengbo. academi aggmail. com
Mwoonjoon_eeehhonee coadddarr..com
Abstract
We study the problem of estimating the body movements of camera wearer from egocentric videos. Current methods for ego-body pose estimation rely on temporally dense sensor data, such as IMU measurements from spatially sparse body parts like the head and hands. However, propose that even temporally sparse observations, such as hand poses captured intermittently from egocentric videos during natural periodic hand movements, body motion. Naively applying diffusion models to generate full-body pose from head pose and sparse hand pose leads to suboptimal results. To overcome this, we develop two-stage approach that decomposes the problem into temporal comple- tion and spatial completion. First, our method employs masked autoencoders impute hand trajectories by leveraging the spatiotemporal correlations between the head pose sequence and intermittent hand poses, providing uncertainty estimates. Subsequently, we employ conditional diffusion models to generate plausible full- body motions based on these temporally dense trajectories of the head and hands, guided by the uncertainty estimates from the imputation. The effectiveness of our method was rigorously tested and validated through comprehensive experiments conducted on various HMD setup with AMASS and Ego- Ego-Exo4D datasets. Project page: https://sgcci, http:://ssch..iih....b.. wtt.hb.b.o/sssssee//////
effectively constrain overall
Introduction
The evolution of augmented reality (AR) devices such as the Apple Vision Pro, Meta Quest 3, Microsoft HoloLens and etc. has dramatically reshaped interactive technologies. These head- mounted displays (HMDs) feature inertial measurement units (IMUs) and video capture capabilities, offering unique egocentric perspective. However, their limited visibility of the user's body parts poses a significant challenge for accurate egocentric body pose estimation key element for immersive AR experiences.
Previous approaches have tackled this problem by spatially reconstructing the entire body from spatially sparse data. For instance, EgoEgo [16] first estimates head poses using SLAM the egocentric video, then generates body poses from these estimated head positions. Other methods, such AvatarPoser [13] and BoDiffusion [3], primarily depend temporally dense tracking signal from spatially sparse body parts, notably the head and hands. This dependency on specific hardware
co secoee authors
work done Honda Research Institute USA.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Hard
Hight Hoae
(a)
(c)
(h)
(d)
Figure 1: Overview of DSPoser. Our goal to estimate ego-body pose without dependency on hand controllers in an HMD environment. (a) Given the egocentric video and head tracking signals as input, (b) our approach first predicts the hand pose in the frames where hands are visible (dark blue). It then estimates the hand poses in frames with invisible hands (light blue) using imputation, and (c) estimates uncertainty associated with the hand poses where the hands are invisible, (d) The predicted and imputed hand pose is then used with head pose to predict the 3D full body pose.
such as head-mounted displays and hand controllers constrains their versatility and diminishes their applicability in broader AR/VR scenarios where hand controllers might not be used, like sports training or analysis applications where the user needs to move freely without holding any devices, or augmented reality experiences in outdoor environments where carrying controllers impractical.
We observe that even temporally sparse observations, such as hand poses captured intermittently from egocentric videos during natural or periodic hand movements, can effectively constrain overall body motion. While it is possible to utilize other visible body parts such as feet or elbows, we opted to rely hand poses. This decision is based on the availability of hand pose detectors [24, 14] and the fact that hands are visible in approximately 20% vidco frames, as demonstrated in Table method incorporates
8. Unlike previous work that concentrated only spatial completion, temporal completion by leveraging the intermittent appearance of hands in egocentric videos. This dual completion approach not only enhances the robustness of body pose estimation under varying conditions but also reduces reliance on specific sensor hardware, making it more adaptable to various AR environments. In our setup, use temporally sparse 3D hand poses from detections in egocentric videos combined with dense head tracking signals to reconstruct the full body. Initially, we temporally complete sparse hand information using Masked Autoencoder (MAE) [11]. which estimates hand pose trajectories by capturing the spatiotemporal correlations between intermittent hand poses and head tracking signals. We develop a probabilistic extension of the MAE provide uncertainty estimates of the predicted hand pose sequence. Subsequently, using a conditional diffusion model, we spatially reconstruct the full body based on the head tracking signal data and imputed hand trajectories along with their predictive uncertainties. We call our approach DSPoser (Doubly Sparse Poser) because can effectively utilize data that is doubly sparse (sparse both temporally and spatially), as shown in Figure 1.
This flexible framework is designed seamlessly adapt diverse AR/VR setups and devices, ranging from spatially sparse scenarios (e.g.. using only head tracking signal or combining it with hand controllers) to doubly sparse scenarios (utilizing head signal data alongside hand detection from egocentric vidco). The key advantage lies in the assumption that the HMD's 's tracking signal is consistently available, enabling our approach function across a wide range of environments and hardware configurations. Extensive experiments have proved our model's versatility and accurate pose estimation capabilities various settings. Furthermore, our ablation studies highlight the significance of incorporating uncertainty estimates, as this crucial information enhances the overall quality of pose estimation, resulting more reliable outputs. By addressing both temporal and spatial completion through our double completion approach, we have developed robust and adaptable solution that reduces dependency on specific sensor hardware, making well-suited for immersive AR experiences in diverse scenarios, such sports training, outdoor environments, and beyond.

---

In summary, our research presents three key contributions:
A robust and versatile framework for egocentric body pose estimation tailored for HMDs. The framework adapts to various AR/VR settings and can leverage tracking signals available in most modern HMD devices without controllers.
We decomposed the problem into temporal completion and spatial completion. Our approach captures the uncertainty from hand trajectory imputation to guide the diffusion model for accurate full-body motion generation.
Extensive evaluations demonstrating the effectiveness of our framework on diverse datasets, outperforming existing methods and underscoring its potential for enhancing user interaction and immersion in AR experiences.
2 Problem Formulation
In our work, we aim to estimate the 3D human pose of a HMD user from sequences of RGB video and head tracking signal. We note that head tracking signal data is commonly accessible from IMU in most HMDs, such Meta Quest and Apple Vision Pro. Suppose we are given an egocentric video Vego = (Vi..... Vr,} where V, is an RGB image and Tw denotes the sequence length, and a corresponding head tracking signal sequence Thead = [Ti...., Tr } where T, Phood and Dhead is the dimension of head tracking signal including 3D pose. Our goal is estimate the full body pose = (P...... Pr. }. where pose state P, E '' at time T, is the number of body joints and D is the dimensionality of pose state. We solve the problem of estimating p(PIVego. Thead) by decomposing it into the stages of imputation and generation, assuming that we have temporally sparse hand data H. from hand detection module f-)) H. f(Vego). We first temporally complete hand trajectory based on and Thead, which can written as p [H] H. Thead). Then, we spatially complete full body pose P from the imputed hands H. and Thead which can be written as p(P|H, Thead). Since H is probabilistic variable, we need to marginalize over H as follows:
p(PPVego eggo: Thead) = p(P(H, Thead )p( (P(FL) Thead)-
3
Methods
Temporal Completion
Spatial Completion
Head tracking signal
Head tracking signal
XT
f
+
Encoder
Decoder
M
Pose
M
Linear
AdaLM
Attention
AdaLN
MLP
Decoder
VQ-VAE
Detector
H
Thead
P
Thcad
Vego
y ZT
Denoising
Zo
MAE
Transformer
Figure 2: Overall pipeline of proposed work DSPoser, composed of Temporal Completion stage and Spatial Completion stage to tackle pose estimation problem from doubly sparse data.
3.1 Detection: Hand Pose Estimation from Egocentric Video
In this work, we estimate the 3D position of the hand from an egocentric camera using a two-step process. First, we use FrankMocap [24] to predict hand poses SMPL-X parameters [22]. from which we extract local 3D hand joint positions relative to the root of the hand model's kinematic use RTM-Pose [14] estimate 2D hand joint positions in the image, H?D 2 Finally, determine the 3D hand joint positions in the camera coordinate system, H3D H.P d by solving for d E 3 that minimizes the reprojection
tree, denoted H3D E 21x3 Simultaneously,

---

error ||7, K + dd)ll. Here, K is the intrinsic matrix, obtained by transforming the original camera parameters into a pinhole model through undistortion.
To better constrain the hand trajectories, we attempted to obtain rotation information from the 3D) hand detection. However, due to the inconsistent quality of hand detection, the rotational information derived from the hand pose was noisy. Therefore, we decided not to incorporate this rotational information into our hand tracking approach on the Ego-Exo4D dataset. We utilized only the 3D wrist location from the Ego-Exo4D dataset, represented by Drand = 3. In contrast, for the AMASS dataset, we leveraged both rotational information and 3D location, as this data readily available, resulting in Drand= = 9.
3.2 Temporal Completion: Hand Trajectory Imputation from Sparse Hand Pose
Masked Auto-Encoder (MAE) In our work, we employed a Masked Autoencoder (MAE) [LJ] to impute missing hand trajectories using head tracking signal Thead and detected hand pose H. Inspired by Vision Transformer (ViT), we treated each T- and H. at time T as a token similar to an image patch in VIT. To accommodate this, we implemented two embedding layers, one for head tracking signal T E R Phcud and the other for hand H, R Phand, both projecting into the common token dimension D M- For the AMASS dataset, we follow the head tracking signal representation Dhead 18 in [13]. For the Ego-Exo4D dataset, Dreat 15, which includes head position and left/right IMU signals. Consequently, the total number of token amounts to x Tw. where 3 accounts for the head and both hands, and Tio is the sequence length. Sinusoidal positional encoding (PH) is used for both the encoder and decoder patches after tests showed it suffices for learning different modalities, compared to learnable PE. In an HMD environment, we assume that the head tracking signal Thead is always available, but hand visibility depends on the egocentric video. Thus, masking applied only the hand tokens based on their visibilities within egocentric view.
In contrast to the MAE [11] training approach. which maintains a consistent number of masked patches due to fixed masking ratio, the count of frames with invisible hand varies across instances in our setup. To address this variability, our encoder selectively applies attention masking to these inputs, ensuring that queries do not attend tokens where hand is invisible. This attention masking technique adapts dynamically to the fluctuating numbers of missing frames across the instances, enhancing the model's ability to handle data sparsity effectively. For decoder, we adopted MAE decoder design except the last projection layer to guide the uncertainty. To capture the uncertainty, we split the final projection layer into two heads for mean and variance of a Gaussian distribution.
Uncertainty-aware MAE Following the [26, 30J, to make the MAE aware of the predictive uncertainty of imputed hand pose sequence, employ the 3-NLL loss [26] function to manage uncertainty by using a set of mean heads (x,xx) and variance heads of (x), which are derived from M models initialized differently, where x = [H; T] is an input to the MAE and i E [1, MM.. The mean heads xx) and variance heads of (x) are trained using the Gaussian negative log- oogliikelhhood loss, which applies to each sample indexed by n with input x,, and ground truth hand pose sequence yn.
LB-NLL x,) = = (INLL(VN: Xa) where,
(2)
LNLL x,) log (xm)) (Hr(xn) y2)
2
(x7
The INLL loss function causes the predicted variance to act as a weighting factor for each data point, emphasizing those with higher variances. The parameter 3 adjusts the intensity of this weighting. The sg(-) function is used to apply the stop-gradient operation, thus preventing gradients from propagating through this part of the computation.
After training, we measure the aleatoric (data) uncertainty (-)e by averaging the variances across models, and epistemic (model) uncertainty Mepi() (-) by calculating variance of model means, and total uncertainty by adding both uncertainties:
(xie = (x)] to M - [x)
(4)
Uepa (x) Var. [x]]]
(5)
Uro (x) = Uate (x) + Lepi (x)
(6)
4

---

Note that Mate (-) and (ep provide uncertainties for each frame and each pose state dimension. The captured uncertainty visualized in Figure 3, demonstrating that MAE effectively captures uncertainty.
3.3 Spatial Completion: Uncertainty-guidet Body Pose Generation from Imputed Hand Trajectories and Head Tracking Signal
We employed the VQ-Diffussion [23] to generate full body poses from imputed hand trajectories and head tracking signal. The exposition of VQ-Diffusion can be found in Section D of the Appendix. As illustrated Figure 2, our motion generation module designed to generate human motion sequences from the temporally dense hand and head trajectories with uncertainty obtained from the MAE model.
VQ-VAE We first train the VQ-VAE to represent human motion with a discrete codebook repre- sentation described in Appendix D.I. We mostly followed the architectural design and training methods of [35]. After the codebook representation is learned by the VQ-VAE, we utilize this latent codebook representation to train denoising diffusion model.
Denoising Transformer Motivated by the work of VQ-Diffusion, we design a denoising trans- former that estimates the distribution p(zo|zz, y). An overview of our proposed model is depicted [4]. To incorporate the diffusion step t into
in Figure 2. We closely follow the implementation the network, we employ the adaptive layer normalization (AdaLN) [2, 15]. We concatenated the estimated hand and head trajectory with codebook after a embedding layer, to match the dimension with codebook representation. Finally, sequence.
the decoder to decode Zg to obtain a full body pose
Uncertainty Guidance We introduce several strategies to guide the denoising process using uncertainty estimates of imputed hand trajectories: sampling, dropout, and distribution embedding.
For sampling, we sample hand sequence from the distribution H NN (x), v (x)) and regard it as the conditioning vector y, where 2² (x) = (x)] x M- p. (x) and l" is measured by one of Eq. (9), (10), and (11). While it would be ideal to sample multiple times to better approximate the marginalization in Equation I, we find just using one sample provides a competitive
performance.
For dropout, set each dimension of (xx) zero with certain probability, which is determined by the corresponding dimension of Ld (x), and denote the result as y. The probability of the d-th dimension of μ(x) being zero is Pa =
(L7 (x) 2t- mma (x)) (2))/LLmmm. (mao (x) mg (x)) where Uz(x) is the d-th dimension of [1" (x). and Ugmin (x), Uggma (x) are the minimum and maximum values over the sequence length, respectively.
For distribution embedding [28], we embed the Gaussian distribution (Nu (x), (x) vector by concatenating the μ² (x) and U (x) in the feature dimension. The resulting embedding will be further concatenated with the head pose sequence form conditioning vector
4 Experiments
4.1 Datasets & Evaluation Metrics
Ego-Exo4D dataset Ego-Exo4D [9] contains simultancous captures of egocentric (first-person) and exocentric (third-person) video perspectives of participants performing complex activities like sports, dance, and mechanical tasks. The dataset comprises 1,422 hours of video ranging from 1 to 42 minutes per video. In addition to video, it provides camera poses, IMU data, and human pose annotations. Specifically for the egopose task, it includes separate training and validation video sets containing 334 and 83 videos respectively. Our problem formulation of ego body pose estimation differs from the ego body pose prediction task from [9], which aims to predict single future frame given a specific time window.
AMASS dataset The AMASS dataset [19] a large human motion database that unifies different existing optical marker-basee based MoCap datasets by converting them into realistic 3D human meshes
5

---

Table 1: Performance comparisons across baseline models for doubly sparse video data on the AMASS test set. We report MPJRE [9] MPJPE [em], and MPJVE [em/s], with the best results highlighted in boldface. Models trained by temporally sparse data, data indicates imputed data, and all other cases involve dense data. T, indicates the sliding window, x indicates the input of our whole pipeline, and y indicates the input of denoising Transformer.
marked with * The notation data denotes
Methads
T.
X
Imputation
MPIPE
MPIVE
MPIRE
VQ-VAE (Recons)
20
Full body
Full body
1.26
11.37
1.81
EgoEgoo [16]
20
19.17
46.17
7.30
Bodiffusion [3]
20
19.27
60.29
8.51
DSPoser (Ours)
20
12,08 04
75.07=0:26
7.04#0.02
DSPoser (Ours)
20
MAE
D 0
7.06-t0.02
28.26#0.05
5.00
Bodiffusion [3]
20
Interpolation
46.45
75.33
17.99
Bodiffusion [3]
20
D
MAE
D&
7.35
31.33
5.47
DSPoser (Ours)
20
0&
MAE
D&
5.51
24.19=00100
4.09
AvatarPoser [13]
0
Interpolation
D
40.42
64.07
16.37
AvatarJLM [36]
D&
Interpolation
25.02
68.42
14.14
AvatarPoser [13]
D&
MAE
&
9.88
62.31
5.98
AvatarJLM [36]
D
MAE
&
7.12
37.60
5.24
DSPoser (Ours)
D&
MAE
& 4
5.87
49.12-0.24
4.31 1:010
represented by SMPL [17] model parameters. Following the AvatarPoser [13] evaluation, used the CMU [5], BMLrub [29], and HDM05 [21] subsets from the AMASS dataset and their preprocessing of tracking signal information. Since AMASS does not include RGB images, we set Dhand assuming that 3D hand position and 6D rotation is available when the hand is "visible". To determine visibility, we compute the angle between the z-axis vector of the head rotation and the vector from the head position to the hand. We define the hand as "visible" if this angle is within 45" range, corresponding to 90" field of view (FoV) of HMD devices.
Evaluation metric We evaluate our results using the following metries: Mean Per Joint Position Error (MPJPE), Mean Per Joint Velocity Error (MPJVE), and Mean Per Joint Rotation Error (MPJRE), following the evaluation of [13, 3]. Since Ego-Exo4D dataset doesn't 't have the annotations for 6D rotation, MPJRE is reported only for AMASS. We report all values with the confidence interval of 95%. We also provide details on MPJPE across hands, upper body above the pelvis, and lower body below the pelvis, denoted as Hand PE, Upper PE, and Lower PE, respectively.
4.2 Full Body Pose Estimation from Doubly Sparse data
To demonstrate the effectiveness of our framework on doubly sparse egocentric video data, we investigated the results of our framework, DSPoser, on the AMASS dataset and Ego-Exo4D, as shown in Table and Table 2, respectively. Since the task of body pose estimation from doubly sparse data is newly introduced in our paper, we compare our results to other baselines, EgoEgo [16], Bodiffusion [3], AvatarPoser [13], and Avatar]LM [3]. Those baselines are designed to estimate human body poses from spatially sparse data. EgoEgo estimates body poses from head poses, and the others estimate body poses from head and hand tracking signals. We report the experimental results using the sampling strategy with alcatoric uncertainty unless otherwise stated. To train the baslines on temporally sparse data, we extend the algorithm as follows: (1) Interpolation: we imputed hand poses with linear interpolation; (2) MAE: we use our trained MAE to impute the hand trajectory. In T. = 1 setup, we report our result after averaging 16 samples while the result in T, 20 setup is from a single sample.
As shown in Table 1, DSPoser consistently outperforms baseline methods on AMASS across all metrics, underscoring the effectiveness of our two-stage approach for ego-body pose estimation. DSPoser achieves notable improvements in MPJPE for both sliding window sizes, T, = 20 and T, = For T, = 20, DSPoser reduces MPJPE from 7.35 em to 5.51 em, significantly outperforming the Bodiffusion extension, which uses MAE to impute invisible hands. For T, 1, DSPoser

---

Table Performance comparisons across baseline models for doubly sparse video data on the Ego- Exo4D validation set. We report MPJPE [em] and MPJVE [em/s], with the best results highlighted in boldface. Models trained by us are marked with * The notation Data denotes temporally sparse data, data indicates imputed data, and all other cases involve dense data.
Methods
T.
X
Imputation
y
MPJPE
MPJVE
VQ-VAE (Recons)
20
Full body
Full body
6.77
33.29
EgoEgo" [16]
20
29.49
47.50
8
Bodiffusion' [3]
20
28.56
109.71
DSPoser (Ours)
20
D
19.12=0.06
18.5.1th.11
DSPoser (Ours)
O
MAE
D&
18.46 #0.06
40.67
Bodiffusion' [3]
20
D O
Interpolation
D
59.81
120.12
Bodiffusion' [36]
20
D&
MAE
D
22.12
53.30
DSPoser (Ours)
D&
MAE
D&
16.84#0.04
39.86=t0.05
AvatarPoser' [13]
D&O
Interpolation
&
47.28
89.34
AvatarJLMM [36]
D&
Interpolation
D
43.01
61.98
AvatarPoser" [13]
D&
MAE
&
24.54
62.34
Avatar]LM" [3]
D&
MAE
21.08
45.77
DSPoser (Ours)
D&
MAE
&
19.09
55.82+0.27
Table 3: Performance comparisons across baseline models on the AMASS test set. We report MPJRE []] MPJPE [cm], and MPJVE [cm/s], with the best results highlighted in boldface. Note that is trained only with dense data without uncertainty.
Methods
MPIPE
MPIVE
MPJRE
Hand PE
Upper PE
Lower
FinalIK [25]
18.09
59.24
16.77
LoBSTR [34]
Q 0
9.02
44.97
10.69
VAE-HMD [7]
Q 0
6.83
37.99
4.11
ColIMoves [1]
5.55
65.28
4.58
AvatarPoser [13]
4.20
28.23
3.08
2.34
1.88
8.06
AvatarJLM [36]
3.35
20.79
2.90
1.24
1.72
6.20
DSPoser (Ours)
&
3.73=0.08
43.43=#0.14
2.94#00.0
3.26
1.92
6.53
achieves superior MPJPE compared Ava though it showswlimiatiions in MPJVE due the stochasticity of the diffusion model. In the experimental results presented in Table 2, our DSPoser model demonstrates superior performance on the Ego-Exo4| validation set. The model outperforms existing baselines, achieving a lower MPJPE of 16.84 cm, which represents an improvement over the next best model by 5.49 em. Additionally, DSPoser achieves an MPJVE of 39.86 cm/s, improving upon the basline of naive extension of Bodiffusion by 7.64 cm/s.
It is evident that by incorporating temporally sparse hand pose data, DSPoser framework signifi- cantly enhances pose estimation accuracy. For instance, on the AMASS dataset, MPJPE improved dramatically from 12.08 em to 5.51 em, while on the Ego-Exo4D dataset, improves from 19.12 em to 16.84 cm in T, 20 setup. This indicates that even sparse hand trajectory data, when effectively utilized, can provide crucial information for refining the accuracy of ego body pose estimation. Our method's 's ability to harness sparsely available data underscores its potential in applications where capturing dense sequence is challenging.
4.3 Full Body Pose Estimation from Spatially Sparse data
To demonstrate the versatility of our framework, we conduct experiments on spatially sparse video data. In the temporally dense data setup, where there no uncertainty regarding hand poses, the dense data directly works as condition for spatial completion on the right side of Figure 2. Table presents the results, demonstrating that DSPoser performs comparably to baseline models designed specifically for dense data setups on MPJPE and MPJRE metrics, underscoring the versatility of our dual approach in handling dense data scenarios. As discussed in Section 4.2, the higher MPJVE error results from the inherent stochasticity of the diffusion model.
7

---

Table Ablation study for uncertainty guidance
Table 5: Ablation study for different types uncer
strategy
tainty
Methods
MPJPE
MPIVE
MPRRE
Methods
MPJPE
MPIVE
MPJRE
w/o Uncertainty
6.05 0.01
30.124t.04
4.36
w/ Uncertainty
6.05
30.12 30.1240.04 12.00
4.3641000
Sample
5,51
24.1920.10
4.09
Epistomic
5.78*0003
27 27.55+0000
4.10
Distribution emb.
5.67=0.02
25.63+0002
4.1644002
Aleatoric
5,51 5,51
24.19*0.10
4.09
Dropout
5.55 5.55-9.02
25.10400.z
4.11 1.110
Total
5.59
25.65
4.16410.z
Table 6: Ablation study for 3 for
uncertainty capturing with MAE.
Table 7: Hand detection accu- racy on Ego- Ego-Exx4D dataset.
Table 8: Hand visibility ratio for
B
MPJPE (cm)
AMASS and Ego-Exo4D dataset
1.00
11.57
MPJPE (cm)
Num. Hands
0
2
Leh Hand 9.51
AMASS
82.77%
12.04%
5.19%
0.50
10.85
Right Hand
9.63
Ego Exo4D
72.95%
19.44%
7.61%
0.25
10.92
4.4 Ablation Studies
Based on the ablation study results shown in Tables 4 and we can analyze the impact of different uncertainty guidance strategies and types of uncertainty on the performance of the model for body pose estimation. The ablation study conducted with AMASS dataset with the sliding window T, = 20 to better analyze the effect of the uncertainty guidance. Table 4 investigates the effects of various uncertainty guidance strategies, including no uncertainty guidance, sample, distribution embedding, and dropout. The results suggest that incorporating uncertainty guidance through these strategies can improve the model's 's performance across different metries. The sampling strategy achieves the best performance, with the lowest MPJPE of 5,51, MPJVE of 24.19, and MPJRE of 4.09, indicating its effectiveness in capturing uncertainty and improving pose estimation accuracy.
Table 5 examines the contributions of different types of uncertainty, including epistemic uncertainty, aleatoric uncertainty, and total uncertainty. The results show that accounting for aleatoric uncertainty leads to the best overall performance. This suggests that considering data uncertainty can provide complementary information and improve the robustness of the pose estimation model. Overall, the ablation study highlights the importance of incorporating uncertainty guidance and considering different types of uncertainty in the model design for accurate and reliable body pose estimation.
In Table 6, we analyzed the effect of different 3 values on the AMASS dataset during the uncertainty capturing process of the Masked auto-Encoder (MAE). The results, shown in the table, indicate that B = 0.5 provides the best temporal completion for head and hand 3D positions from the doubly sparse input. Therefore, we / to 0.5 for training the MAE.
4.5 Hand Detection Accuracy and Hand Visibility Statistics
We investigate the error of the hand detector applied to the Ego-Exo4D dataset in terms of MPJPE, as shown in Table The detection results indicate an average error of less than 10 cm. We also analyze the visibility statistics for the AMASS and Ego-Exode dataset, at least one hand is visible in 18% of all frames with a 90 field of view (FoV), whereas in the Ego-Exo4D dataset, at least one hand is visible in 27% of all frames.
datasets in Table 8. In the AMASS
4.6 Qualitative Results
We visualized the alcatoric uncertainty in Figure 3, captured by a model trained using MAE on the AMASS dataset. In cases of partial visibility, as shown in Figure 3 (a-l) and (a-2), the uncertainty range is notably small. Conversely, in frames where the subject is completely obscured, the uncertainty range increases significantly. Even in fully invisible scenarios, the model captures range of uncertainty, likely influenced by head movements. Most of the estimated frames fall within the +20 range.
We also visualized the qualitative results on the Ego-Exo4D dataset and AMASS dataset in Figure The qualitative results for AMASS show that our method improves the estimation results when sparse
8

---

2-4xis (m)
1.00
ngght hand
right hand
1.3
right hign hand ragns
3.25
1.2-
1.00
2.90
0.75
0.50
2.80
0.9
0.00
0
40
50
100
120
100
2000
300
400
500
20
(a-1) Partially visible
(a-2)
Partially visible
(b) Fully invisible
Figure 3: Uncertainty visualization of the right hand pose captured by the MAE, Gray areas represent frames where the hand invisible, and white areas denote visible frames. We depict aleatoric uncertainty within ranges of +lo and =o from the estimated μ..
Ground Truth
(a) Video Frames
1
Ours
Head Only
(b) Ego-Exo4D Skeleton Groundtruth and Prediction
(c) AMASS Groundtruth and Prediction
Figure 4: (a) Ego-Exo4D video frames, (b) the corresponding skeleton ground truth and our prediction results, and (c) qualitative results on AMASS data under different input conditions. green indicates the ground truth, blue indicates the predicted result, and red indicates the visible hands. Head only estimates body pose from head trajectories, whereas Ours estimates body pose from imputed hand and head trajectories.
hand information is available, compared the Head Only results. Additionally, in the Ego-Exo4D results, the hands are more aligned compared to the lower body when hands are available.
Related Works
Human Pose Estimation from Sparse Input common capture setting in mixed reality involves using head-mounted device and hand controllers. Estimating full-body motion from the sparse input of head and hand movements is challenging. Recently, several methods have been proposed to tackle this: AvatarPoser [13] is the first learning-based method predict full-body poses in world coordinates using only head and hand motion inputs. It uses a Transformer encoder to extract deep features and decouples global motion from local joint orientations, refining arm positions with inverse kinematics for accurate full-body motion. BoDiffusion [3] employs a generative diffusion model for motion synthesis. addressing the under-constrained reconstruction problem. It uses time and space conditioning scheme to leverage sparse tracking inputs, generating smooth and realistic full-body motion sequences. AvatarJLM [36] uses two-stage framework where sparse signals embedded into high- -igh-dimensional features and processed by an MLP to generate joint-level features. These features are then converted into tokens and fed into transformer-bee
network to capture spatial and temporal dependencies, with an SMPL regressor transforming them into 3D full-body pose sequences. HMD-poser [6] combines a lightweight temporal-spatial feature learning network with regression layers and uses forward kinematics to achieve real-time human motion tracking. AGRoL [8] utilized conditional diffusion model to generate full body pose from sparse upper-body tracking signals. worth noting concurrent work, EgoPoser [12], which also addresses ego body pose estimation

---

from doubly sparse observations. Their focus lies in preparing training data through field-of-view (FoV) modeling rather than introducing new algorithms. Our work orthogonal theirs, providing algorithmic contributions through a multi-stage pipeline including an uncertainty-an masked auto-encoder (MAE).
Human Body Pose Estimation from Egocentric Videos Estimating full 3D human body pose from egocentric vidcos is an ill-posed problem due to the partial visibility of wearer's body parts from the camera mounted on wearer's head. Recently, several approaches have been proposed to address this challenge. EgoEgo [16] integrates SLAM and a learned transformer to estimate head motion, then leverages estimated head pose to generate plausible full-body motions using diffusion models. [18] designs a kinematic policy to generate per-frame target motion from egocentric inputs, and leverages a pre-learned dynamics model to distill human dynamics information into the kinematic model. GIMO [37] integrates motion, 3D eye gaze, and 3D scene features to generate gaze informed long term intention-awan human motion prediction. [32] leverages external camera to generate pscudo labels to estimate full 3D body pose from single head mounted fish eye camera using weak supervision. [33] estimates geometry of surrounding objects and extracts 2D body pose features using EgoPW [32] to regress 3D body pose with a voxel-to- toxelll-voxeel network [20].
Conclusion
In this paper, we have addressed the problem of egocentric body pose estimation using temporally sparse observations from head-mounted displays (HMDss). By leveraging both temporal and spatial completion, our approach effectively utilizes intermittent hand pose detections from egocentric videos, alongside consistently available head pose data, reconstruct full-body motions. Through comprehensive experiments on datasets such as AMASS and Ego-Exo4D, we have demonstrated the effectiveness of our framework. Our results indicate significant improvements over existing methods, particularly in scenarios where dense sensor data may not be available or practical. This advancement opens up new possibilities for beneficial augmented reality experiences in various applications. including sports training by providing feedback on body mechanics, and other scenarios where users need move freely without additional sensors such as hand controllers. However, our method has not been explicitly tested for faimess across different demographic groups. Potential biases in the datasets used could result in uneven performance across various user populations. Careful curation of training datasets is necessary to prevent unfair failures for underrepresented groups
7 Limitations
While our proposed method for estimating the body movements of a camera wearer from sparse tracking signals shows promising results, several limitations should be acknowledged. Firstly, our method has been tested with only one type of sparse body part tracking signal, specifically the hand. Incorporating the detection of other body parts, such as feet and elbows, may improve overall body pose estimation. Additionally, variations in lighting, occlusions, and the quality of the egocentric video can impact the accuracy of hand pose detection, subsequently affecting the overall body pose estimation.
The effectiveness of our method was validated using the AMASS and Ego-Exo4D datasets. Although these datasets are comprehensive, they may not encompass the full spectrum of possible real-world variations. Our study focused on pose estimation within a window size of less than few seconds, following standard settings from the literature. It remains unclear how our method will perform with larger window sizes. Furthermore, the scalability of our method with larger datasets has not been thoroughly evaluated. The use of diffusion models for pose estimation may limit its utility for real-time applications due to their inference speed. Additionally, using multiple models to compute epistemic uncertainty can be computationally intensive.
Acknowledgement We acknowledge Feddersen Chair Funds and the US National Science Founda- tion (FW-HTF 1839971, PFI-TT 2329804) for Professor Karthik Ramani. Any opinions, findings, and conclusions expressed this material are those of the authors and do not necessarily reflect the views of the funding agency. We sincerely thank the reviewers for their constructive suggestions.
10

---

References
[1] K. Ahuja, E. Ofek, M. Gonzalez-Franco, C. Holz, and A. D. Wilson. Coolmoves: User motion accentuation in virtual reality. Proceedings of the ACM Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(2):1-23, 2021.
[2] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization, arXiv preprint arXiv:1607.06450, 2016.
[3] A. Castillo, M. Escobar, G. Jeanneret, A. Pumarola, P. Arbeláez, A. Thabet, and A. Sanakoyeu. Bodiffusion: Diffusing sparse observations for full-body human motion synthesis. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 2023.
[4] S. Chi, H. Chi, H. Ma, N. Agarwal, F. Siddiqui, K. Ramani, and K. Lee. M2d2m: Multi-motion generation from text with discrete diffusion models. In European conference on computer vision. Springer, 2024.
[5] CMU Graphics Lab. Cmu graphics lab motion capture database. http://mocap.. cs. cmu. 2000,
[6] P. Dai, Y. Zhang, T. Liu, 7. Fan, T. Du, Z. Su, X. Zheng, and Z. Li. Hmd-poser: On- device real-time human motion tracking from scalable sparse observations. arXiv preprint arXiv:2403.03500 2024.
[7] A. Dittadi, S. Dziadzio, D. Cosker, B. Lundell, J. Cashman, and J. Shotton. Full-body motion from a single head-mounted device: Generating smpl poses from partial observations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 11687- 11697, 2021.
[8] Y. Du, R. Kips, A. Pumarola, S. Starke, A. Thabet, and A. Sanakoyeu. Avatars grow legs: Gen- erating smooth human motion from sparse tracking inputs with diffusion model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 481 481-490, 2023.
[9] K. Grauman, A. Westbury, L. Torresani, K. Kitani, J. Malik, T. Afouras, K. Ashutosh, V. Baiyya, Bansal, B. Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. arXiv preprint arXiv:2311.18255 2023.
[10] S. Gu, D. Chen, Bao, Wen, Zhang, Chen, L. Yuan, and B. Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10696-10706, 2022.
[11] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[12] J. Jiang, P. Streli, M. Meier, and C. Holz. Egoposer: Robust real-time ego-body pose estimation in large scenes. In European conference on computer vision. Springer, 2024.
[13] J. Jiang, P. Streli, H. Qiu, A. Fender, L. Laich, P. Snape, and C. Holz. Avatarposer: Articulated full-body pose tracking from sparse motion sensing. In European conference on computer vision, pages 443-460. Springer, 2022.
[14] T. Jiang, Lu. Zhang, Ma, R. Han, C. Lyu, Y. Li, and K. Chen. Rimpose: Real-time multi-person pose estimation based on mmpose. arXiv preprint arXiv:2303.07399, 2023.
[15] K. Lee, H. Chang, L. Jiang, H. Zhang, Z. Tu, and C. Liu. VITGAN: Training GANs with vision transformers. In International Conference on Learning Representations, 2022.
[16] Li, K. Liu, Jnd J. Wu. Ego-body pose estimation via ego-head pose estimation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 17142-17151, 2023.
11

---

[17] M. Loper, N. Mahmood, J. Romero. G. Pons-Moll, and M. J. Black. Smpl: A skinned multi- person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 851-866. 2023.
[18] Z. Luo, R. Hachiuma, Y. Yuan, and K. Kitani. Dynamics-regulated kinematic policy for egocentric pose estimation. Advances in Neural Information Processing Systems, 34:25019- 25032, 2021.
[19] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. Black. Amass: Archive of motion capture surface shapes. In Proceedings of the IEEE/CVF international conference computer vision, pages 5442-545 1222--45,1110 2019.
[20] G. Moon, J. Y. Chang, and K. M. Lee. V2v-posenet: Voxel-to-voxel prediction network for accurate 3d hand and human pose estimation from single depth map. In Proceedings of the IEEE conference on computer vision and pattern Recognition, pages 5079-5088, 2018.
[21] M. Müller, T. Röder, M. Clausen, B. Eberhardt, B. Krüger, and A. Weber. Documentation mocap database hdm05. Computer Graphics Technical Report CG-2007-2, Universitäl Bonn, 7:11, 2007.
[22] G. Pavlakos, V. Choutas, N. Ghorbani, Bolkart, A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3d hands, face, and body from single image. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pages 10975-10985, 2019.
[23] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[24] Y. Rong, T. Shiratori, and H. Joo. Frankmocap: A monocular 3d whole-body pose estimation system via regressiin and integration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1749-1759, 2021.
[25] RootMotion. Final ik. https://assetstonn htnp:/ttttttteee.t.......tt wwwtt.tiitttpteeeecc http:///aaccaage//ooolleeee. /at.oliioniimmiooo///oomoooooo 1aaal-ik-129900 2018
[26] M. Seitzer, A. Tavakoli, D. Antic, and G. Martius. On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks. arXiv preprint arXiv:2203.0911 096668, 2022.
[27] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256 2265. PMLR, 2015.
[28] B. Sriperumbudur, A. Gretton, K. Fukumizu, B. Schölkopf, and G. Lanckriet. Hilbert space embeddings and metrics on probability measures. Journal of Machine Learning Research, 11:1517-1561, Apr. 2010.
[29] N. F. Troje. Decomposing biological motion: A framework for analysis and synthesis of human gait patterns. Journal of vision, 2(5):2-2, 2002.
[30] M. Valdenegro-Tory and D. S. Mori. A deeper look into aleatoric and epistemic uncertainty disentanglement. In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1508-1516. IEEE, 2022.
[31] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning, Advances in neural information processing systems, 30, 2017.
[32] J. Wang, L. Liu, W. Xu, K. Sarkar, D. Luvizon, and C. Theobalt. Estimating egocentric 3d human pose in the wild with external weak supervision. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13157-13166, 2022.
[33] J. Wang, D. Luvizon, W. Xu, L. Liu, K. Sarkar, and C. Theobalt. Scene-aware egocentric 3d human pose estimation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 13031-13040, 2023.
12

---

[34] D. Yang. D. Kim, and S.-H. Lee. Lobstr: Real-time lower-body pose prediction from sparse upper-body tracking signals. In Computer Graphics Forum, volume 40, pages 265-275, Wiley Online Library, 2021.
[35] J. Zhang, Y. Zhang, X. Cun, Y. Zhang, H. Zhao, H. Lu, X. Shen, and Y. Shan. Generating human motion from textual descriptions with discrete representations. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14730-147400 2023.
[36] X. Zheng, Z. Su, C. Wen, Z. Xue, and X. Jin. Realistic full-body tracking from sparse the IEEE/CVF International Conference
observations via joint-level modeling. In Proceedings the on Computer Vision, pages 14678-14688, 2023.
[37] Y. Zheng, Y. Yang, Mo, Li, T. Yu, Y. Liu, C. Liu, and L. J. Guibas. Gimo: Gaze- informed human motion prediction in context. In European Conference on Computer Vision, pages 676 694. Springer, 2022.
13

---

A Additional Details
A.1
Architecture & Experimental Details
Training step
Times
Module
of Params
MACs
Time
33
100
(ms)
VQ-VAE
17.9 M
3
3 ms
25
5.83
5.92
5.69
8.72
278
33
5.67
5.63
5.58
348
MAE
51.3 M
G
4 ms
50
5.61
5.53
522
VQ-Diffusion
74.2 M
1190.2
958 ms
100
5.51
1013
Table 9: Experimental results for different modules
Table 10:
Performance results for different in-
ference and training step combinations
VQ-VAE We adhered to the architectural details and training protocol of Zhang al. [35], with modifications including setting both the encoder and decoder stride to I, and adjusting the window size 40. For the Ego-Exo4D dataset, we employed wing loss with width of 5 and curvature of 4. For AMASS, we opted for 1.2 loss. Additionally, to generate smooth motion, we applied both velocity and acceleration losses, assigning weights of 10 for each in the AMASS dataset, and weights of for each in the Ego-Exo4D dataset. The input shape is represented as Twr x J x D Datt. where J the number of joints, Tw time window, and Danta is the data representation. For AMASS, which uses 6D rotation representation, J = 22 and Dãata 6. For Ego-Exo4D, J = 17 and Datta 3. We use Tw 40 for both AMASS and Ego-Exo4]]
We employed the same hyperparameters and training specifications as Gu et al. [10] for training VQ-Diffusion. Additionally, we replaced the absolute positional encoding with relative positional encoding, following the implementation of VQ-Diffusion for human motion generation proposed by Chi et al. [4]. We replaced the text condition module with the uncertainty- aware MAE module to feed the imputed trajectory and uncertainty as a conditional input.
VQ-Diffusion
Masked Auto-Encoder We adapted the encoder accommodate a variable number of visible hands and modified the last projection layer to guide the uncertainty, as detailed in the main paper. Otherwise, we followed the training details provided by He al. [11]. We trained models to measure the uncertainty, M = 4.
Dataloader We adopted the dataloader of AMASS and evaluation configurations from Bodiffusion [3] for our experiments. For the Ego-Exo4ID dataset, we employed the dataset implementation from the Ego-Exo4D [9].
Analaysis Computaional Cost We evaluated the computational cost of our approach by ana- lyzing the number parameters, multiply-acemmulate operations (MACs), and inference time for each module in our pipeline. The reported times in Table 10 represent the total measured time for the entire pipeline, whereas Table measures the time only for the corresponding module, excluding overhead between modules. As shown in Table 9, the VQ-Diffusion model is responsible for the majority of MACs and inference time. To mitigate this, we conducted further experiments to explore ways of reducing the VQ-Diffusion model's inference time. Table 10 presents the trade-off between performance and inference time based on the number diffusion steps, offering multiple options. Notably, training with 50 steps and inferring with 25 steps yields approximately 4x faster inference time with only about 3% reduction in performance.
B Compute Resource
We ran our experiments on one workstation, containing AMD Ryzen Threadripper PRO 7975WX.
DDR5 RAM 256GB and 4 NVIDIA GeForce RTX 4090. AMASS dataset takes 512GB of storage and Ego-Exoo Exo4D dataset takes ITB. One training run took around 18 hours on GPU for AMASS and around 12 hours for Ego-Exo4D.. Inference over AMASS validation set takes 40 minutes on 1 GPU and inference over Ego-Exo4D validation set takes 10 minutes on GPU. In total, all experiments including preliminary failed experiments took approximately 300 GPU-hours.
14

---

C Licenses for Assets Used in the Paper
Code We use the code of BoDiffusion [3] which is available https//giihub..om/BBYY Uniandes/BoDfftssion.. Unfortunately, we could not locate the licensing terms for the source code. For the Masked Auto Encoder, we use the implementation available at pyytooo but we could not find the licensing terms for this
source code.
We also employ VQ-Diffusion [10]. available at https://github.co https//github.coo/ciinng/// con/Ccentg///V/V biffuiioatceemmiinnaarrrrdemeeo Program.
which is licensed under Microsoft's Open Source
For VQ-VAE, use the implementation from T2M-GPT [35], which can be found at http:/iiww..cmm/iel//////-----/- and is licensed under the Apache License 2.0.
For 3D hand detection, we use the code of FrankMoCap [24]: https://github. wowm//accboookssar//ttaaeeeeeeee
which licensed under the CC BY-NC 4.0 license. under
We also used RTM-pose [14], which is available at wttp:///www..oooooooommmmaaa///a the Apache License 2.0.
Data We use the Ego-Exo4D dataset [9] https://eoo-exo-deeeo tom (commercial or non-commercial) license. We also use AMASS [19] https//amass.s..ee.mmpp which is licensed under a custom (non-commercial scientific research) license.
which is licensed under a cus-
D
Preliminary
D.1 Discrete Diffusion Model
Discrete diffusion models [10] represent category of diffusion models that progressively introduce noise into data while training to reverse this process. In contrast to continuous models, such as a latent diffusion model [23], which manipulate data in continuous state space, discrete diffusion models operate within discrete state spaces.
VQ-VAF Vector Quantized-Variie Variational Autoencoder (VQ-VAE) [31] is generative model that extends the concept of Variational Autoencoders (VAEs) by incorporating discrete latent representa- tions via vector quantization. The encoder Exx) compresses input data x into discrete latent vectors by mapping each encoded representation to the closest vector Zu to the nearest codebook entry from learned codebook of prototypes using the nearest-neighbor search: Z, - Q(z) = argmin c, cl c,l12 Ck) where K is the total number of codebooks. The decoder D(z,)) recon-
Here, C (ci,..., structs the input data x from these quantized vectors, yielding a reconstructed output & = D(2,)) The optimization process involves minimizing combination of reconstruction loss and commit- ment loss. The reconstruction loss is expressed as ||x - 8||2. while the commitment loss en- sures the encoder commits to the nearest prototype in the codebook: ||sg[za] z/12. where sg is the stop-gradient operator. The overall loss function, which the VQ-VAE model minimizes, is: Lvo ||x x|| +|1 sann-szzzll sgiz]ll2+ \vol sg/zg/ z||2. Here, Avg a coefficient for the commitment loss.
Forward Diffusion Process. Building on the foundation laid by the discrete diffusion models introduced by [27]. VQ-Diffusion [10] refined the diffusion process with and--lee strategy. In VQ-Diffusion, during the forward diffusion process, tokens can either transition other tokens or to special <MASK> token. The transition probability from token z' to z at diffusion step t is defined by the matrix ji.. The transition matrix Q₁ structured in R (K (1),, follows:
Q
Q
0
where Q = a,I 3,111.
Here, a₁ adjusts ensure conservation of probability, such that a, 1 K3 - is the probability of transitioning between tokens, and " governs transitions to the <MASK> token. The transition from step to is expressed as: (24./2-1) = o (z, m(4,))),, where v(z.) e (K+1x1x is an one-hot encoded vector representing the token index of z₁. Using the Markov property, the
15

---

probability of transitioning from any initial step 0 to step is Zo) (2) where Q. QQQ Q1 is the cumulative transition matrix. This defines the cumulative probabilities as Oii Te - I1 (1 7)), and B, (1 0z -k)
Conditional Denoising Process. In the conditional denoising process, a neural network denoted as aims to predict the original, noiseless token Zo given corrupted token and the associated condition, such as a embedded hand trajectories. The posterior distribution for the discrete diffusion process can be defined as:
q( 21 20) g(Zt-1 Zn)g(Ze (z 22:-1 Zo)
g(2e [oo
vv (z:) Q, (Zt (2 (2-
(8)
v (Zz) Q20(Z))
With this, the reverse transition distribution is determined as:
Po(2:-1/2:.3> = -2.=1 20 po @0 Zz, y),
(9)
where the network iteratively denoises tokens from step down to 1, eventually generating token conditioned on To train the network po. the training approach includes not only denoising objective but also the standard variational lower bound objective [27], denoted L.vb- The comprchensive training objective is:
L = Lith + xex,)) (zol
(10)
where À is the coefficient for the denoising loss.
E Additional Experimental results
We illustrate additional uncertainty visualization Fig. 5. In addition, demonstrate the additional qualitative results with our method on Ego-Exo4D dataset and AMASS dataset in Fig. 6 and Fig. & respectively.
Z-axis (m)
right
125
1230
20
200
360
Frame
Figure 5: Additional uncertainty visualization of the right hand pose captured by the MAE, Gray represent frames where the hand is invisible, and white areas denote visible frames. We depict aleatoric uncertainty within ranges of #1 and #20 from the estimated μ..
16

---

Figure 6: Qualitative results showing the groundtruth in green and predicted human pose in blue using our method on the Ego-Exo4D dataset.
17

---

Bodiffusion
High Error
AvatarPoser
AvatarJLM
DSPoser
Error
GroundTruth
Figure 7: Qualitative results on AMASS dataset comparing DSPoser (Ours) against the baselines. Color gradient indicates an absolute positional error, with higher error corresponding to higher blue intensity. Results demonstrate that motions generated by DSPoser exhibit greater similarity to the ground truth. Furthermore, highlights higher errors (indicated with red circles) for baselines when the hand is occluded in the ground truth pose (indicated with black circle).
18

---

Ground Truth
PPPPP!
Ground Truth
RITERI
Ours
Ours
Ground
Ground
Owrs
Figure 8: Qualitative results showing the groundtruth in Green and predicted human pose blue using our method on AMASS dataset, with red indicating the visible hands.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims accurately reflect this paper's empirical contributions. The introduction section discusses contributions made in the paper and important assumptions and limitations. Limitations of work are described in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations our work are described in Section 7.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Implementation details explained in Section 3 and Section A.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Code release is challenging due to our organization's policy.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.c cc/ http://dddssseeeuieee
wwwssssissoolloooeo ooiicyyyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: gwtp///ie////eee// /aaeee///ooeumiiiic iesiissssoooooo
/ /nips.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We specified the experimental setting and details in Section A. and Section 4.1.
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report 95% confidence interval in the experimental tables in Section 4. The confidence interval was computed using closed form formula. The variability due to random drawing of noise vectors in denoising diffusion and uncertainty guidance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources used in this paper can be found in Section B of the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: //te::///eeere http//neurrp..eeee cc httpppulllitiicccipeeeeeepeeeeee
Answer: [Yes]
Justification:
Our research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes],
Justification: Both potential positive societal impacts and negative societal impacts of the work are discussed in Section 6.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We believe this paper does not pose high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Citations and licenses of assets used in this paper be found in Section C of the Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "C3S0eV0qd7", "reviewer": "Reviewer_9VxB", "summary": "The paper introduces an approach for estimating full-body pose from egocentric videos combined with sparse head and hand positions. The key contribution lies in utilizing sparse temporal annotations of hand positions to achieve a complete representation. The method is evaluated on the publicly available Ego-Exo4D and AMASS datasets, demonstrating performance that surpasses current state-of-the-art approaches.", "questions": "**Justification:**\n\nOverall, the paper has clear contributions and thorough quantitative comparisons against previous state-of-the-art approaches. However, the claims about the method's use in AR and VR applications could be substantiated better, and there is a lack of qualitative comparisons.\n\n**Additional comments:**\n\n- There are missing articles (e.g., \"the\") in multiple instances (L84, L124).\n- The phrase \"using the transformer architecture\" in L155 is redundant.\n- Correct \"L161 strategies\" and \"L45 signals\".", "limitations": "The paper adresses the limitations. \n\nOne of the main limitations is the lack of publicly available code, which will hinder progress in the community.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Relevance of the task/scope of the paper:**\n  - The task of estimating full-body pose from egocentric video and sparse positional signals is relevant for the NeurIPS community, and the paper presents this relevance adequately.\n  \n- **Technical novelty of the approach:**\n  - The approach utilizes both images and sparse information from the hands and head, which is a novel idea. The use of a masked autoencoder to complete the information from the hands is also innovative.\n  \n- **Technical correctness of the paper:**\n  - The methods section describes the proposed approach clearly.\n  \n- **Related Work:**\n  - The review of related work is comprehensive, covering all relevant literature.\n  \n- **Experimental validation:**\n  - The experiments are evaluated on the AMASS and Ego-Exo4D datasets, with a thorough comparison against current state-of-the-art approaches. The quantitative results show improvement over previous methods.\n  \n- **Writing and presentation:**\n  - The paper is mostly well-written, and the ideas are conveyed clearly despite a few typographical errors.", "weaknesses": "**Weaknesses:**\n\n- **Technical contributions:**\n  - The key contributions could be consolidated into one. The first and second contributions are similar, and the third discusses the potential for AR experiences without providing experiments involving real-life AR devices.\n  \n- **Experimental validation:**\n  - The paper lacks qualitative comparisons against state-of-the-art approaches and video visualizations of the results. Given the low MPJPE and MPJRE values in current methods, visual assessments are important to ensure the improvements are not just quantitative artifacts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your acknowledgment of our paper's novelty and your valuable suggestions for improving our method.\nWe fully recognize the importance of open-source code in advancing research, as this paper is also built upon the publicly available code of other researchers. \n\n> Q1: Key contributions can be consolidated into one & no experiments involving real-life AR devices.\n\nWe appreciate the feedback regarding the similarity between our first and second contributions. Our primary aim was to highlight different aspects of our approach, but we understand the need for clarity and consolidation.\n Additionally, while the dataset we used in our experiments, EgoExo4D, is collected using Meta's Aria Devices (an HMD/AR device) and demonstrates the potential for AR applications, we recognize that explicit experiments involving real-life AR devices would strengthen our claims. We will consider this in future work.\n\n> Q2: Qualitative comparisons against state-of-the-art approaches and video visualizations of the results.\n\nWe appreciate your comment on the importance of qualitative comparisons. While we have visualized our qualitative results in a video format, we found that it is not permitted to include links during the rebuttal phase. As an alternative, we have included detailed comparative qualitative results in the PDF file attached to this rebuttal. We hope this additional information provides clarity and supports the quantitative improvements we reported. We also ensure that we will add video comparisons against state-of-the-art methods in the final version.\n\n> Q3: Lack of publicly available code.\n\nWe recognize the value of open-source code and plan to release ours upon acceptance. Our organizational policy prioritizes careful review and preparation before any public release. In the meantime, we're committed to providing detailed methodologies in our publications to support reproducibility and further research in the field.\n\n> Q4: Misc.\n\nThank you for pointing out the grammar mistakes and writing issues. Following the reviewer's comments, we will:\n\n1. Conduct a thorough grammar review of the manuscript, including the ones you mentioned.\n\n2. Remove the phrase \"using the transformer architecture\" in L155.\n\nWe believe these changes will improve the clarity and readability of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L6KslCRSsP", "reviewer": "Reviewer_2EAc", "summary": "This paper presents a new method for ego-body pose estimation from egocentric videos. Compared to previous methods that assume hand tracking signals are always available, this paper focused on the case that hand poses captured intermittently from egocentric videos. To solve this, this paper proposes a two-stage method that firstly does temporal completion for hand trajectory and then spatial completion for full-body poses. Experiments show better performances than selected baseline methods.", "questions": "1. The authors acknowledged that using diffusion models could limit the usage of real-time applications. What is the motivation for using diffusion models for this task? Some recent papers like [2] even show AvatarPoser performed better than Diffusion models. In Table 3, the previous method, AvatarJLM, also shows better performance than the proposed method.\n\n    [2] Ma et al. Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wild, arXiv 2024\n\n2. IMU measurements are mentioned multiple times in this paper. Are they used in the paper? If yes, how?\n\n3. Will the code be published?", "limitations": "Limitation has been discussed in Section 7.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and very easy to follow. The figures and tables are well presented.\n2. I like the point that the authors use the hand poses captured intermittently from egocentric video data instead of assuming dense tracking signals. The two-stage method also sounds reasonable for this case.", "weaknesses": "1. It would be interesting to see how the proposed method would compare to the FoV modeling in [1], which focused on the same task setting when the hand tracking signals are intermittent. However, [1] is not discussed in the submission.\n\n    [1] Jiang et al. EgoPoser: Robust Real-Time Ego-Body Pose Estimation in Large Scenes, arXiv 2023\n\n2. In Table 3, when trained only on dense data, the proposed methods performed worse than previous methods. So, I was wondering whether the model generalizes well to different settings.\n\n3. Metrics related to computational complexity, such as the number of parameters, FLOPs, and inference time, are not provided. This is important to ensure a fair comparison with previous methods, and to see the potential in real-world applications.\n\n4. Sota methods like AvatarPoser and AvatarJLM are only compared in Table 3 but not in Tables 1 and 2. It would be better to have consistency when comparing methods.\n\n5. Some technical details seem missing. For example, what are the lengths of input and output frames? How is an evaluation made during the evaluation (e.g., the step size of the sliding window)?\n\n6. There are no video comparisons provided, which are important for this task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewers' insightful comments for our paper.\n\n> Q1: FoV modeling result of EgoPoser (ECCV 2024).\n\n\\begin{array}{c|cc|cc|cc}\\hline\n\\textbf{Strategies}&\\textbf{MPJPE (180°)}&\\textbf{MPJVE (180°)}&\\textbf{MPJPE (120°)}&\\textbf{MPJVE (120°)}&\\textbf{MPJPE (90°)}&\\textbf{MPJVE (90°)} \\\\\\\\\\hline\n\\text{EgoPoser}&5.31&39.69&6.07&46.01&6.60&48.25 \\\\\\\\\n\\text{DSPoser (Ours)}&4.80&22.58&5.28&23.13&5.51&24.19 \\\\\\\\\\hline\n\\end{array}\n\nThank you for bringing EgoPoser (ECCV 2024) to our attention. We were unaware of this work at the time of our submission, and it is encouraging to see parallel interest in ego body pose estimation from intermittent observations. We will ensure to cite this work in the next version of our paper.\n\nRegarding contributions, we understand that EgoPoser's main focus is on preparing training data based on field-of-view (FoV) modeling rather than random masking. While we share an interest in FoV considerations, our work offers distinct algorithmic contributions, including:\n\n1. A multi-stage approach to pose estimation.\n\n2. An uncertainty-aware masked auto-encoder (MAE).\n\nThese aspects of our work were recognized by reviewers as innovative contributions to the field. We believe our approach complements the ideas presented in EgoPoser.\n\nMoreover, the experimental data presented in the preceding table demonstrates that our approach surpasses EgoPoser in performance, with notably superior results in scenarios involving a narrow field-of-view (FoV).\n\n> Q2: Generalization on Ego-Body Pose Estimation given dense data.\n\nPlease see the table in Q3 of the Author Rebuttal.\n\nThe quantitative results of our method are influenced by the performance of the VQ-VAE component. Since Table 3 in the main paper is intended to show the versatility of our framework rather than argue our main contribution, we did not conduct an extensive hyper-parameter search to achieve state-of-the-art performance initially. To address this, we performed hyperparameter tuning on the VQ-VAE to mitigate performance loss attributed to this component. Following these adjustments, we observed improved results compared to other methods.\n\nThe results demonstrate that while our method may not exhibit the best performance across all metrics, it consistently shows at least the second-best performance for all metrics. A one-to-one comparison with other baselines reveals that our method outperforms them in at least 3 out of 4 metrics. Specifically, we included the Jitter metric to provide a comprehensive analysis of our method, as suggested by Reviewer VMPC. Our method significantly achieves better smoothness, indicating that our approach generates smoother motion close to the ground truth in terms of Jitter.\n\n> Q3: Metrics related to computational complexity.\n\nPlease see tables in Q2 of Author rebuttal, or the Table B and C in the pdf.\n\nWe recognize the importance of including metrics related to computational complexity. While we have roughly provided the inference time in the supplementary material, we agree that a more detailed comparison would be beneficial. Including metrics such as the number of parameters, MACs, and inference time will offer a more comprehensive comparison with previous methods and better illustrate the potential of our approach for real-world applications. We will incorporate these details in the revised version of our paper.\n\n> Q4: Consistency between tables\n\nPlease refer to table in Q1 of the Author Rebuttal, or Table A in the pdf file.\n\nThank you for highlighting this inconsistency. Following the reviewer's comment, we plan to update the Tables 1 and 2 to include comparisons with AvatarPoser and AvatarJLM with various imputation method, ensuring that relevant state-of-the-art methods are consistently evaluated throughout the paper. However, due to limited time and resources, we are currently only able to provide baseline results on the AMASS dataset. We are working on training baseline methods on the Ego-Exo4D dataset and will update the results as soon as they are completed.\n\n> Q5: Missing details\n\nThank you for bringing this to our attention. We realize that some technical details were not clearly outlined in our submission. As described in Supplementary Section A., the window size is set to 40 frames, so the lengths of the input and output frames can be inferred as 40 frames from the explanation in the Preliminary section. However, we acknowledge that this is not clear and easy to catch, so we will explicitly state these details in the main paper.\nAdditionally, we will specify the sliding window size, which is set to 20 frames. Upon a thorough review of the paper, we also noticed that some details of the hand detectors, such as how we handled visibility for the Ego-Exo4D dataset, were not clearly stated. We will ensure that these details, along with the step size of the sliding window used during evaluation, are clearly presented in the main text to provide a complete understanding of our methodology.\n\n> Q6: Qualitative comparisons against SoTA\n\nPlease refer to the Q4 of the Author Rebuttal. \n\n> Q7: the motivation for using diffusion models\n\nPlease refer to the Q2 of the Author Rebuttal. \n\n> Q8: How we used IMU information.\n\nThe Ego-Exo4D dataset, collected by Meta using the Aria device, includes head trajectory data processed from IMU measurements. In the paper, we refer to this head trajectory as the tracking signal from the IMU, which is integral to our analysis and experiments. For the AMASS dataset, while IMU data is not used, we follow the approach of previous works such as AvatarPoser, where aggregated data of joint pose, joint velocity, 6D rotation, and angular velocity mimic the tracking signals from IMU. Both AvatarPoser and Avatar JLM have demonstrated that models trained with these signals are applicable to real-world data from AR devices.\n\n> Q9: Lack of publicly available code.\n\nPlease refer to Q5 of the Author Rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rgIsJfL1Mo", "reviewer": "Reviewer_jBAY", "summary": "The paper introduces the task of full-body pose estimation from temporally and spatially sparse tracking inputs. It differs from the prior work in assuming only the partial availability of hand tracking, which is a common scenario for head-mounted displays (HMD) without hand controllers. To address this problem, a two-stage approach is proposed. First, a masked autoencoder (MAE) infills the missing hand joints along with an uncertainty prediction using only the available frames and the head tracking. Then, the imputed tracking data (hands and head) are passed to a VQ-Diffusion model to predict the remaining body. The proposed model, namely DSPoser, is evaluated on the AMASS and Ego-Exo4D datasets where it performs better than the baselines in this new problem setting.", "questions": "Please see my questions in the previous section and address my main concern on the evaluations.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: The paper introduces a new challenge to the full body tracking domain, which has become an active research area due to the increasing number of HMD devices in the market. The proposed solution combines techniques from various works effectively.\n\nQuality: The proposed two-stage approach is practical. The masked autoencoder with uncertainty estimation addresses the temporal sparsity problem and decouples the formerly known spatially sparse body tracking (i.e., hands and head are always available) from the temporal sparse setting.\n\nClarity: The paper is well-organized and easy to follow. There is enough background information to understand the proposed method and make connections to the prior works. The authors also provide experimental details thoroughly which seem to be sufficient for reproducibility. \n\nSignificance: The new problem setting is novel and I expect it to be more commonly addressed in the future. Hence, this paper could be a reference for future works.", "weaknesses": "1- I think the evaluations in Tables 1 and 2 could be better structured and also more fair. To make an apples-to-apples comparison, it would be better to group methods using a particular type of input data. For example, EgoEgo should be compared against the DSPoser with only the head tracking inputs. Similarly, the Bodiffusion could also use the MAE imputation. Considering that the VQ-Diffusion and the motion tokenizer are taken from the prior work, MAE as being the main contribution could be better highlighted in this way. The “naive Bodiffusion extension” is simply too naive. Stronger baselines could be introduced.\n\n2- The runtime performance analysis is missing. Considering that the proposed problem setting aims for real-time applications, a masked autoencoder with additional uncertainty computations followed by a diffusion inference is not the optimal candidate. I acknowledge that this is covered in the limitations section. What could be done about it?\n\n3- After reading the “Uncertainty-aware MAE” section, I assumed the total uncertainty (Eq. 6) is proposed. The ablation study, however, reveals that the aleatoric uncertainty gives better performance. Is it the one used in the experiments? \n\n4- A supplementary video with qualitative comparisons would be very helpful. \n\n5- [1*] and [2*] (as concurrent work) could also be covered in the related work section.\n\nThese are not weaknesses but suggestions:\n- I think mixture density networks [3*] could be applied in this setting which would also simplify the story around the uncertainty. The network predicts parameters of a Gaussian Mixture Model, not very different from the current aleatoric uncertainty. \n- Line #243: “The dropout strategy achieves...” -> “The sampling strategy achieves...”\n\n```\n[1*] Du, Yuming, et al. \"Avatars grow legs: Generating smooth human motion from sparse tracking inputs with diffusion model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[2*] Dai, Peng, et al. \"HMD-Poser: On-Device Real-time Human Motion Tracking from Scalable Sparse Observations.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3*] Bishop, Christopher M. \"Mixture density networks.\" (1994).\n```", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers insightful feedback on our paper. We will improve the draft based on the comments.\n\n> Q1: Stronger baselines & Reorganization of Table 1 and 2\n\nDue to limited time and resources, we are currently only able to provide baseline results on the AMASS dataset. We are currently working on training basline methods on Ego-Exo4D datasets, and we will update as soon as they are completed.\nPlease refer to Q1 in the Author Rebuttal to see our response on stronger baselines.\n\nThank you for your valuable feedback. We appreciate the suggestions for improving the structure and fairness of our evaluations. Since we now have more baselines as shown in the Table A in the pdf, it will be better to re-organize the table for a fair comparison and easier understanding.\n\n> Q2: The runtime performance analysis, and possible direction.\n\nPlease refer to the Q2 of the Author Rebuttal.\n\nThank you for your feedback regarding the runtime performance analysis. We understand the importance of computational efficiency, especially for real-time applications.\n\nIn Table B in the pdf and Q2 of the Author Rebuttal, we present a detailed comparison of the computational complexity of VQ-VAE, MAE, and VQ-Diffusion modules, highlighting the number of parameters, multiply-accumulate operations (MACs), and inference time. Our results show that the overhead introduced by the MAE, including uncertainty computations, is minimal compared to the significant overhead from the diffusion process. Specifically, MAE adds only 3 ms to the inference time, which is negligible compared to the 955 ms required by the VQ-Diffusion module. Therefore, the primary computational burden arises from the diffusion process rather than the MAE with uncertainty computations.\n\nWhen we chose the diffusion model, we recognized the heavy computational cost but concluded it was more appropriate for solving the under-constrained problem of ego-body pose estimation. To mitigate this heavy computation issue, we selected the VQ-Diffusion method, which denoises in discretized latent spaces and is considered more computationally efficient. Additionally, as shown in Table C in the pdf, our approach allows skipping denoising steps using the reparameterization trick, following methods from [1] and [2]. The results show that our method provides four times faster options, effectively balancing the trade-off between MPJPE and inference time.\n\nRecent research, such as [3] and [4], has focused on improving the speed of diffusion inference. We expect that as these advancements continue, the diffusion model's versatility and extendability will become even more beneficial, reducing the cost of the diffusion sampling process.\n\n[1] Austin, Jacob, et al. \"Structured denoising diffusion models in discrete state-spaces.\" Advances in Neural Information Processing Systems 34 (2021): 17981-17993.\n\n[2] Gu, Shuyang, et al. \"Vector quantized diffusion model for text-to-image synthesis.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[3] Zheng, Hongkai, et al. \"Fast sampling of diffusion models via operator learning.\" International conference on machine learning. PMLR, 2023.\n\n[4] Yin, Tianwei, et al. \"One-step diffusion with distribution matching distillation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n> Q3: Clarification on Uncertainty\n\nYes, aleatoric uncertainty with a sampling strategy is used in our Table 1 and Table 2. To avoid confusion, we will clarify the strategy and the type of uncertainty at the beginning of Section 3.3.\n\n> Q4: Qualitative comparisons against state-of-the-art approaches\n\nWe appreciate your comment on the importance of qualitative comparisons. While we have visualized our qualitative results in a video format, we found that it is not permitted to include links during the rebuttal phase. As an alternative, we have included detailed comparative qualitative results in the PDF file attached to this rebuttal. We hope this additional information provides clarity and supports the quantitative improvements we reported. We also ensure that we will add video comparisons against state-of-the-art methods in the final version.\n\n> Q5: Suggestion of concurrent related works\n\nThank you for the suggestion. We will include suggested references in the related work section to ensure comprehensive coverage of concurrent research.\n\n> Q6: Mixed Density Network as Aleatoric Uncertainty\n\n\\begin{array}{l|ccc}\n\\hline\n\\textbf{Methods} & \\textbf{MPJPE} & \\textbf{MPJVE} & \\textbf{MPJRE} \\\\\\\\ \\hline \\hline\n\\text{w/o Uncertinaty}&6.05\\pm0.01&30.12 \\pm 0.04&4.36 \\pm 0.00 \\\\\\\\\n\\text{DSPoser w/ MDN}&5.84\\pm0.05&28.34 \\pm 0.15&4.72 \\pm 0.04 \\\\\\\\ \n\\textbf{DSPoser (Ours)}&5.51\\pm0.02&24.19 \\pm 0.10&4.09 \\pm 0.02 \\\\\\\\\\hline\n\\end{array}\n\nThank you for the interesting idea for uncertainty measurement. We reported the result of the whole pipeline after substituting the head of our MAE with a Mixed Density Network (MDN), setting the number of mixtures \\(M\\) to 4 for fair comparison. Similar to the calculation of aleatoric uncertainty in our paper, we measure the aleatoric uncertainty of MDN by $\\mathcal{U}_{ale}(\\mathbf{x})  \\approx M^{-1} \\sum_i \\pi_i\\sigma_i^2 (\\mathbf{x})$, where $\\pi$ is mixture weight. The results show that MDN improves performance compared to our method without uncertainty; however, it shows worse performance compared to the MAE approaches.\n\n\\begin{array}{cc}\n\\hline\n\\textbf{Methods} & \\textbf{MPJPE} \\\\\\\\\\hline\n\\textbf{MDN} & 13.45 \\\\\\\\\n\\textbf{MAE (Ours)} & 10.85 \\\\\\\\\\hline\n\\end{array}\n\nTo investigate the performance difference, we also analyzed the results of temporal completion of hand trajectories. Different from the MAE, the loss of MDN often diverged, so we early stopped the MDN training at 800 epochs, leading to worse MPJPE for hand trajectories constrast to the MAE model trained fro 4000 epochs.\n\n> Q7: Typos.\n\nWe will revise the sentence as suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yAYuZxq8Xi", "reviewer": "Reviewer_VMPC", "summary": "This paper proposes a system to estimate full-body pose from forward-facing egocentric videos. Dubbed “doubly sparse video data,” such data streams have the distinct characteristic that only the headset pose is persistent, while the hand pose estimation is only occasionally available. The proposed method first infills the hand motion from estimated hand motion from video information, and then uses the infilled motion (with estimated uncertainty) to estimate the full-body pose. The full-body pose estimation is built upon VQ-VAE representation and VQ-diffusion. Experiments show that the proposed pipeline outperforms SOTA methods.", "questions": "If possible, some metric on the velocity/acceleration error of the estimated motion would help indicate the smoothness of the motion.", "limitations": "Limitations are addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work is very well-motivated; estimating the full-body pose from sparse egocentric views could have many applications in AR/VR and animation. The task is also very challenging, as hands are only visible in very few frames.\n- The proposed system is a complete solution to estimate full-body pose (including fingers) from egocentric videos and head tracking. It leverages the persistent signals (headset tracking) and occasional signals (hand pose) well by formulating it as a probabilistic infilling problem. The infilled hand motion then serves as input to a diffusion-based full-body pose estimator.\n    - I find the uncertainty formulation a great addition to the current literature. While most methods just use diffusion-based pose estimation plus masking, the proposed MAE solution seems to be a principled way of obtaining a better hand pose trajectory based on sparse input.\n- Experimental results on the Ego-Exo-4D dataset and AMASS show that the proposed method outperforms SOTA methods. The evaluation is extensive and shows the results from the method well. Showing the results of using dense information (Table 3) also demonstrates the strength of the proposed method.", "weaknesses": "- For pose estimation, it would be very beneficial to provide estimated motion as videos to better judge the quality of the estimated pose and motion. Not providing videos weakens this work.\n- Since a complex system is proposed, it would be great to see some ablation about VQ-VAE.\n- Looking at Figure 7 in the appendix, it appears to me that the proposed method could be overfitting. There is no information on kicking the feet up for the human, but the estimated pose is kicking the feet up.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreicate the acknowledgement of our motivation and the novelty of our method. We will improve the draft based on the valuable comments of the reviewer.\n\n> Q1: Qualitative comparisons against state-of-the-art approaches and video visualizations of the results.\n\nWe appreciate your comment on the importance of qualitative comparisons. While we have visualized our qualitative results in a video format, we found that it is not permitted to include links during the rebuttal phase. As an alternative, we have included detailed comparative qualitative results in the PDF file attached to this rebuttal. We hope this additional information provides clarity and supports the quantitative improvements we reported. We also ensure that we will add video comparisons against state-of-the-art methods in the final version.\n\n> Q2: VQ-VAE ablation studies.\n\n\\begin{array}{l|c|cccc}\n\\hline\n\\textbf{Methods} & \\text{Pipeline} & \\textbf{MPJPE} & \\textbf{MPJVE} & \\textbf{MPJRE} & \\textbf{Jitter} \\\\\\\\\n\\hline \\hline\n\\text{BoDiffusion [3]} & \\text{MAE}  +  \\text{Skeleton Space Diffusion} & 7.35 & 31.33 & 5.47 & 1254.84 \\\\\\\\\n\\text{AvatarJLM [34]} & \\text{MAE} +  \\text{Transformer} & 7.12 & 37.60 & 5.24 & 16.95 \\\\\\\\\n\\textbf{DSPoser (ours)} &  \\text{MAE} +  \\text{VQ-Diffusion} + \\text{VQ-Decoder} &5.51 & 24.19 & 4.09 & 4.27 \\\\\\\\ \n\\hline\n\\end{array}\n\nTo evaluate the effectiveness of our pipeline, we implemented baseline models to tackle the pose estimation problem given doubly sparse data. For these baseline models, we introduced MAE at the initial stage of their methods to complete the temporally sparse data, then fed the imputed trajectory into their respective models. The implementation of BoDiffusion as a baseline can serve as an ablation study for our VQ-VAE, as it applies the diffusion process directly to the skeleton space, contrasting with our approach of applying the diffusion process on the Vector-Quantized latent space. Additionally, AvatarJLM can be considered another ablation study, as it utilizes a Transformer instead of a diffusion model to complete the sparse data. These results demonstrate that VQ-VAE outperforms the other architectural options.\n\n> Q3: Overfitting issue.\n\nThank you for your observation and insightful feedback. While it is possible that overfitting could explain this observation, there are other plausible explanations. To the best of my knowledge, when a human moves, the motion of each joint influences the others. Previous research, such as [1], [2], has demonstrated that joint movements are not only connected to adjacent joints through bones (explicit relationships) but also highly related to distant joints that are not directly connected (implicit relationships) in a certain motion context. Additionally, in [3], even though only head position is utilized for whole body estimation, the results often show very accurate lower body movements. This indicates that even if there appears to be no direct information for reconstructing the kicking motion in our visualized example, the intermittent hand observations and dense head trajectory data may provide sufficient information to reconstruct the kicking motion. Therefore, what may seem like overfitting could actually be the model leveraging these implicit relationships to generate a plausible motion sequence.\n\n[1] Chi, Hyung-gun, et al. \"Infogcn: Representation learning for human skeleton-based action recognition.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[2] Wu, Zhize, et al. \"SelfGCN: Graph Convolution Network with Self-Attention for Skeleton-based Action Recognition.\" IEEE Transactions on Image Processing (2024).\n\n[3] Li, Jiaman, Karen Liu, and Jiajun Wu. \"Ego-body pose estimation via ego-head pose estimation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n> Q4: Evaluation on the velocity/acceleration metric.\n\n\\begin{array}{l|cccc}\n\\hline\n\\textbf{Methods}&\\text{MPJPE}&\\text{MPJVE}&\\text{MPJRE}&\\text{Jitter} \\\\\\\\\n\\hline\n\\text{AvatarPoser [10]}&40.42&64.07&16.37&27.89 \\\\\\\\\n\\text{Bodiffusion [3]}&46.45&75.33&17.99&2793.32 \\\\\\\\\n\\text{AvatarJLM [34]}&25.02&68.42&14.14&32.18 \\\\\\\\\n\\text{AvatarPoser [10]}&9.88&62.31&5.98&37.89 \\\\\\\\\n\\text{Bodiffusion [3]}&7.35&31.33&5.47&1254.84 \\\\\\\\\n\\text{AvatarJLM [34]}&7.12&37.60&5.24&16.95 \\\\\\\\\\hline\n\\textbf{DSPoser (Ours)}&\\textbf{5.51}\\pm0.02&\\textbf{24.19}\\pm0.10&\\textbf{4.09}\\pm0.02&\\textbf{4.27}\\pm0.03 \\\\\\\\\\hline\n\\end{array}\n\n\\begin{array}{l|c|cccc}\n\\hline\n\\text{Methods} & \\mathbf{y} & \\text{MPJPE} & \\text{MPJVE} & \\text{MPJRE} & \\text{Jitter} \\\\\\\\ \\hline \\hline\n\\text{GT} & - & - & - & - & 4.01 \\\\\\\\\n\\text{VQ-VAE (Paper)} & \\text{Full body} & 1.26 & 11.37 & 1.81 & 3.93 \\\\\\\\\n\\text{VQ-VAE (Opt'ed)} & \\text{Full body} & 1.15 & 10.59 & 1.67 & 3.89 \\\\\\\\ \\hline\n\\text{AvatarPoser [10]} & \\text{Dense traj.} & 4.18 & 29.40 & 3.21 & 13.63 \\\\\\\\\n\\text{Bodiffusion [3]} & \\text{Dense traj.} & 3.63 & \\mathbf{14.39} & \\mathbf{\\textcolor{blue}{2.70}} & 493.78 \\\\\\\\\n\\text{AvatarJLM [34]} & \\text{Dense traj.} & \\mathbf{3.35} & 20.79 & 2.90 & \\mathbf{\\textcolor{blue}{8.39}} \\\\\\\\ \\hline\n\\mathbf{DSPoser (Paper)} & \\text{Dense traj.} & 3.61 \\pm 0.01 & 18.36 \\pm 0.03 & 2.81 \\pm 0.02 & 4.08 \\pm 0.02 \\\\\\\\ \n\\mathbf{DSPoser (Opt'ed)} & \\text{Dense traj.} & \n\\mathbf{\\textcolor{blue}{3.48 \\pm 0.01}} & \\mathbf{\\textcolor{blue}{17.86 \\pm 0.03}} & \\mathbf{2.68 \\pm 0.02} & \\mathbf{4.03 \\pm 0.02} \\\\\\\\ \\hline\n\\end{array}\n\nWe deeply appreciate your valuable comments on our paper. Thanks to your suggestion, we found out that our method shows significantly better performance on the Jitter metric, which is often used to measure the smoothness of motion. Jitter is a measure of jerk calculated by the derivative of acceleration, and MPJVE, already reported in our paper, indicates the velocity error. As seen in the table above and Q2 of the VQ-VAE ablation studies, it is demonstrated that our method produces smoother results compared to other methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WIwmLmS7M6", "reviewer": "Reviewer_TnAd", "summary": "This paper presents a framework to estimate full-body human poses from egocentric head mounted display. The input of the system mainly contains 2 parts: head tracking signal given by HMD, and sparse hand pose signal estimated from egocentric video. The algorithm, DSPoser, is composed of two stages: temporal completion and spatial completion. In the temporal completion stage, DSPoser uses an encoder-decoder to generate the gaussian distribution of hand pose states. In spatial completion stage, DSPoser generates fullbody poses from hand trajectories and head tracking signals. Experiments were performed on Ego-Exo4D and AMASS dataset.", "questions": "(1)\tFor hand pose estimation, why use FrankMocap instead of recently introduced ACR (ACR: Attention Collaboration-based Regressor for Arbitrary Two-Hand Reconstruction [CVPR 2023]) or IntagHand (Interacting Attention Graph for Single Image Two-Hand Reconstruction [CVPR 2022]), which are designed for hand mesh recovery only? Does the performance of hand mesh recovery affect the final full-body pose estimation? \n\n(2)\tMay the training on AMASS benefit Ego-Exo4D performance? \n\n(3)\tIn Figure 3(b), why not demonstrate hand pose here? \n\n(4)  Sec.3.3 is titled with \"imputed hand trajectories and head tracking signal\". However, L.146 said \"from imputed hand trajectories\" and did not mention \"head tracking signal\" in this section at all. \n\nMinor questions: \nIn table.1, “x” should also be briefly explained like “y”. \n\nAt L.81, V_1 … V_T_w should be defined (RGB images?) \n\nDefinition of T_w (L. 114) is better moved to its first appearance (L.81)", "limitations": "Limitations are clearly addressed.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "(1)\tThe quantitative results are excellent, especially for doubly sparse data. Table.1 and Table.2 supports the effectiveness of the proposed method. \n\n(2)\tThe overall paper writing is clear. \n\n(3)\tLimitations are well-explained. \n\n(4)\tThe ablation study about aleatoric and epistemic uncertainties is interesting to me.", "weaknesses": "(1)\tThe novelty of “newly introduced task (L.205)” is a bit limited. Both motion imputation and pose completion are widely investigated problems, and the combination of both problems seems not difficult to solve. \n\n(2)\tThe paper only proves that the proposed hand trajectory imputation is better than linear interpolation for the doubly sparse task. As the motion imputation/interpolation/inbetweening is a long-standing task, the contribution of the proposed hand imputation method (Uncertainty-aware MAE) is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable comments aimed at improving our paper. We will revise the draft according to the reviewer's suggestions.\n\n> Q1: Limited novelty\n\nOur novelty lies in our approach to solving the body pose estimation problem given doubly sparse data, specifically in how we address the under-constrained problem by measuring and exploiting uncertainty. Previous methods rely heavily on dense hand signals, requiring hand controllers for ego-body pose estimation. Another approach that only uses head poses to estimate the whole body does not utilize hand pose information. Our proposed solution strikes a novel balance between these two approaches, eliminating the need for hand controllers while achieving better results by incorporating a few constraints from detected hand poses.\n\n> Q2: The effectiveness of Uncertainty Aware MAE\n\nAs the reviewer noted, numerous works have addressed trajectory imputation. However, imputing the hand trajectory itself is not our primary task or the focus of our novelty. Our design of MAE aims to capture uncertainty while imputing the hand trajectory, which differs from other imputation methods such as mask token, in-betweening, and interpolation. Additionally, we introduced a couple of ways to utilize this uncertainty —sampling, dropout, and distribution embedding— in a diffusion model while spatially completing the full body. Given that our newly introduced task of estimating ego-body pose from doubly sparse video data is an under-constrained problem, one of our key motivations is to leverage the \"uncertainty\" that arises from this under-constrained data.\n\n> Q3: Hand pose estimation module and its effectivness\n\n\\begin{array}{l|cc}\n\\hline\n\\textbf{Methods} & \\textbf{MPJPE} & \\textbf{MPJVE} \\\\\\\\\n\\hline \\hline\n\\textbf{FrankMocap} & 16.84 \\pm 0.04 & 39.86 \\pm 0.05 \\\\\\\\\n\\textbf{ACR} & 16.69 \\pm 0.05 & 40.12 \\pm 0.06 \\\\\\\\\n\\textbf{Hand Ground Truth} & 16.43 \\pm 0.02 & 37.49 \\pm 0.04 \\\\\\\\ \n\\hline\n\\end{array}\n\nWe agree that recently introduced hand models can be used instead of FrankMocap. Before we submitted the paper, we compared the results of FrankMocap with the ground truth 3D hand joint location of Ego-Exo4D and concluded that the effect of hand detectors is not significant. We believe this is because we utilized only the wrist 3D location from the detected hand. Therefore, even though FrankMocap was our initial choice only to prove the concept, we decided not to replace it.\nWe visulized a table of comparison for the different inputs of the hand detector, and it shows that the performance differnece is not significant from ground truth and other hand detector models.\n\n> Q4: Training on AMASS benefits Ego-Exo4D performance?\n\nWe greatly appreciate the suggestion of applying transfer learning to the Ego-Exo4D dataset. The Ego-Exo4D paper's baseline implementation of EgoEgo [1] took the second-place in the Ego Body Pose Estimation Challenge. This implementation has demonstrated the benefits of training on the AMASS dataset for improving performance on Ego-Exo4D. This method employs a conditional diffusion model, cross-attention for conditioning, and rotary positional embeddings with SLAM pose input. We recognize the potential of this approach and intend to explore its application in our future work on this task.\n\n[1] Li, Jiaman, Karen Liu, and Jiajun Wu. \"Ego-body pose estimation via ego-head pose estimation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n> Q5: Hand visualization on Figure 3-(c)\n\nWe appreciate the reviewer's feedback. While our primary focus is on body pose estimation, rather than hand pose estimation, we understand the importance of comprehensive visualization. Therefore, we can provide visualizations of the canonical hand poses, similar to the approach we used for mesh visualization in Fig 3-(c), to enhance clarity and completeness.\n\n> Q6: Writing clarity regarding our constraint in Section 3.3.\n\nThank you for the valuable feedback. We acknowledge the oversight in Section 3.3, where the title mentions both \"imputed hand trajectories and head tracking signal,\" but the text only references \"imputed hand trajectories.\" We will clarify our method by including a head tracking signal in this section to ensure consistency and completeness.\n\n> Q7: Suggestions for the clarify.\n\nThank you for pointing out the oversights. Following the reviewer's comments, we will make the following revisions to enhance clarity:\n\n1. Add a brief explanation of $x$ in Table 1.\n\n2. Clarify the dimension of the RGB video data.\n\n3. Move $T_w$​ to its first appearance in the text.\n\nThese changes will help ensure the information is clear and easy to understand.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time and effort in helping us improve the paper. We appreciate your acknowledgment of the novelty and valuable suggestions to improve our method. In this rebuttal, we want to clarify a few common questions raised by reviewers. Please note that **experiments** are conducted on the **AMASS dataset** unless otherwise stated.

> Q1: The Necessity of Stronger Baselines
 
\begin{array}{l|cccc}
\hline
\textbf{Methods}&Imputation&\text{MPJPE}&\text{MPJVE}&\text{MPJRE}&\text{Jitter} \\\\
\hline
\text{AvatarPoser [10]}&interpolation&40.42&64.07&16.37&27.89 \\\\
\text{Bodiffusion [3]}&interpolation&46.45&75.33&17.99&2793.32 \\\\
\text{AvatarJLM [34]}&interpolation&25.02&68.42&14.14&32.18 \\\\
\text{AvatarPoser [10]}&MAE&9.88&62.31&5.98&37.89 \\\\
\text{Bodiffusion [3]}&MAE&7.35&31.33&5.47&1254.84 \\\\
\text{AvatarJLM [34]}&MAE&7.12&37.60&5.24&16.95 \\\\\hline
\textbf{DSPoser (Ours)}&MAE&\textbf{5.51}\pm0.02&\textbf{24.19}\pm0.10&\textbf{4.09}\pm0.02&\textbf{4.27}\pm0.03 \\\\\hline
\end{array}

Please find the detailed table in the pdf file.

We acknowledge that stronger baselines are needed to prove the effectiveness of our methods. Therefore, we implemented the baselines with linear interpolation and MAE imputation. The table demonstrates that our method outperforms these baselines.

> Q2: Motivation of Diffusion Model & Its Computational Complexity

We appreciate the question regarding our choice of diffusion models despite potential limitations for real-time applications. We have two reasons for choosing a diffusion model for this task:

First, we considered the ego-body pose estimation task to be an under-constrained problem, so we chose a diffusion model to leverage the inherent uncertainty of the task.

Second, when we first designed our framework, we planned to incorporate multi-modal inputs, such as image features from ego-centric videos and gaze information, in addition to sparse hand data. This integration would enable our model to generate diverse motion sequences conditioned on multi-modal inputs, which are available in the Ego-Exo4D data. 

We believe that the diffusion model's versatility and extendability make it well-suited for these types of multi-modal integrations. This potential for extension reinforces our choice of diffusion models as a foundational element in our research, providing a flexible and powerful tool for future developments in this field.

\begin{array}{c|ccc}
\hline
\textbf{Module}&\textbf{\\# of Params}&\textbf{MACs}&\textbf{Time} \\\\
\hline 
\text{VQ-VAE}&17.9 \text{ M}&3.6 \text{ G}&3 \text{ ms} \\\\
\text{MAE}&51.3 \text{ M}&23.3 \text{ G}&4 \text{ ms} \\\\
\text{VQ-Diffusion}&74.2 \text{ M}&1190.2 \text{ G}&958 \text{ ms} \\\\
\hline
\end{array}

\begin{array}{c|cccc|c}
\hline
\text{Infer. Steps}&\text{Train. Steps 25}&\text{Train. Steps 33}&\text{Train. Steps 50}&\text{Train. Steps 100}&\text{Infer. Time (ms)}\\\\\hline
25&5.83&5.92&5.69&8.72&278\\\\
33&-&5.67&5.63&5.58&348\\\\
50&-&-&5.61&5.53&522\\\\
100&-&-&-&5.51&1013\\\\\hline
\end{array}

We appreciate the feedback and recognize the importance of including metrics related to computational complexity. While we have roughly provided the inference time in the supplementary material following the NeurIPS 2024 submission policy, we agree that a more detailed comparison would be beneficial. We will incorporate these details in the revised version of our paper to provide a thorough evaluation of our method's computational complexity.

> Q3: Evaluation on Velocity and Accleration.

\begin{array}{l|c|cccc}
\hline
\text{Methods}&\mathbf{y}&\text{MPJPE}&\text{MPJVE}&\text{MPJRE}&\text{Jitter} \\\\\hline 
\text{GT}&-&-&-&-&4.01 \\\\
\text{VQ-VAE (Paper)}&\text{Full body}&1.26&11.37&1.81&3.93 \\\\
\text{VQ-VAE (Opt'ed)}&\text{Full body}&1.15&10.59&1.67&3.89 \\\\\hline
\text{AvatarPoser [10]}&\text{Dense traj.}&4.18&29.40&3.21&13.63 \\\\
\text{Bodiffusion [3]}&\text{Dense traj.}&3.63&\mathbf{14.39}&\mathbf{\textcolor{blue}{2.70}}&493.78 \\\\
\text{AvatarJLM [34]}&\text{Dense traj.}&\mathbf{3.35}&20.79&2.90&\mathbf{\textcolor{blue}{8.39}} \\\\\hline
\mathbf{DSPoser (Paper)}&\text{Dense traj.}&3.61\pm0.01&18.36\pm0.03&2.81\pm0.02&4.08\pm0.02 \\\\
\mathbf{DSPoser (Opt'ed)}&\text{Dense traj.}&\mathbf{\textcolor{blue}{3.48\pm0.01}}&\mathbf{\textcolor{blue}{17.86\pm0.03}}&\mathbf{2.68\pm0.02}&\mathbf{4.03\pm0.02} \\\\\hline
\end{array}

We found that our method shows significantly better performance on the Jitter metric, which is often used to measure the smoothness of motion. Jitter is a measure of jerk calculated by the derivative of acceleration and MPJVE indicates the velocity error as reported in our paper. As seen in the table above and table in Q1, it is demonstrated that our method produces smoother results compared to other methods, closely matching the smoothness of the ground truth, while other methods show significantly higher values on the Jitter metric.

> Q4: Qualitative comparisons against state-of-the-art approaches and video visualizations of the results.

We appreciate your comment on the importance of qualitative comparisons. While we have visualized our qualitative results in a video format, we found that it is not permitted to include links during the rebuttal phase. As an alternative, we have included detailed comparative qualitative results in the PDF file attached to this rebuttal. We hope this additional information provides clarity and supports the quantitative improvements we reported. We also ensure that we will add video comparisons against state-of-the-art methods in the revised version.

> Q5: Lack of publicly available code.

We recognize the value of open-source code and plan to release ours upon acceptance. Our organizational policy prioritizes careful review and preparation before any public release. In the meantime, we're committed to providing detailed methodologies in our publications to support reproducibility and further research in the field.

Meta-review: The submission initially received generally positive reviews; after rebuttal, overall, reviewers became slightly even more positive about the submission.  The AC agrees with the shared recommendations.  The authors should incorporate the rebuttal into the camera ready.
Decision: Accept (poster)