Joint Optimization for 4D Human-Scene Reconstruction in the Wild
Authors: Zhizheng Liu
Joe Lin
Wayne Wu
Bolei Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Human Scene Interaction
Global Human Motion Estimation
Dense Scene Reconstruction
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Reconstructing human motion and its surrounding environment is crucial for understanding human-scene interaction and predicting human movements in the scene. While much progress has been made in capturing human-scene interaction in constrained environments, those prior methods can hardly reconstruct the natural and diverse human motion and scene context from web videos. In this work, we propose JOSH, a novel optimization-based method for 4D human-scene reconstruction in the wild from monocular videos. Compared to prior works that perform separate optimization of the human, the camera, and the scene, JOSH leverages the human-scene contact constraints to jointly optimize all parameters in a single stage. Experiment results demonstrate that JOSH significantly improves 4D human-scene reconstruction, global human motion estimation, and dense scene reconstruction by utilizing the joint optimization of scene geometry, human motion, and camera poses. Further studies show that JOSH can enable scalable training of end-to-end global human motion models on extensive web data, highlighting its robustness and generalizability. The code and model are available at [https://vail-ucla.github.io/JOSH/](https://vail-ucla.github.io/JOSH/]).

Published as a conference paper at ICLR 2026
JOINT OPTIMIZATION FOR 4D HUMAN-SCENE RECON- STRUCTION IN THE WILD
Zhizheng Liu, Joe Lin, Wayne Wu, Bolei Zhou Department of Computer Science, University of California, Los Angeles
Input Video
Global Human Motion
Dense Scene Reconstruction
Camera Poses
A
B
Figure 1: 4D human-scene reconstruction in the wild with JOSH. Given web video captured from a single camera as input, JOSH outputs the global human motion, the dense scene reconstruction, and the camera poses with coherent human scene interaction.
A ABSTRACT
Reconstructing human motion and its surrounding environment is crucial for un- derstanding buman-scern interaction and predicting human movements in the scene. While much progress has been made capturing human-scen seennn-ceene interac- tion in constrained environments, those prior methods can hardly reconstruct the natural and diverse human motion and scene context from web videos. this work, we propose JOSH, novel optimization-basee method for 4D human-scene reconstruction the wild from monocular videos. Compared to prior works that perform separate optimization of the human, the camera, and the scene, JOSH leverages the human-scene contact constraints to jointly optimize all parameters in single stage. Experiment results demonstrate that JOSH significantly im- proves 4D human-scene reconstruction, global human motion estimation, and dense scene reconstruction utilizing the joint optimization of scene geometry, human motion, and camera poses. Further studies show that JOSH can enable scalable training of end-to-end global human motion models extensive web data, highlighting its robustness and generalizability. The code and model are available ht https: http:://vvaluulla. github. 1o//ossoss JOSH/.
INTRODUCTION
Humans constantly interact with their surrounding environments: people sit on benches at plazas, walk through crosswalks intersections, and climb stairs inside buildings. Capturing and understanding human-scene interaction is thus crucial to many applications. For example, analyzing how pedestrians navigate crosswalks and sidewalks is essential for the safe deployment of autonomous driving (Keller & Gavrila, 2013). Meanwhile, designing urban public spaces requires knowing how individuals interact with their surroundings to optimize the group flow, encourage social interaction, and create inviting, accessible environments (Gchl, 2013).
Existing research in human-scene interaction mainly focuses on capturing and synthesizing human motion in pre-scanned 3D scenes (Hassan et al., 2019; Huang et al., 2022; Jiang et al., 2024). They first reconstruct the scene without people in constrained environments with complex sensor setups such multi-viee RGBD cameras and laser scanners and then fit the human motion the environment. Such complex setups limit data accessibility and the capturing of natural and diverse human-scene interactions in the wild. On the other hand, reconstructing global human motion from

---

Published as a conference paper at ICLR 2026
casual web videos has been rrisin topic in recent years (Von Marcard et al., 2018: Kaufmann et al., 2023). However, most previous works only reconstruct motion without scene context (Shin et al., 2024; Wang et al., 2024b; Li et al., 2024; Wang et al., 2025b), and the resulting reconstructed motion lacks grounding and meaning from its surrounding environment.
In this paper, we aim to capture human-scene interactions by tackling monocular 4D human-scene reconstruction, which reconstructs the camera pose, the 4D global human motion, and the 3D scene from web videos. Only few works (Zhao et al., 2024; Luvizon et al., 2023; Liu et al., 2021; Xue et al., 2024) have attempted 4D human-scene reconstruction, by separately performing camera pose estimation, scene reconstruction, and global human motion optimization. However, such approaches ignore the critical interplay between the camera, the human, and the environment, as they can mutually refine each other continuously in a holistic reconstruction. Moreover, as shown in Fig. (a), these approaches can hardly achieve consistency and coherence in detailed human-scene contacts, resulting in physically implausible reconstruction. Many prior works (Shin al., 2024; Shen et al., 2024; Wang et al., 2024b) also only focus reconstructing the motion of one single person without ensuring the consistency of multi-human motion in the same world space, Therefore, is natural and essential to attempt to reconstruct the camera, the scene, and the motions of all humans simultaneously with joint optimization for better accuracy and consistency of 4D human- -eman-scenee reconstruction.
In this work, we propose JOSH (Joint Optimization of Scene Geometry and Human Motion), general optimization framework for 4D) human-scene reconstruction from monocular videos. As shown in Fig. 1, from single video, JOSH can simultaneously reconstruct dense 3D environment, camera poses, and 4D global human motion, producing accurate and coherent human-scene interaction data, JOSH initializes the optimization with local human mesh recovery, human-scene contact labels, depth maps, and point correspondences. Instead of performing sequential optimization and reconstruction, JOSH jointly refines the camera poses, the 4D global human motion of all people, and the dense 3D scene point cloud in a single stage, with the key insight that numan-scenn contact can act as strong constraints that bridge the scene and human motion and guide the joint optimization to produce more precise and consistent 4D human- scene reconstruction results.
JOSH is a general optimization framework, and we experiment with JOSH using different initialization methods. Our results demonstrate that JOSH significantly improves performance in 4D human-scene reconstruction, global human motion estimation, and dense scene reconstruction, compared to approaches that do not apply joint optimization. Noticeably, when initialized with VIMO (Wang et al., 2024b) and MASt3R (Duisterhof et al., 2024). JOSH sets a new state-of-the-arr for global human motion estimation, surpassing previous methods by clear margin. Ablation studies demonstrate the effectiveness of each optimization component in JOSH.
Web videos offer rich and diverse source of real-world global human motion; however, their unstructured nature makes obtaining reliable ground-truth anmotations challenging. Owing to JOSH's strong robustness and generalizability, can facilitate the scalable training of end-to-end global human motion models by providing accurate pseudo-labels. Further experiments demonstrate that the model trained on web data using JOSH's pseudo-labed significantly outperforms the one trained on datasets with ground- on in-the-wild data. We further design an end-to-end model, JOSH3R (JOint Scene and Human 31> Reconstruction), with a lightweight human trajectory head based on MASt3R Leroy et al. (2024) to predict the relative human transformation directly between two frames, allowing real-time inference as trade-off to the estimation accuracy.
labels. This highlights JOSH's ' effectiveness in enabling scalable training
We summarize our contribution as follows: to tackle the challenge of 4D human-scene reconstruction in the wild by jointly optimizing the camera pose, the global human motion, and the scene reconstruction in single stage. 2) Experiments with different initializations show that JOSH brings significant improvement to 4D human--cemt recon- struction, global human motion estimation, and dense scene reconstruction. 3) Further studies show JOSH3R achieves competitive results by training only with labels predicted by JOSH, highlighing JOSH's high potential for scalable training of end-to-end models using extensive web videos.
1) general optimization framework, JOSH, is proposed
2
PRELIMINARIES
4D human scene reconstruction involves both global human motion estimation and dense scene reconstruction. We introduce preliminaries of these two tasks in Sec. 2.1 and See. 2.2, respectively.
2

---

Published as a conference paper at ICLR 2026
2.1 GLOBAL HUMAN MOTION ESTIMATION
In global human motion estimation, the goal is to reconstruct the 4D human motion with SMPL (Loper al., 2023) human parameters in the world coordinate at every timestep { g))==- The SMPL model takes the parameters 9 = {Ty, g> ,3' where 0¹ E (3) 23 is the local pose of the 23 body joints, 3 E R represents the body shape of the human, and TV (Ry 12) (3) is the global orientation and translation of the SMPI. meshes in the world coordinate. The human pose O' and shape 3 can be obtained from human mesh recovery techniques (Goel et al., 2023), as they mainly depend on the local 2D image context. Estimating the global transformation g entangled in a monocular video. We can decompose the global transformation TV as T = P To where P E (3) is the camera extrinsic matrix and T = (R', t²) E SE(3) is the transformation matrix of the SMPL meshes in the local camera coordinate. It is also possible to estimate TV directly from an image (Goel et al., 2023) given the focal length of the camera and the human shape prior. Therefore, accurate reconstruction of the global human motion depends on precise estimation of both the camera poses [Pr I== and local SMPL parameters {0)
(3) and outputs the human body mesh vertices V E 380000x3
is more challenging as camera motion and human motion are
N with O (T: 3*).
2.2 DENSE SCENE RECONSTRUCTION
In dense scene reconstruction, the goal is to recover the 3D geometry of the environment given by the dense point cloud X E R xxxxxxxx X consists of the transformed local point map X of each timestep in the world coordinate:
X x= X-U = (K),
(I)
1=1
where o² E IR is the scale for each frame, K E 3x3 is the camera intrinsic matrix, Z' E ARx is the dense depth map for each frame, and x (-) is the unprojection from 2D pixels of the frame to 3D points in camera coordinate given the camera intrinsics and depth map. A common way for dense scene reconstruction is to first predict the per-frame point maps X in the local camera coordinate and the point correspondences between adjacent frames C = {(x", x² (x E X'. xy E X', (i,j) E E). where & contains pairs of frames in the optimization graph, and is the confidence of the point correspondence. It then performs a global optimization of all parameters with the 3D correspondence loss and 2D reprojection loss:
CAD
Σ
e. c-plo' 0d pix))
c.p( ( KK' 1/5 ((p))-'-''')))
C2D
Σ
(3)
(x p(=(K) Ki xx) o( KC :/2 (pi) (pix)))
Laoene W0p LBD + W2D L2D=
where x (-) projects 3D points in the camera coordinate to the pixel locations with camera intrinsics and p(-) is robust loss to deal with potential outliers.
3 JOINT OPTIMIZATION OF SCENE AND HUMAN
Humans are constantly interacting with the scene. The moving camera captures such interaction and encodes rich geometry information useful both human and scene reconstruction. Therefore. it is necessary to perform joint optimization for 4D human- reconstruction as it enables the camera, the human, and the scene to benefit each other continuously. Existing methods (Shin et al., 2024; Xue et al., 2024; Zhao et al., 2024) handle scene reconstruction, human motion estimation, and camera pose estimation as separate tasks or perform separate optimization. This separation can lead to inconsistent and inaccurate reconstruction results. To better exploit the synergy between the camera, the human, and the scene, we introduce JOSH, a general optimization framework that jointly reconstruction in single stage, allowing all tasks
optimizes all the parameters for 4D human to mutually refine each other. An overview of JOSH is shown in Fig. 2. We first formalize the task of 4D human-scene reconstruction from monocular videos in Sec. 3.1. We then discuss the initialization
3

---

Published as a conference paper at ICLR 2026
Input Video
(a) Initialization
(b) Joint Optimization
4D Human-Scene Reconstruction
Point Nape &t Comespondencee noed (
Auxeal Scare Recorntraction Loss
Skaws Human Proricoss
Soonuee Human Scame Connact Losses
Human Mesh
flaazurry (o:
Contect labels ( S.:
Monamon Frr
Mestmine Ag
(A)
Figure 2: Overview of JOSH. Given an input video, JOSH first initializes the point maps, point correspondences, local human mesh recovery, and contact labels from pre- trairra models. JOSH then jointly optimizes the camera poses, the dense point cloud, and the global human motion with the key human-scone contact losses to predict 4D human-scene reconstruction.
before optimization in Sec. 3.2. We introduce our joint optimization procedure, including the key human-scene contact losses in Sec. 3.3. We finally present JOSH3R, an end-to-end prediction model in Sec. 3.4.
3.1 PROBLEM DEFINITION
We aim to tackle the problem of 4D human-scena reconstruction, which reconstructs the global human motion of all humans, the surrounding scene context, and the camera parameters from in-the-win videos. The input a single monocular video with sequence of images {I' E R N N is the total number of frames. The optimization parameters are {K', Pt of Z' 1 TM 1N including the camera intrinsic and extrinsic parameters, the dense depth maps with global scales, and the local SMPL parameters for each human k among all M humans. The final output the joint reconstruction of the dense scene point cloud X and the global human motion ( 1M 1N
where
c kJ== -
g.k k== 1-=1 For notation simplicity, we will omit the subscript when discussing the human parameters, as they apply to all humans during joint optimization.
3.2 INITIALIZATION
JOSH uses off-the-shelf models to initialize parameters for optimization. We use dense scene re- construction methods initialize the local point maps {X') Jt=1 and point correspondences C. As many dense scene reconstruction methods only work for static environments, humans in monoc- ular videos may introduce additional noise within reconstructed depth maps and result in wrong correspondences, leading degraded reconstruction performance. Hence, first segment out the moving humans using a video segmentation model DEVA (Cheng et al., 2023), resulting in 2D masks M' for each image. We then trim the point cloud and matching results based the masks, i.e., X (ax X T'/T(K' (K' xx) Md), Č {(x', xx, the trimmed results with points in the background for scene reconstruction. We also use human mesh recovery methods to provide initial estimation of the local SMPL parameters { 1 for all humans in the video. To compute the key human-scene contact losses, we finally predict the per-vertex contact labels of the SMPL mesh for all humans (U' V! = from the image using contact prediction model BSTRO (Huang et al., 2022).
Cl X .xd X'), ) and only
JOSH is compatible with different dense scene reconstruction and human mesh recovery methods as long as they can provide the required initializations. In our experiments, we will show the benefit of joint optimization with different initializations.
3.3 JOINT OPTIMIZATION
Human-scene contact is the most natural form of human-scene interaction. It can provide explicit geometry constraints and physical grounding that bridge camera pose estimation, human motion estimation, and scene reconstruction with joint optimization. These contact constraints can help refine body pose in motion estimation, ensuring physically plausible interactions with the environment while improving the accuracy of the scene geometry and the camera poses. Therefore, aim to
4

---

Published as a conference paper at ICLR 2026
leverage human-scene contact by inferring two key human- contact losses for joint optimization given the estimated contact labels {U*) J==1
Contact Scene Loss Human-scene contact can help identify corresponding contact points from the human and the scene, and hence, we can ensure the physical plausibility of the contact by constraining corresponding contact points from the human body mesh and the dense scene point cloud to be close in the final reconstruction with the contact scene loss Le-- We first identify the correspondence between scene contact points æt in the background scene point cloud and the predicted human contact points tt For all predicted contact vertices æf, of the human body mesh in initialization, we require these contact vertices to be visible avoid depth ambiguities, i.e. I (K*, .x,)) (1-M"). Next, we search for the corresponding contact point in the scene background with the closest distance to the projected contact vertex: a argmin x' X | (K*, . K .ri)2 We finally do a filtering, requiring that the depth predictions of the scene contact point and the human contact point are close in monocular depth prior Z' (Bhat et al., 2023): IZ ( ,²))) Z' (m(K', x+))) < e. We empirically find that such a search heuristic generally leads to correct contact correspondences, especially for foot and hand contacts. Then we can obtain the contact correspondences D = {(0f, o²) E U x E X and the contact scene loss Le1 can be computed as follows:
L11
Σ
paah, - -,
(5)
(ar), (D))
Note that as the human contact point 24, is updated during joint optimization, we search for and update the corresponding scene contact point x', in each iteration.
Contact Static Loss Human-scene contact can also help identify body parts that are relatively static to the scene when contact is maintained across frames. This motivates us to design the contact static loss Lea that encourages such contact remains static to reduce sliding motion. We can check if the same vertex remains in contact in adjacent frames and denote the results as E {(xr, xx))) æ e U 2 e U.}. Lee ensures both the human contact points and their corresponding scene contact points remain static as follows:
L22 =
Σ (L
+ -o,)'
Final Loss The final loss for joint optimization includes the scene reconstruction loss, the human prior loss, and the key human-scene contact loss. The scene reconstruction loss Laceme is computed from point correspondences Ć in the static background using Eqn. 4. The human prior loss Lnuman consists of a temporal smoothness loss to ensure smoothness of the global human motion, a SMPL prior loss to constrain the estimated SMPL local parameters close to their initial values, and 2D) key-point reprojection loss to regularize the 2D projection of the SMPL joints:
Chuman
( Ssmoohh 0
-0 1 Wsmpil O - - 12 + W2dkpp W2dkplJán - F2nD.
-
(7)
where O the initial local SMPL parameters from human mesh recovery, JED is the projected SMPL joints, and JiP is the estimated 2D keypoints using ViTPose (Xu et al., 2022). Finally, we use the key human-scene contact losses Lcontact to guide both human and scene reconstruction with Lcontact Woihel + We2Loz. We sum up all aforementioned losses to obtain our final loss:
C = Lavene + Lhuman + Loontact-
(8)
We use this loss to optimize all the parameters (K', Pi,a', Z', O = (T:;((TTT,)) (9, in a single stage with gradient based optimizer. Note that the final result on metric scale the human prior loss Lhuman encodes information about the global scale in metric units.
Optimizing Focal Length As many in-the-wild videos don't provide camera intrinsics information Kt, prior works (Ye et al., 2023; Zhao et al., 2024; Wang et al., 2024b) approximate a fixed focal length such as the diagonal pixel length. However, the root depth of the local human mesh tą predicted by existing human mesh recovery models (Goel et al., 2023; Cai et al., 2023) is proportional to focal length f, wrong initial focal length estimation would also lead to irreoverrblle error
5

---

Published as a conference paper at ICLR 2026
in the human motion. On the contrary, JOSH supports optimizing the focal length of the camera f while updating the local root depth thanks to joint optimization. To be specific, we add of to the optimization parameters and set the local translation of the SMPL mesh t [ t'] in each optimization iteration, where L fun t- and finit is the initial focal length. This formulation ensures that the depth adjustment remains consistent with the focal length update, leading to more accurate 4D human-scene reconstruction. We show the benefit of jointly optimizing focal length in Sec. 4.5.
3.4 END-TO-END PREDICTION WITH JOSH3R
Web videos provide a diverse and large-scale data source for real-world global human motion, but their unstructured nature makes it challenging to get reliable ground-truth annotations. On the other hand, existing real-world global human motion datasets (Dai et al., 2023; Kaufmann et al., 2023; Huang et al., 2022) are small in scale and lack diverse scene backgrounds, real-world motion, and camera movements, making it challenging global human motion in the world coordinate directly from monocular videos. Due to the strong performance of JOSH in in-the-wild settings as shown in Fig. (b), it can generalize to noisy web videos and enable scalable training of end-to-end models from extensive web videos. To this end, we use JOSH to label global human motion from about 20 hours of web videos and train an end-to-end model JOSH3R. Inspired by the strong performance of MASt3R Leroy et al. (2024) in geometric scene understanding. we introduce a new human trajectory head to output the relative local human transformation AT: between adjacent frames, which is in parallel with the original output heads for local scene reconstruction. Assuming the origin of the world coordinate aligns with the camera coordinate of the first frame, the global human transformation and the global camera pose can then be computed iteratively without optimization as:
train efficient end-to-end model for recovering
T =
II AT: T' -T-)
(9)
where T! the local transformation of the SMPL human mesh in the first frame. We leave the implementation details of labeling web videos and JOSH3R in the supplementary material.
4 EXPERIMENTS
We diseuss our experiment setups in Sec. 4.1. We show the effectiveness of JOSH on 4D human-scene reconstruction in Sec. 4.2. Then, we show the benefit of JOSH on global human motion estimation and dense scene reconstruction respectively in Sec. 4.3 and Sec. 4.4, Ablation studies on JOSH can be found in Sec. 4.5. We further show the potential of JOSH to enable scalable training of JOSH3R from web videos in Sec. 4.6. Implementation details about JOSH can be found in Appendix B.
4.1 EXPERIMENT SETUPS
Datasets We use the SLOPER4D dataset (Dai et al., 2023) as the main dataset for ablation and comparison, as it provides ground truth labels for the global human motion, the scene mesh, and the camera parameters annotated from LiDAR point clouds, and we use the 6 publicly released sequences for evaluation. We also provide results on the EMDB (Kaufmann et al., 2023) and RICH (Huang et al., 2022) datasets. The EMDB dataset only provides labels of the global human motion and camera poses without the scene labels, and we use the EMDB-2 subset with 25 sequences for evaluation following previous works (Shin et al., 2024; Wang et al., 2024b). The RICH dataset only provides global human motion labels and pre-captured scene meshes from laser scanner without ground-truth -ttuh camera parameters, and we use the subset of 40 sequences with a moving camera.
Metrics We evaluate on the following aspects of 4D human-scene reconstruction: global human motion estimation accuracy, dense scene reconstruction quality, and physics plausibility. As different datasets contain different labels, we only evaluate the available metrics for each dataset. For global human motion estimation, we follow prior works (Ye et al., 2023; Shin et al., 2024; Wang et al., 2024b) to measure errors in the world coordinate. We split sequences into segments of 100 frames and align the predicted motion with the ground truth using either the first two frames or the entire

---

Published as a conference paper at ICLR 2026
Table I: Effectiveness of JOSH on 4D human-scene reconstruction. We compare different variants of JOSH with the baseline SynCHMR" (Zhao al., 2024) on SLOPER4D (Dai et al., 2023) dataset.
Initialization
Human Metrics
Scene Metrics
Physics Metrics
Methods
Human
Scene
WA-MPIPE
W-MPIPE 1 RTE%
ATE
Abs Rel 4 8e1.25 1 CD
Fitter J.
FFR
SynCHMR
HMR220 DROID SLAM
233.2
1125.4
4.5
17.47
0.25
0.55
17.76
123.9
67.4
9.0
JOSH1
HMR2.0 DROID-SLAN
206.3
1094.2
4.4
17.18
0.25
0.57
16.84
7.6
56.9
3.3
JOSH2
WHAM
MonST3R
210.4
994.3
3.0
14.53
0.22
0.75
9.09
7.8
45.3
2.1
JOSHa
0.16
2.9
a) Input
Ground
Joint Opt)
SynCHMR* (w/o Jeint Opt)
b) Imput Video
JOSH
Input Video
JOSH
Input Video
Figure 3: Qualitative 4D human-scene reconstruction results with the JOSH; variant. a): Qualitative results the RICH (Huang et al., 2022) dataset. We compare the ground truth reconstruction with JOSH with joint optimization and the SynCHMR" (Zhao et al., 2024) baseline without joint optimization. JOSH has better reconstruction quality and coherency between global human motion and dense scene reconstruction. b). Qualitative results of weh videos. JOSH can reconstruct the motion of multiple people and their surrounding environment in the wild. More video qualitative results are included in the supplementary video.
segment and compute the mean-per-joint error in millimeters to obtain W-MPJPE WA-MPJPR for evaluation. We also evaluate the relative Root Translation Error (RTE%) of the whole trajectory. For dense scene reconstruction, as JOSH can reconstruct the scene on metric scale, do not perform global scale alignment. Following previous literature (Wang et al., 2024a; Zhang et al., 2024; Zhao et al., 2024), we first evaluate the video depth estimation using absolute relative eгrоr (AbsRel) and the fraction of inlier points (5<1.25) as well as the camera poses using absolute trajectory error (ATE) in meters as indirect metrics. We then evaluate the dense reconstruction with chamfer distance (CD) in meters. For physics plausibility, we evaluate jittering (Jitter, 10m/s²) and foot sliding during contact (FS, mm) for motion plausibility. Jittering measures the rate of change 3D joint accelcration with respect to time. Foot sliding is the average foot displacement between adjacent frames where the ground truth foot displacement is less then a threshold (1 (10mm). We also evaluate foot floating rate (FFR%) for human-scen interaction plausibility. The foot floating rate computes the minimum distance between the human foot and the scene point cloud and reports the ratio of the distance greater than given threshold (20em).
Initialization Methods To show the benefit of JOSH as a general optimization framework, we build different variants of JOSH with various state-of-the-an human mesh recovery and dense scene reconstruction methods to provide initializations experiment with HMR2.0 (Goel al., 2023), WHAM (Shin et al., 2024). and VIMO (Wang et al., 2024b) to provide local human mesh initializations. For scene initialization, we experiment with DROID-SLAM (Teed & Deng, 2021), MonST3R (Zhang al., 2024), and MASt3R (Leroy et al., 2024). We combine these initializations to get three variants as JOSH, JOSH,, and JOSH;.
shown in Tab. (column "Initialization"). ")) We
4.2 4D HUMAN-SCENE RECONSTRUCTION
To show the effectiveness of JOSH 4D human-scene reconstruction, we compare it with SynCHMR (Zhao et al., 2024) as our main baseline. SynCHMR performs sequential optimiza- tion of scene and human instead of joint optimization. It first calibrates the scale of the scene reconstruction and camera poses using HMR2.0 (Goel et al., 2023) and ZoeDepth (Bhat et al., 2023). It then transforms the local SMPL. meshes to the world coordinates while fixing the scene reconstruc-
7

---

Published as a conference paper at ICLR 2026
(a) Human Motion Estimation on the EMDB Dataset
(b) Scene Reconstruction on the SLOPERAD Dataset
776.1
21.33
700
21
600
Chamfer Distomce
18
16,84
W-MPPE
600
372.3
336,3
13.28
9,09
12.35
500
222,4
SLAHMM W/ original method w/ JOSH variont
WILAM
TRAM
DROID-SLAM
MonST3R
MASOR
masking
IB JOSH variant
Figure 4: Quantitative comparison results on a): global human motion estimation and b): dense scene reconstruction (L). For each method, we compare with the JOSH variant that uses the same local human initialization/ / scene initialization to highlight the effectiveness of joint optimization.
Ground Truth
JOSH, (ours)
MAS13R
MonST3R
Truth
JOSH, (ours)
Figure 5: Qualitative comparisons for global hu- man motion estimation on EMDB. JOSH has the best accuracy compared to the other baselines, while TRAM Wang et al. (2024b) has the wrong global body orientation, and WHAM Shin et al. (2024) has large errors in global translation.
Figure 6: Qualitative comparisons for dense scene reconstruction SLOPER4D. We visualize the re- constructed dense seene point cloud and the camera trajectory. JOSH has the best accuracy compared to the other baselines, while MAS (Leroy al., 2024) has the wrong scale, and MonST3R (Zhang al., 2024) fails produce consistent reconstruction.
tion. As their implementation is not publicly available, we re-implement their method and call it SynCHMR* As shown in Tab. 1, with the same initialization method, JOSH; performs better than SynCHMR* in all metrics, especially in the physics plausibility, which reduces jittering from 1 123.9 to 7.6, foot sliding from 67.4 to 56.9, and foot floating rate from 9.0% to 3.3%. In addition, JOSH is more general framework that can benefit from more recent works in local human mesh recovery and dense scene reconstruction. For example, when using VIMO and MASt3R as initialization, the variant JOSH; reduces error by 46.6% WA-MPJPE, and reduces error by 70.1 % in chamfer distance compared to SynCHMR*. while SynCHMR* can only work with DROID- SLAM (Teed & Deng, 2021). As shown in Fig. (a), as SynCHMR' does not directly optimize the detailed contact between scene and human, the reconstruction result lacks coherency and physics plausibility with significant foot penetration in the scene, which further highlights the value of joint optimization. As JOSH is a more general method, we believe its performance can be further improved with better initializations in future models.
4.3 GLOBAL HUMAN MOTION ESTIMATION
We compare state-of-the-art global human motion estimation methods SLAHMR (Ye et al., 2023), WHAM (Shin et al., 2024), and TRAM (Wang et al., 2024b) with the corresponding JOSH variant that has the same local human initialization. The results are shown in Fig (a). It can be observed that JOSH always leads to better performance than its counterpart method. For example, JOSH, performs better than SLAHMR (Ye et al., 2023) when both use HMR2.0 as initialization, achieving W-MPJPE of 372.3 on the EMDB dataset compared to 776. with SLAHMR. In addition, when using VIMO Wang et al. (2024b) initialization, JOSHa performs better than its counterpart TRAM and sets up new sfate-of-the-as on the EMDB dataset with a 174.7 W-MPJPE. These results demonstrate the general benefit of jointly optimizing the human and the scene in obtaining accurate global human motion estimation. Some qualitative comparisons are shown in Fig. 5. complete comparison table on all datasets can be found in Appendix. C.
8

---

Published as a conference paper at ICLR 2026
2388.3
18.5 10.4
Table 2: Ablation experiments of JOSH SLOPER4D. Row I: Disabling the contact seene loss Let. Row 2: Not optimizing local human motion +.. Row 3: Disabling the contact static loss L.2. Row 4: The full optimization
1148.4
296.1
297.7
with the JOSH₂ variant. Row 5 and Row 6: Optimizing without the given ground truth camera intrinsics, while Row
also optimizes the intrinsics.
Human W.MPJPEJ
Scene 1 Abs Rell, ATTE
Physics
WA-MP (mm)
W-MPUPE (HUM)
RTE%
Variants
[SS FFRS.
JOSHER EMDB) JUSHER (W/ JOSH Labch)
an =WHAM
Figure 7: Scalable training results with JOSH
Le1
1361.4
4.7
0.49
22.47
47.3
92.9
opt 6
486.4
1.8
0.17
3.28
35.6
3.2
labels (L). We compare to JOSH3R trained on the EMDB (Kaufmann et al., 2023) dataset and
-Cu2
448.3
1.9
0.18
3,26
68.2
3.2
JOSH₂
438.3
1.8
0.17
28.2
2.9
WHAM (Shin et al., 2024) on the SLOPER4D (Dai
Intrin.
1220.7
5.5
0.60
19.89
77.3
et al., 2023) dataset.
Opt. Intrin.
1053,4
4.6
0.47
16.91
68.3
6.8
4.4
DENSE SCENE RECONSTRUCTION
Similar to global human motion estimation, we compare state-of-the-art scene reconstruction methods DROID-SLAR (Teed & Deng. 2021), MonST3R (Zhang et al., 2024), and MASO3 (Leroy et al., 2024) with the corresponding JOSH variant that has the same scene initialization in Fig. 4 (b). For a fair comparison, also perform human masking for MASt3R and DROID-SLAM, which cannot work with dynamic objects, so that the only varying factor is joint optimization. The results show the joint optimization of JOSH can benefit different dense reconstruction methods well. For example, JOSH; achieves 57.0% less chamfer distance on the SLOPER4D dataset compared to the original MASt3R, which only optimizes the scene. Also, we can see that the variant JOSH; achieves the best performance on both global human motion estimation and dense scene reconstruction. This further suggests a strong correlation between the two tasks and the fact that performing joint optimization necessary. Some qualitative comparisons are shown in Fig. 6. A complete comparison table on all datasets can be found in Appendix. D.
4.5 ABLATION STUDIES
We conduct ablation studies in Tab. 2 to analyze the benefits of each optimization component of JOSH using the JOSH3 variant. From the comparison between Row and Row 4, show that enforcing consistency between the scene contact point and human contact point with the contact scene loss Le1 can significantly improve the performance in all metrics. For example, RTE improves from 4.7% to 1.8%, ATE improves from 22.47 to 3.21, and FFR has been reduced drastically from 92.9% to 2.9%, as the contact scene loss Le1 can guide toward the metric scale of the camera poses and the depth map, as well as improve the global human motion accuracy with better camera poses, thus boosting human-scene interaction plausibility. From the comparison between Row 2 and Row 3, we show that jointly optimizing the human parameters O also leads to a better performance compared only optimizing the scene and the camera pose, improving W-MPJPE from 486.4 to 438.3, which further underscores the significance of joint optimization. From the comparison between Row 3 and Row 4, we show that the contact static loss L,2 is useful in refining reconstruction quality by enforcing contact correspondences to remain static across time steps, which reduces foot sliding from 68.2 to 28.2. From the comparison between Row 5 and Row 6, we use the estimated camera intrinsics instead of the provided ground truth intrinsics. We can see that jointly optimizing the camera focal length f can lead to better reconstruction performance in all metrics, improving W-MPJPE from 1220.7 to 1053.5, ATE from 19.89 16.91, and FFR from 15.2% to 6.8%. These results show the importance of jointly optimizing camera intrinsics with JOSH for in-the-wild settings, where the ground truth often unavailable.
4.6 SCALABLE TRAINING OF JOSH3R ON WEB VIDEOS
As shown in Fig. 7, we experimented with training JOSH3R using ground-truth labels from EMDB (Kaufmann et al., 2023), compared to scalable training on web videos. The results showed that JOSH3R trained with web videos annotated by JOSH surpasses the performance of the one trained on the ground truth dataset by a large margin, improving WA-MPJPE by a significant 59.2%. It also achieves comparable performance the state-of-the-art global human motion estimation

---

Published as a conference paper at ICLR 2026
Table 3: Performance analysis of JOSH and JOSH3R. We compare the efficiency and accuracy of the JOSH; variant and JOSH3R trained with JOSH labels. "FPS" indicates the amortized frames per second with all modules method. All methods are tested on an Nvidia RTX 4090 GPU. evaluate on EMDB (Kaufmann et al., 2023). SLOPER4D (Dai et al., 2023), and RICH (Huang et al., 2022) datasets.
inference
EMDB W-MPIPEL RTESK
SLOPER4D WA-MPIPEL W-MPJPEL RTESL
RICH WA-MPIPEL
Methods
FPS.
WA- WA-MPIPEL
W-MPIPEJ RTES:
SLAHIMR
0.1
326.9
776.1
10.2
706.9
34010
12.6
132.2
237.1
6.4
1272.3
10.5
196.1
TRAM
1.1
76.4
222.4
215.1
1285.3
3.0
127.8
238.0
6.0
0.8
120.0
438.3
1.8
102.6
184.3
JOSIER
15.4
220.0
661.7
13.1
296.3
1148.4
10.4
190.4
334.9
6.3
Ground-Truth
JOSH3R
Figure 8: Qualitative results of JOSH3R. We visualize the human pose of the right (future) frame in the left (current) frame's 's coordinate.
method WHAM (Shin et al., 2024) with a more accurate human trajectory in terms of RTE (8.8% extensive ground-truth human motion data (Mahmood et al., 2019) for training, JOSH3R'S 's high performance reflects JOSH's ability to generalize well to in-the-wild settings and label diverse and realistic global human motion from web videos without manual effort. We believe the performance of JOSH3R can be further improved by scaling web data annotated by JOSH, and we will leave future work.
10.4%) Considering that WHAM
To further analyze the accuracy and efficiency of JOSH and JOSH3R, we conduct experiments in Tab. 3. Since JOSH3R takes a data-driven approach, it performs best on the SLOPER4D dataset, which is most similar to pedestrian street walking videos from JOSH labels. We can observe that while JOSH achieves the best accuracy, JOSH3R significantly faster than JOSH, with the best efficiency of 15.4 FPS vs. 0.8 FPS, allowing real-time inference. Although JOSH3R's accuracy is currently only on par with traditional approaches, these results highlight a promising path toward training efficient, end-to-end global human motion estimation models using pseudo-labels and strong visual geometry backbones. Some qualitative results of JOSH3R are shown Fig. 8.
5 CONCLUSION
We present JOSH, general optimization framework for 4D human-scene reconstruction in the wild. JOSH leverages human-scene contact to jointly optimize the camera pose, the global multi-human motion, and the scene geometry. The results show that the joint optimization of JOSH can not only produce coherent human-scent interaction data but also achieve better performance both global human motion estimation and dense scene reconstruction. We believe JOSH and JOSH3R could enable scalable training of 4D human-scene reconstruction on web videos with real-world human motion and diverse environments.
Limitations. JOSH uses results from off fif-the-shelf models as initialization. Therefore, JOSH will benefit from future models with better performance, while it could suffer from bad initializations even after joint optimization. In addition, JOSH works best when both the human and scene contact points are visible the 2D image, which could fail in some more challenging settings when the contact correspondences are hard to obtain there is contact between the human and the scene.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our work focuses on reconstructing human motion and surrounding environments from monocular in-the-wild videos. We mindful of ethical considerations regarding the use of human data. All datasets employed in this study (SLOPER4D (Dai et al., 2023), EMDB (Kaufmann et al., 2023), and RICH (Huang et al., 2022)) are publicly released for research purposes with appropriate usage licenses, and we adhere strictly to their terms of use. When leveraging additional web videos for large- scale training, we only utilize publicly available, non-identifialle footage and avoid data containing sensitive private information. The reconstructed human motion data are treated strictly as research outputs for advancing computer vision and graphics techniques, and we do not use or intend them for surveillance, biometric identification, any application that could compromise individual privacy or safety. Finally, our models are designed to study human- -cene interaction in general, rather than focusing specific demographic personal attributes, thus limiting risks of discriminatory outcomes.
REPRODUCIBILITY STATEMENT
We have made careful efforts to ensure the reproducibility of our results. All datasets used in this paper publicly available benchmarks (SLOPER4D (Dai al., 2023), EMDB (Kaufmann et al., 2023), RICH (Huang et al., 2022)) following standard evaluation protocols from prior works global human motion estimation (Shin al., 2024) and dense scene reconstruction (Zhang et al., 2024). We detail our optimization objectives, initialization choices, and ablation settings in Sec. 3 and Sec. Hyperparameters, loss weights, and implementation details are specified in Appendix B. To foster reproducibility, we will release the source code of JOSH and scripts for dataset preprocessing and evaluation upon publication. Moreover, we provide a comprehensive comparison across different initialization methods on various datasets using different metrics in Appendices C and to demonstrate the robustness of our framework under varying experimental conditions. Together, these measures should allow other researchers to faithfully reproduce and extend our results.
ACKNOWLEDGMENTS
The project was supported by NSF Grants 2235022, CCF-2344955, and by the ONR grant N0001425 12166.

---

Published as a conference paper at ICLR 2026
REFERENCES
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.1 202212300.1122888 2023. 5, 7, 16
Zhongang Cai, Wanqi Yin, Ailing Zeng. Chen Wei, Qingping Sun, Wang Yanjun, Hui En Pang, Haiyi Mei, Mingyuan Zhang, Lei Zhang, Chen Change Loy, Lei Yang. and Ziwei Liu. SMPLer-X: Scaling up expressive human pose and shape estimation. In Advances in Neural Information Processing Systems, 2023.
Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon- Yoon-Yong Lee. Tracking anything with decoupled video segmentation. ICCV, 2023. 4, 18
Yudi Dai, Yitai Lin, Chenglu Wen, Siqi Shen, Lan Xu, Jingyi Yu, Yuexin Ma, and Cheng Wang. Hsc4d: Human-centered 4d scene capture in large-scale indoor-outdoor space using wearable imus and lidar. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6792-6802, 2022. 15
Yudi Dai, Yitai Lin, Xiping Lin, Chenglu Wen, Lan Xu. Hongwei Yi, Siqi Shen, Yuexin Ma, and Cheng Wang. Sloper4d: scene-aware dataset for global 4d human pose estimation in urban environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.
June 2023. 6, 7, 9, 10, 11, 15, 16, 17, 18
Bardienus Duisterhof, Lojze Zust, Philippe Weinzaepfel, Vincent Leroy, Yohann Cabon, and Jerome Revaud. Mastrr-sfm: a fully-integrated solution for unconstrained structure-frommmmotion, 2024.
URL httph::///// /arsiv///axiv.../ org/ /tt./////445 19152. 2,
Jan Gehl. Cities for people. Island press, 2013.
Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik.
Humans in 4d: Reconstructing and tracking humans with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14783-14794, 2023. 3, 16, 19
Mohamed Hassan, Vasileios Choutas, Dimitrios Tzionas, and Michael J Black. Resolving 3d human pose ambiguities with 3d scene constraints. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2282-2292. 2019. 1, 15
Chun-Hao P Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael Black. Capturing and inferring dense full-body human-scent contact. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13274 13285, 2022. 1, 4, 6, 7, 10, 11, 15, 16, 17
Nan Jiang, Zhiyuan Zhang. Hongjie Li, Xiaoxuan Ma, Zan Wang, Yixin Chen, Tengyu Liu, Yixin Zhu, and Siyuan Huang. Scaling up dynamic human-scene interaction modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1737-1747, 2024. 1, 15
Manuel Kaufmann, Jie Song. Chen Guo, Kaiyue Shen, Tianjian Jiang, Chengeheng Tang. Juan José Zárate, and Otmar Hilliges. Emdb: The electromagnetic database of global 3d human pose and shape in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14632-14643, 2023. 2. 9, 10, 11, 15, 16, 17, 18
Christoph G Keller and Dariu M Gavrila. Will the pedestrian cross? study pedestrian path prediction. IEEE Transactions on Intelligent Transportation Systems, 15(2):994506, 2013. I
Diederik Kingma. Adam: method for stochastic optimization. arXiv preprint 44Xiv:1412.6900 2014. 16
Vincent Leroy, Yohann Cabon, and Jérôme Revaud, Grounding image matching in 3d with mast3r. arXiv preprint arXiv:2406. 20v:24406009756 2024. 6, 7, 8, 16, 18
Jiefeng Li, Ye Yuan, Davis Rempe, Haotian Zhang, Pavlo Molchanov, Cewu Lu, Jan Kautz, and Umar lqbal. Coin: Control inpaantiip diffusion prior for human and camera motion estimation. arXiv preprint 2408 16426, 2024. 2, 15, 17
12

---

Published as a conference paper at ICLR 2026
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European conference on computer vision, pp. 280-296 Springer, 2022. 18
Miao Liu, Dexin Yang, Yan Zhang, Zhaopeng Cui, James M Rehg, and Siyu Tang. 4d human body capture from egocentric video via 3d scene grounding. In 2021 international conference on 3D vision (3DV), pp. 930-939. IEFE, 2021. 2, 15
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 851-866. 2023. 3
Diogo C Luvizon, Mare Habermann, Vladislav Golyanik, Adam Kortylewski, and Christian Theobalt. Scene-aware 3d multi-human motion capture from a single camera. In Computer Graphics Forum, volume 42, pp. 371-383. Wiley Online Library, 2023. 2, 15
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5442 5251,54511111 2019. 10, 15
Lea Müller, Hongsuk Choi, Anthony Zhang, Brent Yi, Jitendra Malik, and Angjoo Kanazawa. Reconstructing people, places, and cameras. arXiv preprint arXiv:2412.17800 2024. 15
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019. 16
Georgios Pavlakos, Ethan Weber, Matthew Tancik, and Angjoo Kanazawa. The one where they reconstructed 3d humans and environments in tv shows. In European Conference on Computer Vision, pp. 732-749. Springer, 2022. 15
Davis Rempe, Tolga Birdal, Aaron Hertzmann, Jimei Yang, Srinath Sridhar, and Leonidas Guibas. Humor: 3d human motion model for robust pose estimation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1 11488-12 1888-114991 1499, 2021. 15
Daniel Sczepansky. Poptravel. https:: cessed: 2024-11-13 Licensed under CC BY. 18
//ww:. hwtp:wyyouuu.ee youtube. com wttp//t.mmmaeeeeee /tpppttaaaeoooor....oot 2024. Ac-
Zehong Shen, Huaijin Pi, Yan Xia, Zhi Cen, Sida Peng, Zechen Hu, Hujun Bao, Ruizhen Hu, and Xiaowei Zhou. World-grounded human motion recovery via gravity-view coordinates. In SIGGRAPH Asia Conference Proceedings, 2024. 2, 15, 17
Soyong Shin, Juyong Kim, Eni Halilaj, and Michael Black. Wham: Reconstructing gorld-grounded humans with accurate 3d motion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2070-2080, 2024. 2, 3, 6, 7, 9, 10, 11, 15, 17
Yu Sun, Qian Bao, Wu Liu, Tao Mei, and Michael J Black. Trace: 5d temporal regression avatars with dynamic cameras in 3d environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8856-8866, 2023. 17
Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems, 34:16558-16569, 2021. 7, 8, 9, 15, 16
Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European conference on computer vision (ECCV), pp. 601-617, 2018.
Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Con- tinuous 3d perception model with persistent state. arXiv preprint 2XXi:2500111388 2025a. 15
13

---

Published as a conference paper at ICLR 2026
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20697-20709, 2024a. 7
Yufu Wang, Ziyun Wang, Lingjie Liu, and Kostas Daniilidis. Tram: Global trajectory and motion of 3d humans from in-the-wild videos. arXiv preprint arXiv:2403. 17346, 2024b. 2, 5, 6, 7, 8, 15, 16, 17
Yufu Wang, Yu Sun, Priyanka Patel, Kostas Daniilidis, Michael Black, and Muhammed Kocabas. Prompthmr: Promptable human mesh recovery. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025b. 2, 17
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20310-20320, 2024. 15
Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in neural information processing systems, 05:38571-38584 2022. 5
Lixin Xue, Chen Guo, Chengwei Zheng, Fangjinhua Wang, Tianjian Jiang, Hsuan-I Ho, Manuel Kaufmannn, Jie Song, and Hilliges Otmar. HSR: holistic 3d human-scene reconstruction from monocular videos. In European Conference on Computer Vision (ECCV), 2024. 2, 3, 15
Ming Yan, Xin Wang, Yudi Dai, Siqi Shen, Chenglu Wen, Lan Xu, Yuexin Ma, and Cheng Wang. Cimi4d: A large multimodal climbing motion dataset under human-scene interactions. In Proceed- ings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12977-12988, 2023. 15
Ming Yan, Yan Zhang, Shuqiang Cai, Shuqi Fan, Xincheng Lin, Yudi Dai, Siqi Shen, Chenglu Wen. Lan Xu, Yuexin Ma, et al. Relil A comprehensive multimodal human motion dataset and method. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2250-2262, 2024. 15
Fengyuan Yang, Kerui Gu, Ha Linh Nguyen, and Angela Yao. Ofcam: Global human mesh recovery via optimization-Fii camera motion scale calibration. arXiv preprint arXiv:2407.00574, 2024. 15, 17
Vickie Ye, Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa. Decoupling human and camera motion from videos in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 21222-21232, 2023. 8, 15, 16. 17
Wanqi Yin, Zhongang Cai, Ruisi Wang, Fanzhou Wang, Chen Wei, Haiyi Mei, Weiye Xiao, Zhitao Yang, Qingping Sun, Atsushi Yamashita, et al. Whac: World-grounded humans and cameras. arXiv preprint ar 24vv: 12959, 2024. 15, 17
Ye Yuan, Umar Iqbal, Pavlo Molchanov, Kris Kitani, and Jan Kautz. Glamr: Global occlusion aware human mesh recovery with dynamic cameras. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 11038-11049, 2022. 15, 17
Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. arXiv preprint arxiv:2410.03825, 2024. 7, 8, 9, 11, 15, 16
Siwei Zhang, Yan Zhang, Federica Bogo, Mare Pollefeys, and Siyu Tang. Learning motion priors for 4d human body capture in 3d scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 1 11343-1 1353, 2021. 15
Yizhou Zhao, Tuanfeng Yang Wang, Bhiksha Raj, Min Xu, Jimei Yang, and Chun-Hao Paul Huang. Synergistic global-space camera and human reconstruction from videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1216-1226 2024. 3. 7, 15, 17
14

---

Published as a conference paper at ICLR 2026
Appendix
We discuss related works in Sec. A. We provide implementation details of JOSH in Sec. B. More results on global human motion estimation are shown in Sec. C. More results dense scene reconstruction are shown in Sec. D. More comparison on the EMDB (Kaufmann et al., 2023) dataset presented in Sec. E. Sec. F shows an analysis of JOSH's robustness to noisy initializations. We then discuss the procedure to label web videos using JOSH for training JOSH3R in Sec. G. The architecture of JOSH3R and its implementation details are discussed in Sec. H. Additional qualitative results, contact label visualizations, and failure cases can be found in the supplementary video.
We do not use Large Language Models (LLMs) in paper writing.
A RELATED WORK
Monocular Global Human Motion Estimation. Estimating global human motion in the world coordinates from a single monocular video is challenging task as the camera motion and the human motion are entangled in the video, and we need to recover both motions in metric units. GLAMR and HuMoe et al., 2021; Yuan al., 2022) propose to leverage the motion prior to human poses and predict the ego movements between frames, leading to noisy and unreliable predictions, especially for the rare poses. An alternative solution is to estimate the camera motion with SLAM (Teed & Deng, 2021) and then transform the camera-coordinatt motion to the global coordinate with the estimated camera poses. The key challenge here is to resolve the scale ambiguity of the poses: TRAM (Wang et al., 2024b) proposes to approximate scale from monocular depth estimation, WHAC (Yin et al., 2024) gauges the scale from the human motion, and OfCam ( (Yang et al., 2024) estimates the scale from human mesh depth. SLAHMR (Ye et al., 2023) proposes an optimization pipeline that jointly optimizes the scale and human meshes using both the human motion prior and SLAM camera poses. WHAM (Shin al., 2024) performs such optimization implicitly by taking the SLAM camera angular velocity as input and learning the human motion prior from 3D keypoint sequences of AMASS (Mahmood et al., 2019). In contrast, COIN (Li al., 2024) learns the motion prior from a diffusion model. Though many methods (Shin et al., 2024; Wang et al., 2024b; Ye al., 2023) include SL.AM module in their pipeline, they treat SLAM as stand-alone module and do not optimize the scene, the human motion, and the camera motion in a single stage. Unlike existing approaches, our method jointly optimizes the dense scene background, the foreground human motion, and the camera motion for 4D human-scene reconstruction. JOSH also supports jointly optimizing the motion of multiple people simultaneously, while many state-of-the of-the-art approaches ( (Wang et al., 2024b; Shin et al., 2024; Shen et al., 2024) focus only on reconstructing the motion of one person.
Human-Scene Interaction and Reconstruction. Existing human-scene interaction datasets (Huang et al., 2022; Dai et al., 2022; Yan et al., 2023; 2024; Dai et al., 2023; Jiang et al., 2024) set up additional sensors like multi-view RGBD cameras, laser scanners, and IMUs or use multiple shots of the same scene (Pavlakos et al., 2022) to first get ground-truth 3D labels of the environment. The human-scene interaction can then be obtained by fitting SMPL human meshes with the ground-truth scene reconstruction and camera poses (Hassan et al., 2019; Zhang et al., 2021). While these approaches can provide accurate results, they are not scalable to casual web videos where both the dense 3D scene and human motion must be estimated from single camera. Only a few works (Luvizon et al., 2023; Zhao et al., 2024; Liu et al., 2021; Xue et al., 2024) have been proposed to tackle the challenging task of monocular 4D human-scene reconstruction, but they perform sequential optimization by reconstructing the camera pose, the scene, and the global human motion separately. Such separate optimization does not fully exploit the synergy between the scene and the human compared to our proposed joint optimization. Müller et al. (2024) performs joint optimization but only reconstructs static scenes with humans from multi-view cameras. Other recent works address the challenging task of 4D dynamic scene reconstruction with moving objects, where Zhang et al. (2024); Wang et al. (2025a) proposes to learn the dynamic scenes with an end-to-end model, and Wu et al. (2024) reconstructs the scene with dynamic 4D Gaussians Splatting. These methods aim to reconstruct general dynamic scenes without treating humans separately, while our method focuses reconstructing scenes together with the global human motion as fine- iin-ggrined SMPL meshes. In addition, most prior scene reconstruction methods can only recover the scene geometry up unknown scale factor, while JOSH can reconstruct the scene metric scale.
15

---

Published as a conference paper at ICLR 2026
Table 4: Effectiveness of JOSH on global human motion estimation. We evaluate the EMDB (Kaufmann al., 2023). SLOPER4D (Dai et al.. 2023), and RICH (Huang et al., 2022) datasets and compare different variants of JOSH with methods using the same local human mesh recovery initializations.
EMDB
SLOPER4D
RICH
Methods
WA WAIMP
W-MPIPEL RIE
WA MPIPP W-MIIPE RIES
WA-MPIPEL
W-MPIPEL
RTER
HMR220 SLAHMR
326.9
776.1
10.2
706.9
34010
12.6
132.2
237.1
6.4
HMR2.0+ JOSH,
130.8
372.3
206.3
1094.2
4.4
101.3
183.6
WHAM WHAM JOSH2
131.1
335.3
4.1
297.7
1272.3
10.5
108.4
196.1
4.5
103.0
242.1
2.4
210.4
994.3
3.0
92.1
158.5
3.7
VIMO TRAM
76.4
1285.3
3.0
127.8
VIMO 3OSHg
68.9
174.7
1.3
120.0
438.3
1.8
102.6
184.3
3.4
B IMPLEMENTATION DETAILS
The loss weights are set to be W2D 1, WSD 1, We1 1, W22 10, Wampl 10, Wsmooth 0.1, W2dkpp = 0.01 in all experiments by grid search. The depth filtering threshold e is set to 0.2m. We use Adam (Kingma, 2014) as our gradient-based optimizer implemented in PyTorch (Paszke et al., 2019), and the learning rate is set to 0.05, with the number of iterations set to 1000. As it slow and computationally expensive to optimize the dense scene point cloud of the entire long sequence, we split each sequence into segments of 100 frames to perform local reconstruction and concatenate the results afterward. To further improve the processing speed, we sample the keyframe every 0.2s and interpolate the camera poses in between.
To get scene initializations with different variants of JOSH, we use depth and optical flow maps from DROID-SLAM (Teed & Deng, 2021) to get initial point maps and correspondences. We can directly obtain point maps and correspondences from MonST3R (Zhang et al., 2024) and MASt3R (Leroy al., 2024). For comparison in dense scene reconstruction, as DROID-SLAM does not directly provide dense scene reconstruction in metric scale, we combine its camera poses with ZoeDepth (Bhat al., 2023) and scale the pose accordingly similar to TRAM (Wang et al., 2024b) to get the dense reconstruction results. MASt3R can provide metric-scale reconstruction without scaling. For MonST3R, we scale the reconstruction results according to the local human depth get metric-scale results. For scene reconstruction evaluation on SLOPER4D, we find all methods will fail on the challenging "seq009" sequence and lead to significantly larger error than the other sequences. We hence exclude it from the scene reconstruction evaluation so the results are not dominated by single outlier case.
C MORE RESULTS ON GLOBAL HUMAN MOTION ESTIMATION
We provide additional results on global human motion estimation on all datasets in Tab. For fair comparison, we compare each method to the JOSH variant with the same local human initialization. The results show that JOSH always leads better performance than its counterpart method on all metrics. For example, JOSH, achieves 206.3 WA-MPJPE on the SLOPER4D dataset, outperforming SLAHMR (Ye et al., 2023), which both uses HMR2.0 (Goel et al., 2023) as initialization. JOSHz improves RTE from 6.0% to 3.4% on the RICH dataset, outperforming its counterpart TRAM (Wang et al., 2024b), which both uses VIMO (Wang et al., 2024b) as local human initialization. These results further demonstrate the general benefit of jointly optimizing the human and the scene in obtaining an accurate global human motion estimation.
D MORE RESULTS ON DENSE SCENE RECONSTRUCTION
We provide additional results on global human motion estimation on all datasets in Tab. 4. For a fair comparison, we compare each method to the JOSH variant with the same scene initialization. The results show that the joint optimization of JOSH can benefit different dense reconstruction methods as well, especially on the most important Chamfer distance metric. For example, JOSH, with PROID-SLAM, (Teed & Deng, 2021) as scene initialization achieves a 1.57 Chamfer distance on the RICH dataset compared the baseline of 1.81 Chamfer distance without human optimization. JOSH, using MonsT3R (Zhang et al., 2024) as scene initialization also produces more accurate camera poses than the baseline the EMDB dataset, which achieves 1.94 ATE compared 2.33
16

---

Published as a conference paper at ICLR 2026
Table Effectiveness of JOSH dense reconstruction. We evaluate the EMDB (Kaufmann al..
2023), SLOPER4D (Dai al.. 2023), RICH (Huang et al., 2022) datasets and compare different variants JOSH with methods using the same scene reconstruction initializations.
EMDB ATE 3
SLOPERAD
RICH CD 4
Methods
ATE
Abs Rel
8c1.25
CD
DROID-SLAM
8,93
30.00
0.33
0.38
21.35
1.81
DROID-SLAM +JOSH
5.57
17.18
16.84
1.57
MonST3R
2.33
15.06
0.21
0.74
13.28
1.70
MonsT3R +JOSH2
1.94
14.53
0.22
0.75
9.09
1.66
1.37
22.47
1.70
MASH3 +JOSH3
1.02
3.21
0.17
0.87
5.31
1.35
Table 6: Global human motion estimation results EMDB.
EMDB
Methods
WA. MPJPE
W-MPIPE
RTES
Jitter
FS
TRACE
2023)
2987.6
GLAMR (Yuan et 2022) 2022)
280.8
726.6
11.4
46.3
20.7
SLAHMR (Ye et 2023)
WHAM (Shin et 2024)
131.1
335.3
4.1
22.5
4.4
WHAC (Vin et al., 2024)
142.2
389.4
OrCaM (Yang
2024)
108.2
317.9
2.2
COIN (Li et al., 2024)
152.8
407.3
3.5
GVIIMR
al., 2024)
TRAM (Wang et al., 2024h)
76.4
222.4
1.4
18.5
23.4
PrompHHMR- (Wang al., 2025b)
16.3
JOSH3 (ours)
68.9
174.7
1.3
9.1
12.1
ATE. This further suggests a strong correlation between the global human motion estimation and dense scene reconstruction, and the fact that performing joint optimization is necessary.
E MORE COMPARISON ON THE EMDB DATASET
As EMDB (Kaufmann al., 2023) the most widely used dataset for evaluating global human motion estimation, we perform an extensive comparison with other methods in Tab. 6. The results show that JOSH3 sets up new state-of-the-ee on human motion estimation accuracy with a 68.9 WA-MPJPE and a 174.7 W-MPJPE. It also achieves decent motion quality with 9.1 jittering and 12.1 foot skating, which are comparable to other methods.
F ROBUSTNESS TO NOISY INITIALIZATIONS
We conducted additional experiments on the robustness of JOSH to noisy initializations. Specifically, randomly perturb 20% of the local HMR, scene depth, and contact labels. For depth labels, add random unit Gaussian noise to the depth values. For local HMR labels, we add a random unit Gaussian noise the local human translation. For contact labels, we randomly replace contact joint with another one. The results are shown in Tab. 7.
We observe that JOSH is robust contact-label noise, as leverages both the dynamic mask and the monocular depth estimates to filter incorrect contact correspondences as discussed in Sec. 3.3, Contact Scene Loss. In addition, our contact modeling employs a robust loss similar to Mast3R- SfM (Duisterhof et al., 2024), further reducing the impact of outliers. For depth-label noise, the degradation more pronounced scene reconstruction than in global human motion. Conversely, local HMR label noise affects global human motion and physical plausibility more severely than scene reconstruction. Despite these perturbations, JOSH consistently outperforms SyncHMR (Zhao et al., 2024) across most evaluation metrics under all three noise conditions. These results demonstrate that JOSH maintains robustness from noisy inputs, which is crucial for 4D human scene reconstruction in real-world settings.
17

---

Published as a conference paper at ICLR 2026
Table 7: Robustness analysis of JOSH SLOPER4D.
SLOPER4D
Methods
WA-MPIPE W-MPIPE
RIE
ATE
Abs Rel
5< 1.25
CD
Jitler
PS
FFR
3
120.0
438.3
3.21
0.16
5.31
7.1
28.2
2.9
JOSHa (+noisy depth)
245.5
865.9
4.9
12.07
0.41
0.63
6.24
7.8
39.1
3.0
JOSHL (+noisy contact)
0.32
1OSH; (enoisy human)
407.5
910.6
1.8
3.81
0.32
0.84
5.19
19.3
119.3
19.4
SynCHMR
233.2
1125.4
17.47
0.25
0.55
17.76
67.4
123.9
9.0
Image
Label
Figure Web videos annotated by JOSH. The first row shows the initial frame of the sequence with pedestrians walking in urban seenes. The second row shows pscudo-labels of the global human motion predicted by JOSH by projecting the future motion to the initial frame.
G LABELING WEB VIDEOS WITH JOSH
We collect 20 hours of web videos of pedestrians walking in cities worldwide posted by content creators on YouTube (Sczepansky, 2024) with diverse pedestrian movements and scene backgrounds. All videos have a Creative Commons license and we remove all personally identifiable information when processing them. We split each video into 5-second clips (150 frames) and collected 3175 video clips. To annotate the global human motion, we first use VitDet (Li et al., 2022) to detect and DEVA (Cheng et al., 2023) to track the pedestrians and run the JOSH3 variant to optimize the multi- human motion of all pedestrians and the scene background in each video clip. In total, we collect about 460,000 frames of global human motion pscudo-labels, Note that while it possible to collect more labeled data for training the end-to-end human trajectory model JOSH3R, the current data scale and the diversity scene backgrounds and human subjects are already much larger than existing real-world global human motion datasets EMDB2 (Kaufmann al., 2023) and SLOPER4D (Dai et al., 2023) with only about 100,000 annotated frames and a few subjects and scenes. This further shows that the lack of diversity of existing datasets limit the learning of end-to-end global human motion models from video inputs and the high potential of learning such models with large-scale pseudo-labels from web data. Some visualizations of the pseudo labels are depicted in Fig. 9, showing the high diversity of pedestrian movements and scene context from web vidcos.
H ARCHITECTURE OF JOSH3R
more detailed architecture of JOSH3R is shown in Fig. 10. We keep the model design simple as baseline for end-to-end global human motion model. We build on top of the siamese network design of MAS13R (Leroy et al., 2024). JOSH3R first takes the feature maps S S' after the frozen MASt3 encoder and decoder layers and applies ROI pooling with the bounding boxes from ViTDet to obtain the features of the scene within the spatial context of the human. We use an additional feature regressor that learns extract relevant information from these features into human tokens H' H' Then, we concatenate S' with H' and S with H to form a complete set of tokens that encapsulates both human and scene information. These tokens are fed into two separate decoders, which contain cross attention modules where human and scene information can interact. The remaining network consists of several prediction heads. The decoded human tokens H H are each passed into a human prediction head that outputs the human's ss local translation T and T in their respective camera coordinates. The decoded scene tokens S'. S are max- mooled then passed through scene prediction head that outputs o² σ¹, which then be used to scale the original MASt3R pointmap predictions as such: o² X11, g.. X21- We reason that current scene reconstruction methods primarily lack in its scale and hence it suffices predict the factor at which scale the scene. Finally, the two human
18

---

Published as a conference paper at ICLR 2026
Human Heads
T
JOSH3R Decoder;
MASI3R Encoder
MASHR Decodirr
Pool
Heads
Relative Trans Head
AT
Relative Rot
AR
MASL3R Encoder
MASI3R Decodery
Scene Heady
JOSH3R Decoder,
Image /
Human Head,
Figure 10: JOSH3R joint human-sceme prediction architecture.
tokens and two pooled scene tokens are concatenated and inputted into two heads, which together produce relative rotation and translation AT, A.
We train JOSH3R for 50 epochs in-the-wild videos annotated by JOSH. The learning rate is set to le-4 with a batch size of on Nvidia RTX A6000 GPUs. We sample image pairs with temporal intervals between 1-10 frames during training and also apply data augmentations, including random center cropping and color jittering. During inference, the image pairs are sampled every 0.2s, and we also interpolate the human trajectories in between. We combine the global human trajectories from JOSH3R with the predicted local human poses from HMR2.0 (Goel et al., 2023) to get global human motion results.
19


Reviews:
[{"review_id": "KRkGFjJgA8", "reviewer": "Reviewer_u1bR", "summary": "- The paper tackles joint scene and human-mesh reconstruction from video, with a focus on modeling human–scene contacts.\n- The proposed approach, JOSH, is an optimization-based method with two stages: (1) initialize point maps, correspondences, human meshes, and contact labels using off-the-shelf models; (2) jointly optimize the scene point cloud, camera parameters, and mesh parameters using priors and contact losses.\n- The method introduces heuristics and losses—static-contact loss, scene-contact loss, and focal-length optimization—to accurately model the camera and humans in contact with the scene.\n- The evaluations are done on multiple datasets: SLOPER4D, EMDB and RICH.\n- Key Baselines: SLAHMR, WHAM, TRAM.\n- The quantative results show that JOSH consistently outperforms the baselines across various scenarios.", "questions": "My questions are primarily centered around the weaknesses mentioned above:\n\n1. Including JOSH3R in the main narrative. If not, justifications for novelty without it.\n2. Robustness to poor initializations.\n3. Pose-alignment comparison on EMDB-1.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper is well written, organized and easy to follow. The ideas are presented clearly and the introduction section particularly is an insightful read. \n\n- Joint human-scene modeling: The problem of jointly modeling scenes and humans is ambitious and forward-looking. The proposed approach, JOSH, is promising, shows strong in-the-wild results, and should encourage the community to design integrated human–scene methods.\n\n- Simplicity of the approach: The core is simple—and that is its strength: initialize correctly and optimize reliably. This offers a promising path to obtain pseudo-3D (human+scene) supervision at scale from web videos. Although the supervision is noisy (compared to capture studios or synthetic), it remains to be seen how learning-based approaches (e.g., JOSH3R) generalize when trained at scale.\n\n- Strong results: The qualitative results in the manuscript (and supplemental videos) clearly show the performance gap between JOSH and baselines such as TRAM and WHAM. The ability to model contacts over long durations without drift is impressive. Quantitatively, it outperforms both human- and scene-reconstruction baselines (MonST3R, MASt3R) across datasets.", "weaknesses": "- Limited Technical Novelty (without JOSH3R): I am suprised that JOSH3R's details (the learning based module built on top of the data collected using JOSH) is relegated to the appendix. Without JOSH3R as a core contribution, JOSH remains an optimization-based method for joint human–scene reconstruction; it can at best provide pseudo labels (although viable at scale). Standalone, it offers limited new technical insight. The optimization procedures largely mirror those in Hi4D (CVPR 2023), Ego-Exo4D (CVPR 2024), SLAMHR (CVPR 2023) with the key difference that the scene is included in the optimization. I acknowledge that incorporating the scene is nontrivial and requires precise execution at scale; however, this does not substantially increase the technical novelty. I strongly encourage including JOSH3R in the main paper narrative.\n\n- Reliance on off-the-shelf initializations: The method acknowledges it can “suffer from bad initializations.” In practice, it stitches VIMO for humans and MASt3R for scenes, so failures propagate into JOSH. The optimization terms can refine but cannot recover from catastrophic errors.\n\n- Comparison with Human only Methods on EMDB: The paper evaluates global human motion on EMDB, while other human/scene metrics are reported on SLOPER4D (less common in HMR). For a fair comparison to human-only methods, please report MPJPE (not world-MPJPE) on EMDB1 (see Table 3 of EMDB) to assess JOSH’s pose-alignment performance post-optimization against state-of-the-art human-only baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PUD8EF1b1i", "reviewer": "Reviewer_VPyC", "summary": "This paper proposes JOSH, an approach for reconstructing the 4D human motion and the 3D environment from a single video. The paper uses initialization information from off-the-shelf models and jointly optimizes the human motion and the environment. This is done by designing carefully the optimization objectives for this joint optimization. The complete system is benchmarked in different datasets and settings demonstrating strong quantitative performance.", "questions": "- I appreciate the demonstration of using different off-the-shelf models for initializing the camera and human motion in Table 1 (e.g., VIMO, WHAM, etc), but it would be helpful to see some direct comparisons. I understand it might not be easy to do all the combinations, but how does VIMO+MonST3R compares with VIMO+MASt3R?\n- I was underwhelmed from the demonstration on in-the-wild videos. It would be helpful to see results on more videos and more challenging ones. For example, how does the method performs on videos from the PoseTrack dataset?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method achieves strong quantitative performance across the board. The paper provides comparisons with multiple baselines on a few benchmarks and JOSH outperform previous works in the majority of cases.\n- Integrating signals for human contact is not easy, because these estimates are very noisy. The paper does this integration carefully, so that JOSH can benefit from the noisy predictions of the off the shelf systems.\n- There is a helpful ablation that considers different aspects of the algorithm and shows the effect they have in the final result.\n- The paper also introduces the JOSH3R model and trains it with pseudo-labels from running JOSH on in the wild videos. Although JOSH3R is not the main focus of the paper, it is a nice addition and shows the benefit of using JOSH for generating pseudo ground truth.", "weaknesses": "- The proposed approach is relatively straightforward for the most part. This type of optimizations are more traditional (e.g., Rempe et al, ICCV 2021 & Ye et al, CVPR 2023), so integrating better initial estimates from off-the-shelf models or refining the optimization objectives will often lead to better results.\n- The use of human-scene contact relies on the contact being visible in the video. Such contact is often available in the simpler benchmark datasets used for evaluation but less common in in-the-wild videos. To be fair, this is something recognized by the paper as one of its weaknesses.\n- The supplementary video does not provide a lot of results for in-the-wild videos. There are a few in the end, but they tend to be on the simpler side when it comes to the scene and/or the camera motion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0b09zWltEo", "reviewer": "Reviewer_QBLa", "summary": "This paper proposes a single-stage optimization framework that jointly estimates camera intrinsics/extrinsics, global multi-human motion (SMPL), and dense scene geometry from a in-the-wild monocular video. The key novelty is to explicitly enforce human–scene contact through two losses: (i) a contact-scene loss that pulls predicted human contact vertices toward matched scene points, and (ii) a contact-static loss that discourages contact sliding across time. The method also optimizes focal length jointly with root depth to correct metric-scale drift common in monocular setups. On SLOPER4D/EMDB/RICH, JOSH variants (with different off-the-shelf initializers) improve human trajectory errors, scene Chamfer distance, and physical plausibility (jitter, foot sliding) over strong baselines and a re-implementation of SynCHMR. They further show that pseudo-labels produced by JOSH enable scalable training of an end-to-end model that outperforms training on limited GT.", "questions": "1. Evaluate foot-skate error, contact F1, and COM-support margin like PPR do.\n2. Compare results with fixed f versus optimized f to show how intrinsics optimization resolves scale drift relative to SynCHMR.\n3. Include contact visulizations with and without contact modeling.\n4. Randomly perturb 20 % of BSTRO contact labels or depth priors; measure degradation in WA-MPJPE and sliding metrics to show robustness of Lc1/Lc2.\n5. Provide inter-person collision or contact consistency metrics to substantiate multi-human capability.\n\nI will reconsider my recommendation if the authors could kindly address these points.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Clean contact formulation. Explicit vertex–scene distance with visibility + depth-prior gating + a temporal static contact term that mathematically targets sliding, which is practical and easy to implement without a simulator.\n2. Focal-length optimization tied to root depth, addressing metric-scale failures when intrinsics are unknown, high leverage for web videos.\n3. General wrapper. Boosts multiple scene/human initializers; evaluation spans human, scene, and physics plausibility metrics.", "weaknesses": "1. No force/stability reasoning. JOSH’s contacts are geometric (distance/static) without force, friction cone; expect residual artifacts under occlusion or weak depth priors (e.g., hand-on-sofa, compliant supports).\n2. Contact detection & masking sensitivity. JOSH assumes reliable contact vertices and segmentation; occlusion/noisy masks may yield wrong correspondences, and the method’s robustness to such errors isn’t deeply quantified.\n3. Dynamic scenes / non-static supports. geometric matching in JOSH presumes a static background for scene correspondences; cannot handle transient contacts and near-contacts through stability/dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t7NrIbNIcD", "reviewer": "Reviewer_rgPA", "summary": "This paper proposes JOSH, a unified optimization framework for 4D human scene reconstruction in the wild from monocular video. This method introduces a human-scene contact loss, which enforces physical plausibility and consistency between the human scene and the surrounding environment. Several experiments on the SLOPER4D, EMDB, and RICH datasets demonstrate that JOSH significantly improves global human motion estimation, dense scene reconstruction, and physical plausibility metrics compared to previous methods (e.g., SynCHMR, WHAM, and TRAM). Another interesting network is JOSH3R, an end-to-end model that generates scalable pseudo-labels for large-scale web video training.", "questions": "Please clarify why penetration remains despite foot contact loss and discuss the potential of adding hand penetration constraints. Provide results with other new monocular camera scene reconstruction methods. Add comparisons with recent methods (e.g., VGGT, CUT3R). Provide JOSH’s time complexity data and JOSH3R’s qualitative/quantitative results to validate real-time capability.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Josh appears to have a lot of competing work these days, but in terms of timeline, Josh is indeed the first to simultaneously optimize camera pose, human motion, and scene geometry using a joint optimization framework.\n2. Josh3R is a good model that demonstrates the feasibility of automatically labeling real web videos for large-scale model training.\n3. Josh's extensive experiments on multiple datasets (SLOPER4D, EMDB, and RICH) validate the effectiveness of his approach, which I highly recommend.", "weaknesses": "1. The overall visualization and experimental results (WA-MPJPE, MPJPE) are acceptable. However, I observed that even with the foot contact loss, there remains some penetration in the visualization (the 41-second mark in the supplementary video). Additionally, given that your work builds on the Rich model, why not consider adding hand penetration constraints? For instance, the first two visualizations in the supplementary video exhibit significant hand penetration.  \n2. The method relies heavily on the initialization model. While monocular camera scene reconstruction methods are advancing rapidly, JOSH still adopts MASt3R. Also, key experiments comparing with this year’s methods (e.g., VGGT, CUT3R) are missing. Including these experiments would better validate the work’s performance.  \n3. Although optimization-based methods are effective, they are computationally costlier than purely learning-based methods. It would be valuable to see experiments on time complexity to evaluate JOSH’s computational performance.  \n4. Theoretically, JOSH3R is capable of real-time performance. I would appreciate seeing results for JOSH3R—both qualitative visualizations and quantitative metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)