Contact-guided Real2Sim from Monocular Video with Planar Scene Primitives
Authors: Zihan Wang
Jiashun Wang
Jeff Tan
Yiwen Zhao
Jessica K. Hodgins
Shubham Tulsiani
Deva Ramanan
Venue: ICLR 2026 Poster (2026)
Keywords: Human Scene Interaction
4D human motion reconstruction
Physics-based simulation for control
Humanoid
Whole body control
Contact modelling
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Simulation-ready asset for Real2Sim from Monocular Videos

Abstract: We introduce CRISP, a method that recovers simulatable human motion and scene geometry from monocular video. Prior work on joint human--scene reconstruction relies on data-driven priors and joint optimization with no physics in the loop, or recovers noisy geometry with artifacts that cause motion-tracking policies with scene interactions to fail. In contrast, our key insight is to fit simulation-ready convex planar primitives to a depth-based point cloud reconstruction of the scene via a simple clustering pipeline over depth, normals, and flow. To reconstruct scene geometry that might be occluded during interactions, we use human--scene contact modeling (e.g., using human posture to reconstruct the occluded seat of a chair). Finally, we ensure that human and scene reconstructions are physically plausible by using them to drive a humanoid controller via reinforcement learning. Our approach reduces motion-tracking failure rates from 55.2\% to 6.9\% on human-centric video benchmarks (EMDB, PROX), while delivering 43\% faster RL simulation throughput. This demonstrates CRISP's ability to generate physically valid human motion and interaction environments at scale, advancing real-to-sim applications for robotics. Code and interactive demos are available at our project website: https://crisp-real2sim.github.io/CRISP-Real2Sim

Published as a conference paper at ICLR 2026
CONTACT-GUIDED REAL2SIM FROM MONOCULAR VIDEO WITH PLANAR SCENE PRIMITIVES
Zihan Wang"
Jiashun Wang'
Jeff Tan
Yiwen Zhao
Jessica Hodgins
Shubham Tulsiani
Deva Ramanan
Carnegie Mellon University
input frames
input frames
Figure We present CRISP, method for recovering 3D human motion and simulatable scene geometry from monocular video. Our key insight is to fit compact planar primitives (blue) to point- cloud reconstructions the scene. We then use the joint human and scene reconstruction to train humanoid controller with RL in simulation (dynamics rollouts of the humanoid + yellow > pink for different timesteps). Extensive experiments on EMDB and PROX show that CRISP lowers motion-trackin failure rates by 8x compared to prior art.
colored as green
ABSTRACT
We introduce CRISP, method that recovers simulatable human motion and scene geometry from monocular video. Prior work on joint human scene reconstruction relies on data-driven priors and joint optimization with physics in the loop, recovers noisy geometry with artifacts that cause motion tracking policies with scene interactions to fail. In contrast, our key insight is to recover convex, clean, and simulation-reade geometry by fitting planar primitives to a point cloud recon- struction of the scene, via simple clustering pipeline over depth, normals, and flow. To reconstruct scene geometry that might occluded during interactions, human posture
make use of human-scene contact modeling (e.g., reconstruct the occluded seat of a chair). Finally, we ensure that human and scene reconstructions physically-plassee troller via reinforcement learning, Our approach reduces motion tracking failure rates from 55.2% 6.9% on human-centric video benchmarks (EMDB, PROX), while delivering 43% faster RL simulation throughput. We further validate it on ccattued videos, Internet videos, and even
by using them to drive humanoid con-
in-the-wild videos including gera-generated videos. This demonstrates CRISP's ability to generate physically- valid human motion and interaction environments scale, greatly advancing real- to-sim applications for robotics and AR/VR. Code and interactive demos are avail- able at our project website: crisp-real2sim. wwtphgggi/.h.iiiieee. S
"Equal contribution.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
What does it mean to understand a video of human? Although there has been tremendous progress in well-studied tasks such as space-time reconstruction or activity recognition, we argue that true human understanding is physical: a person's foot is not simply stepping down, but is placed in contact with ground surface to provide support. Indeed, humans constantly have close interactions with their surrounding environments - they sit on chairs, lie on sofas, and climb stairs. Given a casual monocular video of such an interaction, our goal is to build a "vid2sim" pipeline for generating human-scene reconstructions. Concretely, we wish to reconstruct the scene with enough fidelity to accurately simulate the human, environment, and their interactions while obeying the laws of physics (c.g., avoiding inter- anter-penetrations, foot sliding, and floating geometry). Doing so would unlock scalable learning for applications such as embodied AI and robotics (Allshire et al. 2025). thysically-plausbbe character animation Yuaa et al. 2023). and AR/VR (Luo et al. 2024).
While there has been tremendous progress in reconstructing scenes (Wang et al. 2025a) and hu-
mans (Shin et al. 2024 Wang et al. 2024a Shen et al., 2024)) the interaction between the two is less well-studied. While notable exceptions exist (Liu et al.][2225) prior art still struggles on videos with parallax and occlusions (due to the human body), producing reconstructions with duplicate structures or missing regions (Wang et al.) (025d). Moreover, we find that the accuracy of recon- structions needs to be even more precise for physical simulation; even small amounts of noise in ground plane reconstructions can (literally!) trip up a physical humanoid simulation (see Fig. 4 As such, prior work on driving humanoid simulators with video input focuses on simplistic environ- ments where there is limited interaction with scene geometry (Luo et al. 2023a). A final subtle point is the efficiency of simulation; collision detection often requires geometry that well approximated by convex primitives, which can quickly become expensive for complicated scene geometries.
Contributions. We introduce CRISP, real-to-sim pipeline that converts monocular human videos into simulation-ready assets by integrating human mesh recovery (HMR), 4D reconstruction, and contact prediction. Unlike most reconstruction works, which yield noisy and non-watertight 2.5D geometry, CRISP explicitly builds geometric outputs ready for physics simulation, by fitting clean, convex and watertight planar primitives to point cloud reconstructions via a clustering algorithm. While conceptually simple, this produces simulation-reate fidelity of physical simulation. Second, we make use of state-of-the-arr monocular depth priors to reduce artifacts such as duplicate structures that plague concurrent methods (Allshire et al. 2025). Third, we improve scene reconstruction by reasoning about occluded geometry with inferred human body shape (c.g., the seated pose can infer an occluded chair seat). To do so, models to detect common human scene interactions such as setting-on-a-chane physically-valic reconstruction, we use reinforcement learning (RL) drive simulated humanoid to follow reconstructed human motions while interacting with the reconstructed scene in simulation.
geometry that significantly improves the
vision- Finally, to produce a
Results. We evaluate CRISP across standard human benchmarks (EMDB, PROX) and demon- strate strong gains in both reconstruction fidelity and real-to-sim performance. CRISP achieves a 93.1% real-to-sim success rate, significantly surpassing baselines that collapse under noisy scene geometry. When integrated into RL training pipelines, CRISP supports 43% faster simulation throughput compared to dense-mesh approaches, while maintaining physically plausible interac- tions. Somewhat surprisingly. physical reasoning improves the quality of both the human and scene reconstruction. These results highlight that CRISP not only bridges the video-to- -ommuulatinooon gap, but also makes RL training from in-the-wiid wiild videos practical and efficient for embodied AI and robotics.
2 RELATED WORK
2.1 MONOCULAR HUMAN MOTION ESTIMATION.
3D human motion recovery is most widely formulated as recovering the parameters of a paramet- ric human model, such as SMPL (Loper et al. 2023) or SMPL-X (Pavlakos et al. 2019). Classic methods (Bogo et al. (016) rely on optimization, fitting body model parameters to match the hu- man shape of the input. More recently, feed-forward HMR methods (Kocabas et al., 2020 Shen et al. 2024) directly regress SMPL parameters using deep neural networks such as transformers. Although these works typically recover 3D humans in camera coordinates, recent focus has shifted to recover metric human trajectories jointly with scene geometry and contacts in world frame (Ye) et al. 2023 Wang et al. 2024a)). For example, TRAM explicitly estimates camera parameters using DROID-SLAM (Teed & Deng 2022 by masking dynamic regions, then uses the camera to
2

---

Published as a conference paper at ICLR 2026
Contacl-ongorttee Recenstruciin with Planar Prinsilives
FMM Network
Filting
Visual SLAM
(world)
motion
Contact Model
Monocolar RGB Video
Contact-Clus for Scrne Hallucination
Learsed Wholr-bady Contral Policy
Figure 2: CRISP pipeline. Given a casual RGB video (left), CRISP reconstructs scene geometry and human motion that are used to drive humanoid controller in simulation. After recovering camera poses, intrinsies, and global point clouds (middle-top), we propose a clustering algorithm to obtain small number (~e 50) of compact planar primitives that enable efficient simulation (right- top). also detect human-scene contacts and explicitly use them to recover interaction surfaces that may be occluded, such stair and its platform (middle-bottom), Finally, and scene reconstructions to drive a humanoid controller in simulation via RL (right-bottom).
human
project people into world coordinates. WHAM (Shin al.) 2024) is trained to predict the likelihood of foot-ground contact using estimated contact labels, using these contact estimates to refine the body pose. (Liu et al., 2025) jointly recovers scene geometry, human motion, and contact points, then jointly optimizes all of these observations using human-scene contact constraints. Despite the recent progress in human-scene reconstruction, existing works largely rely on integrating multiple data-driven priors, such as feed-forward HMR and geometric foundation models. Beyond that, we propose to leverage physics simulators to infer simulatable human-scent interactions.
2.2 HUMAN-SCENE INTERACTION
Realistic human-scene interaction has long been a central challenge in the computer vision com- munity. To achieve accurate human-scene motion modeling, modeling contact is essential (Wang 2021 Zhang et al., 2020b Nam et al. 2024). line of work explicitly works on contact prediction (Huang et al., 2022 Dwivedi et al. 2025). On the other hand, physics-basee based control methods have gained significant attention for their ability to produce natural and physically plausi- ble interactions, thanks to physics simulations. For example, Chao et al. (202T) constructed library of policies capable of tasks such as chair sitting by imitating motion capture trajectories. YYu et al. (222I) targeted more dynamic behaviors, training distinet controllers to reproduce complex parkour movements captured from video. Beyond direct motion tracking, adversarial imitation frameworks encourage physically grounded behaviors in indoor environments, leading more natural and di- verse interactions with objects and surfaces (Hassan et al.) (0233 Xiao et al.) (023). Luo et al. (2022) introduce embodied scene-aware human pose estimation, where agent equipped with propriocep- tion and scene awareness recovers human motion in simulation; however, the surrounding scene is given rather than reconstructed. HILL (Wang al. 2025b) explores hybrid imitation learning by combining motion tracking with adversarial learning to train unified parkour controllers from In- ternet videos, but requires manual scene annotations function effectively. Most relevant to our work is the concurrent VideoMimic framework (Allshire et al.) (025), which proposes a real-to- sim-to-real pipeline that jointly reconstructs humans and environments, producing control policies for humanoids capable of skills such sitting and walking, Compared with VideoMimie, our ap- proach provides more accurate modeling of humans, scenes, and contacts, resulting in substantially improved simulation stability, efficiency, and RL success rates.
METHOD
3.1 OVERVIEW
Given a capture monocular video V (4 E H ing with static scene 8 (e.g. parkour, stair climbing, sitting sofa), goal is recover
[ Ji=1 depicting a human interact-

---

Published as a conference paper at ICLR 2026
Sputially
temporally
split
merge
plane
Figure 3: Planar fitting, Given per-frame pointmaps from a visual SLAM system, we (1) produce candidate planar segments by running K-means on normal maps (computed finiteddifferences on pointmaps); (2) spatially-split segments via DBSCAN on 3D points within each segment; (3) temporally-merge segments across frames with similar planar fits and sufficient optical flow corre- spondences (Zhang et al.] 2025). Notably, physical planes that may appear as multiple segments in different frames are merged into a single temporally consistent planar region (pink and blue seg- ments before merging - blue after merging). We then fit plane to each merged planar region using RANSAC and define a planar cuboid with a default thickness of 0.05m. See Appendix for details.
simulatable 3D human motion and scene geometry in a common world coordinate. We begin by inferring camera poses, intrinsics, and a global scene point cloud. Then, to obtain simulatable com- pact planar primitives from the global scene point cloud, we propose simple clustering algorithm (Sec. 3.2). We make use of human-scene contact modeling (Sec. 3.3 to recover scene geometry that might be occluded during interactions. Finally, we ensure that human and scene reconstructions are physically plausible by using them to drive a humanoid controller via RI. (Sec. 3.4.
Human, Scene, and Camera Initialisation Similar to concurrent work (Allshire et al. 2025), we use MegaSAM (Li et al. 2024) to jointly recover camera intrinsics X E Rेउ per-frame camera poses J. [R; & E (3), and a per-frame dense depth map D (d,)) from unconstrained monocular video. To improve the geometry quality, we replace the depth estimator in the optimiza- tion stage of MegaSAN AM with MoGe (Wang et al. (025c), producing scale-invariant dense point cloud together with calibrated camera parameters { To estimate 3D human pose, we pass the intrinsics K to GVHMR (Shen et al. 2024) to obtain SMPL mesh in camera space, then lift the 3D humans to world frame using the estimated camera poses J, ensuring that the human, scene, and camera share single coordinate system. Although MegaSAM reconstructs P up unknown scale factor, the scale of a typical human is known: we use this cue to recover a metric- scale point cloud T by scaling T such that depth of the human in the scaled MegaSAM point cloud matches the depth of the 3D SMPL mesh from GVHMR.
3.2 NORMAL CRAMAL-BASED PLANAR PRIMITIVE FITTING
Why planar primitives? Although reconstruction pipelines typically output point clouds, physics- based simulators (e.g. Isaac Gym) require meshes for collision detection and force calculation. Typical pipelines convert point clouds to 3D meshes by fusing points into a truncated signed distance function (TSDF), then meshing via Marching Cubes. However, not only are these meshes very large (with hundreds of thousands of triangles), but these pipelines often produce noisy meshes, with oversmoothed surfaces in some regions and unwanted geometry artifacts in others. Such artifacts can cause serious issues when driving a humanoid controller in simulation (Fig. 4 The humanoid may bump into reconstruction artifacts and experience unstable contact forces, causing RL-driven humanoids to fail to reproduce the motions observed in video.
Our key insight is that decomposing the scene into small set (<s 50) of convex primitives solve both issues. Convex primitives are small and efficient to simulate, and standardized primitives also regularize the reconstruction, making it more robust to low-level noise. Our specific choice of prim- itive is based on a planar-world assumption: many human-scene interactions such as sitting, lying down, parkour, climbing stairs, etc. can be represented as interactions with planar surfaces. While prior work has investigated planar decompositions via a combination of 2D segmentation priors and global neural fields (Ye et al., 2025). we find it sufficient simply cluster 3D point cloud recon- structions (from visual SLAM) into planar primitives (Fig. 3) yielding an efficient, lightweight, and simulation-reade reconstruction without per-scene optimization.
3.3 CONTACT AS CUE FOR SCENE COMPLETION
Why contact? Given unconstrained monocular video, critical interaction surfaces in the scene might occluded by the human or other scene geometry. For example, person might stand on ground plane that is out of view, sit onto couch that is now occluded by their body. From
4

---

Published as a conference paper at ICLR 2026
an imput frame 1, where the posed SMPL mesh is estimated as M,, aim to estimate per-vertex contact predictions (v) E {0. 1} and use them to guide the completion scene geometry. Given an image of person (potentially interacting with their environment). InteractVLM (Dwivedi et al.) 2025) predicts a binary contact mask over SMPI. vertices that are in contact with the scene. When naively applying InteractVI.M to frames of video, we tend find over-predicts false positives during "near-contact" frames, presumably because it was not trained on such hard negatives.
Kemepora-kinematii filtering. To reduce false positives, we apply non-maximum suppression to contact predictions across time. Specifically, we keep only those predictions with consistently high confidence for L frames and return the frame t with the smallest amount of human motion By:
arg min Ut x (1,6+ 1.)
3.4 MAYSICS-BASER MOTION TRACKING
Following Peng et al. 2018), train fully-constrrined motion-tracking policy TPC to imitate the full-body motion sequence extracted by our pipeline. Specifically, the policy takes as input the character state 8t and the next K target poses 9t = [fe, ft+*). The output is action a to let the character track the reference motion precisely. The policy is trained using a standard motion tracking reward r, which encourages the character to minimize the difference between the state of the simulated humanoid and the reference motion at each timestep t. We follow MaskedMimic (Tessler al. 2024 Wang et al. 2025b) in the design of observation, action, and reward of the model.
Observation The simulated character is constructed based on the SMPL human model (Loper et al., 2023 Luo al.) 2023b). The robot state is represented by a set of features that describes the configuration of the character's body,
81- = (0. e 0000, (p+ ( e 0000 Ut θ 01001)
where 6r and pr are joint orientations (quaternions) and positions, are the linear and angular velocities, and @ represents quaternion subtraction. The policy additionally conditioned on the next K target poses 9r = [[(+11 f.t2...., with joint-wise targets f? = (0/ e 0/. 0 e
graat (p: p. p))) or (f: p, @ 0pot).
Action. Following prior work (Peng et al. 2018 Tessler et al. 2024), actions are parameterized as desired joint targets for a Proportional-Deriant (PD) controller. The stochastic policy π (a: 8t, 9t) is modelled as a multivariate Gaussian with fixed diagonal covariance matrix Ex ax = 0.055. Reward. The reward function is defined
Ftt upe apllfe -Pell + w.e arll + wpe cullde--p
+ w aw l/ -ull wwe -om lhh + We Σ |T
-
where w(-) and ({-) are weights to balance rewards. The reward encourages the robot to imitate the position p. rotation linear velocity p. angular velocity q. and the root height specified by the reference motion. An energy penalty is applied to encourage smoother motion and mitigate jittering.
Training. Following MaskedMimic (Tessler et al. 2024 Wang al. 2025b), the policy is mod- eled using a transformer encoder architecture, and the critic is modeled with a simple MLP. To enhance training stability and efficiency, we adopt the Reference State Initialization (RSI) and Early Termination (ET) strategies introduced in DeepMimic (Peng et al.) 2018). Specifically, at the begin- ning of each episode, the initial state sampled from the reference motion: with probability 10% from the first frame, and otherwise uniformly along the trajectory. An episode is terminated early any joint position deviates by more than 0.5 meters from the reference world coordinates. To better compare how the reconstructed assets from different methods perform in simulation, we train a separate control policy for each motion clip. We utilize Isaac Gym (Makoviychuk et al., 2021] to simulate all the environments with simulation frequency of 120Hz. Policies operate at 30Hz and are optimized with Proximal Policy Optimization (PPO) (Schulman et al. 2017), using generalized advantage estimator (GAE) (Schulman al. 2015
5

---

Published as a conference paper at ICLR 2026
input frames
VideoMimic
CRISP
(Ours)
input
VideoMimic
(Ours)
Figure 4: Qualitative comparison. We compare VideoMimic with CRISP (ours) six sequences (2x3). For each sequence, the top row lists sequential input frames (green > yellow > pink). In the bottom a simulated dynamic motion rollout at the corresponding frame. To mimic the behavior in input frames, the agents depend accurate scene geometry for faithful physics feedback. Thus, physically plausible reconstruction of VideoMimic will cause simulation failure: the agent often: (a) suffers penetrations and gets stuck in 'ghost surfaces'; (b) bounces off surfaces protruding out of the ground: (c) suffers contact errors from over-smoothed structures that degrade motion recovery; (d) collides with scene artifacts; (e) gets trapped by bumpy terrains and (f) gets trapped local dents. In contrast, CRISP produces simulation-ready assets that preserve contact-faithie human- scene interactions, enabling stable policy rollouts in diverse, complex terrain both in indoor and outdoor scenarios. Please see the project page for interactive and in-the-wild examples.
show the scene reconstructions (blue) from VideoMimic and CRISP, along with
EXPERIMENTS
We evaluate CRISP on its ability to bridge monocular reconstruction and physics-based simulation along three dimensions: (i) world-grounded HMR accuracy, assessing the quality of global hu- man motion estimation; (ii) human-scene interaction fidelity, measuring the geometric and physical consistency between reconstructed humans and environments; and (iii) RL-based motion tracking performance, evaluating whether the reconstructed assets can be used for downstream reinforcement learning. Importantly, both the human and scene reconstructions play central role in reliable real- to-sim transfer: low-quality geometry often results unstable contacts and poor RL convergence, whereas accurate human reconstruction supports robust tracking and physically consistent behavior.

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative comparison. We evaluate the humanoid policy success rates (Success, 1), av- erage simulation throughput (FPS 1), reconstruction quality (eDs; $ and CD one 4), non-penetration accuracy (Non-Pene. 1), and HMR quality (W-MPJPE and WA-MPJPE +) compared to VideoMimic, as well as variants of our method with TSDF, NKSR, and Planar Primitive geometry representations. Compared to VideoMimic, our method has nearly half the chamfer distance error, and experiences much higher humanoid policy success rates. Notably, the reconstruction artifacts of VideoMimic often lead to catastrophic failures during simulation and policy rollout (Fig. + Com- pared to strongest dense mesh baseline NKSR, our 2-way chamfer error is slightly larger because our reconstructions are less complete, but our low 1-way chamfer (Recon GGT) error reveals that our planar primitives consistently lie near the ground
PROX(1I)
EMDB(20)
Method
RL
Success
FPS
Success
CD2
eD...
Non-Pene.
Success f
W-MPJPE on (WA-)
VideoMimic
X
0.337
0.311
0.928
$21.09 (110.64)
Ours
x
0,187
0.174
0.909
179.84 (78. 16)
VideoMimic
44.8%
16K
27.3%
0,311
50.0%.
505.31 (145.23)
Ours (TSDF)
75.9%
15K
72.7%
0,178
0.222
0,925
77.8%
197.77 (75.62)
Ours (NKSR)
79.3%
16K
90.9%
0.163
0.187
0.937
75.0%
185.00 (74.77)
Ours (Planar)
93.1%
23K
90.9%
0.187
0.174
0.947
93.8%
175.93 (70.60)
4.1
EXPERIMENT SETUP
Datasets. We conduct experiments on EMDB (Kuffmann et al. 2023 and PROX (Hassan et al, 2019). EMDB provides ground-truth global human motion without paired scene geometry. Follow- ing prior work (Shin et al. 2024 Wang et al., 2024a), we use the EMDB-2 subset with 21 sequences (4 indoor, 17 outdoor). PROX contains pseudo sseudo-ground-teg ground-trutt truth human motion paired with 3D scene annotations in PROX are known to be noisy, most prior work does report direct comparisons on motion accuracy. In line with this practice, we use PROX primarily for evaluating suman-scens interaction fidelity. We standardize each video 600 frames(20s) on average. During meshification, points beyond the 95th-percentile depth (far from the camera) or farther than 2.5m from the pelvis are treated as non-contact and filtered out.
scans in 12 indoor settings. As the geeudo-ground-trus
Baselines. We evaluate our method against two types of baselines: alternative geometry recon- struction methods and alternative human motion recovery (HMR) methods. For both, we use the same RL-based motion tracking framework for comparison of downstream simulation performance. We also extract reference motion and geometry from CRISP and VideoMimic (Allshire et al. 2025) and send them into the same tracking and RL. training pipeline for fair rystem-to-systens comparison.
Geometry: We consider several common scene reconstruction pipelines: (i) meshes reconstructed from TSDF fusion using VDBFusion (Vizzo et al., 2022) and Marching Cubes (Lorensen & Cline, 1987). (ii point clouds reconstructed via NKSR (Huang et al. 2023). and (iii) dense mesh recon- structions adopted by VideoMimic (Allshire et al.) (025). These baselines are compared to our planar primitive representation, which produces lightweight, simulation-reat geometry.
World groudddr HMR. For global human motion recovery, we compare against prior state-of- ti-hhhhan art methods, including GVHMR (Shen et al., 2024), TRAM (Wang et al.) (024a),, WHAM (Shin al., 2024). and VideoMimic (Allshire et al.] 2025). Beyond raw HMR output from VideoMimic and CRISP (ours), we also include the rollout motion from trained RL policies for comparison.
By combining these baselines with the same RL-based physics refinement, we obtain fair assess- ment of how different human and scene models impact the final real-to-sim performance. Notably, when both human and scene reconstructions are high quality, RL refinement produces motions that are not only physically stable but also elosely resemble those observed in the input video.
Metrics. We evaluate all methods using three categories of metrics:
World grounded HMR quality. Following prior work (Shin et al. 2024 Wang et al.) 2024a), we split sequences into 100-frame segments and align the predicted motion with ground truth, using either the first two frames or the full segment. We report WA-MPJPE and W-MPJPPE (mean per-joint position error in millimeters), root translational error (RTE, normalized by trajectory length), as well temporal smoothness metries such jitter and acceleration error.
Human-scens scene interaction fidelity To measure geometric alignment, we compute Chamfer Dis- tance between reconstructed and ground-truth scenes, including bi-directional and one-way variants
7

---

Published as a conference paper at ICLR 2026
input frames
w/o contact
w/ contact
Figure 5: Ablation contact-guided scene completion. We evaluate impact of contact-guided scene completion on three sequences (three columns). For each sequence, the top row lists sequential input frames (green yellow - pink). Below, overlay the dynamics rollout the corresponding time step on the reconstructed scene geometry (blue). The middle row ("w/o contact") reconstruction omits occluded support surfaces (e.g. platforms), causing the humanoid to fall move unnaturally and leading to physically implausible rollouts. In contrast, the bottom row ("w/ contact") includes additional support geometry (orange), producing stable simulations and motion that better tracks the reference human motion. Please see video on website for more details.
(Recon-GTT and GT-oRec Precision is reflected in low Recon GGT error (every reconstructed surface is supported by ground truth), while completeness is reflected in GT-RReco error. We also report the con-penetration score (Non-Pene) following PLACE (Zhang et al. 2020a), which quantifies how often human meshes intersect with scene geometry.
RL-based motion tracking performance. We assess whether the reconstructed assets can be used for reinforcement learning. We report two metrics: task success rate and training throughput (FPS). Success rate is defined as the percentage of episodes in which the robot's joints remain within 0.5 meters of the reference motion throughout the trajectory. FPS measures the overall system through- put, including both environment simulation and policy learning steps, reflecting the efficiency of RL. training. Strong performance CRISP shows it not only reconstructs geometry and motion, but also produces assets that enable controllable, indooffatthull simulation.
4.2 OVERALL REAL-TO-SIM PERFORMANCE
Table summarizes the overall quantitative results for reconstruction fidelity, contact quality, and downstream RL training. Compared to the concurrent VideoMimic pipeline, our method achieves substantially higher RL success rates and simulation throughput: while VideoMimic attains only 44.8% success with 16K FPS, our planar primitive representation reaches 93.1% % success with 23K FPS, significantly improving both reliability and efficiency.
We further ablate the effect of scene representation in Table I Replacing VideoMimic's dense mesh with TSDF mesh already improves RL success, but the TSDF geometry still exhibits over- smoothing and duplicated structures, which reduce contact precision. Using NKSR sharpens the reconstructed surfaces, yielding stronger non-eenetattion and lower Chamfer distance. Our planar primitives push this trend further: they achieve the lowest one-sided Chamfer (CDome. Recon-+GT) while incurring a slightly worse bidirectional Chamfer C'D This gap is expected, since CDss in- cludes the GT- term that penalizes missing fine-grained structures, which are typically non- contact regions human- -eeann-ccene interaction. In contrast, the one-sided Recon-GT eтror directly reflects how accurate the reconstructed geometry is where it does exist. In simulation, missing tiny non-contact details is largely harmless, whereas extra noisy geometry produces unwanted contact artifacts that destabilize policy rollouts. Consequently, despite slightly worse CDni, pla-
8

---

Published as a conference paper at ICLR 2026
Table 2: World-grounder HMR Evaluation on EMDR. We report WA-MPJPE100 W-MPJPE Noot-Translations. Error (RTE), Jitter (J), and Acceleration error (ACCEL). Our method outper- forms all baselines after RL motion refinement.
Method
RL
WA- W-MPJPE
RTEL
Jitter)
ACCEL
WHAM
X
98.45
267.53
3.30
22.57
5.21
TRAM
X
83.61
249.50
1.93
24.00
4.82
GVHMR
X
74.80
200.71
1.90
15.50
4.39
VideoMimic
A
X
110.64
$21.09
2.12
9.29
4.65
VidcoMimic
145.24
505.32
3.00
8.34
4.17
Ours
X
78.16
179.84
1.88
13.04
4.59
Ours
70.60
175.93
1.90
8.14
4.10
nar representation yields the highest non-penetration score and the best RL success rate, making it particularly well-suited to our contact-rich setting. One remaining limitation is that planar decom- position can leave small gaps between neighboring primitives, so the reconstructions may appear visually incomplete; however, we observe that these gaps do not affect simulation quality because key support and contact surfaces are always modeled.
We present a qualitative comparison with VideoMimic in Fig. including both the reconstructed scenes and the outcomes after RL-based simulation. Additional visualizations available project website. These results reveal clear trend: lower-quality human or scene reconstructions propagate errors into the physics loop, leading to unstable contacts and reduced RL tracking success, whereas our higher-fidelity, simulation-opimmized representations enable robust policy learning.
4.3 GLOBAL HUMAN MOTION ESTIMATION
Since our goal is to build a real-to-sim pipeline that remains faithful to the original video, it is essential to validate the accuracy of human motion recovery. Table 2 reports the global motion estimation results on EMDB using world-grounded metrics. Our method significantly outperforms prior approaches in both joint accuracy and trajectory stability. Without RL refinement, our model already achieves competitive WA-MPiPE110 and W-MPJPE1100 comparable to GVHMR and better than WHAM, TRAM, and VideoMimic. With RL-based motion tracking, errors are further reduced to 70.60 mm and 175.93 mm respectively, yielding the best overall accuracy among all methods.
In addition to joint-level metries, method also improves global trajectory consistency, with the lowest root translational error and substantially reduced temporal jitter (8.14) compared to previ- methods. This indicates that reinforcement learning not only reduces pose drift but also stabi- lizes temporal dynamics, producing smoother and more physically consistent motion. By contrast, VideoMimic exhibits large joint errors (> 500 mm) and high drift, showing that low-quality recon- structions severely degrade world-gn worddggrouned motion estimation.
4.4 ABLATION STUDY ON CONTACT CUES
To assess the role of contact cues, we conduct an ablation study with and without our VLM--baeed contact prediction. As shown in Table 3 explicitly incorporating contact signals improves the ge- ometric alignment of reconstructed scenes, yielding lower Chamfer Distance both Recce and Recon->GT directions. The inclusion of contact priors helps refine surfaces near human in- teraction regions, leading to more faithful reconstructions of support structures. In Figure [5] we illustrate several qualitative examples showing that without these contact-guided planar reconstruc- tions, downstream RI. policies often struggle to finish the entire motion.
CONCLUSION
We present CRISP, a real-to-sim pipeline that transforms unconstrained monocular human videos into simulation-ready assets, enabling physically valid and natural human-scene interactions. Our planar fitting yields compact, high-quality, and simulation-ready convex scene primitives, and our contact-guided reconstruction strategy allows us to recover occluded surfaces, while reinforcement learning validates and refines the reconstructed interactions in simulation. Extensive experiments demonstrate that CRISP substantially improves world- grounded HMR accuracy, human scene in- teraction fidelity, and downstream RL training performance.

---

Published as a conference paper at ICLR 2026
REFERENCES
Arthur Allshire, Hongsuk Choi, Junyi Zhang, David McAllister, Anthony Zhang, Chung Min Kim, Trevor Darrell, Pieter Abbeel, Jitendra Malik, and Angioo Kanazawa. Visual imitation enables contextual humanoid control. arXiv preprint X2xx::2500000729. 2025.
Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J Black. Keep smpl: Automatic estimation of human pose and shape from a single image. In Computer Vision-ECCC 2016: 14th European Conference, Amsterdam, The Netherlands, Octo- ber 11-14, 2016, Proceedings, Part V 14, pp- 561-578. Springer, 2016.
Yu-Wei Chao, Jimei Yang, Weifeng Chen, and Jia Deng. Learning to sit: Synthesizing human- chair interactions via hierarchical control. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 5887-5895, 2021.
Sai Kumar Dwivedi, Dimitrije Antić, Shashank Tripathi, Omid Taheri, Cordelia Schmid, Michael J Black, and Dimitrios Tzionas. Interactvim: 3d interaction reasoning from 2d foundational models. In Proceedings of the Computer Vision and Putlern Recognition Conference, pp. 22605-22615, 2025.
Mohamed Hassan, Vasilcios Choutas, Dimitrios Tzionas, and Michael Black. Resolving 3d human pose ambiguities with 3d scene constraints, 2019. URL /ttpttts//axx...or/////aa//
06963
Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, and Xue Bin Peng. Synthesizing physical character-scen, interactions. In ACM SIGGRAPH 2023 Conference Pro- ceedings. pp. 1-9, 2023.
Chun-Hao Huang, Hongwei Yi, Markus Höschle, Matvey Safroshkin, Tsvetelina Alexiadis, Senya Polikovsky, Daniel Scharstein, and Michael J Black. Capturing and inferring dense full-body human-scene contact. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13274-13285, 2022.
Jiahui Huang, Zan Gojeic, Matan Atzmon, Or Litany, Sanja Fidler, and Francis Williams. Neural kernel surface reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4369-43799 2023.
Manuel Kaufmann, Jie Song, Chen Guo, Kaiyue Shen, Tianjian Jiang, Chengcheng Tang, Juan José Zárate, and Otmar Hilliges. Emdb: The electromagnetic database of global 3d human pose and shape in the wild. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14632-14643, 2023.
Muhammed Kocabas, Nikos Athanasiou, and Michael J Black. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5253-5263, 2020.
Yuxuan Kuang, Haoran Geng, Amine Elhafsi, Tan-Dzung Do, Pieter Abbeel, Jitendra Malik, Marco Pavone, and Yue Wang. Skillblender: Towards versatile humanoid whole-body mac-mmaniullation via skill blending. arXiv preprint arXiv:2506.09366 2025.
Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang. Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. Megasam: Accurate, fast, and robust struc- ture and motion from casual dynamic videos. arXiv preprint arXiv:2412.04445 2024.
Zhizheng Liu, Joe Lin, Wayne Wu, and Bolei Zhou. Joint optimization for 4d human-scene recon- struction in the wild. arXiv preprint arXiv:2501.02/58. 2025.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pp. 851-866. 2023.
10

---

Published as a conference paper at ICLR 2026
William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface struction algorithm. In Proceedings of the 14th Anmual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '87, pp. 163-169, New York, NY, USA, 1987. As- sociation for Computing Machinery. ISBN 0897912276. doi: 00.1145/3440.334222 URL. https::/doi. wtt.//i.org//// /00 wtt.r//....111/.... 11../1343370111 37422
Zhengyi Luo, Shun Iwase, Yuan, and Kris Kitani. Embodied scene-aware human pose estimation.
Advances in Neural Information Processing Systems, 35:6815-6828, 2022.
Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, al. Perpetual humanoid control for real- time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10895-10904, 2023a.
Zhengyi Luo, Jinkun Cao, Alexander Winkler, Kris Kitani, and Weipeng Xu. Perpetual humanoid control for real-time simulated avatars, 2023b. URL
06456
Zhengyi Luo, Jinkun Cao, Rawal Khirodkar, Alexander Winkler, Jing Huang, Kris Kitani, and Weipeng Xu. Real-time simulated avatar from head-mounted sensors, 2024. URL https: //arxiv.org//////...vv..... wwwp://.orr//bb//2440000 06862
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2 200: 10470, 2021.
Hyeongjin Nam, Daniel Sungho Jung, Gyeongsik Moon, and Kyoung Mu Lec. Joint reconstruc- tion of 3d human and object via contact refinement transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10218-10227, 2024.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10975-10985, 2019.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. DeepMimic. ACM Trans- actions on Graphics, 37(4):1-14, jul 2018. doi: 10.114531975117:20111.. URL https: //doi. 111.0/33111149221197777 3201311
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Picter Abbeel. High-
dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.024436 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06347, 2017.
Zehong Shen, Huaijin Pi, Yan Xia, Zhi Cen, Sida Peng, Zechen Hu, Hujun Bao, Ruizhen Hu, and Xiaowei Zhou. World-grounded human motion recovery via gravity-view coordinates. In SIGGRAPH Asia 2024 Conference Papers, 2024.
Soyong Shin, Juyong Kim, Eni Halilaj. and Michael J. Black. WHAM: Reconstructing world- grounded humans with accurate 3D motion. IEEE/VV Conf. on Computer Vision and Pattern Recognition (CVPR), 2024.
Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras, 2022. URL httppsss/aaxiii.og//ggaaa//
Chen Tessler, Yunrong Guo, Ofir Nabati, Gal Chechik, and Xue Bin Peng. Maskedmimic: Uni- fied physics-based character control through masked motion inpainting. ACM Transactions on Graphics (TOG), 2024.
Ignacio Vizzo, Tiziano Guadagnino, Jens Behley, and Cyrill Stachniss. Vdbfusion: Flexible and efficient tsdf integration of range sensor data. Sensors, 22(3), 2022. ISSN 1424-8220. doi: 10.3390/s22031 296. URLbtttps http:://www wwww. mdpi 0111222-2220/22//2233

---

Published as a conference paper at ICLR 2026
Zifu Wan, Yaqi Xie, Ce Zhang. Zhiqiu Lin, Zihan Wang, Simon Stepputtis, Deva Ramanan, and Katia P Sycara. Instructpart: trisntoieeted part segmentation with instruction reasoning. Pro- ceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 24202-24227, 2025.
Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.
2025a.
Jiashun Wang, Huazhe Xu, Jingwei Xu, Sifei Liu, and Xiaolong Wang. Synthesizing long-term 3d human motion and interaction in 3d scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patern Recognition, pp. 9401-9411, 2021.
Jiashun Wang, Yifeng Jiang, Haotian Zhang, Chen Tessler, Davis Rempe, Jessica Hodgins, and Xue Bin Peng. Hil: Hybrid imitation learning of diverse parkour skills from videos. arXiv preprint arxiv:2505.122 12619, 2025b.
Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 5261-5271, 2025c.
Yufu Wang, Ziyun Wang, Lingjie Liu, and Kostas Danilidis. Tram: Global trajectory and motion of 3d humans from in-the-wil whld videos. arXiv preprint arXiv:2403.17346, 2024a.
Zihan Wang. Bowen Li, Chen Wang, and Sebastian Scherer. Airshot: Efficient few-shot detection for autonomous exploration. 2024 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pp. 11654-11661. IEEE, 2024b.
Zihan Wang, Jeff Tan, Tarasha Khurana, Neehar Peri, and Deva Ramanan. Monofusion: Sparse- view 4d reconstruction via monocular fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8252-8263. 2025d.
Haoyang Weng, Yitang Li, Nikhil Sobanbabu, Zihan Wang, Zhengyi Luo, Tairan He, Deva Ra- manan, and Guanya Shi. Hdmi: Learning interactive humanoid whole-body control from human videos. arXiv preprint arXiv:2509. 16777, 2025.
Zeqi Xiao, Tai Wang, Jingbo Wang, Jinkun Cao, Wenwei Zhang, Bo Dai, Dahua Lin, and Jiang- miao Pang. Unified human-scene interaction via prompted chain-of-contacts. arXiv preprint arXiv:2309.079/8 2023.
Hanqiao Ye, Yuzhou Liu, Yangdong Liu, and Shuhan Shen. Neuralplane: Structured 3d recon- struction in planar primitives with neural fields. In The Thirleenth International Conference on Learning Representations, 2025.
Vickie Ye, Georgios Pavlakos, Jitendra Malik, and Angjoo Kanazawa. Decoupling human and camera motion from videos in the wild. 2023.
Ri Yu, Hwangpil Park, and Jehee Lee. Human dynamics from monocular video with dynamic camera movements. ACM Transactions on Graphics (TOG), 40(6):1-14, 2021.
Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided human motion diffusion model, 2023. URL htttpa:///// wttpaaa.wivv........aaao....oo.a wttp:orrr/ab//222.0000000
Siwei Zhang, Yan Zhang, Qianli Ma, Michael Black, and Siyu Tang. Place: Proximity learning of articulation and contact in 3d environments. In 2020 International Conference on 3D Vision (3DV), pp. 642-651. IEEE, 2020a.
Yan Zhang, Mohamed Hassan, Heiko Neumann, Michael J Black, and Siyu Tang. Generating 3d people in scenes without people. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6194-6204, 2020b.
Yuchen Zhang, Nikhil Keetha, Chenwei Lyu, Bhuvan Jhamb, Yutian Chen, Yuheng Qiu, Jay Karhade, Shreyas Jha, Yaoyu Hu, Deva Ramanan, et al. Ufm: A simple path towards unified dense correspondence with flow, arXiv preprint arXiv:2506.09278 2025.
12

---

Published as a conference paper at ICLR 2026
A
APPENDIX
The Use of Large Language Models (LLMs).. The authors confirm that LLMs are used to polish paper writing in abstract and introduction sections. Some captions use output by LLMs.
Reproducibility statement. We have tested all code on publicly-availabe online videos and have validated its success. All experiments were conducted on standard benchmarks and additional pub- licly available internet videos. Code and data will be open-sourced upon acceptance.
B DETAILS OF PLANAR FITTING.
Algorithm 1: Planar Fitting from [T, N. R, with II e (0, IN
Input: Per-frame points {P:)-= with P. E Nx3; optical flows { covisibility masks {.
Output: R E Mx333 t RXX3 S Mx3 and II E {0, (0,1)[NNTMM)
(1) Per-frame segmentation
for t= I....,TTdo
(a) Estimate normals from points: Pt [N[33] 7 Nr [N,3].
y/ [N] (labels in {l,..., ......K K}). (c) Spatial clustering: For each k, take X₁= = {p E P. (p) = k} [N DBSCAN, {
(b) Cluster normals into K groups: N₂ []
KMeans
(2) Cross-Trame association (set view)
for each pair () with Φi-jj and C do
(a) Warp: For each segment a E 8;, warp Si,a from frame i to frame by Dj-+j-
(b) Score & Link: Let A S, B = S, with
A. B. For each E Sj. compute overlap ratio Pab between the warped S,, and Sj.6. and normal cosine Toob = (fi. Nz,6)-
Connect (i, a) + (j. if both scores above fixed thresholds. Set X An
the
warped domain (accepted overlaps) and produce + B - X segments instead.
10
(d) Unify across pairs: Aggregate all (A X) over time = M global planar groups. correspondence
// we have found
11 (3) Primitive fitting (global primitive)
12 for I,..., do
13
(a) Plane fitting (RANSAC): Xm =
+ (n, c) and inliers.
14
(b) Pose: Project inliers to the plane; fit min-area rectangle to get in-plane axes (x, y): set
Rm [x n| hrighd-handed). (Defer center update to (c))
15
(c) Size & Center: Set Sm [S,, Sy. S,] from in-plane coverage and cormal-direction
spread; then set
A S2 n and tm = c A.
16
(d) Split (optional): the footprint is not rectangular, split along the principal in-plane axis and refit on each part.
17 (4) Contact-guided hallucination (opt.) Repeat the above fit on predicted contact points (world coords) to augment planes, with S, 2 0.05 m clamp.
18
return (R, t, S. IT)
C CONTACT-GUDD GUIDED SCENE COMPLETION: RELIABILITY ANALYSIS
Our contact-guided completion predictions. Specifically, we retain windows of length L with consistently high contact confidence (threshold 7) and low body speed (threshold z), then choose the most stationary frame per window. We find this reduces premature "imminent contact" false positives and improves the stability. When does contact guidance help? Contact-guided completion is most beneficial when support surfaces are occluded (c.g.. chair seats, stair treads, platforms). In these cases, completing even a small missing support region can prevent catastrophic falls and significantly increase rollout success. We use RL-based motion tracking to validate and physically ground the recovered human motion within the reconstructed scene. Concretely, RI. optimizes control policy that tracks the recovered
temporal-kinematic filtering suppress colse-positive contact
13

---

Published as a conference paper at ICLR 2026
Table 3: Ablation on Contact (no RL-refine). Green enabled. red = disabled. For PROX, higher Success/ / Non-Pene and lower CD are better; for PKR, we report Success only, CD (two-way) the symmetric Chamfer, while Re-Re and Recon- +GT are the directional terms. Contact modeling on the PROX dataset does not increase the success rate because most sequences involve sitting; when the seat is missing, the motion simply switches to a squatting pose. However, contact modeling still improves physical plausibility quantitatively as shown in Fig. 5
Method
Contact?
PROX
Success
CD (two-way)
CDS -Reon + C'
Non-Pene
VideoMimic
x
27.3%
0.337
0,3625
0.3114
0.928
x
90.9%
0.193
0.211
Ours
v
90.9%
0.187
0.199
0.173
0.947
reference motion while respecting the scene's s contact constraints. This process refines the simulated motion rollouts (e.g., reducing drift/jitter and eliminating physically implausible penetrations), and serves as an end-to-end test of whether the reconstructed assets are simulation-readly Importantly, in the current implementation RL does not directly modify the reconstructed scene primitives; rather, it uses them as collision geometry and assesses whether the motion can be executed stably.
Failure pattern of contact guidance. Although empirically find that contact completion method can improve overall reconstruction quality by hallucinating support surfaces, we find that the estimated contact-augmented planar primitives are occasionally inaccurate. We use an HMR network to infer 3D human pose and shape, then run an off-the-shelf contact prediction module to infer binary contact masks on the canonical SMPL mesh. This paradigm highly depends on the accuracy of HMR and contact estimation: if the human mesh drifts in world coordinates, the resulting
planar primitives would drift as well. Most of the time, this appears as slightly tilted planar primitives with a small amount number of offset translation. For future work, improving the planar fitting or post- post-processing the contact-completed geometry to make it more scene-grounded is promising direction.
D LIMITATIONS AND FUTURE WORK
We summarize several failure cases:
(1) Inherent limitations of planar primitives. While we empirically show that curved or round objects can often be approximated reasonably well (See our website for results), highly curved or organic shapes may appear faceted or under-fitted. Although this limitation does not significantly re- duce locomotion success rates, we believe that integrating our method with more expressive convex primitives (e.g., superquadries) could further improve fine-grained shape details for curved surfaces.
(2) Unable to model fluid and deformable objects. Our method assumes static, rigid scene geometry. Thus, we are not able to faithfully model fluids (e.g., sand in desert, water in a lake) or deformable objects (e.g., folded cloth, soft cushions when in large deformations). Nonetheless, the existing codebase could be extended to dynamic rigid objects or scenes by incorporating time- varying primitives.
(3) Unable to model dynamic objects. CRISP is limited to static human- -cenn interaction, and does not currently support dynamic scenes or moving objects. The RL training is for locomotion only and lacks loco-manipulation ability as in existing pipelines Weng et al. (2025): Kuang et al. 2025).
Future work. Extending CRISP with more expressive primitives (e.g.. superquadrics or other para- metric shapes) and supporting dynamic human-object interaction handling are exciting directions for future work and could further improve local geometric fidelity, especially for modeling of complex scene and coupling affordance prediction (Wan et al. 2025) for manipulation interactions.
E RUNTIME ANALYSIS
We test CRISP on a 300-frame (10s) video with resolution 1440 1920 on a single RTX A6000 GPU. We report the cost for each component of our pipeline in Table 4 Meanwhile, VideoMimic requires 1282.94, .44 on the same machine to prepare geometry and reference motion before RL. From the result, and inferring the feed-forward priors that are required (monocular depth and flow estimation). Our planar fitting algorithm is lightweight, meaning that the core geometric step (from points to planar
conclude that the main computational bottleneck is Visual SLAM (i.e. MegaSAM)
14

---

Published as a conference paper at ICLR 2026
Table 4: Runtime and memory breakdown of CRISP 300-frame (10s) video at 1440
1920
resolution using single NVIDIA RTX A6000,
Module
Runtime (s)
Runtime (min)
VRAM (MiB)
Proportion (%)
1. Prior preparation
297.33
4.96
4,944
32.3
2. Visual SLAM
518.18
8.64
11,936
56.3
3. HMR (GVHMR)
30.51
0.51
4,940
3.3
4. Planar fitting
74.97
1.25
8.1
Total
920.99
15.35
11,936
100.0
primitives) can be run in real time on incoming frames. When coupled with a real-time RGB-D SLAM and detection system (Wang al. (024b), expect that the pipeline can operate in real-time fashion, using sensor depth and flows obtained from relative camera motion.
F SCENE-AWARE POLICY.
Beyond the scene-blind baseline in the paper, we implemented and evaluated scene-aware variant that takes geometry information at runtime. Specifically, we downsample sparse number [N=2.8k] of 3D points from nearby planar primitives, represent these points the humanoid's i local frame, and apply PointNet to extract global features which serve as additional input tokens for the trans- former policy (scene-blind: proprioception + target poses; scene-aware: proprioception + target poses point cloud). This scene-aware policy achieves higher success rates and better obstacle avoidance some sequences; we show the comparisons on the project website. However, the main goal of our work is not train controller that compensate for imperfect reconstructions, but to test how faithfully our reconstructed scenes and motions support direct simulation (a ("real-to-sim"") scene-aware policy can learn avoid obstacles and route around reconstruction errors, but doing so reduces its sensitivity to reconstruction quality and makes it harder to reveal underlying recon- struction issues. Our experiments show that once geometry and motion capture are accurate, the blind policy already behaves robustly; scene-aware inputs provide an optional extra performance gain for deployment (e.g., when targeting sim-to-real), but they are not required to substantiate the central claims of the paper.
G SEAMLESS TRANSFER TO CURRENT SIM-TO-REAL PIPELINE.
We believe that CRISP can be readily integrated into existing sim-to-real robot pipelines: for ex- ample, it is drop-in replacement for stages and 2 of VideoMimic's 4-stage pipeline (1. MoCap pretraining. tracking reconstructed video motions, 3. distillation, 4. RI. finetuning).
H RL TRAINING DETAILS
For RL training, we use discount factor of 0.99 and apply GAE (Schulman et al. 2015) with т 0.95. The policy optimized with a learning rate of x 10-5, Training is conducted in parallel with 2048 environments, and each update step uses batch size of 8192 for the policy and critic.
The tracking reward defined with the following weights:
Wp = 2.5, w, = 1.5, w. = 0.5,
wa = 0.5,
Wh = 1, W. 0.001,
ap 1.5, a, 0.3, ar 0.12, at 0.05, ah 20.
For the policy architecture, adopt transformer encoder with a latent dimension of 256, feed- forward size of 512, two layers, and two self-attention heads. The critic is implemented as a multi- layer perceptron with hidden sizes [1024, 512].
We show the learning curves of our method and VideoMimic on each motion clip in Figure Our reconstructed assets allow the agent achieve higher rewards more quickly, indicating faster conver- gence and better overall tracking quality. In addition, we visualize the torque reward over training steps in Figure 7 Compared to method, VideoMimic consistently requires the robot exert
15

---

Published as a conference paper at ICLR 2026
higher torques to complete the same motions. This reflects the lower quality of the scene sets produced by VideoMimic, which introduce unwanted geometry artifacts and unstable supports, thereby forcing the policy to compensate with excessive control effort.
cutdioy wiit
puddoor long
mettocr aak lnges
Intooz
Dpm
dontfell
big statrs
ouodddor Jallo
Voeafe
Sirps
HB9:112
MPH15
NTHLLL
N2:15
S1ze R059/s
S1pp
S4go
Stan 'Wekroum
N3LBr33V
N/0gp 1iroa
Figure 6: Reward curve.
16

---

Published as a conference paper at ICLR 2026
Jong.wek
35.irdoor
Weps
Stes
autder
d1.cuddorr uxit
35. (5.outnnrr eraignt
20gt
Ztpee
Ztape
Srips
79,0000eer naR reclangte
93.00/deer AER cedee
NPH13
MPH1332
NETELS
Sxppa
Steps
Stepe
N:50fb
Maubrory
N30ppnsrreo
Werkruum
2018
Cuil
Call
Figure 7: Torque reward curve.
17


Reviews:
[{"review_id": "yNZVpUA4SM", "reviewer": "Reviewer_z8C3", "summary": "In this paper, the authors proposed to use planner primitive to approximate the real world scene \"geometry\".\nThe author uses MegaSAM to recover the global scene point cloud and uses planar primitive fitting to approximate the scene geometry.\nCombined with RL training, this allows for recovering the human motion and the scene geometry from a monocular video.", "questions": "1. How fast can the scene reconstruction run and can it be done in real time?\n\n2. Is it possible to do runtime scene reconstruction which only considers a subset of the scene geometry?\n\n3. Is it possible to input the scene information into the policy observation and apply it in a runtime fashion?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. It provides a nice and simple pipeline for recovering the human motion and the scene geometry from a monocular video.\nGenerating motions which handles human-scene interactions is a very challenging task\nand crucial for downstream applicaitons in animation and robot loco-manipulation.\n\n2. The paper is well-written and the results are convincing.\nAs can be seen in the paper,\na rich set of experiments and ablation studies are conducted to evaluate the performance of the proposed method.\n\n3. The method appears robust across a diverse set of static scenes where the agent is vaulting, parkouring, and climbing.\n\n4. This paper provides more robust scene reconstruction in sim compared to baselines such as VideoMimic.", "weaknesses": "1. The paper does not present sim2real results as in VideoMimic.\nConsidering that the two work are very similar in terms of the pipeline,\nit would make presenting only real2sim results rather incremental and less convincing.\n\nI would argue the fps in training is not that important as people have been exploring universal sim RL policy,\nmaking retraining less needed.\nBut I think the proposed method will actually make more sense in real time sim2real setting,\nwhere scene needs to be reconstructed on the fly and perhaps that's where the planer primitive fitting will come in handy.\n\n2. It's hard to tell how scalable and robust the algorithm is given that only a collection of scenes from PROX and EMDB are used.\nIt will be very helpful if the authors could provide more experiments on more diverse scenes,\nand maybe even extend to the in-the-wild videos from youtube for example.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ioRB8E0Sl", "reviewer": "Reviewer_ZtWx", "summary": "The authors introduce CRISP to recover human motion and scene geometry simultaneously from monocular videos. CRISP first fits convex planar primitives to depth-based point clouds, then addresses the occlusions with human-scene contact modeling. Finally, an RL policy is trained to simulate the reconstructed human-scene interaction.", "questions": "- How was K in L232 decided?\n\n- How long does it take to recover one sample?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The overall pipeline is reasonable and practical. The idea of exploiting planar primitives achieves a balance between representativity and efficiency.\n\n- The performance is impressive and superior to its counterparts. \n\n- Comprehensive analyses are provided, revealing meaningful findings for the real2sim of HSI.", "weaknesses": "- The symbol system is a little complicated. Also, there are duplicated symbols like K in L232 and L243. Annotating them in pipeline figures, such as Figure 2, might help.\n\n- In Table 2, the ground truth jitter should be provided for reference.\n\n- It might be better if some failure cases could be shown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q8W6yqwxaz", "reviewer": "Reviewer_9RmM", "summary": "This paper presents a framework for simulation-ready human scene reconstruction from a single video. The authors introduce a planar primitive representation to better model planar regions such as the ground and propose a contact term to improve physical plausibility. The paper is clearly written and provides several qualitative results demonstrating the effectiveness of the approach.", "questions": "What will be the result if the video is a human lying on a yoga ball?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written and easy to follow.\n\n2. The idea of introducing planar primitives to constrain plane reconstruction is intuitive and practical.\n\n3. The proposed contact term helps ensure more physically consistent interactions between humans and the environment.\n\n4. The experiments validate the overall framework, and the inclusion of numerous demos and visualization results makes the paper more understandable and accessible.", "weaknesses": "1. Although I appreciate the effort in building the proposed framework, it somewhat gives the impression of a technical pack, combining many existing components rather than introducing fundamentally new concepts.\n\n2. Table 2: The reported decimal precision is inconsistent across metrics and should be standardized for clarity.\n\n3. Experimental results are not sufficiently comprehensive:\n\n    - The ablation study is limited; for instance, it is unclear how the number of primitives affects reconstruction quality.\n\n    - There is no report of the model’s runtime, computational cost, or efficiency.\n\n    - No failure case analysis is provided.\n\n    - Some of the demos provided by UPP do not seem to align properly.  In several examples, the input video and the output result do not match in duration or content.\n\n4. The paper lacks a discussion of limitations, such as handling complex or irregular objects in the scene (e.g., interactions with yoga balls or sandbags).\n\n5. There are several typos and minor grammatical issues throughout the paper, and need careful proofreading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "saLxyEkBNy", "reviewer": "Reviewer_8a6k", "summary": "The paper introduces CRISP, a method for recovering simulatable 3D human motion and scene geometry from monocular video. Its key point is the use of compact planar primitives for scene representation, which are more efficient and robust for physics simulation than traditional dense meshes. CRISP also employs contact-guided modeling to reconstruct occluded interaction surfaces. By validating the reconstruction through Reinforcement Learning (RL), the method achieves a high real-to-sim success rate and reduces motion tracking failure rates, while speeding up RL simulation throughput. This makes the \"vid2sim\" pipeline practical for embodied AI and robotics.", "questions": "no.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper's strengths lie in its high empirical quality and practical value for the Real-to-Sim pipeline. It replacing noisy dense meshes with convex planar primitives for simulation efficiency. This engineering choice, combined with clever contact-guided scene completion, results in a highly robust system that achieves a  good reduction in motion tracking failures and an acceleration of RL training throughput.", "weaknesses": "The primary weakness of CRISP is its limited conceptual novelty, functioning largely as a highly effective \"set of practical engineering approaches\" that integrates existing components (HMR, 4D reconstruction, contact prediction) rather than introducing a new framework. Specifically, the use of planar primitives, while practical, relies on a simple, non-learned clustering pipeline that may struggle with complex, non-planar geometries (e.g., curved furniture, organic shapes), limiting its generalizability beyond the \"planar-world assumption.\" The Contact-Guided Scene Completion is a heuristic that depends on pre-trained vision-language models for interaction detection, which introduces a dependency on external, potentially brittle, classifiers. The two core designs—reliance on planar primitives and contact-guided scene completion—may boost computational efficiency but fail to address the key challenges in scene reconstruction, leaving the work unable to grasp the true hurdles of such tasks.\nIn addition, the experimental comparison in Figure 2 is not fair to the baseline method (VideoMimic). The authors represent the scene using compact planar primitives, which, while simplifying the optimization for motion tracking, lead to significant scene distortion. This representation creates a larger sim-to-real gap for the subsequent humanoid tracking task. For a more equitable comparison and to better prepare the method for real-world deployment, the authors should consider investigating learning a motion prior from these compact scene interactions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)