Unified 3D Scene Understanding Through Physical World Modeling
Authors: Wanhee Lee
Klemen Kotar
Rahul Mysore Venkatesh
Jared Watrous
Honglin Chen
Khai Loong Aw
Daniel LK Yamins
Venue: ICLR 2026 Poster (2026)
Keywords: 3D Scene Undertstanding
Visual World Models
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Understanding 3D scenes requires flexible combinations of visual reasoning tasks, including depth estimation, novel view synthesis, and object manipulation, all of which are essential for perception and interaction. Existing approaches have typically addressed these tasks in isolation, preventing them from sharing a common representation or transferring knowledge across tasks. A conceptually simpler but practically non-trivial alternative is to unify these diverse tasks into a single model, reducing different tasks from separate training objectives to merely different prompts and allowing for joint training across all datasets. In this work, we present a physical world model for unified 3D understanding and interaction 3WM, formulated as a probabilistic graphical model in which nodes represent multimodal scene elements such as RGB, optical flow, and camera pose. Diverse tasks emerge from different inference pathways through the graph: novel view synthesis from RGB and dense flow prompts, object manipulation from RGB and sparse flow prompts, and depth estimation from RGB and camera conditioning, all zero-shot without task-specific training. 3WM outperforms specialized baselines without the need for finetuning by offering precise controllability, strong geometric consistency, and robustness in real-world scenarios, achieving state-of-the-art performance on NVS and 3D object manipulation. Beyond predefined tasks, the model supports composable inference pathways, such as moving objects aside while navigating a 3D environment, enabling complex geometric reasoning. This demonstrates that a unified model can serve as a practical alternative to fragmented task-specific systems, taking a step towards a general-purpose visual world model.

Published as a conference paper at ICLR 2026
UNIFIED 3D SCENE UNDERSTANDING THROUGH PHYSICAL WORLD MODELING
Wanhee Lee
Klemen Kotar Khai Loong Aw
Rahul Mysore Venkatesh
Jared Watrous
Honglin Chen2*
Daniel L. K. Yamins-
Stanford University
2OpenAl
ABSTRACT
Understanding 3D scenes requires flexible combinations of visual reasoning tasks, including depth estimation, novel view synthesis, and object manipulation, all of which are essential for perception and interaction. Existing approaches have typ- ically addressed these tasks in isolation, preventing them from sharing a common representation or transferring knowledge across tasks. A conceptually simpler but practically non-trivial alternative is to unify these diverse tasks into single model, reducing different tasks from separate training objectives to merely dif- ferent prompts and allowing for joint training we present a physical world model for unified 3D understanding and interaction (3WM), formulated multimodal scene elements such as RGB, optical flow, and camera pose. Diverse tasks emerge from different inference pathways through the graph: novel view synthesis from RGB and dense flow prompts, object manipulation from RGB and sparse flow prompts, and depth estimation from RGB and camera conditioning, all zero-shot without task-specife training. 3WM outperforms specialized baselines without the need for finetuning by offering precise controllability, strong geomet- ric consistency, and robustness in real-world scenarios, achieving tfte--ofttheart performance on NVS and 3D object manipulation. Beyond predefined tasks, the model supports composable inference pathways, such as moving objects aside while navigating a 3D environment, enabling complex geometric reasoning. This demonstrates that unified model can serve a practical alternative to fragmented task-specific systems, taking a step towards general-purper visual world model.
all datasets. In this work,
probabilistic graphical model in which nodes represent
INTRODUCTION
Understanding 3D scenes from visual data fundamental challenge in computer vision and an important step toward building world models that support real-world interaction. Holistic 3D under- standing requires perceiving visible surfaces and reasoning about hidden geometry consistent way. For example, grasp toy cat, must infer the back underside that is not directly visible. An agent navigating a cluttered hallway must estimate depth for free space, rea- son about occluded structure, and manipulate objects to move them out the way when necessary. These scenarios highlight that 3D perception depends on flexible combinations of reasoning modes, and that general vision model capturing this broader structure can provide stronger foundation than systems designed narrowly around specific benchmarks.
physically
Three essential components of 3D understanding are depth estimation, novel view synthesis, and object manipulation, as they collectively capture the challenges of perceiving geometry, inferring unseen views, and reasoning about dynamic interactions. Existing approaches address these tasks in- dependently within specific frameworks and finetuned narrow datasets, which leaves each model with its own limitations. Depth models cannot infer occluded regions Yang et al. (2024): Waag et al. (2024a). novel view synthesis methods struggle with geometric consistency and precise con-
trol Sargent et al. (2024); Yuu et al. (2024). object manipulation models Pandey et al. (2024): Wu (224)) allow localized editing but do not extend to scene-level reasoning, and none of them
"Equal contribution.

---

Published as a conference paper at ICLR 2026
can flexibly handle tasks that they not optimized for. An alternative is to adopt unified frame- work that trains on a superset of all tasks together, free from fixed input- designs and dataset constraints. In such framework, supervision from diverse tasks and datasets may shape a repre- sentation grounded in the physical scene, enabling knowledge transfer across tasks and supporting robust generalization.
Unifying the diverse components of 3D understanding into a single model that works robustly in real-world scenarios is difficult because it requires both flexibility and controllability. Most previous approaches are restricted to a single form of input and output, for example, mapping RGB images depth, and therefore cannot flexibly support the diverse queries that arise in 3D understanding. Additionally, achieving strong controllability in generative models has been challenging, motivating extensive research on incorporating additional conditioning signals and architectural modifications to better steer the generation process, most notably exemplified by ControlNet (Zhang et al. (2023). Yet reliably enforeing such controls in vision models remains unsolved, particularly for 3D under- standing tasks, where outputs often deviate from the intended specification or show distortions in object and scene appearance despite ongoing efforts to address these issues Zhou et al. (2025); Shi et al. (202)). These limitations motivate the need for new general framework that unifies tasks through flexible prompting while ensuring precise geometric control and physical consistency.
In this work, propose a physical world model for 3D understanding (3WM) that provides uni- fied framework supporting diverse inference pathways while remaining precisely controllable. We formulate the model probabilistic graphical model (PGM) in which nodes represent multimodal scene elements such as RGB patches, optical flow patches, and camera pose. The graphical model formulation allows traversal across modalities and spatial locations, making it possible construct flexible inference pathways guided by conditioning. Queries are represented explicitly as tokens rather than being hidden in auxiliary modules, providing a unified physical prompting interface. To make this PGM practical and scalable, we formulate it as an autoregressive next token predictor, enabling efficient training and inference on large-scale data. This framework enables the model to integrate knowledge across tasks and modalities and develop coherent understanding of the 3D world. With this design, tasks such novel view synthesis, object manipulation, and depth estimation emerge naturally within the same system as different forms of zero-shot causal inference rather than predefined objectives. Through extensive evaluation, controllability, strong geometric consistency, and robustness in real-world scenarios, outperforming specialized baselines in NVS and 3D object manipulation. Moreover, the model supports flexible geometric reasoning, including joint object manipulation with NVS, complex egocentric navigation, revealing occluded geometry by removing attached objects, and handling depth uncertainty, all of which are required for reliable interaction and navigation in complex real-world 3D environments. These results suggest that 3WM develops strong 3D understanding of the world and opens the possibility of tackling a wide range of 3D vision problems within single framework, taking step toward general purpose visual world models.
find that model achieves precise
Our work makes the following core contributions to unified physical world modeling.
We introduce 3WM, unified physical world model that jointly represents RGB, optical flow, and camera pose within a single generative autoregressive framework with a shared prompting interface.
We propose local random access sequence formulation that allows the model to condition on, query, and update arbitrary spatial regions, enabling flexible inference pathways in which modalities and spatial locations can be decoded in any order within a GPT-style autoregressive transformer.
3WM performs novel view synthesis, 3D object manipulation, and self-supervised depth estimation in zero-shot manner while achieving state-of-the-art performance real- world benchmarks.
The model further supports flexible geometric reasoning capabilities, including composi- tional camera and object motion, amodal completion, and reasoning about depth uncer- tainty.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
Novel View Synthesis (NVS) has been widely studied as a fundamental task in 3D vision. Regression- based methods Yu et al. (202T); Charatan et al. (2024); Kulhánek et al. (2022); Sajj jadi et al. (2022) perform well for view interpolation but yield blurry reconstructions in unobserved regions. This leads to a shift toward generative models, mostly diffusion-based methods, which enable high-quality and diverse NVS. Zero-I-to-3 Liu et al. (2023). trained on large-scale synthetic datasets Deitke et al. 2023); Chang et al. (2015), predicts novel views from a single image us- ing implicit camera conditioning. ZeroNVS Sargent et al. (2024) integrates Zero-1-to-33 approach with a score distillation sampling framework Poole et al. (2022), and extends the application to real-world scenes. Other approaches, such as MotionCtrl Wang al. (202cc), inject camera em- beddings to guide video diffusion without explicit 3D representations. ViewCrafter Yu et al. (2024) utilizes point-cloud rendering using DUS:3R WWan et al. (2024a) for improved performance with better camera motion control. SEVA Zhou et al. (2025) handles diverse novel view synthesis tasks and produces temporally consistent samples and long videos without requiring 3D distillation. In this work, we explore autoregressive sequence modeling for the NVS problem as an alternative to diffusion-based approaches to overcome the limitations of previous works.
3D Object Manipulation While NVS focuses on generating novel views of the input scene, object manipulation refers to the task of transforming objects the scene while keeping the camera fixed. Drag-based image editing methods Wang et al. (2024c); Wu et al. (2024); Shi et al. (2024); Yin
ict al. (2023) aim to solve this problem by parameterizing object transforms as 2D motion vectors which are then used as conditioning to fine-tune stable diffusion (SD) Rombach al. (2022). These
methods can be naturally extended to more complex 3D transforms by incorporating depth infor- mation into the drag vectors Wang al. (2024b). Another class of models Pandey al. (2024); Koo et al. (205)). performs 3D object manipulations by editing input depth maps according to the desired object transform and utilizing a depth-conditioned diffusion model to generate the edited image. However, these methods heavily rely on inverting the input image into the SD latent space, which often fails on real-world images Mokady et al. (2023).
[mages/Mokadyy
General Motion Control Recently, several works have shown that motion-conditioned diffusion models can be used perform sophisticated image manipulations, Geng et al. (2024); ) Koroglu al. conectooy-connitionedd control net on top of large video diffusion model Bar-Tal al.] (2024). The model demonstrates emergent capabilities such as object and camera control and drag- base- image editing and motion transfer. Another set of recent works Gu et al. (2025); Zhang et al. (2024); Feng et al. (2024) uses 3D point trajectories providing more powerful control over image generation. However, they have not shown robust and precise control in real-world scenarios through extensive evaluations. It is also worth noting that approach is in the same spirit as sequential generative modeling frameworks such Bai et al. (2024), which can perform various perception and generation tasks using visual prompts. Our approach goes further by enabling random access for scalability, incorporating non-visual queries such as camera pose, and providing precise controllability for 3D tasks.
22244; Jin et al. (2025) train a
Language- -riven Semantic 3D Understanding Recent multimodal 3D LLMs, such as Inst3D- LMM Yu et al. (2025), Video-3D LLM /Zheng et al. (025), PointLLLM Xu et al. (2024), and L et al. (2024), focus on semantic reasoning over pre-reconstruuted 3D inputs, address- ing tasks including 3D dense captioning, visual grounding, and question answering. These models interpret an already constructed 3D scene rather than inferring physical geometry from images or predicting how future observations change under camera or object motion. Their goals, inputs, and evaluation settings therefore differ fundamentally from our formulation, which centers on learning physical 3D structure and transformations directly from 2D observations.
METHODS
We first introduce 3WM as a probabilistic graphical model V over RGB, optical flow, and camera pose, represented using strictly local HLQ patch codes. We then describe how these variables are serialized into a Local Random Access Sequence (LRAS) of pointer-value tokens, which allows w to learn conditional predictions next present our flexible inference pathways, with optical flow acting as a controllable intermediate representation. Finally, we show how these pathways enable zero-shot prompts for 3D tasks, includ-
arbitrary nodes in the graph. Building this formulation,
3

---

Published as a conference paper at ICLR 2026
Local Quantization
Random Order Sequence
Autoregressive Modeling
Sequence in Pointer-Token Representstion
Input:
Local Ouun
Model
Transformer
Shuffle
Output:
Figure 1: Local random access sequence modeling. Our modeling framework has three key com- ponents: (a) local patch quantizer trained based on a small convolutional autoencoder; (b) a video serialization process based on "pointer-content representation", which allows arbitrary ordering of the patches during training and generation; and (c) an LLM-like autoregressive transformer to predict the contents of the next patch, trained random sequence order.
ing camera-driven flow prediction for depth from motion, dense flow conditioning for novel view synthesis, and object-specific flow fields for 3D object manipulation.
3.1 LEARNING A PROBABILISTIC GRAPIICCLL MODEL VIA LOCAL RANDOM ACCESS SEQUENCE MODELING
We introduce a unified, precisely controllable world model that treats visual data nodes in Probabilistic Graphical Model (PGM) Koller & Friedman (2009). To efficiently implement such model, we formulate
a GPT-style next token predictor through the use of key innova- tions: (i) a local quantizer that preserves strict patch independence, and (ii) pointer tokens that allow random-access encoding/decoding. Together, they let us phrase wide range of geometric tasks as LLM-style prompts without task-specific heads, losses, or datasets-whi while maintaining precise, patch-level control.
Probabilistic Graphical Modeling. We construct learnable PGM (V) over local visual variables (e.g.. RGB or flow patches) and global control variables (e.g., camera pose). Each variable as- signed unique pointer address (spatiotemporal patch index) from the set P and contains value from the discrete codebook V. We model the function V, which takes as input X-the set of vari- ables (pointer-value pairs) observed so far-and an yet unobserved pointer p from the set P, and outputs the distribution over possible values for node from the codebook V. For example, X might contain all patches from the current frame plus a sparse subset from the next frame, while p could be any masked patch in the next frame that V must predict from this partial observation.
V : (X, & dxm(X)) + [Pr[(p, v) [X] VEV
This enables the model to predict the distribution over values at any node and sample values either simultaneously or through autoregressive sampling, effectively populating all nodes in the graph.
Sequence formulation with pointers. To learn these conditionals efficiently, we serialize pointer-structured data as interleaved sequence of pointer and content tokens (Po, ub Pr, u1,...) and train causal autoregressive transformer:
(X, p) = Pr[vk Po, Pb. Uk-1,
Pointer tokens remove raster-order bias and enable random access: the next pointer can pre- dicted provided conditioning. Thus, decoding order itself becomes a controllable traversal of PGM nodes, supporting fully sequential, fully parallel, or hybrid schedules, while keeping a single next-token objective. This recasts learning nigh-dimensional PGM standard GPT-style training on many random traversals.
Strict locality via HI.Q. Instead of compressing an entire frame into global codes, we use a Hier- archical Local Quantizer (HL.)), convolutional autoencoder whose receptive field is restricted to each patch, ensuring strict locality during encoding. Each patch is encoded into short sequence of four codes, where the first provides coarse preview and the remaining codes progressively add fine detail. This property ensures that local interventions such as masking, overwriting, or resam- pling individual patches behave in predictable way. Moreover, the resulting code structure makes the autoregressive modeling objective better aligned with natural language modeling assumptions of conditional independence, enabling tractable factorization and controllable local edits (see Figure [].
4

---

Published as a conference paper at ICLR 2026
Tokenizing Image Flow
Optical as Conditioning: Image Editing
Optical Target: Depth Estimation
HOS
Powe
nGB
Camera
Input
3WM
3WM
Orleet
RGB
Flow
3D WM
3D WM
Figure 2: Flexible inference pathways across modalities. Our framework allows flexibly construct inference pathways for 3D scene understanding. Using optical flow tokens as conditioning, the model performs image editing by generating the next RGB frame. Conversely, when optical flow tokens serve as the prediction target, the model enables depth estimation by predicting the next flow field from a single RGB image and in-plane camera motion input.
We evaluate the contribution of the local random access sequence design through an ablation study provided in Appendix A.4 Further details datasets, filtering, and training of 3WM are provided in Appendix A
3.2 FLEXIBLE INFERENCE PATHWAYS
Optical flow as a control surface. Optical-flow patches serve as a powerful control mechanism within PGM, providing explicit representation of motion that users directly manipulate. We adopt the causal ordering |RGB, - Flow - RGB in our sequences, where flow acts as an intermediate action space: each patch specifies what moves and by how much that spatial location. This design enables users to provide sparse or dense flow constraints to guide generation, with appearance synthesis and disocelusion resolution emerging naturally from the RGB predictions that follow.
With this flexible design, we realize several inference pathways using the same model:
v (RGBo, 1) RGB1: Dense flow-controlled RGB prediction-gih given an input image and a pre-defined dense flow field (motion instructions), render the next frame.
an input image and sparse flow seeds, complete dense flow field. Can be composed with dense flow- controlled RGB prediction to enable object removal and localized edits.
V (RGBo, Fsparse) + Fo Sparse-to-dense flow completion-give
V (RGBo, Cl + Fo-11: Camera-controlles flow prediction given an input image and camera translation, obtain the globally induced flow field. Can be utilized for depth estimation via induced parallax.
Each inference pathway represents a different conditional query V (X, p) over the same underlying joint distribution, where we condition on different subsets of nodes (RGB, flow, or camera) and sample the remaining. The task itself is defined entirely by which nodes we choose to observe versus predict: dense flow conditioning yields motion control, sparse flow yields completion, and camera conditioning yields mumuturr-from-mottinn This PGM formulation eliminates the need for task-specific architectures; instead, diverse capabilities emerge as different paths through the graph. The effectiveness of optical flow as an intermediate representation validated in Appendix A.4
3.3 ZERO-SHOV PROMPTS FOR 3D TASKS
Novel view synthesis Novel view synthesis can be performed with the V (RGBo, Fo-+1) + RGB inference pathway, by conditioning the model on 2D optical flow fields that represent how the pixels move given a desired camera pose change. To generate these flow fields, we use the following steps: a) estimate depth from the input image using off-the-shelf model Yang et al. (2024); Wang et al. (2024a), and unproject it into 3D) point cloud, b) apply a rigid transformation to the point cloud given camera transformations, c) re-project the transformed point cloud and compute the displacement relative to the pixels of the first frame to obtain the 2D flow (See Figure 2 and Appendix A..)). Finally, 3WM generates the edited image conditioned on the computed flow map and the input image.
3D object manipulation 3D object manipulation can be performed with the (RGB,, Fo--1) RGB, inference pathway by creating flow field where the flow on the surface of the object charac-
5

---

Published as a conference paper at ICLR 2026
Input Image
MotionCtrl
ZeroNVS
ViewCrafter
SEVA
3WM
Ground Truth
Figure 3: Novel view synthesis from single image. The results show that our model performs controllable novel view synthesis with various camera motions in diverse scenes. Compared to other models, the reconstructed images do not show abrupt change in object and scene identity.
terizes the 3D transformation to performed, with the flow of the background set to condition- ing the predictor to move the object, but keep the background fixed. We follow similar procedure described above to produce flow fields for rigid object transformations and use the SegmentAny- thing Kirillov et al. (2023) model to suppress the flow of the background regions (See Figure 2 and Appendix A.2). Additionally, we can use the V (RGBo, Fsparse - Fo--1 pathway to produce dense flow fields corresponding to object motion from sparse flow prompt.
Depth extraction Camera--coditioneed flow generation through the (RGBo, Cin-planc) + Fo--1 inference pathway provides natural method for extracting depth maps without additional finetun- ing. We provide in-plane camera motion as input to 3WM and predict the optical flow induced by camera motion. Then, we compute the magnitude of the optical flow compute the disparity which, when inverted, yields 2.5D depth maps. That is, Daept & Fis where Fnow = v (RGB Cin-plane). In practice, we find that simple downward camera translation is sufficient to generate high-quality depth maps. Additionally, performance can be improved by statistical aggregation over disparity maps generated with different seeds for the same image.
RESULTS
We begin by evaluating novel view synthesis from single image, demonstrating that 3WM outper- forms prior NVS models across object-centric and scene-level benchmarks. We then present results on 3D object manipulation, using our 3DEditBench dataset to compare edit fidelity and geometric adherence against diffusion- and drag-based baselines. Lastly, we examine self-supervised depth estimation, showing that 3WM achieves strong performance on both static and dynamic indoor datasets, despite never using depth supervision during training.
4.1 NOVEL VIEW SYNTHESIS
Evaluation Details. We evaluate novel view synthesis (NVS) from single images two out-of- distribution benchmarks: WildRGB-D for object-centiee NVS and DL3DV for scene-level NVS. We also report results the SEVA benchmark for single-image NVS (small-viewpoint, Reconfusion

---

Published as a conference paper at ICLR 2026
Our Evaluations
SEVA Benchmark
Model
WildRGB-D
DL3DV
REIOK
LLFF
DTU
PSNR
LPIPS
PSNR
LPIPS
PSNR t LPIPS
PSNR t LPIPS & PSNR LPIPS !
MotionCtrl
12.39
0.404
12.62
0.462
ZeroNVS
16.14
0.283
15.62
0.331
ViewCrafter
13.96
0.290
16.59
0.253
20.88
0.287
10.53
0.620
12.66
0.485
SEVA 15.10
0.278
11.82
0,516
18.11
0.308
14.03
0.389
14.47
0.316
3WM (Ours)
18.02
0.185
19.02
0.252
21.54
0.231
15.24
0.490
14.63
0.357
Table I: Comparison of metries for novel view synthesis. The left block reports results on WildRGB-D and DL3DV from our evaluation set. The right block presents SEVA benchmark (Zhouu
et al. (2025) performance the small-viewpoint NVS setting using the Reconfusion split across DTU, ILF.. and RE10K datasets.
split). Quantitative metrics include PSNR and LPIPS Zhang et al. (2018). We compare against MotionCtrl, ZeroNVS, ViewCrafter, and SEVA as baselines. Further details dataset selection and implementation are provided in the appendix A.3
Qualitative and Quantitative Comparisons. As shown in Table T our model achieves the best overall performance on WildRGB-D, DL3DV, and the SEVA benchmark, reflecting both recon- struction quality and precise camera control. Qualitatively (Figure 3). MotionCtrl distorts scenes and objects inconsistently, and despite efforts to optimize scene scales, it fails to accurately con- trol camera motion. ZeroNVS often produces inaccurate 3D reconstructions with artifacts, and its hallucinated regions are blurry and unrealistic. ViewCrafter generates visually appealing images but frequently changes object appearance and global illumination. SEVA performs reasonably on WildRGB-D but requires scale sweeps for alignment and breaks down on DL3DV with more diverse camera trajectories. In contrast, 3WM maintains both object and scene identity while offering robust and reliable camera control.
4.2 3D OBJECT MANIPULATION
Model
PSNR
LPIPS 4
EA
Dragnny
15.13
0.443
0.517
Diffusion Handles
17.82
0.344
0.619
LightningDrag
19.52
0.184
0.722
3WM (ours)
22.73
0.133
0.797
Table 2: Comparison metries for 3D object manipulation.
Evaluation Details. We compare method against DiffusionHandle: Pandey al. (022)) the closest related work that performs 3D object edits using copph-conditioned diffusion models, as well as drag-basee based image editing approaches such LightningDrag (Sh et al. (2024) and DragAny- thing Wu et al. (2024). While the latter cannot be directly conditioned on 3D transforms, we adapt them by providing sparse 2D flow vectors from our dataset annotations, which enables them to per- form 3D manipulations to a reasonable extent. For guantitative evaluation, we follow the metrics used in our NVS experiments, including PSNR and LPIPS. However, prior work work Pandey al. (2024) has shown that these metrics tend to favor perceptual image quality over edit accuracy. To address this, we also report the Edit Adherence (EA) metric, which measures how well the boundaries the manipulated object align with the ground truth.
New Object Editing Benchmark. Most prior work in this area either use human evaluations on small set of images Michel et al. (2023). or synthetic benchmarks [Pandey et al. (2024) to evaluate their method. This can be attributed to the lack of high quality real-world datasets with ground-truth 3D object transform annotations. To address this problem, we collect dataset called 3DEdi teech consisting of 100 image pairs with a diverse set of object types undergoing rotations and translations, and inter-object occlusions. We capture these images in a variety of background and lighting conditions. To obtain the ground-trus truth 3D object transformation for a given pair, we annotate four corresponding points in the two images, unproject them, and least-squares opti-

---

Published as a conference paper at ICLR 2026
Input Image
DragAnything
Lightning Drag
Handles
3WM
Ground Truth
Figure 4: 3D object manipulation from single image. We show that model perform both 3D object translation and rotation. Compared to other methods, our model preserves object identity real world images, and produces more photorealistic generated images with accurate object edits.
mization to find the best-fitting rigid transformation that aligns the two sets of points. This transform then used create flow maps that condition 3WM perform 3D object edits in natural scenes.
Qualitative and Quantitative Comparisons. We find that our model outperforms other methods on all metrics (see Table Qualitative comparisons Figure further show that our model pro- duces more accurate and realistic results, particularly
complex 3D transformations. The Edit Adherence (EA) metric Pandey al. (2024), designed to measure how precisely the edited object matches the intended transformation, strongly favors our model's generations. Interestingly, Diffu- sionHandles Pandey et al. (2024) often struggles on real-world images due to failures in the null-text inversion process, which causes changes in surrounding object appearance and leads to unnatural generations, blurry reconstructions, and incorrect 3D) motion. A similar trend is observed in the drag-based baselines, though LightningDrag degrades less severely. In contrast, 3WM overcomes these issues and produces geometrically consistent edits.
4.3
SELF-SUPERVISED DEPTH ESTIMATION
NYUD-v2
BONN
TUM
Model
AbsRel
61
AbsRel
81
AbsRel
81
SC-DepthVV
0.132
0.828
0.172
0.814
0.257
0.582
IndoorDepth
0.116
0.864
0.154
0.846
0.205
0.697
MotionCtrl
0.232
0.664
0.171
0.793
0.205
0.682
SEVA
0.404
0.574
0.352
0.618
0.503
0.496
3WM (Ours)
0.078
0.940
0.084
0.942
0.137
0.869
Table 3: Comparison of metries for self-supervised monocular depth estimation NYUD-v2, BONN, and TUM datasets.
8

---

Published as a conference paper at ICLR 2026
Input Image
SC-DepthV2
IndoarDepth
MotionCIri
SEVA
3WM
Ground Truth
Figure 5: Self-supervised monocular depth estimation. On both static and dynamic scenes, our model outperforms existing self-supervised depth estimation methods. Compared to other genera- tive model basclines, it shows stronger camera motion controllability and geometric understanding. Yellow artifacts ground truth depth maps are noise and excluded during the evaluation.
Since do we compare our model with other self-supervise depth estimators. We evaluate SC-Depth V2 Bian al. (2021), IndoorDepth Fan al. (2023), MotionCurl [Wang al. (2024c), and SEVA Zhou (2025) as baselines, aligning with the self-supervised setting of our model. To extract depth from MotionCtrl and SEVA, we induce downward in-plane camera motion and compute the disparity between the first and generated images using DPFlow Morimitsu et al. (2025). We evaluate the performance on three datasets: NYUv2 Silberman et al. (2012), BONN Palazzolo al. (2019), and TUM Sturm et al. (2012). NYUv2 is mostly composed of static scenes, whereas BONN and include humans with implied motion. To ensure consistent input resolution across methods, we evaluate only the center-cropped regions of each image. Unlike relative depth estimation methods that apply scale-and-shift alignment during evaluation Ranfil et al. (020), we adopt only a global scale adjustment using the median depth value.
Evaluation Details.
any depth label during the training, for fair comparison,
Qualitative and Quantitative Comparisons. As shown in Table 3 and Figure 5 3WM achieves high-quality depth reconstruction in both static and dynamic settings, outperforming all baselines across diverse evaluations. The self-supervised baseline models exhibit limitations in dynamic scenes because they rely on static geometry consistency, preventing them from extracting train- ing signals from moving objects. In contrast, our model effectively learns depth cues from optical flow unconstrained video. Although MotionCtri and SEVA yield qualitative improvements on dynamic objects compared to other baselines, their overall performance remains weak, indicating limited controllability and geometric understanding. These results demonstrate the strong geometric understanding and controllability of our model compared to existing baselines.
5 EMERGENT GEOMETRIC REASONING ABILITIES
Geometric reasoning in 3D scenes is inherently complex, requiring the ability to address scenar- ios that extend beyond any single benchmark task. These include uncovering occluded geometry, navigating cluttered environments, and handling ambiguous depth.
Figure 6 illustrates several representative cases. (a) 3WM can first perform object manipulation to move obstacles aside and then apply novel view synthesis to the updated scene. This sequential rea- soning exposes free space that previously occluded, allowing navigation through newly opened paths. (b) When the occluding object cannot be physically moved. the model should able to simulate navigation along complex trajectories reveal hidden regions. In such case, 3WM can

---

Published as a conference paper at ICLR 2026
(a) Joint Object Manipulation and NVS
(b) Complex Egocentric Navigation
Input Image
Move Obstacie
Mave Forward
Input Image
Complex Motion
Complex Molion
(c) Revealing Occluded Geometry
(d) Handling Depth Uncertainty
Input Image
Remove Cup
Remove Table
Ingut Image
Depth Seed
Depth Seed
Figure 6: Flexible geometrie reasoning in the complex real-world environment through 3WM. (a) The model moves obstacles aside to reveal free space and simulates navigation through the newly opened path. (b) The model follows complex egocentric trajectories to uncover hidden regions and moves together with objects to capture realistic navigation scenarios. (c) The model removes at- tached objects one by one to reveal the background geometry, performing a form of amodal com- pletion. (d) The model handles depth uncertainty in cases such transparent objects by generating multiple plausible depth outputs.
move forward, lower the viewpoint, and rotate the camera to the right to expose a hallway. The model can also support navigation with objects, as in moving with a bike, where the bike remains fixed relative to the egocentric viewpoint while the background updates with the camera motion. (c) To reveal geometry hidden by attached objects, the model applies large flow displacement to the object targeted for removal and repeats this process iteratively. This allows the model to uncover both the background and the previously occluded surfaces of other objects, effectively performing amodal completion.
(d) When conditioned on an in-plane camera motion, the model can generate multiple optical flow maps corresponding different plausible depth configurations. This captures multimodal depth reasoning, which is essential in cases such as transparent or reflective materials where depth cannot be uniquely defined.
These examples demonstrate that our model is not limited to predefined tasks or narrow benchmarks. By supporting flexible inference pathways, can generalize wide range of challenges that arise in real-workd 3D environments. This ability to reason flexibly about geometry highlights the strength of our framework and points toward general-purpee models for 3D understanding.
DISCUSSION
We present several representative failure cases in Fig. [1] These behaviors occur occasionally rather than systematically, and are already reflected in our quantitative evaluation, where our model still outperforms prior methods. The observed failure modes include motion-induced blur under large displacements, residual object copies at the original location, and sensitivity to segmentation quality.
7 CONCLUSION
In this work, we demonstrate that 3WM can unify novel view synthesis, object manipulation, and depth estimation through diverse inference pathways while maintaining precise controllability and strong geometric consistency. By showing that complex 3D reasoning emerge naturally within a unified system. 3WM provides a step toward general-purpos. visual world models capable of supporting richer interaction with the physical environment.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the following awards: To D.L.K.Y.: Simons Foundation grant 543061, National Science Foundation CAREER grant 1844724, National Science Foundation Grant NCS- 2123963, Office of Naval Research grant S5122, ONR MURI00010S02, ONR MURI S5847, and ONR MURI I 1413866 493027. We also thank the Stanford HAL, Stanford Data Sciences and the Marlowe team, and the Google TPU Research Cloud team for computing support.
REFERENCES
Yutong Bai, Xinyang Geng, Kartrikeya Mangalam, Amir Bar, Alan I. Yuille, Trevor Darrell, Jitendra Malik, and Alexei Efros. Sequential modeling enables scalable learning for large vision models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22861-2283 2024.
Omer Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Guanghui Liu, Amit Raj, et al. Lumiere: A space-time diffusion model for video generation. In SIGGRAPH Asia 2024 Conference Papers, pp. 1-11, 2024.
Daniel M Bear, Kevin Feigelis, Honglin Chen, Wanhee Lee, Rahul Venkatesh, Klemen Kotar, Alex Durango, and Daniel LK Yamins. Unifying (machine) vision via counterfactual world modeling. arXiv preprint arXiv:2306.0/828. 2023.
Jia- Wang Bian, Huangying Zhan, Naiyan Wang, Tat-Jun Chin, Chunhua Shen, and Ian Reid. Auto- rectify network for unsupervised indoor depth estimation. IEEE transactions on pattern analysis and machine intelligence. 44(12):9802-981) 2021.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, al. Shapenet: An information-ries 3d model repository. arXiv preprint arXiv: 15/2.33012 2015.
David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaus- sian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 19457-19467, 2024.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 13142-13153 2023.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE conference on computer vision and pattern recogni- tion, pp. 12873-12883, 2021.
Chao Fan, Zhenyu Yin, Yue Li, and Feiqing Zhang. Deeper into self-supervised monocular indoor depth estimation. arXiv preprint arXiv:2312.01283, 2023.
Wanquan Feng, Tianhao Qi, Jiawei Liu, Mingzhen Sun, Pengqi Tu, Tianxiang Ma, Fei Dai, Songtao Zhao, Siyu Zhou, and Qian He. 12vcontrol: Disentangled and unified video motion synthesis control. arXiv preprini arXiv:2411.17765 2024.
Rao Fu, Jingyu Liu, Xilun Chen, Yixin Nie, and Wenhan Xiong. Scene-llm: Extending language model for 3d visual understanding and reasoning. arXiv preprint arXiv:2403. 11401, 2024.
Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez-Guevare Carl Doersch, Yusuf Aytar, Michael Rubinstein, al. Motion prompting: Con- trolling video generation with motion trajectories. arXiv preprint arXiv:2412.02700. 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv: 1783, 2024.

---

Published as a conference paper at ICLR 2026
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 19383-1940) 2024.
Zekai Gu. Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, et al. Diffusion as shader: 3d-aware video diffusion for versatile video generation control. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers, pp. 1-12, 2025.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000-16009, 2022.
Wonjoon Jin, Qi Dai, Chong Luo, Seung-Hwan Back, and Sunghyun Cho. Flovd: Optical flow meets video diffusion model for enhanced camera-controlled video synthesis. arXiv preprint 00X1:22502.08244, 2025.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 10Xx::1005.06950, 2017.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceed- ings of the IEEE/CVF international conference on computer vision, pp. 4015-4026, 2023,
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Juil Koo, Paul Guerrero, Chun-Hae Hao P Huang. Duygu Ceylan, and Minhyuk Sung. Videohandles: Editing 3d object compositions in videos using video generative priors. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 17692-17701, 2025.
Mathis Koroglu, Hugo Caselles-Dupré, Guillaume Jeanneret Sanmiguel, and Matthieu Cord. On- lyflow: Optical flow based motion conditioning for video diffusion models. arXiv preprint arXiv:2411.10501, 2024.
Jonáš Kulhánek, Erik Demerr, Torsten Sattler, and Robert Babuška. Viewformer: Nerf-free neural rendering from few images using transformers. In European Conference on Computer Vision, pp. 198- 216. Springer, 2022.
Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. D13dv-10k: A large-scale scene dataset for deep learning-basse 3d vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22160-22169, 2024.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2023.
Oscar Michel, Anand Bhattad, Eli VanderBilt, Ranjay Krishna, Aniruddha Kembhavi, and Tanmay Gupta. Object 3dit: Language-guided 3d-aware image editing. Advances in Neural Information Processing Systems, 36:3497-3516, 2023.
Ron Mokady, Amir Hertz, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6038-6047, 2023.
Henrique Morimitsu, Xiaobin Zhu, Roberto M Cesar, Xiangyang Ji, and Xu-Cheng Yin. Dpflow: Adaptive optical flow estimation with a -yraamdd framework. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.
2025.
12

---

Published as a conference paper at ICLR 2026
Emanuele Palazzolo, Jens Behley, Philipp Lottes, Philippe Giguere, and Cyrill Stachniss. Refu- sion: 3d reconstruction in dynamic environments for rgb-d cameras exploiting residuals. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7855-7862. IEEE, 2019.
Karran Pandey, Paul Guerrero, Matheus Gadelha, Yannick Hold-Geoffroy, Karan Singh, and Niloy J Mitra. Diffusion handles enabling 3d edits for diffusion models by lifting activations to 3d. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7695-
7704, 2024.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. /4988, 2022.
Alec Radford, Jong Wook Kim. Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-87 8763. PmLR, 2021.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transac- tions on pattern analysis and machine intelligence, 44(3):1623-1637, 2020.
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Shordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d cat- egory reconstruction. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10901-10911, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp.
2022.
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, Urs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation trans- former: Geometry-free free novel view synthesis through set-latent scene representations. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 222996233, 2022.
Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Hermmann Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, et al. Zeronvs: Zero-shot 360-degree view synthesis from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9420-9429 2024.
Yujun Shi, Jun Hao Liew, Hanshu Yan, Vincent YF Tan, and Jiashi Feng. Lightningdrag: Lightning fast and accurate drag-based image editing emerging from videos. arXiv preprint arXiv:2405. 13722, 2024.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In Computer Eision-ECCH 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pp. 746-760. Springer, 2012.
Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. bench- mark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 573-580. IEEE, 2012.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. Advances in neural information process- ing systems, 15:10078-1009,, 2022.
Shuzhe Wang. Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Ge- ometric 3d vision made easy. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20697-207000 2024a.
13

---

Published as a conference paper at ICLR 2026
Zhouxia Wang, Yushi Lan, Shangchen Zhou, and Chen Change Loy. Objetrl-2.5 d: Training-free object control with camera poses. arXiv preprint arXiv:2412.07721, 2024b.
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024c.
Weijia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation. In European Conference on Computer Vision, pp. 331-348. Springer, 2024.
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. In European Conference on Computer Vision, pp. 131-147. Springer, 2024.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. Advances in Neural Information Processing Systems, 37:21875-21911 2024.
Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high- fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12 12-22, 2023.
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Drag- nuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint ar arXiv:2308.0800 2023.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4578-4587, 2021.
Hanxun Yu, Wentong Li, Song Wang, Junbo Chen, and Jianke Zhu. Inst3d-lmm: Instance-aware 3d scene understanding with multi-modal instruction tuning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14147-14157, 2025.
Wangho Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien- Tsin Wong, Ying Shan, and Yonghong Tian. Viewcrafter: Taming video diffusion models for high-fidelity novel view synthesis. arXiv preprint wwXiv:2409.02044 2024.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, et al. Mvimgnet: A large-scale dataset of multi-view images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9150-9161, 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3836-3847, 2023.
Qihang Zhang, Shuangfei Zhai, Miguel Angel Bautista. Kevin Miao, Alexander Toshev, Joshua Susskind, and Jiatao Gu. World-consistent video diffusion with explicit 3d modeling. arXiv preprint arXiv:2412.01821, 2024.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effecctivnness of deep features perceptual metric. In Proceedings the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018.
Yuchen Zhang, Nikhil Keetha, Chenwei Lyu, Bhuvan Jhamb, Yutian Chen, Yuheng Qiu, Jay Karhade, Shreyas Jha, Yaoyu Hu, Deva Ramanan, et al. Ufm: A simple path towards unified dense correspondence with flow, arXiv preprint arXiv:2506.0977 09278, 2025.
Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d 3d Ilm: Learning nosition-awart video repre- sentation for 3d scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8995-9006, 2025.
14

---

Published as a conference paper at ICLR 2026
Jensen Jinghao Zhou, Hang Gao, Vikram Voleti, Aaryaman Vasishta, Chun-Han Yao, Mark Boss, Philip Torr, Christian Rupprecht, and Varun Jampani. Stable virtual camera: Generative view synthesis with diffusion models. arXiv preprint arXiv:2503.14489, 2025.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint 1805.088800098/11 2018.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DATASETS AND TRAINING DETAILS
Training Datasets. 3WM was pre-trained on combination of large-scale internet video collec- tion, termed BVD (Big Video Dataset), and several 3D vision benchmarks. The latter include the train splits of ScanNet++ Yeshwanth et al. (2023), CO3D Reizenstein al. (202T). RealEstate 10K Zhoou
et al.
2018), MVImgNer Yu et al. (023)) DL3DV Ling et al. (2024). and EgoExo4D Grauman
al. 2024). Optical flow was computed for all videos using DPFlow Morimitsu et al. (2255 and
UFM Zhang et al. (2025).
Big Video Dataset. The BVD consists of roughly 7,000 hours of internet videos automatically crawled using search queries generated by LLaMA 3 Grattafioni et al.] (2024). The queries targeted videos containing rich physical dynamics, diverse environments, and varied objects. Specifically, action categories from Kinetics400 Kay et al. (2017) were expanded with additional sports, physical activities, and product review categories. To ensure training relevance, we filtered videos by requir- ing minimum level of optical flow and by applying CLIP Radford et al. (202T) keyword alignment. Positive keywords included action, activity, motion, and place, while negative keywords included animation, cartoon, face, game menu, graphic, map, newscast, person, and screenshot. Alignment was quantified by the dot product between CLJP embeddings of keywords and video frames.
Training Details. The 3WM sequence model was trained autoregressively using cross-entropy loss on next-token prediction with a batch size of 512 and a sequence length of 4,096. Training for RGB and camera pose tokens ran for 500K steps. The learning rate was linearly warmed up over 2K iterations 3 x 10-4 held constant until 500K steps. Training was then continued for an additional 200K steps with optical flow tokens, where the learning rate stayed at 3 x 10 initially and decayed linearly to zero during the last 100K steps.
The RGB variant of HLQ was trained on combination of ImageNet and OpenImages with a batch size of 512 for 200K iterations. The objective combined an & reconstruction loss, a low-resolution loss, and a Dino V2-based perceptual loss. Optimization was performed using AdamW with a learn- ing rate of x 10-4, including 2K warmup steps followed by 198K steps of decay to zero.
The flow variant of HLQ was trained on the same dataset used for training the sequence model, with a batch size of 512. Optical flow was extracted from video data using DPFlow. Optimization was again performed using AdamW with learning rate of 1 10 Training used 2K warmup steps, followed by 300K iterations with a fixed learning rate and then 200K iterations with linear decay.
Training Sequence. Every training example is formed as a causal token sequence in which all previous tokens serve as observed context and the next token is supervised, identical to standard language- model training. In practice, all RGB and flow tokens after the first frame predicted autoregressively. The only exceptions are the RGB, tokens, camera-pose tokens, and pointer tokens, which we do not supervise in this work, though the framework is not limited to this choice.
Within RGB+flow sequences, the model naturally encounters several supervision patterns depending on whether flow is the final target or an intermediate variable used to produce the next RGB frame. Some sequences terminate with flow (e.g., RGB, + Flowo-11 or RGB, + RGB, + Flow| 1-22), while others place flow mid-sequence and continue autoregressively to the next RGB frame (e.g.. RGB, + Flowp=11 + RGB, or RGB, + RGB, + Flow1-22 + RGB2). When available, pose tokens are inserted between RGB frames or immediately before flow. These variations provide bal- anced supervision across modalities and naturally expose the model to different geometric reasoning configurations.
The curriculum over training steps controls which of these sequence types the model sees. Before 500k steps, training uses RGB-only sequences drawn from 2-4 consecutive frames, where each frame is partially masked according to fixed visible-token ratios ((0.5, 0.3] for 2-frame sequences, [0.5, 0.15, 0.15] for 3-frame sequences, and [0.5, 0.1. 0.1, for 4-ramme sequences). After 500k steps, we introduce flow and train mixture of RGB-onl -olly and RGB+flow sequences, with random allocation of tokens across frames, modalities, and time indices. This particular schedule simply re- flects reasonable choice rather than an optimized design. The same autoregressive formulation
16

---

Published as a conference paper at ICLR 2026
supports arbitrary modality mixtures, token allocations, and sequence orderings, and exploring al- ternative training schedules and sequence constructions would be an interesting direction.
A.2 3D EDITING METHODS
Novel View Synthesis
Conditonin Floe
Novel View
Object Manipulation
Conditooing Flow
Ofject
3WM
warp depth map
3D object motion
Conditining Flow
Navall Viers 2
Conditoning Now
Eltpect Meton
Diee O
Dipthkophhingd
SWM
Dopti
Figure 7: 3D scene editing through optical flow field manipulation: We perform 3D scene edits by constructing optical flow fields corresponding to the desired transformations either camera or object motion in 3D.
We illustrate the procedure for constructing flow fields used both novel view synthesis and 3D object manipulation. For novel view synthesis (Section 333), the flow encodes pixel displacements induced by a desired camera motion. For object manipulation (Section (.3). the flow specifies the motion of object surfaces under 3D transformation while background regions remain fixed.
eigurreproovine a unified visualization of these processes, complementing the method descriptions in the corresponding subsections.
A.3 NVS EVALUATION DETAILS
To ensure a fair evaluation of novel view synthesis (NVS) on out-of-distribation datasets, we se- lected two benchmarks: WildRGB-D for object-centric NVS and DL3DV for scene-level NVS. For WildRGB-D, we randomly sampled 100 scenes from its evaluation split. For DL3DV, since some models were trained on this dataset, we selected 100 scenes from its recently released 1 11K subset, which, to the best of our knowledge, was not used to train any of the compared models. From each video, we extracted 25-frame sequence and used the first frame as the input image and evalu- ated the generated frames. For quantitative evaluation, we measured PSNR and LPIPS Zhang et al. 2018). As baselines, we compared against MotionCtrl, ZeroNVS, ViewCrafter, and SEVA.
To evaluate novel view synthesis, we compared generated images to ground-truth real- world images using known camera poses. While camera rotation unambiguous, camera translation may have arbitrary scale. Therefore, it was necessary to find the right scene scale to perform fair evaluations for all the models. To align MotionCtrl, ZeroNVS, and SEVA results with ground-truth images, swept range of scene scales and took the generated trajectories with the best median LPIPS score across frames. For ZeroNVS, we swept scales in the range 0.1 to 10. multiplying the seale by the ground-truth camera trans lanslations from each evaluation scene. ZeroNVS introduces malization scheme Sargent et al. (2024) at training time to address this scale ambiguity, but does apply inference. For MotionCtrl, we swept the range to 10, smaller translation scales empirically weaken the camera conditioning and lead to incorrect camera pose trajectories. For SEVA, we swept the scale from 0.1 to 2.0 and used their set NVS method for evaluation. Scale alignment for these models often failed for samples with especially poor 3D reconstruction qual- ity. For ViewCrafter, we resolved the scene scale using their method of aligning point clouds from DUS(3R Wang et al. (2024a). For 3WM, we computed single scale value per scene by matching the optical flow computed from the video using DPFlow and the 2D flow computed using the depth from DepthAnything
DUS(3R and relative camera pose changes.
Since ViewCrafter operates on wide rectangular videos, we adapted the input images accordingly. For DL3DV, which consists of wide images, provided the full image to ViewCrafter. For WildRGB-D, whose images have greater height than width, we cropped them into wide rectangu- lar regions match ViewCrafter's aspect ratio. All other models received center-cropped square image as input for both datasets. All evaluation metrics were computed only on the overlapping regions. For WildRGB-D, this region rectangular, and for DL33D it square.
17

---

Published as a conference paper at ICLR 2026
A.4 ABLATION STUDIES
We conduct several ablation studies to validate two key design choices of our framework: the use of local random access sequence modeling and the use of optical flow as an intermediate representation.
Local random access sequence modeling, We compare different tokenization and sequence mod- cling strategies using 1orampareeetrr models in Table 4 Our approach with local random
quence shows clear benefits over the raster order approach. As observed in MAE He et al. (2022), VideoMAE Tong et al. (2022). and CWM Bear al. (023), random masking encourages stronger representation learning while allowing us to represent each frame with fewer tokens. In contrast, raster order models must encode all patches sequentially, leading to inefficiency and degraded per- formance. Moreover, we find that local tokens provide finer control over scene elements than global token alternatives Esser et al. (202T); Rombach et al. (2022), improving both controllability and out- put quality. Together, these results demonstrate that local random access sequence design not only improves training efficiency but also enhances the model's ability to support robust controllability in flexible inference pathways.
WildRGB-D: Novel View Synthesis
Model (100M)
PSNR f
SSIM t
LPIPS 4
Local & Random
17.28
0.530
0.236
Local & Raster
15.00
0.459
0.385
VQGAN & Random
17.16
0.515
0.238
VQGAN Raster
15.71
0.454
0.298
Table 4: Advantage of local random access sequence modeling. Comparison of 100M models with different tokenizers and sequence strategies shows the benefit of random access. Local tokens further improve controllability of the scene, yielding the best overall performance.
Optical flow as causal intermediate. Table 5 shows that using optical flow control signal substantially improves performance across tasks. Flow provides direct handle on scene geometry. enabling more precise inference than camera-only control, which suffers from scale ambiguity. This advantage is most apparent in depth estimation, where predicting flow directly yields more geomet- rically consistent reconstructions than first predicting RGB and deriving flow afterward from the input image and predicted image. These results support our claim that intermediate modalities such as flow enable more controllable and robust inference pathways, grounding the model in physical structure.
WildRGB-D: Novel View Synthesis
NYU Depth Estimation
Model
PSNR
SSIM
LPIPS
Dataset
AbsRel
Log10 + 8
3WM,gb
14.49
0.389
0.346
3WMrgb
0.173
0,064
0.825
3WM
18.02
0.555
0.185
3WM
0.078
0.033
0.940
Table 5: Advantage of optical flow for causal inference. The results highlight the advantage of optical flow as a control signal. Optical flow provides a direct handle on scene geometry, enabling more precise inference than camera-only control, which is affected by scale ambiguity. This benefit also reflected depth estimation, where predicting flow directly outperforms deriving it from predicted image.
A.5 LIGHTING AND APPEARANCE UNDERSTANDING
Our model does not capture lighting effects perfectly across all conditions. At the same time, consistently observe many cases where the model correctly predicts shadows and view-dependent lighting changes, indicating that it leamns meaningful aspects of object appearance (Figure 8). We do not regard the remaining failure cases in challenging lighting as fundamental limitations of the formulation. Rather, they reflect the current scale of the model and the limited diversity of illumina-
18

---

Published as a conference paper at ICLR 2026
(a) Lighting on the Object
(b) Shadow object
Input Image
Prediction
Ground Truth
Input Image
Prediction
Ground Truth
Figure 8: Lighting and appearance understanding. Additional qualitative examples illustrating the model's handling of lighting and appearance. In case (a), specular highlights on objects change appropriately the object moves, and in case (b), cast shadows shift consistently with the object's motion. While some examples still show incomplete specular or shading behavior, many exhibit correct reasoning about shadows and view-dependent appearance. These results suggest that lighting fidelity is primarily constrained by model and data scale rather than by limitations of the approach.
tion conditions in the training data. We expect lighting fidelity to improve as the model is scaled up and trained on datasets with broader lighting variation.
A.6 EXTENDED DISCUSSION ON OBJECT MANIPULATION
(a) Deformable Object
(b) Compositionality
Input Image
Input Flow
Prediction
Inout Image
Input Flow
Prediction
Figure 9: Additional qualitative examples of object manipulation. (a) The model performs deformable-object manipulation by applying flow conditioning only to the deforming regions, which allows it to infer the resulting non-rigid transformations. In the figure, the red circles indicate regions where non-zero flow applied on the object. Other regions are left unspecified, requiring the model to infer their behavior from the local conditioning.
(b) In multi-object scenes, the model achieves selective manipulation by applying flow conditioning to a segmented target object while assigning zero flow to the rest of the scene.
The examples in the main paper focus on rigid object manipulation, reflecting the fact that 3DEditBence primarily contains rigid transformations. However, our framework rigid object manipulation. Because the model operates on local tokens and the conditioning interface supports both sparse and dense flow control, can naturally handle deformable object manipulation. By applying flow conditioning only to the deforming region and leaving the rest of the object un- constrained, the model infers the resulting non-rigid transformation. We additionally set the flow to zero in the four corners of the image to signal that the transformation is local rather than global. We have added qualitative examples that illustrate this behavior in Figure g Finally, we show exam- ples with and without stop patches the blanket to illustrate how these signals affect the model's probabilistic behavior Figure I0
limited to
For stacked multi-object scenes, include additional examples in Figure where the model manipulates a single object while preserving the geometry and appearance the others. these cases, segment the target object and apply full flow conditioning that object, while the rest
19

---

Published as a conference paper at ICLR 2026
of the scene receives flow in the conditioning. These results show that the model selec- tively operate on one object among several and maintain coherent scene structure throughout the manipulation.
Input Image
Preciction
Prediction
Prediction 3
Input Image
Prediction 1 Prediction
Prediction 3
Figure 10: Effect of stop patches on probabilistic deformation. Left: predictions from three random seeds using only the motion patch (red). Right: predictions using both the motion patch (red) and stop patches (blue). Yellow arrow indicates the model's predicted deformation. Without stop patches, the model exhibits uncertainty in how the rest of the blanket moves. Stop patches constrain the motion and lead more stable outcomes.
A.7 AMODAL COMPLETION EVALUATION
Model
AbsRel +
Log10 4
81 f
DragAnything
0.0666
0.0285
0.9480
Diffusion Handles
0.0500
0.0215
0.9680
Lightning Drag
0.0324
0.0145
0.9782
3WM (Ours)
0.0263
0.0120
0,9740
Table 6: Amodal completion depth evaluation 3DEditBench. Metrics computed on the region originally occupied by the moved object.
Here, we show quantitative amodal completion evaluation. Since all baseline editing methods in the paper can perform amodal completion by moving objects. we assess how accurately each method reconstructs the depth behind the removed object on 3DEditBench. We segment the region previously occupied by the object and compare the predicted depth in the edited image with the ground-truth depth of the target view, restricted to this occluded region. Depth maps are estimated using DepthAnything v2 and aligned by median-based scale normalization.
This metric directly measures model's ability to infer the hidden geometry. As shown in Table 6 our model achieves lower AbsRel and Log 10 errors than all baselines while maintaining competitive 51, indicating more accurate reconstruction of the unseen surfaces.
A.8 LIMITATIONS & FUTURE WORK
(b) Duplicated Object Segmentation
(a) Motion Blur
Input Image
Prediction
Ground Truth
Input Image
Segmentation
Prediction
Ground Truth
(c) Wrong Segmentation
Input Image
Segmentation
Prediction
Ground Truth
Figure 11: Additional qualitative examples illustrating current limitations. (a) Motion blur: Because the model trained on real videos, sometimes reproduces motion-induced blur when large object displacements but may be undesirable for fine- iine-grained manipulation. occasionally generate duplicated copy at the original location. (c) Segmentation errors: For rigid- object manipulation, incorrect input segmentation leads to incorrect zero-flow constraints, causing unpredictable or distorted outcomes.
present. This behavior is consistent with the training distribution
(b) Object duplication: The model may
20

---

Published as a conference paper at ICLR 2026
Limitations As shown in Fig. our method exhibits three failure modes: motion-induced blur under large displacements, residual object copies at the original location, and sensitivity to segmen- tation quality. These occur occasionally rather than systematically and are already captured in quantitative results. Beyond visual artifacts, efficiency remains a practical constraint. Our runtime falls within the typical range for generative models but is not real-time, which limits use in interac- tive applications.
Future Work We expect efficiency improvements from standard engineering optimizations widely used in large autoregressive models. Another promising direction is to evaluate the model's s navigation and planning capabilities. We believe that exploring these directions would highly valuable and represents natural next steps for unified physical world models.
A.9 USE OF LARGE LANGUAGE MODELS (LLMS)
We used LLMs as a general-purpose assistance tool for grammar checking and for suggesting al- ternative expressions or synonyms. LLMs did not play role in developing the research ideas, designing experiments, or writing substantive content.
21


Reviews:
[{"review_id": "F42ndFB7B7", "reviewer": "Reviewer_eRy7", "summary": "The paper presents an autoregressive generative model (3WM) for RGB images and optical flow based on an interesting \"pointer-content representation\" that allows a single autoregressive model to generate image or optimal flow output from sparse or dense image or optical flow input. Effectively, the paper trains a single model to learn to approximate a diversity of types of conditional distributions on image and optical flow tokens, via randomization of the training set. The paper demonstrates the resulting model on several tasks including novel-view synthesis, 3D object manipulation, and depth extraction.", "questions": "1. The specific way that training examples were generated from the video dataset seems central to the model. It would help to provide more detail on the type of (latent, observed) pairs that were generated synthetically from the video dataset, and the distribution of these different types of examples at training time?\n2. Since the autoregressive model represents in principle all possible conditional distributions, it would be interesting to study how self-coherent it is. It would be interesting to compare (i) the approximate amortized inferences obtained by sampling from the trained autoregressive model, with (ii) the idealized conditional distribution that is defined by sampling from the unconditioned joint distribution and performing rejection sampling. One way to do this is via estimating expected symmetrized KL divergence on synthetic data generated from the model itself as in \"AIDE: An algorithm for measuring the accuracy of probabilistic inference algorithms\" to compare rejection sampling versus the variational approximation learned by the model (this essentially boils down to sampling from the joint model, then evaluating the conditional probability of the latent tokens given the observed tokens under the autoregressive model). Are there certain classes of queries where it is more self-consistent than others? Can you use the joint model to do a little model-based Monte Carlo (importance sampling) to improve upon proposals sampled from the autoregressive amortized approximation of the conditional distribution?\n3.  Interesting connection: The \"pointer-content\" representation doesn't fit well with standard graphical models, but is more closely related to the address-value representation used in the Gen probabilistic programming language (see e.g. section 4.1 of Gen: A General-Purpose Probabilistic Programming System with Programmable Inference) where random choices are assigned labels (\"addresses\") and generative models are probability distribution on dictionaries that map addresses to values.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The pointer-content representation that allows for a single model to approximate a number of conditional generation tasks on the joint space of (RGB image, optical flow, RGB image) is significant. The approach of training a model to approximate diverse types of conditional distributions via randomly selecting and re-reorder \"pointer-content\" pairs is a useful contribution. The qualitative comparisons to baselines shown in the paper are compelling.", "weaknesses": "The use of \"graphical model\" as the formal framework for describing the work seems somewhat ill-suited, since there are no conditional independences in the model (and not surprisingly, the the paper includes no graphical models visualized). See \"Questions\" for connections to other formal frameworks for generative modeling that seem more aligned with the \"pointer-content\" representation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KMFLgOOLzY", "reviewer": "Reviewer_AqTG", "summary": "The proposed physical world model aims to enable 3D scene understanding enabling improvements in visual reasoning tasks. \nThe method proposes a probabilistic graphical model operating on scene information such as RGB patches, optical flow, and depth maps, implemented as a 7B autoregressive transformer performing next-token prediction. The model is trained on Big video dataset and 3D vision benchmarks such as Co3D, ScanNet++, and others. The model enables novel-view synthesis, 3D object manipulation, and depth estimation. The paper presents strong results for each of the tasks on standard test benchmarks demonstrating improvements over existing methods. The authors also contribute a new dataset of 100 image pairs for 3D object manipulation which could be useful for future research.", "questions": "- Can the model perform transformation of non-rigid objects such as cloth? If so, how can the transformations be performed?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-motivated and proposes a simple yet effective method enabling strong results for Novel-view synthesis, 3D object manipulation, and depth estimation tasks. The paper expresses the ideas with enough detail for understanding each component and its implementation. The evaluation is comprehensive with existing methods on standard benchmarks for the respective tasks. Composable inference pathways for 3D scene understanding as shown in Figure 6 is a strong result and could be useful for downstream robotics tasks. This can be used for complex reasoning and planning in 3D scenes.", "weaknesses": "- The results do not show strong understanding of lighting and appearance understanding based on the results in Figure 3 (the specular highlights on the objects) and 4 (the macbook), as the objects seem to have baked in shading.\n- The object manipulations are limited to rigid transformations and do not show compositional understanding of objects such as stacked objects. The demonstrated examples are limited to the new dataset and would require more evidence to support the claim of improved 3D object manipulation capabilities.\n- The paper should include a discussion on the limitations of the model to better inform the reader about the scope of the contributions.\nSuggestions for improvement of the presentation:\n- I recommend the authors to include a short introduction to each section as an overview of the content.\n- Introduction can include a concise summary of the contributions of the paper.\n\nMinor:\n- L115: an video --> a video", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mI6YCwLNw8", "reviewer": "Reviewer_Dr8R", "summary": "This paper aims to propose a method to unify several important tasks of 3D scene understanding. To this end, the authors propose a probabilistic graphical model, where nodes represent multimodal scene elements such as RGB, optical flow, and camera pose. The combinations of these nodes therefore enables different tasks, including novel view synthesis, 3D object manipulation and depth estimation without task-specific training. Experiments show that the unified model outperforms the respective task-specific baselines. The authors also discover that the model supports composable inference pathways, therefore can help other tasks such as navigation and amodal completion.", "questions": "Generally I like the paper. I would recommend the authors to address my concerns in the weakness section during the rebuttal.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novelty of the problem. I like the story that the authors want to develop a unified model for different 3D understanding tasks.\n\n2. Novelty of the method. The method to unify the different 3D understanding tasks is also novel and interesting to me.\n\n3. Good evaluation results. Experiment results show the developed unified model generally outperforms the task specific models for novel view synthesis, 3D object manipulation and depth estimation, although it has not been trained for the specific tasks. The results are impressive to me.", "weaknesses": "1. Efficiency. It is good to see the authors have developed such a unified model with good performance. It would be better if the authors can also show the efficiency of their model for different tasks compared with the respective task-specific baselines. If the model are going to be applied to embodied AI systems for navigation, for example, the efficiency does matter.\n\n2. Application scenarios. In section 5 of the paper, the authors discuss about the emergent geometric reasoning abilities of the model, and have provided several qualitative examples. However, it increases the paper's contribution and significance if the authors can provide quantitative results on the tasks, such as navigation, and amodal completion, as they mentioned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4PDNfv2SlR", "reviewer": "Reviewer_WZBe", "summary": "This paper proposes 3WM (3D World Model), a unified framework for 3D scene understanding and interaction based on a probabilistic graphical model (PGM) formulation. Unlike task-specific 3D models, 3WM encodes multimodal scene elements (RGB, optical flow, camera pose) as nodes and performs diverse tasks—such as novel view synthesis, depth estimation, and 3D object manipulation—as different inference pathways through the same graph.", "questions": "I am not very familiar with the research areas of Novel View Synthesis and World Model, so my current evaluation mainly focuses on the presentation quality and writing clarity of the paper. I will update my assessment after considering feedback and technical insights from other reviewers.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces a conceptually elegant unification of multiple 3D vision tasks under one probabilistic and autoregressive modeling framework.\n2. The local random-access sequence modeling design and pointer-token formulation are interesting and potentially valuable for scalable multimodal 3D reasoning.\n3. Both quantitative and qualitative experiments have been conducted in sufficient detail.", "weaknesses": "1. The presentation quality is inconsistent. The Introduction section fails to clearly and explicitly highlight the main contributions of the paper, making it difficult for readers to grasp the motivation and significance of the work.\n2. A teaser figure should be included in the Introduction to provide an intuitive overview of the proposed framework and help readers quickly understand the key idea and workflow.\n3. The paper lacks comprehensive ablation studies, which are necessary to fully evaluate the generalization ability and effectiveness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)